diff --git a/.github/workflows/book.yml b/.github/workflows/book.yml index 37d2da7..e940bb2 100644 --- a/.github/workflows/book.yml +++ b/.github/workflows/book.yml @@ -23,6 +23,22 @@ concurrency: cancel-in-progress: true jobs: + # The reading guides' depth rules (CLAUDE.md § Reading-guide depth) have a + # mechanical part — each step declaring its input and output, a collapsed + # answer under every checklist item, a line-number gutter on every quoted + # snippet. Across 230 guides those survive only if a script enforces them. + # `--all` drops the ratchet the rollout ran behind: every guide is converted, + # so a file that does not follow the rules is a new one that skipped them + # rather than one the rollout has not reached. Nothing here needs a + # toolchain, so it runs first and fast. + depth: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Reading guides follow the depth rules + run: python3 tools/check-reading-depth.py --check --all + build: runs-on: ubuntu-latest steps: diff --git a/.gitignore b/.gitignore index acbd3c6..5c89f22 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ target/ /book/ /mermaid.min.js /mermaid-init.js +/.cache/ diff --git a/CLAUDE.md b/CLAUDE.md index 8c44b8f..4abca0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,6 +21,23 @@ A self-paced database-internals learning path, rendered as an mdBook (`book.toml - **Generators are seeded**, so every figure reproduces exactly apart from timings. Lockfiles are committed for the same reason. - **Exercise lanes must degrade, not crash.** A bench binary on a fresh clone prints its provided lanes and a `[stub — ...]` note for the rest, and exits 0. Never let a `todo!()` panic hide a measurement above it. +## Reading-guide depth + +The `reading-*.md` chapters teach from zero: a reader who knows systems but not the chapter's theory must be able to finish without leaving the page. `topics/00-performance-toolbox/reading-criterion.md` is the reference implementation of these rules; match it. + +- **Define every term at first use.** A term of art (t-test, p-value, quartile, IQR, MAD, standard error, null hypothesis, arithmetic intensity, coordinated omission) gets a **bold** name and a one-sentence plain-language definition at the point it first appears, *before* any argument leans on it. A step may use only terms defined in an earlier step or defined on the spot. Borrowed jargon — using a word the guide never defined because the source material used it — is the failure this rule exists to stop. +- **Every step declares its input and output.** Each `### Step N` opens with a `> **In:** … **Out:** …` blockquote naming the dataset it consumes, *which earlier step produced it*, and what it emits. When one stage forks into two datasets used by different downstream steps, the fork gets its own numbered step. "Is this the same data as the previous section?" must never be left to the reader to infer. +- **A formula gets its symbols named and one worked example.** Quote it as the source actually computes it, name every symbol, then run it once on 3–5 concrete numbers so a real answer comes out. Arithmetic printed in a guide is verified like any other number in this repo — compute it, don't estimate it. +- **Anchors are verified against the pinned clone, file *and* line.** Citing the right line of the wrong file is the same error class as inventing a number. Re-grep every anchor before committing; state the version the line numbers belong to. +- **A quoted snippet carries the line numbers it actually occupies, and names the one to look at.** Put the real number in the gutter of every line, mark elided ranges (`// ... 131–139: bookkeeping ...`) rather than silently closing a gap, and say in the prose which line carries the argument ("the line to focus on is 277, its only `return`"). A snippet anchored to the function signature while quoting code forty lines below it leaves the reader unable to find anything. Pseudocode gets a `// ILLUSTRATION — not quoted from the crate` header and a pointer to the real code. +- **Describe what the code does, not what the technique usually does.** criterion 0.5.1's `Slope::fit` is a one-field struct fitting through the origin, so the textbook "the intercept absorbs the overhead" account of least squares is simply false there. Read the implementation before writing the explanation, and prefer the honest, weaker claim over the tidy, wrong one. +- **Every `## Done when` box carries its answer in a collapsed `
` block**, introduced by "Answer each before unfolding it." The checklist is a self-test, so the answer must be reachable without leaving the page but never visible by accident. Answers restate the reasoning rather than pointing back at a step number, and are held to the same standard as the body: real anchors, real numbers, the honest claim. +- **Never trade a definition, a worked example or an answer for brevity.** These chapters have no length target — a guide that assumes vocabulary is not shorter, it is unfinished. Cut redundancy instead. + +The mechanical half of these rules is enforced by `python3 tools/check-reading-depth.py` (step In/Out blockquotes, collapsed answers under every `Done when` item, line-number gutters on quoted snippets, the section spine); run it on a guide before committing it. `--check` is a ratchet — a guide that has started following the rules must follow all of them, and the guides the rollout has not reached yet are reported without failing. The other half — definitions, worked arithmetic, honest claims — is judgement and stays the writer's job. + +Anchors are checked with `python3 tools/pinned-source.py`, which opens a file at the revision the pin table records (`show`, `grep`, `check`, `list`). It uses a real clone under `~/repos` when one is present and otherwise fetches that same commit into a gitignored `.cache/`, so an anchor can be verified on a machine that has not cloned 85 upstream repos. A repo that is not in the pin table — a crate read from the cargo registry — needs `--ref` and a version stated in the guide. + ## Topic package shape Each `topics/NN-name/` contains: `README.md` (study guide, opening with *the problem, measured* — the provided benchmark lane's real output), four to seven `reading-*.md` guides in the concept-first format (framing lead → "the problem in one sentence" → numbered `### Step N` sections → how to read the source → questions → `## Done when` checklist → references), `notes.md` (a `## Baseline (provided lane, , measured )` section recording the real output, *then* the reader's prediction worksheet — leave those cells empty, they are the exercise), and `experiments/` — a Rust crate with **lane 1 implemented and two lanes stubbed**, where the stub tests are the specification and the reference numbers live in `notes.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 675f938..c0b47b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,7 +43,16 @@ Every topic follows the same shape, because the shape is what makes it checkable format: an H1 with the idea in it, a framing lead, **the problem in one sentence**, then `### Step N` sections that build each concept using only terms defined in earlier steps, then how to read the source material with the concepts in hand, - questions to answer, a "done when" checklist, and references. + questions to answer, a "done when" checklist, and references. The depth rules + that make "using only terms defined in earlier steps" enforceable — define every + term at first use, declare each step's input and output, work every formula on + concrete numbers — are in + [CLAUDE.md](https://github.com/AviAvni/database-learning-path/blob/master/CLAUDE.md#reading-guide-depth) + (an absolute link because `CLAUDE.md` is not a book chapter), with + [reading-criterion.md](topics/00-performance-toolbox/reading-criterion.md) as the + reference chapter. Their mechanical half is checked by + `python3 tools/check-reading-depth.py` — run it on a guide before committing, and + see [Building the book](#building-the-book) for the CI gate. - **`notes.md`** — a `## Baseline (provided lane, , measured )` section recording the provided lane's real output with the analysis, then a predictions-vs-measurements worksheet. **The worksheet's cells are meant to be @@ -80,6 +89,18 @@ legitimate shape for a topic; inventing a number to fill the slot is not. `file:line` anchors in the guides mean something. Regenerate it with `python3 tools/pin-table.py` after cloning or updating a reference repo — putting a SHA in each guide instead would mean thousands of them drifting separately. + To read a file at that pinned commit — to write an anchor, or to check one that + is already there — use `python3 tools/pinned-source.py`: + + ```bash + tools/pinned-source.py show lmdb mdb.c -r 1350:1365 # with real line numbers + tools/pinned-source.py grep lmdb 'mdb_env_pick_meta' --path mdb.c + tools/pinned-source.py check lmdb mdb.c:1356 --contains 'meta page' + ``` + + It reads your clone when you have one and otherwise fetches the same commit into + a gitignored `.cache/`, so anchors stay checkable without cloning every upstream + repo the guides cite. - **Generators are seeded.** Anyone must be able to reproduce a figure exactly. - **Notes capture *why* a design wins** and what it trades away — not summaries. @@ -94,7 +115,14 @@ mdbook serve # or: mdbook build CI ([.github/workflows/book.yml](.github/workflows/book.yml)) builds HTML and PDF on every push to `master` and deploys to GitHub Pages. Before committing content, build locally and check that mermaid diagrams render and internal links resolve — a broken -link is invisible in markdown and obvious in the book. +link is invisible in markdown and obvious in the book. The same workflow runs +`tools/check-reading-depth.py --check`, which holds every reading guide that has +started following the depth rules to all of them: + +```bash +python3 tools/check-reading-depth.py topics/03-btree-internals/ # one topic +python3 tools/check-reading-depth.py --stats # rollout progress +``` A second workflow ([verify.yml](.github/workflows/verify.yml)) runs `./verify.sh --summary` and a `-D warnings` build of all 45 crates on every push and diff --git a/FINDINGS.md b/FINDINGS.md index 473bcde..f076dcc 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -28,7 +28,7 @@ instead. | 9 | [Concurrency](topics/09-concurrency/README.md) | A global mutex gets **2.9× slower** from 1 to 16 threads (8.65 → 2.96 Mops/s). Padding "independent" counters to 128 B is worth **17.8×**; 64 B only half-fixes it on M-series. | `./verify.sh 09` | | 11 | [Execution Models](topics/11-execution-models/README.md) | Volcano tops out at **103 M rows/s**, and gets *slower* as selectivity rises (74.7 M at 95%) — surviving the filter is what costs, not the filter. | `./verify.sh 11` | | 12 | [Columnar Analytics](topics/12-columnar-analytics/README.md) | The scan floor is **24–57 GB/s** on a 150 GB/s machine. This lane previously printed **19,047,619 GB/s** — a hoisted loop, caught by its own implausibility. | `./verify.sh 12` | -| 13 | [Graph Engines](topics/13-graph-engines/README.md) | The same two-hop query is **101× slower** from supernodes than from random nodes (4.9 µs → 495 µs) — and reaches *fewer* distinct nodes. | `./verify.sh 13` | +| 13 | [Graph Engines](topics/13-graph-engines/README.md) | The same two-hop query is **101× slower** from supernodes than from random nodes (4.9 µs → 495 µs), because it reaches **77× more** nodes per query — 78,907 against 1022. | `./verify.sh 13` | | 14 | [Vector Search](topics/14-vector-search/README.md) | Brute force: **117 QPS** at recall 1.000. That single point is what every ANN index is betting against. | `./verify.sh 14` | | 15 | [Replication & Consensus](topics/15-replication-consensus/README.md) | Follower fsync policy alone spans **59×** (341 → 20,174 entries/s). Batching fixes the median and leaves the p99 at 2980 µs. | `./verify.sh 15` | | 16 | [Testing & Correctness](topics/16-testing-correctness/README.md) | Seeded crash testing catches planted bugs at **48.8% to 99.6%** per seed — same harness, four wildly different odds of ever finding out. | `./verify.sh 16` | @@ -38,7 +38,7 @@ instead. | 20 | [GraphBLAS](topics/20-graphblas/README.md) | SpMV bandwidth decays **20.7 → 12.3 GB/s** as the graph grows. Hypersparse indexing is **50× smaller** (80.4 MB → 1.59 MB) and sweeps rows **175× faster**. | `./verify.sh 20` | | 21 | [Formal Methods](topics/21-formal/README.md) | The hand-ordered rewriter answers `(a*2)/2` with `(a << 1) / 2` and stops. One locally-excellent rewrite destroys the cancellation — the phase-ordering trap, in four lines. | `./verify.sh 21` | | 22 | [Standard Benchmarks](topics/22-benchmarks/README.md) | TPC-H Q1 and Q6 measured at **5.2–5.7** and **9.0–14.4 GB/s** effective; YCSB-E's p999 is **12.9 µs** against read-only's 4.0 µs. | `./verify.sh 22` | -| 23 | [Full-Text Search](topics/23-fulltext/README.md) | Exhaustive BM25 spans **0.009 ms to 10.378 ms** across four two-term queries — 272,310 postings against 159. Term rarity, not query complexity. | `./verify.sh 23` | +| 23 | [Full-Text Search](topics/23-fulltext/README.md) | Exhaustive BM25 spans **0.009 ms to 10.378 ms** across four queries — 272,310 postings against 159. Term rarity, not query complexity. | `./verify.sh 23` | | 24 | [Graph Algorithms](topics/24-graph-algorithms/README.md) | Same node and edge count, RMAT vs uniform: **15.6 M triangles vs 5428**, and 447 ms vs 195 ms. Degree skew is the workload. | `./verify.sh 24` | | 25 | [Graph ML](topics/25-graph-ml/README.md) | The message-passing kernel *is* an SpMM: **4.31 ms at 16.82 GFLOP/s**, against 5.65 ms for the dense transform beside it. | `./verify.sh 25` | | 26 | [Probabilistic Structures](topics/26-probabilistic/README.md) | A point miss costs **246 ns** (binary search) or **299 ns** (BTreeMap); a 224 MB HashSet does it in **28 ns**. That gap is what a filter is bidding for. | `./verify.sh 26` | @@ -47,7 +47,7 @@ instead. | 29 | [Distributed Transactions](topics/29-distributed-txn/README.md) | The workload's own conflict rate goes **0.3% → 99.6%** as Zipf θ moves 0.5 → 1.3. Contention is a property of the data, before any protocol. | `./verify.sh 29` | | 30 | [Time-Series](topics/30-timeseries/README.md) | delta+varint gives **11.00 B/sample for all four shapes** — a constant series compresses exactly as well as random noise, because only the timestamp is being compressed. | `./verify.sh 30` | | 31 | [CRDTs](topics/31-crdts/README.md) | Last-write-wins on 10 keys with per-write sync loses **94.98%** of writes — 37,991 of 40,000 acknowledged writes that no replica remembers. | `./verify.sh 31` | -| 32 | [HTAP](topics/32-htap/README.md) | One copy, one coarse lock: adding full scans takes writes from **10.5 M per 2 s to 94**, and p99 from 334 ns to **2.7 s**. Every scan is a write outage. | `./verify.sh 32` | +| 32 | [HTAP](topics/32-htap/README.md) | One copy, one coarse lock: adding full scans takes writes from **11.4 M per 2 s to 69**, and p99 from 333 ns to **7.49 s**. Every scan is a write outage. | `./verify.sh 32` | | 33 | [Temporal Graphs](topics/33-temporal-graphs/README.md) | Static reachability reports 25,031 reachable pairs where time-respecting paths number **137** — **99.5% false positives** on the sparse contact graph. | `./verify.sh 33` | | 34 | [Debugging & Diagnosis](topics/34-debugging/README.md) | A closed-loop benchmark reports **p99 = 1.0 µs** where an open-loop one reports **90 ms** on identical work — coordinated omission, a 90,000× lie. | `./verify.sh 34` | | 35 | [Overload Control](topics/35-overload/README.md) | A 10-second outage ends at t=40 s. At 140 QPS (of 300 capacity) goodput stays at **zero until t=161 s**; at 280 QPS it **never recovers** — the outage outlives its own trigger. | `./verify.sh 35` | @@ -57,7 +57,7 @@ instead. | 39 | [Fraud & Identity Graphs](topics/39-fraud-identity-graphs/README.md) | Two row-based rankers fail in *opposite* regimes: degree ranking scores **0.00** precision without camouflage, obscurity ranking **0.00** with it. | `./verify.sh 39` | | 40 | [Security & Attack Graphs](topics/40-security-attack-graphs/README.md) | A directory reporting **8 privileged accounts, forever** has **1969 of 2000 users** holding a path to Domain Admin — and your exposure number depends on how long the collector ran. | `./verify.sh 40` | | 41 | [On-Chain Analytics](topics/41-onchain-analytics/README.md) | The industry-default haircut rule marks **98% of addresses** tainted from one theft; 658 of them are under 0.1% tainted. An 1816 court case does better. | `./verify.sh 41` | -| 42 | [Recommendations & Social](topics/42-recommendations-social/README.md) | Recommending bestsellers to everyone gets **35.3% hit-rate@50** with **92.2% overlap** between users' lists. Popularity is not a weak baseline. | `./verify.sh 42` | +| 42 | [Recommendations & Social](topics/42-recommendations-social/README.md) | Recommending bestsellers to everyone gets **34.0% hit-rate@50** with **92.3% overlap** with the global bestseller list. Popularity is not a weak baseline. | `./verify.sh 42` | | 43 | [Ops Dependency Graphs](topics/43-ops-dependency-graphs/README.md) | One gray failure: **34 of 55 services alert** and the broken one is not among them — it ranks 35th by failure count, 41st by error rate, at exactly the baseline. | `./verify.sh 43` | ## How to read this table diff --git a/PLAN.md b/PLAN.md index 60458c6..97a1dbf 100644 --- a/PLAN.md +++ b/PLAN.md @@ -109,7 +109,7 @@ flowchart TD - **Concepts:** memtable→SST lifecycle, leveled vs tiered vs FIFO compaction, bloom filters (and Monkey's optimal allocation), fractional cascading, compaction debt/write stalls, SST formats & block cache. - **Read code:** fjall (read it ALL — it's small), RocksDB `db/compaction/`, `table/block_based/`. -- **Papers:** "Monkey: Optimal Navigable Key-Value Store" (SIGMOD'17), "Dostoevsky" (SIGMOD'18), RocksDB paper (TODS'21), "Constructing and Analyzing the LSM Compaction Design Space" (VLDB'21). +- **Papers:** "Monkey: Optimal Navigable Key-Value Store" (SIGMOD'17), "Dostoevsky" (SIGMOD'18), RocksDB paper (FAST'21, also ACM Transactions on Storage 17(4)), "Constructing and Analyzing the LSM Compaction Design Space" (VLDB'21). - **Build & bench:** implement a mini-LSM (memtable + SSTs + leveled compaction + bloom filters) — optionally follow skyzh/mini-lsm course; measure write amp with different compaction strategies. - **Capstone M4:** LSM-backed alternative persistence (graph snapshots as SSTs); benchmark B+tree vs LSM backends on graph mutation + bulk-load workloads. @@ -237,7 +237,7 @@ flowchart TD **Why:** The last 10x on a single core. Touched in topic 11 — this is the dedicated deep dive: writing kernels that saturate the CPU. -- **Concepts:** SIMD fundamentals (AVX2/AVX-512 vs ARM NEON/SVE — know both, you're on ARM), autovectorization and why it fails, Rust portable SIMD (`std::simd`) vs intrinsics, branchless selection (masks + compress), SIMD hash probing (SwissTable), SIMD string parsing/comparison, bit-packed decoding at SIMD speed (FastLanes), gather/scatter costs, instruction-level parallelism & dependency chains, Mojo's SIMD-first design (`SIMD[type, width]` as a first-class parametric type — compare its ergonomics vs `std::simd` and intrinsics). +- **Concepts:** SIMD fundamentals (AVX2/AVX-512 vs ARM NEON/SVE — know both, you're on ARM), autovectorization and why it fails, Rust portable SIMD (`std::simd`) vs intrinsics, branchless selection (masks + compress), SIMD hash probing (SwissTable), SIMD string parsing/comparison, bit-packed decoding at SIMD speed (FastLanes), gather/scatter costs, instruction-level parallelism & dependency chains, Mojo's SIMD-first design (`SIMD[dtype, size]` as a first-class parametric type — compare its ergonomics vs `std::simd` and intrinsics). - **Read code:** polars `crates/polars-compute/` kernels, simdjson (the masterclass — read with the paper), hashbrown SIMD group probing, DuckDB compressed-scan kernels, usearch/SimSIMD distance functions, memchr crate, Mojo stdlib + Modular's matmul optimization blog series. - **Papers:** "Rethinking SIMD Vectorization for In-Memory Databases" (SIGMOD'15), "Parsing Gigabytes of JSON per Second" (simdjson, VLDB'19), "The FastLanes Compression Layout" (VLDB'23). - **Build & bench:** write filter-selection and dot-product kernels four ways: naive scalar, autovectorized, `std::simd`, NEON intrinsics; bench with `perf stat` (IPC, vector-lane utilization); then SIMD-ize a bit-packing decoder and compare against topic 12's scalar version; port one kernel to Mojo and compare both the numbers and the code you had to write. diff --git a/SESSION-LOG.md b/SESSION-LOG.md index 9d7a6cc..e307467 100644 --- a/SESSION-LOG.md +++ b/SESSION-LOG.md @@ -9,6 +9,131 @@ Every performance figure quoted below is reproducible with `./verify.sh` (see [README.md](README.md)); timings depend on hardware, everything else is seeded. +## 2026-08-08 — topics 34–43 — the last 40 guides, and the rules turned on for the whole book + +**Final batch of the rollout: topics 34 through 43, 40 guides, 9,163 lines of prose becoming 14,975.** With these the ratchet reads **230 of 230**, and `.github/workflows/book.yml`'s depth job is switched from `--check` (started files only) to `--check --all`, so a new guide that skips the rules now fails CI rather than being quietly exempt. Same contract and method as batches 1–3: one agent per topic on disjoint directories, `tools/check-reading-depth.py` printing *N/N* as the gate, two or three anchors per topic re-verified against the pin afterwards. Every spot-check confirmed the agent. + +**Three claims were the exact reverse of the source.** Topic 43's gray-failure guide sent readers to "§2 for the model, §1 and §3 for the examples"; §2 is the four case studies and §3 is the model (3.1 Terminology, 3.2 Differential observability, 3.3 Temporal evolution). The same guide collapsed §3.1's **observer** and **reactor** into one entity — the observer gathers information, the reactor acts on it, and the gap between them is the whole paper. Topic 40's bloodhound guide used `Contains` as its example of an edge kind *excluded* from pathfinding; at pin `1968388` it is element 60 of `PathfindingRelationships` (`ad.go:1161`), so the example is now `GetChanges`/`GetChangesAll`, which are in `Relationships` and `ACLRelationships` but genuinely not in pathfinding — because only their conjunction is dangerous, and post-processing synthesizes it into `DCSync`. + +**Two control laws were named after the wrong algorithm.** CockroachDB's KV slot adjuster is **AIAD**, not AIMD: `total--` at `kv_slot_adjuster.go:72`, `total++` at `:91`, and the code's own comments say "additive decrease" and "additive increase". Topic 35 carries genuine AIMD as well — DAGOR's Algorithm 1, (1−α)·N down and +β·N up, §4.2.3 — which is precisely why the distinction had to be right. And topic 43's Pivot Tracing 600 → 6 tuples/s headline belongs to **process-level (intermediate) aggregation** of *emitted* tuples (§4), not to the Table 3 rewrites, which reduce tuples *packed into the baggage*; four places in that topic's README and two in its notes credited the wrong optimization, and one exercise went further and called it tuples crossing the join. + +**Three quotations did not survive being checked against the paper.** Topic 35's README quoted Bronson et al. as saying metastable failures "account for many of the largest outages at major web companies" — §1 says they "have caused widespread outages at large internet companies, lasting from minutes to hours". Topic 39's Fellegi-Sunter guide had the decision rule as "at or above T_μ → match, at or below T_λ → nonmatch", which moves both thresholds into the automatic decisions; Eq. 2 is strict at both ends, and the clerical-review band **includes** its boundaries. Its controller definition dropped "(or in exceptional cases multiple entities)", which is exactly the case where Heuristic 1 over-merges, and its safety argument said "private key" where the paper says "private **signing** key" — signing is what a multi-input spend requires and therefore what makes the heuristic sound. + +**Magnitudes.** WeChat's Chinese New Year peak is ≈10× the daily **average**, not 10× the daily peak (§2.3 gives both: peak hours ≈3× average). SLEUTH's 250 bytes and 3 KB per edge belong to **STINGER and NetworkX**, the two main-memory-optimized stores, not to "a general graph database" — Neo4J and Titan are dismissed with no figure at all, so SLEUTH's 10 bytes/edge is 25× better than the *best* main-memory store, which is the stronger claim. Pixie's Algorithm 1 is **eleven** lines, not twenty. BloodHound's `PathfindingRelationships` holds **64** kinds, not 63 (`Relationships` 88, `ACLRelationships` 30, `PostProcessedRelationships` 31, 104 `graph.StringKind` constants in total). And topic 43's dapper guide asserted "113 dependency edges" twice, sourced from nothing — the README says 152 configured, and the claim the exercise rests on is edge recall = 1.000, which is count-agnostic. + +**Two papers contradict themselves, and the guides now say so rather than picking a side silently.** Pixie prints `C = max_{p∈P}|E(p)|` in Eq. 1, which makes step allocation *linear* in degree, and then claims sub-linearity in the next sentence; sub-linearity needs `C = max_p log|E(p)|`, which is what this repo's crate implements. The guide states both and works the example that settles it — degrees 1 and 10,000 with a 10,000-step budget give the low-degree pin 0 steps under the printed definition and ~5 under the intended one. Pixie also cites "lines 12-15 of Algorithm 2" where its printed algorithm has lines 9-14; the guide cites the printed lines and explains the discrepancy. + +**Anchors that had drifted.** DataFusion retired `EnforceDistribution` into `EnsureRequirements` — the old top-level `physical-optimizer/src/enforce_distribution.rs` path does not resolve at the pin, the struct is at `ensure_requirements/mod.rs:166` (`:159` is inside its doc comment), and the hash `RepartitionExec` goes in at `enforce_distribution.rs:1291` behind the `should_add_hash_repartition` guard at `:1281`. Four more repartition anchors pointed at doc comments rather than code: `new_hash_partitioner` is `:679` not `:667`, `new_round_robin_partitioner` is `:710` not `:699`, and the strength-reduced modulo is `partition_reducer.partition_indices` at `:862` — `:675` is prose. `REPARTITION_RANDOM_STATE` is `SeededRandomState::with_seed(0)` (`mod.rs:592`), not a plain `RandomState`, which is the difference between a reproducible partitioning and an irreproducible one. Splink's `expectation_maximisation.py:18` is an *import* line, not the E-step (`:268`); `PostgresDialect` is `dialects.py:573` and the cited `:674` is past the end of a 672-line file; `graph_operations/connected_components.py` does not exist. Six GraphRAG-SDK paths were missing a directory component, and `vector_store.py:485` is `fulltext_search` — the *query* side — where the guide claimed index creation, which is `:133`. Topic 43's three exercise stubs were missing the crate's `src/` component. + +**Numbers corrected in files outside the guides.** `FINDINGS.md` row 42 read "35.3% hit-rate@50 with 92.2% overlap between users' lists"; no run in this repo produced 35.3/92.2 — README's lane-1 table, notes.md and this log all say **0.340** and **0.923** — and the ~0.92 figure is `popularity_overlap`, the overlap with the *global bestseller list*, not between users (that measure is `personalization`, 0.155). Corrected rather than annotated, on the same grounds as rows 23 and 32: it is not a stale measurement, it is a row that matches nothing. Topic 42's README attributed the "classical random walks low degree nodes contribute less signal" quote to §3.1; it is §1. Topic 38's 281-minute indexing figure is for the **Podcast** dataset at a **600-token** chunk window — the 8k window is generation-side — and its notes cited `/tmp/*.pdf` as provenance. Topic 34's `PERF_TIMER_GUARD` row and topic 36's fabricated Twitter α=1.7 were fixed in batch 4's earlier commits. + +**Arithmetic worked rather than asserted**, per rule 3: DAGOR's hidden capacity 300/(1+retries) = 150 QPS with the retry threshold at 280 QPS, 0.5^k shedding, and 30 s × 280 QPS = 8,400 queued; Fellegi-Sunter's per-field weights giving 35.93 bits for the last/first/dob-agree, city-disagree, phone-agree pattern against 45.36 all-agree; the haircut's 10× per-hop dilution reaching 0.1% in three hops, which is where topic 41's 658 sub-0.1% UTXOs come from; BlockSci's 8 B × 1.198e9 = 9.58 GB, i.e. 50.09 → 40.50 GB; Sherlock's (2r)^k = ~80,000 states for r=200, k=2 against 3^200 ≈ 10^95; and dapper's 40000/1024 = 39 traces with a rare path at ≈0.001. + +**Two provenance gaps are disclosed rather than papered over.** Graefe's SIGMOD-1990 Volcano paper is unreachable (ACM 403s; every mirror carries the TKDE-1994 edition, which has no §5 micro-benchmark table), so topic 37's 25.73 µs/record and packet-size sweep are kept on the strength of a prior session's verification, with every Volcano *concept* re-checked against TKDE and a note asking someone with ACM access to spot-check. Ammann/Wijesekera/Kaushik CCS'02 is not open-access anywhere reachable, so topic 40's 5948/68364, 229 bits and `O(|A|^2·|E|)` are cross-corroborated against MulVAL CCS'06 §2 rather than re-read at source; no new Ammann number was introduced. + +**One rendering bug found by the final `mdbook build`, and it was corpus-wide.** `
Answer` opens a CommonMark HTML block that runs to the next **blank line**, so an answer starting on the very next line is raw HTML: backticks stay literal, and any bare `<...>` becomes a tag. `Vec` in topic 23's roaring guide opened a `` element that swallowed the `
` after it. 118 blocks across 21 files were missing the blank line after `` and 58 more files were missing the one before `
`; both are normalised to the reference chapter's shape. Four bare angle brackets in prose took backticks (`Vec`, and `C=A*B` / `C=A*B` in topic 20's GraphBLAS quotation, where `` was being read as the start of ` A["avg_times[] = times[i] / iters[i]
(mod.rs:124–129)"] + S --> D["data = (iters, times) pairs
(mod.rs:140)"] + A --> T["Step 6 · tukey::classify
outlier labels"] + A --> E["Step 7 · estimates()
mean · median · std dev · MAD"] + A --> C["Step 9 · compare.rs
t-test vs baseline"] + D --> R["Step 5 · regression()
slope = ns/iteration"] + R --> H["headline: time: [lo mid hi]"] +``` + +Note the asymmetry, because it surprises everyone: the **headline number you +read comes from `data`** (the slope), while the **regression detection that +tells you it got slower runs on `avg_times`** (the mean). Two different +statistics of the same run. Step 9 comes back to this. + +Why it matters: "which numbers is this step actually looking at?" is the +question that makes the rest of the pipeline legible. Answer it once, here. + +### Step 5 — the slope is the per-iteration cost -### Step 4 — linear regression: the slope is the per-iteration cost +> **In:** `data`, the `(iters, times)` pairs from Step 4 — *not* `avg_times`. +> **Out:** one number: ns per iteration, plus (via Step 7) its interval. -Every sample's total time is really `total_time = overhead + cost × iters`: +Every sample's total time is really `total_time ≈ overhead + cost × iters`: a fixed per-sample overhead (reading the clock, loop setup) plus the true -per-iteration cost times the batch size. Plot the 100 samples as points and -fit a straight line through them — the **slope** of that line is the -per-iteration cost, and the fixed overhead lands in the **intercept**, where -it can't contaminate the answer: - -``` -total_time why slope beats mean-of-averages: - │ × - │ × slope = ns per iteration ← the answer - │ × - │ × - │ × - ├─────────────────────────── iters - ╵← intercept = fixed per-sample overhead - (mean of averages absorbs it; the slope ignores it) -``` - -Compare the naive alternative: averaging the `avg_times` from Step 3 spreads -that fixed overhead across every sample and *inflates* the answer — worst for -the smallest batches. In the code this is `regression()` -(`analysis/mod.rs:269`); it's only valid when `iters` actually varies, so -criterion checks for linear sampling at line 152. The headline -`time: [lo mid hi]` criterion prints is built from this slope. +per-iteration cost times the batch size. **Linear regression** means fitting +a straight line to those points; the **slope** of the line — how much `y` +grows per unit of `x` — is the per-iteration cost. + +**Least squares** picks the line that minimises the sum of squared vertical +distances from the points to it. Criterion's entire regression is four lines — +**focus on 27**, which is the estimate: + +```rust +// stats/bivariate/regression.rs:20–28 (21–22 unpack xs/ys; 23 and 26 blank) +20 pub fn fit(data: &Data<'_, A, A>) -> Slope { +24 let xy = crate::stats::dot(xs, ys); // Σ xᵢyᵢ +25 let x2 = crate::stats::dot(xs, xs); // Σ xᵢ² +27 Slope(xy / x2) // m = Σxᵢyᵢ / Σxᵢ² ← the whole fit +28 } +``` + +Read the type first: `struct Slope(pub A)` — **one** field. This is a fit +of `y = m·x`, a line **forced through the origin**. There is no intercept +term, so criterion is not estimating the per-sample overhead and setting it +aside; it is assuming it away. What saves the estimate is the shape of +`m = Σxᵢyᵢ / Σxᵢ²`: each point's influence is weighted by `xᵢ²`, so the +biggest batches dominate — and the biggest batches are exactly the ones where +a fixed overhead is proportionally smallest. + +Compare the naive alternative, `mean(avg_times)`, on three points with a true +cost of 100 ns/iter and a 500 ns per-sample overhead: + +``` +iters 10 20 30 +times 1500 2500 3500 ns (= 100·iters + 500) +avg 150 125 116.67 ns/iter + +mean(avg_times) = (150 + 125 + 116.67) / 3 = 130.56 → +30.6% +slope = (10·1500 + 20·2500 + 30·3500) / (10² + 20² + 30²) = 121.43 → +21.4% + +same arithmetic on criterion's real ladder (d = 15, cost 70.1 µs, 1 µs overhead): +mean(avg_times) = 70,103.46 ns → +0.0049% | slope = 70,101.00 ns → +0.0014% +``` + +Neither is exact — that is the honest version of this story, and it follows +directly from there being no intercept. But the mean is dragged up hardest by +the *smallest* batch (the 150), where the overhead is a third of the +measurement, while the slope barely notices it; on the real ladder that leaves +the slope ~3.5× less biased. + +This is why linear sampling matters: the fit is only meaningful when `iters` +actually varies, so criterion computes it only under linear sampling +(`analysis/mod.rs:152`). Under flat sampling `estimates.slope` stays `None` +and the headline silently falls back to the mean — `typical()` is +`self.slope.as_ref().unwrap_or(&self.mean)` (`estimate.rs:114`). Why it matters: the slope is a per-iteration estimate that a constant -measurement tax cannot bias — the mean has no such immunity. +measurement tax barely moves, because the largest batches outvote the +smallest ones. The mean has no such defence. -### Step 5 — outliers: label them, never delete them +### Step 6 — outliers: label them, never delete them + +> **In:** `avg_times` from Step 4. +> **Out:** the same 100 values, each tagged with a label. Nothing is removed. An **outlier** is a sample far outside the bulk of the data — usually one of Step 1's noise events (a preemption, a throttle step) landing inside a batch. -Criterion classifies them with **Tukey fences** (`tukey::classify`, -`analysis/mod.rs:141`): compute the quartiles, and flag anything beyond -1.5× the interquartile range as *mild*, beyond 3× as *severe*. That's the -"Found N outliers among 100 measurements" line in the output. - -The crucial policy: outliers are **labeled and reported, never removed**. -Deleting the samples you don't like is how benchmarks lie — maybe that "noise" -is your allocator hitting a slow path every 64th call, i.e. real behavior. - -Why it matters: you get told the data is contaminated *and* you get to see -by how much, instead of the tool silently editing reality. - -### Step 6 — bootstrap resampling: a confidence interval with no assumptions - -A **confidence interval (CI)** is a range — "95% CI [69.6, 70.5] µs" — meaning -the procedure that produced it captures the true value 95% of the time. -Textbook CIs assume the noise is normally distributed (the bell curve); -latency noise isn't (it's skewed — there's a floor but no ceiling). The -**bootstrap** sidesteps the assumption entirely: pretend your 100 samples -*are* the population, resample 100 values from them **with replacement** -(the same sample may be drawn twice, others not at all), recompute the -statistic, and repeat 100,000 times. The spread of those 100,000 recomputed -statistics is an empirical distribution *of the statistic itself* — read -the CI straight off its percentiles: +To say "far outside" precisely you need three definitions: + +- A **percentile** is the value below which a given share of the sorted data + falls; the **median** is the 50th percentile. +- The **quartiles** are the 25th and 75th percentiles, written **q1** and + **q3**. Between them sits the middle half of the data. +- The **interquartile range (IQR)** is `q3 − q1` — the width of that middle + half, and a measure of spread that a few wild values cannot inflate. + +**Tukey's method** (`stats/univariate/outliers/tukey.rs:254`) builds four +**fences** from those, and labels each point by which fences it falls outside: + +``` +inner fences: q1 − 1.5·IQR and q3 + 1.5·IQR outside → mild outlier +outer fences: q1 − 3·IQR and q3 + 3·IQR outside → severe outlier +``` + +Worked on nine sorted `avg_times` in µs (criterion interpolates percentiles, +but with 9 points the quartiles land exactly on data points): + +``` +sample 69.6 69.8 69.9 70.0 70.1 70.2 70.4 70.6 78.3 + q1 median q3 +q1 = 69.9 q3 = 70.4 IQR = 0.5 + +inner fences 69.15 .......................... 71.15 +outer fences 68.40 .......................... 71.90 + 78.3 → SEVERE (high) +``` + +That is the "Found N outliers among 100 measurements" line in the output +(`report.rs:463`). + +The crucial policy: outliers are **labeled and reported, never removed**. The +classified sample flows onward with every point intact. Deleting the samples +you don't like is how benchmarks lie — maybe that "noise" is your allocator +hitting a slow path every 64th call, i.e. real behavior. + +Why it matters: you get told the data is contaminated *and* by how much, +instead of the tool silently editing reality. + +### Step 7 — bootstrap resampling: a confidence interval with no assumptions + +> **In:** `avg_times` (Step 4) for the mean/median/spread estimates; `data` +> (Step 4) for the slope's interval. +> **Out:** for each statistic, a whole distribution of plausible values — from +> which the printed `[lo mid hi]` brackets are read. + +Vocabulary first, because five terms arrive at once: + +- The **population** is what you wish you could measure: every run your + function could ever have. Your 100 samples are a **sample** from it. +- A **statistic** is any number computed from a sample — mean, median, slope. +- A **point estimate** is that single computed number; it says nothing about + how much it would have wobbled had you run again. +- The **sampling distribution** is the spread you *would* see in that statistic + across many repeat runs. Getting at it is the whole game. +- A **confidence interval (CI)** at **confidence level** 95% is a range + produced by a procedure that captures the true value 95% of the time. + +Textbook CIs get there by assuming the noise is normally distributed (the bell +curve). Latency noise isn't: it is skewed, because there is a floor on how +fast code can run but no ceiling on how slow. The **bootstrap** sidesteps the +assumption entirely — pretend your 100 samples *are* the population, and +simulate the repeat runs by drawing from them: ```rust -fn bootstrap_ci(sample: &[f64], nresamples: usize) -> (f64, f64) { - let n = sample.len(); - let mut stats = Vec::with_capacity(nresamples); - for _ in 0..nresamples { // 100_000 in criterion - // resample WITH replacement, same size — pretend the sample IS the population - let stat = mean((0..n).map(|_| sample[rand_below(n)])); - stats.push(stat); // distribution OF THE STATISTIC - } - stats.sort_by(|a, b| a.partial_cmp(b).unwrap()); - (percentile(&stats, 2.5), percentile(&stats, 97.5)) // CI = its percentiles — -} // no normality assumed -``` - -Criterion bootstraps *everything*: `estimates()` (`analysis/mod.rs:300`) -bootstraps the mean/median/std-dev/MAD of the per-iteration averages -(line 321), and the headline `time: [lo mid hi]` is the **slope's** bootstrap -CI — resample the (iters, times) points, refit the line each time. +// ILLUSTRATION — not quoted from the crate; criterion's real loop is +// stats/univariate/resamples.rs:37–41, wrapped by Sample::bootstrap +let n = sample.len(); +for _ in 0..nresamples { // 100_000 in criterion + // resample WITH REPLACEMENT, same size — a value may be drawn twice and + // others not at all; that variation IS the simulated re-run + stats.push(mean((0..n).map(|_| sample[rand_below(n)]))); +} // stats = distribution OF THE STATISTIC +stats.sort_by(|a, b| a.partial_cmp(b).unwrap()); +(percentile(&stats, 2.5), percentile(&stats, 97.5)) // 95% CI = its own percentiles; + // no normality assumed +``` + +Criterion bootstraps *everything*. `estimates()` (`analysis/mod.rs:300`) +resamples `avg_times` 100,000 times and recomputes four statistics each time +(line 321): + +- the **mean** and the **standard deviation** (the square root of the average + squared distance from the mean — the everyday measure of spread); +- the **median** and the **median absolute deviation (MAD)** — the median of + each point's distance from the median, scaled by 1.4826 so that on normal + data it lands on the same scale as the standard deviation + (`stats/univariate/sample.rs:64`). Median and MAD are the outlier-resistant + pair; mean and std dev are not. + +Each of those four gets its own bootstrap distribution, and hence its own CI. +The **standard error** criterion also reports is just the standard deviation +*of the bootstrap distribution* — how much the statistic itself moves around +(`analysis/mod.rs:283`). + +The slope gets the same treatment with one difference: `regression()` +(`analysis/mod.rs:269`) resamples the `(iters, times)` **pairs together** +— index `i` drags both coordinates along (`stats/bivariate/resamples.rs:36–41`) +— and refits the line on each resample. The headline `time: [lo mid hi]` is +that slope distribution's 2.5th, point, and 97.5th values. Why it matters: this is the engine under every bracketed range criterion prints, and it works on ugly, skewed, real-world timing data. -### Step 7 — why a CI beats taking the minimum +### Step 8 — why a CI beats taking the minimum + +> **In:** Step 7's interval versus the rival proposal. +> **Out:** the reason this chapter exists. The rival school (older Python `timeit` advice) says: noise only ever *adds* time, so report the minimum — it's the closest to the true cost. Criterion @@ -174,33 +384,148 @@ rejects that, for four reasons: 2. **Noise isn't strictly additive.** Frequency scaling (Step 1) means early samples can run at a *higher* clock (pre-thermal-throttle) — the min can be an unrepresentatively lucky sample, and on modern laptops often is. -3. **Min is statistically fragile for comparison.** It's an extreme-value - statistic with no usable sampling distribution — you can't compute a - p-value on "min got 2% slower." Step 8's machinery only works because - mean/slope have bootstrap distributions. +3. **Min is statistically fragile for comparison.** It is an **extreme-value + statistic** — a statistic determined entirely by one observation, which + makes its sampling distribution (Step 7) both wild and dependent on sample + size. So you cannot put a number on how surprising "min got 2% slower" is — + Step 9 does exactly that, and its machinery only works because mean and + slope have well-behaved bootstrap distributions. 4. **A point estimate hides confidence.** `[69.6 70.1 70.5] µs` says the measurement is tight; a bare `69.6` hides whether the spread was 1% or 40%. Why it matters: this is the study-guide question, and it's the philosophical core — a benchmark result without an uncertainty estimate is an anecdote. -### Step 8 — regression detection: two gates, not one +### Step 9 — regression detection: two gates, in order + +> **In:** this run's `avg_times` (Step 4) **and** the baseline's `avg_times`, +> recomputed from the saved `sample.json` (`compare.rs:44–49`). +> **Out:** one of three verdicts — *no change detected*, *within noise +> threshold*, or *improved/regressed*. + +First, the lineage trap from Step 4: the comparison runs on **`avg_times`**, +i.e. on means. The slope that produced your headline number is not what gets +compared. A benchmark can print a slope-based time while being judged on its +mean. Detecting "did my change make this slower?" needs two separate questions, because a difference can be statistically real yet too small to care about, -or large but pure noise. Criterion (line 188 → `compare.rs`) loads the saved -baseline and applies two gates: +or large but pure noise. + +**Gate 1 — is the difference real?** This is a **t-test**: given how much +each of two sets of numbers scatters internally, is the gap between their +averages bigger than that scatter can explain? The **t-statistic** is that +gap measured in units of its own uncertainty (`sample.rs:171`): + +``` +t = (x̄ − ȳ) / √(s²ₓ/nₓ + s²ᵧ/nᵧ) + + x̄, ȳ = the two means s² = VARIANCE — the mean squared distance from + nₓ, nᵧ = the two counts the mean, with an n−1 divisor (sample.rs:187) +``` + +Worked on three new samples against three baseline samples, in µs: + +``` +new = [70.1, 70.4, 69.8] x̄ = 70.1000 s²ₓ = 0.090000 +base = [69.2, 69.5, 69.1] ȳ = 69.2667 s²ᵧ = 0.043333 + +numerator = 70.1000 − 69.2667 = 0.8333 +denominator = √(0.090000/3 + 0.043333/3) = √0.044444 = 0.2108 +t = 0.8333 / 0.2108 = 3.95 +``` + +So the gap is about four times the size of its own uncertainty. Is four a +lot? A textbook would look that up in a table — which means assuming a +distribution, exactly what Step 7 refused to do. Criterion bootstraps instead. + +The **null hypothesis** is the boring explanation: there is no real +difference, both sets came from the same population. `mixed::bootstrap` +(`mixed.rs:11`) *builds* that world and measures it: + +```rust +// stats/univariate/mixed.rs — the pooling at 27–28, then the resample loop. +// Lines 66–70 are the non-rayon path; 38–42 are the identical rayon path. +27 c.extend_from_slice(a); +28 c.extend_from_slice(b); // POOL both — erase which run each value came from + // ... 29–65: wrap the pool as a Sample, then rayon/non-rayon dispatch ... +66 let resample = resamples.next(); // draw n_a + n_b, w/ replacement +67 let a: &Sample = Sample::new(&resample[..n_a]); // arbitrarily call these "new" +68 let b: &Sample = Sample::new(&resample[n_a..]); // and these "base" +70 statistic(a, b) // recompute t +``` -1. **Is the difference real?** A bootstrapped two-sample t-test - (`compare.rs`, line 200: `p_value`) — bootstrap the "no difference" - hypothesis and ask how often chance alone produces a gap this big. -2. **Is it big enough to care?** A bootstrapped relative-change estimate - compared against `noise_threshold`. +Pooling then re-splitting at random is what makes it a null distribution: +the two halves now differ by chance alone. Running it 100,000 times shows +exactly how big a `t` chance alone produces. -Both must pass — e.g. `+3781% (p = 0.00 < 0.05)`: significant *and* large. +The **p-value** is then just a rank — the share of those 100,000 chance-only +`t` values that are at least as extreme as the real one +(`stats/mod.rs:63`): + +```rust +// stats/mod.rs, Distribution::p_value — 68–73 map Tails to 1 or 2 +67 let hits = self.0.iter().filter(|&&x| x < t).count(); +74 A::cast(cmp::min(hits, n - hits)) / A::cast(n) * tails // tails = 2 +``` + +`min(hits, n − hits)` takes whichever tail the observation sits in, and the +`× 2` makes it **two-tailed** — criterion asks "different?", not "slower?", +so a speed-up is as detectable as a regression. A p-value of 0.00 means +essentially none of the 100,000 chance-only worlds produced a gap this big. + +The gate: **`p_value < significance_level`** (default 0.05, +`analysis/mod.rs:200` computes it, `report.rs:598` tests it). Fail, and +criterion prints `No change in performance detected.` and stops — gate 2 is +never consulted. + +**Gate 2 — is it big enough to care?** Only reached if gate 1 passed. This +one ignores t entirely and looks at the bootstrapped **relative change** in +the mean — `a.mean() / b.mean() - 1.` resampled 100,000 times +(`compare.rs:108–121`) — and compares its *confidence interval* against the +**noise threshold**, the relative change below which you have declared you do +not care (default 0.01, i.e. 1%). From `report.rs:779`: + +```rust +// report.rs:784–790, inside compare_to_threshold (declared at 779; +// 780–782 pull lb/ub off the confidence interval) +784 if lb < -noise && ub < -noise { // ENTIRE interval below −1% +785 ComparisonResult::Improved +786 } else if lb > noise && ub > noise { // ENTIRE interval above +1% +787 ComparisonResult::Regressed +788 } else { +789 ComparisonResult::NonSignificant // "Change within noise threshold." +790 } +``` + +Note it tests **both bounds**, not the point estimate: an interval straddling +the threshold is not enough. So the three verdicts, in order: + +| Gate 1 (`p < 0.05`) | Gate 2 (whole CI past ±1%) | Printed | +|---|---|---| +| fail | not evaluated | `No change in performance detected.` | +| pass | fail | `Change within noise threshold.` | +| pass | pass | `Performance has improved.` / `regressed.` | + +A line like `+3781% (p = 0.00 < 0.05)` is a run that cleared both. Why it matters: one gate alone produces either false alarms on every 0.3% -wobble or silence on real 5% regressions. +wobble or silence on real 5% regressions — and knowing they are sequential +tells you which message means which failure. + +## The knobs and their defaults + +All set in `lib.rs:427–433`, all overridable per-benchmark or on the CLI: + +| Knob | Default | Step | What it controls | +|------|---------|------|------------------| +| `warm_up_time` | 3 s | 2 | how long the unrecorded calibration loop runs | +| `sample_size` | 100 | 3 | how many batches (`n` in the `d` formula) | +| `measurement_time` | 5 s | 3 | the budget the batch ladder is sized to fill | +| `nresamples` | 100,000 | 7 | bootstrap resamples per statistic | +| `confidence_level` | 0.95 | 7 | the width of every printed `[lo hi]` bracket | +| `significance_level` | 0.05 | 9 | gate 1's p-value cutoff | +| `noise_threshold` | 0.01 | 9 | gate 2's "too small to care" band | ## Where each step lives in the code @@ -209,46 +534,195 @@ step above maps to a call in it: ```mermaid flowchart TD - S["1 · routine.sample() (line 83)
(iters, times) — iters grows [d, 2d, 3d, ...]"] - S --> N["2 · avg_times[i] = times[i] / iters[i] (124–129)"] - N --> T["3 · tukey::classify (141)
label outliers — NEVER remove"] - N --> E["4a · estimates() (300)
mean/median/MAD, each bootstrapped (321)"] - S --> R["4b · regression() (269)
slope of total_time vs iters = ns/iter"] - R --> H["headline: time: [lo mid hi] = slope's bootstrap CI"] - E --> C["5 · compare.rs (188)
bootstrapped t-test vs saved baseline
+ noise_threshold gate"] - H --> C + S["3 · routine.sample() (mod.rs:83)
iters grows d, 2d, 3d, ..."] + S --> N["4 · avg_times = times[i] / iters[i] (124–129)"] + S --> D["4 · data = pairs (140)"] + N --> T["6 · tukey::classify (141)
label outliers — NEVER remove"] + N --> E["7 · estimates() (300)
mean/median/std-dev/MAD, bootstrapped (321)"] + D --> R["5 · regression() (269)
slope of total_time vs iters = ns/iter"] + R --> H["headline: time: lo mid hi = slope's bootstrap CI"] + N --> C["9 · compare.rs (188)
gate 1: bootstrapped t-test (compare.rs:72)
p_value (mod.rs:200)"] + C --> G["9 · gate 2: noise threshold
(report.rs:779)"] ``` Suggested reading order in the crate: -1. `analysis/mod.rs::common` — the spine (Steps 3, 4, 5, 6 in sequence) -2. `stats/univariate/outliers/tukey.rs` — Step 5's fences, ~100 lines -3. `stats/bivariate/regression.rs` — `Slope::fit` is ~10 lines of - least-squares (Step 4) -4. `analysis/compare.rs` + `stats/univariate/mixed.rs` — Step 8's - bootstrapped t-test -5. `routine.rs::warm_up` — see that warm-up (Step 2) is really - iteration-count calibration +1. `analysis/mod.rs::common` — the spine; watch for the Step 4 fork at 124–140 +2. `stats/bivariate/regression.rs` — `Slope::fit`, three lines; note the + one-field struct (Step 5) +3. `stats/univariate/outliers/tukey.rs` — the fences, with an ASCII diagram in + the module docs (Step 6) +4. `analysis/compare.rs` + `stats/univariate/mixed.rs` — the bootstrapped + t-test; `mixed.rs` is where the pooling happens (Step 9) +5. `routine.rs::warm_up` — confirm that warm-up is really calibration (Step 2) + +## Questions to answer + +- Why does criterion report a confidence interval rather than a minimum? + (Step 8 — the README's question for this chapter.) +- Your benchmark prints a headline time built from the slope, but the + regression verdict is computed from the mean. Construct a sample where + those two disagree about the direction of a change. What would the run + print? +- `Slope::fit` has no intercept. What does that assume about the per-sample + overhead, and which batch in the ladder is hurt most when it is wrong? +- Read `Slope::r_squared` (`regression.rs:33`). Line 48 assigns + `ss_tot = ss_res + ...` where every other line accumulates. Is that a bug? + Trace its callers (`report.rs:708`, `html/mod.rs:373`) and decide whether it + can affect a reported *time* or only a plot label. This is the habit the + chapter is really teaching: read the implementation, not the textbook + description of the technique. ## Takeaway -Criterion is built on three ideas: **bootstrap instead of normality assumptions, slope -instead of mean, label outliers instead of dropping them.** +Criterion is built on three ideas: **bootstrap instead of normality +assumptions, slope instead of mean, label outliers instead of dropping +them.** With the caveat you now know: the slope is what it *prints*, the mean +is what it *compares*. ## Done when +Answer each before unfolding it. + - [ ] You can explain why criterion times *batches* and fits a line, rather than timing one iteration. -- [ ] You can give the three reasons taking the minimum is the wrong estimator, not just a noisy one. + +
Answer + + One iteration is both too short and too noisy: the clock's own resolution is + ~20–40 ns, so a short function mostly measures the timer, and any single + reading carries whatever the machine was doing at that instant (Step 1). + Timing a batch amortises both. The batch sizes then grow *linearly* + (`d, 2d, …, 100d`) specifically so that plotting total time against batch + size and fitting a line recovers ns-per-iteration as the **slope** — an + estimate the largest batches dominate, so a fixed per-sample overhead barely + moves it (Steps 3 and 5). + +
+ +- [ ] You can name, for each stage of the pipeline, whether it consumes `avg_times` or the raw `(iters, times)` pairs — and say why the headline and the regression verdict come from different ones. + +
Answer + + `avg_times` feeds `tukey::classify` (Step 6), `estimates()` (Step 7) and the + baseline comparison (Step 9). The raw `(iters, times)` pairs feed + `regression()` (Step 5) and nothing else. + + The headline `time: [lo mid hi]` is the **slope's** bootstrap CI, because + `typical()` returns the slope when it exists (`estimate.rs:114`). The + regression verdict is a t-test on **`avg_times`**, i.e. on means. Same run, + two different statistics — and under flat sampling there is no slope at all, + so the headline silently falls back to the mean. + +
+ +- [ ] You can say what fitting through the origin assumes, and why the slope is still less biased than the mean of the per-iteration averages. + +
Answer + + `Slope` is a one-field struct and `fit` returns `Σxᵢyᵢ / Σxᵢ²`, so the model + is `y = m·x`: it assumes total time is *exactly* cost × iters, i.e. zero + per-sample overhead. When overhead exists the slope is biased upward too — + it is not immune, and the chapter's three-point example shows it landing at + +21.4%. + + It is less biased because each point's influence is weighted by `xᵢ²`, so the + largest batches — where a fixed overhead is proportionally smallest — + dominate. `mean(avg_times)` weights every batch equally, so the *smallest* + batch, where overhead is proportionally largest, drags it up hardest: +30.6% + on the same three points. + +
+ +- [ ] You can give the reasons taking the minimum is the wrong estimator, not just a noisy one. + +
Answer + + It estimates best-case-ever, a state production code never runs in. Noise is + not purely additive — early samples can run at a *higher* pre-throttle clock, + so the min can be an unrepresentatively lucky sample. It is an extreme-value + statistic determined entirely by one observation, so its sampling + distribution is wild and sample-size-dependent, which is why no p-value can + be put on a change in it. And a bare point estimate hides whether the spread + was 1% or 40%. + +
+ - [ ] You can state what a bootstrapped confidence interval assumes about the distribution (nothing) and what it therefore cannot rescue you from. -- [ ] You can name criterion's two regression gates and say why "statistically significant" alone is not one of them. + +
Answer + + It assumes nothing about the *shape* of the noise — that is the point of + resampling the observed data instead of consulting a normal-distribution + table. What it does assume is that your sample is representative of the + population, and it cannot rescue you from a sample that isn't: a + systematically throttled machine, a benchmark measuring the wrong thing, + coordinated omission, or too few samples. Resampling a biased sample yields + a confidently narrow interval around the wrong number. + +
+ +- [ ] You can explain how pooling two samples and re-splitting them at random manufactures a null hypothesis, and what the p-value counts. + +
Answer + + Concatenating the new and baseline samples erases which run each value came + from. Drawing `n_a + n_b` values from that pool with replacement and slicing + them back into two groups produces a pair of samples that differ **by chance + alone** — exactly the "there is no real difference" world. Recomputing `t` + 100,000 times over that world gives the distribution of `t` under the null. + + The p-value is then a rank, not a probability computed from a formula: the + share of those chance-only `t` values at least as extreme as the observed + one — `min(hits, n − hits) / n × 2`, doubled because the test is two-tailed + (criterion asks "different?", not "slower?"). + +
+ +- [ ] You can name criterion's two regression gates, say which one runs first, and map each of the three printed verdicts to the gate that produced it. + +
Answer + + Gate 1 is the bootstrapped t-test: `p_value < significance_level` (0.05). + Gate 2 is the bootstrapped relative mean-change CI with **both** bounds past + ±`noise_threshold` (0.01). Gate 1 runs first and short-circuits. + + | Gate 1 | Gate 2 | Printed | + |---|---|---| + | fail | never evaluated | `No change in performance detected.` | + | pass | fail | `Change within noise threshold.` | + | pass | pass | `Performance has improved.` / `regressed.` | + +
+ - [ ] You have run `cargo bench` in `experiments/` and can point at the warm-up, sample and outlier lines in its output. +
Answer + + Three lines to find, from `report.rs:506`, `:538` and `:463` respectively: + `Benchmarking : Warming up for …` (Step 2's calibration loop); + `Benchmarking : Collecting 100 samples in estimated …` — the iteration + count in that line is `n(n+1)/2 × d` from Step 3; and + `Found N outliers among 100 measurements (…%)` followed by the + low/high × mild/severe breakdown, which is Step 6's Tukey fences reported, + never applied. + +
+ ## References -**Code** -- [criterion.rs](https://github.com/bheisler/criterion.rs) - `src/analysis/mod.rs` (locally: - `~/.cargo/registry/src/index.crates.io-*/criterion-0.5.1/src/analysis/mod.rs`, - 370 lines) — `common()` is the spine; follow the suggested reading - order above through `tukey.rs`, `regression.rs`, `compare.rs`, and - `routine.rs::warm_up` +**Code** — [criterion.rs](https://github.com/bheisler/criterion.rs) **v0.5.1** +(locally `~/.cargo/registry/src/index.crates.io-*/criterion-0.5.1/src/`). Every +line number in this chapter is from that version: + +| File | Lines | What | +|------|-------|------| +| `analysis/mod.rs` | 83, 124–140, 141, 152, 188, 200, 269, 300 | `common()` — sampling, the fork, tukey, linear guard, comparison, p-value, `regression()`, `estimates()` | +| `routine.rs` | 257, 158 | `warm_up`'s doubling loop; `met` | +| `lib.rs` | 427–433, 1362–1428 | defaults; sampling mode and the `d` formula | +| `stats/bivariate/regression.rs` | 20 | `Slope::fit` — least squares through the origin | +| `stats/univariate/outliers/tukey.rs` | 254 | `classify` and the fences | +| `stats/univariate/sample.rs` | 64, 171, 187 | MAD, the t-statistic, variance | +| `stats/univariate/mixed.rs` | 11 | the pooled two-sample bootstrap | +| `stats/mod.rs` | 63 | `p_value` | +| `analysis/compare.rs` | 72 | `t_test` | +| `report.rs` | 463, 598, 779 | outlier line; gate 1's test; gate 2 | diff --git a/topics/00-performance-toolbox/reading-drepper.md b/topics/00-performance-toolbox/reading-drepper.md index 5efb981..9a0bf74 100644 --- a/topics/00-performance-toolbox/reading-drepper.md +++ b/topics/00-performance-toolbox/reading-drepper.md @@ -2,91 +2,342 @@ Every latency table in topic 0 §2 is a compressed version of one 2007 paper — Drepper's "What Every Programmer Should Know About Memory". Before you open -its 114 pages, this chapter builds the eight concepts the paper assumes, one +its 114 pages, this chapter builds the ten concepts the paper assumes, one at a time — then hands you a section-by-section reading lens, and finally a table mapping each concept to the counter and the experiment that identify it in someone else's program. The DDR2 numbers are stale; the cache-organization math, the prefetching rules, and the measurement methodology behind `cache_ladder` are forever. +**Which numbers belong to which era.** Drepper measured on a Pentium 4, a +Pentium M and an early Core 2 in 2007. This repo measured on an Apple M3 Pro +in 2026. Both sets of numbers appear below and they are *never* mixed: a +figure sourced as **(§x.y)** or **(Fig x.y)** is Drepper's, quoted from +`cpumemory.pdf` version 1.0 (November 21, 2007) with the section, figure or +table it came from; a figure sourced as **(notes.md)** or **(FINDINGS row N)** +was measured by a lane in this repo on the M3 Pro on 2026-07-28. Where the repo +has measured the same quantity Drepper did, the repo's number is the one the +argument leans on, and his is kept beside it to show what changed. + ## The problem in one sentence -A modern core executes an instruction in ~0.3 ns, but fetching data from -main memory (DRAM) takes ~80–100 ns — roughly **300 instructions of waiting** -for one load. Everything in this paper is machinery to hide that gap, and -every database trick in later topics (columnar layouts, B-tree fanout, -vectorized execution) is a way of cooperating with that machinery. +The gap between an L1 hit and a DRAM access is a factor of **102×** on the +machine this repo runs on — `cache_ladder` measures 1.02 ns at a 16–128 KB +working set and 104 ns at 128 MB ([notes.md](notes.md)) — so a load that misses +everything costs about as much as a hundred loads that don't. Everything in +Drepper's paper is machinery to hide that gap, and every database trick in +later topics (columnar layouts, B-tree fanout, vectorized execution) is a way +of cooperating with that machinery. ## The concepts, step by step ### Step 1 — the speed gap, and why caches exist -Memory got bigger much faster than it got faster. The fix: put small, -fast memories *between* the core and DRAM, and keep recently-used data there. +> **In:** nothing yet — this step fixes the ladder every later step measures +> against, and separates Drepper's 2007 constants from this repo's 2026 ones. +> **Out:** two latency ladders, nineteen years apart, and the one number that +> did not change: the *ratio* between the top and the bottom. + +Memory got bigger much faster than it got faster. The fix: put small, fast +memories *between* the core and DRAM, and keep recently-used data there. A +**cache hit** means the line was found at that level; a **cache miss** means go +down one level and wait. A **working set** is the set of bytes a program is +actively touching at a given moment — the quantity every graph in the paper +plots along its X axis. + +Drepper's ladder, quoted exactly as §3.2 prints it (page 16, the unnumbered +table introduced with "These are the numbers Intel lists for a **Pentium M**"): + +``` +Drepper §3.2, page 16 — Intel's published Pentium M figures, 2007: + + To Where Cycles + Register ≤ 1 + L1d ~ 3 + L2 ~ 14 + Main Memory ~ 240 +``` + +This repo's ladder, measured by `cache_ladder` on an Apple M3 Pro +([notes.md](notes.md), "Experiment 1"): + +| Working set | ns/access (measured) | Level, and how we know | +|------------:|---------------------:|------------------------| +| 16 KB–128 KB | **1.02** | L1 — the plateau ends exactly at 128 KB, the P-core L1d size | +| 512 KB–1 MB | **5.3–5.8** | L2 | +| 4–8 MB | **7.6–9.0** | still L2 — Apple's per-cluster L2 is 16 MB-class | +| 16 MB | **17.1** | falling out of L2 into the SLC | +| 32 MB | **59.6** | SLC → DRAM transition | +| 64 MB | **87.4** | DRAM | +| 128–512 MB | **104–113** | DRAM plus a growing TLB-miss share (Step 8) | + +The **SLC** (system level cache) is Apple's shared last-level cache — the +structural replacement for the inclusive L3 in Drepper's Fig 3.2, sitting behind +every cluster's L2 rather than inside the CPU complex. Drepper's machines had no +level between L2 and DRAM at all, which is why his Fig 3.4 shows three plateaus +and the table above shows five. + +The two ladders are not comparable unit-for-unit — one is cycles on 2007 +hardware, the other is nanoseconds on a much wider core with a cache level +Drepper's machines did not have. What *is* comparable is the ratio across the +ladder. Drepper's: 240 ÷ 3 = **80×** from L1d to main memory. This repo's: +104 ÷ 1.02 = **102×**. Nineteen years and an instruction-set change later, the +shape is the same and the spread got slightly worse. + +That is the sentence to carry into the rest of the paper: **the constants aged; +the ratio did not.** The whole game is what fraction of your loads hit near the +top. + +### Step 2 — the cache line, and how much of it you actually use + +> **In:** the ladder from Step 1, which priced *one access*. +> **Out:** the unit that access actually moves — a fixed-size line — and a +> utilization fraction that Steps 4 and 6 both consume. + +Caches don't store individual bytes. They store **cache lines** — fixed-size +blocks, the smallest unit that ever moves between levels. Drepper §3.5.2 states +the sizes of his era plainly: "the cache line size is 64 or 128 bytes". His +measured machines all use **64 bytes** (he says so explicitly in §6.2.1: "with +64 bytes for the Core 2 processor"). Apple M-series uses **128 bytes** — a +number this repo did not take on faith but measured: topic 9 found that padding +contended counters to 64 B leaves them **1.8× slower** than padding to 128 B +([topic 9 notes.md](../09-concurrency/notes.md)), which only happens if the +coherence granule is 128. + +Load one byte and the hardware fetches the whole line it lives in. So the +question that decides a data layout is not "how many bytes do I read?" but +"how many bytes did the machine move to give me them?" + +**The formula.** For a loop that touches `e` bytes of every element, with +`s` bytes between the starts of consecutive touched elements (the **stride**), +on a machine with `L`-byte lines: ``` - size latency what it is - registers ~1 KB 0 cycles inside the core - L1 cache ~128 KB ~4 cycles per-core, split data/instruction - L2 cache ~4-16 MB ~14 cycles per-core or per-cluster - L3 / SLC ~24-48 MB ~40 cycles shared by all cores - DRAM GBs ~300 cycles the actual memory + n = max(1, floor(L / s)) elements whose bytes land inside one line + U = (e × n) / L fraction of each transferred line the loop uses + + e = bytes actually read per element + s = stride, bytes between consecutive touched elements + L = cache-line size (64 on Drepper's machines, 128 on Apple M-series) + n = elements per line + U = utilization — 1.0 means nothing was wasted ``` -A "cache hit" = found at that level. A "miss" = go down one level and wait. -The whole game is: what fraction of your loads hit L1? +Worked on five concrete cases. The first two are Drepper's own matrix +multiplication from §6.2.1, which is the whole argument of §6.2 in two lines of +arithmetic: -### Step 2 — the cache line: memory moves in fixed-size chunks +``` +1. §6.2.1 naive inner loop, mul2[k][j], N=1000 doubles, L=64: + e = 8 (one double) + s = 8 × 1000 = 8000 (the inner loop advances the ROW of mul2) + n = max(1, floor(64 / 8000)) = max(1, 0) = 1 + U = (8 × 1) / 64 = 0.125 → 12.5% used, 87.5% of every line wasted + +2. §6.2.1 after transposing mul2 into tmp[j][k], L=64: + e = 8, s = 8 + n = floor(64 / 8) = 8 + U = (8 × 8) / 64 = 64 / 64 = 1.0 → 100% used + +3. Fig 3.11 NPAD=7 — a list whose elements are one line wide, L=64: + e = 8 (the `n` pointer), s = 64 + n = floor(64 / 64) = 1 + U = 8 / 64 = 0.125 → 12.5% + +4. Row layout on M-series: one 8-byte column of a 128-byte row, L=128: + e = 8, s = 128, n = 1 + U = 8 / 128 = 0.0625 → 6.25% used, 93.75% wasted + +5. Column layout on M-series: the same column stored contiguously, L=128: + e = 8, s = 8 + n = floor(128 / 8) = 16 + U = (16 × 8) / 128 = 1.0 → 100% used +``` + +Cases 4 and 5 are topic 12 (columnar storage) in six lines of division: the +same filter, the same 8 bytes of answer, **16× fewer bytes moved**. Cases 1 and +2 are Drepper measuring the same effect on a Core 2 in 2007 — Table 6.2 records +the naive multiply at 16,765,297,870 cycles and the transposed one at +3,922,373,010, which is 23.4% of the original (the paper's own figure; the +division confirms 23.40%). The transpose *added* a full copy of a 1000×1000 +matrix and still won by 4.3×, because it moved case 1's utilization to case 2's. + +Two consequences to carry forward: + +- **Utilization has a floor, not a slope.** Once `s ≥ L`, `n` is pinned at 1 and + `U = e / L` no matter how much bigger the stride gets: 6.25% at stride 128 on + M-series, and still 6.25% at stride 4096. Growing the stride past one line + stops costing you *line* waste — after that it costs you prefetcher coverage + (Step 4) and pages (Step 8) instead. That is the answer to Question 2. +- **Neighbours are free.** Once the line is in L1, the other 120 bytes cost + nothing. Sequential scans exploit this; pointer chasing throws it away. -Caches don't store individual bytes. They store **lines** — fixed 64-byte -blocks (128 bytes on Apple M-series). Load 1 byte and the hardware fetches -the whole line it lives in. +### Step 3 — where can a line live? Sets, ways, and conflict misses + +> **In:** the cache line from Step 2, which now needs somewhere to go. +> **Out:** the address arithmetic that places it, and the third kind of miss — +> the one you can cause on purpose, which the profiler table's row 3 exploits. -Two consequences that shape databases: +A cache can't compare every line's address on every load — that would be too +slow. So it is organized like a hash table with fixed-size buckets: some middle +bits of the address pick a **set** (the bucket), and each set holds `N` lines. +`N` is the **associativity**, and a cache holding `N` lines per set is called +**N-way set-associative**. A new line evicts one of the `N` residents *of its +own set only*. -- **Touching 8 bytes costs a full line.** Filter on one 8-byte column of a - wide row and you waste 94% of every transfer: +Drepper §3.3.1 gives the identity that connects the three: ``` -filter on one 8-byte column, 128 B cache lines (M-series): +Drepper §3.3.1, page 19: -row layout: line = [ a │ b c d e f g ... padding ... ] use 8 B / 128 B → 94% wasted -col layout: line = [ a a a a a a a a a a a a a a a a ] use 128 B → 0% wasted + cache size = cache line size × associativity × number of sets + + O = log2(cache line size) bits of the address used as the line offset + S = log2(number of sets) bits of the address used as the set index ``` - That's topic 12 (columnar storage) in one diagram — Drepper's Fig 3.11. +**Worked on his own example** (§3.3.1 states the answers, so this checks both +the formula and the transcription): -- **Neighbors are free.** Once the line is in L1, the other 120 bytes cost - nothing. Sequential scans exploit this; pointer chasing throws it away. +``` +Drepper's 4 MB, 64-byte-line, 8-way L2: + number of sets = 4,194,304 / (64 × 8) = 4,194,304 / 512 = 8,192 sets + S = log2(8,192) = 13 bits + (§3.3.1, page 19, verbatim: "Given our 4MB/64B cache and 8-way set + associativity the cache we are left with has 8,192 sets and only 13 bits + of the tag are used in addressing the cache set.") + 8 tags compared in parallel per lookup (§3.3.1: "8 tags have to be compared") + +The stride that thrashes it — the set-index period: + period = number of sets × line size = 8,192 × 64 = 524,288 B = 512 KB + ⇒ any two addresses exactly 512 KB apart share a set + ⇒ nine addresses at that spacing overflow an 8-way set: 9 > 8, so every + touch evicts one that will be needed again. The cache is 4 MB and the + working set is nine lines = 576 bytes. + +The same arithmetic for this repo's machine (L1d = 128 KB measured, notes.md; +128-byte lines measured, topic 9 notes.md; 8-way ASSUMED, not measured): + number of sets = 131,072 / (128 × 8) = 128 sets + period = 128 × 128 = 16,384 B = 16 KB + ⇒ a row stride of exactly 16 KB is the pathological one to construct +``` -### Step 3 — where can a line live? Sets, ways, and conflict misses +That last block is where the profiler table's "pad the stride by one line +(4096 → 4096+128)" advice comes from: shifting every row by one line walks the +set index forward instead of repeating it. + +This gives the three miss types their vocabulary: -A cache can't search all its lines on every load — that would be too slow. So -it's organized like a **hash table with fixed-size buckets**: some middle -bits of the address pick a **set** (the bucket), and each set holds N lines -(**N-way associative**, typically 8–16). A new line evicts one of the N -residents of *its own set only*. +- **Cold (compulsory)** — first touch of that line, unavoidable. +- **Capacity** — the working set is simply bigger than the cache. +- **Conflict** — the *set* is full even though the *cache* isn't, as in the + 576-byte working set above. -This gives the three miss types a vocabulary: +Drepper measures how much associativity buys, in Table 3.1 (L2 misses for a +`gcc` run, 32-byte lines). He asserts the trend; the divisions are ours: + +``` +Drepper Table 3.1, 8 MB cache, CL=32 — misses, and the saving from each doubling: + + direct → 2-way: 4,731,904 → 2,690,498 saved 2,041,406 / 4,731,904 = 43.1% + 2-way → 4-way: 2,690,498 → 2,207,655 saved 482,843 / 2,690,498 = 17.9% + 4-way → 8-way: 2,207,655 → 2,111,075 saved 96,580 / 2,207,655 = 4.4% +``` -- **cold** — first touch, unavoidable -- **capacity** — working set simply bigger than the cache -- **conflict** — the set is full even though the cache isn't (bucket - collision: many hot addresses hash to the same set, e.g. a stride that - equals the set-index period) +§3.3.1 calls the first step "almost 44%" — the division on his own table gives +43.1%, close enough that the rounding is the only disagreement. The three +numbers together are the honest version of his prose claim that "the successive +gains are much smaller": the second doubling is worth 2.4× less than the first, +and the third is worth 10× less. For the associativity levels of 2007, §3.3.1 +says "Today processors are using associativity levels of up to 24 for L2 caches +or higher. L1 caches usually get by with 8 sets." — where "8 sets" is a slip for +*8 ways*, as the surrounding paragraph about comparators makes clear. Quote the +number, not the word. ### Step 4 — the prefetcher: hardware that bets on your next load -The memory system watches your access pattern. Sequential or fixed-stride -loads are detected and the *next* lines are fetched before you ask — -hiding DRAM latency entirely. The bet fails on random access: the prefetcher -has nothing to extrapolate, so every miss pays full price. +> **In:** the access pattern implied by Step 2's stride, laid out over Step 3's +> sets. +> **Out:** the measured size of the latency the hardware hides for free, and +> the exact list of patterns it cannot hide — the premise Step 5 removes. + +**Prefetching** is the memory system speculatively fetching lines you have not +asked for, on the bet that your pattern will continue. Drepper §6.3.1 states +the rules his era's hardware followed, and they have not fundamentally changed: + +- The trigger is "a sequence of **two or more** cache misses in a certain + pattern" — one miss never starts a prefetch, because random accesses to + globals are common and would waste bandwidth. +- Fixed strides are recognized, not just adjacent lines, but the recognition + range is bounded: §6.3.1 says the range "has been increased over the years, + but it probably does not make much sense to go beyond the 512 byte window + which is often used today". Treat ~512 B as the outer edge of a stride the + hardware will follow. +- "CPUs today can keep track of **eight to sixteen** separate streams" for the + higher-level caches — and that budget is *shared* with every other core and + hyper-thread on the same cache. +- "Prefetching has one big weakness: **it cannot cross page boundaries**", + because a speculative fetch must never trigger a page fault the program did + not ask for. So you take a miss at every page boundary regardless. +- "Currently prefetch units do not recognize non-linear access patterns." + +**How much is it worth?** Drepper answers this with an accident of Fig 3.10 that +is easy to miss. On his Pentium 4 (16 kB L1d, 1 MB L2), a sequential walk over +a linked list costs about **4 cycles** per element inside L1d, and then — past +the 1 MB L2, where every access is going to main memory — it costs about +**9 cycles** per element. §3.3.2 spells out the comparison itself: "Before we +said that a main memory access takes 200+ cycles. Only with effective +prefetching is it possible for the processor to keep the access times as low as +9 cycles." + +``` +Drepper §3.3.2, Fig 3.10 — what sequential prefetching is worth, 2007: + unhidden main-memory access 200+ cycles (§3.3.2's own figure) + measured sequential walk 9 cycles (Fig 3.10, working set > L2) + hidden 200 / 9 = 22× +``` + +A second detail in the same figure is stranger and more instructive: in the L2 +range the walk shows ~9 cycles per element when the L2's own access latency is +~14 (§3.2's table). The walk is *faster than the cache it is reading from*, +because the next line is already halfway loaded when the loop reaches it. +Prefetching does not just avoid the DRAM trip; it removes the L2 trip from the +critical path. + +The bet fails on random access, and Fig 3.15 measures the failure: + +``` +Drepper Fig 3.15 (§3.3.2), same list, same machine, order shuffled: + sequential, working set ≫ L2 ~9 cycles/element (also Fig 3.10) + random, working set ≫ L2 450+ cycles/element + gap 450 / 9 = 50× + +This repo, Apple M3 Pro, the same comparison in nanoseconds: + streaming scan, topic 12 scan_bench: 800 MB / 0.014 s = 57.1 GB/s + ⇒ one 128 B line every 128 / 57.1e9 = 2.24 ns + dependent random chase, cache_ladder at 128 MB: 104 ns per line + gap 104 / 2.24 = 46× +``` -This is why "sequential vs random" is the single most important distinction -in the topic 0 latency table — same data, same cache, ~10× difference. +Two machines nineteen years apart, and the sequential-vs-random gap is 50× on +one and 46× on the other. This is the correction to a folk figure the earlier +version of this chapter repeated: the gap is **not** "about 10×". It is about +**50×**, and it has been about 50× since 2007. + +Note what Drepper says about *why* random loses even when both are going to +DRAM: §3.3.2 attributes it to three stacked causes — no prefetch, a rising L2 +miss ratio (Table 3.2 puts random at 13.4% miss at a 1 MB working set against +sequential's 0.94%, and 57.8% against 4.67% at 512 MB), and TLB misses (Step 8, +and Fig 3.17, where limiting the randomization to page-sized blocks recovers +"up to 38%"). ### Step 5 — dependent loads: the one latency you cannot hide +> **In:** Step 4's finding that the prefetcher covers predictable patterns. +> **Out:** the cost of a miss when *nothing* can cover it — measured twice on +> the same machine, 11× apart — plus the instrument that measures it, which +> Steps 6 and 8 both reuse. + A load is **dependent** when the core cannot compute its address until an earlier load has come back. `chain[idx]` where `idx` was itself loaded from memory; `node->next->next`; a B-tree descent, where the child pointer lives @@ -94,12 +345,18 @@ inside the parent node you are still waiting for. Why that one word decides everything: an out-of-order core does not run one load at a time. It keeps hundreds of instructions in flight and issues *every* -load whose address it already knows, so several misses sit in the memory -system simultaneously. This is **memory-level parallelism (MLP)**, and it -means the cost of a miss is not a property of the miss — it is a property of -how many other misses could keep it company. +load whose address it already knows, so several misses sit in the memory system +simultaneously. This is **memory-level parallelism (MLP)** — the number of +outstanding misses the machine is servicing at once — and it means the cost of +a miss is not a property of the miss. It is a property of how many other misses +could keep it company. Drepper puts the same point at the head of §6.3: "To +cover the latency of main memory accesses, the command queue would have to be +incredibly long." ``` +ILLUSTRATION — round numbers, not measurements. The measured version of this +diagram is the table immediately below it. + independent loads — a[0], a[1], a[2]: all three addresses computable right now load A ├──────── ~100 ns ────────┤ @@ -115,83 +372,246 @@ dependent loads — B's address IS the value A returned └─► ~300 ns total ⇒ 100 ns each ``` -Same cache, same DRAM, same number of misses — 3× apart here, and ~10× apart -in practice once the out-of-order window is full. Note what the diagram -implies: **latency and bandwidth are different questions.** A chase leaves the -memory bus nearly idle (one 128-byte line per ~104 ns ≈ 1.2 GB/s, against the -24–57 GB/s a single core reaches on a streaming scan in topic 12); it is slow -while doing almost nothing. Nothing you can do about the bus will help it. - -This repo measured both sides of the diagram on the same machine -([`notes.md`](notes.md)): +This repo measured both sides of that diagram on the same machine +([notes.md](notes.md)): | what | working set | ns per access | |------|------------:|--------------:| | `lookup_shootout` `hashmap` at n=1e7 — 1024 **independent** probes | ~160 MB | **9.3** | | `cache_ladder` at 128 MB — a **dependent** chase | 128 MB | **104** | -Both are random DRAM accesses that "should" cost ~100 ns. The 11× gap is -overlap and nothing else. That is why the hash table looked suspiciously flat -at ten million keys, and why a *single* isolated lookup in the capstone would -not enjoy the same number. +Both are random DRAM accesses that "should" cost ~100 ns. 104 ÷ 9.3 = **11.2×**, +and the difference is overlap and nothing else. That is why the hash table +looked suspiciously flat at ten million keys (7.4 ns at n=100 to 9.3 ns at +n=1e7, [notes.md](notes.md)), and why a *single* isolated lookup in the capstone +would not enjoy the same number. The chase is how you measure the un-hidable case. Three properties of -[`cache_ladder`](experiments/benches/cache_ladder.rs) do the work: +[`cache_ladder`](experiments/benches/cache_ladder.rs) do the work, and the file +is short enough to read whole: + +```rust +// topics/00-performance-toolbox/experiments/benches/cache_ladder.rs — chase, 25-31 + 25 fn chase(chain: &[usize], start: usize, steps: usize) -> usize { + 26 let mut idx = start; + 27 for _ in 0..steps { + 28 idx = chain[idx]; + 29 } + 30 idx + 31 } +``` + +Line 28 is the entire experiment. The value loaded *is* the next address, so the +dependency lives in the data, not in the code: it cannot be reordered, hoisted +or speculated around by any compiler or any core, because the next address +genuinely does not exist yet. One miss in flight, always. Line 30 returns `idx` +so the loop is not dead code. + +The second property is in the chain's construction: + +```rust +// topics/00-performance-toolbox/experiments/benches/cache_ladder.rs — make_chain, 14-23 + 14 fn make_chain(len: usize, rng: &mut StdRng) -> Vec { + 15 let mut order: Vec = (0..len).collect(); + 16 order.shuffle(rng); + 17 let mut chain = vec![0usize; len]; + 18 for w in order.windows(2) { + 19 chain[w[0]] = w[1]; + 20 } + 21 chain[order[len - 1]] = order[0]; + 22 chain + 23 } +``` + +Line 16 makes it **random**, which kills the prefetcher (Step 4 — no stride to +extrapolate, and no non-linear pattern is recognized at all). Line 21 makes it +**cyclic** — it closes the permutation into a single cycle covering every slot, +so no short sub-cycle can quietly live in L1 and flatter the big sizes. The +generator is seeded (`StdRng::seed_from_u64(42)`, line 36), so the chain is the +same on every run. + +The third property is the one that was originally wrong: ```rust -fn chase(chain: &[usize], start: usize, steps: usize) -> usize { - let mut idx = start; - for _ in 0..steps { - idx = chain[idx]; // the value loaded IS the next address - } - idx // returned so the loop isn't dead code -} -``` - -1. **The dependency is in the data, not the code.** `idx = chain[idx]` cannot - be reordered, hoisted, or speculated around by any compiler or any core — - the next address genuinely does not exist yet. One miss in flight, always. -2. **`chain` is a random cyclic permutation** (Sattolo's algorithm). Random - kills the prefetcher (Step 4) so nothing arrives early; *cyclic* — one cycle - through every slot — stops a short sub-cycle from quietly living in L1 and - flattering the big sizes. -3. **`idx` is carried across criterion iterations.** The first version of this - benchmark restarted at `idx = 0` each iteration, re-walked the same 65,536 - slots, and reported ~25 ns for "DRAM" — it had measured an ~8 MB hot path - that the benchmark itself created. The fix is in the source comment; - the confession is in `notes.md`. +// topics/00-performance-toolbox/experiments/benches/cache_ladder.rs — the bench closure, 50-57 + 50 // Carry the position across iterations: restarting at 0 every iter + 51 // re-walks the same `steps` slots, which stay cached — at 512MB that + 52 // silently measures an ~8MB hot path instead of DRAM. + 53 let mut idx = 0usize; + 54 b.iter(|| { + 55 idx = chase(black_box(chain), idx, steps); + 56 black_box(idx) + 57 }) +``` + +Line 53 sits *outside* `b.iter`, which is the whole fix. The first version of +this benchmark declared `idx` inside the closure, so every criterion iteration +re-walked the same 65,536 slots (`steps`, line 43) and reported **~25 ns** for +"DRAM" — it had measured an ~8 MB hot path that the benchmark itself created +([notes.md](notes.md), "First version lied"). The correct answer at 512 MB is +113 ns. A benchmark that creates the cache residency it then measures is topic +0's headline failure mode, and this file is where the repo committed it. The readout is unusually direct: with no arithmetic between the loads, `elapsed / steps` **is** the latency of one access at that working-set size. -Sweep the size from 16 KB to 512 MB and the plateaus are the cache levels — -this is Drepper's Fig 3.4, and it is the only number in this repo you can -compare to a datasheet without an argument. +Sweep the size from 16 KB to 512 MB (line 39-42) and the plateaus are the cache +levels. This is the same experiment as Drepper's Fig 3.4 — cycles per operation +against working-set size, with the levels readable off the plateaus — and it is +the only number in this repo you can compare to a datasheet without an argument. The database consequence is the whole curriculum: pointer-chasing layouts (linked lists, naive trees, record-per-node graph stores) pay full latency per hop, while layouts that expose addresses up front (arrays, matrices, page-sized nodes, batched lookup APIs) convert latency into throughput. When a later topic says a design "exposes memory-level parallelism", it means: the -addresses are knowable early enough to overlap the waiting. +addresses are knowable early enough to overlap the waiting. Topic 3 is this +step's bill arriving — B-tree lookups climb **862 → 1101 ns** from 1e6 to 4e6 +keys while the tree's height stays at 3 ([FINDINGS row 3](../../FINDINGS.md)). +Height sets how many pointers you chase; residency sets what each chase costs. + +### Step 6 — latency and bandwidth are opposite questions + +> **In:** the two measured numbers from Step 5 — 9.3 ns overlapped and 104 ns +> serialized — plus Step 2's utilization fraction. +> **Out:** the ratio that says which of the two walls a loop is against, run on +> four of this repo's lanes. The profiler table at the end of the chapter is +> this step applied to code you did not write. + +Look again at Step 5's diagram and notice what it implies about the *bus*. A +dependent chase moves one line per 104 ns and then stops to think. That is +128 ÷ 104e-9 = **1.23 GB/s**, on a machine whose peak memory bandwidth is +150 GB/s ([topic 12 notes.md](../12-columnar-analytics/notes.md)). It is slow +while doing almost nothing. Nothing you can do about the bus will help it. + +Two definitions, because these words get used interchangeably and must not be. +**Latency-bound** means the loop is waiting on the *round trip* of an access it +could not start earlier; the fix is more overlap (more independent addresses, +batched APIs, prefetch hints), and the bus is idle while it happens. +**Bandwidth-bound** means the loop has already saturated the *rate* at which +bytes can arrive; the fix is moving fewer bytes (Step 2's utilization), and more +overlap does nothing at all. + +**Utilization: how close to the wall is this lane?** + +``` + B_peak = 150 GB/s this machine's peak memory bandwidth (topic 12 notes.md) + B_ach = Q / t bytes the lane moved, divided by how long it took + Util = B_ach / B_peak the fraction of the bus the lane is using + +lane Q / t B_ach Util +topic 12 scan_bench, small-range random 800 MB / 0.014 s 57.1 GB/s 38.1% +topic 12 scan_bench, sorted low-card 800 MB / 0.033 s 24.2 GB/s 16.2% +topic 0 lookup_shootout hashmap n=1e7 128 B / 9.3 ns 13.8 GB/s 9.2% +topic 0 cache_ladder at 128 MB 128 B / 104 ns 1.23 GB/s 0.82% + + FINDINGS row 12 states the top two rows as the headline: "The scan floor is + 24–57 GB/s on a 150 GB/s machine." +``` + +(The two `lookup_shootout` and `cache_ladder` rows assume one 128-byte line per +access, which is a floor — a hash probe may touch a second line. All four `Q` +and `t` values come from [notes.md](notes.md) and +[topic 12 notes.md](../12-columnar-analytics/notes.md). The `B_ach` column is +this chapter's own division on the times those files print, so it differs from +the GB/s those files print by the rounding in `t`: 57.1 here against the lane's +57.0, and 24.2 against 24.4. Topic 12's notes also warn that repeat runs put +this lane anywhere from 24 to 76 GB/s depending on machine state, so treat the +utilization column as an order of magnitude, not a constant — which is all the +memory-bound-vs-latency-bound question needs.) + +Read the column. The columnar scan at 38% of peak on a single core is against +the bandwidth wall; buying it more overlap is pointless, and topic 12 spends its +whole chapter moving fewer bytes instead. The pointer chase at 0.82% is against +the latency wall with 99% of the bus sitting idle; it does not need a better +layout, it needs more addresses known earlier. The hash probe at 9.2% is the +interesting one — already overlapping 11× better than the chase (Step 5), and +still using less than a tenth of the bus, which is why Question 4 asks you to +prove there is more MLP left in it. + +**Arithmetic intensity: at what point does a loop stop being memory-bound?** + +``` + W = useful operations the kernel performs + Q = bytes it must move + I = W / Q arithmetic intensity, operations per byte + P = ops/s the core can retire + B = bytes/s the memory system can sustain for this kernel + I* = P / B the RIDGE POINT — the machine's balance + + the loop is memory-bound iff I < I* +``` + +Run it on topic 12's `scan_bench` lane, which folds 100 M `u64` with +`wrapping_add`: + +``` + W = 100,000,000 adds + Q = 100,000,000 × 8 = 800,000,000 bytes + I = 100e6 / 800e6 = 0.125 adds per byte + + B = 57.1 GB/s the best single-core streaming bandwidth this repo has + MEASURED (topic 12 notes.md, small-range random lane) + P = 16e9 adds/s ASSUMED: 4 u64 adds per cycle at 4.0 GHz. This repo has + not measured peak scalar issue rate, so this is a stated + assumption, not a figure. + + I* = 16e9 / 57.1e9 = 0.28 adds per byte + I = 0.125 < I* = 0.28 ⇒ memory-bound, by a factor of 0.28 / 0.125 = 2.24× + + What would it take to reach the ridge? + ops per 8-byte element at the ridge = I* × 8 = 0.28 × 8 = 2.24 + the lane does 1. It would have to more than double its work per element + before bandwidth stopped being the limit. + + Sensitivity, because P was assumed rather than measured: + at 8 adds/cycle, I* = 32e9 / 57.1e9 = 0.56 ⇒ still memory-bound, by 4.5× + at 2 adds/cycle, I* = 8e9 / 57.1e9 = 0.14 ⇒ still memory-bound, by 1.1× + The conclusion does not flip anywhere in that range, which is the point of + computing a ridge instead of asserting one. +``` -### Step 6 — virtual memory: every address you use is fake +That last block is topic 0 §4's roofline paragraph with the division actually +performed. It is also why the profiler flowchart below asks "achieved GB/s near +the machine's peak?" before it asks anything about TLBs: the answer to that one +question separates two diagnoses whose fixes are opposites. + +### Step 7 — virtual memory: every address you use is fake + +> **In:** every address Steps 1–6 loaded from, which they all quietly assumed +> was a real place in DRAM. +> **Out:** the translation those addresses need, priced in dependent loads — +> Step 5's chain, running in silicon in front of your access. Step 8 caches it. + +Every pointer your program holds is a **virtual address** — a number meaningful +only inside your process. The physical DRAM location is decided by the OS, +which gives each process its own address space, maps pages lazily (your `Vec` +allocation may have no physical memory behind it until first touch), shares +pages between processes, and backs some of them with files. So *every* load +needs a translation: virtual page → physical frame. The map is the **page +table**, and it lives, awkwardly, in memory itself. + +**Why a tree and not an array.** Drepper does this arithmetic in §4.2 for his +era: with 4 kB pages on a 32-bit machine the offset is 12 bits, leaving 20 bits +of page number, so a flat table is 2²⁰ entries × 4 bytes = **4 MB** per process — +and "with each process potentially having its own distinct page directory much +of the physical memory of the system would be tied up". The same arithmetic on +this machine is far worse: -Every pointer your program holds is a **virtual address**. The physical DRAM -location is decided by the OS — which gives each process its own address space, -maps pages lazily (your `Vec` allocation may have no physical memory behind it -until first touch), shares pages between processes, and backs some of them with -files. So *every* load needs a translation: virtual page → physical frame. The -map is the **page table**, and it lives, awkwardly, in memory itself. +``` + 47-bit user address space, 16 KB pages (Apple M-series): + pages = 2^47 / 2^14 = 2^33 = 8,589,934,592 pages + flat = 2^33 × 8 bytes = 2^36 = 64 GB of page table, per process +``` -**Why a tree and not an array.** A flat lookup table would be one entry per -page: a 47-bit address space with 16 KB pages is 2³³ pages × 8 bytes = **64 GB -of table per process**. Unaffordable. So it is a **radix tree**: the virtual -address is chopped into fixed-width slices, each slice indexes one level, and -only the sub-tables that actually have mappings are ever allocated. An idle -process's page table is a few KB. You pay for sparsity with depth — and depth -here means *loads*. +Unaffordable, so it is a **radix tree**: the virtual address is chopped into +fixed-width slices, each slice indexes one level, and only the sub-tables that +actually have mappings are ever allocated. An idle process's page table is a +few KB. You pay for sparsity with depth — and depth here means *loads*. +Drepper's §4.2 puts the same trade in one sentence, typo and all: "The level +then form a huge, sparse page directory; address space regions which are not +actually used do not require allocated memory." -**How the address is chopped** (x86-64, 4 KB pages — the paper's case): +**How the address is chopped** (x86-64, 4 kB pages — the paper's case, Fig 4.2): ``` 47 39 38 30 29 21 20 12 11 0 @@ -208,10 +628,13 @@ here means *loads*. │ + offset ──► your data, at last └── CR3: physical address of this process's top table (swapped on context switch) -why 9 bits: one table is one 4 KB page = 4096 / 8 = 512 entries = 2⁹ +why 9 bits: one table is one 4 kB page = 4096 / 8 = 512 entries = 2⁹ + (Drepper §4.2: "on x86-64 with 4kB pages and 512 entries per directory") why dependent: each entry holds the *physical address of the next table*, so load N+1's address is unknown until load N returns — Step 5's chain, in - silicon, running BEFORE your actual access can even issue + silicon, running BEFORE your actual access can even issue. §4.3: "These + accesses cannot be parallelized since they depend on the previous lookup's + result." ``` The same tree, three vocabularies for the same four levels — you will meet all @@ -224,86 +647,302 @@ three while reading: | ↓ | PD | PMD | L2 | | leaf | PT | PTE | L3 | -**Apple Silicon is shallower, because its pages are bigger.** With a 16 KB +**Apple Silicon is shallower, because its pages are bigger.** With a 16 kB granule a table holds 16384 / 8 = 2048 entries = **11 bits** of index, and the offset takes 14 bits. So `14 + 11 + 11 + 11 = 47` — a 47-bit user address space is covered in **three** levels, not four. Bigger pages buy a shorter walk *and* -4× the TLB reach (Step 7) from the same entry count. +4× the TLB reach (Step 8) from the same entry count. Drepper predicted exactly +this in §4.3.2: "There is a second effect of using larger page sizes: the number +of levels of the page table tree is reduced." + +**What a walk costs.** §4.3 prices the best case in his own numbers, and the +paper states the answer, so this checks the transcription: + +``` +Drepper §4.3, page 38: + "on a machine with four page table levels, require at the very least 12 cycles" + ⇒ 4 levels × 3 cycles (§3.2's L1d figure) = 12 cycles, if every level hits L1d + +The other end of the range, using §3.2's own Main Memory figure: + 4 levels × 240 cycles = 960 cycles, if every level misses to DRAM + +The same two bounds on this machine, with three levels and MEASURED latencies +(notes.md): + all-hit: 3 × 1.02 ns = 3.1 ns + all-miss: 3 × 104 ns = 312 ns +``` **Four things keep this from being catastrophic**, and knowing them is the difference between fearing the diagram and predicting it: - **Hardware walks it, not the kernel.** The MMU's page-table walker does those - loads in silicon, costing nanoseconds. The kernel only gets involved when - there is no valid entry — a **page fault**, which is microseconds, a - thousand-fold different event. + loads in silicon, costing nanoseconds; Drepper §4.2 notes x86 and x86-64 "perform + this operation in hardware". The kernel only gets involved when there is no + valid entry — a **page fault**, which is microseconds. §6.2.4 makes the + relative sizes explicit — and then immediately qualifies it, which is the part + usually dropped: "Page faults are orders of magnitude more expensive than TLB + misses but, if a program is running long enough and certain parts of the + program are executed frequently enough, TLB misses can outweigh even page fault + costs." - **The tables are ordinary cacheable memory.** The upper levels are touched by - every access in the region, so they normally sit in L1/L2; only the leaf level - is likely to be cold. + every access in the region, so they normally sit in L1/L2 — which is what + makes the 12-cycle bound the realistic one and the 960-cycle bound the + pathological one. - **There are dedicated page-walk caches** (x86 paging-structure caches, ARM walk caches) holding partial translations, so a walk often skips its first - levels entirely. + levels entirely. This is post-2007 hardware; the paper does not describe it. - **Huge pages truncate the walk.** An entry at the PMD/L2 level can be a *block* descriptor pointing straight at 2 MB of contiguous physical memory - (32 MB with a 16 KB granule) instead of at another table — one fewer load, - and one TLB entry covering 512× more address space. - -So: worst case is ~3–4 dependent DRAM loads *added in front of* your access; -typical case is far less. The measurement is in `notes.md` and it lands where -this predicts — `cache_ladder`'s tail rises **87 → 113 ns** from 64 MB to -512 MB, an added ~26 ns per access once 32K pages overflow the TLB. Not the -+400 ns of a fully cold walk, not zero either. That +26 ns is this diagram, -priced. - -### Step 7 — the TLB: a cache for translations, with tiny reach - -Doing that 4-load walk per access would be absurd, so translations are -cached in the **TLB** (translation lookaside buffer). The catch is -**reach**: ~2K entries × 4 KB pages ≈ only a few MB of address space covered. -Working sets beyond that miss in the TLB *as well as* the caches — the two -penalties stack. This is why databases care about **huge pages** (2 MB/1 GB -pages multiply reach by 512×; Apple's 16 KB base pages already 4× it). - -### Step 8 — multiple cores: coherency and false sharing - -Each core has its own L1/L2, so hardware keeps copies **coherent**: writing -a line invalidates every other core's copy of it. The pathology is **false -sharing** — two threads writing *different* variables that happen to share -one line. The line ping-pongs between cores at ~100-cycle cost per bounce, -and multi-thread scaling collapses with no visible reason in the source. -Padding each thread's data to its own line fixes it. (This pays off in -topic 9, concurrency.) + (32 MB with a 16 kB granule) instead of at another table — one fewer load, and + one TLB entry covering far more address space (Step 8 does that division). -## How to read the paper (with the concepts in hand) +So: worst case is three or four dependent DRAM loads *added in front of* your +access; typical case is far less. The measurement is in +[notes.md](notes.md) and it lands where this predicts: + +``` + cache_ladder's tail, 64 MB → 512 MB: 87.4 → 113 ns + added per access: 113 − 87.4 = 25.6 ns + as a fraction of a fully cold 3-level walk: 25.6 / 312 = 8.2% +``` + +Not the 312 ns of three cold DRAM loads, not zero either. 8.2% of a cold walk is +what it looks like when the upper levels are cached and the walk caches are +doing their job. That +25.6 ns *is* this diagram, priced. + +### Step 8 — the TLB: a cache for translations, with tiny reach + +> **In:** the walk from Step 7, which is far too expensive to do per access. +> **Out:** the reach arithmetic, and the second cliff it puts in Step 1's +> ladder — the one that explains `cache_ladder`'s last two rows. + +Doing that three-or-four-load walk per access would be absurd, so completed +translations are cached in the **TLB** (translation lookaside buffer) — a small, +very fast cache holding virtual-page → physical-frame results. Drepper §4.3 +describes what is stored precisely, and the detail matters: it is not the +directory entries that are cached but "the complete computation of the address +of the physical page", tagged by the virtual address minus its offset bits. + +The catch is **reach** — the total amount of address space the TLB's entries can +cover at once: + +``` + Reach = entries × page size + +Drepper's measurement (§3.3.2, Fig 3.12 — one 64-byte list element per page): + the spike appears when the working set reaches 2^13 bytes + 2^13 / 64 = 128 elements = 128 pages, against 2^12 / 64 = 64 pages just below + §3.3.2's conclusion: "we can compute that the TLB cache has 64 entries" + Reach = 64 × 4,096 = 262,144 B = 256 KB + +This machine (notes.md), where the cliff is measured rather than the entry count: + 512 MB / 16 KB pages = 32,768 pages in the working set + 64 MB / 16 KB pages = 4,096 pages + cache_ladder: 87.4 ns at 64 MB → 113 ns at 512 MB, i.e. +25.6 ns per access + ⇒ reach lies somewhere between those two page counts, and the ladder's last + two rows are the cost of exceeding it +``` + +Working sets beyond reach miss in the TLB *as well as* in the caches, and the +two penalties stack — Drepper §3.3.2 is explicit that "the address translation +penalties are additive to the memory access times", which is why Fig 3.11's +NPAD=31 curve exceeds the machine's own DRAM latency. + +This is why databases care about **huge pages** — pages larger than the OS +default, which multiply reach without needing more TLB entries: + +``` + 4 kB → 2 MB pages: 2,097,152 / 4,096 = 512× the reach + 4 kB → 1 GB pages: 1,073,741,824 / 4,096 = 262,144× the reach + 4 kB → Apple's 16 kB base page: 16,384 / 4,096 = 4× the reach, for free, + on every process, with no configuration at all +``` + +Drepper §4.3.2 lists the cost of the 2 MB version honestly and then names the +one workload it is worth it for: the pages "must be contiguous in physical +memory", which means "finding a free area with 512 contiguous pages ... can be +extremely difficult (or impossible) after the system runs for a while"; on Linux +of that era they had to be reserved at boot via `hugetlbfs`. His conclusion: +"huge pages are the way to go in situations where performance is a premium, +resources are plenty, and cumbersome setup is not a big deterrent. **Database +servers are an example.**" + +### Step 9 — coherence: why a write is a bus event + +> **In:** Step 1's observation that each core has its own L1/L2 — which Steps +> 2–8 never had to think about, because they were single-threaded. +> **Out:** the protocol that keeps those private caches consistent, and the +> message that makes a write expensive. Step 10 is what happens when you +> trigger it by accident. + +Each core has its own L1 (and often L2), so the same line can exist in several +places at once. **Cache coherence** is the hardware guarantee that all those +copies agree: a program cannot observe two different values for one address. +Drepper §3.3.4 describes the protocol every mainstream machine uses, **MESI**, +named for the four states a line can be in: + +- **Modified** — this core has changed the line; it is the only copy anywhere. +- **Exclusive** — unmodified, and known to be in no other core's cache. +- **Shared** — unmodified, and possibly present in other cores' caches. +- **Invalid** — unused. + +The expensive transition has a name. When a core wants to write a line that +other cores may hold, it must first take exclusive ownership by broadcasting a +**Request For Ownership (RFO)** — a message that invalidates every other copy. +§3.3.4 calls it "the infamous ... (RFO) operation" — the elided words are +*Request For Ownership* in the paper's own quotation marks — and notes +that "performing this operation in the last level cache ... is comparatively +expensive". Two situations produce them: a thread migrating between cores, and a +line genuinely needed by two cores. + +The consequence for a write-heavy multithreaded loop: §6.4.1 states it in one +line — "if multiple threads write to a memory location, the cache line must be +in 'E' (exclusive) state in the L1d of each respective core. This means that a +lot of RFO messages are sent, in the worst case one for each write access. So a +normal write will be suddenly very expensive." + +Drepper also measures the aggregate effect on his four-processor box, Table 3.3 +(speed-up at the largest working set, where the theoretical limits are 2 and 4): + +``` +Drepper Table 3.3 (§3.3.4) — measured speed-up, largest working set: + + #Threads Seq Read Seq Inc Rand Add + 2 1.69 1.69 1.54 + 4 2.98 2.07 1.65 + + Random-access work scales 1.54× on two threads and 1.65× on four — + §3.3.4: "it is almost not worth it to scale beyond two threads." +``` + +### Step 10 — false sharing: the pathology with no visible cause -The paper is ~114 pages; §3–§4 are the payload. +> **In:** Step 9's RFO, plus Step 2's cache line as the unit of everything. +> **Out:** the one bug in this chapter that is invisible in the source code, +> and the padding constant this machine actually needs. -- **§3.1–3.2** — skim; this is Steps 1–3 with 2007 diagrams. -- **§3.3 — read carefully.** The famous measurements. Fig 3.4 (sequential vs - random over working-set size) is *exactly* `cache_ladder`; compare his - plateau shapes with yours before explaining your numbers in `notes.md`. - You now know why random loses even in DRAM: no prefetch (Step 4) + TLB - misses (Step 7) + DRAM row activation. -- **§3.3.2** — critical word first / early restart: the CPU resumes as soon - as the needed word arrives, before the rest of the line does. +**False sharing** is the case where two threads write *different* variables that +happen to live in the same cache line. Nothing is shared at the language level; +everything is shared at the hardware level, because the line — not the variable +— is the unit the coherence protocol tracks (Step 2). Every write by one thread +invalidates the other's copy, so the line ping-pongs between cores and +multi-thread scaling collapses with no visible reason in the source. + +Drepper measured it in §6.4.1 with the simplest possible program: N threads, +each incrementing its own memory location 500 million times, pinned to +individual processors on a four-P4 machine. + +``` +Drepper Fig 6.10 (§6.4.1) — same program, locations on one cache line vs on +separate cache lines. The overhead is "computed by dividing the time needed when +using one single cache line versus a separate cache line for each thread": + + 2 threads: 390% + 3 threads: 734% + 4 threads: 1,147% +``` + +This repo's version of the same experiment, on Apple M-series +([topic 9 notes.md](../09-concurrency/notes.md), [FINDINGS row 9](../../FINDINGS.md)): + +``` + packed (all counters in one line): 202.7 ms 197.4 M inc/s + pad128 (one line each): 11.4 ms 3,502.9 M inc/s + ratio: 202.7 / 11.4 = 17.8×, i.e. (17.8 − 1) × 100 = 1,680% overhead + + pad64 is STILL 1.8× slower than pad128 — `#[repr(align(64))]`, the x86 + default that most `CachePadded` types use, only HALF-fixes false sharing on + this machine, because the coherence granule is 128 bytes. +``` + +1,680% on four Apple P-cores in 2026 against 1,147% on four Pentium 4s in 2007: +this is the one pathology in the whole paper that got *worse*, because cores got +faster relative to the interconnect. + +One honest caveat that Drepper supplies himself and that is easy to over-claim +past. Figure 6.11 runs the identical program on a **single** quad-core package +(a Core 2 QX 6700) and finds no scaling problem at all — "there is a slight +overhead when using the same cache line more than once but it does not increase +with the number of cores." His 1,147% needed four separate *sockets*. So the +correct general claim is not "false sharing always costs 10×"; it is "false +sharing costs whatever the path between the two writers costs", which was +enormous across a 2007 front-side bus, negligible within one 2007 package, and +17.8× across the P-cluster of an M3 Pro. Measure it on the machine you have — +the differential in the profiler table takes about ten minutes. + +Padding each thread's hot datum to its own line fixes it, at the cost of +footprint, which §6.4.1 flags as a genuine conflict with the rest of the paper's +advice. Quoted in full, because the elision people usually make hides *what* is +unacceptable: "There is a very simple 'fix' for the problem: put every variable +on its own cache line. This is where the conflict with the previously mentioned +optimization comes into play, specifically, the footprint of the application +would increase a lot. This is not acceptable." It is the footprint increase that +is unacceptable, not the padding itself — the blanket rule costs you everything +Steps 2 and 3 bought. Pad what is written by multiple threads; pack everything +else. (This pays off in topic 9, concurrency.) + +## How to read the paper (with the concepts in hand) + +The paper is 114 pages, version 1.0, dated November 21, 2007; §3–§4 are the +payload. The section numbers below were checked against the PDF's own headings. + +- **§3.1–3.2** — skim; this is Steps 1–3 with 2007 diagrams. Do stop at the + unnumbered cycles table at the end of §3.2 (page 16): ≤1 / ~3 / ~14 / ~240 is + the ladder every later figure is denominated in. Fig 3.4 lives here too — + "Access Times for Random Writes", the first working-set sweep in the paper and + the same *shape* `cache_ladder` produces. +- **§3.3.1 — read carefully.** Associativity, the `size = line × ways × sets` + identity, and Table 3.1. This is Step 3, and it is the only place in the paper + that gives you the arithmetic to *construct* a conflict miss. +- **§3.3.2 — read most carefully of all.** The famous measurements, and the + source of almost every number in this chapter: Fig 3.10 (sequential, ~4 and + ~9 cycles), Fig 3.11 (the same walk with growing element sizes — Step 2's + utilization, measured), Fig 3.12 (the TLB spike, and the 64-entry deduction), + **Fig 3.15 (sequential vs random — the 50× gap)**, Table 3.2 (the miss ratios + behind it) and Fig 3.17 (page-wise randomization, worth "up to 38%"). + Compare his plateau shapes with yours before explaining your numbers in + `notes.md`. You now know why random loses even in DRAM: no prefetch (Step 4) + + rising miss ratio + TLB misses (Step 8). +- **§3.3.4 — read carefully.** MESI, RFO, and the multi-thread scaling + measurements (Figs 3.19–3.22, Table 3.3). This is Step 9. - **§3.4** — instruction cache: skim (matters again at topic 19, JIT). -- **§3.5 — read carefully.** Coherency + false sharing (Step 8) with the - multi-thread scaling-collapse measurements. -- **§4.1–4.3** — Steps 6–7. The key bit is §4.3 on TLB reach. -- **§4.4+, §5, §7** — virtualization and NUMA: skip until a NUMA box matters. -- **§6** — skim for the checklist: sequential > random; hot struct fields - together, sorted by size; padding audits. §6.2's cache-oblivious matrix - transpose is worth 10 minutes — the intellectual ancestor of +- **§3.5.1** — cache and memory bandwidth in bytes/cycle (Figs 3.24–3.29). Worth + ten minutes for the method: he plots 16 B/cycle inside L1d falling to ~5.3 + B/cycle streaming from the FSB, with a visible step at 2¹⁸ bytes "due to the + exhaustion of the DTLB cache". That is Step 6 and Step 8 in one graph. +- **§3.5.2** — critical word first / early restart: the CPU resumes as soon as + the needed word arrives, before the rest of the line does. Note the measured + size of the effect in Fig 3.30 — about **0.7%**. A famous mechanism with a + tiny coefficient is a useful thing to have calibrated. +- **§4.1–4.3** — Steps 7–8. §4.2 is the radix tree, §4.3 prices the walk ("at + the very least 12 cycles"), §4.3.1 covers TLB flushes on context switch, and + §4.3.2 is huge pages, including the sentence that names database servers. +- **§4.4, §5** — virtualization and NUMA: skip until a NUMA box matters. +- **§6.2.1 — read carefully.** The matrix multiplication and Table 6.2. Note + what it actually is: matrix *multiplication*, not a transpose benchmark, and + the blocked version is **cache-aware, not cache-oblivious** — `SM` is defined + as `CLS / sizeof(double)` with `CLS` supplied at compile time by + `getconf LEVEL1_DCACHE_LINESIZE`. Table 6.2's four columns + (100% → 23.4% → 17.3% → 9.47%) are the intellectual ancestor of blocked/vectorized execution (topic 11). - -What's stale vs. forever: DDR2 timings, front-side bus, and Pentium 4 -details aged; the organization math, miss taxonomy, and measurement method -didn't. Keep the Apple Silicon deltas in mind while reading: 128-byte lines -(not 64), no inclusive L3 (shared SLC instead), much larger L1 (128–192 KB). +- **§6.2.2–6.2.4** — instruction cache, higher-level caches, and TLB usage; + skim. §6.3.1 is worth reading in full — it is the only place the paper states + the prefetcher's actual rules (Step 4). +- **§6.4.1 — read carefully.** False sharing, Figs 6.10 and 6.11. This is + Step 10, *not* §3.5 — a mistake this chapter used to make. +- **§7** — memory performance tools: §7.1 (oprofile) and §7.2 (cachegrind) are + the 2007 ancestors of the instrument list in the next section. The tool names + aged; the two-instrument discipline did not. + +What's stale vs. forever: DDR2 timings, front-side bus, and Pentium 4 details +aged; the organization math, miss taxonomy, and measurement method didn't. Keep +the Apple Silicon deltas in mind while reading: 128-byte lines (not 64), no +inclusive L3 (a shared SLC instead), much larger L1 (128 KB measured here), 16 kB +base pages, and a three-level page table. ## Finding these concepts in a real program -Steps 1–8 are visible in a microbenchmark you wrote on purpose. The harder +Steps 1–10 are visible in a microbenchmark you wrote on purpose. The harder skill is spotting them in a program you did not write, where the pathology is one loop among thousands. Three instruments, in the order you should reach for them: @@ -318,17 +957,18 @@ them: showed 21% in SipHash and ~79% in one inlined probe loop, and no amount of staring at it could split "hashing" from "waiting". 2. **Hardware counters** tell you *which wall*. This is the only instrument - that distinguishes the eight concepts directly. On Linux: `perf stat`. On + that distinguishes the ten concepts directly. On Linux: `perf stat`. On macOS there is no `perf` — Instruments → **CPU Counters** template gives you the events, and for real counter work run the same crate in a Linux VM or - container. + container. Drepper's §7.1 is the same idea with `oprofile`. 3. **A differential experiment** — change exactly one thing, re-measure — is the only fully portable instrument, and the one this repo leans on. Each row of the table below has one, because a counter tells you a number is high while a differential proves the *causal* link. -Start with the funnel: two counters (`instructions`, `cycles` → IPC) plus a -branch-miss rate narrow eight suspects to one or two. +Start with the funnel: two counters (`instructions`, `cycles` → **IPC**, the +instructions the core retires per clock cycle) plus a branch-miss rate narrow +ten suspects to one or two. ```mermaid flowchart TD @@ -338,7 +978,7 @@ flowchart TD C -->|no| E{"achieved GB/s near
the machine's peak?"} E -->|yes| F["bandwidth-bound
→ Steps 2, 4: line waste, layout"] E -->|"no — bus mostly idle"| G{"dTLB misses
significant?"} - G -->|yes| H["translation-bound
→ Steps 6, 7: huge pages, smaller reach"] + G -->|yes| H["translation-bound
→ Steps 7, 8: huge pages, smaller reach"] G -->|no| I["latency-bound
→ Step 5: dependent loads, no MLP"] style B fill:#1f6feb,color:#fff style D fill:#8957e5,color:#fff @@ -347,73 +987,391 @@ flowchart TD style I fill:#d29922,color:#000 ``` -The last branch is the one people get wrong: **latency-bound and -bandwidth-bound are opposites.** Both look "memory-bound" in a flamegraph, and -they have opposite fixes — more bandwidth-efficient layouts do nothing for a -pointer chase, and more overlap does nothing for a saturated bus. +The last branch is Step 6, and it is the one people get wrong: **latency-bound +and bandwidth-bound are opposites.** Both look "memory-bound" in a flamegraph, +and they have opposite fixes — more bandwidth-efficient layouts do nothing for a +pointer chase at 0.82% bus utilization, and more overlap does nothing for a scan +already at 38%. | Concept | Signature in a profile | Counters (Linux `perf`) | Differential test that proves it | |---|---|---|---| | **1** Hierarchy at all | Low IPC, flat profile, time on loads | `cycles,instructions` | Shrink the dataset with the algorithm unchanged. Time/op drops sharply ⇒ you were paying the hierarchy, not the code. | -| **2** Cache-line waste | Hot loop touches one field of a wide struct | `cache-references,cache-misses`, plus achieved GB/s vs *useful* bytes | Split hot fields out (AoS→SoA) or shrink the struct. Faster with identical instruction count ⇒ you were paying for bytes you never read. | +| **2** Cache-line waste | Hot loop touches one field of a wide struct | `cache-references,cache-misses`, plus achieved GB/s vs *useful* bytes | Split hot fields out (AoS→SoA) or shrink the struct. Faster with identical instruction count ⇒ you were paying for bytes you never read. Compute `U` first — Step 2's five cases take two minutes. | | **3** Conflict misses | A cliff at a power-of-two size or stride, while the working set still "fits" | `L1-dcache-load-misses` high with a small working set | Pad the stride by one line (row stride 4096 → 4096+128). Faster ⇒ conflict, not capacity. Nothing else moves that. | -| **4** Prefetching | Sequential and random over the *same* data differ ~10× | (vendor-specific prefetch events; weak) | Feed the same loop a sorted vs shuffled index array. The gap *is* the prefetcher's contribution. | +| **4** Prefetching | Sequential and random over the *same* data differ ~50× (Fig 3.15: 50×; this repo: 46×) | (vendor-specific prefetch events; weak) | Feed the same loop a sorted vs shuffled index array. The gap *is* the prefetcher's contribution. | | **5** Dependent loads | One load instruction owns the samples, IPC ≪ 1, **and achieved bandwidth is low** — slow while the bus idles | `cycles,instructions`; on x86 the stall-on-memory events | Run k independent chases interleaved with k cursors. Per-step time falls ~k× until it saturates ⇒ you were latency-bound with spare MLP. Batched/vectorized lookup APIs exist to collect that k×. | -| **6–7** TLB / page walks | A *second*, later cliff after the DRAM plateau has flattened | `dTLB-loads,dTLB-load-misses` (x86: `dtlb_load_misses.walk_completed`) | Enable huge pages (`MADV_HUGEPAGE` / THP) or drop the working set under TLB reach. On macOS: compare above vs below reach — that is `cache_ladder`'s last two rows, 87 → 113 ns. | -| **8** False sharing | Multi-thread scaling collapses; time sits in a *store*; per-thread work is unchanged | `perf c2c` — the purpose-built tool | Pad each thread's datum to its own line (128 B on M-series) and re-plot the scaling curve. Curve straightens ⇒ false sharing. | +| **6** Which wall | Everything looks "memory-bound" | achieved GB/s ÷ peak GB/s | Compute the utilization table of Step 6 for your loop. Under ~10% of peak ⇒ latency; over ~30% ⇒ bandwidth. The two fixes are opposites, so guessing costs you a sprint. | +| **7–8** TLB / page walks | A *second*, later cliff after the DRAM plateau has flattened | `dTLB-loads,dTLB-load-misses` (x86: `dtlb_load_misses.walk_completed`) | Enable huge pages (`MADV_HUGEPAGE` / THP) or drop the working set under TLB reach. On macOS: compare above vs below reach — that is `cache_ladder`'s last two rows, 87.4 → 113 ns. | +| **9** Coherence / RFO | Time sits in *stores*; scaling flattens as soon as two threads write the same structure | `perf c2c`; on x86 the RFO / `mem_inst_retired` events | Make the shared structure per-thread and merge at the end. Scaling straightens ⇒ you were paying RFOs. | +| **10** False sharing | Multi-thread scaling collapses; time sits in a *store*; per-thread work is unchanged | `perf c2c` — the purpose-built tool | Pad each thread's datum to its own line (**128 B** on M-series — 64 B only half-fixes it here) and re-plot the scaling curve. Curve straightens ⇒ false sharing. | Two habits that make this reliable: - **Always pair a counter with a differential.** "Cache misses are high" is not a diagnosis; databases miss cache constantly and are fine. The differential answers the only question that matters — *would fixing it help?* -- **Compute the useful-bytes ratio by hand.** Bytes your algorithm needs ÷ - bytes the machine moved. It needs no profiler, catches Step 2 instantly, and - is the number that decides row-vs-column layouts in topic 12. +- **Compute the useful-bytes ratio by hand.** That is Step 2's `U`: bytes your + algorithm needs ÷ bytes the machine moved. It needs no profiler, catches the + waste instantly, and is the number that decides row-vs-column layouts in + topic 12. ## Questions to answer in notes.md when done -1. Why does `cache_ladder` show *gradual* transitions between plateaus rather than - steps? (Hint: set associativity + random chain touching multiple sets.) -2. Predict: on M-series with 128 B lines, at what stride does a strided-read benchmark - stop getting faster per element? Verify with a quick experiment. -3. How many memory accesses can a single TLB miss add on a 4-level page table, and why - don't we see it in `cache_ladder`? (Hint: 16 KB pages, working set vs TLB reach.) -4. Take `lookup_shootout` at n=1e7 — 9.3 ns per probe over a ~160 MB table — and prove - with the Step 5 differential (not with reasoning) that it is latency-bound with spare - MLP rather than bandwidth-bound: make the probes *dependent* (each key derived from - the previous lookup's result) and report the new ns/probe. Which row of the +1. Why does `cache_ladder` show *gradual* transitions between plateaus rather + than steps? Drepper hit the same thing and diagnosed it in §3.3.2 — Fig 3.15's + random curve "keeps on rising" instead of flattening, and Table 3.2 shows why. + Which of his two causes apply to a random *cyclic* chain, and which does not? +2. Using Step 2's formula with `L = 128`, tabulate `U` for strides + 8, 16, 32, 64, 128, 256 and 4096 bytes over 8-byte elements. At what stride + does `U` stop falling, and why does per-element time keep rising past that + point anyway? Verify with a quick strided-read experiment. +3. How many memory accesses can a single TLB miss add on this machine's page + table, and why is `cache_ladder`'s measured addition only 25.6 ns rather than + the 312 ns of three cold DRAM loads? (Step 7's four mitigations; name the two + that are doing most of the work.) +4. Take `lookup_shootout` at n=1e7 — 9.3 ns per probe over a ~160 MB table, 9.2% + of the bus by Step 6's table — and prove with the Step 5 differential (not + with reasoning) that it is latency-bound with spare MLP rather than + bandwidth-bound: make the probes *dependent* (each key derived from the + previous lookup's result) and report the new ns/probe. Which row of the profiler table did you just walk down? +5. Drepper's §6.2.1 blocking uses `SM = CLS / sizeof(double)` with `CLS` from + `getconf LEVEL1_DCACHE_LINESIZE` — a cache-*aware* parameter, baked in at + compile time. What breaks when that binary runs on a machine with 128-byte + lines, and what would a cache-*oblivious* version do instead? (Table 6.2's + 17.3% column is what is at stake.) ## Takeaway -Every table in topic 0 §2 is a compressed version of this paper. Drepper's method — -plot access cost against working-set size and *explain every inflection* — is the -habit; the numbers you regenerate yourself on your own machine. +Every table in topic 0 §2 is a compressed version of this paper. Drepper's +method — plot access cost against working-set size and *explain every +inflection* — is the habit; the numbers you regenerate yourself on your own +machine. His constants are from 2007 and you should never quote them as +present-day figures; his ratios (80× across the ladder, 50× sequential-to-random, +22× of prefetch coverage) survived nineteen years and an instruction-set change +nearly unchanged, and that is the more surprising result. ## Done when -- [ ] You can recite the latency ladder — L1, L2, L3, DRAM — within 2x, and say which numbers from 2007 have aged and which have not. +Answer each before unfolding it. + +- [ ] You can recite the latency ladder — L1, L2, L3, DRAM — within 2×, and say which numbers from 2007 have aged and which have not. + +
Answer + + This machine, measured by `cache_ladder` ([notes.md](notes.md)): **~1 ns** L1 + (1.02 at 16–128 KB), **~5 ns** L2 (5.3–5.8 at 512 KB–1 MB, and still 7.6–9.0 + out to 8 MB because Apple's per-cluster L2 is 16 MB-class), **~17 ns** at + 16 MB as it falls into the SLC, and **~104 ns** DRAM at 128 MB rising to + 113 ns at 512 MB. + + Drepper's, from the unnumbered table at the end of §3.2 (Intel's published + Pentium M figures): Register ≤1, L1d ~3, L2 ~14, Main Memory ~240 — in + *cycles*, on 2007 hardware, and never to be restated in nanoseconds + or as a modern figure. + + What aged: every absolute constant, plus the entire structure around them — + DDR2, the front-side bus, inclusive L3, 64-byte lines, 4 kB base pages. What + did not: the ratio. His L1d-to-DRAM spread is 240 ÷ 3 = 80×; this machine's is + 104 ÷ 1.02 = 102×. Nineteen years of hardware progress made the gap slightly + worse, not better, which is the reason the paper is still assigned. + +
+ +- [ ] You can compute what fraction of a cache line a strided loop actually uses, and run it on both a row layout and a column layout. + +
Answer + + `U = (e × max(1, floor(L / s))) / L`, where `e` is the bytes read per element, + `s` the stride between touched elements, and `L` the line size — 64 bytes on + Drepper's machines (§6.2.1: "with 64 bytes for the Core 2 processor"), 128 on + Apple M-series, which topic 9 established by measuring that 64-byte padding + leaves contended counters 1.8× slower than 128-byte padding. + + Row layout, filtering on one 8-byte column of a 128-byte row: `s = 128 ≥ L`, + so `n = 1` and `U = 8 / 128 = 6.25%` — 93.75% of every transfer discarded. + Column layout, same filter: `s = 8`, `n = floor(128 / 8) = 16`, + `U = 128 / 128 = 100%`. Same 8 bytes of answer, 16× fewer bytes moved. + + Drepper's version is the same arithmetic at `L = 64`: the naive + `mul2[k][j]` inner loop of §6.2.1 has `s = 8 × 1000 = 8000`, so `U = 12.5%`; + transposing to `tmp[j][k]` gives `s = 8` and `U = 100%`. Table 6.2 prices the + difference at 16,765,297,870 → 3,922,373,010 cycles, or 23.4% of the original, + *including* the cost of copying the whole matrix. + +
+ - [ ] You can explain a conflict miss in terms of sets and ways, and construct a stride that causes one on purpose. -- [ ] You can say why a dependent-load chain is the one latency the prefetcher cannot hide, and why `cache_ladder` is built as a pointer chase for exactly that reason — including all three of its construction choices (data dependency, random *cyclic* permutation, cursor carried across iterations). + +
Answer + + A cache holds `sets` buckets of `ways` lines each, and + `size = line × ways × sets` (§3.3.1). Middle address bits pick the set, so a + line can only be placed in *its* set; a **conflict miss** is an eviction from + a full set while the cache as a whole is nearly empty. + + To construct one, find the set-index period, `sets × line size`, and stride by + it. Drepper's own 4 MB, 8-way, 64-byte-line L2 has + 4,194,304 / (64 × 8) = 8,192 sets — he states this, along with the 13 bits of + set index — so the period is 8,192 × 64 = 524,288 B = **512 KB**. Nine + addresses spaced 512 KB apart use nine lines, 576 bytes total, and thrash a + 4 MB cache, because 9 > 8 ways. + + On this machine, using the measured 128 KB L1d and 128-byte line and *assuming* + 8-way: 131,072 / (128 × 8) = 128 sets, period 128 × 128 = 16 KB. That is why + the profiler table's differential is "pad the row stride by one line" — 4096 → + 4096+128 walks the set index forward instead of landing on it repeatedly. + Associativity is what buys this back: Table 3.1's 8 MB row goes 4,731,904 → + 2,690,498 misses from direct-mapped to 2-way, a 43.1% saving, and then only + 17.9% and 4.4% for the next two doublings. + +
+ +- [ ] You can say what the prefetcher can and cannot cover, and how much latency it hides when it works. + +
Answer + + §6.3.1's rules: it needs **two or more** misses before it starts, it recognizes + strides only within roughly a 512-byte window, it tracks eight to sixteen + streams *shared across all cores on that cache*, it does not recognize + non-linear patterns at all, and — the hard limit — "it cannot cross page + boundaries", because a speculative fetch must never cause a page fault the + program did not ask for. So even a perfect sequential scan takes a miss at + every page boundary. + + How much it is worth: §3.3.2 says main memory costs "200+ cycles" on that + machine, and Fig 3.10's sequential list walk past the 1 MB L2 costs about 9 + cycles per element. 200 ÷ 9 ≈ **22×** hidden. The same figure shows the walk at + ~9 cycles inside the L2 range too, which is faster than the L2's own ~14-cycle + latency (§3.2) — prefetching removes the L2 trip from the critical path, not + just the DRAM trip. + + When it fails, Fig 3.15 measures the bill: the identical list, shuffled, costs + 450+ cycles per element against sequential's ~9 — a **50×** gap. This repo's + equivalent pair is 2.24 ns per 128-byte line streaming (topic 12's 57.1 GB/s) + against 104 ns per line chased (`cache_ladder` at 128 MB), a **46×** gap. + Not the "~10×" this chapter used to claim. + +
+ +- [ ] You can say why a dependent-load chain is the one latency the prefetcher cannot hide, and why `cache_ladder` is built as a pointer chase for exactly that reason — including all three of its construction choices. + +
Answer + + Because there is nothing to extrapolate *and* nothing to overlap. The + prefetcher needs a pattern (Step 4) and the out-of-order core needs a + computable address; in `idx = chain[idx]` the next address is the value the + previous load is still fetching, so exactly one miss is ever in flight. Every + other case in this chapter has some escape: a stride the prefetcher can follow, + or several independent addresses the core can issue together. + + The measurement of the difference is on one machine, in one file: the + `lookup_shootout` hashmap lane does 1024 *independent* random probes over a + ~160 MB table at **9.3 ns** each, while `cache_ladder` does a *dependent* chase + over 128 MB at **104 ns** each — 11.2× apart on the same DRAM + ([notes.md](notes.md)). + + `cache_ladder`'s three choices, at + [`experiments/benches/cache_ladder.rs`](experiments/benches/cache_ladder.rs): + (1) line 28, `idx = chain[idx]` — the dependency is in the *data*, so no + compiler and no core can reorder around it; (2) lines 16 and 21 — the chain is + a random permutation closed into a *single* cycle (Sattolo), where random kills + the prefetcher and cyclic stops a short sub-cycle from living in L1 and + flattering the large sizes; (3) line 53 — `idx` is declared *outside* + `b.iter`, carried across criterion iterations. Without (3) the benchmark + re-walked the same 65,536 slots every iteration and reported ~25 ns for + "DRAM", having measured an ~8 MB hot path it created itself. + +
+ - [ ] You can state the difference between latency-bound and bandwidth-bound, name the one measurement that separates them, and say why a flamegraph never can. -- [ ] You can compute a TLB's reach from entry count and page size, and explain why exceeding it looks like a second, later cliff. -- [ ] Given a strange profile, you can name the counter *and* the differential experiment for each of the eight concepts, without re-reading the table. -- [ ] You wrote answers to all four questions in notes.md. + +
Answer + + Latency-bound: waiting on the round trip of an access that could not be started + earlier; the bus is idle and the fix is more overlap. Bandwidth-bound: the + bytes cannot arrive any faster; the bus is saturated and the fix is moving + fewer bytes. The fixes are opposites, so the distinction is not academic. + + The measurement is achieved bandwidth as a fraction of peak. On this machine + (peak 150 GB/s, [topic 12 notes.md](../12-columnar-analytics/notes.md)): + `cache_ladder` at 128 MB moves one 128-byte line per 104 ns = 1.23 GB/s = + **0.82%** of peak — latency-bound with 99% of the bus idle. Topic 12's + `scan_bench` moves 800 MB in 0.014 s = 57.1 GB/s = **38%** of peak on a single + core — bandwidth-bound. `lookup_shootout` at n=1e7 sits between them at + 13.8 GB/s = **9.2%**, which is Question 4. + + A flamegraph cannot separate them because a sampling profiler attributes a + stall to the instruction that is waiting, and both cases produce the same + picture: one hot load instruction with most of the samples. This repo has the + worked example — the `lookup_shootout` flamegraph shows 21% in SipHash and + ~79% in one inlined probe loop, and no amount of staring at it splits + "hashing" from "waiting on DRAM". + +
+ +- [ ] You can compute a TLB's reach from entry count and page size, explain why exceeding it looks like a second, later cliff, and say what huge pages buy. + +
Answer + + `Reach = entries × page size`. Drepper deduces both factors experimentally in + §3.3.2: Fig 3.12 places one 64-byte list element per page, the cost spikes when + the working set reaches 2¹³ bytes (128 pages, against 64 just below), and he + concludes "the TLB cache has 64 entries". 64 × 4,096 = **256 KB** of reach. + + It is a *second* cliff because translation is a separate cache from the data + caches and the penalties add rather than overlap — §3.3.2: "the address + translation penalties are additive to the memory access times". So once the + DRAM plateau has already flattened, a further rise can only be translation. + That is exactly `cache_ladder`'s tail: 87.4 ns at 64 MB (4,096 pages of 16 KB) + to 113 ns at 512 MB (32,768 pages), **+25.6 ns** per access with the data + latency unchanged. + + Huge pages buy reach multiplicatively without more entries: 2 MB pages are + 2,097,152 / 4,096 = **512×** the reach of 4 kB pages, 1 GB pages are + **262,144×**, and Apple's 16 kB base page is **4×** for free on every process. + They also shorten the walk (§4.3.2). The price, per §4.3.2, is physical + contiguity — "finding a free area with 512 contiguous pages ... can be + extremely difficult (or impossible) after the system runs for a while" — which + is why Drepper's list of workloads worth the setup cost is short, and + "database servers" is on it. + +
+ +- [ ] You can explain false sharing in terms of the coherence protocol, and say what padding constant this machine actually needs. + +
Answer + + MESI (§3.3.4) tracks state per *cache line*, not per variable. To write, a core + must hold the line Exclusive, which means broadcasting a Request For Ownership + that invalidates every other copy. Two threads writing two different variables + that share one line therefore trade RFOs on every write, even though the + program shares nothing — §6.4.1: "This means that a lot of RFO messages are + sent, in the worst case one for each write access. So a normal write will be + suddenly very expensive." + + Drepper's measurement (Fig 6.10, four P4 sockets, 500 M increments per thread, + one line vs separate lines) is 390% / 734% / 1,147% overhead at 2 / 3 / 4 + threads. This repo's (topic 9) is packed 202.7 ms against pad128 11.4 ms — + **17.8×**, i.e. 1,680% — so the pathology got worse, not better, as cores + outran interconnects. + + The constant is **128 bytes** on M-series. `pad64` — which is what + `#[repr(align(64))]` and most x86-derived `CachePadded` types give you — is + still 1.8× slower than `pad128` here, because the coherence granule is 128. + And the honest caveat is Drepper's own Fig 6.11: the identical program on a + *single* quad-core Core 2 package showed no scaling penalty at all, so the + right claim is "false sharing costs whatever the path between the writers + costs", and the differential in the profiler table is how you find out what + yours is. + +
+ +- [ ] Given a strange profile, you can name the counter *and* the differential experiment for each of the ten concepts, without re-reading the table. + +
Answer + + The funnel first: IPC (`instructions` ÷ `cycles`) at ≥2 and scaling with cores + is compute-bound; under ~1, check branch-miss rate; if branches are clean, + check achieved GB/s against peak; if the bus is idle, check dTLB misses; if + those are clean too, it is dependent loads. + + Counters, per concept: `cycles,instructions` for the hierarchy and for + dependent loads; `cache-references,cache-misses` plus achieved-vs-useful bytes + for line waste; `L1-dcache-load-misses` at a small working set for conflict + misses; achieved GB/s ÷ peak for the which-wall question; + `dTLB-loads,dTLB-load-misses` (x86: `dtlb_load_misses.walk_completed`) for + translation; `perf c2c` for coherence and false sharing. + + Differentials, per concept: shrink the dataset (hierarchy); AoS→SoA or shrink + the struct (line waste); pad the stride by one line, 4096 → 4096+128 (conflict); + sorted vs shuffled index array over identical data (prefetching); run k + interleaved cursors and watch per-step time fall ~k× (MLP); compute the + utilization table (which wall); enable huge pages or drop under reach + (translation); make the shared structure per-thread (RFO); pad each thread's + datum to 128 B and re-plot the scaling curve (false sharing). Always pair a + counter with a differential — "cache misses are high" is not a diagnosis, + because databases miss cache constantly and are fine. + +
+ +- [ ] You wrote answers to all five questions in notes.md. + +
Answer + + There is no shortcut here, but three of the five have most of their answer in + this chapter and are really asking you to run the arithmetic yourself: Q2 is + Step 2's `U` formula tabulated over seven strides, Q3 is Step 7's 25.6 ÷ 312 = + 8.2% and naming which mitigations explain it, and Q5 is Table 6.2's 17.3% + column against a compile-time `CLS`. + + Q1 and Q4 need work at the keyboard. Q1 wants you to hold Drepper's §3.3.2 + explanation of the non-flattening random curve — rising L2 miss ratio + (Table 3.2) plus TLB misses (Fig 3.17) — against a chain that is a *single + cycle* over every slot, and decide which of his causes survives that + construction. Q4 is a real edit to `lookup_shootout`: derive each key from the + previous lookup's result, re-run, and report the new ns/probe against the + 9.3 ns baseline. If it lands near `cache_ladder`'s 104 ns you have proved the + 9.3 was overlap, and you have walked down row 5 of the profiler table. + +
## References -**Papers** -- Drepper — "What Every Programmer Should Know About Memory" (Red Hat, - 2007) — [PDF](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf) - (~114 pages — read §3–§4 properly, skim §6, skip the rest; the study - guide's advice stands) +**The paper** +- Ulrich Drepper — "What Every Programmer Should Know About Memory" (Red Hat, + **version 1.0, November 21, 2007**, 114 pages) — + [PDF](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf). + Every section, figure and table number in this chapter was checked against + that PDF. Read §3.3.1–§3.3.4 and §4.1–§4.3 properly, §6.2.1, §6.3.1 and §6.4.1 + carefully, skim the rest. + +| Section / Figure | What this chapter uses it for | +|---|---| +| §3.2, page 16 (unnumbered table) | Intel's Pentium M cycles: Register ≤1, L1d ~3, L2 ~14, Main Memory ~240 — Step 1's 2007 ladder | +| §3.2, Fig 3.4 | "Access Times for Random Writes" — the first working-set sweep; L1d at 2¹³ B, L2 at 2²⁰ B, <10 → ~28 → 480+ cycles. The *shape* `cache_ladder` reproduces | +| §3.3.1 | `size = line × ways × sets`, and the worked 4 MB/8-way/64 B → 8,192 sets, 13 index bits — Step 3 | +| §3.3.1, Table 3.1 | L2 misses vs size, associativity and line size; the 43.1% / 17.9% / 4.4% ladder of diminishing returns | +| §3.3.2, Fig 3.10 | Sequential walk: ~4 cycles in L1d, ~9 past L2 against "200+ cycles" unhidden — Step 4's 22× | +| §3.3.2, Fig 3.11 | The same walk at NPAD 0/7/15/31 (strides 8/64/128/256 B) — Step 2's utilization, measured | +| §3.3.2, Fig 3.12 | One element per page; the spike at 2¹³ B and the deduction "the TLB cache has 64 entries" — Step 8's reach | +| §3.3.2, Fig 3.15 | Sequential vs random, ~9 against 450+ cycles — the 50× gap that corrects this chapter's old "~10×" | +| §3.3.2, Table 3.2 | L2 miss ratios behind Fig 3.15: 0.94% vs 13.42% at 2²⁰, 4.67% vs 57.84% at 2²⁹ | +| §3.3.2, Fig 3.17 | Page-wise randomization; limiting TLB working set is worth "up to 38%" | +| §3.3.4, Fig 3.18 | MESI states and transitions, and the RFO — Step 9 | +| §3.3.4, Table 3.3 | Multi-thread efficiency: 1.69/1.69/1.54 at two threads, 2.98/2.07/1.65 at four | +| §3.5.1, Figs 3.24–3.29 | Bandwidth in bytes/cycle: 16 B/cycle in L1d, ~5.3 streaming, with a DTLB step at 2¹⁸ B | +| §3.5.2, Fig 3.30 | Critical word first / early restart — and the measured effect, about 0.7% | +| §4.2, Fig 4.2 | The four-level radix tree, 512 entries per 4 kB directory, and the 4 MB flat-table counterexample — Step 7 | +| §4.3 | "up to four memory accesses", "at the very least 12 cycles", and why the four loads cannot be parallelized | +| §4.3.2 | Huge pages: 2 MB/4 MB on x86-64, 512 contiguous pages, shorter walks, and "Database servers are an example" | +| §6.2.1, Table 6.2 | Matrix multiplication: 100% → 23.4% → 17.3% → 9.47%, with cache-*aware* blocking `SM = CLS / sizeof(double)` | +| §6.2.1, Fig 6.2 | Spreading one element over two cache lines: ~17% penalty in L2, ~27% in DRAM, 25–35% random | +| §6.3, §6.3.1 | The prefetcher's actual rules: two-miss trigger, ~512 B stride window, 8–16 shared streams, no page crossing, no non-linear patterns — Step 4 | +| §6.4.1, Figs 6.10, 6.11 | False sharing: 390% / 734% / 1,147% across four sockets — and *no* penalty within one quad-core package — Step 10 | +| §7.1, §7.2 | oprofile and cachegrind: the 2007 ancestors of the instrument list above | + +**Measured in this repo** (Apple M3 Pro, 2026-07-28 — see +[FINDINGS.md](../../FINDINGS.md)) + +| Source | Number this chapter uses | +|---|---| +| [`notes.md`](notes.md), `cache_ladder` | The whole measured ladder, 1.02 ns at 16 KB to 113 ns at 512 MB; the 87.4 → 113 ns TLB tail; the ~25 ns lying first version | +| [`notes.md`](notes.md), `lookup_shootout` | 9.3 ns per independent probe at n=1e7 over ~160 MB — the 11.2× MLP fork against the chase | +| [`notes.md`](notes.md), flamegraph | 21% SipHash / ~79% inlined probe loop — why a flamegraph cannot answer Step 6 | +| [`experiments/benches/cache_ladder.rs`](experiments/benches/cache_ladder.rs) | Lines 14-23, 25-31 and 50-57 — the three construction choices | +| [topic 9 `notes.md`](../09-concurrency/notes.md) | packed 202.7 ms vs pad128 11.4 ms = 17.8×; pad64 still 1.8× slower — the 128-byte coherence granule | +| [topic 12 `notes.md`](../12-columnar-analytics/notes.md) | 800 MB folded in 0.014–0.033 s = 24.2–57.1 GB/s single core, against 150 GB/s peak | +| [FINDINGS row 3](../../FINDINGS.md) | B-tree lookups 862 → 1101 ns at constant height — residency, not height | +| [FINDINGS row 9](../../FINDINGS.md) | "Padding 'independent' counters to 128 B is worth **17.8×**; 64 B only half-fixes it on M-series" — Step 10 | +| [FINDINGS row 12](../../FINDINGS.md) | "The scan floor is **24–57 GB/s** on a 150 GB/s machine" — Step 6's utilization column | +| [FINDINGS row 0](../../FINDINGS.md) | "The DRAM latency ladder verified at ~1 / 5 / 100 ns … **21%** of a HashMap lookup is SipHash" — Steps 1 and 5 | **Tools referenced above** - Brendan Gregg — [perf examples](https://www.brendangregg.com/perf.html) — the counter-event cookbook behind the profiler table's middle column. - [`perf c2c(1)`](https://man7.org/linux/man-pages/man1/perf-c2c.1.html) — - purpose-built false-sharing detection (Step 8); no macOS equivalent, so this - is one of the cases worth a Linux VM. + purpose-built false-sharing detection (Steps 9–10); no macOS equivalent, so + this is one of the cases worth a Linux VM. - Instruments → **CPU Counters** template — the macOS substitute for `perf stat`; see topic 0 §4 for the full tool table on this machine. diff --git a/topics/00-performance-toolbox/reading-fair-benchmarking.md b/topics/00-performance-toolbox/reading-fair-benchmarking.md index de9215f..f607576 100644 --- a/topics/00-performance-toolbox/reading-fair-benchmarking.md +++ b/topics/00-performance-toolbox/reading-fair-benchmarking.md @@ -3,29 +3,38 @@ Criterion and Tene cover how a *single measurement* lies; this chapter — built on a 6-page DBTest '18 paper from the future DuckDB authors — covers how a *comparison between systems* lies. Before pointing you at the paper, it builds -the idea of a fair comparison from zero and then walks the eight pitfalls one -at a time: what each one is, a concrete example of how it lies, and how to -avoid it. It is the database-specific companion to topic 0 §1, and the paper's -Appendix A checklist is an artifact you will reuse against every capstone -comparison in this curriculum. +the idea of a fair comparison from zero, pins down the one experimental setup +every demonstration in the paper reuses, and then walks the eight pitfalls one +at a time: what each is, the paper's own measured example of it lying, and how +to avoid it. It is the database-specific companion to topic 0 §1, and the +paper's Appendix A checklist is an artifact you will reuse against every +capstone comparison in this curriculum. + +Every figure below is quoted from the paper with the section, figure or table +it appears in. Where this repo has measured the same failure mode itself, the +repo's number is cited instead of a borrowed one. ## The problem in one sentence Van der Kouwe's survey found benchmarking crimes in **96%** of 50 top-tier -systems papers, and Purohith et al. showed SQLite throughput varies **28×** -on one configuration parameter that 0 of 16 surveyed papers reported — so -"system A is 3× faster than system B" is, by default, a statement about the -experimenters, not the systems. +systems papers (§2.1), and Purohith et al. showed SQLite transaction +throughput varies by a factor of **28** on a single parameter setting that +**none of 16** surveyed papers reported (§2.2) — so "system A is 3× faster +than system B" is, by default, a statement about the experimenters, not the +systems. ## The concepts, step by step ### Step 1 — what a fair comparison even requires +> **In:** nothing yet — this step builds the frame the eight pitfalls hang on. +> **Out:** four requirements, each of which one or more pitfalls violates. +> Step 2 supplies the experimental setup that demonstrates them. + A benchmark comparison is fair only when the *systems* are the only variable — -everything else (data, tuning effort, machine state, what gets timed, and -correctness of the answers) is held equal. That decomposes into four -requirements, and each of the eight pitfalls below is a failure of one of -them: +everything else (the data, the tuning effort, the machine state, what gets +timed, and the correctness of the answers) is held equal. That decomposes into +four requirements: ```mermaid flowchart TD @@ -34,193 +43,514 @@ flowchart TD Q --> CMP["Comparison"] Q --> MEA["Measurement"] Q --> RES["Results"] - SU --> P1["3.1 non-reproducible
(the Escher result)"] - SU --> P2["3.2 untuned baseline
(debug build, default config)"] + SU --> P1["3.1 non-reproducible
(the Escher result, Fig. 2)"] + SU --> P2["3.2 failure to optimize
(debug build, default config)"] CMP --> P3["3.3 apples vs oranges
(kernel vs full system)"] - CMP --> P4["3.4 tuned to the benchmark
(known selectivities)"] - MEA --> P5["3.5 cold/hot conflated"] - MEA --> P6["3.6 restart ≠ cold
(OS page cache warm)"] + CMP --> P4["3.4 overly-specific tuning
(known selectivities)"] + MEA --> P5["3.5 cold and hot conflated"] + MEA --> P6["3.6 restart is not cold
(OS page cache stays warm)"] MEA --> P7["3.7 preprocessing ignored
(index build, auto-imprints)"] - RES --> P8["3.8 fast but wrong
(diff against a trusted engine)"] + RES --> P8["3.8 incorrect code wins
(diff against a trusted engine)"] ``` -- **Setup** must be reproducible and *both* systems must be tuned with equal - effort (pitfalls 3.1, 3.2). -- **Comparison** must pit like against like — same functionality, workloads - neither system was specifically fitted to (3.3, 3.4). -- **Measurement** must control machine state — hot vs cold caches — and time - *all* the work, including preparation (3.5, 3.6, 3.7). -- **Results** must be verified correct, because a wrong answer is free (3.8). +- **Setup** must be reproducible, and *both* systems must be tuned with equal + effort (pitfalls 3.1, 3.2 — Steps 3 and 4). +- **Comparison** must pit like against like: the same functionality, on + workloads neither system was specifically fitted to (3.3, 3.4 — Steps 5, 6). +- **Measurement** must control machine state — hot against cold caches — and + time *all* the work, including preparation (3.5, 3.6, 3.7 — Steps 7, 8, 9). +- **Results** must be verified correct, because a wrong answer is free (3.8 — + Step 10). + +Two definitions the paper leans on from the start. **TPC-H** is the standard +decision-support benchmark: a fixed schema, 22 queries, and generated data at +a chosen **scale factor** — SF1 means roughly 1 GB of data. And Jain's +distinction, quoted at §2.1, frames the whole list: **mistakes** are +"ill-advised but inadvertent choices", while **games** are "deliberate and +purposeful manipulation of the experiment to elicit a specific outcome". The +checklist catches both, which matters because you cannot tell them apart from +the outside — and you are far more likely to commit the first. + +Why it matters: every pitfall below is a failure of exactly one of those four +requirements, which is what makes eight separate mistakes a single checklist. + +### Step 2 — the one setup that produced every number in the paper + +> **In:** the four requirements from Step 1. +> **Out:** the machine, the system versions and the reporting standard behind +> every figure quoted in Steps 3–10 — without which none of them mean +> anything, which is itself pitfall 3.1. + +The paper practises what Step 3 preaches, and states its setup once (§3 +preamble): + +| Component | What | +|---|---| +| CPU | Intel i7-2600K at 3.40 GHz, **one** of its eight hardware threads used | +| Memory | 16 GB | +| OS | Fedora 26, Linux kernel 4.14 | +| Compiler | GCC 7.3.1 | +| Systems | MariaDB 10.2.13, MonetDB 11.27.13, SQLite 3.20.1, PostgreSQL 9.6.1 | +| Workload | mock TPC-H at SF1, single-threaded | +| Reporting | median with non-parametric, quantile-based 95% confidence intervals | +| Artifacts | scripts, results, configs and plotting code, all published | + +Single-threaded is a deliberate fairness choice, stated in the preamble: not +every system supports intra-query parallelism, so giving all of them one +thread removes a variable rather than testing one. + +Two terms in that last row. A **confidence interval** is a range that would +contain the true value in a stated fraction of repeated experiments — a 95% CI +means 95 of 100 repetitions. **Non-parametric** means it is computed without +assuming the measurements follow any particular distribution, here by taking +quantiles of the observed runs directly. That is the same philosophy as +criterion's bootstrap intervals, applied to whole-system runs instead of +function calls: do not assume normality of something you can resample. + +The paper also states, in the same preamble, that including a system in these +experiments implies nothing about the papers that used it. The systems are +props. + +Why it matters: Steps 3–10 quote a dozen timings. Every one of them is *this* +machine, *these* versions, SF1, one thread. A number from this paper carried +into an argument about your hardware is a fresh instance of pitfall 3.1. + +### Step 3 — pitfall 3.1: non-reproducibility + +> **In:** the setup from Step 2. +> **Out:** the Escher result — a ranking that cycles — and the size of the +> single hidden decision that produced it. -The paper's demonstrations all use one setup: mock TPC-H experiments -(TPC-H is the standard decision-support benchmark — fixed schema, 22 queries, -generated data at a scale factor; SF1 ≈ 1 GB) at SF1 against MariaDB, -PostgreSQL, SQLite, and MonetDB, single-threaded. Jain's classic distinction -frames the whole list: *mistakes* (accidental) vs *games* (deliberate) — the -checklist catches both. +A result is reproducible only if someone else can rerun it from the published +hardware description, configuration, code and data. Most papers publish none +of these, and §3.1 notes the aggravating habit of anonymising systems as +"DBMS-X" to avoid a vendor's legal department: even a reader who owns every +system cannot tell which one was measured. -### Step 2 — pitfall 3.1: non-reproducibility +The demonstration is the **Escher result** (Fig. 2, TPC-H Q1 at SF1) — three +pairwise comparisons, every measurement individually true, that together form +a cycle: -A result is reproducible only if someone else can rerun it from the published -hardware description, configuration, code, and data — and most papers publish -none of these. The paper's demonstration is the **Escher result** (Fig. 2): -MariaDB < PostgreSQL < SQLite < MariaDB\* — a cycle where every measurement -is individually "true". The trick: MariaDB\* stored columns as DOUBLE instead -of DECIMAL — *both* allowed by the TPC-H spec, invisible unless the full -setup is published, and worth enough to reorder the ranking. +``` +Fig. 2, median seconds: MariaDB 12.18 + Postgres 9.73 + SQLite 8.19 + MariaDB* 4.70 + +panel 1: Postgres 9.73 < MariaDB 12.18 → P beats M (12.18/9.73 = 1.25×) +panel 2: SQLite 8.19 < Postgres 9.73 → S beats P ( 9.73/8.19 = 1.19×) +panel 3: MariaDB* 4.70 < SQLite 8.19 → M beats S ( 8.19/4.70 = 1.74×) +``` -**How to avoid it:** publish hardware, all configuration parameters, scripts, -and data-generation steps. If a reader can't rebuild the experiment, the -number is an anecdote. +So M < P, P < S, S < M: MariaDB is both the slowest system in the paper and +the fastest, "a contradiction similar to the famous paintings by M.C. Escher". -### Step 3 — pitfall 3.2: failure to optimize the baseline +The hidden decision is one schema choice: MariaDB\* stored the `lineitem` +money columns as `DOUBLE` instead of `DECIMAL`, and MariaDB's decimal +implementation is inefficient. **Both spellings are allowed by the TPC-H +specification** (§3.1 cites [2, sec. 1.3]), so neither run is cheating. Do the +division the paper does not print: + +``` +one schema decision, same system: 12.18 / 4.70 = 2.59× +the largest gap it manufactures: 8.19 / 4.70 = 1.74× +``` + +The undisclosed choice is worth **more than any of the three system gaps it +was used to create**. That is the general shape of this pitfall: the +unpublished variable does not add noise to the comparison, it dominates it. + +**How to avoid it:** publish the hardware, every configuration parameter, the +source or binaries, and the data-generation steps — §3.1's list also names the +OS, how the server was installed, and its version. If a reader cannot rebuild +the experiment, the number is an anecdote. + +### Step 4 — pitfall 3.2: failure to optimize the baseline + +> **In:** the Step 2 setup, now run twice per system — once as an author would +> configure their own system, once as they would configure a competitor's. +> **Out:** two measured gaps that are entirely artifacts of build and config. The baseline system is *the author's competitor*, so nobody spends a week -tuning it — and defaults are terrible. The paper's numbers: MonetDB compiled -in debug mode runs TPC-H Q1 in 1.58 s vs 0.87 s for a release build (1.8× -from a compiler flag); PostgreSQL with default configuration runs Q9 in -0.47 s vs 0.27 s once its memory settings are configured (1.7×). "DBMS A -beats DBMS B" can literally be the same system measured twice. +tuning it. §3.2 states the incentive plainly: "the worse the state of the art +system does, the better the authors' system looks." Two measurements: + +``` +Fig. 3a (Q1) MonetDB debug build 1.58 s + MonetDB* release build 0.87 s 1.58 / 0.87 = 1.82× +Fig. 3b (Q9) Postgres default config 0.47 s + Postgres* configured 0.27 s 0.47 / 0.27 = 1.74× +``` + +Both gaps are between a system and *itself*. The debug build is not merely +"unoptimized": §3.2 explains that MonetDB's debug mode enables sanity-checking +code that **scans entire columns** to verify invariants — work that has +nothing to do with answering the query. Postgres's default configuration +predates the machine it is running on and does not use the available memory. + +An author who published either as "DBMS A beats DBMS B" would have published a +compiler flag. **How to avoid it:** tune both systems with documented, comparable effort — -release builds, memory settings sized to the machine — and publish the -configs (Step 2). +release builds, memory settings sized to the machine — and publish the configs, +which is Step 3's rule again. The Appendix A checklist splits this into exactly +two boxes, *compilation flags* and *system parameters*, because they fail +independently. + +### Step 5 — pitfall 3.3: apples against oranges -### Step 4 — pitfall 3.3: apples vs oranges +> **In:** the Step 4 numbers, now compared against a program that is not a +> database at all. +> **Out:** the largest single gap in the paper, and the reason it is meaningless. + +A comparison is fair only if both systems perform the same functionality. +§3.3 lists what a real DBMS carries that a standalone program does not: +arbitrary queries, transaction isolation, updates, and multiple concurrent +clients. The paper hand-writes TPC-H Q1 as a standalone program, names it +**TimDB**, and measures it against the fairly feature-complete MonetDB: + +``` +Fig. 3c (Q1) MonetDB (release) 0.87 s + 'TimDB' 0.03 s 0.87 / 0.03 = 29× +``` -Comparing a stripped-down kernel against a full system credits the kernel for -all the work it simply doesn't do. The paper hand-writes TPC-H Q1 as a -standalone C++ program ("TimDB"): 0.03 s vs MonetDB's 0.87 s — 29× "faster", -because it skips parsing, query optimization, transactions, overflow -checking, and concurrency control. Any research prototype missing features is -structurally TimDB. +Note which MonetDB that is: the *tuned* 0.87 s from Fig. 3a, not the 1.58 s +debug build. The paper is being scrupulous — and the pitfalls still compound +if you are not: -**How to avoid it:** compare full system vs full system; where feature gaps -exist, state them explicitly next to the numbers; verify both produce +``` +debug MonetDB against a hand-written kernel: 1.58 / 0.03 = 52.7× +``` + +29× of that is pitfall 3.3 and a further 1.8× is pitfall 3.2, and a paper +committing both would report 53× without either being a lie about arithmetic. + +§3.3 also names the subtle version: **overflow handling**. Guaranteeing correct +results regardless of the stored data requires either *overflow checking* +(test each arithmetic result) or *overflow prevention* (prove from the data's +range that none can occur). An implementation with neither is faster and is +not comparable. Any research prototype missing features is structurally TimDB. + +**How to avoid it:** compare full system against full system; ideally integrate +the new algorithm into a complete system before measuring it. Where feature +gaps remain, state them next to the numbers, and verify both systems produce identical results. -### Step 5 — pitfall 3.4: overly-specific tuning +### Step 6 — pitfall 3.4: overly-specific tuning + +> **In:** a standardized benchmark, whose every property is published. +> **Out:** a system whose advantage exists only on that benchmark. Tuning to the benchmark means fitting the *system* to the test's known -properties, so the number stops generalizing. TPC-H's selectivities (the -fraction of rows each filter keeps) and cardinalities (result sizes) are -published constants — so a join-order heuristic can be quietly tuned until -exactly those 22 queries win, while everything else regresses. +properties, so the number stops generalizing. §3.4 lists what TPC-H and TPC-C +publish up front: the workload, the **cardinalities** of intermediate results +(how many rows each step produces), the data distributions, the +**selectivities** of predicates (the fraction of rows a filter keeps), and the +number of groups an aggregation creates. With all of that known, join-order +heuristics can be tuned until exactly those 22 queries win, and data can be +sharded so the work splits evenly *for this benchmark*. + +The failure is invisible from inside the benchmark: the system is genuinely +faster on it, and genuinely slower on the similar queries the benchmark does +not contain. + +**How to avoid it:** run more experiments than the standardized suite — §3.4's +advice is that the standard benchmark is a good *baseline* comparison, with a +set of different queries measured alongside it. Be suspicious of any advantage +that evaporates off-benchmark. + +### Step 7 — pitfall 3.5: conflating cold and hot runs + +> **In:** repeated runs of one query on one system. +> **Out:** two distinct populations of measurement that must not be pooled. + +A **cold run** is the first execution, with nothing cached; a **hot run** has +the data already resident. §3.5 names all four reasons the first is slower: +data must be read from persistent storage, the query must be parsed and +compiled, the buffer pool is empty, and any plan cache is cold. Averaging the +two produces a number describing neither — first-query-of-the-morning and +query-in-a-loop are different user experiences, and both are real. + +**How to avoid it:** report cold and hot *separately*. The checklist's box for +hot runs is "ignore initial runs", which is criterion's warm-up (Step 2 of the +criterion chapter) formalized at the system level: the same idea, one layer up. + +### Step 8 — pitfall 3.6: restarting the server is not a cold run + +> **In:** the cold-run protocol from Step 7. +> **Out:** what stays warm across a restart, and the only protocol that +> actually clears it. + +Subtler than Step 7, and §3.6 gives it its own section because the usual +protocol is wrong. Restarting the database server does **not** produce a cold +run: the operating system uses spare main memory as a cache of disk blocks — +the **page cache** — and that cache belongs to the kernel, not to the process, +so it survives the restart entirely. A restarted server reads its "disk" data +out of RAM. + +The paper's correct protocol, from §3.6 and its footnote 2: -**How to avoid it:** also run queries *outside* the benchmark suite, and be -suspicious of any system whose advantage evaporates off-benchmark. +``` +per cold measurement: + stop the database server + echo 3 > /proc/sys/vm/drop_caches # root, recent Linux + start the server + run and time exactly ONE query + repeat +``` -### Step 6 — pitfall 3.5: conflating cold and hot runs +One query per cycle, because the second query is by definition hot again. §3.6 +also notes the cloud problem: caching also happens on the virtualization host, +where you cannot drop it, so the only option may be to start a fresh virtual +machine — which makes honest cold numbers "very time-consuming and +inconvenient". -A **cold run** starts with empty caches (first query after boot); a **hot -run** has data already cached from previous queries. They can differ by an -order of magnitude, and averaging them produces a number that describes -neither — first-query-of-the-morning and query-in-a-loop are different -user experiences. +**How to avoid it:** flush the OS cache explicitly, per measurement, and treat +any cloud "cold" number as warm until proven otherwise. -**How to avoid it:** report cold and hot *separately*; for hot numbers, -discard the initial iterations — this is criterion's warm-up, formalized at -the system level. +### Step 9 — pitfall 3.7: ignoring preprocessing time -### Step 7 — pitfall 3.6: restart ≠ cold +> **In:** the timed window from Steps 7 and 8. +> **Out:** the work that happens outside it, and the two ways a system gets it +> for free. -Subtler than Step 6: restarting the database server does **not** produce a -cold run, because the OS keeps its own file cache (the page cache) that -survives the process — the restarted server reads "disk" data straight from -the OS's RAM. True cold on Linux = stop server, `echo 3 > -/proc/sys/vm/drop_caches`, start server, run *one* query, repeat the whole -cycle per measurement. (Nearly impossible in the cloud — the hypervisor -caches too, and you can't flush it.) +Excluding preparation — loading, format conversion, index construction — from +the timed window rewards whichever system shifts the most cost into it. §3.7's +statement of the bias: spending more time on index creation generally produces +a faster index, so discarding creation time gives expensive-to-build, +efficient indices an unfair advantage over cheap-to-build, less efficient ones. -**How to avoid it:** flush the OS cache explicitly per cold measurement, and -treat any cloud "cold" number as warm until proven otherwise. +The trap doubles when the preprocessing is *automatic*, and §3.7 gives two +MonetDB examples: -### Step 8 — pitfall 3.7: ignoring preprocessing time +- **Imprints** — a lightweight per-column min/max index that MonetDB builds + automatically the first time a range filter touches a column. Subsequent + range queries on that column are significantly faster. +- **Dictionary encoding** — string columns are stored at load time as integer + offsets into a heap with duplicates eliminated, so string equality in a query + becomes integer comparison. -Excluding preparation work (index builds, data loading, format conversion) -from the timed window rewards systems that shift the most cost into it — an -expensive-to-build index looks free. The trap doubles with *automatic* -preprocessing: MonetDB builds imprints (a min/max index) on the first range -filter and dictionary-encodes strings at load time — so a "cold" first-query -timing silently *includes* that work for MonetDB while a competitor's -equivalent work happened invisibly at load. +Both mean the *first* query pays for an index the later queries enjoy. Discard +the first query as a "cold run" (Step 7) and the index becomes free; keep it +and you charge MonetDB for work a competitor did invisibly at load time. The +two pitfalls interact, which is why they are adjacent in the paper. -**How to avoid it:** either time preprocessing and report it, or verify both -systems did equivalent preparation before the timed window opens. +**How to avoid it:** §3.7's rule is symmetry — either create indexes for both +systems or for neither — plus explicit wariness of automatic index creation. +Where preprocessing is timed, report it. -### Step 9 — pitfall 3.8: incorrect code wins +### Step 10 — pitfall 3.8: incorrect code wins -A fast wrong answer beats every correct system, and nothing in a timing -harness notices. Skipped overflow handling, hardcoded group counts, missed -edge cases — each buys speed and produces plausible-looking output. +> **In:** every number produced by Steps 3–9. +> **Out:** the check that has to happen before any of them count. -**How to avoid it:** diff every result set against a trusted engine, every -run. Correctness checking is part of the benchmark, not a separate activity. +A fast wrong answer beats every correct system, and nothing in a timing +harness notices. §3.8 separates two flavours. An outright bug produces wrong +results and is often *faster* because the bug means less data is touched — and +if the experiment is not reproducible (Step 3), nobody will ever find it. The +subtler flavour is a program correct only for the data it was tested on: +neglected overflow handling (Step 5's term), or hardcoding the number of +groups an aggregation produces. + +**How to avoid it:** §3.8 is specific — compare output against reference +answers, taken either from the benchmark specification or from running the +same query on a well-tested RDBMS such as SQLite or PostgreSQL, and check that +the results stay correct **when the data changes**. Correctness checking is +part of the benchmark, not a separate activity. ## How to read the paper (with the concepts in hand) Six pages, one evening: -- **§1–2** Intro + related work — skim, but note the gems: Jain's *mistakes - vs games* distinction (Step 1); Hoefler & Belli's 12 HPC benchmarking - rules; van der Kouwe's 96%-of-50-papers survey; Purohith et al.'s 28× - SQLite parameter that 0 of 16 papers reported. -- **§3** The eight pitfalls (Steps 2–9), each with its mock TPC-H SF1 - experiment — read carefully; the numbers quoted above all live here. -- **§4 + Appendix A** Conclusions + **the checklist** — the artifact you'll - reuse against every comparison in this repo. - -Methodology to steal from the §3 preamble: their own reporting standard is -**median + non-parametric quantile-based 95% confidence intervals**, all -scripts/configs/plots public — the same philosophy as criterion's bootstrap -CIs, applied to system-level runs. +- **§1–2** Intro and related work — skim, but note the gems: Jain's *mistakes + against games* distinction (§2.1, Step 1); Hoefler and Belli's 12 HPC + benchmarking rules, derived from issues in 120 HPC papers, including their + point that averages are only valid when there is no variance — "which is + almost never the case in benchmarking" (§2.1); van der Kouwe's 96%-of-50 + survey (§2.1); Purohith et al.'s factor-of-28 SQLite parameter that none of + 16 papers reported (§2.2). +- **§3 preamble** The setup of Step 2 — read it before any figure. +- **§3.1–3.8** The eight pitfalls (Steps 3–10), with the mock TPC-H SF1 + experiments in Figures 2 and 3. Every number quoted above lives here. +- **§4 and Appendix A** Conclusions and **the checklist** — the artifact you + will reuse against every comparison in this repo. + +Appendix A's eight groups, condensed: + +| Group | Boxes | +|---|---| +| Choosing your benchmarks | covers the evaluation space; subset justified; stresses the relevant functionality | +| Reproducible | hardware config; DBMS parameters and version; source or binaries; data, schema and queries | +| Optimization | compilation flags; system parameters | +| Apples vs apples | similar functionality; equivalent workload | +| Comparable tuning | different data; various workloads | +| Cold/warm/hot runs | cold and hot differentiated; cold runs flush OS and CPU caches; hot runs ignore initial runs | +| Preprocessing | preprocessing equal between systems; aware of automatic index creation | +| Ensure correctness | verify results; test different data sets; corner cases work | +| Collecting results | several runs; check standard deviation; report robust metrics (median and CIs) | ## Connections to this repo - The capstone's M4 backend shootout and M22 LDBC 3-way FalkorDB comparison - must pass Appendix A — especially 3.2 (tune the *reference* FalkorDB - properly, Step 3) and 3.3 (a young engine missing features is structurally - "TimDB", Step 4 — say so explicitly next to numbers). -- FalkorDB/benchmark audit overlaps: no warmup (3.5 / Step 6), timeout + must pass Appendix A — especially *optimization* (tune the *reference* + FalkorDB properly, Step 4) and *apples vs apples* (a young engine missing + features is structurally TimDB, Step 5 — say so explicitly next to the + numbers). +- FalkorDB/benchmark audit overlaps: no warmup (3.5, Step 7), timeout asymmetry (3.3-ish), uniform keys (3.4's cousin — tuning the *workload* to flatter caches). -- 3.7 (Step 8) is why M0's `workload` crate measures generation throughput +- 3.7 (Step 9) is why M0's `workload` crate measures generation throughput separately from engine time. +- This repo has caught two of these on itself, and both are in + [FINDINGS.md](../../FINDINGS.md): topic 12's scan lane once printed + **19,047,619 GB/s** from a hoisted timing loop — pitfall 3.8, a wrong answer + that was very fast — and now reports **24–57 GB/s** on a 150 GB/s machine. + Topic 6's mmap lane reports p50 **42 ns** against a max of **182 µs**, a + 4300× spread that a mean would have hidden — Hoefler and Belli's point about + averages, measured locally. ## Questions to answer in notes.md -1. Which Appendix A checklist items does FalkorDB/benchmark currently fail? (I count at - least four — list them.) -2. The paper reports medians + CIs; Tene demands full percentile curves + max. When is - each right? (Hint: throughput-style repeated identical runs vs latency under load.) -3. Which "automatic preprocessing" (3.7) exists in FalkorDB that a fair Neo4j - comparison must account for? +1. Which Appendix A checklist items does FalkorDB/benchmark currently fail? (I + count at least four — list them, with the box each one misses.) +2. The paper reports medians with quantile-based CIs; Tene demands full + percentile curves and the max. When is each right? (Hint: repeated identical + runs of one query, against latency under sustained load.) +3. Which "automatic preprocessing" (3.7, Step 9) exists in FalkorDB that a fair + Neo4j comparison must account for? +4. Step 3's arithmetic showed one undisclosed schema choice was worth 2.59×, + more than any system gap it produced. Name an undisclosed variable in this + repo's own lanes that could be worth more than the effect being measured, + and say how you would publish it. +5. `verify.sh` publishes every lane's command and every generator is seeded. + Which Appendix A boxes does that tick, and which does it leave open? ## Takeaway -Appendix A is a reusable review checklist: benchmarks chosen + justified; reproducible -(hardware, params, code, data); both systems optimized; same functionality; cold/hot -separated and correctly collected; preprocessing equalized; results verified; medians + -CIs over several runs. Pin it next to every capstone `notes.md` comparison. +Appendix A is a reusable review checklist: benchmarks chosen and justified; +reproducible (hardware, params, code, data); both systems optimized; same +functionality; cold/hot separated and correctly collected; preprocessing +equalized; results verified; medians and CIs over several runs. Pin it next to +every capstone `notes.md` comparison. ## Done when +Answer each before unfolding it. + - [ ] You can name all eight pitfalls without the paper open. + +
Answer + + Grouped by what they break (Step 1): **setup** — 3.1 non-reproducibility, + 3.2 failure to optimize the baseline; **comparison** — 3.3 apples against + oranges, 3.4 overly-specific tuning; **measurement** — 3.5 conflating cold + and hot runs, 3.6 a restart is not a cold run, 3.7 ignoring preprocessing + time; **results** — 3.8 incorrect code wins. + + The grouping is the recall aid: four requirements for a fair comparison, and + eight ways to fail them. + +
+ - [ ] You can explain why "failure to optimize the baseline" (3.2) is the one that invalidates a result rather than merely weakening it. + +
Answer + + Because both sides of the comparison can be the *same system*. Fig. 3a + measures MonetDB against MonetDB — 1.58 s debug against 0.87 s release, 1.82× + — and Fig. 3b measures Postgres against Postgres, 0.47 s against 0.27 s, + 1.74×. Neither gap contains any information about a system's design. A paper + reporting a 1.8× win over a baseline it built in debug mode has reported a + compiler flag. + + It invalidates rather than weakens because the incentive runs one way. §3.2 + says it: the author has "very little incentive to properly optimize the + current system", so the error is not noise around the truth, it is a bias + that always points at the author's conclusion. A weakness makes a result less + certain; a systematic bias makes it uninformative. + +
+ - [ ] You can state the difference between a restarted process and a cold system (3.6), and name what stays warm across a restart. + +
Answer + + Restarting drops everything the *process* owned — buffer pool, plan cache, + any in-process state. It drops nothing the *kernel* owns, and the kernel owns + the page cache: spare RAM holding recently read disk blocks. The restarted + server therefore reads its data out of memory while believing it read from + disk, and reports a "cold" number that is warm. + + The protocol that actually works (§3.6, footnote 2) is: stop the server, + `echo 3 > /proc/sys/vm/drop_caches` as root, start the server, run and time + exactly one query, repeat — one query per cycle, because the second is hot + again. In a cloud VM even that fails, because the virtualization host caches + too and you cannot reach it; the paper's only suggestion there is to start a + fresh virtual machine. + +
+ - [ ] You can identify which pitfall each of this repo's own lanes is most exposed to — start with topic 6's mmap handicap and topic 12's bandwidth variance. -- [ ] You wrote answers to all three questions in notes.md, including the FalkorDB checklist audit. + +
Answer + + Topic 6's mmap lane reports p50 **42 ns** and a max of **182 µs** + ([FINDINGS.md](../../FINDINGS.md) row 6) — a 4300× spread that is almost + entirely minor page faults. Its exposure is 3.5/3.6: whether the pages were + already resident decides the whole distribution, so any figure from it that + does not say which is a cold/hot conflation. It is also the case Hoefler and + Belli warn about at §2.1 — a mean over that spread describes nothing. + + Topic 12's scan lane reports **24–57 GB/s** on a 150 GB/s machine, and the + same lane once printed **19,047,619 GB/s** from a hoisted timing loop. That + is pitfall 3.8 exactly: incorrect code was very fast, and only an + implausibility check caught it. Its standing exposure now is 3.3 — a + hand-written scan kernel measured against anything that also parses a query + is TimDB. + + More generally: every lane that measures this repo's own code against a + published number inherits 3.1, because the published number's setup (Step 2) + is rarely stated in as much detail as `verify.sh` states ours. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including the FalkorDB checklist audit. + +
Answer + + There is no answer to unfold here — the checklist audit is the exercise. The + bar: for each of the four requirements in Step 1, name the specific + FalkorDB/benchmark behaviour that fails it and the Appendix A box it misses. + An audit that finds nothing has usually confused "I could not reproduce their + setup" (pitfall 3.1, a finding) with "their setup is fine". + +
## References **Papers** - Raasveldt, Holanda, Gubner, Mühleisen — "Fair Benchmarking Considered - Difficult: Common Pitfalls in Database Performance Testing" (DBTest - 2018) — + Difficult: Common Pitfalls in Database Performance Testing" (DBTest 2018) — [PDF](https://hannes.muehleisen.org/publications/DBTEST2018-performance-testing.pdf) — 6 pages, one evening; read §3 carefully, Appendix A is the reusable - artifact. (CWI — Raasveldt & Mühleisen later created DuckDB.) + artifact. (CWI — Raasveldt and Mühleisen later created DuckDB.) + +| Section | What this chapter took from it | +|---|---| +| §2.1 | Jain's mistakes/games distinction; Hoefler & Belli's 12 rules over 120 HPC papers; van der Kouwe's 96% of 50 papers | +| §2.2 | Purohith et al.: SQLite throughput varies by 28×, none of 16 papers reported the parameter | +| §3 preamble | the hardware, versions, single-thread choice and median-with-95%-CI reporting standard (Step 2) | +| §3.1, Fig. 2 | the Escher result: 12.18 / 9.73 / 8.19 / 4.70 s, and the DOUBLE-instead-of-DECIMAL schema choice, both TPC-H-legal | +| §3.2, Fig. 3a-b | MonetDB 1.58 → 0.87 s (debug scans whole columns); Postgres 0.47 → 0.27 s | +| §3.3, Fig. 3c | MonetDB 0.87 s against hand-written 'TimDB' 0.03 s; overflow checking vs prevention | +| §3.4 | what a standardized benchmark publishes: selectivities, cardinalities, group counts | +| §3.5 | why the first run is slower: storage, parse/compile, buffer pool, plan cache | +| §3.6 + fn. 2 | the page cache survives a restart; `drop_caches`; the cloud has no equivalent | +| §3.7 | MonetDB's automatic imprints and load-time dictionary encoding | +| §3.8 | verify against a reference engine, and re-verify when the data changes | +| Appendix A | the checklist, condensed in the table above | **Code** - [pholanda/FairBenchmarking](https://github.com/pholanda/FairBenchmarking) - — the paper's experiment scripts and configs + — the paper's experiment scripts and configs, the artifact §3 preamble + promises. diff --git a/topics/00-performance-toolbox/reading-redis-benchmark.md b/topics/00-performance-toolbox/reading-redis-benchmark.md index 145d6b2..a0673e0 100644 --- a/topics/00-performance-toolbox/reading-redis-benchmark.md +++ b/topics/00-performance-toolbox/reading-redis-benchmark.md @@ -2,72 +2,132 @@ The load generator you'll imitate — and the mistake you'll avoid. In one dependency-free file, redis-benchmark shows a masterclass in cheap pipelining -(one pre-built buffer, patched in place) and, in the same 2000 lines, the +(one pre-built buffer, patched in place) and, in the same 2028 lines, the canonical case of coordinated omission: a closed loop that measures service time and calls it latency. This chapter builds the load-generation concepts -step by step — throughput vs latency, closed loops, pipelining, where the -histogram comes from, and exactly how the numbers go wrong — then hands you -the line-by-line map of the C file. Two questions drive the read: *how does -it implement pipelining, and what does it get wrong about coordinated -omission?* +from zero — throughput against latency, closed loops, pipelining, where the +histogram's samples come from, and exactly how the numbers go wrong — then +hands you the line-by-line map of the C file. Two questions drive the read: +*how does it implement pipelining, and what does it get wrong about +coordinated omission?* + +Every anchor below is Redis **8.6.2** (`src/version.h:1`), the commit +`a176d1225` this repo pins, quoted with the line numbers the code occupies in +that version. ## The problem in one sentence When Redis stalls for 100 ms (a fork for a background save, say), a closed-loop benchmark records *one* bad sample per client instead of the thousands of delayed requests a real workload would have suffered — so the -reported p99 can be 100× better than what users would experience. +reported p99.9 can be hundreds of times better than what users experience. ## The concepts, step by step ### Step 1 — throughput and latency answer different questions -**Throughput** is how many requests per second the server can complete -(capacity: "can Redis do 1M SET/s?"). **Latency** is how long *one* request -takes from the moment a client wants it done to the moment the answer -arrives (experience: "how slow is the p99?" — the p99 being the 99th -percentile, the time that 99% of requests beat and 1% exceed). They are not -two views of one number: a server can post 1M ops/s while some requests take -500 ms, and a load generator built to maximize the first is, as we'll see, -structurally unable to measure the second honestly. +> **In:** nothing yet — this step fixes the vocabulary every later step uses. +> **Out:** two words that are not two views of one number, and the reason the +> tool's design goal decides which of them it can measure. + +**Throughput** is how many requests per second the server completes — a +capacity question, "can Redis do 1M SET/s?". **Latency** is how long *one* +request takes, from the moment a client wanted it done to the moment the +answer arrived — an experience question, "how slow is the p99?". + +A **percentile** is the value a given fraction of samples falls below: the +**p99** is the time 99% of requests beat and 1% exceed; the **p99.9** is the +time 999 requests in 1000 beat. Percentiles are what latency is reported in +because the mean hides exactly the tail users complain about. -Why it matters: redis-benchmark's design goal is throughput; every latency +These are not two views of one number. A server can post 1M ops/s while some +requests take 500 ms — the 1M is an average over a second, the 500 ms is one +request's experience inside it. And a load generator built to maximize the +first is, as Steps 2 to 6 show, structurally unable to measure the second +honestly. + +Why it matters: redis-benchmark's design goal is throughput. Every latency figure it prints has to be read with that in mind. -### Step 2 — closed-loop load generation +### Step 2 — the closed loop: the server sets the send rate + +> **In:** the vocabulary from Step 1. +> **Out:** the send cycle every later step is built on — and the fact that no +> target rate exists anywhere in it. Step 6 turns that absence into the bug. A **closed loop** is the simplest possible client: send a request, wait for -the reply, send the next — the client and server take turns, and at most one -request (per client connection) is ever outstanding. The whole of -redis-benchmark is this cycle: +the reply, send the next. Client and server take turns, so at most one request +per connection is ever outstanding (or one *batch* of them, once Step 3 adds +pipelining). An **event loop** — Redis's own `ae` library, reused here — is +the thing that drives it: a single thread that waits for file descriptors to +become readable or writable and calls a handler for each. + +The whole of redis-benchmark is this cycle: ```mermaid flowchart LR - W["writeHandler (555)
c->start = ustime()"] --> R["readHandler (442)
latency = now − start
on FIRST reply only (452)"] - R --> D["clientDone (420)"] - D --> RC["resetClient (368)"] - RC -->|"next batch starts only after
the previous one finished"| W + W["writeHandler 555
c->start = ustime() at 574"] --> R["readHandler 442
c->latency = ustime()-c->start
at 452, first read event only"] + R --> D["clientDone 420"] + D --> RC["resetClient 368
c->pending = config.pipeline at 374"] + RC -->|"the next batch starts only
after the previous one finished"| W +``` + +The closed loop itself is eight lines, and worth reading in full because the +absence in it is the point: + +```c +// src/redis-benchmark.c — resetClient, the whole closed loop, 368-375 + 368 static void resetClient(client c) { + 369 aeEventLoop *el = CLIENT_GET_EVENTLOOP(c); + 370 aeDeleteFileEvent(el,c->context->fd,AE_WRITABLE); + 371 aeDeleteFileEvent(el,c->context->fd,AE_READABLE); + 372 aeCreateFileEvent(el,c->context->fd,AE_WRITABLE,writeHandler,c); + 373 c->written = 0; + 374 c->pending = config.pipeline; + 375 } ``` -In the code: `clientDone` (line 420) → `resetClient` (line 368 — the closed -loop, in 8 lines) → re-arm the write handler → the next batch starts *after* -the previous one finished. Notice what does **not** exist anywhere in the -cycle: a target request rate or an intended send schedule. The client sends -exactly as fast as the server answers — no faster, and crucially, no matter -what, never *during* a server stall. +Line 372 is the one that closes the loop: it re-arms the *write* handler, so +the next batch is sent the instant this one finished being read. `clientDone` +(420) calls it when `config.keepalive` is set (428-429), and reconnects +instead when it is not (430-437). + +Now look for what is *not* there: no target request rate, no intended send +schedule, no clock the loop is trying to keep up with. The client sends +exactly as fast as the server answers — no faster, and, crucially, not at all +*during* a server stall. -Why it matters: closed loops are trivial to write and great for finding peak -throughput, but the send rate is controlled by the *server* — hold that -thought for Step 5. +Why it matters: closed loops are trivial to write and are the right tool for +finding peak throughput, but the send rate is controlled by the **server**. +Hold that thought for Step 6. -### Step 3 — pipelining: k requests in flight amortize the round trip +### Step 3 — pipelining: one pre-built buffer, k commands deep + +> **In:** the send cycle from Step 2. +> **Out:** a batch of `config.pipeline` commands as the unit that gets sent — +> which Step 4 then treats as a single timing unit. **Pipelining** means sending k requests back-to-back without waiting for -replies, then collecting all k answers — so one network round trip (the -~50–500 µs of wire and kernel time per exchange) is paid once per *batch* -instead of once per request. With a 100 µs round trip and a 1 µs command, -unpipelined throughput caps at ~10K ops/s per connection; `-P 100` lifts it -near 1M. +replies, then collecting all k answers. One **round trip** — the wire, kernel +and syscall time of a single send-and-receive exchange, ~50-500 µs on real +networks and ~10-50 µs on loopback — is then paid once per *batch* instead of +once per request. + +The arithmetic, on a 100 µs round trip and a 1 µs command: + +``` +unpipelined: 1 command / (100 µs RTT + 1 × 1 µs) = 1/101 µs = 9,901 ops/s +-P 100: 100 commands / (100 µs RTT + 100 × 1 µs) = 100/200 µs = 500,000 ops/s +-P 1000: 1000 commands / (100 µs RTT + 1000 × 1 µs) = 1000/1100 µs = 909,091 ops/s +ceiling as k grows: 1 / 1 µs = 1,000,000 ops/s +``` + +So `-P 100` is a 50× lift and gets you *half* way to the 1M/s ceiling, not to +it: the round trip is still half the batch's cost. The ceiling is the command +time alone, and you only approach it once `k × service ≫ RTT`. Topic 7 +measures the real version of this curve on loopback with zero-work requests: +**44k ops/s at P=1 against 12.3M at P=256** ([FINDINGS.md](../../FINDINGS.md) +row 7). redis-benchmark's implementation is the elegant part — there is no request queue at all, just one pre-built buffer: @@ -81,145 +141,433 @@ c->obuf — the whole benchmark is one pre-built buffer, written over and over: trimmed after 1st reply └── randptr[] patch digits in place — no re-serialization ``` -`createClient` (line 625) copies the *same command bytes* `config.pipeline` -times into one output buffer `c->obuf`, sets `c->pending = config.pipeline`, -and the event loop just writes the whole buffer and counts replies back down -(`readHandler`, line 458: `while(c->pending)`). Randomized keys are patched -*in place* through saved pointers into the buffer (`randptr`, lines 377–393 — -writes digits directly into the command bytes, no re-serialization). -Auth/SELECT prefix commands ride in the same buffer once and are trimmed -after the first reply (lines 506–523). - -Cost of the trick: within one batch every pipelined command has the *same* -key randomization per slot of the buffer, and — see Step 4 — the whole batch -becomes one timing unit. - -Why it matters: this is the minimum possible work per event-loop tick, and -it's the part worth stealing for your own load generator. - -### Step 4 — where the latency histogram comes from - -A **latency histogram** counts how many requests fell into each time bucket, -so percentiles (p50, p99, p99.9) can be read off it afterward; -redis-benchmark uses HdrHistogram (a histogram with buckets sized to keep a -fixed relative error at every magnitude — two of them live in `struct -config`, lines 99–100). But a histogram is only as honest as the samples fed -into it, and here's exactly what gets fed: - -- `writeHandler` line 574: `c->start = ustime()` when a batch begins writing. -- `readHandler` line 452: `if (c->latency < 0) c->latency = ustime() - c->start` - — **on the first read event only**. So "latency" = batch send → first - bytes of first reply. Deliberate (the comment says parsing overhead - shouldn't count), but it means the last reply's extra wait is invisible. -- Lines 528–541: that *single* value is recorded into the HdrHistogram - **once per reply** — all `pipeline` requests inherit the first reply's - latency. With `-P 100`, one measurement pretends to be 100. - -`showLatencyReport` (line 830+) then prints beautiful full percentiles — of -that sample. - -Why it matters: the display machinery is state of the art; the *sampling* is -one clock read per batch, duplicated. Good percentiles of a biased sample -are still biased. - -### Step 5 — coordinated omission: the closed loop under-samples the worst moments +`createClient` (625) builds it. The replication is two lines, a hundred lines +into the function — this is the trick worth stealing: + +```c +// src/redis-benchmark.c — inside createClient, 719-731 + 719 c->prefixlen = sdslen(c->obuf); + 720 /* Append the request itself. */ + 721 if (from) { + 722 c->obuf = sdscatlen(c->obuf, + 723 from->obuf+from->prefixlen, + 724 sdslen(from->obuf)-from->prefixlen); + 725 } else { + 726 for (j = 0; j < config.pipeline; j++) + 727 c->obuf = sdscatlen(c->obuf,cmd,len); + 728 } + 729 + 730 c->written = 0; + 731 c->pending = config.pipeline+c->prefix_pending; +``` + +Line 727 is the whole of it: the *same command bytes* appended +`config.pipeline` times into one output buffer. Line 731 sets the reply +counter to match, and the read side just counts back down — +`while(c->pending)` at 458. Note that the counter is `config.pipeline` **plus** +`c->prefix_pending`, because AUTH, SELECT and HELLO 3 ride in the same buffer +(705-717) and are trimmed after their replies arrive (510-521). + +Randomized keys are patched *in place* through saved pointers into that +buffer, so a new key costs twelve stores and no re-serialization: + +```c +// src/redis-benchmark.c — randomizeClientKey, 377-393 + 377 static void randomizeClientKey(client c) { + 378 size_t i; + 379 + 380 for (i = 0; i < c->randlen; i++) { + 381 char *p = c->randptr[i]+11; + 382 size_t r = 0; + 383 if (config.randomkeys_keyspacelen != 0) + 384 r = random() % config.randomkeys_keyspacelen; + 385 size_t j; + 386 + 387 for (j = 0; j < 12; j++) { + 388 *p = '0'+r%10; + 389 r/=10; + 390 p--; + 391 } + 392 } + 393 } +``` + +The line to look at is 388: it writes one decimal digit directly into the +command bytes. `c->randptr[i]` points at the `:rand:` placeholder inside +`c->obuf` itself, so there is no format string, no allocation, and no copy on +the hot path. + +Cost of the trick: the buffer is rewritten once per batch (571, via +`writeHandler`), so every slot gets a fresh key, but all `config.pipeline` +commands in a batch are randomized together and sent together — and, as +Step 4 shows, timed together. + +Why it matters: this is close to the minimum possible work per event-loop +tick, and it is the part worth stealing for your own load generator. + +### Step 4 — one clock read per batch + +> **In:** the batch of `config.pipeline` commands from Step 3. +> **Out:** exactly one number, `c->latency`, per batch — the sample Step 5 +> feeds to the histograms. + +The clock starts when a batch begins writing: + +```c +// src/redis-benchmark.c — inside writeHandler, 561-576 + 561 /* Initialize request when nothing was written. */ + 562 if (c->written == 0) { + // ... 563-568: stop if config.requests has already been issued ... + 570 /* Really initialize: randomize keys and set start time. */ + 571 if (config.randomkeys) randomizeClientKey(c); + // ... 572-573: cluster-mode hash tags and slot epoch ... + 574 c->start = ustime(); + 575 c->latency = -1; + 576 } +``` + +Line 574 is the one that matters, and note the guard on 562: the clock is set +when the *first* byte of a batch is written, not on every partial write. Line +575 arms the sentinel that the read side tests. + +It stops on the first byte of the first reply: + +```c +// src/redis-benchmark.c — the top of readHandler, 449-452 + 449 /* Calculate latency only for the first read event. This means that the + 450 * server already sent the reply and we need to parse it. Parsing overhead + 451 * is not part of the latency, so calculate it only once, here. */ + 452 if (c->latency < 0) c->latency = ustime()-(c->start); +``` + +Line 452 is the whole measurement, and the `< 0` test is what makes it happen +once. So what redis-benchmark calls latency is precisely: **the interval from +"we started writing k commands" to "the first bytes of the first reply came +back"**. The comment on 449-451 says why parsing is excluded, and that +reasoning is sound. What it does not say is the consequence: the k-th reply's +extra wait is outside the interval entirely, and is never measured. + +Why it matters: one `ustime()` pair per batch is the entire sampling +apparatus. Everything printed later is a rendering of these numbers. + +### Step 5 — the fork: one sample, two histograms, k recordings + +> **In:** the single `c->latency` value from Step 4. +> **Out:** two datasets — a cumulative HdrHistogram that +> `showLatencyReport` prints at the end, and a per-second one that drives the +> live line. Both are filled once per *reply*, not once per measurement. + +A **latency histogram** counts how many samples fell into each time bucket, so +percentiles can be read off it afterwards without keeping every sample. +**HdrHistogram** is the standard implementation: its buckets are sized to hold +a fixed *relative* error at every magnitude, so 1 µs and 1 s are both recorded +to the same number of significant digits, in constant time and constant space. + +Two of them live in `struct config` (99-100), and this is where the run's data +forks in two: + +```c +// src/redis-benchmark.c — inside struct config, 99-100 + 99 struct hdr_histogram* latency_histogram; + 100 struct hdr_histogram* current_sec_latency_histogram; +``` + +`latency_histogram` accumulates the whole run and is what `showLatencyReport` +(830) reads for the final percentiles (833-838). `current_sec_latency_histogram` +is reset every second and drives the live progress line. Same samples, two +consumers — so a claim about "the p99 redis-benchmark printed" is a claim +about the first one. + +The recording is inside the reply loop: + +```c +// src/redis-benchmark.c — inside readHandler's while(c->pending), 524-543 + 524 int requests_finished = 0; + 525 atomicGetIncr(config.requests_finished, requests_finished, 1); + 526 if (requests_finished < config.requests){ + 527 if (config.num_threads == 0) { + 528 hdr_record_value( + 529 config.latency_histogram, // Histogram to record to + 530 (long)c->latency<=CONFIG_LATENCY_HISTOGRAM_MAX_VALUE ? (long)c->latency : CONFIG_LATENCY_HISTOGRAM_MAX_VALUE); // Value to record + 531 hdr_record_value( + 532 config.current_sec_latency_histogram, // Histogram to record to + 533 (long)c->latency<=CONFIG_LATENCY_HISTOGRAM_INSTANT_MAX_VALUE ? (long)c->latency : CONFIG_LATENCY_HISTOGRAM_INSTANT_MAX_VALUE); // Value to record + // ... 534-541: the same two calls again, hdr_record_value_atomic, when threads are on ... + 542 } + 543 c->pending--; +``` + +The load-bearing detail is *where* this sits: inside `while(c->pending)` (458), +which spins once per reply. `c->latency` was computed once, at 452, and is not +recomputed — so the same value is recorded `config.pipeline` times. Run with +`-P 100` and a million requests, and the histogram holds 1,000,000 entries +drawn from **10,000** clock readings. One measurement pretending to be a +hundred. +Line 530 adds a second distortion: `CONFIG_LATENCY_HISTOGRAM_MAX_VALUE` is +`3000000L` µs (line 50), so any sample above **3 s** is recorded *as* 3 s. The +worst outliers are truncated, not lost — which flatters the maximum. + +`showLatencyReport` (830) then computes p50, p95, p99 and the max off this +histogram (834-837) and prints them to two decimal places. The display is +state of the art. The samples are one clock read per batch, duplicated. + +Why it matters: good percentiles of a biased sample are still biased, and +nothing downstream of line 452 can recover what line 452 did not measure. + +### Step 6 — coordinated omission: the closed loop under-samples the worst moments + +> **In:** the rate-free cycle (Step 2), the batch as timing unit (Steps 3-4), +> and the histograms (Step 5). +> **Out:** the named defect, its size in requests, and the one structural +> change that fixes it. + +**Service time** is how long the server took once it picked a request up. +**Queueing delay** is how long the request waited before that — behind other +work, or behind a stall. **Latency**, as a user experiences it, is the sum. **Coordinated omission** (Gil Tene's term) is the measurement error where the -load generator, by waiting for the server, silently *coordinates* with it — +load generator, by waiting for the server, silently *coordinates* with it: the requests that would have arrived during a stall are never sent, so the worst -moments are systematically under-sampled and the p99 lies. Steps 2–4 combine -into exactly this. In Tene's terms: - -1. **No target rate exists.** The benchmark always sends as fast as the - server answers, so a stall (fork for RDB save, AOF fsync, slow command) - simply pauses the generator — requests that *would* have arrived during - the stall are never sent, never measured. You get exactly one bad sample - per client per stall instead of thousands. -2. **It measures service time and calls it latency.** Service time = how long - the server took once it picked the request up; latency = service time - *plus the queueing delay* a real open-world client would experience - waiting behind the stall. The queueing delay never appears. -3. **HdrHistogram doesn't save it.** Redis added HdrHistogram (config lines - 99–100) and full percentile output (830+) — good display of a *biased* - sample (Step 4). Correction would require an intended-arrival schedule, - which doesn't exist here. (Compare wrk2, which was written to fix exactly - this; memtier_benchmark has `--rate-limiting`.) -4. Small extra: `hdr_record_value` clamps at - `CONFIG_LATENCY_HISTOGRAM_MAX_VALUE` (line 530) — the worst outliers are - also truncated. +moments are systematically under-sampled and the reported percentiles lie. + +Steps 2 to 5 combine into exactly this, in four parts: + +1. **No target rate exists.** Step 2 found no intended send schedule anywhere + in the cycle. A stall — a fork for an RDB save, an AOF fsync, one slow + `KEYS` — simply pauses the generator. Requests that *would* have arrived + during the stall are never sent, so they are never measured. +2. **It measures service time and calls it latency.** Step 4's interval starts + when the client writes, which for a closed loop is always *after* the + previous reply. The queueing delay of Step 6's definition never appears, + because the client was never queued. +3. **HdrHistogram does not save it.** Redis added two HdrHistograms (99-100) + and full percentile output (830) — excellent display of the biased sample + from Step 5. Correction needs an intended-arrival schedule, which does not + exist here. (Compare wrk2, written specifically to fix this; + memtier_benchmark has `--rate-limiting`.) +4. **The clamp truncates what survives.** Step 5's 3 s ceiling (line 50, used + at 530) caps the few honest outliers a stall does produce. Both loops, distilled to their timing skeletons — the entire bug and the entire fix is *where the clock starts*: ```rust -// closed loop (redis-benchmark): clock starts at SEND — a server stall -// pauses the generator, so the requests that would have queued up behind -// the stall are never sent, never measured. +// ILLUSTRATION — not quoted from Redis. The closed loop is the real cycle of +// src/redis-benchmark.c:368-375 with its clock reads at 574 and 452; the open +// loop is what the tool would need and does not have. + +// closed loop (redis-benchmark): the clock starts at SEND, so a server stall +// pauses the generator and the requests that would have queued behind the +// stall are never sent, never measured. loop { let start = now(); send_batch_and_wait_all_replies(); - record(now() - start); // one bad sample per stall + record(now() - start); // one bad sample per stall } -// open loop (the fix): clock starts at the INTENDED send time — the +// open loop (the fix): the clock starts at the INTENDED send time, and the // schedule advances whether or not the server keeps up. let mut intended = now(); loop { - intended += period; // target rate exists + intended += period; // a target rate exists wait_until(intended); - send_one(); // reply handled async + send_one(); // reply handled asynchronously on_reply(move |t| record(t - intended)); // queueing delay is visible } ``` -Why it matters: worked example of the 100 ms fork stall from the problem -statement — a closed loop with 50 clients records 50 samples of ~100 ms; an -open loop at 100K req/s records ~10,000 samples spanning 0–100 ms of -queueing delay. Same server, same stall; only the second histogram tells the -truth about it. +Worked example — the 100 ms fork stall from the problem statement, on a run of +1,000,000 requests with 50 clients, against an open-loop generator running the +same server at a target 100,000 req/s: + +``` +closed loop: 50 clients × 1 stalled batch = 50 samples of ~100 ms + 50 / 1,000,000 = 0.005% of the histogram + → first visible at the p99.995; p50, p99 and p99.9 are untouched + +open loop: 100,000 req/s × 0.100 s = 10,000 requests delayed + 10,000 / 1,000,000 = 1% of the histogram + delays spread ~uniformly over 0-100 ms, so: + p99.5 = the 5,000th of them = ~50 ms + p99.9 = the 9,000th of them = ~90 ms +``` + +Same server, same 100 ms stall, one run each. The closed loop's p99.9 is a +normal healthy figure — a few hundred microseconds; the open loop's is ~90 ms, +two and a half orders of magnitude worse and correct. Only the second +histogram tells the truth about the stall. + +Why it matters: this is not a bug you can fix downstream. No histogram, no +percentile estimator and no amount of sample count repairs a sample that was +never taken. ## Where each step lives in the code -One file, `src/redis-benchmark.c`, readable top to bottom in an evening: +One file, `src/redis-benchmark.c` (2028 lines at `a176d1225`), readable top to +bottom in an evening: | Lines | What | Step | |-------|------|------| -| 61–108 | `struct config` — all global state, incl. `pipeline`, two HdrHistograms (99–100) | 3, 4 | -| 110–130 | `struct _client` — note `start`, `latency`, `pending` | 2, 4 | -| 368–375 | `resetClient` — the closed loop, in 8 lines | 2 | -| 420–439 | `clientDone` — finished batch → `resetClient` (keepalive) or reconnect | 2 | -| 442–553 | `readHandler` — latency capture + histogram recording | 4, 5 | -| 555–602 | `writeHandler` — batch start, `c->start = ustime()` | 4 | -| 625+ | `createClient` — pipelining via buffer replication | 3 | -| 830+ | `showLatencyReport` — percentiles off HdrHistogram | 4 | -| 946 | `benchmark()` — sets up clients, runs the event loop | 2 | -| 1696 | `main` — test loop over SET/GET/INCR/... | 1 | +| 49-51 | `CONFIG_LATENCY_HISTOGRAM_*` — the 10 µs floor and the 3 s clamp | 5 | +| 61-108 | `struct config` — all global state, incl. `pipeline`, two HdrHistograms (99-100) | 3, 5 | +| 110-130 | `struct _client` — note `start` (120), `latency` (121), `pending` (122) | 2, 4 | +| 368-375 | `resetClient` — the closed loop, in 8 lines | 2 | +| 377-393 | `randomizeClientKey` — digits patched into the buffer in place | 3 | +| 420-439 | `clientDone` — finished batch → `resetClient` (429) or reconnect (430-437) | 2 | +| 442-553 | `readHandler` — latency capture (452), prefix trim (510-521), histogram recording (528-541) | 4, 5 | +| 555-602 | `writeHandler` — batch start, `c->start = ustime()` at 574 | 4 | +| 625-812 | `createClient` — pipelining by buffer replication (726-727) | 3 | +| 830-921 | `showLatencyReport` — percentiles off the cumulative histogram (833-838) | 5 | +| 946-982 | `benchmark()` — allocates both histograms (954-963), runs the event loop, calls the report (976) | 2, 5 | +| 1696 | `main` — the test loop over SET/GET/INCR/… | 1 | Suggested route: `main` (1696) → `benchmark()` (946) → `createClient` (625, -Step 3's buffer trick) → the Step 2 cycle (`writeHandler` 555 → -`readHandler` 442 → `clientDone` 420 → `resetClient` 368) — and as you trace -it, confirm for yourself that no intended-arrival schedule exists anywhere -(Step 5). +for Step 3's buffer trick at 726-727) → then the Step 2 cycle in order, +`writeHandler` (555) → `readHandler` (442) → `clientDone` (420) → +`resetClient` (368). As you trace it, confirm for yourself that no +intended-arrival schedule exists anywhere — that absence is Step 6. + +## Questions to answer in notes.md + +1. `readHandler` computes `c->latency` at line 452 but records it inside the + loop at 528. How many histogram entries does one `-P 100` batch produce, + and how many `ustime()` calls paid for them? +2. The comment at 449-451 justifies measuring only the first read event + ("parsing overhead is not part of the latency"). Is that reasoning right? + What does it cost, for a batch of 100? +3. `c->pending` is set to `config.pipeline` in `resetClient` (374) but to + `config.pipeline + c->prefix_pending` in `createClient` (731). Why the + difference, and what happens to the prefix replies' latencies (506-523)? +4. If you added `--rate` to this tool, which of the two clock reads (574, 452) + would have to move, and what new state would `struct _client` need? +5. Sketch what the histogram from Step 6's worked example looks like in each + loop. Which percentile is the first to differ? ## Takeaway -redis-benchmark is a *throughput* tool with percentile decoration: buffer-replication -pipelining is a masterclass in doing the minimum work per event-loop tick, but the -closed loop means its latency numbers systematically flatter the server under stress. -For the capstone (M7+): keep the obuf trick, add an intended-send schedule. +redis-benchmark is a *throughput* tool with percentile decoration: +buffer-replication pipelining is a masterclass in doing the minimum work per +event-loop tick, but the closed loop means its latency numbers systematically +flatter the server under stress. For the capstone (M7+): keep the `obuf` +trick, add an intended-send schedule. ## Done when +Answer each before unfolding it. + - [ ] You can explain the difference between service time and latency, and say which one `redis-benchmark` reports. + +
Answer + + Service time is how long the server took once it picked the request up; + latency as a user experiences it is service time *plus* the queueing delay + spent waiting to be picked up. redis-benchmark reports service time. Its + clock starts at line 574, when the client begins writing a batch — and in a + closed loop the client only begins writing after the previous reply landed, + so by construction it was never queued. There is no moment in the cycle at + which a request exists but has not been sent, which is exactly the interval + queueing delay would occupy. + +
+ - [ ] You can define coordinated omission and explain the mechanism by which a closed loop under-samples exactly the worst moments. + +
Answer + + Coordinated omission is the error where the load generator waits for the + server and thereby *coordinates* with it: the requests that would have + arrived during a stall are never issued, so the stall is under-represented + in the sample rather than over-represented as it is in production. + + The mechanism is `resetClient` (368-375). Line 372 re-arms the write handler + only after the batch's last reply was consumed, so the send rate is a + function of the server's completion rate. During a 100 ms stall each client + contributes exactly one in-flight batch — 50 clients, 50 bad samples — where + an open loop at 100,000 req/s would have issued 10,000 requests into it. + The stall is 0.005% of the closed loop's histogram and 1% of the open loop's. + +
+ - [ ] You can say why adding HdrHistogram to a closed-loop generator does not fix it. + +
Answer + + Because HdrHistogram is a recording and rendering structure, and the defect + is in the sampling. It buys constant-space storage at fixed relative error + and lets `showLatencyReport` (830) compute exact percentiles at 833-838 — + of whatever it was given. What it was given is one clock read per batch + (452), replicated `config.pipeline` times (528-541), with the 10,000 + requests a stall would have delayed simply absent. Correcting it needs an + intended-arrival schedule to subtract from, and no such timestamp exists in + `struct _client` (110-130). The 3 s clamp at line 530 makes it slightly + worse, capping the honest outliers that do survive. + +
+ - [ ] You can predict what pipelining does to the reported figure, then check your prediction against topic 7's `loopback_bench` — 44k ops/s at P=1 against 12.3M at P=256. + +
Answer + + Throughput rises steeply and then saturates, because the round trip is + amortised over k commands while the per-command service time is not: at a + 100 µs RTT and a 1 µs command, P=1 gives 9,901 ops/s, P=100 gives 500,000, + P=1000 gives 909,091, and the ceiling is 1,000,000. Topic 7's measured + 44k → 12.3M over P=1 → P=256 is the same curve with the per-request cost + being syscalls rather than a network hop. + + The reported *latency* moves the other way and stops meaning what it says: + the interval measured at 452 is now "send 256 commands, get the first reply + back", and it is written into the histogram 256 times. A larger `-P` makes + throughput look better, latency look worse, and the sample count look 256× + more trustworthy than it is. + +
+ - [ ] You can name what you would have to change in the tool to make its latency numbers trustworthy (a target rate, and the arithmetic that goes with it). +
Answer + + Three changes, in dependency order. First, `struct _client` (110-130) needs + an `intended` timestamp alongside `start` (120), advanced by a fixed + `period = 1/rate` regardless of when the previous reply arrived. Second, the + measurement at 452 has to subtract `c->intended`, not `c->start`, so + queueing delay is inside the interval. Third — and this is the part that + makes it a rewrite rather than a patch — `resetClient` (368-375) must stop + gating the next send on the previous reply, which means replies can no + longer be counted down with `while(c->pending)` on a single in-flight batch; + the client needs several batches outstanding and a way to match replies to + their intended times. + + The pipelining machinery survives all of this: `c->obuf` and the + `randptr` in-place patching (377-393) are orthogonal to when you decide to + write the buffer. + +
+ ## References **Code** -- [redis](https://github.com/redis/redis) `src/redis-benchmark.c` (2028 - lines, pinned at Redis 8.6.2 / `a176d1225`) — one file, no dependencies - beyond hiredis + the `ae` event loop; readable top to bottom in an - evening +- [redis](https://github.com/redis/redis) `src/redis-benchmark.c` (2028 lines, + pinned at Redis 8.6.2 / `a176d1225` — version confirmed in + `src/version.h:1`) — one file, no dependencies beyond hiredis and the `ae` + event loop; readable top to bottom in an evening. + +| File | Lines | What | +|------|-------|------| +| `src/redis-benchmark.c` | 49-51 | histogram floor (10 µs) and clamp (3,000,000 µs) | +| `src/redis-benchmark.c` | 99-100 | the two HdrHistograms — the data fork of Step 5 | +| `src/redis-benchmark.c` | 368-375 | `resetClient`, the closed loop | +| `src/redis-benchmark.c` | 377-393 | `randomizeClientKey`, in-place digit patching | +| `src/redis-benchmark.c` | 449-452 | the only latency measurement in the tool | +| `src/redis-benchmark.c` | 528-541 | one sample recorded once per reply | +| `src/redis-benchmark.c` | 574 | `c->start = ustime()` — where the clock starts | +| `src/redis-benchmark.c` | 726-727 | pipelining, by appending the same bytes k times | +| `src/redis-benchmark.c` | 833-838 | the percentiles that get printed | + +**Background** +- Gil Tene, *How NOT to Measure Latency* — the talk that named coordinated + omission, and the source of the open-loop correction sketched in Step 6. +- [wrk2](https://github.com/giltene/wrk2) and + [memtier_benchmark](https://github.com/RedisLabs/memtier_benchmark) + (`--rate-limiting`) — load generators that carry the intended-arrival + schedule redis-benchmark lacks. diff --git a/topics/00-performance-toolbox/reading-rocksdb-db-bench.md b/topics/00-performance-toolbox/reading-rocksdb-db-bench.md index 0bde610..ee4ef94 100644 --- a/topics/00-performance-toolbox/reading-rocksdb-db-bench.md +++ b/topics/00-performance-toolbox/reading-rocksdb-db-bench.md @@ -2,7 +2,7 @@ `fillseq`, `readrandom`, `readwhilewriting` — these workload names started in LevelDB, were extended by RocksDB, and now appear in every LSM paper since. -This chapter is a skim route through the 10,000-line tool that defines them — +This chapter is a skim route through the 10,367-line tool that defines them — but first it builds the concepts step by step: why a shared workload vocabulary exists at all, how every workload reduces to picking an integer, what each name actually stresses, and what to distrust in the numbers the @@ -10,239 +10,929 @@ tool prints. The goal is the *vocabulary* and the measurement shape, not the harness code. Name your own benchmarks in this language and your numbers become comparable to two decades of published results. +Every anchor below is RocksDB at commit **`7c80a5a`**, the revision this repo +pins (`resources/codebases.md`, pin table), quoted with the line numbers the +code occupies in that revision. `tools/db_bench_tool.cc` is **10,367 lines** +there; it churns fast, so on any other commit re-grep before trusting a +number. `tools/pinned-source.py show rocksdb tools/db_bench_tool.cc -r +6107:6119` opens exactly what is quoted here. + ## The problem in one sentence "Our engine does 500K writes/s" is uninterpretable — sequential or random -keys? new inserts or overwrites? uniform or skewed? measured during -compaction or before it? — and each of those choices swings the number by -5–50×, so without a shared workload vocabulary no two papers' numbers can be -compared. +keys? new inserts or overwrites? uniform or skewed? durable or buffered? +measured during compaction or before it? — and this repo has already measured +one of those axes on its own: the durability choice alone moves the ceiling +from **856,898/s** (buffered `write()`) to **44,109/s** (`fsync`) to +**337/s** (`F_FULLFSYNC`), a 2,542× spread +([FINDINGS.md](../../FINDINGS.md) row 5). Without a shared workload +vocabulary, no two papers' numbers name the same experiment. ## The concepts, step by step ### Step 1 — why storage engines need a standard workload vocabulary -A storage engine's performance is not one number but a surface: it depends -on the operation mix (reads vs writes vs scans), the key order (sequential -vs random), whether keys are new or overwrite old ones, and what background -work is running. This matters most for **LSM engines** (log-structured -merge-trees: writes go to sorted files that are later merged — "compacted" — -in the background; **compaction** is that deferred merging work, and it -competes with foreground traffic for IO and CPU). The same LSM can absorb -sequential loads at disk bandwidth and collapse to a tenth of that under -random overwrites, purely because of compaction debt. - -db_bench's fix: give each meaningful point on that surface a *name*, so -"we ran `fillrandom` then `readwhilewriting`" pins down the experiment as -precisely as a chess opening's name pins down twelve moves. +> **In:** nothing yet — this step fixes the vocabulary and the units every +> later step uses. +> **Out:** the axes a benchmark name has to pin down, and the measured size of +> two of them. Step 2 then shows that all of the *key-order* axis reduces to +> one function. + +A storage engine's performance is not one number but a surface: it depends on +the operation mix (reads vs writes vs scans), the key order (sequential vs +random), whether keys are new or overwrite old ones, whether each write is +made durable, and what background work is running. + +Four terms, defined before anything leans on them: + +- An **LSM engine** (log-structured merge-tree) buffers writes in memory, then + writes them out as immutable sorted files that are later merged in the + background. +- **Compaction** is that deferred merging: reading several sorted files and + writing one merged file, discarding shadowed versions. It is real IO and CPU + that competes with foreground traffic, and it happens *after* the write that + caused it was already reported as fast. +- **Write amplification** is bytes actually written to storage ÷ bytes the + application asked to write. It is the price of compaction. Topic 4's notes + give the arithmetic: leveled compaction with size ratio `T` over `L` levels + rewrites each byte about `T/2` times per level, so `T/2 × L` — at `T=10`, + `L=4` that is **~20×** ([topic 4 notes](../04-lsm-deep-dive/notes.md)). +- **Space amplification** is bytes on disk ÷ logical bytes. Topic 1 measured + it end to end on the same 108 MB of records: **0.45× for fjall** (an LSM, + which compresses its sorted runs) against **63.28× for redb** (a + copy-on-write B-tree under random-order inserts) — a 140× spread + ([FINDINGS.md](../../FINDINGS.md) row 1). + +So the same engine can absorb a sequential load near disk bandwidth and +collapse under random overwrites, purely because of compaction debt — and the +same *workload* can produce wildly different verdicts on two engine families. + +db_bench's fix: give each meaningful point on that surface a *name*, so "we +ran `fillrandom` then `readwhilewriting`" pins down the experiment as +precisely as a chess opening's name pins down twelve moves. The menu of names +is one flag, `DEFINE_string(benchmarks, …)` at **115-170**, with its own help +text at **172-273** — that help text is the best documentation the tool has. Why it matters: LevelDB shipped these names in 2011, RocksDB extended them, and every LSM paper since reports in them — the vocabulary *is* the comparability. -### Step 2 — every workload is "pick an integer, format it as a key" - -Under every workload name sits the same skeleton: choose an integer, then -format it as a zero-padded fixed-width key. All the drama — sequential vs -random, insert vs overwrite — lives entirely in *how the next integer is -chosen* (`GenerateKeyFromInt` :3802, `WriteMode` :5869, `KeyGenerator` -:6088): - -```rust -// every fill* workload reduces to how the next integer is chosen -fn next_key(mode: WriteMode, i: u64, n: u64, rng: &mut Rng, perm: &[u64]) -> Key { - let int = match mode { - WriteMode::Sequential => i, // fillseq: in-order, no - // compaction debt - WriteMode::Random => rng.next() % n, // fillrandom/overwrite: - // duplicates → garbage - // → compaction pressure - WriteMode::UniqueRandom => perm[i as usize], // pre-shuffled permutation: - // random order, no dups - }; - generate_key_from_int(int) // zero-padded fixed width -} -``` - -The three modes differ in one property each: `Sequential` arrives in order; -`Random` draws with replacement, so ~37% of a full pass are duplicates — -each duplicate is an overwrite that turns an old value into garbage the -engine must compact away; `UniqueRandom` pre-shuffles `0..n` so arrival -order is random but every key appears exactly once — random placement -without garbage. +### Step 2 — every workload is "pick an integer, then lay it out as a key" + +> **In:** the axes from Step 1, in particular key order. +> **Out:** the single function that decides key order for every `fill*` +> workload, and the duplicate rate it produces — the input to Step 3's four +> names. + +Under every workload name sits the same skeleton: choose an integer, then lay +that integer out as a fixed-width key. All the drama — sequential vs random, +insert vs overwrite — lives entirely in *how the next integer is chosen*. +There are exactly three ways, and they are an enum: + +```cpp +// tools/db_bench_tool.cc — the whole key-order axis, 5869 + 5869 enum WriteMode { RANDOM, SEQUENTIAL, UNIQUE_RANDOM }; +``` + +`KeyGenerator` (**6088-6134**) turns that enum into integers, and its `Next()` +is thirteen lines that contain everything Step 3 will name: + +```cpp +// tools/db_bench_tool.cc — KeyGenerator::Next, 6107-6119 + 6107 uint64_t Next() { + 6108 switch (mode_) { + 6109 case SEQUENTIAL: + 6110 return next_++; + 6111 case RANDOM: + 6112 return rand_->Next() % num_; + 6113 case UNIQUE_RANDOM: + 6114 assert(next_ < num_); + 6115 return values_[next_++]; + 6116 } + 6117 assert(false); + 6118 return std::numeric_limits::max(); + 6119 } +``` + +The line that carries the argument is **6112**: `% num_` is a draw *with +replacement*, so the same key can come up twice. `values_` on 6115 is a +pre-shuffled permutation of `0..num_`, built once in the constructor: + +```cpp +// tools/db_bench_tool.cc — inside the KeyGenerator constructor, 6093-6104 + 6093 if (mode_ == UNIQUE_RANDOM) { + // ... 6094-6097: a comment on the memory cost of materialising the vector ... + 6098 values_.resize(num_); + 6099 for (uint64_t i = 0; i < num_; ++i) { + 6100 values_[i] = i; + 6101 } + 6102 RandomShuffle(values_.begin(), values_.end(), + 6103 static_cast(*seed_base)); + 6104 } +``` + +Line **6098** is the cost of the mode: `UNIQUE_RANDOM` materialises an 8-byte +slot per key before the first write, so `--num=1000000000` wants 8 GB of RAM +just for the permutation. Line 6103 is why the order reproduces: the shuffle +is seeded from `seed_base`. + +**The integer then becomes a key** through `GenerateKeyFromInt` +(**3802-3842**) — and here the tidy story ("zero-padded fixed-width key") is +wrong. The code's own comment at 3797-3801 says what it does: + +```cpp +// tools/db_bench_tool.cc — GenerateKeyFromInt's comment and its tail, 3797-3801 and 3830-3841 + 3797 // - If keys_per_prefix_ is 0, the key is simply a binary representation of + 3798 // random number followed by trailing '0's + 3799 // ---------------------------- + 3800 // | key 00000 | + 3801 // ---------------------------- + // ... 3802-3829: the signature, the --use_existing_keys shortcut, and the + // ... optional prefix block written when keys_per_prefix_ > 0 ... + 3830 int bytes_to_fill = std::min(key_size_ - static_cast(pos - start), 8); + 3831 if (port::kLittleEndian) { + 3832 for (int i = 0; i < bytes_to_fill; ++i) { + 3833 pos[i] = (v >> ((bytes_to_fill - i - 1) << 3)) & 0xFF; + 3834 } + 3835 } else { + 3836 memcpy(pos, static_cast(&v), bytes_to_fill); + 3837 } + 3838 pos += bytes_to_fill; + 3839 if (key_size_ > pos - start) { + 3840 memset(pos, '0', key_size_ - (pos - start)); + 3841 } + 3842 } +``` + +Line **3833** is the one to look at. The shift `(bytes_to_fill - i - 1) << 3` +emits the *most significant* byte first: the integer is written **big-endian +binary**, not decimal digits. The padding on 3840 is ASCII `'0'` (0x30) filling +the remaining `--key_size` bytes (default 16, line 388). Big-endian is the +load-bearing choice: RocksDB's default comparator orders keys byte by byte, so +big-endian layout makes byte order agree with numeric order, and that is +*why* `SEQUENTIAL` integers produce keys that arrive in sorted order. + +**How many duplicates does `RANDOM` produce?** The guide's claim is "~37% of a +full pass", and it is worth deriving rather than asserting, because it is the +whole difference between `fillrandom` and `filluniquerandom`. Line 6112 draws +uniformly from `0..n-1` (up to negligible modulo bias, `n ≪ 2^64`), and +`DoWrite` runs `num_ops = num_` draws per thread (**6160**), each thread with +its own generator over the full key space (**6177-6181**). So, with +`--threads=1`: + +``` +symbols + n = FLAGS_num = the key space size AND the number of draws (6160, 6177) + k = a particular key in 0..n-1 + + P(one draw misses k) = 1 - 1/n each draw is uniform (6112) + P(all n draws miss k) = (1 - 1/n)^n draws are independent + E[keys never written] = n * (1 - 1/n)^n + E[distinct keys written] = n * (1 - (1 - 1/n)^n) + E[duplicate writes] = n - E[distinct] = n * (1 - 1/n)^n + +worked on the guide's own --num=10000000, and on four smaller n to show the limit + + n = 10 (1-1/n)^n = 0.348678 -> 3.49 of 10 keys never written + n = 100 (1-1/n)^n = 0.366032 -> 36.6 of 100 + n = 1000 (1-1/n)^n = 0.367695 -> 368 of 1000 + n = 1000000 (1-1/n)^n = 0.3678793 -> 367,879 of 1,000,000 + n = 10000000 (1-1/n)^n = 0.36787942 -> 3,678,794 of 10,000,000 + + at n = 10,000,000: + expected keys never written = 3,678,794 (36.788%) + expected distinct keys = 6,321,206 (63.212%) + expected duplicate writes = 10,000,000 - 6,321,206 = 3,678,794 (36.788%) + + the limit: (1 - 1/n)^n -> 1/e = 0.36787944... (already matched to 7 digits at n = 1e7) +``` + +Two counts fall out, and they are equal by conservation — every write is +either a key's first appearance or a repeat, so `n − distinct` repeats and +`n × (1−1/n)^n` never-written keys are the same number. So a 10 M-key +`fillrandom` writes **10 M records into 6.32 M distinct keys**, leaving 3.68 M +keys never touched and 3.68 M writes that are overwrites. Each of those +overwrites creates a dead version that compaction must later rewrite and drop +— write amplification (Step 1) with nothing to show for it. + +`UNIQUE_RANDOM` exists precisely to remove that term: 6115 hands out each +integer exactly once, so arrival order is random but the duplicate count is +**zero**. That is the isolation experiment — random *placement* without random +*garbage*. Why it matters: once you see this skeleton, the entire workload menu in -Steps 3–4 is just this function plus an operation type. +Steps 3-4 is this one function plus an operation type, and "random writes" +splits into two genuinely different experiments. ### Step 3 — the fill family: four ways to write -The `fill*` names are write workloads; each stresses a different part of the -write path: - -- **`fillseq`** — sequential-order load (`Sequential` mode). The LSM fast - path: sorted input means files never overlap, so there is no compaction - debt. Papers use it to *build* the database before the real test — it's a - setup step wearing a benchmark's name. -- **`fillrandom`** — random-order inserts. Files overlap, compaction runs - continuously; this is the honest write-throughput number. -- **`overwrite`** — random writes to *existing* keys. Every write creates - garbage (the old version) that compaction must reclaim — maximum - compaction pressure, a different beast from `fillrandom` even though both - are "random writes". -- **`fillsync`** — one fsync (the syscall that forces data to durable media, - ~0.1–5 ms each) per write, run over N/1000 ops. Measures durability cost, - not throughput — expect 3–4 orders of magnitude below `fillseq`. +> **In:** the three key orders from Step 2, and the duplicate arithmetic. +> **Out:** four named write workloads, each pinning one more variable — the +> write-side half of the menu Step 6 composes into a methodology. + +The `fill*` names are write workloads. `Benchmark::Run`'s dispatch chain turns +each name into a method pointer and a couple of flag mutations, and the +mutations are where the meaning lives: + +```cpp +// tools/db_bench_tool.cc — the fill family's arms of the dispatch chain, 4030-4056 + 4030 } else if (name == "fillseq") { + 4031 fresh_db = true; + 4032 method = &Benchmark::WriteSeq; + // ... 4033-4036: fillbatch, the same but with entries_per_batch_ = 1000 ... + 4037 } else if (name == "fillrandom") { + 4038 fresh_db = true; + 4039 method = &Benchmark::WriteRandom; + // ... 4040-4049: filluniquerandom, which forces num_threads to 1 ... + 4050 } else if (name == "overwrite") { + 4051 method = &Benchmark::WriteRandom; + 4052 } else if (name == "fillsync") { + 4053 fresh_db = true; + 4054 num_ /= 1000; + 4055 write_options_.sync = true; + 4056 method = &Benchmark::WriteRandom; +``` + +The two lines that carry the argument are **4050-4051**: `overwrite` and +`fillrandom` call the *same method*, `WriteRandom`. The entire difference is +that `fillrandom` sets `fresh_db = true` on 4038 and `overwrite` does not — +Step 6 shows what `fresh_db` does. So: + +- **`fillseq`** — `SEQUENTIAL` mode (4032 → 5880 → `DoWrite(thread, + SEQUENTIAL)`). The LSM fast path: keys arrive in the comparator's own order + (Step 2's big-endian layout), so successive memtable flushes produce files + with disjoint key ranges and there is nothing for compaction to merge. + Papers use it to *build* the database before the real test — it is a setup + step wearing a benchmark's name. +- **`fillrandom`** — random-order inserts into a fresh DB (4037-4039). Files + overlap, compaction runs continuously, and Step 2's 3.68 M duplicate writes + per 10 M ops add garbage on top. This is the honest write-throughput number. +- **`overwrite`** — the same random writes with **no** fresh DB (4050-4051), + so it runs against whatever the previous entry in the comma list left + behind. Against a DB that already holds the full key space, nearly every + write shadows a live version, which is maximum compaction pressure — a + different beast from `fillrandom` even though both are "random writes", and + the one that matches a steady-state production database. +- **`fillsync`** — random writes with `write_options_.sync = true` (4055) over + `num_ / 1000` ops (4054). The `/1000` is the tell: the author already knew + this workload is far slower and shortened it so the run would finish. + +**How much slower is `fillsync`?** Not a number to guess, and the tidy +folklore answer ("three to four orders of magnitude") is wrong. Two sources +settle it. First, what `sync = true` actually promises — RocksDB's own header +is unusually precise: + +```cpp +// include/rocksdb/options.h — the WriteOptions::sync contract, 2502-2515 + 2502 // If true, the write will be flushed from the operating system + 2503 // buffer cache (by calling WritableFile::Sync()) before the write + 2504 // is considered complete. If this flag is true, writes will be + 2505 // slower. + // ... 2506-2511: what is and is not lost when the process or machine dies ... + 2512 // In other words, a DB write with sync==false has similar + 2513 // crash semantics as the "write()" system call. A DB write + 2514 // with sync==true has similar crash semantics to a "write()" + 2515 // system call followed by "fdatasync()". +``` + +Line **2515** is the one that decides the number: the promise is +`fdatasync`-grade, *not* a drive cache flush. Second, topic 5 measured that +exact ladder ([FINDINGS.md](../../FINDINGS.md) row 5, Apple M3 Pro / APFS): + +``` +per-call p50 and the implied single-threaded commit ceiling (topic 5, notes.md) + + write() only 1.17 µs -> 856,898 commits/s the fillseq/fillrandom rung + fsync 22.67 µs -> 44,109 commits/s 856,898 / 44,109 = 19.4x slower + F_FULLFSYNC 2.97 ms -> 337 commits/s 856,898 / 337 = 2,542x slower + + fillsync's rung, per options.h:2515 = the middle one + so the expected gap is ~19x (1.3 orders of magnitude), not 3-4 orders +``` + +The "3-4 orders" figure belongs to the *bottom* rung — a `Sync()` that really +flushes the drive's volatile cache, which on macOS is `F_FULLFSYNC` and costs +2,542× (3.4 orders). Both rungs get called "fsync" in conversation and they +differ by 131×, so a `fillsync` number is uninterpretable until you know which +one your platform's `WritableFile::Sync()` reached. That is Step 1's point +applied to db_bench itself. Why it matters: a paper quoting "write throughput" without saying which of -these four it ran has told you almost nothing (a 5–50× spread, per Step 1). +these four it ran has told you almost nothing — and the durability axis alone, +measured in this repo, spans 19× to 2,542× depending on a distinction the +benchmark name does not make. ### Step 4 — the read family: point, scan, and interference +> **In:** a database in whatever state the write workloads of Step 3 left it. +> **Out:** the read-side half of the menu, plus the one name whose reported +> number comes from a subset of its threads — which Step 7 revisits. + The read-side names split along two axes — access shape, and whether writes run concurrently: -- **`readrandom` / `readseq` / `readreverse`** — point lookups vs. iterator - scans (forward/backward). Point lookups may probe several LSM levels; - scans stream. -- **`seekrandom`** — the cost of positioning an iterator (Seek touches every - level to set up merge order — a very different profile from a point Get). -- **`multireadrandom`** — MultiGet batching: many keys per call, amortizing - per-call overhead. -- **`readwhilewriting`** — 1 writer + N readers: the "does compaction wreck - my read tail?" test, and the closest thing in the menu to production. The - `*whilemerging`/`*whilescanning` variants isolate other interference - sources. - -Why it matters: read-only numbers (`readrandom` on a freshly-compacted DB) -are the engine's best case; `readwhilewriting` is where LSM read/write -interference — the thing users actually hit — shows up. +- **`readrandom` / `readseq` / `readreverse`** (dispatch at 4078, 4062, 4076) + — point lookups versus iterator scans, forward and backward. A **point + lookup** must consult the memtable and then every level that could hold the + key, so it may touch several files; a **scan** positions once and then + streams. +- **`seekrandom`** (4143) — the cost of positioning an iterator. `Seek` has to + touch every level to build the merging iterator's min-heap before it can + yield the first key (`table/merging_iterator.cc:23-39` states the invariant + every `Seek*()` must restore), so its profile is unlike a point `Get`. +- **`multireadrandom`** (4086) — `MultiGet` batching: `entries_per_batch_` + keys per call, amortising per-call overhead across the batch. +- **`readwhilewriting`** (4158-4160) — the "does compaction wreck my read + tail?" test, and the closest thing in the menu to production. The + `*whilemerging` / `*whilescanning` variants (4161-4166) isolate other + interference sources. + +`readwhilewriting` is worth one more level of detail, because its thread +arithmetic is not what the flag says: + +```cpp +// tools/db_bench_tool.cc — the dispatch arm, 4158-4160, and the method, 8337-8343 + 4158 } else if (name == "readwhilewriting") { + 4159 num_threads++; // Add extra thread for writing + 4160 method = &Benchmark::ReadWhileWriting; + + 8337 void ReadWhileWriting(ThreadState* thread) { + 8338 if (thread->tid > 0) { + 8339 ReadRandom(thread); + 8340 } else { + 8341 BGWriter(thread, kWrite); + 8342 } + 8343 } +``` + +Line **4159** adds one thread beyond `--threads`, and **8338** makes thread 0 +the writer and threads 1..N readers. So `--threads=8 readwhilewriting` runs +nine threads: eight readers and one writer. And the writer removes itself from +the reported figure: + +```cpp +// tools/db_bench_tool.cc — inside BGWriter, 8372-8373 + 8372 // Don't merge stats from this thread with the readers. + 8373 thread->stats.SetExcludeFromMerge(); +``` + +Line **8373** sets the flag that `Stats::Merge` honours at 2484-2486 (Step 7), +so the ops/s `readwhilewriting` prints is a *readers-only* number measured +while a writer ran. That is the right choice — mixing a writer's ops into a +read throughput figure would be meaningless — but it means the line tells you +nothing about what the writer achieved, and the writer's rate is a free +variable unless you also set `--benchmark_write_rate_limit` (1702-1705). + +Why it matters: read-only numbers (`readrandom` on a freshly-compacted DB) are +the engine's best case; `readwhilewriting` is where LSM read/write +interference — the thing users actually hit — shows up, and it is reported +from only part of the run. ### Step 5 — distribution knobs: uniform lies, skew is reality -By default the random modes draw keys **uniformly** (every key equally -likely), but production traffic is **skewed** — a few hot keys dominate, -classically modeled as a **Zipfian distribution** (popularity of the k-th -hottest key falls off as 1/k^s, so a small fraction of keys absorbs most of -the traffic). The difference is not cosmetic: uniform access defeats every -cache (no key is hot enough to stay resident), while skewed access lets -caches shine — the two can disagree on read throughput by an order of -magnitude. - -db_bench's knobs: `read_random_exp_range` (:452) skews reads -exponentially, and **`mixgraph`** (:4133) is the industrial-strength -answer — it models Facebook's *measured* production distributions (the -"Characterizing, Modeling..." FAST'20 paper) with two-term-exponential key -ranges (`keyrange_dist_a..d`, :1708–1717) and Pareto-distributed value -sizes. Same motivation as the capstone's Zipfian `workload` crate: uniform -random keys are the wrong distribution. +> **In:** the read workloads of Step 4, which so far draw keys uniformly. +> **Out:** the two skew models db_bench offers, and the property that separates +> them — a property Step 7's caveats depend on. + +By default the random modes draw keys **uniformly** — every key equally +likely, which is exactly what line 6112's `% num_` gives. Production traffic is +**skewed**: a few hot keys absorb most of the requests. The classic model is a +**Zipfian distribution**, where the popularity of the k-th hottest key falls +off as `1/k^s` for some exponent `s`, so a small fraction of keys takes a large +fraction of the traffic. (The capstone's `workload` crate uses `s = 0.99`, the +YCSB default, for the same reason.) + +The difference is not cosmetic. Uniform access over a key space larger than +memory defeats every cache — no key is hot enough to stay resident — while +skewed access lets the block cache serve most reads. The two can disagree on +read throughput by an order of magnitude on identical hardware. + +db_bench offers two skew knobs, and they differ in a way that matters more +than the shape of the curve. + +**`--read_random_exp_range`** (**452-456**) bends `readrandom`'s draw +exponentially: + +```cpp +// tools/db_bench_tool.cc — GetRandomKey, 7103-7120 + 7103 int64_t GetRandomKey(Random64* rand) { + 7104 uint64_t rand_int = rand->Next(); + 7105 int64_t key_rand; + 7106 if (read_random_exp_range_ == 0) { + 7107 key_rand = rand_int % FLAGS_num; + 7108 } else { + // ... 7109-7115: order = -(uniform in [0,1)) * read_random_exp_range_, then + // ... rand_num = exp(order) * FLAGS_num, which concentrates draws + // ... near 0 — larger flag value, sharper skew ... + 7116 // Map to a different number to avoid locality. + 7117 const uint64_t kBigPrime = 0x5bd1e995; + 7118 // Overflow is like %(2^64). Will have little impact of results. + 7119 key_rand = static_cast((rand_num * kBigPrime) % FLAGS_num); + 7120 } +``` + +The line to focus on is **7119**, and the comment above it on 7116 states the +intent: the hot key *IDs* are deliberately scattered across the key space by +multiplying by a large prime. So this flag gives you hotness **without** +key-space locality — the hot keys are hot, but they live in different SST +blocks. Hold that; it is the exact defect the FAST'20 paper measures. + +**`mixgraph`** (dispatch at **4133**) is the industrial-strength answer. It +models Facebook's *measured* production workloads, from Cao et al., +"Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at +Facebook" (FAST '20). §7.1 of that paper is the indictment: YCSB reproduces +the overall hotness distribution, but "the hot KV-pairs are actually randomly +distributed in the whole key-space", which makes a large number of data blocks +hot and triggers "an extremely large number of block reads" — and the paper +adds explicitly, "db_bench has a similar situation". §7.2 is the fix: partition +the key space into key-ranges sized at the average number of KV-pairs per SST +file, and model the *hotness of the ranges*, so hot keys sit near each other. + +The paper's fitted models (§7.4, on UDB's Assoc workload) map onto db_bench's +flags almost one to one — with one discrepancy the paper wins: + +| db_bench flag (lines) | form in the flag's help text | FAST '20 §7.4's fit | +|---|---|---| +| `keyrange_dist_a..d` (**1708-1719**) | `f(x)=a*exp(b*x)+c*exp(d*x)` — two-term **exponential** | "The average KV-pair access count of key-ranges can be better fit in a two-term **power** model" | +| `key_dist_a`, `key_dist_b` (**1723-1726**) | `f(x)=a*x^b` — simple power | "the distribution of KV-pair access counts follows a power-law that can be fit to the simple power model" ✓ | +| `value_theta/k/sigma` (**1727-1737**) | Generalized Pareto | "Generalized Pareto Distribution best fits the value sizes" ✓ | +| `iter_theta/k/sigma` (**1738-1748**) | Generalized Pareto | "…and Iterator scan length" ✓ | +| `sine_a..d` (**1691-1697**) | `f(x) = A sin(bx + c) + d` | "the QPS variation has a strong diurnal pattern… better fit to the Sine model with a 24-hour period" ✓ | + +The Pareto value sizes are the paper's, confirmed at §7.4 (and §7.2 for +ZippyDB). The key-range model is **not** the paper's two-term power model: +db_bench implements a two-term *exponential*, and the code says so twice — the +help text on 1709-1710 and the call to `gen_exp.InitiateExpDistribution` at +**7944-7946**, guarded by "is any `keyrange_dist_*` non-zero" on 7941-7942. +Both are two-parameter-pair mixtures fitted to the same empirical curve, so +they are close in practice, but if you cite mixgraph as "the paper's model", +this is the term that is yours and not theirs. The `value_k = 0.2615` and +`value_sigma = 25.45` defaults, by contrast, are flagged in the source as +"reasonable defaults based on the mixgraph paper" (1730-1737). + +The property that separates the two knobs: `read_random_exp_range` destroys +locality on purpose (7116-7119); `mixgraph` preserves it by construction +(7963-7965 routes the draw through the key-range distribution first). Same +hotness curve, opposite block-cache behaviour. Why it matters: a benchmark's key distribution silently decides whether the -cache hierarchy participates in the result. +cache hierarchy participates in the result — and *where* the hot keys sit +decides it a second time, independently of how hot they are. ### Step 6 — the comma list is the methodology -db_bench runs its benchmarks *in order against the same database*, so -earlier entries create the state later ones measure: +> **In:** the named workloads of Steps 3-5, as strings. +> **Out:** the state each one runs against — the missing half of every +> published db_bench figure, and the thing Step 7 tells you to ask for. + +`--benchmarks` is split on commas and run in order, against one database that +is opened once: + +```cpp +// tools/db_bench_tool.cc — the top of Benchmark::Run, 3924-3935 + 3924 void Run(ToolHooks& hooks) { + 3925 if (!SanityCheck()) { + 3926 ErrorExit(); + 3927 } + 3928 Open(&open_options_, hooks); + 3929 PrintHeader(open_options_); + 3930 std::stringstream benchmark_stream(FLAGS_benchmarks); + 3931 std::string name; + 3932 std::unique_ptr filter; + 3933 while (std::getline(benchmark_stream, name, ',')) { + 3934 // Sanitize parameters + 3935 num_ = FLAGS_num; +``` + +Line **3928** opens the DB *before* the loop on 3933, so by default every +entry inherits the previous entry's database. But — and this is the correction +that changes how you read a comma list — an entry that set `fresh_db = true` +in Step 3's dispatch destroys it first: + +```cpp +// tools/db_bench_tool.cc — after the dispatch chain, 4295-4317 + 4295 if (fresh_db) { + 4296 DbStateMutationGuard mutation(this); + 4297 if (FLAGS_use_existing_db) { + 4298 fprintf(stdout, "%-12s : skipped (--use_existing_db is true)\n", + 4299 name.c_str()); + 4300 method = nullptr; + 4301 } else { + 4302 if (db_.db != nullptr) { + 4303 db_.DeleteDBs(); + 4304 DestroyDB(FLAGS_db, open_options_); + 4305 } + // ... 4306-4315: the same destroy loop for the --num_multi_db case ... + 4316 Open(&open_options_, hooks); // use open_options for the last accessed + 4317 } +``` + +Line **4304** is the one to look at: `DestroyDB`. Every `fill*` name except +`overwrite` sets `fresh_db` (4031, 4038, 4042, 4053, 4058), so it *wipes* +whatever came before it. So: ``` db_bench --benchmarks=fillseq,readrandom --num=10000000 --value_size=100 --histogram │ │ │ └── measured against the DB fillseq just built - └── builds a compaction-debt-free DB (Step 3) + └── fresh_db = true (4031) → DestroyDB (4304) → build 10M keys in order + +fillseq,fillrandom,readrandom fillrandom wipes fillseq's work (4038) — the + first name contributed nothing at all +fillrandom,overwrite,readrandom overwrite does NOT wipe (4050-4051), so it + shadows fillrandom's 6.32M live keys and + readrandom sees a DB thick with dead versions ``` `fillseq,readrandom` measures reads on a clean, fully-sorted DB; `fillrandom,readrandom` measures reads on a fragmented one — same second -benchmark, very different numbers. That ordering **is** the methodology, and -it's the first thing to check when reproducing a published result. +benchmark, very different numbers. And `--use_existing_db` (4297-4300) turns +the destroy into a *skip*, so the same command line means something different +again. That ordering **is** the methodology, and it is the first thing to +check when reproducing a published result. Why it matters: two papers can both say "readrandom, 10M keys" and still be -measuring different databases. +measuring different databases — and a comma list you have not traced through +4295 may contain a name that did nothing. ### Step 7 — what to distrust in the reported numbers -Knowing how the numbers are produced tells you which claims they can and -cannot support: +> **In:** everything above — the workload, the state it ran against, and the +> threads that ran it. +> **Out:** a list of claims a db_bench figure can and cannot support, and the +> line of code behind each one. + +Knowing how the numbers are produced tells you which claims they can support: ```mermaid flowchart TD - F["--benchmarks=fillseq,readrandom,--histogram
comma list runs IN ORDER against the same DB"] - F --> RUN["Benchmark::Run() (4030)
workload name → method pointer"] - RUN --> RB["RunBenchmark(n, name, method) (4583)
spawn N threads"] - RB --> T1["thread 1
Stats + HistogramImpl (2436)"] + F["--benchmarks=fillseq,readrandom --histogram
split on commas at 3933, run IN ORDER against one DB opened at 3928"] + F --> RUN["Benchmark::Run 3924
dispatch chain 4030-4291: name → method pointer
fresh_db → DestroyDB at 4304"] + RUN --> RB["RunBenchmark 4583
spawn N threads 4608-4634"] + RB --> T1["thread 1
closed loop, e.g. ReadRandom 7147-7229
Stats + HistogramImpl 2436-2452"] RB --> T2["thread 2 ..."] RB --> TN["thread N"] - T1 --> M["merge per-thread histograms (2488)
never average percentiles — Tene's rule"] + T1 --> M["merge_stats.Merge per thread 4649-4652
Stats::Merge 2483-2495 adds histograms at 2491
never averages percentiles — Tene's rule"] T2 --> M TN --> M - M --> OUT["default output: throughput
latency histogram only with --histogram"] -``` - -- **Default output is throughput** (ops/s, MB/s). Per-op latency goes - through `FinishedOps` (:2564) into a plain `HistogramImpl` — reported only - with `--histogram`. A quoted latency without that flag didn't come from - here. -- **It's a closed loop** (each thread issues the next op only after the - previous completes — the same structure as redis-benchmark), so it suffers - coordinated omission (the measurement error where the generator pauses - during server stalls, under-sampling the worst moments): a compaction - stall yields a handful of bad samples instead of the thousands a paced - workload would record. There's a - `--benchmark_write_rate_limit`/read-rate variant for paced writes, but no - coordinated-omission correction — the redis-benchmark critique applies - verbatim. -- **"Latency" here is service time by construction** — db_bench measures - the *embedded* engine, no network, no queueing. Legitimate for engine - work; misleading if quoted as user-facing latency. -- One thing it gets *right*: per-thread histograms are **merged** (:2488), - never averaged — you cannot average percentiles (Tene's rule), and - db_bench doesn't try. + M --> OUT["Stats::Report 2692-2732
always: micros/op mean + ops/sec + MB/s
percentiles only under FLAGS_histogram at 2717"] +``` + +- **Default output is throughput, plus a mean.** `Stats::Report` prints + `micros/op`, `ops/sec`, elapsed seconds and MB/s on 2712-2716 — and the + `micros/op` on 2715 is `seconds_ * 1e6 / done_`, an arithmetic mean. + Percentiles appear only inside `if (FLAGS_histogram)` on **2717-2723**. A + quoted p99 without that flag did not come from here. +- **The per-op clock is an inter-arrival time, not a service time.** + `FinishedOps` (**2564-2584**) computes `micros = now - last_op_finish_` + (2571) and then sets `last_op_finish_ = now` (2583). The interval it records + is "since the previous op finished", so it includes everything the loop did + between the two ops — key generation, the `GetRandomKey` call, the loop + bookkeeping — not just the engine call. +- **It's a closed loop.** Each thread issues the next op only after the + previous one completed — `ReadRandom`'s `while (!duration.Done(1))` at + **7147** runs to `FinishedOps` at **7228** with nothing in between that + waits for a clock. This is the same structure as redis-benchmark + ([reading-redis-benchmark.md](reading-redis-benchmark.md)), so it suffers + **coordinated omission**: the measurement error where the generator, by + waiting for the server, stops issuing requests during a stall and therefore + under-samples exactly the worst moments. A compaction stall yields a handful + of bad samples instead of the thousands a paced workload would record. Topic + 34 measured the size of this on identical work: **closed-loop p99 = 1.0 µs + against open-loop p99 = 90 ms**, a 90,000× understatement + ([FINDINGS.md](../../FINDINGS.md) row 34). +- **The rate limiter does not fix it — it is explicitly excluded from the + measurement.** `--benchmark_write_rate_limit` (1702-1705) does exist, and + `RunBenchmark` builds the limiter at 4590-4598. But look at what `DoWrite` + does immediately after waiting on it: + +```cpp +// tools/db_bench_tool.cc — inside DoWrite, after the rate limiter's Request, 6524-6532 + 6524 if (thread->shared->write_rate_limiter.get() != nullptr) { + 6525 thread->shared->write_rate_limiter->Request( + 6526 batch_bytes, Env::IO_HIGH, nullptr /* stats */, + 6527 RateLimiter::OpType::kWrite); + 6528 // Set time at which last op finished to Now() to hide latency and + 6529 // sleep from rate limiter. Also, do the check once per batch, not + 6530 // once per write. + 6531 thread->stats.ResetLastOpTime(); + 6532 } +``` + + Line **6531** calls `ResetLastOpTime` (2559-2562), whose whole body is + `last_op_finish_ = clock_->NowMicros()`. The comment on 6528-6529 says the + intent out loud: *hide* the pacing wait from the latency. So even the paced + mode measures service time by construction — there is no intended-arrival + timestamp anywhere in `Stats` to subtract from. Coordinated omission here is + not an oversight, it is a documented design choice. +- **"Latency" here is service time by construction** for a second reason: + db_bench measures the *embedded* engine — no network, no queueing, no client + library. Legitimate for engine work; misleading if quoted as user-facing + latency. +- **The histogram is coarse.** `HistogramImpl`'s buckets grow by **1.5×** + (`monitoring/histogram.cc:23-42`, the `bucket_val = 1.5 * bucket_val` on + line 28, rounded to two significant digits on 31-38), and `Percentile` + interpolates *linearly* inside the chosen bucket + (`monitoring/histogram.cc:130-160`, the interpolation on 137-147, clamped to + the observed min and max on 148-155). A p99 that lands mid-bucket is + therefore a linear guess across a range 50% wide. The printed set is fixed at + P50/P75/P99/P99.9/P99.99 (`monitoring/histogram.cc:197-199`) — there is no + p99.999, which is where Step 4's rare compaction stalls would have shown up. +- One thing it gets *right*: per-thread histograms are **merged**, never + averaged. `RunBenchmark` folds each thread's `Stats` in at 4649-4652, and + `Stats::Merge` (2483-2495) adds the *bucket counts* together at 2491. You + cannot average percentiles (Tene's rule), and db_bench does not try. The + same function honours `exclude_from_merge_` at 2484-2486, which is how + Step 4's background writer drops out. Why it matters: db_bench numbers are honest answers to narrow questions; -distrust begins when they're quoted as answers to broad ones. +distrust begins when they are quoted as answers to broad ones. ## Where each step lives in the code — the skim route (30–60 min) -`tools/db_bench_tool.cc` is a ~10,400-line flag-driven monolith — **do not -read it linearly**; hit these anchors: +`tools/db_bench_tool.cc` is a **10,367-line** flag-driven monolith at `7c80a5a` +— **do not read it linearly**; hit these anchors: | Lines | What | Step | |-------|------|------| -| 115–170 | `DEFINE_string(benchmarks, ...)` — the full workload menu; read the help text below it (172+), it's the best documentation | 3, 4 | -| 275–458 | The knobs that define a workload: `num`, `threads`, `value_size`, `histogram`, `read_random_exp_range` (452) | 5, 7 | -| 1708–1717 | `keyrange_dist_a..d` — the mixgraph skew model | 5 | -| 2436 | `class Stats` — per-thread stats, `HistogramImpl` per op type | 7 | -| 2564 | `Stats::FinishedOps` — where each op's micros get recorded | 7 | -| 3802 | `GenerateKeyFromInt` — int → fixed-width key; all key distributions reduce to picking the int | 2 | -| 4030–4140 | `Benchmark::Run()` dispatch: `name == "fillseq"` → method pointer — the map from workload name to implementation | 3, 4, 6 | -| 4583 | `RunBenchmark(n, name, method)` — spawns N threads, merges per-thread `Stats` (histogram merge at 2488, same lesson as Tene: merge histograms, never average percentiles) | 7 | +| 115-170 | `DEFINE_string(benchmarks, …)` — the full workload menu; the help text at 172-273 is the best documentation the tool has | 1, 3, 4 | +| 275-458 | The knobs that define a workload: `num` (275), `threads` (328), `value_size` (337), `key_size` (388), `read_random_exp_range` (452-456), `histogram` (458) | 1, 5, 7 | +| 1691-1697 | `sine_a..d` — the diurnal QPS model, `f(x) = A sin(bx+c)+d` | 5 | +| 1702-1705 | `benchmark_write_rate_limit` — the paced-write flag whose wait is hidden at 6531 | 7 | +| 1708-1719 | `keyrange_dist_a..d` — mixgraph's two-term exponential key-range model | 5 | +| 1723-1748 | `key_dist_a/b` (power), `value_*` and `iter_*` (Generalized Pareto) — the rest of the mixgraph fit | 5 | +| 2436-2452 | `class Stats` — per-thread state; `hist_` is a map of `HistogramImpl` per op type (2450-2452) | 7 | +| 2483-2495 | `Stats::Merge` — histogram buckets added at 2491, `exclude_from_merge_` honoured at 2484-2486 | 4, 7 | +| 2559-2562 | `Stats::ResetLastOpTime` — one line, and the whole coordinated-omission story | 7 | +| 2564-2584 | `Stats::FinishedOps` — `micros = now - last_op_finish_` (2571), recorded only under `FLAGS_histogram` (2569) | 7 | +| 2692-2732 | `Stats::Report` — throughput always (2712-2716), percentiles only at 2717-2723 | 7 | +| 3797-3842 | `GenerateKeyFromInt` — big-endian binary at 3833, `'0'` padding at 3840, not decimal zero-padding | 2 | +| 3924-3935 | `Benchmark::Run` — `Open` once at 3928, comma split at 3933 | 6 | +| 4030-4291 | The dispatch chain: `name == "fillseq"` → method pointer. Fill family 4030-4061, read family 4062-4090, `mixgraph` 4133, `seekrandom` 4143, `readwhilewriting` 4158-4160, unknown-name error 4290-4292 | 3, 4, 5, 6 | +| 4295-4317 | `if (fresh_db)` → `DestroyDB` at 4304 — why some comma-list entries erase the ones before them | 6 | +| 4583-4652 | `RunBenchmark` — rate limiters 4590-4598, thread spawn 4608-4634, per-thread merge 4649-4652 | 7 | | 5869 | `enum WriteMode { RANDOM, SEQUENTIAL, UNIQUE_RANDOM }` | 2 | -| 6088 | `class KeyGenerator` — how UNIQUE_RANDOM permutes the key space | 2 | +| 6088-6134 | `class KeyGenerator` — the shuffle at 6098-6103, `Next()` at 6107-6119 | 2 | +| 6158-6181 | `DoWrite` — one `KeyGenerator` per thread over the whole key space | 2, 3 | +| 6524-6532 | the write rate limiter, and `ResetLastOpTime` hiding its wait | 7 | +| 7103-7120 | `GetRandomKey` — exponential skew, then `kBigPrime` to destroy locality (7116-7119) | 5 | +| 7147-7229 | `ReadRandom`'s loop — a closed loop, `FinishedOps` at 7228 | 7 | +| 7941-7946 | mixgraph's `InitiateExpDistribution` — the two-term exponential, in code | 5 | +| 8337-8343 | `ReadWhileWriting` — tid 0 writes, the rest read | 4 | +| 8372-8373 | `SetExcludeFromMerge` — the writer drops out of the reported number | 4, 7 | + +Suggested route: the menu (115-170) and its help text → `Benchmark::Run` +(3924) → the dispatch chain (4030-4291) for the three or four names you care +about → `fresh_db` (4295-4317) → `KeyGenerator::Next` (6107) → +`GenerateKeyFromInt` (3802) → `RunBenchmark` (4583) → `FinishedOps` (2564) and +`Report` (2692). As you trace it, look for an intended-arrival timestamp +anywhere in `Stats` (2436-2452); its absence is the last bullet of Step 7. + +Three anchors live outside `db_bench_tool.cc` and are worth the detour: +`include/rocksdb/options.h:2502-2515` (what `sync = true` actually promises — +Step 3), `monitoring/histogram.cc:23-42` and `130-160` (how coarse every +printed percentile is — Step 7), and `table/merging_iterator.cc:23-39` (why +`seekrandom` costs what it does — Step 4). + +## Questions to answer in notes.md + +1. `fillrandom` and `overwrite` dispatch to the *same* method, `WriteRandom` + (4039, 4051). Find the one line that differs, then say what + `fillrandom,overwrite,readrandom` measures that `fillrandom,readrandom` + does not. +2. `GenerateKeyFromInt` writes the integer big-endian at 3833. Rewrite line + 3833 as little-endian in your head: which of Step 3's four fill workloads + changes character, and what happens to compaction under `fillseq`? +3. Step 2's arithmetic says a 10 M-op `fillrandom` touches 6.32 M distinct + keys. Run the same derivation for `--num=1000 --writes=10000` (10 draws per + key): what fraction of the key space is still never written, and why does + the answer stop being 1/e? +4. `--benchmark_write_rate_limit` paces writes, and 6531 then hides the pacing + wait from the histogram. If you wanted db_bench to report open-loop latency + instead, which field would you add to `class Stats` (2436-2452), and which + of 2571 and 6531 would have to change? +5. `readwhilewriting` with `--threads=8` runs nine threads and reports eight + of them (4159, 8338, 8373). Design the smallest experiment using only the + flags in 275-458 and 1702-1705 that tells you whether a p99 regression came + from the writer's rate or from compaction. ## Takeaway -db_bench's value is the workload taxonomy, not the harness. When topic 4 (LSM) and M4 -(backend shootout) arrive, name capstone benches in this vocabulary (`fillseq`, -`readrandom`, `readwhilewriting`) so numbers are comparable against published RocksDB -results. +db_bench's value is the workload taxonomy, not the harness. When topic 4 (LSM) +and M4 (backend shootout) arrive, name capstone benches in this vocabulary +(`fillseq`, `readrandom`, `readwhilewriting`) so numbers are comparable against +published RocksDB results — and record the comma list, not just the name, since +Step 6 shows the list is the methodology. ## Done when +Answer each before unfolding it. + - [ ] You can explain why a shared workload vocabulary matters more than any single number db_bench prints. -- [ ] You can name the four members of the fill family and say which one is the adversarial case for a B-tree (and check that against topic 1's measured 63x space amp). -- [ ] You can explain why a uniform key distribution flatters almost every engine, and what changes under Zipf. -- [ ] You can read a db_bench comma list and reconstruct the methodology it encodes. + +
Answer + + Because a single number names a point on a surface without naming the point. + The axes are operation mix, key order, insert-vs-overwrite, durability, and + what background work was running, and this repo has measured two of them + independently: the durability axis alone spans 856,898/s (buffered `write()`) + to 44,109/s (`fsync`) to 337/s (`F_FULLFSYNC`), a 2,542× range + ([FINDINGS.md](../../FINDINGS.md) row 5); the same 108 MB of records lands at + 0.45× space amplification on an LSM and 63.28× on a copy-on-write B-tree + (row 1). A number without the axes is compatible with almost any engine + quality. + + A name fixes the axes. `fillrandom` means `WriteRandom` into a destroyed and + recreated DB (4037-4039 plus the `DestroyDB` at 4304); `overwrite` means the + same method against inherited state (4050-4051); `fillsync` means + `write_options_.sync = true` over `num_/1000` ops (4052-4056). Three names, + one method pointer, three different experiments — and anyone who has read + those twenty lines can reproduce yours. + +
+ +- [ ] You can name the four members of the fill family and say which one is the adversarial case for a B-tree — then check that against topic 1's measured 63.28× space amplification on redb. + +
Answer + + `fillseq` (4030-4032, `SEQUENTIAL`), `fillrandom` (4037-4039, `RANDOM` into a + fresh DB), `overwrite` (4050-4051, `RANDOM` into inherited state), `fillsync` + (4052-4056, `RANDOM` with `sync = true` over `num_/1000` ops). A fifth, + `filluniquerandom` (4040-4049), is the isolation experiment: random order, + zero duplicates, single-threaded by force. + + `fillrandom` is the adversarial case for a B-tree, and topic 1 measured + exactly it: 1.08 M records of 100 B in random key order, batched 1000 at a + time, gave redb (a copy-on-write B-tree) **63.28× space amplification** — + 6,833.9 MB on disk for 108.0 MB of logical data — against fjall's 0.45× + ([FINDINGS.md](../../FINDINGS.md) row 1, + [topic 1 notes](../01-storage-engine-landscape/notes.md)). The mechanism in + those notes is the same one Step 2 derives: random-order inserts touch a new + leaf almost every time, and each batch commit copies every page on the path + to the root without being able to free the old ones yet. Sequential order + (`fillseq`) removes it, which is why using `fillseq` as your headline write + number flatters both engine families and settles nothing. + +
+ +- [ ] You can explain why a uniform key distribution flatters almost every engine, and what changes under skew — including the difference between db_bench's two skew knobs. + +
Answer + + Uniform is the default because line 6112 is `rand_->Next() % num_`, and it is + the *hardest* case for caching: with the key space larger than memory, no key + is requested often enough to stay resident, so the block cache hit rate + collapses toward the ratio of cache size to data size. Real traffic is + skewed — a Zipfian tail where popularity falls as `1/k^s` — so the same + engine on the same hardware serves most reads from cache and posts an + order-of-magnitude better number. Reporting uniform is not conservative; it + measures a workload nobody runs. + + The two knobs differ in *where* the hot keys sit, not how hot they are. + `--read_random_exp_range` (452-456, implemented at 7103-7120) skews the draw + exponentially and then multiplies by `kBigPrime` at 7119, under a comment + that says "Map to a different number to avoid locality" — hotness with the + key-space locality deliberately removed. `mixgraph` (4133) does the opposite: + it models the hotness of *key-ranges* first (7941-7946, 7963-7965), so hot + keys share SST blocks. FAST '20 §7.1 is exactly this measurement — YCSB + reproduces the hotness curve but scatters the hot keys, which triggers "an + extremely large number of block reads", and the paper notes "db_bench has a + similar situation". + +
+ +- [ ] You can read a db_bench comma list and reconstruct the methodology it encodes — including which entries erased the ones before them. + +
Answer + + The list is split on commas at 3933 and run in order against a database + opened once at 3928 — but every entry that set `fresh_db` in the dispatch + chain hits `DestroyDB` at 4304 before it runs. `fillseq` (4031), `fillrandom` + (4038), `filluniquerandom` (4042), `fillsync` (4053) and `fill100K` (4058) + all do; `overwrite` (4050-4051) pointedly does not. + + So `fillseq,readrandom` reads a clean, fully-sorted, compaction-debt-free DB; + `fillrandom,readrandom` reads a fragmented one holding ~6.32 M distinct keys + out of 10 M writes (Step 2's arithmetic); `fillrandom,overwrite,readrandom` + reads a DB thick with dead versions, because `overwrite` inherited + `fillrandom`'s keys and shadowed them. And `fillseq,fillrandom,readrandom` + contains a lie of omission: `fillrandom` destroyed everything `fillseq` + built, so the first name contributed nothing but its own throughput line. One + more flag changes it again — `--use_existing_db` (4297-4300) converts the + destroy into a *skip*, so the fill entry prints "skipped" and the read runs + against whatever was on disk. + +
+ - [ ] You can name three things in a reported db_bench figure you would refuse to take at face value, and what you would ask for instead. +
Answer + + First, **a percentile without `--histogram`**. `Stats::Report` prints ops/s, + MB/s and a mean `micros/op` (2712-2716); the percentile block is inside + `if (FLAGS_histogram)` at 2717-2723, and `FinishedOps` does not even record + the sample otherwise (2569). Ask for the full command line. Then ask which + percentiles: the set is fixed at P50/P75/P99/P99.9/P99.99 + (`monitoring/histogram.cc:197-199`), the buckets grow by 1.5× + (`monitoring/histogram.cc:23-42`), and `Percentile` interpolates linearly + inside them (130-160), so a mid-bucket p99 is a guess across a 50%-wide range. + + Second, **any tail number at all**, because the loop is closed: + `ReadRandom` runs 7147→7228 with nothing waiting on a clock, and even the + paced mode hides the wait — `DoWrite` calls `ResetLastOpTime` right after the + rate limiter (6531), under a comment that says the goal is to "hide latency + and sleep from rate limiter" (6528-6529). Topic 34 measured what that costs + on identical work: p99 of 1.0 µs closed-loop against 90 ms open-loop, a + 90,000× understatement ([FINDINGS.md](../../FINDINGS.md) row 34). Ask for an + open-loop rerun, or treat the figure as service time only. + + Third, **the benchmark name without its comma list and its flags**, per + Step 6: `readrandom` alone does not say whether `DestroyDB` ran at 4304 + before it, whether the keys were uniform or bent by + `--read_random_exp_range` (452), or — for `readwhilewriting` — that the + printed number came from `--threads` readers while a writer that + `SetExcludeFromMerge`'d itself (8373) ran at an unstated rate. Ask for the + whole invocation, not the name. + +
+ +- [ ] You can say what `fillsync` actually measures, and why "three to four orders of magnitude below fillseq" is the wrong number for it. + +
Answer + + `fillsync` is `WriteRandom` with `write_options_.sync = true` (4055) over + `num_ / 1000` ops (4053-4056) into a freshly destroyed DB — so it varies two + things against `fillseq` at once, key order *and* durability, which already + makes the comparison a poor isolation experiment. `fillrandom` is the right + control. + + The size of the durability term is pinned by RocksDB's own header: + `include/rocksdb/options.h:2512-2515` says a write with `sync == true` has + "similar crash semantics to a `write()` system call followed by + `fdatasync()`" — the `fdatasync` rung, not a drive cache flush. Topic 5 + measured that ladder on this hardware: `write()` p50 1.17 µs → 856,898 + commits/s, `fsync` p50 22.67 µs → 44,109/s, `F_FULLFSYNC` p50 2.97 ms → + 337/s ([FINDINGS.md](../../FINDINGS.md) row 5). 856,898 / 44,109 = **19.4×**, + about 1.3 orders of magnitude. The 3–4 orders belongs to the bottom rung: + 856,898 / 337 = 2,542×, or 3.4 orders — and that rung costs 131× more than + the one `sync = true` promises. So quote 19× unless you know the platform's + `WritableFile::Sync()` reached the cache flush, and say which you mean. + +
+ ## References **Papers** -- Cao, Dong, Vemuri, Du — "Characterizing, Modeling, and Benchmarking - RocksDB Key-Value Workloads at Facebook" (FAST 2020) — the measured - production distributions behind `mixgraph`; optional, skim §4-5 +- Cao, Dong, Vemuri, Du — "Characterizing, Modeling, and Benchmarking RocksDB + Key-Value Workloads at Facebook", FAST '20 + ([PDF](https://www.usenix.org/system/files/fast20-cao_zhichao.pdf)) — the + measured production distributions behind `mixgraph`. Read **§7.1** (why YCSB + and db_bench mislead: hot keys scattered across the key space cause "an + extremely large number of block reads"; "db_bench has a similar situation"), + **§7.2** (key-range based modeling, and why the key-range size is the average + number of KV-pairs per SST file), and **§7.4** (the fitted models: two-term + *power* for key-range access counts, simple power within a range, Generalized + Pareto for value sizes and iterator scan lengths, Sine for QPS). Note the + mismatch flagged in Step 5: db_bench implements a two-term *exponential* + where §7.4 reports a two-term *power* fit. **Code** - [rocksdb](https://github.com/facebook/rocksdb) `tools/db_bench_tool.cc` - (~10,400 lines, shallow clone @ `7c80a5a`) — **do not read this - linearly**; it's a flag-driven monolith — follow the skim route table - above (30–60 min) + (**10,367 lines** at `7c80a5a`, the commit in `resources/codebases.md`'s pin + table) — **do not read this linearly**; it is a flag-driven monolith. Follow + the skim route above (30–60 min). + +| File | Lines | What | +|------|-------|------| +| `tools/db_bench_tool.cc` | 115-170 | the workload menu, with its help text at 172-273 | +| `tools/db_bench_tool.cc` | 275-458 | the knobs: `num`, `threads`, `value_size`, `key_size`, `read_random_exp_range`, `histogram` | +| `tools/db_bench_tool.cc` | 1708-1748 | mixgraph's fitted-distribution flags | +| `tools/db_bench_tool.cc` | 2483-2495 | `Stats::Merge` — histograms added, never averaged | +| `tools/db_bench_tool.cc` | 2559-2562 | `ResetLastOpTime` — the pacing wait, hidden | +| `tools/db_bench_tool.cc` | 2564-2584 | `FinishedOps` — the only per-op clock | +| `tools/db_bench_tool.cc` | 2692-2732 | `Stats::Report` — throughput always, percentiles under a flag | +| `tools/db_bench_tool.cc` | 3802-3842 | `GenerateKeyFromInt` — big-endian binary, `'0'`-padded | +| `tools/db_bench_tool.cc` | 4030-4291 | the name → method dispatch chain | +| `tools/db_bench_tool.cc` | 4295-4317 | `fresh_db` → `DestroyDB` | +| `tools/db_bench_tool.cc` | 4583-4652 | `RunBenchmark` — spawn, join, merge | +| `tools/db_bench_tool.cc` | 5869 | `enum WriteMode` | +| `tools/db_bench_tool.cc` | 6088-6134 | `KeyGenerator` — the three key orders | +| `tools/db_bench_tool.cc` | 6524-6532 | the write rate limiter and its hidden wait | +| `tools/db_bench_tool.cc` | 7103-7120 | `GetRandomKey` — exponential skew, locality destroyed | +| `tools/db_bench_tool.cc` | 8337-8343 | `ReadWhileWriting` — tid 0 writes | +| `monitoring/histogram.cc` | 23-42 | 1.5× bucket growth — the resolution of every db_bench percentile | +| `monitoring/histogram.cc` | 130-160 | `Percentile` — linear interpolation inside a 50%-wide bucket | +| `monitoring/histogram.cc` | 197-199 | the fixed printed set: P50, P75, P99, P99.9, P99.99 | +| `include/rocksdb/options.h` | 2502-2515 | `WriteOptions::sync` — "similar crash semantics to a `write()` followed by `fdatasync()`", which is the rung `fillsync` lands on | +| `table/merging_iterator.cc` | 23-39 | the min-heap invariant every `Seek*()` restores — why `seekrandom` is not a point `Get` | + +**Connections** +- [reading-redis-benchmark.md](reading-redis-benchmark.md) — the same + closed-loop defect in a network load generator, with the open-loop fix + sketched. +- [topic 34](../34-debugging/README.md) — coordinated omission measured: + p99 1.0 µs closed-loop against 90 ms open-loop. +- [topic 5](../05-durability-wal/README.md) — the fsync ladder `fillsync` + lands on. +- [topic 1](../01-storage-engine-landscape/README.md) — `fillrandom`'s space + amplification, measured on two engine families. diff --git a/topics/01-storage-engine-landscape/reading-architecture-of-a-dbms.md b/topics/01-storage-engine-landscape/reading-architecture-of-a-dbms.md index e020e1d..30e495b 100644 --- a/topics/01-storage-engine-landscape/reading-architecture-of-a-dbms.md +++ b/topics/01-storage-engine-landscape/reading-architecture-of-a-dbms.md @@ -9,25 +9,56 @@ query, and what breaks without it. Then it routes you through the paper: read the map chapters this week, return per-topic as each box gets built. You are NOT reading all ~120 pages now; budget 2 h. +Every section number below was checked against the PDF of *Foundations and +Trends® in Databases* Vol. 1, No. 2 (2007), pp. 141–259 — the version linked in +the References. **The previous version of this chapter had the section map +wrong**: storage management is §5, not §6, and §6 is transactions. The +corrected routing table is in "How to read the paper" below, and it matters, +because §6 is the section you would land in if you followed the old table +looking for the buffer pool. + ## The problem in one sentence -PostgreSQL is ~1.5 million lines of C, and a single `SELECT name FROM users -WHERE id = 42` passes through five major subsystems on its way to one row — -without a map of those subsystems, every later topic in this curriculum is -a tree with no forest. +The paper's own description of the systems it is surveying is "multi-million +line code bases, most of which are well over a decade old" (§1) — and a single +`SELECT name FROM users WHERE id = 42` passes through five major subsystems of +one of them on its way to one row, so without a map of those subsystems every +later topic in this curriculum is a tree with no forest. ## The concepts, step by step Follow the query. It arrives as bytes on a TCP socket and leaves as a row; -each step is the next box it passes through. +each step is the next box it passes through. The paper follows the same query +in §1.1, using a gate agent at an airport clicking a form to request the +passenger list for a flight — one button click, one single-query transaction, +five boxes. + +Two words the paper defines before it uses them, and so does this chapter. A +**DBMS client** is the library implementing the API an application calls +(JDBC, ODBC, or a driver speaking a proprietary protocol); a **DBMS worker** +is "the thread of execution in the DBMS that does work on behalf of a DBMS +Client" (§2, definitions), and the paper insists on a 1:1 mapping between the +two — one worker handles all SQL requests from one client. Everything in +Step 2 is about what a "worker" is made of. ### Step 1 — the client communications manager: bytes in, rows out +> **In:** nothing yet — an incoming TCP connection carrying an undelimited +> byte stream. +> **Out:** connection state (credentials, current SQL command) plus one framed +> query message, forwarded "deeper into the DBMS" (§1.1, item 1) to Step 2. + The client communications manager is the code that speaks the **wire protocol** — the byte format client and server agree on for shipping queries -in and results out. It accepts connections, authenticates them, frames the -incoming byte stream into discrete query messages, and streams result rows -back: +in and results out. The paper's statement of its job is deliberately narrow +(§1.1, item 1): "to establish and remember the connection state for the caller +(be it a client or a middleware server), to respond to SQL commands from the +caller, and to return both data and control messages (result codes, errors, +etc.) as appropriate." + +**Framing** is the part that word "establish" hides: TCP delivers a stream with +no message boundaries, so the manager must decide where one query ends and the +next begins. ``` client server @@ -37,167 +68,554 @@ back: Concretely: PostgreSQL has its own binary protocol; Redis uses RESP (a text-framed protocol the capstone adopts in topic 7 because it's ~1 page of -spec). The non-obvious job is **result streaming**: a 10-million-row result -must flow out incrementally, and a slow client must back-pressure the -executor instead of forcing the server to buffer everything. - -Without this box there is no way in — and a naive version that buffers whole -results turns one big query into an out-of-memory crash. +spec). The paper adds a tier count you will recognize: client→DBMS directly is +"two-tier", a web server or TP monitor in between makes it "three-tier", and an +application server between those makes four (§1.1, item 1) — which is why "a +typical DBMS needs to be compatible with many different connectivity +protocols". + +The non-obvious job is **result streaming**: a large result must flow out +incrementally rather than being materialized in server memory. §1.1, item 5 +states the mechanism — "for large result sets, the client typically will make +additional calls to fetch more data incrementally from the query, resulting in +multiple iterations through the communications manager, query executor, and +storage manager." §2.1.4 names the shape of that loop: "SQL is typically used +in a 'pull' model: clients consume result tuples from a query cursor by +repeatedly issuing the SQL FETCH request", and most systems work *ahead* of +that stream, using the client communications socket itself as the queue. + +That last detail is where **back-pressure** — a slow consumer forcing the +producer to slow down instead of buffering without bound — comes from for free: +if the enqueue target is the socket, a client that stops reading eventually +fills the socket buffer and stalls the worker. A naive implementation that +buffers the whole result in the server's heap instead turns one big query into +an out-of-memory crash. + +Without this box there is no way in. ### Step 2 — the process manager: who actually runs the query -The process manager decides which OS process or thread executes your query — -it is the mapping between N client connections and M workers. The paper's -taxonomy, still exhaustive today: - -- **process-per-worker** — fork one OS process per connection (classic - PostgreSQL): crash-isolated, but heavy. -- **thread-per-worker** — one thread per connection (MySQL): lighter, shares - one address space. -- **event/async** — a small pool of threads multiplexes thousands of - connections (Redis, most Rust servers): cheapest per connection, hardest - to program. +> **In:** the framed SQL command and connection state from Step 1. +> **Out:** a DBMS worker — an OS process, an OS thread, or a slot in a pool — +> bound to that connection and *admitted* to run, which is the execution +> context Step 3's plan runs inside. + +The process manager decides which unit of OS execution runs your query. The +paper's §1.1 item 2 puts the decision first: "the DBMS must assign a 'thread of +computation' to the command", and "the most important decision that the DBMS +needs to make at this stage in the query regards admission control". + +Three definitions, all §2's, because the taxonomy is meaningless without them. +An **OS process** has a private address space and its own OS resource handles +and security context. An **OS thread** ("k-thread") has neither: it shares the +address space of every other thread in its process, and is scheduled by the +kernel. A **lightweight thread** is scheduled in *user space* by the +application, so switching one costs no kernel mode switch — at the price that +"any blocking operation such as a synchronous I/O by any thread will block all +threads in the process", which is why LWT packages must issue only +asynchronous I/O. A DBMS that ships its own LWT package calls them **DBMS +threads** (§2.2). + +§2.1's taxonomy has exactly three entries, "from the simplest to the most +complex": + +- **process per DBMS worker** — one OS process per connection. Crash-isolated + and debugger-friendly, but the shared structures (buffer pool, lock table) + have to be moved into OS shared memory, "which reduces some of the advantages + of address space separation" (§2.1.1). PostgreSQL "runs the process per DBMS + worker model exclusively on all supported operating systems" (§2.3). +- **thread per DBMS worker** — one multi-threaded process hosts every worker; a + dispatcher thread accepts connections and hands each one a thread (§2.1.2). + MySQL uses this, and DB2 defaults to it where OS threads are good (§2.3). +- **process pool** — "a central process holds all DBMS client connections and, + as each SQL request comes in from a client, the request is given to one of + the processes in the process pool" (§2.1.3). Bounded, often fixed size; a + request arriving when every process is busy waits. + +The third entry is the one an earlier version of this chapter got wrong: it +listed "event/async" as the paper's third model. It is not. The paper's third +model is the **process pool**, and its modern descendant is named in §2.3 as +the pool's thread-based variant: "DBMS workers multiplexed over a thread +pool — Microsoft SQL Server defaults to this model and over 99% of the SQL +Server installations run this way." An event loop over a small thread pool +(Redis, most Rust servers) is that row of the paper's table, not a fourth +family. The paper's own list of exotica is instead about *where the scheduler +lives*: DBMS threads on OS processes (Sybase, Informix) or DBMS threads on OS +threads (SQL Server's "Fibers", §2.3). + +Why does the pool exist at all? §2.1.3 says only that "the memory overhead of +each connection requiring a full process is a clear disadvantage" and §2.1.1 +that "a process has more state than a thread and consequently consumes more +memory" — the paper prints no per-process figure, and neither will this +chapter. But the shape of the argument is arithmetic, so state an assumption +and run it. Assume a per-worker private footprint of 2 MB for a process and +64 KB for a pooled worker's stack, and take §2.3's "tens of thousands of +concurrently connected users" at its low end, 10,000: -The numbers that decide it: 10,000 connections × ~10 MB of per-process -overhead ≈ 100 GB just to hold idle sessions, versus an async pool of 8 -threads. This box also owns **admission control** — deciding that a query -must *wait in a queue* rather than start, so an overloaded server degrades -into higher latency instead of thrashing (all queries slow, none finishing). +``` +process per worker: 10,000 × 2 MB = 20,000 MB = 19.5 GiB of private state +thread/process pool: 200 × 2 MB = 400 MB (200 pooled workers) + + 10,000 × 64 KB = 625 MB (idle connection state) + ─────────── + ~1,025 MB = 1.0 GiB +ratio 20,000 / 1,025 = 19.5× +``` -Without this box, the server accepts every request at once and collapses -under its own concurrency. The choice here directly shapes the capstone -server (M7/M9). +The 2 MB is an assumption, not a measurement; the point that survives any +plausible substitution is that the pool decouples *connections* from +*workers*, so the memory bill grows with the smaller of the two. + +This box also owns **admission control** — refusing to start new work "unless +sufficient DBMS resources are available" (§2.4). Without it a system +**thrashes**: past its peak, throughput "will begin to decrease radically", +usually because the buffer pool cannot hold the working set and the system +"spends all its time replacing pages", sometimes because transactions +"continually deadlock with each other and need to be rolled back and +restarted". With it, §2.4 promises graceful degradation: "transaction latencies +will increase proportionally to the arrival rate, but throughput will remain at +peak." Note the shape of that promise — latency degrades, throughput does not. +Topic 35 measures what happens when it is missing: at 280 QPS against a 300 QPS +capacity, goodput **never recovers** after a 10 s outage ([FINDINGS.md](../../FINDINGS.md) +row 35). + +§2.4 also says admission control is two-tier: a connection-count check in the +dispatcher, and a second controller *inside the query processor* that runs +"after the query is parsed and optimized" and uses the optimizer's estimate of +the query's memory footprint. That second tier is a dependency from Step 2 back +onto Step 3, and it is the reason the boxes are a graph and not a pipeline. + +The choice here directly shapes the capstone server (M7/M9). ### Step 3 — the relational query processor: the database's compiler -The relational query processor turns declarative SQL — you say *what* rows -you want, never *how* to fetch them — into an executable plan. It is a -four-stage compiler pipeline: +> **In:** the SQL text from Step 1, running inside the worker Step 2 admitted. +> **Out:** a query plan — a dataflow graph of operators — executed so that its +> *leaves* issue the record-fetch calls Step 4 answers. + +The relational query processor turns declarative SQL — you say *what* rows you +want, never *how* to fetch them — into an executable plan. §4 splits it into +four stages, one per subsection, and this is the mapping the routing table at +the bottom uses: + +1. **parser** (§4.1) — query text → internal format. Its four tasks, quoted: + "(1) check that the query is correctly specified, (2) resolve names and + references, (3) convert the query into the internal format used by the + optimizer, and (4) verify that the user is authorized to execute the query." + Name resolution means **canonicalization** — expanding `users` into the + four-part name `server.database.schema.table`, which requires the catalog of + Step 5. +2. **rewriter** (§4.2) — expand views, fold constants, simplify. +3. **optimizer** (§4.3) — choose which indexes to use and in what order to + join tables, using statistics about the data. +4. **executor** (§4.4) — run the chosen plan. The paper is specific about how: + "most modern query executors employ the **iterator model** that was used in + the earliest relational systems", where every operator is a subclass of one + four-method interface (§4.4, Fig. 4.2): -1. **parser** — query text → syntax tree; -2. **rewriter** — expand views, fold constants, simplify; -3. **optimizer** — choose which indexes to use and what order to join - tables in, using statistics about the data; -4. **executor** — run the chosen plan, typically as a tree of iterators - each pulling rows from its children. +``` +// ILLUSTRATION — this is the paper's own Fig. 4.2 pseudocode (§4.4, p. 189), +// not code from any engine. Real instances of this interface in this repo: +// topic 11's Volcano executor and turso's `BTreeCursor` (see +// reading-turso-btree.md, core/storage/btree.rs). +class iterator { + iterator &inputs[]; + void init(); + tuple get_next(); + void close(); +} +``` -The stakes are not cosmetic: for `WHERE id = 42`, scanning a 1M-row table -reads ~everything, while descending an index touches 3–4 pages — the -optimizer's choice is routinely a **1000× latency difference** on identical -data. +§4.4.1 draws the consequence the guide's later topics lean on: `get_next()` is +an ordinary procedure call, so "a tuple is returned to a parent in the graph +exactly when control is returned. This implies that only a single DBMS thread +is needed to execute an entire query graph, and queues or rate-matching between +iterators are not needed." Dataflow and control flow are the same edge. Topic 11 +measures what that costs: Volcano tops out at **103 M rows/s** and gets *slower* +as selectivity rises ([FINDINGS.md](../../FINDINGS.md) row 11). -Without this box you'd hand-write the access path for every query — which is -exactly what programming directly against a raw storage engine API is. This -is topics 10–11. +The stakes of stage 3 are not cosmetic, and the arithmetic is worth doing +rather than asserting. Assume a 1,000,000-row table, 100-byte rows, 8 KB pages, +and a B+-tree index on `id` — the same 100-byte record size this topic's own +bench lane uses: -### Step 4 — the transactional storage manager: the box that owns the bytes +``` +rows per page 8192 B / 100 B = 81 rows +pages in a full scan 1,000,000 / 81 = 12,346 pages read +index descent height 3 + 1 leaf = 4 pages read +ratio 12,346 / 4 = 3,087× fewer pages +``` -The transactional storage manager stores the data on disk, caches it in -memory, and guarantees that neither concurrent transactions nor a crash can -corrupt it. It is itself four cooperating sub-managers: +So "the optimizer's choice is worth about 3,000× on this table" — a figure that +falls out of page size, row size and tree height, not out of folklore. It grows +linearly with the table and only logarithmically with the descent, which is why +the gap widens as data grows. Topic 3 measures the caveat: pages touched is not +the same as time, because lookups climb **862 → 1101 ns** from 1e6 to 4e6 keys +while height stays at 3 ([FINDINGS.md](../../FINDINGS.md) row 3). -- **access methods** — the on-disk data structures (B-trees, heaps) that - actually locate rows; -- **buffer pool** — the database's own cache of fixed-size disk pages in - RAM, with its own eviction policy (topic 6); -- **lock manager** — coordinates concurrent transactions (topics 8–9); -- **log manager** — the write-ahead log that makes committed changes - survive a crash (topic 5). +Without this box you would hand-write the access path for every query — which +is exactly what programming directly against a raw storage engine API is. This +is topics 10–11. -This one box is the subject of topics 1–6 and 8–9 — and it is *all* that -fjall and redb are. "Storage engine" names this box, not the database. +### Step 4 — the transactional storage manager: the box that owns the bytes -The paper's §6 adds the fight with the operating system: if the OS also -caches file pages, every hot page sits in RAM **twice** (buffer pool + OS -page cache — "double buffering", half your memory wasted), and the OS may -flush pages to disk in an order that violates the log manager's -write-ahead rule. Hence `O_DIRECT` and databases doing their own IO. +> **In:** the record-fetch and record-modify calls issued by the leaves of +> Step 3's plan. +> **Out:** tuples, read from pages under locks, with log records written for +> anything modified — returned up the iterator stack to Step 3 and out through +> Step 1. + +The transactional storage manager stores the data, caches it in memory, and +guarantees that neither concurrent transactions nor a crash can corrupt it. +§1.1, item 4 lists its parts, and the paper splits them across two whole +sections — **§5 Storage Management** and **§6 Transactions**: + +- **access methods** (§4.5) — the on-disk structures that actually locate + rows: "basic structures like tables and indexes" (§1.1). B-trees are topics + 1 and 3. +- **buffer pool** (§5.3) — "a large shared buffer pool in its own memory + space", "organized as an array of frames, where each frame is a region of + memory the size of a database disk block". Two pieces of per-frame metadata + are worth memorizing now because topic 6 implements both: a **dirty bit**, + set when the page changed since it was read, and a **pin count**, non-zero + meaning "not eligible for participation in the page-replacement algorithm". +- **lock manager** (§6.3) — coordinates concurrent transactions (topics 8–9). +- **log manager** (§6.4) — the write-ahead log. §2.1.4 names its in-memory + half, the **log tail**: an in-memory queue of log entries "periodically + flushed to the log disk(s) in FIFO order", where "a transaction cannot be + reported as successfully committed until a commit log record is flushed to + the log device", and where **group commit** batches several transactions' + commit records into one I/O. Topic 5 measures the price of that flush: + `write()` **857k/s**, `fsync` **44k/s**, `F_FULLFSYNC` **337/s** + ([FINDINGS.md](../../FINDINGS.md) row 5). + +This one box is the subject of topics 1–6 and 8–9 — and it is *all* that fjall +and redb are. "Storage engine" names this box, not the database. It is also +the only box this topic's bench lane measures: the same 108 MB of records +costs fjall **48 MB** on disk and redb **6.8 GB**, space amplification 0.45× +against 63.28× ([FINDINGS.md](../../FINDINGS.md) row 1). Nothing in Steps 1, 2, +3 or 5 moved; the 140× is entirely inside this box. + +§5 is the section that justifies this topic's existence, and it opens with the +two dimensions of control a storage manager is fighting for: + +- **Spatial control** (§5.1) — *where* on the disk a block goes. The reason it + matters is one of the paper's few hard ratios: "sequential bandwidth to and + from disk is between 10 and 100 times faster than random access, and this + ratio is increasing", because density doubles every 18 months and bandwidth + rises as its square root while "disk arm movement... [improves] at about + 7%/year". The maximal answer is **raw-mode access** — bypass the filesystem + and address the block device directly — but §5.1 then measures the + alternative honestly and finds it nearly free: comparing raw access with one + very large file on a mid-sized system, "only a 6% degradation when running + the TPC-C benchmark", and "DB2 reports file system overhead as low as 1% when + using Direct I/O (DIO)". The paper's own conclusion is that vendors "typically + no longer recommend raw storage". +- **Temporal control** (§5.2) — *when* a write actually reaches the disk. + +§5.2 is the section to read twice, because it names **three** distinct problems +with letting the OS buffer your writes, not two: + +1. **Correctness.** "The DBMS cannot guarantee atomic recovery after software + or hardware failure without explicitly controlling the timing and ordering + of disk writes" — the write-ahead logging protocol requires log writes to + precede the corresponding data writes, and OS buffering "can confound the + intention of the DBMS logic by silently postponing or reordering writes". +2. **The prefetch mismatch.** OS read-ahead "depends on the contiguity of + physical byte offsets in files", while the DBMS knows the *logical* future: + the paper's example is scanning B+-tree leaves that are not physically + contiguous, which the query plan can predict and the filesystem cannot. +3. **Double buffering and copy cost.** **Double buffering** is the same page + living in the OS page cache and the DBMS buffer pool at once. §5.2 charges + it twice: "it wastes system memory by effectively reducing the memory + available for doing useful work", and "it wastes time and processing + resources, by causing an additional copying step: on reads, data is first + copied from the disk to the OS buffer, and then copied again to the DBMS + buffer pool. On writes, both of these copies are required in reverse." + +The escape hatches §5.2 names are `mmap`/`msync` and the platform DIO/CIO +interfaces — which is what `O_DIRECT` is on Linux. Topic 6 measures why the +`mmap` half of that answer is a trap: mmap page reads are p50 **42 ns** and max +**182 µs**, a 4300× spread that is entirely minor page faults the database +cannot see or schedule ([FINDINGS.md](../../FINDINGS.md) row 6). §5.2's third +problem is solved; its first is not, because a page fault is still the OS +choosing when to do I/O. ### Step 5 — shared components: the utilities everyone calls -The shared components are the services every other box depends on: the -**catalog** (the database's metadata — tables, columns, indexes, and -statistics, itself stored as ordinary tables), the memory allocator, -replication (topic 22), and admin/monitoring tools. - -The catalog is the load-bearing one: Step 3's 1000× optimizer win is only -possible because the catalog stores row counts and value histograms for the -optimizer to cost plans with. Without it, nothing in the system even knows -what columns a table has. +> **In:** metadata and memory requests arriving from Steps 1, 3 and 4 — the +> parser asking whether a table exists, the optimizer asking how many rows it +> has, every operator asking for scratch memory. +> **Out:** catalog rows, memory contexts, replicated log records and admin +> surfaces — the services with no place in the query's linear path, which is +> why they are drawn beside it rather than in it. + +§7's shared components are the services every other box depends on. Three of +them matter now. + +The **catalog** (§7.1) is the database's metadata — "the names of basic +entities in the system (users, schemas, tables, columns, indexes, etc.) and +their relationships" — and the load-bearing design decision is that it "is +itself stored as a set of tables in the database". The paper's argument for +that is code reuse: "users can employ the same language and tools to +investigate the metadata that they use for other data, and the internal system +code for managing the metadata is largely the same as the code for managing +other tables", and it adds a warning from experience — "this code and language +reuse is an important lesson that is often overlooked in early stage +implementations, typically to the significant regret of developers later on." + +The catalog is not small. §7.1's example: "one major Enterprise Resource +Planning application... has over 60,000 tables, with between 4 and 8 columns +per table, and typically two or three indexes per table." Work that out at the +paper's midpoints — 6 columns and 2.5 indexes per table — and the catalog +alone is 60,000 table rows, 360,000 column rows and 150,000 index rows before +a single user row exists. Which is why §7.1 also says high-traffic parts are +"materialized in main memory... in data structures that 'denormalize' the flat +relational structure of the catalogs into a main-memory network of objects". + +The **memory allocator** (§7.2) is the second, and the paper's point is that +the textbook focus on the buffer pool is misleading: "database systems allocate +significant amounts of memory for other tasks as well" — Selinger-style +optimization builds dynamic-programming state, hash joins and sorts allocate at +runtime. The idiom is a **memory context**: a named region list you allocate +from and free *all at once*, which turns "did every operator free its +temporaries?" into one call. + +Third, **replication services** (§7.4) — topic 15, where follower fsync policy +alone spans **59×** ([FINDINGS.md](../../FINDINGS.md) row 15) — and +**administration, monitoring and utilities** (§7.5). + +The catalog is the one that closes the loop with Step 3: the 3,087× optimizer +win computed above is only possible because something stored the row count and +the fact that an index on `id` exists. Without the catalog, nothing in the +system even knows what columns a table has. ### Step 6 — the assembled map +> **In:** all five boxes, from Steps 1–5. +> **Out:** one diagram, and a reading order for the next thirty topics. + Put the five boxes together and you get the org chart the rest of the -curriculum fills in, box by box — the paper's §1 figure, annotated with -where each box gets built: +curriculum fills in, box by box — the paper's Figure 1.1, annotated with where +each box gets built: ```mermaid flowchart TB - CM["Client communications manager
(topic 7: protocol, RESP)"] --> PC["Process manager
(topic 7/9: threads, admission)"] - PC --> RP["Relational query processor
parse → rewrite → optimize → execute
(topics 10-11)"] - RP --> TS["Transactional storage manager
access methods + buffer + locks + log
(topics 1-6, 8-9)"] - TS --> SC["Shared components
catalog, memory allocator, replication
(topics 15, 22)"] + CM["Client communications manager
§1.1 item 1
(topic 7: protocol, RESP)"] --> PC["Process manager
§2
(topic 7/9: workers, admission)"] + PC --> RP["Relational query processor
§4: parse → rewrite → optimize → execute
(topics 10-11)"] + RP --> TS["Transactional storage manager
§5 storage + §6 transactions
access methods + buffer + locks + log
(topics 1-6, 8-9)"] + TS --> SC["Shared components
§7: catalog, allocator, replication
(topics 15, 22)"] ``` Memorize this diagram; it is the table of contents for topics 3–16. The -punchline for this topic: everything the engine-shootout benchmarks measure -lives inside one box (Step 4) — the capstone builds the other four around -it, milestone by milestone. +punchline for this topic: everything the engine-shootout benchmark measures +lives inside one box (Step 4) — the 140× space-amplification spread of +[FINDINGS.md](../../FINDINGS.md) row 1 is a fact about access methods and +buffering alone, with no query processor, no optimizer and no client protocol +anywhere near it. The capstone builds the other four around it, milestone by +milestone. ## How to read the paper (with the concepts in hand) +**The section numbers below are the corrected ones.** §5 is Storage +Management; §6 is Transactions; §3 is Parallel Architecture, which the old +version of this table did not mention at all. + Read NOW (topic 1): -- **§1 (main components)** — the five-box diagram, i.e. Steps 1–6 in the - authors' own words. Skim fast; you already have the picture — your job is - to attach their vocabulary to it. -- **§2 (process models)** — Step 2 in depth: process- vs thread- vs - event-per-worker, and where admission control lives. Directly informs the - capstone server (M7/M9). -- **§6 (storage management)** — Step 4's fight with the OS: spatial control - (why DBs fight the filesystem), buffer pools vs the OS page cache, the - double-buffering problem. This is the section that justifies this topic's - existence. +- **§1 (introduction, esp. §1.1 the life of a query)** — the five-box + Figure 1.1, i.e. Steps 1–6 in the authors' own words, told through the gate + agent's passenger-list query. Skim fast; you already have the picture — your + job is to attach their vocabulary to it. +- **§2 (process models)** — Step 2 in depth: the definitions block first + (process / OS thread / lightweight thread / DBMS thread / client / worker), + then §2.1's three models, §2.3's who-does-what table, and §2.4 admission + control. +- **§5 (storage management)** — Step 4's fight with the OS: §5.1 spatial + control and the 10–100× sequential/random ratio, §5.2 temporal control and + the three problems with OS buffering, §5.3 the buffer pool's frames, dirty + bits and pin counts. This is the section that justifies this topic's + existence, and it is **not** §6. Skim NOW, return LATER: | Section | Concept | Return at | |---------|---------|-----------| -| §3 parser/rewriter | Step 3, stages 1–2 | topic 10 | -| §4 query processor internals | Step 3, stages 3–4 | topics 10–11 | -| §5 transactions, ACID, locking | Step 4's lock + log managers | topics 8–9 | -| §7 shared components (catalog, replication) | Step 5 | topics 15–16 | +| §3 parallel architecture (shared-memory / shared-nothing / shared-disk / NUMA) | not on the single-node query path at all | topics 36–37 | +| §4.1–4.2 parser, authorization, rewrite | Step 3, stages 1–2 | topic 10 | +| §4.3–4.4 optimizer, executor and the iterator model | Step 3, stages 3–4 | topics 10–11 | +| §4.5 access methods | Step 4's B-trees, from the query processor's side | topics 3, 11 | +| §6 transactions: ACID, serializability, locking, the log manager | Step 4's lock + log managers | topics 5, 8–9 | +| §7 shared components (catalog, allocator, replication) | Step 5 | topics 15–16 | ## Questions to answer in notes.md -1. §6 argues the DBMS should bypass OS caching (O_DIRECT). What are the *two* - distinct problems with letting the OS cache pages? (Double buffering; the OS - evicts/flushes with zero knowledge of WAL ordering.) -2. Which of the five §1 boxes does fjall implement? redb? (Neither has a query - processor or client manager — "storage engine" ≠ "database". The capstone builds - the other boxes on top, milestone by milestone.) -3. 2007 blind spots: name three things the paper couldn't see coming. (Candidates: - NVMe erasing the seek-time mental model, cloud disaggregation — topic 28, columnar - dominance for analytics — topic 12, LSM taking over write paths.) +1. §5.2 argues the DBMS should bypass OS caching. It gives **three** distinct + groups of reasons, not two — name all three, and say which one `O_DIRECT` + fixes and which one `mmap` leaves in place. (Connect the second answer to + topic 6's measured 42 ns / 182 µs mmap spread.) +2. Which of the five §1.1 boxes does fjall implement? redb? Which do they + deliberately not implement, and what does the capstone have to add to turn + one into a database? +3. §5.1 measures raw-device access against one large file at "only a 6% + degradation" on TPC-C, and DB2's DIO overhead "as low as 1%". Given that, + why does §5.1 still spend two pages on spatial control? (Hint: what is the + 6% a measurement *of*, and what does §5.1 say has changed about "raw" + devices since?) +4. §2.4 promises that admission control makes latency degrade proportionally + while *throughput stays at peak*. Topic 35's lane shows goodput at zero for + 121 s after a 10 s outage. Which half of §2.4's promise broke, and what + would have to exist in the loop for it to hold? +5. 2007 blind spots: name three things the paper could not see coming, and for + each say which section would have to be rewritten. (Candidates: NVMe + erasing §5.1's seek-time mental model; cloud disaggregation — topic 28 + measures S3 p50 at 14.17 ms against local NVMe at 0.10 ms; columnar + dominance for analytics — topic 12; LSM taking over write paths — the rest + of this topic.) ## The one-line takeaway -A database is five cooperating managers, and a storage engine is just one of them — -this paper is the org chart for everything the capstone will build. +A database is five cooperating managers, and a storage engine is just one of +them — this paper is the org chart for everything the capstone will build. ## Done when +Answer each before unfolding it. + - [ ] You can draw the five boxes from memory and say which one owns the bytes on disk. + +
Answer + + Client communications manager, process manager, relational query processor, + transactional storage manager, and the shared components drawn beside all + four (§1.1, Figure 1.1). The transactional storage manager owns the bytes: + §1.1 item 4 says it "manages all data access (read) and manipulation (create, + update, delete) calls", and it is the box holding the access methods, the + buffer pool, the lock manager and the log manager. + + The check that you have the boundary right: fjall and redb are *only* that + box. This topic's bench lane changes nothing else in the diagram — same + records, same durability, same client — and still gets 48 MB against 6.8 GB + on disk, 0.45× against 63.28× space amplification + ([FINDINGS.md](../../FINDINGS.md) row 1). A 140× spread with four of the five + boxes absent is the sharpest possible demonstration of where the bytes live. + +
+ - [ ] You can trace one query through all five, naming the four stages inside the query processor. -- [ ] You can state both arguments §6 gives for bypassing OS caching — and connect them to the mmap tail you will measure in topic 6. + +
Answer + + Bytes arrive on a socket; the communications manager frames them into a SQL + command and remembers the connection state (§1.1 item 1). The process manager + assigns a "thread of computation" and decides admission (§1.1 item 2, §2.4). + The query processor runs four stages, one per §4 subsection: parse and + authorize (§4.1), rewrite (§4.2), optimize (§4.3), execute (§4.4). The + executor's leaf operators call into the transactional storage manager, which + takes locks, reads pages through the buffer pool and writes log records (§1.1 + item 4). Then §1.1 item 5's "unwinding the stack": tuples flow back up the + iterator graph into the client communications buffer and out. + + The catalog is touched at three of those stages without appearing in the + path: §4.1 calls it to canonicalize `users` into `server.database.schema.table` + and to type-check expressions, and §4.3 needs its statistics to cost a plan + at all. That is why §7's components are drawn to the side — every box calls + them, no box passes through them. + +
+ +- [ ] You can state all three arguments §5.2 gives against letting the OS buffer writes — and say which section number that discussion is actually in. + +
Answer + + It is **§5.2, Temporal Control: Buffering** — not §6, which is transactions. + Getting this wrong costs you an hour in the wrong chapter, which is why the + routing table above was corrected. + + The three: (1) *correctness* — WAL requires log writes to precede data + writes and commits to return only after the commit record is on the log + device, and OS buffering "can confound the intention of the DBMS logic by + silently postponing or reordering writes"; (2) *the prefetch mismatch* — OS + read-ahead reasons about physical contiguity, while the query plan knows the + logical future, the paper's example being a scan of non-contiguous B+-tree + leaves; (3) *double buffering and copy cost* — the same page in the OS cache + and the buffer pool wastes memory outright and adds a copy in each direction, + and §5.2 insists copies matter because "throughput in a well-tuned + transaction processing DBMS is typically not I/O-bound". + + `O_DIRECT`/DIO fixes (3) and most of (1). `mmap` fixes (3) only: topic 6 + measures mmap page reads at p50 **42 ns** and max **182 µs** + ([FINDINGS.md](../../FINDINGS.md) row 6), and every microsecond of that tail + is the kernel deciding when to do I/O — exactly the control §5.2's first + argument says the DBMS must keep. + +
+ - [ ] You can say which of the five boxes fjall and redb implement, and which they deliberately do not. -- [ ] You wrote answers to both questions in notes.md. + +
Answer + + Both implement Step 4 and nothing else. fjall has access methods (memtable, + SSTs), a buffer/cache layer and a journal — the log manager of §6.4 in the + form of a WAL. redb has access methods (a copy-on-write B-tree), a page + cache and its own durability mechanism. Neither has a client communications + manager (you call them in-process, so there is no wire protocol and no + framing), a process manager (your threads are the workers; there is no + admission control, which is why an unbounded write loop can thrash them), a + relational query processor (no SQL, no plan, no optimizer — you *are* the + access path, which is Step 3's "without this box" clause made literal), and + no catalog beyond a keyspace/table-name registry. + + That is the whole shape of the capstone: M1 defines the storage trait over + this box, and the later milestones add the other four — M7 the protocol + (Step 1), M7/M9 the worker model and admission (Step 2), M10–M11 planning + and execution (Step 3). + +
+ +- [ ] You wrote answers to all five questions in notes.md. + +
Answer + + Nothing to unfold — the questions are the exercise, and they go under + `## Papers → Architecture of a DBMS (2007)` in this topic's `notes.md`. + + The bar for question 1: three groups, named in §5.2's own order, with the + `O_DIRECT`-versus-`mmap` split stated as a claim about *which* problem each + solves rather than a preference. The bar for question 5: a blind spot is only + a blind spot if you can name the section it invalidates. "The paper predates + NVMe" is not an answer; "§5.1's whole spatial-control argument rests on arm + movement improving at 7%/year, and an SSD has no arm" is. + +
## References **Papers** -- Hellerstein, Stonebraker, Hamilton — "Architecture of a Database - System" (Foundations and Trends in Databases, 2007) — - [PDF](https://dsf.berkeley.edu/papers/fntdb07-architecture.pdf) — read - §1–2 + §6 now (2 h); §3–§5 and §7 are reference material to return to - per the table above +- Hellerstein, Stonebraker, Hamilton — "Architecture of a Database System" + (*Foundations and Trends® in Databases*, Vol. 1, No. 2, 2007, pp. 141–259) — + [PDF](https://dsf.berkeley.edu/papers/fntdb07-architecture.pdf) — read §1, + §2 and §5 now (2 h); §3, §4, §6 and §7 are reference material to return to + per the routing table above. + +| Section | What this chapter took from it | +|---|---| +| §1 | "multi-million line code bases, most of which are well over a decade old" | +| §1.1 | Figure 1.1's five components, and the gate agent's query walked through all of them; the communications manager's three jobs; incremental fetch for large result sets | +| §2 (definitions) | OS process / OS thread / lightweight thread / DBMS thread / DBMS client / DBMS worker, and the 1:1 client-to-worker mapping | +| §2.1.1–2.1.3 | the three process models: process per worker, thread per worker, process pool | +| §2.1.4 | shared buffer pool and lock table across process boundaries; the log tail and group commit; SQL's pull model and the socket as result queue | +| §2.3 | PostgreSQL is process-per-worker exclusively; MySQL and DB2 thread-per-worker; SQL Server defaults to a thread pool, "over 99% of installations"; Sybase/Informix DBMS threads on processes; SQL Server Fibers | +| §2.4 | admission control, thrashing, two-tier structure, and the "latency degrades, throughput stays at peak" promise | +| §4.1–4.4 | the parser's four tasks and four-part name canonicalization; the optimizer; the iterator model and Fig. 4.2's four-method interface; §4.4.1 on dataflow coupled to control flow | +| §5.1 | sequential 10–100× random and why the ratio grows (density ×2/18 months, bandwidth ~√density, arm movement 7%/year); raw device vs one large file at 6% on TPC-C; DB2 DIO overhead as low as 1% | +| §5.2 | the three problems with OS buffering: WAL ordering, prefetch mismatch, double buffering plus copy cost; mmap/msync and DIO/CIO as the escape hatches | +| §5.3 | the buffer pool as an array of frames, with a dirty bit and a pin count per frame | +| §6.3–6.4 | lock manager and log manager, as the transactional half of Step 4 | +| §7.1 | the catalog stored as ordinary tables, and the 60,000-table ERP example | +| §7.2 | memory contexts, and why the buffer pool is not the whole memory story | + +**This repo's measurements cited above** +- [FINDINGS.md](../../FINDINGS.md) row 1 (this topic's own 0.45× vs 63.28× + space amplification), row 3 (B-tree height vs cache residency), row 5 (fsync + ladder), row 6 (mmap tail), row 11 (Volcano throughput), row 15 (follower + fsync policy), row 28 (S3 vs local NVMe), row 35 (goodput after an outage). diff --git a/topics/01-storage-engine-landscape/reading-comer-btree.md b/topics/01-storage-engine-landscape/reading-comer-btree.md index 26cd742..462f113 100644 --- a/topics/01-storage-engine-landscape/reading-comer-btree.md +++ b/topics/01-storage-engine-landscape/reading-comer-btree.md @@ -9,129 +9,394 @@ invariants that fix it — one step at a time. Everything in turso's `btree.rs` is a footnote to this paper, and §3's B+ variant is the shape every real engine actually shipped. +Every section reference below was checked against the PDF of *ACM Computing +Surveys* Vol. 11, No. 2 (June 1979), pp. 121–137. **The previous version of +this chapter had the section map wrong** in two places: insertion and deletion +are taught in §1 (under the subheads "Balancing", "Insertion", "Deletion"), +not §2 — §2 is the *cost* analysis — and VSAM is §5, not §4, because §4 is the +multiuser chapter. The corrected reading order is below. Code anchors are +turso at `dd775bc`, the commit this repo's pin table records. + ## The problem in one sentence -Find one record among a million on a 1979 disk, where every disk access -costs ~30 ms: a balanced binary search tree needs ~20 accesses (**600 ms per -lookup**); a B-tree needs 3 (~90 ms then, 3 cached-or-not page reads now) — -and the structure that closes that gap still sits under nearly every -database shipped since. +Find one record among a million on a disk where, in Comer's own framing, "the +time required to access secondary storage is the main component of the total +time required to process the data" (Introduction, *Operations on a File*): a +balanced binary search tree needs ~20 accesses, while Comer's Table I shows a +B-tree of order 50 needs **4 in the worst case** — "later we will see that this +estimate is too high; simple implementation techniques lower the worst case +cost to 3, and the average cost to less" (§2) — and the structure that closes +that gap still sits under nearly every database shipped since. ## The concepts, step by step ### Step 1 — the disk access model: cost = blocks touched -A disk does not hand you bytes; it hands you fixed-size **blocks** (a few -KB), and in 1979 each block fetch costs a mechanical seek plus rotation — -tens of milliseconds — while comparing keys already in memory costs -microseconds. So the only number that matters for a disk-resident structure -is **how many distinct blocks it touches**, not how many comparisons it -does. The RAM model (count comparisons) prices algorithms in the wrong -currency here; the disk model (count block reads) is off by a factor of -~10,000 per operation. - -This is the same observation the turso chapter's Step 1 makes for pages — -"one disk IO" always means "one block/page" — and the same block-transfer -logic as CPU cache lines in topic 0, three orders of magnitude up the -hierarchy. +> **In:** nothing yet — this step fixes the cost currency every later step +> prices things in. +> **Out:** one number, "distinct blocks touched", which Steps 2, 3 and 5 all +> minimize and Step 6 finally drives to ~1. + +A disk does not hand you bytes; it hands you fixed-size **blocks** — a +contiguous run of bytes that the device transfers as one unit, a few KB on +modern hardware. Comer's Introduction states the model and the reason for it in +two sentences: "with current hardware technology, the time required to access +secondary storage is the main component of the total time required to process +the data. Furthermore, most random access devices transfer a fixed amount of +data per read operation, so that the total time required is linearly related to +the number of reads. Therefore, the number of secondary storage accesses serves +as a reasonable cost measure for evaluating index methods." + +Read that carefully, because it is doing two things. It declares the **cost +model** — the thing you count when you compare two algorithms — to be block +accesses rather than comparisons, and it justifies the swap by an *empirical* +claim about the hardware, not a mathematical one. Comer never prints a +millisecond figure anywhere in the paper; a previous version of this chapter +attributed "~30 ms per access, so 600 ms per lookup" to him, and that number is +not his. The honest form of the claim is his: accesses dominate, so count +accesses. + +Comer does list what the model deliberately ignores, and it is worth having the +list: "other less important costs include the time to process data once it has +been placed in main memory, the secondary storage space utilization, and the +ratio of the space required by the index to the space required by the +associated information." Two of those three come back to bite — space +utilization is Step 4, and the index-to-data ratio is why Step 6's B+ shape +won. + +The **RAM model** — count comparisons, assume every memory access costs the +same — prices algorithms in the wrong currency here. This is the same +observation the turso chapter's Step 1 makes for pages ("one disk IO" always +means "one page"), and the same block-transfer logic as CPU cache lines in +topic 0, three orders of magnitude up the hierarchy. ### Step 2 — why binary trees fail on disk -A binary search tree stores one key and two child pointers per node, so -finding one key among n takes ~log₂(n) pointer hops — and on disk, every -hop lands on a different block: +> **In:** the block-counting cost model from Step 1. +> **Out:** two independent failures of the binary tree under that model — +> height and transfer waste — which Step 3 has to fix simultaneously. + +A **binary search tree** stores one key and two child pointers per node, and +the branch taken at a node depends on comparing the query key against the +node's key — Comer's Figure 2 shows exactly this, with the path for the query +"15" darkened. Finding one key among n takes about log₂(n) pointer hops, and +under Step 1's model every hop lands on a different block: ``` -binary tree, 1M keys, nodes scattered on disk: +binary tree, 1,000,000 keys, nodes scattered on disk: - hop 1 → block read (~30 ms) - hop 2 → block read (~30 ms) height = log₂(1,000,000) ≈ 20 - ... ⇒ ~20 block reads ≈ 600 ms/lookup - hop 20 → block read (~30 ms) + hop 1 → block read height ≈ log2(1,000,000) + hop 2 → block read = ln(1e6)/ln(2) + ... = 13.8155 / 0.6931 = 19.93 ⇒ 20 reads + hop 20 → block read - and each read fetches a ~4 KB block to use ~16 bytes of it → 99.6% wasted + and each read fetches a 4096 B block to use 16 bytes of it: + 16 / 4096 = 0.39% used + 4080 / 4096 = 99.6% of the transfer thrown away ``` -Two independent failures: the *height* is 20 (each level is one IO), and -the *transfer* is wasted (one tiny node per big block). Any disk structure -must fix both at once. +Two independent failures, and both are Step 1's currency: the *height* is 20, +and each of those 20 transfers is 99.6% waste. Fixing only the second (pack +several binary nodes per block) still leaves a tree whose height is set by +log₂. Fixing only the first is what Step 3 does — and it fixes the second for +free, which is the elegance. + +Comer builds the fix the same way, as a generalization rather than a +replacement: §1 says it presents the B-tree "as a generalization of the binary +search tree in which more than two paths leave a given node", and Figure 3 +shows the intermediate case — two keys and three branches per node, where "the +query, 15, is less than 42 so the leftmost would be taken at the root." ### Step 3 — the fix: one node = one block, packed with keys -The B-tree's move is to make one tree node exactly one disk block and pack -it with as many sorted keys as fit; the number of children a node can have -is its **fanout**. Now each block read consumes the *entire* transfer, and -the height shrinks from log-base-2 to log-base-fanout: +> **In:** the two failures from Step 2. +> **Out:** the fanout formula and a height in single digits — the budget Step 5's +> mechanics have to preserve and Step 6 finally halves again. + +The B-tree's move is to make one tree node exactly one disk block and pack it +with as many sorted keys as fit. The number of children a node can have is its +**fanout**; Comer's parameter is the **order** *d*, defined in §1 as: "each +node in a B-tree of order d contains at most 2d keys and 2d + 1 pointers… +each must have at least d keys and d + 1 pointers." So order d means fanout +between d + 1 and 2d + 1, and the two ends of that range give two different +heights — a guaranteed one and a typical one. Both are worth computing. + +**The formula, with its symbols named.** Comer derives the height bound in §2 +(*Retrieval Costs*) by counting the minimum number of nodes at each depth — +"the number of nodes at depths 0, 1, 2, … must be at least 2, 2d, 2d², 2d³ …" +— and arrives at + +``` + h ≤ log_d ( (n + 1) / 2 ) + + h the height: the number of nodes visited by a find, i.e. block reads + d the order: the MINIMUM number of keys in a non-root node + n the number of keys in the file +``` + +The `/2` and the `log_d` rather than `log_2d` are the worst case being paid +for: the root may hold as few as one key, and every other node may be only +half full. Run it on Comer's own example, order 50 indexing 10⁶ records: + +``` +h ≤ log_50(500,000.5) = ln(500,000.5) / ln(50) = 13.1224 / 3.9120 = 3.354 ⇒ h = 4 +``` + +which is exactly the `4` in Table I's row for node size 50, column 10⁶. Table I +in full, recomputed from the formula to check the transcription — every cell +below matches the paper: + +| order d | n = 10³ | 10⁴ | 10⁵ | 10⁶ | 10⁷ | +|---|---|---|---|---|---| +| 10 | 3 | 4 | 5 | 6 | 7 | +| 50 | 2 | 3 | 3 | 4 | 4 | +| 100 | 2 | 2 | 3 | 3 | 4 | +| 150 | 2 | 2 | 3 | 3 | 4 | + +**Now derive the fanout from a real page format**, which is the arithmetic this +repo's own topic 3 records and the half Comer leaves to the implementer. +Fanout is not chosen, it is what fits: + +``` + F = floor( (P - H) / (c + s) ) maximum entries in one page + P page size in bytes + H page-header bytes + c bytes per cell (the key plus whatever rides with it) + s bytes per cell pointer in the slot array +``` + +Topic 3's page format (`topics/03-btree-internals/experiments/src/page.rs`, +module docs at lines 1–20) is 4096-byte pages, an 8-byte header, a 2-byte cell +pointer, and an interior cell of `child u32 ∥ key_len u16 ∥ key`. For an 8-byte +key: + +``` +interior cell 4 + 2 + 8 = 14 B +plus its slot 14 + 2 = 16 B per entry +fanout (4096 - 8) / 16 = 4088 / 16 = 255.5 ⇒ F = 255 -- 4 KB block ÷ ~40 bytes per key+child-pointer ≈ **100 keys per node**; -- height = log₁₀₀(1,000,000) = **3** — versus the binary tree's 20; -- at 100-way fanout, 4 levels already index 100⁴ = 100 million keys. +leaf cell 2 + 2 + 8 + 8 = 20 B (key_len, val_len, key, value) +plus its slot 20 + 2 = 22 B per entry +leaf capacity 4088 / 22 = 185.8 ⇒ L = 185 +``` + +`255` and `185` are exactly the numbers in topic 3's recorded fanout table +(`topics/03-btree-internals/notes.md`, *Fanout arithmetic*). Now the height, in +the two-part form a real engine has — leaves first, then interior levels above +them: + +``` +leaves n / L = 1,000,000 / 185 = 5,406 leaf pages +interior log_F(...) = ln(5406)/ln(255) + = 8.5952 / 5.5413 = 1.551 ⇒ 2 interior levels +height 2 + 1 = 3 +``` -Fanout is derived, not chosen: **fanout ≈ block size ÷ entry size**. Bigger -blocks or smaller keys ⇒ flatter tree. The turso chapter's Step 2 draws -this exact tree-of-pages picture (and its Step 3 covers how one page -physically stores variable-length entries — the slotted layout — which -Comer doesn't need and this chapter won't re-explain). +and at n = 10⁹: 10⁹/185 = 5,405,406 leaves, log₂₅₅(5,405,406) = 15.5030/5.5413 += 2.798 ⇒ 3 interior levels ⇒ **height 4**. Both match topic 3's table. Widen +the key to 32 bytes and the same two formulas give F = 4088/40 = 102 and +L = 4088/46 = 88, hence 11,364 leaves and log₁₀₂(11,364) = 9.3383/4.6250 = +2.019 ⇒ 3 interior levels ⇒ **height 4 at a million rows**, one more than the +8-byte key needs. That is the whole cost of a wide key, and it is why suffix +truncation exists. + +Bigger blocks or smaller keys ⇒ flatter tree. Comer's §2 warns that you cannot +simply keep growing the node: "most hardware systems bound the amount of data +that can be transferred with one access", the constant factor grows with the +transfer size, and "each device has some fixed track size which must be +accommodated to avoid wasting large amounts of space", so "optimum node size +depends critically on the characteristics of the system and the devices". + +The turso chapter's Step 2 draws this exact tree-of-pages picture, and its +Step 3 covers how one page physically stores variable-length entries — the +slotted layout — which Comer does not need and this chapter will not +re-explain. ### Step 4 — the invariants: what "B-tree" actually promises -A B-tree of order d enforces three rules at all times: (1) every node -except the root holds between d and 2d keys — **at least half full**; (2) -all leaves sit at the same depth — **perfectly balanced, always**; (3) keys -within a node are sorted, and child subtrees fall strictly between adjacent -keys. +> **In:** the order d and the fanout F from Step 3. +> **Out:** three rules, and the two guarantees they buy — a worst-case height +> and a bounded space overhead — which Step 5's algorithms must maintain on +> every single insert. + +A B-tree of order d enforces three rules at all times: + +1. **Occupancy.** Every node except the root holds between d and 2d keys, so + in Comer's words "each node is at least ½ full" (§1). +2. **Balance.** All leaves sit at the same depth. Comer calls this the point of + the whole structure: "the beauty of B-trees lies in the methods for + inserting and deleting records that always leave the tree balanced" (§1, + *Balancing*). +3. **Order.** Keys within a node are sorted, and each child subtree holds + exactly the keys falling between its two bracketing separators — the + generalization of the binary search tree's left/right split (§1). + +What the rules buy, and it is two distinct things: + +- **The height bound of Step 3 is worst-case, not average-case.** Rule 1 is + what puts the `d` under the logarithm: with a *minimum* of d children per + node, depth i has at least 2dⁱ⁻¹ nodes no matter what order the keys arrived + in. There is no insertion sequence that degrades a B-tree the way sorted + input turns a naive binary search tree into a linked list. +- **Wasted space is capped.** **Storage utilization** — the fraction of the + allocated bytes that hold live entries — is at least 50% by rule 1. The + expected value is better: Comer reports in §3 (*2-3 Trees and Theoretical + Results*) that "extending the analysis to B-trees of higher order, Yao has + shown that the expected storage utilization is ln 2 [≈] 69%" [YAO78]. + +Turn 69% into this topic's currency. **Space amplification** is physical bytes +on disk divided by logical bytes stored, so a structure sitting at ln 2 +occupancy has, from slack alone, -What the rules buy: the height bound of Step 3 is *worst-case*, not -average-case — there is no insertion order that degrades a B-tree the way -sorted input turns a naive binary tree into a linked list. And the -half-full rule caps wasted space: pages are 50–100% full, ~69% (ln 2) on -average in practice — that gap is the B-tree's space overhead, and it's -bounded. +``` + 1 / 0.6931 = 1.443× space amplification from page slack, in steady state +``` + +Hold that number next to what this topic actually measured: redb, a +copy-on-write B-tree fed 108 MB of records in random key order, wrote **6.8 GB** +— space amplification **63.28×** ([FINDINGS.md](../../FINDINGS.md) row 1). The +gap between 1.44× and 63.28× is the measure of how much of a real B-tree's +space cost is *not* the thing Comer analysed. Page slack is bounded and +predictable; copy-on-write page versions retained across 1080 commits are +neither. Comer's 1979 B-tree updates in place, so 1.44× is the whole story +there — and that is precisely the assumption Step 6 of the +[LSM chapter](reading-lsm-paper.md) and this topic's README both attack. ### Step 5 — search, insert, split: the mechanics -Search descends one block per level: read the root, binary-search its keys, -follow the child pointer that brackets your key, repeat until a leaf — -height block reads, exactly the Step 3 budget. Insert descends the same -way, then places the key in a leaf; the interesting case is a full leaf -(2d+1 keys): **split** it into two d-key nodes and push the middle key up -into the parent. The push can overflow the parent too, so splits propagate -upward; splitting the root is the *only* way the tree gets taller — it -grows from the top, which is what keeps all leaves level (invariant 2 for -free). Deletion mirrors it: a node under d keys **borrows** a key from a -sibling or **merges** with one. - -Map to turso: `balance_non_root` (btree.rs:2995) is the "borrow from -siblings first" refinement — Comer explicitly calls out redistribution as -reducing splits, and turso's ≤3-sibling rebalance is that idea implemented. -Cost gradient: one insert usually dirties 1 block, occasionally a split -chain of O(height) blocks. +> **In:** the invariants from Step 4 and the tree shape from Step 3. +> **Out:** three algorithms that all cost O(h) blocks, and the one operation — +> the root split — that is allowed to change h. + +**Search** descends one block per level: read the root, search its keys, +follow the child pointer that brackets your key, repeat until a leaf. That is +h block reads, exactly Step 3's budget. What happens *inside* the node is +Comer's "less important cost" from Step 1, and §3 notes the options anyway: +Clampet suggests binary search rather than linear scan, Knuth's refinement is +that "a binary search might be useful if the node is large, while a sequential +search might be best for small nodes." + +**Insert** descends the same way and places the key in a leaf. The interesting +case is a full leaf, holding its maximum 2d keys: **split** it into two nodes +of d keys each and push the middle key — the **separator**, the key that +divides which subtree a search descends into — up into the parent. The push can +overflow the parent too, so splits propagate upward. Splitting the root is the +*only* way the tree gets taller, which is why it grows from the top, and why +invariant 2 (all leaves level) holds for free: every leaf gains a level at the +same instant. + +**Delete** mirrors it. A node dropping below d keys **borrows** a key from a +sibling (redistribution) or **merges** with one (concatenation). + +The cost, per §2 (*Insertion and Deletion Costs*): an insert or delete "may +require additional secondary storage accesses beyond the cost of a find +operation as it progresses back up the tree. Overall, the costs are at most +doubled, so the height of the tree still dominates". So the gradient is: one +insert usually dirties 1 block, occasionally a split chain of O(h) blocks, and +never more than 2h accesses. + +**Map to turso.** Comer's §3 opens with the refinement turso implements: +"instead of splitting a node as soon as it fills up, keys could merely be +distributed into a neighboring node, splitting only when two neighbors fill." +turso's `balance_non_root` is that idea, and the number of siblings it will +consider is a named constant: + +```rust +// core/storage/btree.rs at turso dd775bc — the constant at 136, and the two +// lines of balance_non_root (2995) that consume it. Elided between them: +// 137-2994, the page/cell format, the cursor and the seek machinery. + 136 pub const MAX_SIBLING_PAGES_TO_BALANCE: usize = 3; +// ... 137-2994: page layout, cursor, seek ... + 2995 fn balance_non_root(&mut self) -> Result> { +// ... 2996-3073: state machine, parent bookkeeping, assertions ... + 3074 let mut pages_to_balance: [Option; MAX_SIBLING_PAGES_TO_BALANCE] = + 3075 [const { None }; MAX_SIBLING_PAGES_TO_BALANCE]; +``` + +The line that carries the argument is **136**: the redistribution window is +three pages wide, fixed at compile time. Line 3074 is where that width becomes +the actual array of pages the balance operates on. A wider window would pack +pages fuller (Step 4's utilization rises toward Step 6's B\*-tree bound) at the +cost of reading more siblings per split — the same tradeoff Comer describes in +prose, with a number attached. ### Step 6 — B-tree vs B+-tree: the variant everyone shipped -In Comer's original B-tree every node stores full records; in the -**B+-tree** variant, interior nodes store only keys (pure routing -information), all records live in the leaves, and the leaves are chained -into a linked list: +> **In:** everything above — a tree whose interior nodes carry records. +> **Out:** the shape every shipped engine uses instead, and the one operation +> (`next`) whose cost it changes by a factor of h. + +In Comer's original B-tree every node stores full records. In the **B+-tree**, +"all keys reside in the leaves. The upper levels, which are organized as a +B-tree, consist only of an index, a roadmap to enable rapid location of the +index and key parts" (§3, *B+-Trees*), and the leaves are "usually linked +together left-to-right". Comer names that chain: "the linked list of leaves is +referred to as the **sequence set**." ``` B-tree: keys+values in ALL nodes B+tree: values ONLY in leaves ┌─────k,v─────┐ ┌──────k──────┐ routing only ┌─k,v─┐ ┌─k,v─┐ ┌──k──┐ ┌──k──┐ - ... [k,v|k,v] ↔ [k,v|k,v] linked leaves + ... [k,v|k,v] ↔ [k,v|k,v] sequence set └── range scan = list walk ``` -Why every real engine chose B+: (a) interior nodes hold only keys → higher -fanout (Step 3's formula with smaller entries) → shorter tree; (b) the -leaf-level linked list → range scans walk sideways without re-descending; -(c) uniform "all data at leaf depth" simplifies everything. +A terminology warning Comer spends a footnote on: "perhaps the most misused +term in B-tree literature is B\*-tree." Knuth's actual **B\*-tree** is a +different thing from the B+-tree — see below — and Comer adopts "B+-tree" for +"Knuth's unnamed implementation" precisely to stop the confusion. So when a +codebase says "B\*", check which one it means. + +Why every real engine chose B+, in the order Comer argues it: + +1. **`next` gets cheap, by a factor of h.** This is the headline, and it is the + problem §2 (*Sequential Processing*) leaves hanging. In a plain B-tree, a + preorder walk "requires space for at least h = log_d(n + 1) nodes in main + memory since it stacks the nodes along a path", and finding the smallest key + means descending from the root to the leftmost leaf (Figure 12). In a + B+-tree, §3 says the structure "retains the logarithmic cost properties for + operations by key, but gains the advantage of requiring **at most 1 access + to satisfy a next operation**. Moreover, during the sequential processing of + a file, no node will be accessed more than once, so space for only 1 node + need be available in main memory." +2. **Higher fanout, shorter tree.** Interior entries carry a separator and a + child pointer instead of a whole record. Run Step 3's formula on topic 3's + format with an 8-byte key and a 100-byte value: the *leaf* capacity falls to + `(2 + 2 + 8 + 100 + 2) = 114 B` per entry, `4088/114 = 35` records per leaf, + while the interior fanout stays at **255** because the value never enters an + interior cell. A B-tree that stored those records in interior nodes too + would have a fanout of 35 there as well — `log₃₅(1e6/35) = 10.259/3.555 = + 2.89 ⇒ 3` interior levels instead of 2, a whole extra IO per lookup on the + same data. Topic 3 records the height as 3 for this shape, which is the + B+ answer. +3. **Uniformity.** All data at leaf depth means deletion never has to hunt for + a record inside an interior node — §3: "the key to be deleted must always + reside in a leaf so its removal is simple", and a stale separator left + behind in the index still routes searches correctly (Comer's Figure 14). + +And the variant that *is* called B\*, since Step 5 already met its mechanism: +Knuth's B\*-tree keeps every node "at least 2/3 full (instead of just 1/2 +full)" by delaying a split until two siblings are full and then "the 2 nodes +are divided into 3, each 2/3 full. This scheme guarantees that storage +utilization is at least 66%" (§3, *B\*-Trees*). Compare it against Step 4's +numbers in this topic's currency: + +``` +B-tree guaranteed 50% ⇒ space amp ≤ 2.000× expected ln 2 = 69% ⇒ 1.443× +B*-tree guaranteed 66% ⇒ space amp ≤ 1.515× +``` + +Comer adds the second-order win: "increasing storage utilization has the side +effect of speeding up the search since the height of the resulting tree is +smaller." The paper's core loop, in the B+ shape §3 argues for — note that the cost of this function is *exactly* its iteration count: ```rust -// height = number of page reads = ceil(log_fanout(n)) — the whole game +// ILLUSTRATION — not quoted from any engine. The real descent this sketches is +// turso's cursor seek in core/storage/btree.rs:2995 (the balance path) and the +// page/cell accessors above it; the height arithmetic in the comment is Step 3's. fn lookup(pager: &Pager, root: PageId, key: u64) -> Option { let mut page = pager.read(root); // each read: 1 potential IO loop { @@ -141,66 +406,234 @@ fn lookup(pager: &Pager, root: PageId, key: u64) -> Option { page = pager.read(page.child(i)); // descend one level } Leaf => return page.find(key), // B+: values ONLY here; - } // leaf link → range scans + } // sequence set → range scans } } -// 4 KB page ≈ 100 keys ⇒ 1 billion rows at height 5, top 3–4 levels cached +// 4 KB page, 8 B key, topic 3's format ⇒ F = 255, L = 185 +// 1e9 rows ⇒ 5,405,406 leaves ⇒ 3 interior levels ⇒ height 4 ``` -And the modern payoff of Steps 3–6 combined: 1 billion rows fit in height -5, and the root plus interior levels are ~1–2% of the data — they stay in -the buffer pool, so a point lookup is typically **one actual disk IO**. +The modern payoff of Steps 3–6 combined is a caching argument, and Comer makes +it himself in §3 (*Virtual B-Trees*): under an LRU policy "the most active +nodes are those close to the root; these tend to stay in memory", and "at +least, the root should remain in main memory since it is accessed for each +search." Size it with the numbers above, at n = 10⁹ and 4 KB pages: + +``` +leaf level 5,405,406 pages × 4096 B = 22.14 GB +interior levels 5,405,406/255 = 21,198 + + 21,198/255 = 84 + + 84/255 = 1 = 21,283 pages × 4096 B = 87.2 MB +interior share 87.2 MB / 22,140 MB = 0.39% of the file +``` + +Under half a percent of the file is routing, so the top three levels fit in any +plausible buffer pool and a point lookup costs **one actual disk IO**. Topic 3 +measures the catch: lookups still climb **862 → 1101 ns** from 1e6 to 4e6 keys +with height pinned at 3 ([FINDINGS.md](../../FINDINGS.md) row 3), because +"resident in the buffer pool" and "resident in CPU cache" are different +questions and Comer's model has no term for the second. ## How to read the paper (with the concepts in hand) -Read in this order: - -1. **§1–2 (the problem + the structure)** — Steps 1–4 in Comer's words: why - balanced trees on disk need high fanout — tree height = number of IOs, - and height = log_fanout(n). A 4 KB page holding ~100 keys ⇒ 1 billion - rows in height 5, of which 3–4 levels cache-resident. This is the whole - game. -2. **§2.1–2.2 (insertion/deletion)** — Step 5's mechanics: split on - overflow, merge/borrow on underflow. Map to turso as you read: - `balance_non_root` (btree.rs:2995) is the "borrow from siblings first" - refinement — Comer calls redistribution out as reducing splits. -3. **§3 (B+-tree, B*-tree variants)** — Step 6; the section that matters - most, because B+ is what every real engine shipped. The B*-tree's - deferred split (redistribute into a sibling before splitting) is - question 2 below. -4. **§4 (applications: VSAM, etc.)** — skim for flavor; 1979's product - landscape. +~15 pages, 2 h. The order below is the corrected one — §4 is concurrency, not +applications. + +1. **Introduction, *Operations on a File*** — Step 1. The four operations + (`insert`, `delete`, `find`, `next`), and the two sentences that declare + block accesses to be the cost measure. `next` is the one to keep in mind; + §3 is where it gets fixed. +2. **§1 The Basic B-Tree** (subheads *Balancing*, *Insertion*, *Deletion*) — + Steps 2, 4 and 5 in Comer's words: the generalization from the binary search + tree (Figures 2 and 3), the order-d definition, and the split/merge + algorithms. Map to turso as you read: `balance_non_root` + (`core/storage/btree.rs:2995`) with `MAX_SIBLING_PAGES_TO_BALANCE = 3` + (`:136`) is §3's "redistribute before splitting" refinement, implemented. +3. **§2 The Cost of Operations** (*Retrieval Costs*, *Insertion and Deletion + Costs*, *Sequential Processing*) — Step 3's height bound and **Table I**, the + single most quotable artifact in the paper. Read *Sequential Processing* + last and notice it ends by deferring `next` to the next section — that + deferral is the reason B+ exists. +4. **§3 B-Tree Variants** — Step 6, and the section that matters most. Read + *B\*-Trees* and *B+-Trees* adjacently so the naming confusion lands, then + *Virtual B-Trees* (the caching argument) and *2-3 Trees and Theoretical + Results* (Yao's ln 2). Skim *Prefix B+-Trees* and *Compression* — they are + topic 3's suffix-truncation exercise. +5. **§4 B-Trees in a Multiuser Environment** — skim now, return at topics 8–9; + this is lock coupling before it had that name. +6. **§5 A General Purpose Access Method Using B+-Trees** — IBM's VSAM. Skim for + flavour; 1979's product landscape. ## Questions to answer in notes.md -1. Why do B-trees guarantee ≥50% page occupancy, and what's the *measured average* - (~69%, ln 2)? Connect to space amplification in the README. -2. B*-tree defers splits by redistributing into siblings. What does turso implement — - B+, B*, or a hybrid? -3. Comer's B-trees assume one page write is atomic. It isn't (torn writes). Which - later machinery patches this hole? (WAL — topic 5; checksums — topic 3.) +1. Comer's height bound is `h ≤ log_d((n+1)/2)` where `d` is the *minimum* + fanout, but Step 3's page arithmetic computes the *maximum*, F. Work both for + topic 3's 8-byte-key format (F = 255, so d = 127) at n = 10⁶ and say how far + apart the guaranteed and typical heights are. Which one does a latency SLO + care about? +2. Why do B-trees guarantee ≥50% page occupancy, and what is the expected + value? (Yao's ln 2 ≈ 69%, §3.) Convert both to space amplification and put + them beside this topic's measured 63.28× for redb — what accounts for the + rest? +3. Comer's §3 describes redistribution-before-split as a way to "delay + splitting and eliminate the associated overhead", and Knuth's B\*-tree as + the 2/3-full version of it. Read turso's `balance_non_root` + (`core/storage/btree.rs:2995`) and decide: is turso B+, B\*, or a hybrid? + Name the line that decides it. +4. Comer's B-trees assume one page write is atomic. It is not — a **torn + write** is a page that hit the disk half-updated after a crash. Which later + machinery patches this hole, and which topic measures its cost? +5. §2's Table I stops at 10⁷ records. Extend it: with topic 3's F = 255 and + L = 185, at what n does the height reach 5, and how big is the file then? + (Step 3 gives you both formulas.) ## The one-line takeaway -The B-tree is the memory hierarchy turned into a data structure: node size = transfer -unit, fanout = whatever fits, height = the IO budget. +The B-tree is the memory hierarchy turned into a data structure: node size = +transfer unit, fanout = whatever fits, height = the IO budget. ## Done when +Answer each before unfolding it. + - [ ] You can state the disk access model in one line — cost is blocks touched, not comparisons made — and use it to explain why a binary search tree is the wrong shape. + +
Answer + + Comer's Introduction: "most random access devices transfer a fixed amount of + data per read operation, so that the total time required is linearly related + to the number of reads. Therefore, the number of secondary storage accesses + serves as a reasonable cost measure." Count block reads, not comparisons. + + The binary search tree fails that model twice over, and the two failures are + independent. Its height is log₂(n) — 19.93, so 20 reads at a million keys — + because each node has two children. And each of those reads pulls a 4096-byte + block to consume roughly 16 bytes of it, throwing away 99.6% of the transfer. + Packing several binary nodes into one block fixes the second and leaves the + first; only making one node *be* one block fixes both, which is Step 3. + +
+ - [ ] You can list the B-tree invariants and say which one forces the >=50% occupancy guarantee. + +
Answer + + Occupancy, balance, order. Occupancy is the one that does the work: §1's + definition is that a node of order d holds "at most 2d keys and 2d + 1 + pointers… each must have at least d keys and d + 1 pointers. As a result, + each node is at least ½ full." A node is sized for 2d keys and never allowed + below d, so the floor is d/2d = 50% by construction. + + That same rule is what makes Step 3's height a *guarantee*: with a minimum of + d children per node, depth i holds at least 2dⁱ⁻¹ nodes regardless of + insertion order, which is the inequality §2 turns into `h ≤ log_d((n+1)/2)`. + Occupancy and worst-case height are the same rule seen from two ends. The + expected occupancy is better than the guarantee — Yao's ln 2 ≈ 69% (§3) — but + the guarantee is what you can quote in a capacity plan: at 50%, space + amplification from slack alone is at most 2.000×, and at 69% it is 1.443×. + +
+ - [ ] You can narrate a split and say where the separator key ends up in a B-tree versus a B+-tree. + +
Answer + + A leaf holding its maximum 2d keys receives one more. It splits into two + nodes of d keys, and the middle key becomes the separator pushed into the + parent. If the parent is also full it splits too, so splits propagate upward; + splitting the root is the only operation that increases the height, and + because it lifts every leaf at once, the "all leaves at the same depth" + invariant is maintained for free. + + The difference is what happens to the middle key itself. In a plain B-tree it + **moves** up: it now lives in the parent and nowhere else, so a search that + matches a key in an interior node stops there. In a B+-tree the algorithm + "promotes a copy of the key, retaining the actual key in the right leaf" (§3), + so the search "does not stop if a key in the index equals the query value. + Instead, the nearest right pointer is followed, and the search proceeds all + the way to a leaf." The consequence Comer draws out is a deletion + simplification: the copy in the index can be left behind as a pure separator + even after the real key is deleted, and searches still land correctly + (Figure 14). + +
+ - [ ] You can compute fanout and height for a given page size and key width, and check yourself against topic 3's measured table (185 leaf cells and fanout 255 for 8 B keys). -- [ ] You wrote answers to both questions in notes.md, including what turso actually implements. + +
Answer + + `F = floor((P - H) / (c + s))` — page size, header, cell bytes, slot-pointer + bytes. Topic 3's format is P = 4096, H = 8, s = 2, interior cell + `child u32 ∥ key_len u16 ∥ key` and leaf cell + `key_len u16 ∥ val_len u16 ∥ key ∥ val`. For an 8-byte key and 8-byte value: + + ``` + interior 4 + 2 + 8 = 14, + 2 slot = 16 ⇒ 4088 / 16 = 255.5 ⇒ F = 255 + leaf 2+2+8+8 = 20, + 2 slot = 22 ⇒ 4088 / 22 = 185.8 ⇒ L = 185 + ``` + + Then height in two parts: `n / L` leaves, and `ceil(log_F(leaves))` interior + levels above them. At n = 10⁶: 5,406 leaves, log₂₅₅(5406) = 1.551 ⇒ 2 interior + levels ⇒ height 3. At n = 10⁹: 5,405,406 leaves, log₂₅₅(…) = 2.798 ⇒ 3 ⇒ + height 4. Both match `topics/03-btree-internals/notes.md`. + + The check that you have understood rather than memorized: widen the key to 32 + bytes and the interior entry becomes 40 B, so F drops to 102 and the height at + 10⁶ rises to 4 — one extra IO on identical data, bought entirely with 24 bytes + of key. Topic 3 records that row too. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including what turso actually implements. + +
Answer + + Nothing to unfold — the questions are the exercise, and they go under + `## Papers → Comer '79` in this topic's `notes.md`. + + The bar for question 3, since it is the one with a checkable answer in code: + turso is a B+-tree with the redistribution refinement, not a B\*-tree. The + deciding evidence is `MAX_SIBLING_PAGES_TO_BALANCE = 3` + (`core/storage/btree.rs:136`, turso `dd775bc`) consumed at + `core/storage/btree.rs:3074` — a fixed three-page redistribution window, which + is §3's "distribute into a neighbouring node" rather than Knuth's 2-into-3 + split with its 66% floor. An answer that says "B\*, because it redistributes" + has confused the mechanism with the guarantee: B\* is defined by the + occupancy bound, and turso never promises one. + +
## References **Papers** -- Comer — "The Ubiquitous B-Tree" (ACM Computing Surveys 1979) — ~15 - pages, 2 h; read §1–3 in order, §3 (the B+/B* variants) matters most, - skim §4 +- Comer — "The Ubiquitous B-Tree" (*ACM Computing Surveys*, Vol. 11, No. 2, + June 1979, pp. 121–137) — ~15 pages, 2 h; read the Introduction, §1, §2 and + §3 in order, §3 (the B+/B\* variants) matters most, skim §4 (multiuser) and + §5 (VSAM). + +| Section | What this chapter took from it | +|---|---| +| Introduction, *Operations on a File* | the four operations; "the number of secondary storage accesses serves as a reasonable cost measure"; the three costs the model ignores | +| §1 | the B-tree as a generalization of the binary search tree (Figures 2–3); order d = "at most 2d keys and 2d + 1 pointers… at least d keys", hence "each node is at least ½ full"; balancing, insertion, deletion | +| §2, *Retrieval Costs* | the node counts 2, 2d, 2d², 2d³ …; the bound `h ≤ log_d((n+1)/2)`; **Table I**, and "a B-tree of order 50 which indexes a file of one million records can be searched with only 4 disk accesses in the worst case… simple implementation techniques lower the worst case cost to 3" | +| §2, *Insertion and Deletion Costs* | costs "at most doubled" over a find; the practical limits on node size (transfer bound, constant factor, track size) | +| §2, *Sequential Processing* | a plain B-tree's `next` may cost log_d n accesses and needs h nodes stacked in memory | +| §3, opening | redistribute into a neighbour before splitting — the refinement turso implements | +| §3, *B\*-Trees* | Knuth's definition: ≥2/3 full, split 2 nodes into 3, "storage utilization is at least 66%"; and the warning that the term is "perhaps the most misused" in the literature | +| §3, *B+-Trees* | all keys in the leaves, upper levels a "roadmap"; the **sequence set**; a promoted *copy* of the separator; "at most 1 access to satisfy a next operation" and space for only 1 node | +| §3, *Virtual B-Trees* | LRU keeps the nodes closest to the root resident; "at least, the root should remain in main memory" | +| §3, *2-3 Trees and Theoretical Results* | Yao's result: "the expected storage utilization is ln 2 [≈] 69%" | **Code** -- [turso](https://github.com/tursodatabase/turso) - `core/storage/btree.rs` — the living counterpart; walked in - [reading-turso-btree.md](reading-turso-btree.md) +- [turso](https://github.com/tursodatabase/turso) at `dd775bc` — + `core/storage/btree.rs:136` (`MAX_SIBLING_PAGES_TO_BALANCE = 3`) and + `core/storage/btree.rs:2995` (`balance_non_root`), the living counterpart; + walked in [reading-turso-btree.md](reading-turso-btree.md). + +**This repo's measurements cited above** +- `topics/03-btree-internals/notes.md` — the fanout table (255/185 for 8 B keys, + 102/88 for 32 B keys, 255/35 for a 100 B value) that Step 3's formula + reproduces, and `topics/03-btree-internals/experiments/src/page.rs:1-20` for + the cell format it is derived from. +- [FINDINGS.md](../../FINDINGS.md) row 1 (redb's 63.28× space amplification) and + row 3 (862 → 1101 ns at constant height). diff --git a/topics/01-storage-engine-landscape/reading-fjall.md b/topics/01-storage-engine-landscape/reading-fjall.md index 3597ac0..1971ef6 100644 --- a/topics/01-storage-engine-landscape/reading-fjall.md +++ b/topics/01-storage-engine-landscape/reading-fjall.md @@ -3,13 +3,22 @@ The LSM protagonist of this topic — a codebase small enough that insert-to-SST is traceable in an afternoon, and layered well enough to steal from. fjall is the *keyspace/journal/scheduling* layer; the actual tree (memtable, SSTs, -blooms, block index) lives in the external `lsm-tree` crate (Cargo.toml:29). -Before touching the code, this chapter builds the LSM machine step by step — -why writes are buffered, what a memtable and journal are, what a flush -produces, how a read finds anything, and why compaction and tombstones exist. -Then it hands you the file and line anchors to watch each step happen. -Reading fjall shows you the LSM *lifecycle*; topic 4 descends into `lsm-tree` -itself. +blooms, block index) lives in the external `lsm-tree` crate, pinned at +`~3.1.6` in `Cargo.toml:29`. Before touching the code, this chapter builds the +LSM machine step by step — why writes are buffered, what a memtable and journal +are, what a flush produces, how a read finds anything, and why compaction and +tombstones exist. Then it hands you file and line anchors to watch each step +happen. Reading fjall shows you the LSM *lifecycle*; topic 4 descends into +`lsm-tree` itself. + +**All line numbers below are from `fjall-rs/fjall@80cf6bc`** (crate version +3.1.6), the commit in this repo's pin table — check with +`python3 tools/pinned-source.py ref fjall`, and read any file at that commit +with `python3 tools/pinned-source.py show fjall -r A:B`. One API caveat: +this topic's own experiment (`experiments/Cargo.toml:7`) pins **fjall 2.x** +(2.11.2 in the lockfile), where the type now called `Keyspace` was called +`Partition` and the type now called `Database` was called `Keyspace`. The +concepts are identical; the names moved in 3.0. ## The problem in one sentence @@ -21,104 +30,166 @@ write on crash, and while still answering point reads in a handful of IOs. ### Step 1 — why buffer writes in memory: sequential beats random -Storage devices reward sequential access and punish random access. An NVMe -SSD streams sequential writes at ~2–5 GB/s, but random 4 KB writes top out -around 50–500K IOPS — and each small random write also forces the device to -rewrite a whole internal flash block (write amplification inside the drive). -An update-in-place engine (like the B-tree in the turso chapter) turns every -insert with a random key into a random page write. The LSM (log-structured -merge) idea inverts this: **never update in place**. Accumulate incoming -writes in RAM — where random access is free — until you have a few MB, then -write them to disk in one big sequential burst. +> **In:** a stream of inserts whose keys arrive in random order, and a block +> device with wildly asymmetric sequential and random throughput. +> **Out:** the reason every LSM starts with a RAM buffer, and the name of the +> cost that buys. -``` +Storage devices reward sequential access and punish random access. An +**update-in-place** engine — one that modifies a record where it already lives, +like the B-tree in the turso chapter — turns every insert with a random key into +a random page write, because the target page is wherever the key sorts. + +The LSM (log-structured merge) idea inverts this: **never update in place**. +Accumulate incoming writes in RAM — where random access is nearly free — +until you have tens of megabytes, then write them to disk in one big sequential +burst. + +```text update-in-place: insert(k₉₃₁), insert(k₀₂), insert(k₅₅₀) ... → 3 random 4 KB page writes, scattered across the file - log-structured: insert(k₉₃₁), insert(k₀₂), insert(k₅₅₀) ... × ~100K - → buffered in RAM, sorted, then ONE 8-64 MB sequential write + log-structured: insert(k₉₃₁), insert(k₀₂), insert(k₅₅₀) ... × ~600K + → buffered in RAM, sorted, then ONE 64 MiB sequential write ``` -What it costs: the data on disk is now *many files written at different -times* instead of one tree — reads and space reclamation get harder. Steps -4–6 are the price being paid. +That 64 MiB is not a guess: fjall's default `max_memtable_size` is +`64 * 1_024 * 1_024` at `src/keyspace/options.rs:91`, and at this topic's +100-byte records that is 64 MiB / 100 B ≈ **671,000 records per flush**. + +What it costs: the data on disk is now *many files written at different times* +instead of one tree — reads and space reclamation get harder. Steps 4–6 are the +price being paid, and [FINDINGS.md](../../FINDINGS.md) row 1 is what the trade +is worth on this topic's workload: on the same 108.0 MB of records, fjall's LSM +occupies **48.4 MB** (space amp 0.45×) against redb's copy-on-write B-tree at +**6,833.9 MB** (63.28×), a **140× spread**. ### Step 2 — the memtable and the journal: RAM for speed, a log for safety +> **In:** the decision to buffer writes in RAM. +> **Out:** the two structures that decision forces (a sorted in-memory map and +> an append-only log), the ordering rule between them, and fjall's actual +> `insert` from line 905. + The in-RAM buffer is the **memtable** — an in-memory *sorted* map (fjall's `lsm-tree` uses a skip list) that absorbs every write and can be range-scanned -in key order. Sorted matters: when it's time to write to disk, the data must -come out in key order (Step 3), and reads must be able to search it. +in key order. Sorted matters twice: the data must come out in key order when it +is written to disk (Step 3), and reads must be able to search it. -RAM alone is a durability hole: crash before the buffer hits disk and the -writes are gone. The fix is the **journal** (also called a WAL, write-ahead -log): an append-only file on disk. The rule that gives it its name — every -write is appended to the journal *before* it enters the memtable. Appending -is sequential (the fast case from Step 1), so durability costs one sequential -append, not a random write. After a crash, replaying the journal rebuilds -the memtable. +RAM alone is a durability hole: crash before the buffer hits disk and the writes +are gone. The fix is the **journal** (also called a WAL, write-ahead log): an +append-only file on disk. The rule that gives it its name — every write is +appended to the journal *before* it enters the memtable — is what makes replay +correct. Appending is sequential (the fast case from Step 1), so durability +costs one sequential append, not a random write. -fjall's `Keyspace::insert()` — `src/keyspace/mod.rs:905` — *is* this step, -de-sugared to ten lines: +Here is `Keyspace::insert` verbatim, with the doc comment and error paths +elided: ```rust -fn insert(&self, key: &[u8], value: &[u8]) -> Result<()> { - let journal = self.journal.lock(); // journal lock BEFORE memtable — - journal.write_raw(key, value)?; // replay order must equal apply order - journal.persist(self.durability)?; // fsync per policy, not per write - let bytes = self.tree.insert(key, value); // memtable: sorted, in RAM - self.write_buffer.fetch_add(bytes); // atomic accounting → backpressure - if self.memtable_over_size_limit() { - self.rotate_memtable(); // seal it + enqueue flush task — - } // event-driven, no polling - Ok(()) -} +// src/keyspace/mod.rs at fjall-rs/fjall@80cf6bc — Keyspace::insert, lines 905-950, +// with the doc comment and two guard clauses (912-914, 921-924) elided. +905 pub fn insert, V: Into>( +906 &self, +907 key: K, +908 value: V, +909 ) -> crate::Result<()> { +919 let mut journal_writer = self.supervisor.journal.get_writer(); +926 let seqno = self.supervisor.seqno.next(); +928 journal_writer.write_raw(self.id, &key, &value, lsm_tree::ValueType::Value, seqno)?; +930 if !self.config.manual_journal_persist { +931 journal_writer +932 .persist(crate::PersistMode::Buffer) +937 ?; +938 } +940 let (item_size, memtable_size) = self.tree.insert(key, value, seqno); +942 self.supervisor.snapshot_tracker.publish(seqno); +944 drop(journal_writer); +946 self.supervisor.write_buffer_size.allocate(item_size); +947 self.maintenance(memtable_size); +949 Ok(()) +950 } ``` -Two costs to notice: every write is written *twice* (journal + eventually an -SST — the first factor of **write amplification**, the ratio of bytes written -to disk per byte of user data), and the fsync policy on line 3 decides -whether durability is per-write or batched — the single biggest write-latency -knob in any LSM. +Five things this function tells you that prose would not: + +1. **Line 919 takes the journal lock before anything else, and line 944 drops it + only after the memtable insert.** That interval is what guarantees replay + order equals apply order. The comment at line 921 explains a second reason — + the poison flag must be checked *after* acquiring the mutex, otherwise + TOCTOU. +2. **Line 926: a sequence number is allocated per write.** Seqnos are the spine + of LSM correctness — they are how newest-wins is decided in Step 4 and how + GC knows what is safe to drop in Step 5. Line 942 publishes it to the + snapshot tracker. +3. **Line 932's default is `PersistMode::Buffer`, not an fsync.** The enum is at + `src/journal/writer.rs:35`: `Buffer` (:41) hands bytes to the OS page cache + only; `SyncData` (:46) calls `sync_data()` (:226); `SyncAll` (:49) calls + `sync_all()` (:220). So out of the box fjall survives a *process* crash but + not a *power* loss — the single biggest write-latency knob in any LSM, and + the one you must match across engines before comparing them. +4. **Line 946 is accounting, not backpressure.** `write_buffer_size.allocate` + just bumps a counter; the actual throttling is in `maintenance` (line 947). +5. **Every write is written twice** — once to the journal, later to a segment. + That is the first factor of **write amplification**: bytes physically written + to disk per byte of user data. ### Step 3 — flush: the memtable becomes an immutable sorted file -When the memtable reaches its size limit (typically 8–64 MB), it is -**rotated**: marked immutable ("sealed"), swapped for a fresh empty memtable, -and handed to a background thread that writes it out as an **SSTable** -(sorted string table; fjall calls them **segments**) — an *immutable* file of -key-value pairs in sorted order, plus two small helpers: +> **In:** a memtable that has hit 64 MiB. +> **Out:** what a segment file contains, why immutability is the point, and +> which fjall defaults set the layout. -``` +When the memtable exceeds `max_memtable_size`, it is **rotated**: marked +immutable ("sealed"), swapped for a fresh empty memtable, and handed to a +background thread that writes it out as an **SSTable** (sorted string table; +fjall calls them **segments** or *tables*) — an *immutable* file of key-value +pairs in sorted order, plus two small helpers: + +```text one segment (SSTable) on disk: ┌───────────────────────────────┬──────────────┬──────────────┐ - │ data blocks (~4 KB each, │ block index │ bloom filter │ - │ sorted key-value pairs) │ first key → │ ~10 bits/key │ - │ [a..f][g..m][n..s][t..z] ... │ block offset │ ≈1% false pos│ + │ data blocks (4 KiB each, │ block index │ bloom filter │ + │ sorted key-value pairs) │ first key → │ 10 bits/key │ + │ [a..f][g..m][n..s][t..z] ... │ block offset │ ⇒ 0.82% FPR │ └───────────────────────────────┴──────────────┴──────────────┘ - 64 MB data ~tens of KB ~80 KB per 64K keys + 64 MiB data ~tens of KB ~840 KB / 671K keys ``` -- The **block index** maps "first key of each ~4 KB block → file offset", so - finding a key inside a segment costs one binary search in RAM plus **one** - disk read. -- The **bloom filter** is a probabilistic set-membership structure (a bit - array written to by k hash functions): "definitely not here" or "maybe - here". At ~10 bits per key it answers "is key X in this file?" with ~1% - false positives — for the cost of a few hashes, no IO. - -Because the segment is immutable, it never needs locking and can be written -as one sequential stream. The journal entries covering the flushed memtable -can now be dropped. Cost: the same key may now exist in several segments -(old versions in older files) — nothing has been overwritten, only shadowed. +Every number in that box is a fjall default you can check: + +- **4 KiB data blocks** — `data_block_size_policy: BlockSizePolicy::all(/* 4 KiB + */ 4 * 1_024)` at `src/keyspace/options.rs:95`. +- The **block index** maps "first key of each block → file offset", so finding a + key inside a segment costs one binary search in RAM plus **one** disk read. + fjall pins the index blocks of the top two levels in memory + (`index_block_pinning_policy: PinningPolicy::new([true, true, false])`, + `options.rs:100`) and partitions them from level 3 down + (`options.rs:103`). +- The **bloom filter** is a probabilistic set-membership structure — a bit array + written by *k* hash functions — that answers "definitely not here" or "maybe + here", never a false negative. At the standard optimal *k*, the false-positive + rate for *m* bits per key is `0.6185^m`, so fjall's default of 10 bits/key + gives `0.6185^10` = **0.82%**, and 671,000 keys × 10 bits = 839 KB of filter + per segment. The policy is at `src/keyspace/options.rs:108–111` and Step 4 + explains why it is an *array*. + +Because the segment is immutable it never needs locking and can be written as +one sequential stream; the journal entries covering the flushed memtable can +then be dropped. Cost: the same key may now exist in several segments — nothing +has been overwritten, only *shadowed* by a higher seqno. ### Step 4 — the read path: newest wins, blooms skip the rest -A key can live in the active memtable, a sealed-but-not-yet-flushed memtable, -or any segment. Since newer always shadows older, a read checks locations -**newest-first** and returns the first hit: +> **In:** a key that may live in the active memtable, a sealed memtable, or any +> segment on any level. +> **Out:** the newest-first search order, the arithmetic of what blooms save, +> and the one fjall default that is Monkey shipped as a product. -``` +Since newer always shadows older, a read checks locations **newest-first** and +returns the first hit: + +```text get(k): active memtable → sealed memtables (newest first) → segments, newest first: bloom says "no"? → skip, zero IO (the common case) @@ -126,127 +197,365 @@ or any segment. Since newer always shadows older, a read checks locations ``` The number of places a single read might have to check is **read -amplification**. Bloom filters are what keep it tolerable: with 20 segments -and 1% false positives, a lookup for an absent key does ~0.2 disk reads -instead of 20. In fjall, `Keyspace::get()` — `src/keyspace/mod.rs:623` — is -two lines delegating to `tree.get(key, SeqNo::MAX)`; the whole -newest-first/bloom dance lives inside `lsm-tree`. But the bloom *policy* is -configured in fjall: `src/keyspace/config/filter.rs:8–43` (`BitsPerKey` vs -`FalsePositiveRate`, per-level policies — Monkey's idea productized; topic 4). +amplification**. Blooms are what keep it tolerable. Concretely, with 20 segments +and fjall's default 10 bits/key (0.82% FPR), a lookup for an *absent* key does +20 × 0.0082 = **0.16 expected disk reads** instead of 20 — a 122× reduction, for +the cost of 20 in-memory hash probes. + +In fjall the search itself is two lines: + +```rust +// src/keyspace/mod.rs at fjall-rs/fjall@80cf6bc — Keyspace::get, lines 623-625. +// SeqNo::MAX means "read the newest version of everything" — a snapshot read +// would pass its own seqno here instead (topic 8's MVCC preview). +623 pub fn get>(&self, key: K) -> crate::Result> { +624 Ok(self.tree.get(key, SeqNo::MAX)?) +625 } +``` + +The whole newest-first/bloom dance lives inside `lsm-tree`. What fjall keeps is +the *policy*, and its default is the interesting part: + +```rust +// src/keyspace/options.rs at fjall-rs/fjall@80cf6bc — KeyspaceCreateOptions::default, +// lines 108-111 and 116. Both policies are ARRAYS indexed by LSM level. +108 filter_policy: FilterPolicy::new([ +109 FilterPolicyEntry::Bloom(BloomConstructionPolicy::FalsePositiveRate(0.0001)), +110 FilterPolicyEntry::Bloom(BloomConstructionPolicy::BitsPerKey(10.0)), +111 ]), +116 data_block_compression_policy: CompressionPolicy::new([CompressionType::None, CompressionType::None, CompressionType::Lz4]), +``` + +Read line 108–111 carefully: **L0 gets a 0.01% false-positive rate; every deeper +level gets 10 bits/key.** Invert the bloom sizing formula +`m/n = −ln(p)/(ln 2)²` and the L0 budget is `−ln(0.0001)/0.4805` = **19.2 +bits/key**, nearly double the deeper levels. That is Monkey's thesis — +non-uniform filter budgets, spend bits where probes are most frequent — shipped +as a library default. Topic 4 derives why. + +Line 116 is the other half of this topic's headline. `CompressionPolicy::new([None, +None, Lz4])` means L0 and L1 are stored raw and **L2 and below are LZ4**, and +the `lz4` feature is on by default (`Cargo.toml:20`, `default = ["lz4"]`). That +is the mechanism behind fjall's measured **0.45× space amp**: 108.0 MB of +records ending up as 48.4 MB on disk is not the LSM defeating information +theory, it is LZ4 on compressible generated values, plus densely packed sorted +runs. Say that, not "LSMs are space-efficient". ### Step 5 — compaction: merging files to bound read cost -Left alone, flushes pile up segments forever: read amplification grows -without bound and shadowed old versions waste disk (**space -amplification** — disk bytes used vs live data bytes). **Compaction** is the -background fix: pick several segments, merge-sort them (they're each sorted, -so this is a streaming k-way merge), keep only the newest version of each -key, and write one new segment; delete the inputs. +> **In:** a directory that accumulates one segment per flush, forever. +> **Out:** what compaction does, fjall's default geometry, where backpressure +> lives when compaction cannot keep up, and the write-amp bill. + +Left alone, flushes pile up segments forever: read amplification grows without +bound and shadowed old versions waste disk (**space amplification** — bytes on +the device per byte of live data). **Compaction** is the background fix: pick +several segments, merge-sort them (each is already sorted, so this is a +streaming k-way merge), keep only the newest version of each key, write one new +segment, delete the inputs. + +fjall's default is **leveled** — `compaction_strategy: Arc::new(Leveled::default())` +at `src/keyspace/options.rs:123–125`. Segments are organised into levels L0, L1, +L2…; each level is roughly an order of magnitude bigger than the previous and, +below L0, levels hold non-overlapping key ranges, so a read checks at most one +segment *per level*. The strategies fjall re-exports are at +`src/compaction/mod.rs:7`: `Fifo`, `Leveled`, `Levelled` (the last is an +alias — British and American spellings both work). + +The write-amplification bill is the LSM's defining trade, and the LSM paper's +Theorem 3.1 prices it exactly: every key is rewritten once per level it +descends, so **write amp = K·(r+1)** for K disk levels at size ratio r — 4 × 11 += 44× for the usual four-levels-at-ten geometry. See +[reading-lsm-paper.md](reading-lsm-paper.md) Step 5 for the derivation. Topic 4 +is entirely about tuning `K` and `r`. + +What happens when ingest outruns compaction is worth reading, because it is +where "LSM absorbs writes fast" stops being true: -fjall's default policy is **leveled**: segments are organized into levels -L0, L1, L2… where each level is ~10x bigger than the previous and, below L0, -levels contain non-overlapping key ranges — so a read checks at most one -segment *per level*. A 100 GB dataset fits in ~4 levels: read amplification -is bounded at ~4 segment probes, most eliminated by blooms. +```rust +// src/keyspace/mod.rs at fjall-rs/fjall@80cf6bc — the backpressure path, +// lines 789-816. Called from maintenance() (line 839) on every single insert. +789 fn check_write_halt(&self) { +790 while self.tree.l0_run_count() >= 30 { +791 std::thread::sleep(Duration::from_millis(10)); +792 } +793 } +795 pub(crate) fn local_backpressure(&self) -> bool { +796 let mut throttled = false; +798 let l0_run_count = self.tree.l0_run_count(); +800 if l0_run_count >= 20 { +801 perform_write_stall(l0_run_count); +802 self.check_write_halt(); +803 throttled = true; +804 } +806 while self.tree.sealed_memtable_count() >= 4 { +811 std::thread::sleep(Duration::from_millis(100)); +812 throttled = true; +813 } +815 throttled +816 } +``` -The cost is the LSM's defining trade: every key is rewritten once per level -it descends through — leveled compaction commonly costs **10–30x write -amplification**. LSMs buy cheap ingest and bounded reads by re-paying write -bandwidth in the background. Topic 4 is entirely about tuning this trade. +Three thresholds, all hard-coded: **stall at 20 L0 runs**, **halt at 30**, and +**halt while 4+ memtables are sealed and waiting to flush**, sleeping 100 ms a +turn. This is the "write stall" every LSM has, and it is why an LSM's latency +distribution has a long tail even when its mean looks excellent. ### Step 6 — tombstones: a delete is just another write +> **In:** immutable segments, and a user who calls `remove(k)`. +> **Out:** why a delete costs a *write*, when the bytes actually come back, and +> the scan pathology that follows. + Immutable files mean you cannot erase a key in place — an older segment may -still hold it. So `delete(k)` *writes* a **tombstone** (a marker record -meaning "k is deleted"), which travels the same path as any write: journal → -memtable → flush → segment. Reads treat a tombstone as "found: not present" -and stop — newest-first ordering makes it shadow every older version. - -The actual bytes are reclaimed only when compaction merges the tombstone -past every older version of the key; only at the bottom level can the -tombstone itself be dropped. Costs: deleted data occupies disk until -compaction catches up, and a range full of tombstones makes scans *slower* -(they must be read and skipped) — the classic "deleting data made my -database slower" LSM surprise. +still hold it, and you would have to rewrite that file to remove it. So +`remove(k)` *writes* a **tombstone**: a marker record meaning "k is deleted", +carried by the same path as any write — journal → memtable → flush → segment, +with its own seqno. fjall writes it as `lsm_tree::ValueType::Tombstone`, the +same call shape as the `ValueType::Value` on line 928 of Step 2. Reads treat a +tombstone as "found: not present" and stop, because newest-first ordering makes +it shadow every older version. + +The actual bytes are reclaimed only when compaction merges the tombstone past +every older version of the key, and only at the bottom level can the tombstone +itself be dropped — until then dropping it could resurrect an older version +sitting below. Deciding when that is safe is what `snapshot_tracker` exists for: +compaction passes `snapshot_tracker.get_seqno_safe_to_gc()` into +`tree.compact(...)` at `src/compaction/worker.rs:34–37`, so no version an open +reader might still see is ever dropped. That exact problem returns as MVCC +vacuuming in topic 8. + +Two costs follow. Deleted data occupies disk until compaction catches up — which +is a *space amplification* charge, i.e. the axis this topic measures. And a +range full of tombstones makes **scans slower**, because each tombstone must be +read and skipped: the classic "deleting data made my database slower" LSM +surprise. ## Where each step lives in the code -``` +```text src/ - ├─ lib.rs module map — start here - ├─ keyspace/mod.rs insert/get/memtable rotation — the heart (steps 2-4) - ├─ journal/writer.rs WAL writes (step 2) - ├─ flush/worker.rs sealed memtable → SST (step 3) - ├─ compaction/worker.rs compaction runs (step 5) - ├─ supervisor.rs background orchestration - ├─ worker_pool.rs flume-channel thread pool - └─ poison_dart.rs panic guard + ├─ lib.rs module map — start here + ├─ keyspace/mod.rs (1113 lines) insert/get/rotation/backpressure (steps 2-5) + ├─ keyspace/options.rs every default in this guide (steps 3-5) + ├─ journal/writer.rs WAL writes + PersistMode (step 2) + ├─ flush/worker.rs (42 lines) sealed memtable → SST (step 3) + ├─ compaction/worker.rs compaction runs (step 5) + ├─ worker_pool.rs flume-channel thread pool + ├─ ingestion.rs bulk load, and a seqno-race comment worth reading + └─ poison_dart.rs (34 lines) panic guard ``` -**Steps 2–3 — the write path.** Start at `Keyspace::insert()` — -`src/keyspace/mod.rs:905`. Read the whole function; it *is* the LSM -write-path diagram from the README: +**Steps 2–3 — the write path.** Start at `Keyspace::insert` — +`src/keyspace/mod.rs:905`. Read the whole function; it *is* the LSM write-path +diagram from the README: ```mermaid flowchart LR I["insert()
mod.rs:905"] --> J["journal write_raw
mod.rs:928"] - J --> P["journal persist/fsync
mod.rs:932"] + J --> P["journal persist
PersistMode::Buffer
mod.rs:932"] P --> M["tree.insert → memtable
mod.rs:940"] - M --> C["check_memtable_rotate
mod.rs:831"] - C -- over limit --> R["request_rotation
mod.rs:818"] - R --> S["inner_rotate_memtable
mod.rs:727
seal + enqueue flush"] + M --> A["maintenance()
mod.rs:947 → 837"] + A --> C["check_memtable_rotate
mod.rs:831"] + A --> B["local_backpressure
mod.rs:795"] + C -- "size > 64 MiB" --> R["request_rotation
mod.rs:818
sends RotateMemtable"] + R --> W["worker_pool.rs:141
receives it"] + W --> S["inner_rotate_memtable
mod.rs:727
seal + enqueue flush"] S --> F["flush::run
flush/worker.rs:12
memtable → SST"] ``` -**Step 4 — the read path.** `Keyspace::get()` — `src/keyspace/mod.rs:623` -(two-line delegation to `lsm-tree`); bloom configuration at -`src/keyspace/config/filter.rs:8–43`. +Note the hop through the worker pool: `request_rotation` +(`mod.rs:818–829`) only *sends* a `WorkerMessage::RotateMemtable`; a pool thread +receives it at `worker_pool.rs:141–145`, re-takes the journal lock, and calls +`inner_rotate_memtable`. The writing thread never blocks on the seal. + +**Step 4 — the read path.** `Keyspace::get` — `src/keyspace/mod.rs:623–625`, a +two-line delegation to `lsm-tree`. Bloom policy at +`src/keyspace/options.rs:108–111`; its wire encoding at +`src/keyspace/config/filter.rs:8–44` (the `BitsPerKey`/`FalsePositiveRate` +variants are serialised there, but *defined* in `crate::config`). **Step 5 — compaction scheduling.** -- Strategies re-exported at `src/compaction/mod.rs:7`: `Leveled`, `Fifo`. -- Worker: `compaction/worker.rs:10` — thin: `tree.compact(strategy, gc_watermark)`. -- Trigger plumbing: `worker_pool.rs:141–145` sends `WorkerMessage::Compact`. +- Strategies re-exported at `src/compaction/mod.rs:7`: `Fifo`, `Leveled`, + `Levelled`. Default chosen at `src/keyspace/options.rs:123–125`. +- Worker: `src/compaction/worker.rs:10` — thin, 60 lines; the real call is + `tree.compact(strategy, snapshot_tracker.get_seqno_safe_to_gc())` at lines + 34–37. +- Backpressure: `src/keyspace/mod.rs:789–816`. -The interesting part is what fjall *doesn't* do: no compaction geometry here — it -delegates policy to `lsm-tree`, keeping fjall pure lifecycle/scheduling. Good -layering to steal for the capstone's storage crate. +The interesting part is what fjall *doesn't* do: no compaction geometry lives +here — it delegates policy to `lsm-tree`, keeping fjall pure +lifecycle/scheduling. Good layering to steal for the capstone's storage crate. **Aha spots** (worth a detour each): -1. **`poison_dart.rs:27–33`** — a `Drop` guard that poisons the whole keyspace if a - background worker panics. Crash-*visibly* instead of serving from corrupt state. -2. **`ingestion.rs:37–51`** — comment explains holding the journal lock across - `finish()` to prevent seqno inversion between writes and bulk ingest. Sequence - numbers are the spine of LSM correctness (MVCC preview, topic 8). -3. **`snapshot_tracker.rs`** — open-snapshot seqno watermark gates GC: compaction - can't drop a version some reader might still see. This exact problem returns in - MVCC vacuuming (topic 8). -4. **`keyspace/mod.rs:746–750`** — rotation immediately enqueues the flush task; no - polling anywhere. Event-driven background work via channels. +1. **`poison_dart.rs:27–33`** — a `Drop` guard whose whole body is + `if std::thread::panicking() { self.poison(); }`. If a background worker + panics, the database is poisoned and every subsequent `insert` returns + `Error::Poisoned` (checked at `mod.rs:922`). Crash *visibly* instead of + serving from corrupt state. The entire file is 34 lines. +2. **`ingestion.rs:36–52`** — an ASCII interleaving diagram in a comment, + explaining why `finish()` holds the journal lock: without it, a concurrent + writer that already took seqno 1 could insert *after* the ingest registered + seqno 2, inverting the ordering that newest-wins depends on. +3. **`worker_pool.rs:155`** — `if journal_writer.pos()? > 64_000_000` rotates + the journal file. A second 64 MB threshold, independent of the memtable's. +4. **`snapshot_tracker`** — the open-snapshot seqno watermark gates GC; + `mod.rs:757–758` pulls the watermark up on every rotation so a database with + no open snapshots does not stall GC forever + (the comment cites fjall discussion #85). ## Questions to answer while reading -- The journal lock is taken *before* the memtable insert. What ordering bug would - reordering them create? (Hint: replay after crash — Step 2's "replay order must - equal apply order".) -- `mod.rs:946` — write buffer accounting is an atomic counter. Where does backpressure - actually happen when writers outrun flushing? -- What durability do you get *per insert* by default — fsync every write, or batched? - Compare with what you'll set in the experiment (durability parity!). +1. The journal lock is taken at `mod.rs:919`, before the memtable insert at + `:940`, and released at `:944`. Construct the concrete corruption that + swapping lines 928 and 940 would allow, in terms of what journal replay + would reconstruct after a crash. +2. `mod.rs:946` bumps an atomic counter, but backpressure is at `:795`. Read + `local_backpressure` and say which of its three thresholds a writer hits + first on this topic's workload (1.08 M records, batches of 1,000), and + whether it stalls or halts. +3. `mod.rs:930–938` calls `persist(PersistMode::Buffer)` unless + `manual_journal_persist` is set. What durability does that actually give you + — against a process crash, and against power loss? Check what this topic's + `experiments/src/lib.rs:57` and `:69` do, and why parity with redb's + `Durability::None` was necessary before the shootout meant anything. +4. `options.rs:108–111` gives L0 a 0.0001 FPR and deeper levels 10 bits/key. + Convert both to bits per key (`m/n = −ln p / (ln 2)²`), then explain why + spending *more* bits on the *smallest* level is the right way round. +5. `options.rs:116` sets `[None, None, Lz4]`. Predict what the measured space + amp would be if the whole array were `Lz4`, and what it would be if the + `lz4` feature were off (`Cargo.toml:20`) — then say which of those two + predictions you could test without changing fjall. ## Done when -You can narrate insert-to-SST without looking, and you know which decisions live in -fjall vs `lsm-tree`. +Answer each before unfolding it. + +- [ ] You can narrate insert-to-SST end to end, naming every function the write passes through and what each one is for. + +
+Answer + +`Keyspace::insert` (`mod.rs:905`) takes the journal writer lock (`:919`), +allocates a seqno (`:926`), appends to the journal (`write_raw`, `:928`), +persists in `PersistMode::Buffer` (`:932`), inserts into the memtable +(`tree.insert`, `:940`), publishes the seqno to the snapshot tracker (`:942`), +drops the journal lock (`:944`), accounts the bytes (`:946`) and calls +`maintenance` (`:947`). `maintenance` (`:837`) calls `check_memtable_rotate` +(`:831`), which — if `size > max_memtable_size`, default 64 MiB +(`options.rs:91`) — calls `request_rotation` (`:818`) to *send* a +`WorkerMessage::RotateMemtable`. A pool thread receives it (`worker_pool.rs:141`), +retakes the journal lock and calls `inner_rotate_memtable` (`:727`), which seals +the memtable, enqueues a `FlushTask` (`:746`) and sends `WorkerMessage::Flush` +(`:750`). `flush::run` (`flush/worker.rs:12`) calls `tree.flush(...)` and frees +the write-buffer bytes. `maintenance` also calls `local_backpressure` (`:795`). + +
+ +- [ ] You know which decisions live in fjall and which live in `lsm-tree`, and can name two of each. + +
+Answer + +**fjall owns lifecycle and policy**: the journal and its `PersistMode` +(`journal/writer.rs:35`), memtable rotation and the worker-pool scheduling +(`mod.rs:818` → `worker_pool.rs:141`), backpressure thresholds +(`mod.rs:789–816`), the poison-on-panic guard (`poison_dart.rs:27`), and the +defaults in `keyspace/options.rs` — block size, filter policy, compression +policy, compaction strategy. + +**`lsm-tree` owns the data structure**: the memtable (skip list), segment +format, block index, bloom implementation, the newest-first search inside +`tree.get`, and every compaction *geometry* — fjall's `compaction/worker.rs` is +60 lines that just call `tree.compact(strategy, gc_watermark)`, and +`compaction/mod.rs:7` merely re-exports `Fifo`/`Leveled`/`Levelled` from +`lsm_tree::compaction`. The dependency is pinned at `~3.1.6` in `Cargo.toml:29`. +That split is the layering worth stealing. + +
+ +- [ ] You can state fjall's default durability and explain why it had to be matched before this topic's shootout meant anything. + +
+Answer + +Default is `PersistMode::Buffer` (`mod.rs:932`) — bytes go to the OS page cache +with no `fsync`. That survives a process crash but not power loss. +`PersistMode::SyncData` (`journal/writer.rs:46` → `sync_data()`, `:226`) and +`SyncAll` (`:49` → `sync_all()`, `:220`) are the durable modes. + +It had to be matched because fsync dominates everything else in a write +benchmark. This topic's harness says so in its own header comment +(`experiments/src/lib.rs:5–7`): fjall runs `PersistMode::Buffer` and redb runs +`Durability::None`, with one `SyncAll` at the very end +(`experiments/src/lib.rs:69`), "so neither pays fsync per batch while the other +doesn't". Without that, the 140× space-amp spread would have been confounded by +a durability difference. + +
+ +- [ ] You can explain fjall's measured 0.45× space amplification with a mechanism and a line number, rather than as "LSMs are compact". + +
+Answer + +[FINDINGS.md](../../FINDINGS.md) row 1: 108.0 MB of records occupy 48.4 MB — +space amp **0.45×**, against redb's 63.28×, a 140× spread. The mechanism is two +things, both checkable: + +1. `data_block_compression_policy: CompressionPolicy::new([None, None, Lz4])` at + `src/keyspace/options.rs:116`, with `default = ["lz4"]` at `Cargo.toml:20` — + so everything that reaches L2 or below is LZ4-compressed, and the generated + values are compressible. +2. Sorted runs are packed densely with no per-page fill-factor slack, unlike a + B-tree's ~69% expected utilisation. + +Below 1.0 is therefore not the LSM beating information theory — it is CPU being +spent to buy space, which the RUM paper explicitly puts *outside* its triangle +(see [reading-rum-conjecture.md](reading-rum-conjecture.md) Step 3). + +
+ +- [ ] You can point at where an LSM's write latency tail actually comes from in this codebase. + +
+Answer + +`local_backpressure` at `src/keyspace/mod.rs:795–816`, called from +`maintenance` (`:839`) on *every insert*. Three hard-coded thresholds: a write +stall once L0 has ≥ 20 runs (`:800`), a hard halt looping on 10 ms sleeps once +it has ≥ 30 (`check_write_halt`, `:789–793`), and a halt looping on 100 ms +sleeps while 4 or more memtables are sealed and unflushed (`:806–813`). A writer +that outruns compaction therefore does not degrade smoothly; it hits a cliff and +sleeps in 10–100 ms units. That is the p99.9 in every LSM benchmark. + +
## References -**Code** +**Code** (all line numbers at `fjall-rs/fjall@80cf6bc`, crate 3.1.6 — the pin +table entry; verify with `python3 tools/pinned-source.py ref fjall`) - [fjall](https://github.com/fjall-rs/fjall) — `src/keyspace/mod.rs` - (write/read paths), `src/journal/writer.rs`, `src/flush/worker.rs`, - `src/compaction/worker.rs` (shallow clone at `~/repos/fjall`; line - numbers from the clone — expect drift) -- the external [`lsm-tree`](https://github.com/fjall-rs/lsm-tree) crate - holds the actual tree (memtable, SSTs, blooms, block index) — topic 4's - territory + (`insert:905`, `get:623`, rotation `:727/:818/:831`, backpressure `:789–816`), + `src/keyspace/options.rs` (every default cited here: memtable size `:91`, + block size `:95`, pinning `:100–101`, filter policy `:108–111`, compression + `:116`, compaction strategy `:123`), `src/journal/writer.rs` (`PersistMode:35`), + `src/flush/worker.rs:12`, `src/compaction/worker.rs:10`, `src/worker_pool.rs:141`, + `src/poison_dart.rs:27`, `src/ingestion.rs:36` +- the external [`lsm-tree`](https://github.com/fjall-rs/lsm-tree) crate, pinned + at `~3.1.6` in `Cargo.toml:29`, holds the actual tree (memtable, SSTs, blooms, + block index) — topic 4's territory + +**This repo** +- [FINDINGS.md](../../FINDINGS.md) row 1 — the 0.45× vs 63.28× measurement this + guide explains; `./verify.sh 01` +- [notes.md](notes.md) — the baseline table and its caveats +- `experiments/src/lib.rs:1–7` — the durability-parity decision, in the + harness's own words; note it pins fjall **2.x**, whose `Partition` is 3.x's + `Keyspace` +- [reading-lsm-paper.md](reading-lsm-paper.md) — the `K·(r+1)` write-amp + derivation Step 5 cites +- [reading-turso-btree.md](reading-turso-btree.md) — the update-in-place engine + Step 1 contrasts against diff --git a/topics/01-storage-engine-landscape/reading-lsm-paper.md b/topics/01-storage-engine-landscape/reading-lsm-paper.md index 19f9f33..3c18f43 100644 --- a/topics/01-storage-engine-landscape/reading-lsm-paper.md +++ b/topics/01-storage-engine-landscape/reading-lsm-paper.md @@ -1,191 +1,588 @@ # The LSM-tree: an IO scheduling policy, not a data structure -Where the origin of the LSM half of the topic's dichotomy gets read on its -own terms. Before the paper, this chapter builds the idea from zero — the -write problem, the buffer-and-flush trick, the merge that keeps reads sane, -and the three amplifications that name the price — then hands you a -section-by-section route. Warning up front: **1996 LSM ≠ 2026 LSM.** The -paper's C0/C1 components are B-trees merged by "rolling merge"; modern LSMs -(LevelDB lineage) use immutable sorted files + whole-file compaction. Read -it for the *cost model* — that part is timeless — and translate the -mechanism as you go. +Where the origin of the LSM half of the topic's dichotomy gets read on its own +terms. Before the paper, this chapter builds the idea from zero — the write +problem, the buffer-and-flush trick, the merge that keeps reads sane, and the +three amplifications that name the price — then hands you a section-by-section +route. Warning up front: **1996 LSM ≠ 2026 LSM.** The paper's C₀/C₁ components +are B-trees joined by a *rolling merge*; modern LSMs (the LevelDB lineage) use +immutable sorted files plus whole-file compaction. Read it for the **cost +model** — that part is timeless, and §3 states it in five equations — and +translate the mechanism as you go. + +Every number below is either from the paper (cited to its section, example, +definition or equation) or from this repo's own measurement +([FINDINGS.md](../../FINDINGS.md) row 1). Nothing is a remembered figure. ## The problem in one sentence The motivating workload is TPC-A account history — a firehose of inserts, -almost never read — and indexing it with a B-tree costs one random disk IO -per insert: at ~5 ms per seek that's **~200 inserts/second per disk**, no -matter how fast the CPU is. +almost never read — and indexing it with a B-tree costs one random disk read +plus one random disk write per insert (§3.2, equation 3.1), which on 1995 +hardware means **50 extra disk arms to sustain 1000 inserts/second, doubling the +cost of the whole system** (§1, Example 1.2). ## The concepts, step by step ### Step 1 — the write problem: random in-place writes -An in-place index like a B-tree updates data where it lives: an insert -reads the target leaf page from disk, modifies it, and writes it back to -the same spot. Because keys arrive in essentially random order, each insert -lands on a random page — and on a 1996 disk a random page access is a -mechanical seek, ~5–10 ms: - +> **In:** an index that must stay sorted, and a stream of inserts whose keys +> arrive in random order. +> **Out:** the number the whole paper is attacking — inserts per second per +> disk arm, and why it is set by mechanics rather than by CPU. + +**In-place** means an index updates a record where that record already lives: +the insert reads the target **leaf page** (the bottom-level, page-sized node +that actually stores entries) from disk, modifies it in a memory buffer, and +writes it back to the same disk address. Because the keys arrive in essentially +random order, each insert lands on a different, unpredictable page, and on a +1996 disk reaching an unpredictable page means moving the arm — a **seek**. + +The paper does not hand-wave this; it prices it. From §1, Example 1.2, with the +paper's own parameters: + +```text +Example 1.2 parameters (paper §1): + insert rate 1,000 index entries / second + accumulation window 20 days × 8 hours + index entry size Se 16 bytes (4 B Acct-ID + 8 B Timestamp + + 4 B History-row RID, §5) + page size Sp 4,096 bytes + + entries = 1000/s × 8 h × 3600 s/h × 20 d = 576,000,000 entries + leaf bytes = 576,000,000 × 16 B = 9,216,000,000 B = 9.2 GB + leaf pages = 9.216e9 / 4096 = 2,250,000 ≈ 2.3 M pages ``` - B-tree insert path, keys arriving in random order: - - insert(k₁) → seek to page 8,312 → read 4 KB → write 4 KB ~10 ms - insert(k₂) → seek to page 41,907 → read 4 KB → write 4 KB ~10 ms - ... - ⇒ ~100–200 inserts/s per spindle — while the SAME disk streams - sequential writes at MB/s ⇒ thousands of entries/s if only - we could write them in file order + +Those three figures — 576,000,000 entries, 9.2 GBytes, "about 2.3 million pages +needed on the index leaf level, even if there is no wasted space" — are the +paper's, restated in §5 verbatim. Now the cost: + +```text + per insert, B-tree: 1 random page read + 1 random page write = 2 I/Os + at 1000 inserts/s : 2,000 random I/Os per second + 1995 disk arm : ~40 usable I/Os per second (§1; peak is 60-70, but + 40 is "the nominal usable rate to avoid long queues", §3.1) + arms required : 2000 / 40 = 50 additional disk arms ``` -That ~100× gap between random IOPS and sequential bandwidth is the topic 0 -ladder again — and note the waste: a 4 KB page is rewritten to change one -~100-byte entry. +and §1's conclusion: that "essentially doubles the disk cost for the TPC +application", because the Account table already needed 50 arms for its own 2,000 +I/Os per second (Example 1.1). + +So the honest 1995 figure is **~20 index inserts per second per disk arm** +(40 I/Os ÷ 2 I/Os per insert), not the "few hundred" that gets quoted from +memory. The same arm streams a 64-page multi-page block in 95 ms — 9.5 ms seek, +5.5 ms rotational delay, 80 ms transfer — which §3.1 works out to **about 1.5 +ms/page, so COST_π/COST_P ≈ 1/10**. That ten-to-one gap between "a page reached +by seeking" and "a page reached as part of a big block" is the topic 0 ladder in +1995 dollars, and it is factor one of two in the paper's headline result. + +Note the second waste, which the paper's 100%-full C₁ pages later attack: a +4,096-byte page is read and rewritten to change one 16-byte entry — a +**256:1 byte-level write amplification** before any merging happens at all. ### Step 2 — the idea: buffer in memory, flush sorted runs sequentially -Instead of updating disk in place, collect inserts in a sorted in-memory -tree — the paper's **C0 component** — and, when it fills, write its -contents out to disk in one big sequential pass. Durability comes from a -**write-ahead log** (an append-only file each insert is written to first — -itself a sequential write, so it doesn't reintroduce the problem). One -flush amortizes thousands of inserts over a single sequential IO burst. The -whole 1996 idea fits in one loop — defer, batch, write sequentially, and -pay for it at read time: +> **In:** Step 1's diagnosis — the cost is *where* and *when* bytes hit disk, +> not how many bytes there are. +> **Out:** the two-component structure (C₀, C₁) and the write-ahead log, and +> what each one is for. + +Instead of updating disk in place, collect inserts in a sorted **in-memory** +tree — the paper's **C₀ component** — and let them migrate out later to the +disk-resident **C₁ component**. §2 is explicit that C₀ need not be a B-tree at +all: "the nodes could be any size: there is no need to insist on disk page size +nodes since the C₀ tree never sits on disk", and it names a (2-3) tree or an +AVL-tree as candidates. Its modern descendant is the **memtable** (usually a +skiplist). + +Durability comes from the **write-ahead log** (WAL): an append-only file every +insert is written to before it is acknowledged. §2 puts it first — "a log record +to recover this insert is first written to the sequential log file in the usual +way" — and the crucial property is that the log is *sequential*, so it does not +reintroduce Step 1's random-seek problem. §4.2 goes further and points out the +LSM does not even need its own index log: the ordinary transactional insert +records already contain every field plus the row's RID, so index entries can be +reconstructed from them. + +The whole 1996 idea fits in one loop — defer, batch, write sequentially, and pay +for it at read time: ```rust -fn insert(&mut self, k: Key, v: Val) { - self.wal.append(&k, &v); // durability: a sequential append - self.c0.insert(k, v); // C0: sorted tree in RAM (≈ memtable) - if self.c0.bytes() > THRESHOLD { - // rolling merge: drain C0 into C1 in key order — pages written - // sequentially, ~100% full; ONE batch amortizes thousands of inserts - merge_into(&mut self.c0, &mut self.c1); - } -} - -fn get(&self, k: &Key) -> Option { - self.c0.get(k).or_else(|| self.c1.get(k)) // the read-amp tax: check -} // EVERY component, newest first +// ILLUSTRATION — pseudocode for the paper's §2 two-component algorithm. +// Not from a repo; the real thing in this repo is +// topics/01-storage-engine-landscape/experiments/src/main.rs:1 (fjall lane). +1 fn insert(&mut self, k: Key, v: Val) { +2 self.wal.append(&k, &v); // §2: sequential log record goes first +3 self.c0.insert(k, v); // C0: sorted tree in RAM, "no I/O cost" (§2) +4 if self.c0.bytes() > THRESHOLD { +5 // §2 rolling merge: drain a contiguous key range of C0 into C1. +6 // C1 leaves are packed 100% full and written to a NEW disk position. +7 rolling_merge(&mut self.c0, &mut self.c1); +8 } +9 } +10 +11 fn get(&self, k: &Key) -> Option { +12 // §2: "any search for an index entry will look first in C0 and then in C1" +13 self.c0.get(k).or_else(|| self.c1.get(k)) +14 } ``` -The `get` half is the fine print — Steps 3 and 4. - -### Step 3 — the merge (compaction): what keeps reads bounded - -Flushing sorted runs forever would litter the disk with hundreds of files, -and a lookup would have to check every one of them — so the engine -continually **merges** freshly flushed data into a larger on-disk sorted -component, the paper's **C1** (modern name for the ongoing process: -**compaction**). Merging two sorted inputs is a single interleaved pass: -sequential reads in, sequential writes out, output pages packed ~100% full. +Line 3 is where the speedup lives — §2: "the operation of inserting an index +entry into the memory resident C₀ tree has no I/O cost." Line 13 is the bill, +and it is Steps 3 and 4. + +### Step 3 — the rolling merge: what keeps reads bounded + +> **In:** a C₀ that fills up, and a C₁ that must absorb it without seeking. +> **Out:** the paper's emptying-block / filling-block mechanism, the two +> properties it buys (100%-full pages, new disk positions), and the batching +> parameter M that those properties make possible. + +Flushing sorted runs and leaving them alone would litter the disk with thousands +of files, and a lookup would have to check every one — so the engine +continuously **merges** newly arrived data into the larger component. §2 calls +this the **rolling merge**, and describes it as a cursor that cycles through C₁ +in key order forever: "subsequent merge steps bring together increasing index +value segments of the C₀ and C₁ components until the maximum values are reached +and the rolling merge starts again from the smallest values." + +Mechanically (§2), a merge step uses two buffers: + +- the **emptying block** — a multi-page block of *old* C₁ leaves read in from + disk. §2 envisions multi-page blocks of **256 KBytes**; +- the **filling block** — a multi-page block of *newly merged* C₁ leaves being + built, written out when full. + +Two properties of that loop matter more than the mechanism: + +1. **C₁ nodes are 100% full.** §2: the C₁ tree "has a comparable directory + structure to a B-tree, but is optimized for sequential disk access, with + nodes 100% full". Compare Comer's B-tree, whose expected utilisation is + ln 2 ≈ 69% — that difference alone is a 1/0.69 = 1.45× space saving. +2. **Merged blocks go to *new* disk positions.** §2: "newly merged blocks are + written to new disk positions, so that the old blocks will not be overwritten + and will be available for recovery in case of a crash." This is copy-on-write + at the block level, and §2 credits the inspiration to Rosenblum and + Ousterhout's Log-Structured File System. It is *not* an in-place rewrite — + a detail that matters when you meet immutable SSTs in Step 5, because that + part of modern LSM design was already here in 1996. + +Property 1 plus the delay is what creates the paper's second batching factor. +**Definition 3.2.1** names it: **M**, "the average number of entries in the C₀ +tree inserted into each single page leaf node of the C₁ tree during the rolling +merge", and equation (3.2) computes it: + +```text +(3.2) M = (Sp / Se) · ( S0 / (S0 + S1) ) + + Sp = page size in bytes S0 = size of the C0 leaf level + Se = index entry size in bytes S1 = size of the C1 leaf level + +paper's own worked case, §3 opening: + Se = 16 B, Sp = 4 KB ⇒ Sp/Se ≈ 250 entries per fully packed node + S0 = S1/25 ⇒ M ≈ 250 / 25 ≈ 10 entries merged per C1 leaf + +paper's §3.2 worked case (Definition 3.2.1): + Sp/Se = 200, S1 = 40·S0 ⇒ M = 200 · 1/41 = 4.88 ≈ 5 +``` -Concretely: without merging, a year of 64 MB memtable flushes is thousands -of separate sorted runs — thousands of places a point read must look. With -merging, a read checks C0 then C1: **two** places. The price is that every -entry gets *rewritten* during merges, over and over — which needs a name. +Both are the paper's numbers, one paragraph apart, and the difference between +them is the whole design knob: M is set by how big you are willing to make the +memory component. ### Step 4 — naming the price: read, write, and space amplification -The trade now has standard names. **Write amplification** (bytes actually -written to disk per byte of user data): every merge rewrites entries that -were already on disk, so one logical insert may be physically written many -times over its lifetime. **Read amplification** (number of components or -pages consulted per lookup, versus the one that actually holds the answer): -a read must check every component, newest first — the paper admits this -openly. **Space amplification** (bytes on disk per byte of live data): -overwritten and deleted entries linger in older components until a merge -finally drops them. - -An LSM buys its ~100× insert speedup by moving cost *into* read and space -amplification; a B-tree makes the opposite trade. That three-way tension is -the RUM conjecture chapter, verbatim. - -### Step 5 — 1996's rolling merge vs modern leveled/tiered - -The paper's merge is a *rolling cursor*: C1 stays one single valid B-tree -at all times, and the merge continuously cycles through it in key order, -rewriting pages in place-ish fashion. Modern LSMs dropped that: they write -**immutable sorted files** (SSTs) and compact by merging whole files into -new files, deleting the inputs — simpler crash recovery (files are never -modified, only created and deleted) in exchange for lumpier IO. Translate -as you read: - -``` +> **In:** an engine that batches writes and merges repeatedly. +> **Out:** three ratios, each defined so you could compute it from a directory +> listing, and this repo's measured value for one of them. + +The trade has standard modern names. Each is a ratio, so each needs a numerator +and a denominator stated: + +- **Write amplification** = bytes physically written to the device ÷ bytes of + user data inserted. Every merge rewrites entries that were already on disk, so + one logical insert is physically written many times over its lifetime. +- **Read amplification** = pages (or components) consulted per lookup ÷ the one + page that actually holds the answer. §2 states the LSM's version plainly: + "any search for an index entry will look first in C₀ and then in C₁." +- **Space amplification** = bytes occupied on the device ÷ bytes of live user + data. Overwritten and deleted entries linger in older components until a merge + drops them. + +Two of the three have a paper number and a repo number. + +*Space amp, paper.* §3.1's Example 3.1 prices the B-tree side: the 20-day index +"requires about 9.2 GBytes of leaf-level entries. Given that a growing tree is +only about 70% full, the entire tree will require 13.8 GBytes" — space amp +13.8/9.2 = **1.5×** for a B-tree under this insert pattern. The LSM side is +Example 3.2's "0.7 GBytes on disk because of closely packed entries", against a +1 GByte B-tree — space amp **0.7×**, below one. + +*Space amp, measured here.* [FINDINGS.md](../../FINDINGS.md) row 1 is this +topic's own version of exactly that comparison, on the same 1.08 M records of +100 bytes: + +| engine | family | logical | on disk | space amp | +|---|---|---|---|---| +| fjall | LSM | 108.0 MB | 48.4 MB | **0.45×** | +| redb | B-tree (CoW) | 108.0 MB | 6833.9 MB | **63.28×** | + +A **140× spread**, and the LSM's figure is below 1.0 — for the same reason the +paper's is: closely packed runs (plus, in fjall's case, LZ4 on the value bytes), +paid for with read cost. redb's 63× is not a defect either; `notes.md` explains +it as a copy-on-write B-tree meeting its adversarial case — random key order, +1,080 separate durable batch commits, no compaction afterwards, so every commit +copies each page on the root-to-leaf path and cannot free the old ones yet. Use +those two numbers, not remembered ones, whenever this topic needs a figure for +"LSM vs B-tree space". + +An LSM buys its insert speedup by moving cost *into* read and space +amplification; a B-tree makes the opposite trade. That three-way tension is the +RUM conjecture chapter, verbatim. + +### Step 5 — from rolling merge to leveled compaction, and the write-amp formula + +> **In:** the 1996 mechanism, and the modern vocabulary you already have. +> **Out:** a term-by-term translation table, and §3.4's Theorem 3.1 worked on +> a concrete leveled geometry. + +The paper's merge is a *cursor* over a single, always-valid C₁ B-tree. Modern +LSMs dropped the cursor but kept the write-to-a-new-place discipline: they write +**immutable sorted files** (SSTs) and compact by merging whole files into new +files, then deleting the inputs. Translate as you read: + +```text paper (1996) modern (LevelDB lineage) ───────────── ──────────────────────── -C0 in-memory AVL/2-3 tree → memtable (skiplist) -C1 on-disk B-tree → a level of immutable SSTs +C0 in-memory (2-3)/AVL tree → memtable (skiplist) +C1 ... CK on-disk B-trees → levels L1 ... LK of immutable SSTs rolling merge cursor → compaction job -filling disk pages ~100% full → SST blocks, sequentially written +emptying / filling blocks → compaction input / output files +multi-page block, 256 KB (§2) → SST data block + readahead +C1 nodes packed 100% full (§2) → SST blocks, sequentially written +size ratio r (§3.4) → size ratio / fanout T (usually 10) +number of disk components K → number of levels L +``` + +§3.4 is the part worth doing on paper, because it *is* modern leveled +compaction's write-amplification formula, derived thirty years early. +Symbols (§3.4): + +- `S_i` = bytes of leaf-level entries in component `C_i`; `S = Σ S_i` +- `r_i = S_i / S_{i-1}` = the size ratio between adjacent components +- `R` = steady insert rate into `C₀`, in bytes per second +- `K` = number of *disk*-resident components (so `K+1` components in all) +- `S_p` = page size in bytes; `H` = total page I/O rate needed for all merges + +**Theorem 3.1**: with `S_K`, `S₀` and `R` fixed, `H` is minimised when all the +`r_i` are equal to one constant `r` — i.e. size the components in a *geometric +progression*. Then + +```text +(3.5) S = S0 · (1 + r + r² + ... + r^K) +(3.6) H = (2R / Sp) · ( K·(1 + r) − 1/2 ) +``` + +Equation (3.6) comes straight out of the proof's per-level accounting, and that +accounting *is* the write-amp derivation. For one merge of `C_{i-1}` into `C_i`, +in pages per second: + +```text + read from C_{i-1} R/Sp (the entries migrating out) + read from C_i r·R/Sp (the cursor passes r× as many C_i pages) + write to C_i (r+1)·R/Sp (both inputs land in the enlarged C_i) + ─────────────────────────────── + per level (2r+2)·R/Sp + over K levels K·(2r+2)·R/Sp = (2R/Sp)·K·(1+r) + minus the C0 read, which is free (C0 is in memory): −(1/2)·(2R/Sp) + ⇒ (3.6) +``` + +So, reading the write line only: + +> **write amplification = K · (r + 1)**, and total I/O amplification (reads plus +> writes) = 2·(K·(1+r) − ½). + +Work it on the geometry the modern default describes — size ratio `T = r = 10`, +four disk levels `K = 4`: + +```text + write amp = K·(r+1) = 4 × 11 = 44× + the usual textbook shorthand = T × L = 10 × 4 = 40× (drops the "+1") + total I/O amp = 2·(K·(1+r) − ½) = 2·(44 − 0.5) = 87× + + sizing, from (3.5), with S0 = 10 MB: + S1..S4 = 100 MB, 1 GB, 10 GB, 100 GB + S = 10 MB × (1+10+100+1000+10000) = 10 MB × 11,111 = 111 GB + and inverting: r = (S_K/S0)^(1/K) = (100 GB / 10 MB)^(1/4) = 10000^(1/4) = 10 ``` -The paper's §4 generalizes to multi-component C0…Ck with a size ratio `r` -between adjacent components — exactly modern leveled compaction's fanout-10 -geometry, and its optimal-`r` derivation prefigures Monkey/Dostoevsky -(topic 4). +44× write amp for a 111 GB index is the price of the insert speedup, and it is +why topic 4 spends its time on the `T`/`K` choice rather than on the merge code. +Note the shape of the tradeoff in (3.6): raising `r` makes each merge more +expensive but reduces `K` (since `K = log_r(S_K/S₀)`), so `H` is a genuine +minimisation problem, not a monotone knob. That derivation is what Monkey and +Dostoevsky later reopened (topic 4). + +The paper is also blunt about when the LSM *loses*. §3.3: if `M < K₁ · +COST_π/COST_P` — which happens when C₀ is tiny relative to C₁, or entries are so +large that few fit per page — "this could even cancel the batching effect of +multi-page disk reads, so we would do better to use a normal B-tree for +inserts". There is no LSM-always-wins claim anywhere in the paper. ### Step 6 — the punchline: an IO scheduling policy, not a data structure -Strip the mechanism away and nothing about the *data* changed — same -entries, same sort order, same queries; the only thing the LSM changed is -**when and in what order bytes reach the disk**. The paper's §3 `COST_π` -algebra makes this precise: with batching, each insert's amortized IO cost -is `~(entry_size / page_size) × WA` *sequential* bytes instead of one -random page read+write — the algebra formalizes "sequential bandwidth is -~100× cheaper than random IOPS", the topic 0 ladder in 1996 dollars. That -is why "LSM vs B-tree" survives every hardware generation: it's a policy -choice about IO scheduling, and the constants change but the policy -question doesn't. +> **In:** everything above. +> **Out:** equation (3.4) as the formal statement of the title claim, and +> Definition 5.1 as the reason the claim survives hardware generations. + +Strip the mechanism away and nothing about the *data* changed — same entries, +same collation order, same queries. The only thing the LSM changed is **when, +and in what order, bytes reach the disk.** §3.2 makes that precise. Against the +B-tree baseline of equation (3.1), `COST_B-ins = COST_P · (D_e + 1)` — where +`D_e` is the *effective depth*, "the average number of pages not found in buffer +during a random key-value search", typically **2** for Example 1.2's index — the +LSM's amortised insert cost is equation (3.3), `COST_LSM-ins = 2·COST_π / M`, +and the ratio is: + +```text +(3.4) COST_LSM-ins / COST_B-ins = K1 · (COST_π / COST_P) · (1 / M) + + K1 = 2/(De + 1) ≈ 2/3 ≈ 0.67 (§3.2, for De ≈ 2) + +worked with the paper's own §3.2 values: + COST_π/COST_P = 1/10 (§3.1: 1.5 ms/page in a 64-page block + vs a full random access) + M = 5 (Sp/Se = 200, S1 = 40·S0) + ratio = 0.67 × 0.1 × 0.2 = 0.0134 ≈ 1/75 +``` + +which is §3.2's "nearly two orders of magnitude". Read the two factors: neither +is a property of the *data structure*. `COST_π/COST_P` is a property of how the +I/O is *issued* (one big block versus many small seeks); `1/M` is a property of +how long you *wait* before issuing it. Both are scheduling decisions. + +§5 gives the same claim its cleanest form. **Definition 5.1** calls an access +method a **Continuum Structure** if it "provides for immediate placement of a +newly inserted index entry in its ultimate collation order, based on key-value, +with all other entries already present" — and then observes that B-trees, +extendible hashing, and Bounded Disorder files are all Continuum Structures, so +all of them pay Step 1's random-page cost, and none of them can escape it by +being cleverer about layout. The LSM's one novelty is that it is *not* one: §1 +calls it "a cascaded series of deferred placements". + +That is why "LSM vs B-tree" survives every hardware generation. The constants +move — COST_π/COST_P was 1/10 on a 1995 SCSI-2 disk and is a different number on +NVMe — but the policy question, *how long do I defer placement and how big a +batch do I place*, does not. ## How to read the paper (with the concepts in hand) +The paper's own plan is at the end of §1. Its real section map — worth having +open, because the numbering is easy to misremember: + +| § | Title | What it actually contains | +|---|---|---| +| 1 | Introduction | The Five Minute Rule; Examples 1.1 and 1.2 (TPC-A) | +| 2 | The Two Component LSM-Tree Algorithm | C₀/C₁, rolling merge, §2.1 growth | +| 3 | Cost-Performance and the Multi-Component LSM-Tree | 3.1 disk model, 3.2 equations 3.1–3.4, 3.3 multi-component, 3.4 Theorems 3.1/3.2 | +| 4 | Concurrency and Recovery in the LSM-tree | 4.1 concurrency, 4.2 checkpoint/recovery | +| 5 | Cost-Performance Comparisons with Other Access Methods | Definition 5.1; TSB-tree, MD/OD R-tree, Bounded Disorder | +| 6 | Conclusions and Suggested Extensions | Figure 6.1, the cold/warm/hot cost graph | + Read in this order: -1. **§1 (intro + The Five Minute Rule)** — the economic argument: pages hot - enough are worth keeping in RAM; LSM works because *recent* data is hot - by construction (Step 2's C0 is exactly the hot set). -2. **§2 (two-component LSM)** — Steps 2–3 in the authors' words: C0, C1, - and the rolling merge. Keep Step 5's translation table open and convert - every term to its modern equivalent as you read. -3. **§3 (cost model)** — the payoff, Step 6. Work the `COST_π` algebra - until "amortized sequential bytes per insert" feels obvious; this is the - timeless part. -4. **§4–5 (multi-component + concurrency/recovery)** — skim. Multi-component - C0…Ck with size ratio `r` is modern leveled compaction (Step 5); the - optimal-`r` derivation prefigures Monkey/Dostoevsky (topic 4). The - concurrency/recovery machinery is what immutable SSTs made obsolete. -5. **§6 (comparison)** — skim; the competitors (MD/1 hashing, TSB-tree) are - dead, the framing (amortized cost per insert) survived. +1. **§1, including The Five Minute Rule** — the economic argument. Read the rule + carefully: by 1995 the paper restates it as **60 seconds**, not five minutes + ("the reason it is smaller now in 1995 than when defined in 1987"), and §3.1 + re-derives τ ≈ 62.5 seconds from its own cost table. LSM works because + *recent* data is hot by construction — Step 2's C₀ is exactly the hot set. +2. **§2 (two-component LSM)** — Steps 2–3 in the authors' words. Keep Step 5's + translation table open and convert every term as you read. +3. **§3.1–3.2 (the disk model and the four equations)** — the payoff, Step 6. + §3.1's cost table (COST_m = $100/MB, COST_d = $1/MB, COST_P = $25 per IO/s, + COST_π = $2.5 per IO/s, 1995 workstation) is what makes equation (3.4) + numeric. Work it until "amortised block I/Os per insert" feels obvious. +4. **§3.3–3.4 (multi-component, Theorem 3.1)** — do not skim this; it is Step + 5's write-amp arithmetic and the direct ancestor of leveled compaction. + Equation (3.6) is the one to be able to re-derive. +5. **§4 (concurrency and recovery)** — skim. Most of this machinery is what + immutable SSTs made obsolete; §4.2's checkpoint scheme is the interesting + survivor. +6. **§5 (comparisons)** — read Definition 5.1 and skip the rest. The competitors + (TSB-tree, MD/OD R-tree, Bounded Disorder files) are dead; the framing — + "is this structure a Continuum Structure?" — is the durable idea. ## Questions to answer in notes.md -1. The paper claims LSM trades *what* for its insert speedup? (It's read amp — find - where the paper admits point reads must check every component.) -2. Rolling merge keeps C1 a valid B-tree at all times. What do modern LSMs give up by - using immutable files instead, and what do they gain? (Hint: crash recovery - complexity vs write pattern.) -3. Derive: at size ratio r between components, an entry is rewritten how many times - before reaching the last component? Relate to leveled WA ≈ r × levels. +1. §1 Example 1.2 gets 2.3 million leaf pages from 1,000 inserts/second. Redo + the arithmetic with 2026 numbers — same entry size, an NVMe device at + 500,000 IOPS instead of 40 — and say whether Example 1.2's conclusion + ("essentially doubles the disk cost") still follows. Which of the paper's + two batching factors survives, and which collapses? +2. Equation (3.2) gives `M = (Sp/Se)·(S0/(S0+S1))`. The paper works two cases + with different answers (M ≈ 10 and M = 5). Reconcile them: which parameter + differs, and what does that tell you about which knob an engine actually + controls at run time? +3. §2 says merged blocks are written to *new* disk positions. Modern LSMs use + immutable SSTs. Name one thing modern engines gained by going further than + the paper (whole files immutable, not just blocks relocated) and one thing + they gave up. Point at the §4.2 machinery that the change made unnecessary. +4. Use Theorem 3.1 to derive the write amplification for a tiered geometry + instead of leveled: if each level holds `r` *separate* runs that are merged + only when the level is full, what happens to the `(r+1)` term? Which + amplification moves in the other direction, and by how much? +5. §3.3 states the condition under which a plain B-tree beats an LSM for + inserts. Write it out, substitute this repo's numbers where you can, and + name one real workload in which it holds. ## The one-line takeaway -LSM is not a data structure, it's an *IO scheduling policy*: convert random writes -into sequential ones by deferring and batching — and pay for it at read time. +LSM is not a data structure, it's an *IO scheduling policy*: defer placement and +batch it, so writes leave the machine in block-sized, sequential units — and pay +for it at read time, in a currency equation (3.4) prices exactly. ## Done when -- [ ] You can explain why random in-place writes are the problem the paper is solving, in terms of what the disk is asked to do. -- [ ] You can define read, write and space amplification precisely enough to compute each one. -- [ ] You can work the §3 cost model far enough to say where the insert speedup comes from and what pays for it. -- [ ] You can explain the title claim — an IO scheduling policy, not a data structure — and defend it against the obvious objection that C0/C1 are clearly data structures. -- [ ] You wrote answers to all questions in notes.md, and can connect the paper's rolling merge to the leveled/tiered choice topic 4 asks you to implement. +Answer each before unfolding it. + +- [ ] You can state, in terms of what the disk arm is asked to do, why random in-place writes are the problem the paper is solving — with the paper's own per-insert I/O count. + +
+Answer + +An in-place index must put each new entry in its final collation position +immediately. With random keys that position is on an unpredictable one of the +index's 2.3 million leaf pages (§1, Example 1.2), so each insert costs one +random read plus, in the steady state, one random write of a dirty page — +equation (3.1), `COST_B-ins = COST_P·(D_e + 1)` with `D_e ≈ 2`. At 1,000 +inserts/second that is 2,000 random I/Os per second, and at the 1995 nominal +rate of 40 usable I/Os per disk arm per second, 50 extra arms — which §1 says +doubles the disk cost of the whole TPC application. The CPU is never the +limit; the arm is. + +
+ +- [ ] You can define read, write and space amplification precisely enough to compute each one, and quote this repo's measured space-amp figures rather than a remembered ratio. + +
+Answer + +Each is a ratio with a stated denominator: write amp = bytes physically written +to the device ÷ bytes of user data inserted; read amp = pages (or components) +consulted per lookup ÷ the one that holds the answer; space amp = bytes occupied +on the device ÷ bytes of live user data. + +The measured figures for this topic are in +[FINDINGS.md](../../FINDINGS.md) row 1: on the same 108.0 MB of records, fjall +(LSM) occupies 48.4 MB — space amp **0.45×** — and redb (CoW B-tree) occupies +6,833.9 MB — space amp **63.28×**. A 140× spread. Below 1.0 is not a paradox: +the LSM packs runs densely and compresses values, spending read cost to buy +space, which is the paper's own Example 3.2 result (0.7 GBytes for a 1 GByte +B-tree) with a 2026 compressor attached. + +
+ +- [ ] You can write equation (3.4) from memory, name every symbol in it, and put a number on each factor. + +
+Answer + +`COST_LSM-ins / COST_B-ins = K₁ · (COST_π/COST_P) · (1/M)`. + +- `COST_P` — disk-arm cost to provide 1 page/second of *random* I/O; `COST_π` — + the same for a page read as part of a multi-page block. §3.1 measures the + ratio at ≈ **1/10** (a 64-page block costs 9.5 ms seek + 5.5 ms rotation + + 80 ms transfer = 95 ms, about 1.5 ms/page). +- `M` — Definition 3.2.1, the average number of C₀ entries merged into each + single-page C₁ leaf. Equation (3.2): `M = (Sp/Se)·(S0/(S0+S1))`. §3.2's case: + `Sp/Se = 200`, `S1 = 40·S0` ⇒ **M = 5**. +- `K₁ = 2/(D_e + 1)`, with `D_e` the effective B-tree depth ≈ 2 ⇒ **0.67**. + +Product: 0.67 × 0.1 × 0.2 = 0.0134 ≈ 1/75, which §3.2 rounds to "nearly two +orders of magnitude". + +
+ +- [ ] You can derive the multi-component write amplification from Theorem 3.1 and evaluate it for T = 10 over 4 levels. + +
+Answer + +Theorem 3.1's per-level accounting: merging `C_{i-1}` into `C_i` reads `R/Sp` +pages/second from `C_{i-1}`, reads `r·R/Sp` from `C_i` (the cursor crosses `r` +times as many `C_i` pages), and writes `(r+1)·R/Sp` to the enlarged `C_i`. The +write line alone gives **write amp = K·(r+1)**; summing all three lines over `K` +levels and subtracting the free in-memory `C₀` read gives equation (3.6), +`H = (2R/Sp)·(K·(1+r) − 1/2)`. + +For `r = T = 10`, `K = 4`: write amp = 4 × 11 = **44×** (the usual shorthand +`T × L` = 40× drops the `+1`), and total read+write I/O amplification = +2 × (44 − 0.5) = **87×**. Equation (3.5) sizes it: with `S₀ = 10 MB` the +components are 100 MB, 1 GB, 10 GB, 100 GB and `S = 10 MB × 11,111 = 111 GB`; +inverting, `r = (100 GB / 10 MB)^(1/4) = 10`. + +
+ +- [ ] You can defend the title claim against the obvious objection that C₀ and C₁ are clearly data structures. + +
+Answer + +Both factors in equation (3.4) are scheduling properties, not structural ones: +`COST_π/COST_P` is about *how* the I/O is issued (one 256 KB block versus many +seeks) and `1/M` is about *how long you wait* before issuing it. Neither changes +the entries, their collation order, or the queries. §5's Definition 5.1 is the +sharp version: a **Continuum Structure** places each new entry in its ultimate +collation order immediately, and B-trees, extendible hashing and Bounded +Disorder files all are one — so all pay the same random-placement cost +regardless of how their nodes are shaped. The LSM's single novelty is being +*not* one; §1 calls it "a cascaded series of deferred placements". C₀ and C₁ are +indeed data structures, but they are the *implementation* of a deferral policy, +and §3.3 proves the point from the other side by giving the condition +(`M < K₁·COST_π/COST_P`) under which the same structures lose to a plain B-tree. + +
+ +- [ ] You wrote answers to all five questions in notes.md, and can connect the paper's rolling merge to the leveled/tiered choice topic 4 asks you to implement. + +
+Answer + +The link is Theorem 3.1. The paper proves that, for a fixed largest component, +the I/O rate is minimised when the size ratios `r_i` are all equal — a geometric +progression, which is exactly leveled compaction's fanout. Topic 4's leveled +implementation is Theorem 3.1's optimum; its tiered implementation is what you +get when you relax the "one sorted run per component" assumption and let a level +hold several runs, trading write amplification down for read and space +amplification up. §3.4's closing note that `K+1` is "the only remaining free +variable" is topic 4's level-count knob, and the optimal-`r` derivation is what +Monkey and Dostoevsky reopened with a per-level Bloom-filter budget. + +
## References **Papers** -- O'Neil, Cheng, Gawlick, O'Neil — "The Log-Structured Merge-Tree - (LSM-Tree)" (Acta Informatica 1996) — - [PDF](https://www.cs.umb.edu/~poneil/lsmtree.pdf) — read §1–3 in - order for the cost model; skim §4–6 and translate the mechanism to - modern terms as you go +- O'Neil, Cheng, Gawlick, O'Neil — "The Log-Structured Merge-Tree (LSM-Tree)" + (Acta Informatica 33(4), 1996) — + [PDF](https://www.cs.umb.edu/~poneil/lsmtree.pdf) — §1 for Examples 1.1/1.2 + and the Five Minute Rule; §2 for C₀/C₁ and the rolling merge; §3.1–3.2 for the + disk model and equations (3.1)–(3.4); §3.4 for Theorem 3.1 and equations + (3.5)–(3.6); §5 for Definition 5.1 (Continuum Structure) +- Gray, Putzolu — "The Five Minute Rule for Trading Memory for Disk Accesses" + (SIGMOD 1987) — reference [13] of the LSM paper, the source of the rule §1 + restates at 60 seconds +- Rosenblum, Ousterhout — "The Design and Implementation of a Log-Structured + File System" (SOSP 1991) — §2 credits it for the write-to-new-locations idea + +**This repo** +- [FINDINGS.md](../../FINDINGS.md) row 1 — the measured space-amp comparison + (fjall 0.45×, redb 63.28×) this guide cites instead of a borrowed figure; + re-derive with `./verify.sh 01` +- [notes.md](notes.md) — the baseline table and the explanation of why redb's + 63× is the adversarial case rather than a defect +- [reading-rum-conjecture.md](reading-rum-conjecture.md) — the three-way + formulation of Step 4's amplifications diff --git a/topics/01-storage-engine-landscape/reading-rocksdb-layout.md b/topics/01-storage-engine-landscape/reading-rocksdb-layout.md index a683350..49c43de 100644 --- a/topics/01-storage-engine-landscape/reading-rocksdb-layout.md +++ b/topics/01-storage-engine-landscape/reading-rocksdb-layout.md @@ -1,158 +1,458 @@ # RocksDB: buy the map before walking the territory -RocksDB is everything fjall and tidesdb do, ~50x larger — too big to read, -too important to skip. This chapter is not a walkthrough but an orientation -map: it first builds, step by step, the concept behind each major component -(so every directory name means something), then gives you the directory map -and two entry points. 30 minutes of `ls` and header-skimming now, so that -when topic 4 (compaction), topic 6 (block cache), and topic 22 (db_bench) +RocksDB is everything fjall and tidesdb do, an order of magnitude larger — too +big to read, too important to skip. This chapter is not a walkthrough but an +orientation map: it first builds, step by step, the concept behind each major +component (so every directory name means something), then gives you the +directory map and two entry points. Thirty minutes of header-skimming now, so +that when topic 4 (compaction), topic 6 (block cache), and topic 22 (db_bench) ask "where does X live?", you already know which directory holds the answer. +**All paths and line numbers are at `facebook/rocksdb@7c80a5a`**, this repo's +pinned commit — check with `python3 tools/pinned-source.py ref rocksdb`, and +read any file at that commit with +`python3 tools/pinned-source.py show rocksdb -r A:B`. Every anchor below +was verified against that commit; a few in earlier drafts of this guide were +not, and are corrected inline. + ## The problem in one sentence -RocksDB runs the same LSM lifecycle you traced in fjall — log, memtable, -SST, compaction — but hardened for services at Meta storing hundreds of -terabytes; your problem, in the next 30 minutes, is to learn which of its -~10 top-level directories owns each piece of that lifecycle, so any future -question costs one `ls` instead of a day of grepping. +RocksDB runs the same LSM lifecycle you traced in fjall — log, memtable, SST, +compaction — but hardened for services storing hundreds of terabytes; your +problem, in the next thirty minutes, is to learn which of its ten top-level +directories owns each piece of that lifecycle, so any future question costs one +`ls` instead of a day of grepping. ## The concepts, step by step ### Step 1 — the same machine, industrialized -RocksDB is an LSM (log-structured merge) engine: writes append to a -write-ahead log and land in an in-memory sorted buffer (the **memtable**); -full memtables are flushed to immutable sorted files (**SSTs**); background -**compaction** merges SSTs to keep reads bounded. Everything fjall does in -~10K lines of Rust, RocksDB does in hundreds of thousands of lines of C++ — -the extra mass is not a different algorithm, it's *options* (every knob -pluggable), *operability* (stats, backups, rate limiting), and *scale* -(column families, multi-threaded everything). So the map to build is: which -directory holds each lifecycle stage, plus which directories hold the -industrial padding. +> **In:** the LSM lifecycle you already traced in fjall. +> **Out:** a measured sense of *where* the extra mass went, so you know what +> you are not reading. + +RocksDB is an LSM (log-structured merge) engine: writes append to a write-ahead +log and land in an in-memory sorted buffer (the **memtable**); full memtables +are flushed to immutable sorted files (**SSTs**); background **compaction** +merges SSTs to keep reads bounded. Everything fjall does, RocksDB does — the +extra mass is not a different algorithm, it is *options* (every knob +pluggable), *operability* (stats, backups, rate limiting), and *scale* (column +families, multi-threaded everything). + +Put a number on "larger" by comparing the two files that play the same role: + +```text + RocksDB @7c80a5a fjall @80cf6bc +the class that owns the engine db/db_impl/db_impl.h src/keyspace/mod.rs + 3,759 lines + db/db_impl/db_impl.cc + 8,238 lines + ───────────────────── ────────────────── + 11,997 lines 1,113 lines 10.8× + +the public option surface include/rocksdb/ src/keyspace/ + options.h options.rs + 3,232 lines 742 lines 4.4× + +the public API header include/rocksdb/db.h src/lib.rs + 2,399 lines +``` + +`include/rocksdb/options.h` alone — just the *declarations* of the knobs — is +three times the size of fjall's entire keyspace module. That is the shape of the +difference, and the reason this chapter is a map rather than a reading. ### Step 2 — DBImpl and column families: where everything is wired together +> **In:** a public `DB` interface and a request that has to reach a memtable. +> **Out:** the one class every path goes through, and RocksDB's name for +> fjall's keyspace. + `DBImpl` is the class that owns the whole engine — memtables, SST metadata, -background threads — and implements the public API. It lives in -`db/db_impl/db_impl.h` and is a ~3.8K-line god class: you never read it top -to bottom, you enter it at one method and follow one path. It also manages -**column families** (independent keyspaces — each with its own memtable, -SSTs, and options — that share one write-ahead log, so a batch spanning -several of them commits atomically; `db/column_family.h`). Column family ≈ -fjall's *keyspace* — same concept, same reason to exist. +background threads — and implements the public `DB` API declared in +`include/rocksdb/db.h` (2,399 lines). It lives in `db/db_impl/db_impl.h` and is +a 3,759-line god class declaration backed by an 8,238-line `.cc`: you never read +it top to bottom, you enter at one method and follow one path. The two entry +points worth bookmarking: + +```cpp +// db/db_impl/db_impl.h at facebook/rocksdb@7c80a5a — the two entry points, +// lines 255-256 and 270-273. Everything else in this 3,759-line header is +// reachable from one of them. +255 using DB::Write; +256 Status Write(const WriteOptions& options, WriteBatch* updates) override; +270 using DB::Get; +271 Status Get(const ReadOptions& _read_options, +272 ColumnFamilyHandle* column_family, const Slice& key, +273 PinnableSlice* value, std::string* timestamp) override; +``` + +Note the `ColumnFamilyHandle*` on line 272. A **column family** is an +independent keyspace — its own memtable, its own SSTs, its own options — that +*shares one write-ahead log with its siblings*, so a `WriteBatch` spanning +several of them commits atomically. It lives in `db/column_family.h` (978 +lines). Column family ≈ fjall's *keyspace*: same concept, same reason to exist, +and the shared-WAL detail is the same one that makes fjall take a single journal +lock across all keyspaces. ### Step 3 — `memtable/`: the write buffer is pluggable -In fjall and tidesdb the memtable is one fixed data structure. In RocksDB -it's an interface with several implementations — the default skip list -(`memtable/skiplist.h`), plus hash-based and vector variants for special -workloads. That is the RocksDB pattern in miniature: every component you -saw as a single choice elsewhere is a *directory of choices* here. Cost: -the option surface (Step 7) explodes combinatorially. +> **In:** fjall's single fixed skip-list memtable. +> **Out:** the RocksDB pattern in miniature — one choice becomes a directory of +> choices — and what that costs. + +In fjall and tidesdb the memtable is one fixed data structure. In RocksDB it is +an *interface*: `MemTableRep` at `include/rocksdb/memtablerep.h:62`, with +`MemTableRepFactory` at `:359`, and several implementations — the default skip +list (`memtable/skiplist.h`, 518 lines), plus hash-based and vector variants for +special workloads. `db/memtable.h` (1,042 lines) is the wrapper that holds one +of them plus the sequence-number and flush machinery. + +That is the RocksDB pattern everywhere: every component you saw as a single +choice elsewhere is a directory of choices here. The cost is Step 7's option +surface, which explodes combinatorially — 3,232 lines of it. ### Step 4 — `table/`: the SST file format -An SST here is the **block-based table** format: ~4 KB data blocks of -sorted key-value pairs, an index block mapping first-keys to block offsets, -a filter block (bloom or ribbon filter — "is this key maybe in this file?" -at ~10 bits/key), and a footer that locates the rest. Exactly tidesdb's -SSTable anatomy, productized with compression, checksums, and partitioned -indexes. The format lives in `table/block_based/` and `table/format.h` — -this is topic 4 and topic 6 territory (the block cache caches precisely -these blocks). +> **In:** a sealed memtable that must become an immutable file. +> **Out:** the four parts of a block-based SST, with RocksDB's own defaults and +> its own filter arithmetic. + +An SST here is the **block-based table** format: data blocks of sorted key-value +pairs, an index block mapping first-keys to block offsets, a filter block, and a +footer that locates the rest. Exactly tidesdb's and fjall's SSTable anatomy, +productised with compression, checksums, and partitioned indexes. + +```text + block-based table (table/block_based/, table/format.h): + ┌──────────────────────┬─────────────┬──────────────┬─────────────┬────────┐ + │ data blocks │ filter │ index block │ metaindex │ footer │ + │ 4 KiB default │ block │ first key → │ block │ │ + │ (table.h:400) │ bloom or │ offset │ │ │ + │ sorted KV pairs │ ribbon │ 4 KiB meta │ │ │ + └──────────────────────┴─────────────┴──────────────┴─────────────┴────────┘ +``` + +- `block_size = 4 * 1024` at `include/rocksdb/table.h:400`; + `metadata_block_size = 4096` at `:423`. +- The reader is `table/block_based/block_based_table_reader.h` (981 lines); the + writer is `table/block_based/block_based_table_builder.h` (245 lines); + `table/format.h` (534 lines) is the footer and block-handle encoding. + +The filter block is where RocksDB has gone furthest past fjall, and +`include/rocksdb/filter_policy.h` states the trade in its own numbers +(lines 169–173): a **Ribbon filter** "saves about 30% space compared to Bloom +filters, with similar query times but roughly 3-4x CPU time … if you pass in 10 +for `bloom_equivalent_bits_per_key`, you'll get the same 0.95% FP rate as Bloom +filter but only using about 7 bits per key." + +Work that: 10 bits/key → 0.95% false positives with Bloom; 7 bits/key → the same +0.95% with Ribbon; 3/10 = **30% of the filter memory given back, paid for in +3–4× filter-construction CPU**. And the header goes on (lines 175–188) to make +it a *per-level* decision via `bloom_before_level` (default `0`, signature at +`:210`): "the space savings of Ribbon filters makes sense for lower (higher +numbered; larger; longer-lived) levels of LSM, whereas the speed of Bloom +filters make sense for highest levels." That is the same Monkey-shaped idea as +fjall's per-level `FilterPolicy` array — see +[reading-fjall.md](reading-fjall.md) Step 4 — with a second dimension added. ### Step 5 — versions and the MANIFEST: which files ARE the database -An LSM's file set changes constantly — every flush adds an SST, every -compaction adds some and deletes others. A **version** is one immutable -snapshot of "these exact SST files, at these levels, are the database right -now", and the **MANIFEST** is an append-only log of version *edits* (+file -/ −file records) so the current version survives a crash. This is the LSM's -answer to "what is authoritative?" — in a B-tree engine it's one file; here -it's a *list of files*, and that list needs its own durability story. -`db/version_set.h` owns it. Reads pin a version (so compaction can't delete -files under them) — the same lifetime problem tidesdb solved with refcounts. +> **In:** a file set that changes on every flush and every compaction. +> **Out:** the LSM's answer to "what is authoritative?", and why that answer +> needs its own durability story. + +An LSM's file set is never stable — every flush adds an SST, every compaction +adds some and deletes others. A **version** is one immutable snapshot of "these +exact SST files, at these levels, are the database right now", and the +**MANIFEST** is an append-only log of version *edits* (+file / −file records) so +the current version survives a crash. + +This is a genuinely different problem from the B-tree world. In a B-tree engine +the authoritative thing is one file and a root pointer inside it; here it is a +*list of files*, and lists do not fsync themselves. Hence a whole subsystem: + +| class | file:line | role | +|---|---|---| +| `VersionEdit` | `db/version_edit.h:705` | one MANIFEST record: files added, files deleted | +| `VersionStorageInfo` | `db/version_set.h:131` | the per-level file layout of one version | +| `Version` | `db/version_set.h:914` | one immutable snapshot, refcounted | +| `VersionSet` | `db/version_set.h:1240` | the chain of versions + MANIFEST writer | + +`db/version_edit.h` is 1,151 lines and `db/version_set.h` is 1,980 — this is not +a footnote, it is comparable in size to the memtable and table code combined. +Reads *pin* a version so compaction cannot delete files out from under them — +the same lifetime problem tidesdb solves with refcounts, and the same problem +fjall's `snapshot_tracker` solves with a seqno watermark. ### Step 6 — `db/compaction/`: picker (policy) vs job (mechanics) +> **In:** the write-amplification knob from the LSM paper. +> **Out:** the one architectural split in this directory worth memorising, and +> where each half lives. + RocksDB splits compaction in two, and the split is the thing to remember: -the **compaction picker** decides *which* files to merge (leveled, -universal, FIFO policies — the geometry that sets write amplification), and -the **compaction job** (`db/compaction/compaction_job.h`) does the k-way -merge and writes the outputs. When topic 4 asks "how does leveled -compaction pick files?", the answer is in the picker; when topic 22's -db_bench shows compaction stalls, the mechanics are in the job. + +- the **compaction picker** decides *which* files to merge — this is the + geometry that sets write amplification. Abstract base at + `db/compaction/compaction_picker.h:48` (346 lines), with three concrete + policies: `LevelCompactionPicker` (`compaction_picker_level.h:18`, a 35-line + header — the policy declaration really is that small), + `UniversalCompactionPicker` (`compaction_picker_universal.h:16`), and + `FIFOCompactionPicker` (`compaction_picker_fifo.h:15`); +- the **compaction job** does the k-way merge and writes the outputs — + `db/compaction/compaction_job.h` (743 lines), over the plan described by + `db/compaction/compaction.h` (694 lines). + +When topic 4 asks "how does leveled compaction pick files?", the answer is in +the picker; when topic 22's db_bench shows compaction stalls, the mechanics are +in the job. The write-amplification bill those pickers are trading against is +`K·(r+1)` — 44× for four levels at size ratio 10; see +[reading-lsm-paper.md](reading-lsm-paper.md) Step 5. ### Step 7 — the supporting cast: cache, IO, options, monitoring -The remaining directories are the industrial padding, each a one-liner: +> **In:** the six directories that are not the lifecycle. +> **Out:** one sentence and one verified anchor each, so none of them is ever +> a mystery again. -- `cache/` — the **block cache** (`cache/lru_cache.h`): keeps hot SST data - blocks in RAM so repeat reads skip the disk entirely (topic 6's subject). +- `cache/` — the **block cache** (`cache/lru_cache.h`, 473 lines): keeps hot SST + data blocks in RAM so repeat reads skip the disk entirely. Topic 6's subject, + and it caches precisely the 4 KiB blocks from Step 4. - `file/` + `env/` — IO helpers and the OS abstraction layer - (`env/env_posix.cc`); every read/write goes through here, which is how - RocksDB runs on posix, Windows, and remote storage alike. -- `options/` — the infamous config surface (`options/db_options.h`): - hundreds of knobs, most of them the pluggability from Steps 3–6. -- `monitoring/` — statistics, histograms, perf context - (`monitoring/statistics.h`): how you *see* write stalls and read amp. -- `util/` — blooms, hashing, compression (`util/bloom_impl.h`). + (`file/filename.h`, 200 lines; `env/env_posix.cc`, 532 lines). Every read and + write goes through here, which is how RocksDB runs on POSIX, Windows and + remote storage alike. +- `options/` — the config plumbing (`options/db_options.h`, 174 lines, holds + `ImmutableDBOptions`). The *user-facing* surface is + `include/rocksdb/options.h` at **3,232 lines** — that is the file people mean + when they complain about RocksDB's knob count, and most of those knobs exist + to select among the pluggable choices from Steps 3–6. +- `monitoring/` — statistics, histograms, perf context. **The anchor to use is + `monitoring/statistics_impl.h:42` (`class StatisticsImpl : public + Statistics`), plus the public `include/rocksdb/statistics.h` (957 lines) and + `monitoring/perf_context_imp.h`.** There is no `monitoring/statistics.h` at + this commit — an earlier version of this guide cited one, and it does not + exist. +- `util/` — blooms, hashing, compression (`util/bloom_impl.h`, 489 lines); the + policy-side glue is `table/block_based/filter_policy_internal.h` (347 lines). - `utilities/` — transactions, backup, checkpoints - (`utilities/transactions/` — topic 8 territory). + (`utilities/transactions/pessimistic_transaction.h`, 369 lines — topic 8 + territory). ## Where each step lives in the code ```mermaid flowchart TB - API["include/rocksdb/db.h
public API"] --> DBI["db/db_impl/db_impl.h
DBImpl — ~3.8K-line god class"] - DBI --> MEM["memtable/
skiplist & friends"] - DBI --> TAB["table/
SST formats
block_based/*"] - DBI --> VS["db/version_set.h
manifest: which SSTs exist"] - DBI --> CMP["db/compaction/
compaction_job.h"] - TAB --> CACHE["cache/
lru_cache.h — block cache"] + API["include/rocksdb/db.h
public API, 2399 lines"] --> DBI["db/db_impl/db_impl.h:256 Write
:271 Get — 3759-line god class"] + DBI --> CF["db/column_family.h
column family = fjall keyspace"] + DBI --> MEM["include/rocksdb/memtablerep.h:62
MemTableRep + memtable/skiplist.h"] + DBI --> TAB["table/block_based/
SST format, table/format.h"] + DBI --> VS["db/version_set.h:1240 VersionSet
db/version_edit.h:705 VersionEdit"] + DBI --> CMP["db/compaction/
picker.h:48 vs compaction_job.h"] + TAB --> FILT["include/rocksdb/filter_policy.h:210
bloom_before_level"] + TAB --> CACHE["cache/lru_cache.h
block cache — topic 6"] DBI --> FILE["file/ + env/
IO + OS abstraction"] - DBI --> MON["monitoring/
statistics, histograms"] + DBI --> MON["monitoring/statistics_impl.h:42
+ include/rocksdb/statistics.h"] ``` -| Dir | What lives there | Anchor | Step | +| Dir | What lives there | Verified anchor | Step | |-----|------------------|--------|------| -| `db/` | engine core: DBImpl, column families, versions, compaction | `db/db_impl/db_impl.h`, `db/column_family.h` | 2, 5, 6 | -| `table/` | SST file formats | `table/block_based/`, `table/format.h` | 4 | -| `memtable/` | memtable representations | `memtable/skiplist.h` | 3 | +| `db/` | engine core: DBImpl, column families, versions, compaction | `db/db_impl/db_impl.h:256`, `db/column_family.h`, `db/version_set.h:1240` | 2, 5, 6 | +| `table/` | SST file formats | `table/block_based/block_based_table_reader.h`, `table/format.h` | 4 | +| `memtable/` | memtable representations | `memtable/skiplist.h`, `include/rocksdb/memtablerep.h:62` | 3 | | `cache/` | block/row cache | `cache/lru_cache.h` | 7 | | `file/` | IO helpers, prefetch, filenames | `file/filename.h` | 7 | | `util/` | blooms, hashing, compression | `util/bloom_impl.h` | 7 | -| `options/` | the infamous config surface | `options/db_options.h` | 7 | +| `options/` | config plumbing (public surface is `include/rocksdb/options.h`) | `options/db_options.h` | 7 | | `env/` | OS abstraction | `env/env_posix.cc` | 7 | -| `monitoring/` | stats/histograms/perf context | `monitoring/statistics.h` | 7 | -| `utilities/` | transactions, backup, checkpoints | `utilities/transactions/` | 7 | +| `monitoring/` | stats/histograms/perf context | `monitoring/statistics_impl.h:42` | 7 | +| `utilities/` | transactions, backup, checkpoints | `utilities/transactions/pessimistic_transaction.h` | 7 | ### The two entry points - `DBImpl::Write()` — `db/db_impl/db_impl.h:256` (write path entry) - `DBImpl::Get()` — `db/db_impl/db_impl.h:271` (read path entry) -Everything you traced in fjall/tidesdb exists here too, ~50x larger: journal ↔ -`db/log_writer.cc`, keyspace ↔ column family, manifest ↔ `version_set`. +Everything you traced in fjall and tidesdb exists here too: fjall's journal ↔ +`db/log_writer.h:75` (`class Writer`), fjall's keyspace ↔ column family, fjall's +`snapshot_tracker` version-pinning ↔ `VersionSet`. When topic 4 asks "how does +leveled compaction pick files?", you should already know the answer lives in +`db/compaction/compaction_picker_level.h` and the file metadata in +`db/version_set.h` — navigation cost paid once, here. + +## Questions to answer in notes.md -## Why orient now +These all require the source open; `python3 tools/pinned-source.py show rocksdb + -r A:B` is the fastest way to answer them. -When topic 4 asks "how does leveled compaction pick files?", you should already know -the answer lives in `db/compaction/` and version metadata in `db/version_set.h` — -navigation cost paid once, here. +1. `db/db_impl/db_impl.h:256` declares `Write(const WriteOptions&, WriteBatch*)` + but `:271`'s `Get` takes a `ColumnFamilyHandle*` and `Write` does not. + Explain from `db/column_family.h` how a `WriteBatch` addresses multiple + column families, and why that design forces them to share one WAL. +2. `include/rocksdb/filter_policy.h:169–188` claims Ribbon gives the same 0.95% + FP rate at 7 bits/key that Bloom gives at 10, for 3–4× construction CPU, and + recommends it for *deeper* levels via `bloom_before_level` (default 0 at + `:210`). Work out the memory saved on a 100 GB LSM with 100-byte records at + both settings, and say why "deeper levels get the cheaper-to-build filter" + is the opposite of what you might guess from Monkey. +3. Open `db/version_edit.h:705` and list the fields of `VersionEdit`. Then + answer: after a crash mid-compaction, what exactly makes the *old* input SSTs + still authoritative? Name the record that would have made the new ones + authoritative and when it is written. +4. `db/compaction/compaction_picker_level.h` is a 35-line header while + `db/compaction/compaction_job.h` is 743 lines. Read enough of + `db/compaction/compaction_picker.h:48` to explain the split, and say which of + the two files you would open to change write amplification and which to + change compaction *throughput*. +5. Compare `include/rocksdb/options.h` (3,232 lines) with fjall's + `src/keyspace/options.rs` (742 lines). Pick three RocksDB options that have + no fjall equivalent and, for each, name the Step-3-to-6 pluggability that + made it necessary. ## Done when -Given any lifecycle question — "where is the bloom filter built?", "what -records that an SST was deleted?" — you can name the directory (and usually -the header) without grepping. +Answer each before unfolding it. + +- [ ] Given any lifecycle question — "where is the bloom filter built?", "what records that an SST was deleted?" — you can name the directory, and usually the header, without grepping. + +
+Answer + +Bloom/Ribbon filters: the policy is `include/rocksdb/filter_policy.h` (public) +and `table/block_based/filter_policy_internal.h`; the bit-twiddling is +`util/bloom_impl.h`; the block is written into the SST by +`table/block_based/block_based_table_builder.h`. + +"An SST was deleted" is recorded by a `VersionEdit` (`db/version_edit.h:705`) +appended to the MANIFEST by `VersionSet` (`db/version_set.h:1240`). The file is +not unlinked until no live `Version` (`db/version_set.h:914`) still references +it. + +
+ +- [ ] You can name the two entry points into `DBImpl` and say what each one is the head of. + +
+Answer + +`DBImpl::Write(const WriteOptions&, WriteBatch*)` at +`db/db_impl/db_impl.h:256` — head of the write path: WAL append via +`db/log_writer.h:75`, then memtable insert, then possibly a flush and +compaction schedule. + +`DBImpl::Get(const ReadOptions&, ColumnFamilyHandle*, const Slice&, +PinnableSlice*, std::string*)` at `db/db_impl/db_impl.h:271` — head of the read +path: memtable, then immutable memtables, then the pinned `Version`'s SSTs level +by level, with filter and block-cache lookups in between. + +Between them they reach every subsystem in the table above, which is why the map +is worth more than any single walkthrough. + +
+ +- [ ] You can map every fjall concept you learned onto its RocksDB counterpart, with a file for each. + +
+Answer + +| fjall | RocksDB | +|---|---| +| `Keyspace` (`src/keyspace/mod.rs`) | column family, `db/column_family.h` | +| `Database` / supervisor | `DBImpl`, `db/db_impl/db_impl.h` | +| journal (`src/journal/writer.rs`) | WAL, `db/log_writer.h:75` | +| memtable (fixed skip list in `lsm-tree`) | `MemTableRep` interface, `include/rocksdb/memtablerep.h:62`; default `memtable/skiplist.h` | +| segment / SST | block-based table, `table/block_based/` | +| filter policy array (`options.rs:108`) | `FilterPolicy` + `bloom_before_level`, `include/rocksdb/filter_policy.h:210` | +| `snapshot_tracker` seqno watermark | pinned `Version`, `db/version_set.h:914` | +| — (no equivalent) | MANIFEST / `VersionEdit`, `db/version_edit.h:705` | +| `Leveled` strategy (`compaction/mod.rs:7`) | `LevelCompactionPicker`, `db/compaction/compaction_picker_level.h:18` | + +The one row with no fjall equivalent is the MANIFEST, because fjall delegates +the entire file-set-durability problem to `lsm-tree`. + +
+ +- [ ] You can explain the picker/job split and say which side you would touch to change write amplification. + +
+Answer + +The **picker** (`db/compaction/compaction_picker.h:48` and its three subclasses) +chooses *which* files to merge and therefore fixes the geometry — level count, +size ratio, how many files per job. That geometry *is* write amplification: +`K·(r+1)` from the LSM paper's Theorem 3.1. The **job** +(`db/compaction/compaction_job.h`, 743 lines) executes a chosen plan +(`db/compaction/compaction.h`) — the k-way merge, output file writing, rate +limiting, subcompaction parallelism — and therefore fixes compaction +*throughput*, not its total volume. + +So: change write amp in the picker; change stall behaviour and CPU usage in the +job. + +
+ +- [ ] You can state the Ribbon-vs-Bloom trade in RocksDB's own numbers and say where the option lives. + +
+Answer + +`include/rocksdb/filter_policy.h:169–173`: a Ribbon filter "saves about 30% +space compared to Bloom filters, with similar query times but roughly 3-4x CPU +time and 3x temporary space usage during construction" — 10 +bloom-equivalent bits/key gives 0.95% FP rate at "about 7 bits per key". + +The knob is `bloom_before_level`, default `0`, declared at `:210` and mutable at +runtime via `db->SetOptions({{"table_factory.filter_policy.bloom_before_level", +"3"}})` (`:192`). Lines 175–181 give the rationale: Bloom for the highest +(smallest, hottest, shortest-lived) levels where build speed matters, Ribbon for +the deeper long-lived levels where the 30% memory saving compounds. + +
+ +- [ ] You checked at least one anchor in this guide yourself with `tools/pinned-source.py`, and know how to re-check the rest. + +
+Answer + +`python3 tools/pinned-source.py ref rocksdb` prints +`facebook/rocksdb@7c80a5a`. Then, for example: + +``` +python3 tools/pinned-source.py show rocksdb db/db_impl/db_impl.h -r 255:273 +python3 tools/pinned-source.py grep rocksdb "class CompactionPicker" --glob 'db/compaction/*.h' +``` + +`show` prints the file's total line count in its header, which is how every +"N lines" figure in this guide was obtained. Do this before trusting any line +number in any guide, including this one — the previous version of this file +cited `monitoring/statistics.h`, which does not exist at this commit. + +
## References -**Code** -- [rocksdb](https://github.com/facebook/rocksdb) (shallow clone @ - `7c80a5a` at `~/repos/rocksdb`) — don't read it yet; orient with the - directory map above. Anchors: `db/db_impl/db_impl.h`, - `db/version_set.h`, `db/compaction/`, `table/block_based/`, - `memtable/skiplist.h`, `cache/lru_cache.h` +**Code** (all at `facebook/rocksdb@7c80a5a` — this repo's pin table entry) +- [rocksdb](https://github.com/facebook/rocksdb) — don't read it yet; orient + with the directory map above. Verified anchors: `db/db_impl/db_impl.h:256` + (`Write`) and `:271` (`Get`), `db/column_family.h`, `db/version_set.h:131/914/1240`, + `db/version_edit.h:705`, `db/compaction/compaction_picker.h:48`, + `db/compaction/compaction_picker_level.h:18`, `db/compaction/compaction_job.h`, + `db/log_writer.h:75`, `table/block_based/block_based_table_reader.h`, + `table/format.h`, `include/rocksdb/table.h:400`, + `include/rocksdb/filter_policy.h:169–210`, + `include/rocksdb/memtablerep.h:62`, `memtable/skiplist.h`, + `cache/lru_cache.h`, `util/bloom_impl.h`, `monitoring/statistics_impl.h:42`, + `env/env_posix.cc`, `file/filename.h`, + `utilities/transactions/pessimistic_transaction.h` + +**This repo** +- [reading-fjall.md](reading-fjall.md) — the same lifecycle at 1/10 the size, + with every default cited; read it first +- [reading-tidesdb.md](reading-tidesdb.md) — the same lifecycle in C, where the + SST anatomy of Step 4 is small enough to read end to end +- [reading-lsm-paper.md](reading-lsm-paper.md) — the write-amplification model + the pickers of Step 6 are trading against +- [FINDINGS.md](../../FINDINGS.md) row 1 — this topic's measured LSM-vs-B-tree + space amplification (0.45× vs 63.28×), the number RocksDB's compaction knobs + exist to move diff --git a/topics/01-storage-engine-landscape/reading-rum-conjecture.md b/topics/01-storage-engine-landscape/reading-rum-conjecture.md index 2dfbffd..d94992c 100644 --- a/topics/01-storage-engine-landscape/reading-rum-conjecture.md +++ b/topics/01-storage-engine-landscape/reading-rum-conjecture.md @@ -2,160 +2,547 @@ After the B-tree and LSM papers give the triangle its concrete corners, this short vision paper names the trade-off every storage structure lives inside: -read, update, and memory overhead cannot all approach optimal at once. It -doesn't build anything — it hands you the design compass the rest of the -curriculum steers by. This chapter defines the three overheads one at a -time with real numbers, puts a structure at each corner, and only then -states the conjecture. Read the paper *after* the two engine papers. +read, update, and memory overhead cannot all be bounded at once. It doesn't +build anything — it hands you the design compass the rest of the curriculum +steers by. This chapter defines the three overheads one at a time on real +numbers, puts a structure at each corner, and only then states the conjecture in +the authors' exact words, which are narrower than the version people quote. Read +the paper *after* the two engine papers. + +Citations are to the EDBT 2016 proceedings version (Athanassoulis, Kester, Maas, +Stoica, Idreos, Ailamaki, Callaghan), six pages, sections §1–§6. ## The problem in one sentence -Every index design promises fast reads, cheap updates, and a small -footprint — this 6-page paper claims the promise is structurally -impossible: push any two of the three overheads toward their ideal of 1.0 -and the third acquires a floor that rises. +Every index design promises fast reads, cheap updates, and a small footprint — +this six-page paper claims the promise is structurally impossible: **an access +method that sets an upper bound on two of the three overheads also sets a lower +bound on the third** (§3), and §2 proves the special case with three +constructions you can check by hand. ## The concepts, step by step -### Step 1 — read overhead (RO): how much you read vs how much you needed - -Read overhead is the ratio of data a structure actually reads to the data -strictly required to answer the query — **RO = bytes read ÷ bytes needed**, -ideal 1.0. Concretely, for a point lookup of one 100-byte row among 1M -rows: - -- **unsorted log**: scan ~half the file — ~50 MB read for 100 bytes needed - ⇒ RO ≈ 500,000; -- **B+tree**: 3 page reads of 4 KB — 12 KB for 100 bytes ⇒ RO ≈ 120; -- **array indexed directly by key**: read the one entry ⇒ RO ≈ 1. +### Step 1 — read overhead (RO): everything you touched vs what you wanted + +> **In:** a query that needs some specific base data, and a structure that +> keeps auxiliary data to find it faster. +> **Out:** RO as a ratio with both sides named, and its value for four +> structures on this topic's own dataset. + +The paper's vocabulary first, because the definitions are relative to it (§2): + +- **base data** — the actual rows/tuples the system stores. +- **auxiliary data** — everything an access method keeps *in addition*, to make + operations faster: index nodes, filters, zone maps, sorted copies. + +**Read overhead (RO)** is then, verbatim from §2, "the ratio between the total +amount of data read including auxiliary and base data, divided by the amount of +retrieved data". Note both halves: the numerator includes the index traversal, +and the denominator is what you actually wanted, not what you scanned. The +paper's own example: "when traversing a B+-Tree to access a tuple, the RO is +given by the ratio between the total data accessed (including the data read to +traverse the tree and the base data) and the base data intended to be read." + +The theoretical minimum is **1.0** — §2: "implying that the base data is always +read and updated directly and no extra bit of memory is wasted". + +Put numbers on it with this topic's own shootout parameters — `N` = 1,080,000 +records of 100 bytes, 4,096-byte pages, so `B` = 40 tuples per block and the base +data is `N/B` = 27,000 blocks. For a **point lookup of one record**: + +```text +structure blocks read bytes read RO = bytes read / 100 B +───────────────── ────────────── ─────────── ─────────────────────── +perfect hash index 1 4,096 41 +B+-tree 4 16,384 164 +levelled LSM (T=10) 17 69,632 696 +sorted column 21 86,016 860 +unsorted column 13,500 55,296,000 552,960 +``` -RO is what you feel as query latency: it counts the IOs and cache lines a -lookup burns. +The block counts are Table 1's complexity column, evaluated (Step 4 shows the +substitutions). RO is what you feel as query latency: it counts the I/Os and +cache lines a lookup burns. + +### Step 2 — update overhead (UO): everything you wrote vs what changed + +> **In:** one logical change to one record. +> **Out:** UO as a ratio, its floor, and the number for the same five +> structures. + +**Update overhead (UO)**, §2: "the ratio between the size of the physical +updates performed for one logical update, divided by the size of the logical +update" — and, crucially, "the amount of updates applied to the auxiliary data +*in addition to* the updates to the main data". So a B+-tree's UO counts the +leaf page *and* every interior node the split dirties *and* the WAL copy, over +the 100 bytes you actually meant to change. + +The paper calls this "the write amplification", which is exactly the term the +LSM guide uses — RUM's contribution is generalising it to any structure, not +just merge trees. Ideal is again **1.0**. + +Same dataset, one insert: + +```text +structure blocks written bytes written UO = bytes written / 100 B +───────────────── ────────────── ───────────── ───────────────────────── +unsorted column 1 4,096 41 +perfect hash index 1 4,096 41 +levelled LSM (T=10) 1.11 4,542 45 +B+-tree 4 16,384 164 +sorted column 13,500 55,296,000 552,960 +``` -### Step 2 — update overhead (UO): how much you write vs how much changed +Note the two columns invert almost exactly: the unsorted column is UO-cheapest +and RO-worst; the sorted column is the reverse. That inversion is the conjecture +in miniature, and Step 5 makes it formal. -Update overhead is bytes physically written per byte logically changed — -**UO = bytes written ÷ bytes updated**, ideal 1.0. For an 8-byte update: +UO is what you feel as write throughput and SSD wear. §2 says so directly: +"storage with limited endurance (like flash-based drives) favors minimizing the +update overhead". -- **append-only log**: write the 8 bytes (plus a small header) ⇒ UO ≈ 1; -- **B-tree**: rewrite the whole 4 KB page holding the entry ⇒ UO = 512 — - before counting the WAL copy or a split; -- **sorted array**: insert in the middle shifts ~n/2 entries ⇒ UO ≈ n/2 — - 50 MB moved to add 100 bytes to a 1M-row array. +### Step 3 — memory overhead (MO): footprint vs live data -UO is what you feel as write throughput and SSD wear — it is write -amplification generalized to any structure. +> **In:** a structure sitting on disk or in RAM. +> **Out:** MO as a ratio, and the gap between what the paper's model predicts +> and what this repo actually measured. -### Step 3 — memory overhead (MO): footprint vs live data +**Memory overhead (MO)**, §2: "the space overhead induced by storing auxiliary +data … the ratio between the space utilized for auxiliary and base data, divided +by the space utilized for base data". The paper also calls it "the space +amplification" — the same quantity `notes.md` reports for this topic. -Memory (space) overhead is total bytes the structure occupies per byte of -live data — **MO = bytes stored ÷ bytes of live data**, ideal 1.0. -Concretely: +Work it for a B+-tree over the same dataset. The index entry is a key plus a +child pointer; take 8 bytes each: -- **densely packed sorted array**: no pointers, no slack ⇒ MO ≈ 1; -- **B-tree**: pages average ~69% full, plus interior nodes ⇒ MO ≈ 1.5; -- **tiered LSM**: overwritten versions linger across runs until compaction - ⇒ MO ≈ 2 or worse — plus Bloom filters, which are *extra* bytes stored - purely to reduce RO. +```text +base data 1,080,000 × 100 B = 108.0 MB +index bytes 1,080,000 × 16 B = 17.3 MB (dense) + ÷ 0.69 expected B-tree fill = 25.0 MB (Comer's ln 2 result) +MO = (108.0 + 25.0) / 108.0 = 1.23× +``` -MO is what you feel as disk and RAM bills — and since caches hold fewer -useful entries when MO is high, bad MO quietly worsens effective RO too. +Table 1's levelled-LSM row gives a closed form instead: index size +`O(N·T/(T−1))`, so at size ratio `T = 10`, MO = 10/9 = **1.11×**. + +Now compare those model figures against what this repo actually measured +([FINDINGS.md](../../FINDINGS.md) row 1), on the same 1.08 M records: + +| engine | family | logical | on disk | space amp (MO) | model said | +|---|---|---|---|---|---| +| fjall | LSM | 108.0 MB | 48.4 MB | **0.45×** | 1.11× | +| redb | B-tree (CoW) | 108.0 MB | 6833.9 MB | **63.28×** | 1.23× | + +A **140× spread**, and both engines miss the model by a wide margin in opposite +directions. Both misses are informative, and neither is a defect in the paper: + +- **fjall lands below 1.0** because it LZ4-compresses value bytes into the + sorted run. §5 anticipates this precisely and refuses to count it as a + counterexample: "Orthogonally to the tension between the three overheads … + compression is often used to reduce the amount of data to be moved. This + tradeoff between computation (compressing/decompressing) and data size does + not affect the fundamental nature of the RUM Conjecture." Compression buys MO + with CPU, which is a fourth axis the triangle deliberately does not draw. +- **redb lands at 63×** because Table 1's `O(N/B)` index size assumes a settled + tree, not one being rebuilt. `notes.md` explains the mechanism: random key + order plus 1,080 separate durable batch commits means each commit copies every + page on the root-to-leaf path and cannot free the old ones until a later commit + releases them. The base-data denominator is right; the auxiliary-data numerator + is dominated by transient copies the complexity column never modelled. + +Whenever this topic needs a space-amp figure, use 0.45× and 63.28×, not a +remembered "B-trees are about 1.5×". + +MO is what you feel as disk and RAM bills — and since caches hold fewer useful +entries when MO is high, bad MO quietly worsens effective RO too. §4's Figure 2 +makes that vertical coupling explicit; Step 6 returns to it. ### Step 4 — one structure per corner -Score any structure on all three axes and a pattern appears: the classics -each pin two overheads near 1 and bleed on the third. A **sorted array** is -read- and memory-optimal (RO ≈ 1 binary search, MO ≈ 1) but update-hostile -(UO ≈ n/2). A **log** is update-optimal (UO ≈ 1) but read-hostile (RO ≈ n) -and MO grows with dead versions. The **B+tree** buys good reads with page -slack (MO) and page-granularity writes (UO); the **LSM** buys good updates -with multi-component reads (RO) and lingering versions (MO). The paper's §3 -maps them onto a triangle — reproduce it: - -``` - RO = 1 (read-optimal) - ▲ - B+tree ● │ ● hash index - │ - LSM leveled ● │ ● sorted array (static) - │ - LSM tiered ● │ ● bitmap/bloom (approximate) - │ - log ●────────────────────┴────────────────────● compressed archive - UO = 1 (update-optimal) MO = 1 (space-optimal) +> **In:** the three ratios from Steps 1–3. +> **Out:** the paper's Figure 1 map with its real corner labels, and Table 1's +> complexities evaluated on this topic's `N` and `B`. + +§4 (*RUM in Practice*, not §3) is where the triangle lives — Figure 1, "Popular +data structures in the RUM space". Its corners and the structures §4's prose +assigns to each: + +```text + Read Optimized + ▲ + │ Point & Tree indexes: + │ hash indexes, B-Trees, Tries, + │ Prefix B-Trees, Skiplists + │ + Adaptive structures (middle region): + Database Cracking, Adaptive Merging, + Adaptive Indexing + │ + ●────────────────────────┴────────────────────────● + Write Optimized Space Optimized + + Differential structures: Approximate / sparse indexes: + LSM, Partitioned B-tree (PBT), Bloom filters, count-min sketches, + MaSM, Stepped Merge, Positional lossy bitmaps, approximate tree + Differential Tree, LA-Tree, FD-Tree indexing, ZoneMaps, SMA, + Column Imprints ``` -Topic 1's dichotomy is just two dots on this map: B-tree near the read -corner, LSM stretched along the update edge (leveled closer to reads, -tiered closer to updates). - -### Step 5 — the conjecture itself - -The conjecture: an access method can push any two of RO, UO, MO toward 1.0, -but the third then has a hard lower bound that *grows* as the other two -approach 1. It is not a proven theorem — hence "conjecture"; the paper is -explicit about this — but no counterexample has shown up, and every fix -you try demonstrates it. Watch it happen: the sorted array has RO ≈ 1 and -MO ≈ 1, so the conjecture says updates must hurt — UO ≈ n/2, check. Fix UO -by buffering updates in a log in front of the array, and you've just -invented an LSM — and RO (check every buffer) and MO (dead versions) rise -on cue. The improvement didn't remove the cost; it moved it. +Two things to notice about the real figure that the folk version loses. First, +the corners are named by *what they optimise*, not by "RO = 1" — no real +structure sits at a vertex; the vertices are the unreachable ideals of Step 5's +propositions. Second, the middle is not empty: §4 gives it to adaptive methods, +which "balance the tradeoffs online across a larger area of the design space" +rather than sitting at one point. + +Table 1 is the quantitative version, and it is worth evaluating rather than +admiring. Its parameters: `N` dataset size in tuples, `m` query result size, `B` +block size in tuples, `P` partition size, `T` the LSM level size ratio, `MEM` +memory in pages. With this topic's shootout — `N` = 1,080,000, 100-byte records, +4,096-byte pages so `B` = 40, and `T` = 10: + +| structure | point query (→ RO) | insert (→ UO) | index size (→ MO) | +|---|---|---|---| +| perfect hash | `O(1)` = **1** | `O(1)` = **1** | `O(N/B)` = 27,000 blocks | +| B+-tree | `O(log_B N)` = log₄₀ 1.08e6 = 3.77 → **4** | `O(log_B N)` = **4** | `O(N/B)` = 27,000 blocks | +| levelled LSM | `O(log_T(N/B)·log_B N)` = 4.43 × 3.77 = **16.7** | `O(T/B · log_T(N/B))` = 0.25 × 4.43 = **1.11** | `O(N·T/(T−1))` = **1.11·N** | +| sorted column | `O(log₂ N)` = **20.0** | `O(N/B/2)` = **13,500** | `O(1)` | +| unsorted column | `O(N/B/2)` = **13,500** | `O(1)` = **1** | `O(1)` | + +Read the LSM row against the B+-tree row: **4.2× worse point reads (16.7 vs 4), +3.6× better inserts (1.11 vs 4)**. That single pair of ratios is topic 1's +dichotomy, in the paper's own complexity model, and it is what the shootout is +measuring. §4's own summary of the table: "ZoneMaps have the smaller size — being +a sparse index, but Hash Indexes offer the fastest point queries, while B+-Trees +offer the fastest range queries." + +### Step 5 — the conjecture itself, and the three propositions under it + +> **In:** the three ratios, each with an ideal of 1.0. +> **Out:** the exact statement (which is about *bounds*, not about *approaching +> 1.0*), plus the three §2 propositions that motivate it, each checked on +> numbers. + +The statement, §3, word for word: + +> **The RUM Conjecture.** An access method that can set an upper bound for two +> out of the read, update, and memory overheads, also sets a lower bound for the +> third overhead. + +Two precisions worth holding onto, because the popular paraphrase loses both. +It is about **bounds**, not about "approaching 1.0" — the claim is that bounding +any two *forces* a floor under the third, whatever values those two bounds take. +And it is a **conjecture**: §6 says the paper "shows through the RUM Conjecture +that creating the ultimate access method is infeasible", but nowhere is a proof +offered, and §5 spends its length on a research roadmap rather than on a +theorem. + +§2 states a strictly weaker **Hypothesis** first — "an access method that is +optimal with respect to one of the read, update, and memory overheads, cannot +achieve the optimal value for both remaining overheads" — and then backs it with +three constructions on a deliberately trivial model: `N` fixed-size integers, +one per block, block ID = `blkID`, workload of point queries, updates, inserts +and deletes. Each construction is checkable by hand: + +**Prop. 1 — `min(RO) = 1.0 ⇒ UO = 2.0 and MO → ∞.`** Store each value in the +block whose `blkID` equals the value itself. Lookup is one direct address, so +RO = 1.0 exactly. But the array is sparse: the paper's example, the relation +{1, 17}, needs 17 blocks to hold 2 values — MO = 8.5 for two elements, and +unbounded in general "since, in the general case, we cannot anticipate what would +be the maximum value ever inserted". With this topic's 8-byte keys the address +space is 2⁶⁴ blocks for 1.08 M live values, so MO ≈ 1.7 × 10¹³. UO is 2.0 +because changing a value must empty the old block and fill the new one. + +**Prop. 2 — `min(UO) = 1.0 ⇒ RO → ∞ and MO → ∞.`** Append every update to a log +and never reorganise. UO = 1.0 exactly. Both other overheads then grow *without +bound as updates arrive*, because every superseded version stays and every read +must consider all of them: "for minimum UO, both RO and MO perpetually increase +as updates are appended." + +**Prop. 3 — `min(MO) = 1.0 ⇒ RO = N and UO = 1.0.`** Store a dense array, keep +no auxiliary data, update in place. MO = 1.0, and UO is *also* 1.0 — you touch +only the base data you meant to. The full price lands on one axis: a worst-case +point query scans everything, RO = `N`. On this topic's dataset that is 27,000 +blocks, matching the "unsorted column" row of Table 1. + +Prop. 3 is the one to sit with, because it is the title of the guide: it pins +**two** overheads at their theoretical optimum simultaneously, and the third +goes to `N`. That is "optimize two, pay with the third" as a construction rather +than a slogan. + +Now watch the conjecture bite on a fix. Start from Prop. 3's dense array: MO = +1.0, UO = 1.0, RO = N. Fix RO by keeping the array sorted and binary-searching +it — RO drops to log₂ N = 20 — and UO immediately jumps to `N/B/2` = 13,500, +exactly Table 1's sorted-column row. Fix *that* by buffering updates in a log in +front of the sorted array and merging periodically, and you have just derived +the LSM: UO falls back to 1.11, while RO rises to 16.7 and MO rises to 1.11·N. +The improvement did not remove the cost; it moved it, every time, and it moved +it to whichever axis you were not bounding. ### Step 6 — how to use it: a compass, not a theorem -The practical payoff is that every tuning knob is a *position on the -triangle*, not a setting with a correct value. Bloom filter bits/key trades -MO for RO. Compaction eagerness (leveled vs tiered) trades UO for RO. Page -fill factor trades MO for UO. So a design review starts with "what does the -workload need?" and then *chooses where to pay* — and Monkey (topic 4) -turns exactly this into a formal optimization problem, allocating memory -across Bloom filters to minimize RO at fixed MO. What the compass rules -out: any claim that a structure improved one overhead with *no* movement -elsewhere — find where the cost went before believing the benchmark. +> **In:** a tuning knob, a benchmark claim, or a memory hierarchy. +> **Out:** three ways the paper says to use the triangle, and the one class of +> claim it lets you reject on sight. + +**Knobs are positions, not settings.** §5 lists the parameters that move a +structure around the space by name: "the fan-out of B+-Trees, the number of +partitions in PBT, the number of sorted runs in MaSM", and, in its wishlist, +"B+-Trees that have dynamically tuned parameters, including tree height, node +size, and split condition, in order to adjust the tree size, the read cost, and +the update cost at runtime". So Bloom-filter bits per key trades MO for RO; +compaction eagerness (levelled vs tiered) trades UO for RO; page fill factor +trades MO for UO. A design review starts with "what does the workload need?" and +then *chooses where to pay*. Monkey (topic 4) turns exactly this into a formal +optimisation: allocate a fixed memory budget across per-level Bloom filters to +minimise RO at fixed MO. + +**The triangle applies per level of the memory hierarchy, not once globally.** +§4's Figure 2 is the part almost nobody quotes: "The RUM tradeoffs, however, +still hold for each level individually … The RUM tradeoffs can also be viewed +vertically rather than horizontally. For example, the RO_n read and the UO_n +update overheads at memory level n can be reduced by storing more data, updates, +or meta-data, at the previous level n−1, which results, at least, in a higher +MO_{n−1}." That is a one-sentence theory of caching, buffer pools, and +memtables: every one of them buys RO and UO at level *n* by spending MO at level +*n−1*. It is also why the topic 0 latency ladder and this paper are the same +argument seen from two angles. + +**What the compass rules out.** Any claim that a structure improved one overhead +with *no* movement elsewhere. Find where the cost went before believing the +benchmark — and if the answer is "compression", §5 says that is a +computation-for-space trade sitting orthogonal to the triangle, so ask what it +cost in CPU instead. fjall's 0.45× is exactly that case. ## How to read the paper (with the concepts in hand) -1. **§1–2** — the RO/UO/MO definitions, i.e. Steps 1–3; make sure you can - compute all three for a plain sorted array (RO≈1, UO≈n/2 shifts, MO≈1) - and a log (UO≈1, RO≈n, MO grows) before moving on. -2. **§3 (the map)** — Step 4: the paper places real structures on the - triangle. Reproduce the diagram from memory. -3. **§4 (moving on the map)** — Steps 5–6, the punchline for this - curriculum: knobs are *positions*, not settings. Bloom bits/key trades - MO for RO. Compaction eagerness trades UO for RO. Page fill factor - trades MO for UO. Monkey (topic 4) turns this into an actual - optimization problem. -4. **§5 (research directions)** — skim; grade its 2016 predictions with 2026 - hindsight (adaptive/learned indexes, versioned data — how did they age?). +Six pages, roughly one hour. The real section map, since the numbering is easy +to misremember: + +| § | Title | Contains | +|---|---|---| +| 1 | Introduction | The framing; the tagline "Optimize Two at the Expense of the Third" | +| 2 | The RUM Overheads | RO/UO/MO definitions; the Hypothesis; **Props. 1, 2, 3** | +| 3 | The RUM Conjecture | Twenty lines. Just the statement | +| 4 | RUM in Practice | **Figure 1** (the triangle), **Table 1**, Figure 2 (memory hierarchy), cache-oblivious methods | +| 5 | Building RUM Access Methods | Figure 3; the roadmap; the compression note | +| 6 | Summary | Two paragraphs | + +Read in this order: + +1. **§2** — Steps 1–3. Make sure you can restate all three definitions with the + *denominator* named, and can re-derive Props. 1–3 on the paper's array of + integers before moving on. This is 60% of the paper's content. +2. **§3** — Step 5. It is one paragraph; read it twice and note that it says + "upper bound"/"lower bound", not "optimal". +3. **§4** — Step 4. Reproduce Figure 1's corners and their regions from memory, + then evaluate Table 1 on your own `N` and `B` rather than reading the + complexities as decoration. Do not skip Figure 2's paragraph on the memory + hierarchy; it is the most reusable idea in the paper. +4. **§5** — Step 6, plus a grading exercise: the roadmap was written in 2016 and + names five specific wishes (tunable B+-trees, updatable approximate indexes, + morphing access methods, update-friendly bitmaps, log-plus-filter methods). + Mark each as delivered, partly delivered, or not, with 2026 evidence. +5. **§6** — skim. ## Questions to answer in notes.md -1. Place your engine_shootout results on the triangle: which measured number is RO, - UO, MO for fjall and redb? -2. Where does FalkorDB's matrix adjacency sit? (Dense-ish matrix: MO poor for sparse - graphs — that's why delta matrices + roaring exist, topics 20/26.) -3. What's the RUM position of a WAL by itself? Why does *every* engine carry one - anyway? (Durability isn't in the triangle — it's an orthogonal axis the paper - deliberately excludes.) +1. Place this topic's shootout results on the triangle. Which measured number + from `notes.md` is an MO, and what would you have to instrument to get RO and + UO for fjall and redb? (Neither is currently measured — say what the lane + would have to record.) +2. §2's Prop. 3 pins *two* overheads at 1.0 simultaneously. Reconcile that with + the §3 conjecture: does Prop. 3 contradict it, and if not, which of the three + quantities is the one being "bounded" in the conjecture's sense? +3. Table 1 gives levelled LSM an index size of `O(N·T/(T−1))` — 1.11·N at + T = 10. fjall measured 0.45×. Explain the direction of the discrepancy using + §5's compression paragraph, and say what the model would have to add to + predict a number below 1.0. +4. The paper never names durability as an axis. Take a WAL: score it on RO, UO + and MO using §2's definitions (is a WAL auxiliary data?), and then say + whether "every engine carries one anyway" is a point on the triangle or + evidence that the model is incomplete. Defend your answer from the §2 text, + not from intuition. +5. Where does FalkorDB's matrix adjacency sit? Score it on all three axes for a + *sparse* graph, and name the two structures later topics use to move it (see + topics 20 and 26). ## The one-line takeaway -There is no best index, only a workload-shaped position on a three-way frontier — -"which engine is better" is an ill-posed question until the workload is named. +There is no best index, only a workload-shaped position on a three-way frontier +— "which engine is better" is an ill-posed question until the workload is named, +and any benchmark showing an improvement with no offsetting cost has simply not +measured the axis that paid. ## Done when -- [ ] You can define RO, UO and MO as ratios and say what the denominator is in each. -- [ ] You can name one real structure per corner of the triangle. -- [ ] You can state what the conjecture does and does not claim (it is a conjecture and a compass, not a proven bound). -- [ ] You have placed this topic's own measured result on the triangle: fjall at 0.45x and redb at 63.28x space amp, and can say which axis each engine is spending. -- [ ] You wrote answers to both questions in notes.md, including where FalkorDB's matrix adjacency sits. +Answer each before unfolding it. + +- [ ] You can define RO, UO and MO as ratios, naming the numerator and denominator of each, in the paper's auxiliary-vs-base-data vocabulary. + +
+Answer + +All three are stated in §2 relative to **base data** (the rows the system +stores) versus **auxiliary data** (anything an access method keeps in addition: +index nodes, filters, sorted copies). + +- **RO** = total data read, *auxiliary plus base*, ÷ the amount of data actually + retrieved. Ideal 1.0. +- **UO** = size of the physical updates performed for one logical update, + *auxiliary plus base*, ÷ the size of the logical update. The paper calls this + "the write amplification". Ideal 1.0. +- **MO** = space used for auxiliary plus base data ÷ space used for base data. + The paper calls this "the space amplification". Ideal 1.0. + +Ideal 1.0 means, in §2's words, "the base data is always read and updated +directly and no extra bit of memory is wasted". + +
+ +- [ ] You can state the conjecture in the authors' words and say precisely what it does *not* claim. + +
+Answer + +§3: "An access method that can set an upper bound for two out of the read, +update, and memory overheads, also sets a lower bound for the third overhead." + +What it does not claim: (a) that the overheads must approach 1.0 — the claim is +about *any* pair of upper bounds, not about optimal ones; (b) that this is +proven — it is a conjecture, and no proof appears in the paper, only §2's three +constructions and §4's survey; (c) that it covers every cost. §5 explicitly puts +compression outside it, as a computation-versus-size trade that "does not affect +the fundamental nature of the RUM Conjecture", and the paper never mentions +durability, concurrency or latency variance at all. + +
+ +- [ ] You can reproduce §2's three propositions and check each on numbers. + +
+Answer + +On the paper's model — `N` fixed-size integers, one per block, addressed by +`blkID`: + +- **Prop. 1**: `min(RO) = 1.0 ⇒ UO = 2.0 and MO → ∞`. Direct addressing, block + ID = value. The paper's own case, the relation {1,17}, occupies 17 blocks for + 2 values (MO = 8.5); with 8-byte keys the address space is 2⁶⁴ blocks for + 1.08 M live values, MO ≈ 1.7 × 10¹³. UO = 2.0 because a value change empties + one block and fills another. +- **Prop. 2**: `min(UO) = 1.0 ⇒ RO → ∞ and MO → ∞`. Append-only log; both other + overheads "perpetually increase as updates are appended". +- **Prop. 3**: `min(MO) = 1.0 ⇒ RO = N and UO = 1.0`. Dense array, in-place + updates, no auxiliary data — two overheads at the optimum at once, and a + full scan for every point query. On this topic's dataset, RO = 27,000 blocks, + which is Table 1's unsorted-column row. + +
+ +- [ ] You can name each corner of Figure 1 by what it optimises, give two real structures per corner, and say what occupies the middle. + +
+Answer + +Figure 1's corners are **Read Optimized** (top), **Write Optimized** (bottom +left), **Space Optimized** (bottom right) — named by goal, not by "RO = 1", +because no real structure reaches a vertex. + +- Read: hash indexes, B-Trees (also Tries, Prefix B-Trees, Skiplists). +- Write, which §4 calls *differential structures*: LSM, Partitioned B-tree + (also MaSM, Stepped Merge, Positional Differential Tree, LA-Tree, FD-Tree). +- Space: Bloom filters, ZoneMaps (also count-min sketches, lossy bitmaps, + approximate tree indexing, Small Materialized Aggregates, Column Imprints). + +The middle holds **adaptive** methods — Database Cracking, Adaptive Merging, +Adaptive Indexing — which §4 says "balance the tradeoffs online across a larger +area of the design space" instead of sitting at one point. + +
+ +- [ ] You can evaluate Table 1 on a concrete `N` and `B` and read topic 1's dichotomy out of two rows. + +
+Answer + +With `N` = 1,080,000 records of 100 B and 4,096-byte pages (so `B` = 40 tuples +per block), `T` = 10: + +- **B+-tree**: point query `O(log_B N)` = log₄₀ 1.08e6 = 3.77 → 4 I/Os; insert + `O(log_B N)` = 4; index size `O(N/B)` = 27,000 blocks. +- **Levelled LSM**: point query `O(log_T(N/B)·log_B N)` = 4.43 × 3.77 = 16.7 + I/Os; insert `O(T/B · log_T(N/B))` = 0.25 × 4.43 = 1.11 I/Os; index size + `O(N·T/(T−1))` = 1.11·N. + +Ratio of the two rows: the LSM pays **4.2× more I/O per point read** and +**3.6× less per insert**. That is topic 1's whole dichotomy, in complexity form, +before a single benchmark runs. + +
+ +- [ ] You have placed this topic's own measured result on the triangle, and can say which axis each engine is spending and why the model missed both. + +
+Answer + +[FINDINGS.md](../../FINDINGS.md) row 1, same 108.0 MB of records: fjall +**0.45×** space amp, redb **63.28×** — a 140× spread. Both are MO measurements. + +fjall is below the model's 1.11× because it LZ4-compresses value bytes; §5 puts +that trade outside the triangle ("this tradeoff between computation … and data +size does not affect the fundamental nature of the RUM Conjecture"), so the real +price is CPU, an axis the figure does not draw. redb is far above the model's +~1.23× because Table 1's `O(N/B)` index size describes a settled tree, and +`notes.md` shows this lane never lets it settle: random key order plus 1,080 +durable batch commits means every commit copies the whole root-to-leaf path and +cannot free the predecessors yet. Neither number refutes the conjecture; both +show that a complexity column is not a measurement. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including the WAL scoring and where FalkorDB's matrix adjacency sits. + +
+Answer + +The WAL question has no clean answer inside the model, and noticing that is the +point: by §2's definition a WAL is auxiliary data, so it inflates UO (every byte +written twice) and MO (the retained tail) while doing nothing for RO — a strict +loss on the triangle. Every engine carries one anyway, which means the axis it +buys — crash durability — is simply not in the model. Say so explicitly rather +than forcing it onto a corner. + +FalkorDB's matrix adjacency is read-optimised: a sparse-matrix representation of +adjacency makes traversal a linear-algebra kernel (low RO for multi-hop), pays +UO on every edge insert into a compressed matrix, and pays MO badly when the +graph is sparse and the representation is not. That MO cost is exactly why delta +matrices and roaring bitmaps appear in topics 20 and 26 — both are moves toward +the Space Optimized corner. + +
## References **Papers** -- Athanassoulis, Kester, Maas, Stoica, Idreos, Ailamaki, Callaghan — - "Designing Access Methods: The RUM Conjecture" (EDBT 2016) — - [PDF](https://stratos.seas.harvard.edu/files/stratos/files/rum.pdf) — - ~6 pages, 1 h; read after the B-tree and LSM papers so the triangle - has concrete corners +- Athanassoulis, Kester, Maas, Stoica, Idreos, Ailamaki, Callaghan — "Designing + Access Methods: The RUM Conjecture" (EDBT 2016, pp. 461–466) — + [PDF](https://openproceedings.org/2016/conf/edbt/paper-12.pdf) — §2 for the + definitions and Props. 1–3; §3 for the one-paragraph conjecture; §4 for + Figure 1, Table 1 and Figure 2; §5 for the roadmap and the compression note. + Six pages, ~1 h; read after the B-tree and LSM papers so the triangle has + concrete corners +- O'Neil, Cheng, Gawlick, O'Neil — "The Log-Structured Merge-Tree" (1996) — + the write-optimised corner's founding member, cited as [44] +- Dayan, Athanassoulis, Idreos — "Monkey: Optimal Navigable Key-Value Store" + (SIGMOD 2017) — the same group turning §5's "tunable RUM balance" into an + actual optimisation problem; topic 4 implements it + +**This repo** +- [FINDINGS.md](../../FINDINGS.md) row 1 — the measured MO figures (fjall + 0.45×, redb 63.28×) this guide scores against Table 1; `./verify.sh 01` +- [notes.md](notes.md) — why redb's 63× is the adversarial case, not a defect +- [reading-lsm-paper.md](reading-lsm-paper.md) — the write-optimised corner's + cost model, including the `K·(r+1)` write-amplification derivation +- [reading-comer-btree.md](reading-comer-btree.md) — the read-optimised corner's + fanout and ln 2 ≈ 69% utilisation results, used in Step 3's MO arithmetic diff --git a/topics/01-storage-engine-landscape/reading-tidesdb.md b/topics/01-storage-engine-landscape/reading-tidesdb.md index 5837148..c355e2a 100644 --- a/topics/01-storage-engine-landscape/reading-tidesdb.md +++ b/topics/01-storage-engine-landscape/reading-tidesdb.md @@ -1,44 +1,75 @@ # tidesdb: the same LSM with nothing abstracted away -The value of this skim (1–2 h) is seeing the machinery you just traced in -fjall rendered in plain C, with *nothing* hidden — memory ordering, pointer -arithmetic, and disk offsets are all in your face. This chapter first -rebuilds the LSM lifecycle step by step, each time pointing at the concrete -C structure that fjall's Rust abstractions wrap. Read it as a contrast -exercise: match each fjall concept to its C twin and notice exactly what -Rust's abstractions buy you, and what they conceal. +The value of this skim (1–2 h) is seeing the machinery you just traced in fjall +rendered in plain C, with *nothing* hidden — memory ordering, pointer arithmetic +and disk offsets are all in your face. This chapter first rebuilds the LSM +lifecycle step by step, each time pointing at the concrete C structure that +fjall's Rust abstractions wrap. Read it as a contrast exercise: match each fjall +concept to its C twin and notice exactly what Rust's abstractions buy you, and +what they conceal. + +**tidesdb *is* in this repo's pin table**, at `tidesdb/tidesdb@810507a` — confirm +with `python3 tools/pinned-source.py ref tidesdb`, list the tree with +`python3 tools/pinned-source.py list tidesdb`, and read any file at that commit +with `python3 tools/pinned-source.py show tidesdb -r A:B`. Every line +number below was checked against that commit; several anchors in the previous +version of this guide were off and are corrected inline. Note the paths all +carry a `src/` prefix — it is `src/tidesdb.c`, not `tidesdb.c`. ## The problem in one sentence Do fjall's job — absorb random-key writes as sequential IO, survive crashes, -answer reads in a handful of file probes — in ~40K lines of C where every -byte offset, atomic barrier, and malloc is spelled out by hand. +answer reads in a handful of file probes — in a 37,702-line C file where every +byte offset, atomic barrier and `malloc` is spelled out by hand. ## The concepts, step by step ### Step 1 — the LSM recipe, restated in plain C terms +> **In:** the LSM lifecycle from the fjall chapter, as concepts. +> **Out:** the five files those concepts live in, with real line counts, so you +> know the size of what you are about to skim. + An LSM (log-structured merge) engine never updates data in place. It appends -every write to a log file for crash safety, buffers the same write in a -sorted in-memory structure (the **memtable**), periodically dumps the full -memtable to disk as an immutable sorted file (an **SSTable**), and merges -those files in the background (**compaction**) to keep reads cheap. (The -fjall chapter builds the *why* of each piece from sequential-vs-random IO; -this one shows each piece as bytes and structs.) In tidesdb every one of -those nouns is a file you can open: the memtable is `skip_list.c`, the log -and SSTables go through `block_manager.c`, the "maybe present?" filter is -`bloom_filter.c`, and the list of which SSTable belongs to which level is -`manifest.c`. Nothing else. That is the whole engine. +every write to a log file for crash safety, buffers the same write in a sorted +in-memory structure (the **memtable**), periodically dumps the full memtable to +disk as an immutable sorted file (an **SSTable**), and merges those files in the +background (**compaction**) to keep reads cheap. The fjall chapter builds the +*why* of each piece from sequential-vs-random IO; this one shows each piece as +bytes and structs. + +In tidesdb every one of those nouns is a file you can open, and the sizes are +the first surprise: + +```text +src/tidesdb.c 37,702 lines write path, read path, compaction, workers +src/skip_list.c 2,929 lines the memtable — skip list + arena allocator +src/block_manager.c 2,004 lines physical block IO: WAL frames and SST blocks +src/tidesdb.h 2,152 lines every public type and every tunable +src/manifest.c 923 lines which SSTable belongs to which level +src/bloom_filter.c 624 lines the whole filter, hash mixing included +``` + +Nothing else. That is the whole engine — plus `src/btree.c` (tidesdb can build a +B-tree-shaped SSTable), `src/clock_cache.c`, `src/compress.c` and the object-store +backends, none of which you need for this pass. Compare with fjall, where +`src/keyspace/mod.rs` is 1,113 lines and *everything below the lifecycle* is +behind the `lsm-tree` dependency: the ratio is not that C is more verbose, it is +that tidesdb has no crate boundary to hide behind. ### Step 2 — the memtable is a skip list you can read -A **skip list** is a sorted linked list with "express lanes": each node gets -a random height, and higher lanes skip over many nodes, so search is -O(log n) like a balanced tree — but insertion never rebalances anything, -which makes it easy to run lock-free (concurrent threads use atomic -pointer swaps instead of locks). +> **In:** the need for a sorted, concurrently-writable in-memory buffer. +> **Out:** why a skip list rather than a tree, and the allocation strategy +> fjall's Rust hides. -``` +A **skip list** is a sorted linked list with "express lanes": each node gets a +random height, and higher lanes skip over many nodes, so search is O(log n) like +a balanced tree — but insertion never rebalances anything, which is what makes +it easy to run lock-free (concurrent threads use atomic pointer swaps instead of +locks). + +```text level 3: head ──────────────────► k₄₀ ─────────────────► nil level 2: head ────────► k₂₂ ────► k₄₀ ────────► k₇₈ ───► nil level 1: head ─► k₀₇ ─► k₂₂ ────► k₄₀ ─► k₅₅ ─► k₇₈ ───► nil @@ -47,152 +78,462 @@ pointer swaps instead of locks). → ~log₂(n) hops instead of n ``` -tidesdb's `skip_list.c` also shows the allocation strategy fjall's Rust -hides: an **arena bump allocator** — one big malloc'd slab, and each insert -just bumps a pointer forward. No per-node free; the whole arena dies when -the memtable is flushed. Cheap allocation, and it makes the "memtable size -limit" check a single pointer comparison. +At this topic's scale that matters concretely: a 64 MiB memtable of 100-byte +records holds ~671,000 entries, so a skip-list probe is log₂(671,000) ≈ **20 +hops** against 671,000 for a linear list. + +`src/skip_list.c` also shows the allocation strategy fjall's Rust hides: an +**arena allocator** — big slabs are allocated up front and each insert bumps a +pointer forward, with no per-node `free`; the whole arena dies when the memtable +is flushed. tidesdb goes one step further and shards the arenas per thread, +caching slot assignments in thread-local storage (`src/skip_list.c:22–42`, +including the comment explaining that a small *set* of cached slots beats a +single one when a thread interleaves writes across arenas). Cheap allocation, +and it makes the memtable-size check a single comparison — +`skip_list_get_size(umt->skip_list)` at `src/tidesdb.c:29846`. ### Step 3 — the write path: a WAL batch is just bytes at an offset -tidesdb groups writes into transactions: `tidesdb_txn_put` stages each -operation in a per-transaction ops array, and `tidesdb_txn_commit` -serializes the whole batch and hands it to `block_manager_write_raw` — a -raw append of length-prefixed bytes to the log file. Only after the log -append do the ops go into the skip list (`apply_ops_to_memtable`) — the -write-ahead rule, visible as two consecutive C calls. +> **In:** a transaction with staged operations, and the write-ahead rule. +> **Out:** the three consecutive calls that implement it, with line numbers, and +> the one memory-layout decision C forces into the open. + +tidesdb groups writes into transactions: `tidesdb_txn_put` +(`src/tidesdb.c:26535`) stages each operation in a per-transaction ops array, +and `tidesdb_txn_commit` (`src/tidesdb.c:29697`) serialises the whole batch and +hands it to `block_manager_write_raw` — a raw append of length-prefixed bytes to +the log file. The write-ahead rule then appears as three calls, forty lines +apart, in commit order: + +```text +src/tidesdb.c:29796 block_manager_write_raw(umt->wal, uwal_batch, uwal_size) + ↑ the WAL append. Comment at :29792 says it uses a raw + write "to avoid malloc/memcpy/free per commit" + +src/tidesdb.c:29814 if (config.unified_memtable_sync_mode == TDB_SYNC_FULL) + tidesdb_unified_wal_group_sync(...) + ↑ the fsync, and only under one of three sync modes. + Comment at :29811: "group-commit durability -- one + fdatasync per batch of concurrent committers" + +src/tidesdb.c:29837 tidesdb_txn_apply_ops_to_unified_memtable(txn, umt->skip_list) + ↑ ONLY NOW does the write become visible in RAM +``` + +That ordering is the entire durability contract, and unlike fjall — where it is +implied by holding a `MutexGuard` across two method calls — here you can point at +the three lines. Note the three sync modes named on line 29814's comment: +`TDB_SYNC_FULL` (fdatasync per commit batch), `TDB_SYNC_INTERVAL` (a background +sync worker), `TDB_SYNC_NONE` (skip). That is the same knob as fjall's +`PersistMode`, and the same knob you must equalise before comparing engines. One detail the C makes load-bearing and explicit: **key and value share one -malloc** (`tidesdb.c:26579`): `op->value = op->key + key_size` — the value -pointer is just the key pointer plus an offset. Layout as pointer -arithmetic. The Rust equivalent would be a single `Box<[u8]>` with split -indices; here you *see* that one allocation per op is a deliberate -throughput decision, not an accident. +malloc**, and the source says why: + +```c +// src/tidesdb.c at tidesdb/tidesdb@810507a — inside tidesdb_txn_put, +// lines 26579-26590. The comment is the source's own. +26579 /*** we coalesce key+value into a single allocation to halve malloc pressure +26580 ** op->value points into the same buffer at offset key_size +26581 * only op->key should be freed (it owns the entire buffer) */ +26582 const size_t kv_alloc_size = key_size + (value_size > 0 ? value_size : 0); +26583 op->key = malloc(kv_alloc_size); +26584 if (!op->key) return TDB_ERR_MEMORY; +26585 memcpy(op->key, key, key_size); +26586 op->key_size = key_size; +26587 +26588 if (value_size > 0) +26589 { +26590 op->value = op->key + key_size; +``` -Cost, same as fjall: every byte is written twice (log now, SSTable later), -and commit latency is the fsync policy on the log. +Line 26590 is layout as pointer arithmetic: the value pointer is the key pointer +plus an offset. The Rust equivalent would be a single `Box<[u8]>` with split +indices; here you *see* that one allocation per op — instead of two — is a +deliberate throughput decision, and that the ownership rule it creates ("only +`op->key` should be freed") has to be maintained by comment rather than by the +type system. + +Cost, same as fjall: every byte is written twice (log now, SSTable later), and +commit latency is the sync mode on the log. ### Step 4 — the SSTable made explicit: build the bloom, write the offsets -When the memtable is over threshold, a worker (`tidesdb_flush_memtable`) -walks the skip list in key order and writes an SSTable: compressed blocks of -sorted key-value pairs, a **block index** (an array of "first key → byte -offset in this file" entries), and a **bloom filter** (a bit array set by k -hash functions; ~10 bits/key gives ~1% false positives on "is this key maybe -in this file?"). - -In fjall both helpers are inside the `lsm-tree` crate; here you can read -them end to end. `bloom_filter.c` is ~600 lines — the hash mixing, the bit -math, all of it. And the block index returns **raw file offsets** -(`tidesdb.c:9835`): the reader binary-searches a struct array and then -`seek()`s to a byte position. No cursor abstraction — the disk format *is* -the data structure. That is what "immutable sorted file" actually means at -the bottom: a byte layout you can compute offsets into. +> **In:** a full memtable and a file to write it into. +> **Out:** what an SSTable is at byte level, and the bloom sizing formula +> evaluated on this topic's numbers — from the source's own code, not a +> remembered rule of thumb. + +When the memtable is over threshold, a worker (`tidesdb_flush_memtable`, +`src/tidesdb.c:24887`) walks the skip list in key order and writes an SSTable: +compressed blocks of sorted key-value pairs, a **block index** (an array of +"first key → byte offset in this file" entries), and a **bloom filter**. + +In fjall both helpers are inside the `lsm-tree` crate. Here `src/bloom_filter.c` +is 624 lines and you can read all of it — the hash mixing (murmur-family prime +at `:37`, a v2 hash that "appends a murmur3 fmix32 finalizer so short keys fully +avalanche", `:39–43`), the packed 64-bit bitset macros (`:29–33`), the +serialisation format with its version sentinel (`:46–59`), and the sizing: + +```c +// src/bloom_filter.c at tidesdb/tidesdb@810507a — bloom_filter_new, lines +// 203-223. p is the target false-positive rate, n the expected key count. +203 /**** we calculate the size of the bitset (m) using the formula +204 *** m = -n * ln(p) / (ln(2)^2) +205 ** +206 */ +207 const double m_double = ceil(-((double)n) * log(p) / (M_LN2 * M_LN2)); +217 (*bf)->m = (unsigned int)m_double; +219 /* we calculate the number of hash functions (h) using the formula +220 * h = (m / n) * ln(2) +221 * +222 */ +223 const double h_double = ceil(((double)(*bf)->m) / n * M_LN2); +``` + +Work it for one flushed memtable at this topic's shape — n = 671,000 records +(64 MiB / 100 B), target p = 1%: + +```text + m = ceil(-671000 · ln(0.01) / (ln 2)²) + = ceil(671000 · 4.6052 / 0.4805) + = 6,431,575 bits = 804 KB of filter ⇒ 9.59 bits per key + h = ceil((m/n) · ln 2) = ceil(9.59 × 0.6931) = ceil(6.65) = 7 hash functions +``` + +So the folk figure "about 10 bits per key for 1%" is *derived*, not assumed — +and 7 is exactly the range the source's own comment at `:225` calls typical +("typical real-world values are 7-15"). + +The block index is the other half, and it returns **raw file offsets**: + +```c +// src/tidesdb.c at tidesdb/tidesdb@810507a — inside tidesdb_sstable_get, +// lines 9832-9837. There is no cursor abstraction; the lookup produces a +// byte position that a seek() consumes directly. +9832 if (sst->block_indexes && sst->block_indexes->count > 0) +9833 { +9834 int64_t start_slot = 0; +9835 if (compact_block_index_find_slot(sst->block_indexes, key, key_size, &start_slot) == 0) +9836 { +9837 start_file_position = sst->block_indexes->file_positions[start_slot]; +``` + +That is what "immutable sorted file" actually means at the bottom: a byte layout +you can compute offsets into. Note also the honesty in the surrounding comment +(`:9838–9840`): the prefix index is *lossy*, so keys sharing a long prefix span +several blocks with identical min/max prefixes and the lookup must walk a run — +a real complication that a `BTreeMap` API would have hidden from you entirely. ### Step 5 — the read path: every potential miss, one function per stop +> **In:** a key that could be hiding in any of five kinds of place. +> **Out:** read amplification as a literal for-loop, with the line number of +> each stop and the arithmetic of what the bloom `continue` saves. + A read must check every place a newer version of the key could hide, -newest-first, and return the first hit. tidesdb performs each stop as a -separate, named function call — the read path *is* the topic README's §1 LSM -read diagram, one function per box. In pseudo-Rust: +newest-first, and return the first hit. tidesdb performs each stop as a separate, +named call — the read path *is* the topic README's LSM read diagram, one +function per box: ```rust -fn get(&self, key: &[u8]) -> Option { - if let Some(v) = self.txn_write_set.get(key) { return Some(v); } // own writes first - if let Some(v) = self.active_memtable.get(key) { return Some(v); } - for mt in self.immutable_memtables.newest_first() { // refcount-pinned - if let Some(v) = mt.get(key) { return Some(v); } - } - for level in &self.levels { - for sst in level.newest_first() { - if !sst.bloom.might_contain(key) { continue; } // skips MOST absent-key IO - let off = sst.block_index.binary_search(key)?; // a raw file offset — - if let Some(v) = sst.read_block_at(off).find(key) { // the disk format IS - return Some(v); // the data structure - } - } - } - None // read amp made concrete: every stop above was a potential miss -} +// ILLUSTRATION — pseudo-Rust for tidesdb's C read path. Each line names the +// real anchor; read them in order at tidesdb/tidesdb@810507a. +1 fn get(&self, key: &[u8]) -> Option { +2 // src/tidesdb.c:26672 — your own uncommitted writes first, via a hash +3 // table for large transactions (linear reverse scan for small ones) +4 if let Some(v) = self.txn_write_set.get(key) { return Some(v); } +5 +6 // src/tidesdb.c:26808 — skip_list_get_with_seq_ref on the ACTIVE memtable, +7 // taken under tidesdb_active_memtable_try_ref (:26804) so a rotation +8 // cannot swap it out mid-probe +9 if let Some(v) = self.active_memtable.get(key) { return Some(v); } +10 +11 // src/tidesdb.c:26845 — immutable memtables, newest first. The comment +12 // there spells out the invariant: pointers snapshotted under one rwlock, +13 // each immutable pinned by refcount "so a concurrent flush-worker +14 // eviction cannot free one out from under the scan" +15 for mt in self.immutable_memtables.newest_first() { +16 if let Some(v) = mt.get(key) { return Some(v); } +17 } +18 +19 // src/tidesdb.c:9756 — tidesdb_sstable_get, once per SSTable per level +20 for level in &self.levels { +21 for sst in level.newest_first() { +22 // src/tidesdb.c:9810 — bloom check; skips MOST absent-key IO. +23 // Note skip_bloom at :9808: redundant when an L1+ boundary +24 // search already identified this file +25 if !sst.bloom.might_contain(key) { continue; } +26 // src/tidesdb.c:9835 — block index → a raw file offset (:9837) +27 let off = sst.block_index.find_slot(key)?; +28 if let Some(v) = sst.read_block_at(off).find(key) { return Some(v); } +29 } +30 } +31 None // read amp made concrete: every stop above was a potential miss +32 } ``` -Count the stops: write set, active memtable, N immutable memtables, then -per level per SSTable a bloom check and maybe one block read. That count is -**read amplification** as a for-loop — and the bloom `continue` is the line -that keeps it affordable (1% false positives ⇒ ~0.2 block reads for an -absent key across 20 SSTables, instead of 20). +Count the stops: write set, active memtable, N immutable memtables, then per +level per SSTable a bloom check and maybe one block read. That count *is* **read +amplification** — the number of places consulted per lookup, against the one +that holds the answer. The bloom `continue` on line 25 is what keeps it +affordable: with 20 SSTables and the 1% filter sized in Step 4, a lookup for an +absent key does 20 × 0.01 = **0.2 expected block reads** instead of 20. ### Step 6 — rotation and compaction: the concurrency is hand-rolled -Two mutation streams run concurrently with reads: memtable **rotation** -(swap a full memtable for a fresh one, hand the full one to the flush -worker) and **compaction** (merge SSTables within/between levels to bound -read amplification and drop shadowed versions). Both need object-lifetime -guarantees — a reader mid-lookup must not have its memtable freed under it. +> **In:** two background mutation streams — rotation and compaction — running +> against live readers. +> **Out:** the exact atomics that make that safe, and what `Arc` was doing for +> you in fjall. + +Two mutation streams run concurrently with reads: memtable **rotation** (swap a +full memtable for a fresh one, hand the full one to the flush worker) and +**compaction** (merge SSTables within and between levels to bound read +amplification and drop shadowed versions). Both need object-lifetime guarantees +— a reader mid-lookup must not have its memtable freed underneath it. + +fjall gets this from `Arc` for free. tidesdb writes it out, with the memory +ordering visible on every line: + +```c +// src/tidesdb.c at tidesdb/tidesdb@810507a — the tail of tidesdb_txn_commit, +// lines 29846-29856. Read the ordering arguments, not just the calls. +29846 const size_t umt_size = (size_t)skip_list_get_size(umt->skip_list); +29847 atomic_fetch_sub_explicit(&umt->writers, 1, memory_order_release); +29848 atomic_fetch_sub_explicit(&umt->refcount, 1, memory_order_release); +29850 if (umt_size >= txn->db->unified_mt.write_buffer_size) +29851 { +29852 /** CAS-based admission, only one thread enters rotation at a time +29853 * same lock-free pattern as per-CF flush in tidesdb_flush_memtable_internal */ +29854 int expected = 0; +29855 if (atomic_compare_exchange_strong_explicit(&txn->db->unified_mt.is_flushing, &expected, +29856 1, memory_order_acquire, +``` -fjall gets this from `Arc` for free. tidesdb writes it out: memtables carry -atomic **refcounts**, and rotation uses a CAS (compare-and-swap) loop with -**memory ordering spelled out** (`tidesdb.c:29761`): -`memory_order_acq_rel` on the memtable refcount during rotation. Rust's -`Arc` hides exactly these barriers — topic 9 makes you write them yourself. +Three things are explicit here that Rust would have made invisible. The +refcount and writer-count decrements on lines 29847–29848 are +`memory_order_release`, which publishes this committer's skip-list writes to +whoever later acquires. The rotation admission on line 29855 is a +compare-and-swap with `memory_order_acquire`, so exactly one thread wins and it +sees everything the releasing writers did. And the *acquire* side of the reader +path is `tidesdb_active_memtable_try_ref` (`src/tidesdb.c:29761`, and again at +`:26804` on the read path), which loops with a bounded attempt count — +`TDB_ACTIVE_REF_MAX_ATTEMPTS` — rather than blocking. Rust's `Arc` hides exactly +these barriers; topic 9 makes you write them yourself. Compaction scheduling is equally visible: -- After a flush, if a level is over capacity, work is enqueued (`tidesdb.c:19910`). -- Queued work is deduplicated via a CAS `is_compacting` flag - (`tidesdb_enqueue_compaction`, `tidesdb.c:25366`) — and the merge geometry - is computed at *dequeue* time, not enqueue, so it reflects current state. -- The worker picks which L_i → L_{i+1} merge to run by SSTable counts - (`tidesdb.c:20143`). - -Cost, same trade as every LSM: background write amplification purchased to -keep the Step 5 for-loop short. +- After a flush, if the level geometry demands it, work is enqueued — + `tidesdb_enqueue_compaction(cf, 0)` at `src/tidesdb.c:19918`, under a comment + calling it an "auto-compaction trigger -- geometry-driven, not a full merge". + The sibling branch at `:19910` steers a key range straight to the bottom level + instead. +- The enqueue itself (`src/tidesdb.c:25366`) deduplicates via an `is_compacting` + flag, and the blocking variant at `:25403` falls through to it. The merge + geometry is computed at *dequeue* time, not enqueue, so it reflects current + state. +- `tidesdb_compaction_worker_thread` (`src/tidesdb.c:20143`) is the worker + entry point; its header comment (`:20139–20141`) states the concurrency rule: + "the `is_compacting` flag ensures only one compaction per CF at a time, but + multiple workers can compact different CFs concurrently." + +Cost, the same trade as every LSM: background write amplification purchased to +keep the Step 5 for-loop short. [FINDINGS.md](../../FINDINGS.md) row 1 is what +that trade is worth on this topic's workload — an LSM at 0.45× space +amplification against a copy-on-write B-tree at 63.28×, a 140× spread on the +same 108 MB of records. ## Where each step lives in the code -| File | Role (steps) | -|------|------| -| `tidesdb.c` (~38K lines) | the whole engine: write/read/compaction orchestration (3, 5, 6) | -| `skip_list.c` | memtable — lock-free skip list, arena bump allocator (2) | -| `block_manager.c` | physical block IO (WAL + SSTs) (3, 4) | -| `bloom_filter.c` | ~600 lines, readable bloom filter (4) | -| `manifest.c` | level metadata: which SST is in which level (6) | - -**Write path (steps 2–4), file:line** - -``` -tidesdb_txn_put tidesdb.c:26535 stage in per-txn ops array -tidesdb_txn_commit tidesdb.c:29780 serialize WAL batch → block_manager_write_raw -apply_ops_to_memtable tidesdb.c:29837 skip-list inserts (atomic refcounts) -rotate check (CAS loop) tidesdb.c:29850 memtable over threshold → rotate -tidesdb_flush_memtable tidesdb.c:24887 worker serializes skip list → compressed SST +| File | Lines | Role (steps) | +|------|-------|------| +| `src/tidesdb.c` | 37,702 | the whole engine: write/read/compaction orchestration (3, 5, 6) | +| `src/skip_list.c` | 2,929 | memtable — skip list, per-thread arena allocator (2) | +| `src/tidesdb.h` | 2,152 | every public type and tunable | +| `src/block_manager.c` | 2,004 | physical block IO (WAL frames + SST blocks) (3, 4) | +| `src/manifest.c` | 923 | level metadata: which SST is in which level (6) | +| `src/bloom_filter.c` | 624 | the whole filter, sizing math included (4) | + +**Write path (steps 2–4)** — all in `src/tidesdb.c`: + +```text +tidesdb_txn_put 26535 stage in per-txn ops array + coalesced key+value malloc 26579 one allocation, value at +key_size +tidesdb_txn_commit 29697 serialize the batch + block_manager_write_raw (WAL) 29796 raw framed append + group fdatasync (TDB_SYNC_FULL only) 29814 one sync per committer batch + apply_ops_to_unified_memtable 29837 skip-list inserts + refcount/writers release 29847 memory_order_release + rotation check + CAS admission 29850 size >= write_buffer_size +tidesdb_flush_memtable 24887 worker: skip list → compressed SST ``` -**Read path (step 5), file:line** +**Read path (step 5)** — `src/tidesdb.c`: -``` -txn write-set check tidesdb.c:26672 your own uncommitted writes first -active memtable tidesdb.c:26808 skip_list_get_with_seq_ref -immutable memtables tidesdb.c:26845 newest-first, refcount-protected -tidesdb_sstable_get tidesdb.c:9756 per level: bloom (9810) → block index - binary search (9832) → scan blocks +```text +txn write-set check 26672 your own uncommitted writes first +active memtable try_ref 26804 pin it before probing + skip_list_get_with_seq_ref 26808 the probe itself +immutable memtables 26845 newest-first, refcount-protected +tidesdb_sstable_get 9756 per level, per SSTable + bloom check (skippable) 9810 the line that bounds read amp + block index find_slot 9835 → raw file offset at 9837 ``` -**Compaction (step 6)**: enqueue at `tidesdb.c:19910`, CAS dedup at -`tidesdb.c:25366`, level-pick at `tidesdb.c:20143`. The three -"C makes it visible" anchors from the steps, collected: one-malloc key+value -`tidesdb.c:26579` (step 3), `memory_order_acq_rel` refcount `tidesdb.c:29761` -(step 6), raw-offset block index `tidesdb.c:9835` (step 4). +**Compaction (step 6)** — `src/tidesdb.c`: trigger at `19918` (steer-to-bottom +branch at `19910`), enqueue + dedup at `25366`, worker thread at `20143`. + +**The three "C makes it visible" anchors**, collected: one-malloc key+value at +`src/tidesdb.c:26579–26590` (step 3), the release/acquire pair around rotation at +`src/tidesdb.c:29847–29856` (step 6), and the raw-offset block index at +`src/tidesdb.c:9835–9837` (step 4). + +## Questions to answer in notes.md + +Each needs the source open. `python3 tools/pinned-source.py show tidesdb +src/tidesdb.c -r A:B` is the fastest way in. + +1. Read `src/tidesdb.c:29792–29837` and identify the exact window during which an + acknowledged write exists in the WAL but not in the memtable. What does a + concurrent reader at `:26808` see during that window, and is that a bug? Name + the field that decides. +2. `src/tidesdb.c:29814` only calls `tidesdb_unified_wal_group_sync` when the + sync mode is `TDB_SYNC_FULL`. Find the other two modes in `src/tidesdb.h`, + and say for each one exactly what is lost on power failure versus process + crash. Then say which mode you would have to select to make a fair comparison + against fjall's `PersistMode::SyncAll`. +3. `src/bloom_filter.c:207` computes `m = ceil(-n·ln(p)/(ln 2)²)` and `:223` + computes `h = ceil((m/n)·ln 2)`. Evaluate both for p = 0.001 at + n = 671,000, compare the filter size against the p = 0.01 case, and say what + that extra memory buys you in expected block reads across a 20-SSTable level. +4. The block-index comment at `src/tidesdb.c:9838–9840` says the prefix index is + *lossy* and a lookup may have to walk a run of blocks. Construct a key + distribution that makes that run long, and say which of this repo's + generators would produce it. What does that do to the Step 5 read-amp count? +5. Compare `src/tidesdb.c:29847–29856` with fjall's + `src/keyspace/mod.rs:940–947`. Both rotate a full memtable. List every + guarantee tidesdb states with an explicit `memory_order_*` argument that + fjall gets from `Arc` and `MutexGuard` — and name one guarantee that is + *harder* to see in the Rust version because of that. ## Done when -You've matched each fjall concept (journal, memtable, rotation, bloom, level) to its -C twin and noticed the abstractions Rust buys you — and what they hide. +Answer each before unfolding it. + +- [ ] You can match each fjall concept — journal, memtable, rotation, bloom, level metadata — to its tidesdb twin, with a file for each. + +
+Answer + +| fjall | tidesdb | +|---|---| +| journal (`src/journal/writer.rs`) | WAL frames via `src/block_manager.c`, appended at `src/tidesdb.c:29796` | +| `PersistMode::{Buffer,SyncData,SyncAll}` | `TDB_SYNC_{NONE,INTERVAL,FULL}`, branched at `src/tidesdb.c:29814` | +| memtable (skip list inside `lsm-tree`) | `src/skip_list.c`, with a per-thread arena allocator (`:22–42`) | +| `Keyspace` | column family (`tidesdb_column_family_t`) | +| rotation (`inner_rotate_memtable`, `mod.rs:727`) | CAS admission at `src/tidesdb.c:29850–29856` | +| bloom policy (`options.rs:108`) | `bloom_filter_new(bf, p, n)`, `src/bloom_filter.c:188` | +| segment / SST | SSTable written by `tidesdb_flush_memtable`, `src/tidesdb.c:24887` | +| level metadata (inside `lsm-tree`) | `src/manifest.c` | +| `snapshot_tracker` seqno watermark | per-memtable atomic refcounts + `try_ref`, `src/tidesdb.c:26804` | + +
+ +- [ ] You can point at the three consecutive lines that implement the write-ahead rule, and say what would break if two of them swapped. + +
+Answer + +`src/tidesdb.c:29796` (WAL append), `:29814` (group fdatasync, `TDB_SYNC_FULL` +only), `:29837` (apply to the skip list). If the memtable apply moved *before* +the WAL append, a crash between them would leave a write that was visible to +readers — possibly read and acted on — but absent from the log, so replay would +silently lose it. That is the whole content of "write-ahead": the log must be +the superset. The `fdatasync` on `:29814` is the separate question of whether +the log's bytes have actually reached the platter; moving it after `:29837` +would not break correctness of replay, only shrink the durability window. + +
+ +- [ ] You can derive a bloom filter's size and hash count from a target false-positive rate, using the source's formulas, and say what it buys in read amplification. + +
+Answer + +`src/bloom_filter.c:207`: `m = ceil(-n·ln(p) / (ln 2)²)`; `:223`: +`h = ceil((m/n)·ln 2)`. + +For one 64 MiB memtable of 100-byte records — n = 671,000 — at p = 1%: +`m = ceil(671000 × 4.6052 / 0.4805)` = 6,431,575 bits = **804 KB**, i.e. 9.59 +bits per key, and `h = ceil(9.59 × 0.6931)` = **7 hash functions** (inside the +source's own "typical real-world values are 7-15" range at `:225`). + +What it buys: at Step 5's `continue` on line 25 of the read-path sketch, a +lookup for an absent key across 20 SSTables costs 20 × 0.01 = **0.2 expected +block reads** instead of 20 — a 100× reduction in read amplification for 804 KB +per file. + +
+ +- [ ] You can name at least three things Rust's abstractions were doing for you that this codebase does by hand — and one thing the C makes clearer. + +
+Answer + +Hidden by Rust: (1) **lifetime pinning** — `Arc` versus tidesdb's explicit atomic +refcounts and `tidesdb_active_memtable_try_ref` with a bounded retry count +(`src/tidesdb.c:26804`, `:29761`); (2) **memory ordering** — every +`memory_order_release`/`acquire` at `src/tidesdb.c:29847–29856` is implicit in +`Arc`'s and `Mutex`'s internals; (3) **allocation and ownership** — the +coalesced key+value buffer at `:26579–26590`, where "only `op->key` should be +freed" is enforced by a comment rather than by `Box`. + +Clearer in C: the *ordering* of the durability contract. In fjall the +write-ahead rule is implied by holding a `MutexGuard` across two method calls +(`src/keyspace/mod.rs:919–944`); in tidesdb it is three numbered lines you can +point at, and the fsync mode is a visible branch rather than an enum argument +threaded through a config struct. + +
+ +- [ ] You can explain how a reader mid-lookup is protected from a concurrent flush or compaction, and where the mechanism is written. + +
+Answer + +By refcount pinning, stated in the source's own comment at +`src/tidesdb.c:26845–26848`: immutable-memtable pointers are snapshotted under a +single rwlock acquisition and each is pinned by refcount "so a concurrent +flush-worker eviction cannot free one out from under the scan". The active +memtable gets the same treatment via `tidesdb_active_memtable_try_ref` +(`:26804`), and the writer side releases with `memory_order_release` +(`:29847–29848`) so the reader's acquire sees a consistent skip list. + +Rotation admission is a separate CAS on `is_flushing` (`:29855`) so exactly one +thread rotates; compaction uses the same trick with `is_compacting`, one per +column family, which is why `src/tidesdb.c:20139–20141` can promise that +"multiple workers can compact different CFs concurrently". + +
## References -**Code** -- [tidesdb](https://github.com/tidesdb/tidesdb) — `tidesdb.c` (~38K - lines, the whole engine), `skip_list.c`, `block_manager.c`, - `bloom_filter.c` (~600 readable lines), `manifest.c` (shallow clone at - `~/repos/tidesdb`; skim-read, 1–2 h) +**Code** (all at `tidesdb/tidesdb@810507a` — this repo's pin table entry; +confirm with `python3 tools/pinned-source.py ref tidesdb`) +- [tidesdb](https://github.com/tidesdb/tidesdb) — `src/tidesdb.c` (37,702 lines, + the whole engine: `tidesdb_txn_put:26535`, `tidesdb_txn_commit:29697`, + WAL append `:29796`, sync `:29814`, memtable apply `:29837`, rotation + `:29850`, `tidesdb_flush_memtable:24887`, `tidesdb_sstable_get:9756`, + compaction enqueue `:25366` and worker `:20143`), `src/skip_list.c` (2,929), + `src/block_manager.c` (2,004), `src/manifest.c` (923), `src/bloom_filter.c` + (624 — `bloom_filter_new:188`, sizing at `:207` and `:223`). Skim-read, 1–2 h + +**This repo** +- [reading-fjall.md](reading-fjall.md) — the same lifecycle in Rust, with the + concepts built from sequential-vs-random IO; read it first +- [reading-rocksdb-layout.md](reading-rocksdb-layout.md) — the same lifecycle + again, industrialised, where each of these single files becomes a directory +- [FINDINGS.md](../../FINDINGS.md) row 1 — the measured LSM-vs-B-tree space + amplification (0.45× vs 63.28×) that all of this machinery exists to move; + `./verify.sh 01` diff --git a/topics/01-storage-engine-landscape/reading-turso-btree.md b/topics/01-storage-engine-landscape/reading-turso-btree.md index 5a380a8..64866a4 100644 --- a/topics/01-storage-engine-landscape/reading-turso-btree.md +++ b/topics/01-storage-engine-landscape/reading-turso-btree.md @@ -1,63 +1,121 @@ # Turso's B-tree: the canonical page engine, in Rust turso re-implements the SQLite file format, so this is a reading of *the* -canonical page-oriented engine — with Rust types instead of C macros. It is -the B-tree protagonist opposite fjall's LSM. Before touching the code, this -chapter builds the machine step by step: why pages exist, how a tree of -pages finds a row, how one page stores variable-length rows, what one insert -does, and how the whole thing survives a crash. Then it hands you the file -and line anchors to watch each step happen. +canonical page-oriented engine — with Rust types instead of C macros. It is the +B-tree protagonist opposite fjall's LSM. Before touching the code, this chapter +builds the machine step by step: why pages exist, how a tree of pages finds a +row, how one page stores variable-length rows, what one insert does, and how the +whole thing survives a crash. Then it hands you the file and line anchors to +watch each step happen. + +Everything below is anchored at `tursodatabase/turso@dd775bc`, this repo's pin +(`python3 tools/pinned-source.py ref turso`). Read any file at that commit with +`python3 tools/pinned-source.py show turso core/storage/btree.rs -r A:B` — that +is more reliable than a local clone, because these files move fast. ## The problem in one sentence -Store a million sorted rows on disk so that finding one costs a handful of -disk reads and inserting one doesn't rewrite the file. +Store a million sorted rows on disk so that finding one costs a handful of disk +reads and inserting one doesn't rewrite the file. ## The concepts, step by step ### Step 1 — the page: disks deal in blocks, so the engine does too -Disks and OSes transfer data in fixed-size blocks, and a crash-safe engine -wants a unit it can read, cache, and write atomically. So the database file -is an array of fixed-size **pages** (SQLite default 4 KB), and "one disk IO" -always means "one page". Every structure that follows is built out of pages -that point at each other by **page number** — a page number is disk's version -of a pointer. +> **In:** a block device that transfers fixed-size chunks and an engine that +> must survive being killed mid-write. +> **Out:** the page as the universal unit — of IO, of caching, of the atomicity +> argument — and the page number as disk's pointer. + +Disks and OSes transfer data in fixed-size blocks, and a crash-safe engine wants +a unit it can read, cache and write atomically. So the database file is an array +of fixed-size **pages** (SQLite's default is 4 KB), and "one disk IO" always +means "one page". Every structure that follows is built out of pages that point +at each other by **page number** — a page number is disk's version of a pointer, +and dereferencing one means asking the pager for that page. + +Two consequences you will meet again in this guide. First, the *page cache* is +sized in pages, not bytes: turso's default is +`DEFAULT_PAGE_CACHE_SIZE_IN_PAGES = 2000` (`core/storage/page_cache.rs:14`, with +a separate 100,000 for wasm at `:16`) — 8.2 MB at 4 KB pages. Second, the tree +has a hard depth bound derived from page arithmetic: +`BTCURSOR_MAX_DEPTH = 20` (`core/storage/btree.rs:133`), justified in the comment +above it as "a maximum database size of 2^31 pages, a minimum fanout of 2 for a +root-node and 3 for all other internal nodes". Anything deeper is declared +corrupt rather than traversed. ### Step 2 — a tree of pages: fanout is everything +> **In:** a million sorted rows and a budget of a few page reads per lookup. +> **Out:** fanout computed from real SQLite cell sizes, the resulting height, +> and the fraction of the file that has to stay cached to make it work. + To find one row among a million with few page reads, arrange the pages as a -sorted tree. Each **interior page** holds ~50–500 separator keys and child -page numbers; each **leaf page** holds the actual rows. Because one page -holds *hundreds* of keys (not 2, like a binary tree node), the tree is -extremely flat — the height is log-base-*fanout*: +sorted tree. Each **interior page** holds separator keys and child page numbers; +each **leaf page** holds the actual rows. Because one page holds *hundreds* of +keys — not 2, like a binary tree node — the tree is extremely flat: the height +is log-base-*fanout*, not log-base-2. + +"Hundreds" is not a hand-wave; it falls out of the page header layout turso +documents at `core/storage/btree.rs:76–124`. Write **P** for page size, **H** +for header bytes (12 interior, 8 leaf — stated on line 76), **s = 2** for a cell +pointer, and **c** for a cell's own bytes. Then + +```text + fanout = floor((P − H) / (c + s)) + + Table-interior cell = 4 B child page number + varint rowid (3 B up to ~2 M) + c = 7 ⇒ (4096 − 12) / (7 + 2) = 453 children per interior page + Table-leaf cell, 100-byte row + = varint payload length (2 B) + varint rowid (3 B) + 100 B payload + c = 105 ⇒ (4096 − 8) / (105 + 2) = 38 rows per leaf page + + Index-interior cell, 16-byte key + = 4 B child + varint payload length (1 B) + 16 B key + c = 21 ⇒ (4096 − 12) / (21 + 2) = 177 children per interior page ``` - ┌────────── root (interior) ──────────┐ - │ k₅₀ → pg7 k₁₀₀ → pg8 ... ×50 │ - └──────────────────┬───────────────────┘ - ┌─────────────────────────┼──── ~50 children ────┐ - interior pg7 interior pg8 ... - (50 keys each) (50 keys each) - │ │ - leaves leaves 50×50×50 ≈ 125K leaves - × ~50 rows = millions - - 1M rows, fanout 50 → height 3-4: a point lookup touches 3-4 pages, - and the root + interiors are ~2% of the data — they stay cached. + +Now the tree for 1,000,000 rows of 100 bytes: + +```text + leaves = ceil(1,000,000 / 38) = 26,316 pages ← 107.8 MB + interior L1 = ceil( 26,316 / 453) = 59 pages + root = ceil( 59 / 453) = 1 page + ───────────────────────────────────────────────── + height 3: root → interior → leaf. A point lookup reads 3 pages. + Interior total = 60 pages = 245.8 KB = 0.23% of the file. ``` +That last line is the whole argument for why B-trees stay fast: the navigational +part of the structure is a quarter of a megabyte, so it lives in the page cache +permanently and only the final leaf read is a real IO. With turso's default +2000-page cache you hold all 60 interior pages *and* 1,940 leaves — 7.4% of the +leaf level — for 8.2 MB. + +Note the old rule of thumb "fanout ≈ 50" is far too pessimistic for a rowid +table: 50 would require a ~70-byte separator. At fanout 50 the same 26,316 +leaves need three interior levels (527 → 11 → 1) and the tree is height 4. So +**fanout is set by key size, and key size sets height** — which is exactly the +lever topic 3 measures. Its worked table for other key/value shapes is in +[topics/03-btree-internals/notes.md](../03-btree-internals/notes.md); topic 3's +own headline is that height stopping at 3 does *not* stop lookups getting +slower, because cache residency, not height, sets what a page touch costs. + That is the entire reason B-trees won: **the tree's shape is dictated by the page size**, so the memory hierarchy's block transfers are never wasted. ### Step 3 — inside one page: the slotted layout -A leaf must hold *variable-length* rows, keep them *sorted*, and absorb -inserts and deletes *in place*. Storing rows back-to-back fails: inserting in -the middle would shift everything. The fix is one level of indirection — a -**slotted page**: +> **In:** one 4 KB page that must hold variable-length rows, keep them sorted, +> and absorb inserts and deletes in place. +> **Out:** the slotted-page layout, the header fields that implement it, and the +> reason B-trees have space amplification. -``` +Storing rows back-to-back fails: inserting in the middle would shift everything +after it. The fix is one level of indirection — a **slotted page**: + +```text ┌────────────┬──────────────────────┬────────────┬─────────────────┐ │ header │ cell pointer array │ free space │ cell content │ │ 8/12 bytes │ u16 offsets, →grows │ │ ←grows, actual │ @@ -66,98 +124,225 @@ the middle would shift everything. The fix is one level of indirection — a two regions grow toward each other; a "full" page = they meet ``` -- The rows ("**cells**") are written wherever there's room, from the right. -- A small array of 2-byte offsets at the front — the **pointer array** — is - kept in sorted-key order. Sorting means moving 2-byte pointers, never the - rows themselves; binary search runs over the pointer array. -- Delete = remove the pointer, *leave the bytes*. The dead bytes are - reclaimed lazily ("defragmentation") only when space runs out. +- The rows ("**cells**") are written wherever there's room, from the right. The + source says why in as many words at `core/storage/btree.rs:112–114`: "SQLite + strives to place cells as far toward the end of the b-tree page as it can, in + order to leave space for future growth of the cell pointer array." +- A small array of 2-byte offsets at the front — the **cell pointer array** — is + kept in sorted-key order. Sorting means moving 2-byte pointers, never the rows + themselves; binary search runs over the pointer array. +- Delete = remove the pointer, *leave the bytes*. The dead bytes are reclaimed + lazily ("defragmentation") only when space runs out. + +turso spells the header out field by field, with an ASCII diagram, in the +`offset` module — read it, it is the file format in twenty lines: + +```rust +// core/storage/btree.rs at tursodatabase/turso@dd775bc, lines 76-124 +// (constants only; the doc comments on each are worth reading in full) +76 /// The B-Tree page header is 12 bytes for interior pages and 8 bytes for leaf pages. +84 pub mod offset { +86 pub const BTREE_PAGE_TYPE: usize = 0; // u8 +98 pub const BTREE_FIRST_FREEBLOCK: usize = 1; // u16 — head of the freeblock chain +101 pub const BTREE_CELL_COUNT: usize = 3; // u16 — how many pointers in the array +115 pub const BTREE_CELL_CONTENT_AREA: usize = 5; // u16 — where content starts (moves LEFT) +120 pub const BTREE_FRAGMENTED_BYTES_COUNT: usize = 7; // u8 +123 pub const BTREE_RIGHTMOST_PTR: usize = 8; // u32 — interior pages only +124 } +``` -turso draws this exact diagram in the source at `core/storage/btree.rs:76–124`. -This layout is also why B-trees have space amplification: the free gap in -the middle of every page is the price of in-place insertion. +Two of those fields exist purely to manage the dead space deletes leave behind, +and the doc comments define them precisely. A **freeblock** +(`BTREE_FIRST_FREEBLOCK`, comment at `:90–97`) is a run of **at least 4 bytes** +inside the cell content area that is no longer in use, chained to the next one — +explicitly *not* the regular free gap in the middle of the page. **Fragments** +(`BTREE_FRAGMENTED_BYTES_COUNT`, `:119`) are "isolated groups of 1, 2, or 3 +unused bytes" — too small to be worth chaining, so they are merely counted. +When the counter or the chain gets bad enough, `defragment_page()` +(`core/storage/btree.rs:8422`) compacts the content area. + +This layout is also why B-trees have space amplification: the free gap in the +middle of every page, plus the freeblocks and fragments, is the price of +in-place insertion. [FINDINGS.md](../../FINDINGS.md) row 1 is that price +measured on this topic's workload — the same 108 MB of records occupies 48 MB +under fjall's LSM (space amp **0.45×**) and 6.8 GB under redb's copy-on-write +B-tree (**63.28×**), a **140× spread**. redb is not turso, and the mechanism +there is copy-on-write rather than slotted-page slack (per-batch commits copy +every page on the root path), but the direction is the same one this layout +sets up: the in-place family spends space to buy in-place updates. ### Step 4 — one insert, mechanically +> **In:** a cell to add, and a leaf page with its two regions. +> **Out:** the four moves that make the common case dirty exactly one page, and +> the single condition that escalates it. + With Steps 1–3, an insert into a leaf is four small moves: ```rust -fn insert_cell(page: &mut Page, idx: usize, cell: &[u8]) -> Result<(), Full> { - let ptrs_end = page.header_len() + 2 * (page.ncells + 1); // ptr array grows → - let content_start = page.content_start - cell.len(); // content grows ← - if content_start < ptrs_end { - return Err(Full); // regions met: time to balance/split - } - page.buf[content_start..content_start + cell.len()].copy_from_slice(cell); - page.shift_pointers_right(idx); // open slot idx — keys stay sorted - page.write_u16(page.ptr_slot(idx), content_start as u16); - page.ncells += 1; - page.content_start = content_start; - Ok(()) -} -// delete = remove the u16 pointer, LEAVE the bytes → fragmentation, -// reclaimed only by defragment_page() — cheap deletes, deferred cleanup +// ILLUSTRATION — the shape of turso's insert, not its source. The real path is +// insert() core/storage/btree.rs:5779 → insert_into_page() :2568 → +// insert_into_cell() :8669; the overflow branch is balance() :2793. +1 fn insert_cell(page: &mut Page, idx: usize, cell: &[u8]) -> Result<(), Full> { +2 let ptrs_end = page.header_len() + 2 * (page.ncells + 1); // ptr array grows → +3 let content_start = page.content_start - cell.len(); // content grows ← +4 if content_start < ptrs_end { +5 return Err(Full); // regions met: time to balance/split +6 } +7 page.buf[content_start..content_start + cell.len()].copy_from_slice(cell); +8 page.shift_pointers_right(idx); // open slot idx — keys stay sorted +9 page.write_u16(page.ptr_slot(idx), content_start as u16); +10 page.ncells += 1; +11 page.content_start = content_start; // BTREE_CELL_CONTENT_AREA, offset 5 +12 Ok(()) +13 } +14 // delete = remove the u16 pointer, LEAVE the bytes → freeblocks + fragments, +15 // reclaimed only by defragment_page() (btree.rs:8422) — cheap deletes, +16 // deferred cleanup. The mirror of line 8 is shift_pointers_left() (:9067), +17 // which is a single copy_within over the 2-byte pointers. ``` -The common case dirties exactly **one page**. `Err(Full)` is the interesting -case — Step 5. +Line 8 is the payoff of the whole layout: keeping the page sorted costs a +`copy_within` over 2-byte pointers, never a move of the records. Line 4 is the +only branch that can escalate — the common case dirties exactly **one page**. +`Err(Full)` is the interesting case, and it is Step 5. ### Step 5 — when the page is full: balance, not naive split -The textbook answer is: split the full page into two half-full pages and add -a separator key to the parent. That works but leaves pages 50% full — space +> **In:** a leaf whose two regions have met. +> **Out:** why SQLite redistributes across siblings instead of splitting, the +> two constants that bound the operation, and the resulting gradient of dirty +> pages per insert. + +The textbook answer is: split the full page into two half-full pages and add a +separator key to the parent. That works, but it leaves pages 50% full — space amplification and a deeper tree. -SQLite (and turso, in `balance_non_root()`) does better: take the full page -**and up to two siblings**, pool all their cells, and redistribute them -evenly across the (possibly one more) pages. Fewer, fuller pages ⇒ shallower -tree. The costs to notice: a balance dirties ~3 pages instead of 1, and in -the rare worst case a split propagates upward until the root itself splits -(`balance_root()`) — the only operation that makes the tree taller. +SQLite, and turso in `balance_non_root()` (`core/storage/btree.rs:2995`), does +better: take the full page **and up to two siblings**, pool all their cells, and +redistribute them evenly across the resulting pages. The bound is a named +constant — `MAX_SIBLING_PAGES_TO_BALANCE: usize = 3` +(`core/storage/btree.rs:136`) — and so is its consequence: + +```rust +// core/storage/btree.rs at tursodatabase/turso@dd775bc, lines 135-139 +135 /// Maximum number of sibling pages that balancing is performed on. +136 pub const MAX_SIBLING_PAGES_TO_BALANCE: usize = 3; +137 +138 /// We only need maximum 5 pages to balance 3 pages, because we can guarantee that cells from 3 pages will fit in 5 pages. +139 pub const MAX_NEW_SIBLING_PAGES_AFTER_BALANCE: usize = 5; +``` + +Line 138 is a proof obligation stated as a comment, and it is worth checking +against Step 2's numbers: three full leaves hold at most 3 × 38 = 114 cells; +after balancing, those 114 cells plus the incoming one spread over at most 5 +pages, i.e. 23 per page, comfortably inside the 38 a page can take. The reason +the bound is 5 rather than 4 is the divider cells that have to be pushed into +the parent. + +Fewer, fuller pages ⇒ a shallower tree and less slack per page. The costs to +notice: a balance dirties ~3 pages instead of 1, and in the rare worst case a +split propagates upward until the root itself splits — `balance_root()` +(`core/storage/btree.rs:4774`), the only operation that makes the tree taller. + +So one insert dirties: + +```text + 1 page common case — cell fits (Step 4, line 4 takes the happy path) + ~3–5 pages balance_non_root: 3 siblings pooled into ≤5, parent updated + O(height) balance_root: propagation reaches the root, tree grows a level + bounded above by BTCURSOR_MAX_DEPTH = 20 (btree.rs:133) +``` -So one insert dirties 1 page (common), ~3 pages (balance), or O(height) -pages (root split). Hold that gradient — it's question 2 below. +Hold that gradient — it is question 2 below, and it is the B-tree half of the +write-amplification story that [FINDINGS.md](../../FINDINGS.md) row 1 measures +from the other end. ### Step 6 — surviving a crash: the pager and the WAL +> **In:** an engine that writes pages in place, and a machine that can lose +> power between two of those writes. +> **Out:** the two components that fix it, the actual call sites, and one +> correction to the folk version of "the write-ahead rule". + Writing pages in place is exactly what makes crashes dangerous: die mid-write -and the old version is *gone*. Two components fix this: +and the old version is *gone*. Two components fix this. + +The **pager** (`Pager`, `core/storage/pager.rs:1335`) owns all page IO: it caches +pages in memory, hands them to the B-tree, and tracks which are **dirty** +(modified but not yet written). Reads go through `read_page()` +(`core/storage/pager.rs:3240`, cache first) or `read_page_no_cache()` (`:3185`); +`add_dirty()` (`:3412`) marks a page modified. -- The **pager** owns all page IO: it caches pages in memory, hands them to - the B-tree, and tracks which are **dirty** (modified but not yet written). -- The **WAL** (write-ahead log) is an append-only file. The rule that gives - it its name: a page's new version is appended to the WAL *before* the - database file is ever touched. Commit = the WAL append is durable. - Later, a **checkpoint** copies WAL frames back into the main file and - truncates the WAL. +The **WAL** (write-ahead log, `WalFile` at `core/storage/wal.rs:2593`) is an +append-only file. The rule that gives it its name: a page's new version is +appended to the WAL *before* the database file is ever touched. Commit = the WAL +append is durable. Later, a **checkpoint** (`core/storage/wal.rs:3795`) copies +WAL frames back into the main file — which is the only time the main file is +written at all. -The punchline for the topic's B-tree-vs-LSM framing: even the in-place -family writes out-of-place *first*, then reconciles. The difference is what -is **authoritative** — here the B-tree file is (the WAL is a temporary -patch); in an LSM the log-structured files are the database. +One correction worth making, because the previous version of this guide got it +wrong and the mistake is instructive. `add_dirty()` does *not* write to the WAL. +Read it: + +```rust +// core/storage/pager.rs at tursodatabase/turso@dd775bc, lines 3412-3420 +3412 pub fn add_dirty(&self, page: &Page) -> Result<()> { +3413 turso_assert!( +3414 page.is_loaded(), +3415 "page must be loaded in add_dirty() so its contents can be subjournaled", +3416 { "page_id": page.get().id } +3417 ); +3418 self.subjournal_page_if_required(page)?; +3419 let mut dirty_pages = self.dirty_pages.write(); +3420 dirty_pages.insert(page.get().id as u32); +``` + +Line 3418 writes the page's *pre-image* to a **subjournal** +(`core/storage/subjournal.rs`, held at `pager.rs:1357`), which exists so a +`SAVEPOINT` or a failed statement can be rolled back *within* an open +transaction. That is a different mechanism from the WAL, with a different +lifetime. The WAL frames are appended later, on the commit path: `cacheflush()` +(`core/storage/pager.rs:3451`) collects the dirty set and calls +`wal.append_frames_vectored(pages, page_sz)` at `pager.rs:3704` (and again at +`:3901`), landing in `core/storage/wal.rs:4333`. So there are *two* journals +here — one for intra-transaction rollback, one for crash recovery — and reading +`add_dirty()` as "the write-ahead rule, visible in code" conflates them. + +The punchline for the topic's B-tree-vs-LSM framing survives the correction, and +is in fact sharper for it: even the in-place family writes out-of-place *first*, +then reconciles. The difference is what is **authoritative** — here the B-tree +file is (the WAL is a temporary patch that `checkpoint()` folds back in); in an +LSM the log-structured files *are* the database and nothing is ever folded back. ## Where each step lives in the code -These files are huge and move fast — expect line-number drift, navigate by -symbol name. - -| File | Size | Role (steps) | -|------|------|------| -| `core/storage/btree.rs` | ~13K lines | cursor, slotted pages, balance (2–5) | -| `core/storage/pager.rs` | ~6.6K lines | page cache, dirty tracking, IO (6) | -| `core/storage/wal.rs` | ~10K lines | WAL frames + checkpoint (6) | -| `core/storage/page_cache.rs` | — | SIEVE-eviction page cache (6) | - -- **Step 3 in code**: the layout diagram at `btree.rs:76–124`; cell parsing in - `read_btree_cell()` — `core/storage/sqlite3_ondisk.rs:816`; delete - fragmentation fixed by `defragment_page()` — `btree.rs:8422`; pointer-array - maintenance via `copy_within` in `shift_pointers_left()` — `btree.rs:9067`. -- **Steps 2+4 in code — the cursor**: every operation moves via `BTreeCursor` - (`btree.rs:714`), with `CursorContext` (`btree.rs:539`) and `PinGuard` - (`btree.rs:375` — pins a page in the cache while the cursor points at it). - Trace one descent in `seek()` (`btree.rs:5681`): root → binary search the - cell pointer array → child page number → pager fetch → leaf. Insert: - `insert()` (`btree.rs:5779`) → `insert_into_page()` (`btree.rs:2568`). +All at `tursodatabase/turso@dd775bc`. Line counts are that commit's; these files +move fast, so navigate by symbol name if you read a different revision. + +| File | Lines | Role (steps) | +|------|-------|------| +| `core/storage/btree.rs` | 13,186 | cursor, slotted pages, balance (2–5) | +| `core/storage/wal.rs` | 10,064 | WAL frames + checkpoint (6) | +| `core/storage/pager.rs` | 6,614 | page cache, dirty tracking, IO (1, 6) | +| `core/storage/sqlite3_ondisk.rs` | 2,449 | cell parsing — the byte format (3) | +| `core/storage/page_cache.rs` | 1,872 | SIEVE-eviction page cache (1, 6) | + +- **Step 1**: `DEFAULT_PAGE_CACHE_SIZE_IN_PAGES = 2000` — + `page_cache.rs:14`; `BTCURSOR_MAX_DEPTH = 20` — `btree.rs:133`. +- **Step 3**: the header field map and its ASCII diagram — `btree.rs:76–124`; + cell parsing in `read_btree_cell()` — `sqlite3_ondisk.rs:816`; delete + fragmentation fixed by `defragment_page()` — `btree.rs:8422` (with + `defragment_page_fast` at `:8273`, `_full` at `:8399`, `_for_insert` at + `:8412`); pointer-array maintenance via `copy_within` in + `shift_pointers_left()` — `btree.rs:9067`. +- **Steps 2 + 4 — the cursor**: every operation moves via `BTreeCursor` + (`btree.rs:714`), with `CursorContext` (`btree.rs:539`, key enum at `:530`) + and `PinGuard` (`btree.rs:375` — pins a page in the cache while the cursor + points at it). Trace one descent in `seek()` (`btree.rs:5681`; trait + declaration at `:653`): root → binary search the cell pointer array → child + page number → pager fetch → leaf. Insert: `insert()` (`btree.rs:5779`) → + `insert_into_page()` (`btree.rs:2568`) → `insert_into_cell()` (`btree.rs:8669`). ```mermaid flowchart LR @@ -165,40 +350,188 @@ flowchart LR D --> PG["pager.read_page
pager.rs:3240"] PG --> L["leaf: insert_into_page
btree.rs:2568"] L -- page overflows --> B["balance
btree.rs:2793"] - B --> BNR["balance_non_root
btree.rs:2995
redistribute ≤3 siblings"] + B --> BNR["balance_non_root
btree.rs:2995
≤3 siblings → ≤5 pages"] + BNR -- propagates to root --> BR["balance_root
btree.rs:4774
tree grows a level"] ``` -- **Step 5 in code**: `balance_non_root()` — `btree.rs:2995` (the ≤3-sibling - redistribution); `balance_root()` — `btree.rs:4774` (grows the tree by one - level). -- **Step 6 in code**: `Pager` struct — `pager.rs:1335`; reads via - `read_page()` — `pager.rs:3240` (cache first) and `read_page_no_cache()` — - `pager.rs:3185`. Dirty tracking in `add_dirty()` — `pager.rs:3412`; the - page is journaled to the WAL *before* modification — the write-ahead rule, - visible in code. WAL: `WalFile` (`wal.rs:2593`), frames appended in - `append_frames_vectored()` (`wal.rs:708`), `checkpoint()` (`wal.rs:3795`) - copies frames back into the main DB file. Page cache: `page_cache.rs:99` — - SIEVE eviction, default 2000 pages (buffer-pool preview, topic 6). - -## Questions to answer - -1. How many pages does a point lookup touch on a 1M-row table (page 4KB, ~50 cells - interior fanout)? Which of those are realistically cached? -2. Why does `balance_non_root` prefer redistribution over splitting? What does it do - to write amplification (3 dirty pages vs 2)? -3. During checkpoint, what blocks writers? (Read `checkpoint()` far enough to answer.) +- **Step 5**: `MAX_SIBLING_PAGES_TO_BALANCE = 3` — `btree.rs:136`; + `MAX_NEW_SIBLING_PAGES_AFTER_BALANCE = 5` — `btree.rs:139`; `balance()` — + `btree.rs:2793`; `balance_non_root()` — `btree.rs:2995`; `balance_root()` — + `btree.rs:4774`. +- **Step 6**: `Pager` — `pager.rs:1335`; `read_page()` — `pager.rs:3240`; + `read_page_no_cache()` — `pager.rs:3185`; `add_dirty()` — `pager.rs:3412` + (**subjournal**, not WAL — see Step 6); the subjournal handle itself — + `pager.rs:1357`. Commit path: `cacheflush()` — `pager.rs:3451`, which calls + `append_frames_vectored` at `pager.rs:3704`. WAL: `WalFile` — `wal.rs:2593` + (shared state `WalFileShared` — `wal.rs:2781`); `append_frames_vectored()` + impl — `wal.rs:4333` (trait declaration `wal.rs:708`); `checkpoint()` impl — + `wal.rs:3795` (trait declaration `wal.rs:715`). Page cache: `PageCache` — + `page_cache.rs:99`, SIEVE eviction described at `:90–98`, `spill_threshold` + at `:109` (buffer-pool preview, topic 6). + +## Questions to answer in notes.md + +Each needs the source open. `python3 tools/pinned-source.py show turso +core/storage/btree.rs -r A:B` is the fastest way in. + +1. Redo Step 2's fanout arithmetic for a **16-byte index key** instead of a + rowid table (index-interior cell = 4 B child + 1 B varint length + key). What + height does 1M rows give, how many pages is the interior level, and how much + of turso's default 2000-page cache does it consume? Then say which of those + two numbers — height or cached fraction — topic 3 found actually predicts + lookup latency. +2. Why does `balance_non_root()` (`btree.rs:2995`) prefer redistribution over + splitting? Check the claim on `btree.rs:138` — that cells from 3 pages always + fit in 5 — against your Step 2 cell sizes, and say what the choice does to + write amplification (≈3–5 dirty pages per balance versus 2 for a naive + split) *and* to space amplification. +3. During a checkpoint, what blocks writers? Read `checkpoint()` + (`core/storage/wal.rs:3795`) far enough to name the mode enum + (`CheckpointMode`, declared at `wal.rs:715`) and say which of its variants + waits for readers. +4. `add_dirty()` (`pager.rs:3412`) subjournals a page; `cacheflush()` + (`pager.rs:3451`) appends WAL frames. Write down, for a transaction that + modifies one page and then hits a statement error, exactly which bytes each + of the two journals holds and when each is discarded. Which one would you + have to disable to get an honest write-amplification measurement? +5. `BTCURSOR_MAX_DEPTH = 20` (`btree.rs:133`) is justified by "2^31 pages, + minimum fanout of 2 for a root and 3 for other internal nodes". Work that + bound: what tree size does depth 20 at fanout 3 actually cover, and how far + is that from the 2^31-page limit? What does the slack tell you about how + defensive this constant is? ## Done when -You can draw the slotted page from memory and explain how one insert can dirty 1 page -(common), 3 pages (balance), or O(height) pages (root split). +Answer each before unfolding it. + +- [ ] You can draw the slotted page from memory, name the six header fields, and say which two exist only to manage dead space. + +
+Answer + +Header (12 bytes interior, 8 leaf — `btree.rs:76`), then a rightward-growing +array of 2-byte cell pointers in key order, then free space, then the cell +content area growing leftward from the end of the page. + +The six fields (`btree.rs:84–124`): `BTREE_PAGE_TYPE` (0, u8), +`BTREE_FIRST_FREEBLOCK` (1, u16), `BTREE_CELL_COUNT` (3, u16), +`BTREE_CELL_CONTENT_AREA` (5, u16), `BTREE_FRAGMENTED_BYTES_COUNT` (7, u8), +`BTREE_RIGHTMOST_PTR` (8, u32, interior pages only). + +The two dead-space fields are `BTREE_FIRST_FREEBLOCK` — head of a chain of +unused runs of **at least 4 bytes** *inside* the content area, explicitly not +the free gap in the middle (`:90–97`) — and `BTREE_FRAGMENTED_BYTES_COUNT`, +which merely counts "isolated groups of 1, 2, or 3 unused bytes" (`:119`) too +small to chain. `defragment_page()` (`btree.rs:8422`) is what reclaims them. + +
+ +- [ ] You can compute fanout from page size and cell size, and give the height of a 1M-row rowid table with 100-byte rows. + +
+Answer + +`fanout = floor((P − H) / (c + s))` with P = 4096, H = 12 interior / 8 leaf, +s = 2 for the cell pointer. + +Interior (4 B child + 3 B varint rowid, c = 7): (4096 − 12)/9 = **453**. +Leaf (2 B varint length + 3 B varint rowid + 100 B, c = 105): +(4096 − 8)/107 = **38**. + +1,000,000 / 38 = **26,316 leaves** (107.8 MB); 26,316 / 453 = **59** interior; +59 / 453 = **1** root. **Height 3** — a point lookup reads 3 pages. The 60 +interior pages are 245.8 KB, **0.23%** of the file, so they stay resident in the +2000-page (8.2 MB) default cache along with 7.4% of the leaves. + +
+ +- [ ] You can explain how one insert can dirty 1 page, 3–5 pages, or O(height) pages, and name the constant that bounds the middle case. + +
+Answer + +**1 page**: the cell fits — the pointer array and the content area have not met, +so the write is `copy_from_slice` into the content area plus a `copy_within` over +the 2-byte pointers. + +**3–5 pages**: the regions met, so `balance_non_root()` (`btree.rs:2995`) pools +the full page with up to two siblings — `MAX_SIBLING_PAGES_TO_BALANCE = 3`, +`btree.rs:136` — and redistributes into at most +`MAX_NEW_SIBLING_PAGES_AFTER_BALANCE = 5` pages (`btree.rs:139`), updating the +parent's divider cells too. + +**O(height)**: the balance propagates upward until `balance_root()` +(`btree.rs:4774`) splits the root, which is the only operation that makes the +tree taller. Bounded above by `BTCURSOR_MAX_DEPTH = 20` (`btree.rs:133`). + +
+ +- [ ] You can state where a B-tree's space amplification comes from, and quote this repo's measured number for it. + +
+Answer + +Structurally: the free gap between the two regions on every page, plus the +freeblocks and fragments deletes leave behind — the price of in-place insertion. +Balancing across 3 siblings instead of splitting is the mitigation; it keeps +pages fuller than the 50% a naive split leaves. + +Measured, from [FINDINGS.md](../../FINDINGS.md) row 1 (`./verify.sh 01`): the +same 108.0 MB of records occupies **48.4 MB** under fjall's LSM (space amp +**0.45×**, below 1.0 because sorted runs are LZ4'd) and **6,833.9 MB** under +redb's copy-on-write B-tree (**63.28×**) — a **140× spread**. redb's mechanism +is copy-on-write rather than page slack: per `topics/01-storage-engine-landscape/notes.md`, +each of 1080 batch commits copies every page on the path to the root under +random-order inserts. Same direction, harsher constant. + +
+ +- [ ] You can distinguish turso's two journals and say which one implements the write-ahead rule. + +
+Answer + +The **subjournal** (`core/storage/subjournal.rs`, handle at `pager.rs:1357`, +written by `subjournal_page_if_required()` inside `add_dirty()`, +`pager.rs:3418`) holds page *pre-images* so a `SAVEPOINT` or a failed statement +can be undone **within** an open transaction. It is discarded when the +transaction ends. + +The **WAL** (`WalFile`, `wal.rs:2593`) holds page *post-images* for crash +recovery. Frames are appended on the commit path — `cacheflush()` +(`pager.rs:3451`) → `append_frames_vectored` (`pager.rs:3704` → `wal.rs:4333`) +— and folded back into the main database file only by `checkpoint()` +(`wal.rs:3795`). + +The write-ahead rule is the WAL's: the new version of a page reaches the log +before the main file is touched at all. Reading `add_dirty()` as the write-ahead +site conflates the two — a mistake this guide previously made. + +
## References -**Code** -- [turso](https://github.com/tursodatabase/turso) — - `core/storage/btree.rs` (~13K lines: cursor, slotted pages, balance), - `core/storage/pager.rs`, `core/storage/wal.rs`, - `core/storage/page_cache.rs`, `core/storage/sqlite3_ondisk.rs` - (shallow clone at `~/repos/turso`; line numbers drift — navigate by - symbol name) +**Code** (all at `tursodatabase/turso@dd775bc` — this repo's pin table entry; +confirm with `python3 tools/pinned-source.py ref turso`) +- [turso](https://github.com/tursodatabase/turso) — `core/storage/btree.rs` + (13,186 lines: header map `76–124`, `BTCURSOR_MAX_DEPTH:133`, + `MAX_SIBLING_PAGES_TO_BALANCE:136`, `PinGuard:375`, `CursorContext:539`, + `BTreeCursor:714`, `insert_into_page:2568`, `balance:2793`, + `balance_non_root:2995`, `balance_root:4774`, `seek:5681`, `insert:5779`, + `insert_into_cell:8669`, `defragment_page:8422`, `shift_pointers_left:9067`), + `core/storage/wal.rs` (10,064 — `WalFile:2593`, `checkpoint:3795`, + `append_frames_vectored:4333`), `core/storage/pager.rs` (6,614 — + `Pager:1335`, `read_page:3240`, `add_dirty:3412`, `cacheflush:3451`), + `core/storage/sqlite3_ondisk.rs` (2,449 — `read_btree_cell:816`), + `core/storage/page_cache.rs` (1,872 — SIEVE, default 2000 pages at `:14`) + +**This repo** +- [reading-fjall.md](reading-fjall.md) — the LSM protagonist opposite this one; + read both before answering the topic's shootout predictions +- [reading-comer-btree.md](reading-comer-btree.md) — Comer 1979, where the + fanout-and-height argument of Step 2 is proved rather than worked +- [topics/03-btree-internals/notes.md](../03-btree-internals/notes.md) — the + same page arithmetic across other key/value shapes, and the measurement that + height alone does not predict lookup latency +- [FINDINGS.md](../../FINDINGS.md) row 1 — LSM vs copy-on-write B-tree space + amplification, 0.45× vs 63.28×; `./verify.sh 01` diff --git a/topics/02-in-memory-structures/notes.md b/topics/02-in-memory-structures/notes.md index c33741f..787a55b 100644 --- a/topics/02-in-memory-structures/notes.md +++ b/topics/02-in-memory-structures/notes.md @@ -42,27 +42,24 @@ does a little migration work. ## Reading answers +Each guide ends with its own `## Questions to answer in notes.md` list, and the +lists differ in length (4 to 6 questions each). Copy the questions from the +guide you are on rather than working from a fixed count here — that way this +file stays right when a guide gains a question. + ### redis dict (reading-redis-dict.md) -1. Insert into ht[0] during rehash — why a bug: -2. pauserehash exists for: -3. empty_visits=10n tail guarantee: ### redis skiplist (reading-redis-skiplist.md) -1. Why skiplist + dict both: -2. Expected search cost at p=0.25, priced vs measured: ### hashbrown (reading-hashbrown.md) -1. 7/8 vs 1.0 load factor: -2. Hash policy paragraph (for M2 decision): -3. DELETED churn ↔ LSM tombstones: ### RocksDB memtable (reading-rocksdb-memtable.md) -1. spans/backward under concurrent CAS: -2. acquire/release vs SeqCst at line 383: -3. Miss estimate vs hashbrown number: -### rax / ART / SwissTable talk -- (questions in each guide) +### redis rax (reading-redis-rax.md) + +### ART paper (reading-art-paper.md) + +### SwissTable talk (reading-swisstable-talk.md) ## Experiment findings diff --git a/topics/02-in-memory-structures/reading-art-paper.md b/topics/02-in-memory-structures/reading-art-paper.md index f2322ab..59a950b 100644 --- a/topics/02-in-memory-structures/reading-art-paper.md +++ b/topics/02-in-memory-structures/reading-art-paper.md @@ -1,160 +1,703 @@ # ART: sorted like a tree, probed like a hash table -The index inside HyPer and DuckDB — a radix tree tuned until it beats hash -tables on some workloads *while staying sorted*. Where rax spends its design -budget on memory, ART spends it on lookup speed: node layouts that adapt to -fanout, each picking the cheapest search its density allows. It is also where -this topic's SwissTable and radix-tree threads literally meet, in Node16's -SIMD probe. This chapter builds the paper's ideas one at a time — the -sparse-node waste, the four adaptive layouts, the two compression tricks, the -key encoding that makes everything radix-able — then routes you through the -sections. +A radix tree tuned until it beats a chained hash table on lookups *while +staying sorted*. Where rax spends its design budget on memory, ART spends it +on lookup latency: four inner-node layouts, each picking the cheapest search +its density allows. It is also where this topic's SwissTable and radix-tree +threads literally meet, in Node16's SSE probe. This chapter builds the paper's +ideas one at a time — the sparse-node waste, the four layouts, the two +collapsing tricks, the key encoding that makes it universal, and the space +proof — then routes you through the sections. + +The paper is **Viktor Leis, Alfons Kemper, Thomas Neumann, "The Adaptive Radix +Tree: ARTful Indexing for Main-Memory Databases", ICDE 2013**, 12 pages, +[PDF](https://db.in.tum.de/~leis/papers/ART.pdf). Every number below is +followed by the section, figure or table it came from; if a claim here has no +such tag, treat it as this guide's own arithmetic and check it. Budget ~2 h. + +The paper's own system is **HyPer** (§V-D). ART is not a museum piece: DuckDB +ships it as one of its two built-in index types, "mainly used to ensure primary +key constraints and to speed up point and very highly selective (i.e., < 0.1%) +queries" +([DuckDB docs, *Indexes*](https://duckdb.org/docs/current/sql/indexes.html)). ## The problem in one sentence -A radix tree that branches on full bytes needs 256 child pointers per node — -2 KB — and a real node averages a handful of children, so a naive -main-memory radix index burns **~98% of its space on null pointers**; shrink -the nodes naively and every level becomes a search, i.e. a B-tree with extra -steps. +A radix tree that branches on a full byte needs a 256-entry pointer array per +inner node — 2064 bytes with a 16-byte header and 8-byte pointers (Table I) — +while a real node often holds a handful of children, so a naive main-memory +radix index spends almost all of its space on null pointers; shrink the span +instead and the tree gets taller, which is the trade §III-B calls "excessive" +in one direction and slow in the other. ## The concepts, step by step -### Step 1 — the tension: radix depth vs radix memory +### Step 1 — the tension: span buys height and costs memory -Recall from the rax chapter: a radix tree finds keys by *spelling* them — -one branch decision per key byte, depth = key length, no comparisons, no -hashing. Branching on a full byte (span 8 bits) keeps depth minimal — ≤8 -levels for an 8-byte integer key, regardless of n — but demands room for 256 -children per node. Binary-comparison trees (B-tree, T-tree) have the -opposite problem: compact nodes, but every level costs a key comparison plus -a dependent cache miss, and depth grows as log₂(n). ART's move: keep the -byte-wise branching, but **make the node's physical size adapt to how many -children it actually has**. +> **In:** the radix-tree idea from the rax chapter — spell the key, one branch +> per chunk, no comparisons. +> **Out:** the span parameter *s*, the height formula it controls, the +> exponential space cost it carries, and the exact key count above which a +> radix tree is shorter than a perfect binary search tree. + +§III-A defines the knob. An inner node is "an array of 2ˢ pointers"; during +traversal an *s*-bit chunk of the key indexes that array, "and thereby +determines the next child node without any additional comparisons". The +parameter *s* is the **span**, and it fixes the height: + +> "A radix tree storing k bit keys has ⌈k/s⌉ levels of inner nodes. With 32 bit +> keys, for example, a radix tree using s = 1 has 32 levels, while a span of 8 +> results in only 4 levels." — §III-A + +So span is pure height leverage — and pure space cost, because the node is +2ˢ pointers wide whether or not the children exist. §III-B: "Space usage can be +excessive when most child pointers are null", illustrated by Figure 3, which +plots height against space for 1M uniformly distributed 32-bit integers and +shows the space axis running from 32 MB to 32 GB as *s* goes from 1 to 32. +Real systems pick a middle value: the Generalized Prefix Tree uses s = 4, the +Linux kernel radix tree s = 6 (§III-B). + +§III-A also gives the comparison against comparison-based trees, and it is +worth doing the arithmetic rather than reading past it. A perfect BST has +height log₂ n; a radix tree has height k/s; they are equal when +n = 2^(k/s), and the paper states radix trees are shorter "for n > 2^(k/s)": + +``` + 32-bit keys, s = 8: height 32/8 = 4 crossover n = 2^4 = 16 keys + 64-bit keys, s = 8: height 64/8 = 8 crossover n = 2^8 = 256 keys +``` + +Two hundred and fifty-six. Above a few hundred 64-bit keys, a byte-wise radix +tree is already shorter than any balanced binary tree can ever be, and it stops +growing entirely. That is the whole motivation, in one division. + +ART's move is to keep s = 8 — "This choice also has the advantage of +simplifying the implementation, because bytes are directly addressable which +avoids bit shifting and masking operations" (§III-C) — and make the node's +*physical* size adapt to how many children it actually has. ### Step 2 — the four node types: one logical node, four layouts -An ART node is logically always "up to 256 children indexed by byte"; its -physical layout is whichever of four types fits the current child count -(§III.A — the core of the paper): +> **In:** a logical inner node that maps up to 256 key bytes to children, and +> a child count that varies wildly across a real tree. +> **Out:** four concrete layouts with their capacity ranges and byte sizes +> from Table I, and the growth/shrink rule between them. +§III-C names four data structures "according to their maximum capacity", and +Table I gives their sizes under the paper's stated assumptions — a **16-byte +header** storing node type, child count and compressed path, and **8-byte +pointers**: + +``` + Table I (§III-G) — SUMMARY OF THE NODE TYPES (16 BYTE HEADER, 64 BIT POINTERS) + + Type Children Space (bytes) + Node4 2-4 16 + 4 + 4·8 = 52 + Node16 5-16 16 + 16 + 16·8 = 160 + Node48 17-48 16 + 256 + 48·8 = 656 + Node256 49-256 16 + 256·8 = 2064 ``` -Node4 keys[4] ┌k┬k┬k┬k┐ linear scan, fits in - ptrs[4] └●┴●┴●┴●┘ one cache line -Node16 keys[16] ┌k×16────────┐ SIMD compare — literally the - ptrs[16] └●×16────────┘ SwissTable group probe trick +Read the arithmetic column, not just the totals — each one tells you the +layout. Node4 and Node16 are "one key part and one pointer part" (§III-C): +`n` key bytes plus `n` pointers, keys sorted and at corresponding positions. +Node48's `256` is not keys, it is a **256-entry index array**: "a 256-element +array is used, which can be indexed with key bytes directly … this array stores +indexes into a second array which contains up to 48 pointers. This indirection +saves space in comparison to 256 pointers of 8 bytes, because the indexes only +require 6 bits (we use 1 byte for simplicity)" (§III-C). Node256's `256·8` is +the plain pointer array with no keys at all. + +Note Node4's minimum is **2**, not 1. That is not arbitrary: Step 4's path +compression guarantees "each inner node has at least two children" (§III-E), +and Step 6's proof leans on it. + +Nodes change type in place as they fill or empty: "When the capacity of a node +is exhausted due to insertion, it is replaced by a larger node type. +Correspondingly, when a node becomes underfull due to key removal, it is +replaced by a smaller node type" (§III-B). Figure 9 shows this as `if +isFull(node) grow(node)` on lines 31-32 of the insert pseudocode. + +**Work the saving.** A node with 4 children stored as a Node256 wastes 252 of +its 256 pointers: -Node48 index[256]┌256 × 1-byte ─┐ byte-indexed indirection: - ptrs[48] └48 × 8-byte ─┘ index[c] → slot in ptrs - -Node256 ptrs[256] ┌●×256────────┐ direct array — no search at all ``` + null pointers: 252 / 256 = 98.44 % of the array + wasted bytes: 252·8 / 2064 = 2016/2064 = 97.67 % of the node + Node256 / Node4: 2064 / 52 = 39.7× larger +``` + +Thirty-nine point seven times. That factor, applied to the many sparse nodes a +real key distribution produces, is what §III-B means by "excessive". -Nodes grow and shrink between types as children are added or removed — a -Node4 gaining a fifth child is copied into a Node16, and so on. Space now -tracks density: a 3-child node costs ~56 bytes, not 2 KB. +### Step 3 — one search strategy per layout -### Step 3 — search strategy per type: pay only what density demands +> **In:** the four layouts from Step 2 and a key byte to find. +> **Out:** four different `findChild` implementations — loop, SIMD, double +> indirection, direct index — and the reason the choice is per-node rather +> than global. -Each layout picks the cheapest search its density allows — the progression is -linear → SIMD → indexed → direct, and one `match` carries the whole idea: +Figure 8 is the whole point of the paper compressed into 21 numbered lines. The +paper's own pseudocode: -```rust -fn find_child(node: &Node, byte: u8) -> Option<&Node> { - match node { - Node4 { keys, ptrs, n } => // ≤4 children: linear scan, - (0..*n).find(|&i| keys[i] == byte) // one cache line - .map(|i| &ptrs[i]), - Node16 { keys, ptrs, .. } => { - let hits = simd_eq(keys, byte); // the SwissTable group probe - one_bit(hits).map(|i| &ptrs[i]) // (≤1 hit here: keys unique) - } - Node48 { index, ptrs } => // byte-indexed indirection - slot(index[byte as usize]).map(|s| &ptrs[s]), - Node256 { ptrs } => // direct — no search at all - ptrs[byte as usize].as_ref(), - } -} +``` + Fig. 8 (§III-F) — findChild(node, byte), abridged; line numbers are the paper's + + 1 if node.type==Node4 // simple loop + 2 for (i=0; isize` is large" — rax and ART reach opposite conclusions about the same +inner loop because they are optimising different corners. ### Step 4 — lazy expansion and path compression: kill the boring levels -Two tricks remove nodes that exist only to spell out bytes (§III.B) — both -are rax ideas with fixed-size discipline: +> **In:** a tree whose height is still the key length in bytes, because every +> byte gets a level whether or not it distinguishes anything. +> **Out:** two independent height reductions from §III-E, the pessimistic / +> optimistic choice for storing skipped bytes, and ART's actual hybrid — which +> the previous version of this chapter had backwards. + +§III-E, titled *Collapsing Inner Nodes*, introduces two techniques: + +- **Lazy expansion** — "inner nodes are only created if they are required to + distinguish at least two leaf nodes". Figure 6 shows it saving two inner + nodes by truncating the path to the leaf "FOO". The catch is stated in the + same paragraph: "because paths to leaves may be truncated, this optimization + requires that the key is stored at the leaf or can be retrieved from the + database". Figure 7's search pseudocode handles it at line 4, + `leafMatches(node, key, depth)`. +- **Path compression** — "removes all inner nodes that have only a single + child", exactly rax's `iscompr`. The removed bytes still have to be dealt + with, and §III-E gives two approaches: + - **Pessimistic**: store a variable-length partial key vector at each inner + node holding the bytes of the removed one-way nodes, and compare it against + the search key before descending. + - **Optimistic**: store only the *count* of removed nodes, skip that many + bytes without comparing, and compare the full key once at the leaf to + catch a "wrong turn". + +Now the sentence to get right, because it is easy to invert: + +> "We therefore use a hybrid approach by storing a vector at each node like in +> the pessimistic approach, but with a constant size (8 bytes) for all nodes. +> Only when this size is exceeded, the lookup algorithm dynamically switches to +> the optimistic strategy." — §III-E + +ART is **pessimistic by default**, with a fixed 8-byte prefix in the header, +and falls back to **optimistic** only when a compressed path is longer than 8 +bytes. The direction matters: pessimistic means "compare as you go and never +be wrong"; optimistic means "skip and verify at the leaf". ART pays 8 bytes of +every header to stay in the safe mode for the common case. The paper's stated +reason for the cap is the same one rax answers differently: the optimistic +approach "requires one additional check, while the pessimistic method uses more +space, and has variable sized nodes leading to increased memory fragmentation". +Fixed-size nodes are non-negotiable in ART; rax took exactly the other side of +that trade with its arbitrary-length runs. + +Both approaches share one guarantee that Step 6 needs: "Both approaches ensure +that each inner node has at least two children." + +How much does this buy? §V-D measured it on TPC-C indexes (Figure 17): "the +height of index 3 would be 40 without any optimizations. Path compression and +lazy expansion reduce the average height to 8.1." Index 3 is a +`int,int,varchar(16),varchar(16),TID` compound key (Table IV) — 40 bytes of key +collapsed to about 8 levels. -- **Lazy expansion**: a subtree containing a *single* key isn't expanded at - all — the leaf stores the key's remaining bytes. (rax equivalent: storing - the key tail as a compressed run.) -- **Path compression**: a chain of one-child inner nodes is collapsed; each - node carries a **prefix** of the skipped bytes. (rax's `iscompr`.) ART caps - the stored prefix at 8 bytes — beyond that it goes "pessimistic": skip the - bytes optimistically and re-check the full key at the leaf. Why cap it? - Fixed-size node headers — ART refuses variable-size node layouts, which is - exactly the trade rax took the other way. +### Step 5 — binary-comparable keys: the encoding that makes it universal -Together these make depth ≈ number of *distinguishing* bytes, not key -length. +> **In:** a structure that iterates in byte-lexicographic order, and data types +> whose byte representation does not sort the way the type does. +> **Out:** the formal definition of a binary-comparable key and the per-type +> transformations, one of which you have already read in redis. + +Section IV is a whole section for a reason: without it, ART's sortedness is +useless for anything but ASCII. The paper's definition: + +> A transformation t : D → {0,1,…,255}^k produces binary-comparable keys if, +> for all x, y ∈ D: x < y ⇔ memcmp_k(t(x), t(y)) < 0, and likewise for > and +> =. — §IV-A + +And the transformations (§IV-B): + +| Type | Transformation | +|------|----------------| +| Unsigned integers | already ordered; **byte-swap on little-endian machines** so bytes run most- to least-significant | +| Signed integers | flip the sign bit — `x XOR 2^(b−1)` — then store as unsigned | +| IEEE 754 floats | classify into 10 non-overlapping classes (±normalised, ±denormalised, NaN, ±∞, 0), compute a rank, store as unsigned; "3 if statements, 1 integer multiplication, and 2 additions" | +| Character strings | UCA sort keys (e.g. ICU's `ucol_getSortKey`); terminate with a byte that appears nowhere else, "because keys must not be prefixes of other keys" | +| Null | give it a rank — e.g. widen only the smallest values: null → `0,0,0,0,0`, previously-smallest 0 → `0,0,0,0,1`, everything else keeps 4 bytes | +| Compound keys | transform each attribute separately and concatenate | + +You have already read the first row in production. redis's +`encodeTimeoutKey` (`src/timeout.c:78-83`, in the rax chapter) calls +`htonu64` on a millisecond timestamp before using it as a rax key, then appends +the client pointer as a tiebreaker — a big-endian unsigned integer followed by +a concatenated second attribute. That is §IV-B rows 1 and 6, written years +earlier without the vocabulary. + +Section IV also makes a claim worth carrying past this topic: binary-comparable +keys are what let you "replace comparison-based sorting algorithms like +quicksort or mergesort with the radix sort algorithm which can be +asymptotically superior". The same encoding buys ordered radix indexes *and* +radix sorting. + +### Step 6 — the space proof: why 52, and why exactly 52 + +> **In:** Table I's four node sizes and their minimum child counts. +> **Out:** the budget argument from §III-G worked on the actual numbers, +> showing both that 52 bytes per key holds and that it is tight. + +§III-G proves a worst-case bound of **52 bytes per key**, "even for arbitrarily +long keys". The mechanism is an amortisation argument: "Think of each leaf as +providing x bytes and inner nodes as consuming space provided by their +children." Formally, the budget of a node is x for a leaf, and otherwise the +sum of its children's budgets minus its own size. If every node's budget stays +non-negative, the tree costs less than x bytes per key. + +The paper says the induction goes through for x = 52 and leaves the four cases +to the reader. Do them — the arithmetic is four lines and it shows *which* node +type is binding: -### Step 5 — binary-comparable keys: the encoding that makes it universal +``` + budget(node) = (min children) · x − size(node), with x = 52 and Table I sizes + + Node4 : 2 · 52 − 52 = 104 − 52 = 52 ← exactly 52: the binding case + Node16 : 5 · 52 − 160 = 260 − 160 = 100 + Node48 : 17 · 52 − 656 = 884 − 656 = 228 + Node256: 49 · 52 − 2064 = 2548 − 2064 = 484 + + all ≥ 52 ⇒ the induction closes, and the bound is 52 bytes per key. +``` + +Node4 comes out at exactly 52, so the bound is **tight** for this node set — +try x = 51 and Node4 gives 2·51 − 52 = 50 < 51 and the induction fails +immediately. The worst case is a tree of minimally-filled Node4s, which is +precisely the shape a sparse key distribution produces, and it is bounded +because path compression forbids one-child nodes (Step 4). + +Footnote 1 says the bound "can be reduced to 34 bytes per key" with six node +types, where "the Node4 type is replaced by the new node types Node2 and +Node5". That number is derivable from the same argument: a Node2 costs +16 + 2 + 2·8 = **34** bytes, and the binding constraint 2x − 34 ≥ x gives +x ≥ 34. Splitting the smallest node type is exactly how you move the bound, +because the smallest type is what binds. + +Table II puts the bound in company: + +``` + Table II (§III-G) — worst-case bytes per key, 64-bit pointers + + k = 32 k → ∞ + ART 43 52 + GPT 256 ∞ + LRT 2048 ∞ + KISS >4096 NA +``` + +GPT and LRT are unbounded "because [they] do not use path compression, the +number of inner nodes is proportional to the length of the keys" (§III-G). +A bound that survives k → ∞ is the thing adaptive nodes plus path compression +buy, and it is what lets a database *promise* an index memory budget. + +The measured side is much better than the bound. The contributions list in §I +claims "often as low as 8.1 bytes per key", and Table IV shows where that comes +from — four of the seven major TPC-C indexes land at exactly 8.1 or 8.3 bytes +per key, all of them dense integers; the worst, index 3's long strings, is +32.6, "well below the worst case of 52 bytes" (§V-D). + +**Derive the 8.1.** Take 1,000,000 dense 32-bit integer keys — the paper's own +best case, "integers ranging from 1 to n" (§III-G) — with s = 8, so four levels: + +``` + level 4 (last key byte) : fully dense ⇒ Node256; ⌈10⁶/256⌉ = 3907 nodes + level 3 : ⌈3907/256⌉ = 16 nodes (Node256) + level 2 : 16 children = 1 node (Node16) + level 1 : one-way ⇒ removed by path compression = 0 nodes + + bytes = 3907·2064 + 16·2064 + 1·160 = 8,097,072 + 160 = 8,097,232 + per key = 8,097,232 / 1,000,000 = 8.097 bytes +``` -A radix tree returns keys in **byte order**, so for sorted iteration and -range scans to be *correct*, byte order must equal logical order. §III.E + -§IV show the transformations: store integers big-endian (most significant -byte first), flip the sign bit for signed ints, massage IEEE floats, null- -terminate strings, concatenate fields for composite keys. Example: as -little-endian bytes, 256 (`00 01 00 ...`) sorts *before* 1 (`01 00 ...`) — -big-endian fixes it. Don't skip this section: the idea is everywhere — -RocksDB comparators, FoundationDB tuples, and your capstone's composite -(entity, attr) keys in M2 are all binary-comparable encodings. - -### Step 6 — the space guarantee: 52 bytes per key, worst case - -Adaptive nodes plus path compression buy a provable bound (§III.B): -worst-case **52 bytes per key** regardless of key distribution — no -adversarial key set can blow the structure up. Compare: your skiplist's -per-node cost (1.33 pointers average + key) is fine on average but has no -such bound story, and a naive radix tree has no bound at all. Bounds like -this are what let a database *promise* memory budgets. +**8.1 bytes per key** — the paper's number, reproduced from Table I and a +division. Dense keys fill Node256s completely, so the amortised cost is +2064/256 = 8.06 bytes of pointer per key plus a rounding error. That is why +§V-D says "the best case of 8.1 bytes … does occur quite frequently because +surrogate integer keys are often dense". + +### Step 7 — what §V actually measured, and its caveats + +> **In:** the claim "comparable to hash tables" from the abstract. +> **Out:** the hardware, the contestants, the two caveats that shape the micro +> benchmarks, and the specific figures — including the one that this repo has +> independently measured. + +The setup (§V): an Intel Core i7 3930K — 6 cores, 12 threads, 3.2 GHz (3.8 GHz +turbo), 12 MB shared L3, 32 GB quad-channel DDR3-1600 — on Linux 3.2, GCC 4.6. +Contestants: a cache-sensitive B⁺-tree (CSB), k-ary search, FAST, the +Generalized Prefix Tree (GPT), a red-black tree, and a **chained hash table +using MurmurHash64A**. + +Two caveats decide how far the micro benchmarks generalise, and the paper +states both plainly: + +1. **32-bit integer keys only**, "because some of the implementations only + support 32 bit integer keys". +2. **Path compression was removed** for the micro benchmarks: "For such very + short keys, path compression usually increases space consumption instead of + reducing it. Therefore, we removed this feature for the micro benchmarks. + Path compression is enabled in the more realistic second part." + +So Figures 10-15 measure an ART without one of its two headline optimisations, +on the key type most favourable to radix trees. Read them accordingly. The +paper also reports separately for **dense** keys (1..n, randomly permuted) and +**sparse** keys (each bit equally likely 0 or 1) — and the gap between those +two bars is the real story. + +Table III is the most useful table in the paper, because it is counters rather +than throughput: + +``` + Table III (§V-A) — performance counters per lookup + 65K keys 16M keys + ART(dense/sparse) FAST HT ART(dense/sparse) FAST HT + Cycles 40 / 105 94 44 188 / 352 461 191 + Instructions 85 / 127 75 26 88 / 99 110 26 + Misp. branches 0.0 / 0.85 0.0 0.26 0.0 / 0.84 0.0 0.25 + L3 hits 0.65 / 1.9 4.7 2.2 2.6 / 3.0 2.5 2.1 + L3 misses 0.0 / 0.0 0.0 0.0 1.2 / 2.6 2.4 2.4 +``` + +Three readings. At 16M keys ART-dense takes **188 cycles** against the hash +table's **191** and FAST's **461** — "comparable to hash tables" is a fair +summary of that column. The dense/sparse split is entirely a cache-miss story: +1.2 versus 2.6 L3 misses, and the paper says so — "With dense keys, ART causes +only half as many cache misses because its compact nodes can be cached +effectively." And ART-sparse carries **0.84 mispredicted branches per lookup** +that ART-dense does not, "which occur during node type dispatch" (§V-A) — the +price of having four node types is a branch the CPU cannot predict when the +types are mixed. That is the RUM bill for adaptivity, paid in the pipeline. + +The other figures worth knowing: + +- **Figure 13, cache pressure.** "With 1/64th of the cache (192KB), ART reaches + only about one third of the performance of the entire cache (12MB)", while + the hash table "is mostly unaffected, as it does not use caches effectively + anyway". Tree structures live on cached upper levels; a shared cache is a + hidden dependency. +- **§V-C, adaptivity's insert cost.** "The impact of adaptive nodes on the + insertion performance (in comparison with only using Node256) is 20% for + trees with 16M dense keys" — the growth/shrink machinery costs a fifth of + insert throughput and the paper calls it "usually a worthwhile trade off". + Bulk loading recovers 2.5× on sparse keys and 17% on dense; sorted dense + insertion reaches "50 million sorted, dense keys … per second". +- **§V-D, TPC-C end to end.** "ART is almost twice as fast as the hash table / + red-black tree combination and almost four times as fast as the red-black + tree alone", and — the sentence to underline — the hash table "introduced + unacceptable rehashing latencies which are clearly visible as spikes in the + graph" (Figure 16). + +That last one is not a claim you have to take on faith: **this repo measured +it**. Topic 2's `rehash_spike` lane inserts 10 M keys into `hashbrown` one at a +time and reports `p50 = 42 ns` against `max = 58.4 ms` — a 1.4-million-fold +spread, with four of ten deciles carrying a multi-millisecond spike at the +power-of-two boundaries. ART's advantage in Figure 16 is not that it is faster +on average; it is that it has no rehash, so it has no tail. Leis et al. +observed the shape in 2013; the lane in this topic reproduces it in 2025 on a +different hash table. ## How to read the paper (with the concepts in hand) -1. **§III.A–B** — node types (Steps 2–3) + lazy expansion / path compression - (Step 4). Map both tricks onto rax as you read; note where ART's 8-byte - prefix cap diverges from rax's unbounded runs and why. -2. **§III.C–D** — insert/delete with node-type transitions. Skim — it's - Step 2's grow/shrink mechanics spelled out. -3. **§III.E + §IV — binary-comparable keys** (Step 5). Don't skip; work the - encodings until you could encode (u64, u16) pairs cold. -4. **§V — evaluation.** Read Fig. 8/9 with topic-0 eyes: where does ART beat - the hash table (dense integer keys — short paths, no hash cost) and where - does it lose (long random strings — depth ∝ length)? +The section numbering matters — several of these are commonly misquoted. + +| Section | Contents | Step | +|---------|----------|------| +| §I | motivation, contributions (the 52 and 8.1 figures appear here first) | — | +| §II | related work — GPT, LRT, KISS-Tree, Judy, Graefe on normalised keys | 1 | +| §III-A | Preliminaries — span, height, the n > 2^(k/s) crossover, Figure 2 | 1 | +| §III-B | Adaptive Nodes — the space/height tradeoff, Figure 3, grow/shrink | 1, 2 | +| §III-C | **Structure of Inner Nodes** — Node4/16/48/256, Figure 5 | 2 | +| §III-D | Structure of Leaf Nodes — single-value, multi-value, combined slots | 2 | +| §III-E | **Collapsing Inner Nodes** — lazy expansion, path compression, hybrid | 4 | +| §III-F | Algorithms — Figures 7 (search), 8 (findChild), 9 (insert); bulk load | 3 | +| §III-G | **Space Consumption** — Tables I and II, the 52-byte proof | 2, 6 | +| §IV | **Constructing Binary-Comparable Keys** — definition and per-type rules | 5 | +| §V-A | Search performance — Figures 10-12, Table III | 7 | +| §V-B | Caching effects — Figures 12, 13 | 7 | +| §V-C | Updates — Figures 14, 15 | 7 | +| §V-D | End-to-end TPC-C in HyPer — Figures 16, 17, Table IV | 6, 7 | + +A route through it: + +1. **§III-A**, two pages. Do the crossover division yourself for 64-bit keys + before looking at Figure 2. +2. **§III-C with Figure 5 open.** Write the four layouts from the Table I + arithmetic (`16 + 256 + 48·8`) rather than from the prose; the arithmetic + tells you what is stored. +3. **§III-F, Figure 8 only.** Twenty-one lines. Map each branch onto its node + type and say what it costs in memory touches. +4. **§III-E.** Read the pessimistic/optimistic paragraph twice and write down + which one ART uses by default. (It is pessimistic, with an 8-byte cap.) +5. **§III-G.** Work the four budget lines from Step 6 on paper. Then work + x = 51 and watch Node4 fail. +6. **§IV.** Work the encodings until you could encode a `(u64, u16)` pair cold, + including the null case. +7. **§V.** Read §V's first two paragraphs for the caveats *before* any figure, + then Table III. Skim the throughput bars. +8. **Aha:** the paper's four node types are not four optimisations, they are + one — pick the cheapest search the density allows — and the price is + Table III's `0.84 mispredicted branches per lookup` on sparse keys, which + is the node-type dispatch. Every adaptive structure pays for its adaptivity + somewhere; ART pays in a branch. Once you see the cost line for the headline + feature, you are reading the paper the way its authors did. + +**Contrast case.** Read §III-C's Node4-through-Node256 progression directly +against `rax.c:150-155` (`raxNodeCurrentLength`) from the previous chapter. rax +has *one* node layout whose size is computed per node — a 4-child rax node is +4 + 4 + 0 + 32 = 40 bytes against ART's fixed Node4 at 52, and a 16-child rax +node is 4 + 16 + 4 + 128 = 152 against ART's Node16 at 160. rax is smaller at +every fanout, and pays for it with a linear scan and variable-size nodes that +fragment; ART is slightly larger and fixed-size, and gets a SIMD probe, a 16-byte +header with room for a compressed path, and a provable bound. Neither is +"better". They are two points on the same curve, and the paper and the C file +each argue their own corner in a comment. ## Questions to answer in notes.md -1. Node16 search is the SwissTable group probe (compare 16 bytes in one SIMD op). - What's the *structural* difference between how ART and SwissTable use the - result? (ART: index into child pointers; Swiss: candidate slots to verify.) -2. Height of ART on 8-byte integer keys is ≤ 8 regardless of n. At what n does - log₂(n) exceed that — i.e., where does a B-tree start losing on depth alone? -3. For the capstone: would ART beat your M2 hash-based attribute store for - (entity id, attr id) → value? Sketch the key encoding and the RUM trade. +1. Node16's probe (Fig. 8, lines 6-12) is instruction-for-instruction the + SwissTable group probe from `reading-hashbrown.md`. State the *structural* + difference in what each does with the resulting bitfield, and say which one + can have more than one bit set and why. +2. §III-A gives the crossover n > 2^(k/s). Compute it for 64-bit keys at + s = 8, s = 4 and s = 1, and say what that implies about the Linux kernel's + choice of s = 6 for a tree indexed by page offsets. +3. Work the Step 6 budget table yourself, then redo it assuming a 32-byte + header instead of 16. What is the new bound, and which node type is binding? +4. §V removed path compression for the micro benchmarks and used 32-bit + integer keys. Name one figure whose conclusion you would expect to change + with 40-byte string keys and path compression on, and say in which + direction. +5. Figure 16's hash-table rehash spikes are the same phenomenon this repo + measured in the `rehash_spike` lane (`p50 = 42 ns`, `max = 58.4 ms`). Which + ART property removes the tail, and what does it cost — name the counter in + Table III that pays for it. +6. For the capstone: would ART beat a hash-based attribute store for + `(entity id, attr id) → value`? Write the §IV-B encoding for that compound + key explicitly, then state the RUM trade in the terms of Table I. + +## Takeaway + +ART is one idea applied four times: pick the cheapest search the local density +allows, and let the node layout follow. That converts the radix tree's +fundamental problem — a big span costs 2ˢ pointers whether or not you use them +— from a global parameter into a per-node decision, which is why Figure 3 shows +ART below *and* to the left of every fixed-span tree. Path compression and lazy +expansion then cap the height at the number of *distinguishing* bytes, and the +combination yields a proof, not just a measurement: 52 bytes per key for any +key set, any key length, worked in four lines of arithmetic in Step 6. The +costs are equally concrete — 0.84 mispredicted branches per sparse lookup for +node-type dispatch, and 20% of insert throughput for the grow/shrink machinery. +Carry the pattern rather than the structure: when a data structure has one +parameter that trades space against time, the interesting move is usually to +make it local. ## Done when -You can name the four node types with their search strategies from memory, and -explain binary-comparable key encoding well enough to encode (u64, u16) pairs. +Answer each before unfolding it. + +- [ ] Name the four node types with their capacity ranges and byte sizes, and + say what search each performs. + +
+Answer + +From Table I (§III-G) and Figure 8 (§III-F): + +| Type | Children | Bytes | Search | +|------|----------|-------|--------| +| Node4 | 2-4 | 16 + 4 + 4·8 = 52 | linear loop over the sorted key array | +| Node16 | 5-16 | 16 + 16 + 16·8 = 160 | one SSE compare of all 16 keys, masked, then `ctz` | +| Node48 | 17-48 | 16 + 256 + 48·8 = 656 | index the 256-byte `childIndex`, then the pointer array | +| Node256 | 49-256 | 16 + 256·8 = 2064 | the key byte *is* the index — no search | + +The 16 in every row is the constant-size header holding node type, child count +and the compressed path (§III-C). + +
+ +- [ ] ART caps its stored path-compression prefix at 8 bytes. What happens + beyond 8 bytes — does it become pessimistic or optimistic, and what does + that mean operationally? + +
+Answer + +It becomes **optimistic**. §III-E: ART stores a constant-size 8-byte partial +key vector "like in the pessimistic approach", and "only when this size is +exceeded, the lookup algorithm dynamically switches to the optimistic +strategy". Pessimistic means the skipped bytes are stored and compared during +descent, so a mismatch is caught immediately (Figure 7, lines 7-8). Optimistic +means only the *count* of skipped bytes is kept, the lookup skips them without +comparing, and the full key is compared once at the leaf to catch a wrong +turn. The default is the safe one; the fallback is the cheap one. Getting this +backwards inverts both the cost model and the failure mode. + +
+ +- [ ] Show that the 52-byte bound is tight, and name the node type that binds + it. + +
+Answer + +With x = 52 and Table I's sizes, the budget of each node type at its minimum +child count is 2·52 − 52 = **52** (Node4), 5·52 − 160 = 100 (Node16), +17·52 − 656 = 228 (Node48), 49·52 − 2064 = 484 (Node256). All are ≥ 52, so the +induction closes. Node4 hits it exactly, so it is **binding**: at x = 51 the +Node4 line gives 2·51 − 52 = 50 < 51 and the argument fails. That is also why +footnote 1's six-type variant reaches 34 — splitting Node4 into a Node2 +(16 + 2 + 2·8 = 34 bytes) makes 2x − 34 ≥ x the new binding constraint, i.e. +x ≥ 34. + +
+ +- [ ] Derive the paper's best-case 8.1 bytes per key from Table I, for 10⁶ + dense 32-bit integer keys. + +
+Answer + +Dense keys fill nodes completely, so every inner node is a Node256 except the +top. With s = 8 and 4-byte keys there are four levels: the last byte needs +⌈10⁶/256⌉ = 3907 Node256s, the level above ⌈3907/256⌉ = 16 Node256s, the level +above that one node with 16 children (a Node16, 160 bytes), and the top level +is a one-way node that path compression removes. Total +3907·2064 + 16·2064 + 160 = 8,097,232 bytes, i.e. **8.097 bytes per key**. +The intuition is the amortised cost of a full Node256: 2064/256 = 8.06 bytes of +pointer per child. Table IV's 8.1 for TPC-C indexes 1, 4 and 5 is this number. + +
+ +- [ ] Adaptivity is not free. Name the two costs the paper measures, with their + figures and sections. + +
+Answer + +(1) **Branch mispredictions from node-type dispatch**: Table III (§V-A) shows +0.84-0.85 mispredicted branches per lookup for sparse keys at both 65K and 16M +keys, against 0.0 for dense keys where every node is a Node256 and the dispatch +is predictable. (2) **Insert throughput**: §V-C, "The impact of adaptive nodes +on the insertion performance (in comparison with only using Node256) is 20% for +trees with 16M dense keys." Both are the price of having four layouts instead +of one, and the paper judges them worth paying — "Since the space savings from +adaptive nodes can be large, this is usually a worthwhile trade off." + +
+ +- [ ] Why must a binary-comparable string key be terminated with a byte that + appears nowhere else? + +
+Answer + +§IV-B(d): "it is important that each string is terminated with a value which +does not appear anywhere else in any string (e.g., the 0 byte). The reason is +that keys must not be prefixes of other keys." If "foo" were a prefix of +"foobar", a radix tree would have to represent "foo" at an *inner* node rather +than a leaf, which breaks lazy expansion (a truncated path can no longer be +resolved by comparing the leaf's key) and breaks the definition in §IV-A, since +`memcmp_k` compares fixed-length vectors. The terminator restores the property +that every key ends at a leaf. Compare rax, which allows a key at any node — +`iskey` is a bit on every node (`rax.h:79`) — and pays for it with a value +pointer slot in the node layout. + +
## References -**Papers** -- Leis, Kemper, Neumann — "The Adaptive Radix Tree: ARTful Indexing for - Main-Memory Databases" (ICDE 2013) — - [PDF](https://db.in.tum.de/~leis/papers/ART.pdf) — ~2 h; §III.A is the - core, don't skip §III.E/§IV (binary-comparable keys), read §V's - figures with topic-0 eyes +**Paper** + +- Viktor Leis, Alfons Kemper, Thomas Neumann — "The Adaptive Radix Tree: ARTful + Indexing for Main-Memory Databases", ICDE 2013. + [PDF](https://db.in.tum.de/~leis/papers/ART.pdf), 12 pages. + +| Where | What | +|-------|------| +| §III-A, Fig. 2 | span, ⌈k/s⌉ height, the n > 2^(k/s) crossover | +| §III-B, Fig. 3 | why a fixed span is either tall or huge; grow/shrink | +| §III-C, Fig. 5 | the four inner-node layouts | +| §III-E, Fig. 6 | lazy expansion, path compression, pessimistic/optimistic hybrid | +| §III-F, Figs. 7-9 | search, `findChild` (the SSE probe), insert, bulk loading | +| §III-G, Tables I-II | node sizes; the 52-byte proof; comparison to GPT/LRT/KISS | +| §IV | binary-comparable keys — definition and per-type transformations | +| §V-A, Table III | per-lookup cycles, instructions, mispredictions, L3 traffic | +| §V-C | 20% insert cost of adaptivity; bulk loading; 50 M sorted inserts/s | +| §V-D, Table IV, Figs. 16-17 | TPC-C in HyPer; 8.1-32.6 bytes/key; height collapse | + +**Related reading verified for this chapter** + +- [DuckDB, *Indexes*](https://duckdb.org/docs/current/sql/indexes.html) — ART + is one of DuckDB's two built-in index types, used for primary-key + constraints and highly selective (< 0.1%) point queries. + +**Measured in this repo** + +- `topics/02-in-memory-structures/README.md` and `notes.md`, the `rehash_spike` + lane: `p50 = 42 ns`, `p99.9 = 1292 ns`, `max = 58.4 ms` inserting 10 M keys + into `hashbrown`. This is §V-D's "unacceptable rehashing latencies", measured + independently on modern hardware and a different hash table. +- `topics/00-performance-toolbox/notes.md`, `lookup_shootout` at n = 10⁶: + `hashmap 8.8 ns`, `btreemap 26.6 ns`, `vec_binary_search 25.8 ns`. The + ordered-versus-unordered gap ART set out to close. Note this repo has no ART + lane — every ART number in this chapter comes from the paper, on 2013 + hardware. Building one is the honest way to hold the claims to account. +- `topics/00-performance-toolbox/notes.md`: 21% of a `HashMap` lookup is + SipHash. ART's cost model has no hash at all, which is part of why Table III + shows it matching a MurmurHash64A table on cycles. + +**Companion chapters** + +- [`reading-redis-rax.md`](reading-redis-rax.md) — the same structure with the + opposite RUM priority; the contrast case above uses its size arithmetic. +- [`reading-hashbrown.md`](reading-hashbrown.md) — Figure 8's lines 6-12, in + Rust, in a hash table. diff --git a/topics/02-in-memory-structures/reading-hashbrown.md b/topics/02-in-memory-structures/reading-hashbrown.md index 3ae683e..1f88c16 100644 --- a/topics/02-in-memory-structures/reading-hashbrown.md +++ b/topics/02-in-memory-structures/reading-hashbrown.md @@ -3,175 +3,754 @@ This IS `std::collections::HashMap` — you profiled it in topic 0 (21% SipHash, rest inlined probe loop), and now you read the probe loop the flamegraph flattened into "everything else". One idea carries the whole design: keep a -dense array of 1-byte tags beside the slots, so one SIMD load filters 8–16 -candidates before a single key byte is touched. This chapter builds that idea -step by step — open addressing, the control byte, group probing, the probe -sequence, tombstones — then maps each step onto the source. +dense array of 1-byte tags beside the slots, so one SIMD load filters a whole +group of candidates before a single key byte is touched. This chapter builds +that idea step by step — open addressing, the control byte, group probing, the +probe sequence, tombstones — then maps each step onto the source. + +Every anchor below is hashbrown **0.17.1** (`Cargo.toml:3`), the commit +`d69025b` this repo pins, quoted with the line numbers the code occupies in +that version. One warning before you start, because it changes half the +numbers people quote about SwissTable: **the group width is a property of the +target, not of the design.** `Group` is `__m128i` on x86 with SSE2 — +16 tags — and `uint8x8_t` on aarch64 NEON or a bare `u64` in the portable +fallback — **8 tags** (`src/control/group/sse2.rs:20`, +`src/control/group/neon.rs:16`, `src/control/group/generic.rs:41`, selected by +the `cfg_if` at `src/control/group/mod.rs:8-46`). This repo measures on an +Apple M3 Pro, so every figure below that depends on width is given for +`Group::WIDTH = 8` first, with the SSE2 value alongside. ## The problem in one sentence -A chained hash table pays 2+ dependent cache misses per lookup (bucket array, -then each malloc'd node) — ~200 ns at 10M keys — when the theoretical minimum -is one miss: the line the entry actually lives on. +A chained hash table pays 2+ dependent cache misses per lookup (the bucket +array, then each malloc'd node), and topic 0's `cache_ladder` priced a +dependent DRAM miss at ~100 ns on this machine +([FINDINGS.md](../../FINDINGS.md) row 0) — when the theoretical minimum is one +miss: the line the entry actually lives on. ## The concepts, step by step ### Step 1 — open addressing: store entries in the array itself -Instead of buckets pointing at malloc'd chain nodes (chaining — the redis -dict chapter), **open addressing** stores the key-value pairs directly in one -flat array of **slots**. On collision (the slot your hash points at is -taken), you don't follow a pointer — you **probe**: try other slots in a -deterministic sequence until you find the key or an empty slot. Wins: no -per-entry malloc, no pointer chase, and probing walks memory the prefetcher -can follow. Costs: deletion gets tricky (Step 5), and performance collapses -as the table fills — near 100% full, probe sequences get long, which is why -every open-addressing table enforces a maximum **load factor** (fraction of -slots occupied; hashbrown: 7/8). +> **In:** nothing yet — this step names the family hashbrown belongs to and +> the one parameter (load factor) that decides what it costs. +> **Out:** a flat array of slots and a probe rule, plus the reason a maximum +> load factor is mandatory. Step 2 makes the probe cheap. + +Instead of buckets pointing at malloc'd chain nodes (**chaining** — the family +the [redis dict chapter](reading-redis-dict.md) covers), **open addressing** +stores the key-value pairs directly in one flat array of **slots** (a slot is +one fixed-size home for one entry, occupied or not). On a **collision** — the +slot your hash points at is already taken — you don't follow a pointer, you +**probe**: try other slots in a deterministic sequence until you find the key +or an empty slot. Wins: no per-entry malloc, no pointer chase, and the probe +walks memory a hardware prefetcher can follow. + +The cost is that performance collapses as the array fills. Under the standard +uniform-hashing model — every probe lands on an independent uniformly random +slot, which is the textbook idealisation, not literally what Step 4's probe +does — the expected number of *slots examined* by an unsuccessful search at +**load factor** α (occupied slots ÷ total slots) is + +``` +E[slots examined, miss] = 1 / (1 − α) + + symbols: α = n/m, load factor n = live entries m = slots + reading: each probe has probability (1 − α) of hitting an empty slot, + so the number of tries until the first empty is geometric +``` + +Worked, with the divisions performed: + +``` +α = 0.50 → 1 / (1 − 0.50) = 1 / 0.500 = 2.0 slots examined +α = 0.75 → 1 / (1 − 0.75) = 1 / 0.250 = 4.0 +α = 0.875 → 1 / (1 − 0.875) = 1 / 0.125 = 8.0 ← hashbrown's limit +α = 0.9375→ 1 / (1 − 0.9375)= 1 / 0.0625 = 16.0 +``` + +Eight slot examinations at hashbrown's 7/8 limit against two at the classic +50%. That is why every open-addressing table before SwissTable capped α at +about 0.5 — and it is exactly the number Steps 2 and 3 make cheap rather than +smaller. Hold on to the 8.0. + +Deletion also gets harder (Step 5), for reasons that fall straight out of the +probe rule. ### Step 2 — the control byte: a dense 1-byte summary of every slot -The naive probe compares full keys slot by slot — touching a cache line of -slot data per step. hashbrown's move: keep a *separate, dense* array with -**one byte per slot** (the **control byte** or tag, `src/control/tag.rs:9–49`) -that answers "is this slot worth touching?" without touching it: +> **In:** the flat slot array and probe rule from Step 1. +> **Out:** a second array — one byte per slot — that answers "is this slot +> worth touching?" without touching it. Step 3 reads 8 or 16 of these bytes +> at once; Step 5 reuses their spare encoding for deletion. + +The naive probe compares full keys slot by slot, touching a cache line of slot +data per step. hashbrown's move is to keep a *separate, dense* array holding +**one control byte per slot** — a **tag**: a one-byte summary that is either +"empty", "deleted", or seven bits of the entry's own hash. + +```rust +// src/control/tag.rs — the whole encoding, 9-12 and 35-49 + 9 pub(crate) const EMPTY: Tag = Tag(0b1111_1111); + // ... 10-11: doc comment ... + 12 pub(crate) const DELETED: Tag = Tag(0b1000_0000); + // ... 13-34: is_full / is_special / special_is_empty, all one-bit tests ... + 35 pub(crate) const fn full(hash: u64) -> Tag { + // ... 36-46: MIN_HASH_LEN, so a 32-bit usize hash still uses its own top bits ... + 47 let top7 = hash >> (MIN_HASH_LEN * 8 - 7); + 48 Tag((top7 & 0x7f) as u8) // truncation + 49 } +``` + +Line 47 is the one that matters: the tag of an occupied slot is the **top 7 +bits of the hash**, and line 48 masks off the eighth so the high bit stays 0. +That single high bit is the entire state machine — `is_full` is +`self.0 & 0x80 == 0` (tag.rs:17), and both special values have it set, which +is why `EMPTY` is `0xff` and `DELETED` is `0x80`: they differ in the *low* bit +(`special_is_empty` at tag.rs:30 tests `self.0 & 0x01`), so one SIMD sign test +finds "empty or deleted" and one low-bit test separates them. ``` -tag values: EMPTY = 0xff DELETED = 0x80 FULL = 0b0xxxxxxx (h2: top 7 hash bits) +tag values: EMPTY = 0xff DELETED = 0x80 FULL = 0b0xxxxxxx (top 7 hash bits) -hash (64 bits): ┌──────── h1: index bits ────────┬─ h2: top 7 ─┐ - └── which group to probe first ──┴─ tag value ─┘ +hash (64 bits): ┌──────── low bits: h1 ───────┬─ top 7 bits ─┐ + └── which slot to probe first ┴─ tag value ──┘ control array: [23|EMPTY|91|07|DELETED|55|23|EMPTY| ... ] - └────────── one 8/16-byte SIMD load ─────────┘ + └───── one 8-byte (NEON/generic) load ─────┘ + └──────────── or 16 bytes on SSE2 ─────────┘ slot array: [ kv | ___ | kv | kv | ___ | kv | kv | ___ ] touched only on tag hit ``` -The hash is split once: the low bits (**h1**) choose where to start probing; -the top 7 bits (**h2**) become the tag of a FULL slot. A probe compares h2 -against tags first, and only a tag match earns a real key comparison. This is -the "dense filter + fat payload" pattern (README §4): the filter array is 1 -byte per slot, so 64 slots of metadata fit in one cache line. +Two naming warnings, because the literature and the code disagree. The +abseil/CppCon vocabulary calls the index bits **h1** and the tag bits **h2**; +hashbrown keeps `h1` (`src/raw.rs:61-64`, and it is simply `hash as usize` — +the *whole* hash truncated, then masked by `bucket_mask`, so effectively the +low bits) but has no `h2`: the tag constructor is `Tag::full` and the local is +called `tag_hash` (raw.rs:2010). Second, the two arrays are not two +allocations. `RawTableInner` holds one pointer: + +```rust +// src/raw.rs — RawTableInner, 566-580 + 566 struct RawTableInner { + 567 // Mask to get an index from a hash value. The value is one less than the + 568 // number of buckets in the table. + 569 bucket_mask: usize, + 570 + 571 // [Padding], T_n, ..., T1, T0, C0, C1, ... + 572 // ^ points here + 573 ctrl: NonNull, + 574 + 575 // Number of elements that can be inserted before we need to grow the table + 576 growth_left: usize, + 577 + 578 // Number of elements in the table, only really used by len() + 579 items: usize, + 580 } +``` + +The comment on lines 571-572 is the layout: slots grow *downward* from the +`ctrl` pointer (T0 immediately before C0) and control bytes upward, one +allocation, one pointer. So the "dense filter, fat payload" split of README §4 +is a split in addressing, not in allocation: the filter array is 1 byte per +slot, so 64 slots' worth of metadata fit in one 64-byte cache line. -### Step 3 — group probing: 16 tags in one SIMD instruction +### Step 3 — group probing: a whole group of tags in one instruction -Because tags are dense bytes, SIMD (single instruction, multiple data — CPU -instructions that operate on 16 bytes at once) can compare h2 against a whole -**group** of 16 tags (8 on ARM NEON) in one instruction, yielding a bitmask -of candidates. The lookup, de-macro'd: +> **In:** the dense control array from Step 2 and the probe obligation from +> Step 1. +> **Out:** the real lookup loop, and the cache-line budget of one lookup. +> Step 4 supplies the `probe_seq.move_next` this loop calls. + +Because tags are dense bytes, **SIMD** (single instruction, multiple data — +one CPU instruction applied to a vector of lanes) can compare the wanted tag +against a whole **group** of adjacent tags at once, producing a bitmask of +candidate lanes. This is the entire lookup: ```rust -fn find(table: &RawTable, hash: u64, key: &K) -> Option { - let h2 = (hash >> 57) as u8; // top 7 bits = the tag - let mut probe = ProbeSeq::new(h1(hash), table.mask); // triangular stride - loop { - let group = Group::load(&table.ctrl[probe.pos]); // ONE dense cache line - for bit in group.match_tag(h2) { // SIMD: 8–16 tags at once - let slot = (probe.pos + bit) & table.mask; - if table.key(slot) == key { return Some(slot); } // 2nd line: the slot - } - if group.match_empty().any_bit_set() { - return None; // EMPTY stops the probe; DELETED does NOT — - } // the key may have been pushed past a tombstone - probe.move_next(table.mask); - } -} -``` - -False-positive rate: 16 slots × 2⁻⁷ ≈ 16/128 per group — a wasted key -comparison ~12% of the time, cheap. Net cache-line budget per lookup: one -line of control bytes + one line of slot data — the theoretical minimum plus -one dense byte. +// src/raw.rs — find_inner, 2009-2046 (safety comments elided) + 2009 unsafe fn find_inner(&self, hash: u64, eq: &mut dyn FnMut(usize) -> bool) -> Option { + 2010 let tag_hash = Tag::full(hash); + 2011 let mut probe_seq = self.probe_seq(hash); + 2012 + 2013 loop { + // ... 2014-2027: SAFETY comment — pos is masked, and the trailing group is + // always readable because of Step 6's extra Group::WIDTH bytes ... + 2028 let group = unsafe { Group::load(self.ctrl(probe_seq.pos)) }; + 2029 + 2030 for bit in group.match_tag(tag_hash) { + // ... 2031-2032: comment: the & is a modulo, buckets being a power of two ... + 2033 let index = (probe_seq.pos + bit) & self.bucket_mask; + 2034 + 2035 if likely(eq(index)) { + 2036 return Some(index); + 2037 } + 2038 } + 2039 + 2040 if likely(group.match_empty().any_bit_set()) { + 2041 return None; + 2042 } + 2043 + 2044 probe_seq.move_next(self.bucket_mask); + 2045 } + 2046 } +``` + +Four lines carry it. **2028** loads one group of control bytes — 8 bytes on +this machine, 16 under SSE2 — with a single unaligned vector load. **2030** +compares all of them against the wanted tag in one instruction and iterates +only the lanes that matched: on aarch64 that is `vceq_u8` against a splatted +tag plus a reinterpret to a `u64` bitmask (`neon.rs:68-73`); on x86 it is +`_mm_cmpeq_epi8` followed by `_mm_movemask_epi8` (`sse2.rs:73-86`). **2035** +is the only place a real key is compared, and it runs only for lanes that +already matched seven hash bits. **2040** is the stopping rule and the subject +of Step 5: an `EMPTY` anywhere in the group means the key cannot be further +along, so the search ends; a `DELETED` does *not* stop it. + +Now the false-positive rate, which is what earns line 2035 its rarity. A tag +collision needs 7 bits to agree, probability 2⁻⁷ = 1/128 per occupied lane, so +per group load, with the table at its 7/8 limit: + +``` +Group::WIDTH = 8 (NEON / generic — this repo's machine) + occupied lanes ≈ 8 × 7/8 = 7 + E[wasted key compares per group] = 7 / 128 = 0.0547 → 5.5% + +Group::WIDTH = 16 (SSE2) + occupied lanes ≈ 16 × 7/8 = 14 + E[wasted key compares per group] = 14 / 128 = 0.109 → 10.9% +``` + +The earlier version of this chapter quoted the 16-wide figure ("~12% of the +time") without saying which backend it belonged to; on aarch64 it is half +that, because the group is half as wide. + +Cache-line budget per lookup: one line of control bytes (the group load at +2028) plus one line of slot data (the key compare at 2035) — the theoretical +minimum, plus one dense byte per slot. That is the number the flamegraph could +not show you. ### Step 4 — the probe sequence: triangular stride, guaranteed coverage -When a group has neither a match nor an EMPTY, probing moves to another -group. Linear probing (always +1) suffers **clustering** — runs of full slots -grow and merge, lengthening everyone's probes. hashbrown's `ProbeSeq` -(`src/raw.rs:76–93`) grows its stride by one group per step (positions follow -triangular numbers: +1, +2, +3, … groups). The comment links the proof that -triangular probing mod a power of two visits every group exactly once — no -cycling, no missed slots — while spreading clusters out. +> **In:** a group load from Step 3 that contained neither the key nor an +> `EMPTY`. +> **Out:** the next group to load, and the guarantee that repeating this +> visits every group exactly once. Step 6's `bucket_mask` power-of-two +> invariant is what makes the guarantee true. + +**Clustering** is the failure mode of linear probing (always try the next +slot): runs of occupied slots grow, adjacent runs merge, and everyone's probe +gets longer — including keys whose own home slot was free. hashbrown avoids it +by growing the stride: + +```rust +// src/raw.rs — ProbeSeq and its only method, 76-93 + 76 struct ProbeSeq { + 77 pos: usize, + 78 stride: usize, + 79 } + 80 + 81 impl ProbeSeq { + 82 #[inline] + 83 fn move_next(&mut self, bucket_mask: usize) { + // ... 84-88: debug_assert that we have not run past the end of the sequence ... + 90 self.stride = self.stride.wrapping_add(Group::WIDTH); + 91 self.pos = self.pos.wrapping_add(self.stride) & bucket_mask; + 92 } + 93 } +``` + +Line 90 adds one *group width* to the stride each time, and line 91 adds the +new stride to the position — so after k steps the probe sits at + +``` +pos_k = h1 + WIDTH × (1 + 2 + … + k) = h1 + WIDTH × k(k+1)/2 (mod m) + + symbols: h1 = the starting slot (raw.rs:2453, h1(hash) & bucket_mask) + WIDTH = Group::WIDTH (8 here, 16 on SSE2) + k = number of move_next calls m = number of slots +``` + +Those are the **triangular numbers** scaled by the group width, and the +comment at raw.rs:66-74 links Fabian Giesen's proof that triangular numbers +mod 2ⁿ hit every residue exactly once — so with m a power of two the sequence +visits every group exactly once, never cycles early, and never misses a slot. +`probe_seq` starts it at `stride: 0` (raw.rs:2449-2456), so the first +`move_next` jumps a single group and the walk begins as a linear scan. + +Now put Step 1's arithmetic together with Step 3's group width, which is the +whole SwissTable argument in one division. Step 1 said an unsuccessful search +at α = 7/8 examines ~8 slots. Those slots are contiguous within a group, so +the number of *group loads* — the thing that costs a cache miss — is: + +``` + slots examined group loads group loads + load factor α (Step 1: 1/(1−α)) at WIDTH = 8 at WIDTH = 16 + ------------------------------------------------------------------------ + 0.50 2.0 2.0/8 = 0.25 2.0/16 = 0.13 + 0.75 4.0 4.0/8 = 0.50 4.0/16 = 0.25 + 0.875 (hashbrown) 8.0 8.0/8 = 1.00 8.0/16 = 0.50 + 0.9375 16.0 16.0/8 = 2.00 16.0/16 = 1.00 +``` + +At the 7/8 limit an 8-wide group resolves an average miss in **one** group +load; a 16-wide group needs one every other lookup. The classic 50% cap bought +2.0 slot examinations where hashbrown buys 8.0 — and then made them free by +looking at eight at a time. Raising the load factor was not a compromise the +SIMD paid for; it is the thing the SIMD *bought*. ### Step 5 — deletion and tombstones: why DELETED ≠ EMPTY -Open-addressing deletion cannot just mark a slot EMPTY: an EMPTY stops every -probe (Step 3's early exit), so erasing a slot mid-probe-chain would make -keys *beyond* it unfindable. The fix is a **tombstone**: the DELETED tag, -which probes skip over but inserts may reuse. The subtleties -(`src/raw.rs:1952–1984, 1033–1043`): inserting over DELETED doesn't consume -`growth_left` (the tombstone already "spent" its capacity), and a table full -of tombstones triggers **rehash-in-place** — rewriting the control array to -reclaim tombstones without growing. Churn-heavy tables otherwise degrade: -same disease as LSM tombstones, same cure (rewrite/compact). +> **In:** Step 3's stopping rule (`match_empty` ends the search) and Step 2's +> spare tag value. +> **Out:** the erase rule, the condition under which a tombstone is *not* +> written, and the cleanup path a churn-heavy table triggers. + +Open-addressing deletion cannot simply mark a slot `EMPTY`: line 2040 stops +every probe at an `EMPTY`, so erasing a slot in the middle of a probe chain +would make keys *beyond* it unfindable. The classic fix is a **tombstone** — a +marker meaning "occupied once, empty now; keep probing" — which is what +`DELETED` is. + +hashbrown is more careful than the classic fix, and this is the part most +retellings get wrong: + +```rust +// src/raw.rs — inside RawTableInner::erase, 3232-3241 and 3279-3289 + 3232 let index_before = index.wrapping_sub(Group::WIDTH) & self.bucket_mask; + // ... 3233-3235: SAFETY ... + 3236 let (empty_before, empty_after) = unsafe { + 3237 ( + 3238 Group::load(self.ctrl(index_before)).match_empty(), + 3239 Group::load(self.ctrl(index)).match_empty(), + 3240 ) + 3241 }; + // ... 3243-3278: the long comment deriving the rule below — read it ... + 3279 let ctrl = if empty_before.leading_zeros() + empty_after.trailing_zeros() >= Group::WIDTH { + 3280 Tag::DELETED + 3281 } else { + 3282 self.growth_left += 1; + 3283 Tag::EMPTY + 3284 }; + // ... 3285-3286: SAFETY ... + 3287 self.set_ctrl(index, ctrl); + 3288 } + 3289 self.items -= 1; +``` -### Step 6 — two closing tricks: load factor 7/8 and the mirrored tail +Line 3279 is the rule: a tombstone is written **only** when the erased slot +sits inside an unbroken window of `Group::WIDTH` occupied-or-deleted slots. If +any `EMPTY` is within a group's reach on either side, a probe would have +stopped there anyway, so line 3283 writes `EMPTY` instead and line 3282 gives +the capacity back. The consequence the comment spells out at 3273-3275: a +table with fewer buckets than the group width can never contain a tombstone at +all, because `index_before == index` there. + +Insertion knows about tombstones too. `find_insert_index` (raw.rs:1952-1984) +takes the first empty-*or*-deleted lane in a group (`match_empty_or_deleted`, +via `find_insert_index_in_group` at raw.rs:1749-1759), and the accounting is +the subtle bit: + +```rust +// src/raw.rs — inside RawTable::insert, 1031-1043 + 1031 let mut index = self.table.find_insert_index(hash); + 1032 + 1033 // We can avoid growing the table once we have reached our load factor if we are replacing + 1034 // a tombstone. This works since the number of EMPTY slots does not change in this case. + // ... 1035-1036: SAFETY ... + 1037 let old_ctrl = *self.table.ctrl(index); + 1038 if unlikely(self.table.growth_left == 0 && old_ctrl.special_is_empty()) { + 1039 self.reserve(1, hasher); + // ... 1040-1041: SAFETY ... + 1042 index = self.table.find_insert_index(hash); + 1043 } +``` + +Line 1038 reads: grow only if we are out of headroom **and** the slot we are +about to fill was genuinely `EMPTY`. Overwriting a `DELETED` slot costs no +capacity — `record_item_insert_at` decrements `growth_left` only for a slot +that `special_is_empty()` (raw.rs:2459-2460) — because Step 3's stopping rule +depends on the count of `EMPTY` slots, not on the count of free ones, and that +count is unchanged. + +So a churn-heavy table fills with tombstones and hits `growth_left == 0` while +holding far fewer live items than its capacity. The cure is +`reserve_rehash_inner`: + +```rust +// src/raw.rs — inside reserve_rehash_inner, 2756-2757 and 2770-2792 + 2756 let full_capacity = bucket_mask_to_capacity(self.bucket_mask); + 2757 if new_items <= full_capacity / 2 { + // ... 2758-2769: comment and SAFETY ... + 2770 unsafe { + 2771 self.rehash_in_place(hasher, layout.size, drop); + 2772 } + 2773 Ok(()) + 2774 } else { + // ... 2775-2783: "conservatively resize to at least the next size up" ... + 2784 unsafe { + 2785 self.resize_inner( + 2786 alloc, + 2787 usize::max(new_items, full_capacity + 1), + // ... 2788-2791: hasher, fallibility, layout ... + 2792 ) +``` -Load factor: hashbrown allows 7/8 = 87.5% occupancy (`src/raw.rs:152–156`) — -versus 50% for classic open addressing — because group probing checks 16 -slots per step, so even near-full tables resolve in ~1 group. That's 1.14 -bytes of overhead per slot where chaining pays a 16+ byte malloc'd node. +Line 2757 is the decision, and it is a *half*, not a threshold on tombstone +count: if the live items would still fit in half the current capacity, the +table is rewritten in place (2771) — `rehash_in_place` (raw.rs:2985) first +converts every FULL tag to DELETED and every DELETED to EMPTY +(raw.rs:2048-2054) and then re-seats each live entry — and no memory is +allocated. Otherwise it really grows (2785-2792). Same disease as LSM +tombstones (topic 1), same cure: rewrite/compact, with a rule for when the +rewrite is worth it. -The trailing mirror (`src/raw.rs:223`): the control array allocates -`buckets + Group::WIDTH` bytes, the tail replicating the head, so a 16-byte -group load starting near the end never wraps around. Branchless boundary -handling, paid in 16 bytes. +### Step 6 — two closing tricks: the 7/8 rule and the mirrored tail + +> **In:** everything above — the probe loop, the stride, the tombstone rules. +> **Out:** the exact capacity function (which is not 7/8 for small tables) and +> the allocation trick that lets Step 3's group load run off the end of the +> array without a branch. + +The load factor is one function, and it has two cases: + +```rust +// src/raw.rs — bucket_mask_to_capacity, 182-191 + 182 fn bucket_mask_to_capacity(bucket_mask: usize) -> usize { + 183 if bucket_mask < 8 { + 184 // For tables with 1/2/4/8 buckets, we always reserve one empty slot. + 185 // Keep in mind that the bucket mask is one less than the bucket count. + 186 bucket_mask + 187 } else { + 188 // For larger tables we reserve 12.5% of the slots as empty. + 189 ((bucket_mask + 1) / 8) * 7 + 190 } + 191 } +``` + +Line 189 is the famous 7/8 = 87.5%; line 186 is the case the slogan omits. +For 1, 2, 4 or 8 buckets the capacity is `bucket_mask` = buckets − 1, so a +4-bucket table holds 3 entries (75%) and an 8-bucket table holds 7 (87.5%, +which happens to agree). One empty slot must always exist or Step 3's `loop` +at 2013 would never terminate — that is also why `RawTable::new` describes +itself as a table with "exactly 1 bucket" whose data pointer may dangle +(raw.rs:585-587). The old anchor for this function in this chapter was +`raw.rs:152-156`; at `d69025b` it is 182-191. + +Compare the overheads at 7/8, per *entry* rather than per slot, since 1/8 of +the slots are empty: + +``` +control bytes per entry = 1 × 8/7 = 1.143 bytes +u64→u64 slot = 16 bytes = 16 × 8/7 = 18.286 bytes + ------ + total 19.43 bytes per entry + +chaining, same map: 8 B bucket-array pointer (at α = 1.0) + + 24 B node {next, key, value}, which a 16-byte-granular + allocator hands out as a 32 B chunk + ------ + 40.00 bytes per entry + + 40.00 / 19.43 = 2.06× — and one fewer dependent miss +``` + +The trailing mirror is the other trick, and it is one line: + +```rust +// src/raw.rs — inside TableLayout::calculate_layout_for, 216-223 + 216 fn calculate_layout_for(self, buckets: usize) -> Option<(Layout, usize)> { + 217 debug_assert!(buckets.is_power_of_two()); + 218 + 219 let TableLayout { size, ctrl_align } = self; + // ... 220-222: ctrl_offset — round the slot region up to ctrl_align ... + 223 let len = ctrl_offset.checked_add(buckets + Group::WIDTH)?; +``` + +Line 223 allocates `buckets + Group::WIDTH` control bytes rather than +`buckets`, and the tail replicates the head, so the group load at raw.rs:2028 +starting on the last bucket reads real bytes instead of running off the +allocation. Branchless boundary handling, paid for in 8 bytes here and 16 +under SSE2 — and line 217 is the `is_power_of_two` assertion that Step 4's +coverage proof and the `& bucket_mask` at 2033 both rest on. ### Step 7 — naming what stalled in your topic 0 flamegraph -Your flamegraph showed the probe loop fully inlined and memory-stall-bound at -10M keys. Now you can name the stalls: the **control-byte load** is the one -guaranteed miss per probe (dense array, ~1 cache line per group); the slot -touch is the second. h2 filtering exists precisely so there's rarely a -*third*. And the 21% SipHash slice is the price of computing h1/h2 at all — -the hash-policy question your capstone must answer. +> **In:** the probe loop from Steps 3-4 and the layout from Steps 2 and 6. +> **Out:** an account of topic 0's measured 10 M-key lookup in terms of named +> lines — and the one hash-policy question your capstone still has to answer. + +Topic 0's flamegraph showed the probe loop fully inlined into the bench +closure, with **21% of samples inside SipHash** +(`core::hash::sip::Hasher::write`) and the other ~79% attributed to the +inlined loop ([topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md)). +Now you can name the parts: + +- The **21%** is the cost of producing the two things Step 2 splits the hash + into: `h1` (raw.rs:61) for the starting position and `Tag::full` + (tag.rs:35-49) for the tag. Rust's default hasher is SipHash-1-3, chosen for + HashDoS resistance, and it is pure overhead on a u64 key you control. +- The first guaranteed miss is the **control-byte load** at raw.rs:2028, on a + dense array of one byte per slot. +- The second is the **slot touch** inside `eq(index)` at raw.rs:2035, and + Step 3's tag filtering exists precisely so that a *third* line is rarely + needed: 5.5% of the time at `WIDTH = 8`. + +That the whole thing measures **8.8 ns at 1e6 keys and 9.3 ns at 1e7** in +topic 0's `lookup_shootout` — nearly flat across a 10× size increase, on a +~160 MB table where a random probe "should" cost a ~100 ns DRAM miss — is not +a contradiction: those 1024 probes are *independent*, so the out-of-order +window overlaps many misses. A single dependent lookup would be far slower. +Two cache lines per lookup is what makes that overlap possible at all; a +chained table's second miss cannot start until its first has landed. ## Where each step lives in the code | What | Where | Step | |------|-------|------| -| `RawTable` | `src/raw.rs:557` | 1 | -| Tag constants + h2 extraction | `src/control/tag.rs:9–49` | 2 | -| Group dispatch (SSE2/NEON/generic) | `src/control/group/mod.rs:8–46` | 3 | -| **NEON match (your machine)** | `src/control/group/neon.rs:78–90` | 3 | -| Probe sequence (triangular) | `src/raw.rs:76–93` | 4 | -| Insert / tombstone reuse | `src/raw.rs:1952–1984, 1033–1043` | 5 | -| Load factor 7/8 | `src/raw.rs:152–156` | 6 | -| Trailing mirror | `src/raw.rs:223` | 6 | +| `h1` — the starting position, low bits | `src/raw.rs:58-64` | 2 | +| `ProbeSeq` and `move_next` (triangular) | `src/raw.rs:66-93` | 4 | +| `bucket_mask_to_capacity` — 7/8, and the small-table case | `src/raw.rs:182-191` | 6 | +| Trailing mirror: `buckets + Group::WIDTH` ctrl bytes | `src/raw.rs:216-223` | 6 | +| `RawTable` | `src/raw.rs:556-562` | 1 | +| `RawTableInner` + the one-allocation layout comment | `src/raw.rs:564-580` | 2 | +| `insert` — the tombstone/`growth_left` rule | `src/raw.rs:1031-1043` | 5 | +| `find_insert_index_in_group` — `match_empty_or_deleted` | `src/raw.rs:1749-1759` | 5 | +| `find_insert_index` — the insert-side probe loop | `src/raw.rs:1952-1984` | 5 | +| **`find_inner` — the lookup, all of it** | `src/raw.rs:2009-2046` | 3 | +| `rehash_in_place`'s tag conversion (FULL→DELETED→EMPTY) | `src/raw.rs:2048-2054`, `2985` | 5 | +| `probe_seq` — where `stride` starts at 0 | `src/raw.rs:2449-2456` | 4 | +| `record_item_insert_at` — `growth_left` only for EMPTY | `src/raw.rs:2459-2460` | 5 | +| `reserve_rehash_inner` — in-place if `items ≤ capacity/2` | `src/raw.rs:2740-2793` | 5 | +| `erase` — DELETED only inside a full group window | `src/raw.rs:3225-3290` | 5 | +| Tag constants + top-7-bit extraction | `src/control/tag.rs:9-49` | 2 | +| Group backend selection (SSE2 / NEON / LSX / generic) | `src/control/group/mod.rs:8-46` | 3 | +| SSE2 group: `__m128i`, 16 wide, `_mm_movemask_epi8` | `src/control/group/sse2.rs:20`, `73-86` | 3 | +| **NEON group (this repo's machine): `uint8x8_t`, 8 wide** | `src/control/group/neon.rs:16`, `68-73` | 3 | +| Generic group: `u64`, 8 wide on 64-bit | `src/control/group/generic.rs:8-21`, `41` | 3 | Read in this order: -1. **`tag.rs`** — EMPTY/DELETED encoding (Step 2). Why is EMPTY `0xff` and - full tags `0b0xxxxxxx`? (So `match_empty_or_deleted` = "high bit set" — - one SIMD sign test.) -2. **`group/neon.rs:78–90`** — the 8-byte NEON group ops (Apple Silicon path, - Step 3). Note x86 SSE2 gets 16-wide groups; ARM gets 8. Measurable? - (Experiment idea.) -3. **`raw.rs:76–93`** — `ProbeSeq` (Step 4): stride grows by one group per - step (triangular numbers); the comment links the coverage proof. -4. **Insert path `raw.rs:1952`** — find first EMPTY *or* DELETED; tombstone - subtlety at `raw.rs:1033–1043` (Step 5). -5. **Aha: the trailing mirror** — `raw.rs:223` (Step 6). +1. **`tag.rs:9-49`** (Step 2) — the encoding. Ask why `EMPTY` is `0xff` and + full tags are `0b0xxxxxxx`; the answer is at tag.rs:17 and tag.rs:30 — + "special" is one sign test, "empty vs deleted" is one low-bit test. +2. **`group/mod.rs:8-46`** (Step 3) — read the `cfg_if` before either + implementation, so you know which `Group` your build gets. Then + `neon.rs:68-73` and `sse2.rs:73-86` side by side: same function, 8 lanes + against 16. The stale-sounding comment at mod.rs:14-16 says NEON was not + worth it — yet lines 24-33 select it; both paths are 8 bytes wide, so the + choice on aarch64 is between two 8-wide implementations. +3. **`raw.rs:2009-2046`** (Step 3) — `find_inner`, the lookup in 12 real + lines. Trace one hit and one miss by hand. +4. **`raw.rs:66-93`** (Step 4) — `ProbeSeq`; follow the link at line 74 to the + coverage proof. +5. **`raw.rs:3225-3290`** (Step 5) — `erase`. The long comment at 3243-3278 is + the best explanation of tombstones anywhere in the crate; line 3279 is the + rule it derives. +6. **Aha: `raw.rs:223`** (Step 6) — the whole boundary problem, solved by + allocating `Group::WIDTH` more bytes than there are buckets. ## Questions to answer in notes.md -1. Why 7/8 load factor rather than redis's 1.0? (Open addressing degrades near full — - probe lengths explode; chaining just grows chains linearly.) -2. Rust 2018 chose SipHash default for HashMap (DoS resistance) — after this reading - plus the 21% flamegraph number, write the one-paragraph policy for the capstone: - where FxHash/ahash, where SipHash stays. -3. What does DELETED do to a long-lived table with churn? Relate to LSM tombstones — - same problem, same fix (rewrite/compact). +1. Why 7/8 rather than redis's 1.0 (dict.c:1653)? Use Step 1's 1/(1−α) and + Step 4's division: what does α = 15/16 cost in group loads at `WIDTH = 8`, + and what would it cost on an SSE2 build? +2. Rust chose SipHash for `HashMap` (HashDoS resistance). After this reading + plus topic 0's 21% flamegraph slice, write the one-paragraph hash policy + for the capstone: where FxHash/ahash, where SipHash stays, and what + property of the *key source* decides it. +3. What does `DELETED` do to a long-lived table with churn? Trace it through + raw.rs:1038 (the growth check), raw.rs:2757 (the in-place threshold) and + raw.rs:3279 (when a tombstone is even written) — then relate it to LSM + tombstones from topic 1. +4. `erase` writes `EMPTY` rather than `DELETED` whenever an `EMPTY` is within + a group's reach (raw.rs:3279). Construct a small table where the same + deletion writes `DELETED` at `WIDTH = 8` and `EMPTY` at `WIDTH = 16`, and + say which build ends up doing more work later. +5. This repo measured hashbrown's insert path at p50 42 ns, max 58.4 ms + ([FINDINGS.md](../../FINDINGS.md) row 2). Which lines produce the 42 ns, + and which produce the 58.4 ms? (Hint: raw.rs:1038 → raw.rs:2785.) + +## Takeaway + +SwissTable's trick is not "SIMD makes hashing fast" — SipHash is still 21% of +a lookup. It is that a dense one-byte-per-slot filter turns *probe length* +from a memory problem into a register problem, so the table can run at 87.5% +occupancy with about one group load per miss. The group width, and therefore +half the numbers people quote, depends on which backend your target selects. ## Done when -You can draw the control-byte array and narrate one lookup from hash to slot, -including both cache lines it touches. +Answer each before unfolding it. + +- [ ] You can draw the control-byte array and narrate one lookup from hash to slot, naming both cache lines it touches. + +
Answer + + The hash is split once. Its low bits become the starting slot — + `h1(hash) & bucket_mask` (raw.rs:61-64, applied at raw.rs:2453) — and its + top 7 bits become the tag, `Tag::full` at tag.rs:47, with the eighth bit + masked off at tag.rs:48 so the tag is distinguishable from `EMPTY` (0xff) + and `DELETED` (0x80). + + The lookup then loads one *group* of control bytes at that position + (raw.rs:2028) — **first cache line**, 8 bytes on NEON or generic, 16 under + SSE2 — and compares all of them against the tag in one instruction + (raw.rs:2030, `vceq_u8` at neon.rs:70 or `_mm_cmpeq_epi8` at sse2.rs:83). + Only lanes that matched earn a real key comparison at raw.rs:2035 — **second + cache line**, the slot itself, which lives *before* the control pointer in + the same allocation (raw.rs:571-572). If the group holds an `EMPTY` + (raw.rs:2040) the search returns `None`; otherwise `move_next` + (raw.rs:83-92) jumps one more group width than last time and the loop + repeats. + +
+ +- [ ] You can state hashbrown's group width without guessing, and say what changes when it is 8 rather than 16. + +
Answer + + It is `Group::WIDTH`, which is `mem::size_of::()`, and `Group` is + chosen by the `cfg_if` at `src/control/group/mod.rs:8-46`: `__m128i` = **16 + bytes** on x86/x86-64 with SSE2 (sse2.rs:20), `uint8x8_t` = **8 bytes** on + little-endian aarch64 with NEON (neon.rs:16), and `u64` = **8 bytes** in the + portable fallback on any 64-bit target (generic.rs:8-21, 41). This repo's + Apple M3 Pro gets 8. + + Three things change with the width. The false-positive rate per group scales + with it: 7/128 = 5.5% of groups cost a wasted key comparison at width 8 + against 14/128 = 10.9% at width 16. The number of group loads per miss + scales inversely: Step 1's 8.0 slot examinations at α = 7/8 become 8.0/8 = + 1.00 group loads at width 8 and 8.0/16 = 0.50 at width 16. And the probe + stride grows by `Group::WIDTH` per step (raw.rs:90), so the two builds walk + physically different sequences over the same table. + +
+ +- [ ] You can explain why 87.5% occupancy is affordable here and was not affordable for `dense_hash_map` at 50%. + +
Answer + + Because the two designs pay for probe length in different units. Under the + uniform-hashing model an unsuccessful search examines 1/(1−α) slots: + 1/0.5 = 2.0 at α = 0.5 and 1/0.125 = 8.0 at α = 0.875. A table that + examines slots one at a time genuinely pays four times more at the higher + load factor, which is why the classic advice caps α near 0.5. + + hashbrown examines slots `Group::WIDTH` at a time, and those slots are + contiguous, so the cost unit is group loads: 8.0/8 = **1.00** group load per + miss at α = 7/8 on this machine, 8.0/16 = 0.50 under SSE2. The extra + occupancy is nearly free in the currency that matters — cache lines touched + — while saving 1/8 of the slot array plus every per-node malloc. Concretely, + a u64→u64 map costs 19.43 bytes per entry here (18.286 for slots at 8/7 + plus 1.143 for control bytes) against about 40 bytes for a chained table + with 24-byte nodes rounded to 32-byte allocator chunks: 2.06×. + +
+ +- [ ] You can say what `DELETED` is for and when hashbrown declines to write one. + +
Answer + + `DELETED` (tag.rs:12, `0x80`) exists because the probe's stopping rule is + "this group contains an `EMPTY`" (raw.rs:2040). Marking an erased slot + `EMPTY` in the middle of a probe chain would stop searches early and hide + every key that had been pushed past it, so a tombstone is written instead: + probes skip it, inserts may reuse it. + + hashbrown writes one only when it has to. `erase` loads the group ending at + the erased slot and the group starting there (raw.rs:3236-3241), and line + 3279 writes `DELETED` only if `empty_before.leading_zeros() + + empty_after.trailing_zeros() >= Group::WIDTH` — that is, only when the slot + is inside an unbroken window of `Group::WIDTH` non-empty slots. Otherwise a + probe would have stopped at a nearby `EMPTY` anyway, so line 3283 writes + `EMPTY` and line 3282 returns the capacity. A consequence spelled out in the + comment at 3273-3275: tables smaller than the group width never hold a + tombstone at all. + +
+ +- [ ] You can trace how a churn-heavy table recovers its capacity, and name the threshold. + +
Answer + + Filling a tombstone costs no capacity — `record_item_insert_at` decrements + `growth_left` only when the old tag `special_is_empty()` (raw.rs:2459-2460), + and `insert` checks the same thing before deciding to grow (raw.rs:1038), + because Step 3's stopping rule depends on the number of `EMPTY` slots and + overwriting a `DELETED` does not change it. So a table that inserts and + erases repeatedly eventually reaches `growth_left == 0` while holding far + fewer live items than its capacity. + + `reserve_rehash_inner` then decides at raw.rs:2756-2757: if + `new_items <= full_capacity / 2` — the live entries would still fit in half + the current table — it calls `rehash_in_place` (raw.rs:2771), which converts + every FULL tag to DELETED and every DELETED to EMPTY (raw.rs:2048-2054) and + re-seats the live entries without allocating. Otherwise it really grows, to + at least `full_capacity + 1` (raw.rs:2785-2787), with the comment at + 2775-2776 explaining the conservatism: resizing up avoids "churning deletes + into frequent rehashes". + +
+ +- [ ] You can account for topic 0's measured numbers in terms of specific lines of this crate. + +
Answer + + The 21% SipHash slice is the work that produces the two hash derivatives the + table needs: `h1` at raw.rs:61-64 and `Tag::full` at tag.rs:35-49. Nothing + in the probe loop can start until both exist, and on a u64 key you generate + yourself it buys only HashDoS resistance you do not need — which is the + capstone's hash-policy question. + + The remaining ~79% is raw.rs:2009-2046 inlined: the control-byte group load + at 2028 (one cache line), the SIMD compare at 2030, and the key touch at + 2035 (a second cache line). `lookup_shootout` measured 8.8 ns at 1e6 and + 9.3 ns at 1e7 — nearly flat, because the 1024 probes are independent and the + out-of-order window overlaps their misses, which two-independent-lines-per + -lookup makes possible and a chain of dependent loads does not. The insert + side's 58.4 ms max ([FINDINGS.md](../../FINDINGS.md) row 2) comes from the + other branch entirely: raw.rs:1038 finding no headroom, then raw.rs:2785 + allocating and re-seating the whole table. + +
## References **Code** -- [hashbrown](https://github.com/rust-lang/hashbrown) (shallow clone at - `~/repos/hashbrown`) — `src/raw.rs` (RawTable, ProbeSeq, insert path), - `src/control/tag.rs`, `src/control/group/neon.rs` (the Apple Silicon - path; SSE2 sibling for x86) +- [hashbrown](https://github.com/rust-lang/hashbrown) — pinned at **0.17.1** / + `d69025b` (version confirmed in `Cargo.toml:3`). `src/raw.rs` is 4627 lines, + most of it SAFETY commentary; the load-bearing parts are listed below. + +| File | Lines | What | +|------|-------|------| +| `src/raw.rs` | 61-64 | `h1` — the starting position is just the truncated hash | +| `src/raw.rs` | 66-93 | `ProbeSeq`, the triangular stride, and the link to its proof | +| `src/raw.rs` | 182-191 | `bucket_mask_to_capacity` — 7/8, plus the small-table case | +| `src/raw.rs` | 223 | `buckets + Group::WIDTH` control bytes — the trailing mirror | +| `src/raw.rs` | 566-580 | `RawTableInner` and the one-allocation layout comment | +| `src/raw.rs` | 1038 | grow only if out of headroom *and* the slot was truly EMPTY | +| `src/raw.rs` | 1952-1984 | `find_insert_index` — the insert-side probe | +| `src/raw.rs` | 2009-2046 | `find_inner` — the entire lookup | +| `src/raw.rs` | 2459-2460 | `growth_left` accounting, tombstones excluded | +| `src/raw.rs` | 2757 | rehash in place iff the live items fit in half the capacity | +| `src/raw.rs` | 3279 | write DELETED only inside a full group-width window | +| `src/control/tag.rs` | 9-49 | EMPTY / DELETED / top-7-bit tags | +| `src/control/group/mod.rs` | 8-46 | which `Group` your target actually gets | +| `src/control/group/sse2.rs` | 20, 73-86 | 16-wide group, `_mm_movemask_epi8` | +| `src/control/group/neon.rs` | 16, 68-73 | 8-wide group, `vceq_u8` | +| `src/control/group/generic.rs` | 8-21, 41 | 8-wide portable fallback (`u64`) | + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 2 — hashbrown insert p50 42 ns, max + 58.4 ms; the max is the resize at raw.rs:2785, not the probe loop. +- [FINDINGS.md](../../FINDINGS.md) row 0 — the ~1 / 5 / 100 ns cache ladder, + and the 21% SipHash slice of a lookup. +- [topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md) + — `lookup_shootout`: HashMap 7.4 ns at n=100 rising only to 9.3 ns at n=1e7. + +**Companion chapters** +- [reading-swisstable-talk.md](reading-swisstable-talk.md) — the design + narrative that produced this code, told as a sequence of rejected designs. +- [reading-redis-dict.md](reading-redis-dict.md) — the chaining family, and + the incremental rehash hashbrown deliberately does not do. diff --git a/topics/02-in-memory-structures/reading-redis-dict.md b/topics/02-in-memory-structures/reading-redis-dict.md index 6e66f86..64c9bbf 100644 --- a/topics/02-in-memory-structures/reading-redis-dict.md +++ b/topics/02-in-memory-structures/reading-redis-dict.md @@ -10,20 +10,35 @@ two-table dance fixes it — then hands you the line anchors to watch each piece in the source. It is also the design you'll replicate in this topic's experiment. +Every anchor below is Redis **8.6.2** (`src/version.h:1`), the commit +`a176d1225` this repo pins, quoted with the line numbers the code occupies in +that version. Re-check any of them yourself with +`tools/pinned-source.py show redis src/dict.c -r 405:434`. + ## The problem in one sentence -Doubling a hash table is O(n) work done inside *one* insert — at 100M entries -and ~100 ns per entry moved, that single insert takes **~10 seconds** while -every other client waits. +Doubling a hash table is O(n) work done inside *one* insert — and this repo +has measured what that feels like: hashbrown's `rehash_spike` lane inserts +10 M keys one at a time and reports p50 **42 ns** with a max of **58.4 ms** +([FINDINGS.md](../../FINDINGS.md) row 2), a 1.4-millionfold spread inside a +single operation type, on a table 12× smaller than the 100M-key case redis +has to survive. ## The concepts, step by step ### Step 1 — a chained hash table: buckets of linked lists +> **In:** nothing yet — this step fixes the vocabulary and the cost model +> every later step reasons with. +> **Out:** the structure redis actually implements, and the one number +> (chain length) that decides what a lookup costs. Step 2 turns that number +> into the reason the table must grow. + A hash table stores key→value pairs so that lookup costs ~constant time: run the key through a **hash function** (a function mapping any key to a well-scrambled fixed-size integer), keep the low bits as an index into an -array of **buckets**, and put the entry there. Two keys landing in the same +array of **buckets** (the fixed-size slot array; each slot holds the head of +whatever landed there), and put the entry there. Two keys landing in the same bucket is a **collision**; **chaining** resolves it by making each bucket a linked list of entries: @@ -40,176 +55,706 @@ linked list of entries: ``` The cache cost (topic 0): every hop down a chain is a **dependent load** — -the next address comes from the previous node — so each hop is a potential -~100 ns DRAM miss that nothing can prefetch. Chains must stay short. +the address of the next node is only known once the current node has arrived, +so nothing can prefetch it and nothing can overlap two hops. Topic 0's +`cache_ladder` measured that ladder at ~1 ns (L1) / ~5 ns (L2) / ~100 ns +(DRAM) on this machine ([FINDINGS.md](../../FINDINGS.md) row 0), so a chain +hop that misses to DRAM costs ~100 ns and a chain of three costs ~300 ns that +no amount of instruction-level parallelism can hide. Chains must stay short. ### Step 2 — load factor: why the table must grow -The **load factor** is entries ÷ buckets. With a decent hash function, the -average chain length ≈ the load factor, so at load factor 1.0 a lookup walks -~1–2 nodes; at 10.0 it walks ~10 — ten dependent misses, ~1 µs per lookup. -The only fix is more buckets: allocate a bigger array (redis doubles: sizes -are powers of two, stored as exponents) and move every entry to its new -bucket. Moving is mandatory because the bucket index is `hash & (size − 1)` -— change the size and half the entries belong somewhere else. That move is -the **rehash**. +> **In:** the chained table from Step 1. +> **Out:** the growth trigger (`ht_used[0] >= size`), and the arithmetic that +> says what a lookup costs at each load factor. Step 3 prices the growth +> itself. + +The **load factor** — written α — is entries ÷ buckets: α = n/m for n entries +in m buckets. With a hash function that spreads keys evenly, the expected +number of entries examined by a *successful* lookup in a chained table is + +``` +E[entries examined] = 1 + α/2 + + symbols: α = n/m, the load factor + n = entries stored m = buckets allocated + reading: you always examine the one you find (the 1), plus on average + half of the others sharing its bucket (α/2) +``` + +Worked on redis's two thresholds, with the ~100 ns dependent-miss cost from +Step 1 and one extra miss for the bucket array itself: + +``` +α = 1.0 (the normal grow trigger, dict.c:1653) + entries examined = 1 + 1.0/2 = 1.5 + misses = 1 (bucket array) + 1.5 (chain) = 2.5 + cold cost = 2.5 × 100 ns = 250 ns + +α = 4.0 (dict_force_resize_ratio, dict.c:45 — the ceiling redis tolerates + while a fork child is alive, Step 7) + entries examined = 1 + 4.0/2 = 3.0 + misses = 1 + 3.0 = 4.0 + cold cost = 4.0 × 100 ns = 400 ns + +α = 10.0 (an un-resizable table left to degrade) + entries examined = 1 + 5.0 = 6.0 + misses = 7.0 cold cost = 700 ns +``` + +So the penalty is linear in α, not catastrophic — chaining degrades +gracefully, which is exactly why redis can afford to let α reach 4 during a +fork. But 700 ns against 250 ns is still a 2.8× lookup regression, and the +only fix is more buckets: allocate a bigger array (redis doubles — sizes are +powers of two, stored as *exponents* in `ht_size_exp`, so "size" is +`1 << exp`) and move every entry to its new bucket. Moving is mandatory +because the bucket index is `hash & (size − 1)`: change the size and half the +entries belong somewhere else. That move is the **rehash**. ### Step 3 — the stop-the-world rehash is a latency outage +> **In:** the obligation to move all n entries, from Step 2. +> **Out:** the cost of doing it inside one operation, in seconds, on two +> different per-entry assumptions — the number Step 4's design exists to +> avoid. + The textbook rehash happens inside whichever insert crosses the threshold: that one operation allocates the new array and moves all n entries before -returning. Almost every insert costs ~100 ns; one insert costs ~10 s at 100M -entries. Throughput barely notices (the O(n) is amortized); **tail latency** -(the slowest percentiles — p99.9, max — the numbers a server promises its -clients) is destroyed. A redis instance frozen for 10 seconds has failed -every health check and dropped every client. The fix cannot be "rehash -faster"; it must be "never do all the work in one operation." +returning. **Tail latency** — the slowest percentiles, p99.9 and max, the +numbers a server actually promises its clients — is what this destroys, while +throughput barely notices, because the O(n) is amortized over the n inserts +that preceded it. + +Put a number on it. This repo's `rehash_spike` lane gives a *measured* +per-entry cost for the friendliest possible case — hashbrown's flat, +malloc-free array, swept linearly. Assume the 58.4 ms maximum +([FINDINGS.md](../../FINDINGS.md) row 2, eighth decile) is the doubling that +happens when the table crosses 2²³ = 8,388,608 buckets, which at hashbrown's +7/8 load rule holds 7,340,032 live entries when it fires: + +``` +measured spike 58.4 ms = 58,400,000 ns +entries moved 2²³ × 7/8 = 7,340,032 +per entry 58,400,000 / 7,340,032 = 7.96 ns + +redis's dict at 100M entries, same 7.96 ns/entry (optimistic — this assumes +chained nodes sweep as cheaply as a flat array, which they do not): + 100,000,000 × 7.96 ns = 0.796 s + +redis's dict at 100M entries, one dependent DRAM miss per chained node +(Step 1's ~100 ns, the honest number for malloc'd chain nodes): + 100,000,000 × 100 ns = 10.0 s +``` + +Both are outages. Almost every insert costs ~100 ns; this one costs between +0.8 and 10 seconds. A redis instance frozen for even the optimistic 0.8 s has +blown through every sane health-check timeout; at 10 s it has dropped every +client. The fix cannot be "rehash faster" — a 12× speedup still leaves an +0.8 s stall. It must be "never do all the work in one operation." ### Step 4 — the fix: two tables and a migration cursor +> **In:** the outage from Step 3, and the load-factor trigger from Step 2. +> **Out:** the five fields of `struct dict` that make a half-migrated table a +> legal state — the state Steps 5 and 6 operate on. + Redis keeps **both** the old and new bucket arrays alive during the resize -and migrates gradually. `dict.h:143–159` — the whole design in one struct: +and migrates gradually. The whole design is visible in one struct: ```c -struct dict { - dictType *type; - void **ht_table[2]; // ht[0] = old, ht[1] = new (during rehash) - unsigned long ht_used[2]; - long rehashidx; // -1 = not rehashing; else next bucket to migrate - int16_t pauserehash; - signed char ht_size_exp[2]; // sizes as exponents: size = 1 << exp -}; +// src/dict.h — struct dict, 143-159 (the whole design; every field matters) + 143 struct dict { + 144 dictType *type; + 145 + 146 dictEntry **ht_table[2]; + 147 unsigned long ht_used[2]; + 148 + 149 long rehashidx; /* rehashing not in progress if rehashidx == -1 */ + 150 + 151 /* Note: pauserehash is a full unsigned so iterator increments + 152 * don't perform RMW on the same storage unit as other bitfields. */ + 153 unsigned pauserehash; /* If >0 rehashing is paused */ + 154 + 155 /* Keep small vars at end for optimal (minimal) struct padding */ + 156 signed char ht_size_exp[2]; /* exponent of size. (size = 1<0 automatic resizing is disallowed (<0 indicates coding error) */ + 158 void *metadata[]; + 159 }; ``` -`rehashidx` is a cursor sweeping ht[0] from bucket 0 upward: everything below -it has already moved to ht[1], everything above hasn't. Every normal -operation nudges the cursor forward one bucket: +The line to look at is 149. `rehashidx` is a cursor sweeping ht[0] from +bucket 0 upward: every bucket *below* it has already moved to ht[1], every +bucket at or above it has not, and `-1` means no migration is in progress at +all. Line 146 is the pair of bucket arrays (ht[0] = old, ht[1] = new during a +rehash), 147 their live counts, 156 their sizes as exponents. Line 153's +`pauserehash` is the "hold still, someone is iterating me" brake (Step 8). + +Every normal operation nudges the cursor forward: ```mermaid flowchart LR - OP["any dictAdd/dictFind
dict.c:635 / dict.c:779"] --> STEP["_dictRehashStepIfNeeded
dict.c:1705"] - STEP --> RH["dictRehash(d, 1)
dict.c:405 — move 1 bucket
ht[0]→ht[1]"] - RH --> DONE{"ht[0] empty?"} - DONE -- yes --> SWAP["free ht[0], ht[1]→ht[0]
rehashidx = -1"] + OP["any dictAddRaw / dictFind
dict.c:526 / dict.c:800"] --> HOOK["_dictRehashStepIfNeeded(d, idx)
dict.c:1705"] + HOOK -- "the bucket you are
already touching" --> BR["_dictBucketRehash(d, idx)
dict.c:473 — cache-friendly"] + HOOK -- "otherwise" --> RH["dictRehash(d, 1)
dict.c:405 — one bucket at the cursor"] + BR --> DONE{"ht_used[0] == 0?"} + RH --> DONE + DONE -- yes --> SWAP["dictCheckRehashingCompleted
dict.c:380 — free ht[0],
ht[1] becomes ht[0], rehashidx = -1"] DONE -- no --> OP ``` +Note the fork at the hook, which the old version of this chapter missed: if +the bucket your operation is *already* going to touch still lives in ht[0], +redis migrates *that* bucket (dict.c:1709-1712) rather than the one under the +cursor, because that memory is about to be in cache anyway. Only when the +visited bucket is already migrated or empty does it fall back to +`dictRehash(d,1)` at the cursor (dict.c:1716). + The O(n) rehash still happens — but as n tiny installments, each attached to an operation that was paying a hash-table visit anyway. ### Step 5 — one migration step, and why its work is bounded -A step moves one bucket: walk its chain, re-hash every entry into ht[1] -(entries move a *bucket* at a time, not one entry). The subtle hazard is a -**sparse** old table — if most buckets are empty, "move one bucket" could -scan thousands of empty slots looking for a non-empty one, silently breaking -the bounded-work-per-operation guarantee. Redis caps that scan at 10 empty -buckets per requested bucket (`empty_visits`, dict.c:406). The whole machine, -distilled: +> **In:** the half-migrated two-table state from Step 4. +> **Out:** a hard bound on the work one operation can be charged, in buckets +> — the tail-latency guarantee that replaces Step 3's 0.8-to-10-second stall. + +A step moves one *bucket*: walk its chain, re-hash every entry into ht[1]. +The subtle hazard is a **sparse** old table — if most buckets are empty, +"move one bucket" could scan thousands of empty slots looking for a non-empty +one, silently breaking the bounded-work guarantee. Redis caps that scan: + +```c +// src/dict.c — dictRehash, 405-434 (the whole bounded-work loop) + 405 int dictRehash(dict *d, int n) { + 406 int empty_visits = n*10; /* Max number of empty buckets to visit. */ + // ... 407-419: the DICT_RESIZE_FORBID / DICT_RESIZE_AVOID gates of Step 7 ... + 420 while(n-- && d->ht_used[0] != 0) { + // ... 421-423: assert rehashidx is still inside ht[0] ... + 424 while(d->ht_table[0][d->rehashidx] == NULL) { + 425 d->rehashidx++; + 426 if (--empty_visits == 0) return 1; + 427 } + 428 /* Move all the keys in this bucket from the old to the new hash HT */ + 429 rehashEntriesInBucketAtIndex(d, d->rehashidx); + 430 d->rehashidx++; + 431 } + 432 + 433 return !dictCheckRehashingCompleted(d); + 434 } +``` + +Line 426 is the one that carries the guarantee: after ten fruitless bucket +loads the function gives up and returns 1 ("still rehashing"), having done +bounded work. Line 406 sets the budget at `n*10` for a request of n buckets, +so the single-bucket call every operation makes (`dictRehash(d,1)`, +dict.c:469 and dict.c:1716) can touch at most ten empty buckets plus one +chain. + +The actual moving is one level down, and it is worth reading because of what +it does *not* do: + +```c +// src/dict.c — rehashEntriesInBucketAtIndex, 336-352 and 368-377 + 336 static void rehashEntriesInBucketAtIndex(dict *d, uint64_t idx) { + 337 dictEntry *de = d->ht_table[0][idx]; + // ... 338-339: locals ... + 340 while (de) { + 341 nextde = dictGetNext(de); + 342 void *storedKey = dictGetKey(de); + 343 /* Get the index in the new hash table */ + 344 if (d->ht_size_exp[1] > d->ht_size_exp[0]) { + 345 const void *key = dictStoredKey2Key(d, storedKey); + 346 h = dictGetHash(d, key) & DICTHT_SIZE_MASK(d->ht_size_exp[1]); + 347 } else { + 348 /* We're shrinking the table. The tables sizes are powers of + 349 * two, so we simply mask the bucket index in the larger table + 350 * to get the bucket index in the smaller table. */ + 351 h = idx & DICTHT_SIZE_MASK(d->ht_size_exp[1]); + 352 } + // ... 353-370: the no_value key-inlining cases; all end at ht_table[1][h] ... + 371 d->ht_table[1][h] = de; + 372 d->ht_used[0]--; + 373 d->ht_used[1]++; + 374 de = nextde; + 375 } + 376 d->ht_table[0][idx] = NULL; +``` + +Line 371 is the move: the entry is *relinked*, not copied — chaining's one +structural gift, since the payload never changes address. Note the asymmetry +at 344-352: growing re-hashes the key (346, and that recomputation is why the +per-entry cost is nearer Step 3's 100 ns than its 8 ns), while **shrinking +just masks the old index** (351), because a smaller power-of-two mask is a +prefix of a larger one. Line 376 empties the source bucket, which is what +makes the cursor's "everything below me has moved" invariant true. + +The machine, distilled to the shape you will re-implement: ```rust +// ILLUSTRATION — not quoted from redis. The real loop is dict.c:405-434 and +// the per-bucket move is dict.c:336-377; this is the same algorithm with the +// entry-encoding cases (dict.c:353-370) removed. fn rehash_step(d: &mut Dict, mut buckets: usize) { - let mut empty_visits = buckets * 10; // cap the sparse-table scan + let mut empty_visits = buckets * 10; // dict.c:406 while buckets > 0 && d.used[0] > 0 { while d.ht[0].bucket(d.rehashidx).is_empty() { d.rehashidx += 1; empty_visits -= 1; - if empty_visits == 0 { return; } // bounded work per op — the point + if empty_visits == 0 { return; } // dict.c:426 — the bound } for entry in d.ht[0].take_bucket(d.rehashidx) { - let idx = entry.hash & d.mask[1]; // re-hash into the NEW table only + let idx = entry.hash & d.mask[1]; // dict.c:346 — NEW table only d.ht[1].push_bucket(idx, entry); } d.rehashidx += 1; buckets -= 1; } - if d.used[0] == 0 { d.swap_tables(); d.rehashidx = -1; } + if d.used[0] == 0 { d.swap_tables(); d.rehashidx = -1; } // dict.c:380-394 } -// every dictAdd/dictFind calls rehash_step(d, 1) — and during the migration, -// every lookup must check BOTH tables ``` -Worst case per operation: one bucket chain moved + 10 empty visits. That is -the tail-latency guarantee, in buckets. +Now price the guarantee against Step 3's stall, at α = 1.0 (so a non-empty +chain holds ~1.5 entries, Step 2): + +``` +worst case for one operation = 10 empty bucket loads + 1 chain of ~1.5 entries + ≈ 11.5 dependent misses × 100 ns ≈ 1.15 µs + +against the one-shot rehash's measured 58.4 ms: + 58,400,000 ns / 1,150 ns ≈ 50,800× smaller +``` + +The bill does not disappear, it is *spread*. Migrating a 2²³-bucket table one +bucket per operation needs 8,388,608 operations to finish; at the 100K ops/s +of the opening sentence that is 8,388,608 / 100,000 = **83.9 seconds** during +which the dict is in the two-table state and every lookup pays Step 6's tax. +That is the trade: a 58 ms cliff becomes 84 seconds of a slightly slower +table. ### Step 6 — correctness during the migration: who pays the tax -While `rehashidx != -1` the key you want may legitimately be in either table, -so **every lookup checks both** (old first, then new) — the read tax. Writes -follow one rule: **new keys go only to ht[1]**. Inserting into ht[0] would be -a correctness bug, not just waste — if the entry lands in a bucket the cursor -has already passed, it will never be migrated and vanishes when ht[0] is -freed. When ht[0] empties, it is freed, ht[1] becomes ht[0], and the dict is -back to single-table operation. Cost model: rehash O(n) total, amortized O(1) -per op, and no operation ever stalls for more than one chain + 10 empty -visits. +> **In:** the partially migrated table Step 5 leaves behind, with `rehashidx` +> somewhere in the middle of ht[0]. +> **Out:** the two rules — where reads look, where writes land — that make +> that state safe, and what each costs. + +**Rule 1: a read may have to check both tables.** The key you want could +legitimately be in either. The lookup path is `dictFindLinkInternal`, and the +loop is more careful than "check ht[0], then ht[1]": + +```c +// src/dict.c — inside dictFindLinkInternal, 778-796 + 778 /* Rehash the hash table if needed */ + 779 _dictRehashStepIfNeeded(d,idx); + 780 + 781 int tables = (dictIsRehashing(d)) ? 2 : 1; + 782 for (table = 0; table < tables; table++) { + 783 if (table == 0 && (long)idx < d->rehashidx) continue; + 784 idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[table]); + 785 + 786 link = &(d->ht_table[table][idx]); + 787 if (bucket) *bucket = link; + 788 while(link && *link) { + // ... 789-794: compare the stored key, walk to the next link ... + 795 } + 796 } +``` + +Line 783 is the line to focus on, and it corrects the simple story: ht[0] is +skipped entirely when the target bucket sits *below* the cursor, because +everything below the cursor has already been migrated (Step 5's invariant, at +dict.c:376). So the read tax is not "always two lookups" — it is two lookups +only for keys whose ht[0] bucket the cursor has not yet reached, and it +shrinks to zero as the migration advances. Line 781 is the other half: when +`rehashidx == -1` the loop runs once and there is no tax at all. + +**Rule 2: a new key goes only into ht[1].** This is not an optimization, it +is a correctness requirement: + +```c +// src/dict.c — inside dictInsertKeyAtLink, 545-549 + 545 /* If rehashing is ongoing, we insert in table 1, otherwise in table 0. + 546 * Assert that the provided bucket is the right table. */ + 547 int htidx = dictIsRehashing(d) ? 1 : 0; + 548 assert(bucket >= &d->ht_table[htidx][0] && + 549 bucket <= &d->ht_table[htidx][DICTHT_SIZE_MASK(d->ht_size_exp[htidx])]); +``` + +Line 547 decides it, and `dictFindLinkForInsert` hands over a bucket in the +same table (dict.c:1766). Why a bug and not merely waste: if a new entry +landed in an ht[0] bucket the cursor had already passed, nothing would ever +migrate it — `rehashidx` only moves forward — and `dictCheckRehashingCompleted` +frees ht[0] wholesale at dict.c:386 the moment `ht_used[0]` hits zero. The +key would be silently lost. (`ht_used[0]` would also never be decremented for +it, so in practice the dict would instead never *finish* rehashing; either +way the invariant "below the cursor, ht[0] is empty forever" is what the rule +protects.) + +When ht[0] empties, `dictCheckRehashingCompleted` (dict.c:380-394) frees it, +copies ht[1] into slot 0, and sets `rehashidx = -1`. Cost model for the whole +scheme: O(n) total rehash work, amortized O(1) per operation, and no single +operation ever stalls for more than one chain plus ten empty visits. ### Step 7 — the resize policy, and a durability interaction -Growth triggers at load factor 1.0 (`ht_used >= size`, dict.c:1638) when -resizing is enabled. The interesting wrinkle: redis *disables* resizing -during fork-based persistence (BGSAVE), because a fork shares memory pages -copy-on-write (parent and child share physical pages until one writes; a -write copies the whole page) — and a rehash touches every entry, forcing a -copy storm of nearly the entire dataset. But an un-resizable table under -write load degrades (Step 2), so a *forced* grow still fires at -`dict_force_resize_ratio` (dict.c:1655). A data-structure knob tuned by a -durability mechanism — worth pausing on. +> **In:** the load-factor trigger from Step 2 and the migration machine from +> Steps 4-6. +> **Out:** the three-state global policy that decides *when* a migration is +> allowed to start at all — and the reason a persistence mechanism gets a +> vote on a data-structure parameter. + +The growth decision lives in one function: + +```c +// src/dict.c — inside dictExpandIfNeeded, 1648-1661 + 1648 /* If we reached the 1:1 ratio, and we are allowed to resize the hash + 1649 * table (global setting) or we should avoid it but the ratio between + 1650 * elements/buckets is over the "safe" threshold, we resize doubling + 1651 * the number of buckets. */ + 1652 if ((dict_can_resize == DICT_RESIZE_ENABLE && + 1653 d->ht_used[0] >= DICTHT_SIZE(d->ht_size_exp[0])) || + 1654 (dict_can_resize != DICT_RESIZE_FORBID && + 1655 d->ht_used[0] >= dict_force_resize_ratio * DICTHT_SIZE(d->ht_size_exp[0]))) + 1656 { + 1657 if (dictTypeResizeAllowed(d, d->ht_used[0] + 1)) + 1658 dictExpand(d, d->ht_used[0] + 1); + 1659 return DICT_OK; + 1660 } + 1661 return DICT_ERR; +``` + +Line 1653 is the normal trigger: α ≥ 1.0. Line 1655 is the escape hatch: +even when resizing is discouraged, α ≥ `dict_force_resize_ratio` (= 4, +dict.c:45) forces it anyway — Step 2's arithmetic says that is a 400 ns +lookup against 250 ns, a degradation redis will tolerate but not exceed. + +What sets `dict_can_resize` is the interesting part, and it is not in dict.c +at all: + +```c +// src/server.c — updateDictResizePolicy, 778-785 + 778 void updateDictResizePolicy(void) { + 779 if (server.in_fork_child != CHILD_TYPE_NONE) + 780 dictSetResizeEnabled(DICT_RESIZE_FORBID); + 781 else if (hasActiveChildProcess()) + 782 dictSetResizeEnabled(DICT_RESIZE_AVOID); + 783 else + 784 dictSetResizeEnabled(DICT_RESIZE_ENABLE); + 785 } +``` + +Three states, not two — the old version of this chapter said redis "disables +resizing during BGSAVE", and line 782 says otherwise. **Copy-on-write** is +the mechanism behind it: `fork()` gives the child a logical copy of the +parent's memory by sharing the physical pages read-only, and the kernel +copies a page only when one side writes to it. A rehash writes to nearly +every page holding entries, so a resize during a background save can force a +copy of most of the dataset — the parent's RSS balloons toward 2× while the +child writes an RDB file. + +So: line 780, inside the forked child itself, resizing is **forbidden** +outright (the child is a snapshot; there is nothing to gain). Line 782, in +the *parent* while any child runs, it is **avoided** — meaning the α ≥ 1.0 +trigger at 1653 is switched off but the α ≥ 4 force at 1655 stays live, and +`dictRehash` refuses to advance a migration that would not have qualified +under the same rule (dict.c:413-418). A durability mechanism tuning a +data-structure knob, with a bounded degradation (Step 2: 250 ns → 400 ns) +chosen as the price. Worth pausing on. ### Step 8 — iterating a table that rehashes under you: dictScan -The last piece: SCAN must iterate the keyspace across many calls, while -buckets migrate and the table may grow between calls. `dictScan` -(dict.c:1518) increments its cursor in **reversed bit order** -(dict.c:1579–1615). The property that makes it work: because bucket index is -the hash's low bits, the entries of bucket `b` at size 2^n split across -buckets `b` and `b + 2^n` at size 2^(n+1) — and reverse-binary increment -visits those siblings adjacently, so buckets already visited at one size map -onto already-visited buckets at the next. Guarantee: every key present for -the whole scan is returned ≥ once (duplicates possible, misses not). Read -the long comment above the function — one of the great comments in open +> **In:** everything above — a table that may be half-migrated *and* may +> change size between two calls of the iterator. +> **Out:** the reverse-binary cursor and the exact guarantee it buys +> (every key present throughout is returned at least once; duplicates are +> possible). + +SCAN must iterate the keyspace across many separate calls, holding no state +between them but a single integer cursor, while buckets migrate and the table +may double or halve in between. Redis's answer, designed by Pieter Noordhuis +and explained in the comment at dict.c:1434-1517, is to increment the cursor +in **reversed bit order** — reverse the bits, add one, reverse back: + +```c +// src/dict.c — inside dictScanDefrag, the non-rehashing branch, 1574-1587 + 1574 if (!dictIsRehashing(d)) { + 1575 htidx0 = 0; + 1576 m0 = DICTHT_SIZE_MASK(d->ht_size_exp[htidx0]); + 1577 dictScanDefragBucket(d, fn, defragfns, privdata, &d->ht_table[htidx0][v & m0]); + 1578 + 1579 /* Set unmasked bits so incrementing the reversed cursor + 1580 * operates on the masked bits */ + 1581 v |= ~m0; + 1582 + 1583 /* Increment the reverse cursor */ + 1584 v = rev(v); + 1585 v++; + 1586 v = rev(v); + 1587 +``` + +Lines 1584-1586 are the whole trick (`rev` itself is dict.c:1424-1432). The +property that makes it work: because a bucket index is the hash's *low* bits, +the entries of bucket `b` at size 2ⁿ split across exactly buckets `b` and +`b + 2ⁿ` at size 2ⁿ⁺¹ — and a reverse-binary increment visits `b` and +`b + 2ⁿ` adjacently, so a bucket already visited at one size maps onto +already-visited buckets at the next. + +Work the four-bit case the comment describes (mask 1111, size 16), counting +in reverse-binary order: + +``` +normal counting: 0000 0001 0010 0011 0100 … (low bit varies fastest) +reverse counting: 0000 1000 0100 1100 0010 … (HIGH bit varies fastest) + +grow 16 → 64 after visiting 1100: + the keys of bucket 1100 are now in 001100, 011100, 101100, 111100 + reverse counting from 1100 never re-emits a cursor ending in 1100, + because those two low bits are the LAST ones it will vary + ⇒ already-scanned work is never repeated, and nothing is skipped +``` + +The rehashing branch (dict.c:1588-1615) reduces the two-table case to the +one-table case: scan the smaller table's bucket, then every bucket of the +larger table that is an expansion of it (1605-1615). And line 1572 explains +`pauserehash`'s other job — the scan pauses rehashing across the callback, in +case the callback itself calls `dictFind` and moves buckets underneath the +iterator. + +Guarantee, stated at dict.c:1443-1445: every element present in the dict for +the whole scan is returned **at least once**; some may be returned more than +once. Read the full comment (1434-1517) — one of the great comments in open source. ## Where each step lives in the code -- **Step 4** — the struct: `dict.h:143–159`; the piggyback hook - `_dictRehashStepIfNeeded` — dict.c:1705. -- **Step 5** — `dictRehash` — dict.c:405; read the whole function (~50 - lines): `empty_visits = n*10` at dict.c:406, the per-bucket chain walk and - re-hash into ht[1] at dict.c:420–431. -- **Step 6** — the payers: `dictAddRaw` — dict.c:635; `dictFind` — - dict.c:779; `dictAddOrFind` — dict.c:1742. Verify both rules in the source: - lookups probe both tables, inserts go to ht[1] only. -- **Step 7** — resize policy — dict.c:1638; forced grow at - `dict_force_resize_ratio` — dict.c:1655. -- **Step 8** — `dictScan` — dict.c:1518; the reverse-binary increment at - dict.c:1579–1615, spec'd by the comment above it. -- **Contrast case**: valkey's client-side dict — - [`~/repos/valkey/deps/libvalkey/src/dict.c`](https://github.com/valkey-io/valkey), - dict.c:103–150 — a *single-table*, full-rehash dict: no rehashidx, no - two-table dance. Fine for a client's small maps; unacceptable for a - server's keyspace. Same structure, different RUM position — latency - requirements are part of the workload. +`src/dict.c` is 2340 lines at `a176d1225`; you need about 400 of them. + +| Lines | What | Step | +|-------|------|------| +| `dict.h:143-159` | `struct dict` — two tables, `rehashidx`, `pauserehash`, sizes as exponents | 4 | +| `dict.h:193-194` | `DICT_HT_INITIAL_EXP` = 2, so a fresh table has 4 buckets | 2 | +| `dict.h:214-216` | `dictPauseRehashing` / `dictResumeRehashing` / `dictIsRehashingPaused` | 8 | +| `dict.c:44-45` | `dict_can_resize`, `dict_force_resize_ratio` = 4 | 7 | +| `dict.c:336-377` | `rehashEntriesInBucketAtIndex` — the actual chain relink; re-hash on grow (346), mask on shrink (351) | 5 | +| `dict.c:380-394` | `dictCheckRehashingCompleted` — free ht[0], promote ht[1], `rehashidx = -1` | 5, 6 | +| `dict.c:405-434` | `dictRehash` — the bounded loop; `empty_visits = n*10` (406), the bail-out (426) | 5 | +| `dict.c:446-458` | `dictRehashMicroseconds` — the *other* client, a time-budgeted rehash from the server cron | 5 | +| `dict.c:468-470` | `_dictRehashStep` — `dictRehash(d,1)`, skipped while paused | 4 | +| `dict.c:473-490` | `_dictBucketRehash` — migrate the bucket you are already touching | 4 | +| `dict.c:526-536` | `dictAddRaw` — the insert path (was cited as 635 here; it is 526) | 6 | +| `dict.c:542-549` | `dictInsertKeyAtLink` — `htidx = rehashing ? 1 : 0`, the write rule | 6 | +| `dict.c:613-617` | `dictAddOrFind` (was cited as 1742; that line is inside `dictFindLinkForInsert`) | 6 | +| `dict.c:761-798` | `dictFindLinkInternal` — the two-table read, and the skip at 783 | 6 | +| `dict.c:800-804` | `dictFind` (was cited as 779) | 6 | +| `dict.c:1149-1157`, `1173-1197` | safe iterators: pause on first `dictNext` (1179), resume on reset (1153) | 8 | +| `dict.c:1424-1432` | `rev()` — the bit-reversal itself | 8 | +| `dict.c:1434-1517` | the `dictScan` comment — read it in full | 8 | +| `dict.c:1518-1524`, `1560-1621` | `dictScan` → `dictScanDefrag`; reverse increments at 1584-1586 and 1608-1612 | 8 | +| `dict.c:1638-1662` | `dictExpandIfNeeded` — α ≥ 1.0 (1653), forced α ≥ 4 (1655) | 7 | +| `dict.c:1705-1718` | `_dictRehashStepIfNeeded` — the piggyback hook and its bucket/cursor fork | 4 | +| `dict.c:1733-1768` | `dictFindLinkForInsert` — same two-table walk, returns an ht[1] bucket (1766) | 6 | +| `server.c:778-785` | `updateDictResizePolicy` — FORBID / AVOID / ENABLE | 7 | + +Suggested route: the struct (`dict.h:143`) → `dictRehash` (405) → +`rehashEntriesInBucketAtIndex` (336) → the hook (1705) → the two payers, +`dictFindLinkInternal` (761) and `dictInsertKeyAtLink` (542) → the policy +(1638, then `server.c:778`) → `dictScan`'s comment (1434) last, on its own. + +**Contrast case**: valkey's *client-side* dict at +`deps/libvalkey/src/dict.c:103-150` (valkey `8891441ab`) — `dictExpand` there +allocates the new table and moves every entry in one `for` loop (123-143), +asserts the old table is empty (144), frees it (145) and swaps (148). No +`rehashidx`, no second table, no cursor: the entire Step 4 machine is absent. +That is the right call for a client library's small maps and unacceptable for +a server's keyspace — same structure, different RUM position, because latency +requirements are part of the workload. ## Questions to answer in notes.md -1. During rehash, `dictAddRaw` inserts only into ht[1]. Why is inserting into ht[0] - a correctness bug, not just a wasted move? -2. What does `pauserehash` exist for? (Hint: safe iterators.) -3. Redis caps `empty_visits` at 10n. What tail-latency guarantee does that give one - operation, in buckets touched? +1. During rehash, `dictInsertKeyAtLink` (dict.c:547) inserts only into ht[1]. + Why is inserting into ht[0] a correctness bug, not just a wasted move? + Trace what `dictCheckRehashingCompleted` (dict.c:380-394) would do to that + entry. +2. What does `pauserehash` exist for? Find its two users (dict.c:1179 and + dict.c:1572) and say what breaks in each if the brake is removed. +3. Redis caps `empty_visits` at `n*10` (dict.c:406). What tail-latency + guarantee does that give one operation, in buckets touched — and redo + Step 5's ~1.15 µs bound for a table that is being *shrunk* rather than + grown, where chains are longer and empty buckets rarer. +4. Line 783 skips ht[0] when `idx < rehashidx`. Sketch the read tax over the + life of a migration: what fraction of lookups touch two tables when the + cursor is 10% / 50% / 90% through ht[0]? +5. `dictRehashMicroseconds` (dict.c:446) rehashes on a *time* budget instead + of a bucket budget, from the server's cron. Which of the two budgets would + you give your own implementation, and what does the other one get wrong? + +## Takeaway + +The two-table dict is a latency structure, not a throughput structure: it +does strictly *more* total work than a stop-the-world rehash (two-table +lookups, a cursor, ten-empty-bucket scans) in exchange for never letting one +operation pay more than a bounded slice of it. That is the shape of almost +every fix in this curriculum — trade mean for max — and you are about to +build it in `experiments/src/incremental_map.rs`. ## Done when -You can implement the two-table scheme from memory — you'll do exactly that in this -topic's experiment. +Answer each before unfolding it. + +- [ ] You can say what `rehashidx` means, including what `-1` means and what is true of every bucket below it. + +
Answer + + `rehashidx` (dict.h:149) is the index of the next bucket of ht[0] to + migrate. `-1` means no migration is in progress, which is what + `dictIsRehashing` tests and what makes the read path a single-table walk + (dict.c:781). + + The invariant is that every bucket of ht[0] strictly below `rehashidx` is + empty and will stay empty: `rehashEntriesInBucketAtIndex` sets + `d->ht_table[0][idx] = NULL` at dict.c:376 after relinking the chain, the + cursor only ever moves forward (dict.c:425, 430), and new keys are never + written into ht[0] while rehashing (dict.c:547). That invariant is exactly + what licenses the read-path shortcut at dict.c:783 — if `idx < rehashidx`, + looking in ht[0] cannot find anything. + +
+ +- [ ] You can explain why inserting a new key into ht[0] during a migration is a correctness bug rather than a wasted move. + +
Answer + + Because the cursor never goes back. If the new entry lands in an ht[0] + bucket below `rehashidx`, no future `dictRehash` step will visit that + bucket — the `while` loop at dict.c:420-431 starts from `d->rehashidx` and + only increments — so the entry is never moved into ht[1]. + + What happens next depends on the bookkeeping. `dictCheckRehashingCompleted` + (dict.c:380-394) fires when `ht_used[0]` reaches 0 and calls + `zfree(d->ht_table[0])` at dict.c:386, taking the orphaned entry's bucket + with it; if the insert also bumped `ht_used[0]`, the counter never reaches 0 + and the dict simply never finishes rehashing, holding both tables forever. + Either outcome is a bug, which is why dict.c:547 makes the table choice a + single unconditional expression and dict.c:548-549 asserts the caller handed + over a bucket from that table. + +
+ +- [ ] You can state the bounded-work guarantee one operation gets, in buckets, and price it against the one-shot rehash this repo measured. + +
Answer + + One operation triggers at most `dictRehash(d,1)`, which is `empty_visits = + 1*10` (dict.c:406) plus one non-empty bucket: **at most ten empty bucket + loads and one chain**. Line 426 returns as soon as the tenth empty visit is + spent, so the work is bounded even on a table that is 99.9% empty. + + Priced with topic 0's ~100 ns dependent DRAM miss and a ~1.5-entry chain at + α = 1.0, that is about 11.5 × 100 ns ≈ 1.15 µs worst case. The one-shot + alternative was measured in this repo at **58.4 ms** + ([FINDINGS.md](../../FINDINGS.md) row 2) for a 7.34 M-entry table — about + 50,800× larger, and that was hashbrown's flat array, which sweeps at 7.96 ns + per entry rather than chasing malloc'd chain nodes. + +
+ +- [ ] You can say what a read costs while a migration is in flight, and why "it always checks both tables" is not quite right. + +
Answer + + `dictFindLinkInternal` sets `tables = 2` only while rehashing + (dict.c:781), and then line 783 skips ht[0] whenever the key's ht[0] bucket + index is below `rehashidx`, because Step 5's invariant guarantees that + bucket is empty. So the tax applies only to keys whose old bucket the cursor + has not reached yet, and it falls linearly as the cursor advances: roughly + 90% of lookups pay it when the cursor is 10% through, and roughly 10% when + it is 90% through. + + The cost when it does apply is one extra bucket-array load plus that + bucket's chain — Step 2's arithmetic makes it about 250 ns rather than + 125 ns for the second probe at α = 1.0. Cheap per operation, and paid for + the whole 83.9 seconds it takes 100K ops/s to walk a 2²³-bucket table one + bucket at a time. + +
+ +- [ ] You can explain why a fork for BGSAVE changes the resize policy, and what the parent is still allowed to do. + +
Answer + + `fork()` shares the parent's pages with the child copy-on-write, so a page + is duplicated only when someone writes it. A rehash relinks nearly every + entry (dict.c:371) and therefore writes nearly every page holding entries, + which would force the kernel to copy most of the dataset and roughly double + resident memory while the child is writing its RDB. + + `updateDictResizePolicy` (server.c:778-785) therefore has three states, not + two. Inside the forked child, `DICT_RESIZE_FORBID` (780) — no resizing at + all. In the parent while any child runs, `DICT_RESIZE_AVOID` (782), which + switches off the α ≥ 1.0 trigger at dict.c:1653 but leaves the forced grow + at dict.c:1655 live, so a table whose load factor reaches + `dict_force_resize_ratio` = 4 still expands. `dictRehash` applies the same + test before advancing an in-flight migration (dict.c:413-418). The bounded + price of that tolerance is Step 2's arithmetic: 3.0 chain entries examined + instead of 1.5, roughly 400 ns instead of 250 ns per cold lookup. + +
+ +- [ ] You can implement the two-table scheme from memory — which is exactly what `experiments/src/incremental_map.rs` asks for. + +
Answer + + There is no answer to unfold: the implementation is the exercise. The bar, + in the order the code needs it — two bucket arrays and a cursor + (dict.h:143-159); a migration step that moves one bucket and gives up after + ten empty ones (dict.c:405-434); a read that consults ht[1] and consults + ht[0] only when the cursor has not passed the key's old bucket + (dict.c:781-783); a write that lands in ht[1] whenever a migration is in + flight (dict.c:547); and a completion check that frees the old table, + promotes the new one and resets the cursor to −1 (dict.c:380-394). + + The measurement that says you got it right is in + [notes.md](notes.md): hashbrown's row is p50 42 ns / max 58.4 ms. Yours + should keep the p50 within a few nanoseconds of that and move the max into + microseconds. If your max is still in milliseconds, the usual cause is a + step that migrates a *chain* rather than a *bucket*, or an `empty_visits` + cap that was never wired up. + +
## References **Code** -- [redis](https://github.com/redis/redis) `src/dict.c`, `src/dict.h` — - line numbers from the local clone; the `dictScan` comment - (dict.c:1518) is one of the great comments in open source -- [valkey](https://github.com/valkey-io/valkey) - `deps/libvalkey/src/dict.c` — the single-table, full-rehash contrast - case +- [redis](https://github.com/redis/redis) `src/dict.c` (2340 lines), + `src/dict.h` (319 lines), `src/server.c` — pinned at Redis 8.6.2 / + `a176d1225`, version confirmed in `src/version.h:1`. + +| File | Lines | What | +|------|-------|------| +| `src/dict.h` | 143-159 | `struct dict` — the two-table state | +| `src/dict.h` | 214-216 | the `pauserehash` macros | +| `src/dict.c` | 45 | `dict_force_resize_ratio = 4` | +| `src/dict.c` | 336-377 | `rehashEntriesInBucketAtIndex` — one bucket moved | +| `src/dict.c` | 380-394 | completion: free ht[0], promote ht[1] | +| `src/dict.c` | 405-434 | `dictRehash` — the bounded step | +| `src/dict.c` | 426 | the `empty_visits` bail-out — the tail-latency guarantee | +| `src/dict.c` | 547 | `htidx = rehashing ? 1 : 0` — the write rule | +| `src/dict.c` | 783 | the read that skips an already-migrated ht[0] bucket | +| `src/dict.c` | 1434-1517 | the `dictScan` design comment | +| `src/dict.c` | 1584-1586 | the reverse-binary cursor increment | +| `src/dict.c` | 1653, 1655 | α ≥ 1.0, and the forced grow at α ≥ 4 | +| `src/dict.c` | 1705-1718 | the piggyback hook, with its bucket/cursor fork | +| `src/server.c` | 778-785 | FORBID / AVOID / ENABLE, decided by fork state | + +- [valkey](https://github.com/valkey-io/valkey) (`8891441ab`) + `deps/libvalkey/src/dict.c:103-150` — the single-table, full-rehash + contrast case: one loop, every entry, inside one call. + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 2 — hashbrown insert p50 42 ns, max + 58.4 ms, the stop-the-world rehash this whole design avoids. +- [FINDINGS.md](../../FINDINGS.md) row 0 — the ~1 / 5 / 100 ns cache ladder + every cost estimate above is priced with. +- [notes.md](notes.md) — the per-decile maxima, showing the spikes land + exactly where the table crossed a power of two. diff --git a/topics/02-in-memory-structures/reading-redis-rax.md b/topics/02-in-memory-structures/reading-redis-rax.md index b539c3f..5a62e82 100644 --- a/topics/02-in-memory-structures/reading-redis-rax.md +++ b/topics/02-in-memory-structures/reading-redis-rax.md @@ -1,30 +1,47 @@ # rax: a radix tree packed into cache lines -Redis's compressed radix tree — behind stream IDs, client tracking keys, and -cluster slot→key maps — is what a trie looks like when memory is the corner -of the RUM triangle you're defending: one variable-size node layout, -deliberately unaligned pointers, path-compressed runs. This chapter builds -the trie idea from zero, compresses it, then packs it byte by byte the way -rax does — before sending you into the layout comment and the walk. Read for -the *layout* (~45 min, skim the insert logic); it's the memory-first contrast -case for the ART paper that follows. +Redis's compressed radix tree — behind stream entries, client-tracking tables, +the blocked-client timeout index and the errors table — is what a trie looks +like when memory is the corner of the RUM triangle you are defending: one +variable-size node layout sized to the byte, path-compressed runs, and a +padding rule that buys pointer alignment for at most seven bytes. This chapter +builds the trie idea from zero, compresses it, then packs it byte by byte the +way rax does — before sending you into the layout comment and the walk. Read +for the *layout* (~45 min, skim the insert logic); it is the memory-first +contrast case for the ART paper that follows. + +Everything below is read against **redis/redis at `a176d1225`**, where +`src/rax.h` is 204 lines and `src/rax.c` is 2098. Line numbers move between +releases; re-check yours with: + +``` +tools/pinned-source.py ref redis +tools/pinned-source.py show redis src/rax.h -r 77:119 +tools/pinned-source.py show redis src/rax.c -r 126:155 +``` + +If a number below does not match your checkout, trust the checkout and record +the drift in `notes.md` — that is the exercise. ## The problem in one sentence -Redis keeps *millions* of small string-keyed maps (one per stream, per -client-tracking table, per cluster slot), so a per-node overhead of even 48 -bytes — a textbook trie node — multiplies into gigabytes; the index must cost -close to the bytes of the keys themselves. +Redis keeps *millions* of small string-keyed maps — one radix tree per stream, +one per tracking prefix, one for every blocked client's timeout — so a +per-node overhead of even 48 bytes multiplies into gigabytes; the index must +cost close to the bytes of the keys themselves. ## The concepts, step by step ### Step 1 — the trie: the key's bytes ARE the path -A **trie** (radix tree) is a tree where you find a key not by *comparing* -keys but by *spelling* them: each node branches on the next byte of the key, -so the path from the root spells the key out. Lookup depth = key length, not -log n; there is no hash function and no full-key comparisons — just one -branch decision per byte: +> **In:** a set of byte-string keys and the need to look one up. +> **Out:** a structure with no hash function and no whole-key comparisons — +> depth proportional to key length — and one glaring cost: a node per byte. + +A **trie** (radix tree) finds a key not by *comparing* keys but by *spelling* +them: each edge is labelled with the next byte, so the path from the root +spells the key out. Lookup depth is key length, not log n; there is no hash +function and no full-key comparison — just one branch decision per byte: ``` keys "foo", "for": root @@ -37,175 +54,716 @@ keys "foo", "for": root [●] [●] independent of how many keys exist ``` -What you gain over a hash table: sorted iteration and prefix scans for free -(all keys under "fo" live in one subtree — topic 23's inverted index will -want this). What it costs so far: one node *per byte* of every key — a -3-level chain of allocations to store "foo". That's the memory disaster to -fix. +Note where the character lives. rax's own header is emphatic about it, because +it is the thing everybody gets backwards: the character is stored **in the +parent's edge**, not in the child. + +What you gain over a hash table: sorted iteration and prefix scans for free — +every key beginning "fo" lives in one subtree (topic 23's inverted index will +want exactly this). What it costs so far: one node *per byte* of every key. A +three-level chain of allocations to store "foo". That is the memory disaster +the rest of the chapter fixes. ### Step 2 — path compression: collapse single-child chains into runs -Most trie nodes in real data have exactly one child (long unique key tails, -shared prefixes) — a chain of one-child nodes spelling "oot" is pure -overhead. **Path compression** replaces any such chain with a single node -holding the whole byte run: +> **In:** the per-byte node chain from Step 1. +> **Out:** one node per *run* of bytes, so node count tracks branch points +> rather than key length — and the exact invariant that keeps it that way +> under writes. + +Most trie nodes in real data have exactly one child — long unique key tails +and shared prefixes both produce chains. **Path compression** replaces any +such chain with a single node holding the whole byte run. rax's header draws +it for the keys "foo", "foobar", "footer": ``` -radix tree (rax), keys "foo", "foobar", "footer": +// src/rax.h — header comment, the compressed representation, 44-50 + 44 * ["foo"] "" + 45 * | + 46 * [t b] "foo" + 47 * / \ + 48 * "foot" ("er") ("ar") "foob" + 49 * / \ + 50 * "footer" [] [] "foobar" +``` + +Square brackets mark a node that **is a key**, parentheses one that is not +(`rax.h:18-19`); a compressed node shows its whole run inside the delimiters. +Six nodes for three keys, where the uncompressed trie at `rax.h:23-35` needed +ten. Depth is now the number of *branch points*, not the key length. + +The invariant that keeps compression from decaying is stated where it is +enforced — on the *delete* side, not the insert side: - [f o o] ← compressed run (iscompr): one node holds the shared prefix - │ - (key: "foo") - ┌─┴──┐ - [b] [t] - │ │ - [a r] [e r] compressed tails +``` +// src/rax.c — recompression rationale in raxRemove, 1107-1114 + 1107 /* Recompression: if trycompress is true, 'h' points to a radix tree node + 1108 * that changed in a way that could allow to compress nodes in this + 1109 * sub-branch. Compressed nodes represent chains of nodes that are not + 1110 * keys and have a single child, so there are two deletion events that + 1111 * may alter the tree so that further compression is needed: + 1112 * + 1113 * 1) A node with a single child was a key and now no longer is a key. + 1114 * 2) A node with two children now has just one child. ``` -Now depth ≈ the number of *branch points*, not key length, and node count ≈ -distinct branches. A compressed node stores a multi-byte run ("foo") with a -**single** child pointer. The remaining question is what one node costs in -bytes — rax's real contribution. +Read line 1109 carefully: a compressible chain is nodes that are **not keys** +*and* have a single child. Both clauses matter. A single-child node that *is* +a key cannot be folded into a run, because a run has one value slot at its +end and nowhere to hang a value in the middle. Insert splits runs apart +(Step 6); delete is where they get glued back (`raxRemove`, the loop at +1150-1175 walks up to the highest compressible node and then forward along the +chain). Insertion never needs to merge, because it only ever adds branch +points. ### Step 3 — the node: a 4-byte header and one flexible array -rax spends four bytes of header, then packs **everything** — child bytes, -child pointers, and the optional value pointer — into one flexible array in -a single allocation. `rax.h:78–111`: +> **In:** the compressed tree from Step 2, still made of unspecified "nodes". +> **Out:** the concrete byte layout — a 32-bit header and a single flexible +> array holding characters, then child pointers, then an optional value +> pointer — and the size formula that follows from it. + +rax spends **four bytes** of header, then packs everything else into one +flexible array in a single allocation: + +``` +// src/rax.h — raxNode, 77-82 and 110-111 + 77 #define RAX_NODE_MAX_SIZE ((1<<29)-1) + 78 typedef struct raxNode { + 79 uint32_t iskey:1; /* Does this node contain a key? */ + 80 uint32_t isnull:1; /* Associated value is NULL (don't store it). */ + 81 uint32_t iscompr:1; /* Node is compressed. */ + 82 uint32_t size:29; /* Number of children, or compressed string len. */ + ... 83-109: the data layout comment — the spec, quoted next ... + 110 unsigned char data[]; + 111 } raxNode; +``` + +Three bits and a 29-bit count in one word, so `sizeof(raxNode)` is 4 and +`RAX_NODE_MAX_SIZE` = 2²⁹ − 1 = **536,870,911** — the largest fanout or +compressed run the `size` field can express. `isnull` earns its bit: a key +whose value is `NULL` stores no value pointer at all, saving 8 bytes on the +very common "membership set" use. + +The layout comment is the spec. Read it in full before any function: -```c -typedef struct raxNode { - uint32_t iskey:1; /* this node terminates a key */ - uint32_t isnull:1; /* key has no associated value */ - uint32_t iscompr:1; /* node is a compressed run */ - uint32_t size:29; /* # children (or run length if iscompr) */ - unsigned char data[]; /* EVERYTHING else lives here */ -} raxNode; +``` +// src/rax.h — data layout comment, 83-108 + 83 /* Data layout is as follows: + ... + 85 * If node is not compressed we have 'size' bytes, one for each children + 86 * character, and 'size' raxNode pointers, point to each child node. + 87 * Note how the character is not stored in the children but in the + 88 * edge of the parents: + 89 * + 90 * [header iscompr=0][abc][a-ptr][b-ptr][c-ptr](value-ptr?) + 91 * + 92 * if node is compressed (iscompr bit is 1) the node has 1 child. + 93 * In that case the 'size' bytes of the string stored immediately at + 94 * the start of the data section, represent a sequence of successive + 95 * nodes linked one after the other, for which only the last one in + 96 * the sequence is actually represented as a node, and pointed to by + 97 * the current compressed node. + 98 * + 99 * [header iscompr=1][xyz][z-ptr](value-ptr?) + ... 100-104: both kinds can carry a key at any level ... + 105 * If the node has an associated key (iskey=1) and is not NULL + 106 * (isnull=0), then after the raxNode pointers pointing to the + 107 * children, an additional value pointer is present (as you can see + 108 * in the representation above as "value-ptr" field). ``` +So, with the padding rule from Step 4 filled in: + ``` non-compressed, size=3 ("abc" branches): compressed run "xyz" (iscompr=1): -┌header┐┌── data[] ─────────────────────┐ ┌header┐┌── data[] ────────────┐ -│4 bytes││a b c pad│ A* │ B* │ C* │ V*? │ │4 bytes││x y z pad│ Z* │ V*? │ -└──────┘└─────────┴────┴────┴────┴─────┘ └──────┘└─────────┴────┴──────┘ - ▲ char bytes first (dense filter!) whole run = ONE child pointer - then pointers, then value if iskey (points past the run) +┌header┐┌──────── data[] ───────────────┐ ┌header┐┌──── data[] ─────────┐ +│4 bytes││a b c │p│ A* │ B* │ C* │ V*? │ │4 bytes││x y z │p│ Z* │ V*? │ +└──────┘└──────┴─┴────┴────┴────┴──────┘ └──────┘└──────┴─┴────┴──────┘ + ▲ char bytes first (dense filter) whole run = ONE child pointer + │ then padding, then pointers (points at the node after it) + │ then value pointer if iskey&&!isnull + 32 bytes total 16 bytes total ``` -The layout comment at rax.h:83–109 is the spec — read it in full. Note the -order: the branch *characters* come first, densely packed, then the -pointers. Choosing a branch scans only the char bytes — the same "dense -filter, fat payload" move as SwissTable control bytes (README §4): the data -you probe is dense; the data you follow is touched once, on a match. +Note the order: the branch *characters* come first, densely packed. Choosing a +branch scans only the char bytes — the same "dense filter, fat payload" move +as SwissTable's control bytes (README §4): the data you probe is dense and +small; the data you follow is touched once, on a match. + +### Step 4 — the padding rule: rax pays for aligned pointers + +> **In:** a `data[]` array whose pointer section starts after a variable number +> of character bytes. +> **Out:** the 0–7 byte padding that restores 8-byte alignment, the exact node +> size formula, and worked sizes for real nodes. + +Because the characters come first and their count is arbitrary, the pointer +section would land at an arbitrary offset. rax does **not** accept that. It +inserts padding: + +``` +// src/rax.c — size macros, 126-155 + 126 /* Return the padding needed in the characters section of a node having size + 127 * 'nodesize'. The padding is needed to store the child pointers to aligned + 128 * addresses. Note that we add 4 to the node size because the node has a four + 129 * bytes header. */ + 130 #define raxPadding(nodesize) ((sizeof(void*)-(((nodesize)+4) % sizeof(void*))) & (sizeof(void*)-1)) + ... 132-133: comment for raxNodeLastChildPtr ... + 134 #define raxNodeLastChildPtr(n) ((raxNode**) ( \ + 135 ((char*)(n)) + \ + 136 raxNodeCurrentLength(n) - \ + 137 sizeof(raxNode*) - \ + 138 (((n)->iskey && !(n)->isnull) ? sizeof(void*) : 0) \ + 139 )) + 140 + 141 /* Return the pointer to the first child pointer. */ + 142 #define raxNodeFirstChildPtr(n) ((raxNode**) ( \ + 143 (n)->data + \ + 144 (n)->size + \ + 145 raxPadding((n)->size))) + ... 147-149: comment: the second line computes the padding after the string ... + 150 #define raxNodeCurrentLength(n) ( \ + 151 sizeof(raxNode)+(n)->size+ \ + 152 raxPadding((n)->size)+ \ + 153 ((n)->iscompr ? sizeof(raxNode*) : sizeof(raxNode*)*(n)->size)+ \ + 154 (((n)->iskey && !(n)->isnull)*sizeof(void*)) \ + 155 ) +``` + +Line 127-128 says it outright: "The padding is needed to store the child +pointers to aligned addresses." Line 145 is where `raxNodeFirstChildPtr` skips +it. `malloc` returns 8-aligned memory, the header is 4 bytes, and +`raxPadding(size)` is chosen so that `4 + size + padding ≡ 0 (mod 8)` — so +every child pointer in the array is 8-aligned. **rax's pointers are aligned, +by construction, and the padding is the price.** + +That price is at most 7 bytes per node: + +``` +raxPadding(size) = (8 - ((size + 4) mod 8)) & 7 + + size: 0 1 2 3 4 5 6 7 8 + padding: 4 3 2 1 0 7 6 5 4 +``` + +**Work a node.** `raxNodeCurrentLength` (line 150-155) is +`4 + size + padding + (iscompr ? 8 : 8·size) + (iskey && !isnull ? 8 : 0)`. + +``` + non-compressed, size=3, not a key : 4 + 3 + 1 + 24 + 0 = 32 bytes + compressed "xyz", not a key : 4 + 3 + 1 + 8 + 0 = 16 bytes + compressed "xyz", key with value : 4 + 3 + 1 + 8 + 8 = 24 bytes + non-compressed, size=1, not a key : 4 + 1 + 3 + 8 + 0 = 16 bytes + leaf: size=0, key with value : 4 + 0 + 4 + 0 + 8 = 16 bytes + full fanout, size=256, not a key : 4 + 256 + 4 + 2048 = 2312 bytes (9.03 B/child) +``` + +Now price Step 2's whole tree — the three keys "foo", "foobar", "footer", +15 bytes of key data: + +``` + compressed (rax.h:44-50, six nodes) uncompressed trie (ten nodes) + ["foo"] compr size=3, non-key 16 7 × single-child non-key 112 + [t b] size=2, key 32 [b t] size=2, key 32 + ("er") compr size=2, non-key 16 2 × leaf size=0, key 32 + ("ar") compr size=2, non-key 16 + [] × 2 size=0, key 16+16 + ───── ───── + 112 176 + + saving: (176 − 112) / 176 = 36.4% +``` -### Step 4 — unaligned pointers, on purpose +Compression buys 36% here, and the gap widens with key length: every extra +byte of a unique tail costs 16 bytes uncompressed and 1 byte inside a run. +Note also the honest direction of the comparison — 112 bytes of index for 15 +bytes of keys is *not* cheap in absolute terms. Radix trees pay a fixed price +per branch point; they win when keys are long and share prefixes, which is +exactly the stream-ID and tracking-key shape redis uses them for. -Because chars come first and there's no padding, the 8-byte child pointers -in `data[]` may start at **any byte offset** — they are not aligned. Redis -reads and writes them with `memcpy` (the `raxNodeFirstChildPtr` / -`raxNodeLastChildPtr` helpers; rax.h:90, 99). Why tolerate that? +rax also keeps its own byte count, which tells you how seriously the memory +corner is taken here: -- One allocation per node; header + chars + pointers usually fit **one cache - line** for small fanouts (4 + 3 + 3×8 = 31 bytes for the 3-child node - above). -- Alignment padding would spread the node across lines; modern ARM/x86 do - unaligned loads nearly free, so the cache line saved is worth more than - the alignment lost. +``` +// src/rax.c — raxNewNode, 161-174 + 161 raxNode *raxNewNode(rax *rax, size_t children, int datafield) { + 162 size_t nodesize = sizeof(raxNode)+children+raxPadding(children)+ + 163 sizeof(raxNode*)*children; + 164 if (datafield) nodesize += sizeof(void*); + 165 size_t usable; + 166 raxNode *node = rax_malloc_usable(nodesize,&usable); + ... 167-171: NULL check and header init ... + 172 if (rax->alloc_size) *rax->alloc_size += usable; + 173 return node; + 174 } +``` -A deliberate trade of CPU convention for memory locality — the whole chapter -in one decision. +Line 166 asks the allocator for the *usable* size, not the requested one, and +line 172 accumulates it into a caller-supplied counter (`rax->alloc_size`, +`rax.h:117`) — so redis reports the true allocator-rounded footprint of a +tree, malloc slack included, rather than the sum of its `nodesize` arguments. ### Step 5 — the walk: the tree's entire read path -Every rax operation starts with `raxLowWalk`: consume the key byte by byte, -scanning char bytes in branching nodes and matching prefixes in compressed -runs. It returns how much of the key it consumed and — crucially for insert — -`splitpos`, where the key diverged *inside* a compressed run: - -```rust -// returns (bytes of key consumed, split position inside a compressed run) -fn low_walk(mut node: &RaxNode, key: &[u8]) -> (usize, usize) { - let mut i = 0; - while i < key.len() { - if node.iscompr() { - let run = node.chars(); // e.g. "oot" — one node - let m = common_prefix(run, &key[i..]); - if m < run.len() { return (i + m, m); } // diverged MID-run: splitpos - i += m; - node = node.child(0); // whole run = ONE pointer - } else { - match node.chars().iter().position(|&c| c == key[i]) { // dense scan: - Some(j) => { node = node.child(j); i += 1; } // chars only, - None => return (i, 0), // ptrs untouched - } - } - } - (i, 0) // consumed the whole key: node.iskey ⇒ hit -} +> **In:** a key `s` of `len` bytes and the tree root. +> **Out:** how many bytes were consumed, the node where the walk stopped, and +> `splitpos` — the offset *inside* a compressed run where it stopped — which +> is what insert needs to cut. + +Every rax operation starts with `raxLowWalk`. It is 42 lines and it is the +whole read path: + +``` +// src/rax.c — raxLowWalk, 465-506 + 465 static inline size_t raxLowWalk(rax *rax, unsigned char *s, size_t len, raxNode **stopnode, raxNode ***plink, int *splitpos, raxStack *ts) { + 466 raxNode *h = rax->head; + 467 raxNode **parentlink = &rax->head; + 468 + 469 size_t i = 0; /* Position in the string. */ + 470 size_t j = 0; /* Position in the node children (or bytes if compressed).*/ + 471 while(h->size && i < len) { + ... 472-473: debug hook; unsigned char *v = h->data ... + 475 if (h->iscompr) { + 476 for (j = 0; j < h->size && i < len; j++, i++) { + 477 if (v[j] != s[i]) break; + 478 } + 479 if (j != h->size) break; + 480 } else { + 481 /* Even when h->size is large, linear scan provides good + 482 * performances compared to other approaches that are in theory + 483 * more sounding, like performing a binary search. */ + 484 for (j = 0; j < h->size; j++) { + 485 if (v[j] == s[i]) break; + 486 } + 487 if (j == h->size) break; + 488 i++; + 489 } + 490 + 491 if (ts) raxStackPush(ts,h); /* Save stack of parent nodes. */ + 492 raxNode **children = raxNodeFirstChildPtr(h); + 493 if (h->iscompr) j = 0; /* Compressed node only child is at index 0. */ + 494 memcpy(&h,children+j,sizeof(h)); + 495 parentlink = children+j; + ... 496-499: reset j to 0 for the next iteration ... + 500 } + ... 501-504: publish stopnode, plink, and splitpos when h is compressed ... + 505 return i; + 506 } +``` + +Three lines carry the design. + +**Line 473 and 484-486** — the scan reads `h->data`, the *character* prefix, +and nothing else. `children` (line 492) is not even computed until a match is +found. That is the dense-filter payoff made concrete: a 256-way branching node +is 2312 bytes, but choosing a child touches only the first 256 of them, and +usually far fewer. + +**Lines 481-483** — the comment defends a linear scan over binary search, "even +when `h->size` is large". Believe it, and know why: the characters are +contiguous bytes, so a linear scan is a sequential read over at most four +cache lines with a perfectly predictable stride, while a binary search is +log₂(256) = 8 unpredictable branches over the same memory. Topic 0's lesson — +branch misprediction and locality dominate instruction count at these sizes. +This is also the exact place where ART diverges: it replaces this loop with a +SIMD compare (Node16) or a direct index (Node256). + +**Line 494** — the child pointer is read with `memcpy` even though Step 4's +padding guarantees it is aligned. That is not an unaligned-access workaround; +it is the standard C idiom for reading a `raxNode*` out of a `char`-typed +buffer without violating strict aliasing, and every compiler turns it into a +single load. + +The stop condition at line 471 is worth a second: the loop ends when the key +runs out *or* the node has no children. `raxFind` then decides whether that +counts as a hit, in one line: + ``` +// src/rax.c — raxFind, 931-941 + 931 int raxFind(rax *rax, unsigned char *s, size_t len, void **value) { + ... 932-935: locals and debug ... + 936 size_t i = raxLowWalk(rax,s,len,&h,NULL,&splitpos,NULL); + 937 if (i != len || (h->iscompr && splitpos != 0) || !h->iskey) + 938 return 0; + 939 if (value != NULL) *value = raxGetData(h); + 940 return 1; + 941 } +``` + +Line 937 is three failure modes in one expression: the key was not fully +consumed; or it *was* consumed but the walk halted part-way through a +compressed run, so the key is a strict prefix of that run and no node +represents it; or a node does represent it but was never marked a key. The +middle clause exists only because runs exist — it is compression's tax on the +lookup path, and it is one comparison. Cost model: one dependent pointer hop per *node* (not per byte, thanks to -compression), and within a node the scan touches only the dense char prefix. +compression), and inside a node the scan touches only the dense character +prefix. ### Step 6 — insert = split machinery -Inserting a key that diverges mid-run must cut the run at `splitpos`, create -a small branching node, and re-hang the tails. `raxGenericInsert` -(rax.c:515–658, skim) enumerates the cases in its long comment; the picture: +> **In:** a key that diverges from the tree part-way through a compressed run, +> plus the `splitpos` that says where. +> **Out:** two algorithms — one for a mismatch inside a run, one for a key that +> *ends* inside a run — and the reason you should read the comment rather than +> the code. + +`raxGenericInsert` (`rax.c:515-913`) is the longest function in the file, and +roughly a quarter of it is a comment enumerating the cases on the example word +"ANNIBALE": ``` -insert "first" into node ["footer"]: split the run at splitpos=1 - [f] ← shared prefix survives as run (or single node) - ┌─┴─┐ - ["ooter"] ["irst"] ← two compressed tails, new branching node +// src/rax.c — the case enumeration, 577-608 + 577 * When inserting we may face the following cases. Note that all the cases + 578 * require the insertion of a non compressed node with exactly two + 579 * children, except for the last case which just requires splitting a + 580 * compressed node. + 581 * + 582 * 1) Inserting "ANNIENTARE" + 583 * + 584 * |B| -> "ALE" -> "SCO" -> [] + 585 * "ANNI" -> |-| + 586 * |E| -> (... continue algo ...) "NTARE" -> [] + ... 588-605: cases 2-4, all mid-run mismatches at different offsets ... + 606 * 5) Inserting "ANNI" + 607 * + 608 * "ANNI" -> "BALE" -> "SCO" -> [] ``` -Every case is "cut the run, make a 2-child branching node, re-hang the -tails". Don't memorize the five cases — just verify the invariant: **after -any insert, no node has exactly one child unless it's compressed** (otherwise -it would be merged into a run). That invariant is what keeps Step 2's -compression from decaying under writes. +Cases 1-4 are the same event at different offsets: the key mismatched inside +the run, so cut the run at `splitpos`, insert a two-child branching node, and +re-hang both tails. Case 5 is different in kind: the key *ran out* inside the +run with no mismatch, so there is nothing to branch — just split the run into +a prefix and a postfix and mark the prefix as a key. That is exactly why the +code has two labelled algorithms: + +- **ALGO 1** (`rax.c:684-685`, guarded by `if (h->iscompr && i != len)`) — the + mismatch cases. Steps are spelled out at lines 613-654: save `$NEXT`, build + the split node, trim or replace the original depending on whether + `$SPLITPOS == 0`, build a postfix node if the remainder is non-empty, then + fall through to the ordinary insertion for the key's own tail. +- **ALGO 2** (lines 656-681) — case 5, the "key ends inside a run" case: build + the postfix node, trim the current node to `$SPLITPOS` characters, and mark + the trimmed node as the key. + +Two details from the code are worth carrying away. First, `if (h->size == 0 && +len-i > 1)` at line 867: when a fresh tail of more than one byte has to be +appended, rax creates it as a *compressed* node immediately rather than as a +chain it would have to fold later — compression is maintained at write time, +not by a cleanup pass. Second, line 870-871 clamps a new run to +`RAX_NODE_MAX_SIZE`, so a key longer than 2²⁹ − 1 bytes simply becomes several +runs; nothing overflows the 29-bit `size` field. + +Do not memorise the five cases. Verify the invariant instead — the one from +Step 2, at `rax.c:1109-1110`: compressed nodes are chains of nodes that are +**not keys** and have a **single child**. Every case above either preserves it +or is the delete-side repair that restores it. + +### Step 7 — binary-comparable keys, in production + +> **In:** a radix tree that orders keys byte-wise, and a workload that wants +> them ordered *numerically*. +> **Out:** the encoding trick that makes those the same thing, and a verified +> list of where redis relies on it. + +A radix tree iterates in **byte-lexicographic** order. That is only useful for +numbers if the number's bytes sort the same way the number does — which +little-endian integers emphatically do not. redis's blocked-client timeout +index shows the fix in four lines: -### Step 7 — the contrast: rax vs ART, opposite RUM corners +``` +// src/timeout.c — encodeTimeoutKey, 75-83 + 75 #define CLIENT_ST_KEYLEN 16 /* 8 bytes mstime + 8 bytes client ID. */ + 76 + 77 /* Given client ID and timeout, write the resulting radix tree key in buf. */ + 78 void encodeTimeoutKey(unsigned char *buf, uint64_t timeout, client *c) { + 79 timeout = htonu64(timeout); + 80 memcpy(buf,&timeout,sizeof(timeout)); + 81 memcpy(buf+8,&c,sizeof(c)); + ... 82: zero padding for 32-bit targets ... + 83 } +``` + +Line 79 converts the millisecond timeout to **big-endian** before it becomes a +key. Now the most significant byte is byte 0, so the radix tree's byte order +*is* numeric order, and `handleBlockedClientsTimeout` can walk the tree from +the smallest key and stop at the first entry not yet due — an ordered scan +over a structure that never compares whole keys. The client pointer at bytes +8-15 is only a tiebreaker, keeping keys unique. + +That transformation has a name in the next chapter: ART calls it +**binary-comparable keys**, and devotes a section to producing them for signed +integers, floats and compound keys. redis got there first, informally, in a +`htonu64` call. + +Verified users of rax in this checkout, so you can see the workload shape it +was built for: + +| Where | What the tree holds | +|-------|---------------------| +| `src/stream.h:37` | `rax *rax;` — the stream itself, keyed by entry ID | +| `src/stream.h:44-45` | consumer groups by name; message ID → group | +| `src/tracking.c:24-25` | `TrackingTable`, `PrefixTable` — client-side caching | +| `src/server.h:1999` | `clients_timeout_table` — the ordered index above | +| `src/server.h:2003` | `clients_index` — active clients by ID | +| `src/server.h:1935` | `errors` — the error-statistics table | +| `src/server.h:1349` | `blocks_index` — replication backlog blocks | + +Every one of them is either long shared-prefix keys (stream IDs, tracking +keys) or a small map that there may be thousands of. Neither is a case where a +`dict` would win. + +### Step 8 — the contrast: rax vs ART, opposite RUM corners + +> **In:** rax's single variable-size node with a linear scan. +> **Out:** the axis along which ART differs, and the specific claims to check +> against the paper in the next chapter. The next chapter's ART is the same structure tuned for the opposite corner: -| | rax | ART (Leis 2013) | +| | rax (`a176d1225`) | ART (Leis et al., ICDE 2013) | |---|-----|-----| -| node sizes | one variable-size layout | adaptive Node4/16/48/256 | -| child search | linear scan of char bytes | SIMD (Node16), direct index (Node256) | -| pointers | unaligned, memcpy'd | aligned arrays | -| optimized for | memory (redis: millions of tiny trees) | lookup speed (main-memory index) | - -Same structure, opposite RUM corner: rax minimizes M, ART minimizes R. Keep -this table in mind while reading the paper. +| node sizes | one layout, sized to `size` exactly | four fixed layouts: Node4/16/48/256 | +| child search | linear scan of the char prefix (`rax.c:484-486`) | SIMD compare, indirection array, or direct index | +| pointer alignment | padded to 8 (`rax.c:130`) | aligned arrays | +| path compression | full runs, arbitrary length | pessimistic/optimistic, bounded prefix | +| optimised for | memory — millions of tiny trees | lookup latency — one big main-memory index | + +Same structure, opposite RUM corner: rax minimises M, ART minimises R. Two +concrete numbers to carry into the paper: a rax node with 4 children costs +4 + 4 + 0 + 32 = **40 bytes**, and with 16 children 4 + 16 + 4 + 128 = +**152 bytes** — check those against ART's Node4 and Node16, which are fixed +sizes regardless of how many slots are occupied. That difference *is* the RUM +trade, in bytes. ## Where each step lives in the code -- **Steps 3–4** — `raxNode` struct: rax.h:78–111; the layout spec comment: - rax.h:83–109 (read in full before any function); unaligned-pointer helpers - `raxNodeFirstChildPtr` / `raxNodeLastChildPtr`: rax.h:90, 99. -- **Step 5** — `raxLowWalk`: the read path every operation shares. -- **Step 6** — `raxGenericInsert`: rax.c:515–658 (skim; the case-enumeration - comment above the code is the map). +| File | Lines | What | Step | +|------|-------|------|------| +| `src/rax.h` | 16-75 | header comment: notation, vanilla trie, compression, splitting | 1, 2 | +| `src/rax.h` | 44-50 | the compressed representation of foo/foobar/footer | 2 | +| `src/rax.h` | 77 | `RAX_NODE_MAX_SIZE` = 2²⁹ − 1 | 3 | +| `src/rax.h` | 78-111 | `raxNode` — 4-byte header + flexible `data[]` | 3 | +| `src/rax.h` | 83-109 | **the layout spec** — read in full before any function | 3 | +| `src/rax.h` | 113-119 | `rax` — head, counts, `alloc_size`, metadata | 4 | +| `src/rax.h` | 121-130 | `raxStack` — parents, because nodes have no parent pointer | 2 | +| `src/rax.c` | 126-130 | `raxPadding` — the alignment rule and its rationale | 4 | +| `src/rax.c` | 134-145 | `raxNodeLastChildPtr` / `raxNodeFirstChildPtr` | 4 | +| `src/rax.c` | 150-155 | `raxNodeCurrentLength` — the node size formula | 4 | +| `src/rax.c` | 161-174 | `raxNewNode` — one allocation, usable-size accounting | 4 | +| `src/rax.c` | 403-434 | `raxCompressNode` — build a run | 2 | +| `src/rax.c` | 436-464 | `raxLowWalk` doc comment — what `splitpos` means | 5 | +| `src/rax.c` | 465-506 | `raxLowWalk` — the entire read path | 5 | +| `src/rax.c` | 515-913 | `raxGenericInsert` | 6 | +| `src/rax.c` | 560-682 | the case enumeration and both algorithms, as a comment | 6 | +| `src/rax.c` | 684, 867-877 | ALGO 1's guard; compressed-tail creation at write time | 6 | +| `src/rax.c` | 931-941 | `raxFind` — the three-clause hit test | 5 | +| `src/rax.c` | 1107-1121 | the compression invariant, stated on the delete path | 2 | +| `src/rax.c` | 1150-1175 | recompression: walk up, then collect the chain | 2 | +| `src/timeout.c` | 75-83 | `encodeTimeoutKey` — big-endian keys for ordered scan | 7 | + +A route through it that builds rather than jumps: + +1. `rax.h:16-75`. The header comment is a tutorial with pictures — trie, then + compressed trie, then the split that "foo"/"first" forces. Read it before + any code. Line 18-19 defines the notation: `[]` is a key, `()` is not. +2. `rax.h:77-111`. Struct, then the layout comment. Write the two layouts on + paper from lines 90 and 99. +3. `rax.c:126-155`. Four macros. Compute `raxNodeCurrentLength` by hand for a + non-compressed 3-child node and check you get 32. +4. `rax.c:465-506`. `raxLowWalk`. Trace a lookup of "footer" against the tree + at `rax.h:44-50`: which nodes are visited, and what are `i`, `j` and + `splitpos` at each stop? +5. `rax.c:931-941`. `raxFind`. Now trace "foot" — a strict prefix — and find + which of line 937's three clauses rejects it. +6. **Aha:** the padding at `rax.c:130` exists so the child pointers are + *aligned*, and it costs 0-7 bytes per node. Once you see that, re-read the + Step 4 size table and notice that a compressed node costs 16 bytes no + matter whether its run is 1 or 3 bytes long — padding absorbs the + difference. The natural run lengths for a memory-tuned structure are + therefore not 1; they are 3, 11, 19, … Every design decision in this file + is that kind of arithmetic. +7. Only then skim `rax.c:560-682`. Read the comment, not the code. + +**Contrast case.** Read `raxLowWalk`'s branching arm (lines 484-486) beside +this repo's own `reading-hashbrown.md`, where the same "find the matching byte" +question is answered with a 16-wide SIMD compare over control bytes. Both are +scanning a dense byte array for a match; one uses a scalar loop and defends it +in a comment, the other uses `_mm_cmpeq_epi8`. The difference is fanout: rax's +scan is over `size` bytes where `size` is usually under 8, hashbrown's is over +a fixed 16. Below about a group width, the scalar loop wins on setup cost +alone — which is precisely why ART introduces Node4 *and* Node16 rather than +one SIMD node. ## Questions to answer in notes.md -1. Why does rax put the char bytes *before* the pointers instead of interleaving - (char,ptr) pairs? (Branch decision reads only chars — one dense scan.) -2. A radix tree has no hash function and no key comparisons — what does it give - up vs a hash table? (Point-lookup cost ∝ key length; but you gain prefix scans - and ordered iteration — which topic 23's inverted index will want.) +1. Why does rax put the char bytes *before* the pointers instead of + interleaving (char, ptr) pairs? Answer in terms of what `raxLowWalk` line + 484-486 touches versus what it does not, and how many cache lines each + layout would read for a 32-child node. +2. `raxPadding` costs 0-7 bytes per node to keep child pointers aligned. Using + the size table in Step 4, compute the total padding in the six-node tree + for foo/foobar/footer, and say what the tree would cost with the pointers + left unaligned. Was the trade worth it here? +3. Lines 481-483 defend a linear scan over binary search "even when `h->size` + is large". Construct the case where that comment is wrong — what fanout, + and what would you have to measure to show it? (Then check what ART chose.) +4. `raxFind` line 937 has three failure clauses. Give a concrete key and tree + for each, and say which one exists only because of path compression. +5. A radix tree has no hash function and no whole-key comparison. Compare + against this repo's measured `lookup_shootout` at n = 10⁶ — `hashmap + 8.8 ns`, `btreemap 26.6 ns` — and say what rax buys that neither offers, + naming the two redis subsystems from Step 7 that need it. + +## Takeaway + +rax is a radix tree that treats every byte as negotiable. Four bytes of header +carry three flags and a 29-bit count; characters and pointers share one +flexible array in one allocation; a compressed node folds an arbitrary run +into a single child pointer; a `NULL` value costs no pointer at all. The one +place it *spends* is `raxPadding` — up to seven bytes per node so the child +pointers stay 8-aligned — which is a good reminder that "memory-optimised" +never means "no padding", it means every byte was priced. The read path is 42 +lines and touches only the dense character prefix of each node, and the entire +insert complexity is the price of keeping runs merged. When you meet ART next, +the question to hold is not "which is better" but "which corner": rax has one +node shape sized to the byte, ART has four shapes sized for the probe. ## Done when -You can sketch a compressed vs non-compressed node's `data[]` layout from memory -and say why the pointers are unaligned on purpose. +Answer each before unfolding it. + +- [ ] Compute `raxNodeCurrentLength` by hand for (a) a non-compressed node with + 3 children that is not a key, and (b) a compressed node holding "xyz" + that is a key with a non-NULL value. + +
+Answer + +The formula (`rax.c:150-155`) is +`4 + size + raxPadding(size) + (iscompr ? 8 : 8·size) + (iskey && !isnull ? 8 : 0)`, +and `raxPadding(3) = (8 − ((3+4) mod 8)) & 7 = 1`. + +(a) 4 + 3 + 1 + 8×3 + 0 = **32 bytes**. +(b) 4 + 3 + 1 + 8 + 8 = **24 bytes**. + +The compressed node holds three characters *and* a value in less space than the +branching node needs for three pointers — which is the whole point of Step 2. + +
+ +- [ ] Are rax's child pointers aligned or unaligned? Point at the line that + decides it, and say what it costs. + +
+Answer + +**Aligned.** `raxPadding` at `rax.c:130` inserts 0-7 bytes after the character +section so that `4 + size + padding` is a multiple of 8; since `malloc` returns +8-aligned memory, every child pointer is 8-aligned. +`raxNodeFirstChildPtr` (line 142-145) skips exactly that padding. The cost is +the padding itself — 4 bytes for a leaf, 3 for a 1-child node, 0 for a +4-child node, averaging 3.5 bytes per node over uniform sizes. The `memcpy` +at `rax.c:494` is a strict-aliasing idiom, not evidence of unaligned access. + +
+ +- [ ] Trace a lookup of "foot" against the tree at `rax.h:44-50`. Where does + `raxLowWalk` stop, and which clause of `raxFind` line 937 rejects it? + +
+Answer + +`raxLowWalk` matches "foo" in the compressed root, descends to `[t b]`, matches +'t' at index 0, and descends into the compressed node `("er")`. Now `i == 4 == +len`, so the loop at line 471 exits on the key-exhausted condition. The stop +node is `("er")`, which is compressed, and `splitpos` is 0 — the walk entered +the run but consumed none of it. + +At line 937 the first clause passes (`i == len`) and the second passes +(`splitpos == 0`), so the rejection comes from the **third**: `!h->iskey`. The +node `("er")` is not a key, because "foot" was never inserted. Had the walk +stopped one byte *into* the run — say looking up "foote" — the second clause +`(h->iscompr && splitpos != 0)` would have rejected it instead, and that +clause exists only because runs exist. + +
+ +- [ ] State the compression invariant precisely, and explain why a + single-child node that is a key cannot be folded into a run. + +
+Answer + +From `rax.c:1109-1110`: compressed nodes represent chains of nodes that are +**not keys** *and* have a **single child**. A run stores `size` characters and +exactly one child pointer, plus at most one value pointer at the very end — so +there is exactly one position, the end of the run, at which a value can hang. +Folding a mid-chain node that carries a value would leave nowhere to store it. +That is also why removing a key can *create* a compression opportunity (case 1 +at line 1113): the node stops being a key, so the chain becomes foldable, and +`raxRemove` walks up at lines 1159-1165 to find the highest node that now +qualifies. + +
+ +- [ ] The characters come before the pointers. Name the other structure in + this topic that makes the same choice, and the one number that decides + whether a scalar or SIMD scan of that dense region is faster. + +
+Answer + +SwissTable/hashbrown: one dense byte of control tag per slot, with the fat +key/value slots elsewhere (README §4 calls it "dense filter, fat payload"; +ART Node16's 16-byte key array is the third instance). The deciding number is +the **fanout** — how many bytes the scan must cover. hashbrown always scans a +full group (16 bytes with SSE2, 8 with NEON or the generic fallback), so a +single SIMD compare pays for itself; rax's non-compressed nodes usually have a +handful of children, where loading a vector register costs more than the loop +it replaces. `rax.c:481-483` states this as a claim without a measurement, +which makes it a good exercise: pick a fanout and measure. + +
## References **Code** -- [redis](https://github.com/redis/redis) `src/rax.h`, `src/rax.c` — - the layout comment at rax.h:83–109 is the spec; read it in full before - the functions + +- [redis](https://github.com/redis/redis) at `a176d1225` — verify with + `tools/pinned-source.py ref redis`. + +| File | Lines | What | +|------|-------|------| +| `src/rax.h` | 16-75 | header comment — the tutorial, with the splitting example | +| `src/rax.h` | 77-111 | `RAX_NODE_MAX_SIZE`, `raxNode`, and the layout spec at 83-109 | +| `src/rax.h` | 113-130 | `rax` (with `alloc_size`) and `raxStack` | +| `src/rax.c` | 126-155 | `raxPadding`, the child-pointer macros, `raxNodeCurrentLength` | +| `src/rax.c` | 161-181 | `raxNewNode` / `raxFreeNode` — usable-size accounting | +| `src/rax.c` | 436-506 | `raxLowWalk` and its doc comment | +| `src/rax.c` | 515-913 | `raxGenericInsert`; the case enumeration is 560-682 | +| `src/rax.c` | 931-941 | `raxFind` | +| `src/rax.c` | 1107-1175 | the compression invariant and the recompression walk | +| `src/timeout.c` | 75-83 | `encodeTimeoutKey` — big-endian for byte-order = numeric order | +| `src/stream.h` | 37, 44-45 | streams and consumer groups, the largest rax users | +| `src/tracking.c` | 24-25 | `TrackingTable` and `PrefixTable` | + +**Measured in this repo** + +- `topics/00-performance-toolbox/notes.md`, `lookup_shootout` at n = 10⁶: + `hashmap 8.8 ns`, `btreemap 26.6 ns`, `vec_binary_search 25.8 ns`. rax has + no lane of its own here — its win is not point-lookup latency, it is bytes + per tree and prefix iteration, neither of which that lane measures. If you + want a number, the exercise is to build one. +- `topics/00-performance-toolbox/notes.md`, cache ladder ~1 / 5 / 100 ns — + the prices behind "one dependent hop per node". + +**Companion chapters** + +- [`reading-art-paper.md`](reading-art-paper.md) — the same structure with the + opposite RUM priority. Bring the 40-byte and 152-byte numbers from Step 8. +- [`reading-hashbrown.md`](reading-hashbrown.md) — the other "dense filter, + fat payload" layout in this topic, and the SIMD answer to Step 5's scan. diff --git a/topics/02-in-memory-structures/reading-redis-skiplist.md b/topics/02-in-memory-structures/reading-redis-skiplist.md index d93ca85..abf368e 100644 --- a/topics/02-in-memory-structures/reading-redis-skiplist.md +++ b/topics/02-in-memory-structures/reading-redis-skiplist.md @@ -4,83 +4,228 @@ The canonical readable skiplist — the structure behind ZADD/ZRANGE/ZRANK in `t_zset.c` — with one addition the textbooks skip: every forward link records how many level-0 nodes it jumps over, so summing spans during an ordinary descent yields a node's rank at no extra cost. This chapter builds the -structure from a plain sorted list upward — express lanes, the descent, -spans, the insert bookkeeping — then anchors each piece in the source. Read -it before the RocksDB memtable chapter to see what a skiplist looks like when -concurrency isn't allowed to take features away. +structure from a plain sorted list upward — express lanes, the descent, spans, +the insert bookkeeping — then anchors each piece in the source. Read it before +the RocksDB memtable chapter to see what a skiplist looks like when +concurrency is not allowed to take features away. + +Every anchor below is Redis **8.6.2** (`src/version.h:1`), the commit +`a176d1225` this repo pins. That matters more here than in most chapters, +because this file is not the 2009 skiplist most write-ups describe. Three +things have changed and all three are load-bearing: the node no longer holds +an `sds ele` pointer (the string is *embedded* in the node allocation), the +`span` field at level 0 has been **repurposed** to hold node metadata, and the +zset's dict now stores skiplist *node pointers* rather than member/score +copies. Where this chapter contradicts an older account, check it yourself: + +``` +tools/pinned-source.py show redis src/server.h -r 1690:1716 +tools/pinned-source.py show redis src/t_zset.c -r 75:114 +``` ## The problem in one sentence -A sorted set needs insert, lookup, range-by-score, *and* "what is element +A sorted set needs insert, lookup, range-by-score *and* "what is element #4,217?" — all in O(log n) — and a plain sorted linked list does every one of -them in O(n): at 1M elements that's ~1M dependent pointer hops, milliseconds -per query. +them in O(n): at 1 M elements that is up to 1 M dependent pointer hops, at +~100 ns each when cold ([FINDINGS.md](../../FINDINGS.md) row 0), which is +milliseconds per query. ## The concepts, step by step -### Step 1 — a sorted linked list, and why it's too slow +### Step 1 — a sorted linked list, and why it is too slow + +> **In:** nothing yet — this step establishes the baseline structure and the +> exact quantity every later step attacks. +> **Out:** ordered iteration for free, search in O(n) *dependent* loads. Step +> 2 attacks the O(n). The simplest ordered structure is a linked list kept in key order: each node holds a key and a pointer to the next. Ordered iteration and range scans are -trivial — but *finding* anything means walking from the head, one node at a -time. Each hop is a dependent load (the next address comes from the current -node — topic 0's pointer chase), so a search at n=1M costs up to 1M -serialized cache misses. Arrays fix search (binary search) but make insert -O(n) memmove. We want list-like inserts with search that skips ahead. +trivial. But *finding* anything means walking from the head one node at a +time, and each hop is a **dependent load** — the address of the next node is +inside the current one, so the CPU cannot begin the second fetch until the +first has landed (topic 0's pointer chase). A search at n = 1 M costs up to +1 M serialized cache misses; no amount of memory bandwidth helps, because +there is only ever one outstanding request. + +The array alternative fixes search (binary search, O(log n)) and breaks +insert (O(n) memmove). What we want is list-like insert with search that can +skip ahead. ### Step 2 — express lanes: give random nodes extra levels +> **In:** the sorted list from Step 1. +> **Out:** a tower of progressively sparser lists over the same nodes, with +> the height distribution and its two constants. Step 3 turns that tower into +> a search algorithm. + A **skiplist** keeps the sorted level-0 list and adds sparser "express lanes" -above it: each node is assigned a random **height**, and a node of height h -appears in levels 0..h−1. Heights follow a geometric distribution — flip a -biased coin (redis: p = 0.25, `ZSKIPLIST_P`, server.h:630, max level 32, -`zslRandomLevel()` t_zset.c:254) until it fails. So ~1/4 of nodes reach level -1, ~1/16 level 2, and so on: +above it. Each node is assigned a random **height** h and appears in levels +0..h−1. Heights follow a geometric distribution — flip a biased coin until it +fails: + +```c +// redis@a176d1225 — src/t_zset.c:250-260, zslRandomLevel + 250 /* Returns a random level for the new skiplist node we are going to create. + 251 * The return value of this function is between 1 and ZSKIPLIST_MAXLEVEL + 252 * (both inclusive), with a powerlaw-alike distribution where higher + 253 * levels are less likely to be returned. */ + 254 static int zslRandomLevel(void) { + 255 static const int threshold = ZSKIPLIST_P*RAND_MAX; + 256 int level = 1; + 257 while (random() < threshold) + 258 level += 1; + 259 return (level target — O(log n) expected + search 55: move right while next < target, else drop a level ``` -Expected pointers per node: 1/(1−p) = 1.33 at p=0.25 — cheaper than a binary -tree's 2, and no rebalancing logic exists at all: balance is probabilistic, -not maintained. +The pointer budget follows from the same distribution. A node reaches level k +with probability p^k, so its expected number of forward pointers is -### Step 3 — the descent: the one search algorithm for everything +``` +E[levels per node] = Σ p^k for k = 0,1,2,… = 1/(1 − p) + + at p = 0.25: 1 / (1 − 0.25) = 1 / 0.75 = 1.333 forward pointers per node +``` + +Cheaper than a binary tree's two child pointers — and there is no rebalancing +code at all. Balance is probabilistic, not maintained. That absence is the +skiplist's real selling point: `t_zset.c` implements insert, delete, range and +rank in about 900 lines with no rotation logic anywhere. + +### Step 3 — the descent: one search algorithm for everything -Every skiplist operation starts the same way: begin at the head's top level, +> **In:** the tower from Step 2. +> **Out:** the single traversal pattern every zsl function opens with, and its +> cost priced against topic 0's ladder. Step 4 hangs rank on it for free. + +Every skiplist operation starts identically: begin at the header's top level, move right while the next node's key is still less than the target, and when -it isn't, drop down one level. At the bottom you're standing immediately -before the target position. Expected cost at p=0.25: ~log₄(n) levels × ~3 -compares per level — at n=1M, ~30 dependent pointer hops. Price it with topic -0's ladder: 30 × ~100 ns if every hop misses to DRAM ≈ 3 µs worst case — -that's why the hashbrown chapter's table beats it 5–10× on point lookups, and -why sortedness (not raw speed) is what a skiplist is for. +it is not, drop down one level. At the bottom you are standing immediately +before the target position. In the source this is the loop at +`t_zset.c:277-285` (inside insert), `:651-660` (`zslGetRank`), `:694-703` +(`zslGetElementByRankFromNode`) and `:415-420` (`zslUpdateScore`) — the same +seven lines, four times. + +The cost, with p = 1/4 and the divisions performed: + +``` +levels to descend: log_{1/p}(n) = log₄(1,000,000) + = ln(1e6)/ln(4) = 13.8155 / 1.3863 = 9.97 levels + +forward steps per level: (1 − p)/p = 0.75 / 0.25 = 3.0 + (you expect to pass ~3 nodes before the next one + overshoots, since each has probability p of being + tall enough to have appeared on the lane above) + +forward hops, total: 9.97 × 3.0 = 29.9 ≈ 30 dependent loads + at n = 1e7: 11.63 × 3.0 = 34.9 ≈ 35 +``` + +Only the *forward* hops are dependent loads. Dropping a level is free: `level[]` +is a flexible array inside the node's own allocation (`server.h:1707`), so +`level[i]` and `level[i−1]` are 16 bytes apart in a line you already have. + +Price the 30 hops with topic 0's ladder. If every one missed to DRAM: +30 × ~100 ns = **~3.0 µs** — a hard upper bound, and a bad model, because the +top of the tower is traversed by *every* search and stays in L1 while the +bottom levels are cold. The honest statement is: a skiplist lookup costs tens +of dependent misses where hashbrown costs two (its +[chapter](reading-hashbrown.md), Step 3), which is why topic 0's +`lookup_shootout` shows HashMap at 8.8 ns and the ordered structures 3-5× +worse at n = 1e6 (BTreeMap 26.6 ns, sorted-vec binary search 25.8 ns — +[topic 0 notes](../00-performance-toolbox/notes.md)). You do not choose a +skiplist for point-lookup speed. You choose it for what Steps 4 and 5 add on +top of a search you were doing anyway. ### Step 4 — spans: count what you skip, and rank is free -A **rank** query ("what is 55's index?", "give me elements 100–110") needs to +> **In:** the descent from Step 3, which already visits O(log n) links. +> **Out:** rank queries in O(log n) with no auxiliary structure — plus the +> level-0 encoding trick that pays for the node metadata. Step 6 pays the +> maintenance bill. + +A **rank** query ("what index is 55?", "give me elements 100-110") needs to know *how many* level-0 nodes each express-lane jump flew over. Redis stores -exactly that: each forward link carries a **span** — the number of level-0 -nodes it skips. The structs (server.h:1699–1716): +exactly that: each forward link carries a **span**, the number of level-0 +nodes it skips. + +```c +// redis@a176d1225 — src/server.h:1690-1709, the node and its info word + 1690 /* ZSETs use a specialized version of Skiplists */ + 1691 + 1692 /* Node info placed in level[0].span since it's unused at level 0 (static assert verified) */ + 1693 typedef struct zskiplistNodeInfo { + 1694 uint16_t sdsoffset; /* Offset from node start to sds data (after sds header) */ + 1695 uint8_t levels; /* Number of levels in this node (1-32) */ + 1696 uint8_t reserved; + 1697 } zskiplistNodeInfo; + 1698 + 1699 typedef struct zskiplistNode { + 1700 double score; + 1701 struct zskiplistNode *backward; + 1702 struct zskiplistLevel { + 1703 struct zskiplistNode *forward; + // ... 1704-1705: comment reproduced in the prose below ... + 1706 unsigned long span; + 1707 } level[]; + 1708 /* sds ele is embedded after level[] array (assist zslGetNodeElement(node) to access it) */ + 1709 } zskiplistNode; +``` + +Note what is *not* there: no `sds ele` field. The member string lives inside +the same allocation, past the end of `level[]` (line 1708), reached through +the byte offset stored in `zskiplistNodeInfo.sdsoffset`. One `zmalloc` per +node holds score, backward pointer, the whole level array and the string — +which is the difference between one cache miss and two when you finally +compare the key. + +And the trick that pays for that offset: **level 0's span is always 1**, so +the field is dead weight, so redis puts the node metadata there instead. ```c -typedef struct zskiplistNode { - sds ele; double score; - struct zskiplistNode *backward; // level-0 doubly-linked - struct zskiplistLevel { - struct zskiplistNode *forward; - unsigned long span; // # of L0 nodes this link jumps over - } level[]; // flexible array: height varies per node -} zskiplistNode; +// redis@a176d1225 — src/t_zset.c:75-81 and 101-104, span synthesised at level 0 + 75 static inline unsigned long zslGetNodeSpanAtLevel(zskiplistNode *x, int level) { + // ... 76-77: comment — at level 0, span stores node info instead of distance ... + 78 if (level > 0) return x->level[level].span; + 79 /* For level 0, if regular node, span is 1. If tail node, span is 0. */ + 80 return x->level[0].forward ? 1 : 0; + 81 } + // ... 83-99: Set / Incr / Decr, each a no-op when level == 0 ... + 101 /* Get zskiplistNodeInfo from node (stored in level[0].span). */ + 102 static_assert(sizeof(zskiplistNodeInfo) <= sizeof(((zskiplistNode *)0)->level[0].span), "Must fit in level[0].span"); + 103 static inline zskiplistNodeInfo *zslGetNodeInfo(const zskiplistNode *node) { + 104 return (zskiplistNodeInfo *)&node->level[0].span; + 105 } ``` -Now the ordinary descent computes rank as a side effect — sum the spans of -every link you traverse: +Line 80 *computes* the level-0 span instead of reading it; lines 85, 91 and 97 +silently skip writes at level 0; line 102 is the static assertion that the +4-byte info struct fits in the 8-byte span slot it is squatting in. Free +metadata, at the cost of every span access going through an accessor — which +is exactly why the insert code in Step 6 never touches `.span` directly. + +With spans present, the ordinary descent computes rank as a side effect: ```rust +// ILLUSTRATION — not quoted from redis; the shape of t_zset.c:645-662. +// The real zslGetRank compares {score, ele} via zslCompareWithNode +// (t_zset.c:120) and reads spans through zslGetNodeSpanAtLevel (t_zset.c:75). fn rank_of(list: &SkipList, target: &Key) -> u64 { let mut node = &list.head; let mut rank = 0u64; @@ -98,79 +243,396 @@ fn rank_of(list: &SkipList, target: &Key) -> u64 { } ``` -That's ZRANK and ZRANGE-by-index in O(log n) with zero extra structure — the -descent was happening anyway. The cost: every insert and delete must keep -every affected span exact (Step 6). +The real `zslGetRank` is `t_zset.c:645-662`; it also checks at every level +whether it has *landed on* the target (line 657) so it can return early. The +inverse operation, ZRANGE-by-index, is the same walk with the comparison +replaced by a running total against the wanted rank +(`zslGetElementByRankFromNode`, `t_zset.c:688-705`). + +The bill for all this is one `unsigned long` per forward link and the +requirement that every insert and delete keep every affected span exact. + +### Step 5 — backward pointers, and rank without comparisons + +> **In:** the level-0 list and the spans from Step 4. +> **Out:** reverse ranges as a plain walk, and a second rank algorithm that +> exploits both. Step 7 explains why neither survives concurrency. + +Level 0 is doubly linked: `backward` (`server.h:1701`) makes ZREVRANGE a plain +walk from `zsl->tail`, with no descent and no cleverness. Only level 0 gets +it; higher levels would double the pointer cost for no query redis runs. + +The backward pointer plus spans also enable a rank algorithm the textbooks do +not have — one that avoids string comparison entirely: + +```c +// redis@a176d1225 — src/t_zset.c:672-685, zslGetRankByNode + 672 unsigned long zslGetRankByNode(zskiplist *zsl, zskiplistNode *x) { + 673 unsigned long distance_to_end = 0; + 674 int level; + // ... 675-676: comment — walk forward to the end, jumping at each node's top level ... + 677 while (x) { + 678 level = zslGetNodeInfo(x)->levels - 1; + 679 distance_to_end += zslGetNodeSpanAtLevel(x, level); + 680 x = x->level[level].forward; + 681 } + 682 + 683 /* Rank = total nodes - nodes after this one */ + 684 return zsl->length - distance_to_end; + 685 } +``` -### Step 5 — backward pointers: reverse ranges as a list walk +Given a node pointer (which, per Step 7, is what the zset's dict hands you), +this walks *forward* to the tail always taking the current node's tallest +lane, sums the spans it crosses, and subtracts from `zsl->length`. Same +O(log n), but zero `sdscmp` calls — the doc comment at 664-671 says so +outright. It is only possible because `levels` is stored in the node +(Step 4's repurposed word) and because spans are exact. -Level 0 is doubly linked: each node's `backward` pointer makes ZREVRANGE a -plain walk from the tail — no descent, no cleverness. Only level 0 gets this -(higher levels would double the pointer cost for no query redis runs). Note -for later: a backward pointer is a *second* pointer that must be updated -atomically-with the forward one — trivial single-threaded, poison for -lock-free designs (Step 7). +There is a warning hidden here for Step 7: `backward` is a *second* pointer +that must change in the same logical instant as the corresponding `forward`. +Trivial when one thread owns the structure; very hard to make atomic without +locks. ### Step 6 — insert: remember the splice points on the way down -`zslInsert` (t_zset.c:265–339) is the heart. One descent records, per level: +> **In:** the descent (Step 3), spans (Step 4), and a node whose height was +> already drawn by `zslRandomLevel`. +> **Out:** the two arrays that make a single descent sufficient, and the three +> distinct span updates an insert must perform. Step 7 explains what makes +> them safe. + +`zslInsertNode` (`t_zset.c:265-321`) is the heart. Note the split: the public +`zslInsert` (`t_zset.c:326-339`) only draws a height (line 335), allocates the +node (336) and delegates — an earlier version of this chapter cited +"`zslInsert`, t_zset.c:265-339", which merges the two functions; 265 is +`zslInsertNode` and 326 is `zslInsert`. + +One descent records, per level i: -- `update[i]` — the rightmost node at level i that precedes the insert point - (the nodes whose forward pointers must be spliced); -- `rank[i]` — cumulative span up to `update[i]` (so new spans can be computed - without re-walking). +- `update[i]` — the rightmost node at level i preceding the insert point (the + nodes whose forward pointers must be spliced), `t_zset.c:284`; +- `rank[i]` — the cumulative level-0 distance travelled to reach `update[i]`, + `t_zset.c:279-281`, so new spans can be computed without re-walking. ``` -insert 55, height 2: update[] captured on the way down +insert 55, height 2: update[]/rank[] captured on the way down L2 ──────► 17 ────────────────► 71 update[2]=17 rank[2]=2 L1 ──────► 17 ────► 42 ─[55]──► 71 update[1]=42 rank[1]=3 splice L0 ─► 8 ─► 17 ─► 29 ─► 42 ─[55]► 71 update[0]=42 rank[0]=3 splice - levels above height: span += 1 only + levels above height 2: span += 1 only +``` + +Then the splice, which is where the span algebra lives: + +```c +// redis@a176d1225 — src/t_zset.c:298-311, splice and span fix-up + 298 /* Insert the node at the found position */ + 299 for (i = 0; i < level; i++) { + 300 node->level[i].forward = update[i]->level[i].forward; + 301 update[i]->level[i].forward = node; + 302 + 303 /* update span covered by update[i] as node is inserted here */ + 304 zslSetNodeSpanAtLevel(node, i, zslGetNodeSpanAtLevel(update[i], i) - (rank[0] - rank[i])); + 305 zslSetNodeSpanAtLevel(update[i], i, (rank[0] - rank[i]) + 1); + 306 } + 307 + 308 /* increment span for untouched levels */ + 309 for (i = level; i < zsl->level; i++) { + 310 zslIncrNodeSpanAtLevel(update[i], i, 1); + 311 } ``` -Then splice the new node in at each level ≤ its height, computing its spans -from the rank differences. Note the span bookkeeping at t_zset.c:304–305: -levels *above* the new node's height don't get a new link, but their spans -still grow by one — a node now exists underneath them. Subtle, and the kind -of invariant your own implementation will get wrong first try. +Read `rank[0] − rank[i]` as "how many level-0 nodes lie between `update[i]` +and the insert point". Line 305 sets `update[i]`'s new span to that distance +plus one (the new node itself); line 304 gives the new node the remainder of +the old span. Their sum is the old span plus one, which is the invariant a +verification routine would check — and redis ships one, `zslDebugVerifyStruct` +at `t_zset.c:4817`, worth reading before you debug your own. + +Lines 309-311 are the case people get wrong first: levels *above* the new +node's height get no new link, but a node now exists underneath them, so +their spans still grow by one. (An earlier version of this chapter cited +304-305 for this; that pair is the splice arithmetic, and the above-height +increment is 309-311.) There is a third case at `t_zset.c:288-296`: when the +new node is taller than the list has ever been, the header's links at the new +levels are created with span `zsl->length` — they leap the entire existing +list. + +Finally, note what `zslUpdateScore` (`t_zset.c:396-430`) does with all this. +If the new score keeps the node between its current neighbours (the two-line +test at 400-401), it writes `node->score` and returns — no splice, no span +touched, O(1). Otherwise it unlinks and reinserts *the same node*, so the +dict's pointer stays valid (comment at 425-426). A structure whose identity +survives repositioning is what makes Step 7's design possible. + +### Step 7 — what single-threading buys: features, and one allocation + +> **In:** every mechanism above — spans, backward pointers, the descent. +> **Out:** the reason all of them are affordable here, and the two design +> choices that follow from it. The RocksDB chapter is the contrast. + +No locks, no CAS (compare-and-swap — the atomic primitive lock-free structures +are built from). Redis is single-threaded on the data path, so this skiplist +can afford operations that touch several pointers at once: an insert writes +`update[i]->level[i].forward`, `node->level[i].forward`, two spans per level, +and two backward pointers. Making that sequence appear atomic to concurrent +readers is exactly the problem RocksDB's `InlineSkipList` avoids by *deleting +the features* — no backward pointers, no spans, no deletes (next chapter). +Concurrency removes features; topic 9 makes that precise. + +The same freedom shows up in the memory layout. `zslCreateNode` +(`t_zset.c:169-205`) computes `node_size + sds_buf_size` and makes **one** +allocation (line 181) holding the node, its level array, and a copy of the +member string placed with `sdsnewplacement` (line 193): -### Step 7 — what single-threading buys: features +``` +one zmalloc: [ score 8 | backward 8 | level[0..h-1] 16h | sds hdr | member bytes ] + ▲ level[0].span holds {sdsoffset, levels} + h = 1 (75% of nodes): 8 + 8 + 16 = 32 B + string + h = 1.333 (expected): 8 + 8 + 16 × 1.333 ≈ 37 B + string +``` -No locks, no CAS (compare-and-swap — the atomic instruction lock-free -structures are built from) — redis is single-threaded on the data path, so -this skiplist is free to use backward pointers and spans, both of which -require multi-pointer updates that are hard to make atomic without locks. -Contrast with RocksDB's `InlineSkipList` (next chapter): concurrent writers -⇒ no backward pointers, no spans, no deletes. Concurrency *removes* features -— a theme topic 9 makes precise. +And the second structure in a zset is now cheaper than the folklore says. The +dict does not store a copy of the member and a copy of the score; it stores +the **node pointer**: + +```c +// redis@a176d1225 — src/t_zset.c:53-64, the zset's dict is a set of node pointers + 53 /* dictType for zset's dict (maps sds to zskiplistNode*) */ + 54 dictType zsetDictType = { + 55 dictSdsHash, /* hash function */ + // ... 56-57: key dup / val dup, both NULL ... + 58 dictSdsKeyCompare, /* compares embedded sds by keyFromStoredKey */ + 59 NULL, /* key destructor - skiplist owns the node memory */ + // ... 60-61: val destructor, allow-to-expand ... + 62 .no_value = 1, /* no values stored (only nodes) */ + 63 .keyFromStoredKey = zslGetNodeElementForDict, /* extract embedded sds from node */ + 64 }; +``` + +`dictAdd(zs->dict, node, NULL)` at `t_zset.c:1486` inserts the *node* as the +key; `keyFromStoredKey` (line 63) tells the dict how to find the sds inside it +(`zslGetNodeElement`, `t_zset.c:129-133`); line 59 records that the skiplist +owns the memory. Because `.no_value = 1`, the dict can store that pointer +directly in the bucket with no `dictEntry` allocation whenever the bucket +holds one key or the key sits at a chain tail (`dict.h:17-25`). So the index +costs roughly the bucket array plus tag bits — not a second copy of every +member — and a ZSCORE returns the node, from which the score is one field +away. ## Where each step lives in the code -- **Steps 2, 4, 5** — the structs: `zskiplistNode` / `zskiplistLevel` — - server.h:1699–1716; `span` and `backward` fields. -- **Step 2** — `zslRandomLevel()` — t_zset.c:254; `ZSKIPLIST_P` (0.25) — - server.h:630, max level 32. Compare: RocksDB uses branching factor 4 (same - p) but caps at 12. -- **Steps 3–4** — the descent pattern opens every zsl function; rank - accumulation visible in `zslGetRank` and inside `zslInsert`. -- **Step 6** — `zslInsert` — t_zset.c:265–339; the above-height span - increment at t_zset.c:304–305. +| Lines | What | Step | +|---|---|---| +| `server.h:629-630` | `ZSKIPLIST_MAXLEVEL` 32, `ZSKIPLIST_P` 0.25 | 2 | +| `server.h:1692-1697` | `zskiplistNodeInfo` — what squats in level[0].span | 4 | +| `server.h:1699-1709` | `zskiplistNode` — no `ele` field; sds embedded after `level[]` | 4 | +| `server.h:1711-1716` | `zskiplist` — header, tail, length, level, `alloc_size` | 2 | +| `t_zset.c:53-64` | `zsetDictType` — `no_value`, `keyFromStoredKey` | 7 | +| `t_zset.c:75-99` | span accessors; level 0 is synthesised, never written | 4 | +| `t_zset.c:101-114` | `zslGetNodeInfo` / `zslSetNodeInfo` + the `static_assert` | 4 | +| `t_zset.c:120-133` | `zslCompareWithNode`, `zslGetNodeElement` (offset → sds) | 3 | +| `t_zset.c:169-205` | `zslCreateNode` — the single allocation | 7 | +| `t_zset.c:250-260` | `zslRandomLevel` — the coin, and the cap | 2 | +| **`t_zset.c:265-321`** | **`zslInsertNode` — descent, `update[]`, `rank[]`, splice** | 6 | +| `t_zset.c:277-285` | the descent, in its canonical form | 3 | +| `t_zset.c:288-296` | new-tallest-level case: header spans = `zsl->length` | 6 | +| `t_zset.c:299-306` | splice + span algebra | 6 | +| `t_zset.c:309-311` | above-height spans += 1 | 6 | +| `t_zset.c:326-339` | `zslInsert` — draw height, allocate, delegate | 6 | +| `t_zset.c:345-366` | `zslUnlinkNode` — the same algebra in reverse | 6 | +| `t_zset.c:396-430` | `zslUpdateScore` — O(1) fast path, node identity preserved | 6 | +| `t_zset.c:645-662` | `zslGetRank` — spans summed during the descent | 4 | +| `t_zset.c:672-685` | `zslGetRankByNode` — forward walk, no string compares | 5 | +| `t_zset.c:688-705` | `zslGetElementByRankFromNode` — the descent, inverted | 4 | +| `t_zset.c:4817` | `zslDebugVerifyStruct` — the invariants, as code | 6 | + +Read in this order: + +1. **`server.h:1690-1716`** (Step 4) — the structs. Ask where the member + string is before reading line 1708. +2. **`t_zset.c:75-114`** (Step 4) — the accessors. Once you see that level 0's + span is computed rather than stored, the rest of the file's insistence on + `zslGetNodeSpanAtLevel` stops looking like ceremony. +3. **`t_zset.c:250-260`** (Step 2) — three lines of randomness, the entire + balance strategy. +4. **`t_zset.c:265-321`** (Step 6) — `zslInsertNode`. Work the example in Step + 6 by hand and check your spans against lines 304-305. +5. **`t_zset.c:645-685`** (Steps 4-5) — the two rank algorithms side by side. + The second one is the payoff for the metadata word in Step 4. +6. **Aha: `t_zset.c:4817`** — `zslDebugVerifyStruct`. Every invariant this + chapter states, written as assertions. Port it into your own + implementation before you port anything else. + +**Contrast case.** Compare `zslInsertNode`'s span bookkeeping with what +`InlineSkipList` does at the equivalent moment (next chapter): nothing, because +it has no spans to keep. Then ask what ZRANK would cost without them — an O(n) +walk of level 0, or a second index — and you have priced the feature. ## Questions to answer in notes.md -1. Why does the zset need *both* the skiplist and a dict (score lookup by member)? - What does that cost in memory, and what's the RUM read? -2. Derive the expected search cost at p=0.25: levels × nodes-per-level ≈ - log₄(n) × ~3 compares. At n=1M: ~30 dependent pointer hops — now price it with - topic 0's ladder (30 × ~100ns if cold). Compare your measured number. +1. Why does a zset need *both* the skiplist and a dict? Look at + `t_zset.c:53-64` and `:1486` before answering the memory half: the dict + stores node pointers with `.no_value = 1`, so what exactly is duplicated + and what is not? State the RUM trade-off in one sentence. +2. Derive the expected search cost at p = 0.25 and check the derivation in + Step 3: log₄(n) levels × (1−p)/p forward steps. Do it for n = 1e6 and + n = 1e7, price both with topic 0's ~100 ns cold-miss figure, then say why + the resulting number is an upper bound rather than an estimate. +3. Level 0's span field holds `zskiplistNodeInfo` instead of a span + (`server.h:1692`, `t_zset.c:75-81`). What does that buy, what does it cost, + and what would break if someone wrote `x->level[0].span = 1` directly? +4. `zslGetRankByNode` (`t_zset.c:672-685`) walks *forward* to the tail rather + than descending from the header. Why is that the same O(log n), and which + property of the height distribution makes it so? +5. Your own skiplist has to choose a node layout. Redis makes one allocation + containing score, levels and the member string (`t_zset.c:169-205`). Write + down what you will do instead and what it costs you in cache misses per + comparison — this is the notes.md line "Implementation trade I chose for + skiplist node layout, and why". + +## Takeaway + +A skiplist is not competitive with a hash table on point lookups and is not +trying to be — topic 0 measured the gap at 3-5× against ordered structures +generally. It is competitive on *what else the descent can carry*. Redis hangs +three things on a traversal it was doing anyway: exact rank (spans), reverse +iteration (backward pointers) and node metadata (a repurposed dead word). +Every one of them costs multi-pointer updates, which is precisely the currency +a concurrent implementation cannot spend. ## Done when -You can explain spans to someone in two sentences, and you know which features your -experiment's skiplist can steal (backward/span) vs what RocksDB's concurrency forbids. +Answer each before unfolding it. + +- [ ] You can explain spans in two sentences, and say what they cost. + +
Answer + + A span is the number of level-0 nodes a given forward link jumps over, so + summing the spans of the links you traverse during an ordinary descent gives + you the rank of where you landed — ZRANK and ZRANGE-by-index in O(log n) + with no auxiliary structure (`zslGetRank`, `t_zset.c:645-662`). The cost is + one `unsigned long` per forward link, plus the obligation that every insert + and delete keep every affected span exact: `t_zset.c:304-305` for the + spliced levels, `:309-311` for the levels above the new node's height, and + `:288-296` when the list grows taller. + +
+ +- [ ] You can say where the member string lives and why the answer is not "in an `sds ele` field". + +
Answer + + There is no `ele` field. `zslCreateNode` (`t_zset.c:169-205`) computes + `node_size + sds_buf_size` and makes a single allocation (line 181) holding + the score, the backward pointer, `level[0..h−1]`, and a copy of the member + placed in-line with `sdsnewplacement` (line 193). The byte offset from the + node's start to the string is stored in `zskiplistNodeInfo.sdsoffset` + (`server.h:1694`) and read back by `zslGetNodeElement` (`t_zset.c:129-133`). + + The payoff is one cache miss instead of two when a comparison finally has to + look at the key — every `zslCompareWithNode` (`t_zset.c:120-126`) that gets + past the score check reads a string already on a line the node touched. + +
+ +- [ ] You can explain why level 0's `span` field does not hold a span. + +
Answer + + Because a level-0 link always jumps exactly one node, so the value is + constant and storing it is waste. Redis puts a `zskiplistNodeInfo` + (`sdsoffset`, `levels`, `reserved` — `server.h:1693-1697`) in that word + instead, guarded by a `static_assert` that it fits (`t_zset.c:102`). + `zslGetNodeSpanAtLevel` therefore *computes* level 0's span — 1, or 0 at the + tail (`t_zset.c:78-80`) — and the setter, incrementer and decrementer all + skip level 0 (`:85`, `:91`, `:97`). + + It buys the node's height and its string offset for free, which is what + makes `zslGetRankByNode` (`t_zset.c:678`) and `zslGetNodeElement` possible. + It costs one indirection on every span access and one very sharp edge: + writing `x->level[0].span` directly would silently corrupt the node's height + and string offset at once. + +
+ +- [ ] You can derive the search cost at p = 0.25 rather than quoting it. + +
Answer + + A level-k lane holds about n·p^k nodes, so the tower is + log_{1/p}(n) levels tall: at n = 1e6 and p = 1/4 that is + ln(1e6)/ln(4) = 13.8155/1.3863 = **9.97** levels. Within a level you expect + to pass (1−p)/p = 0.75/0.25 = **3.0** nodes before the next one is tall + enough to have appeared on the lane above. Total forward hops: + 9.97 × 3.0 = **29.9 ≈ 30**; at n = 1e7, 11.63 × 3.0 ≈ 35. + + Only forward hops are dependent loads — dropping a level reads + `level[i−1]` in the node you are already standing on (`server.h:1707`). + Multiplying 30 × ~100 ns gives ~3.0 µs, which is an *upper* bound assuming + every hop misses to DRAM; the top of the tower is walked by every search and + stays cached, so the real figure is much lower. The comparison that matters + is structural: tens of dependent misses here against two for hashbrown. + +
+ +- [ ] You can name the features single-threading buys, and predict which ones the next chapter loses. + +
Answer + + Redis is single-threaded on the data path, so an operation may touch many + pointers before anyone else looks. That affords: **backward pointers** + (`server.h:1701`), which make ZREVRANGE a plain tail walk; **spans**, which + require updating O(log n) counters per insert (`t_zset.c:299-311`); + **deletes**, which require the same algebra in reverse + (`zslUnlinkNode`, `:345-366`); and **in-place score updates** that keep the + node's address stable so the dict's pointer stays valid (`:396-430`). + + RocksDB's `InlineSkipList` supports concurrent writers, so it drops all + four: no backward pointers, no spans, no deletes, no repositioning. Its + insert is a per-level CAS on a single `next` pointer, which is the largest + update it can make atomically. Concurrency does not make the structure + faster; it makes it do less. + +
## References **Code** -- [redis](https://github.com/redis/redis) `src/t_zset.c` (zslInsert, - zslRandomLevel) — struct definitions in `src/server.h:1699–1716` +- [redis](https://github.com/redis/redis) — pinned at **8.6.2** / + `a176d1225` (`src/version.h:1`). The skiplist is `src/t_zset.c`, its structs + are in `src/server.h`. + +| File | Lines | What | +|---|---|---| +| `src/server.h` | 629-630 | `ZSKIPLIST_MAXLEVEL` = 32, `ZSKIPLIST_P` = 0.25 | +| `src/server.h` | 1692-1697 | `zskiplistNodeInfo` — the word stored in level[0].span | +| `src/server.h` | 1699-1709 | `zskiplistNode`; note the absent `ele` field | +| `src/t_zset.c` | 53-64 | `zsetDictType` — the dict indexes node pointers | +| `src/t_zset.c` | 75-99 | span accessors; level 0 synthesised, never written | +| `src/t_zset.c` | 102 | the `static_assert` that makes the repurposing legal | +| `src/t_zset.c` | 169-205 | `zslCreateNode` — one allocation, embedded sds | +| `src/t_zset.c` | 250-260 | `zslRandomLevel` | +| `src/t_zset.c` | 265-321 | `zslInsertNode` — the descent and all three span cases | +| `src/t_zset.c` | 326-339 | `zslInsert` — the wrapper that draws the height | +| `src/t_zset.c` | 396-430 | `zslUpdateScore` — O(1) when order is unchanged | +| `src/t_zset.c` | 645-662 | `zslGetRank` | +| `src/t_zset.c` | 672-685 | `zslGetRankByNode` — no string comparisons | +| `src/t_zset.c` | 4817 | `zslDebugVerifyStruct` — the invariants as assertions | +| `src/dict.h` | 17-25 | why a `no_value` dict needs no `dictEntry` allocation | + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 0 — the ~1 / 5 / 100 ns cache ladder + used to price the descent. +- [topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md) + — `lookup_shootout` at n = 1e6: HashMap 8.8 ns, BTreeMap 26.6 ns, sorted-vec + binary search 25.8 ns. The ordered-structure penalty, measured. + +**Companion chapters** +- [reading-rocksdb-memtable.md](reading-rocksdb-memtable.md) — the same + structure with concurrent writers, and the features that removes. +- [reading-hashbrown.md](reading-hashbrown.md) — the two-cache-line point + lookup this chapter is compared against. +- [reading-redis-dict.md](reading-redis-dict.md) — the other half of a zset. diff --git a/topics/02-in-memory-structures/reading-rocksdb-memtable.md b/topics/02-in-memory-structures/reading-rocksdb-memtable.md index 339940a..7ce9506 100644 --- a/topics/02-in-memory-structures/reading-rocksdb-memtable.md +++ b/topics/02-in-memory-structures/reading-rocksdb-memtable.md @@ -2,13 +2,25 @@ This is where LSM write throughput lives: every `Put` in half the industry lands in this one header. Two ideas are the whole file — a node layout that -puts the hot pointer and the key on the same cache line by indexing the tower -*negatively*, and a concurrency contract kept simple by one workload +puts the hot pointer and the key in the same allocation by indexing the tower +*negatively*, and a concurrency contract kept small by one workload restriction: memtables never delete, they freeze and drop wholesale. This -chapter builds up to both — what a memtable is, why it's a skiplist, the +chapter builds up to both — what a memtable is, why it is a skiplist, the layout trick, then the lock-free insert — before pointing you at the lines. Budget: 1–2 h. +Everything below is read against **facebook/rocksdb at `7c80a5a`**, where +`memtable/inlineskiplist.h` is 1422 lines. Line numbers move between releases; +re-check yours with: + +``` +tools/pinned-source.py ref rocksdb +tools/pinned-source.py show rocksdb memtable/inlineskiplist.h -r 350:422 +``` + +If a number below does not match what you see, trust your checkout and record +the drift in `notes.md` — that is the exercise, not an error report. + ## The problem in one sentence Eight writer threads must insert into one sorted in-memory structure at @@ -19,159 +31,736 @@ sorted map caps the whole LSM engine at one core. ### Step 1 — the memtable: the sorted buffer every LSM write hits first +> **In:** an LSM engine that must accept writes at memory speed and hand disk +> a *sorted* file. +> **Out:** three requirements — ordered iteration, concurrent insert — and one +> non-requirement, delete, which is the lever the rest of the file pulls. + In an LSM engine (topic 1), every write goes to an in-memory buffer — the -**memtable** — which, when full (RocksDB default 64 MB), is **frozen** (made -immutable), flushed to disk as a sorted file, and then dropped wholesale. -Two requirements follow: the structure must support **sorted iteration** -(the flush writes a sorted file), and it must absorb **concurrent inserts** -from many writer threads. One non-requirement matters just as much: it never -needs to *delete* a node — even a user's Delete is an insert (a tombstone -entry); physical removal happens only when the whole frozen memtable is -dropped at once. +**memtable** — which, when full, is **frozen** (made immutable), flushed to +disk as a sorted file, and then dropped wholesale. The full threshold is +`write_buffer_size`, default **64 MiB**: + +``` +// include/rocksdb/options.h — write_buffer_size, 175-191 + 175 // Amount of data to build up in memory (backed by an unsorted log + 176 // on disk) before converting to a sorted on-disk file. + 177 // + 178 // Larger values increase performance, especially during bulk loads. + ... 179-187: max_write_buffer_number, recovery time, per-column-family note ... + 188 // Default: 64MB + ... 189-190: dynamically changeable through SetOptions() ... + 191 size_t write_buffer_size = 64 << 20; +``` + +Two requirements follow from "converting to a sorted on-disk file": the +structure must support **sorted iteration**, and it must absorb **concurrent +inserts** from many writer threads — RocksDB turns those on by default: + +``` +// include/rocksdb/options.h — allow_concurrent_memtable_write, 1421-1429 + 1421 // If true, allow multi-writers to update mem tables in parallel. + 1422 // Only some memtable_factory-s support concurrent writes; currently it + 1423 // is implemented only for SkipListFactory. Concurrent memtable writes + 1424 // are not compatible with inplace_update_support or filter_deletes. + ... + 1429 bool allow_concurrent_memtable_write = true; +``` + +One **non**-requirement matters just as much: the structure never needs to +*delete* a node. Even a user's `Delete` is an insert (a tombstone entry); +physical removal happens only when the whole frozen memtable is dropped at +once. Hold onto that — Step 6 spends it. ### Step 2 — why a skiplist, not a hash table or B-tree +> **In:** the two requirements from Step 1 — ordered iteration and concurrent +> insert. +> **Out:** the elimination argument that leaves a skiplist, and the property +> that makes it CAS-able: an insert touches independent single words. + A hash table has no ordered iteration — flushing to a sorted file would -require an O(n log n) sort of 64 MB on every flush. A B-tree keeps order but -inserts trigger node splits — multi-node rewrites that need complex latching -under concurrency. A skiplist (previous chapter) keeps order, and an insert -touches only a handful of *independent* forward pointers — each one a single -word that can be swapped atomically with **CAS** (compare-and-swap: an atomic -CPU instruction that writes a new value only if the location still holds the -expected old value, and reports failure otherwise). Independent single-word -updates are exactly what lock-free programming can handle. That's the whole -case: sortedness + CAS-able inserts. +require sorting 64 MiB of entries on every flush. A B-tree keeps order but +inserts trigger node splits: multi-node rewrites that need real latching under +concurrency, because a split must appear atomic to a concurrent reader +descending through it. + +A skiplist (previous chapter) keeps order, and an insert touches only a +handful of *independent* forward pointers — each one a single word that can be +swapped atomically with **CAS** (compare-and-swap: one atomic CPU instruction +that writes a new value only if the location still holds the expected old +value, and reports failure otherwise). Independent single-word updates are +exactly the shape lock-free programming can handle without a multi-word atomic +primitive that no hardware provides. That is the whole case: sortedness plus +CAS-able inserts. + +The price is honest and this repo has measured it. Topic 0's `lookup_shootout` +lane at n = 10⁶ reports `hashmap 8.8 ns`, `btreemap 26.6 ns`, +`vec_binary_search 25.8 ns` per lookup. A skiplist is a *worse* pointer chase +than a B-tree — same dependent misses, less fanout per cache line — so the +ordered structures here are already ~3× the hash table on point lookups, and a +skiplist lands at or above the B-tree. RocksDB accepts that ratio to buy +concurrent writers and a sorted flush. Different RUM position, deliberately +chosen. ### Step 3 — the node layout: one allocation, tower indexed negatively -A textbook skiplist node holds a key pointer and an array of forward -pointers — so a lookup touches the node, then the key, two dependent misses. -InlineSkipList (lines 358–421) makes a node **one allocation, three regions, -with the struct pointing at the middle**: +> **In:** a skiplist node needs a key, a height, and `height` forward +> pointers. +> **Out:** a three-region single allocation with the `Node*` pointing at the +> *middle*, so the hot fields (level-0 link and the key) are adjacent and the +> cold tower sits behind the pointer. + +A textbook skiplist node holds a key *pointer* and an array of forward +pointers — so a lookup touches the node, then chases the pointer to the key: +two dependent misses per comparison. RocksDB's own header opens by naming the +saving it is after: + +``` +// memtable/inlineskiplist.h — file header, 10-18 + 10 // InlineSkipList is derived from SkipList (skiplist.h), but it optimizes + 11 // the memory layout by requiring that the key storage be allocated through + 12 // the skip list instance. For the common case of SkipList this saves 1 pointer per skip list node and gives better cache + 14 // locality, at the expense of wasted padding from using AllocateAligned + 15 // instead of Allocate for the keys. The unused padding will be from + 16 // 0 to sizeof(void*)-1 bytes, and the space savings are sizeof(void*) + 17 // bytes, so despite the padding the space used is always less than + 18 // SkipList. +``` + +The comment above `Node` says how, and it is the single most surprising line +in the file: + +``` +// memtable/inlineskiplist.h — Node layout comment, 352-356 + 352 // The Node data type is more of a pointer into custom-managed memory than + 353 // a traditional C++ struct. The key is stored in the bytes immediately + 354 // after the struct, and the next_ pointers for nodes with height > 1 are + 355 // stored immediately _before_ the struct. This avoids the need to include + 356 // any pointer or sizing data, which reduces per-node memory overheads. +``` + +So one allocation, three regions, `Node*` aimed at the middle: + +``` + raw allocation (AllocateNode, line 868): + + ┌───────────────────────────┬───────────────┬──────────────────┐ + │ tower: next_[-(h-1)]…[-1] │ Node: next_[0]│ key bytes inline │ + └───────────────────────────┴───────────────┴──────────────────┘ + prefix = 8*(h-1) bytes ▲ Node* points HERE + │ + level n reached by NEGATIVE index (&next_[0] - n) line 383 + key reached as (&next_[1]) line 374 +``` + +Both tricks are one line each, and the struct that makes them legal is a +one-element array: ``` - raw allocation (from concurrent arena, line 860-869): +// memtable/inlineskiplist.h — Node accessors, 374-396 and 417-420 + 374 const char* Key() const { return reinterpret_cast(&next_[1]); } + 375 + 376 // Accessors/mutators for links. Wrapped in methods so we can add + 377 // the appropriate barriers as necessary, and perform the necessary + 378 // addressing trickery for storing links below the Node in memory. + 379 Node* Next(int n) { + 380 assert(n >= 0); + 381 // Use an 'acquire load' so that we observe a fully initialized + 382 // version of the returned Node. + 383 return ((&next_[0] - n)->Load()); + 384 } + 385 + 386 void SetNext(int n, Node* x) { + 387 assert(n >= 0); + 388 // Use a 'release store' so that anybody who reads through this + 389 // pointer observes a fully initialized version of the inserted node. + 390 (&next_[0] - n)->Store(x); + 391 } + 392 + 393 bool CASNext(int n, Node* expected, Node* x) { + 394 assert(n >= 0); + 395 return (&next_[0] - n)->CasStrong(expected, x); + 396 } + ... 397-416: no-barrier variants and InsertAfter ... + 417 private: + 418 // next_[0] is the lowest level link (level 0). Higher levels are + 419 // stored _earlier_, so level 1 is at next_[-1]. + 420 Atomic next_[1]; +``` - ┌──────────────────────────┬───────────────┬─────────────────┐ - │ tower: next_[h-1]…next_[1]│ Node: next_[0]│ key bytes inline│ - └──────────────────────────┴───────────────┴─────────────────┘ - ▲ Node* points HERE - levels accessed by NEGATIVE indexing: (&next_[0] - n) line 383 - key accessed as (&next_[1]): line 374 +Line 374 is the payoff: `&next_[1]` is the byte just past the struct, which is +where the key was written. No key pointer exists, so no key pointer can be +missed on. Line 383 is the other half: `&next_[0] - n` walks *backwards* into +the prefix. + +The allocation that sets this up does the arithmetic explicitly: + +``` +// memtable/inlineskiplist.h — AllocateNode, 858-880 + 858 template + 859 typename InlineSkipList::Node* + 860 InlineSkipList::AllocateNode(size_t key_size, int height) { + 861 auto prefix = sizeof(Atomic) * (height - 1); + 862 + 863 // prefix is space for the height - 1 pointers that we store before + 864 // the Node instance (next_[-(height - 1) .. -1]). Node starts at + 865 // raw + prefix, and holds the bottom-mode (level 0) skip list pointer + 866 // next_[0]. key_size is the bytes for the key, which comes just after + 867 // the Node. + 868 char* raw = allocator_->AllocateAligned(prefix + sizeof(Node) + key_size); + 869 Node* x = reinterpret_cast(raw + prefix); + ... 870-877: comment explaining why height need not be stored ... + 878 x->StashHeight(height); + 879 return x; + 880 } ``` -Why: the common case (level-0 traversal + key compare) touches `next_[0]` and -the key — **adjacent, same cache line(s)**. Taller levels (rare: ~1/4 of -nodes at branching factor 4) sit *before* the node, out of the hot path's -way. No separate key allocation, no pointer to the key. This is README §4's -dense-filter/inline-payload pattern again, by an author who priced the cache -lines. +Line 868 is one allocation for tower + node + key. Line 878 is a second trick +worth naming: the height is not a field. It is written *into* `next_[0]` +(`StashHeight`, lines 361-364) and read back by `Insert` (`UnstashHeight`, +lines 368-372) before that slot is used as a pointer — so a node carries no +height field at all, because a search that arrived at level *h* already knows +*h* is valid for this node (the comment at 870-877 says exactly this). + +**Work the saving.** On a 64-bit build `sizeof(void*)` = 8. Node height is +geometric with p = 1/4, so E[height] = 1/(1 − p) = 1/0.75 = **1.333 levels**, +i.e. 8 × 1.333 = **10.67 bytes** of forward pointers per node. The classic +`SkipList` adds one more word for the key pointer: 10.67 + 8 = +**18.67 bytes** of metadata per node. InlineSkipList drops the key pointer and +pays alignment padding of 0–7 bytes instead (header lines 15-17), so with +uniformly distributed key lengths the expected padding is 3.5 bytes and the +expected net saving is 8 − 3.5 = **4.5 bytes per node**. In a 64 MiB memtable +holding 100-byte entries — 67,108,864 / 100 = **671,089 entries** — that is +671,089 × 4.5 = 3,019,899 bytes = **2.88 MiB, or 4.5% of the memtable**. Which +is 4.5% more user data per flush, hence ~4.5% fewer flushes and ~4.5% less L0 +write amplification. The header's "always less than `SkipList`" +(lines 17-18) is the guaranteed version of that: the saving is 8 and the +worst-case padding is 7, so the difference never goes the wrong way. + +This is README §4's "dense filter / inline payload" pattern once more, by an +author who priced the cache lines: the common case — a level-0 step followed +by a key compare — touches `next_[0]` and the key bytes, which are adjacent. +The taller levels, needed by only 1/4 of nodes, sit *before* the node, out of +the hot path. ### Step 4 — the concurrency contract: publish with acquire/release -Lock-free readers and writers share the list through the forward pointers, -so every link is an atomic with ordering semantics: `Next()`/`SetNext()` use -**acquire/release** (lines 383, 390 — release on the writer side guarantees -everything written before the pointer store, i.e. the node's key bytes, is -visible to any reader that acquires the pointer), and `CASNext` (line 395) -does the compare-and-swap. This is the classic publish pattern (topic 9): -fully construct the node, *then* make it reachable with one release-store — -readers can never see a half-built node. +> **In:** readers walking the list with no lock while a writer is building and +> linking a node. +> **Out:** the exact memory ordering on each link operation, and why a +> *relaxed* store appears in the middle of a lock-free insert without breaking +> anything. + +Readers and writers share the list through the forward pointers, so every link +is an atomic with declared ordering. The header states the contract: + +``` +// memtable/inlineskiplist.h — thread safety and invariants, 20-38 + 20 // Thread safety ------------- + 21 // + 22 // Writes via Insert require external synchronization, most likely a mutex. + 23 // InsertConcurrently can be safely called concurrently with reads and + 24 // with other concurrent inserts. Reads require a guarantee that the + 25 // InlineSkipList will not be destroyed while the read is in progress. + 26 // Apart from that, reads progress without any internal locking or + 27 // synchronization. + 28 // + 29 // Invariants: + 30 // + 31 // (1) Allocated nodes are never deleted until the InlineSkipList is + 32 // destroyed. This is trivially guaranteed by the code since we never + 33 // delete any skip list nodes. + 34 // + 35 // (2) The contents of a Node except for the next/prev pointers are + 36 // immutable after the Node has been linked into the InlineSkipList. + 37 // Only Insert() modifies the list, and it is careful to initialize a + 38 // node and use release-stores to publish the nodes in one or more lists. +``` + +The orderings behind `Load`, `Store` and `CasStrong` are one file away: + +``` +// util/atomic.h — Atomic, 104-121 + 104 template + 105 class Atomic : public RelaxedAtomic { + 106 public: + 107 explicit Atomic(T initial = {}) : RelaxedAtomic(initial) {} + 108 void Store(T desired) { + 109 RelaxedAtomic::v_.store(desired, std::memory_order_release); + 110 } + 111 T Load() const { + 112 return RelaxedAtomic::v_.load(std::memory_order_acquire); + 113 } + ... 114-117: CasWeak, acq_rel ... + 118 bool CasStrong(T& expected, T desired) { + 119 return RelaxedAtomic::v_.compare_exchange_strong( + 120 expected, desired, std::memory_order_acq_rel); + 121 } +``` + +So `Next()` is an **acquire** load (line 112), `SetNext()` is a **release** +store (line 109), and `CASNext()` is an **acq_rel** compare-exchange (line +120). This is the classic publish pattern (topic 9): fully construct the node, +*then* make it reachable with one release operation. Release on the writer +side orders everything written before it — the node's key bytes — ahead of the +pointer store; acquire on the reader side means a reader that observes the new +pointer also observes those bytes. No reader can see a half-built node. + +`SeqCst` would add a global total order across *all* atomics, which nothing +here needs and which costs a full barrier on x86 stores and a `dmb ish` on +ARM. Acquire/release is exactly the pairing the invariant requires. + +And now the subtlety that the pseudocode version of this chapter used to get +wrong. Look at lines 404-407: `NoBarrier_SetNext` is a **relaxed** store. It +is used at line 1152 to set the *new node's* outgoing pointer just before the +CAS at 1153 publishes it. That is safe because the new node is not yet +reachable by any reader — there is nothing to order against — and the CAS at +1153 is itself `acq_rel`, so it orders the relaxed store ahead of the moment +the node becomes visible. The release is on the CAS, not on the node's own +pointer write. `InsertAfter` (lines 410-415) states the rule in a comment: +"NoBarrier_SetNext() suffices since we will add a barrier when we publish a +pointer to `this` in prev." ### Step 5 — the lock-free insert: level 0 is the only truth -`InsertConcurrently` (line 913, CAS loop lines 1135–1171) computes a -**splice** (the prev/next pair per level — same role as redis's `update[]`), -then links the node in level by level with CAS, retrying any level where a -concurrent insert changed the neighborhood: - -```rust -fn insert_concurrently(list: &SkipList, node: &Node, height: usize) { - let mut splice = list.find_splice(node.key()); // prev/next per level - for lvl in 0..height { // bottom-up: correctness - loop { // needs only level 0 - node.set_next(lvl, splice.next[lvl]); // prepare BEFORE publish - if splice.prev[lvl] - .cas_next(lvl, splice.next[lvl], node) // release: key bytes are - { // visible before the link - break; - } - splice.recompute(lvl, node.key()); // lost the race — re-find - } // neighbors, retry - } -} -// a node linked at level 0 but not yet above is merely slower to find — -// never incorrect. That asymmetry is what makes the lock-free version small. -``` - -The load-bearing asymmetry: level 0 contains *every* node, so a search that -reaches level 0 finds everything; upper levels are only shortcuts. A node -visible at level i but not i+1 is just slower to find — so partial linking -never breaks readers, and levels can be CAS'd independently, without any -multi-word atomicity. - -### Step 6 — no deletes: the restriction that keeps it ~200 lines - -The header comment (lines 31–33) states the contract: **no deletes, no -unlink**. General lock-free deletion is a research problem — an unlinked node -may still be held by a concurrent reader, so you need hazard pointers or -epochs to know when freeing is safe. InlineSkipList sidesteps all of it with -the Step 1 workload fact: memtables are frozen then dropped wholesale, so -nothing is ever freed while readers run. One workload restriction deletes an -entire class of machinery — constraint-driven simplicity, the design lesson -of the whole file. It's also why the redis skiplist's spans and backward -pointers are absent: both require multi-pointer updates no single CAS can do. - -### Step 7 — the supporting cast: arena and pluggable memtables - -Nodes come from a **concurrent arena** (`memory/concurrent_arena.h:57–68`) — -a bump allocator with per-core shards, so concurrent inserts don't contend on -malloc either (the allocator would otherwise be the next lock). Heights come -from `RandomHeight` (lines 559–573): branching factor 4, max 12 levels. And -the skiplist is just one implementation of the `MemTableRep` interface -(`memtable/skiplistrep.cc:17–397`); siblings in `memtable/`: -`hash_skiplist_rep` (hash → per-bucket skiplists, for point-heavy), -`hash_linklist_rep`, `vectorrep` (bulk-load: append then sort-on-flush). The -memtable is *pluggable* because RUM positions differ per workload — RocksDB -ships four answers. +> **In:** a fully built node with a stashed height, and a *splice* — the +> prev/next pair at every level, the same role as redis's `update[]`. +> **Out:** the bottom-up CAS loop, the reason a partially linked node is never +> incorrect, and the two duplicate checks that only run at level 0. + +`Insert` and `InsertConcurrently` are the same template body with a `UseCAS` +flag; the concurrent one differs only in putting the splice on the stack +instead of reusing the cached one: + +``` +// memtable/inlineskiplist.h — Insert entry points, 907-920 + 907 template + 908 bool InlineSkipList::Insert(const char* key) { + 909 return Insert(key, seq_splice_, false); + 910 } + 911 + 912 template + 913 bool InlineSkipList::InsertConcurrently(const char* key) { + 914 Node* prev[kMaxPossibleHeight]; + 915 Node* next[kMaxPossibleHeight]; + 916 Splice splice; + 917 splice.prev_ = prev; + 918 splice.next_ = next; + 919 return Insert(key, &splice, false); + 920 } +``` + +The single-threaded `Insert` reuses `seq_splice_` — a splice cached on the +list itself (line 842) — because a sequential writer can trust the splice it +computed last time. A concurrent writer cannot, so it gets a fresh stack one. +Note `kMaxPossibleHeight` = 32 (line 70) makes those stack arrays a fixed +512 bytes; that constant exists so this allocation-free path is legal. + +Here is the loop itself: + +``` +// memtable/inlineskiplist.h — Insert, CAS path, 1134-1172 + 1134 if (UseCAS) { + 1135 for (int i = 0; i < height; ++i) { + 1136 while (true) { + 1137 // Checking for duplicate keys on the level 0 is sufficient + 1138 if (UNLIKELY(i == 0 && splice->next_[i] != nullptr && + 1139 compare_(splice->next_[i]->Key(), key_decoded) <= 0)) { + 1140 // duplicate key + 1141 return false; + 1142 } + 1143 if (UNLIKELY(i == 0 && splice->prev_[i] != head_ && + 1144 compare_(splice->prev_[i]->Key(), key_decoded) >= 0)) { + 1145 // duplicate key + 1146 return false; + 1147 } + ... 1148-1151: two asserts that the splice still brackets the key ... + 1152 x->NoBarrier_SetNext(i, splice->next_[i]); + 1153 if (splice->prev_[i]->CASNext(i, splice->next_[i], x)) { + 1154 // success + 1155 break; + 1156 } + 1157 // CAS failed, we need to recompute prev and next. It is unlikely + 1158 // to be helpful to try to use a different level as we redo the + 1159 // search, because it should be unlikely that lots of nodes have + 1160 // been inserted between prev[i] and next[i]. No point in using + 1161 // next[i] as the after hint, because we know it is stale. + 1162 FindSpliceForLevel(key_decoded, splice->prev_[i], nullptr, i, + 1163 &splice->prev_[i], &splice->next_[i]); + 1164 + 1165 // Since we've narrowed the bracket for level i, we might have + 1166 // violated the Splice constraint between i and i-1. Make sure + 1167 // we recompute the whole thing next time. + 1168 if (i > 0) { + 1169 splice_is_valid = false; + 1170 } + 1171 } + 1172 } +``` + +Read line 1135 first: `i` counts **up**, so level 0 is linked before any +express lane. That direction is the correctness argument. Level 0 contains +*every* node, so a search that descends to level 0 finds everything; the upper +levels are only shortcuts. A node that is linked at level 0 but not yet at +level 3 is merely *slower to find* — never missing. That asymmetry is what +lets each level be CAS'd independently, with no multi-word atomicity anywhere. + +Line 1153 is the publish, and the retry at 1162-1163 is deliberately narrow: +on a lost race it re-finds the bracket for **level i only**, starting from the +`prev_[i]` it already has, rather than restarting the whole search. The +comment at 1157-1161 justifies it — the window between `prev[i]` and `next[i]` +is small, so a local re-scan almost always finds the new neighbour in a hop or +two. Lines 1168-1169 pay for that shortcut: a narrowed bracket at level i may +no longer nest inside level i−1's, so the cached splice is marked invalid for +next time rather than silently reused. + +Lines 1137-1147 are the last piece: duplicate detection runs **only at level +0**, guarded by `i == 0`, because level 0 is where every key lives. That is +the same fact used twice — once for correctness of partial linking, once to +avoid three redundant comparisons per insert. + +### Step 6 — no deletes: the restriction that removes an entire literature + +> **In:** invariant (1) from Step 4's header quote. +> **Out:** the class of machinery that invariant deletes, and the two redis +> features it costs. + +Invariant (1) at lines 31-33 states the contract: **allocated nodes are never +deleted until the list is destroyed**. General lock-free deletion is a +research problem, not an implementation detail: an unlinked node may still be +held by a concurrent reader mid-traversal, so freeing it safely needs hazard +pointers, epoch-based reclamation, or RCU — hundreds of lines and a +reclamation thread. InlineSkipList sidesteps all of it with the Step 1 +workload fact: memtables are insert-only until frozen, then dropped wholesale, +so nothing is ever freed while a reader runs. One workload restriction removes +an entire class of machinery. That is the design lesson of the file. + +It is also why the redis skiplist's **spans** and **backward pointers** are +absent here. Both require updating several pointers *as one atomic step*: a +span is a count that every node above the insertion point must increment +together with the link, and a backward pointer means the successor's `back` +field and the predecessor's `next` field must change together. No single CAS +covers two words, so both features are incompatible with this insert loop. +Redis pays a global lock (single-threaded) and buys `ZRANK` in O(log n); +RocksDB pays no lock and gives up rank queries it never needed. + +### Step 7 — the supporting cast: heights, arena, and pluggable memtables + +> **In:** the insert path from Step 5, which still needs a height and memory. +> **Out:** where the height comes from, why the allocator is not the next +> bottleneck, and the interface that makes the whole skiplist swappable. + +Heights come from a coin flip with no loop-carried allocation: + +``` +// memtable/inlineskiplist.h — RandomHeight, 558-573 + 558 template + 559 int InlineSkipList::RandomHeight() { + 560 auto rnd = Random::GetTLSInstance(); + 561 + 562 // Increase height with probability 1 in kBranching + 563 int height = 1; + 564 while (height < kMaxHeight_ && height < kMaxPossibleHeight && + 565 rnd->Next() < kScaledInverseBranching_) { + 566 height++; + 567 } + ... 568-571: sync point and asserts ... + 572 return height; + 573 } +``` + +`kScaledInverseBranching_` is `(Random::kMaxNext + 1) / kBranching_` (line +837), so the comparison at line 565 is a p = 1/`kBranching_` coin without a +division. The defaults are in the constructor signature: `max_height = 12`, +`branching_factor = 4` (lines 77-78), and `kMaxPossibleHeight = 32` (line 70) +is the compile-time cap that sizes the stack arrays in Step 5. + +**Work the numbers.** With p = 1/4, the expected search cost is +log₄(n) levels × (1 − p)/p forward hops per level. At n = 10⁶: +log₄(10⁶) = ln(10⁶)/ln(4) = 13.8155/1.3863 = **9.97 levels**, and +(1 − 0.25)/0.25 = **3.0 hops per level**, so 9.97 × 3.0 = **~30 dependent +pointer hops** per lookup. Now check the cap: `kMaxHeight_` = 12 means the +tallest express lane spans 4¹² = **16,777,216 entries**, and a 64 MiB memtable +of 100-byte entries holds 67,108,864 / 100 = **671,089** entries, needing +log₄(671,089) = **9.68 levels**. The default 12 is sized for the default +memtable with headroom, not for a general-purpose index — raise +`write_buffer_size` far enough and 12 stops being enough, which is why it is a +constructor parameter. + +Compare those ~30 hops with topic 0's measured `lookup_shootout` at n = 10⁶: +`hashmap 8.8 ns`. Thirty dependent accesses cannot happen in 8.8 ns — topic +0's ladder puts DRAM at ~100 ns and L1 at ~1 ns, so the skiplist only survives +because the top levels are tiny and stay cached, while the last few hops miss. +The hash table wins point lookups outright. What it cannot do is iterate in +order for the flush, or absorb eight concurrent writers without a latch. + +Nodes come from a **concurrent arena**, and its class comment is more precise +than "lock-free": + +``` +// memory/concurrent_arena.h — ConcurrentArena, 35-41 and 57-68 + 35 // ConcurrentArena wraps an Arena. It makes it thread safe using a fast + 36 // inlined spinlock, and adds small per-core allocation caches to avoid + 37 // contention for small allocations. To avoid any memory waste from the + 38 // per-core shards, they are kept small, they are lazily instantiated + 39 // only if ConcurrentArena actually notices concurrent use, and they + 40 // adjust their size so that there is no fragmentation waste when the + 41 // shard blocks are allocated from the underlying main arena. + ... + 57 char* AllocateAligned(size_t bytes, size_t huge_page_size = 0, + 58 Logger* logger = nullptr) override { + 59 size_t rounded_up = ((bytes - 1) | (sizeof(void*) - 1)) + 1; + ... 60-62: assert that rounding is correct and pointer-aligned ... + 63 return AllocateImpl(rounded_up, huge_page_size != 0 /*force_arena*/, + ... 64-67: lambda falling back to arena_.AllocateAligned ... + 68 } +``` + +It is a bump allocator behind a **spinlock**, with per-core shards that are +created lazily only once contention is observed (lines 38-39) — not a +lock-free allocator. That is enough: the common path takes a shard-local bump +and never reaches the spinlock, so `malloc` does not become the next +bottleneck behind the lock-free list. `MemTable` holds one directly +(`db/memtable.h:914`, `ConcurrentArena arena_;`), which is also why freeing +the memtable is a single arena teardown — the other half of Step 6's bargain. + +Finally, the skiplist is only *one* implementation of the `MemTableRep` +interface (`memtable/skiplistrep.cc:17`, `class SkipListRep : public +MemTableRep`, in a 425-line file). Its siblings ship in the same directory: +`hash_skiplist_rep.cc` (hash → per-bucket skiplists, for point-heavy +workloads), `hash_linklist_rep.cc`, and `vectorrep.cc` (bulk load: append, +sort on flush). The memtable is *pluggable* because the RUM position differs +per workload — RocksDB ships four answers and lets you pick. Note the +`allow_concurrent_memtable_write` comment from Step 1: only `SkipListFactory` +supports concurrent writes, so choosing a sibling silently costs you the +property this whole chapter is about. ## Where each step lives in the code -- **Step 3** — node layout: lines 358–421; negative tower indexing at line - 383, inline key at line 374; arena allocation at lines 860–869. -- **Step 4** — `Next()`/`SetNext()` acquire/release: lines 383, 390; - `CASNext`: line 395. -- **Step 5** — `InsertConcurrently`: line 913; the CAS loop: lines - 1135–1171. Read it against the Rust skeleton above and answer: which level - is linked first, and why does a *partially linked* node never break - readers? -- **Step 6** — the no-delete contract: header comment lines 31–33. -- **Step 7** — `RandomHeight`: lines 559–573; sharded arena: - `memory/concurrent_arena.h:57–68`; the `MemTableRep` plug-in point and its - three siblings: `memtable/skiplistrep.cc:17–397` and neighbors in - `memtable/`. +All in `memtable/inlineskiplist.h` unless another file is named. + +| Lines | What | Step | +|-------|------|------| +| `options.h:175-191` | `write_buffer_size = 64 << 20` | 1 | +| `options.h:1421-1429` | `allow_concurrent_memtable_write = true` | 1 | +| 10-18 | header: "saves 1 pointer per skip list node" | 3 | +| 20-27 | thread-safety contract | 4 | +| 31-33 | invariant (1): nodes are never deleted | 6 | +| 35-38 | invariant (2): release-stores publish nodes | 4 | +| 70 | `kMaxPossibleHeight = 32` | 5, 7 | +| 77-78 | ctor defaults `max_height = 12`, `branching_factor = 4` | 7 | +| 352-356 | Node layout comment — key after, tower before | 3 | +| 361-372 | `StashHeight` / `UnstashHeight` — height in `next_[0]` | 3 | +| 374 | `Key()` = `&next_[1]` | 3 | +| 379-396 | `Next` acquire / `SetNext` release / `CASNext` | 4 | +| 404-407, 410-415 | `NoBarrier_SetNext` and the comment justifying it | 4 | +| 417-420 | `Atomic next_[1]` and the negative-index comment | 3 | +| `util/atomic.h:104-121` | the actual orderings behind Load/Store/CasStrong | 4 | +| 558-573 | `RandomHeight` | 7 | +| 837 | `kScaledInverseBranching_` — the division-free coin | 7 | +| 853-856 | `AllocateKey` — the caller's entry point | 3 | +| 858-880 | `AllocateNode` — one allocation, `Node*` at `raw + prefix` | 3 | +| 907-920 | `Insert` vs `InsertConcurrently` | 5 | +| 1030-1044 | recover `Node*` from key, unstash height, grow `max_height_` | 5 | +| 1047-1131 | splice validation and `RecomputeSpliceLevels` | 5 | +| 1134-1172 | the CAS path — the heart of the file | 5 | +| 1173-1199 | the non-CAS path, for contrast | 5 | +| `concurrent_arena.h:35-41, 57-68` | spinlock + per-core shards | 7 | +| `db/memtable.h:914` | `ConcurrentArena arena_;` | 7 | +| `skiplistrep.cc:17` | `SkipListRep : public MemTableRep` | 7 | + +A route through it that builds rather than jumps: + +1. Read the header, lines 10-38, in one pass. It is a design document: the + layout rationale, the thread-safety contract, and both invariants. +2. `Node`, lines 352-420. Find line 374 and line 383 and satisfy yourself that + they address opposite sides of the same pointer. +3. `AllocateNode`, 858-880. Confirm line 868 allocates all three regions at + once and line 869 aims `Node*` at the middle. +4. `util/atomic.h:104-121`. Write down which ordering each of `Load`, `Store`, + `CasStrong` carries; you will need them in the next step. +5. The CAS loop, 1134-1172. Follow one insert of a height-3 node: which level + is linked first, what happens on a failed CAS at level 2, and why the + duplicate checks are guarded by `i == 0`. +6. **Aha:** line 1152 is a *relaxed* store immediately before the CAS at 1153 + that publishes the node. Once you see why that is not a bug — the node is + unreachable until 1153 succeeds, and the `acq_rel` CAS orders the relaxed + store ahead of publication — you have understood the file's memory model. + Every other ordering decision follows from the same rule. + +**Contrast case.** Read the non-CAS path at lines 1173-1199 straight after the +CAS path. Same loop, same bottom-up direction, but the link is a plain +`SetNext` at line 1197 with no retry and no `splice_is_valid` bookkeeping, +because external synchronisation guarantees the splice is still accurate. The +diff between the two branches is precisely the cost of lock-freedom in this +design: one retry loop, one narrow re-find, and one invalidation flag. That is +small — and it is small *because* of invariant (1). Compare with redis's +`zslInsert`, which needs no atomics at all because it holds the only thread. ## Questions to answer in notes.md -1. Redis's skiplist has spans + backward pointers; this one has neither. For each, - say exactly what breaks under concurrent CAS inserts. -2. Why acquire/release on the links rather than SeqCst? What reorder is actually - being prevented at line 383? (Reader must see the node's key bytes written - *before* the pointer that publishes it — classic publish pattern, topic 9.) -3. Estimate: at branching 4 and 1M entries, how many dependent misses per lookup, - and why does your hashbrown number from topic 0 beat it? Where does the skiplist - still win? (Sorted iteration for flush; concurrent writers.) +1. Redis's skiplist has spans and backward pointers; this one has neither. For + each, say exactly which line of the CAS loop (1134-1172) would have to + become a multi-word atomic, and why no CAS can provide it. +2. Why acquire/release rather than `SeqCst` on the links? Name the specific + reorder prevented at line 383, and say what `SeqCst` would add that nothing + here consumes. +3. Line 1152 is `NoBarrier_SetNext` — a relaxed store — inside a lock-free + insert. Explain why it is safe, then construct the variant that *would* be + a bug (hint: move the relaxed store after line 1153). +4. Redo Step 7's arithmetic for your own workload: at branching factor 4 and + your `write_buffer_size`, how many levels does log₄(entries) want, and how + much headroom does `kMaxHeight_ = 12` leave? At what memtable size does 12 + stop being enough? +5. Estimate the dependent misses per lookup at 10⁶ entries and compare against + this repo's measured `hashmap 8.8 ns` and `btreemap 26.6 ns` (topic 0 + `lookup_shootout`). Where does the skiplist still win, and what would you + have to measure to show it? + +## Takeaway + +Two ideas carry the file. The layout idea: put the tower *before* the node and +the key *after* it, so the hot pair — level-0 link and key bytes — share an +allocation and the cold tower is out of the way; the height does not even need +a field, because arriving at level *h* proves *h* is valid. The concurrency +idea: level 0 contains every node, so links can be made bottom-up with +independent single-word CASes, and a partially linked node is slow, never +wrong. Both are cheap only because the workload never deletes — one +restriction that removes hazard pointers, epochs, and the redis features +(spans, backward links) that need multi-word atomicity. When a lock-free +structure looks suspiciously small, look for the restriction paying for it. ## Done when -You can explain the negative-index tower AND why insert-only makes lock-free easy — -these two ideas are the file. +Answer each before unfolding it. + +- [ ] Given a `Node*` and a height of 3, name the byte offsets of `next_[0]`, + `next_[-2]`, and the first key byte, relative to the raw allocation. + +
+Answer + +From `AllocateNode` (858-869): `prefix = sizeof(Atomic) * (height - 1)` += 8 × 2 = 16 bytes, and `Node* x = raw + prefix`, so the `Node` starts at +offset 16. `next_[0]` is at offset **16** (it is the first and only declared +member, line 420). `next_[-2]` — level 2 — is 2 words *earlier*: offset +16 − 16 = **0**, the very start of the allocation. The key is `&next_[1]` +(line 374), one word past `next_[0]`: offset **24**. Level 1 sits at offset 8. + +
+ +- [ ] Why can duplicate keys be detected by checking level 0 alone (lines + 1137-1147), and what would break if the check ran at every level? + +
+Answer + +Level 0 is the only level that contains every node — upper levels are a random +subset — so if a duplicate exists anywhere, it is on level 0. Running the +check at every level would not be *incorrect*, just wasteful: three extra key +comparisons per height-4 insert, each a potential cache miss on +`splice->next_[i]->Key()`. Worse, an upper level could miss a duplicate that +level 0 would catch, so the check would still be needed at level 0 — the extra +work buys nothing. The comment at line 1137 says exactly this: "Checking for +duplicate keys on the level 0 is sufficient." + +
+ +- [ ] A reader is walking level 0 while a writer is midway through the loop at + line 1135, having linked its node at level 0 but not yet at level 2. + What does the reader observe, and is it correct? + +
+Answer + +The reader finds the new node — level 0 is linked, and the `acq_rel` CAS at +1153 paired with the acquire `Load` at line 383 guarantees the node's key +bytes are visible to it. A *different* reader descending from level 2 will +step past the new node at that level and land on it after descending to level +0 or 1. So the node is findable by every search, just via a slightly longer +path until the upper links land. Partial linking costs latency, never +correctness — that is the property that makes per-level CAS legal. + +
+ +- [ ] Invariant (1) says nodes are never deleted. Name the concrete machinery + that invariant removes, and the concrete feature it costs. + +
+Answer + +Removed: safe memory reclamation — hazard pointers, epoch-based reclamation, +or RCU, plus the reclamation thread and the per-read overhead of announcing a +hazard. Without deletes there is never an unlinked-but-still-referenced node, +so freeing is a single arena teardown (`db/memtable.h:914`) when the frozen +memtable is dropped. Cost: no in-place delete (a user `Delete` becomes a +tombstone *insert*, which the compaction layer must later resolve), and no +spans or backward pointers, since both need several words updated as one step. + +
+ +- [ ] The header claims the layout saves one pointer per node "despite the + padding". Show the inequality, and turn it into bytes for a 64 MiB + memtable of 100-byte entries. + +
+Answer + +Saving is exactly `sizeof(void*)` = 8 bytes (the eliminated key pointer); +padding is 0 to `sizeof(void*) - 1` = 0 to 7 bytes (header lines 15-17). Since +7 < 8, the net is strictly positive for every key length — that is the "always +less than `SkipList`" claim at lines 17-18. With uniform key +lengths the expected padding is 3.5 bytes, so the expected net saving is +8 − 3.5 = 4.5 bytes/node. A 64 MiB memtable of 100-byte entries holds +67,108,864 / 100 = 671,089 nodes, so the saving is 671,089 × 4.5 = 3,019,899 +bytes = 2.88 MiB, i.e. **4.5% more user data per flush**. + +
## References **Code** -- [rocksdb](https://github.com/facebook/rocksdb) - `memtable/inlineskiplist.h` — the header comment (lines 31–33) states - the no-delete contract; also `memory/concurrent_arena.h:57–68` - (sharded arena) and `memtable/skiplistrep.cc` (the `MemTableRep` - plug-in point and its three siblings) + +- [rocksdb](https://github.com/facebook/rocksdb) at `7c80a5a` — verify with + `tools/pinned-source.py ref rocksdb`. + +| File | Lines | What | +|------|-------|------| +| `memtable/inlineskiplist.h` | 10-38 | header: layout rationale, thread safety, both invariants | +| `memtable/inlineskiplist.h` | 352-420 | `Node` — negative tower index, inline key, stashed height | +| `memtable/inlineskiplist.h` | 558-573 | `RandomHeight` — p = 1/4 coin, capped at `kMaxHeight_` | +| `memtable/inlineskiplist.h` | 853-880 | `AllocateKey` / `AllocateNode` — one allocation, three regions | +| `memtable/inlineskiplist.h` | 907-920 | `Insert` vs `InsertConcurrently` | +| `memtable/inlineskiplist.h` | 1134-1172 | the CAS path — bottom-up, level-0 duplicate check, narrow retry | +| `memtable/inlineskiplist.h` | 1173-1199 | the non-CAS path — the contrast case | +| `util/atomic.h` | 104-121 | release `Store`, acquire `Load`, `acq_rel` `CasStrong` | +| `memory/concurrent_arena.h` | 35-41, 57-68 | spinlock-guarded bump arena with lazy per-core shards | +| `db/memtable.h` | 914 | `ConcurrentArena arena_;` — the memtable owns one | +| `memtable/skiplistrep.cc` | 17 | `SkipListRep : public MemTableRep` — the plug-in point | +| `include/rocksdb/options.h` | 175-191, 1421-1429 | 64 MiB memtable; concurrent writes on by default | + +Siblings in `memtable/`: `hash_skiplist_rep.cc`, `hash_linklist_rep.cc`, +`vectorrep.cc` — three other RUM positions for the same interface. + +**Measured in this repo** + +- `topics/00-performance-toolbox/notes.md`, `lookup_shootout` at n = 10⁶: + `hashmap 8.8 ns`, `btreemap 26.6 ns`, `vec_binary_search 25.8 ns` per + lookup — the ordered/unordered gap this design pays for concurrency. +- `topics/00-performance-toolbox/notes.md`, cache ladder: ~1 ns L1 / ~5 ns L2 + / ~100 ns DRAM — the prices behind "~30 dependent hops". +- `topics/02-in-memory-structures/notes.md` (FINDINGS row 2): the + `rehash_spike` lane, `p50 = 42 ns` against `max = 58.4 ms`. A skiplist has + no rehash, so it has no equivalent tail — worth remembering when the median + says the hash table wins. + +**Companion chapters** + +- [`reading-redis-skiplist.md`](reading-redis-skiplist.md) — the same + structure single-threaded, with spans and backward pointers. Read the two + side by side: every feature redis has and RocksDB lacks is a multi-word + update. +- [`reading-hashbrown.md`](reading-hashbrown.md) — the unordered alternative + ruled out in Step 2, and why it wins point lookups. diff --git a/topics/02-in-memory-structures/reading-swisstable-talk.md b/topics/02-in-memory-structures/reading-swisstable-talk.md index 31408a4..b44183b 100644 --- a/topics/02-in-memory-structures/reading-swisstable-talk.md +++ b/topics/02-in-memory-structures/reading-swisstable-talk.md @@ -1,154 +1,560 @@ # The SwissTable design walk: how benchmarks kill hash tables How Google replaced `std::unordered_map` fleet-wide — told as a sequence of -designs, each rejected by a measurement. This chapter is a watching guide for -Kulukundis's CppCon talk: watch it *after* reading -[`reading-hashbrown.md`](reading-hashbrown.md), because the talk is the design -narrative for the code you just read. Here the narrative is rebuilt step by -step — each design, the benchmark that killed it, and the idea that replaced -it — so you can watch for the beats instead of chasing them. Budget ~60 min -video + 30 min notes. +designs, each one killed by a measurement. Watch Kulukundis's CppCon 2017 talk +*after* [`reading-hashbrown.md`](reading-hashbrown.md), because the talk is the +design narrative for the code you just read. This chapter rebuilds that +narrative step by step — each design, the number that killed it, the idea that +replaced it — so you watch for the beats instead of chasing them. Budget ~60 +min video + 30 min notes. + +**On sourcing.** A talk is not a citable artifact the way a file at a commit +is, so every claim below is grounded in something you can re-check: the +[abseil Swiss Tables design notes](https://abseil.io/about/design/swisstables) +for the design as Google documented it; Google's own +[sparsehash](https://github.com/sparsehash/sparsehash) at `1dffea3d9` for what +`dense_hash_map` really did; hashbrown at `d69025b` for the final shape; and +cppreference for the C++ requirements. Two consequences. First, the +fleet-wide RAM- and CPU-percentage figures this talk is famous for are **not +reproduced here** — they are spoken numbers with no retrievable primary +source, and this repo does not print numbers it has not checked. Note them +yourself as you watch, with the timestamp. Second, where the talk's account +and the code disagree, the code wins; two such places are flagged below. ## The problem in one sentence -`std::unordered_map` costs 2+ dependent cache misses and a malloc'd node per -entry, across a fleet where hash tables hold ~1% of *all* RAM and serve ~4% -of all CPU cycles — and the C++ standard's own API rules forbid fixing it in -place. +`std::unordered_map` costs 2+ dependent cache misses and one malloc'd node per +entry — topic 0's `cache_ladder` priced a dependent DRAM miss at ~100 ns +([FINDINGS.md](../../FINDINGS.md) row 0) — and the three C++ requirements that +force that layout are in the standard, so it cannot be fixed in place. ## The concepts, step by step The design walk, in one picture — each arrow is a benchmark verdict: ``` -std::unordered_map chaining, per-node malloc, iterator stability +std::unordered_map chaining, per-node malloc, reference stability │ "every lookup = 2+ dependent misses" ▼ -dense_hash_map open addressing, quadratic probe, but 2 sentinel - │ keys stolen from the user + 50% max load +dense_hash_map open addressing, quadratic probe — but 2 sentinel + │ keys taken from the user, and a 50% occupancy cap + │ "half the array is slack, and the API steals two key values" ▼ -"store metadata per slot" 1 byte: empty/deleted/full + 7 hash bits - │ "but scanning bytes one at a time is slow" +"store metadata per slot" 1 byte: empty/deleted/full + 7 hash bits (H2) + │ "but scanning bytes one at a time is a branch per byte" ▼ -SwissTable group the bytes, compare 16 at once with SSE2 - → 87.5% load factor, ~1 miss per lookup +SwissTable group the bytes, compare a whole group at once + → 87.5% occupancy, ~1 group load per lookup +``` + +### Step 1 — the incumbent: chaining, mandated by an API contract + +> **In:** nothing yet — this step establishes what is being replaced and why +> the replacement could not be done in place. +> **Out:** three named requirements from the C++ standard and the cost they +> impose. Step 2 is the first attempt to escape them. + +`std::unordered_map` uses **chaining** (buckets holding malloc'd linked-list +nodes — the family the [redis dict chapter](reading-redis-dict.md) covers) not +because its authors preferred it, but because three published requirements +leave almost nothing else. Naming them precisely matters, because "the +standard mandates chaining" is a summary, not a quotation: + +1. **Reference and pointer stability.** "References and pointers to either key + or data stored in the container are only invalidated by erasing that + element, even when the corresponding iterator is invalidated" + (cppreference, `std::unordered_map` → Iterator invalidation → Notes). A + rehash may invalidate iterators but *not* references — so entries can never + move, so they cannot live inline in an array that gets reallocated. +2. **The bucket interface.** `bucket_count()`, `bucket_size(n)`, `bucket(key)` + and `begin(n)`/`end(n)` returning a `local_iterator`, which cppreference + defines as an iterator that "can be used to iterate through a single bucket + but not across buckets". Chains are part of the public API. +3. **Node handles** (C++17): `extract()` returns a `node_type`, a handle that + owns the element's *node* and can be re-inserted into another container + without copying. That only means anything if a per-element node exists. + +The measured cost: a lookup dereferences the bucket array, then a node, then +possibly the next node — each dependent on the last, so the out-of-order +window cannot overlap them — plus a malloc per insert. Lesson zero of the +talk, and the one that generalises past hash tables: **API guarantees are +performance decisions**, made once and un-unmakeable. + +### Step 2 — first replacement: `dense_hash_map` and its two warts + +> **In:** Step 1's diagnosis — the pointer chase has to go. +> **Out:** open addressing working, with two specific costs (sentinel keys, a +> 50% occupancy cap) and the published probe table that explains the cap. +> Steps 3 and 4 remove one wart each. + +Google's earlier answer, `dense_hash_map`, dropped chaining for **open +addressing**: entries live inline in one flat array, collisions are resolved +by probing. Lookups fell to about one miss. Its probe rule is quadratic, and +it is one macro: + +```c +// sparsehash@1dffea3d9 — src/sparsehash/internal/densehashtable.h:115-119 + 115 // The probing method + 116 // Linear probing + 117 // #define JUMP_(key, num_probes) ( 1 ) + 118 // Quadratic probing + 119 #define JUMP_(key, num_probes) ( num_probes ) +``` + +Line 119 makes the k-th step jump k slots, so the offsets from the home slot +are 1, 3, 6, 10, … — the triangular numbers, the same sequence hashbrown walks +at `raw.rs:90`, except that hashbrown counts in *groups* and this counts in +slots. The loop it drives is at densehashtable.h:648-653, and it stops at the +first empty slot. + +The first wart is in the API. The table has no metadata array, so "empty" and +"deleted" have to be encoded *in key space*: the user must donate two key +values that can never appear in real data. + +```c +// sparsehash@1dffea3d9 — densehashtable.h:390, 496 (the two donations) + 390 void set_deleted_key(const key_type &key) { + // ... 391-395: assert it differs from the empty key ... + // ... 496: void set_empty_key(const_reference val) { + 497 // Once you set the empty key, you can't change it + 498 assert(!settings.use_empty() && "Calling set_empty_key multiple times"); +``` + +Forget to call them and the table asserts; choose a value that later shows up +in your data and it silently disappears. An API landmine, and one no standard +container could ever ship. + +The second wart is memory, and the source states the trade-off outright: + +```c +// sparsehash@1dffea3d9 — densehashtable.h:1309-1316 + 1309 // How full we let the table get before we resize. Knuth says .8 is + 1310 // good -- higher causes us to probe too much, though saves memory. + 1311 // However, we go with .5, getting better performance at the cost of + 1312 // more space (a trade-off densehashtable explicitly chooses to make). + // ... 1313-1315: "feel free to play around", then the template header ... + 1316 const int dense_hashtable::HT_OCCUPANCY_PCT = 50; +``` + +Half the array is slack. Why 50 and not Knuth's 80? The file publishes its own +answer — a probe table sitting in the header comment: + +```c +// sparsehash@1dffea3d9 — densehashtable.h:77-84 (the file's own numbers) + 77 // NUMBER OF PROBES / LOOKUP Successful Unsuccessful + 78 // Quadratic collision resolution 1 - ln(1-L) - L/2 1/(1-L) - L - ln(1-L) + // ... 79-80: the same for linear probing ... + 81 // -- enlarge_factor -- 0.10 0.50 0.60 0.75 0.80 0.90 0.99 + 82 // QUADRATIC COLLISION RES. + 83 // probes/successful lookup 1.05 1.44 1.62 2.01 2.21 2.85 5.11 + 84 // probes/unsuccessful lookup 1.11 2.19 2.82 4.64 5.81 11.4 103.6 +``` + +Read line 84 across: an unsuccessful lookup costs 2.19 slot probes at L = 0.50 +and 5.81 at L = 0.80. **2.65× the probes to save 37.5% of the slots** — and +each probe touches a slot, so each is a potential cache miss. Given that +exchange rate, 50% is the right answer. Hold on to the exchange rate; Step 5 +changes it. + +### Step 3 — the metadata byte: state out of key space, 7 hash bits for free + +> **In:** Step 2's two warts — sentinel keys and the occupancy cap. +> **Out:** a dense one-byte-per-slot array that fixes the first wart outright +> and sets up the fix for the second. Step 4 makes reading it cheap. + +The idea that removes the sentinels: stop encoding state in key space and keep +**one metadata byte per slot** in a separate dense array. Google's design +notes describe the split of the 64-bit hash: + +> H1, a 57 bit hash value, used to identify the element index within the table +> itself … H2, the remaining 7 bits of the hash value, used to store metadata +> for this element. … Each metadata entry consists of one byte, which consists +> of a single control bit and the 7 bit H2 hash. +> +> — [abseil, *Swiss Tables Design Notes*](https://abseil.io/about/design/swisstables) + +Two wins at once. No key value is stolen: "empty" and "deleted" are states of +the metadata byte, not of the key. And the 7 H2 bits are a free per-slot +**pre-filter** — a probe compares one byte instead of touching the slot's key, +and is wrong only when 7 bits collide, probability 2⁻⁷ = 1/128 per occupied +slot. This is hashbrown's control byte at `src/control/tag.rs:9-49`, here at +the moment of invention. + +**Where the code diverges from the design note.** hashbrown's H2 is the same — +`Tag::full` takes the top 7 bits (`tag.rs:47`) and masks the eighth +(`tag.rs:48`) — but its H1 is not 57 bits: + +```rust +// hashbrown@d69025b — src/raw.rs:58-64 + 58 /// Primary hash function, used to select the initial bucket to probe from. + // ... 59-60: #[inline] and a clippy allow ... + 61 fn h1(hash: u64) -> usize { + 62 // On 32-bit platforms we simply ignore the higher hash bits. + 63 hash as usize + 64 } ``` -### Step 1 — the incumbent: chaining mandated by an API contract - -`std::unordered_map` uses chaining (buckets of malloc'd linked-list nodes — -the redis dict chapter's family) not because its authors loved it, but -because the C++ standard's API promises force it: **pointer stability** -(references to elements must survive any rehash — impossible if entries live -inline and move) and a **bucket interface** (`bucket_count()`, -`begin(bucket)`, exposing chains as an API). The measured cost: every lookup -is 2+ dependent cache misses (bucket array, then node, then possibly next -node) plus a malloc per insert. Lesson zero of the talk: API guarantees are -performance decisions. - -### Step 2 — first replacement: dense_hash_map and its warts - -Google's earlier answer, `dense_hash_map`, switched to open addressing -(entries inline in one flat array, collisions resolved by probing — see the -hashbrown chapter, Step 1) with quadratic probing. Lookups dropped to ~1 -miss. The measured warts: the user must *donate two sentinel key values* -(one meaning "empty slot", one "deleted slot" — so those keys become -unusable, an API landmine), and it needs a **50% maximum load factor** — -half the table is empty slack, 2× the memory of the entries themselves. -Fast, but RAM-hungry and awkward. The fleet pays for RAM too. - -### Step 3 — the metadata byte: state out of band, 7 hash bits for free - -The fix for both warts: stop encoding empty/deleted *in key space* and store -**one metadata byte per slot** in a separate dense array — 1 bit of state -(empty/deleted/full) plus **7 bits of the hash** (h2). No sentinel keys -stolen from the user; and the 7 hash bits act as a per-slot pre-filter, so a -probe compares 1 byte instead of touching the slot's key: a false positive -only 1/128 of the time. This is hashbrown's control byte (`tag.rs:9–49`), -here at the moment of invention. - -### Step 4 — group probing: scan 16 metadata bytes in one instruction - -The next benchmark verdict: scanning metadata bytes one at a time is still a -loop with a branch per byte. Because the metadata is a dense byte array, -SIMD (16-byte-at-once CPU instructions) can compare a whole **group** of 16 -tags against h2 in one SSE2 compare + one `_mm_movemask_epi8` (turn the -16-lane comparison into a 16-bit integer bitmask — then iterate its set -bits). One instruction filters 16 slots; probing moves group by group. This -is hashbrown's `Group::match_tag` — NEON, 8-wide, on your machine. - -### Step 5 — what the combination buys: 87.5% load and tombstone rules - -With group probing, a probe step examines 16 slots nearly free, so the table -stays fast even when almost full: **load factor rises from 50% to 7/8 = -87.5%** — a fleet-wide RAM cut on its own, on top of removing per-node -mallocs. Deletion uses the DELETED metadata state (a **tombstone**: probes -must skip it, since stopping there would hide keys probed past it; inserts -may reuse it) — the talk's discussion is where hashbrown's -rehash-in-place-when-full-of-tombstones policy (raw.rs:152, 1033) comes -from. The end state: ~1 cache miss per lookup, 1 metadata byte per slot of -overhead, no sentinels — and it still couldn't ship as `unordered_map`, -because Step 1's API contract survives any benchmark. +It is the whole hash truncated to `usize` and then masked by `bucket_mask` +(raw.rs:2453), so the index bits and the tag bits *overlap* on a 64-bit +target — harmless, because the index uses low bits and the tag uses high ones, +but it means "H1 is 57 bits" describes abseil, not hashbrown. Also note that +hashbrown has no identifier called `h2` at all; the local is `tag_hash` +(raw.rs:2010). When you hear "H2" in the talk, translate to `Tag::full`. + +### Step 4 — group probing: one instruction filters a whole group + +> **In:** the dense metadata array from Step 3. +> **Out:** the probe loop as it ships, and the parameter — group width — that +> every remaining number depends on. Step 5 spends what this buys. + +The next verdict: scanning metadata bytes one at a time is still a loop with a +branch per byte. But the bytes are dense, so **SIMD** can compare a whole +**group** of adjacent tags against H2 in one instruction. Abseil's design +notes give both the algorithm and the code: + +``` +1. Use the H1 hash to find the start of the "bucket chain" for that hash. +2. Use the H2 hash to construct a mask. +3. Use SSE instructions and the mask to produce a set of candidate matches. +4. Perform an equality check on each candidate. +5. If no element is found amongst the current candidates, perform probing to + generate a new set of candidates. Note that a deleted element does not + cease probing, though an empty element would. + +MaskMatch(h2_t hash) const { + auto match = _mm_set1_epi8(hash); + return Mask(_mm_movemask_epi8(_mm_cmpeq_epi8(match, metadata))); +} + — abseil, Swiss Tables Design Notes +``` + +`_mm_cmpeq_epi8` compares 16 bytes lane-wise; `_mm_movemask_epi8` collapses +the result to a 16-bit integer whose set bits are the candidate lanes. Step 5 +of that list is the tombstone rule, and it is exactly hashbrown's +`match_empty` stopping condition at `raw.rs:2040`. + +**Sixteen is not a law.** This is the second place the talk and the code +diverge, and it is the one most retellings get wrong. In hashbrown the group +is whatever the target provides: + +| Backend | Type | `Group::WIDTH` | Selected when | +|---|---|---|---| +| SSE2 | `__m128i` (`sse2.rs:20`) | **16** | x86/x86-64 with `sse2` (`mod.rs:17-21`) | +| NEON | `uint8x8_t` (`neon.rs:16`) | **8** | little-endian aarch64 with `neon` (`mod.rs:24-31`) | +| LSX | `m128i` (`lsx.rs:17`) | 16 | nightly + loongarch64 + `lsx` (`mod.rs:34-39`) | +| generic | `u64` (`generic.rs:41`) | **8** on 64-bit | everything else (`mod.rs:42-44`) | + +This repo measures on an Apple M3 Pro, so `Group::WIDTH = 8` here and +`match_tag` is `vceq_u8` plus a reinterpret (`neon.rs:68-73`) rather than +`_mm_movemask_epi8` (`sse2.rs:73-86`). A previous version of this chapter +pointed at `neon.rs:78-90` for the group compare; at `d69025b` that range is +`match_empty`, and `match_tag` is 68-73. + +The width also sets the false-positive rate, because a group holds +`WIDTH × load` occupied lanes and each collides with probability 1/128: + +``` +WIDTH = 8 (NEON / generic): 8 × 7/8 = 7 lanes; 7 / 128 = 0.0547 → 5.5% +WIDTH = 16 (SSE2): 16 × 7/8 = 14 lanes; 14 / 128 = 0.109 → 10.9% +``` + +So the wider group filters more slots per instruction *and* wastes more key +comparisons. Neither number is "the" SwissTable false-positive rate. + +### Step 5 — what the combination buys: the exchange rate flips + +> **In:** Step 2's published probe table and Step 4's group width. +> **Out:** the 87.5% load factor justified by division rather than assertion, +> plus the deletion rules that come with it. Step 6 generalises the method. + +Step 2's exchange rate — more occupancy costs proportionally more probes — was +computed in the currency of *slot probes*. Group probing changes the currency +to *group loads*, and the same arithmetic comes out the other way. Using +densehashtable.h:78's own formula for an unsuccessful lookup, +`1/(1−L) − L − ln(1−L)`, evaluated at SwissTable's load factor: + +``` +L = 0.500 → 1/0.500 − 0.500 − ln(0.500) = 2.000 − 0.500 + 0.693 = 2.193 +L = 0.875 → 1/0.125 − 0.875 − ln(0.125) = 8.000 − 0.875 + 2.079 = 9.204 + + (L = 0.500 reproduces the 2.19 printed at densehashtable.h:84, which is how + we know the formula is being read correctly; 0.875 is not in their table.) + + slot probes: 9.204 / 2.193 = 4.20× more at 87.5% than at 50% + group loads: 9.204 / 8 = 1.15 (WIDTH = 8, this machine) + 9.204 / 16 = 0.58 (WIDTH = 16, SSE2) +``` + +Contiguous slots share a group, so ~9.2 slot probes become **1.15** group +loads at width 8 — barely more than one cache line — or 0.58 at width 16. +(That division is an estimate: the formula assumes each probe lands +independently, whereas a group is a contiguous window. It is the right order +of magnitude, and the direction is not in doubt.) Meanwhile the slot array +shrinks: + +``` +1,000,000 entries at L = 0.500 → 1,000,000 / 0.500 = 2,000,000 slots +1,000,000 entries at L = 0.875 → 1,000,000 / 0.875 = 1,142,857 slots + 2,000,000 / 1,142,857 = 1.75× fewer +``` + +1.75× fewer slots, plus no per-node malloc, for about one cache line per +lookup. hashbrown encodes the 7/8 in `bucket_mask_to_capacity` +(`raw.rs:182-191`, with a separate case for tables of 8 buckets or fewer — +the earlier version of this chapter cited `raw.rs:152`, which is a different +function at this commit). + +Deletion is the bill that comes with open addressing. Because the probe stops +at the first *empty* slot, erasing a slot mid-chain would hide every key +probed past it — hence a **tombstone**, the `DELETED` state, which probes skip +and inserts may reuse. Abseil's list says it in one line: "a deleted element +does not cease probing, though an empty element would." hashbrown then adds +two refinements the talk predates: + +- It writes a tombstone only when it must — `erase` checks whether an `EMPTY` + is already within a group's reach on either side, and if so writes `EMPTY` + and returns the capacity instead (`raw.rs:3279-3284`). +- When tombstones do choke a table, `reserve_rehash_inner` rewrites it in + place rather than growing, but only if the live items would fit in *half* + the current capacity (`raw.rs:2756-2757`); otherwise it really grows + (`raw.rs:2785-2787`). + +Same disease as LSM tombstones from topic 1, same cure: compaction, gated by +a rule about when it pays. + +And after all of it, this could still not ship as `std::unordered_map`, +because Step 1's three requirements survive any benchmark. `absl::flat_hash_map` +is a different type with a different contract — which is the point. ### Step 6 — the method is the takeaway -Every arrow in the design walk is a *measurement*, not an opinion: -hypothesize → benchmark → let the number kill or keep the design — the -topic-0 method applied to data-structure design at fleet scale. Watch the +> **In:** all five verdicts above. +> **Out:** the transferable procedure, and the honest limit of what a talk can +> establish. + +Every arrow in the design walk is a *measurement*, not a preference: +hypothesise → benchmark → let the number keep or kill the design. That is +topic 0's method applied to data-structure design at fleet scale. Watch the talk as a methodology demonstration wearing a hash table as a costume. -## How to watch the talk (with the concepts in hand) +The limit is worth naming, since this chapter is about believing numbers. The +best-known figures from this talk — what fraction of a fleet's RAM and CPU +goes to hash tables — are spoken claims with no retrievable primary source, +which is why they appear nowhere above. Everything else here survived being +checked: the C++ requirements against cppreference, the 50% cap and its +probe table against sparsehash's own header, H1/H2 against Google's design +notes, and every width and line number against hashbrown at `d69025b`. Write +the spoken numbers into notes.md with a timestamp, mark them unverified, and +treat that as the exercise. + +## How to read the talk -Timestamps are approximate across uploads — navigate by slide titles: +Timestamps vary across uploads and this chapter does not assert any, so +navigate by slide content. Each beat below maps to a step above and to +something you can open: -- **"The C++ standard basically mandates chaining"** — Step 1: why - `unordered_map` can't be fixed in place (pointer stability + bucket API - promises). -- **The metadata byte slide** — Step 3: the h2/control-byte idea introduced. -- **The SSE2 `_mm_movemask_epi8` slide** — Step 4: the group probe; this is - hashbrown's `Group::match_tag`, NEON on your machine. -- **Load factor + tombstone discussion** — Step 5: where the 7/8 and - rehash-in-place decisions come from (hashbrown raw.rs:152, 1033). +| Watch for | Step | Open alongside | +|---|---|---| +| "the standard basically mandates chaining" | 1 | cppreference `unordered_map` → Iterator invalidation, and `local_iterator` | +| `dense_hash_map`, `set_empty_key`, 50% load | 2 | `densehashtable.h:390`, `:496`, `:1309-1316`, and the probe table at `:77-84` | +| the metadata byte slide (1 control bit + 7 hash bits) | 3 | `tag.rs:9-49`; the abseil design notes' H1/H2 diagram | +| the `_mm_movemask_epi8` slide | 4 | `sse2.rs:73-86` beside `neon.rs:68-73` | +| load factor and tombstone discussion | 5 | `raw.rs:182-191`, `raw.rs:2756-2757`, `raw.rs:3279` | +| any fleet-wide percentage | 6 | your notes — record it with a timestamp, marked unverified | -Connect each talk moment to the code you already read: +Suggested route: read the [abseil design notes](https://abseil.io/about/design/swisstables) +first (ten minutes, and it is the written form of Steps 3-4), then watch, then +re-open [`reading-hashbrown.md`](reading-hashbrown.md) Step 3 and check the +talk's account against `find_inner` at `raw.rs:2009-2046`. The talk is +`std::unordered_map` → `dense_hash_map` → metadata → SIMD; the code is that +same walk with eight more years of tombstone bookkeeping bolted on. -| Talk moment | You saw it in | -|---|---| -| metadata byte = 1 bit state + 7 bits hash | `tag.rs:9–49` | -| group probe, movemask | `group/neon.rs:78–90` (ARM twist: 8-wide) | -| "deleted vs empty" probe-stop rule | `raw.rs` tombstone logic :1033–1043 | -| iterators break on rehash — API cost | Rust never promised stability, so hashbrown got this for free | +**Contrast case.** Watch how differently redis solves the same growth problem. +SwissTable's answer to "the table is full" is to stop the world and rebuild +(`raw.rs:2785`), which this repo measured as a **58.4 ms** worst-case insert +([FINDINGS.md](../../FINDINGS.md) row 2). Redis instead keeps two tables and +migrates one bucket per operation (`dict.c:405-434`), trading a permanently +slower lookup for the absence of that spike — see +[`reading-redis-dict.md`](reading-redis-dict.md). Neither is wrong; they are +answers to different questions about the p99. ## Questions to answer in notes.md -1. Google couldn't ship this as `std::unordered_map` because the standard's API - promises (pointer stability, bucket interface) mandate chaining. Which redis - `dict` features would SwissTable similarly break? (Incremental rehash needs - stable *entries*? Check — redis moves entries between tables anyway; the real - conflict is `dictScan`'s bucket cursor.) -2. The talk reports big fleet-wide RAM savings from the load-factor jump - (50% → 87.5%) plus removing per-node mallocs. Estimate the bytes-per-entry - difference for a u64→u64 map: chaining with malloc'd nodes vs SwissTable at - 7/8 load. Show the arithmetic in notes. -3. Kulukundis says hash quality matters *more* for open addressing than - chaining — why? (Clustering compounds; a bad h2 also raises false positives.) +1. Google could not ship this as `std::unordered_map` because of the three + requirements in Step 1. Which redis `dict` features would SwissTable + similarly break? (Incremental rehash needs stable *entries*? Check — redis + moves entries between tables anyway, `dict.c:336-377`; the real conflict is + `dictScan`'s bucket cursor, `dict.c:1424-1445`.) +2. Estimate bytes per entry for a u64→u64 map: chaining with malloc'd nodes + versus SwissTable at 7/8. Show the arithmetic — bucket array, node size, + allocator rounding, control bytes, empty-slot slack. Then compare against + the 1.75× *slot* ratio derived in Step 5 and explain why the byte ratio is + larger. +3. Kulukundis argues hash quality matters *more* for open addressing than for + chaining. Give two distinct mechanisms, using the steps above: one about + Step 2's probe sequence, one about Step 3's 7-bit tag. +4. Take the exchange rate from Step 2 (2.19 → 5.81 probes for 0.50 → 0.80) and + redo it in group loads at `WIDTH = 8` and `WIDTH = 16`. At which width does + Knuth's 0.8 stop looking expensive, and what does that say about who the + 1998 advice was written for? + +## Takeaway + +The design walk's real lesson is not "use SIMD". It is that a table's maximum +load factor is not a constant of nature but an exchange rate between two +costs — and that changing the *unit* the probe is billed in (slots → groups) +re-prices every design decision downstream of it. Also: check which unit, and +which group width, any quoted SwissTable number was computed in. ## Done when -You can retell the rejected-design sequence (chaining → dense_hash_map → -metadata bytes → SIMD groups) and give the one-line benchmark reason each step -was taken. +Answer each before unfolding it. + +- [ ] You can retell the rejected-design sequence and give the measured reason for each step. + +
Answer + + `std::unordered_map` → chaining, 2+ *dependent* cache misses per lookup + (~100 ns each on this machine, [FINDINGS.md](../../FINDINGS.md) row 0) plus a + malloc per insert; unfixable because of the three requirements in Step 1. + + → `dense_hash_map`: open addressing with quadratic probing + (`densehashtable.h:119`) cuts it to about one miss, but costs two donated + sentinel keys (`:390`, `:496`) and caps occupancy at 50% + (`HT_OCCUPANCY_PCT = 50`, `:1316`), because at 80% an unsuccessful lookup + costs 5.81 slot probes against 2.19 at 50% (`:84`). + + → metadata byte: one byte per slot holding a control bit and 7 hash bits + (abseil design notes; `tag.rs:9-49`) — the sentinels are gone and a probe + can reject a slot without touching it, wrong only 1/128 of the time. + + → SwissTable: compare a whole group of those bytes in one instruction + (`_mm_cmpeq_epi8` + `_mm_movemask_epi8`, or `vceq_u8` on NEON), which + re-denominates probe cost in group loads and makes 87.5% occupancy cheaper + than 50% used to be. + +
+ +- [ ] You can name the three C++ requirements that made `unordered_map` unfixable, not just say "the standard mandates chaining". + +
Answer + + (1) **Reference and pointer stability**: cppreference states that references + and pointers are invalidated only by erasing that element, even when the + iterator is invalidated — so a rehash may not move elements, which rules out + storing them inline in a reallocated array. (2) **The bucket interface**: + `bucket_count()`, `bucket(key)`, `bucket_size(n)` and `begin(n)` returning a + `local_iterator` that iterates "a single bucket but not across buckets" — + chains are public API. (3) **Node handles** since C++17: `extract()` returns + a `node_type` that owns a per-element node and can be re-inserted elsewhere + without copying, which presupposes that per-element nodes exist. + + None of the three says "use chaining". Together they leave essentially + nothing else, which is the more interesting version of the claim. + +
+ +- [ ] You can say what the metadata byte fixed, and what it did *not* fix on its own. + +
Answer + + It fixed the API wart completely: "empty" and "deleted" become states of a + separate byte rather than reserved key values, so `set_empty_key` / + `set_deleted_key` (`densehashtable.h:390`, `:496`) disappear and no key + value is unusable. It also bought a free 7-bit pre-filter, since the byte + has room for H2 alongside the state bit. + + It did not, by itself, fix the occupancy cap. Reading one byte per slot in a + loop is still a branch per slot; you have swapped a slot touch for a byte + touch. Only Step 4's group compare — many tags per instruction — changes the + unit probe cost is billed in, and only then does raising the load factor + from 50% to 87.5% become affordable. + +
+ +- [ ] You can show, with the divisions performed, why 87.5% is cheaper for SwissTable than 50% was for `dense_hash_map`. + +
Answer + + Using densehashtable.h:78's own formula for unsuccessful quadratic-probe + lookups, `1/(1−L) − L − ln(1−L)`: at L = 0.50 it gives + 2.000 − 0.500 + 0.693 = **2.193** (matching the 2.19 printed at `:84`, which + is how we know we are reading it right), and at L = 0.875 it gives + 8.000 − 0.875 + 2.079 = **9.204**. That is 9.204 / 2.193 = **4.20× more slot + probes**. + + But those slots are contiguous within a group, and a group is examined in + one load: 9.204 / 8 = **1.15** group loads at `WIDTH = 8`, or + 9.204 / 16 = **0.58** at `WIDTH = 16`. Roughly one cache line either way, + against `dense_hash_map`'s ~2.2 independent slot touches. And the array is + smaller: 1,000,000 entries need 1,000,000/0.875 = 1,142,857 slots instead of + 1,000,000/0.500 = 2,000,000 — **1.75× fewer**. The estimate is rough (the + formula assumes independent probes, a group is a contiguous window) but the + direction is not. + +
+ +- [ ] You can state which numbers in this chapter come from a source you can re-open, and which the talk asserts without one. + +
Answer + + Re-openable: the C++ requirements (cppreference); `HT_OCCUPANCY_PCT = 50`, + the quadratic probe table, `JUMP_`, and both sentinel setters (sparsehash at + `1dffea3d9`); the H1/57-bit, H2/7-bit split, the one-byte-per-entry + overhead, the `MaskMatch` code and the "deleted does not cease probing" rule + (abseil's published design notes); every width, constant and line number + (hashbrown at `d69025b`); the ~100 ns dependent miss and the 58.4 ms insert + spike ([FINDINGS.md](../../FINDINGS.md) rows 0 and 2). + + Not re-openable, and therefore absent: the fleet-wide RAM and CPU + percentages the talk opens with. They may well be right; there is no + artifact to check them against, so this chapter does not print them. That is + the same standard [reading-fair-benchmarking.md](../00-performance-toolbox/reading-fair-benchmarking.md) + applies to published speedups. + +
## References -**Papers** -- Kulukundis — "Designing a Fast, Efficient, Cache-friendly Hash Table, - Step by Step" (CppCon 2017 talk) — - [video](https://www.youtube.com/watch?v=ncHmEUmJZf4) — ~60 min; - timestamps vary across uploads, navigate by the slide titles listed - above - -**Code** -- [hashbrown](https://github.com/rust-lang/hashbrown) — the Rust - incarnation of the final design; walked in - [reading-hashbrown.md](reading-hashbrown.md) +**Talk** +- Matt Kulukundis — "Designing a Fast, Efficient, Cache-friendly Hash Table, + Step by Step", CppCon 2017 — + [video](https://www.youtube.com/watch?v=ncHmEUmJZf4) — ~60 min. Slides were + not deposited in the [CppCon2017](https://github.com/CppCon/CppCon2017) + repository, and timestamps differ across re-uploads; navigate by the slide + content in "How to read the talk" above. + +**Primary sources used in place of the talk's audio** +- [abseil, *Swiss Tables Design Notes*](https://abseil.io/about/design/swisstables) + — the H1/H2 split, the one-byte-per-entry metadata, the five-step lookup, + `MaskMatch`, and the deleted-does-not-stop-probing rule. Co-authored by + Kulukundis; the written form of Steps 3-4. +- [cppreference, `std::unordered_map`](https://en.cppreference.com/w/cpp/container/unordered_map) + — reference/pointer stability, `local_iterator`, `node_type`. + +| Source | Lines | What | +|---|---|---| +| sparsehash `1dffea3d9` `src/sparsehash/internal/densehashtable.h` | 77-84 | the published probes-per-lookup table (2.19 at L=0.5, 5.81 at L=0.8) | +| " | 115-119 | `JUMP_` — quadratic probing, one macro | +| " | 390, 496 | `set_deleted_key`, `set_empty_key` — the two donated sentinels | +| " | 648-653 | the probe loop, stopping at the first empty slot | +| " | 1309-1316 | "Knuth says .8 … we go with .5", `HT_OCCUPANCY_PCT = 50` | +| hashbrown `d69025b` `src/control/tag.rs` | 9-49 | the metadata byte, as shipped | +| " `src/control/group/mod.rs` | 8-46 | which group width your target gets | +| " `src/control/group/sse2.rs` | 20, 73-86 | 16-wide, `_mm_movemask_epi8` | +| " `src/control/group/neon.rs` | 16, 68-73 | 8-wide, `vceq_u8` — this repo's machine | +| " `src/raw.rs` | 58-64 | `h1` — the whole hash, not abseil's 57 bits | +| " `src/raw.rs` | 182-191 | `bucket_mask_to_capacity` — the 7/8 rule | +| " `src/raw.rs` | 2009-2046 | `find_inner` — the design walk's destination | +| " `src/raw.rs` | 2756-2757, 3279 | tombstone policy: rehash-in-place, and when not to write one | + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 0 — the ~1 / 5 / 100 ns cache ladder + behind Step 1's "dependent miss". +- [FINDINGS.md](../../FINDINGS.md) row 2 — hashbrown insert p50 42 ns, max + 58.4 ms: the price of Step 5's stop-the-world growth. + +**Companion chapters** +- [reading-hashbrown.md](reading-hashbrown.md) — the final design as code. +- [reading-redis-dict.md](reading-redis-dict.md) — the incumbent family, and + the incremental-rehash answer SwissTable does not give. diff --git a/topics/03-btree-internals/reading-graefe-survey.md b/topics/03-btree-internals/reading-graefe-survey.md index 98587eb..196bf2d 100644 --- a/topics/03-btree-internals/reading-graefe-survey.md +++ b/topics/03-btree-internals/reading-graefe-survey.md @@ -3,172 +3,695 @@ Every "B-trees are simple" take dies in Graefe's ~200-page survey of what production B-trees actually do — compression, latching, logging interactions, bulk loads. **Do not read it all.** This chapter builds the survey's core -ideas one step at a time — the fanout arithmetic first, then the three -key-compression tricks that move it — and then hands you the ~50 pages that -matter for this topic and the capstone (budget: 3 h); you'll come back for -more in topics 5 (logging), 8/9 (latching), and 12 (columnar). +ideas one step at a time — the fanout arithmetic first, then what that +arithmetic does *not* buy, then the four key-compression tricks that move it — +and then hands you the ~45 pages that matter for this topic and the capstone +(budget: 3 h); you'll come back for more in topics 5 (logging), 8/9 +(latching), and 12 (columnar). + +Every section number below is Graefe, *Modern B-Tree Techniques*, +**Foundations and Trends in Databases Vol. 3, No. 4 (2010), pp. 203–402**, +© 2011, DOI `10.1561/1900000028` — the 203-page PDF, whose section numbers and +figure numbers are the ones cited here. Each claim names the section it came +from, because in a survey this long an unsourced number is unfindable. + +Every *measured* number below is this repo's own, from +[`notes.md`](notes.md) (Apple M3 Pro, 2026-07-28) or +[FINDINGS.md](../../FINDINGS.md) row 3. Nothing in the survey was measured on +your machine, and Graefe measures almost nothing at all — it is a survey, and +its own numbers are illustrative calculations (Fig. 3.1) rather than +experiments. ## The problem in one sentence -A B-tree lookup pays one page read per level of the tree, so on a -billion-key tree every byte you shave off the keys stored in interior pages -raises fanout — and shaving 16-byte keys down to 4-byte separators is the -difference between height 4 holding **1 billion** keys and holding **10 -billion**. +A B-tree lookup pays one page read per level, so every byte shaved off the +keys stored in *interior* pages raises fanout and can cost the tree a whole +level — on this repo's own 4 KiB page format, cutting a 32-byte separator to +4 bytes lifts fanout from **102 to 340** and drops the tree holding 10⁹ keys +from height 5 to height 4, one page read saved on every lookup forever. ## The concepts, step by step ### Step 1 — height is priced in page reads -A B-tree stores its keys in fixed-size **pages** (disk blocks, typically -4 KB): **interior pages** hold keys plus child-page pointers, **leaf pages** -hold the actual data, and a point lookup reads one page per level from root -to leaf. The number of children an interior page holds is the **fanout**, and -the tree's **height** — the number of levels — is log-base-fanout of the key -count. +> **In:** nothing yet — a page size, a key size and a record count, which is +> all the survey needs to price a lookup. +> **Out:** two numbers per key shape — leaf capacity `L` and interior fanout +> `F` — plus the height they imply. Step 2 forks these into an interior-page +> budget and a leaf-page budget, which the compression steps then spend. + +A B-tree stores its keys in fixed-size **pages** — disk blocks, 4–8 KB in +traditional designs (§2.2). **Leaf pages** hold the records; **branch nodes** +(Graefe's word; also called internal, intermediate or *interior* nodes, §1.1) +hold **separator keys** plus child-page pointers. A point lookup reads one +page per level from root to leaf. The **fanout** `F` is the number of children +per branch node — "sometimes only in the tens, typically in the hundreds, and +sometimes in the thousands" (§2.2). The **height** is the number of levels; +Graefe warns the word is ambiguous (§2.2: "the height of this B-tree is 2 +(levels above the leaves) or 3 (levels including the leaves)"), so this +chapter always means *levels including the leaf*, which is what a lookup pays. + +The survey's formula, quoted as §2.2 states it: -The arithmetic to internalize: 4KB page, 16-byte keys + 8-byte child pointers -≈ 170 fanout ⇒ 170⁴ ≈ 1B keys in height 4. +``` +§2.2: N records in the tree + L records per leaf + F average children per parent (the fanout) + + leaf nodes = N / L + branch levels = log_F (N / L) "this expression is rounded up" +``` + +Symbols: `N` is the row count, `L` is how many records fit in one leaf page, +`F` is how many child pointers fit in one branch page. Both `L` and `F` come +out of the page format — that is the entire lever. -Why it matters: height is the *only* number a lookup pays for, and it moves -in whole units — an entire page read gained or lost across *every* lookup. -Every technique in this survey is ultimately a lever on fanout, hence on -height. +Work it on the format this topic's experiments actually use, so the numbers +are checkable rather than round. The format is fixed in +`experiments/src/bin/btree_baseline.rs`: -### Step 2 — separators are synthetic: suffix truncation +```rust +// topics/03-btree-internals/experiments/src/bin/btree_baseline.rs, 29-38 + 29 /// Cells per leaf and interior fanout for the page format documented in + 30 /// src/page.rs. Arithmetic, not measurement — labelled as such in the output. + 31 fn geometry(key_len: usize, val_len: usize) -> (usize, usize) { + 32 // leaf cell: key_len u16 ∥ val_len u16 ∥ key ∥ val (+ 2 for its ptr) + 33 let leaf_cell = 2 + 2 + key_len + val_len + 2; + 34 // interior cell: child u32 ∥ key_len u16 ∥ key (+ 2 for its ptr) + 35 let interior_cell = 4 + 2 + key_len + 2; + 36 let usable = PAGE_SIZE - HEADER; + 37 (usable / leaf_cell, usable / interior_cell) + 38 } +``` -An interior page never needs to store real keys — it stores **separators**, -values whose only job is to route a search left or right, so a separator -between two leaf keys can be *any* string that sorts between them, including -one much shorter than either. +Lines 33 and 35 are the two that matter: they say exactly what a byte of key +costs in each region of the tree. `PAGE_SIZE` is 4096 and `HEADER` is 8 +(`btree_baseline.rs:25-26`), so `usable` = 4088. Run it on an 8-byte key with +an 8-byte value: ``` -suffix truncation: separator between "smith,bob" and "smyth,al" - needs only "smy" — interior keys shrink ⇒ fanout grows - ⇒ height shrinks ⇒ every lookup saves a page +leaf cell = 2 + 2 + 8 + 8 + 2 = 22 bytes ⇒ L = 4088 / 22 = 185.8 → 185 +interior = 4 + 2 + 8 + 2 = 16 bytes ⇒ F = 4088 / 16 = 255.5 → 255 + +N = 1,000,000: leaves = ⌈1000000 / 185⌉ = 5406 + branch levels = log_255(5406) = 8.5951 / 5.5413 = 1.551 → 2 + height = 2 + 1 (the leaf level) = 3 +N = 1,000,000,000: + leaves = ⌈1e9 / 185⌉ = 5,405,406 + branch levels = log_255(5405406) = 15.503 / 5.5413 = 2.798 → 3 + height = 3 + 1 = 4 +``` + +Those are exactly the `185`, `255`, `3`, `4` in the first row of +[`notes.md`](notes.md)'s fanout table and in +[README.md](README.md)'s measured block — the baseline binary is running +Graefe §2.2's formula, rounded up, plus one for the leaf level. + +Why it matters: height is the number of *page touches* a lookup pays, it moves +in whole units, and every technique in the rest of this survey is ultimately a +lever on `L` or `F`, hence on height. + +### Step 2 — the fork: two byte budgets, and the one number fanout cannot move + +> **In:** `L` and `F` from Step 1, and the page format that produced them. +> **Out:** two separate byte budgets — the **interior-page budget**, spent by +> Steps 3, 5 and 6, and the **leaf-page budget**, spent by Step 4 — plus a +> third result Step 7 needs: the comparison count is invariant, so every +> CPU-side win must come from cache faults, not from comparisons. + +The two divisors in `geometry()` are different expressions, and that is the +fork. A branch node stores `child u32 ∥ key_len u16 ∥ key`, so a byte of +separator costs one interior slot byte. A leaf stores +`key_len ∥ val_len ∥ key ∥ val`, so a byte of key costs one leaf slot byte +*and the record must stay exact* — a leaf key **is** the data. Two budgets, +two different sets of techniques: + +```mermaid +flowchart TD + S1["Step 1: page format
⇒ L = 185, F = 255"] + S1 --> IB["interior-page byte budget
(separators are synthetic)"] + S1 --> LB["leaf-page byte budget
(keys must stay exact)"] + S1 --> CC["comparison count
= log2(N), invariant (§2.3)"] + IB --> S3["Step 3 — suffix truncation (§3.5)"] + IB --> S5["Step 5 — normalized keys (§3.4)"] + IB --> S6["Step 6 — poor man's normalized key (§3.6)"] + LB --> S4["Step 4 — prefix truncation (§3.5)"] + CC --> S7["Step 7 — node size (§3.1) and cache faults (§3.6)"] ``` -Suffix truncation is small enough to write down whole — the point is that a -separator is *synthetic*, so it only has to sort between its neighbors: +The third output is the one most readings of this survey miss. A **comparison** +here means one key-versus-key test inside a page's binary search. §2.3 counts +them for a whole root-to-leaf search: + +``` +§2.3: comparisons = log_F (N/L) × log_2(F) + log_2(L) + ╰─ levels ─╯ ╰ per branch ╯ ╰ in the leaf ╯ + + "the product term simplifies to log_2(N/L) and then the entire + expression simplifies to log_2(N)" +``` + +Check it on Step 1's numbers, N = 10⁶, L = 185, F = 255: + +``` +log_255(1000000/185) = log_255(5405.4) = 8.5951 / 5.5413 = 1.5511 +log_2(255) = 7.9944 +log_2(185) = 7.5314 + +1.5511 × 7.9944 + 7.5314 = 12.4002 + 7.5314 = 19.9316 +log_2(1000000) = 19.9316 ✓ +``` + +Identical to four decimal places, because the algebra is exact. So: **raising +fanout buys you page touches, and buys you exactly zero comparisons.** A +bigger page means fewer levels and more comparisons per level, and the two +cancel. §2.3 puts it as "the record count is the only primary influence on the +number of comparisons in a root-to-leaf search". + +That is not a footnote — it is why this topic's headline finding looks the way +it does. [FINDINGS.md](../../FINDINGS.md) row 3 records lookups climbing +**862 → 1101 ns** from 1e6 to 4e6 keys with height pinned at 3. Height did not +move and, by §2.3, the comparison count moved by only +log₂(4×10⁶) − log₂(10⁶) = 2 comparisons. Neither of the two things this +chapter's title names changed. What changed is the third channel, and §3.6 +names it: "cache faults contribute a substantial fraction to the cost of +searching within a B-tree page", and "a cache fault may waste 100s of CPU +cycles" (§3.6 summary). A **cache fault** is a memory reference that misses +CPU cache and has to go to DRAM. + +Why it matters: keep two columns, not one. Fanout controls how many pages you +touch; residency controls what a touch costs. The survey has a section for +each, and this topic measured the second one beating the first. + +### Step 3 — separators are synthetic: suffix truncation + +> **In:** the interior-page byte budget from Step 2 — `4 + 2 + key_len + 2` +> bytes per branch slot. +> **Out:** a smaller `key_len` in that expression, and therefore a larger `F` +> feeding back into Step 1's height formula. Step 4 does the same trick on the +> other budget. + +A branch node never needs to store real keys. **Suffix truncation** (§3.5) +means: when a leaf splits and a new separator must be posted to the parent, +choose not the highest key of the left leaf nor the lowest of the right, but +*the shortest string that separates them*. §3.5 states the rule and then the +part that people get wrong: + +> "Any letter can be chosen that is larger than J and not larger than S. **It +> is not required that the letter actually occurs in the current key values.**" +> — §3.5, on Fig. 3.6 + +So the constraint is an ordering constraint, not a shortening constraint: the +separator must satisfy `max(left leaf) < separator ≤ min(right leaf)` under +the inclusive-upper-bound convention §2.3 assumes. Being *short* is the payoff, +not the requirement. §3.5's Fig. 3.6 example: splitting between +`Johnson, Lucy` and `Smith, Eric`, a separator at the exact centre would need +"at least 9 letters, including the first letter of the given name", but if any +split point between the arrows is acceptable, "a single letter suffices" — +because the two candidate keys differ in their first byte. ```rust -// separator between leaf keys "smith,bob" and "smyth,al" → "smy" +// ILLUSTRATION — not quoted from the survey, which gives no code. The +// experiment hook where you will actually implement this is +// experiments/src/page.rs:73 (`split_into`), whose todo!() at :75 reads +// "try suffix-truncating the separator here and measure fanout". fn shortest_separator(left: &[u8], right: &[u8]) -> Vec { let mut i = 0; - while i < left.len() && left[i] == right[i] { + while i < left.len() && i < right.len() && left[i] == right[i] { i += 1; // skip the shared prefix } - right[..=i].to_vec() // one byte past divergence: > left, ≤ right + // one byte past divergence: strictly > every key in `left`'s leaf, + // and <= `right` because it is a prefix of `right`. + right[..=i.min(right.len() - 1)].to_vec() } -// shorter separators ⇒ more fit per interior page ⇒ fanout up ⇒ height down — -// and height is priced in page reads, so EVERY lookup collects the saving ``` -Redo Step 1's arithmetic with truncation: separators cut to 4 bytes ⇒ fanout -~340 ⇒ height 4 still, but now at 10B keys. **Height is the metric; fanout is -the lever; key size is what you control.** This is your experiment for this -topic — and note that separators need only be *shorter than both neighbors*, -not real keys. (Survey §3.1–3.3.) - -### Step 3 — prefix truncation: store the shared prefix once +Two honest caveats the tidy version hides. First, this returns a *prefix of +`right`*, which is one legal answer but not the only one — §3.5 says any +byte in the open interval works, so a real implementation can often stop a +byte earlier by picking `left[i] + 1` when that is still ≤ `right[i]`. Second, +it needs the length guards on both sides; the version without them panics when +one key is a prefix of the other. -Leaf keys must stay exact — they *are* the data — so leaves compress -differently: when every key on a page shares a common prefix, store that -prefix once in the page header and keep only the distinct tails in the cells -(the per-key entries within a page). +Now spend the budget. Interior slot = `4 + 2 + key_len + 2` = `8 + key_len`, +over `usable` = 4088: ``` -prefix truncation: page stores common prefix once - page ["foo/aaa".."foo/zzz"]: header prefix="foo/", cells store "aaa"… +key_len = 32 (full key) slot = 40 ⇒ F = 4088 / 40 = 102.2 → 102 +key_len = 4 (truncated sep) slot = 12 ⇒ F = 4088 / 12 = 340.6 → 340 + lift = 340 / 102 = 3.33× ``` -Same lever, different region of the tree: more keys per leaf ⇒ fewer leaves ⇒ -fewer interior entries ⇒ (eventually) a shorter tree. The cost: the common -prefix must be recomputed whenever a split or merge changes the page's key -range. (Survey §3.1–3.3.) +The `102` is [`notes.md`](notes.md)'s measured-format row for a 32-byte key, +so the left-hand column is not invented. Feed both into Step 1's height +formula, with leaf capacity unchanged at `L` = 88 (leaves keep the whole key — +that is Step 4's problem, not this one): -### Step 4 — normalized keys: comparison becomes one memcmp +``` +N = 10⁶, L = 88 ⇒ leaves = ⌈10⁶/88⌉ = 11,364 + F = 102: ⌈11364/102⌉ = 112 → ⌈112/102⌉ = 2 → ⌈2/102⌉ = 1 height 4 + F = 340: ⌈11364/340⌉ = 34 → ⌈ 34/340⌉ = 1 height 3 + +N = 10⁹, L = 88 ⇒ leaves = 11,363,637 + F = 102: 111,408 → 1,093 → 11 → 1 height 5 + F = 340: 33,423 → 99 → 1 height 4 +``` -A **normalized key** is a re-encoding of a typed, possibly composite key -(say, an integer column plus a case-insensitive string column) into a single -byte string whose plain byte-by-byte order equals the intended sort order — -so one branch-free `memcmp` replaces a typed comparison that dispatches on -column types and collations. +**One whole level at both scales.** That is the prediction +[README.md](README.md) §7 asks you to write down before you run the truncation +experiment ("Predict first: fanout ratio ⇒ height change at 1M keys?"). It is +arithmetic on the stated page format, not a measurement — and Step 2 is the +reason you should not expect the lookup time to fall by a third when you +measure it. + +§3.5 also kills the obvious follow-up: do **not** apply suffix truncation when +splitting a *branch* node. Fig. 3.7 is titled "Incorrect suffix truncation" and +shows a shortened separator `g` routing a search for key `gh` into the wrong +subtree, "obviously incorrectly". Graefe adds the reason not to care: "if 99% +of all B-tree nodes are leaves and 99% of the remaining nodes are immediate +parents of leaves, additional truncation could benefit at most 1% of 1% of all +nodes." + +Why it matters: **height is the metric; fanout is the lever; interior key size +is what you control.** This is your experiment for this topic. + +### Step 4 — prefix truncation: store the shared prefix once + +> **In:** the leaf-page byte budget from Step 2 — `2 + 2 + key_len + val_len + 2` +> per leaf slot, with the key required to stay exact. +> **Out:** a larger `L`, which shrinks `N/L` inside Step 1's log and so shrinks +> height from the other end. + +Leaf keys **are** the data, so they cannot be replaced with a synthetic +separator. **Prefix truncation** (§3.5) is the compression that survives that +constraint: "analyzes the keys in a B-tree node and stores the common prefix +only once, truncating it from all keys stored in the node." §3.5's Fig. 3.5 +shows a node of `Smith, Jack` / `Smith, Jane` / `Smith, Jason` / … stored +instead as `Prefix = Smith, J` plus `ack` / `ane` / `ason` / …. + +§3.5 names two benefits and one design decision: + +- Space: "permits increasing the number of records per leaf and increasing the + fan-out of branch nodes" — it applies to both regions, unlike Step 3. +- CPU: "the truncated key bytes do not need to be considered in comparisons + during a search." +- The decision: truncate against the *actual* keys currently in the page, or + against the **maximal possible key range** the page could ever hold? §3.5 + argues for the second, "in particular for insertions": with actual-key + truncation, "insertion of a new key might force reformatting all existing + keys. In an extreme case, a new record might be much smaller than the free + space in a B-tree page yet its insertion might force a page split." + +The possible range is captured by **fence keys** — copies of the separator +keys posted to the parent when this page was split, retained in the page +itself (§3.5; illustrated in Fig. 4.11 in §4.4). The bytes shared by a page's +two fence keys are shared by every key the page can ever hold, now or later. +Note the circularity that makes the whole scheme work: Step 3's suffix +truncation is what keeps fence keys short, and short fence keys are what keep +their overhead affordable — §3.5 says exactly that. + +§3.5 also gives a version that needs **no format change at all**, which is +worth stealing: **dynamic prefix truncation**. While binary-searching a branch +node, the two separator keys flanking the chosen child pointer are already +compared against the search key. If they agree on some leading bytes, every +key under that pointer agrees on them too, so those bytes can be skipped in +all later comparisons — and "dynamic prefix truncation can be exploited +without adding comparison steps to a root-to-leaf search." + +Cost, from the same section: with actual-key truncation the prefix must be +recomputed whenever a split or merge changes the page's key range; with +fence-key truncation it need not, which is the argument for fence keys. + +Why it matters: same lever, opposite end of the tree. Step 3 shrinks `F`'s +divisor; Step 4 shrinks `N/L` inside the log. Only Step 4 can help a tree whose +keys are already short separators. + +### Step 5 — normalized keys: comparison becomes one hardware instruction + +> **In:** the interior-page budget again, plus the fact from Step 2 that the +> comparison *count* is fixed — so the only thing left to attack is the cost of +> one comparison. +> **Out:** keys as plain byte strings, which is the precondition Steps 3, 4 and +> 6 all quietly assumed. + +A **normalized key** (§3.4) is a re-encoding of a typed, possibly multi-column +key into a single binary string "such that simply binary comparisons suffice" +— the byte order of the string equals the intended sort order. The string +encodes "multiple columns, their sort direction (e.g., descending) and +collation including local characters (e.g., case-insensitive German), string +length or string termination" (§3.4). + +Fig. 3.4's worked row: the tuple `(2, "flow", "error")` becomes ``` -normalized keys: encode (type,collation,composite) into memcmp-able bytes - — comparison becomes branch-free byte compare (SIMD-able, - topic 17) +§3.4, Fig. 3.4: 1 0…0 0000 0000 0010 1 flow\0 1 error\0 + ▲ ▲ ▲ + │ └──────────┴─ 1 = column present + └─ 1 = leading column is not null, so nulls (0) sort first ``` -You've met this idea already: it's the binary-comparable encoding of ART -§III.E. Why it matters: within a page the lookup cost is CPU comparisons, not -IO, and a branch-free byte compare is what hardware executes fast — and what -SIMD can widen later (topic 17). (Survey §3.4.) +Three details §3.4 insists on, each of which is a bug if you skip it: + +1. Strings are terminated (`\0`), never length-prefixed: "A length indicator, + for example, would destroy the main value of normalized keys, namely + sorting with simple binary comparisons." +2. Signed integers and floats need bits flipped: "Signed integers require + reversing some bits to ensure the proper sort order, just like floating + point values require proper treatment of exponent, mantissa, and the two + sign bits." +3. Normalization can be **lossy** — a case-insensitive collation maps two + distinct strings to one key — so §3.4 lists three fixes, of which the third + is the B-tree-specific one: "employ normalized keys only in branch nodes; + recall that key values in branch nodes merely guide the search to the + correct child but do not contain user data." + +That third fix is the same observation as Step 3's, reused. And §3.4's closing +bullet reports it is what production does: "Some systems employ normalized keys +in branch nodes but not in leaf nodes" — because (§3.4) normalized keys "tend +to be longer than the original string values", which is a fanout cost you only +want to pay where the key is synthetic anyway. + +You have met this idea before: it is the binary-comparable encoding of ART +(topic 2). §3.5 opens by pointing out the dependency in the other direction — +"Once keys have been normalized into a simple binary string, another B-tree +optimization becomes much easier to implement, namely prefix and suffix +truncation" — so Step 5 is logically *upstream* of Steps 3 and 4 even though +you meet it later. + +Why it matters: §3.6 explains the payoff in the currency Step 2 said was the +only one left. Typed comparison "can require a large amount of code whereas two +normalized keys can be compared by a single hardware instruction" — so +normalization removes *instruction* cache faults, not comparisons. Topic 17 +widens the same byte compare with SIMD. + +### Step 6 — poor man's normalized key: a filter inside the indirection vector + +> **In:** normalized keys from Step 5 and the slotted-page layout §3.3 calls an +> **indirection vector** (the sorted array of per-record slots; this repo's +> READMEs call it the cell pointer array). +> **Out:** a binary search that usually decides without dereferencing any +> record — the last of the three interior-budget techniques. + +§3.6: "After prefix truncation has been applied, many comparisons in a binary +search are decided by the first few bytes. Even where normalized keys are not +used in the records, e.g., in B-tree leaves, storing a few bytes of the +normalized key can speed up comparisons. If only those few bytes are stored, +not the entire normalized key, such that they can decide many but not all +comparisons, they are called **poor man's normalized keys**." + +The placement is the whole trick: put those bytes "as an additional field in +the elements of the indirection vector", not in the record. Fig. 3.8 shows a +page of European countries with a single letter cached per slot: -### Step 5 — poor man's normalized key: a filter inside the pointer array +``` +§3.6, Fig. 3.8 (page of Belgium / France / Luxemburg): -Cache the first few bytes of each cell's normalized key directly inside that -cell's slot in the page's pointer array (the small sorted array of cell -offsets), so binary search usually decides from the slot alone and touches -the actual cell only on a near-tie. + indirection vector records (variable size) + ┌───┬───┬───┐ ┌──────────────┬────────────┬──────────────┐ + │ B │ F │ L │ ──────────▶ │ 7, "elgium" │ 6, "rance" │ 9,"uxemburg" │ + └───┴───┴───┘ └──────────────┴────────────┴──────────────┘ + one cached byte per slot size + the REST of the key lives here -This is the dense-filter pattern yet again — first bytes of the key cached -IN the pointer array slot, the same move as SwissTable's h2 byte and the -skiplist tower. Why it matters: the pointer array is contiguous and hot in -cache; the cells are scattered across the page — each avoided dereference is -an avoided cache miss. (Survey §3.5. Question 3 below asks you to state the -general principle in one sentence.) + search "Denmark": B < D < F — decided by the vector alone, zero record touches + search "Finland": ties with F — must dereference and find "France" +``` -### Step 6 — node size is a trade, not a constant +§3.6's own two cases, verbatim in substance: a search for "Denmark" "can +eliminate all records by the poor man's normalized keys without incurring +cache faults for the main records"; a search for "Finland" "can rely on the +poor man's normalized key for the binary search but eventually must access the +main record for 'France'." + +How many bytes? §3.6 says one letter is only the figure's simplification: +"2 or 4 bytes seem more appropriate, depending on the page size … in a small +database page optimized for flash storage and its fast access latency, 2 bytes +might be optimal; whereas in large database pages optimized for traditional +disks and their fast transfer bandwidth, 4 bytes might be optimal." + +There is a matching *subtraction* in the same section, and it is the better +lesson: §3.6 argues the record *size* should be moved **out** of the slot and +into the record, "because the record length is hardly ever accessed without +access to the related record". Cache-line budget is zero-sum — you earn the +bytes for a filter by evicting a field nobody reads on its own. + +This is the dense-filter pattern this curriculum keeps meeting: the same move +as SwissTable's `h2` byte and the skiplist tower (topic 2), and the same +principle Question 3 below asks you to write in one sentence. Why it matters: +the indirection vector is contiguous and hot; the records are scattered across +the page — and by Step 2, avoided cache faults are the only CPU win left on +the table. + +### Step 7 — node size is a trade, not a constant + +> **In:** everything above — `L`, `F`, the invariant comparison count, and the +> cache-fault channel. +> **Out:** the one parameter that moves all of them at once, and the survey's +> two answers for the two devices. + +Nothing makes 4 KB sacred. §3.1 gives a one-line heuristic: pick the node size +at which **access latency equals transfer time**, computed by multiplying the +two. It "guarantees a sustained transfer bandwidth at least half of the +theoretical optimum as well as an I/O rate at least half of the theoretical +optimum". Its two worked cases: -Nothing makes 4 KB sacred: page size trades IO efficiency (bigger pages -amortize seeks and raise fanout) against CPU cache behavior (binary search -over a huge page thrashes cache lines) and write cost (one dirty byte -rewrites the whole page). +``` +§3.1: disk 5 ms latency × 200 MB/s = 1 MB node size + flash 0.1 ms × 100 MB/s = 10 KB node size +``` -The survey's resolution (§5.1–5.2): big nodes *plus in-node structure* — a -mini-index inside the page — get both: large IO units for the disk, -cache-sized search steps for the CPU. Hold onto this when topic 12 makes -columnar pages megabytes wide. +Check the first: 0.005 s × 200 × 10⁶ B/s = 1.0 × 10⁶ B. And the second: +1 × 10⁻⁴ s × 100 × 10⁶ B/s = 1.0 × 10⁴ B. Both divisions come out as printed. + +§3.1's Fig. 3.1 then optimizes properly, maximizing "the number of comparisons +per unit of I/O time" — the **node utility** being log₂ of the records per +page, i.e. the comparisons one page read buys. Its assumptions are stated: +pages 70% full, 20-byte records, 5 ms access, 200 MB/s burst: + +``` +§3.1, Fig. 3.1: + page KB records/page utility I/O ms utility/time + 4 143 7.163 5.020 1.427 + 16 573 9.163 5.080 1.804 + 64 2,294 11.163 5.320 2.098 + 128 4,588 12.163 5.640 2.157 ← best + 256 9,175 13.163 6.280 2.096 + 1,024 36,700 15.163 10.120 1.498 + 4,096 146,801 17.163 25.480 0.674 +``` + +§3.1's conclusion is blunter than most textbooks': "Historically common disk +pages of 4 KB are far from optimal for B-tree indexes on traditional disk +drives." Note that the *heuristic* said 1 MB and the *optimization* said 128 KB; +Graefe reports both and does not reconcile them, and the 1,024 KB row's +utility/time of 1.498 is below the 4 KB row's 1.427 only barely — the curve is +flat and broad between 64 and 256 KB, which is the real finding. + +The resolution for the CPU side is §3.6's, not §3.1's: keep the big node for +the device and put a **cache-conscious structure inside it** — "organizes the +indirection vector not as a linear array but as a B-tree of cache lines … The +size of each node in this B-tree is equal to a single cache line or a small +number of them", for which §3.6 reports "search time and cache faults within a +B-tree page may be cut in half compared to node formats not optimized for CPU +caches" (§3.6, citing [24]). That is a *cited* claim, not one Graefe measured; +treat it as a pointer to a paper, not as a number. + +Why it matters: your capstone picks a page size once and lives with it. Hold +this section when topic 12 makes columnar pages megabytes wide, and hold §3.6's +answer for what has to go *inside* them. ## How to read the paper (with the concepts in hand) +The section numbers in the table below are this edition's; a previous version +of this chapter cited several of them wrongly (see the note at the end of this +section). ~45 pages total. + Read now (this topic): -| Section | Pages (approx) | Why | +| Section | Pages | Why | |---|---|---| -| §2 Basic techniques | skim | Step 1 — you know this from the code | -| **§3.1–3.3 Prefix + suffix truncation** | read | Steps 2–3; your experiment; separators need only be *shorter than both neighbors*, not real keys | -| **§3.4 Normalized keys** | read | Step 4; binary-comparable encoding again (ART §III.E) — one memcmp replaces typed comparison | -| §3.5 Poor man's normalized key | read | Step 5; first bytes of the key cached IN the pointer array slot — dense filter pattern yet again | -| **§4.2 Overflow / variable-length records** | skim | you saw SQLite's version | -| §5.1–5.2 Node sizes | read | Step 6; why 4KB? (it's not sacred — CPU cache vs disk trade; big nodes + in-node structure) | +| §2.1–2.3 Data structures, sizes, algorithms | 213–221 | Steps 1–2 — the `log_F(N/L)` formula and the `log_2(N)` comparison invariant | +| §3.1 Node size | 232–233 | Step 7; Fig. 3.1's utility table — 4 KB "far from optimal" for disk | +| §3.3 Variable-length records | 235–237 | the slotted page you already read twice, as the survey states it: indirection vector + records growing toward each other | +| **§3.4 Normalized keys** | 237–239 | Step 5; Fig. 3.4's encoding, and why a length prefix would break it | +| **§3.5 Prefix B-trees** | 239–243 | Steps 3–4; Fig. 3.5 prefix, Fig. 3.6 separator choice, Fig. 3.7 *incorrect* suffix truncation. This is your experiment | +| §3.6 CPU caches | 244–246 | Steps 2, 6, 7; poor man's normalized keys (Fig. 3.8), and the cache-fault channel this topic measured | +| §3.11 Splitting nodes | 258–259 | skim — the split policy the above compression interacts with | Defer (note where, come back later): -- §6 latching & B-link trees → topic 9 (concurrency) -- §7 logging & recovery interplay (fence keys, ghost records) → topic 5 -- §8 bulk load / index creation → topic 12/22 +- **§4.1 Latching and locking**, **§4.6 B^link-trees**, **§4.8 latch coupling** + (265–289) → topic 9 (concurrency) +- **§4.2 Ghost records**, **§4.9 physiological logging**, + **§4.4 fence keys at leaf boundaries** (268–293) → topic 5 (durability/WAL) +- **§6.1 Index creation**, **§6.4 bulk insertions**, **§6.6 defragmentation** + (344–363) → topics 12 and 22 +- **§7.4 Column stores**, **§7.5 large values** (381–390) → topics 12 and, + for the overflow story, this topic's SQLite chapters + +Corrections to the previous edition of this chapter, all verified against the +PDF's table of contents (pp. 203–402): truncation is **§3.5**, not §3.1–3.3 +(§3.1 is node size, §3.2 interpolation search, §3.3 variable-length records); +poor man's normalized keys are **§3.6**, not §3.5; node sizes are **§3.1**, not +§5.1–5.2 (§5.1 is disk-order scans, §5.2 fetching rows); latching is **§4**, +not §6 (§6 is B-tree utilities); logging and ghost records are **§4**, not §7 +(§7 is advanced key structures); bulk load and index creation are **§6**, not +§8 (§8 is the two-page conclusion). Only "§3.4 normalized keys" and "§2 basic +techniques" survived unchanged. ## Questions to answer in notes.md 1. Why does suffix truncation apply to interior separators but prefix truncation mostly to leaf pages? (Separators are synthetic; leaf keys must be exact.) + Then extend it: §3.5's Fig. 3.7 forbids suffix truncation when splitting a + *branch* node — say why, and why Graefe thinks it does not matter. 2. SQLite/turso do neither. Given SQLite's design goals (simplicity, robustness, - integer rowids as the common key), argue whether that's the right call. + integer rowids as the common key), argue whether that's the right call. Use + Step 3's arithmetic: for an 8-byte rowid the interior slot is 16 bytes, so + what is the largest fanout truncation could possibly buy? 3. Poor man's normalized key = SwissTable h2 = skiplist tower = pointer-array-as- filter. Write the general principle in one sentence for the capstone notes. +4. §2.3 proves the comparison count is `log_2(N)` whatever the fanout, yet this + topic measured 862 → 1101 ns from 1e6 to 4e6 keys at constant height. Using + §3.6's vocabulary, name the cost channel that moved and say which section of + the survey you would read to attack it. ## Done when -You can do the fanout→height arithmetic cold, and you've marked which sections -you'll return to in topics 5 and 9. +Answer each before unfolding it. + +- [ ] You can do the fanout→height arithmetic cold: given a page size, a header size, a key size and a record count, produce `L`, `F` and the height, with the logarithm actually evaluated. + +
Answer + + The formula is §2.2's: `leaves = N / L`, `branch levels = log_F(N/L)`, + rounded up, and the height a lookup pays is that plus one for the leaf level. + `L` and `F` come from the page format, and the only skill is counting the + per-slot bytes correctly — including the slot pointer itself. + + On this repo's format (`experiments/src/bin/btree_baseline.rs:31-38`, + `PAGE_SIZE` 4096, `HEADER` 8, so 4088 usable): an 8-byte key with an 8-byte + value gives a leaf cell of `2 + 2 + 8 + 8 + 2 = 22` bytes ⇒ `L` = 4088/22 = + 185, and an interior cell of `4 + 2 + 8 + 2 = 16` ⇒ `F` = 4088/16 = 255. At + N = 10⁶: leaves = ⌈10⁶/185⌉ = 5406, log₂₅₅(5406) = 8.5951/5.5413 = 1.551 → 2 + branch levels, height 3. At N = 10⁹: leaves = 5,405,406, + log₂₅₅(5405406) = 15.503/5.5413 = 2.798 → 3, height 4. Those are the `185`, + `255`, `3`, `4` printed in `notes.md`'s fanout table. + +
+ +- [ ] You can say what a separator key is legally required to be, and what suffix truncation therefore buys — with the fanout numbers for a 32-byte key before and after. + +
Answer + + A separator is required only to *sort strictly between* the two leaves it + divides: `max(left) < separator ≤ min(right)`. §3.5 is explicit that it need + not be a real key at all — "It is not required that the letter actually + occurs in the current key values." Shortness is the payoff, not the rule. + + On this repo's format the interior slot is `4 + 2 + key_len + 2 = 8 + key_len` + bytes over 4088 usable. A full 32-byte key gives a 40-byte slot and + `F` = 102 — the number in `notes.md`'s second row. A 4-byte separator gives a + 12-byte slot and `F` = 340, a 3.33× lift. With leaf capacity unchanged at + `L` = 88, that is height 4 → 3 at 10⁶ keys (11,364 leaves: 112 → 2 → 1 versus + 34 → 1) and height 5 → 4 at 10⁹. + + One caveat §3.5 makes loudly: this applies when splitting *leaves* only. + Fig. 3.7, "Incorrect suffix truncation", shows a shortened branch-level + separator `g` routing a search for `gh` into the wrong subtree. + +
+ +- [ ] You can explain why a bigger fanout does not reduce the number of key comparisons a lookup performs, and name what it does reduce. + +
Answer + + §2.3 counts a whole root-to-leaf search as + `log_F(N/L) × log_2(F) + log_2(L)`: fewer levels, but proportionally more + comparisons per level, and the two cancel exactly — + "the entire expression simplifies to `log_2(N)`". Numerically, at N = 10⁶, + L = 185, F = 255: 1.5511 × 7.9944 + 7.5314 = 19.9316, and + log₂(10⁶) = 19.9316. + + What fanout reduces is **page touches** — height, which is one page read (or + at best one page-cache hit plus a pointer chase) per level. That is the + quantity worth paying for when a touch is expensive, and it is why this + chapter is titled the way it is. It is also why the topic's measured ladder + is not a step function: at 4e6 keys and 270 MB, `notes.md` records 1101 + ns/lookup against 862 at 1e6 keys with height pinned at 3, because the pages + stopped fitting in CPU cache. §3.6 is the section that owns that cost: + "a cache fault may waste 100s of CPU cycles." + +
+ +- [ ] You can state where a poor man's normalized key is stored, why that location and not another, and how many bytes the survey recommends. + +
Answer + + In the **indirection vector** — §3.3's name for the sorted per-record slot + array, this repo's "cell pointer array" — as an extra field in each slot, + never in the record. §3.6's reason is cache faults, not space: the vector is + a small contiguous run that the binary search walks anyway, whereas the + records are scattered across the page, so a comparison decided from the slot + costs no additional cache line. Fig. 3.8's example: searching a page of + Belgium/France/Luxemburg for "Denmark" is settled by the cached first bytes + alone; searching for "Finland" ties against `F` and must dereference the + "France" record. + + §3.6 recommends **2 or 4 bytes**, "depending on the page size" — 2 for small + flash-sized pages, 4 for large disk-sized pages; the single letter in + Fig. 3.8 is the figure's simplification. The same section pays for those + bytes by moving the record *length* out of the slot and into the record, + "because the record length is hardly ever accessed without access to the + related record". + +
+ +- [ ] You can say what page size §3.1's own numbers point at, and why 4 KB survives anyway. + +
Answer + + Two answers that do not agree. §3.1's heuristic — set node size so access + latency equals transfer time, i.e. multiply them — gives 5 ms × 200 MB/s = + 1 MB for a disk and 0.1 ms × 100 MB/s = 10 KB for flash. §3.1's Fig. 3.1 + optimizes utility (comparisons bought) per I/O millisecond instead and peaks + at **128 KB**, with 2.157 against 4 KB's 1.427; §3.1's verdict on the + historical default is "Historically common disk pages of 4 KB are far from + optimal for B-tree indexes on traditional disk drives." + + 4 KB survives because the disk is no longer the only cost. §3.6 is the + counterweight: binary search over a very large page is a sequence of cache + faults, "a cache fault may waste 100s of CPU cycles", and §2.3 already + proved a bigger page buys zero comparisons. The survey's resolution is not + "pick a small page" but "pick the page the device wants and give it internal + structure" — an indirection vector organized as a B-tree of cache lines, + for which §3.6 cites a halving of in-page search time and cache faults. That + is also exactly the split this topic measured: fanout is one lever, + residency is the other. + +
## References **Papers** -- Graefe — "Modern B-Tree Techniques" (Foundations and Trends in - Databases, 2011) — ~200 pages; do NOT read it all — follow the - section table above (§3 truncation + normalized keys and §5 node - sizes now; §6–§8 deferred to topics 9, 5, and 12/22) +- Goetz Graefe — "Modern B-Tree Techniques", *Foundations and Trends in + Databases* Vol. 3, No. 4 (2010), pp. 203–402, © 2011, + DOI [10.1561/1900000028](https://doi.org/10.1561/1900000028) — ~200 pages; + do NOT read it all, follow the section table above. + +| Section | Pages | What this chapter took from it | +|---|---|---| +| §2.2 | 215–216 | `branch levels = log_F(N/L)`, rounded up; node sizes 4–8 KB; ">99% of all nodes are leaves"; 70% average utilization | +| §2.3 | 216–218 | `log_F(N/L) × log_2(F) + log_2(L) = log_2(N)` — the comparison count fanout cannot move | +| §3.1 | 232–233 | latency × bandwidth heuristic (1 MB disk, 10 KB flash); Fig. 3.1's utility table peaking at 128 KB | +| §3.3 | 235–237 | the indirection vector, and the two regions growing toward each other | +| §3.4, Fig. 3.4 | 237–239 | normalized keys; null bits, `\0` termination, why a length prefix breaks it; branch-only normalization | +| §3.5, Figs. 3.5–3.7 | 239–243 | prefix truncation, fence keys, dynamic prefix truncation, shortest separators, and Fig. 3.7's *incorrect* suffix truncation | +| §3.6, Fig. 3.8 | 244–246 | cache faults as a first-class cost; poor man's normalized keys in the indirection vector, 2–4 bytes | + +**This repo** +- [`notes.md`](notes.md) — the fanout table (185/255, 88/102, 35/255) and the + height ladder these steps compute against, Apple M3 Pro, measured 2026-07-28. +- [FINDINGS.md](../../FINDINGS.md) row 3 — 862 → 1101 ns at constant height: + §2.3's invariant and §3.6's cache faults, measured. +- `experiments/src/bin/btree_baseline.rs:29-50` — `geometry()` and `height()`, + the two functions that implement §2.2's formula. diff --git a/topics/03-btree-internals/reading-lmdb.md b/topics/03-btree-internals/reading-lmdb.md index 55c5c73..188c8aa 100644 --- a/topics/03-btree-internals/reading-lmdb.md +++ b/topics/03-btree-internals/reading-lmdb.md @@ -3,196 +3,700 @@ LMDB is the anti-SQLite: no WAL, no page cache of its own, no free-space- within-page — just copy-on-write pages over one big mmap, with crash recovery reduced to picking the newer of two meta pages. This chapter builds that -design one step at a time — the mmap, copy-on-write, the two-meta commit -protocol, page reuse, and the reader table — then hands you the anchors to -read its single 12,846-line file as a *design*, skimming the code (2 h). It -is also the on-disk twin of the capstone reference's in-memory `cow_btree`, -which is exactly M3's comparison exercise. +design one step at a time — the mmap, copy-on-write, the two lists one page +touch produces, the two-meta commit protocol, page reuse, and the reader table +— then hands you the anchors to read its single 12,846-line file as a *design*, +skimming the code (2 h). It is also the on-disk twin of the capstone +reference's in-memory `cow_btree`, which is exactly M3's comparison exercise. + +Every anchor below is `libraries/liblmdb/mdb.c` at the commit this repo pins, +**`LMDB/lmdb@704dc70`** (confirm with `tools/pinned-source.py ref lmdb`), and +the file is 12,846 lines at that revision. Line numbers are the ones the code +occupies there; re-check any you carry elsewhere. ## The problem in one sentence A crash can strike between any two of the hundreds of page writes in a -commit, yet reopening an LMDB database afterwards costs exactly **two page -reads** — read both meta pages, keep the one with the larger valid -transaction id — with no log to replay and no repair step to run. +commit, yet reopening an LMDB database afterwards costs exactly **two reads of +a meta-page-sized buffer** — `mdb_env_read_header` (mdb.c:4673) loops +`NUM_METAS` = 2 times over a `pread` at :4718 and keeps the larger valid +`mm_txnid` at :4749 — with no log to replay and no repair step to run. ## The concepts, step by step ### Step 1 — one big mmap: the OS page cache IS the cache -A **page** is a fixed-size block (4 KB by default) — the unit of disk IO — -and **mmap** is the system call that makes a file addressable as ordinary -memory. LMDB maps the entire database file read-only into the process's -address space once, at open. +> **In:** nothing yet — a file on disk and an `open()`. +> **Out:** an address range, `env->me_map`, that every later step reads +> through; plus a page size, `env->me_psize`, that Step 2's copy cost and +> Step 4's arithmetic are both denominated in. + +A **page** is the fixed-size block LMDB reads, writes and copies as a unit, and +**mmap** is the system call that makes a file addressable as ordinary memory. +`mdb_env_map` (mdb.c:5040) maps the environment once, at open: + +```c +// libraries/liblmdb/mdb.c — inside mdb_env_map, the POSIX branch, 5095-5118 + 5095 int mmap_flags = MAP_SHARED; + 5096 int prot = PROT_READ; + 5097 if (flags & MDB_WRITEMAP) + 5098 prot |= PROT_WRITE; + // ... 5099-5116: MAP_NOSYNC on FreeBSD, the MDB_VL32 partial-map branch, + // and ftruncate when MDB_WRITEMAP is on ... + 5117 env->me_map = mmap(addr, env->me_mapsize, prot, mmap_flags, + 5118 env->me_fd, 0); +``` -A read is then just a pointer dereference into the map: zero-copy, no buffer -pool, no page cache of LMDB's own — the OS page cache IS the cache. Writes do -*not* go through the map by default: they go through `pwrite` (an explicit -write-at-offset system call), or through a writable map only if you opt into -`MDB_WRITEMAP`. +Line 5096 is the one that carries the design: the default protection is +`PROT_READ` and nothing else. A read is then a pointer dereference into the +map — zero-copy, no buffer pool, no page cache of LMDB's own. Writes do *not* +go through the map by default: `mdb_page_flush` (mdb.c:4105) writes dirty +pages with `pwrite`/`pwritev` (mdb.c:4237, :4240) through `env->me_fd` +(:4122). A writable map is opt-in, via `MDB_WRITEMAP` at :5097. + +Two corrections to the folklore, both visible in the code: + +- **The page size is not 4 KB by default; it is the OS page size.** For a new + environment `mdb_env_open` sets `env->me_psize = env->me_os_psize` + (mdb.c:5520), capped at `MAX_PAGESIZE`; for an existing one it takes the + size recorded in the file, `env->me_psize = meta.mm_psize` (:5527). That is + 4096 on x86-64 Linux and **16384 on the Apple Silicon machine this repo + measures on**, so every "4 KB page" below is an assumption you should state + when you reuse it. +- **It maps `me_mapsize`, not the file size** (:5117). `me_mapsize` is a + configured maximum, defaulting to `DEFAULT_MAPSIZE` = 1,048,576 bytes + (mdb.c:788) and normally raised with `mdb_env_set_mapsize`. This is why LMDB + makes you choose a ceiling up front and why `MDB_MAP_FULL` exists — the + address range is fixed at open, and growth past it is an error, not a + remap. Why it matters: topic 6's mmap-considered-harmful paper will argue mmap is -dangerous for *writes* (no control over write-back order) — note that LMDB's -default mode avoids exactly that by using pwrite + the meta protocol of -Step 3, not the writable map. +dangerous for *writes*, because the application has no control over write-back +order. Line 5096 is LMDB's answer: the map is read-only, and ordering is +enforced by the explicit `pwrite` sequence of Step 4. ### Step 2 — copy-on-write: never overwrite a live page +> **In:** the mmap and `me_psize` from Step 1, plus a write transaction and a +> cursor sitting on some page. +> **Out:** for each page the write touches, a *new* page at a *new* page +> number, with the parent repointed at it. Step 3 collects the two lists this +> produces. + **Copy-on-write** (COW) means a transaction never modifies a page that any -committed version of the tree can reach: the first write to a clean page -inside a transaction copies it to a *fresh page number*, and the parent's -child pointer is updated to point at the copy — which works because the -parent was touched first (the descent touches top-down). +committed version of the tree can still reach. The first write to such a page +inside a transaction copies it to a fresh page number, and the parent's child +pointer is updated to point at the copy. `mdb_page_touch` (mdb.c:3015) is the +whole mechanism, and the interesting half is eighteen lines: + +```c +// libraries/liblmdb/mdb.c — inside mdb_page_touch, 3024-3044 + 3024 if (IS_SUBP(mp) || IS_WRITABLE(txn, mp)) + 3025 return MDB_SUCCESS; + 3026 + 3027 if (!IS_MUTABLE(txn, mp)) { + 3028 /* Page from an older snapshot */ + 3029 if ((rc = mdb_midl_need(&txn->mt_free_pgs, 1)) || + 3030 (rc = mdb_page_alloc(mc, 1, &np))) + 3031 goto fail; + 3032 pgno = np->mp_pgno; + // ... 3033-3035: a debug print and an assertion that the pgno really changed ... + 3036 mdb_midl_xappend(txn->mt_free_pgs, mp->mp_pgno); + 3037 /* Update the parent page, if any, to point to the new page */ + 3038 if (mc->mc_top) { + 3039 MDB_page *parent = mc->mc_pg[mc->mc_top-1]; + 3040 MDB_node *node = NODEPTR(parent, mc->mc_ki[mc->mc_top-1]); + 3041 SETPGNO(node, pgno); + 3042 } else { + 3043 mc->mc_db->md_root = pgno; + 3044 } +``` + +The line to focus on is **3041**: the parent's child pointer is edited *in +place*. That is only legal because the parent was itself touched earlier in +the same descent — a cursor walks root-to-leaf, so by the time the leaf is +touched every ancestor already sits at a new page number. Line 3043 is the +base case: no parent means this is the root, so the new page number goes into +`md_root`, which is the field a meta page carries. + +Line 3025's early return is the reason the cost is a *path*, not a *tree*: a +page already writable in this transaction is touched once and reused. The copy +itself happens at :3071 (`mdb_page_copy`), `me_psize` bytes at a time. ``` - modify one key in a height-4 tree: + modify one key in a tree of depth 4: - before: root₀ → int₀ → int₀' → leaf₀ (all shared, read-only) - after: root₁ → int₁ → int₁' → leaf₁ (4 NEW pages = 16 KB written) - root₀ → int₀ → int₀' → leaf₀ (old path still intact) + before: root₀ → branch₀ → branch₀' → leaf₀ (all shared, read-only) + after: root₁ → branch₁ → branch₁' → leaf₁ (4 NEW pages written) + root₀ → branch₀ → branch₀' → leaf₀ (old path still intact, + readers may hold it) ``` -Dirty pages are tracked in a sorted list of page IDs and flushed -sequentially at commit. The cost is right there in the diagram: changing one -key at tree height 4 writes four 4 KB pages — 16 KB — where a WAL engine -would append a ~100-byte log record. That is the price of Step 3's free -recovery. +The tree's depth is not a guess: LMDB records it in `MDB_db.md_depth` +(mdb.c:1330), which lives in the meta page (`mm_dbs`, :1374) and is what +`mdb_stat` prints. + +Compare with the capstone reference's in-memory `cow_btree`: same path copy, +but `Arc` refcounts replace Step 6's freelist, and "commit" is an atomic root +swap instead of Step 4's meta write. Write this comparison in notes — it's +M3's core. + +### Step 3 — the fork: one page touch, two lists + +> **In:** the new and old page numbers produced by every `mdb_page_touch` call +> in Step 2. +> **Out:** two separate lists that go to two different places — `dirty_list`, +> which Step 4's commit writes to disk, and `mt_free_pgs`, which Step 6's +> allocator eventually recycles. Confusing them is the classic misreading of +> LMDB. + +Look again at the block above. It produces two facts per touched page, and +they are consumed by code that has nothing to do with each other: + +```mermaid +flowchart TD + T["mdb_page_touch 3015
one clean page touched"] + T -->|"new pgno, dirtied at :3071"| D["txn->mt_u.dirty_list
(sorted, mdb_page_dirty 2659,
insert at 2670)"] + T -->|"old pgno, appended at :3036"| F["txn->mt_free_pgs
(this txn freed these)"] + D --> C["Step 4: mdb_page_flush 4105
pwritev to disk at commit"] + F --> G["Step 6: mdb_freelist_save 3858
writes them into FREE_DBI,
keyed by this txnid"] +``` -Compare with the capstone reference's in-memory `cow_btree`: same path-copy, -but Arc refcounts replace the freelist, and "commit" is an atomic root swap -instead of a meta-page write. Write this comparison in notes — it's M3's core. +A **dirty page** is a page this transaction has written and must flush. +`mdb_page_dirty` (mdb.c:2659) records it, and line 2670's `mdb_mid2l_insert` +keeps `dirty_list` sorted by page number — which is why Step 4's flush can +coalesce runs of adjacent pages into one `pwritev` (:4240) instead of one +syscall per page. + +A **freed page** is the *old* copy, appended to `txn->mt_free_pgs` at :3036. +It is emphatically not garbage yet: readers pinned to older transactions may +still be walking it (Step 5). `mdb_freelist_save` (mdb.c:3858), called from +the commit path at :4571, writes this list into the freelist database keyed by +this transaction's id — which is the *entire* record of when a page became +reusable. + +Why it matters: the write amplification of Step 4 and the unbounded growth of +Step 6 are the same event seen from two sides. One touched page costs one page +written now and one page number owed to the future. + +### Step 4 — the commit protocol: two meta pages, two durability barriers + +> **In:** `dirty_list` from Step 3, plus a transaction id. +> **Out:** a durable, complete new version of the tree reachable from one meta +> page — and, whatever happens mid-way, the *previous* version still reachable +> from the other. Step 5's readers pick between them. + +A **meta page** stores the root page number of every database in the +environment plus the id of the transaction (**txnid**) that produced them — +the entry point to one complete, immutable version of the tree. LMDB keeps +exactly two, and the comment above the struct is the whole design: + +```c +// libraries/liblmdb/mdb.c — NUM_METAS and the MDB_meta comment, 1351-1358 + 1351 /** Number of meta pages - also hardcoded elsewhere */ + 1352 #define NUM_METAS 2 + 1353 + 1354 /** Meta page content. + 1355 * A meta page is the start point for accessing a database snapshot. + 1356 * Pages 0-1 are meta pages. Transaction N writes meta page #(N % 2). + 1357 */ + 1358 typedef struct MDB_meta { +``` -### Step 3 — the commit protocol: two meta pages, two fsyncs +Line 1356 is the invariant: **pages** 0 and 1 (not byte offsets 0 and 1 — they +sit at byte 0 and byte `me_psize`), and txn N writes meta `N % 2`, so the +previously committed meta is never the one being overwritten. +`mdb_env_write_meta` implements the toggle in one line, :4863, +`toggle = txn->mt_txnid & 1`. + +The ordering, quoted from the commit path: + +```c +// libraries/liblmdb/mdb.c — inside mdb_txn_commit, 4571-4599 + 4571 rc = mdb_freelist_save(txn); + // ... 4572-4582: error handling and a MDB_DEBUG-only mdb_audit() ... + 4583 if ((rc = mdb_page_flush(txn, 0))) + 4584 goto fail; + // ... 4585-4588: assert the loose-page count matches dirty_list ... + 4589 if (!F_ISSET(txn->mt_flags, MDB_TXN_NOSYNC) && + 4590 (rc = mdb_env_sync0(env, 0, txn->mt_next_pgno))) + 4591 goto fail; + // ... 4592-4596: the MDB_TXN_PREPARE early return ... + 4597 prepared: + 4598 if ((rc = mdb_env_write_meta(txn))) + 4599 goto fail; +``` -A **meta page** is a page storing the root page number plus the id of the -transaction (**txnid**) that produced it — the entry point to one complete, -immutable version of the tree. LMDB keeps two, at file offsets 0 and 1, and -txn N writes meta `N % 2` — so the previously committed meta is *never* -overwritten. +Read it as four events: record the freed pages (4571), write the data pages +(4583), **barrier one** (4590, `mdb_env_sync0` — a real `fsync`/`msync`, and +skippable with `MDB_NOSYNC`), then write the meta (4598). + +**Barrier two is not a second `fsync`, and the previous edition of this +chapter said it was.** `mdb_env_write_meta` gets durability from the file +descriptor instead: + +```c +// libraries/liblmdb/mdb.c — inside mdb_env_write_meta, 4918-4937 + 4918 off = offsetof(MDB_meta, mm_mapsize); + 4919 ptr = (char *)&meta + off; + 4920 len = sizeof(MDB_meta) - off; + 4921 off += (char *)mp - env->me_map; + 4922 + 4923 /* Write to the SYNC fd unless MDB_NOSYNC/MDB_NOMETASYNC. + 4924 * (me_mfd goes to the same file as me_fd, but writing to it + 4925 * also syncs to disk. Avoids a separate fdatasync() call.) + 4926 */ + 4927 mfd = (flags & (MDB_NOSYNC|MDB_NOMETASYNC)) ? env->me_fd : env->me_mfd; + // ... 4928-4935: the Windows OVERLAPPED WriteFile branch ... + 4936 retry_write: + 4937 rc = pwrite(mfd, ptr, len, off); +``` -Commit is: write the dirty COW pages → fsync (force to disk) → write the -meta page → fsync. +Line 4927 is the correction: `me_mfd` is a second descriptor onto the same +file, opened `MDB_O_META = O_WRONLY|MDB_DSYNC` (mdb.c:5318), so the `pwrite` +at 4937 *is* the barrier. The comment at 4924-4925 says so. There is one +platform exception, at :4964-4968: on `__APPLE__` LMDB does issue an explicit +`MDB_FDATASYNC(env->me_mfd)`, because Darwin's `O_DSYNC` does not reach the +platter. So "two fsyncs" is true on macOS and false on Linux; "two durability +barriers, independently disableable" is true everywhere — +`MDB_NOSYNC` is documented as "don't fsync after commit" and `MDB_NOMETASYNC` +as "don't fsync metapage after commit" (`lmdb.h:354`, `:358`). + +Lines 4918-4920 also carry a detail worth its own sentence: the meta write is +a **partial-struct write**, starting at `mm_mapsize` and running to the end of +`MDB_meta`. On a 64-bit build that is +`8 (mm_mapsize) + 2 × 48 (mm_dbs) + 8 (mm_last_pg) + 8 (mm_txnid)` = +**120 bytes** — the `48` being `MDB_db` (mdb.c:1327-1336: `4 + 2 + 2` then five +8-byte fields). `mm_magic` and `mm_version` are deliberately *not* rewritten, +because they never change. 120 bytes fits inside a single 512-byte sector, and +that — not a checksum — is the atomicity LMDB depends on. There is no checksum +on a meta page; validation at open is only the `P_META` flag (:4730), the +magic (:4738) and the version (:4743). ``` - crash timeline: recovery = nothing: - write pages ─ fsync ─ write meta ─ fsync open env, read both metas, - ▲crash: old meta wins ▲crash: old pick larger valid txnid - (new pages unreachable) meta wins (mdb_env_pick_meta) + crash timeline: recovery = nothing: + pages ─ fsync(4590) ─ meta pwrite(4937, O_DSYNC) + ▲crash: old meta wins ▲crash: old meta wins + (new pages unreachable) (the other slot is untouched) + + open: mdb_env_read_header 4673 — two preads (4718), keep larger mm_txnid (4749) + then, per read txn: mdb_env_pick_meta 4990 — one comparison, no I/O ``` -No WAL, no redo, no undo. Recovery is *choosing a root pointer*. The price is -paid elsewhere: every commit rewrites the whole root-to-leaf path (Step 2). +Recovery really is one expression: + +```c +// libraries/liblmdb/mdb.c — mdb_env_pick_meta in full, 4985-4995 + 4985 /** Check both meta pages to see which one is newer. + 4986 * @param[in] env the environment handle + 4987 * @return newest #MDB_meta. + 4988 */ + 4989 static MDB_meta * + 4990 mdb_env_pick_meta(const MDB_env *env) + 4991 { + 4992 MDB_meta *const *metas = env->me_metas; + 4993 return metas[ (metas[0]->mm_txnid < metas[1]->mm_txnid) ^ + 4994 ((env->me_flags & MDB_PREVSNAPSHOT) != 0) ]; + 4995 } +``` -The protocol fits on a napkin: +Line 4993 is the entire redo log of this database: a comparison of two +integers. (The `^` on 4994 is the `MDB_PREVSNAPSHOT` debugging flag, which +deliberately picks the *older* meta — proof that the older one is still a +complete, mountable tree.) No WAL, no redo, no undo. -```rust -fn commit(env: &mut Env, txn: Txn) -> Result<()> { - write_pages(&txn.dirty)?; // COW pages at NEW page numbers — - fsync(env.fd)?; // durable before any root sees them - let meta = Meta { txnid: txn.id, root: txn.new_root }; - write_meta_slot(env, (txn.id % 2) as usize, &meta)?; // toggle: never - fsync(env.fd) // overwrite live meta -} +Price it, because "free recovery" is not free. Assume 4096-byte pages (state +this — Step 1 showed it is the OS page size, so it is 16384 on Apple Silicon), +a tree of depth 4, and one key changed: -fn open(env: &Env) -> Root { - let (m0, m1) = read_both_metas(env); - pick_valid_with_larger_txnid(m0, m1).root // recovery IS this line — -} // a crash anywhere above just - // means the old meta still wins +``` +COW path copy 4 pages × 4096 = 16,384 B (Step 2) +freelist DB page ≥1 page × 4096 = 4,096 B (mdb_freelist_save 3858 — + FREE_DBI is itself a COW tree) +meta write 120 B = 120 B (mdb_env_write_meta 4918-4920) + ───────── + ≥ 20,600 B for one key + +a WAL engine's equivalent: one ~100 B log record, appended, then one fsync + ≥ 20,600 / 100 = 206× the bytes ``` -### Step 4 — readers never block writers +Why it matters: recovery is a root-pointer choice precisely *because* commit +paid for it in advance, 206× over in this example. Step 7 shows the same trade +simplifying the split code. -A **read transaction** is just a claim on one version of the tree: it picks -the newest meta, records that meta's txnid in a **reader slot** (one entry -per reader in a shared lock file), and from then on follows pointers through -pages that — by the COW rule — nobody will ever modify. That's the entire -read-txn setup: no locks on the data pages, ever. +### Step 5 — readers never block writers + +> **In:** the two meta pages Step 4 maintains. +> **Out:** a frozen `txnid` per active reader, published in a shared table — +> which is simultaneously what makes reads lock-free and the input Step 6's +> allocator must respect. + +A **read transaction** is just a claim on one version of the tree. It picks a +txnid, records it in a **reader slot** — one `MDB_reader` (mdb.c:869) per +reader in a shared lock file — and from then on follows pointers through pages +that, by Step 2's rule, nobody will ever modify. There are no locks on data +pages, ever. + +`mdb_txn_renew0` (mdb.c:3285) does the setup, and it has two paths. The +previous edition of this chapter cited only the first: + +- **No lock table** (`MDB_NOLOCK`, or a read-only env with no `ti`): :3294-3298 + calls `mdb_env_pick_meta` at :3296 and takes `meta->mm_txnid` directly. +- **The normal path**: :3349-3358 publishes the shared + `ti->mti_txnid` into the slot first — `do r->mr_txnid = ti->mti_txnid; while + (r->mr_txnid != ti->mti_txnid);` at :3349-3351, a retry loop against a racing + committer — and only then derives the meta from it, + `meta = env->me_metas[r->mr_txnid & 1]` at :3356. + +The ordering in the normal path is the load-bearing part: **publish, then +read**. A writer that commits between the two sees the reader's slot already +claiming the older txnid, so Step 6's allocator will not recycle the pages +that reader is about to walk. Writes are the opposite extreme: a single writer mutex allows exactly one -write transaction at a time — LMDB doesn't pretend otherwise. +write transaction at a time. LMDB does not pretend otherwise. Why it matters: readers cost nothing to run and nothing to writers, which is what makes LMDB's read path famous — but each reader's frozen txnid becomes a -liability in Step 5. - -### Step 5 — page reuse: garbage collection as a database +liability in Step 6. -COW keeps producing dead pages (every superseded path), so LMDB stores freed -page IDs in a **freelist database** — an internal B-tree (`FREE_DBI`) keyed -by the txn that freed them. The allocator reuses a freed page only if it was -freed by a txn *older than the oldest active reader* — found by scanning the -reader table of Step 4 for the smallest frozen txnid. +### Step 6 — page reuse: garbage collection as a database -Consequence: a stalled reader pins EVERY page version since its snapshot — -the file grows without bound. (The infamous LMDB "long-lived reader" footgun; -the reference `cow_btree` has the same issue as Arc-pinned snapshots.) +> **In:** `mt_free_pgs` from Step 3, now persisted into `FREE_DBI` and keyed by +> the txnid that freed each page; plus the reader slots from Step 5. +> **Out:** page numbers the allocator may hand out again — and, when a reader +> stalls, a file that grows without bound. -Why it matters: LMDB has no compaction and no vacuum — this -freed-by-txn bookkeeping is the *only* thing standing between COW and -unbounded file growth, and one forgotten read txn defeats it. +COW keeps producing dead pages (every superseded path), so LMDB stores freed +page ids in a **freelist database** — an internal B-tree, `FREE_DBI` = 0 +(mdb.c:1345), keyed by the txn that freed them. `mdb_page_alloc` (mdb.c:2693) +reuses a freed page only if it was freed by a transaction older than the oldest +active reader; the gate is a `break` out of the freelist scan: + +```c +// libraries/liblmdb/mdb.c — inside mdb_page_alloc's freeDB scan, 2800-2810 + 2800 last++; + 2801 /* Do not fetch more if the record will be too recent */ + 2802 if (oldest <= last) { + 2803 if (!found_old) { + 2804 oldest = mdb_find_oldest(txn); + 2805 env->me_pgoldest = oldest; + 2806 found_old = 1; + 2807 } + 2808 if (oldest <= last) + 2809 break; + 2810 } +``` -### Step 6 — search and split: COW makes redistribution pointless +`last` is the txnid key of the freeDB record being considered; line 2808 stops +the moment that key reaches `oldest`. The same guard repeats at :2818-2826 for +the record actually fetched. And `oldest` is a linear scan of the reader table: + +```c +// libraries/liblmdb/mdb.c — mdb_find_oldest in full, 2638-2655 + 2638 /** Find oldest txnid still referenced. Expects txn->mt_txnid > 0. */ + 2639 static txnid_t + 2640 mdb_find_oldest(MDB_txn *txn) + 2641 { + 2642 int i; + 2643 txnid_t mr, oldest = txn->mt_txnid - 1; + 2644 if (txn->mt_env->me_txns) { + 2645 MDB_reader *r = txn->mt_env->me_txns->mti_readers; + 2646 for (i = txn->mt_env->me_txns->mti_numreaders; --i >= 0; ) { + 2647 if (r[i].mr_pid) { + 2648 mr = r[i].mr_txnid; + 2649 if (oldest > mr) + 2650 oldest = mr; + 2651 } + 2652 } + 2653 } + 2654 return oldest; + 2655 } +``` -Search is the standard descent — walk from the root, binary-search each -page, follow the child pointer. Splits promote the median key upward, -cascading toward the root when a parent is also full. +Line 2650 is the whole GC policy: a `min` over every reader slot with a live +pid. One process that opened a read transaction and forgot to close it holds +`mr_txnid` at its snapshot forever, so line 2808 breaks on the very first +freeDB record, and *every* page version since that snapshot stays unreusable. +The file grows without bound. (This is the infamous LMDB long-lived-reader +footgun; the reference `cow_btree` has the identical failure as `Arc`-pinned +snapshots — an unreleased handle keeps every superseded node alive.) + +Note also `mdb_find_oldest` costs an O(`mti_numreaders`) scan and the result is +cached in `env->me_pgoldest` (:2805), refreshed at most once per allocation +(`found_old` at :2803). + +Why it matters: LMDB has no compaction and no vacuum. This freed-by-txn +bookkeeping is the *only* thing standing between COW and unbounded growth, and +one forgotten read transaction defeats it. + +### Step 7 — search, split and delete: COW pays for simpler page code + +> **In:** everything above — a single-writer transaction, a path already being +> copied, and no in-page free list to maintain. +> **Out:** the reason LMDB's page code is a fraction of `btree.c`'s size, and +> the one place it is *not* simpler. + +Search is the standard descent: `mdb_page_search` (mdb.c:7535) walks root to +leaf, calling `mdb_node_search` (mdb.c:6689) to binary-search each page. + +Splits promote the median key: `mdb_page_split` (mdb.c:10662) allocates one +right sibling at :10688 and chooses `split_indx = (nkeys+1) / 2` at +**:10742**. There is one fast path, and it is opt-in rather than detected — +`if (nflags & MDB_APPEND)` at :10735 sets `split_indx = newindx; nkeys = 0;`, +so an append puts the new key alone into the fresh sibling and moves nothing. +(SQLite detects the same case itself, in `balance_quick`; see +[`reading-sqlite-btree.md`](reading-sqlite-btree.md).) + +There is no sibling redistribution *on split*, and Step 2 is the reason: the +root-to-leaf path is being copied anyway, so "redistribute in place to avoid +dirtying a neighbour" saves nothing that has not already been spent. + +Nor is there any free-space structure inside a page. A **freeblock chain** — +SQLite's linked list of reusable holes threaded through the dead bytes — has no +counterpart here, because `mdb_node_del` (mdb.c:9434) compacts immediately: + +```c +// libraries/liblmdb/mdb.c — the end of mdb_node_del, 9467-9481 + 9467 ptr = MP_PTRS(mp)[indx]; + 9468 for (i = j = 0; i < numkeys; i++) { + 9469 if (i != indx) { + 9470 MP_PTRS(mp)[j] = MP_PTRS(mp)[i]; + 9471 if (MP_PTRS(mp)[i] < ptr) + 9472 MP_PTRS(mp)[j] += sz; + 9473 j++; + 9474 } + 9475 } + 9476 + 9477 base = (char *)mp + MP_UPPER(mp) + PAGEBASE; + 9478 memmove(base + sz, base, ptr - MP_UPPER(mp)); + 9479 + 9480 MP_LOWER(mp) -= sizeof(indx_t); + 9481 MP_UPPER(mp) += sz; +``` -This is deliberately *simpler* than SQLite's 3-sibling balance: COW means the -root-to-leaf path is being rewritten anyway, so there's no "redistribute in -place to avoid dirtying neighbors" incentive — and with no free-space -management inside pages (no freeblocks; pages are rebuilt append-style), -there's nothing to compact either. +Line 9478 is the trade: one `memmove` slides every cell below the deleted one +up by `sz` bytes, and 9470-9472 fix the surviving pointers by the same amount. +A page therefore has exactly one free region, between `MP_LOWER` and +`MP_UPPER` — no chain, no fragment counter, no `defragmentPage`. The cost is +paid at delete time instead of amortised, which is affordable *because* the +page was going to be copied wholesale anyway (Step 2). + +The one place LMDB is not simpler is the delete side. `mdb_rebalance` +(mdb.c:10297) *does* redistribute with a neighbour: a leaf below +`FILL_THRESHOLD` — 250 tenths of a percent, i.e. **25% full** +(mdb.c:1130-1136) — or with fewer than `minkeys` entries triggers +`mdb_node_move` (called at :10457) or a merge. So "COW makes redistribution +pointless" holds for splits and not for underflow: emptiness is a property of +the *tree*, not of the write path, and no amount of path copying fixes it. + +Why it matters: Step 4 bought recovery with write amplification; this step +shows the same purchase bought page-format simplicity. Both are the same +decision — "the path is being rewritten anyway" — cashed in twice. ## Where each step lives in the code -One file: `mdb.c`, 12,846 lines. Read it as a design, skim the code — the -`MDB_meta` comment and the reader table carry the whole model. - -- **Step 1 — the mmap**: `mdb_env_map` — mdb.c:5040: one big `PROT_READ` - mmap; writes go through `pwrite` (default) or a writable map with - `MDB_WRITEMAP` (:5097). -- **Step 2 — COW**: `mdb_page_touch` — mdb.c:3015: first write to a clean - page in a txn copies it to a fresh page number; the parent's child pointer - is updated (parent was touched first — the descent touches top-down). - Dirty pages tracked in `mt_u.dirty_list` (sorted ID list; insert at - `mdb_page_dirty` :2670) — flushed sequentially at commit. -- **Step 3 — commit protocol**: two meta pages at file offsets 0 and 1; txn N - writes meta `N % 2` (comment mdb.c:1356, `MDB_meta` struct :1358). - `mdb_txn_commit` → `mdb_page_flush` (write dirty pages) → fsync → - `mdb_env_write_meta` (mdb.c:4847, slot `txnid & 1` at :4863) → fsync. - Recovery: `mdb_env_pick_meta`. -- **Step 4 — readers**: read txn setup in `mdb_txn_renew0` (mdb.c:3285) picks - the newest meta (`mdb_env_pick_meta` :3296), records its txnid in a reader - slot — `MDB_reader` struct :869, one slot per reader in a shared lock file, - holding a frozen `mr_txnid`. -- **Step 5 — page reuse**: freelist database `FREE_DBI` (mdb.c:1345); - `mdb_page_alloc` (mdb.c:2693) reuses freed pages only if freed by a txn - older than the oldest active reader (`mdb_find_oldest` :2640 scans the - reader table `mti_readers`). -- **Step 6 — search/split (skim)**: `mdb_page_search` :7535 → - `mdb_node_search` :6689 (binary search per page); `mdb_page_split` :10662 — - median promotion, cascading up. +One file, `libraries/liblmdb/mdb.c`, 12,846 lines at `704dc70`. Read it as a +design and skim the code; the `MDB_meta` comment and the reader table carry the +whole model. + +| Lines | What | Step | +|---|---|---| +| 788 | `DEFAULT_MAPSIZE` 1,048,576 — the ceiling you must raise | 1 | +| 869-879 | `MDB_reader` — one cache-line-padded slot, `mr_txnid` + `mr_pid` | 5 | +| 1130-1136 | `PAGEFILL` and `FILL_THRESHOLD` 250 (= 25%) | 7 | +| 1327-1336 | `MDB_db` — 48 bytes, incl. `md_depth` and `md_root` | 2, 4 | +| 1345 | `FREE_DBI` 0 — the freelist is database 0 | 6 | +| 1351-1358 | `NUM_METAS` 2 and the comment: "Transaction N writes meta page #(N % 2)" | 4 | +| 2638-2655 | `mdb_find_oldest` — min over live reader slots | 6 | +| 2659-2673 | `mdb_page_dirty` — sorted insert at :2670 | 3 | +| 2693 | `mdb_page_alloc` — the freeDB scan; the `oldest` gate at :2800-2810 and :2818-2826 | 6 | +| 3015-3074 | `mdb_page_touch` — COW; parent repointed at :3041, old pgno freed at :3036, copy at :3071 | 2, 3 | +| 3285 | `mdb_txn_renew0` — reader setup; no-lock path :3296, normal path :3349-3358 | 5 | +| 3858 | `mdb_freelist_save` — `mt_free_pgs` → `FREE_DBI`, called at :4571 | 3, 6 | +| 4105-4240 | `mdb_page_flush` — `pwrite` (:4237) / `pwritev` (:4240) over the sorted dirty list | 3, 4 | +| 4571-4599 | the commit ordering: freelist → pages → `mdb_env_sync0` → meta | 4 | +| 4673-4753 | `mdb_env_read_header` — two `pread`s (:4718), larger txnid wins (:4749) | 4 | +| 4847-4982 | `mdb_env_write_meta` — toggle `& 1` at :4863, partial write at :4918-4920, `me_mfd` at :4927, Apple fdatasync at :4964 | 4 | +| 4985-4995 | `mdb_env_pick_meta` — recovery, in one comparison | 4 | +| 5040-5118 | `mdb_env_map` — `PROT_READ` at :5096, `me_mapsize` at :5117 | 1 | +| 5318 | `MDB_O_META = O_WRONLY|MDB_DSYNC` — why barrier two needs no fsync | 4 | +| 5520, 5527 | `me_psize` = OS page size (new) or `meta.mm_psize` (existing) | 1 | +| 6689 | `mdb_node_search` — binary search within a page | 7 | +| 7535 | `mdb_page_search` — the root-to-leaf descent | 7 | +| 9434-9482 | `mdb_node_del` — immediate compaction, `memmove` at :9478 | 7 | +| 10297-10457 | `mdb_rebalance` — the redistribution that *does* exist, on underflow | 7 | +| 10662-10742 | `mdb_page_split` — new right sibling :10688, `MDB_APPEND` fast path :10735, median :10742 | 7 | + +Suggested route: the `MDB_meta` comment (1354-1356) → `mdb_page_touch` (3015) +→ the commit ordering (4571-4599) → `mdb_env_write_meta` (4847) → +`mdb_env_pick_meta` (4990) → `mdb_find_oldest` (2640) → `mdb_node_del` (9434). +Seven stops, and you have the design. ## Questions to answer in notes.md 1. Why does LMDB's split not bother with SQLite-style sibling redistribution? - (COW already dirties the path; also no freeblocks — append-style page builds.) -2. Double meta + fsync ordering: which of the two fsyncs could you drop, under - what hardware assumption, and what breaks on consumer SSDs? -3. Price a 1-key commit at tree height 4, 4KB pages: bytes written for LMDB vs - a WAL engine (≈ record + fsync). When does LMDB's model win anyway? - (Read-heavy, batch-committed writes.) + (COW already dirties the path; also no freeblocks — `mdb_node_del` :9478 + compacts on the spot.) Then say why `mdb_rebalance` (:10297) exists anyway, + given the same argument. +2. Double meta + the two barriers: which of them could you drop, under what + hardware assumption, and what breaks on consumer SSDs? Name the flag that + drops each (`lmdb.h:354`, `:358`) and say what the 120-byte meta write at + :4918-4920 assumes about sector atomicity. +3. Price a 1-key commit at tree depth 4, 4 KB pages: bytes written for LMDB vs + a WAL engine (≈ record + fsync). Redo it for the 16 KB pages an Apple + Silicon machine would give you (Step 1, mdb.c:5520). When does LMDB's model + win anyway? (Read-heavy, batch-committed writes.) +4. `mdb_txn_renew0` publishes `r->mr_txnid` at :3350 *before* reading the meta + at :3356. Construct the interleaving that would lose data if those two lines + were swapped, using `mdb_find_oldest` (:2646) and the `oldest <= last` gate + at :2808. ## Done when -You can narrate a crash at any point in the commit sequence and say which root -survives, and you can state the reader-pins-pages problem and its capstone twin. +Answer each before unfolding it. + +- [ ] You can narrate a crash at any point in the commit sequence and say which root survives. + +
Answer + + Three windows, from the ordering at mdb.c:4571-4599. A crash *before* the + data pages are durable (during `mdb_page_flush` at :4583, or before + `mdb_env_sync0` at :4590 returns) leaves new pages that no meta references — + garbage in the unallocated tail, and meta `(N-1) % 2` still points at a + complete tree. A crash *between* the barrier at :4590 and the meta write at + :4598 leaves the new pages fully durable but unreachable: the same outcome, + because reachability runs through the meta. A crash *during* the meta write + itself is the interesting one, and it is safe by the toggle at :4863 — + `txn->mt_txnid & 1` selects the slot the *previous* commit did not use, so + the other slot is untouched by construction. + + Recovery then reads both slots (`mdb_env_read_header`, two `pread`s at + :4718), rejects anything whose `P_META` flag (:4730), magic (:4738) or + version (:4743) is wrong, and keeps the larger `mm_txnid` (:4749). Note what + is *not* checked: there is no checksum. The meta write is only 120 bytes + (:4918-4920, `offsetof(MDB_meta, mm_mapsize)` to the end of the struct), and + the safety argument is that a sub-sector write is atomic on the device. That + is the assumption question 2 asks you to attack. + +
+ +- [ ] You can state where the second durability barrier actually is, and why it is not a second `fsync` on Linux. + +
Answer + + It is the `pwrite` at mdb.c:4937 itself. Line 4927 chooses the descriptor: + `mfd = (flags & (MDB_NOSYNC|MDB_NOMETASYNC)) ? env->me_fd : env->me_mfd`, + and `me_mfd` is opened `MDB_O_META = O_WRONLY|MDB_DSYNC` (mdb.c:5318). The + comment at 4923-4925 states the intent: "me_mfd goes to the same file as + me_fd, but writing to it also syncs to disk. Avoids a separate fdatasync() + call." + + The first barrier *is* a conventional one — `mdb_env_sync0` at :4590, guarded + by `MDB_NOSYNC`. The exception is Apple: :4964-4968 issues an explicit + `MDB_FDATASYNC(env->me_mfd)` after the meta write, because Darwin's `O_DSYNC` + does not flush the drive cache. So on macOS there really are two syscalls; on + Linux there is one `fsync` plus one synchronous write. Either way there are + two *barriers*, separately disableable via `MDB_NOSYNC` and `MDB_NOMETASYNC` + (`lmdb.h:354`, `:358`) — which is the knob question 2 is about. + +
+ +- [ ] You can state the reader-pins-pages problem, name the two lines that cause it, and give its capstone twin. + +
Answer + + A read transaction freezes a txnid in its slot (`mdb_txn_renew0` :3350) and + never releases it until the transaction ends. `mdb_find_oldest` (mdb.c:2640) + takes a minimum over every slot with a live pid — the comparison is line + 2650 — and `mdb_page_alloc` refuses to consume any freeDB record whose txnid + key has reached that minimum: `if (oldest <= last) break;` at line **2808** + (repeated at :2824). + + So a process that opens a read transaction and forgets it holds `oldest` + fixed, the `break` fires on the first record, and no freed page from any + later transaction is ever reused. Since every write copies its whole + root-to-leaf path (Step 2), the file grows by roughly `depth` pages per + commit forever. LMDB has no compaction or vacuum to recover from this — the + fix is `mdb_reader_check` or ending the transaction. + + The capstone twin is the reference `cow_btree`: `Arc`-pinned snapshots have + the identical shape, with the refcount playing the reader slot's role. A + retained snapshot handle keeps every superseded node reachable, and memory + grows for exactly the same reason. + +
+ +- [ ] You can price a 1-key commit and say what LMDB bought with those bytes. + +
Answer + + Assume 4096-byte pages and a tree of depth 4 (`md_depth`, mdb.c:1330). + `mdb_page_touch` copies one page per level, so the path costs + 4 × 4096 = 16,384 bytes. `mdb_freelist_save` (:3858, called at :4571) must + record the four freed page numbers into `FREE_DBI`, which is itself a + copy-on-write B-tree, so add at least one more page: 4,096 bytes. The meta + write adds 120 bytes (:4918-4920). Total ≥ 20,600 bytes to change one key. + A WAL engine writes one ~100-byte record and one `fsync`, so this is ≥ 206× + the bytes. On Apple Silicon, where `me_psize` is the 16,384-byte OS page + (:5520), multiply the page terms by four. + + What it bought: recovery is `mdb_env_pick_meta` (:4990), one integer + comparison at line 4993 — no log to replay, no undo, no repair utility, and + a reader that started before the commit still walks a complete, consistent + tree because none of its pages were touched. It also bought page-format + simplicity (Step 7's single free region, no freeblock chain) and lock-free + reads (Step 5). The model wins when reads dominate and writes are batched: + the 20,600 bytes are per *commit*, not per key, so a transaction updating a + thousand keys amortises the path copies over all of them. + +
+ +- [ ] You can draw the LMDB commit diagram from memory and say what is *not* in it. + +
Answer + + In it: two meta pages at page numbers 0 and 1 (mdb.c:1356), a tree hanging + off each root, a fresh root-to-leaf path written to new page numbers, then + `pages → fsync (:4590) → meta[txnid & 1] (:4863) → O_DSYNC pwrite (:4937)`. + + Not in it, and this is the point: no write-ahead log, no undo log, no + checkpoint, no page LSNs, no torn-page detection, no per-page checksum, and + no in-page freeblock chain (Step 7 — `mdb_node_del` compacts with one + `memmove` at :9478 instead). Also absent, and easy to forget: any bound on + file growth. The freelist database (`FREE_DBI`, :1345) is the only + reclamation mechanism, and Step 6 showed a single stalled reader disables it + at line 2808. + +
## References **Code** -- [LMDB](https://github.com/LMDB/lmdb) `libraries/liblmdb/mdb.c` - (12,846 lines, one file; local clone at `~/repos/lmdb`) — read it as a - design, skim the code; the `MDB_meta` comment (:1356) and the reader - table (`MDB_reader` :869) carry the whole model +- [LMDB](https://github.com/LMDB/lmdb) `libraries/liblmdb/mdb.c` — one file, + 12,846 lines, pinned at `LMDB/lmdb@704dc70`. Read it as a design, skim the + code; the `MDB_meta` comment (:1354-1356) and the reader table + (`MDB_reader` :869) carry the whole model. +- `libraries/liblmdb/lmdb.h:344-370` — the environment flag block: + `MDB_NOSYNC` (:354), `MDB_NOMETASYNC` (:358), `MDB_WRITEMAP` (:360), + `MDB_NOLOCK` (:366). Each one names a guarantee this chapter's steps rely + on. + +| File | Lines | What | +|---|---|---| +| `mdb.c` | 1351-1358 | `NUM_METAS` 2; "Transaction N writes meta page #(N % 2)" | +| `mdb.c` | 3024-3044 | `mdb_page_touch` — COW, parent repointed at :3041 | +| `mdb.c` | 3036 | the old page number joins `mt_free_pgs` — Step 3's fork | +| `mdb.c` | 4571-4599 | the commit ordering, all four events | +| `mdb.c` | 4918-4937 | the 120-byte meta write, through the `O_DSYNC` descriptor | +| `mdb.c` | 4985-4995 | `mdb_env_pick_meta` — recovery in one comparison | +| `mdb.c` | 2638-2655 | `mdb_find_oldest` — the reader-table minimum | +| `mdb.c` | 2800-2810 | the `oldest <= last` gate that stalled readers jam | +| `mdb.c` | 9467-9481 | `mdb_node_del` — why LMDB has no freeblock chain | + +**In this curriculum** +- [`reading-sqlite-btree.md`](reading-sqlite-btree.md) — the opposite design: + freeblocks, three balance paths, and a WAL underneath. +- [README.md](README.md) §4 — the commit diagram to reproduce from memory, and + the M3 comparison against the reference `cow_btree`. diff --git a/topics/03-btree-internals/reading-sqlite-btree.md b/topics/03-btree-internals/reading-sqlite-btree.md index 933b23e..d32c651 100644 --- a/topics/03-btree-internals/reading-sqlite-btree.md +++ b/topics/03-btree-internals/reading-sqlite-btree.md @@ -2,180 +2,614 @@ You already know the format from turso — this guided skim (2 h) reads **the original** for the parts turso simplified and for comments that carry two -decades of production experience: the balance_quick fast path, the "25% -faster" right-bias tweak, pointer maps, predecessor-swap deletes. This -chapter builds those production tricks one step at a time, then hands you the -reading route — don't read its 11,633 lines linearly. +decades of production experience: the `balance_quick` fast path, the measured +"about 25% faster" tweak, pointer maps, predecessor-swap deletes. This chapter +builds those production tricks one step at a time — the parsed page, the +descent, in-page free space, the overflow-cell dodge, the three balance paths, +the reverse index, and the delete that forks into two — then hands you the +reading route. Don't read its 11,633 lines linearly. + +Every anchor below is SQLite at the commit this repo pins, **`sqlite/sqlite@951de30`** +(confirm with `tools/pinned-source.py ref sqlite`), where `src/btree.c` is +11,633 lines and `src/btreeInt.h` is 746. All arithmetic uses +`SQLITE_DEFAULT_PAGE_SIZE` = 4096 (`src/sqliteLimit.h:214`) with zero reserved +bytes, so *usable size* = *page size* = 4096. ## The problem in one sentence btree.c must make the single most common write on Earth — appending the next -sequential rowid to a table — nearly free, inside **11,633 lines** of C that -also survive every crash, corrupt page, and pathological key distribution -that twenty years on billions of devices can produce; one measured -distribution tweak in its balance code is worth "about 25%" of whole-database -speed. +sequential rowid to a table — cost one new page and one cell copy instead of +rewriting three siblings' worth of cells (114 of them, at 4 KB pages and +100-byte rows), inside 11,633 lines of C that also survive every crash, +corrupt page and pathological key distribution that twenty years on billions +of devices can produce. ## The concepts, step by step ### Step 1 — MemPage: parse the page once, dispatch without branching -`MemPage` is the in-memory representation of one disk page (a fixed-size -4 KB block): the raw bytes plus decoded header fields, built once when the -page enters the cache so no later operation re-parses the header. Two -production tricks live in this struct: - -- `xCellSize` / `xParseCell` are **function pointers** picked once per page - type at init — a table leaf gets the table-leaf parser, an index interior - gets the index-interior parser — so the per-cell inner loop never - re-checks "what kind of page am I?". Devirtualized dispatch, 1994 style. -- `nFree` (the page's free-byte count) is computed **lazily** — held at −1 - until someone actually needs it, because computing it means walking the - freeblock list (Step 3) and most page visits never ask. - -The companion struct `CellInfo` (`nKey`, `pPayload`, `nLocal`, `nSize`) is -what `xParseCell` fills in: one cell (a single key+row entry on the page) -decoded into fields. Why it matters: cell parsing is the innermost loop of -every search, insert, and balance — this is where cycles go. - -### Step 2 — the search path: descend, binary-search, and skip work when hinted - -A lookup descends from the root page, binary-searching each page's sorted -cells to pick the child pointer to follow, until it lands on a leaf. Two -optimizations mark this as production code: - -- `sqlite3BtreeTableMoveto` takes a **bias hint** parameter: a caller that - knows it's appending (rowids arriving in order — the common case) skips - the binary search entirely and probes the rightmost cell first. When - `lwr >= nCell` after the search, the descent takes the page's rightmost - pointer. -- `sqlite3BtreeIndexMoveto` compares full keys, and uses an - `xRecordCompare` **callback specialized per key shape** — the comparator - for "one integer column" is a different function than the general one. - Same devirtualization move as Step 1. - -Why it matters: comparisons are the entire CPU cost of a descent; picking the -specialized comparator once per query instead of branching per comparison is -free money. +> **In:** 4096 raw bytes from the pager. +> **Out:** a `MemPage` whose header fields are decoded, whose per-cell +> operations are already bound to the right function, and whose free-byte +> count is deliberately *not* computed. Every later step reads this struct +> instead of the bytes. + +The bytes have a fixed shape, and btreeInt.h documents it better than any +other open-source file documents anything: + +```c +// sqlite/sqlite src/btreeInt.h — the page layout and header table, 111-134 + 111 ** | file header | 100 bytes. Page 1 only. + 112 ** |----------------| + 113 ** | page header | 8 bytes for leaves. 12 bytes for interior nodes + 114 ** |----------------| + 115 ** | cell pointer | | 2 bytes per cell. Sorted order. + 116 ** | array | | Grows downward + 117 ** | | v + 118 ** |----------------| + 119 ** | unallocated | + 120 ** | space | + 121 ** |----------------| ^ Grows upwards + 122 ** | cell content | | Arbitrary order interspersed with freeblocks. + 123 ** | area | | and free space fragments. + 124 ** |----------------| + 125 ** + 126 ** The page headers looks like this: + 127 ** + 128 ** OFFSET SIZE DESCRIPTION + 129 ** 0 1 Flags. 1: intkey, 2: zerodata, 4: leafdata, 8: leaf + 130 ** 1 2 byte offset to the first freeblock + 131 ** 3 2 number of cells on this page + 132 ** 5 2 first byte of the cell content area + 133 ** 7 1 number of fragmented free bytes + 134 ** 8 4 Right child (the Ptr(N) value). Omitted on leaves. +``` + +A **cell** is one key+payload entry; a **slot** is what a cell costs in total, +its 2-byte pointer (:115) plus its body. Note line 134: the right-child +pointer is 4 bytes at header offset 8, which is why the interior header is +12 bytes and the leaf header 8 — that difference is `MemPage.childPtrSize`, +"0 if leaf==1. 4 if leaf==0" (btreeInt.h:282), and it reappears as a literal +`+4` in Step 7. + +`MemPage` (btreeInt.h:273-304) is the decoded form, built once when the page +enters the cache. Three of its 30 fields carry production decisions: + +```c +// sqlite/sqlite src/btreeInt.h — inside struct MemPage, 288-303 + 288 int nFree; /* Number of free bytes on the page. -1 for unknown */ + // ... 289-292: nCell, maskPage, aiOvfl[4] ... + 293 u8 *apOvfl[4]; /* Pointers to the body of overflow cells */ + // ... 294-301: pBt, aData, aDataEnd, aCellIdx, aDataOfst, pDbPage ... + 302 u16 (*xCellSize)(MemPage*,u8*); /* cellSizePtr method */ + 303 void (*xParseCell)(MemPage*,u8*,CellInfo*); /* btreeParseCell method */ +``` + +- **Lines 302-303 are devirtualized dispatch, 1994 style.** `xCellSize` and + `xParseCell` are function pointers chosen once per page at init — a table + leaf gets the table-leaf parser, an index interior gets the index-interior + parser — so the per-cell inner loop never re-tests "what kind of page am + I?". A descent parses O(log N × log₂ F) cells; the branch it avoids is the + most-executed branch in the engine. +- **Line 288 is laziness with a stated sentinel.** `nFree` is held at −1 until + someone needs it, because computing it means walking the freeblock chain + (Step 3) and most page visits never ask. `btreeComputeFreeSpace` is called + on demand — you can watch it happen at btree.c:9212 and :9994. +- **Line 293 is Step 4's entire mechanism**, and note the array bound: **four** + overflow cells, not an arbitrary number. + +`CellInfo` (btreeInt.h:480-486) is what `xParseCell` fills in — `nKey`, +`pPayload`, `nPayload`, `nLocal`, `nSize`. The pair `nPayload` (:483) and +`nLocal` (:484) is where a too-big cell forks into a local part and an +overflow chain; the arithmetic of that split belongs to +[`reading-sqlite-file-format.md`](reading-sqlite-file-format.md). + +Why it matters: cell parsing is the innermost loop of every search, insert and +balance. This is where cycles go. + +### Step 2 — the search path: descend, binary-search, bias the first probe + +> **In:** a root page number and a search key. +> **Out:** a leaf page and an index into its cell pointer array — plus a count +> of pages touched, which is the tree's height. + +A lookup descends from the root, binary-searching each page's sorted cell +pointer array to pick the child to follow. Here is the whole loop's skeleton +for a rowid table: + +```c +// sqlite/sqlite src/btree.c — inside sqlite3BtreeTableMoveto, 5917-5968 + 5917 lwr = 0; + 5918 upr = pPage->nCell-1; + 5919 assert( biasRight==0 || biasRight==1 ); + 5920 idx = upr>>(1-biasRight); /* idx = biasRight ? upr : (lwr+upr)/2; */ + 5921 for(;;){ + // ... 5922-5951: decode this cell's rowid, compare, narrow lwr/upr, + // and return early on an exact hit at a leaf ... + 5952 assert( lwr+upr>=0 ); + 5953 idx = (lwr+upr)>>1; /* idx = (lwr+upr)/2; */ + 5954 } + // ... 5955-5963: if this is a leaf, we are done ... + 5964 moveto_table_next_layer: + 5965 if( lwr>=pPage->nCell ){ + 5966 chldPg = get4byte(&pPage->aData[pPage->hdrOffset+8]); + 5967 }else{ + 5968 chldPg = get4byte(findCell(pPage, lwr)); +``` + +Two production marks, and the first one is narrower than it is usually +described: + +- **Line 5920 is the bias hint, and it biases exactly one probe.** With + `biasRight = 1` the first `idx` is `upr` — the rightmost cell — instead of + the midpoint. Every *subsequent* probe is the ordinary midpoint, line 5953. + So an appending caller does not skip the binary search; it wins the common + case in one comparison and otherwise pays the usual log₂. (The parameter is + declared at :5840, "If true, bias the search to the high end".) +- **Lines 5965-5966 are the rightmost pointer.** When the key sorts past every + cell, the descent follows the 4-byte right child at header offset 8 — + the field from Step 1's line 134. + +`sqlite3BtreeIndexMoveto` (btree.c:6068) is the index counterpart; it compares +full records through an `xRecordCompare` callback specialized per key shape, +the same devirtualization move as Step 1. + +Now price the descent, because "height" is the currency. Symbols: `P` = page +size, `H` = page header bytes, `F` = fanout (children per interior page), +`L` = entries per leaf page, `N` = row count. + +``` +P = 4096, zero reserved bytes (SQLITE_DEFAULT_PAGE_SIZE, sqliteLimit.h:214) + +table leaf H = 8 (btreeInt.h:113) + slot = 2 (cell pointer) + 1 (payload-size varint, 100 fits in one byte) + + 3 (rowid varint, 10^6 needs 3) + 100 (payload) = 106 B + L = floor((4096 - 8) / 106) = floor(4088 / 106) = 38 rows/leaf + +table interior H = 12 (the extra 4 = right child, btreeInt.h:134) + slot = 2 (cell pointer) + 4 (child pgno) + 3 (rowid varint) = 9 B + F = floor((4096 - 12) / 9) = floor(4084 / 9) = 453 children/page + +height at N = 10^6: + leaves = ceil(N / L) = ceil(10^6 / 38) = 26,316 + interior levels = ceil(log_F(N/L)) = ceil(log_453(26,315.8)) + = ceil(ln 26,315.8 / ln 453) = ceil(10.178 / 6.116) + = ceil(1.6642) = 2 + total pages touched per lookup = 2 + 1 leaf = 3 + +at N = 10^9 the rowid varint grows to 5 B, so F = floor(4084/11) = 371: + leaves = ceil(10^9 / 38) = 26,315,790 + ceil(log_371(26,315,790)) = ceil(17.086 / 5.916) = ceil(2.8879) = 3 + total pages touched = 4 +``` + +For an *index* b-tree with a 16-byte key the slot is +`2 + 4 + 1 (payload-size varint) + 16 = 23`, so `F = floor(4084/23) = 177` — +and with the textbook 8-byte child pointer instead of SQLite's 4-byte one it +would be `floor(4084/24) = 170`. Fanout is not delicate: halving the pointer +width moved it 4%. + +**A warning this topic has measured.** Height is the count of pages a lookup +*touches*; it is not the time a lookup takes. This topic's own benchmark holds +height at 3 across 10⁶ → 4×10⁶ keys and still watches lookups climb from +862 ns to 1101 ns — see the measured block in [README.md](README.md) and the +ladder in [notes.md](notes.md). Height is a step function; latency is not, +because what a touch *costs* depends on cache residency. Use the arithmetic +above to predict pages read, and nothing else. + +Why it matters: comparisons are the entire CPU cost of a descent, and picking +the specialized comparator once per query instead of branching per comparison +is free money. ### Step 3 — free space within a page: freeblocks, merged on free -When a cell is deleted, its bytes become a **freeblock** — a hole inside the -page, threaded into a linked list (each freeblock stores a 2-byte pointer to -the next hole and its own 2-byte size) so later inserts can reuse the space. +> **In:** a page with cells deleted out of it over time, and a request for +> `nByte` contiguous bytes. +> **Out:** an offset — reached by one of three routes, in a fixed order — or +> a failure that hands control to Step 4. + +Deleting a cell turns its bytes into a **freeblock**: a hole inside the cell +content area, threaded into a singly linked list so later inserts can reuse +it. The format is four bytes of self-description, and it has a floor: + +```c +// sqlite/sqlite src/btreeInt.h — the freeblock and fragment rules, 152-163 + 152 ** Unused space within the cell content area is collected into a linked list of + 153 ** freeblocks. Each freeblock is at least 4 bytes in size. The byte offset + 154 ** to the first freeblock is given in the header. Freeblocks occur in + 155 ** increasing order. Because a freeblock must be at least 4 bytes in size, + 156 ** any group of 3 or fewer unused bytes in the cell content area cannot + 157 ** exist on the freeblock chain. A group of 3 or fewer free bytes is called + 158 ** a fragment. The total number of bytes in all fragments is recorded. + 159 ** in the page header at offset 7. + 160 ** + 161 ** SIZE DESCRIPTION + 162 ** 2 Byte offset of the next freeblock + 163 ** 2 Bytes in this freeblock +``` -- `allocateSpace` satisfies an insert from the freeblock list, then from the - gap between the pointer array and cell content, and only then compacts. -- `freeSpace` **merges adjacent freeblocks** as it inserts the new hole into - the (address-ordered) list — deletes actively fight fragmentation instead - of deferring everything. -- `defragmentPage` is the last resort: rewrite all cells contiguously, reset - the list. +Lines 161-163 explain line 153: a freeblock must hold its own 2-byte next +pointer and 2-byte size, so 4 bytes is the minimum it can describe. Anything +smaller is a **fragment**, unreachable and merely counted — in a *one-byte* +header field (Step 1's line 133), so at most 255 fragment bytes can even be +represented on a page. + +`allocateSpace` (btree.c:1846) then tries three routes, in this order: + +```c +// sqlite/sqlite src/btree.c — the three routes in allocateSpace, 1889-1928 + 1889 if( (data[hdr+2] || data[hdr+1]) && gap+2<=top ){ + 1890 u8 *pSpace = pageFindSlot(pPage, nByte, &rc); + // ... 1891-1903: if a freeblock fit, return its offset ... + 1904 + 1905 /* The request could not be fulfilled using a freelist slot. Check + 1906 ** to see if defragmentation is necessary. + 1907 */ + // ... 1908 ... + 1909 if( gap+2+nByte>top ){ + // ... 1910-1911: asserts ... + 1912 rc = defragmentPage(pPage, MIN(4, pPage->nFree - (2+nByte))); + // ... 1913-1916: recheck top ... + 1917 } + // ... 1918-1924: comment on why the gap allocation is now safe ... + 1925 top -= nByte; + 1926 put2byte(&data[hdr+5], top); + // ... 1927 ... + 1928 *pIdx = top; +``` -Why it matters: this is the machinery that makes delete cheap (unlink a -2-byte pointer, thread a hole) while keeping pages usable for decades of -churn without a vacuum. +Read it as: **freeblock chain first** (:1889-1890, gated on the chain being +non-empty), **compact only if the gap is too small** (:1909-1912), then always +**allocate from the gap** (:1925-1928) between the pointer array and the cell +content. Defrag is the last resort, not a third allocation route. + +`freeSpace` (btree.c:1945) is the other half, and its contract is one line of +comment at :1937 — "Adjacent freeblocks are coalesced." Deletes actively fight +fragmentation as they happen, rather than deferring all of it. + +`defragmentPage` (btree.c:1640) has a fast path worth seeing, because it is +the same instinct as `balance_quick`: at :1674, if the page has at most two +freeblocks and at most `nMaxFrag` fragment bytes, it slides the cells with one +or two `memmove`s (:1693, :1701) and adds a fixed offset to each affected cell +pointer (:1702-1706), instead of rebuilding the page through the temp buffer +at :1712-1740. `allocateSpace` passes `nMaxFrag = MIN(4, ...)` at :1912, so +the fast path only fires on nearly clean pages. + +Why it matters: this machinery is what makes delete cheap — unlink a 2-byte +pointer, thread a hole — while keeping pages usable through decades of churn +with no vacuum. Contrast LMDB, which has no freeblocks at all and pays a full +`memmove` on every delete instead; see +[`reading-lmdb.md`](reading-lmdb.md) Step 7. ### Step 4 — the overflow-cell trick: a page is never physically overfull -When an insert doesn't fit even after Step 3's efforts, SQLite does *not* -grow or reallocate the page — the incoming cell is parked **in memory, -beside the page**, in a small `apOvfl[]` array, and the caller is obligated -to run balance (Step 5) before releasing the page. Balance drains `apOvfl[]` -into its redistribution pool immediately. +> **In:** an insert that Step 3 could not place, even after defragmenting. +> **Out:** a page that is logically overfull but physically valid, plus an +> obligation on the caller to run Step 5 before releasing it. Crucially, the +> on-disk format never learns that overfull pages exist. + +When a cell will not fit, SQLite does not grow the page, reallocate it, or +invent an "overfull" on-disk representation. The incoming cell is parked **in +memory, beside the page**, in `MemPage.apOvfl[4]` (Step 1's line 293), with +`aiOvfl[i]` recording which non-overflow cell it belongs before +(btreeInt.h:291-292). ```rust -// insertCell's trick: a page is never physically overfull +// ILLUSTRATION — not quoted from SQLite; the real code is insertCell, +// sqlite/sqlite src/btree.c:7363, with the array at src/btreeInt.h:293 fn insert_cell(page: &mut MemPage, i: usize, cell: Cell) { - match page.allocate_space(cell.len()) { // freeblocks → gap → defrag - Some(off) => page.write_cell(off, i, &cell), + match page.allocate_space(cell.len()) { // btree.c:1846 — freeblocks, + Some(off) => page.write_cell(off, i, &cell), // defrag, then the gap None => { - page.ap_ovfl.push((i, cell)); // parked IN MEMORY, beside the page - // caller must run balance() before the page is released: the - // balance pool drains ap_ovfl while redistributing ≤3 siblings, - // so the on-disk format never needs an "overfull" representation + page.ap_ovfl.push((i, cell)); // btreeInt.h:293 — 4 slots, + // parked IN MEMORY + // caller must run balance() (btree.c:9162) before the page is + // released; balance drains ap_ovfl while redistributing, so the + // on-disk format never needs an "overfull" representation } } } ``` -Why it matters: the on-disk format never needs an "overfull page" -representation, so every page on disk is always valid — a crash-safety and -simplicity win bought with one tiny in-memory array. +Why it matters: every page on disk is always structurally valid, at every +instant, so a crash can never expose a page shape the reader does not +understand. That crash-safety and simplicity win is bought with one 4-element +in-memory array — and the bound of 4 is itself a statement, since Step 5 is +guaranteed to run before a fifth could be needed. ### Step 5 — balance: read it for the engineering, not the algorithm -**Balance** is what runs when a page overflows (or underflows): pool the -cells of the problem page and its neighbors, redistribute them across -enough pages, push new separator keys to the parent. The `balance()` -dispatcher picks between three production-shaped paths: - -- `balance_quick` — the rightmost-leaf append gets its **own dedicated - path**: just allocate one new leaf on the right and put the new cell there, - touching the minimum possible pages. Sequential inserts are THE common - case — fillseq from topic 1 — so it gets its own code. -- `balance_nonroot` — the general case: pool the overfull page with up to - `NB = 3` siblings and redistribute. Find the comment near :8738: the - right-bias optimization — packing pages fuller on the left so the - rightmost page has room for the *next* append — "makes the database about - 25% faster". A one-line distribution tweak, measured. Topic-0 lesson in - the wild. -- `balance_deeper` — root split: the root's content moves into a new child - and the tree grows *up* by one level, the only operation that increases - height. - -Why it matters: the algorithm is in every textbook; the fast path, the bound -NB=3, and the measured 25% tweak are what two decades of production look like. +> **In:** a page carrying overflow cells from Step 4 (or one left too empty by +> Step 7). +> **Out:** pages that are all physically valid and within the fill rules, and +> new separator keys pushed into the parent — possibly making *it* overfull, +> which is why `balance` is a loop. + +**Balance** pools the cells of the problem page and its neighbours, +redistributes them, and pushes separators up. The `balance()` dispatcher +(btree.c:9162) picks between three paths. + +**`balance_quick` (btree.c:8039)** is the append fast path, and its gate is +five exact conditions: + +```c +// sqlite/sqlite src/btree.c — the balance_quick gate inside balance(), 9216-9222 + 9216 #ifndef SQLITE_OMIT_QUICKBALANCE + 9217 if( pPage->intKeyLeaf + 9218 && pPage->nOverflow==1 + 9219 && pPage->aiOvfl[0]==pPage->nCell + 9220 && pParent->pgno!=1 + 9221 && pParent->nCell==iIdx + 9222 ){ +``` + +Read: a rowid-table leaf (:9217) with exactly one overflow cell (:9218) that +belongs *after* every existing cell (:9219), on a non-root parent (:9220), +and which is that parent's rightmost child (:9221). That is precisely "the +next sequential rowid, appended". The rationale is stated where the function +lives, and it is the argument the previous edition of this chapter attached to +the wrong comment: + +```c +// sqlite/sqlite src/btree.c — the balance_quick header comment, 8022-8037 + 8022 ** Instead of trying to balance the 3 right-most leaf pages, just add + 8023 ** a new page to the right-hand side and put the one new entry in + 8024 ** that page. This leaves the right side of the tree somewhat + 8025 ** unbalanced. But odds are that we will be inserting new entries + 8026 ** at the end soon afterwards so the nearly empty page will quickly + 8027 ** fill up. On average. + // ... 8028-8032: pPage must be the right-most leaf, with one overflow ... + 8033 ** The pSpace buffer is used to store a temporary copy of the divider + 8034 ** cell that will be inserted into pParent. Such a cell consists of a 4 + 8035 ** byte page number followed by a variable length integer. In other + 8036 ** words, at most 13 bytes. Hence the pSpace buffer must be at + 8037 ** least 13 bytes in size. +``` + +Lines 8033-8037 also give you the divider cell's exact budget: 4 bytes of page +number plus a rowid varint of at most 9, so **at most 13 bytes** enter the +parent per split. Price the fast path against the general one at the leaf +geometry of Step 2 (L = 38 rows/leaf): + +``` +balance_quick : allocate 1 page, copy 1 cell, insert ≤13 B into the parent +balance_nonroot : pool NB = 3 siblings ⇒ 3 × 38 = 114 cells re-encoded and + rewritten, plus the parent's dividers rebuilt +saving per append: 114 cell copies → 1 +``` + +**`balance_nonroot` (btree.c:8277)** is the general case: pool the overfull +page with up to `NB = 3` siblings (`#define NB 3` at btree.c:7552, commented +"(NN*2+1): Total pages involved in the balance") and redistribute. Two of its +comments are worth the trip, and they are *different* optimizations: + +```c +// sqlite/sqlite src/btree.c — the measured 25%, and what it is about, 8730-8741 + 8730 /* + 8731 ** Reassign page numbers so that the new pages are in ascending order. + 8732 ** This helps to keep entries in the disk file in order so that a scan + 8733 ** of the table is closer to a linear scan through the file. That in turn + 8734 ** helps the operating system to deliver pages from the disk more rapidly. + 8735 ** + 8736 ** An O(N*N) sort algorithm is used, but since N is never more than NB+2 + 8737 ** (5), that is not a performance concern. + 8738 ** + 8739 ** When NB==3, this one optimization makes the database about 25% faster + 8740 ** for large insertions and deletions. + 8741 */ +``` + +**This is a page-number reassignment, not a fill-bias.** The block that +follows (:8742-8770) is an O(N²) selection sort over at most five pages +(:8747-8751) that renumbers the freshly balanced siblings into ascending page +order, so that a later table scan reads the file closer to sequentially. The +25% is a *physical locality* result — a topic-0 lesson in the wild, and a +sibling of topic 1's fillseq-vs-fillrandom gap. + +The fill bias is a separate thing, twenty lines earlier, and it is explicitly +*not* an optimization: + +```c +// sqlite/sqlite src/btree.c — the packing adjustment in balance_nonroot, 8636-8646 + 8636 /* + 8637 ** The packing computed by the previous block is biased toward the siblings + 8638 ** on the left side (siblings with smaller keys). The left siblings are + 8639 ** always nearly full, while the right-most sibling might be nearly empty. + 8640 ** The next block of code attempts to adjust the packing of siblings to + 8641 ** get a better balance. + 8642 ** + 8643 ** This adjustment is more than an optimization. The packing above might + 8644 ** be so out of balance as to be illegal. For example, the right-most + 8645 ** sibling might be completely empty. This adjustment is not optional. + 8646 */ +``` + +So the left bias is an accident of the greedy first-fit packing loop +(:8580-8634), and the loop at :8647-8688 exists to *correct* it for +correctness, not for speed. Attributing "packs left so the right page has room +for the next append" to the 25% comment fuses two unrelated pieces of code — +the previous edition of this chapter did exactly that, and it was wrong. + +**`balance_deeper` (btree.c:9081)** is the root split: the root's content moves +into a fresh child and the tree grows *up* by one level. It is the only +operation that increases height, it is called at :9190, and the assertion at +:9188 records that it can happen at most once per `balance()` call. + +Why it matters: the algorithm is in every textbook; the five-condition fast +path, the bound `NB = 3`, and the *measured* 25% renumbering are what two +decades of production look like. ### Step 6 — pointer maps: the reverse index turso doesn't have -A **pointer map** is a reverse index — for each page number, which page -points *at* it (its parent, or the overflow page before it) — stored in -dedicated ptrmap pages when auto-vacuum is enabled. +> **In:** the observation that a B-tree has only downward pointers. +> **Out:** a permanent format tax, paid on every page allocation and every +> split, that makes one management operation — auto-vacuum's page relocation — +> possible at all. + +A **pointer map** is a reverse index: for each page number, who points *at* +it. Relocating a page (which vacuum must do to shrink the file) would +otherwise require searching the whole tree for the parent. Entries are five +bytes — a one-byte type plus a four-byte page number — as the offset macro +shows: `PTRMAP_PTROFFSET(pgptrmap, pgno) = 5*(pgno-pgptrmap-1)` +(btreeInt.h:630). The five types are `PTRMAP_ROOTPAGE` … `PTRMAP_BTREE` +(btreeInt.h:664-668), documented at :647-662; note `PTRMAP_OVERFLOW2` (:657), +which chains overflow pages backwards so an overflow page can find its +predecessor. + +The density follows directly: + +```c +// sqlite/sqlite src/btree.c — ptrmapPageno, 1063-1075 + 1063 static Pgno ptrmapPageno(BtShared *pBt, Pgno pgno){ + // ... 1064-1067: locals, mutex assert, and pgno<2 returns 0 ... + 1068 nPagesPerMapPage = (pBt->usableSize/5)+1; + 1069 iPtrMap = (pgno-2)/nPagesPerMapPage; + 1070 ret = (iPtrMap*nPagesPerMapPage) + 2; + 1071 if( ret==PENDING_BYTE_PAGE(pBt) ){ + 1072 ret++; + 1073 } + 1074 return ret; + 1075 } +``` + +Line 1068 is the whole cost model. Evaluate it: + +``` +usable = 4096 +entries per ptrmap page = floor(4096 / 5) = 819 +pages covered by one group = 819 + 1 = 820 +space overhead = 1 / 820 = 0.122 % +``` -B-trees only have downward pointers, so relocating a page (which vacuum must -do to shrink the file) would otherwise require searching the whole tree for -whoever points at it. The cost: one ptrmap page every ~⌊usable/5⌋ pages of -the file. Why it matters: it's a concrete example of paying a permanent -format tax for one management operation — and of why turso hasn't -implemented it (yet). +0.122% of the file is a rounding error. The real price is elsewhere: every +page allocation, every split, and every overflow-chain change must also +*write* its ptrmap page (`ptrmapPut`, btree.c:1087), which turns one dirtied +page into two and gives the balance code an extra failure mode. + +Why it matters: it is a clean example of paying a permanent format tax for one +management operation — and of why turso has not implemented it yet. See +[`reading-turso-btree-deep.md`](reading-turso-btree-deep.md). + +### Step 7 — the fork: one interior delete becomes two page mutations + +> **In:** a delete positioned on an *interior* page. +> **Out:** two independent structural edits — a cell dropped from the interior +> page and a cell promoted out of a leaf — each of which can require its own +> `balance()` call. This is the one place a single logical operation forks into +> two physical ones, and the code is shaped entirely around that. + +An interior cell is not only data; it is also the separator routing searches +between two subtrees. Deleting it therefore cannot just remove it. SQLite's +answer, and its reason, are in the comment: + +```c +// sqlite/sqlite src/btree.c — inside sqlite3BtreeDelete, 9948-9959 + 9948 /* If the page containing the entry to delete is not a leaf page, move + 9949 ** the cursor to the largest entry in the tree that is smaller than + 9950 ** the entry being deleted. This cell will replace the cell being deleted + 9951 ** from the internal node. The 'previous' entry is used for this instead + 9952 ** of the 'next' entry, as the previous entry is always a part of the + 9953 ** sub-tree headed by the child page of the cell being deleted. This makes + 9954 ** balancing the tree following the delete operation easier. */ + 9955 if( !pPage->leaf ){ + 9956 rc = sqlite3BtreePrevious(pCur, 0); + // ... 9957-9958: assert and error check ... + 9959 } +``` -### Step 7 — interior deletes become leaf deletes: the predecessor swap +Lines 9951-9953 are the part textbooks skip: **predecessor rather than +successor**, because the predecessor is guaranteed to live in the subtree +under the child pointer of the very cell being removed. That containment is +what keeps the subsequent rebalancing local. + +The fork then plays out in order: drop the interior cell first (`dropCell` at +:9980), then promote the leaf's *last* cell (`findCell(pLeaf, pLeaf->nCell-1)` +at :10003) into the interior page, then drop it from the leaf: + +```c +// sqlite/sqlite src/btree.c — the promotion, inside sqlite3BtreeDelete, 10003-10013 + 10003 pCell = findCell(pLeaf, pLeaf->nCell-1); + // ... 10004-10010: corruption check, size, temp space, make the leaf writable ... + 10011 rc = insertCell(pPage, iCellIdx, pCell-4, nCell+4, pTmp, n); + // ... 10012 ... + 10013 dropCell(pLeaf, pLeaf->nCell-1, nCell, &rc); +``` -Deleting a key that lives on an *interior* page can't just remove the cell — -that cell is also the separator routing searches between two subtrees. So -SQLite swaps in the key's **predecessor** (the largest key in the left -subtree, always on a leaf), overwriting the interior cell, then deletes the -predecessor from its leaf and rebalances there. +Line 10011 carries a detail worth its own sentence: `pCell-4` and `nCell+4`. +A leaf cell promoted to an interior page **grows by exactly four bytes**, +because interior cells carry a child page number and leaf cells do not — Step +1's `childPtrSize`, "0 if leaf==1. 4 if leaf==0" (btreeInt.h:282). The four +bytes are taken from in front of the cell and filled with `n`, the child pgno. + +Both halves then have to be repaired, so `balance()` can be called twice: + +```c +// sqlite/sqlite src/btree.c — the two balance calls after a delete, 10032-10048 + 10032 assert( pCur->pPage->nOverflow==0 ); + 10033 assert( pCur->pPage->nFree>=0 ); + 10034 if( pCur->pPage->nFree*3<=(int)pCur->pBt->usableSize*2 ){ + 10035 /* Optimization: If the free space is less than 2/3rds of the page, + 10036 ** then balance() will always be a no-op. No need to invoke it. */ + 10037 rc = SQLITE_OK; + 10038 }else{ + 10039 rc = balance(pCur); + 10040 } + 10041 if( rc==SQLITE_OK && pCur->iPage>iCellDepth ){ + // ... 10042-10047: walk the cursor back up to the interior page ... + 10048 rc = balance(pCur); +``` + +Line 10034 states SQLite's underflow threshold in closed form: +`nFree × 3 ≤ usable × 2`, i.e. balance is skipped unless **more than two +thirds of the page is free** — a page under one third full. At usable = 4096 +that is `nFree > 2730`. Compare LMDB, which merges below 25% full +(`FILL_THRESHOLD` 250 tenths of a percent, `mdb.c:1136`); SQLite tolerates +emptier pages before doing structural work. Line 10039 repairs the leaf; line +10048 walks back up and repairs the interior page, but only if the first +balance did not already propagate far enough (:10041). Why it matters: every delete's structural work happens at leaf level, where -balance (Step 5) already knows what to do — one mechanism instead of two. +Step 5 already knows what to do — one mechanism instead of two — and the +price of that reuse is this fork, the two `dropCell`s, and the possibility of +two balances. ## Where each step lives in the code -**Start with btreeInt.h:1–215** — the file-format spec as a comment: page -layout diagram, cell formats, freeblock list, overflow, freelist. This is -the best on-disk-format documentation in open source. Read it entire before -any function. - -- **Step 1**: `MemPage` — btreeInt.h:273–303 (note the `xCellSize` / - `xParseCell` function pointers and lazy `nFree`); `CellInfo` — - btreeInt.h:480–486: `nKey`, `pPayload`, `nLocal`, `nSize`. -- **Step 2**: `sqlite3BtreeTableMoveto` — btree.c:5837–5978. Binary search - :5917–5954; child descent :5965–5971 (`lwr >= nCell` ⇒ rightmost pointer); - the bias-hint parameter. `sqlite3BtreeIndexMoveto` — btree.c:6068–6295 - with its per-key-shape `xRecordCompare` callback. -- **Step 3**: `allocateSpace` — btree.c:1846–1944; `freeSpace` — :1945–2050 - (merges adjacent freeblocks!); `defragmentPage` — :1640–1837. -- **Step 4**: `insertCell` — btree.c:7363–7450 (the `apOvfl[]` parking). -- **Step 5**: `balance()` dispatcher — btree.c:9162–9225; `balance_quick` — - btree.c:8039–8150; `balance_nonroot` — btree.c:8277–8826 with `NB = 3` at - :7552 and the "about 25% faster" comment near :8738; `balance_deeper` — - btree.c:9081. -- **Step 6**: pointer maps (auto-vacuum) — btreeInt.h:653–668, - btree.c:1098–1170. -- **Step 7**: delete — btree.c:9873–10050 (:9954 leaf check, :9956 - predecessor fetch). +**Start with btreeInt.h:1-215** — the file-format spec as a comment: page +layout diagram, header table, cell formats, freeblock list, overflow, freelist. +It is the best on-disk-format documentation in open source. Read it entire +before any function. + +| File | Lines | What | Step | +|---|---|---|---| +| `sqliteLimit.h` | 214 | `SQLITE_DEFAULT_PAGE_SIZE 4096` | all | +| `btreeInt.h` | 110-134 | page layout + the 8/12-byte header table | 1 | +| `btreeInt.h` | 152-163 | freeblocks ≥ 4 B, fragments ≤ 3 B | 3 | +| `btreeInt.h` | 273-304 | `MemPage`: `nFree` −1 at :288, `apOvfl[4]` at :293, `xCellSize`/`xParseCell` at :302-303, `childPtrSize` at :282 | 1, 4, 7 | +| `btreeInt.h` | 480-486 | `CellInfo`: `nKey`, `pPayload`, `nPayload`, `nLocal`, `nSize` | 1 | +| `btreeInt.h` | 630, 647-668 | `PTRMAP_PTROFFSET` (5 B/entry) and the five entry types | 6 | +| `btree.c` | 1063-1075 | `ptrmapPageno` — the density formula at :1068 | 6 | +| `btree.c` | 1087, 1146 | `ptrmapPut`, `ptrmapGet` | 6 | +| `btree.c` | 1640 | `defragmentPage`; the ≤2-freeblock fast path at :1674-1708 | 3 | +| `btree.c` | 1774 | `pageFindSlot` — the freeblock-chain search | 3 | +| `btree.c` | 1846-1930 | `allocateSpace` — freeblocks :1889, defrag :1912, gap :1925 | 3 | +| `btree.c` | 1945 | `freeSpace` — "Adjacent freeblocks are coalesced" (:1937) | 3 | +| `btree.c` | 5837 | `sqlite3BtreeTableMoveto`; `biasRight` declared :5840, used :5920; midpoint :5953; right child :5965-5966 | 2 | +| `btree.c` | 6068 | `sqlite3BtreeIndexMoveto` — `xRecordCompare` per key shape | 2 | +| `btree.c` | 7106 | `fillInCell` — builds the overflow chain before insertion | 4 | +| `btree.c` | 7363 | `insertCell` — the `apOvfl[]` parking | 4 | +| `btree.c` | 7552 | `#define NB 3` | 5 | +| `btree.c` | 8022-8037 | `balance_quick`'s rationale and the ≤13-byte divider | 5 | +| `btree.c` | 8039 | `balance_quick` | 5 | +| `btree.c` | 8277 | `balance_nonroot` | 5 | +| `btree.c` | 8636-8646 | the left-packing bias and why correcting it is "not optional" | 5 | +| `btree.c` | 8730-8741 | the page renumbering that is "about 25% faster" | 5 | +| `btree.c` | 9081 | `balance_deeper` — the only height increase | 5 | +| `btree.c` | 9162 | `balance()` — the dispatcher loop; quick-balance gate :9216-9222 | 5 | +| `btree.c` | 9873 | `sqlite3BtreeDelete` | 7 | +| `btree.c` | 9948-9959 | why the *predecessor*, not the successor | 7 | +| `btree.c` | 10003-10013 | the promotion, growing the cell by 4 bytes at :10011 | 7 | +| `btree.c` | 10034-10048 | the ⅔-free skip, and the two `balance()` calls | 7 | ## Questions to answer in notes.md @@ -183,22 +617,179 @@ any function. inserted into the page. What crash-safety property makes that ordering safe? (Pages only become durable at commit via pager/WAL — nothing here is.) 2. Why does `balance_quick` exist when `balance_nonroot` handles the same case? - Estimate the work saved for a fillseq insert (pages touched, cells copied). -3. SQLite computes `nFree` lazily and validates cells only under - `SQLITE_DEBUG`. What does that say about where btree.c sits on the + Estimate the work saved for a fillseq insert (pages touched, cells copied), + then check your estimate against Step 5's `114 → 1` and against the + five-condition gate at :9216-9222 — which of the five would a `fillrandom` + workload violate first? +3. SQLite computes `nFree` lazily (btreeInt.h:288) and validates cells only + under `SQLITE_DEBUG`. What does that say about where btree.c sits on the trust-the-page-vs-verify spectrum, and what's the corruption story? (`PRAGMA integrity_check` exists for a reason.) +4. The 25% comment at :8739 is about page *renumbering* (:8730-8734), not about + packing. Which topic-1 measurement does that make it a sibling of, and what + would you expect the 25% to become on an NVMe drive where sequential and + random reads differ by far less than on the 2004 hardware it was measured + on? ## Done when -You can explain why NB=3 (bounded work per split, adjacent redistribution beats -cascading splits) and name the two fast paths (bias hint, balance_quick) that -serve sequential inserts. +Answer each before unfolding it. + +- [ ] You can explain why `NB = 3`, and say what bounds it buys. + +
Answer + + `#define NB 3` at btree.c:7552, commented "(NN*2+1): Total pages involved in + the balance" — `NN = 1` sibling on each side, plus the page itself. It bounds + the work per split to a constant: at most 3 sibling pages are read, pooled + and rewritten, at most `NB+2 = 5` pages are renumbered (the comment at + :8736-8737 relies on that bound to justify an O(N²) sort), and the parent + gains at most a bounded number of dividers. + + What it buys is that adjacent redistribution usually *avoids* a split + entirely — the cells simply spread across three pages instead of two — so + the tree grows in height far less often than a naive "split at 100% full" + scheme, without the unbounded cascade that pooling *all* siblings would + cause. The trade is fill factor: three-way redistribution leaves pages + fuller than a plain split does, which is why an append run would degrade + under it, which is why Step 5's fast path exists. + +
+ +- [ ] You can name the two fast paths that serve sequential inserts, and say where each one lives. + +
Answer + + (1) The **bias hint**: `biasRight` (declared btree.c:5840, used at :5920, + `idx = upr>>(1-biasRight)`). It makes the *first* binary-search probe the + rightmost cell rather than the midpoint, so an append finds its position in + one comparison. Note the narrow scope — every later probe is the ordinary + midpoint at :5953, so this does not skip the search. + + (2) **`balance_quick`** (btree.c:8039), gated by five conditions at + :9216-9222: rowid-table leaf, exactly one overflow cell, the overflow cell + sorts after every existing cell, the parent is not page 1, and the page is + the parent's rightmost child. It allocates one page, puts the single new + cell there, and pushes a ≤13-byte divider (:8033-8037) into the parent — + instead of re-encoding ~114 cells across three siblings. Its own comment + (:8022-8027) admits the tree is left "somewhat unbalanced" and bets that the + nearly empty page will fill up: "On average." + +
+ +- [ ] You can say what the "about 25% faster" comment is actually about, and what it is *not* about. + +
Answer + + It is at btree.c:8739 and it belongs to the block at :8730-8741, which + **reassigns page numbers so the freshly balanced siblings end up in + ascending order** — an O(N²) selection sort over at most 5 pages + (:8747-8751) — "so that a scan of the table is closer to a linear scan + through the file" (:8732-8734). It is a physical-locality optimization, + the same phenomenon topic 1 measures as fillseq vs fillrandom. + + It is *not* about packing pages fuller on the left, and it is not about + leaving room for the next append. The left-packing bias is a different + thing, at :8636-8646, and the code that follows it exists to *undo* it: "This + adjustment is more than an optimization. The packing above might be so out + of balance as to be illegal... This adjustment is not optional." The + leave-room-for-the-next-append argument is `balance_quick`'s, at :8022-8027. + Three separate ideas within a hundred lines of each other; the previous + edition of this chapter merged the first two and got both wrong. + +
+ +- [ ] You can state how many pages a lookup touches in a 10⁶-row table, show the arithmetic, and say why that is not a latency prediction. + +
Answer + + With `P` = 4096 (`sqliteLimit.h:214`), a 100-byte payload and rowids below + 2²¹: a table leaf has an 8-byte header and 106-byte slots + (2 pointer + 1 payload-size varint + 3 rowid varint + 100), so + `L = floor(4088/106) = 38`. A table interior page has a 12-byte header + (the extra 4 being the right child, btreeInt.h:134) and 9-byte slots + (2 + 4 + 3), so `F = floor(4084/9) = 453`. Then + `leaves = ceil(10⁶/38) = 26,316` and + `ceil(log_453 26,315.8) = ceil(10.178/6.116) = ceil(1.6642) = 2` interior + levels, for **3 pages touched**. At 10⁹ rows the rowid varint grows to 5 + bytes, `F` falls to `floor(4084/11) = 371`, and the answer is 4. + + Why it is not a latency prediction: this topic measured it. Height stays at + 3 from 10⁶ to 4×10⁶ keys while lookups climb 862 ns → 1101 ns (see the + measured block in README.md and the ladder in notes.md). Height is a step + function of `N`; latency is smooth, because height counts pages *touched* + and says nothing about whether a touch hits L2, L3 or DRAM. Predict pages + read with the arithmetic; predict time with a benchmark. + +
+ +- [ ] You can say what an interior delete costs that a leaf delete does not. + +
Answer + + A leaf delete is one `dropCell` and possibly one `balance()`. An interior + delete forks (Step 7): `sqlite3BtreePrevious` at btree.c:9956 walks down to + the predecessor — chosen over the successor because it is guaranteed to sit + in the subtree under the deleted cell's own child pointer (:9951-9953), which + keeps the repair local. Then there are two `dropCell`s, at :9980 (the + interior cell) and :10013 (the leaf cell), one `insertCell` at :10011 that + grows the promoted cell by exactly 4 bytes (`pCell-4, nCell+4` — an interior + cell carries a child pgno, a leaf cell does not, btreeInt.h:282), and up to + **two** `balance()` calls: :10039 for the leaf, then :10048 after walking the + cursor back up, if the first did not propagate far enough (:10041). + + Both balances are guarded by the threshold at :10034, + `nFree*3 <= usableSize*2` — balance is skipped unless more than two thirds of + the page is free, i.e. the page is under one third full. At usable = 4096 + that means `nFree > 2730`. + +
+ +- [ ] You can state the pointer map's cost in both space and writes. + +
Answer + + Space: entries are 5 bytes (1 type + 4 page number), from + `PTRMAP_PTROFFSET(pgptrmap, pgno) = 5*(pgno-pgptrmap-1)` at btreeInt.h:630, + and `ptrmapPageno` groups the file into runs of + `nPagesPerMapPage = (usableSize/5)+1` (btree.c:1068). At usable = 4096 that + is `819 + 1 = 820`, so one page in 820 is a ptrmap page — **0.122%**. A + rounding error. + + Writes: that is the real cost. Every page allocation, every split that moves + a page, and every change to an overflow chain must also call `ptrmapPut` + (btree.c:1087) and dirty the covering ptrmap page — turning one dirtied page + into two, adding a second failure point to the balance code, and adding + write traffic to the pager on the hottest paths. That is why it is + compile-time optional (`SQLITE_OMIT_AUTOVACUUM`, btree.c:1053) and why turso + has not implemented it. + +
## References **Code** -- [sqlite](https://github.com/sqlite/sqlite) — `src/btree.c` (11,633 - lines; don't read linearly) and `src/btreeInt.h` (746 lines) — - btreeInt.h:1–215 is the best on-disk-format documentation in open - source; read that comment entire before any function + +| File | Lines | What | +|---|---|---| +| `src/btreeInt.h` | 1-215 | the on-disk format as a comment — read it entire, before any function | +| `src/btreeInt.h` | 273-304 | `MemPage` — every production decision of Step 1 | +| `src/btree.c` | 1846-1930 | `allocateSpace` — the three routes, in order | +| `src/btree.c` | 5917-5968 | the descent loop and the one-probe bias | +| `src/btree.c` | 8022-8037 | why the append fast path exists | +| `src/btree.c` | 8636-8646 | the packing bias, and why fixing it is not optional | +| `src/btree.c` | 8730-8741 | the measured 25% — page renumbering, not packing | +| `src/btree.c` | 9948-10013 | the predecessor swap and the 4-byte promotion | + +- [sqlite](https://github.com/sqlite/sqlite), pinned at `sqlite/sqlite@951de30` + — `src/btree.c` (11,633 lines; don't read linearly) and `src/btreeInt.h` + (746 lines). + +**In this curriculum** +- [`reading-lmdb.md`](reading-lmdb.md) — the opposite design: no freeblocks, + no pointer map, no WAL, and a page that is compacted on every delete. +- [`reading-turso-btree-deep.md`](reading-turso-btree-deep.md) — the same + format reimplemented, with the simplifications this chapter is reading + *against*. +- [README.md](README.md) §3 — the splits-and-balance diagram, and the measured + height ladder that Step 2's arithmetic must be read alongside. diff --git a/topics/03-btree-internals/reading-sqlite-file-format.md b/topics/03-btree-internals/reading-sqlite-file-format.md index 59387bb..330fd7c 100644 --- a/topics/03-btree-internals/reading-sqlite-file-format.md +++ b/topics/03-btree-internals/reading-sqlite-file-format.md @@ -2,79 +2,194 @@ The normative spec for what btree.c writes — and the one document in this topic you read with a hex dump open beside it. After two codebases' worth of -slotted pages, this chapter builds the format bottom-up in five steps — -header, page, varint, cell, record — verifies your mental model against the -official text, and ends with the exercise that makes the format yours: -labeling every byte of one cell in a real database file. ~1.5 h. +slotted pages, this chapter builds the format bottom-up in six steps — +header, page, varint, cell, record, and the fork where a payload outgrows its +page — verifies your mental model against the official text, and ends with the +exercise that makes the format yours: labelling every byte of one cell in a +real database file. ~1.5 h. + +Two kinds of anchor appear below. Section numbers such as **§1.3.2** are +sections of *The SQLite Database File Format*, +, which is normative. Line numbers +such as `btreeInt.h:130` are SQLite at the commit this repo pins, +**`sqlite/sqlite@951de30`** (confirm with `tools/pinned-source.py ref sqlite`), +where the same rules appear as the implementation's own comments. When the two +disagree, the document wins — but they do not disagree, and reading them +side by side is the point. ## The problem in one sentence -A two-row table is an **8,192-byte** file (two 4 KB pages), and by the end of -this chapter you must be able to point at every byte that encodes the row -`(500, 'world')` in a raw hex dump — page size, page type, cell pointer, -varints, serial types, payload. +A two-row table is an **8,192-byte** file — two 4,096-byte pages — and by the +end of this chapter you must be able to point at every byte that encodes the +row `(500, 'world')` in a raw hex dump: page size, page type, cell pointer, +the two varints `0x83 0x74`, the record header, the serial type `0x17`, and +the five bytes `w o r l d`. ## The concepts, step by step ### Step 1 — the file is an array of pages, and byte 0 starts a 100-byte header -An SQLite database file is nothing but fixed-size pages laid end to end — -page 1 begins at byte 0, page N at byte `(N−1) × page_size` — and the first -100 bytes of page 1 are the **file header** that says how to read everything -else. The fields to find in your dump: - -- **page size** at offset 16 (big-endian u16 — most significant byte first; - 4096 is stored as `0x10 0x00`); -- the **file change counter** at offset 24, bumped on every write - transaction; -- the **freelist** head page and count at offsets 32–39 (the chain of - wholly-unused pages); -- the **schema cookie**, bumped whenever the schema changes. +> **In:** a file, and nothing else — no schema, no catalogue, no side file. +> **Out:** one number, the page size, from which every other address in the +> file is computed. Steps 2-6 all address in page units. + +An SQLite database file is fixed-size pages laid end to end (§1.2): page 1 +begins at byte 0, page *N* at byte `(N−1) × page_size`. The first 100 bytes of +page 1 are the **database header** (§1.3) — and note the asymmetry, page 1 is +the only page that carries it, which is why `MemPage.hdrOffset` is documented +as "100 for page 1. 0 otherwise" (btreeInt.h:281). + +The fields to find in your dump, with the offsets §1.3 gives them: + +| Offset | Size | Field | Why you care | +|---|---|---|---| +| 0 | 16 | `"SQLite format 3\0"` | how a file(1)-style detector recognises it | +| **16** | **2** | **page size**, big-endian | the unit for every address below (§1.3.2) | +| 18, 19 | 1, 1 | write / read format version | 1 = legacy journal, 2 = WAL | +| **20** | **1** | **reserved bytes per page** | *usable size* = page size − this (§1.3.4) | +| 21, 22, 23 | 1 each | payload fractions, must be 64 / 32 / 32 | Step 6's overflow thresholds (§1.3.5) | +| 24 | 4 | file change counter | bumped on every write txn (§1.3.6) | +| 28 | 4 | in-header database size, in pages | §1.3.7 | +| 32 | 4 | first freelist **trunk** page | head of the free-page chain (§1.3.8) | +| 36 | 4 | total freelist pages | §1.3.8 | +| 40 | 4 | schema cookie | bumped when the schema changes (§1.3.9) | +| 92 | 4 | version-valid-for number | pairs with offset 24 — question 3 (§1.3.16) | + +Two things the older edition of this chapter left out, and both bite in the +exercise: -Why it matters: everything downstream is addressed in page units, and the -page size that defines those units lives in exactly one place — these two -bytes. +- **Page size is big-endian and 4096 is `0x10 0x00`.** §1.3.2 adds a wrinkle: + the value must be a power of two between 512 and 32768, *or the literal + value 1*, which means 65536 — because 65536 does not fit in two bytes. +- **Offset 20 is usually 0 but is not always.** §1.3.4 calls it "Bytes of + unused 'reserved' space at the end of each page. Usually 0." It is + subtracted from the page size to give the **usable size**, which is the + number every later formula actually uses. On macOS's system `sqlite3` (an + Apple build, `3.51.0 …apl`) this byte reads `0x0c`, so usable = 4096 − 12 = + 4084. On a stock build it is 0 and usable = 4096. Check your own byte before + trusting any arithmetic below. + +Why it matters: everything downstream is addressed in page units, and the two +numbers that define those units — offsets 16 and 20 — live in exactly one +place each. ### Step 2 — the b-tree page: one type byte, then the slotted layout -Every page that stores table or index data is a **b-tree page**: its first -byte declares the page type (`0x0D` table leaf, `0x05` table interior, -`0x0A` index leaf, `0x02` index interior), followed by the slotted-page -header you now know from two codebases — cell count, offset where cell -content starts, first-freeblock pointer, fragmented-bytes counter — then the -sorted array of 2-byte cell pointers, a gap, and the cells themselves packed -from the page's end. +> **In:** a page number and the page size from Step 1. +> **Out:** a page type, a cell count, and an array of 2-byte offsets — enough +> to find any cell on the page without decoding a single one of them. + +Every page holding table or index data is a **b-tree page** (§1.6). Its first +byte declares the type, and the four legal values are worth memorising because +you will read them off a dump constantly: + +| Byte | Page | +|---|---| +| `0x02` | interior **index** | +| `0x05` | interior **table** | +| `0x0a` | leaf **index** | +| `0x0d` | leaf **table** | + +"Any other value for the b-tree page type is an error" (§1.6). The header that +follows is the slotted-page header you now know from two codebases, and +btreeInt.h states it in the same layout the document does: + +```c +// sqlite/sqlite src/btreeInt.h — the b-tree page header, 126-134 + 126 ** The page headers looks like this: + 127 ** + 128 ** OFFSET SIZE DESCRIPTION + 129 ** 0 1 Flags. 1: intkey, 2: zerodata, 4: leafdata, 8: leaf + 130 ** 1 2 byte offset to the first freeblock + 131 ** 3 2 number of cells on this page + 132 ** 5 2 first byte of the cell content area + 133 ** 7 1 number of fragmented free bytes + 134 ** 8 4 Right child (the Ptr(N) value). Omitted on leaves. +``` + +Line 134 is why interior pages have a 12-byte header and leaves an 8-byte one. +Then comes the sorted array of 2-byte cell pointers, a gap, and the cells +packed from the page's end: ``` - ┌───────────────┬─────────────────────┬───────┬───────────────────┐ - │ 8/12 B header │ cell ptr array (2B │ free │ cells, packed │ - │ type,ncell, │ each, sorted order) │ gap │ from the right │ - │ content-start │ →grows │ │ ←grows │ - └───────────────┴─────────────────────┴───────┴───────────────────┘ + offset 0 8 or 12 +2·nCell content-start usable + ┌───────────────┬─────────────────────┬───────────────┬───────────────────┐ + │ page header │ cell ptr array (2 B │ unallocated │ cells, packed │ + │ type, nCell, │ each, sorted by KEY │ gap │ from the right │ + │ content-start │ → grows │ │ ← grows │ + └───────────────┴─────────────────────┴───────────────┴───────────────────┘ + ↑ header offset 5 points here ─────────────────┘ ``` -Verify your mental model against the normative text — especially the -**freeblock** rules (a freeblock is a reusable hole left by a delete): a -freeblock must be at least 4 bytes, and leftovers too small to be freeblocks -are counted in the header's fragment counter, capped at 60 before the page -must be defragmented. +The pointer array is sorted by *key*; the cells themselves are in arbitrary +physical order. That separation is the whole point of a slotted page — an +insert in the middle moves 2·k bytes of pointers, never a byte of payload. + +Now verify your model against the normative text, because this is where two +rules are stated with a force people usually get wrong (§1.6): + +> A freeblock requires at least 4 bytes of space. If there is an isolated +> group of 1, 2, or 3 unused bytes within the cell content area, those bytes +> comprise a fragment. … In a well-formed b-tree page, the total number of +> bytes in fragments may not exceed 60. + +**That 60 is a well-formedness invariant, not a defragmentation trigger.** A +page carrying 61 fragment bytes is *corrupt*, not merely untidy. Defragmenting +is described separately and permissively — "SQLite **may** from time to time +reorganize a b-tree page so that there are no freeblocks or fragment bytes" +— and in the implementation it happens only when an allocation cannot +otherwise be satisfied (`allocateSpace`, btree.c:1909-1912; see +[`reading-sqlite-btree.md`](reading-sqlite-btree.md) Step 3). The previous +edition of this chapter said the counter was "capped at 60 before the page +must be defragmented", which fuses a validity rule with an unrelated policy. + +Two more details from §1.6 that the implementation comment (btreeInt.h:152-163) +repeats: a freeblock's 2-byte size field counts **including the 4-byte +header**, and freeblocks are chained "in order of increasing offset". Why it matters: this is where the spec is law — the codebases you read are correct *because* they match these rules, not the other way around. ### Step 3 — the varint: SQLite's variable-length integer -A **varint** is an integer encoded in 1–9 bytes, 7 payload bits per byte, -where a set high bit means "more bytes follow" — small numbers (the common -case: short lengths, low rowids) cost one byte instead of eight. SQLite's -flavor is **big-endian** (most significant group first — unlike protobuf), -and a 9th byte, if reached, contributes all 8 of its bits. +> **In:** a byte offset inside a cell. +> **Out:** a 64-bit value *and* a length, so you know where the next field +> begins. Without this you cannot advance a single field in Steps 4-6. + +A **varint** is, in §2.1's words, "a static Huffman encoding of 64-bit +twos-complement integers that uses less space for small positive values". It +is 1 to 9 bytes: seven payload bits per byte, high bit set meaning "more +follows", most significant group **first** — big-endian, unlike protobuf — and +a ninth byte, if reached, contributes all 8 of its bits. btreeInt.h states the +rule and then, unusually, hands you a test vector: + +```c +// sqlite/sqlite src/btreeInt.h — the varint rule and its worked examples, 170-184 + 170 ** Cell content makes use of variable length integers. A variable + 171 ** length integer is 1 to 9 bytes where the lower 7 bits of each + 172 ** byte are used. The integer consists of all bytes that have bit 8 set and + 173 ** the first byte with bit 8 clear. The most significant byte of the integer + 174 ** appears first. A variable-length integer may not be more than 9 bytes long. + 175 ** As a special case, all 8 bits of the 9th byte are used as data. This + 176 ** allows a 64-bit integer to be encoded in 9 bytes. + 177 ** + 178 ** 0x00 becomes 0x00000000 + 179 ** 0x7f becomes 0x0000007f + 180 ** 0x81 0x00 becomes 0x00000080 + 181 ** 0x82 0x00 becomes 0x00000100 + 182 ** 0x80 0x7f becomes 0x0000007f + 183 ** 0x81 0x91 0xd1 0xac 0x78 becomes 0x12345678 + 184 ** 0x81 0x81 0x81 0x81 0x01 becomes 0x10204081 +``` + +Line 182 is the one to stare at: `0x80 0x7f` and `0x7f` both decode to 127. +The encoding is not canonical, so a decoder must not assume minimal length. -Every cell starts with varints, so carry the decoder in your head into the -exercise: +Carry the decoder into the exercise: ```rust -// SQLite varint: 7 bits/byte, BIG-endian (unlike protobuf), max 9 bytes +// ILLUSTRATION — not quoted from SQLite. The rule is src/btreeInt.h:170-176; +// the real decoder is sqlite3GetVarint in src/util.c. fn read_varint(buf: &[u8]) -> (u64, usize) { let mut v = 0u64; for i in 0..8 { @@ -85,7 +200,20 @@ fn read_varint(buf: &[u8]) -> (u64, usize) { } ((v << 8) | buf[8] as u64, 9) // 9th byte contributes all 8 bits } -// rowid 500 = 0x83 0x74 → (0b0000011 << 7) | 0b1110100 — find it in the dump +``` + +Work the case you will meet in the dump, by hand: + +``` +rowid 500 = 0b1_1111_0100 + split into 7-bit groups, most significant first: + group 1 = 500 >> 7 = 3 = 0b000_0011 + group 0 = 500 & 0x7f = 116 = 0b111_0100 + set the continuation bit on every group but the last: + byte 0 = 0x80 | 3 = 0x83 + byte 1 = 116 = 0x74 + encoded: 0x83 0x74 (2 bytes, versus 8 for a fixed u64) + decode: (3 << 7) | 116 = 384 + 116 = 500 ✓ ``` Why it matters: you cannot find *anything* inside a cell without decoding @@ -94,91 +222,397 @@ one. ### Step 4 — the cell: payload size, rowid, record -A **cell** is one row's on-disk container. In a table leaf it is exactly -three parts, in order: a varint giving the payload size, a varint giving -the **rowid** (the table's hidden 64-bit integer key), then the payload — -the encoded row itself (Step 5). +> **In:** one 2-byte entry from Step 2's pointer array. +> **Out:** a payload byte-range and a rowid — the row's identity and its +> contents, still undecoded. + +A **cell** is one row's on-disk container. btreeInt.h gives the general shape +for all four page types at once: + +```c +// sqlite/sqlite src/btreeInt.h — the general cell layout, 189-196 + 189 ** The content of a cell looks like this: + 190 ** + 191 ** SIZE DESCRIPTION + 192 ** 4 Page number of the left child. Omitted if leaf flag is set. + 193 ** var Number of bytes of data. Omitted if the zerodata flag is set. + 194 ** var Number of bytes of key. Or the key itself if intkey flag is set. + 195 ** * Payload + 196 ** 4 First page of the overflow chain. Omitted if no overflow +``` + +For a table leaf (`0x0d`) three of those five rows survive: no left child +(line 192, it is a leaf), a payload-size varint (193), and — because +`intkey` is set — line 194 becomes *the rowid itself*, as a varint, rather +than a key length. Then the payload, and the 4-byte overflow pointer only if +Step 6 fired. + +So a table-leaf cell is exactly: + +``` + varint payload size, in bytes (the record of Step 5) + varint rowid — the table's hidden 64-bit integer key + bytes the record, `payload size` of them + [4 B] first overflow page, present only when the payload did not fit +``` -Concretely, for the row `(500, 'world')`: payload-size varint, then rowid -500 as the two bytes `0x83 0x74` (check the 7-bit encoding), then the -record. The cell pointer in Step 2's array is what tells you where this cell -begins. +Concretely, for `(500, 'world')` in the exercise below, that reads +`08 | 83 74 | <8 bytes of record>` — payload size 8, rowid 500 as Step 3's two +bytes, then the record. Total cell size 11 bytes. -Why it matters: the cell is the unit the b-tree machinery moves, splits, and -points at — and for an INTEGER PRIMARY KEY table, the rowid varint here *is* -the primary key (question 2 below). +Why it matters: the cell is the unit the b-tree machinery moves, splits and +points at — and for an `INTEGER PRIMARY KEY` table the rowid varint *is* the +primary key, which is question 2 below and which you will see confirmed by a +`0x00` in the record. ### Step 5 — the record: serial types make pages schema-free -The payload is a **record**: a varint giving the header length, then one -**serial type** varint per column (a single number encoding both the -column's type *and* its byte length), then the column values back to back — -so a page can be decoded with no schema in hand. +> **In:** the payload byte-range from Step 4. +> **Out:** typed column values — obtained without consulting the schema, +> which is the property that makes a page self-describing. + +The payload is a **record** (§2.1): a varint giving the header length, then +one **serial type** varint per column, then the column values back to back. A +serial type is a single number encoding both the column's type *and* its byte +length, so the decoder never needs the table definition to know where one +value ends. + +| Serial type *T* | Content bytes | Meaning | +|---|---|---| +| 0 | 0 | NULL | +| 1, 2, 3, 4 | 1, 2, 3, 4 | big-endian twos-complement integer | +| 5 | 6 | 48-bit integer | +| 6 | 8 | 64-bit integer | +| 7 | 8 | IEEE-754 double | +| **8** | **0** | **the integer 0** — value is entirely in the type | +| **9** | **0** | **the integer 1** — likewise | +| 10, 11 | var | reserved; never in a well-formed file | +| *T* ≥ 12, even | (*T*−12)/2 | BLOB of that length | +| *T* ≥ 13, odd | (*T*−13)/2 | text of that length, **no NUL terminator** | + +Three points the older edition of this chapter got loose, and one it omitted: + +- **The header-length varint counts itself.** §2.1: "The varint value is the + size of the header in bytes *including the size varint itself*." Off-by-one + here is the single most common error in the exercise. +- **Name the variable.** Going *up*, a text value of length `n` gets serial + type `T = 2n + 13`; going *down*, a serial type `T` yields length + `(T − 13)/2`. Those are inverses of each other, not two rules — and the + previous edition wrote both with the same letter `n`, which makes them look + contradictory. For `'hello'`, `n = 5`, so `T = 2·5 + 13 = 23 = 0x17`. +- **Types 8 and 9 have a version floor.** §2.1 marks both "(Only available for + schema format 4 and higher.)" — the schema format number is at file offset + 44 (§1.3.10). A booleans-heavy table on an older schema format pays a byte + per value that a modern one does not. +- **Five types are zero-length**, not two: §2.1 lists 0, 8, 9, 12 and 13 — + the last two being the empty blob and the empty string. "If all columns are + of these types then the body section of the record is empty." + +Why it matters: this is the exercise's final boss — once you can read a serial +type and count value bytes, the whole file is legible without a schema. + +### Step 6 — the fork: when a payload outgrows the page + +> **In:** a record from Step 5 that is larger than a page can hold. +> **Out:** two things instead of one — a *local* prefix that stays in the +> cell, and an *overflow chain* of pages holding the rest. Every later reader +> must handle both halves, which is why this fork touches search, delete and +> vacuum alike. + +A cell must fit on a page, and a record need not. §1.7 resolves this by +splitting the payload: the cell keeps a prefix, and the remainder goes into a +linked list of overflow pages, addressed by the 4-byte pointer from Step 4's +line 196. + +```c +// sqlite/sqlite src/btreeInt.h — the overflow chain format, 198-204 + 198 ** Overflow pages form a linked list. Each page except the last is completely + 199 ** filled with data (pagesize - 4 bytes). The last page can have as little + 200 ** as 1 byte of data. + 201 ** + 202 ** SIZE DESCRIPTION + 203 ** 4 Page number of next overflow page + 204 ** * Data +``` -The serial-type table is the heart of §2. Note especially: +The threshold is not a constant; it is computed from Step 1's payload +fractions at header offsets 21, 22 and 23 — the bytes §1.3.5 requires to be +64, 32 and 32, each a fraction of 255. The implementation turns them into +four limits in one place: + +```c +// sqlite/sqlite src/btree.c — the payload limits, inside sqlite3BtreeSetPageSize, 3471-3474 + 3471 pBt->maxLocal = (u16)((pBt->usableSize-12)*64/255 - 23); + 3472 pBt->minLocal = (u16)((pBt->usableSize-12)*32/255 - 23); + 3473 pBt->maxLeaf = (u16)(pBt->usableSize - 35); + 3474 pBt->minLeaf = (u16)((pBt->usableSize-12)*32/255 - 23); +``` -- types **8 and 9** mean literal integer 0 and 1 with **zero bytes of - payload** — the value is entirely in the type number; -- text and blob lengths ride inside the type number via the odd/even - encoding: text of length n has serial type `2n+13` (decode with - `(n−13)/2`), blobs use even numbers (`(n−12)/2`). So `'hello'` (text, - length 5) has serial type 2·5+13 = 23. +Evaluate them, naming every symbol. `U` = usable size = page size − reserved +bytes (Step 1, offsets 16 and 20). The `−12` is the interior page header; the +`−23` is a worst-case cell overhead allowance; `64/255` and `32/255` are the +header's payload fractions. -Why it matters: this is the exercise's final boss — once you can read a -serial type and count value bytes, the whole file is legible. +``` +stock build, reserved = 0, U = 4096: + maxLocal = floor((4096-12) × 64 / 255) - 23 + = floor(4084 × 64 / 255) - 23 = floor(261376/255) - 23 + = 1025 - 23 = 1002 ← index cells spill past 1002 B + minLocal = floor(4084 × 32 / 255) - 23 = floor(130688/255) - 23 + = 512 - 23 = 489 ← at least this much always stays local + maxLeaf = 4096 - 35 = 4061 ← a TABLE leaf keeps up to 4061 B locally + +Apple's system sqlite3, reserved = 12 (offset 20 = 0x0c), U = 4084: + maxLocal = floor(4072 × 64 / 255) - 23 = 1022 - 23 = 999 + maxLeaf = 4084 - 35 = 4049 +``` -## How to read the document (with the concepts in hand) +The asymmetry at line 3473 is the design: a *table* leaf keeps almost the +whole page locally (4061 of 4096 bytes), because a table b-tree is where big +rows live and chasing an overflow chain to read one row would be miserable. +An *index* page keeps at most 1002 bytes — roughly a quarter of the page — +because an index page is searched, and search wants fanout, and fanout wants +small cells. Same format, two policies, both derived from three bytes in the +file header. + +`minLocal` (line 3472) is the anti-thrash floor: at least 489 bytes always +stay local, so a payload that only just exceeds `maxLocal` cannot produce an +overflow page holding four bytes of data. The exact spill formula, and the +"≥ 4 cells per page" reasoning behind the `64/255`, belong to +[`reading-turso-btree-deep.md`](reading-turso-btree-deep.md), which reads a +reimplementation of these same four lines. + +Why it matters: this fork is the reason `CellInfo` has both `nPayload` and +`nLocal` (btreeInt.h:483-484), why the pointer map needs two overflow entry +types (`PTRMAP_OVERFLOW1`/`2`, btreeInt.h:666-667), and why a delete has to +free a chain before it frees a cell. -Read in this order: +## How to read the document (with the concepts in hand) -1. **§1 The database file** — Step 1's 100-byte file header: page size - (offset 16), file change counter, freelist head + count (offsets 32–39), - schema cookie. -2. **§1.6 B-tree pages** — Step 2: the slotted-page spec you now know from - two codebases; verify your mental model against the normative text (esp. - freeblock rules: min 4 bytes, fragment cap 60). -3. **§2 Record format** — Steps 3–5: the serial types table. Note types 8/9 - (literal 0 and 1 — zero bytes of payload!) and the odd/even text/blob - length encoding `(n−13)/2` / `(n−12)/2`. -4. **§1.5 Pointer maps**, **§4.1 WAL vs rollback journal** — skim; WAL is - topic 5. +Section numbers below are the real ones on +; the previous edition of this +chapter had three of them wrong, listed under "corrections" at the end. + +| Read | Section | For | Step | +|---|---|---|---| +| 1st | **§1.2 Pages**, **§1.3 The Database Header** | the 100-byte header table; page size §1.3.2, reserved bytes §1.3.4, payload fractions §1.3.5, change counter §1.3.6, free page list §1.3.8, schema cookie §1.3.9, version-valid-for §1.3.16 | 1 | +| 2nd | **§1.6 B-tree Pages** | the slotted-page spec you know from two codebases — verify against it, especially the freeblock minimum of 4 bytes and the 60-byte fragment *validity* limit | 2 | +| 3rd | **§2.1 Record Format** | varints, the serial-type chart, the header-length-includes-itself rule | 3, 4, 5 | +| 4th | **§1.7 Cell Payload Overflow Pages** | the fork, and its interaction with §1.3.5's fractions | 6 | +| skim | **§1.5 The Freelist**, **§1.8 Pointer Map Pages** | free-page reuse and the reverse index; the ptrmap cost model is in [`reading-sqlite-btree.md`](reading-sqlite-btree.md) Step 6 | — | +| skim | **§3 The Rollback Journal**, **§4 The Write-Ahead Log** | how any of this becomes durable — topic 5 does it properly | — | + +**Corrections to the previous edition of this chapter**, all verified against +the live document's table of contents: the record format is **§2.1**, not §2 +(§2 is "Schema Layer"); pointer maps are **§1.8**, not §1.5 (§1.5 is "The +Freelist"); the rollback journal is **§3** and the WAL is **§4**, so "§4.1 WAL +vs rollback journal" was two sections conflated (§4.1 is "WAL File Format"). +The canonical URL is `fileformat.html`; `fileformat2.html` also resolves but +is not the name the document uses for itself. ## The exercise (30 min, do it) +Write the scratch database inside the repo, not `/tmp`: + ```bash -sqlite3 /tmp/t.db "create table t(a integer primary key, b text); - insert into t values (1,'hello'),(500,'world');" -xxd /tmp/t.db | head -80 +mkdir -p .cache/scratch && rm -f .cache/scratch/t.db +sqlite3 .cache/scratch/t.db \ + "create table t(a integer primary key, b text); + insert into t values (1,'hello'),(500,'world');" +ls -l .cache/scratch/t.db # expect exactly 8192 bytes = 2 × 4096 +xxd -l 112 .cache/scratch/t.db # the file header (Step 1) +xxd -s 4096 -l 16 .cache/scratch/t.db # page 2's header (Step 2) +xxd -s 8154 -l 38 .cache/scratch/t.db # the two cells (Steps 3-5) ``` Find by hand, writing offsets in notes.md: -- page size at offset 16 (big-endian u16); -- page 2's header byte `0x0D` (table leaf), cell count, content-area start; -- the two cell pointers, then decode cell 1: payload-size varint, rowid varint - (rowid 500 needs 2 bytes — check the 7-bit encoding), record header, serial - type for 'hello' (text len 5 ⇒ type 2·5+13 = 23). + +1. **Offset 16** — the page size as a big-endian u16. **Offset 20** — your + build's reserved bytes; compute `usable = page_size − reserved` and use it + everywhere below. +2. **Page 2, offset 0** — the type byte. Then offsets 3-4 (cell count), 5-6 + (cell content area start), 1-2 (first freeblock) and 7 (fragments). +3. **Page 2, offsets 8-11** — the two 2-byte cell pointers. They are page + offsets, so cell *k* begins at file byte `4096 + pointer[k]`. Note which + pointer is larger, and explain it: cells grow leftward from the page's end, + so the *second* row inserted sits at the *lower* offset. +4. Decode both cells: payload-size varint, rowid varint (rowid 500 needs the + two bytes of Step 3), then the record — header-length varint (remember it + counts itself), one serial type per column, then the values. +5. **Close the arithmetic.** Add each cell's total length to its pointer. The + largest such sum must equal your usable size from step 1 — the first cell + ends exactly at the usable boundary. If it does not, you mis-decoded a + varint or misread offset 20. + +Two things to notice while you are in there. The `a integer primary key` +column decodes to serial type **`0x00`, NULL** — the value is not stored +twice; it lives only in the cell's rowid varint (question 2). And the record +header is 3 bytes for both rows: one for its own length, one for each column's +serial type. If you can decode a row from a hex dump, the format is yours. ## Questions to answer in notes.md -1. Why does the format store the cell CONTENT area offset in the header instead - of deriving it from the cell pointers? (Cheap free-space check: `content_start - − ptr_array_end` without scanning.) -2. INTEGER PRIMARY KEY tables store the key only as the rowid varint — the - column itself is NULL in the record. What does this alias buy in bytes/row - and what does it forbid? (WITHOUT ROWID tables exist for the other case.) -3. The change counter (offset 24) and version-valid-for (92) — how do they let - a reader detect a stale in-memory schema without locks? +1. Why does the format store the cell content area offset in the header + (§1.6, offset 5) instead of deriving it from the cell pointers? (Cheap + free-space check: `content_start − (header + 2·nCell)` without scanning — + and compare `MemPage.nFree`'s −1 sentinel, btreeInt.h:288.) +2. `INTEGER PRIMARY KEY` tables store the key only as the rowid varint — the + column itself decodes to serial type 0, NULL, as you just saw in the dump. + What does this alias buy in bytes per row for the exercise's table, and + what does it forbid? (`WITHOUT ROWID` tables, §2.4, exist for the other + case.) +3. The change counter (offset 24, §1.3.6) and the version-valid-for number + (offset 92, §1.3.16) — how do they let a reader detect a stale in-memory + schema without taking a lock? What has to be true about the *order* in + which those two fields are written? +4. Offset 20's reserved bytes shrink the usable size, and every formula in + Step 6 is denominated in usable size. If a build reserved 32 bytes per page + for a checksum, recompute `maxLocal` and `maxLeaf` at a 4096-byte page, and + say how many more index cells per page you would need to lose before the + tree gained a level (use the fanout arithmetic in + [`reading-sqlite-btree.md`](reading-sqlite-btree.md) Step 2). ## Done when -Your notes contain the annotated hex dump with every byte of one cell labeled. +Answer each before unfolding it. + +- [ ] Your notes contain the annotated hex dump with every byte of one cell labelled. + +
Answer + + For `(500, 'world')` on a stock build, the cell is 11 bytes and reads + `08 83 74 03 00 17 77 6f 72 6c 64`: + + | Bytes | Field | Value | + |---|---|---| + | `08` | payload-size varint (Step 4) | 8 bytes of record follow the rowid | + | `83 74` | rowid varint (Step 3) | `(3 << 7) \| 116` = 500 | + | `03` | record header length (§2.1) | 3 — **and it counts itself** | + | `00` | serial type, column `a` | NULL — the rowid alias (question 2) | + | `17` | serial type, column `b` | 23, odd ⇒ text of length (23−13)/2 = 5 | + | `77 6f 72 6c 64` | body | `w o r l d`, no NUL terminator | + + Check it closes: header 3 + body 0 + 5 = 8 = the payload size. And + 1 + 2 + 8 = 11 = the cell's total length, which is what you add to its + pointer in exercise step 5. + +
+ +- [ ] You can state the two numbers in the file header that every later formula depends on, and where they are. + +
Answer + + **Offset 16** (2 bytes, big-endian) is the page size — §1.3.2, a power of + two from 512 to 32768, or the literal `1` meaning 65536 because 65536 does + not fit in two bytes. **Offset 20** (1 byte) is the reserved bytes per page + — §1.3.4, "Usually 0". + + Usable size = page size − reserved bytes, and *that* is the quantity every + formula uses: the cell content area's upper bound, `maxLocal`/`minLocal`/ + `maxLeaf` (btree.c:3471-3474), the ptrmap group size (btree.c:1068), and the + fanout arithmetic. It is worth checking your own byte: macOS's system + `sqlite3` reports `0x0c` there, so usable is 4084, not 4096, and every + derived number shifts. + +
+ +- [ ] You can say what the 60-byte fragment limit means, and what it does not. + +
Answer + + §1.6: "In a well-formed b-tree page, the total number of bytes in fragments + may not exceed 60." It is a **validity invariant** — a page whose one-byte + counter at header offset 7 exceeds 60 is corrupt, and `PRAGMA + integrity_check` will say so. + + It is *not* a defragmentation trigger. §1.6 describes defragmenting + separately and permissively — SQLite "may from time to time reorganize a + b-tree page so that there are no freeblocks or fragment bytes" — and the + implementation only does it when an allocation cannot be satisfied any other + way (`allocateSpace`, btree.c:1909-1912). A fragment exists at all only + because a freeblock needs 4 bytes to hold its own next-pointer and size + (§1.6, btreeInt.h:152-163), so 1-3 stranded bytes have no way to describe + themselves and can only be counted. + +
+ +- [ ] You can explain why a page is decodable without the schema, and name the one thing that is not. + +
Answer + + Because the record format (§2.1) puts a **serial type** varint in front of + every value, and a serial type encodes both the datatype and the byte length: + `T ≥ 13` odd means text of length `(T−13)/2`, `T ≥ 12` even means a blob of + `(T−12)/2`, `T` in 1-7 are fixed widths, and 0, 8, 9 carry their value + entirely in the type number and occupy zero bytes. So a decoder can walk + every column of every row with no table definition in hand — which is what + makes `xxd` a usable tool here at all. + + What is *not* recoverable: the column **names**, their declared types and + affinities, and the table's name. Those live only in the `sqlite_schema` + table on page 1 (§2.6), as ordinary rows holding the original `CREATE TABLE` + text. A page tells you a value is a 5-byte string; only the schema tells you + the string is called `b`. + +
+ +- [ ] You can say when a payload forks into an overflow chain, with the number evaluated. + +
Answer + + When the record exceeds the page's local limit, computed at btree.c:3471-3474 + from the three payload fractions at file header offsets 21, 22, 23 — required + by §1.3.5 to be 64, 32, 32, each over 255. With `U` = usable size = 4096 + (reserved = 0): + + - a **table leaf** keeps up to `maxLeaf = U − 35 = 4061` bytes locally; + - an **index** page keeps up to + `maxLocal = floor((U−12)×64/255) − 23 = floor(261376/255) − 23 = 1025 − 23 = 1002`; + - and whatever spills, at least + `minLocal = floor((U−12)×32/255) − 23 = 512 − 23 = 489` bytes stay behind, + so a marginal overflow cannot create a nearly empty overflow page. + + The asymmetry is deliberate: table leaves hold rows and want them local; + index pages are searched and want fanout, so they cap cells at about a + quarter of a page. The chain itself is a singly linked list, each page a + 4-byte next-pointer followed by data, every page but the last completely + full (btreeInt.h:198-204). + +
## References -**Papers** -- SQLite team — "The SQLite Database File Format" (official - documentation) — https://www.sqlite.org/fileformat2.html — the - normative spec for what btree.c writes; read side-by-side with a real - database file and a hex dump +**The document** +- SQLite team — *The SQLite Database File Format*, + — normative. Read side by side with + a real file and a hex dump. + +| Section | What to take | +|---|---| +| §1.2 | pages are fixed size; page *N* starts at `(N−1) × page_size` | +| §1.3 | the 100-byte header table — offsets 16, 20, 21-23, 24, 32, 36, 40, 92 | +| §1.6 | b-tree page header; freeblocks ≥ 4 B; fragments ≤ 60 B for *validity* | +| §1.7 | cell payload overflow pages | +| §2.1 | varints, the serial-type chart, header length includes itself | +| §2.4 | `WITHOUT ROWID` tables — question 2's other case | + +**Code (the same rules, as the implementation's own comments)** + +| File | Lines | What | +|---|---|---| +| `src/btreeInt.h` | 110-134 | page layout and the header offset table | +| `src/btreeInt.h` | 152-163 | freeblock and fragment rules | +| `src/btreeInt.h` | 170-184 | the varint rule *and* seven worked examples | +| `src/btreeInt.h` | 189-204 | cell layout and the overflow chain | +| `src/btree.c` | 3471-3474 | `maxLocal` / `minLocal` / `maxLeaf` — Step 6's thresholds | + +Pinned at `sqlite/sqlite@951de30`; confirm with `tools/pinned-source.py ref sqlite`. + +**In this curriculum** +- [`reading-sqlite-btree.md`](reading-sqlite-btree.md) — the code that writes + this format, including what it does when a page fills. +- [`reading-turso-btree-deep.md`](reading-turso-btree-deep.md) — the same + format decoded by a second implementation, where Step 6's spill formula is + worked in full. diff --git a/topics/03-btree-internals/reading-turso-btree-deep.md b/topics/03-btree-internals/reading-turso-btree-deep.md index a95cea7..0dae252 100644 --- a/topics/03-btree-internals/reading-turso-btree-deep.md +++ b/topics/03-btree-internals/reading-turso-btree-deep.md @@ -5,221 +5,933 @@ descends into the page mechanics that surface glossed over — the freeblock chain, the exact overflow-spill formulas, the resumable balance state machines, varints, and the whole-page freelist. This chapter builds each mechanism step by step, then maps every step to its anchors. Budget: 2–3 h -across `core/storage/btree.rs`, `sqlite3_ondisk.rs`, and `pager.rs`. +across `core/storage/btree.rs`, `core/storage/sqlite3_ondisk.rs`, +`core/storage/pager.rs`, and `core/types.rs`. + +Every anchor below is turso at `tursodatabase/turso@dd775bc`. Confirm the pin +with `tools/pinned-source.py ref turso` before you start; if it prints a +different SHA, the line numbers in this guide are for a different tree and you +should navigate by the symbol names, which are given for every anchor. Read a +range with, for example: + +``` +tools/pinned-source.py show turso core/storage/btree.rs -r 7592:7687 +``` + +Turso is a Rust rewrite of SQLite that is byte-compatible with the SQLite file +format, so this guide and [reading-sqlite-btree.md](reading-sqlite-btree.md) +describe the *same* format through two implementations. Where turso simplifies, +clarifies, or renames something, this guide says so — those diffs are the +cheapest way to see what is essential in the format and what is C-era +incidental. The on-disk field definitions themselves are in +[reading-sqlite-file-format.md](reading-sqlite-file-format.md); that guide +defers the overflow-spill arithmetic to this one, and Step 6 below pays that +debt. ## The problem in one sentence -A **4,096-byte** page must absorb variable-length rows — up to and including -a 100 KB payload — plus arbitrary deletes and re-inserts, forever, without -the on-disk format ever needing a special case: freeblocks, overflow chains, -and 3-sibling balancing are the entire toolkit. +A **page** — a fixed-size chunk of the file, 4,096 bytes by default — must +absorb variable-length rows, up to and including a 100 KB payload, plus +arbitrary deletes and re-inserts, forever, without the on-disk format ever +needing a special case; freeblocks, overflow chains, and 3-sibling balancing +are the entire toolkit. + +Two size symbols recur throughout and are worth pinning now. `P` is the **page +size** (4,096 by default). `R` is the **reserved region** at the end of every +page, a byte count stored at file-header offset 20 that the b-tree layer is +forbidden to touch. `U = P − R` is the **usable space** — the part of the page +the b-tree actually gets. Almost every formula below is written in `U`, not +`P`, and turso's parameter for it is literally named `usable_space`. A stock +build has `R = 0` and therefore `U = P = 4096`; Apple's system SQLite ships +`R = 12`, so `U = 4084` there. This guide works every number at `U = 4096` and +flags where `R` would move it. ## The concepts, step by step -### Step 1 — the freeblock chain: free space is a linked list in the dead bytes +### Step 1 — the slotted page: a header, a growing pointer array, and a shrinking content area + +> **In:** a raw 4,096-byte page and nothing else. **Out:** the meaning of all +> seven header fields, and the fanout and tree height those bytes buy — +> evaluated, not asserted. -When a cell (one row's on-disk container) is deleted, its bytes become a -**freeblock** — a hole inside the page, threaded into a linked list *through -the dead space itself*: each freeblock's first 4 bytes hold a 2-byte pointer -to the next freeblock and its own 2-byte size. Allocation for a new cell is -first-fit down that chain. +A **slotted page** stores variable-length records in a fixed-size block by +splitting the block in three: a header at the front, an array of 2-byte +offsets ("slots", or the **cell pointer array**) growing rightward from the +header, and the **cell content area** — the records themselves — growing +leftward from the end. The unallocated gap between the two is what is left. +A **cell** is one record's on-disk container: for a table leaf, one row. -Two rules keep the bookkeeping in 4 bytes: a freeblock must be at least -**4 bytes** (smaller leftovers can't hold the next-pointer + size), and those -too-small scraps are instead counted in the page header's -`fragmented_bytes` counter. +Turso names every header offset in one module, `btree.rs:84–124`, with the +layout drawn in the doc comment above it at `btree.rs:76–83`: -The freeblock walk, distilled — first-fit through a linked list threaded -through the dead space itself: +| offset | width | field | meaning | +|---|---|---|---| +| 0 | 1 B | `BTREE_PAGE_TYPE` | leaf/interior × table/index | +| 1 | 2 B | `BTREE_FIRST_FREEBLOCK` | head of the freeblock chain, 0 = none | +| 3 | 2 B | `BTREE_CELL_COUNT` | number of slots in the pointer array | +| 5 | 2 B | `BTREE_CELL_CONTENT_AREA` | offset of the lowest live cell byte | +| 7 | 1 B | `BTREE_FRAGMENTED_BYTES_COUNT` | unusable scraps, ≤ 60 | +| 8 | 4 B | `BTREE_RIGHTMOST_PTR` | interior pages only | + +So the header is **8 bytes on a leaf** and **12 bytes on an interior page** — +the last field exists only where there is an extra child to point at. Turso +gives both, plus the two other geometry constants, real names: ```rust -fn find_free_slot(page: &mut Page, need: usize) -> Option { - let mut prev = FREEBLOCK_HEAD; // header bytes 1–2 - let mut off = page.first_freeblock(); - while off != 0 { - let (next, size) = page.freeblock_at(off); // 2B next-ptr + 2B size - if size as usize >= need { - let rest = size as usize - need; - if rest < 4 { // leftover can't hold a freeblock: - page.unlink(prev, next); // take it all, book the scraps - page.add_fragmented(rest as u8); // (header cap: 60) - return Some(off); - } - page.set_size(off, rest as u16); // carve the tail, keep the block - return Some(off + rest as u16); - } - prev = off; off = next; - } - None // nothing fits: allocate from the middle gap, or defragment -} -``` - -Why it matters: deletes cost 2 bytes of pointer-array edit plus threading one -hole — the cleanup is deferred to Step 2, and paid only when space actually -runs short. - -### Step 2 — defragmentation: compact the holes when first-fit fails - -**Defragmentation** rewrites all live cells contiguously at the page's end, -zeroing the freeblock chain and the fragment counter — turning many scattered -holes into one usable gap. Turso has a fast path when there are ≤2 -freeblocks and a slow path that compacts everything. - -Question to hold while reading: what triggers defrag, and why is it correct -to move cells but never the pointer array? (The pointer array *is* the sorted +// tursodatabase/turso@dd775bc — core/storage/sqlite3_ondisk.rs +// the four constants every page-arithmetic formula in this guide is built from + 80 pub const CELL_PTR_SIZE_BYTES: usize = 2; + 81 pub const INTERIOR_PAGE_HEADER_SIZE_BYTES: usize = 12; + 82 pub const LEAF_PAGE_HEADER_SIZE_BYTES: usize = 8; + 83 pub const LEFT_CHILD_PTR_SIZE_BYTES: usize = 4; +``` + +Two of those constants are easy to skip past and both matter below. Every cell +costs **2 bytes of pointer** in addition to its own bytes — that 2 is +`CELL_PTR_SIZE_BYTES`, and forgetting it is the classic way to overcount +fanout by a few percent. And an interior cell carries a **4-byte child page +number**, `LEFT_CHILD_PTR_SIZE_BYTES`, before anything else. + +Now the arithmetic. Define: + +- `U` = usable space per page = 4096 (stock build). +- `H_leaf` = 8, `H_int` = 12 — the header widths above. +- `p` = 2 — `CELL_PTR_SIZE_BYTES`. +- `c` = 4 — `LEFT_CHILD_PTR_SIZE_BYTES`. +- `L` = **leaf fanout**: how many rows fit on one leaf page. +- `F` = **interior fanout**: how many children one interior page can name. + +Take a concrete row: a table with a 100-byte payload and rowids below +2²¹ = 2,097,152, so the rowid varint (Step 4) is 3 bytes and the payload-size +varint is 1 byte. One table-leaf slot therefore costs: + +```text +slot_leaf = p + size_varint + rowid_varint + payload + = 2 + 1 + 3 + 100 + = 106 bytes + +L = floor((U − H_leaf) / slot_leaf) + = floor((4096 − 8) / 106) + = floor(4088 / 106) + = floor(38.57) + = 38 rows per leaf +``` + +A table-*interior* cell has no payload at all (Step 5): it is a 4-byte child +pointer and a rowid varint, and nothing else. + +```text +slot_int_table = p + c + rowid_varint + = 2 + 4 + 3 + = 9 bytes + +F = floor((U − H_int) / slot_int_table) + = floor((4096 − 12) / 9) + = floor(4084 / 9) + = floor(453.8) + = 453 children per interior page +``` + +That 453 is the number to remember, and it is why the original guide's +hand-wave — "table interior cells are ~13 bytes, so table trees have enormous +fanout" — is worth replacing with a figure. (13 is the *worst case*: 4 bytes of +child plus a 9-byte rowid varint, which only occurs above rowid 2⁵⁶. At that +point `F` falls to `floor(4084/15) = 272`.) + +Height follows. Let `N` be the row count and `d` the number of **pages a +lookup touches**, root included: + +```text +N = 1,000,000 +leaves = ceil(N / L) = ceil(1000000 / 38) = 26,316 + +interior levels = ceil( log(leaves) / log(F) ) + = ceil( log(26316) / log(453) ) + = ceil( 10.178 / 6.116 ) + = ceil( 1.6642 ) + = 2 + +d = 2 interior + 1 leaf = 3 pages touched +``` + +At `N = 10⁹` the rowid varint grows to 5 bytes, so `F = floor(4084/11) = 371`, +`leaves = ceil(10⁹/38) = 26,315,790`, and +`ceil(log(26315790)/log(371)) = ceil(17.086/5.916) = ceil(2.888) = 3`, giving +**4 pages touched**. A thousandfold more data costs exactly one more page +touch. That is the whole argument for B-trees over binary trees, and it is +also, deliberately, only half the story — see the caveat at the end of this +step. + +Turso encodes the same arithmetic as a corruption check: + +```rust +// tursodatabase/turso@dd775bc — core/storage/btree.rs +// the depth bound, and the two balance-width constants used in Step 7 + 126 /// Maximum depth of an SQLite B-Tree structure. Any B-Tree deeper than + 127 /// this will be declared corrupt. This value is calculated based on a + 128 /// maximum database size of 2^31 pages a minimum fanout of 2 for a + 129 /// root-node and 3 for all other internal nodes. + 130 /// + 131 /// If a tree that appears to be taller than this is encountered, it is + 132 /// assumed that the database is corrupt. + 133 pub const BTCURSOR_MAX_DEPTH: usize = 20; + 134 + 135 /// Maximum number of sibling pages that balancing is performed on. + 136 pub const MAX_SIBLING_PAGES_TO_BALANCE: usize = 3; + 137 + 138 /// We only need maximum 5 pages to balance 3 pages, because we can guarantee that cells from 3 pages will fit in 5 pages. + 139 pub const MAX_NEW_SIBLING_PAGES_AFTER_BALANCE: usize = 5; +``` + +Check the 20 against its own stated premises — worst case, not the 453 above: + +```text +root fanout 2, internal fanout 3, so with `k` edges from root to leaf: + leaves ≤ 2 · 3^(k−1) + +k = 19: 2 · 3^18 = 774,840,978 < 2^31 = 2,147,483,648 +k = 20: 2 · 3^19 = 2,324,522,934 > 2^31 +``` + +So 20 edges is the first depth that can address a 2³¹-page file under the +worst legal fanout, and anything deeper is by definition corrupt. The constant +is exact, not a round number someone picked. + +**The caveat, and it is this topic's headline.** Everything above computes +pages *touched*. It does not compute time. This topic's `README.md` records +lookups climbing **862 → 1101 ns** between 1e6 and 4e6 keys with height pinned +at 3 the whole way — a 28% slowdown with no change in `d` at all. Height sets +how many pages a lookup touches; what a touch *costs* is set by whether that +page is in CPU cache, and at 270 MB it is not. Read the measured block in +`README.md` and the height ladder in `notes.md` before you let the tidy +logarithm above convince you that fanout is the only lever. + +Why it matters: `L` and `F` are the only two numbers in this entire guide that +the format designer actually controls. Steps 2 through 6 are all, in the end, +about protecting them. + +### Step 2 — the freeblock chain: free space is a linked list in the dead bytes + +> **In:** a page whose header (Step 1) says where the content area starts. +> **Out:** what a delete costs, and the exact rule that turns leftover bytes +> into unusable fragments. + +When a cell is deleted, its bytes become a **freeblock** — a hole inside the +content area, threaded into a singly linked list *through the dead space +itself*. Each freeblock's first 4 bytes are a 2-byte offset of the next +freeblock (0 = end) and a 2-byte size **that includes those 4 header bytes**. +The list head is `BTREE_FIRST_FREEBLOCK` in the page header, and the chain is +kept in **ascending offset order** — `find_free_slot` treats a non-ascending +next-pointer as corruption (`btree.rs:7629–7631`). + +Two rules keep the bookkeeping inside 4 bytes. A freeblock must be at least +**4 bytes**, because anything smaller cannot hold its own next-pointer and +size (`CELL_SIZE_MIN` at `btree.rs:7597`). And leftovers below that threshold +are instead counted in the header's 1-byte fragment counter — bytes that are +free but unaddressable, because nothing can point at them. + +Allocation is first-fit down the chain, in `find_free_slot` +(`btree.rs:7592–7687`). The interesting branch is what happens when the +request *almost* fills a block: + +```rust +// tursodatabase/turso@dd775bc — core/storage/btree.rs — find_free_slot + 7636 let new_size = size - amount; + 7637 // If the freeblock's new size is < CELL_SIZE_MIN, the freeblock is deleted and the remaining bytes + 7638 // become fragmented free bytes. + 7639 if new_size < CELL_SIZE_MIN { + 7640 if page_ref.num_frag_free_bytes() > 57 { + 7641 // SQLite has a fragmentation limit of 60 bytes. + 7642 // check sqlite docs https://www.sqlite.org/fileformat.html#:~:text=A%20freeblock%20requires,not%20exceed%2060 + 7643 return Ok(None); + 7644 } +``` + +Read the guard at 7640 carefully, because the constant is **57, not 60**. The +format's invariant is that the fragment counter never exceeds 60. The leftover +about to be absorbed here is `new_size`, which this branch has just established +is 1, 2, or 3 bytes. So refusing at `> 57` guarantees `57 + 3 = 60` at worst: +the code checks the *pre-*state against a bound that leaves room for the +largest legal increment. Refusing returns `None`, which sends the caller to +Step 3. + +Otherwise the block is carved and — this is the neat part — +`btree.rs:7669–7682` shrinks the block in place and returns +`Ok(Some(cur + new_size))`, i.e. the allocation is taken from the block's +**tail**. The freeblock's 4-byte header stays at its original offset, so +neither the chain's ascending order nor its predecessor's next-pointer needs +touching. A first-fit allocation that hits this branch relinks nothing at all. + +The mirror operation is `free_cell_range` (`btree.rs:8097–…`), which may +coalesce the freed range into the next freeblock, the previous one, or both. +It also has a case worth noticing at `btree.rs:8118–8125`: if the chain is +empty *and* the freed range starts exactly at the content-area boundary, no +freeblock is created — the content-area pointer simply moves right and the +bytes rejoin the unallocated gap. Deleting the most recently inserted cell +leaves no trace. + +Why it matters: a delete costs a 2-byte pointer-array edit plus threading one +hole. All the cleanup is deferred to Step 3 and paid only when space actually +runs short. That is the same bargain LMDB makes at page granularity and +log-structured stores make at file granularity — defer the compaction, pay it +in a batch. + +### Step 3 — defragmentation: compact the holes when first-fit fails + +> **In:** a page where Step 2's first-fit returned `None` even though the +> total free byte count is sufficient. **Out:** why that situation is possible +> at all, and which of two algorithms turso picks. + +**Defragmentation** rewrites all live cells contiguously against the page's +end, then zeroes the freeblock chain and the fragment counter — turning many +scattered holes into one usable gap. It is the answer to the question Step 2 +leaves open: total free space can exceed a request while *no single freeblock* +does. + +The entry point is `defragment_page` (`btree.rs:8422`), and it chooses between +two algorithms at `btree.rs:8435–8440`: + +- the **fast path**, `defragment_page_fast` (`btree.rs:8273`), used when there + are **at most 2 freeblocks** and the fragment count is within the caller's + `max_frag_bytes` budget. Its doc comment (`btree.rs:8268–8272`) gives the + reasoning: with one or two holes it is cheaper to `memmove` the two or three + surviving runs of cells and add a fixed delta to each affected pointer than + to rebuild the page. Note the last line of that comment — the fast path + **does not reduce the fragment count**, it only moves cells. +- the **full path**, reached otherwise, which reconstructs the page cell by + cell. `defragment_page_full` (`btree.rs:8399–8401`) forces it by passing + `max_frag_bytes = -1`: since `num_frag_free_bytes()` is unsigned and + compared as `isize`, `x <= -1` is never true, so the fast-path test at 8436 + always fails. A sentinel, not a budget. + +Question to hold while reading: what triggers defrag, and why is it correct to +move cells but never the pointer array? (The pointer array *is* the sorted index — cells are only ever reached through it, so rewriting cell offsets in -place is invisible to every reader of the page.) +place is invisible to every reader of the page. The array's *order* is the +data structure; the cells' positions are an implementation detail.) -Why it matters: defrag is O(page size) — the fee for Step 1's cheap deletes, -charged rarely and all at once. +Why it matters: defrag is O(page size) — the fee for Step 2's cheap deletes, +charged rarely and all at once. The fast path exists because the common case +after a single delete-then-insert is exactly one freeblock, and paying a full +4 KB rebuild for that would make the deferred-cleanup bargain a bad one. -### Step 3 — varints and the record format +### Step 4 — varints and the record format -A **varint** is an integer encoded in 1–9 bytes, 7 bits per byte, big-endian -(most significant group first), high bit meaning "more bytes follow" — the -9th byte, if reached, carries a full 8 bits (max 9 bytes for a u64). Small -numbers (short lengths, low rowids) cost 1 byte instead of 8. +> **In:** a byte range that Step 5 will identify as one cell's payload. +> **Out:** the integers and column values inside it, decoded without a schema. -On top of varints sits the **record** — the encoding of one row: a -header-size varint, then one **serial type** varint per column (a single -number encoding both the column's type AND its byte length), then the raw -values. Serial types are why pages are schema-less: any page can be decoded -with no schema in hand. +A **varint** is an integer encoded in 1–9 bytes, big-endian (most significant +group first), 7 payload bits per byte with the high bit meaning "another byte +follows". The 9th byte, if reached, contributes a full 8 bits — so the ceiling +is 8×7 + 8 = 64 bits, exactly a `u64`, in at most 9 bytes. Small numbers — +short lengths, low rowids — cost 1 byte instead of 8, which is precisely why +`L = 38` and `F = 453` in Step 1 rather than the smaller figures fixed-width +integers would give. -Why it matters: every cell begins with varints, and every balance or -overflow computation below starts by decoding them. +```rust +// tursodatabase/turso@dd775bc — core/storage/sqlite3_ondisk.rs — read_varint + 1304 pub fn read_varint(buf: &[u8]) -> Result<(u64, usize)> { + 1305 let mut v: u64 = 0; + 1306 for i in 0..8 { + 1307 match buf.get(i) { + 1308 Some(c) => { + 1309 v = (v << 7) + (c & 0x7f) as u64; + 1310 if (c & 0x80) == 0 { + 1311 return Ok((v, i + 1)); + 1312 } + 1313 } +``` -### Step 4 — the four cell formats +Eight iterations of 7 bits, then a separate ninth-byte case at +`sqlite3_ondisk.rs:1320–1331` that shifts by 8 rather than 7. That case also +carries a canonicalization check at `:1326`: a 9-byte encoding whose top 8 bits +are zero is rejected as corrupt, because such a value had a shorter encoding +and a well-formed writer would have used it. The encoder is `write_varint` +(`sqlite3_ondisk.rs:1379`). + +On top of varints sits the **record** — the encoding of one row. Its shape is: +a header-size varint, then one **serial type** varint per column, then the raw +column values back to back. A serial type is a single number that encodes both +the column's type *and* its byte length; text of length `n` is `2n+13` and a +blob of length `n` is `2n+12`, so lengths ride inside the type tag and no +separate length field exists. Five serial types (0, 8, 9, 12, 13) occupy **zero +bytes** in the value area — NULL, integer 0, integer 1, and the empty +blob/string carry their entire value in the tag. + +Turso's record header walk is in `core/types.rs`, and it pins down the one +detail everyone gets wrong on the first read: -There are two b-trees (table trees keyed by rowid, index trees keyed by -column values) times two page levels (interior, leaf), giving exactly four -cell layouts: +```rust +// tursodatabase/turso@dd775bc — core/types.rs — record header parse + 1651 let (header_size, header_varint_len) = read_varint(payload)?; + 1652 let header_size = header_size as usize; + 1653 + 1654 if header_size > payload.len() + 1655 || header_varint_len > payload.len() + 1656 || header_varint_len > header_size + 1657 { +``` -- table interior: `child u32 ∥ rowid varint` — no payload at all; -- table leaf: `size ∥ rowid ∥ payload`; -- index interior: `child ∥ size ∥ payload` — the full key rides along; -- index leaf: `size ∥ payload`. +The slice taken from this is `&payload[header_varint_len..header_size]` (the +same computation at `types.rs:1196`). Both bounds are measured from the *start +of the record*, which means **the header-size varint counts itself**. The check +at 1656 — `header_varint_len > header_size` is corrupt — is exactly the +statement that the header must be at least big enough to contain its own length +field. Per-serial-type decoding is `read_value_serial_type` +(`sqlite3_ondisk.rs:1101`) and `read_value` (`sqlite3_ondisk.rs:973`). + +Why it matters: serial types are why pages are schema-less — any page can be +fully decoded with no catalogue in hand, which is what lets the b-tree layer, +the pager, and every recovery tool work on bytes alone. Every cell begins with +varints, and every balance or overflow computation below starts by decoding +them. + +### Step 5 — the four cell formats + +> **In:** a page type byte from Step 1 and a slot offset from the pointer +> array. **Out:** which of exactly four layouts to parse, and what that choice +> costs in fanout. + +There are two b-tree flavours — **table** trees keyed by rowid, **index** trees +keyed by column values — times two page levels, giving exactly four cell +layouts. Turso declares them as four structs at +`sqlite3_ondisk.rs:774–812`, and the field lists *are* the format: + +| cell | layout | struct | +|---|---|---| +| table interior | `child u32 ∥ rowid varint` | `:782–785` | +| table leaf | `size varint ∥ rowid varint ∥ payload` | `:788–795` | +| index interior | `child u32 ∥ size varint ∥ payload` | `:798–804` | +| index leaf | `size varint ∥ payload` | `:807–812` | + +Three of the four structs also carry `first_overflow_page: Option` — the +tail of Step 6's chain. The table-interior cell is the exception and carries no +payload at all, so it can never overflow. Parsing is `read_btree_cell` +(`sqlite3_ondisk.rs:816`). + +Two consequences fall straight out of the table: + +1. A table-interior cell is 4 bytes of child plus a rowid varint, so the + `slot_int_table = 9` and `F = 453` of Step 1 hold. Table trees are shallow + because their interior cells are nearly empty. +2. An index-interior cell carries the **whole key**. Redo Step 1's interior + arithmetic for a 16-byte index key: the slot costs + `p + c + size_varint + key = 2 + 4 + 1 + 16 = 23` bytes, so + `F = floor(4084 / 23) = 177` — **2.6× worse than the table tree's 453**, + from key bytes alone. + +Note there is **no prefix or suffix truncation anywhere**: turso, like SQLite, +stores full keys in interior cells. Graefe's survey treats suffix truncation as +standard practice and +[reading-graefe-survey.md](reading-graefe-survey.md) works through what it +buys; this topic's `notes.md` measures the same gap from the other end, where a +32-byte key costs 2.5× the interior slots of an 8-byte key. That missing +optimization is your experiment's opening. + +Why it matters: fanout is not a property of the page size, it is a property of +the *key*, and only index trees pay. This is question 1 below. + +### Step 6 — overflow: the exact spill formulas + +> **In:** a payload from Step 4 that does not fit the page. **Out:** exactly +> how many bytes stay local, how many overflow pages result, and why the +> constants are 64/255 and 32/255. + +When a payload is too big for its page, the excess **overflows** into a chain +of dedicated overflow pages, each holding a 4-byte next-page number (0 +terminates) followed by `U − 4` payload bytes. Only a prefix stays "local" in +the cell, and the last 4 bytes of that local region are the first overflow +page number — verified at `sqlite3_ondisk.rs:951–957`, which reads +`unread[cell_len-4 .. cell_len]` as a big-endian `u32` and hands back +`&unread[..cell_len-4]` as the local payload. + +Two thresholds govern the decision, both at `btree.rs:9010–9043`: + +- `max_local` — the largest payload that stays entirely local. + - index pages: `(U − 12) · 64/255 − 23` + - table pages: `U − 35` +- `min_local` — the smallest local prefix a spilled payload may keep: + `(U − 12) · 32/255 − 23`, the **same formula for all four page types** + (`btree.rs:9040–9042`; the `page_type` parameter is `_page_type`, unused). + +At `U = 4096`: + +```text +max_local(index) = floor((4096 − 12) · 64 / 255) − 23 + = floor(4084 · 64 / 255) − 23 + = floor(261376 / 255) − 23 + = 1025 − 23 + = 1002 bytes + +min_local = floor(4084 · 32 / 255) − 23 + = floor(130688 / 255) − 23 + = 512 − 23 + = 489 bytes + +max_local(table) = 4096 − 35 = 4061 bytes +``` -Note: **no prefix/suffix truncation anywhere** — turso (like SQLite) stores -full keys. That's your experiment's opening. Why it matters: table interior -cells are ~13 bytes, so table trees have enormous fanout; index interior -cells carry whole keys, so fat keys directly cost fanout (question 1 below). +These are the identical figures `reading-sqlite-btree.md` derives from +`sqlite/sqlite@951de30` `src/btree.c:3471–3474`, which is the point: the +formulas are file-format constants, not implementation choices, and a rewrite +that changed them would produce unreadable files. + +**Why 64/255 and 32/255?** The doc comment states the design goal outright at +`btree.rs:9015`: "Give a minimum fanout of 4 for index b-trees". Check it. Four +index-interior cells at the maximum local size cost + +```text +per cell: max_local(index) + p + c + size_varint(1002 is 2 bytes) + = 1002 + 2 + 4 + 2 + = 1010 bytes +4 cells = 4040 bytes ≤ U − H_int = 4084 ✓ (44 bytes spare) +5 cells = 5050 bytes > 4084 ✗ +``` + +So four maximal cells fit and five cannot — the fraction 64/255 = 0.25098 is +"just over a quarter", and the −23 is a conservative allowance for cell +overhead (the real overhead here is 8). The second stated goal at +`btree.rs:9016–9017` is the one usually forgotten: keep enough payload local +that **the record header of Step 4 can normally be read without following the +chain**, so a query that only needs column types never touches an overflow +page. + +Now the spill rule itself, which the doc comment states at +`btree.rs:9034–9036` and the code implements: + +```rust +// tursodatabase/turso@dd775bc — core/storage/sqlite3_ondisk.rs — payload_overflows + 2138 if payload_size <= payload_overflow_threshold_max { + 2139 return (false, 0); + 2140 } + 2141 + 2142 let mut space_left = payload_overflow_threshold_min + 2143 + (payload_size - payload_overflow_threshold_min) % (usable_size - 4); + 2144 if space_left > payload_overflow_threshold_max { + 2145 space_left = payload_overflow_threshold_min; + 2146 } + 2147 (true, space_left + 4) +``` -### Step 5 — overflow: the exact spill formulas +Naming the symbols: `P_size` is the total payload, `M = min_local`, +`X = max_local`, and `K = M + (P_size − M) mod (U − 4)` is the candidate local +size at 2142–2143. The rule is **two-branch**, and the second branch at +2144–2145 is the one usually left out of summaries: -When a payload is too big for its page, the excess **overflows** into a -chain of dedicated overflow pages, each holding a 4-byte next-page number -followed by payload bytes (0 terminates the chain); only a prefix of the -payload stays "local" in the cell. The thresholds are exact formulas: +- if `K ≤ X`, keep `K` bytes local; +- **otherwise keep exactly `M` bytes local.** -- `max_local(index) = (usable−12)·64/255 − 23`, - `max_local(table) = usable − 35`, - `min_local = (usable−12)·32/255 − 23`; -- spill rule: keep `min_local + (payload − min_local) % (usable − 4)` bytes - local — sized so the *last* overflow page is exactly full; -- chain format: the last 4 local bytes = next overflow page number - (0 terminates). +The `+ 4` at 2147 is the overflow page pointer, which the cell must also hold. -Why 64/255 and 32/255? Work it out: they bound local payload so a page -always fits **≥4 cells** — fanout survives fat keys. That's the whole point: -overflow trades extra page reads for one value against tree height for -everyone. +The point of the `K` branch is that `P_size − K` is then an exact multiple of +`U − 4`, so **every overflow page including the last is completely full** — no +partly-used page in the chain. Work a case where it holds, a 5,000-byte table +row: -### Step 6 — balance as a resumable state machine +```text +K = 489 + (5000 − 489) mod 4092 + = 489 + 4511 mod 4092 + = 489 + 419 + = 908 ≤ 4061 = X → first branch +remainder = 5000 − 908 = 4092 = exactly one full overflow page +``` -**Balancing** is what runs when an insert overflows a page: pool the cells -of the overfull page, up to two siblings, and the divider cells between them -(the parent's separator entries), then redistribute evenly. Turso's twist on -SQLite: balancing is a **resumable state machine** (`IOResult`) instead of -synchronous recursion, because every page touch may yield for async IO. +Now work the guide's own headline case, a 100 KB row, and watch the property +fail: + +```text +P_size = 102,400 +K = 489 + (102400 − 489) mod 4092 + = 489 + 101911 mod 4092 + = 489 + 3703 + = 4192 > 4061 = X → SECOND branch +local = M = 489 bytes +remainder = 102400 − 489 = 101,911 +pages = ceil(101911 / 4092) = 25 overflow pages +last page = 101911 − 24·4092 = 3,703 of 4,092 bytes used +``` -- `balance_root()`: root overflow ⇒ copy root into a new child, root becomes - interior pointing at it (tree grows up). -- `balance_non_root()`: the ≤3-sibling pool-and-redistribute — sibling pick - prefers the left neighbor, dividers are pulled from the parent into the - pool, and redistribution may mint one new sibling. +So "sized so the last overflow page is exactly full" is a property of the first +branch only. Roughly one remainder in eight lands above `X` for a table leaf +(`K > 4061` requires the modulus to exceed 3,572, i.e. 519 of 4,092 possible +values) and takes the fallback, where the chain does end with a partial page. +Do not state the packing property unconditionally. + +Why it matters: overflow trades extra page reads for one fat value against +tree height for everyone else. Without it, a single 100 KB row would force a +page size that wrecks `L` and `F` for the other million rows. + +### Step 7 — balance as a resumable state machine + +> **In:** a page that Step 2 and Step 3 together could not make room on. +> **Out:** which pages get rewritten, how many come out, and what the async +> rewrite costs in invariants. + +**Balancing** is what runs when an insert overflows a page: pool the cells of +the overfull page, up to two siblings, and the **divider cells** between them +(the parent's separator entries), then redistribute the pool evenly. Turso's +twist on SQLite is that balancing is a **resumable state machine** returning +`IOResult` rather than synchronous recursion, because every page touch may +yield for async IO. + +The dispatcher is `balance` (`btree.rs:2793`), whose match arms at +`btree.rs:2895–2904` route to three routines: + +- `balance_quick` (`btree.rs:2895–2897`) — the append fast path, when the + overflowing page is the rightmost leaf of its subtree. Its doc comment at + `btree.rs:2909–2915` spells out the four steps: allocate a new right sibling, + put the overflow cell in it alone, insert one divider into the parent, and + move the parent's rightmost pointer. No cells are redistributed at all. + `reading-sqlite-btree.md` measures what this saves on a sequential load. +- `balance_root` (`btree.rs:4774`) — root overflow. Allocate a child, copy the + root's contents into it, and the root becomes an interior page pointing at + it. This is the *only* operation that increases tree height, which is why + B-trees grow at the root rather than the leaves. Note `btree.rs:4789–4790`: + when the root is page 1 the copy must skip the 100-byte file header, which is + the whole reason root splits copy rather than simply reusing the page. +- `balance_non_root` (`btree.rs:2995–4309`) — everything else. + +`balance_non_root` is a five-phase state machine, and knowing the phase +boundaries is the difference between reading it and drowning in it: + +| sub-state | lines | what it does | +|---|---|---| +| `NonRootPickSiblings` | 3014–3271 | choose up to `MAX_SIBLING_PAGES_TO_BALANCE` = 3 neighbours | +| `NonRootDoBalancing` | 3272–3814 | pool cells, size the output pages, decide the split points | +| `NonRootDoBalancingAllocate` | 3815–3855 | allocate any new sibling pages needed | +| `NonRootDoBalancingFinish` | 3856–4281 | write cells into the new pages, rewrite parent dividers | +| `FreePages` | 4282–4309 | return now-empty siblings to Step 8's freelist | + +Inside the pooling phase there is a distinction that is easy to miss and is one +of the more elegant things in the format. At `btree.rs:3507–3517`, **for table +leaves the divider cells are not pooled at all** — they stay in the parent as +bookkeeping — while for index and interior pages they *are* pooled. The reason +is Step 5's table: a table-interior divider is only `(child, rowid)`, and after +redistribution the correct rowid is simply the largest one on the page to its +left, so the divider can be *regenerated* rather than moved. An index divider +carries a real key that exists nowhere else and must be redistributed like any +other cell. The assertions at 3518–3529 exist to catch getting this backwards. + +The output width is bounded by `MAX_NEW_SIBLING_PAGES_AFTER_BALANCE = 5` +(`btree.rs:139`), asserted at `btree.rs:3597–3600` with the message "it is +corrupt to require more than 5 pages to balance 3 siblings". So the contract is +**≤3 pages in, ≤5 pages out**, and every balance is O(1) pages regardless of +tree size. + +The sizing itself happens in two passes, and turso quotes SQLite verbatim for +the second: +```rust +// tursodatabase/turso@dd775bc — core/storage/btree.rs — balance_non_root, sizing pass 2 + 3689 // Comment borrowed from SQLite src/btree.c + 3690 // The packing computed by the previous block is biased toward the siblings + 3691 // on the left side (siblings with smaller keys). The left siblings are + 3692 // always nearly full, while the right-most sibling might be nearly empty. + 3693 // The next block of code attempts to adjust the packing of siblings to + 3694 // get a better balance. + 3695 // + 3696 // This adjustment is more than an optimization. The packing above might + 3697 // be so out of balance as to be illegal. For example, the right-most + 3698 // sibling might be completely empty. This adjustment is not optional. ``` + +Pass one (`btree.rs:3588–3676`) greedily packs cells left until each page is +full, spilling into a new page when needed. Pass two (`btree.rs:3699–3793`) +walks the pages right-to-left moving cells back. The comment is worth taking at +its word: the first pass can produce an *illegal* page, not merely an ugly one, +so the second pass is a correctness step. This is the same comment +`reading-sqlite-btree.md` anchors at `sqlite/sqlite@951de30` +`src/btree.c:8636–8646`; the C original and the Rust copy agree line for line. + +```text balance_non_root, 2 siblings + overfull page: parent: [ ... D1 ... D2 ... ] D = divider cells │ │ │ [sib L] [OVERFULL] [sib R] └──────── pool: L + D1 + full + D2 + R ────────┘ - redistribute evenly ⇒ 2–4 pages, new dividers up + redistribute ⇒ up to 5 pages, new dividers up + (table leaves: D1, D2 stay in the parent — btree.rs:3511) ``` -Why it matters: pooling ≤3 siblings bounds the work per balance while -leaving pages fuller than a naive half/half split — and the state-machine -shape forces every intermediate state to be resumable (question 3 below asks -what invariant that requires). +Why it matters: pooling ≤3 siblings bounds the work per balance while leaving +pages fuller than a naive half/half split would. And the state-machine shape +forces every intermediate state to be resumable — which is question 3 below. + +### Step 8 — the freelist: recycling whole pages -### Step 7 — the freelist: recycling whole pages +> **In:** the pages Step 7's `FreePages` phase emptied, plus whole trees +> dropped by DDL. **Out:** where those pages go, and why the file never +> shrinks. -Separately from Step 1's *within-page* holes, whole pages freed by drops and +Separately from Step 2's *within-page* holes, whole pages freed by drops and balances go on the **freelist** — a chain of **trunk pages**, each holding a -next-trunk u32, a leaf-count u32, and then an array of free page numbers -(the "leaves" are just free page IDs, never read). +next-trunk `u32`, a leaf-count `u32`, and then an array of free page numbers. +The "leaves" are just page IDs; their contents are never read. Turso documents +the layout in a comment and four constants: + +```rust +// tursodatabase/turso@dd775bc — core/storage/sqlite3_ondisk.rs + 85 // Freelist trunk page layout: + 86 // - Bytes 0-3: Page number of next freelist trunk page (0 if none) + 87 // - Bytes 4-7: Number of leaf page pointers on this trunk page + 88 // - Bytes 8+: Array of 4-byte leaf page pointers + 89 pub const FREELIST_TRUNK_OFFSET_NEXT_TRUNK_PTR: usize = 0; + 90 pub const FREELIST_TRUNK_OFFSET_LEAF_COUNT: usize = 4; + 91 pub const FREELIST_TRUNK_OFFSET_FIRST_LEAF_PTR: usize = 8; + 92 pub const FREELIST_TRUNK_HEADER_SIZE: usize = 8; + 93 pub const FREELIST_LEAF_PTR_SIZE: usize = 4; +``` + +Freeing is `Pager::free_page` (`pager.rs:5019–5154`) — note the name, because +there is no `add_page_to_freelist` in this tree. Its capacity test is at +`pager.rs:5103–5104`: + +```text +max_free_list_entries = U / FREELIST_LEAF_PTR_SIZE − RESERVED_SLOTS + = 4096 / 4 − 2 + = 1024 − 2 + = 1022 leaf pointers per trunk page +``` -Allocation pops a leaf number off the current trunk; when a trunk runs -empty, the trunk page ITSELF becomes the allocated page — the list consumes -its own skeleton. Freeing appends to the trunk (or starts a new one). +The `RESERVED_SLOTS = 2` (`pager.rs:5022`) are the next-trunk and leaf-count +`u32`s. If there is room, the page number is appended to the current trunk +(`pager.rs:5106–5126`); if not, the page being freed **becomes a new trunk** +pointing at the old one (`pager.rs:5130–5148`). So the freelist's own index +structure costs nothing extra — it is built out of the pages it is tracking. +One trunk per 1,022 free pages is an overhead of `1/1023` ≈ **0.098%** of the +freed space. + +Allocation is `Pager::allocate_page` (`pager.rs:5250`), which prefers reuse: +read the first trunk (`pager.rs:5301–5314`), and if it has leaves, pop one +(`ReuseFreelistLeaf`, `pager.rs:5390–5447`). Only when the freelist is empty +(`pager.rs:5302–5303`) does it fall through to `AllocateNewPage` +(`pager.rs:5450`) and grow the file. When a trunk runs out of leaves the trunk +page ITSELF is handed out as the allocation — the list consumes its own +skeleton. Why it matters: the file never shrinks on delete; it recycles. This is the -page-granularity mirror of the freeblock story, and the structure your -capstone's pager will need too. +page-granularity mirror of Step 2's freeblock story — same deferred-cleanup +bargain, three orders of magnitude up — and it is the structure your capstone's +pager will need too. It is also why `VACUUM` exists as a separate, explicit, +whole-file operation. ## Where each step lives in the code -Line numbers drift — navigate by symbol name. - -- **Step 1 — slotted page + freeblocks**: header parsing `btree.rs:76–124` - (offsets in README §1); `find_free_slot()` — `btree.rs:7592–7680` walks the - freeblock chain (each freeblock: 2B next-ptr + 2B size, threaded through - the content area). Minimum slot 4 bytes; smaller leftovers become the - header's `fragmented_bytes` counter. -- **Step 2 — defragment**: `btree.rs:8273–8444` — fast path when ≤2 - freeblocks, slow path compacts everything. -- **Step 3 — varints + records**: `read_varint` / `write_varint` — - `sqlite3_ondisk.rs:1304–1336 / 1379–1421`; record decoding (header-size - varint, per-column serial-type varints, then values) — - `sqlite3_ondisk.rs:1101–1237`. -- **Step 4 — cell formats**: structs `sqlite3_ondisk.rs:775–812`, parsing - :826–930. -- **Step 5 — overflow**: thresholds `btree.rs:9019–9042`; spill rule - `sqlite3_ondisk.rs:2130–2148`; chain format `sqlite3_ondisk.rs:951–961`. -- **Step 6 — balance**: `balance_root()` — `btree.rs:4774–4852`; - `balance_non_root()` — `btree.rs:2995–4087`, sibling pick at :3305–3375 - (left preferred, dividers pulled from parent into the pool), - redistribution + new-sibling creation at :3430–3680. Trigger: insert - overflows the page (`btree.rs:2903` — split path after `split_cell()` - can't fit). -- **Step 7 — freelist**: trunk page format `sqlite3_ondisk.rs:89–93`; - `allocate_page()` — `pager.rs:5250–5448`; `add_page_to_freelist()` — - `pager.rs:5101–5145`. +All anchors are `tursodatabase/turso@dd775bc`. Symbol names are given so the +anchors survive a re-pin. + +- **Step 1 — slotted page geometry**: header offset module `btree.rs:84–124` + with the layout diagram at `:76–83`; the four size constants + `sqlite3_ondisk.rs:80–83`; `BTCURSOR_MAX_DEPTH` and the two balance-width + constants `btree.rs:126–139`. +- **Step 2 — freeblocks**: `find_free_slot` `btree.rs:7592–7687` (ascending-order + check `:7629–7631`; the 57-byte fragment guard `:7640`; tail-carve return + `:7669–7682`); `free_cell_range` `btree.rs:8097–…` with the + no-freeblock-needed case at `:8118–8125`; `compute_free_space` + `btree.rs:8689`. +- **Step 3 — defragment**: dispatcher `defragment_page` `btree.rs:8422`, path + choice `:8435–8440`; `defragment_page_fast` `btree.rs:8273` (rationale + `:8268–8272`); `defragment_page_full` `btree.rs:8399–8401`; + `defragment_page_for_insert` `btree.rs:8412`. +- **Step 4 — varints + records**: `read_varint` `sqlite3_ondisk.rs:1304–1337` + (9-byte case `:1320–1331`, canonicalization check `:1326`); `write_varint` + `sqlite3_ondisk.rs:1379`; record header parse `core/types.rs:1650–1660` and + `:1187–1196`; `read_value_serial_type` `sqlite3_ondisk.rs:1101`; `read_value` + `sqlite3_ondisk.rs:973`. +- **Step 5 — cell formats**: `BTreeCell` enum `sqlite3_ondisk.rs:774–779`, the + four structs `:782–812`, parser `read_btree_cell` `:816`. +- **Step 6 — overflow**: `payload_overflow_threshold_max` `btree.rs:9019–9028` + and `payload_overflow_threshold_min` `btree.rs:9040–9043`, with the design + rationale in the doc comments at `:9010–9018` and `:9030–9038`; the spill + rule `payload_overflows` `sqlite3_ondisk.rs:2132–2148` (fallback branch + `:2144–2145`); chain pointer extraction `sqlite3_ondisk.rs:951–957`. +- **Step 7 — balance**: dispatcher `balance` `btree.rs:2793`, arms `:2895–2904`; + `balance_quick` doc `:2909–2915`; `balance_root` `btree.rs:4774` (page-1 + header offset `:4789–4790`); `balance_non_root` `btree.rs:2995–4309` with + sub-states `NonRootPickSiblings` `:3014`, `NonRootDoBalancing` `:3272`, + `NonRootDoBalancingAllocate` `:3815`, `NonRootDoBalancingFinish` `:3856`, + `FreePages` `:4282`; table-leaf divider rule `:3507–3517`; five-page assert + `:3597–3600`; the borrowed SQLite comment and rebalancing pass `:3689–3793`. +- **Step 8 — freelist**: trunk layout `sqlite3_ondisk.rs:85–93`; + `Pager::free_page` `pager.rs:5019–5154` (capacity `:5103–5104`, append + `:5106–5126`, new trunk `:5130–5148`); `Pager::allocate_page` `pager.rs:5250` + (trunk read `:5301–5314`, `ReuseFreelistLeaf` `:5390–5447`, + `AllocateNewPage` `:5450`). ## Questions to answer in notes.md 1. Why do table-btree interior cells store only rowids (no payload) while index-btree interior cells carry the full key? What does that do to fanout? -2. The freeblock minimum is 4 bytes and `fragmented_bytes` caps at 60 in SQLite — - what goes wrong without defragmentation? When must `allocateSpace` defrag even - though total free space suffices? -3. Turso's balance yields mid-operation for IO. What invariant must hold at every - yield point so a concurrent reader (or a crash) never sees a broken tree? - (Hint: WAL — pages aren't durable until commit; in-memory the cursor holds refs.) + Use Step 1's and Step 5's numbers: 453 versus 177 at a 16-byte key. +2. The freeblock minimum is 4 bytes and the fragment counter is capped at 60 — + what goes wrong without defragmentation? Describe a page where the total + free space exceeds a request but `find_free_slot` still returns `None`. +3. Turso's balance yields mid-operation for IO. What invariant must hold at + every yield point so a concurrent reader (or a crash) never sees a broken + tree? (Hint: WAL — pages aren't durable until commit; in-memory the cursor + holds refs.) +4. Step 6's spill rule has two branches. Construct a payload size that takes + each, and say what the chain's last overflow page looks like in both cases. ## Done when -You can write the byte layout of a table-leaf page containing two cells and one -freeblock, from memory, and explain what balance_non_root pools and why ≤3. +Answer each before unfolding it. + +- [ ] Write the byte layout of a table-leaf page holding two cells and one + freeblock, from memory — every header field, the pointer array, and the + freeblock's own 4 bytes. + +
+Answer + +Header, 8 bytes (`btree.rs:84–124`): byte 0 page type (`0x0d` = table leaf); +bytes 1–2 first freeblock offset; bytes 3–4 cell count = 2; bytes 5–6 cell +content area start; byte 7 fragment count. No rightmost pointer — that field +exists only on interior pages, which is what makes the leaf header 8 rather +than 12. + +Bytes 8–11: the cell pointer array, two 2-byte big-endian offsets, in **key +order**, not allocation order. + +Then the unallocated gap, then the content area growing leftward from `U`. The +freeblock sits inside the content area: 2 bytes next-offset (0 if it is the +only one) then 2 bytes size, and **that size includes these 4 header bytes**. +The header's byte 1–2 field points at it. + +Sanity check the arithmetic closes: content-area start + (sum of live cell +sizes) + (sum of freeblock sizes) + fragment count = `U`. + +
+ +- [ ] Explain what `balance_non_root` pools, and why the bound is 3. + +
+Answer + +It pools the cells of the overfull page plus up to two neighbouring siblings, +plus the divider cells separating them in the parent — except for table leaves, +where dividers stay in the parent because a `(child, rowid)` divider can be +regenerated from the largest rowid on the page to its left +(`btree.rs:3507–3517`). + +The bound is `MAX_SIBLING_PAGES_TO_BALANCE = 3` (`btree.rs:136`), with output +bounded by `MAX_NEW_SIBLING_PAGES_AFTER_BALANCE = 5` (`btree.rs:139`), asserted +at `:3597–3600`. Three is the smallest window that lets an underfull page draw +from *both* neighbours, so it can usually be fixed without changing the +parent's cell count; and holding the window at a constant makes every balance +O(1) pages no matter how big the tree is. Wider windows pack pages better but +make each insert's worst case worse. + +
+ +- [ ] Compute the interior fanout `F` of a table b-tree on a 4,096-byte page + with rowids under 2²¹, and the number of pages a lookup touches at + `N = 10⁶`. Name every term. + +
+Answer + +`U = 4096`, interior header `H_int = 12`, cell pointer `p = 2`, child pointer +`c = 4`, rowid varint 3 bytes at that magnitude. Slot = `2 + 4 + 3 = 9`, so +`F = floor((4096 − 12)/9) = floor(4084/9) = 453`. + +Leaves: with a 100-byte payload the leaf slot is `2 + 1 + 3 + 100 = 106` and +`L = floor(4088/106) = 38`, so `leaves = ceil(10⁶/38) = 26,316`. Interior +levels = `ceil(log 26316 / log 453) = ceil(10.178/6.116) = ceil(1.664) = 2`, +so **3 pages touched** including the leaf. + +And the point of Step 1's caveat: that 3 does not predict latency. This topic's +`README.md` shows lookups going 862 → 1101 ns while `d` stays at 3. + +
+ +- [ ] State the fragment-counter guard in `find_free_slot` and explain why the + constant is 57. + +
+Answer + +`if page_ref.num_frag_free_bytes() > 57 { return Ok(None) }` +(`btree.rs:7640`). The branch it guards absorbs a leftover of 1, 2, or 3 bytes +into the fragment counter, and the format's invariant is that the counter never +exceeds 60. Testing the pre-state against 57 leaves headroom for the largest +legal increment: `57 + 3 = 60`. Refusing returns `None`, which sends the caller +to defragmentation (Step 3). + +Note what the 60 is: a **validity invariant** — a page whose counter exceeds it +is corrupt — not a threshold that triggers anything by itself. + +
+ +- [ ] Give the two branches of the overflow spill rule, and say which one + leaves a partially-filled last overflow page. + +
+Answer + +With `M = min_local`, `X = max_local`, `P_size` the payload and +`K = M + (P_size − M) mod (U − 4)` (`sqlite3_ondisk.rs:2142–2143`): if `K ≤ X` +keep `K` local; otherwise keep exactly `M` (`:2144–2145`). Either way the cell +also holds a 4-byte overflow pointer (`:2147`). + +The first branch makes `P_size − K` an exact multiple of `U − 4`, so every +overflow page including the last is full. The **second** branch is the one that +leaves a partial page: a 100 KB row at `U = 4096` gives `K = 4192 > 4061 = X`, +falls back to 489 local bytes, and its 25th and last overflow page holds +3,703 of 4,092 bytes. + +
+ +- [ ] Say where a page freed by a balance goes, and why the file does not + shrink. + +
+Answer + +`balance_non_root`'s `FreePages` sub-state (`btree.rs:4282–4309`) calls +`Pager::free_page` (`pager.rs:5019`), which appends the page number to the +current freelist trunk if it has room — 1,022 leaf pointers at `U = 4096`, +from `U/4 − 2` at `pager.rs:5103–5104` — or turns the freed page into a new +trunk pointing at the old one (`:5130–5148`). + +The file does not shrink because nothing ever truncates it: `allocate_page` +(`pager.rs:5250`) reuses freelist pages first and only extends the file when +the list is empty (`:5302–5303` → `AllocateNewPage` `:5450`). Reclaiming the +space to the filesystem is a separate explicit operation, `VACUUM`. + +
## References **Code** -- [turso](https://github.com/tursodatabase/turso) — - `core/storage/btree.rs` (slotted-page ops, balance state machines), - `core/storage/sqlite3_ondisk.rs` (overflow, varints, cell formats), - `core/storage/pager.rs` (freelist) — local clone at `~/repos/turso`; - line numbers drift, navigate by symbol name. Extends topic 1's - [reading-turso-btree.md](../01-storage-engine-landscape/reading-turso-btree.md) +- [turso](https://github.com/tursodatabase/turso), pinned at `dd775bc` — + `core/storage/btree.rs` (slotted-page ops, overflow thresholds, balance state + machines), `core/storage/sqlite3_ondisk.rs` (cell formats, varints, spill + rule, freelist trunk layout), `core/storage/pager.rs` (page allocation and + freeing), `core/types.rs` (record header walk). Local clone at `~/repos/turso`; + confirm the pin with `tools/pinned-source.py ref turso`. +- Extends topic 1's + [reading-turso-btree.md](../01-storage-engine-landscape/reading-turso-btree.md), + which covers the cursor/seek/insert surface this guide descends beneath. +- [sqlite](https://github.com/sqlite/sqlite), pinned at `951de30` — the C + original. `src/btree.c:3471–3474` carries the same overflow formulas and + `:8636–8646` the same rebalancing comment turso quotes at `btree.rs:3689`. + +**In this topic** +- [reading-sqlite-btree.md](reading-sqlite-btree.md) — the same algorithms in + C, including `balance_quick` and the page-number-ordering optimization turso + has not adopted. +- [reading-sqlite-file-format.md](reading-sqlite-file-format.md) — the on-disk + field definitions, byte offset by byte offset; it defers the spill arithmetic + to Step 6 here. +- [reading-graefe-survey.md](reading-graefe-survey.md) — what suffix truncation + would buy the index-interior fanout of Step 5, and why neither SQLite nor + turso implements it. +- `README.md` and `notes.md` — this topic's measured height ladder, and the + reason Step 1's logarithm is only half the story. + +**Docs** +- [SQLite file format](https://www.sqlite.org/fileformat.html) — §1.6 for the + b-tree page header and the 60-byte fragment bound turso cites verbatim at + `btree.rs:7641–7642`. diff --git a/topics/04-lsm-deep-dive/README.md b/topics/04-lsm-deep-dive/README.md index 9e7823b..dd936a3 100644 --- a/topics/04-lsm-deep-dive/README.md +++ b/topics/04-lsm-deep-dive/README.md @@ -52,21 +52,28 @@ flowchart LR ``` Reads run the same path in reverse: memtable → sealed → L0 (every run!) → one -segment per deeper level (disjoint ⇒ binary search by key range). Every skipped -disk probe is a bloom filter earning its bits. +table per deeper level (disjoint ⇒ binary search by key range). Every skipped +disk probe is a bloom filter earning its bits. ("Table" is the `lsm-tree` +crate's word for what RocksDB calls an SST and what a lot of writing calls a +segment; this topic follows the crate, because the crate is what you will read.) ## 2. Inside an SST ``` - ┌─────────────┬─────────────┬──────┬─────────────┬────────┬─────────┐ - │ data block │ data block │ … │ filter block│ index │ trailer │ - │ (~4KB, LZ4) │ │ │ (bloom) │ block │ /meta │ - └─────────────┴─────────────┴──────┴─────────────┴────────┴─────────┘ + ┌─────────────┬─────────────┬──────┬─────────────┬─────────────┬─────────┐ + │ data block │ data block │ … │ index block │ filter block│ trailer │ + │ (~4KB, LZ4) │ │ │ │ (bloom) │ /meta │ + └─────────────┴─────────────┴──────┴─────────────┴─────────────┴─────────┘ inside a data block (restart interval 16): [FULL key ∥ v][shared=5,rest ∥ v][shared=7,rest ∥ v]…[FULL key]…[restart offsets] ▲ binary search over restart points, linear decode between them ``` +The index comes before the filter because that is the order `Writer::finish` +writes them in — index at `src/table/writer/mod.rs:384`, filter at `:388`. The +trailer is what makes either findable, so the on-disk order is a free choice, +and different engines make it differently; read the writer rather than assuming. + Prefix truncation *inside* blocks (vs topic 3's B-tree pages which stored full keys) works because blocks are immutable — write once, no in-place updates to break the delta chain. Immutability is the LSM superpower: checksums per block, @@ -137,7 +144,7 @@ Optionally follow skyzh/mini-lsm alongside — but the point here is the *measur `Leveled` (ratio 10) and `Tiered` (K=4). 4. **The experiment** (`src/bin/write_amp.rs`): load 10M keys (uniform random overwrite, 3 passes), count bytes written to disk / bytes of user data — - write amp per strategy. Also record: read amp (segments probed per get, + write amp per strategy. Also record: read amp (tables probed per get, bloom hits/misses), space amp (dir size / live data). Fill the RUM table with MEASURED numbers. diff --git a/topics/04-lsm-deep-dive/reading-compaction-design-space.md b/topics/04-lsm-deep-dive/reading-compaction-design-space.md index 621d2e1..3a023e9 100644 --- a/topics/04-lsm-deep-dive/reading-compaction-design-space.md +++ b/topics/04-lsm-deep-dive/reading-compaction-design-space.md @@ -1,12 +1,17 @@ # Compaction is four axes, not two strategies "Leveled vs tiered" is a false binary: a compaction policy is an independent -choice on four design axes — trigger, layout, granularity, movement — and +choice on four design axes — trigger, layout, granularity, data movement — and every system you've read in this topic sits somewhere in that grid. Before the paper, this chapter builds the four axes one at a time, with the systems you already know as coordinates. This is the taxonomy chapter; read it LAST of the four papers, because it organizes the other three. +Every axis name, option list and number below is checked against the paper — +Sarkar, Staratzis, Zhu, Athanassoulis, *Constructing and Analyzing the LSM +Compaction Design Space*, PVLDB 14(11): 2216-2229, 2021 — and cited to the +section, observation or takeaway it came from. + ## The problem in one sentence After three papers and two codebases you have seen at least five distinct @@ -19,121 +24,481 @@ vocabulary. ### Step 1 — a compaction policy is a bundle of independent decisions -Every compaction, in every engine, answers the same four questions: *when* -do we compact, *what shape* must the levels have, *how much* data does one -job move, and *does the data actually get rewritten*. "Leveled" and -"tiered" are bundles — prepackaged answers to all four at once — which hides -the fact that the answers are independently choosable: +> **In:** the compaction behaviours you have already met — lsm-tree's leveled +> strategy, RocksDB's scores and universal compaction, Dostoevsky's K and Z. +> **Out:** four named questions every one of them answers, and the observation +> that the answers are independently choosable — which Steps 2-5 take one at a +> time. -``` - a compaction policy = choice on each axis: +Every compaction, in every engine, answers the same four questions. The paper's +own phrasing (§3.1), verbatim: - 1. TRIGGER when? level saturation / #runs / staleness / space amp - 2. DATA LAYOUT what shape? leveling / tiering / 1-leveling / L-leveling / hybrid - 3. GRANULARITY how much at once? whole level / one file (RocksDB) / few files - 4. DATA MOVEMENT who moves? full merge / trivial move (relink non-overlapping) ``` + 1) Compaction trigger: When to re-organize the data layout? + 2) Data layout: How to lay out the data physically on storage? + 3) Compaction granularity: How much data to move at-a-time during + layout re-organization? + 4) Data movement policy: Which block of data to be moved during + re-organization? + — Constructing and Analyzing…, §3.1 +``` + +"Leveled" and "tiered" are answers to question 2 *only*. They get used as if +they answered all four, which is what hides the other three from view. + +Two structural facts about the grid, from §3.2. First, **data layout is +single-valued; trigger, granularity and data movement policy are +multi-valued** — an engine has exactly one layout but may have several triggers +and several file-picking rules active at once. Second, the space is large: +"Plugging in some typical values for the cardinality of the primitives, we +estimate the cardinality of the compaction universe as **>10⁴**, a vast yet +largely unexplored design space." -Unbundling matters because the axes control *different* observable costs — -Steps 2–5 take them one at a time. +Unbundling matters because the axes control *different* observable costs. ### Step 2 — axis 1, the trigger: what event starts a compaction -The trigger is the predicate that fires a compaction job. The familiar one -is **saturation** — a level exceeds its size target (RocksDB's score ≥ 1.0 -from the compaction chapter). But nothing forces that choice: you can -trigger on **run count** (tiered's "K runs accumulated"), on **staleness** -(data untouched for N hours gets merged — useful for TTL workloads), or -directly on **space amplification** (compact when dir size / live data -exceeds 1.5). The paper's empirical finding worth flagging now: at low write -rates, the *trigger* choice moves point-lookup latency more than the layout -does — because the trigger decides how long overlapping runs linger before -being merged away. +> **In:** the four questions from Step 1. +> **Out:** the five triggers in production use, and the recognition that the +> familiar one (level saturation) is a choice, not a law — Step 3 then shows +> that trigger and layout are genuinely separable. + +The trigger is the predicate that fires a compaction job. §3.1.1 lists the +common ones: + +``` + i) Level saturation: level size goes beyond a nominal threshold + ii) #Sorted runs: sorted run count for a level reaches a threshold + iii) File staleness: a file lives in a level for too long + iv) Space amplification (SA): overall SA surpasses a threshold + v) Tombstone-TTL: files have expired tombstone-TTL + — §3.1.1 +``` + +The familiar one is **level saturation** — RocksDB's score ≥ 1.0 from the +compaction chapter, where the score is bytes-in-level ÷ target-bytes-for-level +(`db/version_set.cc:4136-4137`). The paper notes a wrinkle worth knowing: some +engines measure saturation by *file count* rather than bytes, which "works only +when all immutable files are of equal size, or for systems that have a tunable +file size" — RocksDB's L0 trigger (`level0_file_num_compaction_trigger`, default +4) is exactly this variant, and it is why L0 is special-cased in its scoring +code. + +The other four are not exotic. **#Sorted runs** is tiering's trigger and, with +space amplification, is what RocksDB's universal compaction uses (§3.2: +"compactions are triggered when either (a) the number of sorted runs in a level +or (b) the estimated space amplification in the tree reaches certain +thresholds. This interpretation of tiering is also referred to as universal +compaction in systems like RocksDB"). **Tombstone-TTL** exists because a delete +is not persistent until its tombstone reaches the last level — a compaction +trigger driven by privacy regulation rather than performance, which is a +genuinely different reason for a database to do work. ### Step 3 — axis 2, the layout: what shape the levels are kept in -The layout is the invariant about runs per level — the axis Dostoevsky -already turned into a dial. **Leveling** = 1 run per level; **tiering** = up -to T runs per level; **1-leveling / L-leveling** = tiering with a leveled -first or last level (L-leveling is exactly lazy leveling); hybrids mix per -level. This is the only axis the "leveled vs tiered" vocabulary ever named, -and Steps 4–5 are the two whole axes it left silent. +> **In:** the trigger from Step 2, which decides *when*. +> **Out:** the invariant that decides *what shape* — five options, of which the +> vocabulary in your head names only two — plus the one measured result that +> attributes point-lookup latency to this axis. + +The layout is the invariant about runs per level — the axis Dostoevsky already +turned into a dial. §3.1.2's list: + +``` + i) Leveling: one sorted run per level + ii) Tiering: multiple sorted runs per level + iii) 1-leveling: tiering for Level 1; leveling otherwise + iv) L-leveling: leveling for last level; tiering otherwise + v) Hybrid: a level can be tiering or leveling independently + — §3.1.2 +``` + +Read iii) and iv) carefully, because they are easy to get backwards. +**1-leveling is leveling with a *tiered first level*** — laziness at the top, to +absorb ingest bursts without stalling. **L-leveling is tiering with a *leveled +last level*** — which is exactly Dostoevsky's Lazy Leveling, and Table 1 files +Dostoevsky under L-leveling. They are near-opposites, and the paper reaches for +1-leveling far more often than you would guess, because: + +> **1-Lvl** … is the default data layout for RocksDB. (§3.2) + +RocksDB's L0 tolerates multiple overlapping runs and "is allowed to grow +perpetually in order to avoid write-stalls in ingestion-heavy workloads" +(§3.1.2). So the engine everyone calls "leveled" is, in this taxonomy, a hybrid +— and Table 1 lists its layout as "Leveling / 1-Leveling" for exactly that +reason. + +This is the axis that moves point-lookup latency, and the paper measures it +(**O4**, §5.1.2): point lookups are best with `Full` leveling and worst with +tiering — mean latency **1.1-1.9× higher** for tiering on existing keys and +**~2.2× higher** on non-existing keys. Note that this is *far short* of the +textbook prediction: "For non-empty lookups in a tree with size ratio T, +theoretically, the lookup cost for tiering should be T× higher than its leveling +equivalent." The measured gap is 2.2×, not 10×, and the paper explains why — +RocksDB's tiering keeps fewer sorted runs than textbook tiering, and the block +cache plus lookup temporality absorb much of the rest. A 5× discrepancy between +the asymptotic model and the measurement, explained rather than hidden, is worth +more than either number alone. ### Step 4 — axis 3, granularity: how much data one job moves -Granularity is the size of a single compaction job's input. **Whole-level** -compaction (your mini-LSM, the 1996 paper's rolling merge in spirit) merges -an entire level at once: with a 2.5 GB L2 that is one job occupying the disk -for tens of seconds, and every one of those seconds is back-pressure — -foreground writes stall in bursts. **File-granularity** compaction -(RocksDB: pick *one* ~64 MB file plus its next-level overlaps) does the -same total work as many small jobs spread over time. Same throughput, -radically different p99.9: granularity is a **tail-latency knob, not a -throughput knob** — the paper's cleanest finding, and topic 2's -rehash-spike lesson (one big pause vs many amortized ones) at LSM scale. - -### Step 5 — axis 4, data movement: merge bytes or relink them - -Data movement asks whether a compaction physically rewrites data or merely -re-labels it. A **full merge** reads, merges, and rewrites every input byte -— the default assumption. A **trivial move** applies when an input file -does not overlap anything at the destination level: the engine just edits -metadata to say the file now belongs to the next level — **zero bytes of -IO** (lsm-tree's `Choice::Move`, RocksDB's trivial move). For sequential or -bulk-load ingest this axis dominates everything: a sorted snapshot can -cascade to the bottom level entirely by relinking, write amp 1.0 — which is -your M4 graph-snapshot question answered by an axis the two-word vocabulary -couldn't even express. +> **In:** a layout (Step 3) and a trigger that just fired (Step 2). +> **Out:** how big the resulting job is — the axis that turns out to control +> tail latency, with the measured spread that proves it. + +Granularity is the size of a single compaction job's input. §3.1.3's list: + +``` + i) Level: all data in two consecutive levels + ii) Sorted runs: all sorted runs in a level + iii) Sorted file: one sorted file at a time + iv) Several sorted files: several sorted files at a time + — §3.1.3 +``` + +**Full compaction** (level granularity — your mini-LSM, and the 1996 paper's +rolling merge in spirit) merges an entire level at once: with a 2.5 GB L2 that +is one job occupying the disk for tens of seconds, and every one of those +seconds is back-pressure. **Partial compaction** (file granularity — RocksDB: +pick one ~64 MB file plus its next-level overlaps) does comparable total work as +many small jobs spread over time. + +The measurements, all from §5.1.1 on the setup in Step 6: + +- **O1** — compaction data movement dwarfs the data itself: `Full` moves **63×** + the ingested bytes (32× read + 31× written); `Tier` **23×**. +- **O2** — partial compaction moves **34%-56% less data than `Full`**, for two + reasons the paper separates: (1) a file with no overlap in its parent level is + "only logically merged" — a **pseudo-compaction**, pure metadata, zero IO; and + (2) a smaller granularity lets you *choose* a cheap file (that is Step 5's + axis). Partial strategies run **4× more compaction jobs**, which is the number + of tree levels. +- **TA I** — "Full-level compactions perform about 1/L times fewer compactions + than partial compaction routines, however, full-level compaction moves nearly + **2L times more data per compaction**." +- **O3** — `Full`'s mean compaction latency is 1.2-1.9× higher than partial + leveling and 2.1× higher than tiering. Also: CPU is **~50%** of compaction + time regardless of strategy, dominated by the in-memory sort-merge — so + compaction is not the pure-IO activity it is usually drawn as. +- **TA II**, the number to remember — "Tail write stall for `Tier` is **~25 ms**, + while for partial leveling (`Old`) it is as low as **1.3 ms**." + +That last pair is a **19× spread in tail write latency** between two +configurations of the same engine on the same workload. Granularity is a +**tail-latency knob** — topic 2's rehash-spike lesson (one big pause versus many +amortized ones) at LSM scale. + +One correction to the folklore, from the paper's own numbers: partial compaction +is *not* merely the same work rearranged. §2 describes it that way ("does not +radically change the total amount of data movement… but amortizes this data +movement uniformly over time"), but O2 measures 34-56% *less* total movement, +because finer granularity is what makes pseudo-compactions and cheap-file +picking possible at all. When the background section and the measurement +disagree, take the measurement. + +Pseudo-compaction is the same optimization the reference crates call a **trivial +move** — `lsm-tree`'s `Choice::Move` (`src/compaction/mod.rs:70`, taken at +`src/compaction/leveled/mod.rs:524-527` and `:574-577`) and RocksDB's trivial +move. Note where it sits: the taxonomy does not give it an axis of its own; it +falls out of choosing file granularity. For sequential or bulk-load ingest it +dominates everything — a sorted snapshot can cascade to the bottom level +entirely by relinking, write amp 1.0 — which is the M4 graph-snapshot question +answered by a mechanism the two-word vocabulary could not express. + +### Step 5 — axis 4, data movement: which file gets picked + +> **In:** partial compaction from Step 4, which has just decided to move *one* +> file and now has to say *which*. +> **Out:** seven picking policies, each optimizing a different metric — and the +> paper's negative result about what this axis does *not* affect. + +Data movement policy answers "which block of data to be moved" — in the +literature's more common name, the **file picking policy**. It only exists when +granularity is partial: §3.1.4 opens "When partial compaction is employed, the +data movement policy selects which file(s) to choose for compaction", and §3.2 +notes that a full-level design "by definition, does not need a data movement +policy". The axes are not independent in that one respect. + +``` + i) Round-robin: chooses files in a round-robin manner + ii) Least overlapping parent: file with least overlap with "parent" + iii) Least overlapping grandparent: as above with "grandparent" + iv) Coldest: the least recently accessed file + v) Oldest: the oldest file in a level + vi) Tombstone density: file with #tombstones above a threshold + vii) Tombstone-TTL: file with expired tombstone-TTLs + — §3.1.4 +``` + +Each entry is a different metric being optimized, and §3.1.4 names them: +round-robin and random "do not focus on optimizing for any particular +performance metric, but help in reducing space amplification"; **coldest** +optimizes read throughput; **least overlap** minimizes write amplification; +**tombstone density** reduces space amplification; **tombstone-TTL** bounds +delete latency. One axis, five different goals — and the measured payoff for the +write-amp choice is real but modest: `LO+1` and `LO+2` "move **10%-23% less +data** than other partial compaction strategies" (O2). + +Now the negative result, which is more interesting than the positive one: + +> **TA III:** The point lookup latency is largely unaffected by the data +> movement policy. In presence of Bloom filters (with high enough memory) and +> small enough block cache, the point query latency remains largely unaffected +> by the data movement policy as long as the number of sorted runs in the tree +> remains the same. (§5.1.2) + +File picking changes *which* bytes move, not *how many runs exist* — and reads +pay per run, so reads cannot tell. The axis that moves point lookups is layout +(Step 3); the axis that moves tail writes is granularity (Step 4); this axis +moves write amplification, space amplification and delete latency. Four axes, +four different cost columns — that is the whole reason to have the taxonomy. + +The paper's own filter configuration is worth noting since it lands exactly on +the Monkey chapter's arithmetic: 10 bits per key giving "FPR = 0.8%" (§5.1.2) — +the same 0.819% that `e^(−10·ln²2)` predicts and that fjall's filters deliver at +0.844%. ### Step 6 — using the grid: place every system, then trust only same-engine data -With four axes, every policy you've met becomes a coordinate — your -mini-LSM is (trigger = level size, layout = leveled or tiered, granularity -= whole level, movement = full merge, + trivial move if you stole -`Choice::Move`); RocksDB leveled is (saturation, leveling, one-file, -merge+trivial-move). The paper's second contribution is methodological: it -implements the *whole grid inside one engine* so comparisons vary one axis -at a time — the Fair Benchmarking lesson (topic 0) applied, because -cross-engine comparisons confound all four axes with everything else. And -the headline empirical result across the grid: **no policy wins everywhere** -— the RUM conjecture, empirically, again. +> **In:** all four axes. +> **Out:** a coordinate for every system in this topic, the methodology that +> makes cross-strategy numbers trustworthy, and the headline finding. + +With four axes, every policy you've met becomes a coordinate: + +| system | layout | trigger | granularity | movement | +|---|---|---|---|---| +| your mini-LSM | leveling or tiering | level size | level | n/a (whole level) | +| lsm-tree crate | leveling | level size, run count | several files | — | +| RocksDB default | **1-leveling** | level saturation, #runs, staleness, SA, TS-TTL | file (single/multiple) | round-robin, least-overlap ±1/±2, coldest, oldest, TS-density, TS-TTL | +| RocksDB universal | tiering | #sorted runs + space amp | sorted run | — | +| Dostoevsky | **L-leveling** | per-level (K, Z) | file and level | least-overlap | + +(The RocksDB and Dostoevsky rows are Table 1's, transcribed.) Note the two +entries the two-word vocabulary gets wrong on its own terms: RocksDB "leveled" +is 1-leveling, and RocksDB "universal" *is* the paper's `Tier`. + +The paper's second contribution is methodological, and it is the Fair +Benchmarking lesson (topic 0) applied at scale: they implement **ten** strategies +— `Full`, `LO+1`, `LO+2`, `RR`, `Cold`, `Old`, `TSD`, `TSA`, `Tier`, `1-Lvl` — +*inside one codebase* (modified RocksDB, "more than a hundred design knobs"), +then run **more than 2000 experiments** varying one axis at a time. Cross-engine +comparisons cannot do this: they confound all four axes with the storage format, +the filter implementation, the thread pool and everything else. §5's setup is a +single AWS `t2.2xlarge` (8 vCPU at 3.0 GHz, 32 GB RAM, 45 MB L3, Ubuntu 20.04) +with a 40 GB io2 SSD at 4000 provisioned IOPS; RocksDB at size ratio 10, 8 MB +write buffer, 10 bits/key filters, 8 MB block cache, direct IO, one compaction +thread, 128 B entries, 10 M inserts. + +And the headline, Key Takeaway A (§1): + +> **There is no perfect compaction strategy.** When it comes to selecting a +> compaction strategy for an LSM-engine, there is no single best. Thus, a +> compaction strategy needs to be custom-tailored to specific combinations of +> workload, LSM tuning, and performance goals. + +The RUM conjecture, empirically, again — this time with 12 observations +attached. §6's practical distillation: avoid `Tier` where worst-case latency +matters (its tail is the 25 ms in Step 4, and O10 shows it *worsening* with data +size beyond 8 GB), avoid `LO+2` where predictability matters, and prefer partial +leveling or `1-Lvl` for stable performance. + +Two things the taxonomy does *not* cover, worth noticing because a good taxonomy +makes its own gaps visible. **Filter memory allocation** is not an axis — Monkey +moves on none of the four, so a fifth primitive would be needed to express it. +And **trivial move / pseudo-compaction** has no axis either; it is an emergent +consequence of choosing file granularity (Step 4). Both are real design +decisions with measured effects, sitting outside a design space the paper +estimates at >10⁴ points. ## How to read the paper (with the concepts in hand) -1. §3 — the taxonomy (Steps 1–5 in the authors' terms). Make the table for: - your mini-LSM, lsm-tree crate, RocksDB leveled, RocksDB universal, FIFO. -2. §4 — the benchmark methodology (Step 6): they implement the design space - inside one engine to compare fairly — same engine, one variable. -3. **§5 findings** — the ones worth keeping: - - file-granularity compaction (RocksDB style) smooths write stalls vs - whole-level (spikes) — granularity is a *tail latency* knob, not a - throughput knob (Step 4); - - trigger choice dominates point-lookup latency more than layout at low - write rates (Step 2); - - no policy wins everywhere (the RUM conjecture, empirically, again). -4. Skim the workload sensitivity plots — note which finding you'll test. +Budget about 2 h. Section numbers are the paper's own. + +1. **§3.1** — the four primitives and their option lists (Steps 1-5). Read + Figure 3 first; it is the whole taxonomy on one page. +2. **§3.2 and Table 1** — where twenty-plus real systems land. Fill in the grid + in Step 6 for: your mini-LSM, the lsm-tree crate, RocksDB leveled, RocksDB + universal, FIFO. Table 2 defines the ten codified strategies you will meet + throughout §5. +3. **§4 Benchmarking Compactions** — the one-engine methodology (Step 6). Short, + and the reason to believe §5. +4. **§5 findings** — the keepers: **O1-O3 and TA I-II** for granularity and tail + latency (Step 4); **O4 and TA III** for what moves point lookups and what + does not (Steps 3 and 5); **O10** for how `Tier` degrades with data size. + Read the setup paragraph before quoting anything. +5. **§6 Discussion** — "Avoiding the Worst Choices" is the practitioner's page. +6. Skim the workload-sensitivity plots (§5.2) — note which finding you'll test. ## Questions to answer in notes.md 1. Your write_amp experiment compacts whole levels. Predict, then measure if time allows: what does per-insert p99.9 look like vs a per-file granularity - variant? (This is topic 2's rehash-spike lesson at LSM scale.) -2. Which axis does Dostoevsky's lazy leveling move on? (Layout only — trigger/ - granularity/movement orthogonal.) Which does Monkey move on? (None — it's - a filter-memory axis the taxonomy doesn't cover; where would you add it?) + variant? (This is topic 2's rehash-spike lesson at LSM scale; the paper's + own answer is TA II's 25 ms vs 1.3 ms.) +2. Which axis does Dostoevsky's lazy leveling move on? (Layout only — Table 1 + files it as L-leveling; trigger, granularity and movement stay orthogonal.) + Which does Monkey move on? (None — filter memory is not one of the four; + where would you add it, and what would its option list be?) 3. For M4's graph-snapshot SSTs: bulk-loading a snapshot is one giant sorted - run. Which axis choices make ingest cheap? (Trivial move into the bottom - level — no merge at all.) + run. Which axis choices make ingest cheap? (File granularity, so that + pseudo-compaction / trivial move applies — no merge at all.) ## Done when -Your notes contain the 5-system × 4-axis table and one prediction you could -test with the mini-LSM. +Answer each before unfolding it. + +- [ ] You can name the four primitives, the question each answers, and which one is single-valued. + +
Answer + + From §3.1, verbatim: (1) **compaction trigger** — when to re-organize the data + layout? (2) **data layout** — how to lay out the data physically on storage? + (3) **compaction granularity** — how much data to move at-a-time during layout + re-organization? (4) **data movement policy** — which block of data to be moved + during re-organization? + + **Data layout is single-valued**; trigger, granularity and data movement + policy are multi-valued, so an engine has one layout but can carry several + triggers and several picking rules at once (§3.2). "Leveled" and "tiered" are + answers to question 2 only, which is why the vocabulary hides three quarters + of the design space. The paper estimates the full space at **>10⁴** distinct + strategies. + +
+ +- [ ] You can give the five layouts, and say what 1-leveling and L-leveling actually mean. + +
Answer + + Leveling (one run per level); tiering (multiple runs per level); **1-leveling** + — *tiering for Level 1, leveling otherwise*; **L-leveling** — *leveling for the + last level, tiering otherwise*; hybrid — each level chooses independently + (§3.1.2). + + These two are near-opposites and easy to swap. 1-leveling is laziness at the + *top*, to absorb ingest bursts without stalling — and it is **RocksDB's + default** (§3.2: "1-Lvl … is the default data layout for RocksDB"; Table 1 + lists RocksDB as "Leveling / 1-Leveling"). L-leveling is laziness + *everywhere but the bottom* — which is Dostoevsky's Lazy Leveling, and Table 1 + files Dostoevsky under L-leveling. + +
+ +- [ ] You can say which axis moves which cost, with a number for each. + +
Answer + + **Layout → point lookups.** O4: tiering's mean point-lookup latency is + 1.1-1.9× leveling's on existing keys and ~2.2× on non-existing keys. (Theory + predicts T× = 10×; the measured 2.2× is explained by RocksDB's tiering keeping + fewer runs than textbook tiering, plus the block cache and lookup temporality.) + + **Granularity → tail write latency, and total data moved.** TA II: tail write + stall is ~25 ms for `Tier` against 1.3 ms for partial leveling (`Old`) — a 19× + spread. O1: `Full` moves 63× the ingested bytes, `Tier` 23×. O2: partial + compaction moves 34-56% less than `Full` while running 4× more jobs. TA I: + full-level compaction does ~1/L as many compactions, each moving ~2L times + more data. + + **Data movement policy → write amp, space amp, delete latency — but not + reads.** O2: `LO+1`/`LO+2` move 10-23% less data than other partial + strategies. TA III: "The point lookup latency is largely unaffected by the + data movement policy… as long as the number of sorted runs in the tree remains + the same." Reads pay per run; picking a different file does not change the run + count. + + **Trigger →** when everything above happens, and (via tombstone-TTL) how + quickly a delete becomes persistent. + +
+ +- [ ] You can explain why the paper implements ten strategies in one engine, and what that buys. + +
Answer + + Because a cross-engine comparison confounds all four axes with the storage + format, filter implementation, thread pool and everything else — the Fair + Benchmarking lesson from topic 0. So they integrate ten codified strategies + (`Full`, `LO+1`, `LO+2`, `RR`, `Cold`, `Old`, `TSD`, `TSA`, `Tier`, `1-Lvl`) + into one modified RocksDB codebase exposing "more than a hundred design knobs", + and run **more than 2000 experiments** varying one primitive at a time (§1, + "Experimental Contribution 1"; §4). + + What it buys is attribution. Without it, "tiering has worse tail latency" is a + claim about two products; with it, it is a claim about one primitive with + everything else held fixed — and it is why O4 can go on to say *why* the + measured 2.2× falls short of the theoretical 10×, instead of just reporting a + ratio. + + The setup, for quoting: AWS `t2.2xlarge`, 8 vCPU at 3.0 GHz, 32 GB RAM, 45 MB + L3, Ubuntu 20.04, 40 GB io2 SSD at 4000 provisioned IOPS; RocksDB at size + ratio 10, 8 MB write buffer, 10 bits/key filters (FPR 0.8%), 8 MB block cache, + direct IO, one compaction thread, 128 B entries, 10 M inserts. + +
+ +- [ ] You can state the headline finding and name two design decisions the taxonomy does not cover. + +
Answer + + Key Takeaway A (§1): "**There is no perfect compaction strategy.** … there is + no single best. Thus, a compaction strategy needs to be custom-tailored to + specific combinations of workload, LSM tuning, and performance goals." The RUM + conjecture, arrived at empirically, with 12 observations behind it. §6's + practical version: avoid `Tier` when worst-case latency matters (25 ms tails, + and O10 shows it degrading further past 8 GB), avoid `LO+2` when + predictability matters, prefer partial leveling or `1-Lvl` for stability. + + Not covered: **filter memory allocation** — Monkey moves on none of the four + primitives, so expressing it needs a fifth. And **trivial move / + pseudo-compaction** — real, measured (part of O2's 34-56%), named in the text, + but not an axis; it is an emergent consequence of choosing file granularity. + A taxonomy that makes its own gaps visible is doing its job. + +
## References **Papers** -- Sarkar, Papon, Staratzis, Athanassoulis — "Constructing and Analyzing - the LSM Compaction Design Space" (VLDB 2021) — §3 taxonomy and §5 - findings are the keepers; §4's one-engine methodology is the Fair - Benchmarking lesson applied +- Sarkar, Staratzis, Zhu, Athanassoulis — *Constructing and Analyzing the LSM + Compaction Design Space*, PVLDB 14(11): 2216-2229, 2021. + Artifacts at `https://disc.bu.edu/lsm-compaction`. §3 is the taxonomy, §4 the + one-engine methodology, §5 the twelve observations and seven takeaways, §6 the + practitioner's summary. + +| Claim in this chapter | Source | +|---|---| +| The four primitives and their questions | §3.1 | +| Layout single-valued, other three multi-valued; space >10⁴ | §3.2 | +| Five triggers | §3.1.1 | +| Five layouts, incl. 1-leveling and L-leveling definitions | §3.1.2 | +| `1-Lvl` is RocksDB's default layout; `Tier` is universal compaction | §3.2; Table 1 | +| Four granularity options; partial compaction defined | §3.1.3, §2 | +| Seven data movement policies, and the metric each targets | §3.1.4 | +| `Full` moves 63× ingested bytes, `Tier` 23× | O1, §5.1.1 | +| Partial moves 34-56% less, runs 4× more jobs; `LO±` 10-23% less | O2, §5.1.1 | +| Full does 1/L as many compactions, each ~2L× larger | TA I | +| `Full` mean latency 1.2-1.9× partial, 2.1× tiering; CPU ~50% | O3 | +| Tail write stall 25 ms (`Tier`) vs 1.3 ms (`Old`) | TA II | +| Tiering point lookups 1.1-1.9× / ~2.2× leveling, vs T× predicted | O4, §5.1.2 | +| Point lookup latency unaffected by movement policy | TA III | +| `Tier` degrades past 8 GB | O10, §5.2 | +| Ten strategies, one codebase, >100 knobs, >2000 experiments | §1, §4 | +| EC2 setup, RocksDB config, 10 bits/key → FPR 0.8% | §5, "Experimental Setup"; §5.1.2 | +| "There is no perfect compaction strategy" | Key Takeaway A, §1 | +| Avoid `Tier` / `LO+2`; prefer partial leveling or `1-Lvl` | §6, "Avoiding the Worst Choices" | + +**Code** +- `lsm-tree src/compaction/mod.rs:70` at `8526dd3` — `Choice::Move`, the + pseudo-compaction of O2; taken at `src/compaction/leveled/mod.rs:524-527` + and `:574-577`. +- `rocksdb db/version_set.cc:4136-4137` at `7c80a5a` — the level-saturation + trigger of §3.1.1, as a score. + +**Repo cross-references** +- `topics/04-lsm-deep-dive/reading-rocksdb-compaction.md` — the scoring and + stall machinery this chapter classifies. +- `topics/04-lsm-deep-dive/reading-dostoevsky.md` — L-leveling, from the inside. +- `topics/00-performance-toolbox/reading-fair-benchmarking.md` — why §4's + one-engine methodology is the only way these numbers mean anything. diff --git a/topics/04-lsm-deep-dive/reading-dostoevsky.md b/topics/04-lsm-deep-dive/reading-dostoevsky.md index 3be271e..3b5f408 100644 --- a/topics/04-lsm-deep-dive/reading-dostoevsky.md +++ b/topics/04-lsm-deep-dive/reading-dostoevsky.md @@ -7,6 +7,11 @@ tiered actually promise, which level dominates each cost, why merging eagerly at small levels buys nothing — until "tier the top, level the bottom" is the obvious move — and then the Fluid-LSM dial that generalizes it. +Every formula and number below is checked against the paper — Dayan & Idreos, +*Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores +via Adaptive Removal of Superfluous Merging*, SIGMOD 2018 — and cited to the +section, equation or figure it came from. + ## The problem in one sentence Leveled compaction rewrites every key ~T times per level (T = size ratio, @@ -18,144 +23,646 @@ levels contribute almost nothing to read or space cost, so roughly ### Step 1 — the two classic policies, restated as runs per level -A **run** is a sorted, key-disjoint set of segments — one unit a point read -must probe once (lsm-tree chapter, Step 5). The two classic compaction -policies differ only in *how many runs each level tolerates* before merging: +> **In:** the LSM shape from the lsm-tree chapter — a memtable, flushes, levels +> that grow by a factor of T. +> **Out:** the two named policies expressed as a single integer per level (how +> many runs it tolerates), which is the form Step 6 turns into a dial. + +Vocabulary first, since the paper's whole argument is in these symbols +(its Table 1 is the glossary): + +| symbol | meaning | +|---|---| +| `N` | number of entries in the tree | +| `T` | **size ratio** — each level holds T× the entries of the one above | +| `L` | number of levels on disk | +| `B` | entries per disk block (an IO moves one block) | +| `M` | total main memory given to bloom filters, in bits | +| `p_i` | false-positive rate of the filters at level *i* | +| `s` | size of a range-scan's target range, as a fraction of the key space | + +A **run** is a sorted, key-disjoint set of tables — one unit a point read must +probe once (lsm-tree chapter, Step 5). The two classic compaction policies +differ only in *how many runs each level tolerates* before merging: -- **Leveled**: every level holds exactly **1 run**. Each time data arrives - from above, it is merged into the level's run immediately — and since the - level is up to T× bigger than the arriving data, each incoming byte drags - ~T resident bytes through the merge. Write amplification (bytes physically - written per byte of user data) ≈ O(T·L) over L levels; reads probe 1 run +- **Leveled**: every level holds exactly **1 run**. Each time data arrives from + above it is merged into the level's run immediately — and since the level is + up to T× bigger than the arriving data, each incoming byte drags resident + bytes through the merge with it. Update cost `O(L·T / B)`; reads probe 1 run per level. -- **Tiered**: each level accumulates up to **T runs** of similar size, then - merges them all into one run that moves down a level. Each byte is - rewritten only ~once per level — write amp ≈ O(L) — but reads must probe - up to T runs per level, and the largest level may hold T stale copies of - the same key. +- **Tiered**: each level accumulates up to **T−1 runs** of similar size, and the + T-th arrival triggers a merge of all of them into one run that moves down. + Each byte is rewritten ~once per level — update cost `O(L / B)` — but reads + probe up to T−1 runs per level, and the largest level may hold T−1 stale + copies of the same key, so space amplification is `O(T)`. -Same data, same levels; the whole difference is eagerness of merging. +(Both complexities are Figure 6, rows (A) and (G).) Same data, same levels; the +whole difference is eagerness of merging. + +Worth knowing before you go looking for tiered in this repo's reference crate: +`lsm-tree` at `8526dd3` ships **leveled only** — its tiered strategy is +commented out of the module tree: + +```rust +// fjall-rs/lsm-tree@8526dd3 — src/compaction/mod.rs + 7 pub(crate) mod fifo; + 8 pub(crate) mod leveled; + // ... 9-17: other modules ... + 18 // pub(crate) mod tiered; + 19 pub(crate) mod worker; + 20 + 21 pub use fifo::Strategy as Fifo; + 22 pub use filter::{CompactionFilter, Factory, ItemAccessor, Verdict}; + 23 pub use leveled::Strategy as Leveled; + 24 // pub use tiered::Strategy as SizeTiered; +``` + +So "leveled vs tiered" is a live design argument, not a menu you pick from — +which is exactly why the parameterization in Step 6 is the useful takeaway. ### Step 2 — where each cost actually lives -The three costs an LSM is judged on do not come from all levels equally: +> **In:** the two policies from Step 1 and the four costs an LSM is judged on. +> **Out:** the attribution — which *level* dominates each cost — which is the +> whole evidence base for the diagnosis in Step 3. + +The costs do not come from all levels equally, and this asymmetry is the entire +paper: -- **Space amplification** (bytes on disk per byte of live data) is dominated - by the **largest level** — it holds ~90% of the data at T=10, and dead - versions of keys survive there until a merge drops them. Upper levels are - ~10% of the data total; even fully duplicated they barely matter. +- **Space amplification** (bytes on disk per byte of live data) is dominated by + the **largest level**. The paper's argument (§4.1, "Space-Amplification"): in + the worst case every entry at levels 1…L−1 is an update to an existing entry + at level L, and that fraction is `1/T` of the data, so space amplification is + at most `O(1/T)`. At T = 10 the upper levels are 10% of the data *in total* — + even fully duplicated they barely matter. - **Zero-result point lookups** (the filter-tax workload from Monkey) are - dominated by the largest level too: its filter has the most keys per bit - and thus the highest false-positive contribution. -- **Write amplification** is dominated by the **upper levels**: every byte - passes through L1, L2, … on its way down, getting rewritten at each stop. + dominated by the largest level too. Under the optimal allocation the bottom + level's FPR is `p_L = R·(T−1)/T` (Equation 5) — the bottom level *is* 90% of + the lookup cost at T = 10, by construction, because that is where the entries + are and bits are scarcest per entry. +- **Long range lookups** are dominated by the largest level: it "contains + exponentially more entries than all other levels", so the cost is `O(s/B)` + regardless of what the upper levels look like (§4.1, "Range Lookups"). +- **Update cost** is dominated by the **upper levels**: every byte passes + through L1, L2, … on its way down, getting rewritten at each stop. With + leveling that is L rewrites each dragging a level's worth of resident data. +- **Short range lookups** are the one exception — they touch every run at every + level and so they *do* care about upper-level fragmentation. Keep this one in + your pocket; it is the bill Step 4 pays. -| Level | What merging there improves | Who cares | +| level | what merging there improves | who cares | |---|---|---| -| upper (small) levels | almost nothing — they're small, probes are filtered | nobody | -| **largest level** | space amp (dead versions live here) + zero-result reads | everybody | +| upper (small) levels | short range lookups, and almost nothing else | short scans only | +| **largest level** | space amp, zero-result lookups, long range lookups | everybody | ### Step 3 — the diagnosis: superfluous merging -Superfluous merging is merge work whose cost you pay but whose benefit no -metric reflects — and Step 2 says that's most of what leveled compaction -does. Keeping L1 (0.9% of the data) as one pristine run costs a full T× -rewrite of everything passing through, and buys: a filtered probe avoided -occasionally, on a level whose filter was nearly perfect anyway (Monkey gave -small levels the most bits/key), and space savings on 0.9% of the data. -Meanwhile tiered compaction is lazy *everywhere*, including the one level -where eagerness pays — its largest level fragments into T runs, wrecking -space amp (up to T stale copies of the hottest 90% of data) and zero-result -lookups (T bottom-level filters to get past instead of 1). +> **In:** Step 2's attribution table. +> **Out:** a named defect in *both* classic policies — each is eager or lazy in +> the wrong place — which Step 4 fixes by splitting the difference. + +**Superfluous merging** is merge work whose cost you pay but whose benefit no +metric reflects — and Step 2 says that is most of what leveled compaction does. +The abstract puts it exactly: "merge operations from all levels of LSM-tree but +the largest (i.e., most merge operations) reduce point lookup cost, long range +lookup cost, and storage space by a negligible amount while significantly adding +to the amortized cost of updates." + +Concretely at T = 10, L = 4: keeping L1 (0.09% of the data) as one pristine run +costs a full rewrite of everything passing through it, and buys a filtered probe +avoided occasionally on a level whose filter Monkey already made nearly perfect +(23.9 bits/key, FPR 0.001%), plus space savings on 0.09% of the data. + +Meanwhile tiered compaction is lazy *everywhere*, including the one level where +eagerness pays. Its largest level fragments into T−1 runs, which wrecks space +amplification (`O(T)` instead of `O(1/T)` — a factor of `T²`) and multiplies +zero-result lookup cost by T (Figure 6(D): `O(T·e^(−M/N))` versus +`O(e^(−M/N))`). + +Both classic policies are therefore wrong in the same way: they apply one +eagerness setting to levels whose cost structures differ by a factor of `T^L`. ### Step 4 — the fix: lazy leveling -Lazy leveling applies each policy where it wins: **tiered at the upper -levels** (writes pass through cheaply — nobody needed those levels merged) -and **leveled at the largest level only** (the one place where 1 run buys -space amp and read cost for everybody): +> **In:** the diagnosis from Step 3. +> **Out:** a policy that is tiered above and leveled at the bottom, and the four +> complexities it lands on — one of which (short range lookups) is worse, and +> Step 5 has to buy the point-lookup one back with filter bits. + +**Lazy Leveling** applies each policy where it wins: **tiered at levels +1…L−1** (writes pass through cheaply — nobody needed those levels merged) and +**leveled at level L only** (the one place where 1 run buys space amp and read +cost for everybody). §4.1: "Lazy leveling at its core is a hybrid of leveling +and tiering: it applies leveling at the largest level and tiering at all other +levels." ``` - tiered: leveled: lazy leveled (Dostoevsky): + tiered: leveled: lazy leveled (Dostoevsky): - L1: ▧▧▧▧ K runs L1: ▧ 1 run L1: ▧▧▧▧ K runs ← tiered on top - L2: ▧▧▧▧ L2: ▧ L2: ▧▧▧▧ (writes cheap) - L3: ▧▧▧▧ L3: ▧ L3: ▧ 1 run ← leveled at bottom - (space + reads OK) - WA: O(L) WA: O(T·L) WA: O(L + T) ← T paid once, at bottom + L1: ▧▧▧▧ T−1 runs L1: ▧ 1 run L1: ▧▧▧▧ T−1 runs ← tiered on top + L2: ▧▧▧▧ L2: ▧ L2: ▧▧▧▧ (writes cheap) + L3: ▧▧▧▧ L3: ▧ L3: ▧ 1 run ← leveled at bottom + (space + reads OK) + update: O(L/B) O(L·T/B) O((L+T)/B) ← T paid once, at bottom ``` -Read the write-amp column: the expensive T-fold merge is paid exactly -**once**, at the bottom, instead of at every level. Point-lookup and space -complexity match leveled (the largest level is 1 run — the only level where -it mattered); write cost is close to tiered. At T=10, L=3: leveled WA ≈ 30×, -lazy leveled ≈ 13×, for essentially the same read and space behavior. +The update-cost derivation is one sentence in §4.1: "An updated entry with Lazy +Leveling participates in `O(1)` merge operations per level across Levels 1 to +L−1 and in `O(T)` merge operations at Level L. The overall number of merge +operations per entry is therefore `O(L + T)`." + +The full comparison, transcribed from Figure 6 (all six rows): + +| cost | tiering | leveling | lazy leveling | +|---|---|---|---| +| update | `O(L/B)` | `O(L·T/B)` | `O((L+T)/B)` | +| zero-result point lookup | `O(T·e^(−M/N))` | `O(e^(−M/N))` | `O(e^(−M/N))` | +| point lookup, existing key | `O(1 + T·e^(−M/N))` | `O(1)` | `O(1)` | +| short range lookup | `O(L·T)` | `O(L)` | `O(1 + (L−1)·T)` | +| long range lookup | `O(s·T/B)` | `O(s/B)` | `O(s/B)` | +| space amplification | `O(T)` | `O(1/T)` | `O(1/T)` | + +Read the column: lazy leveling matches **leveling** on four of the six rows, +beats it decisively on updates, and loses only on short range lookups — the one +cost Step 2 flagged as caring about upper-level fragmentation. That is the +paper's claim in full, and it is a genuinely surprising one: three of the four +things people buy leveling *for* did not need leveling above the bottom level. + +Put the update row on this repo's own arithmetic. `topics/04-lsm-deep-dive/notes.md` +uses `T/2 × L` for leveled write amplification (a level is on average half full +when data merges into it, so the resident data dragged through averages T/2×, +not T×) and `L` for tiered: + +``` + T = 10, L = 4 + + leveled T/2 × L = 5 × 4 = 20× (notes.md) + tiered L = 4 = 4× (notes.md) + lazy leveling (L−1) + T/2 = 3 + 5 = 8× (same convention: + tiered above, leveled + once at the bottom) +``` + +**8× instead of 20× — a 2.5× cut in write amplification — while the space, +long-scan and point-read columns above stay exactly where leveling had them.** +That number is arithmetic on this repo's stated model, not a measurement: topic +4 has no `verify.sh` lane, because its benches measure only your code. For a +*measured* sense of what this family of costs looks like in practice, the +nearest lane in this repo is `FINDINGS.md` row 1 (`./verify.sh 01`, Apple M3 +Pro, 2026-07-28): the same 108 MB of records lands as **48 MB** on disk under +fjall's LSM against **6.8 GB** under redb's copy-on-write B-tree — space +amplification 0.45× versus 63.28×, a 140× spread. + +### Step 5 — the filter allocation that keeps the read column honest + +> **In:** lazy leveling's shape from Step 4, which now has T−1 runs to probe at +> every upper level instead of 1. +> **Out:** the FPR assignment that keeps zero-result lookup cost at +> `O(e^(−M/N))` anyway, worked on concrete numbers — and the memory floor below +> which the trick stops working. + +Step 4's table claims lazy leveling matches leveling on point lookups. That +cannot be free: the tree now has `(T−1)·(L−1) + 1` runs to probe instead of `L`. +The cost is paid in **filter allocation**, and §4.1's "Bloom Filters Allocation" +is where the paper earns the claim. + +Start with the objective, Monkey's rule adapted to more runs per level (§4.1, +Equation 3): + +``` + R = p_L + (T−1) · Σ(i=1..L−1) p_i Dostoevsky §4.1, Eq 3 + + R expected wasted IOs per zero-result point lookup + p_i the FPR shared by every run at level i +``` + +The memory model is Monkey's, unchanged — and note the sentence that makes it +work with multiple runs per level: "Since the filters at any given level all +have the same FPR, we can directly apply this equation regardless of the numbers +of runs at a level." A level's filters cost `M_i = −N_i·ln(p_i)/ln(2)²` bits +whether that's one run or nine, because it is the same total number of entries +either way. + +``` + M = −(N / ln²2) · ((T−1)/T) · Σ(i=1..L) ln(p_i) / T^(L−i) §4.1, Eq 4 +``` + +Minimizing Equation 3 subject to Equation 4 (Lagrange multipliers; derivation in +**Appendix A**) gives: + +``` + p_i = R · (T−1)/T for i = L §4.1, Eq 5 + R / T^(L−i+1) for 1 ≤ i < L +``` + +and substituting back gives the closed form for R itself: + +``` + R = e^(−(M/N)·ln²2) · T^(T/(T−1)) / (T−1)^((T−1)/T) §4.1, Eq 6 +``` + +Worked at T = 10, L = 4, N = 10 M, M/N = 10 bits per entry — the same tree and +the same budget as the Monkey chapter's Step 5, so the two are directly +comparable: + +| level | runs | leveling: bits/key, FPR | lazy leveling: bits/key, FPR | +|---|---|---|---| +| 1 | 1 vs 9 | 23.85, 0.0011% | 27.96, 0.00011% | +| 2 | 1 vs 9 | 19.06, 0.0106% | 23.17, 0.0015% | +| 3 | 1 vs 9 | 14.26, 0.1057% | 18.38, 0.0146% | +| 4 | 1 vs 1 | 9.47, 1.0566% | 9.01, 1.3160% | +| | | `R` = **0.01174** | `R` = **0.01465** | + +Equation 6 evaluates to 0.014646 at these parameters, agreeing with the direct +solve to 0.2%. So lazy leveling's zero-result lookups cost **25% more wasted +IOs than leveling at the same memory** — a constant factor, exactly as the paper +says: "the multiplicative term at the right-hand side of Equation 6 is a small +constant for any value of T. Therefore, the cost complexity is `O(e^(−M/N))`, +the same as with leveling despite having eliminated most merge operations." + +Where does the extra memory come from? Read the bits column: lazy leveling +shifts ~0.5 bits/key *off* the bottom level (9.01 vs 9.47) and onto the upper +levels (+4.1 bits/key each), because each upper level now needs a lower FPR to +survive being multiplied by T−1 in Equation 3. Trading 25% more false probes for +2.5× less write amplification is the deal on the table. + +**The trick has a memory floor.** As `M/N` shrinks the optimal FPRs rise toward +1 and filters start disappearing (bottom level first, since its FPR is highest). +Equation 7 gives the threshold: + +``` + M/N threshold = (1/ln²2) · ( ln(T)/(T−1) + ln(T−1)/T ) §4.1, Eq 7 + + T = 10 → 0.99 bits per entry + T = 3 → 1.62 bits per entry ← the global maximum over all T +``` + +The paper's own comment: mainstream stores default to 10 or 16 bits per entry, +"an order of magnitude larger", so the analysis holds everywhere it matters; for +sensors and mobile devices below 1.62 bits/entry, Appendix C adapts it by +merging more at larger levels. It is a rare case of an optimization whose +precondition is *satisfied by an order of magnitude* rather than marginally, and +worth noticing as a modelling habit. + +### Step 6 — Fluid LSM: two knobs make it a dial, not a trick + +> **In:** three named policies (tiered, leveled, lazy leveled) and their costs. +> **Out:** two integers that generate all three and everything between, so a +> tuner can *solve* for a policy instead of picking one from a list. + +**Fluid LSM-tree** generalizes the whole family with two integers (§4.2): +**K** = runs tolerated at each of levels 1…L−1, **Z** = runs tolerated at the +largest level. The paper's parameterization, quoted: + +- `K = 1` and `Z = 1` give leveling. +- `K = T−1` and `Z = T−1` give tiering. +- `K = T−1` and `Z = 1` give Lazy Leveling. + +The mechanism is one detail worth keeping: each level has an *active* run that +incoming runs merge into, with a size threshold of `T/K` percent of the level's +capacity for levels 1…L−1 and `T/Z` percent at level L; when the active run hits +its threshold a new active run starts, and when the level is at capacity all its +runs merge and flush down (§4.2, "Basic Structure"). K and Z are not just +counters — they set how big each run is allowed to get. -### Step 5 — Fluid LSM: two knobs make it a dial, not a trick +Every cost from Step 4 becomes a formula in K and Z (Figure 8): -Fluid LSM generalizes the whole family with two integers: **K** = runs -tolerated at each upper level, **Z** = runs tolerated at the largest level. -Leveled is (K=1, Z=1); tiered is (K=T−1, Z=T−1); lazy leveling is (K=T−1, -Z=1) — and everything between is legal, so the paper can *solve* for K and Z -per workload (write-heavy ⇒ raise Z toward tiered; space-constrained or -lookup-heavy ⇒ Z=1) rather than pick a named strategy. The whole family is -one compaction chooser with two thresholds: +| cost | Fluid LSM | +|---|---| +| update | `O((T/B) · (L/K + 1/Z))` | +| zero-result point lookup | `O(Z · e^(−M/N))` | +| point lookup, existing key | `O(1 + Z · e^(−M/N))` | +| short range lookup | `O(Z + (L−1)·K)` | +| long range lookup | `O(s·Z / B)` | +| space amplification | `O((Z−1) + 1/T)` | + +Check the row against Step 4's table by substituting: `K=Z=1` turns the update +row into `O(T·(L+1)/B) = O(T·L/B)` (leveling); `K=Z=T−1` turns it into +`O((T/(T−1))·(L+1)/B) ≈ O(L/B)` (tiering); `K=T−1, Z=1` gives +`O((T/B)·(L/(T−1) + 1)) ≈ O((L+T)/B)` (lazy leveling). Notice that **`Z` alone +drives every read and space cost** — `K` appears only in the short-range row. +That is Step 2's attribution table restated as algebra, and it is the cleanest +one-line summary of the paper. + +The dial slots into a real interface. `lsm-tree`'s compaction strategy is a +single method returning a `Choice`: + +```rust +// fjall-rs/lsm-tree@8526dd3 — src/compaction/mod.rs + 63 /// Describes what to do (compact or not) + 64 #[derive(Debug, Eq, PartialEq)] + 65 pub enum Choice { + 66 /// Just do nothing. + 67 DoNothing, + 68 + 69 /// Moves tables into another level without rewriting. + 70 Move(Input), + 71 + 72 /// Compacts some tables into a new level. + 73 Merge(Input), + // ... 74-79: Drop(HashSet) for FIFO-style expiry ... + 80 } + // ... 81-95: trait CompactionStrategy, get_name, get_config ... + 96 /// Decides on what to do based on the current state of the LSM-tree's levels + 97 fn choose(&self, version: &Version, config: &Config, state: &CompactionState) -> Choice; +``` + +A Fluid strategy is that method with two thresholds instead of one: ```rust -// K = max runs at upper levels, Z = max runs at the largest level. -// K=Z=1 ⇒ leveled; K=Z=T−1 ⇒ tiered; K=T−1, Z=1 ⇒ lazy leveling. -fn choose(&self, v: &Version) -> Choice { - for lvl in 0..v.last_level() { - if v.runs(lvl) > self.k { // upper levels: tolerate K runs - return Choice::MergeRunsInto(lvl + 1); +// ILLUSTRATION — not quoted from a repo. This is Fluid LSM-tree (Dostoevsky +// §4.2) written against the real trait above, src/compaction/mod.rs:97. +// K = max runs at levels 1..L-1, Z = max runs at the largest level. +// K=Z=1 ⇒ leveling; K=Z=T−1 ⇒ tiering; K=T−1, Z=1 ⇒ lazy leveling. +fn choose(&self, version: &Version, _c: &Config, _s: &CompactionState) -> Choice { + let last = version.last_level(); + for lvl in 0..last { + if version.run_count(lvl) > self.k { // upper levels: tolerate K runs + return Choice::Merge(self.input_for(lvl, lvl + 1)); } } - if v.runs(v.last_level()) > self.z { // largest level: tolerate Z - return Choice::MergeLastLevel; // T paid once, here - } + if version.run_count(last) > self.z { // largest level: tolerate Z + return Choice::Merge(self.input_for(last, last)); // the T-fold cost, + } // paid once, here Choice::DoNothing } ``` -That chooser slots straight into the lsm-tree crate's compaction trait and -your mini-LSM's pluggable strategy — leveled and tiered stop being rivals -and become endpoints of one dial. The costs to keep honest: range scans -still pay for every run at every level (K > 1 hurts them regardless of Z), -and upper-level runs still need filter memory — which is why Monkey and -Dostoevsky compose. +Two honest costs to carry away. Short range scans pay `O(Z + (L−1)·K)` — `K > 1` +hurts them regardless of `Z`, and no filter helps a scan. And upper-level runs +still need filter memory, at a *higher* bits/key than leveling would need +(Step 5's table) — which is why Monkey and Dostoevsky compose rather than +compete: you need Monkey's allocation to make Dostoevsky's shape affordable. + +### Step 7 — Dostoevsky itself, and what the evaluation actually reports + +> **In:** the Fluid design space parameterized by `T`, `K`, `Z`. +> **Out:** how the system picks a point in it, and an honest reading of what the +> paper measured — which is less quantitative than you might expect. + +**Dostoevsky** is the system that searches that space at runtime (§4.3). It +weights the four costs by their observed frequency in the workload — +`w` updates, `r` zero-result lookups, `v` non-zero lookups, `q` range lookups — +and maximizes worst-case throughput: + +``` + τ = Ω⁻¹ · ( w·W + r·R + v·V + q·Q )⁻¹ §4.3, Eq 14 + + Ω time to read one block from storage + W,R,V,Q the update / zero-result / existing-key / range cost formulas +``` + +Two pruning insights make the search cheap: there are only `⌈log₂(N/(P·B))⌉` +meaningful values of `T`, and the objective is **convex** in `K` and `Z` +(lookup costs increase monotonically in both, update cost decreases), so a +divide-and-conquer on each converges logarithmically. Total +`O(log₂(N/(B·P))³)` iterations, which "takes a fraction of a second". Re-tuning +runs every 16 buffer flushes in their implementation. + +Now the evaluation, and read this part carefully because it is *not* what the +other three papers in this topic do (§5): + +- **Implementation**: Dostoevsky built on **RocksDB**, with Equation 9's filter + allocation embedded in the code and Fluid LSM-tree implemented via RocksDB's + event-listener API to schedule custom merges. +- **Setup**: a RAID of 500 GB 7200 RPM disks, 32 GB DDR4, 4 × 2.7 GHz cores with + 8 MB L3, Ubuntu 16.04, ext4 with journaling off; direct IO; 2 MB buffer; + **10 bits per entry** of filter memory; fence pointers one per 32 KB block; + block cache 10% of the dataset. +- **Results**: Figure 10 plots **normalized** throughput — the y-axis is scaled + to Dostoevsky's own result — as the proportion of zero-result lookups sweeps + from 0.5% to 95%. The claim the paper makes is *dominance*, not a factor: + "Dostoevsky dominates all fixed policies by encompassing all of them and + fluidly transitioning among them." The abstract likewise says "strictly + dominates" with no percentage attached. +- The most concrete artifact in Figure 10(A) is the row of chosen tunings + printed above the plot: `T,Z,K` runs from tiering-like settings at the + update-heavy end to `Z=1, K=1` (pure leveling) at the lookup-heavy end, and + "these tunings are all unique to the Lazy Leveling and Fluid LSM-tree design + spaces, except at the edges." + +So: do not quote a speedup number for Dostoevsky, because the paper does not +publish one. Its quantitative content is the complexity table (Figure 6), the +closed-form models (Equations 3-14), and the tunings in Figure 10(A). This repo +adds no measurement of its own here either — topic 4 has no `verify.sh` lane. ## How to read the paper (with the concepts in hand) -1. §2 — the cost table (Table 1): Steps 1–2 as formulas. Reproduce it for - yourself for T=10, L=3: write cost, point read (zero/non-zero result), - range, space. This table IS the paper. -2. §3 — Lazy Leveling analysis (Step 4). Check the claim: same point-read + - space complexity as leveled, write cost close to tiered. -3. §4 — Fluid LSM + the tuning section (Step 5; skim the solver, keep the - knobs). -4. Evaluation — find the throughput-vs-skew plots. +Budget about 2.5 h. Section numbers are the paper's own — note that its +**Table 1 is a glossary of terms, not a cost table**; the cost table is +**Figure 6**. + +1. **§2 Background** — the LSM mechanics and Table 1's symbols. Skim if the + lsm-tree chapter is fresh; do read Table 1 as a glossary. +2. **§3 Design Space and Problem Analysis** — Figures 3-5, which quantify Step + 2's attribution: how much of each cost comes from which level. This is the + evidence for "superfluous". +3. **§4.1 Lazy Leveling** — the core. **Figure 6 IS the paper** (Steps 4-5); + reproduce its six rows for T = 10, L = 4 before moving on. Equations 3-8 are + Step 5; the Lagrange derivation is Appendix A and the closed form for R is + Appendix B. +4. **§4.2 Fluid LSM-tree** — K and Z, and Figure 8's cost table (Step 6). + Substitute the three named policies into every row until the table stops + needing to be looked up. +5. **§4.3 Dostoevsky** — the auto-tuner (Step 7). Skim the solver; keep the + convexity argument, since that is what makes it usable online. +6. **§5 Evaluation** — Figure 10(A) for the tunings, 10(B) for "no single merge + policy rules", 10(C) for scalability. Read the setup paragraph first: 2017 + spinning disks, direct IO, 10 bits/entry. ## Questions to answer in notes.md 1. Your mini-LSM implements leveled and tiered. Using its measured write amp and read amp: on YOUR numbers, what would lazy leveling have scored? - (Compute — upper levels tiered cost + bottom leveled cost.) + (Compute — upper levels tiered cost + bottom leveled cost; Step 4 does this + for `notes.md`'s model, so the method is there.) 2. Why do range scans not benefit from lazy leveling the way point reads do? - (Every run at every level must be merged into the scan regardless.) + (Every run at every level must be merged into the scan regardless — and + Figure 8's short-range row `O(Z + (L−1)·K)` is the only one containing `K`.) 3. RocksDB never shipped lazy leveling as such — universal compaction covers - part of the space. From reading-rocksdb-compaction.md, which universal - knobs approximate K and Z? + part of the space. From reading-rocksdb-compaction.md, which universal knobs + approximate K and Z? ## Done when -You can reproduce Table 1 from memory for the three strategies (writes, point -reads, space) and say in one sentence why "merge lazily except the last level" -dominates. +Answer each before unfolding it. + +- [ ] You can state which level dominates each of the four costs, and why. + +
Answer + + **Space amplification — largest level.** Worst case, every entry at levels + 1…L−1 is an update to something at level L; that fraction is `1/T` of the + data, so space amp is `O(1/T)` under leveling (§4.1, "Space-Amplification"). + + **Zero-result point lookups — largest level.** Under the optimal allocation + the bottom level's FPR is `p_L = R·(T−1)/T` (Equation 5): 90% of the wasted-IO + budget at T = 10, because that is where 90% of the entries are and so where + bits per entry are scarcest. + + **Long range lookups — largest level.** It "contains exponentially more entries + than all other levels", so the cost is `O(s/B)` whatever the upper levels do. + + **Update cost — upper levels.** Every byte is rewritten once per level on its + way down, and under leveling each rewrite drags a level's worth of resident + data with it: `O(L·T/B)`. + + **The exception: short range lookups** care about every run at every level, so + they are the one cost that upper-level fragmentation genuinely hurts. That is + the bill lazy leveling pays. + +
+ +- [ ] You can reproduce Figure 6 for the three policies — update, point lookup, space — and say which row lazy leveling loses. + +
Answer + + | cost | tiering | leveling | lazy leveling | + |---|---|---|---| + | update | `O(L/B)` | `O(L·T/B)` | `O((L+T)/B)` | + | zero-result point lookup | `O(T·e^(−M/N))` | `O(e^(−M/N))` | `O(e^(−M/N))` | + | point lookup, existing key | `O(1 + T·e^(−M/N))` | `O(1)` | `O(1)` | + | short range lookup | `O(L·T)` | `O(L)` | `O(1 + (L−1)·T)` | + | long range lookup | `O(s·T/B)` | `O(s/B)` | `O(s/B)` | + | space amplification | `O(T)` | `O(1/T)` | `O(1/T)` | + + Lazy leveling matches leveling on four rows, strictly beats it on updates, and + loses only on **short range lookups** (`O(1 + (L−1)·T)` against `O(L)`) — + precisely the cost that Step 2 identified as the one caring about upper-level + fragmentation. Everything else people buy leveling for turns out not to need + leveling above the bottom level. + + The one-sentence version of why it dominates: the expensive T-fold merge is + the only thing that improves space amp, long scans and point lookups, and it + only does so at the largest level — so pay it there once and nowhere else. + +
+ +- [ ] You can put the update row on this repo's arithmetic and say what lazy leveling would score. + +
Answer + + `notes.md` models leveled write amplification as `T/2 × L` (a level averages + half full when data merges into it) and tiered as `L`. At T = 10, L = 4: + leveled 20×, tiered 4×. Lazy leveling is tiered above and leveled once at the + bottom, so `(L−1) + T/2 = 3 + 5 = 8×` — a **2.5× cut** against leveling, while + the space, long-scan and point-read rows stay where leveling had them. + + Two honesty notes. That is arithmetic on a stated model, not a measurement: + topic 4 has no `verify.sh` lane, because its benches measure only your code. + And the constant differs from the paper's `O(L+T)`, which does not carry the + `/2`; the asymptotics agree, the constant is this repo's convention. + +
+ +- [ ] You can explain how lazy leveling keeps leveling's point-lookup complexity despite having T−1 runs per upper level, and what it costs. + +
Answer + + Through filter allocation, not through structure. The objective becomes + `R = p_L + (T−1)·Σ(i=1..L−1) p_i` (Equation 3) — upper levels are multiplied + by their run count — but the *memory* model is unchanged, because "the filters + at any given level all have the same FPR… regardless of the numbers of runs at + a level". Optimizing Equation 3 against Equation 4 gives Equation 5: + `p_L = R(T−1)/T`, and `p_i = R/T^(L−i+1)` below. The closed form (Equation 6) + is `R = e^(−(M/N)ln²2) · T^(T/(T−1)) / (T−1)^((T−1)/T)`, whose trailing factor + is "a small constant for any value of T" — hence `O(e^(−M/N))`, same as + leveling. + + The price is that constant. At T = 10, L = 4, 10 bits per entry: leveling gets + `R = 0.01174`, lazy leveling `R = 0.01465` — **25% more wasted IOs** — funded + by moving ~0.5 bits/key off the bottom level onto the upper levels (which need + ~+4.1 bits/key each to survive the ×(T−1)). + + There is also a floor: below `M/N = (1/ln²2)(ln T/(T−1) + ln(T−1)/T)` bits per + entry the FPRs converge to 1 and the argument stops. That threshold peaks at + **1.62 bits/entry** (at T = 3) and is 0.99 at T = 10 — an order of magnitude + below the 10-to-16 bits/entry everyone actually uses (Equation 7 and the + paragraph after it). + +
+ +- [ ] You can state the Fluid LSM parameterization and say which knob drives which costs. + +
Answer + + `K` = runs tolerated at each of levels 1…L−1; `Z` = runs tolerated at the + largest level. `K=1, Z=1` is leveling; `K=T−1, Z=T−1` is tiering; + `K=T−1, Z=1` is lazy leveling (§4.2). Each level keeps an active run that + incoming runs merge into, capped at `T/K` percent of the level's capacity + (`T/Z` at level L). + + From Figure 8: update `O((T/B)(L/K + 1/Z))`, zero-result lookup + `O(Z·e^(−M/N))`, existing-key lookup `O(1 + Z·e^(−M/N))`, short range + `O(Z + (L−1)·K)`, long range `O(s·Z/B)`, space amp `O((Z−1) + 1/T)`. + + **`Z` appears in every row; `K` appears only in the update row and the + short-range row.** So the bottom-level knob controls all the read and space + behaviour, and the upper-level knob is a pure trade between write cost and + short scans — which is Step 2's attribution table restated as algebra. + +
+ +- [ ] You can say what the paper's evaluation actually reports, without inventing a speedup. + +
Answer + + §5: Dostoevsky implemented **on RocksDB** (Equation 9's allocation embedded in + the code, Fluid LSM-tree built on RocksDB's event-listener API); a RAID of + 500 GB 7200 RPM disks, 32 GB DDR4, 4 × 2.7 GHz cores, 8 MB L3, Ubuntu 16.04, + ext4 journaling off; direct IO, 2 MB buffer, 10 bits per entry of filters, + fence pointers per 32 KB block, block cache at 10% of the dataset. + + Figure 10 plots **normalized** throughput against the proportion of + zero-result lookups (0.5% → 95%); the y-axis is scaled to Dostoevsky's own + result, so the figure shows *dominance*, not a factor. The paper's words are + "strictly dominates" (abstract) and "dominates all fixed policies by + encompassing all of them" (§5). No headline speedup percentage exists to + quote — the quantitative content is Figure 6's complexities, Equations 3-14, + and the `T,Z,K` tunings printed above Figure 10(A), which sweep from + tiering-like settings at the update-heavy end to `Z=1, K=1` at the + lookup-heavy end. + +
## References **Papers** -- Dayan & Idreos — "Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree - Based Key-Value Stores via Adaptive Removal of Superfluous Merging" - (SIGMOD 2018) — §2's cost table (Table 1) IS the paper; §3 for the lazy - leveling analysis, §4 for Fluid LSM (skim the solver, keep the knobs) +- Dayan & Idreos — *Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based + Key-Value Stores via Adaptive Removal of Superfluous Merging*, SIGMOD 2018. + §3 for the problem analysis, §4.1 for Lazy Leveling (Figure 6 and Equations + 3-8), §4.2 for Fluid LSM-tree (Figure 8), §4.3 for the tuner, §5 for the + evaluation. Appendix A is the Lagrange derivation, Appendix B the closed form + for R, Appendix C the sub-1.62-bits/entry adaptation. + +| Claim in this chapter | Source | +|---|---| +| Lazy Leveling = tiering above, leveling at level L | §4.1, "Basic Structure" | +| Most merges reduce lookup cost and space "by a negligible amount" | Abstract | +| Update cost `O((L+T)/B)`; leveling `O(L·T/B)`; tiering `O(L/B)` | §4.1 "Updates"; Figure 6(A) | +| Space amp `O(1/T)` for lazy leveling and leveling, `O(T)` for tiering | §4.1 "Space-Amplification"; Figure 6(G) | +| Short range `O(1 + (L−1)·T)`; long range `O(s/B)` | §4.1 "Range Lookups"; Figure 6(B), (C) | +| `R = p_L + (T−1)·Σ p_i` | §4.1, Equation 3 | +| Filter memory model, same regardless of runs per level | §4.1, Equation 4 and preceding sentence | +| `p_L = R(T−1)/T`, `p_i = R/T^(L−i+1)` | §4.1, Equation 5 (Appendix A) | +| Closed form for R; complexity `O(e^(−M/N))` | §4.1, Equation 6 (Appendix B) | +| Memory floor: 0.99 bits/entry at T=10, max 1.62 at T=3 | §4.1, Equation 7 | +| `V = 1 + R − p_L`, `O(1)` | §4.1, Equation 8 | +| K/Z parameterization and the three named policies | §4.2, "Parameterization" | +| Fluid cost table in K and Z | Figure 8 | +| Throughput objective, convexity, `O(log₂(N/BP)³)` search | §4.3, Equation 14 | +| Built on RocksDB; disk/memory setup; 10 bits/entry | §5, "Experimental Infrastructure" and "Implementation" | +| Normalized throughput, "dominates", no speedup factor | §5 and Figure 10; abstract | + +**Code** +- `lsm-tree src/compaction/mod.rs:65-97` at `8526dd3` — the `Choice` enum and + the `CompactionStrategy::choose` signature a Fluid strategy would implement; + `:18` and `:24` show the tiered strategy commented out of the module tree. + +**Repo cross-references** +- `topics/04-lsm-deep-dive/notes.md` — the `T/2 × L` and `L` write-amp model + used in Step 4. +- `FINDINGS.md` row 1 — the measured fjall-vs-redb space-amplification lane borrowed + in Step 4, since topic 4 has no lane of its own. +- `topics/04-lsm-deep-dive/reading-monkey.md` — the filter allocation Step 5 + extends. diff --git a/topics/04-lsm-deep-dive/reading-lsm-tree.md b/topics/04-lsm-deep-dive/reading-lsm-tree.md index 937d37c..7846d81 100644 --- a/topics/04-lsm-deep-dive/reading-lsm-tree.md +++ b/topics/04-lsm-deep-dive/reading-lsm-tree.md @@ -5,235 +5,885 @@ point reads, versioned level metadata, pluggable compaction — exists as a few hundred readable lines in fjall's `lsm-tree` crate. Topic 1 read fjall's keyspace layer; everything LSM-shaped delegates here, and the crate is small enough to read completely. Before you open it, this chapter builds the whole -machine one layer at a time — block, segment, filter, level, compaction, read +machine one layer at a time — block, table, filter, level, compaction, read path — then hands you the file and line anchors to watch each layer in code. +Every anchor below is **fjall-rs/lsm-tree at `8526dd3`**, the commit this repo +pins (`tools/pinned-source.py ref lsm-tree`), quoted with the line numbers the +code occupies in that revision. One naming note before you start: the crate +calls a flushed file a **table** (`src/table/`, `TableId`), not a *segment* — +that is the same object RocksDB calls an SST, and the topic README's "segment" +is the older vocabulary. This chapter uses the crate's word. + ## The problem in one sentence Absorb writes at sequential-disk speed by only ever appending sorted files — and then keep a point read from having to search *dozens* of those files: a -naive pile of 100 flushed files means up to 100 disk probes per `get`, and +naive pile of 100 flushed tables means up to 100 disk probes per `get`, and every mechanism in this crate exists to push that back toward 1. ## The concepts, step by step ### Step 1 — the shape of the machine: buffer, flush, merge -An LSM engine never modifies data on disk; it buffers writes in a sorted -in-memory structure (the **memtable** — your topic 2 skiplist) and, when that -fills (say 8 MB), writes its contents out as one immutable sorted file called -a **segment** (RocksDB calls it an SST, "sorted string table"). Deletes are -writes too: a **tombstone** (a key marked "deleted") is inserted like any -other entry, because you can't erase from files you never modify. Background -**compaction** merges accumulated segments into fewer, bigger ones so reads -stay bounded. Everything below is one of those three verbs — buffer, flush, -merge — made concrete. The cost baked into the shape: a key can now exist in -several places at once (memtable and multiple segments), so every read must -consult them *newest first* and take the first hit. +> **In:** nothing yet — this step fixes the three verbs and the four words +> (memtable, table, tombstone, amplification) every later step leans on. +> **Out:** a stream of sorted key-value pairs arriving at a file writer, which +> is exactly what Step 2 encodes. + +An LSM engine never modifies data on disk. It buffers writes in a sorted +in-memory structure — the **memtable**, your topic 2 skiplist — and when that +fills (fjall's default is 64 MiB, `fjall src/keyspace/options.rs:91`) it writes +the contents out as one immutable sorted file, the **table**. Deletes are +writes too: a **tombstone** is a key stored with the marker "this key is +deleted" instead of a value, because you cannot erase from files you never +modify. The crate's `ValueType` enum has four variants, two of which are +tombstones: + +```rust +// src/value_type.rs — the on-disk marker byte, 5-22 (the `is_tombstone` test is 27-29) + 5 /// Value type (regular value or tombstone) + 6 #[derive(Copy, Clone, Debug, Eq, PartialEq)] + 7 #[cfg_attr(test, derive(strum::EnumIter))] + 8 pub enum ValueType { + 9 /// Existing value + 10 Value, + 11 + 12 /// Deleted value + 13 Tombstone, + 14 + 15 /// "Weak" deletion (a.k.a. `SingleDelete` in `RocksDB`) + 16 WeakTombstone, + 17 + 18 /// Value pointer + 19 /// + 20 /// Points to a blob in a blob file. + 21 Indirection = 4, + 22 } +``` + +Background **compaction** then merges accumulated tables into fewer, bigger +ones so reads stay bounded. Everything below is one of those three verbs — +buffer, flush, merge — made concrete. + +Three cost words, because the rest of the topic argues in them. **Write +amplification** is bytes physically written to disk per byte of user data. +**Read amplification** is places consulted per lookup. **Space amplification** +is bytes on disk per byte of live data. They trade against each other, and the +trade is the topic: this crate's leveled strategy documents itself as "high +write amplification, decent read amplification and great space amplification +(~1.1x)" at `src/compaction/leveled/mod.rs:119`. + +The measured version of that claim, from this repo rather than from the crate's +doc comment: topic 1's lane writes 1.08 M records of 100 B each — 108,000,000 B +logical — through fjall and through redb, and reports **48,429,915 B on disk +for fjall against 6,833,917,952 B for redb**, space amp **0.45× vs 63.28×** +([FINDINGS.md](../../FINDINGS.md) row 1). The LSM lands *below* 1.0 because +Step 2's block compression is on; the copy-on-write B-tree lands at 63× because +1080 random-key commits each copy a root-to-leaf path. That is the machine this +chapter takes apart. + +The cost baked into the shape: a key can now exist in several places at once +(memtable and several tables), so every read must consult them *newest first* +and take the first hit. Steps 4–7 are all about making "several places" cheap. + +### Step 2 — the block: prefix truncation against a restart head + +> **In:** the sorted key-value stream from Step 1, as the table writer receives +> it. +> **Out:** one ~4 KB compressed, checksummed block, plus the offset of each +> restart head — the unit Step 3 assembles into a table. + +A table's data is cut into **blocks**: ~4 KB chunks that are the unit of IO, +checksum and compression (`BlockSizePolicy::all(4_096)`, +`src/config/mod.rs:261`). Inside a block, sorted neighbours share prefixes, so +most entries store `shared_prefix_len + rest` instead of the whole key. That +would break random access if every entry were relative to its predecessor, so +every 16th entry is written in FULL as a **restart head** — a self-contained +entry a search can start from. The **restart interval** is how many entries one +head covers: 16 for data blocks, 1 for index blocks +(`src/config/mod.rs:256-257`). A **binary index** over the restart heads' file +offsets is what makes the block searchable: binary-search the heads, then +linear-scan at most 15 entries. + +The whole encoder is 38 lines, and one of them is the detail the textbook +account gets wrong: -### Step 2 — the block: prefix truncation with restart points +```rust +// src/table/block/encoder.rs — Encoder::write, 122-159 + 122 pub fn write(&mut self, item: &'a Item) -> crate::Result<()> { + 123 // NOTE: Check if we are a restart marker + 124 if self + 125 .item_count + 126 .is_multiple_of(usize::from(self.restart_interval)) + 127 { + 128 self.restart_count += 1; + 129 + 130 if self.restart_interval > 0 { + // ... 131-134: a clippy allow for the u32 cast ... + 135 self.binary_index_builder.insert(self.writer.len() as u32); + 136 } + 137 + 138 item.encode_full_into(&mut *self.writer, &mut self.state)?; + 139 + 140 self.base_key = item.key(); + 141 } else { + 142 let shared_prefix_len = longest_shared_prefix_length(self.base_key, item.key()); + 143 item.encode_truncated_into(&mut *self.writer, &mut self.state, shared_prefix_len)?; + 144 } + // ... 145-158: hash-index bookkeeping (see below) and item_count += 1 ... + 159 } +``` -A segment's data is cut into **blocks** — ~4 KB chunks that are the unit of -IO, checksum, and compression — and inside a block, sorted neighbors share -prefixes, so each entry stores only `shared_prefix_len + rest` relative to -its predecessor. That breaks random access (decoding entry N needs entry -N−1), so every 16th entry is written in FULL as a **restart point**: binary -search jumps between restart points, then linear-decodes at most 15 entries. +The line to look at is **142**, together with **140**. `self.base_key` is +assigned only inside the restart branch, so the shared prefix is computed +against the **restart head**, not against the immediately preceding entry. +That is not the LevelDB scheme this is usually described as: it truncates less +(entry 15 shares only what it shares with entry 0, not with entry 14), and in +exchange decoding any entry needs *only* the restart head, never a chain of +15 predecessors. `parse_truncated` takes exactly that one extra argument, +`base_key_offset` (`src/table/data_block/mod.rs:120-124`). + +Worked example, counting key bytes only. Take 16 consecutive keys +`user:0000001040:profile` … `user:0000001055:profile`, each 23 bytes, one +restart interval's worth. A full entry writes one varint key length plus the +key; a truncated entry writes two varints (shared length, rest length) plus the +rest: ``` - inside one 4 KB block (restart interval 16): - [FULL key ∥ v][shared=5,rest ∥ v][shared=7,rest ∥ v]…[FULL key]…[restart offsets] - ▲ binary search over restart points, linear decode between them +full: 16 × (1 + 23) = 384 bytes +truncated: (1 + 23) head, written in full = 24 + + 9 × (2 + 9) keys …1041-1049 share 14 of 23 bytes = 99 + + 6 × (2 + 10) keys …1050-1055 share 13 of 23 bytes = 72 + ---- + 195 bytes + +saved 384 − 195 = 189 bytes, 49.2% of the key region +average entry: 24.0 bytes full → 12.19 bytes truncated +``` + +Nearly twice as many entries per 4 KB block, and therefore about half as many +blocks per lookup — paid for with a linear scan of up to 15 variable-length +records after the binary search lands. + +Why this is safe here and not in a B-tree page: blocks are **immutable**, so no +in-place update can break the truncation. The same immutability buys the rest of +the block header for free. Each block carries an **xxh3-128 checksum** — a fast +non-cryptographic hash whose only job is detecting corruption +(`src/hash.rs:7-9`) — computed over the *compressed* bytes at +`src/table/block/mod.rs:70` and verified before decompression at `:94-102`, so a +corrupt block is caught without ever being handed to LZ4. Compression itself is +per level, not global: the default policy is `[None, Lz4]` +(`src/config/mod.rs:275-283`), i.e. L0 uncompressed for flush speed and LZ4 +everywhere below. Read the `#[cfg]` on 276-280 before you quote that: lsm-tree's +own `default = []` (`Cargo.toml:20`) leaves the policy at `[None]`, and it is +fjall — `default = ["lz4"]`, `fjall Cargo.toml:20-21` — that turns the second +entry on. The 0.45× space amplification in Step 1 was measured through fjall, so +it includes LZ4. + +One honest correction to the folklore: the crate does embed an optional +per-block **hash index** (a tiny in-block hash table from key to restart index, +`encoder.rs:148-154`), but the default `HashRatioPolicy::all(0.0)` +(`src/config/mod.rs:286`) leaves it switched off, so the shipped read path is +binary search over restart heads. + +### Step 3 — the table: one forward pass, three outputs + +> **In:** the block stream from Step 2, plus every distinct user key as it goes +> past. +> **Out:** the fork this chapter turns on — data blocks (consumed by Step 7's +> disk read), index entries (consumed by Steps 5 and 7 to find the right block), +> and filter bits (consumed by Step 4). One pass, three artifacts, three +> different downstream readers. + +Because a table is immutable it can be written **append-only in a single +forward pass**, and the writer is where the three artifacts diverge: + +```rust +// src/table/writer/mod.rs — inside Writer::write, 266-296 + 266 // NOTE: Check if we visit a new key + 267 if Some(&user_key) != self.current_key.as_ref() { + 268 self.meta.key_count += 1; + 269 self.current_key = Some(user_key.clone()); + // ... 270-274: comment — do not buffer every item's key, there may be + // multiple versions of the same one ... + 275 if self.bloom_policy.is_active() { + 276 self.filter_writer.register_key(&user_key)?; + 277 } + 278 } + // ... 279-287: first_key bookkeeping, chunk push ... + 288 if self.chunk_size >= self.data_block_size as usize { + 289 self.spill_block()?; + 290 } + // ... 291-295: seqno range bookkeeping ... + 296 } ``` -Why this is safe here and not in a B-tree page: blocks are **immutable** — -written once, never edited — so no in-place update can ever break the delta -chain. The payoff is real: keys like `user:1042:profile` shrink to a few -bytes each, so more entries fit per 4 KB block, so fewer blocks per lookup. -Each block also carries an **xxh3 checksum** (a fast non-cryptographic hash -that detects corruption) and optional LZ4 compression — both trivial to add -when nothing mutates. The crate even embeds an optional per-block hash index, -a tiny SwissTable-ish shortcut inside each block; the topic 2 pattern at yet -another scale. +Line **276** is the filter fork and line **288** is the block fork. Note the +guard on 267: `register_key` fires once per *distinct user key*, so ten versions +of one key cost the filter one entry, not ten — a filter sized by live keys, not +by writes. `spill_block` (303-366) then encodes the buffered chunk, writes it +through `Block::write_into`, and registers the block's last key and file offset +with the index writer at **332-337**. That is the third artifact. -### Step 3 — the segment: an SST written in one forward pass +The order the pieces land in the file is decided by `finish`: -A segment is blocks plus a table of contents, and because it is immutable it -can be written **append-only in a single pass**: buffer key-value pairs, -flush a data block whenever ~4 KB accumulate, remember each block's last key -and file offset for the **index block** (which maps key ranges to block -positions), feed every key's hash to the filter builder (Step 4), and write -index + filter + trailer/metadata last: +```rust +// src/table/writer/mod.rs — inside Writer::finish, 374-388 and 414, 518-528 + 374 self.spill_block()?; + // ... 376-380: delete the file and return None if nothing was written ... + 382 // Write index + 383 log::trace!("Finishing index writer"); + 384 let index_block_count = self.index_writer.finish(&mut self.file_writer)?; + 385 + 386 // Write filter + 387 log::trace!("Finishing filter writer"); + 388 let filter_block_count = self.filter_writer.finish(&mut self.file_writer)?; + // ... 390-411: linked blob files, table_version byte ... + 414 self.file_writer.start("meta")?; + // ... 416-514: the meta block — key_count, restart intervals, seqno range … ... + 518 let mut checksum = self.file_writer.into_inner()?; + 519 checksum.inner_mut().get_mut().sync_all()?; + // ... 520-527: take the file's checksum, then a clippy allow for the fsync ... + 528 fsync_directory(self.path.parent().expect("should have folder"))?; +``` + +So the on-disk order at `8526dd3` is data blocks → **index** (384) → **filter** +(388) → meta (414), then `sync_all` on the file (519) and an fsync of the +containing directory (528) so the new file's *name* is durable too: ``` - ┌─────────────┬─────────────┬──────┬──────────────┬────────┬─────────┐ - │ data block │ data block │ … │ filter block │ index │ trailer │ - │ (~4KB, LZ4) │ │ │ (bloom) │ block │ /meta │ - └─────────────┴─────────────┴──────┴──────────────┴────────┴─────────┘ + ┌─────────────┬─────────────┬──────┬────────┬──────────────┬─────────┐ + │ data block │ data block │ … │ index │ filter block │ meta │ + │ (~4KB, LZ4) │ │ │ block │ (bloom) │ /trailer│ + └─────────────┴─────────────┴──────┴────────┴──────────────┴─────────┘ + written first, streaming written at finish(), 384 → 388 → 414 ``` -A point read inside one segment costs: index lookup (usually cached) → one -data block read → binary search over restart points. One segment ≈ one disk -IO. The problem is *how many segments* — Steps 4 and 5. +That is index-before-filter — the reverse of the ASCII diagram in the topic +README, which draws the RocksDB-flavoured layout. Neither order matters to a +reader, because both are found through the meta block at the end; it matters +only if you are checking the guide against the code, which is the point. + +A point read inside one table then costs: index lookup (usually cached) → one +data block read → binary search over restart heads → up to 15 entries of linear +scan. One table ≈ one disk IO. The problem is *how many tables* — Steps 4 and 5. ### Step 4 — the bloom filter: paying DRAM to skip IO +> **In:** the distinct-key stream forked off at Step 3's line 276. +> **Out:** one filter block per table, sized `m` bits with `k` probes — the +> gate Step 7 checks before it is allowed to spend an IO. + A **bloom filter** is a probabilistic set summary: a bit array plus k hash -functions that can answer "definitely not present" or "maybe present" — -never a false negative, occasionally a false positive. Each segment carries -one over all its keys, so a read checks the filter (a few DRAM probes, ~100 -ns) before paying a disk IO (~100 µs) for a segment that probably doesn't -have the key. The classic sizing: 10 bits per key ⇒ ~1% false positive rate -with k≈7 probes (`m,k` from `−n·ln(fpr)/ln²2`). - -Seven hash computations per key would be expensive, so the crate uses -**double hashing**: compute two real hashes and derive all k probe positions -via `h1 += h2; h2 *= i` — k memory probes but only ONE real hash -computation. Compare RocksDB's cache-local bloom, which keeps all k bits in -one cache line (topic 0 priced why: k random probes into a big bit array is -k potential cache misses). The trade to hold: ~1.25 bytes of DRAM per key -buys skipping ~99% of pointless segment reads — Monkey (this topic's paper) -optimizes exactly this budget. - -### Step 5 — runs, levels, and the version: keeping "newest first" cheap - -A **run** is a set of segments whose key ranges are *disjoint* (no overlap), -so finding which segment might hold a key is a binary search over ranges — -**one segment probed per run**. Levels organize runs by age and size: -**L0** is special — every memtable flush lands there as its own tiny run, -and flushes overlap arbitrarily, so a read must probe *every* L0 run; L1 and -deeper are each one disjoint run, each level ~10× larger than the one above. - -``` - L0: [run][run][run][run] ← one run per flush, overlapping: probe ALL - L1: [────── one disjoint run ──────] ← binary search: probe 1 - L2: [───────────── one run, 10× bigger ─────────────] ← probe 1 -``` - -The metadata saying "these segments, in these runs, at these levels" is the -**version** — an immutable snapshot of the tree's file layout. Compaction -never mutates a version; it writes a *new* version file, checksums it, and -atomically renames it into place (`rewrite_atomic` + fsync). This is -RocksDB's MANIFEST in miniature: **compaction commits by publishing a new -version, never by mutating the old one** — copy-on-write again, at the -metadata level. Cost of L0's laxness: at 20+ L0 runs a point read does 20+ -filter checks, which is why every engine eventually stalls writers (topic -README §3). +functions, answering "definitely not present" or "maybe present", never a false +negative. A **false positive** is a "maybe" for a key that is not there, and the +**false-positive rate (FPR)** is how often that happens. Each table carries one +filter over all its keys, so a read checks DRAM (tens of nanoseconds) before +paying an IO for a table that probably does not have the key. + +The sizing formula is the standard one, and the crate computes it in +`calculate_m`: + +``` +m = ceil_to_byte( −n · ln(fpr) / ln²2 ) src/table/filter/standard_bloom/builder.rs:129-150 +k = floor( bits_per_key · ln2 ), minimum 1 :79 and :111 + + n = number of distinct keys the filter will hold + fpr = the target false-positive rate, e.g. 0.01 + m = bits in the array, rounded up to a whole byte (:148) + k = how many bits each key sets and each probe tests + ln2 = 0.6931…, and ln²2 = 0.4805… +``` + +Work it on the crate's own unit test, which asserts +`calculate_m(1_000, 0.01) == 9_592` (`builder.rs:184`): + +``` +n = 1000 keys, fpr = 0.01 +m = −1000 × ln(0.01) / 0.4805 = −1000 × (−4.6052) / 0.4805 = 9584.6 → 9592 bits (byte-aligned) +bits per key = 9592 / 1000 = 9.592 + +k, as the textbook computes it: round(9.592 × 0.6931) = round(6.648) = 7 +k, as the crate computes it: bpk is `(m / n) as f32` at :72 — usize division, + so 9592 / 1000 = 9, and k = (9 × 0.6931) as usize = 6 + +actual FPR = (1 − e^(−k/bpk))^k at bpk = 9.592 + k = 6: (1 − e^(−0.6255))^6 = 0.46500^6 = 1.011% + k = 7: (1 − e^(−0.7298))^7 = 0.51806^7 = 1.000% +``` + +So the crate ships **k = 6, not the optimal 7**, because two truncating casts +(`(m / n) as f32` at :72, `as usize` at :79) round the bits-per-key down before +the multiply — and it costs 0.011 percentage points of FPR, which is why nobody +has noticed. The same arithmetic at the crate's *default* filter policy, +`BloomConstructionPolicy::BitsPerKey(10.0)` (`src/config/mod.rs:288-290`): + +``` +bpk = 10 → k = (10 × 0.6931) as usize = 6 (optimal would be 7) + k = 6: 0.844% false positives + k = 7: 0.819% false positives +``` + +Ten bits per key is 1.25 bytes of DRAM per key, and it buys skipping ~99.2% of +the pointless table reads. That is the budget Monkey (this topic's next chapter) +spends differently. + +Seven hash computations per key would be expensive, so the crate uses **double +hashing**: compute one real hash and derive all k probe positions from it +arithmetically. + +```rust +// src/table/filter/standard_bloom/mod.rs — StandardBloomFilterReader::contains_hash, 102-121 + 102 pub fn contains_hash(&self, mut h1: u64) -> bool { + 103 let mut h2 = secondary_hash(h1); + 104 + 105 for i in 1..=(self.k as u64) { + 106 let idx = h1 % (self.m as u64); + // ... 108-111: a clippy allow for the usize cast ... + 112 if !self.has_bit(idx as usize) { + 113 return false; + 114 } + 115 + 116 h1 = h1.wrapping_add(h2); + 117 h2 = h2.wrapping_mul(i); + 118 } + 119 + 120 true + 121 } +``` + +Lines **116-117** are the trick: `h1 += h2; h2 *= i` walks k positions with two +integer operations each. The only real hash is `xxh3_64` (`src/hash.rs:2-4`, via +`Builder::get_hash` at `builder.rs:172-174`), and `secondary_hash` +(`builder.rs:10-13`) derives `h2` from `h1` with a shift and a multiply. Line +113 is the early exit: the *first* zero bit ends the probe, so a true negative +usually costs fewer than k memory touches. The build side is the identical loop +with `enable_bit` instead of `has_bit` (`builder.rs:153-168`). + +Cost note the crate makes explicit: filter blocks are pinned in memory by +policy, and the default is `PinningPolicy::new([true, false])` +(`src/config/mod.rs:264`), which by `PinningPolicy::get`'s "index by level, last +entry repeats" rule (`src/config/pinning.rs:18-24`) means **pinned for L0 tables +only**. At deeper levels the filter block itself is fetched through the cache +(`src/table/mod.rs:267-275`), so a filter check down there is not unconditionally +free. + +### Step 5 — runs, levels and the version: keeping "newest first" cheap + +> **In:** the tables written by Step 3, now many of them. +> **Out:** the *version* — the immutable list of which tables are in which run +> at which level. Step 6 rewrites it; Step 7 walks it. + +A **run** is a set of tables whose key ranges are *disjoint*, so finding which +table might hold a key is a binary search over ranges — **one table probed per +run**: + +```rust +// src/version/run.rs — Run::get_for_key, 98-103 + 98 /// Returns the table that may possibly contains the given key. + 99 pub fn get_for_key(&self, key: &[u8]) -> Option<&T> { + 100 let idx = self.partition_point(|x| x.key_range().max() < &key); + 101 + 102 self.0.get(idx).filter(|x| x.key_range().min() <= &key) + 103 } +``` + +Line **100** is the binary search (`partition_point` is Rust's), and line 102 is +the "or nobody" case: the run may simply have no table covering that key, which +costs zero IO. A **level** is a list of runs, and it is disjoint exactly when it +holds one: + +```rust +// src/version/mod.rs — GenericLevel::is_disjoint, and the level-count default, 31 and 67-69 + 31 pub const DEFAULT_LEVEL_COUNT: u8 = 7; + ... + 67 pub fn is_disjoint(&self) -> bool { + 68 self.run_count() == 1 + 69 } +``` + +**L0** is the exception by construction: every memtable flush lands there as its +own run, and flushes overlap arbitrarily, so a read must probe *every* L0 run. +L1 and deeper are one run each, each level targeted about 10× larger than the +one above. The leveled strategy's defaults make that concrete — +`l0_threshold: 4`, `target_size: 64 MiB`, `level_ratio_policy: vec![10.0]` +(`src/compaction/leveled/mod.rs:135-143`) — with `level_base_size = target_size +× l0_threshold` (`:183-185`) and each deeper level multiplied by the ratio +(`:196-230`): + +``` + L1 target = 64 MiB × 4 = 256 MiB + L2 target = 256 MiB × 10 = 2.50 GiB + L3 target = 2.5 GiB × 10 = 25.0 GiB + L4 target = 25 GiB × 10 = 250 GiB + + L0: [run][run][run][run] ← one run per flush, overlapping: probe ALL 4 + L1: [────── one disjoint run, 256 MiB ──────] ← binary search: probe 1 + L2: [───────── one run, 2.5 GiB ──────────────────] ← probe 1 +``` + +Read amplification, worked at those defaults: 4 L0 runs at the compaction +trigger plus 6 deeper levels (`DEFAULT_LEVEL_COUNT = 7`, and `choose` asserts +exactly that at `leveled/mod.rs:278`) is **10 runs**, so a `get` for an absent +key does at most 10 filter checks. At Step 4's measured default FPR of 0.844% +per filter, the expected number of *wasted* data-block reads is +`10 × 0.00844 = 0.084` per zero-result lookup — one in twelve. Fill L0 to 20 +runs instead of 4 and it is `26 × 0.00844 = 0.22`. That single multiplication is +the reason write stalls exist (the RocksDB chapter, Step 4) and the reason +Monkey argues about how those bits are split. + +The metadata saying "these tables, in these runs, at these levels" is the +**version**, and compaction never mutates one. It writes a whole new one: + +```rust +// src/version/persist.rs — persist_version, 16-17 and 35-42 (the file is 45 lines) + 16 let path = folder.join(format!("v{}", version.id())); + 17 let file = std::fs::File::create_new(path)?; + ... + 35 let checksum = writer.checksum(); + 36 + 37 let mut current_file_content = vec![]; + 38 current_file_content.write_u64::(version.id())?; + 39 current_file_content.write_u128::(checksum.into_u128())?; + 40 current_file_content.write_u8(0)?; // 0 = xxh3 + 41 + 42 rewrite_atomic(&folder.join(CURRENT_VERSION_FILE), ¤t_file_content)?; +``` + +Read lines 16-17 and 42 together, because that is the commit protocol and it is +worth being precise about: each version is its **own new file** `v{id}`, written +with `create_new` so it can never clobber an existing one, and fsynced along +with its directory at `:32` *before* anything points at it; the only thing +rewritten in place is the 25-byte `current` file (`src/file.rs:12`) holding the +version id and its checksum, and `rewrite_atomic` does that as temp-file → +fsync → rename → fsync → fsync-directory (`src/file.rs:62-90`). Recovery reads +`current`, opens `v{id}`, and decodes levels → runs → tables +(`src/version/recovery.rs:34-94`). This is RocksDB's MANIFEST+CURRENT pair in +miniature with one difference that matters at scale: RocksDB appends *deltas*, +lsm-tree writes a *full snapshot* of the file layout every time. ### Step 6 — compaction: a k-way merge plus one deferred rule -Compaction picks some input segments, merges them (a **k-way merge**: pop -the smallest key across k sorted iterators, using an interval heap — -double-ended, so reverse scans work too), writes new segments, and publishes -a new version. The crate makes the *policy* a trait — `choose(version, -config, state)` returns `Merge | Move | Drop | DoNothing` — so leveled -(L0 trigger 4 runs, size ratio 10) and any other strategy plug in. Note -`Move`: a segment that doesn't overlap the next level is *relinked* into it, -zero bytes of IO. - -The one subtle rule: **tombstones are evicted only when the compaction's -output is the last level** (`evict_tombstones(is_last_level)`). Drop a -tombstone at L1 while an older version of its key still sits in L3, and the -old value is *resurrected* — the delete silently undone. So deleted keys -physically survive, level by level, until a merge finally carries the -tombstone to the bottom. That's space amplification with a purpose, and the -same reasoning your M4 capstone will need. +> **In:** the version from Step 5, plus a policy object. +> **Out:** a `Choice`, and if it is `Merge` or `Move`, a new version — which is +> the input Step 7 reads against. + +Compaction picks some input tables, merges them, writes new tables and publishes +a new version. The crate makes the *policy* a trait with a four-way answer: + +```rust +// src/compaction/mod.rs — Choice and the strategy trait, 63-80 and 87-97 + 63 /// Describes what to do (compact or not) + 64 #[derive(Debug, Eq, PartialEq)] + 65 pub enum Choice { + 66 /// Just do nothing. + 67 DoNothing, + 68 + 69 /// Moves tables into another level without rewriting. + 70 Move(Input), + 71 + 72 /// Compacts some tables into a new level. + 73 Merge(Input), + // ... 75-79: Drop — delete tables without compacting, used by the FIFO strategy ... + 80 } + ... + 87 pub trait CompactionStrategy { + // ... 88-95: get_name and get_config ... + 96 /// Decides on what to do based on the current state of the LSM-tree's levels + 97 fn choose(&self, version: &Version, config: &Config, state: &CompactionState) -> Choice; +``` + +Line **70** is the one worth stealing. A **trivial move** is a compaction that +rewrites nothing: if the input does not overlap anything at the destination +level, the engine relinks the file into it, zero bytes of IO. Leveled returns it +in two places, and both guard it the same way: + +```rust +// src/compaction/leveled/mod.rs — the two trivial-move sites, 524-527 and 574-577 + 524 if target_level_overlapping_table_ids.is_empty() && first_level.is_disjoint() { + 525 return Choice::Move(choice); + 526 } + 527 return Choice::Merge(choice); + ... + 574 if can_trivial_move && level.is_disjoint() { + 575 return Choice::Move(choice); + 576 } + 577 Choice::Merge(choice) +``` + +524 is the L0→L1 case ("nothing in L1 overlaps, and L0 happens to be one +disjoint run"); 574 is the L1+ case, where `pick_minimal_compaction` +(`:19`, called at `:553`) reports whether the tables it chose can be relinked. +Both require `is_disjoint()` — Step 5's `run_count() == 1`. + +The merge itself is a **k-way merge**: pop the smallest key across k sorted +iterators. The crate's is double-ended, on an interval heap: + +```rust +// src/merge.rs — the heap type and Merger::next, 6 and 85-99 (next_back is 102-117) + 6 use interval_heap::IntervalHeap as Heap; + ... + 85 fn next(&mut self) -> Option { + 86 if !self.initialized_lo { + 87 fail_iter!(self.initialize_lo()); + 88 } + 89 + 90 let min_item = self.heap.pop_min()?; + 91 + // ... 92: a clippy allow for the index ... + 93 if let Some(next_item) = self.iterators[min_item.0].next() { + 94 let next_item = fail_iter!(next_item); + 95 self.heap.push(HeapItem(min_item.0, next_item)); + 96 } + 97 + 98 Some(Ok(min_item.1)) + 99 } +``` + +Line 90 pops the minimum and line 93 refills *only* the iterator it came from — +the standard k-way merge, O(log k) per item. An **interval heap** stores both +ends, so `pop_max` at `:108` gives reverse iteration from the same structure, +which is why a range scan can be run backwards without a second merge path. + +The one subtle rule is when a tombstone may finally be discarded: + +```rust +// src/compaction/worker.rs — the tombstone rule, 381-390 + 381 let dst_lvl = payload.canonical_level.into(); + 382 let last_level = opts.config.level_count - 1; + 383 + 384 // NOTE: Only evict tombstones when reaching the last level, + 385 // That way we don't resurrect data beneath the tombstone + 386 let is_last_level = payload.dest_level == last_level; + 387 + 388 merge_iter = merge_iter + 389 .evict_tombstones(is_last_level) + 390 .zero_seqnos(false); +``` + +Line **386** is the entire rule, and the comment on 384-385 is the reason. Drop a +tombstone at L1 while an older version of its key still sits at L3, and the old +value is *resurrected* — the delete silently undone. So deleted keys physically +survive, level by level, until a merge finally carries the tombstone to level 6 +(`level_count - 1`, with the default 7). That is space amplification with a +purpose, and the same reasoning your M4 capstone will need. ### Step 7 — the read path, end to end -A `get` is now just Steps 1–6 executed newest-first: active memtable → -sealed memtables (newest first) → each run of the version, gated by filters. -Two production touches worth noticing: the key's filter hash is computed -**once** and shared across all segment filter checks (the SipHash-cost -lesson from topic 0 applied), and **seqno filtering** happens at every step -— each entry carries a sequence number (a global write counter), and a read -under a snapshot picks the newest version ≤ its snapshot seqno. That's MVCC -(multi-version concurrency control — readers see a frozen point in time) -falling out of the LSM's "never overwrite" design for free. +> **In:** everything: the memtables of Step 1, the tables of Step 3, the filters +> of Step 4, the version of Step 5 as Step 6 last published it. +> **Out:** one `Option`, and an IO count you can now predict. -The whole path, compressed to its shape: +A `get` is Steps 1-6 executed newest-first. `Tree::get` (`src/tree/mod.rs:639`) +delegates to `get_internal_entry` (`:157`), which is this: ```rust -fn get(&self, key: &[u8], snapshot: SeqNo) -> Option { - if let Some(v) = self.active.get(key, snapshot) { return live(v); } - for mt in self.sealed.iter().rev() { // newest sealed first - if let Some(v) = mt.get(key, snapshot) { return live(v); } - } - let h = hash(key); // hashed ONCE for all filters - for run in self.version.runs() { // L0: run per flush; L1+: one - let Some(seg) = run.get_for_key(key) else { continue }; // disjoint ⇒ binary search - if !seg.filter_maybe_contains(h) { continue; } // bloom: skip the IO - if let Some(v) = seg.point_read(key, snapshot) { return live(v); } - } - None // live(): tombstone ⇒ None -} -``` - -Count the cost: memtable probes are free; each L0 run and each deeper level -is one filter check; actual disk IO only where a filter says "maybe". That -is read amplification tamed — the number this whole crate exists to bound. +// src/tree/mod.rs — Tree::get_internal_entry_from_version, 696-714 + 696 pub(crate) fn get_internal_entry_from_version( + 697 super_version: &SuperVersion, + 698 key: &[u8], + 699 seqno: SeqNo, + 700 ) -> crate::Result> { + 701 if let Some(entry) = super_version.active_memtable.get(key, seqno) { + 702 return Ok(ignore_tombstone_value(entry)); + 703 } + 704 + 705 // Now look in sealed memtables + 706 if let Some(entry) = + 707 Self::get_internal_entry_from_sealed_memtables(super_version, key, seqno) + 708 { + 709 return Ok(ignore_tombstone_value(entry)); + 710 } + 711 + 712 // Now look in tables... this may involve disk I/O + 713 Self::get_internal_entry_from_tables(&super_version.version, key, seqno) + 714 } +``` + +Three probes in strict recency order — active memtable (701), sealed memtables +newest-first (707, and the `.rev()` that makes it newest-first is at `:743`), +then disk (713). Each carries `seqno`, the **sequence number**: a global write +counter stamped on every entry, so a read under a snapshot takes the newest +version at or below its own seqno. That is MVCC — multi-version concurrency +control, readers seeing a frozen point in time — falling out of "never +overwrite" for free. `ignore_tombstone_value` (`:67-73`) is what turns a found +tombstone into `None` rather than a value. + +The disk half is nine lines, and contains the production touch: + +```rust +// src/tree/mod.rs — Tree::get_internal_entry_from_tables, 716-736 + 716 fn get_internal_entry_from_tables( + 717 version: &Version, + 718 key: &[u8], + 719 seqno: SeqNo, + 720 ) -> crate::Result> { + 721 // NOTE: Create key hash for hash sharing + 722 // https://fjall-rs.github.io/post/bloom-filter-hash-sharing/ + 723 let key_hash = crate::table::filter::standard_bloom::Builder::get_hash(key); + 724 + 725 for table in version + 726 .iter_levels() + 727 .flat_map(|lvl| lvl.iter()) + 728 .filter_map(|run| run.get_for_key(key)) + 729 { + 730 if let Some(item) = table.get(key, seqno, key_hash)? { + 731 return Ok(ignore_tombstone_value(item)); + 732 } + 733 } + 734 + 735 Ok(None) + 736 } +``` + +Line **723** is the touch: the key is hashed **once**, outside the loop, and the +same `u64` is handed to every filter — the SipHash lesson from topic 0 applied, +since with 10 runs the naive version would hash the key 10 times. Lines 725-728 +are Step 5's structure walked in order: levels, runs within a level, and +`get_for_key`'s binary search reducing each run to at most one candidate table. +`filter_map` drops the runs that cover no such key range before any IO is +considered. + +The filter gate is inside `Table::get`: + +```rust +// src/table/mod.rs — inside Table::get, 280-292 + 280 if let Some(filter_block) = &filter_block { + 281 if !filter_block.maybe_contains_hash(key_hash)? { + // ... 282-287: metrics — filter_queries += 1, io_skipped_by_filter += 1 ... + 288 return Ok(None); + 289 } + 290 } + 291 + 292 let item = self.point_read(key, seqno); +``` + +Line **288** is the whole value proposition of Step 4: the function returns +before `point_read` (292, defined at `:317`) can touch a data block. The metric +next to it is even named `io_skipped_by_filter`. + +Count the cost of one `get` at the defaults: memtable probes are pure DRAM; +at most 10 runs survive `get_for_key`; each survivor costs one filter check; +and only a "maybe" (0.844% of absent keys, from Step 4) pays for a data block. +That is read amplification tamed — the number this whole crate exists to bound. ## Where each step lives in the code -Read the directories in step order — each layer lands before the one that -uses it. - -- **Step 2 — block encoding, `src/table/block/`**: restart intervals + - prefix truncation in `encoder.rs:61–151` — a FULL item every - `restart_interval` items; between restarts, items store - `shared_prefix_len + rest` (`longest_shared_prefix_length`, :142); - optional per-block hash index (:148–154). Per-block header with xxh3 u128 - checksum + sizes in `header.rs:49–60` (the header itself gets a u32 - checksum, :109). LZ4 per block: `mod.rs:60–70, 111–120`. Index blocks: - `index_block/block_handle.rs:20–43` — varint offset+size handles. -- **Step 3 — segment writer/reader, `src/table/`**: `writer/mod.rs:40–95` — - buffer KVs, flush block at size threshold, feed the filter + index - writers, trailer/metadata last (single forward pass — an SST is written - append-only, like everything else in an LSM). Read path with filter: - `mod.rs:245–290` — filter loaded lazily; if `maybe_contains_hash` says no, - the point read never touches a data block (:281–288). +Read the directories in step order — each layer lands before the one that uses +it. Every line number is `8526dd3`. + +- **Step 1 — vocabulary, `src/value_type.rs`**: the four `ValueType` variants + (`:5-22`), `is_tombstone` (`:27-29`). Amplification claim in the crate's own + words: `src/compaction/leveled/mod.rs:119`. +- **Step 2 — block encoding, `src/table/block/`**: `encoder.rs:122-159` — full + entry at 138, truncation against `base_key` at 140/142, hash index at + 148-154. Decode side, showing the restart head is the only dependency: + `src/table/data_block/mod.rs:120-124` (`parse_truncated`'s `base_key_offset`). + Block header with the xxh3-128 checksum: `header.rs:47-60`, its own u32 + checksum at `:109`. Compress on write and verify-then-decompress on read: + `block/mod.rs:60-65`, `:70`, `:94-102`, `:104-118`. Index handles are varints: + `src/table/index_block/block_handle.rs:20-26` (the struct) and `:45-50` (the + encoding). Defaults: `src/config/mod.rs:256-257` (restart interval 16 data, + 1 index), `:261` (4 KB), `:275-283` (None at L0, LZ4 below), `:286` (hash + index off). +- **Step 3 — table writer, `src/table/writer/mod.rs`**: `write` (243-296) — + filter fork at 275-277, block fork at 288-290; `spill_block` (303-366) — + index registration at 332-337; `finish` (371-539) — index 384, filter 388, + meta 414, `sync_all` 519, directory fsync 528. - **Step 4 — bloom filter, `src/table/filter/standard_bloom/`**: - `builder.rs:55–127` — `with_fp_rate` computes m,k from `−n·ln(fpr)/ln²2` - (:58); `with_bpk` direct (:93). Double hashing: `builder.rs:10–13` + - `mod.rs:102–129`. -- **Step 5 — version + levels, `src/version/`**: `mod.rs:42–114` + - `run.rs:51–103` — levels are runs; `get_for_key` (:99–103) - binary-searches segment ranges. Persistence: `persist.rs:9–45` — new - version file written, checksummed, `rewrite_atomic` on - CURRENT_VERSION_FILE + fsync. Recovery: `recovery.rs:34–95`. -- **Step 6 — compaction, `src/compaction/`**: the trait in `mod.rs:87–98` — - `choose(version, config, state) → Merge | Move | Drop | DoNothing`; find - where leveled uses `Move` (`leveled/mod.rs:19` pick_minimal_compaction). - Leveled policy: `leveled/mod.rs:113–143` — L0 trigger 4 runs, ratio 10. - Worker: `worker.rs:382–389` — `evict_tombstones(is_last_level)`. Merge: - `merge.rs:35–99` — k-way merge on an interval heap. -- **Step 7 — read path, `src/tree/mod.rs`**: `get` :639–643 → - `get_internal_entry` :696–750: active memtable → sealed (newest first) → - levels; seqno filtering at every step (:701/707/730); hash computed once - and shared across all segment filter checks (:721–723). Tombstones: - `value_type.rs:8–27`; hidden at read time (`tree/mod.rs:67–72`), dropped - at bottom-level compaction. + `builder.rs:129-150` (`calculate_m`), `:58-86` (`with_fp_rate`, the truncating + `bpk` at :72 and `k` at :79), `:93-127` (`with_bpk`), `:153-168` (set), + `:10-13` (`secondary_hash`), `:184` (the unit test whose numbers Step 4 works + through). Probe side: `mod.rs:102-121`. Defaults and pinning: + `src/config/mod.rs:288-290`, `:264`, `src/config/pinning.rs:18-24`. +- **Step 5 — version and levels, `src/version/`**: `mod.rs:31` + (`DEFAULT_LEVEL_COUNT = 7`), `:42-78` (`GenericLevel`, `is_disjoint` at + 67-69), `run.rs:51-61` and `:98-103` (`get_for_key`). Level sizing: + `src/compaction/leveled/mod.rs:135-143`, `:183-185`, `:196-230`. Persistence: + `version/persist.rs:16-17`, `:35-42`; `src/file.rs:12`, `:62-90`. Recovery: + `version/recovery.rs:34-94`. +- **Step 6 — compaction, `src/compaction/`**: `mod.rs:63-80` (`Choice`), + `:87-97` (the trait). Trivial moves: `leveled/mod.rs:524-527`, `:574-577`, + with `pick_minimal_compaction` at `:19` (called at `:553`). Tombstones: + `worker.rs:381-390`. The k-way merge is **not** under `compaction/` — it is + `src/merge.rs:6` (interval heap), `:85-99` (`next`), `:102-117` (`next_back`). +- **Step 7 — read path, `src/tree/mod.rs`**: `get` (`:639-643`) → + `get_internal_entry` (`:157`) → `get_internal_entry_from_version` + (`:696-714`) → `get_internal_entry_from_tables` (`:716-736`), with the shared + hash at `:723`; sealed memtables newest-first at `:738-750`. Tombstones hidden + at read time: `:67-73`. Filter gate: `src/table/mod.rs:245-278` (acquire), + `:280-290` (skip), `:292`/`:317` (`point_read`). ## Questions to answer in notes.md 1. Why can L0 not be a disjoint run, and what does that cost a point read? - (Flushes overlap arbitrarily ⇒ probe every L0 run ⇒ the stall trigger.) -2. Restart interval 16: derive the trade (space saved by truncation vs linear - decode cost per lookup). Why don't B-tree pages (topic 3) do this? -3. The version file is rewritten whole on every compaction. RocksDB instead - appends VersionEdits to a MANIFEST log. When does lsm-tree's simpler choice - break down? (Huge segment counts; crash mid-rewrite handled by atomic rename.) + (Flushes overlap arbitrarily ⇒ probe every L0 run. Put a number on it with + Step 5's arithmetic: 4 runs at the trigger versus 20, at 0.844% FPR each.) +2. Restart interval 16: derive the trade — Step 2 measured 49.2% of the key + region saved on 23-byte keys, against a scan of up to 15 variable-length + records per lookup. At what key length and what block-cache hit rate does + that stop being worth it? Why don't B-tree pages (topic 3) do this? +3. The version is written as a whole new `v{id}` file on every compaction + (`persist.rs:16-17`) while RocksDB appends VersionEdits to a MANIFEST log. + When does the simpler choice break down? (Count the bytes: a version file + lists every table id, checksum and seqno per run per level — + `recovery.rs:60-94` shows the exact record — so cost scales with *total* + table count, not with the size of the change.) ## Done when -You can trace one `get` from `tree/mod.rs:639` to a data-block binary search, -naming every filter/index consulted, and explain why tombstones die only at -the bottom. +Answer each before unfolding it. + +- [ ] You can trace one `get` from `src/tree/mod.rs:639` to a data-block binary search, naming every filter and index consulted. + +
Answer + + `Tree::get` (`:639-643`) calls `get_internal_entry` (`:157`), which reaches + `get_internal_entry_from_version` (`:696-714`): active memtable at 701, then + sealed memtables newest-first at 707 (the `.rev()` is at `:743`), then + `get_internal_entry_from_tables` at 713. That function hashes the key once at + `:723` and walks `iter_levels().flat_map(|lvl| lvl.iter())` at 725-727, + reducing each run to at most one candidate table with `Run::get_for_key` + (`src/version/run.rs:98-103`, a `partition_point` binary search over key + ranges at :100). + + Inside `Table::get` (`src/table/mod.rs:229`) the filter block is acquired + (pinned, or loaded through the cache, 245-278) and probed at 281; a "no" + returns at 288 having touched no data block. A "maybe" falls through to + `point_read` at 292/`:317`, which uses the index block to find the one data + block that can hold the key, then binary-searches that block's restart heads + and linear-scans at most 15 entries (restart interval 16, + `src/config/mod.rs:256`). + + At the leveled defaults that is at most 10 runs — 4 L0 runs at the trigger + plus 6 deeper levels — so at most 10 filter checks and, for an absent key, + `10 × 0.844% = 0.084` expected data-block reads. + +
+ +- [ ] You can explain why tombstones die only at the bottom level, and name the line that decides it. + +
Answer + + `src/compaction/worker.rs:386`: `let is_last_level = payload.dest_level == + last_level;`, where `last_level = opts.config.level_count - 1` (:382, so level + 6 at the default `DEFAULT_LEVEL_COUNT = 7`, `src/version/mod.rs:31`). It is + passed straight to `evict_tombstones(is_last_level)` at :389. + + The reason is in the comment at 384-385: a tombstone is only a *marker* that a + key was deleted, and older versions of that key can still exist in any level + below. Drop the marker during an L1→L2 merge while an L3 table still holds the + old value, and the next read finds the old value and returns it — the delete is + silently undone. Only a merge whose output *is* the deepest level can prove + nothing older survives. + + The price is space amplification with a purpose: deleted keys occupy disk until + compaction walks them all the way down, which under leveled compaction means + once per level. This is also why a delete-heavy workload can look like it is + not reclaiming anything. + +
+ +- [ ] You can compute the filter's real false-positive rate from `bits_per_key`, and say why the crate's `k` is not the textbook one. + +
Answer + + `m = ceil_to_byte(−n · ln(fpr) / ln²2)` (`builder.rs:129-150`) and + `k = floor(bits_per_key · ln2)` (`:79`, `:111`), then + `FPR = (1 − e^(−k/bpk))^k`. At the crate's own test point, + `calculate_m(1000, 0.01) = 9592` (`:184`), bits per key is 9.592 and the + textbook `k` is `round(9.592 × 0.6931) = 7`, giving 1.000%. + + The crate computes `k = 6` instead, because line :72 writes + `let bpk = (m / n) as f32` where both are `usize` — integer division turns + 9.592 into 9 before the multiply, and `as usize` at :79 truncates + `9 × 0.6931 = 6.24` to 6. The real FPR at bpk 9.592 with k = 6 is 1.011% + against 1.000% at k = 7: an 0.011-point tax nobody would ever see in a + benchmark. At the default `BitsPerKey(10.0)` (`src/config/mod.rs:289`) the + same truncation gives k = 6 and 0.844% where k = 7 would give 0.819%. + + The honest summary is that the sizing is textbook and the probe count is one + short of optimal, for a cost of about 3% more false positives — which is far + smaller than the effect Monkey is about, namely how the bits are split between + levels in the first place. + +
+ +- [ ] You can say what a "version" is, how a compaction commits one, and how that differs from RocksDB. + +
Answer + + A version is the immutable list of which tables sit in which run at which + level. Compaction never edits one. `persist_version` + (`src/version/persist.rs:9-45`) creates a brand-new file named `v{id}` with + `File::create_new` (16-17) — so it cannot clobber an existing version — writes + the whole layout into it, takes an xxh3 checksum (35), and then rewrites the + 25-byte `current` file with `{version id, checksum, checksum type}` (37-42). + `rewrite_atomic` (`src/file.rs:62-90`) does that as temp file → `sync_all` → + rename → `sync_all` → fsync of the directory, so `current` never points at a + half-written version. Recovery is the same sequence backwards + (`recovery.rs:34-94`): read `current`, open `v{id}`, decode level count, run + counts, then each table's id, checksum and global seqno. + + The difference from RocksDB is what gets written, not whether it is atomic. + RocksDB appends a `VersionEdit` — a delta, "add these files, delete those" — to + an append-only MANIFEST log, so a compaction's metadata cost is proportional to + the *change*. lsm-tree writes a full snapshot, so its metadata cost is + proportional to the *total* number of tables. At a hundred tables the snapshot + is simpler and free; at a hundred thousand it is a rewrite of the whole + catalogue on every compaction. + +
+ +- [ ] You can explain what the shared key hash at `src/tree/mod.rs:723` saves, and why it is safe. + +
Answer + + It saves k−1 … well, it saves *hashes*, not probes: the key is hashed once with + `xxh3_64` (`src/hash.rs:2-4`) outside the loop at :723, and the resulting `u64` + is passed into every `Table::get(key, seqno, key_hash)` at :730. Without it, a + lookup crossing 10 runs would hash the key 10 times. Topic 0's finding — 21% of + a HashMap lookup being SipHash ([FINDINGS.md](../../FINDINGS.md) row 0) — is + the reason that is worth a line of code. + + It is safe because every filter in the tree uses the same hash function and + derives its k probe positions arithmetically from that one value: `h2 = + secondary_hash(h1)` (`builder.rs:10-13`), then `h1 += h2; h2 *= i` + (`standard_bloom/mod.rs:116-117`). The per-filter variation comes from `self.m` + in `h1 % (self.m as u64)` at :106, not from the hash, so two filters of + different sizes get different bit positions out of the same input hash. + + The one thing it does *not* save is the memory probes: each filter still walks + its own k bits, up to k cache misses per table, which is exactly the cost + RocksDB's cache-local bloom attacks (see the compaction chapter, Step 7). + +
## References **Code** - [fjall-rs/lsm-tree](https://github.com/fjall-rs/lsm-tree) — the engine under - fjall; read it all (~3 h): `src/table/block/`, `src/table/`, - `src/table/filter/standard_bloom/`, `src/version/`, `src/compaction/`, - `src/tree/mod.rs`. Local shallow clone at `~/repos/lsm-tree`. + fjall, pinned at `8526dd3`; read it all (~3 h). Verify any anchor below with + `tools/pinned-source.py show lsm-tree -r `. +- [fjall-rs/fjall](https://github.com/fjall-rs/fjall) at `80cf6bc` — + `src/keyspace/options.rs:91` for the 64 MiB memtable default that decides how + often Step 3 runs. + +| File | Lines | What | +|------|-------|------| +| `src/value_type.rs` | 5-22, 27-29 | the tombstone markers, and the test that hides them | +| `src/table/block/encoder.rs` | 122-159 | restart heads (138), truncation against `base_key` (140/142) | +| `src/table/block/header.rs` | 47-60, 109 | per-block xxh3-128 checksum, plus a u32 checksum of the header | +| `src/table/block/mod.rs` | 60-65, 70, 94-118 | LZ4 on write, checksum verified *before* decompression | +| `src/table/writer/mod.rs` | 275-277, 288-290, 384-388 | the one-pass fork: filter, data block, then index-before-filter at `finish` | +| `src/table/filter/standard_bloom/builder.rs` | 72, 79, 129-150, 184 | m and k, the truncating casts, and the unit test Step 4 works through | +| `src/table/filter/standard_bloom/mod.rs` | 102-121 | double hashing, and the early exit at 113 | +| `src/version/mod.rs` | 31, 67-69 | 7 levels by default; "disjoint" means exactly one run | +| `src/version/run.rs` | 98-103 | one binary search per run, or no candidate at all | +| `src/version/persist.rs` | 16-17, 35-42 | new `v{id}` file, then an atomic rewrite of `current` | +| `src/compaction/mod.rs` | 63-80, 87-97 | `DoNothing`, `Move`, `Merge`, `Drop`, and the one-method trait | +| `src/compaction/leveled/mod.rs` | 135-143, 183-185, 524-527, 574-577 | ratio 10 / 64 MiB / L0 trigger 4; both trivial-move sites | +| `src/compaction/worker.rs` | 381-390 | tombstones evicted only when the output is the last level | +| `src/merge.rs` | 6, 85-99, 102-117 | interval-heap k-way merge, forwards and backwards | +| `src/tree/mod.rs` | 67-73, 696-714, 716-736 | the read path, newest-first, with the hash computed once at 723 | diff --git a/topics/04-lsm-deep-dive/reading-monkey.md b/topics/04-lsm-deep-dive/reading-monkey.md index 4dd4ca3..252ff69 100644 --- a/topics/04-lsm-deep-dive/reading-monkey.md +++ b/topics/04-lsm-deep-dive/reading-monkey.md @@ -1,138 +1,326 @@ # Monkey: bloom bits where they pay "10 bits/key everywhere" was folklore; Monkey turned bloom-filter sizing into -an optimization problem and won ~2× fewer wasted IOs from the *same* DRAM. -Before the paper, this chapter builds the argument one step at a time — what -a zero-result lookup costs, how bits buy false-positive rate, why a bit -spent at a small level is 10× cheaper than the same bit at a big one — until -the allocation rule ("FPR proportional to level size") falls out. Then it -sets up the per-level-bits experiment in the mini-LSM. +an optimization problem and won a large factor of wasted IOs back from the +*same* DRAM. Before the paper, this chapter builds the argument one step at a +time — what a zero-result lookup costs, how bits buy false-positive rate, why a +bit spent at a small level is T× cheaper than the same bit at a big one — until +the allocation rule ("FPR proportional to level size") falls out. Then it sets +up the per-level-bits experiment in the mini-LSM. + +Every formula, symbol and number below is checked against the paper — +Dayan, Athanassoulis and Idreos, *Monkey: Optimal Navigable Key-Value Store*, +SIGMOD 2017 — and cited to the section, equation or figure it came from. Where +this guide previously disagreed with the paper, the paper won. ## The problem in one sentence A lookup for a key that *doesn't exist* must be told "no" by every level of -the LSM, and each level's bloom filter lies (says "maybe") about 1% of the +the LSM, and each level's bloom filter lies (says "maybe") about 0.8% of the time at the standard 10 bits/key — so with a fixed DRAM budget for filters, the question is: is spreading it uniformly across levels actually the -division that wastes the fewest disk reads? (Answer: no, by ~2×.) +division that wastes the fewest disk reads? (Answer: no — 2.8× worse than +optimal on the worked example below.) ## The concepts, step by step ### Step 1 — the setup: one filter per level, one shared memory budget -An LSM with L levels (say 3), size ratio T (say 10 — each level holds 10× -more keys than the one above), and a total filter memory budget M. Every -level gets a bloom filter (from the lsm-tree chapter: a bit array answering -"definitely not here" or "maybe here"; more bits per key ⇒ fewer wrong -"maybe"s, called **false positives**). Question: how should M be divided -among the levels? The state of practice — same bits/key everywhere — is an -answer nobody had ever justified. +> **In:** the LSM shape from the lsm-tree chapter — L levels, each T× bigger +> than the one above, one bloom filter per run. +> **Out:** the six symbols the rest of the chapter argues in, and a memory +> budget to divide — which Step 2 turns into an objective function. + +Fix the vocabulary first, because the paper's algebra is unreadable without it +and every symbol here is one the paper uses (Figure 6 is its glossary): + +| symbol | meaning | worked value below | +|---|---|---| +| `N` | total number of entries in the tree | 10,000,000 | +| `T` | **size ratio**: each level holds T× the entries of the one above | 10 | +| `L` | number of levels on disk | 4 | +| `p_i` | false-positive rate of the filter at level *i* (1 = smallest) | to be chosen | +| `M_filters` | total main memory for all filters, in bits | 99,990,000 (≈ 11.9 MiB) | +| `R` | expected number of wasted IOs per zero-result lookup | the thing to minimize | + +A **false positive** is a filter answering "maybe" for a key it does not hold; +the **false-positive rate** is how often that happens. Level sizes follow from +`T` alone. The paper states it in §4.1: the last level holds at most +`N·(T−1)/T` entries, and in general level *i* holds at most +`N/T^(L−i) · (T−1)/T`, "because smaller levels have exponentially smaller +capacities by a factor of T". At N = 10 M, T = 10, L = 4: + +``` + level 1 9,000 entries (0.09% of the data) + level 2 90,000 + level 3 900,000 + level 4 9,000,000 entries (90% of the data) + --------- + total 9,999,000 +``` + +Question: how should `M_filters` be divided among these four filters? The state +of practice — same bits/key everywhere — is an answer nobody had ever justified. +The paper is blunt about it in §2: "To the best of our knowledge, all LSM-tree +based key-value stores use the same number of bits-per-entry across all Bloom +filters." ### Step 2 — what a zero-result lookup costs: the sum of the FPRs +> **In:** the per-level FPRs `p_1 … p_L` from Step 1, still unassigned. +> **Out:** the objective function `R = Σ p_i` — one number to minimize, which +> Steps 3 and 4 show is minimized *unevenly*. + A **zero-result lookup** (probing a key that exists nowhere — the common -case for existence checks, inserts-if-absent, and joins) gets the answer +case for existence checks, insert-if-absent, and joins) gets the answer "no" only after every level's filter says no. Each level is one independent -chance of a false positive, and each false positive costs one wasted disk -IO (~100 µs) probing a segment that doesn't have the key. So: +chance of a false positive, and each false positive costs exactly one wasted +disk IO probing a run that does not have the key. The paper's Equation 3 says +precisely that: + +``` + leveling: R = Σ(i=1..L) p_i one run per level + tiering: R = (T−1) · Σ(i=1..L) p_i up to T−1 runs per level, + all the same size, so all the + same FPR + — Monkey §4.1, Equation 3 +``` + +(The published Equation 3 prints the same `(T−1)·Σ` on both branches, which the +surrounding prose contradicts one paragraph earlier: "With leveling every level +has at most one run and so R is simply equal to the sum of FPRs across all +levels." The prose is the correct reading and is what this chapter uses.) + +Why exactly one IO per false positive, regardless of which level? Because the +run's fence pointers — the index of the lsm-tree chapter's Step 3 — take the +lookup straight to the one qualifying page. The paper leans on this explicitly +in §4.1: "the I/O cost of probing any run is the same regardless of its size +(due to the fence pointers we only fetch the qualifying disk page)". That +sentence is the hinge of the entire argument. + +At uniform 10 bits/key the four filters in Step 1 each have FPR 0.8193% (Step 3 +derives that), so ``` - expected wasted IOs per zero-result lookup = fpr(L1) + fpr(L2) + … + fpr(Lmax) + R_uniform = 4 × 0.008193 = 0.0328 wasted IOs per zero-result lookup ``` -At uniform 10 bits/key, that's ~1% + ~1% + ~1% ≈ 0.03 wasted IOs per lookup -for 3 levels. The objective is now precise: **minimize the sum of per-level -FPRs subject to a fixed total number of bits.** That's an optimization -problem, and the two facts in Steps 3 and 4 make it lopsided. +The objective is now precise: **minimize `Σ p_i` subject to a fixed +`M_filters`.** That is an optimization problem, and the two facts in Steps 3 and +4 make its answer lopsided. ### Step 3 — fact one: bits buy FPR exponentially -A bloom filter's false-positive rate falls *exponentially* in bits per key: -`fpr ≈ e^(−bits·ln²2)`, i.e. every ~1.44 extra bits/key *halves* the FPR. -Concretely: 10 bits/key ⇒ ~0.8%, 12 bits/key ⇒ ~0.3%, 8 bits/key ⇒ ~2.2%. +> **In:** one filter, one bits-per-entry budget. +> **Out:** the conversion `p ↔ bits`, in both directions — the substitution that +> turns Step 2's objective into something differentiable. + +A bloom filter's false-positive rate falls *exponentially* in bits per entry. +The paper states it as Equation 2 in §2: + +``` + FPR = e^( −(bits/entries) · ln(2)² ) Monkey §2, Equation 2 + + rearranged in §4.1 to size a filter: + bits = −entries · ln(FPR) / ln(2)² + + ln(2) = 0.693147 + ln(2)² = 0.480453 +``` + +This is the same formula the lsm-tree crate implements as `calculate_m` +(`src/table/filter/standard_bloom/builder.rs:129-150`), which is a useful +reassurance that the paper is describing shipped filters, not an idealisation. + +Two consequences to have at your fingertips: + +``` + bits/key FPR = e^(−bits·0.480453) + 8 2.143% + 10 0.8193% + 12 0.3132% + 14 0.1197% + 16 0.04578% + + halving the FPR costs ln2 / ln²2 = 1/ln2 = 1.4427 bits per key — always, + at any starting point. +``` + Exponential returns mean the *marginal* value of a bit depends enormously on -where it's spent — the first bits at any level are hugely effective, the -20th bit is nearly worthless. Uniform allocation ignores this curvature. +where it is spent. The first bits at any level are hugely effective; the 20th +bit is nearly worthless. Uniform allocation ignores this curvature entirely. ### Step 4 — fact two: levels differ in size by T×, but not in penalty -The bottom level holds ~T× more keys than the level above it (with T=10 and -3 levels: 90% of all keys are in the last level) — but a false positive at -the bottom level costs exactly the same **one disk IO** as a false positive -at a tiny upper level. Now combine with Step 2's objective: lowering a -*small* level's FPR by some amount requires extra bits for few keys; -lowering the *huge* bottom level's FPR by the same amount requires extra -bits for T× more keys. **A unit of FPR reduction is T× cheaper (in bits) at -a smaller level.** Uniform bits/key is therefore spending most of the budget -where it buys the least. +> **In:** Step 2's uniform per-false-positive penalty and Step 3's per-entry +> cost curve. +> **Out:** the asymmetry — FPR reduction is T× cheaper at each shallower level — +> which Step 5 turns into the closed form. + +The bottom level holds ~T× more entries than the level above it (with T = 10 and +L = 4: 90% of all entries are in level 4) — but a false positive at the bottom +level costs exactly the same **one disk IO** as a false positive at the tiny top +level. Combine that with Step 3: halving a level's FPR always costs 1.4427 bits +*per entry in that level*, so halving level 1's FPR costs + +``` + level 1: 1.4427 × 9,000 entries = 12,984 bits + level 4: 1.4427 × 9,000,000 entries = 12,984,300 bits — 1000× more +``` + +for exactly the same reduction in `R`. **A unit of FPR reduction is T× cheaper +(in bits) at each level you move up.** Uniform bits/key is therefore spending +most of the budget where it buys the least: in the worked example, 90% of +`M_filters` goes to level 4's filter, which contributes exactly one quarter of +`R`, the same quarter as level 1's filter that costs a thousandth as much. ### Step 5 — the optimum: FPR proportional to level size -Minimizing the FPR sum under the bit budget (the paper does it with -Lagrange multipliers; the informal version is "shift bits from where they're -expensive to where they're cheap until marginal value equalizes") gives a -clean closed form: **each level's FPR should be proportional to its size**, -which means bits/key *decrease* geometrically toward the bottom: +> **In:** the objective from Step 2 and the two cost facts from Steps 3-4. +> **Out:** the allocation rule, and a concrete table of bits per key per level +> that Step 6 prices. + +Minimizing `R = Σ p_i` subject to fixed `M_filters` is a multivariate +constrained optimization. The paper solves it with Lagrange multipliers — in +**Appendix B**, not in the body — and reports the result as Equations 5 +(leveling) and 6 (tiering) in §4.1. The conclusion in one sentence, quoted from +§4.1: "the optimal FPR at Level i is T times higher than the optimal FPR at +Level i−1. In other words, the optimal FPR for level i is proportional to the +number of elements at level i." + +Substituting Step 3's conversion, "FPR × T per level down" means bits per key +falls by a *constant* per level: ``` - uniform (state of practice): Monkey (optimal): - - L1 (small) 10 bits/key L1 ~14 bits/key (FPR tiny) - L2 10 bits/key L2 ~12 bits/key - L3 (huge) 10 bits/key L3 ~8 bits/key (FPR larger, but - fewer probes land here anyway) - total FPR cost: sum of per-level expected wasted IOs: MINIMIZED — - FPRs, dominated by... all equally exponentially decreasing FPR up the tree + gap between consecutive levels = ln(T) / ln(2)² + = 2.302585 / 0.480453 + = 4.793 bits per key, at T = 10 ``` -In the limit the bottom level may get ~0 bits — its "filter" is the fact -that every lookup for an existing key ends there anyway, so a filter that -mostly says "maybe" was buying nothing. The whole allocation, as the closed -form your mini-LSM can call: +Worked on the Step 1 tree — N = 10 M, T = 10, L = 4, and the *same* +99,990,000-bit budget (10 bits/key on average) in both columns: + +| level | entries | uniform bits/key | uniform FPR | Monkey bits/key | Monkey FPR | +|---|---|---|---|---|---| +| 1 | 9,000 | 10.00 | 0.8193% | **23.85** | 0.001057% | +| 2 | 90,000 | 10.00 | 0.8193% | **19.05** | 0.01057% | +| 3 | 900,000 | 10.00 | 0.8193% | **14.26** | 0.1057% | +| 4 | 9,000,000 | 10.00 | 0.8193% | **9.47** | 1.057% | +| | | `R` = **0.0328** | | `R` = **0.0117** | | + +Read the two FPR columns as the whole idea. Uniform spends 9.47-plus bits on +every one of the nine million bottom-level entries to buy an 0.8193% FPR there, +and the same 10 bits on each of the nine thousand top-level entries. Monkey +takes half a bit per key away from level 4 — barely moving its FPR, from 0.82% +to 1.06% — and spends the 4.5 million bits that frees on levels 1-3, whose FPRs +fall by factors of 8, 78 and 775. The sum drops from 0.0328 to 0.0117: +**2.79× fewer wasted IOs per zero-result lookup, at identical memory.** + +Run it the other way and the same allocation reaches the uniform tree's +`R = 0.0328` using 78.6 Mbit instead of 99.99 Mbit — **21% less filter DRAM for +identical lookup cost**, on this tree. + +Push `R` higher still and the deepest filters vanish entirely. The paper handles +this explicitly (§4.1): for larger `R`, "more of the Bloom filters at the +deepest levels cease to exist as their optimal FPRs converge to 1", and +Equations 5 and 6 carry a term `L_filtered = L − max(0, ⌊R−1⌋)` for exactly that +— they solve the smaller problem on the shallowest `L_filtered` levels and give +the rest no filter at all. A filter that says "maybe" almost always was buying +nothing. + +The allocation, as the closed form your mini-LSM can call: ```rust -// Pick a total zero-result FPR budget; hand each level a share -// PROPORTIONAL TO ITS SIZE, then convert fpr → bits/key. -fn monkey_alloc(level_keys: &[u64], total_fpr: f64) -> Vec { - let n: u64 = level_keys.iter().sum(); - level_keys.iter().map(|&nk| { - let fpr = total_fpr * nk as f64 / n as f64; // p_i ∝ level size - -fpr.ln() / (LN_2 * LN_2) // bits/key: fpr ≈ e^(−bits·ln²2) - }).collect() // small levels get MORE bits/key +// ILLUSTRATION — not quoted from a repo. The bits↔FPR conversion is Monkey §2 +// Equation 2, the same one lsm-tree implements at +// src/table/filter/standard_bloom/builder.rs:129-150. +fn monkey_alloc(level_entries: &[u64], total_fpr: f64) -> Vec { + let n: u64 = level_entries.iter().sum(); + level_entries + .iter() + .map(|&nk| { + let fpr = total_fpr * nk as f64 / n as f64; // p_i ∝ level size (§4.1) + -fpr.ln() / (LN_2 * LN_2) // bits/key, Equation 2 inverted + }) + .collect() // small levels come out with MORE bits per key } ``` ### Step 6 — what it buys, and where the idea stops -Same total DRAM, ~2× fewer expected false probes on zero-result lookups — -that's the paper's headline evaluation number, and it's free: no new data -structure, just arithmetic at filter-build time. The rule to remember is the -marginal one: **equal IO saved per bit spent, everywhere ⇒ FPR proportional -to level size.** Two boundaries to keep in mind: the argument assumes point -lookups (filters don't help range scans at all — a scan must consult every -run regardless); and the paper's §5 goes on to co-tune the merge policy -itself with the same optimization mindset — skim that, because Dostoevsky -(next chapter) does the merging half properly. +> **In:** the allocation from Step 5. +> **Out:** the measured payoff, the asymptotic claim behind it, and the two +> boundaries — one of which is Dostoevsky's opening. + +The asymptotic result is sharper than the worked example and is the reason the +paper exists (§4.3, Table 1): the state of the art has lookup cost +`O(e^(−M/N) · log_T(N·E / M_buffer))`, i.e. proportional to `L`, while Monkey +has `O(e^(−M/N))`. "Monkey shaves a factor of O(L) from the complexity of lookup +cost for both tiering and leveling… lookup cost R in Monkey is asymptotically +independent of the number of levels L." The intuition is Step 5's table +continued downward: because the FPRs decay geometrically going up, their sum +converges instead of growing with `L`. + +The measured payoff, from §5: + +- **Setup** (§5, "Default Set-up"): Monkey implemented on top of LevelDB, + differing *only* in filter allocation; 1 GB of 1 KB entries; 16 K uniformly + random zero-result point lookups; size ratio 2; 1 MB buffer; + `M_filters/N` = 5 bits per element; block cache disabled; a 500 GB 7200 RPM + disk, 32 GB RAM, 4 × 2.7 GHz cores. +- **Lookup latency**: "Monkey reduces lookup latency by an increasing margin as + the data volume grows (**50%–80%** for the data sizes we experimented with)" + — Abstract, and §5 "Monkey dominates LevelDB by up to 80%". +- **IOs per lookup**: Figure 11(A) is annotated **≈1 I/O per lookup for LevelDB + against ≈0.2 for Monkey** at the largest data size — a 5× reduction, and the + clearest single number in the paper. +- **Memory**: Figure 11(C) — Monkey matches LevelDB's lookup performance with up + to **≈60% smaller** filter memory. + +Note what does *not* appear in that list: this repo has no measured lane for +topic 4 (its benches measure only your code), so every number above is the +paper's, on the paper's 2017 spinning disk. The rule to carry away is the +marginal one — **equal IO saved per bit spent, everywhere ⇒ FPR proportional to +level size** — and it is free: no new data structure, just arithmetic at +filter-build time. + +Two boundaries. First, the argument assumes **point** lookups: filters do not +help range scans at all, since a scan must consult every run regardless, so a +scan-heavy workload gets nothing from any of this. Second, Monkey holds the +merge policy fixed while it tunes filters; §4.2-4.3 and Appendix D go on to +co-tune `T` and the merge policy against the same cost model, but the merging +half is done properly by Dostoevsky, the next chapter — which is also where the +sum-of-FPRs objective gets its second term. ## How to read the paper (with the concepts in hand) -1. §1–2 — the LSM cost model (worth it alone: R/W/M costs as formulas in T, - L — Steps 1–2 with full generality). Map each symbol to your mini-LSM's - knobs. -2. §4 — the allocation argument (Steps 3–5). Follow the Lagrange-multiplier - sketch once; then re-derive the "FPR ∝ level size" conclusion informally - yourself. -3. §5 — merging co-tuning (T as a continuum from leveled to tiered). Skim — - Dostoevsky does this better. -4. §6 evaluation — look for the ~2× lookup improvement at equal memory - (Step 6's number). +Budget about 2 h. The section numbers below are the paper's own. + +1. **§2 Background** — Equation 2, the FPR↔bits relation (Step 3), and the + statement that everyone uses uniform bits per entry. Ten minutes. +2. **§3 LSM-tree design space** — the R/W/M cost model in terms of `T` and `L` + (Step 1 with full generality). Map each symbol to your mini-LSM's knobs. +3. **§4.1 Minimizing Lookup Cost** — the heart: Equation 3 (Step 2), Equation 4 + (the memory model), Equations 5-6 (Step 5's optimum). Read Figure 6 first as + a glossary. The Lagrange derivation itself is **Appendix B** — follow it once + if you want it, but re-deriving "FPR ∝ level size" informally from Steps 3-4 + is the version worth being able to reproduce. +4. **§4.3 Scalability and Tunability** — Table 1, where the `O(L)` factor is + shaved (Step 6's asymptotic claim). +5. **§5 Experimental Analysis** — Figure 11(A) for the ≈1 → ≈0.2 IOs per lookup, + Figure 11(C) for the ≈60% memory saving. Check the setup paragraph before + quoting anything: 5 bits per element, size ratio 2, block cache *off*. +6. Skim **§4.2 and Appendix D** (co-tuning `T` and the merge policy) — Dostoevsky + does that half better. §6 is Related Work and §7 the conclusion; neither + carries numbers. ## Questions to answer in notes.md 1. In your mini-LSM (3 levels, T=10, 10M keys), compute uniform-vs-Monkey - expected false probes per zero-result get at 10 bits/key average. Then - *measure* zero-result gets both ways (the experiment supports per-level - bits-per-key for exactly this). + expected false probes per zero-result get at 10 bits/key average — the + 4-level version is worked in Step 5, so redo it at L = 3 and check that the + advantage *shrinks*. Then *measure* zero-result gets both ways (the + experiment supports per-level bits-per-key for exactly this). 2. Monkey assumes point lookups dominate. What breaks for range scans? (Filters don't help ranges at all — prefix blooms exist for a subset.) 3. FalkorDB angle: an attribute store doing existence checks before edge @@ -141,14 +329,145 @@ itself with the same optimization mindset — skim that, because Dostoevsky ## Done when -You can state the allocation rule ("equal *marginal* IO saved per bit ⇒ FPR -proportional to level size") and back it with the measured table from your -mini-LSM. +Answer each before unfolding it. + +- [ ] You can write down `R` for leveling and for tiering, naming every symbol. + +
Answer + + `R` is the expected number of wasted IOs for a *zero-result* point lookup — + one where the key exists nowhere, so every filter consulted that says "maybe" + costs one useless disk read. + + Leveling: `R = Σ(i=1..L) p_i`, where `p_i` is the false-positive rate of level + *i*'s filter and `L` is the number of levels on disk. Tiering: + `R = (T−1) · Σ(i=1..L) p_i`, because tiering keeps up to `T−1` runs per level + (the T-th arrival triggers a merge), all the same size and therefore all with + the same FPR. Monkey §4.1, Equation 3 — whose leveling branch is misprinted as + the tiering one; the prose above it gives the correct reading. + + The step that makes this a *sum* rather than something level-weighted is that + a false positive costs one IO no matter which level it happened at: fence + pointers take the probe straight to the single qualifying page (§4.1). If + probing a big run cost more than probing a small one, the whole optimum would + move. + +
+ +- [ ] You can convert between bits per key and FPR in both directions, and say what 1.4427 bits buys. + +
Answer + + `FPR = e^(−(bits/entries)·ln(2)²)` (Monkey §2, Equation 2), and inverted for + sizing, `bits = −entries · ln(FPR) / ln(2)²` (§4.1). With + `ln(2)² = 0.480453`: 8 bits/key → 2.143%, 10 → 0.8193%, 12 → 0.3132%, + 14 → 0.1197%, 16 → 0.04578%. + + Because the relation is exponential, the cost of *halving* the FPR is constant: + `ln2 / ln²2 = 1/ln2 = 1.4427` bits per key, from any starting point. That + constancy is what makes the optimization clean — the marginal price of an FPR + halving at level *i* is 1.4427 × (entries at level *i*) bits, so it differs + between levels only through the entry count, by exactly a factor of `T` per + level. + + The same formula is what fjall's lsm-tree implements in `calculate_m` + (`src/table/filter/standard_bloom/builder.rs:129-150`), so this is a + description of shipped filters, not an idealisation. + +
+ +- [ ] You can state the allocation rule and produce the bits-per-key ladder for a concrete tree. + +
Answer + + Rule: **equal marginal IO saved per bit spent, everywhere ⇒ each level's FPR + proportional to its number of entries** — "the optimal FPR at Level i is T + times higher than the optimal FPR at Level i−1" (§4.1, from Equations 5-6, + derived by Lagrange multipliers in Appendix B). + + Since FPR × T per level down, bits per key fall by a constant + `ln(T)/ln(2)² = 4.793` per level at T = 10. On the worked tree — N = 10 M, + T = 10, L = 4, entries 9 K / 90 K / 900 K / 9 M, budget 99,990,000 bits (10 + bits/key average): + + | level | uniform | Monkey | + |---|---|---| + | 1 | 10 bits, 0.8193% | 23.85 bits, 0.001057% | + | 2 | 10 bits, 0.8193% | 19.05 bits, 0.01057% | + | 3 | 10 bits, 0.8193% | 14.26 bits, 0.1057% | + | 4 | 10 bits, 0.8193% | 9.47 bits, 1.057% | + + `R` falls from 0.0328 to 0.0117 — 2.79× fewer wasted IOs at identical memory — + or, holding `R` fixed at 0.0328 instead, the Monkey allocation needs 78.6 Mbit + against 99.99 Mbit, 21% less DRAM. Bottom line: giving up half a bit per key at + the level holding 90% of the data funds an 8×-to-775× FPR improvement + everywhere else. + +
+ +- [ ] You can say what the paper actually measured, and on what hardware, without rounding it into folklore. + +
Answer + + §5: Monkey built on LevelDB and differing *only* in filter allocation; 1 GB of + 1 KB entries; 16 K uniformly random zero-result point lookups; size ratio 2; + 1 MB buffer; 5 bits per element of filter memory; LevelDB's block cache + **disabled**; a 500 GB 7200 RPM disk with 32 GB RAM and 4 × 2.7 GHz cores. + + Results: lookup latency 50%–80% lower, with the margin *growing* as data volume + grows (Abstract; §5 "up to 80%"); Figure 11(A) annotated ≈1 IO per lookup for + LevelDB against ≈0.2 for Monkey; Figure 11(C), equal performance at up to ≈60% + less filter memory. + + The caveats that matter when quoting it: a 2017 spinning disk makes a saved IO + worth ~10 ms, so the latency ratio is not transferable to NVMe; the block cache + was off; and the default size ratio was 2, not the 10 this chapter's worked + example uses. This repo adds no measurement of its own here — topic 4 has no + `verify.sh` lane, because its benches measure only your code. + +
+ +- [ ] You can name the two things Monkey does *not* fix. + +
Answer + + First, **range scans**. Every result here is about point lookups, and about + zero-result ones in particular; a range scan must open every run whose key + range overlaps the range regardless of what any filter says, so a scan-heavy + workload sees none of this benefit. Prefix bloom filters help a narrow subset + (scans over a fixed key prefix) and nothing else. + + Second, **the merge policy**. Monkey holds the merge policy fixed while it + optimizes filters; the `R = Σ p_i` objective says nothing about update cost, + and the same tree tuned for lookups may be paying 20× write amplification to + get there (topic 4's `notes.md` works that out for leveled at T = 10, L = 4). + §4.2-4.3 and Appendix D extend the cost model to co-tune `T` and the policy, + but the properly worked answer — different policies at different levels — is + Dostoevsky's, in the next chapter. + +
## References **Papers** -- Dayan, Athanassoulis, Idreos — "Monkey: Optimal Navigable Key-Value - Store" (SIGMOD 2017) — §1–2 for the LSM cost model, §4 for the - allocation argument; skim §5 (Dostoevsky does the merging co-tuning - better) and §6 for the ~2× lookup improvement at equal memory +- Dayan, Athanassoulis, Idreos — *Monkey: Optimal Navigable Key-Value Store*, + SIGMOD 2017. Read §2 (Equation 2), §3 (the design space), §4.1 (Equations 3-6, + the whole argument), §4.3 Table 1 (the `O(L)` saving), §5 Figure 11 (the + numbers). Appendix B is the Lagrange derivation; Appendix C is the iterative + allocator for variable entry sizes; Appendix D co-tunes the size ratio. + +| Claim in this chapter | Source | +|---|---| +| `R` = sum of per-level FPRs (leveling); ×(T−1) for tiering | §4.1, Equation 3 + preceding prose | +| One IO per false positive regardless of level, via fence pointers | §4.1 | +| `FPR = e^(−(bits/entries)·ln²2)` | §2, Equation 2 | +| Level *i* holds `N/T^(L−i) · (T−1)/T` entries | §4.1 | +| Filter memory model `M_filters` in terms of the `p_i` | §4.1, Equation 4 | +| Optimal FPR is T× higher per level down ⇒ ∝ level size | §4.1, Equations 5-6; derived in Appendix B | +| Deepest filters disappear as their optimal FPR → 1 | §4.1, the `L_filtered` term | +| Lookup cost loses its `O(L)` factor | §4.3, Table 1 | +| 50%–80% lower lookup latency; ≈1 → ≈0.2 IOs; ≈60% less memory | Abstract; §5 and Figure 11(A), (C) | + +**Code** +- `lsm-tree src/table/filter/standard_bloom/builder.rs:129-150` at `8526dd3` — + Equation 2, shipped, for comparison with the paper's algebra. diff --git a/topics/04-lsm-deep-dive/reading-rocksdb-compaction.md b/topics/04-lsm-deep-dive/reading-rocksdb-compaction.md index c0d2e67..a96268a 100644 --- a/topics/04-lsm-deep-dive/reading-rocksdb-compaction.md +++ b/topics/04-lsm-deep-dive/reading-rocksdb-compaction.md @@ -7,6 +7,12 @@ MVCC for metadata. Before the guided skim, this chapter builds each addition as its own concept — what problem it solves and what it costs — then maps every one to its file and line. +Every anchor is **facebook/rocksdb at `7c80a5a`**, the commit this repo pins +(`tools/pinned-source.py ref rocksdb`); check any of them with +`tools/pinned-source.py show rocksdb -r `. The defaults quoted +throughout are the shipped ones, and each is cited where it is declared, because +almost every number below is a default someone has since re-tuned. + ## The problem in one sentence Compaction is a background job competing with foreground writes for the same @@ -19,6 +25,11 @@ pushes back. ### Step 1 — compaction debt: why compaction needs a scheduler +> **In:** the LSM shape from the lsm-tree chapter — levels, runs, a flush that +> keeps adding files at the top. +> **Out:** the quantity every later step manipulates: *debt*, measured two ways +> (L0 file count, and bytes over target), which Step 2 turns into a number. + Compaction debt is the gap between what has been written and what has been merged — concretely, bytes sitting in levels that exceed their target size, waiting to be pushed down. Writers add debt (every memtable flush is a new @@ -30,193 +41,816 @@ input is the current shape of the tree, its output is one job. Get the policy wrong — say, round-robin across levels — and a hot level's debt grows unboundedly while the picker dutifully polishes cold ones. +The shape the debt accumulates in, at the shipped defaults: + +``` + level0_file_num_compaction_trigger = 4 include/rocksdb/options.h:255 + max_bytes_for_level_base = 256 MB include/rocksdb/options.h:303 + max_bytes_for_level_multiplier = 10 include/rocksdb/advanced_options.h:671 + + L0 4 files, each ~one memtable, mutually overlapping ← debt in *files* + L1 256 MB target ← debt in *bytes* + L2 2.5 GB + L3 25 GB + L4 250 GB +``` + +Two units, because L0 is a different animal. RocksDB says so itself, in the +comment that opens the scoring function: + +```cpp +// db/version_set.cc — inside VersionStorageInfo::ComputeCompactionScore, 4002-4021 + 4002 if (level == 0) { + 4003 // We treat level-0 specially by bounding the number of files + 4004 // instead of number of bytes for two reasons: + 4005 // + 4006 // (1) With larger write-buffer sizes, it is nice not to do too + 4007 // many level-0 compactions. + 4008 // + 4009 // (2) The files in level-0 are merged on every read and + 4010 // therefore we wish to avoid too many files when the individual + 4011 // file size is small (perhaps because of a small write-buffer + 4012 // setting, or very high compression ratios, or lots of + 4013 // overwrites/deletions). + 4014 int num_sorted_runs = 0; + 4015 uint64_t total_size = 0; + 4016 for (auto* f : files_[level]) { + 4017 total_downcompact_bytes += static_cast(f->fd.GetFileSize()); + 4018 if (!f->being_compacted) { + 4019 total_size += f->compensated_file_size; + 4020 num_sorted_runs++; + 4021 } + 4022 } +``` + +Reason (2) on line 4009 is the whole reason this topic exists: an L0 file is a +sorted run all by itself, so it is one more place every point read must look. +Note line 4018 too — a file already claimed by a running compaction does not +count as debt, because someone is already paying it. + ### Step 2 — score-driven picking: highest debt first +> **In:** Step 1's two debt units, per level. +> **Out:** one `double` per level, sorted descending, and from it a single +> compaction job — the input Step 3 executes. + RocksDB reduces "which level hurts most" to one number per level, the **score** — how far the level is past its trigger, normalized so scores are comparable across levels: -- **L0**: `num_files / level0_file_num_compaction_trigger` — L0 is scored by - *file count*, because every L0 file is an overlapping run that every point - read must probe (topic README §1); 8 files with trigger 4 ⇒ score 2.0. -- **L1+**: `level_bytes / MaxBytesForLevel` — deeper levels are scored by - *bytes over target* (targets grow 10× per level: 256 MB, 2.5 GB, 25 GB…). +- **L0**: `num_sorted_runs / level0_file_num_compaction_trigger` + (`db/version_set.cc:4077-4078`), where `num_sorted_runs` is the count of L0 + files *not already being compacted*. +- **L1+**: `level_bytes_no_compacting / MaxBytesForLevel(level)` + (`:4136-4137`), where the numerator again excludes in-flight files + (`:4129-4134`) and the denominator is the level's target from Step 1. + +```cpp +// db/version_set.cc — the L1+ branch of ComputeCompactionScore, 4125-4137 + 4125 } else { // level > 0 + 4126 // Compute the ratio of current size to size limit. + 4127 uint64_t level_bytes_no_compacting = 0; + 4128 uint64_t level_total_bytes = 0; + 4129 for (auto f : files_[level]) { + 4130 level_total_bytes += f->fd.GetFileSize(); + 4131 if (!f->being_compacted) { + 4132 level_bytes_no_compacting += f->compensated_file_size; + 4133 } + 4134 } + 4135 if (!immutable_options.level_compaction_dynamic_level_bytes) { + 4136 score = static_cast(level_bytes_no_compacting) / + 4137 MaxBytesForLevel(level); +``` -Highest score ≥ 1.0 compacts first; below 1.0 means no debt, do nothing. -One production subtlety: bytes already being compacted by an in-flight job -are subtracted before scoring — double-booking a level wastes IO. The -scoring loop, reduced to its logic: +Work it on a concrete tree, at the Step 1 defaults: -```rust -// Compact the level with the highest score ≥ 1.0; the picker is a -// scheduler, so bytes already being compacted don't count twice. -fn pick_compaction_level(&self, v: &Version) -> Option { - (0..v.num_levels()) - .map(|lvl| { - let score = if lvl == 0 { - v.num_l0_files() as f64 / self.l0_file_trigger as f64 - } else { - (v.level_bytes(lvl) - v.bytes_being_compacted(lvl)) as f64 - / self.max_bytes_for_level(lvl) as f64 - }; - (lvl, score) - }) - .max_by(|a, b| a.1.total_cmp(&b.1)) - .filter(|&(_, score)| score >= 1.0) // below 1.0: no debt, do nothing - .map(|(lvl, _)| lvl) -} -``` - -Once a level is picked, the job's inputs are expanded to clean key -boundaries and the overlapping next-level files are pulled in — a merge must -consume *every* next-level file its key range touches, or it would create -overlap where the level promises disjointness. +``` + L0 8 files, none being compacted score = 8 / 4 = 2.00 + L1 512 MB, none being compacted score = 512 / 256 = 2.00 + L2 1.0 GB score = 1.0 / 2.5 = 0.40 + L3 30 GB score = 30 / 25 = 1.20 + + sorted descending: L0 (2.00), L1 (2.00), L3 (1.20), L2 (0.40) + → pick L0. Now start one L1 compaction covering 256 MB of L1: + L1 recomputed = (512 − 256) / 256 = 1.00, and it drops behind L3. +``` + +That last line is the "subtract in-flight bytes" rule doing its job: without it +the picker would keep choosing L1 and double-book the same files. Two wrinkles +the arithmetic above hides. First, `kScoreScale = 10.0` (`:3996`): under +`level_compaction_dynamic_level_bytes` any score above 1.0 is multiplied by ten +so that the comparison has room to express priorities *within* the "needs +compaction" band — the raw number you read in a log may be 20.0, not 2.0. +Second, the levels are sorted by a **bubble sort** (`:4173-4186`, with RocksDB's +own justification: "the number of entries are small"), and the picker relies on +that order: + +```cpp +// db/compaction/compaction_picker_level.cc — inside SetupInitialFiles, 210-235 and 255-259 + 210 for (int i = 0; i < compaction_picker_->NumberLevels() - 1; i++) { + 211 start_level_score_ = vstorage_->CompactionScore(i); + 212 start_level_ = vstorage_->CompactionScoreLevel(i); + 213 assert(i == 0 || start_level_score_ <= vstorage_->CompactionScore(i - 1)); + 214 if (start_level_score_ >= 1) { + // ... 215-220: skip LBase if an L0→LBase compaction is already pending ... + 221 output_level_ = + 222 (start_level_ == 0) ? vstorage_->base_level() : start_level_ + 1; + 223 bool picked_file_to_compact = PickFileToCompact(); + // ... 224-227: sync point, then "found the compaction!" ... + 228 if (start_level_ == 0) { + 229 // L0 score = `num L0 files` / `level0_file_num_compaction_trigger` + 230 compaction_reason_ = CompactionReason::kLevelL0FilesNum; + 231 } else { + 232 // L1+ score = `Level files size` / `MaxBytesForLevel` + 233 compaction_reason_ = CompactionReason::kLevelMaxLevelSize; + 234 } + 235 break; + // ... 236-254: nothing pickable — clear inputs, and for L0 try an intra-L0 merge ... + 255 } else { + 256 // Compaction scores are sorted in descending order, no further scores + 257 // will be >= 1. + 258 break; + 259 } +``` + +Line **214** is the threshold (`>= 1`, i.e. at or past target) and line **258** +is why the loop can stop early. The two comments on 229 and 232 are the score +formulas restated at the point of use — they are documentation, though; the +arithmetic itself lives in `version_set.cc` as quoted above. + +Once a level is picked, `PickCompaction` +(`db/compaction/compaction_picker_level.cc:531-558`) expands the job: other L0 +files if this is an L0 job (`:542`), then `SetupOtherInputsIfNeeded` (`:548`) +which pulls in *every* overlapping next-level file — a merge must consume all of +them, or it would create overlap where the level promises disjointness. The last +thing `GetCompaction` does is recompute all the scores (`:596`), because +registering this job just changed every `being_compacted` flag it touched. + +One escape hatch worth knowing: when L0 is over its trigger but an L0→LBase +compaction cannot start (one is already running), the picker falls back to +`PickIntraL0Compaction` (`:248-252`, defined at `:924`) — merging L0 files into +*fewer, bigger L0 files*. It moves no data downward and pays no debt, but it +reduces the run count, which is exactly what Step 4 is about to stall on. ### Step 3 — the merge job: mostly not a merge +> **In:** the job Step 2 picked — a set of input files and an output level. +> **Out:** new SST files (Step 5 builds them) and a `VersionEdit` (Step 8 +> commits it). + The core of a compaction job is the k-way merge you already read in lsm-tree (pop the smallest key across k sorted inputs, write output -blocks). What RocksDB wraps around it is the production payload: -**compaction filters** (user callbacks that can drop or rewrite each -key-value pair mid-merge — TTL expiry lives here), **snapshot lists** (a -key's old version can only be dropped if no live snapshot might still read -it — MVCC reaching into compaction), and **sub-compaction splitting** (one -job's key range carved into disjoint sub-ranges so multiple threads merge in -parallel). Skim the function for shape; the lesson is how much of an -industrial compaction is *not* the merge. +blocks). What RocksDB wraps around it is the production payload, and the +proportions are the lesson. + +**Sub-compaction splitting** comes first: one job's key range is carved into +disjoint sub-ranges and each gets a thread. + +```cpp +// db/compaction/compaction_job.cc — inside CompactionJob::RunSubcompactions, 727-741 + 727 const size_t num_threads = compact_->sub_compact_states.size(); + 728 assert(num_threads > 0); + 729 compact_->compaction->GetOrInitInputTableProperties(); + 730 + 731 // Launch a thread for each of subcompactions 1...num_threads-1 + 732 std::vector thread_pool; + 733 thread_pool.reserve(num_threads - 1); + 734 for (size_t i = 1; i < compact_->sub_compact_states.size(); i++) { + 735 thread_pool.emplace_back(&CompactionJob::ProcessKeyValueCompaction, this, + 736 &compact_->sub_compact_states[i]); + 737 } + 738 + 739 // Always schedule the first subcompaction (whether or not there are also + 740 // others) in the current thread to be efficient with resources + 741 ProcessKeyValueCompaction(compact_->sub_compact_states.data()); +``` + +`ProcessKeyValueCompaction` (`:1904`) is therefore the per-thread body, and the +first thing it does is not merging: it resolves the **compaction filter** — a +user callback consulted for every key-value pair mid-merge, where TTL expiry +lives — at `:1920-1924`, then builds a `CompactionIterator` (`:1621-1632`) +handing it the **snapshot list** (`job_context_->snapshot_seqs`, `:1623`). + +The snapshot list is MVCC reaching into compaction: a key's old version can be +dropped only if no live snapshot might still read it. The rule for *tombstones* +is where RocksDB is strictly cleverer than lsm-tree, and it is worth reading +side by side with `evict_tombstones(is_last_level)`: + +```cpp +// db/compaction/compaction_iterator.cc — the early tombstone drop, 1152-1159 and 1164-1170 + 1152 } else if (compaction_ != nullptr && + 1153 (ikey_.type == kTypeDeletion || + 1154 (ikey_.type == kTypeDeletionWithTimestamp && + 1155 cmp_with_history_ts_low_ < 0)) && + 1156 !compaction_->allow_ingest_behind() && + 1157 DefinitelyInSnapshot(ikey_.sequence, earliest_snapshot_) && + 1158 compaction_->KeyNotExistsBeyondOutputLevel(ikey_.user_key, + 1159 &level_ptrs_)) { + // ... 1160-1163: a TODO about this being the only use of compaction_ ... + 1164 // For this user key: + 1165 // (1) there is no data in higher levels + 1166 // (2) data in lower levels will have larger sequence numbers + 1167 // (3) data in layers that are being compacted here and have + 1168 // smaller sequence numbers will be dropped in the next + 1169 // few iterations of this loop (by rule (A) above). + 1170 // Therefore this deletion marker is obsolete and can be dropped. +``` + +Line **1158** is the difference. lsm-tree drops a tombstone only when the output +*is* the bottom level, because that is the cheap sufficient condition. RocksDB +asks the sharper question — `KeyNotExistsBeyondOutputLevel`, "does this user key +appear in any level below the output?" — and can therefore retire a tombstone at +L2 if nothing below L2 holds that key, freeing the space several levels earlier. +The bottommost case is still handled separately at `:1188-1191`. Same +correctness argument (never resurrect data), a tighter test, and one more +`level_ptrs_` cursor to maintain. + +Skim `ProcessKeyValueCompaction` for shape rather than detail; the lesson is how +much of an industrial compaction is *not* the merge. ### Step 4 — write stalls: back-pressure as a feature +> **In:** the debt Step 1 measured and Step 2 is failing to pay down fast +> enough. +> **Out:** a `WriteStallCondition` — `kNormal`, `kDelayed` or `kStopped` — +> applied to foreground writers. + A write stall is the engine deliberately slowing or stopping foreground writes because compaction has fallen behind. It sounds like a bug; it is load-shedding. Without it, debt is unbounded: L0 grows to hundreds of files and *every read* pays for it, indefinitely. The stall converts an unbounded -read-amplification problem into a bounded write-latency problem. RocksDB's -triggers, in escalating order: +read-amplification problem into a bounded write-latency problem. + +The whole policy is one if-else chain, and reading it in order matters, because +the order *is* the priority — the first matching condition wins: + +```cpp +// db/column_family.cc — ColumnFamilyData::GetWriteStallConditionAndCause, 1016-1045 + 1016 if (num_unflushed_memtables >= mutable_cf_options.max_write_buffer_number) { + 1017 return {WriteStallCondition::kStopped, WriteStallCause::kMemtableLimit}; + 1018 } else if (!mutable_cf_options.disable_auto_compactions && + 1019 num_l0_files >= mutable_cf_options.level0_stop_writes_trigger) { + 1020 return {WriteStallCondition::kStopped, WriteStallCause::kL0FileCountLimit}; + 1021 } else if (!mutable_cf_options.disable_auto_compactions && + 1022 mutable_cf_options.hard_pending_compaction_bytes_limit > 0 && + 1023 num_compaction_needed_bytes >= + 1024 mutable_cf_options.hard_pending_compaction_bytes_limit) { + 1025 return {WriteStallCondition::kStopped, + 1026 WriteStallCause::kPendingCompactionBytes}; + // ... 1027-1032: memtable number − 1 → kDelayed ... + 1033 } else if (!mutable_cf_options.disable_auto_compactions && + 1034 mutable_cf_options.level0_slowdown_writes_trigger >= 0 && + 1035 num_l0_files >= + 1036 mutable_cf_options.level0_slowdown_writes_trigger) { + 1037 return {WriteStallCondition::kDelayed, WriteStallCause::kL0FileCountLimit}; + 1038 } else if (!mutable_cf_options.disable_auto_compactions && + 1039 mutable_cf_options.soft_pending_compaction_bytes_limit > 0 && + 1040 num_compaction_needed_bytes >= + 1041 mutable_cf_options.soft_pending_compaction_bytes_limit) { + 1042 return {WriteStallCondition::kDelayed, + 1043 WriteStallCause::kPendingCompactionBytes}; + 1044 } + 1045 return {WriteStallCondition::kNormal, WriteStallCause::kNone}; +``` -- L0 files ≥ `level0_slowdown_writes_trigger` (default 20) → **delayed** - (writes trickled at a reduced rate) -- pending compaction bytes ≥ soft limit → **delayed** -- L0 files ≥ `level0_stop_writes_trigger` (default 36) → **stop** -- pending compaction bytes ≥ hard limit → **stop** +Six conditions, not four, and three causes × two severities: + +| condition | cause | threshold | default | +|---|---|---|---| +| stop | memtable limit | unflushed memtables ≥ `max_write_buffer_number` | 2 (`advanced_options.h:271`) | +| stop | L0 file count | ≥ `level0_stop_writes_trigger` | 36 (`:554`) | +| stop | pending bytes | ≥ `hard_pending_compaction_bytes_limit` | 256 GB (`:717`) | +| delay | memtable limit | ≥ `max_write_buffer_number − 1`, only if that is > 3 | — (`column_family.cc:1027-1031`) | +| delay | L0 file count | ≥ `level0_slowdown_writes_trigger` | 20 (`advanced_options.h:547`) | +| delay | pending bytes | ≥ `soft_pending_compaction_bytes_limit` | 64 GB (`:709`) | + +Two things to take from the table. The stalls are tested **stop-first**: the +chain checks the severe conditions before the mild ones, so a database at 40 L0 +files reports `kStopped`, never `kDelayed`. And the L0 numbers frame the whole +topic — compaction is triggered at 4 files, writers are slowed at 20 and stopped +at 36, so the design intent is that a read never probes more than a few dozen L0 +runs. Put the lsm-tree chapter's filter arithmetic on that: at 0.844% false +positives per run, 4 L0 runs plus 6 levels cost 0.084 wasted block reads per +absent key, while 36 L0 runs plus 6 levels cost 0.35 — four times the wasted IO, +which is the badness the stop trigger exists to bound. + +Compare fjall's version of the same valve, which is the entire file: + +```rust +// fjall src/keyspace/write_delay.rs — the whole valve, 5-16 (fjall-rs/fjall@80cf6bc) + 5 const STEP_SIZE: usize = 10_000; + 6 const THRESHOLD: usize = 20; + 7 + 8 pub fn perform_write_stall(l0_runs: usize) { + 9 if let THRESHOLD..30 = l0_runs { + 10 let d = l0_runs - THRESHOLD; + 11 + 12 for _ in 0..(d * STEP_SIZE) { + 13 std::hint::black_box(()); + 14 } + 15 } + 16 } +``` -Note the unit of debt: *bytes not yet merged*, not files — a direct measure -of outstanding work. Compare fjall's version of the same valve: a spin-loop -delay when L0 reaches 20–30 runs (fjall `src/keyspace/write_delay.rs:8–16`) -— same idea, 100× simpler. Stalls are the honest choice. +Same idea — slow the writer in proportion to how far past 20 runs L0 is — and +100× simpler, but read line 9 carefully: the range pattern `THRESHOLD..30` is +*exclusive*, so a keyspace at 30 or more L0 runs gets **no delay at all**. fjall +has the delay valve and not the stop valve; RocksDB's `kStopped` tier is the +part that makes the bound actually a bound. Stalls are the honest choice, and +the hard stop is the half people leave out. ### Step 5 — building the SST: the same block tricks, plus shortened separators -RocksDB's table builder is lsm-tree's Step 2–3 with the serial numbers still -visible: prefix truncation with restart interval 16 (the same constant — -not convergent evolution; LevelDB is the shared ancestor), blocks flushed at -~4 KB, and an index entry recorded per block. The extra trick: the index -doesn't store each block's last key verbatim — `FindShortestSeparator` -shortens it to the smallest string that still separates this block from the -next (between `"userA...zzz"` and `"userB..."`, the separator `"userB"` -suffices). Shorter separators ⇒ smaller index ⇒ more index in cache. This is +> **In:** the merged key-value stream from Step 3. +> **Out:** a block-based SST — data blocks, an index of shortened separators +> (Step 6 searches it), a filter (Step 7 builds it). + +RocksDB's table builder is lsm-tree's Step 2-3 with the serial numbers still +visible. The constants are identical, and not by convergent evolution — LevelDB +is the shared ancestor: + +``` + block_size = 4 * 1024 include/rocksdb/table.h:400 + block_restart_interval = 16 include/rocksdb/table.h:413 + index_block_restart_interval = 1 include/rocksdb/table.h:416 + metadata_block_size = 4096 include/rocksdb/table.h:423 +``` + +Restart interval and delta encoding are wired into the data-block builder at +`table/block_based/block_based_table_builder.cc:1096-1097`, and the flush +decision is delegated to a policy object at `:1126-1128`. + +The extra trick is in the index. RocksDB does not store each block's last key +verbatim; it stores the shortest string that still *separates* one block from +the next, and it explains itself at the call site: + +```cpp +// table/block_based/block_based_table_builder.cc — inside WriteBlock's caller, 1901-1912 + 1901 if (LIKELY(ok())) { + 1902 // We do not emit the index entry for a block until we have seen the + 1903 // first key for the next data block. This allows us to use shorter + 1904 // keys in the index block. For example, consider a block boundary + 1905 // between the keys "the quick brown fox" and "the who". We can use + 1906 // "the r" as the key for the index block entry since it is >= all + 1907 // entries in the first block and < all entries in subsequent + 1908 // blocks. + 1909 r->index_builder->AddIndexEntry( + 1910 last_key_in_current_block, first_key_in_next_block, r->pending_handle, + 1911 &r->index_separator_scratch, skip_delta_encoding); + 1912 } +``` + +Work RocksDB's own example through the algorithm in `util/comparator.cc:42-101`, +which is four lines of real logic: + +``` + start = "the quick brown fox" (19 bytes) + limit = "the who" + + 1. common prefix scan (:47-50) → diff_index = 4 ('q' vs 'w') + 2. start_byte = 'q' = 0x71, limit_byte = 'w' = 0x77 (:55-56) + 3. start_byte < limit_byte, and 0x71 + 1 = 0x72 < 0x77 (:57, :64) + 4. so: (*start)[4]++ → 'r'; resize to diff_index + 1 = 5 (:65-66) + + separator = "the r" (5 bytes) → 19 − 5 = 14 bytes saved, 74% of the entry +``` + +The guard on line 64 is the case people get wrong: if incrementing the byte +would reach `limit` exactly, the code cannot use it and walks forward looking +for the first non-`0xFF` byte instead (`:67-80`). And `FindShortestSeparator` +operates on *user* keys, so the index builder has to tack a sequence number back +on afterwards to keep the internal-key ordering valid +(`table/block_based/index_builder.cc:78-101`, the fixup at `:89-97`). + +Shorter separators ⇒ smaller index ⇒ more index resident in cache. That is SQLite's interior-page separator idea rediscovered — the truncation topic 3 -experimented with. +experimented with — and it is what makes Step 6's problem tractable at all. ### Step 6 — the read path at scale: partitioning the index -A point read runs filter → index → data block, same as lsm-tree, with one -scale-driven change. On a huge SST (multi-GB), the index block itself -becomes megabytes — too big to pin in cache whole. RocksDB's fix is the -**partitioned index**: cut the index into chunks and build an index *over -the index* — a two-level B-tree, with the small top level pinned in cache -and partitions loaded on demand. The academic alternative (fractional -cascading, threading search hints between levels) never shipped: plain -binary search per level won in practice. Data blocks themselves are probed -through the **block cache** (a shared in-memory cache of decompressed -blocks) before touching disk. +> **In:** the SST Step 5 built, and one key. +> **Out:** at most one data-block read, having consulted a filter and a +> possibly two-level index. + +A point read runs filter → index → data block, same as lsm-tree: + +```cpp +// table/block_based/block_based_table_reader.cc — inside BlockBasedTable::Get, 3039-3053 and 3071 + 3039 const bool may_match = + 3040 FullFilterKeyMayMatch(filter, key, prefix_extractor, get_context, + 3041 &lookup_context, read_options); + 3042 TEST_SYNC_POINT("BlockBasedTable::Get:AfterFilterMatch"); + 3043 if (may_match) { + 3044 IndexBlockIter iiter_on_stack; + // ... 3045-3050: disable BlockPrefixIndex if the prefix extractor changed ... + 3051 auto iiter = + 3052 NewIndexIterator(read_options, need_upper_bound_check, &iiter_on_stack, + 3053 get_context, &lookup_context); + // ... 3054-3070: iterator ownership, timestamp size, blob scratch ... + 3071 for (iiter->Seek(key); iiter->Valid() && !done; iiter->Next()) { +``` + +Line **3043** is the same gate as lsm-tree's `return Ok(None)`: no index +iterator is even constructed for a filtered-out key. The data block itself is +fetched through `NewDataBlockIterator` (`:3092-3096`), which goes to the **block +cache** — a shared in-memory cache of uncompressed blocks — before touching disk +(`GetDataBlockFromCache`, `:2010`, called at `:2345`). + +The scale-driven change is the index. Size it from Step 5's numbers: a 256 MB +SST at 4 KB blocks has ~65,500 data blocks, so with a 20-byte shortened +separator and a handle per entry the index block is on the order of 2 MB — one +allocation, one cache entry, all-or-nothing. RocksDB's fix is the **partitioned +index**: cut that index into ~4 KB partitions (`metadata_block_size`, +`table.h:423`) and build an index *over the index*. + +```cpp +// table/block_based/partitioned_index_reader.h — the two-level reader, 14-15 and 27-28 + 14 // Index that allows binary search lookup in a two-level index structure. + 15 class PartitionIndexReader : public BlockBasedTable::IndexReaderCommon { + ... + 27 // return a two-level iterator: first level is on the partition index + 28 InternalIteratorBase* NewIterator( +``` + +Only the small top level is pinned; partitions are loaded on demand through the +same block cache (`block_based_table_reader.cc:1776-1780` decides the pinning +for `kTwoLevelIndexSearch`). A point read now costs one top-level binary search +plus one partition lookup, in exchange for never having to hold 2 MB of index +resident to answer a single key. + +The academic alternative — fractional cascading, threading search hints from one +level's index into the next — never shipped: plain binary search per level won +in practice, because it composes with a cache and cascading does not. ### Step 7 — filters, industrialized: cache-local bloom and ribbon +> **In:** the keys of one SST as Step 5 writes them. +> **Out:** a filter block whose false-positive rate and build cost are now both +> tuning knobs — the gate Step 6 checks at line 3039. + Two upgrades over the textbook bloom filter, both bought with the topic 0 -price list in hand: - -- **FastLocalBloom**: a classic bloom's k probes hit k random cache lines — - k potential cache misses per lookup. RocksDB's variant confines all k - probes for a key to **one cache line**: one miss max. The price is paid in - statistics — a blocked bloom has slightly worse false-positive rate at - equal bits/key (keys crowd within their line). Sizing is in - `millibits_per_key` — fleet-scale tuning wants sub-bit granularity. -- **Ribbon filters**: a different construction (linear algebra over the key - hashes) that is ~30% smaller for the same false-positive rate but much - slower to *build*; if the equation system fails to solve ("banding - fails"), it retries with a new seed up to 256 times, then falls back to - bloom. A pure CPU-for-DRAM knob: spend build-time CPU during compaction, - save filter memory forever after. +price list in hand. + +**FastLocalBloom** (`table/block_based/filter_policy.cc:365-377`) fixes a memory +problem, not a math one. A classic bloom's k probes hit k random positions in a +multi-megabyte bit array — up to k cache misses per lookup, and the lsm-tree +chapter's double-hashing loop is exactly that shape. RocksDB confines all k +probes for a key to **one 64-byte cache line** — one miss, maximum. The price is +paid in statistics, and the implementation states it to three decimal places: + +``` +// util/bloom_impl.h:105-108, for 10 bits/key and num_probes = 6 + + theoretical best, cache-local, 512-bit bucket 0.9535% + this implementation 0.957% + LegacyLocalityBloomImpl 1.138% + 1024-bit buckets (some ARM cache lines) 0.951% +``` + +Set that against the *non*-local optimum for the same budget — 0.844% at k = 6, +the number the lsm-tree chapter computed for fjall's filter at the same 10 +bits/key — and the trade is exact: **0.844% → 0.957% false positives, about 13% +relatively worse, in exchange for 1 cache miss instead of up to 6.** Sizing is +in `millibits_per_key` (`filter_policy.cc:369`, asserted ≥ 1000 at `:376`), +because fleet-scale tuning wants sub-bit granularity. + +**Ribbon filters** (`Standard128RibbonBitsBuilder`, `:658`) attack the space +instead. The construction solves a linear system over the key hashes rather than +setting independent bits, and the header states the trade: + +``` +// include/rocksdb/filter_policy.h:169-173 + + "saves about 30% space compared to Bloom filters, with similar query times + but roughly 3-4x CPU time and 3x temporary space usage during construction. + For example, if you pass in 10 for bloom_equivalent_bits_per_key, you'll get + the same 0.95% FP rate as Bloom filter but only using about 7 bits per key." +``` + +10 bits/key → 7 bits/key at an unchanged 0.95% false-positive rate: a pure +CPU-for-DRAM trade, spend build-time CPU during compaction and save filter +memory forever after. Two production details make it usable. Solving can fail, +so the builder re-seeds — 256 times, then gives up and builds a bloom instead: + +```cpp +// table/block_based/filter_policy.cc — inside Standard128RibbonBitsBuilder::Finish, 751-762 + 751 bool success = banding.ResetAndFindSeedToSolve( + 752 num_slots, hash_entries_info_.entries.begin(), + 753 hash_entries_info_.entries.end(), + 754 /*starting seed*/ entropy & 255, /*seed mask*/ 255); + 755 if (!success) { + 756 ROCKS_LOG_WARN( + 757 info_log_, "Too many re-seeds (256) for Ribbon filter, %llu / %llu", + 758 static_cast(hash_entries_info_.entries.size()), + 759 static_cast(num_slots)); + 760 SwapEntriesWith(&bloom_fallback_); + 761 assert(hash_entries_info_.entries.empty()); + 762 return bloom_fallback_.Finish(buf, status); +``` + +And the choice is made **per level**, not per database: `bloom_before_level = 0` +by default, meaning bloom for flushes (L0) and ribbon everywhere below +(`filter_policy.h:184-185`, with the rationale at `:175-181` — bloom's speed +where the data is hot and short-lived, ribbon's density where it is not). That +is the same "spend the budget unevenly across levels" instinct Monkey formalises +in the next chapter, applied to filter *construction* rather than filter *size*. ### Step 8 — the MANIFEST: MVCC for metadata -Compaction's final act is swapping files: outputs replace inputs. RocksDB -commits this by appending a **VersionEdit** (a delta record: "add these -files, delete those") to the **MANIFEST** — an append-only log of metadata -changes — then pointing the CURRENT file at it. In-memory, each reader -holds a refcounted **Version** (an immutable snapshot of the file layout); -a compaction publishes a new Version, and readers mid-iteration keep using -their old one until they drop the reference. That is multi-version -concurrency control applied to *metadata*: writers never disturb readers, -and crash recovery replays the edit log. lsm-tree rewrites its whole version -file instead — same atomicity, different scale point: a delta log wins when -you have 100K files, a rewrite wins on simplicity when you have 100. +> **In:** the output files Step 3 produced and the input files it consumed. +> **Out:** a durably committed new `Version`, with readers still safely using +> the old one. + +Compaction's final act is swapping files: outputs replace inputs. The vocabulary +is defined in one comment: + +```cpp +// db/version_edit.h — the definition, 701-708 + 701 // The state of a DB at any given time is referred to as a Version. + 702 // Any modification to the Version is considered a Version Edit. A Version is + 703 // constructed by joining a sequence of Version Edits. Version Edits are written + 704 // to the MANIFEST file. + 705 class VersionEdit { + 706 public: + 707 // Retrieve the table files added as well as their associated levels. + 708 using NewFiles = std::vector>; +``` + +A `VersionEdit` really is just "add these files, delete those": the two tags are +`kDeletedFile = 6` and `kNewFile = 7` in the serialization enum +(`db/version_edit.h:37-47`, with `kNewFile4 = 103` as the current format, +`:52`). `VersionSet::LogAndApply` (`db/version_set.cc:6778`) is the commit +entry point, and the durable part is two lines inside `ProcessManifestWrites`: + +```cpp +// db/version_set.cc — inside VersionSet::ProcessManifestWrites, 6500 and 6508-6513, 6527-6530 + 6500 io_s = raw_desc_log_ptr->AddRecord(write_options, record); + ... + 6508 if (s.ok()) { + 6509 io_s = + 6510 SyncManifest(db_options_, write_options, raw_desc_log_ptr->file()); + 6511 manifest_io_status = io_s; + // ... 6512-6513: sync point callback ... + ... + 6527 if (s.ok() && new_descriptor_log) { + 6528 io_s = SetCurrentFile( + 6529 write_options, fs_.get(), dbname_, pending_manifest_file_number_, + 6530 file_options_.temperature, dir_contains_current_file); +``` + +Read the condition on **6527**: CURRENT is rewritten only when a *new* MANIFEST +file was started. The steady-state commit is one appended record (6500) plus one +fsync (6509) — CURRENT keeps pointing at the same log. That is the delta design +paying off: metadata cost is proportional to the change, not to the database. + +In memory, the swap is refcounting: + +```cpp +// db/version_set.cc — inside VersionSet::AppendVersion, 6093-6108 + 6093 // Make "v" current + 6094 assert(v->refs_ == 0); + 6095 Version* current = column_family_data->current(); + 6096 assert(v != current); + 6097 if (current != nullptr) { + 6098 assert(current->refs_ > 0); + 6099 current->Unref(); + 6100 } + 6101 column_family_data->SetCurrent(v); + 6102 v->Ref(); + 6103 + 6104 // Append to linked list + 6105 v->prev_ = column_family_data->dummy_versions()->prev_; + 6106 v->next_ = column_family_data->dummy_versions(); + 6107 v->prev_->next_ = v; + 6108 v->next_->prev_ = v; +``` + +The old version is *un*referenced, not deleted — `Version::Unref` frees only at +zero (`:4943-4951`) — and every live version stays on the doubly-linked list at +6104-6108. A reader mid-iteration holds a reference and keeps reading the file +layout it started with, while compaction publishes a new one beside it. That is +multi-version concurrency control applied to *metadata*: writers never disturb +readers, and crash recovery replays the edit log. + +lsm-tree writes a whole new version file instead (`persist.rs:16-17`) — same +atomicity, different scale point: a delta log wins when you have 100K files, a +snapshot wins on simplicity when you have 100. ## Where each step lives in the code -- **Steps 1–2 — `db/compaction/compaction_picker_level.cc`**: score - formulas in comments :229–233 (`L0: num_files / - level0_file_num_compaction_trigger`; `L1+: level_bytes / - MaxBytesForLevel`); `LevelCompactionBuilder::PickCompaction` :531 — setup - inputs, expand to clean key boundaries, grab the overlapping next-level - files; :596 — score recomputed accounting for in-flight compactions. -- **Step 3 — `db/compaction/compaction_job.cc:1904`**: - `ProcessKeyValueCompaction` — the k-way merge plus compaction filters, - snapshot lists, sub-compaction splitting. -- **Step 4 — `db/column_family.cc:1019–1043`**: - `GetWriteStallConditionAndCause` — the four triggers in Step 4's order. -- **Step 5 — `table/block_based/block_based_table_builder.cc`**: restart - interval + delta encoding :1096–1097 (default 16); block flush policy - :1127 (~4 KB); index entry = last key of each block, written on flush - :1908–1912, shortened via `FindShortestSeparator`. -- **Step 6 — `table/block_based/block_based_table_reader.cc`**: `Get` :3010 - — whole-table filter check first :3040, then index iterator :3044–3053, - then data block :3071–3096; block cache probe in `GetDataBlockFromCache` - :2345. Partitioned index :1778 + `partitioned_index_reader.h:15`. -- **Step 7 — `table/block_based/filter_policy.cc`**: - `FastLocalBloomBitsBuilder` :365–376 (`millibits_per_key`, one cache line - per key — contrast lsm-tree's double hashing across the whole bit array); - ribbon :658–686 (falls back to bloom after 256 seed attempts). -- **Step 8 — `db/version_set.cc`**: `LogAndApply` :6778 — append a - `VersionEdit` (version_edit.h:37–77, 705–744) to the MANIFEST log, point - CURRENT at it; readers keep iterating their old refcounted Version. +Every line number is `7c80a5a`. + +- **Steps 1-2 — scoring, `db/version_set.cc`**: `ComputeCompactionScore` at + `:3983`; `kScoreScale = 10.0` at `:3996`; the L0 justification and + `num_sorted_runs` at `:4002-4021`; the L0 formula at `:4077-4078`; the L1+ + formula and its in-flight exclusion at `:4125-4137`; the bubble sort at + `:4173-4186`; `MaxBytesForLevel` at `:5354-5360`. Defaults: + `include/rocksdb/options.h:255` (L0 trigger 4), `:303` (256 MB base), + `include/rocksdb/advanced_options.h:671` (multiplier 10). +- **Step 2 — picking, `db/compaction/compaction_picker_level.cc`**: + `SetupInitialFiles` `:207-260` (threshold at `:214`, the two score comments at + `:229`/`:232`, the descending-order break at `:255-258`, the intra-L0 fallback + at `:248-252`); `PickCompaction` `:531-558`; `SetupOtherInputsIfNeeded` + `:481`; the score recompute after registering a job, `:590-597`. +- **Step 3 — merging, `db/compaction/`**: `compaction_job.cc:725-746` + (sub-compaction fan-out), `:1904` (`ProcessKeyValueCompaction`), `:1920-1924` + (compaction filter), `:1621-1632` (`CompactionIterator` with the snapshot + list). Decisions: `compaction_iterator.cc:356` and `:600-630` + (`kRemove` / `kChangeValue` / `kRemoveAndSkipUntil`); tombstone drops at + `:1152-1187` (early, via `KeyNotExistsBeyondOutputLevel`) and `:1188-1191` + (bottommost). +- **Step 4 — stalls, `db/column_family.cc:1010-1046`**: + `GetWriteStallConditionAndCause`, all six branches. Defaults in + `include/rocksdb/advanced_options.h`: `:271`, `:547`, `:554`, `:709`, `:717`. + Contrast: `fjall src/keyspace/write_delay.rs:5-16` at `80cf6bc`. +- **Step 5 — building, `table/block_based/`**: + `block_based_table_builder.cc:1096-1097` (restart interval + delta encoding), + `:1126-1128` (flush policy), `:1901-1912` (index entry, with the "the r" + example); `index_builder.cc:78-101` (`FindShortestInternalKeySeparator`); + `util/comparator.cc:42-101` (the byte-level algorithm). Defaults: + `include/rocksdb/table.h:400`, `:413`, `:416`, `:423`. +- **Step 6 — reading, `table/block_based/block_based_table_reader.cc`**: `Get` + `:3010`; filter check `:3039-3042`; index iterator `:3044-3053`; the block + loop `:3071-3096`; block cache `:2010` and `:2345`; two-level pinning + `:1776-1780`. Partitioned index: `partitioned_index_reader.h:14-15`, `:27-28`. +- **Step 7 — filters, `table/block_based/filter_policy.cc`**: + `FastLocalBloomBitsBuilder` `:365-377` (`millibits_per_key`); + `Standard128RibbonBitsBuilder` `:658-672`; the 256-reseed fallback + `:751-762`. The numbers: `util/bloom_impl.h:99-131`, + `include/rocksdb/filter_policy.h:169-205`. +- **Step 8 — committing, `db/version_set.cc`**: `LogAndApply` `:6778`; + `ProcessManifestWrites` `:6111`, with `AddRecord` `:6500`, `SyncManifest` + `:6509`, `SetCurrentFile` guarded by `new_descriptor_log` `:6527-6530`; + `AppendVersion` `:6082-6109`; `Version::Ref`/`Unref` `:4941-4951`. Format: + `db/version_edit.h:37-78` (tags), `:701-708` (the definition). ## Questions to answer in notes.md 1. Why does leveled compaction pick by *score* rather than round-robin? - Construct a workload where round-robin lets one level grow unboundedly. + Construct a workload where round-robin lets one level grow unboundedly, and + check it against `SetupInitialFiles`' early break at `:255-258` — what does + the descending sort guarantee that a round-robin scan cannot? 2. Partitioned index vs lsm-tree's per-block hash index — both attack "index - too big for cache". Which helps point reads, which helps scans, why? -3. FastLocalBloom does k probes in one cache line — what does that cost in FPR - vs a classic bloom at equal bits/key? (Blocked blooms have slightly worse - FPR — the locality is paid for in statistics.) + too big for cache". Which helps point reads, which helps scans, why? Size + both for the 256 MB / 4 KB-block SST in Step 6. +3. FastLocalBloom does k probes in one cache line — Step 7 puts the cost at + 0.844% → 0.957% at 10 bits/key. Using topic 0's price list, how many extra + *nanoseconds* of avoided cache misses does that 0.113-point FPR increase buy + back, and at what filter size does the trade stop paying? ## Done when -You can list the three stall triggers from memory and explain LogAndApply's -refcounted-Version scheme as "MVCC for metadata". +Answer each before unfolding it. + +- [ ] You can compute both compaction scores from a level's state, and say which files are excluded from the numerator and why. + +
Answer + + L0: `num_sorted_runs / level0_file_num_compaction_trigger` + (`db/version_set.cc:4077-4078`, trigger defaults to 4, + `include/rocksdb/options.h:255`). L1+: + `level_bytes_no_compacting / MaxBytesForLevel(level)` (`:4136-4137`, with L1 + at `max_bytes_for_level_base` = 256 MB and each level 10× the last). + + Both numerators exclude files with `being_compacted` set — the `if + (!f->being_compacted)` guards at `:4018` and `:4131`. The reason is that a + score is a claim about *outstanding* work: a file already assigned to a running + job will be paid for shortly, and counting it again would make the picker + choose the same level repeatedly and double-book its files. This is also why + `GetCompaction` recomputes every score immediately after registering a new job + (`compaction_picker_level.cc:590-597`). + + Worked: 8 L0 files → 2.00; L1 holding 512 MB → 2.00; start one L1 job covering + 256 MB and L1 drops to (512−256)/256 = 1.00. Under + `level_compaction_dynamic_level_bytes`, scores above 1.0 are additionally + multiplied by `kScoreScale = 10.0` (`:3996`), so logs show 20.0 where this + arithmetic gives 2.0. + +
+ +- [ ] You can list all six write-stall conditions with their defaults, and say what order they are tested in. + +
Answer + + From `db/column_family.cc:1016-1045`, in the code's own order — stops first, + then delays: + + 1. unflushed memtables ≥ `max_write_buffer_number` (2) → **stop** + 2. L0 files ≥ `level0_stop_writes_trigger` (36) → **stop** + 3. pending compaction bytes ≥ `hard_pending_compaction_bytes_limit` (256 GB) → + **stop** + 4. unflushed memtables ≥ `max_write_buffer_number − 1`, only when that option + is > 3 → **delay** + 5. L0 files ≥ `level0_slowdown_writes_trigger` (20) → **delay** + 6. pending compaction bytes ≥ `soft_pending_compaction_bytes_limit` (64 GB) → + **delay** + + Defaults at `include/rocksdb/advanced_options.h:271`, `:554`, `:717`, `:547`, + `:709`. Because it is a single if-else chain, the order is the priority: a + database at 40 L0 files matches condition 2 and reports `kStopped`, never + `kDelayed`. Three causes (memtable limit, L0 file count, pending compaction + bytes) × two severities, and the pending-bytes pair is the only one that + measures debt in bytes rather than in objects. + +
+ +- [ ] You can explain `LogAndApply`'s refcounted-Version scheme as "MVCC for metadata", including what is *not* rewritten on a normal commit. + +
Answer + + A `Version` is the immutable file layout; a `VersionEdit` is a delta against it + (`db/version_edit.h:701-704`), serialized with `kDeletedFile = 6` and + `kNewFile = 7` tags (`:43-44`). `LogAndApply` (`db/version_set.cc:6778`) routes + into `ProcessManifestWrites`, which appends the record (`:6500`) and fsyncs the + MANIFEST (`:6509`). + + The thing that is *not* rewritten is CURRENT: `SetCurrentFile` runs only under + `if (s.ok() && new_descriptor_log)` (`:6527`), i.e. only when a fresh MANIFEST + file was started. A steady-state compaction commit is one appended record plus + one fsync. + + In memory, `AppendVersion` (`:6082-6109`) unrefs the outgoing current version, + installs the new one and refs it (6097-6102), then links it into a + doubly-linked list of live versions (6104-6108). `Version::Unref` frees only + when the count hits zero (`:4943-4951`), so a reader that took a reference + before the swap keeps reading its own snapshot of the file layout, and the + files it names cannot be deleted underneath it. Writers never block readers, + and every historical version is reconstructible by replaying edits — the same + two properties MVCC gives to data. + +
+ +- [ ] You can walk `FindShortestSeparator` on a concrete key pair and say how many bytes it saves. + +
Answer + + Using RocksDB's own example from `block_based_table_builder.cc:1904-1908` and + the algorithm at `util/comparator.cc:42-101`: with + `start = "the quick brown fox"` (the last key of one block) and + `limit = "the who"` (the first key of the next), the common-prefix scan + (`:47-50`) stops at `diff_index = 4`, where `'q'` (0x71) meets `'w'` (0x77). + Since `start_byte < limit_byte` and `0x71 + 1 = 0x72` is still below 0x77 + (`:57`, `:64`), the code increments byte 4 to `'r'` and truncates to 5 bytes + (`:65-66`). The index entry becomes `"the r"` — 19 bytes down to 5, a 74% + saving on that entry, and it is still ≥ every key in the first block and < + every key in the next. + + The branch at `:64` is the interesting one: when incrementing would land + exactly on `limit`, no single-byte bump is legal, and the code walks forward to + the first non-`0xFF` byte instead (`:67-80`). Separately, because this operates + on user keys, the index builder must re-append a sequence number afterwards to + keep internal-key ordering valid (`index_builder.cc:89-97`). + + The payoff is that a smaller index is a more cacheable index — which is the + same lever Step 6 pulls again, one level up, by partitioning it. + +
+ +- [ ] You can state the price RocksDB pays for a cache-local bloom filter, in false-positive rate, and for a ribbon filter, in build cost. + +
Answer + + Cache-local bloom: at 10 bits/key with 6 probes, `util/bloom_impl.h:105-108` + records the theoretical best for a 512-bit (one cache line) bucket as 0.9535%, + this implementation at about 0.957%, and the older + `LegacyLocalityBloomImpl` at 1.138%. The non-local textbook optimum for + the same 10 bits/key and k = 6 is 0.844% — the figure the lsm-tree chapter + computed for fjall's filter. So locality costs roughly 0.113 percentage points, + about 13% relatively more false positives, and buys at most one cache miss per + lookup instead of up to six. + + Ribbon: `include/rocksdb/filter_policy.h:169-173` claims about 30% space saving + at equal false-positive rate — 10 bloom-equivalent bits/key becomes about 7 + actual bits/key at the same 0.95% — for "roughly 3-4x CPU time and 3x temporary + space usage during construction", plus a 3 GB-vs-1 GB temporary-memory example + for a 100 M-key filter (`:200-203`). Construction can also fail outright: the + builder tries 256 seeds (`filter_policy.cc:751-754`) and falls back to bloom + (`:760-762`). Hence the default `bloom_before_level = 0` (`filter_policy.h:184`): + bloom for flushes, ribbon for everything below. + +
## References **Code** -- [facebook/rocksdb](https://github.com/facebook/rocksdb) — - `db/compaction/compaction_picker_level.cc`, - `db/compaction/compaction_job.cc`, `db/column_family.cc` (stalls), - `table/block_based/block_based_table_builder.cc`, - `table/block_based/block_based_table_reader.cc`, - `table/block_based/filter_policy.cc`, `db/version_set.cc` (MANIFEST). - Local clone at `~/repos/rocksdb`. -- [fjall-rs/fjall](https://github.com/fjall-rs/fjall) - `src/keyspace/write_delay.rs` — the 100×-simpler stall valve, for - contrast. +- [facebook/rocksdb](https://github.com/facebook/rocksdb), pinned at `7c80a5a`. + Read in the step order above; budget ~3 h and skim `compaction_job.cc` rather + than reading it. +- [fjall-rs/fjall](https://github.com/fjall-rs/fjall) at `80cf6bc` — + `src/keyspace/write_delay.rs` for the 100×-simpler stall valve. + +| File | Lines | What | +|------|-------|------| +| `db/version_set.cc` | 3983-4186 | `ComputeCompactionScore`: both formulas, in-flight exclusion, `kScoreScale`, the bubble sort | +| `db/compaction/compaction_picker_level.cc` | 207-260, 531-558, 590-597 | pick highest score ≥ 1, expand inputs, then rescore | +| `db/compaction/compaction_job.cc` | 725-746, 1621-1632, 1904 | sub-compactions, snapshot list, the per-thread merge | +| `db/compaction/compaction_iterator.cc` | 1152-1191 | tombstones dropped early when no key exists below the output level | +| `db/column_family.cc` | 1010-1046 | six stall conditions, stops tested before delays | +| `table/block_based/block_based_table_builder.cc` | 1096-1097, 1901-1912 | restart interval 16, and the shortened index separator | +| `util/comparator.cc` | 42-101 | `FindShortestSeparator`, byte by byte | +| `table/block_based/block_based_table_reader.cc` | 3010, 3039-3096 | filter gate, index iterator, data-block loop | +| `table/block_based/partitioned_index_reader.h` | 14-15, 27-28 | two-level index, top level pinned | +| `table/block_based/filter_policy.cc` | 365-377, 658-672, 751-762 | cache-local bloom, ribbon, 256 re-seeds then fall back | +| `util/bloom_impl.h` | 99-131 | the exact FPR cost of confining k probes to one cache line | +| `include/rocksdb/filter_policy.h` | 169-205 | ribbon's 30% / 3-4× CPU trade, and `bloom_before_level` | +| `db/version_edit.h` | 37-78, 701-708 | `kNewFile`/`kDeletedFile`, and the definition of a Version | diff --git a/topics/04-lsm-deep-dive/reading-rocksdb-tods.md b/topics/04-lsm-deep-dive/reading-rocksdb-tods.md index 0648e02..7a06e54 100644 --- a/topics/04-lsm-deep-dive/reading-rocksdb-tods.md +++ b/topics/04-lsm-deep-dive/reading-rocksdb-tods.md @@ -1,7 +1,7 @@ # RocksDB's decade: write amp → space amp → CPU -Not a data-structures chapter — a **10-years-of-production** one. RocksDB's -development priorities shifted three times in a decade, and every shift was +Not a data-structures chapter — a **production retrospective**. RocksDB's +development priorities shifted three times in eight years, and every shift was driven by hardware economics rather than better algorithms. Before the paper, this chapter walks the arc one era at a time — what hardware fact made each metric the bottleneck, and what RocksDB changed in response — then points you @@ -9,135 +9,506 @@ at the sections where the fleet-scale lessons live. Read it for what benchmarks don't show: the failure modes, API regrets, and configuration sprawl that only appear at fleet scale. +**A note on which paper this is.** The title is *Evolution of Development +Priorities in Key-value Stores Serving Large-scale Applications: The RocksDB +Experience*, by Dong, Kryczka, Jin and Stumm. It appeared at **USENIX FAST +'21** and, extended, as ACM **Transactions on Storage** 17(4), Article 26 (TOS, +not TODS). Every section number, quotation and figure below is checked against +the openly available FAST '21 version — so if you are reading the journal +version, expect the section numbering to differ slightly. + ## The problem in one sentence The "right" LSM configuration is not a property of the algorithm but of the -hardware bill: over one decade the binding constraint at Facebook moved from -SSD *endurance* (write amp) to SSD *capacity* ($/GB — space amp) to *CPU* -(NVMe made storage faster than the code driving it) — three different -objective functions for the same engine. +hardware bill: over eight years the binding constraint at Facebook moved from +SSD *endurance* (write amp) to SSD *capacity* ($/GB — space amp) to *CPU and +DRAM price* — three different objective functions for the same engine. ## The concepts, step by step ### Step 1 — the arc: one engine, three objective functions -A production storage engine is tuned to whichever resource currently runs -out first — and at fleet scale that resource is decided by procurement, not -computer science. The paper's history of what RocksDB optimized for, in -order: +> **In:** RocksDB as you know it from the compaction chapter — scores, stalls, +> filters, MANIFEST. +> **Out:** the observation that none of those defaults were chosen on algorithmic +> grounds, and the three-era timeline Steps 2-5 walk. + +A production storage engine is tuned to whichever resource currently runs out +first — and at fleet scale that resource is decided by procurement, not computer +science. The paper's own summary, from the abstract: "We describe how and why +RocksDB's resource optimization target migrated from write amplification, to +space amplification, to CPU utilization." ``` -2012 ───────► 2015 ───────► 2018 ───────► 2021 -write amp space amp CPU disaggregated / remote storage -(SSD wear, (SSDs got (storage got (storage moves off-box; - fillrandom cheaper — fast enough topic 28 territory) - benchmarks) $/GB rules) that CPU is - the bottleneck) + 2012 ─────────► ~2015 ─────────► ~2018 ─────────► 2021 + write amp space amp CPU & DRAM disaggregated storage + (flash erase (flash cycles (space-amp wins (CPU and SSD can be + cycles are and IOPS both already banked; provisioned separately; + the budget) turned out to CPU/memory "a current priority") + be slack; $/GB prices rose + rules) relative to SSD) ``` Each shift happened because the *hardware economics* moved, not because the -algorithms improved. This is the RUM triangle (topic 1) steered by -procurement — the same trade-off space, with the weights set by the price -list. Steps 2–5 take the eras one at a time. +algorithms improved. This is the RUM triangle (topic 1) steered by procurement — +the same trade-off space, with the weights set by the price list. + +Do not read the arrow as "the previous metric stopped mattering". §3 is explicit +that write amplification "continues to be an issue" for write-heavy workloads, +and the CPU era is motivated by the space-amp work being *done*, not by space +ceasing to matter. ### Step 2 — the write-amp era: flash wears out -Write amplification (bytes physically written to flash per byte of user -data) was the founding obsession because flash cells have a finite erase -budget — a 2012-era SSD tolerated only a few thousand program/erase cycles -per cell, so an engine with WA 30 wears the drive out 30× faster than the -raw data rate suggests, and at fleet scale that is a hardware replacement -line-item. RocksDB's founding pitch over its LevelDB ancestor was exactly -this: batch more, merge smarter, benchmark `fillrandom` WA. This is the era -your mini-LSM's write-amp experiment recreates. +> **In:** an LSM and a 2012 SSD with a finite erase budget. +> **Out:** why write amplification was the founding metric, with the measured +> range RocksDB actually achieves — and the reason it was not enough. + +Write amplification (bytes physically written to flash per byte of user data) +was the founding obsession because flash cells have a finite program/erase +budget, and at fleet scale that is a hardware replacement line-item. §3: "When we +started developing RocksDB, we initially focused on saving flash erase cycles +and thus write amplification, following the general view of the community at the +time." + +The measured numbers, all §3 "Write amplification": + +``` + SSD-internal write amp (observed) 1.1 – 3 + storage/database software write amp up to 100 + (a full 4/8/16 KB page written for a <100 B change) + RocksDB Leveled Compaction 10 – 30 + RocksDB Tiered Compaction 4 – 10 + — "although with lower read performance" + + RocksDB vs InnoDB, LinkBench on MySQL: + RocksDB issues 5% as many writes per transaction +``` + +Two things to take from that block. First, **10-30× is what leveled compaction +actually costs in production**, which brackets the ~20× that +`topics/04-lsm-deep-dive/notes.md` derives from `T/2 × L` at T = 10, L = 4 — a +rare case where the textbook model and the fleet agree. Second, the paper is +candid that 10-30 "is too high for write-heavy applications. For this reason we +added Tiered Compaction" — the design-space chapter's `Tier`, adopted for a +purely economic reason. + +The B-tree comparison is the one to keep: **5% of InnoDB's writes per +transaction**. That is the LSM's whole pitch in one number, and it lines up with +this repo's own measured lane — `FINDINGS.md` row 1 (`./verify.sh 01`, Apple M3 +Pro, 2026-07-28): the same 108 MB of records occupies 48 MB under fjall's LSM +and 6.8 GB under redb's copy-on-write B-tree. + +This is the era your mini-LSM's `write_amp` experiment recreates. ### Step 3 — the space-amp era: $/GB beats endurance -By ~2015 SSDs had gotten cheap and durable enough that the dominant cost -was simply *how many bytes of flash you must buy* — space amplification -(bytes on disk per byte of live data). This flipped the compaction -preference: leveled compaction's *high* write amp became acceptable because -its space amp is excellent (~1.1×, since the bottom level is one run holding -~90% of data with few stale versions — the Dostoevsky chapter's Step 2 in -production dollars), while tiered's up-to-K× space overhead priced it out of -most Facebook services. Universal (tiered) compaction survived only for -ingest-heavy workloads. When storage is billed by the byte-month, WA is a -tax you pay once; space amp is rent you pay forever. - -### Step 4 — the CPU era: NVMe outran the software - -Around 2018, NVMe drives delivering hundreds of thousands of IOPS at tens of -microseconds stopped being the bottleneck — the CPU cycles spent *per -operation* (merge comparisons, block decode and decompression, filter -hashing at every level, checksum verification) became the limiting resource. -The optimization target moved inside the CPU: cheaper comparators, less -decompression on the read path, filter designs trading build CPU for DRAM -(the ribbon filter from the compaction chapter is this era's artifact). -Reconcile with your topic 0 finding — SipHash at 21%, memory stalls dominant -— and the LSM adds its own CPU stack on top of a hash table's. - -### Step 5 — the remote-storage era: the disk leaves the box - -By 2021 the direction is **disaggregated storage** — SSTs living on shared -remote storage (network-attached), with compute and capacity scaled -independently and compaction potentially offloaded to other machines -("remote compaction"). The economics again: pooled storage beats stranded -per-box capacity at fleet scale. The predictions in §5 are 2021-vintage and -checkable — topic 28 will grade them. +> **In:** the write-amp era's assumption that flash endurance is the scarce +> resource. +> **Out:** the measurement that falsified it, the compaction change it caused, +> and the numbers to quote instead of "space amp ≈ 1.1×". + +By the mid-2010s the assumption had simply stopped being true. §3: "we observed +that for most applications, space utilization was far more important than write +amplification, given that neither flash write cycles nor write overhead were +constraining. In fact the number of IOPS utilized in practice was low compared +to what the SSD could provide." + +Note the shape of that argument: they did not find a better algorithm, they +*measured their own fleet* and discovered they had been optimizing slack. The +supporting evidence is Figure 3 — a survey of **42 different production +deployments** of ZippyDB and MyRocks, each serving a different application, +measured over a month across four axes (flash endurance, read bandwidth, space, +CPU). "Most of the workloads are space constrained." + +The engineering response was **Dynamic Leveled Compaction**: size each level +from the *actual* size of the last level rather than from static targets. (This +is `level_compaction_dynamic_level_bytes`, the branch you saw guarding the score +formula at `rocksdb db/version_set.cc:4135`.) The measured effect, §3 and +Table 4 — RocksDB 5.9, all defaults, constant 2 MB/s write rate, keys chosen +randomly from a prepopulated space: + +| keys | fully compacted | steady state | overhead | +|---|---|---|---| +| 200 M | 12.0 GB | 13.5 GB (dynamic) | **12.4%** | +| 200 M | 12.0 GB | 15.1 GB (LevelDB-style) | 25.6% | +| 1,000 M | 60.1 GB | 67.5 GB (dynamic) | **12.4%** | +| 1,000 M | 60.3 GB | 73.8 GB (LevelDB-style) | 22.4% | + +"Dynamic Leveled Compaction limits space overhead to 13%, while Leveled +Compaction can add more than 25%. Moreover, space overhead in the worst case +under Leveled Compaction can be as high as **90%**, while it is stable for +dynamic leveling." + +So the figure to quote for leveled space amplification is **1.13× with dynamic +leveling, 1.25× with static leveling and up to 1.9× worst case** — not a flat +"~1.1×". And the number that made the business case: "for UDB, one of Facebook's +main databases, the space footprint was reduced to **50%** when InnoDB was +replaced by RocksDB." + +When storage is billed by the byte-month, write amp is a tax you pay once; space +amp is rent you pay forever. This is the Dostoevsky chapter's Step 2 in +production dollars. + +### Step 4 — the CPU era: not the bottleneck, but the price + +> **In:** the popular claim that NVMe outran the software. +> **Out:** the paper's flat rejection of that claim, and the *actual* reason CPU +> became an optimization target — which is a different argument with different +> consequences. + +Here the paper says something more interesting than the story usually told, and +it is worth reading twice because it contradicts the received version: + +> An issue of concern sometimes raised is that SSDs have become so fast that +> software is no longer able to take advantage of their full potential. That is, +> with SSDs, the bottleneck has shifted from the storage device to the CPU, so +> fundamental improvements to the software are necessary. **We do not share this +> concern based on our experience**, and we do not expect it to become an issue +> with future NAND flash based SSDs for two reasons. First, only a few +> applications are limited by the IOPS provided by the SSDs… most applications +> are limited by space. Second, we find that any server with a high-end CPU has +> more than enough compute power to saturate one high-end SSD. **RocksDB has +> never had an issue making full use of SSD performance in our environment.** +> (§3, "CPU utilization") + +So: **CPU did not become the bottleneck.** What happened is a price movement. +The paper's actual argument for the CPU era, same section: + +- "reducing CPU overheads has become an important optimization target, given + that the low hanging fruit of reducing space amplification **has been + harvested**"; +- "until several years ago, the price of CPUs and memory was reasonably low + relative to SSDs, but **CPU and memory prices have increased substantially**, + so decreasing CPU overhead and memory usage has increased in importance"; +- and it "improves the performance of the few applications where the CPU is + indeed constraining" — a minority, per Figure 3's 42 deployments. + +The named early work is filter-side, which ties straight back to the compaction +chapter: "prefix bloom filters, applying the bloom filter **before** index +lookups, and other bloom filter improvements." (The ribbon filter — ~30% less +space for 3-4× the build CPU — is this era's later artifact, and the trade +direction tells you the era: spend CPU, save DRAM, because DRAM got expensive.) + +The paper also names the two cases where CPU *does* bind: a badly balanced host +(one CPU, many SSDs), and intensive write-dominated workloads — for which the +suggested fix is a lighter compression option, or the observation that "the +workload may simply not be suitable for SSDs since it would exceed the typical +flash endurance budget that allows the SSD to last **2-5 years**." + +Reconcile all of this with your topic 0 finding — SipHash at 21% of a HashMap +lookup, memory stalls dominant — and note that the LSM stacks its own CPU costs +(merge comparisons, block decode and decompression, filter hashing at every +level, checksum verification) on top of a hash table's. + +### Step 5 — what comes next: the disk leaves the box + +> **In:** three eras of local, directly-attached flash. +> **Out:** the 2021-vintage forward look, stated as the paper states it — which +> is narrower than the usual paraphrase. + +§3's "Adapting to newer technologies" surveys the candidates and mostly *rejects* +them, with one exception. Open-channel SSDs, multi-stream SSDs and ZNS "would +benefit only a minority of the applications using RocksDB, given that most +applications are space constrained, not erase cycle or latency constrained" — +Step 3's finding used as a filter on the roadmap. In-storage computing: unclear +benefit, would need API changes through the whole stack. + +The exception: + +> **Disaggregated (remote) storage** appears to be a much more interesting +> optimization target and is a current priority… With remote storage, it is +> easier to make full use of both CPU and SSD resources at the same time, +> because they can be separately provisioned on demand (something much more +> difficult to achieve with locally attached SSDs). (§3) + +Note the reason: not throughput, but **independent provisioning** — the same +economics argument as every other era. Storage-class memory gets three +possibilities and no commitment; the paper notes drily that using SCM as main +storage is awkward because "RocksDB tends to be bottlenecked by space or CPU, +rather than I/O". + +And under "Main Data Structure Revisited", the answer to the question this whole +topic circles: "We continuously revisit the question of whether LSM-trees remain +appropriate, but continue to come to the conclusion that they do." The one +concession is **key-value separation** for large objects (WiscKey-style), shipped +as **BlobDB**. + +§8's roadmap includes one line that should make you sit up after the previous two +chapters: "we plan to **unify leveled and tiered compaction** and improve +adaptivity." That is Dostoevsky's Fluid LSM-tree, on RocksDB's own to-do list. +The open questions are worth reading as a research menu — hybrid SSD/HDD, the +cost of long runs of consecutive deletion markers, better write throttling, +efficient replica comparison, SCM, and a generic integrity handoff API. ### Step 6 — the fleet-scale lessons: what only production teaches -The paper's most valuable section (§4) is not about performance at all; -it's what running the engine on hundreds of thousands of machines proves: - -- **Silent corruption is a certainty, not a risk.** At fleet scale, - "unlikely" bit-flips (controller bugs, RAM, kernel) happen daily — - which is why RocksDB checksums at *every* layer independently: per block, - per file, per WAL record, trusting no layer below. -- **API regrets are forever**: sequence numbers and (missing) user - timestamps leaked into the public API early and constrain everything - since. -- **Configuration sprawl is an acknowledged failure**: hundreds of knobs, - most users unable to set them — the price of a decade of "add an option" - compromises (contrast Monkey/Dostoevsky's "solve for the knob" ethos). - -These are the parts benchmarks can't show and the reason to read a TODS +> **In:** the three eras, all of which are performance stories. +> **Out:** the three sections that are not about performance at all — and the +> measured failure rates that make them the most valuable part of the paper. + +§4 (serving large-scale systems), §5 (failure handling) and §6 (the key-value +interface) are what running the engine on hundreds of thousands of machines +proves. + +**Silent corruption has a measured rate.** §5 quantifies it rather than +gesturing at it, by comparing primary and secondary indexes in MyRocks tables +that have both — any inconsistency must have been introduced below the +application: + +> Based on our measurements, corruptions are introduced at the RocksDB level +> roughly **once every three months for each 100 PB of data**. Worse, in **40%** +> of those cases, the corruption had already propagated to other replicas. + +And separately, from one storage-system bug in network-failure handling: +"roughly **17 checksum mismatches for every petabyte of physical data +transferred**." That 40% is the sentence that justifies the whole design: if +corruption reaches replicas before detection, replication is not a safety net. + +**Hence checksums at every layer, each catching a different threat** (§5, +"Multi-layer protection"): *block* checksums (inherited from LevelDB, verified on +**every read**, keeping filesystem-level corruption away from clients); *file* +checksums (added in **2020**, recorded in the MANIFEST's SSTable entry and +validated wherever the file is transferred, so corruption cannot ride a backup +into a replica); *handoff* checksums (passed down to the filesystem so WAL +appends are validated incrementally at write time — "unfortunately, local file +systems rarely support this"); plus a *planned* application-layer checksum. Each +layer distrusts the one below it, by design. + +**API regrets are forever** (§6). RocksDB uses internal **56-bit sequence +numbers**, incremented on every client write and not settable by the +application. Snapshots pin a version — but only going forward: "RocksDB does not +support taking a snapshot of the past, since there is no API to specify a +time-point." And because each instance assigns its own sequence numbers, "it is +essentially impossible to create versions of data that offer cross-shard +consistent reads." Applications work around it by encoding timestamps in the key +(which hurts point lookups) or the value (which hurts scans); the fix under way +is user-defined timestamps as a first-class concept. + +**Configuration sprawl is an acknowledged failure** (§4, "Managing +configurations"). "A common complaint now is that there are far too many options +and that it is too difficult to understand their effects; i.e., it has become +very difficult to specify an 'optimal' configuration." Worse, the optimum depends +on the application above, not just the system embedding RocksDB: across the +**39 ZippyDB deployments** sampled in Table 5 there are **over 25 distinct +configurations** (14 of them differing in the compaction area alone) — despite +"significant efforts… to use uniform configurations wherever possible". Contrast +Monkey and Dostoevsky's "solve for the knob" ethos: the paper's own list of +suggestions that *did not* work out opens with "**Customizability is always good +to users**". + +These are the parts benchmarks can't show, and the reason to read a production retrospective instead of another asymptotic analysis. ## How to read the paper (with the concepts in hand) -1. §1–2 — background + the resource-priority history (Steps 1–5's arc, in - the authors' words). -2. §3 — lessons on compaction: why leveled won at Facebook (space, Step 3), - universal kept for ingest-heavy; the tiered-vs-leveled discussion with - production numbers instead of asymptotics. -3. **§4 — large-scale lessons** (Step 6). The best section: failure - handling and layered checksums, the timestamp/seqno API regrets, - configuration sprawl. -4. §5 — future directions (2021 vintage, Step 5): remote compaction, tiered - storage — check which happened (topic 28 will). +Budget about 2 h. Section numbers below are the **FAST '21** version's. + +1. **§1-2** — background and RocksDB architecture. Skim §2.2 if the compaction + chapter is fresh; do read §2.1 on flash economics. +2. **§3 Evolution of resource optimization targets** — Steps 1-5, the whole arc, + in the authors' words. Read the "CPU utilization" subsection twice: it argues + *against* the popular framing. Figure 3 (42 deployments) and Table 4 (space + overhead) are the two quantitative anchors. +3. **§4 Lessons on serving large-scale systems** — resource management across + many instances on a host, WAL treatment, rate-limited file deletions, data + format compatibility, configuration management (Table 5), replication and + backup. +4. **§5 Lessons on failure handling** (Step 6) — the best section. The corruption + rates, the multi-layer checksum design (Fig. 4), and differentiated error + handling. +5. **§6 Lessons on the key-value interface** — the 56-bit sequence numbers, why + snapshots don't compose across shards, user-defined timestamps. +6. **§8 Future Work** and the appendix — the six open questions, the numbered + "lessons learned", and the short list of "suggestions that did not work out", + which is the most quotable page in the paper. ## Questions to answer in notes.md -1. The paper says CPU became the bottleneck once NVMe arrived. Reconcile with - your topic-0 finding (SipHash 21%, memory stalls dominant): which CPU costs - does an LSM add on top of a hash table's? (Comparisons in merges, block - decode/decompress, filter hashing per level.) +1. The popular story says CPU became the bottleneck once NVMe arrived; §3 says + RocksDB "has never had an issue making full use of SSD performance". Which + argument does the paper actually make for optimizing CPU, and what evidence + backs it? Then reconcile with your topic-0 finding (SipHash 21%, memory + stalls dominant): which CPU costs does an LSM add on top of a hash table's? + (Comparisons in merges, block decode/decompress, filter hashing per level, + checksum verification.) 2. Why does RocksDB checksum at block AND file AND WAL-record level rather - than trusting the filesystem? What's the FalkorDB/redis equivalent story? - (RDB has a CRC; AOF... check.) -3. Pick the lesson from §4 most relevant to the capstone and write one + than trusting the filesystem? (§5's answer is the 40% figure — corruption + reaches replicas before detection.) What's the FalkorDB/redis equivalent + story? (RDB has a CRC; AOF… check.) +3. Pick the lesson from §4-§6 most relevant to the capstone and write one paragraph on how it changes your M4 design. ## Done when -You can narrate the write-amp → space-amp → CPU priority arc with the hardware -reason for each transition. +Answer each before unfolding it. + +- [ ] You can narrate the three-era arc with the hardware reason for each transition. + +
Answer + + **Write amp (2012-)**: flash cells have a finite program/erase budget, so an + engine writing 30× the user data wears a fleet out 30× faster. RocksDB + inherited this priority from the community consensus of the time. + + **Space amp (~2015-)**: they measured their own fleet and found the assumption + false — "neither flash write cycles nor write overhead were constraining. In + fact the number of IOPS utilized in practice was low compared to what the SSD + could provide." Figure 3's 42-deployment survey shows most workloads space + constrained. Flash was now cheap and durable enough that $/GB dominated. + + **CPU (~2018-)**: *not* because the SSD outran the software (§3 explicitly + rejects that), but because the space-amp low-hanging fruit had been harvested + and "CPU and memory prices have increased substantially" relative to SSDs. It + is a cost-per-server argument, not a saturation argument. + + The through-line: every transition was forced by a price list, not by a better + algorithm. Same RUM triangle, different weights. + +
+ +- [ ] You can give real numbers for RocksDB's write and space amplification. + +
Answer + + Write amp (§3): SSD-internal 1.1-3 as observed; software-level "sometimes as + high as 100" when a full 4/8/16 KB page is written for a sub-100-byte change; + **RocksDB leveled compaction 10-30**; **tiered compaction 4-10**, "although + with lower read performance". Against a B-tree: on LinkBench over MySQL, + "RocksDB issues only 5% as many writes per transaction as InnoDB". + + Space amp (§3, Table 4 — RocksDB 5.9, defaults, 2 MB/s constant write rate): + **Dynamic Leveled Compaction holds space overhead to ~13%** (12.4% at both + 200 M and 1000 M keys); static LevelDB-style leveling "can add more than 25%" + (22.4-25.6% measured) and "in the worst case can be as high as **90%**". The + business number: UDB's footprint fell to **50%** when InnoDB was replaced by + RocksDB. + + The 10-30× leveled figure brackets this repo's own `T/2 × L ≈ 20×` model at + T = 10, L = 4 (`notes.md`) — model and fleet agreeing is worth noting, since + they usually don't. + +
+ +- [ ] You can state what the paper says about CPU being the bottleneck, and why that matters. + +
Answer + + It denies it. §3, "CPU utilization": "We do not share this concern based on our + experience… any server with a high-end CPU has more than enough compute power + to saturate one high-end SSD. RocksDB has never had an issue making full use of + SSD performance in our environment." The exceptions named are unbalanced hosts + (one CPU, several SSDs) and write-dominated workloads — for which the paper + suggests lighter compression, or notes the workload may not suit SSDs at all + given a 2-5 year endurance budget. + + CPU became a target for two other reasons: the space-amp work was done, and + CPU/DRAM prices rose relative to flash, so shaving CPU and memory buys + cheaper hardware configurations. Early work was filter-side — prefix bloom + filters, applying the filter *before* index lookups. + + Why it matters: the two framings prescribe different work. "The CPU is the + bottleneck" says rewrite the hot path. "CPU is expensive relative to flash" + says trade CPU *for* DRAM and space where the price list favours it — which is + exactly what the ribbon filter does (~30% less space for 3-4× build CPU), and + it would look like a bad trade under the first framing. + +
+ +- [ ] You can quote the corruption rate and explain the multi-layer checksum design from it. + +
Answer + + §5: corruption is "introduced at the RocksDB level roughly **once every three + months for each 100 PB of data**", measured by comparing primary and secondary + indexes in MyRocks tables — and "in **40%** of those cases, the corruption had + already propagated to other replicas". Separately, one storage-system bug in + network-failure handling produced "roughly **17 checksum mismatches for every + petabyte of physical data transferred**". + + The 40% is the design driver: if corruption reaches replicas before anyone + notices, replication is not a safety net, so detection has to happen *early* + and at *every* layer. Hence: **block** checksums (from LevelDB, verified on + every read, keeping filesystem-level corruption from clients); **file** + checksums (added 2020, stored in the MANIFEST's SSTable entry and validated on + every transfer, so corruption cannot ride a backup into a replica); **handoff** + checksums (passed down with WAL writes for incremental validation — but "local + file systems rarely support this"); plus a planned application-layer checksum. + Each layer distrusts the one below. + +
+ +- [ ] You can name two things the paper admits it got wrong, with specifics. + +
Answer + + **Configuration sprawl** (§4, "Managing configurations"): "a common complaint + now is that there are far too many options and that it is too difficult to + understand their effects". Worse, the optimum depends on the application above + the embedding system — across the 39 ZippyDB deployments in Table 5 there are + over 25 distinct configurations (14 differing in compaction alone), despite + deliberate effort to unify them. The appendix's "suggestions that did not work + out" opens with "Customizability is always good to users." + + **Versioning in the API** (§6): 56-bit sequence numbers are internal, + incremented per client write and not settable; snapshots only pin the present, + because "RocksDB does not support taking a snapshot of the past, since there is + no API to specify a time-point"; and since each instance numbers independently, + "it is essentially impossible to create versions of data that offer cross-shard + consistent reads". Applications encode timestamps in the key (hurting point + lookups) or the value (hurting scans). User-defined timestamps are the fix in + progress. + + Two more from the same appendix list, both worth a moment: "RocksDB can be + blind to CPU bit flips" and "It's OK to panic when seeing any I/O error." + +
## References **Papers** -- Dong, Kryczka, Jin, Stumm — "RocksDB: Evolution of Development - Priorities in a Key-value Store Serving Large-scale Applications" - (ACM TODS 2021) — §4 (large-scale lessons) is the best section; §5's - 2021-vintage future directions are checkable predictions for topic 28 +- Dong, Kryczka, Jin, Stumm — *Evolution of Development Priorities in Key-value + Stores Serving Large-scale Applications: The RocksDB Experience*, USENIX FAST + '21 (open access), extended as ACM Transactions on Storage 17(4), Art. 26. + §3 is the three-era arc, §5 (failure handling) is the best section, §8 and the + appendix hold the open questions and the "suggestions that did not work out". + All citations below are to the FAST '21 version. + +| Claim in this chapter | Source | +|---|---| +| Optimization target migrated write amp → space amp → CPU | Abstract; §3 | +| SSD-internal WA 1.1-3; software WA up to 100 | §3, "Write amplification" | +| Leveled WA 10-30; tiered WA 4-10 | §3, "Write amplification" | +| RocksDB issues 5% of InnoDB's writes per transaction (LinkBench) | §3 | +| IOPS were slack; most workloads space constrained | §3, "Space amplification"; Figure 3 (42 deployments) | +| Dynamic Leveled Compaction: ~13% overhead vs >25%, worst case 90% | §3 and Table 4 | +| UDB footprint reduced to 50% replacing InnoDB | §3 | +| "We do not share this concern" — CPU is not the bottleneck | §3, "CPU utilization" | +| CPU targeted because space-amp fruit harvested and CPU/DRAM prices rose | §3, "CPU utilization" | +| Prefix bloom filters, filter before index lookup | §3, "CPU utilization" | +| SSD endurance budget sized for 2-5 years | §3, "CPU utilization" | +| Open-channel / multi-stream / ZNS benefit only a minority | §3, "Adapting to newer technologies" | +| Disaggregated storage is "a current priority", for independent provisioning | §3 | +| LSM-trees remain appropriate; BlobDB for key-value separation | §3, "Main Data Structure Revisited" | +| Plan to unify leveled and tiered compaction; six open questions | §8 | +| Corruption once per 3 months per 100 PB; 40% already replicated | §5, "Frequency of silent corruptions" | +| 17 checksum mismatches per PB transferred | §5 | +| Block / file (2020) / handoff checksum layers | §5, "Multi-layer protection"; Fig. 4 | +| 56-bit sequence numbers; no snapshot of the past; no cross-shard versions | §6, "Versions and timestamps" | +| 39 ZippyDB deployments, over 25 distinct configurations | §4, "Managing configurations"; Table 5 | +| "Customizability is always good to users" among failed suggestions | Appendix, "Suggestions that did not work out" | + +**Code** +- `rocksdb db/version_set.cc:4135` at `7c80a5a` — the + `level_compaction_dynamic_level_bytes` branch, i.e. Dynamic Leveled + Compaction from §3, in the scoring code. + +**Repo cross-references** +- `FINDINGS.md` row 1 — the measured fjall-vs-redb space comparison used in + Step 2, since topic 4 has no lane of its own. +- `topics/04-lsm-deep-dive/notes.md` — the `T/2 × L ≈ 20×` write-amp model that + §3's 10-30 range brackets. +- `topics/04-lsm-deep-dive/reading-rocksdb-compaction.md` — the scores, stalls + and filters this chapter gives the economic backstory for. +- `topics/04-lsm-deep-dive/reading-dostoevsky.md` — Fluid LSM-tree, which §8 + lists as RocksDB roadmap. diff --git a/topics/05-durability-wal/README.md b/topics/05-durability-wal/README.md index 2c838a3..1872a9e 100644 --- a/topics/05-durability-wal/README.md +++ b/topics/05-durability-wal/README.md @@ -95,8 +95,12 @@ invariant: | macOS `F_FULLFSYNC` | drive cache flushed | ms-scale — measure it! | | `O_DIRECT` + own buffering | bypass page cache | topic 6 | -Group commit exists because of this ladder: if fsync costs 1ms, one fsync per -commit caps you at 1K commits/s — but one fsync can cover N commits. +Group commit exists because of this ladder, and the rung decides the size of the +problem. On this machine one `F_FULLFSYNC` per commit caps you at **337 +commits/s** and one macOS `fsync` per commit at **44,109** — the same code, two +orders of magnitude apart, depending only on which call you made. One flush can +cover N commits either way, so the arithmetic to do is N = λ·T: at an offered +100,000 commits/s, a 2.97 ms flush gathers 297 of them. ```mermaid flowchart LR diff --git a/topics/05-durability-wal/notes.md b/topics/05-durability-wal/notes.md index 7f91490..ee03d3c 100644 --- a/topics/05-durability-wal/notes.md +++ b/topics/05-durability-wal/notes.md @@ -24,6 +24,12 @@ transaction is capped there regardless of how fast the rest of the engine is, which is why group commit is not an optimization but a structural requirement — and why topic 15's follower-fsync table looks the way it does. +And note who is on which rung. Redis defines `redis_fsync` as `fdatasync()` on +Linux but as `fcntl(fd, F_FULLFSYNC)` on Apple (`src/config.h:128-135` at the +pinned revision), so `appendfsync always` on this machine pays the 2.97 ms rung, +not the 22.67 µs one. The same configuration file, on the same version, means +two things 131× apart depending on the kernel underneath it. + ## Predictions (fill BEFORE running fsync_ladder) | Rung | Predicted p50 | Measured p50 | Measured p99 | diff --git a/topics/05-durability-wal/reading-aether.md b/topics/05-durability-wal/reading-aether.md index 8e58ab8..278caaf 100644 --- a/topics/05-durability-wal/reading-aether.md +++ b/topics/05-durability-wal/reading-aether.md @@ -2,178 +2,509 @@ On a multicore, the log is ONE shared object every transaction must append to and flush — so how does it not become the bottleneck? Aether's answer is four -independent fixes that compose, and one of them (consolidation arrays) is the -ancestor of how postgres inserts WAL today. Before the paper, this chapter -builds the four bottlenecks one at a time and then each fix in order of -increasing cleverness — ending with the one that shipped everywhere. +independent fixes that compose, and two of them are the ancestors of how +postgres inserts WAL today. Before the paper, this chapter builds the four +bottlenecks one at a time — in the authors' own lettering, which is not the one +this chapter used to give — and then each fix in order of increasing cleverness, +ending with the measurement that says which one actually mattered. + +Every claim below was checked against the paper as published: **Ryan Johnson, +Ippokratis Pandis, Radu Stoica, Manos Athanassoulis & Anastasia Ailamaki, +"Aether: A Scalable Approach to Logging", *PVLDB* 3(1), VLDB 2010, pp. 681–692.** +Section, figure and table numbers are cited for each. Every timing attributed to +*this machine* comes from `experiments/src/bin/fsync_ladder.rs` as recorded in +`notes.md`. + +## Vocabulary, defined once, before it is used + +| Term | Meaning | +|---|---| +| **WAL** (write-ahead logging) | a change's log record is made durable before the changed page may reach nonvolatile storage; see `reading-aries.md` | +| **LSN** (log sequence number) | a log record's offset in the log; its position in the total order | +| **log buffer** | the in-memory staging area records are copied into before they are written out | +| **buffer acquire** | reserving a byte range of the log buffer — assigns the LSN, must be serial | +| **buffer fill** | copying the record's bytes into that reserved range — needs no serialization | +| **group commit** | letting one durability call cover many transactions' commit records | +| **ELR** (early lock release) | releasing a transaction's database locks at commit-record *creation* rather than *durability* (§3.1) | +| **flush pipelining** | detaching the worker thread from the commit wait; a daemon acks clients after the flush (§4.1) | +| **consolidation array** | an auxiliary array of slots where contending threads combine their buffer requests before touching the log mutex (§5.1) | +| **decoupled buffer fill** | releasing the log mutex immediately after buffer acquire, so fills pipeline (§5.2) | +| `fsync` / `fdatasync` / `F_FULLFSYNC` | the three durability rungs — see below; the paper says "log flush" and means whichever one your system uses | + +**The durability ladder, measured here**, because "log flush latency" is the +paper's central quantity and a bare number for it is worthless: + +| call | what it guarantees | p50 on this machine | implied ops/s | +|---|---|---|---| +| `write()` alone | bytes in the OS page cache — survives `kill -9`, not power loss | 1.17 µs | 856 898 | +| `fdatasync()` / macOS `fsync()` | bytes handed to the drive, whose volatile cache may still hold them | 22.67 µs | 44 109 | +| macOS `fcntl(fd, F_FULLFSYNC)` | the drive flushed its cache to stable media | 2.97 ms | 337 | + +**19.4×** from the first rung to the second; a further **131×** to the third; +**2 542×** end to end. The middle row was measured on macOS as `fsync(2)` — +there is no `fdatasync` on this machine, `fsync_ladder.rs` compiles that lane +out — and it is named above only because it occupies the same rung on Linux. +Aether's own device series (§3.2) is a different ladder — +0 ms ramdisk (40–80 µs of kernel round trip), 100 µs "fast flash drive", 1 ms +"fast magnetic drive", 10 ms "slow magnetic drive" — and this machine's +`F_FULLFSYNC` rung, at 2.97 ms, sits between the paper's last two. ## The problem in one sentence -Every committing transaction must append to a single serial log and wait for -it to reach disk, so on a 32-core machine the log is 32 threads funneling -into one mutex and one ~1 ms fsync — a hard ceiling of ~1K commits/s and -worsening lock contention, no matter how many cores you add. +Every committing transaction must append to a single serial log and wait for it +to reach disk, so on a many-core machine the log is every thread funneling into +one mutex and one flush — and on this machine that flush is 22.67 µs or 2.97 ms +depending only on which system call you chose, which is enough to leave the +paper's 64-context server **75% idle** on lock contention alone (§1.1, Fig. 2). ## The concepts, step by step ### Step 1 — why the log must be serial, and what that costs on a multicore -The entire recovery story of topic 5 rests on the log being one totally -ordered sequence: records are replayed in log order, commit order *is* log -order, and a record is durable only if everything before it is. That total -order is bought with physical serialization — one append point, one flush -frontier. On one core in 1992 this was free; on a multicore it turns the log -into the single object every transaction must touch twice (once to insert +> **In:** the recovery requirements of topic 5 — replay in log order, commit +> order equals log order. +> **Out:** a single append point and a single flush frontier, i.e. exactly the +> shape of object that does not scale with core count. + +The entire recovery story of topic 5 rests on the log being one totally ordered +sequence: records are replayed in log order, commit order *is* log order, and a +record is durable only if everything before it is. Aether states the constraint +sharply while explaining decoupled buffer fill (§5.2): "Log records must be +written to disk in LSN order because **recovery must stop at the first gap it +encounters**; in the event of a crash any committed transactions beyond a gap +would be lost." + +That total order is bought with physical serialization — one append point, one +flush frontier. On one core in 1992 this was free; on a multicore it turns the +log into the single object every transaction must touch twice (once to insert its records, once to await the flush). Aether's contribution starts with *naming* the distinct ways that hurts. +*Why it matters:* "the gap" is the reason none of the four fixes below is +allowed to reorder anything. Every one of them preserves LSN order and attacks +only *waiting*. + ### Step 2 — the four bottlenecks, separated -Four different waits hide inside "commit is slow", with four different -causes — a mutex, a disk, a lock table, and a scheduler: +> **In:** the single observation "commit is slow" on a 64-context server. +> **Out:** four distinct waits with four distinct causes — a disk, a lock table, +> a scheduler, and a mutex — each with its own fix and its own measurement. + +The paper's abstract names them, and this chapter now uses the paper's letters. +(An earlier version of this file assigned A–D in a different order; if you +remember "A = the buffer mutex", relabel.) Quoting the abstract: + +> "(a) the high volume of small-sized I/O requests may saturate the disk, (b) +> transactions hold locks while waiting for the log flush, (c) extensive context +> switching overwhelms the OS scheduler with threads executing log I/Os, and (d) +> contention appears as transactions serialize accesses to in-memory log data +> structures." ``` - txn commits ──► [A] contend on log-buffer insert (one mutex around append) - ──► [B] wait for fsync (I/O latency per commit) - ──► [C] hold locks WHILE waiting for [B] (lock contention amplified) - ──► [D] context switches around the wait + txn commits ──► (a) one small I/O per commit saturates the device + ──► (b) locks held WHILE waiting for (a) lock contention amplified + ──► (c) block/unblock per commit scheduler overload + ──► (d) contend on the log buffer one mutex around append ``` -[A] is CPU-side: every append serializes on the buffer mutex, and memcpy -happens *inside* the critical section. [B] is the disk: ~1 ms of fsync per -commit. [C] is the multiplier: the transaction still holds its row/page -locks while waiting on [B], so one fsync delay cascades into every -transaction queued behind those locks. [D] is the OS: blocking on IO means -descheduling and rescheduling threads at ~µs each. Read the paper as four -independent fixes that compose: +Figure 1 is the paper's own picture of this; §1.1 gives the measurements. Two +are worth carrying: with locks held across the flush the system is left **75% +idle** even at 60 clients on a 64-hardware-context Niagara II; scheduler +overload alone leaves it **20% idle** (§1.1, Fig. 2). Idle, not busy — that is +the signature of a waiting bottleneck rather than a computational one. -| Bottleneck | Fix | Modern descendant | -|---|---|---| -| B: fsync per commit | group commit | postgres `XLogFlush` recheck | -| C: locks held across flush | **Early Lock Release** (ELR) | controversial; see Q2 | -| D: scheduling | flush pipelining (async commit queues) | redis everysec (cruder) | -| A: buffer insert mutex | **consolidation array** | postgres reserve-then-copy | - -### Step 3 — fix B: group commit — one fsync covers N commits - -Group commit attacks the fsync count by noticing that one disk flush makes -durable *every* log record written before it, not just yours. So instead of -one fsync per commit, transactions that arrive while a flush is in progress -simply wait for the *next* flush, which covers all of them: at 1 ms per -fsync and 32 waiting committers, that's 32 commits per fsync ⇒ ~32K -commits/s through the same disk. Every serious engine does this; postgres's -version (recheck the flushed-LSN after acquiring the write lock — most -backends find their work already done) is dissected in -reading-postgres-xlog.md §3. Group commit fixes the *throughput* of [B] but -leaves latency (you still wait ~1 fsync), locks held ([C]) and the insert -mutex ([A]) untouched. - -### Step 4 — fix C: Early Lock Release — stop holding locks through the flush - -ELR releases a transaction's locks at commit-record *creation* (the record -is in the log buffer, ordered) rather than commit-record *durability* (the -record is on disk) — so the ~1 ms flush wait no longer blocks every -transaction queued on those locks. The safety argument is elegant: a -dependent transaction that read your uncommitted-but-logged data cannot -*acknowledge* before you, because its commit record sits **after** yours in -the serial log — a crash that loses your commit necessarily loses theirs -too. The serial log, the thing that seemed like pure bottleneck, doubles as -a free dependency tracker. The catch (question 2): the argument only covers -effects that escape through the log — a read-only transaction that never -writes a commit record can leak unflushed state to a user. Real systems -mostly didn't ship ELR. - -### Step 5 — fix D: flush pipelining — the thread leaves, the commit stays - -Flush pipelining decouples the *worker thread* from the *commit wait*: -instead of blocking on fsync ([D]'s context switches), the worker enqueues -the commit, detaches, and immediately picks up new work; a background daemon -acknowledges each client after the flush covering its commit lands. -Throughput of asynchronous commit, durability of synchronous commit — the -cost is added ack latency and a more complex scheduler, **not** a loss -window. Contrast redis's `appendfsync everysec` (reading-redis-aof-rdb.md), -which acks *before* durability and accepts up to ~1 s of loss — pipelining -is the same "don't block the worker" instinct with the contract kept intact. - -### Step 6 — fix A: consolidation arrays — combine before you contend - -With B, C, D fixed, the remaining wall is the log-buffer mutex itself: -every append still serializes, memcpy included. The insight: even with -group commit, the *insertions* contend one at a time. Fix: threads combine -their requests *before* touching the lock. +| Bottleneck | Aether's fix | Section | Modern descendant | +|---|---|---|---| +| (a) one I/O per commit | group commit (assumed, not the paper's contribution) | — | postgres `XLogFlush`'s three early exits | +| (b) locks held across the flush | **Early Lock Release** | §3 | shipped almost nowhere — see Step 4 | +| (c) scheduler overload | **flush pipelining** | §4 | postgres `synchronous_commit=off`; redis `everysec` (cruder — see Step 5) | +| (d) log-buffer contention | **consolidation array** (§5.1) and **decoupled buffer fill** (§5.2) | §5 | postgres reserve-then-copy = §5.2, *not* §5.1 | + +*Why it matters:* the taxonomy is the paper's most reusable output. When your +own commit path is slow, the first question is which of the four letters it is, +because the four fixes are independent and three of them are cheap. + +### Step 3 — fix (a): group commit — one flush covers N commits + +> **In:** a stream of committing transactions arriving at rate λ, and a +> durability call costing T. +> **Out:** λ·T commits riding each call, so the device stops being the ceiling — +> while each commit still waits about T. + +Group commit attacks the flush *count* by noticing that one durability call +makes durable *every* log record written before it, not just yours. Transactions +that arrive while a flush is in progress simply wait for the next one, which +covers all of them. Every serious engine does this; the paper takes it as given +and does not claim it. + +**Do the arithmetic on real numbers**, because "an fsync costs about a +millisecond" is exactly the kind of unanchored figure that makes this reasoning +useless. Batch size is `λ·T`; throughput with group commit is `λ` (the flush is +no longer the constraint); throughput without it is `1/T`. ``` - naive: T1 ─lock─ memcpy ─unlock─ T2 ─lock─ memcpy ─unlock─ T3 … - consolidated: - T1,T2,T3 meet in a slot array, add up sizes (CAS, no lock), - ONE of them acquires the lock, reserves sum(bytes) once, - each thread memcpys into its own slice IN PARALLEL. +Top rung — macOS F_FULLFSYNC, T = 2.967 ms + offered λ batch = λ·T ceiling without group commit + 1 000/s 2.97 337/s + 5 000/s 14.84 337/s + 20 000/s 59.34 337/s + 100 000/s 296.70 337/s + +Middle rung — fdatasync / macOS fsync, T = 22.67 µs + 100 000/s 2.27 44 109/s + +The old claim in this file was "1 ms per fsync and 32 waiting committers +⇒ ~32K commits/s". Neither number came from anywhere. The honest local +form of the same sentence: + 32 committers riding one F_FULLFSYNC = 32 × 337 = 10 784 commits/s + 32 committers riding one fdatasync = 32 × 44109 = 1 411 488 commits/s ``` -The principle: decouple *sequencing* (assigning log offsets — must be -serial, so make it tiny: one addition) from *copying* (moving the bytes — -needn't be serial, so make it parallel). Postgres's -`ReserveXLogInsertLocation` (a spinlock held for 3 arithmetic ops) + 8 -parallel insertion locks is this idea in production — read -reading-postgres-xlog.md §2 side by side with the paper's §5. The slot -dance, in code: - -```rust -// Combine appends BEFORE the lock; only sequencing stays serial. -fn append(&self, rec: &[u8]) -> Lsn { - let slot = self.slots.join(); // CAS onto an open slot - let my_off = slot.size.fetch_add(rec.len()); // add my bytes — no lock - if my_off == 0 { // first in = group leader - let total = slot.close(); // no more joiners - let base = { - let _g = self.buffer_lock.lock(); // tiny critical section: - self.reserve(total) // ONE reservation for all - }; - slot.publish(base); - } - let base = slot.wait_for_base(); - self.buf_write(base + my_off, rec); // everyone copies IN PARALLEL - Lsn(base + my_off) -} +Read what the table says: the batch grows with offered load *by itself*, which +is why group commit is stable and needs no tuning. What it does **not** fix is +latency — each commit still waits about `T`, so on the top rung every commit +still eats 2.97 ms. That residual latency, held across your locks, is +bottleneck (b); held by a blocked thread, it is bottleneck (c). Postgres's +version of group commit — recheck the flushed LSN after acquiring the write +lock, so most backends find their work already done — is dissected in +`reading-postgres-xlog.md` Step 4. + +*Why it matters:* group commit converts a per-commit cost into a per-batch cost, +which is why every remaining fix in this paper is about *latency* and +*contention* rather than device throughput. + +### Step 4 — fix (b): Early Lock Release — stop holding locks through the flush + +> **In:** a transaction whose commit record is in the log buffer but not yet on +> disk, still holding all its database locks. +> **Out:** the locks, released immediately — with recoverability preserved by +> the log's own total order rather than by waiting. + +ELR releases a transaction's locks at commit-record *creation* rather than +commit-record *durability*, so the flush wait no longer blocks every transaction +queued on those locks. §3.1 attributes the observation to DeWitt et al. [4] and +states it with its caveat attached: a transaction's locks can be released before +its commit record is written to disk, **"as long as it does not return results +to the client before becoming durable."** + +The safety argument is elegant, and it is the reason the serial log — the thing +that looked like pure bottleneck — pays for itself: "Serial log implementations +preserve this property naturally, because the dependant transaction's log +records must always reach the log later than those of the pre-committed +transaction and will therefore become durable later also." A crash that loses +your commit necessarily loses theirs. + +§3.1 gives the formal conditions, from [21] — both must hold: + +1. "Every dependant transaction's commit log record is written to the disk after + the corresponding log record of pre-committed transaction." +2. "When a pre-committed transaction is aborted all dependant transactions must + also be aborted." The paper notes most systems meet this trivially, because + they "do no work after inserting the commit record, except to release locks." + +The catch is condition 1 read carefully: it only covers effects that escape +through *the log*. A read-only transaction writes no commit record, so it has no +place in the total order and can leak unflushed state to a user. + +**Why nobody shipped it.** The paper's own answer (§3.1) is better than the +vague one this chapter used to give: "modern database engines do not implement +ELR and to our knowledge this is the first paper to analyze empirically ELR's +performance. We hypothesize that this is largely due to the effectiveness of +asynchronous commit, which obviates ELR and which nearly all major systems do +provide." In other words the industry bought the same latency win by *giving up +durability* (postgres's `synchronous_commit=off`, redis's `everysec`) rather +than by reasoning about log order. + +**What it is worth** (§3.2, Fig. 3, TPC-B on the 64-context Niagara II, zipfian +skew on the x-axis): ELR's speedup is "maximized (35x) for slower devices, but +remains substantial (2x) even with flash drives if contention is present." §1.2 +gives the headline as **15%–164%** "even when logging to fast flash disks". The +35× figure is not a general claim about ELR — it is the high-skew, 10 ms-device +corner of one figure, and quoting it without both qualifiers is exactly the +error this chapter exists to avoid. + +*Why it matters:* ELR is the topic's cleanest example of a serialization +constraint doubling as a correctness proof. Even if you never implement it, the +argument — "the log order already encodes the dependency, so I do not need to +wait to know about it" — recurs everywhere. + +### Step 5 — fix (c): flush pipelining — the thread leaves, the commit stays + +> **In:** a worker thread that is about to block on a durability call. +> **Out:** the same thread, immediately running the next transaction; a daemon +> that acks each client after the flush covering its commit lands — with the +> durability contract intact. + +Flush pipelining (§4.1) decouples the *worker thread* from the *commit wait*: +instead of blocking, the worker detaches the transaction state, enqueues it, and +picks up new work; a daemon acknowledges each client after the flush covering +its commit record completes. Throughput of asynchronous commit, durability of +synchronous commit — the cost is added ack latency and a more complex scheduler, +**not** a loss window. + +The measurements (§4.2): the baseline leaves **12 of 64** hardware contexts idle +at peak; flush pipelining reaches all **64** (Fig. 4), and delivers "up to 22% +higher performance" (Fig. 5). + +Contrast redis's `appendfsync everysec` (`reading-redis-aof-rdb.md`): same +"don't block the worker" instinct, but it acks *before* durability and accepts +roughly a second of loss. Postgres's `synchronous_commit=off` is the same trade. +Flush pipelining is the version that keeps the contract — which is precisely why +it is more complicated. + +**And it is the fix that mattered most.** §6.4, Fig. 9 (Shore-MT running TATP's +`UpdateLocation`): "For systems today, flush pipelining provides the largest +single performance boost, **68% higher than the baseline**. The scalable log +buffer adds a modest **7%** further speedup by eliminating log contention." +Note also the dependency: "flush pipelining depends on ELR to prevent +log-induced lock contention which would otherwise limit scalability" — which is +why Fig. 9's middle curve is labelled *FlushPipelining + ELR*, not flush +pipelining alone. + +*Why it matters:* this inverts the usual reading of the paper. The famous idea +is the consolidation array; the idea that bought the throughput was scheduler +relief. + +### Step 6 — fix (d): two different ways to stop contending on the log buffer + +> **In:** N threads that each want to append a small record to one shared +> buffer, currently taking one mutex each and memcpying inside it. +> **Out:** two orthogonal designs — one that reduces *how many* threads enter +> the critical section, one that shortens *how long* each stays — and a hybrid. + +With (a), (b) and (c) fixed, the remaining wall is the log buffer itself. §5 +splits the work into two phases and observes that only the first is inherently +serial: + +- **buffer acquire** — reserve a byte range, which assigns the LSN. Serial. +- **buffer fill** — copy the record in. "Buffer fill operations are not + inherently serial (records never overlap)" (§5.2). + +Two independent attacks follow. The paper's Figure 6 labels them, and this +chapter uses those labels: + +``` + (B) Baseline: T1 ─lock─ memcpy ─unlock─ T2 ─lock─ memcpy ─unlock─ T3 … + + (C) Consolidation array (§5.1) — fewer threads enter the critical section + T1,T2,T3 meet in a slot, sum their sizes with CAS (no mutex), + ONE of them takes the mutex and reserves sum(bytes) once, + all three fill their own slices in parallel. + "…effectively bounding contention at the log buffer to the number of + array entries protecting the log buffer, rather than the number of + threads in the system." (§5.1) + Residual cost: groups are still serialized against each other. + + (D) Decoupled buffer fill (§5.2) — the critical section gets shorter + Every thread takes the mutex, reserves its own range, and RELEASES THE + MUTEX IMMEDIATELY; the memcpy happens outside. Fills pipeline. + Cost: buffer *release* becomes a second serialization point, because + regions must be released in LSN order — "recovery must stop at the + first gap it encounters". No mutex needed, but each thread waits for + its predecessor to release. + + (CD) Hybrid (§5.3) — both; bounded contention AND maximum pipelining. ``` -This is the fix that shipped everywhere, because it attacks the only -bottleneck that *scales with core count* — [B] is constant per disk, but -[A] gets worse with every core you add. +The underlying principle is one sentence: decouple *sequencing* (assigning log +offsets — must be serial, so make it tiny) from *copying* (moving bytes — +needn't be serial, so make it parallel). + +**Which one is postgres?** §5.2, not §5.1 — and this chapter used to say the +opposite. `ReserveXLogInsertLocation` (`xlog.c:1172–1184`) holds a spinlock for +one addition and four field moves, then releases it and copies outside; the +format conversions happen at `:1182–1184`, deliberately after the release. That +is decoupled buffer fill exactly. Postgres's answer to §5.2's release-in-order +requirement is `WaitXLogInsertionsToFinish`, which is why every insertion lock +publishes an `insertingAt` value. + +The 8 WAL insertion locks (`NUM_XLOGINSERT_LOCKS` = 8, `xlog.c:157`) are *not* a +consolidation array. Compare how a thread finds its slot: + +| | Aether's consolidation array | postgres's insertion locks | +|---|---|---| +| how you pick a slot | `idx = randn(ARRAY_SIZE)` — probe at random (Algorithm 5 line 3, Appendix A.2) | `MyProcNumber % NUM_XLOGINSERT_LOCKS` on first use, then reuse the same one for cache affinity (`xlog.c:1429–1431`) | +| on contention | join whatever OPEN slot you find; state machine FREE→OPEN→PENDING→COPYING→DONE (§A.2) | move to the next lock, `lockToTry = (lockToTry + 1) % 8` (`xlog.c:1448`), so inserters migrate apart | +| what it bounds | the number of threads reaching the mutex, to the array size | nothing — it partitions the waiting, it does not combine requests | +| requests combined? | **yes** — one reservation serves the whole group, "two or three atomic operations per participating thread" (§A.2) | **no** — every backend makes its own reservation | +| how many | peak performance at **3–4 slots** (§A.4, Fig. 12); the paper fixes it at four | 8, fixed | + +Postgres got §5.2 and a lock-partitioning scheme; it did not get §5.1. + +**What the log buffer is worth, in the paper's own numbers.** §6.3.1: the +average record in their workloads is about **120 B**, and "a high-performance +application generates between 100 and 200MBps of log, or between 800K and 1.6M +log insertions per second" — check the division, 100 MB/s ÷ 120 B = 833 K/s. The +baseline log buffer peaks at roughly **140 MB/s** and then *falls* as contention +grows. The abstract's headline is over **1.8 GB/s** for small records — an order +of magnitude past the baseline. §6.3.2, Fig. 8(right): (C) wins below ~1 kB +records where contention dominates, (D) wins above it where copy cost does, and +the hybrid beats both across the range until all three saturate the memory +system; with the records kept L1-resident the hybrid scales to about **21 GB/s** +before becoming CPU-limited. + +*Why it matters:* this is the fix that attacks the only bottleneck that *grows +with core count* — (a) is constant per device, (c) is bounded by the scheduler, +but (d) gets worse with every core you add. §6.4's own conclusion is that it is +worth only 7% today and that "this bottleneck is growing rapidly with core +counts and will soon dominate." ## How to read the paper (with the concepts in hand) -1. §1–2 for the bottleneck taxonomy (Step 2's table, in the authors' words). -2. §5 consolidation arrays (Step 6) — the durable contribution. -3. §3 ELR (Step 4) — for the *argument* about log order as dependency - tracking. -4. Skim §4 (flush pipelining, Step 5) + evaluation (§6): note which fix buys - what at which core count. +1. **Abstract and §1.1** for the bottleneck taxonomy in the authors' letters + (Step 2's table). Fig. 1 is the map; Fig. 2 is the evidence. +2. **§6.4 and Fig. 9 next, out of order** — 68% from flush pipelining, 7% from + the log buffer. Knowing the scoreboard before you read the mechanisms stops + you from over-weighting the clever one. +3. **§3** — ELR (Step 4). Read it for the *argument*: log order as a free + dependency tracker, plus the two formal conditions and the + "does not return results to the client" caveat. +4. **§4** — flush pipelining (Step 5). Short, and the fix that mattered. +5. **§5** — the log buffer. Read §5.1 and §5.2 as *two* designs, and hold + Figure 6's four panels (B, C, D, CD) in view; then read + `reading-postgres-xlog.md` Step 2 beside §5.2, not §5.1. +6. **Appendix A.2 and A.4** if you intend to build one: the slot state machine, + Algorithm 5, and the finding that 3–4 slots is the peak. ## Questions to answer in notes.md -1. Why does ELR NOT violate durability for the *dependent* transaction? - (Its commit record is behind yours; a crash that loses yours loses its too.) -2. ELR hazard: what if the dependent txn's result escapes to the user by a - channel other than its own commit ack (e.g. a read-only txn that never - logs)? This is why real systems mostly didn't ship it. -3. Consolidation vs postgres's 8 insert locks: both parallelize the copy — - what's the difference in HOW threads find a slot? (CAS-combining into a - shared slot vs hashing onto a fixed lock array; contrast under 8 vs 80 - writers.) -4. Which bottleneck does your M5 group-commit design leave unfixed? (Likely A - — a single mutex around the WAL buffer is fine at graph-workload commit - rates; say at what commits/s it wouldn't be.) +1. Why does ELR NOT violate durability for the *dependent* transaction? State + the paper's two formal conditions (§3.1) and say which one the serial log + satisfies for free. +2. ELR hazard: what if the dependent transaction's result escapes to the user by + a channel other than its own commit ack — say a read-only transaction that + never logs? Relate this to the paper's caveat, "as long as it does not return + results to the client before becoming durable." +3. Consolidation array (§5.1) versus postgres's 8 insertion locks: both reduce + time spent in the critical section, but only one *combines* requests. Work + out what each does under 8 writers and under 80, and explain why the paper + found 3–4 slots optimal (§A.4) while postgres uses 8 locks. +4. Redo Step 3's group-commit table for the durability rung *you* intend to ship + on. At what offered load does the batch first exceed 10? What does that imply + about when group commit is worth implementing at all? +5. Which bottleneck does your M5 group-commit design leave unfixed? (Likely (d) + — a single mutex around the WAL buffer is fine at graph-workload commit rates; + say at what commits/s it wouldn't be, using Step 6's 120 B / 800 K–1.6 M + inserts-per-second figures as the yardstick.) ## Done when -You can name the four bottlenecks from memory, sketch a consolidation array, -and point at the postgres code that embodies it. +Answer each before unfolding it. + +- [ ] Name the four bottlenecks in the paper's own lettering, and say what kind + of resource each one is. + +
Answer + + (a) high volume of small I/O requests saturating the disk — a *device*; (b) + transactions holding locks while waiting for the log flush — a *lock table*; + (c) context switching overwhelming the OS scheduler — a *scheduler*; (d) + contention serializing access to in-memory log data structures — a *mutex*. + Straight from the abstract. Four different resources is the point: the fixes + are independent and compose. + +
+ +- [ ] Which of Aether's fixes bought the most throughput, and by how much? + +
Answer + + Flush pipelining — "the largest single performance boost, 68% higher than the + baseline" (§6.4, Fig. 9), with the scalable log buffer adding "a modest 7% + further speedup". Two caveats: that is on Shore-MT running TATP + `UpdateLocation` on a 64-context Niagara II, and flush pipelining "depends on + ELR to prevent log-induced lock contention", so the 68% curve is + FlushPipelining + ELR. The paper expects the ranking to invert as core counts + rise. + +
+ +- [ ] Sketch a consolidation array, and say what it bounds. + +
Answer + + Contending threads back off to an array of slots, join one at random + (`idx = randn(ARRAY_SIZE)`, Algorithm 5), and CAS their sizes together; the + first thread in acquires the mutex once and reserves the *group's* total, then + every member fills its own slice in parallel and the last one out releases the + region. It bounds "contention at the log buffer to the number of array entries + protecting the log buffer, rather than the number of threads in the system" + (§5.1) — a constant instead of something that grows with core count. Peak at + 3–4 slots (§A.4). + +
+ +- [ ] Point at the postgres code that embodies Aether's log-buffer work — and + name the right section. + +
Answer + + `ReserveXLogInsertLocation` (`xlog.c:1172–1184`): a spinlock held for one + addition and four field moves, with the conversions and the record copy done + after the release. That is **§5.2, decoupled buffer fill**, not §5.1's + consolidation array — postgres does not combine requests. Its answer to + §5.2's release-in-LSN-order requirement is `WaitXLogInsertionsToFinish` plus + the per-lock `insertingAt` values. The 8 insertion locks + (`NUM_XLOGINSERT_LOCKS`, `xlog.c:157`) partition waiting via + `MyProcNumber % 8` with migration on contention (`xlog.c:1429–1448`); they are + a lock array, not a consolidation array. + +
+ +- [ ] Group commit converts a per-commit cost into a per-batch cost. On this + machine's top rung, how many transactions ride one flush at 20 000 + commits/s offered, and what is the ceiling without group commit? + +
Answer + + Batch = λ·T = 20 000 × 2.967 ms = **59.3** transactions per flush. Without + group commit the ceiling is 1/T = **337 commits/s** flat, regardless of + offered load. Note what group commit does not fix: each commit still waits + about 2.97 ms, and that residual wait is bottlenecks (b) and (c). + +
+ +- [ ] Why must the log buffer's *release* be serialized even after §5.2 removes + the mutex from the fill? + +
Answer + + Because regions must be released in LSN order: "Log records must be written to + disk in LSN order because recovery must stop at the first gap it encounters; + in the event of a crash any committed transactions beyond a gap would be lost" + (§5.2). No mutex is required, but each thread must wait for its predecessor to + release before releasing its own region — which is why postgres publishes an + `insertingAt` value per insertion lock and has `WaitXLogInsertionsToFinish`. + +
## References -**Papers** -- Johnson, Pandis, Stoica, Athanassoulis, Ailamaki — "Aether: A Scalable - Approach to Logging" (VLDB 2010) — ~12 pages; §1–2 for the bottleneck - taxonomy, §5 (consolidation arrays) is the durable contribution, §3 for - the ELR argument, skim §4 and the evaluation +**Paper** — Johnson, Pandis, Stoica, Athanassoulis & Ailamaki, "Aether: A +Scalable Approach to Logging", *Proceedings of the VLDB Endowment* 3(1), 2010, +pp. 681–692. + +| section / figure | what this chapter took from it | +|---|---| +| Abstract | the (a)–(d) lettering; 20–69% end-to-end; >1.8 GB/s log insert (Steps 2, 6) | +| §1.1, Figs. 1–2 | 75% idle from lock contention, 20% from scheduler overload (Step 2) | +| §1.2 | ELR worth 15%–164% on fast flash (Step 4) | +| §3.1 | ELR's definition, the DeWitt attribution [4], the client-results caveat, the two conditions from [21], and why nobody shipped it (Step 4) | +| §3.2, Fig. 3 | 35× on slow devices, 2× on flash; the 0/100 µs/1 ms/10 ms device series (Steps 4, and the ladder above) | +| §4.1–§4.2, Figs. 4–5 | flush pipelining; 12-of-64 idle contexts → 64; up to 22% (Step 5) | +| §5.1, §A.2, Algorithm 5 | the consolidation array; contention bounded to the array size; random slot probing (Step 6) | +| §5.2 | decoupled buffer fill; release-in-LSN-order; "recovery must stop at the first gap" (Steps 1, 6) | +| §5.3, Fig. 6 | the (B)/(C)/(D)/(CD) panels and the hybrid (Step 6) | +| §6.1 | platform and method — Sun T5220, Solaris 10, TATP 100K subscribers, TPC-B 100 tellers, ten 30 s runs, all within 2% | +| §6.3.1–§6.3.2, Fig. 8 | 120 B average record, 800 K–1.6 M inserts/s, 140 MB/s baseline, ~21 GB/s in L1 (Step 6) | +| §6.4, Fig. 9 | 68% from flush pipelining, 7% from the log buffer (Steps 5, 6) | +| §A.4, Fig. 12 | peak at 3–4 slots; the paper fixes the array at four (Step 6) | + +**Code** — postgres/postgres@701f021: `src/backend/access/transam/xlog.c` +(`ReserveXLogInsertLocation` `:1172–1184`, `NUM_XLOGINSERT_LOCKS` `:157`, +`WALInsertLockAcquire` `:1411–1450`). Read alongside +`reading-postgres-xlog.md`. + +**Measurements** — `topics/05-durability-wal/notes.md`, "Baseline (provided lane, +Apple M3 Pro / APFS, measured 2026-07-28)", from +`experiments/src/bin/fsync_ladder.rs`; headline in `FINDINGS.md` row 5. diff --git a/topics/05-durability-wal/reading-aries.md b/topics/05-durability-wal/reading-aries.md index 2af64c7..3145c1f 100644 --- a/topics/05-durability-wal/reading-aries.md +++ b/topics/05-durability-wal/reading-aries.md @@ -2,192 +2,618 @@ Postgres escapes undo via MVCC, SQLite-WAL escapes redo via page images, LMDB escapes logging via COW — ARIES is the recovery method for engines that escape -*nothing*: update-in-place, steal, no-force. It is the most-cited recovery -paper and the vocabulary every other design in this topic is defined against; -reading it tells you exactly what each escape hatch is worth. Before the -70 pages, this chapter builds the machine step by step: the two buffer -policies that create the problem, the LSN discipline that makes replay safe, -the three recovery passes, and the CLR trick that lets recovery itself crash. +*nothing*: update-in-place, steal, no-force. It is the most-cited recovery paper +and the vocabulary every other design in this topic is defined against; reading +it tells you exactly what each escape hatch is worth. Before the 70 pages, this +chapter builds the machine step by step: the two buffer policies that create the +problem, the LSN discipline that makes replay safe, the three recovery passes, +the CLR trick that lets recovery itself crash — and then runs all three passes +by hand over an eight-record log so the rules stop being abstract. + +Every section number, figure number and quotation below was checked against the +paper as published: **Mohan, Haderle, Lindsay, Pirahesh & Schwarz, "ARIES: A +Transaction Recovery Method Supporting Fine-Granularity Locking and Partial +Rollbacks Using Write-Ahead Logging", ACM TODS 17(1), March 1992, pp. 94–162.** +Where this chapter uses a term the paper does not (`ATT`, `DPT`, +"physiological"), it says so. + +## Vocabulary, defined once, before it is used + +| Term | Meaning | Where | +|---|---|---| +| **WAL** (write-ahead logging) | a log record describing a change is made durable *before* the changed page may reach nonvolatile storage | §1 | +| **steal** | the buffer manager may write a dirty page to disk *before* its transaction commits ⇒ you owe **undo** | §2, from Haerder & Reuter [36] | +| **no-force** | commit does *not* require the transaction's data pages to reach disk, only its log ⇒ you owe **redo** | §2, [36] | +| **LSN** (log sequence number) | "the address of the first byte of the log record in the ever-growing log address space… monotonically increasing" | §4.1 | +| **page-LSN** | the LSN of the most recent log record applied to a page, stored *in* the page | §4.2 | +| **PrevLSN** | in every log record: the LSN of the same transaction's preceding record — a backward chain per transaction | §4.1 | +| **UndoNxtLSN** | present *only* in CLRs: "the value of PrevLSN of the log record that the current log record is compensating" | §4.1 | +| **CLR** (compensation log record) | the log record describing an undo action; redo-only, never itself undone | §1.1, §3 | +| **idempotent redo** | replay that is safe to repeat, because `page-LSN ≥ record.LSN` proves the change is already on the page | §6.2, Fig. 11 | +| **checkpoint** | a periodic pair of log records that bounds how far back restart must read | §5.4 | +| **fuzzy checkpoint** | a checkpoint that stops nothing and forces no dirty page to disk — it only writes out two small tables | §5.4 | +| **page-oriented redo** | redo touches only the page named in the record; no index retraversal, no other page | §1.1 | +| **logical undo** | undo may act on a *different* page than the original update, so another transaction can move an uncommitted record | §1.1 | +| **physiological logging** | Gray & Reuter's later name for exactly the above pairing — physical to a page, logical within it. **The ARIES paper never uses this word**; its terms are the two above | — | +| **transaction table** | the paper's name for what textbooks call the **ATT**: TransID, State ('P'/'U'), LastLSN, UndoNxtLSN | §4.3 | +| **dirty_pages table** | the paper's name for what textbooks call the **DPT**: PageID, RecLSN | §4.4 | +| **group commit** | letting one durability call cover many transactions' commit records | not in this paper — see `reading-postgres-xlog.md` | + +**And the durability call itself.** ARIES's one hard I/O requirement at commit is +that the log be forced through the commit record. What "forced" costs is this +topic's ladder, measured by `experiments/src/bin/fsync_ladder.rs` on the machine +in `notes.md`: `write()` alone **1.17 µs** (page cache only — survives `kill -9`, +not power loss); `fdatasync()` / macOS `fsync()` **22.67 µs** (handed to the +drive, whose volatile cache may still hold it); macOS `fcntl(fd, F_FULLFSYNC)` +**2.97 ms** (the drive flushed its cache). 856 898 → 44 109 → 337 implied +durable commits/s: **19.4×** then a further **131×**. The middle rung was +measured on macOS as `fsync(2)` — there is no `fdatasync` on this machine — and +`fdatasync` is named only because it occupies the same rung on Linux. Every time +this chapter says "force the log", that is the price, and which rung you are on +decides whether ARIES's commit path costs microseconds or milliseconds. ## The problem in one sentence -An update-in-place engine that lets dirty pages reach disk *before* commit -and doesn't force them to disk *at* commit can crash into a state where the -disk holds half of transaction A's writes and none of transaction B's -committed ones — and recovery must reconstruct exactly which is which from -nothing but an append-only log, even if it crashes again halfway through -doing so. +An update-in-place engine that lets dirty pages reach disk *before* commit and +doesn't force them to disk *at* commit can crash into a state where the disk +holds half of transaction A's writes and none of transaction B's committed ones +— and recovery must reconstruct exactly which is which from nothing but an +append-only log, even if it crashes again halfway through doing so. ## The concepts, step by step ### Step 1 — steal and no-force: two freedoms, two debts -A buffer manager (the component caching disk pages in RAM) faces two policy -questions, and each "convenient" answer creates a recovery obligation. -**Steal** = the cache may evict a dirty page to disk *before* its -transaction commits (freedom: evict whatever page is coldest; debt: the disk -now holds uncommitted data, so after a crash you need **undo** — the ability -to reverse it). **No-force** = commit does *not* require writing the -transaction's pages to disk, only its log records (freedom: commit costs one -sequential log flush, not N random page writes; debt: the disk may lack -committed data, so you need **redo** — the ability to re-apply it). The -2×2 matrix: +> **In:** a buffer manager (the component caching disk pages in RAM) and two +> policy questions about when its pages may or must be written. +> **Out:** a 2×2 matrix in which each convenient answer names a recovery pass +> you now owe. + +**Steal** = the cache may evict a dirty page to disk *before* its transaction +commits. Freedom: evict whatever page is coldest. Debt: the disk now holds +uncommitted data, so after a crash you need **undo** — the ability to reverse +it. **No-force** = commit does *not* require writing the transaction's pages to +disk, only its log records. Freedom: commit costs one sequential log force, not +N random page writes. Debt: the disk may lack committed data, so you need +**redo**. | | force (pages flushed at commit) | no-force | |---|---|---| | **no-steal** | no undo, no redo — but hopeless perf | redo only (your likely M5 design) | | **steal** | undo only | **undo + redo — ARIES's territory** | -High-performance update-in-place engines (InnoDB, SQL Server, Db2) all -choose steal + no-force — both freedoms, both debts. ARIES is how you pay. +High-performance update-in-place engines (InnoDB, SQL Server, Db2) all choose +steal + no-force — both freedoms, both debts. ARIES is how you pay. + +The paper's argument for steal is stronger than the usual "you might run out of +buffer space", and worth having: under fine-granularity (record-level) locking +with overlapping transactions, **"with a no-steal policy, a page may never get +written to nonvolatile storage if the page always contains uncommitted updates"** +(§2). No-steal is not merely inconvenient at high concurrency; it can be +*unsatisfiable*. This is the hinge that makes ARIES's whole apparatus necessary +rather than optional, and it is worth stating precisely because a redo-only +engine (M5, turso) buys its simplicity by refusing record-level locking on +shared pages. + +*Why it matters:* every escape hatch in this topic is a cell in this matrix. +Naming your cell tells you which passes you must write. ### Step 2 — the LSN: one number that orders everything -The **LSN** (log sequence number) is a log record's byte offset in the log — -monotonically increasing, so it doubles as a global timestamp for every -change in the system. The discipline that makes everything else work: every -page on disk carries the LSN of the last log record applied to it -(**pageLSN**). Now "has this page already seen this update?" is one integer -comparison — `pageLSN ≥ record.LSN ⇒ yes, skip` — and replaying the log -becomes **idempotent** (safe to repeat: applying a record twice is -impossible because the first application raised the pageLSN). Each record -also carries its transaction's previous record's LSN (**prevLSN**), chaining -every transaction's history backward through the log for undo to walk. +> **In:** a log record about to be written, and the page it changes. +> **Out:** a monotonically increasing identifier that lets a single integer +> comparison answer "has this page already seen this update?" + +The **LSN** is defined by §4.1 as "the address of the first byte of the log +record in the ever-growing log address space" and is therefore "monotonically +increasing" — a global timestamp for every change in the system, obtained for +free from the log's own geometry. The discipline that makes everything else +work: every page on disk carries the LSN of the last log record applied to it +(§4.2, the **page-LSN**). Now "has this page already seen this update?" is one +comparison — `page-LSN ≥ record.LSN ⇒ yes, skip` — and replaying the log becomes +**idempotent**: applying a record twice is impossible, because the first +application raised the page-LSN past it. + +Two backward pointers complete the structure: + +- **PrevLSN**, in every record, chains a transaction's own history backward + through the log, so undo can walk one transaction without scanning. §4.1 + notes in a footnote that AS/400, Encompass and NonStop SQL *don't* link a + transaction's records, "which makes undo inefficient since a sequential + backward scan of the log must be performed." +- **UndoNxtLSN**, present *only* in CLRs (§4.1), holds "the value of PrevLSN of + the log record that the current log record is compensating" — it is zero when + nothing remains. This one field is the whole of Step 6. + +The paper's own name is `UndoNxtLSN`; earlier versions of this chapter called it +`undoNext`, which appears nowhere in the paper. + +Two more of §1.1's definitions matter, because they explain what the LSN buys. +**Page-oriented redo** means "the log record whose update is being redone +describes which page of the database was originally modified… and the same page +is modified during the redo processing… no other page of the database needs to +be examined" — so redo is one page fetch and one comparison, and pages recover +independently. **Logical undo** is the opposite freedom: undo may act on a +different page, which is what "permit[s] uncommitted updates of one transaction +to be moved to a different page by another transaction." §1.1 states the +trade-off in one line: "In the interest of efficiency, ARIES supports +page-oriented redo and it supports, in the interest of high concurrency, logical +undos." + +*Why it matters:* this is the single idea the rest of the topic borrows. Postgres +stamps pages with LSNs for the same reason; turso's frames are idempotent for a +weaker version of the same reason (a whole page image needs no comparison at +all). ### Step 3 — fuzzy checkpoints: bounding how far back recovery reaches -A checkpoint is a periodic log record that lets recovery start from -somewhere later than the beginning of time. Stopping the system to flush -everything would be a latency crater, so ARIES checkpoints **fuzzily** — -without stopping anything, it just snapshots two small tables into the log: -the **DPT** (dirty page table: which cached pages have unflushed changes, -and the LSN of the *earliest* change each might be missing on disk — its -recLSN) and the **ATT** (active transaction table: which transactions are -in flight, and their last LSN). Cost: a few KB written, zero pause. The DPT -tells redo where to start reading (the minimum recLSN); the ATT tells undo -who its candidates are. +> **In:** a running system with dirty pages and in-flight transactions, and a +> log that would otherwise have to be read from the beginning of time. +> **Out:** two small tables written into the log, and a master record pointing +> at them — with **no pause and no page flush**. + +Stopping the system to flush everything would be a latency crater, so ARIES +checkpoints **fuzzily** (§5.4): it writes a `begin_chkpt` record, then an +`end_chkpt` record carrying the transaction table, the buffer pool's dirty_pages +table, and the file mapping; then the **master record** on disk is updated to +hold the `begin_chkpt` record's LSN. That master record is where restart begins. + +The sentence to carry away is §5.4's: **"ARIES does not require that any dirty +pages be forced to nonvolatile storage during a checkpoint."** Cost: a few KB +written, zero pause, zero forced page I/O. What you buy is a bound: the +dirty_pages table's minimum RecLSN is where redo must start, and the transaction +table names undo's candidates. + +The **RecLSN** of a page is the LSN of the *earliest* change that page might be +missing on disk — recorded when the page first became dirty. §4.4: "the minimum +RecLSN value in the table gives the starting point for the redo pass." + +*Why it matters:* checkpoint cost and recovery bound are the two halves of one +dial, and ARIES's setting of it — pay nothing now, read a bounded amount later — +is why nobody had to invent a "stop the world" checkpoint again. Compare +postgres's `CreateCheckPoint` (`xlog.c:7400–7897`), which *does* flush its buffers +in `CheckPointGuts` and therefore pays much more up front. -## Vocabulary (the paper is unreadable without these) +### Step 4 — pass 1, analysis: rebuild the two tables -| Term | Meaning | -|---|---| -| steal | dirty pages may hit disk BEFORE commit (⇒ need undo) | -| no-force | pages need NOT hit disk at commit (⇒ need redo) | -| LSN | log sequence number; every page stamps the LSN of its last change | -| CLR | compensation log record — undo work is itself logged, redo-only | -| DPT | dirty page table (checkpointed) — which pages might need redo | -| ATT | active transaction table (checkpointed) — who needs undo | +> **In:** the master record's `begin_chkpt` LSN, and every log record from there +> to the end of the log. +> **Out:** the transaction table and dirty_pages table as they stood at the +> instant of the crash, a `RedoLSN`, and a list of losers. No data page is +> touched and no log record is written. -### Step 4 — pass 1, analysis: rebuild the two tables +Analysis (§6.1, Fig. 10) opens the log scan at the `begin_chkpt` record and +replays only the *bookkeeping*: + +``` +RESTART_ANALYSIS (paper Fig. 10, condensed) + Trans_Table, Dirty_Pages := empty + open log scan at Master_Rec.ChkptLSN -- the begin_chkpt record + for each record until end of log: + if trans-related and TransID not in Trans_Table: + insert (TransID, 'U', LSN, PrevLSN) + case update | compensation: + Trans_Table[T].LastLSN := LSN + if update and undoable: Trans_Table[T].UndoNxtLSN := LSN + if compensation: Trans_Table[T].UndoNxtLSN := LogRec.UndoNxtLSN + if redoable and PageID not in Dirty_Pages: + insert (PageID, LSN) -- RecLSN := this record's LSN + case End_Chkpt: merge in the checkpointed Trans_Table and Dirty_PagLst + case prepare: State := 'P' case rollback: State := 'U' + case end: delete the Trans_Table entry + for each entry with State='U' and UndoNxtLSN=0: + write an 'end' record and remove it -- rolled back, end record missing + RedoLSN := minimum(Dirty_Pages.RecLSN) +``` + +Whoever remains in the transaction table with State `'U'` is a **loser** — still +running when the world ended. Note the last loop: a transaction that had already +been fully rolled back before the crash (UndoNxtLSN back to 0) but whose `end` +record never made it is quietly finished off here, without any undo work. + +Two subtleties worth reading for. First, `end` is what removes a transaction — +and §5.3 says a transaction "is committed by writing an **end** record and +releasing its locks", so the `end` record *is* the durable commit point (a +separate `prepare` record is only needed for distributed transactions). Second, +the analysis pass is not strictly required: §6.1 observes the tables could be +rebuilt during redo instead, at the cost of starting redo at +`min(min(checkpoint RecLSNs), LSN(begin_chkpt))` — strictly earlier, so strictly +more work. -After a crash, recovery's first pass reads the log forward from the last -checkpoint, replaying only the *bookkeeping*: it rebuilds the DPT and ATT as -they stood at the instant of the crash — pages that got dirtied after the -checkpoint enter the DPT, transactions that committed leave the ATT, and -whoever remains in the ATT at the end is a **loser** (a transaction that -was still running when the world ended). No data is touched; the pass just -answers two questions: *where must redo start* (min recLSN in the DPT) and -*who must undo roll back* (the losers in the ATT). +*Why it matters:* analysis is the pass that costs nothing and saves everything — +it is one sequential read that converts "the whole log" into "these pages from +this LSN" and "these three transactions." ### Step 5 — pass 2, redo: repeat history, even for losers -Redo reads forward from the DPT's minimum recLSN and re-applies every -update whose page hasn't seen it (the Step 2 comparison: apply iff -`pageLSN < record.LSN`) — **including the updates of doomed loser -transactions**. This "repeating history" is the counterintuitive core of -ARIES: the goal of redo is not "restore committed work" but "restore the -*exact* state at the instant of the crash", losers and all. Why: only from -that exact state can undo run as perfectly ordinary transaction rollback — -the same code path as a user typing ROLLBACK — instead of a special -recovery-only mode reasoning about half-restored pages. Redo pays for this -with some wasted work (re-applying updates it will immediately undo); it -buys one rollback mechanism instead of two. +> **In:** `RedoLSN`, the dirty_pages table, and the log from `RedoLSN` forward. +> **Out:** the database's pages restored to their *exact* state at the instant of +> the crash — losers' updates included. No log record is written. + +Redo (§6.2, Fig. 11) re-applies every update whose page hasn't seen it — +**including the updates of doomed loser transactions**. This "repeating history" +is the counterintuitive core of ARIES: the goal of redo is not "restore +committed work" but "restore the exact state at the instant of the crash". Only +from that state can undo run as perfectly ordinary transaction rollback — the +same code path as a user typing ROLLBACK — instead of a recovery-only mode +reasoning about half-restored pages. Redo pays with some wasted work +(re-applying updates it will immediately undo) and buys one rollback mechanism +instead of two. + +**The redo test is three levels deep, not one.** Earlier versions of this chapter +gave only the third: ``` - log: …──[ckpt: DPT+ATT]────────────────────────► crash - 1. ANALYSIS ────────────────► rebuild DPT/ATT from ckpt forward - 2. REDO ────────────────► repeat HISTORY (even losers!) from - min(recLSN in DPT) — page LSN ≥ record LSN ⇒ skip - 3. UNDO ◄──────────────── roll back losers, writing CLRs; - CLR.undoNext skips already-undone work +RESTART_REDO (paper Fig. 11, condensed) + for each record from RedoLSN to end of log: + (1) type is 'update' or 'compensation', and the record is redoable? + (2) PageID IN Dirty_Pages AND LSN >= Dirty_Pages[PageID].RecLSN ? + -- both are table lookups; if either fails, the page is NOT fetched + (3) fetch and X-latch the page: + IF Page.LSN < LogRec.LSN THEN redo it; Page.LSN := LogRec.LSN + ELSE Dirty_Pages[PageID].RecLSN := Page.LSN + 1 ``` +Levels (1) and (2) are pure in-memory filters whose entire purpose is to *avoid +the page fetch* — "the RecLSN information serves to limit the number of pages +which have to be examined" (§6.2). Level (3) is the idempotence test proper. The +`ELSE` branch is the one everybody forgets: when the page turns out to be newer +than the record, the table's RecLSN was stale (the page was written to disk after +the checkpoint but before the failure), so redo *corrects the table* — and every +later record for that page can then be filtered at level (2) instead of level +(3). Step 7 shows this firing. + +*Why it matters:* the difference between one test and three is the difference +between fetching every page named in the log and fetching only the pages that +might actually need work. On a large buffer pool that is most of restart time. + ### Step 6 — pass 3, undo with CLRs: recovery that survives recovery -Undo walks each loser's prevLSN chain newest-first, reversing every update — -and here is the trick that makes ARIES bulletproof: **each undo action is -itself logged**, as a **CLR** (compensation log record). A CLR is redo-only -(it is never undone — undoing an undo would re-apply the original mistake) -and carries an **undoNext** pointer to the record *before* the one just -compensated. Now crash during undo: the next recovery's redo pass replays -the CLRs (restoring the partial rollback — repeating history again), and -undo resumes exactly at the last CLR's undoNext. No update is ever undone -twice, no matter how many times recovery itself crashes. The three passes, -as one function: - -```rust -fn recover(log: &Log, ckpt: &Checkpoint) { - let (dpt, att) = analysis(log, ckpt); // 1. who was dirty, who was active - for rec in log.from(dpt.min_rec_lsn()) { // 2. REDO: repeat history — - if page_lsn(rec.page_id) < rec.lsn { // even losers' updates. - apply_redo(rec); // pageLSN ≥ recLSN ⇒ skip: - } // idempotence by LSN compare - } - for txn in att.losers() { // 3. UNDO: ordinary rollback, - for rec in txn.updates_newest_first() { // but each undo is LOGGED - let clr = log.append_clr(rec); // as a redo-only CLR - clr.undo_next = rec.prev_lsn; // crash mid-undo? restart - apply_undo(rec); // resumes at undo_next — - } // no double-undo, ever - } -} +> **In:** the transaction table's losers and their `UndoNxtLSN` values. +> **Out:** every loser rolled back, a CLR written for each undone record, and a +> log from which a *second* crash can resume without undoing anything twice. + +Undo reverses every loser's updates newest-first — and here is the trick that +makes ARIES bulletproof: **each undo action is itself logged**, as a **CLR**. A +CLR is redo-only (undoing an undo would re-apply the original mistake) and +carries an `UndoNxtLSN` pointing at the record *before* the one just +compensated. Crash during undo, and the next recovery's redo pass replays the +CLRs (restoring the partial rollback — repeating history again) while its +analysis pass reads each CLR's `UndoNxtLSN` straight into the transaction table +(Fig. 10, the `compensation` case in Step 4); undo then resumes exactly where it +left off. §3 states the resulting bound: "the number of CLRs written will be +exactly equal to the number of undoable log records written during forward +processing" — no matter how many times recovery itself crashes. + +**Undo is one merged backward sweep, not a loop over transactions.** This is the +detail earlier versions of this chapter got wrong. §3 and §6.3 describe it as +"continually taking the **maximum** of the LSNs of the next log record to be +processed for each of the yet-to-be-completely-undone loser transactions": + +``` +RESTART_UNDO (paper Fig. 12, condensed) + WHILE some Trans_Table entry has State = 'U': + UndoLSN := maximum(UndoNxtLSN) over entries with State = 'U' + LogRec := Log_Read(UndoLSN) + case 'update' and undoable: + X-latch the page; Undo_Update(Page, LogRec) + write a 'compensation' record (a CLR) whose UndoNxtLSN := LogRec.PrevLSN + Page.LSN := LSN(the CLR); Trans_Table[T].LastLSN := LSN(the CLR) + Trans_Table[T].UndoNxtLSN := LogRec.PrevLSN + IF LogRec.PrevLSN = 0: write 'end'; delete the Trans_Table entry + case 'compensation': + Trans_Table[T].UndoNxtLSN := LogRec.UndoNxtLSN -- skip the undone run ``` -One refinement worth knowing exists: **nested top actions** let a structural -change (a B-tree split) survive even if the transaction that triggered it -aborts — other transactions may already be using the new page; physical -consistency and logical visibility are different things (paper §10). +One sweep, strictly decreasing in LSN, hopping between losers. This matters for +a practical reason: it reads the log backward *once*, sequentially, instead of +once per loser. + +Note also what the undo pass does **not** do: there is no page-LSN test. Redo +already put the page in its pre-crash state, so the record's effect is known to +be present. §10.1 shows that a scheme *without* repeating history is forced to +test `page_LSN >= record.LSN` before undoing, and that this test is exactly what +breaks — see Step 7's postscript. + +**Nested top actions (§9 — not §10, and not a B-tree split).** Sometimes a +sub-sequence of a transaction's actions must survive the transaction's own +rollback. The paper's worked example (Fig. 14) is **file extension**: once a file +has been extended, other transactions may use the new area, so undoing the +extension "might very well lead to a loss of updates performed by the other +committed transactions." The mechanism is three steps (§9): remember the +transaction's current last-log-record position; log the nested action's records +as ordinary *undo-redo* records; and on completion write a **dummy CLR** whose +`UndoNxtLSN` points back to the remembered position. Rollback then hops straight +over the whole sequence. Crash *before* the dummy CLR, and the incomplete +sequence is undone normally — precisely because its records were undo-redo. "The +dummy CLR in a sense can be thought of as the commit record for the nested top +action", and unlike a real commit the transaction need not wait for it to be +forced. Index and hashing applications are in the companion papers, ARIES/IM and +ARIES/LHS [62, 59] — the B-tree split story lives there, not here. + +*Why it matters:* CLRs are the reason ARIES recovery is restartable, and +restartability is the property that separates a recovery *method* from a +recovery *sketch*. + +### Step 7 — the three passes, worked by hand + +> **In:** an eight-record log, a checkpoint, and a disk whose pages are in a +> specific known state. +> **Out:** every decision all three passes make, with reasons — the only way to +> know you actually understand the rules. + +**The log.** LSNs are shown 10 apart for readability; in reality they are byte +offsets (§4.1). The checkpoint's two tables are empty, so everything interesting +is reconstructed by analysis. + +| LSN | Txn | Type | Page | PrevLSN | note | +|---:|---|---|---|---:|---| +| 10 | — | `begin_chkpt` | — | — | master record points here | +| 20 | — | `end_chkpt` | — | — | Trans_Table ∅, Dirty_PagLst ∅ | +| 30 | T1 | update | P5 | 0 | T1's first record | +| 40 | T2 | update | P7 | 0 | | +| 50 | T1 | update | P5 | 30 | | +| 60 | T2 | `end` | — | 40 | T2 commits (§5.3) | +| 70 | T3 | update | P9 | 0 | | +| 80 | T1 | update | P7 | 50 | ← crash right after this | + +**Disk state at the crash.** The buffer manager wrote P7 out at some point after +LSN 40 was applied to it and before LSN 80 was, so on disk: `P5.LSN = 0`, +`P7.LSN = 40`, `P9.LSN = 0`. + +**Pass 1 — analysis**, from LSN 10, applying Fig. 10: + +| reading | Trans_Table | Dirty_Pages | +|---|---|---| +| 20 `end_chkpt` | ∅ | ∅ | +| 30 T1/P5 | T1: U, Last 30, UndoNxt 30 | P5 → RecLSN **30** | +| 40 T2/P7 | + T2: U, Last 40, UndoNxt 40 | + P7 → RecLSN **40** | +| 50 T1/P5 | T1: Last 50, UndoNxt 50 | P5 already present — RecLSN stays 30 | +| 60 T2 `end` | **T2 deleted** | — | +| 70 T3/P9 | + T3: U, Last 70, UndoNxt 70 | + P9 → RecLSN **70** | +| 80 T1/P7 | T1: Last 80, UndoNxt 80 | P7 already present — RecLSN stays 40 | + +Analysis concludes: **losers = {T1, T3}** (T2 left the table at its `end` +record); **Dirty_Pages = {P5:30, P7:40, P9:70}**; **RedoLSN = min RecLSN = 30**. + +**Pass 2 — redo**, from LSN 30, applying Fig. 11's three levels: + +| LSN | level (2): in DPT and LSN ≥ RecLSN? | level (3): page fetch | outcome | +|---:|---|---|---| +| 30 P5 | yes, 30 ≥ 30 | `P5.LSN = 0 < 30` | **redo**; `P5.LSN := 30` | +| 40 P7 | yes, 40 ≥ 40 | `P7.LSN = 40`, not < 40 | **skip**; `Dirty_Pages[P7].RecLSN := 41` | +| 50 P5 | yes, 50 ≥ 30 | `P5.LSN = 30 < 50` | **redo**; `P5.LSN := 50` | +| 60 | not update/compensation | — | ignored | +| 70 P9 | yes, 70 ≥ 70 | `P9.LSN = 0 < 70` | **redo — and T3 is a loser** | +| 80 P7 | yes, 80 ≥ **41** | `P7.LSN = 40 < 80` | **redo**; `P7.LSN := 80` | + +Two things to see. LSN 70 is redone *even though T3 will be rolled back three +lines from now* — that is repeating history. And LSN 40's `ELSE` branch fixed the +stale RecLSN for P7 from 40 to 41, which is what level (2) then tested LSN 80 +against. + +**Pass 3 — undo**, applying Fig. 12. Entering with T1 (UndoNxt 80) and T3 +(UndoNxt 70): + +| step | `max(UndoNxtLSN)` | undo | CLR written | CLR's `UndoNxtLSN` | table after | +|---|---:|---|---|---:|---| +| 1 | **80** (T1) | T1's update to P7 | LSN 90 | 50 | T1 UndoNxt 50, T3 UndoNxt 70 | +| 2 | **70** (T3) | T3's update to P9 | LSN 100 | 0 | T3 done → `end`, deleted | +| 3 | **50** (T1) | T1's update to P5 | LSN 110 | 30 | T1 UndoNxt 30 | +| 4 | **30** (T1) | T1's update to P5 | LSN 120 | 0 | T1 done → `end`, deleted | + +The order is **80, 70, 50, 30** — one backward sweep that alternates between T1 +and T3, not "finish T1, then start T3." Four undoable records in, four CLRs out, +exactly as §3 promises. + +**Now crash again, immediately after CLR 100.** Restart from scratch: + +- *Analysis* reads to the new end of log. At CLR 90 the `compensation` case sets + `T1.UndoNxtLSN := 50` — the CLR's own field, not its PrevLSN. At CLR 100 it + sets `T3.UndoNxtLSN := 0`, and Fig. 10's trailing loop ("State='U' and + UndoNxtLSN=0") writes T3's `end` record and drops it. **T3 needs no undo work + at all on this restart.** +- *Redo* replays CLRs 90 and 100 if their pages are behind — idempotently, by + the same page-LSN test. +- *Undo* starts at `max(UndoNxtLSN) = 50` and does steps 3 and 4 above. + +No update is undone twice, and the CLR count is still four. That is the whole +argument for CLRs, on concrete numbers. + +**Postscript: why not just skip the losers in redo?** §10.1's Figures 15 and 16 +answer it with three LSNs. A page's disk copy is at LSN 10. Loser T2 updated it +at LSN 20; non-loser T1 updated it at LSN 30. *Selective* redo (System R's +scheme: redo only committed and in-doubt transactions) skips 20, redoes 30, and +leaves the page at LSN 30. Undo then asks its usual question — is +`page_LSN ≥ 20`? — sees 30, and **undoes update 20 even though its effect was +never applied to the page.** The paper's own words: "By not repeating history, +the page_LSN is no longer a true indicator of the current state of the page." +Reversing the pass order doesn't save you either: undoing 20 first writes a CLR +whose LSN exceeds 30, so redo would then skip 30 although it isn't on the page. +Repeating history is not an aesthetic choice; it is what makes the page-LSN mean +something. + +*Why it matters:* if you can run this table without looking, you can read §6 of +the paper at speed, and you can review your own recovery code. ## How to read the paper (with the concepts in hand) 1. This chapter's steps (or Franklin's "Crash Recovery" chapter / CMU 15-445 recovery notes) until the three passes + CLRs feel obvious. -2. Paper §3 ("the problem"): why the naive undo-then-redo attempts fail — - the best catalog of recovery bugs ever assembled. -3. §6: the passes in detail — read for `pageLSN ≥ recLSN ⇒ skip redo` - (Step 2's idempotence via LSN comparison) and the CLR undoNext chain - (Step 6). -4. Skim §10 (nested top actions — how B-tree splits survive rollback: the - split stays even if the insert that caused it aborts). +2. **§1.1** for the definitions — page-oriented redo, logical undo, CLR. Ten + minutes here saves an hour later; the paper uses these terms as primitives + from §2 onward. +3. **§2** for steal/no-force and the non-obvious argument that no-steal is + unsatisfiable under fine-granularity locking. +4. **§6** for the passes in detail. Read Fig. 10, 11 and 12 as code, and check + them against Step 7's table — the three-level redo test (Fig. 11) and the + `maximum(UndoNxtLSN)` sweep (Fig. 12) are the two places a summary will have + lied to you. Fig. 13 is the paper's own worked example (all records on one + page; redo `3 4 4' 3' 5 6`, undo `6 5 2 1`). +5. **§10** — *Recovery Paradigms* — is the catalog of recovery bugs: the System R + paradigms that break under WAL + fine-granularity locking (selective redo; + undo before redo; no CLRs; not logging index and space-management changes; no + LSNs on pages). §10.1's Figures 15 and 16 are the sharpest pages in the paper. + (Earlier versions of this chapter pointed at §3 for this; §3 is the overview.) +6. Skim **§9** for nested top actions — file extension, dummy CLRs. ## Map to what you've read -- postgres: ANALYSIS+REDO yes (xlogrecovery.c), UNDO replaced by MVCC vacuum; - FPIs make redo idempotent even without perfect LSN discipline. -- turso WAL: no passes at all — commit boundary detection only. -- Your M5 WAL: if you chose logical records (reading-turso-wal.md Q3), you owe - ARIES-style idempotent redo: stamp pages with LSNs, skip if page is newer. +- **postgres**: analysis + redo yes (`xlogrecovery.c`, `ApplyWalRecord` at + `:1883` dispatching to `rm_redo` at `:1966`); undo replaced by MVCC + vacuum, + so there is no undo pass and no CLR. Full-page images make redo idempotent + even where LSN discipline alone would not suffice. +- **turso WAL**: no passes at all — recovery is commit-boundary detection over + whole page images (`reading-turso-wal.md`). A page image is idempotent by + construction, which is what buys the simplification. +- **redis AOF**: replay from the start of the log, or from a BASE snapshot; no + LSNs, no undo, and a torn tail is simply truncated + (`reading-redis-aof-rdb.md`). +- **Your M5 WAL**: if you chose logical records, you owe ARIES-style idempotent + redo — stamp pages with LSNs and skip when the page is newer. If you chose + no-steal, you owe no undo pass; say so explicitly, and say what it costs you in + concurrency (Step 1). ## Questions to answer in notes.md -1. Why must CLRs be redo-only (never undone)? Walk a crash-during-undo. -2. Nested top action for a B-tree split: why is letting the split survive an - aborted insert both correct and necessary? (Other txns may already use the - new page; physical consistency ≠ logical visibility.) -3. Which of steal/no-force does YOUR topic-3 B+tree + WAL implement? Derive +1. Why must CLRs be redo-only (never undone)? Walk a crash-during-undo using + Step 7's second crash, and say what would go wrong if the CLR at LSN 90 were + itself undoable. +2. Nested top action for a file extension (§9): why is letting the extension + survive an aborted transaction both correct and necessary? Name the concrete + loss that undoing it would cause. +3. Redo's level (2) test and its `ELSE` branch (Fig. 11) exist purely to avoid + page fetches. On a 100 GB database with a 10 GB buffer pool and a checkpoint + every 5 minutes, estimate how many pages levels (1) and (2) save you from + fetching, and compare that to reading the log itself. +4. Which of steal/no-force does *your* topic-3 B+tree + WAL implement? Derive which passes your recovery needs. (Likely no-steal/no-force at first ⇒ - redo-only — say so explicitly.) + redo-only — say so explicitly, and say which cell of Step 1's matrix you are + in.) +5. ARIES forces the log through the commit record. Using this topic's ladder, + state the commit ceiling for a single-threaded committer on each of the three + rungs, and say what group commit (`reading-postgres-xlog.md`) would change. ## Done when -You can fill the 2×2 steal/force matrix with (undo?, redo?) from memory and -explain repeating history in two sentences. +Answer each before unfolding it. + +- [ ] Fill the 2×2 steal/force matrix with (undo?, redo?) from memory, and give + the paper's non-obvious argument for why steal is not optional. + +
Answer + + no-steal/force: neither pass. no-steal/no-force: redo only. steal/force: undo + only. steal/no-force: both — ARIES's cell. The non-obvious argument (§2): under + fine-granularity locking with overlapping transactions, "with a no-steal + policy, a page may never get written to nonvolatile storage if the page always + contains uncommitted updates." No-steal can be *unsatisfiable*, not merely + slow. + +
+ +- [ ] Explain repeating history in two sentences, and give the concrete failure + that selective redo produces. + +
Answer + + Redo re-applies *every* update from `RedoLSN` forward whose page hasn't seen + it, losers included, so that the pages are in their exact pre-crash state; undo + can then be ordinary transaction rollback rather than a special mode reasoning + about half-restored pages. The failure (§10.1, Fig. 15–16): with a disk page at + LSN 10, a loser's update at 20 and a non-loser's at 30, selective redo skips 20 + and redoes 30, leaving `page_LSN = 30`; undo's `page_LSN ≥ 20` test then says + yes and undoes update 20 **although it was never applied to the page**. + +
+ +- [ ] State the redo test in full — all three levels — and say what the `ELSE` + branch does. + +
Answer + + (1) the record is an `update` or `compensation` and is redoable; (2) + `PageID IN Dirty_Pages AND LSN >= Dirty_Pages[PageID].RecLSN` — both in-memory, + and failing either means the page is never fetched; (3) fetch the page and test + `Page.LSN < LogRec.LSN`; if so redo and set `Page.LSN := LogRec.LSN`. The + `ELSE` branch sets `Dirty_Pages[PageID].RecLSN := Page.LSN + 1`: the page + reached disk after the checkpoint, so the table's RecLSN was stale, and fixing + it lets every later record for that page be filtered at level (2) instead of + being fetched. + +
+ +- [ ] In what order does the undo pass process a log with two losers? Use Step 7. + +
Answer + + One merged backward sweep, strictly decreasing in LSN, taking + `maximum(UndoNxtLSN)` across all `State='U'` entries each iteration (§6.3, + Fig. 12). In Step 7 that is **80 (T1), 70 (T3), 50 (T1), 30 (T1)** — it + alternates between the transactions. It is *not* "roll back T1 completely, then + roll back T3"; that would read the log backward once per loser. + +
+ +- [ ] How many CLRs does a recovery write, and does a second crash change the + answer? + +
Answer + + Exactly one per undoable log record written during forward processing (§3) — + four in Step 7's log. A second crash does not change it: analysis reads each + CLR's `UndoNxtLSN` into the transaction table (Fig. 10's `compensation` case), + so undo resumes at `max(UndoNxtLSN)` and never revisits a compensated record. + A loser whose `UndoNxtLSN` has reached 0 is finished off by Fig. 10's trailing + loop with an `end` record and no undo work at all. + +
+ +- [ ] What does a fuzzy checkpoint write, and what does it deliberately *not* do? + +
Answer + + It writes a `begin_chkpt` record, an `end_chkpt` record carrying the + transaction table, the dirty_pages table and the file mapping, and updates the + master record to the `begin_chkpt` LSN (§5.4). What it does not do: pause the + system, or flush pages — "ARIES does not require that any dirty pages be forced + to nonvolatile storage during a checkpoint." Recovery pays for that later, and + the bill is bounded by `min(RecLSN)`. + +
## References -**Papers** -- Mohan, Haderle, Lindsay, Pirahesh, Schwarz — "ARIES: A Transaction - Recovery Method Supporting Fine-Granularity Locking and Partial - Rollbacks Using Write-Ahead Logging" (ACM TODS 1992) — 70 pages; read a - summary first (Franklin's "Crash Recovery" chapter or CMU 15-445 - recovery notes), then dip into §3 and §6, skim §10 +**Paper** — Mohan, Haderle, Lindsay, Pirahesh & Schwarz, "ARIES: A Transaction +Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using +Write-Ahead Logging", *ACM Transactions on Database Systems* 17(1), March 1992, +pp. 94–162. DOI `10.1145/128765.128770`. + +| section / figure | what this chapter took from it | +|---|---| +| §1.1 | page-oriented redo, logical undo, CLRs as redo-only (Step 2) | +| §2 | steal / no-force, and why no-steal can be unsatisfiable (Step 1) | +| §4.1–§4.4 | LSN, PrevLSN, UndoNxtLSN, page-LSN, the two tables, RecLSN (Steps 2–3) | +| §5.3 | the `end` record as the commit point (Steps 4, 7) | +| §5.4 | fuzzy checkpoints; "no dirty pages need be forced" (Step 3) | +| §6.1, Fig. 10 | the analysis pass, including its trailing cleanup loop (Steps 4, 7) | +| §6.2, Fig. 11 | the three-level redo test and the RecLSN correction (Steps 5, 7) | +| §6.3, Fig. 12 | `maximum(UndoNxtLSN)` — the merged backward sweep (Steps 6, 7) | +| Fig. 13 | the paper's own single-page worked example | +| §9, Fig. 14 | nested top actions; file extension; the dummy CLR (Step 6) | +| §10, §10.1, Figs. 15–16 | the catalog of broken recovery paradigms; why repeat history (Steps 5–7) | +| [36] Haerder & Reuter 1983 | where steal/no-steal/force/no-force come from | +| [59], [62] | ARIES/LHS and ARIES/IM — where the index and hashing applications live | + +**Terminology note** — "ATT", "DPT" and "physiological logging" are textbook +shorthands, not the paper's words; it says *transaction table*, *dirty_pages +table*, and *page-oriented redo with logical undo*. Physiological logging is +Gray & Reuter, *Transaction Processing: Concepts and Techniques* (1993). + +**Measurements** — `topics/05-durability-wal/notes.md`, "Baseline (provided lane, +Apple M3 Pro / APFS, measured 2026-07-28)", from +`experiments/src/bin/fsync_ladder.rs`; headline in `FINDINGS.md` row 5. + +**Secondary reading** — Franklin, "Concurrency Control and Recovery" (in *The +Computer Science and Engineering Handbook*); CMU 15-445 lecture notes on +logging and recovery. diff --git a/topics/05-durability-wal/reading-postgres-xlog.md b/topics/05-durability-wal/reading-postgres-xlog.md index a468bc7..449211a 100644 --- a/topics/05-durability-wal/reading-postgres-xlog.md +++ b/topics/05-durability-wal/reading-postgres-xlog.md @@ -1,166 +1,551 @@ # postgres xlog: reserve-then-copy and the flush recheck -Postgres's WAL is 10,000+ lines of C, but it earns its keep with five -mechanisms: the back-linked record format, the reserve-then-copy insertion +Postgres's WAL is 10,196 lines of C in one file, but it earns its keep with +six mechanisms: the back-linked record format, the reserve-then-copy insertion trick, group commit via a flush recheck, full-page writes after checkpoints, -and fuzzy checkpointing with redo-only recovery. Before the code, this -chapter builds each mechanism as a concept — what problem it solves and what -it costs — then hands you the exact functions and lines to skim. Do NOT read -the file linearly. +fuzzy checkpointing with redo-only recovery, and a configurable sync call that +decides *which rung of the durability ladder* you are actually standing on. +Before the code, this chapter builds each mechanism as a concept — what problem +it solves and what it costs — then hands you the exact functions and lines to +skim. Do NOT read the file linearly. + +Every line number below was read at **postgres/postgres@701f021** +(`tools/pinned-source.py show postgres -r A:B`). Every timing below +comes from this topic's own provided lane, `cargo run --release --bin +fsync_ladder`, on the Apple M3 Pro / APFS machine recorded in `notes.md`. + +**Vocabulary, once, before it is used.** *WAL* (write-ahead log) is the rule +that a change is described in a sequential log, and that log made durable, +*before* the page holding the change may be written; postgres calls its WAL the +"xlog". An *LSN* (log sequence number) is a record's byte offset in that +ever-growing log, so LSNs are monotone and comparable. A *checkpoint* is a +periodic marker saying "everything logged before here is already in the data +files"; recovery need not read behind it. *Group commit* is the trick of +letting one durability call serve many transactions at once. And three system +calls that people say "fsync" for, which are three different things: + +| call | what it guarantees | measured p50 here | +|---|---|---| +| `write()` alone | bytes are in the OS page cache | **1.17 µs** | +| `fdatasync()` / macOS `fsync()` | bytes handed to the *drive*; the drive's volatile cache may still hold them | **22.67 µs** | +| macOS `fcntl(fd, F_FULLFSYNC)` | the drive has flushed its cache to stable media | **2.97 ms** | + +That is a **19.4×** step from the first to the second and a further **131×** +step from the second to the third (856 898 → 44 109 → 337 implied commits/s). +Whenever this chapter says a cost, it says which of the three rungs it means. +One precision, because this rung is the one everyone conflates: the middle row +was measured on macOS as `fsync(2)`. There is no `fdatasync` on this machine — +`fsync_ladder.rs` compiles that lane out — so 22.67 µs is a macOS `fsync` +number, and `fdatasync` is named here only because it occupies the same rung on +Linux. ## The problem in one sentence Hundreds of backend processes must append to one serial log and make their -commits durable, without the log's mutex or its ~1 ms fsync becoming the -ceiling — while also surviving the fact that a 8 KB postgres page can be -half-written when the power dies. +commits durable, without the log's spinlock or its sync call becoming the +ceiling — a per-commit `F_FULLFSYNC` caps the whole server at **337 +commits/s** on this machine — while also surviving the fact that an 8 KB +postgres page can be half-written when the power dies. ## The concepts, step by step ### Step 1 — the record: a backward-linked list with per-record checksums -A WAL record (postgres calls the WAL "xlog") is a self-describing entry: -total length, the transaction id that wrote it, a CRC (a checksum detecting -corruption), and — the interesting field — **`xl_prev`**, the LSN of the -*previous* record, making the log a backward-linked list even though -recovery reads it forward. Why: postgres recycles old 16 MB WAL segment -files by renaming them for reuse, so the tail of a "new" segment can contain -stale-but-internally-valid records from its previous life; a record whose -`xl_prev` doesn't point at the record actually before it is exposed as a -leftover. Records can also attach **block references** — links to the pages -they modify — and, when needed, a **full-page image** (FPI: a complete copy -of a page, Step 4) with the page's free-space hole elided to save bytes. +> **In:** a WAL segment file, possibly recycled from a previous life, being +> read forward byte by byte after a crash. +> **Out:** a decision, per record, of *this is a real record of mine* versus +> *this is where my log ends* — reached with a 24-byte header and no index. + +A WAL record is self-describing: total length, the transaction id that wrote +it, a CRC (a checksum over the record used to detect corruption), and — the +interesting field — **`xl_prev`**, the LSN of the *previous* record, making the +log a backward-linked list even though recovery reads it forward. The fixed +header is `SizeOfXLogRecord` = **24 bytes** (`xlogrecord.h:55`). + +Why carry a back-pointer you never follow? Because postgres does not delete +old WAL segments; it **renames them for reuse**. `InstallXLogFileSegment` is +documented as being used "both to install a newly-created segment (from a temp +file) and to recycle an old segment" and the file is "renamed into place" +(`xlog.c:3586–3600`). A segment is `DEFAULT_XLOG_SEG_SIZE` = 16 MB +(`pg_config_manual.h:20`), so the tail of a "new" segment holds up to 16 MB of +stale-but-internally-valid records from its previous life, each with a +perfectly good CRC. `xl_prev` is what exposes them. The reader says so in +its own words: + +```c +// src/backend/access/transam/xlogreader.c — ValidXLogRecordHeader, the +// sequential-read branch, 1173-1188 + 1173 else + 1174 { + 1175 /* + 1176 * Record's prev-link should exactly match our previous location. This + 1177 * check guards against torn WAL pages where a stale but valid-looking + 1178 * WAL record starts on a sector boundary. + 1179 */ + 1180 if (record->xl_prev != PrevRecPtr) + 1181 { + 1182 report_invalid_record(state, + 1183 "record with incorrect prev-link %X/%08X at %X/%08X", + 1184 LSN_FORMAT_ARGS(record->xl_prev), + 1185 LSN_FORMAT_ARGS(RecPtr)); + 1186 return false; + 1187 } + 1188 } +``` + +Note the two strengths of the test. Reading sequentially (`randAccess == +false`), the prev-link must match **exactly** (`xlogreader.c:1180`). Seeking to +an arbitrary LSN, all postgres can demand is `record->xl_prev < RecPtr` +(`xlogreader.c:1164`, whose comment concedes "we can't exactly verify the +prev-link") — a back-pointer into the future is impossible, but a stale one is +not detectable without knowing where you came from. + +Records can also attach **block references** — links to the pages they +modify (`xlogrecord.h:103`) — and, when needed, a **full-page image** (FPI: a +complete byte-for-byte copy of a page, Step 4) whose header +(`xlogrecord.h:141`) carries a `hole_offset` (`:144`) so the free-space hole in +the middle of the page is elided rather than logged. + +*Why it matters:* the log has no index and no table of contents. Every +structural guarantee recovery gets — where a record starts, where the log ends, +that this record is not litter — is squeezed out of 24 bytes per record. ### Step 2 — insertion: reserve serially, copy in parallel +> **In:** N backends each holding an assembled WAL record of a few dozen to a +> few thousand bytes, all wanting to append to one shared buffer. +> **Out:** each backend's bytes in the shared WAL buffers at a unique LSN +> range, with the serial section reduced to five assignments. + The naive design — one mutex around "append my record to the log buffer" — -serializes the memcpy of every backend (Aether's bottleneck A). Postgres -splits the operation in two: **reservation** — a spinlock held for ~3 -arithmetic operations hands out a byte range in the log ("your record -occupies LSN X..X+len") — and **copying** — the backend then memcpys its -record into that reserved slice of the shared WAL buffers under one of -`NUM_XLOGINSERT_LOCKS = 8` insertion locks, in parallel with up to 7 other -backends. The sequencing stays serial but is made *tiny*; the heavy part is -made parallel. This is Aether's consolidation insight shipped in -production, and topic 2's incremental-rehash move in disguise: keep the -critical section O(1), amortize/parallelize the heavy part. +serializes the memcpy of every backend. That is the bottleneck Aether calls +**(D) log buffer contention** (Johnson et al., VLDB 2010, §1.1). Postgres +splits the operation in two. + +**Reservation** hands out a byte range. The comment above it states the design +goal outright — "the duration the spinlock needs to be held is minimized by +minimizing the calculations that have to be done while holding the lock … +reserving X bytes from WAL is almost as simple as `CurrBytePos += X`" +(`xlog.c:1163–1170`) — and the critical section delivers on it: + +```c +// src/backend/access/transam/xlog.c — ReserveXLogInsertLocation, 1172-1184 + 1172 SpinLockAcquire(&Insert->insertpos_lck); + 1173 + 1174 startbytepos = Insert->CurrBytePos; + 1175 endbytepos = startbytepos + size; + 1176 prevbytepos = Insert->PrevBytePos; + 1177 Insert->CurrBytePos = endbytepos; + 1178 Insert->PrevBytePos = startbytepos; + 1179 + 1180 SpinLockRelease(&Insert->insertpos_lck); + 1181 + 1182 *StartPos = XLogBytePosToRecPtr(startbytepos); + 1183 *EndPos = XLogBytePosToEndRecPtr(endbytepos); + 1184 *PrevPtr = XLogBytePosToRecPtr(prevbytepos); +``` + +Count what is inside the lock: one addition and four field moves, nine lines. +The two conversions from "usable byte position" to a real `XLogRecPtr` — which +must skip page headers, and are the expensive part — happen at 1182–1184, +*outside*. Note also that this is where `xl_prev` gets its value: the +reservation both allocates the range and tells you what came before you +(`xlog.c:899–903`), which is why the record's CRC can only be finished +afterwards (`xlog.c:950–953`). + +**Copying** is the parallel half. The backend memcpys its record into that +reserved slice under one of `NUM_XLOGINSERT_LOCKS = 8` insertion locks +(`xlog.c:157`), acquired at `xlog.c:860` before the reservation and released +after `CopyXLogRecordToWAL` (`xlog.c:1266`, called at `:959`). Which of the 8 +you get is a hash of your backend number — `MyProcNumber % +NUM_XLOGINSERT_LOCKS` (`xlog.c:1430`) — with migration to another slot on +contention (`:1448`). Eight backends copy at once; only the five assignments +above are serial. The whole two-step design is spelled out in the comment at +`xlog.c:826–855`. + +A caution the guide used to get wrong: this is Aether's **§5.2 "Decoupling +Buffer Fill"** — release the buffer-allocation mutex immediately and let fills +pipeline — not its §5.1 consolidation array, in which threads combine their +requests via CAS in an auxiliary slot array *before* touching the mutex. +Postgres has no consolidation array. See `reading-aether.md` Step 5. + +*Why it matters:* this is topic 2's incremental-rehash move in disguise — keep +the unavoidably serial section O(1) and constant-sized, then parallelize the +part whose cost scales with the data. ### Step 3 — group commit: the flush recheck -Commit requires the log flushed through your commit record's LSN — but not -that *you* do the flushing. `XLogFlush`'s heart is one recheck: after -acquiring the write lock (possibly having waited behind another flusher), -look again at the shared flushed-LSN — while you waited, that other backend -probably fsynced past your LSN, and you return having done zero IO. One -fsync covers every backend that queued behind the lock. Group commit is -just that recheck: - -```rust -fn xlog_flush(&self, upto: Lsn) { - if self.flushed_lsn() >= upto { return; } // cheap check, no lock - let _g = self.write_lock.lock(); // maybe wait behind a flusher… - if self.flushed_lsn() >= upto { return; } // …RECHECK: their fsync already - // covered our LSN — free ride - self.write_out_buffers_through(upto); - self.wal_file.fdatasync(); // ONE fsync for every backend - self.advance_flushed_lsn(); // that queued behind the lock -} +> **In:** a backend that has just inserted its commit record at LSN `record`, +> and must not reply to the client until the log is durable through it. +> **Out:** the client's reply, having usually performed **zero** I/O itself. + +Commit requires the log flushed through your commit record's LSN — but not that +*you* do the flushing. `XLogFlush`'s heart is one recheck: after acquiring the +write lock (possibly having waited behind another flusher), look again at the +shared flushed-LSN. While you waited, that other backend probably synced past +your LSN, and you return having done nothing. One sync covers every backend +that queued behind the lock. That is all group commit is: + +```c +// src/backend/access/transam/xlog.c — XLogFlush's loop, 2848-2891 (comments elided) + 2848 for (;;) + 2849 { + 2853 RefreshXLogWriteResult(LogwrtResult); + 2854 if (record <= LogwrtResult.Flush) + 2855 break; + 2865 insertpos = WaitXLogInsertionsToFinish(WriteRqstPtr); + 2874 if (!LWLockAcquireOrWait(WALWriteLock, LW_EXCLUSIVE)) + 2875 { + 2881 continue; + 2882 } + 2884 /* Got the lock; recheck whether request is satisfied */ + 2885 RefreshXLogWriteResult(LogwrtResult); + 2886 if (record <= LogwrtResult.Flush) + 2887 { + 2888 LWLockRelease(WALWriteLock); + 2889 break; + 2890 } +``` + +There are in fact *three* exits before any I/O: a check before the loop is +entered at all (`xlog.c:2820–2821`), the top-of-loop check at 2853–2855, and +the post-lock recheck at 2885–2886. `LWLockAcquireOrWait` (2874) is the unusual +primitive — it returns false when the lock became free without being handed to +you, which sends you back around to re-read the flushed LSN rather than +acquiring a lock you may not need. Postgres's own comment (2867–2872) names the +purpose: "This helps to maintain a good rate of group committing when the +system is bottlenecked by the speed of fsyncing." + +Two optional knobs, `commit_delay` and `commit_siblings`, add a deliberate +pre-flush sleep to grow the batch further (`xlog.c:2901–2906`), guarded by +`MinimumActiveBackends(CommitSiblings)` so a lightly-loaded server never pays +the latency. Then, and only then, `XLogWrite` does the write and the sync +(`xlog.c:2925`). + +**Work the arithmetic on the measured numbers.** Let *T* be the cost of one +sync and λ the rate at which commit records arrive. A flush that starts now +serves everyone who arrives during the previous flush, so the steady-state +batch size is λ·*T* and the durable throughput is min(λ, batch/*T*) — the +system is stable at *any* λ, and the batch grows to absorb it: + ``` +rung = F_FULLFSYNC, T = 2.967 ms (measured p50) + + λ = 1 000 commits/s → batch = 1 000 × 0.002967 = 2.97 commits per sync + λ = 5 000 → batch = 14.84 + λ = 20 000 → batch = 59.34 + λ = 100 000 → batch = 296.70 -Two optional knobs (`commit_delay`/`commit_siblings`) add a pre-flush sleep -to grow the batch further. At 1 ms per fsync and 32 concurrent committers, -the recheck turns ~1K commits/s into ~32K. Your commit_throughput experiment -reimplements exactly this. +without group commit, every commit pays its own 2.967 ms: + ceiling = 1 / 0.002967 s = 337 commits/s, flat, at every λ above + +rung = macOS fsync, T = 22.67 µs (measured p50) + ceiling without group commit = 1 / 0.00002267 = 44 109 commits/s + λ = 100 000 → batch = 100 000 × 0.00002267 = 2.27 commits per sync +``` + +Two lessons fall out. First, group commit is not an optimization, it is what +makes the durable ceiling a function of λ rather than a constant. Second, the +batch is large exactly when the sync is slow: on the `F_FULLFSYNC` rung a +20 000/s workload rides 59 transactions per sync, but on the `fsync` rung the +same workload rides 0.45 — group commit does almost nothing there, because +there is nothing to hide. + +Beware the number that used to be in this file: "at 1 ms per fsync and 32 +concurrent committers, ~1K commits/s becomes ~32K." The arithmetic is fine and +the 1 ms is from nowhere. On the machine in `notes.md`, 32 committers on the +`F_FULLFSYNC` rung get 337 × 32 = **10 784** commits/s at best, and that is a +ceiling on the *batch*, not a promise — it needs the 32 to actually overlap. +Your `commit_throughput` experiment reimplements exactly this recheck. + +*Why it matters:* the recheck is nine lines and it is the difference between a +server that does 337 commits/s and one that does tens of thousands. ### Step 4 — full-page writes: the torn-page defense -A **torn page** is a page half-written at the moment of power loss — the -disk holds 4 KB of new bytes and 4 KB of old, an inconsistent hybrid that -postgres's normal WAL records can't fix, because they are *deltas* ("set -this tuple's field") that assume the page under them is intact. The fix: -the **first** modification of each page after a checkpoint logs a full-page -image instead of a delta (`needs_backup = page_lsn <= RedoRecPtr` — the -page hasn't been touched since the checkpoint's redo point). Recovery then -restores the whole page from the FPI before applying later deltas — a torn -page is simply overwritten wholesale. The cost is the famous **sawtooth**: -WAL volume spikes right after every checkpoint (every hot page owes one -8 KB image), then decays. Alternatives on the same problem: InnoDB's -double-write buffer (write pages twice, once to a scratch area); LMDB and -SQLite-WAL never overwrite pages at all, so they have no torn-page problem -to solve. +> **In:** a page on disk that was half-written when the power died — 4 KB of +> new bytes and 4 KB of old, with a CRC that matches neither. +> **Out:** a correct page, reconstructed without ever reading the broken one. + +A **torn page** is a page half-written at the moment of power loss. Postgres's +normal WAL records cannot fix it, because they are *deltas* ("set this tuple's +field") that assume the page under them is intact — a physiological record, +logical within a page and physical about which page. The fix: the **first** +modification of each page after a checkpoint logs a full-page image instead of +a delta. The test is one comparison: + +```c +// src/backend/access/transam/xloginsert.c — XLogRecordAssemble, 678-694 + 678 /* Determine if this block needs to be backed up */ + 679 if (regbuf->flags & REGBUF_FORCE_IMAGE) + 680 needs_backup = true; + 681 else if (regbuf->flags & REGBUF_NO_IMAGE) + 682 needs_backup = false; + 683 else if (!doPageWrites) + 684 needs_backup = false; + 685 else + 686 { + 692 XLogRecPtr page_lsn = PageGetLSN(regbuf->page); + 693 + 694 needs_backup = (page_lsn <= RedoRecPtr); +``` + +Three overrides come first — an explicit force, an explicit suppress, and +`full_page_writes = off` — and only then the real test. `page_lsn <= +RedoRecPtr` reads as "this page has not been touched since the checkpoint's +redo point", so its on-disk image is the one the checkpoint left and recovery +may have to rebuild it. Recovery restores the whole page from the FPI before +applying later deltas; a torn page is simply overwritten wholesale, never +read. + +The cost is the famous **sawtooth**: WAL volume spikes right after every +checkpoint, because every hot page owes one 8 KB image, then decays as the +working set is covered. Alternatives on the same problem: InnoDB's double-write +buffer (write every page twice, once to a scratch area, so one intact copy +always exists); LMDB and SQLite-WAL never overwrite a page in place at all, so +they have no torn-page problem to solve — see `reading-turso-wal.md`. + +*Why it matters:* full-page writes are the clearest case in the topic of buying +recovery correctness with steady-state write bandwidth, and the exchange rate +is set by one tunable (`checkpoint_timeout`). ### Step 5 — fuzzy checkpoints and redo-only recovery -A postgres checkpoint doesn't stop the world: it sets a **redo point** (the -LSN recovery will start from) under the insert lock, then flushes dirty -buffers over minutes *while WAL keeps rolling* — "fuzzy" because the -checkpoint is a starting point, not a consistent snapshot (ARIES Step 3). -Recovery reads forward from the redo point, dispatching each record to a -per-resource-manager redo handler. Two things to notice: - -- **A bad CRC means "end of log", not "corruption error."** After a crash - the log's tail is *expected* to be garbage (a half-written record); - per-record checksums are how recovery finds the cliff edge and stops. -- **There is no undo pass.** Postgres MVCC never overwrites tuples in - place — an update writes a new tuple version, so a loser transaction's - writes are just dead tuples that vacuum will reap. ARIES's undo machinery - (CLRs, rollback) isn't needed; the log is redo-only. Read - reading-aries.md for what postgres is *not* doing. - -### Step 6 — the sync method: which fsync do you mean? - -The actual durability call is configurable (`wal_sync_method`): `fsync`, -`fdatasync` (skips inode metadata — usually the right default), or -`open_datasync` (the file is opened with O_DSYNC, so every write syncs — no -separate call). These differ by 10× or more on the same hardware, and on -macOS none of them flush the drive cache without F_FULLFSYNC. Your -fsync_ladder experiment measures exactly these — the numbers feed every -design decision in M5. +> **In:** a running server with a dirty buffer pool and a log that never stops. +> **Out:** a durable marker saying "recovery may start here", produced without +> pausing writes. + +A **fuzzy checkpoint** is a checkpoint that does not stop the world: it fixes a +**redo point** (the LSN recovery will start from) and then flushes dirty +buffers over minutes *while WAL keeps rolling*. "Fuzzy" because the result is a +starting point, not a consistent snapshot — the data files at the end of the +checkpoint match no single instant. ARIES §5.4 is where this comes from +(`reading-aries.md` Step 3). + +How postgres fixes the redo point depends on the kind of checkpoint, and this +is the detail the guide previously got wrong. For a **shutdown** checkpoint the +redo point is taken while the insert locks are held (`xlog.c:7529–7562`, the +assignment at `:7561`). For a normal **online** checkpoint the insert locks are +released first (`:7568`) and the redo point is instead the LSN of a dedicated +`XLOG_CHECKPOINT_REDO` record inserted into the log (`:7579–7593`, with +`checkPoint.redo = RedoRecPtr` at `:7601`). Only afterwards does +`CheckPointGuts(checkPoint.redo, flags)` (`:7715`) flush the dirty buffers, and +the checkpoint record itself is inserted at `:7750` and flushed at `:7754`. +`CreateCheckPoint` runs `xlog.c:7400–7897`. + +Recovery reads forward from the redo point in `PerformWalRecovery` +(`xlogrecovery.c:1612`), whose loop calls `ApplyWalRecord` at `:1782`; +`ApplyWalRecord` itself is defined at `:1883` and dispatches with +`GetRmgr(record->xl_rmid).rm_redo(xlogreader)` at `:1966`. Two things to +notice: + +- **A bad CRC means "end of log", not "corruption error."** After a crash the + log's tail is *expected* to be garbage — a record half-written when the power + went. `ValidXLogRecord` (`xlogreader.c:1205–1227`) recomputes the CRC and + compares at `:1218`; a mismatch is the cliff edge, and recovery stops there + having replayed everything before it. +- **There is no undo pass.** Postgres MVCC never overwrites a tuple in place — + an update writes a new tuple version — so a loser transaction's writes are + just dead tuples that vacuum will reap. ARIES's undo machinery (CLRs, + rollback) is not needed; the log is redo-only. Read `reading-aries.md` for + what postgres is deliberately *not* doing, and what it gives up by not doing + it. + +*Why it matters:* checkpoint interval is the one knob that trades steady-state +cost (FPI volume, Step 4) against recovery time, and "fuzzy" is what makes the +knob cheap enough to turn. + +### Step 6 — the sync method: which rung are you on? + +> **In:** a decision that a byte range of WAL must survive a power cut. +> **Out:** one of five different system-call sequences, spanning **19.4×** in +> cost between the two you are most likely to be running. + +The durability call is configurable via `wal_sync_method`, and +`issue_xlog_fsync` (`xlog.c:9361`) has **five** cases, not three +(`xlog.c:9383–9409`): + +| `wal_sync_method` | what it calls | rung | +|---|---|---| +| `fsync` | `pg_fsync_no_writethrough` → `fsync()` | middle | +| `fsync_writethrough` | `pg_fsync_writethrough` → `fcntl(fd, F_FULLFSYNC)` (`fd.c:467`) | **top** | +| `fdatasync` | `fdatasync()` — skips inode metadata | middle | +| `open_sync` | nothing; the file was opened `O_SYNC`, so `issue_xlog_fsync` asserts unreachable (`:9399–9403`) | middle | +| `open_datasync` | nothing; the file was opened `O_DSYNC` | middle | + +So the old claim that "on macOS none of them flush the drive cache without +F_FULLFSYNC" is wrong in the specific: postgres ships that rung as +`fsync_writethrough`. What is true is that it is never the default. The default +is `open_datasync` where `O_DSYNC` exists and differs from `O_SYNC`, otherwise +`fdatasync` (`xlogdefs.h:78–84`); neither `src/template/linux` nor +`src/template/darwin` overrides it. A stock postgres on macOS therefore sits on +the **middle** rung — 22.67 µs, 44 109 implied commits/s, and a drive cache +that has not been flushed. Turning on `fsync_writethrough` moves you to +2.97 ms and 337/s, a **131×** price for the guarantee most people assume they +already had. The macOS `fsync(2)` manual page is the primary source: "while +fsync() will flush all data from the host to the drive … the drive itself may +not physically write the data to the platters for quite some time … For +applications that require tighter guarantees … Mac OS X provides the +F_FULLFSYNC fcntl." + +Your `fsync_ladder` experiment measures exactly these rungs; the numbers feed +every design decision in M5. + +*Why it matters:* every durability claim in this topic — and most of the ones +you will read elsewhere — is meaningless until it says which of these five +lines it ran. ## Where each step lives in the code +Anchors verified at postgres/postgres@701f021. + - **Step 1 — `xlogrecord.h:41–53`**: `XLogRecord` — `xl_tot_len`, `xl_xid`, - `xl_prev`, `xl_crc`. Block references :103; full-page images ride in - `XLogRecordBlockImageHeader` (:141) — note `hole_offset`: the free space - in the middle of a page is elided from the FPI. -- **Step 2 — `xlog.c`**: `ReserveXLogInsertLocation` :1149–1193 — the - spinlock held for ~3 arithmetic ops (:1172–1180) hands out byte ranges. - `CopyXLogRecordToWAL` :1266 — copy into the reserved slice under one of - `NUM_XLOGINSERT_LOCKS = 8` (xlog.c:157) insertion locks. -- **Step 3 — `xlog.c:2800–2891`**: `XLogFlush`; the recheck of - `LogwrtResult.Flush` at :2885; `commit_delay`/`commit_siblings` - :2901–2906. -- **Step 4 — `xloginsert.c:621–700`**: `XLogRecordAssemble`; - `needs_backup = (page_lsn <= RedoRecPtr)` at :694. -- **Step 5 — checkpoint + recovery**: `CreateCheckPoint` — xlog.c:7400–7560, - redo point set under the insert lock (:7561), then dirty buffers flushed - while WAL keeps rolling. `PerformWalRecovery` — - xlogrecovery.c:1612–1806; `ApplyWalRecord` (:1782) dispatches to - per-resource-manager redo handlers. Per-record CRC validation in - xlogreader.c:1207–1227 — invalid CRC = end of log. -- **Step 6 — `xlog.c:9361–9410`**: `issue_xlog_fsync` — fsync / fdatasync / - open_datasync. + `xl_prev`, `xl_info`, `xl_rmid`, `xl_crc`; `SizeOfXLogRecord` = 24 at `:55`. + Block references `:103`; FPI header `XLogRecordBlockImageHeader` `:141`, with + `hole_offset` at `:144`. Prev-link enforcement: + `xlogreader.c:1139–1191` (`ValidXLogRecordHeader`), the exact-match branch at + `:1173–1188`, the random-access branch at `:1160–1171`. Segment recycling: + `xlog.c:3586–3600`; segment size `pg_config_manual.h:20`. +- **Step 2 — `xlog.c`**: `ReserveXLogInsertLocation` `:1149–1193`, spinlock + `:1172–1180`. `CopyXLogRecordToWAL` `:1266`, called from `:959`. + `NUM_XLOGINSERT_LOCKS = 8` at `:157`; `WALInsertLockAcquire` `:1410–1450`, + slot hash `:1430`, migration `:1448`. Design comment `:826–855`. +- **Step 3 — `xlog.c:2800–2930`**: `XLogFlush`; pre-loop exit `:2820–2821`; + loop `:2848`; top-of-loop check `:2853–2855`; `LWLockAcquireOrWait` `:2874`; + **the recheck at `:2885–2886`**; `commit_delay`/`commit_siblings` + `:2901–2906`; `XLogWrite` `:2925`. Cost of extra insertion locks: + `WaitXLogInsertionsToFinish` `:1545`, its loop over all 8 at `:1597`. +- **Step 4 — `xloginsert.c:621`**: `XLogRecordAssemble`; the four-branch backup + decision `:679–694`; `needs_backup = (page_lsn <= RedoRecPtr)` at `:694`. +- **Step 5 — checkpoint + recovery**: `CreateCheckPoint` `xlog.c:7400–7897`; + shutdown redo point under the insert locks `:7529–7562`; online path releases + them at `:7568` and inserts `XLOG_CHECKPOINT_REDO` at `:7579–7593`, assigning + `checkPoint.redo` at `:7601`; `CheckPointGuts` `:7715`; checkpoint record + `:7750`, flushed `:7754`. `PerformWalRecovery` `xlogrecovery.c:1612`, calling + `ApplyWalRecord` at `:1782`; `ApplyWalRecord` defined `:1883`, rmgr dispatch + `:1966`. CRC validation `xlogreader.c:1205–1227`, compare at `:1218`. +- **Step 6 — `xlog.c:9361`**: `issue_xlog_fsync`, five cases at `:9383–9409`; + `pg_fsync_writethrough` → `fcntl(F_FULLFSYNC)` at + `src/backend/storage/file/fd.c:467`; default choice `xlogdefs.h:78–84`. ## Questions to answer in notes.md 1. Why is `xl_prev` needed when records are read forward anyway? (Detects a - valid-looking record left over from a recycled segment file.) -2. FPI sawtooth: checkpoint_timeout ↑ ⇒ WAL volume ↓ but recovery time ↑. - Write the trade as a formula in (dirty rate, checkpoint interval). + valid-looking record left over from a recycled 16 MB segment file — + `xlogreader.c:1176–1179` says so in the source.) Follow-up: why does the + random-access path only check `xl_prev < RecPtr`? +2. FPI sawtooth: `checkpoint_timeout` ↑ ⇒ WAL volume ↓ but recovery time ↑. + Write the trade as a formula in (pages dirtied per second, checkpoint + interval, page size) and evaluate it for 500 pages/s and intervals of 5 and + 30 minutes. 3. The 8 insertion locks: what workload would make you raise the number, and - what does postgres pay for each extra lock at flush time? (Flush must wait - for all in-progress copies below the target LSN — WaitXLogInsertionsToFinish.) + what does postgres pay for each extra lock at flush time? (Every flush walks + all of them — `WaitXLogInsertionsToFinish`, the loop at `xlog.c:1597`.) +4. You set `wal_sync_method = fsync_writethrough` on the machine in + `notes.md`. Using the ladder, what is the new single-threaded commit + ceiling, and how many concurrent committers do you need before group commit + restores the throughput you had on the default setting? ## Done when -You can explain reserve-then-copy, the flush recheck, and needs_backup in -three sentences total — those three lines are the file. +Answer each before unfolding it. + +- [ ] Name the two halves of a WAL insert, and say exactly how many operations + postgres performs inside the reservation spinlock. + +
Answer + + Reservation and copying. Inside `SpinLockAcquire`/`SpinLockRelease` + (`xlog.c:1172–1180`) there is **one addition and four field moves** — read + `CurrBytePos`, add `size`, read `PrevBytePos`, store both back. The two + byte-position-to-`XLogRecPtr` conversions, which have to skip page headers, + are deliberately outside at `:1182–1184`. Copying then happens under one of + `NUM_XLOGINSERT_LOCKS = 8` insertion locks, so eight backends memcpy at once. + +
+ +- [ ] Point at the single line that makes group commit work, and say what a + backend that hits it has just avoided. + +
Answer + + `xlog.c:2885–2886` — `RefreshXLogWriteResult(LogwrtResult); if (record <= + LogwrtResult.Flush)` immediately after `LWLockAcquireOrWait` succeeds. A + backend that breaks there has avoided the write **and the sync**: another + backend's flush already covered its LSN. On the `F_FULLFSYNC` rung that is + 2.97 ms of latency it did not pay, and it is why N committers can exceed the + 337 commits/s that one committer is capped at. + +
+ +- [ ] State the `needs_backup` test in words, and say what recovery does with + the resulting record. + +
Answer + + `needs_backup = (page_lsn <= RedoRecPtr)` (`xloginsert.c:694`) — "this page + has not been modified since the checkpoint's redo point." When true, the + record carries a **full-page image** instead of a delta. Recovery writes that + whole 8 KB page over whatever is on disk before applying any later delta to + it, so a page that was torn mid-write is never read, only overwritten. The + price is the post-checkpoint WAL sawtooth: one 8 KB image per hot page. + +
+ +- [ ] Postgres recovery has no undo pass. Say what makes that possible and + what it costs. + +
Answer + + MVCC: an update writes a *new* tuple version rather than overwriting the old + one, so an aborted transaction leaves dead tuples rather than corrupt data. + Recovery therefore replays forward from the redo point and stops — no CLRs, + no rollback machinery (contrast ARIES, `reading-aries.md`). The cost is paid + elsewhere: dead tuples must be reclaimed by vacuum, tables bloat between + vacuums, and long-running readers pin old versions. + +
+ +- [ ] A colleague says "postgres fsyncs on every commit, and an fsync costs + about a millisecond." Correct both halves. + +
Answer + + *Neither half is anchored.* (a) Postgres does not fsync per commit — it + flushes per *batch*, and any committer whose LSN is already covered performs + no I/O at all (`xlog.c:2885`). (b) "An fsync" is not one thing: + `issue_xlog_fsync` has five cases (`xlog.c:9383–9409`), and on the machine in + `notes.md` the middle rung (`fdatasync`, `open_datasync`, macOS `fsync`) is + **22.67 µs** while the top rung (`fsync_writethrough` → + `fcntl(F_FULLFSYNC)`, `fd.c:467`) is **2.97 ms** — 131× apart. The default is + `open_datasync` or `fdatasync` (`xlogdefs.h:78–84`), i.e. the middle rung, + which on macOS leaves the data in the drive's volatile cache. + +
## References -**Code** -- [postgres/postgres](https://github.com/postgres/postgres) — - `src/backend/access/transam/xlog.c` (10,196 lines — do NOT read - linearly), `src/backend/access/transam/xloginsert.c`, - `src/backend/access/transam/xlogrecovery.c`, - `src/include/access/xlogrecord.h`. Local clone at `~/repos/postgres`. +**Code** — all anchors read at `postgres/postgres@701f021`; local clone at +`~/repos/postgres`, pin recorded in `resources/codebases.md`. + +| file | what this chapter took from it | +|---|---| +| `src/backend/access/transam/xlog.c` (10,196 lines — do NOT read linearly) | insertion (Step 2), flush and group commit (Step 3), checkpoints (Step 5), `issue_xlog_fsync` (Step 6) | +| `src/backend/access/transam/xloginsert.c` | `XLogRecordAssemble` and the `needs_backup` test (Step 4) | +| `src/backend/access/transam/xlogreader.c` | prev-link and CRC validation, i.e. "where does the log end" (Steps 1, 5) | +| `src/backend/access/transam/xlogrecovery.c` | `PerformWalRecovery`, `ApplyWalRecord`, rmgr dispatch (Step 5) | +| `src/backend/storage/file/fd.c` | `pg_fsync_writethrough` → `F_FULLFSYNC` (Step 6) | +| `src/include/access/xlogrecord.h`, `xlogdefs.h`, `src/include/pg_config_manual.h` | record layout, default sync method, 16 MB segment size | + +**Measurements** — `topics/05-durability-wal/notes.md`, "Baseline (provided +lane, Apple M3 Pro / APFS, measured 2026-07-28)", produced by +`experiments/src/bin/fsync_ladder.rs`. `FINDINGS.md` row 5 carries the +headline. + +**Papers** — Mohan et al., "ARIES", *ACM TODS* 17(1), 1992, §5.4 for fuzzy +checkpoints. Johnson et al., "Aether: A Scalable Approach to Logging", *PVLDB* +3(1), 2010, §1.1 for the bottleneck taxonomy and §5.2 for decoupled buffer +fill. + +**Manual pages** — macOS `fsync(2)`, the paragraph beginning "Note that while +fsync() will flush all data from the host to the drive". diff --git a/topics/05-durability-wal/reading-redis-aof-rdb.md b/topics/05-durability-wal/reading-redis-aof-rdb.md index 3fbf98d..d0e04f3 100644 --- a/topics/05-durability-wal/reading-redis-aof-rdb.md +++ b/topics/05-durability-wal/reading-redis-aof-rdb.md @@ -1,164 +1,574 @@ # Redis AOF & RDB: the command stream is the log -Redis logs the *commands themselves* (AOF) and checkpoints by *forking* (RDB) -— and since a graph module's data lives inside redis's keyspace, this is the -durability FalkorDB actually has today. Before the code, this chapter builds -the design step by step: what a command log is, what the fsync policy knob -really promises, why a command log must be rewritten, and how fork+COW turns -the OS into a snapshot engine. Read it as the incumbent your M5 design +Redis logs the *commands themselves* (AOF) and checkpoints by *forking* (RDB) — +and since a graph module's data lives inside redis's keyspace, this is the +durability FalkorDB actually has today. Before the code, this chapter builds the +design step by step: what a command log is, what the fsync policy knob really +promises, why a command log must be rewritten, and how fork + copy-on-write +turns the OS into a snapshot engine. Read it as the incumbent your M5 design competes with. +Every line number below was read at **redis/redis@a176d1225** +(`tools/pinned-source.py show redis -r A:B`). Every timing comes from +this topic's provided lane, `cargo run --release --bin fsync_ladder`, on the +Apple M3 Pro / APFS machine recorded in `notes.md`. + +**Vocabulary, once, before it is used.** A *WAL* (write-ahead log) is a +sequential file a change is written to, and made durable in, before the state it +describes may be considered committed; an *AOF* is redis's WAL, and it logs +commands rather than pages or deltas. A *checkpoint* is a complete rendering of +current state that lets the log before it be discarded; RDB is redis's. *Group +commit* is letting one durability call serve many operations. And the three +rungs of the durability ladder, which this chapter names every time it says +"fsync": + +| call | what it guarantees | measured p50 here | +|---|---|---| +| `write()` alone | bytes in the OS page cache; survives `kill -9`, not power loss | **1.17 µs** | +| `fdatasync()` / macOS `fsync()` | bytes handed to the drive; its volatile cache may still hold them | **22.67 µs** | +| macOS `fcntl(fd, F_FULLFSYNC)` | the drive flushed its cache to stable media | **2.97 ms** | + +**19.4×** from the first to the second, a further **131×** to the third — 856 +898 → 44 109 → 337 implied durable operations per second. The middle row was +measured on macOS as `fsync(2)`; there is no `fdatasync` on this machine +(`fsync_ladder.rs` compiles that lane out), so it is named only because it +occupies the same rung on Linux. Redis picks a different rung *on different +platforms for the same config value*, which is the single most surprising fact +in this chapter (Step 2). + ## The problem in one sentence -Redis serves ~100K+ commands/s from one thread, so it cannot afford either -an fsync per command (~1 ms each would cap it at ~1K/s) or any pause to -write a snapshot — its durability design is entirely shaped by "the main -thread must never wait for the disk," and the price is a stated window of +Redis serves ~100K+ commands/s from one thread, so it can afford neither a +durability call on the command path — on this machine the top rung is 2.97 ms, +which would put a 2.97 ms floor under every write's latency — nor any pause to +write a snapshot; its durability design is entirely shaped by "the main thread +must never wait for the disk," and the price is a stated, configurable window of acknowledged-but-lost writes. ## The concepts, step by step ### Step 1 — the command log: log what was *said*, not what changed -An AOF ("append-only file") is a **command log**: instead of logging page -images (turso) or record deltas (postgres), redis appends the write -commands themselves — literally as RESP protocol text, the same bytes a -client would send. `SET user:42 "avi"` goes into the log as `SET user:42 -"avi"`. Recovery is replay: start an empty server, feed it the file as if a -very fast client were retyping history. The trade is volume-vs-CPU flipped -from the other designs: a command is usually tiny (tens of bytes — the -cheapest possible log record), but replay must re-execute *full command -processing* — parsing, dispatch, data-structure updates — so recovery time -scales with total command count, not with final data size. Every write -command is appended to an in-memory buffer (`server.aof_buf`) during -command execution; what happens to that buffer next is the whole durability -story. +> **In:** a write command that has just executed against the in-memory +> keyspace. +> **Out:** the same command, re-serialised as RESP protocol text, appended to +> an in-memory buffer — with no file I/O on the command path at all. + +An AOF ("append-only file") is a **command log**: instead of logging page images +(turso, `reading-turso-wal.md`) or record deltas (postgres, +`reading-postgres-xlog.md`), redis appends the write commands themselves, +literally as RESP protocol text — the same bytes a client would send. `SET +user:42 "avi"` goes into the log as `SET user:42 "avi"`. Recovery is replay: +start an empty server and feed it the file as if a very fast client were +retyping history. + +The append itself is one `sdscatlen`, and the comment above it states the whole +contract: + +```c +// src/aof.c — feedAppendOnlyFile's tail, 1438-1445 + 1438 /* Append to the AOF buffer. This will be flushed on disk just before + 1439 * of re-entering the event loop, so before the client will get a + 1440 * positive reply about the operation performed. */ + 1441 if (server.aof_state == AOF_ON || + 1442 (server.aof_state == AOF_WAIT_REWRITE && server.child_type == CHILD_TYPE_AOF)) + 1443 { + 1444 server.aof_buf = sdscatlen(server.aof_buf, buf, sdslen(buf)); + 1445 } +``` + +The trade against the other designs is volume-vs-CPU, flipped. A command is +usually tiny — tens of bytes, the cheapest possible log record — but replay must +re-execute *full command processing*: parsing, dispatch, data-structure updates. +Recovery time therefore scales with **total command count**, not with final data +size. A key written a million times costs a million replays; in turso it would +cost one page image, in postgres one delta plus whatever survives checkpointing. + +*Why it matters:* every other property in this chapter follows from putting the +log record on the *command* axis. Rewrite (Step 3) exists only because command +logs grow with history; fork-based snapshotting (Step 5) exists only because +that rewrite must not pause the server. ### Step 2 — the fsync policy: durability as a config knob -Once per event-loop iteration the buffer is `write()`n to the AOF file — -which only reaches the kernel's page cache, not the disk (topic README §3: -the kernel lies until fsync). *When to fsync* is a user-facing policy, and -this is the design's signature: redis makes the durability window a -**config choice** the other systems don't offer: - -- **`always`** — fsync before processing more commands. Durable, and slow: - the main thread eats ~1 ms per batch. -- **`everysec`** — fsync issued at most once per second, **on a background - thread** (the "bio" thread): the main thread never touches the disk. - Window: up to ~2 s of *acknowledged* writes can vanish. -- **`no`** — the kernel flushes when it feels like it. Window: unbounded - (typically ~30 s). - -The three contracts, side by side: - -```rust -// flushAppendOnlyFile: the client was ACKED before any of this runs. -fn flush_aof(&mut self, policy: Fsync) { - self.file.write_all(&self.aof_buf); // into page cache only - self.aof_buf.clear(); - match policy { - Fsync::Always => self.file.fdatasync(), // durable before next ack: slow - Fsync::EverySec => { - if self.last_fsync.elapsed() >= Duration::from_secs(1) { - self.bio.submit(FsyncJob); // background thread — main - } // thread never touches the disk; - } // window: up to ~2 s of ACKED writes - Fsync::No => {} // kernel decides; window unbounded - } -} +> **In:** an `aof_buf` holding one event loop's worth of commands, and a +> configured `appendfsync` value. +> **Out:** bytes in the page cache, plus zero, one, or a deferred durability +> call — and a client reply that is sent *after* whichever of those happened. + +Once per event-loop iteration the buffer is `write()`n to the AOF file — which +only reaches the kernel's page cache, not the drive. *When to fsync* is a +user-facing policy, and this is the design's signature: redis makes the +durability window a **config choice** the other systems in this topic don't +offer. + +```c +// src/aof.c — flushAppendOnlyFile's policy tail, 1329-1354 (logging elided) + 1329 /* Perform the fsync if needed. */ + 1330 if (server.aof_fsync == AOF_FSYNC_ALWAYS) { + 1331 /* redis_fsync is defined as fdatasync() for Linux in order to avoid + 1332 * flushing metadata. */ + 1337 if (redis_fsync(server.aof_fd) == -1) { + 1340 exit(1); + 1341 } + 1344 server.aof_last_incr_fsync_offset = server.aof_last_incr_size; + 1345 server.aof_last_fsync = server.mstime; + 1347 } else if (server.aof_fsync == AOF_FSYNC_EVERYSEC && + 1348 server.mstime - server.aof_last_fsync >= 1000) { + 1349 if (!sync_in_progress) { + 1350 aof_background_fsync(server.aof_fd); + 1351 server.aof_last_incr_fsync_offset = server.aof_last_incr_size; + 1352 } + 1353 server.aof_last_fsync = server.mstime; + 1354 } +``` + +Three policies (`AOF_FSYNC_NO 0`, `AOF_FSYNC_ALWAYS 1`, `AOF_FSYNC_EVERYSEC 2` +— `server.h:634–636`), and note that `no` has no branch at all: redis simply +never issues a durability call and the kernel's writeback timer decides. On +Linux that default is `vm.dirty_expire_centisecs = 3000`, i.e. **30 seconds**. + +**Which rung is `always`?** This is the fact worth carrying away. `redis_fsync` +is a macro, and it is not the same call everywhere: + +```c +// src/config.h — 128-135 + 128 /* Define redis_fsync to fdatasync() in Linux and fsync() for all the rest */ + 129 #if defined(__linux__) + 130 #define redis_fsync(fd) fdatasync(fd) + 131 #elif defined(__APPLE__) + 132 #define redis_fsync(fd) fcntl(fd, F_FULLFSYNC) + 133 #else + 134 #define redis_fsync(fd) fsync(fd) + 135 #endif +``` + +On Linux, `appendfsync always` sits on the **middle** rung (`fdatasync`). On +macOS the identical config line sits on the **top** rung (`F_FULLFSYNC`). The +measurement is only available on one of those platforms at a time — this +machine is a Mac, so `notes.md` records macOS `fsync` (22.67 µs) as the middle +rung and `F_FULLFSYNC` (2.97 ms) as the top, a **131×** gap. The Linux +`fdatasync` rung is not measured here; the safe statement is that redis on +macOS pays the top rung for the same config line that buys a middle rung on +Linux, and that on this hardware the two rungs are 131× apart. Redis is unusual +in reaching for the top rung at all, and honest about it: postgres ships +`F_FULLFSYNC` only behind a non-default `wal_sync_method`, and turso only behind +`PRAGMA fullfsync`. + +**Work the arithmetic.** Redis does not fsync per command — it fsyncs at most +once per event-loop iteration, so the AOF is *already* group-committed by the +loop. At an offered load of 100 000 write commands/s: + +``` +appendfsync always, macOS (F_FULLFSYNC, T = 2.967 ms) + flushes/s ceiling = 1 / 0.002967 = 337 + commands per flush = 100 000 × 0.002967 = 297 + added latency floor per command ≈ 2.97 ms + +appendfsync always, middle rung (macOS fsync measured here at T = 22.67 µs; + Linux fdatasync is the same rung but is NOT measured here) + flushes/s ceiling = 1 / 0.00002267 = 44 109 + commands per flush = 100 000 × 0.00002267 = 2.27 + added latency floor per command ≈ 23 µs + +appendfsync everysec (one bio fsync per second, off the command path) + commands per fsync = 100 000 + added latency floor per command ≈ 0 +``` + +So the old claim in this file — "an fsync per command, ~1 ms each, would cap it +at ~1K/s" — is wrong twice: redis never fsyncs per command, and "1 ms" is from +nowhere. What `always` actually costs is a **latency floor**, not a throughput +cap: 297 commands still ride each 2.97 ms flush. + +**The reply ordering is the contract, and it is not what the old pseudocode in +this file said.** `beforeSleep` flushes the AOF *before* it writes replies, and +says why: + +```c +// src/server.c — beforeSleep, 1958-1962 + 1958 /* Write the AOF buffer on disk, + 1959 * must be done before handleClientsWithPendingWrites and + 1960 * sendPendingClientsToIOThreads, in case of appendfsync=always. */ + 1961 if (server.aof_state == AOF_ON || server.aof_state == AOF_WAIT_REWRITE) + 1962 flushAppendOnlyFile(0); ``` -Sharpen the comparison with postgres: group commit *batches the flush but -never acks early* — the client waits until its LSN is durable. Everysec is -group commit with a *time*-based batch and the **ack before the flush** — -a different contract, not just different tuning. +`handleClientsWithPendingWrites()` runs at `server.c:1998`. So under `always` +the durability call completes *before* the client is told "OK" — a genuine +durable-before-ack, the same contract as postgres group commit. Under +`everysec` and `no`, only the `write()` completes first, and the ack is on the +page cache. + +**The window under `everysec` is not exactly one second.** Two constants set +it. The fsync is issued when at least 1000 ms have passed (`aof.c:1348`). But +if a background fsync is still running, redis postpones the *write* too, and +keeps postponing for up to 2000 ms (`aof.c:1196`) before giving up: + +```c +// src/aof.c — flushAppendOnlyFile's postponement, 1186-1204 (logging elided) + 1186 if (server.aof_fsync == AOF_FSYNC_EVERYSEC && !force) { + 1190 if (sync_in_progress) { + 1191 if (server.aof_flush_postponed_start == 0) { + 1194 server.aof_flush_postponed_start = server.mstime; + 1195 return; + 1196 } else if (server.mstime - server.aof_flush_postponed_start < 2000) { + 1199 return; + 1200 } + 1203 server.aof_delayed_fsync++; + 1204 serverLog(LL_NOTICE,"Asynchronous AOF fsync is taking too long (disk is busy?). ..."); +``` + +So the honest statement of the `everysec` window is: normally up to ~1 s of +acknowledged writes, stretching toward ~2 s when the disk cannot keep up — and +redis tells you when that happens, both in the log and in the +`aof_delayed_fsync` counter. + +**One more silent rung change.** With `no-appendfsync-on-rewrite yes`, the fsync +is skipped entirely while any child process is doing I/O (`aof.c:1326–1327`) — +during a rewrite or a BGSAVE, `always` quietly becomes `no`, i.e. the bottom +rung, for the duration. + +**And an opt-in durable ack.** Redis tracks `fsynced_reploff` +(`server.c:1970–1978`) so a client can issue `WAITAOF` and block until its write +is genuinely fsynced. That is the escape hatch: per-connection postgres +semantics without paying them server-wide. + +Sharpen the comparison with postgres: group commit *batches the flush but never +acks early* — the client waits until its LSN is durable. Redis's `always` is the +same contract at the granularity of an event loop. `everysec` is a genuinely +different contract: a *time*-based batch with the **ack before the flush**. + +*Why it matters:* this is the only design in the topic that lets an operator +choose the durability window, and the choice is legible only if you know which +rung the chosen policy lands on — which, for `always`, depends on the operating +system. ### Step 3 — the rewrite problem: command logs grow with history, not with data -A command log's size is proportional to *everything ever said*, not to the -data: 1M `INCR counter` commands is 1M log records describing one 8-byte -value. So the AOF must periodically be **rewritten** — replaced by the -shortest command sequence that reconstructs the *current* state (one `SET -counter 1000000`). Redis does this without pausing: **fork** the process -(the OS clones it; both copies share memory copy-on-write — see Step 5), -let the child serialize current state into a fresh **BASE** file at its -leisure, while the parent keeps serving and appends new commands to a new -**INCR** file. When the child finishes, BASE + INCR replace the old log. +> **In:** an AOF whose length is proportional to everything ever said. +> **Out:** a fresh BASE file holding the shortest command sequence that +> reconstructs current state, produced without pausing the server. + +A command log's size is proportional to *everything ever said*, not to the data: +1M `INCR counter` commands is 1M log records describing one 8-byte value. So the +AOF must periodically be **rewritten** — replaced by the shortest command +sequence that reconstructs the *current* state (one `SET counter 1000000`). + +Redis does this without pausing, by forking: + +```c +// src/aof.c — rewriteAppendOnlyFileBackground, 2664-2696 (error paths elided) + 2664 /* We set aof_selected_db to -1 in order to force the next call to the + 2665 * feedAppendOnlyFile() to issue a SELECT command. */ + 2666 server.aof_selected_db = -1; + 2667 flushAppendOnlyFile(1); + 2668 if (openNewIncrAofForAppend() != C_OK) { + 2689 if ((childpid = redisFork(CHILD_TYPE_AOF)) == 0) { + 2692 /* Child */ + 2693 redisSetProcTitle("redis-aof-rewrite"); + 2695 snprintf(tmpfile,256,"temp-rewriteaof-bg-%d.aof", (int) getpid()); + 2696 if (rewriteAppendOnlyFile(tmpfile) == C_OK) { +``` + +Read the ordering: a **forced** flush first (`flushAppendOnlyFile(1)` at +`:2667`, the `force` argument that bypasses the everysec postponement of Step +2), then a *new* INCR file is opened (`:2668`), and only then the fork +(`:2689`). The parent keeps serving and appends new commands to the new INCR +file; the child serialises current state into a fresh BASE at its leisure. When +the child finishes, BASE + INCR replace the old log. + +*Why it matters:* the ordering is the correctness argument. If the new INCR +were opened after the fork, commands executed in the gap would be in neither +file. ### Step 4 — multi-part AOF: an LSM in disguise -The modern (7.0+) AOF is not one file but a set — a **manifest** file lists -one BASE file plus one or more INCR files; recovery loads the BASE, then -replays the INCRs in order. Squint and this is topic 4 wholesale: BASE = -the bottom level (a compacted, sorted-out rendering of all history), INCR -files = L0 (recent appends), rewrite = full compaction, manifest = the -MANIFEST. Even the write amplification question transfers: a rewrite's cost -is (entire dataset serialized) per (INCR data absorbed) — exactly a -full-compaction WA. Topic 4's vocabulary was never LSM-specific; it's the -vocabulary of *any* log that must be compacted. +> **In:** one BASE file, N INCR files, and a manifest naming them. +> **Out:** at recovery, the current keyspace — by loading the BASE and then +> replaying each INCR in sequence order. + +The modern (7.0+) AOF is not one file but a set, described by redis's own +header comment: + +```c +// src/aof.c — the AOF Manifest file implementation, 48-70 + 48 * Append-only files consist of three types: + 50 * BASE: Represents a Redis snapshot from the time of last AOF rewrite. The manifest + 51 * file contains at most a single BASE file, which will always be the first file in the + 52 * list. + 54 * INCR: Represents all write commands executed by Redis following the last successful + 55 * AOF rewrite. In some cases it is possible to have several ordered INCR files. + 60 * HISTORY: After a successful rewrite, the previous BASE and INCR become HISTORY files. + 61 * They will be automatically removed unless garbage collection is disabled. + 63 * The following is a possible AOF manifest file content: + 65 * file appendonly.aof.2.base.rdb seq 2 type b + 66 * file appendonly.aof.1.incr.aof seq 1 type h + 69 * file appendonly.aof.4.incr.aof seq 4 type i + 70 * file appendonly.aof.5.incr.aof seq 5 type i +``` + +Squint and this is topic 4 wholesale: + +| redis | LSM equivalent | +|---|---| +| BASE (`type b`) | the bottom level — a compacted rendering of all history | +| INCR (`type i`) | L0 — recent appends, replayed in `seq` order | +| rewrite | full compaction | +| manifest | the MANIFEST | +| HISTORY (`type h`) | obsolete files awaiting GC | + +Note the BASE file's extension in redis's own example: `appendonly.aof.2.base.rdb`. +The BASE of an AOF is an **RDB** file (Step 5) when `aof-use-rdb-preamble` is on +— so a "modern AOF" is literally a checkpoint plus a tail of commands, which is +the structure of every WAL system in this topic. + +Even the write-amplification question transfers: a rewrite's cost is (entire +dataset serialised) per (INCR data absorbed) — exactly a full-compaction WA. +Topic 4's vocabulary was never LSM-specific; it is the vocabulary of *any* log +that must be compacted. + +*Why it matters:* it tells you what to measure. If you can express redis's +durability in topic-4 terms, you can price it with topic-4 arithmetic instead of +inventing new intuitions. ### Step 5 — RDB: checkpoint by fork, priced in COW -An RDB snapshot is durability by checkpoint alone: fork, and let the child -walk the entire keyspace writing a compact binary snapshot (with a CRC64 -trailer — a 64-bit checksum over the file), while the parent serves -traffic. Correctness is delegated to the OS: **copy-on-write** (COW) means -parent and child share all memory pages until the parent *writes* one, at -which point the kernel copies that 4 KB page — so the child sees a frozen -instant of the keyspace for free. The price is paid in page copies under -write load: a write-hot parent duplicates its working set, and worst case a -multi-GB dataset approaches 2× RAM during the snapshot. This cost reaches -all the way down into data-structure design: topic 2's dict *disables -rehashing during BGSAVE* (dict.c:1655), because a rehash touches every -bucket and would COW-copy the whole table. Durability window with RDB -alone: everything since the last snapshot — minutes. +> **In:** a live keyspace under write load. +> **Out:** a consistent point-in-time binary snapshot with a CRC64 trailer, +> written by a child process, paid for in copied memory pages. + +An RDB snapshot is durability by checkpoint alone: fork, and let the child walk +the entire keyspace writing a compact binary snapshot, while the parent serves +traffic. + +```c +// src/rdb.c — rdbSaveBackground, 1859-1878 (bookkeeping elided) + 1859 int rdbSaveBackground(int req, char *filename, rdbSaveInfo *rsi, int rdbflags) { + 1862 if (hasActiveChildProcess()) return C_ERR; + 1868 if ((childpid = redisFork(CHILD_TYPE_RDB)) == 0) { + 1871 /* Child */ + 1872 redisSetProcTitle("redis-rdb-bgsave"); + 1874 retval = rdbSave(req, filename,rsi,rdbflags); + 1878 exitFromChild((retval == C_OK) ? 0 : 1, 0); +``` + +Correctness is delegated to the OS: **copy-on-write** (COW) means parent and +child share all memory pages until the parent *writes* one, at which point the +kernel copies that page — so the child sees a frozen instant of the keyspace for +free. + +The file ends with a **CRC64 trailer** (a 64-bit checksum over the whole file, +written at `rdb.c:1702–1706`) and it is verified on load +(`rdb.c:4025–4038`, compare at `:4034`). That makes an RDB **all-or-nothing**: +a truncated or corrupt snapshot is rejected wholesale, where a truncated AOF is +merely replayed up to the last complete command. Two log formats, two different +answers to "what does a torn tail mean" — and both are defensible, because a +snapshot has no useful prefix while a command log does. + +The price of the fork is paid in page copies under write load: a write-hot +parent duplicates its working set, and worst case a multi-GB dataset approaches +2× RAM during the snapshot. This cost reaches all the way down into +data-structure design. Topic 2's dict does **not** simply "disable rehashing +during BGSAVE" — it raises the threshold: + +```c +// src/server.c — updateDictResizePolicy, 772-785 + 772 /* This function is called once a background process of some kind terminates, + 773 * as we want to avoid resizing the hash tables when there is a child in order + 774 * to play well with copy-on-write (otherwise when a resize happens lots of + 775 * memory pages are copied). ... */ + 778 void updateDictResizePolicy(void) { + 779 if (server.in_fork_child != CHILD_TYPE_NONE) + 780 dictSetResizeEnabled(DICT_RESIZE_FORBID); + 781 else if (hasActiveChildProcess()) + 782 dictSetResizeEnabled(DICT_RESIZE_AVOID); + 783 else + 784 dictSetResizeEnabled(DICT_RESIZE_ENABLE); + 785 } +``` + +Three states, not two. Inside the child, resizing is **forbidden** outright. In +the parent while a child lives, it is **avoided**: `dictExpandIfNeeded` +(`dict.c:1648–1660`) then only expands once used/buckets reaches +`dict_force_resize_ratio`, which is **4** (`dict.c:45`), instead of the normal +1:1. So a rehash during BGSAVE is not impossible, just four times less likely — +because a rehash touches every bucket and would COW-copy the whole table. + +Durability window with RDB alone: everything since the last snapshot — minutes. + +*Why it matters:* this is the topic's clearest case of a durability mechanism +whose real cost shows up somewhere else entirely — in RSS, and in a hash table's +load factor. ### Step 6 — the FalkorDB angle: this is the incumbent +> **In:** a graph whose adjacency matrices live in redis's keyspace as module +> data. +> **Out:** exactly the two mechanisms above, applied to a data structure +> neither was designed for — and the baseline your M5 design must beat. + A graph module's data lives inside redis's keyspace, so its durability *is* -this file: RDB serializes the matrices via module callbacks, and AOF logs -the `GRAPH.QUERY` commands themselves. Two consequences to quantify in -notes (they are the M5 comparison baseline): replaying `GRAPH.QUERY` -commands re-executes *parsing and planning* per command — estimate recovery -time for 10M mutations vs replaying logical records; and an RDB snapshot of -a multi-GB graph forks + COWs the whole matrix set under write load — -measure-or-estimate the stall and the memory spike. +this file: RDB serialises the matrices via module callbacks, and AOF logs the +`GRAPH.QUERY` commands themselves. Two consequences to quantify in `notes.md`, +because they are the M5 comparison baseline: + +1. **Replay re-executes parsing and planning.** A `GRAPH.QUERY` is not a data + mutation, it is a query to be compiled. Estimate recovery time for 10M + mutations replayed as `GRAPH.QUERY` text versus replayed as logical records, + using the per-command cost you can measure. +2. **A snapshot forks and COWs the whole matrix set.** Under write load, a + multi-GB graph approaches 2× RSS during BGSAVE, and the matrices are exactly + the kind of large contiguous allocation that COW handles worst + (one write dirties a page of a structure you then copy in full). + +*Why it matters:* M5 is not a greenfield design. It is an argument that a +purpose-built log beats a command log plus a fork, and that argument needs both +of the numbers above. ## Where each step lives in the code -- **Step 1 — `aof.c:1409–1448`**: `feedAppendOnlyFile` — every write - command appended (as RESP text!) to `server.aof_buf` (:1444). -- **Step 2 — `aof.c:1147–1355`**: `flushAppendOnlyFile` — buffer → write() - (:1218), then the policy (:1330–1354): `AOF_FSYNC_ALWAYS` (:1337); - `AOF_FSYNC_EVERYSEC` (:1350) — fsync on the bio background thread - (`aof_background_fsync`, :983); `AOF_FSYNC_NO`. The "postpone" logic near - :1147 delays *writes* when the bio fsync falls behind (question 1). -- **Steps 3–4 — `aof.c`**: `rewriteAppendOnlyFileBackground` :2652–2720 — - fork at :2689; the child serializes a fresh BASE, the parent accumulates - a new INCR. Multi-part AOF manifest: aof.c:45–71. -- **Step 5 — `rdb.c`**: `rdbSaveBackground` :1859–1892 — fork (:1868), - child walks the keyspace; CRC64 trailer rdb.c:1702–1706. The - rehash-disable during BGSAVE: dict.c:1655. +Anchors verified at redis/redis@a176d1225. + +- **Step 1 — `aof.c:1409–1448`**: `feedAppendOnlyFile`; the RESP serialisation + at `:1436` (`catAppendOnlyGenericCommand`, defined `:1357`); the buffer append + and its ordering comment `:1438–1445`. +- **Step 2 — `aof.c:1147–1355`**: `flushAppendOnlyFile`. Empty-buffer fast path + and the everysec catch-up `:1152–1181`; the write postponement `:1186–1205` + (2000 ms cap at `:1196`); the write `:1218`; `no-appendfsync-on-rewrite` + `:1326–1327`; the policy tail `:1329–1354` — `AOF_FSYNC_ALWAYS` `:1330–1345` + (`redis_fsync` `:1337`, `exit(1)` on failure `:1340`), `AOF_FSYNC_EVERYSEC` + `:1347–1353` (1000 ms interval `:1348`, `aof_background_fsync` `:1350`, + defined `:983` → `bioCreateFsyncJob`). Policy constants `server.h:634–636`. + `redis_fsync` `config.h:128–135`. Reply ordering `server.c:1958–1962` and + `:1998`; `WAITAOF` bookkeeping `server.c:1970–1978`. +- **Steps 3–4 — `aof.c`**: `rewriteAppendOnlyFileBackground` `:2652–2720` — + forced flush `:2667`, new INCR opened `:2668`, fork `:2689`, child writes a + temp BASE `:2695–2696`. Multi-part AOF manifest documentation `:42–71`; + naming constants `:73–75`. +- **Step 5 — `rdb.c`**: `rdbSaveBackground` `:1859–1892`, fork `:1868`; CRC64 + trailer written `:1702–1706`, verified on load `:4025–4038` (compare + `:4034`). COW-driven resize policy `server.c:772–785`; the threshold it + selects `dict.c:1648–1660` with `dict_force_resize_ratio = 4` at `dict.c:45`. ## Questions to answer in notes.md -1. everysec acks before durability. State the exact loss window and why redis - considers delaying *writes* (not acks) when the bio fsync falls behind - (:1147 area — the "postpone" logic). -2. AOF-as-LSM: map BASE/INCR/rewrite/manifest onto topic-4 terms. What's the - "write amp" of an AOF rewrite? -3. Command-log vs page-image vs logical-record WAL: rank recovery speed and - log volume for a graph-mutation workload; justify your M5 choice. +1. `everysec` acks before durability. State the exact loss window from the two + constants in the code (`aof.c:1348` and `aof.c:1196`), and explain why redis + postpones the *write* rather than the ack when the bio fsync falls behind. +2. `appendfsync always` is `fdatasync` on Linux and `F_FULLFSYNC` on macOS + (`config.h:128–135`). Using the ladder in `notes.md`, compute the added + per-command latency floor on each, at an offered load of 100 000 write + commands/s. Which platform's `always` would you be willing to run in + production, and what would you tell a user who benchmarked on the other one? +3. AOF-as-LSM: map BASE / INCR / rewrite / manifest onto topic-4 terms. What is + the "write amp" of an AOF rewrite, and how does `aof-use-rdb-preamble` change + the answer? +4. Command-log vs page-image vs logical-record WAL: rank recovery speed and log + volume for a graph-mutation workload; justify your M5 choice with the + numbers, not the vibe. ## Done when -You can state each appendfsync policy's durability window from memory and -explain AOF rewrite as compaction. +Answer each before unfolding it. + +- [ ] State each `appendfsync` policy's durability window, and say which one + acks the client before the data is durable. + +
Answer + + `always` — zero window; the durability call completes inside + `flushAppendOnlyFile`, which `beforeSleep` runs *before* + `handleClientsWithPendingWrites` precisely for this reason + (`server.c:1958–1962`, `:1998`). `everysec` — normally up to ~1 s + (`aof.c:1348` issues the fsync when 1000 ms have elapsed), stretching toward + ~2 s when a bio fsync is behind and writes are postponed (`aof.c:1196`); the + client is acked after the `write()`, so **this is the one that acks before + durability**. `no` — unbounded; redis issues no durability call and the + kernel's writeback timer decides (Linux default 30 s). + +
+ +- [ ] `appendfsync always` costs the same on Linux and macOS. True or false? + +
Answer + + False, and the gap is huge. `redis_fsync` is `fdatasync(fd)` on Linux and + `fcntl(fd, F_FULLFSYNC)` on macOS (`config.h:128–135`) — the middle and top + rungs of the ladder. On the machine in `notes.md` those two rungs measure + **22.67 µs** and **2.97 ms**, a **131×** gap (the middle rung there is macOS + `fsync`; `fdatasync` is compiled out on this platform, so the Linux figure is + not measured here). Only the macOS call actually flushes the drive's write + cache, so the macOS build is strictly more durable and strictly slower for the + identical config line. + +
+ +- [ ] Explain the AOF rewrite as compaction, and name the ordering constraint + that makes it correct. + +
Answer + + A command log grows with history, not with data, so it is periodically + replaced by the shortest command sequence that reproduces current state — a + full compaction, with BASE as the bottom level and INCR files as L0. The + ordering constraint is at `aof.c:2667–2689`: force-flush the current buffer, + **open the new INCR file, and only then fork**. Any command executed between + opening the INCR and forking is captured by both the new INCR and the child's + BASE (harmlessly, since replay is BASE-then-INCR); a command in the gap the + other way round would be in neither. + +
+ +- [ ] A truncated AOF and a truncated RDB behave differently at load time. Say + how, and why the difference is right. + +
Answer + + An RDB carries a CRC64 over the whole file (`rdb.c:1702–1706`), checked on + load (`rdb.c:4025–4038`); a truncated or corrupt one is rejected **wholesale**. + A truncated AOF is replayed up to the last complete command and the tail is + discarded. The difference is right because a snapshot has no useful prefix — + half a keyspace dump is not half a keyspace — while a command log's prefix is + exactly a valid earlier state. + +
+ +- [ ] Why does redis change a hash-table tuning parameter while a BGSAVE is + running? + +
Answer + + Copy-on-write. A rehash touches every bucket, so it would COW-copy the entire + table into the parent's private memory during the snapshot. + `updateDictResizePolicy` (`server.c:772–785`) therefore sets `DICT_RESIZE_AVOID` + in the parent while any child lives — and `DICT_RESIZE_FORBID` inside the child + itself. Under AVOID, `dictExpandIfNeeded` (`dict.c:1648–1660`) only expands at + `dict_force_resize_ratio = 4` (`dict.c:45`) instead of a 1:1 load factor. It is + not disabled, it is made four times less likely — a durability mechanism + reaching down and retuning a data structure two topics away. + +
## References -**Code** -- [redis](https://github.com/redis/redis) — `src/aof.c` (feed, flush - policies, rewrite, multi-part manifest) and `src/rdb.c` (fork + COW - snapshot, CRC64 trailer). Local clone at `~/repos/redis`. +**Code** — all anchors read at `redis/redis@a176d1225`; local clone at +`~/repos/redis`, pin recorded in `resources/codebases.md`. + +| file | what this chapter took from it | +|---|---| +| `src/aof.c` | the command log (Step 1), fsync policies and postponement (Step 2), rewrite (Step 3), the multi-part manifest (Step 4) | +| `src/config.h` | `redis_fsync` — which rung `appendfsync always` lands on, per platform (Step 2) | +| `src/server.c` | `beforeSleep`'s flush-before-reply ordering and `WAITAOF` bookkeeping (Step 2); `updateDictResizePolicy` (Step 5) | +| `src/server.h` | the three `AOF_FSYNC_*` constants (Step 2) | +| `src/rdb.c` | fork-based snapshot and the CRC64 trailer (Step 5) | +| `src/dict.c` | `dict_force_resize_ratio`, the COW-driven threshold change (Step 5) | + +**Measurements** — `topics/05-durability-wal/notes.md`, "Baseline (provided +lane, Apple M3 Pro / APFS, measured 2026-07-28)", produced by +`experiments/src/bin/fsync_ladder.rs`. `FINDINGS.md` row 5 carries the +headline. + +**Manual pages** — macOS `fsync(2)` for why `F_FULLFSYNC` exists; Linux +`proc(5)` / `vm.dirty_expire_centisecs` for the 30 s default behind +`appendfsync no`. diff --git a/topics/05-durability-wal/reading-turso-wal.md b/topics/05-durability-wal/reading-turso-wal.md index 84c4d02..4081cfa 100644 --- a/topics/05-durability-wal/reading-turso-wal.md +++ b/topics/05-durability-wal/reading-turso-wal.md @@ -2,59 +2,153 @@ This is SQLite's WAL mode in Rust: commits append whole page images as frames, a chained checksum makes the log's valid prefix self-evident, and recovery has -no redo or undo at all — it just decides where the log ends. Of the four +no redo and no undo at all — it just decides where the log ends. Of the four durability designs in this topic, this is the one your experiment should steal -most from — so before the code, this chapter builds it piece by piece: the -frame, the commit marker, the checksum chain and salts, the read path, the -checkpoint, and finally the recovery loop that all of them exist to make +most from, so before the code this chapter builds it piece by piece: the frame, +the commit marker, the checksum chain and salts, the sync call, the read path, +the checkpoint, and finally the recovery loop that all of them exist to make trivial. +Every line number below was read at **tursodatabase/turso@dd775bc** +(`tools/pinned-source.py show turso -r A:B`). Every timing comes from +this topic's provided lane, `cargo run --release --bin fsync_ladder`, on the +Apple M3 Pro / APFS machine recorded in `notes.md`. + +**Vocabulary, once, before it is used.** A *WAL* (write-ahead log) is a +sequential file that a change is written to, and made durable in, before the +page holding that change may be overwritten. A *frame* is turso's unit of WAL +content: one 24-byte header plus one whole database page. A *checkpoint* copies +frames back into the main database file so the WAL can be reused. *Idempotent +redo* means replaying a log record twice is harmless — turso gets it for free, +because replaying a page image just writes the same bytes again. And the two +durability calls this chapter distinguishes throughout, with their measured p50 +on this machine: + +| call | what it guarantees | measured p50 | +|---|---|---| +| `fsync()` (macOS) | bytes handed to the drive; the drive's volatile cache may still hold them | **22.67 µs** | +| `fcntl(fd, F_FULLFSYNC)` | the drive has flushed its cache to stable media | **2.97 ms** | + +**131× apart** — 44 109 versus 337 implied commits/s. Turso lets you choose +between them at runtime, which is why this chapter names the rung every time it +says "sync". + ## The problem in one sentence After `kill -9` mid-commit, the tail of the log file is arbitrary garbage — -half-written frames, stale bytes from a previous log generation — and -recovery must decide, from file contents alone, exactly which prefix of the -log to trust, with zero tolerance for accepting one corrupt or uncommitted -byte. +half-written frames, stale bytes from a previous log generation — and recovery +must decide, from file contents alone, exactly which prefix of the log to +trust, with zero tolerance for accepting one corrupt or uncommitted byte. ## The concepts, step by step ### Step 1 — the frame: log whole page images, not operations +> **In:** a transaction that has modified some set of pages in the buffer pool. +> **Out:** those pages appended verbatim to the WAL file, with the main +> database file untouched. + Turso's WAL is a file of **frames**, and a frame is a complete copy of one -4 KB database page plus a 24-byte header (which page number this is, plus -the fields in Steps 2–3). When a transaction commits, every page it -modified is appended to the WAL as a frame — the main database file is not -touched at all. Compare the alternatives from this topic: postgres logs -*deltas* ("change this tuple") and must replay them onto pages at recovery; -redis logs *commands* and must re-execute them. Page images are the -maximalist choice: the log **is** the data, so recovery needs no -replay logic whatsoever — and, as a free bonus, torn pages can't hurt you -(a half-written frame fails its checksum and is discarded whole; the -previous version of the page still exists untouched elsewhere). The cost is -volume: WAL bytes ∝ *pages touched*, not bytes changed — a 1-byte update -logs 4 KB. +database page (4 KB by default; the header allows "a power of two between 512 +and 65536 inclusive", `sqlite3_ondisk.rs:425–426`) plus a 24-byte header — +`WAL_FRAME_HEADER_SIZE = 24` at `sqlite3_ondisk.rs:405`. When a transaction +commits, every page it modified is appended to the WAL as a frame; the main +database file is not touched at all. + +Compare the alternatives from this topic: postgres logs *deltas* ("change this +tuple's field") and must replay them onto pages at recovery +(`reading-postgres-xlog.md` Step 4); redis logs *commands* and must re-execute +them (`reading-redis-aof-rdb.md` Step 1). Page images are the maximalist +choice, and they buy two things: + +- **Recovery needs no replay logic whatsoever.** Nothing is applied; frames are + the current version of their pages. +- **Torn pages cannot hurt you.** A half-written frame fails its checksum and + is discarded whole (Step 3), and the previous version of that page still + exists untouched — either earlier in the WAL or in the database file. This is + the whole of postgres's full-page-write machinery, obtained by construction + rather than by logging an extra 8 KB image after each checkpoint. + +The cost is volume: WAL bytes ∝ *pages touched*, not bytes changed. A one-byte +`UPDATE` that lands on one page writes 4 KB + 24 bytes. Price it against the +alternative before choosing it for M5: + +``` +one-byte update, 4 KB pages + + turso-style page image : 4 096 + 24 = 4 120 bytes + postgres-style delta : ~24 (header) + ~30 (block ref + payload) ≈ 55 bytes + + ratio ≈ 75× — but the delta owes you idempotent redo, an LSN on every + page, and full-page images after each checkpoint + +break-even: the page-image log wins as soon as a transaction dirties most of a +page, and whenever the engineering cost of a correct redo path matters more +than write bandwidth +``` + +*Why it matters:* every later step in this chapter is cheap *because* of this +choice. The format prepays the complexity that postgres and ARIES pay at +recovery time. ### Step 2 — the commit marker: db_size turns frames into transactions +> **In:** a run of frames in the WAL, some belonging to a completed +> transaction and some to one that was still being appended when the power +> went. +> **Out:** a single frame index that is the last committed state, derived from +> one `u32` field with no separate commit record. + A transaction is multiple frames, and the log needs to say where one ends — -otherwise recovery couldn't tell "committed" from "half-appended". The -frame header's **db_size** field does it with zero extra records: it is 0 -on ordinary frames and holds the database's new size (in pages) on the -*last* frame of a transaction. A frame with `db_size != 0` **is** the -commit record. Recovery's rule follows immediately: the state after a crash -is defined by the last valid frame with `db_size != 0` — frames after it -may be perfectly intact, but with no commit marker following them they are -invisible. No separate commit record, no transaction table. +otherwise recovery could not tell "committed" from "half-appended". The frame +header's **`db_size`** field does it with zero extra records. The struct +documents it exactly: + +```rust +// core/storage/sqlite3_ondisk.rs — WalFrameHeader, 477-500 (doc comments elided) + 477 pub struct WalFrameHeader { + 479 pub(crate) page_number: u32, + 483 pub(crate) db_size: u32, + 486 pub(crate) salt_1: u32, + 489 pub(crate) salt_2: u32, + 492 pub(crate) checksum_1: u32, + 495 pub(crate) checksum_2: u32, + 496 } + 497 + 498 impl WalFrameHeader { + 499 pub fn is_commit_frame(&self) -> bool { + 500 self.db_size > 0 +``` + +`db_size` is documented at `:481–482` as "For commit records, the size of the +database file in pages after the commit. For all other records, zero." — so a +frame with `db_size != 0` **is** the commit record, and `is_commit_frame()` +(`:499–500`) is literally `db_size > 0`. Six `u32`s, 24 bytes, and one of them +does the job postgres needs a commit record and a transaction id for. + +Recovery's rule follows immediately: the state after a crash is defined by the +last *valid* frame with `db_size != 0`. Frames after it may be byte-perfect, +but with no commit marker following them they are invisible. No separate commit +record, no transaction table, no two-phase anything. -### Step 3 — the checksum chain and the salts: making the valid prefix self-evident +*Why it matters:* atomicity here is a property of the reader, not of the +writer. The writer never has to make a multi-frame append atomic; the reader +just refuses to look past the last commit marker. + +### Step 3 — the checksum chain and the salts + +> **In:** a WAL file whose tail may contain a half-written frame, and whose +> middle may contain intact frames left over from a previous generation of the +> same file. +> **Out:** a single stopping point, computed with two `u32` compares and one +> arithmetic pass per frame. Each frame carries a checksum, but not an independent one — checksums are -**cumulative**: frame N's checksum is computed over frame N's contents -*seeded with frame N−1's checksum*. One flipped bit anywhere invalidates -that frame *and every frame after it* — which is exactly what you want: -the log's trustworthy prefix ends at the first bad checksum, and nothing -past a corruption can masquerade as valid. +**cumulative**: frame N's checksum is computed over frame N's contents *seeded +with frame N−1's checksum*. One flipped bit anywhere invalidates that frame +*and every frame after it*, which is exactly what you want: the log's +trustworthy prefix ends at the first bad checksum, and nothing past a +corruption can masquerade as valid. ``` WAL file: [hdr] [frame p5][frame p2][frame p9*] [frame p5][frame p1*] … @@ -64,125 +158,466 @@ past a corruption can masquerade as valid. fails the salt check even if its checksum chain looks plausible ``` +The writer builds the chain in two hops per frame, seeding the page checksum +with the header's: + +```rust +// core/storage/sqlite3_ondisk.rs — prepare_wal_frame, 2073-2088 + 2073 frame[0..4].copy_from_slice(&page_number.to_be_bytes()); + 2074 frame[4..8].copy_from_slice(&db_size.to_be_bytes()); + 2075 frame[8..12].copy_from_slice(&wal_header.salt_1.to_be_bytes()); + 2076 frame[12..16].copy_from_slice(&wal_header.salt_2.to_be_bytes()); + 2077 + 2078 let expects_be = wal_header.magic & 1; + 2079 let use_native_endian = cfg!(target_endian = "big") as u32 == expects_be; + 2080 let header_checksum = checksum_wal(&frame[0..8], wal_header, prev_checksums, use_native_endian); + 2081 let final_checksum = checksum_wal( + 2082 &frame[WAL_FRAME_HEADER_SIZE..WAL_FRAME_HEADER_SIZE + page_size as usize], + 2083 wal_header, + 2084 header_checksum, + 2085 use_native_endian, + 2086 ); + 2087 frame[16..20].copy_from_slice(&final_checksum.0.to_be_bytes()); + 2088 frame[20..24].copy_from_slice(&final_checksum.1.to_be_bytes()); +``` + +Read the seeds carefully: `prev_checksums` (the previous frame's result) seeds +the header checksum over `frame[0..8]`, which then seeds the page checksum. +Note what is *not* covered — bytes 8..16, the two salts. They are not +checksummed because they are compared directly, which is the point of Step 3's +second half. + +The checksum itself is not a CRC. `checksum_wal` +(`sqlite3_ondisk.rs:2169–2197`) is SQLite's two-word additive rolling sum — +`s0 = s0 + (v0 + s1); s1 = s1 + (v1 + s0)` over 8-byte groups +(`:2183–2184`), with a byte-swapping variant for the non-native endianness +(`:2189–2190`). It is a handful of adds per 8 bytes, which is why checksumming +a 4 KB page is not visible next to the write. It is also weaker than a CRC +against adversarial corruption — an acceptable trade for a format whose threat +model is a torn write, not an attacker. + The chain has one blind spot: the WAL file is *reused* after a checkpoint (reset, not deleted), so a frame from the file's previous life can sit at -exactly the right offset with an internally consistent checksum. The fix is -two **salts** — random values in the WAL header, regenerated on every WAL -reset and copied into every frame header. A frame whose salts don't match -the current header's is from a dead generation, whatever its checksum says. -Two cheap u32 comparisons close the hole. - -### Step 4 — commit means fsync — and on macOS, which fsync matters - -A commit is durable only when the OS confirms the frames reached stable -storage, so after appending the commit frame turso fsyncs the WAL file. The -trap this codebase makes explicit in its types: on macOS, plain `fsync()` -does **not** flush the drive's write cache — data can sit in the SSD's -volatile buffer and vanish on power loss. Turso's `FileSyncType` -distinguishes `Fsync` from `FullFsync` (the macOS `F_FULLFSYNC` fcntl, -which does flush the drive cache). Your fsync_ladder experiment will show -the gap; it's not subtle — plain fsync is ~100 µs-fast and weak, -F_FULLFSYNC is ms-scale and honest. A durability design that hasn't chosen -between them hasn't chosen its guarantee. +exactly the right offset with an internally consistent checksum. The fix is two +**salts** — values in the WAL header, copied into every frame header, and +changed on every WAL reset. `restart_snapshot_from_authority` +(`wal.rs:1747–1768`) bumps `checkpoint_seq` by one (`:1752`), increments +`salt_1` (`:1753`), and draws a **fresh random** `salt_2` (`:1754`), then resets +`max_frame` and `nbackfills` to 0 (`:1756–1757`). A frame whose salts don't +match the current header's is from a dead generation, whatever its checksum +says. Two `u32` comparisons close the hole. + +*Why it matters:* the chain answers "is this frame intact?" and the salts +answer "is this frame *mine*?". Neither question is answerable by the other's +mechanism, and a format that asks only the first is the classic WAL bug. + +### Step 4 — commit means sync — and which sync you chose is the guarantee + +> **In:** frames written to the WAL file descriptor, sitting in the OS page +> cache. +> **Out:** either a genuinely durable commit at **2.97 ms**, or a commit that +> survives process death but not power loss at **22.67 µs** — selected by one +> pragma. + +A commit is durable only when the frames have reached stable storage, so after +appending the commit frame turso syncs the WAL file. The codebase makes the +choice explicit in its type system rather than burying it: + +```rust +// core/io/mod.rs — FileSyncType, 124-134 + 124 /// Controls which sync mechanism to use for durability. + 125 /// `FullFsync` only has effect on Apple platforms (uses F_FULLFSYNC fcntl). + 126 /// On other platforms, both variants behave the same (regular fsync). + 127 #[derive(Debug, Clone, Copy, PartialEq, Eq, AtomicEnum)] + 128 pub enum FileSyncType { + 129 /// Regular fsync - flushes to disk but may not flush disk write cache on macOS. + 130 Fsync, + 131 /// Full fsync - on macOS uses F_FULLFSYNC to flush disk write cache. + 132 /// On other platforms, behaves the same as Fsync. + 133 FullFsync, + 134 } +``` + +and honours it at the syscall boundary — `core/io/unix.rs:455–472`, where on +Apple targets `Fsync` becomes `libc::fsync(fd)` (`:460`) and `FullFsync` +becomes `libc::fcntl(fd, libc::F_FULLFSYNC)` (`:462`), while on every other +target both call plain `fsync` (`:470`). The knob is `PRAGMA fullfsync` +(`core/translate/pragma.rs:716–726`), and the whole handler is +`#[cfg(target_vendor = "apple")]` — elsewhere there is nothing to choose. + +**The default is `Fsync`** (`core/storage/pager.rs:1680`, and +`get_sync_type()` is a compile-time constant `FileSyncType::Fsync` off Apple, +`:1780–1784`). So an out-of-the-box turso on macOS is on the **middle** rung: +22.67 µs per commit, 44 109 implied commits/s, and the write still in the +drive's volatile cache. `PRAGMA fullfsync=on` moves you to 2.97 ms and 337/s. +That is a 131× price, and it is the price of the guarantee most people assume +they already had. + +The sync sits inside a documented three-phase commit protocol +(`wal.rs:4148–4158`): *prepare* — serialise frames and compute checksums; +*write + fsync* — "caller submits I/O and waits for durability"; *commit* — +update the WAL index and page metadata. `prepare_wal_finish` +(`wal.rs:4130–4146`) shows why the ordering is not cosmetic: it only calls +`coordination.mark_initialized()` inside the completion callback, and only `if +res.is_ok()` (`:4139–4141`), because "a failed sync must leave the WAL +uninitialized so the header is re-issued before the next append" +(`:4136–4138`). + +*Why it matters:* a durability design that has not chosen between these two +calls has not chosen its guarantee, and the gap between them is larger than +almost any other decision in the engine. ### Step 5 — reads check the WAL first -Until frames are copied back into the database file, the newest version of -a page lives in the WAL — so every read consults an in-memory map of -page number → latest frame; a hit reads the frame from the WAL file, a miss -falls through to the main database file. The consequence worth pricing: a -big uncheckpointed WAL makes *reads* slower — more frames for the map to -cover, an extra lookup on every page access. Checkpointing (Step 6) is -therefore a read optimization, not just space reclamation. +> **In:** a read of page P by a transaction with a snapshot bounded by frames +> `[min_frame, max_frame]`. +> **Out:** either a frame number to read from the WAL, or `None`, meaning +> "read it from the database file". + +Until frames are copied back into the database file, the newest version of a +page lives in the WAL — so every read consults a frame index first. +`find_frame` (`wal.rs:3335–3405`) does three things in order: + +1. **Short-circuit.** If the reader holds read-lock 0 and the WAL has nothing + newer than the backfilled prefix, return `None` immediately and read the + database file (`:3364–3373`). +2. **Bound the search by the snapshot.** `min_frame` and `max_frame` + (`:3374–3375`) are this transaction's visible window; a frame outside it is + another transaction's and must not be seen. +3. **Look up.** Delegate to the coordination layer's frame index + (`:3393–3395`), which is a shared-memory index with a local scanned-cache + fallback used when the shared index has overflowed its reserved space + (`wal.rs:1925–1931` — "keep correctness by consulting the local scanned + cache"). + +A hit is followed by `read_frame` (`wal.rs:3409`) against the WAL file; a miss +falls through to the main database file. + +The consequence worth pricing: a big uncheckpointed WAL makes *reads* slower. +Every page access pays an index lookup, the index covers more frames, and the +pages themselves are scattered through a growing file instead of sitting at +their home offsets. Checkpointing (Step 6) is therefore a read optimisation, +not just space reclamation — which is the opposite of the intuition that a +checkpoint is pure overhead. + +*Why it matters:* it explains why SQLite-family engines have a +`wal_autocheckpoint` threshold at all. Left alone, read latency degrades with +WAL length, and no amount of write tuning fixes it. ### Step 6 — checkpoint: moving frames home -A checkpoint copies committed frames from the WAL back into the main -database file ("backfill"), after which the WAL can be reset. Turso -implements the four SQLite modes — Passive (copy what you can, never -block), Full, Restart, and Truncate (progressively stronger: finish the -backfill, force the next writer to start a fresh WAL, physically shrink the -file) — and copies frames **sorted by page number for locality**: the -database file is written in ascending page order, sequential-ish IO instead -of commit-order scatter. Restart/Truncate change the **salts** — that is -how every old frame in the reused file dies at once (Step 3) without a -single byte being erased. +> **In:** a WAL holding frames `[nbackfills+1 … max_frame]` and a database file +> that is behind. +> **Out:** those pages written into the database file, `nbackfills` advanced, +> and — in Restart/Truncate mode — a WAL whose entire contents have been +> invalidated at once by changing two `u32`s. + +A checkpoint copies committed frames from the WAL back into the main database +file ("backfill"), after which the WAL can be reset. Turso implements the four +SQLite modes (`CheckpointMode`, `wal.rs:154–171`), each documented in place: + +| mode | behaviour (paraphrasing `wal.rs:157–170`) | +|---|---| +| `Passive` | copy as many frames as possible without waiting for any reader or writer; never blocks either | +| `Full` | block until there is no writer and all readers are on the newest snapshot, then checkpoint everything and sync the DB file | +| `Restart` | as `Full`, then block until all readers read from the database file only, so the next writer restarts the log | +| `Truncate` | as `Restart`, then physically truncate the WAL file to zero bytes | + +`should_restart_log()` (`:174–179`) is true for exactly `Restart` and +`Truncate`; `require_all_backfilled()` (`:182–184`) is true for everything +except `Passive`. + +**The ordering, which this guide previously got backwards.** The work list is +built by `iter_latest_frames(min_frame, max_frame)` — the latest visible frame +per page — and then sorted: + +```rust +// core/storage/wal.rs — checkpoint_inner, CheckpointState::Start, 4668-4672 + 4668 let mut to_checkpoint = self + 4669 .coordination + 4670 .iter_latest_frames(oc_min_frame, oc_max_frame); + 4671 // sort by frame_id for read locality + 4672 to_checkpoint.sort_unstable_by(|a, b| (a.1, a.0).cmp(&(b.1, b.0))); +``` + +The list is `Vec<(u64, u64)>` of "page_id + frame_id combinations" +(`wal.rs:2470–2471`, destructured as `let (page_id, target_frame)` at `:4722`), +so `a.1` is the **frame id**: the sort orders the work by position in the WAL, +giving sequential *reads*, not ascending page order. + +Write locality is obtained separately, and more cleverly. Read pages accumulate +in `pending_writes`, a `BTreeMap>` keyed by page id, and +`write_pages_vectored` (`sqlite3_ondisk.rs:658`) coalesces consecutive page ids +into `writev` runs. Its own comment does the arithmetic: + +```rust +// core/storage/sqlite3_ondisk.rs — write_pages_vectored's contract, 648-658 + 648 /// Write a batch of pages to the database file. + 649 /// + 650 /// we have a batch of pages to write, lets say the following: + 651 /// (they are already sorted by id thanks to BTreeMap) + 652 /// [1,2,3,6,7,9,10,11,12] + 653 // + 654 /// we want to collect this into runs of: + 655 /// [1,2,3], [6,7], [9,10,11,12] + 656 /// and submit each run as a `writev` call, + 657 /// for 3 total syscalls instead of 9. + 658 pub fn write_pages_vectored( +``` + +Nine pages, three syscalls — a **3×** reduction on that example, and the +BTreeMap gives the ordering for free rather than by an explicit sort. So the +checkpoint gets sequential reads from the frame-id sort *and* sequential-ish +writes from the page-id map, on opposite sides of the same loop. + +Restart and Truncate then change the **salts** (Step 3, `wal.rs:1752–1754`) — +that is how every old frame in the reused file dies at once, without a single +byte being erased. + +*Why it matters:* checkpointing is where a WAL design's costs come due, and the +two sorts above are the difference between a checkpoint that streams and one +that random-seeks twice. ### Step 7 — recovery: find the cliff edge -Now the payoff — recovery is a single forward scan with no redo and no -undo: validate the WAL header's checksum, then walk frames in order, -checking salts (Step 3) and the cumulative checksum chain, remembering the -position of the last frame with `db_size != 0` (Step 2). First bad frame ⇒ -stop — that's the cliff edge; the answer is the last valid **commit**, not -the last valid frame. A half-written transaction's frames are physically -present but unreachable. The whole recovery, in one loop: +> **In:** a WAL file of unknown validity and a database file that may be behind +> it. +> **Out:** one number — `max_frame`, the index of the last valid commit frame — +> after a single forward pass with no redo and no undo. + +Now the payoff. Recovery validates the WAL header's checksum, then walks frames +in order, checking salts and the cumulative checksum chain, remembering the +position of the last frame with `db_size != 0`. First bad frame ⇒ stop. The +answer is the last valid **commit**, not the last valid frame. + +Both stopping conditions and the commit rule are in one loop +(`sqlite3_ondisk.rs:1790–1866`): ```rust -// Walk frames; the answer is the last valid COMMIT, not the last valid frame. -fn recover(frames: &[Frame], hdr: &WalHeader) -> u64 { - let mut c = hdr.checksum; - let mut last_commit = 0; - for (i, f) in frames.iter().enumerate() { - if f.salts != hdr.salts { break; } // stale frame from an old WAL generation - c = chain(c, f); // cumulative: one bad bit ends the log - if c != f.checksum { break; } // torn frame ⇒ the cliff edge - if f.db_size != 0 { // commit frame - last_commit = i as u64 + 1; // a half-written txn's frames stay - } // present but UNREACHABLE - } - last_commit -} +// core/storage/sqlite3_ondisk.rs — StreamingWalReader::process_frames, 1815-1862 +// (tracing::debug! calls elided) + 1815 if s1 != header.salt_1 || s2 != header.salt_2 { + 1827 break; + 1828 } + 1829 + 1830 let seed = checksum_wal(&fh[0..8], header, st.cumulative_checksum, use_native); + 1831 let calc = checksum_wal(page, header, seed, use_native); + 1832 if calc != (c1, c2) { + 1841 break; + 1842 } + 1843 + 1844 st.cumulative_checksum = calc; + 1845 let frame_idx = st.frame_idx; + 1846 st.pending_frames + 1847 .entry(page_no as u64) + 1848 .or_default() + 1849 .push(frame_idx); + 1850 + 1851 if db_size > 0 { + 1852 st.last_valid_frame = st.frame_idx; + 1853 st.last_valid_checksum = calc; + 1860 self.flush_pending_frames(&mut st); + 1861 } + 1862 st.frame_idx += 1; ``` -Place it on the topic's axis: postgres must *redo* (its log holds deltas, -not page images); ARIES must redo *and undo*; LMDB does nothing at all (the -meta-page flip made commit atomic). Turso's recovery is deciding where the -log ends — the entire complexity was prepaid in the format. +Trace the three exits. A zero page number stops the scan (`:1809–1813`). A salt +mismatch stops it (`:1815`) — that frame belongs to a previous generation. A +chained-checksum mismatch stops it (`:1832`) — that frame, and therefore +everything after it, is untrustworthy. And note `pending_frames` +(`:1846–1849`): frames are *staged* per page and only published by +`flush_pending_frames` when a commit frame arrives (`:1851–1860`). A +half-written transaction's intact frames are read, staged, and then simply +never published. + +`finalize_loading` (`:1893–1936`) commits the answer, and says the essential +thing in a comment: + +```rust +// core/storage/sqlite3_ondisk.rs — finalize_loading, 1903-1923 + 1903 let max_frame = st.last_valid_frame; + 1904 if max_frame > 0 { + 1905 let mut frame_cache = wfs.runtime.frame_cache.lock(); + 1906 for frames in frame_cache.values_mut() { + 1907 frames.retain(|&f| f <= max_frame); + 1908 } + 1921 wfs.metadata.max_frame.store(max_frame, Ordering::SeqCst); + 1922 // use checksum of last valid commit frame, not necessarily the last frame + 1923 wfs.metadata.last_checksum = st.last_valid_checksum; +``` + +The chain is resumed from the last valid **commit** frame's checksum (`:1923`), +not the last frame that happened to verify — so the next append continues a +chain that recovery will agree with. + +Place it on the topic's axis: postgres must *redo* (its log holds deltas, not +page images); ARIES must redo *and* undo; LMDB does nothing at all (the +meta-page flip made commit atomic). Turso's recovery is deciding where the log +ends — the entire complexity was prepaid in the format, in Steps 1–3. + +*Why it matters:* this is the cheapest correct recovery in the topic, and the +reason is not cleverness in the recovery code. It is that Steps 1, 2 and 3 each +removed a class of question recovery would otherwise have had to answer. ## Where each step lives in the code -- **Step 1 — the format**: `WalHeader` — sqlite3_ondisk.rs:411–443 (magic, - version, the two salts, header checksum); `WalFrameHeader` — :477–495 - (24 bytes — page_no, db_size, salts, checksum_1/2). -- **Steps 2–3 — frame write**: sqlite3_ondisk.rs:2058–2090 — cumulative - checksums, each frame's checksum seeding the next (:2080–2088). -- **Step 4 — commit + sync**: `prepare_wal_finish` — wal.rs:4130–4145 - (fsync after the commit frame); `FileSyncType` — io/mod.rs:128–134 - (`Fsync` vs `FullFsync`). -- **Step 5 — reads**: `find_frame` — wal.rs:3335–3404 (the in-memory - page→latest-frame map); hit ⇒ `read_frame` (:3409) from the WAL file. -- **Step 6 — checkpoint**: `CheckpointMode` — wal.rs:160–183 (Passive / - Full / Restart / Truncate); `checkpoint_inner` — wal.rs:4594–4672 — - backfill loop copies frames [nbackfills+1 … max_frame] into the DB file, - sorted by frame for locality; Restart/Truncate change the salts. -- **Step 7 — recovery**: `WalScan` — sqlite3_ondisk.rs:1426–1932: validate - header checksum (:1727), walk frames verifying salt + chained checksum - (:1830–1831), remember the last frame with `db_size > 0` (:1844–1855); - final state = last valid COMMIT (:1923), not last valid frame. +Anchors verified at tursodatabase/turso@dd775bc. + +- **Step 1 — the format**: `WAL_FRAME_HEADER_SIZE = 24` + `sqlite3_ondisk.rs:405`; `WalHeader` `:411–444` (magic, format version, page + size, `checkpoint_seq`, the two salts, header checksum); `WalFrameHeader` + `:472–496`. +- **Step 2 — commit marker**: `db_size` documented `:481–482`; + `is_commit_frame()` `:499–501`. +- **Step 3 — checksums and salts**: `prepare_wal_frame` `:2058–2091`, chain + seeding `:2080–2086`; `checksum_wal` `:2169–2197`; salt regeneration + `wal.rs:1747–1768` (`checkpoint_seq+1` `:1752`, `salt_1+1` `:1753`, random + `salt_2` `:1754`). +- **Step 4 — commit + sync**: `FileSyncType` `core/io/mod.rs:124–134`; the + syscalls `core/io/unix.rs:455–472` (`fsync` `:460`, `F_FULLFSYNC` `:462`, + non-Apple `:470`); `PRAGMA fullfsync` `core/translate/pragma.rs:716–726`; + default `Fsync` `core/storage/pager.rs:1680` and `:1780–1784`; + `prepare_wal_finish` `wal.rs:4130–4146`; three-phase protocol comment + `wal.rs:4148–4158`. +- **Step 5 — reads**: `find_frame` `wal.rs:3335–3405` — short-circuit + `:3364–3373`, snapshot window `:3374–3375`, delegation `:3393–3395`; index + with fallback `wal.rs:1915–1932`; `read_frame` `:3409`. +- **Step 6 — checkpoint**: `CheckpointMode` `wal.rs:154–171`, + `should_restart_log` `:174–179`, `require_all_backfilled` `:182–184`; + `checkpoint_inner` `:4594`, work list and frame-id sort `:4668–4672`, + `pages_to_checkpoint` type `:2470–2471`, processing loop `:4721–4774`; + `write_pages_vectored` `sqlite3_ondisk.rs:647–658`. +- **Step 7 — recovery**: `StreamingState` `sqlite3_ondisk.rs:1614–1625`; + header validation `handle_header_read` `:1703–1735` (checksum compare + `:1727`); `process_frames` `:1790–1866` — zero page `:1809`, salt `:1815`, + chained checksum `:1830–1832`, commit `:1851–1861`; `finalize_loading` + `:1893–1936`, the answer at `:1921–1923`. + (There is no `WalScan` type at this pin; the earlier version of this guide + named one.) ## Questions to answer in notes.md -1. Why do frames carry whole page images instead of deltas? Name the two - things this buys (no redo logic; torn-page immunity — a torn frame fails - its checksum and everything after is discarded) and the one it costs (WAL - volume ∝ pages touched, not bytes changed). +1. Why do frames carry whole page images instead of deltas? Name the two things + this buys (no redo logic; torn-page immunity — a torn frame fails its + checksum and everything after it is discarded) and the one it costs. Then do + the arithmetic from Step 1 for *your* M5 workload: bytes/txn under page + images versus under deltas. 2. Why salts AND checksums? Construct the failure that checksums alone miss. - (WAL reset reuses the file; an old frame at the right offset can have a - valid *internal* checksum — but chains from stale salts.) -3. For your experiment's WAL: page images or logical records? Decide and - justify with the M5 workload (small graph mutations ⇒ logical records win - on volume, but then you owe idempotent redo — LSN-stamped pages). + (WAL reset reuses the file; an old frame at the right offset can have a valid + *internal* checksum — but was chained from stale salts. Note that + `prepare_wal_frame` checksums `frame[0..8]` and the page, never the salt + bytes at 8..16.) +3. Turso's default is `FileSyncType::Fsync`. On the machine in `notes.md`, what + is the single-connection commit ceiling on the default, and on `PRAGMA + fullfsync=on`? Which of those two numbers would you quote in a durability + claim, and why? +4. For your experiment's WAL: page images or logical records? Decide and justify + with the M5 workload — small graph mutations ⇒ logical records win on volume, + but then you owe idempotent redo, which means an LSN on every page and a + full-page-image scheme for torn writes. ## Done when -You can narrate recovery over a WAL containing a torn frame mid-transaction -and a complete-but-uncommitted transaction, and say what survives (everything -up to the last valid commit frame; both damaged suffixes vanish). +Answer each before unfolding it. + +- [ ] Narrate recovery over a WAL that contains, in order: two committed + transactions, a complete-but-uncommitted transaction, and a torn frame. + Say what survives. + +
Answer + + The scan verifies the header checksum (`:1727`), then walks frames. Both + committed transactions pass the salt and chain checks; each ends in a frame + with `db_size > 0`, so at each one `last_valid_frame` advances and the staged + `pending_frames` are published (`:1851–1860`). The third transaction's frames + also verify and are staged — but no commit frame ever follows, so they are + never published. The torn frame fails the chained checksum and `break`s the + loop (`:1832–1842`). `finalize_loading` sets `max_frame` to the *second* + transaction's last frame and resumes the chain from *its* checksum (`:1923`). + Both damaged suffixes vanish; nothing is undone, because nothing was ever + applied. + +
+ +- [ ] Explain why a valid checksum is not sufficient to accept a frame. + +
Answer + + Because the WAL file is *reset*, not deleted, at Restart/Truncate checkpoints. + A frame from the file's previous life can sit at exactly the right offset with + a checksum that verifies against a chain seeded by the *old* header. The salts + are the generation stamp: `restart_snapshot_from_authority` + (`wal.rs:1747–1768`) increments `salt_1` and draws a fresh random `salt_2`, so + a stale frame's copied salts no longer match the header's and the scan stops + at `:1815` before it ever computes a checksum. + +
+ +- [ ] Turso's checkpoint sorts its work list. Say what it sorts by and what + that buys — and what gives it locality on the *other* side of the copy. + +
Answer + + It sorts by **frame id**, not page id — `sort_unstable_by(|a, b| (a.1, + a.0).cmp(&(b.1, b.0)))` with the comment "sort by frame_id for read locality" + (`wal.rs:4671–4672`). That makes the *reads* from the WAL sequential. Write + locality comes from a different mechanism: pages accumulate in a + `BTreeMap>` keyed by page id, and `write_pages_vectored` + merges consecutive runs into `writev` calls — its example turns nine pages + into three syscalls (`sqlite3_ondisk.rs:648–658`). + +
+ +- [ ] Someone benchmarks turso on a Mac, sees 40 000 commits/s, and calls it + durable. What is wrong? + +
Answer + + They are on the default `FileSyncType::Fsync` (`pager.rs:1680`), which on + Apple is `libc::fsync` (`io/unix.rs:460`). Turso's own doc comment says it + "may not flush disk write cache on macOS" (`io/mod.rs:129`). 40 000/s is right + next to the measured `fsync` ceiling of 44 109/s — the giveaway. Real + power-loss durability needs `PRAGMA fullfsync=on` + (`translate/pragma.rs:716–726`) → `fcntl(F_FULLFSYNC)` (`io/unix.rs:462`), + measured at 2.97 ms, a ceiling of **337/s**. Their number is 131× too high + for the guarantee they claimed. + +
+ +- [ ] Turso has no full-page-write machinery and no `checkpoint_timeout` + sawtooth. Say why, in one sentence, and name what it pays instead. + +
Answer + + Because it never overwrites a page in place: a commit appends whole page + images to the WAL, so a torn frame is discarded by its checksum and the + previous version of the page is still intact elsewhere — there is no + half-updated page for a full-page image to repair. It pays for this in steady + state instead of in bursts: every commit writes 4 KB + 24 bytes per dirtied + page, whatever the transaction changed, and read latency degrades with WAL + length until a checkpoint runs (Step 5). + +
## References -**Code** -- [tursodatabase/turso](https://github.com/tursodatabase/turso) — - `core/storage/wal.rs`, `core/storage/sqlite3_ondisk.rs`, - `core/io/mod.rs`. Local clone at `~/repos/turso`. +**Code** — all anchors read at `tursodatabase/turso@dd775bc`; local clone at +`~/repos/turso`, pin recorded in `resources/codebases.md`. + +| file | what this chapter took from it | +|---|---| +| `core/storage/sqlite3_ondisk.rs` | frame and header layout (Steps 1–2), checksum chain (Step 3), the recovery scan (Step 7), `write_pages_vectored` (Step 6) | +| `core/storage/wal.rs` | salt regeneration (Step 3), sync sequencing (Step 4), `find_frame` (Step 5), checkpoint modes and the backfill loop (Step 6) | +| `core/io/mod.rs`, `core/io/unix.rs` | `FileSyncType` and the two system calls behind it (Step 4) | +| `core/translate/pragma.rs`, `core/storage/pager.rs` | `PRAGMA fullfsync` and the default rung (Step 4) | + +**Format specification** — the on-disk layout is SQLite's, documented at +; turso's structs carry the same field names +and the same big-endian encoding, and its checkpoint modes match + (cited in the source at +`wal.rs:4798`). + +**Measurements** — `topics/05-durability-wal/notes.md`, "Baseline (provided +lane, Apple M3 Pro / APFS, measured 2026-07-28)", produced by +`experiments/src/bin/fsync_ladder.rs`. `FINDINGS.md` row 5 carries the +headline. diff --git a/topics/06-buffer-pool/reading-duckdb-buffer.md b/topics/06-buffer-pool/reading-duckdb-buffer.md index 6b37d93..852f629 100644 --- a/topics/06-buffer-pool/reading-duckdb-buffer.md +++ b/topics/06-buffer-pool/reading-duckdb-buffer.md @@ -1,164 +1,404 @@ # DuckDB's buffer pool: eviction by queue of hints -The interesting contrast with postgres: no fixed frame array, no CLOCK — -blocks are heap-allocated, tracked by `shared_ptr`, and eviction is a -concurrent FIFO queue of *hints* that are allowed to go stale. Re-pinning -never removes a queue entry; it invalidates one, and dead nodes get swept in -bulk. Mark now, collect later — the amortization move again, this time inside -the replacement policy itself. This chapter builds the design step by step, -then maps each piece to the C++ files. +The interesting contrast with postgres: no fixed frame array, no clock sweep +— blocks are heap allocations tracked by `shared_ptr`, and eviction is a +lock-free FIFO of *hints* that are allowed to go stale. Re-pinning never +removes a queue entry; it invalidates one, and the corpses get swept in bulk +later. Mark now, collect later — the amortization move again, this time +inside the replacement policy itself. This chapter builds the design step by +step, works out what the staleness actually costs in queue length, then maps +each piece to the C++. + +Read at [`duckdb/duckdb@6c0c1a68`](https://github.com/duckdb/duckdb), the +repo's pinned commit (pin table at the end of `resources/codebases.md`; local +clone at `~/repos/duckdb`). One naming note before you open the files: the +control object was split. **`BlockMemory`** (block_handle.hpp:32) owns the +residency state, the pin count and the eviction sequence number; +**`BlockHandle`** (:251) is the outer handle callers hold. The eviction queue +tracks `BlockMemory`. ## The problem in one sentence -An embedded analytics engine can't pre-allocate a fixed frame array (it +An embedded analytics engine cannot pre-allocate a fixed frame array — it shares RAM with a host process and juggles buffers from 256 KB row groups to -multi-GB hash tables), so DuckDB must track recency and enforce a memory -budget over *heap allocations of arbitrary size* — without a global lock and +multi-GB hash tables — so DuckDB must track recency and enforce a memory +budget over *heap allocations of arbitrary size*, without a global lock and without per-access list surgery. ## The concepts, step by step -### Step 1 — the BlockHandle: residency without frames +### Step 1 — the BlockMemory: residency without frames + +> **In:** nothing yet — this is the unit everything else operates on. +> **Out:** a pin count and a residency state, which Step 2 must observe +> without holding anything and Step 4 must re-verify before freeing. + +Postgres's unit is the **frame**: a fixed slot in a preallocated array +([`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md), Step 1). DuckDB +has no frames. Each block is a separate heap allocation, and the unit of +residency is a control object whose whole job is to answer two questions: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/include/duckdb/storage/buffer/block_handle.hpp, BlockMemory's residency accessors, 69-84 + 69 //! Returns true, if the block state is BLOCK_UNLOADED. + 70 bool IsUnloaded() const { + 71 return state == BlockState::BLOCK_UNLOADED; + 72 } + 73 //! Returns the number of readers. + 74 int32_t GetReaders() const { + 75 return readers; + 76 } + 77 //! Increments the number of readers prior to returning it. + 78 int32_t IncrementReaders() { + 79 return ++readers; + 80 } + 81 //! Decrements the number of readers prior to returning it. + 82 int32_t DecrementReaders() { + 83 return --readers; + 84 } +``` + +`readers` is an `atomic` (:222) — the **pin** count, "in use, don't +evict", the same concept as postgres's refcount and LeanStore's absence of +one. `state` is `BLOCK_LOADED` or `BLOCK_UNLOADED`: the bytes are in memory, +or they are not. Eviction's one question is answered by `CanUnload`: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/buffer/block_handle.cpp, CanUnload, 109-125 + 109 bool BlockMemory::CanUnload() const { + 110 if (GetState() == BlockState::BLOCK_UNLOADED) { + 111 // The block has already been unloaded. + 112 return false; + 113 } + 114 if (GetReaders() > 0) { + 115 // There are active readers. + 116 return false; + 117 } + 118 if (BlockId() >= MAXIMUM_BLOCK && MustWriteToTemporaryFile() && !GetBufferManager().HasTemporaryDirectory()) { + 119 // The block memory cannot be destroyed upon eviction/unpinning. + 120 // In order to unload this block we need to write it to a temporary buffer. + 121 // However, no temporary directory is specified, hence, we cannot unload. + 122 return false; + 123 } + 124 return true; + 125 } +``` -Postgres's unit is the frame — a fixed slot in a preallocated array. DuckDB -has no frames: each block of data is a separate heap allocation, and the -unit of residency is the **BlockHandle** — a small control object that says -whether the block's bytes are currently in memory (`BLOCK_LOADED`) or not -(`BLOCK_UNLOADED`), and how many users are reading it right now (an atomic -`readers` pin count — a pin being "in use, don't evict"). `CanUnload` -answers eviction's one question: loaded, unpinned, and not otherwise -protected. +Line 118 is Step 6 arriving early: some blocks have no home on disk to be +dropped back to, and if there is nowhere to spill them, they are simply not +evictable. -Callers never touch the counter directly: pinning returns a `BufferHandle`, -an RAII guard object whose destructor decrements `readers` — drop the guard -and the block becomes evictable. Rust translation: this is exactly a guard; -your buffer pool's `PageGuard` should work the same way. +Callers never touch `readers` directly. `Pin` returns a `BufferHandle`, an +RAII guard whose destructor unpins — drop the guard and the block becomes +evictable. That is exactly a Rust guard; the capstone's `PageGuard` should +work the same way. -Why it matters: no fixed array means memory can flow between the pool and -the rest of the process — but it also means eviction has no array to sweep -a CLOCK hand over. Something else must remember what's cold. That's Step 2. +Why it matters: no fixed array means memory can flow between the pool and the +rest of the process — but it also means eviction has no array to sweep a +clock hand over. Something else has to remember what is cold. That is Step 2. ### Step 2 — the eviction queue: a FIFO of hints, not truths -DuckDB's replacement policy is a concurrent FIFO queue that approximates -LRU: every time a block is *unpinned*, a `BufferEvictionNode` is pushed — -so blocks unpinned longest ago surface first. The trick is what a node -contains: +> **In:** Step 1's `BlockMemory`, at the moment it is unpinned. +> **Out:** a queue entry that may be a lie by the time anyone reads it — +> which Steps 3 and 4 exist to cope with. + +The replacement policy is a lock-free FIFO that approximates LRU: every time +a block is unpinned, a node is pushed, so blocks unpinned longest ago surface +first. The queue is `duckdb_moodycamel::ConcurrentQueue` +(buffer_pool.cpp:61) — a multi-producer/multi-consumer lock-free queue. The +interesting part is what a node holds: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/buffer/buffer_pool.cpp, BufferEvictionNode and its liveness test, 42-59 + 42 BufferEvictionNode::BufferEvictionNode(weak_ptr block_memory_p, idx_t eviction_seq_num) + 43 : memory_p(std::move(block_memory_p)), handle_sequence_number(eviction_seq_num) { + 44 D_ASSERT(!memory_p.expired()); + 45 } + 46 + 47 bool BufferEvictionNode::IsDeadNode(optional_idx debug_sleep_micros) { + 48 auto shared_memory_p = memory_p.lock(); + 52 if (!shared_memory_p) { + 53 return true; + 54 } + 55 if (handle_sequence_number != shared_memory_p->GetEvictionSequenceNumber()) { + 56 return true; + 57 } + 58 return false; + 59 } +``` + +Two fields, two failure modes: -- a **weak_ptr** to the block (a non-owning reference that can answer "is - this object still alive?" without keeping it alive — a `shared_ptr` here - would make the queue itself pin every block forever: the cache becomes a - leak), and -- the handle's `handle_sequence_number` *as of enqueue time* — a version - stamp. +- a **`weak_ptr`** (a non-owning reference that can answer "is this object + still alive?" without keeping it alive). A `shared_ptr` here would make the + queue itself pin every block forever — the cache would become a leak, which + is question 1 below. Line 52: if the upgrade fails, the block is gone and + the node is dead. +- the block's **eviction sequence number as of enqueue time** — a version + stamp, `atomic eviction_seq_num` (block_handle.hpp:231). Line 55: if + the block's current number has moved on, this node has been superseded. A queue entry is therefore just a *hint*: "this block was cold when I was -pushed." Nothing guarantees it's still true by the time eviction pops it. +pushed." Nothing guarantees it is still true when someone pops it. ### Step 3 — dead nodes: invalidate instead of remove -Here's the concurrency problem the design dodges: when a block is re-pinned -(it turned out to be hot), true LRU would remove its entry from the middle -of the queue — but removing from the middle of a concurrent FIFO needs a -lock or an O(n) search. DuckDB refuses: re-pin **never touches the queue**. -Instead, unpinning again later bumps the handle's sequence number and -enqueues a *fresh* node; the OLD node — still sitting in the queue! — now -has a stale sequence number and has become a **dead node**, a corpse that -eviction will recognize and skip. +> **In:** Step 2's hints, and a block that turns out to be hot again. +> **Out:** a queue that grows corpses, plus the bulk-collection policy that +> bounds how many — the arithmetic below is the point of this step. + +Here is the concurrency problem the design dodges. When a block is re-pinned, +true LRU would remove its entry from the middle of the queue — but removing +from the middle of a concurrent FIFO needs a lock or an O(n) search. DuckDB +refuses: **re-pin never touches the queue.** Unpinning later bumps the +sequence number and enqueues a *fresh* node, and the old node — still sitting +in the queue — now carries a stale number and has become a **dead node**, a +corpse Step 4 will recognise and skip. + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/buffer/buffer_pool.cpp, BufferPool::AddToEvictionQueue, 281-297 + 281 // Count the previous live entry before bumping the sequence number. PurgeIteration + 284 queue.IncrementDeadNodes(); + 296 BufferEvictionNode node(handle->GetMemoryWeak(), ts); + 297 return queue.AddToEvictionQueue(std::move(node)); +``` ``` - re-pin doesn't REMOVE the queue entry (that needs a lock or O(n) search); - it INVALIDATES it with a seq bump and re-enqueues later. + re-pin does NOT remove the queue entry (that needs a lock or an O(n) search); + it INVALIDATES it with a sequence bump and enqueues a fresh node later. → same amortization move as topic 2's incremental rehash and topic 4's tombstones: mark now, collect in bulk later. ``` -Corpses do pile up, so cleanup is amortized: `PurgeIteration` runs once per -`INSERT_INTERVAL = 4096` insertions and bulk-removes dead nodes — O(1) -amortized per operation instead of O(n) per re-pin. +Corpses pile up, so collection is amortized and bounded by four constants +that sit together in the source and are worth reading as a policy: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/buffer/buffer_pool.cpp, the purge policy constants, 115-124 + 115 //! We trigger a purge of the eviction queue every INSERT_INTERVAL insertions + 116 constexpr static idx_t INSERT_INTERVAL = 4096; + 117 //! We multiply the base purge size by this value. + 118 constexpr static idx_t PURGE_SIZE_MULTIPLIER = 2; + 119 //! We multiply the purge size by this value to determine early-outs. This is the minimum queue size. + 120 //! We never purge below this point. + 121 constexpr static idx_t EARLY_OUT_MULTIPLIER = 4; + 122 //! We multiply the approximate alive nodes by this value to test whether our total dead nodes + 123 //! exceed their allowed ratio. Must be greater than 1. + 124 constexpr static idx_t ALIVE_NODE_MULTIPLIER = 4; +``` -### Step 4 — the eviction loop: mostly corpse-skipping +`Purge` (:154) is entered by whichever thread's insertion hits the interval; +everyone else `try_lock`s and leaves (:156–159). `PurgeIteration` (:215) +dequeues `purge_size` nodes in bulk (:225), drops the dead ones, and +re-enqueues the survivors in bulk (:249). The loop's two early-outs are at +:198 and :207. + +**What that policy costs, as arithmetic.** All four constants are load-bearing: -With Steps 2–3 in place, evicting to free N bytes is a pop-and-verify loop — -each pop must survive three liveness checks before it frees anything: - -```rust -fn evict_until(&self, needed: usize) -> bool { - let mut freed = 0; - while freed < needed { - let Some(node) = self.queue.pop() else { return false }; - let Some(block) = node.block.upgrade() else { continue }; // weak_ptr: block - // already gone - if node.seq != block.eviction_seq.load() { continue; } // DEAD: re-pinned - // since enqueue - if !block.can_unload() { continue; } // pinned right now - freed += block.unload(); // write to temp file if no disk home - } - true -} ``` + purge_size = INSERT_INTERVAL × PURGE_SIZE_MULTIPLIER + = 4096 × 2 = 8,192 nodes swept per purge + amortized per unpin = 8,192 / 4,096 = 2 node inspections + (it sweeps twice what it inserts, which is what + stops the queue oscillating — comment at :176) + + minimum queue size before ANY purging happens (:169): + purge_size × EARLY_OUT_MULTIPLIER = 8,192 × 4 = 32,768 nodes + + tolerated corpse ratio — the loop keeps purging while (:207) + alive × (ALIVE_NODE_MULTIPLIER − 1) ≤ dead, i.e. 3·alive ≤ dead + so it stops once dead < 3·alive: + dead nodes may be up to 75% of the queue + the queue may be up to 4× the live set + + what that costs in bytes, for a 16 GB pool of 256 KB row groups: + live blocks 16 GiB / 256 KiB = 65,536 + queue at the 4× ceiling 262,144 nodes + node ≈ weak_ptr (2 pointers) + idx_t = 24 B + queue ≈ 6.3 MB = 0.04% of the pool +``` + +Two inspections per unpin, and 0.04% of the pool spent on corpses, to make +re-pin cost *nothing at all*. That is the trade, and it is a good one for +analytics — question 2 asks when it stops being one. + +### Step 4 — the eviction loop: mostly corpse-skipping -Why it matters: every check is lock-free — a weak_ptr upgrade, an atomic -load, a state check. The cost model inverts postgres's: hits and re-pins pay -nothing; the *evictor* pays for everyone's corpses. Question 2 below asks -when that trade loses. +> **In:** Step 2's queue of hints and Step 3's corpses. +> **Out:** freed bytes — or a failure, which Step 5 turns into a thrown +> query error rather than an OOM kill. + +Evicting is a pop-and-verify loop. Every popped node must survive three +liveness checks before anything is freed, and all three are lock-free: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/buffer/buffer_pool.cpp, EvictionQueue::IterateUnloadableBlocks, 467-506 + 467 for (;;) { + 468 // get a block to unpin from the queue + 469 BufferEvictionNode node; + 470 if (!q.try_dequeue(node)) { + 471 // we could not dequeue any eviction node, so we try one more time, + 472 // but more aggressively + 473 if (!TryDequeueWithLock(node)) { + 474 return; + 475 } + 476 } + 477 + 478 // get a reference to the underlying block pointer + 479 auto handle = node.memory_p.lock(); + 485 if (!handle) { + 486 DecrementDeadNodes(); + 487 continue; + 488 } + 489 + 490 // we might be able to free this block: grab the mutex and check if we can free it + 491 auto lock = handle->GetLock(); + 492 if (node.handle_sequence_number != handle->GetEvictionSequenceNumber()) { + 493 // A newer entry superseded this node: it was counted as dead when that entry was added. + 494 DecrementDeadNodes(); + 495 continue; + 496 } + 499 handle->SetHasLiveQueueEntry(lock, false); + 500 if (!handle->CanUnload()) { + 501 // The block cannot be unloaded right now (e.g. it is pinned). It gets a new queue + 502 // entry when it is unpinned again. + 503 continue; + 504 } + 505 + 506 if (!fn(node, handle, lock)) { +``` -One refinement: there isn't one queue but several, by buffer type -(`EVICTION_QUEUE_TYPES` in buffer_pool.hpp:116–122, in priority order) — -managed buffers and external file caches don't compete for survival in a -single FIFO. +Line 479: the block may already be gone (`weak_ptr` upgrade fails). Line 492: +it may have been re-pinned since (Step 3's corpse). Line 500: it may be +pinned *right now*, in which case the node is simply dropped — line 502 is +the design in one sentence: "It gets a new queue entry when it is unpinned +again." Nothing is ever put back to preserve ordering. + +The callback that actually frees is in `EvictBlocksInternal` (:391): it +early-returns if usage is already under the limit (:397), and otherwise +`Unload`s (:414) until it is (:416). There is a nice special case at +:406–408 — if the victim's allocation is exactly the size being requested, +the memory is handed over directly instead of being freed and re-malloc'd. + +Why it matters: the cost model inverts postgres's. Hits and re-pins pay +nothing; the *evictor* walks everyone's corpses. Postgres pays a usage-count +CAS on every hit so its clock hand never has to skip anything stale +([`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md), Step 4). + +One refinement: there is not one queue but **eight**, in three types — +`BLOCK_AND_EXTERNAL_FILE_QUEUE_SIZE = 1`, `MANAGED_BUFFER_QUEUE_SIZE = 6`, +`TINY_BUFFER_QUEUE_SIZE = 1` (buffer_pool.hpp:116–122), constructed in that +order at buffer_pool.cpp:255–266. Managed buffers are sharded six ways +because they are the contended case; blocks and tiny buffers are not. ### Step 5 — memory reservations: a gate in front of malloc -The memory budget is enforced *before* allocating, not observed after: -`EvictBlocksOrThrow` runs Step 4's loop until the requested reservation -fits under the limit, and if eviction can't free enough, the allocation -**throws** ("could not allocate block of size…") — the query fails rather -than the process OOMing. `Pin` composes the pieces: block loaded ⇒ -`readers++`; unloaded ⇒ reserve memory (evicting as needed), then reload -from disk or temp file. +> **In:** Step 4's eviction loop, as a subroutine. +> **Out:** either a reservation that fits the budget, or a thrown query +> error — never an over-limit allocation. + +The budget is enforced *before* allocating, not observed after: + +```cpp +// duckdb/duckdb@6c0c1a68 — src/storage/standard_buffer_manager.cpp, EvictBlocksOrThrow, 126-137 + 126 TempBufferPoolReservation StandardBufferManager::EvictBlocksOrThrow(QueryContext context, MemoryTag tag, + 127 idx_t memory_delta, unique_ptr *buffer, + 128 ARGS... args) { + 129 auto r = buffer_pool.EvictBlocks(context, tag, memory_delta, buffer_pool.maximum_memory, buffer); + 130 if (!r.success) { + 131 string extra_text = StringUtil::Format(" (%s/%s used)", StringUtil::BytesToHumanReadableString(GetUsedMemory()), + 132 StringUtil::BytesToHumanReadableString(GetMaxMemory())); + 133 extra_text += InMemoryWarning(); + 134 throw OutOfMemoryException(args..., extra_text); + 135 } + 136 return std::move(r.reservation); + 137 } +``` + +If eviction cannot free enough, line 134 throws — the *query* fails, with the +used/max numbers in the message, rather than the process being OOM-killed. +Callers supply the message: "could not allocate block of size %s" at :155 for +`Allocate`, "failed to pin block of size %s" at :364 for `Pin`. + +`Pin` (:337) composes everything: take the block's lock, and if the state is +`BLOCK_LOADED` just `Load` (increment readers) and return (:349–351). If not, +**drop the lock** — the comment at :338–340 explains that returning a +`BufferHandle` while holding the lock would deadlock on its destructor — run +`EvictBlocksOrThrow` (:362), then re-take the lock and *re-check* the state +(:369), because another thread may have loaded the block while eviction was +running. Two checks around one lock gap; the same shape as Step 4's three +re-verifications. Contrast the two accounting philosophies you now know: DuckDB gates -allocations up front; redis (reading-redis-zmalloc.md) counts after the +allocations up front, redis +([`reading-redis-zmalloc.md`](reading-redis-zmalloc.md)) counts after the fact and evicts keys asynchronously. ### Step 6 — spilling: the buffer pool doubles as the swap file -Not every buffer has a home on disk: hash-join tables and sort runs are -*temporary* — evicting them can't just drop the bytes. `WriteTemporaryBuffer` -sends evicted temporary data to the temp-file manager, and Step 5's reload -path brings it back on demand. This is why DuckDB joins bigger than RAM -work: eviction and spilling are one mechanism. Postgres spills per-operator -instead (each sort/hash gets `work_mem` and manages its own temp files) — -two philosophies of the same fallback. +> **In:** Step 4's decision to unload a block, and Step 1's line 118. +> **Out:** why larger-than-RAM joins work at all, and the design contrast +> with postgres's `work_mem`. + +Not every buffer has a home on disk. Hash-join tables and sort runs are +*temporary*: evicting them cannot just drop the bytes, because there is +nowhere to read them back from. `WriteTemporaryBuffer` +(standard_buffer_manager.cpp:501) hands them to the temp-file manager (:508), +and Step 5's reload path brings them back on demand. That is why DuckDB joins +bigger than RAM work at all — eviction and spilling are one mechanism, not +two. It is also why `CanUnload` returns false when there is no temporary +directory (block_handle.cpp:118): with nowhere to spill, an unspillable block +is pinned in effect. + +Postgres spills per operator instead: each sort or hash gets `work_mem` and +manages its own temp files, and the buffer pool never sees them. Two +philosophies of the same fallback — one budget shared and enforced centrally, +versus many budgets enforced locally. + +Both, note, are *the database deciding*. The alternative — letting the OS +page the working set out under you — is what this topic's lane measures: +p50 42 ns against a 182 µs maximum, a 4300× spread of stalls the database can +neither see nor schedule ([FINDINGS.md row 6](../../FINDINGS.md); +[`reading-mmap-paper.md`](reading-mmap-paper.md)). DuckDB throwing an +`OutOfMemoryException` is the deliberate opposite: a failure you can attribute +and a query you can retry with a bigger limit. ## Where each step lives in the code -Local clone at `~/repos/duckdb`: +Read `buffer_pool.cpp` top to bottom — it is 612 lines and contains the whole +policy — then dip into `standard_buffer_manager.cpp` for `Pin` and the +reservation path. | File | What | Steps | |------|------|-------| -| `src/include/duckdb/storage/buffer/block_handle.hpp` | BlockHandle | 1 | -| `src/storage/buffer/buffer_pool.cpp` | queue, purge, eviction | 2–4 | -| `src/include/duckdb/storage/buffer/buffer_pool.hpp` | purge cadence, queue types | 3–4 | -| `src/storage/standard_buffer_manager.cpp` | reservations, pin, spill | 5–6 | - -- **Step 1**: block_handle.hpp — `BlockState` - (BLOCK_LOADED/BLOCK_UNLOADED, :62–71), atomic `readers` pin count - (:73–87), `CanUnload` (:208). -- **Step 2**: `BufferEvictionNode` — buffer_pool.cpp:42 (weak_ptr + - `handle_sequence_number`). -- **Step 3**: `BufferPool::AddToEvictionQueue` — buffer_pool.cpp:271 (seq - bump + fresh node; old node goes dead — :284, IncrementDeadNodes); - `PurgeIteration` — buffer_pool.hpp:104, `INSERT_INTERVAL = 4096` :116. -- **Step 4**: `EvictBlocks`/`EvictBlocksInternal` — buffer_pool.cpp:377+ - (`IterateUnloadableBlocks` pops; dead-seq skip; weak_ptr-fail skip; - `Unload` at :38 in that loop); queue-per-type — - buffer_pool.hpp:116–122. -- **Step 5**: `EvictBlocksOrThrow` — standard_buffer_manager.cpp:126 - (throw at :155); `Pin` — :333/:337. -- **Step 6**: `WriteTemporaryBuffer` — standard_buffer_manager.cpp:501 - (temp-file manager handoff :508). +| `src/include/duckdb/storage/buffer/block_handle.hpp` | `BlockMemory` (:32) and `BlockHandle` (:251) | 1 | +| `src/storage/buffer/block_handle.cpp` | `CanUnload`, `Unload` | 1, 4 | +| `src/storage/buffer/buffer_pool.cpp` | node, queue, purge policy, eviction loop | 2–4 | +| `src/include/duckdb/storage/buffer/buffer_pool.hpp` | how many queues, of which types | 4 | +| `src/storage/standard_buffer_manager.cpp` | reservations, `Pin`, spilling | 5–6 | + +| Step | Symbol | Location | +|---|---|---| +| 1 | `BlockMemory` / `BlockHandle` class split | block_handle.hpp:32, :251 | +| 1 | `IsUnloaded`, `GetReaders`, `Increment/DecrementReaders` | block_handle.hpp:69–84 | +| 1 | `atomic readers`, `atomic eviction_seq_num` | block_handle.hpp:222, :231 | +| 1 | `GetEvictionSequenceNumber` / the bump | block_handle.hpp:111, :116 | +| 1 | `CanUnload` — declaration, then the three conditions | block_handle.hpp:208; block_handle.cpp:109–125 | +| 2 | `BufferEvictionNode` ctor and `IsDeadNode` | buffer_pool.cpp:42, :47–59 | +| 2 | the lock-free queue type | buffer_pool.cpp:61 | +| 3 | `BufferPool::AddToEvictionQueue` — count dead, then re-enqueue | buffer_pool.cpp:271, :284, :296 | +| 3 | `EvictionQueue::AddToEvictionQueue` returns "time to purge" | buffer_pool.cpp:144–147 | +| 3 | the four purge constants | buffer_pool.cpp:115–124 | +| 3 | `Purge` — single-purger `try_lock`, both early-outs | buffer_pool.cpp:154, :156, :169, :198, :207 | +| 3 | `PurgeIteration` — bulk dequeue, drop dead, bulk re-enqueue | buffer_pool.cpp:215, :225, :236, :249 | +| 4 | `IterateUnloadableBlocks` — the three checks | buffer_pool.cpp:465, :479, :492, :500 | +| 4 | `EvictBlocksInternal` — the callback that frees | buffer_pool.cpp:391, :397, :406, :414 | +| 4 | queue counts by type | buffer_pool.hpp:116–122; buffer_pool.cpp:255–266 | +| 5 | `EvictBlocksOrThrow` and the throw | standard_buffer_manager.cpp:126, :134 | +| 5 | the two caller messages | standard_buffer_manager.cpp:155, :364 | +| 5 | `Pin`, its deadlock comment, and the re-check after eviction | standard_buffer_manager.cpp:337, :338–340, :349, :369 | +| 6 | `WriteTemporaryBuffer` → temp-file manager | standard_buffer_manager.cpp:501, :508 | ## Questions to answer in notes.md @@ -171,17 +411,165 @@ Local clone at `~/repos/duckdb`: are pinned. Trace where each behavior comes from and which your capstone pool should adopt (server vs embedded assumptions). +## Takeaway + +Take the frame array away and the clock hand goes with it, so recency has to +live somewhere else: a lock-free FIFO of weak references plus version stamps. +Every entry is a hint that may be stale, and the whole design follows from +refusing to fix stale entries eagerly — re-pin is free, the evictor skips +corpses, and a bulk purge every 4,096 insertions keeps the queue within 4× of +the live set. The budget is then enforced at the only place it can be +enforced without a frame array: in front of `malloc`, with a thrown query +error when eviction cannot make room. + ## Done when -You can explain a dead node, the 4096-insert purge cadence, and why re-pin -never touches the queue — and name the postgres structure each replaces. +Answer each before unfolding it. + +- [ ] You can define a dead node, name its two causes, and say which line detects each. + +
Answer + + A dead node is a queue entry that no longer describes reality. Two causes, + both in `IsDeadNode` (buffer_pool.cpp:47–59) and again inline in + `IterateUnloadableBlocks`: + + 1. **The block is gone.** `memory_p.lock()` on the `weak_ptr` fails — line + 52 in `IsDeadNode`, line 479/485 in the eviction loop. The `BlockMemory` + was destroyed while the node sat in the queue. + 2. **The block was re-pinned and re-enqueued.** The node's + `handle_sequence_number` no longer equals the block's current + `GetEvictionSequenceNumber()` — line 55, and line 492 in the loop. A + newer node for the same block exists further back in the queue, so this + one is a corpse. + + Both paths call `DecrementDeadNodes()` when found, because the corpse was + counted at `IncrementDeadNodes()` (buffer_pool.cpp:284) the moment it was + superseded. + +
+ +- [ ] You can explain why re-pin does not remove the block's queue entry, and what it does instead. + +
Answer + + Because the queue is a lock-free MPMC FIFO + (`duckdb_moodycamel::ConcurrentQueue`, :61) and removing from the middle of + one requires either a lock or an O(n) scan — either of which would put a + contended operation on the hot path, where re-pins are frequent. + + Instead the entry is *invalidated*: `BufferPool::AddToEvictionQueue` (:271) + counts the previous live entry as dead (:284) and bumps the block's + eviction sequence number, then pushes a fresh node carrying the new number + (:296–297). The stale node stays in the queue until someone pops it, and + whoever does — either `PurgeIteration` or the eviction loop — throws it + away in O(1). The cost of correctness has been moved from the frequent + operation (re-pin, now free) to the rare one (purge, batched). + +
+ +- [ ] You can compute the purge cadence, the amortized work per unpin, and the corpse ratio the policy tolerates. + +
Answer + + From the constants at buffer_pool.cpp:115–124: + + - **Cadence**: a purge is triggered when insertions hit a multiple of + `INSERT_INTERVAL = 4096` (:146). One thread wins the `try_lock` at :156; + the rest return immediately. + - **Work per purge**: `purge_size = INSERT_INTERVAL × PURGE_SIZE_MULTIPLIER` + = 8,192 nodes, dequeued in bulk (:225). So **2 node inspections per + unpin**, amortized. It deliberately sweeps twice what it inserted — the + comment at :176 says this is what stops the queue oscillating around the + trigger. + - **Floor**: nothing is purged while the queue is below + `purge_size × EARLY_OUT_MULTIPLIER` = 32,768 nodes (:169), to keep the + LRU characteristic (:168). + - **Corpse ratio**: the aggressive loop stops when + `alive × (ALIVE_NODE_MULTIPLIER − 1) > dead` (:207), i.e. once + `dead < 3 × alive`. So up to **75% of the queue may be corpses**, and the + queue may be up to 4× the live set. For a 16 GB pool of 256 KB row groups + (65,536 live blocks) that is 262,144 nodes of ~24 bytes ≈ 6.3 MB, or + 0.04% of the pool. + +
+ +- [ ] You can name the postgres structure that each DuckDB piece replaces, and say what postgres pays instead. + +
Answer + + | DuckDB | postgres | who pays | + |---|---|---| + | heap allocation + `BlockMemory` | fixed 8 KB frame in the `shared_buffers` array | postgres pays a fixed, pre-committed budget; DuckDB pays per-allocation bookkeeping | + | `atomic readers` | the 18-bit refcount inside the packed state word | same idea, same cost | + | eviction queue of hints | `nextVictimBuffer` + 4-bit usage counts | postgres pays a usage bump inside the pin CAS on **every hit**; DuckDB pays nothing on hits and makes the evictor skip corpses | + | `PurgeIteration` every 4,096 inserts | nothing — the clock array never grows stale | DuckDB's amortized 2 inspections/unpin is the price of not having an array | + | `EvictBlocksOrThrow` | no equivalent; postgres errors only when *all* buffers are pinned (freelist.c:274) | DuckDB fails the query at the limit; postgres relies on the fixed array making the limit unreachable | + | `WriteTemporaryBuffer` | per-operator `work_mem` and private temp files | one central budget vs many local ones | + +
+ +- [ ] You can say what DuckDB does when the budget cannot be met, and why that is the right failure for an embedded engine. + +
Answer + + `EvictBlocksOrThrow` (standard_buffer_manager.cpp:126) runs the eviction + loop first, and if `r.success` is false it throws an `OutOfMemoryException` + at :134 with the used and maximum figures formatted into the message. The + allocation never happens; the query dies, the process does not. + + That is right for an embedded engine because DuckDB is a library inside + someone else's process. Exceeding the budget would not just harm DuckDB — + it would take the host application down with an OOM kill, at a moment the + host cannot attribute or handle. A thrown exception is attributable + (memory tag, used/max in the message), catchable, and retryable with a + higher `memory_limit`. Postgres can afford the opposite default because it + owns its processes and its `shared_buffers` array is preallocated, so the + limit is enforced by construction rather than by checking. + +
+ +- [ ] You wrote answers to all three questions in notes.md. + +
Answer + + Nothing to unfold. Question 2 is the one with real content: the worst case + is `N` corpses per block re-pinned `N` times between purges, bounded in + practice by the 3:1 dead:alive rule at :207 — so the honest answer names + the workload where that bound is reached (many small, repeatedly re-pinned + blocks) and compares it to a clock array, which has *no* stale state to + sweep because its metadata lives with the frame instead of in a queue. + +
## References -**Code** -- [duckdb/duckdb](https://github.com/duckdb/duckdb) — - `src/storage/buffer/buffer_pool.cpp`, - `src/storage/standard_buffer_manager.cpp`, - `src/include/duckdb/storage/buffer/buffer_pool.hpp`, - `src/include/duckdb/storage/buffer/block_handle.hpp`. Local clone at - `~/repos/duckdb`. +**Code** — [duckdb/duckdb](https://github.com/duckdb/duckdb) at `6c0c1a68`. +Local clone at `~/repos/duckdb`; the pin table is at the end of +`resources/codebases.md`. + +| File | Lines | What | +|---|---|---| +| `src/include/duckdb/storage/buffer/block_handle.hpp` | 32, 251 | the `BlockMemory` / `BlockHandle` split | +| `src/include/duckdb/storage/buffer/block_handle.hpp` | 69–87 | residency state and the pin count | +| `src/include/duckdb/storage/buffer/block_handle.hpp` | 111–116, 222, 231 | the eviction sequence number and `readers` | +| `src/storage/buffer/block_handle.cpp` | 109–125 | `CanUnload`'s three conditions | +| `src/storage/buffer/buffer_pool.cpp` | 42–61 | the node, `IsDeadNode`, the queue type | +| `src/storage/buffer/buffer_pool.cpp` | 115–124 | the four constants that define the purge policy | +| `src/storage/buffer/buffer_pool.cpp` | 144–251 | `AddToEvictionQueue`, `Purge`, `PurgeIteration` | +| `src/storage/buffer/buffer_pool.cpp` | 255–266 | eight queues in three types | +| `src/storage/buffer/buffer_pool.cpp` | 271–297 | invalidate-then-re-enqueue | +| `src/storage/buffer/buffer_pool.cpp` | 391–432 | `EvictBlocksInternal` | +| `src/storage/buffer/buffer_pool.cpp` | 465–510 | `IterateUnloadableBlocks` — the three checks | +| `src/include/duckdb/storage/buffer/buffer_pool.hpp` | 116–122 | `EVICTION_QUEUE_TYPES` and the per-type counts | +| `src/storage/standard_buffer_manager.cpp` | 126–137 | `EvictBlocksOrThrow` | +| `src/storage/standard_buffer_manager.cpp` | 337–375 | `Pin`: lock, drop, evict, re-check | +| `src/storage/standard_buffer_manager.cpp` | 501–508 | spilling to the temp-file manager | + +**Related** +- [`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md) — the fixed-array + design this one is the inverse of. +- [`reading-redis-zmalloc.md`](reading-redis-zmalloc.md) — the third + accounting philosophy: count after the fact. +- [FINDINGS.md row 6](../../FINDINGS.md) — what happens when nobody enforces + a budget and the OS pages for you. diff --git a/topics/06-buffer-pool/reading-leanstore-paper.md b/topics/06-buffer-pool/reading-leanstore-paper.md index bef6c12..ba176e2 100644 --- a/topics/06-buffer-pool/reading-leanstore-paper.md +++ b/topics/06-buffer-pool/reading-leanstore-paper.md @@ -6,181 +6,524 @@ pointer swizzling, a cooling stage, and optimistic latches; vmcache (SIGMOD '23), from the same group, is "what we'd do differently five years later" — same goal, mechanism moved into the MMU. This chapter builds the ideas one at a time — the tax a classic pool charges on every hit, then the -three LeanStore ingredients that zero it out, then vmcache as the -retraction-and-fix — before pointing you at the sections that matter. +three LeanStore ingredients that zero it out, then what the ablation actually +measured, then vmcache as the retraction-and-fix — before pointing you at the +sections that matter. + +Section numbers below are Leis, Haubenschild, Kemper, Neumann, +*"LeanStore: In-Memory Data Management Beyond Main Memory"* (ICDE 2018, +12 pages), and Leis, Alhomssi, Ziegler, Loeck, Dietrich, +*"Virtual-Memory Assisted Buffer Management"* (SIGMOD 2023, 14 pages). Every +number carries the section, figure or table it came from. The vmcache code +quoted in Step 8 is the authors' reference implementation +[`viktorleis/vmcache`](https://github.com/viktorleis/vmcache) at commit +`1157828`, with the line numbers it occupies there. ## The problem in one sentence -A classic buffer pool charges a hash lookup + a latch + a pin-count update -on *every page access, even when the page is already in RAM* — measured on -in-memory TPC-C, that overhead is a large fraction of total runtime, which -is why in-memory systems like HyPer simply deleted the buffer manager and -gave up larger-than-RAM data to get the speed back. +A classic buffer pool charges a hash lookup plus a latch plus a pin-count +update on *every page access, even when the page is already in RAM* — and the +LeanStore authors' own ablation puts a number on it: rebuilding LeanStore +with a translation hash table, LRU and traditional latches drops +single-threaded TPC-C from **67K to 30K transactions/s**, and ten-threaded +TPC-C from **597K to 18K** (LeanStore Fig. 7), which is why in-memory systems +like HyPer simply deleted the buffer manager and gave up larger-than-RAM data +to get the speed back. ## The concepts, step by step ### Step 1 — the per-access tax of a classic buffer pool -A buffer pool (the fixed-size in-memory cache of disk pages that the engine -manages itself) translates every page reference through a map: given a -**page id** (the page's number in the file), find the **frame** (the RAM -slot currently holding it). In postgres that means, on every access: hash -the page id, take a partition lock, probe the table, then **pin** the frame -(atomically bump a reference count so eviction can't take it while you look -at it), and unpin after. That's ~2 atomic operations plus a probable cache -miss on the hash bucket — per access, forever, even when 100% of the data -is in RAM and none of this machinery ever does anything useful. +> **In:** nothing yet — this step fixes the vocabulary and prices the thing +> the whole design is trying to delete. +> **Out:** three named costs (translation, replacement bookkeeping, pinning), +> each of which Steps 2, 4 and 5 remove in turn. + +A **buffer pool** is the fixed-size in-memory cache of disk pages the engine +manages itself. It exists to answer one question on every access: given a +**page id** — the page's number in the file, its permanent name — find the +**frame**, the RAM slot currently holding it. In the canonical design (§I, +citing Effelsberg and Härder) that means a hash table lookup per access, and +in typical implementations the structures involved are protected by several +**latches** — short-lived locks over in-memory structures, as opposed to +transactional locks over data. + +Three costs, then, on the hot path of every hit: + +1. **Translation** — hash the page id, take the partition latch, probe. +2. **Replacement bookkeeping** — LRU list surgery, or a Second Chance bit + to set. §III-B: "for frequently accessed pages (e.g., B-tree roots), + updating access tracking information … sometimes becomes a scalability + bottleneck." +3. **Pinning** — `pinPage` increments a per-page reference count so eviction + cannot take the frame while you hold a pointer into it; `unpinPage` + decrements. §III-C counts the damage in Shore-MT: **15 latch acquisitions + for a single-row update transaction**, and names the mechanism — + "cacheline ping-pong", where a line holding a hot latch bounces between + cores because cache coherency must serialize writes to it. + +How big is that? Two measurements, from the two papers. + +LeanStore's Fig. 7 ablation, single-threaded and ten-threaded TPC-C, 100 +warehouses, 16 KB pages, on an Intel Xeon E5-2687W v3 (10 cores, 20 threads): -LeanStore's goal (§I–II): **pay for translation and replacement only on -misses** — the hot path should look like an in-memory system. Three -ingredients, each killing one cost: +``` + baseline +swizzling +lean evict +opt. latch + 1 thread 30K 48K 62K 67K +10 threads 18K 23K 109K 597K + + what the classic design costs, as division (LeanStore Fig. 7): + 1 thread: 1 − 30/67 = 55% of the achievable throughput given up + 10 threads: 597/18 = 33× — and note 18K < 30K: the traditional + design got SLOWER with ten threads than one +``` + +vmcache's Table 2 prices the same tax as a microbenchmark of pure hits — +average instructions, cache misses and latency for a random 4 KB page access, +over 128 GB of data: ``` - 1. pointer swizzling translation cost → 0 (parent holds raw pointer) - 2. cooling stage replacement cost → 0 (no per-access bookkeeping; - random candidates + second-chance FIFO) - 3. optimistic latches pinning cost → 0 (readers validate versions, - hold nothing) + instructions cache misses time + plain memory read 3.3 1.0 219 ns + vmcache full access logic 10.4 2.0 236 ns + hash table (unsynchronized) 27.9 2.6 336 ns + + hash table over a plain read: 336/219 = 1.53×, +117 ns per access + hash table over vmcache: 336/236 = 1.42×, +100 ns per access + instruction ratio: 27.9/10.4 = 2.7× ``` -### Step 2 — pointer swizzling: the parent's pointer IS the translation +The paper's own caveat matters: "our hash table implementation is not +synchronized, and the shown overhead is therefore actually a lower bound for +the true cost of any hash table based design" (§5.4). The 100 ns is what a +*single-threaded, latch-free* hash table costs. -**Swizzling** means storing, in the place where a page id would go, the -actual in-memory pointer to the frame — so following a B-tree parent-to-child -link is one pointer dereference, zero lookups. Each reference slot (a -**swip**) is one u64 that is *either* a raw `BufferFrame*` (page resident: -"hot") *or* a page id with a tag bit set (page on disk: "evicted"). The -buffer pool's mapping table is thereby distributed into the data structure -itself; there is no central hash table on the hot path at all. +LeanStore's goal (§I) is to pay for translation and replacement **only on +misses**. Three ingredients, each killing one of the three costs: -Why it matters: a hot access costs literally what an in-memory system -charges — a dereference. The cost moved entirely to the miss, where a disk -read dwarfs it anyway. +``` + 1. pointer swizzling translation cost → 0 (parent holds a raw pointer) + 2. cooling stage replacement cost → 0 (no per-access bookkeeping; + random candidates + a second-chance FIFO) + 3. optimistic latches pinning cost → 0 (readers validate versions, + write nothing) +``` -### Step 3 — the price of swizzling: one parent, bottom-up eviction +Why it matters: those three lines are the paper's whole structure, and Step 7 +checks each against the ablation that measured it. -If two swips pointed at the same page, evicting it would require finding -and un-swizzling *both* raw pointers — but there's no central table to find -them with. So LeanStore imposes the **one-swip-per-page rule**: every page -has exactly one owner reference. That's natural for a B-tree (each node has -one parent) and awkward for anything graph-shaped. +### Step 2 — pointer swizzling: the parent's pointer IS the translation -Same logic forces **bottom-up eviction**: a parent may only be evicted -after all its children are — an evicted parent's swip slot holds a page id, -and a page id can't point at a hot child's frame; the child would become -unreachable. (§III.B is this argument; hold it for capstone question 4 — -GraphBLAS tiles referenced by row *and* column are a DAG, not a tree.) +> **In:** the translation cost from Step 1. +> **Out:** the **swip** — the 8-byte reference that is either a pointer or a +> page id — which Steps 3 and 4 then have to evict, and which Step 8's +> vmcache deletes again. + +**Pointer swizzling** means storing, in the slot where a page id would go, +the actual in-memory pointer to the frame. Following a B-tree parent-to-child +link is then one dereference and zero lookups. The paper's name for that slot +is a **swip**: "the reference, i.e. the 8-byte memory location referring to a +page" (§IV-B). A swip is **swizzled** when it holds an in-memory pointer and +**unswizzled** when it holds an on-disk page id, and §III-A says exactly how +the two are told apart: "We use pointer tagging (one bit of the 8-byte +reference) to distinguish between these two states." + +One bit, one branch. §III-A's summary sentence is the design in a line: "the +buffer management overhead of accessing a hot page merely consists of one +conditional statement that checks this bit." Or §I, less formally: "accessing +an in-memory page merely involves a simple, well-predicted if statement +rather than a costly hash table lookup." + +The consequence for the architecture is bigger than the branch. The +translation table has not been made faster; it has been **deleted and +scattered into the data structure itself** (§IV-A: "a traditional page +translation table is not needed because its state is embedded in the +buffer-managed data structures"). Note the detail in §III-A that keeps +recovery possible: swizzled pages still have page identifiers, they are just +stored in the frame rather than in the reference. + +Why it matters: a hot access now costs what an in-memory system charges. The +cost has moved entirely onto the miss, where a disk read dwarfs it anyway. + +### Step 3 — the price of swizzling: one owning swip, bottom-up eviction + +> **In:** the swip from Step 2. +> **Out:** two structural constraints — a page has exactly one incoming +> reference, and parents are evicted after children — that decide whether a +> given data structure can live on this design at all. Step 9 asks that +> question about matrix tiles. + +If two swips pointed at the same page, one could be swizzled and the other +unswizzled at the same time, and there is no central table to reconcile them +with. §IV-B works the example: a page `Px` referenced by `Py` and `Pz` "can +be referenced by one swizzled and one unswizzled swip at the same time. +Maintaining consistency, in particular without using global latches, is very +hard and inefficient." So LeanStore imposes the rule: **each page has a +single owning swip**, and the buffer pool "in its entirety a forest of +pages." + +The same reasoning forces **bottom-up eviction**: "we never unswizzle (and +therefore never evict) a page that has swizzled children" (§IV-B). The reason +is not policy but correctness — an evicted parent's swip slot is written to +disk, and if it held a memory pointer, "pages containing memory pointers +might be written out to disk, which would be a major problem because a +pointer is only valid during the current program execution." + +The mechanism is §IV-E's **iteration callback**: every buffer-managed data +structure registers a function that iterates the swips on one of its pages +(a no-op for leaves), and each page carries a marker saying which callback it +belongs to. When a randomly picked inner page turns out to have a swizzled +child, the buffer manager does not give up — Fig. 5: it "will try to unswizzle +one of the encountered swizzled child pages (randomly picking one of these)", +which "implicitly prioritizes inner pages over leaf pages during +replacement", i.e. inner nodes tend to stay resident. Finding the parent to +un-swizzle *through* uses parent pointers stored in the frame, which are +cheap to maintain precisely because children are always unswizzled first and +the pointers are never persisted. + +Two honest caveats the guide's usual summary drops. First, §IV-B says the +single-swip rule "is not a fundamental limitation of our approach" and +sketches two escapes: several parent pointers per frame (enough for +B+tree inter-leaf links), or "fat" swips carrying both a page identifier and +a pointer. Second, the rule is not original to LeanStore — §IV-B credits +Graefe et al.'s swizzling-based buffer manager with the same decision. + +Why it matters: this is the constraint that decides admissibility. A B-tree +is a forest of single-parent pages and fits. A graph, or a matrix tile +referenced by both a row index and a column index, does not — which is +question 4 below, and the reason Step 8's vmcache exists. ### Step 4 — the cooling stage: replacement with zero per-access work -Every classic policy (LRU lists, CLOCK usage bits) does a little -bookkeeping on *each access* to know what's cold later. LeanStore refuses: -instead, a background thread picks buffer frames **at random**, un-swizzles -them into a **cooling FIFO** (a queue holding ~10% of the pool, §III.D's -sizing heuristic). A cooling page is still in RAM; if anyone touches it, the -access path notices the "cool" tag and re-swizzles it cheaply — a **second -chance** that rescues hot pages that were unluckily sampled. Reach the end -of the FIFO untouched and you're written back (if dirty) and evicted. - -``` - random sample ──► cooling FIFO (~10% of pool) ──► evict at the end - │ - └── touched while cool? re-swizzle: second chance +> **In:** the swizzled/unswizzled distinction from Step 2 and the +> bottom-up rule from Step 3. +> **Out:** a replacement policy that writes nothing on a hit — and the +> measured hit-rate bill for that, which Step 7 spends. + +Every classic policy does bookkeeping on *each access* so it can know what is +cold later. §III-B refuses, and states the change of perspective precisely: +"Instead of tracking frequently accessed pages in order to avoid evicting +them, our replacement strategy identifies infrequently-accessed pages." + +The mechanism is speculative un-swizzling. Pick a **random** page in the pool +— no metadata is consulted, because none is maintained — and unswizzle its +reference *without* evicting the page. It is now **cooling**: unswizzled but +still in RAM. Cooling pages sit in a FIFO queue, most recently unswizzled at +the front, and are evicted (after a write-back if dirty) when they reach the +end. Touch one before then and it is pulled out of the queue and re-swizzled, +with no I/O at all: §III-B calls this the **second chance**, "a grace period +before it is evicted", which is what makes a policy this crude survivable. + +Fig. 3's state machine, which is worth being able to draw: + +```mermaid +stateDiagram-v2 + Cold --> Hot: page fault, load, swizzle + Hot --> Cooling: speculatively unswizzle\n(random pick, no metadata) + Cooling --> Hot: accessed — second chance,\nre-swizzle, no I/O + Cooling --> Cold: reaches FIFO end —\nwrite back if dirty, evict ``` -Why it matters: randomness replaces bookkeeping. Fig. 6 shows random + -second-chance FIFO tracks true LRU's hit rate closely on Zipf-skewed -workloads — while charging the hot path *nothing*. +`Hot` = in RAM, swip swizzled. `Cooling` = in RAM, swip *unswizzled*. +`Cold` = on SSD, swip unswizzled. Note that `Cooling` and `Cold` are +indistinguishable from the swip alone — that is what the next bullet is +about. + +Four implementation details the summaries usually lose, all from §IV-C: + +- **The cooling stage is a FIFO *plus a hash table*.** A cooling page's swip + is unswizzled — it holds a page id, not a tag pointing at the frame — so + an accessor cannot tell "cooling" from "on disk" by looking at the swip. + It looks the page id up in the cooling stage's hash table, which maps page + ids to queue entries; a hit there means the page is still in RAM, and it is + removed from both the hash table and the queue before being swizzled. +- **Nothing cools until memory runs short.** "The cooling stage is only used + when the free pages in the buffer pool are running out." +- **The unswizzling is done by worker threads, synchronously, not by a + background thread.** §IV-C considers both and chooses: "We use the second + option in order to avoid the risk of background threads being too slow." + Whenever a thread requests an empty page or swizzles one, it checks whether + the cooling percentage is below the threshold and unswizzles a page if + needed. (The modern LeanStore *code* does use dedicated page-provider + threads — the vmcache paper §5.1 configures it with 8 of them — so if you + read the repo expecting the paper's design, this is where they diverge.) +- **One global latch protects the cooling stage**, and the paper defends it: + the latch is only taken on the cold path, where I/O costs "orders of + magnitude more than a latch acquisition" anyway. + +The target is ~10% of the pool in the cooling state (§III-B), and §VI-B +justifies the number rather than asserting it: throughput was measured with +the cooling stage swept from 1% to 50% across Zipf factors (Fig. 11), +"performance is very stable … in particular for reasonable settings between +5% and 20%". Only around skew 1.6 — where the working set is close to the +buffer pool size — does the setting cost more than 10%. + +Why it matters: randomness replaces bookkeeping. The hot path writes nothing +at all, which is the only way to satisfy §III-C's rule that "programs that +frequently write to memory locations accessed by multiple threads do not +scale." + +### Step 5 — optimistic latches: readers that hold nothing + +> **In:** the pinning cost from Step 1. +> **Out:** readers that write no shared state — and the safety hole that +> creates, which Step 6 closes. + +Pinning exists so eviction cannot yank a page mid-read, but a pin is a write +to a shared cache line on every access, which is the ping-pong of Step 1. +§IV-F replaces it with an **optimistic latch**: the latch is an update +counter incremented after every modification, and "readers can proceed +without acquiring any latches, but validate their reads using the version +counters instead". Read the version, do the work, re-read the version — equal +means the read was consistent, changed means retry. + +The protocol built on top is **Optimistic Lock Coupling** (§IV-F, citing +Leis et al.), which "ensures consistent reads in tree data structures without +physically acquiring any latches during traversal". Writers usually latch +only the page they modify; only structure-modification operations such as +splits latch several. §III-C states the resulting shape: "lookups on swizzled +pages do not acquire any latches at all". + +Why it matters: this is the ingredient that makes swizzling *safe* rather +than merely fast. A reader holding no pin cannot block eviction — but it also +cannot stop it, which is the problem Step 6 exists to solve. + +### Step 6 — epoch-based reclamation: how a page is safely reused + +> **In:** Step 5's readers, which hold nothing, and Step 4's cooling queue. +> **Out:** the rule that decides *when* a cooling page's memory may actually +> be handed to someone else — the piece most retellings of this paper drop. + +If readers neither latch nor pin, what stops the buffer manager from reusing +a frame while a thread is still reading it? §IV-G's answer is **epoch-based +reclamation**, borrowed from latch-free data structures: one global epoch +counter that grows periodically, plus a local epoch per thread. + +The protocol (Fig. 6): before touching any buffer-managed structure, a thread +copies the global epoch into its local one — it has "entered" that epoch — +and on finishing sets its local epoch to ∞, meaning it holds nothing. When a +page is unswizzled into the cooling stage it is tagged with the global epoch +at that moment. Right before it is actually evicted, the buffer manager +checks the *minimum* local epoch across all threads: only when every thread +has moved past the page's epoch can no thread still hold a pointer into it, +and only then may the memory be reused. Note the economy — the paper points +it out — that only cooling pages carry an epoch, never hot ones, so the hot +path again writes nothing. + +Why it matters: "readers hold nothing" is not free; the cost was moved from a +per-access atomic to a per-eviction epoch comparison, which is paid on the +cold path where Step 4 already spends a latch. + +### Step 7 — what the three ingredients actually bought + +> **In:** the three ingredients (Steps 2, 4, 5–6). +> **Out:** the measured in-memory parity, the measured hit-rate loss, and +> the measured out-of-memory behaviour — the evidence Step 8's redesign had +> to preserve. + +**In-memory (§V-B).** Single-threaded TPC-C, 100 warehouses (10 GB), buffer +pool large enough for all of it: LeanStore 67K txns/s against an in-memory +B-tree at 69K — `67/69 = 97%` of a system with no buffer manager at all — +while BerkeleyDB manages 10K and WiredTiger 16K (Fig. 1). The comparison is +clean by construction: §V-A says the in-memory and buffer-managed B-trees +"have the same page layout and synchronization protocol", so the 3% is +buffer management and nothing else. Scaling on the +10-core machine (Fig. 8): BerkeleyDB peaks at 20K with 5 threads (2.4×), +WiredTiger reaches 8.8× at 20 threads, LeanStore 8.8× at 10 threads and 12.6× +with HyperThreading. Fig. 7's ablation, quoted in Step 1, says which +ingredient bought what: single-threaded, swizzling and lean eviction are the +big two (30K → 62K, roughly 2×) and optimistic latches add little because +one thread contends with nobody; at ten threads all three are required, and +missing any one collapses the result. + +**The hit-rate bill (§VI-B).** This is the number to know, because it is the +honest cost of replacing bookkeeping with randomness. The authors traced all +page accesses and simulated other policies — 5 GB data set, 1 GB buffer pool, +Zipf factor 1.0: -### Step 5 — optimistic latches: readers hold nothing +``` + Random FIFO LeanEvict(5%/10%/20%/50%) LRU 2Q OPT + 92.5% 92.5% 92.7 92.8 92.9 93.0 93.1% 93.8% 96.3% -Pinning exists so eviction can't yank a page mid-read. LeanStore replaces it -with **optimistic latches**: each frame carries a version counter; a reader -notes the version, reads *without writing any shared state*, then -re-checks the version — unchanged means the read was consistent, changed -means retry. Writers bump the version. A reader that holds nothing can't -block eviction and costs zero coherence traffic on the hot path (compare a -pin: an atomic increment that bounces the cache line between every reading -core, topic 0's false-sharing lesson). This is topic 9's main subject making -an early appearance. + LeanEvict at its recommended 10% against LRU: + hit rate: 93.1 − 92.8 = 0.3 percentage points + miss rate: 7.2% vs 6.9% = 4.3% more misses, relative + against the theoretical optimum: 96.3 − 92.8 = 3.5 points +``` -### Step 6 — vmcache: keep the goal, drop the swizzling +Nobody is far from anybody except OPT, which is unimplementable. §VI-B draws +the conclusion the arithmetic supports: "the page hit rates do not directly +translate into performance, as more complex strategies like LRU and 2Q would +also result in a higher runtime overhead" — you pay 4.3% more misses to make +every hit free. + +**Out-of-memory (§VI).** With a 20 GB pool and TPC-C growing from 10 GB to +50 GB (Fig. 9), LeanStore "stays close to the in-memory performance although +around 500 MB/sec are written out to the SSD in the background", while the +in-memory B-tree left to Linux swapping "drops severely and is highly +unstable". Fig. 10's lookup benchmark (5 GB data, 1 GB pool, 20 threads) +shows the shape of graceful degradation across skew: 92K lookups/s at 76K +I/Os per second under a uniform distribution, up to 143M lookups/s with zero +I/Os at the highest skew — a 1,554× range set entirely by how much of the +working set the replacement strategy manages to keep. + +Why it matters: "as fast as an in-memory system" is a claim about the hot +path, and these are the three measurements that pin it — parity in memory, a +0.3-point hit-rate loss, and no cliff when the data stops fitting. + +### Step 8 — vmcache: keep the goal, drop the swizzling + +> **In:** everything above — and specifically Step 3's one-swip constraint, +> which is the thing being paid to remove. +> **Out:** the same "pay only on the miss" property with translation done by +> the MMU, and a page-state array that replaces both the swip and the latch. Swizzling works but *infects the whole codebase*: every data structure must -know swips, honor one-parent, cooperate with cooling. vmcache (SIGMOD '23) -keeps "pay only on the miss" and moves translation into the hardware: - -- mmap an **anonymous** virtual range (address space backed by no file — - the CIDR '22 trap doesn't apply because the kernel never sees your file): - `page(pid)` is just `virt + pid * 4096` — the MMU (the address-translation - hardware) is the translation layer, for free. -- BUT the DB — not the kernel — decides residency: an explicit per-page - state word (Evicted/Marked/Locked/Unlocked + version counter), explicit - `pread` into the fixed virtual address on fault, `madvise(DONTNEED)` on - evict. -- The page-state word doubles as the hybrid latch (Step 5's optimistic - version counter — same bits, new home). -- Any page can have any number of references — the one-parent rule dies; - arbitrary graphs are fine (relevant to a graph-store capstone!). - -The whole design fits in one state machine: - -```rust -// Translation is the MMU's job; RESIDENCY is the DB's. -fn page(&self, pid: u64) -> *mut u8 { unsafe { self.virt.add(pid as usize * 4096) } } - -fn fix(&self, pid: u64) { - loop { - let s = self.state[pid].load(); // Evicted/Marked/Locked/Unlocked + version - match s.kind() { - Evicted => if self.state[pid].cas(s, s.locked()) { - pread(self.fd, self.page(pid), 4096, pid * 4096); // into the FIXED addr - self.state[pid].store(s.unlocked_bumped()); // word doubles as - return; // the hybrid latch - }, - Marked | Unlocked => if self.state[pid].cas(s, s.locked()) { return; }, - Locked => core::hint::spin_loop(), // someone else is faulting it in - } - } -} -// evict: write back if dirty, madvise(DONTNEED, page(pid)), state → Evicted +know about swips, honour one-parent, and cooperate with cooling. vmcache +(SIGMOD '23) keeps the goal and moves translation into the hardware. + +- **The mapping is virtual memory.** vmcache reserves an **anonymous** + mapping — `mmap` with `MAP_ANONYMOUS`, backed by no file, so CIDR '22's + trap ([`reading-mmap-paper.md`](reading-mmap-paper.md)) does not apply + because the kernel never learns which file the bytes belong to — at least + as large as the storage. Page *pid* always lives at `virtMem + pid`, and + the **MMU**, the address-translation hardware, does the lookup for free. +- **Residency stays with the DBMS.** A page is brought in with an explicit + `pread` *into that fixed address* (§3.1), and thrown out with + `madvise(…, MADV_DONTNEED)`, having been written back with `pwrite`/libaio + first if dirty. Nothing happens that the buffer manager did not ask for. +- **A page-state array replaces the swip.** One 64-bit word per page: 8 bits + of lock state and 56 bits of version counter (§3.6), so the word *is* + Step 5's optimistic latch. The states are Unlocked, a shared-reader count, + Locked, **Marked** and Evicted. +- **Any page may have any number of references.** Step 3's constraint is + gone, so graphs are fine — which is Table 1's `graphs: yes` column, and the + reason this design matters for a graph-store capstone. + +The state machine is small enough to read in full. This is the authors' +reference implementation, not pseudocode: + +```cpp +// viktorleis/vmcache@1157828 — vmcache.cpp, PageState's constants, 74-81 + 74 struct PageState { + 75 atomic stateAndVersion; + 76 + 77 static const u64 Unlocked = 0; + 78 static const u64 MaxShared = 252; + 79 static const u64 Locked = 253; + 80 static const u64 Marked = 254; + 81 static const u64 Evicted = 255; ``` -### Step 7 — the map of the design space +Line 75 is the whole idea: one atomic word holds the state *and* the version, +so a reader validates residency and consistency in a single load. Values +0–252 double as the shared-reader count, which is why the special states +start at 253. + +```cpp +// viktorleis/vmcache@1157828 — vmcache.cpp, BufferManager::fixX, 682-702 + 682 Page* BufferManager::fixX(PID pid) { + 683 PageState& ps = getPageState(pid); + 684 for (u64 repeatCounter=0; ; repeatCounter++) { + 685 u64 stateAndVersion = ps.stateAndVersion.load(); + 686 switch (PageState::getState(stateAndVersion)) { + 687 case PageState::Evicted: { + 688 if (ps.tryLockX(stateAndVersion)) { + 689 handleFault(pid); + 690 return virtMem + pid; + 691 } + 692 break; + 693 } + 694 case PageState::Marked: case PageState::Unlocked: { + 695 if (ps.tryLockX(stateAndVersion)) + 696 return virtMem + pid; + 697 break; + 698 } + 699 } + 700 yield(repeatCounter); + 701 } + 702 } +``` -Put the three readings of this topic side by side: +The line to look at is **690**, and then 696: both return `virtMem + pid`. +The address does not depend on whether the page was resident — there is +nothing to translate, ever. The only difference between a hit and a miss is +whether `handleFault` (675-680: `ensureFreePages`, `readPage`, insert into +the resident set) ran first. That is "pay only on the miss", expressed as +control flow. + +Replacement is **CLOCK**, not a cooling FIFO: §3.4 marks Unlocked pages +`Marked` ahead of need, any access clears the mark, and pages still marked +when the evictor comes round are taken. Note where the mark lives — in the +same word as the latch, so clearing it is something an access does anyway. +Eviction runs in batches of 64 through the five numbered steps of §3.4, which +are literally the comments in the code (`0. find candidates` at 760, +`1. write dirty pages` at 783, `4. remove from page table` at 804 where the +`madvise(MADV_DONTNEED)` sits at 814). + +The bill for all this is a page-state array plus page tables: §3.6 computes +8.016 bytes of page table per 4 KB of storage, plus 8 bytes of page state — +about **16 bytes of DRAM per 4 KB on disk**, so 4 GB for a 1 TB SSD, or +`1/256` of capacity. + +And the honest weakness, §5.3: basic vmcache is bound by the kernel's +page-table manipulation once misses are frequent. Out-of-memory random +lookups run about **60% faster with the exmap kernel module** than without, +and without exmap vmcache is "substantially slower than LeanStore" on that +workload, though still ahead of WiredTiger and the mmap-based LMDB. For +TPC-C the gap is small "because even vmcache manages to become I/O bound". + +### Step 9 — the map of the design space + +> **In:** all three designs (classic pool, LeanStore, vmcache) and the mmap +> chapter's fourth. +> **Out:** the table you should be able to reconstruct from memory, and the +> question it answers for the capstone. ``` - LeanStore: translation in POINTERS (swips, invasive, tree-shaped data) - vmcache: translation in the MMU (virt addressing, any ref-graph) - both: replacement + residency decided by the DB, never the kernel + classic: translation in a HASH TABLE (a lookup on every hit) + LeanStore: translation in POINTERS (swips; invasive; tree-shaped data) + vmcache: translation in the MMU (virtual addressing; any ref graph) + mmap: translation in the MMU — but RESIDENCY in the kernel too + LeanStore + vmcache: replacement and residency decided by the DB, never the OS ``` -The CIDR '22 mmap paper (reading-mmap-paper.md) is the missing middle: mmap -with *kernel*-controlled residency is the trap; vmcache is mmap-style -addressing with DB-controlled residency. One cost remains: madvise-heavy -eviction pays syscalls and TLB shootdowns — the **exmap** kernel module in -the vmcache paper fixes that; without it, plain vmcache still beats classic -pools. +vmcache's own Table 1 lays the same comparison out over six designs, and the +two rows that matter for a graph store are `graphs` (mmap yes, traditional +yes, pointer swizzling **no**, vmcache yes) and `control` (mmap: OS; +everything else: DBMS). The CIDR '22 mmap paper +([`reading-mmap-paper.md`](reading-mmap-paper.md)) is the missing middle: +mmap with *kernel*-controlled residency is the trap, and vmcache is +mmap-style addressing with DB-controlled residency. -## How to read the papers (with the concepts in hand) +Why it matters: the capstone's data is matrix tiles addressed by row *and* +column. Step 3 says that is not a forest, so the swizzling column is closed +to it, and this table is where you find out which columns are still open. -Read LeanStore first, then vmcache as the retraction-and-fix. You've read -the code (reading-leanstore.md) — in the LeanStore paper focus on: - -- **§I–II** — Step 1's problem statement; skim, you know it. -- **§III.B — read carefully.** The one-swip-per-page ownership rule and why - eviction is bottom-up (Step 3). -- **§III.D — read carefully.** Cooling-stage sizing (the 10% heuristic) and - the hit/second-chance probabilities — Fig. 6 shows random+FIFO tracks LRU - closely on Zipf (Step 4). -- **§V** — evaluation: in-memory TPC-C at parity with a - no-buffer-manager build; the graceful degradation curve as data exceeds - RAM (the money plot). +## How to read the papers (with the concepts in hand) -In vmcache: the page-state machine (Step 6's code is its skeleton), the -argument for why dropping swizzling gives nothing back, and the exmap -numbers for eviction-heavy workloads. +Read LeanStore first, then vmcache as the retraction-and-fix. You have read +the code ([`reading-leanstore.md`](reading-leanstore.md)), so in the LeanStore +paper focus on: + +| Section | How to read it | Step | +|---|---|---| +| §I–II | The problem statement and Fig. 1's four bars. Skim; Step 1 has it. | 1 | +| §III-A | Pointer swizzling in one page. The sentence to keep: hot-page overhead is "one conditional statement". | 2 | +| §III-B | **Read carefully.** The change of perspective — identify *cold* pages instead of tracking hot ones — and the 10% cooling target. | 4 | +| §III-C | Why latches, not I/O, are the bottleneck: 15 latch acquisitions per Shore-MT row update, and cacheline ping-pong. | 1, 5 | +| §IV-B | **Read carefully.** The single-owning-swip rule and why eviction is bottom-up. Note the two escapes it offers (multiple parent pointers, "fat" swips). | 3 | +| §IV-C | The cooling stage as implemented: FIFO **plus a hash table**, unswizzling done synchronously by worker threads, one global latch defended. | 4 | +| §IV-E, Fig. 5 | The iteration callback, and why replacement implicitly favours inner pages. | 3 | +| §IV-F–G | Optimistic latches, then epoch-based reclamation — the second is what makes the first safe. | 5, 6 | +| §V-B, Fig. 7 | The ablation. Read the 10-thread row before the 1-thread row. | 7 | +| §VI-B | The hit-rate table (LeanEvict 92.8% vs LRU 93.1% vs OPT 96.3%) and Fig. 11's cooling-size sweep. | 4, 7 | + +In vmcache: §3.1–3.2 (the primitives and the page-state machine — Step 8's +code is its skeleton), §3.4 (clock over the state array, batched eviction), +§3.6 (why an extra cache miss for the state word is nearly free: both +addresses are known upfront, so the CPU issues them in parallel), Table 1 and +Table 2, then §5.3 for what exmap is actually worth. ## Questions to answer in notes.md @@ -196,19 +539,183 @@ numbers for eviction-heavy workloads. indexes = a DAG, not a tree. Which of the two designs is even admissible, and what would the swizzling workaround cost? +## Takeaway + +Three ingredients, three deleted costs: swizzling deletes translation, +the cooling stage deletes replacement bookkeeping, optimistic latches (plus +epochs) delete pinning. The measured result is 97% of an in-memory B-tree's +single-threaded TPC-C for 0.3 hit-rate points against LRU. vmcache keeps the +property and pays for it in DRAM (16 bytes per 4 KB page) instead of in +invasiveness — which is the trade a graph-shaped capstone has to take. + ## Done when -You can state what each of the three LeanStore ingredients eliminates, and -explain in two sentences why vmcache can drop swizzling without giving back -the hot-path win. +Answer each before unfolding it. + +- [ ] You can state what each of the three LeanStore ingredients eliminates, and which of them the ablation says matters most — at one thread, and at ten. + +
Answer + + Pointer swizzling eliminates **translation**: the parent's swip holds the + frame pointer, so §III-A's "one conditional statement that checks this bit" + replaces a hash probe under a partition latch. The cooling stage eliminates + **replacement bookkeeping**: nothing is written on a hit, because cold + pages are found by random sampling plus a second chance rather than by + tracking hot ones (§III-B). Optimistic latches eliminate **pinning**: + readers validate a version counter instead of incrementing a shared + reference count (§IV-F). + + Fig. 7 measures which matters. At one thread: 30K baseline → 48K with + swizzling → 62K with lean eviction → 67K with optimistic latches, so + swizzling and eviction are ~2× together and the latches add ~8%, because a + single thread contends with nobody. At ten threads: 18K → 23K → 109K → + 597K, a 33× swing, and all three are required — the baseline at ten threads + (18K) is *slower* than the baseline at one (30K), which is what a + contended global hash table and a single LRU list do to scaling. + +
+ +- [ ] You can explain the one-owning-swip rule and derive bottom-up eviction from it, rather than remembering it. + +
Answer + + §IV-B's argument: if a page `Px` were referenced by both `Py` and `Pz`, one + swip could be swizzled and the other unswizzled at the same instant, and + since swizzling deleted the central translation table there is nothing to + reconcile them with — "maintaining consistency, in particular without using + global latches, is very hard and inefficient". So every page has exactly + one incoming reference and the buffer pool is a forest. + + Bottom-up eviction falls out of the same fact. A parent's swip slot is + *part of the page*, so evicting a parent writes its swips to disk. If a + child were still swizzled, the value written would be a raw memory pointer, + and "a pointer is only valid during the current program execution" — the + child would be unreachable after restart. Hence §IV-B: never unswizzle a + page with swizzled children. §IV-E implements it with a per-data-structure + iteration callback, and when an inner page is picked but has a swizzled + child, it unswizzles a randomly chosen child instead — which implicitly + keeps inner nodes resident (Fig. 5). + +
+ +- [ ] You can say what a cooling page's swip contains, and how an accessor discovers the page is still in RAM. + +
Answer + + It contains a **page id**, not a pointer: cooling means *unswizzled but + still resident* (§III-B, Fig. 3). So the swip alone cannot distinguish + "cooling in RAM" from "cold on SSD" — checking the tag bit says only "not + swizzled". + + The discovery happens in the cooling stage, which §IV-C says is a FIFO + queue *and* a hash table mapping page identifiers to queue entries. An + accessor that finds an unswizzled swip looks the page id up there; a hit + means the page is in memory, and it is removed from the hash table and from + the queue, then swizzled — no I/O. That is the second chance. A miss means + a real page fault. This is the detail that separates the paper from the + later code, where the frame reference carries a "cool" tag bit and the + lookup goes away. + +
+ +- [ ] You can price the cooling stage's randomness against LRU with the paper's own hit rates, and say why the trade is worth taking. + +
Answer + + §VI-B's traced simulation, 5 GB data set, 1 GB buffer pool, Zipf 1.0: + random 92.5%, FIFO 92.5%, LeanEvict 92.8% at the recommended 10% cooling, + LRU 93.1%, 2Q 93.8%, OPT 96.3%. So LeanEvict gives up 0.3 percentage points + of hit rate to LRU — in miss terms, 7.2% against 6.9%, which is 4.3% more + misses. + + It is worth taking because the cost is paid on the miss path, where a page + fault costs microseconds, and the saving is paid on the hit path, where + vmcache's Table 2 measures a hash-table access at 336 ns against 236 ns and + 27.9 instructions against 10.4 — for an *unsynchronized* table, i.e. a + lower bound. §VI-B's own conclusion: hit rates "do not directly translate + into performance, as more complex strategies like LRU and 2Q would also + result in a higher runtime overhead". + +
+ +- [ ] You can explain in two sentences why vmcache can drop swizzling without giving back the hot-path win — and name what it pays instead. + +
Answer + + Because the translation that swizzling avoided is done by hardware anyway: + every page lives at the fixed address `virtMem + pid` in an anonymous + mapping, so the MMU resolves it during the load itself and both `fixX`'s + returns are the same expression (`vmcache.cpp:690` and `:696`) whether the + page was resident or not. The DBMS keeps the part mmap gets wrong — + residency — by faulting with an explicit `pread` and evicting with + `madvise(MADV_DONTNEED)` (`vmcache.cpp:814`). + + It pays in DRAM and in kernel time. §3.6: about 16 bytes of DRAM per 4 KB + of storage (8.016 bytes of page table plus an 8-byte page state), which is + 4 GB for a 1 TB SSD. And §5.3: page-table manipulation becomes the + bottleneck when misses are frequent — out-of-memory random lookups are + about 60% faster with the exmap kernel module, and without it vmcache is + substantially slower than LeanStore on that workload. What it buys for that + price is Step 3's constraint: any page may have any number of incoming + references. + +
+ +- [ ] You wrote answers to all four questions in notes.md, including the admissibility verdict for matrix tiles. + +
Answer + + There is no answer to unfold — the verdict is the exercise. The bar: decide + whether a tile addressed by both a row index and a column index has one + owning reference or two, and if two, say which of §IV-B's escapes you would + take ("fat" swips carrying both id and pointer, or several parent pointers + per frame) and what it costs in bytes per frame and in eviction complexity. + "Use vmcache instead" is a legitimate answer, but only with the DRAM + arithmetic from §3.6 attached. + +
## References **Papers** -- Leis, Haubenschild, Kemper, Neumann — "LeanStore: In-Memory Data - Management Beyond Main Memory" (ICDE 2018) — focus on §III.B (one swip - per page, bottom-up eviction), §III.D (cooling-stage sizing, Fig. 6), - §V (the graceful-degradation money plot) -- Leis, Alhomssi, Ziegler, Loeck, Dietrich — "Virtual-Memory Assisted - Buffer Management" (vmcache/exmap, SIGMOD 2023) — read after LeanStore, - as the retraction and the fix +- Leis, Haubenschild, Kemper, Neumann — *"LeanStore: In-Memory Data + Management Beyond Main Memory"* (ICDE 2018, 12 pages) — + [PDF](https://db.in.tum.de/~leis/papers/leanstore.pdf) +- Leis, Alhomssi, Ziegler, Loeck, Dietrich — *"Virtual-Memory Assisted Buffer + Management"* (vmcache/exmap, SIGMOD 2023, 14 pages) — + [PDF](https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/vmcache.pdf) — + read after LeanStore, as the retraction and the fix + +| Where | What this chapter took from it | +|---|---| +| LeanStore §I, Fig. 1 | 67K (LeanStore) vs 69K (in-memory B-tree) vs 16K (WiredTiger) vs 10K (BerkeleyDB) single-threaded TPC-C | +| LeanStore §III-A | swips, pointer tagging with **one** bit, "one conditional statement" per hot access | +| LeanStore §III-B | identify cold pages instead of tracking hot ones; random candidates; the second chance; ~10% cooling | +| LeanStore §III-C | 15 latch acquisitions per Shore-MT single-row update; cacheline ping-pong | +| LeanStore §IV-B | the single owning swip; never unswizzle a page with swizzled children; the two escapes from the rule | +| LeanStore §IV-C | the cooling FIFO *and* its hash table; unswizzling done synchronously by worker threads; the global cooling latch | +| LeanStore §IV-E, Fig. 5 | the iteration callback; inner pages implicitly favoured | +| LeanStore §IV-F–G, Fig. 6 | optimistic latches; epoch-based reclamation, and that only cooling pages carry an epoch | +| LeanStore §V-A | the machine: Xeon E5-2687W v3, 10 cores, 64 GB, Linux 4.8, 16 KB pages, ~4K lines of C++ | +| LeanStore §V-B, Fig. 7 | the ablation: 30/48/62/67K at 1 thread, 18/23/109/597K at 10 | +| LeanStore §V-B, Fig. 8 | scaling: BerkeleyDB 2.4×, WiredTiger 8.8×, LeanStore 8.8× at 10 threads / 12.6× with SMT | +| LeanStore §VI-A, Fig. 9 | 20 GB pool, data growing 10 → 50 GB, ~500 MB/s written back, swapping unstable | +| LeanStore §VI-B, Figs. 10–11 | 92K → 143M lookups/s across skew; the hit-rate table; the 1–50% cooling sweep | +| vmcache §3.1–3.2 | anonymous mapping, `pread` into the fixed address, `MADV_DONTNEED`, the four page states | +| vmcache §3.4 | clock via the Marked state; batched eviction in five steps | +| vmcache §3.6 | 8.016 B page table + 8 B page state per 4 KB ≈ 16 B/page; parallel access to state and data | +| vmcache Table 1 | the six-design comparison; `graphs: no` for pointer swizzling | +| vmcache Table 2 | 219 / 236 / 336 ns and 3.3 / 10.4 / 27.9 instructions per random page access | +| vmcache §5.1, §5.3 | the setup (EPYC 7713, 128 GB pool, 1 TB workloads); exmap worth ~60% on out-of-memory random lookups | + +**Code** +- [viktorleis/vmcache](https://github.com/viktorleis/vmcache) at `1157828` — + the paper's reference implementation, a single 1855-line `vmcache.cpp`. + +| File | Lines | What | +|---|---|---| +| `vmcache.cpp` | 74-81 | `PageState`: one atomic word, states Unlocked/shared/Locked/Marked/Evicted | +| `vmcache.cpp` | 675-680 | `handleFault` — ensure free pages, read, insert into the resident set | +| `vmcache.cpp` | 682-702 | `fixX` — the state machine of Step 8; both returns are `virtMem + pid` | +| `vmcache.cpp` | 755-825 | batched eviction, its five numbered steps as code comments | +| `vmcache.cpp` | 814 | `madvise(MADV_DONTNEED)` — the eviction primitive itself | diff --git a/topics/06-buffer-pool/reading-leanstore.md b/topics/06-buffer-pool/reading-leanstore.md index 627eea3..8a23dbb 100644 --- a/topics/06-buffer-pool/reading-leanstore.md +++ b/topics/06-buffer-pool/reading-leanstore.md @@ -1,156 +1,424 @@ # LeanStore in code: swips, cooling, hybrid latches -The paper claims a hot page access can cost zero atomics; this chapter walks -the classic ICDE '18 codebase to see how — a u64 that is either a pointer or -a page id, a background thread that cools random frames, and latches whose -readers hold nothing. Read the paper guide -([reading-leanstore-paper.md](reading-leanstore-paper.md)) first for the why; -this chapter rebuilds the mechanism step by step, then hands you the file -and line anchors to watch each piece work. +The paper claims a hot page access can cost zero *writes* to shared memory; +this chapter walks the classic ICDE '18 codebase to see how — a `union` that +is either a pointer or a page id, a background thread that samples random +frames, and latches whose readers hold nothing and abort by `longjmp`. Read +the paper guide ([reading-leanstore-paper.md](reading-leanstore-paper.md)) +first for the why; this chapter rebuilds the mechanism step by step, then +hands you the file and line anchors to watch each piece work. + +Read at [`leanstore/leanstore@90fcf18`](https://github.com/leanstore/leanstore), +the repo's pinned commit (the pin table is at the end of +`resources/codebases.md`; local clone at `~/repos/leanstore`). Everything is +under `backend/leanstore/`. Three places the **code differs from the paper** +are flagged as you reach them — they are the most useful part of reading both. ## The problem in one sentence -Postgres charges every page access a hash probe + partition lock + CAS pin -(~2 atomics and a likely cache miss) even when the page is in RAM; -LeanStore's code must deliver the same page, crash-safe and evictable, for -the cost of a single pointer dereference. +Postgres charges every page access a hash probe plus a partition LWLock plus +a CAS pin — two atomics and a likely cache miss even when the page is already +in RAM ([`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md), Step 3) — +and LeanStore's code has to deliver the same page, crash-safe and evictable, +for the cost of a dereference and a version compare. ## The concepts, step by step ### Step 1 — the swip: one u64 that is either a pointer or a page id -A **swip** is a reference slot inside a parent node that holds *either* a -raw in-memory pointer to a buffer frame (a **BufferFrame** — the RAM slot -holding a page plus its header) *or* an on-disk page id, distinguished by -the top two bits of the word: +> **In:** nothing yet — this is the representation everything else reads. +> **Out:** three states (HOT / COOL / EVICTED) encoded in two bits, which +> Step 2 branches on and Step 3 transitions between. + +A **swip** is a reference slot inside a parent node. It holds *either* a raw +in-memory pointer to a **BufferFrame** — the RAM slot holding a page, plus +its header — *or* the page's on-disk **page id**. There is no separate field +saying which; it is a `union`, and two tag bits in the high end of the word +decide how to read it: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/Swip.hpp, the tag bits, 20-34 + 20 // 1xxxxxxxxxxxx evicted, 01xxxxxxxxxxx cooling, 00xxxxxxxxxxx hot + 21 static const u64 evicted_bit = u64(1) << 63; + 22 static const u64 evicted_mask = ~(u64(1) << 63); + 23 static const u64 cool_bit = u64(1) << 62; + 24 static const u64 cool_mask = ~(u64(1) << 62); + 25 static const u64 hot_mask = ~(u64(3) << 62); + 26 static_assert(evicted_bit == 0x8000000000000000, ""); + 27 static_assert(evicted_mask == 0x7FFFFFFFFFFFFFFF, ""); + 28 static_assert(hot_mask == 0x3FFFFFFFFFFFFFFF, ""); + 29 + 30 public: + 31 union { + 32 u64 pid; + 33 BufferFrame* bf; + 34 }; +``` + +Line 31's `union` is the whole trick: the same 8 bytes are a `u64 pid` or a +`BufferFrame*`, and lines 45–47 read the tag to say which. ``` bit 63 (evicted) bit 62 (cool) - 0 0 HOT — raw BufferFrame*: dereference it - 0 1 COOL — frame exists, sits in cooling FIFO - 1 - EVICTED — low bits hold the page id, on disk + 0 0 HOT — the bytes ARE a BufferFrame* + 0 1 COOL — frame exists; mask the bit off (:51) + 1 - EVICTED — low 63 bits are the page id (:49) ``` +The transitions are three one-line mutators: `warm()` clears the cool bit +(:59–63), `cool()` sets it (:65), `evict(pid)` overwrites the word with a +page id plus bit 63 (:67). + +> **Code vs paper #1.** §III-A of the paper uses **one** tag bit and keeps +> cooling pages in a separate hash table; the code uses **two** and encodes +> `COOL` in the swip itself. That single change is why Step 2's COOL arm is a +> bit-clear rather than a hash lookup — the code is strictly cheaper than the +> design it published. + The buffer pool's mapping table is thereby *distributed into the parent -nodes*: no hash lookup, no partition lock, on any hot access. The price: -exactly one swip may reference a page — if two parents held raw pointers, -un-swizzling on eviction couldn't find them both (no central table to -consult). +nodes*: no hash lookup and no partition lock on any hot access. The price is +the paper's §IV-B rule — exactly one swip may reference a page, because if +two parents held raw pointers, un-swizzling on eviction could not find them +both. There is no central table left to consult. ### Step 2 — resolveSwip: the three-arm hot path -`resolveSwip` is the single function every page access goes through, and -its three arms are the swip's three states — only the third ever touches -the disk: +> **In:** a swip in one of Step 1's three states, plus an optimistic `Guard` +> on the parent (Step 4). +> **Out:** a usable `BufferFrame&` — and, on the third arm, a page read that +> Step 3's provider thread will eventually undo. + +`resolveSwip` is the single function every page access goes through. Its +three arms are the swip's three states, and only the third touches the disk: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/BufferManager.cpp, resolveSwip's first two arms, 280-299 + 280 // Returns a non-latched BufferFrame, called by worker threads + 281 BufferFrame& BufferManager::resolveSwip(Guard& swip_guard, Swip& swip_value) + 282 { + 283 if (swip_value.isHOT()) { + 284 BufferFrame& bf = swip_value.asBufferFrame(); + 285 swip_guard.recheck(); + 286 return bf; + 287 } else if (swip_value.isCOOL()) { + 288 BufferFrame* bf = &swip_value.asBufferFrameMasked(); + 289 swip_guard.recheck(); + 290 BMOptimisticGuard bf_guard(bf->header.latch); + 291 BMExclusiveUpgradeIfNeeded swip_x_guard(swip_guard); // parent + 292 BMExclusiveGuard bf_x_guard(bf_guard); // child + 293 bf->header.state = BufferFrame::STATE::HOT; + 294 swip_value.warm(); + 295 return *bf; + 296 } + 297 // ------------------------------------------------------------------------------------- + 298 swip_guard.unlock(); // Otherwise we would get a deadlock, P->G, G->P + 299 const PID pid = swip_value.asPageID(); +``` + +Read the three arms as three price tags: + +- **HOT (283–286)** — a load, a branch, and `recheck()` (Step 4: re-read the + parent's version and compare). No atomic read-modify-write, no shared + write, no lock. This is the case that runs on essentially every access, and + it is the entire point of the design. +- **COOL (287–295)** — the second chance. The page is in RAM but unswizzled, + so the accessor rescues it: three guards (child optimistic, parent + upgraded to exclusive, child upgraded to exclusive), then flip the frame + state and clear the bit. Note **two** structures change — the frame's + `header.state` at 293 *and* the parent's swip at 294 — which is why both + latches are needed and why this arm, unlike HOT, does write shared memory. +- **EVICTED (298 onward)** — the page fault. Line 298's comment is the + interesting part: the parent guard is *released first* because the lock + order here (page → global partition) is the reverse of Step 3's (global → + page), and the resolution is to not hold both. + +The miss path is more careful than "read the page". It takes the partition +mutex (:301), looks the pid up in `partition.io_ht` — an **I/O hash table** +of reads currently in flight (`IOFrame`, Partition.hpp:18–33) — and only if +there is no entry does it pop a free frame (:307), publish an `IOFrame` in +state `READING` (:311), drop the global lock, and call `readPageSync` +(:317). A second thread that wants the same page finds the `READING` entry, +blocks on that `IOFrame`'s mutex (:372–376), and retries — one physical read +serves both. On success the swip is swizzled and the frame marked HOT, in +that order, with the comment shouting why: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/BufferManager.cpp, publishing the loaded page, 341-351 + 341 swip_guard.recheck(); + 342 JMUW> g_guard(partition.ht_mutex); + 343 BMExclusiveUpgradeIfNeeded swip_x_guard(swip_guard); + 344 io_frame.mutex.unlock(); + 345 swip_value.warm(&bf); + 346 bf.header.state = BufferFrame::STATE::HOT; // ATTENTION: SET TO HOT AFTER + 347 // IT IS SWIZZLED IN + 348 // ------------------------------------------------------------------------------------- + 349 if (io_frame.readers_counter.fetch_add(-1) == 1) { + 350 partition.io_ht.remove(pid); + 351 } +``` +If anything in that block fails validation, `jumpmuCatch` (:356) parks the +frame in the `IOFrame` as `READY` and jumps — the read is not wasted, just +handed to whoever retries. + +> **Code vs paper #2.** The frame carries an explicit `STATE` enum — +> `FREE / HOT / COOL / LOADED` (BufferFrame.hpp:19) — *in addition to* the +> swip's tag bits. The paper describes one state machine; the code keeps two +> and asserts they agree (e.g. PageProviderThread.cpp:192–193). Order matters +> at 345–346 for exactly this reason. + +### Step 3 — the page provider: replacement by random sampling, twice + +> **In:** HOT frames from Step 2, and a free list running low. +> **Out:** COOL frames (phase 1) and free frames (phase 2) — the supply Step +> 2's miss path draws on. + +Classic policies do bookkeeping on every access: postgres bumps a usage +counter inside the pin CAS, DuckDB enqueues on every unpin. LeanStore does +*nothing* per access. A background `pageProviderThread` +(PageProviderThread.cpp:28, owning a range of partitions) samples frames at +random instead: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/PageProviderThread.cpp, candidate selection, 40-49 + 40 auto next_bf_range = [&]() { + 41 const u64 BATCH_SIZE = FLAGS_replacement_chunk_size; + 42 cool_candidate_bfs.clear(); + 43 for (u64 i = 0; i < BATCH_SIZE; i++) { + 44 BufferFrame* r_bf = &randomBufferFrame(); + 45 DO_NOT_OPTIMIZE(r_bf->header.state); + 46 cool_candidate_bfs.push_back(r_bf); + 47 } + 48 return; + 49 }; ``` - isHOT (:283) → return the pointed-to frame. Done. ~0 overhead. - isCOOL (:287) → frame exists but sits in the cooling FIFO: - latch parent, clear cool bit (second chance), return. - EVICTED → page fault: grab free frame, readPageSync (:317), - swizzle the swip, return. + +`randomBufferFrame()` at line 44 is the entire replacement policy's input. +There is no LRU list, no usage counter, no access timestamp — none exists to +consult. Line 45's `DO_NOT_OPTIMIZE` prefetches the header the loop is about +to need. The batch is `FLAGS_replacement_chunk_size`, **default 64** +(Config.cpp:75). + +The loop only runs when frames are short (:64): +`dram_free_list.counter < free_bfs_limit`, where the limit is +`FLAGS_free_pct` — **default 1** — percent of the pool, divided across +partitions (BufferManager.cpp:55). Then, per candidate: + +1. Take an optimistic guard; skip anything pinned in memory, being written + back, or exclusively latched (:74). +2. **If it is already COOL, it goes on the evict list (:77–79).** This is + phase 2's entire input. +3. If it is HOT, check its children — `iterateChildrenSwips` (:90) with + `all_children_evicted &= swip.isEVICTED()` (:91). If a child is still HOT, + push *the child* onto the candidate list and repick (:92–97): eviction is + bottom-up, and inner pages therefore drift towards staying resident, which + is Fig. 5 of the paper implemented. +4. Find the parent (`findParent`, :114) so the swip can be reached at all, + then upgrade both parent and child to exclusive and cool the page: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/PageProviderThread.cpp, the cooling itself, 143-154 + 143 BMExclusiveUpgradeIfNeeded p_x_guard(parent_handler.parent_guard); + 144 BMExclusiveGuard r_x_guard(r_guard); + 147 paranoid(r_buffer->header.state == BufferFrame::STATE::HOT); + 152 r_buffer->header.state = BufferFrame::STATE::COOL; + 153 parent_handler.swip.cool(); // Cool the pointing swip before unlocking the current bf + 154 } ``` -The same three arms, as code: - -```rust -// The hot path is a pointer dereference — nothing else. -fn resolve(&self, parent: &HybridGuard, swip: &mut Swip) -> &BufferFrame { - if swip.is_hot() { return swip.frame(); } // raw pointer: ~0 overhead - if swip.is_cool() { - parent.upgrade_exclusive(); // touched while cooling ⇒ - swip.warm(); // second chance: clear the - return swip.frame(); // bit, dodge the FIFO - } - let frame = self.free_frames.pop(); // EVICTED ⇒ page fault: - self.read_page_sync(swip.pid(), frame); // the ONLY case that pays - swip.swizzle(frame); // pid → pointer, in place — - frame // next access is hot -} +Phase 2 then evicts what phase 1 found already cool: `evict_bf` (:171) finds +the parent again, asserts the page is clean (`ensure(!bf.isDirty())`, :190 — +dirty pages are written by `AsyncWriteBuffer` in phase 3 first), +`swip.evict(pid)` (:196), and returns the frame to the free list. + +> **Code vs paper #3, the big one.** The paper's cooling stage is a **FIFO +> queue plus a hash table** (§IV-C), and the paper explicitly rejects +> background threads in favour of synchronous unswizzling by workers. The +> code has **neither**: `struct Partition` (Partition.hpp:65–104) contains an +> I/O hash table, a free list and page-id allocation — and no cooling queue +> at all — while a dedicated `pageProviderThread` (`FLAGS_pp_threads`, +> default 1, Config.cpp:7) does the work. A page is evicted only if it is +> **randomly sampled twice**: once while HOT (→ COOL) and again while still +> COOL. The FIFO's grace period has become a second sampling draw. + +**What that costs, as arithmetic.** Take a 16 GiB pool of 16 KiB pages: +`N = 1,048,576` frames, `FLAGS_free_pct = 1`, chunk 64. + +``` + finding a victim: a draw hits a COOL frame with probability c ≈ the cool + fraction, so + expected draws per eviction = 1/c + at c = 1%: 100 random frame-header reads per page evicted + each draw is a random access into a 16 GiB array: a near-certain cache miss + (≈219 ns for a random page touch over 128 GB, vmcache Table 2), so + ≈ 100 × 219 ns ≈ 22 µs of sampling per eviction + against an NVMe read of ≈100 µs — the sampling is ~20% of the I/O it + enables, and it runs on a background thread, off the critical path. + + the grace period: with k = 64 draws per round, a given frame waits + N/k = 1,048,576/64 = 16,384 rounds + between visits, on average. That interval IS the second chance — any single + access during it calls resolveSwip's COOL arm and re-swizzles the page. + + what randomness gives up: to visit EVERY frame once, uniform sampling needs + N·ln N = 1,048,576 × 13.86 ≈ 14.5M draws (coupon collector) + where postgres's clock hand needs exactly N = 1,048,576 visits — 14× + fewer. LeanStore does not care, because it never needs full coverage; it + needs one victim. ``` -Why it matters: the HOT arm is the whole point of the design — the case -that runs 99%+ of the time costs what an in-memory system costs. Note also -the latching-order comment at BufferManager.hpp:67–68: swizzle vs coolPage -acquire latches in *conflicting* order; the fix is jump-and-retry -(optimistic abort) instead of blocking — deadlock avoidance by restart, the -same philosophy as Step 4's latches. - -### Step 3 — the cooling stage: replacement with zero per-access cost - -Classic eviction policies do bookkeeping on every access (postgres bumps a -usage counter; DuckDB enqueues on every unpin). LeanStore's -`PageProviderThread` does the bookkeeping *for* you, in the background, -keeping ~10% of frames "cool": - -- Pick a **random** buffer frame (:44) — no LRU metadata exists at all. -- Phase 1 (:52): unswizzle it (turn the parent's raw pointer back into a - tagged frame reference with the cool bit set) — but only if all its - children are already evicted (:90–91, `iterateChildrenSwips`): evict - leaves before parents, bottom-up. (An evicted parent's swip slot can't - hold a hot child's pointer — the child would be unreachable.) -- Cool frames enter a per-partition FIFO (Partition.hpp:65+). Touched while - cool ⇒ resolveSwip warms it (Step 2's second chance — a hot page that was - unluckily sampled gets rescued for the cost of one bit flip). Reaches the - FIFO head untouched ⇒ written back if dirty (AsyncWriteBuffer) and - evicted. - -Random + second-chance approximates LRU with zero per-access cost — compare -postgres (per-access usage bump) and DuckDB (per-unpin enqueue). LeanStore -pays *nothing* per access; that's the whole point of the paper. +Why it matters: the per-access cost of replacement is *zero*, and the price +is paid in sampling draws on a background thread. That is the trade the +paper's §VI-B hit-rate table prices at 0.3 percentage points against LRU. ### Step 4 — hybrid latches: readers that hold nothing -A **latch** is a short-lived lock protecting an in-memory structure; a -classic read latch is an atomic increment that bounces the cache line -between every reading core. LeanStore's `HybridLatch` is a version word -(writers CAS it odd, bump on release), and readers in OPTIMISTIC mode -proceed *without writing anything*: read the version, do the work, -revalidate — version changed ⇒ jump (longjmp-style unwind) and retry. +> **In:** every `Guard` the previous three steps constructed. +> **Out:** the safety argument for all of it — and the abort mechanism +> (`jumpmu`) those steps' `jumpmuTry` blocks depend on. + +A **latch** is a short-lived lock protecting an in-memory structure (as +opposed to a transactional lock over data). A classic read latch is an atomic +increment, which bounces the cache line between every reading core. +LeanStore's `HybridLatch` supports three modes in one 64-byte object: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/sync-primitives/Latch.hpp, HybridLatch, 21-43 + 21 constexpr static u64 LATCH_EXCLUSIVE_BIT = 1ull; + 22 constexpr static u64 LATCH_VERSION_MASK = ~(0ull); + 23 // ------------------------------------------------------------------------------------- + 24 using VersionType = atomic; + 25 struct alignas(64) HybridLatch { + 26 VersionType version; + 27 std::shared_mutex mutex; + 41 bool isExclusivelyLatched() { return (version & LATCH_EXCLUSIVE_BIT) == LATCH_EXCLUSIVE_BIT; } + 42 }; + 43 static_assert(sizeof(HybridLatch) == 64, ""); +``` + +`alignas(64)` plus the `static_assert` at 43 mean one latch is exactly one +cache line — no false sharing between neighbouring frames. The version's low +bit doubles as the exclusive flag, so the word is **odd while held**: +`toExclusive` (:155) takes the mutex and CASes `version → version + 1`, and +`unlock` (:93–104) adds the bit again and release-stores, leaving it even +with a new value. + +Optimistic readers write nothing at all. Validation is one function: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/sync-primitives/Latch.hpp, Guard::recheck, 84-91 + 84 void recheck() + 85 { + 86 // maybe only if state == optimistic + 87 assert(state == GUARD_STATE::OPTIMISTIC || version == latch->ref().load()); + 88 if (state == GUARD_STATE::OPTIMISTIC && version != latch->ref().load()) { + 89 jumpmu::jump(); + 90 } + 91 } +``` -This is what makes swizzling safe: a reader holding no pin can't block -eviction — the page can be cooled or evicted under it, and the reader just -fails validation and retries. It's also topic 9's main subject making an -early appearance. +Line 89 is the abort: not an error code, a `longjmp` back to the enclosing +`jumpmuTry` block. That is why Steps 2 and 3 are written as `jumpmuTry` / +`jumpmuCatch` pairs — any validation failure anywhere inside unwinds to the +top and retries the whole operation, and *that* is the deadlock-avoidance +strategy for the conflicting lock orders Step 2's line 298 warned about. The +guard's `faced_contention` flag (:52) even records that it happened, which +feeds contention-split heuristics elsewhere. + +This is what makes swizzling safe. A reader holding no pin cannot block +eviction — the page may be cooled or evicted underneath it, and the reader +simply fails `recheck()` and starts over. It is also topic 9's subject making +an early appearance. ### Step 5 — the frame header: dirtiness derived, not flagged -`BufferFrame` (BufferFrame.hpp:18–99) carries the latch in its header (:27 -— annotated "NEVER DECREMENT": versions only grow), and defines dirty -without a flag: `isDirty()` = `page.PLSN != last_written_plsn` (:84) — a -page is dirty exactly when its latest change-LSN (log sequence number, the -WAL position of its last modification) is newer than the LSN it was last -written back at. No flag to keep in sync with the WAL — the WAL position -*is* the flag. Nice integration detail to steal for your M6 pool. +> **In:** the `BufferFrame` that Steps 2 and 3 pass around. +> **Out:** the one design detail from this codebase worth copying into the +> capstone's own pool. + +`BufferFrame` (BufferFrame.hpp:18) is a header plus a `Page`, and the header +holds everything the previous steps touched: + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/BufferFrame.hpp, header and isDirty, 18-27 + 18 struct BufferFrame { + 19 enum class STATE : u8 { FREE = 0, HOT = 1, COOL = 2, LOADED = 3 }; + 20 struct Header { + 21 WORKERID last_writer_worker_id = std::numeric_limits::max(); // for RFA + 22 LID last_written_plsn = 0; + 23 STATE state = STATE::FREE; // INIT: + 24 std::atomic is_being_written_back = false; + 25 bool keep_in_memory = false; + 26 PID pid = 9999; // INIT: + 27 HybridLatch latch = 0; // INIT: // ATTENTION: NEVER DECREMENT +``` + +```cpp +// leanstore/leanstore@90fcf18 — backend/leanstore/storage/buffer-manager/BufferFrame.hpp, dirtiness without a flag, 84-85 + 84 inline bool isDirty() const { return page.PLSN != header.last_written_plsn; } + 85 inline bool isFree() const { return header.state == STATE::FREE; } +``` + +A **dirty page** is one modified in RAM but not yet written back — and line +84 defines it without a flag at all: the page is dirty exactly when its +current page LSN (`PLSN`, the WAL position of its last modification, +BufferFrame.hpp:68) differs from the LSN it was last written back at +(:22). There is no `is_dirty` bit to keep in sync with the log; the WAL +position *is* the flag, and it is set once on load (BufferManager.cpp:332) +and once per write-back. Compare postgres, which carries a `BM_DIRTY` flag +bit in the state word and must clear it in the right order relative to +`XLogFlush`. + +Two more details worth stealing: the latch's "NEVER DECREMENT" comment at :27 +(versions only ever grow, so a stale reader can never be fooled by +wraparound within a run), and `alignas(512) struct Page` at :67 — page +buffers are sector-aligned because the reads and writes are `O_DIRECT`. ## Where each step lives in the code -All under `backend/leanstore/` in the classic ICDE '18 repo (local clone -at `~/repos/leanstore`): +Read in this order: `Swip.hpp` (78 lines, all of it), then `Latch.hpp`'s +`HybridLatch` and `Guard::recheck`, then `resolveSwip`, then phase 1 of +`pageProviderThread`. -| File | What | Steps | +| File (under `backend/leanstore/`) | What | Steps | |------|------|-------| -| `storage/buffer-manager/Swip.hpp` | the tagged u64 | 1 | -| `storage/buffer-manager/BufferManager.cpp` | resolveSwip | 2 | -| `storage/buffer-manager/PageProviderThread.cpp` | cooling | 3 | -| `storage/buffer-manager/Partition.hpp` | cooling FIFO | 3 | -| `sync-primitives/Latch.hpp` | hybrid latches | 4 | -| `storage/buffer-manager/BufferFrame.hpp` | frame header, LSN-dirty | 5 | - -- **Step 1**: `Swip.hpp:17–67` — `evicted_bit = 1<<63`, `cool_bit = 1<<62` - (:21–26); `isHOT()` (:45), `isCOOL()` (:46), `isEVICTED()` (:47); - `warm()` clears the cool bit (:62), `cool()` sets it (:65), `evict(pid)` - stores a page id + bit 63 (:67). -- **Step 2**: `resolveSwip` — BufferManager.cpp:281–330 (HOT :283, COOL - :287, EVICTED page fault + `readPageSync` :317); the conflicting - latch-order comment — BufferManager.hpp:67–68. -- **Step 3**: PageProviderThread.cpp — random pick :44, phase 1 unswizzle - :52, children check :90–91 (`iterateChildrenSwips`); cooling FIFO — - Partition.hpp:65+. -- **Step 4**: `HybridLatch` — Latch.hpp:26–41 (`LATCH_EXCLUSIVE_BIT` :41); - `Guard` and the OPTIMISTIC read protocol — :51+. -- **Step 5**: `BufferFrame` — BufferFrame.hpp:18–99; latch in header :27; - `isDirty()` from LSNs :84. +| `storage/buffer-manager/Swip.hpp` | the tagged union | 1 | +| `storage/buffer-manager/BufferManager.cpp` | `resolveSwip`, the miss path | 2 | +| `storage/buffer-manager/Partition.hpp` | `IOFrame`, the I/O hash table, the free list — **no cooling FIFO** | 2, 3 | +| `storage/buffer-manager/PageProviderThread.cpp` | random sampling, cooling, eviction | 3 | +| `sync-primitives/Latch.hpp` | hybrid latches, `recheck`, `jumpmu` | 4 | +| `storage/buffer-manager/BufferFrame.hpp` | frame header, LSN-derived dirtiness | 5 | +| `Config.cpp` | the defaults that make the policy concrete | 3 | + +| Step | Symbol | Location | +|---|---|---| +| 1 | `evicted_bit` (1<<63), `cool_bit` (1<<62), the state comment | Swip.hpp:20–25 | +| 1 | the `union { u64 pid; BufferFrame* bf; }` | Swip.hpp:31–34 | +| 1 | `isHOT` / `isCOOL` / `isEVICTED` | Swip.hpp:45, :46, :47 | +| 1 | `asPageID` (mask bit 63), `asBufferFrameMasked` (mask both) | Swip.hpp:49, :51 | +| 1 | `warm()` clears cool, `cool()` sets it, `evict(pid)` overwrites | Swip.hpp:59–63, :65, :67 | +| 2 | `resolveSwip` — HOT / COOL / EVICTED arms | BufferManager.cpp:281, :283, :287, :298 | +| 2 | the reverse-lock-order comment | BufferManager.cpp:298 | +| 2 | `partition.io_ht` lookup, free-frame pop, `readPageSync` | BufferManager.cpp:305, :307, :317 | +| 2 | swizzle-then-mark-HOT, and why the order matters | BufferManager.cpp:345–347 | +| 2 | a second reader waiting on an in-flight read | BufferManager.cpp:372–386 | +| 2 | `IOFrame` states `READING` / `READY`, `readers_counter` | Partition.hpp:18–33 | +| 3 | `pageProviderThread(p_begin, p_end)` | PageProviderThread.cpp:28 | +| 3 | `randomBufferFrame()` in chunks of `replacement_chunk_size` | PageProviderThread.cpp:40–49 | +| 3 | the trigger: free list below `free_bfs_limit` | PageProviderThread.cpp:64; BufferManager.cpp:55 | +| 3 | already-COOL frames become evict candidates | PageProviderThread.cpp:77–79 | +| 3 | `iterateChildrenSwips`; pick a hot child instead | PageProviderThread.cpp:90–97 | +| 3 | `findParent`; the cooling itself | PageProviderThread.cpp:114, :143–153 | +| 3 | phase 2 `evict_bf`; `ensure(!bf.isDirty())`; `swip.evict` | PageProviderThread.cpp:171, :190, :196 | +| 3 | `free_pct` 1, `pp_threads` 1, `replacement_chunk_size` 64 | Config.cpp:5, :7, :75 | +| 4 | `LATCH_EXCLUSIVE_BIT`; `HybridLatch`; one cache line | Latch.hpp:21, :25–42, :43 | +| 4 | `GUARD_STATE`, `LATCH_FALLBACK_MODE` | Latch.hpp:45, :46 | +| 4 | `Guard::recheck` → `jumpmu::jump()` | Latch.hpp:84–91 | +| 4 | `toExclusive` (mutex + version CAS), `unlock` | Latch.hpp:155, :93–104 | +| 5 | `BufferFrame`, `STATE` enum, header fields, the latch | BufferFrame.hpp:18, :19, :20–27 | +| 5 | `OptimisticParentPointer` — how `findParent` gets cheap | BufferFrame.hpp:45–63 | +| 5 | `alignas(512) struct Page`, `PLSN` / `GSN` | BufferFrame.hpp:67, :68–69 | +| 5 | `isDirty()` from LSNs | BufferFrame.hpp:84 | ## Questions to answer in notes.md @@ -168,18 +436,174 @@ at `~/repos/leanstore`): LeanStore survives in it? (Cooling idea stays; swips go; one-parent constraint gone — that's the headline win.) +## Takeaway + +Three mechanisms, each visible in about twenty lines. A tagged union puts the +page table inside the parent node, so a hot access is a load and a compare. +A background thread samples random frames instead of maintaining metadata, so +an access writes nothing — and pays for it in sampling draws (≈1/c per +eviction) rather than per-access bookkeeping. A version-word latch lets +readers hold nothing and abort by `longjmp`, which is what makes the other +two safe. The code is not the paper: two tag bits instead of one, a +page-provider thread instead of synchronous cooling, and a second random draw +instead of a FIFO. + ## Done when -You can draw the swip state machine (HOT/COOL/EVICTED with transitions and -who performs each) and explain why a hot hit costs zero atomics. +Answer each before unfolding it. + +- [ ] You can draw the swip state machine — HOT / COOL / EVICTED, every transition, and which thread performs each — from the three mutators in `Swip.hpp`. + +
Answer + + ``` + HOT ──cool()──────────────► COOL ──evict(pid)──► EVICTED + ▲ PageProviderThread │ PageProviderThread │ + │ phase 1 (:153) │ phase 2 (:196) │ + │ │ │ + └──── warm() ───────────────┘ │ + │ worker thread, resolveSwip's COOL arm (:294) │ + │ │ + └──── warm(&bf) ──────────────────────────────────────┘ + worker thread, after readPageSync (:345) + ``` + + Two mutators belong to the background provider (`cool()` in phase 1, + `evict()` in phase 2, both under exclusive guards on parent *and* child), + and two belong to whichever worker thread happens to touch the page + (`warm()` on the COOL arm, `warm(&bf)` after a read). The transitions are + strictly HOT → COOL → EVICTED going down, and both downward states jump + straight back to HOT going up — there is no COOL → EVICTED shortcut a + worker can take, and no HOT → EVICTED shortcut at all. + +
+ +- [ ] You can say exactly what a hot page access costs, and why "zero atomics" is not quite the right claim. + +
Answer + + `resolveSwip`'s HOT arm (BufferManager.cpp:283–286) is: test two bits of + the swip, dereference it, and call `swip_guard.recheck()` — which is an + atomic **load** of the parent latch's version word and a compare + (Latch.hpp:88). So it costs two loads and a branch. + + What it does *not* cost is the thing that matters: no atomic + read-modify-write, no store to shared memory, no lock acquisition. A pure + load of a shared cache line leaves it in the Shared state on every reading + core; postgres's `PinBuffer` CAS (bufmgr.c:3351) takes it Exclusive and + bounces it. "Zero atomics" is loose; "zero *writes* to shared state" is the + claim, and it is the one that determines scalability. + +
+ +- [ ] You can explain how the code's replacement policy differs from the paper's, and what plays the role of the cooling FIFO. + +
Answer + + The paper (§IV-C) keeps cooling pages in a FIFO queue plus a hash table + from page id to queue entry, and has *worker* threads do the unswizzling + synchronously, explicitly rejecting background threads. The code has no + queue — `struct Partition` (Partition.hpp:65–104) holds an I/O hash table, + a free list and page-id allocation, and nothing else — and runs a dedicated + `pageProviderThread` (`FLAGS_pp_threads`, default 1). + + The FIFO's role is played by **a second random draw**. Phase 1 samples 64 + frames (`replacement_chunk_size`, Config.cpp:75); a HOT one gets cooled + (:153); an already-COOL one is put on the evict list (:77–79). So a page is + evicted only if it is sampled twice and not accessed in between, and the + expected gap between two draws on the same frame — `N/k` rounds, 16,384 for + a 1,048,576-frame pool — *is* the grace period. It is also why the code + needs the second tag bit the paper did not: COOL has to be visible in the + swip, since there is no queue to look in. + +
+ +- [ ] You can work out how many random draws an eviction costs, and say why that is affordable. + +
Answer + + A draw finds an evictable frame with probability equal to the cool + fraction `c`, so the expected number of draws per eviction is `1/c`. With + `FLAGS_free_pct = 1` (Config.cpp:5) and a cool fraction of the same order, + that is ~100 draws. Each is a random read of a `BufferFrame` header + scattered over the whole pool — a cache miss, on the order of 219 ns for a + random page touch over 128 GB (vmcache Table 2) — so roughly 22 µs of + sampling per page evicted. + + Affordable for two reasons. It buys an SSD read of ~100 µs, so it is about + 20% overhead on the operation it enables; and it happens on a background + thread, so it is not on any query's critical path. What it gives up is + coverage: visiting every frame once takes `N·ln N` ≈ 14.5M draws against + the clock hand's exactly `N` ≈ 1.05M visits. LeanStore never needs + coverage — it needs one victim — which is why it can trade 14× thoroughness + for zero per-access bookkeeping. + +
+ +- [ ] You can explain why `isDirty()` needs no flag, and what postgres has to do instead. + +
Answer + + `isDirty()` is `page.PLSN != header.last_written_plsn` (BufferFrame.hpp:84): + the page's current log sequence number against the LSN it was last written + back at. Any modification advances `PLSN` as a side effect of logging, so + dirtiness is *derived* and cannot drift out of sync with the WAL. The two + places it is set are load (`last_written_plsn = page.PLSN`, + BufferManager.cpp:332) and write-back. + + Postgres instead carries a `BM_DIRTY` flag inside the packed state word and + must order its clearing against `XLogFlush(recptr)` in `FlushBuffer` + (bufmgr.c:4585) and against concurrent dirtiers — which is why + `GetVictimBuffer` needs the comment at bufmgr.c:2577–2583 about a backend + re-dirtying the page between the sweep and the invalidation. A derived + predicate has no such race: there is nothing to clear. + +
+ +- [ ] You wrote answers to all four questions in notes.md, including the tree-or-DAG verdict. + +
Answer + + Nothing to unfold — that verdict is the exercise, and it decides whether + this whole design is admissible for the capstone. The bar for question 1: + walk the eviction of a page reachable from two parents and name the exact + line that cannot be executed (`parent_handler.swip.evict(evicted_pid)`, + PageProviderThread.cpp:196 — `findParent` returns *one* handler). For + question 3, the paper's own answer is §VI-B's table (LeanEvict 92.8% vs LRU + 93.1% at Zipf 1.0); yours should come from `benches/eviction.rs`. + +
## References -**Code** -- [leanstore/leanstore](https://github.com/leanstore/leanstore) (the - classic ICDE '18 codebase) — - `backend/leanstore/storage/buffer-manager/`: `Swip.hpp`, - `BufferManager.cpp`, `BufferFrame.hpp`, `PageProviderThread.cpp`, - `Partition.hpp`; latches in - `backend/leanstore/sync-primitives/Latch.hpp`. Local clone at - `~/repos/leanstore`. +**Code** — [leanstore/leanstore](https://github.com/leanstore/leanstore) at +`90fcf18`, the classic ICDE '18 codebase. Local clone at `~/repos/leanstore`; +all paths below are relative to `backend/leanstore/`. + +| File | Lines | What | +|---|---|---| +| `storage/buffer-manager/Swip.hpp` | 20–34 | the two tag bits and the `union` | +| `storage/buffer-manager/Swip.hpp` | 45–67 | the three predicates and the three mutators | +| `storage/buffer-manager/BufferManager.cpp` | 55 | `free_bfs_limit` from `FLAGS_free_pct` | +| `storage/buffer-manager/BufferManager.cpp` | 281–400 | `resolveSwip`, all three arms and the I/O-frame protocol | +| `storage/buffer-manager/Partition.hpp` | 18–33 | `IOFrame` — how two threads share one read | +| `storage/buffer-manager/Partition.hpp` | 65–104 | `Partition` — note what is *not* in it | +| `storage/buffer-manager/PageProviderThread.cpp` | 28–49 | the thread, and random candidate batches | +| `storage/buffer-manager/PageProviderThread.cpp` | 64–108 | the trigger, the COOL shortcut, the children check | +| `storage/buffer-manager/PageProviderThread.cpp` | 143–153 | cooling, under two exclusive guards | +| `storage/buffer-manager/PageProviderThread.cpp` | 171–200 | phase 2 eviction | +| `sync-primitives/Latch.hpp` | 21–43 | `HybridLatch`: version word + `shared_mutex`, one cache line | +| `sync-primitives/Latch.hpp` | 84–104 | `recheck` and `unlock` | +| `sync-primitives/Latch.hpp` | 155–165 | `toExclusive` | +| `storage/buffer-manager/BufferFrame.hpp` | 18–27 | `STATE` enum and the header | +| `storage/buffer-manager/BufferFrame.hpp` | 45–69 | optimistic parent pointer; `alignas(512)` page | +| `storage/buffer-manager/BufferFrame.hpp` | 84 | `isDirty()` | +| `Config.cpp` | 5, 7, 75 | `free_pct` 1, `pp_threads` 1, `replacement_chunk_size` 64 | + +**Related** +- [`reading-leanstore-paper.md`](reading-leanstore-paper.md) — the design as + published, and the three places this code departs from it. +- [`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md) — the classic + pool this is measured against. +- vmcache (SIGMOD '23) Table 2 — the 219 ns random page touch used in + Step 3's sampling arithmetic. diff --git a/topics/06-buffer-pool/reading-mmap-paper.md b/topics/06-buffer-pool/reading-mmap-paper.md index 40b8d05..3a943cf 100644 --- a/topics/06-buffer-pool/reading-mmap-paper.md +++ b/topics/06-buffer-pool/reading-mmap-paper.md @@ -5,167 +5,423 @@ a general-purpose write-heavy DBMS every apparent win reverses. It is short, punchy, and deliberately provocative — so read it adversarially, then construct the counter-evidence yourself (LMDB exists and is excellent). Before you open it, this chapter builds the concepts one at a time — what -mmap actually does, why it tempts database authors, and the four distinct -ways it betrays them — then hands you a section-by-section reading lens. The -payoff is knowing precisely *which* property of a workload makes mmap wrong. +mmap actually does, what it costs *here* on the machine this repo measures, +why it tempts database authors, and the four distinct ways it betrays them — +then hands you a section-by-section reading lens. The payoff is knowing +precisely *which* property of a workload makes mmap wrong. + +The paper is Crotty, Leis and Pavlo, *"Are You Sure You Want to Use MMAP in +Your Database Management System?"*, CIDR 2022 — 7 pages. Every figure quoted +below carries the section, figure or table it came from in that paper. Every +*measured* number is this repo's own: topic 6's `pool_vs_mmap` lane, +[FINDINGS.md](../../FINDINGS.md) row 6, with the full output in +[`notes.md`](notes.md). Nothing here is remembered. ## The problem in one sentence If you let the kernel manage your database's memory via `mmap`, the kernel — -not you — decides when dirty pages hit disk, so write-ahead logging becomes -unenforceable; and even for pure reads, the paper measures mmap plateauing -far below the ~6 GB/s a single NVMe drive can deliver, then *degrading over -time* once eviction starts. +not you — decides when dirty pages reach disk, so write-ahead logging becomes +unenforceable; and even for pure reads the cost of an access is bimodal, this +repo measuring **42 ns at the median against 182 µs at the maximum** on the +same instruction ([FINDINGS.md](../../FINDINGS.md) row 6), with the database +unable to tell the two apart in advance. ## The concepts, step by step ### Step 1 — what mmap actually does -`mmap` asks the OS to map a file directly into your process's virtual -address space: after the call, `file_bytes[i]` is just a pointer -dereference, no `read()` syscall. Nothing is copied up front. The first -touch of each 4 KB region triggers a **page fault** (a hardware trap into -the kernel), the kernel reads that page of the file into its **page cache** -(the kernel's own cache of file data in RAM) and wires the mapping; later -touches are plain memory access. Eviction is also the kernel's job: under -memory pressure it writes dirty pages back and unmaps them — whenever it -likes. +> **In:** nothing yet — this step fixes the vocabulary every later step +> leans on. +> **Out:** the seven-stage access path of the paper's Fig. 1, and the two +> words (*page fault*, *TLB shootdown*) that Steps 2 and 7 turn into numbers. + +`mmap` asks the OS to map a file into your process's **virtual address +space** — the range of addresses your process can name, which the hardware +translates to physical RAM addresses. After the call, `file_bytes[i]` is a +pointer dereference, not a `read()` syscall. Nothing is copied up front. + +The paper's Fig. 1 walks the seven stages (§2.1). Condensed: the program +calls `mmap` and gets a pointer ①; the OS reserves address space but loads no +data ②; the program dereferences the pointer ③; the OS looks for a mapping +④; finding none it takes a **page fault** — a hardware trap into the kernel +raised when a touched address has no valid translation ⑤; the kernel loads +the page into the **page cache**, its own RAM cache of file data ⑥; and adds +the translation to the **page table** (the kernel's per-process map from +virtual to physical addresses) and to the faulting core's **TLB** — the +translation lookaside buffer, a small per-core hardware cache of recent +virtual→physical translations ⑦. + +Two faults, not one, and the distinction runs through this whole chapter. A +**major page fault** needs a disk read because the data is not in RAM at all. +A **minor page fault** needs no I/O: the bytes are already in the page cache, +only *this* process's page-table entry is missing, so the kernel just wires +the mapping up. Minor faults are the cheap kind — and Step 2 shows that +"cheap" still means three orders of magnitude worse than a hit. + +Eviction is the kernel's job too, and it is where the asymmetry lives. When +the OS evicts a page it must remove the translation from the page table *and* +from every core's TLB. Flushing the local core's TLB is easy; remote cores +are the problem, because — as §2.1 states — current CPUs provide no coherence +for remote TLBs, so the OS must send an **inter-processor interrupt** (an +IPI: one core forcibly interrupting another) to make each remote core flush. +That is a **TLB shootdown**, and §3.4 prices it at thousands of cycles, +citing Villavieja et al. Why it matters: you got a demand-paged cache of the file for ~zero code. -Everything below is the bill for the words "whenever it likes". +Everything below is the bill — and note already that faulting a page *in* +touches one core, while throwing one *out* touches all of them. -### Step 2 — what a buffer pool is, and why mmap tempts +### Step 2 — the cost, measured on this machine -A **buffer pool** is the fixed-size in-memory cache of disk pages that the -database engine manages itself: a `page_id → frame` map, pin counts (a -counter saying "this page is in use, don't evict it"), an eviction policy, -and explicit read/write calls. That's thousands of lines of subtle -concurrent code — and mmap seems to make all of it free: no copies between -kernel and user space, no eviction code, pointer access, and the page cache -is shared with every other process. - -Real systems took the bait: MongoDB (MMAPv1 — abandoned), LMDB (kept it, -happily), SQLite (optional), RavenDB. The paper's claim: for a -*general-purpose write-heavy DBMS*, every apparent win reverses. The next -four steps are the four reversals — memorize them: +> **In:** the fault vocabulary from Step 1. +> **Out:** the two numbers — 42 ns resident, 4459 ns at p99.9 — that make +> "the kernel decides" a latency problem rather than an aesthetic one. Steps +> 5 and 7 spend them. + +This repo's topic 6 lane does the smallest honest version of the paper's +experiment: `cargo run --release --bin pool_vs_mmap` maps a 1 GiB file +(262,144 pages of 4 KiB) and performs 2,000,000 Zipf(0.99)-distributed page +reads, touching 8 bytes per page so that the access — not a `memcpy` — +dominates. On an Apple M3 Pro it prints ([`notes.md`](notes.md), baseline +measured 2026-07-28): + +``` +mmap p50 42 ns p99 1500 ns p99.9 4459 ns max 181887 ns +``` + +The file is 1 GiB and the machine's page cache is far larger, so essentially +none of these are major faults. The tail is minor faults: pages the kernel +holds but has not mapped into this process, plus the eviction traffic that +mapping them provokes. + +Now the division that makes the spread mean something: ``` - 1. Transactional safety kernel may flush a dirty page ANY time - ──────────────────── ⇒ can't order page-write after log-write - ⇒ WAL rule unenforceable without COW tricks - 2. I/O stalls page fault = your thread stops; no async, - ──────────────────── no prefetch you control, no admission control - 3. Error handling disk error = SIGBUS in the middle of a memcpy, - ──────────────────── not an error code at a syscall boundary - 4. Performance (§4) the surprise: even READ-ONLY mmap loses at scale +spread, max over median: 181887 / 42 = 4330× ``` -### Step 3 — problem 1: the kernel breaks the WAL rule +A single instruction — the same load, in the same loop — costs either 42 ns +or 182 µs, and nothing in the program can tell which. Ask next how *rare* the +bad case has to be before it stops mattering. Let f be the fraction of +accesses that fault, 42 ns the resident cost, and C the cost of a fault; then + +``` +mean = 42(1 − f) + C·f = 42 + (C − 42)·f [ns per access] + +C = 1500 ns (the measured p99, the cheapest fault in the run): + mean doubles at f = 42 / 1458 = 0.0288 = 2.88% ≈ 1 access in 35 +C = 4459 ns (the measured p99.9): + mean doubles at f = 42 / 4417 = 0.0095 = 0.95% ≈ 1 access in 105 +C = 181887 ns (the measured max): + mean doubles at f = 42 / 181845 = 0.00023 = 0.023% ≈ 1 access in 4330 +``` + +One access in a hundred is enough to double the average cost of every access +in the program. Turn it around and price the tail that was actually measured: +the run's slowest 0.1% is 2,000 samples of at least 4459 ns, so those alone +contribute at least `0.001 × 4459 = 4.46 ns` to the mean — more than a tenth +of the median's *entire* cost, contributed by one access in a thousand. The +slowest 1% (20,000 samples at ≥ 1500 ns) accounts for at least 30 ms, against +the 84 ms an all-median run of 2,000,000 × 42 ns would have taken: **1% of +the accesses, 36% again of the whole idealised run.** + +Why it matters: mmap's median is genuinely excellent, and that is the trap. +The paper's four problems are all arguments about the other 1%, and the other +1% is where the run's time is. + +### Step 3 — what a buffer pool is, and why mmap tempts + +> **In:** the fault costs from Step 2. +> **Out:** the machinery mmap appears to make unnecessary, and the list of +> real systems that took the bet — the setup for the four reversals in Steps +> 4 to 7. + +A **buffer pool** is the fixed-size in-memory cache of disk pages that the +database engine manages itself. Its parts, each of which has a step of its +own in this topic's other chapters: + +- a **page**: the fixed-size unit of transfer between disk and memory + (8 KB in postgres, 4 KiB in this topic's lane, 16 KB in LeanStore's + experiments); +- a **frame**: one slot of RAM that holds one page, plus its bookkeeping; +- a `page_id → frame` map, so a page reference can find its frame; +- **pin** and **unpin**: taking and releasing a reference count on a frame + that makes it ineligible for eviction while you hold a pointer into it; +- a **dirty page**: one modified in RAM whose changes are not yet on disk; +- an **eviction policy**: the rule choosing which unpinned frame to reuse + when a new page must be read in. + +That is thousands of lines of subtle concurrent code — and mmap appears to +make all of it free: no copy between kernel and user space, no eviction code, +no pin counts, pointer access, and a page cache shared with every other +process. + +Real systems took the bait. Table 1 lists ten and the years each used mmap: +MonetDB (2002–), MongoDB (2009–2019), LevelDB (2011–), LMDB (2011–), SQLite +(2013–), SingleStore (2013–2015), QuestDB (2014–), RavenDB (2014–), InfluxDB +(2015–2020) and WiredTiger (2020–). **Table 1 is that list and nothing more** +— it is not a table of verdicts, and the paper's own concessions live in §6, +which Step 8 quotes. + +§2.3 tells the cautionary half. MongoDB's MMAPv1 needed "an overly complex +copying scheme" and could not compress on-disk data, and was deprecated in +2015 and removed in 2019. SingleStore's `mmap` calls took 10–20 ms per query +— *nearly half of total query runtime* — on contention over a shared mmap +write lock, and switching to `read` made the queries CPU-bound. InfluxDB hit +write I/O spikes past a few GB and dropped mmap for IOx. RocksDB was forked +out of LevelDB partly over read bottlenecks caused by the latter's mmap use. + +Why it matters: this is not a hypothesis about what might go wrong. It is a +list of engineering teams who paid for the discovery. + +### Step 4 — problem #1 (§3.1): the kernel breaks the WAL rule + +> **In:** the buffer pool vocabulary from Step 3 — dirty pages in +> particular. +> **Out:** the ordering invariant mmap cannot express, and the three +> workarounds the paper catalogues, one of which is LMDB's and returns in +> Step 8. Write-ahead logging (topic 5) rests on one ordering invariant: a modified page may reach disk only *after* the log record describing the modification -is durable — otherwise a crash leaves a page whose history the log doesn't -contain, and recovery cannot undo it. A buffer pool enforces this trivially: -it controls every page write, so it checks "is the log flushed up to this -page's LSN?" before each one. - -With mmap the kernel flushes dirty pages on its own schedule — memory -pressure, periodic writeback, whenever. There is no hook that says "not this -page, not yet." Your only levers are `msync` gymnastics (flushing -*everything* at barriers) or copy-on-write shadow-paging tricks that give up -in-place updates entirely. - -Why it matters: this problem alone disqualifies mmap for any engine with -in-place updates + WAL — which is postgres, MySQL, and your topic-3/5 stack. - -### Step 4 — problem 2: page faults are I/O you can't schedule - -When your thread touches an unmapped page, it stops dead until the kernel -finishes the disk read — a **stall**. There is no way to say "start fetching -these 8 pages, I'll do other work meanwhile": no async interface, no -prefetch you control (the kernel's readahead guesses, and guesses wrong for -random access), and no admission control (nothing stops 100 threads from -faulting at once and burying the disk). A buffer pool turns every miss into -an explicit I/O request it can batch, reorder, and overlap; mmap turns every -miss into a surprise nap of ~100 µs (NVMe) mid-instruction. - -### Step 5 — problem 3: errors arrive as signals, not return codes - -With explicit I/O, a failed read returns an error code at a syscall -boundary, where you have context and can respond. With mmap, a disk error -surfaces as a **SIGBUS** signal (a hardware-fault signal) delivered in the -middle of whatever instruction touched the page — possibly deep inside a -`memcpy` in a third-party library. Handling that means a process-wide signal -handler that somehow maps a faulting address back to a database operation -and unwinds safely. Nobody does this well; most mmap systems just crash. - -### Step 6 — problem 4: even read-only mmap loses at scale - -You might concede writes and still want mmap for read-only analytics. §4 is -the paper's surprise: three kernel bottlenecks cap read throughput, measured -with fio on multi-NVMe arrays: - -- **page table contention** — parts of the kernel's page-fault path are - effectively single-threaded; concurrent faulting cores serialize. -- **TLB shootdowns** — the TLB (the per-core cache of virtual→physical - translations, topic 0) may hold a stale entry on *any* core after an - unmapping, so evicting one page sends an interrupt (IPI) to every core to - flush it. Eviction cost *scales with core count* — more cores, worse. -- **4 KB granularity** — mmap moves data one page-table-walk-managed 4 KB - page at a time; one explicit 2 MB `pread` does the same work with a single - syscall and no per-page kernel bookkeeping. - -Result: explicit `pread`/O_DIRECT sustains device bandwidth; mmap plateaus -far below on NVMe arrays and *degrades over time* once eviction (and its -shootdowns) begins. - -Why it matters: note the asymmetry for question 2 below — faulting a page -*in* touches only the faulting core's mappings; evicting must chase every -core that might have cached the translation. - -### Step 7 — the rebuttal you must construct: LMDB and the escape hatches +is durable. Otherwise a crash leaves a page whose history the log does not +contain, and recovery cannot undo it. A buffer pool enforces this trivially +because it performs every page write itself, so it can check "is the log +flushed past this page's LSN?" first. -LMDB (topic 3) is mmap-based and wins its niche, so "never mmap" is too -strong; the truth is a checklist. LMDB dodges each bullet: copy-on-write -pages are never overwritten, so problem 1's ordering is a non-problem — the -meta-page flip IS the commit; read-mostly workloads fault once, then it's -just memory (problem 2); a read-only mmap can't SIGBUS on writes (problem -3); and its scale target is "fits mostly in RAM" (problem 4). The paper's -own Table 1 concedes designs like this. +§3.1 states the core issue exactly: "due to transparent paging, the OS can +flush a dirty page to secondary storage at any time, irrespective of whether +the writing transaction has committed. The DBMS cannot prevent these flushes +and receives no warning when they occur." -Map it to what you know: +The obvious lever does not work either. `mlock` pins pages in memory, but +§2.2 records that POSIX (and Linux) permit the OS to flush a dirty page to +the backing file *even while it is locked*. Pinning stops eviction, not +write-back. There is no call that means "not this page, not yet". -| System | Uses | Escapes the trap because | +So mmap-based systems buy the ordering back with a protocol. §3.1 classifies +all of them into three: + +| Protocol | Who uses it (§3.1) | What it costs | |---|---|---| -| LMDB | mmap everything | COW + read-mostly + single writer | -| SQLite | optional mmap for reads | WAL still explicit; mmap read-only | -| postgres | no mmap; shared_buffers | needs write ordering (FPIs, ckpts) | -| LeanStore/vmcache | anonymous mem / virt mapping | explicit residency control | +| **OS copy-on-write** — a second `MAP_PRIVATE` mapping as a private workspace, changes applied there, WAL for durability, a background thread propagating to the primary | MongoDB MMAPv1 | bookkeeping for pages with pending updates; the private workspace grows toward a *second full copy* of the database, needing periodic `mremap` compaction that must itself block pending changes | +| **User space copy-on-write** — copy the affected pages out of the mapping into a user-space buffer, update and log there, copy back after the WAL is durable | SQLite, MonetDB, RavenDB | a page copy per update (some systems apply WAL records straight into the mapping to avoid it) | +| **Shadow paging** — primary and shadow copies both mmap'd; copy the page to the shadow, change it, `msync` the shadow, then flip a pointer to install it as primary | LMDB | the copy per updated page, plus (in LMDB) only a single writer, which is how it keeps transactions from seeing partial updates | + +Every one of those is machinery — which is the paper's rhetorical point: the +thing you adopted mmap to avoid writing, you end up writing anyway, in a +harder form. + +Why it matters: this problem alone disqualifies mmap for any engine that does +in-place updates under a WAL — postgres, MySQL, and your topic-3/5 stack. + +### Step 5 — problem #2 (§3.2): page faults are I/O you cannot schedule + +> **In:** Step 2's measured tail and Step 1's fault taxonomy. +> **Out:** the three workarounds §3.2 evaluates and rejects, plus the +> read-amplification arithmetic of the default `madvise` hint. + +With a buffer pool a miss is an explicit request, so the engine can issue it +asynchronously (`libaio`, `io_uring`), batch it, reorder it, or overlap it +with computation. §3.2's example is a B+tree leaf scan: the reads for +non-contiguous leaves could all be issued at once to mask latency — but +"mmap does not support asynchronous reads". Worse, since the OS may +transparently evict, §3.2 notes that even a *read-only* query can trigger a +blocking fault, "because the DBMS cannot know whether the page is in memory". +That is Step 2's bimodality restated as a scheduling problem: 42 ns or +182 µs, and no way to ask which before committing to the load. + +The paper works through the escape hatches: + +- **`mlock`** — pin the pages you will need again. But the OS limits how much + memory a process may lock, and you must track and unlock pages yourself. +- **`madvise`** — **`madvise` is the call that hands the kernel a hint about + an expected access pattern**, per file or per page range. §2.2 covers three + hints: `MADV_NORMAL`, `MADV_RANDOM`, `MADV_SEQUENTIAL`. They are hints; the + OS may ignore them, and §3.2 warns that the wrong one "can have dire + implications for performance". +- **Prefetch threads** — spawn helpers to touch pages so *they* block instead + of the query thread. It works, and it is a thread pool you now maintain to + simulate the async I/O interface you gave up. + +The default hint is worth doing the arithmetic on. §2.2: under `MADV_NORMAL` +a fault fetches the accessed page **plus the next 16 and the previous 15**. +With 4 KB pages: -The honest conclusion: **mmap is wrong when the DB must control -WRITE-BACK.** Read-only/COW designs escape most of it. And vmcache -(SIGMOD '23, reading-leanstore-paper.md) is the synthesis: keep -virtual-memory *addressing*, but the DB — not the kernel — keeps explicit -control of residency and eviction. +``` +pages moved per fault: 1 + 16 + 15 = 32 pages +bytes moved per fault: 32 × 4 KB = 128 KB (§2.2's figure) +read amplification for a 4 KB random read: = 32× + +what that costs on the paper's own drive (§4: Samsung PM1733, rated +7000 MB/s read), if every fault moves its full 128 KB: + 7,000,000,000 B/s ÷ 131,072 B = 53,406 faults/s +against 4 KB per fault: + 7,000,000,000 B/s ÷ 4,096 B = 1,708,984 faults/s +``` + +For a random-access OLTP workload the default hint spends 97% of the device's +bandwidth on pages nobody asked for — which is why §2.2 recommends +`MADV_RANDOM` for larger-than-memory OLTP and `MADV_SEQUENTIAL` for scans. + +Why it matters: every workaround is *more* code than the buffer-pool call it +replaces, and none of them restores the one thing you wanted: knowing, before +you dereference a pointer, whether it will cost 42 ns or a trip to the +kernel. + +### Step 6 — problem #3 (§3.3): errors arrive as signals, not return codes + +> **In:** Steps 4 and 5 — the kernel writes when it likes and faults when it +> likes. +> **Out:** the third problem, which is about *correctness plumbing* rather +> than performance, and is the one most often shortened to "SIGBUS" and +> thereby under-stated. + +§3.3 makes three distinct points, and the third is the famous one. + +1. **Checksums must be re-validated on every access.** A DBMS that keeps a + page checksum normally verifies it once, when the page is read into the + pool. Under mmap the OS may have transparently evicted and re-read the + page since your last access, so the check has to happen on *every* access + to mean anything. +2. **Corruption is persisted silently.** These systems are typically written + in memory-unsafe languages, and a stray pointer write lands in a mapped + page. A buffer pool can check pages before it writes them out, because it + performs the write; mmap "will silently persist corrupted pages to the + backing file". +3. **I/O errors become signals.** With explicit I/O a failed read returns an + error code at a syscall boundary, and handling can be contained in one + module. With mmap, any code that touches mapped memory can raise a + **`SIGBUS`** — a hardware-fault signal — delivered in the middle of + whatever instruction touched the page, possibly deep inside a third-party + `memcpy`. The handler must map a faulting address back to a database + operation and unwind safely. + +Why it matters: the first two are the ones people forget. mmap does not just +make error *handling* awkward; it moves the checksum from the miss path, +where it is free, onto the hit path, where Step 2 says you have 42 ns to +spend. + +### Step 7 — problem #4 (§3.4, §4): even read-only mmap loses at scale + +> **In:** the TLB-shootdown mechanism from Step 1. +> **Out:** the paper's three named bottlenecks and the measured gaps — +> the numbers Step 8's rebuttal has to survive. + +You might concede writes and still want mmap for read-only analytics. §3.4 +is the surprise, and it names exactly three bottlenecks — worth getting +right, because a fourth is often invented for this list: + +1. **Page table contention** — the OS must synchronize the page table, "which + becomes highly contended with many concurrent threads" (§4.1). +2. **Single-threaded page eviction** — "the OS uses only a single process + (`kswapd`) for page eviction, which was CPU-bound in our experiments" + (§4.1). One kernel thread against 128 hardware threads of demand. +3. **TLB shootdowns** — Step 1's IPI storm, thousands of cycles each (§3.4), + measured through `/proc/interrupts` and plotted in Fig. 2b. + +§3.4 adds the crucial asymmetry: shootdowns "occur during page eviction when +a core needs to invalidate mappings in a remote TLB". Faulting a page *in* +installs a translation on one core. Evicting one must chase every core that +might hold it — so eviction cost grows with core count. That is question 2 +below, and it is why the paper's plots are flat until the page cache fills +and then fall off a cliff. + +The measurements (§4), on an AMD EPYC 7713 (64 cores, 128 hardware threads), +512 GB RAM of which 100 GB was available to Linux 5.11 for its page cache, +and 10 × 3.8 TB Samsung PM1733 SSDs rated 7000 MB/s read, accessed as raw +block devices; the baseline is `fio` 3.25 with `O_DIRECT`: + +| Experiment | fio | mmap | Where | +|---|---|---|---| +| Random reads, 2 TB range, 100 threads (95% of accesses fault) | ~900K reads/s, stable | matched fio for the first 27 s, **dropped to nearly zero for ~5 s**, recovered to about **half** of fio | §4.1, Fig. 2a | +| Sequential scan, 1 SSD | full device bandwidth, stable | matched fio, then fell off after ~17 s | §4.2, Fig. 3 | +| Sequential scan, 10 SSDs (RAID 0) | scales | **~20× worse**, "virtually no improvement over the results from using one SSD" | §4.2, Fig. 4 | + +The fio baseline is worth checking rather than trusting, and §4.1 shows its +work: 100 threads each with one outstanding I/O against an NVMe latency of +roughly 100 µs gives + +``` +100 outstanding I/Os ÷ 0.000100 s = 1,000,000 reads/s (the ceiling) +measured: ~900,000 reads/s = 90% of it +``` + +so `fio` really is saturating the device, and mmap's post-collapse half is +half of a real number. The paper's summary sentence (§4.2): mmap "performs +well only on a single SSD during the initial loading phase. Once page +eviction begins or when using multiple SSDs, mmap is 2–20× worse than fio." + +Note what triggers every collapse: the page cache filling up, i.e. **the +moment eviction starts**. Before that, mmap is fine. This is the same shape +as Step 2's spread on a laptop, three orders of magnitude larger. + +Why it matters: the read-only case was supposed to be mmap's safe harbour, +and it is the case the paper measures losing. + +### Step 8 — the rebuttal you must construct: LMDB and the escape hatches + +> **In:** all four problems (Steps 4–7). +> **Out:** the checklist that decides, per workload, whether the paper's +> conclusion applies to you — and the paper's own version of that checklist. + +LMDB (topic 3) is mmap-based and wins its niche, so "never mmap" is too +strong. The honest form is a checklist, and LMDB dodges each bullet: + +- **Problem 1** — LMDB uses shadow paging (§3.1's third protocol): pages are + never overwritten in place, so no ordering rule can be violated by an + untimely flush. Commit is `msync` of the modified shadow pages followed by + the pointer flip that installs them as primary. A single writer removes the + conflict cases. +- **Problem 2** — read-mostly workloads fault once per page and then run at + Step 2's 42 ns. +- **Problem 3** — a read-only mapping cannot silently persist a corrupted + page, which removes §3.3's second point; the SIGBUS exposure remains. +- **Problem 4** — the collapse in §4 starts when the page cache fills. + LMDB's target is a working set that fits. + +| System | Uses | Escapes the trap because | +|---|---|---| +| LMDB | mmap everything | shadow paging + read-mostly + single writer | +| SQLite | optional mmap for reads | WAL still explicit; mapping used read-only | +| postgres | no mmap; `shared_buffers` | needs write ordering (FPIs, checkpoints) | +| LeanStore / vmcache | anonymous memory, explicit residency | the DB, not the OS, decides eviction | + +And the paper says the same thing itself, in §6, which is where its +concessions actually live (not Table 1). *When you should not use mmap*: you +need transactionally safe updates; you need to handle page faults without +blocking or need explicit control over what is in memory; you care about +error handling; you require high throughput on fast storage. *When you might*: +"Your working set (or the entire database) fits in memory and the workload is +read-only" — or "you need to rush a product to the market and do not care +about data consistency or long-term engineering headaches. Otherwise, never." + +The honest conclusion: **mmap is wrong when the DB must control write-back, +and unpredictable whenever the page cache is under pressure.** Read-only, +shadow-paged, fits-in-RAM designs escape most of it. vmcache (SIGMOD '23, +[`reading-leanstore-paper.md`](reading-leanstore-paper.md) Step 6) is the +synthesis the paper itself gestures at in §5 when it endorses "lightweight +buffer management techniques" like **pointer swizzling** — storing the +in-memory frame pointer where the page id would go, so a resident access +needs no lookup: keep virtual-memory *addressing*, but let the DB keep +explicit control of residency and eviction. ## How to read the paper (with the concepts in hand) -It's a short CIDR position paper — one sitting. +Seven pages, one sitting. -- **§1–2 (the temptation)** — skim; this is Step 2. Note the list of - systems that tried and where each landed. -- **§3 (the four problems)** — read carefully and memorize; Steps 3–5. - For each problem, ask "which mechanism from my topic-5 WAL does this - break?" -- **§4 (performance)** — the part worth re-reading; Step 6. Study the fio - plots: where mmap plateaus, and the degradation-over-time curve once - eviction starts. This is the section people skip and shouldn't. -- **Table 1** — the paper's own concession matrix; check LMDB's row against - Step 7 and argue with any cell you disagree with. +| Section | How to read it | Step | +|---|---|---| +| §1, §2.1 | The mmap mechanism and Fig. 1's seven stages. Skim if Step 1 landed, but read the TLB-shootdown paragraph at the end of §2.1 twice — it is the mechanism behind §4. | 1 | +| §2.2 | The POSIX API. The two sentences worth memorising: `mlock` does not prevent flushes, and `MADV_NORMAL` fetches 32 pages. | 4, 5 | +| §2.3 + Table 1 | Who tried it and what happened. Table 1 is a *list of systems and years*, not a verdict matrix. | 3 | +| §3.1–3.4 | The four problems. For each, ask "which mechanism from my topic-5 WAL does this break?" | 4–7 | +| §4 | The part people skip. Study *when* each line falls over — always at the moment the page cache fills — and Fig. 2b's shootdown rate next to Fig. 2a's throughput. | 7 | +| §5 | Two paragraphs, easily missed: the authors endorse pointer swizzling as the right alternative. That is topic 6's LeanStore thread. | 8 | +| §6 | The prescription. This, not Table 1, is where the paper concedes. | 8 | -Read it adversarially: the authors are deliberately provocative, and the -LMDB rebuttal (Step 7) is *yours* to construct — the paper won't do it for -you. +Read it adversarially: the authors are deliberately provocative, and the LMDB +rebuttal (Step 8) is *yours* to construct — the paper won't do it for you. ## Questions to answer in notes.md @@ -181,12 +437,173 @@ you. ## Done when -You can argue both sides for five minutes each — "never mmap" and "LMDB is -right" — and state precisely which property of your workload picks the side. +Answer each before unfolding it. + +- [ ] You can argue both sides for five minutes each — "never mmap" and "LMDB is right" — and state precisely which property of your workload picks the side. + +
Answer + + The "never mmap" case is §3.1's sentence — the OS can flush a dirty page at + any time, irrespective of whether the writing transaction committed, and + `mlock` does not stop it (§2.2) — plus §4's measurements: mmap matched fio + for 27 seconds and then dropped to nearly zero for five, recovering to half + of fio's ~900K reads/s (Fig. 2a), and was ~20× worse across 10 SSDs + (Fig. 4). + + The "LMDB is right" case is that all four problems are conditional. Shadow + paging (§3.1's third protocol) never overwrites a page, so there is no + ordering to violate; a single writer removes the conflict cases; read-mostly + access faults once per page and then runs at this repo's measured 42 ns + ([FINDINGS.md](../../FINDINGS.md) row 6); and every collapse in §4 begins + when the page cache fills, which a fits-in-RAM working set never does. + + The property that picks the side is **who must control write-back, and + whether the working set fits**. In-place updates under a WAL need the + engine to order page writes against log writes, and mmap cannot express + that order at all. A larger-than-memory working set puts you on the far + side of §4's cliff, where eviction — and its TLB shootdowns — runs + continuously. + +
+ +- [ ] You can name the paper's four problems and the *three* bottlenecks behind the fourth, without inventing a fourth bottleneck. + +
Answer + + The four problems are §3.1 transactional safety, §3.2 I/O stalls, §3.3 + error handling, §3.4 performance. The three bottlenecks §3.4 names behind + the fourth are: **page table contention** (the OS must synchronize the page + table, and it becomes highly contended with many concurrent threads); + **single-threaded page eviction** (Linux evicts with the single `kswapd` + process, which was CPU-bound in the paper's runs); and **TLB shootdowns** + (thousands of cycles each, measured via `/proc/interrupts` in Fig. 2b). + + "4 KB granularity" is *not* on that list, though the temptation to add it + is understandable — §2.2's `MADV_NORMAL` behaviour (fetch the accessed page + plus the next 16 and previous 15, so 128 KB per fault at 4 KB pages) is a + read-amplification argument, and it appears in the paper as a *madvise* + discussion, not as a §3.4 bottleneck. Keep them separate: the first three + are properties of the kernel's paging machinery under concurrency, the + fourth is a tunable hint. + +
+ +- [ ] You can explain why a page fault the database cannot see is worse than a slow read it can, using this topic's own measured numbers. + +
Answer + + Because the cost is bimodal and unannounced. The same load instruction in + the same loop costs 42 ns at the median and 181,887 ns at the maximum + ([FINDINGS.md](../../FINDINGS.md) row 6, full output in + [`notes.md`](notes.md)) — a 4330× spread — and there is no call the engine + can make beforehand to learn which it is about to pay. An explicit read is + slower than 42 ns but it is *scheduled*: it can be issued asynchronously, + batched with its neighbours, or overlapped with computation, all of which + §3.2 says mmap cannot do because "mmap does not support asynchronous + reads". + + The arithmetic says how little of the bad case it takes. With f the + fraction of accesses that fault, the mean is `42 + (C − 42)·f`; at the + measured p99 fault cost of 1500 ns the mean doubles at f = 42/1458 = 2.88%, + and at the p99.9 cost of 4459 ns it doubles at f = 42/4417 = 0.95% — one + access in a hundred. In the run as measured, the slowest 1% (20,000 + accesses at ≥ 1500 ns) accounts for at least 30 ms against the 84 ms an + all-median run would have taken. + +
+ +- [ ] You can state what `mlock` does and does not guarantee, and why that single fact kills the simplest fix for problem #1. + +
Answer + + `mlock` pins pages in memory so the OS will not evict them. §2.2 is + explicit that this is *all* it does: "according to the POSIX standard (and + Linux's implementation), the OS is permitted to flush dirty pages to the + backing file at any time, even if the page is pinned." + + The simplest imagined fix for problem #1 is "lock the dirty pages until the + log is flushed, then unlock them" — a two-line change that would make WAL + ordering enforceable. It does not work, because locking controls + *residency* and the WAL rule is about *write-back*, and mmap exposes no + call that separates the two. That is why §3.1's three protocols are all + structural — a private `MAP_PRIVATE` workspace, a user-space copy, or + shadow paging — rather than a flag. + +
+ +- [ ] You can say which single event triggers every performance collapse in §4, and why it is the eviction side rather than the fault side. + +
Answer + + The page cache filling up. In §4.1 mmap tracked fio for 27 seconds and then + dropped to nearly zero for about five; in §4.2's single-SSD scan the drop + came after about 17 seconds. Both are the moment the OS must start evicting + to make room, and §4.1 says so directly: "This sudden drop in performance + occurred when the page cache filled up, forcing the OS to begin evicting + pages from memory." + + It is the eviction side because of the asymmetry in §2.1. Faulting a page + *in* installs one translation in the page table and caches it in the + faulting core's TLB — a local operation. Evicting must remove the + translation from the page table *and* from every core's TLB, and since + current CPUs provide no coherence for remote TLBs, the OS sends an + inter-processor interrupt to each — a TLB shootdown, thousands of cycles + (§3.4). So fault-in cost is independent of core count while eviction cost + grows with it, which is why Fig. 2b's shootdown rate rises exactly where + Fig. 2a's throughput falls. + +
+ +- [ ] You wrote answers to all four questions in notes.md, including naming the specific `crash_test.rs` case that would fail under mmap. + +
Answer + + There is no answer to unfold here — tracing your own topic-5 tests against + §3.1 is the exercise. The bar: name a test whose assertion is about + *ordering* rather than content, because those are the ones an untimely + kernel flush breaks. A test that crashes after a page is modified but + before its log record is durable, then asserts recovery can undo the + change, is asserting exactly the invariant §3.1 says the OS may violate + "at any time, irrespective of whether the writing transaction has + committed." + + An answer that says "all of them" has not done the work: tests that only + check that committed data survives are fine under mmap, because `msync` at + commit is enough for those. + +
## References **Papers** -- Crotty, Leis, Pavlo — "Are You Sure You Want to Use MMAP in Your DBMS?" - (CIDR 2022) — short position paper; memorize the four problems of §3, - re-read §4 (why even read-only mmap loses at scale) +- Crotty, Leis, Pavlo — *"Are You Sure You Want to Use MMAP in Your Database + Management System?"* (CIDR 2022) — + [PDF](https://db.cs.cmu.edu/papers/2022/p13-crotty.pdf) — 7 pages, one + sitting. Memorize the four problems of §3; re-read §4. + +| Section | What this chapter took from it | +|---|---| +| §2.1, Fig. 1 | the seven-stage access path; TLB shootdowns as IPIs, because CPUs give no coherence for remote TLBs | +| §2.2 | `MAP_SHARED`/`MAP_PRIVATE`; the three `madvise` hints and `MADV_NORMAL`'s 128 KB (page + next 16 + previous 15); `mlock` does not prevent flushes; `msync` | +| §2.3, Table 1 | ten mmap-based DBMSs and their years; MongoDB MMAPv1's removal; SingleStore's 10–20 ms per query on a shared mmap write lock | +| §3.1 | the OS may flush a dirty page at any time; the three update protocols (OS COW / user-space COW / shadow paging) and who uses each | +| §3.2 | no asynchronous reads; read-only queries can block on faults; `mlock`, `madvise` and prefetch threads as partial workarounds | +| §3.3 | checksums must be revalidated per access; corrupted pages persisted silently; SIGBUS mid-instruction | +| §3.4 + §4.1 | the three bottlenecks: page table contention, single-threaded `kswapd` eviction, TLB shootdowns at thousands of cycles | +| §4 preamble | EPYC 7713 (64c/128t), 512 GB RAM with 100 GB page cache, 10 × PM1733 SSDs at 7000 MB/s, Linux 5.11, fio 3.25 `O_DIRECT` | +| §4.1, Fig. 2 | ~900K reads/s for fio at 100 threads (≈100 µs NVMe latency); mmap's 27 s / ~5 s collapse / half-speed recovery; shootdown rate in Fig. 2b | +| §4.2, Figs. 3–4 | the single-SSD drop after ~17 s; ~20× gap on 10 SSDs; "2–20× worse than fio" | +| §5 | the authors' own endorsement of pointer swizzling as the right alternative | +| §6 | the prescription — when not to use mmap, and the two cases where you might | + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 6 — mmap page reads, p50 **42 ns**, + max **182 µs**; full lane output and the handicap note in + [`notes.md`](notes.md), lane source in + `experiments/src/bin/pool_vs_mmap.rs`. + +**Next** +- [`reading-leanstore-paper.md`](reading-leanstore-paper.md) — vmcache, the + design that keeps mmap's addressing and takes back residency control. +- [`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md) — what the + thousands of lines mmap promised to save actually look like. diff --git a/topics/06-buffer-pool/reading-postgres-bufmgr.md b/topics/06-buffer-pool/reading-postgres-bufmgr.md index 4ee3eef..a91df05 100644 --- a/topics/06-buffer-pool/reading-postgres-bufmgr.md +++ b/topics/06-buffer-pool/reading-postgres-bufmgr.md @@ -1,187 +1,463 @@ # postgres bufmgr: a buffer's life in one atomic word -Postgres packs everything CLOCK needs to know about a buffer — refcount, -usage count, flags — into a single atomic u64, so the hit path is one CAS and -the sweep hand reads victims without locks. This chapter builds the classic -shared-buffers design step by step — frames and pins, the packed state word, -the hit path, CLOCK, the foreground dirty-victim flush, scan admission, and -the background writer that exists to hide the flushes — then maps each step -to the exact lines in `bufmgr.c` and `freelist.c`. +Postgres packs everything the clock sweep needs to know about a buffer — +refcount, usage count, flags, and (as of this tree) the content lock itself — +into a single atomic 64-bit word, so the hit path is one CAS and the sweep +hand reads victims without ever locking a header. This chapter builds the +classic `shared_buffers` design step by step — frames and pins, the packed +state word, the hit path, the clock sweep and its arithmetic, the foreground +dirty-victim flush, scan admission, and the background writer that exists to +hide those flushes — then maps each step to the exact lines in `bufmgr.c` and +`freelist.c`. + +Everything below is read at the repo's pinned postgres commit, +[`postgres/postgres@701f021`](https://github.com/postgres/postgres), which is +`20devel` — a development tree, not a release. That matters twice: the buffer +**free list is gone** (only the clock sweep remains, in a file still named +`freelist.c`), and the per-buffer content lock has moved *into* the state +word. Descriptions written against PG ≤ 17 will disagree with the line +numbers here, and in those two places with the design. ## The problem in one sentence -With 128 backends hammering a shared cache of (say) 16 GB / 4 million -buffers, every page access must find, claim, and protect a buffer using at -most a couple of atomic instructions — any lock on the common path would -serialize the whole server. +With 128 backends hammering a shared cache of (say) 16 GB — 2,097,152 frames +of 8 KB — every page access must find, claim, and protect a frame using at +most a couple of atomic instructions, because any lock held on the common +path would serialize the whole server. ## The concepts, step by step ### Step 1 — frames, the mapping table, and pins -Postgres's buffer pool (the fixed-size in-memory cache of disk pages the -engine manages itself, `shared_buffers`) is a big array of fixed 8 KB -**frames** allocated at startup, plus a shared hash table mapping -`(relation, block number) → frame index`. Two operations define everything -else: +> **In:** nothing yet — this step fixes the two operations every later step +> is trying to make cheap. +> **Out:** the lookup (Step 3 shards it) and the pin (Step 2 packs it into +> one word, Step 4 reads it to skip victims). -- **Lookup**: hash the page's identity, probe the table. Found ⇒ hit; - not found ⇒ miss, and some frame must be recycled. +A **buffer pool** is the fixed-size in-memory cache of disk pages that the +engine manages itself; in postgres it is `shared_buffers`, a big array of +fixed 8 KB **frames** — RAM slots, each currently holding one **page**, the +unit the storage layer reads and writes — allocated once at startup. Beside +it sits a shared hash table mapping `(relation, fork, block number) → frame +index`. Two operations define everything else: + +- **Lookup**: hash the page's identity, probe the table. Found ⇒ *hit*; + not found ⇒ *miss*, and some frame must be recycled. - **Pin**: atomically increment the frame's reference count before touching - its bytes. A pinned frame (refcount > 0) is invisible to eviction — the - pin is the only thing standing between your pointer and the frame being - reused for a different page mid-read. Unpin when done. + its bytes. A pinned frame (refcount > 0) is invisible to eviction — the pin + is the only thing standing between your pointer and the frame being reused + for a different page mid-read. **Unpin** when done. (The database sense of + "pin" throughout; nothing to do with `mlock`.) + +A page whose in-RAM copy has been modified but not yet written back is +**dirty**. An **eviction policy** is whatever decides which unpinned frame to +recycle on a miss; Step 4 is postgres's. -Why it matters: on a hot workload these two operations run millions of -times per second across 100+ processes. Everything below is about making -them cost one or two atomics. +Why it matters: on a hot workload these two operations run millions of times +per second across 100+ processes. Everything below is about making them cost +one or two atomics. ### Step 2 — the packed state: one atomic u64 per buffer +> **In:** the pin from Step 1, which naively wants a lock. +> **Out:** a single `pg_atomic_uint64` that Steps 3, 4 and 5 all mutate with +> plain CAS loops instead of taking the header spinlock. + Instead of separate fields guarded by a spinlock, postgres packs a buffer's -entire hot-path state — refcount, usage count, and flag bits (dirty, valid, -locked) — into ONE atomic u64 (`BufferDesc.state`): +entire hot-path state into ONE atomic 64-bit word, `BufferDesc.state` +(buf_internals.h:344). The header comment spells out the division at +:34–42 — and note how little is left over: ``` - ┌──────────── 64-bit state ────────────┐ - │ lock bits │ flags │ usage(4) │ refcount(18) │ - └───────────────────────────────────────┘ - BUF_REFCOUNT_BITS 18 (:49) BUF_USAGECOUNT_BITS 4 (:50) - BM_MAX_USAGE_COUNT 5 (:144) — CLOCK survives ≤5 sweeps + ┌───────────────── 64-bit BufferDesc.state ─────────────────┐ + │ 1 excl │ 1 sh-excl │ 18 share-lock │ 12 flags │ 4 usage │ 18 refcount │ + └───────────────────────────────────────────────────────────┘ + bit 53 bit 21 bit 17 bit 0 + + BUF_REFCOUNT_BITS 18 (:49) BUF_USAGECOUNT_BITS 4 (:50) + BUF_FLAG_BITS 12 (:51) BUF_LOCK_BITS 18+2 (:52) + BM_MAX_USAGE_COUNT 5 (:144) — the sweep gives ≤5 second chances + + 18 + 4 + 12 + 20 = 54 bits used, 10 spare — and there is a + StaticAssertDecl at :54 that keeps the sum ≤ 64. ``` -- **refcount (18 bits)** — the pin count from Step 1; 18 bits because at - most MAX_BACKENDS processes can pin simultaneously (StaticAssert at :130). +- **refcount (18 bits)** — the pin count from Step 1. 18 bits because at most + `MAX_BACKENDS` processes can pin simultaneously, and the compiler is made + to check it: `StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS)` + at :130. - **usage count (4 bits, capped at 5)** — a tiny popularity score for - eviction (Step 4); it saturates harmlessly at 5. - -Why packed: pin/unpin/usage-bump become a single CAS (compare-and-swap — an -atomic "replace this word only if it still holds the value I read") — no -spinlock on the hit path. Same trick as topic-2's SwissTable metadata byte: -cram the hot-path-decidable state into one word. + eviction (Step 4). It saturates harmlessly; the comment at :136–143 + explains the tradeoff, and Step 4 does its arithmetic. +- **lock bits (20)** — in this tree the *content lock* lives here too, with + a `proclist_head lock_waiters` (:358) for the sleepers. Older postgres kept + a separate `LWLock content_lock` in the descriptor; it is gone. + +Why packed: pin, unpin, and usage-bump each become a single **CAS** +(compare-and-swap — an atomic "replace this word only if it still holds the +value I read"), so nothing on the hit path acquires the buffer header +spinlock. The struct comment says it outright at :272–275. Same trick as +topic 2's SwissTable metadata byte: cram the hot-path-decidable state into +one word. ### Step 3 — the hit path: sharded lookup, then one CAS -A hit costs one hash probe plus one CAS. The probe: `BufTableLookup` runs -under one of **`NUM_BUFFER_PARTITIONS = 128`** partition locks — the hash -table is sharded 128 ways so concurrent lookups almost never contend on the -same lock. The claim: `PinBuffer` runs a CAS loop on the state word — -refcount+1, and usage_count+1 if it's below the cap of 5 (:3338–3352). Two -memory operations total; nothing global is written. - -Cost to notice: still ~2 atomics + a probable cache miss on the hash bucket -per access — this is exactly the tax LeanStore's swizzling eliminates -(reading-leanstore-paper.md, Step 1). - -### Step 4 — CLOCK: eviction as a sweeping second-chance hand - -On a miss, some unpinned frame must be recycled — but maintaining a true -LRU list (move-to-front on every hit) would mean list surgery on the hot -path. **CLOCK** approximates LRU with the usage count: one shared atomic -counter, `nextVictimBuffer`, ticks around the frame array like a clock -hand. At each frame it inspects the state word: - -- pinned (refcount ≠ 0) ⇒ skip — invisible to CLOCK; -- usage_count > 0 ⇒ decrement it and move on — the buffer "spends a life"; -- both zero ⇒ victim. - -A hit bumps usage (max 5), so a frequently-used buffer survives up to 5 -full sweeps untouched. The sweep, distilled: - -```rust -// One shared clock hand; a buffer survives ≤5 sweeps untouched. -fn get_victim(&self) -> BufId { - loop { - let id = self.clock_tick(); // fetch_add(1) % NBuffers - let s = self.desc[id].state.load(); - if s.refcount() != 0 { continue; } // pinned: invisible to CLOCK - if s.usage_count() > 0 { // recently used: spend a life - let _ = self.desc[id].state.cas(s, s.dec_usage()); - continue; - } - if self.desc[id].state.cas(s, s.pinned()) { // both zero ⇒ victim; pin it - return id; // caller flushes it if dirty — - } // in the FOREGROUND - } -} -``` - -Why it matters: a hit costs a saturating increment; only misses pay the -sweep. That trade — no per-hit bookkeeping beyond one CAS — is why nobody -ships strict LRU (your `benches/eviction.rs` measures exactly this). +> **In:** the mapping table (Step 1) and the packed word (Step 2). +> **Out:** the measured per-hit tax — one partition lock, one probe, one CAS +> — which is exactly what Step 4 must not add to, and what LeanStore's +> swizzling deletes. + +A hit is a probe plus a CAS. `BufferAlloc` (bufmgr.c:2197) hashes the tag, +picks a partition lock, and takes it **shared**: + +```c +// postgres/postgres@701f021 — src/backend/storage/buffer/bufmgr.c, BufferAlloc, 2218-2240 + 2218 /* determine its hash code and partition lock ID */ + 2219 newHash = BufTableHashCode(&newTag); + 2220 newPartitionLock = BufMappingPartitionLock(newHash); + 2221 + 2222 /* see if the block is in the buffer pool already */ + 2223 LWLockAcquire(newPartitionLock, LW_SHARED); + 2224 existing_buf_id = BufTableLookup(&newTag, newHash); + 2225 if (existing_buf_id >= 0) + 2226 { + 2227 BufferDesc *buf; + 2228 bool valid; + 2235 buf = GetBufferDescriptor(existing_buf_id); + 2236 + 2237 valid = PinBuffer(buf, strategy, false); + 2238 + 2239 /* Can release the mapping lock as soon as we've pinned it */ + 2240 LWLockRelease(newPartitionLock); +``` -### Step 5 — the miss path: the dirty victim is YOUR problem +The mapping table is sharded `NUM_BUFFER_PARTITIONS = 128` ways +(lwlock.h:83), with `BufTableHashPartition` = `hashcode % 128` +(buf_internals.h:248–250), so concurrent lookups of different pages almost +never contend on the same lock. Line 2240 is the point of the packing: the +partition lock is released the instant the pin is in, so it is held for one +hash probe and one CAS, never across I/O. + +The pin itself, inside `PinBuffer` (:3295), is one CAS loop over the state +word — refcount+1, and usage_count+1 if below the cap: + +```c +// postgres/postgres@701f021 — src/backend/storage/buffer/bufmgr.c, PinBuffer's CAS loop, 3330-3352 + 3330 buf_state = old_buf_state; + 3331 + 3332 /* increase refcount */ + 3333 buf_state += BUF_REFCOUNT_ONE; + 3334 + 3335 if (strategy == NULL) + 3336 { + 3337 /* Default case: increase usagecount unless already max. */ + 3338 if (BUF_STATE_GET_USAGECOUNT(buf_state) < BM_MAX_USAGE_COUNT) + 3339 buf_state += BUF_USAGECOUNT_ONE; + 3340 } + 3341 else + 3342 { + 3343 /* + 3344 * Ring buffers shouldn't evict others from pool. Thus we + 3345 * don't make usagecount more than 1. + 3346 */ + 3347 if (BUF_STATE_GET_USAGECOUNT(buf_state) == 0) + 3348 buf_state += BUF_USAGECOUNT_ONE; + 3349 } + 3350 + 3351 if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, + 3352 buf_state)) +``` + +Lines 3341–3348 are Step 6 arriving early: a page pinned through a bulk-read +ring never gets a usage count above 1, so it cannot out-compete a real +working-set page in Step 4's sweep. + +Cost to notice: a hit still costs an LWLock acquire/release, a probable cache +miss on the hash bucket, and a CAS on a shared line. vmcache's Table 2 +measures the shape of this — a random page access through a hash table takes +**336 ns and 27.9 instructions** against 219 ns and 3.3 instructions for a +plain memory read, and that is for an *unsynchronized* table, i.e. a lower +bound. This is precisely the tax LeanStore's pointer swizzling deletes +([`reading-leanstore-paper.md`](reading-leanstore-paper.md), Step 1). + +### Step 4 — the clock sweep: eviction as a second-chance hand + +> **In:** the usage count and refcount from Step 2, which the sweep is the +> only consumer of. +> **Out:** a victim frame, pinned and owned by the caller — which Step 5 +> then discovers may be dirty. + +On a miss, some unpinned frame must be recycled. True **LRU** — a list with +move-to-front on every hit — would mean shared list surgery on the hot path, +which is the one thing Step 3 refuses. **Clock sweep** (also called *second +chance*) approximates it: a single shared counter, `nextVictimBuffer` +(freelist.c:42), ticks around the frame array like the hand of a clock. At +each frame it reads the state word and applies three rules: + +``` + refcount ≠ 0 ⇒ skip (pinned: invisible to the hand) + usage_count > 0 ⇒ decrement it (the buffer spends one of its lives) + both zero ⇒ VICTIM (pin it and return it) +``` + +That is the whole policy, and it is these lines: + +```c +// postgres/postgres@701f021 — src/backend/storage/buffer/freelist.c, StrategyGetBuffer's sweep, 239-246 + 239 /* Use the "clock sweep" algorithm to find a free buffer */ + 240 trycounter = NBuffers; + 241 for (;;) + 242 { + 243 uint64 old_buf_state; + 244 uint64 local_buf_state; + 245 + 246 buf = GetBufferDescriptor(ClockSweepTick()); +``` + +```c +// postgres/postgres@701f021 — src/backend/storage/buffer/freelist.c, the three rules, 263-303 + 263 if (BUF_STATE_GET_REFCOUNT(local_buf_state) != 0) + 264 { + 265 if (--trycounter == 0) + 266 { + 274 elog(ERROR, "no unpinned buffers available"); + 275 } + 276 break; + 277 } + 286 if (BUF_STATE_GET_USAGECOUNT(local_buf_state) != 0) + 287 { + 288 local_buf_state -= BUF_USAGECOUNT_ONE; + 289 + 290 if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, + 291 local_buf_state)) + 292 { + 293 trycounter = NBuffers; + 294 break; + 295 } + 296 } + 297 else + 298 { + 299 /* pin the buffer if the CAS succeeds */ + 300 local_buf_state += BUF_REFCOUNT_ONE; + 301 + 302 if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, + 303 local_buf_state)) +``` + +Note what is *not* there: no list, no lock, no per-buffer metadata beyond the +4 bits. `ClockSweepTick` (:110) is a `pg_atomic_fetch_add_u32` (:120) with a +CAS-based modular wraparound so the counter cannot overflow mid-flight while +`completePasses` (:48) is kept consistent with it. And line 274 is the only +failure mode: if the hand makes a whole lap (`trycounter` counts down from +`NBuffers`) finding every frame pinned, postgres errors out rather than spin +forever. + +**The arithmetic, part 1: does the policy even matter?** Take +`shared_buffers = 16 GB`, so `NBuffers = 16 GiB / 8 KiB = 2,097,152` frames. + +``` + working set 12 GB = 1,572,864 pages < pool + after warm-up every page is resident; over 100M accesses the only + misses are the compulsory first touches: + hit rate = 1 − 1,572,864/100,000,000 = 98.4%, → 100% as the run grows + the eviction policy is never consulted. LRU, CLOCK, random: identical. + + working set 32 GB = 4,194,304 pages > pool, accesses uniform + hit rate = pool / working set = 2,097,152 / 4,194,304 = 50% + and that is the answer for EVERY policy, including OPT: under uniform + access no ordering of victims is better than any other. +``` + +So the policy only earns its keep when the working set exceeds the pool *and* +the accesses are skewed — which is what makes LeanStore's §VI-B measurement +(random 92.5% vs LRU 93.1% vs OPT 96.3%, at Zipf 1.0) the interesting +comparison rather than a uniform one. + +**The arithmetic, part 2: what the sweep costs per miss.** In steady state +the hand must destroy usage counts as fast as hits create them. Per access, +hits create at most `h` increments (hit rate `h`); the hand creates one +decrement per frame it visits. If it visits `S` frames per miss and the miss +rate is `m = 1 − h`, then finding a victim needs `S·m ≥ h + m = 1`: + +``` + m = 50% ⇒ S ≥ 2 frames visited per miss + m = 5% ⇒ S ≥ 20 + m = 1% ⇒ S ≥ 100 + in every case S·m ≈ 1 frame visit per buffer ACCESS — a constant, + which is why the hand is affordable at all +``` + +The better the hit rate, the longer the hand must walk per miss — but the +total work per access stays at about one state-word read. And the cap on +usage explains itself: buf_internals.h:140 warns "it can take as many as +`BM_MAX_USAGE_COUNT`+1 complete cycles of the clock-sweep hand to find a free +buffer", which at 5 is `6 × 2,097,152 = 12.6M` frame visits in the worst case +— tolerable. A cap "comparable to NBuffers" would approximate true LRU +(:139) and make that worst case unbounded. -`BufferAlloc` (bufmgr.c:2197) looks up, misses, and calls `GetVictimBuffer` -(:2548), which runs Step 4's sweep. Now the ugly part: **if the victim is -dirty** (modified in RAM but not yet written to disk), *the backend that -wants a new page writes the old one out itself* — `FlushBuffer`, right -there in the foreground (:2584 onward). Your innocent `SELECT` eats a full -disk write before its read can even start: every dirty eviction is a -user-visible latency spike. +Why it matters: a hit costs one saturating increment inside a CAS it was +doing anyway; only misses pay the walk. That trade is why nobody ships strict +LRU — and your `benches/eviction.rs` lane measures exactly this. -Note the WAL-rule cameo: before flushing, `XLogNeedsFlush(BufferGetLSN(...))` -(~:2633) — a page may not be written until the log covering its changes is -durable (topic 5's invariant; the same one mmap can't enforce, -reading-mmap-paper.md Step 3). +### Step 5 — the miss path: the dirty victim is YOUR problem + +> **In:** Step 4's victim frame. +> **Out:** a user-visible latency spike whose size Step 7's background writer +> is built to hide — and the WAL rule that makes it worse than one write. + +`GetVictimBuffer` (bufmgr.c:2548) reserves pin bookkeeping +(`ReservePrivateRefCountEntry` :2559 — question 2's machinery), calls Step 4's +`StrategyGetBuffer` (:2569), and then hits the ugly part: **if the victim is +dirty, the backend that wants a new page writes the old one out itself**, +right there in the foreground, at `FlushBuffer` (:2634). Your innocent +`SELECT` eats a disk write before its read can even start. + +And it is not one write. Inside `FlushBuffer` (:4526) comes the WAL rule: + +```c +// postgres/postgres@701f021 — src/backend/storage/buffer/bufmgr.c, FlushBuffer's WAL rule, 4565-4585 + 4565 recptr = BufferGetLSN(buf); + 4566 + 4567 /* + 4568 * Force XLOG flush up to buffer's LSN. This implements the basic WAL + 4569 * rule that log updates must hit disk before any of the data-file changes + 4570 * they describe do. + 4584 if (pg_atomic_read_u64(&buf->state) & BM_PERMANENT) + 4585 XLogFlush(recptr); +``` -Also read how reads got faster: `PinBufferForBlock` (:1223) → -`ReadBuffer_common` (:1276) → `StartReadBuffersImpl` (:1371) — v17+ turned -the miss into a vectored/async `ReadBuffersOperation`; the miss path now -streams. +So a dirty eviction can cost a **log flush (an fsync) plus an 8 KB write**, +serially, on a query that only wanted to read. This is topic 5's invariant — +and the same one mmap cannot enforce, because the kernel may write a dirty +page out whenever it likes ([`reading-mmap-paper.md`](reading-mmap-paper.md), +Step 4). + +Put a number on the stall. This topic's own lane measures page-access +latency under mmap at **p50 42 ns and max 182 µs** +([FINDINGS.md row 6](../../FINDINGS.md), `notes.md` baseline) — a 4300× +spread caused by minor page faults the database cannot see. A foreground +dirty-victim flush is a stall of comparable magnitude or worse. The +difference is the entire argument for owning the buffer pool: postgres +*knows* it is about to pay it, can attribute it (`pg_stat_io`'s `IOOP_EVICT`, +counted at :2660), and can arrange for someone else to have paid it already +— which is Step 7. Under mmap the same-sized stall arrives unannounced. + +There is one more branch worth seeing, at :2624–2631: if the victim came from +a strategy ring and reusing it *would* require a WAL flush, +`StrategyRejectBuffer` can hand it back and the sweep starts over. Postgres +would rather evict a stranger than make a bulk scan wait on the log. + +Also read how the read side got faster: `PinBufferForBlock` (:1223) → +`ReadBuffer_common` (:1276) → `StartReadBuffersImpl` (:1371). Recent versions +turned the miss into a vectored/async `ReadBuffersOperation` (note +`PgAioWaitRef io_wref` in the descriptor, buf_internals.h:352); the miss path +now streams instead of blocking one block at a time. ### Step 6 — buffer rings: eviction policy as admission policy +> **In:** the sweep from Step 4, which a sequential scan would otherwise +> feed with the entire pool. +> **Out:** the second of the two places a system can defend itself — and the +> comparison question 3 asks you to settle. + One `SELECT count(*)` on a 100 GB table would, naively, march through the -pool evicting everything — a sequential scan touches each page once and -never again, the worst possible tenant. Postgres's defense is at -*admission*: `GetAccessStrategy` (freelist.c:426) gives bulk scans a private -**ring** of ~256 KB (BAS_BULKREAD, :442–459) — the scan recycles its own 32 -buffers instead of claiming fresh ones, so the other 4 million buffers never -see it. Compare LeanStore, which defends at *eviction* (unlucky pages get a -second chance in the cooling FIFO); question 3 below asks what each misses. +pool evicting everything: a sequential scan touches each page once and never +again, the worst possible tenant. Postgres's defence is not in the policy but +at *admission*. `GetAccessStrategy` (freelist.c:426) hands bulk operations a +private **ring** of buffers that they recycle among themselves: + +``` + BAS_BULKREAD 256 KB base (:459) = 32 frames of 8 KB + + BLCKSZ × io_combine_limit × effective_io_concurrency + for in-flight AIO (:480-481), capped by the pin limit + BAS_BULKWRITE 16 MB (:488) = 2,048 frames + BAS_VACUUM 2 MB (:491) = 256 frames + + a bulk read's blast radius, in a 16 GB pool: + 32 / 2,097,152 = 0.0015% of the frames +``` + +The scan reuses its own 32 frames instead of claiming fresh ones, so the +other two million never see it — and per Step 3's lines 3341–3348, even the +pages it does touch never accumulate a usage count above 1. + +Compare LeanStore, which defends at *eviction* instead: an unlucky page gets +a second chance in the cooling FIFO before it is thrown out +([`reading-leanstore-paper.md`](reading-leanstore-paper.md), Step 4). +Question 3 below asks what each approach misses. ### Step 7 — the background writer: hide the foreground flush -Step 5's foreground flush is the latency killer, so a dedicated process, -`BgBufferSync` (bufmgr.c:3854), runs the *same clock* slightly ahead of the -sweep hand, writing dirty buffers preemptively so that when -`GetVictimBuffer` arrives, victims are already clean. Its pace is an -estimate: `bgwriter_lru_maxpages` (default 100 pages/round) scaled by a -moving average of recent buffer-allocation rate (:3877–3911). It's an -*estimator* — read the long comment; when it guesses low, backends pay -Step 5's spike again. +> **In:** Step 5's foreground flush and Step 4's clock hand. +> **Out:** an estimator, its default ceiling in MB/s, and the conditions +> under which it fails and Step 5's spike comes back. + +Step 5's foreground flush is the latency killer, so a dedicated process runs +the *same clock* slightly ahead of the sweep hand, writing dirty buffers +preemptively so that when `GetVictimBuffer` arrives, the victims are already +clean. That is `BgBufferSync` (bufmgr.c:3854), and `StrategySyncStart` +(freelist.c:326–348) is how it asks where the hand currently is. + +Its pace is a guess, and worth reading as an example of a self-tuning +control loop that can be wrong: + +``` + smoothed_alloc fast-attack, slow-decline EMA of recent allocations, + smoothing_samples = 16 (:3876, :4021-4025) + upcoming_alloc_est = smoothed_alloc × bgwriter_lru_multiplier (2.0, :191) + hard cap bgwriter_lru_maxpages = 100 pages per round (:190) + round length BgWriterDelay = 200 ms (postmaster/bgwriter.c:59) + + default writeback ceiling: + 100 pages × 8 KB / 0.2 s = 800 KB / 0.2 s = 4 MB/s + = 500 dirty pages per second +``` + +Four megabytes a second. Dirty more than 500 pages/s in steady state and the +overflow lands on backends as Step 5's spike, no matter how well the +estimator tracks. (For scale, LeanStore's Fig. 9 reports ~500 MB/s written +back in the background while staying near in-memory throughput — 125× the +postgres default. The defaults are old; the mechanism is not the limit.) + +Why it matters: this is the shape of every "hide the cost" subsystem — an +estimator plus a cap. Read the long comment above :3877; when it guesses low, +or when the cap binds, the work does not disappear, it simply moves back onto +the query that triggered it. ## Where each step lives in the code +Read in this order: `buf_internals.h`'s header comment (:33–147) for the +state word, then `freelist.c` end-to-end (it is only 770 lines and contains +the entire policy), then the three `bufmgr.c` functions. + | File | What | Steps | |------|------|-------| -| `src/include/storage/buf_internals.h` | packed state word :49–147, `BufferDesc` :344, partitions :244–250 | 1–2 | -| `src/backend/storage/buffer/bufmgr.c` | pin, miss path, bgwriter | 3, 5, 7 | -| `src/backend/storage/buffer/freelist.c` | CLOCK + strategies/rings | 4, 6 | -| `src/include/storage/lwlock.h` | `NUM_BUFFER_PARTITIONS = 128` :83 | 3 | - -- **Step 2**: `BUF_REFCOUNT_BITS`/`BUF_USAGECOUNT_BITS` — - buf_internals.h:49–50; `BM_MAX_USAGE_COUNT` :144; the - refcount-vs-MAX_BACKENDS StaticAssert :130. -- **Step 3**: `PinBuffer` — bufmgr.c:3295 (CAS loop :3338–3352); - `BufTableLookup` under partition locks — buf_internals.h:244–250, - lwlock.h:83. Async reads: `PinBufferForBlock` :1223 → `ReadBuffer_common` - :1276 → `StartReadBuffersImpl` :1371. -- **Step 4**: `StrategyControl->nextVictimBuffer` — freelist.c:42; - `ClockSweepTick` — :104–160 (`fetch_add(1) % NBuffers` with a CAS-based - modular wraparound so the counter never overflows mid-flight); - `StrategyGetBuffer` — :184, the sweep loop :246–290, the - "no unpinned buffers available" error when everything's pinned ~:274. -- **Step 5**: `BufferAlloc` — bufmgr.c:2197 (lookup :2224); - `GetVictimBuffer` — :2548 (foreground `FlushBuffer` :2584 onward; WAL - check `XLogNeedsFlush` ~:2633; `ReservePrivateRefCountEntry` :2559 — - question 2's resource-owner machinery). -- **Step 6**: `GetAccessStrategy` — freelist.c:426; BAS_BULKREAD ring - sizing :442–459. -- **Step 7**: `BgBufferSync` — bufmgr.c:3854; pacing - `bgwriter_lru_maxpages` :190 + the moving-average estimator :3877–3911 - (read the long comment). +| `src/include/storage/buf_internals.h` | state-word layout :33–147, `BufferDesc` :326 (`state` :344), partition hashing :244–258 | 1–2 | +| `src/backend/storage/buffer/freelist.c` | the whole replacement policy: clock sweep + rings. **No free list** despite the name | 4, 6, 7 | +| `src/backend/storage/buffer/bufmgr.c` | pin, miss path, flush, bgwriter | 3, 5, 7 | +| `src/include/storage/lwlock.h` | `NUM_BUFFER_PARTITIONS 128` :83 | 3 | + +| Step | Symbol | Location | +|---|---|---| +| 2 | `BUF_REFCOUNT_BITS` 18 / `BUF_USAGECOUNT_BITS` 4 / `BUF_FLAG_BITS` 12 / `BUF_LOCK_BITS` 20 | buf_internals.h:49–52 | +| 2 | size assertion; refcount-vs-`MAX_BACKENDS` assertion | buf_internals.h:54, :130 | +| 2 | `BM_MAX_USAGE_COUNT 5` and the tradeoff comment | buf_internals.h:136–147 | +| 2 | `BufferDesc` with `state`, `io_wref`, `lock_waiters` | buf_internals.h:326–359 | +| 3 | `BufferAlloc` — hash, partition lock, lookup, pin, release | bufmgr.c:2197, :2218–2240 | +| 3 | `PinBuffer` and its CAS loop (ring cap at :3347) | bufmgr.c:3295, :3330–3352 | +| 3 | `BufTableHashPartition` = `hashcode % NUM_BUFFER_PARTITIONS` | buf_internals.h:248–250, lwlock.h:83 | +| 3 | async reads: `PinBufferForBlock` → `ReadBuffer_common` → `StartReadBuffersImpl` | bufmgr.c:1223, :1276, :1371 | +| 4 | `nextVictimBuffer`, `completePasses` | freelist.c:42, :48 | +| 4 | `ClockSweepTick` — `fetch_add(1)` plus CAS wraparound | freelist.c:110–166 (add at :120) | +| 4 | `StrategyGetBuffer`; the sweep; `"no unpinned buffers available"` | freelist.c:184, :239–316, :274 | +| 5 | `GetVictimBuffer`; `ReservePrivateRefCountEntry`; dirty check | bufmgr.c:2548, :2559, :2584 | +| 5 | ring rejection when a WAL flush would be needed | bufmgr.c:2624–2631 | +| 5 | foreground `FlushBuffer` call, then the function | bufmgr.c:2634, :4526 | +| 5 | the WAL rule: `XLogFlush(recptr)` | bufmgr.c:4565–4585 | +| 6 | `GetAccessStrategy`; ring sizes 256 KB / 16 MB / 2 MB | freelist.c:426, :459, :488, :491 | +| 7 | `BgBufferSync`; `StrategySyncStart` | bufmgr.c:3854; freelist.c:326–348 | +| 7 | `bgwriter_lru_maxpages` 100, `bgwriter_lru_multiplier` 2.0, `smoothing_samples` 16 | bufmgr.c:190, :191, :3876 | +| 7 | the EMA itself | bufmgr.c:4021–4028 | ## Questions to answer in notes.md @@ -195,16 +471,186 @@ Step 5's spike again. 4. Postgres double-buffers (shared_buffers + OS page cache). What does `O_DIRECT` (topic 6's io story, debug_io_direct) buy and cost here? +## Takeaway + +The hit path is one shared LWLock acquire on a 1-in-128 shard plus one CAS on +a word that holds refcount, usage, flags and the content lock together. The +policy that word feeds — clock sweep with 5 second chances — costs about one +state-word read per access no matter what the hit rate is, and matters at all +only when the working set exceeds the pool *and* the accesses are skewed. +Everything expensive has been pushed onto the miss, where the real hazard is +not the read but the dirty victim: a WAL fsync plus an 8 KB write, in the +foreground, unless a background writer capped at 4 MB/s got there first. + ## Done when -You can narrate a miss on a dirty victim end-to-end — every lock, atomic, -and I/O in order — and say which step BgBufferSync moves off the hot path. +Answer each before unfolding it. + +- [ ] You can narrate a miss on a dirty victim end-to-end — every lock, atomic and I/O in order — and say which of them `BgBufferSync` moves off the hot path. + +
Answer + + `BufferAlloc` (:2197) hashes the tag, takes the partition LWLock shared + (:2223), probes (`BufTableLookup` :2224), and misses. It calls + `GetVictimBuffer` (:2548), which reserves a refcount entry and resource + owner slot (:2559–2560) and enters `StrategyGetBuffer` (freelist.c:184). + The clock hand (`ClockSweepTick` :110, an atomic fetch-add) walks frames, + CAS-decrementing usage counts (:288–291) and skipping pinned ones (:263), + until it finds refcount 0 and usage 0 and CAS-pins it (:300–303) — no lock + taken anywhere in the walk. + + Back in `GetVictimBuffer`, `buf_state & BM_DIRTY` (:2584) is true. It takes + the content lock conditionally (:2603 — conditionally, to avoid deadlock + with a concurrent page split), then calls `FlushBuffer` (:2634), which + first does `XLogFlush(recptr)` (:4585) — a **WAL fsync** — and only then + `smgrwrite`s the 8 KB page. Then the mapping table is updated under the + *exclusive* partition lock, the buffer is pinned into its new identity, and + the read is issued (`StartReadBuffersImpl` :1371). + + `BgBufferSync` (:3854) removes exactly one of these: the `XLogFlush` + + `smgrwrite` pair, by having written the page already so `BM_DIRTY` is clear + when the hand arrives. It removes neither the sweep, nor the partition + locks, nor the read. + +
+ +- [ ] You can compute the hit rate for a 16 GB pool against a 12 GB and a 32 GB uniform working set, and say what the eviction policy is worth in each case. + +
Answer + + 16 GB / 8 KB = **2,097,152 frames**. A 12 GB working set is 1,572,864 + pages, which fits, so after warm-up the hit rate tends to 100% (over a + 100M-access run, `1 − 1,572,864/100,000,000` = 98.4%) and the policy is + never consulted — every page is resident and nothing is ever evicted. + + A 32 GB working set is 4,194,304 pages, twice the pool. Under *uniform* + access the hit rate is `pool / working set` = **50%** for every policy, + including the unimplementable optimum: with no skew there is no information + to exploit, so choosing victims well is choosing among equals. + + The policy therefore earns its keep only in the third case — working set + larger than the pool *and* skewed. That is why LeanStore evaluates at + Zipf 1.0 rather than uniform (§VI-B: random 92.5%, LRU 93.1%, OPT 96.3%), + and why the honest summary of clock sweep is not "it approximates LRU" but + "it approximates LRU in the only regime where either one matters". + +
+ +- [ ] You can explain why a *higher* hit rate makes the clock hand walk *further* per miss, and why that does not make the sweep more expensive overall. + +
Answer + + Conservation of usage counts. Each hit adds at most one increment (capped + at 5); each frame the hand visits removes at most one. In steady state the + two must balance, and a victim additionally has to be reached: with hit + rate `h`, miss rate `m` and `S` frames visited per miss, `S·m ≥ h + m = 1`, + so `S ≥ 1/m`. At `m = 50%` the hand walks 2 frames per miss; at `m = 5%`, + 20; at `m = 1%`, 100. + + Overall cost is `S·m ≈ 1` state-word read **per access**, independent of + the hit rate — the walk grows exactly as fast as misses become rare. What + the cap bounds is the worst case, not the average: buf_internals.h:140 says + a victim can take `BM_MAX_USAGE_COUNT + 1` = 6 complete cycles, which is + 12.6M frame visits in a 16 GB pool. A cap "comparable to NBuffers" (:139) + would approximate true LRU and make that unbounded, which is the tradeoff + the comment describes. + +
+ +- [ ] You can say what a bulk sequential scan can and cannot do to the pool, and name the two independent mechanisms that limit it. + +
Answer + + It can consume at most its ring: `BAS_BULKREAD` starts at 256 KB + (freelist.c:459) = 32 frames of 8 KB, grown only by + `BLCKSZ × io_combine_limit × effective_io_concurrency` to keep in-flight + AIO from stalling on its own pins (:480–481), and capped by the pin limit. + In a 16 GB pool that is `32 / 2,097,152` = 0.0015% of the frames — the scan + recycles its own buffers instead of claiming fresh ones. + + Two mechanisms, not one. (1) Admission: `GetAccessStrategy` (:426) hands + out the ring at all, and `GetBufferFromRing` (:198) is consulted before the + clock sweep. (2) Usage suppression: `PinBuffer` caps a strategy-pinned + buffer's usage count at 1 (bufmgr.c:3341–3348) with the comment "Ring + buffers shouldn't evict others from pool", so even the pages a scan does + touch cannot out-survive working-set pages in the sweep. A third, smaller + one: if reusing a ring buffer would force a WAL flush, + `StrategyRejectBuffer` gives it back and takes a stranger instead (:2624). + +
+ +- [ ] You can compute the background writer's default writeback ceiling and say what happens above it. + +
Answer + + `bgwriter_lru_maxpages = 100` pages per round (bufmgr.c:190) and + `BgWriterDelay = 200` ms (postmaster/bgwriter.c:59), so the ceiling is + `100 × 8 KB / 0.2 s` = **4 MB/s**, or 500 dirty pages per second. Below + that, the estimator — a fast-attack/slow-decline EMA over + `smoothing_samples = 16` (:3876, :4021–4025) scaled by + `bgwriter_lru_multiplier = 2.0` (:191) — decides how much of the ceiling to + use. + + Above it, the work does not disappear: every dirty page beyond 500/s is + still dirty when the clock hand reaches it, so a backend pays Step 5's + `XLogFlush` + write in the foreground. The estimator can also simply guess + low after a workload shift, with the same result. For scale, LeanStore's + Fig. 9 sustains ~500 MB/s of background writeback while staying near + in-memory throughput — 125× the postgres default, which says the ceiling is + a conservative default rather than a property of the design. + +
+ +- [ ] You wrote answers to all four questions in notes.md. + +
Answer + + Nothing to unfold — the answers are the exercise. Two of them have hints + in the text: question 1's asymmetry is settled by the two `StaticAssertDecl` + lines (buf_internals.h:130 and :146), and question 3's contrast is Step 6's + last paragraph. Question 2 wants you to follow `ReservePrivateRefCountEntry` + into `resowner.h`; question 4 wants `debug_io_direct` and an argument about + where the second copy of every page is currently living. + +
## References -**Code** -- [postgres/postgres](https://github.com/postgres/postgres) — - `src/backend/storage/buffer/bufmgr.c`, - `src/backend/storage/buffer/freelist.c`, - `src/include/storage/buf_internals.h`, - `src/include/storage/lwlock.h`. Local clone at `~/repos/postgres`. +**Code** — [postgres/postgres](https://github.com/postgres/postgres) at +`701f021` (`20devel`). Local clone at `~/repos/postgres`; the pin table is at +the end of `resources/codebases.md`. + +| File | Lines | What | +|---|---|---| +| `src/include/storage/buf_internals.h` | 33–52 | the state word's division, in the header comment | +| `src/include/storage/buf_internals.h` | 130, 146 | the two static assertions that keep the caps honest | +| `src/include/storage/buf_internals.h` | 136–144 | why `BM_MAX_USAGE_COUNT` is 5 and not larger | +| `src/include/storage/buf_internals.h` | 248–258 | `BufTableHashPartition`, `BufMappingPartitionLock` | +| `src/include/storage/buf_internals.h` | 326–359 | `BufferDesc`: tag, `state`, `io_wref`, `lock_waiters` | +| `src/include/storage/lwlock.h` | 77–96 | `NUM_BUFFER_PARTITIONS 128` and its offset in the LWLock array | +| `src/backend/storage/buffer/freelist.c` | 42–48 | `nextVictimBuffer`, `completePasses` | +| `src/backend/storage/buffer/freelist.c` | 110–166 | `ClockSweepTick` | +| `src/backend/storage/buffer/freelist.c` | 184–316 | `StrategyGetBuffer`: ring first, then the sweep | +| `src/backend/storage/buffer/freelist.c` | 426–500 | `GetAccessStrategy` and the three ring sizes | +| `src/backend/storage/buffer/bufmgr.c` | 190–191 | `bgwriter_lru_maxpages`, `bgwriter_lru_multiplier` | +| `src/backend/storage/buffer/bufmgr.c` | 2197–2245 | `BufferAlloc`'s hit path | +| `src/backend/storage/buffer/bufmgr.c` | 2548–2660 | `GetVictimBuffer`, including the foreground flush | +| `src/backend/storage/buffer/bufmgr.c` | 3295–3360 | `PinBuffer` | +| `src/backend/storage/buffer/bufmgr.c` | 3854–4130 | `BgBufferSync` and its estimator | +| `src/backend/storage/buffer/bufmgr.c` | 4526–4600 | `FlushBuffer` and the WAL rule | +| `src/backend/postmaster/bgwriter.c` | 59 | `BgWriterDelay = 200` ms | + +**Measurements cited** +- [FINDINGS.md row 6](../../FINDINGS.md) and this topic's `notes.md` — mmap + page reads p50 42 ns, max 182 µs on this machine; the scale of an + unscheduled stall. +- vmcache (SIGMOD '23) Table 2 — 336 ns / 27.9 instructions for a hash-table + page access vs 219 ns / 3.3 for a plain read. +- LeanStore (ICDE '18) §VI-B and Fig. 9 — the hit-rate comparison at Zipf + 1.0, and ~500 MB/s of background writeback. + +**Next** +- [`reading-leanstore-paper.md`](reading-leanstore-paper.md) — the design + that deletes Step 3 entirely. +- [`reading-mmap-paper.md`](reading-mmap-paper.md) — what happens when you + let the kernel make Steps 4–7's decisions for you. diff --git a/topics/06-buffer-pool/reading-redis-zmalloc.md b/topics/06-buffer-pool/reading-redis-zmalloc.md index 87306bd..d3dbc90 100644 --- a/topics/06-buffer-pool/reading-redis-zmalloc.md +++ b/topics/06-buffer-pool/reading-redis-zmalloc.md @@ -1,13 +1,18 @@ # zmalloc: memory management when there are no pages Redis has no buffer pool — no pages, no frames, no eviction hand. What it has -instead is an allocation *ledger*: every malloc accounted on per-thread -padded counters, `maxmemory` enforced against an allocator statistic, and -key-level eviction after the fact. This chapter builds that ledger step by -step — why accounting replaces caching, how you learn a pointer's size, why -the counters are padded, how eviction hangs off a statistic, and what -fragmentation forces redis to do — plus a bonus: turso's CLOCK page cache in -Rust, the closest existing code to your experiment. +instead is an allocation *ledger*: every malloc accounted on per-thread padded +counters, `maxmemory` enforced against that ledger, and key-level eviction +after the fact. This chapter builds the ledger step by step — why accounting +replaces caching, how you learn a pointer's size, why the counters are padded +(and to *what*, which this repo has measured), how eviction hangs off a +statistic, and what fragmentation forces redis to do — plus a bonus: turso's +Rust page cache, the closest existing code to your experiment. + +Read at [`redis/redis@a176d1225`](https://github.com/redis/redis) and +[`tursodatabase/turso@dd775bc`](https://github.com/tursodatabase/turso), the +repo's pinned commits (pin table at the end of `resources/codebases.md`; local +clones at `~/repos/redis` and `~/repos/turso`). ## The problem in one sentence @@ -20,135 +25,443 @@ kernel's OOM killer does it for them. ### Step 1 — no pages: accounting instead of caching -A buffer pool exists to decide which disk pages live in RAM; redis keeps -*everything* in RAM, so there is nothing to cache and nothing to page. Its -memory-management problem is different: track how much the process has -allocated (the ledger), compare against a budget (`maxmemory`), and shed -load (evict whole keys) when over. `zmalloc` is a thin wrapper around the -allocator (jemalloc in practice) whose whole job is to maintain that ledger -on every allocate and free. - -Why it matters: the mechanisms look superficially like topic 6's — a limit, -an eviction policy — but the *unit* is a variable-size key-value pair, not -a fixed page, and the trigger is an allocator statistic, not a miss. +> **In:** the vocabulary of this topic — buffer pool, page, frame, eviction. +> **Out:** which of those words still mean anything when the dataset is +> already in RAM, which is the frame Steps 2–5 are built in. + +Three definitions, since this guide is where they stop applying. A **buffer +pool** is a fixed region of RAM holding copies of disk **pages** (fixed-size +blocks, 8 KB in postgres) in **frames** (the slots that hold them); an +**eviction policy** picks which resident page to drop when a new one must +come in. All three assume the data has a home on disk and RAM is the scarce +copy of it. + +Redis keeps *everything* in RAM. There is no disk copy to fall back to, so +there is nothing to cache and nothing to page. Its memory problem is a +different one: track how much the process has allocated (the ledger), compare +against a budget (`maxmemory`), and shed load — evict whole keys, losing the +data — when over. `zmalloc` is a thin wrapper around the allocator (jemalloc +in practice) whose entire job is to maintain that ledger on every allocate and +free. + +Why it matters: the mechanisms *look* like a buffer pool's — a limit, an +eviction policy, a sampling scan — but the unit is a variable-size key-value +pair rather than a fixed page, and the trigger is an allocator statistic +rather than a miss. Everything that follows is downstream of that one +substitution. ### Step 2 — the prerequisite: how big is this pointer? -To subtract on `free(p)`, the ledger must answer "how many bytes was `p`?" — -and plain libc historically had no portable way to ask. Two paths -(`PREFIX_SIZE`, zmalloc.c:39–46): +> **In:** Step 1's ledger, which must add on alloc and *subtract* on free. +> **Out:** a size for any pointer — and a per-allocation overhead that is +> either 0 or 8 bytes, which Step 5's fragmentation story then builds on. + +To subtract on `free(p)` the ledger must answer "how many bytes was `p`?", +and portable libc has no way to ask. Redis compiles one of two answers: + +```c +// redis/redis@a176d1225 — src/zmalloc.c, PREFIX_SIZE, 39-48 + 39 #ifdef HAVE_MALLOC_SIZE + 40 #define PREFIX_SIZE (0) + 41 #else + 42 /* Use at least 8 bytes alignment on all systems. */ + 43 #if SIZE_MAX < 0xffffffffffffffffull + 44 #define PREFIX_SIZE 8 + 45 #else + 46 #define PREFIX_SIZE (sizeof(size_t)) + 47 #endif + 48 #endif +``` -- With jemalloc (`HAVE_MALLOC_SIZE`), the allocator itself reports any - pointer's *usable size* (`malloc_usable_size`) ⇒ prefix is 0 bytes. -- Without it, redis prepends an 8-byte size header to every allocation and - reads it back on free — 8 bytes of overhead on *every* allocation, which - for a store full of 40-byte keys is real money. +- **With `HAVE_MALLOC_SIZE`** (jemalloc, and macOS's libc): the allocator + itself reports any pointer's *usable size*, so the prefix is 0 bytes. The + alloc path calls `zmalloc_size(ptr)` and hands the answer straight to the + ledger (zmalloc.c:184–185). +- **Without it**: redis prepends an 8-byte header holding the size to every + allocation, returns `ptr + PREFIX_SIZE` to the caller (:190–193), and reads + the header back on free (:499–503, :521–523). Eight bytes on *every* + allocation — for a store full of 40-byte keys that is a 20% tax on the data + itself. + +Note *which* size the ledger records: the **usable** size, what the allocator +actually reserved including bucket rounding, not what you asked for. Ask +jemalloc for 100 bytes, get a 112-byte bin, and the ledger records 112. That +gap, times millions of allocations, is Step 5. + +### Step 3 — per-thread padded counters: a ledger without coherence traffic + +> **In:** Step 2's byte count, produced on every alloc and free. +> **Out:** a total that Step 4 can read, at a per-allocation cost small +> enough to be invisible — the arithmetic for "small enough" is the point of +> this step. + +One global `used_memory` updated with an atomic `fetch_add` would put the +same cache line in play on every malloc on every core. The line ping-pongs +between cores, and this repo has measured what that costs: topic 9 found +padding "independent" counters apart to be worth **17.8×** +([FINDINGS.md row 9](../../FINDINGS.md)). Redis pays none of it — one counter +struct per thread, each padded to its own cache line: + +```c +// redis/redis@a176d1225 — src/zmalloc.c, the per-thread ledger, 82-96 + 82 #define MAX_THREADS 16 /* Keep it a power of 2 so we can use '&' instead of '%'. */ + 83 #define THREAD_MASK (MAX_THREADS - 1) + 84 #define PEAK_CHECK_THRESHOLD (1024 * 100) /* 100KB */ + 85 + 86 typedef struct used_memory_entry { + 87 redisAtomic long long used_memory; + 88 redisAtomic long long last_peak_check; + 89 char padding[CACHE_LINE_SIZE - sizeof(long long) - sizeof(long long)]; + 90 } used_memory_entry; + 91 + 92 static __attribute__((aligned(CACHE_LINE_SIZE))) used_memory_entry used_memory[MAX_THREADS]; + 93 static redisAtomic size_t num_active_threads = 0; + 94 static redisAtomic size_t zmalloc_peak = 0; + 95 static redisAtomic time_t zmalloc_peak_time = 0; + 96 static __thread long my_thread_index = -1; +``` + +Read line 89 and 92 together: explicit tail padding *and* alignment, so no +two threads' counters can share a line. And `CACHE_LINE_SIZE` is not 64 +everywhere — + +```c +// redis/redis@a176d1225 — src/config.h, CACHE_LINE_SIZE, 38-44 + 38 #ifndef CACHE_LINE_SIZE + 39 #if defined(__aarch64__) && defined(__APPLE__) + 40 #define CACHE_LINE_SIZE 128 + 41 #else + 42 #define CACHE_LINE_SIZE 64 + 43 #endif + 44 #endif +``` -Note that the ledger counts *usable* size (what the allocator actually -reserved, including bucket rounding), not requested size — ask for 100 -bytes, jemalloc hands you a 112-byte bin, the ledger records 112. That gap, -multiplied by millions of allocations, is fragmentation (Step 5). +— which is the same conclusion this repo reached from the other direction: +topic 9's lane found that on M-series, 64-byte padding only *half*-fixes the +sharing and 128 is what actually works ([FINDINGS.md row 9](../../FINDINGS.md)). +Redis's `#ifdef` and our measurement agree, and neither knew about the other. + +The total is computed on read, by summing the live counters: + +```c +// redis/redis@a176d1225 — src/zmalloc.c, zmalloc_used_memory, 567-580 + 567 size_t zmalloc_used_memory(void) { + 568 size_t local_num_active_threads; + 569 long long total_mem = 0; + 570 atomicGet(num_active_threads,local_num_active_threads); + 571 if (local_num_active_threads > MAX_THREADS) { + 572 local_num_active_threads = MAX_THREADS; + 573 } + 574 for (size_t i = 0; i < local_num_active_threads; ++i) { + 575 long long thread_used_mem; + 576 atomicGet(used_memory[i].used_memory, thread_used_mem); + 577 total_mem += thread_used_mem; + 578 } + 579 return total_mem; + 580 } +``` -### Step 3 — per-thread padded counters: the ledger without coherence traffic +The alloc path itself needs that sum, to maintain the peak — so the sum is +*throttled* by the second counter in the struct. `update_zmalloc_stat_alloc` +always bumps the thread's own counter (:111), but only runs the cross-thread +sum once the thread has allocated `PEAK_CHECK_THRESHOLD` more bytes than at +its last check (:116), then records the new watermark (:143). Frees skip the +check entirely — `update_zmalloc_stat_free` (:147–150) is a bare local +decrement. -A single global `used_memory` counter updated with an atomic `fetch_add` -would put the same cache line in play on every malloc on every core — the -line ping-pongs between cores at ~100 cycles a bounce (topic 0's false -sharing), taxing every allocation in the process. Redis instead keeps one -counter *per thread*, each padded out to its own cache line -(`aligned(CACHE_LINE_SIZE)`, a MAX_THREADS array — zmalloc.c:86–92): each -thread bumps its own line, uncontended, and the true total is computed by -summing all counters *when someone reads it* — and reads are rare. +**What the throttle buys, as arithmetic**: -```rust -// One counter per thread, each on its own cache line — a single global -// fetch_add would put coherence traffic on EVERY malloc on EVERY core. -#[repr(align(64))] -struct Padded(AtomicI64); -static USED: [Padded; MAX_THREADS] = /* … */; - -fn zmalloc(size: usize) -> *mut u8 { - let p = unsafe { malloc(size) }; - let real = malloc_usable_size(p); // jemalloc answers "how big is p?" - USED[thread_id()].0.fetch_add(real as i64, Relaxed); // uncontended bump - p -} -fn used_memory() -> i64 { - USED.iter().map(|c| c.0.load(Relaxed)).sum() // the SUM is paid on read, -} // and reads are rare +``` + threshold 100 KB of new allocation per thread (:84) + average redis allocation ~64 B (small key + value + object header) + ⇒ one global sum every 100 KB / 64 B ≈ 1,600 allocations + cost of one global sum ≤ MAX_THREADS = 16 atomic loads + ⇒ amortized 16 / 1,600 = 0.01 loads per malloc + + without the throttle: 16 loads on every malloc, of lines other cores + are actively writing — i.e. the exact coherence storm the padding + was there to avoid, reintroduced by the reader. + + the same allocation on the shared-counter design: + uncontended L1 hit ~1 ns (FINDINGS row 0's latency ladder) + contended line, 8 cores the 17.8× that topic 9 measured + jemalloc tcache malloc tens of ns + ⇒ accounting would cost more than the allocation it accounts for. ``` -Even the occasional full sum is throttled: `update_zmalloc_stat_alloc` -(:105–145) bumps the local counter always, but pays for the cross-thread -sum only occasionally (the peak-check throttle, :109–118). +One sharp edge worth seeing: thread indices are assigned once and masked with +`& THREAD_MASK` (:101), so the 17th thread *shares* a counter — and a cache +line — with the 1st. `MAX_THREADS = 16` is a hard cap, and past it the false +sharing quietly comes back. ### Step 4 — maxmemory: eviction hangs off a statistic -The ledger's sum is what `maxmemory` compares against. When -`used_memory() > maxmemory`, redis evicts — but the unit of eviction is a -**key** (a complete value: a whole hash, a whole list), chosen by -approximate LRU or LFU over sampled keys, freed *after* the limit is -already breached. Compare the two accounting philosophies in this topic: -DuckDB gates every allocation up front (reserve-or-throw, before malloc); -redis counts after and evicts asynchronously. A key can't be partially -resident — there's no analogue of "page out half the value" — which is -question 2 below. +> **In:** Step 3's `zmalloc_used_memory()` sum. +> **Out:** a decision to free whole keys — sampled, approximate, and taken +> *after* the limit is already breached. + +`getMaxmemoryState` reads the ledger and works out the shortfall: + +```c +// redis/redis@a176d1225 — src/evict.c, getMaxmemoryState, 384-419 + 384 int getMaxmemoryState(size_t *total, size_t *logical, size_t *tofree, float *level) { + 385 size_t mem_reported, mem_used, mem_tofree; + 389 mem_reported = zmalloc_used_memory(); + 397 if (mem_reported <= server.maxmemory && !level) return C_OK; + 398 + 399 /* Remove the size of slaves output buffers and AOF buffer from the + 400 * count of used memory. */ + 401 mem_used = mem_reported; + 402 size_t overhead = freeMemoryGetNotCountedMemory(); + 403 mem_used = (mem_used > overhead) ? mem_used-overhead : 0; + 411 if (mem_used <= server.maxmemory) return C_OK; + 413 /* Compute how much memory we need to free. */ + 414 mem_tofree = mem_used - server.maxmemory; + 417 if (tofree) *tofree = mem_tofree; + 419 return C_ERR; + 420 } +``` -### Step 5 — active defrag: moving memory that the allocator can't +Lines 399–403 are the honest part: replica output buffers and the AOF buffer +are *not* the dataset, so they are subtracted before the comparison. The +ledger measures the process; the policy is about the data. + +Then `performEvictions` (:532) frees `mem_tofree` bytes' worth of **keys** — +complete values, a whole hash, a whole list. Two things about that policy are +easy to get wrong: + +- **The default is not to evict.** `maxmemory-policy` defaults to + `MAXMEMORY_NO_EVICTION` (config.c:3192): over the limit, writes get an + error and nothing is discarded. Eviction is opt-in, because for many redis + deployments silently losing data is worse than failing a write. +- **The LRU is sampled, not tracked.** With an eviction policy set, each pass + samples `maxmemory-samples` keys (default **5**, config.c:3223) via + `evictionPoolPopulate` (evict.c:134, called at :602) and merges them into a + 16-entry pool (`EVPOOL_SIZE`, evict.c:36) that persists across calls; the + loop at :621 then walks the pool from best to worst. There is no LRU list — + keeping one would mean a list write on every access, which is precisely the + hot-path cost Step 3 spent all that padding to avoid. + +**How good is 5 samples?** For a key uniformly in the coldest fraction *f*, +the chance that at least one of *k* samples lands there is `1 − (1−f)^k`: + +``` + f = 10% coldest, k = 5 1 − 0.9^5 = 41% + f = 20% coldest, k = 5 1 − 0.8^5 = 67% + f = 20% coldest, k = 10 1 − 0.8^10 = 89% (maxmemory-samples 10) +``` + +A single pass is a poor approximation of LRU. What rescues it is the +16-entry pool surviving across passes, plus the fact that evicting *many* +keys means many independent draws: the probability that a given hot key is +picked before some colder one, repeatedly, falls off fast. Approximation is +affordable here for the same reason it is in DuckDB's queue of hints +([`reading-duckdb-buffer.md`](reading-duckdb-buffer.md)) — being wrong costs +one extra miss, and being exact costs a write on every access. + +Compare the three accounting philosophies now on the table: + +| System | When the budget is checked | What happens at the limit | +|---|---|---| +| postgres | never — the frame array *is* the budget | a new page evicts an old one; error only if all frames are pinned | +| DuckDB | before every allocation | evict, then throw `OutOfMemoryException` if that was not enough | +| redis | after the fact, from a statistic | error (default), or sample keys and evict whole values | + +### Step 5 — active defrag: moving memory that the allocator cannot + +> **In:** Step 2's usable-size rounding and a long-running write workload. +> **Out:** why the ledger and the OS disagree about how much memory redis is +> using, and the one mechanism that can close the gap. **Fragmentation** here means: jemalloc serves allocations from size-class -bins backed by pages; free objects leave holes, and a bin holding 3 live -objects out of 128 slots still pins its pages — RSS (what the OS charges -you) stays high while `used_memory` (live bytes) is low. A normal allocator -can't fix this: it handed out raw pointers and may never move the memory -they target. - -Redis fixes it *cooperatively*: `activeDefragAlloc` (defrag.c:177, and the -:142 comment) asks jemalloc which allocations sit in sparse bins, -re-allocates each one (new pointer, same bytes), frees the old, and — the -expensive part — **rewrites every reference** to it, which redis can do -because it owns all the data structures that hold the pointers. -Defragmentation in userspace, because the allocator can't move memory it -handed out. +bins backed by whole pages; freed objects leave holes, and a bin with 3 live +objects in 128 slots still pins all its pages. RSS — what the OS charges you, +and what the OOM killer looks at — stays high while `used_memory` (live +bytes) is low. A normal allocator cannot fix this: it handed out raw +pointers and may never move what they point at. + +Redis measures the gap explicitly, and *only* over the bins it could actually +do something about: + +```c +// redis/redis@a176d1225 — src/defrag.c, getAllocatorFragmentation, 1226-1234 + 1226 /* Calculate the fragmentation ratio as the proportion of wasted memory in small + 1227 * bins (which are defraggable) relative to the total allocated memory (including large bins). + 1228 * This is because otherwise, if most of the memory usage is large bins, we may show high percentage, + 1229 * despite the fact it's not a lot of memory for the user. */ + 1230 float frag_pct = (float)frag_smallbins_bytes / allocated * 100; + 1231 float rss_pct = ((float)resident / allocated)*100 - 100; + 1232 size_t rss_bytes = resident - allocated; + 1233 if(out_frag_bytes) + 1234 *out_frag_bytes = frag_smallbins_bytes; +``` + +It then fixes it *cooperatively*, one allocation at a time: + +```c +// redis/redis@a176d1225 — src/defrag.c, activeDefragAllocWithoutFree, 142-166 + 142 /* this method was added to jemalloc in order to help us understand which + 143 * pointers are worthwhile moving and which aren't */ + 144 int je_get_defrag_hint(void* ptr); + 151 void* activeDefragAllocWithoutFree(void *ptr) { + 152 size_t size; + 153 void *newptr; + 154 if(!je_get_defrag_hint(ptr)) { + 155 server.stat_active_defrag_misses++; + 156 return NULL; + 157 } + 158 /* move this allocation to a new allocation. + 159 * make sure not to use the thread cache. so that we don't get back the same + 160 * pointers we try to free */ + 161 size = zmalloc_usable_size(ptr); + 162 newptr = zmalloc_no_tcache(size); + 163 memcpy(newptr, ptr, size); + 164 server.stat_active_defrag_hits++; + 165 return newptr; + 166 } +``` + +Line 144 is a function jemalloc grew *for redis*: "is this pointer sitting in +a sparse run worth moving?" Line 162 is the subtle one — allocate bypassing +the thread cache, or you get handed back the very pointer you are trying to +vacate. The caller (`activeDefragAlloc`, :177) frees the old pointer, and +then — the expensive part this code does not show — every *reference* to it +must be rewritten, which redis can do only because it owns every data +structure holding those pointers. Defragmentation in userspace, because the +allocator cannot move memory it handed out. + +**When it runs, as arithmetic.** Two gates and a linear ramp +(`computeDefragCycles`, defrag.c:1369–1388, defaults from config.c:3215–3218 +and :3289): + +``` + gate 1 frag_pct ≥ active-defrag-threshold-lower = 10% + gate 2 frag_bytes ≥ active-defrag-ignore-bytes = 100 MB + ramp cpu_pct = INTERPOLATE(frag_pct, 10, 100, 1, 25) (:1380) + + a 4 GB instance at 15% fragmentation: + frag_bytes ≈ 0.15 × 4 GB = 600 MB ≥ 100 MB ✓ both gates + cpu_pct = 1 + (15−10)/(100−10) × (25−1) = 2.3 → 2% of CPU + + a 500 MB instance at the same 15%: + frag_bytes ≈ 75 MB < 100 MB ✗ gate 2 + → no defrag at all: 75 MB is not worth any CPU + + at 100% fragmentation the ramp saturates at 25% of one core — + a quarter of redis's single-threaded budget spent moving bytes + that are already in RAM. +``` FalkorDB angle: GraphBLAS matrices are big opaque zmalloc blobs — redis can -count them but not defrag them (their internal pointers are GraphBLAS's, -not redis's), and one matrix can blow the maxmemory budget in a single GrB -call. Your capstone owns its allocations; decide what "maxmemory" should -even mean for a graph store. - -### Step 6 — bonus: turso's CLOCK page cache, the Rust reference - -Back in buffer-pool land: turso (topic 1's B-tree) carries a real Rust -CLOCK implementation — the closest existing code to your -`src/buffer_pool.rs` experiment. Compare with yours *after* you build it -(don't copy first): - -- `PageCache` — page_cache.rs:99–116: an intrusive circular list with a - `clock_hand` raw pointer (:107); the comment at :95–98 states the - discipline (insert behind the hand). -- `advance_clock_hand` — :174; `insert` — :204. -- Note what's unsafe (`Send`/`Sync` impls :115–116, raw pointers) and what - your Rust version can do differently with indices into a `Vec` - instead of pointers (safe, and the array is exactly postgres's layout). +count them but cannot defrag them (their internal pointers are GraphBLAS's, +not redis's), and one matrix can blow the `maxmemory` budget inside a single +`GrB` call, between two checks of the ledger. Your capstone owns its +allocations; decide what "maxmemory" should even mean for a graph store. -## Where each step lives in the code +### Step 6 — bonus: turso's page cache, the Rust reference -Local clones at `~/repos/redis` and `~/repos/turso`: +> **In:** everything above, and the buffer-pool vocabulary from Step 1 that +> redis had no use for. +> **Out:** a working Rust implementation to diff your capstone against — +> after you have built yours. + +Back in buffer-pool land: turso (topic 1's B-tree) carries a real Rust page +cache, the closest existing code to your `src/buffer_pool.rs` experiment. +Read its header comment before anything else, because it is not the plain +CLOCK the name suggests: + +```rust +// tursodatabase/turso@dd775bc — core/storage/page_cache.rs, PageCache, 90-116 + 90 /// PageCache implements a variation of the SIEVE algorithm that maintains an intrusive linked list queue of + 91 /// pages which keep a 'reference_bit' to determine how recently/frequently the page has been accessed. + 92 /// The bit is set to `Clear` on initial insertion and then bumped on each access and decremented + 93 /// during eviction scans. + 94 /// + 95 /// The ring is circular. `clock_hand` points at the tail (LRU). + 96 /// Sweep order follows next: tail (LRU) -> head (MRU) -> .. -> tail + 97 /// New pages are inserted after the clock hand in the `next` direction, + 98 /// which places them at head (MRU) (i.e. `tail.next` is the head). + 99 pub struct PageCache { + 100 /// Capacity in pages + 101 capacity: usize, + 102 /// Map of Key -> pointer to entry in the queue + 103 map: HashMap, + 104 /// The eviction queue (intrusive doubly-linked list) + 105 queue: LinkedList, + 106 /// Clock hand cursor for SIEVE eviction (pointer to an entry in the queue, or null) + 107 clock_hand: *mut PageCacheEntry, + 111 /// Conservative estimation of pages that are evictable based on dirty/spilled state. + 112 evictable_count: usize, + 113 } + 114 + 115 unsafe impl Send for PageCache {} + 116 unsafe impl Sync for PageCache {} +``` + +Three things to take from it: + +1. **New pages arrive cold.** `PageCacheEntry::new` sets `ref_bit: CLEAR` + (:58) and `_insert` splices the entry in *after* the hand (:308–311). In + classic **clock sweep** — a circular scan that gives each frame a *second + chance* before evicting it — a new page usually arrives with its reference + bit *set*, so it survives one full revolution. SIEVE's point is that most + new pages are one-hit wonders: make them prove themselves instead. +2. **The "bit" is a counter, exactly like postgres's.** `REF_MAX = 3` (:34); + `bump_ref` saturates at it on every `get` (:65, :412) and `decrement_ref` + walks it down on the sweep (:684). Postgres's usage count is the same idea + with a cap of 5 + ([`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md)). +3. **The sweep is bounded, for the same reason postgres's is.** + `max_examinations = len × (REF_MAX + 1)` (:637) — enough revolutions to + drive any counter to zero, and then `CacheError::Full` rather than an + infinite loop. `evictable()` (:620–626) is the pin check: not dirty (or + already spilled), not locked, not pinned, not the header page, and + `Arc::strong_count == 1` — Rust's refcount standing in for an explicit + **pin** (a "someone is using this, don't evict" marker). + +Note what is `unsafe` — the `Send`/`Sync` impls at :115–116, the raw +`*mut PageCacheEntry` in the map, `cursor_mut_from_ptr` throughout — and note +that your version does not have to be. Indices into a `Vec` are safe, +and give you exactly postgres's layout for free. Diff your design against +this one *after* you have built it; do not copy it first. + +## Where each step lives in the code | File | What | Steps | |------|------|-------| -| `redis/src/zmalloc.c` | the ledger | 2–4 | -| `redis/src/defrag.c` | cooperative defrag | 5 | -| `turso/core/storage/page_cache.rs` | Rust CLOCK | 6 | - -- **Step 2**: `PREFIX_SIZE` — zmalloc.c:39–46; `zmalloc` — :161–193 - (`malloc_usable_size` path vs prefix path). -- **Step 3**: `used_memory` per-thread padded counters — :86–92; - `update_zmalloc_stat_alloc` + peak-check throttle — :105–145. -- **Step 5**: `activeDefragAlloc` — defrag.c:177 (+ the :142 comment). -- **Step 6**: `PageCache` — page_cache.rs:99–116; `advance_clock_hand` - :174; `insert` :204. +| `redis/src/zmalloc.c` | the ledger | 2–3 | +| `redis/src/config.h` | the padding width | 3 | +| `redis/src/evict.c` | `maxmemory` and key eviction | 4 | +| `redis/src/defrag.c` | cooperative userspace defrag | 5 | +| `turso/core/storage/page_cache.rs` | a real Rust SIEVE/clock cache | 6 | + +| Step | Symbol | Location | +|---|---|---| +| 2 | `PREFIX_SIZE` — the two builds | zmalloc.c:39–48 | +| 2 | alloc path: usable-size branch vs prefix branch | zmalloc.c:170–195 (:184, :190) | +| 2 | `zmalloc_size` reads the header back; `zfree` | zmalloc.c:499–503, :509–526 | +| 3 | `MAX_THREADS`, `THREAD_MASK`, `PEAK_CHECK_THRESHOLD` | zmalloc.c:82–84 | +| 3 | `used_memory_entry` — two counters plus padding | zmalloc.c:86–92 | +| 3 | `CACHE_LINE_SIZE` = 128 on Apple aarch64, else 64 | config.h:38–44 | +| 3 | thread index assignment and the `& THREAD_MASK` wrap | zmalloc.c:96–103 | +| 3 | `update_zmalloc_stat_alloc` — local bump, throttled sum | zmalloc.c:105–145 (:111, :116, :143) | +| 3 | `update_zmalloc_stat_free` — no check at all | zmalloc.c:147–150 | +| 3 | `zmalloc_used_memory` — the sum over active threads | zmalloc.c:567–580 | +| 4 | `getMaxmemoryState` — read ledger, subtract non-data | evict.c:384–419 (:389, :402, :414) | +| 4 | `performEvictions` | evict.c:532 | +| 4 | `EVPOOL_SIZE` = 16, `evictionPoolPopulate`, the pool walk | evict.c:36, :134, :602, :621 | +| 4 | `maxmemory-policy` default, `maxmemory-samples` default 5 | config.c:3192, :3223 | +| 5 | `getAllocatorFragmentation` — small bins only | defrag.c:1213–1238 (:1230) | +| 5 | `je_get_defrag_hint`, `activeDefragAllocWithoutFree` | defrag.c:142–166 (:154, :162) | +| 5 | `activeDefragAlloc` — move then free | defrag.c:177–182 | +| 5 | `computeDefragCycles` — two gates and the ramp | defrag.c:1369–1388 (:1374, :1380) | +| 5 | defrag defaults: 1/25% CPU, 10/100% thresholds, 100 MB floor | config.c:3215–3218, :3289 | +| 6 | `PageCache` and the insertion discipline | page_cache.rs:90–116 | +| 6 | `CLEAR`, `REF_MAX`, `bump_ref`, `decrement_ref` | page_cache.rs:33–34, :65, :70–72 | +| 6 | `get` bumps the counter | page_cache.rs:395–413 (:412) | +| 6 | `_insert` splices after the hand | page_cache.rs:253–327 (:301, :308–311) | +| 6 | `advance_clock_hand`, `make_room_for` | page_cache.rs:174, :604 | +| 6 | `evictable`, `evict_one` — the bounded sweep | page_cache.rs:620–626, :629–695 (:637, :654, :684) | ## Questions to answer in notes.md @@ -160,18 +473,192 @@ Local clones at `~/repos/redis` and `~/repos/turso`: 3. After building your CLOCK pool: diff your design against turso's — hand placement on insert, where usage bits live, pin representation. +## Takeaway + +Take pages away and a buffer pool becomes a ledger. The hot path has to be +free, so the counter is sharded per thread and padded to a cache line — 128 +bytes on Apple silicon, which is exactly what this repo's topic-9 lane had to +measure the hard way. Reading the total is expensive, so it is throttled to +roughly once per 1,600 allocations. Knowing the total exactly is impossible +anyway, so eviction samples 5 keys and keeps a 16-entry pool. And because the +allocator will not move memory it handed out, redis moves it itself, gated on +a fragmentation percentage and a floor of 100 MB. Every one of those is the +same trade in a different costume: pay approximately and often, or exactly +and rarely. + ## Done when -You can explain PREFIX_SIZE, why the counters are padded, and what active -defrag can't touch — and you've compared your finished pool to turso's. +Answer each before unfolding it. + +- [ ] You can explain what `PREFIX_SIZE` is for, when it is 0, and what it costs when it is not. + +
Answer + + The ledger has to *subtract* on free, so it must recover an allocation's + size from a bare pointer. `PREFIX_SIZE` (zmalloc.c:39–48) is how: + + - With `HAVE_MALLOC_SIZE` — jemalloc, or macOS libc — the allocator answers + the question itself, so `PREFIX_SIZE` is **0** and the alloc path just + calls `zmalloc_size(ptr)` (:184). + - Without it, redis prepends an 8-byte size header, hands the caller + `ptr + PREFIX_SIZE` (:190–193), and reads the header back in + `zmalloc_size` (:499–503) and `zfree` (:521–523). + + The cost is 8 bytes on **every** allocation, not per key — a 40-byte key + pays 20%. That is a large part of why a jemalloc build is the recommended + one, quite apart from jemalloc's own behaviour. + +
+ +- [ ] You can say why the counters are padded, to what width, and what happens past 16 threads. + +
Answer + + Padded so that no two threads' counters share a cache line: a shared line + written by several cores ping-pongs between them, and topic 9 measured + padding as worth **17.8×** ([FINDINGS.md row 9](../../FINDINGS.md)). The + struct carries explicit tail padding (zmalloc.c:89) *and* the array is + aligned (:92). + + To `CACHE_LINE_SIZE`, which redis defines as **128 on Apple aarch64** and 64 + elsewhere (config.h:38–44) — the same platform split topic 9's lane found + empirically, where 64 only half-fixed the sharing on M-series. + + Past 16 threads: `MAX_THREADS` is 16 and thread indices wrap with + `& THREAD_MASK` (:101), so thread 17 shares both counter and cache line + with thread 1. The false sharing returns, silently, and the ledger stays + correct only because both threads' updates are atomic. + +
+ +- [ ] You can compute how often the expensive global sum actually runs, and why that matters. + +
Answer + + `update_zmalloc_stat_alloc` bumps the thread-local counter unconditionally + (:111) but calls `zmalloc_used_memory()` only when this thread's counter + has advanced past `PEAK_CHECK_THRESHOLD` = 100 KB since its last check + (:84, :116), after which it records the new watermark (:143). Frees never + check at all (:147–150). + + At a ~64-byte average allocation that is one sum per ≈1,600 allocations. + The sum itself is at most 16 atomic loads (:574–578), so amortized it is + **0.01 loads per malloc**. + + It matters because the sum reads *every other thread's* counter — the + exact lines the padding exists to keep private. An unthrottled reader would + reintroduce, from the read side, the coherence storm the write side was + designed to avoid. + +
+ +- [ ] You can state what redis does by default when `maxmemory` is exceeded, and how good its LRU actually is. + +
Answer + + By default it **does not evict**: `maxmemory-policy` defaults to + `MAXMEMORY_NO_EVICTION` (config.c:3192), so writes fail with an error and + no data is lost. Eviction is an opt-in choice. + + With a policy set, the LRU is sampled, not tracked. Each pass draws + `maxmemory-samples` keys (default 5, config.c:3223) in + `evictionPoolPopulate` (evict.c:134) and merges them into a 16-entry pool + (`EVPOOL_SIZE`, evict.c:36) that persists across passes; the loop at :621 + takes the best entry. A single draw of 5 finds a key from the coldest 10% + only `1 − 0.9^5 = 41%` of the time, and from the coldest 20% `67%` of the + time. The persistent pool and the sheer number of evictions are what make + that acceptable — and the reason to accept it is that a true LRU list + would need a list write on every access, on the hot path Step 3 worked so + hard to keep clean. + +
+ +- [ ] You can explain why an ordinary allocator cannot defragment, what redis does instead, and when it bothers. + +
Answer + + An allocator cannot move an allocation because it gave out a raw pointer + and has no idea who holds copies of it. Redis can, because it owns every + structure that stores those pointers: `activeDefragAllocWithoutFree` + (defrag.c:151) asks `je_get_defrag_hint` (:144, a jemalloc entry point + added for redis) whether a pointer sits in a sparse run, copies it to a + fresh allocation taken *outside the thread cache* (:162 — otherwise + jemalloc hands back the pointer you are vacating), and the caller frees + the old one (:177–182) after every reference has been rewritten. + + It bothers only when both gates in `computeDefragCycles` (:1374) pass: + fragmentation ≥ 10% **and** wasted bytes ≥ 100 MB. CPU is then interpolated + linearly from 1% at 10% fragmentation to 25% at 100% (:1380). So a 4 GB + instance at 15% fragmentation (≈600 MB wasted) spends about 2% of a core; + a 500 MB instance at the same 15% (≈75 MB) spends nothing, because 75 MB is + not worth any CPU at all. + +
+ +- [ ] You can name three ways turso's cache differs from a textbook clock sweep, and what your Rust version can do differently. + +
Answer + + It is a SIEVE variation (page_cache.rs:90–93), and: + + 1. **New pages start cold.** `ref_bit: CLEAR` on insert (:58), spliced in + after the hand (:308–311). Textbook clock inserts with the bit *set*, so + a new page survives a full revolution; SIEVE makes it earn that. + 2. **The bit is a saturating counter**, `REF_MAX = 3` (:34), bumped on + every `get` (:412, :65) and decremented on the sweep (:684) — postgres's + usage count with a smaller cap. + 3. **The sweep is explicitly bounded**: `len × (REF_MAX + 1)` examinations + (:637), then `CacheError::Full` instead of spinning — and `evictable()` + (:620–626) treats `Arc::strong_count == 1` as the pin check rather than + keeping a separate pin count. + + Yours can be safe. Turso needs `unsafe impl Send`/`Sync` (:115–116) and raw + `*mut PageCacheEntry` because the intrusive list holds self-references; a + `Vec` with `u32` indices for the hand and the links has none of that + and gives you postgres's array layout for free. + +
+ +- [ ] You wrote answers to all three questions in notes.md. + +
Answer + + Nothing to unfold. Question 2 is the one worth care: the comparison is + unfair because a buffer pool can keep *part* of a value resident — one page + of a large row — while redis can only keep a key whole or not at all, so + the same RAM buys a different kind of hit. Say which unit your capstone + uses before you compare its hit rate to anything. + +
## References -**Code** -- [redis](https://github.com/redis/redis) — `src/zmalloc.c` (the ledger) - and `src/defrag.c` (cooperative userspace defragmentation). Local clone - at `~/repos/redis`. -- [tursodatabase/turso](https://github.com/tursodatabase/turso) — - `core/storage/page_cache.rs`, a real Rust CLOCK to diff against your - experiment *after* you build it (don't copy first). Local clone at - `~/repos/turso`. +**Code** — [redis/redis](https://github.com/redis/redis) at `a176d1225`, +[tursodatabase/turso](https://github.com/tursodatabase/turso) at `dd775bc`. +Local clones at `~/repos/redis` and `~/repos/turso`; the pin table is at the +end of `resources/codebases.md`. + +| File | Lines | What | +|---|---|---| +| `redis/src/zmalloc.c` | 39–48 | `PREFIX_SIZE`: the two ways to size a pointer | +| `redis/src/zmalloc.c` | 82–103 | thread cap, threshold, the padded counter array | +| `redis/src/zmalloc.c` | 105–150 | the throttled alloc stat, and the bare free stat | +| `redis/src/zmalloc.c` | 170–195 | the alloc path's three compile-time branches | +| `redis/src/zmalloc.c` | 495–526 | `zmalloc_size` and `zfree` | +| `redis/src/zmalloc.c` | 567–580 | `zmalloc_used_memory` | +| `redis/src/config.h` | 38–44 | 128-byte lines on Apple aarch64 | +| `redis/src/evict.c` | 36, 134, 384–419, 532, 602–621 | the pool, the state check, the eviction loop | +| `redis/src/config.c` | 3192, 3223, 3215–3218, 3289 | eviction and defrag defaults | +| `redis/src/defrag.c` | 142–182 | the jemalloc hint and the cooperative move | +| `redis/src/defrag.c` | 1213–1238, 1369–1388 | measuring fragmentation, and the CPU ramp | +| `turso/core/storage/page_cache.rs` | 33–34, 58–72 | `CLEAR`, `REF_MAX`, the counter | +| `turso/core/storage/page_cache.rs` | 90–116 | the SIEVE comment and the struct | +| `turso/core/storage/page_cache.rs` | 174, 253–327 | hand movement and insert-after-hand | +| `turso/core/storage/page_cache.rs` | 604–695 | `make_room_for`, `evictable`, `evict_one` | + +**Related** +- [`reading-duckdb-buffer.md`](reading-duckdb-buffer.md) — the other + approximate policy in this topic, and the opposite budget discipline. +- [`reading-postgres-bufmgr.md`](reading-postgres-bufmgr.md) — the usage + count turso's `ref_bit` is a smaller copy of. +- [FINDINGS.md row 9](../../FINDINGS.md) — the 17.8× that justifies line 89. diff --git a/topics/07-networking-protocols/README.md b/topics/07-networking-protocols/README.md index 725da42..0fd4f9b 100644 --- a/topics/07-networking-protocols/README.md +++ b/topics/07-networking-protocols/README.md @@ -88,8 +88,10 @@ The two non-obvious moves: client (`handleClientsWithPendingWrites`). Batching by loop iteration. - **Pipelining falls out for free** — the input buffer may hold 100 commands; `processInputBuffer` loops until the buffer is drained, and all 100 replies - coalesce into one write. This is why `redis-benchmark -P 64` is ~10× -P 1: - same work, 1/64th the syscalls. + coalesce into one write. This is why `redis-benchmark -P 64` is **66.2×** + `-P 1` in this topic's own lane (44,088 → 2,919,728 ops/s): same work, + 1/64th the syscalls. Note it is not 64× — it overshoots, because the win is + not only the syscall count but everything the batch amortises around it. ## 3. Three threading models, one question: what's serialized? diff --git a/topics/07-networking-protocols/reading-bolt-packstream.md b/topics/07-networking-protocols/reading-bolt-packstream.md index 5f47495..332358b 100644 --- a/topics/07-networking-protocols/reading-bolt-packstream.md +++ b/topics/07-networking-protocols/reading-bolt-packstream.md @@ -1,216 +1,861 @@ # Bolt & PackStream: the graph in the type system RESP encodes a node as nested arrays the client must re-interpret; Bolt puts -Node, Relationship, and Path on the wire as first-class types, and makes -result streaming client-driven — backpressure IS the protocol. The reference -implementation here is FalkorDB's own Bolt 5.x server, complete until #2170 -removed it (2026-07-08): read it frozen in time with -`git show 0b11a00b3^:src/bolt/` in `~/repos/FalkorDB`. This chapter -builds the protocol step by step — the serialization format, the graph -types, the message vocabulary, the pull-based streaming — before walking the -C files. +Node, Relationship, and Path on the wire as first-class types, and — on paper — +makes result streaming client-driven, so backpressure IS the protocol. The +reference implementation here is FalkorDB's own Bolt 5.x server, complete until +#2170 removed it on 2026-07-08. This chapter builds the protocol step by step — +the serialization format, the graph types, the message vocabulary, the +pull-based streaming — and then walks the C files to find out how much of that +design FalkorDB's server actually implemented. The answer is the most useful +thing in the chapter, and it is not the answer the design promises. + +Every code anchor below is FalkorDB at commit **`40780e992`** — that is +`0b11a00b3^`, the tree one commit before PR #2170 deleted `src/bolt/`. The +repo's pin table records FalkorDB at `ccb449a9a` (`resources/codebases.md`), +where these files no longer exist, so read them at the older commit: + +```sh +tools/pinned-source.py --ref 40780e992ecc11f598ce3f4f65e04367f9abae2f \ + show FalkorDB src/bolt/bolt.c -r 133:151 +# or, in a clone: git show 0b11a00b3^:src/bolt/bolt.c +``` + +Every protocol claim is checked against Neo4j's Bolt specification +(), cited by page and section. ## The problem in one sentence A graph query returns nodes, relationships, and paths — typed, structured -values — but RESP can only say "array of arrays of strings", so every -FalkorDB client library re-parses nested arrays into graph objects by -convention; and if the result is 10M rows, RESP's server must buffer all of -them, because the client has no way to say "give me 1,000 at a time." +values — but RESP can only say "array of arrays of strings", so every FalkorDB +client library re-parses nested arrays into graph objects by convention; and if +the result is 10M rows, RESP's server must buffer all of them, because the +client has no way to say "give me 1,000 at a time." ## The concepts, step by step ### Step 1 — the two problems Bolt exists to solve -Bolt is Neo4j's binary protocol, and it differs from RESP in exactly the -two places the problem statement names. First, **typing**: the wire format -(PackStream, Steps 3–4) has markers for maps, lists, and *graph types* — -Node, Relationship, Path — so a driver hands you a graph object, not a -string table to re-interpret. Second, **streaming**: after a query runs, -records flow only when the client asks for them (`PULL {n}`, Step 5) — -backpressure designed in, not bolted on (topic 7 §4's problem, solved at -the protocol layer). +> **In:** nothing yet — this step fixes the two axes every later step is +> measured against. +> **Out:** the two design goals (typing, client-driven streaming) that Steps +> 2–5 build, and that Step 6 audits the implementation against. + +A **wire protocol** is the agreement between a client and a server about what +bytes mean: how a receiver finds where one message ends, what messages are +legal, and how values are spelled. Bolt is Neo4j's binary wire protocol, and it +differs from RESP in exactly the two places the problem statement names. + +First, **typing** — whether the wire format carries what a value *is*, or only +what it *looks like*. RESP has five kinds of value (simple string, error, +integer, bulk string, array), so a node arrives as an array of arrays and the +client library rebuilds the object by convention. Bolt's serialization format, +**PackStream** (Steps 3–4), has markers for maps, lists, and *graph types* — +Node, Relationship, Path — so a driver hands you a graph object. + +Second, **streaming**: whether the server may push a result as fast as it can +produce it, or must wait to be asked. **Backpressure** is a receiver's ability +to make a sender slow down; without it the only options are buffer (spend +memory) or die (drop the client). Bolt's answer is a protocol-level **cursor** — +a server-side position in a result the client advances explicitly — driven by +the `PULL {n}` message (Step 5). Why it matters: these are the two axes of the RESP/pgwire/Bolt table in -topic 7 §5 — Bolt is what a protocol looks like when the *data model* lives -in the protocol. +[topic 7 §5](README.md#5-bolt-the-third-answer-resp-vs-pgwire-vs-bolt). Bolt is +what a protocol looks like when the *data model* lives in the protocol — and +Step 6 is what happens when an engine adopts the format but not the cursor. ### Step 2 — the handshake: version negotiation in 20 bytes -A Bolt connection opens with the client sending 4 magic bytes -`0x60 0x60 0xB0 0x17` (so a server can tell Bolt from a stray HTTP request -on byte 1) plus four *proposed* protocol versions, 4 bytes each; the server -answers with the one version it picks, and every byte after that is spoken -in it. FalkorDB's implementation accepts 5.1..5.7 (bolt_api.c:803, version -pick :845–864, clamped to that range). Compare RESP, where versioning is an -optional in-band `HELLO 2|3` command — question 3 asks which design a proxy -can transparently downgrade. +> **In:** a fresh TCP connection, no bytes exchanged. +> **Out:** an agreed protocol version, after exactly 20 bytes from the client +> and 4 from the server — and the first place FalkorDB's implementation is +> narrower than the spec. + +A **handshake** is a fixed exchange that happens before any protocol message, +used to agree on which protocol will be spoken. Bolt's is deliberately tiny and +is itself unversioned (spec, *Handshake*). + +The client sends 4 identification bytes `60 60 B0 17` (spec, *Handshake* — +"the identification consists of the following four bytes"), which lets a server +tell Bolt from a stray HTTP request on byte 1. FalkorDB checks them with one +comparison: + +```c +// src/bolt/bolt_client.c — bolt_check_handshake, 672-680 + 672 // validate bolt handshake + 673 bool bolt_check_handshake + 674 ( + 675 bolt_client_t *client // the client + 676 ) { + 677 ASSERT(client != NULL); + 678 + 679 return ntohl(buffer_read_uint32(&client->read_buf.read)) == 0x6060B017; + 680 } +``` + +Line 679 is the whole check: read four bytes big-endian (**big-endian** = most +significant byte first, which PackStream uses exclusively — spec, *PackStream* +§ *Endianness*) and compare against the magic constant. + +Then the version proposals. The spec is precise: "the client submits exactly +four protocol versions, each encoded as a big-endian 32-bit unsigned integer for +a total of 128 bits" (*Handshake* § *Version negotiation*), and "a server should +assume that the versions … have been sent in order of preference. Therefore, if +a match occurs for more than one version, the first match should be selected." +The arithmetic of the chapter title: + +``` +client → server: 4 magic bytes + 4 versions × 4 bytes = 4 + 16 = 20 bytes +server → client: 1 chosen version = 4 bytes +``` + +FalkorDB reads all 16 bytes and then looks at two of them: + +```c +// src/bolt/bolt_client.c — bolt_read_supported_version, 682-695 + 682 // return the latest supported bolt version + 683 bolt_version_t bolt_read_supported_version + 684 ( + 685 bolt_client_t *client // the client + 686 ) { + 687 ASSERT(client != NULL); + 688 + 689 char data[16]; + 690 buffer_index_read(&client->read_buf.read, data, 16); + 691 bolt_version_t version; + 692 version.minor = data[2]; + 693 version.major = data[3]; + 694 return version; + 695 } +``` + +Lines 692–693 are the ones to look at: `data[2]` and `data[3]` are the low two +bytes of the *first* 4-byte proposal. The other three proposals are read into +the buffer and never examined. The spec's "first match" rule is implemented as +"first proposal or nothing". + +The decision itself is in the handshake handler: + +```c +// src/bolt/bolt_api.c — inside BoltHandshakeHandler, 845-866 + 845 bolt_version_t version = bolt_read_supported_version(client); + 846 if(version.major == (uint)-1 || version.major == 255) { + 847 version.major = 5; + 848 version.minor = 7; + 849 } + 850 if(version.major != 5 || version.minor < 1) { + 851 RedisModule_EventLoopDel(fd, REDISMODULE_EVENTLOOP_READABLE); + 852 raxRemove(clients, (unsigned char *)&client->socket, sizeof(client->socket), NULL); + 853 bolt_client_free(client); + 854 return; + 855 } + // ... 857-859: point `write` at the start of the write buffer ... + 860 if(client->ws) { + 861 buffer_write_uint16(&write, htons(0x8204)); + 862 } + 863 buffer_write_uint16(&write, 0x0000); + 864 buffer_write_uint8(&write, MIN(version.minor, 7)); + 865 buffer_write_uint8(&write, version.major); + 866 buffer_socket_write(&start, &write, client->socket); +``` + +The two lines that carry the argument are 850 and 864, and they say different +things. **Line 850 is the acceptance test**: major must be 5 and minor at least +1 — with *no upper bound*. **Line 864 is the answer**: `MIN(version.minor, 7)`. +So a client proposing 5.9 is not rejected; it is accepted and told "5.7", and +the connection then speaks 5.7. (An earlier version of this chapter said the +server "accepts 5.1..5.7, clamped to that range". The accepted range is 5.1 and +up; only the *reply* is clamped.) + +Line 846 is a second surprise: a first-proposal high byte of 255 is rewritten to +5.7. `FF` in that position is how the spec spells a *manifest*-style handshake +request, introduced in Bolt 5.7 (spec, *Handshake* § *Bolt version 5.7*) — so +FalkorDB answers a manifest request with a plain 4-byte version reply. + +Compare RESP, where versioning is an optional in-band `HELLO 2|3` command sent +after the connection is already usable — question 3 asks which of the two a +proxy can transparently downgrade. ### Step 3 — PackStream: type in the high nibble, size in the low -**PackStream** is Bolt's serialization format — think binary JSON with an -extension point. Every value starts with a **marker byte**: the high nibble -says the type, and for "tiny" variants the low nibble carries the size, so -small values cost one marker byte total. From the FalkorDB source -(bolt.c): NULL is 0xC0 (:11), tiny-string base 0x80 (:21) — so a 5-char -string is marker 0x85 + 5 bytes. +> **In:** an agreed 5.x version from Step 2; from here on both sides spell +> values in PackStream. +> **Out:** the byte-level encoding of scalars — the alphabet Step 4's +> structures and Step 5's messages are both written in. + +**PackStream** is Bolt's serialization format: binary JSON with an extension +point. Every value begins with a **marker byte** — one byte that says what the +value is, and for small values how big it is (spec, *PackStream* § *General +representation*). A **nibble** is half a byte, four bits: the marker's high +nibble picks the type family, and for "tiny" variants the low nibble carries the +size, so a small value costs one marker byte total. + +FalkorDB's markers are one `#define` block, and they match the spec exactly: + +```c +// src/bolt/bolt.c — the marker table, 11-39 + 11 #define NULL_MARKER 0xC0 + // ... 12-13: TRUE_MARKER 0xC3, FALSE_MARKER 0xC2 ... + 14 #define TINY_INT8_MIN 0xF0 + 15 #define TINY_INT8_MAX 0x7F + // ... 16-20: INT8/16/32/64 markers 0xC8-0xCB, FLOAT_MARKER 0xC1 ... + 21 #define TINY_STRING_BASE_MARKER 0x80 + // ... 22-31: STRING8/16/32, TINY_LIST 0x90, LIST8/16/32, BYTES8/16/32 ... + 32 #define TINY_MAP_BASE_MARKER 0xA0 + // ... 33-35: MAP8/16/32 markers 0xD8-0xDA ... + 36 #define STRUCTURE_BASE_MARKER 0xB0 + 37 + 38 #define TINY_SIZE 16 + 39 #define TINY_MARKER_CHECK(base, marker) (marker >= base && marker <= base + 0x0F) +``` -Integers are varint-by-cases: `bolt_reply_int` (bolt.c:133) picks -tiny-int/int8/16/32/64 by value, biased so the common range -16..127 costs -exactly one byte: +Line 38 is the one that explains the shape: `TINY_SIZE` is 16 because a nibble +holds 0–15. A string of 5 characters is marker `0x80 + 5 = 0x85` then 5 bytes; a +list of 3 items is `0x93` then the items; a map of 2 pairs is `0xA2` then four +values. Sixteen or more, and you pay a separate 8-, 16- or 32-bit size field +(spec, *PackStream* § *Sized values*). + +Integers are a **varint-by-cases** encoding — a variable-length integer where +the width is chosen per value rather than fixed — biased so the common range +costs one byte: + +```c +// src/bolt/bolt.c — bolt_reply_int, 130-151 + 130 // write int value to client response buffer + 131 // using the minimal representation + 132 // if the minimal representation is known use it for better performance + 133 void bolt_reply_int + 134 ( + 135 bolt_client_t *client, // client to write to + 136 int64_t data // int value to write + 137 ) { + 138 ASSERT(client != NULL); + 139 + 140 if(data >= TINY_INT8_MIN && data <= TINY_INT8_MAX) { + 141 bolt_reply_tiny_int(client, data); + 142 } else if(INT8_MIN <= data && data <= INT8_MAX) { + 143 bolt_reply_int8(client, data); + 144 } else if(INT16_MIN <= data && data <= INT16_MAX) { + 145 bolt_reply_int16(client, data); + 146 } else if(INT32_MIN <= data && data <= INT32_MAX) { + 147 bolt_reply_int32(client, data); + 148 } else { + 149 bolt_reply_int64(client, data); + 150 } + 151 } +``` + +Line 140 is the one to focus on: `TINY_INT8_MIN` is `0xF0` and `TINY_INT8_MAX` +is `0x7F` (lines 14–15), which as signed bytes are **−16 and +127** — exactly +the range the spec calls TINY_INT and says is "encoded within a single byte". +`bolt_reply_tiny_int` (bolt.c:68–77) then writes the value itself as the marker, +with no separate type byte at all. + +Worked example — three integers through this ladder, checked against the +spec's *optimal representation* table: -```rust -// High nibble = type, low nibble = size for "tiny" variants; ints are -// varint-by-cases, biased so -16..127 costs exactly one byte. -fn write_int(out: &mut Vec, v: i64) { - match v { - -16..=127 => out.push(v as u8), // tiny - _ if i8::try_from(v).is_ok() => { out.push(0xC8); out.push(v as u8); } - _ if i16::try_from(v).is_ok() => { out.push(0xC9); out.extend((v as i16).to_be_bytes()); } - _ if i32::try_from(v).is_ok() => { out.push(0xCA); out.extend((v as i32).to_be_bytes()); } - _ => { out.push(0xCB); out.extend(v.to_be_bytes()); } - } -} -fn write_struct_header(out: &mut Vec, n_fields: u8, tag: u8) { - out.push(0xB0 + n_fields); // marker: tiny structure of n fields - out.push(tag); // 0x4E Node, 0x52 Relationship, 0x50 Path, 0x10 RUN… -} // then the fields follow, each PackStream-encoded ``` + 42 → 140 true (−16 ≤ 42 ≤ 127) → tiny_int 2A 1 byte + 300 → 144 true (INT16 range) → int16 C9 01 2C 3 bytes + −17 → 142 true (INT8 range) → int8 C8 EF 2 bytes +2^40 (1.1e12) → 148 false, falls through → int64 CB 00 00 01 00 ... 9 bytes +``` + +−17 is the interesting one: it misses the tiny range by exactly one, and the +cost of that one step is a doubling, 1 byte → 2. Spec agrees: its table gives +INT_8 for −128…−17 and TINY_INT for −16…+127. -Compare topic 7 §1: RESP optimizes the *parser* (scan for \r\n); -PackStream optimizes the *type round-trip* (marker dispatch table). +Compare [topic 7 §1](README.md#1-resp-a-protocol-optimized-for-the-parser): +RESP optimizes the *parser* (ASCII lengths, `memchr` for CRLF, never scan the +payload); PackStream optimizes the *type round-trip* — one marker byte +dispatches to a decoder that already knows the target type. ### Step 4 — structures: one mechanism for messages AND graph types -PackStream's extension point is the **structure**: a marker byte -`0xB0 + n_fields` (bolt.c:36), then a **tag byte** naming what the -structure *is*, then that many fields, each PackStream-encoded and -arbitrarily nested (lists :198, maps :225, structures :250 in bolt.c). +> **In:** the scalar markers from Step 3. +> **Out:** the one composite form — the structure — that both protocol messages +> (Step 5) and graph values (this step) are built from, plus the byte count that +> shows what typing buys against RESP. + +A **structure** is PackStream's extension point: marker `0xB0 + n_fields`, then +a **tag byte** naming what the structure *is*, then that many +PackStream-encoded fields, arbitrarily nested. The spec is explicit that the +size is a *field count*, not a byte count, and that a structure holds "up to 15 +fields" (spec, *PackStream* § *Structure*). FalkorDB's writer is five lines: + +```c +// src/bolt/bolt.c — bolt_reply_structure, 248-260 + 248 // write structure header to client response buffer + 249 // expected 'size' number of items to follow + 250 void bolt_reply_structure + 251 ( + 252 bolt_client_t *client, // client to write to + 253 bolt_structure_type type, // structure type + 254 uint32_t size // number of items to follow + 255 ) { + 256 ASSERT(client != NULL); + 257 + 258 int8_t values[2] = {STRUCTURE_BASE_MARKER + size, type}; + 259 buffer_write(&client->write_buf.write, values, 2); + 260 } +``` + +Line 258 carries the argument, and note what it does *not* have: unlike +`bolt_reply_string` (170–194), `bolt_reply_list` (198–219) and `bolt_reply_map` +(225–246), which each branch four ways on size, the structure writer has no +large form and no bounds check. `0xB0 + size` is written unconditionally, so a +17-field structure would emit `0xC1` — the FLOAT marker. That is safe only +because the spec caps structures at 15 fields and every call site in this tree +passes a literal 0–8. + +The elegant part is the tag byte's namespace. One enum covers the protocol's +*messages* and its *data types*: + +```c +// src/bolt/bolt.h — bolt_structure_type, 27-49 (elided rows are more messages) + 27 typedef enum bolt_structure_type { + 28 BST_HELLO = 0x01, // hello message from client + // ... 29-30: GOODBYE 0x02, RESET 0x0F ... + 31 BST_RUN = 0x10, // run query message from client + // ... 32-34: BEGIN 0x11, COMMIT 0x12, ROLLBACK 0x13 ... + 35 BST_DISCARD = 0x2F, // discard all message from client + 36 BST_PULL = 0x3F, // pull records message from client + 37 BST_NODE = 0x4E, // node value + 38 BST_PATH = 0x50, // path value + 39 BST_RELATIONSHIP = 0x52, // relationship value + // ... 40-43: POINT2D 0x58, ROUTE 0x66, LOGON 0x6A, LOGOFF 0x6B ... + 44 BST_SUCCESS = 0x70, // success message + 45 BST_RECORD = 0x71, // record message + // ... 46-48: UNBOUND_RELATIONSHIP 0x72, IGNORED 0x7E, FAILURE 0x7F ... + 49 } bolt_structure_type; +``` + +Lines 37–39 sit in the same enum as lines 31 and 44–45: a `RUN` message and a +`Node` value are the same kind of thing on the wire, distinguished only by the +tag. A Path (0x50) is a structure whose fields are lists of Node and +Relationship structures. RESP has no equivalent — there is one composite type, +the array, and no way to label it. -The elegant part: one tag enum (`BST_*`, bolt.h:27) covers both the -protocol's *messages* — HELLO 0x01, RUN 0x10, PULL 0x3F, RECORD 0x71 — and -its *data types* — Node 0x4E, Relationship 0x52, Path 0x50. A Path is a -structure of lists of Node/Relationship structures; a RUN message is a -structure of (query string, params map, extra map). RESP has no -equivalent: a FalkorDB RESP reply encodes a node as nested arrays the -client library must re-interpret; Bolt puts the graph in the type system. +```rust +// ILLUSTRATION — not quoted from FalkorDB. The real writers are +// src/bolt/bolt.c:250-260 (structure header) and src/bolt/bolt.c:133-151 +// (integers); this is the same two decisions in one readable place. +fn write_struct_header(out: &mut Vec, n_fields: u8, tag: u8) { + out.push(0xB0 + n_fields); // bolt.c:258 — no large form, no bounds check + out.push(tag); // 0x4E Node, 0x52 Relationship, 0x50 Path, 0x10 RUN +} // then n_fields values follow, each PackStream-encoded +``` -### Step 5 — RUN/PULL: the client drives the stream +Worked example — what the typing actually costs and saves. Take one node: id +42, label `Person`, properties `name="Alice"` and `age=30`. FalkorDB's Bolt +formatter emits it as a 4-field Node structure (`resultset_replybolt.c:134`, +following the field list in the comment at :127–132): -Bolt splits "execute" from "fetch". `RUN` executes the query, and the -server replies only metadata (`SUCCESS {fields}` — column names, **no rows -sent!**). Rows flow only in response to `PULL {n: 1000}` — n RECORDs, then -`SUCCESS {has_more: true}` — and the client either PULLs again or sends -`DISCARD` (stop paying for rows it doesn't want). The whole session: +``` +B4 4E structure, 4 fields; tag Node 2 bytes +CB 00 00 00 00 00 00 00 2A id — bolt_reply_int64 at :135, NOT the minimal form 9 bytes +91 86 "Person" tiny list of 1, tiny string of 6 8 bytes +A2 84 "name" 85 "Alice" tiny map of 2 pairs; key 4, value 5 + 83 "age" 1E key 3, value 30 as a tiny int 17 bytes +87 "node_42" element_id, built by sprintf at :117 8 bytes + total = 44 bytes +``` + +The same node through the RESP verbose formatter +(`resultset_replyverbose.c:131–168`, whose reply shape is the comment at +:132–138) is a 3-element array of `["id", n]`, `["labels", [...]]`, +`["properties", [[k, v], ...]]`: + +``` +*3\r\n *2\r\n $2\r\nid\r\n :42\r\n 4 + 4 + 8 + 5 = 21 bytes +*2\r\n $6\r\nlabels\r\n *1\r\n $6\r\nPerson\r\n 4 + 12 + 4 + 12 = 32 bytes +*2\r\n $10\r\nproperties\r\n *2\r\n 4 + 17 + 4 + *2\r\n $4\r\nname\r\n $5\r\nAlice\r\n + 4 + 10 + 11 + *2\r\n $3\r\nage\r\n :30\r\n + 4 + 9 + 5 = 68 bytes + total = 121 bytes +``` + +121 / 44 = **2.75×**, and the RESP version does not even carry the element id. +Two details make the comparison honest rather than flattering. Bolt spends 9 +bytes on an id that would fit in one, because `_ResultSet_BoltReplyWithNode` +calls `bolt_reply_int64` (:135) rather than the minimal `bolt_reply_int` from +Step 3 — property *values* do use the minimal form (:45), node ids do not. And +the RESP reply's comment at :136 promises `[name, value, value type]` triples +while the code at :117 replies an array of 2; the third element is not emitted. +Read the code, not the comment. + +Wrapped for the wire, that node costs a little more: `bolt_client_reply_for` +writes a 2-byte chunk-length placeholder plus `B1 71` (a 1-field RECORD +structure), then a `91` list header for the single column, then the 44 bytes, +then a 2-byte terminator — **51 bytes** per single-column RECORD. Step 5 +explains the chunk bytes. + +### Step 5 — RUN/PULL: the client drives the stream (as specified) + +> **In:** the structures from Step 4, now used as messages rather than values. +> **Out:** the message sequence a Bolt driver expects, and the chunk framing +> that carries it. Step 6 checks this sequence against FalkorDB's code and finds +> half of it missing. + +Bolt splits "execute" from "fetch". `RUN` (tag 0x10, spec *Messages* § +*Request message RUN*) carries the query, its parameters and an extras map, and +the server answers only *metadata* — `SUCCESS {fields}`, the column names. +Records flow only in response to `PULL` (tag 0x3F): its `extra` map has `n`, +"how many records to fetch", which "has no default and must be present" since +Bolt 4.0, and `qid`, naming which open result to pull from. The server answers +with zero or more RECORDs and then `SUCCESS {has_more}` — "true if there are +more records to stream" (spec, *Messages* § *Request message PULL*). The client +either pulls again or sends `DISCARD` (0x2F) to stop paying for rows it does not +want. ``` client server - │ 0x60 0x60 0xB0 0x17 + 4 versions │ handshake: bolt_api.c:803, - ├────────────────────────────────────►│ version pick :845-864 - │◄──────────────── chosen version ────┤ (5.1..5.7 accepted) - │ HELLO {auth...} 0x01 │ + │ 60 60 B0 17 + 4×4 version bytes │ 20 bytes in, 4 back: + ├────────────────────────────────────►│ bolt_client.c:679 (magic), + │◄──────────────── chosen version ────┤ :690-693 (proposals), api.c:850/:864 + │ HELLO {auth...} 0x01 │ bolt_api.c:708-710 │◄─────────────── SUCCESS 0x70 ──────┤ - │ RUN "MATCH..." {} {} 0x10 │ bolt_api.c:721 - │◄─────────────── SUCCESS {fields} ───┤ (query ran; no rows sent!) - │ PULL {n: 1000} 0x3F │ bolt_api.c:726 - │◄─────────────── RECORD × n 0x71 ───┤ client-driven streaming: - │◄─────────────── SUCCESS {has_more}──┤ backpressure IS the protocol - │ DISCARD 0x2F │ (or: stop paying for rows) -``` - -This is a pull-based cursor *in the protocol* — pgwire's portal -(Execute {max_rows} / PortalSuspended) rediscovered, with the client -explicitly naming its batch size. The cost: between RUN and the final PULL, -the server holds a suspended result — state per open cursor (question 1: -what does 10K idle cursors cost?). - -One framing detail: PackStream values have no overall message-length -prefix, so messages are wrapped in **chunks** — 2-byte length headers, a -0x0000 chunk as terminator. Chunking lets the server start transmitting a -RECORD before knowing the full message size — it streams records as it -produces them (question 2). - -### Step 6 — the server side: one switch, two decouplings - -The FalkorDB implementation shows how little a Bolt server core is: - -- `BoltRequestHandler` (bolt_api.c:670): one dispatch switch over the - `BST_*` message tags — the protocol state machine is ~10 cases. -- RUN executes the query but replies only metadata (:467–482); records - flow in the PULL handler (:504–521) — result *materialization* and - result *transport* are decoupled server-side too, mirroring Step 5's - wire-level split. -- `ws_handshake` (bolt_api.c:831): the same port sniffs and upgrades - WebSocket — that's how browser clients speak Bolt. -- It's all inside a Redis module: the Bolt socket bypasses RESP entirely - and injects work into the same executor — two protocols, one engine. - -Why it matters for M7: the stretch goal is exactly this shape — a Bolt -listener beside your RESP one, sharing the executor and result set -(question 6). + │ RUN "MATCH..." {} {} 0x10 │ case at bolt_api.c:721-723 + │◄─────────────── SUCCESS {fields} ───┤ resultset_replybolt.c:278-295 + │ PULL {n: 1000} 0x3F │ case at bolt_api.c:726-730 + │◄─────────────── RECORD × n 0x71 ───┤ resultset_replybolt.c:263-275 + │◄─────────────── SUCCESS {has_more}──┤ ← spec says so; FalkorDB never + │ DISCARD 0x2F │ sends has_more (Step 6) +``` + +The framing is the last piece. PackStream values carry no overall message +length, so messages are wrapped in **chunks**: each chunk is a 2-byte +big-endian length header followed by that many bytes, and each message ends with +a zero-length chunk, `00 00`. The spec's reason is the interesting part: "a +message can be divided across multiple chunks, allowing client and server alike +to transfer large messages without having to determine the length of the entire +message in advance" (spec, *Message* § *Chunking*). The header is 16 bits, so +one chunk holds at most 65,535 bytes. + +FalkorDB writes the header *first* and fills it in *afterwards*: + +```c +// src/bolt/bolt_client.c — the chunk header placeholder, inside +// bolt_client_reply_for, 569-575 + 569 msg.bolt_header = client->write_buf.write; + 570 buffer_write_uint16(&client->write_buf.write, 0x0000); + 571 msg.start = client->write_buf.write; + 572 msg.end = client->write_buf.write; + 573 arr_append(client->write_messages, msg); + 574 + 575 bolt_reply_structure(client, response_type, size); +``` + +```c +// src/bolt/bolt_client.c — the back-patch, inside bolt_client_end_message, +// 586-598 + 586 bolt_message_t *msg = client->write_messages + arr_len(client->write_messages) - 1; + 587 msg->end = client->write_buf.write; + 588 uint64_t n = buffer_index_diff(&msg->end, &msg->start); + // ... 589-594: the WebSocket variant writes a frame header too ... + 595 buffer_index_t write_bolt_header = msg->bolt_header; + 596 buffer_write_uint16(&write_bolt_header, htons(n)); + 597 buffer_write_uint16(&client->write_buf.write, 0x0000); + 598 msg->end = client->write_buf.write; +``` + +Line 588 measures the finished message and line 596 writes that length back into +the two bytes reserved at 570; line 597 appends the `00 00` terminator. So this +implementation emits **exactly one chunk per message** and never splits — which +means it buys none of the "without having to determine the length in advance" +property the spec's chunking exists for, and depends on every message fitting in +the 16-bit header (`htons` of a `uint64_t` truncates above 65,535). + +### Step 6 — what FalkorDB's server actually implements + +> **In:** the specified message flow from Step 5. +> **Out:** the audit — which parts of the flow exist in `src/bolt/`, which are +> empty, and what that means for the backpressure claim the chapter opened with. + +The dispatch loop is as small as advertised. `BoltRequestHandler` +(bolt_api.c:670) reassembles a chunked message and switches on its tag — +11 labelled cases plus `default`, at :706–745. Two of those cases are the whole +finding: + +```c +// src/bolt/bolt_api.c — inside BoltRequestHandler's switch, 721-730 + 721 case BST_RUN: + 722 BoltRunCommand(client); + 723 break; + 724 case BST_DISCARD: + 725 break; + 726 case BST_PULL: + 727 BoltPullCommand(client); + 728 client->processing = false; + 729 BoltRequestHandler(client); + 730 break; +``` + +Line 724 is an empty case: `DISCARD` does nothing at all. And line 727 calls +this: + +```c +// src/bolt/bolt_api.c — BoltPullCommand, in full, 539-552 + 539 // handle the PULL message + 540 void BoltPullCommand + 541 ( + 542 bolt_client_t *client // the client that sent the message + 543 ) { + 544 // The PULL message requests data from the remainder of the result stream + 545 // input: + 546 // extra::Dictionary{ + 547 // n::Integer, + 548 // qid::Integer, + 549 // } + 550 + 551 ASSERT(client != NULL); + 552 } +``` + +The line to look at is 551, because it is the only one: the body is an assert. +`n` is documented in the comment and never read. **FalkorDB's Bolt server does +not implement the cursor.** The rows are produced by the query, not by the pull: +`BoltRunCommand` dispatches `graph.QUERY` with a `--bolt` marker argument +(bolt_api.c:526–532, strings created at :979–980), and the result set formatter +emits everything as it goes — + +```c +// src/resultset/formatters/resultset_replybolt.c — ResultSet_EmitBoltRow, 263-275 + 263 void ResultSet_EmitBoltRow + 264 ( + 265 ResultSet *set, + 266 SIValue **row + 267 ) { + 268 bolt_client_t *bolt_client = set->bolt_client; + 269 bolt_client_reply_for(set->bolt_client, BST_PULL, BST_RECORD, 1); + 270 bolt_reply_list(set->bolt_client, set->column_count); + 271 for(int i = 0; i < set->column_count; i++) { + 272 _ResultSet_BoltReplyWithSIValue(bolt_client, set->gc, *row[i]); + 273 } + 274 bolt_client_end_message(bolt_client); + 275 } +``` + +Line 269 is the tell: every RECORD is *labelled* a reply to `PULL`, and this +function is called once per row by the executor as rows are produced. The +matching header (`ResultSet_ReplyWithBoltHeader`, :278–295) labels its metadata +a reply to `RUN` at :283. A driver therefore sees a well-formed Bolt +conversation — SUCCESS-for-RUN, RECORDs-for-PULL, SUCCESS-for-PULL from +`ResultSet_EmitBoltStats` (:297, :301) — while the server behind it never +suspended anything. `has_more` appears nowhere in `src/bolt/` or in the Bolt +formatter; grep both and you get nothing. + +So the honest version of the chapter's opening claim: **Bolt specifies +protocol-level backpressure, and FalkorDB implemented Bolt's typing without it.** +A 10M-row `MATCH` buffers 10M rows into `client->write_buf` exactly as RESP +would — the axe of [topic 7 §4](README.md#4-backpressure--the-part-everyone-forgets) +just falls on a different buffer. + +There is a second thing the code does *not* do, and it costs more than it looks. +`BoltRequestHandler` refuses to start a message while one is in flight: + +```c +// src/bolt/bolt_api.c — the in-flight gate, 676-680 and 704 + 676 // if there is a message already in process or + 677 // not enough data to read the message + 678 if(client->processing || buffer_index_length(&client->read_buf.read) <= 2) { + 679 return; + 680 } + // ... 682-702: reassemble the chunked message into msg_buf ... + 704 client->processing = true; +``` + +Line 678 is **head-of-line blocking** — a queue discipline where one unfinished +item stops every item behind it, however ready they are. One Bolt message per +connection at a time means no **pipelining** (sending k requests back-to-back +without waiting for replies), so every message pays a full **round trip**: the +wire, kernel and wakeup cost of one send-and-receive exchange. The flag is +cleared in `BoltResponseHandler` after the socket write (:894) and, for PULL, +inline at :728. Topic 7's own lane prices that discipline: identical zero-work +requests run at **44,088 ops/s at pipeline depth P=1 and 12,321,414 at P=256** +([FINDINGS.md](../../FINDINGS.md) row 7, full table in [notes.md](notes.md)) — +a 279× swing that is nothing but round trips. A protocol that structurally +forbids depth > 1 has chosen the left-hand end of that curve. + +What the implementation *does* get right is the decoupling that makes a second +protocol cheap, in three parts. First, the listener is its own port riding +redis's event loop rather than a thread of its own: + +```c +// src/bolt/bolt_api.c — inside BoltApi_Register, 965-980 + 965 socket_t bolt = socket_bind(port); + // ... 966-972: bail out if the bind failed; detach a thread-safe context ... + 973 if(RedisModule_EventLoopAdd(bolt, REDISMODULE_EVENTLOOP_READABLE, BoltAcceptHandler, global_ctx) == REDISMODULE_ERR) { + // ... 974-976: log and fail ... + 977 RedisModule_Log(NULL, "notice", "Bolt protocol initialized. Port: %d", port); + 978 + 979 COMMAND = RedisModule_CreateString(global_ctx, "graph.QUERY", 11); + 980 BOLT = RedisModule_CreateString(global_ctx, "--bolt", 6); +``` + +Line 973 is the one that matters: the Bolt socket is registered on the *same* +`ae` loop that serves RESP. `RedisModule_EventLoopAdd` is a thin wrapper that +ends in `aeCreateFileEvent(server.el, …)` — redis `src/module.c:10115`, at the +`a176d1225` pin — so two protocols share one thread and one engine, and +everything [reading-redis-ae-networking.md](reading-redis-ae-networking.md) says +about the loop applies to Bolt unchanged. Second, line 979: RUN turns into the +ordinary `graph.QUERY` command with a `--bolt` marker (:980), so only the result +*formatter* differs. Third, `ws_handshake` (called at bolt_api.c:831, defined in +`src/bolt/ws.c:110`) sniffs a WebSocket upgrade on the same port when the magic +bytes are absent — which is how browser clients speak Bolt. + +Why it matters for M7: the stretch goal is exactly this shape — a Bolt listener +beside your RESP one, sharing the executor and result set — and this +implementation tells you which half is cheap (the listener, the formatter) and +which half is the actual work (the cursor, question 6). ## Where each step lives in the code -All in the removed `src/bolt/` tree — read frozen in time with -`git show 0b11a00b3^:src/bolt/` in `~/repos/FalkorDB`: +All in the removed `src/bolt/` tree plus the Bolt result formatter, at +`0b11a00b3^` = `40780e992`: | Anchor | What | Step | |--------|------|------| -| bolt_api.c:803, :845–864 | handshake magic + version pick (5.1..5.7) | 2 | -| bolt.c:11, :21, :36 | markers: NULL 0xC0, tiny-string 0x80, structure 0xB0 | 3 | -| bolt.c:133 — `bolt_reply_int` | varint-by-cases integers | 3 | -| bolt.c:198 / :225 / :250 | lists, maps, `bolt_reply_structure` | 4 | -| bolt.h:27 — `BST_*` enum | messages + Node 0x4E / Relationship 0x52 / Path 0x50 | 4 | -| bolt_api.c:721 / :726 | RUN and PULL entry points | 5 | -| bolt_api.c:670 — `BoltRequestHandler` | the ~10-case dispatch switch | 6 | -| bolt_api.c:467–482 / :504–521 | RUN replies metadata; PULL streams records | 6 | -| bolt_api.c:831 — `ws_handshake` | WebSocket sniff-and-upgrade | 6 | - -Suggested route: bolt.h (the enum, Step 4) → bolt.c top-down (markers → -ints → containers, Steps 3–4) → bolt_api.c following one session in the -Step 5 diagram's order. +| `bolt_client.c:672-680` — `bolt_check_handshake` | magic `0x6060B017`, compared at :679 | 2 | +| `bolt_client.c:682-695` — `bolt_read_supported_version` | reads 16 bytes, uses `data[2..3]` — the first proposal only | 2 | +| `bolt_api.c:845-866` | accept test at :850 (major 5, minor ≥ 1); reply `MIN(minor,7)` at :864 | 2 | +| `bolt.c:11-39` | marker table; `TINY_SIZE 16` at :38 is why nibbles work | 3 | +| `bolt.c:130-151` — `bolt_reply_int` | varint-by-cases; the tiny test is :140 | 3 | +| `bolt.c:170-194 / :198-219 / :225-246` | string, list, map — each branches four ways on size | 3 | +| `bolt.c:248-260` — `bolt_reply_structure` | `0xB0 + size` at :258, no large form, no bounds check | 4 | +| `bolt.h:27-49` — `BST_*` | messages *and* Node 0x4E (:37) / Path 0x50 (:38) / Relationship 0x52 (:39) | 4 | +| `resultset_replybolt.c:121-161` | the 4-field Node; `bolt_reply_int64` for the id at :135 | 4 | +| `resultset_replybolt.c:109-119` | `element_id` = `sprintf("%s_%llu")` at :117 | 4 | +| `bolt_client.c:569-575 / :586-598` | chunk header placeholder, then back-patched at :596 | 5 | +| `bolt_api.c:670`, switch :706-745 | `BoltRequestHandler` — 11 cases + default | 6 | +| `bolt_api.c:539-552` — `BoltPullCommand` | **empty**: one `ASSERT` at :551 | 6 | +| `bolt_api.c:724-725` | `BST_DISCARD` — an empty case | 6 | +| `bolt_api.c:676-680`, :704, :894 | the `processing` gate: one message in flight per connection | 6 | +| `resultset_replybolt.c:263-275` / :278-295 / :297 | RECORD, header, stats — labelled replies to PULL/RUN | 6 | +| `bolt_api.c:949-984` — `BoltApi_Register` | own port at :965, on redis's event loop at :973 | 6 | +| `bolt_api.c:831`, `ws.c:110` | WebSocket sniff-and-upgrade on the same port | 6 | + +Suggested route: `bolt.h` (the enum, Step 4) → `bolt.c` top-down (markers → +ints → containers, Steps 3–4) → `bolt_api.c` following one session in the +Step 5 diagram's order — and when you reach `BoltPullCommand`, stop and check +whether you believe the chapter's Step 6 or its Step 5. ## Questions -1. RUN/PULL splits "execute" from "fetch". What does the server have to - *hold* between the two, and what does that cost under 10K idle - cursors? (Compare pgwire portals, topic 7 §4.) -2. PackStream has no length prefix on messages — chunking (2-byte chunk - headers, 0x0000 terminator) wraps it. Why chunk instead of - length-prefixing the whole message, for a server that streams records - as it produces them? -3. The handshake proposes four versions, server picks one - (bolt_api.c:845-864 clamps to 5.1..5.7). Compare RESP's HELLO 2/3. - Which design lets a proxy transparently downgrade, and why? -4. Node/Relationship on the wire carry element ids + property maps. - What does this rule out that RESP's "everything is arrays" allows — - and which side of the trade does a *new* graph database want? -5. Why might FalkorDB have removed Bolt (#2170)? List the real costs a - second protocol imposes on an engine (state machines, result - encoders, auth, tests, fuzz surface) — then what you'd need to keep - it cheap. -6. **M7 mapping**: the stretch goal is a Bolt listener beside RESP. Which - pieces of your M7 RESP server are protocol-neutral (executor, - result set) and which need a Bolt twin? Sketch the - `bolt_reply_*`-equivalent trait your result set must implement. +1. RUN/PULL splits "execute" from "fetch". What would the server have to *hold* + between the two to implement it properly, and what does that cost under 10K + idle cursors? (Compare pgwire portals, + [topic 7 §4](README.md#4-backpressure--the-part-everyone-forgets).) +2. PackStream has no length prefix on messages — chunking (2-byte headers, + `00 00` terminator) wraps it. The spec's stated reason is streaming without + knowing the total length; FalkorDB back-patches a single chunk header + instead (bolt_client.c:596). What would have to change in + `bolt_client_end_message` to emit a genuinely streamed multi-chunk RECORD, + and what breaks if a message exceeds 65,535 bytes today? +3. The handshake takes four proposals in preference order and FalkorDB reads + the first (bolt_client.c:692-693), rejecting anything that is not 5.x with + minor ≥ 1. Compare RESP's in-band `HELLO 2|3`. Which design lets a proxy + transparently downgrade a connection, and why? +4. Node/Relationship on the wire carry element ids and property maps. What does + that rule out that RESP's "everything is arrays" allows — and which side of + the trade does a *new* graph database want? +5. Why might FalkorDB have removed Bolt (#2170)? List the real costs a second + protocol imposes (state machines, result encoders, auth, tests, fuzz + surface), then say which of them the *unfinished* parts you found in Step 6 + — the empty PULL, the missing `has_more` — make better or worse. +6. **M7 mapping**: the stretch goal is a Bolt listener beside RESP. Which pieces + of your M7 server are protocol-neutral (executor, result set) and which need + a Bolt twin? Sketch the `bolt_reply_*`-equivalent trait your result set must + implement, and decide whether you implement `PULL {n}` for real. ## Done when +Answer each before unfolding it. + - [ ] You can write a PackStream marker byte from memory and decode type and size from its nibbles. + +
Answer + + The high nibble selects the type family and, for the "tiny" variants, the low + nibble is the size: `0x8_` string, `0x9_` list, `0xA_` map, `0xB_` structure. + So `0x85` is a 5-character string, `0x93` is a 3-item list, `0xA2` is a + 2-pair map, and `0xB4` is a 4-field structure whose *next* byte is the tag. + The cut-off is 16 because a nibble holds 0–15 — `TINY_SIZE` at `bolt.c:38`, + used by `TINY_MARKER_CHECK` at :39 and by each writer's first branch + (`bolt_reply_string` :179, `bolt_reply_list` :205, `bolt_reply_map` :232). + + Scalars have fixed markers rather than sizes: `0xC0` null (`bolt.c:11`), + `0xC2`/`0xC3` false/true, `0xC1` float, `0xC8`–`0xCB` int8/16/32/64. Integers + in −16…+127 have no marker at all — the value *is* the byte + (`bolt_reply_tiny_int`, bolt.c:68–77, guarded at :140 by `TINY_INT8_MIN` + `0xF0` and `TINY_INT8_MAX` `0x7F`). That is why 42 costs 1 byte, −17 costs 2, + and 300 costs 3. + +
+ - [ ] You can explain how one structure mechanism serves both protocol messages and graph types, and why that is more than an aesthetic choice. + +
Answer + + A structure is `0xB0 + n_fields`, a tag byte, then the fields + (`bolt_reply_structure`, bolt.c:250–260, the write at :258). The tag is drawn + from a single enum that contains `BST_RUN = 0x10` (bolt.h:31) and + `BST_RECORD = 0x71` (:45) alongside `BST_NODE = 0x4E` (:37), + `BST_PATH = 0x50` (:38) and `BST_RELATIONSHIP = 0x52` (:39). A message and a + node are the same shape; only the tag differs. + + It is not aesthetic because it makes the *encoder* recursive and the *decoder* + a single dispatch table. `_ResultSet_BoltReplyWithSIValue` + (resultset_replybolt.c:33) is one switch that reaches nodes (:57), edges + (:60) and paths by calling the same primitives that write a RUN's parameter + map — a Path is a structure of lists of Node and Relationship structures, and + nothing special is needed to nest it. On the client side a driver registers + one handler per tag and gets graph objects out of the decoder rather than + reconstructing them: for the node in Step 4, 44 typed bytes against RESP's + 121 untyped ones, with the element id included in the smaller number. + +
+ - [ ] You can say what RUN/PULL buys by splitting execute from fetch, and what the server must therefore hold between them. + +
Answer + + The split buys client-driven flow control. `RUN` (0x10) returns only + `SUCCESS {fields}`; rows arrive only when the client sends `PULL {n}` (0x3F), + whose `n` "has no default and must be present" since Bolt 4.0, and the server + answers `SUCCESS {has_more: true}` if there is more (spec, *Messages* § + *Request message PULL*). The client sizes its own bites and can walk away with + `DISCARD` (0x2F). + + What the server must hold between the two is the expensive part: a suspended + execution — the plan, its iterators, the read transaction or snapshot the rows + are being read under, and enough identity (`qid`) to route a later PULL to the + right one. That is per-connection state that survives across event-loop turns, + which is why 10K idle cursors is a real cost and why pgwire's portals carry + the same liability. + + FalkorDB declined to pay it. `BoltPullCommand` (bolt_api.c:539–552) is one + `ASSERT` and reads neither `n` nor `qid`; `BST_DISCARD` (:724–725) is an empty + case; `has_more` appears nowhere. Rows are emitted by + `ResultSet_EmitBoltRow` (resultset_replybolt.c:263–275) as the executor + produces them, merely *labelled* as replies to PULL at :269. The wire looks + like a cursor; the server is buffer-or-die, the same as RESP. + +
+ - [ ] You can explain how chunking substitutes for a message length prefix, and what that costs a parser. -- [ ] You wrote answers to all questions in notes.md, including the honest cost list for why FalkorDB removed Bolt. + +
Answer + + A PackStream value's marker tells you the size of *that value*, never of the + message containing it, so the transport adds a layer: each chunk is a 2-byte + big-endian length followed by that many bytes, and a message ends with a + zero-length chunk `00 00`. Because a message may span several chunks, a sender + can start transmitting before it knows the total length (spec, *Message* § + *Chunking*); the 16-bit header caps one chunk at 65,535 bytes. + + The cost lands on the reader, which now has two loops: reassemble chunks until + the terminator, *then* parse PackStream. That is exactly `BoltRequestHandler` + at bolt_api.c:690–696 — read a `uint16`, copy that many bytes into `msg_buf`, + repeat until the length reads zero — and it must also handle "the next chunk + has not arrived yet" by returning and resuming (:693). + + FalkorDB's *writer* takes none of the benefit. `bolt_client_reply_for` + reserves two bytes at bolt_client.c:570 and `bolt_client_end_message` + back-patches the finished length at :596 — one chunk per message, always, + which means the whole message was buffered before the header could be + written, and a message over 65,535 bytes truncates in `htons`. + +
+ +- [ ] You can state, from the code rather than the specification, how much of Bolt's backpressure FalkorDB implemented — and what that means for a 10M-row query. + +
Answer + + None of it. The evidence is three anchors: `BoltPullCommand` + (bolt_api.c:539–552) has an empty body, the `BST_DISCARD` case (:724–725) is + empty, and no `has_more` key is written anywhere in `src/bolt/` or + `resultset_replybolt.c`. The RECORDs come out of `ResultSet_EmitBoltRow` + (:263–275) as the executor produces rows, into `client->write_buf`, and the + socket write happens later in `BoltResponseHandler` (bolt_api.c:875, send at + :893). A 10M-row `MATCH` therefore materialises 10M encoded rows in the + module's buffer before a byte is guaranteed to move — the same buffer-or-die + failure mode as RESP, reached through a protocol whose spec exists partly to + prevent it. + + The connection is also strictly one message at a time: `BoltRequestHandler` + returns early while `client->processing` is set (:678, set at :704, cleared at + :894 and inline for PULL at :728). That is head-of-line blocking, so a Bolt + client cannot pipeline, and every message pays a full round trip. Topic 7's + lane prices that at 44,088 ops/s against 12,321,414 for the same zero-work + request at depth 256 ([FINDINGS.md](../../FINDINGS.md) row 7) — a 279× gap + that this design forgoes by construction. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the honest cost list for why FalkorDB removed Bolt. + +
Answer + + The cost list should be concrete, and the code gives you most of it: a second + message state machine (`bolt_change_client_state`, called from + bolt_client.c:576), a second result encoder (`resultset_replybolt.c`, 375 + lines, one branch per SIValue type), a second auth path (HELLO/LOGON at + bolt_api.c:708–713), a second framing layer to fuzz (chunk reassembly at + :690–696, plus the WebSocket variant at :686–689), a second listener and its + configuration (`BoltApi_Register` :949–984), and the version matrix Step 2 + showed is easy to get subtly wrong. + + Then note which way the unfinished parts cut. An *empty* `BoltPullCommand` is + cheap to maintain but is a promise the wire makes and the server does not + keep, so every driver that batches with `n` is silently getting + buffer-everything behaviour — a bug report that costs more than the code it + saved. The way to keep a second protocol cheap is the opposite of what + happened here: share the executor and the result set (which this + implementation does, via `graph.QUERY` at bolt_api.c:979 and + `CommandDispatch` at :532), and make the protocol layer a thin encoder over a + cursor abstraction that *both* protocols use — so `PULL {n}` and a future + `GRAPH.CURSOR` are the same code path. + +
## References -**Papers** -- Neo4j — Bolt Protocol + PackStream specifications - (https://neo4j.com/docs/bolt/current/) — the normative source for - markers, messages, and the handshake +**Specification** +- Neo4j — *Bolt Protocol* and *PackStream* specifications + (). Sections cited above: *Handshake* § + *Version negotiation* (four 4-byte proposals, first match wins) and § *Bolt + version 5.7* (the manifest handshake); *PackStream* § *General + representation*, § *Sized values*, § *Endianness*, § *Integer* (the optimal + representation table) and § *Structure* (tag byte, up to 15 fields); + *Message* § *Chunking* (2-byte headers, `00 00` terminator, 65,535 maximum); + *Messages* § *Request message RUN* (signature 10) and § *Request message + PULL* (signature 3F, `n` and `qid`, `has_more`). **Code** - [FalkorDB/FalkorDB](https://github.com/FalkorDB/FalkorDB) `src/bolt/` - (`bolt.c`, `bolt.h`, `bolt_api.c`) — removed by #2170; read it frozen - in time with `git show 0b11a00b3^:src/bolt/` in - `~/repos/FalkorDB` + (`bolt.c`, `bolt.h`, `bolt_api.c`, `bolt_client.c`, `ws.c`) and + `src/resultset/formatters/resultset_replybolt.c` — removed by #2170 on + 2026-07-08; read at `0b11a00b3^` = `40780e992ecc11f598ce3f4f65e04367f9abae2f` + with `tools/pinned-source.py --ref 40780e992… show FalkorDB `, or + `git show 0b11a00b3^:src/bolt/` in a clone. + +| File | Lines | What | +|------|-------|------| +| `src/bolt/bolt.c` | 11-39 | the marker table, and `TINY_SIZE 16` | +| `src/bolt/bolt.c` | 133-151 | `bolt_reply_int` — varint by cases | +| `src/bolt/bolt.c` | 250-260 | `bolt_reply_structure` — the whole extension point | +| `src/bolt/bolt.h` | 27-49 | one tag enum for messages and graph types | +| `src/bolt/bolt_client.c` | 672-695 | handshake magic and version read | +| `src/bolt/bolt_client.c` | 569-598 | chunk header written, then back-patched | +| `src/bolt/bolt_api.c` | 539-552 | `BoltPullCommand` — the empty cursor | +| `src/bolt/bolt_api.c` | 670-746 | `BoltRequestHandler` — the state machine | +| `src/bolt/bolt_api.c` | 845-866 | version acceptance and reply | +| `src/bolt/bolt_api.c` | 949-984 | second port, same event loop | +| `src/resultset/formatters/resultset_replybolt.c` | 121-161 | the Node structure on the wire | +| `src/resultset/formatters/resultset_replybolt.c` | 263-275 | RECORDs, emitted as rows are produced | + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 7 — 44k ops/s at P=1 against 12.3M at + P=256, the price of the round trips a non-pipelining protocol pays. diff --git a/topics/07-networking-protocols/reading-c10k-thread-per-core.md b/topics/07-networking-protocols/reading-c10k-thread-per-core.md index 24b4663..53d7f8b 100644 --- a/topics/07-networking-protocols/reading-c10k-thread-per-core.md +++ b/topics/07-networking-protocols/reading-c10k-thread-per-core.md @@ -1,166 +1,687 @@ # C10K to thread-per-core: what is a server thread for? -Three readings spanning 1999→2024, one thread: *what should a server thread -be responsible for?* Kegel's C10K catalog explains why event loops became the -default, valkey's 8.0 posts show disciplined Amdahl analysis parallelizing -exactly the profiled majority, and Glauber Costa's thread-per-core essays -take the radical endpoint — share nothing between cores. This chapter builds -the concepts in historical order — what a connection-thread costs, the -polling problem, readiness notification, the async-I/O detour, and the two -modern answers — then tells you how to read each source. Together they span -the shared↔sharded plane M7 must position itself in. +Three readings spanning 1999→2024, one question: *what should a server thread be +responsible for?* Dan Kegel's C10K page is the catalogue of answers that existed +when the question was new; valkey's 8.0 blog posts are a modern answer arrived at +by profiling; Glauber Costa's thread-per-core writing is the radical endpoint — +share nothing between cores. This chapter builds the concepts in the order the +industry found them, then checks each against code you can read: the event loop +FalkorDB and redis actually run, and the numbers this repo actually measured. + +**Which versions this chapter is about.** Kegel's page is a living document with +a stale heartbeat: its RCS log ends at *Revision 1.212, 2006/09/02*, and the +hand-written changelog has one later entry, *2011/07/21 — Added nginx.org*. So it +is a 1999 essay revised through 2006, and anything it says about "current" kernels +is twenty years old. The valkey claims come from two dated posts, 2024-08-05 and +2024-09-13. The redis code is the repo's pin: + +```sh +tools/pinned-source.py ref redis # a176d1225 +tools/pinned-source.py show redis src/ae.c -r 30:44 +tools/pinned-source.py check redis src/config.h:86 --contains 'HAVE_EPOLL' +``` + +Every `file:line` below was re-checked against that pin. Where this chapter used +to state a number with no source, it now either cites one or says the number was +removed. ## The problem in one sentence -In 1999, serving 10,000 concurrent connections with one thread each meant -10,000 stacks (~80 MB at 8 KB minimum each, far more at defaults) and a -scheduler drowning in context switches (~1–10 µs each) — and every design -since is a different answer to "what do we give a thread to do, if not one -connection?" +Kegel's arithmetic: a $1,200 machine of the day — 1000 MHz, 2 GB RAM, 1000 +Mbit/s — divided by 20,000 clients leaves 50 kHz, 100 KB and 50 kbit/s each, +"so hardware is no longer the bottleneck"; the bottleneck is the *unit of work +you hand a thread*, and every design since 1999 is a different answer to "what +do we give a thread to do, if not one connection?" + +(Check his division as you read — 1000 MHz / 20,000 = 50 kHz ✓, 2 GB / 20,000 = +100 KB ✓, 1000 Mbit/s / 20,000 = 50 kbit/s ✓, but $1,200 / 20,000 = **$0.06**, +not the $0.08 he prints. The argument survives the slip; the habit of checking +does not survive not checking.) ## The concepts, step by step -### Step 1 — the cost of thread-per-connection - -The naive server design gives each connection its own thread, which blocks -on `read()` until that client sends something. Simple, and the costs are -per-connection whether the connection is active or idle: a stack (memory -reserved per thread — 10K mostly-idle connections still hold 10K stacks), a -kernel scheduling slot, and a **context switch** (the ~1–10 µs of saving one -thread's registers and loading another's, plus the cache/TLB state the new -thread finds cold) every time attention moves between clients. In 1999 this -died at ~10K connections; Kegel's page is the catalog of escape routes. - -### Step 2 — select/poll: one thread, but O(n) per wakeup - -The first escape: one thread watches *all* the sockets by handing the -kernel the full list of fds (file descriptors — the small integers naming -open sockets) via `select` or `poll`, sleeping until any is ready. That -fixes the 10K-stacks problem but adds a new tax: the fd list is passed and -scanned *on every call* — O(n) in registered connections, even if only 3 -are ready. At 10K mostly-idle connections you burn the CPU scanning 10K -entries to find 3 events, thousands of times per second. - -### Step 3 — readiness notification: epoll/kqueue, the line that won - -**Readiness notification** means the kernel keeps the interest list -*between* calls: you register each fd once (`epoll_ctl`/`kevent`), then each -wait returns *only the ready fds* — O(ready), not O(registered). 10K idle -connections cost nothing per wakeup; 3 ready ones cost 3 dispatches. This -is the line that won: `ae.c` (reading-redis-ae-networking.md) is exactly -this plus a dispatch table, and every mainstream event loop (libuv, mio -under tokio) is the same shape. - -### Step 4 — async I/O: stillborn, then resurrected as io_uring - -The fourth entry in Kegel's menu was **asynchronous I/O**: instead of "tell -me when the fd is ready, then I'll call read()", submit the *operation -itself* ("read 16 KB from fd 7 into this buffer") and get completion -notification. POSIX aio was stillborn on Linux for sockets — but the idea -returned twenty years later as **io_uring**: two lock-free rings shared -with the kernel (submission queue in, completion queue out), so batches of -reads/writes/accepts cost near-zero syscalls. Question 4 asks what ae.c's -design becomes when poll+read+write turn into submission entries. - -The full 1999 menu, with hindsight: - -1. thread per connection — dies at ~10K in 1999 (stacks, context switches) -2. select/poll — O(n) scans of the fd set per wakeup -3. **readiness notification** (epoll/kqueue) — O(ready) not O(registered): - this line wins; ae.c is exactly this + a dispatch table -4. async I/O (POSIX aio) — stillborn on Linux for sockets; the idea returns - as io_uring twenty years later - -### Step 5 — what changed since 1999, and the io-threads answer - -Three assumptions expired: threads got cheaper (10K threads is fine now), -cores multiplied (one loop can't fill a 64-core box — the single-threaded -event loop went from solution to bottleneck), and NICs got faster than a -single core's syscall budget. So the question inverted: not "how does one -thread serve 10K sockets" but "how do 64 cores share one server". - -Valkey's 8.0 answer is the conservative one: keep ONE execution thread -(zero locks in the data structures, commands atomic by construction) and -parallelize only I/O. The blog posts give the measured story: redis-6-style -io-threads (spin-waiting threads, main thread coordinating every batch) -gained modestly at high CPU burn; valkey 8's rework — SPSC handoff, threads -owning the whole read→parse and write path, main-thread prefetching for -batches — claims ~1M+ ops/s/node, ~2–3× redis 7 on the same box. The -reasoning to internalize: they **profiled first** — parse+syscall was the -majority of CPU at high op rates, commands themselves ~30% — and -parallelized exactly the majority and nothing else. Amdahl's law (speedup -is capped by the serial fraction), applied with discipline. +### Step 1 — what a thread costs when it owns a connection + +> **In:** nothing but the naive design — one thread per connection, blocking +> `read()`. +> **Out:** the three costs that design pays per connection, one of them with +> Kegel's own arithmetic and one of them measured in this repo. + +Three words first, because everything below is built from them. + +A **syscall** is a call from your process into the kernel — `read`, `write`, +`epoll_wait`. It is not a function call: it traps into the kernel, which +validates arguments, may copy buffers across the user/kernel boundary, and may +put your thread to sleep. A **context switch** is the kernel taking one thread +off a core and putting another on: registers saved and restored, and — the part +that usually costs more — the cache and TLB state the incoming thread finds +cold. A **round trip** is one complete send-and-receive exchange between two +processes: request out, reply back, including both sides' syscalls and both +sides' wakeups. + +Kegel's fourth strategy, *"Serve one client with each server thread"*, is the +naive design, and he prices it in virtual memory rather than in RAM: + +> "Has the disadvantage of using a whole stack frame for each client, which +> costs memory. Many OS's also have trouble handling more than a few hundred +> threads. If each thread gets a 2MB stack (not an uncommon default value), you +> run out of *virtual memory* at (2^30 / 2^21) = 512 threads on a 32 bit machine +> with 1GB user-accessible VM." +> — Kegel, *I/O Strategies* § 4 + +That is the real 1999 constraint, and it is worth being precise about it: the +wall was **address space**, not physical memory. 2^30 bytes of user-accessible +virtual address space divided by a 2^21-byte stack is 512 threads, and the +stacks are mostly untouched — an idle connection's thread does not fault in 2 MB. +(An earlier version of this chapter said "10,000 stacks, ~80 MB at 8 KB minimum +each". That number had no source, and it also argues the wrong quantity. It is +gone; Kegel's own division replaces it.) + +The second cost is the context switch, and this chapter no longer quotes a +per-switch figure — the "~1–10 µs" it used to print was unsourced. What this +repo *can* say is measured. Topic 7's own lane does nothing at all: a 32-byte +request, an 8-byte reply, no parsing, no store, over loopback. At pipeline depth +1 it runs at **44,088 ops/s, 22.68 µs per request** +([FINDINGS.md](../../FINDINGS.md) row 7; full table in [notes.md](notes.md)). +Every microsecond of that 22.68 is syscalls, wakeups and scheduling on a machine +with no network in it. That is the honest scale for "what attention costs when +it has to move between two runnable things". + +The third cost is the one Kegel's page cannot show you because it was not yet a +problem: a thread that blocks in `read()` is *also* a thread whose work cannot be +batched with anyone else's. Hold that thought until Step 5. + +### Step 2 — Kegel's five strategies, in his order and his words + +> **In:** the per-connection costs from Step 1. +> **Out:** the actual 1999 menu — five strategies, not four, and in an order +> that is not the one this chapter used to print. + +Kegel lists exactly five, under the heading *I/O Strategies*, introduced as "The +following five combinations seem to be popular": + +1. **Serve many clients with each thread, and use nonblocking I/O and + level-triggered readiness notification.** His sub-list: `select()`, `poll()`, + `/dev/poll` (Solaris 2.7+), and **kqueue** (FreeBSD, NetBSD). +2. **Serve many clients with each thread, and use nonblocking I/O and readiness + *change* notification.** His sub-list: **epoll** (Linux 2.6+), Polyakov's + kevent, Drepper's proposal, realtime signals, signal-per-fd, and **kqueue + again**. +3. **Serve many clients with each server thread, and use asynchronous I/O.** +4. **Serve one client with each server thread** (and use blocking I/O). +5. **Build the server code into the kernel.** + +Two things about that list are worth more than the list itself. + +**First, this chapter used to get the order wrong.** It said "the fourth entry in +Kegel's menu was asynchronous I/O". Async I/O is the *third*; the fourth is +thread-per-client. The old ordering — thread-per-connection, select/poll, +readiness notification, async I/O — was a retelling in historical order, not +Kegel's taxonomy, and presenting it as his was simply wrong. It is corrected +above. + +**Second, kqueue appears in both 1 and 2, and that is the real lesson.** +Level-triggered against edge-triggered is a *mode* you select, not a property an +API has. Kegel defines both: + +> "With this scheme, the kernel tells you whether a file descriptor is ready, +> whether or not you've done anything with that file descriptor since the last +> time the kernel told you about it." +> — § 1, on level-triggered +> +> "Readiness change notification (or edge-triggered readiness notification) means +> you give the kernel a file descriptor, and later, when that descriptor +> transitions from not ready to ready, the kernel notifies you somehow. It then +> assumes you know the file descriptor is ready, and will not send any more +> readiness notifications of that type for that file descriptor until you do +> something that causes the file descriptor to no longer be ready." +> — § 2 + +And the sentence that makes the whole family make sense, from § 1: + +> "readiness notification from the kernel is only a hint; the file descriptor +> might not be ready anymore when you try to read from it." + +Two vocabulary items fall out. A **file descriptor** (fd) is the small integer +the kernel gives you to name an open socket or file. **Readiness notification** +means the kernel tells you *when you may call `read`* and you then call it; +**completion notification** — Step 4 — means you hand the kernel the read itself +and it tells you when the bytes have landed. The first still costs you a syscall +per operation; the second does not. + +What strategies 1 and 2 share, and what strategy 1's own `select`/`poll` do not, +is a kernel-side **interest list that survives between calls**. `select` and +`poll` take the whole fd array on every call, so each wakeup is O(registered): +10,000 idle connections cost 10,000 array entries copied and scanned to find the +3 that are ready. `epoll` and `kqueue` register once (`epoll_ctl`, `EV_SET` + +`kevent`) and each wait returns only the ready ones — O(ready). That is the line +that won, and every mainstream event loop is on it. + +An **event loop** is the shape those APIs imply: a single thread that blocks in +one "which of my fds are ready?" call, dispatches a callback per ready fd, and +goes round again. + +### Step 3 — which strategy the pinned redis code is (and on which machine) + +> **In:** Kegel's five strategies from Step 2. +> **Out:** the exact strategy and the exact mode redis chose, read out of +> `src/ae.c` at the pin — including which backend file compiles on the machine +> that produced this topic's numbers. + +`ae.c` picks its backend at compile time, in a nested `#ifdef` ladder with a +comment that states the intended ranking: + +```c +// redis src/ae.c — the backend ladder, 30-44 + 30 /* Include the best multiplexing layer supported by this system. + 31 * The following should be ordered by performances, descending. */ + 32 #ifdef HAVE_EVPORT + 33 #include "ae_evport.c" + 34 #else + 35 #ifdef HAVE_EPOLL + 36 #include "ae_epoll.c" + 37 #else + 38 #ifdef HAVE_KQUEUE + 39 #include "ae_kqueue.c" + 40 #else + 41 #include "ae_select.c" + 42 #endif + 43 #endif + 44 #endif +``` + +Those three macros are set by platform, not by configuration: + +```c +// redis src/config.h — "Test for polling API", 84-99 (accept4 test elided) + 84 /* Test for polling API */ + 85 #ifdef __linux__ + 86 #define HAVE_EPOLL 1 + 87 #endif + // ... 89-95: HAVE_ACCEPT4 ... + 97 #if (defined(__APPLE__) && defined(MAC_OS_10_6_DETECTED)) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined (__NetBSD__) + 98 #define HAVE_KQUEUE 1 + 99 #endif +``` + +Line 85 is the one to internalize before writing anything about `epoll` in this +repo: **`ae_epoll.c` is compiled only on Linux.** The measurements in +[notes.md](notes.md) were taken on an Apple M3 Pro, so the loop that produced +them was `ae_kqueue.c`. Anything this repo says about "how the event loop +behaves" is, on the reader's machine, a statement about kqueue. + +Now the mode. Redis registers interest like this: + +```c +// redis src/ae_kqueue.c — aeApiAddEvent, 102-111 + 102 static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) { + 103 aeApiState *state = eventLoop->apidata; + 104 struct kevent evs[2]; + 105 int nch = 0; + 106 + 107 if (mask & AE_READABLE) EV_SET(evs + nch++, fd, EVFILT_READ, EV_ADD, 0, 0, NULL); + 108 if (mask & AE_WRITABLE) EV_SET(evs + nch++, fd, EVFILT_WRITE, EV_ADD, 0, 0, NULL); + 109 + 110 return kevent(state->kqfd, evs, nch, NULL, 0, NULL); + 111 } +``` + +The flag on lines 107–108 is `EV_ADD` and **not** `EV_CLEAR`. `EV_CLEAR` is +kqueue's edge-triggered switch; without it, kqueue is level-triggered. The Linux +file makes the same choice — `ae_epoll.c:62-67` builds `ee.events` out of +`EPOLLIN`/`EPOLLOUT` and never sets `EPOLLET`; grep the file for `EPOLLET` and +you get nothing. So: + +> **Redis is Kegel's strategy 1 on both platforms — many clients per thread, +> nonblocking I/O, *level-triggered* readiness notification — not strategy 2.** + +This chapter previously implied the opposite by calling Step 3 "readiness +notification: epoll/kqueue, the line that won" and describing strategy 2's +semantics. The family is right; the mode was wrong. Level-triggered is the +forgiving mode — Kegel again: edge-triggered "is a bit less forgiving of +programming mistakes, since if you miss just one event, the connection that event +was for gets stuck forever" — and redis, which reads a bounded 16 KB per wakeup +and comes back later for the rest, *depends* on being told again. + +There is one more platform difference visible in the pinned code, and it is a +nice example of an abstraction leaking. `epoll` merges a fd's interests into one +registration (`mask |= eventLoop->events[fd].mask` at `ae_epoll.c:63`, one +`epoll_ctl` at `:67`) and returns one event per fd. kqueue registers up to two +kevents per fd (lines 104–108 above) and returns them *separately*, so `ae` has +to re-merge them: + +```c +// redis src/ae_kqueue.c — aeApiPoll's merge pass, 142-157 + 142 /* Normally we execute the read event first and then the write event. + 143 * When the barrier is set, we will do it reverse. + 144 * + 145 * However, under kqueue, read and write events would be separate + 146 * events, which would make it impossible to control the order of + 147 * reads and writes. So we store the event's mask we've got and merge + 148 * the same fd events later. */ + 149 for (j = 0; j < retval; j++) { + 150 struct kevent *e = state->events+j; + 151 int fd = e->ident; + 152 int mask = 0; + 153 + 154 if (e->filter == EVFILT_READ) mask = AE_READABLE; + 155 else if (e->filter == EVFILT_WRITE) mask = AE_WRITABLE; + 156 addEventMask(state->eventsMask, fd, mask); + 157 } +``` + +A second pass at `:162-170` walks the same array again, reads the merged mask +back out and clears it. Two O(ready) passes instead of one, on macOS only, +because redis wants to control read-before-write ordering (the `AE_BARRIER` +feature) and kqueue will not give it that for free. + +### Step 4 — the async-I/O line: stillborn, then resurrected as io_uring + +> **In:** readiness notification from Steps 2–3, which still costs one `read` +> syscall per ready socket. +> **Out:** the other half of the taxonomy — completion notification — why it +> failed in 1999, and what it does to an `ae.c`-shaped design when it returns. + +Kegel's **third** strategy is the one that lost: + +> "This has not yet become popular in Unix, probably because few operating +> systems support asynchronous I/O, also possibly because it (like nonblocking +> I/O) requires rethinking your application. Under standard Unix, asynchronous +> I/O is provided by the `aio_` interface […] AIO is normally used with +> edge-triggered completion notification, i.e. a signal is queued when the +> operation is complete." +> — § 3 + +Note his phrasing: **completion** notification, and it is orthogonal to +level/edge. You submit "read 16 KB from fd 7 into this buffer" and are told when +the bytes are *there*, rather than being told the fd is readable and then doing +the read yourself. POSIX aio never worked well for sockets on Linux, and the line +went quiet for twenty years. + +**io_uring** is that line resurrected. Two shared ring buffers, mapped into both +the process and the kernel: a submission queue you push operations onto and a +completion queue the kernel pushes results onto. Because the rings are shared +memory, N operations can be submitted and N results collected with as few as one +syscall — or zero, in polled mode. The arithmetic from Step 1 is why anyone +cares: at P=1 the topic's lane spends 22.68 µs to move 40 bytes, and essentially +all of it is the crossing, not the copying. + +What that does to an `ae.c`-shaped loop is question 4, and it is a real design +question rather than a rhetorical one: `aeApiPoll` returns *fds*, and every +handler then calls `read`/`write` itself. Under io_uring the loop would return +*finished operations*, so `readQueryFromClient` would stop being "the thing that +reads" and start being "the thing that runs after the read", and the buffer it +reads into would have to be pinned and owned by the kernel between submission and +completion. That is not a backend swap; it inverts who owns the buffer. + +### Step 5 — the assumptions that expired, and the arithmetic of one loop + +> **In:** the single-threaded event loop of Steps 2–4. +> **Out:** why one loop stopped being enough, worked two ways — a queueing +> calculation on stated assumptions, and valkey's profile-first answer with the +> numbers its authors published. + +Three of Kegel's premises expired. + +**Threads got cheaper.** Kegel saw it coming himself: "Perhaps in the +not-too-distant future, those who prefer using one thread per client will be able +to use that paradigm even for 10000 clients." His *Limits on threads* section +already puts Linux 2.6 + NPTL at "32000 or so threads" subject to +`/proc/sys/vm/max_map_count`, with the caveat that you need very small stacks +unless you are on a 64-bit processor — which everyone now is, which dissolves the +2^30-of-address-space wall from Step 1 entirely. + +**Cores multiplied**, and this is the one that matters. A single event loop is +one thread, so it can use exactly one core, and the question inverted: not "how +does one thread serve 10,000 sockets" but "how do 64 cores share one server". + +Work the saturation arithmetic before reading anyone's blog post. Model the loop +as one server in a queue. Suppose the per-request service time on the loop thread +— parse, execute, encode, the `read` and the `write` — is s = 1 µs. Then: + +``` + service rate µ = 1/s = 1 000 000 req/s ← hard ceiling, one thread + utilisation ρ = λ/µ + M/M/1 wait W = s / (1 - ρ) + + λ = 500 000 ρ = 0.50 W = 1 µs / 0.50 = 2 µs + λ = 800 000 ρ = 0.80 W = 1 µs / 0.20 = 5 µs + λ = 950 000 ρ = 0.95 W = 1 µs / 0.05 = 20 µs + λ = 990 000 ρ = 0.99 W = 1 µs / 0.01 = 100 µs +``` + +(M/M/1 — Poisson arrivals, exponential service, one server — is the wrong model +for a loop that batches, and it flatters nothing: it is a *lower* bound on how +badly the last 5% of capacity behaves. The shape is the point. Doubling the +arrival rate from 500k to 990k, still under the ceiling, multiplies waiting time +by fifty.) + +Now put the measured numbers next to it. Topic 7's lane at P=1 spends 22.68 µs +per request, which by the same formula is a ceiling of 44,088 req/s — and it is +exactly the measured figure, because the lane's server does no work at all. That +is the whole content of the P=1 row: **when s is dominated by round trips, the +"service time" you are saturating a core with is not your code**. At P=256 the +same server does 12,321,414 ops/s (row 7 again), because 256 requests now share +one round trip. Batching moved the ceiling by 279× without making a single line +of the server faster. + +**Valkey 8's answer** is the conservative one: keep one execution thread — no +locks in the data structures, commands atomic by construction — and parallelize +only I/O. The discipline worth copying is that they profiled first, and their +published profile is not what this chapter used to claim: + +- "Socket polling system calls, such as `epoll_wait`, are expensive procedures. + When executed solely by the main thread, `epoll_wait` consumes more than 20 + percent of the time." — part 1, § *High Level Design* +- After the I/O-thread rework alone, "we observed an increase in the number of + requests per second, reaching up to 780K SET commands per second. Profiling the + execution revealed that Valkey's main thread was spending more than 40% of its + time in a single function: `lookupKey`." — part 2, § *Back to Valkey* +- Prefetching the dictionary chains for a whole batch "reduces the time spent on + `lookupKey` by more than 80%"; the total impact of memory-access amortization + is "almost 50%", taking it "to more than 1.19M rps". — part 2, § *Batching and + interleaving* +- The headline: "Throughput increased by approximately 230%, rising from 360K to + 1.19M requests per second compared to **Valkey 7.2**. […] average latency + decreasing by 69.8% from 1.792 ms to 0.542 ms." Measured with "8 I/O threads, + 3M keys DB size, 512 bytes value size, and 650 clients running sequential SET + commands using AWS EC2 C7g.16xlarge". — part 1, § *Major Upgrade to Valkey + Performance* + +Three corrections come out of that list, and they are the reason to read sources +rather than summaries. This chapter used to say "commands themselves ~30%" — that +figure appears in neither post; the profiled figures are `epoll_wait` > 20% and +`lookupKey` > 40%, and both are *I/O and memory*, not command logic. It used to +say "~2–3× redis 7"; the comparison is against **valkey 7.2**, and it is +approximately 3.3× (1.19M / 360K), stated by its authors as ~230% *increase*. And +it used to attribute the whole gain to threading: 780K of it is threading, and the +step from 780K to 1.19M is the prefetcher, which is a memory-latency trick +(topic 0's territory) that only became possible *because* the I/O threads deliver +commands in batches. + +Amdahl's law is the frame: speedup is capped by the fraction you do not +parallelize. Valkey measured the fraction first and parallelized exactly it. ### Step 6 — thread-per-core: the shared-nothing endpoint -Glauber Costa's essays (Seastar/ScyllaDB, later Glommio) take the radical -position: don't share ANYTHING between cores. One reactor (event loop) per -core, connections pinned to cores, and the *data itself* **sharded by -core** — the keyspace is hash-partitioned, and a request for shard 7 -arriving on core 2 is forwarded to core 7 as a message (cross-core SPSC -again), never accessed under a lock. +> **In:** valkey's shared-keyspace answer from Step 5. +> **Out:** the opposite answer — shard the data itself by core — its two named +> implementations, and the specific costs it accepts. -The trade: no locks ⇒ no lock contention and perfect cache locality — but -also no work stealing, so a hot shard is a hot core, and tail latency now -depends on your partitioning function. The Rust incarnation of the split: -Glommio (io_uring + thread-per-core executors) never moves a task between -cores (locality, pays imbalance); tokio's work-stealing runtime moves tasks -to idle workers (evens load, pays cross-core cache traffic). +**Thread-per-core** means one thread per CPU, usually pinned, with no thread pool +and no migration. Glauber Costa's definition, from the Glommio announcement: + +> "Each core, or CPU, runs a single thread, and often (although not necessarily), +> each of these threads is pinned to a specific CPU. As the Operating System +> Scheduler cannot move these threads around, and there is never another thread +> in that same CPU, there are no context switches." +> — *Introducing Glommio, a thread-per-core crate for Rust and Linux*, +> § *What is thread-per-core?* + +That alone is not the win; the win needs **sharding**, and with it, +**shared-nothing** — an architecture in which no two threads touch the same +mutable state, so there is nothing to lock: + +> "each of the threads in the thread-per-core application becomes responsible for +> a subset of the data […] Anything is possible, so long as two threads never +> share the responsibility of handling a particular request." +> — same article, § *Using Sharding* + +Costa's own account of why locks disappear (same section) is worth reading twice: +sharding alone still needs a lock, because the OS can preempt a thread mid-update +and schedule another thread that touches the same shard. It is *thread-per-core +plus sharding* that removes the lock, because updates to two keys in one shard are +serialized by construction — they run on the same thread, one at a time. + +The two named implementations both state it plainly. Seastar, the C++ framework +under ScyllaDB, leads its home page with "Shared-nothing design: Seastar uses a +shared-nothing model that shards all requests onto individual cores" and +"Message passing: A design for sharing information between CPU cores without +time-consuming locking". DragonflyDB's README says the same in redis terms: "we +use shared-nothing architecture, which allows us to partition the keyspace of the +memory store between threads so that each thread can manage its own slice of +dictionary data. We call these slices 'shards'" — and, for multi-key commands, it +cites the VLL paper: "The choice of shared-nothing architecture and VLL allowed us +to compose atomic multi-key operations without using mutexes or spinlocks." + +The trade is real and this chapter should not soften it. No locks means no lock +contention and excellent cache locality; it also means **no work stealing**, so a +hot shard is a hot core and your tail latency is now a property of your +partitioning function. The contrast in Rust is exact: Glommio is "a Cooperative +Thread-per-Core crate for Rust & Linux based on `io_uring`" that "doesn't use +helper threads anywhere" (its README), while tokio's multi-thread runtime steals: + +> "Each processor maintains its own run queue. Tasks that become runnable are +> pushed onto the current processor's run queue and processors drain their local +> run queue. However, when a processor becomes idle, it checks sibling processor +> run queues and attempts to steal from them. […] Under load, processors operate +> independently, avoiding synchronization overhead. In cases where the load is +> not evenly distributed across processors, the scheduler is able to +> redistribute." +> — tokio blog, *Making the Tokio Scheduler 10x Faster*, § *Work-stealing +> scheduler* + +Evens the load, pays cross-core synchronization. Costa's model never pays the +synchronization and never evens the load. Neither is free. ``` shared keyspace ◄──────────────────────► sharded keyspace - redis/valkey: 1 exec thread DragonflyDB/Scylla: N exec threads, - + io threads, zero data locks keyspace hash-partitioned per core, - cross-shard ops = messages/transactions + redis / valkey 8: DragonflyDB / ScyllaDB: + ONE execution thread, N execution threads, keyspace + N I/O threads, partitioned per core; multi-key + zero data locks ops = VLL transactions, no mutexes + ▲ ▲ + ae.c + io_threads.c Seastar / helio + io_uring ``` -This is the plane M7 positions itself in: shared↔sharded on one axis, -loop↔threads on the other. +This is the plane M7 has to position itself in: shared↔sharded on one axis, +one-loop↔many-threads on the other. FalkorDB sits at the top-left — redis's model, +one execution thread, a graph as one keyspace entry — which is why its concurrency +story is module-level locking rather than partitioning. ## How to read the three resources (with the concepts in hand) -- **Kegel, "The C10K problem"** (kegel.com/c10k.html) — read as history - that explains present defaults. Skim the I/O-strategy section against - Steps 1–4's menu; skip the dated driver patches entirely. -- **Valkey's 8.0 blog series** (valkey.io/blog) — read *after* - reading-valkey-iothreads.md; the posts supply the measurements behind - Step 5. Watch for the profile-first discipline: which numbers justified - parallelizing parse+I/O and nothing else. -- **Glauber Costa's thread-per-core essays** ("The reactor pattern is - dead, long live the reactor"; the shard-per-core posts; later Glommio - writing) — read for Step 6's position and its honest costs. Keep asking - M7's question: what is the sharding unit for a *graph*? +- **Kegel, "The C10K problem"** (kegel.com/c10k.html) — read the opening + arithmetic and the five *I/O Strategies* sections, and nothing else. Check each + strategy against Step 2's list, then skip *LinuxThreads*, *NGPT*, *NPTL*, the + per-OS notes and the driver patches entirely: they are 2006 kernel trivia. + *Limits on threads* is worth two minutes for the `max_map_count` note. Read it + as an artefact — its "current" is twenty years stale, and noticing which of its + premises expired (Step 5) is most of its value. +- **Valkey's 8.0 posts** — "Unlock 1 Million RPS: Experience Triple the Speed + with Valkey" (2024-08-05) and its part 2 (2024-09-13), on valkey.io/blog. Read + after [reading-valkey-iothreads.md](reading-valkey-iothreads.md), because the + posts describe the code that guide reads. Copy down every number *with its + configuration attached* — the 1.19M is 8 I/O threads on a c7g.16xlarge with + 512-byte values and 650 clients running sequential SET, and quoting it without + that is how "~2–3× redis 7" got into this file. +- **Glauber Costa on thread-per-core** — the Glommio announcement on the Datadog + engineering blog is the readable entry point; seastar.io's home page and + DragonflyDB's README give the same design in two other codebases' words. Read + for Step 6's position *and* its costs, and keep asking M7's question: what is + the sharding unit for a *graph*? ## Questions to answer in notes.md -1. Which C10K strategy is tokio's multi-thread runtime? (Careful: it's - readiness-based mio underneath + work-stealing tasks on top — two - layers, name both.) -2. A graph database's unit of work is a *query* (ms-scale), not a GET - (µs-scale). Redo valkey's Amdahl analysis for M7: what fraction of a - GRAPH.QUERY round-trip is parse+I/O, and does ANY threading of the - network layer matter? Where do the threads belong instead (M9)? -3. Thread-per-core for a graph: matrices don't hash-partition like a - keyspace. What's the sharding unit — graph? subgraph? matrix tile? What - does a BFS crossing shards cost in messages? -4. io_uring (the C10K "async I/O" line resurrected): what changes in ae.c's - design if poll+read+write become submission-queue entries? (Topic 6's - O_DIRECT thread rejoins here.) +1. Which C10K strategy is tokio's multi-thread runtime? Careful — it is two + layers, and they are different answers: name what mio does underneath and what + the scheduler does on top. +2. Redis is level-triggered (Step 3). Sketch what would have to change in + `readQueryFromClient` if `ae` registered with `EV_CLEAR`/`EPOLLET` instead. + Which of redis's existing behaviours — the 16 KB bounded read especially — + becomes a bug? +3. A graph database's unit of work is a *query* (ms-scale), not a GET (µs-scale). + Redo valkey's Amdahl analysis for M7: if execution is 1 ms and parse+I/O is + 20 µs, what is the ceiling on any amount of network threading? Where do the + threads belong instead (M9)? +4. Thread-per-core for a graph: matrices do not hash-partition like a keyspace. + What is the sharding unit — graph, subgraph, matrix tile? What does one BFS + frontier step crossing shards cost in messages? +5. io_uring (Step 4): what changes in `ae.c`'s design if poll+read+write become + submission-queue entries? Who owns the read buffer between submission and + completion, and what does that do to `querybuf` reallocation? (Topic 6's + `O_DIRECT` thread rejoins here.) ## Done when -You can place redis, valkey 8, tokio, and DragonflyDB in the -shared↔sharded / loop↔threads plane and argue M7's position in it. +Answer each before unfolding it. + +- [ ] You can list Kegel's five I/O strategies in his order, and say which one + the pinned redis code implements. + +
Answer + + (1) many clients per thread + nonblocking I/O + level-triggered readiness; + (2) many clients per thread + nonblocking I/O + readiness *change* + (edge-triggered) notification; (3) many clients per server thread + + asynchronous I/O; (4) one client per server thread with blocking I/O; + (5) server code in the kernel. + + Redis is **strategy 1** — level-triggered readiness — on both platforms. + `ae_kqueue.c:107-108` registers with `EV_ADD` and no `EV_CLEAR`; + `ae_epoll.c:62-67` builds `ee.events` from `EPOLLIN`/`EPOLLOUT` and never sets + `EPOLLET`. Being in strategy 1 alongside `select` does not make it O(n): the + interest list persists across calls, which is the property that matters, and + which `select`/`poll` lack. + +
+ +- [ ] You can say which multiplexing backend compiled on the machine that + produced this topic's measurements, and prove it from the source. + +
Answer + + `ae_kqueue.c`. `config.h:85` guards `HAVE_EPOLL` with `#ifdef __linux__`, and + `config.h:97` guards `HAVE_KQUEUE` with `__APPLE__ && MAC_OS_10_6_DETECTED` (or + a BSD). The ladder at `ae.c:32-44` therefore falls through `HAVE_EVPORT` and + `HAVE_EPOLL` to `#include "ae_kqueue.c"` on the Apple M3 Pro of + [notes.md](notes.md). Consequence: on that machine `aeApiPoll` runs the + two-pass read/write merge at `ae_kqueue.c:149-170`, which has no counterpart in + the epoll backend, because kqueue delivers `EVFILT_READ` and `EVFILT_WRITE` as + separate events. + +
+ +- [ ] You can explain readiness notification against completion notification + without using the word "async", and say which syscalls each costs you. + +
Answer + + Readiness: you tell the kernel which descriptors you care about; it tells you + which are *ready*; you then perform the `read`/`write` yourself. Cost: one wait + syscall per wakeup plus one I/O syscall per ready descriptor — and Kegel's + warning applies, the readiness is "only a hint", the fd may not be ready by the + time you act, so the fd must be nonblocking. + + Completion: you hand the kernel the operation *and its buffer*; it tells you + when the bytes have moved. Cost: with io_uring's shared submission/completion + rings, N operations can cost one syscall, or zero in polled mode. The price is + ownership — the buffer belongs to the kernel until the completion arrives, so + you cannot resize or free it in between. + +
+ +- [ ] Given a per-request service time, you can compute the arrival rate that + saturates one event-loop thread, and say what happens just below it. + +
Answer + + µ = 1/s. At s = 1 µs the ceiling is 1,000,000 req/s. Below it, waiting time + grows as W = s/(1−ρ): 2 µs at λ=500k, 5 µs at λ=800k, 20 µs at λ=950k, 100 µs + at λ=990k. The ceiling is not where the trouble starts — a 2× rise in arrivals + from 500k to 990k is a 50× rise in queueing delay. + + Then the humbling version: this topic's lane at P=1 has s ≈ 22.68 µs, so its + ceiling is 1/22.68 µs = 44,088 req/s — the measured number + ([FINDINGS.md](../../FINDINGS.md) row 7) — for a server that executes *nothing*. + Its service time is round trips. Batching at P=256 amortizes one round trip over + 256 requests and the same server reaches 12,321,414 ops/s. + +
+ +- [ ] You can state valkey 8.0's published numbers with the configuration + attached, and say which part of the gain is not threading. + +
Answer + + 360K → 1.19M rps, ~230% increase, versus **valkey 7.2** (not redis 7): 8 I/O + threads, 3M keys, 512-byte values, 650 clients, sequential SET, AWS EC2 + c7g.16xlarge; average latency 1.792 ms → 0.542 ms (−69.8%). Part 1, + § *Major Upgrade to Valkey Performance*. + + The I/O-thread rework alone reached 780K (part 2, § *Back to Valkey*). The step + from 780K to 1.19M is the **prefetcher**: `lookupKey` was over 40% of main-thread + time, `dictPrefetch` interleaves the hash-chain walks for a whole batch and cuts + it by more than 80%, worth "almost 50%" overall. That is a memory-latency win, + not a concurrency win — and it is only available because the I/O threads hand + over *batches*. + +
+ +- [ ] You can place redis, valkey 8, tokio and DragonflyDB on the + shared↔sharded / one-loop↔many-threads plane, and argue M7's position. + +
Answer + + Redis ≤5 and M7 v1: shared keyspace, one loop — everything serialized, no locks + needed because there is one thread. Valkey 8: shared keyspace, one *execution* + thread plus N I/O threads — commands still serialized, so still no data locks; + I/O and parsing parallel. Tokio's multi-thread runtime: shared state, N + work-stealing workers — so anything shared needs a lock or a channel, and tasks + migrate between cores. DragonflyDB / ScyllaDB: sharded keyspace, + thread-per-core, no cross-core locks at all; multi-key atomicity comes from a + transaction protocol (VLL) instead of mutexes. + + M7's honest position is valkey's, one step back: one execution thread, because a + graph is one keyspace entry and the executor is the expensive part. The + thread-per-core question is deferred to a real answer for question 4 — what a + graph shard *is* — and until that exists, sharding buys contention, not + throughput. + +
## References -**Papers** -- Dan Kegel — "The C10K problem" (kegel.com/c10k.html, 1999–2003) — skim - the I/O-strategy section, skip the dated driver patches -- Valkey blog — the 8.0 performance/multithreading series - (valkey.io/blog) — read after - [reading-valkey-iothreads.md](reading-valkey-iothreads.md) for the - measured story -- Glauber Costa — thread-per-core essays ("The reactor pattern is dead, - long live the reactor"; the Seastar/ScyllaDB shard-per-core posts and - later Glommio writing) +**Primary sources (web documents, cited by section)** +- Dan Kegel — "The C10K problem", kegel.com/c10k.html. Written 1999; RCS log ends + at *Revision 1.212, 2006/09/02*; last changelog entry *2011/07/21*. Sections + used here: the opening cost arithmetic; *I/O Strategies* §§ 1–4; *Limits on + threads*. +- Dan Touitou & Uri Yagelnik — "Unlock 1 Million RPS: Experience Triple the Speed + with Valkey", valkey.io/blog, 2024-08-05. Sections used: *Major Upgrade to + Valkey Performance* (360K → 1.19M, −69.8% latency, test configuration); + *High Level Design* (`epoll_wait` > 20%, one poller at a time). +- Dan Touitou & Uri Yagelnik — "Unlock 1 Million RPS … part 2", valkey.io/blog, + 2024-09-13. Sections used: *Speculative execution and linked lists* (20.8 s → + <2 s → 1.8 s on Graviton 3; external memory ≈ 50× L1); *Back to Valkey* (780K, + `lookupKey` > 40%); *Batching and interleaving* (>80%, ~50%, 1.19M). +- Glauber Costa — "Introducing Glommio, a thread-per-core crate for Rust and + Linux", Datadog engineering blog. Sections used: *What is thread-per-core?*, + *Using Sharding*. (An earlier version of this chapter cited an essay titled + "The reactor pattern is dead, long live the reactor". That title could not be + verified against any primary source and has been removed.) +- Glommio README, github.com/DataDog/glommio — § *What is Glommio?* +- seastar.io home page — § *Shared-nothing design*, § *Message passing*. +- DragonflyDB README, github.com/dragonflydb/dragonfly — the shared-nothing and + VLL paragraphs. +- "VLL: a lock manager redesign for main memory database systems" (VLDB Journal; + DragonflyDB links `cs.umd.edu/~abadi/papers/vldbj-vll.pdf`) — the paper cited + for multi-key atomicity without mutexes. Not read for this chapter; listed + because Dragonfly's claim rests on it. +- tokio blog — "Making the Tokio Scheduler 10x Faster", § *Work-stealing + scheduler*. + +**Code, at this repo's pins** (`tools/pinned-source.py ref redis` → `a176d1225`) +- `src/ae.c:30-44` — the backend ladder. +- `src/config.h:84-99` — `HAVE_EPOLL` is `__linux__`; `HAVE_KQUEUE` is Apple/BSD. +- `src/ae_kqueue.c:102-111` — `EV_ADD`, no `EV_CLEAR`: level-triggered. +- `src/ae_kqueue.c:142-170` — the two-pass read/write merge kqueue forces. +- `src/ae_epoll.c:54-69` — one merged registration per fd; no `EPOLLET`. +- Read next: [reading-redis-ae-networking.md](reading-redis-ae-networking.md) for + the loop itself, [reading-valkey-iothreads.md](reading-valkey-iothreads.md) for + the code behind Step 5's numbers. + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 7 — 44,088 ops/s at P=1, 12,321,414 at + P=256, 279×. Full table and method in [notes.md](notes.md); the lane is + `experiments/src/bin/loopback_bench.rs`. diff --git a/topics/07-networking-protocols/reading-pgwire-qdrant.md b/topics/07-networking-protocols/reading-pgwire-qdrant.md index fb37993..77de877 100644 --- a/topics/07-networking-protocols/reading-pgwire-qdrant.md +++ b/topics/07-networking-protocols/reading-pgwire-qdrant.md @@ -1,183 +1,727 @@ # pgwire & tonic: sessions, portals, and protocols you don't write Two contrasts with RESP: a protocol with *stateful sessions and streaming* -(postgres wire, via the pgwire Rust crate), and a protocol you don't write at -all (gRPC, via qdrant's tonic setup). Together they bracket RESP's design -point — no handshake, no cursors, buffer-or-die — and fill in the -design-space table M7 has to take a position on. This chapter builds the -concepts step by step — protocol as state machine, framing, the two postgres -query modes, portals, sessions, and generated protocols — then maps them to -the two codebases. +(the postgres wire protocol, read through the `pgwire` Rust crate), and a +protocol you don't write at all (gRPC, read through qdrant's tonic setup). +Together they bracket RESP's design point — no handshake, no cursors, +buffer-or-die — and fill in the design-space table M7 has to take a position on. +This chapter builds the concepts step by step — framing, state machine, the two +query modes, portals, sessions, generated protocols — and checks each against the +pinned source, which corrects several claims an earlier version of this chapter +made from memory. + +**Which versions this chapter is about.** Both trees are pinned, and the +anchors below were re-checked against those pins, not against whatever is on +`main` today: + +```sh +tools/pinned-source.py ref pgwire # sunng87/pgwire@6bb6299 +tools/pinned-source.py ref qdrant # qdrant/qdrant@44ad62f +tools/pinned-source.py show pgwire src/api/query.rs -r 599:637 +tools/pinned-source.py check qdrant lib/api/src/grpc/qdrant.rs:1 --contains 'generated by prost-build' +``` + +Two path corrections fall out immediately, because the earlier version of this +chapter cited files that do not exist at these pins: pgwire's startup handler is +in **`src/api/auth/mod.rs`**, not `src/api/auth.rs` (`auth` is a directory of +seven files), and qdrant's generated protocol crate is **`lib/api/`**, not +`api/`. There are also no local clones at `~/repos/pgwire` or `~/repos/qdrant`; +`tools/pinned-source.py` fetches from the pin. ## The problem in one sentence -A query returning 10M rows must not require the server to hold 10M rows in -a reply buffer (RESP's answer) — postgres's protocol lets the client pull -1,000 rows at a time from a suspended query, and gRPC inherits flow control -from HTTP/2; both bake into the *protocol* the backpressure RESP doesn't -have. +A query returning 10M rows must not require the server to hold 10M rows in a +reply buffer — RESP's only answer — and postgres's protocol says so structurally: +`Execute` carries a `max_rows` count, the server replies `PortalSuspended` and +*stops producing* (pgwire `src/api/query.rs:351-354`), and the client decides +whether to ask for more. ## The concepts, step by step -### Step 1 — a wire protocol is a state machine plus a framing rule +### Step 1 — a wire protocol is a framing rule plus a state machine + +> **In:** RESP, in which any command is legal at any time and every reply is +> complete. +> **Out:** the two questions every wire protocol answers, and postgres's answer +> to the second one read out of pgwire's connection-state enum. + +**Framing** is how the receiver knows where one message ends and the next +begins. A **state machine** is which messages are legal *now*. RESP answers the +first with in-band ASCII lengths and the second with "all of them, always" — a +RESP connection has no state you can name. + +Postgres does have states, and pgwire spells them out: + +```rust +// pgwire src/api/mod.rs — the connection lifecycle, 46-64 + 46 /// States of a PostgreSQL connection lifecycle. + 47 #[derive(Debug, Clone, Copy, Default)] + 48 pub enum PgWireConnectionState { + 49 /// Waiting for an SSL request from the client. + 50 #[default] + 51 AwaitingSslRequest, + 52 /// Waiting for a startup message from the client. + 53 AwaitingStartup, + 54 /// Authentication handshake in progress. + 55 AuthenticationInProgress, + 56 /// Connection is idle and ready for queries. + 57 ReadyForQuery, + 58 /// A query is currently being executed. + 59 QueryInProgress, + 60 /// A COPY operation is in progress. + 61 CopyInProgress(bool), + 62 /// Waiting for a Sync message from the client. + 63 AwaitingSync, + 64 } +``` + +Seven states, and the interesting one is the last. `AwaitingSync` is *error +recovery as a protocol state*, and pgwire's dispatcher quotes the postgres manual +for why it exists: + +```rust +// pgwire src/tokio/server.rs — the dispatch on connection state, 198-214 + 198 match socket.state() { + 199 PgWireConnectionState::AwaitingStartup + 200 | PgWireConnectionState::AuthenticationInProgress => { + 201 authenticator.on_startup(socket, message).await?; + 202 } + 203 // From Postgres docs: + 204 // When an error is detected while processing any extended-query + 205 // message, the backend issues ErrorResponse, then reads and discards + 206 // messages until a Sync is reached, then issues ReadyForQuery and + 207 // returns to normal message processing. + 208 PgWireConnectionState::AwaitingSync => { + 209 if let PgWireFrontendMessage::Sync(sync) = message { + 210 extended_query_handler.on_sync(socket, sync).await?; + 211 // TODO: confirm if we need to track transaction state there + 212 socket.set_state(PgWireConnectionState::ReadyForQuery); + 213 } + 214 } +``` -A wire protocol answers two questions: how does the receiver know where one -message ends (**framing**), and what messages are legal *now* (the **state -machine**). RESP's state machine is trivial — any command, any time, one -reply each. Postgres's is not: a connection moves through startup → auth → -ready → (parse → bind → execute)* → sync, and several messages are only -legal in certain states. Reading pgwire, keep asking: *where does session -state live?* — the crate forces a `ClientInfo` parameter through every -handler call; your RESP server keeps per-connection state implicitly in the -tokio task. Both are answers to "protocol = state machine". +Hold on to lines 203–207, because they turn up again in Step 4: **discard until +Sync** is precisely what makes it safe to send a batch of extended-query messages +without waiting for each reply. RESP has no equivalent rule, which is why a +pipelined RESP batch keeps executing after one of its commands fails. -### Step 2 — postgres framing: one type byte + a binary length +Where does session state live? pgwire threads a `ClientInfo` through every +handler call, and `src/api/mod.rs:66-70` adds a "per-connection typed extension +store, keyed by `TypeId`" for handlers to hang their own state on. Your RESP +server keeps the equivalent implicitly, in the local variables of the tokio task +serving the connection. Both are answers to "protocol = state machine"; only one +of them is written down. -Every postgres message is framed as 1 ASCII type byte + a 4-byte -big-endian i32 length + payload. Like RESP it dispatches on a leading type -byte; unlike RESP the length is *binary* (RESP spells lengths in ASCII -digits terminated by CRLF). Fixed 5-byte headers mean the reader never -scans: read 5 bytes, learn the size, read exactly that many. +### Step 2 — postgres framing: one type byte + a binary length, and resumable -In pgwire this is `src/messages/` — every frontend/backend message as a -typed struct with encode/decode. The crate structure IS the protocol -lesson: one module per message family, one type per message. +> **In:** the need for framing from Step 1. +> **Out:** postgres's 5-byte header, exactly what its length field counts, and +> the partial-message behaviour — worked on real byte counts against RESP. + +Every postgres message after the startup packet is 1 ASCII type byte, a 4-byte +big-endian `i32` length, then the body. pgwire's `Message` trait writes it in one +place: + +```rust +// pgwire src/messages/mod.rs — Message::encode, 121-136 + 121 fn encode(&self, buf: &mut BytesMut) -> PgWireResult<()> { + 122 if let Some(mt) = Self::message_type() { + 123 buf.put_u8(mt); + 124 } + 125 + 126 let len = self.message_length(); + 127 if len > Self::max_message_length() { + 128 return Err(PgWireError::MessageTooLarge( + 129 Self::max_message_length(), + 130 len, + 131 )); + 132 } + 133 + 134 buf.put_i32(len as i32); + 135 self.encode_body(buf) + 136 } +``` + +And reads it in one place, where the exact meaning of the length field is +visible: + +```rust +// pgwire src/messages/codec.rs — decode_packet, 63-84 (signature elided) + 63 pub(crate) fn decode_packet( + // ... 64-71: buf, offset, max_size, decode_fn and their bounds ... + 72 if let Some(msg_len) = get_length(buf, offset) { + 73 if msg_len > max_size { + 74 return Err(PgWireError::MessageTooLarge(max_size, msg_len)); + 75 } + 76 + 77 if buf.remaining() >= msg_len + offset { + 78 buf.advance(offset + 4); + 79 return decode_fn(buf, msg_len).map(|r| Some(r)); + 80 } + 81 } + 82 + 83 Ok(None) + 84 } +``` + +`offset` is 1 when the message has a type byte. Line 77 needs `msg_len + offset` +bytes and line 78 skips `offset + 4`, so **the length counts itself and the body +but not the type byte**. Line 83 is the other half: if the whole message has not +arrived, return `Ok(None)` and leave the buffer untouched — the decoder is +resumable, and `get_length` (`codec.rs:53-59`) deliberately reads the length +*without* advancing the cursor. + +Now the arithmetic, because "fixed header, no scanning" is usually oversold. A +simple `Query` for `SELECT 1` — `message_length` is `5 + query.len()` +(`simplequery.rs:22-24`), the 5 being 4 length bytes plus the trailing NUL: + +``` + 'Q' type byte, not counted in the length 1 byte + 00 00 00 0D length = 5 + 8 = 13 4 bytes + "SELECT 1" 00 cstring body 9 bytes + on the wire = 14 bytes +``` + +A three-column result row `("alice", "30", "NYC")` as a `DataRow` — +`message_length` is `4 + 2 + data.len()` (`data.rs:174-176`), and each field +inside `data` is a 4-byte length followed by the bytes (`api/results.rs:319` +writes `-1` as a placeholder, `:328-330` back-patches the real length): + +``` + 'D' 1 byte + 00 00 00 1C length = 4 + 2 + 22 = 28 4 bytes + 00 03 field count 2 bytes + 00 00 00 05 "alice" 9 bytes + 00 00 00 02 "30" 6 bytes + 00 00 00 03 "NYC" 7 bytes + on the wire = 29 bytes + + the same row in RESP2: + *3\r\n $5\r\nalice\r\n $2\r\n30\r\n $3\r\nNYC\r\n + 4 + 11 + 8 + 9 = 32 bytes +``` + +**29 against 32 — under 10%.** The binary framing is not where postgres wins; +what it buys is that the reader never scans for a delimiter and never parses +ASCII digits, and that a NULL is a 4-byte `-1` inside the row rather than RESP2's +separate `$-1\r\n` form. The wins that matter are in Steps 3–5, and they are +structural, not byte-level. + +One detail worth carrying to M7: pgwire caps message sizes by *type*. +`SMALL_PACKET_SIZE_LIMIT` is 10,000 bytes (`messages/mod.rs:52`) and is the +default `max_message_length` for a frontend message; `Query` overrides it to +`LARGE_PACKET_SIZE_LIMIT`, `0x3fffffff - 1` (`simplequery.rs:27-29`, +`mod.rs:51`), and `DataRow` overrides it to `i32::MAX` (`data.rs:170-172`). A +tiny per-message budget by default, raised deliberately where large payloads are +legitimate — compare redis's single global `proto-max-bulk-len`. ### Step 3 — the simple query protocol: RESP with row framing -`SimpleQueryHandler` (src/api/query.rs:48) is the RESP-like mode: one -`Query` message carrying a SQL string in; a stream of messages out — -`RowDescription` (column names/types), then zero or more `DataRow`s, then -`CommandComplete`. One request, one complete response, no state left -behind. Note what's already better than RESP for a database: rows are -individually framed messages, so the server can *write them as it produces -them* instead of materializing the whole result first. +> **In:** the framing from Step 2. +> **Out:** the RESP-shaped mode of postgres, and the one thing it already does +> better — writing rows as it produces them. + +`SimpleQueryHandler` (`src/api/query.rs:48`) is the familiar shape: one `Query` +message carrying a SQL string, and a stream of messages back — `RowDescription`, +zero or more `DataRow`s, `CommandComplete`. What it does with them is the part +worth reading: + +```rust +// pgwire src/api/query.rs — send_query_response, 568-596 (destructuring elided) + 568 let QueryResponse { + // ... 569-572: command_tag, row_schema, mut data_rows ... + 576 if send_describe { + 577 let row_desc = into_row_description(&row_schema); + 578 client + 579 .send(PgWireBackendMessage::RowDescription(row_desc)) + 580 .await?; + 581 } + 582 + 583 let mut rows = 0; + 584 while let Some(row) = data_rows.next().await { + 585 let row = row?; + 586 rows += 1; + 587 client.feed(PgWireBackendMessage::DataRow(row)).await?; + 588 } + 589 + 590 let tag = Tag::new(&command_tag).with_rows(rows); + 591 client + 592 .send(PgWireBackendMessage::CommandComplete(tag.into())) + 593 .await?; +``` + +Two things. First, `data_rows` is a **stream** — line 584 awaits each row from +the executor, so rows are encoded and handed to the socket as they are produced, +never materialized as a whole result. Second, look at `feed` on line 587 against +`send` on lines 579 and 591: in the futures `Sink` API, `feed` buffers and `send` +buffers *and flushes*. So a thousand-row response is a thousand buffered +`DataRow`s and exactly one flush, at `CommandComplete`. That is the identical +discipline redis's reply buffer implements by hand +([reading-redis-ae-networking.md](reading-redis-ae-networking.md), the write +path) and that this topic's own lane implements as "flush only when drained" +([notes.md](notes.md)) — three codebases, one rule: **never let a syscall +boundary follow a logical record boundary.** + +Third, and easy to miss: `_on_query` loops over `Vec` +(`query.rs:102-121`), because one simple-query message may carry several +semicolon-separated statements. The simple protocol's "one request, one response" +is really "one request, one *stream* of responses". ### Step 4 — the extended query protocol: portals are protocol-level backpressure -`ExtendedQueryHandler` (src/api/query.rs:174) splits "run this SQL" into -five messages — Parse → Bind → Execute → Sync — and that decomposition is -where the power lives: - -- **Parse**: compile SQL into a named **prepared statement** (a compiled - query kept server-side, reusable with different parameters). -- **Bind**: attach concrete parameter values to a statement, producing a - **portal** — a suspended, partially-executed query the server holds. -- **Execute {max_rows}**: pull up to max_rows rows *from* the portal. Not - done? The server replies `PortalSuspended` and *stops producing* — the - client decides when (whether) to pull more. This is backpressure and - cursoring in the protocol itself; RESP has neither (a module either - buffers the whole reply or blocks the loop). -- **Sync**: close out the sequence, recover from errors, get - `ReadyForQuery`. +> **In:** the simple protocol from Step 3, which streams but cannot be paused. +> **Out:** the five-message decomposition, the suspend/resume mechanism read out +> of `_on_execute`, and what the extra messages cost in round trips — worked on +> this topic's measured numbers. + +`ExtendedQueryHandler` (`src/api/query.rs:174`) splits "run this SQL" into +**five** frontend messages, not the four an earlier version of this chapter +listed: **Parse → Bind → Describe → Execute → Sync** (`Describe` is optional in +practice but is a first-class message with its own handler at `query.rs:374`, and +Step 3's `send_describe` flag exists precisely because "not all `Execute` comes +with `Describe`" — `query.rs:555-557`). `Flush` and `Close` are two more. + +- **Parse** compiles SQL into a named **prepared statement** — a parsed, + server-held query, reusable with different parameters. `on_parse` + (`query.rs:185-200`) runs the crate's `QueryParser` and stores the result in the + portal store. +- **Bind** attaches concrete parameter values to a statement, producing a + **portal** — a named, executable instance of that statement. `on_bind` + (`query.rs:206-225`) errors with `StatementNotFound` if you bind a name that + was never parsed: the state machine, enforced. +- **Execute {max_rows}** pulls up to `max_rows` rows *from* the portal. +- **Sync** ends the sequence, drops the unnamed portal and returns + `ReadyForQuery` (`on_sync`, `query.rs:436-452`). + +The suspend/resume is the whole point, and it is eleven lines: ```rust -// The protocol IS a session state machine; portals are protocol-level -// backpressure — a suspended query the client pulls N rows at a time. -match msg { - Parse { name, sql } => { self.stmts.insert(name, prepare(sql)?); } - Bind { portal, stmt, args } => { self.portals.insert(portal, cursor(stmt, args)?); } - Execute { portal, max_rows } => { - let cur = self.portals.get_mut(&portal)?; - for row in cur.take(max_rows) { send(DataRow(row))?; } - if cur.done() { send(CommandComplete)?; } - else { send(PortalSuspended)?; } // client decides when to pull more - } - Sync => { self.close_txn_if_failed(); send(ReadyForQuery)?; } - _ => { /* Describe, Close, Flush … */ } -} -``` - -Cost: five messages instead of one — a round-trip each unless pipelined -(question 3). Buy: prepared statements, parameter binding, binary result -formats, and results in client-sized bites. +// pgwire src/api/query.rs — the fetch half of _on_execute, 341-361 + 341 // Fetch rows from the portal and send to client + 342 if needs_fetch { + 343 let fetch_result = portal.fetch(max_rows).await?; + 344 let mut response = fetch_result.response; + 345 let command_tag = response.command_tag().to_owned(); + 346 let mut row_count = 0; + 347 while let Some(row) = response.data_rows().next().await { + 348 client.feed(PgWireBackendMessage::DataRow(row?)).await?; + 349 row_count += 1; + 350 } + 351 if fetch_result.suspended { + 352 client + 353 .send(PgWireBackendMessage::PortalSuspended(PortalSuspended)) + 354 .await?; + 355 } else { + 356 let tag = Tag::new(&command_tag).with_rows(row_count); + 357 client + 358 .send(PgWireBackendMessage::CommandComplete(tag.into())) + 359 .await?; + 360 } + 361 } +``` + +Line 351 is the branch RESP does not have. `PortalSuspended` means "there are +more rows and I have stopped producing them"; the query's state stays on the +server and the *client* decides whether to spend more. The portal is only started +once — `_on_execute` checks `PortalExecutionState::Initial` at `:271-274` and +runs `do_query` only then; a second `Execute` on the same portal falls to the +`else` at `:336-339` and goes straight to the fetch. The public helper +`send_partial_query_response` (`:599-637`) implements the same loop for handlers +that want it directly, and it documents the protocol's zero convention: +`while max_rows == 0 || rows < max_rows` at `:614` — **`max_rows = 0` means no +limit**. + +Now the cost, worked. A **round trip** is one send-and-wait-for-reply exchange. +Five messages sent one at a time, each awaited, is five round trips. This topic's +lane measures a loopback round trip at **22.68 µs** at pipeline depth 1 +([FINDINGS.md](../../FINDINGS.md) row 7, table in [notes.md](notes.md)): + +``` + unpipelined, loopback: 5 × 22.68 µs = 113.4 µs → 8 818 queries/s + pipelined , loopback: 1 × 22.68 µs = 22.68 µs → 44 088 queries/s (5.0×) + + unpipelined, 100 µs network RTT: 5 × 100 µs = 500 µs → 2 000 queries/s + pipelined , 100 µs network RTT: 1 × 100 µs = 100 µs → 10 000 queries/s (5.0×) +``` + +The server code is identical in both rows. **Pipelining** — writing the next +request before the previous reply arrives — is what collapses the 5 into 1, and +Step 1's discard-until-`Sync` rule is what makes it safe: if `Parse` fails, the +backend throws away `Bind`, `Describe` and `Execute` unread rather than executing +half a batch. That is the design pgwire's `AwaitingSync` state implements at +`src/tokio/server.rs:208-214`. + +So the extended protocol's price is not really five round trips; it is five +*messages*, and a client that batches them pays one. What it buys is prepared +statements, typed parameter binding, binary result formats, and results in +client-sized bites. ### Step 5 — sessions from byte 0: the startup handshake -A postgres connection is a state machine *before the first query*: -`StartupHandler` (src/api/auth.rs; see api/mod.rs:555) processes startup -parameters (user, database, options), runs an auth exchange (possibly -multi-round-trip: cleartext, MD5, SCRAM), and only then reports -ready-for-query. RESP connections have no handshake at all (HELLO is -optional). Count what the handshake costs postgres — connection setup -latency, hence everyone runs connection pools — and what it buys: -per-session GUCs, transaction state, and cancel keys (a token another -connection can use to cancel your running query). +> **In:** a fresh TCP connection. +> **Out:** what postgres does before the first query is legal, what that costs, +> and what it buys that RESP cannot express. + +A postgres connection is a state machine before any query. Step 1's enum starts +at `AwaitingSslRequest` — the very first exchange is not even a postgres message +but an 8-byte "may I use TLS?" probe with **no type byte at all**: a 4-byte +length of 8 followed by the magic number 80877103 (`src/messages/startup.rs:570`, +`:587-593`) — then `AwaitingStartup`, then `AuthenticationInProgress`. +`StartupHandler` (`src/api/auth/mod.rs:22`, selected per server by +`src/api/mod.rs:555`) drives the rest: startup parameters (user, database, +options), then an authentication exchange, and only then `ReadyForQuery`. + +The auth exchange is where "possibly multi-round-trip" stops being a hedge: +pgwire ships `cleartext.rs`, `md5pass.rs`, `noop.rs`, `sasl.rs`, +`sasl/scram.rs` and `sasl/oauth.rs` under `src/api/auth/`, and SCRAM is a +*challenge-response* protocol — client-first, server-first, client-final, +server-final. Count the round trips before the first byte of SQL: TLS probe, +TLS handshake, startup packet, and 2–4 more for SCRAM. At the 100 µs RTT of +Step 4's arithmetic that is roughly half a millisecond of connection setup; at a +1 ms cross-AZ RTT it is 5 ms. **That is why every postgres deployment runs a +connection pool** and no redis deployment needs one — the handshake is the reason +pgbouncer exists. + +What it buys, and RESP cannot express: per-session parameters (`ServerParameterProvider`, +`src/api/auth/mod.rs:36`), transaction status carried on every `ReadyForQuery` +(`send_ready_for_query`, `query.rs:640-655`), and **cancel keys** — a secret +issued at startup that a *second* connection can present to cancel your running +query (`src/api/cancel.rs`; the query path arms it at `query.rs:38-44` and races +it against execution with `select` at `:94-98`). RESP's answer to "cancel my +query" is `CLIENT KILL`, which kills the connection. ### Step 6 — tonic/gRPC: the protocol you don't write -gRPC inverts the whole exercise: you write a `.proto` interface definition, -and the framing, parser, streaming, and backpressure (HTTP/2 flow-control -windows — receiver-advertised byte budgets per stream) are *generated and -inherited*, not written. In qdrant, the services are generated from the -protos in the `api/` crate. The costs of not writing it: every message is -protobuf (field tags, varints — no zero-copy into your value types), and -HTTP/2 framing means you can't debug with `nc`. +> **In:** two hand-written protocols, framing and state machine included. +> **Out:** what generation gives you, measured in lines; what it takes away; and +> two things qdrant had to fix in the protocol it did not write. -Two qdrant deployment details worth noting: +gRPC inverts the exercise. You write a `.proto` interface definition and the +framing, the parser, the streaming and the flow control are *generated and +inherited*. The exchange rate is measurable at qdrant's pin: -- `src/tonic/mod.rs:138` and `:277` — `Server::builder()` **twice**: - separate internal (peer-to-peer raft) and public gRPC servers. Protocol - surface split by trust domain — compare redis exposing admin + data on - one port. -- The middleware layers in mod.rs (auth around :138, logging, telemetry) — - tower's onion model, where each concern wraps the next, vs redis's - "check ACL inside processCommand". +``` + lib/api/src/grpc/proto/*.proto 16 files, 4 446 lines (written) + lib/api/src/grpc/qdrant.rs 17 428 lines (generated) + ──────── + ratio = 3.9× the hand-written IDL +``` + +`lib/api/src/grpc/qdrant.rs:1` says `// This file is @generated by prost-build.` +and `points_service.proto` describes the entire Points API — upsert, delete, get, +search, payload operations, index management — in **107 lines**. + +The costs are real. Every message is protobuf: field tags and varints, decoded +into generated structs, so there is no zero-copy path into your own value types +the way RESP's `$5\r\nalice\r\n` can borrow a slice. HTTP/2 framing means `nc` +and `telnet` are useless and you need `grpcurl`. And the protocol you did not +write has defaults you did not choose — which is the most useful thing in +qdrant's tonic setup: + +```rust +// qdrant src/tonic/mod.rs — inside init_internal, 277-285 + 277 let mut server = Server::builder() + 278 // Internally use a high limit for pending accept streams. + 279 // We can have a huge number of reset/dropped HTTP2 streams in our internal + 280 // communication when there are a lot of clients dropping connections. This + 281 // internally causes an GOAWAY/ENHANCE_YOUR_CALM error breaking cluster consensus. + 282 // We prefer to keep more pending reset streams even though this may be expensive, + 283 // versus an internal error that is very hard to handle. + 284 // More info: + 285 .http2_max_pending_accept_reset_streams(Some(1024)); +``` + +That knob is HTTP/2's defence against the "Rapid Reset" class of abuse — a peer +that opens streams and immediately resets them. qdrant's own cluster traffic +looked like the attack, tripped the library's `ENHANCE_YOUR_CALM` response, and +broke raft consensus. Inheriting a protocol means inheriting its threat model and +its defaults. The same file raises another inherited default six times: +`.max_decoding_message_size(usize::MAX)` on every service (`:179`, `:185`, +`:191`, `:197`, `:203`, `:209`) removes tonic's per-message cap outright. + +Two deployment details, both of which the earlier version of this chapter got +partly wrong: + +- **`Server::builder()` appears twice** — `:138` in `init` (public) and `:277` in + `init_internal` (peer-to-peer: raft, internal points/collections services, + shard snapshots). The protocol surface is split by **trust domain**: two + ports, two service lists, and the public one deliberately advertises only the + public services through reflection ("Only advertise the public services", + `:131-132`). Compare redis, which exposes admin and data commands on one port + and separates them with ACLs. +- **Auth is not at `:138`.** It is a tower layer, `option_layer(...AuthLayer...)` + at `:160-168` for the public server and `:301` for the internal one — and the + internal one is *opt-in*: + +```rust +// qdrant src/tonic/mod.rs — internal auth is off by default, 255-264 + 255 // Only enforce authentication on the internal API when the operator + 256 // explicitly opts in. The API key is still forwarded unconditionally + 257 // on outgoing internal requests, so the cluster keeps working + 258 // across a rolling upgrade while `enforce_internal_auth` is false. + 259 let internal_auth_layer = if settings.service.enforce_internal_auth.unwrap_or_default() + 260 { + 261 AuthKeys::try_create(&settings.service, toc.clone()).map(auth::AuthLayer::new) + 262 } else { + 263 None + 264 }; + 265 +``` + +`unwrap_or_default()` on an `Option` is `false`, so the internal gRPC port +is **unauthenticated unless configured otherwise**, and the comment says why: +during a rolling upgrade, half the cluster would not yet be sending the key. The +split is therefore not "trusted because authenticated" but "trusted because +network-isolated" — a deployment assumption, not a code one. That is worth +knowing before copying the pattern. + +The middleware itself is tower's onion: `ServiceBuilder::new().layer(logging) +.layer(telemetry).option_layer(auth).into_inner()` (`:155-169`), each concern +wrapping the next, applied once at `:172`. Compare redis, which checks ACLs +inside `processCommand` — a call, not a wrapper. + +Finally, the honest note about streaming. gRPC gives you server-streaming RPCs +for free, and qdrant uses **exactly one** in its entire surface: +`rpc ReadBytesStream(...) returns (stream ReadBytesStreamResponse)` +(`storage_read_service.proto:29`). Grep the other fifteen protos for `stream` and +you get nothing. Search, scroll, retrieve — all unary: one request, one fully +materialized response. **Having protocol-level streaming and using it are +different things**, and on the axis this topic cares about, qdrant's public API +behaves like RESP. ### Step 7 — the design space, assembled -The three protocols answer the same questions differently — fill the last -row yourself: +> **In:** all three protocols, read rather than remembered. +> **Out:** the comparison table with the evidence attached, and the last row left +> for you. -| | RESP | pgwire | gRPC | +| | RESP | pgwire | gRPC (qdrant) | |---|---|---|---| -| framing | ASCII len prefixes | type byte + i32 len | HTTP/2 frames | -| parse cost | memchr + atoi | fixed header read | protobuf decode | -| streaming | no (buffer all) | portals, row-at-a-time | HTTP/2 streams | -| backpressure | output-buffer kill | portal suspend | flow-control windows | -| debuggability | `nc` works | needs a tool | needs grpcurl | -| your GRAPH.QUERY | ? | ? | ? | +| framing | ASCII length prefixes, `\r\n` | 1 type byte + `i32` length (`messages/mod.rs:121-136`) | HTTP/2 frames | +| length semantics | length of the payload | includes itself, excludes the type byte (`codec.rs:77-78`) | frame header | +| partial input | resume by re-parsing | `Ok(None)`, buffer untouched (`codec.rs:83`) | h2 handles it | +| 3-field row | 32 bytes | 29 bytes | protobuf, varint-tagged | +| parse cost | `memchr` + `atoi` | fixed 5-byte header read | protobuf decode into generated structs | +| streaming | no — buffer all | rows as produced (`query.rs:584-588`) | available; used once (`storage_read_service.proto:29`) | +| backpressure | output-buffer-limit kill | `PortalSuspended` (`query.rs:351-354`) | HTTP/2 flow control, unused by the unary API | +| session state | none | 7-state enum (`api/mod.rs:46-64`) | per-RPC; none between | +| cancellation | `CLIENT KILL` | cancel key + `select` (`query.rs:94-98`) | HTTP/2 stream reset | +| debuggability | `nc` works | needs a client library | needs `grpcurl` | +| your `GRAPH.QUERY` | ? | ? | ? | ## Where each step lives in the code -Local clones at `~/repos/pgwire` and `~/repos/qdrant`: +No local clones are needed; every row was read with +`tools/pinned-source.py show -r A:B` at the pins named at the top. | Anchor | What | Step | |--------|------|------| -| pgwire `src/messages/` | typed message structs, encode/decode | 2 | -| pgwire `src/api/query.rs:48` — `SimpleQueryHandler` | simple query | 3 | -| pgwire `src/api/query.rs:174` — `ExtendedQueryHandler` | Parse/Bind/Execute/Sync, portals | 4 | -| pgwire `src/api/auth.rs` + `api/mod.rs:555` — `StartupHandler` | startup + auth state machine | 5 | -| qdrant `src/tonic/mod.rs:138`, `:277` | two servers, tower middleware | 6 | -| qdrant `api/` crate | generated services from `.proto` | 6 | - -Read pgwire asking Step 1's question (*where does session state live?*), -and qdrant asking Step 6's (*what did I not have to write, and what did -that cost me?*). +| pgwire `src/api/mod.rs:46-64` | the 7-state connection lifecycle | 1 | +| pgwire `src/tokio/server.rs:198-214` | state dispatch; discard-until-`Sync` | 1, 4 | +| pgwire `src/messages/mod.rs:121-136` | `Message::encode` — type byte + `i32` length | 2 | +| pgwire `src/messages/codec.rs:63-84` | `decode_packet` — what the length counts, resumability | 2 | +| pgwire `src/messages/mod.rs:51-56` | the four packet-size limits | 2 | +| pgwire `src/api/query.rs:48` | `SimpleQueryHandler` | 3 | +| pgwire `src/api/query.rs:568-596` | `send_query_response` — stream rows, feed, flush once | 3 | +| pgwire `src/api/query.rs:174` | `ExtendedQueryHandler` | 4 | +| pgwire `src/api/query.rs:341-361` | the suspend/resume branch | 4 | +| pgwire `src/api/query.rs:599-637` | `send_partial_query_response`; `max_rows == 0` means unlimited | 4 | +| pgwire `src/api/auth/mod.rs:22` + `src/api/mod.rs:555` | `StartupHandler` | 5 | +| pgwire `src/api/cancel.rs`, `query.rs:38-44`, `:94-98` | cancel keys, raced against execution | 5 | +| qdrant `lib/api/src/grpc/proto/*.proto` | 4,446 lines of IDL, 16 files | 6 | +| qdrant `lib/api/src/grpc/qdrant.rs:1` | 17,428 generated lines | 6 | +| qdrant `src/tonic/mod.rs:138`, `:277` | two servers, two trust domains | 6 | +| qdrant `src/tonic/mod.rs:155-172`, `:255-264`, `:277-285` | tower onion; opt-in internal auth; the h2 reset-stream fix | 6 | + +Read pgwire asking Step 1's question — *where does session state live?* — and +qdrant asking Step 6's: *what did I not have to write, and what did that cost me?* ## Questions to answer in notes.md 1. FalkorDB result sets ride RESP arrays — huge ones buffer entirely in the - module. What would a portal-style GRAPH.QUERY cursor look like as RESP - commands? (FalkorDB actually has one — recall GRAPH.QUERY's timeout + - result-set config; design the missing GRAPH.CURSOR anyway.) -2. Why does qdrant run TWO tonic servers instead of one with authz? What - attack/ops story does the split simplify? -3. Extended query's 5 messages cost a round-trip each unless pipelined — - how does pgwire's async design let Parse/Bind/Execute/Sync coalesce, and - what's the RESP equivalent? (MULTI? No — pipelining itself.) + module. Design `GRAPH.CURSOR` as RESP commands: what replaces `Bind` (where + does the portal name come from), what replaces `PortalSuspended` (RESP has no + "I stopped" reply), and who evicts an abandoned cursor? Compare what FalkorDB + actually does today (`GRAPH.QUERY` timeout + result-set size config). +2. Why does qdrant run two tonic servers instead of one with authz — and, given + `enforce_internal_auth` defaults to false (`src/tonic/mod.rs:259`), what is + actually protecting the internal port? Write the deployment assumption down as + if it were a requirement. +3. Step 4 shows five extended-query messages costing one round trip when + pipelined. What has to be true of *your* server for that to be safe? Name + redis's equivalent guarantee for a pipelined RESP batch, and say what happens + there when command 2 of 5 fails. +4. `feed` versus `send` (`query.rs:587` against `:591`) is the same decision as + redis's reply buffer and this topic's "flush only when drained". Find the + place in your M7 server where you make it, and measure the `-P 64` penalty + for getting it wrong. +5. gRPC gave qdrant streaming for free and it used it once + (`storage_read_service.proto:29`). For a graph query returning 1M rows, write + the `.proto` you would need, then ask what the client library does with a + stream that outlives the request — and whether that is different from a + portal. ## Done when -You can fill the table's last row with committed answers for M7 and defend -"RESP + explicit cursor commands" against "just use gRPC" for a graph DB. +Answer each before unfolding it. + +- [ ] You can state exactly what a postgres message header contains and what its + length field counts, and prove it from pgwire's decoder. + +
Answer + + One ASCII type byte (absent only on the startup packet) then a 4-byte + big-endian `i32`. The length **includes its own 4 bytes and the body, and + excludes the type byte**: `decode_packet` requires `buf.remaining() >= msg_len + + offset` (`codec.rs:77`) where `offset` is 1 when a type byte is present, then + advances `offset + 4` (`:78`). If the whole message has not arrived it returns + `Ok(None)` (`:83`) without consuming anything, so the decoder resumes cleanly — + and `get_length` (`:53-59`) peeks the length without moving the cursor. + +
+ +- [ ] You can say how much smaller postgres's framing is than RESP's for a real + row, and therefore what the protocol's actual advantage is. + +
Answer + + For `("alice", "30", "NYC")`: 29 bytes as a `DataRow` (1 type + 4 length + 2 + field count + 3×4 field lengths + 10 bytes of value) against 32 bytes as a + RESP2 array. **Under 10%** — framing is not the advantage. The advantages are + structural: rows stream as they are produced (`query.rs:584-588`), a portal can + be suspended (`:351-354`), the session carries state, and a NULL is an in-row + `-1` rather than a distinct wire form. + +
+ +- [ ] You can explain what a portal is, what `Execute {max_rows}` does when the + rows run out and when they do not, and what `max_rows = 0` means. + +
Answer + + A portal is a bound, executable instance of a prepared statement — the + server-side state of a partially consumed query. `_on_execute` starts it once + (`PortalExecutionState::Initial`, `query.rs:271-274`) and thereafter only + fetches. After sending up to `max_rows` rows it sends `PortalSuspended` if more + remain (`:351-354`) or `CommandComplete` if not (`:355-360`). `max_rows = 0` + means **no limit**: `while max_rows == 0 || rows < max_rows` + (`send_partial_query_response`, `:614`). + + That is protocol-level backpressure: the server stops producing and the client + chooses whether to pay for more. RESP's only options are buffer-everything or + kill the client on the output-buffer limit. + +
+ +- [ ] You can compute what the extended protocol's extra messages cost, and name + the rule that lets a client avoid paying it. + +
Answer + + Five messages awaited one at a time is five round trips. On this repo's + loopback measurement (22.68 µs per round trip at P=1, + [FINDINGS.md](../../FINDINGS.md) row 7) that is 113.4 µs per query — 8,818 + queries/s — against 22.68 µs and 44,088 queries/s when the five are written + together: exactly 5×. At a 100 µs network RTT, 500 µs against 100 µs. + + The rule that makes batching safe is postgres's error recovery: on an error in + any extended-query message the backend "reads and discards messages until a + Sync is reached", quoted in pgwire's dispatcher at `src/tokio/server.rs:203-207` + and implemented as the `AwaitingSync` state. A failed `Parse` cannot leave a + half-executed batch behind. + +
+ +- [ ] You can quantify what gRPC generates for you, and name two things qdrant + had to override in the protocol it did not write. + +
Answer + + 4,446 lines of `.proto` across 16 files produce 17,428 lines in + `lib/api/src/grpc/qdrant.rs` — 3.9× — and the entire Points API is 107 lines of + IDL. + + Overrides: (1) `http2_max_pending_accept_reset_streams(Some(1024))` + (`src/tonic/mod.rs:285`), because the library's Rapid-Reset defence fired on + qdrant's own cluster traffic and broke raft consensus (issue #1907); (2) + `max_decoding_message_size(usize::MAX)` on all six public services (`:179`, + `:185`, `:191`, `:197`, `:203`, `:209`), removing tonic's per-message cap. A + third, if you want it: TLS and compression are configured per server rather + than inherited. + +
+ +- [ ] You can say whether gRPC actually solves the 10M-row problem *in qdrant*, + with evidence. + +
Answer + + No. The capability exists — server-streaming RPCs and HTTP/2 flow-control + windows — but qdrant declares exactly one streaming RPC in its whole surface, + `ReadBytesStream` (`storage_read_service.proto:29`), and it is a storage-read + internal. Search, scroll and retrieve are unary: one request, one fully + materialized response, which is RESP's failure mode with a nicer encoding. + Inheriting backpressure is not the same as using it — and that is the question + M7 has to answer for itself rather than delegate to a framework. + +
+ +- [ ] You can fill the table's last row with committed answers for M7, and defend + "RESP + explicit cursor commands" against "just use gRPC" for a graph DB. + +
Answer + + There is no single right answer; there is a defensible one. RESP + explicit + cursor commands keeps `nc` debuggability, keeps redis clients working, and puts + the cursor where the cost actually is — but it makes you write the cursor + lifetime, eviction and error recovery that postgres's `Sync` rule and gRPC's + stream lifetime give you. gRPC hands you streaming and flow control and takes + away the zero-copy parse path, the single-port deployment and every existing + redis client — and qdrant's one streaming RPC out of sixteen protos is evidence + that having the mechanism does not mean you will use it. + + Whatever you choose, the defence has to name what you are buying and what you + are writing yourself. "It's the standard" is not a row in the table. + +
## References -**Code** -- [sunng87/pgwire](https://github.com/sunng87/pgwire) — `src/messages/`, - `src/api/query.rs`, `src/api/auth.rs`; the crate structure IS the - protocol lesson. Local clone at `~/repos/pgwire`. -- [qdrant/qdrant](https://github.com/qdrant/qdrant) — `src/tonic/mod.rs` - (two servers, tower middleware) plus the generated services in the - `api/` crate. Local clone at `~/repos/qdrant`. +**Code, at this repo's pins** +- [sunng87/pgwire](https://github.com/sunng87/pgwire) @ `6bb6299` — + `src/messages/mod.rs` and `src/messages/codec.rs` (framing), + `src/api/query.rs` (both query protocols), `src/api/auth/mod.rs` (startup and + authentication), `src/api/mod.rs` (connection state, handler selection), + `src/tokio/server.rs` (the dispatcher). The crate layout *is* the protocol + lesson: one module per message family, one type per message. +- [qdrant/qdrant](https://github.com/qdrant/qdrant) @ `44ad62f` — + `src/tonic/mod.rs` (two servers, tower middleware, the inherited-default + overrides) and the generated protocol crate at `lib/api/` (**not** `api/`: + protos in `lib/api/src/grpc/proto/`, generated code in + `lib/api/src/grpc/qdrant.rs`). +- Fetch either with `tools/pinned-source.py show -r A:B`; verify an + anchor with `tools/pinned-source.py check : --contains '…'`. + +**Corrections made to the previous version of this chapter** +- `src/api/auth.rs` → `src/api/auth/mod.rs:22` (`auth` is a directory). +- qdrant "`api/` crate" → `lib/api/`. +- "middleware layers in mod.rs (auth around :138)" → `:138` is + `Server::builder()`; auth is `option_layer` at `:160-168` (public) and `:301` + (internal), and the internal one is disabled by default (`:259`). +- "five messages — Parse → Bind → Execute → Sync" listed four; the five are + Parse, Bind, Describe, Execute, Sync. +- "local clones at `~/repos/pgwire` and `~/repos/qdrant`" — no such clones exist + in this repo's workflow; `tools/pinned-source.py` is the access path. + +**Read alongside** +- [reading-redis-ae-networking.md](reading-redis-ae-networking.md) — the same + buffer-and-flush-once discipline, in C. +- [reading-bolt-packstream.md](reading-bolt-packstream.md) — a third answer, + where the cursor is specified and then not implemented. +- [Topic 7 README §4](README.md#4-backpressure--the-part-everyone-forgets) — the + framing this chapter supplies the evidence for. + +**Measured in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 7 — 44,088 ops/s at P=1, 12,321,414 at + P=256, 279×; the 22.68 µs round trip used in Steps 4 and 5 is the P=1 row of + [notes.md](notes.md). diff --git a/topics/07-networking-protocols/reading-redis-ae-networking.md b/topics/07-networking-protocols/reading-redis-ae-networking.md index 7fe62a2..23626dd 100644 --- a/topics/07-networking-protocols/reading-redis-ae-networking.md +++ b/topics/07-networking-protocols/reading-redis-ae-networking.md @@ -1,191 +1,1148 @@ # The redis event loop: pipelining for free One thread, one poll syscall per iteration, and two buffering decisions — -parse everything the read buffer holds, write nothing until beforeSleep — -give redis pipelining and reply batching without any dedicated machinery. -This chapter builds the loop step by step — what an event loop even is, why -the handler table is an array, how the read path turns one syscall into 100 -command executions, why replies are hoarded instead of written, and where the -whole thing kills a client — then maps each step to the functions. `ae.c` is -~500 lines, so read it fully (a rare luxury); `networking.c` is huge, so -read only the five functions this chapter walks. +parse a batch of commands out of whatever the read buffer holds, and write +nothing until the top of the next loop turn — give redis pipelining and reply +batching without any dedicated machinery. This chapter builds the loop step by +step: what an event loop even is, why the handler table is an array, how the +read path turns one syscall into a batch of command executions, the arithmetic +that turns 44k ops/s into 12.3M, why the parser is resumable, why replies are +hoarded, and where the whole thing kills a client. + +**Which version this chapter is about.** Every anchor below is +`redis/redis@a176d1225`, which is what `tools/pinned-source.py` will hand you: + +``` +$ tools/pinned-source.py ref redis +redis redis/redis a176d1225 + +$ tools/pinned-source.py show redis src/ae.c -r 360:420 +$ tools/pinned-source.py check redis src/networking.c:2802 --contains 'handleClientsWithPendingWrites' +``` + +You do **not** need a local clone. If you have one and its numbers disagree, +your clone is at a different commit — this file's line numbers are only true at +`a176d1225`. `src/ae.c` is 511 lines, so read it end to end (a rare luxury in +this repo). `src/networking.c` is 5,775 lines, so read only the ten functions +this chapter walks. ## The problem in one sentence -One redis thread must serve 10,000 concurrent connections at ~1M ops/s, -which leaves a budget of roughly one microsecond of CPU per command — -so the design's whole job is to spend as few syscalls (~1–2 µs each) as -possible per command, ideally amortizing one syscall over a hundred -commands. +One redis thread must serve ten thousand concurrent connections at a million +operations per second, which leaves roughly one microsecond of CPU per +command — and this repo's own topic-5 lane measured a single `write()` syscall +at 857k/s, i.e. **1.17 µs each** ([FINDINGS.md](../../FINDINGS.md) row 5) — so +the naive two-syscalls-per-command design is already 2.3× over budget before it +parses a byte, and the entire architecture is an argument about how to get the +syscall count per command *below one*. ## The concepts, step by step -### Step 1 — the event loop: one thread, one poll, many sockets - -An **event loop** is a single thread that, instead of dedicating itself to -one connection, repeatedly asks the kernel "which of my sockets have data -waiting right now?" and handles exactly those. The asking is one syscall — -`kqueue` on your Mac, `epoll` on Linux — that takes a set of **fds** (file -descriptors: the small integers the OS uses to name open sockets) and -returns only the *ready* ones, costing O(ready), not O(registered). - -`aeProcessEvents` (ae.c:360) is one turn of the loop: run the -`beforesleep` callback (:377–378, important in Step 5), then `aeApiPoll` -(:398) — **one syscall collects all ready fds** — then dispatch each ready -fd to its registered handler. Timers ride the same loop: the poll timeout -is set to the time until the nearest timer. The OS backend is chosen at -compile time (`ae_kqueue.c` / `ae_epoll.c`) behind an abstraction of just 4 -functions (add/del/poll/name). - -Why it matters: 10K blocked threads would cost stacks and context switches -(topic 7 §3, C10K); one loop thread costs one poll syscall per *batch* of -ready events. - -### Step 2 — `events[fd]`: the handler table is an array, not a hash map - -The loop needs to map each ready fd to its handler and per-connection -state. `aeCreateEventLoop` (ae.c:47) allocates plain arrays indexed by fd — -`events[fd]` — not a hash table, because fds are exactly the keys arrays -love: small dense integers handed out by the OS from the lowest free slot. -`setsize` = maxclients + headroom. - -Beyond speed, the array matches fd *semantics*: when a connection closes, -its fd number is immediately reusable by the next `accept()`, and -`events[fd]` is simply overwritten — a `HashMap` would need -careful delete-before-reinsert to avoid a stale handler firing on the new -connection (question 2 below). - -### Step 3 — the read path: one read() becomes N command executions - -When a client's fd is readable, `readQueryFromClient` (networking.c:3715) -reads up to 16 KB (`PROTO_IOBUF_LEN`, server.h:188) into the client's -**querybuf** (a per-client input accumulation buffer), then calls -`processInputBuffer` (:3529), which **loops**: parse one complete command, -execute it, repeat until the buffer has no complete command left. - -That loop is the entire implementation of **pipelining** (a client sending -many commands without waiting for replies): if the client sent 100 commands -back-to-back, one 16 KB `read()` swallows them all, and the loop executes -all 100 with zero further syscalls. +### Step 1 — the event loop: one thread, one poll syscall, many sockets + +> **In:** a set of open sockets, most of them idle, and one thread. +> **Out:** the subset that has data waiting right now, plus a call to the +> handler registered for each — in O(ready) work, not O(registered). + +A **syscall** is a call into the kernel: the CPU switches privilege level, +saves and restores register state, and runs kernel code on your thread's +behalf. It is not a function call; it costs on the order of a microsecond +(measured: 1.17 µs for `write()`, [FINDINGS.md](../../FINDINGS.md) row 5). +Every syscall in the hot path is a tax you pay per command unless you can +amortize it over several. + +A **file descriptor** (fd) is the small non-negative integer the kernel hands +you to name an open socket, file or pipe. They are allocated from the lowest +free slot, so a process with 10,000 open sockets has fds roughly in the range +0..10,050 — dense, not sparse. That fact drives Step 2. + +An **event loop** is a single thread that, instead of dedicating itself to one +connection and blocking on it, repeatedly asks the kernel "which of my +descriptors are ready?" and services exactly those. The asking is one syscall. +This is **readiness notification**: the kernel tells you a socket *can* be read +without blocking, and you then do the read yourself. (The alternative, +**completion notification**, has you submit the read up front and the kernel +tells you when the bytes have landed — that is `io_uring` and IOCP, and the +c10k chapter of this topic works through why the difference matters.) + +`aeProcessEvents` is one turn of that loop: + +```c +// redis src/ae.c — aeProcessEvents, 360-413 (the shape of one loop turn) + 360 int aeProcessEvents(aeEventLoop *eventLoop, int flags) + 361 { + 362 int processed = 0, numevents; +// ... 363-376: early-out when neither file nor time events were requested, +// and compute the poll timeout from the nearest timer ... + 377 if (eventLoop->beforesleep != NULL && (flags & AE_CALL_BEFORE_SLEEP)) + 378 eventLoop->beforesleep(eventLoop); +// ... 379-395: tvp = 0 if AE_DONT_WAIT, else time until the earliest timer ... + 396 /* Call the multiplexing API, will return only on timeout or when + 397 * some event fires. */ + 398 numevents = aeApiPoll(eventLoop, tvp); +// ... 399-408: zero numevents if file events were not requested; aftersleep ... + 409 for (j = 0; j < numevents; j++) { + 410 int fd = eventLoop->fired[j].fd; + 411 aeFileEvent *fe = &eventLoop->events[fd]; + 412 int mask = eventLoop->fired[j].mask; + 413 int fired = 0; /* Number of events fired for current fd. */ +``` + +Three things to notice, in order of how much they will surprise you: + +1. **`beforesleep` runs *before* the poll, not after the dispatch** (`:377-378`). + The name is accurate — it is the last thing that happens before the thread + goes to sleep in `aeApiPoll`. This is where the entire write path lives + (Step 6). It is wired up once, at `src/server.c:3069` + (`aeSetBeforeSleepProc(server.el, beforeSleep)`), and `aeMain(server.el)` at + `src/server.c:8027` is the whole server's main loop. +2. **One `aeApiPoll` collects *all* ready fds** (`:398`). Not one syscall per + ready socket — one syscall per *batch* of ready sockets. On your Mac that is + a single `kevent()`: + +```c +// redis src/ae_kqueue.c — aeApiPoll, 124-137 (one kevent() for the whole set) + 124 static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) { + 125 aeApiState *state = eventLoop->apidata; + 126 int retval, numevents = 0; + 127 + 128 if (tvp != NULL) { + 129 struct timespec timeout; + 130 timeout.tv_sec = tvp->tv_sec; + 131 timeout.tv_nsec = tvp->tv_usec * 1000; + 132 retval = kevent(state->kqfd, NULL, 0, state->events, eventLoop->setsize, + 133 &timeout); + 134 } else { + 135 retval = kevent(state->kqfd, NULL, 0, state->events, eventLoop->setsize, + 136 NULL); + 137 } +``` + +3. **Timers ride the same poll.** The timeout handed to `kevent` is the time + until the nearest timer (`ae.c:388-395`), so redis needs no timer thread and + no separate `sleep`. One syscall serves both "wake me when a socket is + ready" and "wake me in 100 ms". + +The backend is chosen at compile time behind an abstraction of four functions +(`aeApiCreate` / `aeApiAddEvent` / `aeApiDelEvent` / `aeApiPoll`, plus +`aeApiName`). On macOS you compile `ae_kqueue.c`; `ae_epoll.c` is *not* +compiled on your machine. The c10k chapter of this topic works through the +selection ladder at `ae.c:30-44` and what each backend actually guarantees; +the one fact you need here is that redis registers events **level-triggered** +(`ae_kqueue.c:102-111` uses `EV_ADD` without `EV_CLEAR`), so a socket with +unread bytes reports ready again on the next poll — which is what makes it safe +for Step 3 to stop reading whenever it likes. + +Why it matters: 10,000 threads blocked in `read()` would cost 10,000 stacks and +a context switch per message. One loop thread costs one poll syscall per +*batch* of ready events, and that batch can be large. + +### Step 2 — `events[fd]`: the dispatch table is an array, not a hash map + +> **In:** a bare integer fd that just became readable. +> **Out:** the function to call and the `client *` to call it with, in one +> load — and a dispatch table that costs kilobytes for 10,000 connections. + +Step 1's dispatch loop does `&eventLoop->events[fd]` (`ae.c:411`). That is a +plain array indexed by the raw file descriptor. Two questions: why is that fast, +and why is it *correct*? + +Fast, because fds are exactly the keys arrays love — small, dense integers +handed out by the OS from the lowest free slot. A hash map would compute a hash +and chase a pointer to reach data an array reaches with one shift-and-add. Here +is what a slot costs: + +```c +// redis src/ae.h — the two arrays' element types, 52-57 and 73-76 + 52 typedef struct aeFileEvent { + 53 int mask; /* one of AE_(READABLE|WRITABLE|BARRIER) */ + 54 aeFileProc *rfileProc; + 55 aeFileProc *wfileProc; + 56 void *clientData; + 57 } aeFileEvent; +// ... 58-72: aeTimeEvent ... + 73 typedef struct aeFiredEvent { + 74 int fd; + 75 int mask; + 76 } aeFiredEvent; +``` + +Work the memory. On a 64-bit machine `aeFileEvent` is one `int` padded to 8 +plus three pointers at 8 = **32 bytes**; `aeFiredEvent` is two `int`s = **8 +bytes**. The loop is created with `setsize = maxclients + CONFIG_FDSET_INCR` +(`src/server.c:2937`), and `CONFIG_FDSET_INCR` is `32 + 96 = 128` +(`src/server.h:143`, `:207`). So at `maxclients 10000`: + +``` +setsize = 10000 + 128 = 10,128 slots +events array = 10,128 × 32 bytes = 324,096 B ≈ 317 KiB +fired array = 10,128 × 8 bytes = 81,024 B ≈ 79 KiB + --------------------- +whole dispatch table for 10,000 connections ≈ 405,120 B ≈ 396 KiB +``` + +Compare the thread-per-connection design the c10k chapter takes apart: 10,000 +threads at the usual 2 MiB stack reservation is **19.5 GiB** of virtual address +space. The event loop's *entire* connection-dispatch structure is 396 KiB — +about 1/51,700th of it. (Both figures are address space, not resident memory; +idle thread stacks are mostly never faulted in. The point is not that threads +use 19.5 GiB of RAM, it is that the loop's bookkeeping fits in L2.) + +And it is not even eagerly allocated. Modern redis grows the arrays on demand: + +```c +// redis src/ae.c — aeCreateFileEvent, 145-168 (grow-on-demand, capped at setsize) + 145 int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, + 146 aeFileProc *proc, void *clientData) + 147 { + 148 if (fd >= eventLoop->setsize) { + 149 errno = ERANGE; + 150 return AE_ERR; + 151 } + 152 + 153 /* Resize the events and fired arrays if the file + 154 * descriptor exceeds the current number of events. */ + 155 if (unlikely(fd >= eventLoop->nevents)) { + 156 int newnevents = eventLoop->nevents; + 157 newnevents = (newnevents * 2 > fd + 1) ? newnevents * 2 : fd + 1; + 158 newnevents = (newnevents > eventLoop->setsize) ? eventLoop->setsize : newnevents; + 159 eventLoop->events = zrealloc(eventLoop->events, sizeof(aeFileEvent) * newnevents); + 160 eventLoop->fired = zrealloc(eventLoop->fired, sizeof(aeFiredEvent) * newnevents); + 161 + 162 /* Initialize new slots with an AE_NONE mask */ + 163 for (int i = eventLoop->nevents; i < newnevents; i++) + 164 eventLoop->events[i].mask = AE_NONE; + 165 eventLoop->nevents = newnevents; + 166 } + 167 + 168 aeFileEvent *fe = &eventLoop->events[fd]; +``` + +`nevents` starts at `min(setsize, INITIAL_EVENT)` with `INITIAL_EVENT 1024` +(`ae.c:46`, `:54-56`), so a server configured for 10,000 clients but serving 40 +of them holds a 40 KiB table, and doubles from there — `setsize` is a *cap*, +not an allocation. + +Correctness is the more interesting half. When a connection closes, its fd +number goes straight back on the free list and the very next `accept()` can +return it. With an array, `events[fd]` is simply overwritten by the new +connection's registration — there is no stale entry to find. With a +`HashMap` you would have to guarantee the delete happens before +the reinsert, or a handler belonging to a dead client fires on a live one that +happens to have inherited its number. The array makes the aliasing that fd +reuse creates *unrepresentable*: there is exactly one slot per fd, always. + +### Step 3 — the read path: one `read()` becomes a batch of commands + +> **In:** an fd the poll reported readable. +> **Out:** up to `lookahead` fully parsed commands executed back to back, all +> paid for with a single `read()` syscall. + +When a client's fd is readable, the registered handler is `readQueryFromClient` +(`networking.c:3715`, registered at `:132` inside `createClient`). It sizes a +read, does exactly one, and hands the bytes to the parse-and-execute loop. + +The sizing is not just "16 KB": + +```c +// redis src/networking.c — readQueryFromClient, 3732 and 3780-3798 (one read, sized up) + 3732 readlen = PROTO_IOBUF_LEN; +// ... 3733-3779: if the next thing on the wire is a >= 32 KB bulk argument, +// set readlen to exactly the remaining bytes of that argument +// (Step 5's zero-copy depends on this); otherwise borrow the +// per-thread reusable query buffer ... + 3780 qblen = sdslen(c->querybuf); + 3781 if (!(c->flags & CLIENT_MASTER) && // master client's querybuf can grow greedy. + 3782 (big_arg || sdsalloc(c->querybuf) < PROTO_IOBUF_LEN)) { +// ... 3783-3791: non-greedy growth for the initial allocation and for big args ... + 3792 } else { + 3793 c->querybuf = sdsMakeRoomFor(c->querybuf, readlen); + 3794 + 3795 /* Read as much as possible from the socket to save read(2) system calls. */ + 3796 readlen = sdsavail(c->querybuf); + 3797 } + 3798 nread = connRead(c->conn, c->querybuf+qblen, readlen); +``` + +`PROTO_IOBUF_LEN` is 16 KiB (`server.h:188`) but line 3796 is the real policy: +once the buffer has grown, redis asks for *everything the buffer can hold*, +with the comment stating the motive outright — "to save read(2) system calls". +The read at `:3798` is one syscall, and it is the only one on the read side. + +There is a second memory trick here worth noticing. A client with no partial +command in flight does not own a query buffer at all — it *borrows* a +per-thread reusable one (`networking.c:3766-3776`), which it gives back once +the buffer drains. Ten thousand mostly-idle connections therefore do not cost +ten thousand 16 KiB buffers; they cost one per thread plus whatever the few +mid-command clients hold. + +Now the loop the bytes flow into. This is the part most descriptions of redis +get wrong, because it changed: `processInputBuffer` is not "parse one command, +execute it, repeat". It is a **two-level** loop — an inner loop that parses up +to `lookahead` commands, then an outer loop that executes the parsed ones: + +```c +// redis src/networking.c — processInputBuffer, 3529-3546 and 3563-3567 (the batching) + 3529 int processInputBuffer(client *c) { + 3530 /* We limit the lookahead for unauthenticated connections to 1. + 3531 * This is both to reduce memory overhead, and to prevent errors: AUTH can + 3532 * affect the handling of succeeding commands. Parsing of "large" + 3533 * unauthenticated multibulk commands is rejected, which would cause those + 3534 * commands to incorrectly return an error to the client. */ + 3535 const int lookahead = authRequired(c) ? 1 : server.lookahead; + 3536 + 3537 /* Keep processing while there is something in the input buffer */ + 3538 while ((c->querybuf && c->qb_pos < sdslen(c->querybuf)) || + 3539 c->pending_cmds.ready_len > 0) + 3540 { +// ... 3541-3562: bail out if the client is blocked, closing, or already has a +// command in flight; decide whether to parse more ... + 3563 const int parse_more = !c->pending_cmds.ready_len; + 3564 + 3565 /* Parse up to lookahead commands only if we don't have enough ready commands */ + 3566 while (parse_more && c->pending_cmds.ready_len < lookahead && + 3567 c->querybuf && c->qb_pos < sdslen(c->querybuf)) +``` + +`server.lookahead` defaults to **16** (`REDIS_DEFAULT_LOOKAHEAD`, +`server.h:210`; the config is `lookahead`, `config.c:3246`). The parsed +commands land in `c->pending_cmds`, a list of `pendingCommand` structs +(`server.h:1444-1445`), and only then does the outer loop pull them off the +head and execute them one at a time. + +Why decouple parsing from execution at all? Because it creates a window in +which redis knows *which keys the next sixteen commands will touch* before it +touches any of them — and can prefetch them: + +```c +// redis src/networking.c — processInputBuffer, 3635-3646 (prefetch the parsed batch) + 3635 /* Prefetch the command only when more commands have been parsed and we + 3636 * are in the main thread. If running in an IO thread, prefetch will be + 3637 * deferred until the client is processed by the main thread. Skip prefetch + 3638 * if there are too few commands to avoid meaningless prefetching. */ + 3639 if (parse_more && c->running_tid == IOTHREAD_MAIN_THREAD_ID && + 3640 c->pending_cmds.ready_len > 1) + 3641 { + 3642 /* Prefetch the commands. */ + 3643 resetCommandsBatch(); + 3644 addCommandToBatch(c); + 3645 prefetchCommands(); + 3646 } +``` + +Those three functions live in `src/memory_prefetch.h:22-24`. This is the same +idea the valkey chapter of this topic takes apart at length — hide DRAM latency +by issuing several dependent lookups' cache misses concurrently — and it is +only *possible* because a pipelined client handed the server sixteen commands +in one read. Pipelining does not merely save syscalls; it hands the engine a +batch to be clever with. + +**Pipelining** is the client-side technique of writing many requests without +waiting for each reply. The server needs no feature to support it: the inner +loop at `:3566` drains whatever the buffer holds, and a client that sent 100 +commands back to back has all 100 executed off one `read()`. + +### Step 4 — the round-trip arithmetic: where 44k becomes 12.3M + +> **In:** the measured loopback lane in `notes.md` and a pipeline depth P. +> **Out:** the syscall count and round-trip count *per request*, and the +> division that turns 44,088 ops/s into 12,321,414 ops/s. + +This topic's measured headline ([FINDINGS.md](../../FINDINGS.md) row 7) is +about a benchmark that does *no work at all* — no parsing, no store, 32 bytes +in and 8 bytes out — and still spans 279×: + +| P | ops/s | µs per request | client syscalls per op | vs P=1 | +|---|---|---|---|---| +| 1 | 44 088 | 22.68 | 2.000 | 1.0× | +| 8 | 353 067 | 2.83 | 0.250 | 8.0× | +| 64 | 2 919 728 | 0.34 | 0.031 | 66.2× | +| 256 | 12 321 414 | 0.08 | 0.008 | **279.5×** | + +A **round trip** is one traversal of request-to-server-and-reply-back: the +client cannot send request *n+1* until reply *n* arrives, so its rate is capped +at 1/RTT no matter how fast the server is. Count both quantities per request at +depth P: + +``` +Per BATCH of P requests, client side: 1 × write() + 1 × read() = 2 syscalls +Per BATCH of P requests, server side: 1 × read() + 1 × writev() = 2 syscalls +Per BATCH of P requests: 1 round trip + +So per REQUEST: + client syscalls = 2 / P + total syscalls = 4 / P (both processes) + round trips = 1 / P + + P = 1 : 2.000 client syscalls, 4.000 total, 1.000 RTT per request + P = 8 : 0.250 client syscalls, 0.500 total, 0.125 RTT per request + P = 64 : 0.031 client syscalls, 0.062 total, 0.0156 RTT per request + P = 256 : 0.0078 client syscalls, 0.0156 total, 0.0039 RTT per request +``` + +The "client syscalls per op" column in `notes.md` is exactly this computed +floor, `2.0/P` — it is arithmetic, not an instrumented count. Say so when you +quote it. + +Now the division the headline rests on. Throughput is depth divided by the time +one batch takes: + +``` +ops/s = P / T_batch + +P = 1: T_batch = 1 / 44,088 = 22.681 µs (one request per batch) +P = 256: T_batch = 256 / 12,321,414 = 20.777 µs (256 requests per batch) + +speedup = (256 / 20.777 µs) / (1 / 22.681 µs) + = 256 × (22.681 / 20.777) + = 256 × 1.0916 + = 279.4× ← matches the measured 279.5× to rounding +``` + +Read that middle line again, because it is the whole chapter. **The time to +complete a batch barely changed** — 22.681 µs for one request, 20.777 µs for +two hundred and fifty-six. A batch of 256 requests costs *8% less* than a batch +of 1. Every microsecond of that 22.681 was overhead: two context switches into +the kernel and back on each side, two process wakeups, one loopback traversal. +The payload was never the cost. Check that directly: 40 bytes cross the +loopback per exchange, and even at a conservative 10 GB/s that is 4 ns — 0.02% +of 22.681 µs. + +The 279× therefore decomposes cleanly: **256× from amortizing a fixed per-batch +cost over 256 requests, and a further 1.09× because the batched exchange is +itself slightly cheaper per byte than the single one.** That is why `notes.md` +calls `2/P` a floor on the improvement rather than a ceiling. + +Turn it around and the design constraint from "the problem in one sentence" +falls out. Suppose you want 1,000,000 ops/s from one thread. That is a 1.000 µs +budget per command, all in. At the measured 1.17 µs per `write()` +([FINDINGS.md](../../FINDINGS.md) row 5): + +``` +budget per command at 1M ops/s = 1000 ns +syscall bill at P=1 (2 per command) = 2 × 1170 ns = 2340 ns → 2.3× over budget +syscall bill at P=4 (0.5 per command) = 0.5 × 1170 = 585 ns → 59% of budget, still awful +syscall bill at P=16 (0.125 per cmd) = 0.125 × 1170 = 146 ns → 15% of budget +syscall bill at P=64 (0.031 per cmd) = 0.031 × 1170 = 37 ns → 3.7% of budget +``` + +**A single-threaded server cannot reach 1M ops/s unpipelined, on this hardware, +for arithmetic reasons that have nothing to do with how good its data +structures are.** Any "redis does a million ops per second" claim is either +pipelined or multi-threaded; when you see one, the first question is `-P` what. + +Two corrections to numbers you will meet nearby: + +- This topic's `README.md` §2 says "`redis-benchmark -P 64` is ~10× `-P 1`". + The measured lane says **66.2×** (2,919,728 / 44,088). The "~10×" is folklore + from a different machine and a different server; the number this repo can + defend is 66.2× on an M3 Pro over loopback with a do-nothing server. Real + redis will land lower, because at P=64 real work starts to matter — that gap + is precisely what your `notes.md` prediction table is asking you to guess. +- Latency does not get worse when you pipeline here: per-request time *improves* + from 22.68 µs to 0.08 µs. That is not the usual throughput-for-latency trade, + because what batching removed was pure round-trip overhead, not queueing + behind useful work. Server-side batching (group commit, topic 5) is the real + trade; this is not. + +None of this arithmetic survives if the kernel is allowed to sit on your small +writes. **Nagle's algorithm** delays sending a small TCP segment while an +earlier un-acknowledged segment is still outstanding, coalescing small writes +into fewer packets — excellent for a `telnet` session, catastrophic for a +request/reply protocol, where it interacts with delayed ACKs to add tens of +milliseconds to a round trip. **`TCP_NODELAY`** is the socket option that turns +it off. Redis sets it on every client, unconditionally, at creation: + +```c +// redis src/networking.c — createClient, 121-135 (nodelay + the 16 KiB reply buffer) + 121 client *createClient(connection *conn) { + 122 client *c = zmalloc(sizeof(client)); + 123 +// ... 124-127: comment on NULL conn for fake (Lua/AOF) clients ... + 128 if (conn) { + 129 connEnableTcpNoDelay(conn); + 130 if (server.tcpkeepalive) + 131 connKeepAlive(conn,server.tcpkeepalive); + 132 connSetReadHandler(conn, readQueryFromClient); + 133 connSetPrivateData(conn, c); + 134 } + 135 c->buf = zmalloc_usable(PROTO_REPLY_CHUNK_BYTES, &c->buf_usable_size); +``` + +The `setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, ...)` behind that call is at +`src/anet.c:258`, wrapped as `anetEnableTcpNoDelay` at `:266`. This repo's own +loopback bench sets the same option for the same reason — see the +`TCP_NODELAY` note in the `notes.md` baseline header. Note the consequence: +having disabled the kernel's write coalescing, redis has to do its own, which +is Step 6. + +### Step 5 — the parser: length-prefixed, resumable, zero-copy for big args + +> **In:** whatever bytes happen to be in `querybuf` — possibly half a command, +> possibly nine and a half. +> **Out:** a `pendingCommand` with `argv` populated, or a clean "incomplete" +> that loses nothing. + +`processMultibulkBuffer` (`networking.c:3117`) is the RESP parser, and RESP's +design (topic 7 §1) makes it nearly trivial: read `*`, then per argument +read `$` and then *exactly* len bytes. No payload byte is ever scanned or +compared. Four details repay the read. + +**The argument count sizes `argv` once.** + +```c +// redis src/networking.c — processMultibulkBuffer, 3142-3165 (parse *argc, size argv) + 3142 serverAssertWithInfo(c,NULL,c->querybuf[c->qb_pos] == '*'); +// ... 3143-3152: string2ll the count; reject non-numeric, > INT_MAX, and +// > 10 args from an unauthenticated client ... + 3153 c->qb_pos = (newline-c->querybuf)+2; + 3154 + 3155 if (ll <= 0) return C_OK; + 3156 + 3157 c->multibulklen = ll; + 3158 c->bulklen = -1; + 3159 + 3160 /* Setup argv array on pending command structure. + 3161 * Reallocate argv array when the requested size is greater than current size. */ + 3162 if (c->multibulklen > pcmd->argv_len) { + 3163 zfree(pcmd->argv); + 3164 pcmd->argv_len = min(c->multibulklen, 1024); + 3165 pcmd->argv = zmalloc(sizeof(robj*)*(pcmd->argv_len)); +``` + +Note the `min(..., 1024)` at `:3164`: a client claiming a million arguments +does not get a million-pointer allocation up front. The array grows later, as +arguments actually arrive (`:3276-3279`). + +**Resumability.** TCP is a byte stream with no message boundaries; a command +can arrive split across two `read()`s in any position — mid-length, mid-payload, +between the `\r` and the `\n`. On incomplete input the parser simply returns +`C_ERR` with no `read_error` set, leaves the bytes in `querybuf`, and stores its +progress in two fields of `struct client`: + +```c +// redis src/server.h — struct client, 1459-1461 (the entire parser resume state) + 1459 int reqtype; /* Request protocol type: PROTO_REQ_* */ + 1460 int multibulklen; /* Number of multi bulk arguments left to read. */ + 1461 long bulklen; /* Length of bulk argument in multi bulk request. */ +``` + +That is it — a request type, a count of arguments still expected, and a count +of bytes still expected of the current argument. Three integers are the entire +suspended state of a half-parsed command, which is why redis can afford to keep +ten thousand of them. (They are reset by `resetClientQbufState`, +`networking.c:2848-2852`.) Your Rust parser's partial-input resumption test is +testing exactly this property. + +**Big-argument zero-copy.** For arguments of at least `PROTO_MBULK_BIG_ARG` = +32 KiB (`server.h:191`), redis arranges for the query buffer to contain *only* +that argument, and then hands the buffer itself to the object system instead of +copying out of it: + +```c +// redis src/networking.c — processMultibulkBuffer, 3281-3300 (the sds becomes the object) + 3281 /* Optimization: if a non-master client's buffer contains JUST our bulk element + 3282 * instead of creating a new object by *copying* the sds we + 3283 * just use the current sds string. */ + 3284 if (!(c->flags & CLIENT_MASTER) && + 3285 c->qb_pos == 0 && + 3286 c->bulklen >= PROTO_MBULK_BIG_ARG && + 3287 querybuf_len == (size_t)(c->bulklen+2)) + 3288 { + 3289 (pcmd->argv)[(pcmd->argc)++] = createObject(OBJ_STRING,c->querybuf); + 3290 pcmd->argv_len_sum += c->bulklen; + 3291 c->all_argv_len_sum += c->bulklen; + 3292 sdsIncrLen(c->querybuf,-2); /* remove CRLF */ +// ... 3293-3299: give the client a fresh querybuf, sized for another fat arg +// unless that would be more than maxmemory/32 ... + 3300 sdsclear(c->querybuf); +``` + +Read the four conditions at `:3284-3287` as a specification of when this is +*safe*: not a replication link, the bulk starts at offset 0, it is at least +32 KiB, and the buffer length is *exactly* `bulklen + 2` — the buffer holds this +argument and nothing else. The last condition is not luck. It is manufactured by +the read side: `readQueryFromClient:3739-3752` detects that the next thing on +the wire is a big argument and sets `readlen` to exactly the argument's +remaining bytes, deliberately accepting more `read()` calls to buy the alignment. +The optimization fails, and falls to `createStringObject` at `:3303-3304`, the +moment a pipelined client sends anything after the big `SET` in the same +segment — which is the answer to question 3. + +**The inline fallback.** `processInlineBuffer` (`networking.c:2968`) handles +`PING\r\n` typed into `nc`: `strchr` for the newline (`:2975`), then +`sdssplitargs` on the line (`:2992`). It is the only scanning parser anywhere +in the path, it is capped at `PROTO_INLINE_MAX_SIZE` = 64 KiB +(`server.h:190`, checked at `:2979-2981`), and it exists purely so a human with +a terminal can talk to the server. Which form a client is using is decided by a +single byte: `'*'` means multibulk, anything else means inline +(`networking.c:3570-3575`). + +### Step 6 — the write path: replies are hoarded, then flushed with one `writev` + +> **In:** a batch of executed commands, each of which called `addReply*`. +> **Out:** one gathered `writev()` per client per loop turn, covering every +> reply the client accumulated — not one write per reply. + +Here is the surprise. `addReply` does **not** write to the socket: + +```c +// redis src/networking.c — addReply, 571-587 (append to a buffer, never write) + 571 /* Add the object 'obj' string representation to the client output buffer. */ + 572 void addReply(client *c, robj *obj) { + 573 if (_prepareClientToWrite(c) != C_OK) return; + 574 + 575 if (sdsEncodedObject(obj)) { + 576 _addReplyToBufferOrList(c,obj->ptr,sdslen(obj->ptr)); + 577 } else if (obj->encoding == OBJ_ENCODING_INT) { + 578 /* For integer encoded strings we just convert it into a string + 579 * using our optimized function, and attach the resulting string + 580 * to the output buffer. */ + 581 char buf[32]; + 582 size_t len = ll2string(buf,sizeof(buf),(long)obj->ptr); + 583 _addReplyToBufferOrList(c,buf,len); + 584 } else { + 585 serverPanic("Wrong obj->encoding in addReply()"); + 586 } + 587 } +``` + +`_addReplyToBufferOrList` is the two-tier buffer, and its last three lines say +everything: + +```c +// redis src/networking.c — _addReplyToBufferOrList, 485 and 517-520 (buffer, then spill) + 485 void _addReplyToBufferOrList(client *c, const char *s, size_t len) { +// ... 486-516: refuse for closing clients, disconnect a replica that replied, +// account bytes, divert push messages to a separate list ... + 517 size_t reply_len = _addReplyPayloadToBuffer(c, s, len, PLAIN_REPLY); + 518 if (len > reply_len) + 519 _addReplyPayloadToList(c, c->reply, s + reply_len, len - reply_len, PLAIN_REPLY); + 520 } +``` + +A fixed 16 KiB chunk first (`c->buf`, allocated in `createClient` at +`networking.c:135`, sized `PROTO_REPLY_CHUNK_BYTES` = 16 KiB, +`server.h:189`), and only the overflow goes to a linked list of blocks +(`c->reply`, `server.h:1462`). Small replies — which is nearly all of them — +never allocate. + +The client is then merely *flagged*, and the comment explaining why is the best +three sentences in the file: + +```c +// redis src/networking.c — putClientInPendingWriteQueue, 282-299 (flag, don't write) + 282 void putClientInPendingWriteQueue(client *c) { +// ... 283-290: skip if already flagged, or if a replica cannot receive yet ... + 291 /* Here instead of installing the write handler, we just flag the + 292 * client and put it into a list of clients that have something + 293 * to write to the socket. This way before re-entering the event + 294 * loop, we can try to directly write to the client sockets avoiding + 295 * a system call. We'll only really install the write handler if + 296 * we'll not be able to write the whole reply at once. */ + 297 c->flags |= CLIENT_PENDING_WRITE; + 298 listLinkNodeHead(server.clients_pending_write, &c->clients_pending_write_node); + 299 } +``` + +"Avoiding a system call" — the system call being avoided is the *registration* +of a write event with kqueue/epoll, plus the extra poll wakeup it would cause. +The write itself happens at the top of the next loop turn, in `beforeSleep` +(`server.c:1857`, flush at `:1998`): + +```c +// redis src/networking.c — handleClientsWithPendingWrites, 2802-2843 (one pass, one write each) + 2802 int handleClientsWithPendingWrites(void) { + 2803 listIter li; + 2804 listNode *ln; + 2805 int processed = listLength(server.clients_pending_write); + 2806 + 2807 listRewind(server.clients_pending_write,&li); + 2808 while((ln = listNext(&li))) { + 2809 client *c = listNodeValue(ln); +// ... 2810-2834: skip replicas owned by IO threads, protected and closing +// clients; hand the client to an IO thread if one is available ... + 2835 /* Try to write buffers to the client socket. */ + 2836 if (writeToClient(c,0) == C_ERR) continue; + 2837 + 2838 /* If after the synchronous writes above we still have data to + 2839 * output to the client, we need to install the writable handler. */ + 2840 if (clientHasPendingReplies(c)) { + 2841 installClientWriteHandler(c); + 2842 } + 2843 } +``` + +And `writeToClient` does not issue one write per buffer either. `_writevToClient` +builds an `iovec` array containing `c->buf` **and** the `c->reply` list nodes and +issues a single scatter-gather write: + +```c +// redis src/networking.c — _writevToClient, 2474-2495 (gather buf + list into one writev) + 2474 static int _writevToClient(client *c, ssize_t *nwritten) { + 2475 int iovmax = min(IOV_MAX, c->conn->iovcnt); + 2476 struct iovec iov[iovmax]; + 2477 ReplyIOV reply_iov = {iov, iovmax}; + 2478 + 2479 /* Add c->buf to iov array */ + 2480 if (c->bufpos > 0) { + 2481 if (likely(!c->buf_encoded)) { + 2482 /* Non-encoded buffer - add directly */ + 2483 iov[reply_iov.iovcnt].iov_base = c->buf + c->sentlen; + 2484 iov[reply_iov.iovcnt].iov_len = c->bufpos - c->sentlen; + 2485 reply_iov.iov_bytes_len += iov[reply_iov.iovcnt++].iov_len; +// ... 2486-2493: the copy-avoidance encoded-buffer path ... + 2494 /* Add c->reply list nodes to iov array */ + 2495 if (!replyIOVReachLimit(&reply_iov)) { +``` + +So: a pipeline of 100 `GET`s produces 100 `addReply` calls, one flag, zero +syscalls during execution, and **one `writev()`** at the top of the next loop +turn. Combine with Step 3 and the server's whole syscall bill for that pipeline +is one `read()` plus one `writev()` — the `4/P` arithmetic of Step 4, with the +server's half of it delivered by these two mechanisms. + +Two guards on the batching, both worth knowing: + +- `NET_MAX_WRITES_PER_EVENT` = 64 KiB (`server.h:123`) caps how much a single + normal client may be written per event, so one `KEYS *` over loopback cannot + starve the other 9,999 clients (`networking.c:2718-2734`). The cap is lifted + when over `maxmemory`, and for replicas and monitors, whose buffers would + otherwise grow without bound. +- The writable event is the *exception*, not the rule. `installClientWriteHandler` + is called only at `:2841`, only when the socket refused the whole reply. Redis + registers for write readiness only when the kernel has told it, by short + write, that it must. + +Here is the whole design as one Rust sketch. It is **not** redis code — it is +the shape you should be able to reproduce in your own server: ```rust -// processInputBuffer: drain every COMPLETE command the buffer holds. -// This loop IS pipelining: 100 commands in one read() = 100 executions, -// zero extra syscalls. -fn process_input(&mut self, c: &mut Client) { - loop { - match parse_multibulk(&c.querybuf[c.pos..]) { // *argc, then $len + bytes per arg - Parsed { cmd, consumed } => { - c.pos += consumed; - execute(&cmd, c); // addReply BUFFERS, never writes +// ILLUSTRATION — not quoted from redis. The real loop is redis src/ae.c:360 +// (aeProcessEvents), the real read side is src/networking.c:3529 +// (processInputBuffer), the real flush is src/networking.c:2802 +// (handleClientsWithPendingWrites), called from src/server.c:1998. +loop { + flush_pending_writes(&mut clients); // beforeSleep: ONE writev per client + let ready = poll.wait(next_timer()); // ONE syscall for the whole fd set + for fd in ready { + let c = &mut clients[fd]; // array, not a map — fds are dense + c.querybuf.extend(read_once(fd)); // ONE read, sized to the whole buffer + let mut batch = Vec::new(); + while batch.len() < LOOKAHEAD { + match parse_resp(&c.querybuf[c.pos..]) { + Parsed { cmd, used } => { c.pos += used; batch.push(cmd) } + Incomplete => break, // keep bytes; multibulklen/bulklen resume } - Incomplete => break, // keep the bytes; multibulklen/bulklen remember - } // where we were — resume on the next readable event + } + prefetch_keys(&batch); // the point of parsing ahead + for cmd in batch { + execute(cmd, c); // addReply BUFFERS; it never writes + } } - c.querybuf.drain(..c.pos); - c.pos = 0; } ``` -Why it matters: this is why `redis-benchmark -P 64` is ~10× `-P 1` — same -command work, 1/64th the syscalls. - -### Step 4 — the parser: length-prefixed, resumable, zero-copy for big args - -`processMultibulkBuffer` (:3117) is the RESP parser, and RESP's design -(topic 7 §1) makes it almost embarrassingly simple: read `*argc` (:3123– -3157 — the argument count, so `argv[]` is sized once), then per argument -read `$len` and then *exactly* len bytes — no scanning of payload bytes, -ever. Three details worth the read: - -- **Resumability.** TCP is a byte stream: a command can arrive split across - two `read()`s. On incomplete input the parser returns, leaves the bytes - in querybuf, and stores its progress in two fields — `multibulklen` (args - still expected) and `bulklen` (bytes still expected of the current arg), - :184–185 — resuming on the next readable event. Your Rust parser's - partial-input resumption test mirrors exactly this. -- **Big-arg zero-copy.** Args over `PROTO_MBULK_BIG_ARG` (32 KB, - server.h:191) get the querybuf *repositioned* so the arg's bytes can - become an sds string object without a copy — zero-copy for large SETs. -- **The inline fallback.** `processInlineBuffer` (:2968) handles - `PING\r\n` typed into `nc`: scan for newline (:2975), split on spaces — - the ONLY scanning parser in the path, kept purely for debuggability. - -### Step 5 — the write path: replies are hoarded, then flushed once - -The surprise: `addReply` (:572) does NOT write to the socket. It appends -the reply bytes to a per-client buffer — a fixed 16 KB chunk first -(`PROTO_REPLY_CHUNK_BYTES`), overflowing into a list of blocks so small -replies never allocate — and flags the client as pending-write. - -The actual writing happens at the *top of the next loop iteration*: -beforeSleep (Step 1) calls `handleClientsWithPendingWrites` (:2802), which -walks the pending clients and issues **one `write()` per client for all -replies accumulated this iteration**. A pipeline of 100 GETs = 100 buffered -replies = 1 syscall. Only if the socket's kernel buffer fills does redis -install a write handler and let the poll wake it when writable — the only -time redis uses write events. - -Why it matters: batching by loop iteration is the write-side twin of -Step 3 — together they make the syscall count per iteration ~2 per active -client, independent of pipeline depth. - -### Step 6 — backpressure: the buffer that grows until the axe falls - -Steps 3 and 5 both accumulate unbounded buffers, so redis needs a policy for -clients that produce faster than they consume. Input side: a client -streaming commands faster than execution grows querybuf toward a max — then -is killed. Output side: a slow reader (or one `KEYS *` returning 10M keys) -grows the reply list until `closeClientOnOutputBufferLimitReached` (grep -it) disconnects it. **Buffer-or-die**: RESP has no way to tell a producer -"slow down" (contrast pgwire's portals, topic 7 §4). - -Trace what happens when `GRAPH.QUERY` returns 1M rows through a module: -module → RedisModule_ReplyWith* → these same buffers → possibly the axe. +### Step 7 — backpressure: the buffer that grows until the axe falls + +> **In:** a client that produces faster than the server consumes, or consumes +> slower than the server produces. +> **Out:** a disconnect — because RESP has no way to say "slow down". + +Steps 3 and 6 both accumulate unbounded buffers, so redis needs a policy for +clients on either side of the loop that get out of step. It has exactly one +policy, and it is blunt. + +Input side: a client streaming commands faster than they execute grows +`querybuf`. When it crosses the limit, the client is freed: + +```c +// redis src/networking.c — readQueryFromClient, 3838-3850 (the query-buffer axe) + 3838 if (!(c->flags & CLIENT_MASTER) && +// ... 3839-3842: comment — queued MULTI args count toward the same budget ... + 3843 (c->mstate.argv_len_sums + sdslen(c->querybuf) > server.client_max_querybuf_len || + 3844 (c->mstate.argv_len_sums + sdslen(c->querybuf) > 1024*1024 && authRequired(c)))) + 3845 { + 3846 c->read_error = CLIENT_READ_REACHED_MAX_QUERYBUF; + 3847 freeClientAsync(c); + 3848 atomicIncr(server.stat_client_qbuf_limit_disconnections, 1); + 3849 goto done; + 3850 } +``` + +Note `:3844`: an *unauthenticated* client gets a hard 1 MiB ceiling regardless +of config — a pre-auth client cannot make the server allocate. + +Output side: a slow reader, or one that issued `KEYS *` against a 10M-key +database, grows `c->reply` until: + +```c +// redis src/networking.c — closeClientOnOutputBufferLimitReached, 5215-5239 (the reply axe) + 5215 int closeClientOnOutputBufferLimitReached(client *c, int async) { + 5216 if (!c->conn) return 0; /* It is unsafe to free fake clients. */ +// ... 5217-5221: assert reply_bytes sane; nothing to do if the buffer is empty ... + 5222 if (checkClientOutputBufferLimits(c)) { + 5223 sds client = catClientInfoString(sdsempty(),c); + 5224 + 5225 if (async) { + 5226 freeClientAsync(c); +// ... 5227-5235: log the disconnect at LL_WARNING, sync path calls freeClient ... + 5236 sdsfree(client); + 5237 server.stat_client_outbuf_limit_disconnections++; + 5238 return 1; + 5239 } +``` + +Both counters are exported, which tells you the maintainers expect this to +happen in production: `stat_client_qbuf_limit_disconnections` and +`stat_client_outbuf_limit_disconnections`. + +Call this **buffer-or-die**. RESP has no flow-control message — nothing a +server can send that means "pause". Contrast the pgwire chapter of this topic, +where a *portal* lets the client ask for `n` rows at a time and the server +simply stops after `n` with a `PortalSuspended`; the protocol carries the +backpressure, so nobody has to be killed. + +Now trace a module through it, because that is where this bites in FalkorDB. +`GRAPH.QUERY` returning a million rows calls `RedisModule_ReplyWith*`, and +those are thin wrappers over the same functions: + +```c +// redis src/module.c — RM_ReplyWithLongLong, 3095-3102 (module replies are addReply) + 3095 /* Send an integer reply to the client, with the specified `long long` value. + 3096 * The function always returns REDISMODULE_OK. */ + 3097 int RM_ReplyWithLongLong(RedisModuleCtx *ctx, long long ll) { + 3098 client *c = moduleGetReplyClient(ctx); + 3099 if (c == NULL) return REDISMODULE_OK; + 3100 addReplyLongLong(c,ll); + 3101 return REDISMODULE_OK; + 3102 } +``` + +So the path is: module → `RM_ReplyWith*` → `addReply*` → `c->buf` (16 KiB) → +`c->reply` list → possibly `closeClientOnOutputBufferLimitReached`. A module +cannot stream, cannot yield, and cannot be told the client is slow. It +materializes the whole reply in the server's memory and hopes. That constraint +— not query planning — is what shapes how a graph module has to paginate. ## Where each step lives in the code -Local clone at `~/repos/redis`; `src/ae.c` read fully, `src/networking.c` -only these functions: +Everything is `redis/redis@a176d1225`. Read `src/ae.c` end to end (511 lines); +from `src/networking.c` (5,775 lines) read only these. | Anchor | What | Step | |--------|------|------| -| `aeCreateEventLoop` — ae.c:47 | `events[fd]` arrays, setsize | 2 | -| `aeProcessEvents` — ae.c:360 | beforesleep :377–378, `aeApiPoll` :398 | 1 | -| `ae_kqueue.c` / `ae_epoll.c` | 4-function backend abstraction | 1 | -| `readQueryFromClient` — networking.c:3715 | 16 KB read into querybuf | 3 | -| `processInputBuffer` — networking.c:3529 | the pipelining loop | 3 | -| `processMultibulkBuffer` — networking.c:3117 | RESP parse; resumption state :184–185 | 4 | -| `processInlineBuffer` — networking.c:2968 | the `nc` fallback, newline scan :2975 | 4 | -| `addReply` — networking.c:572 | buffer, don't write | 5 | -| `handleClientsWithPendingWrites` — networking.c:2802 | flush in beforeSleep | 5 | -| `closeClientOnOutputBufferLimitReached` (grep) | the axe | 6 | -| server.h:188/189/191 | `PROTO_IOBUF_LEN`, `PROTO_REPLY_CHUNK_BYTES`, `PROTO_MBULK_BIG_ARG` | 3–5 | - -Suggested route: ae.c top to bottom (it's ~500 lines), then the read path -(Steps 3–4) as one trace, then the write path (Step 5), then grep for the -limits (Step 6). +| `aeProcessEvents` — `src/ae.c:360` | one loop turn: `beforesleep` `:377-378`, `aeApiPoll` `:398`, dispatch `:409-413` | 1 | +| `aeApiPoll` — `src/ae_kqueue.c:124` | the single `kevent()` at `:132`/`:135`; the two-pass read/write merge at `:142-173` | 1 | +| `aeSetBeforeSleepProc` — `src/server.c:3069`; `aeMain` — `:8027` | how `beforeSleep` gets wired to the loop | 1, 6 | +| `aeFileEvent` — `src/ae.h:52-57`; `aeFiredEvent` — `:73-76` | 32 B and 8 B per slot | 2 | +| `aeCreateFileEvent` — `src/ae.c:145` | grow-on-demand at `:155-166`, `setsize` cap at `:148` | 2 | +| `aeCreateEventLoop(maxclients+128)` — `src/server.c:2937` | `CONFIG_FDSET_INCR`, `src/server.h:207` | 2 | +| `readQueryFromClient` — `src/networking.c:3715` | one `connRead` at `:3798`, sized at `:3732`/`:3796`; reusable buffer `:3766-3776` | 3 | +| `processInputBuffer` — `src/networking.c:3529` | the lookahead parse loop `:3563-3567`, prefetch `:3639-3646`, execute `:3672` | 3 | +| `REDIS_DEFAULT_LOOKAHEAD 16` — `src/server.h:210` | config `lookahead`, `src/config.c:3246` | 3 | +| `connEnableTcpNoDelay` — `src/networking.c:129` | Nagle off for every client; `setsockopt` at `src/anet.c:258` | 4 | +| `processMultibulkBuffer` — `src/networking.c:3117` | `*argc` `:3142-3165`, big-arg zero-copy `:3281-3300` | 5 | +| `multibulklen` / `bulklen` — `src/server.h:1460-1461` | the entire parser resume state | 5 | +| `processInlineBuffer` — `src/networking.c:2968` | the `nc` fallback; the only scanning parse, `:2975` | 5 | +| `addReply` — `src/networking.c:572` | → `_addReplyToBufferOrList` `:485`, buffer-then-spill `:517-519` | 6 | +| `putClientInPendingWriteQueue` — `src/networking.c:282` | flag, don't write — the comment at `:291-296` | 6 | +| `handleClientsWithPendingWrites` — `src/networking.c:2802` | called from `beforeSleep`, `src/server.c:1998` | 6 | +| `_writevToClient` — `src/networking.c:2474` | one gathered write over `c->buf` + `c->reply` | 6 | +| `NET_MAX_WRITES_PER_EVENT` — `src/server.h:123` | the 64 KiB fairness cap, enforced at `:2718-2734` | 6 | +| query-buffer limit — `src/networking.c:3838-3850` | plus the 1 MiB pre-auth ceiling at `:3844` | 7 | +| `closeClientOnOutputBufferLimitReached` — `src/networking.c:5215` | the output-side axe | 7 | +| `RM_ReplyWithLongLong` — `src/module.c:3097` | modules reply through the same buffers | 7 | + +Suggested route: `ae.c` top to bottom first — it is short and it is the +skeleton. Then the read path (Steps 3 and 5) as one continuous trace from +`readQueryFromClient` to `processCommandAndResetClient`. Then the write path +(Step 6) backwards from `beforeSleep`. Then grep the two limits (Step 7). ## Questions to answer in notes.md -1. Why write in beforeSleep rather than in addReply? Count syscalls for a - pipeline of 100 GETs both ways. -2. `events[fd]` arrays vs a `HashMap`: why is the array not just - faster but *correct* here? (fd reuse semantics after close.) -3. The big-arg zero-copy: what property of sds + querybuf repositioning - makes it safe? When does it fail (arg spans two reads)? -4. Your tokio server does a write per response future by default — what's - the tokio equivalent of pending-writes batching? (Hint: buffered writer + - flush on yield, or explicit corking.) +1. Count syscalls both ways for a pipeline of 100 `GET`s: (a) as redis does it, + (b) if `addReply` wrote immediately. Then price the difference at the 1.17 µs + per `write()` this repo measured ([FINDINGS.md](../../FINDINGS.md) row 5). + What fraction of a 1 µs-per-command budget does each design spend? +2. `events[fd]` as an array versus a `HashMap`: why is the array not + merely faster but *safer*? Write down the specific bug the array makes + impossible, in terms of fd reuse after `close()`. +3. The big-argument zero-copy at `networking.c:3284-3287` has four conditions. + For each, construct a client that violates only that one, and say what redis + does instead. Which of the four is manufactured by the *read* side, and where? +4. Your tokio server writes once per response future by default. What is the + tokio equivalent of pending-writes batching, and where in your code does the + flush have to go so that it is the analogue of `beforeSleep` rather than of + `addReply`? +5. `notes.md` reports 66.2× for P=64 while this topic's `README.md` §2 says + "~10×". Both cannot be right. Which conditions would make each true, and what + does that tell you about quoting a pipelining speedup without its hardware, + its server and its workload? +6. Redis is registered level-triggered (`ae_kqueue.c:102-111`). Suppose it were + edge-triggered instead. Which single line of Step 3 becomes a bug, and what + would you have to add to `readQueryFromClient` to fix it? ## Done when -You can narrate one loop iteration with 3 pipelined clients — every syscall, -every buffer — and explain where a 101st slow client changes the story. +Answer each before unfolding it. + +- [ ] You can state, without looking, how many syscalls and how many round trips + one request costs at pipeline depth P, and why the syscall figure in + `notes.md` is arithmetic rather than a measurement. + +
+Answer + +Per *batch* of P requests: the client does one `write()` and one `read()`, the +server does one `read()` and one `writev()`, and the pair costs one round trip. +So per *request* it is `2/P` client syscalls, `4/P` total across both processes, +and `1/P` round trips. At P=1 that is 2 / 4 / 1; at P=256 it is 0.0078 / 0.0156 +/ 0.0039. + +The `notes.md` "syscalls per op" column is `2.0/P` computed from that model — +nobody ran `dtrace`. It is the client-side floor: a real client could do worse +(a partial write, a short read), never better. + +
+ +- [ ] You can perform the division that turns 44,088 ops/s into 12,321,414 ops/s + and say which part of the 279× is *not* explained by the syscall count. + +
+Answer + +Throughput is `P / T_batch`. + +``` +P = 1: T_batch = 1 / 44,088 = 22.681 µs +P = 256: T_batch = 256 / 12,321,414 = 20.777 µs +speedup = 256 × (22.681 / 20.777) = 256 × 1.0916 = 279.4× +``` + +256× of it is pure amortization: one fixed per-batch cost divided over 256 +requests. The remaining **1.09×** is *not* explained by syscall count — it is +that a batch of 256 is itself 8% cheaper to move than a batch of 1, because +larger writes amortize per-byte and per-wakeup costs too. This is why `notes.md` +calls `2/P` a floor on the improvement, not a ceiling. + +The payload is irrelevant: 40 bytes at even 10 GB/s is 4 ns, 0.02% of 22.681 µs. +The 22.681 µs is context switches and wakeups. + +
+ +- [ ] You can show, arithmetically, why a single-threaded server cannot reach + 1M ops/s without pipelining on this hardware. + +
+Answer + +1M ops/s on one thread = a 1000 ns budget per command. This repo measured +`write()` at 857k/s = **1170 ns per call** ([FINDINGS.md](../../FINDINGS.md) +row 5). Unpipelined, the server pays 2 syscalls per command: + +``` +2 × 1170 ns = 2340 ns → 2.3× over a 1000 ns budget, before parsing a byte +``` + +At P=16 the bill is `0.125 × 1170 = 146 ns`, 15% of budget, and the target +becomes reachable. So any "1M ops/s" headline is pipelined, multi-threaded, or +measured on hardware with much cheaper syscalls — and the first question to ask +is what `-P` was. + +
+ +- [ ] You can explain why `beforesleep` runs *before* `aeApiPoll` and what would + break if the flush ran after the dispatch loop instead. + +
+Answer + +`beforesleep` (`ae.c:377-378`) is the last thing that happens before the thread +blocks in `aeApiPoll` (`:398`). That is precisely the moment when all replies +generated by the *previous* turn's dispatch are complete and none can still be +appended — so it is the latest possible point at which one `writev()` per client +captures the whole batch. + +Running it after the dispatch loop would still batch, but it would sit *before* +the timer-driven work and the async-free queue that also run in `beforeSleep`, +and any reply those produced would wait a full extra loop turn. More +importantly the ordering is what lets redis skip registering a write event at +all: because the flush is guaranteed to happen before the sleep, `addReply` can +merely set a flag and link the client into `clients_pending_write` +(`networking.c:291-298`) instead of calling into the kernel. + +
+ +- [ ] You can narrate one full loop iteration with three pipelined clients — + every syscall, every buffer — and say what changes when a 101st, slow + client is added. + +
+Answer + +One turn, three clients each with a 100-command pipeline in flight: + +1. `beforeSleep` → `handleClientsWithPendingWrites`: the pending-write list is + empty on the first turn, so nothing happens. **0 syscalls.** +2. `aeApiPoll` → one `kevent()` returns 3 ready fds. **1 syscall.** +3. For each of the three fds: `readQueryFromClient` does one `connRead` into the + borrowed per-thread query buffer, up to whatever the buffer holds. **3 + syscalls.** +4. Each `processInputBuffer` parses up to `lookahead` (16) commands into + `pending_cmds`, calls `prefetchCommands()` on the batch, executes them one at + a time, and loops back to parse the next 16 — 100 commands per client, **0 + syscalls**. Each `addReply` appends into `c->buf` (16 KiB) and flags the + client once. +5. Next turn's `beforeSleep`: three clients on the pending list, one + `_writevToClient` each, gathering `c->buf` and any `c->reply` nodes into a + single `writev`. **3 syscalls.** + +Total: 7 syscalls for 300 commands, ≈ 0.023 per command. + +Add a 101st client that reads slowly: its `writev` returns short, so +`clientHasPendingReplies` is still true and `installClientWriteHandler` +(`networking.c:2841`) registers a *write* event for it — the only situation in +which redis asks the poll about writability. Its unsent bytes accumulate in +`c->reply`. Meanwhile `NET_MAX_WRITES_PER_EVENT` (64 KiB, `server.h:123`) caps +how much it can consume per event so the other 100 still get served. If it never +drains, `closeClientOnOutputBufferLimitReached` (`networking.c:5215`) +disconnects it and bumps `stat_client_outbuf_limit_disconnections` — RESP has no +way to ask it to slow down. + +
+ +- [ ] You can name the four conditions guarding the big-argument zero-copy and + say which one the read path exists to manufacture. + +
+Answer + +`networking.c:3284-3287`: (1) not a master/replication client, (2) `qb_pos == 0` +— the bulk starts at the buffer's origin, (3) `bulklen >= PROTO_MBULK_BIG_ARG` +(32 KiB), (4) `querybuf_len == bulklen + 2` — the buffer contains this argument +and *nothing else*. + +Condition (4) — and by extension (2) — is manufactured by the read side. +`readQueryFromClient:3739-3752` notices that the next thing on the wire is a big +bulk and sets `readlen` to exactly that argument's remaining bytes, accepting +extra `read()` syscalls to buy the alignment. When it holds, the query buffer's +sds *becomes* the string object (`createObject(OBJ_STRING, c->querybuf)`, +`:3289`) with the CRLF trimmed by `sdsIncrLen(..., -2)`, and the client is +handed a fresh buffer. When it fails — a pipelined client sent more bytes after +the big `SET` — the code falls to `createStringObject` at `:3303-3304` and pays +the copy. + +
+ +- [ ] You can say what `TCP_NODELAY` disables, where redis sets it, and why the + measured lane would be a different experiment without it. + +
+Answer + +Nagle's algorithm delays transmitting a small TCP segment while a previous +segment is still unacknowledged, coalescing small writes into fewer packets. +Combined with delayed ACKs on the peer, a request/reply protocol can stall for +tens of milliseconds. `TCP_NODELAY` turns it off. + +Redis sets it on **every** client, unconditionally, in `createClient` +(`networking.c:129`, `connEnableTcpNoDelay`), which reaches +`setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, ...)` at `anet.c:258`. + +Without it, the kernel would be doing coalescing of its own, and the +measured curve would no longer isolate the cost of *explicit* batching — the +P=1 number would improve for reasons that have nothing to do with the server, +and the 279× would shrink into meaninglessness. `notes.md` records +`TCP_NODELAY` in the baseline header for exactly this reason. Note the trade +redis makes: having told the kernel not to coalesce, it must coalesce itself, +which is what Step 6 is. + +
+ +- [ ] You can explain what `lookahead` buys that a one-command-at-a-time loop + cannot, and name the file that spends it. + +
+Answer + +`processInputBuffer` parses up to `server.lookahead` commands (default 16, +`server.h:210`) into `c->pending_cmds` *before* executing any of them +(`networking.c:3563-3567`). That creates a window in which the server knows +which keys the next sixteen commands will touch, so it can issue their +lookups' cache misses concurrently instead of serially: +`resetCommandsBatch()` / `addCommandToBatch()` / `prefetchCommands()` at +`:3643-3645`, declared in `src/memory_prefetch.h:22-24`. + +A one-command-at-a-time loop has no such window — each lookup's cache miss is +on the critical path of the next. The valkey chapter of this topic measures what +that is worth. Note the dependency chain: this optimization only exists because +a *pipelined client* handed the server sixteen commands in one read. Pipelining +buys syscalls first and cache parallelism second. + +
## References -**Code** -- [redis](https://github.com/redis/redis) — `src/ae.c` (read fully), - `src/networking.c` (the five functions above), plus the buffer-size - constants in `src/server.h`. Local clone at `~/repos/redis`. +**Code at this repo's pins** — all `redis/redis@a176d1225`, verified with +`tools/pinned-source.py`: + +- `src/ae.c` (511 lines, read fully) — the loop, the arrays, grow-on-demand. +- `src/ae.h` — `aeFileEvent` (32 B) and `aeFiredEvent` (8 B). +- `src/ae_kqueue.c` — the backend actually compiled on macOS. `ae_epoll.c` is + not built on your machine; see the c10k chapter of this topic. +- `src/networking.c` — `createClient` `:121`, `putClientInPendingWriteQueue` + `:282`, `addReply` `:572`, `_addReplyToBufferOrList` `:485`, + `_writevToClient` `:2474`, `writeToClient` `:2691`, + `handleClientsWithPendingWrites` `:2802`, `processInlineBuffer` `:2968`, + `processMultibulkBuffer` `:3117`, `processInputBuffer` `:3529`, + `readQueryFromClient` `:3715`, `closeClientOnOutputBufferLimitReached` `:5215`. +- `src/server.c` — `aeCreateEventLoop` `:2937`, `aeSetBeforeSleepProc` `:3069`, + `beforeSleep` `:1857` with the flush at `:1998`, `aeMain` `:8027`. +- `src/server.h` — `NET_MAX_WRITES_PER_EVENT` `:123`, `CONFIG_MIN_RESERVED_FDS` + `:143`, `PROTO_IOBUF_LEN` `:188`, `PROTO_REPLY_CHUNK_BYTES` `:189`, + `PROTO_INLINE_MAX_SIZE` `:190`, `PROTO_MBULK_BIG_ARG` `:191`, + `CONFIG_FDSET_INCR` `:207`, `REDIS_DEFAULT_LOOKAHEAD` `:210`, + `multibulklen`/`bulklen` `:1460-1461`. +- `src/anet.c:258` — the `TCP_NODELAY` `setsockopt`, wrapped at `:266`. +- `src/module.c:3097` — `RM_ReplyWithLongLong`, the module path into `addReply`. +- `src/memory_prefetch.h:22-24` — the batch-prefetch API. +- `src/config.c:3246` — the `lookahead` config, min 1, default 16. + +**Measured in this repo:** + +- [FINDINGS.md](../../FINDINGS.md) row 7 — 44k ops/s at P=1, 12.3M at P=256, + **279×**, on identical zero-work requests. Full table, machine and date in + [notes.md](notes.md). +- [FINDINGS.md](../../FINDINGS.md) row 5 — `write()` at **857k/s** (1.17 µs per + call), the syscall price used throughout Step 4. + +**Corrections made to the previous version of this chapter:** + +- `multibulklen` / `bulklen` were cited as `server.h:184-185`. They are at + **`server.h:1460-1461`**; lines 184-191 are the `PROTO_*` size constants. +- "parse one complete command, execute it, repeat" no longer describes + `processInputBuffer`. It parses up to `lookahead` (default 16) commands into + `c->pending_cmds` *first*, prefetches their keys, and only then executes — + `networking.c:3563-3567` and `:3639-3646`. +- "`aeCreateEventLoop` allocates plain arrays … `setsize` = maxclients + + headroom" was half right. `setsize` is a *cap* (`server.c:2937`, + `maxclients + 128`); the arrays start at `min(setsize, 1024)` and double on + demand in `aeCreateFileEvent` (`ae.c:155-166`). +- "reads up to 16 KB" understated it. `PROTO_IOBUF_LEN` is the starting size, + but `networking.c:3796` resets `readlen` to the whole available buffer — + "to save read(2) system calls" — and the big-argument path sets it to exactly + one argument's remaining bytes. +- "one `write()` per client" is now "one `writev()` per client": + `_writevToClient` (`:2474-2495`) gathers `c->buf` **and** the `c->reply` list + into a single scatter-gather call. +- "This is why `redis-benchmark -P 64` is ~10× `-P 1`" — the measured lane says + **66.2×** (2,919,728 / 44,088). The same "~10×" appears in this topic's + `README.md` §2 and is likewise unsupported by anything measured here. +- The unanchored Rust pseudocode is now marked `ILLUSTRATION` and points at the + three real functions it compresses. +- Removed: the claim that syscalls cost "~1–2 µs each" as a general fact. The + only syscall cost this repo has measured is `write()` at 1.17 µs + ([FINDINGS.md](../../FINDINGS.md) row 5), and that is what Step 4 uses. +- Removed: "Local clone at `~/repos/redis`". There is no clone; use + `tools/pinned-source.py`, which pins the commit these line numbers are true at. diff --git a/topics/07-networking-protocols/reading-valkey-iothreads.md b/topics/07-networking-protocols/reading-valkey-iothreads.md index 2fc0746..6b005b5 100644 --- a/topics/07-networking-protocols/reading-valkey-iothreads.md +++ b/topics/07-networking-protocols/reading-valkey-iothreads.md @@ -1,170 +1,1057 @@ # valkey io-threads: parallelize the majority, nothing else -Valkey 8 rewrote redis 6's io-threads and roughly doubled throughput — while -commands still execute on one thread with zero locks in the data structures. -Read it as a case study in *what to parallelize when you refuse to lock the -data structures*: SPSC handoff, batch commit, and a prefetcher that turns -pointer chases into a pipeline. This chapter builds those three ideas step -by step, then maps them to `io_threads.c` and `memory_prefetch.c`. This is -the "great perf PRs to study" item. +Valkey 8 rewrote redis 6's io-threads and published a 3.3× throughput increase +— while commands still execute on one thread, with zero locks in the data +structures. Read it as a case study in *what you are allowed to parallelize +when you refuse to lock the keyspace*: three queues with three different +disciplines, a one-word job handoff, a published-watermark protocol that +replaces locking, an adaptive pool that turns itself on, and a prefetcher that +turns serial pointer chases into overlapping ones. + +**Which version this chapter is about.** Every anchor below is +`valkey-io/valkey@8891441ab`. The design changed substantially between the +2024 blog posts and this commit, and this chapter flags every place they +disagree. Confirm and read with: + +``` +$ tools/pinned-source.py ref valkey +valkey valkey-io/valkey 8891441ab + +$ tools/pinned-source.py show valkey src/io_threads.c -r 1:75 +$ tools/pinned-source.py check valkey src/io_threads.c:514 --contains 'trySendReadToIOThreads' +``` + +`src/io_threads.c` is 918 lines and `src/memory_prefetch.c` is 302 — read both +in full. You do not need a local clone. ## The problem in one sentence -At 1M small ops/s, profiling shows the single redis thread spends the -*majority* of its CPU on `read()`/`write()` syscalls and RESP parsing — -command execution itself is only ~30% — so the ceiling can be roughly -doubled by parallelizing only the I/O layer, if (and only if) the handoff to -worker threads costs less than the work being handed off. +At a million small operations per second, the single valkey thread spends the +majority of its CPU not executing commands but on socket syscalls, RESP parsing +and — measured by the maintainers — `epoll_wait` alone taking "more than 20 +percent of the time", plus over 40% of main-thread time inside `lookupKey`; so +the ceiling can be raised several-fold by parallelizing *only* the I/O layer and +*only* the memory stalls, provided the handoff to worker threads costs +meaningfully less than the work being handed off. ## The concepts, step by step ### Step 1 — the contract: what may move to threads, what must not -The single-threaded command model is redis's core invariant: because -exactly one thread ever touches the keyspace, every dict/rax/listpack -operation runs with zero locks and commands are atomic by construction. -Valkey keeps that contract absolutely — commands still execute ONLY on the -main thread. What moves to io-threads: `read()`, RESP parsing, `write()`, -and (new in valkey 8) memory prefetching. - -Amdahl's law (speedup is capped by the fraction you *don't* parallelize) -says this is worth it exactly when parse+I/O dominates — i.e., small -commands, many clients. GRAPH.QUERY with 50 ms of matrix math? io-threads -buy ~nothing. GET/SET at 1M ops/s? 2×. - -### Step 2 — SPSC queues: handoff without contention - -An **SPSC queue** (single-producer single-consumer: exactly one thread ever -pushes, exactly one ever pops) needs no CAS loops at all — the producer -owns the head index, the consumer owns the tail, and one release-store -publishes each batch. Valkey gives each io-thread its own private SPSC -inbox (`io_private_inbox[IO_THREADS_MAX_NUM]`, io_threads.c:23) fed only by -the main thread: N threads, N uncontended queues, zero shared-queue -contention. - -Compare redis 6's design — worker threads spinning on one shared list with -a busy-wait fence, the main thread coordinating every batch — which burned -CPU for modest gains. The queue discipline alone is a large part of the -rewrite's win. - -Why it matters: the handoff has to be cheaper than the ~1–2 µs syscall it -offloads, or the whole scheme loses. Uncontended SPSC push is ~10 ns. - -### Step 3 — tagged pointers and batch commit: shrinking the handoff further - -Two micro-optimizations make each handoff nearly free: - -- **Tagged job pointers**: a job is one word — a pointer with the job - *type* smuggled into its unused low bits (`untagJob`, :333; pointers to - aligned objects always have zero low bits to spare). One word per job - means a batch of 8 jobs moves in a single cache line — topic 2's - bit-smuggling again. -- **Batch commit**: the producer doesn't publish each enqueue; it buffers - them and `spscCommit` (:61) publishes the whole batch with one - release-store — amortizing the fence, the same group-commit shape as - topic 5's WAL. - -On the consumer side, `IOThreadMain` (:293) drains its inbox in batches of -`BATCH_SIZE` (:320–321, `spscDequeueBatch`). - -### Step 4 — the offload decision: eligible clients, same-thread fallback - -The main thread decides per client, per event, whether to offload: -`trySendReadToIOThreads` (:514) and `trySendWriteToIOThreads` (:550) -offload only if threads are enabled and the client is eligible — and every -call site in networking.c (:2313, :3043, :6408) has a **same-thread -fallback**: if the offload can't happen, the main thread just does the work -itself, redis-style. Threads are an accelerator, not a dependency — they -can even be resized at runtime (`initIOThreads` :489, resize :476). - -### Step 5 — the clever part: prefetching the batch's dict entries - -By the time a batch of parsed commands reaches the main thread, valkey -knows every key the batch will touch — so before executing, it warms the -cache. A dict lookup is a **pointer chase** (hash → bucket → entry → -value: each load's address depends on the previous load's result, so the -~100 ns DRAM misses serialize — topic 0, Step 5). But *across* commands -the chains are independent, so `PrefetchCommandsBatch` -(memory_prefetch.c:26–33) walks all the chains **level by level**, issuing -`__builtin_prefetch` for every batch member at each level — while key A's -bucket line is in flight, it computes key B's hash: - -``` - without: exec(A): miss…wait 100ns… exec(B): miss…wait… serial misses - with: prefetch A.bucket, B.bucket, C.bucket (overlap!) - exec(A) hit, exec(B) hit, exec(C) hit misses paid once -``` - -```rust -// Walk every key's lookup path LEVEL BY LEVEL across the batch: -// while A's bucket line is in flight, compute B's hash — the pointer -// chase becomes a pipeline of overlapping misses, not a chain. -fn prefetch_batch(dict: &Dict, batch: &[Command]) { - let hashes: Vec = batch.iter().map(|c| hash(c.key())).collect(); - for &h in &hashes { - prefetch(dict.bucket_addr(h)); // level 1: all bucket lines - } - for &h in &hashes { - prefetch(dict.entry_addr(h)); // level 2: entries (buckets now warm) - } - // main thread then executes the batch: every lookup hits warm lines -} -``` - -This is software memory-level parallelism: topic 0's MLP finding (10 -independent misses in flight ≈ 10× cheaper per miss) engineered -deliberately. The file comment at memory_prefetch.c:7 states the whole -idea; keys come from up to `max_prefetch_size` commands *across multiple -clients* (question 4 asks why that matters). +> **In:** a server whose correctness rests on "exactly one thread touches the +> keyspace", and a profile saying that thread is mostly not touching the +> keyspace. +> **Out:** a partition of the work into a threadable majority and a +> non-negotiable single-threaded core — plus the Amdahl arithmetic that says +> when the split is worth anything. + +The single-threaded command model is redis's and valkey's core invariant. +Because exactly one thread ever touches the keyspace, every hashtable, rax and +listpack operation runs with **zero locks**, and every command is atomic by +construction — no transaction manager, no latch ordering, no deadlock detector. +That invariant is worth more than any throughput number, and valkey keeps it +absolutely: **commands still execute only on the main thread.** + +What moves to I/O threads is everything that is not command execution. The job +enum is the exact list, and it is short: + +```c +// valkey src/io_threads.h — the complete set of offloadable jobs, 6-22 + 6 typedef enum { + 7 JOB_REQ_READ_CLIENT = 0, + 8 JOB_REQ_WRITE_CLIENT, + 9 JOB_REQ_FREE_ARGV, + 10 JOB_REQ_FREE_OBJ, + 11 JOB_REQ_POLL, + 12 JOB_REQ_ACCEPT, + 13 JOB_REQ_COUNT + 14 } JobRequest; + 15 _Static_assert(JOB_REQ_COUNT <= 8, "JOB_REQ_COUNT must not exceed 7 for pointer arithmetic"); + 16 + 17 typedef enum { + 18 JOB_RES_READ_CLIENT = 0, + 19 JOB_RES_WRITE_CLIENT, + 20 JOB_RES_COUNT + 21 } JobResult; + 22 _Static_assert(JOB_RES_COUNT <= 8, "JOB_RES_COUNT must not exceed 7 for pointer arithmetic"); +``` + +Six request types: read a client (which includes RESP parsing), write a client, +free an `argv`, free an object, run the poll, and accept a connection. Note what +is *not* there: no "execute". Note also `JOB_REQ_POLL` — valkey offloads the +`epoll_wait` itself, which redis does not. The maintainers' stated reason is +that "when executed solely by the main thread, `epoll_wait` consumes more than +20 percent of the time" (*Unlock 1 Million RPS*, part 1, § *High Level Design*), +with the discipline that "at any given time, at most one thread, either an +io_thread or the main thread, executes `epoll_wait`". + +Now the Amdahl accounting, because this is the step where people fool +themselves. Amdahl's law says speedup is capped by the fraction you *do not* +parallelize: if a share `s` of the work stays serial, the ceiling is `1/s` no +matter how many threads you add. Work it on the maintainers' own published +numbers, which come in two stages: + +``` +Stage 1 — I/O threads alone (part 2, § "Back to Valkey"): + "reaching up to 780K SET commands per second" + and then: "Valkey's main thread was spending more than 40% of its + time in a single function: lookupKey" + + The I/O work was parallelized away, and what surfaced underneath was + not command logic — it was memory stalls. That 40% is the new serial + share, so the new ceiling is 1/0.40 = 2.5× over the 780K. + +Stage 2 — memory-access amortization (part 2, § "Batching and interleaving"): + "reduces the time spent on lookupKey by more than 80%" + "In total the impact of the memory access amortization on Valkey + performance is almost 50% and it increased the requests per second + to more than 1.19M rps" + + 780K × 1.5 ≈ 1.17M. Published figure: 1.19M. The arithmetic closes. + +Headline (part 1, § "Major Upgrade to Valkey Performance"): + 360K → 1.19M rps, "approximately 230%" increase, against Valkey 7.2 + average latency 1.792 ms → 0.542 ms + on AWS EC2 c7g.16xlarge, 8 I/O threads, 3M keys, 512-byte values, + 650 clients, sequential SET +``` + +Two things to take from that arithmetic. First, **the comparison base is valkey +7.2, not redis**, and the workload is one specific sequential-SET run on a +64-vCPU Graviton instance; quoting "valkey is 3.3× faster" without those +conditions is exactly the sin this topic's measured lane exists to punish. This +repo's own headline ([FINDINGS.md](../../FINDINGS.md) row 7) makes the same +point from the other direction: identical zero-work requests span **279×** +(44,088 ops/s at P=1 to 12,321,414 at P=256) purely by changing pipeline depth, +so a throughput number without its depth, its client count and its value size +is not a measurement of a server at all. + +Second, the two stages are the whole design. Stage 1 says *parallelize the I/O*. +Stage 2 says *once you have, the bottleneck is DRAM, not CPU* — and Step 6 is +what they did about it. + +So: `GRAPH.QUERY` with 50 ms of matrix math per call? I/O threads buy +approximately nothing — the serial share is ~100%. `GET`/`SET` at a million +ops/s with 650 concurrent clients? That is the case the numbers above describe. +Before you copy this design, measure `s` for *your* workload. + +### Step 2 — three queues, three disciplines + +> **In:** a main thread with jobs to give away and N workers with results to +> give back. +> **Out:** a handoff whose cost is a few atomic operations, chosen per direction +> — because "which threads may touch this queue" is a different question in each +> direction, and the right answer is a different queue. + +Most descriptions of valkey's I/O threads say "each thread gets its own SPSC +queue". That is one third of the truth, and it is the least important third. +The declarations are the whole map: + +```c +// valkey src/io_threads.c — the three queues and who may touch them, 17-27 + 17 static int cur_epoll_thread = 0; + 18 // Main -> IO: Shared Queue (Single Producer Multi Consumer) where all IO threads pull jobs from + 19 static spmcQueue io_shared_inbox = {0}; + 20 // IO -> Main: Response Channel (Multi Producer Single Consumer) used by IO threads to send results back to main-thread + 21 static mpscQueue io_shared_outbox = {0}; + 22 // Main -> IO (Thread-Specific) for tasks that must run on specific IO thread where IO threads check their private inbox before the shared queue + 23 static spscQueue io_private_inbox[IO_THREADS_MAX_NUM] = {0}; + 24 static size_t io_jobs_submitted; + 25 static _Atomic(size_t) io_jobs_finished; + 26 static int io_threads_initialized = 0; + 27 _Atomic long long used_active_time_io_thread[IO_THREADS_MAX_NUM] = {0}; +``` + +Read the three comments as three different answers to "who is allowed to touch +this?": + +- **`io_shared_inbox` — SPMC.** One producer (the main thread), many consumers + (every I/O thread). This is where the *actual work* goes: `JOB_REQ_READ_CLIENT` + and `JOB_REQ_WRITE_CLIENT` are enqueued here (`io_threads.c:534`). It is a + shared queue on purpose: any thread may take any client's read, so a burst of + work is load-balanced automatically with no scheduling decision by the main + thread. +- **`io_shared_outbox` — MPSC.** Many producers (every I/O thread), one consumer + (the main thread). Results come back here (`sendToMainThread`, `:769-775`). + This one *must* tolerate concurrent producers, so it costs more — and when it + is full, the producing thread spills into a thread-local + `pending_io_responses` list (`:14`, `:774-775`) rather than blocking. +- **`io_private_inbox[i]` — SPSC, one per thread.** One producer, one consumer, + the cheapest discipline there is: the producer owns the head index, the + consumer owns the tail, and no CAS loop is ever needed. These carry only jobs + that must land on a *specific* thread — `JOB_REQ_FREE_ARGV` (free the memory + on the thread that allocated it) and `JOB_REQ_POLL`. + +The consumer loop makes the priority explicit, and the comments name the +disciplines: + +```c +// valkey src/io_threads.c — IOThreadMain, 308-345 (private first, then shared) + 308 while (1) { +// ... 309-318: cancellation point; account time spent since the last turn ... + 319 processed = 0; + 320 /* PRIORITY 1: Drain Private SPSC Queue (Batch Processing) */ + 321 while ((batch_count = spscDequeueBatch(&io_private_inbox[id], batch_jobs, BATCH_SIZE)) > 0) { + 322 for (size_t i = 0; i < batch_count; i++) { + 323 void *data; + 324 int type; + 325 untagJob(batch_jobs[i], &data, &type); + 326 + 327 switch (type) { + 328 case JOB_REQ_FREE_ARGV: + 329 IOThreadFreeArgv((robj **)data); + 330 break; + 331 case JOB_REQ_POLL: + 332 IOThreadPoll((aeEventLoop *)data); + 333 break; + 334 default: + 335 serverPanic("Invalid SPSC job type: %d", type); + 336 } + 337 } + 338 processed += batch_count; + 339 } + 340 + 341 /* + 342 * PRIORITY 2: Shared Global Queue (SPMC) + 343 * Only checked after SPSC is drained. + 344 */ + 345 void *tagged_job = spmcDequeue(&io_shared_inbox); +``` + +Note the asymmetry: the private queue is drained in **batches of +`BATCH_SIZE` = 32** (`:152`, `:305`, `:321`), the shared queue one job at a time +(`:345`). Batch dequeue is exactly the SPSC discipline's payoff — the consumer +owns its index, so it can advance it 32 slots with one publish. + +And when a thread finds both queues empty it does not spin. It blocks on a +mutex the main thread holds: + +```c +// valkey src/io_threads.c — IOThreadMain, 377-386 (park on a mutex, do not spin) + 377 /* If both queues were empty (no processing done), wait for signal. */ + 378 if (processed == 0) { + 379 if (unlikely(pending_io_responses)) { + 380 flushPendingIOResponses(0); + 381 } else { + 382 /* If it is locked. We should block until main thread unlocks it. */ + 383 pthread_mutex_lock(&io_threads_mutex[id]); + 384 pthread_mutex_unlock(&io_threads_mutex[id]); + 385 } + 386 } +``` + +That is the single most important difference from redis 6's io-threads, which +had worker threads **busy-waiting** on a shared list with a spin fence: they +burned a core each while idle, which is why the feature was widely disabled in +production. Here an inactive thread costs nothing, which is what makes Step 5's +adaptive pool possible at all. + +Which queue the *poll* job uses is decided by thread count, and the comment +states the trade-off outright: + +```c +// valkey src/io_threads.c — trySendPollJobToIOThreads, 748-763 (SPMC or SPSC, by scale) + 748 /* Use SPMC to minimize polling overhead. At high thread counts, use private SPSC queues for lower latency. */ + 749 if (server.active_io_threads_num <= 9) { + 750 if (unlikely(spmcEnqueue(&io_shared_inbox, job) == false)) { +// ... 751-754: on a full queue, abandon the offload and poll on the main thread ... + 755 } else { + 756 cur_epoll_thread = ((cur_epoll_thread) % (server.active_io_threads_num - 1)) + 1; + 757 if (unlikely(spscIsFull(&io_private_inbox[cur_epoll_thread]))) { +// ... 758-760: same abandon-and-fall-back path ... + 761 } + 762 spscEnqueue(&io_private_inbox[cur_epoll_thread], job, true); + 763 } +``` + +Below ten active threads, contention on the shared queue is cheaper than the +bookkeeping of round-robining private ones; above ten, it is not. That crossover +is a *measured* engineering constant, not a principle — the lesson to steal is +that "SPSC is always better" is false, and which queue wins depends on N. + +### Step 3 — tagged pointers and batch commit: making the handoff nearly free + +> **In:** a job that is a (pointer, type) pair. +> **Out:** a single machine word in a queue slot, and a batch of them published +> with one release-store instead of one per job. + +The handoff has to be cheaper than the work it offloads or the whole scheme +loses. Two micro-optimizations get it down to a few instructions. + +**Tagged job pointers.** A job is one word — the pointer with its type smuggled +into the low bits that alignment guarantees are zero: + +```c +// valkey src/io_threads.c — the tagged-pointer job encoding, 29-42 + 29 /* Job Types for Tagged Pointers + 30 * We use the lower 3 bits of the pointer to store the job type. + 31 * Requires data pointers to be 8-byte aligned (standard for zmalloc/ptrs). */ + 32 #define JOB_TAG_MASK 0x7 + 33 #define JOB_PTR_MASK (~(uintptr_t)JOB_TAG_MASK) + 34 + 35 static inline void *tagJob(void *ptr, int type) { + 36 return (void *)((uintptr_t)ptr | type); + 37 } + 38 + 39 static inline void untagJob(void *tagged_ptr, void **ptr, int *type) { + 40 *type = (int)((uintptr_t)tagged_ptr & JOB_TAG_MASK); + 41 *ptr = (void *)((uintptr_t)tagged_ptr & JOB_PTR_MASK); + 42 } +``` + +Three bits is a budget of eight job types, and Step 1's +`_Static_assert(JOB_REQ_COUNT <= 8, ...)` (`io_threads.h:15`) is what stops +someone spending a ninth. Encoding is one `or`; decoding is two `and`s. This is +topic 2's bit-smuggling in its simplest form. + +Work the cache arithmetic, because it is the actual point. A queue slot is one +8-byte word, so a 64-byte cache line holds **8 jobs**, and `BATCH_SIZE = 32` +jobs occupy exactly **4 cache lines**: + +``` +struct { void *ptr; int type; } → 16 B per slot (12 + padding) + 4 slots per cache line + 32 jobs = 8 cache lines + +tagged single word → 8 B per slot + 8 slots per cache line + 32 jobs = 4 cache lines ← half the traffic +``` + +Every one of those lines is contended — it crosses from the producer's core to +the consumer's. Halving them halves the coherence traffic on the hottest +structure in the design. + +**Batch commit.** The producer does not publish each enqueue. It buffers them +and publishes the batch with one commit: + +```c +// valkey src/io_threads.c — commitIOJobs, 59-63 (one publish per batch, per thread) + 59 void commitIOJobs(void) { + 60 for (int i = 1; i < server.active_io_threads_num; i++) { + 61 spscCommit(&io_private_inbox[i]); + 62 } + 63 } +``` + +`spscCommit` itself lives in `src/queues.h` (included at `io_threads.c:8`); +`:61` is the call site — the point where the main thread makes a whole batch of +private-queue work visible at once. This is the same shape as topic 5's group +commit: the fence is the expensive part, so amortize it over as many items as +you can bear to delay. + +Why it matters, in one line: the handoff must cost less than a syscall. This +repo measured `write()` at **1.17 µs** ([FINDINGS.md](../../FINDINGS.md) row 5). +A tagged-pointer enqueue into an uncontended SPSC ring is a store, an index +bump, and — once per batch — one release fence: tens of nanoseconds, not +microseconds. There is roughly two orders of magnitude of headroom, which is +why the design works at all. (This repo has not measured the enqueue cost +directly; if you want the number, that is an exercise, not a citation.) + +### Step 4 — the offload decision: eligibility, and always a same-thread fallback + +> **In:** a client with a readable socket or pending replies. +> **Out:** either a job on the shared inbox, or nothing at all — in which case +> the main thread does the work itself, exactly as redis would. + +The main thread decides per client, per event. `trySendReadToIOThreads` +(`io_threads.c:514`) is a wall of eligibility checks followed by one enqueue: + +```c +// valkey src/io_threads.c — trySendReadToIOThreads, 514-544 (eligibility, then SPMC enqueue) + 514 int trySendReadToIOThreads(client *c) { + 515 if (server.active_io_threads_num <= 1) return C_ERR; + 516 /* If IO thread is already reading, return C_OK to make sure the main thread will not handle it. */ + 517 if (c->io_read_state != CLIENT_IDLE) return C_OK; + 518 if (c->io_write_state == CLIENT_PENDING_IO) return C_OK; + 519 /* For simplicity, don't offload replica clients reads as read traffic from replica is negligible */ + 520 if (getClientType(c) == CLIENT_TYPE_REPLICA) return C_ERR; + 521 /* With Lua debug client we may call connWrite directly in the main thread */ + 522 if (c->flag.lua_debug) return C_ERR; + 523 /* For simplicity let the main-thread handle the blocked clients */ + 524 if (c->flag.blocked || c->flag.unblocked) return C_ERR; + 525 if (c->flag.close_asap) return C_ERR; +// ... 526-533: stash parse/auth/replication flags on the client, mark it +// CLIENT_PENDING_IO, postpone connection state updates ... + 534 if (unlikely(spmcEnqueue(&io_shared_inbox, tagJob(c, JOB_REQ_READ_CLIENT)) == false)) { + 535 c->read_flags = 0; + 536 c->io_read_state = CLIENT_IDLE; + 537 connSetPostponeUpdateState(c->conn, 0); + 538 return C_ERR; + 539 } + 540 + 541 io_jobs_submitted++; + 542 server.stat_io_reads_pending++; + 543 c->flag.pending_read = 1; + 544 return C_OK; + 545 } +``` + +Every check is a *simplification*, not a correctness requirement — read the +comments: "for simplicity", "for simplicity", "for simplicity". Blocked clients, +replicas and Lua-debug clients are hard to reason about concurrently, so they +are simply not offloaded. That is the correct instinct for a change of this +risk: shrink the concurrent surface until you can hold it in your head. + +Note `:534-539`: if the queue is full, the function **undoes its own state +changes** and returns `C_ERR`. Everything is reversible up to the enqueue, and +the enqueue is the commit point. + +The `C_ERR` matters because every call site has a same-thread fallback: + +```c +// valkey src/networking.c — sendReplyToClient, 3040-3045 (offload, else do it here) + 3040 /* Write event handler. Just send data to the client. */ + 3041 void sendReplyToClient(connection *conn) { + 3042 client *c = connGetPrivateData(conn); + 3043 if (trySendWriteToIOThreads(c) == C_OK) return; + 3044 writeToClient(c); + 3045 } +``` + +```c +// valkey src/networking.c — handleClientsWithPendingWrites, 3258-3272 (same shape, in the flush loop) + 3258 c->flag.pending_write = 0; + 3259 listUnlinkNode(server.clients_pending_write, ln); + 3260 + 3261 if (!clientHasPendingReplies(c)) continue; + 3262 + 3263 /* If we can send the client to the I/O thread, let it handle the write. */ + 3264 if (trySendWriteToIOThreads(c) == C_OK) continue; + 3265 + 3266 /* We can't write to the client while IO operation is in progress. */ + 3267 if (c->io_write_state != CLIENT_IDLE) continue; + 3268 + 3269 processed++; + 3270 + 3271 /* Try to write buffers to the client socket. */ + 3272 if (writeToClient(c) == C_ERR) continue; +``` + +Threads are an **accelerator, not a dependency**. Set `io-threads 1` — which is +the default, "Single threaded by default" (`config.c:3375`) — and every one of +these paths falls through to the redis behaviour the previous chapter of this +topic describes, function for function. + +They are also *adaptive*. `active_io_threads_num` starts at 1 even when +`io-threads` is 8 (`io_threads.c:497`), and threads ignite only when the main +thread is visibly drowning: + +```c +// valkey src/io_threads.c — the ignition thresholds, 148-152 and 171-179 + 148 #define IO_IGNITION_EVENTS 4 + 149 #define IO_IGNITION_CPU_SYS 30.0 + 150 #define IO_IGNITION_CPU_SYS_LOW 5.0 + 151 #define IO_IGNITION_CPU_USER 50.0 + 152 #define BATCH_SIZE 32 +// ... 153-170: IOThreadsAfterSleep; the always-active policy short-circuits here ... + 171 /* Ignition Policy */ + 172 if (server.active_io_threads_num == 1) { + 173 int should_ignite = 0; + 174 #ifdef RUSAGE_THREAD + 175 float cpu_sys = (float)getInstantaneousMetric(STATS_METRIC_MAIN_THREAD_CPU_SYS) / 10000.0; + 176 float cpu_user = (float)getInstantaneousMetric(STATS_METRIC_MAIN_THREAD_CPU_USER) / 10000.0; + 177 /* Ignite IO threads if sys CPU > 30%, or if sys CPU > 5% and user CPU > 50% */ + 178 should_ignite = (cpu_sys > IO_IGNITION_CPU_SYS) || + 179 (cpu_sys > IO_IGNITION_CPU_SYS_LOW && cpu_user > IO_IGNITION_CPU_USER); +``` + +"System CPU above 30%" is Step 1's premise turned into a runtime test: *if the +main thread is spending a third of its life in the kernel, there is I/O worth +stealing.* After ignition the pool scales by queue depth, one thread at a time: + +```c +// valkey src/io_threads.c — the scaling decision, 206-218 (queue depth drives the pool) + 206 /* Decision (Every STATS_METRIC_SAMPLES Samples) */ + 207 if (sample_count % STATS_METRIC_SAMPLES != 0) return; + 208 + 209 size_t avg_q_size = getInstantaneousMetric(STATS_METRIC_IO_WAIT); + 210 size_t active = server.active_io_threads_num; + 211 size_t target = active; + 212 + 213 /* Calculate Target */ + 214 if (avg_q_size > 1 && active < (size_t)server.io_threads_num) { + 215 target++; + 216 } else if (avg_q_size == 0 && (now - last_scale_time > IO_COOLDOWN_MS)) { + 217 if (target > 1) target--; + 218 } +``` + +Scale up when the average queue is non-trivially occupied; scale down, after a +cooldown, when it is empty. `io-threads` is a *ceiling*, not a thread count (max +`IO_THREADS_MAX_NUM` = 256, `config.h:361`), and it is modifiable at runtime via +`updateIOThreads` (`io_threads.c:442`), which refuses while the response queue is +too full to drain safely (`:455-464`) — a deadlock this design has to actively +avoid, and documents. + +### Step 5 — the published watermark: how you share buffers without locking them + +> **In:** a reply buffer the main thread is appending to and an I/O thread is +> about to write to the socket. +> **Out:** a single snapshot value that tells the worker exactly how far it may +> read — and no lock anywhere. + +This is the step the design's popular summaries skip, and it is where the real +concurrency reasoning lives. If commands run on the main thread and writes run +on an I/O thread, they are both touching `c->reply` and `c->buf`. What stops +them tearing? + +Not a lock. A watermark, snapshotted by the main thread *before* the job is +enqueued: + +```c +// valkey src/io_threads.c — trySendWriteToIOThreads, 567-583 (snapshot how far the worker may go) + 567 } else { + 568 /* Save the last block of the reply list to io_last_reply_block and the used + 569 * position to io_last_bufpos. The I/O thread will write only up to + 570 * io_last_bufpos, regardless of the c->bufpos value. This is to prevent I/O + 571 * threads from reading data that might be invalid in their local CPU cache. */ + 572 c->io_last_reply_block = listLast(c->reply); + 573 if (c->io_last_reply_block) { + 574 clientReplyBlock *block = (clientReplyBlock *)listNodeValue(c->io_last_reply_block); + 575 c->io_last_bufpos = block->used; +// ... 576-577: force a fresh header if the block is encoded ... + 578 } else { + 579 c->io_last_bufpos = (size_t)c->bufpos; +// ... 580-582: same, for the static buffer ... + 583 } +``` + +Read the comment at `:569-571` slowly. The worker writes "only up to +`io_last_bufpos`, **regardless of the `c->bufpos` value**". The main thread may +keep appending past that point while the worker is running; the worker will not +look, so it cannot observe a half-written byte range, and it does not need the +main thread's writes to be visible to it at all. The bound was published once, +before the handoff, and the handoff's release-store is what makes everything +written before it visible. + +This is the general pattern, and it is worth naming because it recurs +everywhere in this repo's topics: **you do not need mutual exclusion if you can +partition the data by a value that only one side ever advances.** The main +thread owns "how much exists"; the worker owns "how much has been sent"; the +watermark is the fence between them. Compare topic 5's WAL, where the durable +LSN plays exactly this role, and topic 8's MVCC, where a snapshot timestamp +does. + +The read direction is guarded by a state machine on the client instead: +`io_read_state` and `io_write_state` move between `CLIENT_IDLE`, +`CLIENT_PENDING_IO` and `CLIENT_COMPLETED_IO` (see the guards at +`io_threads.c:517-518` and `:553-554`, and the assert in +`processClientIOReadsDone`, `networking.c:6412`). A client is owned by exactly +one thread at a time, and the state field is the token of ownership. Again: not +a lock — an invariant maintained by whose turn it is. + +### Step 6 — the clever part: prefetching the batch's lookups + +> **In:** a batch of parsed commands from the I/O threads, and a main thread +> about to execute them one at a time. +> **Out:** every one of those commands' hashtable lookups issued *concurrently* +> as cache misses, so the batch pays one DRAM latency instead of `n`. + +Once Step 1's stage 1 removed the I/O, the maintainers found their main thread +spending "more than 40% of its time in a single function: `lookupKey`" (part 2, +§ *Back to Valkey*). A hashtable lookup is a **pointer chase**: hash → bucket → +entry → value, where each load's *address* comes from the previous load's +*result*. The CPU cannot start load `n+1` before load `n` returns, so the misses +serialize. Part 2 puts the scale of the penalty plainly — external memory access +is roughly 50× L1 latency — and demonstrates it on a toy: scanning 16 linked +lists of 10 million elements each takes **20.8 seconds** sequentially on a +Graviton 3, but interleaving the 16 traversals takes **under 2 seconds** — "a 10x +speedup" — and adding `__builtin_prefetch` brings it to **1.8 s**. + +Nothing about the memory got faster. The misses simply overlapped. This repo +measured the same effect from the other end: topic 0's `lookup_shootout` finds a +HashMap probe costing **9.3 ns at n = 1e7** in a ~160 MB table where a single +*dependent* random probe "should" cost a ~100 ns DRAM miss +([FINDINGS.md](../../FINDINGS.md) row 0; the table is in +[topic 0's notes.md](../00-performance-toolbox/notes.md)) — roughly a tenfold +gap, produced by nothing but the probes being independent enough for the +out-of-order window to overlap them. + +Valkey's contribution is to *engineer* that overlap deliberately. Because the +I/O threads hand over a batch, the main thread knows every key the next `n` +commands will touch before it touches any of them. `hashtablePrefetch` then +walks all the lookups **round-robin, one step each**, rather than one lookup to +completion: + +```c +// valkey src/memory_prefetch.c — hashtablePrefetch, 158-168 (round-robin, one step per key) + 158 static void hashtablePrefetch(hashtable **tables) { + 159 initBatchInfo(tables); + 160 KeyPrefetchInfo *info; + 161 while ((info = getNextPrefetchInfo())) { + 162 switch (info->state) { + 163 case PREFETCH_ENTRY: prefetchEntry(info); break; + 164 case PREFETCH_VALUE: prefetchValue(info); break; + 165 default: serverPanic("Unknown prefetch state %d", info->state); + 166 } + 167 } + 168 } +``` + +`getNextPrefetchInfo` (`:98-106`) advances a cursor modulo the batch size and +returns the next key that is not `PREFETCH_DONE`; `prefetchEntry` (`:122-133`) +performs exactly **one** `hashtableIncrementalFindStep` and then calls +`moveToNextKey` (`:87-89`). So the loop's shape is: + +``` +key A: step 1 (issue A's bucket load, do not wait) +key B: step 1 (issue B's bucket load — A's is still in flight) +key C: step 1 ... +key A: step 2 (A's line has arrived by now; issue A's entry load) +key B: step 2 +... +``` + +The chase is not shortened. It is *turned sideways*, so `n` independent chains +progress in lockstep and their misses overlap. Note the API this rests on: +`hashtableIncrementalFindInit` / `…Step` / `…GetResult` (`:118`, `:123`, `:138`) +— the hashtable exposes a *resumable* find precisely so a caller can interleave +several. That is the reusable design lesson: to get memory-level parallelism out +of a data structure, its lookup has to be expressible as a state machine you can +step, not a function you must call to completion. + +There are two more prefetch phases before the hashtable walk, and they are not +about keys at all: + +```c +// valkey src/memory_prefetch.c — prefetchCommands, 181-213 (argv first, then the tables) + 181 static void prefetchCommands(void) { + 182 /* Prefetch argv's for all clients */ + 183 for (size_t i = 0; i < batch->client_count; i++) { + 184 client *c = batch->clients[i]; + 185 if (!c || c->argc <= 1) continue; + 186 /* Skip prefetching first argv (cmd name) it was already looked up by the I/O thread. */ + 187 for (int j = 1; j < c->argc; j++) { + 188 valkey_prefetch(c->argv[j]); + 189 } + 190 } +// ... 191-202: a second pass prefetching argv[j]->ptr for RAW-encoded objects ... + 203 /* Get the keys ptrs - we do it here after the key obj was prefetched. */ + 204 for (size_t i = 0; i < batch->key_count; i++) { + 205 batch->keys[i] = objectGetVal((robj *)batch->keys[i]); + 206 } + 207 + 208 /* Prefetch hashtable keys for all commands. Prefetching is beneficial only if there are more than one key. */ + 209 if (batch->key_count > 1) { + 210 server.stat_total_prefetch_batches++; + 211 /* Prefetch keys from the main hashtable */ + 212 hashtablePrefetch(batch->keys_tables); + 213 } + 214 } +``` + +The `argv` objects were allocated *on an I/O thread's core*, so they are cold in +the main thread's L1 — part 2 names this as a second problem it had to solve +with the same method. And `:209`: with a single key there is nothing to overlap +with, so the whole mechanism is skipped. Prefetching is only ever worth it in +batches. + +Finally, where the batch comes from — and this is the answer to the "why span +multiple clients?" question: + +```c +// valkey src/memory_prefetch.c — addCommandToBatchAndProcessIfFull, 263-289 (batch across clients AND pipelines) + 263 int addCommandToBatchAndProcessIfFull(client *c) { + 264 if (!batch) return C_ERR; + 265 + 266 batch->clients[batch->client_count++] = c; + 267 + 268 /* Client's next command */ + 269 if (c->parsed_cmd && !(c->read_flags & READ_FLAGS_BAD_ARITY)) { + 270 c->read_flags |= READ_FLAGS_PREFETCHED; + 271 addCommandToBatch(c->parsed_cmd, c->argv, c->argc, c->db, c->slot); + 272 } + 273 + 274 /* Commands in the queue. */ + 275 for (int j = c->cmd_queue.off; j < c->cmd_queue.len && batch->key_count < batch->max_prefetch_size; j++) { +// ... 276-279: add each already-parsed pipelined command's keys to the batch ... + 280 } + 281 + 282 /* If the batch is full, process it. + 283 * We also check the client count to handle cases where + 284 * no keys exist for the clients' commands. */ + 285 if (batch->client_count == batch->max_prefetch_size || batch->key_count == batch->max_prefetch_size) { + 286 processClientsCommandsBatch(); + 287 } + 288 + 289 return C_OK; + 290 } +``` + +Both sources count: `:266-272` adds the client's next command, and `:275-280` +adds everything already parsed in that client's *pipeline*. `max_prefetch_size` +is the `prefetch-batch-max-size` config, **default 16**, range 0–128 +(`config.c:3379`). So the batch fills from many clients *and* from one client's +depth — which is another way of saying that this topic's measured 279× pipelining +result ([FINDINGS.md](../../FINDINGS.md) row 7) is not only about syscalls. +Depth also feeds the prefetcher. A client that pipelines gives the server both +fewer syscalls per command *and* a wider batch to overlap misses across. + +One drift to note if you read the blog first: part 2 calls this function +`dictPrefetch` and describes a chained hash of `dictEntry`s. At this pin the +dict has been replaced by an open-addressed `hashtable`, and the function is +`hashtablePrefetch`. The idea is identical; the names and the data structure are +not. ## Where each step lives in the code -Local clone at `~/repos/valkey`: +Everything is `valkey-io/valkey@8891441ab`. Read `src/io_threads.c` (918 lines) +and `src/memory_prefetch.c` (302 lines) in full; from `src/networking.c` (6,665 +lines) read only the call sites. | Anchor | What | Step | |--------|------|------| -| `io_private_inbox[IO_THREADS_MAX_NUM]` — io_threads.c:23 | one SPSC per thread | 2 | -| `spscCommit` — io_threads.c:61 | batched producer commit | 3 | -| `IOThreadMain` — io_threads.c:293 | consumer loop, `spscDequeueBatch` :320–321 | 2–3 | -| `untagJob` — io_threads.c:333 | job type in pointer low bits | 3 | -| `initIOThreads` — io_threads.c:489 (resize :476) | setup, runtime resize | 4 | -| `trySendReadToIOThreads` — :514 / `trySendWriteToIOThreads` — :550 | offload decision | 4 | -| networking.c :2313, :3043, :6408 | call sites, each with same-thread fallback | 4 | -| `PrefetchCommandsBatch` — memory_prefetch.c:26–33 (file comment :7) | batch prefetch | 5 | - -## What to steal for M7 - -- SPSC per worker beats MPMC when you can dedicate pairs — in tokio terms: - per-connection tasks already give you this shape for free; the lesson - applies when you add a worker pool for query execution (M9). -- Batch handoff + commit, not per-item signaling. -- Prefetch only helps when execution is memory-bound on *predictable* - pointer chains — matrix kernels are already streaming; the graph-store - analogue is prefetching node/edge attribute blocks for a batch of lookups. +| `JobRequest` / `JobResult` — `src/io_threads.h:6-22` | the complete list of what may be offloaded; the `<= 8` static asserts | 1, 3 | +| `io_shared_inbox` (SPMC) — `src/io_threads.c:19` | main → any thread; carries the *read and write* jobs | 2 | +| `io_shared_outbox` (MPSC) — `src/io_threads.c:21` | threads → main; `sendToMainThread` `:769` | 2 | +| `io_private_inbox[]` (SPSC) — `src/io_threads.c:23` | main → *one specific* thread; free-argv and poll only | 2 | +| `IOThreadMain` — `src/io_threads.c:293` | private-first priority `:320-339`, shared `:345`, park on mutex `:377-386` | 2 | +| `BATCH_SIZE 32` — `src/io_threads.c:152` | `spscDequeueBatch` at `:321` | 2, 3 | +| `tagJob` / `untagJob` — `src/io_threads.c:35`, `:39` | 3-bit type in the pointer's low bits, `:29-33` | 3 | +| `commitIOJobs` — `src/io_threads.c:59` | `spscCommit` per thread at `:61`; the queue code is in `src/queues.h` | 3 | +| `trySendReadToIOThreads` — `src/io_threads.c:514` | eligibility wall, then `spmcEnqueue` at `:534`; full-queue rollback `:534-539` | 4 | +| `trySendWriteToIOThreads` — `src/io_threads.c:550` | same shape; the watermark snapshot at `:567-583` | 4, 5 | +| `sendReplyToClient` — `src/networking.c:3043` | offload, else `writeToClient` — the fallback pattern | 4 | +| `handleClientsWithPendingWrites` — `src/networking.c:3264` | same fallback inside the flush loop | 4 | +| `postponeClientRead` — `src/networking.c:6408` | the read-side entry point | 4 | +| ignition thresholds — `src/io_threads.c:148-151`, `:171-179` | threads start when main-thread sys CPU > 30% | 4 | +| scaling decision — `src/io_threads.c:206-218` | ±1 thread by average queue depth | 4 | +| `updateIOThreads` — `src/io_threads.c:442` | runtime resize; refuses under load `:455-464` | 4 | +| `io_last_bufpos` — `src/io_threads.c:567-583` | the published watermark that replaces a lock | 5 | +| `io_read_state` / `io_write_state` — `src/io_threads.c:517`, `:553` | ownership as a state machine | 5 | +| `hashtablePrefetch` — `src/memory_prefetch.c:158` | round-robin one step per key | 6 | +| `getNextPrefetchInfo` / `moveToNextKey` — `:98`, `:87` | the cursor that makes it round-robin | 6 | +| `prefetchEntry` / `prefetchValue` — `:122`, `:136` | the two states; `valkey_prefetch` at `:141` | 6 | +| `prefetchCommands` — `:181` | argv pass, argv->ptr pass, then the tables at `:209-213` | 6 | +| `addCommandToBatchAndProcessIfFull` — `:263` | batch spans clients `:266` *and* pipelines `:275` | 6 | +| `prefetch-batch-max-size` (default 16) — `src/config.c:3379` | `io-threads` (default 1) at `:3375` | 4, 6 | + +Suggested route: `io_threads.h` first — 45 lines, and the enums are the design +brief. Then `io_threads.c` lines 1-63 (the three queues and the tagging), then +`IOThreadMain`, then the two `trySend*` functions. Then `memory_prefetch.c` end +to end. Only then the `networking.c` call sites, to see how little they had to +change. + +## What to steal + +- **Pick the queue discipline per direction, not per project.** SPMC where one + producer feeds many workers, MPSC coming back, SPSC only where the job must + land on a named thread — and note that valkey switches between SPMC and SPSC + for the *same* job type at a measured crossover of 9 threads + (`io_threads.c:749`). In tokio terms you get the per-connection shape for + free; the lesson bites when you add a worker pool for query execution. +- **Batch the handoff and the commit, not per-item signalling** — and let idle + workers block on a mutex rather than spin. Redis 6's spinning io-threads are + the counter-example the whole rewrite is arguing against. +- **A published watermark beats a lock** whenever one side only ever advances a + bound. `io_last_bufpos` is the whole synchronization protocol for the write + path. +- **Expose lookups as steppable state machines** if you want callers to be able + to overlap their misses. `hashtableIncrementalFindStep` is what makes + `hashtablePrefetch` possible; a lookup that can only be called to completion + cannot be interleaved. For a graph store the analogue is a batched + node/edge-attribute fetch that can be advanced one level at a time. +- **Prefetching only pays on predictable pointer chains with a batch to + amortize over** — `:209` skips it entirely for a single key, and matrix + kernels are already streaming, so they gain nothing. ## Questions to answer in notes.md -1. Why SPSC queues instead of one MPMC queue? What does the redis-6 design - (shared list + busy spin) pay per job that SPSC doesn't? -2. Tagged job pointers: why smuggle the type in low bits instead of a - struct { type, ptr }? (Queue slot stays one word ⇒ one cache line moves - per batch of 8.) -3. Amdahl accounting for FalkorDB: measure (or estimate) parse+I/O share of - a GRAPH.QUERY round-trip; at what query cost does io-threading stop - mattering? -4. Why must prefetch batches span *multiple clients* to work? (One client's - pipeline is sequential in the buffer, but its keys are independent — - what actually limits batch depth?) +1. Valkey uses three queue disciplines. For each, say what would break (or + merely get slower) if it were replaced by an MPMC queue, and why the poll job + switches between two of them at 9 threads (`io_threads.c:749`). +2. Tagged job pointers: why smuggle the type into the low bits instead of a + `struct { void *ptr; int type; }`? Do the cache-line arithmetic for + `BATCH_SIZE = 32` both ways, and find the assertion that keeps the trick + sound. +3. `io_last_bufpos`: construct the tearing bug that would exist if the I/O + thread used `c->bufpos` instead. Then find the other place in this repo's + topics where the same "publish a bound, don't take a lock" pattern appears. +4. Amdahl accounting for FalkorDB: estimate the parse+I/O share of a + `GRAPH.QUERY` round trip. At what per-query cost does io-threading stop + mattering? Cross-check your estimate against the ignition rule at + `io_threads.c:177-179` — would your workload ever ignite the threads? +5. Why must the prefetch batch span multiple clients *and* each client's + pipeline (`memory_prefetch.c:266`, `:275`)? What actually limits batch depth, + and what does `prefetch-batch-max-size = 16` cost you if you set it to 128? +6. This chapter quotes 360K → 1.19M rps. Write down every condition under which + that was measured, then state the number you would need to see before + believing io-threads would help *your* server. ## Done when -You can explain what valkey parallelized, what it deliberately didn't, and -why the prefetcher is the same insight as topic 0's MLP experiment. +Answer each before unfolding it. + +- [ ] You can say exactly which work valkey moved off the main thread and which + it refused to move, and name the file that enumerates it. + +
+Answer + +Moved: reading a client socket (including RESP parsing), writing a client +socket, freeing `argv`, freeing objects, running the poll (`epoll_wait`), and +accepting connections. That is the complete `JobRequest` enum at +`src/io_threads.h:6-14` — six types, and `_Static_assert(JOB_REQ_COUNT <= 8)` at +`:15` caps it at the 3-bit pointer tag budget. + +Refused: **command execution**. The keyspace is still touched by exactly one +thread, which is what lets every hashtable/rax/listpack operation run without a +single lock and makes commands atomic by construction. Offloading the poll is +the one thing on this list redis does not do; the maintainers measured +`epoll_wait` at "more than 20 percent of the time" on the main thread. + +
+ +- [ ] You can name valkey's three queues, their disciplines, their directions, + and which one carries the actual read and write jobs. + +
+Answer + +From `io_threads.c:19-23`: + +- `io_shared_inbox` — **SPMC**, main → any I/O thread. This is the one that + carries `JOB_REQ_READ_CLIENT` and `JOB_REQ_WRITE_CLIENT` (`spmcEnqueue` at + `:534`). Shared on purpose: any worker takes any client's job, so load + balances itself. +- `io_shared_outbox` — **MPSC**, I/O threads → main. Results come back here via + `sendToMainThread` (`:769`); when it is full the worker spills into a + thread-local `pending_io_responses` list rather than blocking. +- `io_private_inbox[i]` — **SPSC**, main → thread `i`. Carries only work that + must land on a *specific* thread: `JOB_REQ_FREE_ARGV` and `JOB_REQ_POLL` + (`IOThreadMain:327-336`). Drained in batches of `BATCH_SIZE = 32`, because + single-consumer ownership is what makes batch dequeue cheap. + +The common summary "each thread gets its own SPSC queue" describes the *least* +used of the three. + +
+ +- [ ] You can do the cache arithmetic for tagged job pointers and find the line + that keeps the trick sound. + +
+Answer + +A `struct { void *ptr; int type; }` is 16 bytes after padding → 4 slots per +64-byte line → `BATCH_SIZE = 32` jobs span 8 lines. A tagged pointer is 8 bytes +→ 8 slots per line → 32 jobs span **4 lines**. Half the coherence traffic on the +most contended structure in the design, since every one of those lines migrates +from producer core to consumer core. + +The trick works because `zmalloc` returns 8-byte-aligned pointers, so the low 3 +bits are always zero (`io_threads.c:29-33`, `JOB_TAG_MASK 0x7`). Three bits is +eight types, and `_Static_assert(JOB_REQ_COUNT <= 8, ...)` at `io_threads.h:15` +is what stops a future contributor from silently adding a ninth and corrupting +every pointer in the queue. + +
+ +- [ ] You can explain how the write path shares `c->reply` between two threads + with no lock, and construct the bug that would exist without it. + +
+Answer + +Before enqueueing the write job, the main thread snapshots how far the worker +may go into `c->io_last_bufpos` (and `c->io_last_reply_block`), at +`io_threads.c:567-583`. The comment is explicit: the I/O thread writes "only up +to `io_last_bufpos`, **regardless of the `c->bufpos` value**". + +Without it, the worker would read `c->bufpos` live. The main thread keeps +executing commands and appending replies while the worker runs, so the worker +could observe a `bufpos` that has advanced past bytes not yet written, or a +reply-list tail being mutated underneath it — a torn read, and on some +architectures a read of a stale cached line. The watermark makes the two threads +disjoint by *value*: the main thread owns "how much exists", the worker owns +"how much has been sent", and the release-store of the enqueue publishes +everything below the bound. + +Same shape as topic 5's durable LSN and topic 8's snapshot timestamp: publish a +bound only one side advances, and you never need mutual exclusion. + +
+ +- [ ] You can explain what `hashtablePrefetch` does differently from calling + `lookupKey` n times, and why the hashtable needed a new API for it. + +
+Answer + +`lookupKey` n times runs n pointer chases *serially*: hash → bucket → entry → +value, where each load's address depends on the previous load's result, so the +CPU cannot start chase `n+1` before chase `n` finishes. n dependent DRAM misses, +paid one after another. + +`hashtablePrefetch` (`memory_prefetch.c:158-168`) runs the same n chases +**round-robin, one step each**: `getNextPrefetchInfo` (`:98`) advances a cursor +modulo the batch, `prefetchEntry` (`:122`) performs exactly one +`hashtableIncrementalFindStep` and calls `moveToNextKey` (`:87`). While key A's +bucket line is in flight, key B's is issued. The chases are not shortened — they +are overlapped, so the batch pays roughly one DRAM latency instead of n. + +The new API is `hashtableIncrementalFindInit` / `…Step` / `…GetResult` +(`:118`, `:123`, `:138`). A lookup you can only call to completion cannot be +interleaved; making the find a *resumable state machine* is the enabling change, +and that is the transferable lesson. + +
+ +- [ ] You can quote the two published stages of the speedup with their + conditions, and say what surfaced as the bottleneck after stage 1. + +
+Answer + +Stage 1, I/O threads alone: "reaching up to 780K SET commands per second" +(*Unlock 1 Million RPS*, part 2, § *Back to Valkey*). What surfaced underneath +was **not** command logic — profiling showed the main thread "spending more than +40% of its time in a single function: `lookupKey`". The bottleneck moved from +syscalls to DRAM. + +Stage 2, memory-access amortization: prefetching "reduces the time spent on +`lookupKey` by more than 80%"; total impact "almost 50%", taking it "to more +than 1.19M rps". Check: 780K × 1.5 ≈ 1.17M. + +Conditions on the headline (part 1, § *Major Upgrade to Valkey Performance*): +360K → 1.19M rps, "approximately 230%" increase, **against Valkey 7.2** — not +redis — on an AWS EC2 c7g.16xlarge, 8 I/O threads, 3M keys, 512-byte values, 650 +clients, sequential SET, with average latency 1.792 ms → 0.542 ms. Part 2 +reproduces on a c7g.4xlarge with `--io-threads 9`. Every one of those conditions +changes the number. + +
+ +- [ ] You can say why an idle I/O thread costs nothing here and did not in redis + 6, and what that enables. + +
+Answer + +When both its queues are empty, an I/O thread blocks on `pthread_mutex_lock` +(`io_threads.c:378-385`) — a mutex the main thread holds while the thread should +be inactive. It consumes no CPU. Redis 6's io-threads busy-waited on a shared +list behind a spin fence, so an enabled-but-idle thread burned a core; that is +the main reason the feature was widely left off. + +Costless idling is what makes the **adaptive pool** possible. +`active_io_threads_num` starts at 1 even when `io-threads` is 8 +(`io_threads.c:497`); threads ignite only when the main thread's system CPU +exceeds 30%, or exceeds 5% while user CPU exceeds 50% (`:148-151`, `:177-179`); +after that the pool moves ±1 thread based on average queue depth (`:206-218`), +scaling down only after a cooldown. `io-threads` is therefore a ceiling (max 256, +`config.h:361`), not a thread count — and its default is 1 +(`config.c:3375`, "Single threaded by default"). + +
+ +- [ ] You can state what happens when the offload cannot happen, and why that + makes this design safe to adopt. + +
+Answer + +Every `trySend*` path has a same-thread fallback. `trySendReadToIOThreads` +returns `C_ERR` for ineligible clients (replicas, blocked, Lua-debug, +closing — each marked "for simplicity" in the source, `io_threads.c:519-525`) +and, if the queue is full, **rolls back every state change it made** before +returning `C_ERR` (`:534-539`). The enqueue is the commit point. + +Callers then just do the work inline: `sendReplyToClient` falls through to +`writeToClient` (`networking.c:3043-3044`), `handleClientsWithPendingWrites` +does the same inside its loop (`:3264-3272`). + +So with `io-threads 1` — the default — every path degenerates to the redis +behaviour, function for function, and no client is ever *dependent* on a worker +existing. Threads are an accelerator. That is what makes a change of this +blast radius shippable. + +
## References -**Code** -- [valkey-io/valkey](https://github.com/valkey-io/valkey) — - `src/io_threads.c`, `src/memory_prefetch.c` (the file comment at :7 - states the whole idea), plus the grep points in `src/networking.c`. - Local clone at `~/repos/valkey`. +**Primary sources** — the maintainers' own write-up, quoted by section heading: + +- Uri Touitou and Alon Yagelnik, *Unlock 1 Million RPS: Experience Sharing with + Amazon ElastiCache and Valkey* (valkey.io blog, 2024-08-05) — § *Major Upgrade + to Valkey Performance* (360K → 1.19M rps, ~230%, vs Valkey 7.2, c7g.16xlarge, + 8 I/O threads, 3M keys, 512-byte values, 650 clients, sequential SET; latency + 1.792 → 0.542 ms) and § *High Level Design* (`epoll_wait` > 20% of main-thread + time; at most one thread runs it at a time; I/O threads never sleep on epoll). +- *Unlock 1 Million RPS — Part 2* (valkey.io blog, 2024-09-13) — § *Speculative + execution and linked lists* (16 × 10M-element lists: 20.8 s → under 2 s + interleaved, "a 10x speedup", → 1.8 s with `__builtin_prefetch`; external + memory ≈ 50× L1), § *Back to Valkey* (780K SET/s from I/O threads alone; + `lookupKey` > 40% of main-thread time), § *Batching and interleaving* + (prefetch cuts `lookupKey` time by > 80%; total impact "almost 50%", to + > 1.19M rps; "All the relevant code can be found in `memory_prefetch.c`"), and + the reproduce section (c7g.4xlarge, 16 aarch64 cores, `--io-threads 9`). + +**Code at this repo's pin** — all `valkey-io/valkey@8891441ab`, verified with +`tools/pinned-source.py`: + +- `src/io_threads.h` (45 lines) — the job enums and the two static asserts. +- `src/io_threads.c` (918 lines) — read in full. +- `src/memory_prefetch.c` (302 lines) — read in full; the file comment at `:6-9` + states the whole idea. +- `src/networking.c` — the five call sites: `:2313`, `:3043`, `:3227`, `:3264`, + `:6408`. +- `src/config.c:3375` (`io-threads`, default 1) and `:3379` + (`prefetch-batch-max-size`, default 16, range 0-128); + `src/config.h:361` (`IO_THREADS_MAX_NUM` 256). + +**Measured in this repo:** + +- [FINDINGS.md](../../FINDINGS.md) row 7 — 44k ops/s at P=1, 12.3M at P=256, + **279×**, on identical zero-work requests. Full table in [notes.md](notes.md). +- [FINDINGS.md](../../FINDINGS.md) row 5 — `write()` at **857k/s** (1.17 µs), the + syscall cost the handoff has to beat. +- [FINDINGS.md](../../FINDINGS.md) row 0 and + [topic 0's notes.md](../00-performance-toolbox/notes.md) — `lookup_shootout`: + 9.3 ns per *independent* HashMap probe at n = 1e7 over ~160 MB, against a + ~100 ns dependent-miss expectation. That gap is what Step 6 engineers on + purpose. + +**Corrections made to the previous version of this chapter:** + +- "Valkey gives each io-thread its own private SPSC inbox … fed only by the main + thread: N threads, N uncontended queues" described the *least* used of three + queues. Reads and writes go through the **shared SPMC** `io_shared_inbox` + (`:19`, enqueued at `:534`); results return through the **MPSC** + `io_shared_outbox` (`:21`); the private SPSC queues (`:23`) carry only + free-argv and poll jobs. +- `untagJob` was cited as `io_threads.c:333`. That is a *call site*; the + definition is at **`:39-42`**, next to `tagJob` at `:35-37`. +- `spscDequeueBatch` was cited as `:320-321`; it is at **`:321`**, and + `BATCH_SIZE` is **32** (`:152`). +- `PrefetchCommandsBatch` was described as a function at "`memory_prefetch.c:26-33`" + that "walks all the chains level by level". It is a **struct** (`:26-39`); the + walk is `hashtablePrefetch` (`:158-168`), and it is **round-robin one step per + key**, not level-by-level, driven by `hashtableIncrementalFindStep` rather than + by hand-rolled `__builtin_prefetch` on bucket addresses. There are two prefetch + states, `PREFETCH_ENTRY` and `PREFETCH_VALUE` (`:15-19`), not four levels. +- "roughly doubled throughput", "command execution itself is only ~30%", + "~1M+ ops/s/node, ~2-3× redis 7" and "uncontended SPSC push is ~10 ns" were + unsourced. Replaced with the maintainers' published figures and their + conditions (360K → 1.19M vs **Valkey 7.2**; 780K from I/O threads alone; + `lookupKey` > 40%; prefetch > 80% of that; `epoll_wait` > 20%). The SPSC push + cost is *not* replaced with a number, because neither the blog nor this repo + has measured it — the guide now states only the two-orders-of-magnitude + headroom against the 1.17 µs `write()` this repo did measure. +- "the ~1-2 µs syscall it offloads" — the only syscall cost this repo has + measured is `write()` at 1.17 µs ([FINDINGS.md](../../FINDINGS.md) row 5). +- "topic 0's MLP finding (10 independent misses in flight ≈ 10× cheaper per + miss)" was a paraphrase with an invented figure. The measured result is 9.3 ns + per independent probe at n = 1e7 against a ~100 ns dependent expectation. +- Added, because the previous version omitted them entirely: the `io_last_bufpos` + published-watermark protocol (Step 5), the adaptive ignition and scaling policy + (Step 4), the SPMC-vs-SPSC crossover at 9 threads (`:749`), and the fact that + idle threads park on a mutex rather than spin (`:377-386`). +- The unanchored Rust pseudocode has been removed in favour of the real + `hashtablePrefetch` and `prefetchCommands`, quoted with line gutters. +- Removed: "Local clone at `~/repos/valkey`". There is no clone; use + `tools/pinned-source.py`, which pins the commit these line numbers are true at. +- Note for readers of the blog: part 2's `dictPrefetch` over a chained + `dictEntry` hash is `hashtablePrefetch` over an open-addressed `hashtable` at + this pin. Same idea, different names and data structure. diff --git a/topics/08-transactions-mvcc/reading-ansi-critique.md b/topics/08-transactions-mvcc/reading-ansi-critique.md index ec42046..908d381 100644 --- a/topics/08-transactions-mvcc/reading-ansi-critique.md +++ b/topics/08-transactions-mvcc/reading-ansi-critique.md @@ -1,184 +1,831 @@ # Isolation levels, made rigorous: history patterns and write skew Berenson et al.'s SIGMOD '95 critique is the paper that made isolation -rigorous — and, accidentally, the paper that NAMED snapshot isolation and -its flaw, seven years before anyone shipped a fix. Before you open it, this +rigorous — and, in the same ten pages, the paper that first defined snapshot +isolation and named the anomaly that dethrones it. Before you open it, this chapter builds the vocabulary from zero: what a history is, why prose -definitions of isolation fail, the pattern catalog that replaced them, and -where snapshot isolation lands in the resulting hierarchy. Read it before -the SSI chapter or that one won't land. +definitions of isolation fail *in two directions at once*, the pattern catalog +that replaced them, and where snapshot isolation lands in the resulting +hierarchy. Read it before the SSI chapter or that one won't land. + +Carry the paper's own warning while you read: its subject is that the ANSI +phenomena are **ambiguous**. A summary that lists "dirty read, non-repeatable +read, phantom" as three settled bugs has already committed the error the paper +attacks — each of those three names has a strict reading and a broad reading, +the two disagree about real histories, and the paper argues at length (§3, +Remark 4) that only the broad one was intended. Worse, the strict reading of +all three *still* admits the classical inconsistent-analysis bug (§3, history +H1), and none of the six readings mentions the write anomaly the paper has to +add from scratch (P0, §3, Remark 3). + +Every claim below cites the section, table or remark it came from in the +SIGMOD '95 version (pp. 1–10, +[arXiv:cs/0701157](https://arxiv.org/abs/cs/0701157)), read in full for this +chapter. `§4.2` is a section, `Table 4` a table, `Remark 8` one of the paper's +ten numbered results, `H5` one of its named example histories. ## The problem in one sentence -ANSI SQL-92 defined its four isolation levels with three sentences of -English prose so ambiguous that the industry spent a decade shipping -incompatible things under the same names — Oracle sold snapshot isolation -labeled **SERIALIZABLE** for years, silently permitting an anomaly (write -skew) that the standard's authors never wrote down. +ANSI SQL-92 defined its four isolation levels with three sentences of English +prose (§2.2's P1, P2, P3) that each support two incompatible formal readings — +and the paper shows that under the strict reading the levels fail to exclude +executions everyone agrees are wrong (§3, H1/H2/H3), while under either reading +they omit dirty writes entirely (§3, Remark 3) and cannot tell apart isolation +levels that commercial systems were already shipping (§4) — so for a decade +"REPEATABLE READ" was a word two vendors could both honour while behaving +differently on the same workload. ## The concepts, step by step ### Step 1 — a history: concurrency reduced to one interleaved string -A **transaction** is a group of reads and writes that must behave as one -atomic unit, and a **history** is the actual interleaved order in which the -database executed the operations of several concurrent transactions. The -paper's entire method is to stop arguing about prose and write histories in -a four-symbol notation: +> **In:** nothing yet — this step fixes the notation every later step is +> written in. +> **Out:** a four-symbol shorthand for executions, plus the predicate form +> `r1[P]`, which Steps 2–10 use to state every anomaly as a pattern. + +A **transaction** groups a set of actions that transform the database from one +consistent state to another (§2.1). A **history** models the interleaved +execution of a set of transactions as a *linear ordering* of their actions — +reads and writes of specific data items (§2.1). A **data item** is deliberately +broad: "a table row, a page, an entire table, or a message on a queue" (§2.1, +following [EGLT]). + +Two actions **conflict** if they are performed by distinct transactions on the +same data item and at least one is a write (§2.1). A history's **dependency +graph** has committed transactions as nodes and one edge per conflicting pair, +oriented in the order they occurred; two histories are **equivalent** when they +have the same committed transactions and the same dependency graph, and a +history is **serializable** when it is equivalent to some *serial* history — +one that runs the transactions one at a time, in sequence (§2.1). That is the +definition the whole paper is measured against. + +The notation, introduced in §2.2 immediately after the three ANSI phenomena: ``` - r1[x] transaction 1 reads item x - w1[x] transaction 1 writes item x - c1 transaction 1 commits - a1 transaction 1 aborts + r1[x] transaction 1 reads data item x + w1[x] transaction 1 writes data item x (insert, update or delete) + c1 transaction 1 commits + a1 transaction 1 aborts (ROLLBACK) + r1[P] transaction 1 reads the set of records satisfying predicate P + w1[P] transaction 1 writes a record satisfying predicate P + r1[x=50] the same read, with the value it returned — used in §3's examples + ... "and later, in this order" ``` -So `w1[x] r2[x] a1` is a complete, unambiguous description of "T2 read T1's -uncommitted write, and then T1 aborted" — T2 read data that never existed. -One line replaces a paragraph, and two people can now check mechanically -whether a given execution is allowed. That precision is the paper's whole -contribution; everything else follows from it. +So `w1[x] ... r2[x] ... a1` says "T2 read T1's uncommitted write, and then T1 +aborted" — T2 read a value that never committed. One line replaces a paragraph, +and two people can now check *mechanically* whether a given execution matches a +forbidden shape. That precision is the paper's contribution; everything else +follows from it. -### Step 2 — isolation levels are defined by their bugs +Why it matters: from here on, "does my database permit X?" is a pattern-match +against a string, not an argument about English. -An **isolation level** is not a feature list — it is a contract about which -**anomalies** (specific broken history shapes, like the dirty read above) -the database promises to prevent. Lower levels permit more anomalies in -exchange for more concurrency; **serializable**, the top level, promises -the result is equivalent to *some* one-at-a-time execution of the -transactions — no anomaly of any shape. +### Step 2 — what ANSI actually wrote, and the table that made it famous -This bottom-up view matters because it is checkable: given a history, you -can pattern-match it against the forbidden shapes. "Which anomalies does my -level permit?" is the only question that survives contact with a real bug -report. +> **In:** the notation from Step 1. +> **Out:** the three ANSI phenomena in the paper's own words and ANSI's +> four-level table — the object Steps 3–6 dismantle. -### Step 3 — why prose fails: the strict/loose ambiguity +An **isolation level** is a contract about which **phenomena** — action +subsequences that may lead to anomalous, perhaps non-serializable, behaviour +(§1) — a transaction is forbidden to experience. The paper is careful about one +distinction most summaries drop: a **phenomenon** is a shape that *might* lead +to trouble, while an **anomaly** is an actual non-serializable outcome (§1; +"there is a technical distinction between anomalies and phenomena"). Step 3 is +where that distinction becomes the whole argument. -ANSI defined each anomaly in English, and the paper's Section 3 shows the -prose supports two incompatible readings. A *strict* reading — forbid only -the exact completed anomaly sequence — permits histories everyone agrees -are broken (e.g. two uncommitted transactions overwriting each other's -writes, which ANSI never mentions at all). A *loose* reading — forbid any -history that could ever extend into the anomaly — over-forbids and outlaws -harmless executions. Two vendors could both "conform" and behave -differently on the same workload. +ANSI SQL-92 named three, quoted here in compressed form from §2.2: + +- **P1 (Dirty Read)** — T1 modifies a data item; T2 then reads it before T1 + commits or rolls back. If T1 rolls back, T2 has read a data item that was + never committed and so never really existed. +- **P2 (Non-repeatable or Fuzzy Read)** — T1 reads a data item; T2 then + modifies or deletes it and commits; T1 rereads and gets a modified value, or + finds it gone. +- **P3 (Phantom)** — T1 reads a set of data items satisfying some search + condition; T2 then creates data items satisfying that condition and commits; + T1 repeats the read and gets a different set. + +And Table 1 (§2.2) crossed them with four levels: + +``` + Table 1 — ANSI SQL isolation levels, defined by the three original phenomena + P1 (or A1) P2 (or A2) P3 (or A3) + Dirty Read Fuzzy Read Phantom + ANSI READ UNCOMMITTED Possible Possible Possible + ANSI READ COMMITTED Not Possible Possible Possible + ANSI REPEATABLE READ Not Possible Not Possible Possible + ANOMALY SERIALIZABLE Not Possible Not Possible Not Possible +``` + +Two details of that table are already the paper talking back. First, the top +row is not called SERIALIZABLE: §2.2 notes that [ANSI] Subclause 4.28 separately +requires the SERIALIZABLE level to provide "commonly known as fully serializable +execution", and that "the prominence of the table compared to this extra proviso +leads to a common misconception that disallowing the three phenomena implies +serializability" — so the paper renames the phenomena-only level **ANOMALY +SERIALIZABLE** and keeps it distinct. Second, §2.2 warns in place that "Table 1 +is not a final result; Table 3 will supersede it" (Step 6). + +Why it matters: every "the three isolation anomalies are…" listicle you have +read is reproducing Table 1, without the two disclaimers printed on it. -The fix is Section 3's move: redefine every phenomenon as a **history -pattern** in the Step 1 notation — a shape of interleaved operations, with -no interpretation left to the reader. +### Step 3 — the fork: each phenomenon splits into a strict and a broad reading -### Step 4 — the pattern catalog: P0 through A5B +> **In:** the three English phenomena from Step 2. +> **Out:** *two* catalogs from one — strict A1/A2/A3 and broad P1/P2/P3. Step 4 +> tests them against real histories; Step 6 keeps only the broad one; Step 10's +> hierarchy needs both, because snapshot isolation sits between them. -Here is the catalog to internalize (worth the hour alone) — each line is a -forbidden interleaving shape, in Step 1's notation: +§2.2 takes P1 apart. The English does not actually insist that T1 abort — it +says that *if* it does, something unfortunate might follow. So there are two +formalisations: ``` - P0 dirty write w1[x] … w2[x] (both uncommitted) - P1 dirty read w1[x] … r2[x] before c1/a1 - P2 fuzzy read r1[x] … w2[x] before c1 - P3 phantom r1[P] … w2[y in P] predicate P, not item! - P4 lost update r1[x] … w2[x] … w1[x] - A5A read skew r1[x] … w2[x] w2[y] c2 … r1[y] - A5B write skew r1[x] r1[y] … w2[y] … w1[x] (your doctors test!) + A1 (strict): w1[x] ... r2[x] ... (a1 and c2 in either order) + P1 (broad): w1[x] ... r2[x] ... ((c1 or a1) and (c2 or a2) in any order) ``` -Two corrections in this table restructured the field: +The strict form forbids the *actual anomaly*: two of the four possible +commit/abort pairings. The broad form forbids the *phenomenon*: all four, so it +outlaws the interleaving whether or not anything bad ends up happening. §2.2: +"Interpreting (2.2) as the meaning of P1 prohibits an execution sequence if +something anomalous might [happen] in the future." -- **P3, the famous one**: ANSI's phantom was item-based; real phantoms are - **predicate**-based. `r1[P]` means "T1 ran a query whose WHERE clause is - predicate P" — and `w2[y in P]` inserts a *new* row matching P. Locking - every row you read doesn't help, because the dangerous row didn't exist - when you read. You must somehow lock rows that *would have* matched. -- **The lost-update ladder (P4)**: ANSI REPEATABLE READ, as literally - written, permits P4 — a read-modify-write silently overwriting another - transaction's committed write. Every locking implementation of RR - prevents it. The prose and the implementations had diverged for a decade - without anyone noticing, because nobody had written the patterns down. +The same split applies to the other two, giving the six patterns the paper +carries forward (§2.2): -### Step 5 — snapshot isolation: defined here, in its critics' paper +``` + P1: w1[x] ... r2[x] ... ((c1 or a1) and (c2 or a2) in any order) + A1: w1[x] ... r2[x] ... (a1 and c2 in any order) + + P2: r1[x] ... w2[x] ... ((c1 or a1) and (c2 or a2) in any order) + A2: r1[x] ... w2[x] ... c2 ... r1[x] ... c1 + + P3: r1[P] ... w2[y in P] ... ((c1 or a1) and (c2 or a2) in any order) + A3: r1[P] ... w2[y in P] ... c2 ... r1[P] ... c1 +``` -**Snapshot isolation (SI)** is the scheme where every transaction reads -from a frozen **snapshot** — the database exactly as of its start time, -ignoring all later commits — and writes are checked only against writes: -if two overlapping transactions write the same item, the **first committer -wins** and the second aborts. No reader ever blocks a writer or vice versa. +Read A2 and A3 closely: the strict forms contain the *re-read*. That is what +"non-repeatable" and "phantom" mean literally — you have to read twice and get +two answers. The broad forms P2 and P3 do not require a second read at all; +they fire the moment a concurrent write lands on something you read. -Section 4 of this paper is where SI was first formally defined — by the -people about to expose its flaw. Measured against Step 4's catalog, SI -forbids P0–P2, P4, and A5A: no dirty anything, no fuzzy reads, no lost -updates, no read skew. That places it strictly ABOVE ANSI Repeatable Read. -It looks, at first inspection, indistinguishable from serializable. +One more correction is smuggled into P3 (§2.2): "the English statement of ANSI +SQL P3 just prohibits inserts to a predicate, but P3 above intentionally +prohibits any write (insert, update, delete) affecting a tuple satisfying the +predicate once the predicate has been read." -### Step 6 — write skew: the anomaly that dethrones SI +Why it matters: this is the fork the topic README's five-row anomaly table +silently picks a side of. When someone says "repeatable read prevents +non-repeatable reads", ask which of A2 and P2 they mean; the two levels differ +on real histories (Step 4's H2), and they differ again on lost update (Step 7). -Look at A5B again: `r1[x] r1[y] … w2[y] … w1[x]`. T1 reads both items and -writes x; T2 reads both and writes y. The write sets `{x}` and `{y}` are -**disjoint** — so first-committer-wins, which only compares write sets, -sees no conflict and lets both commit. But each transaction's write was -justified by a read the other invalidated. The topic README's -doctors-on-call test is exactly this: invariant "at least one doctor on -call", both transactions verify it against their snapshot, each removes a -different doctor, both commit, invariant dead. +### Step 4 — three histories decide it: the strict reading is untenable -So SI permits A5B while locking-based Repeatable Read (which holds read -locks) forbids it — yet SI forbids phantoms that RR permits. Neither -dominates: the hierarchy is a **partial order**, not a ladder: +> **In:** the two catalogs from Step 3. +> **Out:** the paper's verdict (Remark 4 — the broad readings are the correct +> ones), argued on three concrete histories with real values. Steps 5–6 build +> the repaired catalog on top of that verdict. + +The method: exhibit a history that everybody agrees is broken, and show the +strict catalog permits it. + +**H1, against A1** (§3) — a $40 transfer between two bank balances, x and y, +which should keep x + y = 100: ``` - Serializable - / \ - SI (no P4, allows A5B) Repeatable Read (locking; no A5B via locks, - \ / allows phantoms P3) - Read Committed - | - Read Uncommitted + H1: r1[x=50] w1[x=10] r2[x=10] r2[y=50] c2 r1[y=50] w1[y=90] c1 + + T1's intent: x: 50 − 40 = 10 y: 50 + 40 = 90 10 + 90 = 100 ✓ + T2 reads: x = 10 and y = 50 ⇒ 10 + 50 = 60 ✗ (should be 100) + the shortfall: 100 − 60 = 40 — exactly the amount in flight ``` -Why it matters: Oracle shipped SI *as* "serializable" for years. Postgres -called it "repeatable read" (honest) and later added SSI on top (next -guide). And your `mvcc.rs` experiment implements Section-4 SI verbatim — -including the test that *demonstrates* write skew before you prevent it. +§3 calls this "the classical inconsistent analysis problem". Now check it +against the strict catalog: A1 needs one of the two transactions to abort — +neither does. A2 needs a data item read twice by the same transaction — nothing +is. A3 needs a predicate. **H1 violates none of A1, A2, A3, and is not +serializable.** It does violate P1 (`w1[x] ... r2[x] ...` with both committing), +so the broad reading catches it. -## How to read the paper (with the concepts in hand) +**H2, against A2** (§3) — no dirty data at all this time: + +``` + H2: r1[x=50] r2[x=50] w2[x=10] r2[y=50] w2[y=90] c2 r1[y=90] c1 + + T1 reads: x = 50 (pre-transfer) and y = 90 (post-transfer) ⇒ 140 + the truth: 100 either side of T2 ✗ 40 too much +``` + +T1 never reads anything uncommitted, so P1 is satisfied; it never reads the +same item twice, so A2 does not apply. §3: "The problem with H2 is that by the +time T1 reads y, the value for x is out of date." P2 — `r1[x] ... w2[x] ...`, +no re-read required — disqualifies it at `w2[x=10]`. + +**H3, against A3** (§3) — T1 lists the active employees, T2 inserts one and +updates the stored count z, T1 then reads z: + +``` + H3: r1[P] w2[insert y to P] r2[z] w2[z] c2 r1[z] c1 +``` + +No predicate is evaluated twice, so A3 permits it, and yet T1's list and T1's +count disagree. P3 forbids it. + +Three histories, three strict patterns defeated — hence: + +> **Remark 4** (§3). Strict interpretations A1, A2, and A3 have unintended +> weaknesses. The correct interpretations are the Broad ones. + +Why it matters: "my database forbids dirty reads, non-repeatable reads and +phantoms" is not one claim, it is two, and only the broad one implies anything +about H1 and H2. + +### Step 5 — P0, the phenomenon ANSI forgot + +> **In:** the broad catalog P1–P3, endorsed by Step 4. +> **Out:** one new pattern, P0, that no ANSI level below SERIALIZABLE excludes +> and every real locking system prevents. Step 6 folds it into the repaired +> table. + +§3 opens on a compliment — Remark 2: the locking levels of Table 2 are at least +as strong as the same-named ANSI levels. Then it asks whether they are *more* +isolated, and answers yes, at the very bottom of the ladder: + +``` + P0 (Dirty Write): w1[x] ... w2[x] ... ((c1 or a1) and (c2 or a2) in any order) +``` + +A **dirty write** is one uncommitted transaction overwriting another +uncommitted transaction's write. ANSI excludes it only at SERIALIZABLE (§3); +Locking READ UNCOMMITTED excludes it everywhere, because long-duration write +locks are the one thing every locking system holds. + +Two reasons it must be forbidden, both from §3. First, consistency. Assume a +constraint x = y, and let T1 write 1 to both while T2 writes 2 to both: + +``` + w1[x] w2[x] w2[y] c2 w1[y] c1 + + x: written 1 by T1, then 2 by T2 → 2 (T2's write survives) + y: written 2 by T2, then 1 by T1 → 1 (T1's write survives) + result x = 2, y = 1 — the constraint x = y is broken, and each transaction + alone would have preserved it. +``` + +Second, recovery. Consider `w1[x] w2[x] a1`: you cannot undo `w1[x]` by +restoring x's before-image, because that would wipe out T2's update — and if +you *don't* restore it and T2 later aborts, T2's before-image is now wrong too. +§3: "Even the weakest locking systems hold long duration write locks. +Otherwise, their recovery systems would fail." + +> **Remark 3** (§3). ANSI SQL isolation should be modified to require P0 for +> all isolation levels. + +Why it matters: this is the paper finding a bug not in a database but in the +*standard* — a phenomenon so basic that every implementation prevented it and +nobody noticed the spec didn't ask them to. + +### Step 6 — the repaired catalog, and what the patterns really are + +> **In:** the broad P1–P3 (Step 4) plus P0 (Step 5). +> **Out:** Table 3 — the four-phenomenon table that supersedes Table 1 — and +> Remark 6's punchline about what the patterns encode. Steps 7–9 add levels +> that live *between* these rows. + +§3 restates the four patterns in their final form, dropping the `(c2 or a2)` +clauses that do not restrict anything: + +``` + P0: w1[x] ... w2[x] ... (c1 or a1) Dirty Write + P1: w1[x] ... r2[x] ... (c1 or a1) Dirty Read + P2: r1[x] ... w2[x] ... (c1 or a1) Fuzzy / Non-Repeatable Read + P3: r1[P] ... w2[y in P] ... (c1 or a1) Phantom +``` + +``` + Table 3 — the levels redefined by the four phenomena (supersedes Table 1) + P0 P1 P2 P3 + Dirty Write Dirty Read Fuzzy Read Phantom + READ UNCOMMITTED Not Possible Possible Possible Possible + READ COMMITTED Not Possible Not Possible Possible Possible + REPEATABLE READ Not Possible Not Possible Not Possible Possible + SERIALIZABLE Not Possible Not Possible Not Possible Not Possible +``` + +Then the observation that explains why this table is stable where Table 1 was +not (§3): "For single version histories, it turns out that the P0, P1, P2, P3 +phenomena are disguised versions of locking." Forbidding P0 ≡ long-duration +write locks on items and predicates; forbidding P1 ≡ well-formed reads; +forbidding P2 ≡ long-duration item read locks; forbidding P3 ≡ long-duration +*predicate* read locks. + +> **Remark 6** (§3). The locking isolation levels of Table 2 and the +> phenomenological definitions of Table 3 are equivalent. Put another way, P0, +> P1, P2, and P3 are disguised redefinitions of locking behavior. + +That is the sting in the tail. ANSI's designers "sought a definition that would +admit many different implementations, not just locking" (§2.2) — and the only +repair that makes their phenomena precise turns them back into a description of +a lock manager. Which is exactly why Steps 7–9 have to leave this catalog +behind to describe a multi-version system. + +Note also §2.3 and Remark 1's ladder for locking levels — the **duration** of a +lock is the whole vocabulary there: **long duration** means held until after +commit or abort, **short duration** means released as soon as the action +completes. Locking REPEATABLE READ is precisely "long-duration read locks on +*items*, short-duration read locks on *predicates*" (Table 2) — which is why it +stops fuzzy reads and not phantoms. + +### Step 7 — P4 lost update: the level that lives between two rows + +> **In:** Table 3 from Step 6. +> **Out:** two more patterns (P4, P4C) and the first level that Table 3 cannot +> place — the shape of the argument Step 8 repeats for snapshot isolation. + +§4.1 introduces the anomaly Cursor Stability exists to prevent: + +``` + P4 (Lost Update): r1[x] ... w2[x] ... w1[x] ... c1 + P4C (Cursor Lost Update): rc1[x] ... w2[x] ... w1[x] ... c1 (rc = read cursor) +``` + +Worked on the paper's own history H4 (§4.1), where both transactions are +incrementing a balance: + +``` + H4: r1[x=100] r2[x=100] w2[x=120] c2 w1[x=130] c1 -~1.5 h. The core is §3 and §4; the rest supports them. + T2's increment: 120 − 100 = +20 committed at c2 + T1's increment: 130 − 100 = +30 computed from the stale read r1[x=100] + final x = 130 ⇒ total applied = 130 − 100 = +30 + expected if serial = +20 + 30 = +50 + lost = 50 − 30 = 20 — precisely T2's update +``` + +Where does P4 sit? §4.1: it is possible at READ COMMITTED, because forbidding +P0 or P1 does not exclude H4 — there is no read-after-write of uncommitted +data, and T2 commits before T1's write. But forbidding **P2** does exclude it, +"since w2[x] comes after r1[x] and before T1 commits or aborts". So P4 is +strictly between READ COMMITTED and REPEATABLE READ, and that gap is where a +real, widely shipped level lives: + +> **Remark 7** (§4.1). READ COMMITTED « Cursor Stability « REPEATABLE READ. + +**Cursor Stability** holds a read lock on the row the cursor is currently +positioned on, released when the cursor moves or closes; that alone converts +P4 into P4C for cursor-mediated updates (§4.1). §4.1 also notes the practical +consequence: "READ COMMITTED, in some systems, is actually the stronger Cursor +Stability. The ANSI standard allows this." + +Notation: `L1 « L2` means L1 is **weaker** than L2 — every non-serializable +history allowed by L2 is also allowed by L1, and at least one is allowed by L1 +and not L2 (§2.3). `L1 »« L2` means **incomparable**: each allows a +non-serializable history the other forbids. Step 10 needs both symbols. + +Why it matters: the pattern of this step — an anomaly that a real product's +real level prevents, sitting between two rows of the official table — is +repeated one level up, and that repetition is what produces snapshot isolation. + +### Step 8 — snapshot isolation, defined here, by the people about to break it + +> **In:** the vocabulary of Steps 1–7, and the observation from Step 7 that +> real levels fall between the official rows. +> **Out:** SI's three mechanisms (start timestamp, commit timestamp, +> first-committer-wins), and the reason single-valued histories stop being an +> adequate description. Step 9 attacks it; Step 10 places it. + +§4.2 introduces it in one paragraph, and this is where snapshot isolation gets +its name and its first formal definition: + +- **Start-Timestamp.** Each transaction reads data from a **snapshot** of the + *committed* data as of the time it started — any time before its first read. + "A transaction running in Snapshot Isolation is never blocked attempting a + read as long as the snapshot data from its Start-Timestamp can be + maintained." Updates by other transactions active after that timestamp are + invisible to it. +- **Its own writes are in the snapshot.** A transaction re-reading what it + wrote sees its own value. +- **Commit-Timestamp and first-committer-wins.** At commit T1 takes a + **Commit-Timestamp** larger than any existing start or commit timestamp. It + commits "only if no other transaction T2 with a Commit-Timestamp in T1's + execution interval [Start-Timestamp, Commit-Timestamp] wrote data that T1 + also wrote. Otherwise, T1 will abort." §4.2 names this **first-committer-wins** + and states what it buys: it "prevents lost updates (phenomenon P4)". + +Note what is compared: **write sets against write sets**. Nothing about reads +enters the test. Hold that; it is Step 9's entire content. + +SI needs a richer notation, because a data item now has several versions at +once. §4.2 rewrites Step 4's H1 as a **multi-valued (MV) history**, subscripting +each item with the transaction that produced that version: + +``` + H1.SI: r1[x0=50] w1[x1=10] r2[x0=50] r2[y0=50] c2 r1[y0=50] w1[y1=90] c1 + ↑ ↑ + T1 creates version x1 T2 still reads version x0 + T2 sees x0 + y0 = 50 + 50 = 100 ✓ — consistent, unlike H1's 60 in Step 4 +``` + +Same physical interleaving as H1, opposite verdict — because T2 reads the *old +version*, not the half-transferred one. §4.2 shows the MV history maps to a +serializable single-valued one: + +``` + H1.SI.SV: r1[x=50] r1[y=50] r2[x=50] r2[y=50] c2 w1[x=10] w1[y=90] c1 +``` + +"Mapping of MV histories to SV histories is the only rigorous touchstone needed +to place Snapshot Isolation in the Isolation Hierarchy" (§4.2). + +Why it matters: your `experiments/src/mvcc.rs` implements exactly this §4.2 +definition — snapshot at `begin`, buffered writes, write-set comparison at +commit returning `CommitError::WriteConflict`. + +### Step 9 — read skew and write skew: the anomalies ANSI has no name for + +> **In:** SI's mechanism from Step 8, in particular that only write sets are +> compared. +> **Out:** A5A and A5B, worked on the paper's H5 and on the doctors schedule +> your test suite reproduces. Step 10 uses A5B to place SI in the hierarchy. + +§4.2 first generalises what is going wrong. A **constraint violation** is the +generic anomaly: databases satisfy a constraint predicate C(DB) over multiple +items, every transaction preserves it in isolation, and a transaction that +*reads* a state violating it produces garbage. Then two named shapes (§4.2, +"A5 (Data Item Constraint Violation)"): + +``` + A5A (Read Skew): r1[x] ... w2[x] ... w2[y] ... c2 ... r1[y] ... (c1 or a1) + A5B (Write Skew): r1[x] ... r2[y] ... w1[y] ... w2[x] ... (c1 and c2 occur) +``` -1. **§1–2** — skim; motivation and the ANSI prose being critiqued. You have - the punchline already (Step 3). -2. **§3 — read carefully.** The strict/loose ambiguity argument and the - history-pattern redefinitions (Steps 3–4). Work each P-pattern by - writing out a concrete history that matches it. Don't skip the P3 - discussion — predicate vs item is the correction people still get wrong. -3. **§4 — read carefully.** The formal definition of SI and its placement - in the hierarchy (Steps 5–6). This section is the spec your `mvcc.rs` - implements. -4. **Tables and the level-hierarchy figure** — reproduce them from memory - afterwards; they compress the whole paper. +Read A5B's shape carefully, because it is the one the rest of this topic turns +on: **T1 reads x and writes y; T2 reads y and writes x.** The write sets are +{y} and {x} — disjoint. First-committer-wins compares write sets and finds +nothing. But each transaction's write was justified by a read that the other +transaction invalidated. + +§4.2's own instance, H5, with a bank constraint "x + y must stay positive" +(balances may go negative individually as long as the pair does not): + +``` + H5: r1[x=50] r1[y=50] r2[x=50] r2[y=50] w1[y=−40] w2[x=−40] c1 c2 + + T1's check: x + y = 50 + 50 = 100 > 0, so writing y = −40 leaves + 50 + (−40) = 10 > 0 ✓ (against T1's snapshot) + T2's check: x + y = 50 + 50 = 100 > 0, so writing x = −40 leaves + (−40) + 50 = 10 > 0 ✓ (against T2's snapshot) + committed: x + y = (−40) + (−40) = −80 ✗ — the constraint is dead, + and 10 − (−80) = 90 units of "safety" evaporated +``` + +The doctors-on-call form your tests use is the same pattern with booleans +(`experiments/src/mvcc.rs`, `write_skew_happens_under_snapshot_isolation`, which +*passes when the anomaly occurs* — you must be able to produce the bug before +you prevent it). Mapping it onto A5B with x = `bob_on_call`, y = `alice_on_call`: + +``` + invariant C(DB): at least one doctor on call. Initially alice=1, bob=1. + + T1 T2 A5B symbol + ────────────────────────────── ────────────────────── ────────── + begin (snapshot: alice=1,bob=1) + read bob → 1 r1[x] + begin (same snapshot) + read alice → 1 r2[y] + "bob is on call, so I may + take alice off" + write alice = 0 w1[y] + "alice is on call, so I + may take bob off" + write bob = 0 w2[x] + commit ✓ c1 + commit ✓ c2 + + committed state: alice = 0, bob = 0 — nobody on call. + T1's write set {alice}, T2's write set {bob}: intersection is EMPTY, so + first-committer-wins (Step 8) has nothing to compare and admits both. +``` + +Both transactions commit. Not one, not "the first" — **both**, and the invariant +is broken by a pair of individually correct transactions. That is the whole +indictment. + +Two sharp observations from §4.2. First: "Fuzzy Reads (P2) is a degenerate form +of Read Skew where x = y" — A5A is P2 generalised to two related items. Second, +and this is why A5A/A5B are labelled with an A: "Clearly neither A5A nor A5B +could arise in histories where P2 is precluded, since both A5A and A5B have T2 +write a data item that has been previously read by an uncommitted T1. Thus +phenomena A5A and A5B are only useful for distinguishing isolation levels that +are below REPEATABLE READ in strength." They are not new *locking* phenomena; +they are the resolution needed to describe multi-version levels that the +single-version catalog blurs together. + +Why it matters: write skew is the anomaly ANSI has no name for, and the reason +it has no name is structural — ANSI's three phenomena were written for a +single-version world, and A5B needs two transactions' *read* sets to be visible +to see anything wrong at all. + +### Step 10 — the hierarchy is a partial order, not a ladder + +> **In:** every pattern from Steps 3–9, and SI's mechanism from Step 8. +> **Out:** Table 4 reproduced faithfully, plus the three remarks that place SI +> — the answer to "is snapshot isolation strong or weak?", which is "neither". + +Table 4 (§5) is the paper's final artifact, eight phenomena wide. Reproduced in +full, including the "Sometimes Possible" cells, which are the interesting ones: + +``` + Table 4 — Isolation types characterized by possible anomalies allowed (§5) + + P0 P1 P4C P4 P2 P3 A5A A5B + Dirty Dirty Cursor Lost Fuzzy Phan- Read Write + Write Read Lost Update Read tom Skew Skew + Update + READ UNCOMMITTED Not Poss. Poss. Poss. Poss. Poss. Poss. Poss. + == Degree 1 Poss. + READ COMMITTED Not Not Poss. Poss. Poss. Poss. Poss. Poss. + == Degree 2 Poss. Poss. + Cursor Stability Not Not Not Some- Some- Poss. Poss. Some- + Poss. Poss. Poss. times times times + REPEATABLE READ Not Not Not Not Not Poss. Not Not + Poss. Poss. Poss. Poss. Poss. Poss. Poss. + Snapshot Not Not Not Not Not Some- Not POSS- + Poss. Poss. Poss. Poss. Poss. times Poss. IBLE + SERIALIZABLE Not Not Not Not Not Not Not Not + == Degree 3 Poss. Poss. Poss. Poss. Poss. Poss. Poss. Poss. +``` + +The Snapshot row is the paper's punchline in one line of a table: **seven +"Not Possible" cells and one "Possible", and the one is write skew.** Note the +Phantom cell too — "Sometimes Possible", not "Possible": §4.2 gives the case +that makes it sometimes ("a set of job tasks determined by a predicate cannot +have a sum of hours greater than 8"; two transactions each read the predicate, +each insert a task, neither write set intersects, both commit) and separately +observes that "Snapshot Isolation has no phantoms (in the strict sense of the +ANSI definitions A3)" — the strict A3 needs a *re-read*, and an SI transaction +re-reading a predicate always sees its own frozen snapshot. Strict-vs-broad, +Step 3, deciding a table cell. + +Three remarks place SI, and no two of them say the same thing: + +> **Remark 8** (§4.2). READ COMMITTED « Snapshot Isolation. +> Proof: first-committer-wins precludes P0, the timestamp mechanism prevents +> P1, and A5A is possible under READ COMMITTED but not under SI. +> +> **Remark 9** (§4.2). REPEATABLE READ »« Snapshot Isolation — *incomparable*. +> "Snapshot Isolation histories prohibit histories with anomaly A3, but allow +> A5B, while REPEATABLE READ does the opposite." +> +> **Remark 10** (§4.2). ANOMALY SERIALIZABLE « SNAPSHOT ISOLATION. +> SI precludes A1, A2 *and* A3 — so it is strictly stronger than Table 1's +> phenomena-only "SERIALIZABLE" from Step 2. + +Remark 10 is the sentence to keep. Table 1's checklist — the one every tutorial +reproduces — is *passed* by an isolation level that lets two doctors take +themselves off call simultaneously. Drawn as the partial order those remarks +define (Figure 2 in §5 draws the same lattice, and additionally places Cursor +Stability and Oracle Consistent Read on it): + +``` + SERIALIZABLE == Degree 3 + (== Date/IBM "Repeatable Read") + / \ + gap: P3 (phantom) gap: A5B (write skew) + / \ + REPEATABLE READ »« incomparable »« Snapshot Isolation + (Table 3 / locking) (Remark 9) (§4.2, first defined here) + \ / + Remark 7 \ / Remark 8 + \ / + READ COMMITTED == Degree 2 + | + READ UNCOMMITTED == Degree 1 + | + Degree 0 (no isolation but write atomicity) + + Cursor Stability sits on the left edge, strictly between READ COMMITTED and + REPEATABLE READ (Remark 7). ANOMALY SERIALIZABLE — Table 1's phenomena-only + top row — sits BELOW Snapshot Isolation (Remark 10), not at the top. +``` + +Why it matters: "stronger isolation" is not a dial. Two levels can each forbid +something the other permits, and the ladder picture in most documentation is +the reason people are surprised when a SERIALIZABLE-labelled Oracle transaction +produces write skew — a mislabelling the SSI paper confirms was still true in +2012 ("users requesting SERIALIZABLE mode actually received snapshot isolation +(as they still do in the Oracle DBMS)", Ports & Grittner, VLDB 2012, §2). + +## How to read the paper (with the concepts in hand) + +~1.5 h, ten pages. The core is §3 and §4.2; the rest supports them. + +1. **§1–2.1** — skim; motivation and the serializability vocabulary of Step 1. + Do not skip the sentence distinguishing *phenomenon* from *anomaly* (§1) — + Step 3 is built on it. +2. **§2.2 — read carefully.** The three ANSI phenomena, the notation, the + strict/broad split, Table 1 and the ANOMALY SERIALIZABLE naming (Steps 2–3). +3. **§2.3** — Table 2, the locking levels defined by lock *scope*, *mode* and + *duration*. Skim, but keep the long/short duration distinction; Remark 6 + needs it. +4. **§3 — read carefully.** Work H1, H2 and H3 yourself before reading the + verdict, matching each against A1/A2/A3 and then against P1/P2/P3 (Step 4). + Then P0 and Remark 3, then Table 3 (Steps 5–6). +5. **§4.1** — Cursor Stability, P4, P4C, H4, Remark 7 (Step 7). The shape of + the argument matters more than the level. +6. **§4.2 — read carefully, twice.** The SI definition, H1.SI, H5, A5A/A5B, and + Remarks 8–10 (Steps 8–10). This section is the spec your `mvcc.rs` + implements and the hole the next chapter fills. +7. **§4.3, §5** — Oracle Read Consistency (a *statement*-level snapshot, not a + transaction-level one), then Table 4 and Figure 2. Reproduce Table 4 from + memory afterwards; it compresses the whole paper. ## Questions for notes.md -1. Write the doctors-on-call write skew in the paper's history notation, - and show which forbidden phenomenon it does NOT match (that's why SI - lets it through). -2. Why can't first-committer-wins catch write skew? (One sentence: the - conflict is r→w across txns, not w→w.) -3. Predicate phantoms in a graph: "MATCH (n:Person) WHERE n.age > 40" - ran twice in a txn while another txn CREATEs a matching node. Which - structure would M8 need to lock/validate — a label matrix? an index - range? Is that even expressible as key locks (recall RocksDB guide Q3)? -4. Your mvcc.rs implements exactly Section-4 SI. Which tests map to which - phenomena? (Label each test with its P/A number in a comment.) +1. Write the doctors-on-call write skew in the paper's history notation, and + show which forbidden phenomenon it does NOT match — check it against all + eight columns of Table 4, not just A5B. +2. Why can't first-committer-wins catch write skew? (One sentence: the conflict + is r→w across transactions, not w→w — Step 8's comparison never looks at a + read set.) +3. Table 4 gives Snapshot "Not Possible" for P2 (Fuzzy Read) and "Possible" for + A5B (Write Skew) — but §4.2 says that in the single-valued interpretation, + "forbidding P2 also precludes A5B". Reconcile the two: which interpretation + is each cell written in, and what does that tell you about describing a + multi-version system with a single-version catalog? +4. Predicate phantoms in a graph: `MATCH (n:Person) WHERE n.age > 40` runs twice + in a transaction while another transaction CREATEs a matching node. Which + structure would M8 need to lock or validate — a label matrix? an index + range? Is that even expressible as key locks (recall the RocksDB guide's + Q3)? +5. Your `mvcc.rs` implements exactly the §4.2 definition. Which test maps to + which phenomenon? Label each with its P/A number in a comment — and say + which Table 4 columns your implementation has *no test for*. + +## Takeaway + +The catalog is the artifact: P0/P1/P2/P3 as history patterns (Table 3), P4 and +P4C for the levels between the rows, A5A and A5B for the multi-version levels +the single-version catalog cannot see. Snapshot isolation is defined in §4.2 of +its own critics' paper, and it is *strong* — it beats Table 1's entire +checklist (Remark 10) — but incomparable with locking REPEATABLE READ (Remark +9) and short of serializable by exactly one shape: A5B. + +## Connections to this topic's experiment + +Your `experiments/src/mvcc.rs` is the §4.2 spec in Rust: +`first_committer_wins_on_write_write_conflict` is the Step 8 commit rule, +`write_skew_happens_under_snapshot_isolation` is the A5B history of Step 9 and +passes *when the anomaly occurs*, and `serializable_mode_prevents_write_skew` +closes it by validating the read set — which is strictly stronger (and more +abort-happy) than the SSI machinery of the next chapter. + +What this repo has measured so far is only the *baseline* that MVCC has to +beat: one global `Mutex`, 4 threads × 50 000 transactions × 4 ops, +**623 454 / 594 264 / 676 691 txn/s** on read-heavy 95/5, write-heavy 50/50 and +64-hot-key mixes respectively ([notes.md](notes.md), Apple M3 Pro, 2026-07-28; +[FINDINGS.md](../../FINDINGS.md) row 8). The finding is the *flatness*: a mutex +cannot exploit a read-heavy mix and cannot be hurt by a hot one, because it had +already serialized everything. The `mvcc txn/s` and `aborts` columns are still +`stub` — **this repo has not measured an MVCC implementation beating a mutex, +and no claim here should be read as if it had.** Yours will produce those +numbers, and Step 9's A5B is why the Serializable column will have a non-zero +abort count that the Snapshot column does not. ## Done when -You can define SI in one sentence of history notation and name the exact -anomaly that separates it from serializable — without looking. +Answer each before unfolding it. + +- [ ] You can define snapshot isolation in one sentence of the paper's own vocabulary, and name the exact anomaly that separates it from serializable — without looking. + +
Answer + + §4.2: a transaction reads from a snapshot of the committed data as of its + Start-Timestamp (any time before its first read), sees its own writes in that + snapshot, and at commit takes a Commit-Timestamp and succeeds only if no + other transaction with a Commit-Timestamp inside its execution interval + [Start-Timestamp, Commit-Timestamp] wrote data it also wrote — + first-committer-wins. + + The anomaly is **A5B, Write Skew**: + `r1[x] ... r2[y] ... w1[y] ... w2[x] ... (c1 and c2 occur)`. It is the single + "Possible" cell in Table 4's Snapshot row. It survives because the commit + test compares write sets — {y} against {x}, disjoint — and never looks at + what either transaction read. + +
+ +- [ ] You can state the difference between A2 and P2 and give a history that separates them. + +
Answer + + A2 is the strict reading, `r1[x] ... w2[x] ... c2 ... r1[x] ... c1` — it + requires T1 to actually *re-read* x and get a different answer. P2 is the + broad reading, `r1[x] ... w2[x] ... (c1 or a1)` — it fires as soon as a + concurrent transaction writes something T1 read, re-read or not (§2.2). + + The separating history is H2 from §3: + `r1[x=50] r2[x=50] w2[x=10] r2[y=50] w2[y=90] c2 r1[y=90] c1`. T1 reads + x = 50 before T2's transfer and y = 90 after it, computing a total of 140 + where the truth is 100 on either side. Nothing is read twice, so A2 permits + it; P2 forbids it at `w2[x=10]`. Remark 4 concludes from H1, H2 and H3 that + the broad readings are the ones ANSI must have meant. + +
+ +- [ ] You can say why the paper had to invent P0, and what breaks without it. + +
Answer + + P0 (Dirty Write), `w1[x] ... w2[x] ... (c1 or a1)`, is a second uncommitted + transaction overwriting the first one's uncommitted write. ANSI excludes it + only at SERIALIZABLE (§3), yet every real locking system prevents it at every + level, because long-duration write locks are the one thing they all hold — + so the standard failed to describe even the systems it was written for + (Remark 3). + + Two things break. Consistency: with constraint x = y, T1 writing 1 to both + and T2 writing 2 to both can interleave as `w1[x] w2[x] w2[y] c2 w1[y] c1`, + leaving x = 2 and y = 1 — each transaction correct alone, the constraint dead + (§3). Recovery: in `w1[x] w2[x] a1` you cannot restore x's before-image to + undo T1 without erasing T2's update, and if you skip the restore, T2's own + before-image is now wrong should T2 abort later. §3: "Even the weakest locking + systems hold long duration write locks. Otherwise, their recovery systems + would fail." + +
+ +- [ ] You can place snapshot isolation in the hierarchy using all three of Remarks 8, 9 and 10, and explain why the picture is not a ladder. + +
Answer + + Remark 8: READ COMMITTED « Snapshot Isolation — SI is strictly stronger, + because first-committer-wins precludes P0, the timestamp mechanism precludes + P1, and read skew A5A is possible under READ COMMITTED but not under SI. + Remark 9: REPEATABLE READ »« Snapshot Isolation — *incomparable*: SI forbids + A3 (a re-read of a predicate always returns the frozen snapshot) which + locking REPEATABLE READ permits, while REPEATABLE READ forbids A5B (its + long-duration item read locks conflict with the other transaction's write) + which SI permits. Remark 10: ANOMALY SERIALIZABLE « SNAPSHOT ISOLATION — SI + precludes A1, A2 and A3, so it beats Table 1's entire checklist. + + It is not a ladder because Remark 9's »« is a real incomparability, not a + missing measurement: neither level's permitted-history set contains the + other's. Any single "isolation strength" number would have to order two + levels that genuinely do not compare, which is exactly the mistake that lets + a system advertise Table 1 compliance (Remark 10) while permitting the + doctors bug. + +
+ +- [ ] You can explain why write skew has no ANSI name, structurally — not just historically. + +
Answer + + Because ANSI's three phenomena are written entirely in terms of one + transaction's *reads* colliding with another's *writes on the same item*, and + in A5B — `r1[x] ... r2[y] ... w1[y] ... w2[x]` — no item is both read and + written by the same transaction, and the two write sets are disjoint. There + is nothing for an item-scoped, single-version phenomenon to point at. Seeing + the bug requires holding *both* transactions' read sets and both write sets + at once and noticing that the reads justified the writes. + + §4.2 makes the same point from the other side: A5A and A5B "could [not] arise + in histories where P2 is precluded, since both A5A and A5B have T2 write a + data item that has been previously read by an uncommitted T1. Thus phenomena + A5A and A5B are only useful for distinguishing isolation levels that are + below REPEATABLE READ in strength." In a locking world they are redundant; + they only become *necessary* once a system stops taking read locks at all — + which is the entire design of the multi-version levels §4.2 had to add. + +
## References **Papers** -- Berenson, Bernstein, Gray, Melton, O'Neil, O'Neil — "A Critique of ANSI - SQL Isolation Levels" (SIGMOD 1995, - [arXiv:cs/0701157](https://arxiv.org/abs/cs/0701157)) — ~1.5 h; §3's - history notation and §4's SI definition are the core +- Berenson, Bernstein, Gray, Melton, O'Neil, O'Neil — "A Critique of ANSI SQL + Isolation Levels" (SIGMOD 1995, pp. 1–10, + [arXiv:cs/0701157](https://arxiv.org/abs/cs/0701157)) — ~1.5 h. Anchors used + in this chapter: + +| Where | What | +|---|---| +| §1 | phenomenon vs anomaly; the four ANSI levels | +| §2.1 | transaction, history, conflict, dependency graph, serializable | +| §2.2 | P1/P2/P3 in English; the history notation; A1–A3 vs P1–P3; Table 1; ANOMALY SERIALIZABLE | +| §2.3 | Table 2 — locking levels by scope, mode and duration; Remark 1 | +| §3 | Remark 2; H1, H2, H3; Remark 4 (broad wins); P0 and Remark 3; Table 3; Remark 6 | +| §4.1 | Cursor Stability; P4, P4C; H4; Remark 7 | +| §4.2 | Snapshot Isolation defined; H1.SI, H1.SI.SV; A5A, A5B; H5; Remarks 8, 9, 10 | +| §4.3 | Oracle Read Consistency — a per-statement snapshot, no first-committer-wins | +| §5 | Table 4 and Figure 2 — the full lattice | + +- Ports & Grittner — "Serializable Snapshot Isolation in PostgreSQL" + (VLDB 2012, [arXiv:1208.4179](https://arxiv.org/abs/1208.4179)) — §2 for the + confirmation that SERIALIZABLE-means-SI was still shipping in 2012, and §2.1.1 + for the doctors-on-call figure. The next chapter, + [reading-ssi-postgres.md](reading-ssi-postgres.md), reads it in full. diff --git a/topics/08-transactions-mvcc/reading-inmemory-mvcc.md b/topics/08-transactions-mvcc/reading-inmemory-mvcc.md index 12bbf60..2637679 100644 --- a/topics/08-transactions-mvcc/reading-inmemory-mvcc.md +++ b/topics/08-transactions-mvcc/reading-inmemory-mvcc.md @@ -4,235 +4,877 @@ What does MVCC look like when the disk-era assumptions are deleted? Hekaton (SIGMOD '13) answers with one design — no locks, no latches, no pages; Wu & Pavlo's VLDB '17 evaluation answers with the whole design SPACE, benchmark-backed prices attached. This chapter builds Hekaton's -machine one move at a time, then lays out the five-axis menu. Read Hekaton -first (a design), then Wu/Pavlo (the menu). +machine one move at a time, then walks the four design decisions Wu & +Pavlo isolate and prices each one. Read Hekaton first (a design), then +Wu/Pavlo (the menu). + +Two papers, cited throughout by their own section, table and figure +numbers so you can check every claim: + +- **Hekaton** — Diaconu, Freedman, Ismert, Larson, Mittal, Stonecipher, + Verma, Zwilling, *Hekaton: SQL Server's Memory-Optimized OLTP Engine*, + SIGMOD 2013. Cited as "Hekaton §N". +- **Wu/Pavlo** — Wu, Arulraj, Lin, Xian, Pavlo, *An Empirical Evaluation + of In-Memory Multi-Version Concurrency Control*, VLDB 2017 + ([PDF](https://db.cs.cmu.edu/papers/2017/p781-wu.pdf)). Cited as + "Wu §N", with figure and table numbers. + +Every number below carries the section, table or figure it came from. If a +claim here has no such tag, it is not from the papers and you should +distrust it. ## The problem in one sentence -When every data access costs ~100 ns instead of ~10 ms, the coordination -machinery built for disks — a central lock manager, page latches, a buffer -pool — costs more than the work it protects: the classic "OLTP through the -looking glass" breakdown found under 15% of instructions doing useful work, -so Hekaton's design rule was absolute — **no locks, no latches, no pages, -anywhere on any hot path**. +Hekaton's authors did the arithmetic before they wrote any code (§2): a +10–100× throughput goal cannot be reached by tuning, because "improving +scalability and CPI can produce only a 3–4× improvement", so "to go 10× +faster, the engine must execute 90% fewer instructions… to go 100× faster, +it must execute 99% fewer instructions" — and the only way to delete 90% +of the instructions in an OLTP path is to delete the machinery itself, so +the design rule became **no latches or spinlocks on any performance-critical +path, no lock manager, no lock table** (§2.1.2). ## The concepts, step by step ### Step 1 — MVCC recap, minus the disk -MVCC (multi-version concurrency control) means writers never overwrite: -each update creates a **new version** of the record, and every reader is -handed a consistent point-in-time view, so readers and writers never block -each other. Postgres (previous guide) implements this *on disk*: versions -are heap tuples on 8 KB pages, visibility metadata is transaction ids -(xids) plus a commit log to look them up, and cleanup is a background -vacuum process. - -Delete the disk and every one of those choices is up for renegotiation: -versions can be plain heap-allocated structs linked by pointers, "who -committed when" can be a single 64-bit timestamp comparison instead of a -log lookup, and any thread can free garbage the moment it's provably -unreachable. Hekaton is what you get when you renegotiate all of them at -once. +> **In:** the disk-era MVCC design you met in the previous guide — postgres +> heap tuples, xids, clog, vacuum. +> **Out:** a list of which of those choices are *forced by the disk* and +> therefore up for renegotiation once the database lives in DRAM. + +Definitions used from here on: + +- A **transaction** is a group of reads and writes that must appear to + happen all-at-once or not at all. +- **MVCC** (multi-version concurrency control) means writers never + overwrite: each update creates a **new version** of the record, so + readers and writers never block each other. +- A **version** is one immutable snapshot of a record's contents plus the + metadata saying when it was the truth; the set of versions of one record, + linked, is the **version chain**. +- **Visibility** is the predicate "should transaction T see version V?". +- **Garbage collection** (GC) is reclaiming versions no live transaction can + still see. Postgres's variant is called **vacuum**. + +Postgres implements MVCC *on disk*: versions are heap tuples on 8 KB +pages, the visibility metadata is transaction ids (xids) plus a commit log +(clog) to look those xids up, and cleanup is a background vacuum process. +Every one of those three is a consequence of the page: you need a clog +because a tuple header is too precious to hold a commit timestamp, you need +vacuum because you cannot free half a page. + +Delete the disk and all three are renegotiable. Versions can be +heap-allocated structs linked by pointers; "who committed when" can be a +single 64-bit comparison instead of a log probe; and any thread can free +garbage the moment it is provably unreachable. Hekaton is what you get when +you renegotiate all of them at once. + +Why it matters: the rest of this guide is a list of *what postgres does +because of the disk*, and what each of those choices becomes without it. ### Step 2 — the version record: two timestamps bound a lifetime -Hekaton stamps each version with the time interval during which it was the -truth. A global counter hands out monotonically increasing **timestamps**; -a version created at time 100 and superseded at time 250 is visible to -exactly the transactions reading at times 100–249: +> **In:** the need to answer "was this version the truth when I started?" +> without a commit-log lookup. +> **Out:** the version header — `Begin`, `End`, links, payload — and a +> visibility test that is two integer comparisons. + +Hekaton stamps each version with the interval during which it was the +truth. A monotonically increasing counter hands out **timestamps** (§6.1). +Each version carries two: `Begin` is "the commit time of the transaction +that created the version", `End` is "the commit timestamp of the +transaction that deleted the version (and perhaps replaced it with a new +version)" (§6.1). §4 gives the layout: ``` ┌──────────┬─────────┬──────────────┬─────────┐ - │ begin_ts │ end_ts │ index links │ payload │ + │ Begin │ End │ index links │ payload │ └──────────┴─────────┴──────────────┴─────────┘ - live version: end_ts = ∞ - during update: end_ts = writer's txn-id (acts as the write lock!) - visibility: begin_ts ≤ my_read_ts < end_ts + header Name/City/Amount + + live version: End = ∞ + visibility: Begin < RT and End > RT (§6.1) ``` -The visibility check is one range test — two integer comparisons, no -commit-log probe, no in-progress list scan. Compare postgres, where the -same question costs a clog lookup (cached in hint bits) plus a binary -search over the snapshot's in-progress array. That's the payoff of -timestamps: **the version is self-describing with two u64s**. +where RT is the transaction's **logical read time**, which for every +isolation level Hekaton supports is set to the transaction's start time +(§6.1). + +Work it on the paper's own example. Hekaton Figure 2 is a bank-account +table; here are its John rows, with the amounts and timestamps as printed: + +| Begin | End | Name | City | Amount | +|-------|-----|------|--------|--------| +| 10 | 20 | John | London | 100 | +| 20 | 100 | John | London | 110 | +| 100 | ∞ | John | London | 130 | + +Three reads, three answers, using `Begin < RT and End > RT`: + +- RT = 15 → row 1: `10 < 15` and `20 > 15` → **visible**, Amount 100. + Row 2: `20 < 15` is false → invisible. Row 3: `100 < 15` false → + invisible. +- RT = 50 → row 1: `20 > 50` false → invisible. Row 2: `20 < 50` and + `100 > 50` → **visible**, Amount 110. +- RT = 105 → rows 1 and 2 fail the `End > RT` half; row 3: `100 < 105` + and `∞ > 105` → **visible**, Amount 130. + +Exactly one version is visible at each read time, because "different +versions of a record always have non-overlapping valid times so at most one +version of a record is visible to a read" (§4.1). + +Why it matters: the whole check is two integer comparisons on data already +in the cache line you fetched. Compare postgres, where the same question +costs a clog lookup (usually short-circuited by hint bits) plus a search of +the snapshot's in-progress xid array — see +[`heapam_visibility.c:939`](reading-postgres-heapam.md). That is the payoff +of timestamps: **the version is self-describing with two 64-bit fields**. ### Step 3 — txn-ids double as locks: one CAS does two jobs +> **In:** a system with no lock manager and no lock table (§2.1.2), which +> nonetheless has to stop two writers updating one record. +> **Out:** the trick that fits a write lock inside the `End` field, and the +> reader-side cost it creates. + There is no lock manager, so where does write-write conflict detection -live? Inside `end_ts` itself. Timestamps and transaction ids share the -field, distinguished by one bit (bit-smuggling again). To update a record, -a writer CASes (compare-and-swap — an atomic "replace this value only if -it still equals what I read") its txn-id into the live version's `end_ts`: - -- CAS succeeds → this transaction now "owns" the update; the txn-id sitting - in `end_ts` *is* the write lock, and the writer links its new version. -- CAS fails, or a txn-id is already there → a second writer has the record; - abort or wait. First-writer-wins, detected with zero shared tables. - -One atomic instruction replaces the entire lock-manager conversation: -acquire lock + install version pointer, fused. The cost: readers who -encounter a txn-id in a timestamp field must go ask the transaction table -what state that writer is in — Step 4's fine print. - -### Step 4 — commit processing, not a commit point - -Commit is a *pipeline*, not an instant: (1) acquire a **commit_ts** from -the global counter; (2) **validate** — for serializable, re-read your read -set and re-run your scan predicates to confirm nothing changed since your -snapshot (this is OCC, optimistic concurrency control: check at the end -instead of locking up front); (3) write the log record; (4) **fix up** — -walk your versions replacing your txn-id with the real commit_ts in every -`begin_ts`/`end_ts` you touched. - -Between (1) and (4), other transactions can meet your txn-id in a version. -Instead of blocking, they take a **commit dependency**: "I'll treat this -version as committed at your commit_ts — but if you abort, I abort too." -Visibility becomes speculative, with the speculation resolved by the -writer's fate: +live? Inside the timestamp fields themselves. Hekaton §4.2, describing +Figure 2's in-flight transfer: + +> Note that transaction 75 has stored its transaction Id in the Begin and +> End fields of the new and old versions, respectively. (One bit in the +> field indicates the field's content type.) A transaction Id stored in the +> End field prevents other transactions from updating the same version and +> it also identifies which transaction is updating the version. A +> transaction Id stored in the Begin field informs readers that the version +> may not yet be committed and identifies which transaction created the +> version. + +So one bit distinguishes "this is a timestamp" from "this is a transaction +id" (bit-smuggling again), and installing your txn-id in a live version's +`End` field with an atomic compare-and-swap — **CAS**, "replace this value +only if it still equals what I read" — is simultaneously the lock +acquisition and the conflict test: + +- CAS succeeds → this transaction owns the update; the txn-id sitting in + `End` *is* the write lock, and the writer links its new version in. +- A txn-id is already there → another writer holds the record; this one + aborts. First-writer-wins, detected with zero shared tables. + +Follow Figure 2's transfer of $20 from Larry to John all the way through. +Before commit, four versions are in play (§4.2, Figure 2 as printed): + +``` + old John: Begin=20 End=Tx75 Amount=110 + new John: Begin=Tx75 End=inf Amount=130 + old Larry: Begin=30 End=Tx75 Amount=170 + new Larry: Begin=Tx75 End=inf Amount=150 +``` + +Check the money: John 110 + 20 = 130 ✓, Larry 170 − 20 = 150 ✓, total +before 280, total after 280 ✓ — the transfer conserves the sum, which is +the invariant the transaction exists to protect. Then: "suppose transaction +75 commits with end timestamp 100… transaction 75 returns to the old and +new versions and sets the Begin and End fields, respectively, to 100" +(§4.2). Every `Tx75` above becomes `100`, and old John's valid time becomes +20–100 while new John's becomes 100–∞ — which is exactly the table you +worked in Step 2. + +Why it matters: one atomic instruction replaces the entire lock-manager +conversation — acquire lock and publish the version pointer, fused. The +bill arrives on the read side: a reader that meets a txn-id where it +expected a timestamp must go ask the transaction map what that writer is +doing. That is Step 4. + +### Step 4 — commit is a pipeline, and readers speculate through it + +> **In:** a reader that has just found a txn-id, not a timestamp, in a +> version header. +> **Out:** commit dependencies — the mechanism that lets that reader +> proceed without blocking, and the cascading-abort risk it accepts. + +Commit is a *pipeline*, not an instant (§6.2). In order: + +1. **Get an end timestamp.** "The validation phase begins with the + transaction obtaining an end timestamp. This end timestamp determines + the position of the transaction within the transaction serialization + history." (§6.2.1) +2. **Validate.** A serializable transaction must show two things (§6): + **read stability** — "if T reads some version V1 during its processing, + we must ensure that V1 is still the version visible to T as of the end + of the transaction" — and **phantom avoidance** — "the transaction's + scans would not return additional new versions", checked by repeating + the scans. To make this possible each transaction keeps a *read set* + (pointers to versions read) and a *scan set* (§6.2.1). Note the price + list: "repeatable read requires only read validation and snapshot + isolation and read committed require no validation at all" (§6.2.1). +3. **Log.** "A transaction T is committed as soon as its changes to the + database have been hardened to the transaction log." (§6.2.2) +4. **Post-process.** Walk the *write set* replacing your txn-id with the + real end timestamp in every `Begin`/`End` you touched (§6.2.2) — the + `Tx75 → 100` rewrite from Step 3. + +Between steps 1 and 4 your txn-id is visible to everyone. Hekaton's answer +is not to block: + +> Any transaction T1 that begins while a transaction T2 is in the validation +> phase becomes dependent on T2 if it attempts to read a version created by +> T2 or ignores a version deleted by T2. In that case T1 has two choices: +> block until T2 either commits or aborts, or proceed and take a commit +> dependency on T2. To preserve the non-blocking nature of Hekaton, we have +> T1 take a commit dependency on T2. This means that T1 is allowed to commit +> only if T2 commits. If T2 aborts, T1 must also abort so cascading aborts +> are possible. (§6.2.1) + +Two consequences the paper spells out (§6.2.1): T1 increments a dependency +count and cannot commit until it reaches zero; and because T1 is now +holding uncommitted data, a **read barrier** holds T1's result set back +from the client until the dependencies clear. The non-blocking property is +paid for in latency-at-the-edge, not latency-in-the-engine. + +Sketching the reader side makes the two-field decode concrete: ```rust +// ILLUSTRATION — not quoted from any repo; it is Hekaton §6.1's visibility +// rule plus §6.2.1's commit dependency, written in Rust so the two-way +// decode of the Begin/End fields is explicit. Real implementations of the +// same predicate: postgres heapam_visibility.c:939, and this topic's own +// exercise at experiments/src/mvcc.rs:89. fn visible(v: &Version, read_ts: u64, txns: &TxnTable) -> bool { - let begin = match v.begin_ts { + let begin = match v.begin_field { Stamp(ts) => ts, TxnId(id) => match txns.state(id) { - Committing { commit_ts } => commit_ts, // take a commit DEPENDENCY: - _ => return false, // I abort if the writer does + Committing { end_ts } => end_ts, // take a commit DEPENDENCY: + _ => return false, // I abort if the writer does }, }; - let end = match v.end_ts { + let end = match v.end_field { Stamp(ts) => ts, // superseded at ts TxnId(_) => u64::MAX, // being updated — still the latest for readers }; - begin <= read_ts && read_ts < end + begin < read_ts && read_ts < end } ``` -Why it matters: no reader ever waits on a writer's commit — the cost moved -from blocking (latency) to cascading aborts (wasted work), which is the -right trade when conflicts are rare. +Why it matters: no reader ever waits on a writer's commit. The cost moved +from blocking (latency) to cascading aborts (wasted work) — the right trade +when conflicts are rare, and the wrong one when they are not, which is the +finding Step 8 will price. + +### Step 5 — indexes point at versions, not at chains + +> **In:** a database with no pages, where the only way to reach a record is +> an index probe. +> **Out:** the cost model of Hekaton's choice — every new version is +> inserted into every index — and the correction it forces to a common +> intuition. + +With no pages, "the table" is just the set of version records, and +"records are always accessed via an index lookup" (§4). Hekaton has two +index types: "hash indexes which are implemented using lock-free hash +tables and range indexes which are implemented using Bw-trees, a novel +lock-free version of B-trees" (§4) — the Bw-tree is topic 9's cautionary +protagonist. + +The layout detail that matters: "Each index requires a link field in the +record… Versions that hash to the same bucket are linked together using the +first link field" (§4). So a hash bucket is a chain of *version records*, +not of *logical rows* — Figure 2's "Hash bucket J contains four records: +three versions for John and one version for Jane" (§4). A lookup scans the +bucket and applies Step 2's test; §4.1's worked case: "A lookup for John +with read time 15… would trigger a scan of bucket J that checks every +record in the bucket but returns only the one with Name equal to John and +valid time 10 to 20." + +And crucially, on update: transaction 75 "has created the new versions for +Larry and for John and inserted them into the appropriate buckets in the +index" (§4.2). **New versions go into the index.** Wu §6.2 classifies this +as the *physical pointer* scheme and names Hekaton as a user of it; Wu +Table 1's Hekaton row reads *Physical* under Index Management. Step 11 +prices that choice. + +> **Correction to a claim this guide used to make.** An earlier version of +> this guide said Hekaton's "index entries point at *chains*, not individual +> versions, so a new version doesn't churn the index." That is the *logical +> pointer* scheme (Wu §6.1), which Hekaton does not use — §4.2 shows new +> versions being inserted into the index buckets, and Wu Table 1 files +> Hekaton under Physical. The churn is real, and Step 11 measures what it +> costs. + +Why it matters: this is the axis where "no pages" does *not* buy a free +lunch — deleting the buffer pool made lookups cheap and made updates +touch more index structures, not fewer. + +### Step 6 — cooperative GC: the workload cleans itself, mostly + +> **In:** an MVCC engine with no vacuum daemon and a bounded memory budget. +> **Out:** the two-part GC design, and the specific hole in it that a +> background process still has to fill. + +The correctness rule first (§8.1.1): "the visibility of a version is +determined by its begin and end timestamps. Any version whose end timestamp +is less than the current oldest active transaction in the system is not +visible to any transaction and can be safely discarded." A GC thread +"periodically scans the global transaction map to determine the begin +timestamp of the oldest active transaction" — the **watermark**. + +Removal is in two parts (§8.1.2): + +1. **Cooperative.** "Since regular index scanners may encounter garbage + versions as they scan indexes, index operations are empowered to unlink + garbage versions when they encounter them. If this unlinks a version + from its last index, the scanner may also reclaim it." The paper gives + two reasons: it "naturally parallelizes garbage collection", and it + "ensures that old versions will not slow down future scanners by forcing + them to skip over old versions encountered, for example, in hash index + bucket chains." +2. **Background.** Cooperative cleaning is explicitly "insufficient to + ensure that either (1) 'cold' areas of an index which are not traversed + by scanners are free of garbage, or that (2) a garbage version is + removed from other indexes that it might participate in. Versions in + these 'dusty corners' (infrequently visited index regions) do not need + to be collected for performance reasons, but they needlessly consume + memory." + +So the answer to "what about garbage nobody walks past?" is in the paper: +it is not a performance problem (nobody is walking past it, so it slows +nobody down) but it is a memory problem, and a background sweep exists +precisely for it. + +Contrast postgres on every axis now visible: timestamps vs xid + clog + +hint bits (Step 2); CAS-as-lock vs a lock manager (Step 3); commit-time +validation vs SIREAD locks (Step 4, and see +[`reading-ssi-postgres.md`](reading-ssi-postgres.md)); indexes carrying +every version vs `t_ctid` chains (Step 5); cooperative cleaning vs vacuum +(Step 6). Note the one axis where they *agree*: Wu Table 1 files both +postgres and Hekaton under append-only storage with **O2N** (oldest-to-newest) +version ordering. Step 12 shows that shared choice putting them in the same +place at the bottom of a benchmark. + +Why it matters: this is the last of Hekaton's design decisions, and you +now have a complete point in the design space. Wu/Pavlo's contribution is +the space itself. + +### Step 7 — what a Wu/Pavlo number is measured on + +> **In:** a paper full of percentages you are about to quote. +> **Out:** the machine, system and protocol behind them, so you know what +> each percentage is a percentage *of*. + +Every Wu/Pavlo figure quoted below comes from this setup (§7): + +| Knob | Setting | +|---|---| +| Machine | 4-socket Intel Xeon E7-4820, ten 1.9 GHz cores per socket (**40 cores**), 25 MB L3 per socket, 128 GB DRAM, Ubuntu 14.04 | +| System | Peloton, one codebase implementing every combination — so protocol differences are not vendor differences | +| Isolation | SERIALIZABLE throughout | +| Workloads | YCSB (Zipfian skew parameter θ) and TPC-C | +| Method | 60 s warm-up, then measure; results averaged over five trials | + +Two things follow. First, "θ" is the contention dial: θ=0.2 is near-uniform +access, θ=0.8–0.9 is a hot handful of keys, and *every verdict below is +conditional on θ*. Second, because one system implements all the variants, +a percentage here compares two designs, not two engineering teams — which +is the reason this paper is worth reading at all. + +One instructive control, before any protocol appears. Fig 6a runs a +*read-only* YCSB workload at θ=0.2: "all but one of the protocols scales +almost linearly up to 24 threads. The main bottleneck for all of these +protocols is the cache coherence traffic from updating the memory manager's +counters and checking for conflicts when transactions commit (even though +there are no writes)" (§7.2). Read-only, no conflicts, and the ceiling at +24 of 40 cores is *still* coherence traffic. Fig 6b's counterpart: raising +transactions from 10 to 100 operations "reduced by ∼30×" the throughput but +made all protocols "scale linearly up to 40 threads", because longer +transactions mean less pressure on the shared structures. + +Why it matters: hold that finding next to this repo's own measured lane in +[`notes.md`](notes.md) — a single global mutex delivers ~600k txn/s +*flat* across read-heavy, write-heavy and hot-key workloads. Both results +say the same thing from opposite ends: on a multi-core machine the shared +coordination structure, not the isolation algorithm, sets the ceiling. + +### Step 8 — design decision 1 of 4: the concurrency control protocol + +> **In:** four protocols implemented in one system (Wu §3). +> **Out:** which one to reach for, and the specific contention level at +> which the popular answer collapses. + +The four (§3): + +| Protocol | Mechanism | Extra per-tuple state | +|---|---|---| +| **MVTO** (timestamp ordering) | order by transaction timestamp; abort a writer whose tuple has already been read by a later transaction | `read-ts` (§3.1) | +| **MVOCC** (optimistic) | run, then validate the read set at commit | — (§3.2) | +| **MV2PL** (two-phase locking) | take read/write locks in the tuple header | `read-cnt`, packed with `txn-id` into one 64-bit word (§3.3) | +| **Certifier** (SI+SSN) | snapshot isolation plus a serial-safety-net check | — (§3.4) | + +One implementation detail worth carrying away: MV2PL uses a **no-wait** +deadlock policy — a transaction that cannot get a lock aborts immediately +rather than waiting, so there is no deadlock detector to run (§3.3). + +The measured verdicts: + +- **Contention is what separates them, and only past a threshold.** Fig 7a + (read-intensive, 40 threads): "When θ is less than 0.7, we see that all + of the protocols achieve similar throughput. Beyond this contention level, + the performance of MVOCC is reduced by ∼50%. This is because MVOCC does + not discover that a transaction will abort due to a conflict until after + the transaction has already executed its operations. **There is nothing + about multi-versioning that helps this situation.**" (§7.2) +- **Nothing helps write-write conflicts.** Fig 7b (update-intensive): + "there is not a great difference among the protocols except MV2PL; they + handle write-write conflicts in a similar way and again multi-versioning + does not help reduce this type of conflicts." (§7.2) +- **On TPC-C, MVTO wins.** Fig 10a, 10 warehouses: MVTO achieves 45–120% + higher throughput than the others (§7.2). +- **And nobody ships it.** §8: "Overall, we found that MVTO works well on a + variety of workloads. **None of the systems that we list in Table 1 adopt + this protocol.**" Table 1 lists nine systems. + +Why it matters: this is the axis the literature argues about, and its +verdict is the mildest of the four — below θ=0.7 the protocol barely +matters. Keep that in view while reading Steps 9–11. + +### Step 9 — design decision 2 of 4: version storage + +> **In:** three ways to lay out a version chain (Wu §4). +> **Out:** the axis Wu/Pavlo call the most important one, and the two +> sub-decisions hiding inside it. + +The three schemes (§4), defined: + +- **Append-only** — every version is a full copy of the tuple, all in the + same table space (postgres, Hekaton, MemSQL, NuoDB, HYRISE per Table 1). +- **Time-travel** — full copies again, but the old versions live in a + separate table (SAP HANA). +- **Delta** — the master version is updated in place and only the *changed + attributes* are copied into a separate delta store, like an undo record + (Oracle, MySQL-InnoDB). + +Append-only carries a sub-decision: which end of the chain the index enters. +**O2N** (oldest-to-newest) means the index points at the oldest version and +readers walk forward; **N2O** (newest-to-oldest) means the index points at +the newest and readers usually stop immediately. + +Measured: + +- **N2O always wins.** Fig 12: N2O "always performs better than O2N in both + workloads", and at the highest contention (θ=0.9) "the N2O ordering + achieves 2.4–3.4× better performance" (§7.3). +- **Delta wins wide tables with narrow updates.** Fig 13b: "when the table + has 100 attributes, the delta scheme achieves ∼2× better performance than + append-only and time-travel schemes because it uses less memory" (§7.3). + Fig 14 and Fig 15 refine it: delta is best when few attributes are + *modified*, and degrades fastest as the number of attributes *read* rises. +- **…and loses scans, badly.** Fig 17b (TPC-C, 40 warehouses): "With delta + storage, the latency of the scan queries grows near-linearly with the + increase of number of threads (which is bad), while the append-only and + time-travel schemes maintain a latency that is 25–47% lower when using 40 + threads" (§7.3). Fig 17a: append-only wins TPC-C throughput, because + TPC-C reads many attributes at once. +- **The allocator is a confounding variable.** Fig 16: the delta scheme is + "stable regardless of the number of memory spaces", while append-only and + time-travel throughput is "improved by 1.6–4× when increasing the number + of separate memory spaces from 1 to 20" (§7.3). Allocator contention + masquerades as storage-scheme cost — if you benchmark append-only against + delta with a single shared allocator, you are benchmarking the allocator. +- **Non-inlined attributes should be reference-counted, not copied.** + Fig 11: with the read-intensive workload the DBMS "achieves ∼40% higher + throughput when the number of non-inlined attributes is increased to 50", + and for update-intensive "the performance gap reaches over 100%" (§7.3). + +Why it matters: §8's headline finding is that "the version storage scheme +is one of the most important components to scaling an in-memory MVCC DBMS +in a multi-core environment. This goes against the conventional wisdom in +database research that has mostly focused on optimizing the concurrency +control protocols." + +### Step 10 — design decision 3 of 4: garbage collection + +> **In:** Step 6's Hekaton design plus postgres's vacuum, now as two points +> on a third axis (Wu §5). +> **Out:** the measured cost of each, and the reason the paper prefers the +> option neither system uses. + +Two granularities, and within tuple-level, two triggers (§5): + +- **Tuple-level VAC** — a background vacuum thread scans for expired + versions. (Postgres, Oracle, MySQL, NuoDB, MemSQL per Table 1.) +- **Tuple-level COOP** — cooperative cleaning by whatever worker walks the + chain; Step 6's Hekaton design. §5.1 notes COOP "only works for the O2N + append-only storage" — which is one reason Hekaton's Table 1 row is O2N. +- **Transaction-level** — reclaim in batches, per transaction/epoch, rather + than per tuple. + +Measured, with append-only O2N, MVTO, 40 worker threads and one GC thread +(§7.4): + +- **COOP beats VAC.** Fig 18: "COOP achieves 45% higher throughput compared + to VAC under read-intensive workloads." Fig 19: "COOP has a 30–60% lower + memory footprint per transaction than VAC", and its performance is more + stable because it "amortizes the GC overhead across multiple threads." +- **Transaction-level beats both.** Fig 20a shows a slight edge on + read-intensive, and "the gap increases to 20% in Fig. 20b for the + update-intensive workload. Transaction-level GC removes expired versions + in batches, thereby reducing the synchronization overhead." §8's summary: + "a transaction-level GC provided the best performance with the smallest + memory footprint." +- **GC off is not a speed-up, it is a slow decay.** Fig 18 again: + "performance declines over time when GC is disabled because the DBMS + traverses longer version chains to retrieve the versions. Furthermore, + because the system never reclaims memory, it allocates new memory for + every new version." Both mechanisms "improve throughput by 20–30% + compared to when GC is disabled" (§7.4). + +Why it matters: GC is the axis most likely to be omitted from a benchmark +(it costs nothing in the first 20 seconds) and the one whose absence shows +up as a *downward slope* rather than a lower number — which is why Fig 18's +x-axis is elapsed time, not thread count. Your own benchmarks should copy +that choice. + +### Step 11 — design decision 4 of 4: index management + +> **In:** Step 5's discovery that Hekaton inserts every new version into +> every index. +> **Out:** the measured price of that, and the alternative. + +Two schemes (§6): + +- **Logical pointers** — the index maps key → an indirection slot (or + primary key) that holds the head of the version chain. A new version + updates the slot; the secondary indexes never move. +- **Physical pointers** — the index entry points straight at a version + record, so "when updating any tuple in a table, the DBMS inserts the newly + created version into all the secondary indexes" (§6.2). Hekaton and + MemSQL do this. + +Measured on update-intensive YCSB with MVTO, append-only N2O and +transaction-level GC, varying the number of secondary indexes (§7.5): + +- Fig 22b, **high contention (θ=0.8)**: "logical pointer achieves 25% + higher performance compared to physical pointer scheme." +- Fig 22a, **low contention (θ=0.2)**: "the performance gap is enlarged to + 40% with the number of secondary indexes increased to 20." +- Fig 23, **eight secondary indexes, varying threads**: "for the high + contention workload, the DBMS's throughput when using logical pointers is + 45% higher than the throughput of physical pointers." +- §8's summary: "logical pointer scheme always achieve a higher throughput + especially when processing update-intensive workloads." + +Why it matters: the cost is proportional to *number of secondary indexes*, +a quantity that grows silently over a schema's life. A design that is +free at one index is paying 40% at twenty. + +### Step 12 — the shootout, and where Hekaton actually lands + +> **In:** the four axes, priced. +> **Out:** the paper's own head-to-head of nine real systems' configurations, +> and the result that should revise your opinion of Step 1–6's design. + +Wu/Pavlo's last experiment (§8, Figs 24–25) configures Peloton as each of +Table 1's nine real systems and runs TPC-C — "a good approximation of their +abilities", with the honest caveat that it does not capture "other factors +in the real DBMSs… (e.g., data structures, storage architecture, query +compilation)." + +Table 1's rows for the four systems this topic cares about: + +| System | Protocol | Version storage | GC | Index pointers | +|---|---|---|---|---| +| Oracle / MySQL-InnoDB | MV2PL | Delta | Tuple-level VAC | Logical | +| Postgres | MV2PL / SSI | Append-only, **O2N** | Tuple-level VAC | Physical | +| Hekaton | MVOCC | Append-only, **O2N** | Tuple-level COOP | Physical | +| NuoDB | MV2PL | Append-only, **N2O** | Tuple-level VAC | Logical | + +The result (§8, Fig 24): + +> As shown in Fig. 24, the DBMS performs the best on both the low-contention +> and high-contention workloads with the Oracle/MySQL and NuoDB +> configurations… **Postgres and Hekaton's configurations lead to the worst +> performance**, and the major reason is that the use of append-only storage +> with O2N ordering severely restricts the scalability of the system. + +Hekaton and postgres finish together, at the bottom, for the same reason — +the version-ordering choice from Step 9, not anything in Steps 3, 4 or 6. +That is the sharpest thing this pair of papers teaches: a beautifully +argued protocol design (no locks, no latches, CAS-as-lock, commit +dependencies) is outranked on this benchmark by one layout decision it +inherited. + +And no corner dominates. Fig 25 (scan latency): "the DBMS's performance is +the worst with delta storage. This is because the delta storage has to +spend more time on traversing version chains so as to find the targeted +tuple version attribute." The throughput winners are the latency losers. + +Why it matters: read this as the general lesson for design-space papers — +the axis everyone optimizes (protocol, Step 8) had the mildest verdict, and +the axis nobody writes papers about (storage layout, Step 9) decided the +shootout. + +## Where each claim lives in the papers + +| Step | Source | What | +|---|---|---| +| 1 | — | background from the previous guide | +| 2 | Hekaton §4, §4.1, §6.1, Figure 2 | version header, valid time, visibility rule | +| 3 | Hekaton §4.2, Figure 2 | txn-id in Begin/End, the one type bit, the $20 transfer | +| 4 | Hekaton §6, §6.2.1, §6.2.2, §6.2.3 | read stability, phantom avoidance, commit dependencies, read barriers, post-processing | +| 5 | Hekaton §4, §4.1, §4.2; Wu §6.2, Table 1 | index types, bucket chains of versions, physical pointers | +| 6 | Hekaton §8.1.1, §8.1.2; Wu §5.1, Table 1 | watermark, cooperative unlinking, dusty corners; COOP requires O2N | +| 7 | Wu §7, Fig 6a, Fig 6b | hardware, method, the read-only coherence ceiling | +| 8 | Wu §3.1–§3.4, Fig 7a, Fig 7b, Fig 10a, §8 | the four protocols, θ=0.7 cliff, MVTO's 45–120%, nobody ships MVTO | +| 9 | Wu §4, Fig 11, Fig 12, Fig 13b, Fig 14, Fig 15, Fig 16, Fig 17a, Fig 17b, §8 | storage schemes, N2O 2.4–3.4×, delta ∼2×, scan latency 25–47%, allocator 1.6–4× | +| 10 | Wu §5, §5.1, Fig 18, Fig 19, Fig 20a, Fig 20b, §8 | COOP +45%, 30–60% memory, txn-level +20%, GC-off decay | +| 11 | Wu §6.1, §6.2, Fig 22a, Fig 22b, Fig 23, §8 | logical +25% / +40% / +45% | +| 12 | Wu Table 1, §8, Fig 24, Fig 25 | the shootout; postgres and Hekaton last | -### Step 5 — indexes point at version chains, not rows +## How to read the papers (with the concepts in hand) -With no pages, "the table" is just the set of version chains, and the only -way to find one is through an index. Hekaton's indexes (a lock-free hash -table and the Bw-tree — topic 9's cautionary protagonist) map key → chain -of versions; a lookup walks the chain running Step 4's `visible()` until -it finds its version. Index entries point at *chains*, not individual -versions, so a new version doesn't churn the index. MVCC and lock-free -data structures were co-designed here — the version chain's immutable -"append a new head" discipline is exactly what a CAS-based index can -publish atomically. +**Hekaton first (~1.5 h)** — it is a systems paper; §4 and §6 carry it: + +1. **§4 Storage and Indexing** — Steps 2 and 5 in the authors' words. Check + Figure 2's six version records against Step 2's table and Step 3's + transfer; the numbers are all there. +2. **§6 Transaction Management** — Steps 3 and 4. Read §6.2.1 slowly; + commit dependencies are the subtle part and `visible()` above is your + crib. §6's two properties (read stability, phantom avoidance) are the + definition of "serializable" the rest of the paper uses. +3. **§8 Garbage Collection** — Step 6; note the trigger (a scan passing by) + versus postgres's (a scheduled vacuum), and read §8.1.2's "dusty + corners" paragraph twice. +4. **§9 Experimental Results** — the numbers to keep: §9.1.1 reports a 20× + lookup speed-up for 10+ lookups per call and 10.8× for a single lookup; + §9.1.2 reports "around 30×" for transactions updating 100 or more + records. Both are single-core CPU-efficiency measurements on a 2.67 GHz + Xeon W3520 with 1M-row tables (§9.1) — *not* end-to-end system + throughput. Skim §7 (durability); checkpointing versions to disk is + topic-5 material in new clothes. + +**Then Wu/Pavlo (~1 h)** — read it as a menu with prices. §3–§6 map +one-to-one onto Steps 8–11; §7's graphs are the message. Landmarks: Fig 1 +(the version header — compare Step 2), Table 1 (nine systems on the four +axes), Fig 12 (N2O vs O2N), Fig 13–15 (storage vs update rate and attribute +count), Fig 16 (the allocator confound), Fig 18–21 (GC, plotted against +elapsed time), Fig 22–23 (index pointers vs secondary-index count), Fig +24–25 (the shootout). For each axis, find the crossover workload where the +verdict flips — that is what you are buying. -### Step 6 — cooperative GC: the workload cleans itself +## Questions for notes.md -Old versions pile up (that's MVCC's rent), and there's no vacuum process. -Instead: any thread that *walks past* a version whose `end_ts` is older -than the oldest active transaction's read timestamp unlinks it on the -spot. Cleanup happens in proportion to how much the workload reads, right -where the garbage obstructs traffic, with no separate process to schedule -or throttle. The failure mode to remember: garbage nobody walks past -doesn't get cleaned (Wu/Pavlo call this out — question 4). +1. Hekaton's `End`-as-lock: write the CAS-based first-writer-wins in + pseudocode. Where does your `mvcc.rs` do the same check? (Point at the + line once `commit()` at `experiments/src/mvcc.rs:105` is implemented and + `first_committer_wins_on_write_write_conflict` passes.) +2. Delta storage wins narrow updates of wide tables (Fig 13b); append-only + N2O wins reads and scans (Fig 12, Fig 17b). Which is a GraphBLAS **delta + matrix** (topic 20)? So M8's "copy-on-write + deltas" sits where in Wu's + taxonomy — and what do Fig 14/15 predict about its read path? +3. Logical vs physical index pointers: FalkorDB's node ids *are* logical + indirection into matrices. What does that make "index management" cost + for a graph MVCC — which updates still have to touch indexes, and does + Fig 22a's 40%-at-20-indexes result apply at all? +4. Cooperative GC cleans in proportion to reads: what happens to a + write-only hot key that nobody reads? Hekaton §8.1.2 answers this + directly — find the answer and say why it is a memory problem and not a + throughput problem. +5. Predict, then check Wu §7.2: at 40 cores and high contention, what ruins + MVOCC — validation aborts or timestamp allocation? (Fig 7a's sentence + settles it in one line.) +6. Step 12: Hekaton and postgres finish last in Fig 24 for the same reason. + Name it, and name the two Hekaton design choices that the shootout says + were *not* the problem. + +## Takeaway + +Hekaton shows what MVCC becomes when the disk is deleted: two timestamps +per version, a CAS into a timestamp field standing in for the entire lock +manager, commit as a validate-log-fixup pipeline with speculative readers, +and cleanup done by whoever walks past. Wu & Pavlo show that this beautiful +protocol design is not where the throughput is: across four design +decisions measured in one system, the concurrency control protocol matters +least below θ=0.7, and the version-storage layout matters most — enough to +put Hekaton's configuration at the bottom of their TPC-C shootout next to +postgres, for the one choice the two systems share. + +## Connections to this topic's experiment + +This topic's provided benchmark lane measures a deliberately naive +baseline: a single global `Mutex`, 4 threads × 50 000 transactions +× 4 operations each. On an Apple M3 Pro (measured 2026-07-28, recorded in +[`notes.md`](notes.md)): + +| Workload | Keys | mutex txn/s | +|---|---|---| +| read-heavy 95/5 | 10 000 | 623 454 | +| write-heavy 50/50 | 10 000 | 594 264 | +| write-heavy 50/50 | 64 (hot) | 676 691 | + +**Read that table as a negative result, and be careful what you conclude +from it.** The three numbers are flat — within about 12% of each other — +across workloads that differ enormously in read/write mix and key skew. The +reason is not that the mutex is good; it is that a global mutex has +*already serialized everything*, so the workload's shape cannot influence +the result. That is the finding recorded in +[`FINDINGS.md`](../../FINDINGS.md) row 8. + +What this repo has **not** measured is MVCC beating that mutex. The `mvcc +txn/s` and `aborts` columns in `notes.md` are `stub` — they are the +exercise. Nothing in this guide, and nothing in the topic, is evidence that +this repo's MVCC implementation is faster than the mutex; the Wu/Pavlo +numbers quoted above were measured on a 40-core Xeon running Peloton, not +here. + +Two connections worth holding onto while you implement: + +- Step 7's Fig 6a control (a read-only workload still ceilinged at 24 of 40 + cores by coherence traffic on the *memory manager's* counters) and this + topic's flat mutex line are the same lesson: the shared structure sets the + ceiling. When your `mvcc.rs` lane finally produces a number, ask which + shared structure is setting *its* ceiling before you credit the protocol. +- Step 10's Fig 18 is plotted against elapsed time because GC's absence + shows up as a slope, not a level. `gc_drops_dead_versions_but_respects_active_snapshots` + in `experiments/src/mvcc.rs` is the correctness half of that; a run long + enough to show the slope is the performance half. -Contrast postgres on every axis now visible: timestamps vs -xid+clog+hint-bits (Step 2); CAS-as-lock vs lock manager (Step 3); -commit-time validation vs SIREAD locks (Step 4); new-to-old pointer chains -vs t_ctid old-to-new; cooperative GC vs vacuum (Step 6). +## Done when -### Step 7 — Wu/Pavlo: the same decisions as a menu with prices +Answer each before unfolding it. -Hekaton is one point in a space. Wu & Pavlo implemented *every* combination -of the design axes in one system (Peloton) and benchmarked them — read -their tables as a price list: +- [ ] Name the four design decisions Wu & Pavlo isolate, and say which one + §8 calls the most important for scaling an in-memory MVCC DBMS. -| Axis | Options | Verdict (their workloads) | -|---|---|---| -| concurrency control | MVTO / MVOCC / MV2PL / SI+SSN | MVTO wins TPC-C by 45–120%; MVOCC loses ~50% past contention θ=0.7 (conflicts found only at validation); no protocol helps write-write conflicts | -| version storage | append-only / delta / time-travel | **delta wins for small updates of wide tables** (~2× at 100 attributes) but scan latency grows near-linearly with threads; append-only pays full-tuple copies | -| ordering | newest-to-oldest / oldest-to-newest | N2O wins always — 2.4–3.4× at θ=0.9; O2N walks garbage first, and readers want the newest | -| GC | tuple-level background / cooperative / txn-level / epoch | cooperative: +45% throughput, 30–60% less memory than background vacuum; txn-level epoch: +20% update-intensive, smallest footprint; GC off = throughput *decays over time* | -| index mgmt | logical pointers / physical | logical (indirection) — +25% at high contention, +40% with 20 secondary indexes; physical means every version churns every index | - -Terms: *append-only* stores each version as a full copy (postgres); -*delta* stores only the changed columns (like an undo record); N2O/O2N is -which end of the version chain the pointer enters. The meta-lesson (their -words, roughly): everyone argues about CC algorithms, but **version -storage and GC decide throughput**. Storage layer > protocol. (The RUM -triangle strikes again.) - -Three findings hide outside the axis table: - -- **The allocator is a confounding variable.** Partitioning version - storage into per-thread memory spaces lifted append-only/time-travel - throughput **1.6–4×** (Fig 16) — allocator contention masquerades as - protocol cost. Even on read-only YCSB, scaling flattens past 24 of 40 - cores from cache-coherence traffic on the memory manager's counters, - not from any protocol. -- **Non-inline attributes** (BLOBs, long strings): reference-count them - instead of copying per version — +40% read-intensive, >100% - update-intensive at 50 attributes (Fig 11). -- **The shootout** (§8, Figs 24–25): Peloton configured as each real - system's Table-1 row, on TPC-C. Oracle/MySQL (MV2PL + delta + vacuum + - logical ptrs) and NuoDB (MV2PL + append-N2O) come first; **postgres - comes last** — append-only O2N is the strangler. But the delta winners - post the *worst* scan latency — no corner of the space dominates. And - MVTO, the best all-round protocol, ships in none of the nine systems - surveyed. +
Answer -## How to read the papers (with the concepts in hand) +Concurrency control protocol (§3), version storage (§4), garbage collection +(§5), index management (§6). §8: "the version storage scheme is one of the +most important components to scaling an in-memory MVCC DBMS in a multi-core +environment. This goes against the conventional wisdom in database research +that has mostly focused on optimizing the concurrency control protocols." -**Hekaton first (~1.5 h)** — it's a systems paper; the version format and -commit processing sections carry it: - -1. Storage & indexing section — Steps 2 and 5 in the authors' words; check - the version-record figure against Step 2's diagram. -2. Transaction management — Steps 3–4. Read the commit-processing walk - slowly; commit dependencies are the subtle part, and `visible()` above - is your crib. -3. Garbage collection — Step 6; note what triggers cleaning (a scan - passing by) vs postgres (a scheduled vacuum). -4. Skim the durability/recovery section — checkpointing versions to disk - is topic-5 material wearing new clothes. - -**Then Wu/Pavlo (~1 h)** — read it as a menu with prices: taxonomy -sections §3–§6 map one-to-one onto Step 7's table rows; the §7 graphs are -the message. Landmarks: Fig 1 (the version header — compare Step 2's -diagram), Table 1 (nine real systems placed on the axes), Fig 12 (N2O vs -O2N), Fig 13–15 (storage schemes vs update rate / attribute counts), -Fig 16 (the allocator finding), Fig 18–21 (GC on/off decay over time), -Fig 22–23 (index pointers vs secondary-index count), Fig 24–25 (the -shootout). For each axis, find the crossover workload where the verdict -flips — that's what you're buying. +
-## Questions for notes.md +- [ ] Given the Hekaton Figure 2 John rows — `(Begin 10, End 20, 100)`, + `(20, 100, 110)`, `(100, ∞, 130)` — say which version a read at + RT = 20 sees, and why the rule cannot return two. -1. Hekaton's end_ts-as-lock: write the CAS-based first-writer-wins in - pseudocode. Your mvcc.rs does the same check where? (Point at the line - once implemented.) -2. Delta storage wins for writes; append-only N2O for reads. Which is a - GraphBLAS **delta matrix** (topic 20)? So M8's "copy-on-write + deltas" - sits where in the Wu/Pavlo taxonomy — and what does their data predict - about its read path? -3. Logical vs physical index pointers: FalkorDB's node ids ARE logical - indirection into matrices. What does that make "index management" cost - for a graph MVCC — which updates still have to touch indexes? -4. Cooperative GC in proportion to reads: what happens to a write-only - hot key that nobody reads? (Wu/Pavlo call this out — find the fix.) -5. Predict, then check §7 of Wu/Pavlo: at 40 cores, high contention, what - ruins MVOCC — validation aborts or timestamp allocation? +
Answer -## Done when +The rule is `Begin < RT and End > RT` (§6.1), strict on both sides. At +RT = 20: row 1 fails `End > RT` (20 > 20 is false); row 2 fails `Begin < RT` +(20 < 20 is false); row 3 fails `Begin < RT` (100 < 20 is false). **No +version is visible at RT = 20** — the boundary is exactly the instant the +updating transaction's commit timestamp falls on, and a logical read time is +"any value between the transaction's begin time and the current time" +(§6.1), assigned to a transaction's start time, which is never equal to +another transaction's already-assigned end timestamp. Two versions can never +both qualify because "different versions of a record always have +non-overlapping valid times" (§4.1) — the End of one is the Begin of the +next, and the two comparisons are strict in opposite directions. + +
+ +- [ ] Explain why Hekaton's readers never block on an in-flight writer, and + name the cost that replaces blocking. + +
Answer + +A reader that finds a txn-id (not a timestamp) in a version's Begin or End +field takes a **commit dependency** rather than waiting: "To preserve the +non-blocking nature of Hekaton, we have T1 take a commit dependency on T2. +This means that T1 is allowed to commit only if T2 commits. If T2 aborts, T1 +must also abort so **cascading aborts are possible**" (§6.2.1). The costs +are cascading aborts (wasted work instead of wasted wall-clock) and the +**read barrier** — T1's results are withheld from the client until its +dependency count reaches zero, so the latency reappears at the API edge +rather than inside the engine. + +
+ +- [ ] Hekaton uses physical index pointers. Quote the Wu/Pavlo figure and + number that prices this, and say which workload property makes the + price grow. -You can fill the 5-axis table from memory and place postgres, Hekaton, -and your M8 design in it — one row each. +
Answer + +Physical pointers mean "when updating any tuple in a table, the DBMS inserts +the newly created version into all the secondary indexes" (Wu §6.2), and +Table 1 files Hekaton under Physical. Fig 22b: at high contention (θ=0.8) +logical pointers achieve **25% higher** throughput; Fig 22a: at low +contention the gap "is enlarged to **40%** with the number of secondary +indexes increased to 20"; Fig 23: with eight secondary indexes under high +contention, logical is **45% higher**. The property that grows the price is +the **number of secondary indexes** — each new version must be inserted into +every one of them. + +
+ +- [ ] In Fig 24's shootout, which two configurations finish last, and what + single design choice does §8 blame? + +
Answer + +"Postgres and Hekaton's configurations lead to the worst performance, and +the major reason is that the use of **append-only storage with O2N +ordering** severely restricts the scalability of the system" (§8). The +supporting measurement is Fig 12: N2O "always performs better than O2N", +and at θ=0.9 by **2.4–3.4×**. Note what is *not* blamed: Hekaton's +CAS-as-lock, its commit dependencies, or its cooperative GC — the shootout +indicts a layout decision, not the protocol. + +
+ +- [ ] State the measured result from this repo's own lane, and state what it + does **not** show. + +
Answer + +The global-mutex baseline delivers 623 454 / 594 264 / 676 691 txn/s on +read-heavy 10K-key, write-heavy 10K-key and write-heavy 64-hot-key +workloads respectively (Apple M3 Pro, 2026-07-28; `notes.md`, +[`FINDINGS.md`](../../FINDINGS.md) row 8). It is **flat** because the mutex +already serialized everything, so the workload's shape cannot reach the +result. It does **not** show MVCC beating a mutex — the `mvcc txn/s` and +`aborts` columns are `stub`, and every comparative number in this guide was +measured by Wu & Pavlo on a 40-core Xeon running Peloton, not in this repo. + +
## References **Papers** -- Diaconu, Freedman, Ismert, Larson, Mittal, Stonecipher, Verma, Zwilling - — "Hekaton: SQL Server's Memory-Optimized OLTP Engine" (SIGMOD 2013) — - ~1.5 h; the version format and commit processing sections carry it -- Wu, Arulraj, Lin, Xian, Pavlo — "An Empirical Evaluation of In-Memory - Multi-Version Concurrency Control" (VLDB 2017) — - [PDF](https://db.cs.cmu.edu/papers/2017/p781-wu.pdf) — ~1 h; read it as - a menu with prices, Table 1 and the §7 graphs carry the message + +- Diaconu, Freedman, Ismert, Larson, Mittal, Stonecipher, Verma, Zwilling — + *Hekaton: SQL Server's Memory-Optimized OLTP Engine* (SIGMOD 2013) — + ~1.5 h; §4 (storage and indexing) and §6 (transaction management) carry + it. +- Wu, Arulraj, Lin, Xian, Pavlo — *An Empirical Evaluation of In-Memory + Multi-Version Concurrency Control* (VLDB 2017) — + [PDF](https://db.cs.cmu.edu/papers/2017/p781-wu.pdf) — ~1 h; Table 1 and + the §7 graphs carry the message. + +**Anchors used in this guide** + +| Where | What | +|---|---| +| Hekaton §2, §2.1.2 | the 3–4× / 90% / 99% instruction-count argument; "no latches or spinlocks on any performance-critical path" | +| Hekaton §4, §4.1, §4.2, Figure 2 | version record layout; bucket scan at read time 15; the $20 Larry→John transfer and the `Tx75 → 100` fix-up | +| Hekaton §6, §6.1, §6.2.1–§6.2.3 | read stability and phantom avoidance; `Begin < RT and End > RT`; commit dependencies, read barriers, post-processing, rollback | +| Hekaton §8.1.1, §8.1.2 | oldest-active watermark; cooperative unlinking; "dusty corners" | +| Hekaton §9.1, §9.1.1, §9.1.2 | 2.67 GHz Xeon W3520, 1M rows; 20× / 10.8× lookups; ~30× updates | +| Wu Table 1 | nine systems placed on the four axes; postgres and Hekaton both append-only O2N with physical pointers | +| Wu §3.1–§3.4 | MVTO, MVOCC, MV2PL, SI+SSN; `read-ts`, `read-cnt`, no-wait | +| Wu §4, §5, §5.1, §6.1, §6.2 | version storage schemes; GC granularities; COOP requires O2N; logical vs physical pointers | +| Wu §7, Fig 6–Fig 25 | the whole price list; hardware and method in §7 | +| Wu §8 | the four findings and the Fig 24 shootout verdict | + +**In this repo** + +| Where | What | +|---|---| +| [`notes.md`](notes.md) | the measured mutex baseline and the `stub` MVCC columns | +| [`FINDINGS.md`](../../FINDINGS.md) row 8 | the headline: flat ~600k txn/s, because the mutex already serialized everything | +| `experiments/src/mvcc.rs:105` | `commit()` — where first-committer-wins goes | +| [`reading-postgres-heapam.md`](reading-postgres-heapam.md) | the disk-era design this one is defined against | +| [`reading-ssi-postgres.md`](reading-ssi-postgres.md) | SIREAD locks, the alternative to Step 4's commit-time validation | diff --git a/topics/08-transactions-mvcc/reading-postgres-heapam.md b/topics/08-transactions-mvcc/reading-postgres-heapam.md index 2813e7c..824022e 100644 --- a/topics/08-transactions-mvcc/reading-postgres-heapam.md +++ b/topics/08-transactions-mvcc/reading-postgres-heapam.md @@ -5,145 +5,485 @@ creator and deleter, and visibility is a pure function of (tuple header, snapshot) — no lock manager consulted on the read path. Before you open heapam, this chapter builds that machine step by step — the versioned tuple, the header fields, the snapshot, the visibility function that is -the spec of snapshot isolation, the write paths, and the debt collectors — -then hands you the file:line anchors to watch each piece work. +the spec of snapshot isolation, a worked example on real numbers, the write +paths, and the debt collectors — then hands you the file:line anchors to +watch each piece work. + +**Every line number in this guide was re-verified against +`postgres/postgres@701f021`** (check with `python3 tools/pinned-source.py +ref postgres`). Postgres's heap files are large — `heapam.c` is 9264 lines, +`heapam_visibility.c` is 1753 — so use +`python3 tools/pinned-source.py grep postgres --path ` to +jump, not `show`. ## The problem in one sentence Let hundreds of readers scan a table while writers update it, with no reader ever taking a lock — postgres's answer is to never overwrite a row -and to make "can I see this version?" a pure function two integers wide, -paying for it with dead versions that a vacuum process must collect later. +and to make "can I see this version?" a pure function of two integers in +the row's own header against three fields in the reader's snapshot, paying +for it with dead versions that a vacuum process must collect later. ## The concepts, step by step ### Step 1 — versions live in the table itself -In postgres, an UPDATE never modifies the row in place: it inserts a -complete new copy of the row (a new **version**) elsewhere in the table's -storage (the **heap** — the file of 8 KB pages holding the actual rows), -and marks the old copy as superseded. A DELETE just marks. Nothing is -physically removed at delete/update time; old versions sit in the heap -next to live ones until a cleanup pass reclaims them. - -Consequence: at any instant the heap contains *several versions of the -same logical row*, and every reader must decide, per version, "is this the -one I should see?" — using only information stored in the version itself -plus the reader's own context. That decision procedure is the rest of this -chapter. +> **In:** a table, a stream of UPDATEs, and readers who must not block. +> **Out:** a heap that holds several versions of the same logical row at +> once, and the obligation to decide per version whether a given reader +> should see it. + +The vocabulary this guide uses, defined once: + +- A **transaction** is a group of reads and writes that must appear to + happen all-at-once or not at all. +- **MVCC** (multi-version concurrency control) means writers never + overwrite: each update creates a new **version** of the row. +- A **tuple** is postgres's name for one such version — a physical row + image plus a header. +- The **heap** is the file of 8 KB pages holding those tuples. +- A **version chain** is the successive versions of one logical row, linked + through the header. +- **Visibility** is the predicate "should this reader see this version?". +- **Garbage collection** is reclaiming versions no live reader can still + see; postgres's implementation is called **vacuum**. + +In postgres an UPDATE never modifies the row in place: it inserts a +complete new copy of the row elsewhere in the heap and marks the old copy +as superseded. A DELETE just marks. Nothing is physically removed at +delete/update time; old versions sit in the heap next to live ones until a +cleanup pass reclaims them. + +Consequence: at any instant the heap contains *several versions of the same +logical row*, and every reader must decide, per version, "is this the one I +should see?" — using only information stored in the version itself plus the +reader's own context. That decision procedure is the rest of this chapter. + +Why it matters: everything postgres does differently from an in-memory +engine (Hekaton, previous guide) follows from "the version is a row image +on a page". A page-resident header has no room for a commit timestamp, so +you get xids plus a commit log; a page cannot be half-freed, so you get +vacuum. ### Step 2 — the tuple header: creator, deleter, and a chain pointer -Each row version (a **tuple**) carries a header naming who made it and who -killed it. The identifiers are **xids** (transaction ids — a global 32-bit -counter assigned to each writing transaction in start order): +> **In:** a tuple sitting on a page, with nothing else to consult. +> **Out:** three header fields — `t_xmin`, `t_xmax`, `t_ctid` — that encode +> the tuple's whole MVCC life, and the caveat that makes chain-walking +> defensive. + +Each tuple carries a header naming who made it and who killed it. The +identifiers are **xids** (transaction ids — a global 32-bit counter handing +out one id per writing transaction, in start order): + +```c +// src/include/access/htup_details.h — HeapTupleFields, 124-125 + 124 TransactionId t_xmin; /* inserting xact ID */ + 125 TransactionId t_xmax; /* deleting or locking xact ID */ -- `t_xmin` — the xid of the transaction that *inserted* this version. -- `t_xmax` — the xid of the transaction that *deleted or superseded* it - (0 = still live). -- `t_ctid` — a pointer to the *newer* version of the same row, forming a - version chain through the heap. +// src/include/access/htup_details.h — HeapTupleHeaderData, 161 + 161 ItemPointerData t_ctid; /* current TID of this or newer tuple (or a +``` + +- **`t_xmin`** — the xid of the transaction that *inserted* this version. +- **`t_xmax`** — the xid of the transaction that *deleted, superseded or + locked* it (note the header comment says "deleting **or locking**" — + `t_xmax` is not purely a tombstone, which is why Step 5 has to test + `HEAP_XMAX_IS_LOCKED_ONLY`). +- **`t_ctid`** — a pointer to the *newer* version of the same row, forming + the version chain through the heap. So the tuple's whole MVCC life is three fields: born at `t_xmin`, died at `t_xmax`, successor at `t_ctid`. An UPDATE = insert new version + set old -tuple's `t_xmax` + link `t_ctid`. A caveat the source shouts about -(htup_details.h:86–111): the chain can be broken by cleanup, so following +tuple's `t_xmax` + link `t_ctid`. + +The source shouts a caveat about that last field, in the comment at +`htup_details.h:86-111`: + +```c +// src/include/access/htup_details.h — the t_ctid comment, 86-111 (elided) + 86 * A word about t_ctid: whenever a new tuple is stored on disk, its t_ctid + 88 * its t_ctid is changed to point to the replacement version of the tuple. Or + 93 * t_ctid points to itself (in which case, if XMAX is valid, the tuple is + 94 * either locked or deleted). One can follow the chain of t_ctid links + 98 * tuple. Hence, when following a t_ctid link, it is necessary to check + 105 * t_ctid is sometimes used to store a speculative insertion token, instead + 111 * see a speculative insertion token while following a chain of t_ctid links, +``` + +Two hazards in one field: the chain can be broken by cleanup, so following `t_ctid` requires re-checking that the next tuple's `xmin` equals this -tuple's `xmax` — and `t_ctid` is overloaded for speculative insertion -tokens. Chains are walked defensively. +tuple's `xmax`; and `t_ctid` is overloaded to carry speculative-insertion +tokens and a "moved to another partition" marker. Chains are walked +defensively. + +Why it matters: "the version is self-describing" is only two-thirds true +here. `t_xmin` and `t_xmax` are xids, not timestamps, and an xid does not +say whether its transaction committed. That gap is Step 3. ### Step 3 — hint bits: caching "did that transaction commit?" -An xid alone doesn't say whether its transaction committed or aborted — -that lives in the **clog** (commit log — a global bitmap keyed by xid, -two bits per transaction). Checking clog per tuple per read would add a -(cached, but real) lookup to every visibility test. So the first reader -that pays the clog probe writes the answer *back into the tuple header* as -**hint bits**: `HEAP_XMIN_COMMITTED`, `HEAP_XMIN_INVALID`, and the xmax -equivalents. Every later reader tests one bit and skips clog entirely. +> **In:** an xid in a tuple header, and a global commit log that knows its +> fate. +> **Out:** the caching scheme that keeps the commit-log probe off the hot +> path, and the two costs it creates. + +An xid alone does not say whether its transaction committed or aborted — +that lives in the **clog** (commit log: a global array, two bits per +transaction). Probing the clog per tuple per read would add a lookup to +every visibility test. So the first reader that pays the probe writes the +answer *back into the tuple header* as **hint bits**: + +```c +// src/include/access/htup_details.h — infomask hint bits, 204-208 + 204 #define HEAP_XMIN_COMMITTED 0x0100 /* t_xmin committed */ + 205 #define HEAP_XMIN_INVALID 0x0200 /* t_xmin invalid/aborted */ + 206 #define HEAP_XMIN_FROZEN (HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID) + 207 #define HEAP_XMAX_COMMITTED 0x0400 /* t_xmax committed */ + 208 #define HEAP_XMAX_INVALID 0x0800 /* t_xmax invalid/aborted */ +``` + +Note line 206: the two "impossible together" bits set at once is the +encoding for **frozen** — a tuple whose xmin is so old it needs no +comparison at all. That matters in Step 9. + +Every later reader tests one bit and skips the clog. The costs, both real: + +1. **Reads now write.** Setting a hint bit dirties the page, so a pure + SELECT can generate write I/O — and the comment at + `heapam_visibility.c:106-108` names the failure it has to guard against: + "the page must not be undergoing IO at this time (otherwise we e.g. could + corrupt PG's page checksum or even the filesystem's, as is known to + happen with btrfs)". +2. **Permission has to be acquired**, and that is not free, so it is + amortized across a page: + +```c +// src/backend/access/heap/heapam_visibility.c — SetHintBitsState, 83-99 (elided) + 83 * To be allowed to set hint bits, SetHintBits() needs to call + 84 * BufferBeginSetHintBits(). However, that's not free, and some callsites call + 85 * SetHintBits() on many tuples in a row. For those it makes sense to amortize + 86 * the cost of BufferBeginSetHintBits(). Additionally it's desirable to defer + 87 * the cost of BufferBeginSetHintBits() until a hint bit needs to actually be + 91 typedef enum SetHintBitsState + 93 /* not yet checked if hint bits may be set */ + 94 SHB_INITIAL, + 95 /* failed to get permission to set hint bits, don't check again */ + 96 SHB_DISABLED, + 97 /* allowed to set hint bits */ + 98 SHB_ENABLED, + 99 } SetHintBitsState; +``` -The costs, both real: reads now dirty pages (a SELECT can generate write -IO — question 2), and the bits are only hints, so they can be set lazily -and are batched per page by the `SetHintBits` machinery. Reader-writes- -metadata is the same trick as topic 6's buffer usage counters. +Three states, because "we have not asked yet" and "we asked and were told +no" are different and only the first is worth retrying. Amortize-and-batch, +the same pattern as topic 6's buffer usage counters — and reader-writes- +metadata, the same trick. -### Step 4 — the snapshot: three numbers that freeze time +Why it matters: hint bits are what make Step 5's visibility function cheap +*in the common case*. Every cost below is stated for the hinted path unless +said otherwise. + +### Step 4 — the snapshot: three fields that freeze time + +> **In:** a reader that needs a stable definition of "committed already". +> **Out:** `xmin`, `xmax`, `xip[]` — and the cost model of testing an xid +> against them. A **snapshot** is the reader's definition of "now": a compact description -of exactly which transactions had committed when the snapshot was taken. -It is three parts — `xmin`, `xmax`, and `xip[]`: +of exactly which transactions had finished when the snapshot was taken. + +```c +// src/include/utils/snapshot.h — SnapshotData, 148-165 (elided) + 148 * An MVCC snapshot can never see the effects of XIDs >= xmax. It can see + 149 * the effects of all older XIDs except those listed in the snapshot. xmin + 150 * is stored as an optimization to avoid needing to search the XID arrays + 153 TransactionId xmin; /* all XID < xmin are visible to me */ + 154 TransactionId xmax; /* all XID >= xmax are invisible to me */ + 162 * note: all ids in xip[] satisfy xmin <= xip[i] < xmax + 164 TransactionId *xip; + 165 uint32 xcnt; /* # of xact ids in xip[] */ +``` - every xid `< xmin` — finished before I started: decided (visible if committed); -- every xid `>= xmax` — started after me: **invisible**, unconditionally; -- `xip[]` — the list of xids in progress at snapshot time (between xmin - and xmax but not yet finished): **invisible**, even if they commit later. +- every xid `>= xmax` — started after me: **invisible**, unconditionally, + *even if it has already committed in real time*; +- `xip[]` — the xids in progress at snapshot time: **invisible**, even if + they commit a microsecond later. + +`xcnt` at line 165 is the length of `xip[]`, and it is the cost driver. +Building a snapshot means scanning the shared array of running backends +(`GetSnapshotData`, `procarray.c:2114`) — the scan that was postgres's +multicore scalability wall until the 2020 rework added +`GetSnapshotDataReuse` (`procarray.c:2034`), which reuses the previous +snapshot wholesale when nothing has committed since. + +The membership test is `XidInMVCCSnapshot` (`snapmgr.c:1869`), and it is +worth reading exactly, because it is *not* what you would guess: + +```c +// src/backend/utils/time/snapmgr.c — XidInMVCCSnapshot, 1879-1924 (elided) + 1879 /* Any xid < xmin is not in-progress */ + 1880 if (TransactionIdPrecedes(xid, snapshot->xmin)) + 1881 return false; + 1882 /* Any xid >= xmax is in-progress */ + 1883 if (TransactionIdFollowsOrEquals(xid, snapshot->xmax)) + 1884 return true; + 1899 if (!snapshot->suboverflowed) + 1901 /* we have full data, so search subxip */ + 1902 if (pg_lfind32(xid, snapshot->subxip, snapshot->subxcnt)) + 1903 return true; + 1924 if (pg_lfind32(xid, snapshot->xip, snapshot->xcnt)) + 1925 return true; +``` + +`pg_lfind32` is a **linear** search, SIMD-accelerated — not a binary +search, even though `xip[]` is sorted: + +```c +// src/include/port/pg_lfind.h — pg_lfind32, 158-167 (elided) + 158 /* + 159 * For better instruction-level parallelism, each loop iteration operates + 160 * on a block of four registers. + 161 */ + 163 const uint32 nelem_per_vector = sizeof(Vector32) / sizeof(uint32); + 164 const uint32 nelem_per_iteration = 4 * nelem_per_vector; +``` + +`Vector32` is `__m128i` on x86 and `uint32x4_t` on ARM (`simd.h:30`, +`simd.h:35`) — 128 bits, so `nelem_per_vector` = 4 and +`nelem_per_iteration` = **16 xids per loop iteration**. -Building one means scanning the shared array of running backends -(`GetSnapshotData`) — this scan was postgres's multicore scalability wall -until the 2020 rework, and the in-snapshot test (`XidInMVCCSnapshot`) is a -binary search over `xip[]`, so snapshot cost scales with the number of -concurrent write transactions (question 3 compares Hekaton's one-counter -answer). +Work the cost. With 10 000 concurrent write transactions, `xcnt` = 10 000, +so a worst-case miss costs 10 000 ÷ 16 = **625 iterations**, per tuple, per +visibility test. Both cheap branches fire first (lines 1880 and 1883), so +the scan only runs for xids in the `[xmin, xmax)` window — but a snapshot +taken during a write storm has a wide window. Compare Hekaton, where the +same question is two integer comparisons against fields already in the +cache line ([`reading-inmemory-mvcc.md`](reading-inmemory-mvcc.md) Step 2). +That contrast is question 3. + +Why it matters: this is the one place in postgres's read path whose cost +scales with *concurrency* rather than with data size. It is the reason the +2020 `GetSnapshotDataReuse` work existed and the reason long-running +transactions hurt everyone, not just themselves. ### Step 5 — the visibility function: the spec of snapshot isolation -Now combine Steps 2–4: a version is visible iff its creator is visible to -my snapshot AND its deleter (if any) is not. `HeapTupleSatisfiesMVCC` is -that sentence plus a decade of engineering; its logical skeleton: +> **In:** Step 2's header fields, Step 3's hint bits, Step 4's snapshot. +> **Out:** `HeapTupleSatisfiesMVCC` — a pure function of (tuple, snapshot) +> and the operational definition of snapshot isolation in postgres. + +A version is visible iff its creator is visible to my snapshot AND its +deleter (if any) is not. `HeapTupleSatisfiesMVCC` +(`heapam_visibility.c:939`) is that sentence plus a decade of engineering. +Its skeleton, with the lines that carry the argument: + +```c +// src/backend/access/heap/heapam_visibility.c — HeapTupleSatisfiesMVCC, 956-1092 (heavily elided) + 956 if (!HeapTupleHeaderXminCommitted(tuple)) + 958 if (HeapTupleHeaderXminInvalid(tuple)) + 959 return false; // creator aborted + 963 else if (TransactionIdIsCurrentTransactionId(HeapTupleHeaderGetRawXmin(tuple))) + 965 if (HeapTupleHeaderGetCmin(tuple) >= snapshot->curcid) + 966 return false; /* inserted after scan started */ + 1005 else if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmin(tuple), snapshot)) + 1006 return false; // creator in flight at snapshot time + 1007 else if (TransactionIdDidCommit(HeapTupleHeaderGetRawXmin(tuple))) + 1008 SetHintBitsExt(tuple, buffer, HEAP_XMIN_COMMITTED, ... + 1021 if (!HeapTupleHeaderXminFrozen(tuple) && + 1022 XidInMVCCSnapshot(HeapTupleHeaderGetRawXmin(tuple), snapshot)) + 1023 return false; /* treat as still in progress */ + 1026 /* by here, the inserting transaction has committed */ + 1028 if (tuple->t_infomask & HEAP_XMAX_INVALID) /* xid invalid or aborted */ + 1029 return true; // never deleted + 1031 if (HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask)) + 1032 return true; // xmax is a lock, not a delete + 1071 if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmax(tuple), snapshot)) + 1072 return true; // deleter in flight ⇒ still alive to me + 1089 if (XidInMVCCSnapshot(HeapTupleHeaderGetRawXmax(tuple), snapshot)) + 1090 return true; /* treat as still in progress */ +``` -1. xmin aborted → invisible; xmin in-progress and not me → invisible -2. xmin mine and cid < my command → visible (read-your-own-writes lives - here, via CommandId — statement-level granularity inside a txn) -3. xmin committed but `XidInMVCCSnapshot(xmin)` → invisible (committed - AFTER my snapshot — this is the line that makes it "snapshot") -4. then the same dance for xmax to decide "deleted yet, for me?" +Line **1005** is the one that makes this "snapshot" isolation rather than +"read committed": a creator that committed *after* my snapshot was taken is +invisible to me forever, no matter how long I run. Line 1071 is its mirror +for deletes. + +Two details the skeleton hides, both important: + +- **Line 963 comes before line 1005 for a reason.** Your own xid is never + stored in your own snapshot — the comment at `snapmgr.c:1862-1866` says + "GetSnapshotData never stores either top xid or subxids of our own backend + into a snapshot", so `XidInMVCCSnapshot` would wrongly report your own + in-flight writes as *not* in progress. The current-transaction check must + run first. Read-your-own-writes lives at line 965, at **command** + granularity (`CommandId`), not transaction granularity. +- **Lines 922-936 explain a deliberate non-optimization.** When the + inserting transaction is still running according to your snapshot, + postgres does *not* update the hint bits, even if the transaction has in + fact committed: "Checking the true transaction state would require access + to high-traffic shared data structures, creating contention we'd rather do + without, and it would not change the result of our visibility check + anyway." The same function, minus the hint-bit engineering: ```rust +// ILLUSTRATION — not quoted from postgres. This is the logical skeleton of +// HeapTupleSatisfiesMVCC at heapam_visibility.c:939 and XidInMVCCSnapshot at +// snapmgr.c:1869, with hint bits, MultiXacts, subtransactions and frozen +// xids removed. Read the real thing; this is only a crib for Step 6. fn satisfies_mvcc(t: &Tuple, s: &Snapshot) -> bool { // "visible xid" = committed AND not still in flight at snapshot time let vis = |xid: Xid| committed(xid) && !in_snapshot(xid, s); if t.xmin == s.my_xid { - if t.cmin >= s.cur_cid { return false; } // later command in my own txn + if t.cmin >= s.cur_cid { return false; } // heapam_visibility.c:965 } else if !vis(t.xmin) { return false; // creator invisible to me } match t.xmax { - None => true, // never deleted + None => true, // never deleted (:1028) Some(x) if x == s.my_xid => t.cmax >= s.cur_cid, Some(x) => !vis(x), // deleter invisible ⇒ row lives } } fn in_snapshot(xid: Xid, s: &Snapshot) -> bool { // committed AFTER my snapshot? - xid >= s.xmax || (xid >= s.xmin && s.xip.binary_search(&xid).is_ok()) + if xid < s.xmin { return false; } // snapmgr.c:1880 + if xid >= s.xmax { return true; } // snapmgr.c:1883 + s.xip.contains(&xid) // pg_lfind32: LINEAR SIMD scan } ``` -Note what's absent: no locks, no waiting, no consulting other backends — +Note what is absent: no locks, no waiting, no consulting other backends — a pure function of two arguments. That purity is the entire read-side -scalability story. (There is a second visibility function, -`HeapTupleSatisfiesUpdate`, used by writers to find the latest version and -report "being updated by someone else" — that's where waiting and the -EvalPlanQual re-check originate.) +scalability story. There is a second visibility function, +`HeapTupleSatisfiesUpdate` (`heapam_visibility.c:511`), used by writers to +find the latest version and report "being updated by someone else"; that is +where waiting and the EvalPlanQual re-check originate. And +`HeapTupleSatisfiesMVCCBatch` (`heapam_visibility.c:1690`) runs the same +predicate over a whole page at once — topic 11 foreshadowing. -### Step 6 — the write paths, and the HOT shortcut +Why it matters: this function *is* the isolation level. There is no +separate rulebook; snapshot isolation in postgres is defined by which +branches of these 150 lines return true. -With the machinery above, the write paths are almost anticlimactic: +### Step 6 — the visibility function, executed on numbers -- **insert** — new tuple: xmin = my xid, xmax = 0. (The WAL record is - built AFTER the page change, inside the critical section — topic 5's - reserve-then-copy in action.) -- **delete** — nothing moves; set xmax, clear some flags. A "delete" is a - metadata write. -- **update** — insert + mark + link, per Step 1… with one big exception. +> **In:** one concrete snapshot and six concrete tuple headers. +> **Out:** six visibility decisions, each traced to the line of +> `heapam_visibility.c` that produced it — the exercise you should be able +> to do from memory. -The exception is **HOT** (heap-only tuple) updates: if no indexed column -changed and the new version fits on the *same page*, skip all index -updates. The index keeps pointing at the chain head; readers walk `t_ctid` -within the page to reach the live version: +Take one reader, backend B, whose transaction was assigned xid **105** and +which is executing its **third** command (`curcid = 2`, counting from 0). +Its snapshot S: + +``` + S.xmin = 100 → every xid < 100 has finished; trust the clog + S.xmax = 110 → every xid >= 110 is invisible, unconditionally + S.xip[] = [103, 107] (xcnt = 2) → in flight when S was taken + S.curcid = 2 + my xid = 105 → NOT in xip[]; own xids are never stored (snapmgr.c:1862) +``` + +Six tuples, and the decision each one gets: + +| # | Tuple header | Decision | Line that decides it | +|---|---|---|---| +| 1 | `xmin=95, xmax=103` | **visible** | `:1071` | +| 2 | `xmin=103, xmax=0` | invisible | `:1005` | +| 3 | `xmin=112, xmax=0` | invisible | `:1005` (via `snapmgr.c:1883`) | +| 4 | `xmin=88, xmax=99` | invisible | `:1089` falls through | +| 5 | `xmin=105, cmin=0, xmax=0` | **visible** | `:968` | +| 6 | `xmin=105, cmin=3, xmax=0` | invisible | `:965` | + +Traced one at a time: + +**Tuple 1 — `xmin=95, xmax=103`, hint bit `HEAP_XMIN_COMMITTED` set.** +Line 956's test fails (xmin *is* hinted committed), so we take the `else` +at :1018. Line 1022: `XidInMVCCSnapshot(95, S)` → `snapmgr.c:1880`, +`95 < 100` → **false**, so we do not return early. Line 1026: the inserter +committed. Line 1028: `xmax = 103` is valid, so not `HEAP_XMAX_INVALID`. +Line 1031: not lock-only. Line 1061: xmax has no `HEAP_XMAX_COMMITTED` hint +(103 is still running). Line 1063: `103 ≠ 105`, not mine. Line 1071: +`XidInMVCCSnapshot(103, S)` → not `< 100`, not `>= 110`, and +`pg_lfind32(103, [103,107], 2)` finds it → **true** → `return true`. +**Visible.** The row was deleted by a transaction that had not committed +when S was taken, so as far as B is concerned it is still alive — and it +will stay alive to B even after 103 commits. + +**Tuple 2 — `xmin=103, xmax=0`** (the new version 103 wrote). No hint bit, +so line 956 is entered; line 958 no; line 963 `103 ≠ 105`; line 1005 +`XidInMVCCSnapshot(103, S)` → true → **`return false`. Invisible.** Tuples +1 and 2 together are one version chain, and B sees exactly one of them — +the old one. That is snapshot isolation in a single row. + +**Tuple 3 — `xmin=112, xmax=0`.** Line 1005 → `snapmgr.c:1883`, +`TransactionIdFollowsOrEquals(112, 110)` → true → in-snapshot → **`return +false`. Invisible.** Note what was *not* consulted: whether 112 committed. +It may have committed long before B ran this query; xid 112 started after +S was taken, so it is invisible regardless. + +**Tuple 4 — `xmin=88, xmax=99`, both hinted committed.** Line 1022: +`88 < 100` → not in snapshot → inserter visible. Line 1028: xmax valid. +Line 1061: `HEAP_XMAX_COMMITTED` *is* set, so we take the `else` at :1086. +Line 1089: `XidInMVCCSnapshot(99, S)` → `99 < 100` → false. Falls through +to `return false`. **Invisible** — deleted by a transaction that finished +before B started. + +**Tuple 5 — `xmin=105` (mine), `cmin=0`, `xmax` invalid.** Line 956 entered +(own xid, no hint bit). Line 963: `TransactionIdIsCurrentTransactionId(105)` +→ **true**. Line 965: `cmin = 0 >= curcid = 2`? No. Line 968: +`HEAP_XMAX_INVALID` → **`return true`. Visible.** This is +read-your-own-writes: B inserted this in command 0 and reads it in command +2. + +**Tuple 6 — `xmin=105` (mine), `cmin=3`.** Same path to line 965: +`cmin = 3 >= curcid = 2` → **`return false`** — "inserted after scan +started". Same transaction, same xid, opposite answer, decided entirely by +`CommandId`. If B's statement were `UPDATE t SET n = n + 1`, this is the +line that stops it looping forever over the rows it is itself producing. + +Why it matters: notice how few of the six decisions consulted the clog +(only the unhinted ones), and how many were settled by the two integer +comparisons at `snapmgr.c:1880` and `:1883`. The `xip[]` scan ran exactly +twice out of six. That distribution is why the design works. + +### Step 7 — the write paths, and the HOT shortcut + +> **In:** the read-side machinery of Steps 2–6. +> **Out:** three write paths that are almost anticlimactic, plus the one +> optimization that changes an update's cost by a factor of *number of +> indexes*. + +- **insert** (`heapam.c:2004`) — new tuple: xmin = my xid, xmax = 0. +- **delete** (`heapam.c:2717`) — nothing moves; set xmax, adjust flags. A + "delete" is a metadata write to an existing tuple. +- **update** (`heapam.c:3201`) — insert + mark + link, per Step 1… with one + big exception. + +The exception is **HOT** (heap-only tuple) updates. The decision is two +conditions: + +```c +// src/backend/access/heap/heapam.c — the HOT decision inside heap_update, 3972-3981 + 3972 if (newbuf == buffer) + 3974 /* + 3975 * Since the new tuple is going into the same page, we might be able + 3976 * to do a HOT update. Check if any of the index columns have been + 3977 * changed. + 3978 */ + 3979 if (!bms_overlap(modified_attrs, hot_attrs)) + 3981 use_hot_update = true; +``` + +`modified_attrs` comes from `HeapDetermineColumnsInfo` (called at +`heapam.c:3382`, defined at `heapam.c:4360`). If the new version fits on +the *same page* and no HOT-blocking indexed column changed, the index keeps +pointing at the chain head and readers walk `t_ctid` within the page: ``` HOT chain (one page): index entry ──► lp 1 (root, HOT_UPDATED) @@ -154,86 +494,323 @@ within the page to reach the live version: readers walk the chain under the page latch; prune collapses it later. ``` -Why it matters: a table with 5 indexes turns every non-HOT update into 6 -inserts (heap + 5 index entries); HOT makes it 1. This is why -"UPDATE = INSERT+DELETE" is only *half* true in postgres. - -### Step 7 — the debt collectors: prune and vacuum - -Every update and delete leaves a dead version in the heap — MVCC's rent. -Two collectors, opportunistic and thorough: +The flags are set at `heapam.c:4029-4036`, and the index work is signalled +to the caller at `heapam.c:4159-4167`: + +```c +// src/backend/access/heap/heapam.c — what indexes a HOT update still touches, 4159-4167 + 4159 if (use_hot_update) + 4161 if (summarized_update) + 4162 *update_indexes = TU_Summarizing; + 4163 else + 4164 *update_indexes = TU_None; + 4166 else + 4167 *update_indexes = TU_All; +``` -- **page pruning** (`heap_page_prune_opt`) — any *reader* that notices a - page with prunable garbage cleans that one page in passing: dead - versions removed, HOT chains collapsed to a redirect line pointer. No - vacuum needed for the common case. -- **vacuum** (`heap_vacuum_rel` / `lazy_scan_heap`) — the full pass: - collect dead tuple ids, delete the index entries pointing at them, and - only THEN mark the heap line pointers reusable. Two-phase because an - index entry must never point at a reused slot (question 1 makes you - construct the corruption). +**A correction to a claim this guide used to make**: HOT does *not* always +skip all index updates. `TU_Summarizing` at line 4162 means a HOT update +still has to maintain summarizing indexes (BRIN), because per the comment +at `heapam.c:4154-4157` a summary such as a "minmax bounds of the block may +change with this update". Only `TU_None` skips everything. + +Work the arithmetic anyway, because the order of magnitude is the point. A +table with 5 non-summarizing B-tree indexes: + +- **non-HOT update** → 1 heap insert + 5 index inserts = **6 writes**. +- **HOT update** → 1 heap insert + 0 index inserts = **1 write**, six times + less work, and no index bloat to vacuum later. + +Why it matters: "UPDATE = INSERT + DELETE" is only half true in postgres, +and which half you get depends on two things a schema designer controls — +whether the updated column is indexed, and whether `fillfactor` leaves room +on the page for the new version. + +### Step 8 — the debt collectors: prune and vacuum + +> **In:** a heap accumulating one dead tuple per update and per delete. +> **Out:** two collectors with different scopes, and the ordering +> constraint that forces vacuum to be two-phase. + +- **Page pruning** — `heap_page_prune_opt` (`pruneheap.c:271`). Any + *reader* that touches a page with prunable garbage cleans that one page in + passing: dead versions removed, HOT chains collapsed to a redirect line + pointer. The fast exit is at `pruneheap.c:293-295` (`PageGetPruneXid` + returns invalid → return immediately), so the check costs almost nothing + on clean pages. No vacuum needed for the common case. +- **Vacuum** — `heap_vacuum_rel` (`vacuumlazy.c:624`), scanning via + `lazy_scan_heap` (`vacuumlazy.c:1279`). The full pass, and it is + deliberately two-phase: + +```c +// src/backend/access/heap/vacuumlazy.c — lazy_vacuum, 2454-2461 + 2454 else if (lazy_vacuum_all_indexes(vacrel)) + 2456 /* + 2457 * We successfully completed a round of index vacuuming. Do related + 2458 * heap vacuuming now. + 2459 */ + 2460 lazy_vacuum_heap_rel(vacrel); +``` -The famous failure mode: xids are 32-bit, so the counter wraps; vacuum -also "freezes" old tuples to keep xid comparisons valid — fall too far -behind and the database refuses writes. That is the operational price of -storing versions in the table. +Indexes first (`lazy_vacuum_all_indexes`, `vacuumlazy.c:2494`), *then* the +heap (`lazy_vacuum_heap_rel`, `vacuumlazy.c:2640`). The order is not +stylistic: a heap line pointer that has been marked reusable can be handed +to a brand-new, unrelated row at any moment, so any index entry still +pointing at it would silently return the wrong row. Question 1 makes you +construct that corruption. + +There is even a bypass: at `vacuumlazy.c:2436-2437`, if fewer than +`BYPASS_THRESHOLD_PAGES` worth of pages hold dead items *and* the dead-item +store is under 32 MB, vacuum skips index vacuuming entirely — the comment +at `:2387-2392` explains why ("avoids sharp discontinuities in the duration +and overhead of successive VACUUM operations"). + +Why it matters: pruning is opportunistic and page-local; vacuum is +scheduled and table-global; and the thing that makes vacuum expensive is +not the heap, it is having to walk every index. + +### Step 9 — the bill: 32-bit xids and freezing + +> **In:** a 32-bit counter that hands out one value per writing +> transaction, forever. +> **Out:** why vacuum is not optional, and the one flag combination that +> takes a tuple out of the comparison game. + +Xids are 32-bit, so the counter wraps after about 4.2 billion writing +transactions. Postgres compares xids *modulo* that space +(`TransactionIdPrecedes`, used at `snapmgr.c:1880`), which works only while +every live xid is within half the space of every other. So vacuum has a +second job besides reclaiming space: **freezing** old tuples — setting +`HEAP_XMIN_FROZEN` (`htup_details.h:206`, the two mutually-exclusive bits +set together) to mark an xmin as "older than everything, do not compare". +Step 5's line 1021 is where the frozen check short-circuits the snapshot +test. + +Fall too far behind and the database refuses new writes rather than risk +returning wrong answers. The failsafe path is visible in the vacuum code +around `vacuumlazy.c:2468` ("This happens when relfrozenxid or relminmxid +is too far in the past"). + +Why it matters: this is the operational price of storing versions in the +table with 32-bit ids. Hekaton's 64-bit timestamps do not wrap in any +practical timeframe — one of the clearest cases in this topic where a +representation choice made for the disk became an operational burden. ## Where each step lives in the code Read `HeapTupleSatisfiesMVCC` in full first — it is the spec of SI — and -the :86–111 comment in htup_details.h before chasing t_ctid. ~2.5 h total. - -- **Step 2 — the header** (`src/include/access/htup_details.h`): - `t_xmin`/`t_xmax` :124–125; `t_ctid` :161; the chain-walking caveats in - the big comment :86–111. -- **Step 3 — hint bits**: flag definitions htup_details.h:204–208 - (`HEAP_XMIN_COMMITTED / INVALID`, `HEAP_XMAX_*`); SetHintBits machinery - heapam_visibility.c:83–112 — note `SetHintBitsState`: even hint-bit - writes are batched now, amortizing BufferBeginSetHintBits over a page. - The amortize-and-batch pattern, again. -- **Step 4 — the snapshot**: `SnapshotData` — snapshot.h:138–165 (`xmin` - :153, `xmax` :154, `xip[]`/`xcnt` :164–165); `GetSnapshotData` — - procarray.c:2114 (and `GetSnapshotDataReuse` :2034 — if nothing - committed since, reuse the old snapshot wholesale, the 2020 scalability - fix); `XidInMVCCSnapshot` — snapmgr.c:1869, the three-way check. -- **Step 5 — visibility** (`heapam_visibility.c`): - `HeapTupleSatisfiesMVCC` :939 — read the whole thing; - `HeapTupleSatisfiesUpdate` :511 — the writer-side function. Bonus: - `HeapTupleSatisfiesMVCCBatch` :1690 — visibility vectorized over a page, - topic 11 foreshadowing. -- **Step 6 — write paths** (`heapam.c`): `heap_insert` :2004; - `heap_delete` :2717; `heap_update` :3201 — the long one, skim for the - shape: `HeapDetermineColumnsInfo` :3382 (which indexed columns - changed?), `use_hot_update` :3233/:3981, and :4029 where - HEAP_HOT_UPDATED is set and index inserts are skipped entirely. -- **Step 7 — collectors**: `heap_page_prune_opt` — pruneheap.c:271; - `heap_vacuum_rel` — vacuumlazy.c:624 and `lazy_scan_heap` :1279. +the `:86-111` comment in `htup_details.h` before chasing `t_ctid`. ~2.5 h +total. All anchors verified at `postgres/postgres@701f021`. + +| Step | File | Lines | What | +|---|---|---|---| +| 2 | `src/include/access/htup_details.h` | 124-125, 161 | `t_xmin`, `t_xmax`, `t_ctid` | +| 2 | `src/include/access/htup_details.h` | 86-111 | the t_ctid chain-walking caveats | +| 3 | `src/include/access/htup_details.h` | 204-208 | hint bit definitions; 206 is FROZEN | +| 3 | `src/backend/access/heap/heapam_visibility.c` | 83-99, 101-130 | `SetHintBitsState`, and why permission is batched | +| 4 | `src/include/utils/snapshot.h` | 138, 153, 154, 164-165 | `SnapshotData`; `xmin`, `xmax`, `xip[]`, `xcnt` | +| 4 | `src/backend/storage/ipc/procarray.c` | 2114, 2034 | `GetSnapshotData`; `GetSnapshotDataReuse` (the 2020 fix) | +| 4 | `src/backend/utils/time/snapmgr.c` | 1869, 1880, 1883, 1924 | `XidInMVCCSnapshot`; the two range tests; the `pg_lfind32` scan | +| 4 | `src/include/port/pg_lfind.h` | 153, 163-164 | `pg_lfind32`; 16 xids per iteration | +| 4 | `src/include/port/simd.h` | 30, 35 | `Vector32` = 128 bits | +| 5, 6 | `src/backend/access/heap/heapam_visibility.c` | 939, 922-936, 956-1092 | `HeapTupleSatisfiesMVCC`; the deliberate hint-bit non-optimization | +| 5 | `src/backend/access/heap/heapam_visibility.c` | 511, 1690 | `HeapTupleSatisfiesUpdate`; `HeapTupleSatisfiesMVCCBatch` | +| 7 | `src/backend/access/heap/heapam.c` | 2004, 2717, 3201 | `heap_insert`, `heap_delete`, `heap_update` | +| 7 | `src/backend/access/heap/heapam.c` | 3382, 4360 | `HeapDetermineColumnsInfo` — call site, then definition | +| 7 | `src/backend/access/heap/heapam.c` | 3233, 3972-3981, 4029-4036, 4159-4167 | `use_hot_update`: declared, decided, flagged, and what indexes it still touches | +| 8 | `src/backend/access/heap/pruneheap.c` | 271, 293-295 | `heap_page_prune_opt` and its fast exit | +| 8 | `src/backend/access/heap/vacuumlazy.c` | 624, 1279, 2369, 2436-2437, 2454-2461, 2494, 2640 | `heap_vacuum_rel`, `lazy_scan_heap`, `lazy_vacuum`, the bypass, indexes-then-heap | +| 9 | `src/backend/access/heap/vacuumlazy.c` | 2468 | the relfrozenxid failsafe | ## Questions for notes.md 1. Why must the index-entry deletion happen BEFORE line pointers are - recycled? Construct the corruption if the order flipped. -2. Hint bits make reads write. Which topic-6 lesson does that complicate - (think: checksums, dirty buffers from SELECTs)? -3. A snapshot with 10K concurrent writers makes XidInMVCCSnapshot a binary - search over 10K xids per tuple. What does Hekaton's timestamp design - pay instead? -4. FalkorDB angle: postgres stores versions IN the table (old versions - inflate the heap). For a graph whose "table" is a sparse matrix, where + recycled (`vacuumlazy.c:2454-2461`)? Construct the corruption if the + order flipped — name the two rows involved and the query that returns + the wrong one. +2. Hint bits make reads write. Which topic-6 lesson does that complicate, + and what does `heapam_visibility.c:106-108` say goes wrong if the rule is + broken during page I/O? +3. Step 4's arithmetic: 10 000 concurrent writers means `xcnt = 10 000` and + a worst-case 625 SIMD iterations per `XidInMVCCSnapshot` miss. What does + Hekaton's timestamp design pay for the same question, and what does it + give up to get there? +4. Trace tuple 1 of Step 6 again after xid 103 commits and B takes a *new* + snapshot. Which line's answer flips, and which tuple becomes visible? +5. FalkorDB angle: postgres stores versions IN the table, so old versions + inflate the heap. For a graph whose "table" is a sparse matrix, where would old versions live — and is that closer to append-only (postgres) - or delta (Hekaton per Wu/Pavlo taxonomy)? + or delta (Oracle/InnoDB) in the Wu/Pavlo taxonomy? + +## Takeaway + +Postgres's read path is a pure function: three fields in the tuple header +against three fields in the snapshot, with a commit-log probe cached into +the header the first time anyone pays it. Nothing on that path takes a +lock. The bill arrives everywhere else — dead versions that vacuum must +collect index-first, a snapshot whose cost scales with concurrent writers, +and a 32-bit id space that must be frozen before it wraps. + +## Connections to this topic's experiment + +The exercise in `experiments/src/mvcc.rs` is postgres's read path in +miniature: `Txn::get` (`experiments/src/mvcc.rs:89`) has to answer exactly +the question Step 5 answers, and `snapshot_reads_are_stable` and +`uncommitted_writes_are_invisible` are Step 6's tuples 1 and 2 as tests. + +The topic's *measured* lane is a different thing entirely, and worth being +precise about. The provided benchmark measures a single global +`Mutex` — 4 threads × 50 000 transactions × 4 operations — and on +an Apple M3 Pro (measured 2026-07-28, recorded in [`notes.md`](notes.md)) +it returns: + +| Workload | Keys | mutex txn/s | +|---|---|---| +| read-heavy 95/5 | 10 000 | 623 454 | +| write-heavy 50/50 | 10 000 | 594 264 | +| write-heavy 50/50 | 64 (hot) | 676 691 | + +Those three numbers are **flat** — the read/write mix and the key skew move +them by about 12%, which for workloads this different is no signal at all. +The reason is the negative result recorded in +[`FINDINGS.md`](../../FINDINGS.md) row 8: the mutex had already serialized +everything, so the workload's shape could not reach the measurement. + +This repo has **not** measured MVCC beating that mutex. The `mvcc txn/s` +and `aborts` columns in `notes.md` are `stub` — filling them is the +exercise. Nothing in this guide is evidence that MVCC is faster here; the +argument for postgres's design is a *scalability* argument (readers never +block writers), and a 4-thread benchmark on a laptop is not where that +argument is settled. ## Done when -You can execute `HeapTupleSatisfiesMVCC` on paper for: (a) my own insert, -(b) a commit that landed after my snapshot, (c) a HOT-updated row mid-chain. +Answer each before unfolding it. + +- [ ] Given the snapshot `xmin=100, xmax=110, xip=[103,107]`, decide the + visibility of a tuple with `xmin=95, xmax=103`, and name the line of + `heapam_visibility.c` that returns the answer. + +
Answer + +**Visible**, at `heapam_visibility.c:1071`. The inserter 95 is below +`S.xmin`, so `XidInMVCCSnapshot(95, S)` returns false at `snapmgr.c:1880` +and the creator is visible. The deleter 103 is in `xip[]`, so +`XidInMVCCSnapshot(103, S)` returns true at `snapmgr.c:1924`, and line 1071 +reads that as "the deleting transaction was still in flight when I took my +snapshot" → `return true`. The row stays visible to this reader even after +103 commits, because the snapshot does not change. + +
+ +- [ ] A transaction with xid 105 reads a tuple it inserted itself. Why is + `TransactionIdIsCurrentTransactionId` checked *before* + `XidInMVCCSnapshot`, and what decides the answer instead? + +
Answer + +Because your own xid is never in your own snapshot: "GetSnapshotData never +stores either top xid or subxids of our own backend into a snapshot" +(`snapmgr.c:1862-1866`). If `XidInMVCCSnapshot(105, S)` ran first it would +return false — "not in progress" — and the tuple would be treated as +committed by a stranger. The current-transaction test at +`heapam_visibility.c:963` runs first, and the answer is then decided by +**CommandId** at line 965: `cmin >= snapshot->curcid` → invisible +("inserted after scan started"). So read-your-own-writes has +statement-level granularity, which is what stops `UPDATE t SET n = n + 1` +from looping over its own output. + +
+ +- [ ] `XidInMVCCSnapshot` searches a sorted array. Why does it not binary + search, and what does it cost with 10 000 concurrent writers? + +
Answer + +It calls `pg_lfind32` (`snapmgr.c:1902` and `:1924`), a **SIMD-accelerated +linear** scan: `pg_lfind.h:163-164` sets `nelem_per_iteration = 4 * +nelem_per_vector`, and `Vector32` is 128 bits (`simd.h:30`, `simd.h:35`), so +each iteration compares **16 xids**. A worst-case miss over `xcnt = 10 000` +therefore costs 10 000 ÷ 16 = **625 iterations** — but with no branch +misprediction and perfectly sequential access, which is why a branchy binary +search over ~13 levels is not obviously better at these sizes. The two range +tests at `:1880` and `:1883` fire first and eliminate most xids before the +scan is reached at all. + +
+ +- [ ] Does a HOT update always skip index maintenance? Quote the code. + +
Answer + +No. `heapam.c:4159-4167` sets `*update_indexes` to `TU_None` only when +`summarized_update` is false; otherwise a HOT update yields +`TU_Summarizing`, because per `heapam.c:4154-4157` "the update may still +need to update summarized indexes, lest we fail to update those summaries +and get incorrect results (for example, minmax bounds of the block may +change with this update)" — BRIN. The saving is still large: with 5 +ordinary B-tree indexes, a non-HOT update is 6 writes and a HOT update is 1. + +
+ +- [ ] Why does `lazy_vacuum` vacuum indexes before the heap, and what + breaks if you swap them? + +
Answer + +`vacuumlazy.c:2454-2461` calls `lazy_vacuum_all_indexes` and only then +`lazy_vacuum_heap_rel`. Heap vacuuming is what marks line pointers reusable, +and a reusable line pointer can be handed to a completely unrelated new row +immediately. If the heap were cleaned first, an index entry still pointing +at that slot would resolve to whatever row now occupies it, and an index +scan would silently return the wrong row — corruption with no error. Index +first, heap second, always. + +
+ +- [ ] State this topic's measured result and what it does not show. + +
Answer + +The provided lane measures a global `Mutex` at 623 454 / 594 264 / +676 691 txn/s for read-heavy 10K-key, write-heavy 10K-key, and write-heavy +64-hot-key workloads (Apple M3 Pro, 2026-07-28; `notes.md`). The result is +**flat across all three**, which is the negative finding in +[`FINDINGS.md`](../../FINDINGS.md) row 8: the mutex had already serialized +everything, so workload shape could not influence throughput. It does +**not** show MVCC outperforming a mutex — those columns are `stub`, and the +case for postgres's design is about readers not blocking writers under +concurrency, which this 4-thread lane does not test. + +
## References -**Code** -- [postgres](https://github.com/postgres/postgres) — - `src/backend/access/heap/heapam.c`, `heapam_visibility.c`, - `src/include/access/htup_details.h`, `src/include/utils/snapshot.h`, - `src/backend/utils/time/snapmgr.c`, `pruneheap.c`, `vacuumlazy.c`; - ~2.5 h — read `HeapTupleSatisfiesMVCC` in full first, it is the spec - of SI, and the :86–111 comment in htup_details.h before chasing t_ctid +**Code** — all anchors verified at +[`postgres/postgres@701f021`](https://github.com/postgres/postgres) + +| File | Lines | What | +|---|---|---| +| `src/include/access/htup_details.h` | 86-111, 124-125, 161, 204-208 | tuple header, `t_ctid` caveats, hint bits | +| `src/include/utils/snapshot.h` | 138-165 | `SnapshotData` | +| `src/backend/storage/ipc/procarray.c` | 2034, 2114 | snapshot construction and reuse | +| `src/backend/utils/time/snapmgr.c` | 1862-1866, 1869-1925 | `XidInMVCCSnapshot` | +| `src/include/port/pg_lfind.h` | 89-99, 147-207 | linear and SIMD search helpers | +| `src/include/port/simd.h` | 30, 35 | `Vector32` | +| `src/backend/access/heap/heapam_visibility.c` | 83-130, 511, 917-1092, 1690 | hint-bit state, writer-side check, `HeapTupleSatisfiesMVCC`, batch variant | +| `src/backend/access/heap/heapam.c` | 2004, 2717, 3201-4167, 4360 | insert, delete, update, HOT | +| `src/backend/access/heap/pruneheap.c` | 271-300 | opportunistic pruning | +| `src/backend/access/heap/vacuumlazy.c` | 624, 1279, 2369-2470, 2494, 2640 | vacuum, two-phase ordering, bypass, failsafe | + +Read `HeapTupleSatisfiesMVCC` in full first — it is the spec of SI — and +the `:86-111` comment in `htup_details.h` before chasing `t_ctid`. ~2.5 h. + +**In this repo** + +| Where | What | +|---|---| +| [`notes.md`](notes.md) | the measured mutex baseline and the `stub` MVCC columns | +| [`FINDINGS.md`](../../FINDINGS.md) row 8 | flat ~600k txn/s, because the mutex already serialized everything | +| `experiments/src/mvcc.rs:89` | `Txn::get` — where Step 5's predicate goes | +| [`reading-inmemory-mvcc.md`](reading-inmemory-mvcc.md) | the same problem with the disk deleted | +| [`reading-ssi-postgres.md`](reading-ssi-postgres.md) | what postgres adds on top of this to reach SERIALIZABLE | diff --git a/topics/08-transactions-mvcc/reading-rocksdb-transactions.md b/topics/08-transactions-mvcc/reading-rocksdb-transactions.md index 5303f3f..1ca1efa 100644 --- a/topics/08-transactions-mvcc/reading-rocksdb-transactions.md +++ b/topics/08-transactions-mvcc/reading-rocksdb-transactions.md @@ -8,6 +8,11 @@ numbers give snapshots for free, and how the two schools bolt onto the same skeleton — differing only in WHEN conflicts are detected. Then it hands you the file:line anchors to watch both. +**Every line number below was re-verified against +`facebook/rocksdb@7c80a5a`** (check with `python3 tools/pinned-source.py +ref rocksdb`). Everything is under `utilities/transactions/` unless the +path says otherwise. + ## The problem in one sentence Two transactions write the same key concurrently and only one outcome is @@ -19,29 +24,64 @@ the work), and which is cheaper flips with the conflict rate. ### Step 1 — write buffering: a transaction is a private diff -Atomicity ("all my writes appear together, or none do") is easiest if the -database never sees partial state — so a transaction buffers every write -in a private, in-memory container and applies the whole batch to the DB -atomically at commit. RocksDB's container is a `WriteBatchWithIndex`: an -ordered batch of key/value operations plus a small index over itself, so -the transaction's own reads check the batch first (read-your-own-writes), -then fall through to the DB. +> **In:** the atomicity requirement — all my writes appear together or none +> do — and a storage engine that has no notion of "partially applied". +> **Out:** a private, indexed write batch that makes rollback free and +> uncommitted state invisible, plus a precise statement of what buffering +> does *not* solve. + +Definitions used from here on: + +- A **transaction** is a group of reads and writes that must appear to + happen all-at-once or not at all. +- A **snapshot** is a reader's frozen definition of "what had committed + when I started". +- **Isolation** is the guarantee about which other transactions' effects + you can see. +- A **conflict** is two transactions whose effects cannot both be kept. + +Atomicity is easiest if the database never sees partial state — so a +transaction buffers every write in a private, in-memory container and +applies the whole batch to the DB atomically at commit. RocksDB's container +is a `WriteBatchWithIndex`: an ordered batch of key/value operations plus a +small index over itself, so the transaction's own reads check the batch +first (read-your-own-writes), then fall through to the DB. The switch is +visible in the base class: + +```cpp +// utilities/transactions/transaction_base.h — indexing_enabled_, 455-459 + 455 // If true, future Put/PutEntity/Merge/Delete operations will be indexed in + 456 // the WriteBatchWithIndex. If false, future Put/PutEntity/Merge/Delete + 457 // operations will be inserted directly into the underlying WriteBatch and not + 458 // indexed in the WriteBatchWithIndex. + 459 bool indexing_enabled_; +``` + +The index is what costs; a transaction that never reads its own writes can +turn it off with `DisableIndexing` (`transaction_base.h:274`). Rollback becomes free (drop the batch), and nothing a transaction does is visible to anyone before commit. What buffering does NOT solve: two transactions buffering writes to the same key, each validating against a -world that doesn't yet contain the other. That is the conflict problem, +world that does not yet contain the other. That is the conflict problem, Steps 3–5. +Why it matters: every design below inherits this. Neither school has to +worry about undoing partially-applied work, which is why both fit on one +base class. + ### Step 2 — sequence numbers: the LSM gives MVCC for free +> **In:** an LSM tree (topic 4) that never overwrites in place, and a +> global write counter. +> **Out:** a snapshot that is *one integer*, and the one thing it costs. + Every write in RocksDB is stamped with a global, monotonically increasing -**sequence number** (seq), and — because the LSM (topic 4) never -overwrites in place — old values remain present as entries with older -seqs. So a **snapshot** is just one integer: "the seq at the moment I -began". A read at snapshot S returns, for each key, the newest entry with -seq ≤ S; entries newer than S are simply skipped during the merge across -memtable and SST files. +**sequence number** (seq), and because the LSM never overwrites in place, +old values remain present as entries with older seqs. So a snapshot is just +one integer: "the seq at the moment I began". A read at snapshot S returns, +for each key, the newest entry with seq ≤ S; entries newer than S are +skipped during the merge across memtable and SST files. ``` key k in the LSM: (k, seq=91) ── (k, seq=87) ── (k, seq=52) @@ -50,84 +90,285 @@ memtable and SST files. No version chains to maintain, no vacuum to schedule — old versions are garbage-collected by compaction, and a registered snapshot pins them -against that GC. Postgres built visibility machinery; RocksDB inherited -it from its storage layout. One cost to notice: a long-lived snapshot -blocks compaction from dropping anything newer than it. +against that GC. Postgres built visibility machinery +([`reading-postgres-heapam.md`](reading-postgres-heapam.md) Steps 2–5); +RocksDB inherited it from its storage layout. + +Snapshots can be taken eagerly or lazily: + +```cpp +// utilities/transactions/transaction_base.h — snapshot API, 264-272 + 264 void SetSnapshot() override; + 265 void SetSnapshotOnNextOperation( + 266 std::shared_ptr notifier = nullptr) override; + 267 + 268 void ClearSnapshot() override { + 269 snapshot_.reset(); + 270 snapshot_needed_ = false; + 271 snapshot_notifier_ = nullptr; + 272 } +``` + +`snapshot_needed_` is declared at `transaction_base.h:463` with the comment +"SetSnapshotOnNextOperation() has been called and the snapshot has not yet +been reset" — line 270 above is where `ClearSnapshot` resets it. And note +`TransactionOptions::set_snapshot` defaults to **false** +(`include/rocksdb/utilities/transaction_db.h:299`): by default a +transaction has no snapshot at all until it takes one, and each read sees +the latest committed data. + +Why it matters: the cost of this design is on the *other* side. A +long-lived snapshot pins every version newer than it against compaction, so +"a reader that never blocks a writer" is paid for in space, not in waiting. +The same bill postgres pays through vacuum, RocksDB pays through +compaction. ### Step 3 — the shared skeleton, and the one fork in the road +> **In:** Steps 1 and 2 — buffered writes and integer snapshots. +> **Out:** the base class both flavors *are*, and the single question that +> separates them. + Combine Steps 1–2 and you have the whole base class -(`transaction_base.{h,cc}`): reads go through the batch, then the DB at -the snapshot; writes buffer into the batch; commit applies the batch. Both -transaction flavors ARE this class. Note `SetSnapshot` -(transaction_base.h:264) and `snapshot_needed_` :270 — snapshots can be -taken lazily on first read. +(`transaction_base.{h,cc}`): reads go through the batch, then the DB at the +snapshot; writes buffer into the batch; commit applies the batch. Both +transaction flavors ARE this class, with one method overridden differently. The only question left is write-write conflicts, and it has exactly two answers, named by their attitude: - **pessimistic** — assume conflicts happen: detect at *access time* by - locking each key before buffering the write (the 2PL school — two-phase - locking: acquire locks as you go, release only at the end). -- **optimistic** — assume they don't: detect at *commit time* by checking - whether any buffered key was overwritten since your snapshot (the OCC - school — optimistic concurrency control, Kung & Robinson 1981: + locking each key before buffering the write. The **2PL** school + (two-phase locking: acquire locks in a growing phase, release only in a + shrinking phase; **strict 2PL** releases them all at commit). +- **optimistic** — assume they do not: detect at *commit time* by checking + whether any buffered key was overwritten since your snapshot. The **OCC** + school (optimistic concurrency control, Kung & Robinson 1981: read/validate/write phases). +Why it matters: this is the whole taxonomy. Everything in Steps 4 and 5 is +a consequence of *when* the check runs, not *what* it checks. + ### Step 4 — OCC: validate against the memtable, abort on doubt -At commit, the optimistic flavor asks, for each key in its write batch: -"has this key been written with a seq newer than my snapshot?" The trick -is *where* it asks: the **memtable only** (the LSM's in-RAM write buffer, -which holds the most recent writes). If the answer is there, great — -conflict or no conflict. If the memtable's earliest seq is newer than the -snapshot, the memtable is too young to remember the snapshot's era, and -RocksDB *can't know* — so it aborts conservatively with `TryAgain`: - -```rust -// CheckKey, conceptually: "was this key written after my snapshot?" -fn validate(&self, snap_seq: u64) -> Result<(), Abort> { - for key in self.write_batch.keys() { - if self.db.memtable_min_seq() > snap_seq { - return Err(Abort::TryAgain); // memtable too young to answer — - } // abort conservatively, retry - if self.db.latest_seq(key, /*memtable_only=*/ true) > snap_seq { - return Err(Abort::Busy); // someone committed over me - } - } - Ok(()) // batch → DB, atomically -} +> **In:** a committed-ready write batch and a snapshot seq. +> **Out:** the memtable-only validation, the two distinct failure statuses +> it can return, and the config knob that trades memory for abort rate. + +At commit the optimistic flavor asks, for each key it wrote: "has this key +been written with a seq newer than my snapshot?" The dispatch: + +```cpp +// utilities/transactions/optimistic_transaction.cc — Commit, 60-74 + 60 Status OptimisticTransaction::Commit() { + 64 switch (txn_db_impl->GetValidatePolicy()) { + 65 case OccValidationPolicy::kValidateParallel: + 66 return CommitWithParallelValidate(); + 67 case OccValidationPolicy::kValidateSerial: + 68 return CommitWithSerialValidate(); + 71 } +``` + +- **Serial** (`optimistic_transaction.cc:76`) hands a callback to + `WriteWithCallback` so validation runs inside RocksDB's single writer + queue — correct by serialization, at the price of holding that queue. +- **Parallel** (`optimistic_transaction.cc:93`) takes striped bucket + mutexes over the write set first, then validates, then writes. The + comment at `:122-124` names the discipline that keeps it safe: "in a + single txn, all bucket-locks are taken in ascending order. In this way, + txns from different threads all obey this rule so that deadlock can be + avoided." + +Same serialize-vs-stripe trade as topic 5's group commit. + +The validation itself goes `CheckTransactionForConflicts` +(`optimistic_transaction.cc:192`) → `TransactionUtil::CheckKeysForConflicts` +(`transaction_util.cc:154`) → `TransactionUtil::CheckKey` +(`transaction_util.cc:50`, called at `:188`), with `cache_only = true`: + +```cpp +// utilities/transactions/optimistic_transaction.cc — CheckTransactionForConflicts, 192-201 + 192 Status OptimisticTransaction::CheckTransactionForConflicts(DB* db) { + 195 // Since we are on the write thread and do not want to block other writers, + 196 // we will do a cache-only conflict check. This can result in TryAgain + 197 // getting returned if there is not sufficient memtable history to check + 198 // for conflicts. + 199 return TransactionUtil::CheckKeysForConflicts(db_impl, *tracked_locks_, + 200 true /* cache_only */); + 201 } +``` + +`cache_only` means **memtable only** — the LSM's in-RAM write buffer, which +holds the most recent writes. Never touch SSTs during validation; that is +the whole point. Which forces two different failure modes: + +```cpp +// utilities/transactions/transaction_util.cc — CheckKey, 67-104 and 130-147 (elided) + 67 // Since it would be too slow to check the SST files, we will only use + 68 // the memtables to check whether there have been any recent writes + 69 // to this key after it was accessed in this transaction. But if the + 70 // Memtables do not contain a long enough history, we must fail the + 71 // transaction. + 85 } else if (snap_seq < earliest_seq || min_uncommitted <= earliest_seq) { + 88 need_to_read_sst = true; + 90 if (cache_only) { + 91 // The age of this memtable is too new to use to check for recent + 92 // writes. + 104 result = Status::TryAgain(msg); + 130 } else if (found_record_for_key) { + 131 bool write_conflict = snap_checker == nullptr + 132 ? snap_seq < seq + 133 : !snap_checker->IsVisible(seq); + 145 if (write_conflict) { + 146 result = Status::Busy(); + 147 } ``` -Cheap validation — no disk reads, no lock table — bought with spurious -aborts on long transactions (outlive one memtable flush and every commit -is a `TryAgain`). Two commit modes exist: validate inside the single -writer queue (correct by serialization) or take striped locks on the write -set, validate, then write in parallel — the same serialize-vs-stripe trade -as topic 5's group commit. +Line 85 is "I cannot answer", line 132 is "the answer is yes". They return +**different statuses** and mean different things: + +- `Status::Busy()` (line 146) — a real conflict. Someone committed over + you. Retrying will probably lose again. +- `Status::TryAgain()` (line 104) — no information. The memtable was + flushed and rotated since your snapshot, so its history no longer covers + your era. Retrying immediately will very likely succeed. + +Work it on numbers. Take a transaction that took its snapshot at seq +**1000** and commits later: + +| Memtable `earliest_seq` | Key's latest seq | Line that fires | Result | +|---|---|---|---| +| 500 | none found | — | **OK** — no one touched the key | +| 500 | 1200 | 132: `1000 < 1200` | **`Busy`** — real write-write conflict | +| 500 | 900 | 132: `1000 < 900` false | **OK** — the write predates my snapshot | +| 5000 | *irrelevant* | 85: `1000 < 5000` | **`TryAgain`** — memtable too young | + +Row 4 is the one to remember: **the key was never touched by anybody, and +the commit still fails.** Any transaction that outlives one memtable +rotation gets `TryAgain` on every key, deterministically. The error message +at `transaction_util.cc:99-101` even names the fix — "Increasing the value +of the `max_write_buffer_size_to_maintain` option could reduce the +frequency of this error" — which is the actual trade: **memory retained for +flushed memtables, bought against spurious aborts.** + +Why it matters: OCC's validation is cheap because it refuses to do I/O, and +the price of that refusal is a false-abort rate that scales with +transaction *duration* rather than with contention. That is a different +failure axis from the one OCC is usually criticised on. ### Step 5 — 2PL: lock at access, hold to the end -The pessimistic flavor pays up front: every Put/Delete calls `TryLock` on -the key BEFORE buffering it, and `GetForUpdate` takes a read→write lock. -The locks live in a `PointLockManager` — a striped hash table (many -independently-latched buckets, so lock traffic itself doesn't serialize) -mapping key → `LockInfo`, with acquisition timeouts and **deadlock -detection** via a wait-for graph (T1 waits for T2 waits for T1 ⇒ cycle ⇒ -abort somebody; with locks-held-till-end, deadlock is possible and must be -detected, not just avoided). Locks release only after commit's write -lands — that's **strict 2PL**, which is what makes the commit order equal -the lock order. +> **In:** the same base class, with conflict detection moved to access time. +> **Out:** a striped lock table, the timeout that is the *default* defence, +> and the deadlock detector that is not on by default. + +The pessimistic flavor pays up front: every write locks the key **before** +buffering it. The ordering is explicit: + +```cpp +// utilities/transactions/pessimistic_transaction.cc — WriteCommittedTxn::Operate, 489-519 (elided) + 489 Status WriteCommittedTxn::Operate(ColumnFamilyHandle* column_family, + 490 const TKey& key, const bool do_validate, + 491 const bool assume_tracked, + 492 TOperation&& operation) { + 493 Status s; + 494 if constexpr (std::is_same_v) { + 495 s = TryLock(column_family, key, /*read_only=*/false, /*exclusive=*/true, + 496 do_validate, assume_tracked); + 503 if (!s.ok()) { + 504 return s; + 505 } + 519 return operation(); +``` + +Lock at line 495, buffer at line 519, and bail out in between if the lock +fails. Every `Put`, `Delete`, `Merge` and `SingleDelete` in +`pessimistic_transaction.cc` routes through this one function. +`PessimisticTransaction::TryLock` itself is at `:1151`, and `GetForUpdate` +(`:164` and `:172`, both forwarding to `GetForUpdateImpl` at `:182`) is the +read-side equivalent — it takes an exclusive lock by default +(`include/rocksdb/utilities/transaction.h:411`, `bool exclusive = true`). + +The locks live in a `PointLockManager` +(`lock/point/point_lock_manager.h:110`), which is a **striped** hash table: + +```cpp +// utilities/transactions/lock/point/point_lock_manager.cc — LockMap::GetStripe, 441-443 + 441 size_t LockMap::GetStripe(const std::string& key) const { + 443 return FastRange64(GetSliceNPHash64(key), num_stripes_); +``` + +Each stripe has its own mutex and condition variable, so ordinary lock +traffic does not serialize on one latch. The default is +`num_stripes = 16` (`include/rocksdb/utilities/transaction_db.h:171`) — a +number worth remembering, because it is small. Question 2 makes you work +the pathology. + +**Three defaults that change the story**, all from +`include/rocksdb/utilities/transaction_db.h`: + +| Option | Line | Default | Consequence | +|---|---|---|---| +| `transaction_lock_timeout` | 181 | **1000 ms** | a blocked lock gives up after 1 s and returns `TimedOut` | +| `deadlock_detect` | 304 | **false** | **deadlock detection is off unless you ask for it** | +| `deadlock_detect_depth` | 351 | 50 | how far the wait-for graph is walked when it *is* on | + +So the out-of-the-box defence against deadlock is **the timeout**, not the +detector. Turn detection on and `AcquireWithTimeout` +(`point_lock_manager.h:208`, definition around +`point_lock_manager.cc:630-640`) calls `IncrementWaiters` +(`point_lock_manager.cc:840`), which does a bounded breadth-first walk of +the wait-for graph: + +```cpp +// utilities/transactions/lock/point/point_lock_manager.cc — IncrementWaiters, 870-888 and 921-932 (elided) + 870 for (int tail = 0, head = 0; head < txn->GetDeadlockDetectDepth(); head++) { + 883 if (tail == head) { + 884 return false; // ran out of edges: no deadlock + 887 auto next = queue_values[head]; + 888 if (next == id) { // found a cycle back to me + 921 // Wait cycle too big, just assume deadlock. + 930 dlock_buffer_.AddNewPath(DeadlockPath(deadlock_time, true)); + 932 return true; +``` + +Read lines 921–932 carefully: if the search exhausts +`deadlock_detect_depth` without closing a cycle, RocksDB **declares a +deadlock anyway**. That is a deliberate false positive — a wait chain 51 +transactions long aborts somebody even when no cycle exists. Detected +cycles are recorded in a bounded ring (`DeadlockInfoBufferTempl`, +`point_lock_manager.h:31-101`) readable via `GetDeadlockInfoBuffer` +(`point_lock_manager.h:156`). -Note what's locked: **keys, not predicates** — a lock manager over an -order-preserving keyspace can't stop phantoms (no gap/range locks here; -contrast innodb). And neither flavor validates READ sets by default — -snapshot validation (`SetSnapshotOnNextOperation`) is layered on top for -repeatable reads; write skew is entirely possible (question 3). +Locks release only after commit's write lands (`Commit` — +`pessimistic_transaction.cc:681`); that is **strict 2PL**, which is what +makes commit order equal lock order. -### Step 6 — the design plane: when each school wins +Two limits to carry away: -Both flavors, one 2×2 — the cost just moves between the two columns: +- **Keys are locked, not predicates.** A lock manager over an + order-preserving keyspace with no gap or range locks cannot stop + phantoms — contrast InnoDB's next-key locks. +- **Neither flavor tracks a general read set.** Reads are only validated + for keys you explicitly `GetForUpdate` (which calls `ValidateSnapshot`, + `pessimistic_transaction.cc:1290`, which in turn calls + `TransactionUtil::CheckKeyForConflicts`, `transaction_util.cc:20`). A + plain `Get` is invisible to conflict detection, so write skew is entirely + possible — question 3, and see + [`reading-ssi-postgres.md`](reading-ssi-postgres.md) for what tracking + reads properly costs. + +Why it matters: "pessimistic transactions have deadlock detection" is the +kind of true-sounding claim that is false by default. Read the option +defaults before you reason about a system's failure modes. + +### Step 6 — the design plane: where each school pays + +> **In:** two implementations of one interface. +> **Out:** the 2×2 that explains why both exist, worked on one concrete +> pair of transactions. + +Both flavors, one 2×2 — the cost just moves between the columns: ``` conflict cost paid: at access time at commit time @@ -141,65 +382,281 @@ Both flavors, one 2×2 — the cost just moves between the two columns: contention ↑ ⇒ OCC abort rate ↑ (wasted work); 2PL queue depth ↑ (waits). ``` -Low contention: OCC wins — zero lock traffic, validation almost always -passes. High contention: OCC burns whole transactions per abort while 2PL -merely queues; wasted work vs waiting. That crossover is the entire -"which school" decision, and RocksDB exposing both behind one API is the -admission that no single answer exists. +Run the same pair of transactions through both. T1 and T2 each increment +key `k`, currently value 5 written at seq 900. Both take a snapshot at seq +**1000**. + +**Under OCC:** + +1. T1 reads k = 5, buffers `put(k, 6)`. T2 does exactly the same. Neither + has touched anything shared. +2. T1 commits. Validation: no key written after seq 1000 → OK. The batch + lands at seq **1001**. +3. T2 commits. `CheckKey(k, snap_seq = 1000)` → + `GetLatestSequenceForKey` returns 1001 → `transaction_util.cc:132`, + `1000 < 1001` → **`Status::Busy()`**. +4. T2 has done 100% of its work and keeps 0% of it. Its retry starts from + scratch. + +**Under 2PL:** + +1. T1 calls `Operate` → `TryLock(k)` at `pessimistic_transaction.cc:495` → + acquired. +2. T2 calls `Operate` → `TryLock(k)` → the stripe mutex for `k` is + contended; T2 blocks in `AcquireWithTimeout` for up to + `transaction_lock_timeout` = **1000 ms**. +3. T1 commits at `:681` and releases. T2 wakes, acquires, re-reads k = 6, + writes 7. +4. T2 kept 100% of its work and paid for it in wall-clock time. + +Same outcome, opposite currency. Low contention: OCC wins — zero lock +traffic and validation almost always passes. High contention: OCC burns +whole transactions per abort while 2PL merely queues. That crossover is the +entire "which school" decision, and RocksDB exposing both behind one API is +the admission that no single answer exists. + +Why it matters: notice a third case the 2×2 does not have a column for — +Step 4's `TryAgain`, which is an OCC abort caused by *neither* contention +*nor* the workload, only by elapsed time. Real systems fail in ways the +textbook taxonomy has no cell for. ## Where each step lives in the code -All under `utilities/transactions/`; ~1.5 h. - -- **Steps 1+3 — the skeleton** (`transaction_base.{h,cc}`): the private - `WriteBatchWithIndex` and read-through-batch logic; `SetSnapshot` — - transaction_base.h:264; lazy `snapshot_needed_` :270. -- **Step 4 — OCC** (`optimistic_transaction.{h,cc}`, - `transaction_util.cc`): `CheckTransactionForConflicts` (h:67) → - `TransactionUtil::CheckKeyForConflicts` (transaction_util.cc:20) → - `CheckKey` :50 — the memtable-only validation with the `TryAgain` - conservative abort. Commit modes: optimistic_transaction.cc:66; - `CommitWithSerialValidate` (h:76) vs `CommitWithParallelValidate` - (h:78). -- **Step 5 — 2PL** (`pessimistic_transaction.{h,cc}`, `lock/point/`): - `TryLock` — pessimistic_transaction.cc:1151, called before buffering - (:495 — lock first, then base-class write); `GetForUpdate` read→write - upgrade :1121. `PointLockManager` — lock/point/point_lock_manager.h:110: - striped hash of key → `LockInfo` (h:26), `AcquireWithTimeout` :208, - wait-for-graph deadlock detection (h:216) with a bounded deadlock-info - buffer (h:75–93). `Commit` :681 — locks released only after the write - lands: strict 2PL. +All anchors verified at `facebook/rocksdb@7c80a5a`; ~1.5 h. + +| Step | File | Lines | What | +|---|---|---|---| +| 1, 3 | `utilities/transactions/transaction_base.h` | 274-278, 455-459 | `WriteBatchWithIndex`, `indexing_enabled_`, `DisableIndexing` | +| 2 | `utilities/transactions/transaction_base.h` | 264-272, 463 | `SetSnapshot`, `SetSnapshotOnNextOperation`, `ClearSnapshot`, `snapshot_needed_` | +| 2, 5 | `include/rocksdb/utilities/transaction_db.h` | 171, 181, 299, 304, 325, 341, 351 | `num_stripes`, `transaction_lock_timeout`, `set_snapshot`, `deadlock_detect`, `lock_timeout`, `deadlock_timeout_us`, `deadlock_detect_depth` | +| 4 | `utilities/transactions/optimistic_transaction.cc` | 60-74, 76-91, 93-134, 192-201 | `Commit` dispatch; serial vs parallel validate; ascending bucket-lock order at 122-124; `CheckTransactionForConflicts` | +| 4 | `utilities/transactions/optimistic_transaction.h` | 67, 76, 78 | the three declarations | +| 4 | `utilities/transactions/transaction_util.cc` | 50-152, 154, 188 | `CheckKey` — the `TryAgain` branch at 85-104, the `Busy` branch at 130-147; `CheckKeysForConflicts` | +| 5 | `utilities/transactions/pessimistic_transaction.cc` | 164, 172, 182, 489-519, 681, 1151, 1290 | `GetForUpdate`, `Operate` (lock-then-buffer), `Commit`, `TryLock`, `ValidateSnapshot` | +| 5 | `utilities/transactions/transaction_util.cc` | 20 | `CheckKeyForConflicts` — the *pessimistic* read-validation entry point | +| 5 | `utilities/transactions/lock/point/point_lock_manager.h` | 26-28, 31-101, 110, 156, 169-198, 208, 218 | `LockInfo`/`LockMap`/`LockMapStripe`, deadlock ring buffer, `PointLockManager`, the mandated lock order, `AcquireWithTimeout`, `IncrementWaiters` | +| 5 | `utilities/transactions/lock/point/point_lock_manager.cc` | 195, 326-358, 441-443, 630-640, 840-932 | stripe struct, `LockMap`, `GetStripe`, the deadlock-detect call site, `IncrementWaiters` | +| 5 | `include/rocksdb/utilities/transaction.h` | 402-406, 408-412 | the documented status codes; `GetForUpdate`'s `exclusive = true` default | ## Questions for notes.md 1. Why can OCC validation use the memtable only? What property of LSM seq - numbers makes "not in memtable ⇒ too old to conflict... unless memtable - is too young" sound — and what does the TryAgain path cost a retry loop? -2. The pessimistic lock manager stripes by key hash. What's the pathology - for a graph workload where every txn touches the same super-node's - adjacency entries? -3. Neither flavor validates READ sets by default — so what isolation do - you actually get, and where does write skew sneak in? -4. FalkorDB angle: GRAPH.QUERY writes are single-threaded today (one - writer). If M8 keeps single-writer, which of these two machineries do - you still need? (Hint: none for w-w; what about r-w validation for - serializable reads?) + numbers makes "not in memtable ⇒ too old to conflict" sound — and what + exactly does `transaction_util.cc:85` add to that sentence? Then price + the `TryAgain` retry loop: if a transaction takes longer than one + memtable rotation, what is its steady-state commit rate? +2. The lock manager stripes by key hash into `num_stripes = 16` stripes + (`point_lock_manager.cc:441-443`, + `include/rocksdb/utilities/transaction_db.h:171`). Work the pathology for + a graph workload where every transaction touches the same super-node's + adjacency entries: how many of the 16 stripe mutexes carry the load, and + does raising `num_stripes` help? +3. Neither flavor tracks a general read set — only keys passed to + `GetForUpdate` are validated (`pessimistic_transaction.cc:1290`). So what + isolation level do you actually get, and construct the write skew that + sneaks through. Compare with `reading-ssi-postgres.md`. +4. `deadlock_detect` defaults to false + (`include/rocksdb/utilities/transaction_db.h:304`). What happens to a + genuine two-transaction deadlock under the defaults, how long does it + take, and which status does the loser get? Now read + `point_lock_manager.cc:921-932` — what does the detector do that the + timeout does not, and what false positive does it introduce? +5. FalkorDB angle: GRAPH.QUERY writes are single-threaded today (one + writer). If M8 keeps single-writer, which of these two machineries do you + still need? (Hint: none for write-write; what about read-write validation + for serializable reads?) + +## Takeaway + +One base class, one fork. Buffer the writes and snapshot with an integer, +then choose when to look for conflicts: at access time with a striped lock +table, a 1-second timeout and an optional bounded deadlock walk; or at +commit time with a memtable-only check that returns `Busy` for real +conflicts and `TryAgain` when it simply cannot see far enough back. The +first pays in waiting, the second in wasted work, and the second has a +third failure mode — elapsed time — that the textbook 2×2 has no cell for. + +## Connections to this topic's experiment + +The exercise in `experiments/src/mvcc.rs` is the OCC half of this guide. +`first_committer_wins_on_write_write_conflict` is Step 6's OCC trace with +`CommitError::WriteConflict` standing in for `Status::Busy()`, and +`Mode::Serializable` adds the read-set tracking that RocksDB deliberately +does not do (`CommitError::ReadConflict`, +`experiments/src/mvcc.rs:105`). + +The topic's *measured* lane is something else, and worth stating exactly. +It benchmarks a single global `Mutex` — 4 threads × 50 000 +transactions × 4 operations — and on an Apple M3 Pro (measured 2026-07-28, +recorded in [`notes.md`](notes.md)) it returns: + +| Workload | Keys | mutex txn/s | +|---|---|---| +| read-heavy 95/5 | 10 000 | 623 454 | +| write-heavy 50/50 | 10 000 | 594 264 | +| write-heavy 50/50 | 64 (hot) | 676 691 | + +Those numbers are **flat**: about 12% spread across workloads that differ +completely in read/write mix and key skew. That is the negative result +recorded in [`FINDINGS.md`](../../FINDINGS.md) row 8 — the global mutex had +already serialized everything, so nothing about the workload could reach +the measurement. Note especially that the *hot-key* row is the fastest one; +under any of the schemes in this guide, 64 hot keys is the case that hurts, +and the mutex does not notice, because it has no per-key structure to +contend on. + +This repo has **not** measured OCC or 2PL beating that mutex — the +`mvcc txn/s` and `aborts` columns in `notes.md` are `stub`. When you fill +them, the `aborts` column is the one that matters: Step 6 says OCC's cost +is wasted work, and a throughput number alone cannot tell you whether you +bought it. ## Done when -You can explain, with file:line, where each school pays its conflict cost, -and why both can share one write-buffering base class. +Answer each before unfolding it. + +- [ ] Explain, with file:line, where each school pays its conflict cost, + and why both can share one write-buffering base class. + +
Answer + +**2PL pays at access time**: `WriteCommittedTxn::Operate` +(`pessimistic_transaction.cc:489`) calls `TryLock` at line **495** and only +reaches the buffering call, `operation()`, at line **519**. **OCC pays at +commit time**: `OptimisticTransaction::Commit` +(`optimistic_transaction.cc:60`) dispatches to a validate-then-write path, +and the check is `CheckTransactionForConflicts` +(`optimistic_transaction.cc:192`) → `CheckKey` (`transaction_util.cc:50`). + +They share a base class because Step 1's write buffering makes conflict +detection *orthogonal* to atomicity: nothing a transaction does is visible +until the batch is applied, so it makes no difference to reads, rollback or +durability whether the conflict was caught at line 495 or at line 146 of +`transaction_util.cc`. + +
+ +- [ ] A transaction snapshots at seq 1000, writes one key nobody else + touches, and commits after a memtable flush leaves `earliest_seq = + 5000`. What happens, and why is it not a bug? + +
Answer + +It fails with **`Status::TryAgain`**, at `transaction_util.cc:85`: +`snap_seq (1000) < earliest_seq (5000)` → `need_to_read_sst = true`, and +since `cache_only` is true the error at line 104 is returned instead. It is +not a bug because OCC's validation *refuses to read SSTs* — the comment at +`:67-71` says checking SST files "would be too slow", so when the memtable +history no longer covers the snapshot's era, RocksDB cannot prove the +absence of a conflict and refuses conservatively. The message at +`:99-101` names the tuning knob: `max_write_buffer_size_to_maintain`, +trading memory for a lower spurious-abort rate. Note that `TryAgain` is a +*different status* from `Busy` (line 146) precisely so callers can tell +"retry me, I'll probably win" from "someone beat you". + +
+ +- [ ] Does RocksDB's pessimistic transaction detect deadlocks? Answer with + the option and its default. + +
Answer + +**Not by default.** `TransactionOptions::deadlock_detect = false` +(`include/rocksdb/utilities/transaction_db.h:304`). Out of the box, a real +deadlock is resolved by `transaction_lock_timeout` (line 181, default +**1000 ms**), and the loser gets `Status::TimedOut`. If you enable +detection, `AcquireWithTimeout` calls `IncrementWaiters` +(`point_lock_manager.cc:840`), a breadth-first walk of the wait-for graph +bounded by `deadlock_detect_depth` (line 351, default 50), and the loser +gets `Status::Busy` with `SubCode::kDeadlock` +(`point_lock_manager.cc:634`). The detector also has a deliberate false +positive: `point_lock_manager.cc:921-932`, "Wait cycle too big, just assume +deadlock" — a wait chain longer than the depth limit aborts a transaction +even with no cycle present. + +
+ +- [ ] The lock table has 16 stripes by default. What breaks when every + transaction in the workload touches the same key? + +
Answer + +`LockMap::GetStripe` (`point_lock_manager.cc:441-443`) is +`FastRange64(GetSliceNPHash64(key), num_stripes_)` — a pure function of the +*key*. One key hashes to exactly one stripe, so all lock traffic serializes +on that stripe's single mutex and the other 15 of 16 stripes +(`transaction_db.h:171`) sit idle. **Raising `num_stripes` does not help at +all**, because striping only spreads *distinct* keys; it is a fix for lock +table contention, not for key contention. The only fixes are workload-side: +shard the hot key, or stop taking a lock on it (which is what an +optimistic scheme does — and then it pays in aborts instead). + +
+ +- [ ] Trace T1 and T2 both incrementing key `k` (value 5 at seq 900, both + snapshotting at seq 1000) under each school. Which one throws work + away? + +
Answer + +**OCC**: both buffer `put(k, 6)` with no interaction. T1 commits at seq +1001. T2's `CheckKey` hits `transaction_util.cc:132` — `1000 < 1001` → +`Status::Busy()`. T2 discards everything it did and starts over. **2PL**: +T1's `TryLock` at `pessimistic_transaction.cc:495` succeeds; T2's blocks in +`AcquireWithTimeout` for up to 1000 ms; T1 commits at `:681` and releases; +T2 wakes, re-reads k = 6, writes 7 and keeps all of its work. **OCC throws +work away; 2PL spends wall-clock time.** Same correct outcome, opposite +currency — which is exactly why RocksDB ships both. + +
+ +- [ ] State this topic's measured result and what it does not show. + +
Answer + +A global `Mutex` baseline measures 623 454 / 594 264 / 676 691 +txn/s for read-heavy 10K-key, write-heavy 10K-key and write-heavy +64-hot-key workloads (Apple M3 Pro, 2026-07-28; `notes.md`). It is +**flat** — that is the negative finding in +[`FINDINGS.md`](../../FINDINGS.md) row 8: the mutex had already serialized +everything, so workload shape could not influence throughput. The hot-key +row being *fastest* is the tell. It does **not** show OCC or 2PL beating a +mutex; those columns are `stub`, and any such comparison would need the +`aborts` column too, since OCC's cost is wasted work rather than lower +throughput per completed transaction. + +
## References -**Code** -- [rocksdb](https://github.com/facebook/rocksdb) — - `utilities/transactions/`: `transaction_base.{h,cc}` (shared skeleton), - `optimistic_transaction.{h,cc}` + `transaction_util.cc` (OCC), - `pessimistic_transaction.{h,cc}` + `lock/point/point_lock_manager.h` - (2PL); ~1.5 h +**Code** — all anchors verified at +[`facebook/rocksdb@7c80a5a`](https://github.com/facebook/rocksdb) + +| File | Lines | What | +|---|---|---| +| `utilities/transactions/transaction_base.h` | 264-278, 455-463 | shared skeleton: snapshots, indexing | +| `utilities/transactions/optimistic_transaction.h` | 67, 76, 78 | OCC declarations | +| `utilities/transactions/optimistic_transaction.cc` | 60-134, 192-201 | commit dispatch, both validate modes, conflict check | +| `utilities/transactions/transaction_util.cc` | 20, 50-152, 154, 188 | `CheckKey` and its two entry points | +| `utilities/transactions/pessimistic_transaction.cc` | 164-200, 489-519, 681, 1151, 1290 | lock-then-buffer, commit, `TryLock`, `ValidateSnapshot` | +| `utilities/transactions/lock/point/point_lock_manager.h` | 26-101, 110-218 | lock table types, deadlock buffer, API | +| `utilities/transactions/lock/point/point_lock_manager.cc` | 195, 326-358, 441-443, 630-640, 840-932 | striping and deadlock detection | +| `include/rocksdb/utilities/transaction_db.h` | 171, 181, 296-351 | the defaults that decide behaviour | +| `include/rocksdb/utilities/transaction.h` | 402-412 | documented status codes | **Papers** -- Kung & Robinson — "On Optimistic Methods for Concurrency Control" + +- Kung & Robinson — *On Optimistic Methods for Concurrency Control* (TODS 1981) — the OCC school's founding paper (read/validate/write - phases); RocksDB's OptimisticTransaction is this, verbatim + phases); RocksDB's `OptimisticTransaction` is that structure, with the + validate phase restricted to the memtable. + +**In this repo** + +| Where | What | +|---|---| +| [`notes.md`](notes.md) | the measured mutex baseline and the `stub` columns | +| [`FINDINGS.md`](../../FINDINGS.md) row 8 | flat ~600k txn/s, because the mutex already serialized everything | +| `experiments/src/mvcc.rs:105` | `commit()` — where first-committer-wins and read validation go | +| [`reading-postgres-heapam.md`](reading-postgres-heapam.md) | visibility built by hand, instead of inherited from the storage layout | +| [`reading-ssi-postgres.md`](reading-ssi-postgres.md) | what it costs to track read sets properly | diff --git a/topics/08-transactions-mvcc/reading-ssi-postgres.md b/topics/08-transactions-mvcc/reading-ssi-postgres.md index a47e3fa..566be15 100644 --- a/topics/08-transactions-mvcc/reading-ssi-postgres.md +++ b/topics/08-transactions-mvcc/reading-ssi-postgres.md @@ -1,191 +1,1286 @@ # SSI: serializable snapshot isolation without blocking anyone -How postgres turned SI into SERIALIZABLE with passive markers instead of -blocking locks — Ports & Grittner's VLDB '12 account of productionizing -Cahill's dangerous-structure theorem. Before the paper, this chapter -builds the theory one edge at a time: the hole in SI, the -rw-antidependency, the theorem that reduces every anomaly to one shape, -and the engineering that made detecting that shape cheap enough to ship. -Prereq: the Berenson critique -([reading-ansi-critique.md](reading-ansi-critique.md)) — you need write -skew cold. +How postgres turned snapshot isolation into a real `SERIALIZABLE` level using +passive markers instead of blocking locks — Ports & Grittner's VLDB '12 account +of productionizing the dangerous-structure theorem. Before the paper, this +chapter builds the theory one edge at a time: the hole in SI, the +rw-antidependency, the theorem that reduces every anomaly to one shape, and the +engineering that made detecting that shape cheap enough to ship. Prereq: the +Berenson critique ([reading-ansi-critique.md](reading-ansi-critique.md)) — you +need write skew cold — and the tuple headers from +[reading-postgres-heapam.md](reading-postgres-heapam.md), because half of SSI's +conflict detection is just re-reading `xmin`/`xmax`. + +**Two attributions the literature routinely garbles, and this guide gets right +from the start.** The *theorem* — every serialization anomaly contains two +adjacent rw-antidependencies — is **Fekete et al.**, cited as [10] and stated as +Theorem 1 in §3.2. The *algorithm* that exploits it, SSI, is **Cahill, Röhm and +Fekete** (SIGMOD 2008), cited as [7] in §3.3. The *commit-ordering refinement* is +from **Cahill's thesis** [6], §3.3.1. Calling the whole bundle "Cahill's theorem" +(as the previous version of this guide did) collapses three separate results. + +Line numbers below are `postgres/postgres@701f021`, checked with +`python3 tools/pinned-source.py`. Section numbers are from the arXiv version of +the paper (arXiv:1208.4179). ## The problem in one sentence -Snapshot isolation permits write skew, and the classical fix — two-phase -locking — makes readers block writers again, throwing away SI's whole -value; SSI gets full serializability for ~7% overhead by *watching* for -one specific conflict shape and aborting somebody, blocking no one, ever. +Snapshot isolation permits write skew, and the classical fix — strict two-phase +locking — makes readers block writers again, throwing away everything MVCC +bought; SSI gets full serializability by *watching* for one specific conflict +shape and aborting somebody, blocking no one, ever, and the paper's own +measurements put the bill at **5% throughput on a CPU-bound TPC-C variant** +(§8.2, 25 warehouses in tmpfs), **10–20% CPU on the SIBENCH microbenchmark** +(§8.1), and **nothing measurable at all when the workload is disk-bound** (§8.2, +150 warehouses). + +> The previous version of this guide claimed "~7% overhead". That number is not +> in the paper. It has been replaced above by the three figures the paper +> actually reports, each with its section. ## The concepts, step by step -### Step 1 — the hole, restated as a target - -Under SI (snapshot isolation — every transaction reads a frozen snapshot, -and only write-write conflicts abort), two transactions can each read data -the other is about to change, write disjoint items, and both commit — -write skew, the doctors-on-call bug. The conflict is invisible to -first-committer-wins because it lives in the *read→write* crossings, not -in the write sets. - -So the engineering target is precise: detect harmful read→write crossings -between concurrent transactions, cheaply, without making reads take -blocking locks. Everything below is that one sentence, made rigorous and -then made fast. - -### Step 2 — the rw-antidependency: the edge that matters - -An **rw-antidependency** is the relationship "T read something, then a -concurrent U wrote it": T's snapshot didn't include U's write, so U has, -in effect, *un-read* T's view — if these two were serialized, T would have -to come BEFORE U for T's read to make sense. Draw it as an arrow -`T ──rw──► U`. - -Concrete: T1 reads bob's row (`r1[bob]`), concurrent T2 later writes it -(`w2[bob]`) ⇒ `T1 ──rw──► T2`. Note the asymmetry with ordinary conflicts: -nobody waited, nobody failed — the edge is just a *fact about the -interleaving*, recordable at the moment the write happens if someone -remembered the read. These edges are the raw material; one edge alone is -harmless. - -### Step 3 — the dangerous structure: Cahill's theorem - -Cahill's theorem (SIGMOD '08): every non-serializable SI execution -contains **two consecutive rw-antidependencies** — a transaction with one -inbound AND one outbound rw edge, called the **pivot** — with the further -condition that the downstream transaction commits first: - -``` - rw rw - T_in ────► T_pivot ────► T_out rw edge: T reads x, then U writes x - (U "un-reads" T's snapshot) - … and T_out commits FIRST of the three. -``` - -So you don't need to build the full serialization graph and search it for -cycles (the textbook-correct but expensive answer). Track rw edges; when -any transaction accumulates both directions — it became a pivot — abort -somebody. The whole detector is two flags per transaction and one rule: - -```rust -fn on_rw_antidependency(reader: TxnId, writer: TxnId, g: &mut ConflictGraph) { - g[reader].out_rw = true; // reader ──rw──► writer - g[writer].in_rw = true; - for t in [reader, writer] { - if g[t].in_rw && g[t].out_rw { // t became a pivot: - abort_someone(t, g); // T_in ─rw─► t ─rw─► T_out - } // conservative: false positives yes, - } // missed cycles never -} -``` - -This is **conservative**: some aborted histories were actually fine (a -pivot doesn't guarantee a full cycle) — but it never misses a real one. -Check it against write skew: T1 reads bob's row (later written by T2) ⇒ -`T1 ──rw──► T2`; T2 reads alice's row (later written by T1) ⇒ -`T2 ──rw──► T1`. A cycle of length 2 — *each* transaction is a pivot, and -the detector fires. Why it matters: false positives cost a retry; -false negatives would cost correctness. SSI buys certainty with spare -aborts. - -### Step 4 — SIREAD locks: remembering reads without blocking anyone - -To record rw edges you must remember who read what — that's the **SIREAD -lock**, which despite the name blocks nothing: it is a passive marker -"transaction T read this", checked by writers to generate edges. Three -engineering moves make the memory bill payable: - -- **Granularity with escalation**: markers exist at tuple, page, and - relation level; under memory pressure, many fine markers collapse into - one coarse one. Coarser = more false rw edges = more false aborts — - never wrong results. (Correctness is one-directional here; only - performance degrades.) -- **Predicate reads**: "reads" include rows that *would have* matched — the - phantom problem from the Berenson chapter. SSI handles it by marking the - index RANGE the query scanned (via index pages), so an insert into that - range generates the edge. This is the answer to phantoms that key-based - OCC (RocksDB Q3) cannot give. -- **Outliving commit**: an rw edge can form AFTER the reader commits (a - later writer hits its marker), so SIREAD state must survive commit and - is only cleaned when all overlapping transactions end — question 1 makes - you construct the history that requires this. - -### Step 5 — the paper's refinements on raw Cahill - -Ports & Grittner's production additions (§4–§7): - -1. **Commit ordering refinement** — only abort when T_out actually - committed first of the three (the theorem's full condition); raw - pivot-detection over-fires, this trims false positives. -2. **Safe snapshots & DEFERRABLE** — a read-only transaction has no writes, - hence can never have an *inbound* rw edge, hence can never be a pivot; - if it can additionally prove no concurrent writer endangers it, it - drops ALL tracking. `BEGIN ... DEFERRABLE` waits for such a safe - snapshot: long read-only backups then run at serializable for *zero* - overhead. -3. **Memory bounding** — §7's summarization: when tracking state hits its - RAM budget, collapse old transactions' state into a summary - (conservative again — more false aborts, bounded memory). -4. **Two-phase commit interaction** — prepared transactions can linger - indefinitely between prepare and commit, wedging cleanup; §7 explains - why they make everything worse. - -### Step 6 — the price tag, honestly (§8) - -~7% throughput overhead at low contention on their benchmarks — the -markers and edge checks are cheap. The real cost is **aborts**, and it is -workload-shaped: hot read-write pairs create pivot storms. And the -contract shifts to the application: a serialization failure (SQLSTATE -40001) means "run the whole transaction again" — SSI only delivers -serializable semantics if the app retries in a loop. No retry loop, no -serializability; just errors. +### Step 1 — the vocabulary, nailed down before anything moves + +> **In:** the words this paper uses without defining, because its audience was +> the 2012 VLDB program committee. **Out:** a definition for each, so that +> Step 2 onward can be read literally. + +- **Transaction** — a group of reads and writes the database promises to treat + as one unit: all of it takes effect, or none of it. +- **Snapshot** — a frozen view of the database as of one instant. Under + snapshot isolation every read a transaction makes is answered from its + snapshot, so the same query twice returns the same answer even if the world + moved on. §2.1: "as though the transaction operates on a private snapshot of + the database taken before its first read." +- **MVCC** (multi-version concurrency control) — the mechanism that makes + snapshots possible: writing a row leaves the old copy in place and adds a new + one, so different transactions can be shown different versions. Postgres + replaced its original lock-based storage manager with MVCC in 1999 (§3). +- **Tuple** — postgres's word for one row *version*. See + [reading-postgres-heapam.md](reading-postgres-heapam.md). +- **`xmin` / `xmax`** — the transaction ids stamped into a tuple header saying + which transaction created it and which deleted it. A visibility check is a + function of these two plus your snapshot. +- **Serializable** — the execution produces the same result as *some* serial + order of the same transactions, one after another with no overlap. This is + the property applications actually want, because it is the one that lets you + reason about a transaction on its own. +- **Write skew** — two concurrent transactions each read data the other is + about to change, then write *disjoint* rows. No write-write conflict fires, + both commit, and an invariant spanning both rows is broken. Berenson's + A5B; see [reading-ansi-critique.md](reading-ansi-critique.md). +- **rw-antidependency** (the paper also says *rw-conflict*) — §3.1: "if T1 + writes a version of some object, and T2 reads the previous version of that + object, then T1 appears to have executed *after* T2." Written `T2 --rw--> T1`, + the arrow pointing the way the serial order would have to run. +- **Dangerous structure** — two rw-antidependencies end to end: + `Tin --rw--> Tpivot --rw--> Tout`. The middle transaction is the **pivot**. +- **SSI** — serializable snapshot isolation: run under SI, watch for dangerous + structures, abort somebody when one appears. +- **SIREAD lock** — the marker recording "this transaction read this object". + §3.3: these locks "do not block conflicting writes (thus, 'lock' is somewhat + of a misnomer)." +- **Garbage collection / vacuum** — in postgres, the background reclamation of + dead tuples. SSI has its own parallel problem (Step 12): reclaiming the + *tracking state*, which is not the same thing and has its own rules. + +Why it matters: two of these — pivot and rw-antidependency — carry the entire +argument, and both are directional. Getting an arrow backwards silently inverts +which transaction the system decides to kill. + +### Step 2 — the hole, worked on one concrete schedule + +> **In:** a `doctors` table with two rows and the invariant "at least one doctor +> is on call". **Out:** an interleaving where both transactions commit and the +> invariant is false, with the exact reads and writes named. + +This is the paper's Figure 1 (§2.1.1), which is itself based on Cahill et +al. [7]. Both transactions run the same procedure: count the doctors currently +on call; if that count is at least 2, take yourself off call. + +Initial state, and the two transactions' snapshots: + +``` +doctors T1 (xid 100) T2 (xid 101) + Alice on_call = true begin, snapshot S1 begin, snapshot S2 + Bob on_call = true S1 sees Alice=t, Bob=t S2 sees Alice=t, Bob=t + + t0 T1: x <- SELECT count(*) FROM doctors WHERE on_call -> x = 2 + t1 T2: x <- SELECT count(*) FROM doctors WHERE on_call -> x = 2 + t2 T1: 2 >= 2, so UPDATE doctors SET on_call=f WHERE name='Alice' + t3 T2: 2 >= 2, so UPDATE doctors SET on_call=f WHERE name='Bob' + t4 T1: COMMIT -> ok + t5 T2: COMMIT -> ok under plain SI + +final state: Alice on_call = false, Bob on_call = false. Nobody is on call. +``` + +Run either transaction alone and the invariant holds. Run them in either serial +order and it holds: the second one counts 1, fails the `>= 2` test and writes +nothing. Only the interleaving breaks it. + +Why SI's own defences all miss it (§2.1.1): postgres's write locks are +*tuple*-level, and the two transactions update **different tuples** — Alice's +row and Bob's row — so nothing collides. First-committer-wins compares write +sets; these write sets are disjoint. The damage lives entirely in the +*read → write* crossings, which SI does not look at. + +The paper's own contrast, worth memorising because it is the whole design +space in two sentences (§2.1.1): "in two-phase locking DBMS, each transaction +would take read locks that would conflict with the other's write. Similarly, in +an optimistic-concurrency system, the second transaction would fail to commit +because its read set is no longer up to date." + +Why it matters: those two alternatives are exactly the two things this repo's +`experiments/src/mvcc.rs` makes you build (`Mode::Snapshot` reproduces the bug, +`Mode::Serializable` validates the read set). SSI is a *third* answer, and Step +5 is where it diverges from both. + +### Step 3 — the serialization graph and its three kinds of edge + +> **In:** an execution history. **Out:** a directed graph whose cycles are +> exactly the non-serializable executions. + +From Adya et al. [2], via §3.1. One node per transaction; an edge `T1 -> T2` +means T1 must precede T2 in any serial order equivalent to what happened. +Three ways to earn one: + +| edge | when | what it implies about time | +|---|---|---| +| **wr-dependency** | T1 writes a version, T2 *reads that version* | T1 committed **before** T2's snapshot (§3.2) | +| **ww-dependency** | T1 writes a version, T2 replaces it | T1 committed before T2 wrote — enforced by write locking (§3.2) | +| **rw-antidependency** | T1 writes a version, T2 read the version **before** it | T1 appears to run *after* T2 — and the two **overlapped** (§3.2) | + +The last row is the load-bearing one and its asymmetry is the point. wr and ww +edges can only run from a *finished* transaction to a later one. An rw edge is +the only kind that can exist between two transactions that were **concurrent** — +"one must start while the other was active" (§3.2). Note also, from §3.1, that +"objects" is deliberately more abstract than "tuple": if T1 scans for all rows +with `x = 1` and T2 then inserts a matching row, that is a `T1 --rw--> T2` edge +even though no existing row was touched. Phantoms are rw edges too. + +Applying this to Step 2 (the paper does it for you, §3.1, Figure 3a): T1's +update of Alice is invisible to T2's `SELECT`, so T2 appears to run first — +`T2 --rw--> T1`. T2's update of Bob is invisible to T1's `SELECT`, so +`T1 --rw--> T2`. That is a cycle of length two, and a cycle means the execution +matches no serial order at all: + +``` + rw + T1 -------------> T2 + ^ | + | rw | + +------------------+ + + T1 --rw--> T2 : T1 read Bob (old version), T2 wrote Bob + T2 --rw--> T1 : T2 read Alice (old version), T1 wrote Alice +``` + +Why it matters: cycle detection is the textbook-correct answer and it is +expensive — you must materialise the graph and search it. Step 4 is the result +that lets you skip that. + +### Step 4 — Theorem 1: every anomaly contains the same little shape + +> **In:** the observation that anomalies are cycles. **Out:** a *local* test — +> two adjacent edges — that no anomaly can evade. + +§3.2 quotes it verbatim: + +> **Theorem 1 (Fekete et al. [10]).** Every cycle in the serialization history +> graph contains a sequence of edges `T1 --rw--> T2 --rw--> T3` where each edge +> is a rw-antidependency. Furthermore, T3 must be the first transaction in the +> cycle to commit. + +Read the two halves separately, because the engineering treats them separately. + +1. **Two adjacent rw edges.** Adya [1] had already shown every cycle contains at + least two rw edges; Fekete et al. sharpened that to *adjacent*. Adjacency is + what makes the test local: you never look further than one transaction's own + two edge lists. +2. **T3 commits first — of the entire cycle.** §3.2 flags this as its own + contribution to the reading: "this is actually a stronger statement than that + given by Fekete et al., who state only that T3 must commit before T1 and T2. + Though not explicitly stated, it is a consequence of their proof that T3 must + be the first transaction in the entire cycle to commit." The strengthening is + not pedantry — it is what licenses the commit-ordering optimisation in Step 5 + and the "don't abort anything until T3 commits" rule in Step 10. + +And **Corollary 2**: T1 is concurrent with T2, and T2 is concurrent with T3, +because rw edges only occur between concurrent transactions. Plus the footnote +that matters for Step 2: "T1 and T3 may refer to the same transaction, for +cycles of length 2 such as the one in the write-skew example (Figure 3a)." + +Check Step 2 against it. The cycle is `T1 --rw--> T2 --rw--> T1`. Take +Tin = T1, Tpivot = T2, Tout = T1 (the same transaction, per the corollary). T1 +committed at t4 and T2 at t5, so Tout did commit first. The theorem holds, and +it says the anomaly is detectable by noticing that **T2 has an rw edge in and an +rw edge out**. + +Why it matters: the pivot is a property of one node. That is the difference +between an O(graph) search and two list walks. + +### Step 5 — SSI: check for the structure, not for the cycle + +> **In:** Theorem 1. **Out:** an algorithm, its false-positive rate, and the +> variant postgres declined to build. + +§3.3: SSI "checks for a 'dangerous structure' of two adjacent +rw-antidependency edges. If any transaction has both an incoming +rw-antidependency and an outgoing one, SSI aborts one of the transactions +involved." + +Three consequences the paper draws explicitly, all in §3.3: + +- **It is sound.** Theorem 1 guarantees no cycle can form without a dangerous + structure appearing first, so aborting on the structure cannot miss an + anomaly. +- **It is not complete.** "It may have false positives because not every + dangerous structure is part of a cycle." Some aborted executions were + perfectly serializable. +- **It only needs rw edges.** "Dangerous structures are composed entirely of + rw-antidependencies, so SSI does not need to track wr- and ww-dependency + edges." That halves the bookkeeping and, per §5.3, is also why the summarised + representation in Step 12 can be so small. + +The paper then makes a claim that is easy to skim past and is the best argument +in the whole section — SSI is *more* permissive than either classical school: + +> "Essentially, both S2PL and classic OCC prevent concurrent transactions from +> having rw-conflicts. SSI allows some rw-conflicts as long as they do not form +> a dangerous structure, a less restrictive requirement." (§3.3) + +with a worked instance: take the paper's three-transaction batch-processing +anomaly (§2.1.2, Figure 2) and delete the read-only `REPORT` transaction. What +remains is serializable in the order ⟨T2, T3⟩ despite containing a single rw +edge `T2 --rw--> T3`. "Neither S2PL nor OCC would permit this execution, whereas +SSI would allow it, because it contains only a single rw-antidependency." + +Two refinements, §3.3.1: + +- **Commit ordering** (Cahill's thesis [6]): since Theorem 1 requires T3 to + commit first, a dangerous structure where T1 or T2 committed before T3 is a + false positive and can be ignored. Postgres "use[s] an extension of this + optimization." It does not remove all false positives — there may simply be no + path `T3 ⇝ T1` closing the cycle. +- **PSSI** [18] removes all false positives by building the full graph and + testing for cycles, cutting the abort rate "by up to 40%" on a microbenchmark + built to stress false aborts. Postgres rejected it: it needs wr- and + ww-dependencies too, which costs memory, and §6's optimisations "would not be + compatible with PSSI." The clinching argument is measured, not aesthetic — + "the workloads we evaluate in Section 8 have a serialization failure rate well + under 1%, suggesting additional precision has a limited benefit." + +Why it matters: this is the paper's method in miniature. Every place it takes +a *conservative* shortcut, it argues the cost is more aborts and never a wrong +answer, and then measures how many more aborts. Correctness is one-directional; +performance is negotiable. + +### Step 6 — detecting an rw edge when the write happened first: no locks needed + +> **In:** a serializable transaction reading a heap tuple. **Out:** the branch +> of postgres that derives an rw edge straight from `xmin`/`xmax`, with no read +> tracking at all. + +§5.2 splits detection in two by chronology — "which one is needed depends on +whether the write happens chronologically before the read, or vice versa" — and +the write-first half is free, because the MVCC data already records it: + +> "If the tuple is not visible because the transaction that created it had not +> committed when the reader took its snapshot, that indicates a rw-conflict: the +> reader must appear before the writer in the serial order." (§5.2) + +That paragraph is this function: + +```c +// postgres/src/backend/access/heap/heapam.c — HeapCheckForSerializableConflictOut, 9182-9228 (elided) + 9182 void + 9183 HeapCheckForSerializableConflictOut(bool visible, Relation relation, + 9184 HeapTuple tuple, Buffer buffer, + 9185 Snapshot snapshot) + 9186 { + 9187 TransactionId xid; + 9188 HTSV_Result htsvResult; + 9189 + 9190 if (!CheckForSerializableConflictOutNeeded(relation, snapshot)) + 9191 return; + ... + 9204 htsvResult = HeapTupleSatisfiesVacuum(tuple, TransactionXmin, buffer); + 9205 switch (htsvResult) + 9206 { + 9207 case HEAPTUPLE_LIVE: + 9208 if (visible) + 9209 return; + 9210 xid = HeapTupleHeaderGetXmin(tuple->t_data); + 9211 break; + 9212 case HEAPTUPLE_RECENTLY_DEAD: + 9213 case HEAPTUPLE_DELETE_IN_PROGRESS: + 9214 if (visible) + 9215 xid = HeapTupleHeaderGetUpdateXid(tuple->t_data); + 9216 else + 9217 xid = HeapTupleHeaderGetXmin(tuple->t_data); + ... + 9226 case HEAPTUPLE_INSERT_IN_PROGRESS: + 9227 xid = HeapTupleHeaderGetXmin(tuple->t_data); + 9228 break; +``` + +Line **9208** is the whole idea. `visible` is the answer the ordinary visibility +check already produced; if the tuple is live *and* visible to you, there is no +edge and the function returns. Every other arm picks out the xid of the +transaction whose work you could not see — `xmin` when the tuple was created by +someone you cannot see (9210, 9217, 9227), the *updater's* xid when the tuple is +visible to you but somebody has since deleted it (9215) — and hands it to +`CheckForSerializableConflictOut` (`heapam.c:9263`), which lives in +`predicate.c:3952` and decides whether that xid belongs to a concurrent +serializable transaction: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — XidIsConcurrent, 3900-3917 + 3900 static bool + 3901 XidIsConcurrent(TransactionId xid) + 3902 { + 3903 Snapshot snap; + 3904 + 3905 Assert(TransactionIdIsValid(xid)); + 3906 Assert(!TransactionIdEquals(xid, GetTopTransactionIdIfAny())); + 3907 + 3908 snap = GetTransactionSnapshot(); + 3909 + 3910 if (TransactionIdPrecedes(xid, snap->xmin)) + 3911 return false; + 3912 + 3913 if (TransactionIdFollowsOrEquals(xid, snap->xmax)) + 3914 return true; + 3915 + 3916 return pg_lfind32(xid, snap->xip, snap->xcnt); + 3917 } +``` + +That is literally the snapshot test from +[reading-postgres-heapam.md](reading-postgres-heapam.md) — below `xmin` means +finished before you started, at or above `xmax` means started after you, and in +between the answer is a scan of the in-progress array `xip[]`. Same three lines, +different question: not "can I see it?" but "did we overlap?" + +Why it matters: half of SSI's conflict detection costs nothing, because postgres +was already computing the answer for visibility. The expensive half is Step 7. + +### Step 7 — detecting an rw edge when the read happened first: the SIREAD lock manager + +> **In:** the other chronology — you read a row, and only *later* does somebody +> write it. **Out:** postgres's predicate lock manager, its promotion rule, and +> the arithmetic of when a tuple marker becomes a table-wide one. + +Nothing in the MVCC data records that you read something, so this direction +needs the marker. §5.2: postgres could not reuse any existing lock mechanism — +it "did not previously acquire read locks on data accessed in any isolation +level," and its write locks live in *tuple headers on disk*, not in a lock +table, so there is nothing to match against. Hence a new manager that "stores +only SIREAD locks. It does not support any other lock modes, and hence cannot +block." + +Reads acquire markers on what they touched (`PredicateLockTID`, +`predicate.c:2550`, called from `heapam.c:1750`; `PredicateLockRelation`, `:2505`; +`PredicateLockPage`, `:2528`). Writers check for them. `heap_update` does it at +`heapam.c:3963`, `heap_delete` at `:2959`, and the insert paths at `:2054`, +`:2345`, `:2628`: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — CheckForSerializableConflictIn, 4265-4317 (elided) + 4265 CheckForSerializableConflictIn(Relation relation, const ItemPointerData *tid, BlockNumber blkno) + 4266 { + 4267 PREDICATELOCKTARGETTAG targettag; + ... + 4286 /* + 4287 * It is important that we check for locks from the finest granularity to + 4288 * the coarsest granularity, so that granularity promotion doesn't cause + 4289 * us to miss a lock. The new (coarser) lock will be acquired before the + 4290 * old (finer) locks are released. + ... + 4295 if (tid != NULL) + 4296 { + 4297 SET_PREDICATELOCKTARGETTAG_TUPLE(targettag, ...); + 4302 CheckTargetForConflictsIn(&targettag); + 4303 } + 4304 + 4305 if (blkno != InvalidBlockNumber) + 4306 { + 4307 SET_PREDICATELOCKTARGETTAG_PAGE(targettag, ...); + 4311 CheckTargetForConflictsIn(&targettag); + 4312 } + 4313 + 4314 SET_PREDICATELOCKTARGETTAG_RELATION(targettag, ...); + 4317 CheckTargetForConflictsIn(&targettag); + 4318 } +``` + +**A divergence between the paper and the code, worth noticing.** §5.2.1 says +these checks "must be done in the proper order: coarsest to finest." The comment +at `predicate.c:4287-4290` says the opposite — "from the finest granularity to +the coarsest" — and the code at `:4295`, `:4305`, `:4314` runs tuple, then page, +then relation. The code at `701f021` is what ships; take its ordering as +authoritative and the paper's sentence as thirteen years stale (or a typo). +Either way the *reason* is the same in both: promotion must never open a window +in which a marker is invisible to a checker. + +Three engineering moves make the memory bill payable, and only the first two are +usually quoted: + +1. **Index-range locks for predicate reads** (§5.2.1). Real predicate locking + [9] is not used; instead "index access methods acquire SIREAD locks on the + 'gaps' to detect phantoms. Currently, locks on B+-tree indexes are acquired + at page granularity; we intend to refine this to next-key locking [16] in a + future release." An insert into a range you scanned lands on a page you + marked, and the edge appears. +2. **Granularity promotion.** Markers exist at tuple, page and relation level, + and many fine ones collapse into one coarse one under pressure. +3. **No intention locks** (§5.2.1). "One simplification we were able to make is + that intention locks were not necessary, despite the use of multigranularity + locking (and contrary to a suggestion that intention-SIREAD locks would be + required [7])." Since SIREAD locks cannot block, deadlock detection is + unnecessary too, and the acquisition calls need not be placed away from held + buffer latches. + +**Promotion, worked on the real defaults.** The rule: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — MaxPredicateChildLocks and the promotion test, 2217-2229 + 2284-2295 (elided) + 2217 static int + 2218 MaxPredicateChildLocks(const PREDICATELOCKTARGETTAG *tag) + 2219 { + 2220 switch (GET_PREDICATELOCKTARGETTAG_TYPE(*tag)) + 2221 { + 2222 case PREDLOCKTAG_RELATION: + 2223 return max_predicate_locks_per_relation < 0 + 2224 ? (max_predicate_locks_per_xact + 2225 / (-max_predicate_locks_per_relation)) - 1 + 2226 : max_predicate_locks_per_relation; + 2227 + 2228 case PREDLOCKTAG_PAGE: + 2229 return max_predicate_locks_per_page; + ... + 2284 if (parentlock->childLocks > + 2285 MaxPredicateChildLocks(&targettag)) + 2286 { + ... + 2293 promotiontag = targettag; + 2294 promote = true; + 2295 } +``` + +The defaults are in `postgresql.conf.sample`: `max_pred_locks_per_transaction = +64` (`:877`), `max_pred_locks_per_relation = -2` (`:879`), +`max_pred_locks_per_page = 2` (`:882`). Substituting: + +``` +page threshold = max_pred_locks_per_page = 2 +relation threshold = 64 / 2 - 1 = 31 (negative form, :2223-2226) + +so, reading rows one at a time in a serializable transaction: + + tuple marker #1 in page 7 childLocks(page 7) = 1 1 > 2 ? no + tuple marker #2 in page 7 childLocks(page 7) = 2 2 > 2 ? no + tuple marker #3 in page 7 childLocks(page 7) = 3 3 > 2 ? YES -> one page marker replaces three tuple markers + + ... and the relation's child count counts BOTH tuples and pages (comment, :2203-2204): + + descendant marker #31 in table t 31 > 31 ? no + descendant marker #32 in table t 32 > 31 ? YES -> one relation marker replaces all of them +``` + +So a serializable scan that touches 32 rows spread thinly across a table ends up +holding a single **relation**-level SIREAD lock. From that moment, *every* write +by *any* concurrent transaction to that table generates an rw edge against you, +whether or not it touched a row you read. That is the trade in one image: coarser +markers ⇒ false edges ⇒ false aborts ⇒ never a wrong answer. + +Why 32 and not 32 000? Because the whole lock table is sized once at startup: +`NPREDICATELOCKTARGETENTS()` is `max_predicate_locks_per_xact × +(MaxBackends + max_prepared_xacts)` (`predicate.c:263-264`), and §6 explains the +constraint behind it — postgres puts all shared memory in one System V segment, +whose default OS limit the paper gives as 32 MB on Linux. Promotion is not an +optimisation here; it is the thing that stops the table filling up. + +Why it matters: this is the only part of SSI that is *pure* cost. Everything in +Step 6 was already being computed; the marker table is new memory, new +contention on lightweight locks, and the source of the 10–20% in §8.1. + +### Step 8 — remembering the edges: lists, not bits + +> **In:** an rw edge, just detected by Step 6 or Step 7. **Out:** where postgres +> puts it, and why it stores more than the original algorithm did. + +§5.3 lays out the design space as a spectrum of how much you remember: + +| implementation | per-transaction state | source | +|---|---|---| +| original SSI [7] | two single bits: has-in-edge, has-out-edge | §5.3 | +| Cahill's thesis [6] | two *pointers*, self-pointer meaning "more than one" | §5.3 | +| PSSI [18] | the entire graph, including wr and ww edges | §5.3 | +| **PostgreSQL** | **a list of all rw edges in, and all rw edges out** | §5.3 | + +Postgres's reason is not generosity: "keeping pointers to the other transaction +involved in the rw-antidependency, rather than a simple flag, is necessary to +implement the commit ordering optimization described in Section 3.3 and the +read-only optimization of Section 4.1" (§5.3). You cannot ask "did T3 commit +first?" of a bit. + +The struct: + +```c +// postgres/src/include/storage/predicate_internals.h — SERIALIZABLEXACT, 78-119 (elided) + 78 SerCommitSeqNo commitSeqNo; + 79 + 80 /* these values are not both interesting at the same time */ + 81 union + 82 { + 83 SerCommitSeqNo earliestOutConflictCommit; /* when committed with + 84 * conflict out */ + 85 SerCommitSeqNo lastCommitBeforeSnapshot; /* when not committed or + 86 * no conflict out */ + 87 } SeqNo; + 88 dlist_head outConflicts; /* list of write transactions whose data we + 89 * couldn't read. */ + 90 dlist_head inConflicts; /* list of read transactions which couldn't + 91 * see our write. */ + ... + 108 dlist_head possibleUnsafeConflicts; + ... + 115 TransactionId xmin; /* the transaction's snapshot xmin */ + 116 uint32 flags; /* OR'd combination of values defined below */ +``` + +Read the comments on 88-91 as the definition of the arrow's direction, because +they are the least ambiguous statement of it anywhere: **out** = "write +transactions whose data we couldn't read"; **in** = "read transactions which +couldn't see our write". `earliestOutConflictCommit` at :83 is §6.1's trick — +the one extra number that lets a committed transaction be cleaned up without +losing the ability to answer "does this one have an out-conflict, and when did +it commit?" + +Recording an edge is one function: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — SetRWConflict, 656-677 (elided) + 656 static void + 657 SetRWConflict(SERIALIZABLEXACT *reader, SERIALIZABLEXACT *writer) + 658 { + 659 RWConflict conflict; + 660 + 661 Assert(reader != writer); + 662 Assert(!RWConflictExists(reader, writer)); + 663 + 664 if (dlist_is_empty(&RWConflictPool->availableList)) + 665 ereport(ERROR, + 666 (errcode(ERRCODE_OUT_OF_MEMORY), + ... + 673 conflict->sxactOut = reader; + 674 conflict->sxactIn = writer; + 675 dlist_push_tail(&reader->outConflicts, &conflict->outLink); + 676 dlist_push_tail(&writer->inConflicts, &conflict->inLink); + 677 } +``` + +Lines 673-676: the reader gets the out-edge, the writer gets the in-edge, and +the same `RWConflictData` node is threaded onto both lists. Note :664-668 — the +pool is fixed-size and exhausting it is an *error*, not a fallback. That is the +memory pressure Step 12 exists to relieve. + +Why it matters: `inConflicts` and `outConflicts` are the two lists the pivot +test walks. Every later step is a traversal of these. + +### Step 9 — the pivot test, in the three shapes it can arrive in + +> **In:** an about-to-be-recorded edge `reader --rw--> writer`. **Out:** the +> decision "does adding this edge complete a dangerous structure?", and the +> three distinct cases the code checks. + +Every edge goes through `FlagRWConflict` (`predicate.c:4430`), and its **first** +act is the check — before the edge is even recorded: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — FlagRWConflict, 4429-4444 + 4429 static void + 4430 FlagRWConflict(SERIALIZABLEXACT *reader, SERIALIZABLEXACT *writer) + 4431 { + 4432 Assert(reader != writer); + 4433 + 4434 /* First, see if this conflict causes failure. */ + 4435 OnConflict_CheckForSerializationFailure(reader, writer); + 4436 + 4437 /* Actually do the conflict flagging. */ + 4438 if (reader == OldCommittedSxact) + 4439 writer->flags |= SXACT_FLAG_SUMMARY_CONFLICT_IN; + 4440 else if (writer == OldCommittedSxact) + 4441 reader->flags |= SXACT_FLAG_SUMMARY_CONFLICT_OUT; + 4442 else + 4443 SetRWConflict(reader, writer); + 4444 } +``` + +Lines 4438-4441 are Step 12's summarisation showing through: if one end has +already been collapsed into the shared `OldCommittedSxact` placeholder, the edge +degrades to a single "has a summarised conflict" bit. + +The check itself opens with the theorem, drawn in the comment: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — OnConflict_CheckForSerializationFailure header, 4446-4466 + 4446 /*---------------------------------------------------------------------------- + 4447 * We are about to add a RW-edge to the dependency graph - check that we don't + 4448 * introduce a dangerous structure by doing so, and abort one of the + 4449 * transactions if so. + ... + 4454 * Tin ------> Tpivot ------> Tout + 4455 * rw rw + 4456 * + 4457 * Furthermore, Tout must commit first. + 4458 * + 4459 * One more optimization is that if Tin is declared READ ONLY (or commits + 4460 * without writing), we can only have a problem if Tout committed before Tin + 4461 * acquired its snapshot. + 4462 *---------------------------------------------------------------------------- + 4463 */ + 4464 static void + 4465 OnConflict_CheckForSerializationFailure(const SERIALIZABLEXACT *reader, + 4466 SERIALIZABLEXACT *writer) +``` + +:4457 is Theorem 1's second half and :4459-4461 is Theorem 3 from Step 11 — +both encoded as comments above the function that enforces them. The body then +asks three separate questions, because the new edge can complete the structure +in three different positions: + +| case | shape | lines | the condition | +|---|---|---|---| +| 1 | `R --rw--> W --rw--> T2`, W already committed | `:4485-4487` | writer is committed **and** already has an out-conflict — so the structure is complete and (since the writer committed) we must be the reader | +| 2 | `R --rw--> W --rw--> T2`, the writer just became the pivot | `:4508-4532` | walk `writer->outConflicts`; fail if some T2 is prepared and neither the reader nor the writer committed before it, and (if the reader is read-only) T2 prepared before the reader's snapshot | +| 3 | `T0 --rw--> R --rw--> W`, the **reader** just became the pivot | `:4547-4578` | writer is prepared and reader is not read-only; walk `reader->inConflicts` for a T0 that is not doomed and did not commit before the writer prepared | + +Case 3 is the one people forget: adding an edge makes *two* nodes gain an edge, +so both ends have to be re-examined. Note that "prepared" here is broader than +two-phase commit — `predicate.c:268-271`: "a sxact is marked 'prepared' once it +has passed `PreCommit_CheckForSerializationFailure`, even if it isn't using 2PC. +This is the point at which it can no longer be aborted." + +Why it matters: every clause in cases 2 and 3 that mentions a commit sequence +number is Cahill's commit-ordering optimisation from Step 5, spending memory +(the lists of Step 8) to avoid a false abort. + +### Step 10 — who dies: safe retry, worked on the doctors + +> **In:** a detected dangerous structure. **Out:** the specific transaction +> postgres aborts, the rule that chooses it, and — traced through the real +> functions — the answer for Step 2's schedule. + +§5.4 states the property that decides the choice: + +> **Safe retry:** if a transaction is aborted, immediately retrying the same +> transaction will not cause it to fail again with the same serialization +> failure. + +and derives three rules from it, for the structure `T1 --rw--> T2 --rw--> T3`: + +1. **Do not abort anything until T3 commits.** Needed for the commit-ordering + optimisation, and it also serves safe retry. +2. **Always abort T2 if possible** — the pivot. "T2 must have been concurrent + with both T1 and T3. Because T3 is already committed, the retried T2 will not + be concurrent with it and so will not be able to have a rw-conflict out to + it." Aborting T1 instead would leave it still concurrent with T2, so the same + structure could form again. +3. **If both T2 and T3 have committed, abort T1** — safe, because the retried T1 + is concurrent with neither. + +Rule 1 has a consequence: a structure may be detected and left standing. So +there is a second check at commit time: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — PreCommit_CheckForSerializationFailure, 4659-4696 (elided) + 4659 dlist_foreach(near_iter, &MySerializableXact->inConflicts) + 4660 { + 4661 RWConflict nearConflict = + 4662 dlist_container(RWConflictData, inLink, near_iter.cur); + 4663 + 4664 if (!SxactIsCommitted(nearConflict->sxactOut) + 4665 && !SxactIsDoomed(nearConflict->sxactOut)) + 4666 { + 4667 dlist_iter far_iter; + 4668 + 4669 dlist_foreach(far_iter, &nearConflict->sxactOut->inConflicts) + 4670 { + 4671 RWConflict farConflict = + 4672 dlist_container(RWConflictData, inLink, far_iter.cur); + 4673 + 4674 if (farConflict->sxactOut == MySerializableXact + 4675 || (!SxactIsCommitted(farConflict->sxactOut) + 4676 && !SxactIsReadOnly(farConflict->sxactOut) + 4677 && !SxactIsDoomed(farConflict->sxactOut))) + 4678 { + ... + 4694 nearConflict->sxactOut->flags |= SXACT_FLAG_DOOMED; + 4695 break; +``` + +This is a two-level walk: my in-edges (:4659) give me candidate pivots; each +pivot's own in-edges (:4669) give me the Tin. Line **4694** is rule 2 — the +pivot gets `SXACT_FLAG_DOOMED`. The comment at :4625-4629 gives the reason in +progress terms: "This transaction is committing writes, so letting it commit +ensures progress. If we canceled the far conflict, it might immediately fail +again on retry." + +**The doctors, traced.** Take the schedule from Step 2 and follow it through +these functions. T1 = xid 100, T2 = xid 101. + +``` +t0 T1 SELECTs -> SIREAD markers for T1 on the rows/pages it scanned +t1 T2 SELECTs -> SIREAD markers for T2 on the same objects + +t2 T1 UPDATEs Alice + heapam.c:3963 -> CheckForSerializableConflictIn(rel, &oldtup.t_self, blkno) + predicate.c:4302/4311/4317 find T2's SIREAD marker + FlagRWConflict(reader = T2, writer = T1), predicate.c:4430 + OnConflict case 1 (:4485) T1 committed? no -> no failure + OnConflict case 2 (:4514) walk T1->outConflicts empty -> no failure + OnConflict case 3 (:4547) T1 prepared? no -> no failure + SetRWConflict: record T2 --rw--> T1 + +t3 T2 UPDATEs Bob + same path; finds T1's marker + FlagRWConflict(reader = T1, writer = T2) + case 2 walks T2->outConflicts = { T1 }, but T1 is not prepared yet -> no failure + case 3 needs T2 prepared; it is not -> no failure + SetRWConflict: record T1 --rw--> T2 <- the cycle now exists, undetected + +t4 T1 COMMITs -> PreCommit_CheckForSerializationFailure (predicate.c:4632) + :4648 am I already doomed? no + :4659 walk T1->inConflicts -> nearConflict->sxactOut = T2 + :4664 T2 committed or doomed? no + :4669 walk T2->inConflicts -> farConflict->sxactOut = T1 + :4674 farConflict->sxactOut == MySerializableXact ? YES + :4685 is T2 prepared? no + :4694 T2->flags |= SXACT_FLAG_DOOMED + T1 commits. + +t5 T2 COMMITs -> PreCommit_CheckForSerializationFailure + :4648 SxactIsDoomed(T2) -> ERROR 40001, + "Canceled on identification as a pivot, during commit attempt." + +RESULT: T1 commits, T2 aborts. Alice off call, Bob still on call. + Equivalent to the serial order : T2 re-run now counts 1, + fails its 2 >= 2 test, and writes nothing. Invariant holds. +``` + +Map that back onto the theorem. The structure is `T1 --rw--> T2 --rw--> T1`, so +Tin = T1, Tpivot = T2, Tout = T1 (Corollary 2's length-2 case). Tout committed +first, satisfying rule 1. Rule 2 says abort the pivot: T2. And safe retry is +real, not theoretical — on retry T2 is no longer concurrent with T1, so the +edge that killed it cannot re-form. + +Why it matters: "SSI aborts somebody" is not a coin flip. Which somebody is +chosen is the difference between a retry loop that terminates and one that +livelocks. + +### Step 11 — read-only transactions: the optimisation that made it shippable + +> **In:** the observation that a read-only transaction cannot have an rw edge +> pointing *in*. **Out:** Theorem 3, safe snapshots, `DEFERRABLE`, and the +> measured wait. + +A read-only transaction never writes, so nobody can fail to see its writes, so +it can never have an in-edge, so it can never be the pivot. But §2.1.2's +three-transaction anomaly shows it can still be **Tin** — the read-only `REPORT` +transaction is *essential* to that anomaly, "a surprising result discovered by +Fekete et al. [11]". So read-only transactions cannot simply be exempted. + +§4.1 sharpens it into a theorem of the paper's own: + +> **Theorem 3.** Every serialization anomaly contains a dangerous structure +> `T1 --rw--> T2 --rw--> T3`, where if T1 is read-only, T3 must have committed +> before T1 took its snapshot. + +The proof is three lines and worth following (§4.1): if there is a cycle, some +T0 precedes T1 in it; that edge cannot be rw or ww because T1 wrote nothing, so +it is a wr-dependency; a wr-dependency means T0 committed before T1's snapshot; +and T3 commits before T0 by Theorem 1. Hence T3 committed before T1's snapshot. + +Two things fall out: + +- **Read-only snapshot ordering** — a dangerous structure whose T1 is read-only + can be dismissed unless T3 committed before T1's snapshot. This is the + `SxactIsReadOnly(reader)` clause at `predicate.c:4525-4526`, comparing + `t2->prepareSeqNo` against `reader->SeqNo.lastCommitBeforeSnapshot` — the + union member from Step 8's struct at `predicate_internals.h:85-86`. +- **Safe snapshots** (§4.2) — if no concurrent read/write transaction has, or + could develop, an out-conflict to a transaction that committed before your + snapshot, then you cannot be part of any anomaly. Such a transaction "can read + any data (perform any query) without risk of serialization failure. It cannot + be aborted, and does not need to take SIREAD locks." The catch: safety is not + knowable when the snapshot is taken, only once every concurrent read/write + transaction has finished. So postgres tracks the candidates + (`possibleUnsafeConflicts`, `predicate_internals.h:108`) and, on success, sets + `SXACT_FLAG_RO_SAFE` (`predicate.c:3542`), at which point the read-only + transaction drops its markers and degrades to plain `REPEATABLE READ`. + +`DEFERRABLE` (§4.3) turns that from luck into a guarantee by waiting for it: + +```c +// postgres/src/backend/storage/lmgr/predicate.c — GetSafeSnapshot, 1493-1536 (elided) + 1493 while (true) + 1494 { + 1501 snapshot = GetSerializableTransactionSnapshotInt(origSnapshot, + 1502 NULL, InvalidPid); + 1503 + 1504 if (MySerializableXact == InvalidSerializableXact) + 1505 return snapshot; /* no concurrent r/w xacts; it's safe */ + ... + 1513 MySerializableXact->flags |= SXACT_FLAG_DEFERRABLE_WAITING; + 1514 while (!(dlist_is_empty(&MySerializableXact->possibleUnsafeConflicts) || + 1515 SxactIsROUnsafe(MySerializableXact))) + 1516 { + 1517 LWLockRelease(SerializableXactHashLock); + 1518 ProcWaitForSignal(WAIT_EVENT_SAFE_SNAPSHOT); + 1519 LWLockAcquire(SerializableXactHashLock, LW_EXCLUSIVE); + 1520 } + ... + 1523 if (!SxactIsROUnsafe(MySerializableXact)) + 1526 break; /* success */ + ... + 1535 ReleasePredicateLocks(false, false); + 1536 } +``` + +Line 1504-1505 is the special case §4.2 calls out: a snapshot taken when no +read/write transaction is active is *immediately* safe. Otherwise the loop waits +(:1514-1520) and, if the snapshot turns out unsafe, throws it away and retries +(:1535). §4.3 is candid that this can starve in theory. §8.4 measures it +instead: 1 200 deferrable transactions started against the heavy disk-bound +DBT-2++ workload had a **median wait of 1.98 s**, 90% under 6 s, and none over +20 s. + +Why it matters: `pg_dump` is a long read-only transaction. Without this, a +nightly backup would take SIREAD locks over the whole database and prevent every +other transaction's markers from being released. §4.3 names that failure mode +exactly — long readers "inhibit cleanup of other transactions' SIREAD locks … +this can easily exhaust memory." + +### Step 12 — bounded memory, and the one feature that breaks safe retry + +> **In:** SIREAD state that must outlive its transaction. **Out:** the four +> techniques that bound it, and why `PREPARE TRANSACTION` can make safe retry +> impossible. + +The retention rule comes from Corollary 2: only *concurrent* transactions can +share an rw edge, so a committed transaction's markers stay until every +transaction that overlapped it has finished. §6: "a single long-running +transaction can easily prevent thousands of transactions from being cleaned up." +Two requirements followed — bounded memory, and graceful degradation: "the +system should not fail to process new transactions because it runs out of +memory. Instead, it should be able to accept new transactions, albeit possibly +with a higher false positive abort rate." + +The four techniques, §6: + +1. **Safe snapshots and deferrable transactions** (§4.2) — Step 11. +2. **Granularity promotion** (§5.2) — Step 7. +3. **Aggressive cleanup** (§6.1) — drop a committed transaction's state the + moment it stops being needed. The subtlety: if active T1 has an out-edge to + committed T2, you still need to know whether T2 had an out-edge to some T3 + *and when T3 committed* — and T3 may be long gone. Hence the one extra field, + `earliestOutConflictCommit` (`predicate_internals.h:83-84`). A second + optimisation: when the only remaining active transactions are read-only, all + committed transactions' SIREAD locks can be discarded outright, since no + future write can conflict with them. +4. **Summarisation** (§6.2) — when the fixed slot count for committed + transactions is exhausted, collapse old ones into one shared record. "It is + usually sufficient to discover that a transaction has a conflict with some + previously committed transaction, but not which one." That shared record is + `OldCommittedSxact` (`predicate.c:364`, initialised at `:1275-1287`, fed by + `SerialAdd` at `:839`), and it is exactly what `FlagRWConflict` degrades to at + `:4438-4441`. Cost: a higher false-positive abort rate. + +**Two-phase commit** (§7.1) is the one interaction that costs a *property*, not +just performance. A prepared transaction cannot be aborted, so the pre-commit +check must run before `PREPARE`. Worse, consider + +``` + Tactive --rw--> Tprepared --rw--> Tcommitted +``` + +Rule 2 says abort the pivot — `Tprepared` — and you cannot. The only option is +`Tactive`, which on retry is still concurrent with `Tprepared` and likely to +fail identically. §7.1: this "sometimes makes it impossible to guarantee the +safe retry property." The code carries that concession in two places, killing +the reader instead of the writer at `predicate.c:4599-4610` +("Canceled on conflict out to pivot %u, during read") and committing suicide at +`:4685-4692` ("Canceled on commit attempt with conflict in from prepared +pivot"). And after a crash, §7.1 says postgres "conservatively assume[s] that +any prepared transaction has rw-antidependencies both in and out" — a prepared +transaction that survives a restart is treated as a pivot by default. + +Why it matters: every earlier conservative shortcut cost extra aborts. This one +costs the guarantee that retrying helps — the only place in the paper where the +degradation is qualitative. + +### Step 13 — the price, honestly + +> **In:** §8's three benchmarks. **Out:** every number the paper reports, with +> the ratios worked out, and the part of the bill that is not the database's to +> pay. + +§8 names two sources of overhead up front: tracking read dependencies (CPU, plus +contention on the lock manager's lightweight locks) and retries after +serialization failures, "some of which may be false positives." The comparison +baseline throughout is postgres's own `REPEATABLE READ` (= plain SI), with a +purpose-built S2PL implementation as a third point. + +| benchmark | configuration | result | source | +|---|---|---|---| +| SIBENCH | in-memory (tmpfs), a single `` table | tracking read dependencies costs **10–20% CPU**; SSI stays close to SI while S2PL falls away, because updates and scans cannot run concurrently under locking | §8.1 | +| DBT-2++ | 25 warehouses (3 GB), tmpfs, 4 threads — the CPU-bound worst case | SSI is a **5% slowdown** vs SI; SSI beats S2PL at every read-only fraction | §8.2, Fig 5a | +| DBT-2++ | 150 warehouses (19 GB), disk-bound, 36 threads | "the performance of SSI is indistinguishable from that of SI"; serialization failure rate **under 0.25%** in all cases | §8.2, Fig 5b | +| RUBiS | eBay-like auction site, 85% read-only, 6 GB dataset | see the arithmetic below | §8.3, Fig 6 | +| deferrable | started against the disk-bound DBT-2++ load, 1 200 trials | median **1.98 s** to a safe snapshot, 90% within 6 s, all within 20 s | §8.4 | + +The RUBiS row is the one to work out, because it is an end-to-end application +number rather than a database microbenchmark. Figure 6: + +``` + throughput (req/s) serialization failures + SI 435 0.004 % + SSI 422 0.03 % + S2PL 208 0.76 % + + SSI vs SI = 422 / 435 = 0.970 -> 3.0 % slower than snapshot isolation + S2PL vs SI = 208 / 435 = 0.478 -> 52.2 % slower than snapshot isolation + SSI vs S2PL = 422 / 208 = 2.03 -> SSI serves 2x the requests S2PL does + + failure rates: SSI aborts 0.03 / 0.004 = 7.5x as often as SI, + but S2PL aborts 0.76 / 0.03 = 25x as often as SSI. +``` + +That 2.03× is the paper's case in one number: on a read-heavy workload with +frequent rw-conflicts — §8.3 gives the concrete source, "queries that list the +current bids on all items in a particular category conflict with requests to bid +on those items" — locking loses half the machine, and SSI does not. + +Note also the one honest concession in §8.2: TPC-C as written "is known not to +exhibit anomalies under snapshot isolation [10]", so the authors had to import +the "credit check" transaction from Cahill's TPC-C++ variant to make anomalies +possible at all. The benchmark was modified to make SSI's job *harder*, which is +the right direction, but it means the abort rates are not TPC-C's. + +**And the part the database cannot pay.** A serialization failure arrives as +SQLSTATE 40001 with `errhint("The transaction might succeed if retried.")` +(`predicate.c:4597`, `:4609`, `:4656`, `:4692`). §3 assumes the retry exists: +"users must already be prepared to handle transactions aborted by serialization +failures, e.g. using a middleware layer that automatically retries +transactions." If the application does not retry, `SERIALIZABLE` does not +deliver serializable semantics — it delivers errors. All of §5.4's safe-retry +machinery is an optimisation of a retry loop that somebody else has to write. + +Why it matters: the numbers above are the *tracking* cost. The abort cost is +workload-shaped and unbounded — a workload built to produce pivots will produce +them, and §8's "well under 1%" is a property of TPC-C and RUBiS, not a promise. + +## Where each step lives in the code + +All anchors are `postgres/postgres@701f021`. Verify with +`python3 tools/pinned-source.py show postgres predicate.c -r A:B`. + +| Step | What | File | Lines | +|---|---|---|---| +| 6 | rw edge from MVCC data, write-first | `access/heap/heapam.c` | 9182-9263 | +| 6 | is the writer concurrent with me? | `storage/lmgr/predicate.c` | 3900-3917 | +| 6 | the read-side entry point | `storage/lmgr/predicate.c` | 3920-3936, 3952 | +| 7 | acquire a SIREAD marker | `storage/lmgr/predicate.c` | 2505, 2528, 2550 | +| 7 | writer checks for markers, finest → coarsest | `storage/lmgr/predicate.c` | 4265-4318 | +| 7 | per-target promotion threshold | `storage/lmgr/predicate.c` | 2217-2244 | +| 7 | the promotion test itself | `storage/lmgr/predicate.c` | 2255-2300 | +| 7 | lock table sizing | `storage/lmgr/predicate.c` | 263-264 | +| 7 | the defaults (64, −2, 2) | `utils/misc/postgresql.conf.sample` | 877, 879, 882 | +| 8 | per-transaction SSI state | `include/storage/predicate_internals.h` | 78-119 | +| 8 | the flag bits | `include/storage/predicate_internals.h` | 121-142 | +| 8 | record one rw edge | `storage/lmgr/predicate.c` | 656-677 | +| 9 | check, then record | `storage/lmgr/predicate.c` | 4429-4444 | +| 9 | the theorem, as a comment | `storage/lmgr/predicate.c` | 4446-4463 | +| 9 | case 1 / case 2 / case 3 | `storage/lmgr/predicate.c` | 4485-4487 / 4508-4532 / 4547-4578 | +| 10 | who dies, at edge time | `storage/lmgr/predicate.c` | 4580-4612 | +| 10 | who dies, at commit time | `storage/lmgr/predicate.c` | 4632-4705 | +| 11 | safe-snapshot wait for `DEFERRABLE` | `storage/lmgr/predicate.c` | 1487-1545 | +| 11 | the RO_SAFE hand-off | `storage/lmgr/predicate.c` | 3542 | +| 12 | the summarisation placeholder | `storage/lmgr/predicate.c` | 364, 1275-1287, 839 | +| 12 | "prepared" ≠ two-phase commit | `storage/lmgr/predicate.c` | 268-275 | ## How to read the paper (with the concepts in hand) -~1.5 h. §4–§7 are the production engineering, §8 the honest costs. +~1.5 h. The section numbering is not what a skim suggests: §4 is the read-only +theory (not implementation), §5 is the implementation, §6 memory, §7 feature +interactions, §8 the numbers. -1. **§1–2** — skim; SI background and the write-skew motivation you - already have (Step 1). -2. **§3** — the theory: rw-antidependencies and the dangerous structure - (Steps 2–3). Verify the doctors example against Fig-level detail; - confirm the "T_out commits first" condition. -3. **§4–§5 — read carefully.** SIREAD locks, granularity escalation, - index-range predicate handling, commit-ordering refinement, safe - snapshots (Steps 4–5). This is the part no other system had built. -4. **§6–§7** — memory bounding and 2PC; read §7 for *why* state must - outlive commit — it's the subtlest correctness point in the paper. -5. **§8 — read the numbers.** Overhead vs abort rate; note which - benchmarks stress pivot storms (Step 6). +1. **§1–§2** — skim. §2.1.1 is the doctors (Step 2), §2.1.2 the + three-transaction batch anomaly you need for Step 11, §2.2 the four + application-level workarounds and the Wisconsin Court System motivation: + hundreds of relations, "over 20 full-time programmers", queries auto-generated + by ORMs, so the n² analysis of which transaction pairs can skew "was simply + not feasible." +2. **§3 — read carefully.** §3.1's three edge types, §3.2's Theorem 1 and + Corollary 2, §3.3's algorithm and its two admissions (false positives; no + need for wr/ww edges). Do not skip §3.3.1: PSSI is the road not taken and the + reason is measured. +3. **§4 — read carefully.** Theorem 3 and its three-line proof, safe snapshots, + `DEFERRABLE`. This is the part that is *not* in Cahill. +4. **§5** — the engineering. §5.2's two detection paths are the key structural + idea; §5.2.1 the lock manager; §5.3 the state-size design space; §5.4 the + safe-retry rules. Read §5.4 with `predicate.c:4632` open beside it. +5. **§6–§7** — memory bounding and feature interactions. §7.1 (2PC) is the only + place the paper gives up a property rather than some performance. +6. **§8 — read the numbers**, and note which configuration each belongs to; the + overheads range from 20% to unmeasurable depending on whether the bottleneck + is CPU or disk. -For Cahill et al. (SIGMOD '08): the theorem statement is enough — this -paper productionizes it. +For Cahill, Röhm & Fekete (SIGMOD 2008): the algorithm and the theorem +statement are enough — this paper productionizes both. ## Questions for notes.md 1. Why must SIREAD locks outlive commit? Construct the history where the - dangerous structure completes after the reader committed. -2. Lock escalation trades memory for false aborts. Where's the same trade - in your mvcc.rs Serializable mode (hint: your read-set granularity is - whole keys — what's the graph equivalent of escalating to a relation)? -3. Read-only txns: why can they NEVER be T_pivot? (Which edge can't they - have?) How does that justify the safe-snapshot optimization? -4. M8: FalkorDB is single-writer. With exactly one writer at a time, can - a dangerous structure form at all between two write txns? Between a - writer and concurrent readers? So is SSI machinery needed, or does - single-writer + SI already equal serializable? (Prove it with the - pivot definition — this is the M8 design shortcut.) + dangerous structure completes *after* the reader committed. (§3.3 points at + the `T1 --rw--> T2` edge in Example 2; Corollary 2 tells you how long the + locks must then be kept.) +2. Granularity promotion trades memory for false aborts. Where is the same trade + in your `mvcc.rs` `Serializable` mode? Your read set is whole keys — what + would "escalating to a relation" mean there, and what would it do to + `serializable_mode_prevents_write_skew`? +3. Read-only transactions can never be the pivot. Which edge can they not have, + and why? Then: §2.1.2's anomaly *requires* a read-only transaction. Reconcile + those two statements. (Theorem 3 is the reconciliation.) +4. Postgres stores a *list* of rw edges per transaction where the original SSI + paper stored two bits. Name the two optimisations that the list buys (§5.3), + and say what each would have to give up under the two-bit scheme. +5. M8: FalkorDB is single-writer. With exactly one writer at a time, can a + dangerous structure form between two write transactions? Between a writer and + concurrent readers? Is SSI machinery needed, or does single-writer + SI + already equal serializable? Prove it from the pivot definition and Corollary + 2 — this is the M8 design shortcut. + +## Takeaway + +Serializability does not require blocking. It requires *noticing*. Fekete's +theorem shrinks "is this history serializable?" from a graph search to a +two-list walk at one node, and everything postgres added on top — +commit-ordering checks, read-only theory, granularity promotion, summarisation — +is a trade of precision for memory or memory for precision, each defended with a +number. The one thing never traded is soundness: every shortcut in the paper +produces *more* aborts, never a wrong answer. The bill, on the paper's own +benchmarks, is 3–5% throughput where the CPU is the bottleneck and nothing at +all where the disk is — plus a retry loop the application must write itself. + +## Connections to this topic's experiment + +`experiments/src/mvcc.rs` puts both halves of Step 2 in front of you, and it is +worth being precise about what it does and does not implement. + +- `write_skew_happens_under_snapshot_isolation` (`mvcc.rs:189-207`) is Figure 1, + key for key: `t1` reads `bob_on_call` and writes `alice_on_call`; `t2` reads + `alice_on_call` and writes `bob_on_call`; the test asserts **both commits + succeed** — "SI must ALLOW write skew". You have to be able to produce the bug + before you can prevent it. +- `serializable_mode_prevents_write_skew` (`mvcc.rs:210-224`) prevents it, but + **not the way postgres does**. `Mode::Serializable` is backward OCC: at commit, + validate that nothing in your read set was committed after your snapshot. The + module doc (`mvcc.rs:10-13`) says so — "stricter than postgres SSI, zero false + negatives for write skew; count the false positives later." +- That "stricter" is measurable against this paper. §3.3's Example-2-minus-T1 + execution — a single rw edge, no dangerous structure — is serializable, and + SSI permits it while "neither S2PL nor OCC would." Your `Mode::Serializable` + is in the OCC column: it will abort executions SSI would have allowed. The + end state is the same (`t2` gets `Err(CommitError::ReadConflict)` on the + doctors); the false-positive rate is not. + +**What this repo has measured, and what it has not.** The provided lane +(`FINDINGS.md` row 8, and the baseline table in +[notes.md](notes.md)) is a single global `Mutex`, 4 threads × 50 000 +transactions × 4 operations, on an Apple M3 Pro: + +| mix | global-lock txn/s | mvcc txn/s | aborts | +|---|---|---|---| +| read-heavy 95/5, 10K keys | 623 454 | stub | stub | +| write-heavy 50/50, 10K keys | 594 264 | stub | stub | +| write-heavy 50/50, 64 keys (HOT) | 676 691 | stub | stub | + +**The headline is the flatness, and it is a negative result.** ~600k txn/s on +all three mixes: the mutex does not care whether the workload is 95% reads or +50% writes, or whether it collides on 10 000 keys or 64, because it already +serialized everything. The 64-key row is even the *fastest* — a cache-resident +working set, with no contention penalty to pay because there was only ever one +lock to contend on. + +So: **this repo has not measured MVCC beating a mutex, and has not measured SSI +at all.** The `mvcc txn/s` and `aborts` columns are `stub` because you have not +written that code yet. Nothing in this guide should be read as a repo +measurement — every number above is Ports & Grittner's, on their 2011 hardware, +carrying its section. When you fill in those columns, the prediction worth +writing down first is in [notes.md](notes.md): MVCC should crush the baseline on +row 1, where readers never block, and may well *lose* on row 3, where +first-committer-wins converts key contention into aborted work the mutex never +had to redo. ## Done when -You can draw the dangerous structure from memory, place both write-skew -txns on it, and answer Q4 — it decides how much of this paper M8 needs. +Answer each before unfolding it. + +- [ ] Draw the dangerous structure, label all three transactions, and state the + extra condition Theorem 1 imposes beyond "two adjacent rw edges". +
Answer + + `Tin --rw--> Tpivot --rw--> Tout`, with the extra condition that **Tout is the + first transaction in the cycle to commit** (§3.2). The paper flags this as + stronger than Fekete et al. explicitly stated — they said only that T3 commits + before T1 and T2 — and it is what licenses both the commit-ordering + optimisation (§3.3.1) and safe-retry rule 1, "do not abort anything until T3 + commits" (§5.4). It is drawn in the source at `predicate.c:4454-4457`. + +
+ +- [ ] Place both doctors transactions on that structure, and say which one + postgres aborts and why the retry then succeeds. +
Answer + + The cycle has length 2: `T1 --rw--> T2 --rw--> T1`, so Tin = T1, + Tpivot = T2, Tout = T1 (Corollary 2 allows T1 and T3 to be the same + transaction). T1 commits first, satisfying rule 1. Safe-retry rule 2 says + abort the pivot, so **T2 dies** — concretely, T1's + `PreCommit_CheckForSerializationFailure` walks its in-edges to T2 + (`predicate.c:4659`), walks T2's in-edges back to itself (`:4669`, matching at + `:4674`), and sets `SXACT_FLAG_DOOMED` on T2 at `:4694`. T2 then errors at + `:4648` with SQLSTATE 40001. The retry succeeds because the retried T2 is no + longer concurrent with T1: it sees Alice already off call, counts 1, fails + its `>= 2` test, and writes nothing. Aborting T1 instead would have left it + concurrent with T2, so the same structure could re-form — §5.4's reasoning + for preferring the pivot. + +
+ +- [ ] Postgres detects rw-antidependencies two different ways. Name both, say + which chronology each handles, and which one is free. +
Answer + + §5.2 splits on whether the write or the read came first. **Write first:** the + MVCC data already answers it — if a tuple is invisible to you because its + creator had not committed when you took your snapshot, that *is* the edge. + `HeapCheckForSerializableConflictOut` (`heapam.c:9182`) reads the same + `xmin`/`xmax` the visibility check just read, so this direction is + essentially free. **Read first:** nothing in the tuple records that you read + it, so it needs the SIREAD marker — a passive entry in a dedicated lock + manager that "does not support any other lock modes, and hence cannot block" + (§5.2), checked by every writer via `CheckForSerializableConflictIn` + (`predicate.c:4265`). Only the second direction costs anything, and it is the + source of §8.1's 10–20% CPU overhead. + +
+ +- [ ] A serializable transaction reads 40 rows scattered one per page across a + table, with default settings. What SIREAD lock does it end up holding, and + what does that do to its abort risk? +
Answer + + A single **relation**-level marker. The relation promotion threshold is + `max_pred_locks_per_transaction / -max_pred_locks_per_relation - 1` = + `64 / 2 - 1` = **31** (`predicate.c:2223-2226` with the defaults at + `postgresql.conf.sample:877` and `:879`), and the count includes non-direct + descendants (`:2203-2204`), so the 32nd marker trips + `parentlock->childLocks > MaxPredicateChildLocks(...)` at `:2284-2285` and one + relation marker replaces the lot. (Had those rows been packed 3-to-a-page, + page-level promotion would have fired first, at threshold 2, `:2228-2229`.) + Consequence: from then on, *any* write by *any* concurrent transaction to that + table produces an rw edge against you, even to rows you never read. More false + edges, more false aborts — and never a wrong answer. That one-directionality + is the paper's recurring safety argument. + +
+ +- [ ] Why can a read-only transaction never be the pivot — and why is that not + enough to exempt it from SSI tracking? +
Answer + + It can never have an rw edge pointing *in*, because an in-edge means "a + transaction that couldn't see **our** write" (`predicate_internals.h:90-91`) + and it has no writes. No in-edge, no pivot. But §2.1.2's batch-processing + anomaly needs the read-only `REPORT` transaction as **Tin** — remove it and + the execution is serializable in the order ⟨T2, T3⟩. That read-only + transactions can participate at all was "a surprising result discovered by + Fekete et al. [11]". The exemption therefore has to be conditional, and + Theorem 3 (§4.1) supplies the condition: when Tin is read-only, Tout must have + committed **before Tin's snapshot**. That gives the false-positive filter at + `predicate.c:4525-4526` and the safe-snapshot rule of §4.2, under which a + read-only transaction drops its SIREAD locks and degrades to plain + `REPEATABLE READ`. + +
+ +- [ ] Name the number this guide reports for SSI's throughput cost, and the + number the repo's own lane reports — and say why they cannot be compared. +
Answer + + The paper's cost is configuration-dependent: **10–20% CPU** on SIBENCH (§8.1), + a **5% slowdown** on CPU-bound DBT-2++ (§8.2, 25 warehouses in tmpfs), + **3.0%** on RUBiS (422 vs 435 req/s, §8.3 Fig 6), and **nothing measurable** + on disk-bound DBT-2++ (§8.2, 150 warehouses). The repo's lane + ([`FINDINGS.md`](../../FINDINGS.md) row 8, [notes.md](notes.md)) reports + something else entirely: a global `Mutex` at **623 454 / 594 264 / + 676 691 txn/s** across read-heavy, write-heavy and hot-key mixes — flat, + because a single mutex had already serialized everything. They cannot be + compared because the repo has measured **no MVCC implementation and no SSI at + all**: the `mvcc txn/s` and `aborts` columns are still `stub`. Any claim here + that MVCC beats a mutex would be unmeasured. + +
## References **Papers** - Ports & Grittner — "Serializable Snapshot Isolation in PostgreSQL" - (VLDB 2012, [arXiv:1208.4179](https://arxiv.org/abs/1208.4179)) — - ~1.5 h; §4–§7 are the production engineering, §8 the honest costs -- Cahill, Röhm, Fekete — "Serializable Isolation for Snapshot Databases" - (SIGMOD 2008) — the dangerous-structure theorem this paper - productionizes; the theorem statement is enough + (VLDB 2012, [arXiv:1208.4179](https://arxiv.org/abs/1208.4179)) — ~1.5 h. §3 + the theory, §4 the read-only extension (this paper's own contribution), §5 the + implementation, §6 memory, §7 feature interactions, §8 the numbers. +- Cahill, Röhm & Fekete — "Serializable Isolation for Snapshot Databases" + (SIGMOD 2008) — the SSI *algorithm*, cited as [7] throughout. The + dangerous-structure check and SIREAD locks originate here. +- Fekete, Liarokapis, O'Neil, O'Neil & Shasha — "Making Snapshot Isolation + Serializable" (TODS 2005), cited as [10] — the *theorem* (Theorem 1 in §3.2) + that every cycle contains two adjacent rw-antidependencies. +- Berenson et al. — "A Critique of ANSI SQL Isolation Levels" (SIGMOD 1995) — + write skew as A5B; see [reading-ansi-critique.md](reading-ansi-critique.md). + +**In postgres** (`postgres/postgres@701f021`) + +| File | Lines | What | +|---|---|---| +| `src/backend/storage/lmgr/predicate.c` | 263-264 | lock-table sizing: `max_pred_locks_per_xact × (MaxBackends + max_prepared_xacts)` | +| `src/backend/storage/lmgr/predicate.c` | 268-275 | "prepared" is set by the pre-commit check even without 2PC | +| `src/backend/storage/lmgr/predicate.c` | 656-677 | `SetRWConflict` — one edge, threaded onto both lists | +| `src/backend/storage/lmgr/predicate.c` | 1487-1545 | `GetSafeSnapshot` — the `DEFERRABLE` wait | +| `src/backend/storage/lmgr/predicate.c` | 2217-2300 | promotion thresholds and the promotion test | +| `src/backend/storage/lmgr/predicate.c` | 2505-2550 | acquiring relation / page / tuple SIREAD markers | +| `src/backend/storage/lmgr/predicate.c` | 3900-3917 | `XidIsConcurrent` — the overlap test | +| `src/backend/storage/lmgr/predicate.c` | 3920-4010 | `CheckForSerializableConflictOut(Needed)` | +| `src/backend/storage/lmgr/predicate.c` | 4265-4318 | `CheckForSerializableConflictIn` — finest to coarsest | +| `src/backend/storage/lmgr/predicate.c` | 4429-4444 | `FlagRWConflict` — check first, record second | +| `src/backend/storage/lmgr/predicate.c` | 4465-4613 | the three-case pivot test, and who gets doomed | +| `src/backend/storage/lmgr/predicate.c` | 4632-4705 | `PreCommit_CheckForSerializationFailure` | +| `src/include/storage/predicate_internals.h` | 78-142 | `SERIALIZABLEXACT` and its flag bits | +| `src/backend/access/heap/heapam.c` | 9182-9263 | `HeapCheckForSerializableConflictOut` | +| `src/backend/access/heap/heapam.c` | 1750, 2054, 2345, 2628, 2959, 3963 | where the heap AM calls into SSI | +| `src/backend/utils/misc/postgresql.conf.sample` | 877, 879, 882 | the three predicate-lock defaults | + +**In this repo** + +| File | Lines | What | +|---|---|---| +| `experiments/src/mvcc.rs` | 10-13 | the contract for `Mode::Serializable`, and its own note that it is stricter than SSI | +| `experiments/src/mvcc.rs` | 189-207 | `write_skew_happens_under_snapshot_isolation` — Figure 1, key for key | +| `experiments/src/mvcc.rs` | 210-224 | `serializable_mode_prevents_write_skew` — read-set validation, not SSI | +| [`notes.md`](notes.md) | baseline table | the measured global-mutex lane; `mvcc` columns still `stub` | +| [`FINDINGS.md`](../../FINDINGS.md) | row 8 | the flat ~600k txn/s headline | diff --git a/topics/08-transactions-mvcc/reading-surrealdb-tx.md b/topics/08-transactions-mvcc/reading-surrealdb-tx.md index c05148c..302777e 100644 --- a/topics/08-transactions-mvcc/reading-surrealdb-tx.md +++ b/topics/08-transactions-mvcc/reading-surrealdb-tx.md @@ -1,152 +1,805 @@ # The minimal transactional KV interface: surrealdb's kvs layer -surrealdb doesn't implement MVCC — it *abstracts over* engines that do -(tikv, foundationdb, rocksdb, in-memory...), which forces it to define the -minimal transactional interface a multi-model DB needs. This chapter -builds that interface concept by concept — why abstracting forces -minimalism, how the layers stack, what gets declared up front, and which -primitives make OCC portable — then points you at the exact signatures. -Read this one for ARCHITECTURE, not algorithms: the interface is a good -checklist for M8's storage-backend abstraction (M1). +surrealdb doesn't implement MVCC — it *abstracts over* engines that do, which +forces it to write down the minimal transactional interface a multi-model +database needs. This chapter builds that interface concept by concept: why +abstracting forces minimalism, how the layers stack, what gets declared up +front, which primitives make optimistic concurrency portable, and — the part +that matters most — what the interface deliberately does **not** promise. Read +this one for ARCHITECTURE, not algorithms: the interface is a good checklist for +M8's storage-backend abstraction (M1). + +Line numbers below are `surrealdb/surrealdb@9d9a5b0`, checked with +`python3 tools/pinned-source.py`. Everything lives under +`surrealdb/core/src/kvs/`; that prefix is dropped in the prose below and +restored in the tables. + +**Two corrections to the previous version of this guide, made up front because +the rest depends on them.** (1) `Transactor` (`tr.rs:37`) is a *struct*, not a +trait — it holds `inner: Box`, and `Transactable` +(`api.rs:498`) is the actual trait each engine implements. (2) The engine list +at this pin is **five**, and FoundationDB is not among them: `kv-mem`, +`kv-indxdb`, `kv-rocksdb`, `kv-tikv`, `kv-surrealkv` (`core/Cargo.toml:19-35`, +enumerated as `DatastoreFlavor` at `ds.rs:556-567`). ## The problem in one sentence -One query engine must run transactions over five-plus storage engines with -wildly different concurrency machinery (single-node rocksdb, distributed -tikv/foundationdb, a plain in-memory map) — so what is the *smallest* set -of operations the query layer can demand from all of them? surrealdb's -answer fits in one trait: roughly eight method signatures in `tr.rs`. +One query engine must run transactions over five storage engines with wildly +different concurrency machinery — a plain in-memory map, single-node RocksDB and +SurrealKV, browser IndexedDB, distributed TiKV — so what is the *smallest* set +of operations the query layer can demand from all of them, and what happens to +the semantics that the smallest set cannot pin down? ## The concepts, step by step -### Step 1 — abstraction forces minimalism +### Step 1 — the vocabulary, and why abstraction forces minimalism + +> **In:** the words this layer uses in its own doc comments. **Out:** a +> definition for each, plus the design pressure that produced the interface. + +- **Transaction** — a group of reads and writes the database treats as one + unit. Here it is an object with a lifetime: you get one from the datastore, + call methods on it, and end it with exactly one of `commit` or `cancel`. +- **Snapshot** — the frozen view of the data a transaction reads from. + surrealdb does not implement snapshots; it *requires* them from the engine and + tests for them (`tests/snapshot.rs`, Step 6). +- **MVCC** — the engine-side mechanism that makes snapshots possible: writing a + key leaves the old value in place and adds a new version. See + [reading-postgres-heapam.md](reading-postgres-heapam.md). +- **Version** — one historical value of a key, addressed here by a `u64` + timestamp. `version: Option` appears in every read signature. +- **`commit` / `cancel`** — the two terminal operations. `cancel` "reverses all + changes made within the transaction" (`api.rs:519-522`). +- **CAS (compare-and-set)** — write a new value *only if* the current value + still equals what I expected. Here it is `putc` (`tr.rs:202`). +- **First-committer-wins** — the conflict rule where, among concurrent writers + of the same key, the first to commit succeeds and the rest are aborted. +- **Last-writer-wins** — the opposite: everybody commits, and the final value is + whoever committed last. Step 6 shows both, from the same interface calls. +- **Optimistic / pessimistic** — the two concurrency schools from + [reading-rocksdb-transactions.md](reading-rocksdb-transactions.md): validate + at commit and abort, versus take locks up front and block. +- **Capability** — an interface method that some engines implement and others + reject at runtime. The opposite of a guarantee. -When a database implements its own storage engine, the transaction -interface can be as fat and idiosyncratic as it likes — it has one caller -and one implementor. When it must run over N third-party engines, every -method in the interface must be implementable by ALL of them, so each -method must be either universal (get, set, commit) or explicitly optional -(a **capability** — "engines that can, do; callers must not assume"). +Now the design pressure. When a database owns its storage engine, the +transaction interface can be as fat and idiosyncratic as it likes: one caller, +one implementor. When it must run over N third-party engines, every *required* +method must be implementable by all of them. Anything one engine cannot do has +to become either a capability (implementable as an error) or a derived method +(implementable in terms of the required ones). -That pressure is the reason to read this code: the interface that survives -it is close to the theoretical minimum for "transactional ordered KV", -which is exactly the contract M8's storage-backend trait needs to name. +Why it matters: that pressure is the reason to read this code. The interface +that survives it is close to the theoretical minimum for "transactional ordered +KV" — which is exactly the contract M8's storage-backend trait needs to name. -### Step 2 — the layering: three structs between query and engine +### Step 2 — the layering: what sits between a query and an engine -The stack separates policy (caching, typed keys) from the portable -contract from the engine dispatch: +> **In:** a query that needs to read a row. **Out:** the four types it passes +> through, and which of them is engine-agnostic. ``` - Datastore (ds.rs) ── transaction() :3353 ──► Transaction (tx.rs:94) - │ caching + typed keys - ▼ - Transactor (tr.rs:37) - │ uniform async KV-txn API - ▼ - engine flavor (mem/rocksdb/tikv/fdb…) + Datastore (ds.rs:210) + │ .transaction(TransactionType, LockType) ds.rs:3353 + ▼ + TransactionFactory (ds.rs:302, impl :314) + │ .transaction(write, lock, sequences) ds.rs:348 + │ flattens both enums to bool ds.rs:355-363 + │ builder.new_transaction(write, lock) ds.rs:365 + ▼ + Transaction (tx.rs:94, impl :693) typed keys + a catalog cache + │ .tr: Transactor tx.rs:119 + ▼ + Transactor (tr.rs:37) thin typed wrapper, engine-agnostic + │ .inner: Box tr.rs:39 + ▼ + Transactable (api.rs:498) THE interface — implemented per engine + │ + ▼ + DatastoreFlavor::{Mem, RocksDB, IndxDB, TiKV, SurrealKV} ds.rs:556-567 +``` + +The dispatch is smaller than it looks. `TransactionFactory::transaction` reduces +both declared enums to a pair of booleans and hands them to the builder: + +```rust +// surrealdb/core/src/kvs/ds.rs — TransactionFactory::transaction, 348-375 (elided) + 348 pub async fn transaction( + 349 &self, + 350 write: TransactionType, + 351 lock: LockType, + 352 sequences: Sequences, + 353 ) -> Result { + 354 // Specify if the transaction is writeable + 355 let write = match write { + 356 Read => false, + 357 Write => true, + 358 }; + 359 // Specify if the transaction is lockable + 360 let lock = match lock { + 361 Pessimistic => true, + 362 Optimistic => false, + 363 }; + 364 // Create a new transaction on the datastore + 365 let (inner, local) = self.builder.new_transaction(write, lock).await?; + 366 Ok(Transaction::new( + ... + 371 Transactor { + 372 inner, + 373 }, + ... + 375 } ``` -- `Datastore` owns configuration and mints transactions; its - `TransactionFactory` (ds.rs:314) and builder plumbing (ds.rs:450–571) - are the multi-backend dispatch — M1's `StorageBackend` trait, grown up. -- `Transactor` is the uniform async KV-transaction API — the minimal - interface this chapter is about. -- `Transaction` wraps a Transactor with conveniences (Step 5). +Two booleans (:355-363) are the *entire* per-transaction configuration surface +that reaches an engine. Everything else the query layer wants is expressed as +method calls afterwards. -Why it matters: everything above `Transactor` is engine-agnostic by -construction; porting to a new engine means implementing one trait. +Note also `Transactor`'s `Drop` (`tr.rs:48-58`): dropping a writeable, +unfinished transaction logs `warn!` under `cfg(test)` (`:52-53`) and `error!` +otherwise (`:55-56`) — it does *not* silently roll back at this layer. (The +comment at `:54` says "Panic when running in normal mode"; the code does not +panic. Trust the code.) Ending a transaction is the caller's obligation, which +is a stricter contract than `experiments/src/mvcc.rs`'s ("Dropping a `Txn` +without commit = abort, no effects", `mvcc.rs:14`). + +Why it matters: everything above `Transactable` is engine-agnostic by +construction. Porting to a new engine means implementing one trait — and Step 4 +counts exactly how much work that is. ### Step 3 — declare intent at begin: read/write and the school -Two decisions are parameters of `begin`, not discoveries made mid-flight: - -- `TransactionType` (tr.rs:15) is just `Read | Write`, declared UP FRONT. - Compare postgres, where any transaction may write at any moment. - Declaring intent enables engines to specialize: a single-writer engine - can admit unlimited Read transactions concurrently and serialize only - the Writes; read-only transactions can skip conflict tracking entirely - (the same insight as SSI's safe snapshots). -- `LockType` on `Datastore::transaction()` (ds.rs:3353) is - `Optimistic | Pessimistic` — the CHOICE of concurrency school (the two - RocksDB flavors you just read) is a *per-transaction parameter*, passed - down to engines that support both. The school is workload-dependent - (contention flips the winner), so the interface refuses to hard-code it. - -Cost of declaring: the application must know its intent — a "Read" -transaction that tries to write is an error, not an upgrade. - -### Step 4 — the Transactor API: versioned reads and CAS as primitives - -The portable contract itself (tr.rs) — read the signatures, they ARE the -checklist. Two design decisions stand out: - -- **Versioned reads are public API.** `get`/`getm`/`getr`/`getp` - (:119–155) all take `version: Option` — point-in-time reads are - part of the KV contract, not an engine internal. Only some engines honor - it: a capability, not a guarantee. The price of exposing it: GC can't - drop what an API can still name (question 1). -- **Optimistic primitives are exposed, not hidden.** `set` :166 writes - unconditionally; `put` :190 fails if the key exists; `putc` :202 is - compare-and-set on the current value — write only if the value still - equals what I read. With get + putc alone, an upper layer can build - first-committer-wins snapshot isolation over ANY engine (question 3 - makes you sketch it) — OCC becomes portable because the *primitive* is - in the interface even when the engine's own machinery isn't. - -Plus the lifecycle pair: `commit` :103 / `cancel` :95 — commit is where -engine-level conflict errors surface, and the query layer retries. The -retry loop lives above the interface, matching SSI's lesson: serializable -semantics are a contract between engine AND application. - -### Step 5 — caching under a snapshot: invalidation deleted by MVCC - -`Transaction` (tx.rs:94, impl :693) wraps the Transactor with typed keys -and read-through caches. The thing to notice: caching *inside* a -transaction is trivially correct — the transaction reads a frozen -snapshot, so a value read once is valid for the transaction's whole life; -a within-txn cache never invalidates. Topic 6's hardest problem — -invalidation — deleted outright by MVCC's semantics. That is the kind of -simplification you get to collect when the layer below guarantees -snapshot reads. +> **In:** `ds.transaction(Write, Optimistic)`. **Out:** why both parameters are +> declared up front rather than discovered mid-flight, and what each buys. + +```rust +// surrealdb/core/src/kvs/tr.rs — the two begin-time enums, 13-34 + 13 /// Specifies whether the transaction is read-only or writeable. + 14 #[derive(Copy, Clone, Eq, PartialEq)] + 15 pub enum TransactionType { + 16 Read, + 17 Write, + 18 } + 19 + 20 /// Specifies whether the transaction is optimistic or pessimistic. + 21 #[derive(Copy, Clone)] + 22 pub enum LockType { + 23 Pessimistic, + 24 Optimistic, + 25 } + 26 + 27 impl From for LockType { + 28 fn from(value: bool) -> Self { + 29 match value { + 30 true => LockType::Pessimistic, + 31 false => LockType::Optimistic, + 32 } + 33 } + 34 } +``` + +- **`TransactionType`** (`:15`) is declared UP FRONT. Compare postgres, where + any transaction may write at any moment and the system discovers it (recall + `MyXactDidWrite = true` being set on the first write in + [reading-ssi-postgres.md](reading-ssi-postgres.md)). Declaring intent lets an + engine specialise: a single-writer engine can admit unlimited `Read` + transactions concurrently and serialize only the `Write`s, and a read-only + transaction can skip conflict tracking entirely — the same insight as SSI's + safe snapshots, but paid for by the caller instead of inferred. +- **`LockType`** (`:22`) makes the *choice of concurrency school* a + per-transaction parameter — the two RocksDB flavours you just read, selected + per call rather than per build. The interface refuses to hard-code it because + the right answer is workload-dependent: contention flips the winner. + +The cost of declaring is enforced, not advisory: `Transactor::writeable` +(`tr.rs:87`) is checked at the top of every mutating method, and a write on a +`Read` transaction is `Error::TransactionReadonly` (`api.rs:516`, raised at +e.g. `mem/mod.rs:354-356`). A `Read` transaction that tries to write is an +error, not an upgrade. + +Why it matters: this is the cheapest optimisation in the whole layer. One enum +at begin buys the engine the right to make an entire class of transactions +free — and it costs the application only the discipline of knowing its own +intent. + +### Step 4 — the `Transactable` trait: 19 required, 19 derived + +> **In:** the trait an engine must implement. **Out:** the exact split between +> what every engine must supply and what the trait builds for it, with the +> counts. + +The old version of this guide said the interface "fits in one trait: roughly +eight method signatures." The real shape is more interesting, and countable. At +`9d9a5b0`, `Transactable` (`api.rs:498-1082`) declares **38 methods: 19 with no +body (required) and 19 with a default body (derived)**. + +The 19 an engine must write: + +| group | methods | lines | +|---|---|---| +| introspection | `kind`, `closed`, `writeable` | `api.rs:500`, `:508`, `:517` | +| lifecycle | `cancel`, `commit` | `:522`, `:527` | +| point reads | `exists`, `get` | `:530`, `:533` | +| point writes | `set`, `put`, `putc`, `del`, `delc` | `:536`, `:539`, `:542`, `:545`, `:549` | +| range | `keys`, `keysr`, `scan`, `scanr` | `:558`, `:573`, `:588`, `:603` | +| savepoints | `new_save_point`, `release_last_save_point`, `rollback_to_save_point` | `:1010`, `:1013`, `:1016` | + +Everything else is *derived* — written once, in the trait, in terms of those 19: + +```rust +// surrealdb/core/src/kvs/api.rs — three derived methods, 668-676 + 726-735 (elided) + 666 /// Insert or replace a key in the datastore. + 668 fn replace(&self, key: Key, val: Val) -> BoxFut<'_, Result<()>> { + 669 Box::pin(async move { self.set(key, val).await }) + 670 } + 671 + 672 /// Delete all versions of a key from the datastore. + 674 fn clr(&self, key: Key) -> BoxFut<'_, Result<()>> { + 675 Box::pin(async move { self.del(key).await }) + 676 } + ... + 726 fn getp(&self, key: Key, version: Option) -> BoxFut<'_, Result> { + 727 Box::pin(async move { + ... + 733 let range = util::to_prefix_range(&key)?; + 734 self.getr(range, version).await + 735 }) + 736 } +``` + +`getm` (`:693`) is a loop over `get`. `getr` (`:745`) is a loop over +`batch_keys_vals`. `open_keys_cursor` (`:624`) wraps `keys`/`keysr` and advances +`range.start` between batches, with the doc comment noting that "backends that +can keep a native iterator alive across batches override this method" +(`:636-638`). `delp`, `delr`, `clrp`, `clrr`, `count`, `batch_keys`, +`batch_keys_vals`, `timestamp`, `safe_timestamp`, `timestamp_impl`, `compact` +round out the 19. + +Two of the derived methods are the cleanest statement of what a *capability* +means in this codebase: + +- `compact` (`:1079-1081`): the default body is + `bail!(Error::CompactionNotSupported)`. An engine that has a compaction + primitive overrides it; the doc comment (`:1076-1078`) says "the call is + advisory — callers must not rely on it for correctness." +- `safe_timestamp` (`:1054-1056`): the default returns `timestamp()`, and the + doc comment (`:1047-1053`) says exactly which engines that is correct for + ("mem, rocksdb, surrealkv") and which class "MUST override this … or the + router can miss notifications." + +Why it matters: 19 is the number to carry into M1. It says a transactional +backend trait needs two lifecycle methods, seven point operations, four range +operations, three savepoint operations and three introspection methods — and +that everything else you were tempted to put in the trait can be written once, +above it. + +### Step 5 — the two interesting primitives: versioned reads and CAS + +> **In:** the required point-operation signatures. **Out:** the two design +> decisions in them that are not obvious, and what each costs. + +**Versioned reads are public API.** Every read in the trait — `exists`, `get`, +`keys`, `keysr`, `scan`, `scanr`, and the derived `getm`/`getp`/`getr` — takes +`version: Option`. Point-in-time reads are part of the KV contract, not an +engine internal. But only some engines honour it, and the *shape* of the refusal +differs by engine: + +```rust +// surrealdb/core/src/kvs/mem/mod.rs — the capability gate, 51-58 + 51 impl Transaction { + 52 fn ensure_versioned(&self, version: Option) -> Result<()> { + 53 if !self.versioned && version.is_some() { + 54 return Err(Error::UnsupportedVersionedQueries); + 55 } + 56 Ok(()) + 57 } + 58 } +``` + +`mem` (`:52`), `rocksdb` (`rocksdb/mod.rs:165`) and `surrealkv` +(`surrealkv/mod.rs:58`) all carry that identical gate, keyed on a per-datastore +`versioned: bool` — they *can* time-travel if the datastore was opened for it. +TiKV cannot, at all, and says so inline in every read: + +```rust +// surrealdb/core/src/kvs/tikv/mod.rs — Transactable::get, 711-724 (elided) + 711 fn get(&self, key: Key, version: Option) -> BoxFut<'_, Result>> { + 712 Box::pin(async move { + 713 // TiKV does not support versioned queries. + 714 if version.is_some() { + 715 return Err(Error::UnsupportedVersionedQueries); + 716 } + ... + 724 let res = inner.tx.get(key).await?; +``` + +So `version: Option` is a *capability* with three tiers at this pin: +always-available (none), available-if-configured (mem, rocksdb, surrealkv), +never (tikv). `Error::UnsupportedVersionedQueries` is declared once, at +`err.rs:69`, and is the whole vocabulary for the refusal. The price of exposing +it at all: garbage collection cannot drop what an API can still name — question 1. + +**Optimistic primitives are exposed, not hidden.** Three writes, ordered by how +much they assume: + +- `set` (`tr.rs:166`) — "insert or update", unconditional. +- `put` (`tr.rs:190`) — "insert a key if it doesn't exist". +- `putc` (`tr.rs:202`) — "update a key … if the current value matches a + condition". Compare-and-set, with the expected value passed as + `chk: Option`. + +`putc`'s semantics are three arms, and the `None`/`None` case is the one people +miss: + +```rust +// surrealdb/core/src/kvs/mem/mod.rs — Transactable::putc, 347-367 (elided) + 347 fn putc(&self, key: Key, val: Val, chk: Option) -> BoxFut<'_, Result<()>> { + 348 Box::pin(async move { + ... + 359 // Set the key if valid + 360 match (inner.get(&key)?, chk) { + 361 (Some(v), Some(w)) if v == w => inner.set(key, val)?, + 362 (None, None) => inner.set(key, val)?, + 363 _ => return Err(Error::TransactionConditionNotMet), + 364 }; + ... + 367 } +``` + +Line 361 is the ordinary CAS. Line 362 is "I expected this key to be absent, and +it is" — `putc(k, v, None)` is `put` with the absence made explicit. Everything +else is `Error::TransactionConditionNotMet` (:363). + +With `get` + `putc` alone, a layer above can build first-committer-wins over an +engine that does not implement it: + +```rust +// ILLUSTRATION — not quoted from surrealdb. The real primitives are +// Transactor::get (tr.rs:119) and Transactor::putc (tr.rs:202); the +// three-arm CAS semantics are at mem/mod.rs:360-364. +async fn compare_and_swap(tx: &Transactor, key: &[u8], f: impl Fn(Option) -> Val) + -> Result<()> +{ + let before = tx.get(key, None).await?; // the value my decision is based on + let after = f(before.clone()); // my new value + tx.putc(key, after, before).await // Err(TransactionConditionNotMet) +} // if anyone changed it meanwhile +``` + +That is the whole trick: the *primitive* is in the interface even when the +engine's own conflict machinery isn't, so the retry loop can live above the +abstraction. Question 3 makes you extend it to a multi-key write set. + +Why it matters: `putc` is the reason this interface can offer uniform +concurrency semantics without demanding uniform engines. It is also, per Step 6, +not enough on its own. + +### Step 6 — the same calls, opposite outcomes: what the interface does not promise + +> **In:** three concurrent writers of one key, issuing identical calls. +> **Out:** two different, both-correct final states, and the cargo feature that +> decides which one you get. + +This is the sharpest thing in the `kvs` directory, and it is in the test suite +rather than the source. Two test files contain the *same* transaction script and +opposite assertions. Here is the script, with the assertions from both: + +``` + ds.transaction(Write, Optimistic); set("test", "some text"); commit -> ok + tx1 = transaction(Write, Optimistic); tx1.set("test", "other text 1") + tx2 = transaction(Write, Optimistic); tx2.set("test", "other text 2") + tx3 = transaction(Write, Optimistic); tx3.set("test", "other text 3") + tx1.commit() tx2.commit() tx3.commit() + + multiwriter_same_keys_conflict.rs multiwriter_same_keys_allow.rs + #![cfg(any(kv-mem, kv-rocksdb, #![cfg(kv-tikv)] :1 + kv-surrealkv))] :1 + tx1.commit().unwrap() :27 tx1.commit().unwrap() :27 + tx2.commit().unwrap_err() :28 tx2.commit().unwrap() :28 + tx3.commit().unwrap_err() :29 tx3.commit().unwrap() :29 + read back -> b"other text 1" :33 read back -> b"other text 3" :33 + + = FIRST-COMMITTER-WINS = LAST-WRITER-WINS +``` + +Identical API calls; three commits succeed on one engine and one succeeds on the +others; the surviving value differs. Nothing in `Transactable` forbids either. +`commit` is declared as `fn commit(&self) -> BoxFut<'_, Result<()>>` +(`api.rs:527`) with the doc comment "This attempts to commit all changes made +within the transaction" — *attempts* is the entire specification of its failure +behaviour. + +What the interface *does* pin down is snapshot reads, and there is a test for +that too — `tests/snapshot.rs`, which runs unconditionally rather than under a +`cfg`: + +``` + set("test", "some text"); commit :12-14 + tx1 = transaction(Read, ...); tx1.get("test") == b"some text" :16-19 + txw = transaction(Write, ...); txw.set("test", "other text") :21-23 + tx2 = transaction(Read, ...); tx2.get("test") == b"some text" :25-27 + tx3 = transaction(Read, ...); tx3.get("test") == b"some text" :29-31 + txw.set("test", "extra text") :33 + tx1.get("test") == b"some text" <- STILL, after two writes :35-36 + txw.commit() :42 +``` + +Line 35-36 is the guarantee: a reader's answer does not move under it, no matter +how many times a concurrent writer writes. `tx2` and `tx3`, begun *after* `txw` +wrote but *before* it committed, also see the old value (:25-31) — uncommitted +writes are invisible. That is snapshot isolation, stated as an executable +assertion at the portable layer. + +So the honest summary of the contract is a three-way split: + +| property | status in `Transactable` | evidence | +|---|---|---| +| snapshot reads; uncommitted writes invisible | **guaranteed** — tested for every engine | `tests/snapshot.rs` (no `cfg` gate) | +| point-in-time reads by version | **capability** — refused by name | `err.rs:69`, `tikv/mod.rs:713-716` | +| write-write conflict behaviour at commit | **unspecified** — engine-defined | the two test files above | + +Why it matters: this is the real cost of abstracting over engines, and it is not +the one the old version of this guide named. The *operations* port cleanly. The +*isolation semantics* do not, and the layer's response is not to paper over the +difference but to write two tests and gate them by cargo feature. If M8 takes +this trait shape, it inherits this decision — and had better make it +deliberately. + +### Step 7 — caching under a snapshot: what MVCC deletes, and what it doesn't + +> **In:** a read-through cache living inside a transaction. **Out:** which +> class of invalidation snapshot reads eliminate, and the two classes they +> don't. + +`Transaction` (`tx.rs:94`) wraps the `Transactor` with typed keys and a cache +(`tx.rs:121`). The cache is a *catalog* cache, not a row cache: its keys are +schema lookups — `Nss` (namespaces), `Dbs`, `Tbs`, `Ixs`, `Fds`, `Nds` +(cluster nodes) and about thirty more (`cache/tx/lookup.rs:11-70`). The pattern +is the ordinary read-through: + +```rust +// surrealdb/core/src/kvs/tx.rs — NodeProvider::all_nodes, 2173-2192 (elided) + 2173 fn all_nodes(&self) -> BoxProviderFut<'_, Result>> { + 2174 Box::pin( + 2175 async move { + 2176 let qey = cache::tx::Lookup::Nds; + 2177 match self.cache.get(&qey) { + 2178 Some(val) => val.try_into_nds(), + 2179 None => { + 2180 let beg = crate::key::root::nd::prefix(); + 2181 let end = crate::key::root::nd::suffix(); + 2182 let val = self.getr(beg..end, None).await?; + ... + 2184 let entry = cache::tx::Entry::Nds(Arc::clone(&val)); + 2185 self.cache.insert(qey, entry); + 2186 Ok(val) + 2187 } + 2188 } +``` + +The thing worth noticing is what is *absent* from :2177-2187: any check that the +cached answer is still current. There is no version, no generation counter, no +subscription to an invalidation channel. That is what a snapshot buys — +concurrent writers cannot change the answer, so the cache can never go stale +*because of someone else*. Topic 6's hardest problem, cross-actor invalidation, +is deleted outright by the layer below. + +**But "a within-txn cache never invalidates" — the previous version of this +guide's claim — is not true at this pin, in two ways.** + +1. **Self-invalidation is real, and explicit.** A transaction that writes the + catalog must invalidate its own cached view of it, because read-your-own-writes + means the cached answer is stale *to itself*: + +```rust +// surrealdb/core/src/kvs/tx.rs — after removing a namespace definition, 1399-1409 (elided) + 1399 self.set( + 1400 &rc, + ... + 1405 .await?; + 1406 // Invalidate cached namespace lookups so the removal is observed. + 1407 self.cache.remove(&cache::tx::Lookup::Nss); + 1408 self.cache.remove(&cache::tx::Lookup::NsByName(&ns_def.name)); + 1409 Ok(Some(ns_def.namespace_id)) +``` + + The same pattern appears for databases (`tx.rs:1450-1451`) and indexes + (`:1503-1505`), and `clear_cache` (`:2143-2147`) drops the lot. + +2. **It is a bounded cache, so entries can be evicted.** It is a `quick_cache` + with an estimated capacity and a weight budget (`cache/tx/mod.rs:41-46`). + Eviction is not invalidation, but it means a cache hit is never guaranteed. + +The one thing the snapshot *does* buy the implementation is worth quoting, +because it is a design decision most caches cannot make: `shards(1)` +(`cache/tx/mod.rs:44`), justified at `:35-39` — "The cache is per-transaction +and not concurrently accessed across threads, so `shards = 1` is used. The +default `available_parallelism() * 4` would allocate hundreds of sharded +`CacheShard` structs per transaction on large boxes, all of which then get +dropped at commit or cancel." + +Why it matters: the correct statement of the simplification is narrower and more +useful than the old one. Snapshot reads delete **cross-transaction** +invalidation. They do nothing about **self**-invalidation, which you still have +to get right by hand — and the comment at `tx.rs:1406` is what that looks like +when you do. ## Where each step lives in the code -All under `surrealdb/core/src/kvs/`; ~1 h. Read the Transactor signatures -in tr.rs first — they ARE the interface checklist. +All paths are relative to `surrealdb/core/src/`; ~1 h. Read the 19 required +`Transactable` signatures in `api.rs` first — they ARE the interface checklist. -- **Step 2 — layering**: `Datastore::transaction()` — ds.rs:3353; - `TransactionFactory` — ds.rs:314; builder plumbing — ds.rs:450–571; - `Transaction` — tx.rs:94; `Transactor` — tr.rs:37. -- **Step 3 — intent**: `TransactionType` — tr.rs:15 (`Read | Write`); - `LockType` (`Optimistic | Pessimistic`) on ds.rs:3353. -- **Step 4 — the contract**: versioned reads `get`/`getm`/`getr`/`getp` — - tr.rs:119–155; `set` :166, `put` :190, `putc` :202; `commit` :103, - `cancel` :95. -- **Step 5 — caching**: `Transaction` impl — tx.rs:693; note which - methods read through the cache and that nothing ever invalidates it. +| Step | What | File | Lines | +|---|---|---|---| +| 2 | `Datastore` and its `transaction()` entry point | `kvs/ds.rs` | 210, 3353 | +| 2 | `TransactionFactory` struct / impl / flattening | `kvs/ds.rs` | 302, 314, 348-375 | +| 2 | the five engine flavours | `kvs/ds.rs` | 556-567 | +| 2 | `Transaction` struct / impl | `kvs/tx.rs` | 94, 693 | +| 2 | `Transactor` and its `Drop` warning | `kvs/tr.rs` | 37-40, 48-58 | +| 3 | `TransactionType` and `LockType` | `kvs/tr.rs` | 13-34 | +| 3 | `writeable()`, and the readonly error | `kvs/tr.rs`, `kvs/api.rs` | 87, 510-517 | +| 4 | the `Transactable` trait, all 38 methods | `kvs/api.rs` | 498-1082 | +| 4 | the 19 required ones | `kvs/api.rs` | 500-603, 1010-1016 | +| 4 | `compact` — a capability with a default refusal | `kvs/api.rs` | 1071-1081 | +| 5 | versioned reads, in every read signature | `kvs/api.rs` | 530, 533, 558-609 | +| 5 | the capability gate, three engines | `kvs/mem/mod.rs`, `kvs/rocksdb/mod.rs`, `kvs/surrealkv/mod.rs` | 52, 165, 58 | +| 5 | TiKV's flat refusal | `kvs/tikv/mod.rs` | 713-716 | +| 5 | `set` / `put` / `putc` / `del` / `delc` | `kvs/tr.rs` | 166, 190, 202, 215, 226 | +| 5 | `putc`'s three arms | `kvs/mem/mod.rs` | 360-364 | +| 6 | first-committer-wins | `kvs/tests/multiwriter_same_keys_conflict.rs` | 1, 27-33 | +| 6 | last-writer-wins | `kvs/tests/multiwriter_same_keys_allow.rs` | 1, 27-33 | +| 6 | snapshot reads, ungated | `kvs/tests/snapshot.rs` | 16-42 | +| 7 | the transaction cache | `kvs/tx.rs`, `kvs/cache/tx/mod.rs` | 121, 27-56 | +| 7 | read-through, no staleness check | `kvs/tx.rs` | 2173-2192 | +| 7 | self-invalidation | `kvs/tx.rs` | 1407-1408, 1450-1451, 1503-1505, 2146 | + +## How to read the code + +1. **`api.rs:498-1082`** — the trait, top to bottom. Mark each method as + required or derived as you go; you should reach 19 and 19. +2. **`tr.rs:1-230`** — the typed wrapper. It is almost entirely + `key.into_vec()` then a delegation, which is the point: `Transactor` adds + types, not behaviour. +3. **`mem/mod.rs`** — the smallest engine implementation. Read `commit` + (`:183`), `putc` (`:347`) and `ensure_versioned` (`:52`); that is enough to + see the whole shape of an implementation. +4. **`kvs/tests/`** — read `snapshot.rs`, then the two `multiwriter_same_keys_*` + files back to back. This is Step 6 and it is the fastest 5 minutes in the + directory. +5. **`ds.rs:302-376`** — the dispatch, if you want to see how the two begin-time + enums reach an engine. +6. **`tx.rs`** — skim. It is 4 973 lines of typed catalog accessors; read + `all_nodes` (`:2173`) for the cache pattern and one `remove` site + (`:1406-1408`) for the invalidation, and move on. ## Questions for notes.md -1. `version: Option` on every read: what does time-travel-as-API cost - the engines that support it (GC can't drop what an API can name)? -2. Read/Write declared at begin: what optimizations does that unlock for - a single-writer engine? What does FalkorDB's GRAPH.RO_QUERY vs - GRAPH.QUERY split already encode? -3. putc (CAS) as the portable OCC primitive: sketch how you'd build - first-committer-wins snapshot isolation on top of ONLY get/putc. -4. M1 retrospective: does your storage-backend trait from topic 1 admit a - transactional backend, or did you bake in auto-commit? What would you - change now? +1. `version: Option` on every read makes time travel part of the public + API. What does that cost the engines that support it? (Think about what + garbage collection is allowed to drop once an API can still name an old + version — and compare postgres's answer in + [reading-postgres-heapam.md](reading-postgres-heapam.md), where the bound is + the oldest *snapshot*, not the oldest nameable timestamp.) +2. Read/Write declared at begin: what optimisations does that unlock for a + single-writer engine, and what does FalkorDB's `GRAPH.RO_QUERY` vs + `GRAPH.QUERY` split already encode? +3. `putc` as the portable OCC primitive: sketch first-committer-wins over ONLY + `get`/`putc`, for a transaction with a write set of *n* keys. Where does your + sketch break, and what does that tell you about why real engines do this + below the interface rather than above it? +4. Step 6 showed the same script producing first-committer-wins on three engines + and last-writer-wins on a fourth. If you were writing the query layer above + this trait, which of the two would you have to code against — and what would + you have to add to the trait to stop having to guess? +5. M1 retrospective: does your storage-backend trait from topic 1 admit a + transactional backend, or did you bake in auto-commit? Compare its method + count against this trait's 19 required. What would you change now? + +## Takeaway + +The minimum viable transactional KV interface is 19 methods: two lifecycle, +seven point, four range, three savepoint, three introspection. Everything else +— prefix scans, multi-gets, cursors, range deletes, compaction hints — is +derivable and belongs in the trait's default bodies, not in each engine. But the +interface's real lesson is negative: a portable set of *operations* does not +give you portable *semantics*. surrealdb pins down snapshot reads and tests them +for every engine; it exposes point-in-time reads as a named capability that +engines may refuse; and it leaves write-write conflict behaviour completely +unspecified, to the point of shipping two contradictory test files gated by +cargo feature. If you build an abstraction like this, the operations are the +easy part. Deciding, and writing down, which guarantees survive the abstraction +is the work. + +## Connections to this topic's experiment + +`experiments/src/mvcc.rs` is the layer *below* this one — the thing an engine +would have to implement to sit under `Transactable`. Mapping the two makes both +clearer: + +| `Transactable` (`api.rs`) | `mvcc.rs` | note | +|---|---|---| +| `commit` (`:527`) | `Txn::commit` (`mvcc.rs:105`) | yours returns a typed `CommitError`; the trait returns an opaque `Result` | +| `cancel` (`:522`) | dropping a `Txn` (`mvcc.rs:14`) | surrealdb warns on an undropped write txn instead (`tr.rs:48-58`) | +| `get(key, version)` (`:533`) | `Txn::get` (`mvcc.rs:89`) | yours has no `version` parameter — it is the always-`None` tier of Step 5 | +| `set` / `del` (`:536`, `:545`) | `Txn::put` / `Txn::delete` (`mvcc.rs:94`, `:99`) | same shape | +| `putc` (`:542`) | — | you have no CAS; Step 5's question 3 is about adding one | +| — | `Mvcc::gc` (`mvcc.rs:70`) | garbage collection is below the interface, which is exactly why versioned reads complicate it | + +Your `Mode::Snapshot` commit rule — first-committer-wins +(`mvcc.rs:7-9`) — is the `multiwriter_same_keys_conflict.rs` column of Step 6's +table, and `first_committer_wins_on_write_write_conflict` is the same assertion +as that file's `tx2.commit().unwrap_err()` at `:28`. Your `Mode::Serializable` +has no counterpart anywhere in `kvs`: nothing in `Transactable` validates a read +set, and nothing in it could, since the trait never sees which keys you read. + +**What this repo has measured, and what it has not.** The provided lane +(`FINDINGS.md` row 8, and the baseline table in [notes.md](notes.md)) is a +single global `Mutex`, 4 threads × 50 000 transactions × 4 operations, +on an Apple M3 Pro: + +| mix | global-lock txn/s | mvcc txn/s | aborts | +|---|---|---|---| +| read-heavy 95/5, 10K keys | 623 454 | stub | stub | +| write-heavy 50/50, 10K keys | 594 264 | stub | stub | +| write-heavy 50/50, 64 keys (HOT) | 676 691 | stub | stub | + +**The headline is the flatness, and it is a negative result.** ~600k txn/s on +all three mixes: the mutex does not care whether the workload is 95% reads or +50% writes, or whether it collides on 10 000 keys or 64, because it had already +serialized everything. The 64-key row is even the *fastest* — a cache-resident +working set, with no contention penalty to pay because there was only ever one +lock to contend on. + +So: **this repo has not measured MVCC beating a mutex, and has measured nothing +at all about surrealdb.** The `mvcc txn/s` and `aborts` columns are `stub` +because that code is yours to write. Nothing in this guide is a repo +measurement — this chapter contains no timings, only counted and quoted source. +When you fill in those columns, the prediction worth writing down first is in +[notes.md](notes.md): MVCC should crush the baseline on row 1, where readers +never block, and may well *lose* on row 3, where first-committer-wins converts +key contention into aborted work the mutex never had to redo. ## Done when -You can list the 6–8 operations a transactional KV interface needs to -support a multi-model DB, and say which are capabilities vs guarantees. +Answer each before unfolding it. + +- [ ] List the operations a transactional KV interface needs, grouped, and give + the count of required versus derived methods in `Transactable`. +
Answer + + **19 required** (`api.rs`): introspection `kind`/`closed`/`writeable` + (`:500`, `:508`, `:517`); lifecycle `cancel`/`commit` (`:522`, `:527`); point + reads `exists`/`get` (`:530`, `:533`); point writes + `set`/`put`/`putc`/`del`/`delc` (`:536`–`:549`); range + `keys`/`keysr`/`scan`/`scanr` (`:558`–`:603`); savepoints + `new_save_point`/`release_last_save_point`/`rollback_to_save_point` + (`:1010`–`:1016`). **19 derived**, with default bodies written in terms of + those: `replace`, `clr`, `clrc`, `getm`, `getp`, `getr`, `delp`, `delr`, + `clrp`, `clrr`, `count`, `batch_keys`, `batch_keys_vals`, `open_keys_cursor`, + `open_vals_cursor`, `timestamp`, `safe_timestamp`, `timestamp_impl`, + `compact`. Note that `Transactor` (`tr.rs:37`) is a struct wrapping + `Box`, not the trait itself. + +
+ +- [ ] Name one guarantee, one capability and one unspecified behaviour in this + interface, and the evidence for each. +
Answer + + **Guarantee — snapshot reads.** `tests/snapshot.rs` has no `cfg` gate, so + every engine must pass it: a reader's answer does not change under concurrent + writes (`:35-36`), and uncommitted writes are invisible to readers begun after + them (`:25-31`). **Capability — point-in-time reads.** `version: Option` + is in every read signature, but mem/rocksdb/surrealkv gate it on a + per-datastore flag (`mem/mod.rs:52-55` and its two twins) and TiKV refuses it + outright (`tikv/mod.rs:713-716`), both via `Error::UnsupportedVersionedQueries` + (`err.rs:69`). **Unspecified — write-write conflict behaviour.** `commit`'s + doc comment says only that it "attempts to commit" (`api.rs:524-527`), and the + test suite ships both outcomes: `multiwriter_same_keys_conflict.rs` (mem, + rocksdb, surrealkv) asserts `unwrap_err()` for the second and third committer, + `multiwriter_same_keys_allow.rs` (tikv) asserts `unwrap()` for all three. + +
+ +- [ ] Three transactions each `set` the same key, then commit in order. Give + both possible final values, and say which engines produce which. +
Answer + + `b"other text 1"` under first-committer-wins — `kv-mem`, `kv-rocksdb`, + `kv-surrealkv`, per `multiwriter_same_keys_conflict.rs:1` and its assertions + at `:27-29` (`unwrap`, `unwrap_err`, `unwrap_err`) and `:33`. Or + `b"other text 3"` under last-writer-wins — `kv-tikv`, per + `multiwriter_same_keys_allow.rs:1`, `:27-29` (three `unwrap`s) and `:33`. The + API calls are identical in both files; only the cargo feature and the + assertions differ. That is the interface declining to specify the semantics. + +
+ +- [ ] What does `putc(key, val, None)` mean, and how does it differ from + `putc(key, val, Some(old))` and from `put`? +
Answer + + `mem/mod.rs:360-364` gives all three arms. `putc(k, v, Some(w))` succeeds only + if the current value is exactly `w` (`:361`) — ordinary compare-and-set. + `putc(k, v, None)` succeeds only if the key is currently **absent** (`:362`) — + the same effect as `put` (`api.rs:539`, "insert a key if it doesn't exist"), + but with the expectation written down rather than implied. Anything else is + `Error::TransactionConditionNotMet` (`:363`). The reason `putc` matters is + that `get` + `putc` is enough to build first-committer-wins *above* the + interface, on an engine that does not provide it below. + +
+ +- [ ] Snapshot reads delete one class of cache invalidation. Which — and which + classes remain? +
Answer + + They delete **cross-transaction** invalidation: a concurrent writer cannot + change the answer to a read you already made, so `all_nodes` + (`tx.rs:2173-2192`) can cache with no version, no generation counter and no + subscription. What remains: (1) **self**-invalidation, because + read-your-own-writes makes your own catalog writes stale your own cache — + hence the explicit `self.cache.remove(...)` calls at `tx.rs:1407-1408`, + `:1450-1451`, `:1503-1505` with the comment "Invalidate cached namespace + lookups so the removal is observed"; and (2) **eviction**, since the cache is + a bounded `quick_cache` with a weight budget (`cache/tx/mod.rs:41-46`). "A + within-transaction cache never invalidates" is therefore too strong. + +
+ +- [ ] What has this repo measured about surrealdb, and what does the topic's + measured lane actually report? +
Answer + + **Nothing.** This chapter contains no timing at all — only counted and quoted + source at `surrealdb/surrealdb@9d9a5b0`. The topic's measured lane + ([`FINDINGS.md`](../../FINDINGS.md) row 8, [notes.md](notes.md)) is a global + `Mutex` at **623 454 / 594 264 / 676 691 txn/s** across read-heavy, + write-heavy and hot-key mixes — flat, because a single mutex had already + serialized everything, and *fastest* on the 64-key row because that working + set is cache-resident. The `mvcc txn/s` and `aborts` columns are `stub`: this + repo has **not** measured MVCC beating a mutex. + +
## References -**Code** -- [surrealdb](https://github.com/surrealdb/surrealdb) — - `surrealdb/core/src/kvs/`: `ds.rs`, `tr.rs`, `tx.rs`; ~1 h — read the - Transactor signatures in tr.rs, they ARE the interface checklist +**Code** (`surrealdb/surrealdb@9d9a5b0`, under `surrealdb/core/src/`) + +| File | Lines | What | +|---|---|---| +| `kvs/api.rs` | 498-1082 | `Transactable` — the interface: 19 required, 19 derived | +| `kvs/api.rs` | 1071-1081 | `compact` — a capability whose default body refuses | +| `kvs/tr.rs` | 13-34 | `TransactionType` and `LockType` | +| `kvs/tr.rs` | 37-58 | `Transactor` (a struct, not a trait) and its `Drop` warning | +| `kvs/tr.rs` | 95-230 | the typed wrapper: `cancel`, `commit`, reads, `set`/`put`/`putc`/`del`/`delc` | +| `kvs/ds.rs` | 302-376 | `TransactionFactory` — two enums flattened to two bools | +| `kvs/ds.rs` | 556-567 | `DatastoreFlavor` — the five engines | +| `kvs/tx.rs` | 94-141, 693 | `Transaction` — typed keys and the catalog cache | +| `kvs/tx.rs` | 2173-2192 | read-through caching with no staleness check | +| `kvs/tx.rs` | 1406-1408 | self-invalidation, with the comment that explains it | +| `kvs/cache/tx/mod.rs` | 27-56 | the per-transaction cache, `shards(1)` and why | +| `kvs/mem/mod.rs` | 52-57, 347-367 | the versioned-read gate and `putc`'s three arms | +| `kvs/tikv/mod.rs` | 711-716 | an engine refusing a capability inline | +| `kvs/err.rs` | 69, 80 | `UnsupportedVersionedQueries`, `CompactionNotSupported` | +| `kvs/tests/snapshot.rs` | 16-42 | snapshot isolation as an ungated assertion | +| `kvs/tests/multiwriter_same_keys_conflict.rs` | 1, 27-33 | first-committer-wins (mem, rocksdb, surrealkv) | +| `kvs/tests/multiwriter_same_keys_allow.rs` | 1, 27-33 | last-writer-wins (tikv) | +| `core/Cargo.toml` | 19-35 | the five `kv-*` features | + +**In this repo** + +| File | Lines | What | +|---|---|---| +| `experiments/src/mvcc.rs` | 6-15 | the commit contract your engine has to satisfy | +| `experiments/src/mvcc.rs` | 60-107 | the operations, mapped against `Transactable` above | +| [`notes.md`](notes.md) | baseline table | the measured global-mutex lane; `mvcc` columns still `stub` | +| [`FINDINGS.md`](../../FINDINGS.md) | row 8 | the flat ~600k txn/s headline | + +**Related chapters** +- [reading-rocksdb-transactions.md](reading-rocksdb-transactions.md) — the two + schools `LockType` selects between, one level down. +- [reading-ssi-postgres.md](reading-ssi-postgres.md) — what it takes to add read-set + tracking, which this interface has no hook for. +- [reading-postgres-heapam.md](reading-postgres-heapam.md) — the version chains + `version: Option` reads from, and why GC is the price. diff --git a/topics/09-concurrency/README.md b/topics/09-concurrency/README.md index 5506b9a..b9ad5b0 100644 --- a/topics/09-concurrency/README.md +++ b/topics/09-concurrency/README.md @@ -33,9 +33,13 @@ never "add threads". The counter table is the same physics one level down: three layouts where every thread owns its own counter and touches nobody else's, spanning 17.8×. `pad64` -— the x86-default `CachePadded` — is still 1.8× off `pad128`, because M-series -coherence granularity is 128 B. Check that assumption on your own hardware -before trusting any padding. +— a hand-written `#[repr(align(64))]`, the textbook "pad to one cache line" +advice — is still 1.8× off `pad128`, because M-series coherence granularity is +128 B. Crossbeam's `CachePadded` already knows this: it is +`repr(align(128))` on x86-64 *and* aarch64 (`crossbeam-utils/src/cache_padded.rs` +lines 70–77 and 87–94, at the pinned revision), because Sandy Bridge onwards +prefetches 64-byte lines in pairs. It is the 64-byte assumption in your own +head, not the crate's, that this lane is testing. And note which structure is *slowest* single-threaded: crossbeam's lock-free set, at 4.21 vs the mutex's 8.65. Atomics and epoch bookkeeping cost real diff --git a/topics/09-concurrency/notes.md b/topics/09-concurrency/notes.md index 22333ea..030a9d1 100644 --- a/topics/09-concurrency/notes.md +++ b/topics/09-concurrency/notes.md @@ -15,9 +15,11 @@ Predict FIRST, then measure. - **17.8× packed → pad128.** "Independent" counters sharing a line are not independent — this is the whole reason redis pads `used_memory`. - **pad64 is still 1.8× slower than pad128**: Apple M-series coherence - granularity is 128 B. `#[repr(align(64))]`, the x86 default, only HALF-fixes - false sharing on this machine. Check every `CachePadded` assumption against - the hardware you are actually on. + granularity is 128 B. A hand-written `#[repr(align(64))]` — the textbook "pad + to one cache line" — only HALF-fixes false sharing on this machine. Note that + crossbeam's `CachePadded` is *not* the thing being caught out here: it is + `repr(align(128))` on x86-64 and aarch64 alike. Check the 64-byte assumption + wherever you wrote it yourself. - **Run-to-run variance is large on the packed row** and worth knowing about: an earlier run of this same binary recorded 636 ms / 63 M inc/s, i.e. a 59× ratio rather than 17.8×. Contended-line throughput depends on how the threads diff --git a/topics/09-concurrency/reading-bwtree.md b/topics/09-concurrency/reading-bwtree.md index 50c98fb..9c79a34 100644 --- a/topics/09-concurrency/reading-bwtree.md +++ b/topics/09-concurrency/reading-bwtree.md @@ -8,67 +8,163 @@ latches cost, what CAS can and cannot do, and how the memory hierarchy prices the alternatives. The arc is this topic's thesis in miniature — the memory hierarchy, not elegance, decides which concurrency scheme survives. +This is a papers chapter, so the standard is different: **download the PDFs +and check the section numbers.** Every figure below is followed by the +section or table it came from, and where the two papers disagree the +disagreement is the point. The OLC protocol has real pinned source — +`leanstore/leanstore@90fcf18`, `backend/leanstore/sync-primitives/Latch.hpp` +— so Step 8 reads code, not pseudo-code. + ## The problem in one sentence Every thread traversing a latched B-tree *writes* the root's latch — even pure readers — so the root's cache line ping-pongs between all cores at -~100 cycles per bounce and read throughput stops scaling right when core -counts explode; the Bw-tree bet everything on eliminating latches, and -five years later an honest benchmark showed an optimistically-latched -B+tree beating it 1.5–4× with ~10× less code. +**38.3 ns per transfer on this machine** and read throughput stops scaling +right when core counts explode; the Bw-tree bet everything on eliminating +latches, and five years later an honest benchmark found it +**under-performing lock-based indexes by 1.5–4.5×** (Wang & Pavlo et al., +SIGMOD '18, §1) while requiring "an order of magnitude more code" than a +B-tree with optimistic lock coupling (Leis et al., IEEE Data Eng. Bull. +2019, §3.2). ## The concepts, step by step ### Step 1 — the enemy: latch traffic on hot cache lines +> **In:** a B-tree, N cores, and a read-only workload. +> **Out:** why *readers* are the problem, and the measured cost of the +> line they fight over. + A **latch** is a short-duration lock protecting a data structure's -physical integrity (nanoseconds, held across one node access — unlike -topic 8's transaction locks, which protect logical content for seconds). -The classic B-tree protocol, latch coupling, acquires the parent's latch, -then the child's, then releases the parent — correct, deadlock-free by -ordering, and a scaling disaster: acquiring even a *read* latch is a write -to the latch word, so every traversal by every thread writes the root's -cache line. Cache coherency (topic 0, Step 8) then bounces that line -between cores at ~100 cycles per transfer. With 64 cores doing lookups, -the root's latch line is the whole bottleneck — reads that share data -perfectly still serialize on the metadata. - -Both designs below are answers to exactly this. They differ in how much -of the latch they remove: all of it (Bw-tree) or just the reader's writes +*physical* integrity — nanoseconds, held across one node access — unlike +topic 8's transaction locks, which protect *logical* content for seconds. +The Bw-tree paper's follow-up states the convention outright: "in this +paper, we always use the term 'lock' when referring to 'latch'" (SIGMOD +'18, footnote 1, p. 1). Read "latch-free" and "lock-free" as synonyms +throughout. + +The classic B-tree protocol, **latch coupling**, acquires the parent's +latch, then the child's, then releases the parent — correct, deadlock-free +by ordering, and a scaling disaster. Leis et al. name the mechanism +precisely (§3): "lock acquisition and release require writing to the shared +memory location that implements the lock. This write causes exclusive +ownership of the underlying cache line and invalidates copies of it on all +other processor cores… the lock of the root node becomes a point of +physical contention — even in read-only workloads and even when read/write +locks are used." + +A **cache line** is the unit the memory system moves and owns. The +**coherence protocol** (MESI: Modified / Exclusive / Shared / Invalid) says +a line may be Modified on at most one core at a time, so writing it +requires invalidating every other copy. That invalidation is the cost, and +this topic measured it rather than quoting folklore: + +``` + false_sharing lane, 8 threads × 5M increments on their OWN counters: + packed (all 8 in one line) 202.7 ms → 40.54 ns per increment + pad128 (own line each) 11.4 ms → 2.28 ns per increment + ─────────────────────────────────────────────────────────────── + one cache-line ownership transfer = 40.54 − 2.28 = 38.3 ns +``` + +38.3 nanoseconds, not "~100 cycles" — the cycle figure is machine-specific +folklore and this machine's real number is roughly 4× larger than it. Now +apply it to a 4-level tree: latch-coupling a lookup takes and releases 4 +latches, ≈8 writes to shared words. If each is contended, 8 × 38.3 ns = +**306 ns of pure coherence traffic** for a lookup whose useful work is a +handful of binary searches. + +Leis et al. isolate exactly this in their Table 4 (a 100M-key B-tree, 10 +threads, per-operation counters): + +| lookup, 10 threads | Mop/s | cycles | instructions | L1 misses | +|---|---|---|---|---| +| no synchronization | 15.48 | 2058 | 283 | 38.6 | +| optimistic lock coupling | 14.60 | 2187 | 370 | 43.8 | +| traditional lock coupling | 5.71 | **5591** | 379 | 54.2 | + +Read the middle two columns together: lock coupling and OLC execute +**almost the same number of instructions** (379 vs 370) and lock coupling +burns **2.6× the cycles** (5591 vs 2187). The work is identical; the +difference is entirely the memory system, and it shows up as 10.4 extra L1 +misses per lookup — about 2.6 per latch acquire/release pair on a +four-level tree. At 20 threads, "lookup with OLC is 3.9× faster than +traditional lock coupling" (§4). + +Both designs below are answers to exactly this. They differ in how much of +the latch they remove: all of it (Bw-tree) or just the reader's writes (OLC). ### Step 2 — the lock-free toolkit: CAS, and its one-pointer limit +> **In:** the wish to remove latches entirely. +> **Out:** what CAS can do, what "lock-free" does and does not promise, +> and the one-word constraint that dictates the whole Bw-tree design. + **CAS** (compare-and-swap) is the atomic CPU instruction "replace this -64-bit word with a new value only if it still equals the value I read" — -the primitive from which all lock-free structures are built. A **lock-free** -structure guarantees system-wide progress with no latches at all: threads -publish changes by CASing a pointer; losers retry. - -The catch that shapes everything: CAS swaps ONE word. A B-tree update -often touches several (modify a node in place, or split one node and -update its parent — two pointers, two nodes). So a lock-free B-tree must +64-bit word with a new value only if it still equals the value I read". The +ICDE '13 paper defines it in a footnote on first use (footnote 1, §II.C), +which is the right instinct. + +**Lock-free** is a *progress* guarantee, not a description of instructions: +some thread always makes progress in a bounded number of steps, whatever +any other thread does — including being descheduled mid-operation. +**Wait-free** is the stronger property that *every* thread finishes in a +bounded number of its own steps. A CAS retry loop is lock-free but not +wait-free: one unlucky thread can lose every race indefinitely while the +structure races ahead. This distinction is the whole argument of Step 7 — +lock-freedom promises the *system* progresses, not that *your* insert does. + +The catch that shapes everything: **CAS swaps ONE word.** A B-tree update +often touches several — modify a node in place, or split one node and +update its parent (two nodes, two pointers). So a lock-free B-tree must recast every multi-word operation as a chain of single-pointer -publications — which is precisely the Bw-tree's design, and the source of +publications. That is precisely the Bw-tree's design, and the source of both its elegance and its downfall. +The related hazard, named once: the **ABA problem**. A thread reads pointer +`A`, is descheduled; another frees `A`, allocates a new node at the same +address, links it in; the first thread's CAS against `A` *succeeds* — value +unchanged, meaning changed. Every scheme in this chapter avoids it by +delaying reuse (epochs), not by detecting it. + ### Step 3 — the mapping table: indirection makes every change one CAS +> **In:** the one-word CAS limit from Step 2. +> **Out:** the Bw-tree's first move, and the second-order costs it buys. + The Bw-tree's first move: nodes are identified by logical **PIDs** (page ids), and a central **mapping table** maps PID → pointer to the node's current in-memory representation. All inter-node links store PIDs, never -raw pointers. Now "change node P17" = CAS the single mapping-table slot -for P17 — one word, exactly what CAS can do — and no parent or sibling -ever needs updating when a node's physical location changes. (Wu/Pavlo's -"logical pointers" verdict from topic 8 — same lesson: indirection -decouples updaters.) +raw pointers. The paper (ICDE '13, §II.B) puts it this way: "We use PIDs in +the Bw-tree to link the nodes of the tree. For instance, all downward +'search' pointers between Bw-tree nodes are PIDs, not physical pointers… +The mapping table severs the connection between physical location and +inter-node links." + +Now "change node P17" = CAS the single mapping-table slot for P17 — one +word, exactly what CAS can do — and no parent or sibling ever needs +updating when a node's physical location changes. §II.B calls this "relocation" tolerance, and notes that it "directly +enables both delta updating of the node in main memory and log structuring +of our stable storage". (Wu/Pavlo's "logical pointers" verdict from topic 8 +— same lesson: indirection decouples updaters.) + +**What it costs, measured.** Every node access is now two dependent loads +instead of one: read the PID out of the parent, then read the mapping table +to get the pointer. SIGMOD '18 removed the indirection to price it (§6.3, +"Disabling the Mapping Table"): "Read performance increases by 18% due to +fewer cache misses for load instructions (L1: 32% lower; L3: 52% lower)." +Half the L3 misses come from the indirection alone. ### Step 4 — delta chains: updates without touching the node +> **In:** a mapping table whose slot can be CASed. +> **Out:** the delta-chain representation, and the read cost it creates. + Second move: never modify a node in place. An update allocates a small -**delta record** ("insert k₁", "delete k₂") pointing at the node's current -representation, and CASes the mapping-table slot to point at the delta — -prepending to a chain: +**delta record** — a heap object describing one change, "insert k₁" or +"delete k₂" — that points at the node's current representation, and CASes +the mapping-table slot to point at the delta, prepending to a chain: ``` mapping table: PID ─► pointer update = CAS the PID slot: @@ -79,98 +175,338 @@ prepending to a chain: no in-place writes, no latches anywhere. ``` -Readers reconstruct the node by walking the chain down to the **base -node**, applying deltas as they go; when a chain grows too long, -**consolidation** folds it into a fresh base node (published, again, by -one CAS). Reclamation of replaced deltas/nodes uses epochs — the -crossbeam-epoch guide's scheme, and you know why: a reader may still be -walking the old chain. The cost is already visible if you've internalized -topic 0: a K-delta chain turns one node read into K dependent pointer -chases — K potential DRAM misses at ~100 ns each. +Readers reconstruct the node by walking the chain down to the **base node**, +applying deltas as they go. When a chain grows too long, **consolidation** +folds it into a fresh base node — published, again, by one CAS — and the +old chain is retired. This pattern, **install-and-consolidate**, is the +whole idiom: writers install cheap increments, and someone occasionally +pays to fold them back into a compact form. + +Reclamation of replaced deltas and nodes uses **epoch-based reclamation** — +the crossbeam-epoch guide's scheme, and ICDE '13 §II.C cites the same +lineage ("We use a form of epoch to accomplish safe garbage collection"). +You know why it is needed: a reader may still be walking the old chain. + +The 2013 paper argues this *helps* the cache (§II.A): "Avoiding +update-in-place reduces CPU cache invalidation, resulting in higher cache +hit ratios. Reducing cache misses increases the instructions executed per +cycle." Hold that claim; Step 7 measures it. + +The cost is already visible if you have internalised topic 0 §2: a K-delta +chain turns one node read into K *dependent* pointer chases, each a +potential DRAM miss. And K is not small in practice. SIGMOD '18 Table 2, +Insert-only with 20 threads, measures the **average leaf delta chain length +at 11.38** on the Rand-Int workload (0 on Mono-Int, 0.34 on the +high-contention one). Eleven dependent misses to read one node. ### Step 5 — SMOs: multi-node changes as cooperative state machines -Splits and merges (**SMOs** — structure modification operations) touch two -nodes and a parent, but CAS publishes one word — so the Bw-tree breaks -them into a sequence of individually-CASable steps: a half-split first -posts a split-delta on the child (readers now route around it), then a -separate CAS installs the new separator in the parent. Between steps, the -tree is in a valid-but-incomplete state — and any thread that stumbles on -a partial SMO must **help complete it** before proceeding (waiting for the -original thread would reintroduce blocking — question 2). Latched critical -sections become cooperative state machines: correct, and brutally hard to -write, test, and tune. - -### Step 6 — the reality check: SIGMOD '18 measures it honestly - -CMU rebuilt the design (OpenBw-Tree) and benchmarked it against an OLC -B+tree, Masstree, ART, and a skiplist. Findings: - -- **Delta chains murder cache locality**: a point read is a pointer chase - through K deltas (each hop a potential DRAM miss — topic 0's ladder) vs - a B+tree's two cache-resident binary searches. -- **The mapping table just relocates contention**: under skew, the hot - PID's slot is a hot cache line being CASed by everyone — you moved the - ping-pong from the latch word to the mapping slot, not removed it. -- **Consolidation policy is a whole tuning surface** — their §4.2 - component breakdown is the useful table; read it as a bill of costs. -- Verdict: **the OLC B+tree is 1.5–4× faster** on most workloads and ~10× - simpler. "Lock-free" bought worse constants, not scalability. - -### Step 7 — OLC: the modest protocol that won - -**Optimistic lock coupling** keeps the latch but makes readers stop -writing it. Per node: one u64 holding a version counter + a lock bit -(LeanStore's HybridLatch from topic 6 IS this). A writer CASes the lock -bit, mutates, and releases by incrementing the version. A reader never -acquires anything: it reads the version, reads the node, then re-checks -the version — if unchanged, nothing mutated underneath it; if changed, -RESTART from a safe ancestor: - -```rust -fn read_node(n: &Node, read: impl Fn(&Node) -> T) -> T { - loop { - let v1 = n.version.load(Acquire); - if v1 & LOCKED != 0 { spin_wait(); continue; } // writer active - let out = read(n); // read optimistically... - if n.version.load(Acquire) == v1 { - return out; // ...nothing moved: done - } // else a writer intervened: - } // restart — the only cost -} +> **In:** a split, which must change a child, a new sibling, and a parent. +> **Out:** the half-split protocol, the helping rule, and why "just wait +> for the owner" is not available. + +Splits and merges — **SMOs**, structure modification operations — touch two +nodes and a parent, but CAS publishes one word. ICDE '13 §II.D states the +problem and the fix: "we cannot install a page split with a single CAS… +To deal with this problem, we break an SMO into a sequence of atomic +actions, each installable via a CAS. We use a B-link design to make this +easier. With a side link in each page, we can decompose a node split into +two 'half split' atomic actions." + +A half-split first posts a split-delta on the child, so readers route +around it via the side link; a separate CAS then installs the new separator +in the parent. Between the two, the tree is in a valid-but-incomplete +state, and any thread that stumbles on a partial SMO must **help complete +it** before proceeding — §II.D: "In order to make sure that no thread has +to wait for an SMO to complete, a thread that sees a partial SMO will +complete it before proceeding with its own operation." + +Read that as a *consequence*, not a design flourish. Waiting for the owner +would reintroduce blocking, and blocking is what lock-freedom means you +gave up the right to do: the owner may have been descheduled by the OS for +a full time slice, so a waiter would be blocked by a thread that is not +running. Helping is the only option once you have committed to +lock-freedom — and it means every thread must contain a correct +implementation of every other thread's half-finished operation. Latched +critical sections become cooperative state machines: correct, and brutally +hard to write, test, and tune. (Question 2.) + +### Step 6 — read the 2013 evidence on its own terms + +> **In:** the ICDE '13 performance section (§VI). +> **Out:** what was actually measured, against what, on what — before you +> read what happened next. + +Rule six of this repo's reading standard says report the negative result. +Here the negative result *is* the story, so the honest way to tell it is to +state the 2013 claims precisely first, and then let the follow-up land. + +**The headline numbers (§VI.C, Fig. 6):** Bw-tree 10.4 M ops/s against +BerkeleyDB's 555 K on the Xbox LIVE workload — an **18.7× speedup**; 8.6× +on the deduplication trace; 5.8× on the synthetic workload. Against a +latch-free skip list (§VI.D, Table II): 3.83 vs 1.02 M ops/s on synthetic +(**3.7×**) and 5.71 vs 1.30 read-only (**4.4×**). Cache efficiency (§VI.E, +Fig. 7): "Almost 90% of its memory reads come from either the L1 or L2 +cache, compared to 75% for the skip list." + +**Now read the setup (§VI.A), which is where the claims live or die.** + +- The baseline is **BerkeleyDB in B-tree mode, non-transactional**, using + "page-level latching (the lowest latch granularity in BerkeleyDB)". A + disk-oriented storage engine with whole-page latches is not a + state-of-the-art in-memory index; it is topic 0's fair-benchmarking + pitfall — an unoptimised baseline, and an apples-to-oranges one. +- The machine is an **Intel Xeon W3550, four cores hyperthreaded to eight**, + and "we use 8 worker threads for each workload". A design whose entire + premise is that core counts are exploding was never evaluated above four + physical cores. +- The implementation is "approximately 10,000 lines of C++ code" (§VI.A) — + a number worth remembering when you reach the code-size comparison. + +**And read the retry data (§VI.B.2, Table I), because it is the premise +Step 7 overturns.** + +| workload | failed splits | failed consolidates | failed updates | +|---|---|---|---| +| Dedup | 0.25% | 1.19% | 0.0013% | +| Xbox | 1.27% | 0.22% | 0.0171% | +| Synthetic | 8.88% | 7.35% | 0.0003% | + +The paper's own reading: "The record update failure rate… is extremely low, +below 0.02% for all workloads… we believe these rates are still +manageable." **On the evidence presented, that is correct.** Work the +arithmetic: expected CAS attempts is `E[attempts] = 1/(1 − p)`, so at +p = 0.0002 you get 1.0002 attempts — one retry per five thousand updates. +Retries genuinely are free here. + +The defect is not the arithmetic. It is that the three workloads never +contained the case where p is large — and Step 7 found it. + +### Step 7 — the reality check: SIGMOD '18 measures it honestly + +> **In:** the 2013 design, reimplemented by people who did not write it. +> **Out:** which premises survived, which did not, and the section number +> for each. + +CMU rebuilt the design as **OpenBw-Tree**, because "the source code has not +been released" and the original description was "missing important details" +(§1). Their machine: two Intel Xeon E5-2680 v2, 10 cores × 2 HT each, 128 +GB, threads pinned to one socket unless stated (§5). Workloads: YCSB A/C/E +over 52M keys with Zipfian skew (§5.1). Their own optimisations bought +1.1–2.5× over a good-faith reimplementation of the original (§1) — so this +is the *charitable* version of the design. + +**Finding 1 — delta chains cost more than they save (§6.3).** They disabled +features one at a time: "by eliminating delta chains for the Read-only +workload, the performance increases by **23%**. If we backport this +modification to the original Bw-Tree, the performance improvement will +even be greater (**45%**)." Replacing delta updates with in-place updates +raises Insert-only throughput **40%**. That is 2013's §II.A claim — +avoiding update-in-place preserves caches — measured and reversed. + +**Finding 2 — the mapping table costs 18% of reads (§6.3).** Quoted in Step +3: L1 misses 32% lower, L3 52% lower without it. + +**Finding 3 — and this one is the deepest.** Disabling CAS "neither +Insert-only nor Read-only operations become significantly faster. This +seems to contradict the common belief that atomic operations like CaS +usually takes more cycles on some ISAs. Our experiments, however, pins the +worker thread on a single core, and therefore, the CPU can perform the CaS +locally, requiring almost no cache coherence overhead." + +Read that twice, because it is this topic's whole thesis stated by +accident: **the atomic instruction is not the cost; the contended line +is.** This topic's own lanes say the same thing in nanoseconds — an atomic +RMW on a line you own is 2.28 ns; the same instruction on a line another +core owns is 40.54 ns. It also means the §6.3 decomposition, being +single-threaded and pinned, *cannot* measure coherence at all — a limitation +the authors state rather than hide. + +**Finding 4 — lock-freedom loses hardest exactly where it was supposed to +win (§6.2).** They built a high-contention workload: every thread appends +monotonically increasing keys, the commonest OLTP insert pattern there is. +"OpenBw-Tree suffers from an extremely high abort rate as threads contend +for the head of the Delta Chain. Table 2 shows that the abort rate is over +1000%, i.e., on average there are more than 10 aborts for every insert." +And: "all lock-free indexes struggled more than any lock-based indexes; for +example, the SkipList failed to make progress in this high-contention +workload." Under contention the winners are Masstree, then ART, then the +OLC B+Tree — all lock-based. + +Now finish the arithmetic Step 6 started. Table 2's exact abort rates are +1.05% (Mono-Int), 1.44% (Rand-Int) and **1078.63%** (Mono-HC): + +``` + E[attempts] = 1/(1 − p) p = failure probability per attempt + ICDE'13 record updates, p = 0.0002 → 1.0002 attempts (1 retry / 5,000) + SIGMOD'18 Rand-Int, p = 0.0142 → 1.0144 attempts + SIGMOD'18 Mono-HC: 10.7863 aborts/insert ⇒ 11.79 attempts + ⇒ p = 1 − 1/11.79 = 0.915 +``` + +From p = 2 × 10⁻⁴ to p = 0.915 — a **5,900× increase in attempts per +insert** produced by nothing but a change of key distribution. That is the +lesson: lock-free retry cost is not a property of the algorithm, it is a +property of the workload's contention on one word, and the 2013 evaluation +had no workload that exercised it. Both papers are honest; only one of them +tested the case. + +**Finding 5 — the residue (§6.3, closing).** "Overall, after disabling +these lock-free features, the Bw-Tree is still **15%–19% slower** than the +B+Tree with OLC synchronization. We conjecture that the simplicity of OLC +upper bounds the number of instructions for every operation, while for the +Bw-Tree, even Read-only operations perform considerable bookkeeping." + +**The verdict, stated exactly (§1 and §6.1).** §1: "the overhead of the +Bw-Tree's indirection layer and delta records causes it to under-perform +the lock-based indexes by **1.5–4.5×**." §6.1 breaks that down: "ART is more +than 4× faster than the OpenBw-Tree for point lookups (though ART is slower +on Scan/Insert)… The OpenBw-Tree is also slower than the Masstree and the +B+Tree, often by a factor of **∼2×**." So the 4× belongs to ART, not to the +OLC B+Tree, and against the B+Tree the honest figure is about 2×. §8 +concludes: "lock-freedom does not always pay off in comparison with modern +lock-based synchronization techniques." + +The code-size claim is from a different paper and should be cited there: +Leis et al. §3.2 — "OpenBw-Tree, an open source implementation of the +Bw-tree, requires an order of magnitude more code than a B-tree based on +OLC" (with both implementations named in footnote 5). Alongside ICDE '13's +own "approximately 10,000 lines of C++" (§VI.A), that is the substantiated +version of "≈10× simpler". "Lock-free" bought worse constants, not +scalability — and a great deal more code. + +### Step 8 — OLC: the modest protocol that won + +> **In:** Step 1's diagnosis — the problem is the *reader's write*. +> **Out:** the protocol that removes only that, read from LeanStore's +> shipped implementation. + +**Optimistic lock coupling** keeps the latch but makes readers stop writing +it. Leis et al. §3.1 gives the whole protocol in six lines. A read-only +node access: (1) read the lock version, restart if the lock is not free; +(2) access the node; (3) read the version again and validate it has not +changed. A write: (1) acquire the lock, waiting if necessary; (2) write; +(3) increment the version and unlock. + +LeanStore ships this as `HybridLatch`, and it is pinned, so read the real +thing (this is topic 6's HybridLatch, now with the concurrency): + +```cpp +// backend/leanstore/sync-primitives/Latch.hpp:21-43 — the latch itself + 21 constexpr static u64 LATCH_EXCLUSIVE_BIT = 1ull; + 24 using VersionType = atomic; + 25 struct alignas(64) HybridLatch { + 26 VersionType version; + 27 std::shared_mutex mutex; + 41 bool isExclusivelyLatched() { return (version & LATCH_EXCLUSIVE_BIT) == LATCH_EXCLUSIVE_BIT; } + 42 }; + 43 static_assert(sizeof(HybridLatch) == 64, ""); +``` + +The lock bit is **the low bit of the version counter**, which makes the +whole protocol arithmetic. Acquiring adds 1 (setting the bit, `:161`, +`:147`); releasing adds 1 again (clearing it and bumping the version, +`:96-97`); so every completed write advances the version by exactly 2 and +an odd version means "locked". A reader that sees the same even version +before and after knows no writer touched the node in between. + +```cpp +// backend/leanstore/sync-primitives/Latch.hpp:84-115 — validate, release, and the optimistic entry + 84 void recheck() + 85 { + 87 assert(state == GUARD_STATE::OPTIMISTIC || version == latch->ref().load()); + 88 if (state == GUARD_STATE::OPTIMISTIC && version != latch->ref().load()) { + 89 jumpmu::jump(); + 90 } + 91 } + 93 inline void unlock() + 94 { + 95 if (state == GUARD_STATE::EXCLUSIVE) { + 96 version += LATCH_EXCLUSIVE_BIT; + 97 latch->ref().store(version, std::memory_order_release); + 98 latch->mutex.unlock(); + 106 inline void toOptimisticSpin() + 109 version = latch->ref().load(); + 110 if ((version & LATCH_EXCLUSIVE_BIT) == LATCH_EXCLUSIVE_BIT) { + 112 do { + 113 version = latch->ref().load(); + 114 } while ((version & LATCH_EXCLUSIVE_BIT) == LATCH_EXCLUSIVE_BIT); + 115 } ``` -Note what the reader never does: write shared memory. The root's cache -line stays Shared in every core's L1 — Step 1's enemy is dead, with a -plain B+tree's memory layout intact. "Coupling" survives as validation -order: validate the parent's version AFTER reading the child pointer — -the pair (read child ptr, revalidate parent) replaces "hold parent latch -while grabbing child". Two residual costs: restarts (rare — a restart -needs a writer to hit *your* path mid-read; question 3 has you compute -how rare), and torn reads of freed memory must be survivable, so node -reclamation still needs epochs or never-freed node memory. +Note what the reader never does: **write shared memory.** `toOptimisticSpin` +(`:106-117`) and `recheck` (`:84-91`) are plain loads only, so the root's +cache line stays Shared in every core's L1 and Step 1's enemy is dead — +with a plain B+tree's memory layout intact. The spin at `:112-114` is +test-and-test-and-set again: spin on ordinary reads, never on the atomic +(the LWLock guide, Step 4, has the same loop in C). + +"Coupling" survives as *validation order*: validate the parent's version +AFTER reading the child pointer — the pair (read child ptr, revalidate +parent) replaces "hold parent latch while grabbing child". `recheck()` at +`:84` is that call, and `jumpmu::jump()` at `:89` is LeanStore's restart: +a longjmp back to a registered restart point, i.e. the whole traversal +begins again. + +Three pieces of fine print, all of which people forget: + +- **Torn reads of freed memory must be survivable.** A speculatively-read + pointer may be garbage, so dereferencing it can segfault; Leis §3.3 shows + the extra validation (Figure 2, line 25) that prevents this, and §3.4 + confirms node reclamation still needs "epoch-based reclamation, hazard + pointers, or optimized hazard pointers" — the crossbeam-epoch guide's + subject, once more. +- **OLC can fall back; lock-free cannot.** §3.4: restarts can be capped and + the operation can drop to pessimistic locking "in cases of very heavy + contention. The ability to fall back to traditional locking is a major + advantage of OLC in terms of robustness over lock-free approaches, which + do not have this option." Compare Step 7's 1078% abort rate, which had + nowhere to fall back to. LeanStore has this: `toOptimisticOrShared` + (`:128-140`) and `toOptimisticOrExclusive` (`:141-154`) take the real + `std::shared_mutex` when the version says the node is contended. +- **The alignment is a bug on this machine.** `alignas(64)` and the + `static_assert(sizeof(HybridLatch) == 64)` at `:25`/`:43` pad each latch + to one *64-byte* line — the textbook advice. This topic's `false_sharing` + lane measures 64-byte padding at **1.8× slower** than 128-byte padding on + Apple M-series (20.4 ms vs 11.4 ms), because M-series cores prefetch + lines in 128-byte pairs, so two latches 64 B apart still travel together. + Postgres uses 128 (`pg_config_manual.h:217`) and crossbeam's + `CachePadded` is `repr(align(128))` on aarch64 *and* x86-64 + (`crossbeam-utils/src/cache_padded.rs:70-77`). LeanStore was written for + server x86; on an M3 you would change this constant and measure again. The arc, in one line: indirection + deltas (Bw) lost to versions + -restarts (OLC) because the memory hierarchy prices pointer chases higher -than optimistic retries. +restarts (OLC) because the memory hierarchy prices dependent pointer +chases higher than optimistic retries — and because a design that cannot +fall back has no answer when its retry probability goes to 0.9. ## How to read the papers (with the concepts in hand) -Read in arc order — design, autopsy, winner: - -1. **Levandoski et al., ICDE '13 (§II–IV)** — Steps 3–5 in the authors' - words: mapping table (§II), delta updates and consolidation (§III), - SMOs and helping (§IV). Read it 2013-generously: latch-free looked - inevitable, and the flash-friendly log-structured page store (§V, skim) - was half the motivation. -2. **Wang, Pavlo et al., SIGMOD '18** — the reality check, Step 6. The - §4.2 component breakdown is the table to study — read it as a bill of - costs; then the head-to-head graphs. Note *which* workloads are closest - for the Bw-tree and why (write-heavy, low skew). -3. **Leis et al., IEEE Data Eng. Bulletin 2019** — short; the OLC - protocol, Step 7. Map every rule to `read_node()` above, and note the - restart-safety requirements (survivable torn reads) — that's the fine - print people forget. +Read in arc order — design, autopsy, winner. Budget ~3 h. Download all +three; do not read summaries of them. + +1. **Levandoski, Lomet, Sengupta, ICDE '13** — Steps 3–5 in the authors' + words: architecture and the mapping table (§II.B), delta updating + (§II.C), SMOs (§II.D, then §IV for the full protocols), in-memory + latch-free pages and GC (§III). Skim §V (the log-structured store — half + the original motivation was flash, which is easy to forget when the + design is discussed as a pure in-memory index). **Then read §VI.A before + §VI.C**, so you know what the 18.7× is against. +2. **Wang, Pavlo et al., SIGMOD '18** — the autopsy, Steps 6–7. §3 lists + what the original description left out; §5.1 the workloads; §6.1 the + head-to-head; §6.2 high contention (Table 2 is the retry data); **§6.3 + is the component decomposition** — the bill of costs, and the section to + study. §8 for the conclusion in their own words. +3. **Leis, Haubenschild, Neumann, IEEE Data Eng. Bull. 2019** — short, ten + pages. §3.1 is the protocol; §3.2 has the code-size comparison; §3.3 the + dangling-pointer correctness argument; §3.4 the fallback and reclamation + fine print; §4 Table 4 and Figure 3 for the numbers used in Step 1. Map + every rule onto `Latch.hpp` as you go. + +Where the papers disagree, note which one measured. §II.A of ICDE '13 +argues delta records *improve* cache behaviour; §6.3 of SIGMOD '18 measures ++23%/+45% read throughput from removing them. Both statements are made in +good faith; only one is an experiment. ## Questions for notes.md @@ -194,15 +530,195 @@ Read in arc order — design, autopsy, winner: You can argue both sides — why Bw-tree looked inevitable in 2013 and why OLC won by 2018 — with the cache-line-level reasons, not slogans. +Answer each before unfolding it. + +- [ ] Give the 2018 verdict as a number, and say who it is against. +
Answer + + **1.5–4.5×**, and it is against "the lock-based indexes" *plural* — §1 of + Wang & Pavlo et al.: "the overhead of the Bw-Tree's indirection layer and + delta records causes it to under-perform the lock-based indexes by + 1.5–4.5×." + + Do not attribute the top of that range to the OLC B+Tree. §6.1 splits it: + "ART is more than 4× faster than the OpenBw-Tree for point lookups + (though ART is slower on Scan/Insert)… The OpenBw-Tree is also slower + than the Masstree and the B+Tree, often by a factor of ∼2×." Against the + OLC B+Tree specifically the honest figure is about **2×**. The + "order of magnitude more code" claim is from a different paper entirely — + Leis et al. §3.2, footnote 5. + +
+ +- [ ] Both papers measured CAS failure rates. Give both, convert each to + expected attempts per operation, and explain the gap. +
Answer + + ICDE '13 Table I (§VI.B.2): record-update failures **below 0.02%** on all + three workloads (splits and consolidates 0.22–8.88%). SIGMOD '18 Table 2 + (§6.2): 1.05% Mono-Int, 1.44% Rand-Int, and **1078.63%** on the + high-contention Mono-HC workload — "more than 10 aborts for every + insert". + + With `E[attempts] = 1/(1 − p)`: p = 0.0002 gives **1.0002** attempts, one + retry per 5,000 inserts; 10.7863 aborts per insert means 11.79 attempts, + i.e. p = 1 − 1/11.79 = **0.915**. A **5,900×** change, caused by nothing + but the key distribution — Mono-HC has every thread appending + monotonically increasing keys, so all of them CAS the same delta-chain + head. The 2013 paper's arithmetic was right; its three workloads simply + never contained the case. Contention is a property of the workload's + concentration on one word, not of the algorithm. + +
+ +- [ ] SIGMOD '18 found that disabling CAS changed nothing. Why, and why is + that the most important sentence in §6.3? +
Answer + + Because the experiment "pins the worker thread on a single core, and + therefore, the CPU can perform the CaS locally, requiring almost no cache + coherence overhead" (§6.3, "Disabling CaS"). A CAS on a line you already + own is nearly free. + + It matters because it separates the two things people conflate. The + *instruction* is cheap: this topic's `false_sharing` lane measures an + uncontended atomic RMW at **2.28 ns**. The *contended line* is not: the + same instruction on a line another core owns costs **40.54 ns**, a 38.3 ns + ownership transfer. Every optimisation in this chapter — padding, pinning + once per operation, readers that never write — targets the line, not the + instruction. It also means the whole §6.3 decomposition, being + single-threaded and pinned, cannot measure coherence effects at all; + the authors say so rather than letting the reader assume otherwise. + +
+ +- [ ] What did the 2013 Bw-tree get compared against, on what machine, and + why does that matter? +
Answer + + §VI.A: **BerkeleyDB in B-tree mode, non-transactional, with page-level + latching** ("the lowest latch granularity in BerkeleyDB"), plus a + latch-free skip list. The machine is an **Intel Xeon W3550 — four cores, + hyperthreaded to eight** — and all workloads use 8 worker threads. + + It matters twice. First, a disk-oriented engine holding whole-page + latches is not a state-of-the-art in-memory index; the 18.7× on the Xbox + workload (§VI.C, Fig. 6) is measured against an unoptimised, differently- + shaped baseline — topic 0's fair-benchmarking pitfalls in one sentence. + Second, a design justified by exploding core counts was never tested + above four physical cores; SIGMOD '18 ran 20 and 40 threads (§5), which + is where §6.2's 1078% abort rate appeared. + +
+ +- [ ] An OLC reader traverses four nodes. How many shared cache lines does + it write, and what is the measured consequence? +
Answer + + **Zero.** `toOptimisticSpin` (`Latch.hpp:106-117`) and `recheck` + (`:84-91`) contain only `latch->ref().load()` — plain atomic loads. The + latch lines stay in the Shared state in every core's L1. + + Leis et al. Table 4 measures the consequence at 10 threads: lock coupling + and OLC execute nearly identical instruction counts (379 vs 370) but lock + coupling burns **5591 cycles against OLC's 2187** — 2.6× — and 54.2 L1 + misses against 43.8. The work is the same; the difference is entirely the + coherence traffic of writing four latches twice each. At 20 threads OLC + is **3.9× faster** (§4). Priced with this topic's constant, 8 contended + writes × 38.3 ns ≈ 306 ns of pure interconnect per lookup. + +
+ +- [ ] LeanStore's `HybridLatch` is `alignas(64)`. Is that right on the + machine you are reading this on? +
Answer + + **No.** `Latch.hpp:25` and the `static_assert` at `:43` pin each latch to + a 64-byte line, which is the textbook rule and correct for the x86 + servers LeanStore targets. + + On Apple M-series it leaves money on the table. This topic's + `false_sharing` lane: 8 counters packed = 202.7 ms, padded to 64 B = 20.4 + ms, padded to 128 B = **11.4 ms**. So 64-byte padding recovers 9.9× of + the available 17.8× and is still **1.8× slower** than 128 — M-series cores + prefetch adjacent lines in 128-byte pairs, so two latches 64 B apart are + dragged around together. Postgres already uses 128 + (`pg_config_manual.h:217`, with the reasoning at `:208-215`) and + crossbeam's `CachePadded` is `repr(align(128))` on aarch64 *and* x86-64 + (`crossbeam-utils/src/cache_padded.rs:70-77`). "Pad to a cache line" is + not the rule; "pad to 128 bytes, then measure" is. + +
+ +- [ ] Why must a thread that finds a half-finished SMO complete it rather + than wait? +
Answer + + Because the owner may not be running. Lock-freedom's guarantee is that + *some* thread progresses regardless of what any other thread does — + including being descheduled by the OS mid-SMO. A waiter would then be + blocked on a thread that will not run for a full time slice, which is + precisely the blocking the design exists to eliminate. ICDE '13 §II.D: + "In order to make sure that no thread has to wait for an SMO to complete, + a thread that sees a partial SMO will complete it before proceeding with + its own operation." + + The price is that every thread must contain a correct implementation of + every other thread's half-finished operation, for every SMO, in every + intermediate state. That is a large part of what Leis et al. mean by "an + order of magnitude more code" (§3.2), and what SIGMOD '18's §3 ("Missing + Components") had to reconstruct from a paper that did not state it. + +
## References -**Papers** -- Levandoski, Lomet, Sengupta — "The Bw-Tree: A B-tree for New Hardware - Platforms" (ICDE 2013) — the design; §II–IV -- Wang, Pavlo et al. — "Building a Bw-Tree Takes More Than Just Buzz - Words" (SIGMOD 2018) — the reality check; §4.2's component breakdown - is the useful table, read it as a bill of costs -- Leis et al. — "Optimistic Lock Coupling: A Scalable and Efficient - General-Purpose Synchronization Method" (IEEE Data Eng. Bulletin 2019) - — short; the protocol that won +**Papers** — download all three; the section numbers below are checked +against the PDFs. + +| Paper | Sections | What is in them | +|---|---|---| +| Levandoski, Lomet, Sengupta — *The Bw-Tree: A B-tree for New Hardware Platforms* (ICDE 2013) | §II.A–II.D | modern-hardware rationale, mapping table, delta updating, SMO decomposition | +| | §III, §IV | latch-free pages, consolidation, epoch GC; full split/merge protocols | +| | §V | the log-structured flash store — skim, but know it was half the motivation | +| | **§VI.A** | the setup: BerkeleyDB baseline, 4-core machine, ~10,000 lines of C++ | +| | §VI.B.2, Table I | CAS failure rates: updates < 0.02%, splits/consolidates 0.22–8.88% | +| | §VI.C–VI.E | 18.7×/8.6×/5.8× vs BerkeleyDB; 3.7×/4.4× vs skip list; cache distribution | +| Wang, Pavlo et al. — *Building a Bw-Tree Takes More Than Just Buzz Words* (SIGMOD 2018) | §1 | the verdict: under-performs lock-based indexes by **1.5–4.5×** | +| | §2, §3 | Bw-tree essentials; what the original description omitted | +| | §5, §5.1 | machine (2× Xeon E5-2680 v2), YCSB A/C/E, 52M keys | +| | §6.1 | head-to-head: ART > 4×; Masstree and B+Tree ∼2× | +| | **§6.2, Table 2** | high contention: abort rate **1078.63%**; avg leaf delta chain 11.38 | +| | **§6.3, Fig. 18** | the decomposition: −DC +23%/+45%, −MT +18%, −DU +40%, −CAS ≈0; residue 15–19% | +| | §8 | "lock-freedom does not always pay off" | +| Leis, Haubenschild, Neumann — *Optimistic Lock Coupling* (IEEE Data Eng. Bull. 2019) | §3, §3.1 | why latch writes are the problem; the six-line protocol | +| | §3.2 | "an order of magnitude more code" than a B-tree with OLC | +| | §3.3, §3.4 | dangling speculative pointers; fallback to pessimistic locking; reclamation | +| | §4, Table 4 | 15.48 / 14.60 / 5.71 Mop/s and the cycle and miss counts used in Step 1 | + +**Code** (pinned at `leanstore/leanstore@90fcf18`) + +| File | Lines | What | +|---|---|---| +| `backend/leanstore/sync-primitives/Latch.hpp` | 21–43 | `HybridLatch` — version word with the lock in its low bit | +| | 84–91 | `recheck()` — the validation, and `jumpmu::jump()` as restart | +| | 93–104 | `unlock()` — release by incrementing, so a write advances the version by 2 | +| | 106–117 | `toOptimisticSpin()` — plain-load spin, no shared writes | +| | 128–154 | `toOptimisticOrShared` / `toOptimisticOrExclusive` — the fallback Leis §3.4 calls OLC's advantage | +| | 155–176 | `toExclusive()` — CAS the version, restart on failure | +| `backend/leanstore/sync-primitives/PageGuard.hpp` | — | how the guards compose into a traversal; read after `Latch.hpp` | + +**Measurements** — from this topic's own lanes; `notes.md` has the full +output, `FINDINGS.md` row 9 the headline. + +| Lane | Figure used above | +|---|---| +| `false_sharing` | uncontended padded atomic RMW **2.28 ns**; contended **40.54 ns**; transfer **38.3 ns** | +| `false_sharing` | pad64 20.4 ms vs pad128 11.4 ms — 64 B is **1.8× slower** on M-series | +| `scaling` | global mutex 8.65 → 2.96 Mops/s (1 → 16 threads): latching that scales *backwards* | + +**Cross-topic** — topic 0 §2 for the memory hierarchy that prices all of +this and topic 0 §3 for the fair-benchmarking pitfalls Step 6 applies; +topic 6 for `HybridLatch` read as a buffer-manager primitive; topic 8 for +logical pointers and for the lock/latch distinction; the crossbeam-epoch +guide for the reclamation both designs still need. diff --git a/topics/09-concurrency/reading-concurrent-skiplists.md b/topics/09-concurrency/reading-concurrent-skiplists.md index 17f3eca..4f547e6 100644 --- a/topics/09-concurrency/reading-concurrent-skiplists.md +++ b/topics/09-concurrency/reading-concurrent-skiplists.md @@ -4,201 +4,589 @@ Same structure, two schools of coordination: RocksDB's memtable skiplist links nodes with per-level CAS and never deletes; memgraph's skiplist — the spine of its whole graph store — uses per-node spinlocks, state bits, and real deletion with GC. Before you open either file, this chapter builds -both designs one concept at a time — the skiplist shape, the CAS toolkit, -each school's insert protocol, and the deletion problem only one of them -has to solve — then hands you the line anchors to watch each piece in -production code. Read RocksDB first (you know this file from topic 2 — now -the concurrency), then memgraph as the contrast. +both designs one concept at a time — the skiplist shape *as these two +implementations actually parameterise it*, the CAS toolkit, each school's +insert protocol, and the deletion problem only one of them has to solve — +then hands you the line anchors to watch each piece in production code. +Read RocksDB first (you know this file from topic 2 — now the concurrency), +then memgraph as the contrast. + +Everything below is read at the pinned commits **`facebook/rocksdb@7c80a5a`** +(`memtable/inlineskiplist.h`, 1422 lines) and +**`memgraph/memgraph@8f87f6a`** (`src/utils/skip_list.hpp`, 1854 lines). +Confirm with `python3 tools/pinned-source.py ref rocksdb`. ## The problem in one sentence Keep one sorted in-memory structure correct while 32 writer threads insert into it and readers traverse it at full speed — a single mutex around it -caps a 32-core machine at the throughput of one core, so both designs -coordinate at the granularity of individual pointers instead. +caps a 32-core machine at *less* than the throughput of one core (this +topic's `scaling` lane measures a global `Mutex` going from 8.65 +Mops/s at one thread to **2.96 Mops/s at sixteen**, 2.9× *backwards*), so +both designs coordinate at the granularity of individual pointers instead. ## The concepts, step by step -### Step 1 — the skiplist: a sorted linked list with express lanes +### Step 1 — the skiplist, with the constants these two actually chose + +> **In:** the need for a sorted structure whose inserts never move existing +> data. +> **Out:** the shape, and the two very different probability constants +> RocksDB and memgraph picked — plus why both are right. A **skiplist** is a sorted linked list where each node also gets a random number of stacked "express lane" links — a **tower**. A node's tower height -is chosen by coin flips at creation (height ≥ h with probability ~2⁻ʰ), so -level 1 skips ~half the nodes, level 2 skips ~three quarters, and so on: +is drawn from a geometric distribution at creation, so the higher lanes hold +exponentially fewer nodes: ``` level 3: head ─────────────────────► 50 ──────────────────────► nil level 2: head ─────────► 20 ───────► 50 ─────────► 80 ────────► nil level 1: head ──► 10 ──► 20 ──► 30 ─► 50 ──► 60 ──► 80 ──► 90 ─► nil - (level 1 = every node; search: top-left, go right until - you'd overshoot, drop a level — ~2·log₂ n ≈ 40 hops at 1M keys) + level 1 = every node. Search: start top-left, go right until + you'd overshoot, drop a level, repeat. +``` + +Generic write-ups say "height ≥ h with probability 2⁻ʰ". **Neither of these +implementations uses that.** Read the real constants: + +```cpp +// memtable/inlineskiplist.h:70-78 — RocksDB's defaults + 70 static const uint16_t kMaxPossibleHeight = 32; + 76 explicit InlineSkipList(Comparator cmp, Allocator* allocator, + 77 int32_t max_height = 12, + 78 int32_t branching_factor = 4); +``` + +```cpp +// memtable/inlineskiplist.h:559-573 — the coin, once per level + 559 int InlineSkipList::RandomHeight() { + 560 auto rnd = Random::GetTLSInstance(); + 562 // Increase height with probability 1 in kBranching + 563 int height = 1; + 564 while (height < kMaxHeight_ && height < kMaxPossibleHeight && + 565 rnd->Next() < kScaledInverseBranching_) { + 566 height++; + 567 } +``` + +`branching_factor = 4` means **p = 1/4**, not 1/2, and `max_height = 12` +caps the tower at 12 (not the 32 `kMaxPossibleHeight` allows). memgraph +went the other way: + +```cpp +// src/utils/skip_list.hpp:106-116 — one RNG draw, ffs gives a geometric height + 106 static uint8_t gen_height() { + 110 uint32_t value = thread_local_mt19937()(); + 111 if (value < 1UL << (32 - kSkipListMaxHeight)) return kSkipListMaxHeight; + 112 // The value should have exactly `kSkipListMaxHeight` bits. + 113 value >>= (32 - kSkipListMaxHeight); + 114 // ffs = find first set + 116 return static_cast(__builtin_ffs(value)); + 117 } +``` + +`__builtin_ffs` (find-first-set) of a uniform 32-bit value returns 1 with +probability ½, 2 with probability ¼, and so on — **p = 1/2**, with +`kSkipListMaxHeight = 32` (`:61`). One RNG call per node instead of a loop, +which is the trick the linked blog comment at `:101` is about. + +**Work the search cost on both and the answer is surprising.** The expected +number of hops for a skiplist with parameter p over n keys is +`(1/p) · log_{1/p}(n)`. At the 50 000-key preload of this topic's +`scaling` lane: + +``` + RocksDB, p = 1/4: log₄(50 000) = 7.80 levels × 4 hops/level = 31.2 hops + memgraph, p = 1/2: log₂(50 000) = 15.61 levels × 2 hops/level = 31.2 hops ``` -Why both engines picked it for concurrency: a B-tree (topic 1) keeps sorted -order by shifting rows inside pages and splitting full pages — bulk moves of -existing data. A skiplist keeps order purely with pointers, so an insert -never moves anything that already exists: it is one pointer swing per level, -and single pointer swings are exactly what atomic hardware instructions can -do. The cost: ~40 *dependent* pointer hops per search — up to 40 potential -cache misses (topic 0) — versus a B-tree's few cache-friendly binary -searches. Step 5 shows RocksDB clawing that back. +**Identical.** The expected hop count is `(1/p)·ln n / ln(1/p)`, and for +p = 1/2 and p = 1/4 the factor `(1/p)/ln(1/p)` is 2/0.693 = 2.885 and +4/1.386 = 2.885 — the function has its minimum around p = 1/e and is flat +between these two. So why did RocksDB pick 1/4? **Memory.** Expected tower +height is `1/(1−p)`: + +``` + p = 1/2: E[height] = 2.00 pointers per node + p = 1/4: E[height] = 1.33 pointers per node → 33% fewer pointers +``` -### Step 2 — CAS and the publication idiom +A memtable is size-capped and flushed when full, so a third fewer pointers +is a third more *keys* per memtable, which is directly fewer SST files. +memgraph's lists hold vertices and edges that live for the lifetime of the +database and are traversed constantly, so it buys the shallower, wider +search instead. Same structure, opposite pressure, and the constant is where +the difference is written down. + +The reason both engines picked a skiplist over a B-tree (topic 1) for +*concurrency* is separate: a B-tree keeps sorted order by shifting rows +inside pages and splitting full pages — bulk moves of existing data. A +skiplist keeps order purely with pointers, so an insert never moves anything +that already exists: it is one pointer swing per level, and single pointer +swings are exactly what atomic hardware instructions can do. The cost is +those ~31 *dependent* pointer hops per search — up to 31 serialised cache +misses (topic 0 §2) — versus a B-tree's few cache-friendly binary searches. +Step 5 shows RocksDB clawing that back. + +### Step 2 — CAS, memory ordering, and the publication idiom + +> **In:** a node built in private memory that must become visible to +> readers running right now. +> **Out:** the three orderings both files use, the one-word limit that +> splits the two schools, and where RocksDB's orderings actually live. **CAS** (compare-and-swap) is the atomic CPU instruction "replace this one 64-bit word with a new value only if it still equals the value I read" — if -another thread changed it in between, the CAS fails and you retry. Three -**memory orderings** appear in every listing below: **Relaxed** (the write -is atomic but promises nothing about *other* writes), **Release** (all my -earlier writes become visible to anyone who reads this value), and -**Acquire** (I see all writes that happened before the Release I just -read). - -Together they form the **publication idiom**: build your object privately -with plain/Relaxed writes, then *publish* it with one Release operation; -readers Acquire-load and are guaranteed to see a fully-built object. The -catch that splits the two schools: CAS swings ONE word, but a height-4 -tower is four links — a multi-pointer insert cannot be atomic, so each +another thread changed it in between, the CAS fails and you retry. + +**Memory ordering** decides what *else* a thread sees when it sees your +write. **Relaxed** guarantees only that the write is atomic — nothing about +the order of your other writes. **Release** on a store guarantees that +everything you wrote before it is visible to any thread that observes the +store. **Acquire** on a load is the other half: observe a Release-store and +you observe everything that preceded it. `SeqCst` adds a single total order +all threads agree on, and costs more. + +Together Release/Acquire form the **publication idiom**: build your object +privately with plain or Relaxed writes, then *publish* it with one Release +store; readers Acquire-load and are guaranteed a fully-built object. +RocksDB says exactly this in the comments on its accessors: + +```cpp +// memtable/inlineskiplist.h:379-396 — where the barriers live, and why + 379 Node* Next(int n) { + 381 // Use an 'acquire load' so that we observe a fully initialized + 382 // version of the returned Node. + 383 return ((&next_[0] - n)->Load()); + 384 } + 386 void SetNext(int n, Node* x) { + 388 // Use a 'release store' so that anybody who reads through this + 389 // pointer observes a fully initialized version of the inserted node. + 390 (&next_[0] - n)->Store(x); + 391 } + 393 bool CASNext(int n, Node* expected, Node* x) { + 395 return (&next_[0] - n)->CasStrong(expected, x); + 396 } +``` + +Two things worth stopping on. First, `Load`/`Store`/`CasStrong` are +RocksDB's own wrappers in `util/atomic.h` — `Load` is acquire (`:111-113`), +`Store` is release (`:108-110`), and `CasStrong` is **acq_rel** +(`:118-121`), not the release/relaxed pair you might guess. Second, look at +the addressing: `&next_[0] - n`. **The tower grows downward in memory.** +The comment at `:352-356` explains: the key is stored in the bytes +immediately after the struct and the higher `next_` pointers immediately +*before* it, so a node is one allocation with no separate tower array and +no stored height. `NoBarrier_SetNext` (`:399`, wrapping `StoreRelaxed` at +`util/atomic.h:60`) is the Relaxed variant used for the not-yet-published +half of the idiom. + +The catch that splits the two schools: **CAS swings ONE word, but a height-4 +tower is four links.** A multi-pointer insert cannot be atomic, so each school must decide what readers are allowed to see in between. ### Step 3 — the CAS school: link one level at a time (RocksDB) -RocksDB's answer: don't make the tower atomic — link it bottom-up, one CAS -per level, and let readers see partial towers. `CASNext` (:393) is the -linking primitive — one `compare_exchange_strong` per level. Per level: -read pred/succ, set `new->next = succ` (Relaxed — unpublished, so a plain -write is fine), CAS `pred->next` from succ to new (Release — the publish); -on failure, re-find just that level and retry: - -```rust -fn link_at_level(mut pred: &Node, new: &Node, lvl: usize) { - loop { - let succ = pred.next[lvl].load(Acquire); - new.next[lvl].store(succ, Relaxed); // unpublished yet: plain write - if pred.next[lvl] - .compare_exchange(succ, new, Release, Relaxed) // publish - .is_ok() { return; } - pred = refind_pred(new.key, lvl); // lost the race — re-find - } // ONLY this level, then retry -} +> **In:** a new node and a `Splice` of (pred, succ) per level. +> **Out:** RocksDB's actual insert loop, what a lost race costs, and why +> partial towers are harmless here. + +RocksDB's answer: don't make the tower atomic. Link it bottom-up, one CAS +per level, and let readers see partial towers. This is the real loop, not a +paraphrase: + +```cpp +// memtable/inlineskiplist.h:1134-1172 — Insert, asserts elided + 1134 if (UseCAS) { + 1135 for (int i = 0; i < height; ++i) { + 1136 while (true) { + 1137 // Checking for duplicate keys on the level 0 is sufficient + 1138 if (UNLIKELY(i == 0 && splice->next_[i] != nullptr && + 1139 compare_(splice->next_[i]->Key(), key_decoded) <= 0)) { + 1141 return false; + 1142 } + 1143 if (UNLIKELY(i == 0 && splice->prev_[i] != head_ && + 1144 compare_(splice->prev_[i]->Key(), key_decoded) >= 0)) { + 1146 return false; + 1147 } + 1152 x->NoBarrier_SetNext(i, splice->next_[i]); + 1153 if (splice->prev_[i]->CASNext(i, splice->next_[i], x)) { + 1155 break; + 1156 } + 1157 // CAS failed, we need to recompute prev and next. ... + 1162 FindSpliceForLevel(key_decoded, splice->prev_[i], nullptr, i, + 1163 &splice->prev_[i], &splice->next_[i]); + 1168 if (i > 0) { + 1169 splice_is_valid = false; + 1170 } + 1171 } + 1172 } ``` -Why bottom-first makes partial towers harmless *for a set*: level 1 (every -node) is the ground truth, and the node is findable the instant its -bottom link lands — upper levels are only shortcuts, so a reader that -doesn't see node 35 at level 2 yet still finds it at level 1: +Read four things off it: + +- **`NoBarrier_SetNext` then `CASNext`** (`:1152-1153`) is the publication + idiom, verbatim: relaxed write into the unpublished node, then one + acq_rel CAS that makes it reachable. +- **A lost race re-searches ONE level, from where it already was** + (`:1162-1163`): `FindSpliceForLevel` starts at `splice->prev_[i]`, not at + the head. The comment at `:1157-1161` gives the reasoning — it is unlikely + that many nodes landed between prev and next, so scanning forward from the + old prev beats restarting. No thread ever waits for another; there is no + full restart. +- **Duplicate detection happens only at level 0** (`:1138-1147`, and the + comment says so). That is the linearization point: whoever wins the level-0 + CAS owns the key. +- **A failed CAS above level 0 invalidates the splice** (`:1168-1170`), + because narrowing the bracket at level i may break the `Splice` invariant + stated at `:341-346` — `prev_[i+1].key <= prev_[i].key < next_[i].key <= + next_[i+1].key`. + +Why bottom-first makes partial towers harmless *for a set*: level 0 (every +node) is the ground truth, and the node is findable the instant its bottom +link lands — upper levels are only shortcuts, so a reader that doesn't see +node 35 at level 2 yet still finds it at level 0: ``` inserting 35, tower height 3: - level 3: 20 ─────────────► 50 (not linked yet — readers skip 35 here) - level 2: 20 ────► 35 ────► 50 (linked) - level 1: 30 ────► 35 ────► 50 (linked FIRST — 35 is now findable) + level 2: 20 ─────────────► 50 (not linked yet — readers skip 35 here) + level 1: 20 ────► 35 ────► 50 (linked) + level 0: 30 ────► 35 ────► 50 (linked FIRST — 35 is now findable) +``` + +**Now price the retry, because the folklore here is wrong.** "Lock-free +means you burn CPU on retries" is the standard worry. Evaluate it. Expected +CAS attempts under a per-attempt failure probability p is a geometric mean, +`E[attempts] = 1/(1−p)`. Estimate p for the `scaling` lane: 16 threads, +19.28 Mops/s, 10% writes ⇒ 1.93 M inserts/s spread over a 50 000-key +keyspace ⇒ about 39 inserts per second land at any one level-0 link point. +The CAS window — load succ, store, CAS — is maybe 5 ns. So +p ≈ 39 × 5e-9 ≈ **2 × 10⁻⁷**, and E[attempts] = 1.0000002: **one retry per +five million inserts**. Even a pathological workload where *every* insert +targets the same link point gives p ≈ 1.93e6 × 5e-9 = 9.6 × 10⁻³ and +E[attempts] = 1.0097 — a 1% overhead. + +**Retries are never the cost. The contended cache line is.** The same lane's +`false_sharing` companion measures one cross-core line transfer at +**38.3 ns** against a 2.28 ns uncontended atomic — so a *single* extra +bounced line costs as much as 17 wasted CAS attempts. Optimise for lines +touched, not for attempts avoided. (Step 6 of the Bw-tree guide has the +counter-example where p really does approach 1, and what it costs.) + +The contract comment states the guarantee the whole design buys: + +```cpp +// memtable/inlineskiplist.h:20-27 — the thread-safety contract + 20 // Thread safety ------------- + 22 // Writes via Insert require external synchronization, most likely a mutex. + 23 // InsertConcurrently can be safely called concurrently with reads and + 24 // with other concurrent inserts. Reads require a guarantee that the + 25 // InlineSkipList will not be destroyed while the read is in progress. + 26 // Apart from that, reads progress without any internal locking or + 27 // synchronization. ``` -The cost profile: a lost race costs a re-find of *one level* (a few hops), -never a full restart, and no thread ever waits for another. The contract -comment (:23) states the guarantee: `InsertConcurrently` is safe with -concurrent reads AND writes. +"Reads progress without any internal locking or synchronization" is the +prize: a reader writes **nothing**, so it never takes a cache line away from +anyone. Contrast the LWLock guide's Step 3, where a *shared* acquisition is +still a read-modify-write on a shared word. ### Step 4 — what the workload let RocksDB not build: deletion -The same contract (:23) hides the enabling assumption: memtable entries -are **never deleted**. In the LSM (topic 4), a delete is a *tombstone -insert*, and the whole memtable dies wholesale at flush — its arena is -freed in one shot. No delete ⇒ no "when may I `free()` a node some reader -still holds?" problem ⇒ no epochs, no GC, nothing. This is why the crate -you'll read next (crossbeam-epoch) is absent here. The discipline to carry -into every code read: always ask **"what did the workload let them NOT -solve?"** +> **In:** the contract comment above. +> **Out:** the assumption three lines below it that deletes an entire +> subsystem. + +The invariants immediately following that contract carry the enabling +assumption: + +```cpp +// memtable/inlineskiplist.h:29-38 — the invariants, and the one that matters + 29 // Invariants: + 31 // (1) Allocated nodes are never deleted until the InlineSkipList is + 32 // destroyed. This is trivially guaranteed by the code since we never + 33 // delete any skip list nodes. + 35 // (2) The contents of a Node except for the next/prev pointers are + 36 // immutable after the Node has been linked into the InlineSkipList. + 37 // Only Insert() modifies the list, and it is careful to initialize a + 38 // node and use release-stores to publish the nodes in one or more lists. +``` + +Memtable entries are **never deleted**. In the LSM (topic 4), a delete is a +*tombstone insert*, and the whole memtable dies wholesale at flush — its +arena is freed in one shot. No delete ⇒ no "when may I `free()` a node some +reader still holds?" problem ⇒ no epochs, no hazard pointers, no reference +counts, nothing. That is why the crate you read next (crossbeam-epoch) has +no counterpart anywhere in this file. + +Invariant (2) is the other half of the same bargain: nodes are immutable +after linking, so a reader that reaches a node never has to re-validate +anything it read. Both invariants are gifts from the LSM's write path, not +properties of skiplists. + +The discipline to carry into every code read: always ask **"what did the +workload let them NOT solve?"** — and then check whether *your* workload +grants the same permission. For `concurrent_set.rs` it does not: your tests +require `remove`, so you inherit Step 7's problem and must solve it with +epochs. ### Step 5 — the splice: amortizing the search across nearby inserts -The search (Step 1's ~40 dependent hops) dwarfs the CASes (one per level), -so RocksDB caches it. A `Splice` (:64) is a cached array of (pred, succ) -per level left over from the previous insert; sequential writers reuse it -(`Insert(key, splice, ...)` :1028, hint variant :113) and -`RecomputeSpliceLevels` (:331/:1016) repairs only the levels the new key -invalidated — nearby keys share most of their path, so most levels survive. -Amortize the O(log n) search across nearby inserts. +> **In:** the ~31 dependent hops from Step 1, dwarfing the ~2 CASes an +> insert needs. +> **Out:** the cached search path, when it survives, and the compile-time +> door that removes the atomics entirely. + +Do the arithmetic that motivates this. An insert at p = 1/4 costs ~31 +dependent pointer hops (Step 1) but only `E[height] = 1.33` CASes. If a hop +is an L2/LLC miss at ~15 ns, the search costs ~465 ns and the CASes ~3 ns. +**The search is 99% of the insert.** So RocksDB caches it: + +```cpp +// memtable/inlineskiplist.h:340-350 — Splice: a cached search path, with its invariant + 340 struct InlineSkipList::Splice { + 341 // The invariant of a Splice is that prev_[i+1].key <= prev_[i].key < + 342 // next_[i].key <= next_[i+1].key for all i. That means that if a + 343 // key is bracketed by prev_[i] and next_[i] then it is bracketed by + 344 // all higher levels. It is _not_ required that prev_[i]->Next(i) == + 345 // next_[i] (it probably did at some point in the past, but intervening + 346 // or concurrent operations might have inserted nodes in between). + 347 int height_ = 0; + 348 Node** prev_; + 349 Node** next_; + 350 }; +``` -One more workload door: `Insert` :908 vs `InsertConcurrently` :913 are the -same template with a `UseCAS` flag — single-writer mode skips the atomics -entirely, a *compile-time* choice. (M9 note: FalkorDB's single writer can -take exactly this door.) +A `Splice` is the (pred, succ) pair per level left over from the previous +insert. `InsertWithHint` (`:111`) and `InsertWithHintConcurrently` (`:117`) +take one; `RecomputeSpliceLevels` (`:331`, defined `:1016`) repairs **only +the levels the new key invalidated**. Sequential or near-sequential writers +— the common memtable pattern, since keys arrive roughly in order within a +column family — keep most levels and re-search only the bottom one or two. +The last sentence of the invariant is the concurrency subtlety: a splice may +be *stale* (someone inserted between prev and next) without being *wrong*, +which is exactly why the CAS at `:1153` can fail and re-search just its own +level. + +One more workload door: `Insert` (`:908`) and `InsertConcurrently` (`:913`) +are the same template (`:1028`) selected by a `UseCAS` bool — look back at +Step 3's `if (UseCAS)` at `:1134` and the `else` at `:1173`. Single-writer +mode skips the atomics entirely, and it is a **compile-time** choice, so +there is not even a branch. (M9 note: FalkorDB's single writer can take +exactly this door.) ### Step 6 — the locking school: lazy locking (memgraph, Herlihy et al.) +> **In:** the same insert problem, and a workload that needs whole towers to +> appear at once. +> **Out:** memgraph's optimistic-find / lock / validate / link / publish +> protocol — and five places where its authors found the published paper +> wrong. + memgraph takes the other road: make the whole tower appear atomically by -briefly locking the neighbors. A **spinlock** is a lock you busy-wait on -instead of sleeping — right for critical sections measured in nanoseconds. -Each Node (:156) carries a per-node `SpinLock` (:163), TWO state bits — -`marked` (:164) and `fully_linked` (:165) — and the flexible-array tower -`nexts[0]` (:169), the same intrusive-tower trick as RocksDB. - -Insert (:1335) is **optimistic**: `find_node` (:1285) collects preds/succs -with *no* locks held, then LOCKs the preds bottom-up, **re-validates** -(each pred still points at its succ, nobody got marked in between — the -optimistic read may be stale), links ALL levels while holding the locks, -and finally PUBLISHes with `fully_linked.store(true, release)` (:1398). -Readers ignore half-linked nodes — the publication idiom from Step 2, with -a bit instead of a CAS'd pointer, and the entire tower appears at once. -The cost profile flips Step 3's: a failed validation unlocks everything -and restarts the whole insert (vs the CAS school's one-level re-find), and -lock-order discipline (always bottom-up) is what prevents deadlock. - -### Step 7 — real deletion, and the scorecard - -memgraph must delete for real, and it does it in two phases. Remove -(:1655): lock, then `marked.store(true, release)` (:1672) — **logical -delete** first (readers skip marked nodes, so the node vanishes from the -set before any pointer moves), THEN physically unlink. Deletion exists -here, so reclamation must too — the problem RocksDB dodged in Step 4: - -- **Accessor-id GC** (:244–246, `SkipListGc` :257, `Collect` :367): every - `Accessor` (:877) gets a monotonically increasing id; a retired node - records the newest alive accessor id; free when all older accessors - are gone. Epoch reclamation with transaction-scoped pins — compare - crossbeam's 3-epoch scheme; same idea, coarser pin. -- `kSkipListGcHeightTrigger` (:69) and `create_chunks` (:817–955 — - chunked parallel iteration for analytics) show this is the SPINE of - memgraph: vertices, edges, and indexes all live in these lists. +briefly locking the neighbours. A **spinlock** is a lock whose waiter +busy-waits instead of sleeping — right for critical sections measured in +nanoseconds, and wrong for anything that might block (see the LWLock +guide's Step 4 for the test-and-test-and-set discipline a good one needs). + +Each node (`skip_list.hpp:156`) carries a per-node `SpinLock` (`:163`), two +state bits — `marked` (`:164`) and `fully_linked` (`:165`) — and the +flexible-array tower `nexts[0]` (`:169`), the same intrusive-tower trick as +RocksDB (memgraph's grows upward; RocksDB's downward). + +Insert is **optimistic**: find with no locks held, then lock, then check +that what you found is still true. + +```cpp +// src/utils/skip_list.hpp:1334-1399 — insert: find, lock, validate, link, publish + 1334 while (true) { + 1335 int layer_found = find_node(object, preds, succs); + 1360 for (int layer = 0; valid && (layer < top_layer); ++layer) { + 1361 TNode *pred = preds[layer]; + 1362 TNode *succ = succs[layer]; + 1363 if (pred != previous_locked) { + 1364 pred->lock.lock(); + 1367 previous_locked = pred; + 1368 } + 1369 // Existence test is missing in the paper. + 1370 valid = !pred->marked.load(std::memory_order_acquire) && + 1371 pred->nexts[layer].load(std::memory_order_acquire) == succ && + 1372 (succ == nullptr || !succ->marked.load(std::memory_order_acquire)); + 1373 } + 1375 if (!valid) continue; + 1390 for (int layer = 0; layer < top_layer; ++layer) { + 1391 new_node->nexts[layer].store(succs[layer], std::memory_order_release); + 1392 } + 1393 for (int layer = 0; layer < top_layer; ++layer) { + 1394 preds[layer]->nexts[layer].store(new_node, std::memory_order_release); + 1395 } + 1396 } + 1398 new_node->fully_linked.store(true, std::memory_order_release); + 1399 size_.fetch_add(1, std::memory_order_acq_rel); +``` + +The protocol, in order: `find_node` (`:1285`) collects preds/succs with *no* +locks held (so it may be stale); the loop at `:1360-1373` locks each +distinct pred **bottom-up** (deduplicated by `previous_locked` at `:1363`, +because one node is often the pred at several levels) and re-validates that +the pred still points at the succ and neither is marked; a failed validation +drops every lock via the `OnScopeExit` guard at `:1352-1356` and +`continue`s — **a full restart**; on success every level is written +(`:1390-1395`, both stores Release) while the locks are held; and the node +is finally PUBLISHed by `fully_linked.store(true, release)` at `:1398`. + +Readers ignore not-yet-`fully_linked` nodes — the publication idiom from +Step 2, with a bit instead of a CAS'd pointer, so the entire tower appears +at once. The cost profile is the exact inverse of Step 3: a lost race costs +a whole-insert restart rather than a one-level re-find, and lock-order +discipline (always bottom-up) is what prevents deadlock. + +**And then read the comments.** `:1358-1359` says "The paper has a wrong +condition here. In the paper it states that this loop should have `(layer <= +top_layer)`, but that isn't correct." `:1369`: "Existence test is missing in +the paper." `:1388-1389`: "The paper is also wrong here." Two more in +`remove` (`:1648-1649`, `:1679-1680`, `:1694-1695`). That is **five separate +corrections to a peer-reviewed algorithm** (Herlihy, Lev, Luchangco & +Shavit, SIROCCO 2007), found by people who had to run it. Take it as the +topic's recurring lesson in its most literal form: published concurrent +algorithms are specifications with bugs, and the errata live in the source +files of whoever shipped them. + +### Step 7 — real deletion, reclamation, and the scorecard + +> **In:** a node that must be removed from a list readers are traversing +> right now. +> **Out:** the two-phase delete, memgraph's accessor-id GC, and the +> side-by-side comparison. + +memgraph must delete for real, and it does it in two phases: + +```cpp +// src/utils/skip_list.hpp:1662-1701 — remove: mark, then unlink, then collect + 1662 while (true) { + 1663 int layer_found = find_node(key, preds, succs); + 1664 if (is_marked || (layer_found != -1 && ok_to_delete(succs[layer_found], layer_found))) { + 1665 if (!is_marked) { + 1666 node_to_delete = succs[layer_found]; + 1667 top_layer = node_to_delete->height; + 1668 node_guard = std::unique_lock{node_to_delete->lock}; + 1669 if (node_to_delete->marked.load(std::memory_order_acquire)) { + 1670 return false; + 1671 } + 1672 node_to_delete->marked.store(true, std::memory_order_release); + 1673 is_marked = true; + 1674 } + 1681 for (int layer = 0; valid && (layer < top_layer); ++layer) { + 1688 valid = !pred->marked.load(std::memory_order_acquire) && + 1689 pred->nexts[layer].load(std::memory_order_acquire) == succ; + 1690 } + 1692 if (!valid) continue; + 1696 for (int layer = top_layer - 1; layer >= 0; --layer) { + 1697 preds[layer]->nexts[layer].store(node_to_delete->nexts[layer].load(std::memory_order_acquire), + 1698 std::memory_order_release); + 1699 } + 1700 gc_.Collect(node_to_delete); +``` + +`marked.store(true, release)` at `:1672` is the **logical delete** and the +linearization point: readers skip marked nodes, so the key leaves the set +before any pointer moves, and the check-then-set under the node's own lock +(`:1668-1673`) is what makes "exactly one caller returns true" true. Note +also that the unlink at `:1696-1698` runs **top-down**, the mirror of +insert's bottom-up — remove the shortcuts first, the ground truth last, so +the node is never reachable at a high level but absent at level 0. + +Deletion exists, so reclamation must too — the problem RocksDB dodged in +Step 4. memgraph's answer is **accessor-id GC**, and its doc comment is the +clearest short description of the family: + +```cpp +// src/utils/skip_list.hpp:241-253 — the reclamation scheme, in its own words + 241 /// The skip list doesn't have built-in reclamation of removed nodes (objects). + 242 /// This class handles all operations necessary to remove the nodes safely. + 244 /// Each accessor is given a monotonically increasing ID. When a node is + 245 /// collected (after the skip list has already unlinked it so no new accessor + 246 /// can reach it) the ID of the newest currently-alive accessor is recorded. + 247 /// The node can be freed once that accessor has been destroyed; older ones + 248 /// must have been destroyed too (ReleaseId records a strict prefix of dead ids). + 251 /// alive/dead bits for ~500k accessors. ReleaseId is lock-free (atomic + 252 /// fetch_or). GC walks the blocks to find `live_horizon` (one past the last + 253 /// released id) and frees every pending node whose tag is < live_horizon. +``` + +Compare crossbeam-epoch line by line and it is the same scheme with a +coarser clock: an `Accessor` (`:877`, taking an id at `:881` and releasing +it in `~Accessor` at `:890`) is a *pin*; the accessor id is the *epoch*; the +`live_horizon` is `is_expired`'s `>= 2` threshold. The differences are that +memgraph's ids are per-accessor rather than a global counter, and that GC +runs opportunistically from `insert` when a tall node appears — +`if (top_layer >= kSkipListGcHeightTrigger) gc_.Run();` (`:1333`, with +`kSkipListGcHeightTrigger = 16` at `:69`). A height-16 tower at p = 1/2 +occurs once per 2¹⁶ = 65 536 inserts, so that is a "run maintenance roughly +every 65k inserts" trigger with no counter at all — the same amortisation +crossbeam gets from `PINNINGS_BETWEEN_COLLECT = 128`, sampled from the +existing randomness instead of counted. + +That this list is memgraph's *spine* — vertices, edges, and indexes all live +in these lists — is visible in what got bolted on: `create_chunks` +(accessor wrappers at `:944` and `:955`, implementations at `:1716` and +`:1731`, and `create_chunks_` at `:1780`) splits the list into ranges at +max-height elements for parallel analytics scans. The comparison table (fill it in notes.md): -| | RocksDB | memgraph | +| | RocksDB `InlineSkipList` | memgraph `SkipList` | |---|---|---| -| writers coordinate by | CAS per level | per-node spinlocks | -| readers see partial insert? | yes — per-level linking is independent (fine for a set) | no — fully_linked gate | -| delete | never (tombstones) | marked bit + unlink | -| reclamation | none needed (arena dies at flush) | accessor-id GC | -| failure/retry | re-find level, re-CAS | unlock all, restart | +| p, max height | 1/4, 12 (`:76-78`) | 1/2, 32 (`:61`, `:106-116`) | +| writers coordinate by | one CAS per level (`:1153`) | per-node spinlocks (`:1364`) | +| readers see partial insert? | yes — levels link independently (fine for a set) | no — `fully_linked` gate (`:1398`) | +| readers write shared memory? | never (`:26-27`) | never (they only read the bits) | +| delete | never (tombstones; invariant (1) `:31-33`) | `marked` bit, then unlink (`:1672`, `:1696`) | +| reclamation | none needed (arena dies at flush) | accessor-id GC (`:241-253`) | +| failure/retry | re-find *one level* from prev (`:1162`) | unlock all, restart insert (`:1375`) | +| single-writer escape | `UseCAS=false`, compile-time (`:1134`) | none | ## Where each step lives in the code -Read RocksDB first, then memgraph as the contrast. - -**RocksDB InlineSkipList** — -[`~/repos/rocksdb/memtable/inlineskiplist.h`](https://github.com/facebook/rocksdb) - -- **Steps 3–4**: start at the contract comment (:23) — the guarantee AND - the never-delete assumption in one place; then `CASNext` (:393), the - one-CAS-per-level linking primitive behind `link_at_level` above. -- **Step 5**: `Splice` (:64); `Insert(key, splice, ...)` (:1028) and the - hint variant (:113); `RecomputeSpliceLevels` (:331/:1016); the - `Insert` :908 vs `InsertConcurrently` :913 template pair with the - `UseCAS` flag. +Read RocksDB first, then memgraph as the contrast. Budget ~2 h. In both +files, read the doc comment above the class before the class. -**memgraph SkipList** — -[`~/repos/memgraph/src/utils/skip_list.hpp`](https://github.com/memgraph/memgraph) -— one header holds the list, the accessors, and the GC. +**RocksDB `InlineSkipList`** — `memtable/inlineskiplist.h` at `7c80a5a` -- **Step 6**: Node (:156), `SpinLock` (:163), `fully_linked` (:165), tower - `nexts[0]` (:169); insert (:1335) via `find_node` (:1285); the publish - at (:1398). -- **Step 7**: `marked` (:164); Remove (:1655) with the logical delete at - (:1672); GC at (:244–246), `SkipListGc` (:257), `Collect` (:367), - `Accessor` (:877); `kSkipListGcHeightTrigger` (:69); `create_chunks` - (:817–955). +| Step | What | Line | +|---|---|---| +| 1 | `kMaxPossibleHeight = 32`; the ctor defaults `max_height=12, branching_factor=4` | `:70`, `:76-78` | +| 1 | `RandomHeight` — the p = 1/4 coin | `:559-573` | +| 2 | `Next`/`SetNext`/`CASNext` and their barrier comments | `:379-396` | +| 2 | the tower grows *downward*: `&next_[0] - n` | `:352-356`, `:383` | +| 2 | `NoBarrier_SetNext` / `NoBarrier_Next` | `:399-410` | +| 2 | the orderings themselves | `util/atomic.h:60`, `:108-113`, `:118-121` | +| 3 | the thread-safety contract | `:20-27` | +| 3 | `Insert` — the whole CAS loop | `:1134-1172` | +| 3 | one-level re-find on CAS failure | `:1162-1163`; the reasoning `:1157-1161` | +| 4 | invariant (1): nodes are never deleted | `:29-38` | +| 5 | `Splice` and its invariant | `:340-350`; fwd decl `:64` | +| 5 | `InsertWithHint` / `InsertWithHintConcurrently` | `:111`, `:117` | +| 5 | `RecomputeSpliceLevels` | `:331` (decl), `:1016` (defn) | +| 5 | `Insert` / `InsertConcurrently` / the shared template | `:908`, `:913`, `:1028` | + +**memgraph `SkipList`** — `src/utils/skip_list.hpp` at `8f87f6a`; one header +holds the list, the accessors, and the GC. + +| Step | What | Line | +|---|---|---| +| 1 | `kSkipListMaxHeight = 32`; `gen_height` (p = 1/2, one draw) | `:61`, `:106-117` | +| 6 | `SkipListNode`, `SpinLock`, `marked`, `fully_linked`, `nexts[0]` | `:156`, `:163`, `:164`, `:165`, `:169` | +| 6 | `find_node` — optimistic, no locks | `:1285` | +| 6 | `insert` — lock bottom-up, validate, link, publish | `:1328-1402`; validate `:1370-1372`; publish `:1398` | +| 6 | the paper's bugs, as found by the implementers | `:1358-1359`, `:1369`, `:1388-1389` | +| 7 | `ok_to_delete` and its paper bug | `:1647-1652` | +| 7 | `remove` — mark, validate, unlink top-down, collect | `:1655-1707`; mark `:1672`; unlink `:1696-1698`; collect `:1700` | +| 7 | the GC scheme, described by its author | `:241-255`; `SkipListGc` `:257`; `Collect` `:367` | +| 7 | `Accessor` — the pin | `:877`; `AllocateId` `:881`; release in `~Accessor` `:890` | +| 7 | `kSkipListGcHeightTrigger = 16`, and where it fires | `:69`, `:1333` | +| 7 | `create_chunks` — parallel scan support | `:944`, `:955` (accessors); `:1716`, `:1731`, `:1780` | ## Questions for notes.md @@ -217,16 +605,165 @@ Read RocksDB first, then memgraph as the contrast. You can fill the table from memory and explain what each system's workload allowed it to NOT build. +Answer each before unfolding it. + +- [ ] State each list's height distribution parameter and max height, and + say why they differ. +
Answer + + RocksDB: **p = 1/4, max height 12** — the constructor defaults are + `max_height = 12, branching_factor = 4` (`inlineskiplist.h:76-78`), and + `RandomHeight` (`:559-573`) increases the height "with probability 1 in + kBranching". memgraph: **p = 1/2, max height 32** — + `kSkipListMaxHeight = 32` (`skip_list.hpp:61`) and `gen_height` + (`:106-117`) takes `__builtin_ffs` of a uniform 32-bit draw, which is 1 + with probability ½, 2 with ¼, and so on. + + They differ on *memory*, not on search cost. Expected hops are + `(1/p)·log_{1/p}(n)`, which at n = 50 000 is 4 × 7.80 = 31.2 for p = 1/4 + and 2 × 15.61 = 31.2 for p = 1/2 — identical, because `(1/p)/ln(1/p)` is + flat between those values. But expected tower height is `1/(1−p)`: 1.33 + pointers per node at p = 1/4 against 2.00 at p = 1/2. A size-capped + memtable buys 33% more keys per flush; a long-lived graph store spends + the pointers. + +
+ +- [ ] RocksDB's insert loses a CAS at level 3. What exactly does it redo? +
Answer + + Only level 3, and not from the head. `Insert` calls + `FindSpliceForLevel(key_decoded, splice->prev_[i], nullptr, i, …)` + at `inlineskiplist.h:1162-1163` — the search restarts from the *old* + `prev_[i]` and scans forward. The comment at `:1157-1161` gives the + reason: it is unlikely that many nodes were inserted between prev and + next, and `next_[i]` is known stale so it is useless as a hint. + + It also sets `splice_is_valid = false` when `i > 0` (`:1168-1170`), + because narrowing the bracket at level i can break the `Splice` invariant + at `:341-346` — so the *next* insert recomputes the whole path rather + than trusting a cache that may now be inconsistent between levels. + Levels 0, 1 and 2, already linked, are untouched. + +
+ +- [ ] Estimate the expected number of CAS attempts per insert on this + topic's `scaling` workload, and say what that implies about where the + cost is. +
Answer + + `E[attempts] = 1/(1−p)` where p is the per-attempt failure probability. + The lane runs 16 threads at 19.28 Mops/s with 10% writes ⇒ 1.93 M + inserts/s over a 50 000-key keyspace ⇒ ~39 inserts/s at any one level-0 + link point. With a CAS window of ~5 ns, p ≈ 39 × 5e-9 ≈ **2 × 10⁻⁷** and + E[attempts] ≈ **1.0000002** — one retry per five million inserts. Even if + every insert hit the *same* link point, p ≈ 9.6 × 10⁻³ and E[attempts] ≈ + 1.0097. + + Implication: **retry cost is noise, and "lock-free wastes work on + retries" is the wrong worry.** One cross-core cache-line transfer costs + 38.3 ns on this machine against a 2.28 ns uncontended atomic — a single + bounced line is worth ~17 wasted CAS attempts. Count lines touched, not + attempts avoided. + +
+ +- [ ] Name the two invariants at the top of `inlineskiplist.h` and the + subsystem each one deletes. +
Answer + + (1) at `:31-33`: "Allocated nodes are never deleted until the + InlineSkipList is destroyed." That deletes *all* of memory reclamation — + no epochs, no hazard pointers, no reference counts. It is granted by the + LSM (topic 4): a delete is a tombstone insert, and the whole memtable's + arena is freed in one shot at flush. + + (2) at `:35-38`: node contents other than the next pointers are immutable + once linked. That deletes re-validation — a reader that reaches a node + never has to check whether what it read is still true, which is precisely + the check memgraph's `insert` must perform at `skip_list.hpp:1370-1372`. + Neither invariant is a property of skiplists; both are gifts from the + write path. + +
+ +- [ ] memgraph's `insert` finds preds and succs with no locks held. What + are the three conditions it re-checks after locking, and what breaks if + you skip them? +
Answer + + `skip_list.hpp:1370-1372`, per level: the pred is not `marked`; the + pred's `nexts[layer]` still equals the succ we found; and the succ (if + non-null) is not `marked`. + + Skip the middle one and you lose inserts: a concurrent insert of a + smaller key between pred and succ would be silently overwritten when + `:1394` stores the new node into `pred->nexts[layer]`. Skip the marked + checks and you link into a node that is being unlinked, so the new node + is reachable through a corpse and vanishes when the corpse is spliced + out. The third check is not in the published algorithm at all — the + comment at `:1369` says "Existence test is missing in the paper" — one of + five errata this file records against Herlihy et al. (`:1358-1359`, + `:1369`, `:1388-1389`, `:1648-1649`, `:1679-1680`). + +
+ +- [ ] memgraph triggers GC from `insert`, not from a timer or a counter. + What is the trigger, and how often does it fire? +
Answer + + `if (top_layer >= kSkipListGcHeightTrigger) gc_.Run();` — + `skip_list.hpp:1333`, with `kSkipListGcHeightTrigger = 16` at `:69`. The + trigger is the randomly-drawn height of the node being inserted. + + At p = 1/2, a tower of height ≥ 16 occurs with probability 2⁻¹⁶, so GC + runs roughly **once per 65 536 inserts** — and it needs no counter, no + clock, and no shared state to decide, because it reuses randomness the + insert had to generate anyway. Compare crossbeam's + `PINNINGS_BETWEEN_COLLECT = 128` (`internal.rs:335`), which buys the same + amortisation with an explicit thread-local counter. Both are the same + move: make maintenance a bounded, rare, opportunistic step on an existing + hot path. + +
## References **Papers** -- Herlihy, Lev, Luchangco, Shavit — "A Simple Optimistic Skiplist - Algorithm" (SIROCCO 2007) — the lazy-locking design memgraph implements + +| Paper | What to take | +|---|---| +| Herlihy, Lev, Luchangco, Shavit — *A Simple Optimistic Skiplist Algorithm* (SIROCCO 2007) | the lazy-locking design memgraph implements — read it **next to** `skip_list.hpp`, whose comments record five corrections to it | +| Pugh — *Skip Lists: A Probabilistic Alternative to Balanced Trees* (CACM 1990) | where `(1/p)·log_{1/p}(n)` and the p = 1/4 recommendation come from | **Code** -- [rocksdb](https://github.com/facebook/rocksdb) - `memtable/inlineskiplist.h` — start at the :23 contract comment -- [memgraph](https://github.com/memgraph/memgraph) - `src/utils/skip_list.hpp` — one header holds the list, the accessors, - and the GC + +| File | Lines | What | +|---|---|---| +| `memtable/inlineskiplist.h` (`rocksdb@7c80a5a`) | 20–38 | the contract and the two invariants — start here | +| | 70–78, 559–573 | p = 1/4, max height 12 | +| | 340–350 | `Splice` and its invariant | +| | 352–410 | `Node` — downward tower, barrier-carrying accessors | +| | 1134–1172 | `Insert` — the CAS loop, in full | +| | 908, 913, 1028 | `Insert` / `InsertConcurrently` / the shared template | +| `util/atomic.h` (`rocksdb@7c80a5a`) | 60, 108–121 | what `Load`, `Store`, `CasStrong` actually order | +| `src/utils/skip_list.hpp` (`memgraph@8f87f6a`) | 61, 69, 106–117 | max height, GC trigger, `gen_height` | +| | 156–169 | the node: lock, `marked`, `fully_linked`, tower | +| | 241–255, 257, 367 | the accessor-id GC, described by its author | +| | 877–890 | `Accessor` — the pin, and its release | +| | 1328–1402 | `insert` — and three of the paper's bugs | +| | 1647–1707 | `ok_to_delete` and `remove` — and two more | + +**Measurements** — from this topic's lanes; see `notes.md` and `FINDINGS.md` +row 9. + +| Lane | Figure used above | +|---|---| +| `scaling` | crossbeam `SkipSet` 4.21 → 19.28 Mops/s (1 → 16 threads); global mutex 8.65 → **2.96** | +| `scaling` | keyspace 100 000, 50 000-key preload, 10% writes — the inputs to the retry estimate | +| `false_sharing` | one cross-core line transfer = **38.3 ns**; uncontended padded atomic = 2.28 ns | + +**Cross-topic** — topic 2 for this same RocksDB file read for its *layout*; +topic 4 for the LSM that grants invariant (1); the crossbeam-epoch guide for +the reclamation memgraph hand-rolled; the LWLock guide, Step 4, for what a +production spinlock has to do that a naive one does not. diff --git a/topics/09-concurrency/reading-crossbeam-epoch.md b/topics/09-concurrency/reading-crossbeam-epoch.md index 6736df5..18acdf3 100644 --- a/topics/09-concurrency/reading-crossbeam-epoch.md +++ b/topics/09-concurrency/reading-crossbeam-epoch.md @@ -2,12 +2,17 @@ Lock-free deletion's boss fight is reclamation — when is it safe to `free()` a node some reader might still hold? crossbeam-epoch answers with -three garbage bags and a global epoch counter, and it's the crate your -`concurrent_set.rs` builds on — read it first so `pin()` isn't magic. This -chapter builds the scheme one concept at a time — why `free()` is the hard -part, what a pin promises, where retired memory waits, and what the epoch -clock actually counts — then maps each piece onto the crate's three source -files. +a global epoch counter and a queue of sealed garbage bags, and it's the +crate your `concurrent_set.rs` builds on — read it first so `pin()` isn't +magic. This chapter builds the scheme one concept at a time — why `free()` +is the hard part, what a pin promises, where retired memory waits, what the +epoch clock actually counts, and what the whole thing costs — then maps +each piece onto the crate's three source files. + +Everything below is read at the pinned commit +**`crossbeam-rs/crossbeam@6b7458d`** +(`python3 tools/pinned-source.py ref crossbeam`); `crossbeam-epoch/src/internal.rs` +is 636 lines there. ## The problem in one sentence @@ -15,16 +20,32 @@ A lock-free reader holds a raw pointer to a node another thread just unlinked — free it immediately and you get a use-after-free; never free it and a set retiring 1M nodes/s at 64 bytes each leaks **64 MB every second** — so the whole game is deciding *when* an unlinked node becomes -untouchable by everyone. +untouchable by everyone, without making the reader pay to say so. ## The concepts, step by step ### Step 1 — why free() is the hard part of lock-free -A **lock-free** structure lets readers traverse pointers while holding no -lock at all — that's the whole point (see the skiplists guide: RocksDB's -readers never write shared memory). But it means a deleter cannot know who -is looking: +> **In:** a shared structure whose readers hold no lock. +> **Out:** the exact reason unlinking is easy and freeing is not, plus the +> lock-free / wait-free / obstruction-free vocabulary. + +**Lock-free** is a progress guarantee, not a description of the +instructions used: a structure is lock-free if *some* thread always makes +progress in a bounded number of steps, no matter what any other thread +does — including being descheduled mid-operation. **Wait-free** is the +stronger guarantee that *every* thread makes progress in a bounded number +of its own steps. The distinction is not academic: a CAS retry loop is +lock-free but not wait-free, because one unlucky thread can lose every race +forever while the structure as a whole races ahead. Postgres uses the term +precisely in its own header — `lwlock.c:38-39` claims "wait-free shared +lock acquisition for locks that aren't exclusively locked", because a +shared acquisition is a bounded number of CAS attempts against a bounded +number of contenders. + +Lock-freedom means readers traverse pointers while holding nothing (see the +skiplists guide: RocksDB's readers never write shared memory). But it also +means a deleter cannot know who is looking: ``` reader: p = head.load(Acquire) ───────────► *p ← use-after-free @@ -35,121 +56,369 @@ is looking: Unlinking is safe — it only stops *future* readers from reaching the node. Freeing is not: a *current* reader captured the pointer before the unlink. -Garbage-collected languages solve this with a tracing GC; in Rust/C you +Garbage-collected languages solve this with a tracing GC; in Rust or C you need an explicit protocol, and getting it wrong is the worst bug class -there is (silent memory corruption, not a crash at the fault site). +there is — silent memory corruption, discovered somewhere else entirely, +minutes later. + +This is also where the **ABA problem** lives, and it is worth naming now +because reclamation and ABA are the same wound. A thread reads pointer `A`, +is descheduled; another thread frees `A`, allocates a new node at the same +address, and links it in; the first thread's CAS comparing against `A` +*succeeds* — the value is unchanged, the meaning is not. Any scheme that +delays reuse long enough (which is what epochs do) makes ABA impossible for +free; schemes that don't must smuggle a version tag into spare pointer bits +instead. ### Step 2 — the pin: readers announce themselves for pennies -The protocol's reader side is one call. `epoch::pin()` (default.rs:42) -returns a `Guard` (guard.rs:70), and the promise is: **while a guard -lives, no garbage from the current epoch is freed**. Cost: ~one SeqCst -fence (a CPU ordering instruction that makes the announcement visible to -all cores before any subsequent load — a few ns) + a thread-local counter -bump. Crucially there is no shared-memory write per *pointer* — you pin -once per OPERATION, not per pointer, so a lookup traversing 40 skiplist -nodes pays the fence once. That is what makes lock-free reads effectively -free. - -### Step 3 — retire now, free later: deferred destruction and the bags - -The deleter side: after unlinking, don't free — **retire**. -`Guard::defer_destroy(ptr)` (guard.rs:271) / `defer` (:90 — arbitrary -closures, unchecked variant :189) mean "free this when safe". Where the -retired memory waits: each thread has a `Local` (internal.rs:293) holding -its pinned epoch + a garbage bag, registered in a global intrusive list of -threads. `defer` (:382) drops garbage into the LOCAL bag first (no -contention with other threads), and only when the bag fills is it sealed -into the global queue, tagged with the current epoch — the tag is what -Step 4 needs. - -### Step 4 — the epoch clock: three bags of garbage +> **In:** a thread about to traverse the structure. +> **Out:** `Guard`, and the price of announcing "I am reading" — measured +> in cache lines, not in locks. + +The protocol's reader side is one call. `epoch::pin()` (`default.rs:42`) +returns a `Guard` (`guard.rs:70`), and the promise is: **while a guard +lives, no garbage retired in the current epoch (or the one before) is +freed**. + +What that costs is the interesting part, and it is best read straight out +of `Local::pin`: + +```rust +// crossbeam-epoch/src/internal.rs:403-459 — Local::pin, cfg branches elided + 403 pub(crate) fn pin(&self) -> Guard { + 404 let guard = Guard { local: self }; + 406 let guard_count = self.guard_count.get(); + 407 self.guard_count.set(guard_count.checked_add(1).unwrap()); + 409 if guard_count == 0 { + 410 let global_epoch = self.global().epoch.load(Ordering::Relaxed); + 411 let new_epoch = global_epoch.pinned(); + 446 self.epoch.store(new_epoch, Ordering::Relaxed); + 447 atomic::fence(Ordering::SeqCst); + 451 let count = self.pin_count.get(); + 452 self.pin_count.set(count + Wrapping(1)); + 456 if count.0 % Self::PINNINGS_BETWEEN_COLLECT == 0 { + 457 self.global().collect(&guard); + 458 } + 459 } +``` + +Three facts fall out of those lines, and together they are the reason +epoch reclamation is cheap: + +1. **A nested pin is free.** `guard_count` is a plain `Cell` + (`:306`) — not an atomic — because only the owning thread touches it. If + the thread is already pinned, `pin()` bumps a non-atomic counter and + returns. The whole `if guard_count == 0` body is skipped. +2. **The announcement is a store to the thread's *own* line.** `Local.epoch` + is `CachePadded` (`:317`), so it lives alone on its own + 128-byte line; `Global.epoch` is `CachePadded` too (`:173`). A reader + writes nothing that another reader is also writing, which is the whole + difference from Step 3 of the LWLock guide, where every shared + acquisition contends for one word. +3. **The ordering is one fence.** Line 446–447 is `store(Relaxed)` followed + by `fence(SeqCst)`. + +**Memory ordering, defined once.** `Relaxed` guarantees only atomicity — +no ordering with respect to any other access. `Release` on a store, paired +with `Acquire` on a load of the same location, guarantees that everything +the storing thread wrote *before* the store is visible to a thread that +*sees* the store — that pairing is how you publish an initialised node. +`SeqCst` additionally puts the operation into one total order that all +threads agree on. A `SeqCst` *fence* orders the accesses around it without +naming a location, which is exactly what a pin needs: the announcement must +land before any subsequent pointer load, or a reader could load a pointer +the collector already believes nobody can see. + +`pin()` is a *per-operation* cost, not a per-pointer one. A skiplist lookup +traversing 31 nodes (see the skiplists guide's arithmetic) pays the fence +once. Compare hazard pointers, where each pointer a reader holds is +individually published with a store-plus-fence — 31 fences for the same +traversal. That trade is the entire design space, and Step 7 prices it. + +### Step 3 — retire now, free later: the local bag + +> **In:** a node that has just been unlinked and must eventually be freed. +> **Out:** where it waits, and why it does not touch shared memory on the +> way there. + +The deleter side: after unlinking, don't free — **retire**. +`Guard::defer_destroy(ptr)` (`guard.rs:271`) means "drop and free this +when safe"; `Guard::defer` (`guard.rs:90`, with the unchecked variant at +`:189`) does the same for an arbitrary closure. + +**Epoch-based reclamation** (EBR), stated in one sentence: readers stamp +themselves with the current epoch while they read, retired objects are +stamped with the epoch they were retired in, and an object may be freed +once no reader is stamped with an epoch old enough to have seen it. + +Where the retired memory waits: + +```rust +// crossbeam-epoch/src/internal.rs:382-389 — Local::defer, into the thread-local bag + 382 pub(crate) unsafe fn defer(&self, mut deferred: Deferred, guard: &Guard) { + 383 let bag = self.bag.with_mut(|b| unsafe { &mut *b }); + 385 while let Err(d) = unsafe { bag.try_push(deferred) } { + 386 self.global().push_bag(bag, guard); + 387 deferred = d; + 388 } + 389 } +``` + +`self.bag` is an `UnsafeCell` in the thread's own `Local` (`:303`), +holding at most `MAX_OBJECTS = 64` deferred functions (`:66`). Retiring is +therefore an append to a thread-private array **63 times out of 64**. Only +on the 64th does `push_bag` run, and only then does anything shared get +written: + +```rust +// crossbeam-epoch/src/internal.rs:191-198 — sealing a full bag with the epoch + 191 pub(crate) fn push_bag(&self, bag: &mut Bag, guard: &Guard) { + 192 let bag = mem::replace(bag, Bag::new()); + 194 atomic::fence(Ordering::SeqCst); + 196 let epoch = self.epoch.load(Ordering::Relaxed); + 197 self.queue.push(bag.seal(epoch), guard); + 198 } +``` + +`bag.seal(epoch)` (`:197`) is where the timestamp gets attached — this is +the tag Step 4 reads. Note the fence at `:194` *precedes* the epoch load: +the sealing thread must not read an epoch older than the unlinking it just +performed, or it would under-stamp its own garbage. + +This is amortize-and-batch, the same move as valkey's SPSC batches (topic +7) and redis's incremental rehash (topic 2): make the common case +thread-local and pay the shared cost once per N. + +### Step 4 — the epoch clock, and why the answer is two + +> **In:** a queue of bags, each stamped with a retirement epoch, and a +> global counter. +> **Out:** the exact predicate that decides a bag is freeable, and the +> proof sketch behind its constant. The **global epoch** is a counter E that stands in for time. Every pin -records which epoch the thread pinned in; every sealed bag records which -epoch its garbage was retired in. The freeing rule is then purely -arithmetic — pop bags **≥ 2 epochs old**: +records the epoch the thread pinned in; every sealed bag records the epoch +its garbage was retired in. The freeing rule is then pure arithmetic, and +it is six lines of source: + +```rust +// crossbeam-epoch/src/internal.rs:155-162 — the entire freeing rule + 155 impl SealedBag { + 156 /// Checks if it is safe to drop the bag w.r.t. the given global epoch. + 157 fn is_expired(&self, global_epoch: Epoch) -> bool { + 158 // A pinned participant can witness at most one epoch advancement. Therefore, any bag that + 159 // is within one epoch of the current one cannot be destroyed yet. + 160 global_epoch.wrapping_sub(self.epoch) >= 2 + 161 } + 162 } +``` ``` global epoch: E thread A: pinned @ E ─┐ - thread B: pinned @ E ├─ all @ E ⇒ advance to E+1 + thread B: pinned @ E ├─ all pinned participants @ E ⇒ advance to E+1 thread C: unpinned ─┘ - bags: [E-2: freeable] [E-1: wait] [E: filling] - one thread stuck pinned @ E-1 ⇒ epoch NEVER advances ⇒ unbounded garbage - (the epoch weakness; hazard pointers bound garbage instead) + bags: [E-2 and older: FREE] [E-1: wait] [E: filling] + one thread stuck pinned @ E-1 ⇒ E never advances ⇒ unbounded garbage ``` -Why two epochs of grace and not one: a thread still pinned at E-1 may have -loaded pointers to nodes that were retired at E-1 *after* it pinned — -garbage from E-1 is only provably unreachable once nobody pinned at E-1 -remains. (Question 1 has you construct the exact interleaving.) +**Why the constant is 2, in the comment's own words: "a pinned participant +can witness at most one epoch advancement".** Unpack that. A thread pins at +epoch `p`. While it stays pinned, `try_advance` (Step 5) refuses to move +past `p+1`, because the scan would see this thread pinned at `p ≠ p+1` and +bail. So the global epoch can reach at most `p+1` — one advancement — while +that thread lives. Now take a bag stamped `b` with `E − b ≥ 2`. Any thread +still pinned satisfies `E ≤ p + 1`, hence `p ≥ E − 1 > b`: every live +reader pinned strictly *after* the bag was sealed, so it cannot have loaded +a pointer into that bag. One epoch of grace would not be enough — a thread +pinned at `E−1` may have loaded pointers to nodes retired at `E−1` after it +pinned. (Question 1 has you construct that interleaving explicitly.) + +The comparison is `wrapping_sub`, not `-`: the epoch is a wrapping counter, +so the arithmetic must be too. ### Step 5 — try_advance: the O(threads) scan that moves the clock -Someone has to move E forward, and it's the readers themselves: every -`PINNINGS_BETWEEN_COLLECT = 128` pins (:335, check at :454–456), the -pinning thread calls `collect` (:208) → `try_advance` (:237): scan ALL -registered threads; if anyone is pinned in an OLDER epoch, bail; -otherwise bump the global epoch. +> **In:** a global epoch nobody is advancing. +> **Out:** who advances it, how often, and the one thread behaviour that +> wedges the whole scheme. + +Someone has to move E forward, and it is the readers themselves — on a +cold path, every 128th pin. Look back at `Local::pin` in Step 2, lines +456–457: `if count.0 % Self::PINNINGS_BETWEEN_COLLECT == 0` calls +`Global::collect`, where `PINNINGS_BETWEEN_COLLECT = 128` (`:335`). +`collect` (`:208`) calls `try_advance` and then pops at most +`COLLECT_STEPS = 8` expired bags (`:178`, loop at `:217-225`) — bounded +work, so no single pin can be ambushed by a huge free storm. ```rust -fn try_advance(global: &Global) -> Epoch { - let e = global.epoch.load(Acquire); - for thread in global.registered_threads() { - let local = thread.epoch.load(Acquire); - if local.is_pinned() && local != e { - return e; // a reader still lives in e-1: - } // its pointers may reach that garbage - } - global.epoch.store(e.next(), Release); // everyone at e ⇒ advance; - e.next() // bags two epochs back are free -} +// crossbeam-epoch/src/internal.rs:237-287 — try_advance, sanitizer cfgs elided + 237 pub(crate) fn try_advance(&self, guard: &Guard) -> Epoch { + 238 let global_epoch = self.epoch.load(Ordering::Relaxed); + 239 atomic::fence(Ordering::SeqCst); + 249 for local in self.locals.iter(guard) { + 250 match local { + 251 Err(IterError::Stalled) => { + 255 return global_epoch; + 256 } + 257 Ok(local) => { + 258 let local_epoch = local.epoch.load(Ordering::Relaxed); + 262 if local_epoch.is_pinned() && local_epoch.unpinned() != global_epoch { + 263 return global_epoch; + 264 } + 268 } + 269 } + 270 } + 276 atomic::fence(Ordering::Acquire); + 285 let new_epoch = global_epoch.successor(); + 286 self.epoch.store(new_epoch, Ordering::Release); + 287 new_epoch + 288 } ``` -Here the diagram's weakness becomes concrete: one thread that stays pinned -(blocked on I/O, a wedged scan) fails the scan forever, E never advances, -and garbage grows without bound. **Hazard pointers** (the main alternative -scheme, where readers publish each individual pointer they hold) bound -garbage instead — at a per-pointer cost epochs refuse to pay. +Two things here are easy to get wrong when you write your own, and both +are visible only in the real source: + +- **The per-`Local` loads are `Relaxed` (`:258`), not `Acquire`.** The + ordering is supplied once by the `SeqCst` fence at `:239` and once by the + `Acquire` fence at `:276`, not per-load. Fences instead of per-access + ordering is the standard trick when you are about to touch N locations + and want to pay for ordering once; it is the same instinct as pinning + once per operation instead of once per pointer. +- **A stalled iterator is a bail-out, not a retry** (`:251-256`). The + `Local` list is itself lock-free, so iteration can be disturbed by a + concurrent unregister; rather than spin, `try_advance` returns and leaves + the job to whoever disturbed it. Advancing the epoch is never urgent — + it is pure optimisation — so every failure path here is "give up + cheaply". + +The comment at `:281-284` justifies the unconditional `store` at `:286`: a +racing thread may have advanced it already, in which case this store writes +the same value, because the caller of `try_advance` is *itself* pinned in +`global_epoch` and so the epoch cannot have run two steps ahead. + +**The failure mode is now concrete.** One thread that stays pinned — blocked +on I/O, a wedged scan, a debugger breakpoint — fails the test at `:262` +forever, E never advances, `is_expired` is never true, and garbage grows +without bound. **Hazard pointers**, the main alternative scheme, publish +each individual pointer a reader holds and free anything not currently +published; they bound garbage by construction, at a per-pointer cost +epochs refuse to pay. Neither is strictly better. Epochs are cheap and +unbounded; hazard pointers are bounded and expensive. ### Step 6 — the Rust twist: the borrow checker enforces the protocol -`Atomic` / `Shared<'g, T>`: an atomic pointer whose loads are -lifetime-tied to a guard — the borrow checker enforces "no pointer -outlives its pin" *at compile time*. Unpin while still holding a -`Shared<'g, T>` and the program doesn't compile. This is the Rust-shaped -part that C++ epoch libraries and hazard pointers lack: Step 1's bug class -isn't just detected, it's unrepresentable (question 2). +> **In:** the protocol from Steps 2–5, which a C programmer must follow by +> discipline. +> **Out:** how `Shared<'g, T>` turns Step 1's bug class into a compile +> error. + +`Atomic` (`atomic.rs`) is an atomic pointer whose `load` returns +`Shared<'g, T>` — a pointer whose lifetime `'g` is tied to the `Guard` that +authorised the load. The borrow checker then enforces "no pointer outlives +its pin" *at compile time*: drop the guard while a `Shared<'g, T>` derived +from it is still live and the program does not compile. + +That is the Rust-shaped part C++ epoch libraries and hazard-pointer +libraries lack. In C++ the equivalent mistake — retaining a pointer past +the end of the critical section — compiles cleanly and corrupts memory +under load a week later. Here it is not detected; it is unrepresentable +(question 2). + +The protocol still has one thing the type system cannot check: **duration**. +A guard held for ten seconds is perfectly typed and completely wrong, for +the reason Step 5 just gave. That is what `Guard::repin` (`:329`) and +`repin_after` (`:366`) are for — they unpin and re-pin, giving the +collector a window, and their signatures deliberately invalidate every +`Shared` you were holding, so the compiler forces you to re-load your +pointers afterwards. `Guard::flush` (`:295`) is the other escape hatch: push +the local bag to the global queue now rather than at 64. + +### Step 7 — the costs, worked + +> **In:** the whole scheme. +> **Out:** per-operation numbers on this machine, and the amortisation +> argument that makes them small. + +Price the pieces with this topic's own measured constants (from +`false_sharing`: an uncontended atomic RMW on a thread's own 128 B-padded +line costs **2.28 ns**; moving a line between cores costs **38.3 ns**). + +**Per pin, steady state.** One `Relaxed` store plus one `SeqCst` fence to +the thread's own `CachePadded` line — no other thread writes it, so no +transfer: on the order of **2–3 ns**. Nested pins: a `Cell` increment, +~0 ns. + +**Per retire.** One append to a thread-local array, 63 times in 64. On the +64th, one `SeqCst` fence and one lock-free queue push. + +**Per `try_advance`.** O(T) `Relaxed` loads, one per registered thread. +Each `Local.epoch` is `CachePadded` (`:317`) and was last written by its +owner, so each load is a cross-core transfer: T = 16 threads ⇒ 16 × 38.3 ns +≈ **610 ns**. That sounds enormous until you divide by +`PINNINGS_BETWEEN_COLLECT`: + +``` + 610 ns per try_advance ÷ 128 pins between collects = 4.8 ns per operation + ...and try_advance is #[cold] (:236), so it isn't in the I-cache hot set +``` -### Step 7 — the costs, and where they're amortized +**4.8 ns amortised**, against a `scaling`-lane operation that costs +1/19.28 Mops/s ≈ 52 ns at 16 threads — under 10%, and it *shrinks* as +threads get busier because pins get more frequent while T stays fixed. -The scheme's economics, for your `concurrent_set.rs`: +Now the comparison that decides the design. `try_advance` is O(threads) — +and O(threads) is exactly what hazard pointers pay **per free**, since +freeing a pointer requires checking it against every thread's published +hazard set. Epochs pay it per *advance attempt*, i.e. once per 128 pins, +and each advance can retire an unbounded number of objects. Same asymptotic +scan, radically different divisor. **Amortisation is the whole argument**, +and it is the recurring lesson of this curriculum — the same one behind +RocksDB's splice (skiplists guide, Step 5), postgres's batched wakeups +(LWLock guide, Step 6), and every buffer in topic 7. -- Amortize-and-batch AGAIN: local bag → sealed batch → global queue → - collect every 128 pins. Compare valkey's SPSC batches (topic 7) and - redis incremental rehash (topic 2). -- `try_advance` is O(threads) — that's the cost hazard pointers pay per - FREE; epochs pay it per ADVANCE attempt. Amortization decides winners. -- Read `Guard`'s docs on repinning (`repin`/`repin_after`) — long-running - readers (a full graph scan!) must repin or they wedge the collector. - This is M9's "reader holds a snapshot for 10 s" problem in miniature. +The bill epochs hand you in exchange: unbounded garbage under a stalled +reader, and no way to bound it from inside the library. For M9 that is the +question — FalkorDB queries can run for seconds (question 4). ## Where each step lives in the code Read in this order — API surface first, machinery second; ~1.5 h total: -`default.rs` → `guard.rs` → `internal.rs`. - -- **Step 2**: `epoch::pin()` — default.rs:42; `Guard` — guard.rs:70. -- **Step 3**: `defer_destroy` — guard.rs:271; `defer` — guard.rs:90 - (unchecked variant :189); `Local` — internal.rs:293; the local-bag path - in `defer` — internal.rs:382. -- **Steps 4–5**: `PINNINGS_BETWEEN_COLLECT = 128` — internal.rs:335 - (checked at :454–456); `collect` — internal.rs:208; `try_advance` — - internal.rs:237 (compare it line-by-line with the snippet above). -- **Steps 6–7**: `Atomic` / `Shared<'g, T>` in atomic.rs; the - repinning docs on `Guard` (`repin`/`repin_after`) in guard.rs — read - them, they are the long-reader contract. +`default.rs` → `guard.rs` → `internal.rs`. `internal.rs` rewards being read +bottom-up: `Local::pin` at `:403` is the whole scheme in 60 lines, and +everything above it is support. + +| Step | What | Where | +|---|---|---| +| 2 | `epoch::pin()` — the public entry point | `crossbeam-epoch/src/default.rs:42` | +| 2 | `Guard` and its contract | `guard.rs:70` | +| 2 | `Local::pin` — counter, store, fence, collect | `internal.rs:403-462` | +| 2 | `guard_count` / `pin_count` are plain `Cell`s | `internal.rs:306`, `:314` | +| 2 | `Local.epoch` is `CachePadded` | `internal.rs:317`; `Global.epoch` `:173` | +| 2 | the x86 `lock cmpxchg` hack vs the aarch64 fence | `internal.rs:416-448` | +| 3 | `defer_destroy` / `defer` / `flush` | `guard.rs:271`, `:90`, `:189`, `:295` | +| 3 | `Local::defer` → thread-local bag | `internal.rs:382-389` | +| 3 | `MAX_OBJECTS = 64` | `internal.rs:66` | +| 3 | `push_bag` — fence, load epoch, seal | `internal.rs:191-198` | +| 4 | `SealedBag::is_expired` — the `>= 2` rule | `internal.rs:157-161` | +| 5 | `PINNINGS_BETWEEN_COLLECT = 128`, and its check | `internal.rs:335`, `:456` | +| 5 | `Global::collect`, `COLLECT_STEPS = 8` | `internal.rs:208`, `:178`, `:217-225` | +| 5 | `try_advance` — the scan | `internal.rs:237-288`; the bail `:262-264` | +| 5 | the `Local` list is intrusive and lock-free | `internal.rs:167`, `:292-295` | +| 6 | `Atomic` / `Shared<'g, T>` | `crossbeam-epoch/src/atomic.rs` | +| 6 | `repin` / `repin_after` — the long-reader contract | `guard.rs:329`, `:366` | +| 7 | `CachePadded` is 128 B on x86-64 **and** aarch64 | `crossbeam-utils/src/cache_padded.rs:70-77`, `:87-94` | + +### What to steal for M9 + +- pin once per *operation*, never per pointer — that is the whole cost + advantage over hazard pointers +- keep the per-thread state on its own **128-byte** line; crossbeam does, + and this topic's `false_sharing` lane measures 17.8× for getting it wrong +- make every maintenance path `#[cold]`, bounded (`COLLECT_STEPS`), and + bail-out-happy — reclamation is never urgent +- decide the repin policy *before* you have a ten-second query, not after ## Questions for notes.md @@ -167,11 +436,155 @@ Read in this order — API surface first, machinery second; ~1.5 h total: You can explain, without the source, why `defer_destroy` in epoch E can free at E+2, and what single thread behavior wedges the whole scheme. +Answer each before unfolding it. + +- [ ] Quote the freeing rule as a predicate on two epochs, and justify its + constant. +
Answer + + `global_epoch.wrapping_sub(self.epoch) >= 2` — `internal.rs:160`, the + whole of `SealedBag::is_expired`. + + The constant is 2 because, as the comment at `:158-159` puts it, "a + pinned participant can witness at most one epoch advancement". A thread + pinned at `p` blocks `try_advance` at `:262` for as long as it stays + pinned, so the global epoch cannot exceed `p + 1`. Given a bag sealed at + `b` with `E − b ≥ 2`, every live reader satisfies `p ≥ E − 1 > b` — it + pinned strictly after the bag was sealed, so it never held a pointer into + it. With a threshold of 1 the argument fails: a reader pinned at `E−1` + may have loaded pointers to nodes retired at `E−1` after it pinned. + +
+ +- [ ] Roughly what does a `pin()` cost when the thread is already pinned, + and why? +
Answer + + Essentially nothing — a non-atomic increment. `Local::pin` reads + `guard_count` (`internal.rs:406`), a plain `Cell` at `:306`, and + the entire announcement block is behind `if guard_count == 0` (`:409`). + Only the *outermost* pin does the store-plus-fence at `:446-447`. + + It can be a plain `Cell` because only the owning thread ever touches it — + the same reasoning that lets `pin_count` (`:314`) and the bag (`:303`) be + non-atomic. Everything a reader writes frequently is thread-private; + everything shared is either read-mostly or `CachePadded`. That is the + design, in one sentence. + +
+ +- [ ] `try_advance` is O(threads). Compute what that costs per operation at + 16 threads on this machine, and say why the answer is not alarming. +
Answer + + The scan does one load per registered `Local` (`internal.rs:249-270`). + Each `Local.epoch` is `CachePadded` (`:317`) and was last written by its + own thread, so each load pulls a line across cores — **38.3 ns** on this + machine (`false_sharing`: 40.54 ns contended minus 2.28 ns uncontended). + 16 threads ⇒ ≈ **610 ns** per `try_advance`. + + It is not alarming because of the divisor. `try_advance` runs once per + `PINNINGS_BETWEEN_COLLECT = 128` pins (`:335`, `:456`), so 610 / 128 ≈ + **4.8 ns per operation** — under 10% of the ~52 ns a `scaling`-lane + operation costs at 16 threads — and it is `#[cold]` (`:236`), so it stays + out of the hot instruction path. Hazard pointers pay the same O(T) scan + **per free** rather than per 128 pins; that divisor is the entire + argument for epochs. + +
+ +- [ ] Name the single thread behaviour that wedges the scheme, and the two + API calls that exist to prevent it. +
Answer + + A thread that **stays pinned**: blocked on I/O, running a multi-second + scan, or stopped in a debugger. The check at `internal.rs:262` sees it + pinned in an older epoch and returns early forever, so the global epoch + freezes, `is_expired` (`:160`) is never true, and the queue of sealed + bags grows without bound. Nothing in the library bounds it — that is the + price of not publishing per-pointer. + + `Guard::repin` (`guard.rs:329`) and `Guard::repin_after` + (`guard.rs:366`) exist for exactly this: they unpin and re-pin, giving + the collector a window. Their signatures invalidate every `Shared` you + held, so the compiler makes you re-load your pointers — the protocol + violation you would otherwise commit is a type error. + +
+ +- [ ] `try_advance` loads each thread's epoch with `Relaxed`. Where does + the ordering come from, and why is it done that way? +
Answer + + From two fences, not from the loads: `atomic::fence(Ordering::SeqCst)` at + `internal.rs:239`, before the scan, and `atomic::fence(Ordering::Acquire)` + at `:276`, after it. The per-`Local` load at `:258` is `Relaxed`. + + It is done that way because ordering is paid per *fence*, not per access. + Attaching `Acquire` to each of T loads would emit T ordering constraints + where one suffices — the same economy as pinning once per operation + rather than once per pointer (Step 2), and the same reason `push_bag` + puts its fence at `:194` before a single `Relaxed` epoch load at `:196`. + If you copy this pattern into your own collector, copy the fences too; + dropping them is the classic bug that passes on x86 (where loads are + acquire-ish anyway) and fails on the ARM Mac you are running on. + +
+ +- [ ] Retiring a node touches shared memory how often, and what happens on + the exception? +
Answer + + **Once every 64 retires.** `Local::defer` (`internal.rs:382-389`) pushes + into `self.bag`, an `UnsafeCell` in the thread's own `Local` + (`:303`) that holds `MAX_OBJECTS = 64` entries (`:66`). The `while let + Err(d) = bag.try_push(deferred)` loop at `:385` only takes its body when + the bag is full. + + On the 64th, `push_bag` (`:191-198`) swaps in a fresh bag, executes a + `SeqCst` fence, loads the current global epoch, and pushes the sealed bag + onto the global lock-free queue. The fence comes *before* the epoch load + on purpose: without it the thread could stamp its garbage with an epoch + older than the unlink it just performed, and under-stamped garbage is + garbage freed too early. + +
## References -**Code** -- [crossbeam](https://github.com/crossbeam-rs/crossbeam) — - `crossbeam-epoch/src/`: `default.rs` (pin), `guard.rs` (Guard, - defer_destroy — read its repinning docs), `internal.rs` (Local, - try_advance); ~1.5 h +**Code** (pinned at `crossbeam-rs/crossbeam@6b7458d`) + +| File | Lines | What | +|---|---|---| +| `crossbeam-epoch/src/default.rs` | 42 | `pin()` — the entry point everything else serves | +| `crossbeam-epoch/src/guard.rs` | 70 | `Guard` — read the doc comment, it is the contract | +| | 90, 189, 271 | `defer`, `defer_unchecked`, `defer_destroy` | +| | 295, 329, 366 | `flush`, `repin`, `repin_after` — the long-reader escape hatches | +| `crossbeam-epoch/src/internal.rs` | 66 | `MAX_OBJECTS = 64` — the local bag's capacity | +| | 155–162 | `SealedBag::is_expired` — the entire freeing rule | +| | 165–174 | `Global` — the `Local` list, the bag queue, the padded epoch | +| | 178, 208–226 | `COLLECT_STEPS = 8` and the bounded `collect` loop | +| | 191–198 | `push_bag` — fence, then stamp with the epoch | +| | 237–288 | `try_advance` — the O(T) scan, its bail-outs, its fences | +| | 291–318 | `Local` — what is `Cell`, what is atomic, what is `CachePadded` | +| | 335, 456 | `PINNINGS_BETWEEN_COLLECT = 128`, and where it is checked | +| | 382–389 | `Local::defer` — the thread-local fast path | +| | 403–462 | `Local::pin` — read this first if you read nothing else | +| `crossbeam-epoch/src/atomic.rs` | — | `Atomic`, `Owned`, `Shared<'g, T>` | +| `crossbeam-utils/src/cache_padded.rs` | 70–77, 87–94 | why `CachePadded` is 128 B on aarch64 too | + +Read order: `default.rs` → `guard.rs` → `internal.rs` (bottom-up from +`Local::pin`). About 1.5 h. + +**Measurements** — see `notes.md` for full lane output, `FINDINGS.md` row 9 +for the headline. + +| Lane | Figure used above | +|---|---| +| `false_sharing` | uncontended padded atomic RMW = **2.28 ns**; one cross-core line transfer = **38.3 ns** | +| `false_sharing` | packed vs pad128 = **17.8×**; pad64 vs pad128 = **1.8×** | +| `scaling` | crossbeam `SkipSet` 4.21 → 19.28 Mops/s from 1 to 16 threads (≈52 ns/op at 16) | + +**Cross-topic** — the skiplists guide for the structure this collector is +protecting; the LWLock guide, Step 3, for what a *shared* line costs when +you do not pad; topic 0 §2 for the memory hierarchy the 38.3 ns sits in. diff --git a/topics/09-concurrency/reading-postgres-lwlock.md b/topics/09-concurrency/reading-postgres-lwlock.md index 8124ddc..079fa87 100644 --- a/topics/09-concurrency/reading-postgres-lwlock.md +++ b/topics/09-concurrency/reading-postgres-lwlock.md @@ -2,150 +2,537 @@ lwlock.c is the latch under every buffer, WAL insert, and proc-array scan you met in topics 5–8. One u32 of state, a CAS fast path, and an intrusive -wait queue — read it as the reference answer to "how do I build a fair -rwlock that doesn't melt at 128 cores". This chapter builds the lock one -concept at a time — what a latch even is, the packed state word, the CAS -fast path, the wait queue, and the lost-wakeup race the whole design -orbits — then pins each piece to its line in the file. +wait queue — read it as the reference answer to "how do I build a +reader-writer lock that doesn't melt at 128 cores". This chapter builds the +lock one concept at a time — what a latch even is, the packed state word, +the CAS fast path, the wait queue, the lost-wakeup race the whole design +orbits, and the padding that keeps neighbouring locks off each other's +cache lines — then pins each piece to its line in the file. + +Everything below is read at the pinned commit **`postgres/postgres@701f021`** +(`python3 tools/pinned-source.py ref postgres`). `lwlock.c` is 1939 lines +there. Line numbers in other releases will differ; three of the flag names +changed as recently as PG 17, so check before you trust a number from a +blog post. ## The problem in one sentence -A postgres scan takes and releases a reader-writer lock on **every buffer -it touches** — millions of acquisitions per second per core, each held for -tens of nanoseconds — so a lock whose fast path costs a syscall (~1 µs) or -even one extra contended cache line would cost more than the work it -protects. +A postgres sequential scan takes and releases a reader-writer lock on +**every buffer it touches** — millions of acquisitions per second per core, +each held for tens of nanoseconds — so a lock whose fast path costs a +syscall (~1 µs) or even one extra contended cache line would cost more than +the work it protects. Topic 9's own `scaling` lane measures exactly that +failure in miniature: one global `Mutex` around a `BTreeSet` runs at +**8.65 Mops/s on one thread and 2.96 Mops/s on sixteen** — 2.9× *slower* +with 16× the hardware. ## The concepts, step by step ### Step 1 — a latch: a nanosecond-scale reader-writer lock +> **In:** a shared data structure that many backends read and few mutate. +> **Out:** the vocabulary — lock, latch, reader-writer lock, spinlock, futex +> — and the reason database people insist on the lock/latch distinction. + A **reader-writer lock** (rwlock) admits many simultaneous readers OR one exclusive writer — the right shape for data that is read constantly and -written rarely. A **latch** is an rwlock used to protect a data -structure's *physical* integrity for nanoseconds — unlike topic 8's -transaction locks, which protect *logical* content for seconds and come -with deadlock detection, queues in the lock manager, and recursion. -Latches get none of that (question 3 asks why recursion in particular is -banned). lwlock.c — "lightweight lock" — is postgres's latch: the thing -under every buffer pin, WAL insert, and proc-array scan from topics 5–8. - -### Step 2 — the packed state word (:49, :96–118) +written rarely. + +Database papers split the word "lock" in two, and the split is not +pedantry, it is two different subsystems: + +| | **lock** (topic 8) | **latch** (this file) | +|---|---|---| +| protects | *logical* content — a row, a range | *physical* integrity — a page, a list | +| held for | milliseconds to seconds | tens of nanoseconds | +| deadlock | detected, waits-for graph | avoided by protocol, never detected | +| recursion | allowed | banned (question 3) | +| lives in | the lock manager, hash table of lock tags | one word next to the object | + +The Bw-tree papers state the convention outright: "in this paper, we always +use the term 'lock' when referring to 'latch'" (*Buzz Words*, SIGMOD 2018, +footnote 1 on p. 1). Read "latch-free" and "lock-free" as the same word. + +Two more terms you need before the code. A **spinlock** is a lock whose +waiter burns CPU re-reading the lock word instead of sleeping — correct +only when the hold time is far shorter than the cost of sleeping. A +**futex** ("fast userspace mutex") is the Linux primitive that lets a lock +be a plain word in memory *until* it is contended, at which point one +syscall parks the waiter on that address; `std::sync::Mutex` on Linux is +built on it. Postgres does not use a futex — its waiters park on a per-backend +SysV semaphore (`PGSemaphoreLock(proc->sem)`, `lwlock.c:1269`), because the +queue has to work between *processes* sharing a memory segment, not threads +in one address space. + +**LWLock** — "lightweight lock" — is postgres's latch: the thing under every +buffer pin, WAL insert, and proc-array scan from topics 5–8. "Lightweight" +is relative to the heavyweight lock manager, not to a mutex. + +### Step 2 — the packed state word + +> **In:** a lock that must record "free / N readers / one writer / someone +> is queued" and be updatable atomically. +> **Out:** why all of that fits in one `uint32`, and what each bit is. **CAS** (compare-and-swap) is the atomic CPU instruction "replace this one -word with a new value only if it still equals the value I read" — it can -update exactly ONE word atomically. So the design's first move is to make -the entire lock state fit in one u32: +word with a new value only if it still equals the value I read". It updates +exactly ONE word atomically. Everything else in this file follows from that +constraint: if the whole lock state does not fit in one word, no single +instruction can move the lock from one legal state to another, and you need +a lock to protect your lock. + +So the design's first move is to make the entire state fit in one u32: + +```c +/* lwlock.c:96-108 — the whole lock state, one word */ + 96 #define LW_FLAG_HAS_WAITERS ((uint32) 1 << 31) + 97 #define LW_FLAG_WAKE_IN_PROGRESS ((uint32) 1 << 30) + 98 #define LW_FLAG_LOCKED ((uint32) 1 << 29) + 99 #define LW_FLAG_BITS 3 + 100 #define LW_FLAG_MASK (((1< **In:** the state word from Step 2 and a mode (shared or exclusive). +> **Out:** `LWLockAttemptLock`, and the measured cost of the cache line it +> touches. With the state in one word, acquiring in the uncontended case is one CAS -loop: load state, compute desired (:788 exclusive add, :792 free -check for shared), compare-exchange, retry on spurious/contended failure. -No syscall, no queue touch. THE hot path — every buffer pin in a scan -goes through here. This is what "doesn't melt" means: the common case is -a handful of instructions on a cache line that, for readers of a -read-mostly lock, everyone can keep shared. - -### Step 4 — slow path furniture: the intrusive wait queue - -When the CAS says "held", the thread must wait — and waiting needs a queue -of waiters. An **intrusive list** embeds the list links inside a structure -that already exists instead of allocating a node, and that's what postgres -uses: - -- `LWLockQueueSelf` :1018 — add me to `proclist` (:680 — an intrusive - list of PGPROC entries, no allocation: the waiter structure lives in - the proc array, same idea as intrusive skiplist nodes). -- The wait-list itself is protected by a SPINLOCK with backoff: - :860–880 `perform_spin_delay` — spin, then sleep escalation; stats - count `spin_delay_count` (:246) so contention is observable. - -No allocation on the slow path matters twice: the queue lives in shared -memory (any backend can wake any other), and a lock you take millions of -times per second cannot afford malloc on its unhappy path either. +loop and nothing else — no syscall, no queue touch, no allocation: + +```c +/* lwlock.c:774-808, comments elided — the entire fast path */ + 774 old_state = pg_atomic_read_u32(&lock->state); + 776 /* loop until we've determined whether we could acquire the lock or not */ + 777 while (true) + 778 { + 782 desired_state = old_state; + 784 if (mode == LW_EXCLUSIVE) + 785 { + 786 lock_free = (old_state & LW_LOCK_MASK) == 0; + 787 if (lock_free) + 788 desired_state += LW_VAL_EXCLUSIVE; + 789 } + 790 else + 791 { + 792 lock_free = (old_state & LW_VAL_EXCLUSIVE) == 0; + 793 if (lock_free) + 794 desired_state += LW_VAL_SHARED; + 795 } + 807 if (pg_atomic_compare_exchange_u32(&lock->state, + 808 &old_state, desired_state)) +``` + +`LWLockAttemptLock` returns `false` when it got the lock and `true` when it +must wait (`:817`, `:820`) — read the return as "mustwait", which is what +the caller names it. + +**A claim to unlearn: a shared acquisition does *not* leave the cache line +shared.** It is tempting to say that on a read-mostly lock every reader can +keep the line in the Shared state and nobody pays coherence. That is false +here, twice over. First, a reader *increments* the count, which is a write. +Second, the comment at `:797-806` says the code deliberately swaps in the +value even when it saw the lock as busy, "the reason that we always swap in +the value is that this doubles as a memory barrier". Every acquisition, +shared or exclusive, successful or not, is a `compare_exchange` — an atomic +read-modify-write that must take the line exclusive on the acquiring core +and invalidate every other copy. + +That is the traffic this topic measures. A **cache line** is the unit the +memory system moves and owns — 64 B on x86-64, 64 B (with 128 B pairing, +Step 7) on Apple M-series. The **coherence protocol** (MESI: Modified / +Exclusive / Shared / Invalid) is the hardware rule that a line may be +Modified on at most one core at a time; to write, a core must first get the +others to Invalidate their copies. **False sharing** is when two logically +independent variables land in the same line and therefore fight over that +one ownership token. + +Work it on this machine's numbers. The `false_sharing` lane has 8 threads +each doing 5 000 000 `fetch_add`s on *its own* counter: + +``` + total time per increment what the line is doing + packed (8 × u64) 202.7 ms 40.54 ns one line, 8 owners + pad64 (64 B apart) 20.4 ms 4.08 ns one line each, sort of + pad128 (128 B apart) 11.4 ms 2.28 ns one line each, really + + cost of one ownership transfer = 40.54 − 2.28 = 38.3 ns +``` + +Now price a *contended* LWLock with that. Every acquire and every release +is one RMW on `lock->state`. If two backends on different cores are hitting +the same lock, each pays ~38 ns of coherence on top of the ~2 ns the +instruction would cost uncontended — 19× — and the lock protects work +measured in *tens* of nanoseconds. This is why the file's entire fast path +is one word: not to save memory, but because every extra word touched is +another line to drag across the interconnect. + +The `scaling` lane puts a number on the handoff, too. A global mutex runs +at 8.65 Mops/s single-threaded — 115.6 ns per operation, all of it real +work plus an uncontended lock. At 16 threads it runs at 2.96 Mops/s — +337.8 ns per operation. The extra **222 ns per operation** is pure handoff: +about six line transfers at 38.3 ns each, and comfortably *under* the ~1 µs +a syscall would cost, which tells you most of those handoffs never park the +waiter at all. They spin, bounce the line, and win. That is what a lock +costs when it is doing nothing wrong. + +### Step 4 — the wait list, and the lock inside the lock + +> **In:** a failed `LWLockAttemptLock` — the thread has to wait. +> **Out:** where the waiter record lives, and why guarding the queue costs +> no extra cache line. + +Waiting needs a queue of waiters. An **intrusive list** embeds the links +inside a structure that already exists instead of allocating a node, and +that is what postgres uses: the links live in the waiter's own `PGPROC`. +`LWLockQueueSelf` (`:1018`) pushes the current backend onto +`lock->waiters`. + +Two details make this shared-memory-correct rather than merely tidy: + +- The list links are **backend index numbers, not pointers** + (`src/include/storage/proclist_types.h:28-42`). The proc array is mapped + at a different virtual address in every backend, so a pointer written by + one backend would be meaningless to another. An index into the array is + not. +- There is no allocation anywhere on the slow path. A lock taken millions + of times per second cannot afford `malloc` on its unhappy path either. + +The queue must itself be mutated atomically, and here the file does +something worth stealing: **the wait-list guard is a bit in the same word**, +not a separate spinlock object. + +```c +/* lwlock.c:845-866, LWLockWaitListLock — test-and-test-and-set on LW_FLAG_LOCKED */ + 845 while (true) + 846 { + 847 /* + 848 * Always try once to acquire the lock directly, without setting up + 849 * the spin-delay infrastructure. ... + 850 */ + 852 old_state = pg_atomic_fetch_or_u32(&lock->state, LW_FLAG_LOCKED); + 853 if (likely(!(old_state & LW_FLAG_LOCKED))) + 854 break; /* got lock */ + 855 + 856 /* and then spin without atomic operations until lock is released */ + 857 { + 858 SpinDelayStatus delayStatus; + 860 init_local_spin_delay(&delayStatus); + 862 while (old_state & LW_FLAG_LOCKED) + 863 { + 864 perform_spin_delay(&delayStatus); + 865 old_state = pg_atomic_read_u32(&lock->state); + 866 } +``` + +Line 852 is the atomic attempt; line 865 is a **plain load**. That shape — +one atomic try, then spin on ordinary reads until the word looks free — +is *test-and-test-and-set*, and its whole purpose is coherence traffic. A +naive spinlock re-runs the atomic RMW in the loop, so N spinners generate N +ownership transfers per iteration; here the spinners sit in the Shared +state reading their own cached copy and generate **zero** traffic until the +holder's release invalidates them once. With the 38.3 ns figure from Step +3: 16 spinners × 1000 iterations costs ~610 µs of interconnect the naive +way and essentially nothing this way. + +`perform_spin_delay` (`:864`, defined in `s_lock.c`) escalates: a few +`pg_spin_delay()` pause instructions, then `pg_usleep`, doubling. It also +bumps `spin_delay_count` (`:246`), which `LWLOCK_STATS` builds print — the +file makes contention *observable*, which is the point topic 0 keeps +making about measurement. ### Step 5 — the lost wakeup, and the double-check dance -Here is the race the whole file orbits. Naive slow path: attempt, fail, -enqueue, sleep. But if the holder *releases between your failed attempt -and your enqueue*, it finds an empty queue, wakes nobody — and you sleep -forever on a free lock. That is a **lost wakeup**. - -The fix — **the double-check dance** in `LWLockAcquire` :1150: attempt → -queue self → attempt AGAIN → only then sleep. Without the second attempt, a -release between attempt and enqueue leaves you sleeping forever. -`LWLockDequeueSelf` :1061 handles the "won on the recheck" undo. This -pattern (test, enqueue, re-test) is THE lesson of the file: - -```rust -fn acquire(lock: &LwLock, mode: Mode) { - loop { - if try_cas(lock, mode) { return; } // fast path: one CAS, no queue - queue_self(lock); // slow: enqueue FIRST... - if try_cas(lock, mode) { // ...then attempt AGAIN — - dequeue_self(lock); // a release may have slipped in - return; // between attempt and enqueue - } - sleep_until_woken(); // safe now: our queue entry is - } // visible, releaser must wake us -} +> **In:** a waiter that has failed the fast path and wants to sleep. +> **Out:** the interleaving that would lose its wakeup, and the enqueue → +> re-attempt → sleep ordering that closes it. + +Here is the race the whole file orbits. The naive slow path is: attempt, +fail, enqueue, sleep. Now interleave two backends: + +``` + T1 (waiter) T2 (holder) + ───────────────────────────────── ───────────────────────────────── + LWLockAttemptLock → mustwait + release: state -= LW_VAL_EXCLUSIVE + HAS_WAITERS not set → wake nobody + (T2 leaves; lock is FREE) + LWLockQueueSelf ... + sleep on proc->sem ... + ── sleeps forever on a free lock ── ``` -Why sleeping is safe *after* the recheck: your queue entry is visible -before your final attempt, so any release that happens after your failed -recheck must see `HAS_WAITERS` and wake you. The two orders (enqueue -before re-test; release checks waiters after clearing the lock) interlock -so that no release can fall in the gap. +That is a **lost wakeup**. The fix is to make the queue entry visible +*before* the last check, so that any release able to slip into the gap is +forced to see it: + +```c +/* lwlock.c:1207-1247 — the loop in LWLockAcquire, comments elided */ + 1207 for (;;) + 1208 { + 1215 mustwait = LWLockAttemptLock(lock, mode); + 1217 if (!mustwait) + 1218 { + 1220 break; /* got the lock */ + 1221 } + 1234 /* add to the queue */ + 1235 LWLockQueueSelf(lock, mode); + 1237 /* we're now guaranteed to be woken up if necessary */ + 1238 mustwait = LWLockAttemptLock(lock, mode); + 1240 /* ok, grabbed the lock the second time round, need to undo queueing */ + 1241 if (!mustwait) + 1242 { + 1245 LWLockDequeueSelf(lock); + 1246 break; + 1247 } +``` -### Step 6 — release, batched wakeups, and fairness +The file's own comment at `:1223-1232` states the invariant precisely: "if +we still couldn't grab it, we know that the other locker will see our queue +entries when releasing since they existed before we checked for the lock." + +Read it as an ordering argument, not as luck. `LWLockQueueSelf` sets +`LW_FLAG_HAS_WAITERS` in the state word; `LWLockRelease` reads the state +word as it subtracts. Both touch the same u32. So either T1's +`HAS_WAITERS` write lands before T2's release-read (T2 sees a waiter and +wakes it), or it lands after — in which case T2's decrement landed before +T1's second `LWLockAttemptLock`, and T1 sees a free lock and takes it. +There is no third case, because a single memory location has a single +modification order. The comment at `:1237` — "we're now guaranteed to be +woken up if necessary" — is that argument in five words. + +`LWLockDequeueSelf` (`:1061`) is the undo for the "won on the recheck" +branch, and it is not free: it has to take the wait-list lock and walk the +list. That is fine, because it only runs when a release landed in a window +a few nanoseconds wide. + +Two smaller facts that fall out of the same design: + +- `LWLockQueueSelf` **PANICs** if the backend is already queued on another + lock (`:1028-1029`). One `PGPROC`, one queue link, so one wait at a time. +- The sleep at `:1267-1273` loops. Semaphores can be signalled for other + reasons, so waking is not proof of acquisition; the code re-checks + `proc->lwWaiting` and counts `extraWaits` to put back afterwards. + +### Step 6 — release, batched wakeups, and the fairness you do *not* get + +> **In:** a held lock and a queue of waiters. +> **Out:** what `LWLockRelease` actually promises about ordering — which is +> much less than "arrival order". + +Release is not a CAS loop. It is a single atomic subtract, and then a +decision made from the value that subtract returned: + +```c +/* lwlock.c:1793-1815 — release, then decide whether anyone needs waking */ + 1793 /* + 1794 * Release my hold on lock, after that it can immediately be acquired by + 1795 * others, even if we still have to wakeup other waiters. + 1796 */ + 1797 if (mode == LW_EXCLUSIVE) + 1798 oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_EXCLUSIVE); + 1799 else + 1800 oldstate = pg_atomic_sub_fetch_u32(&lock->state, LW_VAL_SHARED); + 1808 /* + 1809 * Check if we're still waiting for backends to get scheduled, if so, + 1810 * don't wake them up again. + 1811 */ + 1812 if ((oldstate & LW_FLAG_HAS_WAITERS) && + 1813 !(oldstate & LW_FLAG_WAKE_IN_PROGRESS) && + 1814 (oldstate & LW_LOCK_MASK) == 0) + 1815 check_waiters = true; +``` -`LWLockRelease` :1767 → `LWLockWakeup` :904: wakes the queue head; a -released shared lock wakes waiting readers as a batch, and RELEASE_OK -prevents wakeup storms (a woken waiter that hasn't run yet shouldn't -trigger more wakeups). The queue is what buys **fairness**: waiters are -served in arrival order, so a stream of barging readers cannot starve a -queued writer forever — the failure mode a naive CAS-only rwlock hits at -128 cores. +Three things to take from that. + +**The `WAKE_IN_PROGRESS` flag is a wakeup-storm damper.** A backend that +has been signalled is not running yet — it is on a run queue. Without the +flag, every release in that window would take the wait-list lock and signal +the same waiter again. `LWLockWakeup` sets the flag when it dispatches +signals; the woken backend clears it when it loops round to retry +(`:1276`, `pg_atomic_fetch_and_u32(&lock->state, ~LW_FLAG_WAKE_IN_PROGRESS)`). +Older postgres spelled this `LW_FLAG_RELEASE_OK` with the sense inverted — +same mechanism, complemented bit. + +**Wakeups are batched by mode.** `LWLockWakeup` (`:904`) walks the queue +under the wait-list lock and wakes a *run* of waiters: it stops adding +after the first exclusive waiter (`:954-955`) and skips further exclusive +waiters behind it (`:920-921`). So a released lock hands a whole block of +readers through at once, and never signals two writers who can only +serialise. + +**Postgres LWLocks are not FIFO-fair. They allow barging.** The guide you +are replacing said waiters are served in arrival order; the code says +otherwise, in three places: + +1. `LWLockAttemptLock` never inspects `LW_FLAG_HAS_WAITERS` (re-read Step + 3 — there is no such test). A backend arriving fresh takes the lock if + the count is zero, no matter how many backends are queued. +2. `:1793-1795`: "after that it can immediately be acquired by others, even + if we still have to wakeup other waiters." +3. The design NOTE at `:1195-1205` chose this deliberately: handing the + lock to the woken waiter "means a process swap for every lock + acquisition when two or more processes are contending", and since a + backend must be able to "acquire and release the same lock many times + during a single CPU time slice", throughput beats fairness. The + reference is to a pgsql-hackers thread from 29-Dec-01. + +What the queue *does* buy is that a queued waiter is eventually signalled +and re-tries, so no waiter is silently dropped — and among waiters that are +already queued, `LWLockWakeup` walks the list in order +(`src/backend/storage/lmgr/README:29-30` describes this intent). Progress, +not fairness. If you need FIFO under sustained load you build it on top; you +do not get it from here. + +### Step 7 — padding: postgres pads to 128 bytes, and so should you + +> **In:** an array of thousands of LWLocks (one per buffer, per WAL insert +> slot, per lock-manager partition). +> **Out:** why each gets its own line, and why 64 B is the wrong number on +> this machine. + +Steps 2–6 shrank the lock to one word. That creates a new hazard: 512 +LWLocks now fit in one 64 B run of memory, and two *unrelated* locks +sharing a cache line contend as hard as one lock would. That is false +sharing, and postgres avoids it by padding: + +```c +/* src/include/storage/lwlock.h:62-72 — every lock gets a whole line */ + 62 #define LWLOCK_PADDED_SIZE PG_CACHE_LINE_SIZE + 63 + 64 StaticAssertDecl(sizeof(LWLock) <= LWLOCK_PADDED_SIZE, + 65 "Miscalculated LWLock padding"); + 66 + 67 /* LWLock, padded to a full cache line size */ + 68 typedef union LWLockPadded + 69 { + 70 LWLock lock; + 71 char pad[LWLOCK_PADDED_SIZE]; + 72 } LWLockPadded; +``` -## Where each step lives in the code +The actual `LWLock` is 12 bytes (`lwlock.h:41-50`: a `uint16` tranche, a +`pg_atomic_uint32` state, a `proclist_head` of waiters). The union pads it to +`PG_CACHE_LINE_SIZE`, and the comment at `:52-61` gives both reasons — +alignment "ensures that individual LWLocks don't cross cache line +boundaries", and "in some cases, it's useful to add even more padding so +that each LWLock takes up an entire cache line… for example, in the main +LWLock array, where the overall number of locks is small but some are +heavily contended." + +So what is a cache line, according to postgres? + +```c +/* src/include/pg_config_manual.h:217 — with the reasoning at :208-215 */ + 217 #define PG_CACHE_LINE_SIZE 128 +``` + +The comment above it is explicit that this is a chosen upper bound, not a +measurement: "Too small a value can hurt performance due to false sharing, +while the only downside of too large a value is a few bytes of wasted +memory. The default is 128, which should be large enough for all supported +platforms." Postgres spends **10.7× the size of the lock** on padding, on +purpose. + +**Postgres's 128 is right and the folklore 64 is wrong, and this topic +measured it.** Re-read the table in Step 3. Padding the eight counters to +64 B apart — one *nominal* cache line each — takes the lane from 202.7 ms +to 20.4 ms, a 9.9× win, and it still leaves it **1.8× slower** than padding +to 128 B (20.4 ms vs 11.4 ms). Only the 128 B version reaches the +uncontended 2.28 ns/increment. Apple M-series cores prefetch adjacent lines +in **128-byte pairs**, so two variables 64 B apart are still dragged around +together. crossbeam draws the same conclusion in its own source: +`CachePadded` is `#[repr(align(128))]` on x86-64 (Intel's spatial +prefetcher, `crossbeam-utils/src/cache_padded.rs:70-71`) **and** on aarch64 +("big" cores have 128-byte cache lines, `:77`). + +So: "pad to a cache line" is not the rule. The rule is **pad to 128 bytes**, +and if you write 64 you have bought 9.9× of the available 17.8× and left the +rest on the table. -One file — `src/backend/storage/lmgr/lwlock.c`, ~1.5 h. Start at the -state-word definitions, then `LWLockAttemptLock`, then `LWLockAcquire`. +## Where each step lives in the code -- **Step 2**: state-word layout and constants — :49, :96–118; the static - assert — :117. -- **Step 3**: `LWLockAttemptLock` — :764; the exclusive add — :788; the - shared free-check — :792. -- **Step 4**: `LWLockQueueSelf` — :1018; `proclist` — :680; - `perform_spin_delay` — :860–880; `spin_delay_count` — :246. -- **Step 5**: the double-check dance in `LWLockAcquire` — :1150; - `LWLockDequeueSelf` — :1061. -- **Step 6**: `LWLockRelease` — :1767; `LWLockWakeup` — :904. +One file — `src/backend/storage/lmgr/lwlock.c` (1939 lines at `701f021`), +about 1.5 h. Read it in this order, not top to bottom: the file's own +header comment at `:60-74` sketches the four-phase protocol before any of +it, and is worth the two minutes. + +| Step | What | Where | +|---|---|---| +| 1 | latch vs lock, hold times | `lwlock.c:36-58`; `README:1-40` | +| 1 | the waiter's sleep primitive | `lwlock.c:1269` `PGSemaphoreLock` | +| 2 | state-word constants | `lwlock.c:96-108` | +| 2 | the compile-time assertions | `lwlock.c:111-118` | +| 2 | the `LWLock` struct itself | `src/include/storage/lwlock.h:41-50` | +| 3 | `LWLockAttemptLock` | `lwlock.c:764`; exclusive add `:788`; shared check `:792` | +| 3 | "always swap in, it doubles as a barrier" | `lwlock.c:797-806` | +| 4 | `LWLockQueueSelf` | `lwlock.c:1018`; the one-wait PANIC `:1028-1029` | +| 4 | proclist links are indexes | `src/include/storage/proclist_types.h:28-42` | +| 4 | `LWLockWaitListLock`, test-and-test-and-set | `lwlock.c:835`, atomic try `:852`, plain spin `:862-866` | +| 4 | `perform_spin_delay`, `spin_delay_count` | `lwlock.c:864`, `:246` | +| 5 | the double-check dance | `lwlock.c:1207-1247`; the invariant `:1223-1232` | +| 5 | `LWLockDequeueSelf` | `lwlock.c:1061` | +| 5 | the semaphore wait loop | `lwlock.c:1267-1273` | +| 6 | `LWLockRelease` — atomic subtract | `lwlock.c:1767`, `:1797-1800` | +| 6 | the wake decision | `lwlock.c:1812-1814` | +| 6 | `LWLockWakeup`, batching | `lwlock.c:904`, `:916-956` | +| 6 | barging is deliberate | `lwlock.c:1195-1205`, `:1793-1796` | +| 7 | `LWLockPadded` | `src/include/storage/lwlock.h:62-72` | +| 7 | `PG_CACHE_LINE_SIZE 128` | `src/include/pg_config_manual.h:208-217` | +| — | non-recursion: `held_lwlocks` | `lwlock.c:157`, `:167`, `:1301-1302`, `:1778-1789` | ### What to steal for M9 - one-word state + CAS fast path for your HybridLatch-style version latch -- intrusive wait queues (no allocation on the slow path) +- intrusive wait queues (no allocation on the slow path), and links that are + *indexes* if the queue could ever cross an address space +- test-and-test-and-set for any spin loop — spin on plain loads, never on + the atomic +- pad to **128 B**, not 64, and put the number behind one named constant so + a future machine can move it - observable contention counters from day one ## Questions for notes.md @@ -164,10 +551,157 @@ state-word definitions, then `LWLockAttemptLock`, then `LWLockAcquire`. You can draw the full acquire path — fast CAS, queue, recheck, sleep, wakeup — from memory, and name the race each step exists to close. +Answer each before unfolding it. + +- [ ] Name the three flag bits at `701f021` and say what each one is for. +
Answer + + `LW_FLAG_HAS_WAITERS` (bit 31, `:96`) — at least one backend is queued, so + a releaser must take the wait-list lock and look. `LW_FLAG_WAKE_IN_PROGRESS` + (bit 30, `:97`) — a wakeup has been signalled but the woken backend has not + been scheduled yet, so further releases should not signal again; it is set + by `LWLockWakeup` and cleared by the waiter at `:1276`. `LW_FLAG_LOCKED` + (bit 29, `:98`) — the wait-list guard, taken with + `pg_atomic_fetch_or_u32` at `:852`. + + If you answered `LW_FLAG_RELEASE_OK` you read an older tree. That flag + carried the same information with the opposite polarity. + +
+ +- [ ] A backend takes the lock in SHARED mode on a lock nobody else holds. + How many cache lines change ownership, and roughly what does that cost on + this machine? +
Answer + + One line, and it goes exclusive on the acquiring core — not shared. The + acquisition is `pg_atomic_compare_exchange_u32` (`:807`), an atomic + read-modify-write, and it adds `LW_VAL_SHARED = 1` to the count. The + comment at `:797-806` notes the code swaps in a value even when the lock + looked busy, because that doubles as the memory barrier. + + Cost: if the line was already owned by this core, ~2.28 ns — the + `false_sharing` pad128 figure. If another core owned it, ~40.5 ns (the + packed figure), of which **38.3 ns is the ownership transfer**. There is + no configuration in which readers of an rwlock all keep the line Shared; + a counter-based rwlock writes on every read acquisition by construction. + +
+ +- [ ] Write the lost-wakeup interleaving as a two-thread timeline, then say + which single line of `LWLockAcquire` makes it impossible. +
Answer + + T1 calls `LWLockAttemptLock` and is told to wait. Before T1 enqueues, T2 + releases: `state -= LW_VAL_EXCLUSIVE` at `:1798`, reads back an `oldstate` + with `LW_FLAG_HAS_WAITERS` clear (`:1812`), decides `check_waiters = + false`, and leaves. T1 then enqueues and sleeps — on a lock that is free, + with nobody left to wake it. + + Line **1238** — the *second* `LWLockAttemptLock`, after `LWLockQueueSelf` + at `:1235`. Both the enqueue (which sets `HAS_WAITERS`) and the release + (which reads the state as it subtracts) touch the same `uint32`, and a + single location has one modification order: either the release-read sees + `HAS_WAITERS` and wakes T1, or the decrement preceded T1's second attempt + and T1 takes the free lock. The file states this at `:1223-1232`. + +
+ +- [ ] `LWLockWaitListLock` spins. Why does the loop at `:862-866` read with + `pg_atomic_read_u32` instead of retrying the `fetch_or`? +
Answer + + Because retrying the atomic would be the thing it is trying to avoid. A + `fetch_or` is a read-modify-write: it must take the line exclusive, so N + spinners generate N ownership transfers per loop iteration and the holder + — who also needs the line — is starved by the very threads waiting for it. + A plain load leaves every spinner in the Shared state, reading its own + cached copy at L1 speed and generating **zero** interconnect traffic until + the holder's release invalidates them all once. + + That is test-and-test-and-set: one atomic attempt (`:852`), then a + non-atomic wait (`:862-866`). At 38.3 ns per transfer, 16 spinners + ×1000 iterations is the difference between ~610 µs of coherence traffic + and essentially none. + +
+ +- [ ] True or false: a backend that has been queued on an LWLock for a + while is guaranteed to get it before a backend that arrives now. Cite the + code. +
Answer + + **False.** Postgres LWLocks allow barging. `LWLockAttemptLock` (`:764-822`) + tests only `LW_LOCK_MASK` / `LW_VAL_EXCLUSIVE` and never looks at + `LW_FLAG_HAS_WAITERS`, so a fresh arrival takes a free lock over the heads + of the whole queue. `LWLockRelease` says so out loud at `:1793-1795`: + "after that it can immediately be acquired by others, even if we still + have to wakeup other waiters." + + It is a deliberate trade, argued at `:1195-1205`: granting the lock to the + woken waiter "means a process swap for every lock acquisition when two or + more processes are contending", and LWLocks are meant to be taken and + released many times inside one time slice. What you *do* get is that + queued waiters are signalled and re-try (progress, not starvation-freedom), + and that `LWLockWakeup` walks the queue in order among those already on + it. + +
+ +- [ ] Your own lock array is 8 bytes per lock. What alignment do you give + it, and what does the wrong answer cost? +
Answer + + **128 bytes.** Postgres pads *every* LWLock to `PG_CACHE_LINE_SIZE`, which + is **128** (`pg_config_manual.h:217`), via `LWLockPadded` + (`lwlock.h:62-72`) — 128 B of storage for a 12-byte lock. crossbeam's + `CachePadded` is `repr(align(128))` on x86-64 *and* aarch64 + (`cache_padded.rs:70-77`). + + The wrong answers, priced from the `false_sharing` lane: no padding costs + **17.8×** (202.7 ms vs 11.4 ms) because eight counters share one line and + every increment is an ownership transfer. Padding to 64 B — the textbook + "one cache line" — recovers most of it but is still **1.8× slower** than + 128 B (20.4 ms vs 11.4 ms), because M-series cores prefetch lines in + 128-byte pairs, so 64 B-apart variables still travel together. Half the + advice, most of the win, and a residual you will never find by reading + about MESI. + +
## References -**Code** -- [postgres](https://github.com/postgres/postgres) - `src/backend/storage/lmgr/lwlock.c` — ~1.5 h; start at the state-word - definitions (:49, :96–118), then `LWLockAttemptLock` and `LWLockAcquire` +**Code** (pinned at `postgres/postgres@701f021`) + +| File | Lines | What | +|---|---|---| +| `src/backend/storage/lmgr/lwlock.c` | 36–58 | design comment: why one atomic word | +| | 60–74 | the four-phase locking protocol, in the file's own words | +| | 96–118 | the state word, and the assertions that keep it packed | +| | 764–824 | `LWLockAttemptLock` — the whole fast path | +| | 835–880 | `LWLockWaitListLock` — test-and-test-and-set on a flag bit | +| | 904–956 | `LWLockWakeup` — batched, mode-aware | +| | 1018–1105 | `LWLockQueueSelf` / `LWLockDequeueSelf` | +| | 1150–1300 | `LWLockAcquire` — the double-check dance and the sleep | +| | 1767–1830 | `LWLockRelease` — subtract, then decide | +| `src/include/storage/lwlock.h` | 41–50, 62–72 | the struct, and `LWLockPadded` | +| `src/include/storage/proclist_types.h` | 28–42 | queue links are backend indexes, not pointers | +| `src/include/pg_config_manual.h` | 208–217 | `PG_CACHE_LINE_SIZE 128`, with its reasoning | +| `src/backend/storage/lmgr/README` | 1–40 | lock manager vs LWLock, in postgres's own words | + +Read order: `README` → the `lwlock.c` header comment → `:96-118` → +`LWLockAttemptLock` → `LWLockAcquire`. About 1.5 h. + +**Measurements** — every timing above is from this topic's own lanes; see +`notes.md` for the full output and `FINDINGS.md` row 9 for the headline. + +| Lane | Figure | +|---|---| +| `false_sharing` | packed 202.7 ms / pad64 20.4 ms / pad128 11.4 ms → 17.8× and 1.8× | +| `false_sharing` | one ownership transfer = 40.54 − 2.28 = **38.3 ns** | +| `scaling` | global mutex 8.65 → 2.96 Mops/s from 1 to 16 threads (2.9× *slower*) | +| `scaling` | handoff cost = 337.8 − 115.6 = **222 ns/op** | + +**Cross-topic** — topic 0 §2 for the memory hierarchy these numbers sit in; +topic 6 for the buffer-state word that uses the same packing trick; topic 8 +for the lock manager this file is explicitly *not*. diff --git a/topics/10-query-planning/reading-duckdb-optimizer.md b/topics/10-query-planning/reading-duckdb-optimizer.md index 08bed40..43ac263 100644 --- a/topics/10-query-planning/reading-duckdb-optimizer.md +++ b/topics/10-query-planning/reading-duckdb-optimizer.md @@ -1,33 +1,58 @@ # The readable optimizer: DuckDB's pass pipeline and join-order DP DuckDB's `src/optimizer/` is the clearest production optimizer you can -read: ~25 ordered rewrite passes, each verified after it runs, feeding a -DPccp join enumerator with a greedy escape hatch and a cost model that is -just cardinality. Before you open the code, this chapter builds the seven -concepts an optimizer is made of — plan trees, rewrites, pushdown, the -query graph, the join-order DP, cardinality estimation, and the cost -model — one at a time, then hands you the file and line anchors to watch -each one run. +read: a fixed, hand-ordered list of rewrite passes, each re-verified after +it runs, feeding a connected-subgraph join enumerator with two escape +hatches, and a cost model that is essentially cardinality. Before you open +the code, this chapter builds the seven concepts an optimizer is made of — +plan trees, rewrites, pushdown, the query graph, the join-order DP, +cardinality estimation, and the cost model — one at a time, works the +combinatorics on real numbers, then hands you the file and line anchors to +watch each one run. + +Every anchor below is DuckDB at the commit this repo pins, **`6c0c1a68`** +(`tools/pinned-source.py ref duckdb`), quoted with the line numbers the code +occupies in that revision. This topic has no measured lane — its only binary +runs *your* planner — so every number here comes from the pinned source, from +a paper section named on the spot, or from arithmetic performed in the guide +on stated assumptions. ## The problem in one sentence -For a query joining n tables there are Catalan-many tree shapes times n! -orderings — a 20-way join has ~10¹⁸ possible plans — and the best and -worst of them differ by 100×–1000× in runtime, so the optimizer must find -a near-best one in single-digit milliseconds. +For a query joining n tables the number of candidate plans grows faster than +exponentially — 3.6 million left-deep orders at n = 10, 1.3 *trillion* at +n = 15 (Step 5 does the factorials) — and the gap between the best and worst +of them is not academic: on the Join Order Benchmark the *average* ratio +between the worst and the best plan of a query was **101× with no indexes, +115× with primary-key indexes, and 48,120× with foreign-key indexes** +(Leis et al., VLDB 2015, §6.1) — so the optimizer must find a near-best plan +in single-digit milliseconds. ## The concepts, step by step ### Step 1 — the plan tree: logical says WHAT, physical says HOW -A query is compiled into a **plan** — a tree of operators where data -flows from the leaves (table scans) up to the root (the result). The -distinction everything hangs on: +> **In:** nothing yet — this step fixes the two words every later step is +> phrased in. +> **Out:** the logical/physical split, and the statement that optimization is +> a search problem in two phases — which Steps 2-3 and Steps 4-7 then fill in +> separately. + +A query is compiled into a **plan** — a tree of operators where data flows +from the leaves (table scans) up to the root (the result). **Relational +algebra** is the small set of operators that tree is built from: scan +(σ-free base access), *selection* σ (keep rows matching a predicate), +*projection* π (keep columns), *join* ⋈ (pair rows from two inputs matching +a predicate), aggregate, sort. Every SQL query is a tree of these, and every +rewrite in this chapter is an algebraic identity — a rule saying two +different trees compute the same relation. + +The distinction everything hangs on: - a **logical plan** describes *what* to compute — pure algebra: - `Join(A, B, a.x = b.y)` names no algorithm; -- a **physical plan** describes *how* — `HashJoin(build=B, probe=A)` - picks an algorithm and a side. + `Join(A, B, a.x = b.y)` names no algorithm, only the relation it denotes; +- a **physical plan** describes *how* — `HashJoin(build=B, probe=A)` picks an + algorithm, an order of inputs, and therefore a cost. ``` logical (WHAT) physical (HOW) @@ -40,60 +65,184 @@ distinction everything hangs on: ``` One logical plan maps to *many* physical plans, and they are not close in -cost. Optimization is therefore a search problem in two phases: first -transform the logical plan with rewrites that are always safe, then pick -among alternatives with a cost model. Everything below is one of those +cost — the 48,120× of the problem statement is exactly this spread. +Optimization is therefore a search problem in two phases: first transform the +logical plan with rewrites that are always safe (Steps 2-3), then pick among +alternatives with a cost model (Steps 4-7). Everything below is one of those two phases. -### Step 2 — rewrite passes: transformations that never need a cost model +Why it matters: the two phases have completely different failure modes. A +rewrite that is wrong is a *correctness* bug; a cost model that is wrong is a +*performance* bug that ships silently. This topic's whole point is that the +second kind is the one that costs 100×. + +### Step 2 — the pass pipeline: transformations that never need a cost model + +> **In:** the logical plan from the binder, in the shape Step 1 described. +> **Out:** the same plan, algebraically simplified and with filters as low as +> they will go — the input Step 4 turns into a query graph. The pipeline's +> *order* is the output that matters, because Step 3 depends on where in it +> pushdown sits. + +A **rewrite pass** is a whole-plan transformation believed to be always at +least as good as its input, so no cost estimate is needed and you can just run +it. The classic menu: **predicate pushdown** (move a filter as close to the +scan as possible), unused-column elimination, constant folding (`1+1` becomes +`2` at plan time), and turning a cross product plus a filter that mentions both +sides into a real join. + +DuckDB runs these in one fixed, hand-tuned order and — in production — +re-verifies the plan's column bindings after every single one. The wrapper is +seven lines and is where the discipline lives: + +```cpp +// src/optimizer/optimizer.cpp — RunOptimizer, 119-140 + 119 void Optimizer::RunOptimizer(OptimizerType type, const std::function &callback) { + // ... 120-127: bail on interrupt; skip if this optimizer is disabled ... + 128 auto &profiler = QueryProfiler::Get(context); + 129 { + 130 auto optimizer_timer = profiler.StartTimerInternal("optimizer." + StringUtil::Lower(EnumUtil::ToString(type))); + 131 callback(); + 132 } + 133 if (plan) { + 134 Verify(*plan); + 135 } + 136 } + 137 + 138 void Optimizer::Verify(LogicalOperator &op) { + 139 ColumnBindingResolver::Verify(context, op); + 140 } +``` -A **rewrite pass** is a whole-plan transformation that is always at least -as good as the input — no cost estimate needed, so you just run them in -sequence. The classic menu: **predicate pushdown** (move a filter as -close to the scan as possible — a 1%-selective filter applied *before* a -join shrinks every downstream operator's input 100×), unused-column -elimination, constant folding (`1+1` becomes `2` at plan time), and -turning cross products plus filters into real joins. +Line 134 is the one to look at: *every* pass is followed by a full +`ColumnBindingResolver::Verify` of the resulting plan. Line 130 is the second +one — every pass is separately timed, which is why `EXPLAIN ANALYZE` can tell +you which optimizer cost you the millisecond. -DuckDB runs ~25 such passes in one fixed, hand-tuned order, and — in -production — re-verifies the plan's well-formedness after every single -pass. The order itself tells a story: +The list itself lives in `Optimizer::RunBuiltInOptimizers` (:178), reached from +`Optimizer::Optimize` (:441) at :458. **Count them before you trust any summary +of this file: there are 39 `RunOptimizer(OptimizerType::…)` calls between :197 +and :435, covering 37 distinct optimizer types** — `CTE_INLINING` and +`COLUMN_LIFETIME` each run twice, at different points. (An older version of this +chapter said "~25 passes"; the pinned tree says 39.) The order tells a story: ``` - expression rewriter → cte inlining → FILTER PULLUP → FILTER PUSHDOWN → - in-clause → deliminator (decorrelation cleanup) → … - → JOIN_ORDER (:285) → … → unused columns → common subexpressions → - build/probe side (:334) → limit pushdown → TOP_N (:367) + :197 expression rewriter → :200 cte inlining → :212 FILTER PULLUP → + :218 FILTER PUSHDOWN → :236 in-clause → :242 deliminator (decorrelation) → + :272 projection pullup → :278 outer-join simplification → + :285 JOIN_ORDER → … → :309 unused columns → :321 common subexpressions → + :334 build/probe side → :350 limit pushdown → :367 TOP_N → … + → :411 reorder filter → :423 join filter pushdown → :435 type pushdown ``` -Two things to notice. Pullup runs BEFORE pushdown — it looks backwards, -but hoisting filters through outer-join simplifications first lets -pushdown then sink them *further* than either pass alone could. And join -ordering runs mid-pipeline, on a plan already scrubbed of noise. This is -an order-dependent heuristic pipeline, not a fixpoint engine — contrast -DataFusion's run-until-nothing-changes loop and Cascades' memo (both in -this topic's other guides). +Three things to notice. **Pullup runs before pushdown** (:212 before :218) — it +looks backwards, but hoisting a filter out of a subtree first lets pushdown then +sink it into *both* branches of a join it could not previously cross. **Join +ordering runs mid-pipeline** (:285), on a plan already scrubbed of noise; the +comment above it (:283-284) notes that the join-order pass "also rewrites cross +products + filters into joins and performs filter pushdowns", so the pipeline's +two halves are not perfectly separated. And **build/probe side selection is a +separate, later pass** (:334) — Step 7 comes back to what that costs. -### Step 3 — filter pushdown mechanics: a bag of filters sinking until blocked +This is an order-dependent heuristic pipeline, not a fixpoint engine — contrast +DataFusion's run-until-nothing-changes loop and Cascades' memo (this topic's +other guides). -Pushdown is not "move one filter one step". DuckDB's implementation -carries a *bag* of accumulated filter expressions down the tree: at each -operator it asks "can these filters pass through you?", pushes the ones -that can, and deposits the rest as a Filter node right above the blocker. -Per-operator rules decide passability, and **outer joins** are where -correctness bites: a filter on the NULL-padded side of a left join cannot -sink below the join, because rows it would remove are *created* by the -join (as NULL padding), not present in the input. +Why it matters: a fixed order is a fixed set of bugs. If pass B only fires on +output pass A produces, and B runs first, the rewrite is simply never found — +and nothing in the system reports that. -Cost of getting this wrong: a filter evaluated one operator too late -means every intermediate row between the two positions was materialized, -hashed, or copied for nothing — this is the single highest-leverage -rewrite in the pipeline. +### Step 3 — filter pushdown mechanics: a bag of filters sinking until blocked -### Step 4 — the query graph: joins become a graph problem +> **In:** the plan as it stands at :218, the FILTER_PUSHDOWN slot in Step 2's +> list. +> **Out:** the same plan with every filter as deep as it can legally go — and, +> where a filter proves a NULL-padded row impossible, with an outer join +> rewritten to an inner one. That plan is what Step 4 reads relations out of. + +Pushdown is not "move one filter one step". DuckDB's implementation carries a +*bag* of accumulated filter expressions down the tree: at each operator it asks +"can these pass through you?", pushes the ones that can, and deposits the rest +as a `Filter` node right above the blocker. `FilterPushdown::Rewrite` (:106) is +a bare `switch` on operator type, one case per rule, and the default case is the +honest one: + +```cpp +// src/optimizer/filter_pushdown.cpp — the default arm of Rewrite, 148-151, +// and the function it calls, 339-347 + 148 default: + 149 return FinishPushdown(std::move(op)); + 150 } + 151 } + // ... 153-338: PushdownJoin, PushdownProjection, PushdownGet, … one per operator ... + 339 unique_ptr FilterPushdown::FinishPushdown(unique_ptr op) { + 340 // unhandled type, first perform filter pushdown in its children + 341 for (auto &child : op->children) { + 342 FilterPushdown pushdown(optimizer, convert_mark_joins); + 343 child = pushdown.Rewrite(std::move(child)); + 344 } + 345 // now push any existing filters + 346 return PushFinalFilters(std::move(op)); + 347 } +``` -Before ordering joins, DuckDB extracts from the plan a **query graph**: -one node per base relation (table being joined), one edge per join +Line 342 is the argument: an operator this pass does not understand gets a +*fresh, empty* `FilterPushdown` for each child, so the current bag of filters +cannot cross it, and :346 deposits that bag above the blocker. Unknown operator +⇒ nothing sinks through. That is the safe default, and it is why adding an +operator to DuckDB cannot silently break pushdown correctness. + +**Outer joins** are where the interesting case lives. A **left outer join** +emits every left row, padding the right columns with NULL when nothing matched +— so a filter on the right-hand columns cannot simply sink below the join: the +rows it would remove are *created by* the join, not present in its input. But +`PushdownLeftJoin` (`src/optimizer/pushdown/pushdown_left_join.cpp:107`) does +something cleverer than refuse: + +- :132 classifies each filter by which side its bindings come from; +- :141 pushes LEFT-side filters into the left child, unconditionally; +- :152 asks `FilterRemovesNull(...)` of a right-side filter — *would this + predicate reject a NULL-padded row?* If yes, no padded row can survive the + filter anyway, so the outer join is downgraded to an inner join at :154 and + the whole thing is re-run as `PushdownInnerJoin` (:164), where everything + sinks; +- :167 keeps the filters that do *not* remove NULLs, and they stay above the + join. + +So the rule is not "filters never cross an outer join"; it is "a +NULL-rejecting filter converts the join, and then crosses it". + +Why it matters: a filter evaluated one operator too late means every +intermediate row between the two positions was materialised, hashed or copied +for nothing. With a 1%-selective filter and a 1M-row input, that is 990,000 rows +of pure waste per join above it. This is the highest-leverage rewrite in the +pipeline, and the only one in this chapter that needs no estimate to be worth +doing. + +### Step 4 — the fork: the plan becomes a graph *and* a bag of statistics + +> **In:** the rewritten plan from Step 3, at the JOIN_ORDER slot (:285). +> **Out:** *two* datasets, and they go to different places. (a) A **query +> graph** — relations and predicate edges — consumed by the enumerator in +> Step 5. (b) A per-relation `RelationStats` — row counts and distinct counts — +> consumed by the estimator in Step 6. Everything after this point sees one or +> the other, never the plan tree. + +`QueryGraphManager::Build` (`query_graph_manager.cpp:129`) is the fork, and it +is short enough to read as a table of contents: + +- :132 `relation_manager.ExtractJoinRelations(...)` walks the plan and pulls out + the **base relations** — the leaves that will be re-ordered — plus a + `can_reorder` flag; +- :134-137 bails out entirely if there are fewer than two relations or something + said "don't reorder me"; +- :139 `relation_manager.ExtractEdges(...)` turns the collected filter operators + into **edges**: one per join predicate, tagged with the relations on each side; +- :141-145 binds each predicate's endpoints and materialises the hypergraph + (`CreateHyperGraphEdges` :280, which adds each edge in *both* directions at + :284-285). + +The result is a **query graph**: one node per base relation, one edge per join predicate connecting two of them. ``` @@ -104,147 +253,593 @@ predicate connecting two of them. E ... ``` -The graph's *shape* controls how hard ordering is: join orders worth -considering correspond to **connected subgraphs** (subsets of relations -linked by predicates — joining unconnected sets means a cross product, -almost always a disaster). A chain of 10 relations has few connected -subgraphs; a star or clique has exponentially many. Hold that — it -decides when Step 5's exact algorithm gives up. - -### Step 5 — join ordering by dynamic programming, with a greedy escape hatch - -The search uses **dynamic programming** (DP: solve big problems by -combining stored solutions of subproblems) over relation *sets*: the best -plan for a set S of relations must be built from the best plans of two -connected subsets that partition S. So enumerate connected subgraphs -small-to-large, and for each set keep exactly one entry — the cheapest -plan found — in a **memo** (a table keyed by relation set). This is the -DPccp algorithm ("DP over connected complement pairs"), and unlike -Selinger's original left-deep-only search it considers **bushy** trees -(joins whose both inputs are themselves joins) — which graph-pattern -queries especially want. - -DP is exact but its work is proportional to the number of connected -subgraph pairs — fine for chains, explosive for cliques. DuckDB's own -comment at `plan_enumerator.cpp:234`: "when the amount of pairs gets too -large we exit the dynamic programming and resort to a greedy algorithm" — -repeatedly join the pair with the smallest estimated intermediate result. -Greedy is O(n² log n)-ish and can be badly wrong, but a mediocre plan -beats an optimizer that runs longer than the query. +The graph's *shape* is not decoration — it is the input size of Step 5. Join +orders worth considering correspond to **connected subgraphs**: subsets of +relations linked by predicates. Joining an unconnected set means a cross +product, which multiplies cardinalities instead of dividing them, and is almost +always a disaster. A chain of 10 relations has 55 connected subgraphs; a +10-clique has 1,023 (Step 5 counts them). Hold that. + +Why it matters: this fork is why the join-order optimizer cannot see anything +the plan tree knew and the graph does not carry. Correlation between two +columns of different relations, for instance, has no representation in either +output — which is exactly the "join-crossing correlation" the JOB paper (§4.4) +names as the open frontier. + +### Step 5 — join ordering by dynamic programming, with two escape hatches + +> **In:** the query graph from Step 4a — relations and predicate edges only. +> **Out:** one chosen join *order* (a tree over the relations), written into +> the `plans` memo and read back by `QueryGraphManager::Reconstruct` (:302). +> No algorithm choice is made here; that is Step 2's :334 pass. + +**Dynamic programming** (DP) means solving a big problem by combining stored +solutions to smaller subproblems, each solved once. Here the subproblems are +*sets of relations*: the best plan for a set S must be a join of the best plans +for two disjoint subsets whose union is S. A **memo** is the table that stores +them — keyed by relation set, holding the best plan found for it. + +DuckDB's memo is `plans`, and `EmitPair` is the whole memo discipline in one +function: + +```cpp +// src/optimizer/join_order/plan_enumerator.cpp — inside EmitPair, 193-207 + 193 auto &new_set = query_graph_manager.set_manager.Union(left, right); + 194 // create the join tree based on combining the two plans + 195 auto new_plan = CreateJoinTree(new_set, info, *left_plan->second, *right_plan->second); + 196 // check if this plan is the optimal plan we found for this set of relations + 197 auto entry = plans.find(new_set); + 198 auto new_cost = new_plan->cost; + 199 double old_cost = NumericLimits::Maximum(); + 200 if (entry != plans.end()) { + 201 old_cost = entry->second->cost; + 202 } + 203 if (entry == plans.end() || new_cost < old_cost) { + 204 // the new plan costs less than the old plan. Update our DP table. + 205 plans[new_set] = std::move(new_plan); + 206 return *plans[new_set]; + 207 } + // ... 208-222: a tiebreaker for equal-cost plans, to keep LEFT joins LEFT ... +``` + +The line that defines the whole architecture is 205: the memo is keyed by +`new_set` and holds **exactly one plan per relation set**, the cheapest. That is +the design decision postgres does *not* make (see `reading-postgres-optimizer.md` +— it keeps one plan per interesting order too), and Step 7 explains why DuckDB +can afford it. + +Unlike Selinger's original left-deep-only search, this enumerator considers +**bushy** trees — joins whose *both* inputs are themselves joins, as opposed to +**left-deep** trees where the right input of every join is a base relation. +Graph-pattern queries especially want bushy plans. The file names its own +source: the comment at :529-531 says the enumeration is "a straight +implementation of the paper *Dynamic Programming Strikes Back* by Guido +Moerkotte and Thomas Neumann" — i.e. DPhyp, the hypergraph member of the +DPccp family, whose function names (`EmitCSG` :243, `EnumerateCSGRecursive`, +`EnumerateCmpRecursive` :295, `TryEmitPair` :227, `EmitPair` :185) you will +recognise from that literature. A **csg-cmp-pair** is one unit of that search: +a connected subgraph and a connected *complement* subgraph, disjoint, with at +least one edge between them — one candidate join. + +**Now the arithmetic, because this is where exhaustive search dies.** Let n be +the number of base relations. + +``` +left-deep orderings (permutations) = n! +bushy trees over n labelled leaves = (2n-2)! / (n-1)! +DP joins considered, left-deep = Σ(k=2..n) C(n,k)·k = n·2^(n-1) - n +DP memo entries (all subsets) = 2^n - 1 + + n n! bushy DP considered memo + 5 120 1,680 75 31 +10 3,628,800 17,643,225,600 5,110 1,023 +15 1,307,674,368,000 3,497,296,636,753,920,000 245,745 32,767 +``` + +Read the two right-hand columns against the two left-hand ones. At n = 10, DP +looks at 5,110 combinations instead of 3,628,800 orderings — **710× less work** +for a provably optimal left-deep plan. At n = 15 it is 245,745 against +1.3 × 10¹², **5.3 million× less**. That is the whole reason Selinger's 1979 +idea is still in every engine. But look at the memo column: it doubles every +time you add a relation, and *that* is what dies. (Note also that the framing +"Catalan-many shapes × n! orderings ≈ 10¹⁸ for a 20-way join" mixes two +counts: 20! = 2.4 × 10¹⁸ is the *left-deep* count; the bushy count at n = 20 is +4.3 × 10²⁷.) + +DPhyp does better than the table's `2^n` by enumerating only *connected* +subgraph pairs, so the real work depends on the graph's shape. These counts are +enumerated exhaustively rather than estimated: + +``` +csg-cmp-pairs the enumerator must emit, by query-graph shape: + + n chain star clique DuckDB's budget is 10,000 pairs + 5 20 32 90 (plan_enumerator.cpp:233) + 9 120 1,024 9,330 ← clique just fits + 10 165 2,304 28,501 ← clique blows it by 2.85× + 12 286 11,264 261,625 + 15 560 — — +``` + +A chain never troubles it. A 9-relation clique emits 9,330 pairs and completes. +A 10-relation clique emits 28,501 and does not. That is escape hatch one: + +```cpp +// src/optimizer/join_order/plan_enumerator.cpp — TryEmitPair, 227-241 + 227 bool PlanEnumerator::TryEmitPair(JoinRelationSet &left, JoinRelationSet &right, + 228 const vector> &info) { + 229 pairs++; + // ... 230-232: comment on keeping emission going until a final plan is produced ... + 233 if (pairs >= 10000) { + 234 // when the amount of pairs gets too large we exit the dynamic programming and resort to a greedy algorithm + 235 // FIXME: simple heuristic currently + 236 // at 10K pairs stop searching exactly and switch to heuristic + 237 return false; + 238 } + 239 EmitPair(left, right, info); + 240 return true; + 241 } +``` + +Line 233 is the budget and line 229 is the counter it spends. `false` propagates +up through `EnumerateCmpRecursive` (:315) and `SolveJoinOrderExactly` (:375) as a +plain "I gave up". + +Escape hatch two is cruder and fires *first*, on relation count alone: + +```cpp +// src/optimizer/join_order/plan_enumerator.cpp — SolveJoinOrder, 532-543 + 532 void PlanEnumerator::SolveJoinOrder() { + 533 bool force_no_cross_product = Settings::Get(query_graph_manager.context); + 534 auto swap_to_approximate_threshold = + 535 Settings::Get(query_graph_manager.context); + 536 + 537 // first try to solve the join order exactly + 538 if (query_graph_manager.relation_manager.NumRelations() >= swap_to_approximate_threshold) { + 539 SolveJoinOrderApproximately(); + 540 } else if (!SolveJoinOrderExactly()) { + 541 // otherwise, if that times out we resort to a greedy algorithm + 542 SolveJoinOrderApproximately(); + 543 } +``` + +Line 538 is the gate, and `approximate_join_order_threshold` defaults to **12** +(`src/include/duckdb/main/settings.hpp:261-267`) — the *same* number as +postgres's `geqo_threshold`. So the two hatches divide the work: the relation +gate catches every query with 12 or more tables regardless of shape, and the +10,000-pair budget catches the dense graphs below that. A 15-relation *chain* +emits only 560 pairs and would be trivially solvable exactly — but at n = 15 +the :538 gate has already sent it to the heuristic. + +The heuristic is `SolveJoinOrderApproximately` (:398), and the comment at +:400-401 names it: **Greedy Operator Ordering** — start with every base relation +as its own tree, then repeatedly combine the pair with the lowest cost. The +complexity is stated in the code at :407-409, not guessed here: "This is O(r^2) +per step, and every step will reduce the total amount of relations to-be-joined +by 1, so the total cost is O(r^3)". Greedy can be badly wrong, but a mediocre +plan beats an optimizer that runs longer than the query. + +Why it matters: every engine in this topic has this same shape — exact search +with a cliff, and a heuristic behind the cliff. Knowing *where your* cliff is +(12 relations, or 10,000 pairs) is the difference between a planner you can +reason about and one that surprises you in production. ### Step 6 — cardinality estimation: the numbers the whole search runs on -A **cardinality estimate** is the planner's guess at how many rows an -operator will produce — every cost the DP compares is built from these -guesses. DuckDB's estimator is deliberately simple: start from base-table -row counts, and for each join predicate divide by the **distinct count** -(NDV — the number of different values in the column), i.e. assume every -value is equally frequent (**uniformity**). Columns linked by equality -predicates are grouped into equivalence sets so the same denominator -isn't applied twice. No histograms in the join-order path. And when a -predicate is something the estimator can't reason about, it falls back to -a constant `DEFAULT_SELECTIVITY` guess — DuckDB's version of postgres's -famous 0.005 (a **selectivity** is the fraction of input rows a predicate -keeps). - -Why it matters: these estimates are the *only* signal ranking 10¹⁸ -candidate plans, and on correlated real data they can be off by 10⁴–10⁶ -(see reading-how-good-optimizers.md — errors multiply up the tree). - -### Step 7 — the cost model is just cardinality (Cout) - -With estimates in hand, the **cost model** ranks plans. DuckDB's is one -line: the cost of a join is its estimated *output* cardinality plus its -children's costs — i.e. the sum of all intermediate result sizes, known -in the literature as **Cout**. No CPU weights, no IO constants. Steps 6 -and 7 in one function: - -```rust -fn cost(plan: &Node) -> f64 { - match plan { - Scan(t) => t.estimated_rows, - Join(l, r, preds) => { - let mut card = rows(l) * rows(r); - for p in preds { - card /= distinct_count(p) as f64; // total denominators from - } // matching equivalence sets - card + cost(l) + cost(r) // output size + children: - } // cardinality IS the cost model - } -} -``` - -This is defensible — VLDB'15 showed that with *true* cardinalities even a -trivial Cout model picks plans within ~2× of optimal — and damning: -cardinality IS the cost model, so cardinality error is plan error, -one-for-one. Note also what it ignores: it ranks join *orders* only; -hash-join build-vs-probe side is chosen by a separate later pass -(Step 2's pipeline, the `build/probe side` entry). +> **In:** Step 4b's per-relation `RelationStats`, plus the edges from Step 4a. +> **Out:** one `double` per relation set — the estimated row count — memoised in +> `relation_set_2_cardinality`. Step 7 turns these, and nothing else, into cost. + +**Cardinality** is the number of rows a relation or subplan contains; a +**cardinality estimate** is the planner's guess at it before running anything. +**Selectivity** is the fraction of its input a predicate keeps, so +`cardinality_out = selectivity × cardinality_in`. A **histogram** is a +per-column summary — value ranges and the row count in each — that lets an +engine estimate a range predicate's selectivity without scanning; DuckDB's +join-order path does **not** use one. + +What it uses instead is a numerator over a denominator: + +```cpp +// src/optimizer/join_order/cardinality_estimator.cpp — the whole estimate, +// 889-911 (the leading comment names the method's source) + 889 // Cardinality is calculated using logic based on + 890 // https://blobs.duckdb.org/papers/tom-ebergen-msc-thesis-join-order-optimization-with-almost-no-statistics.pdf + // ... 891-895: INNER equality predicates use transitive equality classes; composite + // same-pair equalities can apply an FK/PK cap; disconnected predicate + // subgraphs are merged by cross product; LEFT/SEMI/ANTI adjust the numerator ... + 896 template <> + 897 double CardinalityEstimator::EstimateCardinalityWithSet(JoinRelationSet &new_set) { + 898 double result; + 899 auto it = state->relation_set_2_cardinality.find(new_set); + 900 if (it != state->relation_set_2_cardinality.end()) { + 901 result = it->second.cardinality_before_filters; + 902 } else { + // ... 903-906: comments on zero cardinalities and semi/anti numerators ... + 904 auto denom = GetDenominator(new_set); + 907 auto numerator = GetNumerator(denom.numerator_relations); + 908 result = numerator / denom.denominator; + 909 state->relation_set_2_cardinality[new_set] = CardinalityHelper(result); + 910 } + 911 return ApplyOrFilterSelectivities(new_set, result); + 912 } +``` + +Line 908 is the entire model. `GetNumerator` (:337-350) is a plain product of +the base relations' row counts — line 347, `numerator *= cardinality_before_filters`. +`GetDenominator` (:880) walks the edges and multiplies in one **total domain** +per equality-equivalence class: the estimated distinct-value count of the +columns that equality predicates have linked together. Grouping into classes is +what stops `a.x = b.x AND b.x = c.x` from dividing by the same domain twice. + +So, symbol by symbol, for a set S of relations joined by equality predicates: + +``` + card(S) = Π |R| ÷ Π tdom(E) + R∈S E∈classes(S) + + |R| rows in base relation R, after its own local filters + tdom(E) the total domain of equality class E — the number of distinct + values the columns in E are estimated to hold +``` + +**Worked example**, on this topic's own three-table schema (the one +`experiments/src/explain.rs` plans and `notes.md` asks you to predict): +`users` 10,000 rows, `orders` 50,000 rows, `items` 200,000 rows; +`users.id` has 10,000 distinct values, `orders.id` has 50,000. + +``` +{users, orders} joined on users.id = orders.user_id + numerator = 10,000 × 50,000 = 500,000,000 + tdom = 10,000 (distinct users.id) + card = 500,000,000 / 10,000 = 50,000 + +{orders, items} joined on orders.id = items.order_id + numerator = 50,000 × 200,000 = 10,000,000,000 + tdom = 50,000 (distinct orders.id) + card = 10,000,000,000 / 50,000 = 200,000 +``` + +Both are right — each foreign key matches exactly one parent, so the join +preserves the child's row count. Now add the two filters `users.city = 7 AND +users.age = 30`, with 100 distinct cities and 50 distinct ages. **Independence** +— the assumption that predicates on different columns are unrelated, so their +selectivities multiply — gives: + +``` +sel(city = 7) = 1/100 = 0.01 +sel(age = 30) = 1/50 = 0.02 +sel(both), independent = 0.01 × 0.02 = 0.0002 +|users| after filters = 10,000 × 0.0002 = 2 rows + +{users, orders} now = (2 × 50,000) / 10,000 = 10 rows +{orders, items} unchanged = 200,000 rows +``` + +Ten against two hundred thousand: the filters flip which pair the enumerator +joins first, by four orders of magnitude. That is the prediction `notes.md` +asks you to commit to before running `explain`. + +Now break independence, which is what real data does. Suppose city 7 is a +university town where half the users are 30. Then `sel(age=30 | city=7) = 0.5`, +not 0.02, and: + +``` +true sel(both) = 0.01 × 0.5 = 0.005 +true |users| after filters= 10,000 × 0.005 = 50 rows +estimate = 2 rows truth = 50 rows +error factor = 50 / 2 = 25× +``` + +A 25× underestimate from *one* correlated pair of columns, in a schema with +three tables. The JOB paper measures what this does at six joins +(`reading-how-good-optimizers.md`); the short version is that the factors +multiply. + +One constant deserves correcting, because it is the most-repeated wrong claim +about this file. DuckDB does have a `DEFAULT_SELECTIVITY`, but it is +**0.2** — `src/include/duckdb/optimizer/join_order/relation_statistics_helper.hpp:55`, +`static constexpr double DEFAULT_SELECTIVITY = 0.2;` — **not** postgres's 0.005. +It is 40× larger, and it is not a general "unknown predicate" fallback either. +It is applied in exactly two places: `relation_statistics_helper.cpp:259-262`, +where a base table has non-equality filters and no equality filter to estimate +from, and `cardinality_estimator.cpp:915-918`, where a relation set is covered +by an OR filter. Everything else goes through the numerator/denominator above. + +Why it matters: these estimates are the *only* signal ranking the plans Step 5 +enumerates, and the guess above is arithmetic over two statistics per column. +There is no histogram anywhere in this path. + +### Step 7 — the cost model is (almost) just cardinality + +> **In:** Step 6's estimate for the combined set, plus the already-computed +> costs of the two child plans from the Step 5 memo. +> **Out:** one `double` — the number `EmitPair` compares at :198-203 to decide +> what stays in the memo. This closes the loop: Steps 5, 6 and 7 run +> interleaved, once per emitted pair. + +The **cost model** is the function that turns estimated cardinalities into one +comparable number per plan. **Cout** is the classic minimal one: the cost of a +plan is the sum of the sizes of all its intermediate results, and nothing else — +no CPU weights, no IO constants. DuckDB's is Cout plus one correction, and the +file is 50 lines long: + +```cpp +// src/optimizer/join_order/cost_model.cpp — ComputeCost, the whole model, 37-48 + 37 // Currently cost of a join mostly factors in the cardinalities. + 38 // LEFT joins need an explicit RHS input component because their output cardinality preserves the LHS, + 39 // which otherwise makes early LEFT joins over large RHS inputs look almost free. + 40 double CostModel::ComputeCost(DPJoinNode &left, DPJoinNode &right, JoinRelationSet &combination, + 41 const vector> &possible_connections) { + 42 auto join_card = cardinality_estimator.EstimateCardinalityWithSet(combination); + 43 auto join_cost = join_card; + 44 if (query_graph_manager.GetPredicateModel().HasLeftJoinPredicates()) { + 45 join_cost += GetLeftJoinInputCost(cardinality_estimator, possible_connections); + 46 } + 47 return join_cost + left.cost + right.cost; + 48 } +``` + +The line that carries the argument is 47: *this join's output cardinality, plus +the two children's costs*. The recursion bottoms out at zero — a leaf +`DPJoinNode` is constructed with `cost(0)` (`join_node.cpp:10`) and +`InitLeafPlans` sets `join_node->cost = 0` explicitly at +`plan_enumerator.cpp:521` — so base-table scans are *free* in this model and a +plan's cost is precisely the sum of its intermediate join outputs. Cout, +exactly. Lines 44-46 are the one exception, and the +comment at :38-39 explains it honestly: a LEFT join's output cardinality equals +its left input's, so without a term for the right input, joining a huge RHS +early looks free. (A summary that says "cost is cardinality, that's it" is +right about :43 and wrong about :45; the pinned tree has both.) + +Run it on the worked example above, on the filtered `users`: + +``` +plan A: (users ⋈ orders) ⋈ items + intermediate {users,orders} = 10 ← Step 6 + final {u,o,i} = 10 × 200,000 / 50,000 = 40 + Cout = 10 + 40 = 50 + +plan B: (orders ⋈ items) ⋈ users + intermediate {orders,items} = 200,000 ← Step 6 + final {u,o,i} = 40 + Cout = 200,000 + 40 = 200,040 +``` + +200,040 / 50 = **4,001× more expensive**, and the only thing separating the two +plans is which intermediate the model was told about. That is the sense in which +cardinality *is* the cost model here: change the estimate, change the plan, and +nothing else in the file gets a vote. + +This is defensible — the JOB paper found that in a main-memory setting a +deliberately trivial cost function, given *true* cardinalities, produced query +runtimes 34% faster than PostgreSQL's 4,000-line model in geometric mean +(Leis et al. §5.4) — and damning: cardinality error is plan error, one for one. + +Note also what this model does *not* rank. It compares join *orders*. Whether a +hash join builds on the left or the right input is decided later, by the +`BUILD_SIDE_PROBE_SIDE` pass at `optimizer.cpp:334`, using column-lifetime +information the join-order pass did not have. + +Why it matters: every knob you might want to tune in this optimizer — a CPU +weight, an IO constant, a sort penalty — does not exist. The only thing you can +improve is Step 6. ## Where each step lives in the code -Read in this order: `optimizer.cpp`, then `filter_pushdown.cpp`, then -the `join_order/` subdirectory — the payoff. - -- **Step 2 — the pipeline** (`optimizer.cpp`): `Optimizer::Optimize` - runs the ~25 passes; every one is wrapped in `RunOptimizer` (:119) - which profiles it and `Verify`s (:134–139) column bindings afterward. - Read :197–367 top to bottom for the order — pullup at :212, pushdown - at :218, JOIN_ORDER at :285, build/probe side at :334, TOP_N at :367. -- **Step 3 — pushdown** (`filter_pushdown.cpp`): `Rewrite` (:106) - dispatches on operator type → per-operator pushdown (`PushdownFilter` - :112); non-pushable operators get a fresh child `FilterPushdown` - (:130–137) — the bag of filters sinking until something blocks. - Look at `pushdown/` for the per-operator rules (pushdown_left_join - etc. — the outer-join correctness cases). -- **Step 4 — the query graph** (`join_order/`): - `query_graph_manager.cpp` / `relation_manager.cpp` extract relations - + edges (predicates) from the plan. -- **Step 5 — the DP** (`join_order/plan_enumerator.cpp`): - `SolveJoinOrderExactly` :375 — DPccp-style dynamic programming: - enumerate connected subgraphs, `EnumerateCmpRecursive` :295, - `TryEmitPair` :227 / `EmitPair` :185 keep the best plan per - relation-SET (the memo). The escape hatch comment at :234; - `SolveJoinOrderApproximately` :398 is the greedy - (smallest-intermediate-result-first); `SolveJoinOrder` :532 picks - between them. -- **Step 6 — estimation** (`join_order/cardinality_estimator.cpp`): - `EstimateCardinalityWithSet` :897 — product of base cardinalities × - per-predicate selectivities, with **total denominators** from matching - equivalence sets; unknown predicates get `DEFAULT_SELECTIVITY` (:917 — - DuckDB's 0.005 moment). Base stats from - `relation_statistics_helper.cpp`. -- **Step 7 — the cost model** (`join_order/cost_model.cpp`): - `ComputeCost` :40 — cost = estimated cardinality of the join output + - children costs. That's it. +Read in this order: `optimizer.cpp`, then `filter_pushdown.cpp`, then the +`join_order/` subdirectory — the payoff. All line numbers are `6c0c1a68`. + +| Step | File | Lines | What | +|---|---|---|---| +| 2 | `src/optimizer/optimizer.cpp` | 119-140 | `RunOptimizer` — profiles each pass and `Verify`s the plan after it | +| 2 | `src/optimizer/optimizer.cpp` | 178 | `RunBuiltInOptimizers` — the pass list's own function | +| 2 | `src/optimizer/optimizer.cpp` | 197-435 | the 39 `RunOptimizer` calls, in order: pullup :212, pushdown :218, JOIN_ORDER :285, build/probe :334, TOP_N :367 | +| 2 | `src/optimizer/optimizer.cpp` | 441-472 | `Optimizer::Optimize` — extensions, then `RunBuiltInOptimizers` at :458 | +| 3 | `src/optimizer/filter_pushdown.cpp` | 106-151 | `Rewrite` — the dispatch table, one arm per operator | +| 3 | `src/optimizer/filter_pushdown.cpp` | 339-347 | `FinishPushdown` — the safe default: fresh child pushdown, filters deposited | +| 3 | `src/optimizer/pushdown/pushdown_left_join.cpp` | 107-171 | side classification :132, LEFT push :141, `FilterRemovesNull` → INNER :152-164 | +| 4 | `src/optimizer/join_order/query_graph_manager.cpp` | 129-147 | `Build` — relations :132, edges :139, hypergraph :145 | +| 4 | `src/optimizer/join_order/query_graph_manager.cpp` | 280-287 | `CreateHyperGraphEdges` — each predicate added in both directions | +| 4 | `src/optimizer/join_order/relation_manager.cpp` | 267, 674 | `ExtractJoinRelations`, `ExtractEdges` — where the fork's two datasets are built | +| 5 | `src/optimizer/join_order/plan_enumerator.cpp` | 185-225 | `EmitPair` — the memo; :205 keeps one plan per relation set | +| 5 | `src/optimizer/join_order/plan_enumerator.cpp` | 227-241 | `TryEmitPair` — the 10,000-pair budget at :233 | +| 5 | `src/optimizer/join_order/plan_enumerator.cpp` | 243, 295, 375 | `EmitCSG`, `EnumerateCmpRecursive`, `SolveJoinOrderExactly` — the DPhyp enumeration | +| 5 | `src/optimizer/join_order/plan_enumerator.cpp` | 398-527 | `SolveJoinOrderApproximately` — Greedy Operator Ordering, O(r³) per its own comment at :407-409 | +| 5 | `src/optimizer/join_order/plan_enumerator.cpp` | 529-543 | the DPhyp citation, and `SolveJoinOrder`'s two escape hatches | +| 5 | `src/include/duckdb/main/settings.hpp` | 261-267 | `approximate_join_order_threshold`, default `"12"` | +| 6 | `src/optimizer/join_order/cardinality_estimator.cpp` | 889-912 | `EstimateCardinalityWithSet` — `numerator / denominator` at :908 | +| 6 | `src/optimizer/join_order/cardinality_estimator.cpp` | 337-350, 880-887 | `GetNumerator` (product of base rows), `GetDenominator` (total domains) | +| 6 | `src/include/duckdb/optimizer/join_order/relation_statistics_helper.hpp` | 55 | `DEFAULT_SELECTIVITY = 0.2` | +| 7 | `src/optimizer/join_order/cost_model.cpp` | 37-48 | `ComputeCost` — Cout at :43/:47, the LEFT-join correction at :44-46 | +| 7 | `src/optimizer/join_order/join_node.cpp` | 9-11 | a leaf `DPJoinNode` is built with `cost(0)` — base scans are free | + +Suggested route: `optimizer.cpp:441` → `:178` and read the list top to bottom +→ `filter_pushdown.cpp:106` → `pushdown_left_join.cpp:107` → then +`join_order/`: `query_graph_manager.cpp:129` → `plan_enumerator.cpp:532` (start +at the *end*, where the two hatches are) → `:375` → `:185` → +`cardinality_estimator.cpp:897` → `cost_model.cpp:40`. ## Questions for notes.md -1. Why does pullup-then-pushdown beat pushdown alone? Find one operator - in `pullup/` where hoisting first enables a deeper sink. -2. The DP keeps one best plan per relation set. What plan property does - that discard that Selinger kept (hint: interesting orders) — and why - does DuckDB get away with it (what physical op dominates)? -3. Exact→greedy threshold: what workload shape triggers it — star schema - (one fact, k dims) or chain? Count connected subgraphs for both at - n=10. -4. Cost = output cardinality only: no distinction between hash-join - build sides at this stage (that's the later BUILD_SIDE_PROBE_SIDE - pass :334). What does splitting order-choice from side-choice lose? -5. M10: a Cypher chain `(a)-[:R]->(b)-[:S]->(c)` is a chain query graph - over edge relations. Which DuckDB piece maps to anchor-node selection - — the enumerator or the cardinality estimator? +1. Why does pullup-then-pushdown beat pushdown alone? Find one operator in + `src/optimizer/pullup/` where hoisting first enables a deeper sink, and name + the file. +2. The memo keeps one best plan per relation set (`plan_enumerator.cpp:205`). + What plan property does that discard that postgres keeps (hint: interesting + orders) — and why does DuckDB get away with it? Which physical operator's + dominance is the answer? +3. Step 5's table says a 9-clique emits 9,330 csg-cmp-pairs and a 10-clique + 28,501. At what n does a *star* schema cross the 10,000-pair budget, and + does the :538 relation gate fire before or after that? Work it from the + table. +4. Cost is output cardinality only (`cost_model.cpp:43`), so build-vs-probe + side is chosen later, at `optimizer.cpp:334`. What does splitting + order-choice from side-choice lose? Construct a case where the best order + under Cout has the worse build side. +5. M10: a Cypher chain `(a)-[:R]->(b)-[:S]->(c)` is a chain query graph over + edge relations. Which DuckDB piece maps to anchor-node selection — the + enumerator (`plan_enumerator.cpp`) or the cardinality estimator? Justify + with the fork in Step 4. + +## Takeaway + +The pipeline is 39 verified passes in a fixed order; the join search is DPhyp +with a 12-relation gate and a 10,000-pair budget behind it; the cost model is +Cout plus one LEFT-join correction; and the estimator is a product of row +counts over a product of distinct counts, with no histogram in sight. For M10: +copy the fork (Step 4) and the budget (Step 5), and expect every plan bug you +have to be an estimate bug. ## Done when -You can list the pass order from memory (coarse buckets), and explain -DPccp + the greedy fallback + the cardinality formula in three sentences. +Answer each before unfolding it. + +- [ ] You can name the pass order in coarse buckets, and say why filter pullup runs before filter pushdown. + +
Answer + + Coarse buckets, in the order `RunBuiltInOptimizers` (`optimizer.cpp:178`) + runs them: expression-level simplification (:197), CTE handling (:200), + filter movement (pullup :212, pushdown :218), subquery decorrelation + (deliminator :242), structural simplification (projection pullup :272, outer + join simplification :278), **join ordering** (:285), column pruning (:309) + and common-subexpression work (:321), physical-ish choices (build/probe side + :334), then limit/top-n (:350, :367) and a final cleanup tail out to :435. + Thirty-nine `RunOptimizer` calls, 37 distinct types — `CTE_INLINING` and + `COLUMN_LIFETIME` each appear twice. + + Pullup runs first because it *un*-blocks pushdown. A filter stranded inside + one branch of a subtree can often be hoisted to a point where it applies to + both branches; once hoisted, the pushdown pass at :218 can sink it into each + of them, deeper than it started. Running pushdown alone would leave it where + it was. The pipeline's order is expert knowledge encoded as a list, which is + also its weakness: a rewrite that only becomes possible after a later pass is + simply never found, and nothing reports that. + +
+ +- [ ] You can explain DPhyp, the greedy fallback, and both escape hatches — with the actual thresholds. + +
Answer + + The enumerator walks *connected subgraph / connected complement* pairs of the + query graph, smallest first, and for each pair calls `EmitPair` + (`plan_enumerator.cpp:185`), which builds the join and keeps it in the `plans` + memo at :205 only if it beats what is already stored for that relation set. + One plan per set, cheapest wins. The file cites *Dynamic Programming Strikes + Back* (Moerkotte and Neumann) at :529-531 as its source. + + Two hatches, in the order they fire. `SolveJoinOrder` (:532) first compares the + relation count against `approximate_join_order_threshold`, default **12** + (`settings.hpp:261-267`), at :538 — twelve or more tables goes straight to + greedy, whatever the shape. Otherwise it tries exact, and `TryEmitPair` + (:227) aborts the moment its `pairs` counter reaches **10,000** (:233), + returning `false` all the way up so :540 falls through to greedy at :542. + + The greedy is `SolveJoinOrderApproximately` (:398): Greedy Operator Ordering, + which starts with every relation as its own tree and repeatedly merges the + cheapest pair. Its own comment at :407-409 states the complexity — O(r²) per + step, O(r³) overall. Concretely: a 9-relation clique emits 9,330 csg-cmp-pairs + and is solved exactly; a 10-relation clique emits 28,501 and is not; a + 15-relation chain emits only 560 but is sent to greedy anyway, by the :538 + gate. + +
+ +- [ ] You can write DuckDB's cardinality formula from memory and run it on three tables, with and without the independence assumption. + +
Answer + + `card(S) = (Π over R in S of |R|) / (Π over equality classes E of tdom(E))` — + `cardinality_estimator.cpp:908`, with `GetNumerator` (:337-350) supplying the + product of base row counts at :347 and `GetDenominator` (:880) supplying one + total domain per equality-equivalence class. `|R|` is R's row count after its + own local filters; `tdom(E)` is the estimated number of distinct values the + columns linked by equality in class E hold. + + On this topic's schema — `users` 10,000, `orders` 50,000, `items` 200,000, + 10,000 distinct `users.id`, 50,000 distinct `orders.id`: + `{users, orders}` = 10,000 × 50,000 / 10,000 = 50,000, and + `{orders, items}` = 50,000 × 200,000 / 50,000 = 200,000. Add + `users.city = 7 AND users.age = 30` with 100 cities and 50 ages: independence + multiplies the selectivities, 0.01 × 0.02 = 0.0002, so `users` becomes 2 rows + and `{users, orders}` becomes 2 × 50,000 / 10,000 = 10 — four orders of + magnitude below `{orders, items}`, which is why the filters flip the join + order. + + Break independence and the same arithmetic lies. If city 7 is a town where + half the users are 30, the true conditional selectivity is 0.5, not 0.02: the + true filtered `users` is 10,000 × 0.01 × 0.5 = 50 rows against an estimate of + 2, a factor of 25 from a single correlated column pair. Under Cout the two + candidate orders here differ by 200,040 / 50 ≈ 4,001×, so an error of that + size is entirely capable of picking the wrong one. + +
+ +- [ ] You can state what DuckDB's `DEFAULT_SELECTIVITY` actually is, and where it is used — without repeating the postgres number. + +
Answer + + It is **0.2**, declared at + `src/include/duckdb/optimizer/join_order/relation_statistics_helper.hpp:55`. + Postgres's `DEFAULT_EQ_SEL` is 0.005; the two are not the same constant and + DuckDB's is 40× larger. Anything that calls DuckDB's "its version of + postgres's famous 0.005" is repeating a claim the source does not support. + + It is also not a general fallback for predicates the estimator cannot reason + about. It appears in exactly two places. At + `relation_statistics_helper.cpp:259-262`, a base relation that has + non-optional filters but no equality filter to estimate from has its + cardinality set to `max(base_cardinality × 0.2, 1)`. At + `cardinality_estimator.cpp:915-918`, `ApplyOrFilterSelectivities` multiplies a + relation set's estimate by 0.2 once per OR filter covering it. Everything + else in the join-order path goes through `numerator / denominator` at :908. + +
+ +- [ ] You can say why the join-order search cannot fix a bad estimate, and where in the code the two are wired together. + +
Answer + + Because search and estimation are not two independent quality knobs — the + search *consumes* the estimate as its only ranking signal. `ComputeCost` + (`cost_model.cpp:40`) calls `EstimateCardinalityWithSet` at :42, adds the + children's costs at :47, and returns; `EmitPair` (`plan_enumerator.cpp:185`) + compares that number at :198-203 and keeps the smaller. There is no other + input. A perfectly exhaustive search over wrong numbers returns the plan that + is optimal *for the wrong numbers*. + + The JOB paper measured exactly this separation. With true cardinalities + injected, exhaustive dynamic programming produced the optimal plan at the + median, the 95th percentile and the maximum — 1.00 / 1.00 / 1.00 in their + Table 3. The same exhaustive DP driven by PostgreSQL's estimates, on the same + queries with foreign-key indexes, scored 1.66 median and **186,367× at the + maximum**. Nothing about the search changed. Meanwhile the *heuristics* given + true cardinalities cost only 1.02-1.20 at the median. Search quality is worth + tens of percent; estimate quality is worth five orders of magnitude. + +
## References **Code** -- [duckdb](https://github.com/duckdb/duckdb) — `src/optimizer/`: - `optimizer.cpp` (the pass pipeline, read :197–367 top to bottom), +- [duckdb](https://github.com/duckdb/duckdb) at `6c0c1a68` — `src/optimizer/`: + `optimizer.cpp` (the pass pipeline, read :178-438 top to bottom), `filter_pushdown.cpp` + `pushdown/`, and `join_order/` - (`plan_enumerator.cpp`, `cardinality_estimator.cpp`, `cost_model.cpp`); - ~2 h + (`plan_enumerator.cpp`, `cardinality_estimator.cpp`, `cost_model.cpp`); ~2 h. +- Tom Ebergen, *Join Order Optimization with (Almost) No Statistics* (MSc + thesis) — the method `cardinality_estimator.cpp:889-890` cites for its + numerator/denominator model. +- Moerkotte and Neumann, *Dynamic Programming Strikes Back* (SIGMOD 2008) — the + enumeration algorithm `plan_enumerator.cpp:529-531` says it implements. + +**Papers** +- Leis, Gubichev, Mirchev, Boncz, Kemper, Neumann — *How Good Are Query + Optimizers, Really?* (VLDB 2015). Used here for three figures, each from a + named section: §6.1 (average worst-to-best plan ratio 101× / 115× / 48,120× + by index configuration), §5.4 (a trivial main-memory cost model with true + cardinalities beats PostgreSQL's own by 34% in geometric mean), and Table 3 + (exhaustive DP scores 1.00/1.00/1.00 with true cardinalities and + 1.66/169/186,367 with PostgreSQL's). Read in full via + `reading-how-good-optimizers.md`. diff --git a/topics/10-query-planning/reading-how-good-optimizers.md b/topics/10-query-planning/reading-how-good-optimizers.md index 72ddf02..2bb7c40 100644 --- a/topics/10-query-planning/reading-how-good-optimizers.md +++ b/topics/10-query-planning/reading-how-good-optimizers.md @@ -1,193 +1,660 @@ # Cardinality is the whole ballgame: the JOB audit -The humbling paper. Leis et al. (VLDB '15) built the Join Order Benchmark -(JOB) — 113 queries over IMDB, REAL correlated data instead of TPC-H's -synthetic uniformity — and audited every layer of the classical optimizer -stack. The verdict reorders this whole topic: cardinality error dwarfs -cost-model error dwarfs search-space limits. Before the paper, this -chapter builds the layers being audited and the estimator being indicted, -step by step — then hands you the reading route. +The humbling paper. Leis, Gubichev, Mirchev, Boncz, Kemper and Neumann +(VLDB '15) built the Join Order Benchmark (JOB) — 113 queries over IMDB, real +correlated data instead of TPC-H's synthetic uniformity — and audited every +layer of the classical optimizer stack. The verdict reorders this whole topic: +cardinality error dwarfs cost-model error dwarfs search-space limits. Before +the paper, this chapter builds the layers being audited and the estimator being +indicted, step by step — then hands you the reading route. + +**Every number below is cited to the section, figure or table of the paper it +came from**, and every number *not* in the paper is arithmetic done here on +assumptions stated here. This guide is written against the VLDB proceedings +version (PVLDB vol. 9 no. 3, pp. 204–215); if you have the extended VLDB +Journal 2018 version, section numbers shift. **Topic 10 has no measured lane** — +its benchmark harness measures only your own code — so nothing here is a +measurement taken on this machine, and none of these figures appear in +`FINDINGS.md`. ## The problem in one sentence -Every cost-based optimizer ranks plans using guessed row counts, and on -real correlated data those guesses are off by 10⁴–10⁶ after a handful of -joins — so the question the paper asks is: of the optimizer's three parts -(estimates, cost model, search), which one is actually responsible for -bad plans? +Every cost-based optimizer ranks plans using guessed row counts; on real +correlated data those guesses go wrong in a way that gets *systematically +worse the more joins you add* — the paper measures PostgreSQL estimates wrong +by 10× or more for 16% of one-join subplans, 32% at two joins and 52% at three +(§3.2) — so the question the paper asks is: of the optimizer's three parts +(estimates, cost model, search), which one is actually responsible for slow +queries? ## The concepts, step by step ### Step 1 — the three claims an optimizer makes -A classical cost-based optimizer is three separable components, each -making its own claim: - -1. **cardinality estimation** — a guess at how many rows each subplan - produces ("this filter keeps ~500 of 6M rows"); -2. a **cost model** — a formula turning cardinalities into a comparable - cost number ("a join producing 500 rows from these inputs costs X"); -3. **plan search** — an algorithm (DP, greedy, genetic) that explores - the space of join orders and tree shapes and keeps the cheapest. +> **In:** a SQL query and a set of catalog statistics. +> **Out:** a decomposition of "the optimizer" into three separately testable +> components, which is the paper's whole experimental design. + +Two terms first, because everything below is stated in them. + +- A **logical plan** is an expression in **relational algebra** — the small + algebra of operators over sets/bags of tuples: select (σ, a filter), project + (π, choose columns), join (⋈), aggregate, union. It says *what* result you + want. `σ_year>2000(title) ⋈ movie_info` is a logical plan. +- A **physical plan** picks an *algorithm* for each logical operator: this join + is a hash join, that scan is an index scan, this aggregate sorts first. Two + physical plans with wildly different runtimes can compute the identical + logical result. The optimizer's job is choosing among them. + +A classical cost-based optimizer is three separable components, each making its +own claim: + +1. **cardinality estimation** — **cardinality** is simply the number of rows a + (sub)expression produces. Estimation is a guess at that number for each + subplan, before running it: "this filter keeps ~500 of 6M rows". +2. a **cost model** — a formula turning estimated cardinalities into a single + comparable number, intended to be monotone in runtime: "a hash join + producing 500 rows from these inputs costs X". It is a *proxy*: you cannot + run every candidate plan to time it, so you score them. +3. **plan search** — an algorithm (dynamic programming, greedy, genetic) that + walks the space of join orders and tree shapes, scores each candidate with + the cost model, and keeps the cheapest. ``` estimates ──► cost model ──► search ──► chosen plan - (guessed (formula (which of 10^18 - row counts) over guesses) candidates wins) + (guessed (formula (which of the + row counts) over guesses) candidate plans wins) ``` -Each layer consumes the previous one's output, so a failure anywhere -poisons everything downstream — and until this paper, nobody had measured -*which* layer fails in practice. That is the entire contribution. +Each layer consumes the previous one's output, so a failure anywhere poisons +everything downstream — and until this paper, nobody had *measured* which layer +fails in practice on real data. That is the entire contribution. Everything +after Step 4 is machinery for answering it. + +Why it matters: this decomposition is the reason the paper can assign blame at +all. If you only measure end-to-end runtime, a slow query tells you nothing +about which of the three to go fix. ### Step 2 — how every system estimates joins: three assumptions -The estimator all audited systems run (postgres and three commercial -engines) rests on the same three assumptions. **Uniformity**: every value -of a column is equally frequent, so an equality predicate keeps 1/NDV of -the rows (NDV = number of distinct values in the column). -**Independence**: predicates don't correlate, so combined selectivity -(the fraction of rows a predicate keeps) is the product of individual -selectivities. **Containment**: in a join, every value of the smaller -side's key set appears in the larger side's. In code: +> **In:** per-column catalog statistics — row counts, distinct-value counts, +> and **histograms** (a table of value ranges with the row count falling in +> each, so that a range predicate can be answered by summing buckets). +> **Out:** a single estimated cardinality per subplan, and three named +> assumptions to hold responsible when it is wrong. + +**Selectivity** is the fraction of its input rows a predicate keeps: a +selectivity of 0.01 on a 1M-row table means an estimated 10,000 rows out. The +estimator that all five audited systems run (PostgreSQL and three unnamed +commercial engines, "DBMS A/B/C", plus HyPer) rests on the same three +assumptions, which the paper names in §2.3: + +- **Uniformity** — every value of a column is equally frequent (once you are + inside a histogram bucket), so an equality predicate keeps 1/NDV of the rows, + where NDV is the number of distinct values in the column. +- **Independence** — predicates do not correlate, so a conjunction's + selectivity is the *product* of the individual selectivities. +- **Principle of inclusion** (the paper's phrase; you will also see + "containment") — in a join between key sets, every value of the smaller + domain appears in the larger one, so nothing is lost to non-matching keys. + +Those three assumptions collapse into one formula, which the paper prints in +§2.3 for PostgreSQL's equi-join: + +``` + |T1 ⋈_{x=y} T2| = |T1| · |T2| / max( dom(x), dom(y) ) + + |T| cardinality of T (its row count) + x, y the join columns + dom(x) the number of distinct values of x (postgres's n_distinct) +``` + +An implementation is a handful of lines: ```rust -// the estimator every audited system runs, and why it under-shoots -fn estimate_join_card(tables: &[Table], preds: &[EquiPred]) -> f64 { - let mut card: f64 = tables.iter().map(|t| t.rows as f64).product(); - for p in preds { - card /= p.ndv_left.max(p.ndv_right) as f64; // uniformity: 1/NDV - } // each predicate applied INDEPENDENTLY — on correlated data the - card // true overlap is larger, so factors compound toward zero -} -``` - -Cheap to compute (a few stats per column), and exactly right on uniform, -independent data. Real data is neither. - -### Step 3 — errors are multiplicative, so they grow exponentially with joins - -The standard error metric is **q-error**: max(estimate/truth, -truth/estimate) — a q-error of 100 means off by 100× in *some* direction. -Because each join's estimate is built by multiplying the previous -estimate by another guessed factor, per-predicate errors *compound*: a -2× error per predicate is 2⁶ = 64× after six joins if you're lucky, far -worse when correlations align. The paper measures exactly this: median -q-error degrades **exponentially with join count, reaching 10²–10⁴ at 6 -joins** across all tested systems — and **underestimation dominates**, -because independence multiplies selectivities toward zero while -correlated predicates actually overlap (city = 'Paris' AND country = -'France' is not sel × sel; it's ~sel). - -Why underestimation is the dangerous direction: an optimizer told "only -3 rows will come out of this" happily picks a nested-loop join — which -then runs 10⁴× more iterations than promised. +// ILLUSTRATION — not quoted from any repo in this course. This is the §2.3 +// formula above, transcribed. The real implementations are postgres +// src/backend/utils/adt/selfuncs.c (eqjoinsel) and duckdb +// src/optimizer/join_order/cardinality_estimator.cpp:908, which divides a +// product of base cardinalities by a product of per-equivalence-class domains. + 1 fn estimate_join_card(tables: &[Table], preds: &[EquiPred]) -> f64 { + 2 let mut card: f64 = tables.iter().map(|t| t.rows as f64).product(); + 3 for p in preds { + 4 card /= p.ndv_left.max(p.ndv_right) as f64; // uniformity: 1/NDV + 5 } // each predicate applied INDEPENDENTLY — on correlated data the + 6 card // true overlap is larger, so factors compound toward zero + 7 } +``` + +Cheap to compute (a few scalars per column), and exactly right on uniform, +independent data. Real data is neither. The paper measures how wrong the +*base-table* half of this is before any join is involved, over 629 base-table +selections (§3.1, Table 1), reporting q-error (defined in Step 3) quantiles: + +``` + Table 1 (§3.1) — q-error of base-table selection estimates + median 90th 95th max + PostgreSQL 1.00 2.08 6.10 207 + DBMS A 1.01 1.33 1.98 43.4 + DBMS B 1.00 6.03 30.2 104,000 + DBMS C 1.06 1,677 5,367 20,471 + HyPer 1.02 4.47 8.00 2,084 +``` + +Read that table twice. The *median* is essentially perfect everywhere — half of +all base-table estimates are dead on. It is the tail that kills you, and the +tail is already four to five orders of magnitude wide before a single join has +happened. + +Why it matters: every later error in this paper is this error, multiplied. + +### Step 3 — q-error, and why errors compound with join count + +> **In:** an estimated cardinality and the true cardinality for the same +> subplan. +> **Out:** one scale-free error number per subplan, and the argument for why +> that number's *distribution* widens exponentially in the number of joins. + +The standard metric is **q-error**. The paper defines it in §3.1 as "the factor +by which an estimate differs from the true cardinality. For example, if the true +cardinality of an expression is 100, the estimates of 10 or 1000 both have a +q-error of 10." Formally: + +``` + q-error(est, true) = max( est/true , true/est ) (always ≥ 1) +``` + +It is deliberately symmetric and multiplicative: being 10× low and 10× high are +both "q-error 10", and — unlike absolute or relative error — it does not care +whether the true value is 100 or 100 million. That is what makes it summable +across a whole workload. + +**Work it on a real pair.** The paper's footnote 6 (§3.2) reports that for one +JOB two-join query with a **true cardinality of 2,600**, PostgreSQL produced +estimates of **3, 9, 128 or 310** — *for the identical query*, varying only the +textual order of relations in `FROM` and predicates in `WHERE`: + +``` + true = 2600 + est = 3 q-error = max(3/2600, 2600/3) = 2600/3 = 866.67 + est = 9 q-error = max(9/2600, 2600/9) = 2600/9 = 288.89 + est = 128 q-error = max(128/2600, 2600/128) = 2600/128 = 20.31 + est = 310 q-error = max(310/2600, 2600/310) = 2600/310 = 8.39 +``` + +Every one of these is an *underestimate*, and the best of them is still 8× low. +Note also what the spread means: two syntactically identical queries get +estimates 100× apart, so the estimator is not even a function of the query's +semantics. + +**Now the compounding, worked on three concrete predicates.** Take a `title` +table and three filters. The row count and the three selectivities below are my +assumptions, chosen to have IMDB's *shape*; they are not measured figures from +the paper. + +``` + |title| = 2,500,000 rows (assumption) + + p1: production_year BETWEEN 2000 AND 2005 sel(p1) = 0.20 + p2: country = 'FR' sel(p2) = 0.05 + p3: genre = 'Drama' sel(p3) = 0.25 + + INDEPENDENCE (what the estimator computes): + sel(p1 ∧ p2 ∧ p3) = 0.20 × 0.05 × 0.25 = 0.0025 + estimate = 2,500,000 × 0.0025 = 6,250 rows + + CORRELATED (what the data actually is), written as conditionals: + P(p2) = 0.05 French titles + P(p3 | p2) = 0.50 French cinema skews to drama (0.25 overall) + P(p1 | p2 ∧ p3) = 0.30 these cluster in the 2000s (0.20 overall) + true sel = 0.05 × 0.50 × 0.30 = 0.0075 + truth = 2,500,000 × 0.0075 = 18,750 rows + + q-error = 18,750 / 6,250 = 3.0 +``` + +Two mild correlations — one factor of 2, one of 1.5 — produced a 3× error on a +*single* table. Now push it through joins. Cardinalities multiply, so their +errors multiply too, and if each of six joins carries an independent 3× error +in the same direction: + +``` + 3^6 = 729× composed error after six joins + 2^6 = 64× the same arithmetic with a mild 2× error per join +``` + +That is arithmetic on my assumption of a constant per-join factor, not a +measurement. **The paper's actual measurement** is Figure 3 (§3.2), which +boxplots over 100,000 estimates from all five systems, grouped by the number of +joins in the subplan (0 through 6). Read three things off it: + +1. The vertical axis spans **underestimation by 10⁸ and overestimation by 10⁴** + — the error is wildly asymmetric, and it is asymmetric toward *under*. +2. The boxes get taller monotonically with join count. In the paper's words, + the errors "grow exponentially… as the number of joins increases", and "for + all systems we routinely observe misestimates by a factor of 1000 or more". +3. The quantified progression for PostgreSQL: **16%** of one-join estimates are + wrong by ≥10×, **32%** at two joins, **52%** at three. DBMS A is better and + fails the same way: 15%, 25%, 36%. + +Be precise about what grows. It is **not** the median — the medians stay near 1 +and drift downward. It is the *width of the distribution*, and it widens +asymmetrically downward. A guide that says "median q-error reaches 10²–10⁴ at +six joins" is misreading Figure 3. + +Why underestimation is the dangerous direction: an optimizer told "only 3 rows +come out of this" cheerfully picks a nested-loop join, which then runs orders +of magnitude more iterations than promised. Overestimation makes you buy a hash +table you did not need; underestimation makes you buy a quadratic algorithm. + +Why it matters: q-error is the unit the rest of the paper is denominated in, +and its asymmetry is the reason Step 7's mitigation works. ### Step 4 — the benchmark: real data is part of the method -TPC-H, the standard benchmark, is *generated* data: uniform value -distributions, independent columns. On it, Step 2's assumptions are -true by construction and estimates look fine — the standard benchmark -was structurally incapable of detecting the standard failure. So the -authors built **JOB**: 113 queries (3–16 joins each) over the real IMDB -dataset, where actors correlate with genres correlate with production -years. **Benchmark data distribution is part of the benchmark** — the -lesson to carry to every benchmark you ever build. - -### Step 5 — the method: inject ground truth, isolate the blame - -The experimental design is worth copying forever. Factor the optimizer -into its three claims (Step 1) and test each *in isolation* by feeding -the layers below it perfect inputs: - -1. cardinality estimates — compare against TRUE cardinalities (computed - offline) for every subplan; -2. cost model — feed it TRUE cardinalities, see if better cost = faster; -3. plan space — with perfect estimates, how much do bushy trees / - exhaustive search matter? - -Injecting ground truth at each layer isolates the blame: if plans become -good the moment true cardinalities are injected, the estimator was the -problem, not the cost model or the search. (This is the fair-benchmarking -discipline of topic 0, applied to a brain.) - -### Step 6 — the verdict: cardinality ≫ cost model ≫ search space - -- **Cardinality is the whole ballgame** — Step 3's 10²–10⁴ q-errors - translate directly into catastrophic plan choices. -- **The cost model barely matters**: with true cardinalities, even a - trivial cost model (they use Cout — the sum of intermediate result - cardinalities, nothing else) picks plans within ~2× of optimal. - Cost-model tuning is polishing the wrong layer. (DuckDB's - `cost_model.cpp:40` being literally Cout is this finding, shipped.) -- **Plan space matters at the margins**: exhaustive beats - greedy/quickpick meaningfully; bushy trees beat left-deep-only by - ~10–40% on some queries. But all of it is noise next to cardinality - error. - -``` - error source typical impact on runtime - cardinality (6-way) 10×–1000× (catastrophic plans) - cost model ~2× - search space ~1.1×–1.4× +> **In:** the observation that nobody had caught this before. +> **Out:** a dataset and a query set on which the failure is *detectable* — +> the paper's first contribution, prior to any measurement. + +TPC-H, the standard benchmark of the era, is *generated* data: uniform value +distributions, independent columns. On it, Step 2's three assumptions are true +by construction, and estimates look fine. The paper confirms this directly in +§3.3 — TPC-H estimates are far better behaved than JOB's. **The standard +benchmark was structurally incapable of detecting the standard failure.** + +So the authors built JOB on the real IMDB dataset (§2.1): a May 2013 snapshot, +**21 tables**, 3.6 GB as CSV, with `cast_info` at 36M rows and `movie_info` at +15M rows. The correlations are the point — actors correlate with genres +correlate with production years correlate with countries. + +The queries (§2.2) are **33 query structures × 2–6 variants = 113 queries**, +with **3 to 16 joins each and 8 joins on average**. Variants of one structure +share a join graph and differ only in their selection predicates, which +isolates estimation quality from plan-shape effects. + +Why it matters: this is the transferable lesson even if you never touch a +relational optimizer. **The data distribution is part of the benchmark.** A +generator that produces independent uniform columns cannot falsify an +independence assumption, no matter how many queries you run through it. + +### Step 5 — the method: extract ground truth, then inject it + +> **In:** the 113 queries and the five systems. +> **Out:** *two* datasets that feed different later steps — (a) each system's +> estimated cardinality for every subplan, compared against the true +> cardinality, which is all of §3; and (b) a modified PostgreSQL that can be +> *fed* any cardinalities you like, which is the instrument for §4, §5 and §6. + +This is the fork in the paper, and it is the piece worth stealing. + +First, extraction (§2.4). For every one of the 113 queries they enumerate its +subplans and run a `COUNT(*)` query per subplan to obtain the **true** +cardinality of every intermediate result. Separately they read each system's +*estimate* for the same subplans out of its `EXPLAIN` output. Dataset (a) is +the pairing of those two, and §3 is simply its q-error distribution. + +Second, injection. They patched PostgreSQL so that its cardinality estimates +can be overridden from outside — you hand it a table of cardinalities and it +optimizes as if those were its own estimates. Now every later question becomes +a controlled experiment: + +1. **Is the estimator to blame?** Run each query twice, once with PostgreSQL's + own estimates and once with true cardinalities injected, and compare + *runtimes* of the resulting plans (§4). +2. **Is the cost model to blame?** Hold cardinalities at truth and vary only + the cost function (§5). Any difference that remains is the cost model's. +3. **Is the search to blame?** Hold cardinalities at truth and vary only the + enumeration algorithm and the admissible tree shapes (§6). + +The experimental setup is worth noting so you can judge what transfers (§2.5): +two Intel Xeon X5570 @ 2.9 GHz (8 cores), 64 GB RAM, PostgreSQL 9.4, one core +per query, `work_mem` 2 GB, `shared_buffers` 4 GB, `effective_cache_size` +32 GB — and `geqo_threshold` raised to 18 so that PostgreSQL runs its dynamic +programming rather than its genetic optimizer on the big queries. This is a +*main-memory* setting, which matters a lot for §5. + +Why it matters: injecting ground truth one layer at a time is how you assign +blame in any layered system. If plans become good the moment true cardinalities +arrive, the estimator was the problem and no amount of cost-model tuning would +have helped. (This is topic 0's fair-benchmarking discipline applied to a +brain rather than a loop.) + +### Step 6 — the verdict: cardinality ≫ cost model ≫ search + +> **In:** the injection instrument from Step 5. +> **Out:** a ranking of the three components by measured impact on runtime, +> with a number attached to each. + +**Cardinality is the whole ballgame (§4).** With PostgreSQL's own estimates +replaced by true cardinalities, the distribution of runtime change across the +113 queries is (§4.1): + +``` + slowdown of a system's estimates vs. true cardinalities, share of queries + <0.9 [0.9,1.1) [1.1,2) [2,10) [10,100) >100 + PostgreSQL 1.8% 38% 25% 25% 5.3% 5.3% + DBMS A 2.7% 54% 21% 14% 0.9% 7.1% + DBMS B 0.9% 35% 18% 15% 7.1% 25% + DBMS C 1.8% 38% 35% 13% 7.1% 5.3% + HyPer 2.7% 37% 27% 19% 8.0% 6.2% +``` + +Roughly a tenth of queries are 10× or worse off because of estimation error +alone, and for DBMS B a quarter of the workload is over 100× off. + +**The cost model matters, but an order of magnitude less (§5).** PostgreSQL's +cost model is "over 4000 lines of C" (§5.1), and with true cardinalities +injected its median runtime-prediction error is **38%** (§5.2). Tuning its CPU +cost parameters up by 50× for a main-memory machine takes that to **30%** +(§5.3) — the defaults imply that processing a tuple is 400× cheaper than +reading it from a page, which was true of 1990s disks and is not true here. + +Then §5.4 replaces the whole thing with a deliberately trivial model, `Cmm`, +which counts only tuples flowing through operators: + ``` + Cmm(R) = τ·|R| base scan or selection + Cmm(T1 ⋈HJ T2) = |T| + Cmm(T1) + Cmm(T2) hash join + Cmm(T1 ⋈INL R) = Cmm(T1) + λ·|T1|·max(|T1⋈R|/|T1|, 1) index-nested-loop + + τ = 0.2 scans are cheaper per tuple than joins + λ = 2 an index lookup costs ~2× a hash-table lookup +``` + +Note this is *not* pure Cout (the sum of intermediate cardinalities and nothing +else); it discounts scans by τ and prices index lookups at λ. On true +cardinalities, geometric mean over all queries: the **tuned** PostgreSQL model +is **41% faster** than the standard one, and the trivial `Cmm` is **34% +faster** (§5.4). The paper's own conclusion from those two numbers: the +improvement "is not insignificant, but… it is dwarfed by improvement in query +runtime observed when we replace estimated cardinalities with the real ones". + +**Search matters least, and mostly by not being catastrophic (§6).** Two +results. First, join order is not free: over 10,000 random plans per query +(§6.1), the average ratio between the worst and best plan is **101×** with no +indexes, **115×** with primary-key indexes and **48,120×** with PK+FK indexes — +so a *randomly* chosen order is a disaster. But an optimizer only has to find a +good one, not the best one, and the share of random plans within 1.5× of +optimal is **44% / 39% / 4%** for those three index configurations. + +Second, tree shape. A **left-deep** plan is one where every join's right input +is a base table, so the tree is a single spine — this is System R's restriction +and it makes the DP tractable. A **bushy** plan allows a join whose *both* +inputs are themselves joins. Table 2 (§6.2), all with true cardinalities and +normalized against the optimal bushy plan: + +``` + Table 2 (§6.2) — slowdown of restricted tree shapes vs. optimal bushy + PK indexes PK + FK indexes + median 95% max median 95% max + zig-zag 1.00 1.06 1.33 1.00 1.60 2.54 + left-deep 1.00 1.14 1.63 1.06 2.49 4.50 + right-deep 1.87 4.97 6.80 47.2 30,931 738,349 +``` + +Left-deep costs you essentially nothing at the median and 2.5× at the 95th +percentile with FK indexes. Right-deep-only is a catastrophe. So "bushy trees +beat left-deep by 10–40%" is not what this table says: bushy's advantage is in +the tail, not the median. + +Third, enumeration algorithm. Table 3 (§6.3) compares exhaustive dynamic +programming against Quickpick-1000 (1000 random plans, keep the best) and GOO +(greedy operator ordering), each normalized by that configuration's optimal +plan: + +``` + Table 3 (§6.3) — plan quality by search algorithm + PK indexes PK + FK indexes + PG estimates true card. PG estimates true card. + med 95% max med 95% max med 95% max med 95% max + Dyn. Prog. 1.03 1.85 4.79 1.00 1.00 1.00 1.66 169 186,367 1.00 1.00 1.00 + QuickPick-1000 1.05 2.19 7.29 1.00 1.07 1.14 2.52 365 186,367 1.02 4.72 32.3 + Greedy (GOO) 1.19 2.29 2.36 1.19 1.64 1.97 2.35 169 186,367 1.20 5.77 21.0 +``` + +This one table is the paper's whole thesis in numbers. Compare *along a row*: +exhaustive DP goes from 1.66 median / 186,367 max with PostgreSQL's estimates +to a perfect 1.00 / 1.00 with true cardinalities. Now compare *down a column*: +with true cardinalities, swapping the world's best search for a greedy +heuristic costs you 1.20 at the median. The search algorithm is worth ~20%; the +estimates are worth five orders of magnitude. + +``` + what the paper actually measured, ranked + cardinality estimates 10× or worse for ~11% of queries (§4.1); + DP max 186,367 → 1.00 on truth (Table 3) + cost model 34–41% geometric-mean runtime (§5.4) + search algorithm 1.00 → 1.20 median, DP → greedy on truth (Table 3) + tree shape 1.00 → 1.06 median, bushy → left-deep (Table 2) +``` + +Why it matters: it tells you where to spend engineering effort, and it explains +why DuckDB ships a cost model that fits on one screen +(`src/optimizer/join_order/cost_model.cpp:40-48`) while spending real +complexity on its cardinality estimator. ### Step 7 — living with wrong estimates: robust plans -Since estimates can't be fixed cheaply, the paper's pragmatic mitigation -is to prefer plans **robust to misestimation**: a hash join costs roughly -linearly in input size, so a 10⁴× underestimate makes it 10⁴× slower — -bad; a nested-loop join costs the *product* of its inputs, so the same -underestimate is quadratically catastrophic. When unsure, take the -algorithm whose worst case degrades gracefully. Postgres's notorious -nested-loop disasters are exactly underestimates of 10⁴ feeding "it's -only 3 rows" decisions. +> **In:** the finding that estimates cannot be fixed cheaply. +> **Out:** two concrete mitigations the paper measures, both of which trade a +> little best case for a lot of worst case. + +Since Step 6 says the estimates are the problem and Step 3 says they are +structurally hard, the pragmatic move is to prefer plans that are **robust to +misestimation** — plans whose cost degrades gracefully when the estimate is +wrong, rather than plans that are optimal if the estimate is right. + +The argument is asymptotic, and the paper makes it in §4.1: a hash join is O(n) +in its input size, while a non-index nested-loop join is O(n²). Under a 100× +underestimate the hash join is ~100× slower than predicted; the nested-loop +join is ~10,000× slower. Same wrong estimate, quadratically different +consequence. The measurements: + +- Disabling non-index nested-loop joins removed **all** timeouts from the + workload (§4.1, Figure 6b) — the entire catastrophic tail was one operator + choice. +- Adding rehashing to the hash table, so an undersized hash table grows instead + of degrading, brought it to **less than 4%** of queries off by more than 2× + (§4.1, Figure 6c). +- And the price of this insurance is small: fully cached, a hash join's + advantage over an index-nested-loop join is at most **5× in PostgreSQL and 2× + in HyPer** (§4.2). You are giving up at most 5× best case to remove an + unbounded worst case. + +One caution before you conclude "so just collect better statistics": §3.4 shows +the authors injecting *true* distinct-value counts into PostgreSQL, which made +underestimation **worse**, not better. The uniformity error and the +independence error had been partially cancelling — two wrongs making a right. +Fixing one input of a wrong model does not give you a right model. + +Why it matters: this is the shape of every mitigation in a system with +irreducible uncertainty — you do not chase the best expected case, you bound +the worst one. ## How to read the paper (with the concepts in hand) -~1.5 h. The methodology (§2–3) is worth as much as the findings — -injecting ground truth per layer isolates the blame. - -- **§2 (the benchmark)** — Step 4. Note *why* each JOB query family - exists: correlations are chosen deliberately, not sampled at random. -- **§3 (cardinality estimation) — read carefully.** Steps 2–3 measured: - the q-error-vs-join-count figures are the paper's core result. Check - that underestimation dominates and that all systems (including the - commercial ones) degrade the same way. -- **§4 (cost model)** — Step 6's second bullet: watch Cout with true - cardinalities land within ~2× of the full-blown model. -- **§5 (plan space)** — Step 6's third bullet: bushy vs left-deep, - exhaustive vs heuristic, all quantified with truth injected. -- **§6 (discussion)** — Step 7's robustness argument, plus the pointers - to sampling-based estimation. The learned-cardinality papers in the - topic README (Kipf '19, Neo, Bao) are this section's direct - descendants — read them after, not instead. +~1.5 h. The methodology (§2) is worth as much as the findings. Section numbers +below are the PVLDB version's. + +- **§2 — Background and Methodology** (Steps 4 and 5). §2.1 the IMDB data, + §2.2 the queries, §2.3 PostgreSQL's estimator and its three assumptions, + §2.4 the extraction/injection instrument, §2.5 the hardware. Note *why* each + query family exists: the correlations are chosen deliberately. +- **§3 — Cardinality Estimation. Read carefully; this is the core.** Table 1 + for base tables (§3.1), **Figure 3 for joins (§3.2)** — spend real time on + Figure 3, it is the paper's central result. §3.3's TPC-H comparison is the + control. §3.4 is the "two wrongs make a right" trap from Step 7. +- **§4 — When Do Bad Cardinality Estimates Lead to Slow Queries?** Step 6's + first result and Step 7's mitigations. Figure 6's three panels are the + argument: (a) estimates vs. truth, (b) nested loops disabled, (c) rehashing. +- **§5 — Cost Models** (Step 6's second result). §5.1 the 4000 lines, §5.2 the + 38% prediction error, §5.3 the tuning, §5.4 the trivial `Cmm` — note it is + *not* pure Cout. +- **§6 — Plan Space** (Step 6's third result). §6.1 how much join order + matters, §6.2 Table 2 on bushy vs left-deep vs right-deep, §6.3 Table 3 on + DP vs Quickpick vs greedy. Table 3 is the single best summary of the paper. +- **§7–8 — Related work and conclusions.** §4.4's "join-crossing correlations" + is the open problem the authors flag. The learned-cardinality papers in the + topic README (Kipf '19, Neo, Bao) are its direct descendants — read them + after, not instead. ## Questions for notes.md -1. Why does independence UNDERestimate join sizes on correlated data? - Construct a 2-table example where sel(a)×sel(b) is 100× low. -2. Cout (sum of intermediate sizes) as the whole cost model: which of - your engines' knobs does that validate (DuckDB cost_model.cpp:40 is - literally this)? -3. "Robust plans": hash join degrades linearly with a bad estimate, - nested-loop quadratically. Frame it as a minimax decision — what's - the regret matrix? -4. Design JOB-for-graphs: what's the correlated-data equivalent for - Cypher patterns (degree skew × label correlation × triangle - density)? Sketch 3 queries where independence-based nnz estimation - (matrix-product size) blows up the same way. This is the M10/M22 - benchmark seed — write it down properly. +1. Why does independence UNDERestimate on correlated data rather than + overestimate? Construct a two-predicate example where sel(a)×sel(b) is 100× + low, and state the conditional probability that makes it so. +2. Table 3 shows DP with PostgreSQL's estimates at a median of 1.66 and greedy + with true cardinalities at 1.20. Write the one-sentence engineering + conclusion, then say which of your engines' behaviour it explains. +3. §3.4: injecting *true* distinct counts made estimates worse. What does that + tell you about the practice of "improving statistics" in isolation, and what + is the analogous trap in a system you have tuned? +4. "Robust plans": hash join degrades linearly with a bad estimate, + nested-loop quadratically, and the price is ≤5× (§4.2). Frame it as a + minimax decision — what is the regret matrix? +5. Design JOB-for-graphs: what is the correlated-data equivalent for Cypher + patterns (degree skew × label correlation × triangle density)? Sketch three + queries where independence-based nnz estimation (matrix-product size) blows + up the same way. This is the M10/M22 benchmark seed — write it down + properly. + +## Takeaway + +The optimizer's three layers fail by wildly different amounts, and the paper +measured the gap: with true cardinalities injected, exhaustive DP finds the +optimal plan every time (Table 3, 1.00/1.00/1.00), while with real estimates +its worst case is 186,367× optimal. Swapping the search algorithm for a greedy +heuristic costs 20% at the median; swapping the 4000-line cost model for a +three-line one costs 34–41% in geometric mean. Cardinality estimation is not +*a* problem in query optimization — on real data it is *the* problem, and the +right response is not better guesses but plans whose cost does not explode when +the guess is wrong. ## Done when -You can rank cardinality/cost/search by measured impact, explain WHY -independence fails low, and have the graph-JOB sketch in notes.md. +Answer each before unfolding it. + +- [ ] State the definition of q-error, and compute it for an estimate of 310 + against a true cardinality of 2,600. +
Answer + + q-error(est, true) = max(est/true, true/est), so it is always ≥ 1 and treats + a 10× under- and a 10× overestimate identically. Here 310 < 2600, so the + max is 2600/310 = **8.39**. This is the paper's own footnote-6 example + (§3.2): the same two-join query, true cardinality 2,600, got estimates of 3, + 9, 128 or 310 depending only on the textual order of relations in `FROM` — + q-errors of 866.67, 288.89, 20.31 and 8.39. Even the *best* of the four is + 8× low. + +
+ +- [ ] Figure 3 shows the error growing with join count. Say precisely what + grows — and what does *not*. +
Answer + + The **distribution widens**; the median does not move much. The medians in + Table 1 (base tables, §3.1) are 1.00–1.06 across all five systems, and in + Figure 3 they stay near 1 and drift *downward* with join count. What grows + is the spread, and it grows asymmetrically: the axis spans underestimation + by 10⁸ against overestimation by only 10⁴. The paper's quantified version + (§3.2) is the fraction of estimates wrong by ≥10×: PostgreSQL 16% at one + join, 32% at two, 52% at three; DBMS A 15%, 25%, 36%. Saying "median q-error + reaches 10²–10⁴ at six joins" misreads the figure. + +
+ +- [ ] Three predicates with selectivities 0.20, 0.05 and 0.25 on a 2,500,000-row + table. Give the independence estimate, then the true count if + P(p3 | p2) = 0.50 and P(p1 | p2 ∧ p3) = 0.30, and the resulting q-error. +
Answer + + Independence multiplies: 0.20 × 0.05 × 0.25 = 0.0025, so the estimate is + 2,500,000 × 0.0025 = **6,250 rows**. The correlated truth chains conditionals + instead: 0.05 × 0.50 × 0.30 = 0.0075, so 2,500,000 × 0.0075 = **18,750 + rows**. q-error = 18,750/6,250 = **3.0**. Two mild correlations (a 2× and a + 1.5×) produced a 3× error on a single table with no joins at all — and + because cardinalities multiply along a join tree, a constant 3× per join + compounds to 3⁶ = 729× after six. (The selectivities here are stated + assumptions for the arithmetic, not measurements from the paper; the paper's + measured version is Figure 3.) + +
+ +- [ ] Rank cardinality estimation, cost model and search by measured impact, + with one number each. +
Answer + + **Cardinality** first, by orders of magnitude: ~11% of queries are 10× or + worse purely from estimation error (§4.1), and in Table 3 exhaustive DP goes + from a max of 186,367× optimal on PostgreSQL's estimates to exactly 1.00 on + true cardinalities. **Cost model** second: with truth injected, a tuned model + is 41% faster and the trivial three-line `Cmm` is 34% faster than + PostgreSQL's 4000-line one (§5.4) — real, but the paper itself calls it + "dwarfed". **Search** last: with truth injected, replacing DP with greedy GOO + costs 1.20 at the median (Table 3), and restricting bushy to left-deep costs + 1.00–1.06 at the median (Table 2). Note that all three of these are measured + *with the layers below held at truth* — that is what makes them comparable. + +
+ +- [ ] The paper's cost-model replacement is often called "Cout". Why is that + wrong, and what is it? +
Answer + + Cout is the sum of all intermediate result cardinalities and nothing else. + The paper's model (§5.4) is `Cmm`, which is Cout *plus two constants*: scans + and selections are charged τ·|R| with **τ = 0.2**, discounting a scan + relative to a join, and an index-nested-loop join is charged + λ·|T1|·max(|T1⋈R|/|T1|, 1) with **λ = 2**, pricing an index lookup at twice a + hash-table lookup. Only the hash join term, |T| + Cmm(T1) + Cmm(T2), is pure + Cout. It is also explicitly a *main-memory* model — it does not model I/O at + all, which is why §2.5's fully-cached setup matters when you decide whether + the result transfers to your system. + +
+ +- [ ] Why could TPC-H never have found this result? +
Answer + + TPC-H's data is generated with uniform value distributions and independent + columns, so Step 2's uniformity and independence assumptions are true *by + construction* and the estimator is right for the right reasons. §3.3 shows + exactly this — TPC-H estimates are far better behaved than JOB's. The + benchmark could not falsify the hypothesis it was implicitly testing. That is + why the paper's first contribution is a *dataset* (real IMDB: 21 tables, + 36M-row `cast_info`) and a *query set* (33 structures × 2–6 variants = 113 + queries, 3–16 joins, 8 on average) rather than a measurement. The data + distribution is part of the benchmark. + +
## References **Papers** - Leis, Gubichev, Mirchev, Boncz, Kemper, Neumann — "How Good Are Query - Optimizers, Really?" (VLDB 2015) — ~1.5 h; the methodology (§2–3) is - worth as much as the findings — injecting ground truth per layer - isolates the blame + Optimizers, Really?" PVLDB 9(3):204–215, 2015. ~1.5 h. The methodology (§2) + is worth as much as the findings — extracting true cardinalities and + injecting them one layer at a time is what makes the blame assignable. + +- Leis et al. — "Query optimization through the looking glass, and what we + found running the Join Order Benchmark", VLDB Journal 27(5), 2018. The + extended version; same results, more configurations, different section + numbers. +- Moerkotte, Neumann, Steidl — "Preventing Bad Plans by Bounding the Impact of + Cardinality Estimation Errors", PVLDB 2009. Where the q-error metric and its + optimality properties come from. + +**Code** +- `duckdb/duckdb@6c0c1a68` — `src/optimizer/join_order/cost_model.cpp:40-48`, + a cost model small enough to read in one sitting, which is this paper's §5 + conclusion shipped as a product. +- `postgres/postgres@701f021` — `src/backend/utils/adt/selfuncs.c`, the + estimator being audited; `src/backend/optimizer/` for the search it feeds. + +**In this topic** +- `reading-postgres-optimizer.md` — the estimator and search this paper + measures, read as code. +- `reading-duckdb-optimizer.md` — a modern optimizer built with this paper's + conclusion already assumed. +- `reading-selinger-cascades.md` — the 1979 design whose assumptions this paper + finally tested at scale. diff --git a/topics/10-query-planning/reading-postgres-optimizer.md b/topics/10-query-planning/reading-postgres-optimizer.md index fcd36fb..d3e519e 100644 --- a/topics/10-query-planning/reading-postgres-optimizer.md +++ b/topics/10-query-planning/reading-postgres-optimizer.md @@ -1,173 +1,619 @@ # Postgres's optimizer: Selinger '79, still in production -Forty-five years on, postgres's join search is still Selinger's DP — -level-by-level over relation sets, interesting orders kept as extra DP -state, a genetic-algorithm escape hatch for big joins. Before the code, -this chapter builds the six ideas the source assumes — access paths, the -level-by-level DP, interesting orders, the two-cost path, the genetic -fallback, and the default selectivities — then maps each to its -file:line. Read it for the search skeleton and for the honesty of the -default constants that run the world when stats are missing. +Forty-six years on, postgres's join search is still Selinger's DP — +level-by-level over relation sets, interesting orders kept as extra DP state, a +genetic-algorithm escape hatch for big joins. Before the code, this chapter +builds the six ideas the source assumes — access paths, the level-by-level DP, +interesting orders, the two-cost path, the genetic fallback, and the default +selectivities — then maps each to its file:line. Read it for the search +skeleton and for the honesty of the default constants that run the world when +stats are missing. + +**Every `file:line` below was read at `postgres/postgres@701f021`** using +`python3 tools/pinned-source.py show postgres -r A:B`. Line numbers move +between releases; re-run the tool rather than trusting the numbers. **Topic 10 +has no measured lane** — nothing here is a timing taken on this machine, and +none of these figures appear in `FINDINGS.md`. Where a runtime number is +needed, it is cited to the JOB paper (`reading-how-good-optimizers.md`). ## The problem in one sentence -Postgres must pick, in milliseconds and often with no statistics at all, -one plan out of an exponential space where the best and worst differ by -1000× — and when it knows nothing about a predicate it literally guesses -0.5%. +Postgres must pick, in milliseconds and often with no statistics at all, one +plan out of a space that grows factorially — and when it knows nothing about an +equality predicate it substitutes the compiled-in constant `0.005`, a number +chosen not to be *accurate* but to be small enough that index scans still get +picked (`src/include/utils/selfuncs.h:24-34`). ## The concepts, step by step ### Step 1 — access paths: even one table has many ways to be read -An **access path** is one concrete way to produce a single table's rows: -a **sequential scan** (read every page front to back) or an **index -scan** (walk an index — a sorted side-structure — to fetch matching rows -one at a time). For a 10M-row table with a filter matching 100 rows, the -index scan does ~100 page reads and the seqscan ~100K; flip the filter to -match 5M rows and the seqscan's sequential IO wins by 10×. So the first -thing the optimizer does is cost every access path *per table* — this is -literally what Selinger's paper title, "access path selection", means. -One path property matters beyond cost: an index scan delivers rows -*already sorted* by the index key. Hold that for Step 3. +> **In:** one base relation, its catalog statistics (row count, page count), +> its indexes, and the filter predicates that apply to it alone. +> **Out:** a *list* of costed **paths** for that relation — a `RelOptInfo` with +> a populated `pathlist`. This is level 1 of Step 2's DP. + +Two vocabulary items, because the whole file is written in them. + +- A **logical plan** is an expression in **relational algebra** — the algebra + of select (σ, filter), project (π, choose columns), join (⋈), aggregate, + union over bags of tuples. It says *what* rows you want. +- A **physical plan** picks an algorithm per logical operator. Postgres calls a + physical subplan a **`Path`**: a node with a cost, an output row estimate, and + an output sort order. + +An **access path** is a `Path` for a single table: a **sequential scan** (read +every page front to back) or an **index scan** (walk a sorted side-structure, +then fetch the matching heap rows). Which wins depends entirely on how many +rows survive the filter, and postgres decides with five compiled-in constants +(`src/include/optimizer/cost.h:24-28`): + +``` + seq_page_cost 1.0 one sequentially-read 8 KB page + random_page_cost 4.0 one randomly-fetched 8 KB page + cpu_tuple_cost 0.01 processing one tuple + cpu_index_tuple_cost 0.005 processing one index entry + cpu_operator_cost 0.0025 evaluating one operator/function +``` + +**Work it.** Take a 10,000,000-row table at 100 tuples per page — the density +`selfuncs.h:26` itself assumes — so 100,000 pages, and one filter predicate. +`cost_seqscan` (`src/backend/optimizer/path/costsize.c:300`, `:306-307`, +totalled at `:339`) is: + +``` + total = seq_page_cost·pages + (cpu_tuple_cost + qual_per_tuple)·tuples + + seqscan = 1.0 × 100,000 + (0.01 + 0.0025) × 10,000,000 + = 100,000 + 125,000 + = 225,000 +``` + +Note what is *absent* from that formula: selectivity. A seqscan costs 225,000 +whether the filter keeps 100 rows or 5 million — it always reads every page and +evaluates the qual on every tuple. Now the index scan at a selectivity of +0.00001 (100 rows out), assuming the worst case for locality, one random heap +fetch per row: + +``` + idxscan ≈ random_page_cost × 100 = 400 + + (cpu_tuple_cost + cpu_index_tuple_cost + + cpu_operator_cost) × 100 = 1.75 + ≈ 402 (plus a few pages of B-tree descent) + + 225,000 / 402 ≈ 560× in the index's favour +``` + +Flip the filter to keep 5,000,000 rows and the index's random-fetch term alone +is `4.0 × min(5,000,000, 100,000) = 400,000` — already 1.8× the seqscan's +*entire* cost before any CPU terms, and that is why postgres switches. (The +real index cost uses the Mackert–Lohman approximation at `costsize.c:898` to +account for pages revisited within one scan; the bound above is enough to see +the crossover.) The single input that decides a 560× difference is the +selectivity estimate — which Step 6 shows is often a guess. + +One path property matters beyond cost: an index scan delivers rows *already +sorted* by the index key. Hold that for Step 3. + +Why it matters: this is literally what Selinger's paper title, "access path +selection", means, and it is the base case the DP builds on. ### Step 2 — the join search: dynamic programming, level by level -To order n joins, postgres uses Selinger's **dynamic programming** (DP): -the best plan for a *set* of relations can only be built from best plans -of its subsets, so compute best plans for all 1-relation sets, then all -2-relation sets from level 1, then level 3 from levels 2+1 (left-deep -bias) *and* — postgres extends Selinger here — bushy combinations of -levels 2+2: +> **In:** the level-1 `RelOptInfo`s from Step 1, and the query's join +> predicates. +> **Out:** one `RelOptInfo` for the full relation set, whose `pathlist` +> contains the surviving complete plans — `standard_join_search`'s return +> value. + +**Dynamic programming** (DP) is the technique of solving a problem by solving +each distinct subproblem once and memoizing the answer, exploiting the fact +that an optimal solution is built from optimal solutions to subproblems. Here +the subproblem is *a set of relations*, and the principle of optimality is: +the best plan for `{A,B,C}` can only be built from best plans for its subsets. +Postgres's own comment says so (`path/allpaths.c:3964-3968`): + +> We employ a simple "dynamic programming" algorithm: we first find all ways to +> build joins of two jointree items, then all ways to build joins of three items +> (from two-item joins and single items), then four-item joins, and so on until +> we have considered all ways to join all the items into one rel. + +**Why bother.** A **left-deep** plan is one where every join's right input is a +base table, so the tree is a single spine; there are n! of them for n +relations. The DP does not enumerate plans, it enumerates *sets*, so it +considers far fewer: + +``` + left-deep orders enumerated one at a time n! + (set, last-relation) pairs the DP considers n·2^(n-1) − n + DP memo entries (subsets of n relations) 2^n − 1 + + n n! DP considered memo n! / DP + 5 120 75 31 1.6 + 10 3,628,800 5,110 1,023 710.1 + 15 1,307,674,368,000 245,745 32,767 5,321,265.4 +``` + +At n = 5 the DP barely pays for itself. At n = 10 it does 710× less work than +naive enumeration; at n = 15, five million times less. And it is *still* +exponential — the memo column is `2^n − 1` — which is exactly why Step 5 exists. +(These are counts over a complete join graph and are computed here, not +measured; the real counts are smaller because Step 2's connectedness test +prunes disconnected subsets.) + +Postgres extends Selinger by also building **bushy** plans — a join whose +*both* inputs are themselves joins. `join_search_one_level` +(`path/joinrels.c:78`) does the level in two passes: ``` - level 1: {A} {B} {C} best path(s) per single rel - level 2: {AB} {AC} {BC} join_search_one_level (joinrels.c:78): - level 3: {ABC} combine level k-1 rels with level 1 - (left-deep bias) AND k-2 with 2 (bushy) - each set keeps: cheapest total path, cheapest startup path, plus one - path per INTERESTING ORDER (sorted output that a later merge join / - ORDER BY could exploit — the DP state postgres kept and DuckDB dropped) + level 1: {A} {B} {C} {D} Step 1's access paths, one RelOptInfo each + + pass 1 (joinrels.c:96-143) "left-sided and right-sided plans": every + level-1 rel of the previous level joined + against each *initial* (single) relation + it has a join clause with [:123] + — no join clause at all? cartesian + product against every initial rel [:139] + + pass 2 (joinrels.c:153-198) "bushy plans": for k = 2, 3, ... while + k <= level-k, join every level-k rel to + every level-(level-k) rel it shares a + join clause with. Halts at the halfway + point because make_join_rel(x,y) handles + both orders [:161] ``` -Connectedness prunes the space: only pair sets linked by a join -predicate, unless a cartesian product is forced at the end. Cost: the DP -memo holds one entry per relation subset — exponential in n, which is -why Step 5's escape hatch exists. +Two precision points people get wrong. First, pass 1 is *not* purely left-deep: +the code's own comment (`:90-91`) says "left-sided **and** right-sided plans", +because `make_join_rel` is symmetric. Second, pass 2 is not just "k−2 with 2" — +it is *every* split from k = 2 up to level/2, so at level 6 it considers 2+4 +**and** 3+3. + +Connectedness prunes the space: pass 2 only pairs rels that share a join clause +or a join-order restriction, explicitly "in order to avoid unreasonable growth +of planning time" (`:149-151`). Pass 1 falls back to cartesian products only +for a rel with no join clauses at all. + +Each set's surviving paths live in `root->join_rel_level[lev]` +(`allpaths.c:3974-3976`, driven by the loop at `:3978-3987`), and `set_cheapest` +(`util/pathnode.c:268`) is called on each finished joinrel before the next +level begins. + +Why it matters: this is the skeleton every cost-based optimizer since 1979 has +either used or deliberately replaced. DuckDB's is the same idea with a +different enumeration order (`reading-duckdb-optimizer.md`). ### Step 3 — interesting orders: why one "best" plan per set isn't enough -An **interesting order** is a sort order of a subplan's output that some -*later* operator could exploit — a merge join (joins two sorted inputs by -scanning them in lockstep), an ORDER BY, a GROUP BY. Keeping only the -single cheapest plan per relation set would be a bug: a subplan that -costs 20% more but delivers rows already sorted can win *globally* by -saving a full sort later. So the DP cell keeps MULTIPLE surviving paths — -one per useful ordering — and a new path survives unless some existing -path beats it on *every* axis. This is `add_path`, conceptually: - -```rust -fn add_path(rel: &mut RelOptInfo, new: Path) { - let dominated = rel.paths.iter().any(|p| - p.total_cost <= new.total_cost - && p.startup_cost <= new.startup_cost // LIMIT-friendly axis - && p.ordering.subsumes(&new.ordering)); // sorted output IS DP state - if !dominated { - rel.paths.retain(|p| !new.dominates(p)); - rel.paths.push(new); // a pricier-but-sorted path survives here, - } // to win later at a merge join or ORDER BY -} +> **In:** a candidate `Path` for some relation set, and the paths already +> surviving for that set. +> **Out:** an updated `pathlist` — the candidate inserted, or rejected, and any +> old paths it dominates removed. This is what makes each DP cell hold +> *several* plans instead of one. + +An **interesting order** is a sort order of a subplan's output that some *later* +operator could exploit — a merge join (which joins two sorted inputs by scanning +them in lockstep), an `ORDER BY`, a `GROUP BY`. Postgres calls the +representation **pathkeys**. + +Keeping only the single cheapest plan per relation set would be a bug: a +subplan that costs 20% more but delivers rows already sorted can win *globally* +by saving a full sort later. So the DP cell keeps multiple surviving paths, and +a new one survives unless some existing path beats it on *every* axis. That is +`add_path` (`util/pathnode.c:459`), and the axes are documented in its header +comment at `:391-412` and implemented at `:518-533`: + +``` + add_path's dominance axes (util/pathnode.c:391-412, :518-533) + 1. disabled_nodes how many disabled node types the path uses — + a higher-order component of cost [:399-406] + 2. startup_cost cost before the first row [:396-398] + 3. total_cost cost of the whole result [:396-398] + 4. pathkeys output sort order — Step 3's whole point [:511-513] + 5. required_outer parameterization (which outer rels it needs) [:518] + 6. rows a path producing fewer rows can win [:524] + 7. parallel_safe [:525] ``` +Three things about this that a summary usually loses: + +- It is **seven axes, not two**. A guide that shows `add_path` comparing only + cost and ordering is describing the 1979 paper, not this file. +- The cost comparison is deliberately **fuzzy**: `compare_path_costs_fuzzily` + (`:181`) is called with `STD_FUZZ_FACTOR`, defined as `1.01` at `:47`. Costs + within 1% of each other count as equal. The comment at `:545-548` gives the + reason — an exact comparison "results in annoying platform-specific plan + variations due to roundoff in the cost estimates". +- Parameterized paths are treated as having **no** pathkeys (`:472-473`, + policy stated at `:420-424`), specifically to keep the pathlist small. + The cost of this refinement is a fatter memo (a handful of paths per set -instead of one); the payoff is that merge-join plans are findable at all. +instead of one); the payoff is that merge-join plans and index-order plans are +findable at all. + +Why it matters: "interesting orders" is the one piece of Selinger that a naive +reimplementation always drops, and dropping it silently removes an entire join +algorithm from the search space. ### Step 4 — two costs per path: startup and total -Every path carries a pair `(startup_cost, total_cost)`: what it costs -before the *first* row comes out, and what the *whole* result costs. The -distinction exists because of LIMIT: for `ORDER BY x LIMIT 10`, an index -scan on x has near-zero startup (rows stream out sorted immediately) -even if its total cost is high, while sort-everything has all its cost -in startup. One number can't represent both queries; two numbers is the -underrated design decision — it's the second dominance axis in Step 3's -`add_path`. +> **In:** a costed `Path`. +> **Out:** the pair `(startup_cost, total_cost)` that Step 3's axes 2 and 3 +> compare — and the reason a single scalar cannot rank plans. + +Every path carries two numbers: what it costs before the *first* row comes out, +and what the *whole* result costs. `cost_seqscan` sets both explicitly +(`costsize.c:338-339`). + +The distinction exists because of `LIMIT`. For `ORDER BY x LIMIT 10`: + +- an index scan on `x` has near-zero **startup** cost — rows stream out already + sorted — even if its total cost is high, because you stop after 10; +- a seqscan-then-sort has *all* of its cost in startup, because a sort cannot + emit its first row until it has consumed its last input row. + +Reverse the query to "no LIMIT, return everything" and the ranking flips. One +number cannot represent both queries; two numbers can. This is why +`add_path` keeps a path that is cheaper on startup even when it loses on total +(`pathnode.c:396-398`: "if one path is cheaper in one of these aspects and +another is cheaper in the other, we keep both") — and it is guarded by +`consider_startup`, so the extra paths are only retained when a `LIMIT`-like +construct actually exists (`:426-429`). + +Why it matters: it is the cheapest possible generalization from "cost" to "cost +*function* of how much of the result you consume", and it costs one extra +`double` per path. ### Step 5 — when n is big: the genetic escape hatch -The DP's memo is exponential in the number of relations, so at -`geqo_threshold` relations (default 12) postgres abandons it for **geqo** -— a **genetic algorithm** (randomized search that "evolves" a population -of candidate solutions by recombining good ones): join orders are encoded -as chromosomes, evaluated with the normal cost model, and bred for a -fixed number of generations. Join order as TSP-style chromosome -evolution; nobody's proud of it, everybody ships a fallback (DuckDB's is -greedy — see reading-duckdb-optimizer.md). The trade: geqo explores -*tree-shaped* candidates rather than committing to one greedy sequence, -at the price of nondeterministic plans. +> **In:** the relation count `levels_needed` and the `initial_rels` list. +> **Out:** *either* the exhaustive DP of Step 2, *or* a genetic search that +> returns one `RelOptInfo` and no optimality guarantee — the branch is a single +> `else if`. + +Step 2's memo is `2^n − 1` entries, so postgres gates on relation count +(`path/allpaths.c:3913-3918`, inside `make_rel_from_joinlist` at `:3847`): + +```c +// postgres/postgres@701f021 — src/backend/optimizer/path/allpaths.c +3913 if (join_search_hook) +3914 return (*join_search_hook) (root, levels_needed, initial_rels); +3915 else if (enable_geqo && levels_needed >= geqo_threshold) +3916 return geqo(root, levels_needed, initial_rels); +3917 else +3918 return standard_join_search(root, levels_needed, initial_rels); +``` + +`geqo_threshold` defaults to **12** (`src/backend/utils/misc/guc_parameters.dat:1191`, +`boot_val => '12'`). Note the extension point on line 3913: an extension can +replace the join search wholesale. + +**geqo** is a **genetic algorithm** — a randomized search that maintains a +population of candidate solutions, breeds new ones by recombining good ones, +and keeps the fitter offspring. Postgres's parameters are computed, not fixed +(`geqo/geqo_main.c:328-350` and `:360-367`): + +``` + pool_size = clamp( 2^(n+1), 10·Geqo_effort, 50·Geqo_effort ) + generations = pool_size + Geqo_effort defaults to 5 (src/include/optimizer/geqo.h:57) + + so at n = 12: 2^13 = 8192, clamped to 50 × 5 = 250 individuals + and 250 generations +``` + +One new individual is bred and evaluated per generation (`geqo_main.c:192` +loop, `geqo_eval` at `:230`, `spread_chromo` at `:233`), so the whole search +costs roughly `pool_size + generations` ≈ 500 cost evaluations at n = 12 — +against a DP that would have considered tens of thousands of set pairs. + +**The detail everybody states backwards.** A geqo chromosome is a **sequence**, +not a tree: `Gene *tour` (`geqo_eval.c:140`), recombined with **ERX**, edge +recombination crossover (`geqo.h:46`, applied at `geqo_main.c:198-204`) — an +operator borrowed from the travelling salesman problem. The tree is *derived* +from the tour afterwards by `gimme_tree` (`geqo_eval.c:163`), which walks the +tour maintaining "clumps" of already-joined relations and adds each new +relation to the first clump it can legally join (`:171-178`). The comment at +`:146-157` records the history: the original implementation joined strictly in +tour order and "could never produce a 'bushy' plan", which broke queries whose +only valid plans are bushy; the clump heuristic was added to fix that, "and as +a nice side-effect it seems to materially improve the quality of the generated +plans". + +The trade: nondeterministic plans (the same query can be planned differently +twice) in exchange for bounded planning time. Nobody is proud of it; everybody +ships a fallback. DuckDB's is greedy operator ordering with the same threshold +of 12 (`reading-duckdb-optimizer.md`). + +Why it matters: the escape hatch is where the "optimal plan" guarantee actually +ends, and knowing the threshold is 12 tells you exactly which of your queries +have no guarantee at all. ### Step 6 — the constants that run the world -All of the above consumes **selectivity** estimates (the fraction of rows -a predicate keeps). With statistics, `selfuncs.c` uses histograms + -MCV lists (most-common-values: the top values and their actual -frequencies) — single-column skew handled. Without stats — fresh table, -default-typed expression, anything opaque — postgres falls back to -compiled-in constants in `include/utils/selfuncs.h`: +> **In:** a predicate and whatever the catalog knows about its columns. +> **Out:** a **selectivity** in [0,1] — the fraction of rows the predicate +> keeps — which Step 1 multiplies into row counts and Step 2 propagates up the +> whole tree. + +With statistics, `utils/adt/selfuncs.c` uses **histograms** (a table of value +ranges plus the row count falling in each, so a range predicate is answered by +summing buckets) and **MCV lists** (most-common-values: the top values with +their measured frequencies). Single-column skew is handled well by these. + +Without stats — a fresh table, an opaque expression, a function call postgres +cannot see into — it falls back to compiled-in constants in +`src/include/utils/selfuncs.h`: + +```c +// postgres/postgres@701f021 — src/include/utils/selfuncs.h + 23 /* + 24 * Note: the default selectivity estimates are not chosen entirely at random. + 25 * We want them to be small enough to ensure that indexscans will be used if + 26 * available, for typical table densities of ~100 tuples/page. Thus, for + 27 * example, 0.01 is not quite small enough, since that makes it appear that + 28 * nearly all pages will be hit anyway. Also, since we sometimes estimate + 29 * eqsel as 1/num_distinct, we probably want DEFAULT_NUM_DISTINCT to equal + 30 * 1/DEFAULT_EQ_SEL. + 31 */ + 32 + 33 /* default selectivity estimate for equalities such as "A = b" */ + 34 #define DEFAULT_EQ_SEL 0.005 + 35 + 36 /* default selectivity estimate for inequalities such as "A < b" */ + 37 #define DEFAULT_INEQ_SEL 0.3333333333333333 + 38 + 39 /* default selectivity estimate for range inequalities "A > b AND A < c" */ + 40 #define DEFAULT_RANGE_INEQ_SEL 0.005 + // ... 41-51: MULTIRANGE_INEQ, MATCH_SEL, MATCHING_SEL ... + 52 #define DEFAULT_NUM_DISTINCT 200 +``` + +Read the comment before you sneer at the number. `0.005` is not an estimate of +reality; it is a value **chosen to be small enough that index scans still get +picked** at ~100 tuples/page. `0.01` was tried and rejected as too large. And +`DEFAULT_NUM_DISTINCT 200` at `:52` is not independent — the comment says it +exists to satisfy `DEFAULT_NUM_DISTINCT = 1/DEFAULT_EQ_SEL`, and indeed +`1/0.005 = 200`. The constants are a *consistent* fiction, not a careless one. + +`DEFAULT_INEQ_SEL` is a third, spelled to sixteen decimal places — the digits +are the only part of it that looks like a measurement. -- `DEFAULT_EQ_SEL 0.005` :34 — "col = ?" with no stats: 0.5%. -- `DEFAULT_INEQ_SEL 0.3333…` :37 — "col < ?": one third. A COIN FLIP - wearing three decimal places. -- `DEFAULT_RANGE_INEQ_SEL 0.005` :40. +**Work the composition, because this is where the constants bite.** Postgres +assumes **independence** across predicates, so a conjunction's selectivity is +the product of the parts. Three unknown equality predicates on a 10,000,000-row +table: -And even with full stats, CROSS-column correlation is assumed away -(independence) unless you manually `CREATE STATISTICS`. VLDB'15's 10⁴× -errors (reading-how-good-optimizers.md) live exactly in that gap — a -constant guess powering million-dollar plan choices. +``` + sel(p1 ∧ p2 ∧ p3) = 0.005 × 0.005 × 0.005 = 1.25e-7 + estimated rows = 10,000,000 × 1.25e-7 = 1.25 + → clamped to 1 by clamp_row_est (costsize.c:215) +``` + +Postgres now believes one row comes out of a ten-million-row table, and will +happily put that on the inner side of a nested loop. That is the exact shape of +the failure the JOB paper measured (`reading-how-good-optimizers.md`, §4.1): +disabling non-index nested-loop joins removed *all* of their timeouts. + +Now the same arithmetic *with* statistics, on this topic's schema (`notes.md`: +`users` 10,000 rows, NDV(city) = 100, NDV(age) = 50): + +``` + sel(city = 'Paris') = 1/100 = 0.01 uniformity + sel(age = 30) = 1/50 = 0.02 uniformity + sel(both), independence = 0.01 × 0.02 = 0.0002 + estimated rows = 10,000 × 0.0002 = 2 +``` + +Perfect statistics, and still wrong the moment `city` and `age` correlate — +because cross-column correlation is assumed away unless you manually +`CREATE STATISTICS`. The JOB paper's Figure 3 lives exactly in that gap. + +Why it matters: everything upstream — Step 1's 560× access-path decision, +Step 2's whole DP, Step 5's genetic fitness function — consumes this number. It +is the cheapest and least accurate input in the entire optimizer. ## Where each step lives in the code -- **Steps 1–2 — the skeleton** (`path/allpaths.c`): `make_one_rel` :183 - — the whole story in one function name: from base relations to ONE - final rel. First `set_base_rel_pathlists` :384 (every table gets its - access paths: seqscan, index paths — Step 1), then the join search. -- **Step 5 gate, then Step 2** (`path/allpaths.c`): the dispatcher - (:3915): if `enable_geqo && levels_needed >= geqo_threshold` (default - 12) → GENETIC algorithm (the `geqo/` directory); else - `standard_join_search` :3952 — the level-by-level DP of Step 2's - diagram, with `join_search_one_level` (`path/joinrels.c:78`) doing the - per-level pairing and enforcing connectedness. -- **Steps 3–4** — `add_path` (in `util/pathnode.c`, but internalize the - Rust sketch above): multi-path DP cells, dominance across - (total_cost, startup_cost, ordering). -- **Step 6** — `src/include/utils/selfuncs.h` :34/:37/:40 for the - defaults; `selfuncs.c` for the histogram + MCV machinery. +All paths relative to the repo root of `postgres/postgres@701f021`. + +| Step | File | Lines | What is there | +|---|---|---|---| +| 1 | `src/include/optimizer/cost.h` | 24-28 | the five cost constants: 1.0 / 4.0 / 0.01 / 0.005 / 0.0025 | +| 1 | `src/backend/optimizer/path/costsize.c` | 300, 306-307, 339 | `cost_seqscan` — pages, CPU per tuple, and the total | +| 1 | `src/backend/optimizer/path/costsize.c` | 898 | `index_pages_fetched` — the Mackert–Lohman approximation | +| 1 | `src/backend/optimizer/path/costsize.c` | 215 | `clamp_row_est` — why an estimate is never 0 rows | +| 1-2 | `src/backend/optimizer/path/allpaths.c` | 183 | `make_one_rel` — the whole story in one name: base rels → one final rel | +| 1 | `src/backend/optimizer/path/allpaths.c` | 384 | `set_base_rel_pathlists` — every table gets its access paths | +| 5 | `src/backend/optimizer/path/allpaths.c` | 3847, 3913-3918 | `make_rel_from_joinlist` and the three-way dispatch (hook / geqo / DP) | +| 2 | `src/backend/optimizer/path/allpaths.c` | 3952, 3964-3968, 3974-3987 | `standard_join_search` — the level loop, and its own "dynamic programming" comment | +| 2 | `src/backend/optimizer/path/joinrels.c` | 78, 90-143 | `join_search_one_level` pass 1 — left- and right-sided plans, cartesian fallback at :139 | +| 2 | `src/backend/optimizer/path/joinrels.c` | 145-198 | pass 2 — bushy plans for every k from 2 to level/2 | +| 2 | `src/backend/optimizer/path/joinrels.c` | 699 | `make_join_rel` — symmetric in its two arguments, which is why pass 1 gets right-sided plans free | +| 3 | `src/backend/optimizer/util/pathnode.c` | 459, 391-412, 518-533 | `add_path` and its seven dominance axes | +| 3-4 | `src/backend/optimizer/util/pathnode.c` | 47, 181, 491-492 | `STD_FUZZ_FACTOR 1.01` and `compare_path_costs_fuzzily` | +| 2-3 | `src/backend/optimizer/util/pathnode.c` | 268 | `set_cheapest` — run per joinrel at the end of each level | +| 5 | `src/backend/utils/misc/guc_parameters.dat` | 1187-1194 | `geqo_threshold`, `boot_val => '12'` | +| 5 | `src/backend/optimizer/geqo/geqo_main.c` | 192, 198-204, 230, 328-350, 360-367 | the GA loop, ERX crossover, pool-size and generation formulas | +| 5 | `src/backend/optimizer/geqo/geqo_eval.c` | 140, 146-160, 163 | `gimme_tree` — tour in, clumped (possibly bushy) tree out | +| 5 | `src/include/optimizer/geqo.h` | 46, 57 | `#define ERX`, `DEFAULT_GEQO_EFFORT 5` | +| 6 | `src/include/utils/selfuncs.h` | 23-30, 34, 37, 40, 52 | the rationale comment, then the constants | +| 6 | `src/backend/utils/adt/selfuncs.c` | — | the histogram + MCV machinery the constants stand in for | + +Reproduce any row with: + +``` +python3 tools/pinned-source.py show postgres src/backend/optimizer/path/joinrels.c -r 145:198 +``` ## Questions for notes.md -1. Interesting orders: construct the query where the globally-cheapest - {AB} subplan loses — a sorted-but-pricier {AB} wins at level 3. -2. Why does geqo exist instead of DuckDB-style greedy? What does genetic - search preserve that greedy can't (hint: it searches TREES, not - sequences)? -3. Two costs (startup, total): which plan flips between `LIMIT 10` and - full result — index scan vs sort — and why does one number fail? -4. MCV lists fix single-column skew. Give the graph-shaped failure that - remains: super-node degree skew is a JOIN skew, invisible to - per-column stats. What stat would M10 need instead (degree histogram - per label?). +1. Interesting orders: construct the query where the globally-cheapest `{AB}` + subplan loses — a sorted-but-pricier `{AB}` wins at level 3 by feeding a + merge join. Which of `add_path`'s seven axes keeps it alive? +2. geqo encodes a join order as a *tour* and recombines it with ERX, a TSP + operator; `gimme_tree` then derives the tree. Why is searching sequences and + deriving trees a reasonable compromise, and what does `geqo_eval.c:146-157` + say went wrong with the strictly-in-tour-order version? +3. Two costs (startup, total): which plan flips between `LIMIT 10` and the full + result — index scan vs sort — and why does one number fail? Then explain why + postgres gates the extra path on `consider_startup` (`pathnode.c:426-429`) + rather than always keeping it. +4. `STD_FUZZ_FACTOR` is 1.01, and the comment blames "platform-specific plan + variations due to roundoff". What does that tell you about how much + confidence to place in a cost difference of 5%? +5. MCV lists fix single-column skew. Give the graph-shaped failure that + remains: super-node degree skew is a *join* skew, invisible to per-column + stats. What statistic would M10 need instead — a degree histogram per label? + +## Takeaway + +Postgres's optimizer is Selinger's 1979 skeleton with three additions that all +turned out to matter more than the skeleton: bushy plans in pass 2 of +`join_search_one_level`, a seven-axis dominance test in `add_path` instead of +"cheapest wins", and a genetic escape hatch for when `2^n` stops fitting. The +DP's advantage over naive enumeration is real and enormous — 5.3 million× at +n = 15 — but it is exponential either way, which is why the threshold at +`geqo_threshold = 12` exists. And every one of those decisions is driven by a +selectivity number that, absent statistics, is the constant `0.005` chosen in +1996 to keep index scans attractive. ## Done when -You can walk standard_join_search for A⋈B⋈C on paper, keeping two paths -per set (cheapest, interesting-order), and name the three default -selectivities from memory. +Answer each before unfolding it. + +- [ ] Walk `standard_join_search` for `A ⋈ B ⋈ C` on paper. How many levels, + what happens at each, and where do the bushy plans come from? +
Answer + + Three levels. Level 1 is `initial_rels`, assigned directly at + `allpaths.c:3976` — one `RelOptInfo` per base table, each holding the access + paths Step 1 built (`set_base_rel_pathlists`, `:384`). The loop at `:3978` + then runs `join_search_one_level` for lev = 2 and lev = 3 + (`joinrels.c:78`). At lev = 2, pass 1 pairs each level-1 rel with each other + initial rel it shares a join clause with (`:123`), producing `{AB}`, `{AC}`, + `{BC}` — pass 2 does nothing because it requires `k <= level - k`, i.e. + `2 <= 0`, false. At lev = 3, pass 1 joins each level-2 rel to the remaining + initial rel, and pass 2 again does nothing (k = 2 needs other_level = 1, + and `k > other_level` breaks at `:161`). So **A⋈B⋈C has no bushy plans at + all** — bushy first becomes possible at level 4, as 2+2. After each level, + `set_cheapest` (`pathnode.c:268`) runs on every finished joinrel. + +
+ +- [ ] For n = 5, 10 and 15 relations, compare n! against the number of + (set, last-relation) pairs the DP considers. Where does exhaustive search + die? +
Answer + + The DP considers `n·2^(n-1) − n` pairs and memoizes `2^n − 1` subsets: + + ``` + n n! DP considered memo n! / DP + 5 120 75 31 1.6 + 10 3,628,800 5,110 1,023 710.1 + 15 1,307,674,368,000 245,745 32,767 5,321,265.4 + ``` + + At n = 5 the DP is barely worth the bookkeeping. At n = 10 it is 710× less + work, at n = 15 it is 5.3 million× less. But look at the memo column: it is + still `2^n`, so the DP buys you roughly five more relations, not unlimited + scaling — which is precisely why `geqo_threshold` is 12 + (`guc_parameters.dat:1191`) and not 50. (Counts computed here for a complete + join graph; connectedness pruning makes the real numbers smaller.) + +
+ +- [ ] Name the three default selectivities and explain why `0.005` rather than + something rounder. +
Answer + + `DEFAULT_EQ_SEL 0.005` (`selfuncs.h:34`), `DEFAULT_INEQ_SEL 0.3333333333333333` + (`:37`), `DEFAULT_RANGE_INEQ_SEL 0.005` (`:40`). The comment at `:24-30` gives + the reason, and it is not accuracy: the values are "small enough to ensure + that indexscans will be used if available, for typical table densities of + ~100 tuples/page… 0.01 is not quite small enough, since that makes it appear + that nearly all pages will be hit anyway". It is a *policy* constant + disguised as an estimate. The same comment ties `DEFAULT_NUM_DISTINCT` to it: + 200 = 1/0.005 (`:52`), so the two fallbacks agree with each other. + +
+ +- [ ] Three unknown equality predicates on a 10,000,000-row table. What does + postgres estimate, and what does that make it do? +
Answer + + Independence makes selectivity multiplicative: `0.005³ = 1.25e-7`, so + `10,000,000 × 1.25e-7 = 1.25` rows, clamped to 1 by `clamp_row_est` + (`costsize.c:215`). Postgres now believes a ten-million-row table yields one + row, which makes it an ideal inner side for a nested loop — and if the truth + is 100,000 rows, that nested loop runs 100,000× more iterations than planned. + This is the mechanism behind the JOB paper's finding (§4.1) that disabling + non-index nested-loop joins removed *every* timeout from their workload. + +
+ +- [ ] `add_path` is often summarized as "keep the cheapest path plus one per + interesting order". What does that summary leave out? +
Answer + + Five of the seven axes. The real dominance test (`pathnode.c:391-412`, + `:518-533`) compares `disabled_nodes` (a *higher-order* term above cost, so a + path using no disabled node type wins regardless of cost, `:399-406`), + `startup_cost` and `total_cost` separately, `pathkeys`, `required_outer` + parameterization (`:518`), output `rows` (`:524`) and `parallel_safe` + (`:525`). It also compares costs **fuzzily** at `STD_FUZZ_FACTOR = 1.01` + (`:47`, used at `:491`), so a 1% cost difference is treated as a tie — the + comment at `:545-548` blames platform-specific floating-point roundoff. And + parameterized paths are forced to have no pathkeys (`:472-473`) so they + cannot win on sort order. + +
+ +- [ ] Above `geqo_threshold` relations, what exactly is being searched — trees + or sequences? +
Answer + + **Sequences.** A chromosome is a `Gene *tour` (`geqo_eval.c:140`), a + permutation of the relations, and the crossover operator is ERX — edge + recombination crossover (`geqo.h:46`, `geqo_main.c:198-204`) — lifted from + the travelling salesman problem. The *tree* is derived from the tour by + `gimme_tree` (`geqo_eval.c:163`), which greedily adds each relation to the + first "clump" it can legally join to (`:171-178`); that clump heuristic is + what allows bushy shapes. The comment at `:146-157` says the original version + joined strictly in tour order, "could never produce a 'bushy' plan", and + broke on queries whose only legal plans are bushy. Sizing at n = 12: + `pool_size = clamp(2^13, 50, 250) = 250`, `generations = pool_size = 250` + (`geqo_main.c:339-349`, `:366`), one evaluation per generation (`:230`) — + roughly 500 cost evaluations total. + +
## References -**Code** -- [postgres](https://github.com/postgres/postgres) — - `src/backend/optimizer/`: `path/allpaths.c` (make_one_rel, - standard_join_search), `path/joinrels.c` (join_search_one_level), - plus `src/include/utils/selfuncs.h` for the default selectivities; - ~1.5 h +**Code** — `postgres/postgres@701f021`, ~1.5 h +- `src/backend/optimizer/path/allpaths.c` — `make_one_rel`, + `make_rel_from_joinlist`, `standard_join_search`. Start here. +- `src/backend/optimizer/path/joinrels.c` — `join_search_one_level`; read both + passes. +- `src/backend/optimizer/util/pathnode.c` — `add_path`; read the header comment + before the body. +- `src/backend/optimizer/geqo/` — `geqo_main.c` and `geqo_eval.c`. +- `src/include/utils/selfuncs.h` and `src/backend/utils/adt/selfuncs.c` — the + constants and the statistics machinery. +- `src/backend/optimizer/README` — postgres's own prose explanation of the + above; read it if any of Step 3 was unclear. + +**Papers** +- Selinger, Astrahan, Chamberlin, Lorie, Price — "Access Path Selection in a + Relational Database Management System", SIGMOD 1979. The design this file + still implements; see `reading-selinger-cascades.md`. +- Leis et al. — "How Good Are Query Optimizers, Really?", PVLDB 2015. What + happens to Step 6's estimates on real data; see + `reading-how-good-optimizers.md`. + +**In this topic** +- `reading-selinger-cascades.md` — where the DP and interesting orders come + from, and the rule-driven alternative. +- `reading-duckdb-optimizer.md` — the same job, done in 2024, with a different + enumeration order and the same threshold of 12. diff --git a/topics/10-query-planning/reading-rust-planner-stack.md b/topics/10-query-planning/reading-rust-planner-stack.md index a6236d2..3e5fd64 100644 --- a/topics/10-query-planning/reading-rust-planner-stack.md +++ b/topics/10-query-planning/reading-rust-planner-stack.md @@ -1,182 +1,565 @@ # The Rust planner stack: Pratt parsing, rule traits, lazy frames -Three codebases, three Rust-shaped answers: sqlparser-rs (the parser -you'll use directly in the experiments), DataFusion's rules-as-a-trait -optimizer, and polars' rewrites-only lazy frames. M10's Cypher planner -will face every design choice DataFusion made. Before the code, this -chapter builds the five ideas these codebases embody — recursive-descent -parsing, Pratt expression parsing, rules as a trait, fixpoint driving, -and rewrites-only optimization — then maps each to its file:line. Read -for the shapes, not the SQL details. +Three codebases, three Rust-shaped answers: sqlparser-rs (the parser you'll use +directly in the experiments), DataFusion's rules-as-a-trait optimizer, and +polars' rewrites-only lazy frames. M10's Cypher planner will face every design +choice DataFusion made. Before the code, this chapter builds the five ideas +these codebases embody — recursive-descent parsing, Pratt expression parsing, +rules as a trait, fixpoint driving, and rewrites-only optimization — then maps +each to its file:line. Read for the shapes, not the SQL details. + +**Every `file:line` below was read at these pins:** +`apache/datafusion-sqlparser-rs@aeb616f`, `apache/datafusion@1e77af8`, +`pola-rs/polars@f8bcc3d`. Re-verify with +`python3 tools/pinned-source.py show -r A:B` rather than trusting +the numbers — these are fast-moving crates. **Topic 10 has no measured lane**; +nothing here is a timing taken on this machine, and none of it appears in +`FINDINGS.md`. The counts below (grammar rules, file proportions, precedence +levels) are exact figures read out of the pinned source. ## The problem in one sentence -Between "a string of SQL" and "a plan the executor can run" sit three -design decisions — how to parse expressions without 40 grammar rules, -how to organize dozens of rewrite rules so they stay testable, and when -you can skip cost-based planning entirely — and each of these three -codebases answers one of them well. +Between "a string of SQL" and "a plan the executor can run" sit three design +decisions — how to parse expressions without one grammar nonterminal per +precedence level (sqlparser-rs's table has 16 distinct levels), how to organize +dozens of rewrite rules so they stay testable, and when you can skip cost-based +planning entirely — and each of these three codebases answers one of them well. ## The concepts, step by step ### Step 1 — parsing: text to AST, by hand -A **parser** turns the query string into an **AST** (abstract syntax -tree — a tree of typed nodes mirroring the query's structure: a -SELECT node holding a list of expression nodes, a FROM node, and so on). -The two ways to build one: feed a grammar to a **parser generator** -(a tool that emits parser code from grammar rules), or write a -**recursive-descent** parser by hand — one function per grammar -construct, each consuming tokens and calling the functions for its -sub-constructs. sqlparser-rs is hand-written recursive descent, and this -is the norm, not the exception: postgres's gram.y aside, DuckDB -(libpg_query) and most production systems that started generated went -manual — because hand-written parsers give precise, human error messages -("expected ON after JOIN near line 3"), and parse errors are a database's -single most user-facing surface. sqlparser-rs also threads a `Dialect` -trait through every decision point — one AST, many SQLs — and its -`src/ast/` types are the de-facto Rust standard (DataFusion consumes -them directly). - -### Step 2 — Pratt parsing: precedence climbing in 30 lines +> **In:** a SQL string and a `Dialect`. +> **Out:** a `Vec` — an AST — or a `ParserError` with a position. +> This is `Parser::parse_sql`'s entire contract. + +A **parser** turns the query string into an **AST** (abstract syntax tree — a +tree of typed nodes mirroring the query's structure: a `Select` node holding a +list of expression nodes, a `From` node, and so on). Note what an AST is *not*: +it is not yet a **logical plan**, which is an expression in **relational +algebra** (select, project, join, aggregate) with resolved column references. +Binding the AST against a catalog to produce a logical plan is a separate stage, +which is why sqlparser-rs can be a standalone crate at all. + +The two ways to build a parser: feed a grammar to a **parser generator** (a +tool that emits parser code from grammar rules — postgres uses Bison, +`src/backend/parser/gram.y`), or write a **recursive-descent** parser by hand — +one function per grammar construct, each consuming tokens and calling the +functions for its sub-constructs. + +sqlparser-rs is hand-written recursive descent, and this is the norm, not the +exception. Hand-written parsers give precise, human error messages ("expected ON +after JOIN near line 3"), and parse errors are a database's single most +user-facing surface. The entry chain is three functions deep: + +``` + Parser::parse_sql(dialect, sql) src/parser/mod.rs:582 + → parse_statements() src/parser/mod.rs:531 -> Vec + → parse_statement() src/parser/mod.rs:626 -> Statement +``` + +Two design details worth stealing. First, a `Dialect` trait is threaded through +every decision point — one AST, many SQLs — including the precedence table +itself (Step 2). Second, recursion is bounded: `DEFAULT_REMAINING_DEPTH = 50` +(`src/parser/mod.rs:213`, installed at `:417`), decremented by a guard at the +top of `parse_subexpr` (`:1431`). A query with 51 levels of nested parentheses +gets a clean `ParserError` instead of a stack overflow — the correct answer to +"what does your parser do on hostile input". + +The `src/ast/` types are the de-facto Rust standard; DataFusion consumes them +directly. + +Why it matters: this is the layer you will actually write for M10, and the +tokenizer/AST split is the cheapest structural decision in the whole stack. + +### Step 2 — Pratt parsing: precedence climbing in one loop + +> **In:** a token stream positioned at the start of an expression, plus a +> minimum binding precedence (`0` at the top, from `prec_unknown`). +> **Out:** one `Expr` tree, correctly parenthesized by precedence and +> associativity, with the token stream positioned just past it. Expressions are the part of a grammar where recursive descent gets ugly: -`a + b * c > d AND e` must parse as `((a + (b*c)) > d) AND e`, and -encoding "* binds tighter than +" as grammar rules takes one rule per -precedence level — ~40 rules for SQL. **Pratt parsing** (also called -precedence climbing) replaces them all with one loop and a precedence -table: parse a prefix (literal, identifier, unary op, parenthesized -expression), then repeatedly ask "does the next token bind tighter than -the operator that called me?" — if yes, consume it and recurse with the -new precedence: +`a + b * c > d AND e` must parse as `((a + (b*c)) > d) AND e`, and encoding +"`*` binds tighter than `+`" in a classical layered grammar takes **one +nonterminal per precedence level**. + +**Count sqlparser-rs's levels.** The precedence table is a single match at +`src/dialect/mod.rs:981-1002`: + +``` + Period 100 Between 20 + DoubleColon 50 Eq 20 + AtTz 41 Like 19 + MulDivModOp 40 Is 17 + PlusMinus 30 PgOther 16 + Xor 24 UnaryNot 15 + Ampersand 23 And 10 + Caret 22 Or 5 + Pipe 21 (unknown) 0 prec_unknown, mod.rs:1005-1007 + Colon 21 + + 18 named variants, 16 distinct numeric levels +``` + +A layered grammar needs 16 nonterminals plus a primary-expression rule — 17 +productions, each of which must be edited and re-layered when you add an +operator. **Pratt parsing** (also called precedence climbing) replaces all 17 +with one loop plus that table: parse a prefix (literal, identifier, unary op, +parenthesized expression), then repeatedly ask "does the next token bind +tighter than the precedence I was called with?" — if yes, consume it and +recurse. + +```rust +// apache/datafusion-sqlparser-rs@aeb616f — src/parser/mod.rs +// (elided; the real body also handles compound field access and COLLATE) +1430 pub fn parse_subexpr(&mut self, precedence: u8) -> Result { +1431 let _guard = self.recursion_counter.try_decrease()?; +1433 let mut expr = self.parse_prefix()?; + // ... 1435-1445: parse_compound_expr, then optional COLLATE ... +1448 loop { +1449 let next_precedence = self.get_next_precedence()?; +1452 if precedence >= next_precedence { +1453 break; +1454 } + // ... 1456-1460: the Period operator is left to compound field access ... +1462 expr = self.parse_infix(expr, next_precedence)?; +1463 } +1464 Ok(expr) +1465 } +``` + +and the recursion is inside `parse_infix`, at the plain binary-operator arm: ```rust -fn parse_subexpr(&mut self, min_prec: u8) -> Expr { - let mut lhs = self.parse_prefix(); // literal, ident, unary, (…) - loop { - let prec = self.get_next_precedence(); // 0 if next isn't infix - if prec <= min_prec { return lhs; } // caller binds tighter: stop - let op = self.next_token(); - let rhs = self.parse_subexpr(prec); // recurse with MY precedence: - lhs = Expr::binary(lhs, op, rhs); // higher-prec ops bind first, - } // left-assoc falls out of <= -} -``` - -This is the 30-line answer to expression grammars that would take 40 -grammar rules; steal it verbatim for Cypher expressions in M10 — you -only write the precedence table. +// apache/datafusion-sqlparser-rs@aeb616f — src/parser/mod.rs +4049 Ok(Expr::BinaryOp { +4050 left: Box::new(expr), +4051 op, +4052 right: Box::new(self.parse_subexpr(precedence)?), +4053 }) +``` + +**Work the trace**, with the real numbers from the table above. `parse_expr` +(`:1404-1406`) enters at `prec_unknown() = 0`: + +``` + parse_subexpr(0) prefix -> a + next '+' = 30; 0 >= 30? no -> parse_infix(a, 30) + parse_subexpr(30) prefix -> b + next '*' = 40; 30 >= 40? no -> parse_infix(b, 40) + parse_subexpr(40) prefix -> c + next '>' = 20; 40 >= 20? YES -> break, return c + -> (b * c) + next '>' = 20; 30 >= 20? YES -> break, return (b * c) + -> (a + (b * c)) + next '>' = 20; 0 >= 20? no -> parse_infix(., 20) + parse_subexpr(20) prefix -> d + next AND = 10; 20 >= 10? YES -> break, return d + -> ((a + (b * c)) > d) + next AND = 10; 0 >= 10? no -> parse_infix(., 10) + parse_subexpr(10) prefix -> e + next EOF = 0; 10 >= 0? YES -> break, return e + -> (((a + (b * c)) > d) AND e) + next EOF = 0; 0 >= 0? YES -> break +``` + +Every `break` above is line 1452-1453 firing, and every descent is line 4052. + +**Now the associativity, which is the subtle part.** The comparison is `>=`, +not `>`. Take `a - b - c`, both operators at precedence 30: + +``` + parse_subexpr(0) prefix -> a + next '-' = 30; 0 >= 30? no -> parse_infix(a, 30) + parse_subexpr(30) prefix -> b + next '-' = 30; 30 >= 30? YES -> break <-- equality stops the recursion + -> (a - b) + next '-' = 30; 0 >= 30? no -> parse_infix((a-b), 30) -> c + -> ((a - b) - c) LEFT-associative +``` + +Equal precedence terminates the inner call, so the operator is left-associative. +Recurse at `precedence - 1` instead and the same operator becomes +right-associative. That single comparison is the whole associativity mechanism — +one character of code per associativity class. + +Why it matters: steal this verbatim for Cypher expressions in M10. You write +one loop, one `parse_prefix`, and a table; adding an operator is one match arm, +not a grammar re-layering. ### Step 3 — rewrite rules as a trait: one file, one rule, one test suite -An optimizer is a pile of **rewrite rules** (plan transformations that -are always safe — pushdown, constant folding; see the DuckDB guide). The -organizational question is how to keep 30+ of them from becoming one -giant pass. DataFusion's answer: every rule is an implementation of one -trait — +> **In:** a `LogicalPlan` and an `OptimizerConfig`. +> **Out:** a `Transformed` — the (possibly rewritten) plan plus a +> flag recording whether anything changed. That is the entire per-rule +> contract; everything else is the driver's job (Step 4). + +An optimizer is a pile of **rewrite rules**. Two kinds, and the distinction +matters in Step 5: a **transformation rule** turns a logical plan into an +equivalent logical plan (push a filter down, eliminate a cross join), while an +**implementation rule** turns a logical operator into a physical one (a join +becomes a hash join). Everything in this step is a transformation rule. + +The organizational question is how to keep 30+ of them from becoming one giant +pass. DataFusion's answer: every rule implements one trait +(`datafusion/optimizer/src/optimizer.rs:83`): + +```rust +// apache/datafusion@1e77af8 — datafusion/optimizer/src/optimizer.rs + 83 pub trait OptimizerRule: Debug { + 85 fn name(&self) -> &str; + // ... 86-90: doc comment ... + 91 fn apply_order(&self) -> Option { + 92 None + 93 } + // ... 94-134: doc comments ... + 135 fn rewrite( + // &self, plan: LogicalPlan, config: &dyn OptimizerConfig, + // ) -> Result> +``` + +Three parts, and the guide-level summaries usually mention only the middle one. + +- **`name`** (`:85`) — used for logging and, critically, for the error context + when a rule fails (`:671`, `:718`). You always learn *which* rule broke the + plan. +- **`apply_order`** (`:91`) — `Some(ApplyOrder::TopDown)`, + `Some(ApplyOrder::BottomUp)` or `None` (`:265-270`). This is real ordering + machinery: a rule declares how it wants the plan walked, and the driver + performs the traversal on its behalf (`:625-662`). `None` means "I recurse + myself". +- **`rewrite`** (`:135`) — returns `Transformed`, a wrapper + carrying a `transformed: bool`. + +The payoff is structural: one file per rule, each unit-testable in isolation. +And "each file's bottom half is its tests" is if anything an understatement — +measured at this pin: + +``` + push_down_filter.rs 4399 lines, #[cfg(test)] at :1424 -> 68% tests + eliminate_cross_join.rs 1558 lines, #[cfg(test)] at :490 -> 69% tests +``` + +The rest of the menu is the same rewrite set DuckDB's pipeline runs +(`reading-duckdb-optimizer.md`): `extract_equijoin_predicate.rs` +(`impl OptimizerRule` at `:51`), `decorrelate_predicate_subquery.rs` (`:56`). + +The cost of the design: rules cannot see each other, so all cooperation has to +happen through the driver. + +Why it matters: it is the cheapest way to make an optimizer contributable by +people who do not understand the whole optimizer — which is exactly the +position you are in when you start M10. + +### Step 4 — the fixpoint driver: repeat until the plan repeats + +> **In:** the initial `LogicalPlan` and the ordered `Vec>`. +> **Out:** the final plan, after at most `max_passes` full sweeps, plus an +> invariant check that the output schema still matches the input's. + +Given independent rules, who decides the order and when to stop? DataFusion's +`Optimizer::optimize` (`optimizer.rs:581`) runs *all* rules in sequence +(`:615`), then repeats the whole sequence. A **fixpoint loop** iterates until +the output stops changing. The interesting question is how it detects that. ```rust -trait OptimizerRule { - fn rewrite(&self, plan: LogicalPlan, config: &dyn OptimizerConfig) - -> Result>; -} +// apache/datafusion@1e77af8 — datafusion/optimizer/src/optimizer.rs + 598 let mut previous_plans = HashSet::with_capacity(16); + 599 previous_plans.insert(LogicalPlanSignature::new(&new_plan)); + // ... 601-603: stash the starting schema, i = 0 ... + 604 while i < options.optimizer.max_passes { + // ... 615-723: for rule in &self.rules { ... apply it ... } + 726 // HashSet::insert returns, whether the value was newly inserted. + 727 let plan_is_fresh = + 728 previous_plans.insert(LogicalPlanSignature::new(&new_plan)); + 729 if !plan_is_fresh { + 730 // plan did not change, so no need to continue trying to optimize + 731 debug!("optimizer pass {i} did not make changes"); + 732 break; + 733 } + 734 i += 1; + 735 } ``` -— where the `Transformed` wrapper carries a flag recording -"did anything actually change". The payoff is structural: one file per -rule (`push_down_filter.rs`, `eliminate_cross_join.rs`, -`extract_equijoin_predicate.rs`, `decorrelate_predicate_subquery.rs` — -the same rewrite menu as DuckDB's pipeline), each unit-testable in -isolation — each file's bottom half *is* its tests. The cost: rules -can't see each other, so cooperation must happen through the driver. +**This is the detail most summaries of DataFusion get wrong, so read lines +598 and 727 carefully.** The loop does *not* terminate on the `Transformed` +flag. It maintains a `HashSet` of the **signature of every plan it has ever +seen** and stops when a pass produces a plan it has seen before. The +`transformed` boolean is consumed only for logging (`:692-700`). + +That is strictly stronger than a change flag, because it also catches +**cycles**: if rule A rewrites P→Q and rule B rewrites Q→P, every pass reports +"changed" forever, but the signature set sees P on pass 2 and breaks. A +`LogicalPlanSignature` is a pair — `node_number` and a `plan_hash` from +`DefaultHasher` (`datafusion/optimizer/src/plan_signature.rs:31-33`, built at +`:62-69`, node count at `:74`) — so it is a hash comparison, not a deep plan +walk. + +`max_passes` defaults to **3** (`datafusion/common/src/config.rs:1559`, +`default = 3`), overridable via `with_max_passes` (`optimizer.rs:226`). -### Step 4 — the fixpoint driver: repeat until nothing changes +Compare the three engines in this course, all verified at their pins: + +``` + driver style detection of "done" bound + ──────────────────── ───────────────────────────────────── ────────────── + DataFusion plan-signature set max_passes = 3 + optimizer.rs:604 optimizer.rs:598, :727-733 config.rs:1559 + polars boolean `changed` flag none — loops + stack_opt.rs:34 stack_opt.rs:23, :36, :45 until stable + DuckDB n/a — no fixpoint at all each pass runs + optimizer.cpp:178 hand-ordered list of 39 calls exactly once +``` -Given independent rules, who decides the order? DataFusion's driver runs -ALL rules in sequence, then repeats the whole sequence up to `max_passes` -times (default 3) or until a full pass reports no change — a **fixpoint -loop** (iterate until the output stops changing). Compare DuckDB: each -pass runs exactly once, in a hand-tuned order (pullup deliberately before -pushdown). The trade: +The trade, restated honestly now that the mechanism is right: ``` fixpoint of all rules (DataFusion) once, in order (DuckDB) ────────────────────────────────── ───────────────────────── - no ordering cleverness needed order encodes expert knowledge + no global ordering to hand-tune order encodes expert knowledge catches rule-enables-rule chains misses them unless ordered right pays repeated plan traversals one traversal per pass - rules must be idempotent-ish rules may assume predecessors ran + needs a termination oracle terminates by construction + rules should be idempotent-ish rules may assume predecessors ran ``` -The `Transformed` flag is what makes the fixpoint terminate — a rule that -always reports "changed" spins the driver to max_passes every time -(question 4 below). +Note that DataFusion has *not* escaped ordering entirely: the rule vector is +still an ordered list (`Optimizer::new`, `:280`; `with_rules`, `:325`), and +`apply_order` (Step 3) hand-picks a traversal direction per rule. What the +fixpoint buys is tolerance of a *slightly wrong* order, not freedom from order. + +Why it matters: if you build M10 rule-by-rule, this is the decision that +determines whether adding rule 31 can break rules 1-30. ### Step 5 — rewrites-only optimization: what polars gets away with +> **In:** a lazy `IR` plan built by dataframe method calls, plus `OptFlags`. +> **Out:** an optimized `IR` — with no join reordering, because the join order +> was in the input. + polars is a dataframe library with a real query optimizer hiding inside: -`.lazy()` builds a plan IR instead of executing eagerly, `.collect()` -optimizes then executes. Its optimizer directory reads like a mini -DuckDB — `predicate_pushdown/`, `projection_pushdown/`, -`simplify_expr/`, `cse/`, `collapse_and_project.rs`, -`delay_rechunk.rs` — but what's MISSING is the lesson: no cost-based -join reordering to speak of. It can skip it because a dataframe program -*is* an explicit plan — the user already wrote the join order, method -call by method call. Rewrites-only optimization is viable exactly when -the API hands you the order. The M10 corollary: Cypher gives no such -luck — a MATCH pattern names *relationships*, not an order, so pattern → -expansion order is a genuine cost-based choice (anchor selection). A -FalkorDB planner cannot be polars. +`.lazy()` builds a plan IR instead of executing eagerly, `.collect()` optimizes +then executes. Its optimizer module list +(`crates/polars-plan/src/plans/optimizer/mod.rs:8-37`) reads like a mini +DuckDB — `predicate_pushdown` (`:30`), `projection_pushdown` (`:31`), +`simplify_expr` (`:32`), `cse` (`:14`), `collapse_and_project` (`:11`), +`delay_rechunk` (`:8`), `cluster_with_columns` (`:10`), `slice_pushdown_lp` +(`:35`), `fused` (`:19`). + +The top-level `optimize` (`:85`) is an explicit hand-ordered sequence gated on +`OptFlags` — `simplify_expr` (`:134`), `comm_subplan_elim` (`:142`), +`predicate_pushdown` (`:176`), `projection_pushdown` (`:208`), then +`simplify_expr` *again* (`:224`) — while the expression-level rules run under +`StackOptimizer::optimize_loop` (`stack_opt.rs:16`), a boolean-flag fixpoint +(`:23`, `:34`, `:36`, `:45`). So polars is both shapes at once: ordered +pipeline outside, fixpoint inside. + +**What is MISSING is the lesson.** Grep the module list for join reordering and +you find exactly one join-named entry, `join_utils` (`:20`), which re-exports +`ExprOrigin` — a helper that classifies which side of a join an expression came +from. There is no cost model, no cardinality estimator, no join enumeration. + +It can skip all of that because a dataframe program *is* an explicit plan — the +user already wrote the join order, method call by method call. `df.join(a).join(b)` +is not a declarative request that the system is free to reorder; it is an +instruction. Rewrites-only optimization is viable exactly when the API hands you +the order. + +The M10 corollary: Cypher gives no such luck. A `MATCH` pattern names +*relationships*, not an order — `MATCH (a)-[:R]->(b)-[:S]->(c)` says nothing +about whether to start from `a`, `b` or `c`. Pattern → expansion order is a +genuine cost-based choice (anchor selection), so a FalkorDB planner cannot be +polars; it has to be at least Step 3 + Step 4, and probably Selinger +(`reading-postgres-optimizer.md`). + +Why it matters: it tells you exactly which half of this topic you are allowed +to skip, and the test is a property of your *API*, not of your engine. ## Where each step lives in the code -- **Step 1 — sqlparser-rs** (`src/parser/mod.rs`): entry `parse_sql` - :582 → `parse_statements` :531 → `parse_statement` :626 — the - hand-written recursive descent. The `Dialect` trait plumbing is - threaded throughout; AST types in `src/ast/`. -- **Step 2 — the heart**: `parse_subexpr` :1428–1450 with - `get_next_precedence` :1449 — match the Rust sketch above against the - real thing. -- **Steps 3–4 — DataFusion** (`optimizer/src/optimizer.rs`): - `OptimizerRule` :83, its `rewrite` returning `Transformed` - (:135); the driver `optimize` :581 with `max_passes` :604. Then skim - the one-file-per-rule menu: `push_down_filter.rs`, - `eliminate_cross_join.rs`, `extract_equijoin_predicate.rs`, - `decorrelate_predicate_subquery.rs`. -- **Step 5 — polars** (`crates/polars-plan/src/plans/optimizer/`): read - the directory listing as much as the code — `predicate_pushdown/`, - `projection_pushdown/`, `simplify_expr/`, `cse/`, - `collapse_and_project.rs`, `delay_rechunk.rs` — and note what isn't - there. +| Step | Repo @ pin | File | Lines | What is there | +|---|---|---|---|---| +| 1 | sqlparser-rs @ `aeb616f` | `src/parser/mod.rs` | 582 | `parse_sql` — the entry point | +| 1 | sqlparser-rs | `src/parser/mod.rs` | 531 | `parse_statements` → `Vec` | +| 1 | sqlparser-rs | `src/parser/mod.rs` | 626 | `parse_statement` — the recursive-descent root | +| 1 | sqlparser-rs | `src/parser/mod.rs` | 213, 417, 1431 | `DEFAULT_REMAINING_DEPTH = 50` and its guard | +| 2 | sqlparser-rs | `src/parser/mod.rs` | 1404-1406 | `parse_expr` — enters at `prec_unknown()` | +| 2 | sqlparser-rs | `src/parser/mod.rs` | **1430-1465** | `parse_subexpr` — the Pratt loop; break at :1452 | +| 2 | sqlparser-rs | `src/parser/mod.rs` | **4452** | `get_next_precedence` (definition; :1449 is the call) | +| 2 | sqlparser-rs | `src/parser/mod.rs` | 3833, 4049-4053 | `parse_infix`, and the binary-op recursion | +| 2 | sqlparser-rs | `src/dialect/mod.rs` | 981-1002, 1005 | `prec_value` — the whole precedence table | +| 3 | datafusion @ `1e77af8` | `datafusion/optimizer/src/optimizer.rs` | 83, 85, 91, 135 | `OptimizerRule`: `name`, `apply_order`, `rewrite` | +| 3 | datafusion | `datafusion/optimizer/src/optimizer.rs` | 265-270 | `ApplyOrder::{TopDown, BottomUp}` | +| 3 | datafusion | `datafusion/optimizer/src/push_down_filter.rs` | 761, 1424 | the rule, then 68% of the file in tests | +| 3 | datafusion | `datafusion/optimizer/src/eliminate_cross_join.rs` | 77, 490 | same shape, 69% tests | +| 4 | datafusion | `datafusion/optimizer/src/optimizer.rs` | 581, 604, 615 | `optimize`, the pass loop, the rule loop | +| 4 | datafusion | `datafusion/optimizer/src/optimizer.rs` | 598, 727-733 | the plan-signature set — the real termination test | +| 4 | datafusion | `datafusion/optimizer/src/plan_signature.rs` | 31-33, 62-69, 74 | `LogicalPlanSignature` = (node_number, plan_hash) | +| 4 | datafusion | `datafusion/common/src/config.rs` | 1559 | `max_passes, default = 3` | +| 5 | polars @ `f8bcc3d` | `crates/polars-plan/src/plans/optimizer/mod.rs` | 8-37 | the module list — read it as a menu | +| 5 | polars | `crates/polars-plan/src/plans/optimizer/mod.rs` | 85, 134, 142, 176, 208, 224 | `optimize` — the hand-ordered sequence | +| 5 | polars | `crates/polars-plan/src/plans/optimizer/stack_opt.rs` | 16, 23, 34, 36, 45 | `optimize_loop` — a boolean-flag fixpoint | + +Reproduce any row with: + +``` +python3 tools/pinned-source.py show sqlparser-rs src/dialect/mod.rs -r 981:1007 +``` ## Questions for notes.md -1. Trace `a + b * c > d AND e` through parse_subexpr by hand (precedence - table lookups included). Now write the Cypher expression subset you - need for M10 and its precedence table. -2. DataFusion's fixpoint-of-all-rules vs DuckDB's once-in-order: which - catches `filter → (rewrite exposes new filter) → filter` chains, and - what's the worst-case cost? -3. Why can polars skip join reordering but FalkorDB can't? Where exactly - does Cypher hide the join order decision (pattern → expansion order)? -4. The `Transformed` flag: why does a fixpoint driver need rules to - report changes honestly — what breaks with a rule that always says - "changed"? +1. Trace `a + b * c > d AND e` through `parse_subexpr` by hand with the real + numbers (40/30/20/10/0) and check your tree against the trace in Step 2. + Then write the Cypher expression subset you need for M10 and its precedence + table — how many distinct levels? +2. `parse_subexpr` breaks on `precedence >= next_precedence`. Show that this + makes `-` left-associative, then show what you would change to make `^` + right-associative. How many characters is the diff? +3. DataFusion's plan-signature fixpoint vs DuckDB's once-in-order pipeline: + which catches `filter → (rewrite exposes new filter) → filter` chains, what + is the worst-case cost, and which one can loop forever if you get it wrong? +4. Why can polars skip join reordering but FalkorDB can't? Point at the exact + place Cypher hides the join order decision, and name the polars module that + *would* have to exist. +5. `push_down_filter.rs` is 68% tests. What does that ratio tell you about the + real cost of adding rule number 31 to an optimizer, and how should that + change your M10 plan? + +## Takeaway + +Three separable decisions, three verified answers. Expression parsing: one loop +plus a 16-level table (`parser/mod.rs:1430`, `dialect/mod.rs:981`) replaces 17 +grammar nonterminals, and one `>=` decides associativity. Rule organization: +one trait with `name`/`apply_order`/`rewrite` (`optimizer.rs:83`) buys you +one-file-per-rule and 68%-tests-by-line. Fixpoint termination: not a change +flag but a set of plan signatures (`optimizer.rs:598`, `:727-733`), which is +what makes rule cycles terminate rather than spin. And the whole cost-based +half of this topic is skippable exactly when your API already specifies the +join order — which polars' does and Cypher's does not. ## Done when -You can parse an expression with Pratt precedence on paper, and argue -rules-as-trait-with-fixpoint vs ordered-pass-pipeline for M10 (pick one, -justify in notes.md). +Answer each before unfolding it. + +- [ ] Parse `a + b * c > d AND e` on paper using sqlparser-rs's real precedence + numbers. What are the numbers, and where does each recursion stop? +
Answer + + From `src/dialect/mod.rs:981-1002`: `*` = MulDivModOp = 40, `+` = PlusMinus = + 30, `>` = Eq = 20, `AND` = And = 10, and the top-level entry is + `prec_unknown() = 0` (`:1005-1007`). `parse_subexpr(0)` takes `a`, sees `+` + (30 > 0) and recurses at 30; that call takes `b`, sees `*` (40 > 30) and + recurses at 40; that call takes `c`, sees `>` at 20 and **breaks because + 40 >= 20**, yielding `(b*c)`; back at 30, `>` at 20 breaks again, yielding + `(a + (b*c))`. Then `>` is consumed at 0, `d` is parsed at 20 and stops at + `AND` (20 >= 10), and finally `AND` is consumed and `e` parsed. Result: + `(((a + (b * c)) > d) AND e)`. Every stop is line 1452-1453; every descent is + line 4052. + +
+ +- [ ] Why is `a - b - c` parsed left-associatively, and what one change would + make it right-associative? +
Answer + + Both `-` are at precedence 30, and the loop test is + `if precedence >= next_precedence { break; }` (`:1452`). The inner + `parse_subexpr(30)` sees the second `-` at 30, finds `30 >= 30` **true**, and + breaks — so the inner call returns just `b` and the outer loop builds + `(a - b)` before consuming the second `-`, giving `((a - b) - c)`. Equal + precedence terminating the recursion *is* left-associativity. To make an + operator right-associative you recurse at `precedence - 1` instead of + `precedence` (line 4052), so the equal-precedence operator is `>` the + threshold and gets absorbed by the inner call. One character of arithmetic + per associativity class. + +
+ +- [ ] How many grammar productions does Pratt parsing save here, exactly? +
Answer + + sqlparser-rs's table (`dialect/mod.rs:981-1002`) has **18 named variants at + 16 distinct numeric levels** — 100, 50, 41, 40, 30, 24, 23, 22, 21, 20, 19, + 17, 16, 15, 10, 5 — plus `prec_unknown() = 0`. A classical layered expression + grammar needs one nonterminal per distinct level plus a primary rule: **17 + productions**, every one of which has to be edited and re-layered to insert a + new operator. Pratt replaces them with `parse_subexpr` (`:1430-1465`, ~36 + lines including the compound-expression and COLLATE handling) and the table. + Adding an operator is one match arm. + +
+ +- [ ] What actually terminates DataFusion's optimizer loop? (It is not the + `Transformed` flag.) +
Answer + + A `HashSet` of every plan seen so far. It is seeded + before the loop (`optimizer.rs:598-599`) and re-inserted after each full pass + (`:727-728`); `HashSet::insert` returning `false` means this plan was already + seen, and the loop breaks (`:729-733`). The `transformed` boolean from + `Transformed` is only used for logging (`:692-700`). This matters + because it makes **cycles** terminate: if rule A rewrites P→Q and rule B + rewrites Q→P, every pass honestly reports "changed" forever, and a + change-flag driver would spin to `max_passes` every single time. A + `LogicalPlanSignature` is `(node_number, plan_hash)` + (`plan_signature.rs:31-33`, `:62-69`), so the check is a hash lookup. The + hard bound is `max_passes`, default **3** (`common/src/config.rs:1559`). + +
+ +- [ ] Does DataFusion's rule trait really eliminate ordering concerns? +
Answer + + No, it relocates them. Two mechanisms survive. First, the rule list is still + an ordered `Vec` applied in sequence each pass (`optimizer.rs:615`, built by + `Optimizer::new` at `:280` / `with_rules` at `:325`). Second, each rule + declares an `apply_order` (`:91`) of `TopDown`, `BottomUp` or `None` + (`:265-270`), and the driver performs that traversal on the rule's behalf + (`:625-662`) — filter pushdown and projection pushdown genuinely want + opposite directions. What the fixpoint buys is *tolerance of a slightly wrong + order*, since a rule enabled by a later rule gets another chance next pass. + It does not buy order-independence. + +
+ +- [ ] polars ships a full pushdown optimizer but no join reordering. Why is + that not a bug? +
Answer + + Because a dataframe program is already an explicit plan. `df.join(a).join(b)` + is an instruction, not a declarative request the system may reorder — the + user chose the order when they wrote the method chain. The module list + (`crates/polars-plan/src/plans/optimizer/mod.rs:8-37`) confirms the absence: + the only join-named entry is `join_utils` (`:20`), an `ExprOrigin` helper, + and there is no cost model or cardinality estimator anywhere in the + directory. Rewrites-only optimization is viable exactly when the API supplies + the order. Cypher does not: `MATCH (a)-[:R]->(b)-[:S]->(c)` names + relationships, not a traversal order, so anchor selection and expansion order + are genuine cost-based choices — which is why M10 needs Step 3 + Step 4 at + minimum and probably Selinger's DP as well. + +
## References **Code** -- [sqlparser-rs](https://github.com/apache/datafusion-sqlparser-rs) — - `src/parser/mod.rs` (parse_subexpr is the heart), `src/ast/` -- [datafusion](https://github.com/apache/datafusion) — - `optimizer/src/optimizer.rs` (OptimizerRule trait + fixpoint driver), - then skim the one-file-per-rule menu -- [polars](https://github.com/pola-rs/polars) — - `crates/polars-plan/src/plans/optimizer/` — read the directory listing - as much as the code; what's MISSING is the lesson +- [sqlparser-rs](https://github.com/apache/datafusion-sqlparser-rs) @ `aeb616f` + — `src/parser/mod.rs` (`parse_subexpr` at :1430 is the heart), + `src/dialect/mod.rs:981-1002` (the precedence table), `src/ast/`. ~40 min. +- [datafusion](https://github.com/apache/datafusion) @ `1e77af8` — + `datafusion/optimizer/src/optimizer.rs` (the trait at :83, the driver at + :581), `datafusion/optimizer/src/plan_signature.rs`, then skim the + one-file-per-rule menu. ~40 min. +- [polars](https://github.com/pola-rs/polars) @ `f8bcc3d` — + `crates/polars-plan/src/plans/optimizer/mod.rs`: read the module list at + :8-37 and the `optimize` sequence at :85 as much as the code; what's MISSING + is the lesson. ~20 min. + +**In this topic** +- `reading-duckdb-optimizer.md` — the same rewrite menu as an ordered, + run-once pipeline, plus the cost-based join enumeration polars omits. +- `reading-postgres-optimizer.md` — what Step 5 says M10 cannot avoid. +- `reading-selinger-cascades.md` — where "transformation rule vs implementation + rule" comes from, and the rule-driven optimizer generator DataFusion's trait + is a distant descendant of. diff --git a/topics/10-query-planning/reading-selinger-cascades.md b/topics/10-query-planning/reading-selinger-cascades.md index 88356c2..d1dce2f 100644 --- a/topics/10-query-planning/reading-selinger-cascades.md +++ b/topics/10-query-planning/reading-selinger-cascades.md @@ -1,204 +1,722 @@ # Selinger and Cascades: the two optimizer architectures -Two papers, 16 years apart, that define the design space every optimizer -lives in: Selinger '79 invented cost-based join search as bottom-up DP; -Graefe's Cascades '95 turned the whole optimization process into rules -firing in a memo. Before the papers, this chapter builds the eight ideas -they contributed, one at a time — cost as a number, selectivity factors, -access paths and interesting orders, the DP itself, then the memo, rules, -and top-down search — and closes with the comparison that decides M10. -Read Selinger closely (it's short and shockingly modern), then Cascades -for the generalization. +Two papers, 16 years apart, that define the design space every optimizer lives +in: Selinger '79 invented cost-based join search as bottom-up DP; Graefe's +Cascades '95 turned the whole optimization process into rules firing in a memo. +Before the papers, this chapter builds the eight ideas they contributed, one at +a time — cost as a number, selectivity factors, access paths and interesting +orders, the DP itself, then the memo, rules, and top-down search — and closes +with the comparison that decides M10. Read Selinger closely (it's short and +shockingly modern), then Cascades for the generalization. + +**Every figure and formula below is quoted from one of these papers and named +to its section, table or figure**, or is arithmetic done here on stated +assumptions. Where a *runtime* number is needed it comes from the JOB paper +(`reading-how-good-optimizers.md`) with its section cited, because **topic 10 +deliberately has no measured lane** — its harness measures only your own code — +so nothing here is a timing from this machine and none of it appears in +`FINDINGS.md`. Selinger is quoted from the SIGMOD 1979 proceedings text; +Cascades from the IEEE Data Engineering Bulletin 18(3), 1995. ## The problem in one sentence -In 1979 nobody knew how to make a machine choose among the exponentially -many ways to evaluate one declarative query — Selinger's answer (estimate -costs, search bottom-up with DP) still ships in postgres, DuckDB, and -SQLite, and Cascades' answer (make the search itself programmable) ships -in SQL Server and CockroachDB. +In 1979 nobody knew how to make a machine choose among the exponentially many +ways to evaluate one declarative query — Selinger's answer (estimate costs, +search bottom-up over subsets rather than over the n! orderings) still ships in +postgres, DuckDB and SQLite, and Cascades' answer (make the search itself +programmable, as rules firing into a memo) ships in SQL Server, CockroachDB and +Orca. ## The concepts, step by step ### Step 1 — cost as a single number: weighted IO + CPU -To compare plans, you need each plan reduced to one comparable number. -Selinger's formula: `COST = PAGE FETCHES + W × RSI CALLS` — disk page -reads, plus CPU work (RSI calls — tuple-fetch calls into System R's -storage interface) scaled by a tuning weight W that says how many CPU -operations equal one IO. One formula, two resources. Everything since is -elaboration: modern engines still argue about W, and as storage moves -NVMe→RAM the right W shifts by ~100× — enough to flip plan choices -(question 1 below). +> **In:** a candidate access plan, plus System R's catalog statistics. +> **Out:** one scalar, so that two plans become comparable with `<`. This is +> what a **cost model** is: a formula from plan to number, intended to be +> monotone in runtime. + +To compare plans you need each plan reduced to one comparable number. Selinger's +formula, stated verbatim in §3: + +``` + COST = PAGE FETCHES + W * (RSI CALLS) + + PAGE FETCHES disk pages read — the I/O term + RSI CALLS tuples returned across the Research Storage Interface, + System R's tuple-at-a-time storage API — the CPU proxy + W "an adjustable weighting factor between I/O and CPU" +``` + +The paper's justification for using tuple count as the CPU term: "Since most of +System R's CPU time is spent in the RSS, the number of RSI calls is a good +approximation for CPU utilization." One formula, two resources, one knob. The +paper gives **no numeric value for W** — it is left as a tuning parameter +throughout. + +**Work out what W has become, and what happens when it moves.** Postgres's +modern equivalents (`src/include/optimizer/cost.h:24-28` at `701f021`) are +`seq_page_cost = 1.0`, `random_page_cost = 4.0`, `cpu_tuple_cost = 0.01`. In +Selinger's units — cost denominated in sequential page fetches — that makes +`W = 0.01`: one page fetch is worth 100 tuples of CPU. Against a *random* page +it is `4.0/0.01 = 400`, which is exactly the ratio the JOB paper flags in §5.3 +when it says PostgreSQL's defaults imply "processing a tuple is 400× cheaper +than reading it from a page". JOB §5.3 then *measures* the correction for a +main-memory machine: scaling the CPU cost parameters up by **50×** improved the +median runtime-prediction error from 38% to 30%. So `W: 0.01 → 0.5`. + +Now push that through Selinger's own formula. Take a 10,000,000-row table in +100,000 pages, and compare a full segment scan against a non-clustered index +scan returning `k` rows (the paper's assumption for a large relation: one page +fetch per tuple retrieved, §4's TABLE 2 discussion): + +``` + segment scan = 100,000 + W × 10,000,000 + index scan = k + W × k = k(1 + W) + + W = 0.01 scan = 200,000 crossover at k = 198,020 = 2.0% of table + W = 0.50 scan = 5,100,000 crossover at k = 3,400,000 = 34.0% of table +``` + +A 50× change in one constant moves the selectivity at which the optimizer +abandons the index from 2% of the table to 34% — a **17× shift in the crossover +point**. That is why W is not a detail: it is the parameter that decides which +half of your plans are index scans. (Arithmetic done here on the stated +assumptions; the 50× is JOB §5.3's measurement, the constants are postgres's.) + +Why it matters: everything since is elaboration on this one line, and the +elaboration is mostly about W. ### Step 2 — selectivity factors: guessing what a predicate keeps -Costs depend on how many rows flow between operators, so Selinger's §4 -introduces the **selectivity factor** — the estimated fraction of rows a -predicate keeps. For `col = value` with an index: 1/ICARD(index), where -ICARD is the number of distinct keys — i.e., assume every value equally -frequent. This is the **uniformity assumption**, born here, and it is -still the default in every engine you'll read. So are the fallback -constants: the paper's table says equality with no information = 1/10 — -the direct grandparent of postgres's `DEFAULT_EQ_SEL = 0.005`. A guess -from 1979, wearing modern clothes, still powering plan choices. +> **In:** a boolean factor (one conjunct of the WHERE clause in conjunctive +> normal form) and whatever the catalog knows. +> **Out:** a **selectivity factor** F in [0,1] — "the expected fraction of +> tuples which will satisfy the predicate" (§4) — from which **cardinality** +> (the row count of an intermediate result) follows by multiplication. + +Costs depend on how many rows flow between operators, so §4 introduces the +selectivity factor and gives TABLE 1, which is worth reading as a list of +assumptions rather than a list of formulas: + +``` + TABLE 1 (§4), the entries that matter + + column = value F = 1 / ICARD(column index) if an index exists + F = 1/10 otherwise + column1 = column2 F = 1 / MAX(ICARD1, ICARD2) if both indexed + F = 1 / ICARD(i) if only one indexed + F = 1/10 otherwise + column > value F = (high - value)/(high - low) if arithmetic & known + F = 1/3 otherwise + col BETWEEN a AND b F = (b - a)/(high - low), else 1/4 + col IN (list) F = n × F(=), capped at 1/2 + p1 OR p2 F = F1 + F2 - F1 × F2 + p1 AND p2 F = F1 × F2 + NOT p F = 1 - F +``` + +Four assumptions are born in that table, and all four are still the default in +every engine you will read: + +- **Uniformity.** `F = 1/ICARD` for equality — ICARD is the index's count of + distinct keys — with the paper's own gloss: "This assumes an even + distribution of tuples among the index key values." A **histogram** (a table + of value ranges with the row count in each) is precisely the later fix for + this, and postgres's MCV lists are the fix for its worst case. +- **Independence.** The `AND` row is a bare product, and the paper says so + directly: "Note that this assumes that column values are independent." +- **The containment / inclusion assumption**, stated for joins: "This assumes + that each key value in the index with the smaller cardinality has a matching + value in the other index." Note that `F = 1/MAX(ICARD1, ICARD2)` is + *literally* the formula the JOB paper audits 36 years later as PostgreSQL's + join estimator (`|T1||T2| / max(dom(x), dom(y))`, JOB §2.3). +- **Honest fallback constants.** The paper does not pretend these are + measurements. On the 1/3: "There is no significance to this number, other + than the fact that it is less selective than the guesses for equal predicates + for which there are no indexes, and that it is less than 1/2. We hypothesize + that few queries use predicates that are satisfied by more than half the + tuples." + +**Trace the lineage, because one constant survived to the digit.** Postgres at +`701f021` defines (`src/include/utils/selfuncs.h`): + +``` + Selinger 1979 postgres 701f021 verdict + column > value F = 1/3 DEFAULT_INEQ_SEL 0.3333333333333333 identical + (selfuncs.h:37) to the digit + column = value F = 1/10 DEFAULT_EQ_SEL 0.005 20× tighter + (selfuncs.h:34) +``` + +The inequality guess is unchanged after 46 years. The equality guess was +tightened 20×, and postgres's comment at `selfuncs.h:24-30` says why — not +because 1/10 was measured wrong, but because the defaults must be "small enough +to ensure that indexscans will be used if available". It is a policy constant, +same as Selinger's was. + +Why it matters: this is the input that JOB §3 shows is wrong by orders of +magnitude on real data. Reading TABLE 1 tells you *which* assumption each error +came from. ### Step 3 — access path selection, plus the interesting-orders refinement -For each single relation, cost every way to read it — each index versus -a full segment scan (an **access path** is one such concrete way) — and -keep the cheapest. But Selinger keeps more than one: also the cheapest -path per **interesting order** — a sort order of the output that some -later operator could exploit (a merge join, ORDER BY, GROUP BY). A -pricier-but-sorted path can win globally by saving a sort later, so -sortedness becomes part of the DP state. This one refinement is what -makes merge-join plans findable at all, and it's the state postgres kept -and DuckDB dropped (see those guides). +> **In:** one relation, its indexes, and the boolean factors that apply to it. +> **Out:** *several* surviving plans, not one — the cheapest unordered access +> path plus the cheapest path producing each **interesting order**. That plural +> is the whole content of this step. + +An **access path** is one concrete way to read a single relation: one of its +indexes, or a full segment scan. Cost every one with §4's TABLE 2 formulas and +keep the cheapest — that is what "access path selection", the paper's title, +means. + +But Selinger keeps more than the cheapest, and here is where summaries get +imprecise. The paper's definition is **enumerable and explicit**, not a vague +"any order a later operator might like". Stated first for single relations: + +> "We say that a tuple order is an **interesting order** if that order is one +> specified by the query block's GROUP BY or ORDER BY clauses." + +and then extended, in the join section: + +> "As in the single relation case, 'interesting' orders are those listed in the +> query block's GROUP BY or ORDER BY clause, if any. **Also every join column +> defines an 'interesting' order.**" + +So the set is exactly: `ORDER BY columns ∪ GROUP BY columns ∪ every join +column`. Finite, computable from the query text before search begins, and +usually small. That precision is what makes the refinement affordable — you are +not keeping a plan per *possible* sort order, you are keeping one per member of +a short list. + +The paper is equally explicit about the payoff and the alternative: "If there +are GROUP BY or ORDER BY clauses, then the cost for producing that interesting +ordering must be compared to the cost of the cheapest unordered path **plus** +the cost of sorting QCARD tuples into the proper order." A pricier-but-sorted +path wins globally exactly when it saves more than the sort would cost. + +Note the important negative: "If there are no GROUP BY or ORDER BY clauses on +the query, then there will be no interesting orderings, and the cheapest access +path is the one chosen." The refinement costs nothing on queries that cannot +use it. + +Why it matters: sortedness becomes part of the search state, which is what +makes merge-join plans findable at all. It is the state postgres kept (as +pathkeys, `reading-postgres-optimizer.md`) and DuckDB dropped +(`reading-duckdb-optimizer.md`), and Step 7 shows Cascades reinventing it from +the other direction. ### Step 4 — the DP: best plans compose from best subplans -The join search (§5) is **dynamic programming**: the best plan joining a -set of n relations must be "best plan for some (n−1)-subset" joined with -the remaining relation — so compute and memoize best plans for sets of -size 1, then 2, then 3… Selinger restricts to **left-deep** trees (the -right input of every join is a base relation, never another join) — -smaller space, and every intermediate result pipelines into the next -join — and defers cartesian products (joins with no connecting predicate) -to last. The DP, as code: +> **In:** the per-relation plan lists from Step 3 and the query's join +> predicates. +> **Out:** one plan per (relation set, interesting order), built for +> successively larger sets — and finally the plan for the full set. + +**Dynamic programming** is solving each distinct subproblem once and memoizing +the answer, justified by a principle of optimality. Selinger states that +principle in §5 in his own terms, and it is worth reading slowly: + +> "once the first k relations are joined, the method to join the composite to +> the k+1-st relation is independent of the order of joining the first k; i.e. +> the applicable predicates are the same, the set of interesting orderings is +> the same, the possible join methods are the same, etc. Using this property, an +> efficient way to organize the search is to find the best join order for +> successively larger subsets of tables." + +Note "**successively larger subsets**" — Selinger's DP is **bottom-up**, level +by level. (The sketch below matches the paper; a top-down memoized recursion +computes the same answer but is not what §5 describes, and the difference +becomes the whole story in Step 7.) + +Two restrictions make it affordable. + +**Left-deep only.** A **left-deep** plan is one where the right ("inner") input +of every join is a base relation, so the tree is a single spine; a **bushy** +plan allows a join whose both inputs are themselves joins. The paper does not +use either word — that vocabulary came later — but it states the restriction +plainly: "two relations are joined together, the resulting composite relation is +joined with the third relation, etc. At each step of the n-way join it is +possible to identify the outer relation (which in general is composite) and the +inner relation (**the relation being added to the join**)." The inner is always +a single relation. That is left-deep, and the paper gives the pipelining +motivation: intermediate composites "are physically stored only if a sort is +required for the next join step", otherwise materialized "one tuple at a time". + +**Cartesian products last.** The paper's heuristic, conditions (1) and (2) of +§5, is that a relation is only added if it has a join predicate with something +already joined — unless nothing does. "This means that all joins requiring +Cartesian products are performed as late in the join sequence as possible." ```rust -fn best_plan(rels: RelSet, memo: &mut HashMap) -> Plan { - if let Some(p) = memo.get(&rels) { return p.clone(); } - let mut best = Plan::infinite_cost(); - for r in rels.iter() { - let rest = rels.without(r); - if !has_join_predicate(rest, r) { continue; } // defer cartesians - let p = cheapest_join(best_plan(rest, memo), access_paths(r)); - if p.cost < best.cost { best = p; } // left-deep: (n−1) ⋈ 1 - // Selinger also keeps the cheapest plan per INTERESTING ORDER here — - // a pricier-but-sorted subplan can win at a later merge join - } - memo.insert(rels, best.clone()); - best -} +// ILLUSTRATION — not quoted from any repo in this course. This is Selinger +// §5's "successively larger subsets", written as code. The production +// version of exactly this is postgres standard_join_search at +// src/backend/optimizer/path/allpaths.c:3952, with the per-level pairing in +// src/backend/optimizer/path/joinrels.c:78. + 1 // memo maps (relation set, interesting order) -> cheapest plan + 2 for level in 2..=n { + 3 for set in subsets_of_size(rels, level) { + 4 for r in set.iter() { + 5 let rest = set.without(r); // left-deep: (k) join 1 + 6 if !has_join_predicate(rest, r) { continue; } // §5 heuristic + 7 for sub in memo.plans_for(rest) { // one per interesting order + 8 for method in [NestedLoop, MergeScan] { + 9 let p = join(sub, access_paths(r), method); + 10 memo.keep_if_cheapest(set, p.order, p); + 11 } // "cheapest unordered" AND "cheapest per order" both kept + 12 } + 13 } + 14 } + 15 } +``` + +**Work the counting, because this is the argument the whole step rests on.** +The paper opens §5's search discussion with "If a query block has n relations +in its FROM list, then there are n factorial permutations of relation join +orders." Evaluate that, against what the DP actually explores: + +``` + n! all left-deep orderings, enumerated one by one + n·2^(n-1) − n (set, last-relation) pairs the DP considers + 2^n − 1 memo entries — one per non-empty subset + (2n-2)!/(n-1)! all BUSHY trees, for comparison (Selinger excludes these) + + n n! bushy DP considered memo n!/DP + 5 120 1,680 75 31 1.6 + 10 3,628,800 17,643,225,600 5,110 1,023 710.1 + 15 1,307,674,368,000 3,497,296,636,753,920,000 245,745 32,767 5.3e6 ``` -Complexity: the famous "n joins considered in O(2ⁿ)-ish sets" — fine to -~12 relations, then every real system bolts on a fallback. Reading -exercise: follow §5's OPTIMAL plan tables by hand once; it's the same +Three readings, and they are the reason this step exists: + +1. **At n = 5 the DP barely pays for itself** — 75 vs 120. If you only ever join + five tables, enumerate and go home. +2. **At n = 10 it is 710× cheaper, at n = 15 five million times.** This is a + qualitative change, not a constant factor, and it is why the technique was + worth a paper. +3. **It is still exponential.** The memo column is `2^n − 1`, so the DP buys + about five more relations, not unlimited scaling. Every real system bolts on + a fallback above roughly 12 relations: postgres's genetic optimizer at + `geqo_threshold = 12`, DuckDB's greedy operator ordering at the same + threshold. Not a coincidence — it is where `2^n` stops fitting in a planning + budget. + +Note also the bushy column: by excluding bushy trees Selinger is giving up a +space 2.7 million× larger at n = 15. JOB §6.2 Table 2 later measured what that +costs — left-deep is at the *median* 1.00× optimal with PK indexes and 1.06× +with PK+FK, and 1.63×/4.50× at the maximum. Cheap, and the pipelining argument +above is why. + +Reading exercise: follow §5's OPTIMAL-plan tables by hand once. It is the same table your experiments' `reorder_joins` builds. +Why it matters: this is the single most-copied algorithm in database history, +and the counting above is why nobody has replaced it for small n. + ### Step 5 — what Selinger punted: nested queries -§6 handles correlated subqueries (a subquery referencing the outer row) -by simply re-evaluating the subquery *per outer row* — correct, and -O(outer × inner). This is the pre-decorrelation world: turning correlated -subqueries into joins (DuckDB's "deliminator" pass) took decades to get -right, and it's still the hardest rewrite family in any pipeline. Reading -§6 tells you exactly what problem that machinery exists to escape. +> **In:** a query block containing a correlated subquery — one that references +> a column of the enclosing block's row. +> **Out:** a plan that re-runs the subquery, in the general case once per outer +> tuple. Correct, and O(outer × inner). + +§6 handles nested queries by evaluating the inner block per outer row: "A +correlation subquery must in principle be re-evaluated for each [tuple of the +enclosing block]." The paper does add optimizations — an *uncorrelated* +subquery is evaluated once, and re-evaluation "can be made conditional… to +avoid re-evaluating subqueries unnecessarily" when the correlated values repeat +— but the baseline semantics is a nested loop, and the cost is multiplicative. + +This is the pre-decorrelation world. Turning correlated subqueries into joins — +DuckDB's "deliminator" pass (`OptimizerType::DELIMINATOR`, +`src/optimizer/optimizer.cpp:242`), DataFusion's +`datafusion/optimizer/src/decorrelate_predicate_subquery.rs:130` — took decades to get right and is still +the hardest rewrite family in any pipeline. + +Why it matters: read §6 as the "before" picture. It tells you exactly what +problem all that machinery exists to escape, and why an engine's subquery +support is a fair proxy for its optimizer's maturity. ### Step 6 — Cascades' memo: the search space as data -Sixteen years later, Graefe's move is to make optimization itself -programmable. The core structure is the **memo**: a set of **groups**, -each group being an equivalence class of plan fragments that all produce -the same result (and therefore share one cardinality estimate). Group -members can reference other groups as inputs — so one memo compactly -encodes exponentially many complete plans, duplication-free: +> **In:** the original query tree. +> **Out:** a **memo** — the search space itself, stored as data that rules can +> read and write, rather than a control flow that a search loop walks. + +Sixteen years later Graefe's move is to make optimization itself programmable. +Three terms, which must be kept distinct because Cascades papers use them +precisely: + +- An **expression** is one operator with its inputs — but the inputs are + *group* references, not sub-expressions. `Join(G2, G3)` is one expression. +- A **group** is an equivalence class: the set of all expressions that produce + the same logical result. Because they are logically equivalent, they share one + **cardinality** estimate — which is exactly the property that makes the memo + compact. +- The **memo** is the whole collection of groups. `optimize()` "first copies the + original query into the internal 'memo' structure" (§2) and everything after + that is rules adding expressions to groups. ``` - memo: G1 = {Join(G2,G3), Join(G3,G2), HashJoin(G2,G3), ...} - G2 = {Scan(A), IndexScan(A)} groups = equivalence classes, - G3 = {Scan(B)} members share cardinality + memo: G1 = { Join(G2,G3), Join(G3,G2), HashJoin(G2,G3), MergeJoin(G2,G3) } + G2 = { Scan(A), IndexScan(A) } groups = equivalence classes + G3 = { Scan(B) } members share a cardinality ``` -Selinger's memo keyed by relation-set is a special case; Cascades' -groups can hold *any* logically-equivalent expressions, not just join -orders. +Because expressions reference groups rather than trees, one memo encodes +exponentially many complete plans without duplication: G1 above stands for +2 × 1 × 4 = 8 complete plans in seven stored nodes. Selinger's memo keyed by +relation-set is the special case where the only equivalences considered are +join reorderings; Cascades' groups can hold *any* logically equivalent +expressions — including rewritten predicates, since the paper explicitly allows +"logically equivalent forms of all expressions, e.g., of a predicate". + +Why it matters: once the search space is data, "add a capability" means "add a +rule that writes into the memo", and that is the entire extensibility argument. ### Step 7 — everything is a rule; search is top-down and goal-driven -In Cascades, the optimizer's knowledge lives in **rules** of two kinds: -**transformation rules** (logical→logical: commute a join, associate) -and **implementation rules** (logical→physical: Join→HashJoin). Adding -an operator or algorithm = adding rules, not editing a search loop. -Search runs **top-down**: "optimize group G under requirement R (e.g. -sorted by x)" spawns tasks that fire rules into the memo; promise -heuristics order the firing; **branch-and-bound pruning** kills any -subtree already costlier than the best known complete plan (essential — -unlike Selinger's small space, the rule-generated space is unbounded). -Requirements are met by **enforcers** — sort (or, in distributed engines, -exchange/shuffle) inserted by the search itself to satisfy a required -property. Enforcers are Step 3's interesting orders, generalized: instead -of *keeping* sorted plans as extra state bottom-up, top-down search -*asks* for sortedness and inserts a sort when nothing provides it — and -that is how distributed engines later got shuffle planning for free. +> **In:** the memo, a rule set, and one root **optimization goal**. +> **Out:** the cheapest complete physical plan satisfying that goal — produced +> by six task types pushing each other onto a stack. + +Cascades' knowledge lives in **rules**, of two kinds: + +- a **transformation rule** rewrites logical → logical (commute a join, + associate two joins, push a predicate); +- an **implementation rule** rewrites logical → physical (`Join` → `HashJoin`). + +Adding an operator or an algorithm means adding rules, not editing a search +loop. Rules are objects (§1's contribution list: "Rules as objects"), and the +paper lists schema- and even query-specific rules as supported. + +**The goal is richer than "optimize this group".** §2: an optimization task +"combines a group or expression with a **cost limit** and with **required and +excluded physical properties**". Three components, and the excluded ones are +the part usually forgotten. + +Search is a set of six task types (§2, Figure 1): + +``` + Optimize Group find the best plan for any expression in a group + Optimize Expression optimize a single new expression + Explore Group derive logical expressions matching a pattern + Explore Expression the same, for one expression + Apply Rule fire one rule + Optimize Inputs recurse into inputs, accumulate cost +``` + +Tasks are **objects, not procedure calls** — "A task object exists for each +task that has yet to be done; all such task objects are collected in a task +structure", currently "a last-in-first-out stack". The paper is explicit that +this is an implementation choice, not a requirement: the structure could be "a +graph that captures dependencies… and permit efficient parallel search", and +the stack exists only "in order to obtain a working system fast". + +Three mechanisms are load-bearing: + +- **Memoization is in the Optimize-Group task.** "Before initiating + optimization of all a group's expressions, it checks whether the same + optimization goal has been pursued already" — so Cascades is *also* dynamic + programming, just keyed by (group, goal) instead of by relation set. +- **Branch-and-bound pruning via the cost limit.** In Optimize Inputs, "Each + time after an input has been optimized, the optimize inputs task obtains the + best execution cost derived, and derives a new cost limit for optimizing the + next input. Thus, pruning is as tight as possible." Unlike Selinger's bounded + space, the rule-generated space has no a-priori bound, so this is not an + optimization — it is what makes termination practical. +- **Enforcers.** Required properties are met by rules that insert an operator + to produce them: "Consider the inputs to a merge-join's inputs, which must be + sorted. An enforcer rule may insert a sort operation." And crucially, + "enforcers such as sorting are normal operators in all ways" — they are + costed and optimized like everything else. + +**Enforcers are Step 3's interesting orders, inverted.** Selinger, going +bottom-up, has to *guess in advance* which orders will be wanted and keep extra +plans for each. Cascades, going top-down, already knows what the parent wants — +it is in the goal — so it *asks* for sortedness and inserts a sort when nothing +in the group provides it. Same problem, opposite direction, and the top-down +version generalizes for free: replace "sorted by x" with "partitioned by x" and +you have shuffle planning in a distributed engine, which is how later systems +got it without new machinery. + +**The Cascades-over-Volcano delta, which is easy to miss.** Volcano's optimizer +generator ran two phases: exhaustively generate *all* logically equivalent +expressions, then optimize. Cascades explores **on demand and by pattern** — "A +group is explored using transformation rules only on demand, and it is explored +only to create all members of the group that match a given pattern." The paper's +own criticism of its predecessor: "The Volcano technique generates all +equivalent logical expressions exhaustively in the first phase. Even if [only a +few are needed]…" — with join associativity the exhaustive set is the whole +factorial space. Lazy, pattern-directed exploration is what makes the rule-based +architecture affordable at all. + +Why it matters: this is the architecture you would copy if M10's rule set is +going to keep growing, and the on-demand exploration is the part that makes it +tractable. ### Step 8 — the design space, in one table +> **In:** Steps 1-7. +> **Out:** the one decision M10 actually has to make, and the evidence for +> either answer. + | | Selinger (bottom-up) | Cascades (top-down) | |---|---|---| -| search | DP over relation sets | memoized task recursion | -| space | joins only; rewrites separate | rewrites + physical, one space | -| pruning | none needed (small space) | branch-and-bound essential | +| search | DP over relation sets, level by level | memoized task recursion over goals | +| memo key | relation set × interesting order | group × (cost limit, required/excluded props) | +| space | joins only; rewrites are a separate phase | rewrites + physical choice, one space | +| ordering | interesting orders kept as extra state | required properties, met by enforcers | +| pruning | none needed — space is bounded by `2^n` | branch-and-bound cost limits, essential | | extensibility | edit the enumerator | add a rule | +| exploration | implicit in the level loop | on demand, pattern-directed | | shipped in | postgres, DuckDB, SQLite | SQL Server, CockroachDB, Orca | -The pattern in the last row is not accidental: bottom-up DP is simple -and predictable — debuggable by whoever inherits it; Cascades pays -complexity for extensibility, which pays off where dedicated optimizer -teams write rules for a living (question 4 below). +The pattern in the last row is not accidental. Bottom-up DP is simple and +predictable — `standard_join_search` is one readable loop, debuggable by +whoever inherits it. Cascades pays real complexity (six task types, a rule +language, a pattern matcher, cost-limit plumbing) for extensibility, which pays +off where a dedicated optimizer team writes rules for a living. -## How to read the papers (with the concepts in hand) +And Step 4's counting says something about *when* the trade is even live: below +about 12 relations, Selinger's space is small enough that the search algorithm +is not your problem. JOB Table 3 (§6.3) makes the same point from the other +end — with true cardinalities, swapping exhaustive DP for a greedy heuristic +costs 1.20× at the median. The architecture choice matters far less than the +estimates it consumes. -**Selinger first** — read it all; it's short. +Why it matters: it means "which architecture" is a maintainability question +first and a plan-quality question second. -- **§2–3** — System R context; skim. -- **§4 — read carefully**: the selectivity-factor table (Step 2). Notice - how many of the constants you can name modern descendants of. -- **§5 — the core**: access paths + interesting orders (Step 3) feeding - the DP (Step 4). Work the OPTIMAL plans tables by hand — the single - best exercise in this topic. +## How to read the papers (with the concepts in hand) + +**Selinger first** — read it all; it's twelve pages. + +- **§1-2** — System R context and the RSS/RSI split; skim, but note the RSI + because Step 1's cost formula is denominated in it. +- **§3** — the cost formula (Step 1). One paragraph, and W is in it. +- **§4 — read carefully.** TABLE 1's selectivity factors (Step 2), then TABLE 2's + single-relation cost formulas and the interesting-order definition (Step 3). + For each constant, name its modern descendant. +- **§5 — the core.** The n! opening, the independence-of-prefix argument, the + Cartesian-deferral heuristic, and the OPTIMAL-plan tables (Step 4). Work the + tables by hand — the single best exercise in this topic. - **§6** — nested queries (Step 5); read as the "before" picture of decorrelation. -**Then Cascades** — a framework paper, denser and drier. - -- The memo and groups first (Step 6), then the task structure and rule - kinds (Step 7). Don't chase implementation details of the task - scheduler; the durable content is memo + rules + enforcers + +**Then Cascades** — a framework paper, denser and drier, and only ten pages of +which §2-4 matter. + +- **§1's bullet list** — the contribution list. Read it as a diff against + Volcano; several bullets are Step 7's mechanisms in one line each. +- **§2 — the algorithm.** Figure 1's six tasks, the goal as + (group, cost limit, required/excluded properties), the memoization check, and + the cost-limit derivation in Optimize Inputs. The explore-on-demand + discussion is here too and is the paper's real contribution. +- **§3-4 — the interface and rules.** Rules as objects, enforcer rules, + promise-ordered moves, group merging. Don't chase the task scheduler's + implementation details; the durable content is memo + rules + enforcers + branch-and-bound. -- Keep Step 8's table beside you and, for every mechanism, ask "what is - the Selinger equivalent, and why doesn't it scale to rules?" +- Keep Step 8's table beside you and, for every mechanism, ask "what is the + Selinger equivalent, and why doesn't it scale to rules?" ## Questions for notes.md -1. Selinger's W (CPU weight): what happens to plan choice as storage - moves NVMe→RAM (topic 6's numbers)? Which plans flip? -2. Interesting orders are DP state. What's the Cascades equivalent - (required physical properties), and why is top-down more natural for - propagating them? -3. Cascades promises "adding an operator = adding rules". Check it: - list the rules M10 needs to add for `Expand` (graph traversal as an - operator) — transformation (Expand commutes with Filter?) and - implementation (Expand → mxv? → per-node lookup?). -4. Why did the simple architecture (bottom-up DP) win in open source and - the complex one in commercial engines? (Consider: who writes the - rules, who debugs the search.) -5. M10 decision to record: Selinger-style enumerator or mini-Cascades - for the Cypher planner? (FalkorDB today: heuristic + label-cardinality - anchor selection — which architecture is that closer to?) +1. Selinger's W: redo Step 1's crossover arithmetic for your own machine's + numbers (topic 6 has the latency figures). At what selectivity does the + index stop winning, and which of your queries sit near that line? +2. Interesting orders are DP state; required physical properties plus enforcers + are the Cascades equivalent. Write out why top-down is more natural for + propagating them — what does the bottom-up version have to guess that the + top-down version is told? +3. Cascades promises "adding an operator = adding rules". Check it: list the + rules M10 needs for `Expand` (graph traversal as an operator) — the + transformation rules (does `Expand` commute with `Filter`? with another + `Expand`?) and the implementation rules (`Expand` → mxv? → per-node + adjacency lookup?). +4. Why did the simple architecture win in open source and the complex one in + commercial engines? Consider who writes the rules and who debugs the search + at 2 a.m. +5. M10 decision to record: Selinger-style enumerator or mini-Cascades for the + Cypher planner? FalkorDB today is heuristic plus label-cardinality anchor + selection — which architecture is that closer to, and what would it cost to + move? + +## Takeaway + +Selinger's contribution was not the cost formula (one line) or the selectivity +table (admittedly guessed constants, two of which postgres still ships). It was +the observation that the best plan for a *set* of relations does not depend on +how that set was built — which converts `15! = 1.3 × 10¹²` orderings into +245,745 considerations, a 5.3-million-fold saving, at the price of an +exponential memo that runs out at about 12 relations. Cascades' contribution +was to notice that the same memoization works when the memo's keys are +equivalence classes and its contents are produced by rules rather than by a +loop — which makes the optimizer extensible, requires branch-and-bound pruning +to stay finite, and turns Selinger's interesting orders into required +properties that the search asks for rather than guesses at. ## Done when -You can run Selinger's DP on a 3-table join by hand, and describe a memo -group's contents for the same query in Cascades terms. +Answer each before unfolding it. + +- [ ] Run Selinger's DP on a three-table join by hand. What is in the memo + after each level, and how many entries are there in total? +
Answer + + Level 1 holds three entries, `{A}`, `{B}`, `{C}`, and each entry holds the + cheapest unordered access path *plus* one path per interesting order + (§4: ORDER BY ∪ GROUP BY columns, plus §5's "every join column defines an + interesting order"). Level 2 holds `{AB}`, `{AC}`, `{BC}` — each built by + taking a level-1 entry as outer and adding a single relation as inner + (left-deep), trying nested loops and merging scans, and skipping any pair with + no join predicate unless nothing qualifies. Level 3 holds `{ABC}`, built from + each level-2 entry plus the remaining relation. Total memo entries: + `2^3 − 1 = 7`, against `3! = 6` orderings — at n = 3 the DP is not yet + winning, which is exactly Step 4's point. + +
+ +- [ ] Define an interesting order the way the paper does, not the way summaries + do. +
Answer + + Selinger defines it twice, and both times as an explicit finite set, not as + "any order a later operator might exploit". Single relations (§4): "a tuple + order is an interesting order if that order is one specified by the query + block's GROUP BY or ORDER BY clauses". Joins (§5): "As in the single relation + case… Also **every join column** defines an 'interesting' order." So the set + is `ORDER BY columns ∪ GROUP BY columns ∪ every join column` — computable + from the query text before search starts, and usually small. That finiteness + is what makes keeping one plan per order affordable. The paper also states the + negative: with no GROUP BY or ORDER BY there are no interesting orders from + the query block at all, and the cheapest path simply wins. + +
+ +- [ ] For n = 5, 10 and 15, give n!, the number of considerations the DP makes, + and the memo size. Where does exhaustive search die, and where does the DP + die? +
Answer + + ``` + n n! DP considered memo n!/DP + 5 120 75 31 1.6 + 10 3,628,800 5,110 1,023 710.1 + 15 1,307,674,368,000 245,745 32,767 5.3e6 + ``` + + DP considered is `n·2^(n-1) − n` (one per (set, last-relation) pair); memo is + `2^n − 1`. Exhaustive enumeration dies between n = 10 and n = 15 — a trillion + orderings is not a planning budget. The DP dies later but for the same reason: + its memo is still `2^n`, so it buys roughly five more relations, which is why + postgres switches to a genetic algorithm at `geqo_threshold = 12` and DuckDB + switches to greedy at the same count. For context, Selinger also excludes all + bushy trees, a space of `(2n-2)!/(n-1)!` = 3.5 × 10²¹ at n = 15; JOB §6.2 + Table 2 measured that exclusion as costing 1.00-1.06× at the median. + +
+ +- [ ] Which of Selinger's 1979 constants is still in postgres unchanged, and + which one moved? +
Answer + + **Unchanged to the digit:** the open-ended-comparison fallback, `F = 1/3`, + survives as `DEFAULT_INEQ_SEL 0.3333333333333333` + (`src/include/utils/selfuncs.h:37` at `701f021`). Selinger's own note on it — + "There is no significance to this number, other than… it is less than 1/2. We + hypothesize that few queries use predicates that are satisfied by more than + half the tuples" — is still the only justification anyone has. **Moved:** + equality with no index was `F = 1/10`; postgres uses `DEFAULT_EQ_SEL 0.005` + (`:34`), 20× tighter. The reason is in postgres's own comment at `:24-30`, and + it is policy rather than measurement: the defaults must be "small enough to + ensure that indexscans will be used if available", and 0.01 was tried and + found too large. + +
+ +- [ ] In Cascades, distinguish a group, an expression and the memo — and say + why the distinction buys compactness. +
Answer + + An **expression** is one operator whose inputs are *group references*, e.g. + `Join(G2, G3)` — not a subtree. A **group** is an equivalence class holding + every expression that produces the same logical result; because they are + logically equivalent they share one cardinality estimate. The **memo** is the + set of all groups, seeded by copying the original query into it (§2). The + compactness follows from the indirection: a group holding 4 expressions whose + inputs are groups of 2 and 1 stands for 8 complete plans in 7 stored nodes, + and the factor is multiplicative down the tree. Selinger's relation-set memo + is the special case where the only equivalence is join reordering; a Cascades + group can hold any logically equivalent expressions, "e.g., of a predicate". + +
+ +- [ ] Cascades needs branch-and-bound pruning and Selinger does not. Why? +
Answer + + Because their search spaces are bounded differently. Selinger's space is + fixed before search begins — left-deep trees over subsets, `2^n − 1` memo + entries — so exhaustive is affordable by construction. Cascades' space is + generated by rules, and a rule set containing join associativity and + commutativity generates without a-priori bound, so something must cut it off. + The mechanism is the **cost limit** carried in every optimization goal + alongside the required and excluded physical properties (§2): in the Optimize + Inputs task, "Each time after an input has been optimized, the optimize + inputs task obtains the best execution cost derived, and derives a new cost + limit for optimizing the next input. Thus, pruning is as tight as possible." + The second half of the answer is exploration: Cascades explores groups "only + on demand… only to create all members of the group that match a given + pattern", which is precisely what Volcano did *not* do, and what made the + exhaustive first phase untenable. + +
+ +- [ ] Enforcers and interesting orders solve the same problem. State the + difference in one sentence, then say what the top-down version gets for + free. +
Answer + + Selinger, searching bottom-up, must **guess in advance** which orders a + not-yet-chosen parent will want and pay to keep an extra plan for each; + Cascades, searching top-down, is **told** what the parent requires as part of + the optimization goal and inserts an enforcer — "An enforcer rule may insert a + sort operation" — only when no member of the group already provides it. The + free lunch is generalization: the goal carries arbitrary *required physical + properties*, so replacing "sorted by x" with "partitioned by x" gives you + exchange/shuffle planning in a distributed engine with no new machinery, and + because "enforcers such as sorting are normal operators in all ways" they get + costed and optimized like anything else. + +
## References **Papers** -- Selinger, Astrahan, Chamberlin, Lorie, Price — "Access Path Selection - in a Relational Database Management System" (SIGMOD 1979) — read it - all; it's short, and §4's selectivity factors + §5's DP are the core -- Graefe — "The Cascades Framework for Query Optimization" (IEEE Data - Engineering Bulletin 1995) — the memo, rules, and top-down task model +- Selinger, Astrahan, Chamberlin, Lorie, Price — "Access Path Selection in a + Relational Database Management System", SIGMOD 1979, pp. 23-34. Read it all; + it's short. §3's cost formula, §4's TABLE 1 selectivity factors and the + interesting-order definition, and §5's DP are the core. ~1 h. +- Graefe — "The Cascades Framework for Query Optimization", IEEE Data + Engineering Bulletin 18(3), 1995, pp. 19-29. The memo, the six tasks, rules + as objects, enforcers, and explore-on-demand. ~1 h. +- Graefe, McKenna — "The Volcano Optimizer Generator: Extensibility and + Efficient Search", ICDE 1993. Read second if Cascades' repeated criticisms of + "the Volcano technique" are opaque; it is the two-phase design Cascades + replaces. +- Leis et al. — "How Good Are Query Optimizers, Really?", PVLDB 2015. Measures + what Step 2's assumptions cost on real data, and Table 2/Table 3 quantify + Step 4's and Step 8's restrictions. See `reading-how-good-optimizers.md`. + +**Code — the descendants** +- `postgres/postgres@701f021` — `src/backend/optimizer/path/allpaths.c:3952` + (`standard_join_search`) is Step 4, `src/include/utils/selfuncs.h:34-40` is + Step 2, still. See `reading-postgres-optimizer.md`. +- `duckdb/duckdb@6c0c1a68` — `src/optimizer/join_order/` is Step 4 with a + different enumeration order and no interesting orders. See + `reading-duckdb-optimizer.md`. +- `apache/datafusion@1e77af8` — `datafusion/optimizer/src/optimizer.rs` is a + distant, memo-less descendant of Step 7's rule architecture. See + `reading-rust-planner-stack.md`. diff --git a/topics/11-execution-models/notes.md b/topics/11-execution-models/notes.md index 82a2a63..5880e3d 100644 --- a/topics/11-execution-models/notes.md +++ b/topics/11-execution-models/notes.md @@ -24,16 +24,16 @@ this box) does batching close, and what does the remainder consist of? ## Predictions (fill BEFORE implementing vectorized.rs / kernels.rs) -Measured baseline (provided volcano, release, 50M rows, sel 50%): -**0.277 s = 180.7 M rows/s** — already fast! ~5.5 ns/row including two -virtual calls per tuple. Modern branch predictors eat stable indirect -calls; the Volcano tax on an M-series core is NOT mostly call overhead. +Measured baseline (provided volcano, release, 50M rows, sel 50%), from the +Baseline block above: **0.484 s = 103.3 M rows/s** — already fast, ~9.7 ns/row +including two virtual calls per tuple. Modern branch predictors eat stable +indirect calls; the Volcano tax on an M-series core is NOT mostly call overhead. Where will the vectorized win actually come from? (SIMD, ILP, no per-row branch.) Predict accordingly: | engine | predicted M rows/s | predicted ratio vs volcano | actual | actual ratio | |---|---|---|---|---| -| volcano (sel 50) | — | 1× | 180.7 | 1× | +| volcano (sel 50) | — | 1× | 103.3 | 1× | | vectorized (sel 50) | | | | | | kernel (sel 50) | | | | | diff --git a/topics/11-execution-models/reading-compiled-vs-vectorized.md b/topics/11-execution-models/reading-compiled-vs-vectorized.md index 555774c..df3d6e9 100644 --- a/topics/11-execution-models/reading-compiled-vs-vectorized.md +++ b/topics/11-execution-models/reading-compiled-vs-vectorized.md @@ -9,52 +9,142 @@ decide every round — registers versus intermediates, and cache-miss overlap — step by step; the residual differences, not the headline winner, are what decide M11 and M19. +There is no headline winner. The paper's own summary opens "To our +surprise, the performance of vectorized and data-centric compiled query +execution is quite similar in OLAP workloads" (§10), and every number +below is checked against the section, table or figure it came from. Two +figures this guide used to carry were not in the paper at all; both +corrections are called out where they occur. + ## The problem in one sentence By 2018 both modern execution models claimed to have killed -interpretation overhead — the question is what separates them once the -100× interpretation tax is gone, and the answer turns out to be -second-order hardware effects worth "only" 2–3× per operator, plus -everything operational. +interpretation overhead — the question is what separates them once that +tax is gone, and the answer is second-order hardware effects that leave +the two within a factor of 1.74 in the *worst* case (Typer 74% faster on +Q1, Tectorwise 32% faster on Q9, §4.1), against a HyPer-vs-Postgres gap +of one to two orders of magnitude — plus a set of operational +differences (§8) that are not about rows per second at all. ## The concepts, step by step ### Step 1 — the common enemy: interpretation overhead -Both models exist to kill the same cost. A classic (Volcano-style) -engine processes one row at a time through a tree of operators, paying -per row: an indirect function call per operator, plus walking the -expression tree (`f < 50` evaluated by recursing over plan-time -objects). That overhead is ~20–100 ns per row while the useful work (a -compare, an add) is ~1 ns — the engine spends >90% of its time deciding -what to do rather than doing it. Both contenders eliminate this, by -opposite means: amortize it over a batch, or compile it away entirely. +> **In:** two engine designs that look nothing alike, and a reason to +> care which one you build. +> **Out:** the single cost both were invented to remove, sized on this +> repo's own measurement — so that Step 4's near-tie reads as "the war +> is over", not "the difference is small because nothing matters". + +A classic **Volcano** (**iterator**) engine composes operators as a tree +in which each exposes `next()` returning **one tuple**, and evaluates +`f < 50` by walking plan-time expression objects. Per row it pays an +indirect call per operator plus that walk, while the useful work is a +single compare. + +This repo has the tax on a machine you can hold: + +``` + FINDINGS.md row 11 / notes.md — exec_bench volcano lane, + Apple M3 Pro, 50M rows, scan -> filter -> group-by-sum: + + 5% selectivity 0.386 s 129.4 M rows/s 7.72 ns per scanned row + 50% 0.484 s 103.3 M rows/s 9.68 ns + 95% 0.669 s 74.7 M rows/s 13.38 ns + + at ~4 GHz: 7.72 ns = ~31 cycles to move one row through + two dyn calls and one accumulate +``` + +Two things to carry forward. First the size: ~31 cycles for work that is +a compare and an add. X100's profile of MySQL puts a single addition at +49 cycles ([reading-x100.md](reading-x100.md)); the paper here notes the +resulting whole-system gap between HyPer and Postgres is "between one and +two orders of magnitude" (§4.1). That is the war both contenders won. + +Second, the direction: the lane gets **slower as selectivity rises**. +Surviving the filter is what costs — a rejected row costs one `next()` +call inside the filter's own loop, a survivor pays a second `next()` up +the chain plus the aggregate's hash and accumulate. "High selectivity = +less work" is exactly backwards for a tuple-at-a-time engine, and the +95% column is the number to keep in mind when Step 5 explains why the +*probe* is where the two models actually separate. ### Step 2 — vectorized execution: interpret once per thousand rows -The vectorized model (X100 lineage — see reading-x100.md) keeps an -interpreter, but each operator call processes a **vector** (a batch of -~1000–2048 values of one column, stored as a plain array) instead of one -row. The query becomes a sequence of **primitives** — precompiled, -branch-free loops like `filter_lt(f_vec, 50)` — each doing one simple -operation over the whole vector. Interpretation still happens, but once -per vector: the ~100 ns dispatch cost divides by 2048 rows ≈ 0.05 ns/row, -while the loops themselves are simple enough for the compiler to -auto-vectorize (emit SIMD — single instructions operating on multiple -values at once). The price: each primitive writes its result to an -intermediate array for the next primitive to read — memory traffic that -Step 3's model avoids. +> **In:** the per-tuple tax from Step 1. +> **Out:** the first cure — keep the interpreter, enlarge its unit — +> and the two hard constraints it imposes on every line of engine code, +> which are what generate all of Tectorwise's losses later. + +The vectorized model (X100 lineage — see +[reading-x100.md](reading-x100.md)) keeps Volcano's tree, but each +operator call processes a **vector**: an array of ~1,000-2,048 values of +one column. §2 states the goal as "to amortize the DBMS's interpretation +decisions by performing as much as possible inside the data manipulation +methods… hash 1000s of values, compare 1000s of string pairs, update a +1000 aggregates". The work is done by **primitives** — precompiled, +type-specialized, branch-light loops. + +The amortization is not a hope, it is measured: + +``` + §4.2, profiler over the query set at SF=10: + interpreted part of runtime < 1.5% + time inside primitives 98.5% +``` + +**Correction:** this guide previously estimated the residue as "~100 ns +of dispatch ÷ 2048 rows ≈ 0.05 ns/row". The estimate was invented; the +paper measured the thing directly, and 1.5% is the number to quote. + +§4.2 then goes further, and this is the sentence the rest of the guide +hangs on. Tectorwise executes more instructions per tuple than Typer — +but since 98.5% of time is inside primitives, and "primitives know all +involved types at compile time", the extra instructions **are not +interpretation**. They "are rather due to the load/store instructions for +materializing primitive results into vectors". Vectorization did not +leave a little interpretation behind; it traded interpretation for +memory traffic. + +The trade is forced by two constraints §2.1 spells out. A vectorized +function (i) can only work on **one data type** — "the number of +combinations grows exponentially" otherwise — and (ii) must process +multiple tuples. Figure 1 shows what that costs on `color = 'green' AND +tires = 4`: + +``` + Figure 1a, generated code — one loop, both predicates in one if: + for i in 0..n: if col[i]=="green" && tir[i]==4: res.append(i) + + Figure 1b, vectorized — constraint (i) forbids the mixed-type if, + so it must become two primitives with a selection vector between: + s = sel_eq_string(col, "green") // writes positions + res = sel_eq_int(tir, 4, s) // reads positions + + cost: one intermediate array written and read that the fused + version kept in a register — "The resulting materialization + of intermediates makes fast caches very important for + vectorized engines" (§2.1) +``` ### Step 3 — compiled execution: fuse the pipeline into one loop -The compiled model (HyPer lineage) deletes the interpreter: at query -time, generate machine code (**JIT** — just-in-time compilation) that -fuses each **pipeline** (a chain of operators between materialization -points, e.g. scan→filter→aggregate) into ONE loop, in which the row's -values live in CPU **registers** (the ~16 named storage slots inside the -core — zero-latency, but scarce) from scan to sink. No calls, no -intermediates, no dispatch. Here are both models on one query, -`SELECT k, SUM(v) FROM t WHERE f < 50 GROUP BY k`: +> **In:** the same per-tuple tax, attacked from the other side. +> **Out:** the second cure — delete the interpreter — and the resource +> it spends instead of memory: registers, of which there are sixteen. + +The compiled model (HyPer lineage) generates machine code at query time +that "fuses all adjacent non-blocking operators of a query pipeline into +a single, tight loop" (§2). A **pipeline** is a chain of operators +between materialization points (scan → filter → aggregate); a +**pipeline breaker** is an operator that must consume its whole input +before producing output (a hash-join build, a sort), and it ends the +pipeline. Within the loop a row's values live in CPU **registers** — of +which x86-64 has 16 general-purpose — from scan to sink. No calls, no +intermediate arrays, no dispatch. + +Both models on `SELECT k, SUM(v) FROM t WHERE f < 50 GROUP BY k`: ``` Typer (compiled) Tectorwise (vectorized) @@ -67,115 +157,330 @@ intermediates, no dispatch. Here are both models on one query, across all operators simple, branch-free, SIMD-able ``` -The price of compilation: generating and compiling that loop takes -100s of milliseconds (LLVM), paid before the first row moves. +**JIT** is just-in-time compilation: emitting machine code (HyPer emits +LLVM IR) after the query arrives. The bill arrives before the first row +moves, and Step 7 has what the paper does and does not say about its +size. ### Step 4 — the fair fight: build both, share everything else -Prior comparisons raced whole systems (HyPer vs Vectorwise), where -storage formats, hash tables, and compilers all differ — attribution -impossible. This paper's method: implement Typer and Tectorwise with the -**same algorithms and same data structures**, differing ONLY in loop -structure (Step 2's four loops vs Step 3's one), then run TPC-H. That's -what makes the comparison fair — the topic 0 discipline of changing one -variable. Headline result: **nearly tied** — TPC-H geometric mean within -~10–20%. The 100× war of X100-vs-MySQL is over; both models kill -interpretation (Step 1). Everything interesting is in where they -*differ* — Steps 5–7. +> **In:** two designs, and a literature of comparisons between whole +> systems where storage format, hash table, parallelization framework +> and compiler all differ at once. +> **Out:** one variable changed, five queries measured, and a result +> that is a range rather than a winner. + +Prior comparisons raced HyPer against VectorWise, where attribution is +impossible. §3's method: implement Typer and Tectorwise in **one test +system**, with "the same algorithms and data structures" and "the same +physical query plans", so that "the only difference between Tectorwise +and Typer is the query execution method". Both were even given the same +parallelization framework — morsel-driven (§6.1) — specifically to +remove it as a variable. That is topic 0's discipline: change one thing. + +Two caveats the paper is explicit about, both of which shape how you may +quote it. §3: "We do not include query parsing, optimization, code +generation, and compilation time in our measurements" — so every runtime +below is *execution only*, and compilation is free by construction. +§3.3: the workload is five representative TPC-H queries, not the suite — +Q1 (fixed-point arithmetic, 4-group aggregation), Q6 (selective filters), +Q3 (join, 147 K build / 3.2 M probe), Q9 (join, 320 K build / 1.5 M +probe), Q18 (high-cardinality aggregation, 1.5 M groups). + +The result (§4.1, Figure 3, SF=1, 1 thread): + +``` + relative single-thread performance, per query: + Q1 Typer faster by 74% (arithmetic, in-cache aggregation) + Q18 Typer faster (Table 1: 30 vs 48 cycles/tuple = 60%) + Q6 tie (Table 1: 11 vs 11 cycles/tuple) + Q3 Tectorwise faster by 4% (join) + Q9 Tectorwise faster by 32% (join) + + the paper's framing of that spread: + "these are not large differences… the difference between HyPer and + PostgreSQL is between one and two orders of magnitude" + "neither paradigm is clearly dominated by the other" +``` + +**Correction:** this guide previously reported "TPC-H geometric mean +within ~10-20%". No such figure appears in the paper — there is no +geometric mean over TPC-H in it, and the honest summary is the range +above: 1.74× the worst way for Tectorwise, 1.32× the worst way for +Typer, direction depending on the query. Reporting a mean would also +have destroyed the finding, since the two halves of the spread point +opposite ways for opposite reasons. Those reasons are Steps 5 and 6. ### Step 5 — memory-level parallelism: why vectorized wins hash probes -**Memory-level parallelism** (MLP — a modern core's ability to have ~10 -cache misses in flight simultaneously, making 10 overlapped misses cost -about as much as one) is the deciding hardware effect for memory-bound -operators. A vectorized probe hashes 2048 keys in loop 2, then issues -2048 independent hash-table lookups in loop 3 — the out-of-order core -overlaps many misses at once. The compiled fused loop handles one row -end-to-end: its single probe miss must resolve before the row finishes, -so it has ONE miss in flight — unless you contort the loop with software -prefetching (manually issuing "fetch this address" hints ahead of use; -they cite group prefetching / AMAC). **Hash join probe is the great -equalizer**: both models end up memory-bound on the HT's random -accesses, with Tectorwise slightly ahead because batching misses is its -natural shape. This is the same MLP lesson as topic 0's -lookup_shootout. +> **In:** Q3 and Q9, where "both engines use exactly the same hash table +> layout and therefore also have an almost identical number of last +> level cache misses" (§4.1) — so the difference cannot be the algorithm. +> **Out:** a hardware effect that turns identical miss *counts* into +> different miss *costs*, and the counter that proves it. + +**Memory-level parallelism** (MLP) is a core's ability to have several +cache misses outstanding at once, so that overlapped misses cost far +less than their sum. §4.1 explains the mechanism in both directions: + +- Tectorwise's "hash table probing code is only a simple loop. It + executes only hash table probes thus the CPU's out-of-order engine can + speculate far ahead and generate many outstanding loads." +- Typer's "code has more complex loops. Each loop can contain code for a + scan, selection, hash-table probe, aggregation and more. The + out-of-order window of each CPU fills up more quickly with complex + loops thus they generate less outstanding loads." + +**Correction:** this guide previously said the fused loop has "ONE miss +in flight". The paper's claim is *fewer*, not one — the out-of-order +window fills faster because each iteration carries more instructions. +The distinction matters: the cure is not "batch or lose", it is anything +that keeps the reorder window free, which is why software prefetching +(group prefetching / AMAC) works for compiled probes at the cost of +contorting the loop. + +The counter that settles it is memory stall cycles, and the SSB table in +§4.4 (1 thread, SF=30, per tuple) shows it as arithmetic: + +``` + cycles IPC instr L1miss branchmiss mem stall + Q3.1 Typer 55 0.7 40 1.1 0.24 27.95 = 51% stalled + Q3.1 TW 53 1.3 71 1.7 0.41 15.68 = 30% stalled + Q4.1 Typer 78 0.5 39 1.8 0.38 45.91 = 59% stalled + Q4.1 TW 59 1.0 61 2.5 0.63 19.48 = 33% stalled + + read Q4.1 as the whole thesis in one row: + Tectorwise runs 61/39 = 1.56x the instructions + and takes 2.5/1.8 = 1.39x the L1 misses + and still finishes in 59/78 = 0.76x the cycles, + because it waits 45.91 - 19.48 = 26.4 fewer cycles per tuple +``` + +§4.1 adds that the advantage grows with the hash table: "Tectorwise's +join advantage increases up to 40% for larger data (and hash table) +sizes". Same lesson as topic 0's `lookup_shootout` — the miss count is +not the cost; the miss *schedule* is. ### Step 6 — registers vs intermediates: why compiled wins expressions -The opposite case: compute-heavy work. In the fused loop a value loaded -once stays in registers through every operator that touches it; in the -vectorized engine every primitive boundary is a store of the whole -result vector + a load by the next primitive. For an expression-heavy -query (or a "wide" pipeline carrying 10 columns through 3 operators), -that's dozens of extra loads/stores per row — Tectorwise's registers -went to array bookkeeping (question 3 below). So **compilation wins**: -expression-heavy work, wide pipelines, and OLTP-style point work (no -per-vector setup cost amortizable over 3 rows). A related sobering -finding: explicit SIMD gained less than hoped — most operators are -memory-bound (Step 5's regime), and SIMD only helps compute-bound -primitives. +> **In:** Q1, the opposite regime — fixed-point arithmetic over a +> four-group aggregation that never leaves cache. +> **Out:** the cost of Step 2's constraint (i) in instructions per +> tuple, plus a warning about the metric you would naturally reach for +> to measure it. -### Step 7 — the operational column: everything that isn't rows/second +When there are no misses to hide, MLP buys nothing and Step 2's +materialization is pure cost. Table 1 (TPC-H SF=1, 1 thread, per tuple): -The differences that decide real deployments are not in the inner loop: +``` + cycles IPC instr L1miss LLCmiss branchmiss + Q1 Typer 34 2.0 68 0.6 0.57 0.01 + Q1 TW 59 2.8 162 2.0 0.57 0.03 + Q6 Typer 11 1.8 20 0.3 0.35 0.06 + Q6 TW 11 1.4 15 0.2 0.29 0.01 + Q3 Typer 25 0.8 21 0.5 0.16 0.27 + Q3 TW 24 1.8 42 0.9 0.16 0.08 + Q9 Typer 74 0.6 42 1.7 0.46 0.34 + Q9 TW 56 1.3 76 2.1 0.47 0.39 + Q18 Typer 30 1.6 46 0.8 0.19 0.16 + Q18 TW 48 2.1 102 1.9 0.18 0.37 + + Q1, the extremes the paper quotes as "up to 2.4x" and "up to 3.3x": + instructions 162 / 68 = 2.4x + L1 misses 2.0 / 0.6 = 3.3x + LLC misses 0.57 = 0.57 — identical, so this is not about DRAM + result 59 / 34 = 1.74x slower +``` -- **compile latency** — Tectorwise starts in ~0 ms; Typer pays 100s of - ms of LLVM per query (deadly for short queries and for interactive - use). -- **profiling** — perf on Tectorwise shows time per named primitive; - compiled code is one opaque JIT blob. -- **adaptivity** — a vectorized engine can swap a primitive mid-query - (e.g. switch filter implementation when selectivity shifts); compiled - code must recompile. -- **engineering** — no LLVM dependency vs hundreds of hand-written - kernels. DuckDB chose vectors partly on exactly these grounds. +The LLC row is the tell: both engines miss last-level cache equally +often on Q1, so Tectorwise's 94 extra instructions and 1.4 extra L1 +misses per tuple are entirely the write-and-reread of intermediates +between primitives. §4.1: "In Tectorwise intermediate results must be +materialized, which is similarly expensive as the computation itself." + +Now the warning, because it is the most reusable thing in the paper. +Tectorwise's **IPC on Q1 is 40% higher** — 2.8 against 2.0 — while being +74% slower. §4.1: "having a higher IPC is not always better… one has to +be cautious when using IPC to compare database systems' performance. It +is a valid measure of the amount of free processing resources, but should +not be used as the sole proxy for overall query processing performance." +A model that executes 2.4× the instructions can retire them beautifully +and still lose. + +Two more results belong here, both of which kill plausible stories. + +**Instruction cache is not the differentiator.** Generated code is +bigger, so you would expect Typer to thrash L1i; recent work found i-cache +misses to be a real problem for OLTP [43]. §4.2 measured it and found +"instruction cache misses are negligible, thus not a performance +bottleneck for OLAP queries. For all queries measured, the L1 instruction +cache (32 KB) was large enough to contain all hot code." One 32 KB +number retires the whole hypothesis for this workload — and note it is +workload-specific, not a law. + +**Branch misses do not line up with the winner either.** Read the +branch-miss column above: Tectorwise wins it on Q3 (0.08 vs 0.27) and +loses it on Q9 (0.39 vs 0.34) and Q18 (0.37 vs 0.16) — yet Tectorwise +wins Q3 *and* Q9 and loses Q18. The correlation that does hold across +every row is the mem-stall column of §4.4. Prefer the counter that +tracks the outcome. + +**SIMD does not rescue the vectorized side either** (§5). Primitives are +tight typed loops, so they are the natural home for **SIMD** (one +instruction over many values), and the micro-benchmarks deliver: up to +8.4× in isolation, 2.3× for hashing. But gather instructions give only +1.1× "because the memory system of the test machine can perform at most +two load operations per cycle — regardless of whether SIMD gather or +scalar loads are used", the full probe primitive gains 1.4×, and +end-to-end the gains "almost vanish", landing "around 10% for join +queries" — even though 55-65% of runtime is inside SIMD-optimized +primitives. Figure 9's sweep says why: SIMD helps while the working set +is in cache and stops helping once it is not. §5.4's conclusion is that +"SIMD does not shift the balance in favor of vectorization much". + +Auto-vectorization is worse news, and worth knowing before you reach for +it: of GCC 7.2, Clang 5.0 and ICC 18, only ICC vectorized a fair share of +primitives, and only with AVX-512; it cut instructions per tuple by +20-60% and produced **no significant runtime improvement**, sometimes +running slower (§5.3). -The scorecard: +### Step 7 — the operational column: everything that isn't rows/second -| dimension | compiled (Typer) | vectorized (Tectorwise) | -|---|---|---| -| computation-heavy | **wins** (registers) | loses (intermediates) | -| memory-bound (probes) | loses (1 miss in flight) | **wins** (miss overlap) | -| compile latency | 100s of ms (LLVM) | **zero** | -| profiling/debugging | opaque blob | **per-primitive** | -| adaptivity | recompile | **swap primitives** | -| implementation effort | LLVM dependency, codegen bugs | 100s of kernels | +> **In:** a performance comparison that came out a tie, which is +> precisely what makes §8 the deciding section. +> **Out:** five dimensions on which the models are *not* tied, and the +> one place this guide previously invented a number. + +§8's opening states the situation: "The performance differences are not +large enough to make a general recommendation whether to use +vectorization or compilation. Therefore, as a practical matter, other +factors… may be of greater importance." + +- **OLTP (§8.1)** — compilation wins outright. A vectorized engine needs + many vectors of values to be efficient, and "for OLTP workloads, + vectorization has little benefit over traditional Volcano-style + iteration", while compilation can fuse an entire stored procedure into + one machine-code fragment. The evidence offered is organizational: + Microsoft SQL Server already had a vectorized engine (Apollo) and the + team "felt compelled to additionally integrate the compilation-based + engine Hekaton". +- **Compile time (§8.2)** — vectorization wins, because primitives are + precompiled. **Correction:** this guide previously priced Typer's + compilation at "100s of ms of LLVM per query". That figure is not in + the paper, which excludes compilation time from every measurement + (§3). What §8.2 does say is the *shape*: LLVM compile time is "often + super-linear to code size", and code size grows with operator count — + or with column count, since "a small SQL query such as `SELECT * FROM + T` can produce a lot of code if table T has thousands of columns". The + mitigations are the interesting part: HyPer switches off LLVM passes + including register allocation for its own more scalable algorithm, and + ships an LLVM IR interpreter that runs the first morsels — "if that is + enough to answer the query, full LLVM compilation is omitted". Spark + falls back to tuple-at-a-time interpretation above 8 KB of generated + Java bytecode. +- **Profiling (§8.3)** — vectorization wins. Per-primitive cycle counts + "adds only marginal overhead, as each call to the function works on a + thousand values". For compiled code "it is currently not possible in + Spark SQL to know the individual contributions to execution time of + relational operators, since the system can only measure performance on + a per-pipeline basis". +- **Adaptivity (§8.4)** — vectorization wins, "the idea of adaptive + execution works best in systems that interpret a query". The worked + example is why VectorWise beat Tectorwise on Q1 (Table 2): during + aggregation it tries to partition a vector's tuples into one selection + vector per group-by key, backing off exponentially if there are too + many groups; when it succeeds, hash aggregation becomes ordered + aggregation with the running sum in a register, so "the aggregates are + just updated once per vector". +- **Implementation (§8.5)** — a wash, differently shaped. Compiled + systems are "code that generates code, thus … harder to comprehend and + debug"; vectorized systems must keep control logic out of primitives + and live with constraint (i). The paper's own example of that + constraint biting: composite sort keys, where a multi-column + comparison must be decomposed into several primitives communicating + through a boolean vector — extra materialization that a compiled sort + specialized to the record format avoids entirely. + +The scorecard, restated to match §8.6's table and the sections above: + +| dimension | compiled (Typer) | vectorized (Tectorwise) | evidence | +|---|---|---|---| +| computation-heavy, in cache | **wins** — registers, 2.4× fewer instr | loses — materialization | Table 1 Q1: 34 vs 59 cycles/tuple | +| memory-bound probes | loses — window fills, fewer loads in flight | **wins** — miss overlap | Q9 74 vs 56; §4.4 mem stalls | +| SIMD headroom | — | small: ~10% on joins, 8.4× only in isolation | §5.2, §5.4 | +| compile latency | super-linear in code size; mitigations required | **zero** — primitives precompiled | §8.2 (not measured here) | +| OLTP / stored procedures | **wins** | little benefit over Volcano | §8.1 | +| profiling | per-pipeline only | **per-primitive**, marginal overhead | §8.3 | +| adaptivity | recompile | **swap primitives mid-flight** | §8.4 | +| implementation | codegen indirection | constraints on every primitive | §8.5 | + +One last result that shrinks the performance column further: with +morsel-driven parallelism on 20 hyper-threads (Table 3, SF=100), "for all +but one query, the performance gap between the two systems becomes +smaller… For the join queries Q3 and Q9, the performance benefit of +Tectorwise is cut in half". Q1's ratio moves from 0.56 to 0.66, Q18's +from 0.75 to 0.97. Hyper-threading hides microarchitecturally +sub-optimal code — so the more cores you have, the less the choice +costs you. Topic 19 revisits compilation; M11 goes vectorized. ## How to read the paper (with the concepts in hand) -~1.5 h. The scorecard sections matter more than the geometric means. - -- **§1–2** — the two models (Steps 2–3) and the shared-everything-else - methodology (Step 4). Verify the fairness claims: same hash table, - same storage. -- **§3 (micro-architectural analysis) — read carefully.** This is - Steps 5–6 measured: cache misses in flight, instructions per cycle, - loads/stores per row. The hash-probe and expression subsections are - the paper's core. -- **§4 (SIMD)** — the smaller-than-hoped gains; note *which* primitives - benefit (compute-bound only). -- **§5 (other factors) — don't skip.** Step 7 lives here: compile time, - profiling, adaptivity. For choosing an architecture, this section - outweighs the benchmarks. -- **§6–7** — related work and summary; skim, then re-read the scorecard - and argue with it. +~1.5 h. The scorecard sections matter more than the aggregate runtimes — +which is fortunate, since there are no aggregate runtimes. + +| Section | What is there | Step | +|---|---|---| +| §1-2 | the two models, and Figure 1's multi-predicate example — the cheapest illustration of constraint (i) in the paper | 2, 3 | +| §2.1-2.2 | why a primitive can only handle one type, and the hash join / group by pseudo-code for both engines | 2 | +| §3 | the fairness argument. Read §3 itself for the two caveats: compilation time excluded, five queries not the suite | 4 | +| §4.1 + Table 1 | **read carefully.** The 74%/32% spread; instructions and L1 misses "up to 2.4×/3.3×"; the out-of-order-window explanation; the IPC warning | 4, 5, 6 | +| §4.2 | interpretation is <1.5% of runtime, and the extra instructions are load/stores not dispatch; the 32 KB i-cache non-result | 2, 6 | +| §4.3 + Figure 5 | Tectorwise's own vector-size sweep — 1,000 default, degradation below 64 and above 64 K. X100's U-curve, re-measured 13 years later | 2 | +| §4.4 | the SSB table, with the mem-stall column that actually tracks the winner | 5 | +| §5 | SIMD: 8.4× in isolation, ~10% end-to-end, and §5.3's auto-vectorization result | 6 | +| §6 | both engines given morsel-driven parallelism; HyPer 11.7× vs VectorWise 7.2× is *exchange vs morsel*, not compiled vs vectorized | 4 | +| §8 | **don't skip.** The whole operational column; §8.4's adaptive-aggregation example is the best concrete thing in it | 7 | +| §9-10 | hybrid models (Figure 13's design space) and the five-bullet summary; read §10 last and check it against your own notes | — | + +## Takeaway + +The interesting result is the *shape* of the tie. Tectorwise runs up to +2.4× the instructions and 3.3× the L1 misses of Typer and still wins the +join queries, because the instructions it wastes are load/stores it +issues while the memory system is busy anyway, and its simple loops keep +the out-of-order window free to overlap misses. Typer wins wherever +there is nothing to overlap: Q1's in-cache arithmetic, where identical +LLC-miss counts prove the gap is pure materialization. + +So the question to ask of your own workload is not "which model" but +"which regime": does this query stall on memory, or compute in cache? +Graph traversal — probes and expands over a large adjacency structure — +lives in the stalled column, which is the M11 argument. And if the answer +is genuinely mixed, note that morsel parallelism and hyper-threading each +shrink the gap (Table 3), while §8's operational column does not shrink +at all. That is why a project with no LLVM budget and a need for +per-operator profiling can choose vectorization without losing an +argument about rows per second. ## Questions for notes.md 1. Why does vectorized probing overlap misses but the compiled loop doesn't? Connect to lookup_shootout (topic 0): what did MLP do for - HashMap throughput there? + HashMap throughput there? (§4.1's out-of-order-window sentence is the + mechanism; §4.4's mem-stall column is the proof.) 2. Software prefetching rescues compiled probes (they cite group prefetching / AMAC). Why is prefetching EASY in a vectorized kernel (you have the whole vector of hashes) and CONTORTED in a fused loop? 3. The "wide pipeline" case: 10 carried columns through 3 operators — - count the loads/stores per row for each model. Where did Tectorwise's - registers go? + count the loads/stores per row for each model. Check your count + against Table 1's Q1 row: 94 extra instructions and 1.4 extra L1 + misses per tuple, at identical LLC misses. §4.2 says where they go. 4. Your kernels.rs is a HAND-compiled Typer pipeline for one fixed query. Predict from the paper: will it beat your vectorized.rs on the - filter+sum workload (compute-bound, k dense)? By how much? + filter+sum workload (compute-bound, k dense)? By how much? (Q1 is the + closest analogue: 1.74×. Q6 — a pure selective filter — is a tie.) 5. M11 (and topic 19's JIT milestone): FalkorDB queries are pattern-matching heavy — probes and expands, memory-bound. Which column of the scorecard do graph workloads live in, and what does @@ -183,14 +488,127 @@ Topic 19 revisits compilation; M11 goes vectorized. ## Done when -You can argue BOTH sides for a graph engine in 3 sentences each, then -commit to one (spoiler: the scorecard's memory-bound row + operational -column point vectorized for M11; revisit at topic 19). +Answer each before unfolding it. + +- [ ] You can state the paper's result as a range with directions, not as a winner or a mean — and say why a mean would have been the wrong summary. + +
Answer + + §4.1, single-threaded, five TPC-H queries: Typer faster by 74% on Q1 + and by ~60% on Q18 (Table 1: 30 vs 48 cycles/tuple); a tie on Q6 (11 vs + 11); Tectorwise faster by 4% on Q3 and 32% on Q9. The paper's own + framing: "neither paradigm is clearly dominated by the other", against + a HyPer-vs-Postgres gap of one to two orders of magnitude. + + A mean is the wrong summary because the two halves of the spread have + *opposite causes* — register residency on compute-bound queries, + miss overlap on memory-bound ones — so averaging them reports a number + that predicts nothing about the next query. It would also imply a + ranking the paper spent twenty pages refusing to produce. + +
+ +- [ ] You can explain what Tectorwise's extra instructions actually are, with the measurement that rules out the obvious answer. + +
Answer + + They are load/store instructions materializing each primitive's result + into a vector — not interpretation. §4.2 rules interpretation out by + measurement: a profiler puts the interpreted part at "less than 1.5% of + the query runtime" at SF=10, so 98.5% of time is inside primitives, and + primitives "know all involved types at compile time". Whatever the + extra instructions are, they are executing inside typed loops. + + Table 1's Q1 row confirms the mechanism: 162 vs 68 instructions per + tuple and 2.0 vs 0.6 L1 misses, at an *identical* 0.57 LLC misses. The + extra traffic never reaches DRAM — it is the write-and-reread of + intermediates through L1, exactly what §2.1's Figure 1 predicts when + constraint (i) forces one `if` into two primitives. + +
+ +- [ ] You can say why Tectorwise wins Q9 despite running more instructions and taking more cache misses, and name the counter that shows it. + +
Answer + + Because identical miss counts do not mean identical miss costs. Both + engines use the same hash table and take almost the same number of LLC + misses (Table 1 Q9: 0.47 vs 0.46), but §4.1 explains that Tectorwise's + probe loop "is only a simple loop… the CPU's out-of-order engine can + speculate far ahead and generate many outstanding loads", while Typer's + fused loop carries scan, selection, probe and aggregation, so the + out-of-order window "fills up more quickly… thus they generate less + outstanding loads". + + The counter is memory stall cycles. §4.4's SSB Q4.1 row is the cleanest + case: Tectorwise runs 1.56× the instructions and takes 1.39× the L1 + misses, and finishes in 0.76× the cycles, because it stalls 19.48 + cycles per tuple against Typer's 45.91. The advantage grows with the + table — §4.1 measures it "up to 40% for larger data (and hash table) + sizes". + +
+ +- [ ] You can explain why a higher IPC does not mean a faster engine, using the paper's own example. + +
Answer + + Table 1, Q1: Tectorwise's IPC is 2.8 against Typer's 2.0 — 40% higher — + and it is 74% slower, because it executes 162 instructions per tuple + against 68. Retiring more instructions per cycle is worthless if the + extra instructions are load/stores you would not have issued in the + other design. + + §4.1's own conclusion: IPC "is a valid measure of the amount of free + processing resources, but should not be used as the sole proxy for + overall query processing performance". Two neighbouring guides make the + same point from other directions — [reading-x100.md](reading-x100.md), + where a low IPC is diagnostic of interpretation, and this repo's + FINDINGS row 17, where the branchless filter wins on work done, not on + instructions retired. + +
+ +- [ ] You can list the operational dimensions that do *not* shrink with better hardware, and say which one decides M11. + +
Answer + + §8's five: OLTP/stored procedures and multi-language support (both to + compilation); compile time, profiling and adaptivity (all three to + vectorization); implementation effort a wash with different shapes — + codegen indirection versus per-primitive constraints. These are + structural consequences of the architecture, so unlike the performance + column they do not move when you add cores. The performance column + does: Table 3 shows the gap narrowing at 20 hyper-threads for four of + five queries, Q9's Tectorwise advantage cut roughly in half. + + For M11 the deciding pair is the memory-bound row and compile time. + Graph pattern-matching is probes and expands over a structure larger + than cache, which is the regime where §4.1 puts vectorization ahead; + and vectorization's zero compile time plus §8.3's per-primitive + profiling are worth more to a project without an LLVM budget than a + ≤32% win on the queries that would have gone the other way. Topic 19 + reopens the question with JIT in hand. + +
## References **Papers** - Kersten, Leis, Kemper, Neumann, Pavlo, Boncz — "Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid - to Ask" (VLDB 2018) — ~1.5 h; the scorecard sections matter more than - the geometric means + to Ask" (VLDB 2018) — ~1.5 h. §4.1 with Table 1 and §8 are the two + sections to internalize; §3's two caveats (compilation time excluded, + five queries) govern how you may quote everything else + +**In this repo** +- [reading-x100.md](reading-x100.md) — the vectorized contender's origin, + and the 2.3× it left on the table that compilation went after +- [reading-morsel-parallelism.md](reading-morsel-parallelism.md) — the + parallelization framework §6 gives to both engines so it stops being a + variable +- [reading-duckdb-execution.md](reading-duckdb-execution.md) — a + production vectorized engine, choosing the §8 column deliberately +- [FINDINGS.md](../../FINDINGS.md) row 11 — the interpretation tax both + models exist to remove, measured here; row 17 for the branchless-filter + counterpart to Step 6's IPC warning diff --git a/topics/11-execution-models/reading-duckdb-execution.md b/topics/11-execution-models/reading-duckdb-execution.md index beff83f..b459fdf 100644 --- a/topics/11-execution-models/reading-duckdb-execution.md +++ b/topics/11-execution-models/reading-duckdb-execution.md @@ -8,65 +8,251 @@ salt-tagged join hash table — the data plane first, then the control plane, then the operator where the tricks pay off. Then it hands you the anchors to watch each step run. +Every anchor below is duckdb at the commit this repo pins, +`6c0c1a68` (`resources/codebases.md`). Quoted C++ carries its real line +numbers in the gutter; elisions are marked. Where a number could be +remembered wrong — the vector size, the salt width, the morsel size — +the guide quotes the constant instead of asserting it. + ## The problem in one sentence Run an analytical query over 100M rows without paying the per-row -interpretation tax (~20–100 ns × 100M rows × 5 operators = minutes of -overhead) and without materializing whole 800 MB intermediate columns to -RAM — the answer is to move data in 2048-row units that fit in cache. +interpretation tax — [reading-postgres-executor.md](reading-postgres-executor.md) +works it out at 2.5 s of pure dispatch for a 5-node plan, before any +column is touched — and without materializing whole 800 MB intermediate +columns to RAM. DuckDB's answer is to move data in units of 2048 rows, +sized so that the unit stays in cache between operators. ## The concepts, step by step ### Step 1 — the DataChunk: `next()` returns 2048 rows -DuckDB keeps the classic iterator structure (operators composed in a -tree, data flowing up), but the unit that moves between operators is a -**DataChunk**: a set of **vectors** — one contiguous array per column — -plus a row count, at most `STANDARD_VECTOR_SIZE = 2048` rows. All -per-call overhead (dispatch, operator state checks) now divides by 2048, -and the loops inside each operator are tight `for` loops over arrays — -auto-vectorizable, prefetcher-friendly. Why 2048 and not a million: a -chunk of 8 columns × 8 bytes × 2048 rows = 128 KB — sized so an -operator's working set stays in L1/L2 *between* operators. It's the -cache ladder of topic 0 turned into an engine constant, and the single -most consequential number in the codebase. +> **In:** a Volcano operator tree, where the unit crossing every operator +> boundary is one tuple and the per-call dispatch cost is therefore paid +> per row. +> **Out:** the same tree with the unit replaced by a **DataChunk** — up +> to 2048 rows of every column at once — and the dispatch cost divided by +> 2048. This is the change that defines the model; Steps 2-6 are all +> consequences of it. + +**Vectorized execution** means each operator call processes a *batch* of +rows rather than one, and the batch is laid out column-wise so the inner +loops are `for` loops over contiguous arrays. DuckDB's batch is the +`DataChunk`: a set of **vectors** — one contiguous array per column, all +of the same length — plus a row count. + +```cpp +// src/include/duckdb/common/types/data_chunk.hpp — the doc comment, 26-44, +// elided to the two sentences that matter, then the class and its payload. + 26 //! A Data Chunk represents a set of vectors. + 27 /*! + 28 The data chunk class is the intermediate representation used by the + 29 execution engine of DuckDB. It effectively represents a subset of a relation. + 30 It holds a set of vectors that all have the same length. +// ... 31-35: how Initialize allocates ... + 36 the chunk. The reason for this behavior is that the underlying vectors can + 37 become referencing vectors to other chunks as well (i.e. in the case an + 38 operator does not alter the data, such as a Filter operator which only adds a + 39 selection vector). +// ... 40-43: rest of the comment ... + 44 class DataChunk { + 45 public: + 46 //! Creates an empty DataChunk + 47 DUCKDB_API DataChunk(); + 48 DUCKDB_API ~DataChunk(); + 49 + 50 //! The vectors owned by the DataChunk. + 51 vector data; +``` -### Step 2 — vector type flags: metadata instead of work +Note lines 37-39 already: an operator that does not alter data — a filter +— produces its output by *adding a selection vector*, not by copying. That +is Step 3, promised in the class comment. + +The batch size is a compile-time constant, and worth reading rather than +remembering, because it is overridable and because the `#ifndef` says who +is allowed to override it: + +```cpp +// src/include/duckdb/common/vector_size.hpp — the whole constant, 15-25. + 15 //! The default standard vector size + 16 #define DEFAULT_STANDARD_VECTOR_SIZE 2048U + 17 + 18 //! The vector size used in the execution engine + 19 #ifndef STANDARD_VECTOR_SIZE + 20 #define STANDARD_VECTOR_SIZE DEFAULT_STANDARD_VECTOR_SIZE + 21 #endif + 22 + 23 #if (STANDARD_VECTOR_SIZE & (STANDARD_VECTOR_SIZE - 1) != 0) + 24 #error The vector size must be a power of two + 25 #endif +``` + +So `STANDARD_VECTOR_SIZE` is 2048 unless the build overrode it, and the +`#error` at 23-25 fixes it to a power of two — which matters later, because +the hash table's `bitmask` modulo trick and the row-group arithmetic in +Step 4 both assume it divides evenly. + +**Why 2048 divides the dispatch tax to nothing.** Take the postgres +guide's arithmetic and change only the unit. Same assumptions: 100M rows, +a 5-operator plan, `c = 20` cycles per operator call (a predicted indirect +call, the callee prologue, the reload of the row pointer), `f = 4 GHz`: + +``` + tuple-at-a-time: 100e6 × 5 = 500,000,000 calls + 500e6 × 20 / 4e9 = 2.5 s = 25 ns/row + vector of 2048: (100e6 / 2048) × 5 = 244,140 calls + 244,140 × 20 / 4e9 = 0.0012 s = 0.012 ns/row +``` + +Two thousand-fold on that term. A cost that dominated the query is now +below the noise floor of a single cache miss. -A vector doesn't have to be a plain array. Each carries a type flag, and -kernels dispatch on it — representing structure instead of expanding it: +**Why 2048 and not 64K.** The chunk is not free to grow, because it has +to *stay resident* between operators: the filter writes it, the aggregate +reads it, and if it was evicted in between, the second read is a memory +access rather than a cache hit. At 8 columns of 8-byte values: ``` - FLAT plain columnar array - CONSTANT one value stands for the whole vector (literals, and any - op whose inputs were constant — never expanded) - DICTIONARY selection vector over another vector (filter output, - decompressed dictionary data — flows through unexpanded) - SEQUENCE start + increment (row ids) - FSST still-compressed strings (topic 12) + bytes per row = 8 cols × 8 B = 64 B + 64 rows → 4 KB 2048 rows → 128 KB + 1024 rows → 64 KB 65536 rows → 4 MB ``` -The payoff is arithmetic: `2 * price` with a CONSTANT `2` runs one loop -over `price` and never materializes 2048 copies of the literal; -dictionary-compressed data flows through the engine without being -decompressed. The cost is combinatorial: a binary kernel faces -{flat, constant, dictionary}² input shapes. Question to hold: how does -DuckDB avoid writing 9 loops per operation? (Look for -`UnifiedVectorFormat` / `ToUnifiedFormat` — the normalize-then-one-loop -dodge, at the price of an indirection.) +Hold those against this machine's *measured* ladder, from +[topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md): +the L1d plateau reads 1.02 ns and "ends exactly" at 128 KB; 512 KB-1 MB +reads 5.3-5.8 ns; 4-8 MB reads 7.6-9.0 ns; the DRAM plateau reads ~25 ns. +A 2048-row × 8-column chunk is 128 KB — exactly one L1d. Widen the chunk +to 64K rows and the same eight columns are 4 MB, so every inter-operator +handoff is an L2 trip. Narrow it to 64 rows and you are back to paying +dispatch 32× more often. 2048 is where those two curves cross, and it is +the single most consequential number in the codebase. + +**What batching does not fix.** This repo's own Volcano lane +([FINDINGS.md](../../FINDINGS.md) row 11) tops out at 103.3 M rows/s and +gets *slower* as selectivity rises — 74.7 M rows/s at 95%. The cost that +grows is per *surviving* row, not per *evaluated* predicate. Dividing +dispatch by 2048 attacks the per-call term; it does nothing on its own +about the work each surviving row still causes downstream. Steps 3 and 6 +are about that half. + +### Step 2 — vector type flags: metadata instead of work + +> **In:** a DataChunk whose vectors are all plain arrays, so a literal +> `2` in `2 * price` must be expanded to 2048 copies of `2` before the +> multiply loop can run. +> **Out:** vectors that carry a *type flag*, so a literal is stored once, +> a filter result is stored as indices, and row ids are stored as +> `start + increment` — structure represented rather than expanded. + +A vector does not have to be a plain array. Each carries a `VectorType`, +and kernels dispatch on it: + +```cpp +// src/include/duckdb/common/enums/vector_type.hpp — the whole enum, 15-22. + 15 enum class VectorType : uint8_t { + 16 FLAT_VECTOR, // Flat vectors represent a standard uncompressed vector + 17 FSST_VECTOR, // Contains string data compressed with FSST + 18 CONSTANT_VECTOR, // Constant vector represents a single constant + 19 DICTIONARY_VECTOR, // Dictionary vector represents a selection vector on top of another vector + 20 SEQUENCE_VECTOR, // Sequence vector represents a sequence with a start point and an increment + 21 SHREDDED_VECTOR // Shredded variant vector + 22 }; +``` + +Six kinds, not the five this guide used to list: `SHREDDED_VECTOR` — the +shredded representation of a `VARIANT` column, where a semi-structured +value's common sub-fields are stored as real typed columns — is present at +this pin. + +The payoff is arithmetic. `2 * price` with a CONSTANT `2` runs one loop +over `price` and never writes 2048 copies of the literal: 2048 × 8 B = +16 KB of stores, and 16 KB of L1 that stays available for real data, +saved per chunk per constant. Dictionary-compressed data flows through +the engine still encoded (topic 12), and a SEQUENCE row-id vector costs +16 bytes instead of 16 KB. + +The cost is combinatorial: a binary kernel faces {flat, constant, +dictionary, sequence, …}² input shapes, and nobody writes 36 loops per +operation. The dodge is a normalizing view: + +```cpp +// src/include/duckdb/common/vector/unified_vector_format.hpp — the payload +// of the normalized view, 22-35 elided to its data members. + 22 struct UnifiedVectorFormat { +// ... 23-30: constructors, copy deleted, move kept ... + 31 const SelectionVector *sel; + 32 const_data_ptr_t data; + 33 ValidityMask validity; + 34 SelectionVector owned_sel; + 35 PhysicalType physical_type; +``` + +`Vector::ToUnifiedFormat` (`src/include/duckdb/common/types/vector.hpp:127`) +turns *any* vector kind into this `(sel, data, validity)` triple, and every +kernel then writes one loop of the shape `data[sel->get_index(i)]`. The +price is an indirection on every element — which is why the hot kernels +still specialize on FLAT and only fall back to the unified path. ### Step 3 — selection vectors: filtering without copying -A filter that copied its ~50 survivors out of 2048 rows into fresh -arrays would pay a copy per operator per chunk. Instead a filter's -entire output is a **SelectionVector** — a small index array `sel[]` -naming the surviving row positions — over the *same untouched* data -vectors. Every downstream kernel takes `(data, sel, count)` and iterates -`sel` instead of 0..2048; zero bytes of column data move until some -operator genuinely must materialize: +> **In:** a filter that must hand its ~1024 survivors, out of a 2048-row +> chunk, to the next operator. +> **Out:** the survivors expressed as a **selection vector** — an array +> of surviving row *positions* over the same untouched data vectors — so +> that zero bytes of column data move. This is **late materialization**: +> defer the copy until an operator genuinely needs a dense array. + +A **selection vector** is a small index array `sel[]` naming which +positions of the underlying vectors are live. Every downstream kernel +takes `(data, sel, count)` and iterates `sel` instead of `0..2048`: + +```cpp +// src/include/duckdb/common/types/selection_vector.hpp — write, read, and +// the storage. 124-127 and 134-140 in full, then the private member at 175. + 124 inline void set_index(idx_t idx, idx_t loc) { // NOLINT: allow casing for legacy reasons + 125 D_ASSERT(idx < capacity); + 126 sel_vector[idx] = UnsafeNumericCast(loc); + 127 } +// ... 128-133: swap(i, j) ... + 134 inline idx_t get_index(idx_t idx) const { // NOLINT: allow casing for legacy reasons + 135 return sel_vector ? get_index_unsafe(idx) : idx; + 136 } + 137 inline idx_t get_index_unsafe(idx_t idx) const { // NOLINT: allow casing for legacy reasons + 138 D_ASSERT(idx < capacity); + 139 return sel_vector[idx]; + 140 } +// ... 141-174: data(), Slice, Verify, Sort ... + 175 sel_t *sel_vector; +``` + +Line 135 is the piece worth stopping on: a null `sel_vector` means the +identity mapping. "No selection" is not a special case the caller must +test for — it is a selection vector whose `get_index(i)` returns `i`, so +the one kernel loop covers both the filtered and the unfiltered path. + +The economics, with `sel_t` being `uint32_t` +(`src/include/duckdb/common/typedefs.hpp:30`), 8 columns of 8-byte values, +and half the chunk surviving: + +``` + copy out 1024 survivors = 1024 rows × 8 cols × 8 B = 65,536 B written + build a selection vector = 1024 × sizeof(sel_t) = 4,096 B written + ratio = 16× less traffic +``` + +— and the 16× is the *floor*, because the copy also has to be paid again +by the next filter in the chain, while a selection vector composes: filter +two selects from filter one's output positions and still moves no data. + +The kernel shape, in Rust, so the loop is legible: ```rust -// every kernel takes (data, sel, count); a filter's OUTPUT is a new sel +// ILLUSTRATION — not quoted from duckdb; the real select loops are the +// templates in src/include/duckdb/common/vector_operations/unary_executor.hpp:310 +// (`SelectLoopSelSwitch`), which handle the vector-type and validity cases +// this omits. The (data, sel, count) shape and the branch-free body are real. fn filter_lt(v: &[i64], t: i64, sel: &[u32], out_sel: &mut [u32]) -> usize { let mut n = 0; for &i in sel { @@ -77,21 +263,31 @@ fn filter_lt(v: &[i64], t: i64, sel: &[u32], out_sel: &mut [u32]) -> usize { } ``` -Note the branch-free body: write unconditionally, advance the counter -only on match — no branch for the predictor to miss on 50%-selective -data (topic 0's branch_misprediction lesson, in production). A -DICTIONARY vector (Step 2) is this same trick promoted to a vector -representation. +The branch-free body is the point: write unconditionally, advance the +counter only on match, so there is no branch for the predictor to miss on +50%-selective data. [FINDINGS.md](../../FINDINGS.md) row 17 puts a number +on what that avoids — a branchy filter runs 0.95 GB/s where a branchless +one runs ~10 GB/s on the same data. A DICTIONARY vector (Step 2) is this +same trick promoted from a call argument to a vector representation. -### Step 4 — pipelines: the plan splits at materialization points +### Step 4 — pipelines: the plan splits at its breakers -Some operators can stream chunk-by-chunk (filter, projection); others -must consume *all* input before producing anything — a hash-join build -must see the whole build side, a sort must see every row. These are -**materialization points**, and they cut the plan into **pipelines**: -each pipeline is a source (scan, or a previous pipeline's output) → a -chain of streaming operators → a **sink** (the materializing operator). -For `SELECT k, SUM(v) FROM t JOIN s ... GROUP BY k`: +> **In:** an operator tree, which says what to compute but not what may +> run at the same time as what. +> **Out:** the same tree cut into **pipelines** at its **pipeline +> breakers**, each pipeline a schedulable unit with a parallelism degree +> and a dependency on the pipelines that must finish first. + +Some operators **stream**: a filter or a projection can consume one chunk +and emit its result immediately. Others must consume *all* their input +before they can emit anything — a hash-join build must see every build +row before the first probe is legal; a sort must see every row before it +knows what comes first. Those are **pipeline breakers**, and the act of +accumulating their whole input into memory is **materialization**. + +A **pipeline** is what lies between breakers: a **source** (a scan, or a +previous pipeline's materialized result) → a chain of streaming operators +→ a **sink** (the breaker). For `SELECT k, SUM(v) FROM t JOIN s ... GROUP BY k`: ``` pipeline 1: scan(s) ──────────────────► build hash table (sink) @@ -99,83 +295,327 @@ For `SELECT k, SUM(v) FROM t JOIN s ... GROUP BY k`: (runs only after pipeline 1's sink is complete) ``` -Pipelines are the scheduling unit: each can be run by many threads in -parallel, and dependencies (build before probe) gate execution. This is -also where morsel-driven parallelism plugs in -(reading-morsel-parallelism.md): the source hands out row-group-sized -work units (122880 rows = 60 vectors) that worker threads pull. +Pipelines are the scheduling unit: each may be run by many threads at +once, and dependencies (build before probe) gate execution. +`Pipeline::ScheduleParallel` (`src/parallel/pipeline.cpp:136-153`) decides +the degree by asking *both* ends — `TryGetMaxThreads` (`:101-134`) starts +from the source's `MaxThreads()`, lets every intermediate operator lower +it, clamps to `TaskScheduler::NumberOfThreads()`, and lets the sink lower +it again — and falls back to `ScheduleSequentialTask` (`:95`) when any +participant says no. + +This is where **morsel-driven parallelism** plugs in +([reading-morsel-parallelism.md](reading-morsel-parallelism.md)): rather +than partitioning the input once and pinning a partition per thread, the +source hands out small work units — **morsels** — that idle workers pull, +which is how the scheme both **work-steals** (a thread that finishes early +takes the next morsel instead of idling) and stays **NUMA-local** (a +worker prefers morsels whose pages are on its own socket). DuckDB's morsel +is a row group, and the size is computed, not hard-coded: + +```cpp +// src/storage/data_table.cpp — how many parallel work units a table offers. + 276 idx_t DataTable::MaxThreads(ClientContext &context) const { + 277 idx_t row_group_size = GetRowGroupSize(); + 278 idx_t parallel_scan_vector_count = row_group_size / STANDARD_VECTOR_SIZE; + 279 if (ClientConfig::GetConfig(context).verify_parallelism) { + 280 parallel_scan_vector_count = 1; + 281 } + 282 idx_t parallel_scan_tuple_count = STANDARD_VECTOR_SIZE * parallel_scan_vector_count; + 283 return GetTotalRows() / parallel_scan_tuple_count + 1; + 284 } +``` + +With `DEFAULT_ROW_GROUP_SIZE` at 122880 +(`src/include/duckdb/storage/storage_info.hpp:26`, and `:394` asserts it is +a multiple of the vector size), the arithmetic is exact: + +``` + vectors per row group = 122880 / 2048 = 60 vectors + morsel = 2048 × 60 = 122,880 rows + units for the 50 M-row lane = 50e6 / 122880 + 1 = 408 work units +``` + +408 units across 8 hardware threads is 51 morsels each — enough +granularity that a straggler costs at most one morsel of tail latency, and +coarse enough that the per-morsel scheduling cost is amortized over +122,880 rows. That is the "what bounds morsel size from below and above" +question, answered on this table. ### Step 5 — the executor protocol: push within, pull between -Inside a pipeline task, DuckDB is **push-based**: the executor fetches a -chunk from the source and pushes it through the operator chain into the -sink. Between tasks it's pull-based: workers pull work units from the -source. (Compare textbook Volcano: pull all the way down.) Pushing needs -a protocol for operators whose output size doesn't match their input — -each operator call returns an `OperatorResultType`: +> **In:** pipelines that need to run, and the awkward fact that operators +> do not preserve cardinality — a join can turn one 2048-row input chunk +> into ten output chunks. +> **Out:** a **push**-based loop inside each task (the executor drives +> chunks *down* into the sink) wrapped in a **pull**-based one between +> tasks (workers pull morsels), plus the four-state protocol that keeps +> memory bounded when the shapes do not match. + +In a **pull** model — the textbook Volcano one — control flows downward: +the root calls `next()` on its child, which calls `next()` on its child. +In a **push** model, control flows upward from the source: the executor +fetches a chunk and hands it to operator 0, then hands operator 0's result +to operator 1, and so on into the sink. DuckDB is push *inside* a pipeline +task and pull *between* tasks (workers pull morsels from the source). + +The reason push needs a protocol at all is cardinality. Each operator call +returns an `OperatorResultType`: + +```cpp +// src/include/duckdb/common/enums/operator_result_type.hpp — the contract, +// stated in its own comment. 15-27 in full. + 15 //! The OperatorResultType is used to indicate how data should flow around a regular (i.e. non-sink and non-source) + 16 //! physical operator + 17 //! There are four possible results: + 18 //! NEED_MORE_INPUT means the operator is done with the current input and can consume more input if available + 19 //! If there is more input the operator will be called with more input, otherwise the operator will not be called again. + 20 //! HAVE_MORE_OUTPUT means the operator is not finished yet with the current input. + 21 //! The operator will be called again with the same input. + 22 //! FINISHED means the operator has finished the entire pipeline and no more processing is necessary. + 23 //! The operator will not be called again, and neither will any other operators in this pipeline. + 24 //! BLOCKED means the operator does not want to be called right now. e.g. because its currently doing async I/O. The + 25 //! operator has set the interrupt state and the caller is expected to handle it. Note that intermediate operators + 26 //! should currently not emit this state. + 27 enum class OperatorResultType : uint8_t { NEED_MORE_INPUT, HAVE_MORE_OUTPUT, FINISHED, BLOCKED }; +``` + +Four states, not three — `BLOCKED` (24-26) is the async-I/O escape hatch, +and the comment is explicit that intermediate operators should not emit +it, so in practice the streaming operators you will read use three. +`HAVE_MORE_OUTPUT` is the interesting one: it exists so that an operator +which explodes its input does *not* buffer the explosion internally. The +executor calls it again with the same input. + +The loop that implements this: + +```cpp +// src/parallel/pipeline_executor.cpp — the source-to-sink loop inside +// Execute(max_chunks) (260). 296-301 fetch, 319-323 push. + 296 } else if (!exhausted_pipeline || next_batch_blocked) { + 297 SourceResultType source_result = SourceResultType::BLOCKED; + 298 if (!next_batch_blocked) { + 299 // "Regular" path: fetch a chunk from the source and push it through the pipeline + 300 source_chunk.Reset(); + 301 source_result = FetchFromSource(source_chunk); +// ... 302-318: BLOCKED/FINISHED handling and the batch-index path ... + 319 if (exhausted_pipeline && source_chunk.size() == 0) { + 320 continue; + 321 } + 322 + 323 result = ExecutePushInternal(source_chunk, chunk_budget); +``` + +`ExecutePushInternal` (`:375-422`) then loops the chunk through the +operator chain via `Execute(input, result, idx)` (`:483`) and into +`Sink` — and its own `do … while (chunk_budget.Next())` at 387-420 is the +`HAVE_MORE_OUTPUT` loop in the flesh: it re-executes the *same* input +until the operator says `NEED_MORE_INPUT` (417-419) or the budget runs out. + +Why the protocol beats internal buffering, on numbers: memory in flight +per thread is one chunk per operator. -- `NEED_MORE_INPUT` — done with this chunk, push me the next; -- `HAVE_MORE_OUTPUT` — I wasn't finished with this input (a join that - exploded one 2048-row chunk into many output chunks); call me again - with the SAME input before fetching more; -- `FINISHED` — this pipeline can stop early (a LIMIT was satisfied). +``` + 5 operators × 128 KB/chunk = 640 KB per worker thread + × 8 worker threads = 5.1 MB for the whole query + the same join buffering internally: 50e6 rows × 10× fanout × 64 B = 32 GB +``` -`HAVE_MORE_OUTPUT` exists because operators must not buffer unbounded -output internally — memory stays bounded at ~one chunk per operator, -and the ownership of chunks stays with the executor (question 3 below). +Bounded memory is not a nice property here; it is the difference between +running and not running. ### Step 6 — the join hash table: salt bits before pointer chases -The hash join is where the vectorized machinery pays off. Build side: -each thread collects its chunks into partitioned row-format storage -(thread-local, no contention), merged at the end — the morsel-driven -two-phase pattern. The table itself stores 8-byte entries = a pointer to -the tuple + **salt** bits (a few bits of the key's hash smuggled into -the entry — topic 2's bit-smuggling): a probe compares the salt FIRST, -and since most non-matching probes fail the salt compare, they are -rejected without ever dereferencing the pointer — no cache miss on the -tuple. The probe is vectorized end to end: hash all 2048 keys, gather -all their buckets, salt-compare en masse, build a selection vector of -candidates, compare actual keys only for those. Batching the bucket -gathers is exactly what lets the core overlap the cache misses — -memory-level parallelism, the reason vectorized probes win in the -VLDB'18 shootout (reading-compiled-vs-vectorized.md). +> **In:** a probe side of 2048 keys per chunk and a hash table far larger +> than cache, where the naive probe costs two *dependent* cache misses per +> key — load the entry, then dereference its pointer to compare the key. +> **Out:** a probe that answers most non-matches from the first load +> alone, because the entry's unused pointer bits carry a **salt** — a +> slice of the key's hash — and a mismatched salt proves a mismatched key. + +Build side: each thread collects its chunks into thread-local partitioned +row-format storage (no contention), and the partitions are merged at the +end — `JoinHashTable::Merge` (`src/execution/join_hashtable.cpp:149-187`), +whose `sink_collection->Combine` is line 169. That is the morsel-driven +two-phase pattern of Step 4 applied to a hash table. + +The table itself is **open-addressed with linear probing** — not chained — +and each slot is one 8-byte word doing two jobs: + +```cpp +// src/include/duckdb/execution/ht_entry.hpp — the split, 33-37, and the +// extraction, 73-80. + 33 #else + 34 //! Upper 16 bits are salt, lower 48 bits are the pointer + 35 static constexpr const hash_t SALT_MASK = 0xFFFF000000000000; + 36 static constexpr const hash_t POINTER_MASK = 0x0000FFFFFFFFFFFF; + 37 #endif +// ... 38-72: constructors, IsOccupied, GetPointer, SetPointer ... + 73 // Returns the salt, leaves upper salt bits intact, sets lower bits to all 1's + 74 static inline hash_t ExtractSalt(const hash_t &hash) { + 75 return hash | POINTER_MASK; + 76 } + 77 + 78 inline hash_t GetSalt() const { + 79 return ExtractSalt(value); + 80 } +``` + +So the salt is **16 bits** wide (line 34, and `SALT_MASK` at 35 spells the +split out), living in the top of a pointer that only needs 48. This is +topic 2's bit-smuggling, and the payoff is a probe loop that dereferences +nothing until the salt agrees: + +```cpp +// src/execution/join_hashtable.cpp — the salted probe, 243-266, inside +// ProbeForPointersInternal (232). + 243 if (USE_SALTS) { + 244 // increment the ht_offset of the entry as long as the next entry is occupied and salt does not match + 245 while (true) { + 246 const ht_entry_t entry = entries.get()[row_ht_offset]; + 247 const bool occupied = entry.IsOccupied(); + 248 + 249 // the entry is empty -> no match possible + 250 if (!occupied) { + 251 break; + 252 } + 253 + 254 const hash_t row_salt = ht_entry_t::ExtractSalt(row_hash); + 255 const bool salt_match = entry.GetSalt() == row_salt; + 256 if (salt_match) { + 257 // we know that the entry is occupied and the salt matches -> compare the keys + 258 auto row_index = GetOptionalIndex(row_sel, i); + 259 AddPointerToCompare(state, entry, pointers_result_v, row_ht_offset, keys_to_compare_count, + 260 row_index); + 261 break; + 262 } + 263 + 264 // full and salt do not match -> continue probing + 265 IncrementAndWrap(row_ht_offset, ht.bitmask); + 266 } +``` + +Line 250 breaks on an empty slot; 254-255 compare salts; only 259 records +a pointer for the key comparison to follow. A non-matching key whose salt +differs never touches the tuple. + +Worked, with the real `k = 16`: + +``` + spurious salt match on a non-matching key = 2^-16 = 1 / 65,536 + of a 2048-key probe chunk, expected false hits = 0.031 keys + tuple dereferences avoided per chunk of all-misses ≈ 2047.97 + at the measured ~25 ns DRAM plateau (topic 0 notes) ≈ 51 µs saved +``` + +That the salt is a *cache-miss* optimization and not a compare +optimization is not an inference — DuckDB turns it off when there are no +misses to save: + +```cpp +// src/include/duckdb/execution/join_hashtable.hpp — when salting is worth it. + 93 //! only compare salts with the ht entries if the capacity is larger than 8192 so + 94 //! that it does not fit into the CPU cache + 95 static constexpr const idx_t USE_SALT_THRESHOLD = 8192; +``` + +Check the threshold against the ladder: 8192 entries × 8 B = 64 KB, which +fits inside this machine's measured 128 KB L1d. Below the threshold the +entry load is an L1 hit, the pointer chase is an L2 hit, and the salt +compare would be pure added instructions — so it is compiled out +(`UseSalt()`, `:370-373`, selects between two template instantiations). + +And the whole probe is vectorized, which is what makes the misses overlap: + +```cpp +// src/execution/join_hashtable.cpp — GetRowPointersInternal (300): probe the +// whole vector, compare the whole vector, re-probe only the non-matches. + 323 do { + 324 const idx_t keys_to_compare_count = ProbeForPointers(state, ht, entries, pointers_result_v, row_sel, + 325 elements_to_probe_count, has_row_sel); + 326 + 327 // if there are no keys to compare, we are done + 328 if (keys_to_compare_count == 0) { + 329 break; + 330 } + 331 + 332 // Perform row comparisons, after Match function call salt_match_sel will point to the keys that match + 333 keys_no_match_count = 0; + 334 const idx_t keys_match_count = + 335 ht.row_matcher_build.Match(keys, key_state.vector_data, state.keys_to_compare_sel, keys_to_compare_count, + 336 pointers_result_v, &state.keys_no_match_sel, keys_no_match_count); +// ... 337-350: append the matches to match_sel ... + 351 for (idx_t i = 0; i < keys_no_match_count; i++) { + 352 const auto row_index = state.keys_no_match_sel.get_index(i); + 353 auto ht_offset_and_salt = ht_offsets_and_salts[row_index]; + 354 IncrementAndWrap(ht_offset_and_salt, ht.bitmask | ht_entry_t::SALT_MASK); + 355 hashes_dense[i] = ht_offset_and_salt; // populate dense again + 356 } + 357 + 358 // in the next iteration, we have a selection vector with the keys that do not match + 359 row_sel = state.keys_no_match_sel; + 360 has_row_sel = true; + 361 + 362 elements_to_probe_count = keys_no_match_count; + 363 + 364 } while (DUCKDB_UNLIKELY(keys_no_match_count > 0)); +``` + +Read 351-362: linear probing's "keep walking until you find it" is +expressed as a *selection vector of the keys that have not yet resolved*, +fed back into the next round. That is Step 3's mechanism reused as control +flow. And because the 2048 entry loads at 246 are issued from a loop with +no dependency between iterations, the core can have many of them in flight +at once — **memory-level parallelism**, the reason a vectorized probe beats +a fused compiled loop that resolves one key at a time +([reading-compiled-vs-vectorized.md](reading-compiled-vs-vectorized.md)). ## Where each step lives in the code Read in this order: the vector types (the data plane), then the pipeline -executor (the control plane), then the join hash table. - -- **Step 1**: `src/include/duckdb/common/vector_size.hpp:16–20` — - `STANDARD_VECTOR_SIZE = 2048`; - `src/include/duckdb/common/types/data_chunk.hpp:44` — `DataChunk` = - a set of vectors + count. This is what `next()` returns. -- **Step 2**: `src/include/duckdb/common/enums/vector_type.hpp:15` — - the vector kinds; chase `UnifiedVectorFormat` / `ToUnifiedFormat` - from any kernel. -- **Step 3**: `src/include/duckdb/common/types/selection_vector.hpp:31` - — `SelectionVector`, the filter-without-copying mechanism. -- **Step 4**: `src/parallel/pipeline.cpp:136` — - `Pipeline::ScheduleParallel`: asks source AND sink whether they - support parallelism, creates one `PipelineTask` per allowed thread; - `:95` sequential fallback. - `src/execution/operator/scan/physical_table_scan.cpp:77` — - `MaxThreads` from the source's global state: ~one unit per row group - (122880 rows, `storage_info.hpp:26`) — DuckDB's morsel size. -- **Step 5**: `src/parallel/pipeline_executor.cpp:260` — - `Execute(max_chunks)`: the main loop. Fetch a chunk from source - (`:281`), push it through the operator chain — `ExecutePushInternal - :375`, which walks operators via `Execute(input, result, idx) :483` — - into the sink. The `OperatorResultType` protocol lives here. -- **Step 6**: `src/execution/join_hashtable.cpp` — build side `Sink` - collects chunks into partitioned row-format storage - (`sink_collection`; `:169` `Combine` merges thread-local partitions). - `ht_entry_t::ExtractSalt` `:195` — the 8-byte pointer+salt entry. - Probe (`ProbeState`, header `:206`): hash a whole chunk - (`VectorOperations::CombineHash` `:393`), gather entries, salt-compare - en masse, then compare actual keys via selection vector; chains - handled with `ResidualPredicateProbeState` selection juggling (header - `:74–:80`). +executor (the control plane), then the join hash table (where it pays). +All anchors are duckdb `6c0c1a68`. + +| File | Lines | What is there | Step | +|---|---|---|---| +| `src/include/duckdb/common/vector_size.hpp` | 15-25 | `DEFAULT_STANDARD_VECTOR_SIZE 2048U`, overridable, power-of-two enforced | 1 | +| `src/include/duckdb/common/types/data_chunk.hpp` | 26-51 | `DataChunk` = `vector data` + count; the comment already names the filter/selection-vector case | 1 | +| `src/include/duckdb/common/enums/vector_type.hpp` | 15-22 | the six `VectorType` kinds | 2 | +| `src/include/duckdb/common/vector/unified_vector_format.hpp` | 22-35 | `(sel, data, validity)` — the normalize-then-one-loop dodge | 2 | +| `src/include/duckdb/common/types/vector.hpp` | 127 | `ToUnifiedFormat` — the entry point to it | 2 | +| `src/include/duckdb/common/types/selection_vector.hpp` | 124-140, 175 | `set_index`, `get_index` (null `sel` = identity, 135), `sel_t *sel_vector` | 3 | +| `src/include/duckdb/common/typedefs.hpp` | 30 | `typedef uint32_t sel_t` — 4 bytes per selected row | 3 | +| `src/include/duckdb/common/vector_operations/unary_executor.hpp` | 310 | `SelectLoopSelSwitch` — the real select kernels | 3 | +| `src/parallel/pipeline.cpp` | 95, 101-134, 136-153 | sequential fallback; `TryGetMaxThreads`; `ScheduleParallel` | 4 | +| `src/storage/data_table.cpp` | 276-284 | `MaxThreads` — morsels are row groups, computed here | 4 | +| `src/include/duckdb/storage/storage_info.hpp` | 26, 394 | `DEFAULT_ROW_GROUP_SIZE 122880ULL`; asserted a multiple of the vector size | 4 | +| `src/include/duckdb/common/enums/operator_result_type.hpp` | 15-27 | the four-state operator contract, with its own rationale | 5 | +| `src/parallel/pipeline_executor.cpp` | 260, 301, 375-422, 483 | `Execute(max_chunks)`; `FetchFromSource`; `ExecutePushInternal`; the per-operator `Execute` | 5 | +| `src/execution/join_hashtable.cpp` | 149-187 | `Merge` — thread-local partitions combined at 169 | 6 | +| `src/include/duckdb/execution/ht_entry.hpp` | 33-37, 73-80 | 16 salt bits / 48 pointer bits in one word; `ExtractSalt` | 6 | +| `src/execution/join_hashtable.cpp` | 232-279 | `ProbeForPointersInternal` — the salted linear probe | 6 | +| `src/execution/join_hashtable.cpp` | 300-368, 370-373 | vectorized probe rounds driven by a non-match selection vector; `UseSalt()` | 6 | +| `src/include/duckdb/execution/join_hashtable.hpp` | 93-95 | `USE_SALT_THRESHOLD = 8192` — and *why*, in the comment | 6 | + +## Takeaway + +The whole engine is one decision propagated: make the unit 2048 rows, and +then live with the consequences. Dispatch stops mattering (Step 1), so the +representation of a batch becomes worth optimizing (Step 2). A batch is +too expensive to copy, so filters return indices instead (Step 3). Batches +must be scheduled, so the plan is cut at its breakers (Step 4) and +operators need a protocol for not matching cardinalities (Step 5). And a +batch of independent probes is exactly what a core needs to overlap cache +misses, which is where the model's largest wins actually come from (Step +6) — not from the dispatch it saved, but from the memory-level parallelism +it made possible. + +What it does *not* buy you is on the other side of the ledger. This repo's +Volcano lane ([FINDINGS.md](../../FINDINGS.md) row 11) shows the +tuple-at-a-time cost rising with selectivity — 103.3 M rows/s at 50%, +74.7 M at 95% — because the expensive thing is a row *surviving*, not a +predicate being evaluated. Batching divides the per-call term by 2048; the +per-surviving-row term is what Steps 3 and 6 are for. ## Questions for notes.md @@ -186,23 +626,133 @@ executor (the control plane), then the join hash table. 3. `HAVE_MORE_OUTPUT`: which operators need it and why can't they just buffer internally? (Memory bound + who owns the chunk.) 4. The salt trick: with 64-bit hashes and k salt bits, what fraction of - non-matching probes still chase a pointer? Pick k. + non-matching probes still chase a pointer? Pick k — then check yours + against DuckDB's 16. 5. M11: your Expand operator explodes one source node into deg(n) results — that's `HAVE_MORE_OUTPUT` shaped. Sketch the state it must keep between calls. ## Done when -You can draw a pipeline for `SELECT k, SUM(v) FROM t JOIN s ... GROUP BY k` -(two pipelines, which is the sink of which), and explain selection -vectors + the salt trick in two sentences each. +Answer each before unfolding it. + +- [ ] You can say what a DataChunk is, what bounds its size from above and from below, and quote the constant rather than remembering it. + +
Answer + + A DataChunk is a set of vectors — one contiguous array per column, all + the same length — plus a row count; it is the unit that crosses every + operator boundary (`data_chunk.hpp:44-51`). Its length is + `STANDARD_VECTOR_SIZE`, which `vector_size.hpp:16` sets to + `DEFAULT_STANDARD_VECTOR_SIZE 2048U` unless the build overrides it, and + which lines 23-25 require to be a power of two. + + From below, the bound is dispatch amortization: at 100M rows through a + 5-operator plan and 20 cycles a call, tuple-at-a-time costs 2.5 s of pure + dispatch (25 ns/row) and a 2048-row unit costs 0.0012 s (0.012 ns/row). + Shrink the chunk to 64 rows and you give 32× of that back. + + From above, the bound is cache residency between operators. Eight 8-byte + columns is 64 B/row, so 2048 rows is exactly 128 KB — the size at which + topic 0's measured latency ladder says this machine's L1d plateau ends. + At 65,536 rows the same chunk is 4 MB, so every handoff from one operator + to the next reads from L2 (7.6-9.0 ns measured) rather than L1 (1.02 ns). + +
+ +- [ ] You can explain how a filter produces output without copying data, and why "no selection vector" is not a special case. + +
Answer + + Its output is a `SelectionVector` — an array of `sel_t` (`uint32_t`, + `typedefs.hpp:30`) holding the *positions* that survived — layered over + the same, untouched data vectors. Downstream kernels take + `(data, sel, count)` and iterate `sel`. For 1024 survivors of 8 8-byte + columns that is 4,096 bytes written instead of 65,536: 16× less traffic, + and it composes, because a second filter selects from the first's + positions and still moves nothing. + + "No selection" is not special because `get_index` + (`selection_vector.hpp:134-136`) returns `idx` when `sel_vector` is null. + An absent selection vector *is* the identity mapping, so one kernel loop + serves the filtered and unfiltered paths and there is no branch on + "was there a filter" in the hot loop. + +
+ +- [ ] You can draw the two pipelines for `SELECT k, SUM(v) FROM t JOIN s ... GROUP BY k` and say which operator is the sink of which. + +
Answer + + Pipeline 1: `scan(s)` → build the hash table, and the build is the sink. + Pipeline 2: `scan(t)` → probe the hash table → the hash aggregate, which + is *its* sink. Pipeline 2 depends on pipeline 1 having finished, because + the build is a pipeline breaker: it cannot emit anything until it has + seen every build row. The probe, by contrast, streams — it is an + intermediate operator in pipeline 2, not a sink. + + The cut is always at a breaker, and a breaker is exactly an operator + that must materialize its whole input (build, sort, the aggregate's hash + table) before producing a first row. + +
+ +- [ ] You can explain the salt trick in two sentences, give the real salt width, and say why DuckDB switches it off for small tables. + +
Answer + + DuckDB stores each hash-table slot as a single 8-byte word whose low 48 + bits are the tuple pointer and whose high 16 bits are a slice of the + key's hash — the salt (`ht_entry.hpp:34-36`). A probe compares the salt + first (`join_hashtable.cpp:254-255`), and since a differing salt proves a + differing key, all but 2^-16 = 1/65,536 of non-matching probes are + rejected from the entry load alone, never dereferencing the pointer. + + It is switched off below `USE_SALT_THRESHOLD = 8192` + (`join_hashtable.hpp:93-95`) because the saving is a *cache miss*, not a + comparison: 8192 entries × 8 B is 64 KB, which fits in this machine's + measured 128 KB L1d, so there is no miss to avoid and the salt compare + would be pure added work. `UseSalt()` (`:370-373`) picks between two + template instantiations, so the branch does not exist at run time either. + +
+ +- [ ] You can say why `HAVE_MORE_OUTPUT` exists rather than letting operators buffer their own output. + +
Answer + + Because buffering is unbounded and the executor's memory is not. An + operator that returns `HAVE_MORE_OUTPUT` + (`operator_result_type.hpp:20-21`) is telling the executor "call me again + with the *same* input" — so the explosion is drained one chunk at a time + through `ExecutePushInternal`'s loop + (`pipeline_executor.cpp:387-420`) instead of accumulating. Peak memory is + therefore one chunk per operator per thread: 5 operators × 128 KB × 8 + threads ≈ 5.1 MB, against 50e6 rows × 10× join fanout × 64 B = 32 GB if + the join buffered its own output. + + The second reason is ownership: chunks belong to the executor, which + reuses them (`final_chunk.Reset()`, `:390`). An operator that buffered + would have to own — and allocate — its output instead. + +
## References **Code** -- [duckdb](https://github.com/duckdb/duckdb) — the data plane: - `src/include/duckdb/common/vector_size.hpp`, +- [duckdb](https://github.com/duckdb/duckdb) at `6c0c1a68` — the data + plane: `src/include/duckdb/common/vector_size.hpp`, `enums/vector_type.hpp`, `types/data_chunk.hpp`, - `types/selection_vector.hpp`; the control plane: - `src/parallel/pipeline.cpp`, `src/parallel/pipeline_executor.cpp`; - the payoff: `src/execution/join_hashtable.cpp`; ~2 h + `types/selection_vector.hpp`, `vector/unified_vector_format.hpp`; the + control plane: `src/parallel/pipeline.cpp`, + `src/parallel/pipeline_executor.cpp`, `src/storage/data_table.cpp`; the + payoff: `src/execution/join_hashtable.cpp`, + `src/include/duckdb/execution/ht_entry.hpp`; ~2 h + +**In this repo** +- [FINDINGS.md](../../FINDINGS.md) row 11 — the Volcano ceiling this model + is trying to beat, and the direction it moves in +- [FINDINGS.md](../../FINDINGS.md) row 17 — branchy vs branchless filter + throughput, the number behind Step 3's branch-free kernel +- [topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md) + — the measured cache ladder every size argument here is checked against diff --git a/topics/11-execution-models/reading-morsel-parallelism.md b/topics/11-execution-models/reading-morsel-parallelism.md index 5b288d8..ca3fe3c 100644 --- a/topics/11-execution-models/reading-morsel-parallelism.md +++ b/topics/11-execution-models/reading-morsel-parallelism.md @@ -7,163 +7,648 @@ work units instead of receiving static partitions — and everything else falls out of it. This chapter builds the six concepts behind that sentence, then routes you through the paper. +Every figure below is checked against the paper and cited to its section +or table. Two claims this guide used to carry did not survive that check, +and one of them is the most useful thing here: **a morsel is not a +vector, and morsel size is not a cache parameter.** The correction is in +Step 3. + ## The problem in one sentence -Split one query across 8 cores by statically giving each core 1/8 of the -data, and one skewed partition leaves 7 cores idle while 1 grinds — the -query runs at 1/8 speed exactly when parallelism was supposed to pay. +Split one query across 64 hardware threads by statically giving each +1/64 of the data and the query finishes when the *slowest* thread does — +which §5.4 measures at 36.8% slower than ideal from nothing more exotic +than one unrelated single-threaded process occupying one of the 64 +cores. ## The concepts, step by step ### Step 1 — the classical answer: exchange operators and static partitions -Volcano-era parallelism (the "exchange" model): the OPTIMIZER picks a -**degree of parallelism** (DOP — the number of threads working on the -query) at plan time, and inserts **exchange operators** — special plan -nodes that split data into static partitions, run copies of the plan -fragment on each, and merge results. Parallelism lives *in the plan*. -Three costs follow directly: the DOP is frozen at optimize time while -the machine's load changes per second; exchange operators materialize -and copy rows between workers; and the plan itself explodes into -parallel variants the optimizer must now reason about. +> **In:** a query plan and a machine with more than one core. +> **Out:** the industry-standard way to connect them — parallelism baked +> into the plan at compile time — and the three costs that follow from +> *when* the decision is made, before any argument about skew. + +Volcano-era parallelism is what §1 calls **plan-driven**: "the optimizer +statically determines at query compile-time how many threads should run, +instantiates one query operator plan for each thread, and connects these +with exchange operators". The **degree of parallelism** (DOP — how many +threads work on this query) is a plan property. An **exchange operator** +is a plan node that routes tuple streams between threads, splitting the +input into partitions and merging results, so that "operators are kept +largely unaware of parallelism". + +That is a real virtue — §1 notes the approach "allows to implement +parallelism without affecting existing query operators", which is why +Oracle, SQL Server and Vectorwise all use it. Three costs come with it, +and all three follow from the decision being made *early*: + +- the DOP is frozen at optimize time, while the machine's load changes + by the second; +- exchange operators materialize and copy rows between workers, and §1 + argues the on-the-fly partitioning they perform "does not always lead + to the optimal plan (as partitioning effort does not always pay off)"; +- the plan explodes into parallel variants the optimizer must reason + about. + +### Step 2 — the failure mode: the slowest thread sets the runtime + +> **In:** a static split of the input across N threads. +> **Out:** the actual failure mode, which is *not* the one this guide +> used to name — plus the paper's measurement of it on a workload with +> no skew in it at all. + +The obvious story is **skew**: some partitions carry far more work than +others — a hot key range, a filter that passes 90% in one region and 1% +elsewhere — so the thread holding the hot partition grinds while the +rest idle. That story is true, and it is not the whole failure. + +**Correction:** this guide previously attributed stranding to skew +alone. TPC-H is, in the paper's words, "fully uniform" — there is no +data skew to blame — and exchange-based Vectorwise still strands +workers on it. §5.2, on the *trivially* parallelizable scan-only Q6: +"the slowest thread often finishes work 50% before the last. While in +real-world scenarios it is usually data skew that challenges load +balancing, this is not the case in the fully uniform TPC-H." + +§1 names the real enemy, and it is broader: perfect load balancing must +survive "uncertain size distributions of intermediate results, as well as +the hard-to-predict performance of modern CPU cores that varies even if +the amount of work they get is the same". Equal work is not equal time. +Frequency scaling, an SMT sibling, a noisy neighbour, or a P-core versus +an E-core all break a static split, and none of them is visible to an +optimizer. + +§5.4 puts a number on it by emulating the exchange model inside the +morsel engine — setting morsel size to `n/t`, one chunk per thread: + +``` + §5.4, TPC-H on 4-socket Nehalem EX (32 cores, 64 hardware threads): + + single query at a time, uniform data, nothing else running + static (morsel = n/t) no significant difference + -> a static split is fine exactly when nothing perturbs it + + same runs, with one unrelated single-threaded process on one core + static (morsel = n/t) 36.8% slower + dynamic (morsel = 100K) 4.7% slower -### Step 2 — the failure mode: skew strands workers + one core out of 64 disturbed. 1/64 = 1.6% of the machine, + and the static plan loses 36.8% of the query. +``` -**Skew** (some partitions having far more work than others — a hot key -range, a filter that passes 90% in one region and 1% elsewhere) breaks -static partitioning: the thread holding the hot partition grinds while -the other N−1 finish and idle. The whole comparison in one table: +The whole comparison in one table: ``` exchange model morsel model ───────────── ──────────── plan fixes DOP at optimize time DOP changes per SECOND - static partitions → skew strands workers PULL 100K-row morsels; - workers (one hot partition = one fast workers just pull more - busy thread, N-1 idle) + static partitions -> the slowest workers PULL 100K-row morsels; + thread sets the runtime, from fast workers just pull more + skew OR from core-speed variance exchange = extra materialization + same pipeline object shared by copying between workers all workers, zero exchange ops plan explosion (parallel variants) one plan, parallelism is runtime ``` -You already measured this without naming it: topic 9's scaling.rs — -static key-range split vs the shootout's shared-queue pulling. +And the end-to-end scoreboard, §5.2 (both systems on the same machine, +scalability = speedup from 1 thread to 64): + +``` + system geo. mean sum scalability + HyPer (morsel-driven) 0.45 s 15.3 s 28.1x + Vectorwise 2.5 (exchange) 2.84 s 93.4 s 9.3x + Vectorwise, full-disclosure settings 1.19 s 41.2 s 8.4x +``` + +Note what this is and is not evidence for. Both systems have "similar +single-threaded performance" (§5.2); the 6.3× gap in geometric mean is +almost entirely the 28.1× versus 9.3× in the last column. It is a +comparison of *parallelization frameworks*, not of execution models — +Vectorwise is vectorized and HyPer is compiled, but four years later the +same group built both models on top of morsel parallelism and found them +tied ([reading-compiled-vs-vectorized.md](reading-compiled-vs-vectorized.md) +§6). Morsel parallelism is orthogonal to Steps 2-3 of that guide, which +is exactly why it won. + +You already measured the underlying effect without naming it: topic 9's +`scaling.rs` — static key-range split vs the shootout's shared-queue +pulling. ### Step 3 — the morsel: work units small enough to rebalance +> **In:** the need to decide *who does what* later than plan time. +> **Out:** the unit of that decision, the loop that consumes it, and — +> the part most summaries get wrong — the fact that its size is a +> scheduling parameter with a floor, not a cache parameter with an +> optimum. + The fix inverts control: instead of *assigning* data to workers, workers -**pull**. A **morsel** is a small run of input (~100K tuples); a -dispatcher keeps a queue of them per pipeline (a pipeline being the -chain of operators between materialization points — see the DuckDB -guide); each worker grabs one morsel, runs it through the WHOLE pipeline -(scan → filter → probe → partial aggregate), then grabs the next. -Pipelines with dependencies (build before probe) gate on completion -events. Skew now dissolves by construction — a slow morsel just means -that worker pulls fewer; there is no partition to be stuck with. The -worker loop IS the design: +**pull**. A **morsel** is a small run of input tuples — §3: "We +experimentally determined that a morsel size of about 100,000 tuples +yields good tradeoff between instant elasticity adjustment, load +balancing and low maintenance overhead." A **pipeline** is the chain of +operators between materialization points (see +[reading-duckdb-execution.md](reading-duckdb-execution.md)); a +**pipeline breaker** is the operator that ends one by having to consume +its whole input first. A worker grabs one morsel, runs it through the +*whole* pipeline, materializes into the next pipeline breaker, and grabs +the next. + +Three structural details that summaries drop, all from §3 and §3.2: + +- **The threads are fixed and pinned.** One worker per hardware thread, + pre-created and permanently bound, so "the level of parallelism of a + particular query is not controlled by creating or terminating threads, + but rather by assigning them particular tasks of possibly different + queries" — and "no unexpected loss of NUMA locality can occur due to + the OS moving a thread to a different core" (§1). +- **The dispatcher is not a thread.** It "is implemented as a lock-free + data structure only. The dispatcher's code is then executed by the + work-requesting query evaluation thread itself", because a real + dispatcher thread "would need a core to run on" and "could become a + source of contention, in particular if the morsel size was configured + quite small". The `QEPobject` that gates pipelines on their + dependencies is likewise "a passive state machine", run on the worker + that just found the queue empty. +- **Morsels are cut on demand.** The per-core lists in Figure 5 are + illustrative; the implementation keeps "storage area boundaries for + each core/socket and segment[s] these large storage areas into morsels + on demand". + +The worker loop is the design: ```rust +// ILLUSTRATION — not quoted from any repo. The paper gives no worker +// loop in code; this is Figure 5 plus §3-§4.4 written out. The shipped +// shapes are polars' work-stealing executor +// (crates/polars-async/src/executor/mod.rs:236 try_steal_task) and its +// Morsel type (crates/polars-stream/src/morsel.rs:82) — see +// reading-rust-execution-stack.md. fn worker(dispatcher: &Dispatcher, ht: &BuildHt) { - let mut local_agg = PartialAgg::new(); // thread-local: no contention - while let Some(m) = dispatcher.pull(my_socket()) { // prefer LOCAL morsels, - let chunk = scan(m); // steal remote when starved - let sel = filter(&chunk); // the WHOLE pipeline runs - let matches = probe(ht, &chunk, &sel); // here, one thread, so - local_agg.update(&matches); // intermediates stay hot - } // commit unit = one morsel: - dispatcher.combine(local_agg); // that's the elasticity + let mut local_agg = PartialAgg::new(); // §4.4 phase 1: fixed-size, + while let Some(m) = dispatcher.pull(my_core()) { // spills when full + let chunk = scan(m); // per-CORE list of morsels + let sel = filter(&chunk); // allocated on this socket; + let matches = probe(ht, &chunk, &sel); // steal remote when starved + local_agg.update(&matches); // whole pipeline, one thread + } // preemption happens HERE, + dispatcher.flush_partitions(local_agg); // at morsel boundaries only } ``` -Morsel size is a trade: too small and per-morsel scheduling overhead +Now the part to unlearn. **Correction:** this guide previously said +morsel size is "a trade: too small and per-morsel scheduling overhead dominates; too big and you're back to coarse partitions that can't -rebalance (question 1 below). +rebalance", and question 1 below asked for the bound "above" in terms of +cache. The cache half is contradicted by §3.3, which opens by +distinguishing morsels from vectors precisely on this point: + +``` + §3.3, in full on the point that matters: + "In contrast to systems like Vectorwise and IBM's BLU, which use + vectors/strides to pass data between operators, there is no + performance penalty if a morsel does not fit into cache. Morsels + are used to break a large task into small, constant-sized work + units to facilitate work-stealing and preemption. Consequently, + the morsel size is not very critical for performance, it only + needs to be large enough to amortize scheduling overhead while + providing good response times." + + Figure 6 (select min(a) from R, 64 threads, Nehalem EX, morsel + size swept 100 .. 10M) is therefore a FLOOR, not a U: + below ~10,000 work-stealing structure overhead shows + above ~10,000 flat — "the morsel size should be set to the + smallest possible value where the overhead is + negligible, in this case to a value above 10,000" + far too large "results in underutilized threads but does not + affect throughput of the system if enough + concurrent queries are being executed" +``` + +A vector's size is set by the cache +([reading-x100.md](reading-x100.md) derives 8K × 40 B = the machine's +320 KB); a morsel's is set by scheduling overhead below and response +time above. They are different parameters answering different questions, +which is why DuckDB carries both — a 2048-value `DataChunk` *and* a +122,880-row row group as its work unit +([reading-duckdb-execution.md](reading-duckdb-execution.md)) — and why +120× separates them. + +§3.3 also explains why the shared work-stealing structure does not +become the bottleneck: work is initially split so each thread "temporarily +owns a local range", each range is **cache-line aligned** so "conflicts +at the cache line level are unlikely", and stealing only starts when a +local range is exhausted. + +One more consequence, cheaply won: **query cancellation**. A cancelled +query is marked in the dispatcher and "the marker is checked whenever a +morsel of that query is finished", so workers stop within a morsel and — +unlike killing threads — "this approach allows each thread to clean up". ### Step 4 — NUMA awareness: run the pipeline where the data lives -On multi-socket machines, memory is **NUMA** (non-uniform memory access: -each socket has local RAM, and touching another socket's RAM costs ~2× -the latency and shares an interconnect). The morsel design absorbs this -with one preference rule: morsels are *placed* on sockets, and a worker -prefers pulling morsels local to its socket, stealing remote ones only -when starved (`dispatcher.pull(my_socket())` above). Because the same -thread runs the whole pipeline on its morsel, intermediate results stay -socket-local automatically — no exchange operator ever ships them -across the interconnect. +> **In:** a machine where "the computer has become a network in itself" +> (§1) — four sockets, four memory controllers, an interconnect between +> them. +> **Out:** the single preference rule that makes the morsel design +> NUMA-aware, and the measurement showing it worked. + +**NUMA** (non-uniform memory access) means each socket has local RAM; +reaching another socket's RAM costs more latency and consumes +interconnect bandwidth that other threads are also using. On the paper's +Sandy Bridge EP some pairs are not directly connected at all, so "some +memory accesses (e.g., from socket 0 to socket 2) require two hops" +(§5.1). + +The design absorbs this with a preference, not a partitioning. §3.1: "For +each core a separate list exists to ensure that a work request of, say, +Core 0 returns a morsel that is allocated on the same socket as Core 0." +**Work stealing** is the escape hatch: "If, for some reason, a core +finishes processing all morsels on its particular socket, the dispatcher +will 'steal work' from another core… On some NUMA systems, not all +sockets are directly connected with each other; here it pays off to steal +from closer sockets first. Under normal circumstances, work-stealing from +remote sockets happens very infrequently; nevertheless it is necessary to +avoid idle threads." + +Because one thread runs the whole pipeline on its morsel, intermediates +never cross the interconnect — and outputs follow the *worker*, not the +input: "a red morsel turns blue if it was processed by a blue core in the +process of stealing work from the core(s) on the red socket" (§3.2). + +The evidence is bandwidth, §5.3 (Table 1, Nehalem EX): + +``` + TPC-H Q1 (aggregates the largest relation), HyPer: + read bandwidth achieved 82.6 GB/s + theoretical maximum of the machine 100 GB/s = 83% of peak + remote accesses low across most queries + most heavily used QPI link not saturated + + Vectorwise on the same query: + remote accesses 75% + -> "shows that its buffer manager is not NUMA-aware" +``` + +**Correction:** this guide previously priced a remote access at "~2× the +latency". The paper gives no such ratio; what it measures is the +*fraction* of accesses that go remote and the QPI saturation that +results. Quote those instead. ### Step 5 — elasticity: the commit unit is one morsel -Since a worker commits to only one morsel at a time, the engine can -change effective DOP *mid-query*: a new query arrives, and workers -simply finish their current ~100K-row morsel (a millisecond or two) and -switch queues. Compare canceling or rebalancing a static-partition plan -mid-flight — the partition is the commit unit, and it's the whole -input/DOP. "Elasticity" means precisely this: **commit granularity = one -morsel**. +> **In:** a running query holding all 64 threads, and a new query +> arriving. +> **Out:** the property that makes reassignment cheap, expressed as a +> time bound you can compute. + +Since a worker commits to only one morsel at a time, "preemption of a +task occurs at morsel boundaries – thereby eliminating potentially costly +interrupt mechanisms" (§3). §3.1: the engine can "gracefully decrease the +degree of parallelism of, say a long-running query `Ql` at any stage of +processing in order to prioritize a possibly more important interactive +query `Q+`", and when `Q+` finishes "the pendulum swings back". Figure 13 +is the profiler trace: four workers on TPC-H Q13, Q14 arrives, workers 2 +and 3 finish their current morsels and switch, then return to Q13. + +The load-balancing consequence is a *bound*, and this is what makes it +different in kind from "the fast workers do more". §3.2: all threads on +one pipeline job "run to completion in a 'photo finish': they are +guaranteed to reach the finish line within the time period it takes to +process a single morsel." + +Size that bound with this repo's own per-row cost as a stand-in: + +``` + FINDINGS.md row 11 — 9.68 ns per scanned row through + scan -> filter -> group-by-sum at 50% selectivity (M3 Pro). + + one morsel = 100,000 rows x 9.68 ns = 0.97 ms + <- the maximum any thread can be left waiting + + a 600M-row scan (TPC-H SF-100 lineitem) on 64 threads: + ideal per-thread work = 600e6 / 64 x 9.68 ns = 90.8 ms + photo-finish window = 0.97 / 90.8 = 1.1% + + the same query, static split, with one of the 64 cores + running at half speed: + that thread's share takes 2 x 90.8 = 181.5 ms + the query waits for it = +100% + morsel version: the other 63 absorb its work = +0.8% + + the paper's measured version of that last pair, §5.4: + static 36.8% slower vs dynamic 4.7% slower +``` + +"Elasticity" therefore means precisely this: **commit granularity = one +morsel**, so both the reassignment latency and the load-imbalance +penalty are bounded by one morsel's processing time rather than by the +input size. ### Step 6 — shared state only at pipeline breakers -Within a morsel, a worker touches only its own data — zero -synchronization. Sharing is confined to pipeline BREAKERS (the -materializing sinks): for aggregation, either thread-local partial hash -tables merged at pipeline end (the `combine` call above), or one shared -global hash table with atomic inserts for the join build — the paper -uses the latter, lock-free (topic 9's toolbox). Which of the two wins -depends on group count: 64 groups fit in every thread's cache and merge -in microseconds; 64M groups make merging cost real (question 2). +> **In:** morsels flowing through pipelines independently. +> **Out:** the two places threads must actually meet — and the fact that +> the paper solves them with *different* mechanisms, chosen for a reason +> worth stealing. + +Within a morsel a worker touches only its own data: zero +synchronization. Sharing is confined to pipeline breakers, and the paper +builds two of them. + +**Hash join build — one shared table, lock-free (§4.1, §4.2).** Two +phases. First, build-side tuples are materialized into a *thread-local* +storage area, "this requires no synchronization". Then, since the input +size is now known exactly, "an empty hash table is created with the +perfect size… much more efficient than dynamically growing hash tables, +which incur a high overhead in a parallel setting". Second, each thread +scans its own area and inserts pointers with atomic compare-and-swap: + +``` + Figure 7 — lock-free insertion into the tagged hash table: + + insert(entry) { + slot = entry->hash >> hashTableShift + do { + old = hashTable[slot] + entry->next = removeTag(old) // chain + new = entry | (old&tagMask) | tag(entry->hash) // OR in the tag + } while (!CAS(hashTable[slot], old, new)) + } + + the pointer layout that makes one CAS enough: + 16 bit tag | 48 bit pointer = 64 bits, one atomic word +``` + +The tag is an early filter — every element of a bucket list sets its bit +in it — so "for selective probes… the filter usually reduces the number +of cache misses to 1 by skipping the list traversal". Encoding it inside +the pointer "saves space and, more importantly, allows to update both the +pointer and the tag using a single atomic compare-and-swap operation". + +The 16/48 split is worth remembering because you have already seen it: +DuckDB's `ht_entry` uses the same 16-bit-plus-48-bit word +([reading-duckdb-execution.md](reading-duckdb-execution.md)). The +mechanisms differ — the paper's table is *chained*, and the tag is a +per-bucket-list filter; DuckDB's is open-addressed with linear probing, +and its 16 bits are a *salt* compared against the probe key's own salt. +Same word layout, same motivation (one cache line, one atomic), different +collision strategy. + +**Aggregation — partitioning, not a shared table (§4.4).** Also two +phases, but built the other way round. Phase 1 is a thread-local +*fixed-size* hash table that "efficiently aggregates heavy hitters", and +"when this small pre-aggregation table becomes full, it is flushed to +overflow partitions". After all input is partitioned, partitions are +exchanged between threads; phase 2 has each thread aggregate a whole +partition into a thread-local table, repeating since "there are more +partitions than worker threads", and pushing each finished partition +downstream immediately so "the aggregated tuples are likely still in +cache". + +The design note is the transferable part: "the aggregation operator is +fundamentally different from join in that the results are only produced +after all the input has been read. Since pipelining is not possible +anyway, we use partitioning – not a single hash table as in our join +operator." And the whole shape is chosen "without relying on query +optimizer estimates" — few groups are absorbed by the fixed-size local +table and never spill; many groups spill and get partitioned. The +structure handles both cases rather than picking one from a cardinality +estimate. + +**What the paper deliberately does not do.** Bushy parallelism — running +two independent pipelines of the same query at once — is available and +declined (§3.2): "the number of independent pipelines is usually much +smaller than the number of cores, and the amount of work in each +pipeline generally differs. Furthermore, bushy parallelism can decrease +performance by reducing cache locality. Therefore, we currently avoid to +execute multiple pipelines from one query in parallel." Intra-pipeline +parallelism at morsel granularity is enough. ## How to read the paper (with the concepts in hand) -~1 h. §2–3 for the design, skim the NUMA eval if you live on a laptop. +~1 h. §3 and §3.3 are the two sections to read carefully; the NUMA +evaluation is skimmable on a laptop, but §5.4's static-vs-dynamic +experiment is not — it is the paper's cleanest single result. -- **§1–2** — the case against exchange (Steps 1–2) and the morsel - design (Step 3). The figures showing per-socket morsel queues are - Steps 3–4 in one picture. -- **§3 — the core**: dispatcher, pipeline gating, the NUMA preference - rule (Step 4), elasticity (Step 5), and the lock-free shared build HT - (Step 6). -- **§4–5 (evaluation)** — skim unless you have sockets; note the skew - experiments confirming Step 2's failure mode and its dissolution. +| Section | What is there | Step | +|---|---|---| +| §1 | the case against plan-driven parallelism, and the sentence about cores "that varies even if the amount of work they get is the same" — the real enemy | 1, 2 | +| §2 + Figures 1-4 | the three-pipeline example query; how a plan becomes pipeline jobs; the QEPobject observing dependencies | 3 | +| §3 + Figure 5 | **the core.** Pinned workers, tasks = (pipeline job, morsel), preemption at morsel boundaries, the 100,000-tuple figure, the three scheduling goals | 3, 4, 5 | +| §3.1-3.2 | elasticity; per-core morsel lists; the lock-free dispatcher with no thread of its own; the "photo finish" bound; work stealing; why bushy parallelism is declined; query cancellation | 3, 4, 5 | +| §3.3 + Figure 6 | **read carefully.** Why a morsel is not a vector, and why its size is a floor rather than an optimum | 3 | +| §4.1-4.2 + Figure 7 | two-phase build, exact-size table, the 16-bit tag inside the 48-bit pointer, CAS insertion | 6 | +| §4.4 + Figure 8 | two-phase aggregation by partitioning, and the sentence explaining why it differs from the join | 6 | +| §5.1-5.2 | the 28.1× vs 9.3× scalability table, and Q6's load-imbalance result on uniform data | 2 | +| §5.3 + Table 1 | NUMA: 82.6 GB/s of a 100 GB/s peak, remote-access percentages, QPI saturation | 4 | +| §5.4 + Figure 13 | **don't skip.** The 36.8% vs 4.7% emulation of static assignment, and the profiler trace of a query yielding cores mid-flight | 2, 5 | -Where you've already seen the idea shipped: +Where you have already seen the idea shipped: -- DuckDB: row-group (122880 rows) work units + `MaxThreads` on sources — - morsels without the NUMA half (laptops don't have sockets). -- polars-stream: `Morsel` + `MorselSeq` + source tokens — morsels with - explicit ordering and backpressure (reading-rust-execution-stack.md). -- Your topic 9 scaling.rs: you measured the skew-stranding effect +- **DuckDB**: row-group (122,880 rows) work units + `MaxThreads` on + sources — morsels without the NUMA half (laptops do not have sockets). +- **polars-stream**: `Morsel` + `MorselSeq` + source tokens, over its own + work-stealing executor — morsels with explicit ordering and + backpressure ([reading-rust-execution-stack.md](reading-rust-execution-stack.md)). + Its `DEFAULT_IDEAL_MORSEL_SIZE` is 100,000, the paper's number + unchanged after a decade. +- **Your topic 9 `scaling.rs`**: you measured the stranding effect without naming it. +## Takeaway + +The paper's contribution is not "pull instead of push" — it is moving +one decision from compile time to run time and discovering how much falls +out. Once the unit of assignment is a morsel rather than a partition, load +balancing becomes a bound rather than a hope ("photo finish", one morsel +wide), elasticity is free (preemption at morsel boundaries, no interrupt +mechanism), NUMA locality is a preference on a queue rather than a +partitioning pass, and cancellation is a flag checked at the same +boundary. Four features, one decision. + +Two things to carry into your own scheduler. First, size the work unit +for *scheduling*, not for cache — §3.3 is explicit that a morsel +overflowing cache costs nothing, and Figure 6 shows a floor around +10,000 with a flat plateau above it. Copying a vector-size intuition here +is the classic error. Second, the thing that breaks static splits is not +mainly skewed data; it is that equal work is not equal time. §5.4's one +busy core costing 36.8% is a laptop-scale result: a P-core and an E-core +handed identical partitions will finish at different moments, and only +the pulling design absorbs it. + ## Questions for notes.md -1. Morsel size tradeoff: 100K rows vs DuckDB's 122880 vs your topic 7 - batch findings — what bounds it below (scheduling overhead per - morsel) and above (load-balance granularity + cache)? Same - amortize-and-batch curve as everywhere else. -2. Two-phase aggregation (thread-local HTs + merge) vs the paper's - shared lock-free build HT: which wins for 64 groups? For 64M groups? - (Contention vs merge cost — your exec_bench has 64 dense groups; - predict.) +1. Morsel size: DuckDB's 122,880-row row group and the paper's 100,000 + tuples land in the same place, while X100's vector is 1,024. §3.3 + says the morsel bound below is scheduling overhead (Figure 6: above + ~10,000) and that cache does *not* bound it above — so what does? + State the bound above in units of *time*, then check it against your + topic 7 batch findings and say which of those bounds are really + vector bounds. +2. The paper uses a shared lock-free table for the join build (§4.2) and + partitioning for aggregation (§4.4). Write the sentence from §4.4 + that explains the difference, then predict: for `exec_bench`'s 64 + dense groups, does phase 1's fixed-size local table ever spill? For + 64M groups? Which phase dominates in each case? 3. Ordering: morsel pulling destroys tuple order. What does the paper - (and polars' MorselSeq) do when ORDER BY needs it back, and what does - that cost? + (and polars' `MorselSeq`) do when ORDER BY needs it back, and what + does that cost? (§4.5 for the parallel merge sort; note the paper + sorts *only* for `order by` / top-k.) 4. On a MacBook (no NUMA, but P-cores vs E-cores): does the heterogeneous-core problem look MORE like NUMA or more like skew? - Which mechanism (locality preference vs dynamic pulling) addresses - it? + §1's "hard-to-predict performance of modern CPU cores that varies + even if the amount of work they get is the same" and §5.4's 36.8% + are the paper's own answer — which mechanism (locality preference or + dynamic pulling) addresses it, and which is dead weight on a laptop? 5. M11: FalkorDB is single-writer, many-reader (M8/M9 decisions). A read query's Expand over a big frontier — morselize the FRONTIER? - What's the natural morsel for SpMV (row-block of the matrix?). This + What's the natural morsel for SpMV (row-block of the matrix?). Note + §3.2's reason for declining bushy parallelism before deciding. This is the M11 parallelism design question — write a paragraph. ## Done when -You can explain skew-stranding with the one-hot-partition picture and -say precisely what "elasticity" means (commit granularity = one morsel). +Answer each before unfolding it. + +- [ ] You can state what actually strands workers under static partitioning, with the paper's measurement on a workload that has no skew in it. + +
Answer + + Not skew — or not only skew. §1: load balancing must survive "uncertain + size distributions of intermediate results, as well as the + hard-to-predict performance of modern CPU cores that varies even if the + amount of work they get is the same". Equal work is not equal time. + + The proof is on uniform data. §5.2, on the trivially parallel scan-only + TPC-H Q6 under Vectorwise: "the slowest thread often finishes work 50% + before the last. While in real-world scenarios it is usually data skew + that challenges load balancing, this is not the case in the fully + uniform TPC-H." And §5.4 isolates it: emulate static assignment by + setting morsel size to `n/t` and TPC-H barely changes — until one + unrelated single-threaded process occupies one of 64 cores, at which + point static loses 36.8% and morsel-driven loses 4.7%. One core in 64 + is 1.6% of the machine. + +
+ +- [ ] You can say why morsel size is not the same kind of parameter as vector size, and give the paper's rule for setting it. + +
Answer + + §3.3 draws the distinction itself: "In contrast to systems like + Vectorwise and IBM's BLU, which use vectors/strides to pass data + between operators, there is no performance penalty if a morsel does not + fit into cache. Morsels are used to break a large task into small, + constant-sized work units to facilitate work-stealing and preemption. + Consequently, the morsel size is not very critical for performance." + + So the rule is a floor, not an optimum: "the morsel size should be set + to the smallest possible value where the overhead is negligible, in + this case to a value above 10,000" (Figure 6, `select min(a) from R` + on 64 threads — chosen because it "stresses the work-stealing data + structure as much as possible"). Above the floor the curve is flat, and + an over-large morsel "results in underutilized threads but does not + affect throughput of the system if enough concurrent queries are being + executed". + + A vector size, by contrast, is derived from cache — X100's 8K × 40 + bytes = the machine's 320 KB. Different question, different answer; + DuckDB carries both constants at once. + +
+ +- [ ] You can state the load-balancing guarantee as a bound and compute it for a concrete query. + +
Answer + + §3.2: all threads working on one pipeline job "run to completion in a + 'photo finish': they are guaranteed to reach the finish line within the + time period it takes to process a single morsel". The worst-case + imbalance is one morsel, independent of input size — which is exactly + what a static split cannot promise, since there the bound is one + partition. + + Using this repo's measured 9.68 ns per scanned row (FINDINGS row 11) as + a stand-in: one 100,000-row morsel is 0.97 ms. A 600M-row scan on 64 + threads gives each thread 90.8 ms of ideal work, so the photo-finish + window is 0.97/90.8 = 1.1% of the runtime. Under a static split, one + core running at half speed doubles the query's runtime; under morsels + the other 63 threads absorb its share for about +0.8%. §5.4's measured + 36.8% vs 4.7% is the same experiment on real hardware. + +
+ +- [ ] You can explain why the dispatcher has no thread of its own, and what it costs to preempt a worker. + +
Answer + + §3.2 rejects a dispatcher thread on two grounds: "(1) the dispatcher + itself would need a core to run on or might preempt query evaluation + threads and (2) it could become a source of contention, in particular + if the morsel size was configured quite small". So it is "implemented + as a lock-free data structure only" and "the dispatcher's code is then + executed by the work-requesting query evaluation thread itself", on the + core that is momentarily between morsels. The `QEPobject` gating + pipelines on their dependencies is likewise a passive state machine run + by whichever worker discovers the queue empty. + + Preemption costs nothing beyond finishing the current morsel: §3 says + it "occurs at morsel boundaries – thereby eliminating potentially + costly interrupt mechanisms", and Figure 13 shows workers 2 and 3 + leaving TPC-H Q13 for Q14 and returning. Query cancellation rides the + same boundary — a marker "checked whenever a morsel of that query is + finished", which unlike killing a thread "allows each thread to clean + up". + +
+ +- [ ] You can say why the join build and the aggregation use different shared-state strategies, in the paper's own terms. + +
Answer + + §4.4: "the aggregation operator is fundamentally different from join in + that the results are only produced after all the input has been read. + Since pipelining is not possible anyway, we use partitioning – not a + single hash table as in our join operator." + + The join build is a shared, lock-free, *tagged* table: materialize + build tuples thread-locally with no synchronization, size the table + exactly (input size is now known — "much more efficient than + dynamically growing hash tables"), then CAS pointers in, with a 16-bit + tag packed into the same 64-bit word as the 48-bit pointer so one + atomic updates both. The tag is an early filter that "usually reduces + the number of cache misses to 1" on selective probes. + + Aggregation instead pre-aggregates into a fixed-size thread-local table + that spills to overflow partitions when full, exchanges partitions + between threads, then aggregates each partition thread-locally and + pushes it downstream immediately while it is still cache-hot. The point + of the shape is robustness "without relying on query optimizer + estimates": few groups never spill, many groups partition. + +
## References **Papers** - Leis, Boncz, Kemper, Neumann — "Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age" (SIGMOD 2014) — - ~1 h; §2–3 for the design, skim the NUMA eval if you live on a laptop + ~1 h. §3 and §3.3 for the design and the morsel-vs-vector distinction, + §4.2 and §4.4 for the two shared-state strategies, §5.4 for the single + cleanest experiment in the paper + +**In this repo** +- [reading-rust-execution-stack.md](reading-rust-execution-stack.md) — + polars-stream's `Morsel`, `MorselSeq` and work-stealing executor, with + the paper's 100,000 still the default +- [reading-duckdb-execution.md](reading-duckdb-execution.md) — row groups + as morsels, and the 16/48 hash-table word in a different collision + scheme +- [reading-compiled-vs-vectorized.md](reading-compiled-vs-vectorized.md) + — §6 there gives morsel parallelism to both execution models, which is + the evidence that this framework is orthogonal to that argument +- [FINDINGS.md](../../FINDINGS.md) row 11 — the per-row cost used to size + the photo-finish bound in Step 5 diff --git a/topics/11-execution-models/reading-postgres-executor.md b/topics/11-execution-models/reading-postgres-executor.md index 435c369..39af2d6 100644 --- a/topics/11-execution-models/reading-postgres-executor.md +++ b/topics/11-execution-models/reading-postgres-executor.md @@ -4,26 +4,45 @@ Tuple-at-a-time execution, still shipping: postgres's executor is the honest per-tuple baseline your benchmark's `volcano.rs` models. Before the code, this chapter builds the iterator model and its two dispatch costs — a function pointer per plan node per tuple, an opcode per -expression step — one concept at a time, ending at the one place -postgres already fought back (the computed-goto expression interpreter). -Then it hands you the file:line anchors. +expression step per tuple — one concept at a time, ending at the one +place postgres already fought back (the computed-goto expression +interpreter). Then it hands you the file:line anchors. + +Every anchor below is postgres master at commit `701f021` — the commit +this repo pins, which `configure.ac:20` calls `20devel` — quoted with the +line numbers the code occupies in that tree. ## The problem in one sentence -Postgres pays ~1 indirect function call per plan node per tuple plus an +Postgres pays one indirect function call per plan node per tuple plus an interpreted opcode per expression step per tuple — negligible for a -3-row OLTP lookup, but a 5-node plan over 100M rows burns 500M indirect +3-row OLTP lookup, and a 5-node plan over 100M rows burns 500M indirect branches before any useful work happens. ## The concepts, step by step ### Step 1 — the iterator (Volcano) model: `next()` returns one tuple -In the Volcano model (Graefe, 1990), every operator — scan, filter, -aggregate, join — implements the same three-call interface: -`open() / next() / close()`, where `next()` returns exactly ONE tuple -(row). The root operator's `next()` calls its child's `next()`, which -calls *its* child's, down to the scan: +> **In:** nothing yet — this step fixes the vocabulary every later step +> costs out. +> **Out:** the three-call operator interface, and the one design decision +> (the unit that crosses it) that Steps 2 to 7 price. + +An **operator** is one node of a physical plan — a scan, a filter, an +aggregate, a join — that consumes rows and produces rows. In the +**Volcano model** (Graefe, 1990; also called the **iterator model**), +every operator implements the same three calls: `open()` to set up, +`next()` to produce, `close()` to tear down. **`next()` returns exactly +one tuple** — one row — which is what makes this a **tuple-at-a-time** +engine: the unit that crosses the operator boundary is a single row. + +Control flows the other way from data. Execution is **pull-based** +(demand-driven): the root asks its child for a tuple, that child asks +*its* child, and the request travels down to the scan, which returns one +row back up the chain. The opposite arrangement, **push-based**, has the +source hand a batch downward into the operators that consume it — that +is what DuckDB does inside a pipeline (reading-duckdb-execution.md +Step 5). Postgres pulls, all the way down: ``` Project.next() @@ -35,58 +54,294 @@ calls *its* child's, down to the scan: ``` The elegance is real: operators compose arbitrarily (any tree of -next()-speaking boxes works), execution is demand-driven (a LIMIT stops +`next()`-speaking boxes works), execution is demand-driven (a LIMIT stops pulling and everything upstream stops), and memory stays bounded (one -tuple in flight per operator). The cost is the subject of this guide — -and of this entire topic. +tuple in flight per operator, so a 100 GB sort-free query needs no 100 GB +buffer). The cost is the subject of this guide — and of this topic. + +Why it matters: every later step is a price tag attached to one of those +three properties, and every alternative in this topic keeps the tree and +changes the unit. ### Step 2 — the price: an indirect call per node per tuple -In postgres, "call the child's next()" is `ExecProcNode(node)`, which is -just `return node->ExecProcNode(node);` — a call through a function -pointer (an **indirect call**: the target address is data, loaded at -runtime, so the CPU must predict where it's going; a misprediction -flushes the pipeline for ~15 cycles). That's one per plan node per -tuple: a 5-node plan over 100M rows = **500M indirect branches** before -any work happens. Worse, between two `next()` calls the tuple's values -leave CPU registers entirely — every operator re-loads what its child -just had in hand. At ~20 ns of such overhead per tuple per operator, -100M rows × 5 operators = minutes spent NOT computing. This is the -number vectorization divides by 2048. - -### Step 3 — a production wart worth stealing: self-modifying dispatch - -Postgres's node dispatch has a cute optimization: every node is -*initialized* with its function pointer set to `ExecProcNodeFirst`, a -wrapper that performs one-time checks (stack depth, instrumentation -setup) and then REPLACES the node's pointer with the real -`ExecProcNodeReal` — so the steady-state path never pays for the checks -again. Self-modifying dispatch: the first call does setup, then swaps -itself out. You've seen the pattern as lazy statics and memoized FFI +> **In:** the operator tree from Step 1. +> **Out:** two numbers — the dispatch count a plan generates, and this +> repo's own measured cost per row — plus the surprise in the second one +> that Step 7 returns to. + +In postgres, "call the child's `next()`" is `ExecProcNode(node)`, a +four-line inline function: + +```c +// src/include/executor/executor.h — the whole of ExecProcNode, 314-329 + 314 /* ---------------------------------------------------------------- + 315 * ExecProcNode + 316 * + 317 * Execute the given node to return a(nother) tuple. + 318 * ---------------------------------------------------------------- + 319 */ + 320 #ifndef FRONTEND + 321 static inline TupleTableSlot * + 322 ExecProcNode(PlanState *node) + 323 { + 324 if (node->chgParam != NULL) /* something changed? */ + 325 ExecReScan(node); /* let ReScan handle this */ + 326 + 327 return node->ExecProcNode(node); + 328 } + 329 #endif +``` + +The line that carries the argument is **327**, and note that it is not +the whole function: 324-325 test `chgParam` on every call, because a +correlated subplan's parameter may have changed since the last tuple. +`node->ExecProcNode` is a field, so 327 is an **indirect call** — a call +whose target address is loaded from data at runtime rather than encoded +in the instruction, which means the CPU must *predict* where it is going +and pays a pipeline refill when it predicts wrong. A **branch +misprediction** is exactly that refill: the core has already begun +executing down the wrong path and must throw that work away. + +One such call per plan node per tuple. The formula, with its symbols +named: + +``` + dispatches = N × D + where N = rows pulled through the pipeline + D = plan depth (operators between the scan and the root) + + t_dispatch = dispatches × c / f + where c = cycles per call+return+callee prologue + f = core clock in cycles per second +``` + +Worked, on the topic's own shape — 100M rows through a 5-node plan, and +`c = 20` cycles as a stated assumption (a predicted indirect call, the +callee prologue, and the reload of the tuple pointer the caller no longer +had in a register), `f = 4e9`: + +``` + dispatches = 100e6 × 5 = 500,000,000 indirect calls + cycles = 500e6 × 20 = 10,000,000,000 cycles + t_dispatch = 1.0e10 / 4.0e9 = 2.5 s of dispatch alone + per row = 2.5 / 100e6 = 25 ns/row before any work +``` + +Now the same arithmetic with the call amortised over a **vector** — a +batch of values of one column handed across the operator boundary +instead of a single row (Step 1's unit, replaced). At a vector of 1024: + +``` + dispatches = (100e6 / 1024) × 5 = 488,281 indirect calls + cycles = 488,281 × 20 = 9,765,620 cycles + t_dispatch = 9.77e6 / 4.0e9 = 0.0024 s + per-row share of dispatch: 25 ns / 1024 = 0.024 ns +``` + +A thousandfold on that term. That is the entire arithmetic of this topic, +and the reason the rest of it is about what *remains* after the term goes +away. + +Because this repo measures the same shape, check the assumption against +the measurement rather than trusting it. `topics/11-execution-models`'s +provided lane runs 50 M rows through a three-operator Volcano chain +(`experiments/src/volcano.rs`: `Scan`, `FilterOp`, `AggOp`, composed +through `Box` at `:118-120`), and +[notes.md](notes.md)'s baseline table records: + +``` + selectivity time rows/s ns per scanned row (t / 50e6) + 5% 0.386 s 129.4 M/s 7.72 ns + 50% 0.484 s 103.3 M/s 9.68 ns + 95% 0.669 s 74.7 M/s 13.38 ns +``` + +9.68 ns per row for a two-`dyn`-call chain says the 20-cycles-per-call +assumption is the right order of magnitude, not a fantasy. + +And then the surprise, which is [FINDINGS.md](../../FINDINGS.md) row 11: +**the engine gets slower as more rows pass the filter.** Do the marginal +division the table invites: + +``` + extra survivors from 5% to 95%: (0.95 − 0.05) × 50e6 = 45,000,000 rows + extra time: 0.669 − 0.386 = 0.283 s + marginal cost per surviving row: 0.283 / 45e6 = 6.29 ns +``` + +Every row that *survives* costs 6.29 ns more than a row that is rejected +— at the assumed 4 GHz, about 25 cycles. That is the second `dyn` call +(`AggOp` pulling through `FilterOp`, `volcano.rs:63-70`) plus the +aggregate's read-modify-write into `sums[k]` (`:96`), neither of which a +rejected row ever reaches: `FilterOp::next` loops internally on a +rejection and returns nothing. **Evaluating the predicate is the cheap +part; passing it is what costs.** Any account of this model that says +"a selective filter does more work" has it backwards. + +Why it matters: the per-tuple tax is proportional to rows *delivered* +through the operator boundary, not rows examined — so it is worst on +exactly the analytic queries that deliver the most. + +### Step 3 — self-modifying dispatch: the wrapper that swaps itself out + +> **In:** the indirect call of Step 2, line 327 of `executor.h`. +> **Out:** what the function pointer at 327 actually points at, and the +> pattern worth stealing. + +Postgres's node dispatch has a cute optimization. Every node is +*initialized* with its `ExecProcNode` pointer set to a wrapper, and the +node's real method stashed beside it: + +```c +// src/backend/executor/execProcnode.c — ExecSetExecProcNode, 429-440 + 429 void + 430 ExecSetExecProcNode(PlanState *node, ExecProcNodeMtd function) + 431 { + 432 /* + 433 * Add a wrapper around the ExecProcNode callback that checks stack depth + 434 * during the first execution and maybe adds an instrumentation wrapper. + 435 * When the callback is changed after execution has already begun that + 436 * means we'll superfluously execute ExecProcNodeFirst, but that seems ok. + 437 */ + 438 node->ExecProcNodeReal = function; + 439 node->ExecProcNode = ExecProcNodeFirst; + 440 } +``` + +Lines 438-439 are the swap: the real method moves to `ExecProcNodeReal`, +and the pointer the hot path reads gets the wrapper. `ExecInitNode` +(`:141`) routes every node through this on the way out — +`ExecSetExecProcNode(result, result->ExecProcNode)` at `:391`, which is +why "every node starts as `ExecProcNodeFirst`" is true of all node types +without any of them knowing. + +The wrapper's whole job is to run once: + +```c +// src/backend/executor/execProcnode.c — ExecProcNodeFirst, 447-470 + 447 static TupleTableSlot * + 448 ExecProcNodeFirst(PlanState *node) + 449 { + // ... 450-456: comment — the stack check is not cheap on x86, so do it once ... + 457 check_stack_depth(); + 458 + // ... 459-463: comment — swap in a wrapper only if one is still needed ... + 464 if (node->instrument) + 465 node->ExecProcNode = ExecProcNodeInstr; + 466 else + 467 node->ExecProcNode = node->ExecProcNodeReal; + 468 + 469 return node->ExecProcNode(node); + 470 } +``` + +Line **467** is the one to look at: the node's own function-pointer field +is overwritten with the real method, so `executor.h:327` never reaches +this wrapper again. Note that the swap is *conditional* — 464-465 install +`ExecProcNodeInstr` instead when `EXPLAIN ANALYZE` asked for +per-node timing, which is how instrumentation costs nothing when it is +off and is a permanent wrapper when it is on. Line 469 then makes the +call that the first tuple was waiting for. + +Self-modifying dispatch: the first call does setup, then replaces itself. +You have seen the pattern as lazy statics and memoized FFI symbol resolution (question 3 below). -### Step 4 — tuple slots: paying for deforming per access +Why it matters: it removes the one-time checks from the hot path without +adding a branch to it — the check is not skipped, the *pointer* changed. +It does nothing about the indirect call itself, which is Step 2's number +and stays. + +### Step 4 — tuple slots: deforming, and how often it really happens -Tuples travel between operators as `TupleTableSlot` — an abstraction -over heap tuples (the on-disk packed format), minimal tuples, and -virtual tuples (just an array of column pointers). The catch: attribute -access may **deform** the tuple — unpack the packed on-disk bytes to -find column k, which requires walking columns 1..k−1 when earlier -columns are variable-length. Postgres pays this per attribute access, -per tuple; a vectorized engine deforms once per column per 2048-row -chunk and then works on flat arrays. Same work, amortized 2048×. +> **In:** the tuple that `ExecProcNode` returns at `executor.h:327` — a +> `TupleTableSlot *`, not a row of values. +> **Out:** the cost of getting a column out of it, and the correction to +> a claim this guide used to make. + +Tuples travel between operators as a **`TupleTableSlot`** — a container +that can hold a heap tuple (the packed on-disk byte layout), a minimal +tuple, or a **virtual tuple** (no bytes at all, just arrays of `Datum` +values and null flags). To read column *k* out of a packed tuple you must +**deform** it: walk the tuple's bytes computing where each column starts, +because a variable-length or nullable column ahead of *k* means offset +*k* cannot be computed from the schema alone. + +The slot caches that work in `tts_values[]`/`tts_isnull[]` and remembers +how far it got in `tts_nvalid`: + +```c +// src/include/executor/tuptable.h — slot_getattr, 413-428 + 413 /* + 414 * slot_getattr - fetch one attribute of the slot's contents. + 415 */ + 416 static inline Datum + 417 slot_getattr(TupleTableSlot *slot, int attnum, + 418 bool *isnull) + 419 { + 420 Assert(attnum > 0); + 421 + 422 if (attnum > slot->tts_nvalid) + 423 slot_getsomeattrs(slot, attnum); + 424 + 425 *isnull = slot->tts_isnull[attnum - 1]; + 426 + 427 return slot->tts_values[attnum - 1]; + 428 } +``` + +Line **422** is the correction this guide owes you. An earlier version of +this chapter said postgres "pays this per attribute access, per tuple". +It does not: the `attnum > tts_nvalid` test means the deform happens only +when the requested column is past the high-water mark, and every access +below it is two array reads (425, 427). `slot_getsomeattrs` (`:375-381`) +guards the same way, and the deform routine says so itself: + +```c +// src/backend/executor/execTuples.c — the contract of slot_deform_heap_tuple, 1004-1007 + 1004 * This is essentially an incremental version of heap_deform_tuple: + 1005 * on each call we extract attributes up to the one needed, without + 1006 * re-computing information about previously extracted attributes. + 1007 * slot->tts_nvalid is the number of attributes already extracted. +``` + +The expression interpreter pushes it further: it hoists the deform into +one dedicated step per slot, emitted ahead of every `Var` reference, so +the interpreter deforms once and then reads flat arrays (Step 5's +`EEOP_INNER_VAR` says this in its own comment at `execExprInterp.c:693-698`). + +So the honest claim is: **postgres deforms once per tuple per slot, up to +the highest column the plan touches.** That is still a per-tuple cost, and +it is still what the vectorized engines amortise — DuckDB deforms once +per column per 2048-row chunk — but it is a factor of "columns touched" +smaller than the version this guide used to assert. + +Why it matters: the honest number is the one worth beating. Per tuple, +not per access, is what your `vectorized.rs` has to divide by its batch +size. ### Step 5 — expressions as flat steps, dispatched by computed goto -Expressions (`a.x + 1 > b.y`) are the second interpretation layer, and -here postgres already fought back. Instead of walking the expression -*tree* per tuple (recursive calls mirroring the syntax), postgres -compiles each expression once, at plan time, into a linear array of -**steps** — opcodes like "fetch attribute 2", "add", "compare" — then -interprets that flat program per tuple: +> **In:** one deformed tuple in a slot, from Step 4. +> **Out:** the answer to `f < t` for that tuple, produced by the second +> interpretation layer — the one postgres already optimised. + +Expressions (`a.x + 1 > b.y`) are the second interpreter, and here +postgres fought back. Rather than walking the expression *tree* per tuple +(recursion mirroring the syntax, one call per node), postgres compiles +each expression once, at plan time, into a flat array of **steps** — an +**opcode** (a small integer naming an operation) plus its operands, like +"deform up to column 3", "fetch attribute 2", "call `int4gt`", "if false, +bail". Then it interprets that linear program once per tuple. + +The shape, in Rust because the C is macro-heavy: ```rust -// expressions compile to FLAT STEPS, then interpret — once per tuple +// ILLUSTRATION — not quoted from postgres. The real loop is +// src/backend/executor/execExprInterp.c:630-2289, whose dispatch macros +// are quoted below (104-131) and one of whose opcode blocks is quoted at +// 689-704. Expressions compile to FLAT STEPS, then interpret per tuple. fn interp(steps: &[Step], row: &Row, regs: &mut [Datum]) -> Datum { let mut ip = 0; loop { @@ -102,19 +357,174 @@ fn interp(steps: &[Step], row: &Row, regs: &mut [Datum]) -> Datum { // vectorization = the SAME flat steps, applied per 2048 rows instead ``` -Two refinements in the real thing. First, where the compiler supports -it, dispatch is a **computed goto** (each opcode's implementation ends -with `goto *dispatch_table[op->opcode]` rather than looping back to one -central `switch`): every opcode *site* gets its own branch-predictor -entry, which learns "an AddI64 here is usually followed by GtI64" — -where a single switch's one shared indirect branch predicts far worse. -The classic interpreter trick (same reason redis' RESP parsing stays -cheap; it's exactly what a JIT removes entirely — topic 19). Second, a -peephole: step patterns matching common shapes (fetch-inner + -fetch-outer + compare) get dedicated fast-path routines that skip the -interpreter altogether. +The real dispatch is two sets of macros chosen by whether the compiler +supports label-as-values: + +```c +// src/backend/executor/execExprInterp.c — the two dispatch schemes, 104-131 + 104 #if defined(EEO_USE_COMPUTED_GOTO) + // ... 105-117: the jump-target lookup tables ... + 118 + 119 #define EEO_SWITCH() + 120 #define EEO_CASE(name) CASE_##name: + 121 #define EEO_DISPATCH() goto *((void *) op->opcode) + 122 #define EEO_OPCODE(opcode) ((intptr_t) dispatch_table[opcode]) + 123 + 124 #else /* !EEO_USE_COMPUTED_GOTO */ + 125 + 126 #define EEO_SWITCH() starteval: switch ((ExprEvalOp) op->opcode) + 127 #define EEO_CASE(name) case name: + 128 #define EEO_DISPATCH() goto starteval + 129 #define EEO_OPCODE(opcode) (opcode) + 130 + 131 #endif /* EEO_USE_COMPUTED_GOTO */ +``` -### Step 6 — the ladder, and why postgres gets away with it +Line **121** against line **128** is the whole difference. **Switch +threading** (128) returns to one shared `switch` after every step, so the +program has exactly *one* indirect branch, and the predictor has one +history slot to describe every opcode transition in every query. +**Computed goto**, also called **direct threading** (121), ends each +opcode's block with a jump straight to the next opcode's block, so there +are as many indirect-branch *sites* as there are opcode implementations, +each with its own predictor entry that can learn "a `SCAN_VAR` here is +followed by a `FUNCEXPR_STRICT_2`". Postgres's own header comment makes +the claim (`:19-28`): a single dispatch location causes "more jumps and +bad branch prediction". + +Concretely, the dispatch table in `ExecInterpExpr` has **121** entries +and the function contains **123** `EEO_CASE` blocks (counted at this +commit), spread over lines 470-2289 — a single 1820-line C function. So +switch threading gives that whole opcode set one predictor entry; +computed goto gives it 121. Setup happens once per expression, not per +tuple, at `:440-454`, where each step's `opcode` field is overwritten +with the address of its block. + +The cost of getting this wrong is measurable, and this repo measured a +close cousin of it. [FINDINGS.md](../../FINDINGS.md) row 17: branchy +filtering collapses to **0.95 GB/s** at 50% selectivity while a +branchless kernel stays flat at **~10 GB/s**. On the 4-byte elements +that lane uses: + +``` + branchy: 0.95e9 B/s / 4 B = 237.5e6 elem/s → 1 / 237.5e6 = 4.21 ns/elem + branchless: 10.0e9 B/s / 4 B = 2.50e9 elem/s → 1 / 2.50e9 = 0.40 ns/elem + gap: 3.81 ns/elem + at 50% selectivity, ~1 mispredict per 2 elements → ≤ 7.62 ns per mispredict + at the assumed 4 GHz → ≤ 30 cycles +``` + +Read that as an *upper* bound, not a measurement of the misprediction +penalty: the branchless lane also autovectorizes, so part of the 3.81 ns +is SIMD rather than prediction. It is still the right order for what one +unpredictable indirect branch per interpreter step costs, and it is why +121 predictor entries beat 1. + +Why it matters: this is the one layer where postgres is not naive. The +step list *is* the flat program a vectorized engine runs — postgres just +runs it one tuple at a time. + +### Step 6 — the fork: a prepared expression becomes one of two things + +> **In:** the flat step list from Step 5, at the moment +> `ExecReadyInterpretedExpr` finishes building it. +> **Out:** two different callables — a hand-written fast path for simple +> shapes, or the full interpreter — and only the second one pays Step 5's +> dispatch at all. + +Before installing the interpreter, postgres pattern-matches the step list +and, for a handful of shapes, swaps in a dedicated C function that skips +the interpreter entirely. This is a **peephole optimization**: a +transformation that looks at a short window of instructions and replaces +it with something better, without any understanding of the whole program. + +```c +// src/backend/executor/execExprInterp.c — inside ExecReadyInterpretedExpr, 288-308 + 288 /* + 289 * Select fast-path evalfuncs for very simple expressions. "Starting up" + 290 * the full interpreter is a measurable overhead for these, and these + 291 * patterns occur often enough to be worth optimizing. + 292 */ + 293 if (state->steps_len == 5) + 294 { + // ... 295-299: read out steps[0..3]'s opcodes ... + 300 if (step0 == EEOP_INNER_FETCHSOME && + 301 step1 == EEOP_HASHDATUM_SET_INITVAL && + 302 step2 == EEOP_INNER_VAR && + 303 step3 == EEOP_HASHDATUM_NEXT32) + 304 { + 305 state->evalfunc_private = (void *) ExecJustHashInnerVarWithIV; + 306 return; + 307 } + 308 } +``` + +The dispatch key is the *length* of the step list — 5 at 293, 4 at 309, +3 at 337, 2 at 399 — and then the opcode sequence. The simplest branch is +the easiest to read: a three-step program that fetches and returns one +inner column becomes `ExecJustInnerVar`: + +```c +// src/backend/executor/execExprInterp.c — the 3-step patterns, 337-347 + 337 else if (state->steps_len == 3) + 338 { + 339 ExprEvalOp step0 = state->steps[0].opcode; + 340 ExprEvalOp step1 = state->steps[1].opcode; + 341 + 342 if (step0 == EEOP_INNER_FETCHSOME && + 343 step1 == EEOP_INNER_VAR) + 344 { + 345 state->evalfunc_private = ExecJustInnerVar; + 346 return; + 347 } +``` + +Line **345** is the fork: `evalfunc_private` now names a hand-written +function, and this expression will never enter `ExecInterpExpr`. Every +shape that falls through all the tests lands on `:456`, +`state->evalfunc_private = ExecInterpExpr;`, and pays Step 5's dispatch +per tuple forever after. Twenty such fast paths are declared at +`:159-178`, and they are all *projections and hash steps* — fetching a +column, assigning a column, hashing a join key. Not one of them evaluates +a predicate. + +Which tells you where the peephole's authors found the volume: emitting a +column into an output slot is the most common expression in any plan, and +"start the interpreter" was measurable against it. + +Worked, on the query in question 1 — `SELECT sum(x) FROM t WHERE y > 10`. +The `WHERE` clause compiles to five steps, and the reason there are five +and not six is at `execExpr.c:2760-2770`, where a `Const` argument is +written straight into the function's `fcinfo` at *init* time ("Don't +evaluate const arguments every round; especially interesting for +constants in comparisons") and a two-argument strict function gets the +specialised opcode at `:2788-2789`: + +``` + EEOP_SCAN_FETCHSOME deform t up to column y (interp block :662) + EEOP_SCAN_VAR y → the compare's arg 0 (interp block :719) + EEOP_FUNCEXPR_STRICT_2 int4gt(arg0, 10) (interp block :996) + EEOP_QUAL false → bail to the end (interp block :1182) + EEOP_DONE_RETURN return the boolean (interp block :632) +``` + +Five steps, five dispatches, per tuple — none matching a fast-path +pattern, because all 20 of them are projections and hash steps. Add the plan's own +`ExecProcNode` calls (Step 2) and a 3-node plan over 100M rows is +`100e6 × (3 + 5) = 800,000,000` indirect branches for a query whose +useful work is one integer compare and one add per row. + +Why it matters: the fork is the whole argument for compilation in +miniature. Postgres pattern-matched 20 shapes by hand; a JIT (topic 19) +pattern-matches every shape by construction. + +### Step 7 — the ladder, and why postgres gets away with it + +> **In:** both dispatch costs — node-level from Step 2, step-level from +> Steps 5 and 6. +> **Out:** where postgres sits on the interpretation ladder, what it +> already compiles, and the workload boundary where the model stops being +> defensible. Linearizing the expression is *half* of vectorization — postgres just still applies it one tuple at a time: @@ -126,54 +536,238 @@ still applies it one tuple at a time: ↘ JIT (topic 19) compiles the steps ``` -Why it survives: for OLTP, per-tuple overhead × 3 tuples is nothing, and -the buffer manager / WAL / locking dominate writes anyway. For analytics -it does NOT get away with it — that's the market gap DuckDB drove a -truck through. (JIT via LLVM exists for expressions — `jit_above_cost` — -but not for the operator loop.) +Why it survives: for OLTP a point query touches three tuples, so Step 2's +25 ns/row and Step 6's five dispatches are noise beside a buffer-pool +lookup, and writes are dominated by WAL and locking anyway. For analytics +it does not get away with it, and that is the market gap DuckDB drove a +truck through. + +Postgres does JIT, and the scope is narrower than "queries" and wider +than this guide used to say. `src/backend/jit/README:249-251`: +"Currently expression evaluation and tuple deforming are JITed" — so +Step 4's deform and Steps 5-6's step list both get compiled, and the +operator loop of Step 2 does not. The trigger is a cost threshold, not a +row count: `src/backend/jit/jit.c:40` sets `jit_above_cost = 100000`, +with `jit_expressions` and `jit_tuple_deforming` defaulting to true at +`:37` and `:39`. The README's own future list (`:262-263`) names "later +compiling larger parts of queries" as not-yet-done. + +Why it matters: postgres has already conceded Steps 4-6 to compilation +and kept Step 2. Your `vectorized.rs` attacks the opposite half. ## Where each step lives in the code -- **Steps 1–2**: `src/include/executor/executor.h:322` — - `ExecProcNode(node)` is just `return node->ExecProcNode(node);` — the - indirect call per tuple per node. -- **Step 3**: `src/backend/executor/execProcnode.c:439` — nodes - initialized with `ExecProcNode = ExecProcNodeFirst` (`:448`), the - wrapper doing one-time checks (stack depth `:457`, instrumentation) - then replacing the pointer with `ExecProcNodeReal`. -- **Step 4**: `TupleTableSlot` — follow it from any node's - `ExecProcNodeReal`; watch for `slot_getattr` deforming. -- **Step 5**: `src/backend/executor/execExprInterp.c` — read the `:14` - header comment first. Computed-goto dispatch at `:86–:126` - (`EEO_SWITCH`/`EEO_CASE`, `:119–:126`); `ExecInterpExpr` `:146` — the - giant opcode loop itself; the peephole fast paths at `:300`. +Read in this order: the dispatch (small), the slot (small), then the +interpreter (large, and worth an hour on its own). + +| File | Lines | What | Step | +|---|---|---|---| +| `src/include/executor/executor.h` | 314-329 | `ExecProcNode` — the indirect call is 327; `chgParam` recheck at 324 | 1, 2 | +| `src/backend/executor/execProcnode.c` | 141 | `ExecInitNode` — builds the `PlanState` tree | 1 | +| `src/backend/executor/execProcnode.c` | 391 | `ExecSetExecProcNode(result, result->ExecProcNode)` — every node gets the wrapper here | 3 | +| `src/backend/executor/execProcnode.c` | 429-440 | `ExecSetExecProcNode` — real method to `ExecProcNodeReal` (438), wrapper into the hot field (439) | 3 | +| `src/backend/executor/execProcnode.c` | 447-470 | `ExecProcNodeFirst` — `check_stack_depth()` once (457), pointer swap (464-467), then the deferred call (469) | 3 | +| `src/include/executor/tuptable.h` | 375-381 | `slot_getsomeattrs` — deform only past `tts_nvalid` | 4 | +| `src/include/executor/tuptable.h` | 413-428 | `slot_getattr` — the `attnum > tts_nvalid` guard at 422 | 4 | +| `src/backend/executor/execTuples.c` | 995-1108 | `slot_deform_heap_tuple` — the incremental contract is stated at 1004-1007; `tts_nvalid` advanced at 1106-1108 | 4 | +| `src/backend/executor/execExprInterp.c` | 6-46 | the file header — read it first; it argues switch vs direct threading (19-28) and names the fast paths (35-38) | 5, 6 | +| `src/backend/executor/execExprInterp.c` | 104-131 | `EEO_SWITCH`/`EEO_CASE`/`EEO_DISPATCH` — computed goto at 121, the switch fallback at 128 | 5 | +| `src/backend/executor/execExprInterp.c` | 252-457 | `ExecReadyInterpretedExpr` — the peephole (288-438), the direct-threading rewrite (440-454), the fallback at 456 | 6 | +| `src/backend/executor/execExprInterp.c` | 469-2289 | `ExecInterpExpr` — 1820 lines, 121 dispatch-table entries (484+), 123 opcode blocks; loop entry at 626-631 | 5 | +| `src/backend/executor/execExprInterp.c` | 689-704 | `EEOP_INNER_VAR` — reads `tts_values[attnum]` directly, and says why in 693-698 | 4, 5 | +| `src/backend/executor/execExpr.c` | 2754-2790 | const arguments folded into `fcinfo` at init (2760-2770); `EEOP_FUNCEXPR_STRICT_2` chosen at 2788-2789 | 6 | +| `src/backend/jit/jit.c` | 37-40 | `jit_expressions`, `jit_tuple_deforming`, `jit_above_cost = 100000` | 7 | +| `src/backend/jit/README` | 246-263 | what is JITed and what is not | 7 | + +Suggested route: `executor.h:314` → `execProcnode.c:429` and `:447` → +`execExprInterp.c`'s header comment (`:6`) → the macros (`:104`) → +`ExecReadyInterpretedExpr` (`:252`) → then dip into `ExecInterpExpr` +(`:469`) at three or four opcode blocks only. Do not read all 1820 lines. ## Questions for notes.md 1. Count the indirect branches per tuple for `SELECT sum(x) FROM t WHERE y > 10`: plan nodes × 1 + expression - steps. Then per 2048 tuples for the DuckDB equivalent. + steps. Then per 2048 tuples for the DuckDB equivalent. (Step 6 counts + the `WHERE` clause's five steps for you and names the interpreter + block each one lands in — the `sum(x)` transition steps are yours.) 2. Computed goto vs switch: WHY does one predictor entry per opcode site - help? (Think topic 0's branch_misprediction bench.) + help? (Think topic 0's branch_misprediction bench, and the 121 + dispatch-table entries of `ExecInterpExpr`.) 3. `ExecProcNodeFirst`'s pointer swap is bit-smuggling's cousin — self-modifying dispatch. Where else have you seen "first call does setup, then replaces itself"? (Hint: lazy statics, memoized FFI resolution.) 4. M11: your eval.rs will interpret property predicates over batches. - Linear steps or closure tree? What does postgres' `:300` peephole - suggest about the 3 shapes worth special-casing for Cypher + Linear steps or closure tree? All 20 of postgres's fast paths + (`:159-178`) are projections and hash steps, not predicates — what + does that suggest about the 3 Cypher shapes worth special-casing (`n.prop = lit`, `n.prop > lit`, label check)? +## Takeaway + +Two interpretation layers, one already half-fixed. The node layer costs +an indirect call per operator per tuple and postgres has not touched it; +the expression layer costs an opcode per step per tuple and postgres has +flattened it, direct-threaded it, peepholed 21 shapes out of it, and JITs +it above `jit_above_cost`. Vectorization attacks the first layer, which +is the one still standing — and this repo's row 11 says the bill lands on +rows that *pass* the filter, so the tax is worst on exactly the queries +that return the most. + ## Done when -You can explain the two dispatch costs (node-level ExecProcNode, -step-level opcode) and name the mitigation for each (vectorization / -computed goto + JIT). +Answer each before unfolding it. + +- [ ] You can explain the two dispatch costs (node-level `ExecProcNode`, step-level opcode) and name the mitigation for each. + +
Answer + + The node-level cost is one indirect call per plan node per tuple, at + `src/include/executor/executor.h:327` — `return node->ExecProcNode(node);`, + a call through a function-pointer field. A 5-node plan over 100M rows + issues 500M of them; at a stated 20 cycles each on a 4 GHz core that is + 2.5 s, 25 ns/row, before any work. Its mitigation is *not* in postgres: + it is the vector. Handing 1024 rows across the boundary instead of one + turns 500M calls into 488,281 and the per-row share of dispatch from + 25 ns into 0.024 ns. + + The step-level cost is one interpreted opcode per expression step per + tuple: `WHERE y > 10` is five steps (`execExpr.c:2760-2789` folds the + constant in at init, so it is five and not six), each ending in + `EEO_DISPATCH()`. Postgres has three mitigations here, all shipping: + computed-goto dispatch (`execExprInterp.c:121`) giving each of 121 + opcode sites its own predictor entry instead of one shared indirect + branch (`:128`); the 20 hand-written fast paths that skip the + interpreter for simple shapes (`:159-178`, installed at `:288-438`); + and LLVM JIT of expressions and tuple deforming above + `jit_above_cost = 100000` (`jit/jit.c:40`, scope stated in + `jit/README:249-251`). + +
+ +- [ ] You can say why postgres's Volcano executor gets *slower* as a filter passes more rows, and give the marginal cost per surviving row from this repo's own lane. + +
Answer + + Because the per-tuple tax is paid by rows that cross an operator + boundary, not by rows that are examined. In the provided lane's chain + (`experiments/src/volcano.rs`), a rejected row costs one `dyn` call + into `Scan::next` and one predicate compare; `FilterOp::next` (`:63-70`) + loops internally and never returns. A surviving row costs that plus the + return through the second `dyn` call and the aggregate's + read-modify-write of `sums[k]` (`:96`). + + [notes.md](notes.md)'s baseline table, 50 M rows: 0.386 s at 5% + selectivity, 0.484 s at 50%, 0.669 s at 95% — 129.4, 103.3 and + 74.7 M rows/s. The marginal division is + `(0.669 − 0.386) / ((0.95 − 0.05) × 50e6) = 0.283 / 45e6 = 6.29 ns` per + additional surviving row, about 25 cycles at an assumed 4 GHz. That is + [FINDINGS.md](../../FINDINGS.md) row 11, and it inverts the intuition + that a permissive filter is the cheap case. + +
+ +- [ ] You can state how often postgres deforms a tuple, and correct the "once per attribute access" version of the claim. + +
Answer + + Once per tuple per slot, up to the highest column the plan references — + not once per access. `slot_getattr` (`tuptable.h:416-428`) tests + `attnum > slot->tts_nvalid` at 422 and only then calls + `slot_getsomeattrs`; every access at or below the high-water mark is + two array reads, 425 and 427. `slot_deform_heap_tuple` states the + contract in its own comment (`execTuples.c:1004-1007`): "an incremental + version of heap_deform_tuple ... without re-computing information about + previously extracted attributes". + + The expression interpreter tightens it further, hoisting the deform + into a single `EEOP_*_FETCHSOME` step emitted before any `Var` + reference (`execExprInterp.c:644-651`), after which `EEOP_INNER_VAR` + (`:689-704`) reads `innerslot->tts_values[attnum]` with an `Assert` + and no branch — the comment at 693-698 says exactly why. The honest + comparison to a vectorized engine is therefore "once per tuple against + once per column per 2048-row chunk", not "once per access". + +
+ +- [ ] You can explain what direct threading buys over switch threading, in predictor entries, and name the line where postgres chooses. + +
Answer + + `execExprInterp.c:128` is switch threading: `EEO_DISPATCH()` expands to + `goto starteval`, returning to one shared `switch`, so the entire + opcode set is dispatched from a single indirect branch with a single + predictor history. `:121` is direct threading: + `goto *((void *) op->opcode)`, a jump from the *end of each opcode's own + block*, so there are as many indirect-branch sites as opcode + implementations — 121 dispatch-table entries and 123 `EEO_CASE` blocks + in `ExecInterpExpr` at this commit. Each site's predictor entry can + learn its own successor distribution: a `SCAN_VAR` block that is always + followed by `FUNCEXPR_STRICT_2` becomes predictable, where the shared + branch sees every transition in every query mixed together. + + The choice is made at compile time — `#ifdef HAVE_COMPUTED_GOTO` at + `:90-92` sets `EEO_USE_COMPUTED_GOTO`, and the rewrite that replaces + each step's opcode with a label address happens once per expression at + `:440-454`, not per tuple. The header comment argues it at `:19-28`. + For the size of the effect, this repo's row 17 bounds one unpredictable + branch at ≤7.62 ns (≤30 cycles at 4 GHz) from the 0.95 GB/s versus + ~10 GB/s branchy/branchless gap on 4-byte elements — an upper bound, + since the branchless lane also autovectorizes. + +
+ +- [ ] You can say what postgres already JITs and what it does not, without saying "expressions". + +
Answer + + It JITs expression evaluation *and* tuple deforming — + `src/backend/jit/README:249-251` names both, and `jit/jit.c:37-39` + gives each its own GUC (`jit_expressions`, `jit_tuple_deforming`, both + true by default). Deforming is the interesting half: the README's + argument (`:255-257`) is that a JIT knows the number of columns and + their types, so it can emit a straight-line deform with the branches + removed — which is Step 4's cost, compiled away. + + What it does not JIT is the operator loop: `ExecProcNode`'s indirect + call at `executor.h:327` survives compilation, so the per-node, + per-tuple dispatch of Step 2 is unchanged no matter how expensive the + query. The README lists "later compiling larger parts of queries" among + future avenues (`:262-263`). The trigger is a plan-cost threshold, not + a row count — `jit_above_cost = 100000` at `jit/jit.c:40` — which means + a cheap plan over a lot of rows can miss it entirely. + +
## References **Code** -- [postgres](https://github.com/postgres/postgres) — - `src/backend/executor/`: `execProcnode.c` (the dispatch), - `execExprInterp.c` (the computed-goto interpreter — read the :14 - header comment first), plus `src/include/executor/executor.h`; ~1 h +- [postgres](https://github.com/postgres/postgres) — pinned at `701f021` + (`configure.ac:20` says `20devel`). Read + `src/backend/executor/execProcnode.c` (the dispatch, ~970 lines) and + `src/backend/executor/execExprInterp.c` (the interpreter — read the + `:6-46` header comment first, then the macros and + `ExecReadyInterpretedExpr`; do not read all 5990 lines), plus + `src/include/executor/executor.h` and + `src/include/executor/tuptable.h`; ~1 h. + +| File | Lines | What | +|---|---|---| +| `src/include/executor/executor.h` | 327 | the indirect call, once per node per tuple | +| `src/backend/executor/execProcnode.c` | 438-439 | real method aside, wrapper installed | +| `src/backend/executor/execProcnode.c` | 464-467 | the wrapper replacing itself | +| `src/include/executor/tuptable.h` | 422 | the `tts_nvalid` guard that makes deforming incremental | +| `src/backend/executor/execTuples.c` | 1004-1007 | the incremental-deform contract, in postgres's words | +| `src/backend/executor/execExprInterp.c` | 121 / 128 | computed goto against switch | +| `src/backend/executor/execExprInterp.c` | 288-438 | the 20-shape peephole fork | +| `src/backend/executor/execExprInterp.c` | 456 | the fallback: everything else pays the interpreter | +| `src/backend/executor/execExpr.c` | 2760-2770 | constants folded into `fcinfo` at init | +| `src/backend/jit/jit.c` | 40 | `jit_above_cost = 100000` | + +**Background** +- Graefe, *Volcano — An Extensible and Parallel Query Evaluation System* + (TKDE 1994) — the model this executor implements. +- This repo: [FINDINGS.md](../../FINDINGS.md) row 11 (the Volcano ceiling + and its selectivity curve) and row 17 (the branchy/branchless collapse + used to bound a misprediction in Step 5). diff --git a/topics/11-execution-models/reading-rust-execution-stack.md b/topics/11-execution-models/reading-rust-execution-stack.md index 740869f..ea7e9db 100644 --- a/topics/11-execution-models/reading-rust-execution-stack.md +++ b/topics/11-execution-models/reading-rust-execution-stack.md @@ -9,6 +9,12 @@ embody — the batch contract, async scheduling, SIMD kernel shape, static vs dynamic parallelism, and the group-by-as-arrays pattern — then maps each to its file:line. +Anchors are polars at `f8bcc3d` and datafusion at `1e77af8`, the commits +this repo pins (`resources/codebases.md`; confirm with +`tools/pinned-source.py ref polars`). Quoted Rust carries its real line +numbers; elisions are marked. Where a batch size or a default could be +misremembered, the constant is quoted rather than asserted. + ## The problem in one sentence A vectorized engine in Rust must decide four things — what the batch @@ -21,91 +27,538 @@ on almost every axis, which is exactly what makes reading both worth it. ### Step 1 — the batch contract: what travels between operators -Every vectorized engine moves data in **batches** (a fixed-capacity set -of column arrays plus a row count — DuckDB's DataChunk, seen in this -topic's other guides), but the *contract* attached to the batch differs -per system: DuckDB's `DataChunk` is 2048 rows and nothing else; polars' -`Morsel` is a DataFrame plus a sequence number plus a backpressure -token; DataFusion's Arrow `RecordBatch` is ~8K rows with ordering left -to a stream contract. What the batch carries determines what the -scheduler must reconstruct later — ordering, flow control, provenance — -so read each system's batch type first; it's the systems' design in -miniature. +> **In:** three engines that agree on the big decision — move data in +> batches, not tuples — and therefore look interchangeable from a +> distance. +> **Out:** the three *batch types*, read side by side, which turn out to +> disagree about size by a factor of fifty and about what a batch carries +> at all. What the batch carries is what the scheduler does not have to +> reconstruct later. + +A **batch** is a fixed-capacity set of column arrays plus a row count — +DuckDB's `DataChunk`, seen in +[reading-duckdb-execution.md](reading-duckdb-execution.md). All three +systems move batches. They differ on two axes, and both are worth reading +out of the source rather than remembering: + +``` + system batch type default size where the size lives + DuckDB DataChunk 2048 compile-time #define + polars Morsel 100,000 runtime config, env-overridable + DataFusion RecordBatch 8192 runtime config, session-settable +``` + +```rust +// crates/polars-config/src/lib.rs — polars' default, 33-35. + 33 const IDEAL_MORSEL_SIZE: &str = "POLARS_IDEAL_MORSEL_SIZE"; + 34 const STREAMING_CHUNK_SIZE: &str = "POLARS_STREAMING_CHUNK_SIZE"; // Backwards compatibility. + 35 const DEFAULT_IDEAL_MORSEL_SIZE: u64 = 100_000; +``` + +```rust +// datafusion/common/src/config.rs — DataFusion's default, 733. + 733 pub batch_size: ConfigNonZeroUsize, default = non_zero_usize_default(8192) +``` + +Both are *runtime* knobs where DuckDB's is a `#define` +(`vector_size.hpp:16`) — the first real design difference, and it follows +from the second: a compile-time size lets DuckDB size stack buffers and +unroll to it; a runtime one lets polars pick a morsel size from the data. + +Now put the sizes on the same ruler. At eight 8-byte columns, a batch +costs 64 B/row: + +``` + DuckDB 2048 rows × 64 B = 128 KB + DataFusion 8192 rows × 64 B = 512 KB + polars 100000 rows × 64 B = 6.10 MB +``` + +Against this machine's measured ladder +([topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md)): +128 KB is exactly where the L1d plateau ends (1.02 ns), 512 KB reads +5.3-5.8 ns, 4-8 MB reads 7.6-9.0 ns. Only DuckDB's batch is an L1 +residency bet. DataFusion's and polars' are L2 bets — which is coherent, +because they are not the same *kind* of unit. DuckDB's 2048 is a **kernel** +grain: the array a `for` loop runs over. polars' 100,000 is a +**scheduling** grain: the quantum of work a thread claims before going +back for more (see [reading-morsel-parallelism.md](reading-morsel-parallelism.md), +where the paper's own recommendation is checked). Comparing them as if +they were the same number is the classic mistake with this table. ### Step 2 — polars-stream: the morsel as a first-class type -polars-stream (the streaming executor behind `.lazy().collect()`) -promotes the work unit to a type: `Morsel` = `DataFrame` + `MorselSeq` + -`SourceToken`. Each piece answers a question the paper version -(reading-morsel-parallelism.md) left implicit: - -- **`MorselSeq`** — a sequence number: parallel workers pull morsels in - whatever order, and order-sensitive sinks (ORDER BY, LIMIT) reassemble - by seq while order-insensitive ones ignore it. DuckDB keeps this - implicit; polars writes it down. -- **`SourceToken`** — backpressure: a sink can ask sources to stop - producing (topic 7's output-buffer problem, solved politely instead of - by killing clients). -- `get_ideal_morsel_size` is a config knob, not a compile-time constant - — contrast `STANDARD_VECTOR_SIZE = 2048`. - -The physical plan is an explicit `Graph` of nodes connected by pipes; -nodes are **async tasks** (cooperatively-scheduled functions that yield -while waiting) and pipes are channels — pipeline parallelism falls out -of the async runtime rather than a hand-rolled scheduler. What async -buys: blocking sources (network, files) integrate for free. What it -costs: poll overhead and fuzzier buffer ownership (question 1 below). +> **In:** the morsel-driven idea from the SIGMOD'14 paper, where "a +> morsel" is a size and an informal convention, and everything the +> scheduler needs to know about a unit of work is implicit. +> **Out:** a `Morsel` *struct* whose four fields each make one of those +> implicit things explicit — the data, its order, who produced it, and +> when it was consumed — plus the graph and executor that move them. + +polars-stream is the streaming executor behind `.lazy().collect()`, and +it promotes the work unit to a type: + +```rust +// crates/polars-stream/src/morsel.rs — the type, 81-95. + 81 #[derive(Debug, Clone)] + 82 pub struct Morsel { + 83 /// The data contained in this morsel. + 84 df: DataFrame, + 85 + 86 /// The sequence number of this morsel. May only stay equal or increase + 87 /// within a pipeline. + 88 seq: MorselSeq, + 89 + 90 /// A token that indicates which source this morsel originates from. + 91 source_token: SourceToken, + 92 + 93 /// Used to notify someone when this morsel is consumed, to provide backpressure. + 94 consume_token: Option, + 95 } +``` + +Four fields, not the three this guide used to list, and the fourth is the +one that matters most for flow control. Take them in turn. + +**`MorselSeq` — order, written down.** Parallel workers pull morsels and +finish them out of order; an ORDER BY or a LIMIT has to put them back. +DuckDB reconstructs this from batch indices; polars carries it: + +```rust +// crates/polars-stream/src/morsel.rs — the sequence number, 15-21. + 15 /// A token indicating the order of morsels in a stream. + 16 /// + 17 /// The sequence tokens going through a pipe are monotonely non-decreasing and are allowed to be + 18 /// discontinuous. Consequently, `1 -> 1 -> 2` and `1 -> 3 -> 5` are valid streams of sequence + 19 /// tokens. + 20 #[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Debug, Default)] + 21 pub struct MorselSeq(u64); +``` + +Read 17-19 carefully: the guarantee is *monotonely non-decreasing and +allowed to be discontinuous*, so `1 → 3 → 5` is legal. That is weaker +than "consecutive", which is what makes it cheap — a source that splits or +drops morsels does not have to renumber anything. `MorselSeq::new` +(`:26-28`) multiplies by two, reserving the low bit for a future +"last morsel with this sequence number" flag (`:24-25`, `:32-33`) — +topic 2's bit-smuggling again, pre-allocated. + +**`SourceToken` — a stop request, not backpressure.** + +```rust +// crates/polars-stream/src/morsel.rs — the token, 51-57, and what it does, 72-78. + 51 /// A token indicating which source this morsel originated from, and a way to + 52 /// pass information/signals to it. Currently it's only used to request a source + 53 /// to stop with passing new morsels this execution phase. + 54 #[derive(Clone, Debug)] + 55 pub struct SourceToken { + 56 stop: Arc>, + 57 } +// ... 58-71: Default and new() ... + 72 pub fn stop(&self) { + 73 self.stop.store(true); + 74 } + 75 + 76 pub fn stop_requested(&self) -> bool { + 77 self.stop.load() + 78 } +``` + +The doc comment (51-53) is explicit that this is "only used to request a +source to stop", which is the LIMIT case: a sink that has enough rows sets +the flag and the scan notices. **Backpressure** — slowing a fast producer +so a slow consumer's queue stays bounded — is the *other* token, the +`consume_token: Option` on line 94: a producer that wants to +throttle waits on it and is woken when the morsel is actually consumed. +This guide previously attributed backpressure to `SourceToken`; the fields +are separate because the two mechanisms are. + +**The size is a config knob, not a constant** (`:11-13`, resolving to the +`100_000` of Step 1) — contrast DuckDB's `#define`. + +The plan is an explicit graph: + +```rust +// crates/polars-stream/src/graph.rs — the graph, 16-24, and a node, 163-169. + 16 /// Represents the compute graph. + 17 /// + 18 /// The `nodes` perform computation and the `pipes` form the connections between nodes + 19 /// that data is sent through. + 20 #[derive(Default)] + 21 pub struct Graph { + 22 pub nodes: SlotMap, + 23 pub pipes: SlotMap, + 24 } +// ... 25-162: add_node, port wiring, state update ... + 163 /// A node in the graph represents a computation performed on the stream of morsels + 164 /// that flow through it. + 165 pub struct GraphNode { + 166 pub compute: Box, + 167 pub inputs: Vec, + 168 pub outputs: Vec, + 169 } +``` + +Nodes implement `ComputeNode`, and the trait is where polars' answer to +DuckDB's `OperatorResultType` lives — not a return code, but two methods: + +```rust +// crates/polars-stream/src/nodes/mod.rs — the operator contract, 99-114. + 99 /// If this node (in its current state) is a pipeline blocker, and whether + 100 /// this is memory intensive or not. + 101 fn is_memory_intensive_pipeline_blocker(&self) -> bool { + 102 false + 103 } + 104 + 105 /// Spawn the tasks that this compute node needs to receive input(s), + 106 /// process it and send to its output(s). Called once per execution phase. + 107 fn spawn<'env, 's>( + 108 &'env mut self, + 109 scope: &'s TaskScope<'s, 'env>, + 110 recv_ports: &mut [Option>], + 111 send_ports: &mut [Option>], + 112 state: &'s StreamingExecutionState, + 113 join_handles: &mut Vec>>, + 114 ); +``` + +`update_state` (`:92-97`) negotiates port readiness before a phase; +`spawn` (105-114) then creates **async tasks** — cooperatively-scheduled +functions that yield at `.await` points rather than blocking a thread — +one set per execution phase. `is_memory_intensive_pipeline_blocker` +(99-103) is polars' name for DuckDB's pipeline breaker. + +And the execution is *phased*: `execute_graph` (`execute.rs:301`) loops — +update all port states, find a runnable subgraph +(`find_runnable_subgraph`, `:106`), run it, repeat until nothing is +runnable (`:328-360`). The graph is not run once; it is run in waves, +which is how a breaker's downstream gets to start only after it finishes. + +**One correction worth making loudly.** It is tempting to say pipeline +parallelism here "falls out of the async runtime rather than a hand-rolled +scheduler". It does not. polars wrote its own work-stealing executor: + +```rust +// crates/polars-async/src/executor/mod.rs — the scheduler polars wrote, 236-240. + 236 fn try_steal_task(&self, thread: usize, rng: &mut R) -> Option { + 237 // Try to get a global task. + 238 loop { + 239 match self.global_high_prio_task_queue.steal() { + 240 Steal::Empty => break, +``` + +`try_steal_task` (236) drains a global high-priority queue, then a +low-priority one, then steals from a randomly chosen sibling thread +(`:254-265`), then parks with one last steal attempt before sleeping +(`:319-321`). tokio appears in `polars-stream` only with the `sync` +feature (`Cargo.toml:32`) and as the runtime for blocking I/O +(`ASYNC.block_in_place_on`, `execute.rs:339`). So `async` here buys the +*task representation* — a suspended operator is a state machine the +compiler wrote — while the scheduling is as hand-rolled as DuckDB's. +What async genuinely buys is that a blocking source integrates for free; +what it costs is poll overhead and fuzzier buffer ownership (question 1). ### Step 3 — what a SIMD kernel actually looks like -Down at the leaf, a kernel is a loop that must survive three hazards: -nulls, selectivity, and the compiler failing to vectorize. polars-compute's -float sum shows the production answers: - -- **Masked variants**: every kernel comes in a pair — - `sum_block_vectorized` and `sum_block_vectorized_with_mask` — because - columns have null masks (a bitmask marking missing values). The masked - sum SELECTS values into SIMD lanes (blend the value or 0.0 per lane) - rather than branching per element — no branch misprediction at 50% - nulls. This masked/unmasked pairing is the columnar equivalent of - selection vectors. -- **Multiple independent accumulators**: fixed-size blocks accumulated - into several SIMD registers in parallel, reduced once at the end. - One accumulator would serialize on the ~4-cycle add latency; 4–8 - independent ones keep the arithmetic ports full — topic 0's MLP - lesson applied to arithmetic instead of memory. -- `vector_horizontal_sum` — the final reduce of one SIMD register to a - scalar, shaped "to map to good shuffle instructions". -- The fine print: float addition isn't associative, so the vectorized - sum ≠ the sequential sum bit-for-bit. Engines document this away. +> **In:** a column of floats with a null mask, and an ambition to add +> them up at memory speed. +> **Out:** the three things a production kernel does about it — SIMD +> lanes instead of a scalar loop, a select instead of a branch for nulls, +> and a recursion that both breaks the dependency chain and bounds the +> floating-point error. + +polars-compute's float sum is small enough to read whole and shows all +three. Two constants set the shape: + +```rust +// crates/polars-compute/src/float_sum.rs — the two shape constants, 13-14. + 13 const STRIPE: usize = 16; + 14 const PAIRWISE_RECURSION_LIMIT: usize = 128; +``` + +**Masked variants, not branches.** Every kernel comes in a pair, because +a column carries a **null mask** — a bitmask marking which values are +missing: + +```rust +// crates/polars-compute/src/float_sum.rs — the pair, 65-69, and the masked +// implementation, 87-98. + 65 // As a trait to not proliferate SIMD bounds. + 66 pub trait SumBlock { + 67 fn sum_block_vectorized(&self) -> F; + 68 fn sum_block_vectorized_with_mask(&self, mask: BitMask<'_>) -> F; + 69 } +// ... 70-86: the impl header and the unmasked variant ... + 87 fn sum_block_vectorized_with_mask(&self, mask: BitMask<'_>) -> F { + 88 let zero = Simd::default(); + 89 let vsum = self + 90 .chunks_exact(STRIPE) + 91 .enumerate() + 92 .map(|(i, a)| { + 93 let m: Mask = mask.get_simd(i * STRIPE); + 94 m.select(Simd::from_slice(a).cast_generic::(), zero) + 95 }) + 96 .sum::>(); + 97 vector_horizontal_sum(vsum) + 98 } +``` + +Line 94 is the whole trick: `m.select(values, zero)` blends per lane — +a null contributes `0.0` and the sum is unchanged — so there is no branch +for the predictor to miss when nulls are scattered. +[FINDINGS.md](../../FINDINGS.md) row 17 puts the scale on what that +avoids: 0.95 GB/s branchy against ~10 GB/s branchless on the same data. +This masked/unmasked pairing is the columnar counterpart of DuckDB's +selection vectors — same problem, different representation. + +**Lanes, and the dependency chain they leave behind.** Work the block out: + +``` + block = PAIRWISE_RECURSION_LIMIT = 128 elements + lanes = STRIPE = 16 + chunks per block = 128 / 16 = 8 SIMD adds + those 8 adds are a *dependent* chain into one accumulator (83, 96) + at an assumed 3-4 cycle FP-add latency: 24-32 cycles per 128 elements + = 4.0-5.3 elements/cycle + a scalar loop, same latency, chain of 128: 0.25-0.33 el/cycle +``` + +So the 16 lanes are not worth 16× on their own — a single accumulator +turns them into roughly 4-5 elements per cycle, because each add waits for +the previous one. The independence has to come from somewhere else, and it +comes from the recursion: + +```rust +// crates/polars-compute/src/float_sum.rs — the recursion, 205-210. + 205 unsafe { + 206 let blocks = f.len() / PAIRWISE_RECURSION_LIMIT; + 207 let left_len = (blocks / 2) * PAIRWISE_RECURSION_LIMIT; + 208 let (left, right) = (f.get_unchecked(..left_len), f.get_unchecked(left_len..)); + 209 pairwise_sum(left) + pairwise_sum(right) + 210 } +``` + +`pairwise_sum` (`:189`) splits the slice in half down to 128-element +blocks and adds the halves (209). The two subtree sums are independent, so +the machine can have several accumulator chains in flight at once — this +is topic 0's memory-level-parallelism lesson applied to arithmetic ports. + +But note *why* the code is shaped that way, because the guide used to get +this backwards: pairwise summation is a **numerical** technique first. Its +error bound grows as O(log n) where a naive running sum grows as O(n); the +instruction-level parallelism is a side effect the author got for free. +The same honesty applies to the reduce: + +```rust +// crates/polars-compute/src/float_sum.rs — the final reduce, 44-63. + 44 fn vector_horizontal_sum(mut v: V) -> T + 45 where + 46 V: IndexMut, + 47 T: Add + Sized + Copy, + 48 { + 49 // We have to be careful about this reduction, floating + 50 // point math is NOT associative so we have to write this + 51 // in a form that maps to good shuffle instructions. + 52 // We fold the vector onto itself, halved, until we are down to + 53 // four elements which we add in a shuffle-friendly way. + 54 let mut width = STRIPE; + 55 while width > 4 { + 56 for j in 0..width / 2 { + 57 v[j] = v[j] + v[width / 2 + j]; + 58 } + 59 width /= 2; + 60 } + 61 + 62 (v[0] + v[2]) + (v[1] + v[3]) + 63 } +``` + +The comment at 49-53 says it outright: float addition is not associative, +so the fold order is chosen to "map to good shuffle instructions" *and* +to be a defensible order. The consequence is one every engine documents +away — a vectorized sum does not equal the sequential sum bit for bit. ### Step 4 — DataFusion: Volcano's shape, async clothes, static partitions -DataFusion keeps the iterator model's *shape* and changes the unit: the -`ExecutionPlan` trait's `execute(partition, ctx)` returns a -`SendableRecordBatchStream` — that's `open()` returning a stream, and -the stream's `poll_next` is `next()`. Volcano survived; what changed is -the payload (Arrow `RecordBatch`, ~8K rows) and the dispatch (async -poll, amortized over the batch, so the per-call cost stops mattering — -one poll per 8K rows is 0.01 ns/row even at 100 ns/poll). +> **In:** the iterator model, which DataFusion had no reason to abandon — +> its shape is a good fit for a composable plan. +> **Out:** the same shape with two substitutions: the payload becomes an +> Arrow `RecordBatch` (so `next()` is amortized), and `next()` becomes an +> async `poll_next` (so a blocking source does not block a thread). The +> parallelism, though, is decided statically, which is the axis on which +> DataFusion differs from both others. -Parallelism is **partition-per-stream**: `execute(i)` for i in 0..N -spawns one task per partition — STATIC partitioning, the very thing -morsel-driven scheduling exists to avoid. Skew hurts more than in -DuckDB/polars-stream; `RepartitionExec` operators patch it up mid-plan -by reshuffling batches across partitions. +```rust +// datafusion/physical-plan/src/execution_plan.rs — the trait, 97, and the +// one method that is the whole model, 478-482. + 97 pub trait ExecutionPlan: Any + Debug + DisplayAs + Send + Sync { +// ... 98-477: name, properties, children, doc examples ... + 478 fn execute( + 479 &self, + 480 partition: usize, + 481 context: Arc, + 482 ) -> Result; +``` + +Read the signature as Volcano with two words changed: `execute` is +`open()`, and the returned stream's `poll_next` is `next()`. What changed +is the payload — a `RecordBatch` of 8192 rows by default (Step 1) — and +so the per-call cost is amortized over 8192 rows exactly as DuckDB's is +over 2048: + +``` + one poll per 8192 rows, at an assumed 100 ns per poll + (async wake, stream plumbing, an Arrow batch handoff) + per-row share = 100 ns / 8192 = 0.012 ns/row + compare tuple-at-a-time dispatch, from the postgres guide: 25 ns/row +``` + +Same conclusion as everywhere else in this topic: batching kills the +dispatch term, whatever the dispatch mechanism happens to be. This is why +"is it async?" is the wrong question to argue about. + +The interesting difference is the first argument. `execute(partition, ctx)` +is **static partitioning**: the plan declares N output partitions, the +runtime calls `execute(0..N)` and spawns one task each, and a task owns its +partition to the end. That is the thing morsel-driven scheduling exists to +avoid. N defaults to the machine's parallelism +(`target_partitions`, `datafusion/common/src/config.rs:768`), and skew is +patched *inside* the plan by `RepartitionExec` +(`datafusion/physical-plan/src/repartition/mod.rs:1150`), inserted by the +optimizer under the `repartition_joins` / `repartition_aggregations` flags +(`config.rs:1443`, `:1455`, both default `true`). + +Worked, so the cost of static is concrete. Take the 50 M-row lane and +eight workers: + +``` + static, balanced: 8 partitions × 6.25 M rows → wall clock = 6.25 M rows + static, one partition holds 3× the mean: + that partition has 18.75 M rows → wall clock = 18.75 M = 3.0× worse + morsel-driven, DuckDB's 122,880-row grain (see the DuckDB guide): + 408 units over 8 threads = 51 each; a straggler costs at most + one unfinished morsel = 122,880 rows = 0.25% of a thread's share +``` + +Static partitioning is not a mistake — it removes a scheduler, and a +`RepartitionExec` fixes the common cases — but its worst case is a +multiple, and morsel-driven's is a rounding error. ### Step 5 — group-by as array arithmetic: intern, then index flat states -The heart of any aggregation engine, and DataFusion's -`GroupedHashAggregateStream` is the cleanest statement of the modern -shape. Per input batch: **intern** the group keys — one hash-table probe -per row that maps each key to a dense integer group index (0, 1, 2, … -in first-seen order) — then update aggregate states that live in flat -columnar arrays indexed by that integer: +> **In:** a `GROUP BY` with several aggregates, and the obvious +> implementation — a hash map from key to a little struct of running +> totals, probed once per aggregate per row. +> **Out:** DataFusion's shape, which probes once per *row* no matter how +> many aggregates there are, and keeps every aggregate's state in a flat +> `Vec` indexed by a dense integer. + +The move is **interning**: map each group key to a dense integer group id +— 0, 1, 2, … in first-seen order — and then never touch the key again. + +```rust +// datafusion/physical-plan/src/aggregates/group_values/mod.rs — the contract, +// 85-100. + 85 /// # Group Ids + 86 /// + 87 /// Each distinct group in a hash aggregation is identified by a unique group id + 88 /// (usize) which is assigned by instances of this trait. Group ids are + 89 /// continuous without gaps, starting from 0. + 90 pub trait GroupValues: Send { + 91 /// Calculates the group id for each input row of `cols`, assigning new + 92 /// group ids as necessary. + 93 /// + 94 /// When the function returns, `groups` must contain the group id for each + 95 /// row in `cols`. + 96 /// + 97 /// If a row has the same value as a previous row, the same group id is + 98 /// assigned. If a row has a new value, the next available group id is + 99 /// assigned. + 100 fn intern(&mut self, cols: &[ArrayRef], groups: &mut Vec) -> Result<()>; +``` + +"Continuous without gaps, starting from 0" (88-89) is the load-bearing +sentence: it is what lets a group id be an *array index* rather than a map +key. The per-batch loop then interns once and hands the same +`group_indices` to every accumulator: + +```rust +// datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs — inside +// group_aggregate_batch (845): intern once, 884-888 … + 884 // calculate the group indices for each input row + 885 let starting_num_groups = self.group_values.len(); + 886 self.group_values + 887 .intern(group_values, &mut self.current_group_indices)?; + 888 let group_indices = &self.current_group_indices; +// ... 889-912: ordering bookkeeping and metrics ... + 913 for ((acc, values), opt_filter) in t { + 914 let opt_filter = opt_filter.as_ref().map(|filter| filter.as_boolean()); + 915 + 916 // Call the appropriate method on each aggregator with + 917 // the entire input row and the relevant group indexes + 918 if self.mode.input_mode() == AggregateInputMode::Raw + 919 && !self.spill_state.is_stream_merging + 920 { + 921 acc.update_batch( + 922 values, + 923 group_indices, + 924 opt_filter, + 925 total_num_groups, + 926 )?; +``` + +And the accumulator's state really is a flat array: + +```rust +// datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs +// — the state, 46-47, and the update, 99-113. + 46 /// values per group, stored as the native type + 47 values: Vec, +// ... 48-98: the other fields, constructors, and the update_batch header ... + 99 // update values + 100 self.values.resize(total_num_groups, self.starting_value); + 101 + 102 // NullState dispatches / handles tracking nulls and groups that saw no values + 103 self.null_state.accumulate( + 104 group_indices, + 105 values, + 106 opt_filter, + 107 total_num_groups, + 108 |group_index, new_value| { + 109 // SAFETY: group_index is guaranteed to be in bounds + 110 let value = unsafe { self.values.get_unchecked_mut(group_index) }; + 111 (self.prim_fn)(value, new_value); + 112 }, + 113 ); +``` + +`values: Vec` (47), grown to `total_num_groups` (100), written +through a raw index (110-111). No per-group heap object exists to chase, +so the aggregate's state is as SIMD-friendly and prefetchable as the input. + +Worked, for the second win — sharing the probe. Take 50 M rows, four +aggregates, and a group count large enough that the group table does not +fit in cache, so each probe costs about one miss at the ~25 ns this +machine measures for its DRAM plateau (topic 0 notes): + +``` + probe per aggregate: 50e6 × 4 = 200,000,000 probes × 25 ns = 5.00 s + intern once per row: 50e6 × 1 = 50,000,000 probes × 25 ns = 1.25 s + saved = 3.75 s +``` + +That is an *upper* bound — real probes hit in cache some of the time, and +the interned path still pays four sequential array writes per row — but +the shape of the saving is right, and it grows linearly with the number of +aggregates, which is why the pattern is universal. + +In the shape M11 will want it: ```rust -// group-by IS array arithmetic: intern keys → dense ids → flat states +// ILLUSTRATION — not quoted from datafusion. The real loop is +// grouped_hash_stream.rs:884-926 (intern, then one update_batch per +// accumulator) and prim_op.rs:99-113 (the flat-array update). This +// collapses both into one function to show the data flow. fn update_batch(&mut self, keys: &Column, vals: &[i64]) { let gids = self.group_values.intern(keys); // ONE HT probe per row, for (i, &g) in gids.iter().enumerate() { // shared by all aggregates @@ -115,57 +568,86 @@ fn update_batch(&mut self, keys: &Column, vals: &[i64]) { } ``` -Two wins: 4 aggregates share ONE probe per row instead of probing 4 -times (question 4), and states are dense arrays — cache-friendly, -SIMD-able, no per-group heap objects to chase. This is exactly the shape -your `vectorized.rs` group-by should have. - ### Step 6 — the comparison that matters +> **In:** five design decisions read one system at a time, which is how +> you learn them and not how you use them. +> **Out:** the three systems on one page, with the axis each one bet +> differently on — and, for each row, the cost that came with the win, so +> that M11's fourth column is chosen rather than copied. + | | DuckDB | polars-stream | DataFusion | |---|---|---|---| -| unit | DataChunk 2048 | Morsel (config) | RecordBatch ~8K | -| parallelism | morsel pull | async graph + tokens | static partitions | -| scheduling | own scheduler | async runtime | tokio | -| ordering | implicit | MorselSeq | stream contract | +| batch type | `DataChunk` | `Morsel` | Arrow `RecordBatch` | +| default size | 2048, compile-time | 100,000, config | 8192, config | +| parallelism | morsel pull | morsel pull, async graph | static partitions + `RepartitionExec` | +| scheduling | own task scheduler | own work-stealing executor over async tasks | tokio | +| ordering | implicit (batch index) | explicit `MorselSeq` | stream contract | +| flow control | `OperatorResultType` return codes | `SourceToken` (stop) + `WaitToken` (backpressure) | async backpressure via `poll_next` | +| operator state | executor-owned chunks | node-owned, per phase | stream-owned | -No row of this table has a free winner: morsel pulling beats static -partitions on skew but demands your own scheduler; async graphs -integrate blocking sources but give up precise control of buffers; an -explicit `MorselSeq` costs a u64 per batch and buys ordered sinks. M11 -must fill in a fourth column — that's the point of reading all three. +No row has a free winner. Morsel pulling beats static partitions on skew +(Step 4's 3.0× against 0.25%) but you must write the scheduler — polars +and DuckDB both did. Async tasks integrate blocking sources for free and +cost poll overhead plus fuzzier buffer ownership. An explicit `MorselSeq` +costs eight bytes per batch and buys ordered sinks without a global sort. +M11 has to fill in a fourth column, which is the point of reading all +three. ## Where each step lives in the code -- **Steps 1–2 — polars-stream** (`crates/polars-stream/src/`): - `morsel.rs:82` — `Morsel`; `MorselSeq` at `:21`; - `get_ideal_morsel_size` at `:11`. The graph: `graph.rs:21,:165` - (`Graph`, `GraphNode`), pipes in `pipe.rs`, `execute.rs:301` - `execute_graph` drives it. Skim `nodes/` for the operator - implementations. -- **Step 3 — polars-compute** - (`crates/polars-compute/src/float_sum.rs`): `:44` - `vector_horizontal_sum`; `:67` `SumBlock` trait with - `sum_block_vectorized` + `sum_block_vectorized_with_mask` — the mask - is a `BitMask`, selected into lanes, not branched. -- **Step 4 — DataFusion** - (`datafusion/physical-plan/src/execution_plan.rs`): `trait - ExecutionPlan` :97; `execute(partition, ctx) -> - SendableRecordBatchStream` (:478). `RepartitionExec` for the - skew patch. -- **Step 5 — DataFusion aggregates** - (`aggregates/grouped_hash_stream.rs:275`) — - `GroupedHashAggregateStream`; `poll_next` (`:641`) pulls input - batches; key interning via `group_values/mod.rs:90` (`trait - GroupValues`); states updated with vectorized - `update_batch(values, group_indices)`. +Read polars first (the morsel type is the topic's vocabulary made +concrete), then DataFusion's aggregate (the pattern you will copy). +Anchors are polars `f8bcc3d`, datafusion `1e77af8`. + +| File | Lines | What is there | Step | +|---|---|---|---| +| `crates/polars-config/src/lib.rs` | 33-35 | `DEFAULT_IDEAL_MORSEL_SIZE = 100_000`, and its env var | 1 | +| `datafusion/common/src/config.rs` | 733 | `batch_size` default 8192 | 1 | +| `crates/polars-stream/src/morsel.rs` | 11-13 | `get_ideal_morsel_size` — a config read, not a constant | 1, 2 | +| `crates/polars-stream/src/morsel.rs` | 15-21, 24-35 | `MorselSeq`, non-decreasing and discontinuous; the reserved low bit | 2 | +| `crates/polars-stream/src/morsel.rs` | 51-57, 72-78 | `SourceToken` — a *stop* request, not backpressure | 2 | +| `crates/polars-stream/src/morsel.rs` | 81-95 | `Morsel` — four fields; `consume_token` (94) is the backpressure one | 2 | +| `crates/polars-stream/src/graph.rs` | 16-24, 163-169, 171-185 | `Graph`, `GraphNode`, `LogicalPipe` | 2 | +| `crates/polars-stream/src/nodes/mod.rs` | 92-97, 99-114 | `ComputeNode::update_state` and `spawn` — the operator contract | 2 | +| `crates/polars-stream/src/execute.rs` | 106, 301, 328-360 | `find_runnable_subgraph`; `execute_graph`; the phase loop | 2 | +| `crates/polars-async/src/executor/mod.rs` | 236-265, 313-321 | the work-stealing scheduler polars wrote itself | 2 | +| `crates/polars-compute/src/float_sum.rs` | 13-14 | `STRIPE = 16`, `PAIRWISE_RECURSION_LIMIT = 128` | 3 | +| `crates/polars-compute/src/float_sum.rs` | 44-63 | `vector_horizontal_sum` — and why the fold order is chosen | 3 | +| `crates/polars-compute/src/float_sum.rs` | 65-69, 87-98 | `SumBlock`; the masked variant selects into lanes (94) | 3 | +| `crates/polars-compute/src/float_sum.rs` | 189-211 | `pairwise_sum` — accuracy first, ILP as a bonus | 3 | +| `datafusion/physical-plan/src/execution_plan.rs` | 97, 478-482 | `trait ExecutionPlan`; `execute(partition, ctx)` | 4 | +| `datafusion/common/src/config.rs` | 768, 1443, 1455 | `target_partitions`; the repartition flags | 4 | +| `datafusion/physical-plan/src/repartition/mod.rs` | 1150 | `RepartitionExec` — the skew patch | 4 | +| `.../aggregates/group_values/mod.rs` | 85-100 | `GroupValues::intern`; ids dense from 0 | 5 | +| `.../aggregates/grouped_hash_stream.rs` | 275, 641, 845, 884-926 | the stream; `poll_next`; `group_aggregate_batch`; intern once, then every accumulator | 5 | +| `.../groups_accumulator/prim_op.rs` | 46-47, 99-113 | `values: Vec` — flat state indexed by group id | 5 | + +## Takeaway + +The three engines agree completely on the decision that matters — batch, +don't tuple — and the arithmetic shows why they can afford to disagree +about everything else: at 2048, 8192 or 100,000 rows per call the dispatch +term is 0.01 ns/row and no longer participates in the argument. What is +left to design is what the batch *carries* (order, stop signals, +backpressure), who hands batches to whom (own scheduler vs static +partitions), and what the innermost loop looks like (lanes, selects, and +a recursion that is really about float error). + +Two of this guide's earlier claims did not survive reading the code: +polars-stream does *not* get its parallelism from an off-the-shelf async +runtime — it ships its own work-stealing executor — and `float_sum`'s +independent accumulators come from a *pairwise* recursion whose first +purpose is numerical accuracy. Both are the kind of thing that reads +plausibly and is wrong, which is the argument for the file:line habit. ## Questions for notes.md 1. Async operators (polars/DF) vs hand-rolled state machines (DuckDB's OperatorResultType): what does async buy (blocking sources) and cost (poll overhead, buffer ownership)? Which fits M11 — remember topic 7's - one-threadpool decision. + one-threadpool decision. Note that polars pays for *both*: async tasks + **and** its own scheduler. 2. MorselSeq: which graph query results are order-sensitive? (ORDER BY obviously — anything else in Cypher? LIMIT without ORDER BY?) 3. The masked-kernel pattern: your batches will have selection vectors @@ -180,18 +662,129 @@ must fill in a fourth column — that's the point of reading all three. ## Done when -You can name the batch unit + parallelism strategy of all three systems -from the table WITHOUT the table, and describe the -intern-then-flat-arrays group-by shape in two sentences. +Answer each before unfolding it. + +- [ ] You can name the batch unit and parallelism strategy of all three systems without the table, and say why comparing 2048 to 100,000 directly is a category error. + +
Answer + + DuckDB: `DataChunk`, 2048 rows fixed at compile time + (`vector_size.hpp:16`), morsel-pull parallelism with its own scheduler. + polars-stream: `Morsel`, default 100,000 rows from config + (`polars-config/src/lib.rs:35`), morsel-pull over an async graph on its + own work-stealing executor. DataFusion: Arrow `RecordBatch`, default + 8192 (`config.rs:733`), static partition-per-stream with + `RepartitionExec` for skew. + + The category error is that 2048 is a *kernel* grain — the array length a + `for` loop runs over, chosen so eight 8-byte columns are 128 KB and stay + in L1 between operators — while 100,000 is a *scheduling* grain, the + quantum a thread claims before going back for more work. They are + answers to different questions; only DataFusion's 8192 is directly + comparable to 2048. + +
+ +- [ ] You can say what each of `Morsel`'s four fields is for, and which one actually implements backpressure. + +
Answer + + From `morsel.rs:82-95`: `df` is the data; `seq: MorselSeq` is the + ordering token, guaranteed monotonely non-decreasing but explicitly + allowed to be discontinuous (15-19), so order-sensitive sinks can + reassemble and order-insensitive ones can ignore it; `source_token: + SourceToken` identifies the producing source and carries a *stop* + request (51-53, 72-78) — the LIMIT case; and `consume_token: + Option` (93-94) is the backpressure mechanism, notifying a + waiting producer only once the morsel has actually been consumed. + + Stop and backpressure are different fields because they are different + mechanisms: one ends production, the other paces it. + +
+ +- [ ] You can explain why a masked SIMD kernel selects rather than branches, and why 16 lanes do not give 16×. + +
Answer + + It selects because a branch on a scattered null mask is unpredictable: + `m.select(values, zero)` (`float_sum.rs:94`) makes a null contribute + `0.0` with no control flow at all. FINDINGS row 17 measures the + alternative at 0.95 GB/s branchy against ~10 GB/s branchless. + + 16 lanes do not give 16× because a 128-element block + (`PAIRWISE_RECURSION_LIMIT`) is 8 chunks of `STRIPE = 16` summed into + *one* accumulator (83, 96) — a dependent chain of 8 FP adds. At 3-4 + cycles of add latency that is 24-32 cycles per 128 elements, about 4-5 + elements per cycle. The independent chains come from `pairwise_sum` + (189-211) splitting the input recursively, and that recursion exists + primarily to bound floating-point error at O(log n) rather than O(n) — + the instruction-level parallelism is a bonus. + +
+ +- [ ] You can describe the intern-then-flat-arrays group-by in two sentences, and count the probes it saves. + +
Answer + + Per input batch, DataFusion interns the group keys once — one hash probe + per row mapping each key to a dense group id, contiguous from 0 + (`group_values/mod.rs:88-100`) — and then hands the same + `group_indices` slice to every accumulator + (`grouped_hash_stream.rs:884-926`). Each accumulator's state is a flat + `Vec` indexed by that id (`prim_op.rs:47`, written at 110-111), so there + is no per-group heap object and the update is array arithmetic. + + With four aggregates over 50 M rows, probing per aggregate is 200 M + probes against 50 M — and if the group table exceeds cache so each probe + costs about one ~25 ns miss, that is 5.00 s against 1.25 s, an upper + bound of 3.75 s saved. The saving scales with the number of aggregates, + which is why the pattern is universal. + +
+ +- [ ] You can state static partitioning's worst case against morsel pulling's, on numbers. + +
Answer + + DataFusion's `execute(partition, ctx)` + (`execution_plan.rs:478-482`) gives a task one partition for the whole + query, with N defaulting to the machine's parallelism + (`config.rs:768`). If the work splits evenly across 8 partitions, 50 M + rows is 6.25 M each. If one partition holds 3× the mean it holds + 18.75 M, and since the query ends when the slowest partition does, the + wall clock is 3.0× the balanced case. + + Morsel pulling bounds the same imbalance by one morsel. At DuckDB's + 122,880-row grain the 50 M-row lane is 408 units, 51 per thread, so a + straggler costs at most one unfinished morsel — 0.25% of a thread's + share. DataFusion patches the gap inside the plan instead, with + `RepartitionExec` (`repartition/mod.rs:1150`) inserted under the + `repartition_joins` / `repartition_aggregations` flags (`config.rs:1443`, + `:1455`). + +
## References **Code** -- [polars](https://github.com/pola-rs/polars) — - `crates/polars-stream/src/` (`morsel.rs`, `graph.rs`, `execute.rs`, - `nodes/`) and `crates/polars-compute/src/float_sum.rs` for what a - SIMD kernel actually looks like -- [datafusion](https://github.com/apache/datafusion) — +- [polars](https://github.com/pola-rs/polars) at `f8bcc3d` — + `crates/polars-stream/src/` (`morsel.rs`, `graph.rs`, `nodes/mod.rs`, + `execute.rs`), `crates/polars-async/src/executor/mod.rs` for the + scheduler, and `crates/polars-compute/src/float_sum.rs` for what a SIMD + kernel actually looks like +- [datafusion](https://github.com/apache/datafusion) at `1e77af8` — `datafusion/physical-plan/src/execution_plan.rs` (the trait) and - `aggregates/` (`GroupedHashAggregateStream`, `group_values/`) — the + `aggregates/` (`grouped_hash_stream.rs`, `group_values/`) — the engine's heart; ~1.5 h for both + +**In this repo** +- [reading-duckdb-execution.md](reading-duckdb-execution.md) — the C++ + system these two are answering, and where 2048 comes from +- [reading-morsel-parallelism.md](reading-morsel-parallelism.md) — the + paper polars' `Morsel` type is a transcription of +- [FINDINGS.md](../../FINDINGS.md) row 17 — branchy vs branchless + throughput, the number behind Step 3's select-don't-branch +- [topics/00-performance-toolbox/notes.md](../00-performance-toolbox/notes.md) + — the measured cache and latency ladder every size argument is checked + against diff --git a/topics/11-execution-models/reading-x100.md b/topics/11-execution-models/reading-x100.md index 2092add..4432a97 100644 --- a/topics/11-execution-models/reading-x100.md +++ b/topics/11-execution-models/reading-x100.md @@ -8,167 +8,573 @@ step by step: the profile that started it, the two failure modes it threads between, the vector, the primitive, and the health metric — then routes you through the sections. +Every number below is checked against the paper — Boncz, Zukowski, Nes, +*MonetDB/X100: Hyper-Pipelining Query Execution*, CIDR 2005 — and cited +to the section, table or figure it came from. Several figures this guide +used to carry did not survive that check; the corrections are called out +where they occur, because being wrong in a memorable way is how these +numbers propagate. + ## The problem in one sentence In 2005 a database evaluating TPC-H Q1 — a plain scan + filter + -arithmetic + group-by, no join — ran **45× slower than a hand-written C -loop over the same data**, and ~90% of that time was interpretation -overhead, not computation. +arithmetic + group-by, no join — ran **121× slower than a hand-written C +loop over the same data** (Table 1: MySQL 4.1 at 26.6 s against +hand-coded at 0.22 s, same AthlonMP), and only 10% of that time was +computation: §3.1 finds the five operations doing the actual work account +for 10% of execution time, the rest being record navigation and +hash-table machinery. ## The concepts, step by step ### Step 1 — the profile: databases ran below 10% of the hardware -The paper opens with measurement, not design. Profiling TPC-H Q1 on -MySQL shows ~90% of time in interpretation overhead: per-tuple function -calls, attribute extraction, expression-tree walking — the machinery of -deciding what to do, not doing it. The health metric they use is **IPC** -(instructions per cycle — how many instructions the core actually -retires per clock; a superscalar core of that era could sustain 3+): -MySQL ran at ~0.7 IPC, because dependent loads and unpredictable -indirect branches stall the pipeline. The famous framing: databases were -running BELOW 10% of what a hand-coded loop achieves ON THE SAME DATA. +> **In:** a query so simple that no system can blame its optimizer — +> one scan, a 98%-selective filter, seven arithmetic expressions, and a +> group-by with four groups (§3). +> **Out:** four measured runtimes on one machine spanning a factor of +> 121, and a profile saying where the missing time went. Everything after +> this step is an attempt to close that gap. + +The paper opens with measurement, not design. Q1 was chosen because "all +database systems operate on a level playing field and mainly expose their +expression evaluation efficiency" (§3): it is a scan of SF×6M `lineitem` +rows, selecting SF×5.9M of them, computing two column-to-constant +subtractions, one addition, three column-to-column multiplications, and +eight aggregates over just four group combinations — small enough that +"accessing the hash-table" costs no cache misses. + +That 98% is not incidental, and this repo has measured why. `exec_bench`'s +Volcano lane ([FINDINGS.md](../../FINDINGS.md) row 11) sweeps selectivity +over 50M rows and finds the model *slowest* at the high end — 74.7 M +rows/s at 95% selectivity against 129.4 M at 5% — because a rejected row +costs one `next()` call inside the filter loop while a survivor pays a +second `next()` up the chain plus the aggregate's work +([notes.md](notes.md)). Q1 is therefore near the worst case for a +tuple-at-a-time engine, which is precisely what makes it a good +microbenchmark: the per-tuple tax is at full strength. + +Table 1, restricted to the one machine that makes the comparison honest +(AthlonMP 1533 MHz, SF=1, 1 CPU): + +``` + Table 1 (AthlonMP 1533MHz, SF=1): + hand-coded C UDF (§3.3) 0.22 s <- the roofline + MonetDB/X100 0.50 s <- 2.3× off it + MonetDB/MIL 3.7 s <- full-column materialization + MySQL 4.1 26.6 s <- tuple-at-a-time interpretation + "DBMS X" (commercial) 28.1 s + + the gaps the paper's argument turns on: + 26.6 / 0.22 = 121× engine tax over hand-written C + 26.6 / 3.7 = 7.2× what column-at-a-time alone buys + 0.50 / 0.22 = 2.3× what X100 still leaves on the table +``` + +**Correction:** this guide previously printed ~0.6 s for both the +hand-coded loop and X100, and called the gap 45×. All three are wrong. +The hand-coded UDF is 0.22 s (§3.3, "a stunning 0.22 seconds"), X100 is +0.50 s, and §3.3's own summary of the gap is that X100 "is able to get +within a factor 2 of this hand-coded implementation" — it does *not* +reach the roofline. The abstract's claim is "between one and two orders +of magnitude higher than previous technology", which the 121× and 7.2× +above bracket. + +A **roofline** is the hardware's actual capacity for a given +computation — here, a hand-written loop over the same arrays. Everything +above it is engine tax, and §3.1's gprof trace of MySQL says what the tax +is spent on: ``` - hand-written C for Q1: ~0.6 s <- the roofline (topic 0!) - MySQL (Volcano, rows): ~27 s <- 45x of pure interpretation tax - MonetDB (full-column): ~3.7 s <- better, but materializes - X100 (vectors): ~0.6 s <- reaches the roofline + Table 2 — MySQL 4.1 gprof trace of Q1, SF=1 (MIPS R12000): + the five "work" operations (+, -, *, SUM, AVG) 10% of time + creation and lookup in the aggregation hash table 28% + record navigation (rec_get_nth_field and friends) 62% + + Item_func_plus::val: 38 instructions per addition, IPC 0.80 + the paper's own division: 38 / 0.8 = 49 cycles per add + what the same machine can do: = 3 cycles per multiply + (3 int/fp + 1 ld/st per cycle) + ratio = 16× ``` -The hand-written loop is a **roofline** — the hardware's actual capacity -for this query. Everything above it is engine tax. +§3.1 then explains the 49 cycles rather than merely reporting them: a +double addition is four dependent RISC instructions (two loads, an add, a +store) at ~5 cycles of latency each, and because the routine performs +exactly one addition per call, the compiler cannot pipeline the loop — +"empty pipeline slots must be generated (stalls) to wait for the +instruction latencies, such that the cost of the loop becomes 20 instead +of 3 cycles". The remaining ~29 cycles are the call itself: "the cost of +the routine call (in the ballpark of 20 cycles) must be amortized over +only one operation, which effectively doubles the operation cost." + +That last sentence is the whole thesis in one line. Both halves of the +cost — the un-pipelined dependent chain *and* the unamortized call — are +consequences of the same decision, which Step 2 names. + +The health metric is **IPC** (instructions per cycle: how many +instructions the core actually retires per clock). §2 reports that +"query execution in commercial DBMS systems get an IPC of only 0.7", +against scientific computation extracting "average IPCs of up to 2". +**Correction:** this guide previously said a superscalar core of that era +could sustain "3+" and that X100 achieved ~2 IPC. Neither is in the +paper. Three is the R12000's *issue width* for int/fp ops (§3.1), not a +sustained rate; and the paper reports no IPC figure for X100 at all — its +X100 measurements are in cycles per tuple (Step 6). ### Step 2 — failure mode one: tuple-at-a-time (Volcano) -The Volcano model — every operator exposing `next()`, each call -returning ONE tuple — pays its overhead per tuple: an indirect function -call per operator, expression-tree interpretation per row, tuple values -leaving CPU registers between operators (see -reading-postgres-executor.md for this model in production). Overhead × -N_rows, with overhead ~20–100 ns against ~1 ns of useful work. That is -MySQL's 27 s: it dies of interpretation. +> **In:** the 26.6 s and the 49-cycle addition from Step 1, which need a +> cause rather than a scapegoat. +> **Out:** the cause — an expression interpreter whose granularity is one +> tuple — and the two independent penalties that follow from it. + +The **Volcano** or **iterator model** composes operators as a tree in +which each exposes `next()`, and each call returns **one tuple** +(see [reading-postgres-executor.md](reading-postgres-executor.md) for +this model still in production). §3.1 derives the cost from the model's +generality rather than from any implementation flaw: a `ScanSelect(R, b, P)` +learns the shape of `R`, the predicate `b` and the projections `P` only +at query time, so "DBMS implementors must in fact implement an expression +interpreter that can handle expressions of arbitrary complexity", and +"one of the dangers of such an interpreter, especially if the granularity +of interpretation is a tuple, is that the cost of the 'real work' … is +only a tiny fraction of total query execution cost." + +The two penalties §3.1 lists are worth keeping separate, because +vectorization fixes them by different mechanisms: + +``` + penalty 1 — no loop pipelining. One addition per call means the + compiler cannot software-pipeline; four dependent instructions at + ~5 cycles latency stall into ~20 cycles instead of ~3. + penalty 2 — unamortized call. ~20 cycles of call overhead divided + by one operation. + total ~49 cycles per addition (measured, + Table 2: 38 instructions at IPC 0.80) +``` + +Penalty 2 is the one everybody quotes; penalty 1 is the larger surprise, +because it is not overhead at all — it is the *same* arithmetic, run +badly, because the compiler was denied the loop it needed. ### Step 3 — failure mode two: full-column-at-a-time (old MonetDB) -MonetDB — the authors' own previous system — had already fixed -interpretation by going to the opposite extreme: each operator processes -an ENTIRE column at once (the BAT algebra), so per-tuple overhead is -zero. The new problem: every operator **materializes** its full -intermediate result — writes a complete result column to memory for the -next operator to read back. For Q1 over 6M rows with ~10 intermediates, -that's hundreds of MB streamed to and from DRAM per query; every op -reads and writes DRAM-sized arrays. IPC is fine; **memory bandwidth** -becomes the wall (question 2 has you compute the seconds of pure memory -traffic). It dies of bandwidth. +> **In:** the obvious cure for Step 2 — stop interpreting per tuple by +> making the unit an entire column. +> **Out:** MonetDB/MIL, which does exactly that, has no interpretation +> problem at all, and is still 17× off the roofline — because it moved +> the bottleneck from the CPU to memory rather than removing it. + +MonetDB — the authors' own previous system — stores each column as a +BAT (Binary Association Table) and evaluates in a column algebra, MIL, +whose operators "always consume a number of materialized input BATs and +materialize a single output BAT" (§3.2). **Materialization** is that +last step: writing a complete intermediate column to memory for the next +operator to read back. + +§3.2 establishes the diagnosis by a beautiful experiment: rerun the same +plan at SF=0.001, so every column and intermediate fits in cache. +"MonetDB/MIL then becomes almost twice as fast" — the work did not +change, so the missing time was memory traffic. Table 3's own columns +make it arithmetic: + +``` + Table 3 (20 MIL invocations spanning >99% of Q1, AthlonMP, SF=1): + total measured time 3724 ms + sum of the per-operator MB column (inputs + outputs) 1361 MB + sustained bandwidth the paper reports MIL stuck at 500 MB/s + "the maximum bandwidth sustainable on this hardware" + + 1361 MB / 500 MB/s = 2.72 s + 2.72 s / 3.724 s = 73% of the query + + in cache at SF=0.001 the same operators exceed 1.5 GB/s +``` + +**Correction:** this guide previously described "~10 intermediates" and +"hundreds of MB". The trace shows 20 MIL invocations and 1361 MB — more +than a gigabyte of DRAM traffic to answer a query whose result is four +rows. + +The single multiply makes the failure vivid. §3.2 works it out: at +500 MB/s, `[*]()` moving 16 bytes in and 8 out manages 20M tuples/s, +"thus 75 cycles per multiplication on our 1533MHz CPU, which is even +worse than MySQL" — worse than the 49 cycles of the model this design was +supposed to beat. Interpretation was cured and bandwidth killed it +instead. ### Step 4 — the vector: small enough for cache, big enough to amortize -X100 threads between the two failure modes: operators still compose via -`next()`, but each call returns a **vector** — ~1000 values of one -column in a plain array. Two constraints pin the size from opposite -sides: - -- big enough that per-call interpretation divides into insignificance — - ~100 ns of dispatch over 1000 values is 0.1 ns/value; -- small enough that all the vectors in flight between the pipeline's - operators — intermediates included — stay resident in L1/L2, never - round-tripping through DRAM. Pipelining THROUGH the cache: - "hyper-pipelining". - -Their vector-size sweep makes the trade visible: performance vs vector -length is U-shaped. Length 1 = MySQL (interpretation tax), length ∞ = -old MonetDB (bandwidth wall); the sweet spot is where (vectors × columns -in flight) ≈ cache size. Vector size is a CACHE parameter, not a tuning -constant — which is why DuckDB's 2048 and X100's ~1000 are the same -decision on different hardware. Your exec_bench should reproduce this -curve's shape — sweep 1 / 64 / 1024 / 64K. +> **In:** two failure modes at opposite extremes of one dial — the unit +> of work, at 1 tuple and at a whole column. +> **Out:** the dial turned to ~1000, and the two constraints that pin it +> there from opposite sides. The paper measures the whole dial, which is +> what makes this a result rather than a preference. + +X100 keeps Volcano's pipelining but changes the payload: each `next()` +returns a **vector** — a plain array of values of one column, "e.g. 1000 +values" (§4.1.1). Two constraints, in §5.1.1's own words: "Preferably, +all vectors together should comfortably fit the CPU cache size, hence +they should not be too big. However, with really small vector sizes, the +possibility of exploiting CPU parallelism disappears. Also, in that case, +the impact of interpretation overhead in the X100 Algebra `next()` +methods will grow." + +Both ends are measured (Figure 10, Q1 on Itanium2 and AthlonMP, vector +size swept from 1 to 4M). The paper's findings, quoted rather than +paraphrased: the default is **1024**; "the optimal vector size seems to +be 1000, but all values between 128 and 8K actually work well"; and at +the far end, "at the extreme vector size of 4M tuples, MonetDB/X100 +behaves very similar to MonetDB/MIL". The curve is U-shaped, and its two +walls are exactly Steps 2 and 3. + +The best part is that §5.1.1 tells you *where* the right-hand wall is and +the arithmetic checks out: + +``` + §5.1.1: "The total width of all vectors used in Query 1 is + just over 40 bytes." + + AthlonMP, combined L1+L2 = 320 KB (the paper's figure): + 8K × 40 B = 327,680 B = 320 KB ← exactly where degradation starts + 4K × 40 B = 163,840 B = 160 KB ← comfortably inside + Itanium2, 16 KB L1 / 256 KB L2 / 3 MB L3: + 256 × 40 B = 10,240 B = 10 KB ← inside L1 + 64K × 40 B = 2,621,440 B = 2.5 MB ← the edge of L3, and §5.1.1 says + the decline runs "until data does + not fit even in L3 (after 64K × 40 bytes)" +``` + +Vector size is therefore a **cache parameter**, not a tuning constant: +the right value is whatever makes (vector length × total vector width in +flight) fit the cache you have. That is why X100's 1024 and DuckDB's 2048 +([reading-duckdb-execution.md](reading-duckdb-execution.md)) are the same +decision on different hardware — DuckDB's chunks are wider per row, and +its L1 is larger. Sweep it in `exec_bench` at 1 / 64 / 1024 / 64K and the +shape should reappear. + +And the left-hand wall is Step 2 measured on X100 itself: "Just like +MySQL, interpretation overhead also hits MonetDB/X100 strongly if it uses +tuple-at-a-time processing (i.e. a vector size of 1)." The model is not +magic; it is a dial, and 1 is the setting that makes it MySQL. ### Step 5 — primitives: interpretation happens per vector, work per value -Inside each `next()`, the work is done by **primitives**: precompiled, -type-specialized loops — `map_add_int_vec_int_vec` — selected once at -plan time. The interpreter's job shrinks to choosing which primitive to -call per vector; the primitive itself is branch-free and -auto-vectorizable (the compiler emits SIMD for it): - -```rust -// a primitive: picked once at plan time, then runs branch-free per vector -fn map_add_i64_vec(a: &[i64], b: &[i64], out: &mut [i64], - sel: Option<&[u32]>) -> usize { - match sel { - None => { for i in 0..a.len() { out[i] = a[i] + b[i]; } a.len() } - Some(s) => { - for (o, &i) in s.iter().enumerate() { - out[o] = a[i as usize] + b[i as usize]; - } - s.len() - } - } -} // interpretation: ONE dispatch per ~1000 values, not per value; - // ~1000 × 8 B per operand keeps the intermediates in L1 -``` - -Note the `sel` parameter: **selection vectors** appear here first — -filters produce index lists over untouched data, and every primitive -takes an optional sel (DuckDB inherits this wholesale). The cost of the -primitive scheme is combinatorics: types × operations generate hundreds -of monomorphized loops — the C++ template / Rust generics trick, paid in -compile time and binary size (question 3). - -### Step 6 — the discipline: IPC as the health metric - -The paper's lasting methodological lesson: measure IPC, not just -runtime. X100 runs at ~2 IPC where MySQL managed 0.7 — same hardware, -same data, ~3× more of the silicon actually working. Runtime tells you -*that* you're slow; IPC (plus cache-miss and branch-miss counters) tells -you *which wall* you're against — interpretation (low IPC, high -branches), bandwidth (low IPC, high misses), or genuinely compute-bound -(high IPC: you're done optimizing dispatch). This is your flamegraph + -`instruments`/counters angle for the experiments. +> **In:** an operator that has been handed a vector and must now do +> arithmetic on it. +> **Out:** a **primitive** — a precompiled, type-specialized loop chosen +> once at plan time — plus the two design consequences that come with it: +> a selection-vector convention that avoids copying, and a combinatorial +> explosion handled by code generation. + +Inside `next()`, the work is done by primitives. §4.2 prints one, and it +is short enough to read whole: + +``` + §4.2, the generated code for vectorized floating-point addition: + + map_plus_double_col_double_col(int n, + double*__restrict__ res, + double*__restrict__ col1, double*__restrict__ col2, + int*__restrict__ sel) + { + if (sel) { + for(int j=0;j____`, as the trace in Table 5 +confirms (`map_mul_flt_col_flt_col`, `map_sub_flt_val_flt_col`, +`select_lt_date_col_date_val`). + +Three things to take from those ten lines. + +**`__restrict__` is load-bearing.** §3.3 notes that the hand-coded +baseline passes `__restrict__` pointers "such that the C compiler knows +that they are non-overlapping. Only then can it apply loop-pipelining!" +The primitives get the same treatment, which is how they recover Step 2's +penalty 1 — not by removing overhead but by giving the compiler back the +loop it needs. + +**The `sel` parameter is where selection vectors enter the world.** "All +X100 vectorized primitives allow passing such selection vectors. The +rationale is that after a selection, leaving the vectors delivered by the +child operator intact is often quicker than copying all selected data +into new (contiguous) vectors" (§4.2). Note precisely what the loop does +with it: it reads `col1[i]` and writes `res[i]` at the *same* index — +§4.1.1 says the results are written "at the same positions in the output +vector as they were in the input one", and the selection vector is then +propagated onward to the aggregate. It does **not** compact. This guide's +earlier Rust sketch wrote survivors to `out[0..n]`, which is the opposite +convention and would have forced exactly the copy §4.2 is avoiding. + +**The cost is combinatorics, paid by a generator.** "X100 contains +hundreds of vectorized primitives. These are not written (and maintained) +by hand, but are generated from primitive patterns" (§4.2) — a pattern +like `any::1 +(any::1 x, any::1 y) plus = x + y` plus a file of requested +signatures (`+(double*, double*)`, `+(double, double*)`, …). That is the +C++ template trick with a makefile instead of a compiler, and question 3 +asks what the Rust equivalent costs. + +§4.2 also names the ceiling that Step 1's 2.3× gap sits against, and this +is the paper's most under-quoted paragraph. A simple binary primitive is +**load/store bound**: "for simple 2-ary calculations, each vectorized +instruction requires loading two parameters and storing one result (1 +work instruction, 3 memory instructions). Modern CPUs can typically only +perform 1 or 2 load/store operations per cycle." + +``` + per output value, a 2-ary primitive issues: + 1 arithmetic instruction + 3 memory instructions + at 2 load/stores per cycle, memory is the binding constraint: + 3 / 2 = 1.5 cycles per value, whatever the ALU could have done + + compound primitives (e.g. /(square(-(double*,double*)), double*)) + keep intermediates in registers, loading and storing only at the + edges of the expression graph — §4.2 measures them "often … twice as + fast", and notes "this factor 2 is similar to the difference between + MonetDB/X100 and the hand-coded implementation" (0.50 / 0.22 = 2.3) +``` + +So the hand-coded loop's remaining advantage is not mystery: it is one +fused expression, and X100's per-primitive boundaries force a store and +two loads that fusion would have kept in registers. That is the same +argument the 2018 shootout re-runs at book length +([reading-compiled-vs-vectorized.md](reading-compiled-vs-vectorized.md)). + +### Step 6 — the discipline: measure cycles per tuple, not just seconds + +> **In:** a runtime, which tells you *that* you are slow. +> **Out:** a per-primitive cycles-per-tuple trace, which tells you +> *which wall* you are against — and the specific numbers X100 hits, so +> you have something to compare your own kernels to. + +X100 "implements detailed tracing and profiling support using low-level +CPU counters" (§5.1), and Table 5 is the output for Q1 on the Itanium2. +Read it against MySQL's Table 2 and MIL's Table 3: + +``` + cycles per tuple for the same multiply, three systems, same query: + MonetDB/MIL 75 cycles (§3.2, derived from 500 MB/s) + MySQL 49 cycles (§3.1, 38 instructions at IPC 0.80) + MonetDB/X100 2.2 cycles (Table 5, map_mul_flt_col_flt_col) + + the rest of Table 5's range (Itanium2, SF=1): + map_fetch (enum fetch-joins) 1.9 cycles/tuple + select_lt 3.0 + map_sub / map_add 2.3 / 2.4 + aggr_sum 6.1-6.6 + aggr_count 4.3 + + bandwidth on the same multiply operator: + MonetDB/MIL 500 MB/s (RAM-bound) + MonetDB/X100 >7.5 GB/s on Itanium2, ~5 GB/s on AthlonMP (§5.1) +``` + +**Correction:** this guide previously claimed "X100 runs at ~2 IPC where +MySQL managed 0.7". The 0.7 is real (Table 2, §2). The 2 is not an X100 +measurement — §2 offers it as what *scientific computing* achieves. +Replace the comparison with the one the paper actually makes: 2.2 cycles +per multiply against 49, which §5.1 states as "way better than the 49 +cycles per tuple achieved by MySQL". + +The methodological lesson survives the correction intact, and is the +transferable part: a runtime is a scalar with no diagnosis in it. Cycles +per tuple, IPC, cache misses and branch misses tell you which wall you +are against — interpretation (low IPC, high branch count), bandwidth (low +IPC, high miss count, and the SF=0.001 experiment as the confirming +test), or genuinely compute-bound (high IPC: stop optimizing dispatch). +§3.2's cache-resident rerun is the cleanest example of the discipline in +the paper: change only the working-set size, and the hypothesis proves +itself. ## How to read the paper (with the concepts in hand) -~1 h. The TPC-H Q1 profile and the vector-size sweep figure are the two -things to internalize. - -- **§1–2 (the problem + how CPUs work)** — Steps 1–2. The 2005 CPU - tutorial is dated in constants, current in structure; skim if topic 0 - is fresh. -- **§3 (microbenchmark: TPC-H Q1) — read carefully.** Step 1's table - measured: MySQL's per-operation profile (the famous "90% overhead" - breakdown), old MonetDB's bandwidth wall (Step 3). -- **§4 (the X100 architecture)** — Steps 4–5: vectors, primitives, the - in-cache pipeline. Watch for the selection-vector plumbing. -- **§5 (evaluation)** — the vector-size sweep: find the U-curve, label - both ends with the failure modes, note where ~1000 sits relative to - their cache sizes. -- **§6** — skim; the DSM/NSM storage discussion feeds topic 12. +~1 h. The TPC-H Q1 profile (Tables 1-3) and the vector-size sweep +(Figure 10 with §5.1.1's cache arithmetic) are the two things to +internalize. + +| Section | What is there | Step | +|---|---|---| +| §1-2 | the problem, and a 2005 super-scalar CPU tutorial. Dated in constants, current in structure — skim if topic 0 is fresh. The IPC 0.7-vs-2 framing is at the end of §2 | 1 | +| §3 intro | why Q1: 6M rows, 98% selective, 4 groups, no join, so systems expose only expression evaluation | 1 | +| §3.1 + Table 1, 2 | **read carefully.** The four runtimes; the 10 / 28 / 62% breakdown; 38 instructions and 49 cycles per addition, and *why* | 1, 2 | +| §3.2 + Table 3 | MIL's bandwidth wall; the SF=0.001 cache-resident rerun; 500 MB/s; 75 cycles per multiply | 3 | +| §3.3 + Figure 4 | the hand-coded UDF: 0.22 s, `__restrict__`, and "within a factor 2" | 1, 5 | +| §4.1.1 + Figure 6 | the worked pipeline — watch the selection vector propagate from Select to Aggr without the data being copied | 5 | +| §4.2 | the generated primitive, the `sel` convention, the pattern generator, and the load/store ceiling on 2-ary primitives | 5 | +| §5.1 + Table 5 | cycles per tuple per primitive; 2.2 for multiply; >7.5 GB/s | 6 | +| §5.1.1 + Figure 10 | **read carefully.** The U-curve: default 1024, 128-8K all fine, 40 bytes of vector width per tuple, 8K × 40 B = the AthlonMP's 320 KB | 4 | +| §6 | related work; the DSM/NSM storage discussion feeds topic 12 | — | + +## Takeaway + +The paper is an argument by elimination on a single dial. Turn the unit +of work to one tuple and you get MySQL: 26.6 s, 49 cycles per addition, +90% of the time spent deciding what to do. Turn it to a whole column and +you get MonetDB/MIL: 3.7 s, no interpretation problem at all, and 1361 MB +of DRAM traffic pinned at the machine's 500 MB/s ceiling — 75 cycles per +multiply, worse than the model it replaced. Turn it to about a thousand +and both terms disappear at once, because the call amortizes *and* the +compiler gets a loop it can pipeline: 0.50 s, 2.2 cycles per multiply. + +Two things are worth carrying past the constants. First, the right vector +size is derived, not chosen — §5.1.1's "just over 40 bytes" times 8K is +the AthlonMP's 320 KB of cache, and that is the whole rule. Second, X100 +did not reach the roofline, and §4.2 says why in a sentence about +load/store ports: a per-primitive boundary costs a store and two loads +that a fused loop keeps in registers. That unfinished 2.3× is the opening +the compiled-execution literature walks through thirteen years later. ## Questions for notes.md 1. Reproduce the arithmetic: 8-col chunk of 8-byte values — what vector length keeps 3 operators' intermediates inside your M-series L1 - (128 KB data)? Does DuckDB's 2048 fit? -2. Full-column MonetDB dies of bandwidth. Compute: Q1 over 6M rows, - ~10 intermediate columns materialized — GB moved vs your Mac's - ~100 GB/s. Seconds of pure memory traffic? + (128 KB data)? Does DuckDB's 2048 fit? (§5.1.1 does the same sum with + 40 bytes of width against 320 KB.) +2. Full-column MonetDB dies of bandwidth. Table 3 measures 1361 MB moved + and 500 MB/s sustained = 2.72 s of the 3.724 s query. Redo it for your + Mac: same 1361 MB against topic 12's measured ~50 GB/s — how many + seconds, and what does that say about whether materialization is still + the failure mode it was in 2005? 3. Primitives are monomorphized per type combination — the C++ template - trick. What's the Rust equivalent, and what does it do to compile - time / binary size? (You'll hit this writing kernels.rs.) + trick, or in X100's case a pattern file and a makefile (§4.2). What's + the Rust equivalent, and what does it do to compile time / binary + size? (You'll hit this writing kernels.rs.) 4. X100 pre-dates SIMD-everywhere: which of its wins does the compiler now deliver FREE via autovectorization of the primitive loops, and what still needs explicit `std::simd`? (Answer after writing - kernels.rs — compare autovec asm vs your manual version.) + kernels.rs — compare autovec asm vs your manual version. §4.2's + load/store ceiling is the thing to check first.) ## Done when -You can draw the U-curve from memory with the two failure modes labeled, -and explain why vector size is a CACHE parameter, not a tuning constant. +Answer each before unfolding it. + +- [ ] You can draw the U-curve from memory with both failure modes labelled, and say where the right-hand wall is *and why it is there*. + +
Answer + + Figure 10 sweeps vector size from 1 to 4M for Q1. At size 1 the curve is + at its worst — §5.1.1: "Just like MySQL, interpretation overhead also + hits MonetDB/X100 strongly if it uses tuple-at-a-time processing" — and + at 4M "MonetDB/X100 behaves very similar to MonetDB/MIL", the + materialization wall. In between, "the optimal vector size seems to be + 1000, but all values between 128 and 8K actually work well"; the default + is 1024. + + The right-hand wall is at a *computable* place, not an empirical one. + Q1's vectors are "just over 40 bytes" of total width per tuple, so 8K + vectors are 8192 × 40 B = 320 KB, which is exactly the AthlonMP's + combined L1+L2. On the Itanium2 (16 KB L1, 256 KB L2, 3 MB L3) the + decline starts earlier and runs "until data does not fit even in L3 + (after 64K × 40 bytes)" = 2.5 MB. + +
+ +- [ ] You can explain why vector size is a cache parameter rather than a tuning constant, and say what that implies about DuckDB's 2048. + +
Answer + + Because the constraint that sets it is `vector length × total width of + all vectors in flight ≤ cache`. Neither side of that inequality is a + property of the query engine: the width comes from the query's columns, + the cache from the chip. §5.1.1 derives 8K from 40 bytes and 320 KB, and + had either number differed the answer would have moved with it. + + So X100's 1024 and DuckDB's 2048 are the *same* decision evaluated on + different hardware, not a disagreement — DuckDB's chunks carry more + bytes per row and its target L1 is larger (2048 × 64 B = 128 KB for + eight 8-byte columns, against the AthlonMP's 320 KB shared between L1 + and L2). Anyone porting either constant without redoing the sum is + copying an answer rather than a method. + +
+ +- [ ] You can state the two independent penalties tuple-at-a-time pays, and say which one is *not* overhead. + +
Answer + + §3.1 separates them for the 49-cycle addition. Penalty 2 is the obvious + one: the routine call costs "in the ballpark of 20 cycles" and is + amortized over a single operation, which "effectively doubles the + operation cost". Penalty 1 is the interesting one and is not overhead at + all — with one addition per call the compiler cannot software-pipeline + the loop, so the four dependent instructions (two loads, an add, a + store) stall on ~5-cycle latencies and the arithmetic itself costs "20 + instead of 3 cycles". + + It matters because the two are fixed by different things. Batching fixes + penalty 2 arithmetically. Penalty 1 is fixed only if the batched code is + a loop the compiler can pipeline — which is why §3.3 and §4.2 both make + a point of `__restrict__`. + +
+ +- [ ] You can say why MonetDB/MIL, which has no interpretation problem, was still 17× off the roofline — with the number that proves it. + +
Answer + + Because every MIL operator materializes its output: it "always consume[s] + a number of materialized input BATs and materialize[s] a single output + BAT" (§3.2). Table 3's own MB column sums to 1361 MB of traffic for a + query returning four rows, and MIL is "stuck at 500 MB/s, which is the + maximum bandwidth sustainable on this hardware" — so 1361/500 = 2.72 s + of the measured 3.724 s, about 73%, is memory movement. + + The proof is §3.2's control experiment rather than the arithmetic: + rerunning the identical plan at SF=0.001, where everything fits in + cache, makes MonetDB/MIL "almost twice as fast" and lifts the operators + above 1.5 GB/s. Same instructions, less memory, large speedup — the + bottleneck is located, not guessed. The single worst case is the + multiply at 75 cycles per tuple, which §3.2 notes is "even worse than + MySQL". + +
+ +- [ ] You can explain the `sel` argument every X100 primitive takes, and what the primitive does *not* do with it. + +
Answer + + `sel` is a selection vector: an array of `n` selected positions produced + by a Select operator. Every X100 primitive accepts one, and when it is + non-NULL the loop iterates `sel` instead of `0..n`. §4.2 gives the + rationale: "after a selection, leaving the vectors delivered by the + child operator intact is often quicker than copying all selected data + into new (contiguous) vectors." + + What it does *not* do is compact. The printed loop reads `col1[i]` and + writes `res[i]` at the same index `i = sel[j]`, and §4.1.1 confirms the + results are written "at the same positions in the output vector as they + were in the input one", with the selection vector propagated onward to + the aggregate. A primitive that wrote survivors densely into `res[0..n]` + would have performed exactly the copy the convention exists to avoid. + +
## References **Papers** - Boncz, Zukowski, Nes — "MonetDB/X100: Hyper-Pipelining Query - Execution" (CIDR 2005) — ~1 h; the TPC-H Q1 profile and the - vector-size sweep figure are the two things to internalize + Execution" (CIDR 2005) — ~1 h. Tables 1-3 (the three failure profiles), + §4.2 (the primitive and its `sel` convention), Table 5 and Figure 10 + with §5.1.1 (cycles per tuple, and the vector-size U-curve derived from + cache size) are the parts to internalize + +**In this repo** +- [reading-duckdb-execution.md](reading-duckdb-execution.md) — the same + design twenty years later, with `STANDARD_VECTOR_SIZE` where X100 has + 1024 +- [reading-postgres-executor.md](reading-postgres-executor.md) — Step 2's + failure mode, still shipping +- [reading-compiled-vs-vectorized.md](reading-compiled-vs-vectorized.md) + — the 2.3× X100 left on the table, re-measured in 2018 +- [FINDINGS.md](../../FINDINGS.md) row 11 — this repo's own + tuple-at-a-time ceiling, and the direction it moves in diff --git a/topics/12-columnar-analytics/README.md b/topics/12-columnar-analytics/README.md index 6080e2b..f470d1c 100644 --- a/topics/12-columnar-analytics/README.md +++ b/topics/12-columnar-analytics/README.md @@ -46,8 +46,9 @@ repeat runs put this lane anywhere from 24 to 76 GB/s depending on machine state. A bandwidth-bound single number wants an error bar; take the high end as the target. -Second, **this lane used to print 19 047 619 GB/s** — roughly 20 000× the -machine's bandwidth. The timing loop let LLVM hoist the pure fold out of its own +Second, **this lane used to print 19 047 619 GB/s** — 19,047,619 / 150 ≈ +**127 000×** the machine's peak memory bandwidth, and ~250 000× the 76 GB/s this +lane actually reaches. The timing loop let LLVM hoist the pure fold out of its own repetition loop, so two of three reps timed nothing and best-of-3 reported 0.000 s. A `black_box` on the input fixed it. Topic 0's first failure mode, found in this repo's own code, which is the best argument going for why the diff --git a/topics/12-columnar-analytics/reading-arrow-parquet.md b/topics/12-columnar-analytics/reading-arrow-parquet.md index f4eaeb3..f79baec 100644 --- a/topics/12-columnar-analytics/reading-arrow-parquet.md +++ b/topics/12-columnar-analytics/reading-arrow-parquet.md @@ -1,43 +1,79 @@ # Arrow & Parquet: the layout compute wants, the bytes disk wants -Two open formats split the columnar world: Arrow is "the layout -kernels compute on" (in memory, O(1) random access, almost no -encoding), Parquet is "the layout bytes rest in" (on disk, encoded -then block-compressed, stats for pruning). Before you open arrow-rs — -one Rust repo, both crates — this chapter builds each format's design -one concept at a time, then the boundary between them, which is where -engines actually differ. +Two open formats split the columnar world: Arrow is "the layout kernels +compute on" (in memory, O(1) random access, almost no encoding), Parquet is +"the layout bytes rest in" (on disk, encoded then block-compressed, statistics +for pruning). Before you open arrow-rs — one Rust repo, both crates — this +chapter builds each format's design one concept at a time, works the encoding +arithmetic on a single concrete column so the ratios are numbers rather than +adjectives, and then examines the boundary between the two formats, which is +where engines actually still differ. + +Every code anchor below is **arrow-rs 59.1.0**, the commit `fed7862` this repo +pins (`Cargo.toml:71` carries the version), quoted with the line numbers the +code occupies in that revision. Every claim about the *format* rather than this +implementation is cited to the specification at +**`apache/parquet-format@apache-parquet-format-2.11.0`**, by section name and +line, because a Rust crate's defaults are not the standard and this chapter is +careful about which is which. ## The problem in one sentence -Compute kernels want every value reachable in O(1) with zero decode, -while disks and networks want the fewest possible bytes — one layout -cannot be both (a delta-encoded value can't be read without its -predecessors), so the ecosystem standardized TWO layouts and one -question: where do you decode? +Compute kernels want every value reachable in O(1) with zero decode, while +disks and networks want the fewest possible bytes — one layout cannot be both +(a delta-encoded value cannot be read without its predecessors), so the +ecosystem standardised TWO layouts and one question: where do you decode? ## The concepts, step by step ### Step 1 — two jobs, two formats -A **memory format** is a contract about where bytes sit in RAM so that -independently written code (a Rust kernel, a Python library, a JDBC -driver) can compute over the same buffers with no conversion; a **file -format** is a contract about bytes at rest so that data survives, -ships, and can be read selectively. Arrow is the first, Parquet the -second, and the design pressures are opposite: Arrow forbids anything -that breaks O(1) random access (a kernel must jump to value 173,205 -directly); Parquet embraces any encoding that shrinks bytes, because -disk reads are the cost. Why it matters: every "why does Arrow/Parquet -do X" question in this chapter resolves to which side of this split X -lives on. +> **In:** nothing yet — this step fixes the vocabulary and the one axis every +> later step is positioned on. +> **Out:** the memory-format/file-format split, and the rule that decides which +> side any given design choice belongs to. Step 2 starts building the memory +> side. + +A **row store** keeps all of a row's columns adjacent, so reading one row is +one contiguous read and reading one column of a million rows touches a million +scattered places. A **column store** does the opposite: each column's values +are contiguous, so a query that names 3 of 100 columns reads only those 3. +Reading only the named columns is called **projection** — the relational +operator that drops columns, and in a column store the physical act of never +fetching them. (C-Store's 2005 paper uses "projection" for a completely +different thing — a sorted stored copy of the table; that clash is flagged in +[reading-cstore-compression.md](reading-cstore-compression.md).) + +Both formats here are column stores. What separates them is what they are +column stores *for*: + +- A **memory format** is a contract about where bytes sit in RAM so that + independently written code — a Rust kernel, a Python library, a JVM reader — + can compute over the same buffers with no conversion. Arrow is one. +- A **file format** is a contract about bytes at rest, so data survives, ships, + and can be read selectively. Parquet is one. + +The design pressures are opposite. Arrow forbids anything that breaks O(1) +random access, because a kernel must be able to jump straight to value +173,205. Parquet embraces any encoding that shrinks bytes, because the cost it +optimises is bytes fetched from a disk or an object store. + +Why it matters: every "why does Arrow/Parquet do X" question in this chapter +resolves to which side of this split X lives on, and the two formats disagree +on almost every choice below precisely because they are answering different +questions. ### Step 2 — an Arrow array is a recipe of buffers -Arrow represents a column ("array") as a small descriptor — data type, -length, null count — plus a fixed list of raw, contiguous **buffers**; -there are no per-value objects and no pointers between values. Every -array type is just a different recipe: +> **In:** the memory-format side of Step 1. +> **Out:** `ArrayData` — a descriptor plus a list of flat buffers — which is +> the object Steps 3, 4 and 5 each add one buffer kind or one consumer to. + +Arrow represents a column (an "array") as a small descriptor plus a fixed list +of raw, contiguous **buffers** — a buffer being an untyped, reference-counted +byte region with no internal structure. There are no per-value objects and no +pointers between values. Every array type is just a different recipe over the +same primitive: ``` Int64Array [validity bitmap][values i64 * n] @@ -46,31 +82,88 @@ array type is just a different recipe: ListArray [validity][offsets][child array] DICTIONARY vector ``` -That descriptor is `ArrayData` in arrow-rs: data type + length + null -count + `buffers` + child data. A 1M-row `Int64Array` is exactly two -allocations: a 125 KB bitmap and an 8 MB values buffer. Why it -matters: "layout as contract" is the whole product — kernels (topic -11's polars-compute) run on these buffers directly, from any language, -with zero conversion. +The descriptor is one struct, and it is worth reading in full because its field +list *is* the contract: + +```rust +// arrow-data/src/data.rs — ArrayData, the whole struct, 208-254 + 208 pub struct ArrayData { + 209 /// The data type + 210 data_type: DataType, + 211 + 212 /// The number of elements + 213 len: usize, + 214 + // ... 215-218: doc comment — the offset applies to buffers and child_data, + // ... but explicitly NOT to nulls ... + 219 offset: usize, + 220 + // ... 221-232: doc comment — which buffers a type uses is per-type, and the + // ... buffer may be larger than `len` needs ... + 233 buffers: Vec, + 234 + // ... 235-243: doc comment — non-empty only for nested types ... + 244 child_data: Vec, + 245 + 246 /// The null bitmap. + 247 /// + 248 /// `None` indicates all values are non-null in this array. + // ... 249-252: the rest of the comment — NullBuffer always covers exactly + // ... `len` elements even when internally sliced ... + 253 nulls: Option, + 254 } +``` + +The line to look at is 233: `buffers: Vec`. Everything else in the +struct is metadata *about* those bytes — type (210), element count (213), a +starting offset (219), children for nested types (244), and nulls (253). + +One correction to make while the struct is open, because an earlier version of +this chapter got it wrong: `ArrayData` does **not** store a null *count*. It +stores `nulls: Option` (253), and `None` is the encoding of "no +nulls at all" (248). A count is derivable from the buffer; it is not a field. + +A 1M-row `Int64Array` is therefore exactly two allocations: 1,000,000 × 8 B = +8,000,000 B of values, and 1,000,000 bits = 125,000 B of validity bitmap. + +Why it matters: "layout as contract" is the whole product. Kernels — topic 11's +polars-compute among them — run on these buffers directly, from any language, +with zero conversion, because there is nothing to convert. ### Step 3 — validity bitmaps: nulls without branches or holes -Arrow marks NULLs with a separate **validity bitmap** (one bit per -row: 1 = value present) rather than sentinel values or by omitting the -slot — null slots still occupy their full width in the values buffer. -For 1M rows that's 125 KB of bitmap regardless of how many nulls there -are, and value *i* is always at offset `i × 8` no matter what precedes -it. That's what makes kernels branch-free: compute everything, mask -nulls afterwards (polars `float_sum`'s masked variant, topic 11). Why -it matters: the "wasted" bytes for null slots buy unconditional O(1) -addressing — Step 1's memory-side priority, chosen explicitly over -compactness. +> **In:** the `ArrayData` buffer list from Step 2, specifically `nulls` (253). +> **Out:** the rule that value *i* always sits at byte offset `i × width`, +> which is the property Steps 4 and 5 both depend on and the thing Parquet's +> encodings in Step 7 give up. + +Arrow marks NULLs with a **validity bitmap** — one bit per row, 1 meaning the +value is present — kept in a separate buffer rather than encoded as a sentinel +value or as an omitted slot. Null slots still occupy their full width in the +values buffer. + +That costs 125 KB of bitmap per million rows regardless of how many nulls there +are, plus 8 bytes of dead space per null in an Int64 column. It buys the +property that makes kernels branch-free: value *i* is at offset `i × 8` no +matter what precedes it, so a kernel can compute over every slot unconditionally +and mask the nulls afterwards (polars's masked `float_sum`, topic 11). + +Why it matters: the "wasted" bytes buy unconditional O(1) addressing. This is +Step 1's memory-side priority chosen explicitly over compactness, and it is the +first place the two formats visibly diverge — Parquet stores nulls as +run-length-encoded definition levels precisely because it does *not* have to +support random addressing. -### Step 4 — offset-based strings, zero-copy slices, and IPC +### Step 4 — offset-based strings: two buffers for a million strings -Variable-length data avoids per-value allocations by concatenating all -bytes into ONE buffer and adding an **offsets** buffer of n+1 integers -— string *i* is `bytes[offsets[i] .. offsets[i+1]]`: +> **In:** the buffer discipline of Steps 2–3. +> **Out:** the variable-length recipe — one bytes buffer plus one offsets +> buffer — completing the set of buffers that Step 5 then hands to two +> different consumers. + +Variable-length data avoids per-value allocation by concatenating all bytes +into ONE buffer and adding an **offsets** buffer of n+1 integers; string *i* is +`bytes[offsets[i] .. offsets[i+1]]`: ``` values "ab", "", "xyz": @@ -78,176 +171,717 @@ bytes into ONE buffer and adding an **offsets** buffer of n+1 integers bytes [a b x y z] 1M strings = 2 allocations, not 1M ``` -Compare redis SDS (topic 2) — same "length-prefixed, cache-friendly" -instinct, different scale. Two consequences of the everything-is-plain- -buffers rule: +The empty string at index 1 is visible as the repeated `2` — no special case, +no sentinel. Compare redis SDS (topic 2): the same "length-prefixed, +cache-friendly" instinct, applied to a million values at once rather than one. + +The offsets buffer costs 4 bytes per row for `StringArray` (i32 offsets), which +is why a 1M-row string column carries 4,000,004 B of offsets before a single +character of payload — the same trade as Step 3, bytes spent to keep addressing +unconditional. + +Why it matters: this is the last buffer kind, and it completes the claim that +an Arrow array is *only* flat buffers. Everything in Step 5 follows from there +being no pointers to fix up. -- **Zero-copy slicing**: an array is `offset` + `len` over shared, - reference-counted (Arc'd) buffers — the same buffer serves many - arrays; slicing allocates nothing. -- **IPC** (inter-process communication — Arrow's wire format, in - `arrow-ipc/`): ship the buffers as-is; serialization = memcpy. The - whole point of a standard memory layout is that it's *already* the - wire format. +### Step 5 — the fork: the same buffers serve slices and the wire -### Step 5 — Parquet: a hierarchy built for selective reading +> **In:** the complete buffer set from Steps 2–4. +> **Out:** two consumers of those identical bytes — in-process slices, and the +> IPC wire format — which is why Arrow's layout choices are load-bearing twice. -A Parquet file splits data twice before storing anything — first -horizontally into **row groups** (~1M rows each), then per column into -**column chunks**, whose bytes are stored as **pages** (~1 MB units of -encoding/compression) — with a **footer** at the end of the file -holding all metadata plus min/max statistics: +The buffers built in Steps 2–4 now fork, and the fork is worth its own step +because the two branches are used by different readers and neither copies: + +``` + the SAME Arc'd buffers + │ + ┌───────────────────────┴───────────────────────┐ + │ │ + in-process slices IPC / Flight + ArrayData::slice — offset+len over write the buffers + shared buffers, no allocation as-is: serialise = memcpy + (data.rs:605-643) (arrow-ipc/) +``` + +**Zero-copy slicing**: an array is an `offset` (219) plus a `len` (213) over +shared, atomically reference-counted buffers, so a slice allocates nothing: + +```rust +// arrow-data/src/data.rs — ArrayData::slice, the non-nested branch, 605-643 + 605 /// Creates a zero-copy slice of itself. This creates a new + 606 /// [`ArrayData`] pointing at the same underlying [`Buffer`]s with a + 607 /// different offset and len + // ... 608-611: panic documentation ... + 612 pub fn slice(&self, offset: usize, length: usize) -> ArrayData { + // ... 613-617: checked add, and the assert that the slice is in bounds ... + 618 if let DataType::Struct(_) = self.data_type() { + // ... 619-633: the nested case — recurse into child_data so the offset + // ... propagates down to children ... + 634 } else { + 635 let mut new_data = self.clone(); + 636 + 637 new_data.len = length; + 638 new_data.offset = offset + self.offset; + 639 new_data.nulls = self.nulls.as_ref().map(|x| x.slice(offset, length)); + // ... 640-643: return new_data ... +``` + +Lines 637-638 are the whole mechanism: a clone that changes two integers. The +`Vec` cloned at 635 is a vector of `Arc`s, so the byte regions are +shared, not duplicated. + +**IPC** — inter-process communication, Arrow's wire format, in `arrow-ipc/` — +is the second consumer: it ships those same buffers as-is, so serialisation is +a length-prefixed `memcpy` rather than an encode. The whole point of +standardising a *memory* layout is that it is already the wire format. + +Why it matters: Steps 3 and 4 spent bytes to keep addressing unconditional, and +this step is where that spending is repaid twice — once for every kernel that +slices, once for every process boundary that would otherwise have serialised. + +### Step 6 — Parquet: a hierarchy built for selective reading + +> **In:** nothing from Arrow — this step crosses to the file-format side of +> Step 1 and starts again from the disk's cost model. +> **Out:** the file → row group → column chunk → page hierarchy, and the footer, +> which Step 7 fills with encodings and Step 8 fills with statistics. + +A Parquet file splits data twice before storing anything. The specification's +glossary is three sentences long and defines all of it (`README.md:72-85`): + +- a **row group** is "a logical horizontal partitioning of the data into rows" + containing exactly one column chunk per column (`README.md:72-74`); +- a **column chunk** is "a chunk of the data for a particular column", living + in one row group and "guaranteed to be contiguous in the file" + (`README.md:76-77`); +- a **page** is the subdivision of a column chunk, "conceptually an indivisible + unit (in terms of compression and encoding)" (`README.md:79-81`). + +The **footer** is the file metadata, and the spec's layout diagram +(`README.md:95-111`) shows exactly where it sits: the file ends with the +metadata, then a 4-byte little-endian length of that metadata, then the magic +`PAR1`. It is at the *end* so that a writer can stream data in one pass and +still record every chunk's location (`README.md:118`), which means every reader +starts by seeking to the last 8 bytes. ``` file - └─ row group (~1M rows) RowGroupMetaData - └─ column chunk (1 col × 1 rg) ColumnChunkMetaData - └─ pages (~1MB) encoding per page - footer: thrift metadata + min/max stats -``` - -The hierarchy exists so a reader can grab *pieces*: want 3 columns of -2 row groups out of a 500-column, 1000-row-group file? Read the -footer, then exactly 6 column chunks — a few MB from a multi-GB file. -Why it matters: on disk (or S3), the unit of cost is bytes fetched, -and the layout is organized so most bytes never get fetched. - -### Step 6 — page encodings, and the RLE/bit-packing hybrid - -Each page's values are encoded with a scheme chosen from a fixed menu: -PLAIN (raw), RLE_DICTIONARY (dictionary ids, run-length encoded), -DELTA_BINARY_PACKED (deltas, bit-packed), BYTE_STREAM_SPLIT (floats: -transpose the bytes so byte 0 of every value sits together — similar -bytes adjacent compress better; the columns-beat-rows argument, one -level down). The workhorse called "RLE" is actually a **hybrid** that -alternates per group between run-length runs and bit-packed literals — -runs when the data repeats, packed groups when it doesn't, so -non-repetitive stretches don't explode into length-1 runs: + └─ row group RowGroupMetaData (metadata/mod.rs:630) + └─ column chunk (1 col x 1 rg) ColumnChunkMetaData (:808) + └─ pages encoding chosen per page + footer: thrift metadata + statistics, then a 4-byte length, then "PAR1" +``` + +The sizes are worth pinning down, because the spec and this implementation do +not agree and both numbers get quoted as if they were one: + +| quantity | value | where | +|---|---|---| +| recommended row group size | 512 MB – 1 GB | spec `README.md:280` | +| recommended data page size | 8 KB | spec `README.md:289` | +| arrow-rs default row group | 1024 × 1024 = 1,048,576 rows | `properties.rs:48` | +| arrow-rs default page size limit | 1024 × 1024 = 1 MiB | `properties.rs:30` | +| arrow-rs default page row limit | 20,000 rows | `properties.rs:42` | + +The spec argues for small pages because "smaller data pages allow for more fine +grained reading (e.g. single row lookup)" and for large row groups because they +"allow for larger column chunks which makes it possible to do larger sequential +IO" (`README.md:278-289`). arrow-rs ships a 1 MiB page limit anyway — a +128× larger page than the spec recommends — which tells you the modern reader +is assumed to be scanning, not doing single-row lookups. + +The hierarchy exists so a reader can grab *pieces*. Want 3 columns of 2 row +groups out of a 500-column, 1000-row-group file? Read the footer, then exactly +6 column chunks. The spec makes the intended parallelism explicit +(`README.md:87-90`): file/row group for MapReduce, column chunk for IO, page +for encoding and compression. + +Why it matters: on disk or in an object store the unit of cost is bytes +fetched, and every level of this hierarchy exists so that most bytes are never +fetched at all. + +### Step 7 — page encodings, and the RLE/bit-packing hybrid + +> **In:** one page's worth of values from the column chunk of Step 6. +> **Out:** that page's bytes, encoded — the first of the two compression layers +> Step 8 stacks, and the input to the decode loop every Parquet scan runs. + +Each page's values are encoded with a scheme from a fixed, enumerated menu. +That menu is one Rust enum, and it is the shortest complete statement of what +Parquet can do: ```rust -// parquet "RLE" is really RLE + bit-packing, alternating per group: -// runs when the data repeats, packed literals when it doesn't -fn decode_hybrid(r: &mut BitReader, width: u32, out: &mut Vec) { - while let Some(header) = r.read_uleb128() { - if header & 1 == 0 { - let count = header >> 1; // RLE group: - let value = r.read_le_bytes(width); // one value, - out.extend(repeat(value).take(count)); // count copies - } else { - let literals = (header >> 1) * 8; // bit-packed group: - for _ in 0..literals { // 8-value multiples, - out.push(r.read_bits(width)); // width bits each - } - } - } -} -``` - -Each group's header low bit picks one of the two worlds. Why it -matters: this decode loop (and the `get_batch` bit-unpacker under it) -is the tight loop under every Parquet scan you'll ever profile. - -### Step 7 — two compression layers, and stats as cross-file zone maps - -Parquet compresses twice: first the **semantic** layer (Step 6's -encodings — the scan can still make sense of the bytes), then an -optional **block** layer (zstd/snappy over the whole encoded page — -opaque bytes, whole-page decompress to read anything). Only the first -layer is scannable; the second buys ratio at rest. DuckDB skips the -second layer for its own storage — the `fetch_row` random-access -constraint from the DuckDB chapter, again. - -On top, the footer keeps min/max statistics per column chunk and per -page — Parquet's zone maps. A reader evaluates predicates against -footer stats and prunes whole row groups BEFORE reading any data -pages: predicate pushdown across a file (even an S3) boundary. A -`WHERE ts >= '2026-01-01'` on a date-sorted file can skip 95% of row -groups for the cost of reading a footer measured in KB. - -### Step 8 — the boundary: where do you decode? - -Reading Parquet into Arrow is a decode from the disk layout to the -compute layout — and *when* to do it is the late-materialization -decision every engine answers differently. Two shortcuts exist: -Parquet dictionary pages can map DIRECTLY to Arrow DictionaryArrays -(no decode!), and RLE-encoded null levels decode straight into -validity bitmaps. Beyond that: - -| system | strategy | -|---|---| -| DuckDB | own format; scans execute over encodings, decode per-vector | -| polars/DataFusion | Parquet → Arrow at scan, engine sees Arrow only | -| ClickHouse | own format; decompress granules, engine sees flat columns | - -Why it matters: the formats are standardized; the boundary is where -engines still compete. Decode too early and you move decoded bytes -through the whole plan; decode too late and every operator must -understand every encoding. +// parquet/src/basic.rs — enum Encoding, doc comments elided, 388-452 + 388 enum Encoding { + // ... 389-396: PLAIN's per-type byte layout ... + 397 PLAIN = 0; + // ... 398-403: the deprecated PLAIN_DICTIONARY = 2 ... + 405 /// Group packed run length encoding. + 406 /// + 407 /// Usable for definition/repetition levels encoding and boolean values. + 408 RLE = 3; + // ... 409-425: the deprecated BIT_PACKED = 4, with its bit-order warning ... + 426 /// Delta encoding for integers, either INT32 or INT64. + 427 /// + 428 /// Works best on sorted data. + 429 DELTA_BINARY_PACKED = 5; + // ... 430-438: DELTA_LENGTH_BYTE_ARRAY = 6, DELTA_BYTE_ARRAY = 7 ... + 439 /// Dictionary encoding. + 440 /// + 441 /// The ids are encoded using the RLE encoding. + 442 RLE_DICTIONARY = 8; + // ... 443-450: BYTE_STREAM_SPLIT's doc — K byte-streams, K = sizeof(type) ... + 451 BYTE_STREAM_SPLIT = 9; + 452 } +``` + +Four of these are the working set, and each is a term this curriculum uses +everywhere, so define them here once: + +- **Run-length encoding (RLE)** stores each maximal run of equal values as one + `(value, count)` pair instead of repeating the value. In Parquet it is + deliberately *not* general: line 407's comment restricts it, and the spec is + blunter — RLE is supported only "for repetition and definition levels, + dictionary indices, [and] boolean values in data pages" + (`Encodings.md:122-127`). +- **Bit-packing** stores each integer in exactly `w` bits rather than its + natural 32 or 64, where `w` is the smallest width that fits the largest + value. +- **Dictionary encoding** stores each distinct value once in a dictionary page + and replaces the column with integer ids into it. The spec puts the two + halves in different pages: dictionary page in PLAIN, data page as RLE-encoded + ids (`Encodings.md:57-60`), and it "will fall back to the plain encoding" if + the dictionary grows too large (`Encodings.md:53-54`). +- **Delta encoding** stores each value's difference from its predecessor; the + enum's own comment says it "works best on sorted data" (428), and the crate's + header warning (`basic.rs:381-385`) is that delta encodings "sacrifice encode + and decode performance for improved storage efficiency", particularly for + record skipping under predicate pushdown. + +`BYTE_STREAM_SPLIT` is the odd one and the most interesting: it does not shrink +anything. The spec is explicit — "this encoding does not reduce the size of the +data but can lead to a significantly better compression ratio and speed when a +compression algorithm is used afterwards" (`Encodings.md:342-343`). It creates +K streams for a K-byte type and scatters each value's *i*-th byte to the *i*-th +stream (`Encodings.md:345-351`), so that the exponent bytes of a million +doubles — which barely vary — end up adjacent. That is the columns-beat-rows +argument applied one level down, to the bytes inside a value. + +#### The hybrid, and the arithmetic on one concrete column + +The workhorse called "RLE" in the enum is really a **hybrid** that alternates, +group by group, between run-length runs and bit-packed literals. The +specification's grammar (`Encodings.md:71-87`, copied verbatim into +`parquet/src/encodings/rle.rs:18-35`) is the whole format: + +``` +bit-packed-header := varint-encode( << 1 | 1) +rle-header := varint-encode( (rle-run-len) << 1) +repeated-value := value repeated, using round-up-to-next-byte(bit-width) +``` + +The low bit of each header selects the world. arrow-rs's decoder is the +grammar, executed: + +```rust +// parquet/src/encodings/rle.rs — RleDecoder::reload, 610-638 + 610 #[inline] + 611 fn reload(&mut self) -> Result { + // ... 612-615: take the BitReader, or error out ... + 617 if let Some(indicator_value) = bit_reader.get_vlq_int() { + // ... 618-623: fastparquet writes zero padding at page end; treat + // ... indicator 0 as end-of-data rather than an error ... + 624 if indicator_value & 1 == 1 { + 625 self.bit_packed_left = ((indicator_value >> 1) * BIT_PACK_GROUP_SIZE as i64) as u32; + 626 } else { + 627 self.rle_left = (indicator_value >> 1) as u32; + 628 let value_width = bit_util::ceil(self.bit_width as usize, u8::BITS as usize); + 629 self.current_value = bit_reader.get_aligned::(value_width); + // ... 630-632: error if the page ended mid-value ... + 633 } + 634 Ok(true) + // ... 635-637: no varint left — the page is exhausted ... + 638 } +``` + +Line 624 is the whole dispatch: one bit of one varint decides whether the next +group is bit-packed or a run. Line 625 multiplies by `BIT_PACK_GROUP_SIZE`, +which is 8 (`rle.rs:48`) because the spec always packs a multiple of 8 values; +line 628 rounds the run's stored value up to a whole number of bytes, which is +where "round-up-to-next-byte(bit-width)" from the grammar lives in code. + +The decode itself is `RleDecoder::get_batch` (`rle.rs:426-461`), and its two +branches are why this is called **vectorized decompression** — decoding a batch +of values with a handful of wide instructions instead of a branch per value. An +RLE run at 434 is `buffer[..].fill(repeated_value)`, a memset; a bit-packed run +at 445 defers to `BitReader::get_batch` (`parquet/src/util/bit_util.rs:696`), +the batch unpacker that is the tight loop under every Parquet scan you will +ever profile. + +Now the arithmetic, on one column carried through the rest of this topic's +chapters. **The column:** 1,000,000 INT64 values; 200 distinct values; average +run length 8, hence 1,000,000 / 8 = 125,000 runs; and every value lies in +[1,000,000,000 … 1,000,000,899], so `max − min + 1` = 900. + +First the two widths, evaluated rather than asserted: + +``` +dictionary width = ceil(log2(distinct)) = ceil(log2(200)) = 8 bits + because 2^7 = 128 < 200 <= 256 = 2^8 + +frame width = ceil(log2(max - min + 1)) = ceil(log2(900)) = 10 bits + because 2^9 = 512 < 900 <= 1024 = 2^10 +``` + +Then the sizes: + +``` +PLAIN (basic.rs:397) + 1,000,000 x 8 B = 8,000,000 B 1.00x + +RLE_DICTIONARY, every group bit-packed (the pessimistic bound) + codes 1,000,000 x 8 bits = 8,000,000 bits = 1,000,000 B + headers 1,000,000 / 512 -> 1,954 groups x 2 B = 3,908 B + dict page 200 x 8 B = 1,600 B + ----------- + 1,005,508 B 7.96x + +RLE_DICTIONARY, every group a run (the optimistic bound) + headers 125,000 x 1 B varint(8 << 1) = 16 = 125,000 B + values 125,000 x 1 B round-up-to-byte(8) = 125,000 B + dict page 200 x 8 B = 1,600 B + ----------- + 251,600 B 31.80x + +frame of reference + bit-packing (not a Parquet page encoding — this is what +DuckDB and BtrBlocks do to the raw values; it is here for comparison) + payload 1,000,000 x 10 bits = 10,000,000 bits = 1,250,000 B + the frame one stored minimum = 8 B + ----------- + 1,250,008 B 6.40x +``` + +**Frame of reference** is the encoding named in that last block: store the +group's minimum once, then bit-pack each value's offset from it. The 512 in the +pessimistic bound is `MAX_GROUPS_PER_BIT_PACKED_RUN` = `1 << 6` = 64 groups +(`rle.rs:51`) × 8 values per group, and 1,000,000 / 512 = 1953.125 → 1,954 +headers of 2 bytes each, because the ULEB128 encoding of `64 << 1 | 1` = 129 +does not fit in one byte. + +Read the two dictionary bounds together: the same encoding on the same column +spans 7.96× to 31.8× depending only on how the values are ordered. Sorting the +column does not change one byte of the dictionary — it changes which side of +line 624 the decoder spends its time on. + +The pessimistic bound also answers the worst-case question directly. Pure RLE +on non-repeating data would emit one 1-byte header plus one value per row — +worse than PLAIN. The hybrid's floor is instead 1 byte per 8-bit code plus 2 +bytes per 512 codes, i.e. 1,005,508 / 1,000,000 = **1.0055× the packed payload**, +an 0.55% overhead. That is why the format alternates instead of committing. + +Why it matters: this decode loop is where a Parquet scan spends its time, and +the arithmetic above is the entire reason anyone tolerates it — an 8× smaller +column is 8× fewer bytes off the disk. + +### Step 8 — two compression layers, and statistics as cross-file zone maps + +> **In:** the encoded pages of Step 7 and the footer skeleton of Step 6. +> **Out:** a fully written file — bytes twice-compressed and annotated with +> min/max statistics — plus the honest accounting of what "GB/s" means once +> bytes on disk and bytes processed differ by 8×. + +Parquet compresses twice: + +1. the **semantic** layer — Step 7's encodings, which a scan can still make + sense of, because a dictionary id is still an id and a bit-packed integer is + still an integer; +2. an optional **block** layer — a general-purpose byte compressor (snappy, + zstd, gzip) applied to the whole encoded page. A **block compressor** treats + its input as opaque bytes and achieves better ratios than any encoding, at + the price that nothing inside a block is readable until the whole block is + inflated. + +Only the first layer is scannable. The second buys ratio at rest and is exactly +what DuckDB refuses for its own storage, for the `fetch_row` reason set out in +[reading-duckdb-compression.md](reading-duckdb-compression.md). + +On top of both, Parquet keeps min/max statistics — **zone maps**, also called +min-max indexes: a per-region summary of the values in that region, used to +prove that no row in it can match a predicate, so the region is never read. +Parquet keeps them at two granularities, in two different structures, and this +chapter previously conflated them: + +```rust +// parquet/src/file/metadata/mod.rs — the per-chunk statistics, inside +// ColumnChunkMetaData, 808-841 (most fields elided) + 808 pub struct ColumnChunkMetaData { + // ... 809-819: descriptor, encodings, file path/offset, num_values, + // ... compression codec, sizes, page offsets ... + 820 statistics: Option, + // ... 821-840: geo statistics, encoding stats, bloom filter and page index + // ... offsets, level histograms, encryption fields ... + 841 } +``` + +```rust +// parquet/src/file/metadata/mod.rs — the per-page index, 1455-1461 + 1455 pub struct ColumnIndexBuilder { + 1456 column_type: Type, + 1457 null_pages: Vec, + 1458 min_values: Vec>, + 1459 max_values: Vec>, + 1460 null_counts: Vec, + 1461 boundary_order: BoundaryOrder, +``` + +Line 820 is the footer's per-column-chunk summary — one min and one max for a +whole chunk. Lines 1458-1459 are the **PageIndex** (the doc comment at +1451-1454 links the spec's `PageIndex.md`): one min and one max *per page*, in +parallel vectors. A reader prunes row groups with the first and pages within a +surviving chunk with the second. A `WHERE ts >= '2026-01-01'` on a date-sorted +file skips most row groups for the cost of reading a footer measured in +kilobytes — predicate pushdown across a file, or an S3, boundary. + +For string columns those statistics are stored **truncated**: arrow-rs's +default is 64 bytes (`properties.rs:54`, `DEFAULT_COLUMN_INDEX_TRUNCATE_LENGTH += Some(64)`). Truncating a minimum is safe — a prefix of the true minimum is +still ≤ every value. Truncating a maximum is not, so the writer increments it: + +```rust +// parquet/src/column/writer/mod.rs — increment, 1860-1874 + 1860 /// Try and increment the bytes from right to left. + 1861 /// + 1862 /// Returns `None` if all bytes are set to `u8::MAX`. + 1863 fn increment(mut data: Vec) -> Option> { + 1864 for byte in data.iter_mut().rev() { + 1865 let (incremented, overflow) = byte.overflowing_add(1); + 1866 *byte = incremented; + 1867 + 1868 if !overflow { + 1869 return Some(data); + 1870 } + 1871 } + 1872 + 1873 None + 1874 } +``` + +Line 1868 carries the argument: the first byte that does not overflow ends the +loop, and the result is a valid upper bound. Line 1873 is the failure case — +all bytes were `0xFF`, no upper bound of that length exists — and +`truncate_max_value` (`:1218`) handles it by falling back to the untruncated +value (`:1233`). The UTF-8 path (`increment_utf8`, `:1844`) additionally refuses +to widen a code point (`:1849`), so it can also fail and fall back. A writer +that truncated the max and *forgot* to increment would produce a max smaller +than a real value in the page, and a reader would prune a page containing +matches — a silently wrong query result. + +#### Which GB/s? + +The measured floor in this topic is a raw fold: 800,000,000 bytes in 0.014 s, +which is 57 GB/s, on a machine whose peak memory bandwidth is 150 GB/s +([FINDINGS.md](../../FINDINGS.md) row 12, and the same table in +[notes.md](notes.md)). Apply the 7.96× from Step 7 to the same column and the +accounting forks: + +``` +bytes actually read 800,000,000 / 7.96 = 100,500,000 B +time, if the machine still moves 57 GB/s of real bytes = 0.00176 s +"bandwidth", logical 800,000,000 / 0.00176 = 454 GB/s +"bandwidth", physical 100,500,000 / 0.00176 = 57 GB/s +``` + +Both figures describe the same run. 454 GB/s is three times the machine's peak +and is *not* a lie — it counts the logical bytes the query is defined over. +57 GB/s counts the bytes that crossed the bus. A "GB/s" number with no stated +denominator is unusable, and the difference is exactly the compression ratio. +(The 454 also ignores decode cost, so it is a ceiling, not a prediction: every +instruction spent in `get_batch` moves the real number down.) + +This is also the sanity check that caught this topic's own worst bug. The +`scan_bench` lane once printed **19,047,619 GB/s**, which is 19,047,619 / 150 = +about 127,000× the machine's peak — impossible under *either* denominator, +since neither logical nor physical bytes can exceed the bus by five orders of +magnitude. It was a hoisted timing loop, fixed with `black_box`; the story is +recorded in [FINDINGS.md](../../FINDINGS.md) row 12 on purpose. + +Why it matters: compression makes throughput ambiguous, and the ambiguity is +where the impossible numbers hide. + +### Step 9 — the boundary: where do you decode? + +> **In:** a written Parquet file (Steps 6–8) and Arrow's in-memory contract +> (Steps 2–5). +> **Out:** the one design decision the formats do not standardise — and the +> reason engines still differ after agreeing on both layouts. + +Reading Parquet into Arrow is a decode from the disk layout to the compute +layout, and *when* to perform it is the **late materialization** decision: +keeping data in its compact, encoded form as deep into the query plan as +possible, and reconstructing full values only for the rows that survive. + +Two shortcuts exist, and both are narrow: + +- A Parquet dictionary page can map straight onto an Arrow `DictionaryArray` + with no decode. arrow-rs's `make_byte_array_dictionary_reader` + (`parquet/src/arrow/array_reader/byte_array_dictionary.rs:77`) does this, but + its own doc comment states the two conditions that break it (`:70-73`): a + read that spans multiple column chunks, or a chunk containing any + non-dictionary-encoded page. The recommended workaround (`:75-76`) is to make + the read batch size a divisor of the row group size. +- RLE-encoded definition levels decode straight into a validity bitmap, because + Step 3's bitmap and the level encoding agree on one bit per row. + +Beyond those, somebody decodes: + +```mermaid +flowchart LR + P["Parquet pages
encoded + block-compressed"] + P -->|"decode at scan"| A["Arrow arrays
polars, DataFusion:
the engine sees Arrow only"] + P -->|"own on-disk format"| D["DuckDB: scans execute
over the encoding,
decode per vector"] + P -->|"own on-disk format"| C["ClickHouse: decompress
a granule, engine sees
flat columns"] +``` + +Decode too early and every operator in the plan moves full-width values; +decode too late and every operator must understand every encoding — which is +the maintainability problem SIGMOD '06 solved with a properties API, Step 7 of +[reading-cstore-compression.md](reading-cstore-compression.md). + +Why it matters: the formats are standardised and the boundary is not. That is +where the engines in this topic still compete, and where your own engine has a +choice to make. ## Where each step lives in the code -One repo — [arrow-rs](https://github.com/apache/arrow-rs) — both -crates; a fresh shallow clone is enough. - -- **Step 2** — `arrow-data/src/data.rs:208` — `ArrayData`: data type + - length + null count + `buffers` + child data. The buffer recipes - above are its interpretation rules per type. -- **Steps 3–4** — validity and offsets are `ArrayData` buffers (same - file); zero-copy shipping in `arrow-ipc/`. -- **Step 5** — `parquet/src/file/metadata/mod.rs`: - `RowGroupMetaData` (`:630`), `ColumnChunkMetaData` (`:808`), - min/max stats at `:1458` (`min_values`/`max_values`). -- **Step 6** — encodings enum: `parquet/src/basic.rs:397+` — PLAIN, - RLE (`:408` — the hybrid), RLE_DICTIONARY, DELTA_BINARY_PACKED - (`:429`), BYTE_STREAM_SPLIT. The hybrid encoder/decoder: - `parquet/src/encodings/rle.rs:55/:342`; the batch bit-unpacker: - `util/bit_util.rs:696` `get_batch` — the tight loop under - everything. -- **Step 7** — stats: same metadata anchors as Step 5; block - compression wraps the encoded page in the page writer/reader paths - next to `rle.rs`. - -Read order: `data.rs` first (the memory contract is one struct), then -`basic.rs` for the encoding menu, then `rle.rs` until the hybrid -decode loop is obvious, then the metadata module for the stats. +One repo — [arrow-rs](https://github.com/apache/arrow-rs) at `fed7862` +(59.1.0) — carries both crates; a fresh shallow clone is enough. The spec is a +second, much smaller repo. + +| Anchor | What | Step | +|---|---|---| +| `arrow-data/src/data.rs:208-254` | `ArrayData` — the whole memory contract in one struct | 2 | +| `arrow-data/src/data.rs:233` | `buffers: Vec` — the field everything else describes | 2 | +| `arrow-data/src/data.rs:253` | `nulls: Option` — `None` means no nulls; there is no count field | 3 | +| `arrow-data/src/data.rs:605-643` | `ArrayData::slice` — zero copy, two integers change | 5 | +| `arrow-ipc/` | the wire format: the same buffers, memcpy'd | 5 | +| `parquet/src/basic.rs:388-452` | `enum Encoding` — the complete page-encoding menu | 7 | +| `parquet/src/encodings/rle.rs:18-35` | the hybrid grammar, copied from the spec into the source | 7 | +| `parquet/src/encodings/rle.rs:48,:51` | `BIT_PACK_GROUP_SIZE` = 8, `MAX_GROUPS_PER_BIT_PACKED_RUN` = 64 | 7 | +| `parquet/src/encodings/rle.rs:610-638` | `reload` — line 624 is the run/literal dispatch | 7 | +| `parquet/src/encodings/rle.rs:426-461` | `get_batch` — `fill()` for runs (434), `BitReader` for literals (445) | 7 | +| `parquet/src/util/bit_util.rs:696` | `BitReader::get_batch` — the batch bit-unpacker under everything | 7 | +| `parquet/src/file/properties.rs:30,:42,:48,:54` | writer defaults: 1 MiB page, 20k rows/page, 1,048,576 rows/row group, 64-byte stat truncation | 6, 8 | +| `parquet/src/file/metadata/mod.rs:630` | `RowGroupMetaData` | 6 | +| `parquet/src/file/metadata/mod.rs:808-841` | `ColumnChunkMetaData`; `statistics` at 820 | 6, 8 | +| `parquet/src/file/metadata/mod.rs:1455-1461` | `ColumnIndexBuilder` — per-*page* min/max, the PageIndex | 8 | +| `parquet/src/column/writer/mod.rs:1187,:1218` | `truncate_min_value` / `truncate_max_value` | 8 | +| `parquet/src/column/writer/mod.rs:1844,:1863` | `increment_utf8`, `increment` — making a truncated max a valid bound | 8 | +| `parquet/src/arrow/array_reader/byte_array_dictionary.rs:66-77` | dictionary preservation, and the two cases that defeat it | 9 | + +And in the specification, `apache/parquet-format@apache-parquet-format-2.11.0`: + +| Anchor | What | Step | +|---|---|---| +| `README.md:64-85` | the glossary: row group, column chunk, page | 6 | +| `README.md:87-90` | unit of parallelisation per level | 6 | +| `README.md:92-118` | the file layout, and why the footer is last | 6 | +| `README.md:277-289` | recommended row group (512 MB–1 GB) and page (8 KB) sizes | 6 | +| `Encodings.md:26` | Plain (PLAIN = 0) | 7 | +| `Encodings.md:50-63` | Dictionary encoding, and the fallback to PLAIN | 7 | +| `Encodings.md:66-144` | the RLE/bit-packing hybrid: grammar (71-87), bit order (89-111), where RLE is legal (122-127) | 7 | +| `Encodings.md:175` | Delta encoding (DELTA_BINARY_PACKED = 5) | 7 | +| `Encodings.md:338-365` | Byte Stream Split, with its worked 3-float example | 7 | + +Read order: `data.rs` first (the memory contract is one struct), then the +spec's `README.md` glossary and file layout, then `basic.rs` for the encoding +menu, then `Encodings.md:66-144` beside `rle.rs` until the hybrid is obvious, +then the metadata module for the statistics. ## Questions for notes.md -1. Why does Arrow have almost NO encodings (just dictionary + REE) - while Parquet has many? What would delta-encoded values break for - an O(1)-random-access compute kernel? -2. Parquet's RLE hybrid: why alternate runs with bit-packed groups - instead of pure RLE? (What input kills pure RLE — and what's the - worst-case size vs PLAIN?) -3. BYTE_STREAM_SPLIT: why does splitting f64s into 8 byte-planes help - zstd? Connect to why columns compress better than rows — it's the - same argument one level down. -4. min/max stats on a string column: why do engines store truncated - prefixes, and what bug lurks if truncation isn't handled on the max - side? (Hint: "abc\xff…" — increment-the-prefix.) -5. M12: property columns for FalkorDB — Arrow-style validity bitmaps - for optional properties, or a separate presence structure - (roaring bitmap keyed by node id)? What does each cost when 1% vs - 99% of nodes have the property? +1. Why does Arrow have almost NO encodings (just dictionary and run-end) while + Parquet has nine? Take `DELTA_BINARY_PACKED` specifically: what does it + break for a kernel that assumes value *i* is at offset `i × 8` + (`data.rs:208-254`)? +2. Parquet's RLE hybrid alternates runs with bit-packed groups instead of using + pure RLE. Work the worst case for both on the Step 7 column: what does pure + RLE cost on 1M non-repeating 8-bit codes, and what does the hybrid cost + (`rle.rs:48,:51` give you the group sizes)? +3. `BYTE_STREAM_SPLIT` does not shrink anything (`Encodings.md:342-343`) yet it + is in the menu. Why does scattering a double's 8 bytes into 8 streams help + the block compressor of Step 8, and how is that the same argument as + columns-beat-rows one level down? +4. Statistics on a string column are truncated to 64 bytes by default + (`properties.rs:54`). Walk `increment` (`:1863`) on the truncated prefix of + `"zzz…z\xff\xff"`: what does it return, what does `truncate_max_value` + (`:1218`) do with that, and what would go wrong if a writer truncated the + max without incrementing it? +5. M12: property columns for FalkorDB — Arrow-style validity bitmaps for + optional properties (Step 3: 125 KB per million rows, always), or a separate + presence structure such as a roaring bitmap keyed by node id? Compute both + at 1% and at 99% density for a million nodes and say which you would ship. + +## Takeaway + +Arrow spends bytes to keep addressing unconditional; Parquet spends CPU to keep +bytes few. Neither is a compromise, because they are answering different +questions — and the only unstandardised part, where you decode between them, is +the part that still decides an engine's performance. ## Done when -You can draw both hierarchies (buffers / file→rg→chunk→page), explain -the two compression layers and why only one is scannable, and name -where the Parquet→Arrow decode happens in polars vs DuckDB. +Answer each before unfolding it. + +- [ ] You can draw both hierarchies — Arrow's buffer recipes and Parquet's file → row group → column chunk → page — and say what sits in the footer and why it is at the end of the file. + +
Answer + + Arrow: an `ArrayData` (`arrow-data/src/data.rs:208-254`) is a data type + (210), a length (213), an offset (219), a `Vec` (233), child arrays + for nested types (244) and `nulls: Option` (253). Each type is a + recipe over that buffer list: `Int64Array` is [validity][values], + `StringArray` is [validity][offsets i32 × (n+1)][utf8 bytes], `ListArray` is + [validity][offsets][child]. A 1M-row Int64 column is exactly two allocations, + 8,000,000 B of values and 125,000 B of bitmap. + + Parquet: a file holds row groups, each holding exactly one column chunk per + column, each chunk divided into pages that are indivisible for encoding and + compression (spec `README.md:72-85`). The footer holds the thrift metadata — + `RowGroupMetaData` (`metadata/mod.rs:630`), `ColumnChunkMetaData` (`:808`) + with its `statistics` (`:820`) — followed by a 4-byte little-endian length + and the magic `PAR1` (`README.md:109-111`). + + It is last because metadata "is written after the data to allow for single + pass writing" (`README.md:118`): the writer cannot know a chunk's byte offset + until it has written the chunk. The cost is that every reader begins with a + seek to the end of the file, which is why footer size, not file size, is what + a wide-schema reader complains about. + +
+ +- [ ] You can explain the two compression layers, say which one a scan can still make sense of, and name the constraint that makes DuckDB refuse the second for its own storage. + +
Answer + + Layer one is semantic: the page encodings of `enum Encoding` + (`parquet/src/basic.rs:388-452`) — PLAIN, RLE_DICTIONARY, DELTA_BINARY_PACKED, + BYTE_STREAM_SPLIT. After this layer a dictionary id is still an id, so a + filter can compare ids without materialising strings, and an RLE run can be + decoded with `fill()` (`rle.rs:434`) or skipped whole. + + Layer two is a block compressor — snappy or zstd over the entire encoded page. + It treats bytes as opaque and gives up all of the above: no value in the page + is readable until the whole page is inflated. + + DuckDB refuses layer two by default because its compression contract requires + every encoding to serve `fetch_row`, a single-row random access — see + [reading-duckdb-compression.md](reading-duckdb-compression.md). A block codec + turns "give me row 1907" into "inflate 100 KB", so zstd survives there only as + a last-resort fallback for columns nothing else catches. + +
+ +- [ ] You can compute, for a column of 1M INT64s with 200 distinct values and average run length 8, the dictionary bit width and the encoded size under both bounds of the RLE hybrid — and say what turns one into the other. + +
Answer + + The width is `ceil(log2(200))` = 8 bits, because 2^7 = 128 < 200 ≤ 256 = 2^8; + the spec stores it as one byte at the head of the data page + (`Encodings.md:59-60`). + + All-bit-packed: 1,000,000 × 8 bits = 1,000,000 B of codes, plus one header + per 512 values — 64 groups (`rle.rs:51`) × 8 values (`rle.rs:48`) — so + 1,000,000/512 → 1,954 headers × 2 B = 3,908 B, plus a 200 × 8 B = 1,600 B + dictionary page: 1,005,508 B, or 7.96× against PLAIN's 8,000,000 B. + + All-runs: 125,000 runs, each a 1-byte header (`varint(8 << 1)` = 16) and a + 1-byte value (`round-up-to-next-byte(8 bits)`), plus the same 1,600 B + dictionary: 251,600 B, or 31.80×. + + What turns one into the other is the *order* of the rows, nothing else. The + dictionary is identical either way. Sorting the column moves the decoder from + the bit-packed branch of `reload` (`rle.rs:625`) to the run branch (`:627`), + and multiplies the ratio by four on this column. + +
+ +- [ ] You can say why a "GB/s" figure for a compressed scan is ambiguous, and use that ambiguity to explain how this topic's 19,047,619 GB/s was caught. + +
Answer + + Because two different byte counts are in play. The topic's raw fold moves + 800,000,000 B in 0.014 s = 57 GB/s on a 150 GB/s machine + ([FINDINGS.md](../../FINDINGS.md) row 12). Encode the same column at the + 7.96× above and it occupies 100,500,000 B; at the same 57 GB/s of *real* + traffic that is 0.00176 s. Divide the logical 800,000,000 B by that time and + you get 454 GB/s — three times the machine's peak, and honest, because it + counts bytes the query is defined over rather than bytes that crossed the + bus. Divide the compressed bytes by the same time and you get 57 GB/s again. + Both are "the bandwidth"; neither means anything unless the denominator is + stated. + + 19,047,619 GB/s survives neither reading. Against the machine's 150 GB/s peak + it is about 127,000× too fast, and no compression ratio available on + 1M-value columns is anywhere near five orders of magnitude. The cause was a + timing loop that let the compiler hoist the fold out of its own repetition + loop, so two of three repetitions measured nothing; `black_box` on the input + fixed it. The lesson kept in [FINDINGS.md](../../FINDINGS.md) row 12 is that + the number was caught by its own implausibility, which only works if you know + what the hardware's ceiling is. + +
+ +- [ ] You can name where the Parquet → Arrow decode happens in polars/DataFusion against DuckDB, and state the two conditions under which arrow-rs cannot preserve a dictionary across that boundary. + +
Answer + + polars and DataFusion decode at the scan: the reader turns pages into Arrow + arrays and every operator above sees Arrow only. DuckDB does not use Parquet + as its own storage at all — its segments carry their own encodings and its + operators execute over them, decoding per 2,048-value vector, so the + equivalent boundary sits inside the executor rather than at the file edge. + ClickHouse likewise decompresses a granule and hands flat columns up. + + arrow-rs can skip the decode entirely for dictionary-encoded byte arrays: + `make_byte_array_dictionary_reader` + (`parquet/src/arrow/array_reader/byte_array_dictionary.rs:77`) hands the ids + straight to a `DictionaryArray`. Its doc comment (`:70-73`) names the two + conditions that defeat it: a single read spanning multiple column chunks + (each chunk has its own dictionary page, so the ids mean different things), + and a column chunk containing any non-dictionary-encoded page (the writer + fell back to PLAIN, as `Encodings.md:53-54` allows when the dictionary grows + too big). The documented mitigation for the first is to choose a read batch + size that divides the row group size (`:75-76`). + +
## References +**Specification** +- [parquet-format](https://github.com/apache/parquet-format) at + `apache-parquet-format-2.11.0` — `README.md` for the glossary (64-85), the + file layout (92-118) and the size recommendations (277-289); `Encodings.md` + for every page encoding, in particular the RLE/bit-packing hybrid grammar + (66-144) and Byte Stream Split (338-365) + +**Code** +- [arrow-rs](https://github.com/apache/arrow-rs) at `fed7862` (59.1.0) — one + repo, both crates: `arrow-data/src/data.rs` (`ArrayData`, the layout + contract, and `slice`), `arrow-ipc/` (zero-copy shipping), + `parquet/src/basic.rs` (the encoding enum), + `parquet/src/encodings/rle.rs` + `parquet/src/util/bit_util.rs` (the hybrid + and its batch unpacker), `parquet/src/file/metadata/mod.rs` (footer and page + statistics), `parquet/src/file/properties.rs` (writer defaults), + `parquet/src/column/writer/mod.rs` (statistic truncation); a fresh shallow + clone is enough + **Papers** - Melnik et al. — "Dremel: Interactive Analysis of Web-Scale Datasets" - (VLDB 2010) — optional; the repetition/definition-level encoding for - nested data that Parquet adopted wholesale (skipped here — graphs - are flat) + (VLDB 2010) — optional; the repetition/definition-level encoding for nested + data that Parquet adopted wholesale, skipped here because graphs are flat -**Code** -- [arrow-rs](https://github.com/apache/arrow-rs) — one repo, both - crates: `arrow-data/src/data.rs` (ArrayData, the layout contract), - `arrow-ipc/` (zero-copy shipping), `parquet/src/basic.rs` - (encodings), `parquet/src/encodings/rle.rs` + `util/bit_util.rs` - (the hybrid), `parquet/src/file/metadata/mod.rs` (footer stats); a - fresh shallow clone is enough +**Measurements in this repo** +- [FINDINGS.md](../../FINDINGS.md) row 12 — the scan floor of 24–57 GB/s on a + 150 GB/s machine, and the 19,047,619 GB/s that preceded it +- [notes.md](notes.md) — the same table with the per-shape timings diff --git a/topics/12-columnar-analytics/reading-btrblocks-fsst.md b/topics/12-columnar-analytics/reading-btrblocks-fsst.md index 05ed4dc..86308ff 100644 --- a/topics/12-columnar-analytics/reading-btrblocks-fsst.md +++ b/topics/12-columnar-analytics/reading-btrblocks-fsst.md @@ -1,220 +1,605 @@ # FSST & BtrBlocks: compress harder, stay random-access -Dictionary encoding dedups whole strings; LZ catches partial overlap -but kills random access. FSST closes that gap — LZ4-class ratios on -similar-but-distinct strings with every single string decodable alone — -and BtrBlocks (same group, three years later) shows what happens when -you cascade such encodings recursively and pick per block by sampling. -This chapter builds both ideas step by step — the gap, the symbol -table, why static tables are the trick, the cascade, and the sampling -argument — then routes you through the two papers (FSST first: it's a -component; then BtrBlocks: the composition). +Two papers from the same research line, read together because the second one uses the +first as a component: + +- **FSST** — Peter Boncz, Thomas Neumann, Viktor Leis, *FSST: Fast Random Access String + Compression*, PVLDB 13(11), 2020. Code: . +- **BtrBlocks** — Maximilian Kuschewski, David Sauerwein, Adnan Alhomssi, Viktor Leis, + *BtrBlocks: Efficient Columnar Compression for Data Lakes*, SIGMOD 2023. Code: + . + +Read FSST first (13 pages, short) and BtrBlocks second (14 pages). Every number below +carries the section, table or figure it came from; if you find one that does not, treat it +as unverified and delete it. + +Two terms, since the papers use them interchangeably and this guide will too. +**Compression factor** (or ratio) is `uncompressed bytes / compressed bytes`, so bigger is +better and a factor below 1.0 means the "compressor" made the data *larger*. **Random +access** means fetching value *i* without touching values 0…*i*−1. + +--- ## The problem in one sentence -A column of 1M distinct URLs defeats dictionary encoding (nothing -repeats *whole*) and zstd would shrink it ~4× but forces you to -decompress a whole block to read ONE string — the gap is a string -encoding with LZ-class ratios where any single value decodes alone. +A general-purpose byte compressor such as LZ4 or zstd needs a large window of surrounding +bytes to find redundancy, so it only pays off when you compress thousands of values as one +opaque block — and then reading one value costs decompressing the whole block; FSST buys +back per-value random access by replacing the back-reference with a *static* 255-entry +symbol table, and BtrBlocks asks what a whole file format looks like when every scheme in +it has that property. + +The FSST paper measures exactly how badly the alternatives fail. On its `urls` column, +compressing each string *individually* with LZ4 gives a compression factor **below 1.0** — +the output is bigger than the input (§6.2, Figure 4), because a single URL is too short to +contain the repetition LZ4 needs. Chopping the same column into fixed-size blocks and +compressing each block recovers the ratio, but only once the blocks are big: + +| LZ4 block size (bytes) | 16 | 64 | 256 | 1 K | 4 K | 16 K | 64 K | +| ---------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | +| compression factor on `urls` | 0.46 | 0.78 | 1.14 | 1.59 | 2.03 | 2.45 | 2.73 | + +*(FSST §6.1.)* Below 256-byte blocks LZ4 inflates the data. To beat 2× you need blocks of +several KB — which is hundreds of URLs, all of which must be decompressed to read one. + +--- ## The concepts, step by step -### Step 1 — the gap: whole-string dedup vs block compression - -Dictionary encoding (topic 12's staple: store each distinct string -once, reference it by integer id) dedups WHOLE strings — useless when -strings are distinct but SIMILAR: URLs, emails, file paths, where the -redundancy is shared *substrings* ("http://www.", "@gmail.com"). -LZ-family compressors (LZ4, zstd) catch exactly that redundancy — they -replace repeated substrings with back-references into a sliding -history window — but the back-references are the poison: to decode -string 5,000 you must first decode everything its references point -into, i.e. the whole block. That breaks `fetch_row`-style random -access and rules them out as a scan-path vector format. Why it -matters: real analytics columns are full of medium-cardinality similar -strings, and until 2020 the menu offered no encoding that was both -compact and randomly accessible for them. - -### Step 2 — the symbol table: 255 substrings, 1-byte codes - -FSST (**fast static symbol table**) compresses strings with a small -fixed table of at most 255 **symbols** — each a 1–8 byte substring — -and encodes each string by greedily replacing matched substrings with -the 1-byte code of the symbol; code 255 is an escape marker meaning -"the next byte is a literal that matched no symbol": +### Step 1 — Why a byte compressor cannot give you per-value access + +> **In:** a string column, and the wish to read value *i* without reading its neighbours. +> **Out:** the reason LZ4/zstd cannot serve that wish, and the shape of a scheme that can. + +LZ4 and zstd are **block compressors**: they encode the input as a stream of literals and +*back-references* ("copy 12 bytes from 300 bytes ago"). Two consequences follow, and both +are structural, not implementation details. + +1. **The dictionary is the data itself.** A back-reference is only meaningful relative to + the bytes already decompressed, so decoding is inherently sequential: to produce byte + *n* you must have produced bytes 0…*n*−1. The FSST paper puts it exactly: LZ4 + decompression mutates internal state, "which precludes cheap point access" (§3). +2. **Short inputs have nothing to point at.** A 60-byte URL contains almost no internal + repetition. The redundancy lives *across* URLs, and a per-string compressor cannot see + it. Hence the sub-1.0 factor in the table above. + +The way out is to move the shared redundancy into a structure that lives *outside* the +compressed bytes and *does not change* while you decode: a symbol table. ``` - "http://www.example.com/index.html" - [http://www.] [example] [.com/] [index] [.html] - 3 17 9 42 51 -> 5 bytes + table -``` +LZ4 (block): [ ....... 64 KB of literals+backrefs ....... ] + ^ to read value 900 you decode from here -34 bytes → 5 bytes, ~7×, and the table itself is tiny: 255 × 8 B ≈ -2 KB, shared by the whole block. Why it matters: the compression unit -dropped from "whole string" (dictionary) to "substring" (LZ) *without* -adopting LZ's history window — the table IS the entire shared state. - -### Step 3 — static is the trick: random access and vectorized decode - -Because the symbol table is trained once and then immutable -(**static**), decoding is a pure per-code table lookup with no -history: any single string decodes alone, in isolation, and the 2 KB -table sits in L1 cache the whole time: - -```rust -// FSST decode: a table lookup per code, NO history window — -// which is exactly why one string decodes without its neighbors -fn decode(codes: &[u8], sym: &[[u8; 8]; 255], len: &[u8; 255]) -> Vec { - let mut out = Vec::new(); - let mut i = 0; - while i < codes.len() { - match codes[i] { - 255 => { out.push(codes[i + 1]); i += 2; } // escape: literal byte - c => { - let n = len[c as usize] as usize; // symbol = 1..8 bytes - out.extend_from_slice(&sym[c as usize][..n]); - i += 1; - } - } - } - out -} +FSST: symbol table (fixed, shared) + [c][c][c] [c][c] [c][c][c][c] ... + val 0 val 1 val 2 + ^ to read value 900, jump to it and decode 1-byte codes ``` -Contrast an adaptive (LZ78-style) scheme, where the table *evolves* as -you decode — every code's meaning depends on all prior codes, and -random access dies. This single property — decode one string without -its neighbors — is why DuckDB ships FSST both as a storage encoding -and as a VECTOR TYPE (FSST_VECTOR, topic 11): compressed strings flow -through the executor itself. Why it matters: the design constraint -(random access) dictated the mechanism (static table), not the other -way around. - -### Step 4 — training the table: greedy search on a sample - -The table is built by an iterative search over a small SAMPLE of the -data: start with single-byte symbols, repeatedly extend and merge -symbols, scoring each candidate by its estimated gain (frequency × -length), for a bounded number of iterations — greedy with restarts, -not an optimal search. The claims to verify in the paper: ~LZ4-class -ratios on string data, *faster* decompression than LZ4, and random -access on top; check their table for where FSST loses (long-range -redundancy, already-compressed data). The cost side: training is a -few passes over a sample — cheap, but nonzero, and a bad sample means -a bad table for the whole block. Why it matters: this is the topic's -recurring pattern — spend bounded work at write time choosing a -representation, harvest it on every scan. - -### Step 5 — BtrBlocks: encoder outputs are columns too, so recurse - -BtrBlocks starts from an observation about open formats: Parquet picks -conservative, one-shot encodings; you can do much better if the format -may choose AGGRESSIVELY per block. Its scheme: for each 64K-value -block, try every applicable encoder, pick the best — then **cascade**: -the *outputs* of one encoding (dictionary codes are an int column; FOR -residuals are an int column; dictionary entries are a string column) -get the same treatment recursively, up to depth 3: +### Step 2 — The symbol table: 255 symbols, 1-byte codes + +> **In:** a corpus of strings, and a symbol table already trained (Step 4 builds it). +> **Out:** how a string is represented, and the exact size of the decoder state. + +FSST replaces *substrings of 1 to 8 bytes* — the **symbols** — with **1-byte codes** +(§3). Fixing the code width at one byte is the design decision everything else follows +from: + +- A code is 1 byte, so there can be at most **256** codes. One is reserved as the + **escape**, leaving **255** real symbols (§3, §3.2). +- Symbols are 1–8 bytes and sit on byte boundaries — no bit-level packing, no alignment + work at decode time. +- The **escape** (code 255) means "the next input byte is a literal, emit it as-is" + (§3.2). So an unrepresentable byte costs **2 bytes** — escape + literal. + +Decoding is an array lookup and an unconditional store. The paper is precise about the +state that has to stay hot: symbols are held as 8-byte words in a **2048-byte** array +(256 × 8) with a separate **256-byte** length array, and both fit in L1 (§3.1). The +decoder writes a full 8-byte word unconditionally and then advances the output pointer by +the symbol's real length — branch-free, at the cost of writing up to 7 bytes it will +overwrite. ``` - strings ─ dictionary ─┬─ codes (ints) ─ FOR ─ bit-pack - └─ dict entries ─ FSST - doubles ─ pseudodecimal ─ (mantissa ints) ─ ... <- their new float trick +// ILLUSTRATION — not quoted from cwida/fsst; this is FSST §3.1 in Rust-shaped +// pseudocode. The real decoder is fsst.h `fsst_decompress` in cwida/fsst, and the +// production integration this repo pins is duckdb/duckdb@6c0c1a68 +// src/storage/compression/fsst.cpp:470 (`duckdb_fsst_decompress`). +let mut out = 0; +for &code in codes { + if code == 255 { // escape: next input byte is a literal + out_buf[out] = next_literal_byte(); + out += 1; + } else { + // unconditional 8-byte store, then advance by the symbol's true length + out_buf[out..out + 8].copy_from_slice(&symbols[code as usize]); + out += lengths[code as usize] as usize; + } +} ``` -So FSST slots in as one component of a larger composition — exactly -how DuckDB's `dict_fsst/` uses it. Why it matters: no single encoding -is the answer; the win compounds — dictionary might give 5×, then -bit-packed codes another 4× — and the cascade finds the composition -per block instead of per format revision. - -### Step 6 — sampling, not full analysis - -To choose among cascades without reading each block many times, -BtrBlocks estimates each candidate's ratio on small random SAMPLES — -and shows that a handful of small slices drawn from *different* -positions (not one contiguous slice!) predicts the full-block ratio -well. Compare the three answers to "who picks the encoding": DuckDB -analyzes everything (full extra pass at ingest), ClickHouse makes you -declare, BtrBlocks samples — near-optimal choice at a fraction of the -ingest cost, risking only an unrepresentative sample. (The topic 0 -sampling lesson: representative beats exhaustive.) Why it matters: -choice quality vs ingest cost is a dial, and sampling sits at its -sweet spot for data lakes where ingest volume is huge. - -### Step 7 — no block compressor on top: the CPU-vs-network bet - -BtrBlocks deliberately puts NO general-purpose byte compressor over -its cascade — everything on disk stays scannable and SIMD-decodable — -and still reaches Parquet+zstd-class ratios with ~4× faster -decompression. The bet: in the object-storage era (topic 28), network -bandwidth to S3 is plentiful and CPU is the scarce resource at scan -time, so trading a few percent of ratio for 4× cheaper decode wins. -This is Parquet's two-layer design (semantic + block) with the second -layer amputated on purpose. Why it matters: it closes the arc this -topic opened — compression IS performance, and the last block codec -standing gets cut when it stops paying for its CPU. - -## How to read the papers (with the concepts in hand) - -**FSST (VLDB '20)** — read first; it's a component. - -1. The scheme itself is Steps 2–3; the paper's contribution beyond - them is the table-construction search (Step 4) and the engineering - for vectorized decode — read both carefully. -2. Check the evaluation table for where FSST *loses* (long-range - redundancy, already-compressed data) — the honest boundary of the - technique. - -**BtrBlocks (SIGMOD '23)** — read second; it's the composition. - -1. The cascade (Step 5) and the sampling argument (Step 6) are the - core — the sampling section is the part to work through slowly - (why multiple small slices beat one contiguous slice). -2. The evaluation's Parquet+zstd comparison is Step 7's bet - quantified — note it's the same group as the VLDB '15 / LeanStore - papers, and the hardware-conscious style shows. +Two numbers worth holding onto. The symbol table's worst-case serialised size is +`8 × 255 + 255` = **2295 bytes** (§3.4: 8 bytes per symbol plus one length byte each); +typical tables are a few hundred bytes because the average symbol length is about 2 +(§3.4). And a table is per-block, so it is amortised over the whole block — DuckDB's +integration sizes this explicitly at `src/storage/compression/fsst.cpp:198-199`, dividing +the estimated payload by the block size to count how many symbol tables it will pay for. + +### Step 3 — "Static" is the whole trick + +> **In:** the symbol table from Step 2. +> **Out:** the three capabilities that follow from it never changing, and their limits. + +The table is built once per block and then **immutable** — it never adapts while +compressing, unlike LZ4's implicit sliding window. Three things fall out. + +**Random access.** Decoding value *i* needs only value *i*'s codes and the shared table. +No state carries over between values, so `fetch_row(i)` is `O(len(i))`, not +`O(bytes before i)`. + +**Selectivity-proportional work.** FSST §6.2 (Figure 5) measures this directly: FSST's +output rate is unaffected by how selective the query is, while block-LZ4 must decompress +an entire block regardless of how few rows survive the predicate. A 1-in-10,000 lookup on +LZ4 blocks of 64 K values does 64,000 values' worth of decode work; FSST does one. + +**Comparison on compressed data.** Because the mapping from string to codes is +deterministic given a table, two strings compressed with the *same* table are equal iff +their code sequences are equal (§3.4). So an equality predicate can compress the constant +once and compare bytes — no decompression at all. This is the payoff for **late +materialization** (deferring the conversion back to user-visible values until after the +filters have run). + +The limits are stated just as plainly, and they are what the exercises should check: + +- The equality trick holds only "as long as both operands are compressed with the same + symbol table" (§3.4). Two blocks trained separately have different tables, so + cross-block equality needs decompression — this is exactly what bit the paper's own + TPC-H join experiment, where "the two join predicate columns use different dictionaries" + and had to be decompressed (§6.6). +- Range comparisons, `LIKE`, and sorting are *not* supported on compressed form; §3.4 + leaves automata-based `LIKE` to future work. + +### Step 4 — Training the table: gain, iterations, sampling + +> **In:** a raw corpus (or a sample of it). +> **Out:** a 255-symbol table, and the cost of producing it. + +Choosing the best 255 symbols is circular: a symbol's worth depends on which *other* +symbols exist, because a longer symbol steals occurrences from its own prefixes (§4.1). +FSST sidesteps this by measuring worth empirically (§4.2): + +1. Start with an **empty** table. Compressing with it escapes every byte, so the first + pass produces output exactly **twice** the input size (§4.3) — this is also the + algorithm's worst case, and the honest answer to "what if my data is incompressible?" +2. Compress the corpus with the current table, counting how often each *code* occurs and + how often each *pair* of successive codes occurs. +3. Build the next generation from the top 255 candidates by **apparent gain** + = `frequency × length`, where candidates are the surviving symbols, all concatenations + of observed pairs, and every single byte plus single-byte extensions (§4.2). +4. Repeat. At least 3 iterations are needed to reach the 8-byte maximum symbol length, + and **5 iterations** converge in practice (§4.4). + +Sampling makes this cheap: the shipped utility trains on a **16 KB sample per 4 MB +chunk**, growing the sample from 6% to 100% of that sample linearly across the 5 +iterations (§4.4). The reasoning is a nice piece of statistics-free intuition — a symbol +frequent in the whole corpus is very unlikely to be absent from the sample. + +The escape code earns its keep here too: because unseen bytes are always representable, +a table trained on a sample is *valid* for data it never saw, which is what makes sampling +sound in the first place (§3.2). + +### Step 5 — What FSST actually buys, measured + +> **In:** the mechanism from Steps 2–4. +> **Out:** the numbers, and the two places the popular summary of this paper is wrong. + +Setup for everything below (§6): the "dbtext" corpus of 23 real string columns, 8 MB per +file, Intel i9-7900X (10 cores, 3.3 GHz), 32 GB RAM, LZ4 1.8.1, g++ 8.3.1 `-O3 +-march=native`, single-threaded. + +**Table 1** is the headline: + +| | LZ4 | FSST | +| --- | --- | --- | +| compression factor, average over 23 columns | **1.70×** | **2.28×** | +| compression speed, average | 608 MB/s | 977 MB/s | +| decompression speed, average | 1857 MB/s | 1942 MB/s | + +Per-column, FSST's factor ranges from **1.63×** (`yago`, a column of Wikipedia entity +names) to **3.84×** (`c_name`, TPC-H customer names). LZ4's range on the same columns is +1.14× to 3.08×. + +Two corrections to the way this paper is usually summarised, both from §6.1: + +- **"FSST gets LZ4-class ratios" understates it.** FSST is **34% better** on average + (2.28 / 1.70 = 1.34). It is not a tie; it wins. +- **"FSST decompresses faster than LZ4" is not what was measured.** The paper's own + wording: "FSST is faster on some data sets and LZ4 is on others – with the average being + almost identical" (1942 vs 1857 MB/s is 4.6%, inside the noise of a column-to-column + swing). The measured wins are **34% on ratio** and **60% on compression speed**. The + decompression story is *equal throughput plus random access* — which is the better claim + anyway, because random access is the thing LZ4 cannot do at any speed. + +Where FSST loses, also measured, also worth stating (§6.3, Silesia corpus): FSST is about +**10% better than LZ4 on text files** but **25% worse on binaries**, and on large XML/JSON +files its factor is **2–2.5× worse** than LZ4's. The premise FSST needs is *many short +strings with shared substrings*. Give it one huge document and the block compressor's +long-range matching wins. + +End-to-end, in the Umbra prototype on TPC-H SF10 with 20 threads (§6.6, Table 4): the +string pool is 4.1 GB uncompressed, 1.5 GB with LZ4, **0.69 GB with FSST**; Q19 (which +filters heavily on string columns) gets **30% faster** (99 ms → 69 ms) because compression +saves scan bandwidth *and* lets the filter push down; Q13's `LIKE` on `o_comment`, which +must decompress, slows by only **3%** (228 ms → 235 ms). + +### Step 6 — BtrBlocks, part 1: encoder output is just another column + +> **In:** a 64,000-value block and a pool of encoding schemes. +> **Out:** a cascade of schemes, and the rule that stops it. + +BtrBlocks splits each column into fixed-size blocks of **64,000 values** (§2.2) and +compresses each block independently, so the scheme can follow a changing data +distribution. Its pool is seven existing schemes plus one new one (§1): **RLE** +(run-length encoding — store `(value, run length)` instead of a repeated value), +**One Value** (the degenerate case: a whole block of one value), **Dictionary** (replace +each distinct value with a small integer code into a lookup table), **Frequency** +(BtrBlocks' variant stores the single dominant value, a bitmap of where it occurs, and the +exceptions), **FOR** (frame of reference — subtract a base so the residuals are small) with +**bit-packing** (store each residual in exactly as many bits as the widest one needs), +SIMD-FastPFOR / SIMD-FastBP128 for patched, SIMD-friendly versions of the same, **FSST** +for strings, **Roaring bitmaps** for NULLs and exception positions, and the paper's new +**Pseudodecimal** encoding for doubles. + +The structural insight is that most of these emit *more columns*. RLE on +`[3.5, 3.5, 18, 18, 3.5, 3.5]` produces a value array `[3.5, 18, 3.5]` and a run-length +array `[2, 2, 2]` (§3.2). Both are columns. Both can be compressed again — the run-length +array by One Value, the value array by Dictionary, and the resulting code array by +FastBP128. That is **cascading compression**, and BtrBlocks applies it recursively with a +default **maximum depth of 3**; when the depth is exhausted, the remaining data is stored +**uncompressed** (§3.2). + +That last clause is important and easy to skim past: the cascade terminates in raw bytes, +*not* in zstd. Step 8 is about why. + +### Step 7 — BtrBlocks, part 2: choose by sampling, not by trying everything + +> **In:** a 64,000-value block and the scheme pool from Step 6. +> **Out:** one chosen scheme per cascade level, at 1.2% of compression CPU. + +Picking the best cascade exactly would mean compressing the block with every scheme and +every combination — exponential in the cascade depth (§3). BtrBlocks instead runs, at each +recursion level (§3): + +1. Collect statistics in one pass: min, max, unique count, average run length. +2. Filter non-viable schemes by heuristic — exclude RLE if average run length < 2, exclude + Frequency if ≥ 50% of values are unique (§3.1). +3. Compress a **sample** with each surviving scheme and keep the best observed ratio. +4. Compress the whole block with the winner. +5. If the output is itself compressible, recurse from step 1. + +The sample's *shape* matters as much as its size. Random individual tuples destroy runs, +so RLE looks useless; a single contiguous range is badly biased. BtrBlocks takes +**10 runs of 64 values** from random positions in non-overlapping parts of the block — +640 values, **1% of 64,000** (§3.1, Figure 2). §6.3 scores strategies by how often they +pick the optimal scheme (or one within 2% of it) and finds that "sampling multiple small +chunks across the entire block improves accuracy compared to other strategies, though +there is little difference between strategies that choose chunks of ≥ 16 tuples". + +The measured cost/benefit of that choice (§6.3): scheme selection consumes **1.2%** of +compression CPU time, picks the correct scheme **77%** of the time, and the resulting +files are only **3.3% larger** than the best cascade achievable by exhaustive search. +Paying 1.2% to get within 3.3% of optimal is the trade the whole design rests on. + +### Step 8 — BtrBlocks, part 3: no block compressor on top, and why + +> **In:** a fully cascaded block. +> **Out:** the bet BtrBlocks is making, and the two different "GB/s" it forces you to +> distinguish. + +Parquet and ORC lean on a general-purpose compressor — Snappy or zstd — layered over their +encodings (§1). BtrBlocks does not: the cascade bottoms out uncompressed (§3.2). The +trade, measured on the Public BI Benchmark: + +| | compression factor | decompression, vs BtrBlocks | +| --- | --- | --- | +| BtrBlocks | **7.06×** | 1.0× (baseline) | +| Parquet + Snappy | 6.88× | BtrBlocks is **3.6×** faster | +| Parquet + Zstd | **8.24×** | BtrBlocks is **3.8×** faster | +| Parquet (encodings only) | — | BtrBlocks is **2.6×** faster | + +*(Factors from §6.4; decompression speedups from §6.6, averaged over Public BI. On TPC-H +the speedups are 2.6× / 3.9× / 4.2× respectively, §6.6.)* + +So BtrBlocks gives up about **14%** of Parquet+Zstd's ratio (7.06 vs 8.24) to decompress +**3.8×** faster. Whether that is a good trade depends on a metric §6.7 defines carefully, +and this is the part of the paper to read twice: + +- **T_u = uncompressed size / decompression time.** The rate at which *logical* data + appears. This is what Figure 8 plots and what a data consumer feels. +- **T_c = compressed size / decompression time** — i.e. `T_u / compression factor`. The + rate at which the decompressor can *consume bytes off the wire*. + +The distinction decides the design. Every Parquet variant reaches over 50 GB/s of T_u, +which looks comfortably above the 12.5 GB/s of a 100 Gbit link — the paper calls that "a +false conclusion stemming from the definition of decompression throughput" (§6.7). What +must exceed the network rate is **T_c**, and Table 5 shows only BtrBlocks gets close: + +| Format | T_u [GB/s] | T_c [Gbit/s] | scan cost [$] | normalized | +| --- | --- | --- | --- | --- | +| BtrBlocks | 174.6 | **86.2** | 0.97 | 1.00× | +| Parquet | 56.1 | 52.6 | 2.47 | 2.61× | +| Parquet + Snappy | 77.6 | 33.2 | 1.74 | 1.84× | +| Parquet + Zstd | 78.6 | **24.8** | 1.70 | 1.77× | + +The S3 client saturates at 91 Gbit/s on uncompressed data, so BtrBlocks' 86.2 Gbit/s uses +**95%** of the available link while Parquet+Zstd's 24.8 Gbit/s uses **27%** — the CPU, not +the network, is the bottleneck for zstd, and you pay for the idle network in instance +hours. Note the units differ between the columns: dividing T_u by T_c in the same units +recovers the aggregate compression factor those five workbooks achieved, e.g. +174.6 ÷ (86.2 / 8) = **16.2×** for BtrBlocks and 78.6 ÷ (24.8 / 8) = **25.4×** for +Parquet+Zstd — which is exactly the point, zstd compresses harder and still costs more. + +One more control from §6.8, because it is the obvious objection: is BtrBlocks fast only +because it uses SIMD? They reimplemented every decompression routine in scalar form. That +slowed in-memory decompression by **17%** — and the scalar version was still **2.3×** +faster than the fastest Parquet variant. The win is the format, not the intrinsics. + +--- + +## How to read the papers + +Read **FSST** in this order: + +1. §3 (the format) and §3.1 (decompression). Ten minutes, and it is the whole idea. +2. §3.4 "Useful Properties" — the shortest, highest-value section in the paper. Every + claim about pushing predicates into compressed data traces back to it. +3. §4.2's Figure 2 — four iterations on the toy corpus `tumcwitumvldb`, with the symbol + table after each. Work through it by hand; it is the fastest way to internalise + "apparent gain". +4. §6.1's Table 1, then §6.3. Skip §5 (AVX512 encoding) on a first pass unless SIMD is why + you came. + +Read **BtrBlocks** in this order: + +1. §2.2 (the scheme pool) and Figure 3 (the per-type decision trees). +2. §3, §3.1, §3.2 — selection, sampling, cascading. Listing 1's `pickScheme` is 10 lines + and is the entire selection algorithm. +3. §6.7 — the T_u / T_c argument. Read it even if you skip the rest of the evaluation. +4. §4 (Pseudodecimal) only if you care about floats; it is self-contained. + +For the code, this repo pins **duckdb/duckdb@6c0c1a68**, which contains a production FSST +integration you can read against the paper: + +| Idea from the papers | Where to look | +| --- | --- | +| Train a symbol table on a sample | `src/storage/compression/fsst.cpp:170` (`duckdb_fsst_create`) — the sample rate is `ANALYSIS_SAMPLE_SIZE = 0.25` at `:38`, applied at `:119` | +| Estimate the compressed size before committing | `src/storage/compression/fsst.cpp:149-203` (`StringFinalAnalyze`) | +| Refuse the scheme unless it clearly wins | `src/storage/compression/fsst.cpp:37` — `MINIMUM_COMPRESSION_RATIO = 1.2`, applied to the returned score at `:202` | +| Pay for one symbol table per block | `src/storage/compression/fsst.cpp:198-199` | +| Decode | `src/storage/compression/fsst.cpp:470` (`duckdb_fsst_decompress`) | +| FSST layered on a dictionary (BtrBlocks' `Dict+FSST`) | `src/storage/compression/dict_fsst/compression.cpp` | + +Confirm the pin before you read: `python3 tools/pinned-source.py ref duckdb`. + +--- + +## Work the numbers yourself + +Do this before reading §6 of either paper. It takes five minutes and makes the results +legible. + +**The escape's worst case.** FSST codes are 1 byte, and an escape is 2 bytes (escape + +literal). So for input where *nothing* matches a symbol, the output is 2 bytes per input +byte: a compression factor of **0.5×**, i.e. 2× inflation. §4.3 confirms this from the +other direction — the first training iteration uses the empty table and "the result will +be twice the input size". Compare with Parquet's RLE-hybrid worst case of about +1.0055× overhead (see `reading-arrow-parquet.md`, which works the same arithmetic on a +concrete column): FSST bets much more aggressively, and its safety net is *selection* — +DuckDB simply refuses to use FSST unless the sample shows at least a 1.2× win +(`fsst.cpp:37`). + +**The symbol table's fixed cost.** Serialised worst case `8 × 255 + 255` = **2295 bytes** +(§3.4); in-memory decode state `256 × 8 + 256` = **2304 bytes** (§3.1). On a 64 KB block +that is a 3.5% overhead; on a 4 MB chunk, 0.05%. This is why block size and symbol table +size are the same conversation. + +**The same concrete column as the rest of this topic.** 1,000,000 values, 200 distinct, +average run 8. As *strings* averaging 12 bytes, the raw column is +`1,000,000 × 12` = **12,000,000 bytes**. Three ways to shrink it: + +| Scheme | Arithmetic | Bytes | Factor | +| --- | --- | --- | --- | +| Dictionary alone | `1,000,000 × 1 B` codes + `200 × 12 B` dict = 1,000,000 + 2,400 | 1,002,400 | **11.97×** | +| Dictionary → RLE on the codes | `125,000 runs × 2 B` + 2,400 | 252,400 | **47.54×** | +| FSST on the raw strings (at the paper's 2.28× average) | `12,000,000 / 2.28` | 5,263,158 | **2.28×** | + +The code width is `ceil(log2(200))` = **8 bits** = 1 byte exactly (2⁷ = 128 < 200 ≤ 256 = +2⁸), so the dictionary codes need no bit-packing at all here. + +Now read the table again, because it explains BtrBlocks' Table 4 better than any prose. On +a *low-cardinality* column, dictionary encoding beats FSST by 5×, and dictionary→RLE beats +it by 20×. FSST is not a competitor to dictionary encoding; it is what you reach for when +the dictionary itself is huge — which is precisely why BtrBlocks' most common string +scheme in Table 4 is `Dict+FSST`: dictionary-encode the column, then FSST the *dictionary*. +Read Table 4's `NYC/Community Board` row (`Dict+FSST`, ratio 8.0×, 15.0 GB/s) next to +`Motos/Medio` (`OneValue`, ratio 5048.8×, 30.8 GB/s) and the point lands: scheme selection +is worth orders of magnitude, and no single scheme is the answer. + +**Which GB/s?** FINDINGS row 12 records this topic's measured scan floor of **24–57 GB/s +on a machine with ~150 GB/s of memory bandwidth**. That figure is *logical* bytes per +second: the lane folds 800 MB of decoded `u64`s. Compress the column 7.96× and the same +scan reads only ~100 MB of real traffic, so the same 57 GB/s of logical rate corresponds to +about 7.2 GB/s of memory traffic — comfortably under the bus. Quote the number the other +way round and you get an "effective bandwidth" of 454 GB/s, three times what the hardware +can deliver. Both numbers are arithmetically correct; only one of them is a memory rate. +This is the same ambiguity BtrBlocks names as T_u versus T_c (§6.7), and it is the same +class of error as this topic's own famous bug — FINDINGS row 12 also records the +**19,047,619 GB/s** a hoisted timing loop once printed here, which is about **127,000×** +the machine's peak and therefore impossible on its face. When you write down a bandwidth, +write down which bytes you counted. + +--- ## Questions for notes.md -1. FSST vs dictionary on: (a) 1M distinct URLs sharing 20 prefixes, - (b) country codes with NDV 200, (c) UUIDs. Pick the winner per case - and say why (BtrBlocks would cascade — which cascade for (a)?). -2. Why must FSST's table be STATIC (immutable after training) for - random access + vectorized decode? What would adaptive (LZ78-style) - codes break? -3. BtrBlocks samples; DuckDB analyzes everything; ClickHouse makes you - declare. Place the three on an ingest-cost / ratio-quality / - operator-burden triangle. -4. The escape byte: worst-case FSST inflation on incompressible input? - Compare with Parquet RLE-hybrid's worst case from the - arrow-parquet guide. -5. M12: property values in FalkorDB are often short similar strings - (emails, category names). Sketch the cascade for a string property - column and mark which stages allow predicate-on-encoded execution - (`= 'x'` on dict codes: yes; on FSST codes: trickier — why? unequal - code lengths, but equality CAN compare encoded bytes if the table - is shared — when is it?). +1. FSST versus dictionary encoding on (a) 1 M distinct URLs sharing 20 prefixes, + (b) country codes with 200 distinct values, (c) UUIDs. Pick the winner per case and say + why — the arithmetic table above gives you (b) directly, and FSST Table 1's `urls` row + (FSST 2.16×, LZ4 2.77×) is the calibration for (a). Which cascade would BtrBlocks build + for (a), and which for (c)? +2. Why must FSST's table be *static* — immutable after training — for random access and + vectorized decode? Name what an adaptive, LZ78-style code would break, using the + argument in §3 ("precludes cheap point access") and the branch-free decode loop of + §3.1. +3. BtrBlocks samples (§3.1: 640 values, 1.2% of CPU, 77% correct), DuckDB analyses a + fraction of everything (`fsst.cpp:38`, `:119`), and ClickHouse makes you declare the + codec in DDL. Place the three on an ingest-cost / ratio-quality / operator-burden + triangle, and say which one you would want on a column whose distribution changes + monthly. +4. The escape byte: what is FSST's worst-case inflation on incompressible input, and where + does §4.3 confirm it? Compare with the Parquet RLE-hybrid's worst case from + `reading-arrow-parquet.md`. Then say which mechanism — not which scheme — protects a + production system from each. +5. **M12**: property values in FalkorDB are often short, similar strings (emails, category + names). Sketch the cascade for a string property column and mark which stages allow + predicate-on-encoded execution. `= 'x'` on dictionary codes is easy; on FSST codes it is + trickier because codes have unequal lengths — yet §3.4 says equality *can* compare + encoded bytes. Under exactly what condition, and what does §6.6's join experiment show + happens when that condition fails? + +--- + +## Takeaway + +Both papers make the same move: they give up some compression ratio to keep the decoder's +state small, fixed and independent per value. FSST gives up the sliding window and gets +random access, predicate evaluation on compressed bytes, and — measured, not asserted — +34% better ratios than LZ4 at roughly equal decompression speed. BtrBlocks gives up zstd +and gets 3.8× faster decompression for 14% worse ratios, which turns out to be the better +end of the trade once you measure throughput in the units the network actually charges +you for. The general lesson for this topic: a compression scheme's *access pattern* is +usually worth more than its ratio, and the only way to know is to be specific about which +bytes you are counting per second. + +--- ## Done when -You can explain FSST in three sentences (symbol table, 1-byte codes, -random access), BtrBlocks in two (sample per block, cascade), and -argue when each beats plain dictionary + zstd. +Answer each before unfolding it. + +- [ ] An FSST-compressed column and an LZ4-compressed column have the same size on disk. + Name two things you can do with the FSST one that you cannot do with the LZ4 one. + +
Answer + +**Fetch a single value** without decompressing its neighbours: FSST decoding is a lookup +in an immutable table, so value *i* is independent of values 0…*i*−1, while LZ4's +back-references make decoding sequential — "which precludes cheap point access" (FSST §3). +Figure 5 (§6.2) measures the consequence: FSST's output rate is independent of query +selectivity, block-LZ4's is not. + +**Evaluate an equality predicate without decompressing at all**: compress the constant +with the same symbol table and compare code sequences (§3.4). §6.6 uses exactly this in +Umbra, which is why Q19 gets *faster* (99 ms → 69 ms) when the column is compressed. + +The caveat on the second one: it requires both operands to share a symbol table (§3.4). +Across blocks with independently trained tables, you are back to decompressing — which is +what happened in the §6.6 join experiment. + +
+ +- [ ] Compress 8 MB of strings that share no substrings whatsoever. What does FSST produce, + and what stops a real system from shipping that result? + +
Answer + +Every byte escapes, and an escape is 2 bytes (escape marker + literal), so the output is +**16 MB — a compression factor of 0.5×**, i.e. 2× inflation. FSST §4.3 states the same +thing from the training side: the first iteration compresses with the empty table, and +"the result will be twice the input size". + +What stops it shipping is *scheme selection*, not the scheme. DuckDB analyses a 25% sample +(`src/storage/compression/fsst.cpp:38`, applied at `:119`), estimates the compressed size +(`:149-203`), and multiplies the estimate by `MINIMUM_COMPRESSION_RATIO = 1.2` (`:37`, +applied at `:202`) so FSST only wins the comparison when it is clearly better. BtrBlocks +does the equivalent by testing every viable scheme on a 640-value sample and keeping the +best observed ratio (§3.1). + +
+ +- [ ] Where does the popular one-line summary "FSST is like LZ4 but with random access" + get the measurements wrong? + +
Answer + +In two places, both from FSST §6.1 / Table 1. + +It **understates the ratio**: FSST averages **2.28×** against LZ4's **1.70×** over the 23 +dbtext columns — 34% better, not equal. + +It **overstates decompression**: the averages are 1942 MB/s (FSST) vs 1857 MB/s (LZ4), and +the paper's own summary is "FSST is faster on some data sets and LZ4 is on others – with +the average being almost identical". The clean measured wins are 34% on ratio and 60% on +compression speed. + +It also hides the failure case (§6.3): on the Silesia binaries FSST is 25% *worse* than +LZ4, and on large XML/JSON files 2–2.5× worse. FSST's premise is many short strings with +shared substrings. + +
+ +- [ ] BtrBlocks spends 1.2% of its compression time deciding which scheme to use. Why is + that a bargain, and what is the measured cost of the shortcut? + +
Answer + +The alternative is exhaustive search: compressing each block with every scheme *and* every +cascade combination, which §3 calls "prohibitively slow" and which grows exponentially in +the cascade depth. The bargain is measured in §6.3 — sampling 10 runs of 64 values (640 +values, 1% of a 64,000-value block) costs **1.2%** of compression CPU, picks the optimal +scheme (or one within 2% of it) **77%** of the time, and yields files only **3.3% larger** +than the best achievable cascade. + +The shortcut's cost is that 3.3%, plus the risk of a mis-estimate on data whose local +structure differs from the sample — which is why the sample is 10 *runs* rather than 640 +scattered values: a run-length estimate needs locality to be meaningful (§3.1, Figure 2). + +
+ +- [ ] Parquet+Zstd compresses better than BtrBlocks (8.24× vs 7.06×) yet costs 1.77× more + to scan from S3. Explain, using the paper's own two throughput metrics. + +
Answer + +Because compression ratio and *scan cost* are connected through the metric §6.7 calls +**T_c = compressed size / decompression time = T_u / compression factor**, not through the +ratio alone. + +Parquet+Zstd's T_u of 78.6 GB/s looks far above a 100 Gbit link's 12.5 GB/s, but that +counts *uncompressed* bytes. Per byte arriving off the wire it manages only **24.8 +Gbit/s** (Table 5), against a client that can pull **91 Gbit/s** — so the link sits ~73% +idle while the CPU works, and you rent the instance for the whole time. BtrBlocks reaches +**86.2 Gbit/s** of T_c, ~95% of the link, and finishes the scan for **$0.97 against +$1.70** (Table 5). + +The general form: when data arrives over a channel, the decompressor must keep up +*measured in channel bytes*. Better ratios make that harder, not easier, because each +channel byte expands into more work. + +
+ +--- ## References -**Papers** -- Boncz, Neumann, Leis — "FSST: Fast Random Access String Compression" - (VLDB 2020) — the scheme, the table-construction search, and the - table of where it loses -- Kuschewski, Sauerwein, Alhomssi, Leis — "BtrBlocks: Efficient - Columnar Compression for Data Lakes" (SIGMOD 2023) — the sampling - argument and the cascade; same group as the VLDB '15 / LeanStore - papers - -**Code** -- [fsst](https://github.com/cwida/fsst) — the authors' reference - implementation; [btrblocks](https://github.com/maxi-k/btrblocks) — - the paper's artifact (both optional — DuckDB's `fsst.cpp` in - [reading-duckdb-compression.md](reading-duckdb-compression.md) is the - production integration) +- Peter Boncz, Thomas Neumann, Viktor Leis. *FSST: Fast Random Access String Compression*. + PVLDB 13(11): 2649–2661, 2020. +- Maximilian Kuschewski, David Sauerwein, Adnan Alhomssi, Viktor Leis. *BtrBlocks: + Efficient Columnar Compression for Data Lakes*. SIGMOD 2023. + +- FSST reference implementation: +- BtrBlocks implementation: +- DuckDB's FSST integration, pinned at `duckdb/duckdb@6c0c1a68`: + `src/storage/compression/fsst.cpp`, `src/storage/compression/dict_fsst/` +- `FINDINGS.md` row 12 — this topic's measured scan floor (24–57 GB/s on a ~150 GB/s + machine) and the 19,047,619 GB/s hoisted-loop bug. +- `reading-arrow-parquet.md` in this topic — the Parquet RLE-hybrid worst case that the + escape's 2× inflation is compared against. diff --git a/topics/12-columnar-analytics/reading-clickhouse-mergetree.md b/topics/12-columnar-analytics/reading-clickhouse-mergetree.md index 3d2cea0..780a328 100644 --- a/topics/12-columnar-analytics/reading-clickhouse-mergetree.md +++ b/topics/12-columnar-analytics/reading-clickhouse-mergetree.md @@ -1,224 +1,625 @@ # MergeTree: brute force, organized -ClickHouse's storage engine is topic 4's LSM shapes at analytics -scale: immutable sorted parts, background merges, and — because the -workload is scans, not point reads — an index that is deliberately -SPARSE. Before you open `src/Storages/MergeTree/` (the codebase is -huge; read ONLY what's anchored below), this chapter builds the -machine step by step: what a part is, what's inside one, why reads -happen 8192 rows at a time, how a sparse index answers "which 8192 -rows" instead of "which row", why every mark is two offsets, what the -background merges do, and who picks the compression. Then it hands -you the file and line anchors. +ClickHouse's storage engine is topic 4's LSM shapes at analytics scale: immutable sorted +parts, background merges, and — because the workload is scans, not point reads — an index +that is deliberately **sparse**. + +`src/Storages/MergeTree/` is enormous. Read only what is anchored here. Every anchor below +was checked against **ClickHouse/ClickHouse@4d598fb2c**; confirm the pin before you start: + +``` +python3 tools/pinned-source.py ref clickhouse +``` + +Two words used constantly below. A **zone map** (or **min-max index**) is a small summary — +usually just the minimum and maximum value — attached to a chunk of rows, letting a reader +skip the chunk when a predicate cannot be satisfied inside those bounds. **Write +amplification** is the total bytes written to storage divided by the bytes of user data +inserted; **read amplification** is the analogous ratio on the read side. + +--- ## The problem in one sentence -Serve `GROUP BY` scans over billions of rows arriving at millions of -inserts per second — a B-tree that pays a page write per row can't -ingest that, and a per-row index would be bigger than the data, so -ClickHouse indexes only every 8192nd row and makes scanning the rest +Serve `GROUP BY` scans over billions of rows arriving at millions of inserts per second — +a B-tree that pays a page write per row cannot ingest that, and a per-row index would rival +the data in size, so ClickHouse indexes only every 8192nd row and makes scanning the rest cheap enough not to care. -## The concepts, step by step - -### Step 1 — the part: an insert writes new files, never modifies old ones +Put a number on the second clause. A dense index over 10 billion `UInt64` keys costs +`10,000,000,000 × 8 B` = **80 GB** before any tree overhead — it cannot stay in memory. One +entry per 8192 rows costs `10,000,000,000 / 8192` = 1,220,703 entries × 8 B = **9.8 MB**, +which is smaller than a CPU's last-level cache on a large server. That factor of 8192 is +the whole design, and everything in this guide is a consequence of it. -A **part** is one self-contained directory of files holding a batch of -rows, sorted by the table's declared sort key (`ORDER BY`); every -INSERT creates a brand-new part, and existing parts are never modified -— they are **immutable**. A table is just the set of its current -parts, and background **merges** combine small parts into bigger ones -(and delete the inputs). +--- -``` - table = set of immutable sorted PARTS (sorted by ORDER BY key) - INSERT -> writes a NEW part (no in-place anything; topic 4's - immutability), background MERGES combine parts -``` +## The concepts, step by step -This is an LSM (log-structured merge design, topic 4: absorb writes -into new sorted files, merge in background) where: memtable ≈ the -insert block, SSTable ≈ part, compaction ≈ merge — but no WAL-per-row, -no point-read path. Why it matters: ingest is pure sequential file -writes at disk bandwidth, and everything read-side can assume sorted, -immutable data. +### Step 1 — The part: an insert writes new files, never modifies old ones -### Step 2 — inside a part: one file per column, sorted by the ORDER BY key +> **In:** a batch of rows arriving in an `INSERT`. +> **Out:** a new immutable directory on disk, and a table that is now the set of its parts. -Within a part, each column is stored in its own file — the columnar -split — and all files are ordered by the same sort key, so row *i* of -every column file belongs to the same logical row: +A **part** is one self-contained directory of files holding a batch of rows, sorted by the +table's declared sort key (`ORDER BY`). Every `INSERT` creates a brand-new part; existing +parts are never modified — they are **immutable**. A table *is* the set of its currently +active parts, and background **merges** combine small parts into bigger ones, then delete +the inputs. ``` - part = one directory: one file per column + primary.idx + marks + table = set of immutable sorted PARTS (each sorted by the ORDER BY key) + INSERT ──▶ writes a NEW part (no in-place anything) + background MERGE ──▶ reads N parts, writes 1, drops the N ``` -The `ORDER BY` key you declare at table creation decides the physical -sort — and therefore the clustering — of every part forever. That is -the price of admission ClickHouse states upfront: you must know your -main filter column at schema time (DuckDB's zone maps have the same -clustering dependency, just undeclared). Why it matters: sorting is -what makes the sparse index (Step 4) and the compression both work. +This is an LSM design (topic 4: absorb writes into new sorted files, merge in the +background) with the vocabulary shifted: memtable ≈ the insert block, SSTable ≈ part, +compaction ≈ merge. What it drops relative to a key-value LSM is just as important: no +write-ahead log per row and no point-read path worth optimising. + +**Why it matters:** ingest becomes pure sequential file writes at disk bandwidth, and every +read-side structure may assume its input is sorted and will never change under it. Steps +3–6 all cash in that assumption. + +### Step 2 — Inside a part: Wide or Compact, and why the answer is not always "one file per column" + +> **In:** one part directory. +> **Out:** the file layout inside it, and the size threshold that switches between two +> layouts. + +The textbook answer is "one file per column" — that is the **column store** arrangement, +where each column's values are contiguous, as opposed to a **row store** where a row's +fields are contiguous. ClickHouse does that, but only above a size threshold. There are two +part formats: + +| Format | Layout | When | +| --- | --- | --- | +| **Wide** | one `.bin` per column, one marks file per column | part ≥ `min_bytes_for_wide_part` | +| **Compact** | *all* columns in a single `data.bin`, all marks in `data.mrk3` | part below that threshold | + +`MergeTreeDataPartCompact.h:8-16` states it directly: "In compact format all columns are +stored in one file (`data.bin`). Data is split in granules and columns are serialized +sequentially in one granule… It's considered to store only small parts in compact format +(up to 10M)." The threshold is `default_min_bytes_for_wide_part = 10485760` — **10 MiB** — +at `MergeTreeSettings.cpp:34`, wired to the `min_bytes_for_wide_part` setting at `:76`. + +The reason is file-count economics, not query performance: a thousand small inserts into a +200-column table would otherwise create 200,000 tiny files. Compact parts are transient — +merges promote them to Wide once they exceed 10 MiB. + +Even inside a Compact part, ClickHouse tries to keep columnar skipping alive: +`compress_per_column_in_compact_parts` (default `true`, +`MergeTreeSettings.cpp:878-883`) starts a new compressed block for each column within a +granule so a reader can still skip columns it does not need — at the cost of compression +ratio, exactly as the setting's own doc says. + +The `ORDER BY` key you declare at table creation fixes the physical sort — and therefore +the clustering — of every part forever. That is the price of admission ClickHouse states +upfront: you must know your main filter column at schema time. (DuckDB's zone maps have +the same clustering dependency; it is simply never declared.) + +### Step 3 — The granule: at most 8192 rows *and* at most 10 MiB + +> **In:** a sorted part. +> **Out:** the read quantum, and the two independent caps that define it. + +A **granule** is the read quantum: the engine never reads or indexes anything smaller. A +**mark** is the per-granule, per-column bookmark saying where that granule's bytes begin. + +The number everyone quotes is 8192, and it is real — +`MergeTreeSettings.cpp:70` declares `index_granularity` with default `8192`. But read the +doc string on the next line: "**Maximum** number of data rows between the marks of an +index." There is a second, independent cap: `index_granularity_bytes`, default +`10 * 1024 * 1024` = **10 MiB** (`MergeTreeSettings.cpp:1676`), described as "Maximum size +of data granules in bytes", with a floor of `min_index_granularity_bytes = 1024` +(`:1681`). Mixed (adaptive) granularity is on by default — +`enable_mixed_granularity_parts` at `:1714`, whose doc explains it "improves ClickHouse +performance when selecting data from tables with big rows (tens and hundreds of +megabytes)". + +So a granule holds `min(8192 rows, whatever fits in 10 MiB)`. Work it: + +| Average row size | Rows in 10 MiB | Granule size | +| --- | --- | --- | +| 100 B | 104,857 | **8192 rows** (the row cap binds) | +| 1 KiB | 10,240 | **8192 rows** (the row cap binds, barely) | +| 4 KiB | 2,560 | **2560 rows** (the byte cap binds) | +| 1 MiB | 10 | **10 rows** | + +The bet behind 8192: decompressing and scanning that many rows with vectorized code costs +microseconds, so tracking anything finer buys nothing. The byte cap exists because that bet +is about *bytes touched*, not rows, and a table of 1 MiB blobs would otherwise make a +"granule" mean 8 GB. + +**Why it matters:** every read-side structure now scales with `rows / 8192` rather than +with `rows` — three orders of magnitude smaller. Step 4 spends that budget. + +### Step 4 — The sparse primary index: which granules, not which row + +> **In:** the sorted key column and the granule boundaries from Step 3. +> **Out:** an in-memory array of one key per granule, small enough to keep resident +> forever. + +The **sparse primary index** (`primary.idx`) stores the sort key of the **first row of each +granule** — one entry per 8192 rows. It is loaded into memory and kept there; +`IMergeTreeDataPart.h:424` is `getIndex()`, and `:425` is `loadIndexToCache()`, which +places it in a dedicated `PrimaryIndexCache`. -### Step 3 — the granule: reads happen 8192 rows at a time +``` + rows: 0 .... 8191 | 8192 .. 16383 | 16384 .. 24575 | ... + granule: 0 | 1 | 2 | + primary.idx: key[0] | key[8192] | key[16384] | ← one entry per granule +``` -A **granule** is the read quantum — a fixed slice of 8192 consecutive -rows (`index_granularity`); the engine never reads or indexes anything -smaller. A **mark** is the per-granule, per-column bookmark saying -where that granule's bytes start in the column file. +The arithmetic from the problem statement, now in context: 10 billion rows, `UInt64` key, +8192-row granules → 1,220,703 entries × 8 B = **9.8 MB**, against **80 GB** for a dense +index. A compound key of three `UInt64` columns triples it to 29.3 MB — still resident. + +This is what "sparse" costs and buys. A B-tree answers *which row*. This answers *which +8192 rows*, and you always over-read up to a full granule. For a scan workload that +over-read is noise; for OLTP it would be fatal, which is the honest reason ClickHouse is +not an OLTP engine. + +### Step 5 — From predicate to mark ranges: two different search algorithms + +> **In:** the in-memory index from Step 4 and a `WHERE` clause. +> **Out:** a list of `MarkRange`s to read — computed by one of two algorithms, depending on +> the predicate's *shape*. + +This is the read path's payload, and it is where the popular summary of MergeTree is +wrong. "The index is binary-searched" is true only for one class of predicate. +`MergeTreeDataSelectExecutor::markRangesFromPKRange` +(`MergeTreeDataSelectExecutor.cpp:1725`, called at `:189` and `:1070`) branches on +`key_condition.matchesExactContinuousRange()` at `:2131`: + +**Case A — the predicate is one continuous key interval** (`user_id = 42`, +`user_id BETWEEN 10 AND 20` on `ORDER BY (user_id, …)`). Binary search for the left and +right endpoints. The code says so at `:2180-2182`: "In case when SELECT's predicate defines +a single continuous interval of keys, we can use binary search algorithm to find the left +and right endpoint key marks of such interval. The returned value is the minimum range of +marks, containing all keys for which KeyCondition holds." It is tagged +`SearchAlgorithm::BinarySearch` at `:2184`. + +**Case B — anything else** (`user_id % 2 = 0`; or a predicate on the *second* key column +only). Binary search is meaningless because the qualifying granules are not contiguous, so +ClickHouse runs a **generic exclusion search** (`:2136-2176`): recursively split each mark +range into `merge_tree_coarse_index_granularity` subranges — default **8**, +`src/Core/Settings.cpp:1593` — and discard any subrange in which the condition provably +cannot be true. ``` - granule = 8192 rows (index_granularity) - mark = (offset_in_compressed_file, offset_in_decompressed_block) - one mark per granule per column +// ILLUSTRATION — not quoted from ClickHouse/ClickHouse; this is the shape of the two +// branches in MergeTreeDataSelectExecutor.cpp:2131-2200 rendered in Rust. The real +// binary-search loop starts at :2190; the exclusion search is delegated to +// genericExclusionSearch() at :2156 (src/Storages/MergeTree/GenericExclusionSearch.h). +fn mark_ranges(idx: &[Key], cond: &KeyCondition) -> Vec { + if cond.matches_exact_continuous_range() { // :2131 + // one interval: find its two endpoints, return the minimal covering range + let lo = lower_bound(idx, cond.left()); + let hi = upper_bound(idx, cond.right()); + vec![MarkRange { begin: lo, end: hi }] // :2184 BinarySearch + } else { + // no interval to bracket: split coarsely and drop what cannot match + generic_exclusion_search(idx, cond, /* coarse = */ 8) // :2156 + } +} ``` -The bet behind the number: decompressing and scanning 8192 rows with -vectorized code costs microseconds, so it is never worth tracking -anything finer. Why it matters: every read-side structure now scales -with `rows / 8192` — three orders of magnitude smaller than the data. - -### Step 4 — the sparse primary index: which granules, not which row - -The **sparse primary index** (`primary.idx`) stores only the sort key -of the FIRST row of each granule — one entry per 8192 rows, so it's -8192× smaller than the key column and always stays in memory. A -predicate on the key becomes two binary searches over this array, -producing a *range of granules* to read: - -```rust -// primary_idx[g] = ORDER BY key of granule g's FIRST row — 8192x -// smaller than the data, always in memory -fn mark_range(primary_idx: &[Key], lo: &Key, hi: &Key) -> Range { - let first = primary_idx.partition_point(|k| k < lo).saturating_sub(1); - let last = primary_idx.partition_point(|k| k <= hi); - first..last // for each granule: seek marks[g].compressed_offset, -} // decompress the block, skip to row — then just scan +Two details worth carrying away, because they are the difference between the idea and a +production implementation: + +- The exclusion search has a **step budget**, + `merge_tree_generic_exclusion_search_max_steps` (`Settings.cpp:1600`, default `0` = + unlimited). Its doc at `:1603` is unusually candid about the trade: "When it is + exhausted, the ranges that were not fully analyzed are **accepted as a whole**, so the + query stays correct but may read more granules than an unlimited search would select." + Index analysis is itself a cost centre that can be capped, and the failure mode is + reading too much, never reading too little. +- Surviving ranges that are close together get **fused**, because a seek costs more than + reading through the gap. `min_marks_for_seek` is computed at `:2143` from + `merge_tree_min_rows_for_seek` / `merge_tree_min_bytes_for_seek` + (`Settings.cpp:1579`, `:1586`, both default `0`): "If the distance between two data + blocks to be read in one file is less than … then ClickHouse does not seek through the + file but reads the data sequentially." + +### Step 6 — Marks: two offsets, because compression blocks ≠ granules + +> **In:** a `MarkRange` from Step 5 and a column file. +> **Out:** a byte position to seek to, and the reason it takes two numbers rather than one. + +Column files are a sequence of independently compressed blocks. Block boundaries are chosen +by **size**, granule boundaries by **rows**, and the two grids do not align — so a mark +carries two coordinates: + +```c +// ClickHouse/ClickHouse@4d598fb2c — src/Formats/MarkInCompressedFile.h:14-21 + 14 /** Mark is the position in the compressed file. The compressed file consists of adjacent compressed blocks. + 15 * Mark is a tuple - the offset in the file to the start of the compressed block, the offset in the decompressed block to the start of the data. + 16 */ + 17 struct MarkInCompressedFile + 18 { + 19 size_t offset_in_compressed_file; + 20 size_t offset_in_decompressed_block; + 21 ``` -``` - WHERE user_id = 42 (ORDER BY user_id): - primary.idx: [1, 800, 1600, ...] -> binary search -> granules 3..4 - read marks[3..4] per needed column -> decompress ~16K rows, scan them +Read: seek to `offset_in_compressed_file`, decompress that block, then skip +`offset_in_decompressed_block` bytes to reach the granule's first row. + +The block sizes come from `src/Core/Settings.cpp`: `min_compress_block_size` = **65,536** +(`:108`) and `max_compress_block_size` = **1,048,576** (1 MiB, `:123`). The `:108` doc +contains the worked example that makes the two-offset design obvious, and it is worth +quoting because it *is* the arithmetic: + +> "We are writing a UInt32-type column (4 bytes per value). When writing 8192 rows, the +> total will be 32 KB of data. Since `min_compress_block_size` = 65,536, a compressed block +> will be formed for **every two marks**." +> +> "We are writing a URL column with the String type (average size of 60 bytes per value). +> When writing 8192 rows, the average will be slightly less than 500 KB of data. Since this +> is more than 65,536, a compressed block will be formed **for each mark**." + +So on the `UInt32` column, every odd-numbered granule starts 32,768 bytes into its block — +`offset_in_decompressed_block = 32768` — and one offset could not have found it. On the URL +column, every mark's second offset is 0. That is the whole story, and it is also the +concrete cost of layering block compression under a row-addressed index. Parquet grows the +same two-level addressing for the same reason (see `reading-arrow-parquet.md`). + +Then there is a detail that rewards reading the header to the end. The in-memory array of +marks is **itself compressed with the schemes this topic is about**: + +```c +// ClickHouse/ClickHouse@4d598fb2c — src/Formats/MarkInCompressedFile.h:51-63 + 51 /** We need to store a sequence of marks, each consisting of two 64-bit integers: + 52 * offset_in_compressed_file and offset_in_decompressed_block. We'll call them x and y for + 53 * convenience, since compression doesn't care what they mean. The compression exploits the + 54 * following regularities: + 55 * * y is usually zero. + 56 * * x usually increases steadily. + 57 * * Differences between x values in nearby marks usually fit in much fewer than 64 bits. + 58 * + 59 * We split the sequence of marks into blocks, each containing MARKS_PER_BLOCK marks. + 60 * (Not to be confused with data blocks.) + 61 * For each mark, we store the difference [value] - [min value in the block], for each of the + 62 * two values in the mark. Each block specifies the number of bits to use for these differences + 63 * for all marks in this block. ``` -Sparse = you always over-read up to a granule; the bet is that -decompress+scan of 8192 rows is cheap (vectorized) and the index stays -resident. A B-tree answers "which row"; this answers "which 8192 -rows". Why it matters: for a scan workload the over-read is noise, and -in exchange the entire index for a 10-billion-row table is ~1.2M -entries — RAM-resident forever. - -### Step 5 — marks: two offsets, because compression blocks ≠ granules - -Column files are stored as a sequence of independently compressed -blocks, and a granule's rows can start in the *middle* of one — so a -mark must carry two coordinates: seek to `offset_in_compressed_file`, -decompress that block, then skip `offset_in_decompressed_block` bytes -to reach the granule's first row. One offset can't work because -compression block boundaries are chosen by size (~64 KB–1 MB), not by -row count, and the two grids don't align. Why it matters: it's the -concrete cost of layering block compression under a row-addressed -index — every format that compresses in blocks (Parquet pages, next -chapter) grows the same two-level addressing. - -### Step 6 — merges: the metabolic cycle, and work done during them - -Background merges continuously take several parts and merge-sort them -into one bigger part — the LSM compaction — steering between two -failure modes: merge too eagerly and you rewrite the same rows over -and over (write amplification); too lazily and scans must visit too -many parts (read amplification). Topic 4's dial, at part granularity. - -The distinctly ClickHouse move: since a merge already streams every -row through memory, **do other work while you're there**. Specialized -engines run computation inside the merge — `ReplacingMergeTree` dedups -rows, `SummingMergeTree` / `AggregatingMergeTree` pre-aggregate them — -compaction-as-computation, and the mechanism behind materialized views -(the paper's answer to "scans are still too slow": precompute during -ingest/merge). That's the architecture triangle: brute-force scan -speed (ClickHouse) vs precomputation (Pinot/Druid star-trees) vs -embedded convenience (DuckDB). It's also the trick FalkorDB could -steal for graph statistics. Why it matters: merge bandwidth is the -system's metabolism — background IO converted into query speed. - -### Step 7 — codec chains: the user declares, nothing analyzes - -Each column carries a declared chain of codecs that compose left to -right — `CODEC(Delta, LZ4)` means delta-encode (store differences from -the previous value), then LZ4 the result. The menu includes -time-series specials: DoubleDelta (deltas of deltas — near-zero for -regular timestamps), Gorilla (XOR consecutive floats — sensor values -barely change), FPC, GCD, ALP (topic 30 material). Contrast the -previous chapter: ClickHouse makes YOU declare the chain (or takes the -default LZ4) — there is no analyze-and-score pass. Why it matters: -it's the third answer to "who picks the encoding" — user-declared -(ClickHouse) vs full-analyze (DuckDB) vs sampled (BtrBlocks) — and the -right answer depends on who knows the data's shape. +Subtract a per-block minimum, then bit-pack the residuals at a per-block width: that is +**frame-of-reference plus bit-packing**, the same pair of schemes Parquet and DuckDB apply +to user data, applied here to the index. The measured payoff is in the class comment at +`:38-41`: "~3 bytes/mark for integer columns, ~5 bytes/mark for string columns, ~0.3 +bytes/mark for trivial marks in auxiliary dict files of LowCardinality columns" — against +16 bytes for the naive two-`size_t` struct, a **5.3×** reduction on integer columns. For +the 10-billion-row table: 1,220,703 marks × 3 columns × 3 B = **11 MB** instead of 58.6 MB. + +### Step 7 — Merges: the metabolic cycle, and the work done during it + +> **In:** a growing pile of parts. +> **Out:** fewer, larger parts — plus, optionally, computed results. + +Background merges take several parts and merge-sort them into one bigger part. This is +topic 4's compaction dial at part granularity, steering between two failure modes: merge +too eagerly and you rewrite the same rows repeatedly (write amplification); too lazily and +every scan must visit too many parts (read amplification). + +`MergeTreeDataMergerMutator::selectPartsToMerge` +(`MergeTreeDataMergerMutator.cpp:272`) makes the choice. The knobs and their defaults, all +in `MergeTreeSettings.cpp`, turn the abstract dial into numbers: + +| Setting | Default | Line | What it bounds | +| --- | --- | --- | --- | +| `merge_selector_base` | 5.0 | `:860` | "Affects write amplification of assigned merges" | +| `max_parts_to_merge_at_once` | 100 | `:637` | fan-in of a single merge | +| `max_bytes_to_merge_at_max_space_in_pool` | 150 GiB | `:475` | largest merge ever attempted | +| `parts_to_delay_insert` | 1000 | `:886` | inserts get an artificial sleep past this | +| `parts_to_throw_insert` | 3000 | `:908` | inserts are **rejected** past this | + +The last two are the "too lazy" failure mode made concrete, and they are back-pressure, not +tuning: `:893-894` says ClickHouse "artificially executes `INSERT` longer (adds 'sleep') so +that the background merge process can merge parts faster than they are added". If merges +cannot keep up, ingest is throttled and then refused — the system chooses to stop accepting +writes rather than let read amplification grow without bound. + +Merges are also prioritised, and the comment at `MergeTask.h:78-82` explains the policy in +plain words: "A priority is simple - the lower the size of the merge, the higher priority. +So, if ClickHouse wants to merge some really big parts into a bigger part, then it will be +executed for a long time, because the result of the merge is not really needed immediately. +It is better to merge small parts as soon as possible." The `MergeTask` class itself is at +`MergeTask.h:84`; it is a resumable state machine so a huge merge can be suspended between +blocks. + +The distinctly ClickHouse move: since a merge already streams every row through memory, +**do other work while you are there**. Specialized engines run computation inside the +merge — `ReplacingMergeTree` deduplicates rows, `SummingMergeTree` and +`AggregatingMergeTree` pre-aggregate them. Compaction-as-computation, and the mechanism +behind incrementally maintained materialized views. That is the architecture triangle: +brute-force scan speed (ClickHouse) versus precomputation (Pinot/Druid star-trees) versus +embedded convenience (DuckDB) — and it is the trick FalkorDB could steal for graph +statistics. + +### Step 8 — Codec chains: the user declares, nothing analyzes + +> **In:** a column and a `CODEC(...)` clause in the DDL. +> **Out:** a chain of transforms applied left to right, and a self-describing header that +> lets the reader undo them. + +Each column carries a declared chain of codecs. `CODEC(Delta, LZ4)` means delta-encode +(**delta encoding**: store each value's difference from the previous one), then LZ4 the +result. Composition is literal — `CompressionCodecMultiple::doCompressData` +(`src/Compression/CompressionCodecMultiple.cpp:44-67`) loops the codecs in order, swapping +its input and output buffers each time (`:60-61`), and writes a header of +`[codec count][method byte × N]` before the data (`:49`, `:55`, `:64`). Decompression walks +the same list **backwards** — `for (int idx = compression_methods_size - 1; idx >= 0; +--idx)` at `:86`. + +The overhead is exactly `1 + codecs.size()` bytes per compressed block (`:66`). On a +64 KiB block with a two-codec chain that is 3 bytes, or **0.0046%** — the reason nobody +thinks about it. + +The menu (`src/Compression/`) includes the general-purpose codecs — LZ4 +(`CompressionCodecLZ4.cpp`, and the default: `CompressionFactory.cpp:257` does +`default_codec = get("LZ4", {})`) and ZSTD — plus a set of *preprocessors* that shrink +nothing by themselves: `Delta`, `DoubleDelta` (deltas of deltas — near-zero for regular +timestamps), `Gorilla` (XOR consecutive floats — sensor values barely change), `FPC`, +`GCD`, `T64`, and `ALP` (topic 30 material). + +"Preprocessor" is the codebase's own word, not a gloss. `CompressionCodecDelta.cpp:30` +declares `bool isCompression() const override { return false; }`, and `:36` returns the +description "Preprocessor (should be followed by some compression codec). Stores difference +between neighboring values; good for monotonically increasing or decreasing data." This is +the exact same idea as Parquet's `BYTE_STREAM_SPLIT`, which also does not shrink anything — +both exist to make the *next* stage's job easier. + +**Why it matters:** this is the third answer to "who picks the encoding". ClickHouse makes +*you* declare the chain (or takes the LZ4 default); DuckDB analyses every column +(`reading-duckdb-compression.md`); BtrBlocks samples 1% of each block +(`reading-btrblocks-fsst.md`). Which is right depends entirely on who knows the data's +shape, and each system's answer is really a statement about who its users are. + +--- ## Where each step lives in the code -All under `src/Storages/MergeTree/` unless noted; a fresh shallow -clone is enough. - -- **Step 3** — `index_granularity = 8192`: `MergeTreeSettings.cpp:70`. -- **Step 4** — the in-memory index: `IMergeTreeDataPart.h:424` - `getIndex` / `:425` `loadIndexToCache`. Predicate → granule ranges: - `MergeTreeDataSelectExecutor::markRangesFromPKRange` (`:1725`, used - at `:189`) — turns a predicate into a list of `MarkRanges` to read. -- **Step 5** — the two-offset mark: - `src/Formats/MarkInCompressedFile.h:17`. -- **Step 6** — merge selection: - `MergeTreeDataMergerMutator::selectPartsToMerge` (`:272`) — the - heuristics balancing write amplification vs part count — plus - `MergeTask.h:84`. The specialized engines (ReplacingMergeTree, - AggregatingMergeTree, SummingMergeTree) are siblings in the same - directory. -- **Step 7** — codec chains: `src/Compression/`, composition in - `CompressionCodecMultiple.cpp`. - -Read order: settings → `markRangesFromPKRange` (the read path is the -payload) → `MarkInCompressedFile.h` → merge selection → codecs. The -design rationale behind all of it is the VLDB '24 paper — -[reading-clickhouse-paper.md](reading-clickhouse-paper.md), read after -this code walk. +All paths relative to the repo root of **ClickHouse/ClickHouse@4d598fb2c**. + +| Step | Anchor | What you will see | +| --- | --- | --- | +| 2 | `src/Storages/MergeTree/MergeTreeDataPartCompact.h:8-16` | the Compact layout, in a comment | +| 2 | `src/Storages/MergeTree/MergeTreeSettings.cpp:34`, `:76` | `default_min_bytes_for_wide_part` = 10 MiB | +| 3 | `src/Storages/MergeTree/MergeTreeSettings.cpp:70` | `index_granularity` = 8192 — note "Maximum" at `:71` | +| 3 | `src/Storages/MergeTree/MergeTreeSettings.cpp:1676`, `:1681`, `:1714` | the 10 MiB byte cap and adaptive granularity | +| 4 | `src/Storages/MergeTree/IMergeTreeDataPart.h:424`, `:425` | `getIndex()` / `loadIndexToCache()` | +| 5 | `src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:1725` | `markRangesFromPKRange`, used at `:189` and `:1070` | +| 5 | same file, `:2131`, `:2136-2176`, `:2178-2200` | the two search algorithms and the choice between them | +| 5 | `src/Core/Settings.cpp:1593`, `:1600`, `:1579`, `:1586` | coarse granularity 8, step budget, seek thresholds | +| 6 | `src/Formats/MarkInCompressedFile.h:14-21` | the two-offset mark | +| 6 | same file, `:38-41`, `:51-67` | FOR + bit-packing applied to the marks themselves | +| 6 | `src/Core/Settings.cpp:108`, `:123` | compression block sizes, with a worked example in the doc | +| 7 | `src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp:272` | `selectPartsToMerge` | +| 7 | `src/Storages/MergeTree/MergeTask.h:78-82`, `:84` | merge priority, and the resumable task | +| 7 | `src/Storages/MergeTree/MergeTreeSettings.cpp:475`, `:637`, `:860`, `:886`, `:908` | the merge and back-pressure limits | +| 8 | `src/Compression/CompressionCodecMultiple.cpp:44-67`, `:86` | chain composition, and undoing it in reverse | +| 8 | `src/Compression/CompressionCodecDelta.cpp:30`, `:36` | a codec that admits it is not a compressor | +| 8 | `src/Compression/CompressionFactory.cpp:257` | LZ4 is the default | + +**Read order:** `MergeTreeSettings.cpp:70` and `:1676` (the two granule caps) → +`markRangesFromPKRange` (the read path is the payload; skim to `:2131` and read both +branches) → `MarkInCompressedFile.h` end to end, it is 156 lines → +`selectPartsToMerge` → `CompressionCodecMultiple.cpp`. `ReplacingMergeTree`, +`SummingMergeTree` and `AggregatingMergeTree` are siblings in +`src/Storages/MergeTree/` if you want to see compaction-as-computation. + +Use `tools/pinned-source.py grep` and `show -r A:B` rather than cloning; whole-file `show` +on this repository will bury you. + +--- + +## Work the numbers yourself + +Do this before opening the code. All four calculations use one table: **10 billion rows**, +`ORDER BY user_id` (a `UInt64`), and a query that reads three `UInt64` columns. + +**1. The index.** `10e9 / 8192` = **1,220,703 granules**. `primary.idx` = 1,220,703 × 8 B += **9.8 MB**, resident forever. Dense alternative: 10e9 × 8 B = **80 GB**. Ratio: 8192×, +by construction. + +**2. The marks.** Three columns × 1,220,703 granules = 3,662,109 marks. Naive +(`2 × size_t`) = 58.6 MB; at the measured ~3 B/mark for integer columns +(`MarkInCompressedFile.h:39`) = **11 MB**. Note that marks scale with *columns × granules*, +so a 200-column table has 244 million marks — which is why compressing them mattered enough +to write a bespoke scheme. + +**3. The point-query over-read.** `WHERE user_id = 42` matching exactly one row still +decompresses one full granule of each of the three columns: `8192 × 8 B × 3` = +**196,608 bytes** = 192 KiB, to return 24 bytes of payload. Read amplification **8192×**. +For a scan of a billion rows that ratio is invisible; for an OLTP workload doing 100,000 +point lookups per second it is 19.7 GB/s of pure waste. Same number, opposite verdict — +which is the entire argument for why OLTP and OLAP engines cannot be the same engine. + +**4. Which GB/s?** `FINDINGS.md` row 12 records this topic's measured scan floor: **24–57 +GB/s on a machine with roughly 150 GB/s of memory bandwidth**. Those are *logical* bytes — +values processed after decoding. A MergeTree scan of an LZ4'd column at, say, 4× compression +reads a quarter as many bytes off disk as it processes, so the same scan can report "40 +GB/s" (logical) while moving 10 GB/s of real traffic, or report "10 GB/s" (physical) for +identical work. Both numbers are correct; neither is meaningful alone. The discipline is to +say which bytes you counted, every time — and to sanity-check against the hardware, since +`FINDINGS.md` row 12 also preserves this topic's own **19,047,619 GB/s**, printed by a +hoisted timing loop, which is about **127,000×** the machine's peak bandwidth and therefore +impossible on its face. An implausible bandwidth is a bug in the benchmark, not a discovery. + +--- ## Questions for notes.md -1. Sparse index over-read: worst case rows decompressed for a point - query with granularity 8192 and a 3-column read? Why is that fine - here and fatal for OLTP? -2. Two offsets per mark: why can't it be one? (Compression block - boundaries ≠ granule boundaries.) -3. ORDER BY choice: `(user_id, ts)` vs `(ts, user_id)` — which queries - does each serve, and what happens to zone maps on the second column? - (Same clustering lesson as DuckDB zone maps, but declared upfront.) -4. Merge heuristics: what goes wrong with too-eager merging - (write amp) vs too-lazy (read amp)? Topic 4's leveled-vs-tiered, at - part granularity. -5. M12/M22: FalkorDB stores matrices per relationship type. What's the - "part" equivalent if property columns become mergeable segments — - and could a merge pre-aggregate degree stats the way - SummingMergeTree does? +1. **Sparse index over-read.** Worst case rows decompressed for a point query with + granularity 8192 and a 3-column read? Compute it in bytes for `UInt64` columns, then + redo it for a table whose rows average 4 KiB (where `index_granularity_bytes` binds + instead — `MergeTreeSettings.cpp:1676`). Why is that fine here and fatal for OLTP? +2. **Two offsets per mark: why can't it be one?** Use the `min_compress_block_size` worked + example at `src/Core/Settings.cpp:109-117`: for a `UInt32` column at 8192 rows/granule, + which marks have a non-zero `offset_in_decompressed_block`, and what is its value? Then + explain why `MarkInCompressedFile.h:55` can say "y is usually zero". +3. **`ORDER BY (user_id, ts)` vs `(ts, user_id)`** — which queries does each serve? Now + connect it to Step 5: `Settings.cpp:1601` says the generic exclusion search runs "when + it uses key columns other than the first one". Which of your two orderings forces which + algorithm, for `WHERE ts > now() - 1h`? +4. **Merge heuristics.** What goes wrong with too-eager merging (write amp) versus too-lazy + (read amp)? Put numbers on the lazy end using `parts_to_delay_insert` = 1000 and + `parts_to_throw_insert` = 3000 (`MergeTreeSettings.cpp:886`, `:908`): what does a client + observe as the part count crosses each? Compare with topic 4's leveled-vs-tiered dial. +5. **M12/M22.** FalkorDB stores matrices per relationship type. What is the "part" + equivalent if property columns become mergeable segments — and could a merge + pre-aggregate degree statistics the way `SummingMergeTree` does? Say what you would give + up (hint: the same thing ClickHouse gave up in Step 1). + +--- + +## Takeaway + +MergeTree is what a storage engine looks like when you accept that you will never do a fast +point read and optimise everything else without that constraint. Sorted immutable parts +make ingest sequential; an index at 1/8192 resolution stays in RAM at any table size; +granules make the read quantum big enough that vectorized code amortises every per-call +cost; two-offset marks pay the small, unavoidable tax for layering block compression +underneath; merges are both garbage collection and a compute opportunity; and codecs are +declared by the person who actually knows the data. Each decision is legible only in terms +of the workload — which is the general lesson worth carrying to the next system. + +--- ## Done when -You can draw part → granule → mark → compressed block, walk a point -query through the sparse index, and name what ClickHouse traded away -(point reads, in-place updates) for scan throughput. +Answer each before unfolding it. + +- [ ] Someone tells you a MergeTree granule is 8192 rows. When are they wrong, and what + makes them wrong? + +
Answer + +Whenever the rows are wide. `index_granularity` = 8192 is a **maximum** — its own doc +string at `MergeTreeSettings.cpp:71` reads "Maximum number of data rows between the marks +of an index". A second cap, `index_granularity_bytes`, defaults to 10 MiB +(`:1676`), and adaptive granularity is on by default via `enable_mixed_granularity_parts` +(`:1714`). A granule is `min(8192 rows, ~10 MiB)`. + +Concretely: at 100 B/row the row cap binds (8192 rows ≈ 800 KiB). At 4 KiB/row the byte cap +binds and a granule is 2560 rows. At 1 MiB/row it is 10 rows. Without the byte cap, a +"granule" of 8192 × 1 MiB would be an 8 GB read quantum. + +
+ +- [ ] `WHERE user_id = 42` and `WHERE user_id % 2 = 0` on `ORDER BY user_id` take different + code paths through the index. Name both and say what determines the choice. + +
Answer + +The switch is `key_condition.matchesExactContinuousRange()` at +`MergeTreeDataSelectExecutor.cpp:2131`. + +`user_id = 42` is a single continuous key interval, so ClickHouse binary-searches for the +left and right endpoint marks (`:2178-2200`, tagged `SearchAlgorithm::BinarySearch` at +`:2184`). The comment at `:2180-2182` describes exactly this. + +`user_id % 2 = 0` is not an interval — qualifying granules are scattered — so there is +nothing to bracket. ClickHouse runs a **generic exclusion search** (`:2136-2176`), +recursively splitting each mark range into `merge_tree_coarse_index_granularity` subranges +(default 8, `src/Core/Settings.cpp:1593`) and discarding subranges where the condition +provably cannot hold. It is bounded by `merge_tree_generic_exclusion_search_max_steps` +(`Settings.cpp:1600`); when the budget runs out, unanalysed ranges are "accepted as a +whole" (`:1603`) — correct, but reading more granules than necessary. + +
+ +- [ ] A mark is two 64-bit offsets = 16 bytes. A 200-column table with 10 billion rows has + 244 million marks. Why is that not 3.9 GB of RAM? + +
Answer + +Because the in-memory mark array is compressed with this topic's own schemes. +`MarkInCompressedFile.h:51-63` lists the regularities it exploits: `y` (the offset within +the decompressed block) is usually zero, `x` (the offset in the file) increases steadily, +and differences between neighbouring `x` values fit in far fewer than 64 bits. So marks are +split into fixed-size blocks; each block stores the per-block minimum of `x` and `y` and +then the residuals bit-packed at a per-block width — **frame-of-reference plus +bit-packing**, exactly what Parquet and DuckDB apply to user data. + +Measured result, from the class comment at `:38-41`: ~3 bytes/mark for integer columns, +~5 for string columns, ~0.3 for trivial marks in LowCardinality dictionary files. At 3 +B/mark the 244 million marks cost ~732 MB rather than 3.9 GB, and random access is still +O(1) (`:33-34`). + +
+ +- [ ] Ingest outruns merges. Trace what a client sees, in order, and say what ClickHouse is + protecting. + +
Answer + +Active parts in one partition accumulate. At **1000** (`parts_to_delay_insert`, +`MergeTreeSettings.cpp:886`) inserts are artificially slowed — the doc at `:893-894` says +ClickHouse "adds 'sleep' so that the background merge process can merge parts faster than +they are added". At **3000** (`parts_to_throw_insert`, `:908`) inserts are rejected +outright. + +It is protecting read amplification. Every query must consult every active part's index and +merge their outputs, so query cost grows with part count. Rather than let scans degrade +without bound, the engine converts the problem into back-pressure on writers — a +deliberate choice to fail the ingest path loudly instead of the query path quietly. This is +topic 4's write-vs-read amplification dial with the thresholds written down. + +
+ +- [ ] ClickHouse's `Delta` codec and Parquet's `BYTE_STREAM_SPLIT` both shrink nothing. + Why does either exist? + +
Answer + +Because they are **preprocessors** for the stage that follows. ClickHouse says so in its own +type system: `CompressionCodecDelta.cpp:30` is `bool isCompression() const override +{ return false; }`, and the description at `:36` is "Preprocessor (should be followed by +some compression codec)." + +`Delta` turns a monotonically increasing sequence into a run of small, similar numbers, +which LZ4 or ZSTD can then encode in far fewer bytes. `BYTE_STREAM_SPLIT` scatters each +double's 8 bytes into 8 separate streams so that the (highly repetitive) exponent bytes sit +next to each other. Same move, different axis: neither removes information, both increase +*local* similarity so a general-purpose compressor finds more matches. + +The practical consequence is that `CODEC(Delta)` alone is close to a no-op — the chain +mechanism (`CompressionCodecMultiple.cpp:44-67`) exists precisely so these can be composed +with a real compressor, and decompression undoes them in reverse order (`:86`). + +
+ +--- ## References **Papers** + - The VLDB '24 system paper gets its own chapter: - [reading-clickhouse-paper.md](reading-clickhouse-paper.md) — read it - after this code walk - -**Code** -- [ClickHouse](https://github.com/ClickHouse/ClickHouse) — - `src/Storages/MergeTree/` (the anchors above: - `MergeTreeSettings.cpp`, `IMergeTreeDataPart.h`, - `MergeTreeDataSelectExecutor.cpp`, - `MergeTreeDataMergerMutator.cpp`, `MergeTask.h`), - `src/Formats/MarkInCompressedFile.h`, and `src/Compression/` for the - codec chains; a fresh shallow clone is enough + [reading-clickhouse-paper.md](reading-clickhouse-paper.md) — read it after this code + walk; it supplies the *why* for every *what* above. + +**Code** — all at `ClickHouse/ClickHouse@4d598fb2c` + +- `src/Storages/MergeTree/` — `MergeTreeSettings.cpp`, `IMergeTreeDataPart.h`, + `MergeTreeDataPartCompact.h`, `MergeTreeDataSelectExecutor.cpp`, + `MergeTreeDataMergerMutator.cpp`, `MergeTask.h` +- `src/Formats/MarkInCompressedFile.h` — 156 lines, read all of it +- `src/Core/Settings.cpp` — the query-level knobs (`:108`, `:123`, `:1579`, `:1586`, + `:1593`, `:1600`) +- `src/Compression/` — `CompressionCodecMultiple.cpp`, `CompressionCodecDelta.cpp`, + `CompressionFactory.cpp` + +**In this topic** + +- [reading-arrow-parquet.md](reading-arrow-parquet.md) — the same two-level addressing + problem, solved in a file format +- [reading-duckdb-compression.md](reading-duckdb-compression.md) and + [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md) — the other two answers to "who + picks the encoding" +- `FINDINGS.md` row 12 — the measured scan floor (24–57 GB/s on a ~150 GB/s machine) and + the 19,047,619 GB/s hoisted-loop bug diff --git a/topics/12-columnar-analytics/reading-clickhouse-paper.md b/topics/12-columnar-analytics/reading-clickhouse-paper.md index 2096a61..f85b9d7 100644 --- a/topics/12-columnar-analytics/reading-clickhouse-paper.md +++ b/topics/12-columnar-analytics/reading-clickhouse-paper.md @@ -1,108 +1,362 @@ # ClickHouse: the case for brute force -The system paper, 15 years in — the design rationale behind the -mechanisms you just read in -[reading-clickhouse-mergetree.md](reading-clickhouse-mergetree.md), -plus the parts you didn't read code for (mutations, replication, -scaling). Read it AFTER the code guide, paired with a local -`clickhouse local` session and ClickBench. Before the paper, this -chapter builds the five arguments it makes — one at a time — because -its two-sentence thesis is this topic's strongest counterpoint to -index-everything instincts. +The system paper, fifteen years in: Robert Schulze, Tom Schreiber, Ilya Yatsishin, Ryadh +Dahimene, Alexey Milovidov, *ClickHouse — Lightning Fast Analytics for Everyone*, PVLDB +17(12): 3731–3744, 2024. + +Read it **after** [reading-clickhouse-mergetree.md](reading-clickhouse-mergetree.md): the +code guide shows you *what* the storage engine does, and this paper supplies the *why*, +plus the parts you did not read code for — pruning beyond the primary key, mutations, +replication, and the benchmark results. Every number below carries the section or figure +it came from. + +One vocabulary note, because the paper uses the word without defining it. A **projection** +in ClickHouse is an alternative copy of a table's rows sorted by a *different* key (§3.2) — +the same idea C-Store built its whole design on, which is why +[reading-cstore-compression.md](reading-cstore-compression.md) is worth reading beside +this. + +--- ## The problem in one sentence -If a vectorized engine scans compressed columns at multiple GB/s per -core, a query over a 100M-row table is a sub-second *scan* — so how -much of classical database machinery (per-row indexes, transactional -updates, row-level replication) should you simply refuse to build? +If a vectorized engine can scan compressed columns fast enough that a query over a +100-million-row table is a sub-second *scan*, how much of classical database machinery — +per-row indexes, transactional updates, row-level replication — should you simply refuse to +build? + +The paper does not frame itself as "brute force"; §1 lists **five key challenges** it claims +to address: (1) huge data sets with high ingestion rates, (2) many simultaneous queries +expecting low latency, (3) diverse data stores, locations and formats, (4) a convenient +query language with performance introspection, and (5) industry-grade robustness and +versatile deployment. Read the rest of the paper as five answers, and notice how many of +them are the same answer: *make the immutable sorted part the unit of everything*. + +--- ## The concepts, step by step -### Step 1 — the brute-force bet: make scanning cheap instead of avoiding it - -ClickHouse's founding bet is that with vectorization (topic 11) + -compression (this topic) + parallelism across all cores, scanning is -fast enough that you rarely need per-row indexes at all. Arithmetic: -16 cores × ~2 GB/s of decompressed scan throughput each ≈ 32 GB/s — -a 10-byte-per-row hot column over 1B rows scans in ~0.3 s *with no -index*. The sparse primary index (one key per 8192 rows, previous -chapter) prunes coarse ranges; from there it's bandwidth. Your -scan_bench measures exactly this bet in miniature. Why it matters: -every other argument in the paper is a consequence of refusing -per-row machinery — read them as corollaries, not separate features. - -### Step 2 — everything happens at merge time - -Because parts are immutable and background merges already stream every -row (previous chapter, Step 6), ClickHouse routes ALL maintenance work -through merges: TTL enforcement (expired rows dropped as merges -rewrite parts), dedup (`ReplacingMergeTree`), pre-aggregation -(`Summing`/`AggregatingMergeTree` — the substrate for materialized -views), recompression of cold parts to heavier codecs. Merges are the -system's metabolic cycle — background bandwidth converted into query -speed. (Topic 4's compaction-as-computation, fully weaponized.) Why it -matters: work that OLTP systems do per-write (and pay for in latency) -is batched into sequential IO the system was doing anyway — but it -makes merge bandwidth the resource everything competes for. - -### Step 3 — who picks the codec: the user, explicitly - -ClickHouse exposes per-column codec CHAINS — `CODEC(Delta, ZSTD)`, -Gorilla/DoubleDelta for time series — and makes the USER declare what -DuckDB's analyze pass discovers automatically. That completes the -three answers to "who chooses the encoding": user-declared -(ClickHouse — zero ingest cost, assumes the operator knows the data), -full-analyze (DuckDB — pays a pass, needs no knowledge), sampled -(BtrBlocks — the middle). Why it matters: it's the same performance -philosophy as `ORDER BY`-at-creation — ClickHouse consistently trades -operator burden for machine efficiency, and the paper is explicit -that this targets operators who profile. - -### Step 4 — updates are batch jobs, not transactions - -A **mutation** (`ALTER TABLE ... UPDATE/DELETE`) is executed by -asynchronously rewriting every affected part in the background — a -single-row update can rewrite gigabytes, and there is no -read-your-write guarantee on it. This is the honest scope statement: -immutable parts made ingest and scans fast (Steps 1–2), and this is -the bill — point updates became bulk jobs. Why it matters: this is -what "giving up OLTP" concretely means; when a vendor benchmark shows -ClickHouse-class scan numbers, this is the capability that was traded -for them. - -### Step 5 — replication ships parts, not rows - -Replicas coordinate through Keeper (their RAFT-ish ZooKeeper -replacement) on a shared log of *actions* — "part X was inserted", -"parts Y+Z merged into W" — and fetch whole part files from peers; -shards are shared-nothing on top. Contrast topic 15's menu: redis -ships commands, postgres ships WAL records, ClickHouse ships FILES — -state-machine replication at part granularity. The granularity is -acceptable precisely because parts are immutable (a fetched file is -never patched) and the workload tolerates second-scale replica lag. -Why it matters: replication design is downstream of the storage -design — immutability made the coarsest, simplest unit the right one. +### Step 1 — The bet: pruning is a menu, and the headline benchmark barely orders from it + +> **In:** a table of 100 million rows and a filter. +> **Out:** the three pruning techniques ClickHouse offers, and the measured fact that its +> best-known benchmark result uses almost none of them. + +§3.2 lists exactly three ways to avoid reading rows: + +1. **The sparse primary key index.** One entry per granule, locally clustered — the + structure you read code for. The paper's own sizing: "only 1000 entries are required to + index 8.1 million rows". Check it: `8,100,000 / 8192` = **988.8** entries. The number is + arithmetic, not a benchmark. +2. **Projections** — "alternative versions of a table that contain the same rows sorted by + a different primary key", which speed up queries filtering on a non-key column "at the + cost of an increased overhead for inserts, merges, and space consumption". Two details + make them affordable where C-Store's were not: they are "populated lazily only from + parts newly inserted into the main table" unless you materialize them in full, and "the + query optimizer chooses between reading from the main table or a projection based on + estimated I/O costs", falling back to the main table for any part that lacks one. +3. **Skipping indices** — metadata over *multiple consecutive granules*, with a + configurable number of granules per index block. Three types: **min-max** (a **zone map**: + the minimum and maximum of an index expression per block, good for "locally clustered + data with small absolute ranges"), **set** (a bounded number of distinct values per + block, for "clumped together" values), and **Bloom filter** (row, token or n-gram, with + a configurable false-positive rate — and, unlike the other two, "cannot be used for + range or negative predicates"). + +Now the part that makes this step worth its own place. §6.2.1 states how the ClickBench +results were produced: "The physical database design is tuned only lightly, for example, we +specify primary keys, but **do not** change the compression of individual columns, create +projections, or skipping indexes." + +So the benchmark ClickHouse wins is run with the sparse index and the default LZ4, and with +options 2 and 3 switched off. That is the brute-force claim, stated by the authors against +their own interest, and it is the strongest evidence in the paper for the thesis of this +topic: if scanning is cheap enough, most index machinery is optional. + +### Step 2 — What "cheap enough" costs, on the paper's own hardware + +> **In:** the ClickBench setup in §6.2.1. +> **Out:** the actual bandwidth ceiling those numbers were measured against, and why +> "GB/s" is ambiguous until you say which bytes. + +§6.2.1 gives the machine: a single-node AWS EC2 **c6a.4xlarge** — **16 vCPUs, 32 GB RAM, +5000 IOPS / 1000 MiB/s disk** — running **43 queries** against a table of **100 million** +anonymized page hits, with the Linux page cache flushed before each *cold* run. + +Read those three numbers together, because they decide everything: + +| Resource | Ceiling | Per vCPU | +| --- | --- | --- | +| Disk | 1000 MiB/s ≈ **1.05 GB/s** | 65.5 MB/s | +| RAM | 32 GB — smaller than many ClickBench columns uncompressed | — | +| Cores | 16 | — | + +A cold query on that instance cannot exceed **~1 GB/s of compressed bytes off disk**, no +matter how many cores are scanning. `FINDINGS.md` row 12 records this topic's measured +in-memory scan floor at **24–57 GB/s on a machine with roughly 150 GB/s of memory +bandwidth** — between **24× and 57×** the c6a.4xlarge's disk rate. So on a cold run the +scan engine is idle most of the time and compression ratio is the only lever that matters: +at a 5× ratio, 1.05 GB/s of disk feeds 5.25 GB/s of logical rows; at 10×, 10.5 GB/s. On a +hot run the page cache removes the disk entirely and the memory-bandwidth story of +`FINDINGS.md` row 12 takes over. This is why the paper reports **cold and hot geometric +means separately** (Figure 10) — they are measuring two different machines. + +The discipline this forces is the one this whole topic is about: **say which bytes you +counted.** "1 GB/s" (compressed, off disk) and "5 GB/s" (logical, after decoding) can +describe the identical query. `FINDINGS.md` row 12 also preserves this topic's own +cautionary figure — a hoisted timing loop once printed **19,047,619 GB/s**, roughly +**127,000×** the machine's peak memory bandwidth. A throughput that exceeds the hardware is +never a discovery; it is a bug in the measurement. + +### Step 3 — Not an LSM hierarchy: all parts are equal, and there is no WAL + +> **In:** an `INSERT`. +> **Out:** a part on disk, and two deliberate departures from the LSM designs of topic 4. + +§3.1 says a part "is created whenever a set of rows is inserted", parts are +"self-contained… include all metadata required to interpret their content without +additional lookups to a central catalog", and merges continue "until a configurable part +size is reached (**150 GB** by default)" — the same number as +`MergeTreeSettings.cpp:475`'s `max_bytes_to_merge_at_max_space_in_pool`. + +Two sentences in §3.1 are the ones to underline, because both contradict the LSM mental +model you brought from topic 4: + +- **"ClickHouse treats all parts as equal instead of arranging them in a hierarchy. As a + result, merges are no longer limited to parts in the same level."** No L0/L1/L2. The + paper immediately names the price: "Since this also forgoes the implicit chronological + ordering of parts, alternative mechanisms for updates and deletes not based on tombstones + are required (see Section 3.4)." A tombstone only works if you can tell which record is + newer, and a flat pile of parts cannot. Step 5 is the consequence. +- **"ClickHouse writes inserts directly to disk while other LSM-tree-based stores typically + use write-ahead logging."** And §3.7 completes it: ClickHouse does "not forcing a commit + (`fsync`) of newly inserted parts to disk by default, allowing the kernel to batch writes + at the cost of forgoing atomicity", justified because "most of ClickHouse's write-heavy + decision making use cases even tolerate a small risk of losing new data in case of a + power outage". + +Also worth having in hand: clients "are encouraged to insert tuples in bulk, e.g. 20,000 +rows at once", and an **asynchronous insert mode** exists that buffers rows from many +`INSERT`s server-side and forms a part on a size or time threshold — the answer to +"thousands of monitoring agents continuously sending small amounts of event data". + +§3.5 adds a nice piece of engineering economy: idempotent inserts, implemented by keeping +"hashes of the N last inserted parts (e.g. **N=100**)" and ignoring re-inserts of a known +hash. The paper explicitly contrasts this with per-tuple uniqueness indexes, whose "space +and update overhead becomes prohibitive for large data sets and high ingest rates". A +100-entry hash set replaces a billion-key index because the *part*, not the row, is the +unit of identity. + +### Step 4 — Everything happens at merge time + +> **In:** a merge that is already streaming every row through memory. +> **Out:** a list of maintenance jobs that ride along for free, and the guarantee they give +> up. + +Because parts are immutable and merges already touch every row, ClickHouse routes +maintenance through them (§3.3): + +| Merge strategy | What it does | +| --- | --- | +| **Replacing** | keeps only the newest version of a tuple (by containing part's creation timestamp, or an explicit version column); "commonly used as a merge-time update mechanism" | +| **Aggregating** | collapses rows with equal primary key values into a **partial aggregation state** — e.g. a sum and a count for `avg()` — combined pairwise as merges proceed | +| **TTL** | processes one part at a time; actions are: move the part to another volume, **re-compress** it with a heavier codec, delete it, or roll it up by aggregating | + +Aggregating merges are the substrate for materialized views, and the paper is precise about +what makes them different from everyone else's: "Unlike other databases, ClickHouse does +not refresh materialized views periodically with the entire content of the source table. +Materialized views are rather updated **incrementally** with the result of the +transformation query when a new part is inserted into the source table." The `-State` / +`-Merge` function suffixes are the user-visible seam: `avgState()` emits a partial +aggregate, `avgMerge()` folds the partials into an answer. + +The honest limitation, stated in §3.3: "Merge-time data transformation does not compromise +the performance of `INSERT` statements, **but it cannot guarantee that tables never contain +unwanted (e.g. outdated or non-aggregated) values**. If necessary, all merge-time +transformations can be applied at query time by specifying the keyword `FINAL`." So +`ReplacingMergeTree` does not give you a unique key; it gives you a promise of eventual +deduplication, plus a `FINAL` escape hatch that pays the cost per query instead. + +**Why it matters:** work that OLTP systems do per write — and pay for in write latency — is +batched into sequential I/O the system was doing anyway. But it makes **merge bandwidth the +resource everything competes for**, which is the failure mode Step 7 of the code guide put +numbers on (`parts_to_delay_insert` = 1000, `parts_to_throw_insert` = 3000). + +### Step 5 — Updates are batch jobs, not transactions + +> **In:** an `ALTER TABLE … UPDATE` or `DELETE`. +> **Out:** two mechanisms, and the exact guarantee each provides. + +§3.4 opens by conceding the point — "The design of the MergeTree\* table engines favors +append-only workloads, yet some use cases require to modify existing data occasionally, +e.g. for regulatory compliance" — and then offers two mechanisms, "neither of which block +parallel inserts". + +**Mutations** "rewrite all parts of a table in-place". Read the guarantee carefully, +because it is easy to overstate in either direction: + +- They are **not atomic**: "to prevent a table (delete) or column (update) from doubling + temporarily in size, this operation is non-atomic, i.e. parallel `SELECT` statements may + read mutated and non-mutated parts." +- They *are* durable in the end: "Mutations guarantee that the data is physically changed + at the end of the operation." +- They are expensive in a specific way: "Delete mutations are still expensive as they + rewrite **all columns in all parts**" — an update touches one column's files, a delete + touches every column's. + +**Lightweight deletes** "only update an internal bitmap column", and ClickHouse "amends +`SELECT` queries with an additional filter on the bitmap column". Physical removal waits for +"regular merges at an unspecified time in future". The trade is stated plainly: "Depending +on the column count, lightweight deletes can be much faster than mutations, at the cost of +slower `SELECT`s." + +And the scope statement that should end any argument about using ClickHouse for OLTP: +"Update and delete operations performed on the same table are expected to be **rare and +serialized** to avoid logical conflicts." + +§3.7 completes the picture: queries run against a snapshot of all parts taken at query +start, with reference counting to keep them alive — "formally, this corresponds to snapshot +isolation realized by an MVCC variant based on versioned parts" — but "statements are +generally **not ACID-compliant** except for the rare case that concurrent writes at the time +the snapshot is taken each affect only a single part." + +**Why it matters:** this is what "giving up OLTP" concretely means. When a vendor benchmark +shows ClickHouse-class scan numbers, this paragraph is the capability that was traded for +them. + +### Step 6 — Replication ships state transitions, and sometimes recomputes instead + +> **In:** a cluster of nodes and a stream of local operations. +> **Out:** a replicated table, and the granularity choice that made it simple. + +§3.6: replication is based on **table states**, "which consist of a set of table parts and +table metadata". Nodes advance a state with exactly three operations — inserts add a part; +merges add one and delete several; mutations and DDL add, delete and/or change metadata. +Each is "performed locally on a single node and recorded as a sequence of state transition +in a global replication log". + +The log lives in **ClickHouse Keeper** — "typically three" processes, using the **Raft** +consensus algorithm, described in §2 as "a drop-in replacement for Apache Zookeeper written +in C++" and coordinating a multi-master scheme. Replicas replay the log **asynchronously**, +so "replicated tables are only eventually consistent, i.e. nodes can temporarily read old +table states while converging towards the latest state" — though operations can optionally +run synchronously until a quorum adopts the new state. + +Compare with topic 15's menu: Redis ships commands, Postgres ships WAL records, ClickHouse +ships a log of *part-level actions* plus the part files themselves. But there is a nuance +that the one-line version of this story drops. §3.6's three optimizations: + +1. New nodes do **not** replay the log from scratch — "they simply copy the state of the + node which wrote the last replication log entry." +2. "**Merges are replayed by repeating them locally or by fetching the result part from + another node.** The exact behavior is configurable and allows to balance CPU consumption + and network I/O. For example, cross-data-center replication typically prefers local + merges to minimize operating costs." So it is not always file shipping — the log entry + is a *description* of the transition, and each replica chooses whether to re-derive it + or download it. +3. Mutually independent log entries are replayed in parallel. + +**Why it matters:** replication design is downstream of storage design. Immutability is +what makes a coarse unit safe — a fetched part file is never patched afterwards — and +determinism is what makes "recompute instead of download" a legal substitution. Neither +option would exist if parts were mutable. + +### Step 7 — What the benchmarks actually show, including the losses + +> **In:** §6.2's three benchmark families. +> **Out:** where ClickHouse wins, where it does not, and what is actually responsible. + +**ClickBench** (§6.2.1): 43 queries, 100 million page hits, c6a.4xlarge, lightly tuned as +described in Step 1. Figure 10 compares cold and hot geometric means against MySQL, +PostgreSQL, Druid, Pinot, Umbra, Snowflake (size S) and Redshift (ra3.4xlarge). The result, +in the paper's own words: "**While the research database Umbra achieves the best overall +hot runtime**, ClickHouse outperforms all other production-grade databases for hot and cold +runtimes." Note the shape of that sentence — it is a claim about *production* systems, and +it concedes first place on hot runtime to a research system. + +**VersionsBench** (§6.2.1): four benchmarks (ClickBench; 15 MgBench queries; 13 queries on a +600-million-row denormalized Star Schema Benchmark fact table; 4 queries on 3.4 billion NYC +Taxi rides), run monthly across **77 releases from March 2018 to March 2024**. Result: +"The performance of VersionBench improved by **1.72×** over the past six years" — which is +`1.72^(1/6)` = **9.5% per year**, compounding, an honest and unglamorous number. The +biggest single jump, August 2022, "was caused by the column-by-column filter evaluation +technique described in Section 4.4": evaluating filters sequentially in descending estimated +selectivity so each predicate sees fewer rows, applied "only when at least one highly +selective predicate is present; otherwise, the latency of the query would deteriorate". + +**TPC-H at scale factor 100** (§6.2.2), on a c6i.16xlarge (64 vCPUs, 128 GB RAM) against +Snowflake at warehouse size L. This is where the losses live, and they are catalogued: + +| Outcome | Queries | Count | Reason given | +| --- | --- | --- | --- | +| Excluded | Q2, Q4, Q13, Q17, Q20, Q21, Q22 | 7 | correlated subqueries "which are not supported as of ClickHouse v24.6" | +| Excluded | Q7, Q8, Q9, Q19 | 4 | need join reordering and join predicate pushdown, "both missing as of ClickHouse v24.6" | +| Faster in ClickHouse | — | 5 | — | +| Faster in Snowflake | — | 6 | — | + +The arithmetic closes: `7 + 4` = **11 excluded**, `22 − 11` = **11 run**, `5 + 6` = 11. So +**half the benchmark could not be executed at all**, and the half that ran is a near tie. + +The important thing is the *cause*. Not the sparse index, not compression, not the scan — +the **query optimizer**: correlated-subquery decorrelation and join reordering, both +acknowledged as missing and both "planned for implementation in 2024". §6.2.2 opens by +saying so itself — "normalized tables are an emerging use case for ClickHouse". A system +built on "make scanning fast" has, predictably, the weaknesses of a system that did not +spend its first decade on join planning. + +### Step 8 — "For everyone": four deployment modes, one of them credited to DuckDB + +> **In:** the paper's title. +> **Out:** what "for everyone" actually names, and what it concedes. + +§2 lists four operating modes: **on-premise** (single server or sharded/replicated +cluster), **cloud** (ClickHouse Cloud, deferred to a follow-up paper), **standalone** — +"turns ClickHouse into a command line utility for analyzing and transforming files, making +it a SQL-based alternative to Unix tools like `cat` and `grep`" — and **in-process**, chDB, +"for interactive data analysis use cases like Jupyter notebooks with Pandas dataframes". + +The sentence to notice: "**Inspired by DuckDB**, chDB embeds ClickHouse as a +high-performance OLAP engine into a host process… this allows to pass source and result +data between the database engine and the application efficiently without copying as they +run in the same address space." + +A paper claiming a general-purpose analytics engine names a single-file embedded database +as the design it is copying for one whole deployment mode. That is not a threat to DuckDB's +niche; it is a citation of it — and it tells you the niche is real enough that the +big-cluster system had to grow into it. + +--- ## How to read the paper (with the concepts in hand) -1. **Architecture/storage sections** — skim; you read the code - (previous chapter). Confirm the part/granule/mark story matches. -2. **The performance discussion** — read as Step 1's bet: where do - they credit vectorization vs compression vs pruning? Note where - the sparse index is *not* the hero. -3. **Merge-time features** (TTL, Replacing/Summing/Aggregating, - recompression) — Step 2; list every job they route through merges. -4. **Codecs** — Step 3; note the time-series specials (Gorilla, - DoubleDelta) and what they assume about the data. -5. **Mutations** — Step 4; read for the honest limits, not the - mechanism. -6. **Replication/scaling** — Step 5; watch for what Keeper stores (log - of part actions, not data). -7. **Evaluation** — skim against your own ClickBench numbers (below); - vendor evals are hypotheses, yours are measurements. - -## The experiments to run alongside (this topic's "run something real") +Budget about two hours. Order: + +1. **§1** — the five challenges. Two pages, and the frame for everything else. +2. **§3.1–3.2** — skim; you read the code (previous chapter). Confirm the part/granule/mark + story matches, and note the three numbers that appear in both: 10 MB Compact-part + threshold, 1 MB block size, 150 GB maximum merged part. +3. **§3.3** — Step 4. List every job routed through merges. Six, if you count re-compression + and roll-up separately. +4. **§3.4, §3.7** — Steps 5. Read for the *limits*, not the mechanism. +5. **§3.6** — Step 6. Watch for what Keeper stores: a log of part actions, never data. +6. **§4.4** — the densest section in the paper. Its "Primary key index evaluation" + paragraph is the prose version of the two search algorithms you found in + `MergeTreeDataSelectExecutor.cpp` — "the range is split into sub-ranges which are + analyzed recursively" is `merge_tree_coarse_index_granularity` at + `src/Core/Settings.cpp:1593`. Also note the monotonicity and preimage tricks + (`toYear(k) = 2024` rewritten as `k >= 2024-01-01 && k < 2025-01-01`) and the "over 30" + hash table variants. +7. **§6.2** — Step 7. Read the setup paragraphs before the graphs; the tuning disclosure is + more informative than the bars. +8. **§5** (integration layer) and **§4.5** (workload isolation) — skim unless you have a + specific interest. + +--- + +## The experiments to run alongside + +This topic's "run something real". The point is to replace the paper's numbers with your +own on your own hardware. ```bash # duckdb + clickbench slice (see ../duckdb-clickbench.md notes file): @@ -112,43 +366,198 @@ design — immutability made the coarsest, simplest unit the right one. # record all of it in notes.md ``` +Two things to record beyond the runtimes, because they are what makes the numbers +interpretable: + +- **Your disk's sequential read rate and your RAM bandwidth.** The paper's instance is + capped at 1000 MiB/s (§6.2.1); yours is probably very different, and the cold/hot gap you + measure is a direct function of that ratio. +- **Which GB/s you are quoting.** Compressed bytes read, or logical rows processed? Write + both into `notes.md`. `FINDINGS.md` row 12 is this topic's reference for what the local + machine can actually do (24–57 GB/s against ~150 GB/s of memory bandwidth), and any + figure above that ceiling is a bug, not a result. + +--- + ## Questions for notes.md -1. The paper's own numbers: where does ClickHouse lose (or barely win) - on ClickBench-class queries, and is the cause ever the sparse index - (vs e.g. string handling)? -2. Merges do TTL/dedup/aggregation — what's the failure mode when merge - bandwidth can't keep up with ingest (too many parts)? Which topic 4 - stall mechanism is the analogue? -3. Part-shipping replication: what does it give up vs WAL shipping - (replication lag granularity, partial-part visibility) and why is - that acceptable for analytics? -4. User-declared codecs vs analyze-and-score vs sampling: which would - you ship for a GRAPH database where property columns arrive via - MERGE statements with unknown distributions? (M12 decision — commit - to one and note why.) -5. The "for everyone" claim: what did they add to serve small/embedded - use (chdb, clickhouse-local), and does it threaten DuckDB's niche or - validate it? +1. **Where does ClickHouse barely win — or lose?** Use §6.2.2's TPC-H table: which queries + were excluded and why, and of those that ran, what is the score against Snowflake? Is + the sparse index ever the named cause, or is it always something else? Name the two + optimizer features the paper admits are missing. +2. **Merge starvation.** Merges do TTL, dedup and aggregation (§3.3). What is the failure + mode when merge bandwidth cannot keep up with ingest? Put the code guide's thresholds on + it (`parts_to_delay_insert` = 1000, `parts_to_throw_insert` = 3000) and name the topic 4 + stall mechanism this is the analogue of. +3. **Part-shipping replication.** What does it give up versus WAL shipping — think + replication lag granularity and partial-part visibility — and why is that acceptable for + analytics? Then account for §3.6's second optimization: when a replica *recomputes* a + merge instead of fetching it, what property of merges is being relied on, and what would + break it? +4. **M12 decision.** User-declared codecs (ClickHouse) versus analyze-and-score (DuckDB) + versus sampling (BtrBlocks): which would you ship for a **graph** database where property + columns arrive via `MERGE` statements with unknown distributions? Commit to one and note + why. Consider that §3.1's `LowCardinality(T)` is itself a *declared* dictionary encoding + — does that change your answer? +5. **The "for everyone" claim.** What did they add to serve small and embedded use + (`clickhouse-local`, chDB — §2), and does it threaten DuckDB's niche or validate it? The + paper's own phrasing about chDB is the evidence; quote it. + +--- + +## Takeaway + +The two-sentence thesis: immutable sorted parts make ingest sequential and every read-side +structure simple, and merges convert background bandwidth into query speed; indexes are +sparse because a vectorized scan over a granule is cheap enough that finding the exact row +is not worth the machinery. Everything the paper concedes — non-atomic mutations, eventual +consistency, no ACID, a join optimizer that cannot run half of TPC-H — is the bill for that +design, and it is presented as such rather than hidden. + +The transferable lesson is not "brute force wins". It is that ClickHouse chose one unit — +the part — and made it the unit of insertion, merging, deduplication, TTL, replication, +snapshot isolation and identity. The simplicity compounds. When you are designing a storage +layer, the question worth asking is not "which index?" but "what is the unit, and how many +jobs can it do?" + +--- ## Done when -You can give the two-sentence ClickHouse thesis (immutable sorted -parts + merge-time work + brute-force vectorized scans; indexes only -sparse), and you have ClickBench-on-DuckDB numbers recorded in -notes.md. +Answer each before unfolding it. + +- [ ] State the ClickHouse thesis in two sentences, then name the single design decision + that the other four sections are consequences of. + +
Answer + +Two sentences: *Tables are sets of immutable parts sorted by a declared key, so inserts are +sequential file writes and background merges do all maintenance — dedup, aggregation, TTL, +re-compression — while streaming rows they were reading anyway. Indexes are sparse (one key +per 8192-row granule) because a vectorized scan of a granule is cheap enough that locating +the exact row is not worth the space or update cost.* + +The decision everything else follows from is **immutability of the part**. Merges can do +work because parts never change (§3.3). Mutations are expensive rewrites for the same +reason (§3.4). Replication can ship whole files, or re-derive them locally, because a part +is deterministic and never patched (§3.6). Snapshot isolation is reference-counting on +versioned parts (§3.7). Idempotent inserts are a 100-entry hash set over parts rather than +an index over rows (§3.5). + +
+ +- [ ] ClickBench is the benchmark ClickHouse leads. Which of its own pruning features does + §6.2.1 say were switched off, and why does that strengthen rather than weaken the + result? + +
Answer + +§6.2.1: "The physical database design is tuned only lightly, for example, we specify primary +keys, but do not change the compression of individual columns, create projections, or +skipping indexes." So of §3.2's three pruning techniques, only the first — the sparse +primary key index — is in play; **projections and skipping indices are off**, and the +default LZ4 codec is used everywhere. + +It strengthens the result because the claim being tested is precisely that scanning is +cheap enough to make the rest optional. Winning with the optional machinery disabled is +evidence for the thesis; winning with it enabled would only show that indexes work. + +The honest caveat in the same section: Umbra, a research system, still posts the best hot +geometric mean. ClickHouse's claim is bounded to production-grade databases. + +
+ +- [ ] ClickHouse could not run 11 of the 22 TPC-H queries. What was missing, and what does + that tell you about where a decade of engineering went? + +
Answer + +Two gaps, both named in §6.2.2 as of v24.6. **Correlated subqueries** are unsupported, +excluding Q2, Q4, Q13, Q17 and Q20–Q22 — seven queries. **Join reordering and join predicate +pushdown** are missing, so Q7–Q9 and Q19 "depend on extended plan-level optimizations… to +achieve viable runtimes" — four more. That is 11 of 22 excluded. Of the 11 that ran, 5 were +faster in ClickHouse and 6 in Snowflake (warehouse size L) — a near tie, on a normalized +schema. + +What it tells you: none of the failures are scan failures. The sparse index, the codecs and +the vectorized engine are not implicated anywhere. The gaps are all **plan-level +optimization** — the part of a database you need when data is normalized and queries have +joins, which is exactly the workload ClickHouse spent fifteen years not having. §6.2.2 says +"automatic subquery decorrelation and better optimizer support for joins are planned for +implementation in 2024". + +
+ +- [ ] `ReplacingMergeTree` deduplicates by primary key. Why is that not a unique + constraint, and what does it cost to get one? + +
Answer + +Because deduplication happens **when a merge happens to run**, and merges are asynchronous +and unscheduled. §3.3 says it directly: merge-time transformation "cannot guarantee that +tables never contain unwanted (e.g. outdated or non-aggregated) values". Between an insert +and the merge that collapses it, a `SELECT` will see both versions. + +The escape hatch is `FINAL` in the `SELECT`, which "applies all merge-time transformations +at query time" (§3.3) — correct results, paid per query instead of once per merge. + +There is a structural reason it cannot be cheaper. §3.1 notes that ClickHouse "treats all +parts as equal instead of arranging them in a hierarchy" and thereby "forgoes the implicit +chronological ordering of parts", which is why "alternative mechanisms for updates and +deletes not based on tombstones are required". A tombstone needs to know which record is +newer; a flat set of parts does not carry that. Replacing merges substitute the part's +creation timestamp, or an explicit version column you supply. + +
+ +- [ ] A replica needs to catch up on a merge another node performed. Give both ways it can + do that and the property that makes the choice legal. + +
Answer + +§3.6, optimization 2: "Merges are replayed by **repeating them locally** or by **fetching +the result part from another node**. The exact behavior is configurable and allows to +balance CPU consumption and network I/O. For example, cross-data-center replication +typically prefers local merges to minimize operating costs." + +The property that makes them interchangeable is that a merge is a **deterministic function +of its immutable inputs**: given the same source parts and the same merge strategy, every +replica produces the same output bytes. Immutability guarantees the inputs are identical; +determinism guarantees the outputs are. + +This is also the boundary condition worth naming: anything non-deterministic inside a merge +— a wall-clock reference in a TTL expression, a random tie-break in a replacing merge — +would break the substitution and force file shipping. Which is why the replication log +records *state transitions* ("parts Y+Z merged into W"), not the data itself. + +
+ +--- ## References **Papers** -- Schulze, Schreiber, Yatsishin, Dahimene, Milovidov — "ClickHouse: - Lightning Fast Analytics for Everyone" (VLDB 2024) — read for the - arguments above, not the mechanisms; skim the eval against your own - ClickBench numbers + +- Robert Schulze, Tom Schreiber, Ilya Yatsishin, Ryadh Dahimene, Alexey Milovidov. + *ClickHouse — Lightning Fast Analytics for Everyone*. PVLDB 17(12): 3731–3744, 2024. + + Read §1, §3, §4.4 and §6.2 closely; skim the rest. **Code** -- [ClickHouse](https://github.com/ClickHouse/ClickHouse) — the code - side is covered by - [reading-clickhouse-mergetree.md](reading-clickhouse-mergetree.md); - [ClickBench](https://github.com/ClickHouse/ClickBench) for the - queries to run alongside + +- [ClickHouse](https://github.com/ClickHouse/ClickHouse) — the code side is covered by + [reading-clickhouse-mergetree.md](reading-clickhouse-mergetree.md), pinned at + `ClickHouse/ClickHouse@4d598fb2c`. The paper's §3.1 numbers (10 MB Compact threshold, + 1 MB block, 150 GB max merged part) all appear there as settings defaults. +- [ClickBench](https://github.com/ClickHouse/ClickBench) — the 43 queries and the public + results dashboard for over 45 systems. + +**In this topic** + +- [reading-cstore-compression.md](reading-cstore-compression.md) — projections, twenty years + earlier, and why C-Store could not afford them +- [reading-duckdb-compression.md](reading-duckdb-compression.md) and + [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md) — the other two answers to "who + picks the encoding", for question 4 +- `FINDINGS.md` row 12 — the measured scan floor (24–57 GB/s on a ~150 GB/s machine) and the + 19,047,619 GB/s hoisted-loop bug diff --git a/topics/12-columnar-analytics/reading-cstore-compression.md b/topics/12-columnar-analytics/reading-cstore-compression.md index b4c2733..633fb49 100644 --- a/topics/12-columnar-analytics/reading-cstore-compression.md +++ b/topics/12-columnar-analytics/reading-cstore-compression.md @@ -1,106 +1,295 @@ # C-Store: operate on compressed data -Every system in this topic descends from two papers out of the same -lab, read here as a pair: C-Store proposes the column-store -architecture, and the SIGMOD '06 follow-up proves the thesis this -topic is named for — the executor should OPERATE ON compressed data, -not just store it. Twenty years on, the value is seeing which of the -original bets survived, and in what disguise. Before you open either -paper, this chapter builds the ideas step by step, then hands you a -reading route through both. +Two papers out of the same lab, read here as a pair: + +- Mike Stonebraker, Daniel Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel + Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O'Neil, Pat O'Neil, Alex Rasin, + Nga Tran, Stan Zdonik. *C-Store: A Column-oriented DBMS*. VLDB 2005 — the architecture. +- Daniel Abadi, Samuel Madden, Miguel Ferreira. *Integrating Compression and Execution in + Column-Oriented Database Systems*. SIGMOD 2006 — the thesis this topic is named for: the + executor should **operate on** compressed data, not merely store it. + +Twenty years on, the value is seeing which of the original bets survived and in what +disguise — and the 2006 paper is unusually good at giving you the *numbers* to score them +with. Before you open either paper, this chapter builds the ideas step by step, then hands +you a reading route through both. + +Vocabulary you will need, all defined at the step that introduces them: **row store**, +**column store**, **projection**, **segment**, **storage key**, **join index**, +**run-length encoding**, **bit-vector encoding**, **delta encoding**, **dictionary +encoding**, **null suppression**, **position list**, **bitstring**, **late +materialization**, **eager** vs **lazy decompression**. + +--- ## The problem in one sentence -2005's row-store OLTP engines read every column of every row to answer -analytics that touch 3 columns of 100 — a 30× bandwidth waste before -any work happens — and C-Store's answer (store columns separately, -sorted, compressed) raised a second question the follow-up paper -answers: once the data is compressed, must you decompress it to -compute? +A 2005 **row store** — a system that stores all attributes of a tuple consecutively — must +read every column of every row to answer an analytic query that mentions three columns out +of a hundred; C-Store's answer was to store each column separately, sorted, and compressed, +which immediately raised the harder question the 2006 follow-up answers: **once the data is +compressed, must you decompress it to compute on it?** + +--- ## The concepts, step by step -### Step 1 — the column store: read only what the query touches - -A **column store** keeps each column of a table in its own contiguous -file, so a query reads only the columns it mentions — a -`SELECT sum(price) WHERE date > X` on a 100-column, 400-byte-row table -reads ~16 of every 400 bytes instead of all of them, a 25× IO cut -before any cleverness. The second, less obvious win: a column is -SELF-SIMILAR — one type, similar values, often sorted or clustered — -which is exactly the shape every lightweight encoding (RLE, -dictionary, bit-packing) feeds on; rows interleave types and kill -every trick. Why it matters: everything in this topic — DuckDB, -ClickHouse, Parquet — is a descendant of this one storage decision. - -### Step 2 — projections: store the table several times, each sorted differently - -A **projection** in C-Store is a copy of some columns of the table -stored physically sorted on a chosen key — and the same table can have -several projections, each sorted differently, so each query picks the -copy whose sort order serves it. This is worth dwelling on because -**sort order is THE enabler**: a column sorted (or clustered) on the -filter key gets long RLE runs (compression) and tight min/max zones -(zone-map pruning) — clustering decides compressibility, stated in -2005; ClickHouse's mandatory `ORDER BY` is the same lesson made a -schema requirement. The cost that killed the idea in its full form: -every extra sorted copy multiplies storage and, worse, write -amplification — each insert lands in every projection. Why it matters: -the idea died as a default and survived as an option (ClickHouse's -secondary "projections" feature is literally named after it, paid for -by the merge machinery). - -### Step 3 — WS/RS: writes and reads live in different structures - -C-Store splits the store in two: a small **WS** (writeable store — an -uncompressed, insert-friendly structure that absorbs writes) and a -large **RS** (read store — the compressed, sorted projections), with a -background **tuple mover** batch-migrating WS contents into RS. Sound -familiar? It's the LSM shape (topic 4: mutable in-memory buffer + -immutable sorted bulk + background merge) invented independently for -analytics — surviving today as delta + main (SAP HANA) and inserts + -parts (ClickHouse). Why it matters: every read-optimized layout in -this book, columnar or graph (FalkorDB's Delta_Matrix, topic 13), -grows this same two-structure answer, because a layout can be -write-friendly or read-optimal but not both. - -### Step 4 — positions: the currency of late materialization - -A **position** is a row's ordinal within a projection (row 0, 1, 2 …), -and C-Store's operators exchange **position lists or bitmaps** — "rows -17, 204, 9,881 survived the filter" — instead of assembled tuples. -**Late materialization** is the resulting discipline: run filters and -joins on the cheap columns first, carry positions through the plan, -and fetch the wide payload columns only for final survivors. At 1% -selectivity on 1M rows, that's 10K payload fetches instead of 1M -decodes. Survives as DuckDB's selection vectors (topic 11) and -Parquet's late decode. Why it matters: positions are what let the next -step's compressed operators avoid ever building a row. - -### Step 5 — the SIGMOD '06 thesis: execute per-run, not per-row - -The follow-up paper's experiment: implement RLE, dictionary, -bit-packing, LZ, and null suppression in a column executor, then -compare **decompress-then-process** against **process-compressed** — -operators that understand the encoding and work on it directly: - -``` - decompress-then-process: [decode all] -> [scan rows] bandwidth + work per ROW - process-compressed: [scan runs/codes directly] work per RUN / per code -``` - -Operating on RLE is a different complexity class: `SUM` over a run = -value × length; a predicate evaluates ONCE per run, not per row — -sorted low-cardinality columns get speedups proportional to average -run length (the paper shows order-of-magnitude wins). The whole thesis -fits in one loop — a filtered SUM over RLE that never materializes a -row: +### Step 1 — The column store, and the four encodings C-Store shipped + +> **In:** a table, and a query that mentions 3 of its 100 columns. +> **Out:** the column-store layout, and the 2×2 rule C-Store used to pick an encoding for +> each column. + +A **column store** is "one in which each attribute is stored in a separate column, such +that successive values of that attribute are stored consecutively on disk" (Abadi §1) — as +opposed to a row store, where "values of different attributes from the same tuple are +stored consecutively". The first-order win is obvious: read 3 columns, not 100. + +The second-order win is the one this whole topic runs on, and Abadi §1 states it plainly: +"Compression ratios are also generally higher in column-stores because consecutive entries +in a column are often quite similar to each other, whereas adjacent attributes in a tuple +are not." A column is *self-similar* — one type, one domain, often sorted — which is +exactly the shape every lightweight encoding feeds on. A row interleaves types and kills +every trick. And §1 adds the structural reason a row store cannot copy this: "In a +row-oriented database, such schemes do not work as well because an attribute is stored as a +part of an entire tuple, so combining the same attribute from different tuples together +into one value would require some way to 'mix' tuples." + +C-Store §3.1 then does something worth copying: it makes the encoding choice a **2×2 table** +on two properties of the column — is it sorted by *its own* values (self-order) or by some +other column's (foreign-order), and does it have few or many distinct values? + +| | few distinct values | many distinct values | +| --- | --- | --- | +| **self-order** | **Type 1** — RLE triples `(v, f, n)`: value, first position, run count | **Type 3** — **delta encoding**: store each value as its difference from the previous one, block-oriented, with the first value of each block stored whole | +| **foreign-order** | **Type 2** — **bit-vector encoding**: one bitmap per distinct value, marking the positions where it occurs; each bitmap is itself run-length encoded because it is sparse | **Type 4** — leave it uncompressed ("we are still investigating possible compression techniques for this situation") | + +Two terms defined by that table. **Run-length encoding (RLE)** replaces a run of identical +values with a single record describing the run — C-Store's is a *triple*, `(4, 12, 7)` +meaning "the value 4 occupies positions 12 through 18". **Bit-vector encoding** turns +`1132231` into three bitmaps: `1100001` for value 1, `0001100` for 2, `0010010` for 3 +(Abadi §4.4). + +Note what is *not* in C-Store's 2005 table: dictionary encoding, and any heavyweight codec. +Those arrive in the 2006 paper. Type 4 — "leave it alone" — is the honest admission of a +gap, and Steps 5–7 are the paper that fills it. + +**Why it matters:** every system in this topic descends from this one storage decision, and +every one of them still asks the same two questions C-Store asked — how sorted, how many +distinct values. DuckDB's analyze pass, BtrBlocks' sampler and ClickHouse's `LowCardinality` +hint are three different mechanisms for filling in the same 2×2. + +### Step 2 — Projections: there is no base table + +> **In:** a logical table `EMP(name, age, salary, dept)`. +> **Out:** what C-Store actually stores on disk, and the machinery needed to get a row back. + +This is the step where most summaries of C-Store go wrong, so read the paper's own sentence +first (C-Store §2): "Whereas most row stores implement physical tables directly and then add +various indexes to speed access, **C-Store implements only projections**." And two sentences +later: "we use the term projection slightly differently than is common practice, as **we do +not store the base table(s) from which the projection is derived**." + +A **projection** is "anchored on a given logical table, T, and contains one or more +attributes from this table", plus optionally attributes from other tables reachable by a +chain of n:1 foreign-key relationships — so a projection may be pre-joined. It has the same +number of rows as its anchor table, duplicates retained. Every column of the projection is +stored column-wise and **sorted on a common sort key**, written after a vertical bar: + +``` +EMP1(name, age | age) +EMP2(dept, age, DEPT.floor | DEPT.floor) +EMP3(name, salary | salary) +DEPT1(dname, floor | floor) +``` + +Each projection is horizontally cut into **segments**, value-partitioned on the sort key. +Within a segment, every value carries a **storage key** — its ordinal position, 1, 2, 3, … +— which in the read store is "not physically stored, but inferred from a tuple's physical +position in the column". + +Now the consequence. If there is no base table, reconstructing a row means stitching +several differently-sorted projections together, and C-Store does that with **join +indexes**: "a collection of `(sid, storage_key)` pairs", one per tuple, mapping each row of +projection T1 to the corresponding row of T2. "An alternative view of a join index is that +it takes T1, sorted in some order O, and logically resorts it into the order O' of T2." + +That is the cost that killed the design in its full form. A join index is a permutation with +one entry per row per projection pair — and §7 notes that the tuple mover's merge-out +assigns *new* storage keys in the rebuilt read store, "thereby requiring join index +maintenance". Every projection multiplies storage, every insert lands in every projection, +and every rebuild rewrites the permutations that tie them together. + +**Why it matters:** sort order is the enabler for everything downstream — a column sorted or +clustered on the filter key gets long RLE runs and tight min/max ranges. C-Store's mistake +was not the idea but the price: paying for *k* sort orders with *k* full copies plus *k²* +permutations. ClickHouse's `ORDER BY` makes one sort order mandatory and free; its +"projections" feature — literally named after this — buys extra ones lazily, on the merge +machinery it was already paying for (see +[reading-clickhouse-paper.md](reading-clickhouse-paper.md), §3.2). + +### Step 3 — WS and RS: an LSM tree, named otherwise + +> **In:** a stream of inserts arriving at a store optimized for reads. +> **Out:** the two-structure answer, and the honest caveat attached to its benchmark. + +C-Store splits the store in two: **WS**, the writeable store, "efficiently updatable +transactionally", with storage keys explicitly materialized; and **RS**, the read store, +compressed and sorted, storage keys inferred from position. A background **tuple mover** +migrates WS into RS. + +The paper does not leave the resemblance implicit — §1: "we use a variant of the +**LSM-tree** concept [ONEI96], which supports a **merge out** process that moves tuples from +WS to RS in bulk by an efficient method of merging ordered WS data objects with large RS +blocks, resulting in a new copy of RS that is installed when the operation completes." + +Map it onto topic 4's vocabulary: + +| C-Store | LSM | +| --- | --- | +| WS — small, updatable, column-organized | memtable | +| RS — large, compressed, sorted, immutable | sorted runs on disk | +| tuple mover / merge-out process (MOP) | compaction | +| high/low water mark epoch numbers | the visibility snapshot a compaction may drop below | + +§7 spells out the old-master/new-master discipline: MOP reads blocks from the RS segment, +drops rows deleted at or before the low water mark, merges in the WS values, writes a new +segment RS′, then "the system cuts over from RS to RS′. The disk space used by the old RS +can now be freed." Immutable inputs, a new output, an atomic swap — ClickHouse's merge, in +2005 clothing. The paper's own justification: "This old-master/new-master approach will be +more efficient than an update-in-place strategy, since essentially all data objects will +move." + +And the caveat that any honest reading has to carry. §9: "At the present time, we have a +storage engine and the executor for RS running. We have an **early implementation of the WS +and tuple mover; however they are not at the point where we can run experiments on them**." +§1 says the same: "we have not fully integrated the WS and tuple mover, **whose overhead +may be significant**." So every C-Store number in Step 7's tables is a *read-only* number +from RS alone. The write side of the design was designed, not measured. + +**Why it matters:** every read-optimized layout in this book — columnar or graph, including +FalkorDB's `Delta_Matrix` in topic 13 — converges on this same two-structure answer, because +a layout can be write-friendly or read-optimal and not both. What differs between systems is +only who pays for the mover. + +### Step 4 — Bitstrings and Mask: late materialization before it had a name + +> **In:** a predicate on one column and a `SELECT` list naming three others. +> **Out:** the currency C-Store's operators actually exchange, and the optimizer decision it +> creates. + +A **position** is a value's ordinal offset within a column (Abadi §5.1 defines it exactly +that way). A **position list** is a set of them — "rows 17, 204, 9,881 survived" — and a +**bitstring** is the same set as one bit per row. + +C-Store's operator set (§8.1) is built around them. Of its ten node types, three matter here: + +- **`Select`** "is equivalent to the selection operator of the relational algebra (σ), but + rather than producing a restriction of its input, instead **produces a bitstring + representation of the result**." +- **`Mask`** "accepts a bitstring B and projection Cs, and restricts Cs by emitting only + those values whose corresponding bits in B are 1." +- **`Permute`** "permutes a projection according to the ordering defined by a join index." + +Plus `BAnd` / `BOr` / `BNot` for combining bitstrings without touching data. Joins are the +same idea: Abadi §3 — "Joins produce positions rather than values", and §5.2 shows the +output of a join being a pair of position columns which are then "sent to other columns from +the input relations… to extract the values at these positions". + +**Late materialization** is the modern name for this discipline: run the filters and joins +on the cheap columns, carry positions through the plan, and fetch the wide payload columns +only for the survivors. Neither paper uses the phrase — C-Store calls it `Mask` placement, +Abadi §6.5 calls it **position filtering**, and Abadi §2 credits the general idea of holding +data compressed in memory to Graefe and Shapiro under the name **lazy decompression**. The +name "late materialization" arrives in the 2007 follow-up (Abadi, Myers, DeWitt, Madden, +*Materialization Strategies in a Column-Oriented DBMS*, ICDE 2007); it is worth knowing that +the idea predates its label by two years. + +What makes this a *concept* and not a trick is that C-Store §8.2 turns it into an explicit +optimizer decision: "the optimizer must decide **where in the plan to mask a projection** +according to a bitstring. For example, in some cases it is desirable to push the `Mask` early +in the plan… while in other cases it is best to delay masking until a point where it is +possible to feed a bitstring to the next operator in the plan (e.g., `COUNT`) that can +produce results solely by processing the bitstring." + +One implementation detail worth keeping: "C-Store iterators return **64K blocks** from a +single column. This approach preserves the benefit of using iterators… while changing the +granularity of data flow to better match the column-based model" (§8.1). Vectorized +execution, arrived at from the compression side rather than the CPU side. + +**Why it matters:** positions are what make Step 5's compressed operators possible at all. +An operator that never assembles a row never has to decode one. + +### Step 5 — The 2006 experiment: eager decompression versus direct operation + +> **In:** one aggregation query, six encodings, and two executor policies. +> **Out:** the measured gap between decompress-then-process and process-compressed. + +The experiment (Abadi §6) is a single-column aggregation: + +```sql +SELECT SUM(C) FROM TABLE GROUP BY C +``` + +over **100 million 32-bit integers**, with the six encodings of §4 — null suppression, LZ, +RLE, bit-vector, dictionary, none — and two policies: + +``` + eager decompression: [decode everything off disk] -> [scan rows] work per ROW + direct operation: [scan runs / codes / bitmaps directly] work per RUN / CODE +``` + +The data is generated so that two parameters can be dialled independently: the **number of +distinct values** (2 to 40 in the first set) and the **average sorted run length** (50, 100, +500, 1000). The rationale is worth noting because it is the same clustering argument as Step +2: "if column C is tertiarily sorted and the first column in the projection has 500 unique +values and the second column in the projection has 1000 unique values then C will have +average sorted runs of size 100000000/(500*1000)=200." + +The measured result, §6.2, on the data with 1000-record sorted runs — average improvement +from *not* eagerly decompressing: + +| Encoding | speed-up from direct operation | +| --- | --- | +| bit-vector | **10.3×** | +| dictionary, group-by-self (multi-value) | **3.94×** | +| RLE | **3.3×** | +| dictionary, value-at-a-time (single-value) | **1.1×** | +| LZ, null suppression | 1.0× — "LZ and NS cannot operate on encoded data" | + +And the reason, stated as a complexity claim rather than a speed-up (§6.2): the CPU cost of +the aggregation is proportional to *n*, where *n* is + +- `num_tuples` for the uncompressed and row-oriented schemes, +- `num_tuples / avg_run_len` for RLE, +- `num_tuples / dict_entry_size` for dictionary multi-value, +- `num_distinct_values` for bit-vector encoding. + +That last line is why bit-vector wins by 10.3×: at 40 distinct values, a `GROUP BY` over +100 million rows becomes 40 popcounts. It is a different complexity class, not a constant +factor. + +The paper then runs the same queries **with CPU contention** (§6.2, Figure 6(c)) and finds +that the schemes with executor shortcuts barely degrade, while LZ, null suppression and +value-at-a-time dictionary degrade most — and confirms with performance counters that +"competition for cache lines accounted for less than 2% of the increase in query time", so +it is genuinely CPU cycles. The conclusion drawn is the durable one: + +> "while normal compression simply trades 'expensive' I/O time for 'cheap' CPU, operating +> directly on compressed data reduces **both** I/O and CPU cycles. This suggests that even +> on a machine with a much faster I/O or a much slower CPU, compressing data and operating +> directly on it will be beneficial." + +The whole thesis fits in one loop — a filtered `SUM` over RLE that never materializes a row: ```rust +// ILLUSTRATION — pseudocode, not quoted from any repo. The paper's own version is +// the Count aggregator in Figure 2 of Abadi et al., SIGMOD 2006. For a real +// implementation of the same idea see duckdb/duckdb@6c0c1a68 +// src/storage/compression/rle.cpp:113 (RLEFinalAnalyze) and rle.cpp:99. struct Run { value: u64, len: u32 } // decompress-then-process is O(rows); this is O(runs). -// sorted low-cardinality columns: runs ≪ rows, often by 1000x fn sum_where_gt(runs: &[Run], threshold: u64) -> u64 { let mut sum = 0; for r in runs { @@ -112,91 +301,521 @@ fn sum_where_gt(runs: &[Run], threshold: u64) -> u64 { } ``` -Dictionary codes compose with Step 4's late materialization: compare -encoded ints, decode only survivors — string predicates become int -predicates (your scan_bench reproduces both effects). Why it matters: -compression stops being a storage tax paid at scan time and becomes -the executor's fast path. - -### Step 6 — the lightweight/heavyweight split, proven - -The same experiment condemns heavyweight codecs for the scan path: -LZ-class compression saved IO but cost CPU per block and offered **no -execution shortcuts** — there is no "sum a gzip block" trick, you must -inflate it. Lightweight encodings both shrink bytes AND admit -per-run/per-code execution; gzip-class codecs belong at rest. Why it -matters: this 2006 finding is Parquet's two compression layers -(semantic then block) and DuckDB's zstd-as-last-resort, decided twenty -years in advance. - -### Step 7 — the abstraction that makes it maintainable - -The naive implementation of process-compressed needs encodings × -operators variants — 5 encodings × 20 operators = 100 -implementations, unmaintainable. The paper's fix: operators consume -"compressed blocks" through an API exposing *properties* (isRLE? -isSorted? oneValue?) so each operator writes a few property-driven -cases, not one per encoding. DuckDB's vector-type flags -(FLAT/CONSTANT/DICTIONARY/FSST, topic 11) are this API, shipped in -production. Why it matters: this is the difference between a benchmark -paper and an architecture — the abstraction is what let the idea -survive into real engines. +Note one difference from the paper: C-Store's RLE record is a *triple* `(value, start_pos, +run_length)`, because positions have to be addressable for Step 4's `Mask` to work. DuckDB's +is a pair — `rle.cpp:113-116` sizes a compressed segment as +`(sizeof(rle_count_t) + sizeof(T)) * seen_count` — because DuckDB reconstructs positions by +running prefix sums instead of storing them. Step 6 prices both. + +**Why it matters:** compression stops being a storage tax paid back at scan time and becomes +the executor's fast path. That inversion is the reason this topic exists. + +### Step 6 — Do the sizes yourself + +> **In:** one concrete column — 1,000,000 `INT64` values, 200 distinct, average run 8. +> **Out:** the byte count under each encoding, with the multiplication shown, and the two +> break-evens worth memorising. + +Use the same column the rest of this topic uses, so the numbers compose across guides. +1,000,000 values, 200 distinct, average run length 8 ⇒ `1,000,000 / 8` = **125,000 runs**. +Baseline: `1,000,000 × 8 B` = **8,000,000 B**. + +**Dictionary encoding** replaces each value with an index into a table of the distinct +values. The code width is `ceil(log2(distinct))` = `ceil(log2(200))` = `ceil(7.64)` = **8 +bits**, because 2⁷ = 128 < 200 ≤ 256 = 2⁸. So: + +``` +codes 1,000,000 × 8 bits = 1,000,000 B +dictionary 200 × 8 B = 1,600 B + ----------- + 1,001,600 B 8,000,000 / 1,001,600 = 7.99x +``` + +**Run-length encoding**, C-Store's three-field triple `(value: 8 B, start_pos: 4 B, +run_len: 4 B)` = 16 B per run: + +``` +125,000 runs × 16 B = 2,000,000 B 8,000,000 / 2,000,000 = 4.00x +``` + +DuckDB's two-field pair `(value: 8 B, count: 4 B)` = 12 B per run: + +``` +125,000 runs × 12 B = 1,500,000 B 8,000,000 / 1,500,000 = 5.33x +``` + +Storing the start position costs **33% of the compressed size** — the price of Step 4's +random access into a run, and exactly the trade DuckDB declines. + +**Bit-vector encoding**, one bitmap per distinct value: + +``` +200 values × 1,000,000 bits = 200,000,000 bits = 25,000,000 B ratio 0.32x +``` + +Three times *larger* than the raw column. The break-even is worth deriving because Abadi §6.1 +states it as an observed fact and it is really arithmetic: bit-vector costs `c` bits per row +for cardinality `c`, against `w` bits per row raw, so it only shrinks when `c < w`. The paper +on 32-bit data: "as soon as the column cardinality is more than 32, type-2 compression is no +longer more compressed than the original 32-bit data." ✓ For our 64-bit column the +break-even is 64, and at 200 distinct values we are `200 / 64` = 3.1× over it. (Which is why +Abadi §4.4 notes their bitmaps are left *un*-further-compressed: "one needs the bit-maps to +be fairly sparse (on the order of 1 bit in 1000) in order for query performance to not be +hindered".) + +**The second break-even — position list versus bitstring** (Step 4's currency, and question +4). Over 1,000,000 rows a bitstring costs `1,000,000 / 8` = **125,000 B**, flat, whatever the +selectivity. A position list at 4 B per surviving row costs `4 × s`. Setting them equal: + +``` +4 x s = 125,000 => s = 31,250 rows = 3.125% selectivity +``` + +At 1% (10,000 survivors) the list is `4 × 10,000` = 40,000 B — **3.1× smaller** than the +bitstring. At 10% (100,000 survivors) it is 400,000 B — **3.2× larger**. Below ~3%, ship +positions; above, ship bits. That single crossover explains why C-Store carries both +representations and why Abadi §5.2's `isPosContig()` property exists at all. + +**One more, from the paper's own §4.2.** C-Store's dictionary packs several codes into a +machine word and keeps entries **byte-aligned**, choosing 1, 2, 3 or 4 bytes per entry "by +requiring the dictionary to fit in the L2 cache". Their worked example: 32 values ⇒ 5-bit +codes ⇒ 1 code fits in 1 byte, 3 in 2 bytes, 4 in 3 bytes, 6 in 4 bytes; picking 3-per-2-bytes +makes the dictionary `32³` = **32,768 entries** = **524,288 B**, "which is half of the L2 +cache on our machine (1MB)". Run the same rule on our column: 8-bit codes pack exactly 1 per +byte with no waste, and a 2-codes-per-entry dictionary would need `200²` = 40,000 entries +(640,000 B — borderline), while 3 codes per entry needs `200³` = 8,000,000 entries, hopeless. +The cache, not the bit width, is what caps the trick. + +**Why it matters:** every ratio in Step 7's tables is one of these five multiplications with +different inputs. Doing them once means you can predict the paper's results before reading +them — and catch the ones that do not follow. + +### Step 7 — Lightweight versus heavyweight, proven, and the decision tree + +> **In:** the same aggregation, run across the cardinality × run-length grid. +> **Out:** the 2×2 result table, the condemnation of heavyweight codecs on the scan path, +> and the heuristic the paper distilled. + +Abadi §6.3's summary table — aggregation query times in seconds, best in **bold**. "High" +and "low" cardinality are 10,000 and 37 distinct values; "runs" means average run length 14. + +| Data | RLE | LZ | Dictionary | Bit-vector | No compression | +| --- | --- | --- | --- | --- | --- | +| no runs, low cardinality | 17.67 | 9.30 | **7.49** | 12.02 | 10.86 | +| runs, low cardinality | **2.43** | 3.93 | 3.29 | 9.83 | 7.59 | +| no runs, high cardinality | 32.48 | 15.05 | **11.25** | N/A | 13.31 | +| runs, high cardinality | **2.56** | 4.48 | 4.56 | N/A | 9.52 | + +Read the columns, not the rows. RLE swings from **worst** (32.48 s) to **best** (2.56 s) on +the same cardinality purely because runs appeared — "for RLE and LZ, run-length is a better +indicator of performance than cardinality". Dictionary is the safe default when there is no +locality. Bit-vector is unusable above ~40 distinct values. And LZ is never best in any of +the four cells. + +That is the condemnation of heavyweight codecs on the scan path, and §6.2 gives the reason: +"since LZ and NS cannot operate on encoded data, their performance for these experiments was +identical" to the eager-decompression case. There is no "sum a gzip block" shortcut; you must +inflate first. The conclusion (§7) states the trade as a recommendation: "**Sacrificing the +compression ratio of heavy-weight schemes for the efficiency light-weight schemes in +operating on compressed data is a good trade-off to make.**" + +The join experiment (§6.5) is the most dramatic number in the paper. A foreign-key join with +predicates on both sides, times in seconds: + +| Encoding of the fact-table join column | 50 distinct keys | 50,000 distinct keys | +| --- | --- | --- | +| RLE | **0.06** | **0.07** | +| Bit-vector | 0.97 | N/A | +| Dictionary | 3.15 | 3.86 | +| No compression | 4.08 | 4.3 | + +`4.08 / 0.06` = **68×** at 50 keys, `4.3 / 0.07` = **61×** at 50,000. The mechanism is Step +4's: an RLE run joins once and emits a whole position range, so the join does work per *run* +where the uncompressed plan does work per *row*. + +§6.5 also carries the finding that most summaries drop. The same query with the two columns' +roles swapped (Figure 9(b)) makes bit-vector encoding go from fastest to catastrophically +slow, "because the query requires the values of the bit-vector column **in position order** +which forces decompression". The lesson the authors draw: "the proper choice of encoding type +for a column depends not just on data characteristics, but also on **the expected query +workload**… It also indicates that redundantly storing the same column in the same sort order +using different compression schemes might be a good idea." + +Figure 10 distils all of it into a decision tree, whose two non-obvious predicates are worth +memorising: "**exhibits good locality**" means the column is a sort column, correlated with +one, or otherwise repetitive; "**likely to be used in a position contiguous manner**" means +it must be read in parallel with another column — true for a `SELECT`-list column read +through a sorted position list, false for a column that only appears in the `WHERE` clause. + +**Why it matters:** this 2006 finding is Parquet's two compression layers (semantic encoding, +then optional block codec) and DuckDB's zstd-as-last-resort, decided twenty years in advance. +It is also the reason `FINDINGS.md` row 12's benchmarks measure *scan* rates rather than +storage ratios: the ratio is not the figure of merit if the executor has to inflate. + +### Step 8 — Three booleans instead of n² operators + +> **In:** an executor with *n* encodings and a set of binary operators. +> **Out:** the abstraction that stopped the combinatorics, and its modern descendant. + +Abadi §5.2 states the engineering problem before the solution: "Every time a new compression +scheme is added to the system, all operators that operate directly on this type of data have +to be supplemented to handle the new scheme. Without careful engineering, there would end up +being **n versions of each operator** – one for each type of compression scheme that can be +input to the operator. Operators that take two inputs (like joins) would need **n² +versions**." + +Figure 1's nested-loop join pseudocode makes it vivid, ending with the line "etc. etc. for +every possible combination of encoding types". + +The fix is a **compressed block API** (Table 1) with three groups of methods: + +| Properties | Iterator access | Block information | +| --- | --- | --- | +| `isOneValue()` | `getNext()` | `getSize()` | +| `isValueSorted()` | `asArray()` | `getStartValue()` | +| `isPosContig()` | | `getEndPosition()` | + +Only **three** properties, and — this is the point — none of them names an encoding. An +operator never asks "is this RLE?"; it asks "does this block hold one value at many +positions?" The properties table (§5.2): + +| Encoding | sorted? | one value? | position contiguous? | +| --- | --- | --- | --- | +| RLE | yes | yes | yes | +| bit-string | yes | yes | **no** | +| null suppression | data-dependent | no | yes | +| Lempel-Ziv | data-dependent | no | yes | +| dictionary | data-dependent | no | yes | +| uncompressed | data-dependent | no | data-dependent | + +RLE and bit-vector differ in exactly one bit of that table — `isPosContig()` — and that single +bit is what selects between Figure 1's second and third optimization. The paper says so: +those optimizations work "in general, not just for RLE" and "in general, not just for +bit-vector compression". + +The payoff is the `Count` aggregator of Figure 2, which is four lines and knows nothing about +encodings: if `isOneValue()`, add `getSize()` to that value's counter; otherwise fall back to +`asArray()` and iterate. "Note that despite RLE and bit-vector encoding being very different +compression techniques, the pseudocode in Figure 2 need not distinguish between them, pushing +the complexity of calculating the block size into the compressed block code." + +**Why it matters:** this is the difference between a benchmark paper and an architecture. The +API is what let the idea survive into production — DuckDB's vector types (`FLAT`, `CONSTANT`, +`DICTIONARY`, `FSST`; topic 11) are the same three questions asked with different names, +`CONSTANT` being `isOneValue()` and the selection vector being the negation of +`isPosContig()`. + +### Step 9 — Which GB/s? Twenty years of hardware, one surviving conclusion + +> **In:** the 2006 machine's disk rate and this topic's measured scan floor. +> **Out:** the factor between them, and why the paper's conclusion still holds anyway. + +Abadi §6 gives the hardware: "a 3.0 GHz Pentium IV, running RedHat Linux, with 2 Gbytes of +memory, 1MB L2 cache, and 750 Gbytes of disk. **The disk can read cold data at 50-60 +MB/sec.**" + +That number explains the shape of every graph in the paper. The uncompressed column is +`100,000,000 × 4 B` = 400 MB; at 55 MB/s that is `400 / 55` = **7.3 seconds of pure I/O**, +and the measured no-compression times sit at 7.59–13.31 s. In 2006, a scan was an I/O +problem with a CPU epilogue. + +Now put this topic's own measurement beside it. `FINDINGS.md` row 12 records a scan floor of +**24–57 GB/s on a machine with roughly 150 GB/s of memory bandwidth**. Against the paper's +disk: + +``` +57 GB/s / 55 MB/s = 57,000 MB/s / 55 MB/s = ~1,036x +``` + +Three orders of magnitude. Every premise of the 2006 experiment has moved — and the +conclusion did not, because the paper's own §6.2 anticipated exactly this: "even on a machine +with a much faster I/O or a much slower CPU, compressing data and operating directly on it +will be beneficial." The reason is the complexity argument, not the bandwidth one: shrinking +`num_tuples` to `num_tuples / avg_run_len` is an algorithmic win that survives any change in +the constant. + +Which brings the discipline this topic keeps insisting on: **say which bytes you counted.** +Our Step 6 column compresses 7.99× under dictionary encoding, so 8 MB of logical `INT64` +values live in 1,001,600 B on disk. Report the same scan two ways: + +``` +physical: 1,001,600 B read logical: 8,000,000 B processed +at 24 GB/s logical => 3.0 GB/s of compressed bytes actually moved +at 3.0 GB/s physical => 24 GB/s of "effective" throughput +``` + +Both sentences describe one query. Neither is wrong; a report that omits which one it means +is. And `FINDINGS.md` row 12 preserves this topic's cautionary tale for the case where the +arithmetic is not just ambiguous but impossible: a hoisted timing loop once printed +**19,047,619 GB/s**, which is `19,047,619 / 150` ≈ **127,000×** the machine's peak memory +bandwidth. A throughput above the hardware's ceiling is never a result. + +**Why it matters:** it is the reason to re-derive the 2006 findings on your own machine +rather than quote them. The ranking survived; the absolute numbers did not, and this topic's +benchmarks exist to replace them. + +--- ## How to read the papers (with the concepts in hand) -**C-Store (VLDB '05)** — read for the architecture bets and score -them against twenty years of history: - -| C-Store bet | survived as | -|---|---| -| columns, not rows, for reads | everything in this topic | -| projections: same table stored MULTIPLE times, each sorted differently | mostly died (storage cost); echoes in ClickHouse ORDER BY + materialized views, secondary "projections" feature literally named after it | -| WS/RS split: writeable store + read store, tuple mover between | LSM-shaped! delta + main (SAP HANA), parts + inserts (ClickHouse) | -| compression per column, chosen by data properties | DuckDB's analyze/score | -| late materialization: join on position lists, fetch payload last | DuckDB selection vectors, Parquet late decode | -| k-safety via projection redundancy instead of RAID | died; replication won | - -Read the storage model (projections, WS/RS) carefully — Steps 2–3; -skim the k-safety and recovery sections (that bet died; replication -won). Watch for positions-as-join-currency (Step 4) — selection -vectors avant la lettre. - -**SIGMOD '06** — read the experiment design, then internalize the -findings list: per-run execution (Step 5), the lightweight/heavyweight -split (Step 6), and the properties API (Step 7). The graphs showing -speedup vs average run length are the quantitative core — compare -them against your own scan_bench numbers. +Budget about three hours for the pair. **Read the 2006 paper first** if you only read one — +it is the one with the thesis and the numbers; C-Store 2005 is the architecture it extends. + +**C-Store (VLDB 2005)** — read for the bets, and score them against twenty years of history: + +| C-Store bet | § | survived as | +| --- | --- | --- | +| columns, not rows, for reads | 2, 3 | everything in this topic | +| **only** projections — no base table, several sort orders, join indexes to reassemble | 2 | mostly died: the storage and the permutation maintenance. Echoes in ClickHouse's mandatory `ORDER BY` and its lazily-populated "projections" feature, which is literally named after this | +| four encodings picked by (self/foreign order) × (few/many distinct values) | 3.1 | DuckDB's analyze-and-score, BtrBlocks' sampler — the mechanism changed, the two questions did not | +| WS / RS split with a tuple mover, "a variant of the LSM-tree concept" | 1, 4, 7 | delta + main (SAP HANA), parts + merges (ClickHouse) — and never benchmarked in this paper | +| `Select` → bitstring, `Mask` placement as an optimizer decision | 8.1, 8.2 | DuckDB selection vectors, Parquet late decode, every late-materialization plan since | +| K-safety through redundant overlapping projections instead of RAID | 1, 6.3 | died; replication won | +| snapshot isolation to avoid 2PC and locking for queries | 1, 6.1 | survived everywhere, including ClickHouse §3.7's versioned parts | + +Read §2 (data model) and §3.1 (encodings) carefully — Steps 1–2. Read §7 (tuple mover) — +Step 3. Read §8.1's operator list — Step 4; it is one page and it is the whole idea. Skim +§6.1–6.3 (snapshot isolation, locking, recovery) and §5 (grid allocation) unless you have a +specific interest. Read §9's first paragraph before its tables, so the read-only caveat is in +place before the 164× lands. + +**SIGMOD 2006** — read the experiment design, then internalise the findings: + +1. **§4** — the six schemes, and *how* each is implemented. §4.2's byte-alignment argument is + the surprising one: "column stores are so I/O efficient that even a small amount of + compression is enough to make queries on that column become CPU-limited", so they + deliberately waste bits to save shifts. +2. **§5.1–5.2** — Step 8. Table 1 and the properties table are the two things to copy into + `notes.md`. +3. **§6.1 vs §6.2** — the same experiment twice, eager then direct. The gap between the two + figures *is* the paper. +4. **§6.3's summary table and §6.5's join table** — Step 7. These are the numbers to cite. +5. **§7 and Figure 10** — the decision tree, and the three closing observations. The third + one — "cost models that only take into account I/O costs will likely perform poorly in the + context of column-oriented systems since CPU cost is often the dominant factor" — is the + sentence the next twenty years of the field spent proving. + +Compare the graphs of speed-up versus average run length against your own `scan_bench` +numbers from this topic's `experiments/` crate. They will not match; the ranking should. + +--- ## Questions for notes.md -1. SUM over RLE runs is O(runs). Which OTHER aggregates stay - run-shortcuttable (min/max? count? avg?) and which break (distinct? - median?)? -2. Projections died of write amplification. ClickHouse's projections - feature revives them WITH the merge machinery paying the cost — - what changed to make it affordable? (Background merges as the - universal work-absorber.) -3. The WS/RS + tuple-mover design is an LSM with different names. Map - the four components onto topic 4's vocabulary. -4. Position lists vs bitmaps for intermediate results: when does each - win? (Selectivity — connect to your topic 11 select-vs-compact - question.) -5. M12: `WHERE n.country = 'IL'` on a dictionary-encoded property - column — write the process-compressed plan (code lookup, int - compare, positions out) and count decodes for 1% selectivity. +1. **`SUM` over RLE runs is O(runs).** Which *other* aggregates stay run-shortcuttable and + which break? Work through `MIN`/`MAX` (what does `isValueSorted()` buy — Abadi §5.2's + Figure 3 says "finding the maximum or minimum value in a sorted block is a single + operation"), `COUNT` (Figure 2's aggregator), `AVG`, then `COUNT DISTINCT` and `MEDIAN`. + For each, say whether the shortcut needs `isOneValue()`, `isValueSorted()`, both, or + whether no block property saves you. +2. **Projections died of write amplification and join-index maintenance.** ClickHouse revives + them with the merge machinery paying the cost — what changed? Name the two specific + mechanisms in the ClickHouse paper's §3.2 (lazy population from newly inserted parts only; + the optimizer choosing per part on estimated I/O cost) and say which of C-Store's two + costs each one addresses. Does either remove the need for join indexes, or did ClickHouse + sidestep that by never splitting the base table? +3. **WS/RS + tuple mover is an LSM with different names** — the paper says so itself in §1. + Map the four components onto topic 4's vocabulary, then find the one place the analogy + fails: what does C-Store's low-water-mark epoch do that an LSM compaction's sequence + numbers do not? +4. **Position lists versus bitstrings** for intermediate results: derive the crossover for + your own row count and pointer width, as Step 6 does for 1 M rows and 4-byte positions + (3.125%). Then connect it to your topic 11 select-vs-compact question — is it the same + crossover? +5. **M12.** `WHERE n.country = 'IL'` on a dictionary-encoded property column of 1 M nodes. + Write the process-compressed plan — code lookup, integer compare, positions out — and + count the decodes at 1% selectivity, against the decompress-then-process plan. Then say + which of Abadi §5.2's three properties your plan relied on. + +--- + +## Takeaway + +The 2006 thesis in one sentence: **expose the properties of compressed blocks to operators, +execute per run and per code, and decode the losers never.** + +The 2005 architecture in one more: store only sorted, compressed projections, reassemble rows +from join indexes, absorb writes in a small updatable store, and move them to the read store +in bulk. + +Score them separately, because they aged differently. The compression thesis is now +universal — Parquet, DuckDB, ClickHouse and BtrBlocks all implement it, and BtrBlocks in 2023 +is still filling in C-Store's Type 4 cell. The architecture half is more mixed: the WS/RS +split won under other names, projections lost on cost, and join indexes disappeared entirely +because everyone else kept the base table. + +The transferable habit is the properties API. Two decades of encodings — FSST, FastPFOR, +Roaring, Pseudodecimal — have been added to column stores since, and none of them required a +new join operator, because the 2006 paper made operators depend on three booleans instead of +on an encoding list. + +--- ## Done when -You can state the SIGMOD '06 thesis in one sentence ("expose encoding -properties to operators; execute per-run/per-code, decode losers -never"), and map C-Store's four big bets to their modern descendants. +Answer each before unfolding it. + +- [ ] State the SIGMOD '06 thesis in one sentence, and say what the alternative was called. + +
Answer + +*Expose the properties of compressed blocks to operators, execute per run and per code, and +decode the losers never.* + +The alternative is **eager decompression** — the classical design in which "data would be +compressed on disk and then eagerly decompressed upon being read into memory… everything read +into memory had to be decompressed whether or not it was actually used" (Abadi §2). The +intermediate position, credited there to Graefe and Shapiro and to MonetDB/X100, is **lazy +decompression**: keep it compressed in memory, decode only what an operator actually needs. +The 2006 paper's contribution is the step past that — for RLE, bit-vector and dictionary +data, many operators need never decode at all. + +§6.2 measures the gap on 1000-record sorted runs: **10.3×** for bit-vector, **3.94×** for +group-by-self dictionary, **3.3×** for RLE, **1.1×** for value-at-a-time dictionary, and +nothing at all for LZ and null suppression, which "cannot operate on encoded data". + +
+ +- [ ] C-Store stores "only projections". Say what that costs, and name the structure that + pays the cost. + +
Answer + +§2: "C-Store implements only projections… we do not store the base table(s) from which the +projection is derived." A projection is anchored on one table, may pre-join columns from +others through n:1 foreign keys, has the same row count as its anchor, and is sorted on a +declared key. + +The cost is reassembly. With no base table, answering a query that needs columns from two +differently-sorted projections means resorting one into the other's order, and the structure +that does it is the **join index**: "a collection of `(sid, storage_key)` pairs", one entry +per row, per projection pair. §2 describes it as taking "T1, sorted in some order O, and +logically resort[ing] it into the order O′ of T2". + +That is a permutation per pair of projections, and it is not static: §7's merge-out process +assigns new storage keys in the rebuilt read store, "thereby requiring join index +maintenance". So `k` sort orders cost `k` copies of the data *plus* the permutations tying +them together *plus* rewriting those permutations on every merge. That is what did not +survive — ClickHouse keeps a base table and buys extra sort orders as optional, lazily +populated projections instead. + +
+ +- [ ] Bit-vector encoding beat every other scheme by 10.3× in one experiment and was + catastrophically slow in another. Explain both. + +
Answer + +It wins when the query wants **a set of positions**, and loses when the query wants **values +in position order**. + +The win (§6.2): for a `GROUP BY` on a column with `c` distinct values, the aggregation cost is +proportional to `num_distinct_values`, not `num_tuples` — a `COUNT` per group is the size of +one bitmap. At 40 distinct values over 100 million rows that is a different complexity class, +hence 10.3×. §6.5 extends it to predicates: bit-vector encoding "is already storing the result +of the predicate as it already contains a position list for each unique value in the column", +so an equality predicate is a projection, not a scan. + +The loss (§6.5, Figure 9(b)): reverse the roles so the bit-vector column is the one being +*position filtered* for its values, and "the query requires the values of the bit-vector +column in position order which forces decompression, which has already been shown to be +slow". Reading the *i*-th value means consulting every bitmap. + +The sizing constraint is separate and just as fatal: `c` bits per row versus `w` bits raw +means it only shrinks below cardinality `w` — "as soon as the column cardinality is more than +32, type-2 compression is no longer more compressed than the original 32-bit data" (§6.1). + +The authors' own conclusion is the one to keep: "the proper choice of encoding type for a +column depends not just on data characteristics, but also on the expected query workload." + +
+ +- [ ] The properties API has exactly three predicates. Name them, and say why none of them + mentions an encoding. + +
Answer + +`isOneValue()` — the block holds a single value at many positions. `isValueSorted()` — the +block's values are sorted (trivially true when there is one value). `isPosContig()` — the +block covers a consecutive range of the column. + +They avoid naming encodings because naming them is what causes the combinatorial explosion +§5.2 opens with: "there would end up being n versions of each operator… Operators that take +two inputs (like joins) would need n² versions." + +The reason three booleans suffice is that the optimizations were never really *about* the +encodings. RLE and bit-vector both "encoded multiple positions for the same value" — RLE +consecutively, bit-vector not — so both admit the same shortcut, differing in exactly one +predicate, `isPosContig()`. Figure 3's optimizations are stated accordingly: they work "in +general, not just for RLE" and "in general, not just for bit-vector compression". Figure 2's +`Count` aggregator branches on `isOneValue()` alone and handles both. + +The practical test of the abstraction is that FSST, FastPFOR, Roaring bitmaps and +Pseudodecimal have all been added to column stores since 2006 without anyone writing a new +join operator. + +
+ +- [ ] The 2006 machine read cold data at 50–60 MB/s. Compute the factor against this topic's + measured scan floor, and say why the paper's conclusion survives it. + +
Answer + +`FINDINGS.md` row 12 records a scan floor of **24–57 GB/s** on a machine with roughly +**150 GB/s** of memory bandwidth. Against the paper's disk: `57 GB/s / 55 MB/s` = `57,000 / +55` ≈ **1,036×**. Three orders of magnitude, and the bottleneck moved from disk to memory +bandwidth on the way. + +The conclusion survives because it was never a bandwidth argument. §6.2 states the mechanism +as a complexity claim — aggregation cost is proportional to `num_tuples` uncompressed, but to +`num_tuples / avg_run_len` for RLE, `num_tuples / dict_entry_size` for multi-value dictionary +and `num_distinct_values` for bit-vector. Dividing the work by the run length is an +algorithmic win, immune to changes in the constant. The authors said so explicitly: "even on +a machine with a much faster I/O or a much slower CPU, compressing data and operating +directly on it will be beneficial." + +What did *not* survive is every absolute number, which is why this topic re-measures rather +than quotes. And the corollary discipline: always say whether a GB/s figure counts compressed +bytes moved or logical bytes processed — at our Step 6 column's 7.99× ratio the same scan is +honestly describable as 3.0 GB/s or 24 GB/s. When it is describable as 19,047,619 GB/s — +about 127,000× a 150 GB/s bus — it is a hoisted loop, not a discovery. + +
+ +--- ## References **Papers** -- Stonebraker et al. — "C-Store: A Column-oriented DBMS" (VLDB 2005) - — read for the architecture bets and which survived twenty years -- Abadi, Madden, Ferreira — "Integrating Compression and Execution in - Column-Oriented Database Systems" (SIGMOD 2006) — the - compression-aware-execution experiment; internalize the findings list - above + +- Mike Stonebraker et al. *C-Store: A Column-oriented DBMS*. VLDB 2005. + + Read §2 (data model), §3.1 (encodings), §7 (tuple mover), §8.1–8.2 (operators and + optimizer), §9 (results — with its read-only caveat). Skim §5, §6. +- Daniel Abadi, Samuel Madden, Miguel Ferreira. *Integrating Compression and Execution in + Column-Oriented Database Systems*. SIGMOD 2006. + + Read §4 (the six schemes), §5 (the compressed block API), §6.1–6.5 (the experiments), + §7 + Figure 10 (the decision tree). +- Daniel Abadi, Daniel Myers, David DeWitt, Samuel Madden. *Materialization Strategies in a + Column-Oriented DBMS*. ICDE 2007 — where "late materialization" gets its name. Optional; + Step 4 has what you need. + +**Code** + +- [duckdb/duckdb](https://github.com/duckdb/duckdb) @ `6c0c1a68` — + `src/storage/compression/rle.cpp:113` sizes an RLE segment as + `(sizeof(rle_count_t) + sizeof(T)) * seen_count`, the two-field pair to C-Store §3.1's + three-field triple. Covered properly in + [reading-duckdb-compression.md](reading-duckdb-compression.md). + +**In this topic** + +- [reading-clickhouse-paper.md](reading-clickhouse-paper.md) — projections, twenty years + later and lazily populated (§3.2), for question 2 +- [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md) — what finally filled C-Store's + Type 4 cell, "many distinct values, foreign order" +- [reading-duckdb-compression.md](reading-duckdb-compression.md) — the properties API as + shipped, and who picks the encoding +- `FINDINGS.md` row 12 — the measured scan floor (24–57 GB/s on a ~150 GB/s machine) and the + 19,047,619 GB/s hoisted-loop bug diff --git a/topics/12-columnar-analytics/reading-duckdb-compression.md b/topics/12-columnar-analytics/reading-duckdb-compression.md index f6cb471..11fca4a 100644 --- a/topics/12-columnar-analytics/reading-duckdb-compression.md +++ b/topics/12-columnar-analytics/reading-duckdb-compression.md @@ -1,187 +1,459 @@ # DuckDB's encoding zoo: analyze, score, commit -Who picks the encoding? DuckDB's answer: nobody — race every candidate -encoder over the column and let the byte estimates decide, per column, -per row group. Before you open the C++, this chapter builds the ideas -one at a time: what a lightweight encoding is, what unit the choice is -made for, the two-pass lifecycle that makes racing affordable, the -random-access constraint that shapes the whole menu, two encoders -end-to-end (RLE, bit-packing), the string stack, and the zone maps -that filter pushdown lands on. Then it hands you the file and line -anchors to watch each piece in the source. +Who picks the encoding? ClickHouse says *you do*, in the DDL. BtrBlocks says *a sampler +does*. DuckDB's answer is the third one: **nobody guesses — run every candidate encoder over +the real data, take the smallest estimate, and only then compress.** It is +benchmark-before-committing, wired into a production storage engine's checkpoint path. + +Before you open the C++, this chapter builds the ideas one at a time: what a lightweight +encoding is, the four nested units the decision is made for, the two-pass lifecycle that +makes racing affordable, the deliberate bias in the scoring, the random-access constraint +that shapes the whole menu, two encoders end to end, the string stack, and the zone maps +that filter pushdown physically lands on. Then it hands you file-and-line anchors for each +piece. + +All anchors are `duckdb/duckdb@6c0c1a68`; check them with +`tools/pinned-source.py show duckdb/duckdb -r A:B`. + +Read [reading-cstore-compression.md](reading-cstore-compression.md) first if you have not. +Step 6 below is Abadi's 2006 thesis running in production, and it is much more striking when +you have read the paper it descends from. + +--- ## The problem in one sentence -Analytics scans are memory-bound, so bytes moved ≈ time — an encoding -that shrinks a column 4× makes the scan up to 4× faster — but the -right encoding differs per column and per chunk of rows, and a wrong -guess can *inflate* the data; DuckDB refuses to guess. +Analytic scans are memory-bound, so bytes moved ≈ time — an encoding that shrinks a column +6× can make the scan several times faster — but the right encoding differs per column *and* +per 2,048 rows, a wrong guess can *inflate* the data, and the only way to know which is +right is to try them all on the actual bytes. + +--- ## The concepts, step by step -### Step 1 — lightweight encoding: compression the scan can execute over +### Step 1 — Lightweight encoding: compression the scan can execute over + +> **In:** a column of values, and a scan loop that has to touch every one of them. +> **Out:** the distinction between an encoding and a block compressor, and why only one of +> them belongs inside the loop. + +An **encoding** here is not gzip. It is a reversible rewrite of a column's values that +exploits a *pattern in the data* — repetition, a narrow range, few distinct values — and +whose decode is a handful of arithmetic instructions, cheap enough to run inside the scan +loop. A **block compressor** (gzip / zstd class) treats bytes as opaque, achieves better +ratios on average, and must inflate a whole block before you can read anything in it. + +Two terms you will need throughout, defined here: + +- **Run-length encoding (RLE)** — store each maximal run of equal values as one + `(value, count)` pair. +- **Bit-packing** — store integers in `ceil(log2(range))` bits instead of the type's full + width, with no byte alignment between them. + +Both appear in DuckDB's menu; so do **frame-of-reference**, **delta encoding**, **dictionary +encoding** and **FSST**, each defined at the step that uses it. + +DuckDB's registered menu, in the order the engine considers them +(`src/function/compression_config.cpp:17-35`): `CONSTANT`, `UNCOMPRESSED`, `RLE`, +`BITPACKING`, `DICTIONARY`, `CHIMP`, `PATAS`, `ALP`, `ALPRD`, `FSST`, `ZSTD`, `ROARING`, +`EMPTY` (all-valid validity masks), `DICT_FSST`. Thirteen real functions plus the `AUTO` +sentinel. Exactly one of them — `ZSTD` — is a block compressor, and Step 5 explains why it +is there at all. + +**Why it matters:** since a scan's cost is bytes moved, a lightweight encoding is not a +space feature that costs time; it *is* the performance feature. That inversion is this +topic's thesis, and DuckDB is where you can watch it being decided per column. + +### Step 2 — Four nested units, and the one the decision is made for -An **encoding** here is not gzip. It is a reversible rewrite of a -column's values that exploits a *pattern in the data* — repetition, -small ranges, few distinct values — and whose decode is a handful of -arithmetic instructions, cheap enough to run inside the scan loop. -Contrast a **block compressor** (gzip/zstd class): it treats bytes as -opaque, achieves great ratios, but must decompress a whole block -before you can read anything. +> **In:** a table being checkpointed to disk. +> **Out:** the four sizes that matter, with the arithmetic relating them. -Concrete: a sorted column of 1M timestamps where each value repeats -~1000 times stores as ~1000 `(value, run_length)` pairs — 16 KB -instead of 8 MB, a 500× reduction — and decoding a run is one branch. -Why it matters: since scans move bytes, a lightweight encoding is not -a space feature, it *is* the performance feature. The topic's thesis -— compression IS performance — starts here. +| Unit | Size | Defined at | +| --- | --- | --- | +| **row group** — a horizontal slice of the table | **122,880** rows | `src/include/duckdb/storage/storage_info.hpp:26` — `#define DEFAULT_ROW_GROUP_SIZE 122880ULL` | +| **column segment** — one column's data within a row group, encoded one way | ≤ a row group | the checkpoint unit; see Step 3 | +| **vector** — the execution and analyze unit | **2,048** values | `src/include/duckdb/common/vector_size.hpp:16` — `#define DEFAULT_STANDARD_VECTOR_SIZE 2048U` | +| **bitpacking group** — the *mode* unit inside a bit-packed segment | **2,048** values | `src/storage/compression/bitpacking.cpp:25` | -### Step 2 — the unit of choice: row group, column segment, vector +The relation is exact, and the codebase asserts it: `storage_info.hpp:394` refuses to compile +unless `DEFAULT_ROW_GROUP_SIZE % STANDARD_VECTOR_SIZE == 0`. So -DuckDB doesn't pick one encoding for a whole table, or even a whole -column. Tables are stored as **row groups** (horizontal slices of -122,880 rows); within a row group each column is its own sequence of -**segments** (contiguous encoded blocks); and all execution moves data -in **vectors** of 2,048 values (topic 11's unit). The encoding -decision is made **per column, per row group**. +``` +122,880 / 2,048 = 60 vectors per row group +``` + +— sixty `analyze` calls per candidate encoder per column, and sixty independently-moded +bitpacking groups inside a bit-packed segment. ``` table - └─ row group (122,880 rows) ← the decision unit - ├─ column "ts" → segments encoded as DELTA_FOR - ├─ column "city" → segments encoded as DICTIONARY - └─ column "id" → segments encoded as BITPACKING + └─ row group (122,880 rows = 60 vectors) <- the encoding decision unit + ├─ column "ts" -> segment encoded as BITPACKING, 60 groups, each with its own mode + ├─ column "city" -> segment encoded as DICT_FSST + └─ column "id" -> segment encoded as BITPACKING ``` -Why it matters: data shape drifts within a table (early rows sorted, -late rows random; one column low-cardinality, another unique), so a -global choice is always wrong somewhere. Per-row-group choice bounds -the damage of any one bad fit to 122,880 rows. +Note the third line of `bitpacking.cpp:25`: the group size is +`STANDARD_VECTOR_SIZE > 512 ? STANDARD_VECTOR_SIZE : 2048`, so on a build with a smaller +vector size the group is still 2,048 — the mode granularity is pinned independently of the +execution granularity. + +**Why it matters:** data shape drifts within a table — early rows sorted, late rows random; +one column low-cardinality, the next unique. A global choice is always wrong somewhere. +Per-row-group choice bounds the damage of a bad fit to 122,880 rows, and bitpacking's +per-group modes bound it further, to 2,048. + +### Step 3 — analyze → score → compress: one scan, every candidate + +> **In:** a column being checkpointed, and the thirteen registered functions. +> **Out:** the selection loop, and the four ways a candidate can lose. + +The contract is documented in the framework header itself, +`src/include/duckdb/function/compression_function.hpp:130-138`: -### Step 3 — analyze → score → compress: race the encoders, cheapest estimate wins +> "The analyze functions are used to determine whether or not to use this compression +> method… 1. The `init_analyze` is called to initialize the analyze state of every candidate +> compression method. 2. The `analyze` method is called with all of the input data in the +> order in which it must be stored. `analyze` can return 'false'. In that case, the +> compression method is taken out of consideration early. 3. The `final_analyze` method is +> called, which should return a score for the compression method… The system then decides +> which compression function to use based on the analyzed score." -For each column of each row group, DuckDB runs *every* candidate -encoder over the data in a dry-run **analyze** pass that only counts -what the encoded size *would* be, picks the smallest estimate, and -only then lets the winner actually **compress**. The selection loop, -condensed: +Three typedefs implement it — `compression_init_analyze_t` at `:139`, +`compression_analyze_t` at `:140`, `compression_final_analyze_t` at `:141` — and the winner's +`compression_compress_data_t` at `:148`. + +The loop that drives them is `ColumnDataCheckpointer::DetectBestCompressionMethod`, +`src/storage/table/column_data_checkpointer.cpp:172-278`. The shape is not one pass per +encoder; it is **one scan, feeding every candidate the same vector**: ```rust -// per column, per row group: race every encoder, cheapest estimate wins -fn choose(col: &RowGroupColumn, candidates: &[&dyn Encoder]) -> &dyn Encoder { - let mut best = (f64::INFINITY, candidates[0]); - for enc in candidates { - let mut st = enc.init_analyze(); - if !col.vectors().all(|v| enc.analyze(&mut st, v)) { - continue; // encoder drops out early +// ILLUSTRATION — Rust sketch of C++ control flow, not quoted. The real loop is +// duckdb/duckdb@6c0c1a68 src/storage/table/column_data_checkpointer.cpp:172-278; +// the single shared scan is at :200-217 and the score comparison at :245-256. +fn detect_best(col: &Column, mut candidates: Vec) -> Encoder { + let mut states: Vec> = candidates.iter().map(|e| e.init_analyze()).collect(); + + for vector in col.vectors() { // ONE pass over the data (:200) + for (enc, st) in candidates.iter().zip(states.iter_mut()) { + if let Some(s) = st { + if !enc.analyze(s, vector) { *st = None; } // dropped for good (:211-214) + } } - let score = enc.final_analyze(st); // ESTIMATED bytes — no compressing yet - if score < best.0 { best = (score, *enc); } } - best.1 // winner re-reads the whole column in compress_data + + let mut best = (usize::MAX, None); + for (enc, st) in candidates.iter().zip(states) { + let Some(s) = st else { continue }; + let score = enc.final_analyze(s); // ESTIMATED bytes (:245) + if score == INVALID_INDEX { continue } // self-disqualified (:248-250) + if score < best.0 { best = (score, Some(enc)); } // strict < : ties go to the + } // earlier-registered function + best.1.expect("no suitable compression method") // FatalException at :265-268 } ``` -The lifecycle, as the framework header documents it: +Four ways to lose, all in that loop: + +1. **Drop out mid-scan** — `analyze` returns `false` and the state is nulled for the rest of + the pass (`:211-214`), so a hopeless candidate costs nothing further. `BitpackingAnalyze` + at `bitpacking.cpp:318-334` does this when one group would not fit in a block, and again + whenever a value overflows the state. +2. **Self-disqualify at the end** — `final_analyze` returns `DConstants::INVALID_INDEX` + (`:247-250`). `BitpackingFinalAnalyze` at `bitpacking.cpp:337-344` does exactly this when + its final `Flush` fails. +3. **Lose on score** — `:252`, `score < best_score`, strictly. On a tie the **earlier** + entry in `compression_config.cpp:17-35` wins, which is why `CONSTANT` and `UNCOMPRESSED` + head the list. +4. **Never run at all** — `:196`, `skip_scan`. If the column's DDL names a compression type, + the analyze scan is skipped outright. That is the ClickHouse-style declaration escape + hatch, sitting inside the automatic system. `PRAGMA force_compression` (`:185-188`) is the + session-level version, and is how you run this topic's experiments. + +The cost is honest and stated by the design: this is a **two-pass** ingest. DuckDB reads the +whole column once to choose and once to compress. And if nothing qualifies, `:265-268` +throws — which is why `UNCOMPRESSED` is in the menu, as the candidate that always scores. + +**Why it matters:** it is the same discipline this repo's `verify.sh` enforces on itself — +measure, then commit — implemented in a storage engine's hot path. And it is the direct +alternative to the two other answers in this topic, which Step 4 prices. + +### Step 4 — The score is deliberately biased, and sometimes sampled rather than measured + +> **In:** a `final_analyze` about to return an estimated byte count. +> **Out:** the two ways that number is not a byte count, and why both are correct. + +**The bias.** Dictionary encoding's `final_analyze`, `dictionary_compression.cpp:85-98`, +computes the real required space and then inflates it on the way out: + +```cpp +// duckdb/duckdb@6c0c1a68 src/storage/compression/dictionary_compression.cpp:92-97 + 92 auto width = BitpackingPrimitives::MinimumBitWidth(state.current_unique_count + 1); + 93 auto req_space = DictionaryCompression::RequiredSpace(state.current_tuple_count, state.current_unique_count, + 94 state.current_dict_size, width); + 95 + 96 const auto total_space = state.segment_count * state.info.GetBlockSize() + req_space; + 97 return LossyNumericCast(DictionaryCompression::MINIMUM_COMPRESSION_RATIO * float(total_space)); +``` + +with `MINIMUM_COMPRESSION_RATIO = 1.2F` +(`src/include/duckdb/storage/compression/dictionary/common.hpp:20`). FSST does the same, +`src/storage/compression/fsst.cpp:37` and `:202`. So both schemes report themselves **20% +larger than they are**, and must beat the alternatives by more than 20% to be chosen. + +Work it on this topic's shared column — 1,000,000 `INT64` values, 200 distinct, average run +8. Dictionary's true cost (Step 8 derives it) is 1,001,600 B and bit-packing's is +≈1,259,664 B: + +``` +true: 1,259,664 / 1,001,600 = 1.258 dictionary wins by 25.8% +reported: 1,259,664 / 1,201,920 = 1.048 dictionary wins by 4.8% + ^ 1,001,600 x 1.2 +``` + +Still a win — but a 5% margin instead of a 26% one, and a column only slightly less +favourable would flip. The bias buys back the costs that do not appear in a byte count: +dictionary and FSST both add an indirection on every decoded value, and `fetch_row` on them +is slower than on a bit-packed segment. A scheme whose score is *its own size* would be +chosen too often. + +**The sampling.** The header says `analyze` "is called with all of the input data", and for +RLE, bit-packing and dictionary that is literally true — every one of the 60 vectors. Two +encoders quietly do less: + +| Encoder | Sample | Fraction of a row group | +| --- | --- | --- | +| FSST — `fsst.cpp:38`, `ANALYSIS_SAMPLE_SIZE = 0.25` | 25% of the strings | 25% | +| ALP — `src/include/duckdb/storage/compression/alp/alp_constants.hpp:19-23` | `RG_SAMPLES = 8` vectors × `SAMPLES_PER_VECTOR = 32` values | `8 × 32 / 122,880` = **0.208%** | +| BtrBlocks, for comparison (SIGMOD 2023 §3.1) | 10 runs of 64 values per 64,000-value block | 1% | + +ALP's sampling is *five times sparser than BtrBlocks'*, in the system that is otherwise the +poster child for exhaustive analysis. Its stride is spelled out at `:22-23`: "We calculate +how many equidistant vector we must jump within a rowgroup", `(122,880 / 8) / 2,048` = +`15,360 / 2,048` = **7** vectors between samples. + +**Why it matters:** "DuckDB analyzes, BtrBlocks samples" is the tidy version of the story and +it is not quite true. The real design rule is *sample when training the model is the +expensive part* — FSST has to build a symbol table, ALP has to search an exponent/factor pair +— and *measure exhaustively when the analyze pass is just counting*. Bring that distinction +to question 1 rather than the tidy version. + +### Step 5 — `fetch_row`: the random-access constraint that shapes the menu + +> **In:** an operator that wants row 1907 of a segment and nothing else. +> **Out:** the contract entry that decides which encodings are admissible at all. + +Every encoding must implement `compression_fetch_row_t` +(`compression_function.hpp:171-173`), documented one line above as "Function prototype used +for reading a single value". Late-materialized fetches and index joins ask for single rows, +not vectors — Step 4 of [reading-cstore-compression.md](reading-cstore-compression.md) is +where that pattern comes from. + +The header states the consequence explicitly at `:174-176`, in the doc comment for +`compression_skip_t`: "Function prototype used for skipping 'skip_count' values, **non-trivial +if random-access is not supported for the compressed data.**" + +That single requirement explains the menu's shape. RLE can binary-search its run counts; +bit-packing computes a bit offset; dictionary indexes its selection buffer; FSST decodes one +string because its symbol table is static and stateless (see +[reading-btrblocks-fsst.md](reading-btrblocks-fsst.md)). A block compressor cannot: fetching +one row means inflating the whole block, and its internal state is path-dependent. + +The framework has two more entries that matter here, both added since the original design: +`compression_select_t` at `:164-166`, "reading a subset of the values of a vector indicated +by a selection vector", and `compression_filter_t` at `:167-170`, "**applying a filter to a +vector while scanning that vector**". Those are filter pushdown reaching all the way into +the encoding — topic 10's plan-level rewrite, landing on physical bytes. Step 6 shows what +RLE does with it. + +**Why it matters:** the storage format is negotiated with the *executor*, not chosen for +ratio alone. Zstd is in the menu (`compression_config.cpp:29`) and it does win sometimes — +but it wins on score, against a 20% handicap on its rivals, and it pays for every point +lookup afterwards. + +### Step 6 — RLE end to end: Abadi 2006, shipped + +> **In:** `src/storage/compression/rle.cpp`, 638 lines. +> **Out:** the whole framework contract in its simplest instance, plus two optimisations you +> have already read the paper for. + +RLE is the smallest encoder that exercises every part of the contract. Read it first; every +other encoder repeats its registration pattern. + +**Analyze** is a run counter. `RLEAnalyzeState` at `:86-91` wraps an `RLEState`; +`RLEAnalyze` at `:99-110` walks the vector calling `Update`, which increments `seen_count` +on each new value. **Score**, `RLEFinalAnalyze` at `:113-116`, is one line: ``` - for each candidate encoder: (per column, per row group) - init_analyze - analyze(vector) per vector — may return false = drop out early - final_analyze -> SCORE (estimated bytes; lower wins) - winner runs compress_data over the same data again - scans use scan_vector / scan_partial; - point lookups use fetch_row <- random access into encodings! +return (sizeof(rle_count_t) + sizeof(T)) * rle_state.state.seen_count; ``` -The cost: this is a **two-pass design** — DuckDB pays a full extra -read of the data at ingest just to CHOOSE the encoding. That is -benchmark-before-committing, built into a production storage engine. -(You can override it with `PRAGMA force_compression` for experiments.) +Bytes = runs × (count size + value size). No bias multiplier — RLE reports its true size. +Note what it is *not*: C-Store's RLE record is a **triple** `(value, start_pos, run_length)` +(C-Store §3.1), because C-Store needed positions addressable. DuckDB stores a **pair** and +reconstructs positions by walking the counts, which is a third less space and a slower +`fetch_row`. + +**Compress** writes two arrays into the segment — values from the header forward, counts from +`rle_count_offset` on (`RLECompressState` at `:126`, the scan state's `data_pointer` and +`index_pointer` at `:313-314`). + +**Registration**, `GetRLEFunction` at `:568-576`, is the pattern to grep for in every other +encoder: one `CompressionFunction` constructor call bundling `RLEInitAnalyze`, `RLEAnalyze`, +`RLEFinalAnalyze`, the three compress functions, `RLEInitScan`, `RLEScan`, `RLEScanPartial`, +`RLEFetchRow`, `RLESkip`, and — at `:574-575` — `RLESelect` and `RLEFilter`. (`:578-584` +then disables `filter` for `BOOL`.) + +Now the two things worth the trip. Both are the 2006 paper, in C++: + +**`isOneValue()`, by another name.** `CanEmitConstantVector` at `:333-347` asks whether the +current run covers an entire 2,048-value vector; if so `RLEScanConstant` at `:349-359` sets +`VectorType::CONSTANT_VECTOR` and writes **one** value: + +``` +result.SetVectorType(VectorType::CONSTANT_VECTOR); +result_data[0] = scan_state.data_pointer[scan_state.entry_pos]; +scan_state.position_in_entry += scan_count; +``` + +2,048 rows produced, one value written. Abadi §5.2's first block property, shipped. + +**Predicate once per run.** `RLEFilter` at `:447-490` is the clearest statement of the whole +thesis anywhere in this topic. Its own comments: + +> "we haven't applied the filter yet — **apply the filter to all RLE values at once**" +> (`:456-457`) +> "**execute the filter over all runs at once**" (`:463`) +> "early-out, **no runs match the filter so the filter can never pass**" (`:478`) -### Step 4 — fetch_row: the random-access constraint that shapes the menu +It builds a `matching_runs` bool array over the run *values* (`:460-475`), caches it on the +scan state (`:310`), and can abandon the entire segment at `:477-481` without decoding a +single row. A predicate over a segment with 125,000 runs and 1,000,000 rows is evaluated +125,000 times, not 1,000,000. That is Abadi §6.2's `num_tuples / avg_run_len`, nineteen years +later, in a shipping engine. -Every encoding must also answer a point request: "give me row 1907 of -this segment" (`fetch_row` in the framework contract), because -operators like index joins and late-materialized fetches ask for -single rows, not whole vectors. An encoding qualifies for the menu -only if it can decode one value without decoding everything before it -— or fake it acceptably. +**Why it matters:** if you read only one file in DuckDB's compression directory, read this +one. It is where the paper stops being history. -This single constraint explains the menu's shape: RLE, dictionary, -bit-packing, and FSST can all jump to (or near) a single value; -a heavyweight block codec (zstd) cannot — fetching one row means -decompressing the whole block. That is why zstd is the **last -resort** fallback, not a default. Why it matters: the storage format -is negotiated with the *executor*, not chosen for ratio alone. +### Step 7 — Bit-packing: four modes decided by one comparison -### Step 5 — RLE: the simplest complete encoder +> **In:** a group of 2,048 integers. +> **Out:** the four modes, the single arithmetic test that chooses between them, and the +> function that is analyze and compress at the same time. -RLE (**run-length encoding** — store each maximal run of equal values -as one `(value, count)` pair) is the smallest example that exercises -the entire framework contract. Its analyze pass just counts runs; its -score is `runs × (value_size + count_size)`; its compress pass writes -two interleaved arrays (values, counts). +`BitpackingMode` is an enum of six values, four of them real +(`src/include/duckdb/storage/compression/bitpacking.hpp:15`): ``` - input: 7 7 7 7 7 7 9 9 9 2 2 2 2 ... (1M rows, ~1000 runs) - encoded: values [7, 9, 2, ...] counts [6, 3, 4, ...] - score: 1000 × (8 B + 2 B) = 10 KB vs 8 MB raw +enum class BitpackingMode : uint8_t { INVALID, AUTO, CONSTANT, CONSTANT_DELTA, DELTA_FOR, FOR }; ``` -RLE wins on sorted or low-cardinality data and loses catastrophically -on random data (1M runs of length 1 = *bigger* than raw — which is -exactly what the score detects before any bytes are written). Read -this encoder first in the source; every other encoder repeats its -registration pattern. +Two definitions first. **Frame of reference (FOR)** stores the group's minimum once, then +bit-packs each value's offset from it — turning 1,000,000,007 … 1,000,000,900 into 10-bit +offsets, 64 bits down to 10, a 6.4× cut. **Delta encoding** stores each value's difference +from its predecessor; `DELTA_FOR` delta-encodes first and then applies FOR to the deltas, +which is what catches timestamps and sequences. -### Step 6 — bit-packing: four encodings in one function +The mode is chosen in `BitpackingState::Flush`, `bitpacking.cpp:204-271`, and it is a +**priority cascade**, not a race: -Bit-packing stores integers in `ceil(log2(max - min + 1))` bits -instead of 64 — but DuckDB's `bitpacking.cpp` is really four encodings -picked *per group of 2,048 values* by trying each and computing the -width it would need: +| Order | Line | Test | Mode | What it stores | +| --- | --- | --- | --- | --- | +| 1 | `:209` | `all_invalid \|\| maximum == minimum` | `CONSTANT` | one value + a metadata word | +| 2 | `:219` | `maximum_delta == minimum_delta` | `CONSTANT_DELTA` | base + delta + a metadata word | +| 3 | `:230-237` | `!prefer_for` | `DELTA_FOR` | FOR value + width + delta offset + packed deltas | +| 4 | `:257` | `can_do_for` | `FOR` | packed offsets + FOR value + width | +| — | `:270` | none of the above | `return false` | the encoder disqualifies itself | + +The one comparison that decides 3 versus 4 is `:230-235`: ``` - all equal -> CONSTANT (store 1 value) - equal deltas -> CONSTANT_DELTA (store base + delta) - clustered -> FOR: store min, bit-pack (value - min) - sequential-ish -> DELTA_FOR: delta-encode, then FOR the deltas +delta_required_bitwidth = MinimumBitWidth(min_max_delta_diff) +regular_required_bitwidth = MinimumBitWidth(min_max_diff) +prefer_for = can_do_for && delta_required_bitwidth >= regular_required_bitwidth ``` -FOR (**frame of reference** — store the group's minimum once, then -only each value's offset from it) turns values like -1,000,000,007…1,000,000,900 into 10-bit offsets: 64 bits → 10 bits, -a 6.4× cut. DELTA_FOR (delta-encode first — store differences from -the previous value — then FOR the deltas) catches timestamps and -sequences. Per-2048-group modes mean **one column segment mixes -encodings** — the decision granularity is even finer than the -row-group race of Step 3. Why it matters: the arithmetic that picks -the mode is the same score-then-commit discipline, recursed one level -down. +Ties go to plain `FOR` — note the `>=`. That is the right default: `DELTA_FOR` stores an +extra `sizeof(T)` delta offset (`:249`) and its decode needs a prefix sum (`:664`, `:824`, +`:899`), so at equal width it is strictly worse. + +And the detail worth the whole step: **analyze and compress are the same function.** +`BitpackingFinalAnalyze` at `:337-344` calls `Flush()` and returns +`total_size`. `EmptyBitpackingWriter` at `:47-63` is a struct whose `WriteConstant`, +`WriteConstantDelta`, `WriteDeltaFor` and `WriteFor` all have empty bodies. So the estimate is +not a model of the encoder — it is the encoder, with the stores compiled out. The score cannot +drift from reality, because there is only one code path. + +Each group's chosen mode is packed into a single 32-bit word with its offset — `EncodeMeta` +at `:34-39` puts the mode in the high 8 bits, the offset in the low 24 (`0x00FFFFFF`), and +`DecodeMeta` at `:40-45` pulls them back out. Four bytes of metadata per 2,048 values. + +**Why it matters:** this is the score-then-commit discipline of Step 3 recursed one level +down, at 2,048-value granularity, and implemented so that the two passes cannot disagree. +It is the pattern to copy the next time you write an estimator. + +### Step 8 — The string stack: dictionary retired, FSST, both, then give up + +> **In:** a `VARCHAR` column. +> **Out:** the cascade of string encodings, and one fact the docs will not tell you. + +**Dictionary encoding** stores each distinct string once in a dictionary and replaces the +column with integer ids into it. The segment layout is drawn in a comment at +`dictionary_compression.cpp:14-44`: a header, a **selection buffer** (`uint16_t` per tuple → +index-buffer slot), an **index buffer** (`uint16_t` per distinct string → offset into the +dictionary), and the dictionary itself, "the string data without lengths". Its score, +`:85-98`, bit-packs the ids at `MinimumBitWidth(unique_count + 1)` — mind the `+1`, which +means 255 distinct values need **9** bits, not 8 — and applies the 1.2× bias of Step 4. + +Now the fact worth checking the source for. `DictionaryCompressionStorage::StringInitAnalyze` +at `:70-78` opens with: + +```cpp +// duckdb/duckdb@6c0c1a68 src/storage/compression/dictionary_compression.cpp:70-77 + 70 unique_ptr DictionaryCompressionStorage::StringInitAnalyze(ColumnData &col_data, PhysicalType type) { + 71 auto &storage_manager = col_data.GetStorageManager(); + 72 if (StorageManager::TargetAtLeastVersion(StorageVersion::V1_3_0, storage_manager.GetStorageVersion())) { + 73 // dict_fsst introduced - disable dictionary + 74 return nullptr; + 75 } + 76 + 77 return make_uniq(col_data.GetBlockManager()); +``` + +Plain `DICTIONARY` is **retired** for storage version 1.3.0 and later. It returns a null +analyze state, which the loop at `column_data_checkpointer.cpp:237-239` skips. It is still in +the menu, and still the encoder to read for the mechanism — but on a database you create +today it never wins, because `DICT_FSST` (`compression_config.cpp:33-34`) supersedes it: a +dictionary whose entries are themselves FSST-compressed, in `src/storage/compression/dict_fsst/`. + +**FSST** — a fast static symbol table mapping up to 255 substrings of 1–8 bytes to one-byte +codes — catches the case dictionary encoding cannot: strings that are *distinct but similar*, +like URLs and email addresses, where there is nothing to deduplicate but plenty to +substitute. Its DuckDB integration is in `fsst.cpp`; the paper and the algorithm get their own +chapter, [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md). Three numbers to carry over: +`MINIMUM_COMPRESSION_RATIO = 1.2` at `:37`, `ANALYSIS_SAMPLE_SIZE = 0.25` at `:38`, and +`duckdb_fsst_decompress` at `:470`. + +**Zstd** (`src/storage/compression/zstd.cpp`, registered at `compression_config.cpp:29`) is +the heavyweight fallback for whatever nothing else catches — accepted because sometimes ratio +genuinely beats access, and constrained by the `fetch_row` cost of Step 5. + +**Why it matters:** the string stack is where the ratio-versus-access trade is sharpest, +because strings are where the ratios are largest. It is also a live part of the tree — +`DICT_FSST` replacing `DICTIONARY` is a change you can date from the source, and the kind of +thing a guide written from documentation would miss. -### Step 7 — the string stack: dictionary, FSST, both, then give up - -Strings get a cascade of increasingly aggressive encodings. **Dictionary -encoding** (store each distinct string once in a dictionary; the -column becomes integer ids into it) wins when there are few distinct -values — 1M rows of 200 country names become 1M small ints (which then -get bit-packed, Step 6) plus a 200-entry dictionary; string -comparisons become int comparisons. **FSST** (fast static symbol table -— a 255-entry table mapping 1–8-byte substrings to 1-byte codes) -catches columns where strings are *distinct but similar* (URLs, -emails) that dictionary can't dedup, while keeping every string -individually decodable — the Step 4 constraint again. `dict_fsst/` -stacks both: a dictionary whose entries are FSST-compressed. And -`zstd.cpp` is the heavyweight fallback for whatever nothing else -catches — accepted only because sometimes ratio beats access. (FSST -gets its own chapter: [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md).) - -### Step 8 — zone maps: skip the segment instead of decoding it - -A **zone map** is a per-segment min/max statistic kept alongside the -data; before scanning a segment, the engine checks the filter against -the min/max and can skip the whole segment — no read, no decode: +### Step 9 — Zone maps: five answers, not three + +> **In:** a filter and a segment about to be scanned. +> **Out:** the three-valued-logic result the storage layer returns, and the two extra cases +> NULLs force. + +A **zone map** (equivalently a **min-max index**) is a per-segment summary of the values it +holds; before scanning, the engine checks the filter against it and may skip the segment +entirely — no read, no decode: ``` WHERE ts BETWEEN '2026-01-01' AND '2026-01-02' @@ -190,82 +462,376 @@ the min/max and can skip the whole segment — no read, no decode: seg 2 [ts: 2026-01-05 .. 2026-02-11] -> skip ``` -DuckDB's check returns three-valued answers: **always-false** (skip -the segment), **always-true** (scan it *and drop the filter* — every -row passes, so why test each one), or no-pruning. String columns keep -min/max *prefixes*, not full values. The catch: zone maps only prune -if the data is clustered on the filter column — on random data every -zone spans the whole domain and nothing skips. Why it matters: this is -where topic 10's filter pushdown physically lands — a plan-level -rewrite becomes a storage-level skip. +`ColumnData::CheckZonemap` at `src/storage/table/column_data.cpp:423-462` does it, delegating +to `expr_filter.CheckStatistics(...)` at `:442-443`. The return type is +`FilterPropagateResult`, and it has **five** values, not three +(`src/include/duckdb/common/enums/filter_propagate_result.hpp:15-21`): + +| Value | Meaning | What the scan does | +| --- | --- | --- | +| `NO_PRUNING_POSSIBLE` | the range straddles the predicate | scan, and evaluate the filter per row | +| `FILTER_ALWAYS_TRUE` | every value in the segment passes | scan, and **drop the filter** — do not test rows | +| `FILTER_ALWAYS_FALSE` | no value can pass | skip the segment | +| `FILTER_TRUE_OR_NULL` | passes except that NULLs are unresolved | scan; only the validity mask needs testing | +| `FILTER_FALSE_OR_NULL` | fails except for NULLs | as above, inverted | + +The last two exist because SQL comparison is three-valued: a min/max range says nothing about +NULLs, so a filter that is decided for every *value* may still be undecided for every *row*. +`FILTER_ALWAYS_TRUE` is the underrated one — it removes per-row filter evaluation entirely, +which is pure win on a highly selective-in-reverse predicate where skipping is impossible. + +Three implementation details that change how you reason about this: + +- **`state.segment_checked` (`:424`, set at `:433`).** The zone map is consulted once per + segment per scan — except for **dynamic filters**, detected at `:431-432`, which are + re-checked every time "as it can always change". Those are the filters a join build side + tightens mid-query; a static `segment_checked` would freeze them at their loosest. +- **Updates invalidate pruning (`:448-461`).** If the column has an update segment, the + filter is evaluated against the update statistics too, and unless both agree the result is + downgraded to `NO_PRUNING_POSSIBLE`. Pruning is only sound over data the statistics + actually cover. +- **String zone maps are prefixes.** `src/include/duckdb/storage/statistics/string_stats.hpp:34` + sets `CURRENT_MAX_STRING_MINMAX_SIZE = 12` (the legacy format used 8, `:35`), and + `string_stats.cpp:391-396` marks anything longer `TRUNCATED_STATS`. So a min/max on a URL + column compares the first 12 bytes — `https://www.` for most of the web, which prunes + nothing. + +The catch that governs all of it: zone maps only prune when the data is **clustered on the +filter column**. On randomly ordered data every zone spans the whole domain and nothing +skips — the same clustering premise C-Store built projections for and ClickHouse enforces +with a mandatory `ORDER BY`. + +**Why it matters:** this is where topic 10's filter pushdown physically lands. A plan-level +rewrite becomes a storage-level skip — or does not, depending on a property of the data no +optimizer controls. + +### Step 10 — Do the sizes yourself, and say which GB/s + +> **In:** one concrete column — 1,000,000 `INT64` values, 200 distinct, average run 8, value +> range 900 wide. +> **Out:** each encoder's score, with the multiplication shown, and an unambiguous way to +> report the scan rate. + +Baseline: `1,000,000 × 8 B` = **8,000,000 B**. Average run 8 ⇒ `1,000,000 / 8` = +**125,000 runs**. + +**RLE**, scored by `rle.cpp:113-116` as `(sizeof(rle_count_t) + sizeof(T)) × seen_count`. +`rle_count_t` is 4 bytes, `T` is 8: + +``` +125,000 x (4 + 8) = 1,500,000 B 8,000,000 / 1,500,000 = 5.33x +``` + +**Bit-packing**, `FOR` mode. The width is `ceil(log2(900))` = **10 bits**, because +2⁹ = 512 < 900 ≤ 1024 = 2¹⁰. Groups: `1,000,000 / 2,048` = 488.28 ⇒ **489 groups**. Per group, +`bitpacking.cpp:263-265` charges the packed payload plus `sizeof(T)` for the frame of +reference plus an aligned width field: + +``` +payload/group: 2,048 x 10 bits = 20,480 bits = 2,560 B +overhead/group: 8 B frame of reference + 8 B aligned width field = 16 B +per group: 2,576 B +total: 489 x 2,576 = 1,259,664 B 8,000,000 / 1,259,664 = 6.35x +``` + +**Dictionary**, scored by `dictionary_compression.cpp:85-98`. Width is +`MinimumBitWidth(unique_count + 1)` = `MinimumBitWidth(201)` = **8 bits**: + +``` +selection buffer: 1,000,000 x 8 bits = 1,000,000 B +dictionary: 200 x 8 B = 1,600 B + ----------- +true: 1,001,600 B ratio 7.99x +reported (x1.2): 1,201,920 B the score the loop compares +``` + +So on this column the ranking the checkpointer sees is dictionary 1,201,920 < bit-packing +1,259,664 < RLE 1,500,000 < uncompressed 8,000,000 — dictionary wins, by 4.8%. Lengthen the +average run to 32 and RLE's score falls to `31,250 × 12` = 375,000 B and it wins by 3.2×. The +same column, a different sort order, a different encoder: that is the whole argument for +deciding per row group. + +**Now the bandwidth, and the ambiguity.** `FINDINGS.md` row 12 records this topic's measured +scan floor: **24–57 GB/s on a machine with roughly 150 GB/s of memory bandwidth**. Take the +dictionary-encoded column above, at 7.99×, and describe one scan two ways: + +``` +physical bytes read: 1,001,600 B logical values produced: 8,000,000 B +at 24 GB/s logical => 3.0 GB/s of compressed bytes actually moved +at 3.0 GB/s physical => 24 GB/s of "effective" throughput +``` + +Both sentences are true of the same query. A report that does not say which one it means is +not a measurement. And `FINDINGS.md` row 12 keeps this topic's cautionary case, where the +number is not merely ambiguous but impossible: a hoisted timing loop once printed +**19,047,619 GB/s**, which is `19,047,619 / 150` ≈ **127,000×** the machine's peak memory +bandwidth. Sanity-check every throughput against the hardware ceiling before you write it +down. + +**Why it matters:** you can now predict which encoder wins for a column before running +`PRAGMA storage_info`, and check the engine against your own arithmetic. That is the point of +the exercise lanes in `experiments/`. + +--- ## Where each step lives in the code -Read the framework header first — the lifecycle contract is documented -in it — then the encoders, then zone maps. - -| File | Role (steps) | -|------|------| -| `src/include/duckdb/function/compression_function.hpp` | the lifecycle contract (3, 4) | -| `src/storage/compression/rle.cpp` | the simplest complete encoder (5) | -| `src/storage/compression/bitpacking.cpp` | four modes in one (6) | -| `src/storage/compression/dictionary_compression.cpp`, `fsst.cpp`, `dict_fsst/`, `zstd.cpp` | the string stack (7) | -| `src/storage/table/column_data.cpp` | zone maps (8) | - -- **Step 3** — `compression_function.hpp:130–141`: `init_analyze` - (`:139`), `analyze` per vector, `final_analyze` (`:141`) returning - the score; the winner's `compress_data` (`:148`). Force a choice via - `PRAGMA force_compression` for your experiments. -- **Step 4** — scans via `scan_vector` (`:159`) / `scan_partial` - (`:162`); point lookups via `fetch_row` (`:172`) — the constraint - that keeps zstd a last resort. -- **Step 5** — `rle.cpp`: `RLEAnalyzeState :86` / `RLEAnalyze :99` - count runs; `RLEFinalAnalyze :113` returns bytes = runs × (value + - count size); `RLECompressState :126` writes the two interleaved - arrays. The `CompressionFunction` registration at `:570` bundles all - the function pointers — grep this pattern in every other encoder. -- **Step 6** — `bitpacking.cpp`: `BitpackingMode` (`:103`, decode - `:42`); AUTO picks per group of 2048 values (`:209–:264`); the mode - decision arithmetic at `:219–:237` computes each variant's width and - picks the smallest. `ForceBitpackingModeSetting :312` for - experiments. -- **Step 7** — `dictionary_compression.cpp:48` (ids then bit-packed); - `fsst.cpp:40–:47,:72` (train a symbol table on a sample, encode all - strings); `dict_fsst/` (both at once); `zstd.cpp` (the fallback). -- **Step 8** — `column_data.cpp:423` `ColumnData::CheckZonemap`: - consults per-segment stats (`numeric_stats.cpp` / - `string_stats.cpp` — note strings keep min/max PREFIXES) and returns - a `FilterPropagateResult`: always-true (drop the filter too!), - always-false (skip the segment), or no-pruning. +Read the framework header first — the lifecycle contract is documented in it — then the +selection loop, then the encoders, then zone maps. + +| File (`duckdb/duckdb@6c0c1a68`) | Role (steps) | +| --- | --- | +| `src/function/compression_config.cpp` | the menu and its order (1, 3) | +| `src/include/duckdb/storage/storage_info.hpp`, `src/include/duckdb/common/vector_size.hpp` | the units (2) | +| `src/include/duckdb/function/compression_function.hpp` | the lifecycle contract (3, 5) | +| `src/storage/table/column_data_checkpointer.cpp` | the selection loop (3, 4) | +| `src/storage/compression/rle.cpp` | the simplest complete encoder (6) | +| `src/storage/compression/bitpacking.cpp` | four modes in one `Flush` (7) | +| `src/storage/compression/dictionary_compression.cpp`, `fsst.cpp`, `dict_fsst/`, `zstd.cpp` | the string stack (8) | +| `src/storage/table/column_data.cpp`, `src/storage/statistics/string_stats.cpp` | zone maps (9) | + +- **Step 1** — `compression_config.cpp:17-35`, the `internal_compression_methods` array. The + order is the tie-break order. +- **Step 2** — `storage_info.hpp:26` (`DEFAULT_ROW_GROUP_SIZE 122880ULL`), `:394` (the + divisibility assert); `vector_size.hpp:16` (`2048U`); `bitpacking.cpp:25` + (`BITPACKING_METADATA_GROUP_SIZE`). +- **Step 3** — `compression_function.hpp:130-138` documents the lifecycle; `:139` + `init_analyze`, `:140` `analyze`, `:141` `final_analyze`, `:148` `compress_data`. The loop + is `column_data_checkpointer.cpp:172-278`: the shared scan at `:200-217`, drop-out at + `:211-214`, `skip_scan` at `:196`, `PRAGMA force_compression` at `:185-188`, the score + comparison at `:245-256`, the "no suitable method" throw at `:265-268`. +- **Step 4** — `dictionary/common.hpp:20` and `fsst.cpp:37` (`MINIMUM_COMPRESSION_RATIO + = 1.2F`); `dictionary_compression.cpp:97` and `fsst.cpp:202` apply it. Sampling: + `fsst.cpp:38` (`ANALYSIS_SAMPLE_SIZE = 0.25`), `alp_constants.hpp:19-23` (8 vectors × 32 + values, jumping 7 vectors). +- **Step 5** — `compression_function.hpp:171-173` `fetch_row`; `:174-176` `skip`, whose doc + states the random-access constraint; `:164-166` `select`; `:167-170` `filter`. +- **Step 6** — `rle.cpp`: `RLEAnalyzeState :86-91`, `RLEAnalyze :99-110`, `RLEFinalAnalyze + :113-116` (the score), `RLECompressState :126`, scan-state pointers `:313-314`, + `CanEmitConstantVector :333-347`, `RLEScanConstant :349-359`, `RLEFilter :447-490` (the + per-run predicate and the whole-segment early-out at `:477-481`), registration + `GetRLEFunction :568-576`. +- **Step 7** — `bitpacking.hpp:15` the enum; `bitpacking.cpp` `EncodeMeta :34-39` / + `DecodeMeta :40-45`, `EmptyBitpackingWriter :47-63`, `Flush :204-271` (CONSTANT `:209`, + CONSTANT_DELTA `:219`, the width comparison `:230-235`, DELTA_FOR `:237`, FOR `:257`, + give-up `:270`), `BitpackingAnalyze :318-334`, `BitpackingFinalAnalyze :337-344`, + `ForceBitpackingModeSetting :312`. +- **Step 8** — `dictionary_compression.cpp:14-44` the layout, `:48-65` the storage struct, + `:70-78` the V1.3.0 retirement, `:85-98` the score; `fsst.cpp:37,:38,:470`; `dict_fsst/`; + `zstd.cpp`. +- **Step 9** — `column_data.cpp:423-462` `CheckZonemap` (`segment_checked` `:424`/`:433`, + dynamic filters `:431-432`, the update downgrade `:448-461`); + `filter_propagate_result.hpp:15-21` the five results; `string_stats.hpp:34-35` and + `string_stats.cpp:391-396` for prefix truncation. + +--- ## Questions for notes.md -1. The analyze pass doubles ingest cost. What does BtrBlocks do instead - (sampling) and what does it risk? -2. `fetch_row` on DELTA_FOR: decoding row 1907 of a 2048 group requires - what? Why is this fine for OLAP (how often does fetch_row run — - think late materialization: fetch AFTER filter). -3. RLE score vs dictionary score on a column of 50% NULLs: which wins - and why does validity (empty_validity.cpp) change the answer? -4. Zone map always-true result removes the FILTER — when does that - matter more than segment skipping? (Selectivity ~100% — filter cost - itself.) -5. M12: which of the four bitpacking modes fits node-id columns in a - graph adjacency payload? (Ids are dense-ish and clustered by - creation time.) +1. **The analyze pass doubles ingest cost.** What does BtrBlocks do instead, and what does + it risk? Then complicate it: DuckDB already samples for two encoders (FSST at 25%, ALP at + 0.208% — Step 4). What distinguishes the encoders that sample from the ones that measure + everything, and does that rule explain BtrBlocks' choice too? +2. **`fetch_row` on `DELTA_FOR`.** Decoding row 1907 of a 2,048-value group requires what, + exactly? (Follow `bitpacking.cpp:824` and `:899`.) Why is the cost acceptable for OLAP — + think about *when* `fetch_row` runs under late materialization, i.e. after the filter, on + survivors only. Then say what would change if the same segment served an OLTP point-lookup + workload. +3. **RLE score versus dictionary score on a column of 50% NULLs.** Which wins, and why does + validity change the answer? Note that DuckDB stores validity as its own column with its own + compression function (`COMPRESSION_EMPTY`, `compression_config.cpp:31-32`, plus + `COMPRESSION_ROARING`), so the NULLs may not be in the data column's score at all. +4. **A zone map returning `FILTER_ALWAYS_TRUE` removes the filter.** When does that matter + more than segment skipping? (Think selectivity near 100% — the cost being removed is + per-row filter *evaluation*, not I/O.) Then account for `FILTER_TRUE_OR_NULL`: what does + the scan still have to do, and what does it get to skip? +5. **M12.** Which of the four bit-packing modes fits the node-id payload columns in a graph + adjacency structure, where ids are dense-ish and clustered by creation time? Work the + width for a plausible id range with Step 10's arithmetic, and say which mode `Flush`'s + cascade would actually reach and why — including whether `DELTA_FOR` or `FOR` wins the + `:235` comparison for your numbers. + +--- + +## Takeaway + +DuckDB's answer to "who picks the encoding" is *the data does, measured, per 122,880 rows* — +and the implementation is more interesting than the slogan. The analyze pass is one shared +scan, not one per candidate. The scores are deliberately 20% pessimistic for the schemes that +cost something at decode time. Two encoders sample rather than measure, because training a +model is the expensive part. And bit-packing's estimator *is* its compressor with the writes +compiled out, so the two passes cannot disagree. + +The transferable idea is the last one. Every system that estimates a cost and then does the +work has two implementations of the same logic and a bug waiting in the gap between them. +`Flush` closes that gap by construction, and it costs one template +parameter. + +--- ## Done when -You can recite the analyze→score→compress lifecycle, the four -bitpacking modes with their triggers, and explain why fetch_row shapes -the whole encoder menu. +Answer each before unfolding it. + +- [ ] Recite the analyze → score → compress lifecycle, and name the four ways a candidate + encoder can fail to be chosen. + +
Answer + +`init_analyze` per candidate → `analyze` per vector, on every candidate, from **one** shared +scan → `final_analyze` returns an estimated byte count → the smallest score wins → the winner +runs `compress_data` over the same data again. Documented at +`compression_function.hpp:130-138`, implemented at `column_data_checkpointer.cpp:172-278`. + +Four ways to lose: +1. **Drop out mid-scan** — `analyze` returns false; the state is nulled and the function + removed for the remainder of the pass (`:211-214`). +2. **Self-disqualify** — `final_analyze` returns `DConstants::INVALID_INDEX` (`:247-250`), as + `BitpackingFinalAnalyze` does at `bitpacking.cpp:341` when `Flush` fails. +3. **Lose on score** — `:252` compares with strict `<`, so ties go to the earlier entry in + `compression_config.cpp:17-35`. +4. **Never be considered** — `:196`'s `skip_scan`, when the DDL or `PRAGMA force_compression` + names a type outright. + +And the safety net: `:265-268` throws `FatalException` if nothing qualifies, which is why +`UNCOMPRESSED` is permanently in the menu. + +
+ +- [ ] `final_analyze` is supposed to return estimated bytes. Give two encoders where the + number it returns is not the estimated bytes, and say why each is right to lie. + +
Answer + +**Dictionary** (`dictionary_compression.cpp:97`) and **FSST** (`fsst.cpp:202`) both multiply +their honest estimate by `MINIMUM_COMPRESSION_RATIO = 1.2` +(`dictionary/common.hpp:20`, `fsst.cpp:37`) before returning it. They report themselves 20% +bigger than they are, so they must beat the alternatives by more than 20% to win. + +Right, because a byte count is not the whole cost. Both add an indirection to every decoded +value, and both make `fetch_row` more expensive than a bit-packed segment's arithmetic. A +scheme scored purely on size would be picked at margins where it loses on time. + +A different kind of not-really-bytes: **ALP** samples 256 of a row group's 122,880 values +(`alp_constants.hpp:19-23` — 8 vectors × 32 values, 0.208%) and **FSST** analyses 25% +(`fsst.cpp:38`). Their scores are extrapolations, not measurements. The rule is that encoders +whose *training* is expensive sample; encoders whose analyze pass is just counting (RLE, +bit-packing, dictionary) measure everything. + +
+ +- [ ] `RLEFilter` is the 2006 paper running in production. Say what it does and what the + complexity is. + +
Answer + +`rle.cpp:447-490`. On the first filtered scan of a segment it evaluates the predicate over +the **run values array**, once — its own comments say "apply the filter to all RLE values at +once" (`:456-457`) and "execute the filter over all runs at once" (`:463`) — records the +result in a `matching_runs` bool array cached on the scan state (`:310`, `:460-475`), and +then, at `:477-481`, returns `sel_count = 0` immediately if no run matched, abandoning the +whole segment without decoding a row. + +Complexity: `O(runs)`, not `O(rows)`. On this topic's shared column — 1,000,000 values, +125,000 runs — the predicate is evaluated 125,000 times instead of 1,000,000, an 8× cut that +grows linearly with average run length. + +That is exactly Abadi §6.2's accounting, which puts the aggregation cost at +`num_tuples / avg_run_len` for RLE against `num_tuples` uncompressed. Its sibling is +`CanEmitConstantVector` at `:333-347`: when a run spans a full 2,048-value vector, +`RLEScanConstant` (`:349-359`) emits a `CONSTANT_VECTOR` holding one value — Abadi's +`isOneValue()` block property, by another name. + +
+ +- [ ] Bit-packing has four modes but only one arithmetic comparison decides between the two + interesting ones. Give it, and say which way ties go and why. + +
Answer + +`bitpacking.cpp:230-235`: + +``` +delta_required_bitwidth = MinimumBitWidth(min_max_delta_diff) +regular_required_bitwidth = MinimumBitWidth(min_max_diff) +prefer_for = can_do_for && delta_required_bitwidth >= regular_required_bitwidth +``` + +If `prefer_for` is false, `Flush` takes the `DELTA_FOR` branch at `:237`; otherwise it falls +through to `FOR` at `:257`. `CONSTANT` (`:209`) and `CONSTANT_DELTA` (`:219`) are cheap +special cases tested before either. + +Ties go to plain **`FOR`**, because of the `>=`. That is correct: `DELTA_FOR` stores an extra +`sizeof(T)` delta offset (`:249`) and its decode needs a prefix sum over the group +(`:664`, `:824`, `:899`), so at equal bit width it is strictly more expensive in both space +and time. + +The structural point is that `Flush` is a **priority cascade**, not a race — it takes the +first mode that applies rather than scoring all four. And `BitpackingFinalAnalyze:337-344` +runs this very function with `EmptyBitpackingWriter` (`:47-63`, all method bodies empty), so +the analyze estimate and the compressor are literally the same code. + +
+ +- [ ] `CheckZonemap` returns five results, not three. Name the two extra ones and say what + forces them to exist. + +
Answer + +`filter_propagate_result.hpp:15-21`: `NO_PRUNING_POSSIBLE`, `FILTER_ALWAYS_TRUE`, +`FILTER_ALWAYS_FALSE`, and the two extras — **`FILTER_TRUE_OR_NULL`** and +**`FILTER_FALSE_OR_NULL`**. + +They exist because SQL comparison is three-valued and a min/max range says nothing about +NULLs. A zone map can prove that every non-NULL value in a segment satisfies the predicate, +but a row whose value is NULL still evaluates to unknown, so the segment cannot be blanket +accepted; the scan still has to consult the validity mask, but it can skip evaluating the +predicate itself. Without these two states that whole case collapses to +`NO_PRUNING_POSSIBLE` and the saving is lost. + +Two related things `column_data.cpp:423-462` reveals: the check is memoised per segment via +`state.segment_checked` (`:424`, `:433`) **except** for dynamic filters (`:431-432`), which +can tighten mid-query and are re-checked; and if the column has updates, the result is +downgraded to `NO_PRUNING_POSSIBLE` unless the update statistics agree (`:448-461`). And +string zone maps are only the first 12 bytes (`string_stats.hpp:34`, truncation marked at +`string_stats.cpp:391-396`), so a URL column's min/max usually compares `https://www.` and +prunes nothing. + +
+ +--- ## References -**Code** -- [duckdb](https://github.com/duckdb/duckdb) — read - `src/include/duckdb/function/compression_function.hpp` first (the - lifecycle contract is documented in the header), then the encoders in - `src/storage/compression/` (`rle.cpp`, `bitpacking.cpp`, - `dictionary_compression.cpp`, `fsst.cpp`, `dict_fsst/`, `zstd.cpp`); - zone maps in `src/storage/table/column_data.cpp` +**Code** — all anchors at `duckdb/duckdb@6c0c1a68`; verify with +`tools/pinned-source.py show duckdb/duckdb -r A:B`. + +- [duckdb/duckdb](https://github.com/duckdb/duckdb). Read in this order: + `src/include/duckdb/function/compression_function.hpp` (the lifecycle contract is in the + header comments), `src/storage/table/column_data_checkpointer.cpp` (the selection loop), + then `src/storage/compression/rle.cpp` and `bitpacking.cpp`, then + `dictionary_compression.cpp` / `fsst.cpp` / `dict_fsst/` / `zstd.cpp`, then + `src/storage/table/column_data.cpp` for zone maps. `src/function/compression_config.cpp` is + the index to all of it. +- `PRAGMA storage_info('')` reports the encoding actually chosen per segment; + `PRAGMA force_compression` overrides the race. Both are how you turn this chapter into an + experiment. + +**Papers** + +- Abadi, Madden, Ferreira. *Integrating Compression and Execution in Column-Oriented Database + Systems*. SIGMOD 2006 — §5.2's block-properties API is what Step 6's `RLEFilter` and + `CanEmitConstantVector` implement. Covered in + [reading-cstore-compression.md](reading-cstore-compression.md). +- Kuschewski, Sauerwein, Alhomssi, Leis. *BtrBlocks: Efficient Columnar Compression for Data + Lakes*. SIGMOD 2023 — §3.1's sampler, for question 1. Covered in + [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md). + +**In this topic** + +- [reading-btrblocks-fsst.md](reading-btrblocks-fsst.md) — FSST itself, and the sampling + alternative +- [reading-clickhouse-mergetree.md](reading-clickhouse-mergetree.md) — the third answer to + "who picks the encoding": the user, in the DDL +- `FINDINGS.md` row 12 — the measured scan floor (24–57 GB/s on a ~150 GB/s machine) and the + 19,047,619 GB/s hoisted-loop bug diff --git a/topics/13-graph-engines/README.md b/topics/13-graph-engines/README.md index bcbbc2c..b84c14a 100644 --- a/topics/13-graph-engines/README.md +++ b/topics/13-graph-engines/README.md @@ -8,8 +8,8 @@ and what does pattern matching (multi-way Expand) cost? ## The problem, measured (bench lane 1, provided — runs today) `cargo run --release --bin hop_bench` — preferential-attachment graph, 1 M nodes -/ 16.0 M directed edges, two-hop `COUNT(DISTINCT)` from 1000 sources. Max degree -6565, p50 degree 11: +/ 16.0 M directed edges, two-hop `COUNT(DISTINCT)` from 10 000 random sources and +from the 100 highest-degree nodes. Max degree 6565, p50 degree 11: ``` impl source set ns/query distinct reached @@ -24,13 +24,26 @@ whatever the degree distribution under your start node says, and on a scale-free graph that distribution is a power law with no useful mean. The median node has 11 neighbours. The top one has 6565. -Now look at the last column: the slow case reaches *fewer* distinct nodes — 7.9 M -against 10.2 M — while taking 101× longer. High-degree neighbourhoods overlap -heavily, so the extra work is redundant rather than productive. That redundancy -is the opening the CSR and masked-SpMV lanes attack, and it is why graph engines -are built around set operations on sorted adjacency rather than around pointer -chasing. It is also why "supernode" is a word in this field and not in the -others. +Now the last column, carefully — because it is a trap this repo walked into. The +checksums are **sums over unequal query counts**: 10 000 random sources against +100 supernodes. Divide before comparing. Per query, random reaches +10 220 457 / 10 000 = **1022** distinct nodes and a supernode reaches +7 890 665 / 100 = **78 907** — **77× more**, not fewer. The earlier reading of +this table ("the slow case reaches fewer nodes, so the work is redundant") was +an artifact of comparing a sum of 10 000 with a sum of 100. + +What the lane does show is cleaner. Cost per distinct node reached is +4914 / 1022 = **4.81 ns** from random sources against 495 378 / 78 907 = +**6.28 ns** from supernodes — only **1.31×** worse. So the 101× is 77× more +work and 1.3× worse cost per unit of it, and *that* is the finding: a two-hop +traversal costs what its reachable neighbourhood costs, and the degree +distribution decides the neighbourhood. The 1.31× residual is the interesting +part, and this lane cannot say whether it is re-walked overlap or the cache +pressure of a 79 000-node frontier — it counts distinct nodes, not edges +traversed. Separating the two is an exercise below. The CSR and masked-SpMV +lanes attack the residual, which is why graph engines are built around set +operations on sorted adjacency rather than pointer chasing. It is also why +"supernode" is a word in this field and not in the others. ## 1. The adjacency representation menu @@ -157,6 +170,15 @@ over three representations, same power-law graph: 5. Compare externally: same query on FalkorDB (`GRAPH.QUERY ... MATCH (a)-[*1..2]->(b) RETURN count(DISTINCT b)`) and neo4j if handy — record in notes.md. +6. YOU add: a second counter alongside the distinct-node checksum that + counts **edges traversed**, and report both normalised per query. + The provided lane measures 1022 against 78 907 distinct nodes per + query and 4.81 against 6.28 ns per distinct node — it cannot say + whether that 1.31× residual is re-walked overlap or the cache cost + of a 79 000-node frontier. Edges-per-distinct-node separates them: + if the supernode ratio is much higher, the work really is redundant + and the visited bitmap is the fix; if it is flat, the residual is + memory and only layout helps. Predict which before you measure. ## Reading guides diff --git a/topics/13-graph-engines/notes.md b/topics/13-graph-engines/notes.md index 7807e82..edc15d5 100644 --- a/topics/13-graph-engines/notes.md +++ b/topics/13-graph-engines/notes.md @@ -3,12 +3,13 @@ ## Baseline (provided lane, Apple M3 Pro, measured 2026-07-28) `cargo run --release --bin hop_bench` — preferential-attachment graph, 1 M -nodes / 16.0 M directed edges, two-hop `COUNT(DISTINCT)` from 1000 sources. +nodes / 16.0 M directed edges, two-hop `COUNT(DISTINCT)` from 10 000 random +sources and from the 100 highest-degree nodes. Max degree 6565, p50 degree 11. -| impl | source set | ns/query | checksum | +| impl | source set | ns/query | checksum (sum over the set) | |---|---|---|---| -| adj_list (oracle) | random | 4 914 | 10 220 457 | +| adj_list (oracle) | random (10 000 sources) | 4 914 | 10 220 457 | | adj_list (oracle) | supernodes (top-100 degree) | **495 378** | 7 890 665 | | CSR (yours) | | | stub | | masked SpMV (yours) | | | stub | @@ -21,18 +22,27 @@ under your start node says it costs, and on a scale-free graph that is a power law with no useful average. The p50 node has 11 neighbours; the top node has 6565. -Note the supernode checksum is *smaller* (7.9 M vs 10.2 M distinct nodes -reached) while taking 101× longer — high-degree neighbourhoods overlap heavily, -so the work is redundant, not productive. That redundancy is what the CSR and -SpMV lanes are able to attack; the adjacency-list oracle cannot. +Note the checksum column is a **sum over the source set**, and the two sets are +different sizes — 10 000 random sources against 100 supernodes. Normalise before +comparing anything: random reaches **1022** distinct nodes per query, +a supernode reaches **78 907** — **77× more**. (An earlier version of this file +read the raw sums as "the slow case reaches fewer nodes"; it does not, and that +mistake is worth keeping in mind every time a benchmark prints a total.) + +So the 101× decomposes: **77× more nodes reached**, and **1.31× worse cost per +node reached** (4.81 ns against 6.28 ns). The second factor is the one worth +chasing, and this lane cannot attribute it — it counts distinct nodes, not edges +traversed, so re-walked overlap and the cache cost of a 79 000-node frontier look +identical from here. Exercise 6 in the README adds the edge counter that +separates them. Checksums must match across all three implementations per source set — that is the correctness gate before any timing comparison means anything. ## Predictions (fill BEFORE implementing csr.rs / matrix.rs) -Baseline (provided, measured): adj_list 3484 ns/query random, -294885 ns/query supernodes (85× tail); graph 1M nodes / 16M directed +Baseline (provided, measured): adj_list 4914 ns/query random, +495378 ns/query supernodes (101× tail); graph 1M nodes / 16M directed edges, max degree 6565, p50 degree 11. | impl | sources | predicted vs adj_list (×) | actual ns/query | @@ -46,7 +56,7 @@ edges, max degree 6565, p50 degree 11. |---|---|---| | does csr beat adj_list at all? (per-node vecs are already contiguous — where's the win?) | | | | matrix vs csr: what does the frontier materialization cost? | | | -| supernode ratio: does CSR shrink the 85× tail or just shift it? | | | +| supernode ratio: does CSR shrink the 101× tail or just shift it? | | | | CSR build time vs adj_list build time | | | ## Implementation log diff --git a/topics/13-graph-engines/reading-graphblas-internals.md b/topics/13-graph-engines/reading-graphblas-internals.md index 1c9c57c..6d45f5e 100644 --- a/topics/13-graph-engines/reading-graphblas-internals.md +++ b/topics/13-graph-engines/reading-graphblas-internals.md @@ -11,6 +11,15 @@ masks, the write problem, and the delta overlay that solves it — then hands you the file anchors. It's also the topic-20/M20 preview: read for the shape now, the kernels later. +Every anchor below was read at the revisions pinned in +[`resources/codebases.md`](../../resources/codebases.md): +**SuiteSparse:GraphBLAS at `1fd5475`** and **FalkorDB at `ccb449a9a`**. +Line numbers move; check one with +`python3 tools/pinned-source.py show GraphBLAS Include/GraphBLAS.h -r 1664:1667` +before you trust it. Three claims in the previous version of this +chapter did not survive that check, and each is called out where it +used to be. + ## The problem in one sentence Answer "who are the neighbors of these 10,000 nodes?" as one streaming @@ -22,32 +31,68 @@ the streaming possible. ### Step 1 — the graph is a boolean matrix; traversal is multiplication -A directed graph on n nodes can be stored as an n×n **adjacency -matrix** A where `A[i][j] = true` iff there is an edge i→j; row i then -IS node i's outgoing neighbor list. The GraphBLAS move: once the graph -is a matrix, traversals become linear algebra. Put a set of source -nodes into a boolean vector x (the **frontier**); then `y = xA` -(**SpMV** — sparse matrix-vector multiply) computes, in one operation, -the union of all their neighbors — a whole BFS step. Two hops = `xA²`, -triangles = `A ⊙ A²` (elementwise AND of A with A²): +> **In:** a directed graph: a set of nodes, and a set of ordered pairs +> (edges) over them. +> **Out:** the same graph as an n×n boolean matrix, and every traversal +> rewritten as a sparse matrix product. + +An **adjacency matrix** A for a graph on n nodes is the n×n boolean +matrix with `A[i][j] = true` exactly when there is an edge i→j. Row i +of A is therefore node i's outgoing neighbour list, spelled as a row +instead of as a linked list. + +Two more definitions before the algebra means anything: + +- A **frontier** is the set of nodes a breadth-first search is + currently standing on, written as a boolean vector x of length n + (`x[i] = true` iff node i is in the set). +- **SpMV** is sparse matrix–vector multiply. Over the boolean + semiring — where "+" is OR and "×" is AND — the entry `(A^T x)[j]` + is `OR over i of (A[i][j] AND x[i])`, which is true exactly when + *some* frontier node i has an edge to j. + +So one SpMV is one whole BFS level, for the entire frontier at once: ``` BFS frontier expansion = SpMV: y<¬visited> = A^T x - 2-hop = A², triangles = A ⊙ A² + 2-hop = A², triangles = A ⊙ A² (⊙ = elementwise AND) ``` +`A²` is the two-hop reachability matrix: `A²[i][k]` is true iff there +is some j with i→j and j→k. `A ⊙ A²` keeps a two-hop pair only when +the two endpoints are also directly connected — which is the +definition of a triangle. + Why it matters: one engine — the sparse-multiply kernels — serves every traversal, so every kernel optimization (SIMD, parallelism, format tricks) speeds up every query. That's FalkorDB's whole -architectural bet. +architectural bet, and it is the bet this topic's headline stresses: +the same two-hop query costs 4 914 ns from random sources and 495 378 +ns from supernodes ([FINDINGS.md](../../FINDINGS.md) row 13). The +matrix spelling does not make that ratio go away — but it changes +*which* part of the machine you can attack. ### Step 2 — CSR: the matrix is stored as offsets + neighbors -Storing n² booleans is absurd for a real graph (1M nodes, 16M edges: -n² = 10¹² cells, 99.998% empty), so sparse matrices store only present -entries. The standard format is **CSR** (compressed sparse row — one -`offsets` array of n+1 positions plus one `neighbors` array of m -column indices; row i is the slice between its offsets): +> **In:** the n×n boolean matrix from Step 1, with m true entries out +> of n² cells. +> **Out:** two contiguous integer arrays that hold only the m true +> entries, and give row i's neighbours as one slice. + +Storing n² booleans is absurd for a real graph. On this topic's own +bench graph — 1 M nodes, 16.0 M directed edges +([notes.md](notes.md), baseline table) — that is: + +``` + cells = n² = 1e6 × 1e6 = 1e12 + filled = m = 16.0e6 + density = 16.0e6 / 1e12 = 1.6e-5 → 99.9984% of the matrix is empty +``` + +So sparse matrices store only the present entries. The standard format +is **CSR** (compressed sparse row): one `offsets` array of n+1 +positions plus one `targets` array of m column indices, where row i's +neighbours are the slice `targets[offsets[i] .. offsets[i+1]]`. ``` CSR for 0->{5,9}, 1->{5}, 2->{}: @@ -56,32 +101,87 @@ column indices; row i is the slice between its offsets): neighbors(i) = targets[offsets[i] .. offsets[i+1]] one slice, zero chase ``` -16M edges as CSR: 4 MB offsets + 64 MB targets — contiguous, -prefetchable arrays (topic 0's sequential-beats-random, structurally -guaranteed). Why it matters: "sparse matrix" and "read-optimized -adjacency" are the same object; the algebra of Step 1 runs over -exactly this layout. +Work the size on the bench graph, with 4-byte indices (a 1 M-node +graph needs 20 bits per id, so 32-bit indices fit with room to spare): + +``` + offsets: (n + 1) × 4 B = 1 000 001 × 4 B = 4.00 MB + targets: m × 4 B = 16 000 000 × 4 B = 64.00 MB + total = 68.00 MB + bytes per edge = 68.00 MB / 16.0e6 = 4.25 B/edge +``` + +Both arrays are contiguous and prefetchable — topic 0's +sequential-beats-random, structurally guaranteed rather than hoped +for. Why it matters: "sparse matrix" and "read-optimized adjacency" +are the same object; the algebra of Step 1 runs over exactly this +layout, and 4.25 B/edge is the number every other representation in +this topic gets compared against. ### Step 3 — four sparsity formats, switched by density at runtime -SuiteSparse doesn't commit to CSR: it keeps each matrix in one of four -formats and switches automatically when the matrix's density (fraction -of present entries) crosses thresholds: +> **In:** a sparse matrix and its current occupancy — how many of its +> n vectors are non-empty, and how many entries it holds. +> **Out:** one of four physical layouts, chosen by SuiteSparse without +> the caller asking. + +SuiteSparse doesn't commit to CSR. `Include/GraphBLAS.h:1664-1667` +names four **sparsity formats**, and a matrix may be told which +subset it is allowed to take: + +```c +// GraphBLAS.h, the sparsity-control bitmask values + 1664 #define GxB_HYPERSPARSE 1 // store entries in a list of non-empty vectors + 1665 #define GxB_SPARSE 2 // store entries in a compressed form (CSR/CSC) + 1666 #define GxB_BITMAP 4 // store entries in a bitmap + 1667 #define GxB_FULL 8 // store all entries, no need for indices +``` + +Read those as: **hypersparse** keeps an explicit list of only the +non-empty rows (so a matrix with 1 000 non-empty rows out of 1 000 000 +costs 1 000 offsets, not 1 000 001); **sparse** is plain CSR/CSC; +**bitmap** is one presence bit per cell plus a values array — random +writes cost a bit-flip because there is no structure to shift; **full** +drops the index arrays entirely because every cell is present. + +The switch is not vibes. `GraphBLAS.h:1715-1728` states the rule, and +`Source/include/GB_defaults.h:20` gives the constant: + +```c +// GraphBLAS.h:1715-1728 — the hyper_switch rule, paraphrasing the comment + 1715 // ... let k be the number of non-empty vectors, n the number of + 1716 // vectors, and h the hyper_switch: + // ... 1717-1722: elided — the same rule stated for GxB_Matrix_Option_set ... + 1723 // hypersparse -> sparse: if n <= 1 || k > 2*n*h + 1724 // sparse -> hypersparse: if n > 1 && k <= n*h + // ... 1725-1727: elided ... + 1728 // +``` + +```c +// Source/include/GB_defaults.h + 20 #define GB_HYPER_SWITCH_DEFAULT (0.0625) +``` -- `GxB_HYPERSPARSE` — offsets stored only for NON-empty rows (graphs - where most node IDs have no edges of a given type — e.g. a rare - relationship type touching 1K of 1M nodes) -- `GxB_SPARSE` — plain CSR/CSC -- `GxB_BITMAP` — dense bitmap of present entries + values array - (fast random writes, no structure to shift) -- `GxB_FULL` — every entry present, no index arrays at all +h = 0.0625 = 1/16. Now put the bench graph's numbers in. The matrix +is 1 M × 1 M, so n = 1 000 000 and the sparse→hypersparse threshold is: ``` - density → hypersparse | sparse (CSR) | bitmap | full - ~n rows m ≈ O(n) m/n²>τ m = n² + n · h = 1 000 000 × 0.0625 = 62 500 non-empty vectors ``` -Crossing a threshold flips the format on the next wait/computation. +- The full adjacency matrix has roughly 1 M non-empty rows (nearly + every node has an out-edge). k ≈ 1 000 000 > 62 500 → stays + **sparse** (CSR). +- A rare relationship type touching 1 000 of the 1 M nodes has + k = 1 000 ≤ 62 500 → flips to **hypersparse**, and its offsets array + costs 1 000 slots rather than 1 000 001. That is a 1 000× saving on + the offsets array, for free, decided by the library. +- Note the two thresholds differ by a factor 2 (`k > 2*n*h` going + back the other way). That gap is hysteresis: without it, a matrix + sitting at exactly k = n·h would flip format on every insert and + delete. + This is the same menu as topic 12's encodings: representation follows data shape, chosen by measurement, invisible above the API. Why it matters: a label matrix with 3 labels and a supernode-heavy adjacency @@ -89,144 +189,356 @@ matrix get different physical layouts for free. ### Step 4 — dot vs saxpy: two ways to multiply, picked per call -Sparse matrix multiply has two classic algorithms, and SuiteSparse -picks per operation. **dot** computes each output entry C(i,j) as an -inner product of a row of A' with a column of B — good when the output -is small or masked, because you compute *only the entries you need*. -**saxpy** (Gustavson's algorithm) scatters each input entry's -contributions into a per-row accumulator — good when the output is big -and dense-ish; a hash-based accumulator variant covers the -too-sparse-for-a-dense-scratch-row case. - -BFS mapping: frontier × adjacency with a small frontier wants dot -guided by the mask (compute only unvisited candidates); a huge -frontier wants saxpy (stream everything). Why it matters: the SAME -`GrB_mxm` call is executed by different algorithms at frontier size 10 -vs 10⁶ — the engine re-plans per step, which hand-written BFS code -never does. +> **In:** a `GrB_mxm` call — operands A and B, an optional mask M, and +> the sparsity formats each of them currently holds. +> **Out:** one of four kernels (saxpy, dot2, dot3, dot4), chosen by a +> per-call control function, with different asymptotic cost. + +Sparse matrix multiply has two classic algorithm families. + +- **dot** computes each output entry `C(i,j)` as an inner product of a + row of A' with a column of B. You compute *only the entries you + ask for* — so dot is the right shape when the output is small or + masked. +- **saxpy** (Gustavson's algorithm) walks the input and *scatters* + each entry's contribution into a per-row accumulator. It touches + the output implicitly, so it is the right shape when the output is + large. + +**Correction.** The previous version of this chapter quoted +`Source/mxm/GB_AxB_meta.c:20-21` as "the algorithm menu", with the +text *"generic: for any semiring; dot2/dot3: does `C=A'*B`, +`C=A'*B` … saxpy: Gustavson + Hash"*. At pin `1fd5475` those two +lines say something else entirely: + +```c +// Source/mxm/GB_AxB_meta.c + 20 // The method is chosen automatically: a gather/scatter saxpy method + 21 // (Gustavson), or a dot product method. +``` + +The real menu, with the real asymptotics, is at the top of +`Source/mxm/GB_AxB_dot.c`: + +```c +// Source/mxm/GB_AxB_dot.c + 21 // The dot product method for C=A'*B, C=A'*B, or C=A'*B computes + 22 // C(i,j) = A(:,i)'*B(:,j) for each entry C(i,j). dot2 computes C=A'*B + 23 // and C=A'*B, taking Omega(m*n) time ... + 24 // ... dot3 computes C=A'*B, and only examines entries in the + 25 // mask M, taking Omega(nnz(M)) time ... + 26 // ... dot4 computes C+=A'*B when C is full ... +``` + +and the saxpy side names its three variants in its own signature: + +```c +// Source/mxm/GB_AxB_saxpy.c + 18 GrB_Info GB_AxB_saxpy // C = A*B using Gustavson/Hash/Bitmap +``` + +The *choice* is made per call in +`Source/mxm/GB_AxB_meta_adotb_control.c`: saxpy is the default (`:36` +sets `GB_USE_SAXPY`), and dot4 (`:72-77`), dot3 (`:78-82`) and dot2 +(`:83-87`) each override it under stated conditions. The dot3 +condition is spelled out in `Source/mxm/GB_mxm.h:235-243`: dot3 is +eligible iff there is a mask, the mask is not complemented, and the +mask is sparse or hypersparse. + +Work the Ω's on the triangle query over the bench graph, where the +mask is the adjacency matrix itself (`C = A²`, Step 5): + +``` + m = n = 1 000 000 (the matrix is 1 M × 1 M) + nnz(M) = nnz(A) = 16 000 000 + + dot2: Omega(m · n) = 1e6 × 1e6 = 1.0e12 cell visits + dot3: Omega(nnz(M)) = 1.6e7 cell visits + ratio = 1.0e12 / 1.6e7 = 62 500× +``` + +Sixty-two thousand times less work, from the same `GrB_mxm` call, for +no reason other than that a sparse mask was supplied and SuiteSparse +noticed. BFS mapping: a small frontier against a big adjacency matrix +gives a sparse mask and wants dot3; a huge frontier makes the mask +useless and wants saxpy. Why it matters: the SAME `GrB_mxm` call is +executed by different algorithms at frontier size 10 vs 10⁶ — the +engine re-plans per step, which hand-written BFS code never does. ### Step 5 — masks: the predicate pushed into the kernel -A **mask** is a boolean matrix/vector passed alongside any GraphBLAS -operation that restricts WHERE output may be produced — -`C = A · B` computes A·B only at positions where M is true, and -never materializes the rest. In BFS, the `¬visited` complement mask -does the dedup/visited check inside the multiply; in triangle -counting, `C = A²` evaluates A² only at positions where an edge -already exists — never building the full (potentially enormous) A². -Masks are how GraphBLAS fuses `filter ∘ compute` into one pass — no -materialized intermediate. Why it matters: this is topic 10's -predicate pushdown, one level down — the filter reaches the innermost -loop of the kernel, and it's the mechanism behind the WCOJ -equivalence in [reading-wcoj.md](reading-wcoj.md). +> **In:** an operation `C = A · B` plus a boolean matrix M of the same +> shape as C. +> **Out:** `C = A · B` — output produced only where M is true, with +> the rest never computed rather than computed and discarded. + +A **mask** is a boolean matrix or vector passed alongside any +GraphBLAS operation, restricting WHERE output may be produced. In BFS +the complement mask `¬visited` performs the visited check inside the +multiply. In triangle counting `C = A²` evaluates A² only at +positions where an edge already exists, so the full A² — which on the +bench graph would be up to 10¹² cells — is never built. + +There is a subtlety worth reading the source for, because it decides +whether masking actually saves work. There are two places a mask can +be applied: + +1. **Inside the kernel**, by dot3, which walks the mask and computes + nothing else. That is the Ω(nnz(M)) path from Step 4. +2. **After the fact**, by `GB_masker`, which computes Z = A·B in full + and then merges. `Source/mask/GB_masker.c:2` and `:10` say what it + does; `:14-15` says who calls it — only `GB_mask`, which is called + only from `GB_accum_mask`. And `GB_AxB_meta.c:15-18` warns that the + algorithm *may* choose this late path. + +```c +// Source/mask/GB_masker.c + 2 // R = masker (C, M, Z): compute C=Z, returning the result in R. + // ... 3-9: elided — argument description ... + 10 // R, M, and Z can be sparse, hypersparse, bitmap, or full ... does R=C ; R=Z + // ... 11-13: elided ... + 14 // GB_masker is only called by GB_mask, which itself is only called + 15 // by GB_accum_mask. +``` + +So "I passed a mask" and "the mask saved me work" are different +claims. Only path 1 saves the Ω. Why it matters: this is topic 10's +predicate pushdown, one level down — when the mask reaches the +innermost loop of the kernel, it is also the mechanism behind the +WCOJ equivalence in [reading-wcoj.md](reading-wcoj.md); when it does +not, you paid for the intermediate anyway. ### Step 6 — the write problem: CSR hates single-edge inserts -CSR's strength — everything contiguous — is exactly why it can't -absorb writes: inserting one edge i→j means shifting the tail of the -`targets` array and bumping every offset after row i — O(m) work, -~64 MB of memmove on the 16M-edge graph, *per edge*. A graph database -takes single-edge writes constantly, so raw CSR is unusable as the -live structure. The generic fix (topic 4's LSM idea, applied to -adjacency): keep the read-optimized structure immutable, buffer -changes in a small mutable overlay, merge in the background. Every -system in this topic grows this mechanism — kuzu's CSR buffers, -GraphBLAS's own internal "pending tuples" — and FalkorDB builds its -own explicit one. Why it matters: the overlay design decides write +> **In:** a live CSR adjacency matrix and one `CREATE (a)-[:R]->(b)`. +> **Out:** the cost of applying that one edge in place — and the +> reason no graph database does it that way. + +CSR's strength — everything contiguous — is exactly why it cannot +absorb writes. Inserting one edge i→j means shifting the tail of the +`targets` array and bumping every offset after row i: + +``` + targets memmove: on average half of 64.00 MB = 32.0 MB + offsets bump: on average half of 4.00 MB = 2.0 MB + per single-edge insert = 34.0 MB touched +``` + +At a generous 20 GB/s of achievable memmove bandwidth that is ~1.7 ms +of pure memory traffic *per edge*. A graph database takes single-edge +writes constantly, so raw CSR is unusable as the live structure. + +The generic fix is topic 4's LSM idea applied to adjacency: keep the +read-optimized structure immutable, buffer changes in a small mutable +overlay, merge in the background. Every system in this topic grows +this mechanism. SuiteSparse has its own version — `GB_PENDING_INIT` +at `Source/include/GB_defaults.h:27` is the initial size of a +matrix's pending-tuple list, 256 entries — and FalkorDB builds an +explicit one on top. Why it matters: the overlay design decides write latency, read overhead, AND when the expensive merge happens. ### Step 7 — Delta_Matrix: main + additions + deletions -FalkorDB wraps every graph matrix in a `Delta_Matrix` — THREE -GraphBLAS matrices (+ optionally the same trio transposed): the main -matrix M (read-optimized, CSR inside), `delta_plus` DP (pending adds, -kept in the write-friendly bitmap/hypersparse world), and -`delta_minus` DM (pending deletes — deleting from CSR in place would -be Step 6's problem again, so deletes are *recorded*, not applied): +> **In:** one logical graph matrix, and a stream of single-entry sets +> and removes. +> **Out:** three GraphBLAS matrices whose combination is the logical +> matrix, with writes O(1)-ish and the rebuild deferred behind a +> counted threshold. + +FalkorDB wraps every graph matrix in a `Delta_Matrix`. The struct is +at `src/graph/delta_matrix/delta_matrix.h:108-115` — **not** at +`:17-22`, which is where the accessor macros live: + +```c +// src/graph/delta_matrix/delta_matrix.h + 108 struct _Delta_Matrix { + 109 bool dirty; // Indicates if matrix requires sync + 110 GrB_Matrix matrix; // Underlying GrB_Matrix + 111 GrB_Matrix delta_plus; // Pending additions + 112 GrB_Matrix delta_minus; // Pending deletions + 113 struct _Delta_Matrix *transposed; + 114 pthread_mutex_t mutex; // Lock + 115 }; +``` ``` - M main matrix (read-optimized, CSR inside) - delta_plus pending adds - delta_minus pending deletes + M matrix (read-optimized, CSR inside) + DP delta_plus pending adds + DM delta_minus pending deletes read(i,j) = (M(i,j) OR DP(i,j)) AND NOT DM(i,j) ``` -The whole contract in three functions: +**Correction.** The previous version of this chapter said the deltas +are "kept in the write-friendly bitmap/hypersparse world". They are +not allowed to be bitmap. `delta_get_set.c:44-53` pins them: + +```c +// src/graph/delta_matrix/delta_get_set.c, inside Delta_Matrix_setElement + 44 // Force delta matrices to be hypersparse + // ... 45: elided ... + 46 info = GxB_set(A->delta_plus, GxB_SPARSITY_CONTROL, GxB_HYPERSPARSE); + 47 info = GxB_set(A->delta_plus, GxB_HYPER_SWITCH, GxB_ALWAYS_HYPER); + // ... 48-51: elided — the hyper-hash is disabled on both deltas ... + 52 info = GxB_set(A->delta_minus, GxB_SPARSITY_CONTROL, GxB_HYPERSPARSE); + 53 info = GxB_set(A->delta_minus, GxB_HYPER_SWITCH, GxB_ALWAYS_HYPER); +``` -```rust -// read = (M ∪ DP) ∖ DM — three probes, never a flush -fn get(g: &DeltaMatrix, i: u64, j: u64) -> bool { - (g.m.get(i, j) || g.dp.get(i, j)) && !g.dm.get(i, j) -} +M is left free to be `GxB_SPARSE | GxB_HYPERSPARSE`; DP and DM are +pinned to hypersparse with `GxB_ALWAYS_HYPER` (Step 3's constant, +forced). That is the right choice for the reason Step 3 gave: a delta +holding a few thousand entries has a few thousand non-empty rows out +of a million, so a CSR offsets array would be 1 000 001 slots of +almost entirely zeros. -fn set(g: &mut DeltaMatrix, i: u64, j: u64) { - if g.dm.remove(i, j) { return; } // re-add of a pending delete - if !g.m.get(i, j) { g.dp.insert(i, j); } // never touch the CSR -} +**Correction.** The previous version's `set()` pseudocode keyed the +branch on DM ("if the entry is in DM, clear it"). The real code keys +on **M**: -fn wait(g: &mut DeltaMatrix) { // the LSM compaction: - g.m = (&g.m | &g.dp) - &g.dm; // whole-matrix rebuild — - g.dp.clear(); // expensive, so DEFERRED - g.dm.clear(); // behind a sync policy -} +```c +// src/graph/delta_matrix/delta_set_element_bool.c + // ... 1-30: elided — argument checks and matrix extraction ... + 31 bool in_m; + 32 info = GxB_Matrix_isStoredElement(m, i, j); + // ... 33-35: elided ... + 36 info = GrB_Matrix_removeElement(dm, i, j); // re-add: drop the tombstone + // ... 37-38: elided ... + 39 info = GrB_Matrix_setElement_BOOL(dp, true, i, j); // never touch M ``` -An entry may be in M, in DP, or in M+DM (deleted but not yet flushed) -— never in both DP and DM; the ASCII state diagrams in the header -enumerate the legal states. `wait()` is the compaction: M = (M ∪ DP) -∖ DM, a whole-matrix rebuild, deliberately deferred behind a sync -policy so FalkorDB controls WHEN it pays. Even matrix multiply has a -delta-aware variant that accounts for pending changes without -flushing. And this IS topic 4's LSM: DP the memtable, M the SST, DM -the tombstones, `wait()` the compaction. Why it matters: this overlay -is what makes "graph as matrices" viable as a *database* rather than -an analytics batch tool — reads stay algebraic, writes stay O(1)-ish, -and the rebuild bill is paid on FalkorDB's schedule. +The distinction matters: re-adding an entry that M already holds is a +DM removal, and adding a genuinely new entry is a DP insert. Neither +path touches M, which is the whole point. +`delta_remove_element.c:36-43` is the mirror image — in M means set +DM, not in M means remove from DP. -## Where each step lives in the code +Reads are also not three probes. `delta_isStored.c` short-circuits +DP → DM → M (`:26`, `:32`, `:39`), and `delta_extract.c` does the same +at `:25`, `:31`, `:38`. An entry that lives in DP costs **one** probe, +not three. -**GraphBLAS** ([SuiteSparse](https://github.com/DrTimothyAldenDavis/GraphBLAS), -shallow clone): +**Correction.** The previous version described `wait()` as a single +`M = (M ∪ DP) ∖ DM` rebuild. `delta_wait.c` (218 lines) is two +independent flushes, each gated on its own counter: -- **Step 3** — `Include/GraphBLAS.h`: `GxB_HYPERSPARSE` (`:1664`), - `GxB_BITMAP` (`:1666`), plus `GxB_SPARSE`/`GxB_FULL` nearby; switch - thresholds `GxB_HYPER_SWITCH` (`:1556`), `GxB_BITMAP_SWITCH` - (`:1559`). -- **Step 4** — `Source/mxm/GB_AxB_meta.c:20-21`, the header comment IS - the algorithm menu: +```c +// src/graph/delta_matrix/delta_wait.c + 13 static void Delta_Matrix_sync_deletions(Delta_Matrix C) { + // ... 14-28: elided ... + 29 info = GrB_transpose(m, dm, NULL, m, GrB_DESC_RSCT0); // M = M .* !DM + // ... 30-32: elided ... + 33 info = GrB_Matrix_clear(dm); + // ... 34-35: elided ... + 36 static void Delta_Matrix_sync_additions(Delta_Matrix C) { + // ... 37-50: elided ... + 51 info = GrB_Matrix_assign(m, dp, NULL, dp, GrB_ALL, nrows, + 52 GrB_ALL, ncols, GrB_DESC_S); // M |= DP + // ... 53-55: elided ... + 56 info = GrB_Matrix_clear(dp); + // ... 57-88: elided — Delta_Matrix_sync begins at :59 ... + 89 if(delta_minus_nvals >= delta_max_pending_changes) { + // ... 90-96: elided ... + 97 if(delta_plus_nvals >= delta_max_pending_changes) { + // ... 98-102: elided ... + 103 info = GrB_wait(m, GrB_MATERIALIZE); + 104 info = GrB_wait(dm, GrB_MATERIALIZE); + 105 info = GrB_wait(dp, GrB_MATERIALIZE); +``` - > generic: for any semiring; dot2/dot3: does `C=A'*B`, `C=A'*B` ... - > saxpy: Gustavson + Hash +Deletions flush via a masked transpose, additions via an assign, and +with `force_sync == false` each side flushes **only when its own +pending count crosses a threshold**. The threshold is a config knob +with a default you can read: - The `dot2/dot3/dot4` files sit in the same `Source/mxm/` directory; - dot3 is the masked variant driven BY the mask. -- **Step 5** — `Source/mask/GB_masker.c:2,10` — computes - `R = masker(C, M, Z)`, i.e. `R = Z`: entries of Z where M is - true, entries of C elsewhere. +```c +// src/configuration/config.h + 19 #define DELTA_MAX_PENDING_CHANGES_DEFAULT 10000 +``` -**FalkorDB** ([repo](https://github.com/FalkorDB/FalkorDB), local at -`~/repos/FalkorDB`): +That number is the amortisation. Work it against Step 6's cost: -- **Step 1** — `src/graph/graph.h:48-52` — the graph IS matrices: +``` + flush-per-write: 34.0 MB touched per edge + flush per 10 000: one rebuild touches O(nnz) ≈ 16.0e6 entries + amortised = 16.0e6 / 10 000 = 1 600 entries/insert + vs a per-write CSR rebuild of 16.0e6 entries/insert + improvement factor = 10 000× (exactly the threshold, by construction) +``` + +Even the multiply is delta-aware. `delta_mxm.c:44` states the +identity `(A * (M + 'delta-plus'))` — but read `:47` +before believing that the deltas are free on both sides: ```c -Delta_Matrix adjacency_matrix; // all connections -Delta_Matrix *labels; // one boolean matrix per label -Delta_Matrix node_labels; // node id → label id mapping -Tensor *relations; // one matrix per relation type +// src/graph/delta_matrix/delta_mxm.c + 44 // C = A * (M + 'delta-plus') + // ... 45-46: elided ... + 47 ASSERT(Delta_Matrix_Synced(A)); // A must already be flushed + // ... 48-73: elided ... + 74 info = GrB_mxm(mask, NULL, NULL, semiring, a, dm, NULL); // mask = A·DM + // ... 75-85: elided ... + 86 info = GrB_mxm(accum, NULL, NULL, semiring, a, dp, NULL); // accum = A·DP + // ... 87-103: elided ... + 104 info = GrB_mxm(_C, mask, NULL, semiring, a, m, GrB_DESC_RSC); + // ... 105-106: elided ... + 107 info = GrB_eWiseAdd(_C, NULL, NULL, plus, _C, accum, NULL); ``` -- **Step 7** — `src/graph/delta_matrix/delta_matrix.h:17-22` (the - trio), `:26-80` (the ASCII state diagrams — the spec); - `delta_set_element_bool.c` (writes go to DP, or clear DM if - re-adding); `delta_remove_element.c` (deletes set DM, or clear DP); - `delta_wait.c` / `delta_will_wait.c` (the flush, triggered by the - sync policy — `graph.h:46` `SyncMatrixFunc`); `delta_mxm.c` (mxm - that accounts for pending deltas without flushing). +Only **B**'s deltas are handled without flushing; A is asserted +synced. An entry may be in M, in DP, or in M+DM (deleted but not yet +flushed) — never in both DP and DM; the ASCII state diagrams at +`delta_matrix.h:26-106` enumerate the legal states and are the real +specification. And this IS topic 4's LSM: DP the memtable, M the SST, +DM the tombstones, `Delta_Matrix_sync` the compaction, 10 000 the +compaction trigger. Why it matters: this overlay is what makes "graph +as matrices" viable as a *database* rather than an analytics batch +tool — reads stay algebraic, writes stay O(1)-ish, and the rebuild +bill is paid on FalkorDB's schedule. -Read order: `graph.h` (30 lines tell you the whole architecture) → -`delta_matrix.h` state diagrams → the four delta C files → then -GraphBLAS's format/algorithm anchors as the layer below. +## Where each step lives in the code + +**GraphBLAS** ([SuiteSparse](https://github.com/DrTimothyAldenDavis/GraphBLAS), +pinned at `1fd5475`): + +| Step | Anchor | What is there | +|---|---|---| +| 3 | `Include/GraphBLAS.h:1664-1667` | the four sparsity format constants | +| 3 | `Include/GraphBLAS.h:1556`, `:1559` | `GxB_HYPER_SWITCH`, `GxB_BITMAP_SWITCH` field ids | +| 3 | `Include/GraphBLAS.h:1715-1728` | the exact switch rule, both directions | +| 3 | `Include/GraphBLAS.h:1734` | `GxB_ALWAYS_HYPER` / `GxB_NEVER_HYPER` | +| 3 | `Source/include/GB_defaults.h:20` | `GB_HYPER_SWITCH_DEFAULT (0.0625)` | +| 4 | `Source/mxm/GB_AxB_dot.c:21-26` | dot2 Ω(m·n), dot3 Ω(nnz(M)), dot4 | +| 4 | `Source/mxm/GB_AxB_saxpy.c:18` | Gustavson / Hash / Bitmap | +| 4 | `Source/mxm/GB_AxB_meta_adotb_control.c:36`, `:72-87` | the per-call choice | +| 4 | `Source/mxm/GB_mxm.h:235-243` | `GB_AxB_dot3_control` — when dot3 is legal | +| 5 | `Source/mask/GB_masker.c:2`, `:10`, `:14-15` | the *late* mask path and its only caller | +| 5 | `Source/mxm/GB_AxB_meta.c:15-18` | the warning that masking may be deferred | +| 6 | `Source/include/GB_defaults.h:27` | `GB_PENDING_INIT 256` — GraphBLAS's own overlay | + +**FalkorDB** ([repo](https://github.com/FalkorDB/FalkorDB), pinned at +`ccb449a9a`): + +| Step | Anchor | What is there | +|---|---|---| +| 1 | `src/graph/graph.h:44` | `struct Graph` opens | +| 1 | `src/graph/graph.h:48-51` | the four matrix members (was cited as `:48-52`) | +| 7 | `src/graph/graph.h:42` | `SyncMatrixFunc` typedef (was cited as `:46`) | +| 7 | `.../delta_matrix/delta_matrix.h:17-22` | accessor **macros**, not the struct | +| 7 | `.../delta_matrix/delta_matrix.h:108-115` | `struct _Delta_Matrix` — the actual trio | +| 7 | `.../delta_matrix/delta_matrix.h:26-106` | the ASCII state diagrams (was cited as `:26-80`) | +| 7 | `.../delta_set_element_bool.c:31-39` | write path, branching on M | +| 7 | `.../delta_remove_element.c:36-43` | delete path, the mirror image | +| 7 | `.../delta_isStored.c:26,32,39` | short-circuiting DP → DM → M read | +| 7 | `.../delta_extract.c:25,31,38` | the same order, for range extraction | +| 7 | `.../delta_wait.c:13-34`, `:36-57`, `:89`, `:97`, `:103-105` | the two flushes and their thresholds | +| 7 | `src/configuration/config.h:19` | `DELTA_MAX_PENDING_CHANGES_DEFAULT 10000` | +| 7 | `.../delta_get_set.c:44-53` | DP/DM pinned hypersparse | +| 7 | `.../delta_mxm.c:44`, `:47`, `:74`, `:86`, `:104`, `:107` | delta-aware multiply, and the A-must-be-synced assert | + +Read order: `graph.h:44-53` (ten lines tell you the whole +architecture) → `delta_matrix.h:26-106` state diagrams → `delta_wait.c` +(the only file that tells you *when* the price is paid) → the three +smaller delta C files → then GraphBLAS's format/algorithm anchors as +the layer below. ## Questions (answer in notes.md) @@ -243,26 +555,131 @@ GraphBLAS's format/algorithm anchors as the layer below. ## Done when -- [ ] You can write the `read = (M ∪ DP) ∖ DM` identity and explain what each of the three matrices holds. -- [ ] You can explain why CSR is hostile to single-edge inserts, and why that fact alone forces something like Delta_Matrix. -- [ ] You can say when dot beats saxpy for a BFS step, in terms of frontier size against matrix dimension. -- [ ] You can explain what a mask pushes into the kernel and what it saves — connect it to the masked-SpMV lane in this topic's bench. +Answer each before unfolding it. + +- [ ] You can write the `read = (M ∪ DP) ∖ DM` identity, explain what each of the three matrices holds, and say how many probes a read actually costs. + +
Answer + + `read(i,j) = (M(i,j) OR DP(i,j)) AND NOT DM(i,j)`. M is the + read-optimized main matrix (CSR or hypersparse inside); DP holds + pending additions; DM holds pending deletions — tombstones, because + removing an entry from M in place is Step 6's problem again. Both + deltas are pinned hypersparse at `delta_get_set.c:46-53`. + + A read is **not** three probes. `delta_isStored.c` tests DP first + (`:26`), then DM (`:32`), then M (`:39`), returning early. An entry + that was just written costs one probe; only an entry absent from + both deltas pays all three. + +
+ +- [ ] You can explain why CSR is hostile to single-edge inserts, and compute the cost on this topic's 16 M-edge graph. + +
Answer + + CSR stores row i's neighbours as a contiguous slice, so inserting + one entry into row i means shifting every later element of + `targets` and bumping every later element of `offsets`. On the + bench graph (1 M nodes, 16.0 M edges, 4-byte indices) `targets` is + 16e6 × 4 B = 64.0 MB and `offsets` is 1 000 001 × 4 B = 4.0 MB; an + average insert lands mid-array, so ≈ 32.0 MB + 2.0 MB = 34.0 MB is + touched per edge. + + That is the argument for Delta_Matrix, and FalkorDB's threshold + makes the saving explicit: `DELTA_MAX_PENDING_CHANGES_DEFAULT` is + 10 000 (`config.h:19`), so one rebuild is amortised over 10 000 + writes — a 10 000× reduction in rebuild work per write, by + construction. + +
+ +- [ ] You can say when dot beats saxpy for a BFS step, in terms of frontier size against matrix dimension, and quote the two Ω's. + +
Answer + + `GB_AxB_dot.c:22-25`: dot2 computes `C=A'*B` and `C=A'*B` in + **Ω(m·n)**; dot3 computes `C=A'*B` examining only entries of M, + in **Ω(nnz(M))**. `GB_mxm.h:235-243` says dot3 is only eligible when + a non-complemented, sparse-or-hypersparse mask exists. + + So a small frontier gives a small sparse mask and wants dot3: on + the triangle query over the bench graph, dot3's Ω(nnz(A)) = 1.6e7 + against dot2's Ω(n²) = 1.0e12, a factor of 62 500. A frontier of + 10⁶ on a 10⁶-node graph makes the mask nearly full, the dot3 + eligibility test fails or stops paying, and saxpy — the default set + at `GB_AxB_meta_adotb_control.c:36` — streams the whole thing + instead. + +
+ +- [ ] You can explain what a mask pushes into the kernel, what it saves, and the one case where passing a mask saves nothing. + +
Answer + + A mask restricts where output may be produced, so `C = A²` + computes only the 16 M positions where an edge already exists + rather than the up-to-10¹² cells of A². That is predicate pushdown + reaching the innermost loop, and it is the reason the masked-SpMV + lane in this topic's bench has something to attack on supernodes — + the oracle spends 6.28 ns per distinct node reached there against + 4.81 ns from random sources (78 907 against 1022 nodes per query), + and a mask makes re-walking unrepresentable. + + The case where it saves nothing: `GB_masker.c` is the *late* path — + it computes Z in full and then merges `R = C ; R = Z`. It is + reached from `GB_accum_mask` (`GB_masker.c:14-15`), and + `GB_AxB_meta.c:15-18` warns the algorithm may choose to defer the + mask to it. A deferred mask costs the intermediate you were hoping + to avoid. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the Delta_Matrix-to-LSM vocabulary mapping. +
Answer + + DP is the memtable (small, mutable, hypersparse, absorbs writes); + M is the SST/level (large, immutable in practice, read-optimized); + DM is the tombstone set; `Delta_Matrix_sync` (`delta_wait.c:59`) is + the compaction; `DELTA_MAX_PENDING_CHANGES_DEFAULT` = 10 000 + (`config.h:19`) is the compaction trigger. + + The one place the analogy is tighter than LSM: the two sides flush + *independently* (`delta_wait.c:89` for deletions, `:97` for + additions), so a delete-heavy workload can compact tombstones + without rewriting for additions. The one place it is looser: there + is no level hierarchy — one M, one DP, one DM, full stop. + +
+ ## References **Papers** - Davis — "Algorithm 1000: SuiteSparse:GraphBLAS: Graph Algorithms in the Language of Sparse Linear Algebra" (ACM TOMS 2019) — optional - companion; the code comments above cover the same ground - -**Code** -- [GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) - (SuiteSparse, shallow clone) — `Include/GraphBLAS.h` for the four - formats and switch thresholds, `Source/mxm/GB_AxB_meta.c` (the - header comment is the algorithm menu), `Source/mask/GB_masker.c` -- [FalkorDB](https://github.com/FalkorDB/FalkorDB) — - `src/graph/graph.h`, `src/graph/delta_matrix/delta_matrix.h` (the - ASCII state diagrams in the header are the spec), plus - `delta_set_element_bool.c`, `delta_remove_element.c`, - `delta_wait.c`, `delta_mxm.c` + companion; the code comments cited above cover the same ground and + are the ones that were actually checked for this chapter. + +**Code** (all line numbers verified at the pins named at the top) + +| Repo | File | Lines | What | +|---|---|---|---| +| GraphBLAS | `Include/GraphBLAS.h` | 1556, 1559, 1664-1667, 1715-1728, 1734 | format constants, switch fields, the switch rule | +| GraphBLAS | `Source/include/GB_defaults.h` | 20, 27 | hyper switch default 0.0625; pending-tuple init 256 | +| GraphBLAS | `Source/mxm/GB_AxB_dot.c` | 21-26 | dot2/dot3/dot4 and their Ω's | +| GraphBLAS | `Source/mxm/GB_AxB_saxpy.c` | 18 | Gustavson / Hash / Bitmap | +| GraphBLAS | `Source/mxm/GB_AxB_meta_adotb_control.c` | 36, 72-87 | which kernel, per call | +| GraphBLAS | `Source/mxm/GB_AxB_meta.c` | 15-18, 20-21 | saxpy-or-dot; mask may be deferred | +| GraphBLAS | `Source/mxm/GB_mxm.h` | 235-243 | `GB_AxB_dot3_control` | +| GraphBLAS | `Source/mask/GB_masker.c` | 2, 10, 14-15, 21-33 | the late mask path and its truth table | +| FalkorDB | `src/graph/graph.h` | 42, 44, 48-53 | `SyncMatrixFunc`; the graph as four matrices | +| FalkorDB | `src/graph/delta_matrix/delta_matrix.h` | 17-22, 26-106, 108-115 | macros; state diagrams; the struct | +| FalkorDB | `src/graph/delta_matrix/delta_set_element_bool.c` | 31-39 | write path | +| FalkorDB | `src/graph/delta_matrix/delta_remove_element.c` | 36-43, 50-81 | delete path, bulk delete | +| FalkorDB | `src/graph/delta_matrix/delta_isStored.c` | 26, 32, 39 | short-circuiting read | +| FalkorDB | `src/graph/delta_matrix/delta_extract.c` | 25, 31, 38 | same order for extraction | +| FalkorDB | `src/graph/delta_matrix/delta_wait.c` | 13-34, 36-57, 59-113, 89, 97, 103-105 | the two flushes and their gates | +| FalkorDB | `src/graph/delta_matrix/delta_get_set.c` | 44-53 | DP/DM pinned hypersparse | +| FalkorDB | `src/graph/delta_matrix/delta_mxm.c` | 44, 47, 74, 86, 104, 107 | delta-aware mxm | +| FalkorDB | `src/configuration/config.h` | 19, 33 | the flush threshold and its config enum | diff --git a/topics/13-graph-engines/reading-kuzu.md b/topics/13-graph-engines/reading-kuzu.md index 5ba701b..af267c5 100644 --- a/topics/13-graph-engines/reading-kuzu.md +++ b/topics/13-graph-engines/reading-kuzu.md @@ -9,6 +9,15 @@ design step by step: edges as a columnar table, CSR as an index over it, the per-node-group update fix, the Intersect operator, and factorization. +Every code anchor below is **kuzu pinned at `89f0263`** +([`resources/codebases.md`](../../resources/codebases.md)); every +paper claim is from Feng, Jin, Chen, Liu & Salihoğlu, *KÙZU Graph +Database Management System*, CIDR 2023, cited by section. Mind the +gap: the paper is from January 2023 and the pin is much later, so +where the two disagree the code wins and the chapter says so. Two +numbers the previous version of this chapter asserted — the node +group size, and the shape of the CSR — did not survive the check. + ## The problem in one sentence If edges are just rows of a columnar table clustered by source node, @@ -22,11 +31,22 @@ kuzu ships one mechanism for each. ### Step 1 — adjacency is a columnar table clustered by source -kuzu stores a relationship table the way DuckDB stores any table — -in **node groups** (horizontal slices, ≈ DuckDB's row groups from -topic 12), each column separately — with one twist: the rows are -edges, sorted by source node id, and column 0 is the neighbor id, -column 1 the rel id, edge properties follow: +> **In:** a relationship table, i.e. a set of (src, dst, type, +> properties) rows. +> **Out:** those rows laid out as columns, horizontally sliced into +> node groups and sorted by source — so that "expand node i" is a +> range of rows rather than a search. + +kuzu stores a relationship table the way DuckDB stores any table — in +**node groups** (horizontal slices, ≈ DuckDB's row groups from topic +12), each column separately — with one twist: the rows are edges, +sorted by source node id, and the first two columns are fixed: + +```cpp +// src/include/storage/table/csr_node_group.h + 162 static constexpr common::column_id_t NBR_ID_COLUMN_ID = 0; + 163 static constexpr common::column_id_t REL_ID_COLUMN_ID = 1; +``` ``` rel table (one node group), sorted by src: @@ -36,89 +56,369 @@ column 1 the rel id, edge properties follow: props: ...columns like any table... ``` +The paper states the same design and adds the part the header does +not: the edges are **double indexed**, forward and backward, and the +edge *properties* are stored in parallel CSR-shaped structures of +their own — + +> "Edges are double indexed and stored in CSR-based adjacency list +> indices …, which are the core join indices in the system to join +> node records. … Edge properties are similarly stored in 'parallel' +> but separate CSR-based structures and double-indexed … This has +> storage and update costs yet ensures that we can scan any node's +> edges and properties of these edges sequentially in both forward +> and backward directions." +> — CIDR '23 §2, *Storage and Indices* + +"Storage and update costs" is the paper's own admission that double +indexing doubles the write. That is the same trade memgraph makes by +keeping `in_edges` and `out_edges` +([reading-memgraph-storage.md](reading-memgraph-storage.md) Step 3) — +kuzu just pays it in columns instead of in per-vertex vectors. + Because all of node 3's edges are adjacent rows, "expand node 3" is a -contiguous slice of every column — and compression, zone maps, and -vectorized scans apply to edges for free. Why it matters: three other -engines in this topic built custom edge storage; kuzu's bet is that -the columnar machinery already solved storage, and graphs only need -two extra operators on top. +contiguous slice of every column. Why it matters: three other engines +in this topic built custom edge storage; kuzu's bet is that the +columnar machinery already solved storage, and graphs only need two +extra operators on top. ### Step 2 — the CSR header: turning sorted rows into O(1) expand +> **In:** a node group's sorted edge rows and a bound node id. +> **Out:** that node's row range — by array indexing, not by search — +> plus the reason kuzu's CSR is not the textbook one. + To find node 3's slice without searching, each node group carries a -**CSR header** (compressed sparse row — an offsets array where entry -i holds the position where node i's edges start; node i's edges are -rows `offsets[i] .. offsets[i+1]`). So adjacency = a columnar table -clustered by src **with a CSR index on top**: +**CSR header**: an offsets array where entry i holds the position +where node i's edges start, so node i's edges are rows +`offsets[i] .. offsets[i+1]`. kuzu stores it as two columns — + +```cpp +// src/include/storage/table/csr_node_group.h + 148 struct CSRNodeGroupCheckpointState final : NodeGroupCheckpointState { + 149 Column* csrOffsetColumn; + 150 Column* csrLengthColumn; + 151 + 152 std::unique_ptr oldHeader; + 153 std::unique_ptr newHeader; +``` + +— offsets *and* lengths, which is already a hint: textbook CSR does +not need lengths, because `offsets[i+1] - offsets[i]` is the length. +Storing both means the rows for node i do **not** run right up to the +rows for node i+1. There is slack between them, and the slack is the +whole update story. + +**Correction.** The previous version of this chapter described plain +CSR. What kuzu builds is a **packed CSR** — a +packed-memory-array-style layout with deliberate gaps, governed by a +calibrator tree and density thresholds: + +```cpp +// src/include/storage/table/csr_node_group.h + 99 // TODO(Guodong): Serialize the info to disk. This should be a config per node group. + 100 struct PackedCSRInfo { + 101 static_assert(common::StorageConfig::NODE_GROUP_SIZE_LOG2 > + 102 common::StorageConfig::CSR_LEAF_REGION_SIZE_LOG2); + 103 uint64_t calibratorTreeHeight = common::StorageConfig::NODE_GROUP_SIZE_LOG2 - + 104 common::StorageConfig::CSR_LEAF_REGION_SIZE_LOG2; + 105 double highDensityStep = (common::StorageConstants::LEAF_HIGH_CSR_DENSITY - + 106 common::StorageConstants::PACKED_CSR_DENSITY) / + 107 static_cast(calibratorTreeHeight); + 108 + 109 constexpr PackedCSRInfo() noexcept = default; + 110 }; +``` + +Every constant in that expression is readable. Two live in +`common/constants.h`: + +```cpp +// src/include/common/constants.h, struct StorageConstants + 78 static constexpr double PACKED_CSR_DENSITY = 0.8; + 79 static constexpr double LEAF_HIGH_CSR_DENSITY = 1.0; +``` + +and two come from a CMake-configured header, whose *defaults* are in +the top-level `CMakeLists.txt`: + +```cpp +// cmake/templates/system_config.h.in, struct StorageConfig + 47 static constexpr uint64_t NODE_GROUP_SIZE_LOG2 = @KUZU_NODE_GROUP_SIZE_LOG2@; + 48 static constexpr uint64_t NODE_GROUP_SIZE = static_cast(1) << NODE_GROUP_SIZE_LOG2; + 49 // The number of CSR lists in a leaf region. + 50 static constexpr uint64_t CSR_LEAF_REGION_SIZE_LOG2 = + 51 std::min(static_cast(10), NODE_GROUP_SIZE_LOG2 - 1); +``` + +```cmake +# CMakeLists.txt + 126 option(KUZU_NODE_GROUP_SIZE_LOG2 "Log2 of the vector capacity." 17) + 127 if(NOT KUZU_NODE_GROUP_SIZE_LOG2) + 128 set(KUZU_NODE_GROUP_SIZE_LOG2 17) +``` + +Now do the arithmetic, because it is all determined: ``` - offsets: [0, 2, 3, 3, 6, ...] neighbors(3) = rows 3..6 of the group + NODE_GROUP_SIZE_LOG2 = 17 → NODE_GROUP_SIZE = 131 072 nodes + CSR_LEAF_REGION_SIZE_LOG2 = min(10, 16) = 10 + → CSR_LEAF_REGION_SIZE = 1 024 CSR lists + calibratorTreeHeight = 17 − 10 = 7 + cross-check: 131 072 / 1 024 = 128 leaf regions, and 2^7 = 128 ✓ + highDensityStep = (1.0 − 0.8) / 7 = 0.02857… ``` -The header costs one array per node group and turns expand into slice -arithmetic — no binary search, no pointer chase. Why it matters: this -is the same CSR as FalkorDB's matrices-in-CSR, arrived at from the -relational direction — the representations converge; what differs is -the machinery around them. +**Correction.** The previous version said a node group is "say, 64K +nodes". It is 2^17 = **131 072** at the default build, twice that. + +The densities say what the slack is for: a leaf region is kept at +0.8 occupancy, and the allowed density ramps from 0.8 up to 1.0 over +the 7 levels of the calibrator tree in steps of 0.0286 — so a small +insert fills local slack, a bigger one redistributes within a leaf +region, and only a large one rebalances a whole subtree. Size the +slack on this topic's graph (1 M nodes, 16.0 M directed edges, +[notes.md](notes.md)): + +``` + node groups needed = 1 000 000 / 131 072 = 7.63 → 8 groups + edges per group = 16.0e6 / 8 = 2.0e6 + slots reserved at density 0.8 = 2.0e6 / 0.8 = 2.5e6 + free slots per group = 2.5e6 − 2.0e6 = 500 000 (25% headroom) +``` + +Why it matters: this is the same CSR as FalkorDB's matrices-in-CSR, +arrived at from the relational direction — but with the gaps that +make it writable built into the layout rather than bolted on as a +separate overlay matrix. ### Step 3 — surviving updates: persistent CSR + transient overlay, per node group -CSR hates single-edge inserts (everything after the insertion point -shifts), so kuzu splits each node group in two: **persistent data** -(the checkpointed, CSR-formatted chunk) plus **transient data** -(in-memory chunked buffers, append-only, with a `csrIndex` mapping -bound node → row indices); reads merge both. At checkpoint, the -transient rows are merged into a rebuilt CSR *for that node group -only* — **update pain is bounded per node group**, not per graph: +> **In:** a stream of single-edge inserts arriving between +> checkpoints. +> **Out:** rows appended to an in-memory chunk plus an index entry, +> with the CSR rebuild deferred to checkpoint and bounded to one node +> group. + +The slack of Step 2 absorbs *some* inserts. For the rest, kuzu splits +each node group in two, and the header says so in five lines that are +worth reading before any of the code: + +```cpp +// src/include/storage/table/csr_node_group.h + 165 // Data in a CSRNodeGroup is organized as follows: + 166 // - persistent data: checkpointed data or flushed data from batch insert. `persistentChunkGroup`. + 167 // - transient data: data that is being committed but kept in memory. `chunkedGroups`. + 168 // Persistent data are organized in CSR format. + 169 // Transient data are organized similar to normal node groups. Tuples are always appended to the end + 170 // of `chunkedGroups`. We keep an extra csrIndex to track the vector of row indices for each bound + 171 // node. + 172 class CSRNodeGroup final : public NodeGroup { +``` + +The `csrIndex` entry per bound node is `NodeCSRIndex` +(`csr_node_group.h:30-59`), and it has a nice compression of its own: +if the node's transient rows happen to be consecutive it stores +`isSequential = true` and just a (start, length) pair; otherwise it +stores the explicit row list. ``` - read(node i) = persistent CSR slice ∪ transient rows for i + read(node i) = persistent CSR slice ∪ transient rows from csrIndex[i] checkpoint = rebuild ONE node group's CSR (oldHeader -> newHeader) ``` +The rebuild granularity is the point, and now it is a number: + +``` + FalkorDB `Delta_Matrix_sync`: rebuilds a whole matrix + ≈ 16.0e6 entries on this graph + kuzu checkpoint: rebuilds one node group + = 16.0e6 / 8 = 2.0e6 rows + ratio = 8× smaller worst-case stall, and it does not grow with the + graph — it grows with 131 072 nodes' worth of edges, full stop +``` + Same LSM-shaped answer as FalkorDB's Delta_Matrix (read-optimal core + -mutable overlay + deferred merge), with a different merge granularity: -FalkorDB rebuilds a whole matrix on `wait()`; kuzu rebuilds one node -group of, say, 64K nodes. Why it matters: merge granularity decides -the worst-case write stall — bounding it per group is the -disk-friendly choice for a system that checkpoints. +mutable overlay + deferred merge) with a different merge granularity. +Why it matters: merge granularity decides the worst-case write stall — +bounding it per group is the disk-friendly choice for a system that +checkpoints. The failure mode it does *not* fix is a supernode: all +of one node's edges live in the node group of its **source id**, so a +node with 6 565 edges concentrates 6 565 rows in one group's slack +budget, and repeated inserts on it will force that group's rebuild +over and over while the other seven sit idle. ### Step 4 — the Intersect operator: worst-case optimal joins where they pay -For cyclic patterns, binary join plans are asymptotically wrong — the +> **In:** a set of bound node pairs and their sorted neighbour lists. +> **Out:** the intersection of those lists — the third variable of a +> cyclic pattern, produced directly instead of enumerated and +> filtered. + +For cyclic patterns, binary join plans are asymptotically wrong. The triangle `(a)->(b), (b)->(c), (a)->(c)` via pairwise joins can -materialize Θ(m²) intermediate (a,b,c-candidate) pairs when the true -output is at most m^1.5 (the AGM bound — see -[reading-wcoj.md](reading-wcoj.md) for the theory). kuzu's fix is a -physical `Intersect` operator: binary-join to get (a,b) pairs, then -for each pair **intersect N(a) ∩ N(b)** — the sorted neighbor lists -from Step 2's CSR slices — to produce c directly, never enumerating -candidates a later edge would kill. The build side -(`intersect_build.h`) prepares sorted adjacency lists in a hash table -keyed by node. - -Note it's a **hybrid**: the optimizer picks Intersect only where +materialize Θ(m²) intermediate pairs when the true output is at most +m^1.5 — the AGM bound, whose statement and attribution are in +[reading-wcoj.md](reading-wcoj.md) Step 2. On this topic's graph: + +``` + m = 16.0e6 edges + binary-join intermediate: m² = 2.56e14 pairs + AGM ceiling on the output: m^1.5 = 16.0e6 × √(16.0e6) + = 16.0e6 × 4 000 = 6.4e10 + ratio = m² / m^1.5 = √m = 4 000× +``` + +kuzu's physical answer is an `Intersect` operator. Read it in the +header first: + +```cpp +// src/include/processor/operator/intersect/intersect.h + 29 class Intersect : public PhysicalOperator { + // ... 30-53: elided — constructor, init, getNextTuplesInternal, copy ... + 54 private: + 55 // For each build side, probe its HT and return a vector of matched flat tuples. + 56 void probeHTs(); + 57 // Left is always the one with less num of values. + 58 static void twoWayIntersect(common::nodeID_t* leftNodeIDs, common::SelectionVector& lSelVector, + 59 common::nodeID_t* rightNodeIDs, common::SelectionVector& rSelVector); + 60 void intersectLists(const std::vector& listsToIntersect); +``` + +The kernel is a plain sorted merge — which is why the lists must be +sorted, and the source is unambiguous about it: + +```cpp +// src/processor/operator/intersect/intersect.cpp + 65 void Intersect::twoWayIntersect(nodeID_t* leftNodeIDs, SelectionVector& lSelVector, + 66 nodeID_t* rightNodeIDs, SelectionVector& rSelVector) { + 67 KU_ASSERT(lSelVector.getSelSize() <= rSelVector.getSelSize()); + // ... 68-71: elided — buffers and cursors ... + 72 while (leftPosition < lSelVector.getSelSize() && rightPosition < rSelVector.getSelSize()) { + // ... 73-74: elided ... + 75 if (leftNodeID < rightNodeID) { + 76 leftPosition++; + 77 } else if (leftNodeID > rightNodeID) { + 78 rightPosition++; + 79 } else { + // ... 80-82: elided — record the match in both selection vectors ... + 83 leftPosition++; + 84 rightPosition++; + 85 outputValuePosition++; + 86 } + 87 } +``` + +Two details make this the skew-aware version rather than the naive +one: + +```cpp +// src/processor/operator/intersect/intersect.cpp + 103 static std::vector swapSmallestListToFront(std::vector& lists) { + // ... 104-107: elided ... + 108 for (auto i = 1u; i < lists.size(); i++) { + 109 if (lists[i].numElements < lists[smallestListIdx].numElements) { + 110 smallestListIdx = i; + 111 } + 112 } +``` + +The smallest list goes first, so the fold starts from the tightest +constraint — which bounds the running intermediate by the *smallest* +degree in the pattern rather than the largest. That is the same idea +as Generic Join picking the smallest candidate set per variable +([reading-wcoj.md](reading-wcoj.md) Step 3). + +And the sortedness the merge assumes is guaranteed on the build side, +not hoped for: + +```cpp +// src/include/processor/operator/intersect/intersect_build.h + 35 class IntersectBuild final : public HashJoinBuild { + // ... 36-44: elided — type tag and constructor ... + 45 uint64_t appendVectors() final { + 46 KU_ASSERT(keyVectors.size() == 1); + 47 return hashTable->appendVectorWithSorting(keyVectors[0], payloadVectors); + 48 } +``` + +`IntersectBuild` *is* a `HashJoinBuild` with one method overridden — +`appendVectorWithSorting` instead of the plain append. That single +override is the entire difference, and it is why Step 2's CSR +ordering is a precondition rather than a nicety. + +Note it is a **hybrid**: the optimizer picks Intersect only where cyclic patterns make binary joins asymptotically wrong; chains and trees stay ordinary binary hash joins (the topic-10/11 machinery). -Why it matters: WCOJ (worst-case optimal join — a join algorithm whose -runtime matches the AGM output bound) is a scalpel, not a religion — -kuzu shows it slotting into a standard vectorized plan as one more -operator. +Note also the honest gap between this code and the paper: CIDR '23 §1 +describes the wco join as being built on **ASP-Join** +(accumulate-semijoin-probe, a three-pipeline hash join using sideways +information passing), which is not what the `Intersect` operator at +this pin does. Read the operator for what it is — a multiway sorted +intersection over hash-table-resident sorted lists — and read the +paper for the direction the system was heading. Why it matters: WCOJ +is a scalpel, not a religion — kuzu shows it slotting into a standard +vectorized plan as one more operator. ### Step 5 — factorization: defer the cross product -One-to-many expands multiply rows: flat execution of -`MATCH (a)-[]->(b)-[]->(c)` materializes Σ deg(a)·deg(b) tuples — on a -power-law graph, billions of rows to represent what is structurally -"for each a: a list of b's; for each b: a list of c's". kuzu keeps -vectors **factorized**: a DataChunk can carry "unflat" vectors — a -group (all b's for one a) with a multiplicity — deferring the cross -product until an operator truly needs flat tuples (topic 11's -vector-type flags, pushed further). Aggregations never need it: +> **In:** a multi-hop pattern whose flat result is a product of +> degrees. +> **Out:** a factorized intermediate — groups plus multiplicities — +> whose size is a *sum* of degrees, with the product deferred until +> some operator actually demands flat tuples. + +One-to-many expands multiply rows. The paper's own worked example is +the cleanest statement of the problem: + +> "Consider a 𝑘-regular database, where a node 𝑣ᵢ has 𝑘 +> outgoing/incoming neighbors … Suppose, Karim has one account 𝑣₁, so +> the output has 𝑘² tuples. Figure 1 shows both the flat and the +> succinct factorized representation of this output." +> — CIDR '23 §1 + +The factorized form the paper writes for that example is +`T_{v₁} = {k backward neighbours} × (v₁, Karim) × {k forward +neighbours}` — a product *expression*, not a product. Count both +representations: + +``` + flat: k² tuples + factorized: k + 1 + k = 2k + 1 values + + k = 11 (this graph's p50 degree, notes.md): + flat 121, factorized 23 → 5.3× + k = 6 565 (this graph's max degree): + flat 43 099 225, factorized 13 131 → 3 282× +``` + +The saving is not a constant factor — it is k²/(2k+1) ≈ k/2, so it +grows with the skew that this topic's headline is about. kuzu keeps +vectors factorized in its `DataChunk`s, and the whole mechanism is one +two-valued enum plus the flag that carries it: + +```cpp +// src/include/common/data_chunk/data_chunk_state.h + 8 // F stands for Factorization + 9 enum class FStateType : uint8_t { + 10 FLAT = 0, + 11 UNFLAT = 1, + 12 }; + // ... 13-24: elided — the class, its capacity ctor, size init ... + 25 bool isFlat() const { return fStateType == FStateType::FLAT; } + 26 void setToFlat() { fStateType = FStateType::FLAT; } + 27 void setToUnflat() { fStateType = FStateType::UNFLAT; } +``` + +An `UNFLAT` chunk *is* the group — all b's for one a, carried once +with the a-side held flat beside it — and `setToFlat()` is where an +operator pays for the cross product it had been deferring (topic 11's +vector-type flags, pushed further). Aggregations never call it: ```rust -// factorized count(*) for a 2-hop: multiply group SIZES, never -// materialize the Σ deg(a)·deg(b) tuples a flat plan would build +// ILLUSTRATION — not kuzu source. The real flag is +// src/include/common/data_chunk/data_chunk_state.h:9-12 (FLAT/UNFLAT); +// the point here is only the arithmetic that UNFLAT makes legal. fn two_hop_count(csr: &Csr) -> u64 { (0..csr.n) .map(|a| { @@ -131,33 +431,48 @@ fn two_hop_count(csr: &Csr) -> u64 { ``` FalkorDB's matrix spelling of the same fact: A² holds PATH COUNTS as -its values — the algebra factorizes for you. Why it matters: -factorization is the executor-level answer to the blowup that WCOJ -answers at the plan level; together they're why kuzu can run -multi-hop patterns a flat vectorized engine chokes on. +its values, so the grand sum of A² *is* this number — the algebra +factorizes for you. Why it matters: factorization is the +executor-level answer to the blowup that WCOJ answers at the plan +level; together they are why kuzu can run multi-hop patterns a flat +vectorized engine chokes on. ## Where each step lives in the code -A shallow clone of [kuzu](https://github.com/kuzudb/kuzu); two headers -carry the chapter: - -- **Steps 1–3** — `src/include/storage/table/csr_node_group.h`: - - `:165-171` — the design comment (read it first: persistent CSR - chunk + transient in-memory chunked groups + `csrIndex`, reads - merge both — the storage story in one comment) - - `:172` — `class CSRNodeGroup final : public NodeGroup` — rel - tables reuse the node-group machinery; `:162-163` — column 0 is - neighbor id, column 1 rel id, properties follow - - `InMemChunkedCSRHeader` (`:117`, `:141`) — offsets+lengths built - in memory; checkpoint's per-group rebuild via - `oldHeader`/`newHeader` (`:152-153`) -- **Step 4** — `src/include/processor/operator/intersect/intersect.h:29` - `class Intersect : public PhysicalOperator`, plus - `intersect_build.h:35` (building sorted adjacency lists into a hash - table keyed by node). -- **Step 5** — no single anchor; the CIDR '23 paper's - §vectorization/factorization discussion is the part the code doesn't - narrate — read it after the two headers. +[kuzu](https://github.com/kuzudb/kuzu) pinned at `89f0263`. + +| Step | Anchor | What is there | +|---|---|---| +| 1 | `src/include/storage/table/csr_node_group.h:162-163` | column 0 = neighbour id, column 1 = rel id | +| 1 | `src/include/storage/table/csr_node_group.h:21-24` | `csr_list_t` — a (startRow, length) pair | +| 2 | `src/include/storage/table/csr_node_group.h:99-110` | `PackedCSRInfo` — calibrator tree height, density step | +| 2 | `src/include/common/constants.h:78-79` | `PACKED_CSR_DENSITY = 0.8`, `LEAF_HIGH_CSR_DENSITY = 1.0` | +| 2 | `cmake/templates/system_config.h.in:47-55` | node group size, leaf region size, chunk capacity | +| 2 | `CMakeLists.txt:114-130` | the defaults: page 2^12, vector 2^11, node group 2^17 | +| 2 | `src/include/storage/table/csr_node_group.h:114-146` | `CSRNodeGroupScanState`; `header` at `:117`, built at `:141-142` | +| 3 | `src/include/storage/table/csr_node_group.h:165-171` | the design comment — read this first | +| 3 | `src/include/storage/table/csr_node_group.h:172-174` | `class CSRNodeGroup`, `DEFAULT_PACKED_CSR_INFO` | +| 3 | `src/include/storage/table/csr_node_group.h:30-59` | `NodeCSRIndex` — sequential or explicit row list | +| 3 | `src/include/storage/table/csr_node_group.h:148-160` | checkpoint state: `oldHeader` `:152`, `newHeader` `:153` | +| 4 | `src/include/processor/operator/intersect/intersect.h:29, 56-60` | the operator and its three private kernels | +| 4 | `src/processor/operator/intersect/intersect.cpp:65-90` | `twoWayIntersect` — the sorted merge | +| 4 | `src/processor/operator/intersect/intersect.cpp:103-118` | `swapSmallestListToFront` — the skew heuristic | +| 4 | `src/include/processor/operator/intersect/intersect_build.h:35, 45-48` | `appendVectorWithSorting` — where sortedness comes from | +| 5 | `src/include/common/data_chunk/data_chunk_state.h:8-12, 25-27` | `FStateType::FLAT`/`UNFLAT` — the factorization flag | +| 5 | CIDR '23 §1 and §3.1 | the factorized-vector design the code does not narrate | + +Read order: the design comment at `csr_node_group.h:165-171`, then +`PackedCSRInfo` at `:99-110` with `constants.h:78-79` and +`system_config.h.in:47-55` open beside it so the constants resolve, +then `intersect.cpp:65-118` — the merge and the smallest-list-first +heuristic are forty readable lines and they are the whole of Step 4. + +One number worth carrying between chapters: kuzu's page size is 4 KiB +(`CMakeLists.txt:114-116`, `KUZU_PAGE_SIZE_LOG2 = 12`, corroborated by +CIDR '23 §2 "fixed page sizes (4KB)"), against neo4j's 8 KiB +(`PageCache.java:49`). Halving the page halves the read amplification +of a random single-record lookup and doubles the number of pages a +sequential scan must fault. ## Questions (answer in notes.md) @@ -176,22 +491,131 @@ carry the chapter: ## Done when -- [ ] You can explain how a CSR header turns sorted rows into an O(1) expand. -- [ ] You can describe the persistent-CSR-plus-transient-overlay scheme per node group, and state its worst-case update cost. -- [ ] You can say why Intersect requires sorted adjacency lists, and what breaks without that. +Answer each before unfolding it. + +- [ ] You can explain how a CSR header turns sorted rows into an O(1) expand, and say why kuzu's header stores lengths as well as offsets. + +
Answer + + The offsets array makes node i's rows the range + `offsets[i] .. offsets[i+1]` — two array reads and a slice, no + binary search and no pointer chase, in every column at once because + the columns are row-aligned. + + It stores lengths too (`csr_node_group.h:149-150`) because this is a + *packed* CSR: `PackedCSRInfo` (`:99-110`) keeps leaf regions at + `PACKED_CSR_DENSITY = 0.8` (`constants.h:78`), so there is slack + between one node's rows and the next node's, and `offsets[i+1] − + offsets[i]` would count the gap. The length column is what makes the + gaps invisible to a reader. + +
+ +- [ ] You can describe the persistent-CSR-plus-transient-overlay scheme per node group, and state its worst-case update cost in rows. + +
Answer + + `csr_node_group.h:165-171`: persistent data is the checkpointed + chunk in CSR format; transient data is appended to in-memory + `chunkedGroups`, with a `csrIndex` mapping each bound node to its + transient row indices. A read merges both. At checkpoint the group's + CSR is rebuilt, `oldHeader` → `newHeader` (`:152-153`). + + The cost is bounded by one node group. Defaults: + `KUZU_NODE_GROUP_SIZE_LOG2 = 17` (`CMakeLists.txt:126`) → + 131 072 nodes per group, so on a 1 M-node / 16 M-edge graph that is + 8 groups and ~2.0 M rows per rebuild — 8× smaller than FalkorDB's + whole-matrix `Delta_Matrix_sync`, and it stops growing once the + graph is bigger than one group. + + The pattern that still hurts: repeated inserts on one supernode. + All of a node's out-edges live in the group indexed by its source + id, so a 6 565-degree node keeps forcing the same group's rebuild + while the other seven groups do nothing. + +
+ +- [ ] You can say why Intersect requires sorted adjacency lists, what breaks without that, and which line guarantees it. + +
Answer + + `twoWayIntersect` (`intersect.cpp:65-90`) is a two-cursor merge: it + advances whichever side holds the smaller id and emits on equality. + That is O(|A| + |B|) *only* if both sides are sorted. On unsorted + input it does not merely get slower — it silently produces the wrong + answer, because it advances past values it will never revisit. + + The guarantee is `intersect_build.h:45-48`: `IntersectBuild` + subclasses `HashJoinBuild` and overrides `appendVectors()` to call + `appendVectorWithSorting`. One method. That is also why Step 2's + CSR ordering matters — the storage already delivers sorted lists, + so the build side is cheap. + +
+ - [ ] You can estimate intermediate sizes for a triangle count under a binary plan against a WCOJ plan on this topic's 16 M edge graph. + +
Answer + + Binary plan: joining two of the three edge relations first + materializes up to m² = (16.0e6)² = 2.56e14 candidate pairs before + the third edge filters them. + + WCOJ: the AGM bound caps the *output* at m^{ρ*} with ρ* = 3/2 for + the triangle, i.e. m^1.5 = 16.0e6 × 4 000 = 6.4e10, and a + worst-case-optimal algorithm runs in time proportional to that + bound rather than to the intermediate. + + Ratio m²/m^1.5 = √m = 4 000×. The attribution matters and is in + [reading-wcoj.md](reading-wcoj.md): the upper bound is Grohe–Marx's, + the matching lower bound is Atserias–Grohe–Marx's, and "AGM bound" + names the pair. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + Question 4's matrix expression: the factorized 2-hop count is + `Σ_a Σ_{b ∈ N(a)} deg(b)`, which is the grand sum of A² — i.e. + `1ᵀ A² 1` over the integer semiring, where A² accumulates path + counts as its values. That is the same number `hop_bench` computes, + and it is computable without ever materializing the + `Σ_a deg(a)·deg(b)` tuples a flat plan would build. + + Question 5: within one node group the neighbour-id column is sorted + *within each source's run* but restarts at every new source, so it + is piecewise-monotonic rather than monotonic. Frame-of-reference or + delta encoding per run wins; a single global delta encoding does + not, because it hits a large negative delta at every run boundary. + +
+ ## References **Papers** -- Feng, Gupta, Jin, et al. — "KÙZU Graph Database Management System" - (CIDR 2023) — the §vectorization/factorization discussion is the - part the code doesn't narrate - -**Code** -- [kuzu](https://github.com/kuzudb/kuzu) (shallow clone) — - `src/include/storage/table/csr_node_group.h` (the design comment at - :165-171 is the storage story), - `src/include/processor/operator/intersect/intersect.h` + - `intersect_build.h` (the WCOJ operator) +- Feng, Jin, Chen, Liu, Salihoğlu — "KÙZU Graph Database Management + System" (CIDR 2023). §1 has the k-regular factorization example and + the ASP-Join description; §2 *Storage and Indices* has the + double-indexed CSR, the 4 KB page size and the GClock buffer + manager; §3.1 covers factorized vectors. Note the paper predates the + pinned revision — where they disagree, the code is the authority. + +**Code** (all line numbers verified at kuzu `89f0263`) + +| File | Lines | What | +|---|---|---| +| `src/include/storage/table/csr_node_group.h` | 21-24, 30-59 | `csr_list_t`, `NodeCSRIndex` | +| `src/include/storage/table/csr_node_group.h` | 99-110 | `PackedCSRInfo` | +| `src/include/storage/table/csr_node_group.h` | 114-146 | scan state; header at 117, built at 141-142 | +| `src/include/storage/table/csr_node_group.h` | 148-160 | checkpoint state, `oldHeader`/`newHeader` | +| `src/include/storage/table/csr_node_group.h` | 162-163, 165-172 | column ids; the design comment; the class | +| `src/include/common/constants.h` | 78-79 | packed-CSR densities | +| `cmake/templates/system_config.h.in` | 25, 30-31, 47-55 | vector capacity, page size, node group / leaf region | +| `CMakeLists.txt` | 114-130 | the defaults for all three | +| `src/include/processor/operator/intersect/intersect.h` | 29, 54-60 | the operator, its kernels | +| `src/processor/operator/intersect/intersect.cpp` | 65-90, 103-118 | the sorted merge; smallest-list-first | +| `src/include/processor/operator/intersect/intersect_build.h` | 35, 45-48 | the one overridden method | +| `src/include/common/data_chunk/data_chunk_state.h` | 8-12, 25-27 | `FStateType` — flat vs unflat | +| `src/antlr4/Cypher.g4` | 917 lines | the grammar, if you want to see the surface language | diff --git a/topics/13-graph-engines/reading-ldbc-snb.md b/topics/13-graph-engines/reading-ldbc-snb.md index a57598e..e572da9 100644 --- a/topics/13-graph-engines/reading-ldbc-snb.md +++ b/topics/13-graph-engines/reading-ldbc-snb.md @@ -4,10 +4,18 @@ A benchmark only referees if it forces the hard parts: updates flowing during reads, power-law data with real correlations, audited full disclosure. LDBC SNB is that referee for graph engines. Before you skim the spec, this chapter builds what makes it one, step by step — -the three workloads, why the correlated data generator is the whole -point, the update requirement that closes the biggest cheat, the audit +the workloads, why the correlated data generator is the whole point, +the update requirement that closes the biggest cheat, the audit rules, and what M22's shootout should steal. +Every claim below is cited to a numbered section of **The LDBC Social +Network Benchmark, version 0.3.6** +([arXiv:2001.02299](https://arxiv.org/abs/2001.02299), 144 pp), which +was read to check it. Three things the previous version of this +chapter asserted — how many workloads SNB has, what SF1 weighs, and +what distribution the friend degrees follow — did not survive that +check and are corrected in place. + ## Why this matters M22 runs an LDBC-style shootout against FalkorDB. Read this now so @@ -26,6 +34,10 @@ audited disclosure, or the numbers mean nothing. ### Step 1 — a referee benchmark forces the parts vendors skip +> **In:** the three ways a vendor can make its own engine look good. +> **Out:** the list of rules a benchmark must impose to close them — +> which is also the reading order for the rest of the spec. + A **benchmark** is only as honest as the shortcuts it forbids. The three standard graph-benchmark cheats: run read-only over a frozen, pre-built structure (no update machinery to pay for); generate uniform @@ -33,75 +45,298 @@ synthetic data (no supernodes, no correlations — every plan looks fine); self-report unaudited numbers with undisclosed warmup, drivers, and scale. LDBC (the Linked Data Benchmark Council — an industry consortium, engines' vendors included) exists to close all three, the -way TPC did for relational systems. Why it matters: each following -step is one closed loophole — read the spec as a list of cheats it -outlaws. +way TPC did for relational systems. The spec's own table of contents +is the checklist: + +> "This document contains: • A detailed specification of the data +> used in the whole LDBC SNB benchmark. • A detailed specification of +> the workloads. • A detailed specification of the execution rules of +> the benchmark. • A detailed specification of the auditing rules and +> the full disclosure report's required contents." +> — Executive Summary, p.3 + +Four documents, four loopholes. Why it matters: each following step is +one closed loophole — read the spec as a list of cheats it outlaws. + +### Step 2 — two workloads, two different questions + +> **In:** the question "is this graph engine fast?" +> **Out:** the two workloads SNB actually defines, the different +> primary metric each one reports, and the boundary where SNB stops. -### Step 2 — three workloads, three different questions +**Correction.** The previous version of this chapter listed *three* +SNB workloads, with Graphalytics as one of them. The spec's abstract +is explicit that there are two: -SNB (the Social Network Benchmark) splits into workloads because "is -it fast?" is three questions with three different answers: +> "LDBC SNB consists of **two workloads** that focus on different +> functionalities: the Interactive workload (interactive transactional +> queries) and the Business Intelligence workload (analytical +> queries)." +> — Abstract, p.2 + +Graphalytics is a *sibling* LDBC benchmark, not an SNB workload: +"Initially, a graph analytics workload was also included in the +roadmap of LDBC SNB, but this was finally delegated to the +Graphalytics benchmark project [34, 35], which was adopted as an +official LDBC graph analytics benchmark" (§1.1, p.10), and §1.4 +*Related Projects* lists it alongside the Semantic Publishing +Benchmark as a separate thing. + +The distinction is not pedantry, because the two workloads report +**incomparable primary metrics**: ``` - SNB Interactive OLTP-ish: 2-hop neighborhoods, short paths, - + concurrent inserts (people, posts, likes) - SNB BI analytics: global scans/aggregations over the graph - Graphalytics pure algorithms: BFS, PageRank, WCC, CDLP, SSSP + SNB Interactive three query classes (§5, p.45): + complex read-only (IC 1 … IC 14) + short read-only (IS 1 … IS 7) + transactional inserts + primary metric: "Operations per second for a given + SF (throughput)" — §5, p.45 + + SNB BI reads (BI 1 … BI 20, §6.4) + refreshes + (inserts and deletes, §6.5) + metric: "the system is characterized by TWO metrics: + the geometric mean of the read query execution times + and the geometric mean of the time required to load + daily batches" — §6.3, p.69 ``` -Interactive is the one FalkorDB-shaped engines care about: latency per -query with updates flowing. Its complex reads (IC1–IC14) are mostly -anchored multi-hop patterns with property filters and aggregation — -i.e. exactly scan-anchor-then-expand plus M12's property columns. -(Graphalytics is topic 24's referee.) Why it matters: an engine's rank -can flip between workloads — quoting "the LDBC number" without naming -the workload is itself a benchmarketing move. +One number versus two; throughput versus geometric-mean latency plus +geometric-mean load time. There is no arithmetic that converts +between them. Interactive is the one FalkorDB-shaped engines care +about, and the spec describes its complex reads exactly as this +topic's benchmark does: + +> "This workload consists of a set of relatively complex read-only +> queries, that touch a significant amount of data – often the +> **two-step friendship neighbourhood** and associated messages –, but +> typically in close proximity to a single node. Hence, the query +> complexity is **sublinear to the dataset size**." +> — §5, p.45 + +That is `hop_bench` with a social-network schema bolted on: a single +anchor node, two expands, aggregate. The "sublinear to the dataset +size" claim is the one this topic's headline stress-tests — sublinear +in *n*, yes, but linear in the anchor's two-hop neighbourhood size, +which is why the same query costs 4.9 µs from a random node and +495 µs from a supernode. Why it matters: an engine's rank can flip +between workloads — quoting "the LDBC number" without naming the +workload is itself a benchmarketing move. -### Step 3 — correlated power-law data is the point +### Step 3 — correlated data is the point, and the degrees are not a power law + +> **In:** a target number of Persons and three simulated years. +> **Out:** a graph whose degree distribution, attribute correlations +> and temporal bursts all break independence assumptions — plus the +> exact mechanism the generator uses to produce each. The datagen produces a graph that is skewed AND correlated, because -both properties break engines in ways uniform data can't. **Power-law -degree distribution** (a few nodes have enormous degree — supernodes — -while most have little): your tail latency becomes a graph-shape -property, which is why hop_bench deliberately includes the 100 -highest-degree sources. **Correlation** (attribute values predict -structure): people named "Wang" cluster in China, friendships -correlate with universities, activity spikes around events — so -cardinality estimates that assume independence are wrong, and the -errors *compound* through multi-hop patterns even faster than in JOB -(topic 10's Leis lesson — uniform synthetic data hides planner sins — -applied to graphs). Why it matters: an engine tuned on uniform data -meets reality's supernodes and correlations in production, at p99. - -### Step 4 — updates run during reads: no frozen-CSR cheating - -Interactive's driver interleaves inserts (people, posts, likes) with -the read queries, with **dependency tracking** — an insert must be -visible to reads scheduled after it — so the engine must serve reads -over a structure that is being mutated, with correctness constraints -on visibility. This single rule is why every architecture in this -topic grew a delta mechanism (kuzu's transient buffers, FalkorDB's -Delta_Matrix, memgraph's MVCC): a read-only CSR would win every -frozen-graph benchmark and be disqualified here. Inserts are scheduled -at spec'd timestamps, not fired as fast as possible — throughput comes -from meeting a schedule, not from batching liberties. Why it matters: -this is the requirement that makes the benchmark measure a *database* -rather than a data structure. +both properties break engines in ways uniform data can't. But be +precise about which distribution does what. + +**Correction.** The previous version said the degree distribution is a +power law. The spec does not say that. It says the knows-degree +follows a Facebook-shaped empirical distribution, and the power law +it *does* specify is for something else entirely — comment timing: + +> "…the number of knows relationships of every person, which is +> guided by a degree distribution function **similar to that found in +> Facebook** [68]." +> — §3.3.2 *Graph Generation*, p.22 + +> "Comment always occur within γ days of their parent message +> following a **power-law distribution**…" +> — §3.6.5, and Figure 3.3 "The power-law used to generate comments" + +The distinction is worth keeping straight: the spec anchors the +knows-degree to a measured empirical distribution from a real social +network (reference [68]) rather than to a closed-form power law, and +does not state its tail exponent or maximum anywhere in the document. +So do not assume the generator will reproduce this topic's +6 565-degree preferential-attachment tail — if you need that shape, +measure the generated graph rather than inferring it from the spec. + +The correlations are the deeper point, and the spec names the +mechanism. Edges are drawn by **homophily**: + +> "…similar persons (with similar interests and behaviors) tend to be +> connected. This is known as the **Homophily principle** [46, 14], +> and implies the presence of a larger amount of **triangles** than +> that expected in a random network." +> — §3.3.2, p.22 + +implemented by sorting persons under a similarity function M(p) and +picking connections from the K nearest positions with a geometric +distribution over ranked distance — and split across exactly three +axes: + +> "In Datagen, **three correlated dimensions** are chosen: the first +> one depends on where the person studied and when, and the second +> correlation dimension depends on the interests of the person, and +> the third one is random (to reproduce the random noise present in +> real data)." +> — §3.3.2, p.23 + +Plus temporal bursts — "**flash mob** events" assigned a random tag, +around which activity volume spikes (§3.3.2, p.23). + +Every one of those is an independence assumption a cost model would +otherwise make. Attribute-value filters are not independent of graph +position; two-hop expansion is not degree² because triangles close; +timestamps are not uniform. Cardinality errors compound through +multi-hop patterns even faster than in JOB (topic 10's Leis lesson — +uniform synthetic data hides planner sins — applied to graphs). Why +it matters: an engine tuned on uniform data meets reality's +supernodes and correlations in production, at p99. + +### Step 4 — updates run during reads, on a schedule, with curated parameters + +> **In:** a stream of timestamped update operations and a set of +> substitution parameters per query template. +> **Out:** a query mix whose issue times are fixed by the spec rather +> than chosen by the vendor — the rule that makes this a database +> benchmark rather than a data-structure benchmark. + +Interactive's driver interleaves inserts with the read queries, and +the inserts are not fired as fast as possible: + +> "Update queries' issue times are taken from the update streams +> generated by the data generator. **These are the times where the +> actual event happened during the simulation of the social +> network.** Complex reads' times are expressed in terms of update +> operations." +> — §4.4 *Load Definition*, p.43 + +So the engine must serve reads over a structure that is being mutated, +on someone else's clock. This single rule is why every architecture in +this topic grew a delta mechanism (kuzu's transient buffers, +FalkorDB's Delta_Matrix, memgraph's MVCC): a read-only CSR would win +every frozen-graph benchmark and be disqualified here. + +There is a second mechanism the previous version of this chapter +missed entirely, and it is the cleverest thing in the spec. Because +query cost varies wildly with the parameter you plug in — the very +effect this topic measures at 101× — LDBC does not sample parameters +at random. It **curates** them, to three stated properties: + +> "**P1:** the query runtime has a bounded variance … **P2:** the +> runtime distribution is stable … **P3:** the optimal logical plan +> (optimal operator order) of the queries is the same … As a result, +> the amount of data that the query touches is roughly the same for +> every parameter binding … Such effects could arise due to the +> **data skew and correlations** between values in the generated +> dataset." +> — §4.3 *Substitution Parameters*, p.42 + +Parameter Curation runs in two stages: compute intermediate-result +sizes for every candidate binding as a side effect of generation, then +greedily select bindings with similar counts. Read that against this +topic's headline and the trade is stark: LDBC deliberately *removes* +the supernode-versus-random-node spread so that a single mean is +meaningful. The 101× gap is real and LDBC hides it on purpose — which +is exactly why your own `hop_bench` reports both lanes separately. + +The other scheduling knob is the frequency table (Table 4.1, p.43), +where "a frequency value is assigned which specifies the relation +between the number of updates performed per complex read" — i.e. the +number of updates between two instances of that query, so a *larger* +number means a *rarer* query. It is scale-dependent, and in opposite +directions: + +``` + Table 4.1 (updates per complex read): + SF1 SF1000 + IC 8 45 1 → 45× MORE frequent at SF1000 + IC 9 157 967 → 6.2× RARER at SF1000 + IC 1 26 26 → unchanged + + the IC8 : IC9 ratio in the mix + at SF1: 157/45 = 3.5 IC8s per IC9 + at SF1000: 967/1 = 967 IC8s per IC9 + the mix shifts by 967 / 3.5 = 277× +``` + +The mix at SF1000 is a different workload from the mix at SF1, by +design — expensive queries are throttled so "faster query types" are +not made "purposeless" (§4.4, p.43). Why it matters: this is the +requirement that makes the benchmark measure a *database* rather than +a data structure, and the reason two SFs are two experiments. ### Step 5 — audit, disclosure, and pinned scale factors -An official LDBC result requires an **audit** — an independent -reviewer reruns the benchmark under the published rules — plus full -disclosure of drivers, warmup, configuration; results are reproducible -or they're not results. **Scale factors** (SF1 … SF30K — dataset sizes -with a defined generator seed) pin the dataset exactly, so comparisons -must name their SF: an SF1 (~3 GB) number and an SF1000 number are -different experiments, not two scores on one leaderboard. Why it -matters: this is the machinery that separates a referee from a blog -post — and the checklist to apply to any vendor claim you read. +> **In:** a claimed result. +> **Out:** the specific checklist that makes it citable — dataset +> size, run length, warm-up, on-time percentage, and who verified it. + +An official LDBC result requires an **audit** by a trained, certified +auditor (§7.2.1) plus a Full Disclosure Report (§7.4.8) carrying the +system description and pricing, data generation and loading, driver +details, performance metrics, validation results, ACID compliance, and +a supplementary package with a README and the database configuration +files — "to ensure reproducibility of the audited results". + +**Scale factors** pin the dataset, and the definition is not a node +count: + +> "For both workloads, **the SF1 data set is 1 GiB**, the SF100 is +> 100 GiB, and the SF10000 data set is 10000 GiB (not 10 TiB)." +> — §3.4.1, p.25 + +**Correction.** The previous version wrote "an SF1 (~3 GB) number". +SF1 is 1 GiB of serialized CSV, not 3 GB. And the *composition* of +that gibibyte differs by workload — Interactive counts 90% initial +data plus the 10% update streams with the `csv-singular-merged-fk` +serializer; BI counts a 97% initial snapshot plus refresh operations +with `csv-composite-merged-fk` (§3.4.1). Same SF number, different +bytes on disk. The proposed SFs are 1, 3, 10, 30, 100, 300, 1000, +3000, 10000, 30000, plus 0.003, 0.1 and 0.3 for validation; all SFs +cover three years starting in 2010, and scaling the SF scales the +number of Persons. + +The run rules are equally specific, and they interlock: + +``` + §7.4.1.1 validation run on SF10 + §7.4.1.1 audited benchmark runs on SF30 or larger + §7.4.7.1 valid run ≥ 2 hours wall clock + §7.4.7.1 95% on-time requirement + actual_start_time − scheduled_start_time < 1 second + for 95% of issued queries + §7.4.7.2 warm-up ≥ 30 min, then a 2-hour measurement window +``` + +The SF30 floor is not arbitrary; the spec derives it, and the +derivation is worth reproducing because it is the whole benchmark in +one calculation: + +``` + §7.4.7.2: "The SNB Datagen produces 3 years worth data of which 10% + is used for updates, i.e. approximately 3×365×0.1 = 109.5 days + = 2628 hours." + + Time Compression Ratio (TCR) replays those updates faster: + playback wall clock = 2628 h × TCR + spec floor = TCR ≥ 0.001 + → shortest possible run = 2628 × 0.001 = 2.628 hours + + required: 30 min warm-up + 2 h measurement = 2.5 hours + 2.628 ≥ 2.5 ✓ — with 7.7 minutes to spare +``` + +"System that can achieve a better compression (i.e. lower TCR value) +on a given scale factor should use larger SFs for their benchmark +runs – otherwise their total runs will be less than 2.5 hours, making +them unsuitable for auditing" (§7.4.7.2). A fast engine is *forced* +onto a bigger dataset. Why it matters: this is the machinery that +separates a referee from a blog post — and the checklist to apply to +any vendor claim you read. ### Step 6 — what to steal for M22 +> **In:** the five preceding steps' rules. +> **Out:** the two or three of them that are worth the implementation +> cost for a single-developer shootout, and the ones to skip. + M22 shouldn't implement all of SNB — it should steal the load-bearing ideas (record decisions in notes.md): @@ -113,24 +348,66 @@ ideas (record decisions in notes.md): - report: throughput at bounded p99, not just mean — the supernode tail is the honest number -Why it matters: the shootout's credibility comes from adopting the -referee's *constraints* (updates flowing, skewed data, tail -reporting), not its full query set. +The one place to deliberately *depart* from LDBC is Parameter +Curation. SNB curates the skew out (Step 4, P1–P3) so a mean is +meaningful; this topic's whole finding is what lives in the skew. +Keep the two lanes — random sources and highest-degree sources — +reported separately, and you get both the referee's comparability and +the number LDBC's design suppresses. + +Scale calibration, from the spec's own entity counts (Appendix B.1, +Table B.1 — real numbers, unlike the still-TODO Table 3.12): + +``` + SF persons person_knows_person rows rows / person + 1 11 000 452 622 41.1 + 10 73 000 4 654 416 63.8 + 30 184 000 14 212 356 77.2 + 100 499 000 46 598 276 93.4 + 1000 3 600 000 447 163 916 124.2 + + this topic's graph: 1 000 000 nodes, 16.0e6 directed edges → 16.0 + + → in edge count the topic graph sits just above SF30; in NODE count + it is 5.4× SF30 and 0.28× SF1000. It is a sparser, wider graph + than any SNB scale factor — worth stating explicitly before + claiming any result transfers. +``` + +Note the drift in the last column: SNB's density *rises* with scale +(41 → 124 rows per person, a 3.0× increase from SF1 to SF1000) +because the simulated period is fixed at three years while the +population grows. Why it matters: the shootout's credibility comes +from adopting the referee's *constraints* (updates flowing, skewed +data, tail reporting), not its full query set — and from saying +plainly where your graph is not theirs. ## How to read the spec (with the concepts in hand) -1. **Data generation section** — read properly; it's Step 3 +| Step | Section | Pages | +|---|---|---| +| 1 | Executive Summary; §1.1 Scope; §1.4 Related Projects | 3, 10 | +| 2 | Abstract; §5 opening; §5.1–5.3; §6.3 Target metric | 2, 45, 69 | +| 3 | §3.3.2 Graph Generation (homophily, three correlation dimensions, flash mobs) | 22-23 | +| 4 | §4.3 Substitution Parameters (P1/P2/P3, Parameter Curation); §4.4 Load Definition + Table 4.1 | 42-44 | +| 5 | §3.4.1 scale factors; §7.2.1 auditors; §7.4.1.1 SF10/SF30; §7.4.7 timing; §7.4.8 FDR | 25, 94, 101, 106 | +| 6 | Appendix B.1 Table B.1 (per-SF entity counts) | 128+ | + +1. **Data generation section (§3.3)** — read properly; it's Step 3 operationalized (which correlations exist, how degrees are drawn). This is the part most readers skip and the part that matters most. -2. **Interactive workload definition** — skim all 14 complex reads, - then read 2–3 closely (IC5-ish friends-of-friends is question 2 - below); note the anchor + expand + filter shape. -3. **Driver / dependency tracking** — read enough to answer why - inserts are scheduled with timed dependencies (Step 4; question 1). -4. **Audit rules and SF definitions** — skim, but internalize the - checklist for reading vendor claims (Step 5). -5. The SIGMOD 2015 paper is the narrative version: read its - correlated-generation and choke-point sections; skim the rest. +2. **Interactive workload definition (§5)** — skim all 14 complex + reads, then read 2–3 closely (IC5-ish friends-of-friends is + question 2 below); note the anchor + expand + filter shape. +3. **Driver / load definition (§4.3–4.4)** — read enough to answer why + inserts are scheduled with timed dependencies (Step 4; question 1), + and read Parameter Curation properly; it is the subtlest idea in + the document. +4. **Audit rules and SF definitions (§3.4.1, §7.4)** — skim, but + internalize the checklist for reading vendor claims (Step 5). +5. The SIGMOD 2015 paper is the narrative version (the spec cites it + at §5, p.45 as reference [24]): read its correlated-generation and + choke-point sections; skim the rest. ## Questions (answer in notes.md) @@ -148,25 +425,135 @@ reporting), not its full query set. ## Done when -- [ ] You can name the three workloads and the different question each one asks. -- [ ] You can explain why correlated power-law data is the point rather than a realism garnish — and connect it to the 101x supernode gap this topic measures. -- [ ] You can say what running updates during reads prevents a vendor from doing. -- [ ] You can state what a pinned scale factor and an audit rule are for. +Answer each before unfolding it. + +- [ ] You can name the workloads SNB actually defines and the different question each one asks — including which metric each reports. + +
Answer + + **Two**: Interactive and Business Intelligence (Abstract, p.2). + Graphalytics is a separate LDBC benchmark, delegated out of the SNB + roadmap (§1.1, p.10; §1.4). + + Interactive: complex reads (IC 1–14), short reads (IS 1–7) and + transactional inserts; primary metric "Operations per second for a + given SF (throughput)" (§5, p.45). + + BI: 20 read queries plus refresh operations (inserts and deletes); + metric is a *pair* — the geometric mean of read query execution + times and the geometric mean of daily-batch load time (§6.3, p.69). + Nothing converts one metric into the other. + +
+ +- [ ] You can explain why correlated data is the point rather than a realism garnish — and connect it to the 101x supernode gap this topic measures. + +
Answer + + §3.3.2 gives the mechanism: homophily (similar persons connect, + producing more triangles than a random graph) implemented over + three correlation dimensions — where and when the person studied, + their interests, and random noise — plus flash-mob temporal bursts. + Each one breaks an independence assumption that a cost model makes + for free, and multi-hop patterns compound the error. + + The degrees are *Facebook-like*, not a power law (§3.3.2, p.22); + the spec's power law is for comment delay (Figure 3.3). So SNB has + supernodes but a bounded tail, where this topic's preferential + attachment generator produces a 6 565-degree node on 1 M nodes and + the 101× two-hop gap that follows from it. + +
+ +- [ ] You can say what running updates during reads prevents a vendor from doing, and what Parameter Curation deliberately removes. + +
Answer + + Updates prevent shipping a frozen read-only CSR: §4.4 fixes the + insert issue times to the simulated event times, and complex read + times are expressed in updates, so the engine must serve reads over + a mutating structure on the spec's clock rather than batching at + its convenience. Every engine in this topic answers with a delta + mechanism. + + Parameter Curation (§4.3) removes the parameter-dependent variance: + P1 bounded runtime variance, P2 stable runtime distribution across + streams, P3 same optimal logical plan for every binding — chosen by + matching intermediate-result sizes. It is the 101× effect, + deliberately engineered out so that a mean is a meaningful summary. + Your own bench keeps the two lanes apart instead. + +
+ +- [ ] You can state what a pinned scale factor and an audit rule are for, and reproduce the spec's own derivation of the 2.5-hour floor. + +
Answer + + SF pins the dataset by *serialized size*: SF1 = 1 GiB, SF100 = + 100 GiB, SF10000 = 10000 GiB (§3.4.1, p.25) — with different + composition per workload (Interactive 90% initial + 10% streams, + `csv-singular-merged-fk`; BI 97% snapshot + refreshes, + `csv-composite-merged-fk`). Comparisons must name the SF and the + workload. + + Audit rules: validation on SF10, audited runs on SF30 or larger + (§7.4.1.1); ≥ 2 h wall clock with a 95% on-time requirement + (`actual_start_time − scheduled_start_time < 1 s`) (§7.4.7.1); + ≥ 30 min warm-up then a 2 h measurement window (§7.4.7.2). + + The derivation: 3 years × 365 × 10% = 109.5 days = 2628 hours of + updates; TCR ≥ 0.001, so the shortest legal replay is 2.628 hours, + which just covers the 0.5 + 2 = 2.5 hours required. A faster engine + must move to a bigger SF or run out of updates. + +
+ - [ ] You wrote answers to all questions in notes.md, including what you intend to steal for M22. +
Answer + + Question 5 needs real counts, and Appendix B.1 Table B.1 has them: + SF10 is 73 000 persons and 4 654 416 `person_knows_person` rows; + SF30 is 184 000 and 14 212 356; SF100 is 499 000 and 46 598 276. + Multiply by your per-edge estimates — memgraph's are computable + from `sizeof(Vertex) == 80` plus a 24-byte `EdgeTriple` per + direction, a CSR's are 8 bytes per edge plus 8 per node, a + Delta_Matrix's are GraphBLAS hypersparse (index + value per entry, + times the number of live matrices). + + For M22, steal: the operation mix with dependency tracking, two or + three representative queries, and p99 reporting. Skip: Parameter + Curation — deliberately, because the skew it removes is this + topic's actual finding. + +
+ ## References **Papers** +- **The LDBC Social Network Benchmark, version 0.3.6** + ([arXiv:2001.02299](https://arxiv.org/abs/2001.02299)) — the + authority for everything above. §3.3.2 data generation; §3.4.1 + scale factors; §4.3 Parameter Curation; §4.4 load definition and + Table 4.1; §5 Interactive; §6.3 BI target metric; §7.4 auditing; + Appendix B.1 Table B.1 per-SF entity counts - Erling et al. — "The LDBC Social Network Benchmark: Interactive - Workload" (SIGMOD 2015) -- LDBC SNB specification - ([ldbcouncil.org/benchmarks/snb](https://ldbcouncil.org/benchmarks/snb)) - — skim the query set, read the data-generation section -- Iosup et al. — "LDBC Graphalytics" (VLDB 2016) — topic 24's referee; - noted here for the boundary + Workload" (SIGMOD 2015) — the narrative version; the spec cites it + at §5, p.45 as its detailed description of the Interactive workload +- Iosup et al. — "LDBC Graphalytics" (VLDB 2016) — a *separate* LDBC + benchmark (§1.4), topic 24's referee; noted here for the boundary **Code** - [ldbc_snb_datagen_spark](https://github.com/ldbc/ldbc_snb_datagen_spark) and the audited implementations under [github.com/ldbc](https://github.com/ldbc) — the driver's - dependency-tracking is the part worth reading for M22 + scheduling of update streams against `scheduled_start_time` (§7.4.7.1) + is the part worth reading for M22 + +**Cross-references in this topic** +- [reading-kuzu.md](reading-kuzu.md), [reading-memgraph-storage.md](reading-memgraph-storage.md), + [reading-graphblas-internals.md](reading-graphblas-internals.md) — + the three delta mechanisms Step 4's update rule forces +- [notes.md](notes.md) — the 4.9 µs / + 495 µs baseline that Step 2's "sublinear to the dataset size" claim + should be read against diff --git a/topics/13-graph-engines/reading-memgraph-storage.md b/topics/13-graph-engines/reading-memgraph-storage.md index 5249523..5d1c59a 100644 --- a/topics/13-graph-engines/reading-memgraph-storage.md +++ b/topics/13-graph-engines/reading-memgraph-storage.md @@ -10,6 +10,15 @@ object-per-vertex model, the struct that holds everything, edges stored twice, undo-delta MVCC, and the ledger of what all this buys and costs. +Every anchor below is **memgraph pinned at `8f87f6a`** +([`resources/codebases.md`](../../resources/codebases.md)). This is +the one chapter in the topic where the source hands you a *hard* +number rather than an estimate — `vertex.hpp:73` is a `static_assert` +on `sizeof(Vertex)` — so Step 2 spends its arithmetic reconstructing +that number field by field. Doing so also breaks a claim the previous +version of this chapter made about `small_vector`; Step 3 shows the +line that breaks it. + ## The problem in one sentence Serve many concurrent transactions mutating the graph — edge inserts, @@ -21,82 +30,245 @@ in pointer-chasing bandwidth. ### Step 1 — no pages, no CSR: the graph is a heap of vertex objects +> **In:** a node id (a `Gid`). +> **Out:** a pointer to a heap object holding that node's entire +> state — with no page, no slot, and no global adjacency structure in +> between. + memgraph represents each node as a plain heap-allocated C++ object holding *everything* about that node — labels, both edge lists, properties, a lock, a version-chain pointer — and the "table" is a concurrent skip list (topic 9's lazy-locking accessor/GC design) keyed -by Gid (the node's global id). There is no page layout to respect, no -global read-optimized structure to rebuild on write: mutating node -42's state touches node 42's object, full stop. Why it matters: this -is the maximally write-friendly end of the topic's spectrum — every -other engine in this topic maintains some shared read-optimized -structure and therefore needs delta machinery; memgraph's "delta -machinery" is just... objects, plus MVCC (Step 4). +by `Gid`, the node's global id (`id_types.hpp:56` defines `Gid` over +`uint64_t`). + +There is no page layout to respect and no global read-optimized +structure to rebuild on write: mutating node 42's state touches node +42's object, full stop. Contrast the other two engines in this topic — +neo4j must place the record in a page and thread it into two chains; +FalkorDB must route the write into `delta_plus` and eventually rebuild +a matrix. Why it matters: this is the maximally write-friendly end of +the topic's spectrum — every other engine here maintains some shared +read-optimized structure and therefore needs delta machinery; +memgraph's "delta machinery" is just... objects, plus MVCC (Step 4). ### Step 2 — the Vertex struct: the whole per-node state in one place -The entire chapter is one struct — every field is a design decision: +> **In:** the 83-line header `src/storage/v2/vertex.hpp`. +> **Out:** the seven fields of `Vertex`, their individual sizes, and a +> reconstruction of the 80 bytes the file asserts they total. + +The entire chapter is one struct, and every field is a design +decision: ```cpp -struct Vertex { - const Gid gid; - utils::small_vector labels; // :41 inline until it spills - Edges in_edges; // :43 small_vector of triples - Edges out_edges; // :44 - PropertyStore properties; // :46 packed blob, not columns - mutable utils::RWSpinLock lock; // :47 per-vertex latch - utils::PointerPack delta_; // :66 MVCC chain head + 2 flag bits -}; -``` - -Notes on the choices: `small_vector` stores its first few elements -*inline* in the struct (no heap allocation) and spills to the heap -only past that — a big win because power-law degree distributions mean -MOST nodes have few labels/edges. Properties are a packed per-node -blob, not columns — great for "load this node's properties", useless -for topic 12-style columnar filters. And `PointerPack` -smuggles two flag bits (`kDeletedBit`, `kNonSeqDeltasBit`, `:62-63`) -into the alignment bits of the delta pointer — the bit-packing ledger -again. Why it matters: one struct = one cache-line-friendly home for -the OLTP hot path; every access pattern beyond single-node suffers -for it. +// src/storage/v2/vertex.hpp + 32 struct Vertex { + // ... 33-38: elided — the constructor and its MG_ASSERT, see Step 4 ... + 39 const Gid gid; + 40 + 41 utils::small_vector> labels; + 42 + 43 Edges in_edges; + 44 Edges out_edges; + 45 + 46 PropertyStore properties; + 47 mutable utils::RWSpinLock lock; + // ... 48-60: elided — delta()/SetDelta()/deleted() accessors ... + 61 private: + 62 static constexpr int kDeletedBit = 0; + 63 static constexpr int kNonSeqDeltasBit = 1; + 64 + 65 utils::PointerPack delta_; + 66 }; +``` + +**Correction.** The previous version of this chapter placed `delta_` +at `:66`. Line 66 is the closing brace; `delta_` is at **`:65`**. + +Ten lines further down the file states its own size, and this is the +one number in this topic that cannot drift without the build failing: + +```cpp +// src/storage/v2/vertex.hpp + 72 static_assert(alignof(Vertex) >= 8, "The Vertex should be aligned to at least 8!"); + 73 static_assert(sizeof(Vertex) == 80, "If this changes documentation needs changing"); +``` + +Reconstruct the 80. Each size below is read from its own header, not +guessed: + +| field | type | size | where the size comes from | +|---|---|---|---| +| `gid` | `Gid` | 8 | `id_types.hpp:56` — `Gid` wraps `uint64_t` | +| `labels` | `small_vector` | 16 | `small_vector.hpp:609-610` — `static_assert(sizeof(small_vector) == 16)` | +| `in_edges` | `Edges` | 16 | same, `small_vector` is always 16 | +| `out_edges` | `Edges` | 16 | same | +| `properties` | `PropertyStore` | 12 | `property_store.hpp:193` — `std::array` = 4 + 8 | +| `lock` | `RWSpinLock` | 4 | `rw_spin_lock.hpp:113, 122` — one `uint32_t lock_status_` | +| `delta_` | `PointerPack` | 8 | one pointer, two flag bits stolen from its alignment | + +``` + 8 + 16 + 16 + 16 + 12 + 4 + 8 = 80 ✓ and 80 % 8 == 0, so alignof ≥ 8 holds +``` + +The `small_vector` is always 16 bytes regardless of element type +because of its layout: + +```cpp +// src/utils/small_vector.hpp + 599 uint32_t size_{}; // max 4 billion + 600 uint32_t capacity_{kSmallCapacity}; // max 4 billion + 601 + 602 union { + 603 value_type *buffer_; + 604 uninitialised_storage + 605 small_buffer_[kSmallCapacity ? kSmallCapacity : 1]; + 606 }; + 607 }; + // ... 608: elided ... + 609 static_assert(sizeof(small_vector) == 16); +``` + +4 + 4 + 8 = 16, where the 8 is either a heap pointer *or* the inline +small buffer — never both. Note what that means: **the inline +capacity can never exceed 8 bytes' worth of elements**, because it +shares a union with a pointer. + +Other choices worth naming while you're in the struct. +`PropertyStore` is a packed per-node blob (a 4-byte size plus an +8-byte pointer, `property_store.hpp:193`), not columns — great for +"load this node's properties", useless for topic 12-style columnar +filters. `PointerPack` smuggles two flag bits — `kDeletedBit` +and `kNonSeqDeltasBit` at `:62-63`, read through `deleted()` at `:53` +and `has_uncommitted_non_sequential_deltas()` at `:57` — into the +alignment bits of the delta pointer. That is the bit-packing ledger +again, the same move neo4j makes with its header byte +([reading-neo4j-record-store.md](reading-neo4j-record-store.md) +Step 3). + +Why it matters: 80 bytes is one and a quarter 64-byte cache lines, so +one struct is very nearly one cache-line-friendly home for the OLTP +hot path — and every access pattern beyond single-node suffers for +what is *not* in those 80 bytes, namely any of the actual edges. ### Step 3 — every edge is stored twice: per-endpoint vectors -Each edge appears in BOTH endpoints' vectors — `Edges` is a -`small_vector` of `(EdgeTypeId, Vertex*, EdgeRef)` triples -(`vertex.hpp:29`), so both "who do I point at?" (out_edges) and "who -points at me?" (in_edges) are answered locally, without a global -reverse index. Compare neo4j's two chains threading one shared record: -memgraph instead duplicates the entry but makes each copy *contiguous -per vertex*. Expand of one node = walk one contiguous vector — better -locality than neo4j's scattered records. The catch: each entry is a -16-byte triple whose `Vertex*` target points anywhere in the heap, so -the moment you *follow* the neighbors (2-hop, frontier), you're back -to a cache miss per hop: +> **In:** one edge (type, source, target). +> **Out:** two entries — one in the source's `out_edges`, one in the +> target's `in_edges` — and a per-entry byte cost that decides whether +> the vector is inline or on the heap. + +Each edge appears in BOTH endpoints' vectors, so "who do I point at?" +and "who points at me?" are answered locally, without a global reverse +index: + +```cpp +// src/storage/v2/vertex.hpp + 29 using EdgeTriple = std::tuple; + 30 using Edges = utils::small_vector>; +``` + +**Correction.** The previous version of this chapter called this "a +16-byte triple" (three times: in this step, in Step 5's ledger, and in +question 4). Size the three members from their own headers: + +| member | type | size | source | +|---|---|---|---| +| `EdgeTypeId` | `uint32_t` wrapper | 4 | `id_types.hpp:59` | +| `Vertex *` | pointer | 8 | — | +| `EdgeRef` | `union { Gid gid; Edge *ptr; }` | 8 | `edge_ref.hpp:33-36`; `Gid` is `uint64_t` | + +``` + 4 + 8 + 8 = 20, rounded up to the 8-byte alignment of its widest member + sizeof(EdgeTriple) = 24 bytes, not 16 +``` + +**Correction, and it is the bigger one.** The previous version said +`small_vector` "stores its first few elements inline … a big win +because power-law degree distributions mean MOST nodes have few +labels/edges." That is true of `labels` and **false of edges**, and +one line says why: + +```cpp +// src/utils/small_vector.hpp + 583 // kSmallCapacity can be 0; in that case we disable the small buffer + 584 constexpr static std::uint32_t kSmallCapacity = sizeof(value_type *) / sizeof(value_type); + // ... 585-592: elided ... + 593 constexpr static bool usingSmallBuffer(uint32_t capacity) { + 594 return kSmallCapacity != 0 && capacity == kSmallCapacity; + 595 } +``` + +The inline capacity is a pointer's worth of elements, integer-divided: + +``` + labels: value_type = LabelId (4 B) → kSmallCapacity = 8 / 4 = 2 + → 2 labels stored inline, no allocation + edges: value_type = EdgeTriple (24 B) → kSmallCapacity = 8 / 24 = 0 + → usingSmallBuffer() is constant false; the small buffer is + disabled entirely, and EVERY non-empty edge vector is a + separate heap allocation +``` + +So on the bench graph's degree distribution (p50 degree 11, max 6 565, +[notes.md](notes.md)) the p50 node's 11 out-edges are *not* inline — +they are one heap block of 11 × 24 = 264 bytes reached through +`buffer_`. Expanding one vertex is therefore two dependent loads +(vertex → buffer) and then a contiguous walk, not one. + +Compare neo4j's two chains threading one shared record: memgraph +instead duplicates the entry but makes each copy *contiguous per +vertex*. That is still a real win — expand of one node walks one +contiguous array instead of neo4j's scattered chain — but the +`Vertex*` in each triple points anywhere in the heap, so the moment +you *follow* the neighbours you are back to a cache miss per hop: + +``` + expand(A): vertex → heap buffer → walk 24 B triples — contiguous + expand 10K frontier: 10K vertex objects (scattered) + + 10K heap buffers (scattered) + + Vertex* targets that point anywhere +``` + +Price the memory against CSR on this topic's graph (1 M nodes, +16.0 M directed edges, [notes.md](notes.md)): ``` - expand(A): walk A's vector — contiguous, prefetchable - expand 10K frontier: 10K scattered vector headers - + Vertex* targets that point anywhere + memgraph: each edge stored twice, 24 B per entry + 2 × 16.0e6 × 24 B = 768 MB of triples + + 1e6 × 80 B of Vertex structs = 80 MB + + one malloc header per non-empty edge vector + CSR: offsets (1e6+1) × 4 B + targets 16e6 × 4 B = 68 MB + (reading-neo4j-record-store.md Step 1's comparison basis) + ratio ≈ 848 MB / 68 MB = 12.5× ``` Why it matters: "contiguous per vertex" is enough for OLTP-shaped 1-hop reads, and structurally incapable of the streaming that CSR gives frontier-scale traversals — this one step is most of the -memgraph-vs-kuzu/FalkorDB performance story. +memgraph-vs-kuzu/FalkorDB performance story, and the 12.5× memory +ratio is the other half of it. ### Step 4 — MVCC by undo deltas (topic 8 cashed in) +> **In:** a vertex object holding the *newest* state, and a reader +> with an older snapshot timestamp. +> **Out:** the view that reader is entitled to, reconstructed by +> undoing deltas backwards — at a cost proportional to how stale the +> reader is. + memgraph keeps the NEWEST version of each vertex in place and hangs a -chain of **undo deltas** off it — each delta says how to reverse one -change (N2O ordering: newest-to-oldest, topic 8) — so a reader with an -older snapshot walks the chain backwards, undoing changes until the -state is old enough for its timestamp: +chain of **undo deltas** off it. Each delta says how to reverse one +change (**N2O** = newest-to-oldest ordering, topic 8), so a reader +with an older snapshot walks the chain backwards, undoing changes +until the state is old enough for its timestamp: ```rust -// N2O read: start from the newest (in-place) state and UNDO backwards -// until the chain is old enough for this reader's snapshot +// ILLUSTRATION — not memgraph source. The chain head is +// src/storage/v2/vertex.hpp:65 (`delta_`), read via `delta()` at :49; +// the delta actions are src/storage/v2/delta.hpp. This is the shape of +// the N2O walk those declarations imply. fn read_vertex(v: &Vertex, snapshot_ts: u64) -> VertexView { let mut view = v.current_state(); // newest version, in place let mut d = v.delta_head(); // PointerPack: flags in low bits @@ -109,58 +281,114 @@ fn read_vertex(v: &Vertex, snapshot_ts: u64) -> VertexView { } ``` -The constructor even asserts a new vertex starts with a -`DELETE_OBJECT` delta (`vertex.hpp:33-37`) — a fresh vertex's undo is -"didn't exist." Old deltas are GC'd once no snapshot needs them. -Combined with the per-vertex `RWSpinLock`, writers never block readers -— exactly topic 8's design, at vertex granularity. Why it matters: -N2O bets that most readers are fresh (0 undo hops) — the right bet for -OLTP — and delta chains per *object* mean a hot vertex's history is -one locality-friendly chain rather than scattered version rows. +The constructor asserts that a new vertex starts life with a +delete-shaped delta — a fresh vertex's undo is "didn't exist": + +```cpp +// src/storage/v2/vertex.hpp + 33 Vertex(Gid gid, Delta *delta) : gid(gid), delta_(delta) { + 34 MG_ASSERT(delta == nullptr || delta->action == Delta::Action::DELETE_OBJECT || + 35 delta->action == Delta::Action::DELETE_DESERIALIZED_OBJECT, + 36 "Vertex must be created with an initial DELETE_OBJECT delta!"); + 37 } +``` + +Note there are **two** accepted actions, not one: `DELETE_OBJECT` and +`DELETE_DESERIALIZED_OBJECT` — the second is the on-recovery path, +where the vertex was reconstructed from a snapshot rather than +created by a transaction. The assertion message only names the first, +which is exactly the kind of drift that makes reading the condition +rather than the message worth the habit. + +Work N2O's bet on numbers. Suppose a hot vertex takes 1 000 updates +per second and the median reader's snapshot is 1 ms old: + +``` + median reader: 1 000 updates/s × 0.001 s = 1 delta to undo + a 100 ms-stale analytics reader: 1 000 × 0.1 = 100 deltas to undo + a reader from 10 s ago: 1 000 × 10 = 10 000 deltas +``` + +N2O makes the fresh reader free and the stale reader linear in +staleness — the correct trade for OLTP, and the wrong one for long +analytical scans. Combined with the per-vertex `RWSpinLock` +(`vertex.hpp:47`, which is writer-friendly per its own doc comment at +`rw_spin_lock.hpp:24-26`), writers never block readers. Old deltas +are GC'd once no snapshot needs them — the topic-9 accessor machinery, +reused. Why it matters: delta chains per *object* mean a hot vertex's +history is one locality-friendly chain rather than version rows +scattered across a heap. ### Step 5 — the ledger: what this architecture buys and costs -Put the four steps against the CSR/matrix side of the topic: +> **In:** the four steps above. +> **Out:** a row-by-row comparison against the CSR/matrix side of the +> topic, with every row traceable to a field or a line. ``` - memgraph CSR/matrix engines - add edge push to 2 vectors delta overlay + merge - delete edge swap-remove tombstone (DM) - expand 1 node walk contiguous vec slice (same-ish!) - expand frontier pointer soup SpMV, streams - memory ptr-heavy, per-obj offsets+targets, dense - durability snapshot + WAL checkpoint matrices + memgraph CSR/matrix engines + add edge push to 2 vectors (24 B each) delta overlay + merge + delete edge swap-remove from 2 vectors tombstone (DM) + expand 1 node vertex → buffer → contig. walk slice (one indirection fewer) + expand frontier pointer soup SpMV, streams + memory / edge 2 × 24 B = 48 B 4 B (CSR targets) + per-node overhead 80 B Vertex + malloc headers 4 B offsets entry + durability snapshot + WAL checkpoint matrices ``` The verdict the table encodes: single-object operations are memgraph's -home turf (no overlay, no merge, no rebuild — just object mutation -under a spinlock), and per-vertex expand is genuinely competitive -because the vector is contiguous. The losses are at frontier scale — -10K frontier nodes = 10K scattered headers, no batch-level structure -to stream — and in memory (16-byte triples with pointers vs 8-byte -offsets; per-object allocator overhead on top). Why it matters: this -is the cleanest existence proof in the topic that the mutation-vs-scan -tension is architectural, not an implementation detail — memgraph -simply picked the other end from FalkorDB. +home turf — no overlay, no merge, no rebuild, just object mutation +under a spinlock — and per-vertex expand is genuinely competitive +because the buffer is contiguous. The losses are at frontier scale +(10 K frontier nodes = 10 K scattered vertex objects *plus* 10 K +scattered heap buffers, with no batch-level structure to stream) and +in memory, where the ratio is the 12× computed in Step 3. + +Tie it to the topic's headline. The 101× supernode penalty +([FINDINGS.md](../../FINDINGS.md) row 13) was measured on an +adjacency-list oracle whose per-node neighbours are contiguous — that +is memgraph's shape, not neo4j's. So the headline is roughly the +*best case* for this architecture: even with contiguous per-vertex +edges, a two-hop from supernodes costs 495 378 ns against 4 914 ns +from random nodes. Normalised per query that is 78 907 distinct nodes +reached against 1022 — 77× more work — at 6.28 against 4.81 ns per node +reached, so 1.31× of the gap is not explained by volume. Nothing in +Step 1–4's design attacks that residual; only a set-structured +representation (CSR with a visited bitmap, or a masked SpMV) can. + +Why it matters: this is the cleanest existence proof in the topic +that the mutation-vs-scan tension is architectural, not an +implementation detail — memgraph simply picked the other end from +FalkorDB. ## Where each step lives in the code -One file carries the whole chapter — -`src/storage/v2/vertex.hpp` in the [memgraph](https://github.com/memgraph/memgraph) -clone from topic 9: - -- **Step 2** — the `Vertex` struct at `vertex.hpp:32`: labels `:41`, - `in_edges`/`out_edges` `:43-44`, `properties` `:46`, `lock` `:47`, - `delta_` `:66`; the smuggled flag bits at `:62-63`. -- **Step 3** — `vertex.hpp:29`: - `Edges = small_vector>`. -- **Step 4** — the `DELETE_OBJECT` constructor assertion at - `vertex.hpp:33-37`; the delta types and GC are the topic-9 machinery - reused (skip-list vertex store, accessor-based GC). - -Read order: the struct top to bottom, pausing at each field to name -the decision it encodes — then re-read Step 5's table and check every -row against a field. +[memgraph](https://github.com/memgraph/memgraph) pinned at `8f87f6a` +(the clone from topic 9). `src/storage/v2/vertex.hpp` is 83 lines and +carries most of the chapter. + +| Step | Anchor | What is there | +|---|---|---| +| 1 | `src/storage/v2/id_types.hpp:56-59` | `Gid`=uint64_t, `LabelId`/`PropertyId`/`EdgeTypeId`=uint32_t | +| 2 | `src/storage/v2/vertex.hpp:32` | `struct Vertex` opens | +| 2 | `src/storage/v2/vertex.hpp:39, 41, 43-44, 46-47, 65` | the seven fields (`delta_` is `:65`, not `:66`) | +| 2 | `src/storage/v2/vertex.hpp:53, 57, 62-63` | the two smuggled flag bits and their accessors | +| 2 | `src/storage/v2/vertex.hpp:72-73` | `alignof(Vertex) >= 8`, `sizeof(Vertex) == 80` | +| 2 | `src/storage/v2/property_store.hpp:193` | `PropertyStore` is a 12-byte packed blob handle | +| 2 | `src/utils/rw_spin_lock.hpp:19-26, 113, 122` | the lock's doc, its `uint32_t` status, the member | +| 2, 3 | `src/utils/small_vector.hpp:599-610` | the 16-byte layout and its `static_assert` | +| 3 | `src/storage/v2/vertex.hpp:29-30` | `EdgeTriple` and `Edges` | +| 3 | `src/storage/v2/edge_ref.hpp:22, 33-36` | `EdgeRef` is a `Gid`/`Edge*` union — 8 B | +| 3 | `src/utils/small_vector.hpp:583-584, 593-595` | `kSmallCapacity` = 8/sizeof(T); 0 disables the small buffer | +| 3 | `src/storage/v2/vertex.hpp:68-70` | `kEdgeTypeIdPos`/`kVertexPos`/`kEdgeRefPos` — how the tuple is unpacked | +| 4 | `src/storage/v2/vertex.hpp:33-37` | the constructor's two accepted initial delta actions | +| 4 | `src/storage/v2/vertex.hpp:49, 51` | `delta()` / `SetDelta()` — the chain head accessors | + +Read order: `vertex.hpp` top to bottom, pausing at each field to name +the decision it encodes; then jump to `small_vector.hpp:583-584` and +work out `kSmallCapacity` for `LabelId` and for `EdgeTriple` before +reading on; then re-read Step 5's table and check every row against a +field. ## Questions (answer in notes.md) @@ -178,14 +406,116 @@ row against a field. 5. Sketch what an analytics query (PageRank) costs on this layout vs a matrix. Where does the memory bus time go? +> Question 2 and question 4 both encode the mistake Step 3 corrects. +> Answer 2 for `labels` (where the premise holds, `kSmallCapacity` = 2) +> and then say why it fails for edges; answer 4 with the real +> `sizeof(EdgeTriple)` = 24, not 16. + ## Done when -- [ ] You can draw the Vertex struct and say what per-node state lives in one place. -- [ ] You can explain why every edge is stored twice and which query shape breaks if it is not. +Answer each before unfolding it. + +- [ ] You can draw the Vertex struct, say what per-node state lives in one place, and reconstruct its asserted size. + +
Answer + + Seven fields (`vertex.hpp:39-65`): `gid`, `labels`, `in_edges`, + `out_edges`, `properties`, `lock`, `delta_`. Everything about a node + — identity, labels, both adjacency directions, properties, its + latch, and its version chain — is reachable from one pointer. + + `vertex.hpp:73` asserts `sizeof(Vertex) == 80`, and it checks out: + 8 (`Gid`, `id_types.hpp:56`) + 16 + 16 + 16 (three `small_vector`s, + `small_vector.hpp:609`) + 12 (`PropertyStore`, + `property_store.hpp:193`) + 4 (`RWSpinLock`'s `uint32_t`, + `rw_spin_lock.hpp:113`) + 8 (`PointerPack`) = 80. + +
+ +- [ ] You can explain why every edge is stored twice, which query shape breaks if it is not, and what an entry actually costs. + +
Answer + + Both `in_edges` and `out_edges` are stored per vertex + (`vertex.hpp:43-44`), so an edge appears in two vectors. Without + `in_edges`, any pattern that traverses backwards — + `MATCH (a)<-[:FOLLOWS]-(b)` — would need a global reverse index or + a full scan. FalkorDB's answer to the same requirement is the + optional transposed matrix in `struct _Delta_Matrix` + (`delta_matrix.h:113`), which is one bit per edge rather than a + second copy of the entry. + + Cost per entry: `EdgeTriple` = `EdgeTypeId` (4) + `Vertex*` (8) + + `EdgeRef` (8) → 24 B after alignment, so 48 B of adjacency per + logical edge, versus 4 B per directed edge in a CSR `targets` + array. + +
+ - [ ] You can state the difference between per-object delta chains here and per-version rows in postgres, and what each makes cheap. + +
Answer + + memgraph hangs one chain of undo deltas off each *object* + (`vertex.hpp:65`), newest state in place, N2O. Postgres writes a new + *row version* per update and leaves the old one in the heap page, + O2N. + + Per-object chains make a hot object cheap to read fresh (0 hops) and + keep its history in one place — good for a supernode taking + concurrent edge inserts, where all the contention is on one vertex. + Per-version rows make *scans* cheap (no chain to walk; visibility is + a per-tuple test) and updates cheap to abort, but scatter a hot + row's history across pages. + + The price of N2O is stated by the arithmetic in Step 4: a reader + that is Δt stale on an object updated at rate r pays r·Δt undo + hops. Fresh readers pay nothing; a 10-second-stale analytics reader + on a 1 000 update/s vertex pays 10 000. + +
+ - [ ] You can compare the cost of expanding one vertex here against kuzu's CSR slice, and say which workload each layout is built for. + +
Answer + + memgraph: load the `Vertex` (80 B), read `out_edges`' `buffer_` + pointer, then walk *n* × 24 B triples in the heap block. Two + dependent loads before the walk starts, because `kSmallCapacity` for + `EdgeTriple` is 8/24 = 0 and the small buffer is disabled + (`small_vector.hpp:583-584`) — so the block is never inline. + + CSR: `offsets[i]` and `offsets[i+1]` are two adjacent 4-byte reads + in one array, then one contiguous slice of 4-byte ids. Six times + less adjacency data per edge, one indirection fewer, and — the real + win — *neighbouring nodes' slices are adjacent*, so a frontier scan + streams while memgraph's frontier scan visits a fresh heap block per + node. + + memgraph is built for concurrent single-object mutation; CSR is + built for frontier-scale traversal. Neither is a bug. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the PageRank cost sketch. +
Answer + + PageRank is *n* rounds of "for every edge, add source rank / source + degree to target". On this layout each round visits 1 M scattered + `Vertex` objects, dereferences 1 M scattered heap buffers, and then + chases 16 M `Vertex*` pointers to scattered destinations — every one + of them a dependent load the prefetcher cannot issue early. The bus + time goes into latency, not bandwidth: the machine is idle waiting. + + As a matrix it is `r ← A^T r`, one SpMV per round, over 68 MB of + contiguous CSR — the bus time goes into bandwidth, which is the + resource you can actually saturate. That is the same argument as + [reading-graphblas-internals.md](reading-graphblas-internals.md) + Step 1, arriving from the other direction. + +
+ ## References **Papers** @@ -196,8 +526,19 @@ row against a field. shows memgraph making (N2O ordering, delta vs. full-copy version storage, GC strategy) — place memgraph in Wu/Pavlo's 5-axis table -**Code** -- [memgraph](https://github.com/memgraph/memgraph) (cloned for - topic 9) — `src/storage/v2/vertex.hpp` is the whole chapter in one - struct; the skip-list vertex store and delta GC are the topic-9 - machinery reused +**Code** (all line numbers verified at memgraph `8f87f6a`) + +| File | Lines | What | +|---|---|---| +| `src/storage/v2/vertex.hpp` | 29-30 | `EdgeTriple`, `Edges` | +| `src/storage/v2/vertex.hpp` | 32-37 | the struct and its constructor assertion (two accepted actions) | +| `src/storage/v2/vertex.hpp` | 39-47, 65 | the seven fields | +| `src/storage/v2/vertex.hpp` | 49-59, 62-63 | delta accessors and the two packed flag bits | +| `src/storage/v2/vertex.hpp` | 68-70, 72-73 | tuple positions; `sizeof(Vertex) == 80` | +| `src/storage/v2/edge_ref.hpp` | 22, 33-36 | the `Gid`/`Edge*` union | +| `src/storage/v2/id_types.hpp` | 56-59 | id widths | +| `src/storage/v2/property_store.hpp` | 193 | the 12-byte property handle | +| `src/utils/small_vector.hpp` | 30-42 | the ASCII layout diagram | +| `src/utils/small_vector.hpp` | 583-584, 593-595 | `kSmallCapacity`, and when the small buffer is disabled | +| `src/utils/small_vector.hpp` | 599-610 | fields and the 16-byte `static_assert` | +| `src/utils/rw_spin_lock.hpp` | 19-26, 113, 122 | writer-friendly RW spinlock in one `uint32_t` | diff --git a/topics/13-graph-engines/reading-neo4j-record-store.md b/topics/13-graph-engines/reading-neo4j-record-store.md index 1c81817..c02c14e 100644 --- a/topics/13-graph-engines/reading-neo4j-record-store.md +++ b/topics/13-graph-engines/reading-neo4j-record-store.md @@ -8,6 +8,16 @@ relationship record, what an expand actually costs in cache misses, and where the design genuinely wins — then anchors each piece in the source. +Every anchor below is **neo4j pinned at `eccd584a`** +([`resources/codebases.md`](../../resources/codebases.md)), community +edition, under +`community/record-storage-engine/src/main/java/org/neo4j/kernel/impl/store/` +unless another path is given. Record widths and store layout are the +folklore-richest part of this topic — the "15-byte node record" is +repeated everywhere and is *format-specific*, so Step 2 checks which +format is actually the default before quoting any width, and Step 3 +corrects a layout diagram that was missing a byte. + ## The problem in one sentence "Index-free adjacency" — neighbors reachable by following direct @@ -20,53 +30,186 @@ streams at GB/s, and the bet inverts. ### Step 1 — the bet, and the hardware that aged out from under it +> **In:** a query that has found one node and now wants its +> neighbours. +> **Out:** the two candidate cost models — 2010 disk and 2026 DRAM — +> and the observation that the same design scores oppositely under +> them. + **Index-free adjacency** means each node stores a direct physical pointer to its relationships, so expanding a node never consults an -index — you pay one pointer dereference per edge, period. On 2010 -spinning disks this was unbeatable: ANY access was a ~10 ms seek, so -one pointer (1 seek) beat a B-tree descent (3–4 seeks) every time. On -DRAM the cost model changed shape (topic 0): a dependent pointer -dereference is a ~110 ns cache miss the prefetcher can't hide, while -a contiguous array scan streams at ~10 GB/s. The pointers didn't get -slower — *sequential* got 100× faster, and pointers can't be -sequential. Why it matters: every design decision below is downstream -of this bet, and judging them requires the 2026 cost model, not the -2010 one. +index — you pay one pointer dereference per edge, period. + +On 2010 spinning disks this was unbeatable. Any access was a ~10 ms +seek, so: + +``` + index-free: 1 seek × 10 ms = 10 ms + B-tree: 3–4 seeks × 10 ms = 30–40 ms + advantage: 3–4×, and it does not depend on how you lay records out, + because nothing is contiguous when everything is a seek +``` + +On DRAM the cost model changed shape (topic 0). A **dependent load** — +a load whose address came out of the previous load, so the +prefetcher cannot issue it early — costs a ~110 ns last-level miss. +A contiguous array scan streams at ~10 GB/s. Redo the comparison for +one thousand neighbours: + +``` + chain walk: 1000 dependent loads × 110 ns = 110 000 ns = 110 µs + CSR slice: 1000 × 4 B = 4 000 B at 10 GB/s = 400 ns = 0.4 µs + advantage: 275×, and it now runs the other way +``` + +The pointers didn't get slower — *sequential* got 100× faster, and +pointers can't be sequential. Why it matters: every design decision +below is downstream of this bet, and judging them requires the 2026 +cost model, not the 2010 one. ### Step 2 — fixed-size records: the store IS the index -neo4j stores every node in exactly 15 bytes and every relationship in -exactly 34 bytes, so a record's disk/file position is pure arithmetic -— `address = id × RECORD_SIZE` — and looking up "node 42" or -"relationship 1,000,007" needs no index structure at all: +> **In:** a record id, e.g. "node 42". +> **Out:** the (page, offset) pair holding it — computed, not looked +> up — plus the reason the arithmetic is not the one you expect. + +neo4j's record formats are fixed-width, so a record's location is +arithmetic rather than an index probe. The two widths, with the field +inventories the source itself writes above them: ```java -public static final int RECORD_SIZE = 15; // NodeRecordFormat.java:32 -public static final int RECORD_SIZE = 34; // RelationshipRecordFormat.java:35 +// format/standard/NodeRecordFormat.java + 30 public class NodeRecordFormat extends BaseOneByteHeaderRecordFormat { + 31 // in_use(byte)+next_rel_id(int)+next_prop_id(int)+labels(5)+extra(byte) + 32 public static final int RECORD_SIZE = 15; ``` +```java +// format/standard/RelationshipRecordFormat.java + 30 public class RelationshipRecordFormat extends BaseOneByteHeaderRecordFormat { + 31 // record header size + 32 // directed|in_use(byte)+first_node(int)+second_node(int)+rel_type(int)+ + 33 // first_prev_rel_id(int)+first_next_rel_id+second_prev_rel_id(int)+ + 34 // second_next_rel_id+next_prop_id(int)+first-in-chain-markers(1) + 35 public static final int RECORD_SIZE = 34; ``` - Node (15 B): inUse | nextRel(35b) | nextProp(36b) | labels(40b) | flags - Rel (34 B): inUse | firstNode | secondNode | type - | firstPrevRel | firstNextRel ← chain @ first node - | secondPrevRel | secondNextRel ← chain @ second node - | nextProp + +Check the two inventories add up before trusting them: + +``` + node: 1 (in_use) + 4 (next_rel) + 4 (next_prop) + 5 (labels) + 1 (extra) = 15 ✓ + rel: 1 (header) + 4 (first_node) + 4 (second_node) + 4 (rel_type) + + 4 (first_prev) + 4 (first_next) + 4 (second_prev) + 4 (second_next) + + 4 (next_prop) + 1 (chain markers) = 34 ✓ +``` + +**Which format is that, though.** The widths above are the +`format/standard/` package. At this pin the standard family is +deprecated and is *not* the default: + +```java +// format/FormatFamily.java + 28 STANDARD("standard", true /* isDeprecated */), + 29 ALIGNED("aligned", false), + 30 HIGH_LIMIT("high_limit", true /* isDeprecated */), +``` + +```java +// format/RecordFormatSelector.java + 66 private static final RecordFormats DEFAULT_FORMAT = PageAligned.LATEST_RECORD_FORMATS; ``` -Squeezing pointers into 15/34 bytes forces bit tricks: pointers are -35 bits (2³⁵ records max per store), with the high bits smuggled into -the inUse byte — the bit-packing ledger again (compare postgres's -tuple header, topic 8). Why it matters: fixed size buys O(1) id→record -access and trivial free-space management — but note what a node record -does NOT contain: its neighbors. It contains only the head pointer of -a chain. +`PageAligned.LATEST_RECORD_FORMATS` is `PageAlignedV5_0` +(`format/aligned/PageAligned.java:28`), and `PageAlignedV5_0` +constructs the *same* two formats with the alignment flag set: + +```java +// format/aligned/PageAlignedV5_0.java + 49 /** + // ... 50-57: elided — the javadoc explaining the difference from standard ... + 58 * Pages are padded at the end instead of letting a record span 2 pages. + 59 */ + // ... 60-68: elided ... + 69 return new NodeRecordFormat(true); + // ... 70-78: elided ... + 79 return new RelationshipRecordFormat(true); +``` + +So: **15 and 34 are correct for the default community format**, but +for the alignment reason, not because "neo4j records are 15 bytes". +Neo4j Enterprise also ships a *block format* whose records are not +these at all — its existence shows up even in the community settings +file, e.g. `GraphDatabaseSettings.java:794-795` marks a setting "Not +applicable for the block format". Do not carry the number outside the +family it belongs to. + +**Correction.** The previous version of this chapter said the address +is `id × RECORD_SIZE`. It is not — the store is paged, and a record +never straddles a page, so the id is split: + +```java +// RecordPageLocationCalculator.java + 35 public static long pageIdForRecord(long id, int recordsPerPage) { + 36 return id / recordsPerPage; + 37 } + // ... 38-47: elided — javadoc ... + 48 public static int offsetForId(long id, int recordSize, int recordsPerPage) { + 49 return (int) (id % recordsPerPage) * recordSize; + 50 } +``` + +`recordsPerPage` is not a constant either — it is computed from the +page size and the record size: + +```java +// format/BaseRecordFormat.java + 107 @Override + 108 public int getFilePageSize(int pageSize, int recordSize) { + 109 return pageAligned ? pageSize : Math.min(pageSize, pageSize - pageSize % recordSize); + 110 } +``` + +```java +// CommonAbstractStore.java + 377 int filePageSize = recordFormat.getFilePageSize(pageCache.pageSize(), recordSize); + // ... 378-391: elided — the paged file is mapped ... + 392 recordsPerPage = (filePageSize - pagedFile.pageReservedBytes()) / recordSize; +``` + +Work it with the default page size — `PageCache.java:49` in +`community/io/` sets `int PAGE_SIZE = 8192`, and take reserved bytes +as 0: + +``` + nodes: 8192 / 15 = 546 records per page + 546 × 15 = 8190 B used, 8192 − 8190 = 2 B padding per page + rels: 8192 / 34 = 240 records per page + 240 × 34 = 8160 B used, 8192 − 8160 = 32 B padding per page + + node 42: page = 42 / 546 = 0 + offset = (42 % 546) × 15 = 42 × 15 = 630 + node 1 000 000: page = 1 000 000 / 546 = 1831 + 1 000 000 − 1831 × 546 = 1 000 000 − 999 726 = 274 + offset = 274 × 15 = 4110 +``` + +Padding overhead: 2/8192 = 0.02% for nodes, 32/8192 = 0.39% for +relationships. Cheap, and it buys the guarantee that reading a record +is one page access rather than possibly two. + +Why it matters: fixed size buys O(1) id→record access and trivial +free-space management — but note what a node record does NOT contain: +its neighbors. It contains only the head pointer of a chain. ### Step 3 — the relationship record: one edge on two linked lists +> **In:** the 34 raw bytes of a relationship record. +> **Out:** the decoded fields — including the ones that do not fit in +> their nominal width, and the byte the previous diagram forgot. + Each 34-byte relationship record sits on TWO doubly-linked lists -simultaneously — one chain per endpoint — using the four -prev/next fields you saw in the layout: `firstPrevRel/firstNextRel` -thread it into the first node's chain, `secondPrevRel/secondNextRel` +simultaneously — one chain per endpoint. `firstPrevRel`/`firstNextRel` +thread it into the first node's chain, `secondPrevRel`/`secondNextRel` into the second node's: ``` @@ -76,22 +219,107 @@ into the second node's: ``` One physical record, two logical list memberships — so both endpoints -can enumerate their edges without storing the edge twice. The records -of one node's chain, however, live wherever *insertion order* put them -in the file; there is no locality guarantee whatsoever. Why it -matters: the chain is the data structure every traversal walks — its -memory layout (scattered) is the whole performance story of Step 4. +can enumerate their edges without storing the edge twice. + +**Correction.** The previous version's layout diagram ended at +`nextProp` and omitted the trailing byte. It is there, it is +load-bearing, and the source documents its four flags: + +```java +// format/standard/RelationshipRecordFormat.java, inside read() + 63 byte headerByte = cursor.getByte(); + // ... 64-69: elided — in-use flag, first-node and next-prop high bits ... + 70 long firstNode = cursor.getInt() & 0xFFFFFFFFL; + 71 long firstNodeMod = (headerByte & 0xEL) << 31; + // ... 72-74: elided ... + 75 // [ xxx, ][ , ][ , ][ , ] second node high order bits, 0x70000000 + // ... 76-79: elided — the same map for the four chain pointers ... + 80 // [ , ][ , ][xxxx,xxxx][xxxx,xxxx] type + 81 long typeInt = cursor.getInt(); + // ... 82: elided ... + 83 int type = (int) (typeInt & 0xFFFF); + // ... 84-99: elided — the four chain pointers and nextProp, each with its mod ... + 100 // [ , x] 1:st in start node chain, 0x1 + 101 // [ , x ] 1:st in end node chain, 0x2 + 102 // [ , x ] first is guaranteed dense, 0x4 + 103 // [ ,x ] second is guaranteed dense, 0x8 + 104 byte extraByte = cursor.getByte(); +``` + +The corrected diagrams, with the real field widths: + +``` + Node (15 B): header(1) | nextRel(4) | nextProp(4) | labels(5) | extra(1) + nextRel = 32 low bits + 3 bits from the header → 35 bits + nextProp = 32 low bits + 4 bits from the header → 36 bits + labels = 32 low bits + 8 bits (hsbLabels) → 40 bits + extra bit 0 = dense + + Rel (34 B): header(1) | firstNode(4) | secondNode(4) | typeInt(4) + | firstPrevRel(4) | firstNextRel(4) ← chain @ first node + | secondPrevRel(4)| secondNextRel(4) ← chain @ second node + | nextProp(4) | extraByte(1) + type = typeInt & 0xFFFF → 16 bits + the other 16 bits of typeInt carry 3 bits each of secondNode, + firstPrevRel, firstNextRel, secondPrevRel, secondNextRel + extraByte bits: first-in-start-chain, first-in-end-chain, + first-guaranteed-dense, second-guaranteed-dense +``` + +The bit-smuggling has a ceiling, and the ceiling is declared: + +```java +// format/standard/StandardFormatSettings.java + // ... 20-28: elided ... + 29 public static final int NODE_MAXIMUM_ID_BITS = 35; + 30 public static final int RELATIONSHIP_MAXIMUM_ID_BITS = 35; + 31 public static final int PROPERTY_MAXIMUM_ID_BITS = 36; + // ... 32-37: elided ... + 38 public static final int RELATIONSHIP_TYPE_TOKEN_MAXIMUM_ID_BITS = 16; + 39 public static final int RELATIONSHIP_GROUP_MAXIMUM_ID_BITS = 35; + // ... 40-48: elided ... + 49 static long bitsToMaxId(int bits) { + 50 return (1L << bits) - 1; + 51 } +``` + +So the store caps are computable, and worth computing because they +are design constraints, not trivia: + +``` + nodes / relationships: 2^35 − 1 = 34 359 738 367 ≈ 34.4e9 + properties: 2^36 − 1 = 68 719 476 735 ≈ 68.7e9 + relationship types: 2^16 − 1 = 65 535 ← the tight one + labels field is 40 bits, but it is an inline label *set*, not an id +``` + +65 535 relationship types is the constraint that bites first — a +schema that encodes data into type names (`:LIKED_2024_01`) runs out. +Compare postgres's tuple header (topic 8): the same bit-packing +ledger, the same habit of stealing high bits from a flags byte. + +The records of one node's chain, however, live wherever *insertion +order* put them in the file; there is no locality guarantee +whatsoever. Why it matters: the chain is the data structure every +traversal walks — its memory layout (scattered) is the whole +performance story of Step 4. ### Step 4 — expand = one dependent load per edge +> **In:** a node record and the relationship store. +> **Out:** that node's neighbour ids, and a count of dependent loads +> — which is the real currency. + Expanding a node means walking its chain: read a record, look at which endpoint you are, follow the corresponding next pointer — and each next address is unknown until the current record arrives, so the CPU cannot prefetch anything: ```rust -// expand(A) in a record store: a linked-list walk where every hop -// is a dependent load — the CPU cannot prefetch what it hasn't read +// ILLUSTRATION — not neo4j source. The real decode is +// format/standard/RelationshipRecordFormat.java:63-104 and the chain +// fields are record/RelationshipRecord.java:39-42; this is the shape of +// the walk those two files imply. fn expand(rels: &[RelRecord], node: &NodeRecord) -> Vec { let mut out = Vec::new(); let mut r = node.next_rel; @@ -109,69 +337,163 @@ fn expand(rels: &[RelRecord], node: &NodeRecord) -> Vec { } ``` -The arithmetic: **one 34-byte record read — one potential cache/page -miss — per edge**. A supernode with 100K edges = 100K dependent loads -≈ 11 ms of pure memory latency. The CSR (compressed sparse row — -adjacency as one offsets array plus one contiguous neighbors array) -spelling of the same operation is `targets[offsets[i]..offsets[i+1]]` -— one slice, hardware prefetcher does the rest, ~40 µs for the same -100K neighbors at 10 GB/s. This is Drepper's pointer-chase-vs-stream -distinction (topic 0) elevated to an architecture. Why it matters: -this per-edge miss is the line item FalkorDB's matrices and kuzu's -CSR exist to delete. +**One 34-byte record read — one potential cache/page miss — per +edge.** Two ways to price it, and they differ by three orders of +magnitude, so state which you mean: + +``` + in-memory, records cached, scattered: + 100 000 edges × 110 ns dependent load = 11.0 ms + CSR, same 100 000 neighbours, 4 B ids, 10 GB/s: + 400 000 B / 10e9 B/s = 40 µs + ratio = 275× + + cold, records on disk, 240 rels/page (Step 2): + worst case one page fault per record = 100 000 page reads + best case, chain perfectly clustered = 100 000/240 = 417 page reads + the 240× spread between those two is exactly what "no locality + guarantee" costs you, and nothing in the format decides it — insertion + order does +``` + +Now anchor it against this topic's own measurement. The bench's +adjacency-list oracle stores each node's neighbours contiguously and +still takes 495 378 ns for a two-hop from supernodes +([notes.md](notes.md); [FINDINGS.md](../../FINDINGS.md) row 13). +Divide: + +``` + 495 378 ns / 110 ns per dependent load = 4 503 dependent loads' worth +``` + +A top-100-degree node in that graph has up to 6 565 first-hop edges +alone, and the two-hop expansion is far larger than that. So the +oracle is plainly *not* paying one miss per edge — it is streaming +contiguous slices, and 495 µs is a **lower bound** on what the same +query would cost a record store, which adds a dependent load per edge +on top. Why it matters: this per-edge miss is the line item +FalkorDB's matrices and kuzu's CSR exist to delete. ### Step 5 — chain maintenance: deletes, lookups, and dense nodes +> **In:** a relationship to delete, or a "is there an edge between A +> and B" question. +> **Out:** the cost, and the extra record types neo4j added to keep +> that cost bounded. + The chains create their own bookkeeping costs, which neo4j itself -acknowledges. Deleting a relationship must unlink it from BOTH -doubly-linked chains — up to 4 neighbor records touched and rewritten. -Finding a *specific* relationship between two given nodes means -walking a chain until you hit it — O(degree) — so neo4j stores the -degree for "dense" nodes and walks the shorter endpoint's chain (see -`RelationshipGroup` records, which also split chains per relationship -type/direction). Why it matters: linked structures make every -structural query a walk; the mitigations (degree caches, relationship -groups) are extra record types patching the base design's asymptotics. +acknowledges. + +- **Delete** must unlink the record from BOTH doubly-linked chains, so + up to 4 neighbour records are touched and rewritten. That is O(1) — + the record ids are in the record you already have — but it is 4 + scattered writes. +- **Find a specific relationship between two given nodes** means + walking a chain until you hit it: O(degree). +- **Dense nodes** get a mitigation. Past a threshold, a node's + relationships are grouped by type and direction into + `RelationshipGroup` records: + +```java +// format/standard/RelationshipGroupRecordFormat.java + 31 // [type+inUse+highbits,next,firstOut,firstIn,firstLoop,owningNode] + // ... 32-37: elided ... + 38 public static final int RECORD_SIZE = 25; +``` + +The threshold is a setting with a default you can read: + +```java +// configuration/GraphDatabaseSettings.java (community/configuration/...) + 796 public static final Setting dense_node_threshold = newBuilder( + 797 "db.relationship_grouping_threshold", INTEGER, 50) +``` + +Fifty. Put that against the bench graph's degree distribution +(p50 degree 11, max degree 6 565, [notes.md](notes.md)): + +``` + p50 node, degree 11 → 11 < 50 → not dense, one flat chain + top node, degree 6565 → ≥ 50 → dense, grouped by type+direction + a group record is 25 B and buys per-type chain heads, so + "expand only :FOLLOWS out-edges" stops walking the other types +``` + +Why it matters: linked structures make every structural query a walk; +the mitigations (degree caches, relationship groups) are extra record +types patching the base design's asymptotics — and they only engage +above 50, which is to say only on the tail that this topic's headline +is about. ### Step 6 — where records win -Be fair (topic 0's benchmarking lesson) — the design has a real -home turf: +> **In:** the same design, judged on the mutation path instead of the +> traversal path. +> **Out:** the three workloads where fixed-size records are the right +> answer, and the symmetric price the other camp pays. + +Be fair (topic 0's benchmarking lesson) — the design has a real home +turf: - **single-edge insert**: write one 34 B record + patch 2–4 chain - pointers — no CSR shifting, no delta-overlay machinery needed at - all -- **update-in-place**: fixed-size slots never move; MVCC/undo is - page-based, not copy-the-adjacency -- **uniform record access**: "get relationship by id" is O(1) - arithmetic (Step 2) + pointers. Price it against the CSR alternative on the bench graph's + 16.0 M edges with 4 B ids: + + ``` + record store: 34 B write + 4 × 34 B pointer patches = 170 B touched + raw CSR: average memmove of half of 16e6 × 4 B = 32 MB touched + ratio ≈ 188 000× + ``` + + That is why CSR engines need an overlay at all + ([reading-graphblas-internals.md](reading-graphblas-internals.md) + Step 7) and neo4j needs none. +- **update-in-place**: fixed-size slots never move, so MVCC/undo is + page-based rather than copy-the-adjacency. +- **uniform record access**: "get relationship by id" is the Step 2 + page/offset arithmetic — no index, no search. The trade in one sentence: neo4j optimized the OLTP mutation path and pays on every traversal; CSR/matrix engines optimize traversal and -need an overlay (kuzu's buffers, FalkorDB's Delta_Matrix) to survive -writes. Why it matters: neither side dodges the tension — they pick -opposite ends and buy back the other end with extra machinery. +need an overlay (kuzu's transient node groups, FalkorDB's +Delta_Matrix) to survive writes. Why it matters: neither side dodges +the tension — they pick opposite ends and buy back the other end with +extra machinery. ## Where each step lives in the code -Everything lives under +Paths are relative to `community/record-storage-engine/src/main/java/org/neo4j/kernel/impl/store/` -in a shallow clone of [neo4j](https://github.com/neo4j/neo4j): - -- **Step 2** — `format/standard/NodeRecordFormat.java:32` - (`RECORD_SIZE = 15`) and - `format/standard/RelationshipRecordFormat.java:35` - (`RECORD_SIZE = 34`). Read both files' `readRecord` methods — they - ARE the layout diagrams above, including the 35-bit pointer - reassembly from the inUse byte's high bits. -- **Steps 3–5** — `record/RelationshipRecord.java:39-44` — the four - chain fields (firstPrevRel/firstNextRel, - secondPrevRel/secondNextRel). For the dense-node mitigation, grep - for `RelationshipGroup`. - -Read order: the two `readRecord` methods first (they make the byte -layouts concrete), then `RelationshipRecord.java` for the chain -fields, then trace Step 4's walk mentally against them. +except where noted, in [neo4j](https://github.com/neo4j/neo4j) pinned +at `eccd584a`. + +| Step | Anchor | What is there | +|---|---|---| +| 2 | `format/standard/NodeRecordFormat.java:31-32` | field inventory comment + `RECORD_SIZE = 15` | +| 2 | `format/standard/RelationshipRecordFormat.java:31-35` | field inventory comment + `RECORD_SIZE = 34` | +| 2 | `format/FormatFamily.java:28-30` | STANDARD and HIGH_LIMIT are deprecated; ALIGNED is not | +| 2 | `format/RecordFormatSelector.java:66` | `DEFAULT_FORMAT = PageAligned.LATEST_RECORD_FORMATS` | +| 2 | `format/aligned/PageAligned.java:28` | that resolves to `PageAlignedV5_0` | +| 2 | `format/aligned/PageAlignedV5_0.java:49-58, 69, 79` | "padded at the end"; same 15/34 formats, aligned flag | +| 2 | `RecordPageLocationCalculator.java:35-37, 48-50` | `pageIdForRecord`, `offsetForId` — the real address arithmetic | +| 2 | `format/BaseRecordFormat.java:107-110` | `getFilePageSize` — where the padding comes from | +| 2 | `CommonAbstractStore.java:377, 392` | `filePageSize`, `recordsPerPage` | +| 2 | `community/io/.../pagecache/PageCache.java:49` | `PAGE_SIZE = 8192` | +| 3 | `format/standard/NodeRecordFormat.java:55-70` | the decode: 35-bit nextRel, 36-bit nextProp, 40-bit labels, dense flag | +| 3 | `format/standard/RelationshipRecordFormat.java:63-104` | the decode, the bit map comments, and the extra byte | +| 3 | `format/standard/StandardFormatSettings.java:29-31, 38-39, 49-51` | the id-width ceilings and `bitsToMaxId` | +| 3–4 | `record/RelationshipRecord.java:39-42` | the four chain fields (was cited as `:39-44`) | +| 3–4 | `record/RelationshipRecord.java:43-44` | `firstInFirstChain`, `firstInSecondChain` — the extra byte's first two flags | +| 5 | `format/standard/RelationshipGroupRecordFormat.java:31-33, 38` | the group layout comment and `RECORD_SIZE = 25` | +| 5 | `community/configuration/.../GraphDatabaseSettings.java:796-797` | `db.relationship_grouping_threshold`, default 50 | + +Read order: the two `read` methods first (they make the byte layouts +concrete — `NodeRecordFormat.java:55-70`, then +`RelationshipRecordFormat.java:63-104`), then +`StandardFormatSettings.java` for why the bit-stealing stops where it +does, then `RecordPageLocationCalculator.java` for the address +arithmetic, then trace Step 4's walk mentally against +`RelationshipRecord.java:39-42`. ## Questions (answer in notes.md) @@ -191,19 +513,121 @@ fields, then trace Step 4's walk mentally against them. ## Done when +Answer each before unfolding it. + - [ ] You can explain the index-free adjacency bet and name the hardware assumption that aged out from under it. -- [ ] You can compute expand cost for a 1000-edge node as dependent loads, and compare it to a CSR slice. -- [ ] You can say what the doubly-linked relationship chain buys and what it costs on insert. -- [ ] You can state the modern version of the index-free adjacency argument — the one that survives the disk era ending. + +
Answer + + The bet: store a direct physical pointer from node to relationship + so expansion never consults an index. The assumption: that *any* + access costs the same, because it is a ~10 ms seek — under which one + pointer hop (1 seek) beats a B-tree descent (3–4 seeks) by 3–4×. + + What aged out is not the pointer, it is the alternative. Sequential + access got ~100× faster while dependent random access did not, so + the same thousand neighbours are 1000 × 110 ns = 110 µs as a chain + walk and 4 000 B / 10 GB/s = 0.4 µs as a CSR slice — a 275× reversal. + +
+ +- [ ] You can compute expand cost for a 1000-edge node as dependent loads, compare it to a CSR slice, and say what decides the cold-cache case. + +
Answer + + Warm: 1 000 dependent loads × 110 ns = 110 µs versus 4 000 B at + 10 GB/s = 0.4 µs — 275×. + + Cold, the answer is a range rather than a number, and the range is + the point. `CommonAbstractStore.java:392` and + `BaseRecordFormat.java:107-110` give 8192/34 = 240 relationship + records per 8 KiB page. If the chain is perfectly clustered, 1 000 + records is 1000/240 = 5 page reads (rounding up from 4.17). If it is + scattered, it is up to 1 000 page reads. Nothing in the record format + decides which — insertion order does, because a relationship record + is appended where the free list puts it, not where its chain + neighbours are. + +
+ +- [ ] You can say what the doubly-linked relationship chain buys and what it costs on insert and delete, with the byte counts. + +
Answer + + It buys: one physical 34 B record serving both endpoints' + enumeration, so an edge is stored once, not twice; and O(1) + unlinking given the record, because `firstPrevRel`/`firstNextRel`/ + `secondPrevRel`/`secondNextRel` (`RelationshipRecord.java:39-42`) + name the neighbours directly. + + It costs: delete touches up to 4 other records (170 B of scattered + writes); "is there an edge A→B" is O(degree) with no shortcut; and + the chain has no locality, which is Step 4's whole problem. + + Insert is where it wins outright: ~170 B touched versus ~32 MB for + an in-place CSR insert on a 16 M-edge graph — about 188 000×. + +
+ +- [ ] You can state the modern version of the index-free adjacency argument — the one that survives the disk era ending — and name the format caveat on the 15/34 numbers. + +
Answer + + What survives: *no index probe on the mutation path*. Insert, + delete and get-by-id are pure arithmetic plus a bounded number of + record writes, with no B-tree to split and no adjacency structure to + rebuild. That is a real advantage and it is why the CSR camp has to + bolt an overlay on. + + What died: the claim that pointer-following is the *fastest way to + read* neighbours. On DRAM it is the slowest way, by 275× against a + contiguous slice. + + The caveat: 15 and 34 belong to `format/standard/` and the aligned + formats built from it. `FormatFamily.java:28-30` marks STANDARD + deprecated, `RecordFormatSelector.java:66` makes `PageAligned` the + default (same widths, page-end padding — + `PageAlignedV5_0.java:58`), and Enterprise's block format is a + different layout entirely. Quote the number with its family. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the 15 B versus 34 B field accounting. +
Answer + + The accounting is written in the source comments and both add up: + node = 1 + 4 + 4 + 5 + 1 = 15 (`NodeRecordFormat.java:31`); + relationship = 1 + 4×8 + 1 = 34 + (`RelationshipRecordFormat.java:32-34`). + + The asymmetry is structural, not arbitrary: a node stores one chain + *head* (4 B) and nothing about its neighbours, while a relationship + must name two endpoints (8 B), a type, and *four* chain pointers + (16 B) because it is a member of two doubly-linked lists at once. + Sixteen of the relationship record's 34 bytes — 47% — are chain + maintenance. That is the storage cost of Step 4's design, paid on + every edge. + +
+ ## References -**Code** -- [neo4j](https://github.com/neo4j/neo4j) (shallow clone) — everything - lives under - `community/record-storage-engine/src/main/java/org/neo4j/kernel/impl/store/`: - `format/standard/NodeRecordFormat.java`, - `format/standard/RelationshipRecordFormat.java` (read both - `readRecord` methods for the layouts), - `record/RelationshipRecord.java` +**Code** (all line numbers verified at neo4j `eccd584a`) + +| File (under `community/record-storage-engine/.../impl/store/` unless noted) | Lines | What | +|---|---|---| +| `format/standard/NodeRecordFormat.java` | 31-32, 55-70 | width, field inventory, decode | +| `format/standard/RelationshipRecordFormat.java` | 31-35, 63-104 | width, field inventory, decode, extra byte | +| `format/standard/StandardFormatSettings.java` | 29-31, 38-39, 49-51 | id-width ceilings, `bitsToMaxId` | +| `format/standard/RelationshipGroupRecordFormat.java` | 31-33, 38 | dense-node group record, 25 B | +| `format/FormatFamily.java` | 28-30 | which families are deprecated | +| `format/RecordFormatSelector.java` | 66 | the actual default format | +| `format/aligned/PageAligned.java` | 28 | → `PageAlignedV5_0` | +| `format/aligned/PageAlignedV5_0.java` | 49-58, 69, 79 | page-end padding; same 15/34 formats | +| `format/BaseRecordFormat.java` | 107-110 | `getFilePageSize` | +| `RecordPageLocationCalculator.java` | 35-37, 48-50 | page id and offset from record id | +| `CommonAbstractStore.java` | 377, 392 | `filePageSize`, `recordsPerPage` | +| `record/RelationshipRecord.java` | 36-44 | the two endpoints, type, four chain fields, two chain flags | +| `community/io/.../pagecache/PageCache.java` | 49 | `PAGE_SIZE = 8192` | +| `community/configuration/.../GraphDatabaseSettings.java` | 794-795, 796-797 | block-format caveat; dense-node threshold 50 | diff --git a/topics/13-graph-engines/reading-query-languages.md b/topics/13-graph-engines/reading-query-languages.md index 0c4ab8f..677f8d3 100644 --- a/topics/13-graph-engines/reading-query-languages.md +++ b/topics/13-graph-engines/reading-query-languages.md @@ -6,9 +6,19 @@ and composability. This chapter builds each fault line step by step — ending with what each language lets a planner do — because the same two-hop pattern returns three different counts depending on semantics the language may not even let you spell. The route runs the family -tree from Cypher through GQL (the first new ISO database language -since SQL, 1987); keep kuzu's `src/antlr4/Cypher.g4` open as the -concrete grammar (a full Cypher in 690 lines). +tree from Cypher through GQL, the first new ISO database language +since SQL itself (ISO 9075:**1987**); keep kuzu's +`src/antlr4/Cypher.g4` open as the concrete grammar. + +Every standards claim below is cited to Deutsch et al., *Graph Pattern +Matching in GQL and SQL/PGQ* (SIGMOD 2022, +[arXiv:2112.06217](https://arxiv.org/abs/2112.06217)) by section or +figure; every ISO designation was checked against iso.org; every +grammar anchor is **kuzu pinned at `89f0263`** +([`resources/codebases.md`](../../resources/codebases.md)). Four +numbers in the previous version of this chapter — the size of +`Cypher.g4`, the date on Cypher, the list of GQL restrictors, and the +year on the GQL standard — did not survive that check. ## The problem in one sentence @@ -21,116 +31,304 @@ the answer, and most languages don't let you say which one you meant. ### Step 1 — fault line one: what a graph even is (property graph vs RDF) +> **In:** one fact with an attribute on the relationship — +> `alice KNOWS bob since 2019`. +> **Out:** its representation in each model, counted in storage units +> and in joins, which is where the two models diverge for good. + A **property graph** makes nodes AND edges first-class objects that carry labels and key-value properties — `since: 2019` lives *on* the KNOWS edge. **RDF** (Resource Description Framework — data as subject-predicate-object **triples** like `:alice :knows :bob`) has no place to put an edge property: a triple is atomic. The workarounds are **reification** (create a statement-node representing the edge, then -hang properties off it — one edge becomes 4 triples and every edge -query becomes extra joins) or RDF-star's edge-triples -(`<< :a :knows :b >> :since 2019`). Why it matters: this single -modeling choice — the edge as a first-class citizen — is most of why -property graphs won the application market, and it decides query -*shape*: SPARQL plans tend toward many small self-joins ("SPARQL is -10 joins where Cypher is 2 expands"). +hang properties off it) or RDF-star's edge-triples +(`<< :a :knows :b >> :since 2019`). Count the reification, since +question 3 asks for exactly this: + +``` + property graph: 1 node record for alice, 1 for bob, + 1 edge record carrying {since: 2019} + + RDF reification: :s rdf:type rdf:Statement . + :s rdf:subject :alice . + :s rdf:predicate :knows . + :s rdf:object :bob . + :s :since 2019 . + = 5 triples for one edge, and the original + (:alice :knows :bob) triple is usually kept too → 6 + + traversal cost: property graph 1 edge dereference + reified RDF 4 self-joins on the statement node + to recover (subject, object) + filter +``` + +The paper puts the split in institutional terms: "Unlike RDF with its +query language SPARQL, which is a W3C standard, property graph systems +possess disparate storage models and querying facilities" (§1, p.2) — +one model got a standard early and one got a decade of dialects. Why +it matters: this single modeling choice — the edge as a first-class +citizen — is most of why property graphs won the application market, +and it decides query *shape*: SPARQL plans tend toward many small +self-joins where Cypher does two expands. ### Step 2 — fault line two: matching semantics — the same pattern, three answers +> **In:** one graph, one pattern, and three rules about what may +> repeat. +> **Out:** three different integers, computed by hand — the +> demonstration that "matching semantics" is a number, not a +> philosophy. + **Matching semantics** is the rule for which subgraph assignments count as matches — specifically, whether pattern variables may repeat graph elements. Three standard choices: **homomorphism** (anything may repeat — nodes and edges), **isomorphism** (no repeated nodes), -**trail** (no repeated *edges*). They give different answers on the -same data: +**trail** (no repeated *edges*). + +The previous version of this chapter tried to demonstrate the split on +a 2-edge pattern. That does not work: on a simple graph, a 2-path with +distinct edges automatically has distinct endpoints, so trail and +node-isomorphism agree by construction. You need three edges. Take the +undirected triangle and count properly: ``` -graph: a triangle 1 ──► 2 ──► 3 ──► 1 + graph: K3 — nodes 1,2,3; undirected edges {1,2}, {2,3}, {3,1} + query: MATCH (a)-[e1]-(b)-[e2]-(c)-[e3]-(d) — 3-edge walks + + homomorphism (anything may repeat): + pick a: 3 ways; each node has degree 2, so each step has 2 choices + 3 × 2 × 2 × 2 = 24 + cross-check with the matrix spelling: the number of length-3 walks + is 1ᵀA³1. A = J − I has eigenvalues 2, −1, −1, and 1 is the + eigenvector for 2, so A³1 = 2³·1 = 8·1 and 1ᵀA³1 = 8 × 3 = 24 ✓ + + trail (no repeated EDGE): + three distinct edges in a 3-edge graph means using all of them, + i.e. walking the triangle: 3 starting nodes × 2 directions = 6 -query: MATCH (a)-[]->(b)-[]->(c) — count the 2-paths + node-isomorphism (no repeated NODE): + the pattern needs 4 distinct nodes; the graph has 3 + = 0 -homomorphism (nodes+edges may repeat): 1-2-3, 2-3-1, 3-1-2, - and a=c ones like 1-2-1? no edge 2→1 — but - add a back-edge and a=c matches appear -isomorphism (no repeated nodes): only node-distinct walks -trail (no repeated edges): Cypher's [*] var-length rule + 24 / 6 / 0 — same graph, same pattern, three answers ``` +Note what the matrix cross-check implies: `A³`'s grand sum *is* the +homomorphism count. Linear algebra is homomorphism-native, which is +why FalkorDB +([reading-graphblas-internals.md](reading-graphblas-internals.md)) +gets that semantics for free and has to work for any other. + This is the SIGMOD'22 paper's core: **matching semantics is a language -parameter, not folklore**. Cypher hard-coded a hybrid in 2012 — -homomorphism for nodes, trail for variable-length edge patterns — a -semantics decision disguised as a default, and every engine since has -had to reverse-engineer the corner cases. Why it matters: two engines -can both "support Cypher patterns" and return different counts; if you -build an engine (M13), you must *pick* and document. - -### Step 3 — GQL makes the semantics syntax: restrictors and selectors - -GQL (ISO 39075:2024) and SQL/PGQ turn Step 2's parameter into explicit -syntax: a **restrictor** names which matches are legal -(TRAIL / ACYCLIC / SIMPLE, or ALL for homomorphism) and a **selector** -names which of the legal matches to return (ANY SHORTEST, ALL -SHORTEST, ANY k) — `MATCH ALL TRAIL (a)-[]->{1,5}(b)` says both out -loud. The other first-class addition: **quantified path patterns** — -`(a) (-[:KNOWS]->){1,5} (b)` — regular-expression-style repetition -over a path segment, replacing Cypher's `[*1..5]` with a composable -form. Why it matters: restrictors aren't just documentation — they're -*prunable*: TRAIL/ACYCLIC bound the search on supernodes where -unrestricted expansion explodes; and M13's capstone rule (keep the AST -GQL-shaped: quantified path patterns + an explicit path-mode field) -exists so M10's parser survives GQL compatibility without a rewrite. +parameter, not folklore**. Cypher hard-coded a hybrid — homomorphism +for nodes, no-repeated-relationship for edges — a semantics decision +disguised as a default, and every engine since has had to +reverse-engineer the corner cases. + +**Correction.** The previous version dated that decision to "Cypher +2012". The paper does not support a year for Cypher specifically; what +it says is that declarative property graph languages appeared "since +2010" — "Cypher from Neo4j, GSQL from TigerGraph, and PGQL from +Oracle, as well as industry/academia prototypes such as G-CORE" — and +that "the **2015** openCypher project has led to a widening industrial +use of Cypher as a language for property graphs, but did not succeed +on its own in establishing a standard" (§1, p.2). Use 2010 for the +category and 2015 for openCypher; drop the 2012. + +Why it matters: two engines can both "support Cypher patterns" and +return different counts; if you build an engine (M13), you must *pick* +and document. + +### Step 3 — GQL makes some of the semantics syntax: restrictors and selectors + +> **In:** an unbounded quantifier like `-[t:Transfer]->*` over a graph +> with a cycle — a query with infinitely many matches. +> **Out:** the two GPML devices that force finiteness, their exact +> keyword lists, and the asymmetry between them that decides which one +> can turn a non-empty answer into an empty one. + +GQL and SQL/PGQ share a pattern sublanguage the paper calls **GPML** +(§1, p.2), and GPML turns part of Step 2's parameter into explicit +syntax. The motivation is not elegance, it is termination: + +> "Written without any restrictions, GPML queries may not terminate as +> they will return infinitely many matches. … To prevent this +> behaviour, GPML queries must demonstrably terminate; in particular, +> the number of matches must be finite. To achieve this, GPML uses +> restrictors and selectors. **Every unbounded quantifier (such as * +> above) must be contained in the scope of either a restrictor or a +> selector or both.**" +> — §5, p.17 + +A **restrictor** is "a path predicate … such that the number of +matches cannot be infinite" (§5.1). **Correction:** the previous +version listed the restrictors as "TRAIL / ACYCLIC / SIMPLE, or ALL +for homomorphism". There is no `ALL` restrictor. Figure 7 lists +exactly three: + +| Keyword | Description (Fig. 7, p.19) | +|---|---| +| `TRAIL` | No repeated edges. | +| `ACYCLIC` | No repeated nodes. | +| `SIMPLE` | No repeated nodes, except that the first and last nodes may be the same. | + +A **selector** is "an algorithm that conceptually partitions the +solution space on the endpoints and selects a finite set of matches +from each partition" (§5.1). Figure 8 lists six: +`ANY SHORTEST`, `ALL SHORTEST`, `ANY`, `ANY k`, `SHORTEST k`, and +`SHORTEST k GROUP` — of which only `ALL SHORTEST` and +`SHORTEST k GROUP` are marked **deterministic**; the other four +explicitly are not. + +The two compose in a fixed order — "restrictors can be seen as +operating *during* pattern matching while selectors operate afterwards +… if combined, selectors are always applied after restrictors" (§5.1) +— so `MATCH ALL SHORTEST TRAIL p = …` means "the shortest among the +trails", not "the trails among the shortest". And that ordering has a +consequence worth memorizing, because it is the one property that +distinguishes the two devices: + +> "Consider a query Q with no selector or restrictor, and assume that +> Q has matches. Then, adding a **selector** to Q might reduce the +> number of matches, but the resulting query will **always have at +> least one match**. On the other hand, adding a **restrictor** to Q +> might yield a query with **no matches at all**." +> — §5.1, p.19 + +A selector is a projection; a restrictor is a filter. Get them +backwards in an optimizer and you will "optimize" a query into +returning nothing. + +The other first-class addition is **quantified path patterns** — +`(a) (-[:KNOWS]->){1,5} (b)` — "quantifiers similar to those in Perl +and other common 'regex' tools … written as postfix operators on +either a single edge pattern or a parenthesized path pattern" (§4, +p.14), replacing Cypher's `[*1..5]` with a composable form. + +**Correction — scope.** The previous version implied GQL makes +matching semantics fully configurable. It does not. Restrictors +constrain repetition *within a path pattern*; constraining repetition +*across* the whole graph pattern is listed as a **Language +Opportunity**, i.e. deferred out of the shipped standard: + +> "…a sample of LOs pertaining to GPML: • Constraining a graph pattern +> through the introduction of **isomorphic match modes**: for example, +> an edge-isomorphic match requires all edges matched across all +> constituent path patterns in the graph pattern to differ from each +> other." +> — §7.1, p.28 + +So Step 2's isomorphism column is still not spellable in GPML as the +paper describes it. Why it matters: restrictors aren't just +documentation — they're *prunable*: TRAIL/ACYCLIC bound the search on +supernodes where unrestricted expansion explodes; and M13's capstone +rule (keep the AST GQL-shaped: quantified path patterns + an explicit +path-mode field) exists so M10's parser survives GQL compatibility +without a rewrite. ### Step 4 — the family tree: two standards, one MATCH grammar -The 2024 standards landscape collapses to one fact: SQL/PGQ (property +> **In:** the two ISO projects and their dates. +> **Out:** the one structural fact that makes targeting both cheap — +> plus an honest note about what the source paper predicted and what +> actually shipped. + +The standards landscape collapses to one fact: SQL/PGQ (property graphs *inside* SQL — a `GRAPH_TABLE(...)` clause whose MATCH returns -a table you join like any other; DuckDB ships it as duckpgq, Oracle -too) and GQL (a standalone graph language with graph DDL and -graph-to-graph queries) share the SAME pattern-matching grammar, -written by a shared committee: +a table you join like any other) and GQL (a standalone graph language +with graph DDL and graph-to-graph queries) share the SAME pattern +matching sublanguage, by policy: + +> "In 2019 the Joint Technical Committee 1 of ISO/IEC … approved a +> project to create GQL, a standard property graph query language with +> full CRUD … and catalog capability. GQL builds on prior graph +> languages, as well as **a new part 16 of SQL, in development since +> 2017, called SQL/PGQ**." +> — §1, p.2 + +> "Both language projects have been assigned to the **ISO/IEC JTC1 +> SC32 … Working Group for Database Languages (WG3)** which continues +> to be responsible for maintaining and enhancing SQL as a whole. +> **This structure serves a policy that GPML be kept identical in GQL +> and SQL/PGQ.**" +> — §1, p.3 ```mermaid graph TD - SQL["SQL (ISO 9075)"] --> PGQ["SQL/PGQ 2023
GRAPH_TABLE(...)"] - C[Cypher 2012] --> OC[openCypher] --> GQL["GQL ISO 39075:2024"] - G[G-CORE 2018
research consensus] --> GQL - PGQ <-->|"same MATCH grammar
(shared committee)"| GQL + SQL["SQL (ISO 9075:1987)"] --> PGQ["SQL/PGQ
ISO/IEC 9075-16
GRAPH_TABLE(...)"] + C["Cypher (declarative PG
languages, since 2010)"] --> OC["openCypher project, 2015"] --> GQL["GQL
ISO/IEC 39075:2024"] + G["G-CORE, SIGMOD 2018
research consensus"] --> GQL + PGQ <-->|"GPML: same pattern sublanguage
(policy of SC32 WG3)"| GQL SPARQL["SPARQL 1.1 (W3C, RDF)"] -.->|"paths, not property graphs"| GQL ``` +**Correction — the year.** The previous version dated SQL/PGQ to 2023. +The paper's Figure 10 (p.28) is a *projected* timeline, carrying +footnote 6: "The schedule depends on work that has not been completed +and so could change." It projected the SQL/PGQ IS for 2023-03-13 and +the GQL IS for 2023-09-10. What actually published is **ISO/IEC +39075:2024**, *Information technology — Database languages — GQL* — +GQL slipped past its own projection by roughly a year. SQL/PGQ is +**ISO/IEC 9075-16**, *Information technology — Database languages SQL +— Part 16: Property Graph Queries (SQL/PGQ)*; its publication year is +not stated in the paper and is not quoted here. Cite the part number, +not a year you cannot source. + +The WG3 membership is worth knowing when you weigh how binding this +is: expert members represent the national standards bodies of China, +Denmark, Finland, Germany, Japan, Korea, the Netherlands, Sweden, the +UK and the USA, with a liaison relationship to LDBC — the same LDBC +whose benchmark is [reading-ldbc-snb.md](reading-ldbc-snb.md) (§1, +p.3). The benchmark council and the language committee are the same +room. + So an engine that implements the shared MATCH grammar once (with Step 3's restrictors as first-class AST nodes) speaks to both worlds. -Why it matters: for the first time since 1987 there is an ISO answer -to "what query language should a graph engine target" — and it is -close enough to openCypher that M13 can target openCypher now and +Why it matters: for the first time since ISO 9075:1987 there is an ISO +answer to "what query language should a graph engine target" — and it +is close enough to openCypher that M13 can target openCypher now and converge later. ### Step 5 — fault line three: composability, from Gremlin to Datalog +> **In:** the question "can a query's output be another query's +> input?" +> **Out:** a ranking of the six languages, and the observation that +> the ranking predicts what an optimizer is allowed to touch. + **Composability** is whether a query's output can feed another query as a first-class input. The spectrum: **Gremlin** sits at the bottom — a traversal like `g.V().out().out()` *is* an execution order (pipelines compose, but every step names machine behavior). **Cypher** composes weakly — `CALL {}` subqueries were bolted on. **SQL/PGQ** -inherits SQL's full composability (MATCH returns a table). **Datalog** -is the ceiling: every rule's output is a relation usable by any other -rule, and recursion is native — a fixpoint (iterate rules until -nothing new derives; semi-naive evaluation only re-derives from the -newest facts — topic 27's incremental cousin) rather than a special -path operator. Why it matters: composability decides what the -*optimizer* may reorganize — which is Step 6 — and what users can -build without engine changes. +inherits SQL's full composability: the paper describes PGQ as +specifying "how to define graph views over an SQL tabular schema, and +to run **read-only** queries over such views, that can be projected by +an SQL SELECT statement" (§1, p.2) — note *read-only*, which is +precisely the capability gap GQL was chartered to fill ("full CRUD … +and catalog capability"). **Datalog** is the ceiling: every rule's +output is a relation usable by any other rule, and recursion is native +— a fixpoint (iterate rules until nothing new derives; semi-naive +evaluation only re-derives from the newest facts — topic 27's +incremental cousin) rather than a special path operator. Why it +matters: composability decides what the *optimizer* may reorganize — +which is Step 6 — and what users can build without engine changes. ### Step 6 — what each language lets the planner do +> **In:** the three fault lines. +> **Out:** one table, whose rightmost columns are the only thing that +> shows up in a flame graph. + The fault lines land in one place: how much freedom the planner has. | | model | matching | composable? | pushdown-friendly? | |---|---|---|---|---| -| Cypher/openCypher | property graph | homomorphism, rel-trail for var-length | weak (`CALL {}` bolted on) | good | -| GQL (ISO 39075:2024) | property graph | configurable: ALL/TRAIL/ACYCLIC + quantified path patterns | graph tables | good | -| SQL/PGQ | property graph *inside* SQL | GQL's MATCH in `GRAPH_TABLE(...)` | full SQL | inherits SQL | -| SPARQL | RDF triples | homomorphism (BGP) | subqueries | union-heavy plans | +| Cypher/openCypher | property graph | homomorphism on nodes, no-repeated-relationship on edges | weak (`CALL {}` bolted on) | good | +| GQL (ISO/IEC 39075:2024) | property graph | per-path-pattern: TRAIL / ACYCLIC / SIMPLE + 6 selectors + quantified path patterns; cross-pattern isomorphism is still an LO | graph tables, full CRUD | good | +| SQL/PGQ (ISO/IEC 9075-16) | property graph *view over* an SQL schema | the same GPML in `GRAPH_TABLE(...)` | full SQL, but read-only over the graph view | inherits SQL | +| SPARQL 1.1 (W3C) | RDF triples | homomorphism (BGP) | subqueries | union-heavy plans | | Gremlin | property graph | imperative traversal | pipelines | almost none — you ARE the plan | | Datalog | relations | homomorphism + fixpoint | **total** — rules feed rules | recursion-native | @@ -148,17 +346,51 @@ the query says what, not how. ## How to read the papers (with the concepts in hand) 1. **Deutsch et al., SIGMOD'22 (GQL and SQL/PGQ pattern matching)** — - the core is Step 2's semantics-as-parameter argument and Step 3's - restrictor/selector taxonomy; read those sections carefully, skim - the formal grammar. Keep the triangle example in hand and re-derive - the three counts as you read. -2. **G-CORE (SIGMOD'18)** — skim as history: the research consensus - (paths as first-class values, graph-to-graph composability) that - GQL absorbed — Step 4's middle arrow. -3. **kuzu's `Cypher.g4`** — not a paper, but read it like one: find - where variable-length patterns (`[*1..5]`) live in the grammar, - then ask where a GQL restrictor field would attach — that's - question 6. + §1 for the standards history and the WG3/GPML structure, §4 for + quantifiers, **§5.1 for restrictors and selectors** (Figures 7 and + 8 are the two tables to memorize), §5.2 for prefilters versus + postfilters, §7.1 for the Language Opportunities — i.e. what is + *not* in the standard. Keep the K3 example from Step 2 in hand and + re-derive 24 / 6 / 0 as you read. +2. **G-CORE (SIGMOD'18, + [arXiv:1712.01550](https://arxiv.org/abs/1712.01550))** — skim as + history: the research consensus (paths as first-class values, + graph-to-graph composability) that GQL absorbed — Step 4's middle + arrow. +3. **kuzu's `src/antlr4/Cypher.g4`** — not a paper, but read it like + one. **Correction:** it is **917 lines**, not the 690 the previous + version claimed. Go straight to the relationship rules, because + kuzu has already done question 6's exercise: + +```antlr +// src/antlr4/Cypher.g4 (kuzu @ 89f0263) + 413 oC_RelationshipDetail + 414 : '[' SP? ( oC_Variable SP? )? ( oC_RelationshipTypes SP? )? ( kU_RecursiveDetail SP? )? ( kU_Properties SP? )? ']' ; + // ... 415-427: elided — properties, rel types, node labels ... + 428 kU_RecursiveDetail + 429 : '*' ( SP? kU_RecursiveType)? ( SP? oC_RangeLiteral )? ( SP? kU_RecursiveComprehension )? ; + 430 + 431 kU_RecursiveType + 432 : (ALL SP)? WSHORTEST SP? '(' SP? oC_PropertyKeyName SP? ')' + 433 | SHORTEST + 434 | ALL SP SHORTEST + 435 | TRAIL + 436 | ACYCLIC ; + 437 + 438 oC_RangeLiteral + 439 : oC_LowerBound? SP? DOTDOT SP? oC_UpperBound? + 440 | oC_IntegerLiteral ; +``` + + Read `kU_RecursiveType` against Figures 7 and 8 and two facts fall + out. kuzu implements two of the three restrictors (`TRAIL`, + `ACYCLIC` — no `SIMPLE`) and three selector-shaped modes + (`SHORTEST`, `ALL SHORTEST`, weighted `WSHORTEST`). And it puts + them in **one alternation**, so exactly one may be chosen — whereas + GPML's §5.1 explicitly allows `ALL SHORTEST TRAIL`, a selector + applied after a restrictor. That single `|` is the gap between a + Cypher-shaped AST and a GQL-shaped one, and it is what question 6 + is asking you to design away. ## Questions @@ -184,23 +416,148 @@ the query says what, not how. ## Done when -- [ ] You can state the three matching semantics and count the 2-paths in a triangle under each. -- [ ] You can explain what GQL's restrictors and selectors make explicit that Cypher left implicit. +Answer each before unfolding it. + +- [ ] You can state the three matching semantics and count the matches of a 3-edge pattern on a triangle under each. + +
Answer + + Homomorphism (anything repeats), node-isomorphism (no repeated + node), trail (no repeated edge). On K3 with + `(a)-[e1]-(b)-[e2]-(c)-[e3]-(d)`: + + - homomorphism 3 × 2 × 2 × 2 = **24**, cross-checked as 1ᵀA³1 with + A = J − I whose leading eigenvalue is 2, so 2³ × 3 = 24 + - trail: three distinct edges in a 3-edge graph means all of them, + i.e. walking the triangle — 3 starts × 2 directions = **6** + - node-isomorphism: the pattern needs 4 distinct nodes and the graph + has 3 → **0** + + Do not try this with a 2-edge pattern: on a simple graph, distinct + edges force distinct endpoints, so trail and node-isomorphism agree + and the demonstration collapses. + +
+ +- [ ] You can explain what GQL's restrictors and selectors make explicit that Cypher left implicit — and the asymmetry between them. + +
Answer + + Cypher's semantics is a hard-coded hybrid (homomorphism on nodes, + no-repeated-relationship on edges) with no syntax to change it. + GPML makes it a keyword, because it has to: §5 requires every + unbounded quantifier to sit inside a restrictor or a selector or + both, otherwise the query need not terminate. + + Restrictors (Fig. 7): `TRAIL`, `ACYCLIC`, `SIMPLE` — three, and no + `ALL`. Selectors (Fig. 8): `ANY SHORTEST`, `ALL SHORTEST`, `ANY`, + `ANY k`, `SHORTEST k`, `SHORTEST k GROUP`. Restrictors act during + matching, selectors after; combined, selectors apply last. + + The asymmetry (§5.1, p.19): adding a *selector* to a query that has + matches always leaves at least one match; adding a *restrictor* can + leave none. A selector projects, a restrictor filters. + + What is still not expressible: cross-pattern isomorphic match modes + — §7.1 lists them as a Language Opportunity, deferred. + +
+ - [ ] You can say what property graphs and RDF actually disagree about, beyond syntax. + +
Answer + + Whether an edge is a first-class object that can carry properties. A + triple is atomic, so `since: 2019` on `:alice :knows :bob` needs + reification — an `rdf:Statement` node plus `rdf:subject`, + `rdf:predicate`, `rdf:object` and the property, i.e. five extra + triples and four self-joins to walk one edge — or RDF-star's + edge-triples. + + The downstream effect is plan shape: SPARQL's basic graph patterns + are triple-at-a-time so plans become many small self-joins, where a + property-graph MATCH becomes a couple of expands. The paper frames + the split institutionally too: RDF/SPARQL had a W3C standard while + property graph systems had "disparate storage models and querying + facilities" until GQL (§1, p.2). + +
+ - [ ] You can name, for each language, one thing its semantics lets the planner do that another's forbids. + +
Answer + + Cypher/GQL/PGQ are declarative, so the planner owns join order, + expansion direction and index choice — which is what makes kuzu's + WCOJ `Intersect` a legal substitution for a binary-join plan. + Gremlin's traversal *is* the plan, so an optimizer can only + peephole. SPARQL exposes triple patterns, so the planner reorders + self-joins but cannot see an "expand" at all. Datalog exposes the + fixpoint itself, so magic sets and demand transformation can rewrite + *through* recursion — nothing else in the table can. + + SQL/PGQ inherits SQL's full composability but the graph view is + read-only (§1, p.2), which is the gap GQL's full-CRUD charter fills. + +
+ - [ ] You wrote answers to all questions in notes.md, including this topic's 2-hop query written in more than one language. +
Answer + + Question 6's shape falls out of reading `Cypher.g4:431-436` against + Figures 7 and 8. kuzu makes path mode a single alternation — + `SHORTEST | ALL SHORTEST | TRAIL | ACYCLIC | WSHORTEST(prop)` — so + at most one may be given. GPML separates the two axes and allows + `ALL SHORTEST TRAIL`. A GQL-shaped AST therefore needs *two* + optional fields, not one enum: + + ``` + struct PathPattern { + restrictor: Option, // Trail | Acyclic | Simple + selector: Option, // AnyShortest | AllShortest | Any + // | AnyK(n) | ShortestK(n) + // | ShortestKGroup(n) + quantifier: Quantifier, // {lo,hi} — covers Cypher's [*1..5] + ... + } + ``` + + and the evaluator must apply the restrictor during matching and the + selector afterwards, in that order, or the "adding a selector never + empties the result" property breaks. + +
+ ## References **Papers** - Deutsch et al. — "Graph Pattern Matching in GQL and SQL/PGQ" (SIGMOD 2022, [arXiv:2112.06217](https://arxiv.org/abs/2112.06217)) - — the matching-semantics-as-parameter argument + — the authority for everything in Steps 3 and 4. §1 standards + history and the WG3/GPML policy; §4 quantifiers; §5 termination; + §5.1 restrictors (Fig. 7) and selectors (Fig. 8); §5.2 prefilters + and postfilters; §7.1 Language Opportunities. Figure 10's timeline + is a *projection* made in December 2021 — check it against what + actually shipped - Angles et al. — "G-CORE: A Core for Future Graph Query Languages" (SIGMOD 2018, [arXiv:1712.01550](https://arxiv.org/abs/1712.01550)) — the research consensus GQL absorbed -- GQL overview at [gqlstandards.org](https://www.gqlstandards.org) -**Code** -- [kuzu](https://github.com/kuzudb/kuzu) `src/antlr4/Cypher.g4` — a - full Cypher grammar in one 690-line file; keep it open while reading +**Standards** (designations checked at iso.org) +- ISO 9075:1987 — the first ISO edition of SQL; the baseline the "first + new ISO database language since SQL" claim is measured from +- ISO/IEC 39075:2024 — *Information technology — Database languages — + GQL* +- ISO/IEC 9075-16 — *Information technology — Database languages SQL — + Part 16: Property Graph Queries (SQL/PGQ)* + +**Code** (verified at kuzu `89f0263`) + +| File | Lines | What | +|---|---|---| +| `src/antlr4/Cypher.g4` | 917 total | a full Cypher grammar in one file | +| `src/antlr4/Cypher.g4` | 413-414 | `oC_RelationshipDetail` — where `[...]` is parsed | +| `src/antlr4/Cypher.g4` | 428-429 | `kU_RecursiveDetail` — `*`, path mode, range | +| `src/antlr4/Cypher.g4` | 431-436 | `kU_RecursiveType` — TRAIL/ACYCLIC/SHORTEST as one alternation | +| `src/antlr4/Cypher.g4` | 438-440 | `oC_RangeLiteral` — Cypher's `[*1..5]` bounds | diff --git a/topics/13-graph-engines/reading-wcoj.md b/topics/13-graph-engines/reading-wcoj.md index 235f8ce..c7a138d 100644 --- a/topics/13-graph-engines/reading-wcoj.md +++ b/topics/13-graph-engines/reading-wcoj.md @@ -11,6 +11,12 @@ it. Pure paper material — the code anchors are kuzu's Intersect operator ([reading-kuzu.md](reading-kuzu.md)) and FalkorDB's masked matrix multiply ([reading-graphblas-internals.md](reading-graphblas-internals.md)). +Every claim below is cited to a section, lemma or theorem of a paper +that is linked in the References and was read to check it. Two things +the previous version of this chapter got loose — who proved which half +of "the AGM bound", and what the bound is a bound *on* — are corrected +in place and flagged. + ## The problem in one sentence Counting triangles on a 16M-edge graph with pairwise joins can @@ -22,12 +28,16 @@ build a two-edge intermediate the third edge would have filtered. ### Step 1 — the triangle query breaks every pairwise plan +> **In:** the triangle query `Q(a,b,c) = R(a,b) ⋈ S(b,c) ⋈ T(a,c)` over +> a graph with m edges, and any plan built from two-relation joins. +> **Out:** a lower bound on what such a plan must materialize — and the +> conclusion that the join *order* is not the free variable. + A **binary (pairwise) join plan** combines relations two at a time — join R with S, then join the result with T — which is how every relational optimizer since System R builds plans. On the triangle -query `Q(a,b,c) = R(a,b) ⋈ S(b,c) ⋈ T(a,c)` (each relation the same m -edges), any pairwise plan must first materialize a two-relation -intermediate: +query (each relation the same m edges), any pairwise plan must first +materialize a two-relation intermediate: ``` R ⋈ S → all paths a->b->c → can be Θ(m²) rows @@ -35,38 +45,136 @@ intermediate: …then filter by T → output was ≤ m^1.5 all along ``` -The star graph is the killer: a hub with degree 1M makes R ⋈ S produce -10¹² two-edge paths, of which the final result keeps a vanishing -fraction. Why it matters: the waste is *structural* — the plan commits -to enumerating pairs before the third relation gets a say — and topic -10's optimizer is innocent; reordering the joins just picks which Θ(m²) -intermediate to build. +The survey states the gap as a settled fact rather than an +observation: + +> "A first bound is to say that there are at most N edges, and hence +> at most O(N³) triangles. A bit more thought suggests that every +> triangle is indexed by any two of its sides and hence there at most +> O(N²) triangles. However, the correct, tight, and non-trivial +> asymptotic is O(N^{3/2}). … In contrast, traditional databases +> evaluate joins pairwise, and as has been noted by several authors, +> this forces them to run in time Ω(N²) on some instance of the +> triangle query." +> — Ngo, Ré & Rudra, *Skew Strikes Back*, §1, p.1 + +Note the shape of that Ω(N²): it is a lower bound on *some instance*, +not on every one. The instance is a star. And this topic's own graph +is a mild version of one — count the two-edge paths it contains, which +is exactly what an `R ⋈ S` intermediate enumerates: + +``` + two-edge paths through a node v = deg(v)² (in × out, here both = deg) + this graph (notes.md): 1e6 nodes, 16.0e6 directed edges, + p50 degree 11, max degree 6 565 + + through the median node: 11² = 121 + through the max node: 6 565² = 43 099 225 + ratio 356 192× +``` + +One node out of a million contributes 43 M intermediate rows on its +own. That is the same fact the topic headline measures from the other +end — the 101× two-hop slowdown from supernodes — and it is why the +waste is *structural*: the plan commits to enumerating pairs before +the third relation gets a say. Topic 10's optimizer is innocent; +reordering the joins just picks which Θ(m²) intermediate to build. ### Step 2 — the AGM bound: how big can the output actually be? -The **AGM bound** (Atserias–Grohe–Marx) gives the maximum possible -output size of a join query as a product of relation sizes raised to a -**fractional edge cover** — an assignment of weights to relations such -that every variable is "covered" by total weight ≥ 1 across the -relations containing it. For the triangle, weights (½, ½, ½) cover -each of a, b, c (each variable appears in two relations, ½ + ½ = 1), -giving: +> **In:** a join query's hypergraph and the sizes |R| of its relations. +> **Out:** a provable ceiling on |Q(D)| for *every* database instance +> D, obtained by solving a small linear program — plus a matching +> instance proving the ceiling is not slack. + +The **AGM bound** gives the maximum possible output size of a join +query as a product of relation sizes raised to a **fractional edge +cover**. The cover is not folklore; it is the feasible set of an +explicit LP, and the fractional edge cover number ρ*(Q) is its +optimum: + +``` + LQ : minimise Σ_R x_R + subject to Σ_{R : a ∈ A_R} x_R ≥ 1 for every attribute a + x_R ≥ 0 for every relation R +``` +— Atserias, Grohe & Marx, §3.1, linear program (3.1) + +The bound itself: + +> **Lemma 2 ([10]).** Let Q be a join query with schema σ and let D be +> a σ-instance. Then for every fractional edge cover (x_R : R ∈ σ) of +> Q we have |Q(D)| ≤ ∏_{R∈σ} |R(D)|^{x_R}. +> — AGM §3.1 + +**Correction — attribution.** The previous version of this chapter +attributed the bound to "AGM (Atserias–Grohe–Marx)" without +qualification. The upper bound is not theirs: AGM's own paper cites it +as Lemma 2 **[10]**, i.e. Grohe & Marx, *Constraint solving via +fractional edge covers* (SODA 2006), and reproves it via Shearer's +lemma (AGM §3.1, "The proof of Lemma 2 is based on a combinatorial +lemma known as Shearer's lemma"). AGM's contribution is the *matching +lower bound*: + +> **Lemma 4.** … for every N₀ ∈ ℕ there is a σ-instance D such that +> |D| ≥ N₀ and |Q(D)| ≥ ∏_{R∈σ} |R(D)|^{x_R}. +> — AGM §3.1, proved by LP duality against the dual program (3.2) + +So "the AGM bound" names the *pair*: the Grohe–Marx ceiling plus the +AGM instance that reaches it. That is what makes it a target worth +building an algorithm to — a bound with slack would not be. The +survey's history agrees and reaches back further, to Friedgut–Kahn +(1990s), and to the Loomis–Whitney inequality of the 1940s +(*Skew Strikes Back* §1, p.1). + +For the triangle, solve the LP by hand. Each of a, b, c appears in +exactly two of the three relations, so x = (½, ½, ½) is feasible +(½ + ½ = 1 for each variable) with cost 3/2: ``` |Q| ≤ |R|^½ · |S|^½ · |T|^½ = m^(3/2) + + m = 16.0e6: + AGM ceiling m^1.5 = 16.0e6 × √(16.0e6) = 16.0e6 × 4 000 = 6.4e10 + pairwise plan m² = 2.56e14 + gap m² / m^1.5 = √m = 4 000× ``` -For m = 16M ≈ 2²⁴: output ≤ 2³⁶ ≈ 64G in theory, but the point is the -*gap* — binary plans can produce Θ(m²) = 2⁴⁸ intermediates, √m ≈ 4000× -above the bound. Why it matters: the bound is a target — an algorithm -whose runtime is O(AGM bound) is **worst-case optimal**, and Step 1 -proved no pairwise plan can be. +**Correction — what the bound bounds.** The previous version wrote +"output ≤ 2³⁶ ≈ 64G in theory". That is right as an arithmetic +statement but easy to misread: `m^1.5` is a *worst-case* ceiling over +all instances with m edges, not an estimate of this graph's triangle +count, which is far smaller. The number that matters is the ratio, not +either endpoint. + +The gap is also not universal. Do the same LP for question 2's +4-cycle `R(a,b) S(b,c) T(c,d) U(d,a)`: take x_R = x_T = 1 and +x_S = x_U = 0 — a covers via R, b via R, c via T, d via T — for cost +2, so |Q| ≤ m². And the dual (3.2) certifies that 2 is optimal: set +y_a = y_c = 1, y_b = y_d = 0, and every relation's constraint +Σ_{a∈A_R} y_a ≤ 1 holds with equality, giving dual value 2 = ρ*. On +the 4-cycle the AGM ceiling *equals* the pairwise intermediate, so +worst-case optimality buys nothing asymptotically. Why it matters: the +bound is a target — an algorithm whose runtime is O(AGM bound) is +**worst-case optimal**, Step 1 proved no pairwise plan can be for the +triangle, and the LP is how you find out whether a given pattern is +one where that distinction pays. ### Step 3 — Generic Join: intersect one variable at a time +> **In:** the relations, pre-indexed consistently with one global +> attribute order. +> **Out:** the query answer, in time proportional to the AGM bound — +> by binding one *variable* at a time via intersection, never +> materializing a pair a later relation would kill. + **Generic Join** meets the AGM bound by changing the unit of work from -"join two relations" to "bind one *variable*, by intersecting -everything known about it": +"join two relations" to "bind one variable, by intersecting everything +known about it". The survey gives it as Algorithm 3 (§4.2, p.14): if +the query has one variable left, return `⋂_{F∈E} R_F`; otherwise split +the variables into I and J, recurse on the projection onto I, and for +each tuple t_I recurse on the residual relations `R_F ⋈ t_I`. For the +triangle, unrolled with I taken one variable at a time: ``` for a in R.a ∩ T.a: # values for variable a @@ -75,30 +183,75 @@ everything known about it": emit (a,b,c) ``` -For the triangle this runs in O(m^1.5) — worst-case optimal. The whole -trick in one line: never enumerate (a,b,c-candidate) pairs that a -later relation kills; **intersect FIRST**. The data-structure -requirement: each relation must be accessible sorted or hashed by any -prefix of the variable order — which for graphs means sorted adjacency -= CSR slices (compressed sparse row — offsets array + sorted neighbors -array), exactly what kuzu's build side guarantees. Why it matters: -this is a different *operator set*, not a smarter plan — the fix lives -below the optimizer. +Two lines of the survey's analysis are the whole reason this works, +and they are worth memorizing because they reappear as code in Step 4: + +> "Given the indices, when |V| = 1 computing ⋂_{F∈E} R_F can easily be +> done in time Õ(m · min |R_F|) = Õ(m · ∏_{F∈E} |R_F|^{x_F})." +> — *Skew Strikes Back* §4.2 + +**min**, not sum, not max. The base case is charged to the *smallest* +participating list — so the intersection kernel must never do work +proportional to the big side. That single word is the specification +that kuzu's `swapSmallestListToFront` +(`intersect.cpp:103-118`, [reading-kuzu.md](reading-kuzu.md) Step 4) +implements, and the property EmptyHeaded names the "min property" +(§1). Overall the algorithm runs in Õ(m·n·∏|R_F|^{x_F}), where Õ +hides a log factor of the input size — for the triangle, Õ(m^1.5). + +The data-structure requirement is the other half: + +> "Both NPRR and Leapfrog Triejoin algorithms do this by fixing a +> global attribute order and build a B-tree-like index structure for +> each input relation consistent with this global attribute order. +> NPRR also described an hash-based indexing structure so as to remove +> a log-factor from the final run time." +> — *Skew Strikes Back* §4.2 + +For graphs that means sorted adjacency = CSR slices (compressed sparse +row — offsets array + sorted neighbors array), exactly what kuzu's +build side guarantees with one overridden method. Note the honest +caveat about the log factor: Veldhuizen's own abstract says leapfrog +triejoin is worst-case optimal "**up to a log factor**, in the sense +of NPRR", and it exhibits a class of instances where LFTJ runs in +O(n log n) while NPRR runs in Θ(n^1.375) — the two algorithms are not +ordered, they are optimal against different granularities of +constraint. Why it matters: this is a different *operator set*, not a +smarter plan — the fix lives below the optimizer. ### Step 4 — the intersection kernel: merge vs galloping -Everything now reduces to intersecting two sorted lists of sizes -d1 ≤ d2, and there are two algorithms: **merge** (walk both in -lockstep, O(d1+d2)) and **galloping** (for each element of the small -list, exponentially probe then binary-search the big list, -O(d1 log d2)). Galloping wins when d1 ≪ d2 — intersect a degree-20 -node with a degree-100K supernode: merge does ~100K steps, galloping -~20 × 17 ≈ 340: +> **In:** two sorted lists of node ids with sizes d1 ≤ d2. +> **Out:** their intersection, and a rule for which of two algorithms +> to use — the rule that decides whether Step 3's asymptotics survive +> contact with the machine. + +Everything now reduces to intersecting two sorted lists, and there are +two algorithms: **merge** (walk both in lockstep, O(d1+d2)) and +**galloping** (for each element of the small list, exponentially probe +then binary-search the big list, O(d1 log d2)). Only the second one +satisfies Step 3's `min` requirement. Price both on this topic's +actual degree distribution — a median node meeting the supernode: + +``` + d1 = 11 (p50 degree, notes.md) + d2 = 6 565 (max degree, notes.md) + + merge: d1 + d2 = 11 + 6 565 = 6 576 steps + galloping: d1 · log2(d2) = 11 × 12.68 ≈ 140 steps + ratio ≈ 47× + + and when the two sides are the same size (d1 = d2 = 11): + merge: 22 steps galloping: 11 × 3.46 ≈ 38 steps → merge wins +``` + +The crossover is real, which is why nobody ships only one kernel. ```rust -// the inner kernel of every WCOJ engine: sorted-set intersection. -// galloping wins when d1 ≪ d2 — on power-law graphs (leaf ∩ supernode) -// that's the common case, and skew is exactly what WCOJ defends against +// ILLUSTRATION — not from any pinned repo. The production version of +// this decision is kuzu's Intersect operator, whose sorted-merge kernel +// is src/processor/operator/intersect/intersect.cpp:65-90 and whose +// smallest-list-first heuristic is intersect.cpp:103-118. fn intersect(small: &[u32], big: &[u32], out: &mut Vec) { let mut lo = 0; for &x in small { // O(d1 log d2) @@ -114,26 +267,74 @@ fn intersect(small: &[u32], big: &[u32], out: &mut Vec) { ``` On power-law graphs (leaf ∩ supernode) the skewed case IS the common -case — fitting, since skew is exactly what WCOJ defends against -("Skew Strikes Back" is the survey's title for a reason). Why it -matters: the asymptotics of Step 3 are delivered or squandered right -here, in the inner loop. +case — fitting, since skew is exactly what WCOJ defends against. The +survey is explicit that this is the whole story, not a side remark: + +> "Connections of join size to arcane geometric bounds may reasonably +> lead a practitioner to believe that the cause of suboptimality is a +> mysterious force wholly unknown to them—but it is not; it is the old +> enemy of the database optimizer, skew." +> — *Skew Strikes Back* §1, p.2 + +Why it matters: the asymptotics of Step 3 are delivered or squandered +right here, in the inner loop. ### Step 5 — EmptyHeaded: the kernel must be hardware-conscious +> **In:** neighborhood sets of wildly varying density on one machine +> with a fixed SIMD width. +> **Out:** a *choice of representation per set* — and a measured +> answer to how much that choice is worth. + EmptyHeaded compiled whole queries down to set intersections over a -trie/CSR-like layout and chose the intersection *representation* by -density: sorted uint arrays for sparse sets, bitsets for dense ones — -SIMD both ways (topic 17 preview). Its lesson: WCOJ is only fast if -the intersection kernel is hardware-conscious; **the asymptotics get -you in the door, bandwidth wins the fight**. A bitset intersection of -two dense neighborhoods is 64 comparisons per cycle-ish; a scalar -merge is 1. Why it matters: this is the topic-0 discipline applied to -a theory result — a 4000× asymptotic win can still lose to a 50× -constant-factor loss if the kernel ignores the machine. +trie/CSR-like layout. Its first measurement is the one that justifies +the rest of the paper: + +> "For common graph queries over real data, we found that set +> intersection typically accounts for over 95% of the overall +> runtime." +> — Aberger et al., *EmptyHeaded*, §1, p.1-2 + +So it chose the intersection *representation* by density — defined as +"the cardinality of the set divided by its range" (§1) — using `uint` +arrays for sparse sets and a two-level `bitset` (blocks of, "say, 128 +bits", each a bitvector) for dense ones, SIMD both ways (topic 17 +preview). And it makes that choice at **three granularities**: graph +level, set level, and block level. The payoff for descending from the +first to the second is measured, and it is enormous where the data is +skewed and small where it is not: + +``` + set-level vs graph-level representation choice (EmptyHeaded §1): + Google+ (highly skewed): 13.4× + LiveJournal (sparse): 1.6× + ratio between the two datasets: 8.4× +``` + +Their optimizer lands within 2× of an infeasible oracle. The hardware +it was written for: "the current Intel Ivy Bridge architecture +supports CPUs with 12 cores and a SIMD register width of 256 which +execute a staggering 14.7 trillion bitwise comparisons per second when +running at 2.4GHz" (§1) — a 2015 number, quoted here because the +*ratio* it implies is the lesson: a bitset intersection moves 256 bits +per instruction where a scalar merge moves one comparison. + +Its lesson: WCOJ is only fast if the intersection kernel is +hardware-conscious; **the asymptotics get you in the door, bandwidth +wins the fight**. Why it matters: this is the topic-0 discipline +applied to a theory result — a 4000× asymptotic win can still lose to +a constant-factor loss if the kernel ignores the machine, and +EmptyHeaded's own 13.4× says exactly how large that constant can get +on one representation decision. ### Step 6 — the matrix spelling: `C = A²` is Generic Join +> **In:** the adjacency matrix A and the mask mechanism from the +> GraphBLAS chapter. +> **Out:** the observation that a masked SpGEMM computes precisely +> Step 3's innermost intersection — the same algorithm, arrived at +> from linear algebra instead of from relational theory. + FalkorDB never wrote an Intersect operator — because masked matrix multiply already is one. `C = A²` (compute A², but only at positions where the mask A has an edge — the mask mechanism from @@ -149,6 +350,17 @@ Same algorithm, three syntaxes: GraphBLAS: C = A·A with a PAIR/AND semiring ``` +The equivalence is only real if the mask is applied *early*, and +GraphBLAS has a specific method for that: `GB_AxB_dot3`, whose work is +Ω(nnz(M)) — proportional to the mask's nonzeros, i.e. to m, not to the +m² of the unmasked product (`Source/mxm/GB_AxB_dot.c:21-26`; see +[reading-graphblas-internals.md](reading-graphblas-internals.md) for +how dot3 gets selected and for the late-masking path that would +silently forfeit the whole argument). EmptyHeaded's abstract makes the +same identification from its side, calling out "the link between +general-purpose worst-case-optimal join algorithms and Boolean +algebra" (§1). + Why it matters: this equivalence is the deepest tie in the topic — the relational world's WCOJ literature and the linear-algebra world's masked-SpGEMM literature converged on the same computation from @@ -157,14 +369,26 @@ optimality without ever naming it. ## How to read the papers (with the concepts in hand) +| Step | Where to read it | +|---|---| +| 1 | *Skew Strikes Back* §1 p.1 (the N³ → N² → N^{3/2} paragraph) and §2 | +| 2 | AGM §3.1 — LP (3.1), Lemma 2 (upper, ← Grohe–Marx), Lemma 4 (lower, AGM's) | +| 2 | AGM §1 Theorem 1 for the four equivalent characterisations; §3.2 Theorems 6 and 7 for why join-project plans work and join-only plans do not | +| 3 | *Skew Strikes Back* §4.2, Algorithm 3 and the `Õ(m · min |R_F|)` base case | +| 3 | Veldhuizen, *Leapfrog Triejoin*, abstract + §1 for the "up to a log factor" caveat and the O(n log n) vs Θ(n^{1.375}) separation from NPRR | +| 4 | *Skew Strikes Back* §1 p.2 ("it is the old enemy … skew") and EmptyHeaded §1 on the min property | +| 5 | EmptyHeaded §1 (95% of runtime; the three granularities; 13.4× vs 1.6×) then its layout section | +| 6 | [reading-graphblas-internals.md](reading-graphblas-internals.md) on dot3 and masking, then [reading-kuzu.md](reading-kuzu.md) on `Intersect` | + 1. **Ngo, Ré, Rudra — "Skew Strikes Back" (SIGMOD Record 2013)** — read THIS one, it's the readable survey. The triangle example is Steps 1–2; Generic Join is Step 3. Work their skew discussion against Step 4 — skew is both the villain (kills binary plans) and the reason galloping wins. -2. **AGM (FOCS 2008)** — dip in only for the fractional edge cover - definition and the bound statement (Step 2); the proofs are - optional. Try computing the cover for a 4-cycle (question 2). +2. **AGM (FOCS 2008; SICOMP version)** — dip in only for the LP (3.1), + Lemma 2 and Lemma 4 (Step 2); the proofs are optional, but read + enough of Lemma 4's opening to see LP duality doing the work, since + you need the dual anyway for question 2's 4-cycle. 3. **EmptyHeaded (SIGMOD 2016)** — read the layout section and the density-adaptive intersection (Step 5); skim the compiler machinery. Compare their array-vs-bitset crossover against your @@ -191,28 +415,151 @@ optimality without ever naming it. ## Done when +Answer each before unfolding it. + - [ ] You can explain why every pairwise plan loses on the triangle query, using intermediate sizes rather than intuition. -- [ ] You can state the AGM bound and compute the fractional edge cover for the triangle. -- [ ] You can narrate Generic Join as one variable at a time, and say where the intersections happen. -- [ ] You can say when galloping beats a merge intersection, in terms of the two list lengths. + +
Answer + + Any pairwise plan must build a two-relation intermediate first, and + for the triangle every such intermediate is a set of two-edge paths. + A node of degree d contributes d² of them, so on a star the + intermediate is Θ(m²) while the output is O(m^1.5) — *Skew Strikes + Back* §1 p.1 states the Ω(N²) lower bound for pairwise evaluation + and the tight O(N^{3/2}) output asymptotic side by side. + + Reordering does not help: symmetry means every choice of "which two + first" produces the same shape of intermediate. On this topic's + graph the max-degree node alone contributes 6 565² = 43 099 225 + intermediate rows. + +
+ +- [ ] You can state the AGM bound, compute the fractional edge cover for the triangle, and say which half of it is due to whom. + +
Answer + + |Q(D)| ≤ ∏_R |R(D)|^{x_R} for any feasible solution x of the LP + (3.1): minimise Σ x_R subject to Σ_{R ∋ a} x_R ≥ 1 for every + attribute a. For the triangle each variable sits in two relations, + so (½,½,½) is feasible with cost ρ* = 3/2 and the bound is m^1.5. + + Attribution: the upper bound is AGM's **Lemma 2 [10]** — Grohe & + Marx, SODA 2006, reproved via Shearer's lemma. AGM's own result is + **Lemma 4**, the matching lower bound: an instance exists that + attains the product, constructed by LP duality against (3.2). "The + AGM bound" is shorthand for the pair. + +
+ +- [ ] You can narrate Generic Join as one variable at a time, say where the intersections happen, and quote its base-case cost. + +
Answer + + Algorithm 3 of *Skew Strikes Back* §4.2: with one variable left, + return the intersection of all relations; otherwise pick a variable + subset I, recurse on the projections onto I, and for every tuple + t_I recurse on the relations semi-joined with t_I. Unrolled for the + triangle: intersect for a, then for b given a, then — the one that + matters — c ∈ S[b].c ∩ T[a].c. + + Base case cost: Õ(m · **min** |R_F|). "min" is the whole point: the + work is charged to the smallest list, which is what forces + galloping in Step 4 and smallest-list-first ordering in kuzu. + Overall Õ(m·n·∏|R_F|^{x_F}), Õ hiding a log factor. + +
+ +- [ ] You can say when galloping beats a merge intersection, in terms of the two list lengths, with a worked number. + +
Answer + + Merge is O(d1+d2), galloping O(d1 log d2); galloping wins once + d2 ≫ d1 log d2. On this topic's graph, d1 = 11 (p50) against + d2 = 6 565 (max degree): merge 6 576 steps versus 11 × log2(6 565) + ≈ 140, a 47× difference. At d1 = d2 = 11 it inverts — 22 versus + ≈ 38 — which is why engines keep both kernels and pick per call. + + Power-law degree distributions make the skewed case the common one, + which is the answer to question 3. + +
+ - [ ] You can explain why `C = A²` is the same algorithm in matrix spelling — and connect it to the masked-SpMV lane here. + +
Answer + + Each entry of A² at position (a,b) is a dot product of A's row a + with A's column b, i.e. |N(a) ∩ N(b)| — Step 3's innermost + intersection. Restricting the computation to positions where the + mask A is nonzero means only existing edges (a,b) are ever + considered, which is intersect-first rather than + enumerate-then-filter. + + The mechanism is `GB_AxB_dot3`, whose work is Ω(nnz(M)) — the mask's + nonzeros, i.e. m — instead of the m² of an unmasked product + (`Source/mxm/GB_AxB_dot.c:21-26`). It only holds if the mask is + applied *early*; the late path through `GB_accum_mask`/`GB_masker` + computes the full product first and then discards, forfeiting the + argument entirely. + +
+ - [ ] You wrote answers to all questions in notes.md. +
Answer + + Question 2's 4-cycle: ρ* = 2, so |Q| ≤ m². Primal certificate + x_R = x_T = 1, x_S = x_U = 0 (a and b covered by R, c and d by T). + Dual certificate for optimality, using AGM's program (3.2): + y_a = y_c = 1, y_b = y_d = 0 satisfies Σ_{a ∈ A_R} y_a ≤ 1 with + equality on all four relations, value 2. + + The lesson is the one Step 2 draws: m² is also what a pairwise plan + materializes, so the 4-cycle has no polynomial gap and worst-case + optimality is not automatically a win. That is the "detectable + trigger" question 5 wants — cyclicity is necessary but the honest + test is ρ*(pattern) versus the intermediate size the best binary + plan would build. + +
+ ## References **Papers** - Atserias, Grohe, Marx — "Size Bounds and Query Plans for Relational - Joins" (FOCS 2008) — the AGM bound + Joins" (FOCS 2008; SICOMP, + [arXiv:1711.03860](https://arxiv.org/abs/1711.03860)) — §3.1 has LP + (3.1), Lemma 2 (upper bound, credited to Grohe–Marx SODA 2006) and + Lemma 4 (AGM's matching lower bound); §3.2 Theorems 6 and 7 for + join-project versus join-only plans - Ngo, Ré, Rudra — "Skew Strikes Back: New Developments in the Theory of Join Algorithms" (SIGMOD Record 2013, [arXiv:1310.3314](https://arxiv.org/abs/1310.3314)) — the readable - survey; read THIS one + survey; read THIS one. §1 p.1 for N^{3/2} and the Ω(N²) pairwise + lower bound, §1 p.2 for the skew thesis, §4.2 Algorithm 3 for + Generic Join and its `Õ(m · min |R_F|)` base case +- Ngo, Porat, Ré, Rudra — "Worst-case Optimal Join Algorithms" + (PODS 2012, [arXiv:1203.1952](https://arxiv.org/abs/1203.1952)) — + NPRR, the first algorithm to match the bound +- Veldhuizen — "Leapfrog Triejoin: A Simple, Worst-Case Optimal Join + Algorithm" ([arXiv:1210.0481](https://arxiv.org/abs/1210.0481)) — + worst-case optimal *up to a log factor*, implementable on ordinary + B-trees; the abstract's O(n log n) vs Θ(n^{1.375}) separation is + worth knowing before you assume the algorithms are ordered - Aberger et al. — "EmptyHeaded: A Relational Engine for Graph - Processing" (SIGMOD 2016) — the hardware-conscious intersection - kernels + Processing" (SIGMOD 2016, + [arXiv:1503.02368](https://arxiv.org/abs/1503.02368)) — the + hardware-conscious intersection kernels; §1 for the 95%-of-runtime + measurement, the uint/bitset choice at graph/set/block granularity, + and the 13.4× (Google+) vs 1.6× (LiveJournal) payoff **Code** -- No repo for this chapter — the code anchors are - [kuzu](https://github.com/kuzudb/kuzu)'s Intersect operator - ([reading-kuzu.md](reading-kuzu.md)) and FalkorDB's masked mxm - ([reading-graphblas-internals.md](reading-graphblas-internals.md)) +- No repo of its own for this chapter. The anchors live in the two + chapters this one binds together: + [kuzu](https://github.com/kuzudb/kuzu)'s `Intersect` operator — + `src/processor/operator/intersect/intersect.cpp:65-90` (sorted + merge) and `:103-118` (smallest list first), + [reading-kuzu.md](reading-kuzu.md) — and GraphBLAS's masked SpGEMM, + `Source/mxm/GB_AxB_dot.c:21-26` (dot3's Ω(nnz(M))), + [reading-graphblas-internals.md](reading-graphblas-internals.md) diff --git a/topics/14-vector-search/notes.md b/topics/14-vector-search/notes.md index dc46ce7..ca45340 100644 --- a/topics/14-vector-search/notes.md +++ b/topics/14-vector-search/notes.md @@ -27,8 +27,8 @@ before you write either. ## Predictions (fill BEFORE implementing hnsw.rs / quant.rs) -Baseline (provided, measured): brute force 185 QPS at recall 1.0 -(100K × 128-d f32 = 51 MB per scan, 500 queries in 2.70 s). +Baseline (provided, measured): brute force 117 QPS at recall 1.0 +(100K × 128-d f32 = 51 MB per scan, 500 queries in 4.28 s). | config | predicted recall@10 | predicted QPS | actual recall | actual QPS | |---|---|---|---|---| @@ -40,7 +40,7 @@ Baseline (provided, measured): brute force 185 QPS at recall 1.0 | question | prediction | actual | |---|---|---| -| hnsw build time for 100K (vs 2.7 s for one brute sweep) | | | +| hnsw build time for 100K (vs 4.28 s for one brute sweep) | | | | ef=16→256: how many × QPS lost for how much recall gained? | | | | u8 scan ×4: above or below the hnsw curve? (it's O(n) but 4× fewer bytes) | | | | max_level with m=16 on 100K points (ln n / ln m ≈ ?) | | | diff --git a/topics/14-vector-search/reading-diskann.md b/topics/14-vector-search/reading-diskann.md index 85f277f..22bac05 100644 --- a/topics/14-vector-search/reading-diskann.md +++ b/topics/14-vector-search/reading-diskann.md @@ -11,98 +11,236 @@ distances rank the results. This chapter assumes [reading-hnsw-paper.md](reading-hnsw-paper.md) (greedy graph search, beams, ef) and [reading-pq.md](reading-pq.md) (PQ codes, ADC). +**Three names, three things — do not blur them.** *Vamana* is the +graph-construction algorithm (§2 of the paper). *DiskANN* is the +SSD-resident system built on it (§3). *FreshDiskANN* is a **later, +separate paper** (Singh et al., 2021) about streaming inserts and +deletes; nothing in this chapter's source covers it, so if you find +yourself explaining how DiskANN handles updates, you have wandered +into a different paper. + +Every claim below cites Subramanya, Devvrit, Kadekodi, Krishnaswamy +& Simhadri, *"DiskANN: Fast Accurate Billion-point Nearest Neighbor +Search on a Single Node"*, NeurIPS 2019, by section, algorithm or +figure. There is **no DiskANN clone in `resources/codebases.md`**, so +unlike the qdrant and usearch chapters this one has no `file:line` +anchors — every number here is the paper's, and the paper is the only +thing being verified against. + ## The problem in one sentence A billion 128-d vectors need ~512 GB for the vectors plus ~100 GB for HNSW links — far beyond one machine's RAM — but naively paging -HNSW to SSD turns each query's ~200–500 hops into 2+ random 100 µs -reads apiece, i.e. **~50–100 ms per query**, 10–20× too slow. +HNSW to SSD turns each query's hundreds of hops into two dependent +random reads apiece, and §3.3's own figure for an SSD round trip is +*"few hundred microseconds"*. + +Make the strawman concrete before reading the fix: + +``` + vectors 1e9 × 128 × 4 B = 512 GB + HNSW links 1e9 × 151 B (reading-hnsw-paper.md, §4.2.3, M=16) + = 151 GB + --------------------------------------------------------------- + RAM needed 663 GB vs one machine's 64 GB → 10× over + + paged to SSD, per hop: 1 read for the vector + 1 for the links, + and the second cannot start until the first is parsed. + 2 × 200 µs × 300 hops = 120 ms/query +``` + +DiskANN's target for the same dataset, from the abstract: *"> 5000 +queries a second with < 3ms mean latency and 95%+ 1-recall@1 on a 16 +core machine"* — on a machine with 64 GB of RAM and two consumer +NVMe drives (§4: an HP z840, dual Xeon E5-2620v4, 16 cores, 2 × +Samsung 960 EVO in RAID-0). Note the metric: **1-recall@1**, the +fraction of queries whose *single* true nearest neighbour is +returned, which is a different and stricter quantity than the +recall@10 this topic's bench reports. ## The concepts, step by step ### Step 1 — why HNSW can't just go to disk -HNSW search is a beam of *dependent* point lookups: you can't know +> **In:** an HNSW index too large for RAM. **Out:** the one metric +> the redesign optimises — SSD round trips per query — and why the +> obvious paging strategy fails on it. + +HNSW search is a beam of *dependent* point lookups: you cannot know which node to read next until the current node's distances are -computed — topic 0's pointer chase, at SSD latency. On disk each hop -needs the node's vector AND its neighbor list, which in an -RAM-designed layout live in different places — two random reads per -hop: +computed — topic 0's pointer chase, at SSD latency rather than DRAM +latency. On disk each hop needs the node's vector AND its neighbour +list, which in a RAM-designed layout live in different places. ``` HNSW paged to SSD, per hop: - read vector block ~100 µs ┐ dependent — can't overlap - read links block ~100 µs ┘ - × ~300 hops/query ⇒ ~60 ms/query — dead on arrival + read vector block ~200 µs ┐ dependent — can't overlap + read links block ~200 µs ┘ + × ~300 hops/query ⇒ ~120 ms/query — dead on arrival ``` -DiskANN's redesign targets exactly the metric that matters: -**number of SSD round trips per query**. Every idea below either -removes reads (Steps 2–3) or overlaps them (Step 4). +The paper is explicit about the currency. §3.3 says a naive port +*"requires many rountrips to SSD (which take few hundred +microseconds) resulting in higher latencies"*, and the entire design +is organised around reducing them. It also gives the fact that makes +the fix possible: *"fetching a small number of random sectors from an +SSD takes almost the same time as one sector"* — SSDs have queue +depth, so **concurrent** reads are nearly free while **dependent** +ones are not. + +Every idea below either removes reads (Steps 2–3) or overlaps them +(Step 4). ### Step 2 — Vamana: a flat graph built for few hops -Vamana is DiskANN's graph: no hierarchy — one flat graph with degree -bound R (max links per node, ~64), built so greedy search converges -in few hops. The builder's pruning rule is **RobustPrune**: +> **In:** the requirement "few hops, no hierarchy". **Out:** +> RobustPrune's α parameter, the two-pass build, and an honest +> statement of what the α > 1 convergence result does and does not +> cover. + +**Vamana** is DiskANN's graph: no hierarchy — one flat graph with +degree bound `R`, built so greedy search converges in few hops. The +builder's pruning rule is **RobustPrune** (Algorithm 2), and the +α-slack line is the whole idea: ``` - RobustPrune(p, candidates, α, R): - while candidates and |out(p)| < R: - p* = closest remaining candidate; add edge p→p* - remove every c with α·d(p*, c) ≤ d(p, c) ← the α slack + RobustPrune(p, V, α, R): # Algorithm 2 + V ← (V ∪ N_out(p)) \ {p}; N_out(p) ← ∅ + while V ≠ ∅: + p* ← argmin_{p'∈V} d(p, p') # closest remaining + N_out(p) ← N_out(p) ∪ {p*} + if |N_out(p)| = R: break + for p' ∈ V: + if α · d(p*, p') ≤ d(p, p'): remove p' from V ← the α slack ``` -α = 1 gives HNSW's Alg-4-style directional pruning (one edge per -direction). The new move: **α > 1 (≈1.2) keeps LONGER edges** — a -candidate is only pruned if the kept edge gets you *α times* closer -to it, so surviving edges shrink the distance to any target -geometrically. Each greedy hop must cut the remaining distance by -≥ α, so hop count is O(log_α of the distance ratio) — the graph -trades extra degree for provably fewer hops. - -Build: two passes over random-order points (second pass with final -α), each pass: greedy search from the **medoid** (the dataset's most -central point, the fixed entry) to find candidates, RobustPrune the -visited set, add back-edges. +At α = 1 this is HNSW's Algorithm-4-style directional pruning — keep +`p'` only if `p'` is closer to the new point than to an already-kept +neighbour. The new move is **α > 1**: `p'` survives unless the kept +edge gets you `α` times closer to it, so fewer candidates are pruned +and **longer edges are retained**. §2.2 states the intent as making +distance to the target *"decrease by a multiplicative factor of α > 1 +at every node along the search path, instead of merely decreasing as +in the SNG"*. + +Here is the caveat the folk version drops. §2.2's convergence claim +is conditional: + +> *"if the out-neighbors of every p ∈ P are determined by +> RobustPrune(p, P \ {p}, α, n − 1), then GreedySearch(s, p, 1, 1)… +> would converge to p ∈ P in logarithmically many steps, if α > 1. +> However, this would result in [Õ(n²) work]"* + +— so the logarithmic bound is proved for the version that considers +*every* point as a candidate with *no* degree bound, which is exactly +the version Vamana cannot afford. §2.2 continues that the real +algorithm *"invokes RobustPrune(p, V, α, R) for a carefully selected +V with far fewer than n − 1 nodes"*. Vamana therefore inherits the +*motivation* for the bound, not the bound. Say it that way; the +empirical support is §4.2's measurement that Vamana makes **2–3× +fewer hops** than HNSW and NSG at the same 98% 5-recall@5 with W=4. + +The build (Algorithm 3, §2.3) has three details people skip: + +1. The graph is **initialised to a random R-regular directed graph**, + not to an empty one — greedy search has to have somewhere to walk + on the first insert. +2. The fixed entry point `s` is the dataset's **medoid**. +3. There are **two passes** over a random permutation of the points: + the first with **α = 1**, the second with the user's α ≥ 1. Each + pass greedy-searches from `s`, RobustPrunes the visited set, and + adds back-edges (which are themselves RobustPruned when they + overflow R). + +§2.4 places this against the neighbours: HNSW and NSG *"implicitly +use α = 1"*, and HNSW additionally restricts its pruning candidate +set V to the final search result list, where Vamana and NSG use the +whole visited set. Levels vs slack, the design fork: HNSW buys few hops with a hierarchy (extra RAM, layered layout); Vamana buys it with edge slack (extra degree, flat layout — exactly what disk wants). -### Step 3 — the layout: one node's everything in one block +### Step 3 — the layout: one node's everything in one sector -With hops minimized, make each hop cost exactly one read: store each -node's full vector and its neighbor list *adjacent*, in one -SSD-page-aligned block: +> **In:** Vamana's flat graph and 512 GB of vectors. **Out:** the +> on-disk record format, the RAM-resident PQ oracle, and the real +> reason the padding is not waste. + +With hops minimised, make each hop cost exactly one read. §3.2 gives +the record layout in one sentence: *"for each point i, we store its +full precision vector x_i followed by the identities of its ≤ R +neighbors. If the degree of a node is smaller than R, we pad with +zeros, so that computing the offset within the disk of the data +corresponding to any point i is a simple calculation, and does not +require storing the offsets in memory."* ``` - RAM: PQ codes for ALL points (~16-32 B each) ← steers the walk - SSD: per-node block: [ full f32 vector | R neighbor ids ] + RAM: PQ codes for ALL points (§3.1: "e.g., 32 bytes per data point") + SSD: per-node record: [ full f32 vector | ≤R neighbour ids | zero pad ] node's data + links CO-LOCATED — one read per hop ``` -The arithmetic: d·4 + R·4 bytes per block — d=128, R=64 → ~768 B — -padded to one 4 KB page. Alignment IS the schema (topic 3's -slotted-page lesson). The RAM side is the PQ trick: at ~16–32 bytes -per point, PQ codes for ALL billion points fit in ~16–32 GB — a full -in-RAM (approximate) distance oracle. The topic-13 echo is exact: -node + adjacency co-located per block = kuzu's CSR node groups; -PQ-in-RAM = the sparse index steering to the right block -(ClickHouse marks, topic 12). +Note precisely what the padding is for: **fixed-size records so +offsets are computed rather than stored**. It is *not* "pad each node +to its own 4 KB page". Work the paper's own example from §3.5 — +degree 128, d=128: + +``` + neighbour ids 4 × 128 = 512 B (§3.5's "4*128 bytes long + for degree 128 graphs") + full f32 vector 128 × 4 = 512 B + ------------------------------------ + record 1024 B → fits in one 4 KB sector, with room to + spare for more records +``` + +§3.5's argument for why this is free: *"reading 4KB-aligned disk +address into memory is no more expensive than reading 512B, and the +neighborhood of a vertex … and full-precision coordinates can be +stored on the same disk sector."* The unused remainder of a sector is +not a tax you pay for one-read hops; it is capacity the SSD's minimum +transfer size gives you whether you use it or not. Alignment IS the +schema (topic 3's slotted-page lesson) — but the schema's job here is +*offset arithmetic*, not page ownership. + +The RAM side is the PQ trick (§3.1): at 32 bytes per point, codes for +all billion points are a full in-RAM approximate distance oracle. + +``` + 1e9 points × 32 B = 32 GB ← fits the z840's 64 GB + vs 1e9 × 128 × 4 = 512 GB ← does not +``` + +One subtlety §3.1 states and most summaries drop: *"Vamana uses +full-precision coordinates when building the graph index"* — the +compression is a search-time device only. A graph built on PQ +distances would bake the quantization error into its topology. + +The topic-13 echo is exact: node + adjacency co-located per record = +kuzu's CSR node groups; PQ-in-RAM = the sparse index steering to the +right block (ClickHouse marks, topic 12). ### Step 4 — the search loop: PQ steers, f32 ranks, W reads in flight -Search is beam search with width W (≈4–8): pick the W best -unexpanded candidates *by PQ distance* (RAM, essentially free), -fetch their SSD blocks **as one batch of concurrent reads** — -memory-level parallelism for disks — then use the exact f32 vectors -that just arrived to rank results, and the neighbor ids in the same -blocks to extend the frontier: +> **In:** the SSD index and a RAM-resident PQ oracle. **Out:** +> BeamSearch, the beam width W, and the division of labour that keeps +> the PQ error out of the final ranking. + +§3.3's **BeamSearch** fetches the neighbourhoods of the W closest +unexpanded candidates *in one shot* rather than one at a time. The +candidates are chosen by PQ distance (RAM, essentially free); the +blocks that come back carry both the exact f32 vector — used for +ranking — and the neighbour ids — used to extend the frontier. ```rust -// the disk loop: PQ (RAM) decides where to walk, f32 (SSD) decides the -// ranking — the approximation never touches the final order +// ILLUSTRATION — not quoted from any file; this is Algorithm 1 with +// §3.3's BeamSearch modification and §3.5's implicit re-ranking, as +// Rust. There is no DiskANN clone in resources/codebases.md, so the +// authority is the paper (Alg. 1, §3.3, §3.5). The nearest real code +// you can read is the same beam WITHOUT the disk parts, in qdrant: +// lib/segment/src/index/hnsw_index/graph_layers.rs:109 +// (search_on_level) and search_context.rs:32 (process_candidate). fn search(q: &[f32], k: usize, w: usize) -> Vec<(f32, Id)> { let mut cands = MinHeap::from([(pq_dist(q, MEDOID), MEDOID)]); let mut seen = HashSet::from([MEDOID]); @@ -120,37 +258,95 @@ fn search(q: &[f32], k: usize, w: usize) -> Vec<(f32, Id)> { } ``` -The division of labor is the deep idea: PQ error only affects WHERE -YOU WALK, never the final ranking — rescoring is fused into -traversal, since the exact vector arrives in the block you had to -read anyway. W is the ef of the disk world: wider beams overlap more -SSD reads (latency hiding) but waste reads on candidates that won't -survive. +The division of labour is the deep idea, and §3.5 names it: *"full +precision coordinates essentially piggyback on the cost of expanding +the neighborhoods."* PQ error affects only WHERE YOU WALK, never the +final ranking, because the exact vector arrives in the block you had +to read anyway. Re-ranking is fused into traversal rather than bolted +on — §3.5 contrasts this with the alternative of fetching all +re-ranking vectors in one shot, *"which would result in hundreds of +random disk accesses all in one shot."* + +W is the `ef` of the disk world, and the paper bounds it from both +sides in §3.3: *"a small number, W (say 4, 8)"*; *"If W = 1, this +search resembles normal greedy search"*; and *"if W is too large, say +16 or more, then both compute and SSD bandwidth could be wasted."* +Work the latency arithmetic: + +``` + SSD round trip (§3.3) ≈ 200-300 µs (a "few hundred") + target mean latency (abstract) < 3 ms + ⇒ rounds available per query ≈ 3000 / 250 ≈ 12 beam iterations + nodes visited at W = 8 ≈ 12 × 8 = 96 SSD records + vs the strawman's 300 dependent reads at 2 reads each = 600 +``` + +Twelve dependent round trips is the entire budget, which is why hop +count (Step 2) and reads-per-hop (Step 3) both had to be attacked. +§3.3 reports the system running at 30–40% SSD load with threads +spending 40–50% of query time in I/O, on drives capable of 500K+ +random reads/s — i.e. tuned so neither the drive nor the CPU is the +sole bottleneck. + +§3.4 adds one more RAM-side lever: cache all vertices within +`C = 3 or 4` hops of the fixed start point, so the first couple of +beam iterations never touch the SSD at all. ### Step 5 — the numbers to retain -- **~5 ms mean latency, 95%+ recall@1** on billion-scale SIFT, one - 64 GB machine — the headline; compare Step 1's ~60 ms strawman. -- Hop count ~O(log_α): tens of beam iterations, each one batch of W - ~100 µs reads overlapped — the latency budget adds up to - single-digit ms. -- ~R·4 + d·4 bytes per SSD block: R=64, d=128 → ~768 B, padded to a - 4 KB page — ~80% of each read is padding, the price of one-read - hops. +> **In:** §4's evaluation. **Out:** the six figures worth quoting, +> each with the configuration that produced it — because every one of +> them is conditional on a build parameter. + +| number | where | configuration | +|---|---|---| +| **> 5000 QPS, < 3 ms mean latency, 95%+ 1-recall@1** | abstract | SIFT1B, 16-core z840, 64 GB RAM, 2 × 960 EVO RAID-0 | +| **1-recall@1 of 98.68% at < 5 ms** | §4.3 | the *single* (one-shot) billion index: L=125, R=128, α=2 | +| merged index costs **≤ 20% extra latency** | §4.3 | 40 k-means shards, ℓ=2, R=64 → 348 GB, avg degree 92.1 | +| **2–3× fewer hops** than HNSW/NSG | §4.2 | measured at 98% 5-recall@5 with W=4 | +| Vamana builds DEEP1M in **149 s** vs HNSW 219 s, NSG 480 s | §4.1 | in-memory; Vamana L=125, R=70, C=3000, α=2; HNSW M=128, efC=512 | +| **> 95% 1-recall@1 in under 3.5 ms** at 32-byte codes | §4.4 | vs IVFOADC+G+P-32's plateau at 62.74% and -16's at 37.04% | + +Two of these are routinely misquoted, so state them carefully. + +**The "~5 ms" figure is not the headline.** The abstract's number is +`< 3 ms` mean latency at 95%+ 1-recall@1; the `< 5 ms` belongs to +§4.3's much stronger 98.68% recall point on the one-shot index. Which +one you cite changes both the latency and the recall. + +**The "5%" figure is not about SSD residency.** §4.3's sentence is +about *edge locality inside the merged index*: the single index beats +the merged one *"possibly because the in- and out-edges of each node +in the merged index are limited to about ℓ/k = 5% of all points"*, +where k=40 shards and ℓ=2 assignments per point give +`ℓ/k = 2/40 = 5%`. It says nothing about what fraction of the data is +read per query. + +The build costs are worth retaining too, because they are the reason +the merged construction exists at all (§4.3): the one-shot billion +index needed **~2 days on an M64-32ms with ≈1100 GB peak RAM**, while +the 40-shard merge produced a comparable index in **~5 days on the +z840 with memory staying under 64 GB**. Sharded build trades wall +clock for a machine you can actually rent. ## How to read the paper (with the concepts in hand) -- **§1–2 (motivation + Vamana)** — Steps 1–2. Read RobustPrune - twice; the α-slack line is one condition and it carries the whole - hop bound. The two-pass build detail matters for reproducing - quality. -- **§3 (the SSD design)** — Steps 3–4: the block layout, PQ-in-RAM, - beam search with batched reads. This is the section to read - line-by-line — it's where "disk-layout discipline applied to - graphs" lives. -- **§4 (evaluation)** — the headline numbers (Step 5); skim the - ablations but note the beam-width and α sweeps — they're the - paper's knobs-vs-curve section. +| paper | step | what to extract | +|---|---|---| +| §1, §2.1 | 1 | the three desiderata; why hierarchy is the wrong answer on disk | +| §2.2, Alg. 2 | 2 | RobustPrune's α line — and the `RobustPrune(p, P\{p}, α, n−1)` qualifier on the log bound | +| §2.3, Alg. 3 | 2 | random R-regular init, the medoid, the two passes (α=1 then α), back-edges | +| §2.4 | 2 | HNSW and NSG *"implicitly use α = 1"*; the candidate-set difference | +| §3.1 | 3 | PQ in RAM at ~32 B/point; Vamana builds on full precision | +| §3.2 | 3 | the record layout and *why* it is zero-padded (computed offsets) | +| §3.3 | 4 | BeamSearch; W = 4 or 8; the "few hundred microseconds" round trip | +| §3.4 | 4 | caching everything within C = 3–4 hops of the start | +| §3.5 | 4 | implicit re-ranking; *"no more expensive than reading 512B"* | +| §4.1–4.2 | 5 | in-memory comparison, build times, the 2–3× hop reduction | +| §4.3–4.4 | 5 | the billion-scale numbers, and the ℓ/k = 5% sentence in its real context | + +Read §2.2 and §3.5 twice; they are the two paragraphs the rest of the +paper rests on, and both contain a qualifier that summaries drop. ## Questions (answer in notes.md) @@ -168,21 +364,152 @@ survive. ## Done when +Answer each before unfolding it. + - [ ] You can say why HNSW does not survive being put on disk, in reads per hop rather than in generalities. +
Answer + + Two reads per hop, and they are *dependent*. An RAM-designed HNSW + keeps a node's vector and its neighbour list in separate + allocations, so a hop needs one read for the vector, then — after + computing distances — another for the links, then the next hop's + address is only known once those return. At §3.3's *"few hundred + microseconds"* per SSD round trip and a few hundred hops, that is + ~120 ms per query. DiskANN attacks both factors: Step 3 makes it + one read per hop by co-locating vector and links in one record + (§3.2), and Step 4 makes W of them concurrent rather than + sequential (§3.3), exploiting the fact that *"fetching a small + number of random sectors from an SSD takes almost the same time as + one sector."* The budget that remains is about twelve dependent + round trips for a 3 ms query. + +
+ - [ ] You can explain what `α > 1` does to greedy walk length, and why that is the property Vamana is buying. +
Answer + + Algorithm 2 drops a candidate `p'` only when + `α·d(p*, p') ≤ d(p, p')`. Raising α above 1 makes that test harder + to satisfy, so more candidates survive and the retained edge set + keeps *longer* edges. §2.2's stated intent is that distance to the + target then *"decrease[s] by a multiplicative factor of α > 1 at + every node along the search path"*, so the number of hops is + logarithmic in the distance ratio rather than linear in it. **The + honest qualifier**: §2.2 proves that only for + `RobustPrune(p, P\{p}, α, n−1)` — every point a candidate, no + degree bound — which costs Õ(n²) and is not what Vamana runs. + Vamana uses *"a carefully selected V with far fewer than n − 1 + nodes"*, so it inherits the motivation, not the theorem. The + empirical replacement is §4.2: 2–3× fewer hops than HNSW and NSG at + 98% 5-recall@5. Why this property: on SSD, hops are round trips, so + hop count *is* latency, and Vamana buys it with degree (flat + layout) instead of with a hierarchy (RAM). + +
+ - [ ] You can describe the block layout and count the SSD reads per hop it achieves. +
Answer + + §3.2: per point, the full-precision vector followed by up to R + neighbour ids, zero-padded to a fixed size *so that offsets are + computed rather than stored in RAM*. One read per hop, because both + halves of what a hop needs are in the same record. §3.5's own + worked case is degree 128 at d=128: `4 × 128 = 512 B` of ids plus + `128 × 4 = 512 B` of vector = 1024 B, which *"can be stored on the + same disk sector."* The remainder of a 4 KB sector is not overhead + charged to this design — §3.5's premise is that *"reading + 4KB-aligned disk address into memory is no more expensive than + reading 512B"*, so it is capacity the transfer size gives you for + free, and the implementation packs further records into it. Do not + say "each node is padded to a 4 KB page"; the paper does not. + +
+ - [ ] You can explain the division of labour in the search loop: PQ steers, f32 ranks, W reads in flight — and what recall failure each part is responsible for. +
Answer + + PQ codes live in RAM (§3.1, *"e.g., 32 bytes per data point"* — 32 + GB for a billion points) and choose which W candidates to fetch. + The SSD read returns both the neighbour ids (frontier) and the + exact f32 vector, so ranking is done on exact distances with *"no + extra reads"* (§3.5) — the re-ranking piggybacks on traversal. + Failure modes split cleanly: an f32 ranking error is impossible, + because the final order uses exact distances; the residual risk is + entirely in *steering*. If PQ error exceeds the spacing between + neighbourhoods, the beam is pointed at the wrong region and the + true neighbour is never fetched, so no amount of exact re-ranking + recovers it. W controls how much slack the steering gets: §3.3 + suggests 4 or 8, says W=1 degenerates to plain greedy search, and + warns that *"if W is too large, say 16 or more, then both compute + and SSD bandwidth could be wasted."* + +
+ +- [ ] You can state the paper's headline numbers with the configuration attached, and name the two that are commonly misquoted. +
Answer + + Headline (abstract): **> 5000 QPS, < 3 ms mean latency, 95%+ + 1-recall@1** on SIFT1B, 16-core z840 with 64 GB RAM and two + consumer NVMe drives. The two traps: (1) the *"< 5 ms"* figure + belongs to a different point — §4.3's one-shot index at **98.68%** + 1-recall@1 — so quoting "~5 ms" as the headline understates the + latency claim and the recall claim at once; (2) the *"5%"* figure + is §4.3's `ℓ/k = 2/40` **edge locality inside the merged index**, + offered as a possible reason the merged index is slower, and has + nothing to do with what fraction of the SSD is read per query. + Also worth attaching: 1-recall@1 is a stricter metric than the + recall@10 this topic's bench measures. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the M28 object-storage preview. +
Answer + + For question 5, the arithmetic is the point: at 250 µs per read a 3 + ms budget allows ~12 dependent rounds; at a 50 ms S3 GET the same + budget allows zero, and even a 500 ms budget allows ten. The knobs + that move are W (up hard, since object stores have effectively + unbounded concurrency and §3.3's "16 or more wastes bandwidth" + warning was about a device with 500K IOPS, not about a network), + and §3.4's cache radius C (up, since the first hops are the ones + you can most cheaply keep local). What breaks is the *dependency + chain*: DiskANN's design assumes round trips are cheap enough that + a dozen of them fit in the latency budget, and that assumption is + the thing object storage removes. + +
## References **Papers** - Subramanya, Devvrit, Kadekodi, Krishnaswamy, Simhadri — "DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single - Node" (NeurIPS 2019) — §2 Vamana + RobustPrune, §3 the SSD design; - the eval headline numbers are in §4 + Node" (NeurIPS 2019) + +| where | what it says | +|---|---| +| abstract | *"> 5000 queries a second with < 3ms mean latency and 95%+ 1-recall@1 on a 16 core machine"* | +| §2.2, Alg. 2 | RobustPrune; `if α·d(p*,p') ≤ d(p,p')` | +| §2.2 | the log-convergence result holds for `RobustPrune(p, P\{p}, α, n−1)`, which costs Õ(n²) | +| §2.3, Alg. 3 | random R-regular init; medoid start; two passes, α=1 then α | +| §2.4 | HNSW and NSG *"implicitly use α = 1"* | +| §3.1 | PQ in RAM, *"e.g., 32 bytes per data point"*; Vamana builds on full precision | +| §3.2 | vector then ≤R ids, zero-padded so offsets are computed not stored | +| §3.3 | BeamSearch; W *"(say 4, 8)"*; *"few hundred microseconds"*; *"16 or more"* wastes | +| §3.4 | cache everything within C = 3 or 4 hops of the start | +| §3.5 | *"reading 4KB-aligned disk address … no more expensive than reading 512B"*; the 4·128 B + vector sector calculation | +| §4.1 | Vamana L=125, R=70, C=3000, α=2 vs HNSW M=128, efC=512; DEEP1M build 149 s / 219 s / 480 s | +| §4.2 | Vamana makes 2–3× fewer hops, at 98% 5-recall@5, W=4 | +| §4.3 | one-shot index 98.68% 1-recall@1 at < 5 ms; merged ≤ 20% extra latency; `ℓ/k = 5%` edge locality; build costs (~2 days / 1100 GB vs ~5 days / < 64 GB) | +| §4.4 | > 95% 1-recall@1 under 3.5 ms at 32-byte codes; IVFOADC+G+P plateaus at 62.74% (-32) and 37.04% (-16) | + +- Singh, Subramanya, Krishnaswamy, Simhadri — "FreshDiskANN" + (2021) — the *separate* paper on streaming updates. Not covered + here; do not attribute its results to the 2019 paper. **Code** - [DiskANN](https://github.com/microsoft/DiskANN) — Microsoft's - production implementation of the paper (optional; the paper is - self-contained) + production implementation. **Not pinned in + `resources/codebases.md`**, so this chapter cites no line numbers + from it; if you read it, treat what you find as a *different + artifact* from the paper and record the commit you read. diff --git a/topics/14-vector-search/reading-hnsw-paper.md b/topics/14-vector-search/reading-hnsw-paper.md index fd181bb..b483796 100644 --- a/topics/14-vector-search/reading-hnsw-paper.md +++ b/topics/14-vector-search/reading-hnsw-paper.md @@ -11,38 +11,97 @@ maps the paper's five algorithms onto those concepts. They map almost line-for-line onto usearch's implementation ([reading-usearch.md](reading-usearch.md)), so read the two together. +Every paper claim below carries a section, algorithm-line, or figure +number from Malkov & Yashunin, *"Efficient and robust approximate +nearest neighbor search using Hierarchical Navigable Small World +graphs"*, IEEE TPAMI 42(4), 2018 — read here as +[arXiv:1603.09320v4](https://arxiv.org/abs/1603.09320). Code anchors +are `qdrant/qdrant@44ad62f` and `unum-cloud/usearch@9fd6b01`, the +revisions pinned in `resources/codebases.md`. Where the paper and an +implementation disagree, this guide says so rather than smoothing it +over. + ## The problem in one sentence -Return the k nearest of 1M 128-dimensional vectors without computing -1M distances per query — exact search streams 512 MB and does 128M -multiply-adds every single time, while HNSW answers in a few hundred -distance computations at recall@10 (the fraction of the true 10 -nearest neighbors the approximate answer actually contains) above -0.95. +Return the k nearest of n high-dimensional vectors without computing +n distances per query — because the exhaustive alternative, measured +on this topic's own bench, runs at **117 QPS at recall 1.000**, and +that single point is what every ANN index is betting against. + +That number is not borrowed. `./verify.sh 14` builds 100 000 +random 128-dimensional f32 vectors (51 MB), issues 500 queries for +k=10, and the brute-force lane takes **4.28 s** on an Apple M3 Pro +(`topics/14-vector-search/notes.md`, baseline measured 2026-07-28). +Work out what the machine was doing: + +``` + distances per query = 100 000 vectors + multiply-adds each = 128 dimensions + queries = 500 + ------------------------------------------------ + total multiply-adds = 500 × 100 000 × 128 = 6.4 × 10⁹ + wall clock = 4.28 s + throughput = 6.4e9 / 4.28 = 1.50 × 10⁹ MAC/s + query rate = 500 / 4.28 = 117 QPS +``` + +1.5 G multiply-adds per second is a *healthy* number for one core — +the scan is not slow, there is simply too much of it. You cannot fix +117 QPS by making the loop faster; a 4× SIMD win buys 468 QPS, still +three orders of magnitude short of a production vector store. The +only lever with the right exponent is touching less data, and that is +what a proximity graph sells. HNSW's claim is a few hundred distance +computations per query instead of 100 000 — a ~300× reduction in +work, paid for with recall@10 slightly below 1.0. + +**recall@10** here means the fraction of the true 10 nearest +neighbours that the approximate answer actually contains, averaged +over queries; **QPS** is completed queries per second, single +threaded unless said otherwise. Both are defined this way in +`topics/14-vector-search/README.md` and measured that way by the +bench. ## The concepts, step by step ### Step 1 — k-NN search, and why "approximate" is the product -k-nearest-neighbor (k-NN) search takes a query vector and returns -the k database vectors with the smallest distance to it (l2, dot, or -cosine — the algorithm won't care, see Step 7). Exact k-NN has -exactly one implementation: compute all n distances, keep the k -best. That's a memory-bound streaming scan — topic 12's lesson, now -per query. The entire ANN (approximate nearest neighbor) field is -one trade: accept recall < 1.0 in exchange for touching a *tiny, -query-dependent subset* of the data. Every algorithm is a point on -the recall-vs-QPS curve in the topic README; HNSW's claim to fame is -generating the best points on it while keeping the trade adjustable -per query. +> **In:** a query vector `q`, a dataset of `n` vectors, an integer +> `k`. **Out:** the vocabulary for the rest of the guide — *exact +> k-NN*, *ANN*, *recall*, and the reason the approximate answer is +> the product rather than a concession. + +**k-nearest-neighbour (k-NN) search** takes a query vector and +returns the k database vectors with the smallest distance to it (l2, +dot, or cosine — the algorithm will not care; see Step 7). Exact k-NN +has exactly one implementation: compute all n distances, keep the k +best. That is a memory-bound streaming scan — topic 12's lesson, now +paid once per query, and it is the 117 QPS above. + +**Approximate nearest neighbour (ANN)** search is the entire field +built on one trade: accept recall < 1.0 in exchange for touching a +*tiny, query-dependent subset* of the data. Every algorithm is a +point on the recall-vs-QPS curve in the topic README. + +The word "approximate" is doing something specific. It is not that +the index is a lossy cache of a correct answer you could get later; +it is that *the recall you want is an input*. HNSW's distinguishing +property is that this input is supplied **per query**, after the +index is built (Step 4), so one index serves a 0.90-recall +autocomplete and a 0.999-recall reranker at different latencies. ### Step 2 — the proximity graph: navigate instead of scan -A proximity graph connects each vector to a handful of its near -neighbors, and search becomes navigation: start anywhere, repeatedly -hop to whichever neighbor is closest to the query, stop when no -neighbor improves. Each hop only computes distances for one node's -~16–32 neighbors — that's the sublinearity. +> **In:** the n vectors and a distance function. **Out:** a graph +> where each node links to a handful of near neighbours, and a search +> procedure — greedy routing — whose cost is *degree × hops* instead +> of n. + +A **proximity graph** connects each vector to a handful of its near +neighbours, and search becomes navigation: start anywhere, +repeatedly hop to whichever neighbour is closest to the query, stop +when no neighbour improves. Each hop computes distances for one +node's neighbours only — with degree 16 and, say, 40 hops that is +640 distance computations instead of 100 000. ``` q × @@ -53,107 +112,190 @@ neighbor improves. Each hop only computes distances for one node's ●───●───● ← entry nearest neighbor ``` -This is NSW, the paper's predecessor: one navigable graph, greedy -routing from a random entry. It worked, but with two flaws: node -degree grew polylogarithmically with n (early-inserted nodes -accumulated links), and quality depended on insertion order. Long -routes across the dataset and short local links were tangled in one -graph. +That is **NSW**, the paper's own predecessor (§3 recaps it). It +worked, but §3 names two flaws: node degree grew polylogarithmically +with n, because early-inserted nodes kept accumulating links from +later ones, and the graph's quality depended on insertion order. Long +routes across the dataset and short local links were tangled into one +structure, so you could not tune them separately. ### Step 3 — the skip-list fix: layers with geometrically fewer nodes +> **In:** NSW's single tangled graph. **Out:** L layers, each a +> proximity graph over a geometrically smaller sample, plus the level +> assignment rule and the constant `mL` that sets the ratio. + A skip list (topic 2) fixes slow linked-list search by adding -express lanes: each element gets a random level, higher levels have +express lanes: each element gets a random level, higher levels hold geometrically fewer elements, and search descends from sparse to -dense. HNSW applies exactly this fix to the proximity graph — that's -the "Hierarchical" in the name: +dense. HNSW applies exactly this fix to the proximity graph — the +"Hierarchical" in the name: ``` L2: ●────────────────● sparse "highways" \ \ - L1: ●──●─────●────────●──● each node: level ~ -ln(U)·mL + L1: ●──●─────●────────●──● each node: level ⌊-ln(U)·mL⌋ \ \ \ \ \ - L0: ●─●─●─●─●─●─●─●─●─●─●─●─● dense base layer, M0 = 2M links + L0: ●─●─●─●─●─●─●─●─●─●─●─●─● dense base layer, Mmax0 = 2M links ``` +Algorithm 1 line 4 of the paper is the rule, and the floor matters: + ``` skip list: express lanes over a linked list, level ~ Geometric(p) - HNSW: express graphs over a proximity graph, level ~ ⌊-ln(U)·mL⌋ + HNSW: l ← ⌊-ln(unif(0..1)) · mL⌋ (Alg. 1, line 4) ``` -with `mL = 1/ln(M)` — chosen so level occupancy drops by factor M -(the per-node link budget, Step 6's table), exactly a skip list's -p = 1/M. Upper layers hold long links between far-apart points; -layer 0 holds everyone with short links. Search cost becomes -O(log n) descent plus a constant-quality local search at L0 — -and, unlike NSW, it no longer depends on insertion order. +**mL** is the level normalisation constant. §4.1: *"a simple choice +for the optimal mL is `1/ln(M)`, this corresponds to the skip list +parameter p = 1/M with an average single element overlap between the +layers."* Work out why that ratio falls out — this is question 1, and +it is four lines: + +``` + level l = ⌊ -ln(U) · mL ⌋, U ~ Uniform(0,1), mL = 1/ln M + + P(l ≥ j) = P( -ln(U)/ln M ≥ j ) substitute mL + = P( -ln(U) ≥ j·ln M ) + = P( U ≤ e^(-j ln M) ) exponentiate, flip + = M^(-j) + + so layer j holds n·M^(-j) nodes in expectation: each layer up + is M× thinner, exactly p = 1/M. + + top layer = the j where n·M^(-j) ≈ 1 ⇒ j = ln n / ln M + + n = 1 000 000, M = 16: + ln(1e6)/ln(16) = 13.8155 / 2.7726 = 4.98 ⇒ ~5 layers +``` + +Five layers to descend, then one bounded search at the base. Unlike +NSW, none of this depends on insertion order: the level is drawn +from a distribution, not earned by arriving early. + +One implementation note to carry into +[reading-qdrant-hnsw.md](reading-qdrant-hnsw.md): qdrant does not +floor. `graph_layers_builder.rs:392` calls `.round()` on the same +`-ln(U)·level_factor` expression, which raises the fraction of nodes +promoted above layer 0 from `1/(M−1)` to roughly `M^(−1/2)` — for +M=16, from 6.7% to about 25%. usearch floors, via a C++ cast to an +integer type (`index.hpp:4339`). Same paper, two different graphs. ### Step 4 — search: greedy descent, then a bounded best-first beam -The query path (paper's Alg 5) has two phases. Phase one: from the -top layer's entry point, greedily descend — on each upper layer keep -just the single closest node found (a beam of width 1), then drop a -layer. Phase two, on layer 0: best-first search with **ef** (the -"expansion factor", the number of candidate results kept while -searching — THE recall/latency knob), tracked by two heaps: a -min-heap of candidates to expand (nearest on top) and a bounded -max-heap of the best ef results seen (worst on top). Stop when the -nearest unexpanded candidate is farther than the worst kept result — -no expansion can improve the answer. +> **In:** the layered graph, a query `q`, and two integers `k` and +> `ef`. **Out:** the k nearest found, plus the reason `ef` is the +> only knob you turn at query time. + +The query path (Algorithm 5, K-NN-SEARCH) has two phases. Phase one: +from the top layer's entry point, greedily descend — `for lc ← L … +1`, each layer calling SEARCH-LAYER with **ef = 1**, keeping just the +single closest node found, then dropping a layer. Phase two, on layer +0: one SEARCH-LAYER call with the user's `ef`, then return the K +nearest elements from the result set W. + +**ef** ("size of the dynamic candidate list" in the paper's own +words) is the number of candidate results kept while searching — the +recall/latency knob. Algorithm 2 tracks it with two structures: a +min-heap `C` of candidates to expand and a bounded set `W` of the +best ef found so far. The stop test is Algorithm 2 lines 7–8: +extract the nearest candidate, and if it is farther than the worst +element of W, break — no unexpanded candidate can improve the answer. ```rust +// ILLUSTRATION — not quoted from any file; this is Algorithms 2 and 5 +// condensed into one function. The real ones are the paper's +// pseudocode, and in code at usearch include/usearch/index.hpp:4629 +// (search_to_find_in_base_) and qdrant +// lib/segment/src/index/hnsw_index/graph_layers.rs:109 (search_on_level). fn search(idx: &Hnsw, q: &[f32], k: usize, ef: usize) -> Vec { let mut ep = idx.entry_point; for level in (1..=idx.max_level).rev() { - ep = greedy_closest(idx, level, ep, q); // upper layers: ef=1, just descend + ep = greedy_closest(idx, level, ep, q); // Alg 5: ef=1 descent } - let mut cands = MinHeap::from([(dist(q, ep), ep)]); // nearest candidate on top - let mut best = BoundedMaxHeap::new(ef); // worst-of-ef on top - let mut visited = VisitedSet::from([ep]); // THE hot structure + let mut cands = MinHeap::from([(dist(q, ep), ep)]); // Alg 2's C + let mut best = BoundedMaxHeap::new(ef); // Alg 2's W + let mut visited = VisitedSet::from([ep]); // Alg 2's v while let Some((d, c)) = cands.pop() { - if d > best.worst() { break; } // nearest cand can't improve: stop + if d > best.worst() { break; } // Alg 2, lines 7-8 for n in idx.neighbors(0, c) { if !visited.insert(n) { continue; } let dn = dist(q, idx.vec(n)); - if dn < best.worst() || !best.full() { + if dn < best.worst() || !best.full() { // Alg 2, line 13 cands.push((dn, n)); - best.push_evicting((dn, n)); // ef bounds BOTH heaps + best.push_evicting((dn, n)); // ef bounds BOTH } } } - best.take_top(k) // hence ef ≥ k + best.take_top(k) // Alg 5: K nearest of W } ``` -The costs to notice: ef is per-QUERY — the recall/latency trade is -decided at search time, not build time; nothing in the index -changes. And the visited set is the hot structure — allocated and -cleared once per query, which is why qdrant and usearch both pool it -(topic 13's stamp trick). +Two costs to notice. First, `ef` is per-*query*: the recall/latency +trade is decided at search time, nothing in the index changes. Second, +`visited` is the hot structure — it is touched once per neighbour +examined and must be cleared once per query, which is why both qdrant +and usearch pool it rather than allocating (topic 13's stamp trick; +qdrant's is `lib/segment/src/index/visited_pool.rs:78`, a `u8` +generation counter that only really zeroes the array every 255 +queries). + +Note the asymmetry Algorithm 5 creates: `W` holds at most `ef` +elements, and the function returns `K` of them. If `ef < K` there are +not enough elements to return, which is why every implementation +clamps `ef` up to at least `k`. At exactly `ef = k`, `W` is full from +the first k neighbours examined and the line-13 admission test +degenerates to "strictly better than the current worst" — the beam +can never hold a candidate that is temporarily bad but leads +somewhere good, and recall falls off sharply. ### Step 5 — insert: draw a level, search down, connect -Insert (paper's Alg 1) reuses search. Draw the new point's level -ℓ = ⌊-ln(U)·mL⌋ (Step 3). From the top entry point, greedily descend -(ef=1) to layer ℓ+1 — just finding the neighborhood. Then from layer -ℓ down to 0, run the Step 4 beam with `ef_construction` (a -build-time ef, typically ~100–128), pick M neighbors from the beam's -results (how to pick is Step 6), add bidirectional links, and shrink -any neighbor that now exceeds its budget (M on upper layers, M0 = 2M -on layer 0). Cost: an insert is roughly one search plus O(M) link -edits — building the index is ~n searches, which is why build time -is one of the three currencies (RAM, build time, recall). - -### Step 6 — the neighbor-selection heuristic: directions, not distances - -The load-bearing detail (paper's Alg 4): when connecting a new point -to M neighbors, do NOT take the M nearest. Take candidates -nearest-first, and keep candidate c only if -`d(c, new) < d(c, kept)` for every already-kept neighbor — c must be -closer to the new point than to anything already chosen. Effect: -neighbors cover DIRECTIONS, not just distances — a dense nearby -cluster gets one representative edge, and the remaining budget buys -long links outward: +> **In:** a new vector and the existing graph. **Out:** the graph +> with the new point linked in, and the cost model that makes build +> time one of the three currencies. + +Insert (Algorithm 1) reuses search. Draw the new point's level +`l = ⌊-ln(U)·mL⌋` (Step 3). From the top entry point, greedily +descend with ef=1 down to layer `l+1` — just locating the +neighbourhood. Then from layer `min(L, l)` down to 0, run the Step 4 +beam with **efConstruction** (a build-time ef), pick M neighbours +from the beam's results with SELECT-NEIGHBORS (Step 6), add +bidirectional links, and shrink any neighbour that now exceeds its +budget — `Mmax` on upper layers, `Mmax0` on layer 0. + +Cost: an insert is roughly one search plus O(M) link edits, so +building is ~n searches. §4.1 is explicit that efConstruction has no +canonical default — the guidance is to *"select an efConstruction +value that is large enough to produce K-ANNS recall close to unity +during the construction process (0.95 is enough for most +use-cases)."* The paper's own experiments use whatever that turned +out to be: §5's 200M SIFT run uses efConstruction=500 (5.6 hours) and +a cheaper efConstruction=40 run (42 minutes) on the same hardware. +The frequently quoted "100" comes from Fig. 10's 10M SIFT example +(3 minutes on four 10-core Xeon E5-4650 v2), not from a stated +default. + +### Step 6 — the neighbour-selection heuristic: directions, not distances + +> **In:** the efConstruction candidates found in Step 5 and a budget +> M. **Out:** which M of them become edges — and why "the M nearest" +> is the wrong answer. + +This is the load-bearing detail. The paper gives two selectors: + +- **Algorithm 3, SELECT-NEIGHBORS-SIMPLE** — return the M nearest. + The strawman. +- **Algorithm 4, SELECT-NEIGHBORS-HEURISTIC** — walk candidates + nearest-first and keep candidate `e` only if, per line 11, *"e is + closer to q compared to any element from R"*, the set already kept. + In other words: `d(e, q) < d(e, r)` for every kept `r`. + +Effect: neighbours cover **directions**, not just distances. A dense +nearby cluster gets one representative edge — every other member of +that cluster is closer to the representative than to the new point, +so the test rejects it — and the remaining budget buys long links +outward: ``` M-nearest: new ●══▶ ○○○ (all 3 links into one cluster; @@ -162,60 +304,119 @@ long links outward: └────────▶ ● far cluster stays connected) ``` -Without it, inter-cluster navigability dies — greedy routing from -one cluster can never reach another, and recall collapses no matter -how big ef gets. `extendCandidates` and `keepPrunedConnections` are -the paper's own knobs over the heuristic. This is also where -implementations differ or cheat: qdrant's `use_heuristic` flag -(graph_layers_builder.rs:41-42) makes it optional; usearch always -applies it. +Without it, inter-cluster navigability dies: greedy routing from one +cluster can never reach another, and recall collapses no matter how +large `ef` gets — the beam explores a component that does not contain +the answer. + +Algorithm 4 takes two further flags, and both are usually off. +`extendCandidates` widens the candidate set to the candidates' +neighbours; the paper says it is *"set to false by default"* and is +useful only for extremely clustered data. +`keepPrunedConnections` back-fills the budget with rejected +candidates so every node reaches exactly M links. Neither appears in +qdrant's implementation of the heuristic +(`lib/segment/src/index/hnsw_index/links_container.rs:47`); qdrant +implements the plain Algorithm 4 and makes the whole heuristic +optional behind a `use_heuristic` flag +(`graph_layers_builder.rs:41-42`). ### Step 7 — parameters, memory, and the warts -The ecosystem froze the paper's advice into defaults: - -| param | paper | usearch default | meaning | -|---|---|---|---| -| M | 5-48 | 16 (`connectivity`) | links/node upper layers | -| M0 | 2M | 32 (`connectivity_base`) | links at layer 0 | -| ef_construction | ~100 | 128 (`expansion_add`) | build-time beam | -| ef | ≥ k | 64 (`expansion_search`) | query-time beam — THE knob | - -Three properties round out the picture: - -- **Metric-agnostic**: distance only enters via comparisons, so HNSW - works for any metric-ish function — cosine/dot/l2 are one - codebase. -- **Memory hunger**: links cost n·(M0 + M·E[levels]) ids on top of - the raw vectors — for n=1M, d=128, M=16 that's ~512 MB of vectors - plus ~140 MB of u32 links, RAM-resident by design (DiskANN exists - because of this — [reading-diskann.md](reading-diskann.md)). -- **Deletes are the unsolved wart**: the paper has none; real - systems tombstone + rebuild (qdrant has a - graph_layers_healer.rs) — the CSR-update-pain story (topic 13) - again. +> **In:** everything above. **Out:** the numbers you will actually +> type into a config, where the paper's advice ends and the +> ecosystem's convention begins, and the two things HNSW does badly. + +The paper gives ranges and one derived constant; the ecosystem froze +particular values into defaults. Keep the columns separate: + +| param | what it is | paper (§4.1) | qdrant default | usearch default | +|---|---|---|---|---| +| M | links/node, upper layers | *"a reasonable range of M is from 5 to 48"* | 16 (`types.rs:1412`) | 16 (`index.hpp:1563`, `connectivity`) | +| Mmax0 | links at layer 0 | *"simulations suggest 2·M is a good choice"* | `m * 2` (`config.rs:46`) | `connectivity * 2` (`index.hpp:1591`) | +| mL | level constant | `1/ln(M)` | `1/ln(max(m,2))` (`graph_layers_builder.rs:317`) | `1/log(connectivity)` (`index.hpp:4149`) | +| efConstruction | build-time beam | no default; *"large enough to produce recall ≈ 0.95 during construction"* | 100 (`types.rs:1409`) | 128 (`index.hpp:1568`) | +| ef | query-time beam — THE knob | ≥ k, chosen per query | defaults to `ef_construct` (`config.rs:48`) | 64 (`index.hpp:1573`) | + +Three properties round out the picture. + +**Metric-agnostic.** Distance enters only through comparisons, so +one codebase serves cosine, dot and l2. usearch takes this furthest: +ten metric kinds in one enum (`index_plugins.hpp:114-133`). + +**Memory hunger.** §4.2.3 gives the formula directly: average memory +per element is `(Mmax0 + mL·Mmax) · bytes_per_link`, which for +4-byte ids and M in 6..48 the paper reports as *"about 60-450 bytes +per object"*. Work the standard configuration: + +``` + n = 1 000 000, d = 128, f32 vectors, M = 16 + Mmax = M = 16, Mmax0 = 2M = 32, mL = 1/ln 16 = 0.36067 + bytes_per_link = 4 (u32 id) + + vectors : 1e6 × 128 × 4 B = 512.0 MB + links : 1e6 × (32 + 0.36067×16) × 4 B + = 1e6 × (32 + 5.771) × 4 B + = 1e6 × 151.08 B = 151.1 MB + --------- + total 663.1 MB + vectors / links = 3.39× +``` + +Two honest caveats on that 151 MB. The paper's formula uses `mL` +where the *expected number of layers above 0* is actually +`Σ_{j≥1} M^(-j) = 1/(M−1) = 1/15 = 0.0667`, because Algorithm 1 +floors the draw. Using the flooring value gives +`(32 + 0.0667×16) × 4 = 132.3 B`, so the paper's figure is ~14% high +— it is the reserved capacity, not the occupied one, and every +implementation that preallocates per-level link arrays actually pays +the higher number. And it counts only ids: qdrant's real level-0 +container also stores a length, and usearch's tape adds a 10-byte +node header per point (`index.hpp:2341` — an 8-byte key plus a 2-byte +level) and a 4-byte count per level +(`index.hpp:4150-4151`). Whatever the accounting, the conclusion +holds: **vectors dominate links by roughly 3–4×**, which is why +quantizing the vectors (topic 14's other half, +[reading-pq.md](reading-pq.md)) is the memory lever and shrinking the +graph is not. + +**Deletes are the unsolved wart.** The paper has no delete algorithm +at all — Algorithms 1–5 cover insert and search only. Real systems +tombstone and rebuild; qdrant carries a whole +`graph_layers_healer.rs` to repair the links a removed point leaves +dangling. It is the CSR-update-pain story from topic 13 again: a +structure optimised for a static read layout is expensive to mutate. ## How to read the paper (with the concepts in hand) -The paper numbers its pseudocode; the steps above are its reading -lens: - -- **§1–3 (intro, related work, NSW recap)** — skim; Steps 1–2. The - one thing to extract is *why* NSW's degree grew and how the layers - fix it (Step 3). -- **Alg 1 (INSERT)** — Step 5. Note the two phases: ef=1 descent to - layer ℓ+1, then ef_construction beams from ℓ down to 0. -- **Alg 2 (SEARCH-LAYER)** — Step 4's two heaps, in the authors' - words. Match each line against the Rust condensation above. -- **Alg 3 vs Alg 4 (SELECT-NEIGHBORS simple vs heuristic)** — Step 6. - Alg 3 is the strawman (M nearest); Alg 4 is the product. Work the - two-cluster picture by hand. -- **Alg 5 (K-NN-SEARCH)** — Step 4's phase structure: descent + one - Alg 2 call at layer 0 with ef. -- **§4 (complexity)** — the mL = 1/ln(M) derivation; question 1 - below. -- **§5 (evaluation)** — skimmable; the recall/QPS curves are the - topic README's curve, measured. +The paper numbers its pseudocode; the steps above are the reading +lens. + +| paper | step | what to extract | +|---|---|---| +| §1–3 (intro, related work, NSW) | 1–2 | *why* NSW's degree grew with n, and how layers fix it | +| Alg. 1 (INSERT) | 5 | the two phases, and the floor in line 4 | +| Alg. 2 (SEARCH-LAYER) | 4 | lines 7–8 (the stop test) and line 13 (the admission test) | +| Alg. 3 vs Alg. 4 | 6 | Alg 3 is the strawman; Alg 4 line 11 is the product | +| Alg. 5 (K-NN-SEARCH) | 4 | ef=1 descent, then one Alg 2 call at layer 0 | +| §4.1 | 3, 7 | mL = 1/ln(M), Mmax0 = 2M, M ∈ 5..48, the efConstruction guidance | +| §4.2.1 | 3 | the O(log N) argument — and its assumption | +| §4.2.3 | 7 | the memory formula and the 60–450 bytes/object range | +| §5 (evaluation) | — | skimmable; the recall/QPS curves are the topic README's curve, measured | + +One thing to read carefully rather than accept. §4.2.1's O(log N) +scaling is derived *under the assumption of exact Delaunay graphs* — +which HNSW does not build, precisely because constructing a Delaunay +graph in high dimensions is intractable. The paper's own text says +the argument is confirmed by simulations on low-dimensional data and +that further analytic evidence is required for the high-dimensional +case. So "HNSW is O(log N)" is a claim about an idealised relative, +supported empirically for the real thing. The mechanism is worth +holding onto even so: with `p = exp(−mL)` the probability that a +greedy step stays in the same layer, the expected number of steps per +layer is bounded by `S = 1/(1 − exp(−mL))`, and the number of layers +scales as `log N` — a constant amount of work per layer, logarithmically +many layers. ## Questions (answer in notes.md) @@ -230,25 +431,161 @@ lens: ## Done when +Answer each before unfolding it. + - [ ] You can explain what makes "approximate" the product rather than a compromise, using this topic's measured 117 QPS brute-force floor. +
Answer + + The brute-force lane is not badly written — 6.4 × 10⁹ multiply-adds + in 4.28 s is 1.5 G MAC/s on one core. It is 117 QPS because it does + 500 × 100 000 × 128 units of work, and no constant-factor + optimisation changes that exponent: perfect 4-wide SIMD gives 468 + QPS, still far from a production store. Approximation is the only + lever that changes the amount of data touched — a few hundred + distance computations instead of 100 000. And because `ef` is a + per-query argument (Alg. 5), the amount of approximation is chosen + by the caller after the index exists, so it is a product feature + rather than a defect: one index serves a cheap 0.90-recall path and + an expensive 0.999-recall path. + +
+ - [ ] You can derive why `mL = 1/ln(M)` gives an expected max level of `ln(n)/ln(M)`. +
Answer + + Level is `l = ⌊-ln(U)·mL⌋` with `U ~ Uniform(0,1)` (Alg. 1, line + 4). Then + `P(l ≥ j) = P(-ln U ≥ j/mL) = P(U ≤ e^(-j/mL)) = e^(-j/mL)`. + Substituting `mL = 1/ln M` gives `e^(-j·ln M) = M^(-j)`. So layer + j holds `n·M^(-j)` nodes in expectation — each layer is M× thinner, + which is the skip list's `p = 1/M`, and §4.1 says exactly that. The + top non-empty layer is where the expectation reaches 1: + `n·M^(-j) = 1 ⇒ j = ln n / ln M`. For n = 10⁶ and M = 16 that is + `13.8155 / 2.7726 = 4.98`, about five layers. + +
+ - [ ] You can state what Algorithm 4's neighbour selection does differently from taking the M nearest, and what breaks if you take the nearest. +
Answer + + Alg. 3 returns the M nearest. Alg. 4 walks candidates + nearest-first and keeps `e` only if, per its line 11, `e` is closer + to the new point than to any already-kept neighbour. On two + well-separated clusters, every member of the near cluster is closer + to the first-kept member than to the new point, so all but one are + rejected — one edge represents the whole cluster and the remaining + budget goes to the far cluster. Take the M nearest instead and all + M links land inside the near cluster; the far cluster becomes + unreachable by greedy routing, so recall collapses and no increase + in `ef` fixes it, because the beam is exploring a component that + does not contain the answer. The paper's two extra flags, + `extendCandidates` and `keepPrunedConnections`, are off by default + (§4.1) and absent from qdrant's implementation + (`links_container.rs:47-71`). + +
+ - [ ] You can say why `ef >= k` is required and what happens at exactly `ef = k`. +
Answer + + Algorithm 5's last step returns the K nearest elements *of W*, and + Algorithm 2 bounds `|W| ≤ ef`. With `ef < k` there are simply not k + elements to return. At `ef = k` exactly, W fills from the first k + neighbours examined and Alg. 2's line-13 admission test + (`distance(e,q) < distance(f,q) or |W| < ef`, where f is W's + furthest) reduces to "strictly better than the current worst". The + beam can no longer hold a candidate that is temporarily worse but + routes toward a better region, so the search behaves like plain + greedy descent and recall drops sharply. Implementations clamp: for + example usearch defaults `expansion_search` to 64 + (`index.hpp:1573`), comfortably above a typical k=10. + +
+ - [ ] You can account for HNSW's memory at n=1M, d=128, M=16, splitting vectors from links. +
Answer + + Vectors: `1e6 × 128 × 4 B = 512 MB`. Links, by §4.2.3's formula + `(Mmax0 + mL·Mmax) · bytes_per_link` with Mmax0=32, Mmax=16, + mL=0.36067, 4-byte ids: `(32 + 5.771) × 4 = 151.08 B` per element, + so 151.1 MB — inside the paper's stated 60–450 bytes/object range + for M ∈ 6..48. Total ≈ 663 MB, vectors dominating links by 3.39×. + Caveat worth stating: the formula uses `mL`, whereas the expected + number of layers above 0 under the floored draw is `1/(M−1) = + 0.0667`, which would give 132.3 B — the paper's figure is the + reserved capacity, ~14% above the occupied one. Either way the + ratio is what matters: quantizing vectors is the memory lever + ([reading-pq.md](reading-pq.md)), shrinking the graph is not. + +
+ +- [ ] You can name one place where a production implementation deviates from the paper, and say what the deviation changes. +
Answer + + Several are verifiable at the pinned revisions. (a) Alg. 1 line 4 + *floors* the level draw; qdrant *rounds* + (`graph_layers_builder.rs:392`), which promotes roughly 25% of + points above layer 0 at M=16 instead of 6.7% — a taller, wider + hierarchy and more link memory. (b) §4.1 gives no efConstruction + default; qdrant picks 100 (`types.rs:1409`) and usearch picks 128 + (`index.hpp:1568`). (c) The paper's `extendCandidates` and + `keepPrunedConnections` do not exist in qdrant's heuristic + (`links_container.rs:47-71`). (d) qdrant's serve-time `ef` defaults + to `ef_construct` rather than to anything derived from k + (`config.rs:48`). + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions are mirrored in + `topics/14-vector-search/notes.md`. Questions 1, 3 and 4 have + worked arithmetic above and should be re-derived rather than + copied; question 2 wants the two-cluster picture drawn by hand; + question 5's skip-list analogue is that a skip list also has a + single fixed head — the entry point is the top-level sentinel, and + descending from it is exactly Alg. 5's ef=1 phase. + +
## References **Papers** - Malkov, Yashunin — "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" - (IEEE TPAMI 2018, - [arXiv:1603.09320](https://arxiv.org/abs/1603.09320)) — Algorithms - 1-5 are the chapter; the eval is skimmable - -**Code** -- [usearch](https://github.com/unum-cloud/usearch) — the paper's - algorithms map to functions almost line-for-line; walked in - [reading-usearch.md](reading-usearch.md) -- [qdrant](https://github.com/qdrant/qdrant) — the production version, - walked in [reading-qdrant-hnsw.md](reading-qdrant-hnsw.md) + (IEEE TPAMI 42(4), 2018, + [arXiv:1603.09320](https://arxiv.org/abs/1603.09320)) + +| where | what it says | +|---|---| +| Alg. 1, line 4 | `l ← ⌊-ln(unif(0..1))·mL⌋` — the floored level draw | +| Alg. 2, lines 7–8 | the stop test: nearest candidate worse than W's furthest | +| Alg. 2, line 13 | the admission test: `d(e,q) < d(f,q) or |W| < ef` | +| Alg. 3 / Alg. 4 | M-nearest vs the heuristic; Alg. 4 line 11 is the rule | +| Alg. 4 params | `extendCandidates` *"set to false by default"* | +| Alg. 5 | ef=1 descent `for lc ← L … 1`, one ef search at layer 0 | +| §4.1 | mL = 1/ln(M); Mmax0 = 2M; M ∈ 5..48; efConstruction guidance | +| §4.2.1 | O(log N) — under the exact-Delaunay assumption, low-d simulations | +| §4.2.3 | memory = `(Mmax0 + mL·Mmax)·bytes_per_link`, 60–450 B/object | +| Fig. 10 / §5 | efConstruction=100 example; 500 and 40 in the 200M SIFT runs | + +**Code** (pins in `resources/codebases.md`) + +| file:line | repo | what | +|---|---|---| +| `include/usearch/index.hpp:1563,1568,1573,1591` | usearch@9fd6b01 | M=16, efConstruction=128, ef=64, Mmax0=2M | +| `include/usearch/index.hpp:4149` | usearch@9fd6b01 | `inverse_log_connectivity` — mL | +| `include/usearch/index.hpp:4336-4340` | usearch@9fd6b01 | `choose_random_level_`, the floored draw | +| `lib/segment/src/types.rs:1409-1422` | qdrant@44ad62f | m=16, ef_construct=100 | +| `lib/segment/src/index/hnsw_index/config.rs:46,48` | qdrant@44ad62f | `m0 = m*2`, `ef = ef_construct` | +| `lib/segment/src/index/hnsw_index/graph_layers_builder.rs:317,392` | qdrant@44ad62f | `level_factor`, and `.round()` where the paper floors | +| `lib/segment/src/index/hnsw_index/links_container.rs:47-71` | qdrant@44ad62f | Algorithm 4, without the two flags | + +**Companion guides** +- [reading-usearch.md](reading-usearch.md) — the algorithms as C++, + almost line-for-line +- [reading-qdrant-hnsw.md](reading-qdrant-hnsw.md) — the production + version, with filtering +- [reading-diskann.md](reading-diskann.md) — what to do when the + 512 MB of vectors will not fit diff --git a/topics/14-vector-search/reading-pq.md b/topics/14-vector-search/reading-pq.md index 354c652..bd9c961 100644 --- a/topics/14-vector-search/reading-pq.md +++ b/topics/14-vector-search/reading-pq.md @@ -10,82 +10,198 @@ on codes without decoding, and the residual system the paper actually ships. Topic 12's dictionary encoding, but the dictionary is learned and the code is a concatenation. +Paper claims below cite Jégou, Douze & Schmid, *"Product +Quantization for Nearest Neighbor Search"*, IEEE TPAMI 33(1), 2011 — +read here as the author manuscript +[inria-00514462v2](https://inria.hal.science/inria-00514462v2). The +paper numbers its sections in Roman numerals, so **§II** is the +quantizer, **§III** is SDC/ADC, **§IV** is IVFADC and **§V** is the +evaluation; equation and table numbers are the paper's own. Code +anchors are `qdrant/qdrant@44ad62f`, the pin in +`resources/codebases.md`. + ## The problem in one sentence -A billion 128-d f32 vectors is **512 GB** — they don't fit in RAM, +A billion 128-d f32 vectors is **512 GB** — they do not fit in RAM, and even if they did, exact distances cost 128 multiply-adds each — so we need a code a few *bytes* long per vector that still supports distance computation, and a plain quantizer capable of that fidelity -would need more centroids than there are atoms in a datacenter. +would need more centroids than a datacenter could store. + +Two definitions to fix before Step 1. A **centroid** is one of the k +representative points a quantizer maps vectors onto; the set of them +is the **codebook**. **Recall@R** in this paper means something +narrower than the repo's usual usage — §V-A defines it as *"the +proportion of query vectors for which the nearest neighbor is ranked +in the first R positions"*, i.e. a 1-NN measure with a re-ranking +shortlist of size R, not the k-NN recall the topic bench reports. +When you compare a PQ number against this topic's brute-force +recall 1.000, you are comparing two different quantities; say which +one you mean. ## The concepts, step by step ### Step 1 — vector quantization: replace a vector with its nearest centroid -A vector quantizer maps each vector to the nearest of k -representative points called **centroids** (learned by k-means: -alternate "assign each vector to its nearest centroid" and "move -each centroid to the mean of its assignees"). The set of centroids -is the **codebook**; the stored code is just the centroid's index — -⌈log₂ k⌉ bits. Distance to a quantized vector ≈ distance to its -centroid, so the quantization error IS the accuracy loss. - -The wall: fidelity needs many centroids, but a codebook with k -centroids costs k·d floats to store and k·d multiply-adds to encode -one vector. k = 2²⁰ (a 20-bit code) is about the practical limit — -and 20 bits is nowhere near enough to describe a 128-d vector well. -For a 64-bit code you'd need k = 2⁶⁴ centroids: unstorable, -unlearnable. +> **In:** a set of d-dimensional vectors. **Out:** a codebook of k +> centroids, a code per vector of ⌈log₂ k⌉ bits, and the wall that +> makes plain quantization useless at the fidelity we need. + +A **vector quantizer** is a function `q` mapping a d-dimensional +vector to one of k centroids (§II-A, Eq. 1-2). The centroids are +learned by k-means — the paper calls it the Lloyd quantizer and +states the two optimality conditions it satisfies: assign each vector +to its nearest centroid (Eq. 4), and set each centroid to the mean of +its assignees (Eq. 5). The stored code is just the centroid's index. +§II-A: *"The memory cost of storing the index value, without any +further processing (entropy coding), is ⌈log₂ k⌉ bits. Therefore, it +is convenient to use a power of two for k."* + +Distance to a quantized vector ≈ distance to its centroid, so the +quantization error *is* the accuracy loss. + +The wall is stated in §II-B's opening, with SIFT as the example. A +quantizer producing 64-bit codes for a 128-dimensional vector — the +paper's phrasing, *"only 0.5 bit per component"* — needs +`k = 2^64` centroids. Table I gives the two costs that kills: + +``` + codebook storage (k-means) = k·D floats + assignment cost per vector = k·D multiply-adds + + k = 2^64, D = 128, f32: + storage = 2^64 × 128 × 4 B = 9.4 × 10^21 bytes = 9.4 zettabytes + and you would need several times k training samples to learn it +``` + +The paper's own summary: *"it is impossible to use Lloyd's algorithm +or even HKM… It is even impossible to store the D×k floating point +values representing the k centroids."* ### Step 2 — the product move: quantize subspaces independently -Product quantization splits the d dimensions into m contiguous -chunks and runs a *separate small quantizer* (k* = 256 centroids, -so each chunk's code is exactly one byte) on each chunk. The full -code is the concatenation of the m chunk codes: +> **In:** the impossible k = 2^64 codebook. **Out:** m small +> codebooks whose Cartesian product has the same cardinality, at +> storage that grows *linearly* in m. + +§II-B, Eq. 8: split the input vector `x` into m distinct subvectors +`u_j` of dimension `D* = D/m` (D a multiple of m), and quantize each +with its own subquantizer `q_j`. The full code is the concatenation +of the m chunk codes. Eq. 9 makes the codebook the Cartesian product +`C = C_1 × … × C_m`, and Eq. 10 gives its size: **`k = (k*)^m`**, +where `k*` is the per-subquantizer centroid count. ``` - x (d=128) → [x¹ | x² | ... | x¹⁶] m=16 chunks of 8 dims + x (d=128) → [x¹ | x² | ... | x¹⁶] m=16 chunks of D* = 8 dims q¹(x¹) q²(x²) ... — each an 8-bit centroid id effective centroids: 256¹⁶ = 2¹²⁸ stored: 16 bytes/vector - codebook cost: m · 256 · (d/m) = 256·d floats — tiny ``` -The implied codebook is the Cartesian product of the m small ones: -256¹⁶ = 2¹²⁸ distinct representable points, from codebooks totalling -256·d floats (128 KB for d=128). The exponential codebook for linear -storage is the whole paper. Same energy as topic 12's dictionary -encoding, but the dictionary is LEARNED (k-means per subspace) and -the code is a concatenation. The cost: the product structure assumes -the chunks are roughly statistically independent — correlated -dimensions split across chunks waste code space (question 2; OPQ -exists to fix this, Step 5). +Work both sides of the trade with real numbers: + +``` + d = 128, m = 16, D* = d/m = 8, k* = 256 + + code length = m · log₂ k* = 16 × 8 = 128 bits = 16 bytes + effective k = (k*)^m = 256^16 = 2^128 centroids + codebook storage = m · k* · D* = k* · d = 256 × 128 + = 32 768 floats = 128 kB + encode one vector= k* · D = 256 × 128 = 32 768 mult-adds + + compare Step 1's plain quantizer at the same effective k = 2^128: + storage 2^128 × 128 × 4 B — not a number worth writing down +``` + +128 kB of codebook and 32 768 operations to encode, for a codebook +of cardinality 2^128. That exchange — exponential effective codebook, +linear storage — is the whole paper. Table I states it in general +form: product k-means costs `m k* D* = k^(1/m) D`, where plain +k-means costs `kD`. + +Why `k* = 256` specifically? Two reasons, both in §II-B. First, +`log₂ 256 = 8`, so each chunk's code is exactly one byte and the +concatenation needs no bit-shifting. Second, the paper measured which +side of the trade to be on and says: *"for a fixed number of bits, it +is better to use a small number of subquantizers with many centroids +than having many subquantizers with few bits"* — a claim repeated +with recall numbers in §V-B. It then names the convention everyone +inherited: *"Using k* = 256 and m = 8 is often a reasonable +choice."* + +There is a ceiling on k* too, and it is a cache argument the paper +makes itself in §II-B: high k* *"increase[s] the memory usage of +storing the centroids (k* × D floating point values), which further +reduces the efficiency if the centroid look-up table does no longer +fit in cache memory."* Step 3 puts a number on that. + +The cost of the product structure: it assumes the chunks are roughly +statistically independent — correlated dimensions split across +chunks waste code space (question 2; OPQ exists to fix this, Step 5). +§II-B is candid that the fix is sometimes a rotation and sometimes +nothing: *"One way to ensure this property is to multiply the vector +by a random orthogonal matrix prior to quantization. However, for +most vector types this is not required and not recommended, as +consecutive components are often correlated by construction and are +better quantized together with the same subquantizer."* ### Step 3 — SDC vs ADC: where you eat the approximation -Distances on codes come in two flavors, differing in whether the -*query* gets quantized too: - -- **SDC** (symmetric distance computation): quantize the query as - well; distance = a precomputed centroid-to-centroid table lookup - per chunk. Fastest possible, but TWO approximations (query error + - database error). -- **ADC** (asymmetric distance computation): keep the query exact. - Once per query, build the `[m × 256]` table of exact sub-distances - `‖qʲ - cⱼ,ᵢ‖²` from each query chunk to every centroid; then any - database vector's distance ≈ m table lookups + adds. ONE - approximation — strictly better recall for the same codes. - Everyone ships ADC (qdrant's `EncodedQueryPQ`, - encoded_vectors_pq.rs:39-41). +> **In:** two PQ codes, or one code and one raw query. **Out:** two +> distance estimators with the same asymptotic cost and materially +> different accuracy, and the reason every production system ships +> the second one. + +§III-A gives both, and the difference is exactly whether the *query* +gets quantized. + +- **SDC** (symmetric distance computation, Eq. 12): quantize the + query too, so `d̂(x,y) = d(q(x), q(y))`, read from a table of + centroid-to-centroid distances. The table holds all `(k*)²` squared + distances per subquantizer, though footnote 1 notes only + `k*(k*−1)/2` need be stored by symmetry. **Two** approximations — + query error and database error. +- **ADC** (asymmetric distance computation, Eq. 13): keep the query + exact, so `d̃(x,y) = d(x, q(y))`. Once per query, build the + `[m × k*]` table of exact sub-distances from each query chunk to + every centroid in that chunk's codebook; then any database vector's + distance is m table lookups plus adds. **One** approximation. + +Table II is explicit that SDC does not buy speed: encoding the query +costs `k*D` for SDC and 0 for ADC, but computing the query's +sub-distances costs 0 for SDC and `k*D` for ADC — *"SDC and ADC have +the same query preparation cost, which does not depend on the dataset +size n"*, and both scan at `nm`. §III-A's conclusion is a +recommendation, not a hedge: *"The only advantage of SDC over ADC is +to limit the memory usage associated with the queries… one should +then use the asymmetric version, which obtains a lower distance +distortion for a similar complexity."* + +Table V measures it, on GIST with 64-bit codes (m=8, k*=256): + +| method | search time (ms) | code comparisons | recall@100 | +|---|---|---|---| +| SDC | 16.8 | 1 000 991 | 0.446 | +| ADC | 17.2 | 1 000 991 | **0.652** | + +Same code length, same scan, 2.4% more time, and recall@100 goes +from 0.446 to 0.652. §V-B puts the same result the other way round: +*"For m=8 we obtain the same accuracy for ADC and k*=64 as for SDC +and k*=256"* — ADC buys you two bits per subquantizer for free. +That is why nobody ships SDC, and it answers question 5. ```rust -// ADC: pay m·256 exact sub-distances ONCE per query… +// ILLUSTRATION — not quoted from any file; this is Eq. 13 of the PQ +// paper as Rust. The production version is qdrant's +// lib/quantization/src/encoded_vectors_pq.rs:515-537 (the table) and +// :474-489 (the scan). + +// ADC: pay m·k* exact sub-distances ONCE per query… fn adc_table(q: &[f32], cb: &Codebook) -> Vec<[f32; 256]> { (0..cb.m).map(|j| { let qj = &q[j * cb.sub_d..(j + 1) * cb.sub_d]; std::array::from_fn(|i| l2_sq(qj, cb.centroid(j, i))) - }).collect() // [m × 256] f32 — small enough to live in L1 + }).collect() // [m × 256] f32 } // …then EVERY candidate costs m byte-indexed lookups, zero float math @@ -94,68 +210,158 @@ fn adc_dist(code: &[u8], table: &[[f32; 256]]) -> f32 { } ``` -For d=128, m=16: 16 KB of tables built once, then each candidate -costs 16 byte-indexed L1 loads instead of 128 multiply-adds — PQ -trades float math for L1-resident lookups. The paper also derives -the distance ESTIMATOR bias (ADC underestimates on average) and a -correction — worth knowing it exists; most systems skip the -correction and oversample instead. +The table's size is the number to keep in your head, because §II-B's +cache warning lands here: + +``` + LUT bytes = m · k* · sizeof(f32) = m × 256 × 4 = 1024·m + + d=128, m=16 (16-byte codes) : 16 × 1024 = 16 kB → fits a 32-48 kB L1d + d=128, m=8 (8-byte codes) : 8 × 1024 = 8 kB → fits comfortably + d=128, m=128 (qdrant's X4) :128 × 1024 = 128 kB → L2 at best + + build cost = m · k* · D* = k* · d = 256 × 128 = 32 768 mult-adds + per-candidate cost = m adds = 16 +``` + +So the per-query table build costs the same as encoding one vector, +and it pays for itself after `32 768 / 16 = 2 048` candidates — below +a two-thousand-candidate shortlist, ADC is dominated by its own setup +and you may as well compute exact distances. That is question 3, and +it is the reason IVFADC's `w` (Step 4) cannot be too small. + +§III-C is worth reading precisely because the paper argues itself out +of its own result. It derives a bias correction (Eq. 25): the ADC +estimator systematically underestimates, and adding the mean +distortion `ξ_j` of each subquantizer removes the bias. Figure 4 +measures both on 10 000 SIFT vectors with m=8, k*=256: bias goes from +**−0.044** to **0.002**, but the variance goes *up*, from +`σ² = 0.00146` to `0.00155`. The paper's verdict: *"In our +experiments, we observe that the correction returns inferior results +on average. Therefore, we advocate the use of Equation 13 for the +nearest neighbor search. The corrected version is useful only if we +are interested in the distances themselves."* Nobody ships the +correction because the authors told them not to — not because the +industry ignored it. ### Step 4 — IVFADC: coarse cells + residual encoding -ADC still scans every code; the paper's shipped system adds a -**coarse quantizer** — a plain k-means with nlist cells (Step 1's -kind) — to make the scan sublinear. Each vector is assigned to its -nearest cell and stored in that cell's **inverted list** (the list -of all vectors in the cell). Query: find the nprobe nearest cells, -ADC-scan only their lists. +> **In:** ADC, which still touches every code. **Out:** a two-level +> system that touches `n·w/k′` of them, and the reason what gets +> encoded is a residual rather than a vector. + +§IV: ADC's scan is still exhaustive, so the shipped system adds a +**coarse quantizer** `q_c` — a plain k-means of the Step 1 kind, with +`k′` centroids, *"typically ranges from k′ = 1 000 to k′ = 1 000 +000"* for SIFT. Each vector goes into the **inverted list** of its +nearest coarse centroid (§IV-B). A query is assigned to its `w` +nearest coarse centroids (the **multiple assignment** of §IV-C, `w` +being IVF's version of `ef`) and only those lists are scanned. -The subtle move: what gets PQ-encoded is not the vector but its -**residual** `x - c(x)` — the offset from its cell's centroid: +The subtle move (§IV-A, Eq. 28-29): what gets PQ-encoded is not the +vector but its **residual** `r(y) = y − q_c(y)`, the offset from its +cell's centroid, so the stored approximation is +`ÿ = q_c(y) + q_p(y − q_c(y))`. ``` - query ─► nearest nprobe cells ─► ADC over residual codes ─► top-k - (coarse index) (16 B/vector, L1 LUTs) + query ─► nearest w cells ─► ADC over residual codes ─► top-k + (coarse index) (m bytes/vector, LUT per cell) ``` -Residuals matter: they're centered around 0 with much smaller -variance than raw vectors, so 256 centroids per subspace go further. -This is frame-of-reference (topic 12's FOR bit-packing) in learned -form: subtract the predictable part, encode the residual cheaply. -The cost knob is nprobe — more cells probed buys recall with scan -time, IVF's version of ef. +§IV's own justification: *"encoding the residual is more precise than +encoding the vector itself"*, because *"the energy of the residual +vector is small compared to that of the vector itself"* (§IV-A). This +is frame-of-reference — topic 12's FOR bit-packing — in learned form: +subtract the predictable part, encode the cheap remainder. The paper +makes the analogy itself: *"the coarse quantizer provides the most +significant bits, while the product quantizer code corresponds to the +least significant bits."* -### Step 5 — what survived twenty years +§IV-C gives the scan cost directly — *"about n×w/k′ entries have to +be parsed"* — which is the whole point: -Four pieces of this 2011 paper are load-bearing in 2026 systems: +``` + n = 1 000 000 000, k′ = 1024, w = 8 + entries scanned = n·w/k′ = 1e9 × 8 / 1024 = 7.81 × 10^6 + vs flat ADC = 1e9 + reduction = k′/w = 128× +``` -- **ADC lookup tables** — unchanged everywhere; qdrant's - `encoded_vectors_pq.rs` is Step 3 verbatim. -- **Residual encoding** — DiskANN keeps PQ codes in RAM to steer SSD - reads ([reading-diskann.md](reading-diskann.md)). -- **OPQ** (rotate the space before chunking so subspaces - decorrelate) — the main refinement worth knowing exists; it - attacks Step 2's independence assumption directly. -- **The recall gap at high k** — why oversample+rescore became the - standard pipeline - ([reading-qdrant-quantization.md](reading-qdrant-quantization.md) §4). +Table V measures the same shape on GIST, and shows both knobs: + +| method | search time (ms) | code comparisons | recall@100 | +|---|---|---|---| +| ADC (flat) | 17.2 | 1 000 991 | 0.652 | +| IVFADC k′=1024, w=1 | 1.5 | 1 947 | 0.308 | +| IVFADC k′=1024, w=8 | 8.8 | 27 818 | 0.682 | +| IVFADC k′=1024, w=64 | 65.9 | 101 158 | 0.744 | +| IVFADC k′=8192, w=8 | 10.2 | 2 709 | 0.516 | + +Read the first two rows together: `w=1` scans 514× fewer codes than +flat ADC and is 11× faster, but drops recall@100 from 0.652 to 0.308 +— the query's true neighbour is often in a *neighbouring* cell. +`w=8` recovers it and then some. Note also the timing floor: `w=1` on +k′=1024 scans 1 947 codes in 1.5 ms, which is nowhere near +1 947 × m adds — most of that 1.5 ms is the coarse assignment and the +ADC table build, exactly the 2 048-candidate break-even from Step 3. + +One implementation consequence that catches people: because the ADC +table is built from `x − q_c(y)` (Eq. 30-31), it depends on *which +cell* is being scanned, so IVFADC rebuilds the `m × k*` table once +per probed cell, not once per query. With `w = 8` that is eight table +builds. + +### Step 5 — what survived twenty years + +> **In:** the 2011 paper. **Out:** which four of its parts you will +> meet again in a 2026 codebase, and which one thing it got argued +> out of. + +- **ADC lookup tables** — unchanged everywhere. qdrant's + `lib/quantization/src/encoded_vectors_pq.rs:515-537` is Eq. 13 + verbatim: `lut_capacity = vector_division.len() * centroids.len()` + at :516, then an exact sub-distance per (chunk, centroid) pair at + :518-534. The scan at :474-489 is one f32 load per code byte, + strided by `centroids_count`, summed. +- **k* = 256** — qdrant hard-codes `CENTROIDS_COUNT = 256` + (`encoded_vectors_pq.rs:30`), and its user-facing + `CompressionRatio` enum (`lib/segment/src/types.rs:749-757`) is + really a choice of `D*`: `get_bucket_size` + (`lib/segment/src/vector_storage/quantized/quantized_vectors.rs:2314-2322`) + maps X4→1, X8→2, X16→4, X32→8, X64→16 dimensions per chunk. At + d=128, qdrant's X64 *is* the paper's recommended m=8, k*=256, + 64-bit code. +- **Residual encoding** — the idea outlived IVF. DiskANN keeps PQ + codes in RAM to steer SSD reads + ([reading-diskann.md](reading-diskann.md)). +- **OPQ** (Ge, He, Ke, Sun, CVPR 2013 — rotate the space before + chunking so subspaces decorrelate) — the main refinement worth + knowing exists; it attacks Step 2's independence assumption + directly, and §II-B's own remark about random orthogonal matrices + is where the thread starts. +- **What did *not* survive**: the §III-C bias correction, rejected by + its own authors. What replaced it in practice is + oversample-and-rescore + ([reading-qdrant-quantization.md](reading-qdrant-quantization.md)), + which fixes ranking errors rather than distance bias. ## How to read the paper (with the concepts in hand) -- **§2 (the quantizer)** — Steps 1–2. The distortion formalism is - denser than it needs to be; keep the "product of small codebooks = - exponential effective codebook" picture in front of you and the - math follows. -- **§3 (SDC/ADC and the estimator)** — Step 3. Read the ADC part - carefully — it's the part every production system runs. The bias - correction (§3.3) is skimmable; note it exists, note nobody ships - it. -- **§4 (IVFADC)** — Step 4. The residual argument is two paragraphs; - translate it to FOR terms as you read. The nprobe/recall curves - here are the paper's version of the topic README's recall-vs-QPS - curve. -- **§5 (evaluation)** — skim; SIFT1B was the headline dataset, and - the numbers set the baseline DiskANN later chased. +| paper | step | what to extract | +|---|---|---| +| §II-A | 1 | Eq. 4-5 (Lloyd conditions); the ⌈log₂ k⌉-bit code | +| §II-B | 2 | Eq. 8 (the split), Eq. 10 (`k = (k*)^m`), Table I, and the "k*=256, m=8" recommendation | +| §III-A | 3 | Eq. 12 (SDC) vs Eq. 13 (ADC), Table II's cost columns, and the closing recommendation | +| §III-B, §III-C | 3 | the distortion analysis and Eq. 25's correction — then Fig. 4 and the sentence rejecting it | +| §IV-A | 4 | Eq. 28-29, the residual argument; translate it into FOR terms as you read | +| §IV-B, §IV-C | 4 | the inverted-list entry layout, and `n·w/k′` | +| §V-A | — | Table III (SIFT: d=128, 1M base, 10k queries) and the recall@R *definition* | +| §V-B | 2, 3 | the m-vs-k* trade at fixed code length; "ADC k*=64 ≈ SDC k*=256" | +| §V-E | 3, 4 | Table V — SDC/ADC/IVFADC times, comparisons and recall in one place | + +The distortion formalism in §II is denser than it needs to be. Keep +"product of small codebooks = exponential effective codebook" in +front of you and the algebra follows. ## Questions (answer in notes.md) @@ -174,24 +380,169 @@ Four pieces of this 2011 paper are load-bearing in 2026 systems: ## Done when +Answer each before unfolding it. + - [ ] You can explain the product move: why quantizing subspaces independently gives 2^128 effective centroids from 16 bytes. +
Answer + + §II-B, Eq. 8-10. Split d=128 into m=16 chunks of D*=8, run a + separate 256-centroid k-means on each, concatenate the 16 one-byte + codes. The implied codebook is the Cartesian product (Eq. 9), so + its cardinality is `(k*)^m = 256^16 = 2^128` (Eq. 10), while + storage is `m·k*·D* = k*·d = 256 × 128 = 32 768` floats = 128 kB + and encoding costs `k*·d = 32 768` multiply-adds. A plain quantizer + at the same effective k would need `k·D` floats, which for + `k = 2^128` is not a storable number. Table I states the general + form: `k^(1/m)·D` versus `k·D`. + +
+ - [ ] You can state the difference between SDC and ADC and say where each eats its approximation. +
Answer + + SDC (Eq. 12) quantizes the query as well and looks up + centroid-to-centroid distances, so it eats *two* approximations — + query error plus database error — and its per-subquantizer table + holds `(k*)²` entries (or `k*(k*−1)/2` by symmetry, footnote 1). + ADC (Eq. 13) leaves the query exact and builds an `[m × k*]` table + of query-chunk-to-centroid distances per query, eating *one* + approximation. Table II shows their costs are the same shape: SDC + pays `k*D` to encode the query, ADC pays `k*D` to build its table, + both scan at `nm`. Table V measures the payoff on GIST at 64-bit + codes: 16.8 ms / recall@100 0.446 for SDC versus 17.2 ms / 0.652 + for ADC. §III-A's own words: *"one should then use the asymmetric + version."* + +
+ - [ ] You can explain why chunks must be roughly statistically independent, and what correlated dimensions do to the code. +
Answer + + The product quantizer's error decomposes as + `MSE(q) = Σ_j MSE(q_j)` (Eq. 11) *because the subspaces are + orthogonal*, and each subquantizer spends its 8 bits describing + variation within its own chunk only. If two chunks carry + near-duplicate information, both spend bits encoding the same + degree of freedom and the effective code length is shorter than + `m·log₂ k*`. Conversely, if one chunk carries most of the variance + and another almost none, the low-variance chunk's 256 centroids are + wasted while the high-variance chunk is under-resolved — which is + why §II-B says each subvector should have *"on average, a + comparable energy"*. OPQ learns a rotation that equalises and + decorrelates before splitting. The topic 12 analogue is + BYTE_STREAM_SPLIT: regrouping bytes so each stream is internally + homogeneous, letting a per-stream encoder do its job. + +
+ - [ ] You can compute the per-query ADC table build cost and say at what candidate count it stops mattering. +
Answer + + Build cost is `m · k* · D* = k* · d` — the m cancels — so at + k*=256, d=128 it is 32 768 multiply-adds, exactly the cost of + encoding one vector (Table I). Per-candidate cost is `m` table + lookups and adds; at m=16 that is 16 operations. Break-even is + `32 768 / 16 = 2 048` candidates. Below a ~2 000-candidate + shortlist you are paying more to build the table than to use it, + which is visible in Table V: IVFADC with k′=1024, w=1 scans only + 1 947 codes yet still takes 1.5 ms. The table also has to fit in + cache to be worth it — `m × 256 × 4 B` is 16 kB at m=16, and §II-B + warns explicitly about the case where it *"does no longer fit in + cache memory."* + +
+ - [ ] You can say why IVFADC encodes residuals rather than raw vectors, in terms of the quantizer's dynamic range. +
Answer + + §IV-A, Eq. 28-29: the coarse quantizer already captures where in + the space the vector lives, so the PQ only has to describe the + offset within one Voronoi cell — *"the energy of the residual + vector is small compared to that of the vector itself"*. The same + 256 centroids per subspace therefore cover a much smaller range and + quantize it finer, so §IV can claim residual encoding *"slightly + improves the search accuracy"* on top of the speedup. In topic 12's + vocabulary this is frame-of-reference: subtract a per-block + reference value, encode the small remainder in fewer bits. The + paper's own analogy is positional: coarse code = most significant + bits, PQ code = least significant bits. The consequence to remember + is that the ADC table depends on the probed cell (Eq. 30-31), so it + is rebuilt `w` times per query, not once. + +
+ +- [ ] You can say what the paper derives in §III-C and why it then tells you not to use it. +
Answer + + §III-C derives an unbiased estimator (Eq. 25) by adding each + subquantizer's mean distortion `ξ_j` to the ADC estimate, since + `E[d(x,y)²] = d̃(x,y)² + ξ(q, q(y))` (Eq. 24). Figure 4 measures it + on 10 000 SIFT vectors at m=8, k*=256: the bias drops from −0.044 + to 0.002, but the variance rises from 0.00146 to 0.00155 — the + classic bias/variance exchange. Worse, the correction is largest + for rare codes, so it penalises exactly the vectors most likely to + be true near neighbours. The paper concludes *"the correction + returns inferior results on average… we advocate the use of + Equation 13"* and keeps the corrected form only for when you want + the distances themselves rather than a ranking. This is the repo's + "report the negative result" rule, in a 2011 paper. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + Question 1's arithmetic, since it is the one most often fudged: at + a fixed 16 bytes = 128 bits, m=16 means `128/16 = 8` bits per + subquantizer (k*=256, D*=8), while m=64 means `128/64 = 2` bits + (k*=4, D*=2). Same storage, same effective `(k*)^m = 2^128`, but + §II-B and §V-B both say the m=16 side wins: *"for a fixed number of + bits, it is better to use a small number of subquantizers with many + centroids."* What changes is the LUT (16 kB vs 64 × 4 × 4 = 1 kB), + the scan cost (16 adds vs 64), and the resolution within each + subspace — 4 centroids cannot describe an 2-d chunk usefully. + +
## References **Papers** - Jégou, Douze, Schmid — "Product Quantization for Nearest Neighbor - Search" (IEEE TPAMI 2011) — §2 the quantizer, §3 SDC/ADC and the - estimator, §4 IVFADC; the paper everyone builds on + Search" (IEEE TPAMI 33(1):117-128, 2011; + [inria-00514462v2](https://inria.hal.science/inria-00514462v2)) + +| where | what it says | +|---|---| +| §II-A, Eq. 4-5 | Lloyd conditions; code is ⌈log₂ k⌉ bits | +| §II-B, Eq. 8 | the split into m subvectors of D* = D/m | +| §II-B, Eq. 10 | `k = (k*)^m` — the exponential effective codebook | +| §II-B, Table I | `mk*D* = k^(1/m)D` storage vs plain k-means's `kD` | +| §II-B | *"Using k* = 256 and m = 8 is often a reasonable choice"*; the LUT-in-cache warning | +| §III-A, Eq. 12 | SDC; `(k*)²` table per subquantizer (footnote 1: `k*(k*−1)/2`) | +| §III-A, Eq. 13 | ADC; `[m × k*]` table built per query | +| §III-A, Table II | equal query-prep cost; *"one should then use the asymmetric version"* | +| §III-C, Eq. 25, Fig. 4 | the bias correction: −0.044 → 0.002 bias, 0.00146 → 0.00155 variance, and the recommendation against it | +| §IV-A, Eq. 28-29 | residual `r(y) = y − q_c(y)`; coarse = MSBs, PQ = LSBs | +| §IV-A | `k′` from 1 000 to 1 000 000 for SIFT | +| §IV-B | inverted-list entry: identifier (8-32 bits) + code (`m⌈log₂ k*⌉` bits) | +| §IV-C | multiple assignment `w`; *"about n×w/k′ entries have to be parsed"* | +| §V-A, Table III | SIFT d=128 / 1M base / 10k queries; recall@R is a 1-NN measure | +| §V-B | *"for a fixed number of bits, … a small number of subquantizers with many centroids"*; ADC k*=64 ≈ SDC k*=256 | +| §V-E, Table V | GIST 64-bit codes: SDC 16.8 ms/0.446, ADC 17.2 ms/0.652, IVFADC rows | + - Ge, He, Ke, Sun — "Optimized Product Quantization" (CVPR 2013) — optional; the rotation refinement worth knowing exists -**Code** -- [qdrant](https://github.com/qdrant/qdrant) - `lib/quantization/src/encoded_vectors_pq.rs` — the production ADC, - walked in - [reading-qdrant-quantization.md](reading-qdrant-quantization.md) +**Code** — `qdrant/qdrant@44ad62f`, pinned in `resources/codebases.md`. + +| file:line | what | +|---|---| +| `lib/quantization/src/encoded_vectors_pq.rs:30` | `CENTROIDS_COUNT = 256` | +| `lib/quantization/src/encoded_vectors_pq.rs:38-43` | `EncodedQueryPQ { lut }` — the ADC table | +| `lib/quantization/src/encoded_vectors_pq.rs:515-537` | building it, Eq. 13 | +| `lib/quantization/src/encoded_vectors_pq.rs:474-489` | the ADC scan | +| `lib/segment/src/types.rs:749-757` | `CompressionRatio { X4 … X64 }` | +| `lib/segment/src/vector_storage/quantized/quantized_vectors.rs:2314-2322` | ratio → dimensions per chunk (D*) | + +Walked in +[reading-qdrant-quantization.md](reading-qdrant-quantization.md). diff --git a/topics/14-vector-search/reading-qdrant-hnsw.md b/topics/14-vector-search/reading-qdrant-hnsw.md index 5a4940f..2a6d59f 100644 --- a/topics/14-vector-search/reading-qdrant-hnsw.md +++ b/topics/14-vector-search/reading-qdrant-hnsw.md @@ -6,62 +6,194 @@ watching a query planner appear inside an index; before the code, it builds the ideas in order — the build/serve split, the pooled search machinery, why filters shatter graphs (percolation), and the per-query decision that picks HNSW / brute force / ACORN from an -estimated cardinality. Everything lives under -`lib/segment/src/index/hnsw_index/`; this chapter assumes +estimated cardinality. This chapter assumes [reading-hnsw-paper.md](reading-hnsw-paper.md). +Every `file:line` below was read at **`qdrant/qdrant@44ad62f`**, the +revision pinned in `resources/codebases.md`; re-verify any of them +with `python3 tools/pinned-source.py show qdrant -r A:B`. Most +paths are under `lib/segment/src/index/hnsw_index/`, but not all — +`visited_pool.rs` sits one directory up, in +`lib/segment/src/index/`, and the quantization pieces are in a +separate crate ([reading-qdrant-quantization.md](reading-qdrant-quantization.md)). + ## The problem in one sentence `WHERE category = X AND vec NEAR q` breaks a graph index: rejecting -non-matching nodes during traversal effectively deletes them, and a -graph with average degree K disconnects once only ~1/K of its nodes -survive — so at 5% selectivity on an M0=32 graph, greedy search -strands in an island and recall falls off a cliff. +non-matching nodes during traversal effectively deletes them from the +walk, and a random graph with average degree K disintegrates once +only ~1/K of its nodes survive — so on qdrant's default `m0 = 32` +graph the cliff sits near 3% selectivity, and a query below it +strands in an island while the brute-force alternative it is trying +to beat still runs at this topic's measured **117 QPS**. + +Two definitions used throughout. **Selectivity** is the fraction of +indexed points that pass the filter (qdrant computes it as +`cardinality / available_vector_count`, `hnsw/search.rs:80`) — so +*low* selectivity means a *restrictive* filter. **Percolation +threshold** is the survival fraction below which a random graph stops +having one giant connected component. ## The concepts, step by step ### Step 1 — build structure ≠ serve structure -A graph being *built* needs concurrent mutation (parallel inserts -locking individual nodes); a graph being *queried* needs compact, -immutable, cache-friendly layout. qdrant makes them two types: - -- `GraphLayersBuilder` (graph_layers_builder.rs:35) — per-node - `RwLock`'d link lists so threads insert in parallel; holds the - paper's build parameters: `ef_construct` (:38), - `level_factor = 1/ln(M)` (:317 — the paper's mL), - `get_random_layer` (:385, `-ln(sample) * level_factor` at :391), - and `link_new_point` (:414) — Alg 1. The Alg 4 heuristic is a - *flag*: `use_heuristic` (:41-42) — find - `select_candidates_with_heuristic` below it and match the paper. -- `GraphLayers` (graph_layers.rs:74) — the FROZEN serve-side graph: - `search_on_level` (:109), `search_entry` (:248 — the ef=1 greedy - descent). - -The same builder/immutable split as CSR (topic 13): pay a +> **In:** the paper's single conceptual graph. **Out:** two Rust +> types with different layouts, and the reason a builder and a served +> index cannot be the same object. + +A graph being *built* needs concurrent mutation — parallel inserts +each locking individual nodes. A graph being *queried* needs a +compact, immutable, cache-friendly layout. qdrant makes them two +types: + +- **`GraphLayersBuilder`** (`graph_layers_builder.rs:35`) — its link + storage is `links_layers: Vec>>` (:43), + one lock per node per level, so threads insert in parallel. It + holds the paper's build parameters: `ef_construct` (:38), + `level_factor` (:39-40), `use_heuristic` (:41-42), and + `link_new_point` (:414), which is Algorithm 1. +- **`GraphLayers`** (`graph_layers.rs:74`) — the frozen serve-side + graph. `search_on_level` (:109) is Algorithm 2, `search_entry` + (:248) is Algorithm 5's ef=1 descent — its doc comment says + *"Beam size is 1"*. + +The same builder/immutable split as CSR in topic 13: pay a conversion step once, serve reads from the compact form forever. +Two details in the builder are worth checking against the paper +before you trust your mental model. + +```rust +// graph_layers_builder.rs — the level constant and the level draw, +// 317 and 384-393, with the body of new_with_params elided. + 317 level_factor: 1.0 / (max(hnsw_m.m, 2) as f64).ln(), +// ... 318-383: the rest of the constructor, and link bookkeeping ... + 384 fn get_random_layer(&self, rng: &mut R) -> usize + 385 where + 386 R: Rng + ?Sized, + 387 { + 388 let distribution = Uniform::new(0.0, 1.0).unwrap(); + 389 let sample: f64 = rng.sample(distribution); + 390 let picked_level = -sample.ln() * self.level_factor; + 391 picked_level.round() as usize + 392 } +``` + +`level_factor` at :317 is the paper's `mL = 1/ln(M)` exactly (§4.1), +with a floor of `ln 2` so M=1 cannot divide by zero. But :391 calls +**`.round()`** where Algorithm 1 line 4 **floors**. That is not +cosmetic. Under flooring, `P(level ≥ 1) = M^(-1) = 1/16 = 6.3%`; +under rounding, a point is promoted whenever `-ln(U)·mL ≥ 0.5`, i.e. +`U ≤ M^(-1/2) = 1/4`, so **25%** of points get a level above 0 — four +times as many, and correspondingly more upper-layer link memory. + +The Algorithm 4 heuristic is a *flag*, not a given: `use_heuristic` +(:41-42) selects between `link_with_heuristic` (:529) and +`link_without_heuristic` (:555). The heuristic itself lives one file +over, in `links_container.rs`, and its load-bearing line is a single +`continue`: + +```rust +// links_container.rs — fill_from_sorted_with_heuristic, 47-71, +// with the setup lines elided. This is the paper's Algorithm 4. + 47 pub fn fill_from_sorted_with_heuristic( +// ... 48-58: signature, clearing self.links, the outer loop header ... + 59 'outer: for candidate in candidates { + 60 for &existing in self.links.iter() { + 61 if score(candidate.idx, existing) > candidate.score { + 62 continue 'outer; + 63 } + 64 } + 65 self.links.push(candidate.idx); + 66 if self.links.len() >= level_m { + 67 break; + 68 } + 69 } +``` + +Line 61 is Algorithm 4 line 11 (*"if e is closer to q compared to any +element from R"*) with the inequality flipped, because qdrant's +`score` is a **similarity** — higher means closer — so "closer to an +already-kept neighbour than to the query" reads as +`score(candidate, existing) > candidate.score`. Note also what is +*not* here: neither of the paper's `extendCandidates` nor +`keepPrunedConnections` flags exists in this implementation. + ### Step 2 — the search machinery: two heaps and a pooled visited set -Serving a query needs the paper's Alg 2 state: `SearchContext` -(search_context.rs:8) holds the two bounded heaps (candidates -min-heap, results max-heap of size ef). The hot structure is the -**visited set** — every scored node checks and sets membership. -Allocating and zeroing one per query would dominate small searches, -so qdrant pools them: `VisitedListHandle` (visited_pool.rs:9) hands -out reusable lists, "cleared" by bumping a generation stamp instead -of zeroing (:14's comment says exactly this). Your hop_bench stamp -trick, productionized with a pool because queries are concurrent — -each in-flight query borrows its own list. +> **In:** a frozen `GraphLayers` and a query. **Out:** the two +> structures Algorithm 2 needs, and why one of them is pooled rather +> than allocated. + +`SearchContext` (`search_context.rs:8`) is Algorithm 2's state in +one struct: `nearest`, a `FixedLengthPriorityQueue` of size ef (:10 — +the paper's W), and `candidates`, a `BinaryHeap` (:12 — the paper's +C). `new(ef)` is :16-21, `lower_bound` (:23-28) is the stop test's +input, and `process_candidate` (:32-40) is the line-13 admission +test. + +The hot structure is the **visited set**: every scored node checks +and sets membership, so it is touched more often than anything else +in the query. Allocating and zeroing one per query would dominate +small searches, so qdrant pools them. `VisitedListHandle` +(`lib/segment/src/index/visited_pool.rs:9`) hands out reusable lists, +and the "clearing" is a generation stamp: + +```rust +// visited_pool.rs — VisitedList and next_iteration, 19-22 and 78-84, +// with check/check_and_update_visited elided. + 19 pub struct VisitedList { + 20 current_iter: u8, + 21 visit_counters: Vec, + 22 } +// ... 23-77: new(), resize(), check(), check_and_update_visited() ... + 78 fn next_iteration(&mut self) { + 79 self.current_iter = self.current_iter.wrapping_add(1); + 80 if self.current_iter == 0 { + 81 self.current_iter = 1; + 82 self.visit_counters.fill(0); + 83 } + 84 } +``` + +The counters are `u8`, one byte per point, and a query is "cleared" +by incrementing `current_iter`. Work the cost: + +``` + points in the segment n = 100 000 + bytes per point 1 (u8 counter) + ------------------------------------------------ + visited list size 100 000 B = 100 kB + stamps before a real wipe 255 + amortised zeroing per query 100 000 / 255 = 392 bytes +``` + +392 bytes of memset per query instead of 100 kB — a 255× reduction, +and that is only the *amortised* case. `VisitedPool` +(`visited_pool.rs:97-99`) wraps a `RwLock>`, `get` +(:108-120) borrows one, and `return_back` (:122-127) puts it back +subject to `POOL_KEEP_LIMIT`. That last part is the difference from +topic 13's single-threaded stamp trick: queries are concurrent, so +each in-flight query needs its *own* list, and the pool bounds how +many the process keeps alive. ### Step 3 — percolation: why filters shatter graphs +> **In:** an HNSW graph with average level-0 degree K, and a filter +> passing a fraction p of points. **Out:** the critical p below which +> greedy search cannot work at all, and the measurement qdrant takes +> at build time instead of assuming it. + Percolation theory studies when a graph falls apart as you randomly -delete nodes: a random graph with average degree K stays connected -while more than ~1/K of nodes survive, and disintegrates into -islands below that. A filter that rejects nodes during traversal IS -node deletion from the walk's point of view. So each filter has a -critical **selectivity** (the fraction of points that pass): +delete nodes. A random graph with average degree K stays connected +while more than ~1/K of nodes survive and disintegrates into islands +below that. A filter that rejects nodes during traversal *is* node +deletion from the walk's point of view — and in qdrant it literally +is: `point_scorer.rs:231` does +`point_ids.retain(|id| self.filters.check_vector(*id))` **before +scoring**, so a rejected neighbour never enters the candidate heap +and the edge through it does not exist. ``` survival fraction p: 1.0 ────────── ~1/K ──────────── 0.0 @@ -70,61 +202,138 @@ critical **selectivity** (the fraction of points that pass): │ recall cliff ``` -qdrant doesn't assume the threshold — it computes it, in a comment -citing the theory (hnsw/build.rs:378-386): +qdrant does not assume the threshold — it computes one and then +measures around it: ```rust -// According to percolation theory, random graph becomes disconnected -// if 1/K points are left, where K is average number of links per point -let percolation = 1. - 2. / (average_links_per_0_level_int as f32); +// hnsw/build.rs — the percolation sampling point and the +// connectivity measurement, 378-397, with the debug! line elided. + 378 // According to percolation theory, random graph becomes disconnected + 379 // if 1/K points are left, where K is average number of links per point + 380 // So we need to sample connectivity relative to this bifurcation point, but + 381 // not exactly at 1/K, as at this point graph is very sensitive to noise. + 382 // + 383 // Instead, we choose sampling point at 2/K, which expects graph to still be + 384 // mostly connected, but still have some measurable disconnected components. + 385 + 386 let percolation = 1. - 2. / (average_links_per_0_level_int as f32); + 387 + 388 let required_connectivity = if average_links_per_0_level_int >= 4 { + 389 let global_graph_connectivity = [ + 390 graph_layers_builder.subgraph_connectivity(rng, &all_points, percolation), + 391 graph_layers_builder.subgraph_connectivity(rng, &all_points, percolation), + 392 graph_layers_builder.subgraph_connectivity(rng, &all_points, percolation), + 393 ]; +// ... 395: debug!("graph connectivity: ...") ... + 397 global_graph_connectivity +``` + +Read :386 carefully, because the variable name inverts the meaning: +`percolation` is the **drop** fraction, so the *survival* fraction it +samples at is `2/K`, exactly what the comment says. Put qdrant's own +default in: + +``` + K = average links per point on level 0 ≈ m0 = 2·m = 32 + (config.rs:46 sets m0 = m*2; types.rs:1412 sets m = 16; + :369 measures the real average rather than assuming it) + + percolation threshold 1/K = 1/32 = 3.1% ← the cliff + sampling point 2/K = 2/32 = 6.3% ← where qdrant looks + `percolation` variable 1 − 2/32 = 0.9375 ← the DROP fraction ``` -Then it MEASURES: sample subgraph connectivity at the 2/K survival -point (:390-392, three samples, take max), and if the main graph -would shatter for an indexed payload category, add extra -category-aware links (`payload_m`, hnsw.rs:93) so each category's -subgraph is navigable on its own. The failure mode is measured -during build — topic 0 discipline inside an index builder. +So the code deletes 93.75% of points three times (:389-393, +different RNG draws), measures the largest surviving component each +time, and takes the max (:397-400). The guard at :388 skips this +entirely for graphs with average degree below 4, where the +arithmetic is meaningless. + +If the main graph would shatter for an indexed payload category, the +build adds extra category-aware links — `payload_m` (`hnsw.rs:93`, +declared at `types.rs:684-686`, with `payload_m0 = payload_m * 2` at +`config.rs:52`) — so each category's subgraph is navigable on its +own. The failure mode is *measured during build*: topic 0's +discipline, inside an index builder. ### Step 4 — the per-query decision: a planner inside the index -With the cliff located, each query picks its algorithm from an -estimated filter cardinality (the number of points expected to pass -— topic 10's central quantity), in hnsw/search.rs:55-84: +> **In:** a query with an optional filter. **Out:** one of three +> algorithms, chosen from an estimated cardinality — topic 10's +> optimiser, living inside a vector index. + +With the cliff located, each query picks its algorithm. The whole +decision is 27 lines: ```rust -let mut algorithm = SearchAlgorithm::Hnsw; -if acorn_enabled && let Some(filter) = filter { - let query_point_cardinality = - payload_index.with_view(|v| v.estimate_cardinality(filter, ...))?; // :74 - let selectivity = cardinality / available_vector_count; // :80 - if selectivity <= acorn_max_selectivity { algorithm = Acorn; } -} -``` - -Topic 10 inside the vector index: **estimate cardinality, then pick -the plan**. The full menu: - -- selectivity high → normal HNSW, `FilteredScorer` rejects - non-matching points during traversal (the graph stays connected - well above 1/K, so this is safe) -- selectivity low → `search_plain_batched` (:264) — brute-force the - filtered id list; below `full_scan_threshold` the graph can't - help, and scanning 500 survivors exactly beats walking a - shattered graph approximately -- middle → ACORN, Step 5 - -The cost of getting this wrong is asymmetric: brute-forcing a 90% -filter scans ~900K points for nothing; HNSW-ing a 1% filter returns -garbage. Hence a planner, not a constant. +// hnsw/search.rs — the algorithm choice, 59-85, with the NOTE +// comment at 64-67 elided. + 59 let mut algorithm = SearchAlgorithm::Hnsw; + 60 if acorn_enabled + 61 && self.config.m0 != 0 + 62 && let Some(filter) = filter + 63 { +// ... 64-67: a NOTE about unfiltered searches on heavily-deleted segments ... + 69 let available_vector_count = vector_storage.available_vector_count(); + 70 let selectivity = if available_vector_count == 0 { + 71 1.0 + 72 } else { + 73 let query_point_cardinality = + 74 payload_index.with_view(|v| v.estimate_cardinality(filter, &hw_counter))?; + 75 let query_cardinality = adjust_to_available_vectors( + 76 query_point_cardinality, + 77 available_vector_count, + 78 id_tracker.available_point_count(), + 79 ); + 80 query_cardinality.exp as f64 / available_vector_count as f64 + 81 }; + 82 if selectivity <= acorn_max_selectivity { + 83 algorithm = SearchAlgorithm::Acorn; + 84 } + 85 } +``` + +Line 74 is topic 10's cardinality estimator, called from inside a +vector index. Line 80 divides its *expected* value (`.exp`, the +point estimate of a `CardinalityEstimation`) by the live vector count +to get selectivity, after :75-79 rescales the payload index's +estimate — which counts *points* — to the number of *vectors* +actually present in this segment. Line 82's threshold defaults to +`ACORN_MAX_SELECTIVITY_DEFAULT = 0.4` (`lib/segment/src/types.rs:556`). + +The full menu: + +| selectivity | plan | why | +|---|---|---| +| > 0.4 | plain HNSW, `FilteredScorer` rejecting during traversal | the graph stays connected well above 1/K, so deleting nodes from the walk is safe | +| ≤ 0.4, above the size floor | ACORN (Step 5) | connected enough to walk, too sparse to walk naively | +| below `full_scan_threshold` | `search_plain_batched` (`hnsw/search.rs:264`) | the surviving id list is small enough to score exactly, and exact beats an approximate walk over a shattered graph | + +The cost of getting this wrong is asymmetric, which is the argument +for a planner rather than a constant: brute-forcing a 90% filter +scans ~900k points for nothing, while HNSW-ing a 1% filter returns +garbage — and unlike the first mistake, the second one is silent. + +One anchor to keep straight: `get_oversampled_top` appears at +`hnsw/search.rs:57`, but that is only the call site. Its definition +is in a different module, +`lib/segment/src/index/vector_index_search_common.rs:27-45`, and it +belongs to quantization rather than to the planner — see +[reading-qdrant-quantization.md](reading-qdrant-quantization.md). ### Step 5 — ACORN: traverse through the blocked nodes -For the middle band, ACORN (`search_on_level_acorn`, -graph_layers.rs:155) keeps the walk connected WITHOUT extra links: -when expanding a node, also consider its 2-hop neighborhood — -neighbors-of-neighbors — treating filtered-out nodes as passable -wires rather than walls: +> **In:** a query in the awkward band — restrictive enough to shatter +> the graph, broad enough that scanning is wasteful. **Out:** a walk +> that stays connected without any extra links, and the scoring bill +> it runs up. + +For the middle band, qdrant implements **ACORN-1** — the paper's +cheap variant, and `graph_layers.rs:155`'s doc comment names it +that specifically, so do not read the generic ACORN paper and expect +a match. The idea: when expanding a node, treat filtered-out +neighbours as passable wires rather than walls, and reach their +neighbours instead. ``` 1-hop, filtered: ● ──✗── ✗ ──✗── ● walk stops at the wall @@ -132,43 +341,148 @@ wires rather than walls: pass-through only ● gets scored ``` -If a p fraction of nodes pass, 1-hop expansion sees ~K·p useful -edges but 2-hop sees ~K²·p — squaring the degree pushes the -percolation threshold from ~1/K down to ~1/K². The price: more -distance computations per expansion (the 2-hop frontier is bigger), -paid only on queries in the awkward band. Compare `payload_m`'s -extra links (Step 3): RAM paid at build time for known categories vs -CPU paid at query time for arbitrary filters — question 2. +The implementation is precise about who pays: + +```rust +// graph_layers.rs — search_on_level_acorn's expansion, 199-240, +// with the closure boilerplate kept because it is the point. + 199 _ = self.try_for_each_link(candidate.idx, level, |hop1| { + 200 if hop1_visited_list.check_and_update_visited(hop1) { + 201 return ControlFlow::Continue(()); + 202 } + 203 + 204 if points_scorer.filters().check_vector(hop1) { + 205 to_score.push(hop1); + 206 if to_score.len() >= hop1_limit { + 207 return ControlFlow::Break(()); + 208 } + 209 } else { + 210 to_explore.push(hop1); + 211 } + 212 ControlFlow::Continue(()) + 213 }); +// ... 215-218: the 2-hop loop header and the stop check ... + 219 let total_limit = to_score.len() + hop2_limit; + 220 _ = self.try_for_each_link(hop1, level, |hop2| { +// ... 221-226: skip anything either visited list has already seen ... + 227 if points_scorer.filters().check_vector(hop2) { + 228 hop1_visited_list.check_and_update_visited(hop2); + 229 to_score.push(hop2); + 230 if to_score.len() >= total_limit { + 231 return ControlFlow::Break(()); + 232 } + 233 } + 234 ControlFlow::Continue(()) + 235 }); + 236 } + 237 + 238 points_scorer + 239 .score_points_unfiltered(&to_score) + 240 .for_each(|score_point| search_context.process_candidate(score_point)); +``` + +The `else` at :209-211 is the whole design: **only neighbours that +FAIL the filter go into `to_explore`**. A neighbourhood where +everything passes produces an empty `to_explore`, the 2-hop loop +never runs, and ACORN costs exactly what plain HNSW costs. The +expansion is bought only where the filter actually removed something. + +Two limits keep the bill bounded: `hop1_limit` and `hop2_limit` are +both set to `get_m(level)` (:181-182), and the 2-hop pass stops at +`total_limit = to_score.len() + hop2_limit` (:219), so at most about +`2M` points are scored per expansion rather than the `M²` the naive +formulation suggests. Two pooled visited lists (:167, :173) keep the +hop-1 and hop-2 frontiers from re-scoring each other. + +Why it works: if a fraction p of nodes pass, 1-hop expansion sees +~K·p useful edges but 2-hop reaches ~K²·p candidates, pushing the +percolation threshold from ~1/K down toward ~1/K². Put in the +defaults: + +``` + K = m0 = 32 + 1-hop cliff 1/K = 1/32 = 3.1% selectivity + 2-hop cliff 1/K² = 1/1024 = 0.098% selectivity +``` + +Roughly a 32× extension of the usable band — which is why the +threshold at `search.rs:82` can be as generous as 0.4 without the +walk falling apart. The price is more distance computations per +expansion, paid only on queries in the awkward band. Compare +`payload_m`'s extra links from Step 3: RAM paid at build time for +*known* categories versus CPU paid at query time for *arbitrary* +filters — question 2. ### Step 6 — the scar tissue worth grepping -Production remainders, each a small chapter of its own: +> **In:** a working filtered index. **Out:** the four remainders that +> production forces on you, each a small chapter of its own. + +**`full_scan_threshold`** (`hnsw/build.rs:95-104`) decides when not +to build a graph at all. The config knob is in KiB — `types.rs:667` +says *"Minimal size threshold (in KiloBytes)"* with the aside *"Note: +1Kb = 1 vector of size 256"* — and the code turns it into a point +count at build time: + +```rust +// hnsw/build.rs — full_scan_threshold, KiB of vectors to a point +// count, 95-104. + 95 let full_scan_threshold = vector_storage_ref + 96 .size_of_available_vectors_in_bytes() + 97 .checked_div(total_vector_count) + 98 .and_then(|avg_vector_size| { + 99 hnsw_config + 100 .full_scan_threshold + 101 .saturating_mul(BYTES_IN_KB) + 102 .checked_div(avg_vector_size) + 103 }) + 104 .unwrap_or(1); +``` -- `hnsw/build.rs:95-109` — `full_scan_threshold` derives the - "indexing threshold": tiny segments never build a graph at all - (brute force wins below ~thousands of points — the build cost - never amortizes). -- `graph_links.rs` — serialized link format: delta-compressed, - topic 12 encodings applied to graph edges. -- `gpu/` — GPU-built HNSW (topic 18 preview). -- `graph_layers_healer.rs` — repairing the graph around deleted +:97 computes the average vector size in bytes, and :99-102 divides +the configured KiB budget by it. With +`DEFAULT_FULL_SCAN_THRESHOLD = 10_000` (`types.rs:1872`) and +`BYTES_IN_KB = 1024` (`lib/segment/src/common/mod.rs:239`): + +``` + budget = 10 000 KiB × 1024 = 10 240 000 bytes + + d = 128 f32: avg_vector_size = 512 B → 10 240 000 / 512 = 20 000 points + d = 1536 f32: avg_vector_size = 6144 B → 10 240 000 / 6144 = 1 666 points +``` + +That is why the knob is in bytes rather than points: the thing a +brute-force scan is actually limited by is bytes streamed (topic +12), and a 1536-dimensional collection hits the same memory-bandwidth +cost at one twelfth the point count. Note `BYTES_IN_KB` is 1024, so +the doc comment's "KiloBytes" means KiB. + +The other three: + +- **`graph_links.rs`** — the serialised link format, delta-compressed; + topic 12's encodings applied to graph edges. +- **`gpu/`** — GPU-built HNSW (topic 18 preview). +- **`graph_layers_healer.rs`** — repairing the graph around deleted points instead of rebuilding: the paper's deletes wart, patched. ## Where each step lives in the code -All paths relative to `lib/segment/src/index/hnsw_index/`: +Paths are relative to `lib/segment/src/index/hnsw_index/` **except** +where marked. All verified at `qdrant/qdrant@44ad62f`. | step | anchors | |---|---| -| 1 build side | `graph_layers_builder.rs:35` (builder), `:38` ef_construct, `:41-42` use_heuristic, `:317` level_factor, `:385/:391` get_random_layer, `:414` link_new_point | -| 1 serve side | `graph_layers.rs:74` GraphLayers, `:109` search_on_level, `:248` search_entry | -| 2 machinery | `search_context.rs:8` SearchContext, `visited_pool.rs:9/:14` VisitedListHandle | -| 3 percolation | `hnsw/build.rs:378-386` threshold, `:390-392` connectivity sampling, `hnsw.rs:93` payload_m | -| 4 the planner | `hnsw/search.rs:55-84` algorithm choice, `:74` estimate_cardinality, `:80` selectivity, `:264` search_plain_batched | -| 5 ACORN | `graph_layers.rs:155` search_on_level_acorn | -| 6 scar tissue | `hnsw/build.rs:95-109`, `graph_links.rs`, `gpu/`, `graph_layers_healer.rs` | - -Read order: Step 4's `search.rs:55-84` first (it's 30 lines and the +| 1 build side | `graph_layers_builder.rs:35` builder, `:38` ef_construct, `:41-42` use_heuristic, `:43` the RwLock'd link layers, `:317` level_factor, `:384-393` get_random_layer (`.round()` at :391), `:414` link_new_point, `:529`/`:555` the two link paths | +| 1 heuristic | `links_container.rs:47-71` — Algorithm 4; `:61` is the test | +| 1 serve side | `graph_layers.rs:74` GraphLayers, `:109` search_on_level, `:248` search_entry, `:531` search | +| 2 machinery | `search_context.rs:8` SearchContext, `:10`/`:12` the two heaps, `:32-40` process_candidate; **`../visited_pool.rs`** (i.e. `lib/segment/src/index/visited_pool.rs`) `:9` handle, `:19-22` the u8 counters, `:78-84` next_iteration, `:97-127` the pool | +| 3 percolation | `hnsw/build.rs:366-370` measured average degree, `:378-386` the 2/K sampling point, `:388-400` three samples and the max; `hnsw.rs:93` payload_m; `../../types.rs:684-686` its config | +| 4 the planner | `hnsw/search.rs:59-85` the algorithm choice, `:74` estimate_cardinality, `:80` selectivity, `:264` search_plain_batched; `../../types.rs:556` ACORN_MAX_SELECTIVITY_DEFAULT = 0.4; **`../vector_index_search_common.rs:27-45`** get_oversampled_top (the call at `hnsw/search.rs:57` is not the definition) | +| 4 filtering | **`point_scorer.rs:231`** — `retain` before scoring; this is why the graph disconnects | +| 5 ACORN-1 | `graph_layers.rs:155` search_on_level_acorn, `:167`/`:173` two pooled visited lists, `:181-182` the hop limits, `:199-213` 1-hop, `:216-236` 2-hop, `:238-240` scoring | +| 6 scar tissue | `hnsw/build.rs:95-104`; `graph_links.rs`; `gpu/`; `graph_layers_healer.rs` | + +Read order: Step 4's `search.rs:59-85` first (it is 27 lines and the thesis), then chase each branch to its implementation. ## Questions (answer in notes.md) @@ -188,11 +502,109 @@ thesis), then chase each branch to its implementation. ## Done when +Answer each before unfolding it. + - [ ] You can explain why the build structure and the serve structure differ, and what freezing buys. +
Answer + + `GraphLayersBuilder` (`graph_layers_builder.rs:35`) stores links as + `Vec>>` (:43) — a lock and a growable + container per node per level, so N threads can insert concurrently. + That layout is terrible to read: a pointer chase and an atomic per + neighbour list. `GraphLayers` (`graph_layers.rs:74`) is the frozen + form, with links serialised into the compact `graph_links.rs` + representation, no locks, and no growth. Freezing buys sequential + access on the hottest inner loop and removes per-read + synchronisation, at the cost of a one-time conversion and an + inability to mutate — which is exactly why deletes need + `graph_layers_healer.rs`. Same trade as topic 13's CSR. + +
+ - [ ] You can explain percolation: why a selective filter shatters a proximity graph rather than just shrinking it. +
Answer + + Because filtering removes *edges*, not just candidates. + `point_scorer.rs:231` retains only passing ids before scoring, so a + rejected neighbour never enters the candidate heap and every path + through it is gone. A random graph with average degree K loses its + giant connected component once the survival fraction drops below + ~1/K. With qdrant's default `m0 = 2·m = 32` (`config.rs:46`, + `types.rs:1412`) that cliff is at 1/32 = 3.1% selectivity. Below + it, greedy search does not return *worse* results — it returns + results from whichever island the entry point happened to land in, + and increasing `ef` does not help, because the answer is not in the + reachable component. qdrant measures rather than assumes: + `hnsw/build.rs:386` samples at survival 2/K (the variable named + `percolation` is the drop fraction, `1 − 2/K = 0.9375`), three + times with different RNG draws, taking the max (:389-400). + +
+ - [ ] You can describe the per-query decision qdrant makes and name the inputs it uses (cardinality estimate, thresholds). +
Answer + + `hnsw/search.rs:59-85`. Inputs: (1) is there a filter at all (:62); + (2) `estimate_cardinality` from the payload index (:74) — topic + 10's estimator; (3) `available_vector_count` (:69), used both to + rescale the point-level estimate to vectors (:75-79) and as the + denominator of selectivity (:80); (4) + `ACORN_MAX_SELECTIVITY_DEFAULT = 0.4` (`types.rs:556`) at :82; (5) + separately, `full_scan_threshold` (`hnsw/build.rs:95-104`), which + routes tiny survivor sets to `search_plain_batched` (:264). + Outputs: plain HNSW above 0.4 selectivity, ACORN-1 at or below it, + exact scan when the survivor list is small enough. The asymmetry + justifies the machinery: a wrong brute-force choice wastes CPU + visibly; a wrong HNSW choice returns wrong answers silently. + +
+ - [ ] You can say what ACORN's 2-hop expansion costs in scoring work and what it buys in connectivity. +
Answer + + Buys: the reachable neighbourhood grows from ~K·p to ~K²·p, moving + the percolation cliff from 1/K = 3.1% to roughly 1/K² = 0.098% at + m0 = 32 — about a 32× wider usable selectivity band. Costs: extra + link traversals and extra scoring, but bounded. `graph_layers.rs` + sets `hop1_limit = hop2_limit = get_m(level)` (:181-182) and caps + the 2-hop pass at `total_limit = to_score.len() + hop2_limit` + (:219), so an expansion scores at most about 2M points, not M². + Crucially :209-211 puts only *failing* neighbours into + `to_explore`, so a fully-passing neighbourhood triggers no 2-hop + work at all — the cost is proportional to how much the filter + actually removed. Versus `payload_m`: extra links are RAM spent at + build time and only for payload keys you declared; ACORN is CPU + spent at query time and works for arbitrary filters. + +
+ +- [ ] You can compute where the percolation cliff sits for qdrant's default graph, and what the code samples instead. +
Answer + + Default `m = 16` (`types.rs:1412`), `m0 = m*2 = 32` + (`config.rs:46`), so K ≈ 32 — though `build.rs:368-370` measures + the real average rather than assuming it. Cliff: 1/K = 1/32 = + 3.1% survival. qdrant samples at 2/K = 6.3% survival, because the + comment at :380-384 says the graph is too noise-sensitive exactly + at the bifurcation. The variable `percolation` at :386 is + `1 − 2/K = 0.9375`, the fraction *dropped*, which is easy to + misread as the survival fraction. + +
+ - [ ] You wrote answers to all five questions in notes.md, including why `full_scan_threshold` is expressed in bytes. +
Answer + + For question 4 specifically: because the cost a brute-force scan is + bounded by is bytes streamed, not points visited (topic 12). + `hnsw/build.rs:95-104` converts a KiB budget into a point count at + build time by dividing by the measured average vector size, so the + same 10 000 KiB default (`types.rs:1872`) means 20 000 points at + d=128 f32 (512 B each) but only 1 666 points at d=1536 (6144 B + each). A point-count knob would silently mean twelve times more + work on the larger collection. + +
## References @@ -200,15 +612,27 @@ thesis), then chase each branch to its implementation. - Patel, Kraft, Guestrin, Zaharia — "ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data" (SIGMOD 2024, - [arXiv:2403.04871](https://arxiv.org/abs/2403.04871)) — optional; - the 2-hop-expansion idea qdrant adopted + [arXiv:2403.04871](https://arxiv.org/abs/2403.04871)). Optional, + and note qdrant implements the cheaper **ACORN-1** variant, as + `graph_layers.rs:155` says in its own doc comment. - The HNSW paper itself is - [reading-hnsw-paper.md](reading-hnsw-paper.md) - -**Code** -- [qdrant](https://github.com/qdrant/qdrant) — everything under - `lib/segment/src/index/hnsw_index/`: `graph_layers_builder.rs`, - `graph_layers.rs`, `search_context.rs`, `visited_pool.rs`, - `hnsw/search.rs` (the per-query algorithm choice), `hnsw/build.rs` - (the percolation measurement), `graph_links.rs`, - `graph_layers_healer.rs` + [reading-hnsw-paper.md](reading-hnsw-paper.md). + +**Code** — all `qdrant/qdrant@44ad62f`, pinned in +`resources/codebases.md`. + +| file:line | what | +|---|---| +| `lib/segment/src/index/hnsw_index/graph_layers_builder.rs:35,43,317,384-393` | builder, RwLock'd links, mL, the rounded level draw | +| `lib/segment/src/index/hnsw_index/links_container.rs:47-71` | Algorithm 4; `:61` is the heuristic test | +| `lib/segment/src/index/hnsw_index/graph_layers.rs:74,109,155,248` | frozen graph, Alg 2, ACORN-1, the ef=1 descent | +| `lib/segment/src/index/hnsw_index/search_context.rs:8-40` | Algorithm 2's two heaps | +| `lib/segment/src/index/visited_pool.rs:9,19-22,78-84,97-127` | the u8-stamp visited list and its pool | +| `lib/segment/src/index/hnsw_index/point_scorer.rs:231` | `retain` before scoring — the reason filters disconnect | +| `lib/segment/src/index/hnsw_index/hnsw/build.rs:95-104,366-400` | full_scan_threshold; the percolation measurement | +| `lib/segment/src/index/hnsw_index/hnsw/search.rs:59-85,264` | the planner; the exact-scan fallback | +| `lib/segment/src/index/hnsw_index/hnsw.rs:93` | payload_m | +| `lib/segment/src/index/hnsw_index/config.rs:46,48,52` | m0 = 2m, ef defaults to ef_construct, payload_m0 | +| `lib/segment/src/index/vector_index_search_common.rs:27-45` | get_oversampled_top's definition | +| `lib/segment/src/types.rs:556,667-673,684-686,1409-1422,1872` | ACORN threshold, full_scan_threshold docs, payload_m, HNSW defaults | +| `lib/segment/src/common/mod.rs:239` | `BYTES_IN_KB = 1024` | diff --git a/topics/14-vector-search/reading-qdrant-quantization.md b/topics/14-vector-search/reading-qdrant-quantization.md index 63e299d..6e052ad 100644 --- a/topics/14-vector-search/reading-qdrant-quantization.md +++ b/topics/14-vector-search/reading-qdrant-quantization.md @@ -2,161 +2,446 @@ Topic 12's thesis — compression IS performance — with a new twist: here compression is LOSSY, so the system needs machinery to claw the -recall back. This chapter climbs qdrant's three-rung ladder step by +recall back. This chapter climbs qdrant's compression ladder step by step — why lossy codes pay, scalar u8 and the score-without-decode trick, PQ, binary — and ends with the oversample+rescore pipeline that makes lossy codes safe; that pipeline shape is what M14 copies. The encoders live in their own crate, `lib/quantization/src/`; the wiring into search is `lib/segment/src/vector_storage/quantized/`. +Every `file:line` below was read at **`qdrant/qdrant@44ad62f`**, the +pin in `resources/codebases.md`; re-check any of them with +`python3 tools/pinned-source.py show qdrant -r A:B`. Several +figures in the older version of this guide were off — where a number +here contradicts folklore (u8 is *not* 256 levels; the per-vector +extra is *not* Σvᵢ), the anchor is given so you can settle it +yourself. + ## The problem in one sentence A million 1536-d f32 embeddings is **6 GB** of vectors that every HNSW hop pokes at random, so bytes-per-vector is the real cost unit — but every byte saved is precision lost, and distances computed on -compressed codes return the *wrong nearest neighbors* unless +compressed codes return the *wrong nearest neighbours* unless something puts the recall back. +Work the size claim, because it is the reason the chapter exists: + +``` + n = 1 000 000, d = 1536, f32 + vectors = 1e6 × 1536 × 4 B = 6.14 GB + HNSW links (M=16, from reading-hnsw-paper.md's §4.2.3 arithmetic) + = 1e6 × 151 B = 0.15 GB +``` + +Forty times more bytes in vectors than in graph, and unlike the +graph, the vectors are touched in random order — one cache miss per +distance computation. That ratio is why quantization, not graph +compaction, is the lever. + ## The concepts, step by step ### Step 1 — the ladder: three compression rungs, one recall knob -Lossy vector compression trades bytes for distance accuracy, and -qdrant ships three rungs — the topic README's table: +> **In:** f32 vectors. **Out:** the three encodings qdrant ships, +> their real compression ratios, and the one property all three must +> have to be worth anything. + +Lossy vector compression trades bytes for distance accuracy. qdrant +ships three families: + +| scheme | stored per vector (d=128) | ratio | distance on encoded | recall cost | +|---|---|---|---|---| +| scalar u8 | d + 4 = 132 B | 3.88× | integer dot + affine postprocess | small | +| PQ, D* dims/chunk | d/D* bytes; D* ∈ {1,2,4,8,16} | 4×–64× | LUT sums — d/D* lookups | real | +| binary, 1 bit/dim | d/8 = 16 B | 32× | XOR + popcount | large, needs rescore | + +The ratios are not round numbers, and two of them differ from what +gets repeated. Scalar u8 is **3.88×**, not 4×, because +`ADDITIONAL_CONSTANT_SIZE = size_of::()` +(`encoded_vectors_u8.rs:22-23`) prepends four bytes per vector — +`get_quantized_vector_size` is `actual_dim + 4` +(`encoded_vectors_u8.rs:593-596`), and `actual_dim` is itself d +rounded up to a multiple of `ALIGNMENT = 16` (:21, :589-591). PQ's +range is **4×–64×**, set by the `CompressionRatio` enum +(`lib/segment/src/types.rs:749-757`), whose variants map to +dimensions per chunk in +`lib/segment/src/vector_storage/quantized/quantized_vectors.rs:2314-2322`: +X4→1, X8→2, X16→4, X32→8, X64→16. + +Two things make the rungs *fast* rather than merely small. + +First, distance must be computable **on the codes** — decoding to +f32 per candidate would eat the savings, since the decode is the same +arithmetic you were trying to avoid. Each rung has its own trick for +this: Step 2's algebraic expansion, Step 3's lookup table, Step 4's +popcount. + +Second, moving fewer bytes *is* the speedup. HNSW is memory-bound — +this topic's brute-force lane manages 117 QPS while running at 1.5 G +multiply-adds per second, which is not an arithmetic problem — so 4× +smaller codes means 4× more of the working set resident at each level +of cache. + +The recall each rung loses is recovered by one shared mechanism, +Step 5's pipeline, which is why the riskier rungs are usable at all. -| scheme | bytes/dim (f32=4) | distance on encoded | recall cost | -|---|---|---|---| -| scalar u8 | 1 | integer dot + affine postprocess | tiny | -| PQ (m chunks × 256 centroids) | ~0.06–0.5 | LUT sums — d/m table lookups | real | -| binary | 1 bit | XOR + popcount | big, needs rescore | +### Step 2 — scalar u8: the affine trick, and scoring without decode -Two things make the rungs *fast* rather than merely small. First, -distance must be computable ON the codes — decoding to f32 per -candidate would eat the savings. Second, moving fewer bytes is -itself the speedup: HNSW is memory-bound, so 4× smaller codes ≈ 4× -more of the index in cache. The recall each rung loses is recovered -by one shared mechanism — Step 5's pipeline — which is why the -riskier rungs are usable at all. +> **In:** an f32 vector and a per-index scale/offset pair. **Out:** +> one byte per dimension plus one f32 correction term, and a dot +> product computed entirely in integers. -### Step 2 — scalar u8: the affine trick, and scoring without decode +Scalar quantization maps each f32 dimension onto a small integer +through an affine transform. `alpha` (the scale) and `offset` live +in `MetadataInt8` (`encoded_vectors_u8.rs:83-90`, fields at :86-87), +and the encode is two lines: -Scalar quantization maps each f32 dimension to one byte through an -affine transform: store a shared `alpha` (scale) and `offset` -(encoded_vectors_u8.rs:86-87), encode `i = (value - offset) / alpha` -(:95) — 4× fewer bytes, ~256 distinguishable values per dimension. -The clever part is scoring WITHOUT decode — expand the dot product -algebraically: +```rust +// encoded_vectors_u8.rs — encode_value and postprocess_score, 93-102. + 93 #[inline] + 94 pub fn encode_value(&self, value: f32) -> u8 { + 95 let i = (value - self.offset) / self.alpha; + 96 i.clamp(0.0, 127.0).round() as u8 + 97 } + 98 + 99 #[inline] + 100 fn postprocess_score(&self, score: f32, query_offset: f32, vector_offset: f32) -> f32 { + 101 self.multiplier * score + query_offset + vector_offset + 102 } +``` + +Read :96 carefully: the clamp is to **127**, not 255. It is a `u8` +container holding a **7-bit** code, so there are **128 +distinguishable levels per dimension, not 256** — the widely repeated +"256 levels" figure is wrong for this code path. The range fitting +confirms it: `alpha_offset_from_min_max` (:501-505) sets +`alpha = (max - min) / 127.0` and `offset = min`. The reason is +headroom in the SIMD kernels — products of two 7-bit values +accumulate in `i32` (`impl_score_dot`, :780-788) without the +saturation care an 8-bit×8-bit product would need. + +The clever part is scoring WITHOUT decode. Expand the dot product of +two decoded vectors algebraically — qdrant writes the expansion in +its own comments at :208-216: ``` dot(q, v) ≈ Σ (α·qᵢ + off)(α·vᵢ + off) = α² Σ qᵢvᵢ + α·off·(Σqᵢ + Σvᵢ) + d·off² - ↑ integer dot ↑ per-vector precomputed sums + ↑ integer dot ↑ per-vector term ↑ index constant ``` +Only the first term depends on both operands, so only it has to be +computed per candidate — and it is an integer dot product over +bytes. The `multiplier` that scales it back is chosen per metric: + ```rust -// score u8 codes WITHOUT decoding: integer dot + affine correction -fn dot_u8(q: &Encoded, v: &Encoded, alpha: f32, off: f32, d: usize) -> f32 { - let int_dot: u32 = q.codes.iter().zip(&v.codes) - .map(|(&a, &b)| a as u32 * b as u32) - .sum(); // the u8 loop SIMD loves - alpha * alpha * int_dot as f32 - + alpha * off * (q.sum + v.sum) // Σqᵢ, Σvᵢ: stored per vector - + d as f32 * off * off // constant for the whole index +// encoded_vectors_u8.rs — the per-metric multiplier, 207-217. + 207 let multiplier = match vector_parameters.distance_type { + 208 // (alpha*x - offset) * (alpha*y - offset) = alpha^2*x*y - alpha*offset*x - alpha*offset*y + offset^2 + 209 // multiplier is applied to xy term only, so we need to multiply score by alpha^2 + 210 DistanceType::Dot | DistanceType::Cosine => alpha * alpha, + 211 // |(alpha*x - offset) - (alpha*y - offset)| = alpha*|x - y| + 212 // multiplier is applied to |x - y| term only, so we need to multiply score by alpha + 213 DistanceType::L1 => alpha, + 214 // ((alpha*x - offset) - (alpha*y - offset))^2 = alpha^2*(x - y)^2 = alpha^2*x^2 - 2*alpha^2*xy + alpha^2*y^2 + 215 // multiplier is applied to (x - y)^2 term only, so we need to multiply score by -2*alpha^2 + 216 DistanceType::L2 => -2.0 * alpha * alpha, + 217 }; +``` + +Now the correction. It is **not** a stored `Σvᵢ`, as is often +claimed — everything metric-dependent is folded into a *single* f32 +written at the front of each encoded vector: + +```rust +// encoded_vectors_u8.rs — the per-vector correction term, 253-275, +// with the invert branch at 267-271 elided. + 253 let vector_offset = match vector_parameters.distance_type { + 254 DistanceType::Dot | DistanceType::Cosine => { + 255 let elements_sum = encoded_vector.iter().map(|&x| f32::from(x)).sum::(); + 256 elements_sum * alpha * offset + 257 } + 258 DistanceType::L1 => 0.0, + 259 DistanceType::L2 => { + 260 let elements_sqr_sum = encoded_vector + 261 .iter() +// ... 262-265: .map(|&x| x*x).sum() * alpha * alpha ... + 266 }; +// ... 267-271: negate if vector_parameters.invert ... + 272 // apply `a^2` shift + 273 let vector_offset = metadata.get_shift() + vector_offset; + 274 encoded_vector[0..ADDITIONAL_CONSTANT_SIZE] + 275 .copy_from_slice(&vector_offset.to_ne_bytes()); +``` + +`elements_sum` at :255 is Σ of the *encoded* bytes, immediately +multiplied by `alpha·offset` (:256), then the whole-index constant +`d·off²` from `get_shift()` (:115-130) is added in at :273 and the +result stored as four bytes at :274-275. For L2 it is a sum of +squares instead (:259-265). So what sits in front of each vector is +one already-scaled f32, not a raw sum — and `postprocess_score` +(:100-102) is a single fused multiply-add-add. + +```rust +// ILLUSTRATION — not quoted from any file; this is the algebra above +// in one function. The real code is split: the integer loop is +// encoded_vectors_u8.rs:780-788 (impl_score_dot), the correction is +// :100-102 (postprocess_score), and the SIMD kernels at :800-813 are +// `unsafe extern "C"` — implemented in C, not Rust. +fn dot_u8(q: &Encoded, v: &Encoded, multiplier: f32) -> f32 { + let int_dot: i32 = q.codes.iter().zip(&v.codes) + .map(|(&a, &b)| a as i32 * b as i32) + .sum(); // the byte loop SIMD loves + multiplier * int_dot as f32 // alpha² for Dot/Cosine + + q.correction // one f32, prepended + + v.correction // to each encoded vector } ``` -`postprocess_score` (:61, :100) applies the affine correction using -per-vector sums stored alongside the codes. The payoff is double: -4× fewer bytes moved AND an integer inner loop that vectorizes -beautifully (topic 17 will SIMD exactly this). One refinement: -`quantile.rs` picks the encoding range from quantiles rather than -min/max, so one outlier dimension doesn't waste alpha on the tails. -Recall cost: tiny — which is why u8 is the default rung for HNSW. +Work the storage: + +``` + d = 128, f32 source + actual_dim = 128 rounded up to a multiple of ALIGNMENT(16) = 128 + stored = actual_dim + ADDITIONAL_CONSTANT_SIZE = 128 + 4 = 132 B + original = 128 × 4 = 512 B + ratio = 512 / 132 = 3.88× + + d = 100 (not a multiple of 16) + actual_dim = 112, stored = 116 B, original = 400 B → 3.45× +``` + +One refinement worth knowing: `find_quantile_interval` +(`lib/quantization/src/quantile.rs:35-80`) picks alpha and offset from +a quantile of a *sample* rather than from min/max, trimming +`cut_index` values off each tail (:62-66) so a single outlier +dimension does not spend the whole 0..127 range on empty space. +`ScalarQuantizationConfig.quantile` (`lib/segment/src/types.rs:766-779`) +is the user-facing knob, validated to [0.5, 1.0], and :42 short-circuits +for fewer than 127 vectors, where there is nothing to estimate. + +Recall cost: small — which is why scalar is the default rung for +HNSW. ### Step 3 — product quantization: bytes per vector, not per dimension -PQ (the full derivation is [reading-pq.md](reading-pq.md); here, -the qdrant-shaped summary) splits the vector into m chunks and -replaces each chunk with the id of its nearest learned centroid — -`CENTROIDS_COUNT = 256` (encoded_vectors_pq.rs:30) so each chunk -codes as exactly one byte. Codebooks come from k-means over a 10K -sample (:27-29 — BtrBlocks-style sampling, topic 12), max 100 -iterations. Scoring uses ADC (asymmetric distance computation): -`EncodedQueryPQ` (:39-41) precomputes a `[chunks × 256]` table of -exact sub-distances per query; each candidate then costs -d/chunk_size table lookups + adds, no float math (:32 -`EncodedVectorsPQ` holds the codes, :46 `Metadata.centroids` the -codebooks). - -PQ trades multiply-adds for L1-resident lookups and reaches -16–64× compression. The cost surfaces in the graph: distances -become approximate EVERYWHERE, so HNSW traversal itself degrades — -wrong distances mean wrong hops, and errors compound along the walk -— which is why qdrant defaults to scalar for HNSW and PQ mostly for -memory-starved setups (question 2). +> **In:** the full PQ derivation from [reading-pq.md](reading-pq.md). +> **Out:** where each piece of it is in qdrant, and the one property +> that makes PQ riskier inside a graph than inside a flat scan. + +PQ splits the vector into chunks and replaces each chunk with the id +of its nearest learned centroid. qdrant's constants are at the top of +the file: `CENTROIDS_COUNT = 256` (`encoded_vectors_pq.rs:30`) so +each chunk codes as exactly one byte, with codebooks from k-means +over a `KMEANS_SAMPLE_SIZE = 10_000` sample (:27), capped at +`KMEANS_MAX_ITERATIONS = 100` (:28) and `KMEANS_ACCURACY = 1e-5` +(:29). Sampling rather than full-corpus training is BtrBlocks-style +(topic 12). + +Scoring is ADC: `EncodedQueryPQ` (:38-43) holds a `lut: Vec` +whose doc comment says exactly what it is — *"Lookup table is a +distance from each query chunk to each centroid related to this +chunk"*. `encode_query` (:515-537) builds it, sizing it at :516 as +`vector_division.len() * centroids.len()` and filling each cell at +:518-534 with an exact sub-distance. `Metadata` (:45-50) holds the +codebooks: `centroids` at :47 and `vector_division` — the chunk +ranges — at :48. Note qdrant stores each of the 256 centroids as a +*full-dimension* vector and slices it per chunk at :522, rather than +storing m separate small codebooks. + +The scan (`score_point_simple`, :474-489) is one f32 load per code +byte, strided by `centroids_count`, summed; :407-440 and :442-472 are +the SSE and NEON versions, unrolling four chunks at a time. + +Put qdrant's compression settings into the LUT arithmetic: + +``` + LUT bytes = (d / D*) × 256 × 4 + + d = 128, X64 (D* = 16) → m = 8 chunks → 8 kB ✓ L1 + d = 128, X32 (D* = 8) → m = 16 chunks → 16 kB ✓ L1 (32-48 kB typical) + d = 128, X4 (D* = 1) → m = 128 chunks → 128 kB ✗ L2 at best +``` + +So the highest-fidelity PQ setting is the one whose lookup table +stops fitting in L1 — the trade is not monotonic, and §II-B of the PQ +paper warns about precisely this. + +The cost that is specific to graphs: PQ makes distances approximate +**everywhere**, including inside HNSW traversal, so wrong distances +mean wrong hops and the errors compound along the walk. A flat IVF +scan uses approximate distances only to *rank* a fixed candidate set, +so an error changes a position; in a graph an error changes which +candidates you ever see. That asymmetry is question 2, and it is why +qdrant defaults to scalar for HNSW and reaches for PQ mainly in +memory-starved setups. ### Step 4 — binary: one bit per dimension +> **In:** an f32 vector. **Out:** d bits, a Hamming distance, and the +> reason this rung is unusable without Step 5. + The bottom rung keeps only the sign of each dimension: -`EncodedVectorsBin` (encoded_vectors_binary.rs:26), 32× compression. -Distance collapses to Hamming distance (the count of differing -bits), computed as XOR + popcount — `xor_popcnt` (:144) with -SSE/NEON paths (:165-190): a 1536-d comparison becomes ~48 64-bit -XOR+popcount ops, a few cycles total. The recall cost is big by -construction — 1 bit can't rank close neighbors — so binary is only -sane WITH rescoring (Step 5), and mainly for high-d embeddings -(1024-d+) where sign patterns carry most of the angle information. +`EncodedVectorsBin` (`encoded_vectors_binary.rs:26`). At one bit per +dimension the ratio against f32 is `32/1 = 32×`. + +Two corrections to the folk version, both from the pinned file. +First, binary quantization here is a *family*, not a single scheme: +`Encoding` (:33-39) offers `OneBit`, `TwoBits` and +`OneAndHalfBits`, so "32×" is the `OneBit` default and the others are +16× and about 21×. Second, the query need not use the same encoding +as storage: `QueryEncoding` (:47-53) offers `SameAsStorage`, +`Scalar4bits` and `Scalar8bits`, which is the PQ paper's +*asymmetric* idea applied to bits — keep the stored side at one bit, +spend more on the query, get better ranking for free +(`xor_popcnt_scalar`, :151, and the dispatch at :336-395). + +The symmetric case collapses to Hamming distance, computed as XOR +plus popcount: + +```rust +// encoded_vectors_binary.rs — BitsStoreType::xor_popcnt for u8, +// 158-209, with the NEON branch at 186-203 elided. + 158 fn xor_popcnt(v1: &[Self], v2: &[Self]) -> usize { + 159 debug_assert!(v1.len() == v2.len()); + 160 + 161 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))] + 162 if is_x86_feature_detected!("sse4.2") { + 163 unsafe { + 164 if v1.len() > 16 { + 165 return impl_xor_popcnt_sse_uint128( + 166 v1.as_ptr(), + 167 v2.as_ptr(), + 168 (v1.len() as u32) / 16, + 169 ) as usize; +// ... 170-184: the /8 and /4 fallbacks, then NEON at 186-203 ... + 205 let mut result = 0; + 206 for (&b1, &b2) in v1.iter().zip(v2.iter()) { + 207 result += (b1 ^ b2).count_ones() as usize; + 208 } + 209 result + 210 } +``` + +Count the work for a realistic embedding, since the usual "~48 ops" +claim does not survive division: + +``` + d = 1536, OneBit + bits = 1536 + bytes = 1536 / 8 = 192 B + u128 lanes= 192 / 16 = 12 ← what :164-169 dispatches + u64 words = 192 / 8 = 24 ← 24 XORs + 24 popcounts + + compare f32: 1536 × 4 = 6144 B and 1536 multiply-adds +``` + +Twelve SIMD iterations against 1 536 multiply-adds, on 192 bytes +instead of 6 144. There is also a `u128` `BitsStoreType` impl (:287) +that dispatches to AVX-512 `vpopcntdq` when available (:292, :299). + +The recall cost is large by construction — one bit cannot rank close +neighbours, only separate hemispheres — so binary is only sane WITH +rescoring, and mainly for high-dimensional embeddings (1024-d and +up) where the sign pattern still carries most of the angular +information. ### Step 5 — oversample + rescore: the recall clawback +> **In:** a quantized index that returns approximately-ranked +> results. **Out:** exactly-ranked results, for a per-query cost that +> does not scale with n. + The shared safety net: search the quantized index for MORE than you need, then re-rank the shortlist with the exact vectors. -`get_oversampled_top` -(lib/segment/src/index/hnsw_index/hnsw/search.rs:57) fetches -`top × oversampling` candidates using the cheap codes, rescores that -shortlist with original f32 vectors, and cuts to `top`: ``` query ──► HNSW over u8/PQ/bin codes ──► top·x candidates - │ rescore with f32 - ▼ - top k + │ rescore with f32 + ▼ + top k ``` -The arithmetic that makes it free-ish: at top=10, oversampling 4× -means exactly 40 f32 distance computations per query — noise next to -the ~thousands of code-distance computations the traversal did. The -quantization error only has to keep the true neighbors *inside the -top 40*, a far weaker demand than ranking them correctly. This is -late materialization (topic 12): cheap representation for the scan, -expensive one only for survivors. `quantized_scorer_builder.rs` -picks the scorer per collection config; storage variants -(RAM/mmap/chunked) live next to it. +The multiplier comes from `get_oversampled_top`. The call site is +`lib/segment/src/index/hnsw_index/hnsw/search.rs:57`, but the +**definition is in a different module** — +`lib/segment/src/index/vector_index_search_common.rs:27-45` — which +is worth knowing before you go looking: + +```rust +// vector_index_search_common.rs — get_oversampled_top, 27-45. + 27 pub fn get_oversampled_top( + 28 quantized_storage: Option<&QuantizedVectors>, + 29 params: Option<&SearchParams>, + 30 top: usize, + 31 ) -> usize { + 32 let quantization_enabled = is_quantized_search(quantized_storage, params); +// ... 34-37: read params.quantization.oversampling, else the default ... + 39 match oversampling_value { + 40 Some(oversampling) if quantization_enabled && oversampling > 1.0 => { + 41 (oversampling * top as f64) as usize + 42 } + 43 _ => top, + 44 } + 45 } +``` + +Three guards at :40: quantization has to be on, the factor has to +exceed 1.0, and it has to be set at all — otherwise `top` passes +through unchanged (:43). `is_quantized_search` (:15-25) is where the +`exact` and `ignore` search params turn quantization off entirely. + +The arithmetic that makes it nearly free: + +``` + top = 10, oversampling = 4.0, d = 1536 + candidates rescored = (4.0 × 10) = 40 + f32 work for the rescore = 40 × 1536 = 61 440 multiply-adds + code-distance work already done by an ef=64 HNSW walk + ≈ thousands of candidates × 192 B each + + and the brute-force alternative for n = 1e6: + 1e6 × 1536 = 1.54 × 10⁹ multiply-adds — 25 000× more +``` + +Forty exact distances is noise. And note what the weaker demand +buys: quantization error only has to keep the true neighbours +*inside the top 40*, which is a far weaker requirement than ranking +them correctly. This is late materialization (topic 12): cheap +representation for the scan, expensive one only for survivors. + +`postprocess_search_result` +(`vector_index_search_common.rs:48`) is where the shortlist is +trimmed back to `top`; +`lib/segment/src/vector_storage/quantized/quantized_scorer_builder.rs` +picks the scorer per collection config, and the RAM/mmap/chunked +storage variants live beside it. ## Where each step lives in the code -Encoders (`lib/quantization/src/`): +All at `qdrant/qdrant@44ad62f`. + +**Encoders** (`lib/quantization/src/`): -- **Step 2 — scalar**: `encoded_vectors_u8.rs` — `:86-87` - alpha/offset, `:95` the quantize expression, `:61/:100` - `postprocess_score`; `quantile.rs` — the outlier-clipping range. -- **Step 3 — PQ**: `encoded_vectors_pq.rs` — `:30` CENTROIDS_COUNT, - `:27-29` the k-means sample, `:32` EncodedVectorsPQ, `:39-41` - EncodedQueryPQ (the ADC table), `:46` Metadata.centroids. -- **Step 4 — binary**: `encoded_vectors_binary.rs` — `:26` - EncodedVectorsBin, `:144` xor_popcnt, `:165-190` the SSE/NEON - paths. +| step | anchors | +|---|---| +| 2 scalar | `encoded_vectors_u8.rs:21-23` ALIGNMENT / ADDITIONAL_CONSTANT_SIZE, `:83-90` MetadataInt8, `:93-97` encode_value (**clamp to 127**), `:100-102` postprocess_score, `:115-130` get_shift, `:207-217` the per-metric multiplier, `:253-275` the folded correction term, `:501-505` alpha from min/max, `:589-596` actual_dim and `+ 4`, `:780-788` impl_score_dot, `:800-813` the C SIMD kernels | +| 2 range | `quantile.rs:35-80` find_quantile_interval; `:42` the <127 short-circuit | +| 3 PQ | `encoded_vectors_pq.rs:27-30` k-means constants and CENTROIDS_COUNT, `:32` EncodedVectorsPQ, `:38-43` EncodedQueryPQ (the LUT), `:45-50` Metadata (centroids at :47, vector_division at :48), `:474-489` the ADC scan, `:515-537` encode_query | +| 4 binary | `encoded_vectors_binary.rs:26` EncodedVectorsBin, `:33-39` Encoding (OneBit / TwoBits / OneAndHalfBits), `:47-53` QueryEncoding, `:144`/`:151` the trait methods, `:158-210` the u8 impl, `:287-321` the u128 impl with AVX-512 vpopcntdq | -Wiring (`lib/segment/src/`): +**Wiring** (`lib/segment/src/`): -- **Step 5 — the pipeline**: `index/hnsw_index/hnsw/search.rs:57` - `get_oversampled_top`; - `vector_storage/quantized/quantized_scorer_builder.rs` (scorer - selection) and the RAM/mmap/chunked storage variants beside it. +| step | anchors | +|---|---| +| 1 config | `types.rs:749-757` CompressionRatio, `:761-764` ScalarType, `:766-779` ScalarQuantizationConfig, `:798-805` ProductQuantizationConfig | +| 3 ratio → D* | `vector_storage/quantized/quantized_vectors.rs:2314-2322` get_bucket_size | +| 5 pipeline | `index/vector_index_search_common.rs:15-25` is_quantized_search, **`:27-45` get_oversampled_top** (the call at `index/hnsw_index/hnsw/search.rs:57` is not the definition), `:48` postprocess_search_result; `vector_storage/quantized/quantized_scorer_builder.rs` scorer selection | -Read order: `encoded_vectors_u8.rs` end to end first (it's the +Read order: `encoded_vectors_u8.rs` end to end first (it is the smallest and carries the score-without-decode idea), then -`get_oversampled_top`, then PQ/binary as variations. +`get_oversampled_top`, then PQ and binary as variations. ## Questions (answer in notes.md) @@ -174,24 +459,133 @@ smallest and carries the score-without-decode idea), then ## Done when +Answer each before unfolding it. + - [ ] You can name the three rungs of the ladder and the compression ratio each achieves. +
Answer + + Scalar u8: one byte per dimension plus a 4-byte per-vector + constant, so `512/132 = 3.88×` at d=128, not the round 4× — the + extra is `ADDITIONAL_CONSTANT_SIZE` + (`encoded_vectors_u8.rs:22-23`, size at :593-596), and `actual_dim` + is d rounded up to a multiple of 16 (:589-591). PQ: one byte per + chunk, with `CompressionRatio` X4…X64 (`types.rs:749-757`) mapping + to 1, 2, 4, 8, 16 dimensions per chunk + (`quantized_vectors.rs:2314-2322`), so **4×–64×**. Binary: one bit + per dimension, `32×` for the `OneBit` default — but + `encoded_vectors_binary.rs:33-39` also offers `TwoBits` (16×) and + `OneAndHalfBits`, so "binary = 32×" needs the qualifier. + +
+ - [ ] You can derive the u8 affine dot-product expansion and say what must be stored per vector for it to work. +
Answer + + With `v ≈ α·code + offset`, + `dot(q,v) = α²·Σqᵢvᵢ + α·off·(Σqᵢ + Σvᵢ) + d·off²` — qdrant writes + the same expansion in comments at `encoded_vectors_u8.rs:208-216`. + Only the first term is a function of both operands, so only it runs + per candidate, as an integer dot over bytes (`impl_score_dot`, + :780-788, accumulating in i32). What is actually stored per vector + is **not** a raw `Σvᵢ`: :253-266 computes `Σcode · α · off` for + Dot/Cosine (or `Σcode² · α²` for L2), :273 folds in the whole-index + constant `d·off²` from `get_shift()` (:115-130), and :274-275 + writes the single resulting f32 into the four bytes at the front of + the encoded vector. `postprocess_score` (:100-102) then costs one + multiply and two adds. Also worth stating: the code is clamped to + **127** (:96), so it is 7-bit — 128 levels per dimension, not 256. + +
+ - [ ] You can explain why PQ hurts HNSW traversal more than it hurts a flat IVF scan. +
Answer + + Because in a graph the distance function chooses which vectors you + ever look at. A flat IVF scan visits a candidate set determined by + the coarse quantizer, and PQ error only perturbs the *ranking* + within it — a mistake costs you a position. In HNSW, each hop + selects the next node from the distance estimates, so an error + routes the walk somewhere else, that wrong node's neighbourhood + supplies the next candidates, and the error compounds along the + path. Increasing `ef` mitigates but does not fix it, because the + beam is being steered by the same corrupted signal. This is why + qdrant treats scalar as the default rung for HNSW: at 128 levels + per dimension the ranking of near neighbours is usually preserved, + so the hops are the same hops. + +
+ - [ ] You can say what oversample-and-rescore claws back, and predict where it lands against this topic's brute-force point before implementing `quant.rs`. +
Answer + + It claws back *ranking*, not *reachability*: the quantized search + only has to place the true neighbours somewhere inside the + oversampled shortlist, and the exact f32 pass then orders them + correctly. `get_oversampled_top` + (`vector_index_search_common.rs:27-45`) computes the shortlist as + `(oversampling × top)` at :41, gated at :40 on quantization being + enabled and the factor exceeding 1.0. Cost: at top=10 and + oversampling 4.0, that is 40 exact distances — `40 × 1536 = 61 440` + multiply-adds at d=1536, against `1e6 × 1536 = 1.54 × 10⁹` for the + brute-force alternative, a 25 000× difference. So the rescore is + effectively free and the whole recall/latency question stays with + the quantized traversal. Against this topic's 117 QPS / recall + 1.000 point, expect the quantized+rescore lane to sit far to the + right on QPS with recall close to but below 1.000 — and record the + measured pair rather than this prediction. + +
+ +- [ ] You can say where `get_oversampled_top` is actually defined, and why the distinction matters. +
Answer + + Defined in `lib/segment/src/index/vector_index_search_common.rs:27-45`; + `lib/segment/src/index/hnsw_index/hnsw/search.rs:57` is only the + call site. It matters because the function is shared machinery — + `is_quantized_search` (:15-25) and `postprocess_search_result` + (:48) live beside it and are used by the non-HNSW index paths too, + so oversampling is a property of quantized search in general rather + than of the HNSW planner. Reading it inside `hnsw/search.rs` would + suggest the planner owns it, which it does not. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the M14 rung decision. +
Answer + + For question 4 specifically, the arithmetic is + `LUT = m × 256 × 4 B`: at m=16 that is 16 kB, which fits a typical + 32-48 kB L1d alongside the codes being scanned. At m=64 it is + 64 kB, which does not — every lookup becomes an L2 access and the + "free" table lookup stops being free. qdrant reaches m=128 at d=128 + with `CompressionRatio::X4` (128 kB of table), which is the + configuration where the highest nominal fidelity buys the worst + locality. + +
## References **Papers** -- Jégou, Douze, Schmid — the PQ paper (IEEE TPAMI 2011) — gets its - own chapter: [reading-pq.md](reading-pq.md) - -**Code** -- [qdrant](https://github.com/qdrant/qdrant) — encoders in - `lib/quantization/src/` (`encoded_vectors_u8.rs`, - `encoded_vectors_pq.rs`, `encoded_vectors_binary.rs`, - `quantile.rs`); wiring in - `lib/segment/src/vector_storage/quantized/` - (`quantized_scorer_builder.rs` and the storage variants) and - `lib/segment/src/index/hnsw_index/hnsw/search.rs` - (`get_oversampled_top`) +- Jégou, Douze, Schmid — the PQ paper (IEEE TPAMI 33(1), 2011) — + gets its own chapter: [reading-pq.md](reading-pq.md). §II-B's + warning about the lookup table outgrowing cache is the one to have + in mind while reading Step 3. + +**Code** — all `qdrant/qdrant@44ad62f`, pinned in +`resources/codebases.md`. + +| file:line | what | +|---|---| +| `lib/quantization/src/encoded_vectors_u8.rs:93-97` | `encode_value` — the clamp is to **127**, so 7-bit | +| `lib/quantization/src/encoded_vectors_u8.rs:100-102,115-130` | `postprocess_score`, `get_shift` | +| `lib/quantization/src/encoded_vectors_u8.rs:207-217` | per-metric multiplier, with the algebra in comments | +| `lib/quantization/src/encoded_vectors_u8.rs:253-275` | the single folded f32 correction, written at the front | +| `lib/quantization/src/encoded_vectors_u8.rs:501-505,589-596` | `alpha = (max-min)/127`; size = `actual_dim + 4` | +| `lib/quantization/src/quantile.rs:35-80` | quantile range fitting instead of min/max | +| `lib/quantization/src/encoded_vectors_pq.rs:27-30,38-50,474-489,515-537` | k-means constants, the LUT, the ADC scan | +| `lib/quantization/src/encoded_vectors_binary.rs:26,33-39,47-53,158-210,287-321` | the binary family, asymmetric query encodings, XOR+popcount | +| `lib/segment/src/types.rs:749-757,766-779,798-805` | CompressionRatio and the two quantization configs | +| `lib/segment/src/vector_storage/quantized/quantized_vectors.rs:2314-2322` | compression ratio → dimensions per chunk | +| `lib/segment/src/index/vector_index_search_common.rs:15-48` | `is_quantized_search`, `get_oversampled_top`, `postprocess_search_result` | +| `lib/segment/src/index/hnsw_index/hnsw/search.rs:57` | the call site only | diff --git a/topics/14-vector-search/reading-usearch.md b/topics/14-vector-search/reading-usearch.md index 6f2a729..97e43c6 100644 --- a/topics/14-vector-search/reading-usearch.md +++ b/topics/14-vector-search/reading-usearch.md @@ -10,6 +10,21 @@ concurrent inserts. This chapter assumes [reading-hnsw-paper.md](reading-hnsw-paper.md) — the algorithms (Alg 1/2/4, M, M0, ef) are used here by name. +Every `file:line` below is from **unum-cloud/usearch@9fd6b01**, the +revision pinned in `resources/codebases.md`. Two files matter: +`include/usearch/index.hpp` (5033 lines — the graph) and +`include/usearch/index_plugins.hpp` (4275 lines — metrics, scalar +types, SIMD dispatch). Reproduce any snippet with +`python3 tools/pinned-source.py show usearch include/usearch/index.hpp -r A:B`. + +**Terms used below, defined once.** *Slot* — usearch's internal +integer index for a member (`compressed_slot_t`, `std::uint32_t` at +`index.hpp:2128`), distinct from the user-visible *key* +(`default_key_t`, `std::uint64_t` at `:2127`). *Tape* — the single +byte buffer holding everything about one node's graph presence. +*Connectivity* — usearch's name for the paper's M. *Expansion* — +usearch's name for the paper's ef. + ## The problem in one sentence Every hop in an HNSW search is a random memory access, so if a @@ -19,130 +34,459 @@ misses (~100 ns apiece) instead of 1 — and at ~300 node visits per query that's the difference between ~30 µs and ~90 µs before a single distance is computed. +Hold that against this topic's measured baseline: brute force does +**117 QPS at recall 1.000** — 8.5 ms per query — by streaming 51 MB +of contiguous f32 at 1.50 × 10⁹ MAC/s. A graph index wins only if +its pointer chase costs far less than that stream. 90 µs of pure +misses is still 90× faster than 8.5 ms, which is exactly why the +layout question is worth a whole chapter rather than a footnote: it +decides how much of that 90× margin you keep. + ## The concepts, step by step ### Step 1 — what an HNSW node must store -An HNSW index is, per node: a level (how high the node reaches in -the hierarchy), and one neighbor list per layer from its level down -to 0 — up to M ids per upper layer, M0 = 2M at layer 0 — plus the -vector itself. The natural first implementation is a -`Vec>` per node (one inner Vec per level): easy to grow, -but every inner Vec is its own heap allocation somewhere else in -memory. Search touches nodes in data-dependent order (topic 0's -pointer chase), so layout — where those lists physically live — is -the entire performance story of an in-RAM implementation. +> **In:** the paper's data model — levels, per-level neighbor lists, +> the vector. **Out:** the list of fields a node must carry, and why +> the obvious `Vec>` encoding is a layout decision disguised +> as a data-structure choice. + +An HNSW index is, per node: a **key** (the caller's id), a **level** +(how high the node reaches in the hierarchy), and one neighbor list +per layer from its level down to 0 — up to M ids per upper layer, +M0 = 2M at layer 0 — plus the vector itself. usearch fixes the widths +in three `using` declarations: + +```cpp +// index.hpp — the width of every field on the tape, 2127-2128, 2335-2341 + 2127 using default_key_t = std::uint64_t; + 2128 using default_slot_t = std::uint32_t; +// ... 2129-2334: allocators, queues, distance types ... + 2335 using neighbors_count_t = std::uint32_t; + 2336 using level_t = std::int16_t; + 2337 + 2338 /** + 2339 * @brief How many bytes of memory are needed to form the "head" of the node. + 2340 */ + 2341 static constexpr std::size_t node_head_bytes_() { return sizeof(vector_key_t) + sizeof(level_t); } +``` + +So the head is `8 + 2 = 10` bytes: **key first, then level**. That +ordering matters in Step 2 and is the thing most descriptions of the +tape get wrong. + +The natural first implementation is a `Vec>` per node (one +inner Vec per level): easy to grow, but every inner Vec is its own +heap allocation somewhere else in memory. Search touches nodes in +data-dependent order (topic 0's pointer chase), so layout — where +those lists physically live — is the entire performance story of an +in-RAM implementation. ### Step 2 — the node tape: one allocation, all levels adjacent +> **In:** the field list from Step 1. **Out:** the exact byte layout, +> the offset arithmetic that replaces pointer hops, and a +> byte-for-byte comparison against `Vec>`. + usearch stores everything about a node's graph presence in one -contiguous byte buffer — the "tape": the level first, then each -layer's neighbor slot (a count followed by ids), preallocated to the -connectivity limits so nothing ever moves: +contiguous byte buffer — the "tape". The authoritative description is +the doc comment on `node_t`: +```cpp +// index.hpp — the tape layout, verbatim, 2364-2372 + 2364 /** + 2365 * @brief A loosely-structured handle for every node. One such node is created for every member. + 2366 * To minimize memory usage and maximize the number of entries per cache-line, it only + 2367 * stores to pointers. The internal tape starts with a `vector_key_t` @b key, then + 2368 * a `level_t` for the number of graph @b levels in which this member appears, + 2369 * then the { `neighbors_count_t`, `compressed_slot_t`, `compressed_slot_t` ... } sequences + 2370 * for @b each-level. + 2371 */ + 2372 class node_t { ``` - node tape: ┌───────┬────────────────┬──────────┬─────┐ - │ level │ L0: cnt + M0×id │ L1: cnt+M×id │ ... │ - └───────┴────────────────┴──────────┴─────┘ + +``` + node tape: ┌─────┬───────┬────────────────────┬──────────────────┬─────┐ + │ key │ level │ L0: cnt + M0 × slot │ L1: cnt + M × slot │ ... │ + │ 8 B │ 2 B │ 4 + 32×4 = 132 B │ 4 + 16×4 = 68 B │ │ + └─────┴───────┴────────────────────┴──────────────────┴─────┘ one allocation, all levels adjacent ``` -Finding layer l's neighbors is pure offset arithmetic — no pointer -hops: +The sizes are not guesses; `precompute_` computes them once per +index and `node_bytes_` turns them into an offset: -```rust -// the tape: level header, then per-level slots preallocated to the -// connectivity limit — neighbors(l) is offset arithmetic, not Vec hops -struct NodeTape<'a> { bytes: &'a [u8] } // one allocation per node - -impl NodeTape<'_> { - fn neighbors(&self, l: usize, m: usize, m0: usize) -> &[u32] { - let slot = |links: usize| (1 + links) * 4; // count + ids - let start = 2 + if l == 0 { 0 } // 2 = level header - else { slot(m0) + (l - 1) * slot(m) }; - let cnt = read_u32(self.bytes, start) as usize; - cast_u32(&self.bytes[start + 4..start + 4 + cnt * 4]) - } // one miss to reach the tape; the rest prefetches -} -``` - -One miss to reach the tape, then the neighbor ids stream in -sequentially — the prefetcher's favorite pattern. Compare qdrant -(per-level `Vec>` in the builder, serialized compressed -later) and neo4j's scattered records (topic 13): usearch picks -"everything about a node in one place" — one pointer chase per node -visit, then streaming. The cost: slots are preallocated to the max -(M or M0), so a node with 3 links pays for 16 — memory traded for -predictable layout and lock-free growth (Step 5). - -### Step 3 — defaults: the paper's advice, frozen into constants - -usearch hard-codes the parameter choices the ecosystem converged on -— the same table as the paper chapter, now as source constants: -`default_connectivity() = 16` (M), `connectivity_base = 2 × M = 32` -(M0), `default_expansion_add() = 128` (ef_construction), -`default_expansion_search() = 64` (ef). Worth internalizing: the -tape's size per node is fixed the moment M is chosen — question 1 -makes you count the bytes. +```cpp +// index.hpp — where the tape's dimensions come from, 4147-4163 + 4147 inline static precomputed_constants_t precompute_(index_config_t const& config) noexcept { + 4148 precomputed_constants_t pre; + 4149 pre.inverse_log_connectivity = 1.0 / std::log(static_cast(config.connectivity)); + 4150 pre.neighbors_bytes = config.connectivity * sizeof(compressed_slot_t) + sizeof(neighbors_count_t); + 4151 pre.neighbors_base_bytes = config.connectivity_base * sizeof(compressed_slot_t) + sizeof(neighbors_count_t); + 4152 return pre; + 4153 } +// ... 4154-4157: span typedef and the node_t overload ... + 4158 inline std::size_t node_bytes_(level_t level) const noexcept { + 4159 return node_head_bytes_() + node_neighbors_bytes_(level); + 4160 } +// ... 4161: the node_t overload ... + 4162 inline std::size_t node_neighbors_bytes_(level_t level) const noexcept { + 4163 return pre_.neighbors_base_bytes + pre_.neighbors_bytes * level; + 4164 } +``` + +Finding layer l's neighbors is then pure offset arithmetic — no +pointer hops. `neighbors_ref_t` is the view that does it: + +```cpp +// index.hpp — the neighbor-slot view over raw bytes, 2404-2426 + 2404 class neighbors_ref_t { + 2405 byte_t* tape_; +// ... 2406: the misaligned-load helper ... + 2407 static constexpr std::size_t shift(std::size_t i = 0) { + 2408 return sizeof(neighbors_count_t) + sizeof(compressed_slot_t) * i; + 2409 } +// ... 2410-2415: iterator typedefs ... + 2416 neighbors_ref_t(byte_t* tape) noexcept : tape_(tape) {} +``` + +Now do the arithmetic the paper chapter's memory formula only sketched +(M = 16, so `connectivity_base = 2M = 32`, `compressed_slot_t` = 4 B, +`neighbors_count_t` = 4 B): + +``` + node_head_bytes_ = sizeof(u64 key) + sizeof(i16 level) + = 8 + 2 = 10 B + neighbors_base_bytes = 32 × 4 + 4 = 132 B + neighbors_bytes = 16 × 4 + 4 = 68 B + + a level-0-only node = 10 + 132 = 142 B + a node reaching L1 = 10 + 132 + 68 = 210 B + + E[extra levels] = 1/(M-1) = 1/15 = 0.0667 + (geometric with p = 1/M, floored — Step 3) + average bytes / node = 142 + 68 × 0.0667 = 146.5 B + + the same node as Vec> (M=16, one inner Vec): + outer Vec header 24 B + inner Vec header 24 B + + heap block for 32 u32 128 B + 2 allocator headers ≈ 32 B + ≈ 208 B + in 2 allocations, 2 dependent + misses to reach a neighbor id +``` + +So the tape is ~1.4× smaller *and* costs one miss instead of two per +node visit. One miss to reach the tape, then the neighbor ids stream +in sequentially — the prefetcher's favorite pattern. Compare qdrant +(per-level `Vec>>` in the builder at +`lib/segment/src/index/hnsw_index/graph_layers_builder.rs:43`, +serialized compressed later) and neo4j's scattered records (topic 13): +usearch picks "everything about a node in one place". The cost: slots +are preallocated to the max (M or M0), so a node with 3 links pays for +16 — memory traded for predictable layout and lock-free growth +(Step 5). + +### Step 3 — defaults, and the level draw that matches the paper + +> **In:** the paper's parameter advice. **Out:** usearch's four +> constants, the `validate()` rule linking two of them, and the +> rounding detail that makes usearch's hierarchy differ from qdrant's. + +usearch hard-codes the parameter choices the ecosystem converged on, +as source constants: `default_connectivity() = 16` (M) at +`index.hpp:1563`, `connectivity_base = default_connectivity() * 2` += 32 (M0) at `:1591`, `default_expansion_add() = 128` +(efConstruction) at `:1568`, `default_expansion_search() = 64` (ef) +at `:1573`. `index_config_t::validate()` at `:1600-1620` is the one +that carries a rule rather than a number: `:1604` recomputes +`checked_mul(connectivity, 2)` and `:1609-1612` rejects a config +whose base connectivity is below it. + +Note where these sit relative to the paper: the paper's §4.1 gives no +efConstruction default at all (100 is a figure caption, and §5's own +experiments use 500 and 40), so usearch's 128 is a *library* choice, +not a paper one. M = 16 is squarely inside §4.1's *"reasonable range +of M is from 5 to 48"*, and M0 = 2M is exactly §4.1's *"2·M is a good +choice for Mmax0"*. + +The level draw is where implementations quietly diverge: + +```cpp +// index.hpp — the level draw, 4336-4340 + 4336 level_t choose_random_level_(std::default_random_engine& level_generator) const noexcept { + 4337 std::uniform_real_distribution distribution(0.0, 1.0); + 4338 double r = -std::log(distribution(level_generator)) * pre_.inverse_log_connectivity; + 4339 return (level_t)r; + 4340 } +``` + +`pre_.inverse_log_connectivity` is `1/ln(connectivity)` from `:4149` +— the paper's **mL**. The C-style cast on `:4339` is a +double→integer **truncation**, i.e. a floor, which is what the +paper's Algorithm 1 line 4 specifies. qdrant, at +`graph_layers_builder.rs:391`, calls `.round()` instead. Compute the +difference at M = 16: + +``` + P(level ≥ 1) with floor: exp(-1/mL) = exp(-ln 16) = 1/16 + = 6.3% promoted + P(level ≥ 1) with round: draw ≥ 0.5, so = exp(-0.5·ln 16) + = 16^-0.5 = 25% promoted +``` + +Four times as many nodes reach layer 1 in qdrant as in usearch at the +same M. Same paper, same formula, one cast apart. ### Step 4 — the three walks: the paper's algorithms as three functions -The whole engine is three traversals, each a direct transcription of -a paper algorithm: - -- **`search_to_insert_`** — Alg 1's per-level beam during insert; - `form_links_to_closest_` applies the Alg 4 heuristic and - back-links (shrinking overfull neighbors back to their slot - limits). -- **`search_to_find_in_base_`** — Alg 2 on layer 0, with an optional - `predicate` parameter — filtering exists here too, but ONLY as - filter-during-traversal: the predicate rejects nodes as they're - scored. There is no cardinality planner and no ACORN; a selective - filter simply disconnects the walk (percolation — compare qdrant's - search.rs:55-84; that gap IS qdrant's moat, walked in - [reading-qdrant-hnsw.md](reading-qdrant-hnsw.md)). -- **The greedy descent loops** (`level >= 0; --level`) — the ef=1 - upper-layer descent, including the update path: usearch supports - in-place vector updates, rare among HNSW libraries. +> **In:** the paper's Algorithms 1, 2, 4 and 5. **Out:** the four +> functions that implement them, with the definition sites (not the +> call sites) so you can read them. + +The whole engine is a handful of traversals, each a transcription of a +paper algorithm. Definition sites, all in `index.hpp`: + +| function | defined | paper | what it is | +|---|---|---|---| +| `search_for_one_` | `:4406` | Alg. 5's descent | the ef=1 greedy walk down the upper layers | +| `search_to_insert_` | `:4455` | Alg. 1's per-level beam | called from `:3234` during insert | +| `form_links_to_closest_` | `:4262` | Alg. 4 heuristic + back-links | called from `:3239` and `:3366`; shrinks overfull neighbors back to their slot limits | +| `search_to_find_in_base_` | `:4629` | Alg. 2 on layer 0 | the query path; called from `:3446` | +| `search_exact_` | `:4704` | — | the brute-force fallback, for when the graph is smaller than the query | The mapping is the point: one paper algorithm ↔ one function, no architecture in between. That's what "reference implementation for your hnsw.rs" means concretely. -### Step 5 — concurrency: striped locks for writers, lock-free readers +**Now the claim this chapter exists to correct.** +`search_to_find_in_base_` takes an optional `predicate`, and it is +tempting — and wrong — to say that a selective filter therefore +disconnects usearch's walk. Read where the predicate is actually +applied: + +```cpp +// index.hpp — the neighbor loop of search_to_find_in_base_, 4681-4695 + 4681 for (compressed_slot_t successor_slot : candidate_neighbors) { + 4682 if (visits.set(successor_slot)) + 4683 continue; + 4684 + 4685 distance_t successor_dist = context.measure(query, citerator_at(successor_slot), metric); + 4686 if (top.size() < top_limit || successor_dist < radius) { + 4687 // This can substantially grow our priority queue: + 4688 next.insert({-successor_dist, successor_slot}); + 4689 if (is_dummy() || + 4690 predicate(member_cref_t{node_at_(successor_slot).ckey(), successor_slot})) { + 4691 top.insert({successor_dist, successor_slot}, top_limit); + 4692 radius = top.top().distance; + 4693 } + 4694 } + 4695 } +``` + +`next.insert` on `:4688` is **outside** the predicate test on +`:4689-4690`. Rejected nodes still enter the expansion frontier; +only the result list `top` is filtered. usearch filters the +**results**, not the traversal — so the walk is not disconnected and +recall does not percolate away. + +What does break is the stopping rule: + +```cpp +// index.hpp — the stop test that a selective predicate defeats, 4660-4666 + 4660 while (!next.empty()) { + 4661 + 4662 candidate_t candidate = next.top(); + 4663 if ((-candidate.distance) > radius && top.size() == top_limit) + 4664 break; + 4665 + 4666 next.pop(); +``` + +The loop exits only when `top` is **full** (`top.size() == top_limit`) +and the frontier's best is worse than `radius`. With a 1% predicate, +`top` rarely fills, the `:4663` break never fires, and the search +walks until `next` empties — degrading toward an exhaustive scan. The +failure mode is **cost, not recall**. + +Contrast qdrant, which does the opposite and therefore has the +opposite problem: + +```rust +// lib/segment/src/index/hnsw_index/point_scorer.rs — qdrant cuts the frontier, 231 + 231 point_ids.retain(|id| self.filters.check_vector(*id)); +``` + +qdrant drops filtered ids *before scoring*, so they never become +frontier — the walk really does disconnect, which is why qdrant needs +a cardinality planner and ACORN-1 +(`graph_layers.rs:155`, walked in +[reading-qdrant-hnsw.md](reading-qdrant-hnsw.md)). usearch has +neither, and does not need them for *recall*; what it lacks is a +plan B for the cost blow-up. + +### Step 5 — concurrency: striped spin locks for writers, lock-free readers + +> **In:** concurrent inserts mutating neighbor lists. **Out:** the +> exact lock structure, how many stripes exist, and the invariant that +> lets readers take no lock at all. Concurrent inserts mutate neighbor lists, so writes need exclusion — but one global lock would serialize the build. usearch uses -**striped locks** (`striped_locks_gt`): a fixed array of ~threads × -connectivity mutexes, each node hashing to one stripe — writers take -only their stripe, so unrelated inserts proceed in parallel. -Searches take NO locks: they read published tapes, which never move -(Step 2's preallocation pays off here — growth never reallocates). -Simpler than qdrant's RwLock-per-node builder; the cost is -update-vs-read races, handled by slot versioning in -`index_dense.hpp`. +**striped spin locks**, not mutexes: + +```cpp +// index.hpp — the lock array, 660-680 + 660 /** + 661 * @brief Cache-line-padded striped spin-lock array for concurrent graph mutations. + 662 * Maps node slots to lock stripes via Fibonacci hashing, with each stripe + 663 * occupying its own cache line to eliminate false sharing. + 664 * The number of stripes is proportional to `threads * connectivity`, not + 665 * graph size, keeping the lock array comfortably within L2/L3 cache. + 666 */ + 667 template , std::size_t cache_line_ak = 128> // + 668 class striped_locks_gt { +// ... 669-672: allocator typedefs and a byte-size static_assert ... + 673 static constexpr std::uint64_t fibonacci_k = 0x9E3779B97F4A7C15ull; + 674 + 675 using atomic_flag_t = std::atomic; + 676 struct alignas(cache_line_ak) padded_lock_t { + 677 atomic_flag_t flag{0}; + 678 char padding_[cache_line_ak - sizeof(atomic_flag_t)]; + 679 }; + 680 static_assert(sizeof(padded_lock_t) == cache_line_ak, "Lock stripe must be exactly one cache line"); +``` + +An `std::atomic` spun on, one per 128-byte cache line: a +single byte of state padded 128× to buy false-sharing immunity. The +stripe index is Fibonacci hashing of the slot (`:693-695`), and the +count comes from the constructor: + +```cpp +// index.hpp — how many stripes exist, 716-729 + 716 striped_locks_gt(std::size_t threads, std::size_t connectivity) noexcept { + 717 checked_size_result_t desired = checked_mul(threads, connectivity); + 718 desired = desired ? checked_mul(desired.value, std::size_t{4}) : desired; +// ... 719-723: overflow bail-out ... + 724 checked_size_result_t count = checked_ceil2((std::max)(desired.value, 256)); +// ... 725-728: overflow bail-out ... + 729 count_ = count.value; +``` + +Work the size on this topic's machine (Apple M3 Pro, 12 threads, +M = 16): + +``` + desired = threads × connectivity × 4 = 12 × 16 × 4 = 768 + count = ceil2(max(768, 256)) = 1024 stripes + bytes = 1024 × 128 B (one cache line each) = 128 KiB +``` + +128 KiB of lock array — sized by *thread count*, never by graph size, +exactly as the doc comment on `:664-665` claims. A billion-node index +uses the same 128 KiB. + +Searches take **no locks at all**: `search_to_find_in_base_` +(`:4629-4699`) contains no lock acquisition anywhere. It can do that +because Step 2's preallocation means a published tape never moves — +growth writes into slots that already exist. Simpler than qdrant's +`RwLock`-per-node builder +(`graph_layers_builder.rs:43`); the cost is update-vs-read races, +handled by slot versioning in `index_dense.hpp`. + +### Step 6 — metrics and the SIMD that isn't there by default + +> **In:** the belief that usearch is "the SIMD one". **Out:** the +> actual metric set, and the preprocessor gate that decides whether +> any hand-written SIMD runs at all. + +The metric set is finite and enumerated — ten real metrics, plus a +sentinel: + +```cpp +// index_plugins.hpp — every metric usearch knows, 114-133 + 114 enum class metric_kind_t : std::uint8_t { + 115 unknown_k = 0, + 116 // Classics: + 117 ip_k = 'i', + 118 cos_k = 'c', + 119 l2sq_k = 'e', + 120 + 121 // Custom: + 122 pearson_k = 'p', + 123 haversine_k = 'h', + 124 divergence_k = 'd', + 125 + 126 // Sets: + 127 hamming_k = 'b', + 128 tanimoto_k = 't', + 129 sorensen_k = 's', + 130 }; +``` + +(The comment groups them: three classic vector metrics, three custom +ones — including `haversine_k`, for lat/long — and three set metrics +over bit vectors.) The scalar types are broader: +`scalar_kind_t` at `:139-164` spans `b1x8` bit vectors through +`f64`, including `bf16` and four minifloat formats. + +Now the dispatch. Check the gate before believing anything about +vectorization: + +```cpp +// index_plugins.hpp — the SIMD backend is OPT-IN, 25-27 + 25 #if !defined(USEARCH_USE_NUMKONG) + 26 #define USEARCH_USE_NUMKONG 0 + 27 #endif +``` + +The external kernel library at this pin is **NumKong** (included at +`:59`), and it defaults to **off**. So the fallback path in +`metric_punned_t::builtin` is the one a stock header-only build takes: + +```cpp +// index_plugins.hpp — metric_punned_t::builtin, 2916-2930 + 2916 inline static metric_punned_t builtin(std::size_t dimensions, metric_kind_t metric_kind = metric_kind_t::l2sq_k, + 2917 scalar_kind_t scalar_kind = scalar_kind_t::f32_k) noexcept { +// ... 2918-2925: fill in the routed function pointer, dimensions, kinds ... + 2927 if (!metric.configure_with_numkong()) + 2928 metric.configure_with_autovec(); + 2929 + 2930 return metric; +``` + +`configure_with_numkong()` returns false when the macro is 0, so +`configure_with_autovec()` runs — plain C++ loops handed to the +compiler's auto-vectorizer. This is the topic-11 argument in +miniature: usearch **templates** the metric so the compiler +specializes it (compiled), where qdrant enum-dispatches into +`unsafe extern "C"` hand-written kernels (vectorized). Neither is +free: usearch pays in compile time and binary size and depends on +the auto-vectorizer's judgment; qdrant pays a dispatch per batch and +maintains the intrinsics by hand. ## Where each step lives in the code -Everything is in `include/usearch/index.hpp` (line numbers from the -walked revision; navigate by symbol name when they drift): - -- **Step 1–2 (the tape)**: `:2242` `class index_gt` — the whole - index: a vector of node pointers + per-node tapes; `:2404` - `neighbors_ref_t` — the view over raw bytes (`tape_`, :2416) that - the Rust sketch above transcribes. -- **Step 3 (defaults)**: `:1563` `default_connectivity() = 16` (M); - `:1591` `connectivity_base = 2 × M` (M0) — computed at :1604; - `:1568` `default_expansion_add() = 128` (ef_construction); `:1573` - `default_expansion_search() = 64` (ef). -- **Step 4 (the walks)**: `:3234` `search_to_insert_`; `:3239` - `form_links_to_closest_` (defined :4262) — Alg 4 + back-links; - `:3446` `search_to_find_in_base_` — Alg 2 with the `predicate`; - `:3232`, `:3354` — the greedy descent loops, including the update - path. -- **Step 5 (concurrency)**: `:664-717` `striped_locks_gt`; the - type-erased/quantized wrapper and slot versioning live in - `index_dense.hpp`. +All paths relative to the `unum-cloud/usearch@9fd6b01` clone. + +| step | file | lines | what | +|---|---|---|---| +| 1 | `include/usearch/index.hpp` | `:2127-2128` | `default_key_t` = u64, `default_slot_t` = u32 | +| 1 | `include/usearch/index.hpp` | `:2335-2341` | `neighbors_count_t` = u32, `level_t` = i16, `node_head_bytes_()` = 10 | +| 2 | `include/usearch/index.hpp` | `:2364-2371` | the tape doc comment — key, then level, then per-level slots | +| 2 | `include/usearch/index.hpp` | `:2372-2392` | `class node_t`, `tape_`, `neighbors_tape()`, key/level accessors | +| 2 | `include/usearch/index.hpp` | `:2404-2435` | `neighbors_ref_t` — `shift()`, `size()`, `push_back` | +| 2 | `include/usearch/index.hpp` | `:4147-4164` | `precompute_`, `node_bytes_`, `node_neighbors_bytes_` | +| 2 | `include/usearch/index.hpp` | `:4166-4182` | `node_malloc_` / `node_make_` — where a tape is born | +| 3 | `include/usearch/index.hpp` | `:1563`, `:1568`, `:1573`, `:1591` | connectivity 16, expansion_add 128, expansion_search 64, connectivity_base 2M | +| 3 | `include/usearch/index.hpp` | `:1600-1620` | `index_config_t::validate()` | +| 3 | `include/usearch/index.hpp` | `:4336-4340` | `choose_random_level_` — mL and the truncating cast | +| 4 | `include/usearch/index.hpp` | `:4406`, `:4455`, `:4262`, `:4629`, `:4704` | the five walks, at their definitions | +| 4 | `include/usearch/index.hpp` | `:4681-4695` | the predicate filters `top`, not `next` | +| 4 | `include/usearch/index.hpp` | `:4660-4664` | the stop test a selective predicate defeats | +| 5 | `include/usearch/index.hpp` | `:660-695` | `striped_locks_gt` — spin flags, Fibonacci hashing | +| 5 | `include/usearch/index.hpp` | `:716-730` | the stripe count: `ceil2(max(threads × M × 4, 256))` | +| 6 | `include/usearch/index_plugins.hpp` | `:114-133`, `:139-164` | `metric_kind_t` (10 metrics), `scalar_kind_t` | +| 6 | `include/usearch/index_plugins.hpp` | `:25-27`, `:2916-2931` | `USEARCH_USE_NUMKONG` defaults to 0; `builtin` falls back to autovec | ## Questions (answer in notes.md) @@ -163,21 +507,133 @@ walked revision; navigate by symbol name when they drift): ## Done when +Answer each before unfolding it. + - [ ] You can list what an HNSW node must store and compute bytes per node for M=16, M0=32. +
Answer + + Key (u64), level (i16), and one `{count, slots…}` block per layer + from the node's level down to 0 — `index.hpp:2364-2371`. Widths at + `:2127-2128` and `:2335-2336`; `node_head_bytes_()` at `:2341` is + `8 + 2 = 10`. From `precompute_` at `:4149-4151`: + `neighbors_base_bytes = 32 × 4 + 4 = 132`, + `neighbors_bytes = 16 × 4 + 4 = 68`. `node_bytes_(level)` at + `:4158-4163` is `10 + 132 + 68 × level`, so a level-0-only node is + **142 B** and a level-1 node is **210 B**. At the floored geometric + draw's `E[extra levels] = 1/(M−1) = 0.0667`, the average is + `142 + 68 × 0.0667 = 146.5 B`. The vector itself is stored + separately — the tape is graph structure only. + +
+ - [ ] You can explain what the node tape buys over `Vec>` per level, in allocations and in locality. +
Answer + + Allocations: one per node instead of `1 + levels`. Misses: one to + reach the tape, after which the count and every neighbor id stream + in from the same cache lines; `Vec>` costs a dependent miss + to the outer Vec's buffer and another to the inner Vec's heap block + before any id is visible. Bytes: 146.5 B on the tape versus roughly + 208 B for the Vec version (24 B outer header + 24 B inner header + + 128 B for 32 u32 + ~32 B of allocator headers). And the second-order + win is Step 5's: a tape that never reallocates is a tape readers can + walk without a lock. + +
+ - [ ] You can say why link slots are preallocated to the maximum rather than grown. +
Answer + + Because `node_bytes_(level)` at `:4158-4163` must be a *pure + function of the level* for two separate reasons. First, offsets: + layer l's block is at a computable distance from the tape start + only if every earlier block has its maximum width — that is what + `neighbors_base_bytes + neighbors_bytes * level` at `:4163` means. + Second, concurrency: growth would mean reallocation, reallocation + would mean a moving tape, and a moving tape would force readers to + take a lock. The price is slack — a node with 3 links occupies + slots for 16 — paid in bytes to buy both O(1) addressing and + lock-free reads. + +
+ - [ ] You can describe the concurrency scheme — striped writer locks, lock-free readers — and what it assumes about readers. +
Answer + + Writers take one stripe of a `striped_locks_gt` array + (`:667-680`): a `std::atomic` **spin flag** — not a mutex — + padded to a full 128-byte cache line so stripes cannot false-share. + The slot picks its stripe by Fibonacci hashing (`:693-695`), and the + array holds `ceil2(max(threads × connectivity × 4, 256))` stripes + (`:716-729`) — 1024 stripes, 128 KiB, at 12 threads and M=16, and + the same 128 KiB no matter how large the graph grows. Readers take + nothing: `search_to_find_in_base_` at `:4629-4699` acquires no lock + anywhere. The assumption that makes that sound is Step 2's — a + published tape never moves, so a reader can at worst observe a stale + neighbor count, never a dangling pointer. Cross-checking a + concurrent *update* (as opposed to insert) is what `index_dense.hpp`'s + slot versioning is for. + +
+ +- [ ] You can say where usearch applies a search predicate, and why that makes its filtered-search failure mode the opposite of qdrant's. +
Answer + + At `index.hpp:4689-4690`, guarding only `top.insert` on `:4691`. + `next.insert` on `:4688` is *outside* that guard, so rejected nodes + still expand the frontier: usearch filters the **result list**, not + the traversal. Recall therefore does not percolate away. What breaks + is termination — the loop's exit at `:4663` requires + `top.size() == top_limit`, and with a 1% predicate `top` rarely + fills, so the walk continues until `next` empties and the query + degrades toward exhaustive. **Cost blow-up, not recall collapse.** + qdrant does the reverse: `point_scorer.rs:231` `retain`s ids before + scoring, cutting the frontier itself, which is genuine percolation — + and precisely why qdrant needs a cardinality planner and ACORN-1 + (`graph_layers.rs:155`) and usearch does not. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your own tape-or-vec decision for `hnsw.rs`. +
Answer + + For question 4, check the gate before answering: `index_plugins.hpp:25-27` + defaults `USEARCH_USE_NUMKONG` to 0, so `builtin` at `:2927-2928` + falls through to `configure_with_autovec()` — a stock build has no + hand-written SIMD at all, only what the compiler finds in templated + loops. For question 5, the honest answer depends on whether your + `hnsw.rs` will ever be read concurrently with writes: if not, + `Vec>` is ~60 B/node more and one extra miss per visit, and + is far easier to get right; if yes, the tape's + never-reallocate invariant is doing work no amount of tuning + recovers. Whichever you choose, write down the M you fixed it at — + `node_bytes_` shows the tape's size is frozen the moment M is. + +
## References **Papers** - Malkov, Yashunin — the HNSW paper ([arXiv:1603.09320](https://arxiv.org/abs/1603.09320)) — gets its - own chapter: [reading-hnsw-paper.md](reading-hnsw-paper.md) + own chapter: [reading-hnsw-paper.md](reading-hnsw-paper.md). + §4.1's *"2·M is a good choice for Mmax0"* and *"reasonable range of + M is from 5 to 48"* are the two lines usearch's defaults implement; + Algorithm 1 line 4's floor is what `:4339`'s cast reproduces. -**Code** -- [usearch](https://github.com/unum-cloud/usearch) — all of it in - `include/usearch/index.hpp` (+ `index_dense.hpp` for the - type-erased/quantized wrapper); C++ templates, but small enough to - hold in your head +**Code** — `unum-cloud/usearch@9fd6b01`, pinned in +`resources/codebases.md` +- `include/usearch/index.hpp` (5033 lines) — the entire graph: + config and defaults (`:1563-1620`), the tape (`:2335-2435`, + `:4147-4182`), the walks (`:4262`, `:4406`, `:4455`, `:4629`, + `:4704`), the locks (`:660-730`) +- `include/usearch/index_plugins.hpp` (4275 lines) — `metric_kind_t` + (`:114-133`), `scalar_kind_t` (`:139-164`), the NumKong gate + (`:25-27`) and `metric_punned_t::builtin` (`:2916-2931`) +- `index_dense.hpp` — the type-erased/quantized wrapper and the slot + versioning that makes concurrent *updates* safe (not read here; + named because Step 5's answer depends on it) +- qdrant, for the two contrasts drawn above: + `lib/segment/src/index/hnsw_index/point_scorer.rs:231` (filter + before scoring) and `graph_layers_builder.rs:391` (`.round()` on the + level draw) diff --git a/topics/15-replication-consensus/README.md b/topics/15-replication-consensus/README.md index 1754b30..51f5151 100644 --- a/topics/15-replication-consensus/README.md +++ b/topics/15-replication-consensus/README.md @@ -51,9 +51,9 @@ survive the failure cases each choice creates. ```mermaid stateDiagram-v2 - Follower --> Candidate: election timeout\n(randomized!) + Follower --> Candidate: election timeout
(randomized!) Candidate --> Leader: votes from majority - Candidate --> Follower: saw higher term /\ncurrent leader + Candidate --> Follower: saw higher term /
current leader Candidate --> Candidate: split vote, new term Leader --> Follower: saw higher term ``` diff --git a/topics/15-replication-consensus/notes.md b/topics/15-replication-consensus/notes.md index 71274cf..f2c1247 100644 --- a/topics/15-replication-consensus/notes.md +++ b/topics/15-replication-consensus/notes.md @@ -82,14 +82,18 @@ Surprises / dead ends: 1. Why no fsync/sockets/threads in the library: 2. maybe_commit on matched=[7,5,5,3,2] → commit index: -3. next_idx decrement optimization (§5.3 footnote): +3. next_idx decrement optimization (§5.3 body, not a footnote — and + the paper doubts it is necessary, while raft-rs implements it): 4. advance_append pipelining — what still can't reorder: 5. Ready → M15 stage 2 mapping: ### qdrant consensus (reading-qdrant-consensus.md) 1. 10K upserts/s through raft = ? (use the 3 ms fsync above): -2. Active/Dead/Partial ↔ Progress replicate/probe/snapshot: +2. `ReplicaState`'s eleven variants (`replica_set_state.rs:100-133`) ↔ + Progress replicate/probe/snapshot — start with Active/Dead/Partial, + then say where `ActiveRead` (readable, not a source of truth, `:125`) + and `ManualRecovery` land: 3. qdrant vector-read consistency: 4. WAL-through-raft: right for a graph DB? FalkorDB's answer: 5. Storage impl behind ConsensusStateRef: diff --git a/topics/15-replication-consensus/reading-ddia-repl.md b/topics/15-replication-consensus/reading-ddia-repl.md index 03e323d..965bae9 100644 --- a/topics/15-replication-consensus/reading-ddia-repl.md +++ b/topics/15-replication-consensus/reading-ddia-repl.md @@ -9,6 +9,17 @@ actually promise (ch. 9). Read ch. 5 alongside valkey's `replication.c` and ch. 9 alongside the Raft paper; ch. 8 is the connective tissue. +**On sourcing.** *Designing Data-Intensive Applications* (O'Reilly, +2017) is a copyrighted book and nothing here quotes it — chapters are +cited by number and every idea is restated in this repo's own words. +That has a happy side effect: **every number below comes from +somewhere you can re-run or re-open** — this topic's `repl_lag` +bench, topic 5's fsync ladder, the Raft paper, or pinned source. +Anchors are valkey at `8891441ab` and raft-rs at `ad13f3d` (the pin +table in `resources/codebases.md`); paths starting `experiments/` are +this topic's own crate. Check any of them with +`python3 tools/pinned-source.py show …`. + ## The problem in one sentence An async replica is always some milliseconds (or, during a @@ -22,136 +33,415 @@ precisely-named guarantees, each with a price. ### Step 1 — replication lag: the gap between ack and everywhere -Replication lag is the delay between a write committing on the -leader and becoming visible on a given replica. Under async -replication (valkey's design, previous chapter) lag is unbounded by -construction — the ack never waited. Normal lag is milliseconds; the -tail is the problem: a replica doing a full resync, hitting disk, or -GC-pausing can lag minutes. Lag is invisible to anyone who only -talks to the leader; it becomes real the moment reads are scaled out -to replicas — which is the entire reason to have replicas. So the -question "what does a reader see?" needs a taxonomy — Step 2. +> **In:** a write that the leader has already acked. **Out:** a +> definition of replication lag, the measured size of it in this +> topic's own bench, and the reason lag is invisible until you scale +> reads out. + +**Replication lag** is the delay between a write committing on the +leader and becoming visible on a given replica. Under **asynchronous +replication** — the leader acks the client without waiting for any +replica, valkey's design and the previous chapter's subject — lag is +unbounded by construction, because the ack never waited for anything. +Under **synchronous** or **semi-synchronous** replication the ack +waits for one or more replicas, which bounds lag at the cost of +putting a network round trip (and possibly an fsync) on the write +path. + +You do not have to guess how big lag is; this topic measured it. +`./verify.sh 15` runs `repl_lag` with 2000 entries × 128 B, +group-commit every 64, WAIT-1 semantics, and varies only the +*follower's* fsync policy: + +| follower fsync | entries/s | ack p50 | ack p99 | +|---|---|---|---| +| every entry | 341 | 2967.0 µs | 3889.5 µs | +| every 8 | 2730 | 22.2 µs | 2979.8 µs | +| every 64 | 12187 | 14.0 µs | 2133.0 µs | +| never | 20174 | 13.8 µs | 64.5 µs | + +Read the p50/p99 gap as the honest picture of lag: at *every 8* the +median ack is 22 µs and the 99th percentile is 2980 µs — **134× +worse**. Lag is not a number, it is a distribution with a long tail, +and the tail is what your users hit. The tail sources are the ones +ch. 5 lists: a replica doing a full resync, a replica that hit disk, +a GC pause. + +Convert the tail into staleness — the quantity a reader actually +cares about: + +``` + How far behind is a replica at p99, in entries? + + inputs (notes.md baseline, Apple M3 Pro / APFS, 2026-07-28): + throughput at "fsync every 64" = 12,187 entries/s + ack p99 at that setting = 2,133 us = 2.133e-3 s + + entries the leader accepts inside one p99 window: + 12,187 x 2.133e-3 = 26.0 entries + + So a reader landing on that replica at the wrong moment sees a + snapshot ~26 writes old. At "every entry" the throughput collapses + to 341/s and the same arithmetic gives 341 x 3.8895e-3 = 1.3 + entries — the replica is nearly current, because the system is + barely moving. Bounding lag by slowing down is a real option and a + terrible one. +``` + +Lag is invisible to anyone who only talks to the leader; it becomes +real the moment reads are scaled out to replicas — which is the +entire reason to have replicas. So the question "what does a reader +see?" needs a taxonomy — Step 2. ### Step 2 — the anomaly catalog: three ways lag bites readers -Each anomaly is a specific reader experience, and each has a -specific, priced fix: +> **In:** a client issuing a sequence of reads while lag is nonzero. +> **Out:** three named anomalies, the guarantee that kills each, and +> the specific thing each guarantee costs you. + +Ch. 5's catalog is three reader experiences, each with a specific, +priced fix. The names are the point — "eventually consistent" is not +a specification, these are: ``` - anomaly fix - ────────────────────────────────────────────────────────── - read-your-writes session stickiness, or read-after - (I posted, refresh, -my-offset (track repl offset per - it's gone) session — valkey WAIT-ish) - monotonic reads pin session to one replica - (time goes backward - across refreshes) - consistent prefix causally-ordered delivery (or - (answer before single-partition ordering) - question) + anomaly guarantee that kills it what it costs + ───────────────────────────────────────────────────────────────── + read-your-writes read-your-writes / offset bookkeeping + (I posted, refresh, read-after-write per session; reads + it's gone) may block or divert + to the leader + monotonic reads monotonic reads load-balancing + (time goes backward freedom — the + across refreshes) session is pinned + to one replica + consistent prefix consistent prefix reads ordering machinery + (answer arrives across partitions, + before question) or one partition ``` -Read them as contracts weaker than "no lag visible at all" -(linearizability, Step 5) but individually purchasable: -read-your-writes costs offset bookkeeping per session; monotonic -reads costs load-balancing freedom; consistent prefix costs ordering -machinery. Question per anomaly: which does our M15 stage-1 follower -exhibit, and what does the fix cost? +Two definitions worth being exact about, because they are commonly +blurred. **Read-your-writes** says a client sees its *own* writes; it +says nothing about anyone else's. **Monotonic reads** says a client +never sees the clock run backwards; it also says nothing about +freshness — a session pinned to a replica that is 26 entries behind +(Step 1's arithmetic) gets monotonic reads and stale data at the same +time. Neither is linearizability (Step 5); they are strictly weaker, +which is exactly why they are affordable. + +The implementable version of read-your-writes is an **offset token**: +the client remembers the replication offset its write reached and +refuses any replica behind that offset. valkey exposes the raw +material — `getClientWriteOffset` (`src/replication.c:4953`) is how +WAIT learns the offset a client's write landed at, and each replica's +progress is tracked as `repl_ack_off`, counted by +`replicationCountAcksByOffset` (`src/replication.c:4962-4975`). What +valkey does *not* ship is the read-side check; that is your M15 +stage-2 work. + +Price the fix with the measured table. If a session must not read a +replica more than one entry stale, the wait is the ack latency: +13.8 µs at p50 in the *never fsync* row, 64.5 µs at p99. If the +follower fsyncs every 8, the same wait is 22.2 µs at p50 but +2979.8 µs at p99 — **the fix's cost is set by the durability policy, +not by the read path.** Question per anomaly: which does our M15 +stage-1 follower exhibit, and what does the fix cost? ### Step 3 — what actually ships: statements, WAL bytes, or rows -Chapter 5's other half is the replication-log format menu, and this -topic implements two of the three: +> **In:** a committed write on the leader. **Out:** three candidate +> encodings for putting it on the wire, what each one breaks, and the +> exact line in valkey where the statement-shipping tax gets paid. + +Ch. 5's other half is the replication-log format menu, and this topic +implements two of the three: -- **Statement-based** — ship the commands; compact, but - nondeterminism must be rewritten first. This is valkey - (post-`propagateNow` — previous chapter). -- **Physical WAL** — ship the storage engine's own log bytes; - deterministic by construction, but coupled to the engine version - and page layout. This is our M15 stage 1. -- **Logical (row-based)** — ship "row X became Y"; decoupled from - the engine, the format change-data-capture wants. +- **Statement-based** — ship the commands. Compact, and readable in + `MONITOR`, but any **nondeterminism** (a random choice, `NOW()`, an + auto-increment, a side effect that depends on local state) must be + rewritten into a deterministic form before it leaves the leader, or + the replicas diverge. This is valkey. +- **Physical WAL** — ship the storage engine's own log bytes. + Deterministic by construction, because the replica is not + re-deciding anything; but the stream is coupled to the engine + version and page layout, so a replica must run compatible code. + This is our M15 stage 1, and it is topic 5's WAL wearing a network + card. +- **Logical (row-based)** — ship "row X became Y". Decoupled from the + engine, therefore upgradable and consumable by outsiders; this is + the format change-data-capture wants, and it is the fattest of the + three on the wire. + +The nondeterminism tax is not abstract — you can open it. valkey's +`SPOP` removes a *random* member, so shipping the command verbatim +would give every replica a different set. The rewrite happens +per-command, inside the command implementation: `spopCommand` picks +the member at `src/t_set.c:970` and then immediately rewrites the +client's own command vector into a deterministic `SREM` at +`src/t_set.c:975`: + +```c +// valkey src/t_set.c — spopCommand, 969-978 (verbatim, no elisions) + 969 /* Pop a random element from the set */ + 970 ele = setTypePopRandom(set); + 971 + 972 notifyKeyspaceEvent(NOTIFY_SET, "spop", c->argv[1], c->db->id); + 973 + 974 /* Replicate/AOF this command as an SREM operation */ + 975 rewriteClientCommandVector(c, 3, shared.srem, c->argv[1], ele); + 976 + 977 /* Add the element to the reply */ + 978 addReplyBulk(c, ele); +``` -The tradeoff table maps onto topic 5's logging choices one-to-one. -The chapter's multi-leader and leaderless sections preview topic 31 -(CRDTs) — skim them on this pass. +Line 975 is the whole idea of statement-based replication in one +call, and the comment above it at 974 says so out loud: what the +replica receives is never the command the client sent. +The count variant does the same thing in bulk — +`spopWithCountCommand` batches `SREM`s through `alsoPropagate` +(`src/t_set.c:922` and `:937`) or turns the whole thing into a +`DEL`/`UNLINK` when the set is emptied (`:790-791`), and suppresses +the original with `preventCommandPropagation` at `:949`. **Every one +of those is a place a contributor can forget**, which is the argument +against statement shipping stated as engineering rather than theory. + +The tradeoff table maps onto topic 5's logging choices one-to-one +(physical vs logical redo is the same fork). Ch. 5's multi-leader and +leaderless sections preview topic 31 (CRDTs) — skim them on this pass. ### Step 4 — partial failure: timeouts guess, clocks lie, tokens fence -Chapter 8 is one argument: in a distributed system you cannot -distinguish {slow node, dead node, slow network, lost packet} — all -four look like silence. Three consequences to extract: - -- **Timeouts are the only failure detector**, and every timeout is a - guess (our sim.rs makes this concrete: `election_timeout` ticks). - Guess short and you declare live nodes dead; guess long and real - failures stall the system. -- **Process pauses**: a GC or VM pause makes a live leader - dead-then-alive — it wakes *believing it still leads*. The defense - is a **fencing token**: a monotonically increasing number issued - with each grant of authority, checked by everyone downstream, so - the stale leader's older token is rejected. Raft terms ARE fencing - tokens (question: walk how). valkey has nothing in this slot — - hence split-brain during failover. -- **Clock skew**: wall clocks drift and jump, so "leader for the - next 5 seconds" (a lease) requires bounded clock error, while - ReadIndex (Step 5) needs no clock at all — it uses a message round - instead of time. +> **In:** silence from a node. **Out:** why silence is +> undiagnosable, what a fencing token is, and the two lines of pinned +> code where Raft has one and valkey's replication stream does not. + +Ch. 8 is one argument: in a distributed system you cannot distinguish +{slow node, dead node, slow network, lost packet} — all four look +like silence. Three consequences to extract: + +**Timeouts are the only failure detector**, and every timeout is a +guess. Guess short and you declare live nodes dead; guess long and +real failures stall the system. Your own crate makes the guess +explicit: `ELECTION_TIMEOUT_MIN = 10` and `ELECTION_TIMEOUT_MAX = 20` +ticks (`experiments/src/raft.rs:33-34`) against +`HEARTBEAT_INTERVAL = 3` (`:35`). + +``` + What ratio of "detector" to "heartbeat" are you actually running? + + experiments/src/raft.rs:33-35 + election timeout 10..20 ticks + heartbeat 3 ticks + ratio 10/3 = 3.3x to 20/3 = 6.7x + + raft-rs src/config.rs:112,115-116 (pinned defaults) + const HEARTBEAT_TICK = 2 + election_tick HEARTBEAT_TICK * 10 = 20 + heartbeat_tick HEARTBEAT_TICK = 2 + ratio 10x, written literally as "* 10" in the source + + Raft §5.6 requires broadcastTime << electionTimeout. 3.3x is thin: + ONE dropped heartbeat plus one late one starts an election. Note + also that experiments/src/raft.rs:79 currently derives the timeout + from (id + seed) rather than drawing from `rng` — deterministic, so + the tests are reproducible, but NOT the randomization Raft §5.2 + asks for. Fixing that is part of the exercise; the docstring at + experiments/src/raft.rs:8 says so. +``` + +**Process pauses**: a GC or VM pause makes a live leader +dead-then-alive — it wakes *believing it still leads*. Scale it: at +qdrant's `tick_period_ms: 100` (`config/config.yaml:359`) a 2-second +stop-the-world pause is 20 ticks, past `ELECTION_TIMEOUT_MAX`, so the +pause *alone* elects someone else and the sleeper wakes stale. The +defense is a **fencing token**: a monotonically increasing number +issued with each grant of authority, checked by everyone downstream, +so the stale leader's older token is rejected. + +Raft terms ARE fencing tokens, and you can point at the check. In +raft-rs, `Raft::step` tests `m.term < self.term` at `src/raft.rs:1416` +and, in the general case, drops the message: + +```rust +// raft-rs src/raft.rs — Raft::step, the stale-term arm, 1416-1477 + 1416 } else if m.term < self.term { + 1417 if (self.check_quorum || self.pre_vote) + ... + 1466 } else { + 1467 // ignore other cases + 1468 info!( + 1469 self.logger, + 1470 "ignored a message with lower term from {from}", + ... + 1476 } + 1477 return Ok(()); +``` + +The woken leader's `MsgAppend` carries its old term, hits line 1416, +and dies at 1477 without touching any log. (The two branches above +1466 are refinements, not exceptions: 1417-1443 replies to a +stale-term heartbeat so the sender learns it is behind, and 1444 +handles pre-vote, whose whole job is to *avoid* the term inflation +this rule punishes.) + +valkey's replication stream has no such number. Its identity is +`replid`, 40 hex characters (`CONFIG_RUN_ID_SIZE = 40`, +`src/server.h:152`), and `changeReplicationId` +(`src/replication.c:2063-2066`) fills it with `getRandomHexChars` — +**random, therefore unordered**. Two replids cannot be compared to +decide which is newer, so a replica cannot reject a stale primary on +the strength of its id; it can only detect that the history differs +and full-resync. valkey does own a real fencing token, but it lives +in the cluster gossip layer, not the replication stream: +`currentEpoch` and `configEpoch` (`src/cluster_legacy.h:278-281`) are +monotonic `uint64_t`s. Standalone replication, the subject of the +previous chapter, has nothing in this slot — which is why its +failover story ends in split-brain and Raft's does not. + +**Clock skew**: wall clocks drift and jump, so "leader for the next 5 +seconds" (a **lease**) requires bounded clock error, while ReadIndex +(Step 5) needs no clock at all — it uses a message round instead of +time. That is not a philosophical preference; it is the difference +between an assumption you can test and one you can only hope for. ### Step 5 — linearizability: the single-copy illusion, defined -Linearizability is the strongest single-object guarantee: the system -behaves as if there were exactly ONE copy of the data, with every -operation taking effect atomically at some instant between its start -and its ack. The test-worthy form: there exists a single total order -of operations, consistent with real time — once any read returns a -value, all later reads return it or newer. +> **In:** a Raft cluster whose *writes* are already linearizable. +> **Out:** the definition, the reason reads are a separate problem, +> and the two priced fixes as they appear in raft-rs's own config. + +**Linearizability** is the strongest single-object guarantee: the +system behaves as if there were exactly ONE copy of the data, with +every operation taking effect atomically at some instant between its +start and its ack. The test-worthy form: there exists a single total +order of operations, consistent with real time — once any read +returns a value, all later reads return it or newer. + +Say what it is *not*, because ch. 7 and ch. 9 get conflated. It is a +**recency** guarantee about one object, not an **isolation level** +about many. Serializability says a set of multi-object transactions +is equivalent to *some* serial order; linearizability additionally +pins that order to real time. You can have either without the other. The trap this topic keeps stepping on: Raft gives linearizable WRITES, but reading from the leader without care is NOT linearizable — a deposed leader partitioned from the majority can serve stale -reads while a new leader commits fresh writes (walk the timeline — -it's question territory). The fixes, priced per read: **ReadIndex** -(confirm leadership with a heartbeat round before serving: one -network round, no clock assumptions) or **leader leases** (serve -freely within a time window: free reads, but correctness now rests -on bounded clock error — Step 4's problem). Async replicas serve -stale reads by design; that's not a bug, it's the A in Step 6. +reads while a new leader commits fresh writes. Your own test builds +exactly that world: +`stale_leader_uncommitted_entry_is_overwritten` +(`experiments/src/raft.rs:180`) strands a leader with one buddy in a +2-of-5 minority, lets the majority commit under a higher term, then +heals and asserts the stale entry is gone. Everything that test does +to *writes*, an uninstrumented read would have exposed to a client. + +raft-rs prices the two fixes as an enum. `ReadOnlyOption` +(`src/read_only.rs:26-37`) has exactly two variants, and its own +doc comments state the trade: + +```rust +// raft-rs src/read_only.rs — ReadOnlyOption, 24-37 + 24 /// Determines the relative safety of and consistency of read only requests. + 25 #[derive(Debug, PartialEq, Eq, Clone, Copy, Default)] + 26 pub enum ReadOnlyOption { + 27 /// Safe guarantees the linearizability of the read only request by + 28 /// communicating with the quorum. It is the default and suggested option. + 29 #[default] + 30 Safe, + 31 /// LeaseBased ensures linearizability of the read only request by + 32 /// relying on the leader lease. It can be affected by clock drift. + 33 /// If the clock drift is unbounded, leader might keep the lease longer than it + 34 /// should (clock can move backward/pause without any bound). ReadIndex is not safe + 35 /// in that case. + 36 LeaseBased, + 37 } +``` + +And it spends them at `src/raft.rs:2168-2182`: `Safe` calls +`bcast_heartbeat_with_ctx` (`:2174`) — one heartbeat round to the +quorum before the read is answered — while `LeaseBased` answers +immediately from `self.raft_log.committed` (`:2177-2180`). Before +either, `:2146-2154` refuses the read outright if the leader has not +yet committed an entry in its own term (`commit_to_current_term()`), +which is Raft §8's no-op-entry rule showing up as a guard clause. + +The defaults tell you which one the authors trust: + +``` + raft-rs pinned defaults, src/config.rs + read_only_option : ReadOnlyOption::Safe (:124, and #[default] at read_only.rs:29) + check_quorum : false (:120) + + And LeaseBased is not merely discouraged, it is REFUSED unless you + opt in: src/config.rs:204-206 returns a config error for + read_only_option == LeaseBased && !check_quorum + + Cost per read, n = 5: + Safe 1 broadcast (4 msgs) + acks from a quorum + -> ~1 RTT added to EVERY read; no clock assumption + LeaseBased 0 messages + -> free, and correctness now rests on bounded clock + error, i.e. Step 4's unverifiable assumption + + Using this topic's measured ack p50 of 13.8 us as an RTT proxy, Safe + costs ~14 us of added latency per read. That is the price of not + trusting a clock. +``` + +Async replicas serve stale reads by design; that's not a bug, it's +the A in Step 6. ### Step 6 — CAP, consensus equivalence, and the FLP dodge +> **In:** a network partition, and a pile of impossibility results. +> **Out:** CAP stated narrowly enough to be true, the equivalence +> that closes the escape hatches, and why FLP does not doom Raft. + The closing vocabulary, three items: - **CAP, properly**: during a network Partition, choose Available-but-stale or Consistent-but-unavailable on the minority - side. valkey chose A; Raft chose C. Our - `minority_partition_cannot_commit` test IS the C choice, executed - — three nodes keep committing, two freeze. -- **Consensus ≡ atomic broadcast ≡ CAS**: ch. 9's equivalence - proofs. Solve any one and you've solved the others — which is why - "just use a CAS register" is not an escape from consensus. + side. It is a claim about one failure mode, not a general + three-way menu — with no partition you get both. valkey chose A; + Raft chose C. Your `minority_partition_cannot_commit` test + (`experiments/src/raft.rs:158-177`) IS the C choice, executed: it + strands a leader with one buddy, proposes, and asserts + `committed() == []` — three nodes keep committing, two freeze, + forever, by design. +- **Consensus ≡ atomic broadcast ≡ linearizable compare-and-set**: + ch. 9's equivalence results. Solve any one and you have solved the + others, and — the direction that matters — needing any one means + you need consensus. "Just use a CAS register" and "just use a + totally-ordered log" are not escapes; they are the same problem + renamed. Worth carrying into design review, where it kills a lot of + proposals in one sentence. - **FLP**: in a fully asynchronous system (no timing assumptions at all), no deterministic consensus protocol can be *guaranteed* to terminate. Raft's randomized timeouts are the practical dodge — - termination with probability 1, not certainty — not a refutation. - One-sentence version for question 4. + the protocol is no longer deterministic, so termination with + probability 1 is available — and timeouts smuggle in the timing + assumption FLP forbids. Raft §5.6 states the assumption openly as + `broadcastTime ≪ electionTimeout ≪ MTBF`. FLP says you cannot get + *guaranteed* termination for free; Raft agrees and buys it. One- + sentence version for question 4. ## How to read the chapters (with the concepts in hand) - **Ch. 5 (Replication)** — Steps 1–3. Read the anomaly catalog - slowly and the log-format section with valkey's `propagateNow` - open ([reading-valkey-replication.md](reading-valkey-replication.md) - Step 2 is the same fork). Skim multi-leader/leaderless — they - return in topic 31. + slowly and the log-format section with valkey's `SPOP` rewrite open + ([reading-valkey-replication.md](reading-valkey-replication.md) + Step 2 is the same fork, and `src/t_set.c:975` is the punchline). + Skim multi-leader/leaderless — they return in topic 31. - **Ch. 8 (The Trouble with Distributed Systems)** — Step 4. The chapter is long; extract exactly three things — timeouts as guesses, pauses + fencing tokens, clock skew vs leases — and move - on. + on. When it reaches fencing tokens, stop and open + `raft-rs src/raft.rs:1416`; the chapter's argument and that line + are the same claim. - **Ch. 9 (Consistency and Consensus)** — Steps 5–6, with the Raft paper ([reading-raft-paper.md](reading-raft-paper.md)) beside it. The linearizability definition deserves a re-read until the - deposed-leader timeline is obvious; the equivalence section can be - read for the statements alone, proofs skimmed. + deposed-leader timeline is obvious; then run + `experiments/src/raft.rs:180` in your head. The equivalence section + can be read for the statements alone, proofs skimmed. ## Questions for notes.md @@ -169,18 +459,252 @@ The closing vocabulary, three items: ## Done when -- [ ] You can name the three read anomalies lag produces and give a user-visible symptom for each. -- [ ] You can state what statement, WAL-byte and row shipping each make hard, and which one valkey uses. -- [ ] You can explain what a fencing token prevents that a timeout cannot. +Answer each before unfolding it. + +- [ ] You can name the three read anomalies lag produces, give a user-visible symptom for each, and say what the fix costs. + +
Answer + + **Read-your-writes**: you post a comment, refresh, and it is gone — + the refresh landed on a replica behind your write. Fix: track the + replication offset your write reached and refuse replicas behind + it (valkey exposes the raw material at `src/replication.c:4953` + and `:4962-4975`). Cost: per-session bookkeeping, plus a wait or a + divert to the leader. + + **Monotonic reads**: you refresh twice and the second refresh shows + *less* than the first — two reads landed on replicas at different + offsets. Fix: pin the session to one replica. Cost: load-balancing + freedom, and no freshness guarantee at all — a pinned session can + be monotonic *and* 26 entries stale (Step 1's arithmetic). + + **Consistent prefix**: you see the answer before the question, + because two causally-related writes went to differently-lagged + partitions. Fix: causally-ordered delivery, or keep the causal set + in one partition. Cost: ordering machinery across partitions. + + None of the three is linearizability; all three are cheaper, which + is the point of naming them separately. + +
+ +- [ ] You can state what statement, WAL-byte and row shipping each make hard, and point at the line where valkey pays the statement tax. + +
Answer + + Statement-based makes **nondeterminism** hard: every random choice, + clock read, or local side effect must be rewritten before it ships. + Physical WAL makes **version coupling** hard: the replica must + understand the leader's page layout and engine version. Logical/row + makes **size** hard, and needs a schema-aware encoder — in exchange + it is the only one an outside consumer can read, which is why CDC + uses it. + + valkey is statement-based, and the tax is paid per command inside + the command: `spopCommand` picks a random member at + `src/t_set.c:970` and rewrites the command into a deterministic + `SREM` at `src/t_set.c:975`. The count variant does it in bulk — + batched `alsoPropagate` SREMs at `src/t_set.c:922` and `:937`, a + `DEL`/`UNLINK` when the set empties at `:790-791`, and + `preventCommandPropagation` at `:949` to suppress the original. + + M15 stage 1 ships physical WAL bytes, so it has none of this + problem and all of the coupling one. + +
+ +- [ ] You can explain what a fencing token prevents that a timeout cannot, and name the pinned line where Raft checks one. + +
Answer + + A timeout decides *when* to stop believing in a node; it cannot + stop a node that has already stopped believing in itself and then + changed its mind. The GC-paused leader wakes convinced it still + leads, and no timeout on any other node can prevent it from sending + a write. A **fencing token** — a monotonically increasing number + issued with authority and checked by every downstream recipient — + can, because the sleeper's number is old. + + In raft-rs the token is the term, and the check is + `} else if m.term < self.term {` at `src/raft.rs:1416`, whose + general arm logs "ignored a message with lower term" and returns at + `:1466-1477` without touching the log. + + valkey's replication stream has no ordered token: `replid` is 40 + random hex chars (`CONFIG_RUN_ID_SIZE = 40`, `src/server.h:152`; + `changeReplicationId` calls `getRandomHexChars` at + `src/replication.c:2063-2066`), so two ids cannot be ranked. + valkey's real epochs — `currentEpoch` / `configEpoch`, + `src/cluster_legacy.h:278-281` — are monotonic but live in cluster + gossip, not in the replication stream. + +
+ - [ ] You can define linearizability precisely enough to say why it is a recency guarantee and not an isolation level. + +
Answer + + There exists a single total order over all operations on the + object, consistent with real time, such that each operation appears + to take effect atomically at one instant between its invocation and + its response. Consequence: once any read returns a value, every + later read returns that value or a newer one. + + It is about **one object** and about **recency**. Serializability + is about **many objects** and about **equivalence to some serial + order**, with no requirement that the order respect wall-clock + precedence — a serializable system may legally order your + transaction before one that finished an hour earlier. + Strict serializability is the conjunction of the two. + + So "we're serializable" does not answer "will my read see my + write", and "we're linearizable" does not answer "can two of my + updates interleave". + +
+ +- [ ] You can state the cost per read of ReadIndex versus a leader lease, and say which one raft-rs makes you opt into. + +
Answer + + **ReadIndex** (`ReadOnlyOption::Safe`, `src/read_only.rs:26-30`): + the leader records its commit index, broadcasts a heartbeat with a + context (`bcast_heartbeat_with_ctx`, `src/raft.rs:2174`), and + answers only once a quorum has replied. Cost: one round trip per + read — roughly 14 µs using this topic's measured ack p50 of + 13.8 µs as an RTT proxy. Assumption: none about clocks. + + **Leader lease** (`LeaseBased`, `read_only.rs:31-36`): answer + immediately from `self.raft_log.committed` + (`src/raft.rs:2177-2180`). Cost: zero messages. Assumption: + bounded clock error — and the enum's own doc says an unbounded + drift means "ReadIndex is not safe in that case". + + raft-rs defaults to `Safe` (`#[default]` at `read_only.rs:29`, + `src/config.rs:124`) and actively refuses `LeaseBased` unless you + also set `check_quorum`, which defaults to `false` + (`src/config.rs:120`, error at `:204-206`). Both paths are gated by + `commit_to_current_term()` at `src/raft.rs:2146-2154` — a leader + that has not yet committed in its own term serves no reads at all. + + For M22, `Safe` is the default answer: one RTT is cheap next to the + measured 2133 µs p99 the write path already carries, and it costs + no assumption you cannot test. + +
+ - [ ] You can fill in the 2x3 matrix of {async, semi-sync, raft} x {read-your-writes, monotonic reads, consistent prefix}. + +
Answer + + The trap is that none of the three rows gives you any of the three + columns *by itself* — every cell is "only if you also do X", and + naming X is the exercise. + + Async replication with replica reads: none of the three hold. It + gives read-your-writes only if the session is routed to the leader + or gated on an offset; monotonic reads only if the session is + pinned; consistent prefix only within a single partition's stream. + + Semi-sync (valkey `WAIT n`): read-your-writes becomes *purchasable* + — the write blocks until n replicas ack — but note what the ack + means. `replicationCountAcksByOffset` + (`src/replication.c:4962-4975`) counts replicas whose + `repl_ack_off` has passed the offset, i.e. **received**, not + fsynced. `WAITAOF` (`waitaofCommand`, `src/replication.c:5030`, + counting via `:4979`) is the durability-aware sibling. Monotonic + reads and consistent prefix are unchanged: still routing problems. + + Raft with ReadIndex reads: all three hold, because linearizability + implies all three. Raft with *unguarded leader reads*: none are + guaranteed, per Step 5's deposed-leader case — which is the whole + reason `ReadOnlyOption` exists. + +
+ +- [ ] You can trace the WAIT-1-then-failover sequence and name both the ch. 5 guarantee that broke and the ch. 9 property that would have held. + +
Answer + + Sequence: the client writes and calls `WAIT 1`; replica B acks + receipt at the right offset, so `replicationCountAcksByOffset` + (`src/replication.c:4962-4975`) returns 1 and `waitCommand` + (`src/replication.c:4996-5026`) succeeds. The primary dies. The + operator promotes replica **C**, which never received the write. + `shiftReplicationId` (`src/replication.c:2082-2095`) gives C a new + random `replid` and history continues without the entry. The client + reads and its acked write is gone. + + Broken: durability of an acknowledged write, and with it + read-your-writes. The ch. 9 property that would have prevented it + is Raft's **Leader Completeness** — a node lacking a committed + entry cannot win an election, enforced by the up-to-dateness check + in §5.4.1. valkey has no such restriction because it has no votes; + promotion is whatever the operator or sentinel says. + + Two further traps in the same sequence. `WAIT` counts *received*, + not fsynced (`WAITAOF`, `src/replication.c:5030`, is the one that + counts `repl_aof_off`); and even a fsynced ack is only durable if + the flush was a real one — topic 5 measured that on macOS/APFS + `F_FULLFSYNC` runs at 337 commits/s, which is why this topic's + per-entry-fsync row sits at 341 entries/s. A cheap `fsync(2)` that + returns in microseconds on that platform proved nothing. + +
+ - [ ] You wrote answers to all five questions in notes.md, including why FLP does not doom Raft in practice. +
Answer + + One sentence: FLP forbids *guaranteed* termination for a + deterministic protocol in a fully asynchronous model, and Raft is + neither — its election timeouts assume bounded-enough timing + (§5.6's `broadcastTime ≪ electionTimeout ≪ MTBF`) and its + randomization makes it non-deterministic, so it terminates with + probability 1 rather than by proof. + + The empirical half of the answer is Figure 16 of the extended Raft + paper: with no randomness the 5-server cluster consistently took + over 10 s to elect; with 5 ms of randomness the median downtime was + 287 ms; with a 12–24 ms timeout the average was 35 ms and the worst + case 152 ms. FLP is not violated by any of that — none of it is a + *guarantee* — and none of it matters to an operator. + + Your own crate is where the assumption becomes a constant: + `ELECTION_TIMEOUT_MIN/MAX = 10/20` ticks against + `HEARTBEAT_INTERVAL = 3` (`experiments/src/raft.rs:33-35`), a + 3.3–6.7× ratio where raft-rs ships 10× (`src/config.rs:112,115-116`, + where the constant is literally `HEARTBEAT_TICK * 10`). + +
+ ## References **Papers / Books** -- Kleppmann — "Designing Data-Intensive Applications" (O'Reilly - 2017) — ch. 5 (Replication), ch. 8 (The Trouble with Distributed - Systems), ch. 9 (Consistency and Consensus); pair ch. 5 with - [reading-valkey-replication.md](reading-valkey-replication.md) and - ch. 9 with [reading-raft-paper.md](reading-raft-paper.md) +- Martin Kleppmann — *Designing Data-Intensive Applications* + (O'Reilly, 2017) — ch. 5 (Replication), ch. 8 (The Trouble with + Distributed Systems), ch. 9 (Consistency and Consensus). Pair ch. 5 + with [reading-valkey-replication.md](reading-valkey-replication.md) + and ch. 9 with [reading-raft-paper.md](reading-raft-paper.md). + Copyrighted; cited by chapter number and paraphrased here, never + quoted. +- Ongaro, Ousterhout — *In Search of an Understandable Consensus + Algorithm (Extended Version)* — §5.4.1 (up-to-dateness, the + Leader Completeness enforcement), §5.6 (the timing inequality), + §8 (the no-op entry behind `commit_to_current_term`), Figure 16 + (measured election downtime). See + [reading-raft-paper.md](reading-raft-paper.md) for the + extended-vs-ATC'14 disambiguation. + +**Code** — all anchors are valkey at `8891441ab` and raft-rs at +`ad13f3d`; `experiments/` is this topic's own crate +- [valkey](https://github.com/valkey-io/valkey) — + `src/t_set.c:970,975` (the SPOP rewrite), `src/replication.c:4953`, + `:4962-4975`, `:4996-5026`, `:5030` (WAIT / WAITAOF), + `:2063-2066`, `:2082-2095` (replid), `src/cluster_legacy.h:278-281` + (the epochs that *are* fencing tokens) +- [raft-rs](https://github.com/tikv/raft-rs) — `src/raft.rs:1416` + and `:1466-1477` (terms as fencing tokens), + `src/read_only.rs:26-37` and `src/raft.rs:2146-2182` (ReadIndex vs + lease), `src/config.rs:120`, `:124`, `:204-206` (the defaults that + encode the authors' opinion) diff --git a/topics/15-replication-consensus/reading-qdrant-consensus.md b/topics/15-replication-consensus/reading-qdrant-consensus.md index 904de91..efdc03e 100644 --- a/topics/15-replication-consensus/reading-qdrant-consensus.md +++ b/topics/15-replication-consensus/reading-qdrant-consensus.md @@ -5,24 +5,36 @@ cluster METADATA only — collection schemas, shard placement, peer membership. The vectors themselves replicate OUTSIDE raft, through replica sets with an ack-count knob. This chapter builds the design step by step — the split, the arithmetic that forces it, the -production raft-rs driving loop, and the weaker data-path contract — -walking `src/consensus.rs` (the loop from +production raft-rs driving loop, the *real* ordering inside +`on_ready`, and the weaker data-path contract — walking +`src/consensus.rs` (the loop from [reading-raft-rs.md](reading-raft-rs.md), in production) and `lib/collection`. Assumes both the Raft paper and raft-rs chapters. +Every `file:line` below is **qdrant at `44ad62f`**, the revision in +this repo's pin table (`resources/codebases.md`). Check any of them +with `python3 tools/pinned-source.py show qdrant src/consensus.rs -r +926:1007`. At this pin `src/consensus.rs` is 1605 lines. + ## The problem in one sentence Pushing every vector upsert through Raft costs a majority round trip -plus a log fsync per write — at topic 5's ~1 ms-ish fsync floor -that's a ceiling around **~1K sequential commits/s** against a bulk -ingest doing 10K+ upserts/s — so qdrant routes the 10K/s through a -cheaper path and reserves Raft for the ~1/minute decisions that -must never fork. +plus a log fsync per write — and this repo has measured that floor, +not guessed it: topic 5's `F_FULLFSYNC` rung is **337 commits/s** and +topic 15's own `repl_lag` bench gets **341 entries/s** when the +follower fsyncs every entry — so a bulk ingest doing 10K upserts/s +would be running ~29× over the ceiling, and qdrant routes the 10K/s +through a cheaper path while reserving Raft for the ~1/minute +decisions that must never fork. ## The concepts, step by step ### Step 1 — two planes: what must agree vs what must flow +> **In:** every kind of write a vector database takes. **Out:** the +> partition of those writes by the cost of a disagreement, and the +> two consistency stories that partition creates. + Split the system's writes by what a disagreement would cost. If two nodes disagree on *where shard 3 lives* or *whether collection X exists*, the cluster is broken — routing forks, splits brain. If two @@ -35,7 +47,7 @@ vectors — high volume, tolerates repair → replica sets): ┌─ raft (consensus.rs) ──────────────────────────┐ │ topology: which peers exist, which shard lives │ │ where, collection create/drop, replica state │ - │ (Active/Dead/Partial) — LOW volume │ + │ (11 variants, Step 5) — LOW volume │ └────────────────────────────────────────────────┘ ┌─ data path (NO raft) ──────────────────────────┐ │ point upserts → forwarded to ALL replicas of │ @@ -46,87 +58,256 @@ vectors — high volume, tolerates repair → replica sets): Same call as kafka (controller raft vs ISR data path). The cost: the system now has TWO consistency stories, and every failure -scenario must be reasoned about across both (Step 5's question). +scenario must be reasoned about across both — and as Step 5 shows, +the two are not actually independent, because the data path escalates +into consensus when a replica fails. ### Step 2 — the arithmetic that forces the split +> **In:** two write rates — metadata changes and point upserts — and +> this repo's measured fsync ladder. **Out:** the ratio that makes +> one of them affordable through Raft and the other not, computed +> rather than asserted. + Metadata changes happen when an operator creates a collection or a -node dies — call it once a minute. Point upserts arrive at 10K+/s -during ingest. Raft's per-commit price (majority RTT + leader and -follower log fsyncs, serialized by the log) is irrelevant at -1/minute and fatal at 10K/s — batching helps but the log is still -one serialized sequence through one leader, for writes that don't -need a total order in the first place: upserts to different points -commute. Consensus buys a property (one agreed order, no acked-write -loss) that the data plane doesn't need at a price it can't pay — -question 1 makes you run the numbers with topic 5's fsync -measurements. +node dies. Point upserts arrive at 10K+/s during ingest. Put real +numbers on both sides: + +``` + Measured floors from this repo (Apple M3 Pro / APFS, 2026-07-28): + + topic 5, F_FULLFSYNC rung 337 commits/s + topic 15, repl_lag, fsync-every-entry + 341 entries/s ack p50 2967.0 us + topic 15, repl_lag, fsync every 64 12,187 entries/s ack p50 14.0 us + + Note the first two agree to ~1%. That is the point: a Raft commit + with a durable follower ack IS a durable flush, so the consensus + path cannot beat topic 5's fsync rung. + + Control plane: 1 metadata change / minute = 0.017 ops/s + 0.017 / 341 = 0.005% of one node's commit budget + + Data plane: 10,000 upserts/s + 10,000 / 341 = 29.3x over the ceiling + + Batching moves the ceiling, not the shape: group-commit at 64 gives + 12,187 entries/s, still 1.2x short of 10K/s with zero headroom for + the p99 (2133.0 us at that setting), and it does nothing about the + log being ONE serialized sequence through ONE leader. +``` + +That last clause is the deeper reason. Even a free fsync would leave +consensus imposing a total order on writes that do not need one: +upserts to different points commute. Consensus buys a property (one +agreed order, no acked-write loss) that the data plane does not need, +at a price it cannot pay. Question 1 makes you redo this with your +own hardware's numbers. ### Step 3 — the driving loop: raft-rs's contract, in production +> **In:** a `RawNode` and a process that must feed it. **Out:** the +> event sources, the batching constants, and the tick-to-milliseconds +> conversion for qdrant's actual config. + `Consensus` (consensus.rs:48) owns `type Node = RawNode` (:36) and runs the loop the raft-rs -chapter promised someone must write: a thread selecting over -{incoming raft messages, a proposal channel, a tick timer}, calling -`step`/`tick`, then draining Ready (:537 the loop, :877 `on_ready`): +chapter promised someone must write. `Consensus::start` (:481-562) is +that loop: ```rust -// the whole of consensus.rs, condensed: raft-rs decides, this loop does -fn run(&mut self) { - loop { - match self.select_with_timeout(TICK) { - Recv::RaftMsg(m) => self.node.step(m).ok(), // network in - Recv::Propose(op) => self.node.propose(vec![], op.encode()), - Recv::Timeout => self.node.tick(), // clock in - } - if !self.node.has_ready() { continue; } - let mut rd = self.node.ready(); - self.storage.persist(rd.entries(), rd.hs()); // 1. fsync FIRST - self.transport.send(rd.take_messages()); // 2. then talk - for e in rd.take_committed_entries() { - self.topology.apply(e); // 3. committed → cluster metadata - } - self.node.advance(rd); // 4. done - } +// ILLUSTRATION — the shape of Consensus::start, not a quote. +// The real loop is consensus.rs:499-561; advance_node is :564-631; +// recv_update's tokio::select! is :633-641; on_ready is :877-902. +loop { + let raft_messages = self.advance_node(tick_period)?; // :501 + // ... elapsed-tick bookkeeping, :504-529 ... + for _ in 0..report_ticks { self.node.tick(); } // :532-534 + let (stop_consensus, is_idle) = self.on_ready()?; // :537 + if stop_consensus { return Ok(()); } + // ... idle-cycle counting, :543-560 ... } ``` -The "state machine" being replicated is the cluster topology map — -`apply(e)` mutates which peers exist and where shards live. Question: -find where snapshots trigger — what happens when a new peer joins -and the raft log has been compacted? +Three things in that loop are worth the read rather than the summary. + +**Batching.** `advance_node` (:564-631) drains up to +`RAFT_BATCH_SIZE = 128` events per iteration (:575, :625), waiting +`tick_period` for the first and only `tick_period / 10` for each +subsequent one (:578-585). A conf-change breaks the batch early +(:597-608, :625) because raft-rs allows only one in flight. + +**Tick arithmetic.** raft-rs's constants are unitless; qdrant +supplies the unit: + +``` + qdrant config/config.yaml:359 tick_period_ms: 100 + raft-rs config.rs:112-116 heartbeat_tick 2, election_tick 20 + raft-rs config.rs:147-163 election range [election_tick, 2 x election_tick) + raft-rs raft.rs:2854-2866 draws uniformly from [20, 40) + + heartbeat = 2 x 100 ms = 200 ms + election = [20, 40) x 100 ms = [2.0 s, 4.0 s) + + The Raft paper (§5.6, Fig 16) recommends 150-300 ms. qdrant's floor + is 2.0 s — about 7x the paper's ceiling. +``` + +Why so slack? Because the loop body at :537 does disk work — WAL +appends, snapshot application, WAL compaction (:899) — and a tight +timeout would misfire whenever a write is slow. qdrant says so in the +code: the comment at :509-519 explains that reported ticks are capped +at `election_tick - 5` = 15 (:521-529) precisely so that "if last +iteration of the loop took too long to complete" it does not "trigger +unnecessary leader election." -### Step 4 — on_ready: the ordering rules, obeyed and optimized +**What the state machine is.** The replicated state machine is the +cluster topology map — `handle_committed_entries` (:997, :1044) +mutates which peers exist and where shards live. Compaction is at +:899 with `compact_wal_entries: 128` (config.yaml:365), which is what +lets a new peer join from a snapshot rather than replaying history. -Follow `on_ready` (:877-1017) and check the raft-rs contract's -ordering: persist entries + HardState → send messages → apply -committed entries → advance. This is the part the library couldn't -enforce, done right in production — the fsync-before-send rule from -the raft-rs chapter, visible as real code. The optimization: -`LightReady` (:928, vs full Ready handling at :885/:1017) is -raft-rs's `advance_append` split in action — messages that don't -depend on fresh persistence go out without waiting for the fsync -round, pipelining the raft log the way topic 5 group-commits a WAL. +### Step 4 — on_ready: the ordering rules, as actually written + +> **In:** a `Ready` bundle from `RawNode::ready()`. **Out:** the true +> order of the seven operations qdrant performs on it, which of them +> is a safety requirement, and the one that contradicts the naive +> "persist before send" rule. + +`on_ready` (:877-902) is three calls: `process_ready` (:926-1007), +then `process_light_ready` (:1015-1050), then `process_role_change` +(:904-918). Both inner functions open with the same warning comment +(:922, :1011): *"The order of operations in this functions is +critical, changing it might lead to bugs."* + +Read the actual order in `process_ready`, because it is **not** +persist-then-send: + +```rust +// consensus.rs — process_ready, 939-1005 (logging and error paths elided) + 939 if !ready.messages().is_empty() { + 941 self.send_messages(ready.take_messages()); // 1. SEND + 942 } + 944 if !ready.snapshot().is_empty() { + 950 if let Err(err) = store.apply_snapshot(&snapshot)? { // 2. snapshot + 953 } + 955 if !ready.entries().is_empty() { + 961 .append_entries(ready.take_entries()) // 3. log append + 963 } + 965 if let Some(hs) = ready.hs() { + 971 .set_hard_state(hs.clone()) // 4. HardState + 973 } + 984 if !ready.persisted_messages().is_empty() { + 990 self.send_messages(ready.take_persisted_messages()); // 5. SEND (gated) + 991 } + 993 let committed_entries = ready.take_committed_entries(); + 997 let stop_consensus = handle_committed_entries(...) // 6. apply + 1005 let light_rd = self.node.advance(ready); // 7. advance +``` + +Line **941 sends before line 961 persists.** That is not a bug and it +is not qdrant being sloppy — it is raft-rs's leader exemption, +consumed correctly. `Ready::messages()` is non-empty only when +`is_persisted_msg` is false, which `RawNode::ready` sets as +`raft.state != StateRole::Leader` (raft-rs raw_node.rs:555, citing +Ongaro's *dissertation* §10.2.1 at :554). So the list drained at 941 +is a leader's, and a leader may replicate before its own disk write. + +The safety-critical ordering is between **3/4 and 5**: +`persisted_messages()` at 984 is drained *after* the append at 961 +and the HardState write at 971. Those are the messages a follower or +candidate sends — the acks and vote responses the leader will count — +and they must not leave before the write. A second ordering +constraint is spelled out in the comment at :996: committed entries +are handled after the HardState save "so that `applied` index is +never bigger than `commit`." + +`process_light_ready` (:1015-1050) then does commit index (:1029-1036) +→ send (:1038) → apply (:1040-1045) → `advance_apply` (:1048). This is +raft-rs's `advance_append`/`advance_apply` split in action: entries +were made durable in the first phase, so the second phase's messages +and applies do not wait on another disk round. + +Where the fsync actually happens is worth checking yourself, because +neither write in that listing is obviously durable. +`ConsensusOpWal::append_entries` +(lib/storage/src/content_manager/consensus/consensus_wal.rs:160-262) +ends with **one** `self.wal.flush_open_segment()` at :259 — one flush +per Ready batch, not per entry. That is group commit, and it is why +the arithmetic in Step 2 uses the batched rung. The HardState is +separate: `Persistent::save` +(lib/storage/src/content_manager/consensus/persistent.rs:375-384) +serialises `{term, vote, commit}` (the `HardStateDef` at :455-459) as +JSON through `atomicwrites::AtomicFile`, and the only flush in +qdrant's own code is a `BufWriter::flush` at :379 — a userspace +flush. Whether that is durable depends on the `atomicwrites` crate's +temp-file-and-rename policy, not on anything in this tree. That is +question 5's real target. ### Step 5 — the data plane: replica sets with a knob, membership by raft +> **In:** a point upsert for a shard with three replicas. **Out:** +> the ack rule, the real replica-state enum, and the moment the data +> path stops being independent of consensus. + A point upsert goes to ALL replicas of its shard; `write_consistency_factor` of them must ack before the client does — valkey's WAIT as a per-write policy (previous chapters' axis: WHO -acks). The twist that makes it better than plain WAIT: **replica -state lives in raft**. A replica that misses writes is marked Dead -*through consensus* — every node agrees it's Dead — and must -complete a shard transfer (re-sync) before becoming Active again: +acks). The rule is `minimal_success_count = +write_consistency_factor.min(replica_count)` +(lib/collection/src/shards/replica_set/update.rs:460), so the factor +is **clamped** to the replica count and cannot make a write fail for +asking more acks than there are replicas. + +Both defaults are **1**: `default_replication_factor` +(lib/collection/src/config.rs:223-225) and +`default_write_consistency_factor` (:227-229), matching +`config/config.yaml:211`. Out of the box qdrant is a single-copy +system; the knob only starts meaning something once you raise the +replication factor. + +Replica state is not the three-state triangle it is usually drawn as. +`ReplicaState` +(lib/collection/src/shards/replica_set/replica_set_state.rs:100-133) +has **eleven** variants: `Active`, `Dead`, `Partial`, `Initializing`, +`Listener`, `PartialSnapshot`, `Recovery`, `Resharding`, +`ReshardingScaleDown`, `ActiveRead`, `ManualRecovery`. Three +predicates carve them up — `is_active` (:138-153, source of truth), +`is_readable` (:156-171), `is_updatable` (:173-) — and they do not +agree: `ActiveRead` is readable but not a source of truth, +`ReshardingScaleDown` is both. The simplified triangle is a teaching +aid, not the enum: ``` + ILLUSTRATION of the main cycle only — the real enum has 11 variants + at replica_set_state.rs:100-133 + Active ──(missed writes, marked via raft)──► Dead ▲ │ └──(shard transfer completes)── Partial ◄────┘ ``` -That closes plain WAIT's nastiest hole: valkey can promote a replica -nobody agrees is current, silently dropping acked writes; qdrant's -failover choices are constrained by an agreed replica-state map. +Now the part that makes this better than plain WAIT, and the part +that makes the two planes *not* independent. When a write succeeds on +enough replicas but fails on others, qdrant deactivates the failed +ones **through consensus** and blocks the client until that +deactivation commits (update.rs:530-590). If it does not commit in +`DEFAULT_SHARD_DEACTIVATION_TIMEOUT` = 30 s (update.rs:30) the client +gets an error whose text is the honest summary of the whole design: + +``` + "Some replica of shard N failed to apply operation and deactivation + timed out after 30s. Consistency of this update is not guaranteed. + Please retry." — update.rs:585-586 +``` + +So a qdrant upsert on the happy path pays no consensus cost, and a +qdrant upsert that touches a failing replica pays a full Raft round +trip before it can answer. The escalation is what closes plain WAIT's +nastiest hole — valkey can promote a replica nobody agrees is +current, silently dropping acked writes; qdrant's failover choices +are constrained by an agreed replica-state map. + What remains open: a write acked at `write_consistency_factor = 1` that dies with its only holder during a failover race — the consensus layer agrees on *who is Dead*, not on *every write* (the @@ -135,50 +316,241 @@ stage 2 pushes the WAL itself through raft). ## Where each step lives in the code +All anchors are qdrant at `44ad62f`. + | anchor | what it is | step | |---|---|---| -| consensus.rs:36 | `type Node = RawNode` | 3 | -| consensus.rs:48 | `struct Consensus` — the driving loop owner | 3 | -| consensus.rs:537 | the ready loop: tick / step / process | 3 | -| consensus.rs:877 | `on_ready` — drain the Ready bundle | 4 | -| consensus.rs:885/928/1017 | Ready vs LightReady handling | 4 | -| lib/collection | shard replication, `write_consistency_factor`, replica states | 5 | - -Read order: the loop at :537 with the condensed Rust above in hand, -then `on_ready` checking the 1-2-3-4 ordering, then grep -`lib/collection` for `write_consistency_factor` and the -Active/Dead/Partial state machine. Also hunt the Storage impl behind -`ConsensusStateRef` — where the raft log and HardState actually get -persisted (question 5). +| src/consensus.rs:36 | `type Node = RawNode` | 3 | +| src/consensus.rs:48 | `struct Consensus` — the driving loop owner | 3 | +| src/consensus.rs:481-562 | `start` — the loop; `on_ready` called at :537 | 3 | +| src/consensus.rs:509-529 | why reported ticks are capped at 15 | 3 | +| src/consensus.rs:564-631 | `advance_node` — batch of 128, conf-change breaks early | 3 | +| src/consensus.rs:633-641 | `recv_update` — the `tokio::select!` | 3 | +| src/consensus.rs:877-902 | `on_ready` — the three-call skeleton | 4 | +| src/consensus.rs:899 | `compact_wal` | 3 | +| src/consensus.rs:904-918 | `process_role_change` | 3 | +| src/consensus.rs:922 / 1011 | "the order of operations ... is critical" | 4 | +| src/consensus.rs:939-1005 | `process_ready` — send, snapshot, entries, HardState, persisted-send, apply, advance | 4 | +| src/consensus.rs:996 | why apply follows the HardState save | 4 | +| src/consensus.rs:1015-1050 | `process_light_ready` — commit index, send, apply, advance_apply | 4 | +| config/config.yaml:359 / 365 | `tick_period_ms: 100`, `compact_wal_entries: 128` | 3 | +| config/config.yaml:211 | `write_consistency_factor: 1` | 5 | +| lib/collection/src/config.rs:223-229 | replication/write-consistency defaults, both 1 | 5 | +| .../replica_set/replica_set_state.rs:100-133 | `ReplicaState` — eleven variants | 5 | +| .../replica_set/replica_set_state.rs:138-171 | `is_active` / `is_readable` — and where they disagree | 5 | +| .../replica_set/update.rs:30 | `DEFAULT_SHARD_DEACTIVATION_TIMEOUT` = 30 s | 5 | +| .../replica_set/update.rs:452-460 | `minimal_success_count` and the clamp | 5 | +| .../replica_set/update.rs:530-590 | deactivate-through-consensus, and the client block | 5 | +| .../consensus/consensus_wal.rs:160-262 | `append_entries`; one `flush_open_segment` at :259 | 4 | +| .../consensus/persistent.rs:375-384 | `save` — HardState via `AtomicFile`, `BufWriter::flush` | 4 | +| .../consensus/persistent.rs:455-459 | `HardStateDef { term, vote, commit }` | 4 | + +Read order: the loop at :481 first, then `process_ready` line by line +against raft-rs's `raw_node.rs:553-555` open in another window (that +is where line 941 stops looking wrong), then `update.rs:452-590` for +the data path. Finally hunt the two persistence sites named above — +they are question 5, and one of them has no fsync in qdrant's own +code. ## Questions for notes.md 1. Why is metadata volume low enough for raft but point writes not? Estimate: 10K upserts/s × majority fsync (topic 5 numbers) = ? -2. Replica states Active/Dead/Partial — map each to a Raft Progress - state (replicate/probe/snapshot). Same problem, different layer? +2. Replica states — map the main ones onto a Raft `Progress` state + (Replicate/Probe/Snapshot). Same problem, different layer? 3. What consistency does a qdrant READ get on vectors? Is it linearizable? Under what config? 4. For the capstone: M15 puts the WAL itself through raft (stage 2) — qdrant chose not to. Which is right for a graph database's write volume, and why might FalkorDB's answer differ from qdrant's? -5. Where does qdrant persist the raft log and HardState? Find the - Storage impl behind ConsensusStateRef. +5. Where does qdrant persist the raft log and HardState, and is + either write actually durable? Find the flush in each. ## Done when -- [ ] You can state the arithmetic that forces the metadata/data plane split — why metadata volume fits raft and point writes do not. -- [ ] You can map the replica states Active/Dead/Partial onto Raft's Progress states. -- [ ] You can say what consistency a qdrant vector read actually gets, and whether it is configurable. -- [ ] You can describe the `on_ready` ordering rules and name which of them is a safety requirement rather than an optimization. +Answer each before unfolding it. + +- [ ] You can state the arithmetic that forces the metadata/data plane split, using measured numbers rather than an estimate. + +
Answer + + The ceiling is this repo's own measurement, not a guess: topic 5's + `F_FULLFSYNC` rung is 337 commits/s and topic 15's `repl_lag` bench + gets 341 entries/s with the follower fsyncing every entry. Those + agree because a Raft commit with a durable follower ack *is* a + durable flush. + + Control plane at 1 change/minute is 0.017 ops/s — 0.005% of that + budget. Data plane at 10,000 upserts/s is 29.3× over it. Group + commit at 64 raises the rung to 12,187 entries/s, still short of + 10K/s once you leave headroom for the 2133.0 µs p99. + + And the ratio is not the whole argument. Raft imposes one + serialized order through one leader on writes that commute — + upserts to different points have no ordering requirement at all, so + even a free fsync would leave the data plane paying for a property + it does not use. + +
+ +- [ ] You can state qdrant's heartbeat and election timeouts in milliseconds, and explain why they are far above the paper's recommendation. + +
Answer + + `tick_period_ms: 100` (config/config.yaml:359) times raft-rs's + unitless constants: `heartbeat_tick` 2 and `election_tick` 20 + (raft-rs config.rs:112-116), with the randomized draw taken from + `[election_tick, 2 × election_tick)` = `[20, 40)` (config.rs:147-163, + raft.rs:2854-2866). That is a 200 ms heartbeat and a 2.0–4.0 s + election timeout, against the paper's §5.6 recommendation of + 150–300 ms. + + The reason is in the code. The loop body at consensus.rs:537 does + disk work — WAL append, snapshot apply, `compact_wal` at :899 — so + a single iteration can be slow. The comment at :509-519 says + reported ticks are capped (at `election_tick - 5` = 15, :521-529) + so that a long iteration does not "trigger unnecessary leader + election." + + The paper's own framing covers this: §5.6's `broadcastTime ≪ + electionTimeout` puts the fsync inside broadcastTime, so a system + with slow durable writes must widen the election timeout to match. + +
+ +- [ ] You can describe the real `process_ready` ordering, and say why sending before persisting is correct there. + +
Answer + + consensus.rs:939-1005, in order: send `ready.messages()` (:941), + apply snapshot (:950), append entries (:961), set HardState (:971), + soft state (:981), send `ready.persisted_messages()` (:990), handle + committed entries (:997), `advance` (:1005). Both `process_ready` + and `process_light_ready` open with "The order of operations in this + functions is critical" (:922, :1011). + + The send at 941 precedes the append at 961, which contradicts the + naive rule — and is correct. `Ready::messages()` is non-empty only + when `is_persisted_msg` is false, and raft-rs sets that as + `raft.state != StateRole::Leader` (raw_node.rs:555), citing + Ongaro's dissertation §10.2.1 at :554. A leader may replicate before + its own disk write, because the commit needs a majority of disks and + its own is not required to be among them. + + The safety-critical order is 961/971 before 990: + `persisted_messages()` carries a follower's or candidate's acks and + vote responses, which are the evidence the leader counts. The second + constraint is at :996 — apply after the HardState save, so `applied` + never exceeds `commit`. + +
+ +- [ ] You can say how many replica states qdrant really has, and name a case where "active" and "readable" disagree. + +
Answer + + Eleven, at replica_set_state.rs:100-133: `Active`, `Dead`, + `Partial`, `Initializing`, `Listener`, `PartialSnapshot`, + `Recovery`, `Resharding`, `ReshardingScaleDown`, `ActiveRead`, + `ManualRecovery`. The Active/Dead/Partial triangle is a teaching + simplification of the main transfer cycle. + + `ActiveRead` is the disagreement: `is_readable` (:156-171) returns + true for it, `is_active` (:138-153) returns false. Its comment + (:125) says "Active for readers, Partial for writers" — it can serve + a query but is not a source of truth for a recovery. + `ReshardingScaleDown` goes the other way and is true for both. + + The mapping worth writing in notes: `Partial` is Raft's + `ProgressState::Snapshot` (catching up by bulk transfer), `Active` + is `Replicate`, and `Dead` has no Raft analogue at all — Raft never + removes a voter for lagging, it just keeps probing. + +
+ +- [ ] You can explain how a failed replica turns a consensus-free write path into one that blocks on a Raft commit. + +
Answer + + update.rs:530-590. When `successes.len() >= minimal_success_count` + but some replicas failed, `handle_failed_replicas` (:544) proposes + their deactivation *through consensus*, and if the client asked for + a callback the request then blocks on + `replica_state.wait_for(...)` (:563-579) until every failed peer is + no longer `can_be_source_of_truth()`. + + The timeout is `DEFAULT_SHARD_DEACTIVATION_TIMEOUT` = 30 s + (update.rs:30), and on expiry the client gets an explicit + "Consistency of this update is not guaranteed. Please retry." + (:585-586). + + So the two planes are not independent. The happy path pays nothing + for consensus; the failure path pays a full Raft round trip + *synchronously*, because the alternative — acking a write while some + replica still claims to be a source of truth without it — is how + you lose acked data during the next failover. That is precisely the + hole plain valkey WAIT leaves open. + +
+ +- [ ] You can say where the raft log and HardState are persisted, and whether either write is demonstrably durable. + +
Answer + + The log: `ConsensusOpWal::append_entries` + (consensus_wal.rs:160-262), which ends with a single + `self.wal.flush_open_segment()` at :259 — one flush per Ready + batch, not per entry. That is group commit, and it is the reason + Step 2's arithmetic uses the batched rung rather than the + fsync-every-entry one. + + The HardState: `Persistent::save` (persistent.rs:375-384), which + writes `{term, vote, commit}` plus the ConfState (the `HardStateDef` + at :455-459) as JSON through `atomicwrites::AtomicFile`. + + Only one of the two is demonstrably durable from this tree. The + only flush in `save` is `writer.flush()` at :379 — a `BufWriter` + flush into the file descriptor, not an fsync. Durability there rests + entirely on what the `atomicwrites` crate does on commit, which is + outside qdrant's source. Given Figure 2 lists `votedFor` as + must-be-durable-before-responding, that is the line to go read. + +
+ - [ ] You wrote answers to all five questions in notes.md, including where the raft log and HardState are persisted. +
Answer + + Question 3 is the one with no single code answer, which is itself + the finding. A read served by a replica in `ActiveRead` or `Partial` + can be stale, because the data path has no commit index — there is + no equivalent of `commitIndex` for vectors, only per-replica + acceptance. + + What *is* linearizable is the metadata: collection existence, shard + placement and replica state all go through the Raft log at + consensus.rs:997/1044. So "which replicas may answer" is agreed even + when "what those replicas contain" is not. Writing that sentence + down is the point of the question. + +
+ ## References **Code** -- [qdrant](https://github.com/qdrant/qdrant) — `src/consensus.rs` - (the driving loop; the anchor map above) and `lib/collection` - (shard replication, `write_consistency_factor`, replica states) +- [qdrant](https://github.com/qdrant/qdrant) at `44ad62f` — + `src/consensus.rs` (the driving loop; the anchor map above), + `lib/collection/src/shards/replica_set/` (`update.rs` for the ack + rule and the deactivation escalation, `replica_set_state.rs` for the + eleven-variant enum), `lib/collection/src/config.rs` (the defaults), + `lib/storage/src/content_manager/consensus/` (`consensus_wal.rs` and + `persistent.rs` — the two persistence sites), `config/config.yaml` - The library it embeds is [raft-rs](https://github.com/tikv/raft-rs) - — walked in [reading-raft-rs.md](reading-raft-rs.md) + — walked in [reading-raft-rs.md](reading-raft-rs.md); `raw_node.rs:553-555` + is what makes `process_ready`'s first send legal + +**Papers** +- Diego Ongaro, *Consensus: Bridging Theory and Practice* (Stanford + PhD dissertation, 2014), §10.2.1 — the authority raft-rs cites for + the leader's parallel disk write, and therefore for consensus.rs:941 diff --git a/topics/15-replication-consensus/reading-raft-paper.md b/topics/15-replication-consensus/reading-raft-paper.md index bc1aadb..df0af1b 100644 --- a/topics/15-replication-consensus/reading-raft-paper.md +++ b/topics/15-replication-consensus/reading-raft-paper.md @@ -6,10 +6,18 @@ election, log replication, and safety as separable concerns, plus a strong-leader design that forbids the log-repair cases Paxos allows. Before the paper, this chapter builds the algorithm one concept at a time — the replicated log, terms, elections, the consistency check, -and the two safety rules — ending on the Fig 8 trap that every -homegrown Raft falls into. Read the extended version — the ATC '14 -paper is a cut-down of the tech report; ~18 pages, but §5 is the -whole game. +the two safety rules, and the timing inequality — ending on the +Fig 8 trap that every homegrown Raft falls into. + +**Read the extended version.** There are two documents and they are +not interchangeable. The ATC '14 conference paper is 16 pages; the +extended version at [raft.github.io/raft.pdf](https://raft.github.io/raft.pdf) +is 18 and adds §7 (log compaction) with Figures 12–13. Figures 1–11 +carry the same numbers in both, but the evaluation figures are +renumbered: **extended Fig 14/15/16 = ATC '14 Fig 12/13/14**. Every +figure and section number below is the **extended** version, and +Ongaro's PhD dissertation is a *third* document — do not quote a +section number across them. §5 is the whole game. ## The problem in one sentence @@ -23,12 +31,16 @@ replicating can take acked writes to the grave. ### Step 1 — the replicated log: agree on order, and state follows -A replicated state machine keeps several servers identical by a +> **In:** several servers that must end up holding identical data. +> **Out:** the reduction of "identical state" to "identical log", the +> definition of *committed*, and Raft's one structural restriction. + +A **replicated state machine** keeps several servers identical by a simple trick: if every server starts from the same state and applies the same commands *in the same order*, they end in the same state. So the servers don't replicate state — they replicate a **log** (a -numbered, append-only sequence of commands), and consensus reduces -to one question: what is entry #i? +numbered, append-only sequence of commands, first index 1 per +Figure 2), and consensus reduces to one question: what is entry #i? ``` index: 1 2 3 4 @@ -39,9 +51,14 @@ to one question: what is entry #i? An entry is **committed** when the protocol guarantees it will never be removed from anyone's log; only committed entries are applied to -the state machine. Raft's structural simplification over Paxos: only -one node — the **leader** — ever appends, and entries flow one -direction, leader → followers: +the state machine. Figure 2 splits this into two volatile counters: +`commitIndex`, the highest entry known committed, and `lastApplied`, +the highest actually fed to the state machine. Raft's structural +simplification over Paxos: only one node — the **leader** — ever +appends, and entries flow one direction, leader → followers. §5.4.1 +states the consequence outright: "log entries only flow in one +direction, from leaders to followers, and leaders never overwrite +existing entries in their logs." ``` Paxos: any replica can propose → logs converge by proof gymnastics @@ -49,80 +66,212 @@ direction, leader → followers: (entries flow one direction: leader → followers) ``` +The whole basic algorithm needs exactly **two** RPCs — `RequestVote` +(§5.2) and `AppendEntries` (§5.3), both boxed in Figure 2. A third, +`InstallSnapshot`, arrives only with log compaction in §7. + ### Step 2 — terms: a logical clock that fences dead leaders +> **In:** a cluster whose leader can be partitioned away and come +> back. **Out:** the definition of a term, the two comparison rules +> that fence a stale leader, and the price in bytes. + Leaders fail, so leadership must be handed over — and the cluster needs to distinguish the current leader's messages from a stale one's. A **term** is a monotonically increasing integer that acts as a logical clock: time divides into numbered terms, each with at most -one leader. Every message carries the sender's term; every node -tracks the highest it has seen. Two rules do all the fencing: -see a *higher* term → you are stale, become follower and adopt it; -see a *lower* term → the sender is stale, reject. A leader deposed -by a partition can't damage anything after healing: its term is old, -so everyone rejects it. Cost: two integers of state and a comparison -per message — the cheapest fencing token in systems. - -### Step 3 — elections: randomized timeouts, one persisted vote - -Each node is a follower, candidate, or leader (the README's state -diagram). Followers expect periodic heartbeats from the leader; -a follower that hears nothing for an **election timeout** increments -the term, becomes candidate, and asks everyone for votes; a majority -of votes makes it leader. Two details carry the correctness: - -- **One vote per term, persisted.** Each node grants at most one - vote per term, and `voted_for` is fsynced to disk *before* the - vote is sent — a crash+restart must not free the node to vote - twice in the same term, or two leaders could win one term - (question: construct the double-vote scenario if `voted_for` were - volatile). -- **Randomized timeouts** (150–300 ms in the paper) break symmetry. - If all nodes timed out together, votes would split, nobody would - get a majority, and the cycle would repeat — a livelock. - Randomization makes one node usually fire first and win cleanly. - (Question: why randomize per-election rather than assigning fixed - distinct timeouts per node? Hint: what happens after a partition - heals with two live candidates?) - -Elections cost nothing during normal operation; the price of this -design is unavailability for ~1 timeout when a leader dies. - -### Step 4 — log replication: the consistency check +one leader (that is Figure 3's *Election Safety*, §5.2). Every +message carries the sender's term; every node tracks the highest it +has seen in `currentTerm`. Two rules do all the fencing: see a +*higher* term → you are stale, become follower and adopt it; see a +*lower* term → the sender is stale, reject (Figure 2's `RequestVote` +receiver rule 1 and `AppendEntries` receiver rule 1 are both "Reply +false if term < currentTerm"). A leader deposed by a partition can't +damage anything after healing: its term is old, so everyone rejects +it. Cost: two integers of state and a comparison per message — the +cheapest fencing token in systems. + +### Step 3 — what must be on disk before you answer + +> **In:** Figure 2's State box. **Out:** the three fields that must +> be fsynced before an RPC reply, the two that must not bother, and +> the concrete double-vote failure that justifies the split. + +Figure 2 names its state box "Persistent state on all servers" and +parenthesises the obligation: *"Updated on stable storage before +responding to RPCs."* The three fields are: + +| field | why it must be durable | +|---|---| +| `currentTerm` | forgetting it lets you re-enter an old term | +| `votedFor` | forgetting it lets you vote twice in one term | +| `log[]` | forgetting it un-acks entries you told the leader you had | + +And the two that are explicitly volatile — `commitIndex` and +`lastApplied` — need not be, because both are *recomputable*. After +a restart a node relearns its commit index from the next +`AppendEntries` (Figure 2's rule: `commitIndex = min(leaderCommit, +index of last new entry)`), and re-applies from the log. Losing them +costs work, not correctness. This is exactly the distinction raft-rs +encodes in `must_sync()` — see +[reading-raft-rs.md](reading-raft-rs.md) Step 4. + +Construct the double-vote failure to see why `votedFor` is in the +first column and not the second: + +``` + term 5, five nodes S1..S5. S1 and S2 both campaign. + S3 grants its vote to S1, replies, then crashes before the + votedFor write reaches the platter. + S3 restarts with votedFor = null and grants its vote to S2. + + votes for S1: S1, S3, S4 = 3 of 5 → majority → leader(term 5) + votes for S2: S2, S3, S5 = 3 of 5 → majority → leader(term 5) + + Two leaders in term 5. Election Safety (Figure 3, §5.2) is gone, + and with it every argument built on top of it. The double-count is + possible only because ONE node's vote was counted twice; that is + what the fsync prevents. +``` + +On this machine that write is not free. Topic 5 measured the ladder: +a real durable flush on macOS/APFS needs `F_FULLFSYNC`, not +`fsync(2)`, and costs enough to cap commits at **337/s** — which is +why this topic's own `repl_lag` bench sees **341 entries/s** when the +follower fsyncs every entry, and **20,174/s** when it never does. A +59× span, from one line of Figure 2's fine print. + +### Step 4 — elections: randomized timeouts, and the numbers behind them + +> **In:** a follower that has stopped hearing heartbeats. **Out:** +> the election procedure, the up-to-dateness comparison quoted +> exactly, and the paper's own measurements of what randomization +> buys. + +Each node is a follower, candidate, or leader (§5.1; the README's +state diagram). Followers expect periodic heartbeats — Figure 2 +describes these as "AppendEntries RPCs that carry no log entries". +A follower that hears nothing for an **election timeout** increments +`currentTerm`, becomes candidate, votes for itself, and sends +`RequestVote` to everyone; a majority makes it leader. + +Two details carry the correctness. **One vote per term, persisted** — +Step 3. And **the up-to-dateness test**, which §5.4.1 states in two +sentences worth memorising: *"If the logs have last entries with +different terms, then the log with the later term is more up-to-date. +If the logs end with the same term, then whichever log is longer is +more up-to-date."* Term first, length second — and the ordering of +those two clauses is what Step 6's trap turns on. + +**Randomized timeouts** break symmetry. If all nodes timed out +together, votes would split, nobody would reach a majority, and the +cycle would repeat. §5.2 says timeouts are "chosen randomly from a +fixed interval (e.g., 150–300ms)". The paper does not leave that as +folklore; Figure 16 measures it on 5 servers with a broadcast time of +roughly 15 ms, 1000 trials per configuration: + +``` + randomness added result + ---------------- ------------------------------------------- + none leader election consistently took > 10 s + (many split votes) + 5 ms median downtime 287 ms + 50 ms worst case over 1000 trials 513 ms + timeout 12–24 ms 35 ms average, longest trial 152 ms + + Read the first two rows as the whole argument for randomization: + 10,000 ms → 287 ms is a factor of ~35, bought with 5 ms of + jitter. Read rows 3 and 4 as the tradeoff: more randomness + improves the WORST case, a lower timeout improves the AVERAGE. + + The paper still recommends 150–300 ms, ~10× the aggressive + 12–24 ms that measured better, because below that "leaders have + difficulty broadcasting heartbeats before other servers start + new elections." +``` + +### Step 5 — the timing inequality: what "enough" means + +> **In:** the three timescales in a real deployment. **Out:** §5.6's +> inequality, the paper's own bounds for each term, and the reason +> the middle one cannot simply be minimised. + +§5.6 states the whole availability requirement as one inequality: + +``` + broadcastTime ≪ electionTimeout ≪ MTBF + + broadcastTime time to send RPCs to every server in parallel and + receive their responses. §5.6: 0.5–20 ms, + "because Raft's RPCs typically require the + recipient to persist information to stable + storage" ← the fsync is INSIDE broadcastTime + electionTimeout §5.6: likely 10–500 ms + MTBF mean time between failures of a single server; + §5.6: typically several months + + Check it with this repo's own numbers. Topic 15 measured a WAIT-1 + ack p99 of 3889.5 us with the follower fsyncing every entry — + call broadcastTime ~3.9 ms, at the top of the paper's range and + entirely because of the follower's fsync. With the paper's + recommended 150 ms minimum election timeout: + + 150 ms / 3.9 ms ≈ 38x headroom + several months / 150 ms ≈ 10^7 x headroom + + Now drop the follower's fsync to one per 64 entries: ack p99 falls + to 2133.0 us, and the headroom rises to ~70x. The left-hand ≪ is + bought with exactly the durability the right-hand side assumed. +``` + +The inequality is why the election timeout cannot simply be driven to +zero: shrink it toward broadcastTime and leaders start losing +elections they should have won. + +### Step 6 — log replication: the consistency check + +> **In:** a leader with a new client command and a follower whose log +> may diverge. **Out:** the two fields that guard every append, the +> induction they support, and the repair loop's cost. The leader appends a client command to its own log, then sends `AppendEntries` to followers. The heart of Raft is one guard on that message: ``` - AppendEntries carries (prev_log_index, prev_log_term) - follower: my log has an entry at prev_log_index with prev_log_term? + AppendEntries carries (prevLogIndex, prevLogTerm) + follower: my log has an entry at prevLogIndex with prevLogTerm? yes → append (truncating any conflicting suffix) - no → reject; leader decrements next_index and retries + no → reject; leader decrements nextIndex and retries ``` -By induction this gives the **Log Matching Property**: if two logs -have the same (index, term) at one position, they are identical up -to that position — the follower only accepted each entry after -proving the previous one matched. A follower with a divergent -suffix (appended by some dead leader, never committed) gets it -*truncated* and overwritten. The follower side, in full: +By induction this gives Figure 3's **Log Matching Property**: "if two +logs contain an entry with the same index and term, then the logs are +identical in all entries up through the given index" (§5.3) — the +follower only accepted each entry after proving the previous one +matched. A follower with a divergent suffix (appended by some dead +leader, never committed) gets it *truncated* and overwritten. The +follower side, as our stub writes it: ```rust -// the consistency check — Log Matching by induction, one RPC at a time +// ILLUSTRATION — not quoted from the paper; this is the shape of +// Figure 2's AppendEntries receiver rules 1-5 as our experiments/ +// stub implements them. The production version is raft-rs +// src/raft.rs:2499 (handle_append_entries). fn handle_append(&mut self, m: AppendEntries) -> bool { - if m.term < self.term { return false; } // stale leader: fenced + if m.term < self.term { return false; } // Fig 2 rule 1 match self.log.get(m.prev_index) { - None => false, // hole → leader backs up - Some(e) if e.term != m.prev_term => false, // divergent history + None => false, // Fig 2 rule 2: hole + Some(e) if e.term != m.prev_term => false, // Fig 2 rule 2: mismatch _ => { for (i, new) in m.entries.iter().enumerate() { let idx = m.prev_index + 1 + i as u64; if self.log.term_at(idx) != Some(new.term) { - self.log.truncate(idx); // conflicting suffix DIES - self.log.push(new.clone()); // (it was never committed) + self.log.truncate(idx); // Fig 2 rule 3 + self.log.push(new.clone()); // Fig 2 rule 4 } } + // Fig 2 rule 5 self.commit_index = m.leader_commit.min(self.log.last_index()); true } @@ -130,71 +279,139 @@ fn handle_append(&mut self, m: AppendEntries) -> bool { } ``` +Cost accounting: one round trip per batch of entries in the common +case; **O(divergence) round trips** to repair a lagging follower, +because `nextIndex` walks back one entry at a time. §5.3 offers a fix +in its body text — the rejecting follower returns the term of its +conflicting entry and the first index it stores for that term, so the +leader skips a whole term per round trip — and then hedges: "In +practice, we doubt this optimization is necessary." It is not a +footnote and the doubt did not hold; raft-rs implements it +(raft.rs:2539-2554 and raft_log.rs:222-248), and its comment at +raft.rs:1783-1789 says naive probing "can easily result in hours of +time spent probing and can even cause outright outages." + Question: why must a follower *truncate* conflicting entries rather -than skip them? Construct the divergent-log picture from the paper's -Fig 7. Cost accounting: one round trip per batch of entries in the -common case; O(divergence) retries to repair a lagging follower. +than skip them? Construct the divergent-log picture from Figure 7. + +### Step 7 — safety rule 1: the election restriction -### Step 5 — safety rule 1: the election restriction +> **In:** a committed entry and a leader that just died. **Out:** the +> voting rule that protects it, and the two-majority intersection +> argument in full. Committed entries must survive leader changes, so Raft never lets a node that *lacks* a committed entry become leader. A voter refuses -any candidate whose log is less up-to-date than its own — compare -last entry's term first, then log length. The quorum argument does -the rest: a committed entry lives on a majority; a winning candidate -convinced a majority; the two majorities intersect in at least one -node, and that node's vote-check blocked any candidate missing the -entry. So an elected leader already contains every committed entry — -which is why Raft never needs to copy entries *into* a new leader -(contrast VSR, [reading-vsr.md](reading-vsr.md), which chose the -opposite). - -### Step 6 — safety rule 2: only current-term entries count for commit - -The subtle one (§5.4.2): "replicated on a majority" is NOT -sufficient to commit an entry from an *older* term. A leader may -only advance `commit_index` by majority-replicating an entry *from -its own term*; older entries then commit indirectly, riding below -it. Figure 8 is the counterexample that forces the rule — work it by -hand: - -``` - term 2 entry replicated to 2/5 by S1 → S1 crashes - S5 elected (term 3), appends locally, crashes - S1 re-elected (term 4), replicates the OLD term-2 entry to 3/5 - — is it committed? NO. S5 can still win (its term-3 entry - is "newer" by last-term comparison) and truncate it. -``` - -The failure is quorum arithmetic: the election restriction compares -*last terms*, and a majority holding an old-term entry can still -vote for a candidate whose newer-term entry outranks it. Replicating -one current-term entry on a majority closes the hole — now any -future winner provably holds everything below it. Our `raft.rs` test -`stale_leader_uncommitted_overwritten` is exactly this shape. Every -homegrown Raft that skips §5.4.2 loses acked writes here. +any candidate whose log is less up-to-date than its own, by Step 4's +term-then-length test. §5.4.1 gives the argument in one move: "A +candidate must contact a majority of the cluster in order to be +elected, which means that every committed entry must be present in at +least one of those servers." + +Work the arithmetic on five nodes: + +``` + |committed set| ≥ 3 (a majority of 5, by definition of commit) + |voter set| ≥ 3 (a majority of 5, to win the election) + 3 + 3 = 6 > 5 → the two sets share at least 6 − 5 = 1 node + + That node holds the committed entry. If the candidate's log were + missing it, the shared node's log would end at a later term (or the + same term but longer), so it refuses the vote — and without that + vote the candidate cannot reach 3. +``` + +Hence Figure 3's **Leader Completeness** (§5.4): a committed entry is +present in the logs of the leaders of all higher terms. Raft never +needs to copy entries *into* a new leader — contrast VSR +([reading-vsr.md](reading-vsr.md)), which chose the opposite and +transfers a log during the view change. §5.4.1 names the tradeoff: +the alternatives "contain additional mechanisms to identify the +missing entries and transmit them to the new leader... this results +in considerable additional mechanism and complexity." + +### Step 8 — safety rule 2: only current-term entries count for commit + +> **In:** a leader that sees an old entry replicated on a majority. +> **Out:** Figure 8's five panels with the paper's own server names, +> and the exact place the Step 7 argument stops working. + +The subtle one (§5.4.2): "replicated on a majority" is NOT sufficient +to commit an entry from an *older* term. The paper's own summary +sentence: **"Raft never commits log entries from previous terms by +counting replicas."** A leader may only advance `commitIndex` by +majority-replicating an entry *from its own term*; older entries then +commit indirectly, riding below it. + +Figure 8 is the counterexample that forces the rule. The paper's +caption, panel by panel — note that only **term 3** is named for S5; +the term S1 holds in (c) is not stated in the caption, so do not +quote one: + +``` + (a) S1 is leader and partially replicates the log entry at index 2. + (b) S1 crashes. S5 is elected leader for TERM 3 with votes from + S3, S4, and itself, and accepts a different entry at index 2. + (c) S5 crashes. S1 restarts, is elected leader, and continues + replication. The term-2 entry at index 2 is now replicated on + a MAJORITY of the servers — "but it is not committed." + (d) If S1 crashes here, S5 can be elected leader (votes from S2, + S3, and S4) and OVERWRITE index 2 with its own term-3 entry. + (e) But if S1 first replicates an entry from its CURRENT term on a + majority, that entry is committed, S5 cannot win an election, + and "all preceding entries in the log are committed as well." +``` + +Where Step 7's argument breaks: the intersection argument is still +true — S5's voting majority in (d) does share a node with the set +holding the term-2 entry. What fails is the *inference from sharing +to refusal*. Step 4's up-to-dateness test compares last terms first, +and S5's last entry is from term 3 while the shared node's is from +term 2. A node holding the old entry therefore votes for S5 quite +happily. Replicating one current-term entry closes the hole because +it raises the shared node's last term to the leader's own, so no +surviving candidate can outrank it. + +Our `raft.rs` test `stale_leader_uncommitted_overwritten` is exactly +panel (d). Every homegrown Raft that skips §5.4.2 loses acked writes +here. In raft-rs the rule is one boolean at `src/raft_log.rs:526`. ## How to read the paper (with the concepts in hand) +Section and figure numbers are the **extended** version. + | section | what to extract | step | |---|---|---| -| §5.1 | the three states + RPC menu (only 2 RPCs!) | 1, 3 | -| §5.2 | elections: terms, randomized timeouts | 2–3 | -| §5.3 | log replication: the consistency check + repair | 4 | -| §5.4 | safety — read TWICE, especially §5.4.2 | 5–6 | +| §5.1 | the three states + the RPC menu (only 2!) | 1, 4 | +| §5.2 | elections: terms, randomized timeouts, 150–300 ms | 2, 4 | +| §5.3 | log replication: the consistency check, repair, the term-skip optimisation | 6 | +| §5.4 | safety — read TWICE, especially §5.4.2 | 7–8 | +| §5.4.1 | the up-to-dateness definition, quoted in Step 4 | 4, 7 | +| §5.6 | the broadcastTime ≪ electionTimeout ≪ MTBF inequality | 5 | | §6 | membership changes (joint consensus) — skim | — | -| §7 | log compaction / snapshots — skim, topic 5 déjà vu | — | +| §7 | log compaction / snapshots + `InstallSnapshot` — skim, topic 5 déjà vu | — | | Fig 2 | the whole algorithm on one page — print it | all | +| Fig 3 | the five safety properties with their section numbers | 7–8 | +| Fig 7 | the divergence zoo (six follower logs, a–f) | 6 | +| Fig 8 | the five-panel commit trap | 8 | +| Fig 16 | the election-timeout measurements (= ATC '14 Fig 14) | 4 | + +Figure 3's five properties, in the paper's own order and with its own +section attributions, are the checklist to hold every implementation +against: **Election Safety** (§5.2), **Leader Append-Only** (§5.3), +**Log Matching** (§5.3), **Leader Completeness** (§5.4), **State +Machine Safety** (§5.4.3). Note the last one is §5.4.**3**, not §5.4 — +it is the property, distinct from the leader-completeness lemma that +implies it. Fig 2 is the spec that raft-rs implements ([reading-raft-rs.md](reading-raft-rs.md)) — keep it printed next to -you for both chapters. Fig 7 is Step 4's divergence zoo; Fig 8 is -Step 6, and worth an hour. +you for both chapters. ## Questions to answer in notes.md -1. Why persist `(current_term, voted_for, log)` but NOT - `commit_index`? What recomputes commit_index after restart? +1. Why persist `(currentTerm, votedFor, log)` but NOT `commitIndex`? + What recomputes commitIndex after restart? 2. Fig 8 step-by-step: which specific quorum-intersection argument fails without the current-term rule? 3. Why does a leader never overwrite/delete its OWN log entries, and @@ -207,24 +424,186 @@ Step 6, and worth an hour. ## Done when +Answer each before unfolding it. + - [ ] You can explain why agreeing on log order is sufficient for state-machine convergence. + +
Answer + + Because a state machine is deterministic: same start state plus + same commands in the same order gives the same end state. So the + replicas never have to compare or reconcile state — they only have + to agree on the contents of entry #i, for every i. + + That is the reduction the whole paper rests on, and it is why + Figure 2's state box contains a `log[]` and not a snapshot of the + data. `lastApplied` is the pointer that turns the agreed log back + into agreed state. + +
+ - [ ] You can say what a term is and what it fences. -- [ ] You can state both safety rules — the election restriction and the current-term commit rule — and explain the quorum-intersection argument behind Figure 8. -- [ ] You can say exactly which state must be persisted before responding, and why the rest need not be. + +
Answer + + A monotonically increasing integer acting as a logical clock: time + divides into numbered terms, each with at most one leader (Figure + 3, *Election Safety*, §5.2). Every message carries the sender's + term. + + It fences a deposed leader. A leader partitioned away and returning + carries an old term, so every receiver applies Figure 2's rule 1 — + "Reply false if term < currentTerm" — and its writes go nowhere. It + also fences the node itself: seeing a higher term forces it back to + follower and adopts the new term. Two integers of state buys the + entire stale-leader problem. + +
+ +- [ ] You can state exactly which state must be persisted before responding, and why the rest need not be. + +
Answer + + Figure 2, "Persistent state on all servers (Updated on stable + storage before responding to RPCs)": `currentTerm`, `votedFor`, + `log[]`. + + `commitIndex` and `lastApplied` are listed as volatile because both + are recomputable. After a restart the next `AppendEntries` carries + `leaderCommit` and Figure 2's rule 5 rebuilds `commitIndex = + min(leaderCommit, index of last new entry)`; `lastApplied` catches + up by replaying the log. Losing them costs work, not correctness. + + `votedFor` is the one whose loss is unrecoverable: a node that + forgets its vote can grant a second one in the same term, and two + candidates can each reach a majority that counts that node — two + leaders in one term. + +
+ +- [ ] You can state the up-to-dateness comparison in the paper's own order, and both safety rules. + +
Answer + + §5.4.1: if the last entries have different terms, the later term + wins; if the terms are equal, the longer log wins. Term first, + length second. + + Rule 1, the **election restriction** (§5.4.1): a voter refuses a + candidate whose log is less up-to-date than its own. Two majorities + of five intersect in at least 6 − 5 = 1 node; that node holds every + committed entry, so it blocks any candidate missing one. Result: + Figure 3's Leader Completeness. + + Rule 2, **§5.4.2**: "Raft never commits log entries from previous + terms by counting replicas." A leader advances `commitIndex` only by + majority-replicating an entry from its own term; older entries + commit indirectly beneath it. + +
+ +- [ ] You can walk Figure 8's five panels and name the exact inference that fails without the current-term rule. + +
Answer + + (a) S1 partially replicates index 2 (term 2). (b) S1 crashes; S5 is + elected for **term 3** on votes from S3, S4 and itself, and accepts + a different entry at index 2. (c) S5 crashes; S1 restarts, is + re-elected, and continues replication — index 2 now sits on a + majority "but it is not committed". (d) S1 crashes; S5 is elected + on votes from S2, S3, S4 and overwrites index 2. (e) Had S1 first + replicated a current-term entry on a majority, that entry is + committed, S5 cannot win, and everything below it commits too. + + The intersection argument itself survives — S5's majority does + share a node with the majority holding the term-2 entry. What fails + is the step from *sharing* to *refusal*: §5.4.1 compares last terms + first, and S5's term-3 last entry outranks the shared node's term-2 + one, so that node votes for S5. Replicating a current-term entry + raises the shared node's last term to the leader's, restoring the + inference. + +
+ - [ ] You can explain why a leader never overwrites its own entries, and what that means for the follower repair loop. + +
Answer + + Figure 3, *Leader Append-Only* (§5.3): "a leader never overwrites or + deletes entries in its log; it only appends new entries." It is + safe to state as an invariant because of the election restriction — + a new leader already holds every committed entry, so there is + nothing it would need to delete. + + The consequence for repair is that all the truncation happens on + the follower. The leader walks `nextIndex` backwards until the + `(prevLogIndex, prevLogTerm)` check passes, and the follower + truncates its divergent suffix (Figure 2, AppendEntries rule 3). + Cost: O(divergence) round trips, which §5.3's term-skip + optimisation reduces to O(diverging terms) and raft-rs implements + at raft.rs:2539-2554. + +
+ +- [ ] You can state §5.6's timing inequality with the paper's bounds, and say what sits inside broadcastTime. + +
Answer + + `broadcastTime ≪ electionTimeout ≪ MTBF`. §5.6 gives broadcastTime + as 0.5–20 ms, election timeout as likely 10–500 ms, and single-server + MTBF as typically several months. + + The important sentence is why broadcastTime is that large: "Raft's + RPCs typically require the recipient to persist information to + stable storage." The follower's fsync is inside the term. This + topic's own bench shows it: WAIT-1 ack p99 is 3889.5 µs when the + follower fsyncs every entry and 2133.0 µs at one fsync per 64, + which moves the headroom against a 150 ms timeout from ~38× to ~70×. + + Figure 16 measures the middle term directly: no randomness gives + >10 s elections, 5 ms of randomness gives a 287 ms median, 50 ms of + randomness caps the worst case over 1000 trials at 513 ms — and the + paper still recommends 150–300 ms rather than the 12–24 ms that + measured best, to keep the left-hand ≪ comfortable. + +
+ - [ ] You wrote answers to all five questions in notes.md, and can predict what `partition_test` must show: 99 never commits, and is truncated everywhere after the heal. +
Answer + + The test is Figure 8 panel (d) in miniature. The minority-side + leader appends entry 99 and can never reach a majority, so + `commitIndex` never covers it and it is never applied. After the + heal the surviving leader's `AppendEntries` fails its + `(prevLogIndex, prevLogTerm)` check on that node, `nextIndex` walks + back, and rule 3 truncates the suffix. + + The assertion worth writing is the negative one: no replica ever + *applied* 99, so no client could have observed it. A test that only + checks the logs converge would pass even for an implementation that + applied and then un-applied it. + +
+ ## References **Papers** -- Ongaro, Ousterhout — "In Search of an Understandable Consensus - Algorithm" (USENIX ATC 2014) — read the extended version (the tech - report); §5 twice, Fig 2 printed, Fig 8 worked by hand -- Ongaro — "Consensus: Bridging Theory and Practice" (Stanford PhD - dissertation, 2014) — optional; the long-form version with the - membership-change fixes +- Diego Ongaro, John Ousterhout — "In Search of an Understandable + Consensus Algorithm", USENIX ATC 2014. Read the **extended + version** ([raft.github.io/raft.pdf](https://raft.github.io/raft.pdf), + 18 pp.), not the 16-page conference paper: it adds §7 and Figures + 12–13, and renumbers the evaluation figures (extended 14/15/16 = + ATC '14 12/13/14). §5 twice, Fig 2 printed, Fig 8 worked by hand. +- Diego Ongaro — "Consensus: Bridging Theory and Practice", Stanford + PhD dissertation, 2014. **A third, different document** — its + section numbers do not correspond to the paper's. §10.2.1 is what + raft-rs cites for the leader's parallel disk write; §10.2 gives the + cost model (disk 100 µs–10 ms, network RTT 5 µs–400 ms). **Code** - The production implementation is [raft-rs](https://github.com/tikv/raft-rs) — walked in - [reading-raft-rs.md](reading-raft-rs.md) + [reading-raft-rs.md](reading-raft-rs.md). Figure 2's persistent + state is its `HardState`; §5.4.2 is `src/raft_log.rs:526`; §5.3's + term-skip optimisation is `src/raft_log.rs:222-248`. diff --git a/topics/15-replication-consensus/reading-raft-rs.md b/topics/15-replication-consensus/reading-raft-rs.md index a71c942..74d92fd 100644 --- a/topics/15-replication-consensus/reading-raft-rs.md +++ b/topics/15-replication-consensus/reading-raft-rs.md @@ -7,9 +7,15 @@ you a `Ready` bundle of work to do. That inversion is what makes consensus testable — and what our sim-based raft.rs stub imitates. Before the anchors, this chapter builds the design in steps: why I/O-free, the driving contract, the ordering rules that carry -safety, and where the paper's Fig 2 lives in the source. Assumes -[reading-raft-paper.md](reading-raft-paper.md) — terms, the -consistency check, and §5.4.2 are used by name. +safety, the repair loop, and where the paper's Fig 2 lives in the +source. Assumes [reading-raft-paper.md](reading-raft-paper.md) — +terms, the consistency check, and §5.4.2 are used by name. + +Every `file:line` below is **raft-rs at `ad13f3d`**, the revision in +this repo's pin table (`resources/codebases.md`). Check any of them +with `python3 tools/pinned-source.py show raft-rs src/raft.rs -r +939:950`. Numbers from other revisions will not line up; `src/raft.rs` +is 2966 lines at this pin and `src/raw_node.rs` is 840. ## The problem in one sentence @@ -24,9 +30,18 @@ fix is to make the algorithm a pure state machine whose every input ### Step 1 — sans-io: the algorithm as a pure state machine -The sans-io pattern (before the name existed): the library performs -no I/O — it *describes* I/O. raft-rs has no `fsync`, no sockets, no -timers, no threads; you feed it inputs and it returns instructions: +> **In:** the idea that a consensus algorithm is a function of its +> inputs. **Out:** the shape of raft-rs's public surface — two input +> methods, one output bundle — and the name of the thing the library +> deliberately does not own. + +**Sans-io** is the pattern (the phrase postdates raft-rs) of writing +a protocol as a state machine that *describes* I/O rather than +performing it. Nothing in `src/` opens a socket, starts a thread, or +calls `fsync`. `Raft` (raft.rs:263) is generic over a +**`Storage`** trait — a read-only interface the library calls to +*fetch* log entries it has already handed you, never to write them. +Writing is your job, described by the `Ready` bundle: ``` ┌────────────── your code ──────────────┐ @@ -36,8 +51,9 @@ timers, no threads; you feed it inputs and it returns instructions: │ │ │ │ has_ready()? │ │ ▼ │ - │ Ready { messages, entries-to-append, │ - │ committed_entries, hs, ss } │ + │ Ready { messages, persisted_messages │ + │ entries, snapshot, hs, ss, │ + │ committed_entries } │ │ 1. persist entries + hardstate │ │ 2. send messages │ │ 3. apply committed_entries │ @@ -45,88 +61,285 @@ timers, no threads; you feed it inputs and it returns instructions: └───────────────────────────────────────┘ ``` +**HardState** is the paper's Fig-2 "persistent state on all servers" +made concrete: `{ term, vote, commit }` — the three fields that must +survive a crash. **SoftState** is the derived, throwaway pair +`{ leader_id, raft_state }`. The split exists so an embedder knows +exactly which bytes it is obliged to fsync. + Deterministic by construction: the same sequence of tick/step calls always produces the same Ready bundles — which is exactly why our `sim.rs` can test consensus without threads (and why topic 16's DST loves this shape). What it costs: every embedder must implement the driving loop and get its ordering rules right (Step 4) — the library -moved the hard-to-test part out, not away. +moved the hard-to-test part out, not away. Step 1's ordering above is +the *approximate* contract; Step 4 is where it turns out to have a +deliberate exception. ### Step 2 — driving it: tick and step, the only two inputs +> **In:** a running process with a timer and a socket. **Out:** the +> two calls that carry all of it into the library, the real dispatch +> order inside `step`, and the tick-to-milliseconds arithmetic that +> turns "election timeout" into a wall-clock number. + Time and network collapse into two methods. `tick()` — you call it -on your own timer; enough ticks without a heartbeat and the state -machine decides "election timeout" and emits vote requests (in the -next Ready). `step(msg)` — you received a Raft message; hand it -over. Internally, `step` (raft.rs:1346) first handles term logic — -higher term → become_follower, lower term → mostly ignore/reject — -and only THEN dispatches on role -(`step_leader/step_candidate/step_follower`). Compare our `raft.rs` -stub: same shape, `match self.role`. Question: which messages must -be handled *before* the role dispatch, and why? (Term comparison is -role-independent — Fig 2's "all servers" rules.) +on your own timer; the library counts ticks, and enough of them +without a heartbeat makes the state machine decide "election +timeout" and emit vote requests in the next Ready. `step(msg)` — you +received a Raft message; hand it over. + +The library has **four** roles, not the paper's three +(raft.rs:61-71): `Follower`, `Candidate`, `Leader`, and +**`PreCandidate`** — the extra state implements pre-vote, where a +node polls for votes *without* bumping its term, so a partitioned +node cannot return and force a term change. Pre-vote is off by +default (`pre_vote: false`, config.rs:121). + +`step` (raft.rs:1346-1537) does **not** simply do "term logic, then +role dispatch". Read the order: + +```rust +// raft.rs — Raft::step, the dispatch skeleton, 1346-1537 (bodies elided) + 1346 pub fn step(&mut self, m: Message) -> Result<()> { + 1348 // Handle the message term, which may result in our stepping down to a follower. + .... // ... 1348-1478: term comparison, become_follower, pre-vote replies ... + 1483 match m.get_msg_type() { + 1484 MessageType::MsgHup => self.hup(false), + 1485 MessageType::MsgRequestVote | MessageType::MsgRequestPreVote => { + .... // ... 1486-1528: vote decision, log up-to-dateness, reply ... + 1529 } + 1530 _ => match self.state { + 1531 StateRole::Candidate | StateRole::PreCandidate => self.step_candidate(m)?, + 1532 StateRole::Follower => self.step_follower(m)?, + 1533 StateRole::Leader => self.step_leader(m)?, + 1534 }, + 1535 } +``` + +The load-bearing line is **1530**: role dispatch is the `_` arm, the +*last* case. Two message types are handled before it. `MsgHup` +(1484) is the local "your election timer fired" signal — it has no +sender and no term, so role dispatch would have nothing to dispatch +on. `MsgRequestVote` / `MsgRequestPreVote` (1485-1529) are handled +role-independently because the paper's Fig 2 states the voting rule +under "all servers": grant at most one vote per term, and only to a +log at least as up-to-date as yours. Putting that in one place is +what makes Election Safety a property of `step` rather than of three +separate role handlers agreeing. + +Worked arithmetic — turning ticks into milliseconds. The inputs, all +from `src/config.rs`: + +``` + heartbeat_tick = 2 (config.rs:112, 116) + election_tick = 2 × 10 = 20 (config.rs:115) + min_election_tick() = election_tick = 20 (config.rs:147-153) + max_election_tick() = 2 × election_tick = 40 (config.rs:157-163) + + reset_randomized_election_timeout (raft.rs:2854-2866) draws + uniformly from [min, max) = [20, 40) ticks. + + A tick is whatever period YOU call tick() on. At qdrant's + tick_period_ms: 100 (qdrant config/config.yaml:359): + + heartbeat interval = 2 × 100 ms = 200 ms + election timeout ∈ [20, 40) × 100 ms = [2.0 s, 4.0 s) + + At a 10 ms tick the same constants give 20 ms heartbeats and a + 200–400 ms election timeout — the Raft paper's §5.6 recommendation + of 150–300 ms. The constants are unitless; the tick period is the + whole configuration. +``` + +So "raft-rs's default election timeout" is not a duration at all. +Anyone quoting one without naming a tick period has skipped a +multiplication. ### Step 3 — Progress: what the leader knows about each follower -The leader tracks, per follower, two indexes (tracker/progress.rs:8-12): +> **In:** a leader that has appended entry 7 locally and heard back +> from some followers. **Out:** the two per-follower indexes, the +> three-file path from those indexes to a commit index, and the +> arithmetic on a concrete `matched` vector. + +The leader tracks, per follower, two indexes (tracker/progress.rs:10, +12) and a state (progress.rs:22): ``` - matched highest index KNOWN replicated on that follower - next_idx next index to send (optimistic; decremented on reject) + matched highest index KNOWN replicated on that follower (:10) + next_idx next index to send (optimistic; decremented on reject) (:12) + state Probe | Replicate | Snapshot (tracker/state.rs:22-30) ``` -`next_idx` implements the paper's repair loop — send from `next_idx`, -on rejection decrement and retry until the consistency check passes. -`matched` feeds commitment: `maybe_commit` (raft.rs:939) sorts all -matched values descending, takes the majority-th one, and commits it -*only if that entry's term is the current term* — §5.4.2 as three -lines of code: +**Probe** means "I am unsure where this follower's log diverges — +send one entry and wait"; **Replicate** means "I know, pipeline +freely"; **Snapshot** means "the follower is so far behind that the +entries it needs have been compacted away, so send it a snapshot +instead". Each state is an answer to a different repair cost. + +`matched` feeds commitment — and the commit computation is spread +across **three** files, which is the thing to trace rather than +memorise: + +```rust +// raft.rs — Raft::maybe_commit, 939-950, the entry point + 939 pub fn maybe_commit(&mut self) -> bool { + 940 let mci = self.mut_prs().maximal_committed_index().0; + 941 if self.r.raft_log.maybe_commit(mci, self.r.term) { + ... // update own Progress, return true + 947 return true; + 948 } + 949 false + 950 } +``` + +```rust +// quorum/majority.rs — MajorityConfig::committed_index, 70-98 (setup elided) + 94 // Reverse sort. + 95 matched.sort_by(|a, b| b.index.cmp(&a.index)); + 96 + 97 let quorum = crate::majority(matched.len()); + 98 let quorum_index = matched[quorum - 1]; +``` ```rust -// §5.4.2, executable: the majority-replicated index counts only if -// the entry there is from MY term — older entries then ride along -fn maybe_commit(&mut self) -> bool { - let mut matched: Vec = - self.progress.values().map(|p| p.matched).collect(); - matched.sort_unstable_by(|a, b| b.cmp(a)); // descending - let quorum_idx = matched[self.quorum() - 1]; // majority-th highest - if quorum_idx > self.commit_index - && self.log.term_at(quorum_idx) == Some(self.term) - { - self.commit_index = quorum_idx; - return true; // Fig 8 cannot happen - } - false -} +// raft_log.rs — RaftLog::maybe_commit, 524-526, where §5.4.2 actually lives + 524 /// Attempts to commit the index and term and returns whether it did. + 525 pub fn maybe_commit(&mut self, max_index: u64, term: u64) -> bool { + 526 if max_index > self.committed && self.term(max_index).is_ok_and(|t| t == term) { +``` + +Line **526** is §5.4.2 as one boolean: the majority-replicated index +counts only if the entry sitting there is from the *current* term. +Note where it is not — `raft.rs:939` computes the index and delegates +the safety test; the check is in `raft_log.rs`, not in the function +named `maybe_commit` on `Raft`. If you go looking for §5.4.2 in +raft.rs you will not find it. + +Worked example. Five voters, `matched = [7, 5, 5, 3, 2]`, leader +term 4: + ``` + 1. reverse sort (majority.rs:95) → [7, 5, 5, 3, 2] + 2. majority(5) → (5 / 2) + 1 = 3 (util.rs:117-119) + 3. matched[quorum - 1] = matched[2] → 5 (majority.rs:98) + 4. raft_log.maybe_commit(5, 4) → commits iff term(5) == 4 + (raft_log.rs:526) + + Read step 3 as: at least 3 of the 5 have index ≥ 5, because the + vector is sorted descending and position 2 is the third element. + If term(5) == 3, nothing commits — Figure 8 is exactly the case + where committing it would be wrong. +``` + +The library's own doc comment carries a second example to check +yourself against (majority.rs:68, repeated at tracker.rs:282): +`[2,2,2,4,5]` returns **2**. -Worked example (question 2): 5 nodes, matched = [7,5,5,3,2] → -majority-th (3rd) highest = 5 → commit 5, if entry 5 is -current-term. Progress also carries probe/replicate/snapshot states — -question: what problem does each state solve for a lagging follower? +### Step 4 — the Ready contract: ordering is the safety, with one exception -### Step 4 — the Ready contract: ordering is the safety +> **In:** a `Ready` bundle in hand. **Out:** which of its two message +> lists you may send before your fsync completes, why the leader is +> exempt, and the citation that authorises the exemption. `has_ready()` (raw_node.rs:562) polls for pending work; `ready()` -(:487) hands you the bundle; `advance()` (:663) confirms you did it. -The ordering rules between those calls are load-bearing — they are -where the paper's durability requirements become YOUR obligations: - -- **Persist entries + HardState BEFORE sending messages** that - reference them. The HardState holds `(term, voted_for, commit)` — - a vote you didn't fsync can be double-cast after a crash, electing - two leaders in one term (the paper's Step 3 persistence rule, - enforced by discipline alone). -- **Apply committed_entries in order; never apply above what's - persisted.** -- `advance()` tells the library the batch is done; `advance_append` - (:678) lets you ack persistence asynchronously — group-commit the - raft log, topic 5's fsync ladder applied to consensus. - -Question: what specific safety violation occurs if you send the -vote-response message before fsyncing `voted_for`? - -### Step 5 — what our raft.rs keeps / drops +(:487-558) hands you the bundle; `advance()` (:663) confirms you did +it. The obvious rule — *persist everything before sending anything* — +is what most write-ups state, and it is **not** what raft-rs +implements. `Ready` carries **two** message lists, and one line +decides which one your messages land in: + +```rust +// raw_node.rs — the tail of RawNode::ready, 553-556 + 553 // Leader can send messages immediately to make replication concurrently. + 554 // For more details, check raft thesis 10.2.1. + 555 rd.is_persisted_msg = raft.state != StateRole::Leader; + 556 rd.light = self.gen_light_ready(); +``` + +`Ready::messages()` (raw_node.rs:184-190) returns the list **only +when `is_persisted_msg` is false** — i.e. only for a leader. +`Ready::persisted_messages()` (:205-211) returns it only when true — +i.e. for a follower, candidate or pre-candidate — and its doc comment +(:202-203) states the obligation: "outbound messages to be sent AFTER +the HardState, Entries and Snapshot are persisted to stable storage." + +So the real contract is asymmetric: + +| your role | list | may you send before your own fsync? | +|---|---|---| +| Leader | `messages()` | **yes** | +| Follower / Candidate / PreCandidate | `persisted_messages()` | **no** | + +The citation at line 554 is Ongaro's **dissertation** §10.2.1 (a +different document from the ATC '14 paper — see the paper chapter), +"Writing to the leader's disk in parallel", pp. 141-142 with Figure +10.2. The argument: an entry is committed when a *majority* has it on +disk; the leader's own disk is one of those, but not a required one. +Quoting the thesis directly: "The leader may even commit an entry +before it has been written to its own disk, if a majority of +followers have written it to their disks; this is still safe. +LogCabin implements this optimization." A follower gets no such +exemption, because its `PREPARE`-equivalent reply (`MsgAppendResponse`) +is the *evidence* the leader counts — acking bytes you have not +persisted is a lie the leader will act on. Likewise a vote response: +`voted_for` must be on disk before `MsgRequestVoteResponse` leaves, +or a crash-and-restart lets the node vote twice in one term and elect +two leaders. + +There is a second knob for the same tradeoff. `must_sync()` +(raw_node.rs:223-232) is **false** iff (a) no HardState changed, or +only its `commit` field did, **and** (b) there are no entries and no +snapshot — in which case an asynchronous HardState write is +permissible. It is set true at :517 (vote or term changed), :543 +(snapshot present) and :549 (entries present). A `commit`-only +HardState change is not worth an fsync because a lost commit index is +recomputable from the log; a lost `vote` is not. + +`advance()` (:663) is `advance_append` + `advance_apply_to`; +`advance_append` (:678-681) lets you ack persistence separately from +apply, which is how you group-commit the raft log — topic 5's fsync +ladder applied to consensus, and directly the reason topic 15's own +measured table moves from 341 entries/s at one fsync per entry to +12,187 entries/s at one per 64. + +### Step 5 — the repair loop, and the optimisation the paper doubted + +> **In:** a newly elected leader whose log diverges from a follower's +> at index 100 of 1000. **Out:** the round-trip cost of naive +> probing, the paper's fix, and the four line numbers proving +> raft-rs implements it. + +`next_idx` implements the paper's repair loop — send from `next_idx`, +and on rejection decrement and retry until the consistency check +passes. Naively that is **one round trip per diverging entry**: 900 +entries of divergence at a 1 ms RTT is 900 ms of probing, and the +raft-rs authors are blunter than that. Their comment +(raft.rs:1783-1789) says naive probing "can easily result in hours of +time spent probing and can even cause outright outages." + +The fix is in the Raft paper's **§5.3 body text** — a paragraph, not +a footnote, and the paper adds "In practice, we doubt this +optimization is necessary". The rejecting follower returns the *term* +of its conflicting entry plus the first index it stores for that +term; the leader then skips a whole term per round trip instead of +one entry. + +raft-rs implements it. Follower side, `handle_append_entries` +(raft.rs:2539-2554): compute `hint_index = min(m.index, last_index)`, +call `find_conflict_by_term` (raft_log.rs:222-248, which walks the +log down while `term > t`), and set both `reject_hint` and `log_term` +on the response. Leader side: read the hint at raft.rs:1747-1750 and +feed it to `pr.maybe_decr_to` at raft.rs:1799. Four anchors, one +optimisation — so the answer to "does raft-rs implement it" is yes, +and the paper's own doubt did not survive contact with production. + +### Step 6 — what our raft.rs keeps / drops + +> **In:** the library as walked above. **Out:** the subset our +> `experiments/` stub keeps, and the rule for telling a safe +> simplification from a latent bug. The stub in `experiments/` is this library minus everything the sim makes unnecessary: @@ -135,37 +348,62 @@ makes unnecessary: |---|---| | Ready bundle + advance | direct send via `Sim` (no I/O to defer) | | Storage trait + persistence | in-memory `Vec<(term, cmd)>` | -| Progress probe/snapshot states | just `next_idx` decrement | +| Progress probe/replicate/snapshot states | just `next_idx` decrement | +| `find_conflict_by_term` fast backup | absent — O(divergence) probes | | joint-consensus membership | fixed peer set | -| pre-vote, leases, learners | absent | - -Same invariants pinned by tests; ~10× less plumbing. The exercise of -the topic is noticing which drops are safe *because the sim is -deterministic* and which would be real bugs in production. +| pre-vote (the `PreCandidate` role), leases, learners | absent | + +Same invariants pinned by tests; ~10× less plumbing. The test to +apply to each row: *does the sim make this unobservable, or merely +unlikely?* Dropping the Ready bundle is safe because the sim has no +I/O to defer — there is no window in which a message can outrun a +write. Dropping `find_conflict_by_term` is safe because it is a +performance optimisation with no safety content. Dropping pre-vote is +**not** in the same category: it changes which terms get created +under a partition, so a sim that exercises partitions will see +different histories with and without it. ## Where each step lives in the code +All anchors are raft-rs at `ad13f3d`. + | anchor | what it is | step | |---|---|---| +| raft.rs:61-71 | `StateRole` — four variants, incl. `PreCandidate` | 2 | +| raft.rs:263 | `Raft` — the actual state machine | 1 | | raw_node.rs:293 | `RawNode` — the public wrapper | 1 | -| raw_node.rs:487 | `ready()` — collect pending work | 4 | +| raw_node.rs:184-190 / 202-211 | `messages()` vs `persisted_messages()` | 4 | +| raw_node.rs:223-232 | `must_sync()` — when an async HardState write is legal | 4 | +| raw_node.rs:487-558 | `ready()` — collect pending work | 4 | +| raw_node.rs:553-555 | the leader exemption (`is_persisted_msg`) | 4 | | raw_node.rs:562 | `has_ready()` — the poll predicate | 4 | | raw_node.rs:663 | `advance()` — "I did the work" | 4 | -| raw_node.rs:678 | `advance_append` — split persistence ack | 4 | -| raft.rs:263 | `Raft` — the actual state machine | 1 | -| raft.rs:939 | `maybe_commit` — §5.4.2 lives here | 3 | +| raw_node.rs:678-681 | `advance_append` — split persistence ack | 4 | +| raft.rs:939-950 | `maybe_commit` — computes the index, delegates the test | 3 | +| tracker.rs:284-288 | `maximal_committed_index` | 3 | +| quorum/majority.rs:95/98 | reverse sort, then `matched[quorum-1]` | 3 | +| util.rs:117-119 | `majority(total) = (total / 2) + 1` | 3 | +| raft_log.rs:526 | the §5.4.2 current-term test | 3 | | raft.rs:1148/1176/1226 | `become_follower/candidate/leader` | 2 | | raft.rs:1283 | `campaign` | 2 | -| raft.rs:1346 | `step` — the message dispatch root | 2 | +| raft.rs:1346-1537 | `step` — term logic, MsgHup, votes, then roles | 2 | +| raft.rs:1530-1534 | the role dispatch, as the `_` arm | 2 | | raft.rs:1539 | `hup` — election timeout fires | 2 | +| raft.rs:1747-1750 / 1799 | leader consumes the backup hint | 5 | +| raft.rs:1783-1789 | "hours of time spent probing" | 5 | | raft.rs:2045/2291/2348 | `step_leader/candidate/follower` | 2 | -| raft.rs:2499 | `handle_append_entries` | 3 | -| tracker/progress.rs:8-12 | `Progress { matched, next_idx }` | 3 | +| raft.rs:2539-2554 | follower builds the backup hint | 5 | +| raft.rs:2854-2866 | `reset_randomized_election_timeout` | 2 | +| raft_log.rs:222-248 | `find_conflict_by_term` | 5 | +| tracker/progress.rs:10/12/22 | `Progress { matched, next_idx, state }` | 3 | +| tracker/state.rs:22-30 | `ProgressState { Probe, Replicate, Snapshot }` | 3 | +| config.rs:112-116, 147-163 | tick defaults and the election-tick range | 2 | Read order: `raw_node.rs` around `ready()`/`advance()` first (the -contract), then `raft.rs:1346` `step` and follow one message type -down each role branch, then `maybe_commit`. qdrant's production -driving loop for this exact API is the next chapter +contract, including line 555), then `raft.rs:1346` `step` and follow +one message type down each role branch, then the three-file +`maybe_commit` chain. qdrant's production driving loop for this exact +API is the next chapter ([reading-qdrant-consensus.md](reading-qdrant-consensus.md)). ## Questions for notes.md @@ -175,8 +413,8 @@ driving loop for this exact API is the next chapter 2. `maybe_commit`: write out the sorted-matched-index computation for 5 nodes with matched = [7,5,5,3,2]. Commit index? 3. next_idx decrement-and-retry is O(divergence) round trips — what - optimization does the paper's §5.3 footnote suggest, and does - raft-rs implement it? + optimization does the paper's §5.3 suggest, and does raft-rs + implement it? 4. advance_append: how does splitting the persistence ack enable pipelining, and what must you still NOT reorder? 5. Map Ready → M15 stage 2: which parts of your WAL commit path @@ -184,22 +422,189 @@ driving loop for this exact API is the next chapter ## Done when +Answer each before unfolding it. + - [ ] You can explain what sans-io buys and why raft-rs contains no fsync, no sockets and no threads. -- [ ] You can write out the `maybe_commit` sorted-matched-index computation from memory. -- [ ] You can state the Ready contract's ordering rules and say which reorderings are safety violations rather than performance bugs. + +
Answer + + It makes the algorithm a deterministic function of its inputs. + Every input that would normally be ambient — the clock, the + network, the disk — becomes an argument: time arrives via `tick()`, + messages via `step(msg)`, and all output is a `Ready` value rather + than a syscall. `Raft` (raft.rs:263) is generic over a + trait that only *reads*. + + The payoff is that a bug which needs a specific interleaving of + timeout, crash and reorder can be reproduced by replaying a + sequence of `tick`/`step` calls, with no threads and no wall clock + involved. Our `sim.rs` is the same trick at a smaller scale. + + The cost is that the untestable part did not vanish, it moved: every + embedder writes its own driving loop and must get Step 4's ordering + right. qdrant's is `Consensus::start` (qdrant `src/consensus.rs:481`) + and it is roughly 500 lines. + +
+ +- [ ] You can write out the `maybe_commit` sorted-matched-index computation from memory, and name the file each of its three stages lives in. + +
Answer + + `Raft::maybe_commit` (raft.rs:939-950) asks the tracker for a + candidate index, then hands it to the log. `ProgressTracker:: + maximal_committed_index` (tracker.rs:284-288) forwards to + `MajorityConfig::committed_index` (quorum/majority.rs:70-124), which + reverse-sorts the matched vector at :95 and takes `matched[quorum-1]` + at :98, with `majority(total) = (total / 2) + 1` (util.rs:117-119). + + For `[7,5,5,3,2]`: sorted descending it is unchanged, `majority(5)` + is 3, so `matched[2]` = **5** — at least three of five have index + ≥ 5. + + The §5.4.2 test is *not* in either of those. It is + `RaftLog::maybe_commit` (raft_log.rs:526): commit only if + `max_index > self.committed` **and** `term(max_index) == term`. With + a leader in term 4 and entry 5 from term 3, nothing commits. + +
+ +- [ ] You can state the Ready contract's ordering rules, and say precisely who is allowed to send before persisting and on whose authority. + +
Answer + + Persist entries, snapshot and HardState; send messages; apply + committed entries in order and never above what is persisted; then + `advance()`. The exception is the ordering of the first two, and it + depends on your role. `RawNode::ready` sets `rd.is_persisted_msg = + raft.state != StateRole::Leader` (raw_node.rs:555), which routes a + leader's outbound messages into `Ready::messages()` (:184-190, + sendable immediately) and everyone else's into + `Ready::persisted_messages()` (:205-211, whose doc comment at + :202-203 requires the write first). + + The authority is cited in the code at raw_node.rs:554: Ongaro's + dissertation §10.2.1, pp. 141-142 — "The leader may even commit an + entry before it has been written to its own disk, if a majority of + followers have written it to their disks; this is still safe." + + A follower has no such licence, because its `MsgAppendResponse` is + the evidence the leader counts toward the majority. Same for a vote: + send `MsgRequestVoteResponse` before `voted_for` is durable and a + crash-restart lets the node vote twice in one term. + +
+ +- [ ] You can say when an asynchronous HardState write is legal, and why the exception is safe. + +
Answer + + `must_sync()` (raw_node.rs:223-232) is false iff no HardState field + other than `commit` changed **and** there are no entries and no + snapshot in the bundle. It is forced true at :517 when `vote` or + `term` changed, :543 for a snapshot, :549 when entries are present. + + It is safe because `commit` is recoverable and `vote`/`term` are + not. After a crash a node re-derives its commit index from the log + and from the leader's next AppendEntries — losing it costs a little + re-apply work. Losing `vote` costs Election Safety. + +
+ - [ ] You can explain how splitting the persistence acknowledgement (`advance_append`) enables pipelining without breaking the contract. -- [ ] You can say what the `next_idx` decrement-and-retry loop costs in round trips, and what optimization fixes it. + +
Answer + + `advance()` (raw_node.rs:663) is `advance_append` plus + `advance_apply_to`. Calling `advance_append` (:678-681) on its own + says "the append is durable" without also claiming "the committed + entries are applied", so the two can proceed at different rates: you + can batch many Ready-worth of entries into one fsync and ack them + together, while apply runs behind. + + What must not be reordered is the pairing itself. You may not + `advance_append` before the write actually returns, and a follower + may not release its `persisted_messages()` on the strength of a + queued write. The gain is exactly topic 15's measured ladder: one + fsync per entry is 341 entries/s, one per 64 is 12,187. + +
+ +- [ ] You can say what the `next_idx` decrement-and-retry loop costs in round trips, what fixes it, and whether raft-rs bothered. + +
Answer + + Naively one round trip per diverging entry — the raft-rs comment at + raft.rs:1783-1789 says this "can easily result in hours of time + spent probing and can even cause outright outages." + + The Raft paper's §5.3 body text (not a footnote) describes the fix: + the follower returns the term of its conflicting entry and the first + index it holds for that term, so the leader skips a whole term per + round trip. The paper then says "we doubt this optimization is + necessary." + + raft-rs implements it anyway. Follower side raft.rs:2539-2554 + (`hint_index`, `find_conflict_by_term`, `reject_hint` + `log_term`), + with the walk itself at raft_log.rs:222-248; leader side + raft.rs:1747-1750 reading the hint and :1799 calling + `pr.maybe_decr_to`. + +
+ +- [ ] You can convert raft-rs's tick constants into a wall-clock election timeout for a given tick period. + +
Answer + + The constants are unitless. `heartbeat_tick` is 2 and + `election_tick` is `2 × 10 = 20` (config.rs:112-116); + `min_election_tick()` returns `election_tick` and + `max_election_tick()` returns `2 × election_tick` + (config.rs:147-163); `reset_randomized_election_timeout` + (raft.rs:2854-2866) draws uniformly from `[20, 40)`. + + Multiply by your tick period. qdrant ticks every 100 ms + (`config/config.yaml:359`), giving 200 ms heartbeats and a + 2.0–4.0 s election timeout. A 10 ms tick gives 20 ms heartbeats and + 200–400 ms, which lands on the paper's §5.6 recommendation of + 150–300 ms. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the Ready-to-M15 mapping. +
Answer + + The mapping worth writing down: your WAL's `append` is + `Ready::entries`; your fsync is the persist step; your ack to the + follower is `persisted_messages()` and your ack to the leader's + peers is `messages()`; your apply loop is `committed_entries`; and + your "the batch is durable" callback is `advance_append`. + + The part with no analogue yet is `HardState`. M15 stage 2 needs a + durable `{term, vote, commit}` beside the log, written under the + same rules as raw_node.rs:223-232 — synchronously when `vote` or + `term` moves, lazily when only `commit` does. + +
+ ## References **Papers** - The Raft paper itself is [reading-raft-paper.md](reading-raft-paper.md) — Fig 2 is the spec - this code implements + this code implements, §5.3 is Step 5's optimisation, §5.4.2 is + raft_log.rs:526 +- Diego Ongaro, *Consensus: Bridging Theory and Practice*, Stanford + PhD dissertation, 2014 — **a different document from the paper**. + §10.2.1 "Writing to the leader's disk in parallel" (pp. 141-142, + Figure 10.2) is what raw_node.rs:554 cites. **Code** -- [raft-rs](https://github.com/tikv/raft-rs) — `src/raw_node.rs` (the - Ready contract), `src/raft.rs` (the state machine; the anchor map - above), `src/tracker/progress.rs`; qdrant's embedding of it is +- [raft-rs](https://github.com/tikv/raft-rs) at `ad13f3d` — + `src/raw_node.rs` (the Ready contract), `src/raft.rs` (the state + machine), `src/raft_log.rs` (the §5.4.2 test and + `find_conflict_by_term`), `src/quorum/majority.rs`, + `src/tracker/progress.rs`, `src/config.rs`; the anchor map above +- qdrant's embedding of it is [reading-qdrant-consensus.md](reading-qdrant-consensus.md) diff --git a/topics/15-replication-consensus/reading-valkey-replication.md b/topics/15-replication-consensus/reading-valkey-replication.md index ce47514..3e4623d 100644 --- a/topics/15-replication-consensus/reading-valkey-replication.md +++ b/topics/15-replication-consensus/reading-valkey-replication.md @@ -4,10 +4,18 @@ The canonical async leader/follower design: ack the client immediately, ship the command stream best-effort, survive disconnects with a backlog. Everything Raft pays for, valkey skips — and this chapter builds each skip as its own concept: the zero-RTT ack, the -command stream, the shared buffer, resumable sync, the opt-in -semi-sync escape hatch, and the failover dance that consensus would -have made unnecessary. Then it hands you the anchor map into -`replication.c` (~5600 lines, sliced, never read linearly). +command stream, the shared buffer, resumable sync, the full-sync +fork, the opt-in semi-sync escape hatch, and the failover dance that +consensus would have made unnecessary. Then it hands you the anchor +map into `replication.c`, sliced, never read linearly. + +Every `file:line` below is **valkey at `8891441ab`**, the revision in +this repo's pin table (`resources/codebases.md`). Check any of them +with `python3 tools/pinned-source.py show valkey src/replication.c -r +449:552`. At this pin `src/replication.c` is **5726** lines, +`src/server.c` is 7937 and `src/t_set.c` is 1659. Several config +names and defaults changed between Redis 6 and Valkey — the ones +below are read out of `src/config.c` at this pin, not from any blog. ## The problem in one sentence @@ -21,9 +29,13 @@ resumable, and (only if you ask) bounded. ### Step 1 — async leader/follower: the ack races the stream -Asynchronous replication means the primary executes a write, replies -to the client, and *then* ships the write to replicas — the ack does -not wait for anyone: +> **In:** a client write arriving at a primary with two replicas. +> **Out:** the ordering of ack against replication, the name for the +> gap it creates, and the price list that ordering buys. + +**Asynchronous replication** means the primary executes a write, +replies to the client, and *then* ships the write to replicas — the +ack does not wait for anyone: ``` client write → primary executes → ack client ← ZERO repl RTT @@ -38,106 +50,300 @@ not wait for anyone: Contrast Raft (previous chapter): majority ack BEFORE commit, one round trip plus an fsync on every write. Valkey's price list is the inverse: write latency is a pure single-node number, replicas are -always some bytes behind (**replication lag** — the repl_lag -experiment measures its floor), and a failover to a lagging replica -silently discards the tail of acked writes. Everything below is the -machinery that manages — never eliminates — that loss window. +always some bytes behind — **replication lag**, the byte distance +between the primary's stream offset and a replica's acked offset — +and a failover to a lagging replica silently discards the tail of +acked writes. + +This topic's `repl_lag` bench prices the other side of that trade. +With WAIT-1 semantics (Step 6) and the follower fsyncing every entry, +throughput is **341 entries/s** and ack p99 is 3889.5 µs; with the +follower never fsyncing, **20,174 entries/s** and p99 64.5 µs. Async +replication is the configuration that does not pay either number on +the client's critical path — it moves the whole ladder off the write +path and into the loss window. Everything below is the machinery that +manages, never eliminates, that window. ### Step 2 — the stream is commands, not pages +> **In:** a `SPOP myset` executed on the primary. **Out:** what +> actually enters the replication stream, the two-layer machinery +> that puts it there, and the WAL analogy. + What flows to replicas is the *command stream* itself -(statement-based replication): the same RESP commands clients sent, -re-executed by each replica. Nondeterministic commands would diverge -replicas — `SPOP` pops a *random* member, so two replicas executing -it disagree forever. `propagateNow` (server.c:3609) is the fix: -rewrite nondeterminism before it enters the stream (SPOP → SREM of -the specific member the primary chose). This is topic 5's -logical-vs-physical WAL choice, made at the replication layer: -statements are compact and human-readable; physical WAL frames +(**statement-based replication**): RESP commands, re-executed by each +replica. Nondeterministic commands would diverge replicas — `SPOP` +pops a *random* member, so two replicas executing it disagree +forever. + +The fix is **per command, at the command's own site**, not in a +central rewriter. `spopCommand` calls `setTypePopRandom` to choose +the member and immediately rewrites itself: + +```c +// t_set.c — spopCommand, 969-975: choose, then rewrite + 969 /* Pop a random element from the set */ + 970 ele = setTypePopRandom(set); + ... + 974 /* Replicate/AOF this command as an SREM operation */ + 975 rewriteClientCommandVector(c, 3, shared.srem, c->argv[1], ele); +``` + +Line **975** is the whole idea: the command the replica sees is +`SREM myset `, which is +deterministic. The multi-element form is messier and worth reading — +`spopWithCountCommand` rewrites to `DEL`/`UNLINK` when it empties the +set (t_set.c:790-791), otherwise emits a batch of `SREM`s through +`alsoPropagate` (t_set.c:922, 937) and then calls +`preventCommandPropagation(c)` (t_set.c:949) so the original `SPOP` +never reaches the stream. + +Two layers sit below that. `alsoPropagate` (server.c:3663) queues +extra commands; `propagatePendingCommands` (server.c:3729) drains the +queue and wraps a multi-command batch in `MULTI`/`EXEC` (server.c:3751, +3762) so replicas apply it atomically. `propagateNow` +(server.c:3609-3650) is the low-level dispatcher at the bottom — it +does **not** rewrite anything; it fans one already-final command out +to `feedAppendOnlyFile` (:3647), `replicationFeedReplicas` (:3648) +and `clusterFeedSlotExportJobs` (:3649). If you go looking for the +SPOP rewrite in `propagateNow` you will not find it. + +This is topic 5's logical-vs-physical WAL choice, made at the +replication layer: statements are compact and human-readable but need +a determinism audit for every command ever added; physical WAL frames (what M15 stage 1 ships) are dumb but deterministic by construction (question 1). ### Step 3 — one buffer, many cursors -N replicas must each receive the stream, but N private copies of -every write would multiply memory by N. Pre-6.2 valkey did exactly -that — each replica had its own output buffer. Now -(`feedReplicationBufferWithObject`, :352-366; append + wake at :449) -there is ONE shared list of buffer blocks; each replica holds just a +> **In:** N replicas that each need every byte of the stream. +> **Out:** the data structure that avoids N copies, the three lines +> that hand a replica its cursor, and what a stuck replica costs. + +N private copies of every write would multiply memory by N. Pre-6.2 +valkey did exactly that — each replica had its own output buffer. Now +there is ONE shared list of buffer blocks; each replica holds a *cursor* (block + offset) into it, and so does the backlog (Step 4). -A slow replica now costs O(1) bookkeeping instead of O(stream) bytes -— same shape as topic 7's client output buffers (question: what -else do the two share?). Blocks are freed once every cursor has -passed them; one stuck replica can still pin the list, which is what -replica output buffer limits are for. -### Step 4 — PSYNC: resumable replication via (replid, offset) +`feedReplicationBuffer` (replication.c:449-552) is the hot path; +`feedReplicationBufferWithObject` (:354-367) is the thin wrapper for +`robj` inputs. The cursor handout is the load-bearing part: + +```c +// replication.c — feedReplicationBuffer, 518-537 (loop body elided) + 518 while ((ln = listNext(&li))) { + 519 client *replica = ln->value; + ... + 521 /* Update shared replication buffer start position. */ + 522 if (replica->repl_data->ref_repl_buf_node == NULL) { + 523 replica->repl_data->ref_repl_buf_node = start_node; + 524 replica->repl_data->ref_block_pos = start_pos; + 525 /* Only increase the start block reference count. */ + 526 ((replBufBlock *)listNodeValue(start_node))->refcount++; + 527 } + 528 + 529 /* Check output buffer limit only when add new block. */ + 530 if (add_new_block) closeClientOnOutputBufferLimitReached(replica, 1); + ... + 533 /* For replication backlog */ + 534 if (server.repl_backlog->ref_repl_buf_node == NULL) { + 535 server.repl_backlog->ref_repl_buf_node = start_node; + 536 /* Only increase the start block reference count. */ + 537 ((replBufBlock *)listNodeValue(start_node))->refcount++; +``` + +Lines 522-527 and 534-537 are the same three moves twice: a replica +and the backlog are *the same kind of reader*. Blocks are freed once +every refcount drops; one stuck replica pins the list, which is what +line **530**'s output-buffer-limit kill exists to bound. + +Block sizing is worth the arithmetic (replication.c:486-487): + +``` + limit = max(repl_backlog_size / 16, PROTO_REPLY_CHUNK_BYTES) + size = min(max(len, PROTO_REPLY_CHUNK_BYTES), limit) + + With the default repl-backlog-size = 10 MB (config.c:3453) and + PROTO_REPLY_CHUNK_BYTES = 16 KB: -Disconnects are routine, and restarting replication from scratch -(full snapshot) on every blip would be unusable. So the stream is -addressable: every byte has an **offset**, the primary's history has -an id (**replid**), and the backlog (created at :137) keeps the last -N MB of stream in a ring. A reconnecting replica says -`PSYNC ` and the primary -(`primaryTryPartialResynchronization`, :854) decides: + limit = max(10 MB / 16, 16 KB) = max(640 KB, 16 KB) = 640 KB + a 100-byte write → size = max(100, 16 KB) = 16 KB, capped + at 640 KB → 16 KB block + a 2 MB write → size = max(2 MB, 16 KB) = 2 MB, capped + at 640 KB → 640 KB block + So small writes are batched into 16 KB blocks (one refcount per + 16 KB of stream, not per write) and a huge write is chopped so no + single block can pin more than 1/16 of the backlog budget. ``` - replid matches (or matches replid2 within second_replid_offset) - AND offset still inside the backlog ring - → +CONTINUE: replay backlog from offset (cheap) - else - → +FULLRESYNC: fork, RDB snapshot, then stream (expensive) + +The append does not itself wake anyone. `prepareReplicasToWrite()` +(replication.c:336) is the wake, and it is called from +`replicationFeedReplicas` at :589 — *before* `feedReplicationBuffer` +at :590. Same shape as topic 7's client output buffers (question: +what else do the two share?). + +### Step 4 — PSYNC: resumable replication via (replid, offset) + +> **In:** a replica reconnecting after a 30-second network blip. +> **Out:** the two-part identity it presents, the exact inequality +> that decides cheap-vs-expensive, and what the check cannot detect. + +Disconnects are routine, and a full snapshot on every blip would be +unusable. So the stream is addressable: every byte has an **offset**, +the primary's history has an id (**replid**, a 40-char hex run id), +and the **backlog** — created in `createReplicationBacklog` +(replication.c:135-146) — keeps the last N bytes of stream as a ring +view over the shared blocks of Step 3. A reconnecting replica sends +`PSYNC ` and `primaryTryPartialResynchronization` +(:854) decides. + +Two tests, in order. The identity test (:866-867): the replid must +match `server.replid`, or match `server.replid2` **and** have +`psync_offset <= server.second_replid_offset`. Then the range test — +this is the inequality to memorise: + +```c +// replication.c — primaryTryPartialResynchronization, 889-891 + 889 /* We still have the data our replica is asking for? */ + 890 if (!server.repl_backlog || psync_offset < server.repl_backlog->offset || + 891 psync_offset > (server.repl_backlog->offset + server.repl_backlog->histlen)) { +``` + +Read line 890-891 as its negation, the success condition: + ``` + backlog->offset ≤ psync_offset ≤ backlog->offset + backlog->histlen + + i.e. the requested byte is still inside the ring. Turn it into a + sizing rule with the defaults (config.c:3453, 3477): + + repl-backlog-size = 10 MB repl-backlog-ttl = 3600 s -```rust -// PSYNC: (replid, offset) is (term, index) with the safety stripped — -// a matching offset is ASSUMED to mean matching history, never checked -fn try_partial_resync(&self, replid: &str, offset: u64) -> Sync { - let id_ok = replid == self.replid - || (replid == self.replid2 && offset <= self.second_replid_offset); - if id_ok && self.backlog.contains(offset) { - Sync::Continue(self.backlog.since(offset)) // replay the ring: cheap - } else { - Sync::Full(self.fork_rdb_snapshot()) // fork + RDB + stream - } -} + partial resync succeeds iff write_rate × disconnect_seconds + ≤ repl-backlog-size + + At 5 MB/s of replication stream: + 10 MB / 5 MB/s = 2 seconds of tolerable disconnect. + At 100 KB/s: + 10 MB / 0.1 MB/s = 100 seconds. + + A 30-second blip at 5 MB/s needs 150 MB of backlog to stay cheap. + The default survives it only if your stream is under 341 KB/s. ``` +On success the primary writes `+CONTINUE` (:935/937); on failure +`+FULLRESYNC %s %lld` (:840) and Step 5's fork. + `replid2` is the failover trick: a promoted replica keeps its old -primary's replid as replid2, so *siblings* of the old primary can -still partial-resync from the new one. The Raft comparison is exact -and damning: (replid, offset) is (term, index) with the safety -stripped — Raft's consistency check *verifies* that prev_index holds -prev_term before appending; PSYNC just assumes a matching offset -means matching history (question 2: what divergence can it not -detect?). - -### Step 5 — full sync and the replica handshake - -When partial resync is refused, the primary forks (`syncCommand`, -:1077): the child serializes an RDB snapshot at a frozen -point-in-time (copy-on-write does the freezing — topic 5), while the -parent accumulates new writes in the replication buffer to stream -after the snapshot. The replica side is a textbook nonblocking state -machine driven by the event loop (topic 7), one state per handshake -stage (:3731+): - -``` - REPL_STATE_CONNECT → CONNECTING → RECEIVE_PING_REPLY → ... +primary's replid as replid2 with `second_replid_offset` marking where +its own history diverged, so *siblings* of the old primary can still +partial-resync from the new one — but only for offsets at or below +that mark, which is what the `<=` at :867 enforces. + +The Raft comparison is exact and damning: `(replid, offset)` is +`(term, index)` with the safety stripped. Raft's consistency check +*verifies* that `prevLogIndex` holds `prevLogTerm` before appending; +PSYNC checks only that the replid matches and the offset is in range — +it never compares the *content* at that offset (question 2: what +divergence can it not detect?). + +### Step 5 — full sync: two forks, and a config default that flipped + +> **In:** a `+FULLRESYNC` decision. **Out:** the fork, the two +> transports and which one is now the default, and the exact moment +> the replica's dataset disappears. + +When partial resync is refused, `syncCommand` (:1077) leads to +`startBgsaveForReplication` (:988). It picks a transport at +:1002-1004: + +``` + socket_target = (mincapa & REPLICA_CAPA_EOF) + && (server.repl_diskless_sync + || filtered RDB + || rdbver != RDB_VERSION) + + true → rdbSaveToReplicasSockets() (:1018) — diskless + false → rdbSaveBackground() (:1021) — via a file +``` + +**Both paths fork.** Diskless does not mean fork-less; it means the +child writes the RDB straight into the replica sockets instead of to +a file first. The child serialises a frozen point-in-time snapshot — +copy-on-write does the freezing (topic 5) — while the parent +accumulates new writes in the Step 3 buffer to stream afterwards. + +Config defaults at this pin, straight out of `src/config.c` — the +first one is the Redis-6 trap: + +| config | default | line | +|---|---|---| +| `repl-diskless-sync` | **enabled (1)** | config.c:3274 | +| `repl-diskless-sync-delay` | 5 s | config.c:3393 | +| `repl-diskless-sync-max-replicas` | 0 (no limit) | config.c:3417 | +| `repl-diskless-load` | **disabled** | config.c:3352 | +| `dual-channel-replication-enabled` | no (0) | config.c:3275 | +| `repl-backlog-size` | 10 MB | config.c:3453 | +| `repl-backlog-ttl` | 3600 s | config.c:3477 | + +Diskless *sync* is on by default here; diskless *load* is not. So the +primary streams the RDB without touching its own disk, and the +replica still writes it to a file before loading. + +The replica side is a nonblocking state machine driven by the event +loop (topic 7). The states are `server.h:389-407` — thirteen of them, +with the handshake sub-range explicitly bracketed by comments at +:393 and :404 — and the driver is `syncWithPrimary` +(replication.c:4077-4197), which carries an ASCII state diagram in +its header comment. (`replication.c:3726` is a *different*, dual-channel +variant, `dualChannelSetupMainConnForPsync`; do not read it as the +main path.) + +``` + REPL_STATE_CONNECT → CONNECTING → RECEIVE_PING_REPLY → SEND_HANDSHAKE + → RECEIVE_AUTH_REPLY → RECEIVE_PORT_REPLY → RECEIVE_IP_REPLY + → RECEIVE_CAPA_REPLY → RECEIVE_VERSION_REPLY + → [RECEIVE_NODEID_REPLY, cluster only] → SEND_PSYNC → RECEIVE_PSYNC_REPLY → TRANSFER → CONNECTED ``` -Note the brutal step: on full sync the replica flushes its ENTIRE -dataset before loading the RDB. Cost of a too-small backlog, made -visible: one disconnect longer than the ring → fork + full RDB + -full reload (question 2's inequality). +The brutal step: on full sync the replica flushes its ENTIRE dataset. +Precisely — `emptyData()` runs at `rdb.c:3169-3173`, *after* the RDB +magic and version check passed at :3160-3167, which return +`RDB_INCOMPATIBLE` without clearing anything. So an incompatible RDB +leaves the old data intact; a compatible one wipes it before the +first key is loaded. During the wipe the replica keeps the link alive +by sending bare newlines (`replicationEmptyDbCallback`, +replication.c:2122-2128). Cost of a too-small backlog, made visible: +one disconnect longer than Step 4's inequality → fork + full RDB + +full reload, with a window where the replica holds nothing at all. ### Step 6 — WAIT: semi-sync as an opt-in, after the fact -`WAIT numreplicas timeout` (:4996) is the bounded-loss escape hatch: -block *the client* until n replicas have acked the primary's current -offset (acks arrive via `REPLCONF ACK`, requested at :4947). The -asymmetry vs consensus is the whole lesson: +> **In:** a client that has already received `+OK` for its write. +> **Out:** what WAIT counts, what it provably does not promise, and +> the sibling command that counts something stronger. + +`WAIT numreplicas timeout` (:4996-5026) is the bounded-loss escape +hatch: block *the client* until n replicas have acked the primary's +current offset. It tries a non-blocking count first (:5013-5017) and +only then blocks via `blockClientForReplicaAck` (:5021). The offset +it waits for is `getClientWriteOffset` (:4953), i.e. `c->woff` — the +stream position after that client's own last write. + +Two mechanisms underneath. `replicationRequestAckFromReplicas` +(:4947-4949) does **not** send anything; it sets +`server.get_ack_from_replicas = 1`, and the comment at :4943-4946 +explains why — the actual `REPLCONF GETACK` broadcast is grouped in +`beforeSleep()`, so many waiting clients cost one broadcast. And the +counting rule is `replicationCountAcksByOffset` (:4962-4975): a +replica counts if `repl_state == REPLICA_STATE_ONLINE` **and** +`repl_ack_off >= offset`. + +That second condition is the whole lesson. `repl_ack_off` is how many +bytes the replica has *received and processed* — not how many it has +fsynced. ``` WAIT: execute → ack replicas → unblock client (write ALREADY applied) @@ -145,49 +351,99 @@ asymmetry vs consensus is the whole lesson: ``` WAIT cannot un-apply anything — it only *informs* the client how far -replication got. WAIT returning 1 of 2 means "one replica has it"; -it does not mean the surviving topology after a failover contains -that replica (question: can the write still be lost? — yes, walk -it). Raft's commit is a promise about the future; WAIT is a report -about the present. +replication got. `WAIT 1 0` returning 1 means "one replica has these +bytes in memory". It does not mean the bytes are on that replica's +disk, and it does not mean the surviving topology after a failover +contains that replica (question: can the write still be lost? — yes, +walk it). Raft's commit is a promise about the future; WAIT is a +report about the present. + +Valkey has the stronger sibling: `WAITAOF` (:5030) counts through +`replicationCountAOFAcksByOffset` (:4979) against `repl_aof_off` — +bytes the replica has fsynced to its AOF. That is the command whose +cost this topic's table actually measures: the 341-vs-20,174 +entries/s span is the difference between counting fsynced bytes and +counting received bytes. ### Step 7 — failover: the coordination consensus would have given free -`FAILOVER` (:5565) hand-coordinates what Raft's election does -automatically: pause writes → wait for the target replica to catch -up to the primary's offset → send it `PSYNC FAILOVER` (take over the -replid) → demote self to replica. Each step exists to close a loss -window: skip the pause and writes keep racing ahead; skip the -catch-up and the tail of the stream dies with the demotion. And this -is the *manual, graceful* path — an unplanned primary death has no -pause and no catch-up, which is where Step 1's loss window cashes -out. Question: which Raft mechanism replaces this entire dance, and -what does it cost per write? +> **In:** an operator who wants to move the primary role without +> losing writes. **Out:** the four documented steps in the order the +> code performs them, and the one that has no unplanned equivalent. + +`failoverCommand` (:5565) hand-coordinates what Raft's election does +automatically. The happy path is documented in the function's own +header comment (:5542-5549) — note step 3 precedes step 4, i.e. the +primary demotes *itself* before asking the target to take over: + +``` + 1. primary initiates a client pause write, stopping replication traffic + 2. primary periodically checks whether any replica has consumed the + entire replication stream, via acks + 3. once a replica has caught up, the primary itself becomes a replica + 4. primary sends PSYNC FAILOVER to the target, which if accepted makes + the replica the new primary and starts a sync +``` + +Each step closes a loss window: skip the pause and writes keep racing +ahead of the catch-up check; skip the catch-up and the tail of the +stream dies with the demotion. `FAILOVER ABORT` (:5571-5579) is the +only escape, because `REPLICAOF` is disabled during a failover, and +`FORCE` skips step 2 — which is precisely opting back into the loss +window. `abortFailover` (:5523-5536) unwinds via +`replicationUnsetPrimary` if the failover had already reached +`FAILOVER_IN_PROGRESS`. + +And this is the *manual, graceful* path. An unplanned primary death +has no pause and no catch-up, which is where Step 1's loss window +cashes out. Question: which Raft mechanism replaces this entire +dance, and what does it cost per write? ## Where each step lives in the code +All anchors are valkey at `8891441ab`. + | anchor | what it is | step | |---|---|---| -| server.c:3609 | `propagateNow` — the rewrite point | 2 | -| replication.c:352-366 | `feedReplicationBufferWithObject` — one buffer, many readers | 3 | -| replication.c:449 | `feedReplicationBuffer` — append + wake replicas | 3 | -| replication.c:137 | `createReplicationBacklog` — the resync ring | 4 | +| t_set.c:969-975 | `spopCommand` — the SPOP→SREM rewrite, at the command's own site | 2 | +| t_set.c:790-791, 922, 937, 949 | `spopWithCountCommand` — DEL rewrite, batched SREMs, propagation suppressed | 2 | +| server.c:3663 / 3729 / 3751-3762 | `alsoPropagate`, `propagatePendingCommands`, the MULTI/EXEC wrap | 2 | +| server.c:3609-3650 | `propagateNow` — the dispatcher (AOF, replicas, cluster); **not** the rewriter | 2 | +| replication.c:336 | `prepareReplicasToWrite` — the actual wake | 3 | +| replication.c:354-367 | `feedReplicationBufferWithObject` — the robj wrapper | 3 | +| replication.c:449-552 | `feedReplicationBuffer` — one buffer, many cursors | 3 | +| replication.c:486-487 | block-size clamp: `backlog/16`, `PROTO_REPLY_CHUNK_BYTES` | 3 | +| replication.c:518-537 | replica cursor and backlog cursor, same three moves | 3 | +| replication.c:560-630 | `replicationFeedReplicas`; sub-replica early return at :572 | 2, 3 | +| replication.c:671-692 | `replicationFeedStreamFromPrimaryStream` — chaining verbatim | 3 | +| replication.c:135-146 | `createReplicationBacklog` — the resync ring | 4 | | replication.c:854 | `primaryTryPartialResynchronization` — PSYNC accept/deny | 4 | -| replication.c:1077 | `syncCommand` — full sync: fork + RDB + stream | 5 | -| replication.c:3731+ | replica-side `REPL_STATE_*` handshake machine | 5 | +| replication.c:866-867 | the replid / replid2 identity test | 4 | +| replication.c:889-891 | the backlog range inequality | 4 | +| replication.c:840 / 935 / 937 | `+FULLRESYNC` and `+CONTINUE` replies | 4 | +| replication.c:1077 | `syncCommand` — full sync entry point | 5 | +| replication.c:988, 1002-1004, 1018, 1021 | `startBgsaveForReplication`: transport choice, both forks | 5 | +| server.h:389-407 | the 13 `REPL_STATE_*` values, handshake range bracketed | 5 | +| replication.c:4077-4197 | `syncWithPrimary` — the replica-side handshake machine | 5 | +| rdb.c:3160-3173 | version check, *then* `emptyData()` | 5 | +| replication.c:2122-2128 | `replicationEmptyDbCallback` — newlines during the wipe | 5 | | replication.c:4564 | `replicaofCommand` — topology is a runtime command | 5 | -| replication.c:4947 | `replicationRequestAckFromReplicas` | 6 | -| replication.c:4996 | `waitCommand` — the semi-sync opt-in | 6 | -| replication.c:5565 | `failoverCommand` — coordinated manual failover | 7 | +| replication.c:4947-4949 | `replicationRequestAckFromReplicas` — sets a flag, `beforeSleep` broadcasts | 6 | +| replication.c:4962-4975 | `replicationCountAcksByOffset` — counts *received*, not fsynced | 6 | +| replication.c:4996-5026 | `waitCommand` — the semi-sync opt-in | 6 | +| replication.c:4979 / 5030 | `replicationCountAOFAcksByOffset` / `waitaofCommand` | 6 | +| replication.c:5542-5549 | the FAILOVER happy path, in comments | 7 | +| replication.c:5565 | `failoverCommand` | 7 | +| config.c:3274/3275/3352/3393/3417/3453/3477 | the replication config defaults | 4, 5 | Slice, don't read linearly: start at `feedReplicationBuffer` (the -hot path), then `primaryTryPartialResynchronization` (the decision), -then `waitCommand` and `failoverCommand` (the two attempts to buy -back what async gave up). +hot path), then `primaryTryPartialResynchronization` (the decision, +and its two-test structure), then `waitCommand`/`waitaofCommand` and +`failoverCommand` (the two attempts to buy back what async gave up). ## Questions for notes.md -1. Replication is statement-shipping after `propagateNow` rewrites — +1. Replication is statement-shipping after the per-command rewrites — what's the analogue of topic 5's logical-vs-physical WAL choice? 2. Backlog sizing: repl-backlog-size vs write rate vs disconnect duration — write the inequality for "partial resync succeeds". @@ -200,20 +456,204 @@ back what async gave up). ## Done when +Answer each before unfolding it. + - [ ] You can explain what "ack first, replicate later" means for a client that received a success reply. -- [ ] You can describe PSYNC's `(replid, offset)` scheme and say what makes a partial resync possible or impossible. + +
Answer + + It means the reply is a statement about one machine. The primary + executed the write and answered; the bytes then entered the shared + replication buffer (`feedReplicationBuffer`, replication.c:449-552) + and will reach replicas whenever their sockets drain. + + If the primary dies in that gap, the write is gone and the client + was told otherwise. The gap is measured in bytes as replication lag + — the distance between the primary's stream offset and a replica's + `repl_ack_off`. + + What it buys is that the write path never contains a network round + trip or a follower fsync. This topic's bench shows the size of what + was avoided: forcing a durable follower ack per entry takes + throughput to 341 entries/s with a 3889.5 µs p99. + +
+ +- [ ] You can name where nondeterministic commands get rewritten, and why it is not one central place. + +
Answer + + At each command's own implementation, because only the command + knows which random choice it made. `spopCommand` calls + `setTypePopRandom` (t_set.c:970) and rewrites itself to `SREM + ` at t_set.c:975. + `spopWithCountCommand` rewrites to `DEL`/`UNLINK` when it empties the + set (t_set.c:790-791), otherwise batches `SREM`s via `alsoPropagate` + (t_set.c:922, 937) and suppresses the original with + `preventCommandPropagation` (t_set.c:949). + + `propagateNow` (server.c:3609-3650) is *not* the rewrite point. It + is the dispatcher that fans an already-final command out to + `feedAppendOnlyFile` (:3647), `replicationFeedReplicas` (:3648) and + `clusterFeedSlotExportJobs` (:3649). The public queueing API is + `alsoPropagate` (server.c:3663), drained by + `propagatePendingCommands` (server.c:3729), which wraps multi-command + batches in MULTI/EXEC (:3751, :3762). + + The cost of this design is that determinism is an obligation on + every command ever added, checked by review rather than by + construction — which is exactly what a physical WAL avoids. + +
+ +- [ ] You can describe PSYNC's `(replid, offset)` scheme, state the range inequality, and say what the check cannot detect. + +
Answer + + A replica sends `PSYNC `. + `primaryTryPartialResynchronization` (replication.c:854) applies two + tests. Identity (:866-867): the replid must equal `server.replid`, + or equal `server.replid2` with `psync_offset <= + server.second_replid_offset`. Range (:889-891, read as its + negation): + + backlog->offset ≤ psync_offset ≤ backlog->offset + backlog->histlen + + Pass both and the primary replies `+CONTINUE` (:935/937); fail + either and `+FULLRESYNC` (:840) with Step 5's fork. + + What it cannot detect is content divergence at a matching offset. + Raft's `AppendEntries` verifies that `prevLogIndex` holds + `prevLogTerm` before appending; PSYNC compares an id and a byte + count and assumes the bytes below are identical. `replid2` plus + `second_replid_offset` is the narrow patch for the one case where + that assumption predictably breaks — a promoted replica's siblings. + +
+ - [ ] You can size the replication backlog from a write rate and a tolerable disconnect window. -- [ ] You can explain why full sync forks, and connect it to copy-on-write. -- [ ] You can say precisely what WAIT does and does not guarantee — then check it against this topic's measured table, where WAIT 1 with per-entry follower fsync costs 341 entries/s. + +
Answer + + write_rate × disconnect_seconds ≤ repl-backlog-size + + The default is 10 MB (`config.c:3453`), with a 3600 s TTL + (`config.c:3477`) after which an idle backlog is freed entirely. + + At 5 MB/s of replication stream that is 2 seconds of tolerable + disconnect; at 100 KB/s it is 100 seconds. To survive a 30-second + blip at 5 MB/s you need 150 MB. Note the rate is *stream* bytes, + not client bytes — the MULTI/EXEC wrapping and the SPOP→SREM + rewrites change the size. + + The cost of getting it wrong is not a slow resync, it is a fork + plus a full RDB plus a full reload, during which (rdb.c:3169-3173) + the replica has flushed its dataset and holds nothing. + +
+ +- [ ] You can explain why full sync forks, which transport is the default at this pin, and connect it to copy-on-write. + +
Answer + + It forks to freeze a point-in-time snapshot without stopping the + primary: the child inherits a copy-on-write view of the heap, so + the parent keeps serving writes and only the modified pages are + duplicated (topic 5). Meanwhile the parent accumulates the new + writes in the Step 3 buffer to stream after the snapshot. + + `startBgsaveForReplication` (replication.c:988) chooses the + transport at :1002-1004: `rdbSaveToReplicasSockets` (:1018) when the + replica advertises `REPLICA_CAPA_EOF` and `repl_diskless_sync` is + on, else `rdbSaveBackground` (:1021) via a file. **Both fork** — + diskless removes the file, not the fork. + + At this pin `repl-diskless-sync` defaults to **enabled** + (config.c:3274), which changed since Redis 6. `repl-diskless-load` + is still **disabled** (config.c:3352), so the replica writes the RDB + to a file before loading it. + +
+ +- [ ] You can say precisely what WAIT does and does not guarantee, and name the command that guarantees more. + +
Answer + + `waitCommand` (replication.c:4996-5026) blocks the client until n + replicas have acked the offset of that client's own last write + (`getClientWriteOffset`, :4953). `replicationCountAcksByOffset` + (:4962-4975) counts a replica if it is `REPLICA_STATE_ONLINE` and + `repl_ack_off >= offset` — bytes **received and processed**, not + fsynced. The GETACK broadcast is not sent by + `replicationRequestAckFromReplicas` (:4947-4949) itself; that only + sets a flag which `beforeSleep()` acts on, so many blocked clients + cost one broadcast. + + So WAIT does not promise durability on the replica, and it does not + promise that the acking replica survives the next failover. It is a + report about the present, after the write was already applied; a + Raft commit is a promise about the future, made before it was. + + `WAITAOF` (:5030) is the stronger one: + `replicationCountAOFAcksByOffset` (:4979) counts `repl_aof_off`, + bytes fsynced to the replica's AOF. That is the axis this topic's + table measures — 341 entries/s at one follower fsync per entry + versus 20,174 with none. + +
+ +- [ ] You can list FAILOVER's four steps in the order the code performs them, and say which has no unplanned equivalent. + +
Answer + + From `failoverCommand`'s own header comment + (replication.c:5542-5549): (1) pause client writes; (2) poll acks + until some replica has consumed the whole stream; (3) the primary + makes *itself* a replica; (4) send `PSYNC FAILOVER` to the target, + which promotes it. Steps 3 and 4 are in that order — the demotion + precedes the handoff. + + Step 2 is the one with no unplanned equivalent. A crashed primary + cannot poll acks, so an unplanned failover promotes whatever replica + the operator or sentinel picks, at whatever offset it had reached. + `FORCE` opts out of step 2 deliberately and is the same bargain. + + Raft replaces the whole dance with the election restriction: a + candidate whose log is not up-to-date cannot win a vote, so + "catch-up before promotion" is enforced by every voter on every + election rather than by a coordinating primary that may be dead. + The price is one majority round trip on every write. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + Question 3 is the one with a crisp code answer. A sub-replica gets + the primary's byte stream proxied *verbatim*: + `replicationFeedStreamFromPrimaryStream` (replication.c:671-692) + takes the raw buffer and calls `prepareReplicasToWrite()` (:689) and + `feedReplicationBuffer()` (:690) on it unchanged. It never + re-encodes commands. + + That is why offsets stay coherent down a chain — every node in the + chain is measuring the same byte sequence. The matching guard is at + `replicationFeedReplicas` :572, which returns early on a node that + has a primary of its own, so an intermediate replica cannot generate + its own stream and desynchronise the numbering. + +
+ ## References **Code** -- [valkey](https://github.com/valkey-io/valkey) — `src/replication.c` - (~5600 lines; slice it with the anchor map above rather than reading - linearly) and `src/server.c` (`propagateNow`, the statement-rewrite - point) +- [valkey](https://github.com/valkey-io/valkey) at `8891441ab` — + `src/replication.c` (5726 lines; slice it with the anchor map above + rather than reading linearly), `src/server.c` (the propagation + dispatcher), `src/t_set.c` (the SPOP rewrite), `src/rdb.c` (the + flush-before-load), `src/server.h` (the `REPL_STATE_*` enum), + `src/config.c` (every default quoted above) **Papers** - None — this is a pure code walk; the consensus counterpoint is diff --git a/topics/15-replication-consensus/reading-vsr.md b/topics/15-replication-consensus/reading-vsr.md index 45ed0ea..cf91395 100644 --- a/topics/15-replication-consensus/reading-vsr.md +++ b/topics/15-replication-consensus/reading-vsr.md @@ -1,136 +1,350 @@ # Viewstamped Replication: same invariants, opposite choices The other consensus protocol — actually the FIRST (VR 1988 predates -Paxos's publication). Read it AFTER Raft: same invariants, opposite -engineering choices at almost every fork. This chapter builds those -forks one at a time — the vocabulary mapping, deterministic -round-robin leadership instead of elections, logs shipped at view -change instead of repaired after, and (the shocker) durability -without disk. TigerBeetle ships VSR in production, so this is not a -museum piece. +Paxos's 1998 publication). Read it AFTER Raft: same invariants, +opposite engineering choices at almost every fork. This chapter +builds those forks one at a time — the vocabulary mapping, +deterministic round-robin leadership instead of elections, logs +shipped at view change instead of repaired after, and (the shocker) +durability without disk. TigerBeetle ships VSR in production, so this +is not a museum piece. + +**Two documents, and they are not the same protocol.** Every claim +below names which one it comes from: + +- **VR Revisited** — Liskov & Cowling, *Viewstamped Replication + Revisited*, MIT-CSAIL-TR-2012-021, 16 pp. State-machine + replication, three sub-protocols, no disk in normal operation *or* + view change. This is the one to read. +- **VR 1988** — Oki & Liskov, *Viewstamped Replication: A New Primary + Copy Method to Support Highly-Available Distributed Systems*, MIT + LCS / PODC 1988, 10 pp. A *transactional* system: its actors are + "cohorts", a view is a **set** of cohorts plus a designated primary + (not just a number), and operations carry **viewstamps**. It does + write to stable storage during a view change. + +VR Revisited says so itself, in §4.3: "The original VR specification +used a protocol that wrote to disk during the view change but did not +require writing to disk during normal case processing." Quoting the +2012 no-disk result as a 1988 result is the standard error. ## The problem in one sentence The same problem as Raft — an acked write must survive any f of 2f+1 nodes dying — but VSR asks how much of Raft's machinery is *forced* and how much is *chosen*: no randomized timeouts, no votes, -and in the pure protocol **zero fsyncs**, versus Raft's fsync of -`voted_for` and log on every vote and append. +and in VR Revisited's protocol **zero disk I/O in normal operation +and view change**, versus Raft's fsync of `votedFor` and log on every +vote and append. ## The concepts, step by step ### Step 1 — same machine, different words +> **In:** Raft's vocabulary from the previous chapters. **Out:** the +> decoder that makes VR Revisited readable in one sitting, plus the +> two 1988 terms that decode to nothing in Raft. + VSR replicates a log through a distinguished node exactly like Raft; only the names differ. Keep this decoder open for the whole paper: -| Raft | VSR | +| Raft | VR Revisited (2012) | |---|---| -| term | view | +| term | view-number | | leader | primary | | election | view change | | log index | op-number | -| commit_index | commit-number | +| commitIndex | commit-number | +| — | status ∈ {normal, view-change, recovering} | | RequestVote / AppendEntries | STARTVIEWCHANGE / DOVIEWCHANGE / PREPARE / PREPAREOK | -A **view** is a numbered epoch with one primary — Raft's term, -eleven years earlier. The protocol splits into three sub-protocols -(normal operation, view change, recovery), and the next three steps -take them in order. +A **view** in VR Revisited is a numbered epoch with one primary — +Raft's term, twenty-four years earlier. A **status** is the extra +concept with no Raft counterpart: §4.1 opens by saying replicas +"participate in processing of client requests only when their status +is normal", and calls that constraint "critical for correctness". +Raft has no such flag; a Raft node is always willing to append. + +Figure 2 of VR Revisited lists the whole per-replica state: the +configuration, the replica number, view-number, status, op-number, +the log, the commit-number, and the client-table. Two of those +deserve a second look. The **configuration** is "a sorted array +containing the IP addresses of each of the 2f + 1 replicas" — sorted, +because Step 3's rotation is an index into it. The **client-table** +records each client's most recent request number and its result, and +exists because §4 allows a client "just one outstanding request at a +time"; it is how VSR deduplicates retries, a problem Raft's paper +leaves to §8. + +Two 1988 words that do not decode: a **cohort** is a replica, and a +**viewstamp** is the pair ⟨viewid, timestamp⟩ that gave the protocol +its name. VR Revisited §4.2 explains that it dropped them: "VR as +originally defined used a slightly different approach: it assigned +each operation a viewstamp... At any op-number, VR retained the +request with the higher viewstamp. VR got its name from these +viewstamps." The 2012 protocol takes the whole log from the latest +previous active view instead. + +Concretely: your own `struct Node` in +`experiments/src/raft.rs:44-60` already holds most of Figure 2. +`term` (:48) is view-number, `log` (:50) is the log, `commit_index` +(:51) is commit-number, `peers` (:46) is the configuration. Three +fields have no VSR counterpart — `voted_for` (:49), `votes_received` +(:59), and the randomized `election_timeout` (:57) — and three VSR +fields are missing: op-number, status, and the client-table. That +diff is the whole chapter in one screen. ### Step 2 — normal operation: the same wire shape as AppendEntries -Client sends the request to the primary; the primary assigns the -next op-number, appends to its log, and broadcasts PREPARE; each -replica appends and answers PREPAREOK; on f PREPAREOKs (f+1 copies -counting the primary = a majority of 2f+1), the primary commits, -executes, and replies to the client: +> **In:** a client request arriving at the primary of view v. +> **Out:** the five message types in order, the exact quorum count +> (which is *f*, not f+1), and the two ways a view-number mismatch is +> handled. + +VR Revisited §4.1, step by step. The client sends +⟨REQUEST op, c, s⟩ to the primary. The primary checks the +client-table (a stale request-number is dropped; the most recent one +is answered from the cached result), advances op-number, appends, and +broadcasts ⟨PREPARE v, m, n, k⟩ — where `n` is the new op-number and +`k` is the current commit-number, so commits piggyback on the next +prepare. Backups process PREPAREs **in order**, doing state transfer +if they are missing earlier entries, then reply ⟨PREPAREOK v, n, i⟩. + +The quorum count is the detail people misquote: ``` - client ─► primary: PREPARE(view, op-number, request) ─► replicas - ◄─ f × PREPAREOK ─┘ - commit, execute, reply (1 round trip, same as Raft) + §4.1 step 5: "The primary waits for f PREPAREOK messages from + different backups; at this point it considers the operation (and + all earlier ones) to be committed." + + f PREPAREOKs, not f+1 — the primary's own copy is the +1. + + f = 2, n = 2f + 1 = 5 + 2 PREPAREOKs + the primary itself = 3 copies = majority of 5 ✓ + + Compare Raft: matched[majority(5) - 1] = matched[2], where the + leader counts ITSELF in the matched vector. Same 3, arrived at by + counting a different thing. Get this wrong by one in either + direction and you have either a stall or a split brain. ``` +Then ⟨REPLY v, s, x⟩ to the client. Backups learn of the commit from +the `k` in the next PREPARE; if no client request arrives "in a timely +way" the primary sends ⟨COMMIT v, k⟩ instead — VSR's heartbeat, and +note it exists to carry the commit-number, not to prove liveness. + +The mismatch handling is the other thing to take from §4.1: "Replicas +only process normal protocol messages containing a view-number that +matches the view-number they know. If the sender is behind, the +receiver drops the message. If the sender is ahead, the replica +performs a state transfer." Raft's rule is symmetric — a higher term +always makes you a follower — where VSR distinguishes *stale sender* +(drop) from *stale self* (go fetch, §5.2). + Same quorum arithmetic as Raft, same one-round-trip latency. The differences are all in what happens when this smooth path breaks. ### Step 3 — view change: the next primary is scheduled, not elected +> **In:** replicas that have stopped hearing from the primary of view +> v. **Out:** the formula that names the next primary, the two +> distinct quorum sizes in the protocol, and the log-selection rule +> with its tie-break. + Raft elects: candidates race, randomized timeouts break ties, votes -are persisted. VSR schedules: the primary of view v is simply -**replica v mod n** — deterministic, known to everyone in advance. -Suspecting the primary, replicas send STARTVIEWCHANGE for view v+1; -once a replica has seen f+1 of those, it sends DOVIEWCHANGE — *with -its entire log* — to the scheduled next primary. That primary picks -the best log among the f+1 it received (highest last-normal-view, -then highest op-number — Raft's election restriction, applied after -the fact) and installs it everywhere via STARTVIEW: +are persisted. VSR schedules. VR Revisited §4 states it plainly: "The +identity of the primary isn't recorded in the state but rather is +computed from the view-number and the configuration... The primary is +chosen round-robin, starting with replica 1, as the system moves to +new views." Replicas are numbered by sorted IP address, smallest +first. + +§4.2's three-message protocol, with **two different quorum sizes**: + +1. A replica noticing the need advances its view-number, sets status + to `view-change`, and broadcasts ⟨STARTVIEWCHANGE v, i⟩. +2. On receiving STARTVIEWCHANGE for its view-number **from f other + replicas**, it sends ⟨DOVIEWCHANGE v, l, v', n, k, i⟩ to the node + that will be primary — where `l` is its whole log and `v'` is "the + view number of the latest view in which its status was normal". +3. The new primary waits for **f + 1 DOVIEWCHANGE messages from + different replicas (including itself)**, then selects the log from + the message with the largest `v'`, breaking ties on the largest + `n`. It takes the largest commit-number it saw, sets status + `normal`, and broadcasts ⟨STARTVIEW v, l, n, k⟩. ```rust -// the next primary is DETERMINED: view mod n. it just needs f+1 logs +// ILLUSTRATION — not quoted from anything. This is VR Revisited §4.2 +// step 3 written in the idiom of your own Raft node; the state it +// mutates is the VSR analogue of experiments/src/raft.rs:44-60 +// (`struct Node`), and the method it would replace is the vote +// tally reached from experiments/src/raft.rs:95 (`receive`). +// Check it against the paper's own wording before trusting it. fn install_view(&mut self, view: u64, msgs: &[DoViewChange]) { - assert!(msgs.len() >= self.f + 1); // quorum intersects commits + assert!(msgs.len() >= self.f + 1); // §4.2 step 3 let best = msgs.iter() .max_by_key(|m| (m.last_normal_view, m.op_number)) - .unwrap(); // Raft's election restriction, - self.log = best.log.clone(); // applied AFTER the fact — - self.op_number = best.op_number; // logs ship at view change, - self.commit_number = // where Raft repairs later - msgs.iter().map(|m| m.commit_number).max().unwrap(); + .unwrap(); // largest v', then largest n + self.log = best.log.clone(); + self.op_number = best.op_number; + self.commit_number = // largest k received, + msgs.iter().map(|m| m.commit_number).max().unwrap(); // not best's self.broadcast(StartView { view, log: &self.log }); } ``` -Note what's missing: no votes, no randomized timeouts, no -split-vote livelock — determinism removed them. The costs traded: -DOVIEWCHANGE ships whole logs (bandwidth per view change, where -Raft's election ships nothing and repairs followers lazily), and a -down node's turn in the rotation forces another view change -(question 1). The safety argument is the same quorum intersection as -Raft's: f+1 logs must include at least one node holding any -committed entry. +The subtlety the assert hides: `f` and `f + 1` are both quorum sizes +in this protocol and they are not interchangeable. Step 2's threshold +is f *other* replicas (f+1 including self, a majority); step 3's is +f+1 *including* self. Getting step 3 down to f would let a new +primary install a log without a majority behind it, and the +intersection argument — f+1 logs must include at least one node +holding any committed entry — collapses. + +On receiving STARTVIEW, replicas replace their log wholesale, and if +it contains uncommitted operations they send PREPAREOK for them +(§4.2 step 5) — which is how the new primary learns what to commit. + +What is missing: no votes, no randomized timeouts, no split-vote +livelock — determinism removed them. What it costs is bandwidth, and +the bandwidth is computable: + +``` + Take this topic's own bench shape: 2000 entries x 128 B = 256 KB + of log, f = 2, n = 5. + + VSR view change: + DOVIEWCHANGE inbound (f + 1) x 256 KB = 768 KB + STARTVIEW outbound (n - 1) x 256 KB = 1024 KB + total ~ 1.75 MB per view change + + Raft election: + RequestVote carries only (term, candidateId, lastLogIndex, + lastLogTerm) — 4 integers. Broadcast to 4 peers: + ~ 128 BYTES + + ~14,000x more bytes per leadership change. Raft pays it back later, + one AppendEntries at a time, only to the followers that actually + diverged; VSR pays it up front, always, to everyone. +``` + +That is the real trade — not "VSR is wasteful" but *when* the repair +cost is paid. If view changes are rare and logs are long, Raft wins; +if divergence is common and logs are short, shipping them once beats +probing. And a down node still takes its turn in the rotation, +forcing another view change (question 1). ### Step 4 — recovery: durability from replication, not disk -The shocker. Raft fsyncs `voted_for` and log entries before -answering — a crashed node reads its promises back from disk. VSR's -pure protocol writes NOTHING to disk: a committed entry lives in -f+1 memories, and the protocol tolerates f failures, so *some* -survivor always remembers it. A crashed replica doesn't trust its -disk at all — it runs the **recovery protocol**: rejoin, send a -RECOVERY message with a nonce (a fresh random number that -distinguishes this recovery from any earlier one — question 4), and -rebuild its state from f+1 responses, rejoining only when caught up. - -The catch, stated honestly: "f failures" must mean f *independent* -failures. Whole-cluster power loss is f+1 simultaneous memory wipes -— everything is gone, where fsync-per-write Raft replays its disk -(question 3 makes you construct the exact losing sequence). Which is -why TigerBeetle adds disk back but keeps VSR's recovery *thinking*: -a node that cannot trust its own storage (checksum failure, torn -write) recovers from its peers — a fault model Raft ignores -entirely, since Raft assumes whatever was fsynced reads back -faithfully. +> **In:** a replica that has just rebooted with an empty memory. +> **Out:** the three-message recovery protocol, the two quorum +> conditions on its responses, and the exact sentence in which the +> paper qualifies the no-disk claim. + +The shocker. Raft fsyncs `votedFor` and log entries before answering +— a crashed node reads its promises back from disk. VR Revisited's +protocol writes nothing to disk in normal operation or view change: a +committed entry lives in f+1 memories, and the protocol tolerates f +failures, so *some* survivor always remembers it. A crashed replica +does not trust its own memory at all — it sets status `recovering` +and runs §4.3: + +1. Send ⟨RECOVERY i, x⟩ to all other replicas, where `x` is a + **nonce**. +2. A replica replies **only if its status is `normal`**, with + ⟨RECOVERYRESPONSE v, x, l, n, k, j⟩ — and `l`, `n`, `k` are + **nil unless j is the primary of its view**. Only the primary + ships a log. +3. The recovering replica waits for **at least f + 1** + RECOVERYRESPONSEs from different replicas, all carrying its own + nonce, **including one from the primary of the latest view it + learns of in these messages**. Then it updates from the primary's + message and sets status `normal`. + +Two conditions on step 3, and both are load-bearing. f+1 responses +give the intersection argument. The "including the primary of the +latest view" clause is what makes the log it copies authoritative — +without it a recovering node could rebuild from f+1 backups that are +all behind. And while recovering it "does not participate in either +the request processing protocol or the view change protocol", which +has a consequence §4.3 spells out: if the recovering replica would be +the primary of a view change in progress, that view change cannot +complete, and the group must do a further one. + +The nonce is not decoration. §4.3: "The protocol uses the nonce to +ensure that the recovering replica accepts only RECOVERYRESPONSE +messages that are for this recovery and not an earlier one. It can +produce the nonce by reading its clock; this will produce a unique +nonce assuming clocks always advance. Alternatively, it could +maintain a counter on disk and advance this counter on each +recovery." Without it, responses from a *previous* crash-and-recover +cycle — stale logs still in flight — would be accepted as current. + +The catch, stated by the paper itself and not softened. §4.3 gives +the alternative it rejected: fsync before PREPARE at the primary and +before PREPAREOK at the backups, which "adds a delay to normal case +processing". Then the justification, with its condition attached: + +> the disk write is "unnecessary because the state is also stored at +> the other replicas and can be retrieved from them, using a recovery +> protocol. Retrieving state will be successful **provided replicas +> are failure independent**, i.e., highly unlikely to fail at the same +> time. If all replicas were to fail simultaneously, state will be +> lost if the information on disk isn't up to date." + +Named mitigations, all outside the protocol: UPSs, non-volatile +memory, and placing replicas in different geographic locations. So +the honest summary is that VSR moved a durability requirement from +the storage layer to the deployment, and the deployment has to hold +up its end. + +Price that against what fsync costs here. Topic 5 measured a real +`F_FULLFSYNC` on macOS/APFS at 337 commits/s, and this topic's +`repl_lag` bench gets 341 entries/s with the follower fsyncing every +entry versus 20,174 with none. That 59× is exactly the number VSR is +declining to pay — and the UPS is what it pays instead. + +The 1988 paper reached the same place by a different road and +described the failure mode more precisely than most retellings do. +Its §4.2 ("Stable Storage") assumes most cohort state is volatile, +defines a *catastrophe* as a majority crashing simultaneously, and +then says something surprising: "a catastrophe does not cause a group +to enter a new view missing some needed information. Rather, it +causes the algorithm to never again form a new view." It stalls; it +does not silently lose. Its conclusion is candid about the whole +experiment: "we chose to avoid the use of stable storage as much as +possible because we were interested in understanding the extent to +which having several replicas eliminated the need for stable storage. +We found that catastrophes... could sometimes occur in our system." ### Step 5 — the forks in the road, side by side +> **In:** both protocols, understood. **Out:** the four decisions +> that differ, with the invariant that is identical underneath each +> — and therefore the evidence that each was a choice. + The reason to read this paper is the table — every row is a place where two correct protocols chose differently, which proves the choice was engineering, not necessity: ``` - choice Raft VSR (Revisited) - ───────────────────────────────────────────────────────────── - who leads next any up-to-date node ROUND-ROBIN: view mod n - that wins votes (deterministic!) - log transfer new leader repairs new primary RECEIVES logs - followers forward in DOVIEWCHANGE, picks best - durability fsync log before ack NO DISK REQUIRED — - durability from replication; - recovery protocol replaces it - vote persistence voted_for fsynced view number in memory; - recovery rejoins carefully + choice Raft VR Revisited (2012) + ───────────────────────────────────────────────────────────────── + who leads next any up-to-date node ROUND-ROBIN: computed + that wins votes from view-number and + the sorted configuration + log transfer new leader repairs new primary RECEIVES f+1 + followers forward, logs in DOVIEWCHANGE and + one AppendEntries picks max (v', n) + at a time + durability fsync currentTerm, NO DISK in normal + votedFor, log before operation or view change; + responding (Fig 2) recovery protocol replaces + it, "provided replicas are + failure independent" + stale participation always willing to status must be `normal`; + append a recovering replica + answers nothing ``` The invariants underneath are identical: one primary per @@ -140,23 +354,48 @@ model. What differs is *where each protocol spends*: Raft spends fsyncs and election randomness; VSR spends view-change bandwidth and a stricter independence assumption. +TigerBeetle is the third answer. It ships VSR in Zig +(`src/vsr/replica.zig`, `docs/internals/vsr.md`) and puts disk back — +but keeps VSR's recovery *thinking* and extends it to a fault Raft's +model excludes entirely: storage that lies. Its docs cite the CTRL +protocol from Alagappan et al., *Protocol-Aware Recovery for +Consensus-Based Storage* (FAST '18), and state the rule that follows +from it — a replica does **not** nack a corrupt log entry, "since it +_might_ be the prepare being requested". Raft's Figure 2 has no +vocabulary for "I have an entry but cannot read it"; VSR's +recover-from-peers instinct does. TigerBeetle is **not** in this +repo's pin table, so no line anchors are given for it and `main` +moves — treat those two paths as pointers, not citations. + ## How to read the paper (with the concepts in hand) -Read "Viewstamped Replication Revisited" (2012), not the 1988 -original: +Read *Viewstamped Replication Revisited* (2012). Section numbers +below are that document's. - **§1–3 (intro, background, the model)** — skim; Step 1's decoder - makes it fast. -- **§4 (the protocol)** — the payload. §4.1 normal operation is - Step 2 — map every message onto the AppendEntries flow you know. - §4.2 view change is Step 3 — check the `install_view` condensation - above against the real message rules. §4.3 recovery is Step 4 — - read for the nonce and for what a recovering replica may NOT do. -- **§5 (pragmatics)** — read §5.1 (efficient recovery) and the - discussion of when disk is reintroduced; this is where the - no-disk argument gets its fine print. -- **§6–7 (reconfiguration, discussion)** — skim; membership change - is Raft §6's joint consensus by another road. + makes it fast. §3 is where the 2f+1 / f fault model is fixed. +- **§4 (the protocol)** — the payload, and Figure 2 (replica state) + is the page to keep open. §4.1 normal operation is Step 2 — map + every message onto the AppendEntries flow you know, and note the + quorum is f PREPAREOKs. §4.2 view change is Step 3 — check the + `install_view` illustration against the real message rules, and + read the viewstamp paragraph at the end for the 1988 contrast. + §4.3 recovery is Step 4 — read for the nonce, for the "including + one from the primary" clause, and for what a recovering replica may + NOT do. §4.4 covers non-deterministic operations, which is valkey's + SPOP problem in another vocabulary. +- **§5 (pragmatics)** — §5.1 is efficient recovery (§4.3's protocol + is expensive precisely because logs are big), §5.2 is state + transfer, and this is where the recovery cost gets bounded. +- **§6 (optimizations)** and **§7 (reconfiguration)** — skim; + reconfiguration is Raft §6's joint consensus by another road. +- **§8 (correctness)** — read the paragraph on why `status` must + gate participation; it is the argument Step 1 flagged. + +The 1988 paper is worth 20 minutes only for §4.2 ("Stable Storage") +and the conclusions, where the no-disk experiment is stated in the +authors' own words — and for seeing how different a *transactional* +formulation looks. Throughout, keep asking Step 5's question: is this rule forced by the invariants, or is it a choice? That habit is the transferable @@ -164,9 +403,10 @@ skill — it's how you'll evaluate M15 stage 2's design decisions. ## Questions for notes.md -1. Round-robin primary (view mod n): what does this remove from the - protocol (no vote-splitting, no randomized timeouts) and what - does it cost (a down node's turn)? +1. Round-robin primary (computed from view-number and the sorted + configuration): what does this remove from the protocol (no + vote-splitting, no randomized timeouts) and what does it cost (a + down node's turn)? 2. DOVIEWCHANGE ships whole logs to the new primary — Raft ships nothing at election, repairing later. Bandwidth vs latency: when is each better? @@ -179,23 +419,196 @@ skill — it's how you'll evaluate M15 stage 2's design decisions. ## Done when -- [ ] You can state which VSR concepts are Raft's under other names, and which are genuinely different choices. -- [ ] You can explain what round-robin primary selection removes from the protocol, and what it costs. -- [ ] You can compare DOVIEWCHANGE's whole-log shipping against Raft's incremental repair and say when each is cheaper. -- [ ] You can write the failure sequence that the no-disk recovery argument depends on, and say what makes it safe. -- [ ] You can say why the recovery protocol needs a nonce. +Answer each before unfolding it. + +- [ ] You can state which VSR concepts are Raft's under other names, which are genuinely different, and which belong only to the 1988 paper. + +
Answer + + Renames: view-number = term, primary = leader, view change = + election, op-number = log index, commit-number = commitIndex. + + Genuinely different in VR Revisited: **status** ∈ {normal, + view-change, recovering}, which gates participation (§4.1 calls + that "critical for correctness") and has no Raft counterpart; the + **client-table**, which deduplicates client retries inside the + protocol; and the **configuration** as a sorted IP array, because + the primary is an index into it rather than an election winner. + + Only in 1988: **cohort** for replica, a **view** as a *set* of + cohorts plus a designated primary rather than a number, and the + **viewstamp** ⟨viewid, timestamp⟩ that named the protocol. VR + Revisited §4.2 says it replaced viewstamps with "take the log from + the latest previous active view". + +
+ +- [ ] You can state VSR's normal-operation quorum exactly, and say why it looks smaller than Raft's. + +
Answer + + §4.1 step 5: "The primary waits for **f** PREPAREOK messages from + different backups". With f = 2 and n = 5 that is 2 messages — plus + the primary's own copy, which is 3, a majority of 5. + + It looks smaller because it counts a different set. Raft's + `maybe_commit` puts the leader's own `matched` into the vector and + takes `matched[majority(5) - 1] = matched[2]`, i.e. it counts the + leader. VSR counts only the backups and adds the primary + implicitly. Same 3 copies either way. + + The other §4.1 rule worth memorising is the view-number mismatch + handling: sender behind → drop the message; sender ahead → do a + state transfer (§5.2) before processing. Raft's rule is symmetric — + a higher term always demotes you. + +
+ +- [ ] You can explain what round-robin primary selection removes from the protocol, what it costs, and where the two different quorum sizes appear. + +
Answer + + It removes candidacy entirely: no votes, no randomized timeouts, no + split-vote livelock, no persisted `votedFor`. The primary of a view + is computed from the view-number and the sorted configuration + (§4), so every replica already knows who it is. + + The cost is that the rotation is blind. If the scheduled next + primary is down, the group must burn another view change to skip + it — and §4.3 adds a nastier case: a *recovering* replica does not + answer DOVIEWCHANGE, so if it is the scheduled primary the view + change stalls until another one is triggered. + + Two quorum sizes, §4.2: a replica sends DOVIEWCHANGE after seeing + STARTVIEWCHANGE from **f other** replicas (step 2); the new primary + installs the view after **f + 1 including itself** (step 3). Both + are majorities of 2f+1, counted from different starting points. + +
+ +- [ ] You can compare DOVIEWCHANGE's whole-log shipping against Raft's incremental repair, with numbers, and say when each is cheaper. + +
Answer + + Take a 256 KB log (this topic's 2000 × 128 B bench), f = 2, n = 5. + A VSR view change moves (f+1) × 256 KB = 768 KB inbound as + DOVIEWCHANGE plus (n−1) × 256 KB = 1 MB outbound as STARTVIEW — + about 1.75 MB. A Raft election broadcasts RequestVote carrying four + integers to four peers: roughly 128 bytes. Four orders of magnitude. + + Raft does not avoid the cost, it defers and targets it: the new + leader repairs only the followers that actually diverged, one + AppendEntries at a time, and §5.3's term-skip optimisation bounds + even that. VSR pays up front, unconditionally, to everyone. + + So: long logs and rare view changes favour Raft; short logs and + frequent divergence favour shipping once. The selection rule is + also strictly simpler in VSR — max on `(v', n)` in one place versus + Raft's per-follower `nextIndex` walk. + +
+ +- [ ] You can write the failure sequence the no-disk argument depends on, and quote the condition the paper attaches to it. + +
Answer + + The losing sequence: 3 of 5 replicas hold entry 42 in memory, the + primary has replied to the client, and then the whole rack loses + power. Nothing was on disk, so entry 42 is gone despite having been + acked. Raft-with-fsync replays it from any of the three logs. + + VR Revisited §4.3 does not hide this. Its justification for + skipping the write is that the state "is also stored at the other + replicas and can be retrieved from them, using a recovery protocol. + Retrieving state will be successful **provided replicas are failure + independent**, i.e., highly unlikely to fail at the same time. If + all replicas were to fail simultaneously, state will be lost if the + information on disk isn't up to date." The mitigations it names — + UPSs, non-volatile memory, geographic separation — are all outside + the protocol. + + The 1988 paper's §4.2 describes the same event and calls the + outcome different: "a catastrophe does not cause a group to enter a + new view missing some needed information. Rather, it causes the + algorithm to never again form a new view." Stall rather than silent + loss. + + The price being declined is measurable here: 337 commits/s at a + real `F_FULLFSYNC` (topic 5), 341 entries/s with a per-entry + follower fsync versus 20,174 without (this topic). + +
+ +- [ ] You can say why the recovery protocol needs a nonce, and what the second condition on its responses is for. + +
Answer + + §4.3: "The protocol uses the nonce to ensure that the recovering + replica accepts only RECOVERYRESPONSE messages that are for this + recovery and not an earlier one." Without it, replies still in + flight from a *previous* crash-recover cycle would be accepted, and + the node could rebuild from a log that was current two crashes ago. + The paper suggests generating it from the clock, or from a counter + kept on disk — which is, amusingly, the one disk write the protocol + will admit to. + + The second condition is that the f+1 responses must **include one + from the primary of the latest view the recovering replica learns + of**. Only the primary sends a log at all (`l`, `n`, `k` are nil in + a backup's response, §4.3 step 2), so this is what makes the copied + state authoritative rather than merely majority-endorsed. + + And responders must have status `normal` — a replica in the middle + of its own view change or recovery answers nothing. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the TigerBeetle checksum point. +
Answer + + The TigerBeetle answer: Raft's Figure 2 assumes stable storage is + faithful — whatever was fsynced reads back. TigerBeetle assumes it + is not, and the VSR feature that makes that survivable is + recover-from-peers: a replica whose own log entry fails its + checksum is in the same position as a replica that never had it, + and §4.3's recovery already knows how to refill from f+1 peers. + + TigerBeetle's `docs/internals/vsr.md` cites the CTRL protocol from + Alagappan et al., *Protocol-Aware Recovery for Consensus-Based + Storage* (FAST '18), and states the consequence: a replica does not + nack a corrupt entry, "since it _might_ be the prepare being + requested". Nacking would let the cluster conclude an entry was + never accepted when it was. + + Connect to topic 5's torn page: the write that half-landed is + exactly this fault, and a single-node WAL can only detect it + (checksum) and then truncate. Replication is what lets you *repair* + it. TigerBeetle is not in this repo's pin table, so treat these as + pointers rather than pinned anchors. + +
+ ## References **Papers** -- Liskov, Cowling — "Viewstamped Replication Revisited" - (MIT-CSAIL-TR-2012-021, 2012) — the version to read; the three - sub-protocols plus the no-disk argument -- Oki, Liskov — "Viewstamped Replication: A New Primary Copy Method" - (PODC 1988) — optional; the original, for the historical claim +- Barbara Liskov, James Cowling — *Viewstamped Replication Revisited*, + MIT-CSAIL-TR-2012-021, 2012 (16 pp.) — **the version to read**. + Figure 2 is the replica state; §4.1 normal operation (f PREPAREOKs), + §4.2 view change (f others, then f+1 including self), §4.3 recovery + and the failure-independence caveat. +- Brian M. Oki, Barbara H. Liskov — *Viewstamped Replication: A New + Primary Copy Method to Support Highly-Available Distributed + Systems*, MIT LCS / PODC 1988 (10 pp.) — **a different protocol**: + transactional, cohorts, viewstamps, and it *does* write to stable + storage during a view change. Read §4.2 and the conclusions. +- Ramnatthan Alagappan et al. — *Protocol-Aware Recovery for + Consensus-Based Storage*, USENIX FAST 2018 — the CTRL protocol + TigerBeetle cites for the corrupt-entry rule in Step 5. **Code** - [tigerbeetle](https://github.com/tigerbeetle/tigerbeetle) — VSR in - production Zig, with the storage-fault model bolted on; `src/vsr/` - if you want to see the protocol shipped + production Zig, with the storage-fault model bolted on; + `src/vsr/replica.zig` and `docs/internals/vsr.md`. **Not in this + repo's pin table**, so no line anchors are given and the paths are + read at a moving `main`. diff --git a/topics/16-testing-correctness/README.md b/topics/16-testing-correctness/README.md index 078c638..3bd99e8 100644 --- a/topics/16-testing-correctness/README.md +++ b/topics/16-testing-correctness/README.md @@ -17,7 +17,7 @@ Every technique in this topic is one choice of generator + oracle: |---|---|---| | property testing | random ops | in-memory model | | DST | random ops + FAULTS + sim clock | model + invariants | -| PQS (SQLancer) | random query around a pivot row | "pivot row must appear" | +| PQS (SQLancer) | random query around a pivot row per table | "pivot row must appear" | | TLP / metamorphic | one query, three partitions | self-consistency | | fuzzing | coverage-guided byte mutation | "doesn't crash" | | Jepsen/elle | concurrent client histories | linearizability checker | @@ -82,17 +82,24 @@ The test-oracle problem: for a random query, who knows the right answer? SQLancer's insight — you don't need one. You need a second query whose result must RELATE to the first: -- **PQS** (pivoted query synthesis): pick a random existing row (the - pivot), *synthesize* a WHERE clause that evaluates TRUE on it - (rectify NULLs as you go), assert the pivot appears in the result. - Finds: expression-evaluation bugs. Needs: an expression evaluator - of your own (the cost of PQS). -- **TLP** (ternary logic partitioning): any predicate p splits rows - three ways — `p`, `NOT p`, `p IS NULL` (SQL is 3-valued!). So - `Q ≡ Q where p ∪ Q where NOT p ∪ Q where p IS NULL`. Finds: - optimizer logic bugs. Needs: nothing but a union. +- **PQS** (pivoted query synthesis): pick one existing row from *each* + table (§3.1 — not one row overall), *synthesize* a WHERE clause that + evaluates TRUE on that combination (rectify NULLs as you go), assert + the pivot appears in the result. Finds: expression-evaluation bugs — + 61 of the paper's 99 confirmed bugs, §4.2 Table 3. Needs: an + expression evaluator of your own, which is the cost of PQS and the + reason SQLancer now lists it as unmaintained (`sqlancer/README.md:80`); + the oracle is still present, in eight `Test*PQS.java` files. +- **TLP** (the paper's title is *Query Partitioning*; ternary logic is + the mechanism): any predicate p splits rows three ways — `p`, + `NOT p`, `p IS NULL` (SQL is 3-valued!). So + `Q ≡ Q where p ⊎ Q where NOT p ⊎ Q where p IS NULL`, where `⊎` is + *multiset* union — with plain set union the identity would not hold + on duplicates. Finds: optimizer logic bugs. Needs: nothing but a union. - **NoREC**: run the query optimized (`WHERE p`) and unoptimized - (`SELECT (p) FROM t` counted as booleans) — counts must match. + (`SELECT (p IS TRUE) FROM t`, summed) — counts must match. The + `IS TRUE` is load-bearing: it collapses NULL to false so the sum + counts exactly the rows `WHERE p` would return. Finds: predicate-pushdown/index bugs. ## 3. Fuzzing diff --git a/topics/16-testing-correctness/notes.md b/topics/16-testing-correctness/notes.md index b282d05..cef5d6f 100644 --- a/topics/16-testing-correctness/notes.md +++ b/topics/16-testing-correctness/notes.md @@ -61,7 +61,7 @@ Surprises / dead ends: ### FDB / Antithesis (reading-fdb-simulation.md) 1. Disk-lies vs Raft's assumptions (+ VSR/TigerBeetle answer): -2. Why BUGGIFY branches don't invalidate the test: +2. Why `buggify()` branches don't invalidate the test: 3. The four escapes (compiler/kernel/sim-bug/wall-clock) — who catches: 4. Why simulation outruns real time: 5. Our engine's remaining nondeterminism sources for M16: diff --git a/topics/16-testing-correctness/reading-fdb-simulation.md b/topics/16-testing-correctness/reading-fdb-simulation.md index c17803e..3efd56c 100644 --- a/topics/16-testing-correctness/reading-fdb-simulation.md +++ b/topics/16-testing-correctness/reading-fdb-simulation.md @@ -11,36 +11,81 @@ semantic level, and how Antithesis pushes the same determinism down to a hypervisor so unmodified systems get it for free. It's the "in the large" version of what our `dst.rs` stub does in miniature. +Two sourcing notes, because this topic has more folklore than any +other in the book. Every architectural claim below is cited to §4 +("Simulation Testing") of *FoundationDB: A Distributed Unbundled +Transactional Key Value Store* (SIGMOD 2021) — which contains, note +carefully, **no numbers at all**; it is entirely qualitative. Every +code anchor is `apple/foundationdb` at commit **`4c775a9`**, the +revision this repo pins. Where the talks and blog posts claim +figures the paper does not, this chapter says so rather than +repeating them. + ## The problem in one sentence A distributed database's worst bugs need a partition, a machine kill, and a recovery to overlap within milliseconds — an event a real test cluster might produce once a month and never again — so FDB rebuilt the system to make that event schedulable, seeded, and -replayable millions of times per night. +replayable. ## The concepts, step by step ### Step 1 — why distributed systems defeat example-based testing +> **In:** N nodes exchanging messages, each of which may be +> delayed, dropped, or reordered. +> **Out:** a space of *orderings*, not a space of inputs — and the +> dangerous orderings are the rare ones. + A distributed system's behavior depends not just on inputs but on *orderings*: which message arrived first, which node paused, whether -a disk write completed before the crash. With N nodes exchanging -messages, the number of possible interleavings explodes -combinatorially, and the dangerous ones — partition during leader -election, crash mid-recovery — are vanishingly rare on healthy -hardware. Unit tests check one ordering; production eventually -explores all of them. The gap is where the bugs live. Worse, when a -rare ordering does fail, it's gone: real clocks, real threads, and -real networks never replay. +a disk write completed before the crash. Unit tests check one +ordering; production eventually explores all of them. The gap is +where the bugs live. + +Put a number on "explodes". Take a single round in which each of `n` +nodes sends one message, and ask only how many delivery orders exist: + +``` + n = 3 nodes, 1 message each 3! = 6 enumerable + n = 5 5! = 120 enumerable + n = 5, three rounds (5!)^3 ≈ 1.7×10^6 borderline + n = 5, three rounds, each message may also be dropped + × 2^15 = 32,768 + ≈ 5.7 × 10^10 not enumerable + + and a real recovery involves hundreds of messages, not fifteen. +``` + +Worse, when a rare ordering does fail, it's gone: real clocks, real +threads, and real networks never replay. You get a stack trace and +no way back. + +Why it matters: the problem is not "we need more tests". It is that +the axis you must cover is not an axis your test framework can +address, and no amount of examples fixes that. ### Step 2 — the bet: the database and its test harness are ONE artifact -FoundationDB (2010s) decided not to bolt testing on afterward but to -design the system so the entire cluster — every node, disk, network -— runs single-threaded inside one process, scheduled by a seeded -event loop (an RNG-driven scheduler; a seed is the one number that -reproduces the whole random stream): +> **In:** permission to constrain how the production code is +> written. +> **Out:** a whole cluster that fits in one thread, whose entire +> execution is a function of one seed. + +FoundationDB decided not to bolt testing on afterward but to design +the system so the entire cluster runs single-threaded inside one +process, scheduled by a seeded event loop (an RNG-driven scheduler; +a **seed** is the one number that reproduces the whole random +stream). The paper's §4 states the constraint and the abstraction +boundary in two sentences: + +> "All database code is deterministic; … one database node is +> deployed per core." … "the simulator … abstracts away all sources +> of nondeterminism — network, disk, time, and PRNG." + +Four sources, named exactly. That list is the checklist for anything +you build yourself: ``` ┌─ one OS process, one thread ────────────────────────┐ @@ -49,24 +94,37 @@ reproduces the whole random stream): │ SimNetwork — seeded delays, drops, PARTITIONS │ │ SimDisk — seeded corruption, torn writes, │ │ "disk that lies" (bit rot) │ - │ + BUGGIFY(p) — code-embedded chaos macros │ + │ + buggify() — code-embedded chaos, Step 5 │ └──────────────────────────────────────────────────────┘ ``` One thread means no OS scheduler in the picture — every interleaving of "concurrent" events is chosen by the simulator's RNG, so a u64 seed reproduces a whole-cluster failure, including the partition -timings. (Our topic 15 sim.rs is this in the small.) +timings. (Our topic 15 `sim.rs` is this in the small.) The +production build swaps the same interfaces for real ones: §4 says +"The production implementation is a simple shim to the relevant +system calls" — the *simulated* implementation is the elaborate one. + +Why it matters: this is the only step that costs anything. Steps 3 +through 6 are what you get for free once you have paid it. ### Step 3 — the mechanism: a seeded event loop over a time-ordered heap +> **In:** a seed and a set of pending events. +> **Out:** one exact execution — and, for IO-bound work, one that +> completes faster than the wall-clock interval it simulates. + Strip the architecture to its core and it is a priority queue of -future events plus one macro. The "cluster" advances by popping the -next event; logical time *teleports* to that event's timestamp — +future events plus one predicate. The "cluster" advances by popping +the next event; logical time *teleports* to that event's timestamp — nothing ever sleeps: -```rust -// the "cluster" advances by popping the next event — no threads, no sleeps +```text +// ILLUSTRATION — the shape of a seeded event loop, not quoted from +// FoundationDB (whose real version is Flow's Net2 runner). The real +// per-site chaos predicate is flow/include/flow/Buggify.h:92-96, +// quoted verbatim in Step 5. fn run(seed: u64) { let mut rng = ChaCha8Rng::seed_from_u64(seed); let mut events = BinaryHeap::new(); // min-heap on fire_time @@ -77,72 +135,286 @@ fn run(seed: u64) { } } } - -fn buggify(rng: &mut impl Rng, p: f64) -> bool { - cfg!(simulation) && rng.random_bool(p) // rare paths made common; -} // compiled out in production ``` Because time jumps instead of passing, an IO-bound workload runs -*faster than real time* — a simulated 30-second recovery costs -however long the CPU takes to process its events, often -milliseconds. That's how the famous claim works: the simulator ran -*millions of cluster-years* of compressed chaos before release, -which is why FDB found so few bugs in production. +*faster than real time*. Work the ratio with this topic's own +measured harness, which is the same idea at small scale: + +``` + crash_matrix (this topic, notes.md baseline): + 5,000 seeds × 40 ops, five bug variants + wall clock ≈ 0.02 s per 5,000-seed sweep + → ≈ 200,000 simulated crash-recoveries per second + + a real crash-recovery on real hardware: ≈ 1 s (process restart + WAL replay) + speedup ≈ 200,000 × — i.e. one second of simulator ≈ 2.3 days of cluster + + the ratio is not magic: it is the fraction of "elapsed" time that was + a sleep. Replace sleeps with a heap pop and that time costs nothing. +``` + +Now the discipline. The famous claim is that FDB simulated *millions +of cluster-years* before release. **That number is not in the +paper** — §4 is entirely qualitative — so this chapter does not +assert it. What the paper *does* claim, in §6.2 and with a named +deployment, is checkable: + +> CloudKit deployed FoundationDB for "more than 0.5M disk years +> without a single data corruption event." + +Disk years of *production*, not simulated cluster-years. That is the +number to quote. + +Why it matters: the speedup is the entire economic argument for DST, +and it is computable from your own harness rather than borrowed from +a conference talk. ### Step 4 — Flow: making the language deterministic +> **In:** C++, which has threads, blocking syscalls, and no way to +> stop you using them. +> **Out:** a dialect in which the scheduler is the only thing that +> can make progress. + The event loop only works if no code path can escape it — no pthreads, no blocking syscalls in the data path. Flow is FDB's C++ -dialect built for exactly this: **actors** (independent state -machines that communicate only by messages) and **futures** (values -that arrive later) compile down to deterministic state machines, and -every `wait()` yields control back to the simulator's scheduler -instead of blocking a thread. The same discipline raft-rs reaches by -being sans-io (reading-raft-rs.md): logic that never touches the -outside world directly can be driven by anything — including a -seeded heap. +dialect built for exactly this. §4's description: + +> "a novel syntactic extension to C++ adding async/await-like +> concurrency primitives" + +**Actors** (independent state machines that communicate only by +messages) and **futures** (values that arrive later) compile down to +deterministic state machines, and every `wait()` yields control back +to the simulator's scheduler instead of blocking a thread. The same +discipline raft-rs reaches by being sans-io +([reading-raft-rs.md](../15-replication-consensus/reading-raft-rs.md)): logic +that never touches the outside world directly can be driven by +anything — including a seeded heap. The cost is total: FDB rewrote itself in a private language to buy -determinism. Hold that price — Step 7's table is about who else pays -how much. +determinism. And the paper is honest that the boundary of the +rewrite is the boundary of the testing (§4, Limitations): + +> simulation "cannot test the performance of the real system"; it +> "cannot test third-party libraries or code that is not written in +> Flow"; and "several bugs have resulted from the true operating +> system contract being weaker than it was believed to be." + +That last clause is the one to remember: a simulated disk implements +the fsync contract *you believe in*. If the kernel's is weaker, your +simulator is wrong in exactly the direction that hurts. + +Hold that price — Step 7's table is about who else pays how much. -### Step 5 — BUGGIFY: the SUT cooperates with its tester +Why it matters: "make it deterministic" is not a code style. It is a +language-level property, and every project in this topic pays for it +at a different layer. + +### Step 5 — buggify: the SUT cooperates with its tester + +> **In:** a rare branch in production code that a black-box tester +> could never reach on demand. +> **Out:** a one-token annotation that makes that branch common — in +> simulation only, and reproducibly per seed. Fault injection from outside (kill a process, return EIO from a syscall) only reaches the failures the environment can express. -BUGGIFY goes further: ~800 macros *inside* the FDB codebase that, in -simulation only, make rare paths common — "pretend the buffer is -full", "return commit_unknown_result", "trigger recovery now". The -system under test cooperates with the tester by exposing its own -rare branches as injectable events, at the semantic level where the -interesting states live. Question: why is injecting at the semantic -level (commit_unknown_result) more powerful than at the syscall -level (EIO)? +"Buggification", the paper's own word (§4), goes further: annotations +*inside* the FDB codebase that, in simulation only, make rare paths +common. The system under test cooperates with the tester by exposing +its own rare branches as injectable events, at the semantic level +where the interesting states live. + +First correction to the folklore: **there is no `BUGGIFY` macro at +`4c775a9`.** It is now a function, and reading it repays the minute +it takes: + +```c +// flow/include/flow/Buggify.h — the per-site activation macro and buggify(), 51-96 (elided) + 51 #define __GENERATE_BUGGIFY_VARIABLES(TYPE, Type, type) \ + 52 inline double P_##TYPE##_BUGGIFIED_SECTION_ACTIVATED{ 0.25 }; \ + 53 inline double P_##TYPE##_BUGGIFIED_SECTION_FIRES{ 0.25 }; \ + 54 inline double P_##TYPE##_ENABLED{ false }; \ + 55 inline std::unordered_map Type##_SBVars; \ +// ... 56-67: is/enable/disable/clear accessors over P_##TYPE##_ENABLED ... + 68 inline bool get##Type##SBVar(const char* file, const int line) { \ + 69 const BuggifySection section{ file, line }; \ + 70 const auto sectionItr = Type##_SBVars.find(section); \ + 71 if (sectionItr != Type##_SBVars.end()) [[likely]] { \ + 72 return sectionItr->second; \ + 73 } \ + 75 const double rand = deterministicRandom()->random01(); \ + 76 const bool activated = rand < P_##TYPE##_BUGGIFIED_SECTION_ACTIVATED; \ + 77 Type##_SBVars.emplace(section, activated); \ + 78 g_traceBatch.addBuggify(activated, line, file); \ +// ... 79-84: dump the trace, return activated ... + 92 inline bool buggify(double probability = P_GENERAL_BUGGIFIED_SECTION_FIRES, + 93 const std::source_location location = std::source_location::current()) { + 94 return isGeneralBuggifyEnabled() && getGeneralSBVar(location.file_name(), static_cast(location.line())) && + 95 deterministicRandom()->random01() < probability; + 96 } +``` + +Three conditions on line 94–95, and they are not the same condition: + +1. `isGeneralBuggifyEnabled()` — the global switch (`:54`, default + `false`, so buggify is inert outside simulation). +2. `getGeneralSBVar(file, line)` — a **per-site, memoized** coin. + Line 70–73 looks the site up in a map keyed by `(file, line)` + (`BuggifySection`, `:38-43`); only on first encounter does line + 75–76 flip a coin at `P_GENERAL_BUGGIFIED_SECTION_ACTIVATED` + (0.25) and *remember it for the whole run*. +3. `deterministicRandom()->random01() < probability` — a fresh coin + per *call*, defaulting to `P_GENERAL_BUGGIFIED_SECTION_FIRES` + (0.25). + +So compute the odds an unconditional buggify site fires on a given +execution: + +``` + P(site activated for this run) = 0.25 (memoized once, line 76) + P(a given call fires | activated) = 0.25 (line 95) + + P(a given call fires) = 0.25 × 0.25 = 0.0625 = 1 in 16 + + but activation is per RUN, not per call, so over many calls in ONE run: + 3 runs in 4: the site NEVER fires, however often it is reached + 1 run in 4: the site fires on ~1 call in 4 + + EXPENSIVE_VALIDATION (:98) is different: P_EXPENSIVE_VALIDATION = 0.05 + (:36) with NO memoization — a fresh 1-in-20 coin every single call. +``` + +That two-level structure is the whole idea, and it is Step 6's +**swarm testing** in miniature: a run in which a site is *never* +buggified explores the normal path deeply; a run in which it *is* +explores the rare path repeatedly. Flipping the coin per call would +give every run the same shallow mixture. + +Second correction: how many sites are there? "About 800" is the +folklore figure and it is not in the paper. What is checkable is the +tree. Grepping `buggify(` across `fdbserver/` at `4c775a9` returns +**369 call sites**, of which **246 are in a single file**, +`fdbserver/core/ServerKnobs.cpp`. That file is not a fault injector +at all — it is the tuning-parameter randomizer: + +```c +// fdbserver/core/ServerKnobs.cpp — a representative knob, 164 + 164 init( MAX_COMMIT_BATCH_INTERVAL, 2.0 ); if( randomize && buggify() ) MAX_COMMIT_BATCH_INTERVAL = 0.5; // ... +``` + +Read that line twice. Two thirds of FDB's buggify sites exist to +make *tuning constants* wrong on purpose, which is §4's point that +"randomization of tuning parameters also ensures that specific +performance tuning values do not accidentally become necessary for +correctness". The remaining ~123 sites across `storageserver`, +`TLogServer`, `VersionedBTree`, `DiskQueue`, `CoordinatedState` and +the workloads are the semantic fault injectors people mean when they +say BUGGIFY. + +Question: why is injecting at the semantic level +(`commit_unknown_result`) more powerful than at the syscall level +(EIO)? + +Why it matters: the memoization at line 77 is the difference between +a chaos monkey and a search strategy, and it is four lines of code. ### Step 6 — oracles as workloads: assert invariants, not outputs -With chaos injected, who decides a run failed? Not expected outputs -— nobody knows the "right answer" of a randomized cluster-year. FDB -ships **workloads** that assert *invariants* (properties that must -hold in every legal execution): a read at version v sees all commits -≤ v; the cluster recovers to availability after any tolerated fault -set; swizzled clogging (partition, then heal in random order) never -loses acked data; machine kills mid-recovery never fork history. -Dumb sanity workloads plus invariants beat clever expected-value -tests because they stay valid under any interleaving — this is the -generator + oracle framing of the topic README, at cluster scale. +> **In:** a randomized cluster-hour with faults injected throughout. +> **Out:** a verdict — from invariants, because nobody knows the +> "right answer". + +With chaos injected, who decides a run failed? Not expected outputs. +FDB ships **workloads** that assert *invariants* (properties that +must hold in every legal execution). §4 defines the class precisely: + +> "the test oracle … verifies invariants that can only be maintained +> through proper atomicity and isolation", plus checks that the +> cluster recovers within a set time. + +Concretely: a read at version v sees all commits ≤ v; the cluster +recovers to availability after any tolerated fault set; swizzled +clogging (partition, then heal in random order) never loses acked +data; machine kills mid-recovery never fork history. Dumb sanity +workloads plus invariants beat clever expected-value tests because +they stay valid under any interleaving — this is the generator + +oracle framing of the topic README, at cluster scale. + +The fault menu §4 names is worth copying verbatim into your own +design doc: machine, rack and datacenter **fail-stop failures and +reboots**; network faults, **partitions**, and latency; disk +**corruption of unsynchronized writes on reboot**; and randomized +event times. Then the sentence everyone skips: + +> "Fault injection distributions are carefully tuned to avoid +> driving the system into a small state-space caused by an excessive +> fault rate." + +Too much chaos is *worse* than too little — a cluster that is always +partitioned only ever exercises the "we are partitioned" path. Your +fault probability is a tuning parameter with an interior optimum, +not a dial to turn to 11. This topic's own `crash_matrix` is the +same lesson from the other side: at a 10% crash rate over 40 ops the +`TornWriteAccepted` bug is caught in only 2,442 of 5,000 seeds +(48.8%) while `NoSyncOnCommit` is caught in 4,980 (99.6%) — same +harness, same fault rate, an order-of-magnitude difference in how +often you find out. + +The randomization is *coordinated*, which the paper calls **swarm +testing** (citing Groce et al.): + +> "each cluster is randomly configured with different cluster sizes, +> configurations, workloads, fault injection parameters, tuning +> parameters, and enables and disables a different random subset of +> buggification points." + +That last clause is Step 5's `P_GENERAL_BUGGIFIED_SECTION_ACTIVATED` +in prose: each run gets a *different subset* of chaos, not the +average of all of it. + +Coverage is tracked at the same granularity, with a macro whose +literal form the paper gives: + +``` + TEST( buffer.is_full() ); // buffer is full +``` + +— which counts, in the paper's words, "the number of distinct +simulation runs" that reached the condition. Not lines. Not +branches. *Runs that reached the interesting state*, which is the +only coverage metric that means anything once every run is a +different configuration. + +Why it matters: this step is the transferable one. You will not +write Flow; you will absolutely write invariant workloads and a +fault-rate tuning curve. ### Step 7 — Antithesis: buy determinism at the hypervisor instead +> **In:** a system you are not allowed to rewrite. +> **Out:** the same reproducibility, purchased at a lower layer for +> a different price. + Same founders, next act: if you can't rewrite your system in Flow, put the WHOLE VM under a deterministic hypervisor — every syscall, interrupt, and thread interleaving is recorded and replayable, so *unmodified* binaries get FDB-grade reproducibility. On top, coverage-guided exploration ("multiverse debugging" — fork the simulation at interesting branch points and explore the divergent -universes) decides which random branches to push deeper. turso runs -its Dockerfile.antithesis image there. +universes) decides which random branches to push deeper. + +turso runs there, and you can check it rather than take it on faith: +`Dockerfile.antithesis` at the repo root and +`.github/workflows/antithesis.yml` (a scheduled run with a +240-minute default duration and an optional `diff_base` for +targeted coverage), driving the workloads in +`testing/antithesis/bank-test/` and +`testing/antithesis/stress-composer/`, with +`scripts/antithesis/diff_to_targeted_coverage.py` turning a diff into +a coverage target. The whole design space is one table — determinism boundary vs rewrite cost: @@ -157,20 +429,46 @@ rewrite cost: ``` Lower boundary = more of the world captured (Antithesis catches -thread races Flow defines away); higher boundary = cheaper to adopt -but more nondeterminism left uncorralled. +thread races Flow defines away, and — per Step 4's Limitations — +third-party libraries Flow cannot see); higher boundary = cheaper to +adopt but more nondeterminism left uncorralled. + +One measured data point on why the boundary matters, from the +paper's §6.2: FDB originally used Zookeeper for coordination, and +"fault injection found two independent bugs (circa 2010)" in it — +after which Zookeeper was deleted and replaced with a de novo Paxos +implementation *written in Flow*. The lesson is not that Zookeeper +was bad; it is that the moment a component sits outside your +determinism boundary, you either move it inside or stop testing it. + +Why it matters: this is the decision you actually face on your own +codebase, and the table's second column is what your team will +argue about. ## How to read the sources (with the concepts in hand) -1. **FDB "Simulation and Testing" + "Testimony" docs** — the design - philosophy in the authors' words. Read with Steps 2–3 in hand: - every section is either the event loop, BUGGIFY (Step 5), or a - workload oracle (Step 6). No clone needed. -2. **`flow/README.md`** in the FDB repo — skim for the - `wait()`-yields-to-scheduler discipline (Step 4) rather than the - C++ details; the point is what a language must give up to be - simulatable. -3. **Antithesis blog** — read one or two posts for the +1. **The SIGMOD 2021 paper, §4 ("Simulation Testing")** — three + pages, and the authoritative source for every claim in Steps 2, + 4, 5 and 6. Read it noticing what is *not* there: no bug counts, + no cluster-years, no coverage percentages. Anyone quoting a + number and citing "the FoundationDB paper" is quoting something + else. +2. **FDB "Simulation and Testing" / "Testimony" docs** — the same + philosophy with more colour and less rigour; use them for + intuition, not for figures. +3. **`flow/include/flow/Buggify.h`** (133 lines) — read it all. + Step 5 walks the two-level coin; also note `EXPENSIVE_VALIDATION` + (`:98`) and the separate `CLIENT_BUGGIFY` axis (`:100-102`). +4. **`flow/include/flow/CodeProbe.h`** and + **`flow/SimBugInjector.cpp`** — the modern successors to the + paper's `TEST()` macro and the hand-rolled injectors. +5. **`fdbserver/core/ServerKnobs.cpp`** — skim any 40 lines. This + is where two thirds of the buggify sites live, and seeing that + they are knob randomizers rather than fault injectors permanently + fixes the mental model. +6. **`flow/README.md`** — skim for the `wait()`-yields-to-scheduler + discipline (Step 4) rather than the C++ details. +7. **Antithesis blog** — read one or two posts for the deterministic-hypervisor claim and multiverse debugging (Step 7); map every capability they advertise onto the table above. @@ -193,25 +491,199 @@ but more nondeterminism left uncorralled. ## Done when +Answer each before unfolding it. + - [ ] You can explain why example-based tests lose to distributed systems, in terms of interleaving count. + +
Answer + + Because the axis that matters is *ordering*, not input. Five nodes + sending one message each gives `5! = 120` delivery orders; three + such rounds gives `(5!)^3 ≈ 1.7 × 10^6`; allow each of those + fifteen messages to also be dropped and it is `× 2^15`, about + `5.7 × 10^10`. A real recovery involves hundreds of messages. + + An example-based test pins one point in that space — whichever one + your machine happened to produce. And the failing point is not + recoverable afterwards: real clocks, threads and networks do not + replay, so even a production failure gives you a stack trace and no + way back to it. + +
+ - [ ] You can state the bet: the database and its test harness are one artifact, and say what that forbids in the production code. + +
Answer + + The bet: make the *production* code deterministic so the whole + cluster can run inside one thread under a seeded scheduler. §4: + "All database code is deterministic; … one database node is + deployed per core." + + What it forbids: threads in the data path, blocking syscalls, and + any direct use of the four nondeterminism sources §4 names — + **network, disk, time, and PRNG**. Every one of those goes through + an interface whose production implementation is, per §4, "a simple + shim to the relevant system calls" and whose simulated + implementation is the interesting one. + + It also forbids third-party code in the data path — which is why + §4's Limitations concede simulation "cannot test third-party + libraries or code that is not written in Flow", and why Zookeeper + was eventually replaced by a Paxos implementation in Flow (§6.2). + +
+ - [ ] You can describe the seeded event loop over a time-ordered heap, and say why simulation runs *faster* than real time. -- [ ] You can explain what BUGGIFY is and give the argument for why compiling it out of production is not cheating. + +
Answer + + A min-heap of `(fire_time, event)`. Pop the earliest, set logical + time *to* it, process it, push whatever follow-up events it + generates. Nothing ever sleeps and nothing ever blocks, so the only + cost is CPU spent processing events. + + It runs faster than real time exactly to the extent that the + workload was waiting rather than computing. A simulated 30-second + recovery that is 99.9% waiting costs 30 ms of CPU. + + This topic's own harness measures the ratio: `crash_matrix` sweeps + 5,000 seeds × 40 ops in about 0.02 s, roughly 200,000 simulated + crash-recoveries per second, against a wall-clock crash-recovery of + order one second. Note the corollary: for a *CPU-bound* workload + the speedup is 1× or worse, because there was no waiting to delete. + +
+ +- [ ] You can explain what buggify is and give the argument for why compiling it out of production is not cheating. + +
Answer + + It is an in-source annotation that, in simulation only, takes a + rare branch. At `4c775a9` it is a function, not a macro: + `buggify(probability, source_location)` at + `flow/include/flow/Buggify.h:92-96`, gated on three conditions — + the global enable (`:54`, default `false`), a memoized per-`(file, + line)` activation coin at 0.25 (`:68-84`), and a per-call firing + coin at 0.25 (`:95`). + + Why it is not cheating: buggify never adds behaviour, it only + *selects among behaviours the production code already contains*. + `MAX_COMMIT_BATCH_INTERVAL = 0.5` (`ServerKnobs.cpp:164`) is a + legal value of a legal knob; `commit_unknown_result` is a status + the client must already handle. The branch taken under buggify is + a branch production can take — it is simply one that needs a + once-a-year coincidence to reach. + + The honest caveat is the paper's own (§4, Limitations): "several + bugs have resulted from the true operating system contract being + weaker than it was believed to be." Buggify explores *your model* + of the rare paths. If the model is wrong, so is the exploration. + +
+ +- [ ] You can compute how often an unconditional buggify site fires, and explain why the odds are structured in two levels rather than one. + +
Answer + + `0.25 × 0.25 = 0.0625`, one call in sixteen — but that flat number + hides the structure. `P_GENERAL_BUGGIFIED_SECTION_ACTIVATED` (0.25, + `Buggify.h:52`) is drawn **once per site per run** and memoized in + `General_SBVars` keyed by `(file, line)` (`:68-77`). + `P_GENERAL_BUGGIFIED_SECTION_FIRES` (0.25, `:53`) is drawn per + call (`:95`). + + So in three runs out of four the site never fires no matter how + often it is reached; in the fourth it fires on about one call in + four. Two levels, not one, because that is **swarm testing**: §4 + says each run "enables and disables a different random subset of + buggification points". A single per-call coin would give every run + the same thin mixture of chaos and explore nothing deeply. + + Contrast `EXPENSIVE_VALIDATION` (`:98`), which uses + `P_EXPENSIVE_VALIDATION = 0.05` (`:36`) with **no** memoization — + a flat 1-in-20 per call, because it is a check, not a behaviour + change, and there is nothing to explore deeply. + +
+ - [ ] You can name three bug classes simulation provably cannot catch. + +
Answer + + §4's Limitations gives three directly: + + 1. **Performance bugs** — "cannot test the performance of the real + system", because logical time is not real time and the whole + point of Step 3 is that waiting costs nothing. + 2. **Anything outside the determinism boundary** — "cannot test + third-party libraries or code that is not written in Flow". + Zookeeper (§6.2) is the worked example. + 3. **Wrong assumptions about the environment** — "several bugs have + resulted from the true operating system contract being weaker + than it was believed to be." The simulated disk implements the + fsync semantics you coded, not the ones your kernel has. + + Add a fourth that follows from Step 2: a bug in the simulator + itself is invisible, because the simulator is the oracle's notion + of reality. This is why turso runs `crash_matrix`-style harnesses + *and* Antithesis *and* a real-hardware suite. + +
+ - [ ] You wrote answers to all five questions in notes.md, including which of our IO traits already sit in the right place for M16. +
Answer + + No unfoldable answer — this one is the writing. Use §4's four-item + list as the audit checklist: **network, disk, time, PRNG**. For + question 5, the ones people forget are the last two: `HashMap` + iteration order (Rust randomizes the seed per process), and any + `rand::thread_rng()` reachable from a plan or a hash table. The + thread pool from M9 is the expensive one, because corralling it + means the M9 work has to become sans-io or single-threaded under a + flag — which is Step 4's "rewrite cost" column arriving on your own + codebase. + +
+ ## References **Papers & docs** +- Zhou et al. — "FoundationDB: A Distributed Unbundled Transactional + Key Value Store" (SIGMOD 2021) — **§4 is the authoritative source** + for the determinism constraint, the four nondeterminism sources, + Flow, buggification, swarm testing, the `TEST()` coverage macro, + and the Limitations; §6.2 for CloudKit's 0.5M disk years and the + Zookeeper replacement; §1 for the `f+1` replication choice - FoundationDB — "Simulation and Testing" + "Testimony" docs ([apple.github.io/foundationdb](https://apple.github.io/foundationdb/testimony.html)) - — the design-philosophy source; no clone needed + — intuition, not figures - Antithesis blog ([antithesis.com/blog](https://antithesis.com/blog)) — by the FDB founders; the deterministic-hypervisor generalization and "multiverse debugging" -**Code** -- [foundationdb](https://github.com/apple/foundationdb) — - `flow/README.md` — the Flow language: actors + futures compiled to - deterministic state machines; skim for the `wait()`-yields-to- - scheduler discipline rather than the C++ details +**Code** — [foundationdb](https://github.com/apple/foundationdb) @ +`4c775a9` + +| File | Lines | What | +|---|---|---| +| `flow/include/flow/Buggify.h` | 36 | `P_EXPENSIVE_VALIDATION{0.05}` — no memoization | +| `flow/include/flow/Buggify.h` | 38-49 | `BuggifySection{file, line}` and its hash — the memo key | +| `flow/include/flow/Buggify.h` | 51-84 | the per-axis variable generator: activation 0.25, fires 0.25, memoized `get*SBVar` | +| `flow/include/flow/Buggify.h` | 92-96 | `buggify()` — the three-condition predicate | +| `flow/include/flow/Buggify.h` | 98, 100-102 | `EXPENSIVE_VALIDATION` and the separate `CLIENT_BUGGIFY` axis | +| `fdbserver/core/ServerKnobs.cpp` | 164 | a representative knob randomizer — 246 of the tree's 369 `buggify(` sites live in this file | +| `flow/include/flow/CodeProbe.h` | — | the modern successor to the paper's `TEST()` coverage macro | +| `flow/SimBugInjector.cpp` | — | simulator-side injection | +| `flow/include/flow/DeterministicRandom.h` | — | the seeded PRNG every coin above draws from | +| `flow/README.md` | — | Flow: actors + futures compiled to deterministic state machines | + +**Code** — [turso](https://github.com/tursodatabase/turso) @ `dd775bc` + +| File | What | +|---|---| +| `Dockerfile.antithesis` | the image Antithesis runs | +| `.github/workflows/antithesis.yml` | scheduled run, 240-minute default, optional `diff_base` | +| `testing/antithesis/bank-test/`, `stress-composer/` | the workloads, i.e. Step 6's oracles | +| `scripts/antithesis/diff_to_targeted_coverage.py` | turns a diff into a coverage target | diff --git a/topics/16-testing-correctness/reading-jepsen.md b/topics/16-testing-correctness/reading-jepsen.md index 4026447..87ae03e 100644 --- a/topics/16-testing-correctness/reading-jepsen.md +++ b/topics/16-testing-correctness/reading-jepsen.md @@ -11,23 +11,33 @@ then routes you through two reports worth reading in full: Redis-Raft (the catalog of consensus-plumbing bugs) and Dgraph (the graph-DB cautionary tale). +Every number below is quoted from a primary source: the Elle paper +(Kingsbury & Alvaro, VLDB 2020, `arXiv:2003.10554`) with its section +number, or the Jepsen report itself with the issue number Redis Labs +or Dgraph filed. There is no `elle` clone in this repo's pin table, +so the code anchors are turso's own Elle integration at commit +`dd775bc` — which emits Elle's EDN and hands it to `elle-cli`. + ## The problem in one sentence Databases routinely claim "serializable" or "linearizable" and lose acked writes the first time a network partition lands mid-failover — -Jepsen's redis-raft analysis alone found acked-write loss in a -system built directly on the Raft paper's math. +and Jepsen's Redis-Raft analysis found 21 issues, five of them +losing committed updates, in a system built directly on the Raft +paper's math. ## The concepts, step by step ### Step 1 — the method: real cluster, real faults, recorded history -Jepsen is black-box testing: it needs no source code, no -instrumentation — just client access to an unmodified binary running -on a real cluster. It spawns concurrent clients issuing operations, -while a **nemesis** process injects real environmental faults, and -records everything into a **history** — a timestamped log of every -operation's start, end, and result: +> **In:** an unmodified binary running on a real cluster, plus +> client access. No source, no instrumentation. +> **Out:** a **history** — a timestamped log of every operation's +> invocation, completion, and result. + +Jepsen is black-box testing. It spawns concurrent clients issuing +operations while a **nemesis** process injects real environmental +faults, and records everything: ``` generators → concurrent client ops (read/write/cas/txn) @@ -41,21 +51,79 @@ operation's start, end, and result: Note the fault menu is topic 15's failure catalog made physical: iptables rules for partitions, SIGSTOP for the process that's alive but not responding (the GC-pause / VM-migration stand-in a crash -doesn't model). +doesn't model). The Redis-Raft test design section names the exact +menu — "process pauses, crashes, network partitions, clock skew, and +membership changes" — on "five-node Debian 9 clusters, on both LXC +and EC2". + +The three-part record per operation is the load-bearing detail. An +operation has an **invocation** time, a **completion** time, and an +outcome that may be `ok`, `fail` (definitely did not happen), or +`info` (**indeterminate** — the client never learned). Indeterminate +is not a nuisance; it is the normal outcome of a partition, and any +checker that cannot represent it will either miss bugs or invent +them. + +Why it matters: everything Jepsen finds, it finds because it refused +to trust the system's own account of what happened. The history is +the only evidence. ### Step 2 — the checker problem: verifying a history is the hard part +> **In:** a recorded history of `n` operations with `c` of them +> concurrent at any moment. +> **Out:** a verdict — reachable only by searching orderings, and +> the search is exponential. + **Linearizability** (every operation appears to take effect -atomically at some instant between its start and end) sounds -checkable — but given a history of concurrent operations, deciding -whether *any* legal ordering explains it is NP-complete in general: -each concurrent window multiplies the orderings to try. Jepsen's -first checker, Knossos, did exactly this search and exploded on long -histories — histories had to stay short, which is the opposite of -what fault-finding wants. elle is the escape. +atomically at some instant between its invocation and completion) +sounds checkable — but given a history of concurrent operations, +deciding whether *any* legal ordering explains it is NP-complete in +general. The Elle paper §1 states the same for the isolation side: +"Serializability checking is also (in general) NP-complete." + +Jepsen's first checker, Knossos, did exactly this search. Work the +cost, using the paper's own framing — "given c concurrent +transactions, the number of permutations to evaluate is c!": + +``` + c = 10 concurrent txns 10! = 3,628,800 feasible + c = 15 15! ≈ 1.3 × 10^12 hours + c = 20 20! ≈ 2.4 × 10^18 no + + measured (§7.5), 24-core Xeon / 128 GB, 100 s runtime cap: + Knossos "often timed out or ran out of memory after a few + hundred transactions" + "many Knossos runs involved search spaces on the order of 10^24" + "With 40+ concurrent processes, even histories of 5000 + transactions were (generally) uncheckable" +``` + +An earlier attempt using the Gecode constraint solver fared no +better: "Histories of more than a hundred-odd transactions quickly +become intractable" (§1). + +Histories therefore had to stay short — which is the opposite of +what fault-finding wants, because a partition takes seconds to land +and the interesting interleavings are rare. elle is the escape, and +the measured gap is stark: Elle "checked hundreds of thousands of +transactions in tens of seconds" under the same 100-second cap, and +is "primarily linear in the length of a history" (§7.5). The +Redis-Raft report describes it the same way: "a new type of +consistency checker, which operates in linear (rather than +exponential) time". + +Why it matters: the checker's complexity is what caps how long you +can run a test, and how long you can run a test is what caps which +bugs you can find. This is a *performance* constraint on a +*correctness* tool. ### Step 3 — elle's trick: design the workload so dependencies are visible +> **In:** freedom to choose what operations the clients issue. +> **Out:** a workload whose *results* directly reveal the +> dependency edges, so nothing has to be searched for. + Don't check arbitrary histories — DESIGN the operations so the outcome itself reveals what ordered what. elle's workload is **list-append**: every write is `append(k, v)` with a globally @@ -68,60 +136,234 @@ inferred). Plain registers (get/set of a single value) hide all of this — each write destroys the evidence of the previous one; lists keep the whole lineage. +Count what one read buys, for a key whose list has grown to length +`n`: + +``` + read of k = [v1, v2, ..., vn] + + ww edges recovered n − 1 (v1→v2, v2→v3, … : the list IS the write order) + wr edges recovered n (this txn read every one of those writes) + rw edges implied ≥ 1 (any later appender to k comes after this read) + + n = 20 → 19 + 20 = 39 dependency facts from ONE read + + same read against a register: + ww edges recovered 0 (the previous value is gone) + wr edges recovered 1 (you saw *someone's* write; which one is ambiguous + unless values are unique) +``` + +That ratio — 39 to 1 — is the whole technique. The paper's +recoverability property is what makes it sound: because appends are +unique and lists are never overwritten, the *version order* of each +key is directly readable off the data, rather than being something +the checker must guess. + +The concrete data format is short enough to read in full. turso's +simulator implements the Jepsen side of this to feed `elle-cli`: + +```rust +// simulator/testing/concurrent-simulator/elle.rs — ElleOp and to_edn, 18-67 (elided) + 18 pub enum ElleOp { + 19 /// Append a value to a list identified by key (list-append model) + 20 Append { key: String, value: i64 }, + 21 /// Read a list by key, result is None before execution and Some after (list-append model) + 22 Read { + 23 key: String, + 24 result: Option>, + 25 }, +// ... 26-29: Write / RwRead — the weaker rw-register model, kept for comparison ... + 30 } +// ... 32-35: doc comment giving the two target forms ... + 36 pub fn to_edn(&self) -> String { + 37 match self { + 38 ElleOp::Append { key, value } => { + 39 format!("[:append \"{}\" {}]", escape_edn_string(key), value) + 40 } + 41 ElleOp::Read { key, result } => { + 42 let result_str = match result { + 43 None => "nil".to_string(), +// ... 44-53: Some(vals) → "[1 2 3]", empty → "[]" ... + 54 format!("[:r \"{}\" {}]", escape_edn_string(key), result_str) + 55 } +``` + +Line 24 is the one to stare at: `result: Option>` — the +`Option` is Step 1's "not yet completed", and the `Vec` is Step 3's +whole point. The `nil` at line 43 is how an *invoked but +uncompleted* read is written down; the checker needs to see the +invocation even when the result never arrived. + +`ElleEventType` at `:72-79` carries the other half — `Invoke`, `Ok`, +`Fail`, `Info` — which is exactly Step 1's three outcomes plus the +invocation record. + +Question: why do unique values + list semantics make wr/ww edges +*directly observable* where plain registers hide them? + +Why it matters: this is a *test design* insight, not an algorithm. +The exponential search in Step 2 didn't get a better algorithm; it +got deleted by choosing a different workload. + ### Step 4 — the serialization graph: a cycle IS an anomaly +> **In:** the ww / wr / rw edges recovered in Step 3, plus real-time +> ordering from Step 1's timestamps. +> **Out:** a directed graph — and any cycle in it is a named +> anomaly, found in near-linear time. + Collect those ww/wr/rw facts into a directed graph over transactions (the **serialization graph** — an edge T1 → T2 means T1 must come before T2 in any serial order). If the graph has a cycle, no serial -order exists — and the cycle's *edge types* name the anomaly from -the isolation literature: G0 (dirty write, ww cycle), G1c (cyclic -information flow), G-single (read skew), pure-rw cycles (write -skew). The whole checker, structurally: +order exists — and the cycle's *edge types* name the anomaly. The +paper's §6 gives the taxonomy precisely, and it is by *composition +of edge types around the cycle*: -```rust -// a read of k = [1, 3] by txn T makes dependency edges OBSERVABLE: -fn check(history: &History) -> Result<(), Cycle> { - let mut g = Graph::new(); - for read in history.reads() { - for w in read.list.windows(2) { - g.add(writer(w[0]), writer(w[1]), Ww); // list order = write order - } - if let Some(&last) = read.list.last() { - g.add(writer(last), read.txn, Wr); // T saw last's write - } - // and T -> writer(v) for any v appended after: an rw anti-dep - } - g.find_cycle() // a cycle = an anomaly; its edge types NAME it -} +| cycle | edges it contains | classic name | +|---|---|---| +| **G0** | **all** ww | dirty write / write cycle | +| **G1c** | ww or wr (at least one wr, no rw) | cyclic information flow | +| **G-single** | **exactly one** rw, rest ww/wr | read skew | +| **G2** | **one or more** rw | anti-dependency cycle (incl. write skew) | + +Note G-single is not "a cycle with rw edges" — it is a cycle with +*exactly one*, which is what makes it a distinct and much more +common finding than general G2. Elle also reports non-cycle +anomalies §6.1 names directly: garbage reads, duplicate writes, and +internal inconsistency (a transaction not seeing its own earlier +writes), plus §4.3.1's aborted read, intermediate read, and dirty +update. + +The paper's claim for the whole scheme, from the abstract: Elle "can +detect every anomaly in Adya et al's formalism (except for +predicates)". + +Detection is **Tarjan's strongly-connected-components algorithm** +followed by a BFS within each SCC to extract a short, human-readable +cycle (§6). Both are near-linear: + +``` + Knossos: O(c!) in the concurrency c — Step 2's 10^24 + Elle: Tarjan SCC O(V + E) + + BFS per SCC O(V + E) + ≈ linear in the number of transactions and edges + + measured (§7.5, same 100 s cap, 24-core Xeon): + Knossos a few hundred transactions before timeout/OOM + Elle hundreds of thousands of transactions in tens of seconds + ratio ~10^3 more history, checked in less time ``` -Cycle detection is polynomial — the NP-complete search of Step 2 is -gone, bought entirely by Step 3's workload design. And the -counterexample is human-readable ("this txn read state that implies -it ran both before and after that one"). Question: why do unique -values + list semantics make wr/ww edges *directly observable* where -plain registers hide them? +And the counterexample is human-readable ("this txn read state that +implies it ran both before and after that one") rather than "no +linearization exists", which is what a search-based checker gives +you. -### Step 5 — what the method finds: redis-raft, 2020 +Why it matters: the cycle is both the *proof* and the *explanation*. +A checker that only says yes/no produces bug reports nobody can act +on; §7's results depended on being able to hand a vendor four +transactions. -The Redis-Raft analysis is the catalog of consensus-*integration* -bugs — none were in the Raft paper's math, ALL were in the plumbing: +### Step 5 — what the method finds: Redis-Raft, 2020 -- acked writes lost on failover (stale-leader window) -- reads served by deposed leaders (no ReadIndex — topic 15 §4!) -- log divergence after membership changes -- the infamous "Raft on top of a system with its own replication" - impedance +> **In:** Redis-Raft development builds `1b3fbf6` through `e0123a9`, +> five-node clusters, the Elle append workload over `RPUSH`/`LRANGE`. +> **Out:** 21 issues — and a lesson about where they were. -Question: for each finding, which of our topic-15 raft.rs tests (or -which MISSING test) covers it? +The Redis-Raft analysis is the catalog of consensus-*integration* +bugs. None were in the Raft paper's math; all were in the plumbing. +The report's own tally: + +> "we found twenty-one issues, including long-lasting unavailability +> in healthy clusters, eight crashes, three cases of stale reads, +> one case of aborted reads, five bugs resulting in the loss of +> committed updates, one infinite loop, and two cases where +> logically corrupt responses could be sent to clients." + +Four worth knowing by their mechanism, because the mechanism is the +teaching: + +- **#14, total data loss on failover.** Not a stale-leader window — + a **missing re-entrancy check**. Redis-Raft intercepts `SET k v`, + rewrites it to `RAFT SET k v`, replicates it, then unwraps and + applies it — whereupon the interception code saw `SET k v` again + and re-wrapped it. With proxying off, followers rejected the + re-wrapped op, so nothing ever reached a follower's state machine + and *any* failover elected a leader with empty state. The same + bug with proxying *on* was #13: an infinite loop that ballooned + the log on every write. One missing check, two catastrophes. +- **#19, stale reads with no faults at all.** A leader is supposed + to commit a no-op entry on election to learn what is committed; + the bundled Raft library didn't. Report example: T1 appended 11 to + key 1 and completed **3.25 seconds** before T2 began, and T2 read + `[5 8 9]`. This is the ReadIndex-adjacent hole from topic 15 — but + note the fault column in the report's table says **None**. +- **#17, split-brain via membership change.** `RAFT_LOGTYPE_REMOVE_NODE` + was left out of the set of log entry types counted as voting + configuration changes, so a leader could remove every other node + unilaterally and declare itself a single-node cluster. "Given *n* + nodes and a sufficiently pathological operator, Redis-Raft could + split into *n* separate clusters." +- **#28, split-brain redux.** Reads of key 81 on `n1` returned lists + beginning `[171 172 176 …]` while `n5` returned `[176 …]` — and + appends of 178 and 208 landed on **both** divergent prefixes. The + underlying library assumed nodes would be demoted *then* removed, + rather than removed directly. + +Now count the fault column of the report's summary table: -The Dgraph analysis is the graph-DB cautionary tale: per-key Raft -groups + cross-group txns = lost writes and read skew — a preview -of topic 29's distributed-transaction problems. +``` + 21 issues, by the fault needed to trigger them: + None 5 (#13, #19, #21, #25, #42) + Failover only 1 (#14) + crash / partition / pause / membership 15 + + fraction needing NO fault injection = 5 / 21 = 23.8% + fraction of the first tested build's two headline bugs + that needed no fault 2 / 2 (#13 and #14: "essentially + unusable" before a nemesis ran) +``` + +Nearly a quarter of the findings needed no nemesis at all — they +needed only *a workload with an oracle*. That is the cheapest +lesson in this topic: before you build fault injection, build the +checker. + +Question: for each finding, which of our topic-15 `raft.rs` tests +(or which MISSING test) covers it? + +The Dgraph analysis is the graph-DB cautionary tale, and its punch +line is even better. Dgraph shards by predicate into per-group Raft +clusters, with a separate Zero Raft cluster allocating timestamps +(an Omid Reloaded design) and claims **snapshot isolation**. The +bank test lost money — a $100 total reading as $102, then 70–80% of +balances vanishing — after a routine predicate migration, with "no +network or node failures". And the cause: + +> "Losing all but the most recently inserted value is a suspicious +> bug to say the least, and its cause turned out **not to be a +> distributed systems problem at all**! … the temporary data +> structure for serialization received Go slices (i.e. pointers) to +> a mutable loop variable which identified the key for that triple. +> This meant that before serialization, *every* triple shared the +> most recently iterated key." + +A Go loop-variable aliasing bug, surfaced as a distributed +consistency violation. Which is Step 5's actual thesis: the +consistency checker is an *end-to-end* oracle, and end-to-end +oracles catch bugs that have nothing to do with the layer you +suspected. + +Why it matters: you will be tempted to test the consensus algorithm. +Both reports say the algorithm was fine and the wrapping was not. ### Step 6 — Jepsen vs DST: complements, not competitors +> **In:** two testing methods that both find concurrency bugs. +> **Out:** a division of labour — and a specific gap each leaves +> that the other fills. + The comparison that matters for M16: | | Jepsen | DST (turso/FDB) | @@ -137,19 +379,62 @@ believes nothing you told it (real kernel, real network, real binary). A serious engine wants both: the bug classes barely overlap. +The Redis-Raft report names its own gap, and it is exactly the gap +this topic's `crash_matrix` measures: + +> "We have not explored single-node faults, such as filesystem +> corruption or the loss of un-fsynced data written to disk. Both +> might be of interest for Redis-Raft, whose correctness hinges +> (like most consensus systems) on single-node durability." + +Put the two side by side with this topic's own numbers. Jepsen's +Redis-Raft campaign ran for months across a dozen builds and +produced 21 issues; `crash_matrix` sweeps 5,000 seeds × 40 ops in +about 0.02 s — roughly 200,000 simulated crash-recoveries per second +— and catches a planted `NoSyncOnCommit` bug in 4,980 of 5,000 +seeds (99.6%). Those are not competing numbers; they are numbers +about different things. `NoSyncOnCommit` is precisely the +un-fsynced-write fault Jepsen said it had not explored, and no +amount of `iptables` would have found it. + +And the discipline runs the other way too. The report's closing +caveat is the sentence to carry into M16: + +> "Jepsen takes an experimental approach to safety verification: we +> can prove the presence of bugs, but not their absence." + +Which is also true of `crash_matrix`, and is why topic 21's solver +exists. + +Why it matters: choosing between them is a category error. Choosing +*which one to build first* is not — and the answer is whichever +covers the fault your durability story depends on. + ## How to read the analyses (with the concepts in hand) -1. **elle paper (VLDB 2020)** — read §on the dependency-graph - construction with Steps 3–4 in hand; the anomaly taxonomy section - is a topic-8 isolation refresher with better names. -2. **"Redis-Raft 1b3fbf6" (2020)** — read in full. For every finding, - identify which Step 4 edge types formed the cycle, and which - plumbing layer (election, log, membership) produced it. -3. **"Dgraph 1.0.2" (2018)** — read as the graph-DB case: watch how - per-key Raft groups turn single-system anomalies into - distributed-transaction ones. -4. **elle README** — the anomaly taxonomy (G0/G1/G2) is the fastest - refresher when the reports start naming cycles. +1. **Elle paper (VLDB 2020) §4 and §6** — the dependency-graph + construction with Steps 3–4 in hand; §6's cycle taxonomy is a + topic-8 isolation refresher with better names, and §6.1's + non-cycle anomalies (garbage read, duplicate write, internal + inconsistency) are the ones you would not have thought to check. +2. **Elle paper §7.5** — the performance section. This is the + argument for the whole design, in measurements: Knossos's few + hundred transactions against Elle's hundreds of thousands. +3. **"Redis-Raft 1b3fbf6" (2020)** — read in full. For every + finding, identify which Step 4 edge types formed the cycle, and + which plumbing layer (election, log, membership, *proxying*) + produced it. Then check the fault column: five needed none. +4. **"Dgraph 1.0.2" (2018)** — read as the graph-DB case, then read + the "Migration Read Skew & Write Loss" section twice: the + distributed-looking symptom, the single-node cause. +5. **Elle paper §7's case studies** — TiDB 2.1.7–3.0.0-beta.1 + (G-single from two default-on automatic retry mechanisms, fixed + in 3.0.0-rc2), YugaByte DB 1.3.1 (G2-item on master crash, from a + fresh master briefly advertising an empty capabilities set), + FaunaDB 2.6.0 (internal inconsistency with **no faults at all**), + and Dgraph 1.1.1 (cyclic version orders from shard migration). + Note the paper's own summary: "Elle revealed anomalies in every + system we tested." ## Questions for notes.md @@ -169,23 +454,198 @@ overlap. ## Done when +Answer each before unfolding it. + - [ ] You can explain why checking a recorded history is the hard half of the method, not collecting one. + +
Answer + + Collecting is `n` clients writing to a log. Checking asks whether + *any* serial order explains the observations, and both + linearizability and serializability checking are NP-complete in + general (Elle §1). The search space is the permutations of + concurrent operations: "given c concurrent transactions, the number + of permutations to evaluate is c!" (§7.5). + + The measured consequence, from §7.5 on a 24-core Xeon with 128 GB + and a 100-second cap: Knossos "often timed out or ran out of memory + after a few hundred transactions", "many Knossos runs involved + search spaces on the order of 10^24", and "with 40+ concurrent + processes, even histories of 5000 transactions were (generally) + uncheckable". + + That caps test *duration*, which caps which bugs you can reach — + a correctness tool bounded by a performance problem. + +
+ - [ ] You can describe elle's workload trick and say why append-and-read-full-list makes dependencies visible. + +
Answer + + Every write is `append(k, v)` with a globally unique `v`; every + read returns the entire list for `k`. The list *is* the version + order, written down by the database itself. + + One read of an `n`-element list yields `n − 1` ww edges (adjacent + pairs), `n` wr edges (this transaction saw each of those writes), + and at least one rw anti-dependency (any later appender follows + this read). For `n = 20` that is 39 dependency facts from a single + operation. The same read of a register yields at most one wr edge + and zero ww edges, because each write destroyed its predecessor's + evidence. + + turso's `ElleOp::Read { key, result: Option> }` + (`testing/concurrent-simulator/elle.rs:22-25`) is the type: the + `Vec` is the recovered version order, the `Option` is "invoked but + not completed". + +
+ - [ ] You can explain why a cycle in the serialization graph *is* an anomaly, and identify which isolation level a pure-rw cycle violates. + +
Answer + + An edge `T1 → T2` asserts "T1 must precede T2 in any serial order". + A cycle asserts a transaction must precede itself — so no serial + order exists, and the history is by definition not serializable. + + A cycle of rw anti-dependencies is **G2** (Elle §6: "one or more + rw"), which in its two-edge form is write skew. Snapshot isolation + *permits* it — that is SI's defining hole, and it is why Dgraph's + upsert test needed index entries treated as conflictable objects. + Serializable forbids it. G-single, one rw edge exactly, is read + skew and is forbidden by SI. + + Detection is Tarjan's SCC plus a BFS inside each component (§6) — + near-linear, versus the `c!` of Step 2, and it hands you the + offending transactions rather than a bare "no". + +
+ - [ ] You can say why Jepsen uses SIGSTOP/SIGCONT rather than kill -9 for certain faults. + +
Answer + + A `kill -9` node is *gone*: it stops holding leases, stops + responding, and its peers correctly conclude it is dead. A + SIGSTOPped node is alive and will **resume**, still believing + whatever it believed before — that it is the leader, that its lease + is valid, that its in-flight write succeeded. + + That is the GC-pause / VM-migration / hypervisor-steal failure, and + it is the one that breaks leases and produces two leaders. It is + the reason for fencing tokens (DDIA ch. 8): a paused leader that + wakes up must be *rejected by the storage layer*, because nothing + it can check locally will tell it time has passed. + + Redis-Raft's nemesis list includes pauses explicitly, and issue #51 + (`EntryCacheAppend` assertion) has "Pause" alone in its fault + column — a crash would not have found it. + +
+ - [ ] You can state what elle cannot check, and where DST complements rather than competes. + +
Answer + + Elle's own stated boundary (abstract): it detects "every anomaly in + Adya et al's formalism (**except for predicates**)" — predicate + anti-dependencies, the phantom-adjacent class, are out of scope + because the workload observes keys, not predicates. It also cannot + check a system that only exposes registers with the same power (see + the previous answer), and being experimental it "can prove the + presence of bugs, but not their absence" (Redis-Raft, Discussion). + + The complement is the fault axis, and the Redis-Raft report names + it: "We have not explored single-node faults, such as filesystem + corruption or the loss of un-fsynced data written to disk." That + is precisely what this topic's `crash_matrix` sweeps — and the + planted `NoSyncOnCommit` bug is caught in 4,980 of 5,000 seeds + (99.6%) at roughly 200,000 simulated crash-recoveries per second. + No `iptables` rule reaches it. + + Deterministic simulation also makes reproduction free (a seed) and + the real-time order exact, where Jepsen must reason about clock + uncertainty between machines. + +
+ +- [ ] You can name the actual root cause of Redis-Raft's total data loss and of Dgraph's write loss, and say what both have in common. + +
Answer + + **Redis-Raft #14**: a missing re-entrancy check. Commands were + intercepted and wrapped as `RAFT SET k v`; after commit they were + unwrapped to `SET k v` and applied — and the interception code + wrapped them *again*. With proxying off, followers rejected the + re-wrapped op, so no follower ever applied anything and every + failover produced an empty leader. + + **Dgraph 1.0.2**: a Go slice aliasing a mutable loop variable + during predicate migration, so every triple in a batch ended up + sharing the most recently iterated key. The report is explicit: + "its cause turned out not to be a distributed systems problem at + all!" + + What they share: neither is a flaw in Raft or in snapshot + isolation. Both are ordinary programming bugs in the *integration* + layer, and both were found by an end-to-end consistency oracle that + did not know or care which layer it was testing. Test the claim, + not the algorithm. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the redis-raft stale-read history. +
Answer + + No unfoldable answer — this one is the writing. For the stale-read + history, the report hands you the shape: `T1: [:append 1 11]` + completing 3.25 seconds before `T2: [:r 1 [5 8 9]]` begins. Write + it as an Elle history with invoke/ok events and say which edge is + missing (the real-time edge T1 → T2 that the wr edge contradicts), + and note the fault column: **None**. + + For question 5, the thing the deterministic simulator makes trivial + is exactly the thing Elle spends §7.5's budget on: in a simulator + the total real-time order of every event is *known by + construction*, so real-time edges are exact rather than inferred + from wall-clock windows with uncertainty at both ends. What you + give up is Step 1's whole premise — you are no longer testing an + unmodified binary on a real kernel. + +
+ ## References **Papers** - Kingsbury & Alvaro — "Elle: Inferring Isolation Anomalies from Experimental Observations" (VLDB 2020, - [arXiv:2003.10554](https://arxiv.org/abs/2003.10554)) -- Jepsen analyses ([jepsen.io/analyses](https://jepsen.io/analyses)) - — read TWO: "Redis-Raft 1b3fbf6" (2020) and a graph one, - "Dgraph 1.0.2" (2018) + [arXiv:2003.10554](https://arxiv.org/abs/2003.10554)) — §1 for the + NP-completeness and the Gecode history, §4 and §6 for the graph + construction and the G0/G1c/G-single/G2 taxonomy, §6.1 for the + non-cycle anomalies, §7 for the four case studies, §7.5 for the + Knossos-vs-Elle measurements + +**Reports** ([jepsen.io/analyses](https://jepsen.io/analyses)) — read +TWO in full: +- "Redis-Raft 1b3fbf6" (2020) — 21 issues; the Discussion section's + tally and the per-issue fault column are the parts to reason over +- "Dgraph 1.0.2" (2018) — the bank test, the predicate-migration + write loss, and the Go loop-variable cause **Code** - [elle](https://github.com/jepsen-io/elle) — the checker itself; - the README's anomaly taxonomy is the fastest G0/G1/G2 refresher + not pinned in `resources/codebases.md`, so nothing here cites it + by line +- turso @ `dd775bc` — the Jepsen-side integration, walked in + [reading-turso-simulator.md](reading-turso-simulator.md) + +| File | Lines | What | +|---|---|---| +| `testing/concurrent-simulator/elle.rs` | 1-7 | module doc: G0/G1/G2/G-Single, "export to EDN for analysis with elle-cli" | +| `testing/concurrent-simulator/elle.rs` | 18-30 | `ElleOp` — list-append and rw-register models side by side | +| `testing/concurrent-simulator/elle.rs` | 36-67 | `to_edn` — `[:append "k" v]` and `[:r "k" [1 2 3]]` | +| `testing/concurrent-simulator/elle.rs` | 72-79 | `ElleEventType` — Invoke / Ok / Fail / Info | +| `.github/workflows/elle.yml` | — | the check running in CI | diff --git a/topics/16-testing-correctness/reading-pqs-tlp-papers.md b/topics/16-testing-correctness/reading-pqs-tlp-papers.md index 7ac108e..a3c4429 100644 --- a/topics/16-testing-correctness/reading-pqs-tlp-papers.md +++ b/topics/16-testing-correctness/reading-pqs-tlp-papers.md @@ -11,42 +11,89 @@ gives you a reading route through both. Pair with [reading-sqlancer.md](reading-sqlancer.md) — the code makes the papers concrete. +Every number below is quoted from the papers themselves, with the +section, table or figure it came from. Two of the three are open +access: PQS is [arXiv:2001.04174](https://arxiv.org/abs/2001.04174), +and preprints of TLP and NoREC are at `manuelrigger.at/preprints/`. +Download them before reading further — this chapter is a route +through the papers, not a substitute for them. + ## The problem in one sentence -PQS alone found ~100 bugs in SQLite/MySQL/Postgres in ~4 months — -in the three most-tested database engines on earth — because until -2020 nobody had a scalable answer to "what should this random query -return?" +PQS alone found 123 bugs in SQLite/MySQL/Postgres in about three +months — in the three most-tested database engines on earth — +because until 2020 nobody had a scalable answer to "what should this +random query return?" ## The concepts, step by step ### Step 1 — the test-oracle problem, and why differential testing fails +> **In:** a generator that emits syntactically valid random SQL at +> microsecond cost. +> **Out:** no verdict — the missing piece is not inputs, it is +> ground truth. + An **oracle** is the component of a test that decides whether an output is wrong; for randomly generated SQL, no such component -existed. Prior art (RAGS, 1998) used **differential testing**: run -the same query on multiple DBMSs and flag disagreements. Two +existed. Prior art (RAGS, Slutz 1998) used **differential testing**: +run the same query on multiple DBMSs and flag disagreements. Two failures killed it: dialects legitimately diverge (MySQL returns 0/1 booleans, SQLite coerces types by "affinity" — a disagreement is usually not a bug), and a bug all systems share produces no -disagreement at all. Both PQS and TLP need only ONE system — that's -the breakthrough. +disagreement at all. + +The scale of the second problem is worth stating precisely. PQS §4.7 +Table 4 reports the SQLancer implementation cost per DBMS: SQLite +6,501 LOC against SQLite's 49,703 LOC (13.1%), MySQL 3,995 against +707,803 (0.6%), PostgreSQL 4,981 against 329,999 (1.5%) — and only +918 LOC shared between them. A "reference implementation" of SQL is +not a thing you can cheaply have; a *relationship* is. + +Both PQS and TLP need only ONE system — that's the breakthrough. + +Why it matters: this is the constraint that shapes every oracle in +topic 16, including the crash-recovery oracle `crash_matrix` +measures. You never get a second correct system for free. ### Step 2 — SQL's third truth value -One database fact both papers pivot on: a SQL predicate (a WHERE -condition) evaluates to TRUE, FALSE, or **NULL** ("unknown" — -`NULL = 5` is neither true nor false), and WHERE keeps only rows -where it is TRUE. Rows evaluating FALSE *or NULL* vanish. Any -two-valued mental model — including the one inside an optimizer -author's head — is wrong in exactly these cases, which is where the -bugs cluster. +> **In:** a **predicate** — a boolean expression in a `WHERE` +> clause — and a row. +> **Out:** TRUE, FALSE, or NULL — three outcomes, of which only one +> keeps the row. + +One database fact both papers pivot on: a SQL predicate evaluates to +TRUE, FALSE, or **NULL** ("unknown" — `NULL = 5` is neither true nor +false), and `WHERE` keeps only rows where it is TRUE. Rows +evaluating FALSE *or NULL* vanish. Any two-valued mental model — +including the one inside an optimizer author's head — is wrong in +exactly these cases, which is where the bugs cluster. + +Count the arms of the case analysis a correct optimizer rewrite has +to survive: + +``` + two-valued reasoning: p ∈ {T, F} 2 cases + SQL reasoning: p ∈ {T, F, N} 3 cases + a rewrite over p AND q: 2×2 = 4 cases assumed, 3×3 = 9 real + → 5 of 9 cases (56%) involve a NULL + and are the ones nobody wrote a test for +``` + +Why it matters: Step 4's rectification and Step 6's partition are +both, structurally, "handle the third case" — and PQS's evaluator +and TLP's third query exist for no other reason. ### Step 3 — PQS: verify ONE row you chose in advance +> **In:** a database where every table holds at least one row. +> **Out:** a **pivot row** — one row from *each* table — and a +> query that provably must return it. + Pivoted Query Synthesis inverts the problem. Don't verify the whole -result set of a random query; pick a random existing row (the -**pivot**), then construct a query that provably must return it: +result set of a random query; pick a pivot, then construct a query +that provably must return it: ``` pick pivot row r @@ -54,21 +101,47 @@ result set of a random query; pick a random existing row (the if r ∉ result(SELECT ... WHERE p) → bug ``` +"Pivot row" is more specific than "a random row", and the difference +matters as soon as there is a `JOIN`. §3.1: "We ensure that each +table holds at least one row. We then select a random row from each +of the tables (see step 2), to which we refer as the pivot row." The +pivot is a row of the cross product — one component per table in the +`FROM` clause — which is what makes `t0.c1` and `t1.c0` both +substitutable in step 3. + Ground truth for one row of one query is cheap to compute — and because generation costs microseconds, "one row per query" times -millions of queries covers the input space in expectation. +millions of queries covers the input space in expectation. The +evidence that this is enough is §4.3: the reduced bug-triggering +test cases averaged **3.71 lines of code**, 13 of them needed a +single line, and the largest was 8 statements (with one 27-statement +outlier). Bugs did not need big inputs; they needed the right one. -### Step 4 — rectification: make ANY random predicate TRUE on the pivot +Why it matters: PQS is the only oracle in this topic that knows a +*fact* about the answer rather than a relationship — and Step 5 is +the bill for that. -The § on *rectified queries* is the algorithmic core. Generate a -random expression tree, evaluate it bottom-up on r's concrete values -under the DBMS's own semantics (dialect-specific NULL rules, casts, -collation — all of it), then **rectify**: TRUE → keep, FALSE → wrap -`NOT`, NULL → wrap `IS NULL` (Step 2's third value gets its own -wrapper): +### Step 4 — rectification: make ANY random predicate TRUE on the pivot -```rust -// rectify: ANY random predicate becomes TRUE-on-the-pivot +> **In:** a randomly generated expression tree and the pivot row's +> concrete values. +> **Out:** a predicate guaranteed to evaluate TRUE on the pivot — +> whatever the original expression did. + +The section on *rectified queries* (§3.2) is the algorithmic core. +Generate a random expression tree, evaluate it bottom-up on r's +concrete values under the DBMS's own semantics (dialect-specific +NULL rules, casts, collation — all of it), then **rectify**: TRUE → +keep, FALSE → wrap `NOT`, NULL → wrap `IS NULL` (Step 2's third +value gets its own wrapper). That is the paper's Algorithm 3, +`rectifyCondition`, driven by Algorithm 1's `generateExpression` and +Algorithm 2's per-node `execute`: + +```text +// ILLUSTRATION — pseudocode for PQS §3.2 Algorithm 3, rectifyCondition. +// Not quoted from SQLancer; the running equivalent is +// src/sqlancer/common/oracle/PivotedQuerySynthesisBase.java:125 +// (abstract getRectifiedQuery, "steps 2-5 of the PQS paper"). fn rectify(p: Expr, pivot: &Row) -> Expr { match eval3(&p, pivot) { // eval under the DBMS's OWN dialect rules True => p, @@ -79,21 +152,92 @@ fn rectify(p: Expr, pivot: &Row) -> Expr { // then: pivot ∉ result(SELECT * FROM t WHERE rectify(p, pivot)) → BUG ``` +Do the arithmetic on what rectification buys. Suppose a generated +predicate is TRUE on the pivot a third of the time: + +``` + without rectification: keep only TRUE-valued predicates + usable fraction ≈ 1/3 + generated per usable query ≈ 3 + and the discarded 2/3 are exactly the FALSE and NULL cases — + the ones Step 2 says the bugs live in + + with rectification: every predicate becomes usable + usable fraction = 1 + generated per usable query = 1 + speedup ≈ 3×, and the NULL arm is now + over-represented rather than absent +``` + +The `IS NULL` arm is the one that matters: without it a NULL-valued +predicate is unusable, and a NULL-valued predicate is the shape of +test that finds NULL-blind optimizer rewrites. + Question: why does rectification make EVERY randomly generated expression usable rather than discarding the ~2/3 that aren't TRUE? +Why it matters: this is the trick, and it is three lines. Everything +expensive about PQS is in `eval3`, not here. + ### Step 5 — what PQS costs, and what it cannot see +> **In:** a working PQS implementation for one DBMS. +> **Out:** an expression evaluator you now own, and one bug class +> the oracle structurally cannot see. + Two prices. First, that `eval3` is a full expression evaluator *per -dialect* — weeks of work for each DBMS, re-implementing exactly the -quirks (MySQL's 0/1 booleans, SQLite's type affinity) you're testing. -Second, containment-not-equality blindness: PQS asserts the pivot -appears in the result — a bug that returns the pivot row plus -GARBAGE rows passes. Results to internalize anyway: ~100 bugs in -~4 months, most in SQLite — which then fixed its test suite. +dialect* — re-implementing exactly the quirks (MySQL's 0/1 booleans, +SQLite's type affinity) you're testing. §3.2 measures one operator: +"the implementation of the LIKE regular expression operator has over +50 LOC in SQLancer." Multiply by an operator table. + +Second, containment-not-equality blindness. PQS asserts the pivot +*appears* in the result; a bug that returns the pivot row plus +garbage rows passes. The paper says so itself: "we cannot detect +logic bugs where a DBMS erroneously fetches duplicate rows." + +Now correct the folklore about the results, because the real numbers +are more interesting than "~100 bugs". From the abstract and §4.2 +Table 2: + +``` + reports opened 123 + true bugs (fixed or verified) 99 = 77 code fixes + 8 doc fixes + 14 confirmed + of which SQLite 65 fixed + MySQL 15 fixed + 10 verified + PostgreSQL 5 fixed + 4 verified + not bugs 24 = 12 "intended behaviour" + 12 duplicates + testing period ~3 months (§4.1) +``` + +And §4.2 Table 3 splits the 99 by *which* oracle caught them: + +``` + Contains (the pivot-row oracle) 61 SQLite 46, MySQL 14, PostgreSQL 1 + Error (unexpected error) 34 + SEGFAULT (crash) 4 + ── + 99 + + the pivot oracle's share = 61 / 99 = 61.6% +``` + +So the headline technique found under two thirds of the bugs the +harness found; a third came free from "the DBMS raised an error it +shouldn't have", which needs no oracle at all. Any harness you build +should log unexpected errors even before it has an oracle — that arm +is a third of the yield for a day of work. + +Why it matters: both prices are *design* costs, not bug counts, and +they are what TLP was written to remove. ### Step 6 — TLP: partition by any predicate, make the DBMS check itself +> **In:** one query with its `WHERE` clause cleared, and any +> randomly generated predicate `p`. +> **Out:** four result sets that must reconcile — with no evaluator +> anywhere. + Ternary Logic Partitioning removes both costs with self-consistency. Any predicate p splits a query's rows into exactly three disjoint groups — TRUE, FALSE, NULL (Step 2) — so: @@ -109,26 +253,163 @@ optimizer — seeing four different queries — plans each differently. The ternary part is the SQL-specific insight: two-valued partitioning (p / NOT p) is WRONG in SQL — NULL rows vanish from both branches, and real optimizer bugs live exactly in that gap -(NULL-blind predicate pushdown, our tlp.rs stub's injected bug). +(NULL-blind predicate pushdown, our `tlp.rs` stub's injected bug). + +Two corrections to carry into the paper. First, the paper's title is +"Finding Bugs in Database Systems via **Query Partitioning**": the +general framework is query partitioning, and TLP is the instance +where the partitioning is done by three-valued logic. Second, the +results, from the abstract and §4 Table 3: + +``` + reports opened 181 + true bugs 175 MySQL, TiDB, SQLite, CockroachDB, DuckDB + fixed 125 + of which logic bugs 77 (Table 4) +``` + +Why it matters: TLP needs no per-dialect evaluator, which is why it +is the technique the SQLancer README calls "among the most widely +adopted testing techniques" (`README.md:82`) while PQS is +"currently unmaintained" (`:80`). ### Step 7 — recombination operators: TLP beyond WHERE -The paper generalizes the identity clause by clause: aggregate TLP -(MAX over partitions = MAX of partition MAXes; AVG canNOT be -recombined from partition AVGs — it needs SUM/COUNT recombination), -DISTINCT, GROUP BY. Each needs a *recombination operator* ⊎ -appropriate to the clause. Question: why is AVG the canonical -example of a non-decomposable aggregate, and what does that echo -from topic 11's partial aggregation? +> **In:** a clause other than `WHERE` — `GROUP BY`, `DISTINCT`, +> `HAVING`, an aggregate. +> **Out:** a different composition operator `⋄` per clause, and one +> aggregate that needs a rewrite instead. + +The paper generalizes the identity clause by clause. **Table 1** +lists nine oracles with the operator each uses to put the partitions +back together (`⊎` is `UNION ALL`, `∪` is `UNION`): + +| oracle | partitions on | ⋄ | note | +|---|---|---|---| +| WHERE | the `WHERE` predicate | `⊎` | duplicates must survive | +| WHERE Extended | predicate + `ORDER BY`/`LIMIT` interplay | `⊎` | | +| GROUP BY | the grouping | `∪` | duplicate groups collapse anyway | +| HAVING | the `HAVING` predicate | `⊎` | | +| DISTINCT | the predicate, under `DISTINCT` | `∪` | | +| MIN / MAX | the predicate | `MIN` / `MAX` of the parts | self-decomposable | +| SUM | the predicate | `SUM` of the parts | self-decomposable | +| COUNT | the predicate | **`SUM`** of the parts | note the operator changes | +| AVG | the predicate | `SUM(s)/SUM(c)` | needs a rewrite — see below | + +`COUNT` is the small surprise: you recombine counts with `SUM`, not +with `COUNT`. `AVG` is the interesting one, and the paper's +vocabulary for it is precise (§2.1, quoting Jesus et al. 2015): + +> "An aggregate function `f` is **self-decomposable** when a merge +> operator `⊕` exists so that, given two non-empty multi-sets X and +> Y, the following holds: `f(X ⊎ Y) = f(X) ⊕ f(Y)`. … An aggregate +> function `f` is **composable** if for some function `g` and a +> self-decomposable aggregate function `h`, it can be expressed as +> `f = g ∘ h`." + +So AVG is not "non-decomposable" — it is **composable but not +self-decomposable**, with `h({x}) = (x, 1)` and `g((s,c)) = s/c`: + +``` + partition sizes: |A| = 3, sum 30 AVG(A) = 10 + |B| = 1, sum 100 AVG(B) = 100 + + wrong (recombine the AVGs): (10 + 100) / 2 = 55 + right (recombine (sum,count)): (30 + 100) / (3 + 1) = 32.5 + true AVG of A ⊎ B: 130 / 4 = 32.5 +``` + +That is exactly topic 11's partial aggregation: you must ship the +partial state `(sum, count)` between workers, not the finished +average. Same algebra, different reason for caring. + +Now the payoff table. §4 Table 4 splits TLP's 77 logic bugs by +oracle: + +``` + WHERE 60 + Aggregate 10 + HAVING 3 + GROUP BY 2 + DISTINCT 2 + ── + 77 (plus, separately, 62 error bugs and 25 crashes) + + the WHERE oracle's share = 60 / 77 = 77.9% +``` + +The paper's own summary: "The WHERE oracle detected 60 bugs … +the most effective one"; "The other oracles detected 17 bugs in +total". If you are porting TLP to a new query language, port the +`WHERE` oracle and stop until it stops finding things. + +Why it matters: the recombination operator is the *only* part of TLP +that has to be redesigned per clause — and per query language, which +is precisely M16's problem. ### Step 8 — the meta-lesson: completeness traded for portability +> **In:** two oracles that both work against a single system. +> **Out:** a design axis — how much you know about the answer versus +> how much it costs to know it. + A metamorphic oracle trades *completeness* for *portability*: PQS knows ground truth for one row of one query; TLP knows only that three queries must reconcile. Both beat differential testing because they need ONE system — no second implementation to disagree with. + +The third point on the axis is NoREC (ESEC/FSE 2020), which knows +even less — one integer — and is worth reading between the two. Its +transformation (§3.1) is smaller than folklore suggests: `SELECT * +FROM t0 WHERE φ` becomes `SELECT (φ IS TRUE) FROM t0`, and the +comparison is the *cardinality* of the first against the count of +TRUEs in the second. Content comparison is §3.3, an extension. Its +results: 159 true bugs of 168 reported, 141 fixed, of which **51** +were logic bugs (§4.3 Table 3: SQLite 39, CockroachDB 7, MariaDB 5, +PostgreSQL 0). + +Line the three up: + +``` + oracle knows per-dialect evaluator logic bugs found + PQS one row must be present YES 61 (of 99, §4.2 T3) + NoREC two counts must be equal no 51 (of 159, §4.3 T3) + TLP three partitions must reconstruct no 77 (of 175, §4 T4) + ─── + 189 logic bugs + total true bugs across the three papers: 123 + 159 + 175 = 457 +``` + +And one more measurement, because it disciplines the whole +enterprise. TLP §5.3 ran each configuration against DuckDB for 10 +hours and measured line coverage: + +``` + database generation alone, no oracle at all 48.3% + any single TLP oracle 55.3% – 55.9% (spread 0.6%) + all oracles together 56.1% + PQS, for comparison (PQS §4.7) 23.7% (PostgreSQL) – 43.0% (SQLite) + + marginal coverage from adding every oracle beyond the first: + 56.1% − 55.9% = 0.2 percentage points + marginal coverage from having any oracle at all: + 55.3% − 48.3% = 7.0 percentage points +``` + +Most of the coverage comes from *generating databases and queries*, +not from the oracle. The oracle's job is not to reach new code — it +is to notice when the code it already reached is wrong. Jung et al. +found DBMS cores exceed 95% coverage after tens of queries; SQLite +has 100% MC/DC coverage and TLP still found bugs in it. Coverage is +a bad proxy for oracle quality, and this is the measurement that +says so. + This is the design space our M16 Cypher oracles live in. +Why it matters: when you pick an oracle for M16 you are picking a +point on this axis, and the axis is "what do I know" versus "what +does knowing cost", not "which found more bugs". + ## How to read the papers (with the concepts in hand) Read PQS first; TLP is partly a response to PQS's costs. @@ -136,17 +417,23 @@ Read PQS first; TLP is partly a response to PQS's costs. 1. **PQS (OSDI '20) §1–2** — the test-oracle problem statement (Step 1) is the keeper; the RAGS comparison tells you why differential testing was a dead end. -2. **PQS §on rectified queries** — the algorithmic core (Step 4). +2. **PQS §3.2, Algorithms 1–3** — the algorithmic core (Step 4). + `generateExpression`, `NotNode::execute`, `rectifyCondition`. Work one rectification by hand with a NULL-valued pivot column. -3. **PQS evaluation** — note the bug counts per DBMS and *where* - they cluster (expression evaluation, exactly what Step 5 - predicts). -4. **TLP (OOPSLA '20) §core identity** — Step 6 in the authors' - words; check that the three partitions are provably disjoint and - exhaustive under three-valued logic. -5. **TLP §generalizations** — the recombination table (Step 7); this - is the part you'll port to Cypher, so read it with M16's - `count(*)`/`collect` in mind. +3. **PQS §4.2, Tables 2 and 3** — the bug counts, and the split + showing only 61 of 99 came from the containment oracle (Step 5). + §4.7 Table 4 for the implementation cost per DBMS. +4. **TLP (OOPSLA '20) §2.1 and Table 1** — Step 6 and Step 7 in the + authors' words; check that the three partitions are provably + disjoint and exhaustive under three-valued logic, and read the + self-decomposable / composable definitions carefully. +5. **TLP §4 Table 4 and §5.2–5.3** — where the bugs actually came + from (60 of 77 from `WHERE`), the five cases where NoREC's + record-count comparison was insufficient, and the coverage + measurement that says coverage is the wrong metric. +6. **NoREC (ESEC/FSE '20) §3.1** — optional but short: the third + point on Step 8's axis, and the one whose transformation you can + hold in your head. ## Questions for notes.md @@ -166,24 +453,199 @@ Read PQS first; TLP is partly a response to PQS's costs. ## Done when +Answer each before unfolding it. + - [ ] You can state the test-oracle problem and explain why differential testing against another DBMS is not a solution. + +
Answer + + The oracle problem: generating a valid random query is cheap; + deciding whether its result is *correct* requires knowing the + answer, and nothing knows the answer. + + Differential testing fails twice. **False positives**: dialects + legitimately differ (MySQL's 0/1 booleans, SQLite's type + affinity), so most disagreements are not bugs and triaging them + costs more than the bugs are worth. **False negatives**: any bug + the two implementations share — and shared misreadings of the + standard are common — produces no disagreement. + + There is a third, quieter reason: you would have to *have* a second + implementation. PQS §4.7 Table 4 shows SQLancer needed 6,501 LOC + for SQLite alone, against SQLite's 49,703 — and that is just an + expression evaluator, not an engine. + +
+ - [ ] You can explain rectification: how any random predicate is made TRUE on the pivot row, and why three-valued logic makes that delicate. + +
Answer + + Evaluate the generated expression on the pivot's concrete values + using the DBMS's own semantics, then wrap by the result: TRUE → + leave it, FALSE → wrap in `NOT`, NULL → wrap in `IS NULL` (PQS + §3.2, Algorithm 3). + + Three-valued logic makes it delicate because the NULL arm cannot + use `NOT`: `NOT NULL` is `NULL`, not TRUE, so a two-valued + rectifier would silently produce a predicate that drops the pivot + and report a bug on every NULL. It needs a *different operator*, + `IS NULL`, which is the only thing in SQL that converts unknown to + known. + + The payoff: instead of discarding roughly two thirds of generated + predicates, all of them become usable — and the NULL-valued third, + which is where the bugs are, becomes over-represented rather than + absent. + +
+ - [ ] You can construct a bug PQS provably misses, using containment rather than equality. + +
Answer + + Any bug that returns *extra* rows. PQS checks that the pivot row is + contained in the result set; a `JOIN` that emits every matching row + twice still contains the pivot, so the containment query returns + non-empty and the oracle is satisfied. + + The paper concedes exactly this: "we cannot detect logic bugs where + a DBMS erroneously fetches duplicate rows." + + The symmetric point is worth making: TLP catches that one (the + partitions won't reconstruct) but misses bugs that are *symmetric* + across all three partitions — if a scan drops the same row from the + whole and from the `p` partition, both sides shrink together and + the identity still holds. Neither oracle is complete; they have + different holes. + +
+ - [ ] You can write the TLP identity and say why `col = col` is a useless partitioning predicate. + +
Answer + + `result(Q) = result(Q WHERE p) ⊎ result(Q WHERE NOT p) ⊎ result(Q + WHERE p IS NULL)`, where `⊎` is multiset addition — `UNION ALL`, + per TLP Table 1's `WHERE` row. + + `col = col` is degenerate because it is TRUE for every non-NULL row + and NULL for every NULL row: the `NOT p` partition is always empty, + and the partition boundary falls exactly on "is this column NULL", + which the engine already special-cases. The identity still holds, + so no bug is found — it burns a run. + + The general lesson: the value of a partitioning predicate is + proportional to how *unevenly and unpredictably* it cuts the rows, + which is why the generator wants deep, mixed-type expressions + rather than simple comparisons. It is also why TLP's coverage + barely moves when you add oracles (§5.3: 55.3% → 56.1%) but + collapses without a generator (48.3% from generation alone). + +
+ - [ ] You can state the trade the two papers make — completeness for portability — and which one you would reach for first. + +
Answer + + PQS buys *completeness for one row* by paying for a per-dialect + expression evaluator (§3.2: the `LIKE` operator alone is over 50 + LOC in SQLancer). TLP buys *portability* by knowing nothing about + the answer beyond an identity three queries must satisfy. + + Reach for TLP first, and the evidence is not opinion: TLP's `WHERE` + oracle alone found 60 of its 77 logic bugs (§4 Table 4), it needs + no evaluator, and the SQLancer README calls it "among the most + widely adopted testing techniques" (`README.md:82`) while PQS is + "currently unmaintained" (`:80`). + + Reach for PQS when you have a bug class TLP is structurally blind + to — most usefully, when you suspect the *expression evaluator* + rather than the optimizer, since TLP runs the same expression on + all four sides and a wrong-but-consistent evaluator satisfies it. + +
+ +- [ ] You can say what fraction of each paper's bugs came from its headline oracle, and what the rest came from. + +
Answer + + **PQS** (§4.2 Table 3): 61 of 99 true bugs (61.6%) from the + containment oracle; 34 from unexpected errors, 4 from segfaults. + + **TLP** (§4 Table 4): of 175 true bugs, 77 were logic bugs, 62 were + error bugs and 25 were crashes — and within the 77, the `WHERE` + oracle found 60 (77.9%). + + **NoREC** (§4.3 Table 3): of 159, only 51 were logic bugs; 58 were + error bugs and 50 were crashes (23 release, 27 debug). + + The pattern: in all three papers roughly half the yield is "the + generator made the engine crash or raise an error it shouldn't + have" — which requires no oracle at all. Build the generator and + the error-log arm first; the oracle is the second half of the + value, not the first. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your first three TLP recombinations for M16. +
Answer + + No unfoldable answer — this one is the writing. For question 5, TLP + Table 1 is the template: pick the clause, then the `⋄` that + reconstructs it. `WHERE` → `⊎` (`UNION ALL` of three `MATCH`es); + `count(*)` → `SUM` of the three counts, not `COUNT`; `collect` → + list concatenation, which is `⊎` again but forces you to decide + whether order is part of the contract. + + Whichever three you pick, check them against + [reading-sqlancer.md](reading-sqlancer.md)'s Step 4: SQLancer's own + comparator checks size then `HashSet` equality + (`ComparatorHelper.java:91, 108-112`), which is weaker than `⊎`. + Write yours to compare multiplicities and you will already have a + sharper oracle than the reference implementation. + +
+ ## References **Papers** - Rigger & Su — "Testing Database Engines via Pivoted Query Synthesis" (OSDI 2020, - [arXiv:2001.04174](https://arxiv.org/abs/2001.04174)) — the - rectified-queries section is the algorithmic core + [arXiv:2001.04174](https://arxiv.org/abs/2001.04174)) — §3.1 pivot + selection, §3.2 Algorithms 1–3 (the rectified-queries core), §4.2 + Tables 2–3 (123 reports / 99 true / 61 from containment), §4.3 + (3.71 LOC average reduced test), §4.7 Table 4 (LOC and coverage + per DBMS) +- Rigger & Su — "Detecting Optimization Bugs in Database Engines via + Non-Optimizing Reference Engine Construction" (ESEC/FSE 2020) — + §3.1 the `(φ IS TRUE)` transformation, §3.3 the content-comparison + extension, §4.3 Tables 2–3 (159 true / 141 fixed / 51 logic) - Rigger & Su — "Finding Bugs in Database Systems via Query - Partitioning" (OOPSLA 2020) — Ternary Logic Partitioning; read - after PQS + Partitioning" (OOPSLA 2020) — §2.1 self-decomposable vs + composable, Table 1 the nine oracles and their `⋄`, §4 Tables 3–4 + (175 true / 125 fixed / 77 logic, 60 of them from `WHERE`), §5.2 + the five NoREC-insufficient cases, §5.3 the DuckDB coverage + measurement **Code** -- [sqlancer](https://github.com/sqlancer/sqlancer) — both papers as - running code; walked in [reading-sqlancer.md](reading-sqlancer.md) +- [sqlancer](https://github.com/sqlancer/sqlancer) @ `af6ae85` — + both papers as running code; walked in + [reading-sqlancer.md](reading-sqlancer.md) +- turso's independent TLP implementation — + `simulator/generation/property.rs:1073-1177`, walked in + [reading-turso-simulator.md](reading-turso-simulator.md) Step 5 + +| Paper section | What to take from it | +|---|---| +| PQS §3.1 | pivot = one row from *each* table, not one row total | +| PQS §3.2 | Algorithms 1–3; `LIKE` alone is >50 LOC of evaluator | +| PQS §4.2 T2–T3 | 123 reported, 99 true, 61 from the containment oracle | +| PQS §4.7 T4 | 6,501 LOC for SQLite; 43.0% / 24.4% / 23.7% line coverage | +| NoREC §3.1 | `SELECT * FROM t WHERE φ` → `SELECT (φ IS TRUE) FROM t` | +| NoREC §4.3 T3 | 159 true, 51 logic — the rest errors and crashes | +| TLP §2.1 | self-decomposable vs composable; AVG needs `(sum, count)` | +| TLP Table 1 | nine oracles, `⊎` vs `∪`, COUNT recombines with SUM | +| TLP §4 T4 | 77 logic bugs; `WHERE` found 60 of them | +| TLP §5.3 | 48.3% coverage from generation alone, 56.1% with every oracle | diff --git a/topics/16-testing-correctness/reading-sqlancer.md b/topics/16-testing-correctness/reading-sqlancer.md index 41a68ef..fd1de56 100644 --- a/topics/16-testing-correctness/reading-sqlancer.md +++ b/topics/16-testing-correctness/reading-sqlancer.md @@ -9,6 +9,29 @@ the oracle base classes (`src/sqlancer/common/oracle/`), not the per-DBMS adapters. The comparative table at the end is what you carry into M16's Cypher oracles. +Every anchor below is SQLancer at commit **`af6ae85`**, the revision +this repo pins (`resources/codebases.md`), quoted with the line +numbers the code occupies at that commit. Where the code and the +papers disagree — and they do, twice — this chapter shows both. + +Start with the headline, because it is the first thing to make +honest. The repo's own README claims only that "SQLancer has found +hundreds of bugs" (`README.md:6`); there is no 450 anywhere in the +tree. The number is a sum of the three founding papers' own +evaluations: + +``` + PQS (OSDI '20, §4.2 Table 2) 123 true bugs SQLite, MySQL, PostgreSQL + NoREC (ESEC/FSE '20, §4.3 Table 2) 159 true bugs SQLite, MariaDB, PostgreSQL, CockroachDB + TLP (OOPSLA '20, §4 Table 3) 175 true bugs SQLite, MySQL, CockroachDB, TiDB, DuckDB + ─── + 457 true bugs across three papers +``` + +457, from three oracles, in five years of engine-decades. The DBMS +list in the title is right as a union but incomplete: MariaDB and +TiDB belong on it too. + ## The problem in one sentence Generating a million random SQL queries is trivial; knowing the @@ -20,126 +43,370 @@ from nothing, and found 450+ real bugs doing it. ### Step 1 — the test-oracle problem +> **In:** a generator that can emit valid random SQL at +> microsecond cost. +> **Out:** nothing usable — until something can decide whether a +> result is wrong. + An **oracle** is whatever tells a test harness that a result is wrong. For random inputs, the oracle is the hard part: a generator can emit `SELECT * FROM t0 JOIN t1 ON ... WHERE (c3 << 2) IS NOT FALSE` in microseconds, but nothing knows what that should return. -Comparing two DBMSs against each other (differential testing) fails -for SQL — dialects legitimately diverge, and a bug both systems -share is invisible. The escape is **metamorphic testing**: instead -of knowing Q's answer, know a *relationship* between Q and a derived -query Q' that must hold if the engine is correct. All three SQLancer -oracles are one choice of relationship each. + +**Differential testing** — comparing two DBMSs against each other — +fails for SQL, for two independent reasons: dialects legitimately +diverge (a disagreement is usually not a bug), and a bug both +systems share is invisible. The escape is **metamorphic testing**: +instead of knowing Q's answer, know a *relationship* between Q and a +derived query Q' that must hold if the engine is correct. All three +SQLancer oracles are one choice of relationship each. + +Why it matters: every design decision downstream — including which +oracle is cheap enough to keep maintaining — follows from the fact +that ground truth is the scarce resource, not test inputs. ### Step 2 — SQL is three-valued: TRUE, FALSE, and NULL -Every SQLancer oracle leans on one fact a smart programmer from -outside databases won't expect: a SQL predicate (a boolean -expression in a WHERE clause) evaluates to one of THREE values — -TRUE, FALSE, or NULL ("unknown": `NULL = 5` is neither true nor -false). WHERE keeps only rows where the predicate is TRUE; rows -where it is FALSE *or NULL* are dropped. Most real optimizer bugs -live exactly in the NULL cases, because programmers — including the -ones writing optimizers — reason two-valued by default. - -### Step 3 — PQS: pick one row, force the query to contain it - -Pivoted Query Synthesis manufactures ground truth for exactly one -row. Pick a random existing row (the **pivot**), then *synthesize* a -WHERE clause guaranteed TRUE on it — and if the pivot doesn't come -back, the engine is wrong. The skeleton -(`PivotedQuerySynthesisBase.check()`, :37): +> **In:** a **predicate** — a boolean expression in a `WHERE` +> clause. +> **Out:** one of *three* values, not two. -``` - 1. pick pivotRow from an existing table (random row) - 2. getRectifiedQuery(): synthesize WHERE that is TRUE on pivotRow - — generate a random expression, EVALUATE it yourself on the - pivot; if it's FALSE wrap NOT, if NULL wrap IS NULL (rectify) - 3. getContainmentCheckQuery(): wrap the DB's own result to ask - "is pivotRow in there?" - 4. containsRows == false → reportMissingPivotRow → BUG +Every SQLancer oracle leans on one fact a smart programmer from +outside databases won't expect: a SQL predicate evaluates to TRUE, +FALSE, or NULL ("unknown": `NULL = 5` is neither true nor false). +`WHERE` keeps only rows where the predicate is TRUE; rows where it +is FALSE *or NULL* are dropped. + +Most real optimizer bugs live exactly in the NULL cases, because +programmers — including the ones writing optimizers — reason +two-valued by default. Concretely: `p OR NOT p` is a tautology in +two-valued logic and is **not** one in SQL, because it evaluates to +NULL whenever `p` does. + +Why it matters: Steps 3, 4 and 5 each need a different answer to +"what do I do with the third value", and each answer is a different +line of code. + +### Step 3 — PQS: pick one row per table, force the query to contain it + +> **In:** a populated database, at least one row per table. +> **Out:** a query guaranteed to return a chosen row — and a +> containment check that fails if it doesn't. + +**Pivoted Query Synthesis** manufactures ground truth for exactly +one row. The paper's step 2 is more specific than "pick a row": "We +then select a random row from **each** of the tables (see step 2), +to which we refer as the pivot row" (PQS §3.1). With three tables in +the `FROM` clause the pivot is a row of the cross product, one +component per table — which is what makes `t0.c0, t1.c0, t2.c1` a +legal thing to compare against. + +The skeleton lives in one abstract class: + +```java +// src/sqlancer/common/oracle/PivotedQuerySynthesisBase.java — check(), 36-53 + 36 @Override + 37 public final void check() throws Exception { + 38 rectifiedPredicates.clear(); + 39 Query pivotRowQuery = getRectifiedQuery(); +// ... 40-42: optional logging of the rectified query ... + 43 Query isContainedQuery = getContainmentCheckQuery(pivotRowQuery); +// ... 44-47: logging ... + 48 // combines step 6 and 7 described in the PQS paper + 49 boolean pivotRowIsContained = containsRows(isContainedQuery); + 50 if (!pivotRowIsContained) { + 51 reportMissingPivotRow(pivotRowQuery); + 52 } + 53 } ``` -Step 2's rectification wrappers (`NOT` for FALSE, `IS NULL` for -NULL) mean every randomly generated expression is usable, not just -the ~1/3 that happen to be TRUE. The price: step 2 requires -SQLancer to implement its OWN expression evaluator per DBMS dialect -(constant folding over one concrete row). That's why PQS finds -*evaluation* bugs — it re-implements evaluation, and disagreement is -a bug in one of the two. Question: whose bug? How does SQLancer -triage false positives where its OWN evaluator is wrong? +Four lines of logic. `getRectifiedQuery` (declared abstract at +`:125`, documented as "steps 2-5 of the PQS paper") synthesizes a +`WHERE` clause the pivot must satisfy; `getContainmentCheckQuery` +(`:114`) wraps it into a query that returns *at least one row* iff +the pivot is present; `containsRows` (`:66-73`) reduces the whole +oracle to "did anything come back". The failure report at `:75-99` +dumps not just the pivot but every rectified predicate with the +value PQS expected it to take — which is the difference between a +bug report and a bug report someone can act on. + +Rectification (Step 4 of +[reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md)) is what +makes every randomly generated expression usable rather than the +third that happen to be TRUE. The price is the class comment at +`:19-22`: `rectifiedPredicates` holds "the predicates used in WHERE +and JOIN clauses, which yield TRUE for the pivot row" — SQLancer has +to *know* they yield TRUE, which means implementing its own +expression evaluator per dialect. + +That price is why PQS is where it is today. The repo README is +explicit: + +> "PQS effectively detects bugs, but requires more implementation +> effort than other testing approaches that follow a metamorphic +> testing or differential testing methodology. Thus, it is currently +> unmaintained." — `README.md:80` + +**Unmaintained, not removed.** At `af6ae85` the base class is live +and eight DBMS still have PQS tests (`test/sqlancer/dbms/Test*PQS.java` +for Databend, Doris, Materialize, MySQL, OceanBase, Postgres, +SQLite, YSQL) against fifteen for TLP. Reading it is still the +fastest way to understand what the other two oracles bought by +giving up ground truth. + +Why it matters: this is the only oracle in the tree that knows a +*fact* about the answer. Everything else knows only a relationship. ### Step 4 — TLP: partition by a predicate, demand the pieces sum -Ternary Logic Partitioning needs no evaluator at all. Any predicate -p splits a query's rows into exactly three disjoint groups — the -rows where p is TRUE, FALSE, and NULL (Step 2) — so the whole must -equal the union of the parts (`TLPWhereOracle.check()`, :76): +> **In:** one original query with **no** `WHERE` clause, and one +> randomly generated predicate `p`. +> **Out:** four result sets — the original, and three partitions — +> that must reconcile. + +**Ternary Logic Partitioning** needs no evaluator at all. Any +predicate `p` splits a query's rows into exactly three disjoint +groups — the rows where `p` is TRUE, FALSE, and NULL (Step 2) — so +the whole must equal the union of the parts: ``` - Q: SELECT * FROM t [JOIN ...] + Q: SELECT FROM t [JOIN ...] -- WHERE explicitly cleared Q_p: ... WHERE p Q_notp: ... WHERE NOT p Q_null: ... WHERE p IS NULL - assert multiset(Q) == Q_p ⊎ Q_notp ⊎ Q_null + paper's assertion (OOPSLA '20 Table 1, WHERE row): + RS(Q) = RS(Q_p) ⊎ RS(Q_notp) ⊎ RS(Q_null) -- ⊎ = multiset addition +``` + +The generic implementation is 44 lines: + +```java +// src/sqlancer/common/oracle/TLPWhereOracle.java — check(), 75-118 (elided) + 75 @Override + 76 public void check() throws SQLException { +// ... 77-87: pick non-empty tables, generate the select, joins, from-list ... + 88 select.setWhereClause(null); + 89 + 90 String originalQueryString = select.asString(); +// ... 91-93: run it, keep firstResultSet ... + 95 boolean orderBy = Randomly.getBooleanWithSmallProbability(); +// ... 96-98: if orderBy, attach ORDER BY clauses ... + 100 TestOracleUtils.PredicateVariants predicates = TestOracleUtils.initializeTernaryPredicateVariants(gen, + 101 gen.generateBooleanExpression()); + 102 select.setWhereClause(predicates.predicate); + 103 String firstQueryString = select.asString(); + 104 select.setWhereClause(predicates.negatedPredicate); + 105 String secondQueryString = select.asString(); + 106 select.setWhereClause(predicates.isNullPredicate); + 107 String thirdQueryString = select.asString(); + 108 + 109 List combinedString = new ArrayList<>(); + 110 List secondResultSet = ComparatorHelper.getCombinedResultSet(firstQueryString, secondQueryString, + 111 thirdQueryString, combinedString, !orderBy, state, errors); + 112 + 113 ComparatorHelper.assumeResultSetsAreEqual(firstResultSet, secondResultSet, originalQueryString, combinedString, + 114 state); ``` -The DB is checked against ITSELF — the optimizer sees three -different queries and may plan each differently (push p into an -index, rewrite NOT p, …); any semantic slip breaks the identity: - -```rust -// TLP: no ground truth needed — the DB is its own oracle -fn tlp_check(db: &Db, q: &Query, p: &Pred) -> Result<(), Bug> { - let whole = db.run(q); // SELECT * FROM t … - let mut parts = db.run(&q.filter(p)); // WHERE p - parts.extend(db.run(&q.filter(¬(p)))); // WHERE NOT p - parts.extend(db.run(&q.filter(&is_null(p)))); // WHERE p IS NULL ← 3-valued! - if multiset(&whole) != multiset(&parts) { - return Err(Bug::PartitionMismatch); // optimizer changed RESULTS - } - Ok(()) -} +Line 88 is the one people miss: the "original" query is built by +*clearing* the `WHERE` clause, so the original and the three +partitions differ by exactly the predicate under test. The three +variants come from one generator call (`:100-101`) into +`TernaryLogicPartitioningOracleBase`'s trio — `predicate`, +`negatedPredicate`, `isNullPredicate` +(`TernaryLogicPartitioningOracleBase.java:19-21`, built at `:34-51` +via `gen.negatePredicate` and `gen.isNull`). + +The `!orderBy` at line 111 selects the recombination strategy: +`getCombinedResultSet` (`ComparatorHelper.java:144-163`) builds one +`firstQuery UNION ALL secondQuery UNION ALL thirdQuery` string when +`asUnion` is true (`:148-152`), and otherwise runs the three +separately and concatenates client-side (`:153-161`). An `ORDER BY` +inside a `UNION ALL` arm is not portable, so the presence of an +`ORDER BY` forces the client-side path. + +Now read the assertion, because it is weaker than the `⊎` above: + +```java +// src/sqlancer/ComparatorHelper.java — assumeResultSetsAreEqual, 89-112 (elided) + 89 public static void assumeResultSetsAreEqual(List resultSet, List secondResultSet, + 90 String originalQueryString, List combinedString, SQLGlobalState state) { + 91 if (resultSet.size() != secondResultSet.size()) { +// ... 92-105: format and throw "The size of the result sets mismatch (%d and %d)!" ... + 106 } + 107 + 108 Set firstHashSet = new HashSet<>(resultSet); + 109 Set secondHashSet = new HashSet<>(secondResultSet); + 110 + 111 boolean validateResultSizeOnly = state.getOptions().validateResultSizeOnly(); + 112 if (!validateResultSizeOnly && !firstHashSet.equals(secondHashSet)) { ``` -Extensions in the codebase: TLP for aggregates (SUM over partitions -must sum), DISTINCT, GROUP BY. Question: why does TLP need the -partitioning predicate p to be deterministic and side-effect free — -what breaks with `random() > 0.5`? +Equal size, then equal *set*. Work the gap: -### Step 5 — NoREC: run the same predicate with the optimizer off +``` + whole = {a, a, b} size 3, HashSet {a, b} + parts = {a, b, b} size 3, HashSet {a, b} + + line 91 3 == 3 → no throw + line 112 {a,b}.equals({a,b}) → no throw + verdict: PASS + + multiset addition would require count(a) = 2 on both sides. + The code never counts. +``` + +A duplicate-multiplicity bug passes. turso's independent +implementation of the same paper has the identical gap — size at +`generation/property.rs:1138`, containment both ways at `:1146-1177` +(see [reading-turso-simulator.md](reading-turso-simulator.md) Step +5). Two implementations, one paper, the same corner cut. + +One more narrowing worth knowing: the "result set" being compared is +one column wide. `getResultSetFirstColumnAsString` +(`ComparatorHelper.java:39-87`) reads `result.getString(1)` — column +1 only, line 61 — and strips trailing zeros from decimals at line +63, with the comment "as many DBMS treat it as non-bugs". That is +why `TLPWhereOracle` calls `gen.generateFetchColumns(true)` with +`shouldCreateDummy = true` (`:84-85`): the query is shaped to +produce one comparable column. + +Why it matters: "we implemented TLP" is a claim about which queries +you send. Whether you implemented the *oracle* is a claim about how +you compare — and the comparison is where two independent +implementations both stopped short. + +### Step 5 — NoREC: run the same predicate where the optimizer can't help + +> **In:** one randomly generated predicate `φ` and a table. +> **Out:** two integers that must be equal. NoREC ("non-optimizing reference engine construction") targets the optimizer specifically, by making the engine compute the same predicate two ways — once where the planner can optimize, once where -it can't: +it can't. The paper's transformation (§3.1) is `SELECT * FROM t0 +WHERE φ` → `SELECT (φ IS TRUE) FROM t0`; SQLancer's SQLite +implementation wraps that in a `SUM`: + +```java +// src/sqlancer/sqlite3/gen/SQLite3ExpressionGenerator.java — 783-792 + 783 @Override + 784 public String generateUnoptimizedQueryString(SQLite3Select select, SQLite3Expression whereCondition) { + 785 SQLite3PostfixUnaryOperation isTrue = new SQLite3PostfixUnaryOperation(PostfixUnaryOperator.IS_TRUE, + 786 whereCondition); + 787 SQLite3PostfixText asText = new SQLite3PostfixText(isTrue, " as count", null); + 788 select.setFetchColumns(Arrays.asList(asText)); + 789 select.setWhereClause(null); + 790 + 791 return "SELECT SUM(count) FROM (" + select.asString() + ")"; + 792 } +``` + +So the pair actually sent is: ``` - optimized: SELECT COUNT(*) FROM t WHERE p (planner ON) - unoptimized: SELECT SUM(CASE WHEN p THEN 1 ELSE 0) (scan + eval) + optimized: SELECT COUNT(*) FROM t WHERE φ (planner ON) + -- or SELECT * FROM t WHERE φ, counted client-side + unoptimized: SELECT SUM(count) FROM ( + SELECT (φ) IS TRUE as count FROM t ) (full scan + per-row eval) +``` + +`IS TRUE` (line 785) is the load-bearing operator and the reason +this is a *SQL* technique rather than a generic one: it collapses +Step 2's three values to two, mapping both FALSE and NULL to 0, so +the sum is exactly the count of rows `WHERE φ` would have kept. Drop +`IS TRUE` and NULL rows would poison the sum. + +The comparison is cardinality only: + +```java +// src/sqlancer/common/oracle/NoRECOracle.java — check(), 72-93 (elided) + 72 boolean shouldUseAggregate = Randomly.getBoolean(); + 73 String optimizedQueryString = gen.generateOptimizedQueryString(select, randomWhereCondition, + 74 shouldUseAggregate); +// ... 75-79: logging ... + 80 String unoptimizedQueryString = gen.generateUnoptimizedQueryString(select, randomWhereCondition); +// ... 81-83: logging ... + 85 int optimizedCount = shouldUseAggregate ? extractCounts(optimizedQueryString, errors, state) + 86 : countRows(optimizedQueryString, errors, state); + 87 int unoptimizedCount = extractCounts(unoptimizedQueryString, errors, state); +// ... 89-91: a -1 from either side means "ignore this run" ... + 93 if (unoptimizedCount != optimizedCount) { ``` +`countRows` (`:123-146`) counts result rows; `extractCounts` +(`:148-170`) sums `rs.getInt(1)` across rows (line 157). The +`shouldUseAggregate` coin at line 72 decides whether the optimized +side is `COUNT(*)` (summed) or `SELECT *` (row-counted) — two +different plan shapes for the same predicate, for free. + Forcing the predicate into the SELECT list defeats index use and -predicate pushdown (moving a filter earlier in the plan) — same +**predicate pushdown** (moving a filter earlier in the plan) — same semantics, no optimizer. A count mismatch means the optimizer -changed RESULTS, not just speed. Question: which of our topic 10 -rewrite rules would NoREC exercise, and which are invisible to it -(ordering? LIMIT?)? +changed RESULTS, not just speed. + +The cardinality-only comparison is a real, measured limitation, and +the TLP paper measured it: re-deriving NoREC test cases from the 60 +bugs TLP's `WHERE` oracle found, "in 5 of these cases, comparing the +record count was insufficient to detect the bug; also the contents +had to be compared, contrary to prior suggestions" (TLP §5.2). Five +of forty-eight. + +Why it matters: NoREC is the cheapest of the three to implement and +the second most productive (51 logic bugs, NoREC §4.3 Table 3) — and +its blind spot is a single design choice you can see in one line. -### Step 6 — composition: three lenses, one schema +### Step 6 — composition: three lenses, one schema, round-robin + +> **In:** a list of `TestOracle`s and one generated +> schema-plus-data. +> **Out:** each generated database state exercised by every oracle +> in turn. Each oracle has a blind spot, and they don't overlap: -| oracle | needs own evaluator | finds | blind to | -|---|---|---|---| -| PQS | YES (per dialect) | expression eval bugs | bugs off the pivot row | -| TLP | no | optimizer logic bugs | bugs symmetric across partitions | -| NoREC | no | pushdown/index bugs | anything both paths share | +| oracle | needs own evaluator | compares | finds | blind to | +|---|---|---|---|---| +| PQS | YES (per dialect) | containment of one pivot row | expression-evaluation bugs | anything about rows *other* than the pivot | +| TLP | no | size + set of the first column | optimizer logic bugs, aggregates, DISTINCT, GROUP BY | duplicate multiplicity; bugs symmetric across all three partitions | +| NoREC | no | one integer | pushdown / index / filter bugs | content bugs that preserve cardinality; anything both paths share | + +They compose — and the composition is deterministic, not random: + +```java +// src/sqlancer/common/oracle/CompositeTestOracle.java — check(), 19-31 + 19 @Override + 20 public void check() throws Exception { + 21 try { + 22 oracles.get(i).check(); + 23 iLast = i; + 24 boolean lastOracleIndex = i == oracles.size() - 1; + 25 if (!lastOracleIndex) { + 26 globalState.getManager().incrementSelectQueryCount(); + 27 } + 28 } finally { + 29 i = (i + 1) % oracles.size(); + 30 } + 31 } +``` -They compose: run all three on the same generated schema/data -(`CompositeTestOracle.java`). That composition — cheap oracles with -disjoint blind spots over one generator — is the design M16's Cypher -oracles copy. +Line 29 is a **round-robin** in a `finally` block: the index advances +even when the oracle throws, so a crashing oracle can't monopolise +the rotation. With `k` oracles registered, each generated database +state gets `1/k` of the checks — which is the argument for keeping +`k` small and each oracle cheap. + +That composition — cheap oracles with disjoint blind spots over one +generator — is the design M16's Cypher oracles copy. Note also what +`af6ae85` has grown beyond the three: the README table +(`README.md:78-87`) lists eight techniques — PQS, NoREC, TLP, DQE +(ICSE '23), QPG (ICSE '23), CERT (ICSE '24), DQP (SIGMOD '24), and +CODDTest (SIGMOD '25) — and `src/sqlancer/common/oracle/` carries +base classes for `CERTOracle`, `CODDTestBase` and `DQEBase` +alongside the three this chapter reads. + +Why it matters: the blind-spot table, not the bug count, is what +tells you which oracle to write next. ## Where each step lives in the code @@ -147,17 +414,30 @@ Read the base classes, not the per-DBMS adapters: | anchor | step | what it is | |---|---|---| -| PivotedQuerySynthesisBase.java:14 | 3 | the PQS skeleton | -| PivotedQuerySynthesisBase.java:30 | 3 | `pivotRow` — the chosen row | -| PivotedQuerySynthesisBase.java:37-51 | 3 | `check()`: rectified query → containment query → "pivot missing" = bug | -| TernaryLogicPartitioningOracleBase.java | 4 | generates p / NOT p / p IS NULL | -| TLPWhereOracle.java:76-92 | 4 | `check()`: original result vs 3-way partition union | -| NoRECOracle.java (reproducer) | 5 | `optimizedQuery != unoptimizedQuery` → bug | -| CompositeTestOracle.java | 6 | run all oracles over one schema/data | +| `README.md:6` | — | "SQLancer has found hundreds of bugs" — the repo's own claim | +| `README.md:78-87` | 6 | the eight-technique table (PQS…CODDTest) with venues | +| `README.md:80` | 3 | PQS "is currently unmaintained" — the reason, in the authors' words | +| `common/oracle/PivotedQuerySynthesisBase.java:14` | 3 | the PQS skeleton class declaration | +| `common/oracle/PivotedQuerySynthesisBase.java:19-22` | 3 | `rectifiedPredicates` — "yield TRUE for the pivot row" | +| `common/oracle/PivotedQuerySynthesisBase.java:30` | 3 | `pivotRow` — the chosen row | +| `common/oracle/PivotedQuerySynthesisBase.java:36-53` | 3 | `check()`: rectified query → containment query → "pivot missing" = bug | +| `common/oracle/PivotedQuerySynthesisBase.java:66-73` | 3 | `containsRows` — the whole oracle is "did anything come back" | +| `common/oracle/PivotedQuerySynthesisBase.java:75-99` | 3 | the failure report: pivot + every predicate's expected value | +| `common/oracle/TernaryLogicPartitioningOracleBase.java:19-21` | 4 | `predicate` / `negatedPredicate` / `isNullPredicate` | +| `common/oracle/TernaryLogicPartitioningOracleBase.java:34-51` | 4 | the trio built via `negatePredicate` and `isNull` | +| `common/oracle/TLPWhereOracle.java:75-118` | 4 | `check()`: clear WHERE (`:88`), three variants, compare | +| `ComparatorHelper.java:39-87` | 4 | `getResultSetFirstColumnAsString` — **column 1 only** (`:61`) | +| `ComparatorHelper.java:89-130` | 4 | `assumeResultSetsAreEqual` — size (`:91`) then `HashSet` (`:108-112`) | +| `ComparatorHelper.java:144-163` | 4 | `getCombinedResultSet` — one `UNION ALL` or three client-side runs | +| `common/oracle/NoRECOracle.java:59-111` | 5 | `check()`: two counts, `!=` is the bug (`:93`) | +| `common/oracle/NoRECOracle.java:123-170` | 5 | `countRows` vs `extractCounts` (`SUM` of column 1 at `:157`) | +| `sqlite3/gen/SQLite3ExpressionGenerator.java:765-792` | 5 | the actual SQL: `COUNT(*)` vs `SUM(count)` over `(φ) IS TRUE` | +| `common/oracle/CompositeTestOracle.java:19-31` | 6 | round-robin over oracles, advancing in `finally` | Reading order: PQS base class first (it's the most mechanical), then -the TLP pair, then NoREC — each `check()` is under 20 lines once you -skip the adapter plumbing. +the TLP pair plus `ComparatorHelper` — the comparator is where the +oracle actually is — then NoREC with one concrete generator beside +it, because the base class alone never shows you the SQL. ## Questions for notes.md @@ -175,23 +455,165 @@ skip the adapter plumbing. ## Done when -- [ ] You can describe all three oracles — PQS, TLP, NoREC — in one sentence each. -- [ ] You can explain why checking one row per query is enough in expectation. -- [ ] You can write the TLP identity for `COUNT(*)` and for `MAX(c)`, and say which one is harder and why. -- [ ] You can explain what NoREC compares and why turning the optimizer off is a valid oracle. +Answer each before unfolding it. + +- [ ] You can describe all three oracles — PQS, TLP, NoREC — in one sentence each, and say what each one compares. + +
Answer + + **PQS**: pick one row per table, synthesize a `WHERE` clause you + have proved TRUE on that pivot, and check the pivot comes back — + comparing *containment of one row* + (`PivotedQuerySynthesisBase.java:49-52`). + + **TLP**: run a query with no `WHERE`, then the same query filtered + by `p`, `NOT p`, and `p IS NULL`, and check the three partitions + reconstruct the whole — comparing *size plus set of the first + column* (`ComparatorHelper.java:91, 108-112`). + + **NoREC**: run `... WHERE φ` against `SELECT SUM(count) FROM + (SELECT (φ) IS TRUE as count FROM t)` and check the two agree — + comparing *one integer* (`NoRECOracle.java:93`). + +
+ +- [ ] You can state PQS's current status in SQLancer and why, without using the word "removed". + +
Answer + + It is **unmaintained but present**. `README.md:80`: "PQS effectively + detects bugs, but requires more implementation effort than other + testing approaches that follow a metamorphic testing or differential + testing methodology. Thus, it is currently unmaintained." + + At `af6ae85`, `PivotedQuerySynthesisBase.java` is a live 138-line + class and eight DBMS still carry `Test*PQS.java` — against fifteen + `Test*TLP.java`. The cause is Step 3's price: PQS is the only oracle + that needs a per-dialect expression evaluator, and there are + nineteen supported DBMS (`README.md:72`). + +
+ +- [ ] You can explain why NoREC's `IS TRUE` wrapper is not decoration. + +
Answer + + `SQLite3ExpressionGenerator.java:785` wraps the predicate in + `IS_TRUE` before summing it. That collapses SQL's three values + (Step 2) to two: TRUE → 1, and **both** FALSE and NULL → 0. The sum + is then exactly the number of rows `WHERE φ` would have kept, which + is the quantity the optimized side produces. + + Without it, a NULL-valued predicate would contribute NULL to the + sum and — depending on the engine — either poison the total or be + silently skipped. Either way the two sides would no longer be + comparing the same thing, and every NULL-bearing row would generate + a false alarm. + +
+ +- [ ] You can say why TLP's implemented check is weaker than the paper's identity, and construct an input that slips through. + +
Answer + + The paper's composition operator for the `WHERE` oracle is `⊎`, + multiset addition (OOPSLA '20 Table 1). `assumeResultSetsAreEqual` + checks `resultSet.size() != secondResultSet.size()` + (`ComparatorHelper.java:91`) and then `HashSet` equality + (`:108-112`) — size plus set, which is strictly weaker. + + `{a, a, b}` vs `{a, b, b}`: same size (3), same set (`{a, b}`). + Passes. Any bug that changes *how many copies* of a row come back — + a join emitting a duplicate, a `UNION ALL` arm dropping one copy — + is invisible. + + It is narrower still than that: only column 1 is compared + (`ComparatorHelper.java:61`), and trailing decimal zeros are + stripped (`:63`). And `--validate-result-size-only` (`:111`) + degrades TLP to a NoREC-style cardinality check on purpose. + +
+ +- [ ] You can explain how the oracles are scheduled when several are enabled, and why that argues for keeping each one cheap. + +
Answer + + `CompositeTestOracle.check()` (`:19-31`) is a **round-robin**: + `i = (i + 1) % oracles.size()` at line 29, inside a `finally`, so + the index advances even when an oracle throws. It is not a random + choice, and one oracle cannot starve the others. + + Consequence: with `k` oracles, each generated database state is + examined by each oracle once every `k` checks — so the marginal + oracle costs `1/k` of every other oracle's throughput. That is the + argument for the TLP paper's own finding (§5.3) that the `WHERE` + oracle alone found 60 of 77 logic bugs while all five oracles + together raised DuckDB line coverage only from 55.3% to 56.1%. + +
+ - [ ] You can sketch a Cypher TLP partition for `MATCH (a)-[e]->(b) WHERE p` and name what makes graph patterns harder than SQL rows here. + +
Answer + + The partition itself transfers directly: run the pattern with no + predicate, then with `p`, `NOT p`, and `p IS NULL`, and require the + three to reconstruct the whole under multiset addition — being + careful to *count* multiplicities rather than repeating + `ComparatorHelper`'s size-plus-set shortcut. + + What is harder: in SQL, NULL arises from a value; in a property + graph, a property can be **absent from the node entirely**, and + `a.age > 30` on a node with no `age` is a third state that also has + to land in the `IS NULL` partition. And the "row" being compared is + a whole path binding `(a, e, b)`, so equality is structural — which + makes the multiplicity question sharper, not softer, than in SQL. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + No unfoldable answer — this one is the writing. Question 4's mapping + is the one to get right, and + [reading-turso-simulator.md](reading-turso-simulator.md) Step 4 + gives it away: `SelectSelectOptimizer` is NoREC (its doc comment at + `model/property.rs:142-148` cites the NoREC paper by name), and + `WhereTrueFalseNull` is TLP (`:153-160` cites the TLP paper). turso + implements no PQS-shaped property at all — consistent with + `README.md:80`'s reason. + +
+ ## References **Papers** -- Rigger & Su — the PQS (OSDI 2020) and TLP (OOPSLA 2020) papers - behind these classes — see +- Rigger & Su — "Testing Database Engines via Pivoted Query + Synthesis" (OSDI 2020) — §3.1 for pivot selection, §4.2 Table 2 + for the 123 bugs +- Rigger & Su — "Detecting Optimization Bugs in Database Engines via + Non-Optimizing Reference Engine Construction" (ESEC/FSE 2020) — + §3.1 for the `(φ IS TRUE)` transformation, §4.3 Table 3 for the 51 + logic bugs +- Rigger & Su — "Finding Bugs in Database Systems via Query + Partitioning" (OOPSLA 2020) — Table 1 for the composition + operators, §5.2 for the count-vs-content measurement +- All three walked in [reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md) **Code** -- [sqlancer](https://github.com/sqlancer/sqlancer) — - `src/sqlancer/common/oracle/` — read the base classes - (`PivotedQuerySynthesisBase`, `TLPWhereOracle`, - `TernaryLogicPartitioningOracleBase`, `NoRECOracle`), not the +- [sqlancer](https://github.com/sqlancer/sqlancer) @ `af6ae85` — + `src/sqlancer/common/oracle/` — read the base classes, not the per-DBMS adapters + +| File | Lines | What | +|---|---|---| +| `README.md` | 6, 72, 78-87 | bug claim, supported DBMS, the eight-technique table | +| `common/oracle/PivotedQuerySynthesisBase.java` | 14-53, 66-99 | PQS: rectify → containment → report | +| `common/oracle/TernaryLogicPartitioningOracleBase.java` | 19-21, 34-51 | the three predicate variants | +| `common/oracle/TLPWhereOracle.java` | 75-118 | the TLP `WHERE` oracle end to end | +| `ComparatorHelper.java` | 39-87, 89-130, 144-163 | first-column extraction, the actual comparison, recombination | +| `common/oracle/NoRECOracle.java` | 59-111, 123-170 | NoREC: two counts, `!=` is the bug | +| `sqlite3/gen/SQLite3ExpressionGenerator.java` | 765-792 | the concrete optimized / unoptimized SQL pair | +| `common/oracle/CompositeTestOracle.java` | 19-31 | round-robin scheduling of oracles | diff --git a/topics/16-testing-correctness/reading-turso-simulator.md b/topics/16-testing-correctness/reading-turso-simulator.md index a3666e8..6383b90 100644 --- a/topics/16-testing-correctness/reading-turso-simulator.md +++ b/topics/16-testing-correctness/reading-turso-simulator.md @@ -10,6 +10,12 @@ map. Read it as the reference implementation for our `dst.rs` stub and for M16 — every piece here has a miniature counterpart in the experiments. +Every anchor below is turso at commit **`dd775bc`**, the revision +this repo pins (`resources/codebases.md`), quoted with the line +numbers the code occupies at that commit. Several of them contradict +what the simulator's own documentation implies; where they do, this +chapter says so and shows the line. + ## The problem in one sentence A crash-recovery bug that needs one specific interleaving of writes, @@ -21,22 +27,28 @@ failure a single u64 you can re-run forever. ### Step 1 — determinism: make the program a pure function of a seed -Deterministic simulation testing (DST) is the discipline of removing -every source of randomness the program doesn't control — wall-clock -time, thread scheduling, IO timing, OS errors — and replacing each -with values drawn from one seeded pseudo-random number generator -(RNG: an algorithm that turns one starting number, the **seed**, -into an endless reproducible stream of "random" numbers). The system -under test (SUT — the code being tested) touches the outside world -only through interfaces, and in test those interfaces are backed by -the RNG: +> **In:** a program that reads the wall clock, spawns threads, and +> calls `pwrite`/`fsync` against a real kernel. +> **Out:** the same program, where every one of those reads is +> served by one seeded RNG — so the whole run is a pure function of +> a `u64`. + +**Deterministic simulation testing (DST)** is the discipline of +removing every source of randomness the program doesn't control — +wall-clock time, thread scheduling, IO timing, OS errors — and +replacing each with values drawn from one seeded **pseudo-random +number generator** (an algorithm that turns one starting number, the +**seed**, into an endless reproducible stream of "random" numbers). +The **system under test** (SUT — the code being tested) touches the +outside world only through interfaces, and in test those interfaces +are backed by the RNG: ``` real: code → syscalls → kernel (time, threads, fsync — nondeterministic) - DST: code → traits ──→ SimClock (ChaCha8 from seed) - ├──→ SimFile (buffered; crash DROPS unsynced, - │ may TEAR the last write) - └──→ SimNet (topic 15's sim.rs already did this) + DST: code → traits ──→ SimulatorClock (ChaCha8 from seed) + ├──→ SimulatorFile (buffered; crash DROPS unsynced, + │ may TEAR the last write) + └──→ SimNet (topic 15's sim.rs already did this) ⇒ failure = a u64 seed. Re-run seed = same bug, every time. ``` @@ -45,112 +57,485 @@ assertion failure prints its seed; re-running that seed reproduces the exact same interleaving, faults and all. Debugging a one-in-a-million bug becomes ordinary single-run debugging. The cost is architectural: the SUT must own NO nondeterminism, which is why -turso routes all IO and time through traits (dependency injection). +turso routes all IO and time through traits (**dependency +injection**: the caller supplies the implementation, so the test can +supply a different one). + +The boundary is never quite total, and turso's is a good example of +a partial one. `impl Clock for SimulatorIO` +(`testing/simulator/runner/io.rs:114-122`) routes +`current_time_wall_clock` through the simulated clock but returns +`MonotonicInstant::now()` — the *real* monotonic clock — from +`current_time_monotonic`. And `SimulatorClock::new` seeds its +starting instant from the real `Utc::now()` (`clock.rs:18`), so +absolute timestamps differ between runs of the same seed; only the +*deltas* are deterministic. Anything in the SUT that branches on an +absolute wall-clock value, or on monotonic elapsed time, escapes the +seed. + +Why it matters: the seed is only worth as much as the boundary is +tight, and the boundary is a property you can read off the code — +count the escapes before you trust a reproducer. ### Step 2 — simulated time: every `now()` is a seeded random jump +> **In:** a `SimulatorClock` holding the current instant, a +> `ChaCha8Rng`, and a `[min_tick, max_tick)` range. +> **Out:** a `DateTime` that is strictly greater than the last +> one returned, by a seeded random amount. + Once the clock is behind a trait, "time" is just data the simulator makes up. turso's `SimulatorClock` advances the current time by a -random tick on *every read* — no wall clock exists anywhere: +random tick on *every read* — no wall clock exists anywhere after +construction: ```rust -// time is data: every now() consumes seeded randomness and ADVANCES -struct SimClock { - curr: Duration, - rng: ChaCha8Rng, // portable, versioned — never the default RNG - min_tick: Duration, - max_tick: Duration, -} - -impl SimClock { - fn now(&mut self) -> Instant { - self.curr += self.rng.random_range(self.min_tick..self.max_tick); - Instant::from(self.curr) // monotone progress: timeout loops terminate - } -} -``` - -Two design points hide in those ten lines. First, ChaCha8 — a -specific, versioned RNG — not `rand`'s default: the default -algorithm can change between crate releases, silently changing what -every archived seed means. Second, `now()` must ADVANCE rather than -return a fixed value: any loop of the form "retry until deadline" -polls the clock, and if time never moves, the simulation livelocks. -Question: why must `now()` ADVANCE time rather than return a fixed -value? (What loops forever if time never moves? Think timeout code.) +// testing/simulator/runner/clock.rs — the whole clock, 7-13 and 25-34 + 7 #[derive(Debug)] + 8 pub struct SimulatorClock { + 9 curr_time: RefCell>, + 10 rng: RefCell, + 11 min_tick: u64, + 12 max_tick: u64, + 13 } +// ... 15-23: new() seeds curr_time from the REAL Utc::now() (line 18) ... + 25 pub fn now(&self) -> DateTime { + 26 let mut time = self.curr_time.borrow_mut(); + 27 let nanos = self + 28 .rng + 29 .borrow_mut() + 30 .random_range(self.min_tick..self.max_tick); + 31 let nanos = std::time::Duration::from_micros(nanos); + 32 *time += nanos; + 33 *time + 34 } +``` + +The file is 35 lines long; that is the entire clock. Three design +points hide in it. + +First, **ChaCha8** — a specific, versioned RNG — not `rand`'s +default: the default algorithm is allowed to change between crate +releases, which would silently change what every archived seed +means. + +Second, `now()` must ADVANCE rather than return a fixed value. Any +loop of the form "retry until deadline" polls the clock, and if time +never moves, the simulation livelocks. Note the consequence: reading +the clock is not free of side effects, and it *consumes a draw from +the RNG stream*. Add a `tracing` call that reads the clock and every +subsequent random decision in the run shifts — the seed still +reproduces, but it reproduces a different execution. + +Third, line 31 is worth staring at. The variable is called `nanos`; +the constructor is `Duration::from_micros`. The unit is +**microseconds**. Combined with the profile defaults in +`testing/simulator/profiles/io.rs:45-53` — `min_tick: 1`, +`max_tick: 30` — that fixes the scale of simulated time: + +``` + tick range [1, 30) µs profiles/io.rs:45-53 + mean tick (1 + 29) / 2 = 15 µs + 1,000 now() calls 1,000 × 15 µs = 15 ms of simulated time + 1,000,000 calls 1e6 × 15 µs = 15 s of simulated time +``` + +So a run that touches the clock a million times has "aged" fifteen +seconds — while costing the CPU only the work of a million integer +draws. That ratio is the reason DST is cheap: simulated time is +bought at the price of arithmetic, not of sleeping. + +Why it matters: time is the single most common leak in a +determinism boundary, and it is also the cheapest thing to make up. ### Step 3 — fault injection at the file layer, per operation -Fault injection means deliberately making an operation fail the way -hardware and kernels really fail — and the realistic granularity is -*per IO operation*, not "kill the process". In turso every simulated -file can, under seeded control: fail a single `pread` or `pwrite`, -fail a `sync` (fsync — the syscall that forces buffered data to -disk), or delay any operation into a `DelayedIo` queue so it -completes later and out of order. A master switch -(`fault: Cell`, io.rs:14) arms injection; a selective variant -targets faults at one file stem only (the WAL but not the database -file, or vice versa). - -This is exactly the fault model our `sim_fs.rs` copies -(buffered-until-sync + tear-on-crash), and it covers the crash -matrix from topic 5 — automatically, exhaustively, on demand: -torn writes, short reads, fsync failures the kernel will never give -you when you want them. Question: which topic 5 crash-matrix cell -does each of {pwrite fault, sync fault, delayed write + crash} -correspond to? +> **In:** a `SimulatorFile` wrapping a real file, plus a `fault` +> flag and a `latency_probability`. +> **Out:** each `pread` / `pwrite` / `sync` / `pwritev` / `truncate` +> either succeeds now, fails with an injected error, or is pushed +> onto a queue to complete later. + +**Fault injection** means deliberately making an operation fail the +way hardware and kernels really fail — and the realistic granularity +is *per IO operation*, not "kill the process". Two independent +mechanisms sit on every simulated file. + +The first is a fault flag. `pub(crate) fault: Cell` +(`runner/io.rs:14`) is the master switch; +`inject_fault` / `inject_fault_selective` (`runner/io.rs:64-80`) set +it, the selective variant matching on a **file stem** so faults can +be aimed at the WAL but not the database file, or the reverse. The +per-op counters that record what fired are declared on the file, not +the IO layer: `runner/file.rs:19-34` holds six of them +(`nr_pread_faults`, `nr_pwrite_faults`, `nr_sync_faults`, and the +matching call counters). + +The second is latency. Every op consults +`generate_latency_duration`, and on a hit is deferred into a +`DelayedIo` queue so it completes later and out of order: + +```rust +// testing/simulator/runner/file.rs — generate_latency_duration, 99-109 + 99 #[instrument(skip_all, level = Level::TRACE)] + 100 fn generate_latency_duration(&self) -> Option { + 101 let mut rng = self.rng.borrow_mut(); + 102 // Chance to introduce some latency + 103 rng.random_bool(self.latency_probability as f64 / 100.0) + 104 .then(|| { + 105 let now = self.clock.now(); + 106 let sum = now + std::time::Duration::from_millis(rng.random_range(5..20)); + 107 sum.into() + 108 }) + 109 } +``` + +Line 103 is the load-bearing one: `latency_probability` is declared +`pub latency_probability: u8` (`file.rs:40`) and divided by 100, so +it is a **percent**, not a per-mille or a float. The profile default +is `latency_probability: 1` (`profiles/io.rs:45-53`) — one percent. +Line 106 sets the delay itself: uniform in `[5, 20)` milliseconds. +Do the arithmetic before you tune anything: + +``` + P(delay) per op 1% profiles/io.rs:45-53 + file.rs:103 + delay when it fires U[5, 20) ms file.rs:106, mean 12.5 ms + mean delay per op 0.01 × 12.5 ms = 125 µs + vs. a mean clock tick 15 µs (Step 2) + + ⇒ one injected delay ≈ 12.5 ms / 15 µs ≈ 833 clock ticks of + simulated time. A single 1%-probability delay reorders the + IO queue by roughly a thousand ticks' worth of other work. +``` + +The delay path is duplicated per operation rather than factored: +pread at `file.rs:149-158`, pwrite at `175-184`, sync at `200-215`, +pwritev at `233-244`, truncate at `257-266`. + +Now the correction. **Sync faults do not fire at this revision.** A +previous version of this chapter said turso can "fail a `sync` +(fsync)". It cannot: + +```rust +// testing/simulator/runner/file.rs — sync(), 192-199 + 192 self.nr_sync_calls.set(self.nr_sync_calls.get() + 1); + 193 if self.fault.get() { + 194 // TODO: Enable this when https://github.com/tursodatabase/turso/issues/2091 is fixed. + 195 tracing::debug!( + 196 "ignoring sync fault because it causes false positives with current simulator design" + 197 ); + 198 self.fault.set(false); + 199 } +``` + +The armed fault is swallowed *and cleared* (line 198), so it does +not even survive to the next operation. `nr_sync_faults` is declared +at `file.rs:34` and never incremented; the stats table hard-codes a +zero for it, with the comment `// No fault counter for sync` +(`file.rs:87-91`). The profile still defaults `sync: true` in its +fault-enable set (`profiles/io.rs:70-78`), which is exactly why this +is worth knowing: the configuration says the fault is on and the +code says it is off. + +Why it matters: the fault your harness *reports* injecting and the +fault it *actually* injects are two different facts, and topic 16's +own baseline is the argument for checking. `NoSyncOnCommit` is the +easiest planted bug in `crash_matrix` to catch — 99.6% of seeds +find it — precisely because sync behaviour is where crash-recovery +bugs live. A harness that silently declines to perturb `sync` is +declining to look in the richest place. ### Step 4 — the generator: interaction plans with properties woven in -A generator is the machine that produces inputs no human would write -by hand. turso's generator (`generation/`) emits an **interaction -plan**: a workload-distributed sequence of SQL statements -interleaved with **property** checks. A property here is a -metamorphic oracle (an oracle that doesn't know the right answer, -only a relationship two results must satisfy — topic README §2): - -- `SelectSelectOptimizer` — two spellings of the same query must - agree (TLP-shaped). -- `WhereTrueFalseNull` — the three-valued partition identity. -- `UnionAllPreservesCardinality` — row counts must add up. -- `ReadYourUpdatesBack` — a session guarantee (DDIA ch. 5 — same - anomaly, single node). -- `FsyncNoWait` / `FaultyQuery` (property.rs:270) — fault-flavored - properties that assert behavior *under* injection. -- `DoubleCreateFailure` — pins error-path behavior. - -Note the generation trick: unrelated random queries are interleaved -WITHOUT breaking property invariants — coverage and oracles coexist -in one plan. That's the shape M16's Cypher properties need. - -### Step 5 — doublecheck: determinism itself is the cheapest oracle - -Run the identical plan twice from the same seed; the two outputs -must match byte-for-byte (`runner/doublecheck.rs`). This oracle -needs NO model of correct behavior — only the promise from Step 1. -Any divergence means nondeterminism leaked into the SUT: HashMap -iteration order, uninitialized memory, a hidden wall-clock read, a +> **In:** a seed and a workload distribution. +> **Out:** an **interaction plan** — a sequence of SQL statements +> interleaved with property checks, each property carrying its own +> assertion. + +A **generator** is the machine that produces inputs no human would +write by hand. turso's (`testing/simulator/generation/`) emits a +plan of statements interleaved with **properties**. A property here +is a **metamorphic oracle**: an oracle that doesn't know the right +answer, only a relationship two results must satisfy (topic README +§2). + +The `Property` enum (`testing/simulator/model/property.rs:11-212`) +has sixteen variants. The ones worth naming, with the line each is +declared on: + +| variant | line | what it asserts | +|---|---|---| +| `InsertValuesSelect` | 27 | inserted rows come back | +| `ReadYourUpdatesBack` | 49 | UPDATE success *and* failure | +| `TableHasExpectedContent` | 61 | model vs engine, one table | +| `DoubleCreateFailure` | 87 | the error path is pinned | +| `SelectLimit` | 100 | LIMIT n returns ≤ n | +| `DeleteSelect` | 120 | deleted rows are gone | +| `DropSelect` | 137 | dropped table is gone | +| `SelectSelectOptimizer` | 149 | NoREC — see below | +| `WhereTrueFalseNull` | 157 | TLP — see below | +| `UnionAllPreservesCardinality` | 167 | counts add up | +| `FsyncNoWait` | 179 | behaviour under fault | +| `FaultyQuery` | 182 | behaviour under fault | +| `SavepointRollback` | 189 | nested rollback | +| `SequenceMonotonicity` | 203 | sequences never go back | + +Two of those need correcting, and the corrections come from the +doc comments the code itself carries. + +`SelectSelectOptimizer` is **NoREC, not TLP.** Its doc +(`model/property.rs:142-148`) names the paper: "As highlighted by +Rigger et al. in Non-Optimizing Reference Engine +Construction(NoREC), SQLite tends to optimize `where` statements +while keeping the result column expressions unoptimized." It runs +`SELECT FROM ` against `SELECT * FROM WHERE +` and — per the same doc — "is successful if the two +queries return the same number of rows". Cardinality only. That is +the NoREC oracle exactly (see +[reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md) Step 6). + +`WhereTrueFalseNull` is the TLP one, and its doc says so +(`model/property.rs:153-160`): "canonically called Ternary Logic +Partitioning (TLP)". + +`ReadYourUpdatesBack` is **not** a session guarantee. Its doc +(`model/property.rs:39-53`) spells out both arms: on UPDATE success +the after-rows carry the new values; on UPDATE failure +`select_before == select_after`. The second arm is a rollback +check — an atomicity property, not a read-your-writes one. + +Why it matters: a property's *name* is a hypothesis about what it +tests; its doc comment and its assertion are the facts. Two of +sixteen names here mislead. + +### Step 5 — the assertion is weaker than the identity it names + +> **In:** two result sets — the original query's and the recombined +> partition's. +> **Out:** a pass/fail — but under a comparison that is *not* +> multiset equality. + +TLP's published identity is `RS(Q) = RS(Q_p) ⊎ RS(Q_¬p) ⊎ +RS(Q_p IS NULL)` where `⊎` is **multiset addition** (Rigger & Su, +OOPSLA 2020, Table 1, WHERE row). turso builds the three partitions +faithfully: + +```rust +// testing/simulator/generation/property.rs — the three partitions, 1073-1083 + 1073 let old_predicate = select.body.select.where_clause.clone(); + 1074 + 1075 let p_true = Predicate::and(vec![old_predicate.clone(), predicate.clone()]); + 1076 let p_false = Predicate::and(vec![ + 1077 old_predicate.clone(), + 1078 Predicate::not(predicate.clone()), + 1079 ]); + 1080 let p_null = Predicate::and(vec![ + 1081 old_predicate, + 1082 Predicate::is(predicate.clone(), Predicate::null()), + 1083 ]); +``` + +— and stitches them with `UNION ALL` (`generation/property.rs:1094-1115`). +But the check that follows is not `⊎`: + +```rust +// testing/simulator/generation/property.rs — the assertion, 1138 and 1146-1147, 1162-1163 + 1138 if select_rows.len() != select_tlp_rows.len() { +// ... 1139-1144: report a row-count mismatch ... + 1145 // Check if any row in select_rows is not in select_tlp_rows + 1146 for row in select_rows.iter() { + 1147 if !select_tlp_rows.iter().any(|r| r == row) { +// ... 1148-1160: report "in select but not in select_tlp" ... + 1161 // Check if any row in select_tlp_rows is not in select_rows + 1162 for row in select_tlp_rows.iter() { + 1163 if !select_rows.iter().any(|r| r == row) { +``` + +That is **equal cardinality plus mutual set containment** — which is +strictly weaker than multiset equality. Work an example: + +``` + whole = {a, a, b} len 3, set {a, b} + parts = {a, b, b} len 3, set {a, b} + + line 1138 3 == 3 → pass + line 1146 every row of whole appears in parts → pass + line 1162 every row of parts appears in whole → pass + verdict: PASS — but a is duplicated on one side and b on the other. + + multiset equality would require count(a) = 2 on both sides. It does not. +``` + +A duplicate-multiplicity bug — a join that emits a row twice, a +`UNION ALL` that drops one copy of a duplicate — passes this +property. That is a real gap against the published oracle, and it is +not turso-specific: SQLancer's own comparator has the same shape +(`src/sqlancer/ComparatorHelper.java:91` size, then `:108-112` +`HashSet` equality — see +[reading-sqlancer.md](reading-sqlancer.md) Step 6). + +Why it matters: "we implemented TLP" is a claim about the queries; +whether you implemented the *oracle* is a claim about the +comparison. Read the comparison. + +### Step 6 — doublecheck: determinism itself is the cheapest oracle + +> **In:** one plan and one seed. +> **Out:** two `SimulatorEnv`s stepped in lockstep, compared +> per-interaction and then file-to-file. + +This oracle needs NO model of correct behaviour — only the promise +from Step 1. But it is not "run the plan twice and diff stdout". The +mechanism (`testing/simulator/runner/doublecheck.rs`) is: + +- Two environments, both turso. `main.rs:484-491` builds the second + with `env.clone_as(SimulationType::Default)` — this is *not* a + differential test against SQLite. +- They are advanced **in lockstep, interaction by interaction, in + one process** (`doublecheck.rs:104-176`), not run to completion + and diffed at the end. +- Each interaction's result values are compared as they are produced + (`compare_results`, `doublecheck.rs:178-201`; the mismatch report + is at line 193). +- At the end, the two on-disk database **files** are compared + byte-for-byte (`doublecheck.rs:56-73`). + +Lockstep is the design decision that matters: it localises the +divergence to the interaction that caused it rather than to the +whole run. Any divergence means nondeterminism leaked into the SUT — +`HashMap` iteration order, uninitialised memory, a hidden wall-clock +read (Step 1 listed two that are still open by construction), a stray thread. It costs one extra run and catches the class of bug that silently invalidates every *other* seed-based result. -Question: what class of bug does doublecheck catch that the model -oracle misses? (Hint: iteration order, uninitialized memory, hidden -wall-clock reads.) - -### Step 6 — shrinking and the bug base: from failure to reproducer to regression suite - -A failing seed typically produces a plan with hundreds of -interactions, most irrelevant. **Shrinking** (the `shrink/` module) -minimizes it: repeatedly delete chunks of the plan and re-run, -keeping any smaller plan that still fails — delta debugging — until -what remains is a minimal reproducer. Shrinking stateful op -sequences is harder than shrinking pure inputs because later ops -depend on state earlier ops created (drop the `CREATE TABLE` and -every subsequent statement changes meaning). - -Found bugs then persist in `runner/bugbase.rs` as seeds — the -regression suite is literally a list of u64s. Compare: our topic 15 -sim tests hardcode seeds 42/7/11/13. That's a bug base, four entries -long. + +Why it matters: every claim in this topic's `notes.md` baseline is +"same harness, different seed". If the harness isn't deterministic, +the whole table means nothing — so the cheapest oracle is the one +that checks the assumption the others rest on. + +### Step 7 — shrinking: greedy, linear, and honest about it + +> **In:** a failing plan of hundreds of interactions and the error +> string it produced. +> **Out:** a smaller plan that produces *the same error string*. + +A failing seed typically produces a plan where most interactions are +irrelevant. **Shrinking** minimises it. The previous version of this +chapter called turso's shrinker delta debugging; it is not, and the +code says so in its own comment. + +Phase 1, `shrink_interaction_plan` (`shrink/plan.rs:24-100`), is +purely static: truncate everything after the failing interaction, +then drop properties that don't touch the tables the failing +interaction depends on. **No re-runs at all.** Its comment at line +25 reads "this is a very naive implementation". + +Phase 2, `brute_shrink_interaction_plan` (`shrink/plan.rs:103-143`) +driving `iterative_shrink` (`146-173`), removes **one whole property +at a time, in reverse order**, re-runs, and keeps the removal only +if the shrunk plan reproduces the same failure. The equality that +defines "same failure" is a string compare on the error +(`test_shrunk_plan`, `shrink/plan.rs:175-201`; the test is +`e1 == e2` at line 198). + +The cost model follows directly, and it is not ddmin's: + +``` + n properties in the truncated plan + ddmin (Zeller & Hildebrandt): partition, halve, ~O(n log n) re-runs, + shrinks toward a 1-minimal subset + turso phase 2: one linear reverse pass, exactly n re-runs, + each keeping or discarding one property + + n = 200 properties → 200 re-runs, one pass. No second pass, so a + removal that only becomes possible after an earlier removal is + never found. +``` + +Shrinking stateful op sequences is harder than shrinking pure inputs +because later ops depend on state earlier ops created — drop the +`CREATE TABLE` and every subsequent statement changes meaning, which +is exactly why phase 1 reasons about table dependencies before phase +2 starts deleting. The string-equality criterion at line 198 is the +other honest limitation: a shrink that changes the error *message* +while preserving the bug is rejected. + +Why it matters: shrinking quality is the difference between a +reproducer a human will read and one they won't. Knowing it's a +single greedy pass tells you when to shrink again by hand. + +### Step 8 — the bug base: a directory per seed, not a list of seeds + +> **In:** a seed that failed, plus the CLI options that produced it. +> **Out:** a directory under `.bugbase` holding the plan, the shrunk +> plan, and the run history. + +The previous version of this chapter said "the regression suite is +literally a list of u64s". It isn't. `runner/bugbase.rs` locates a +`.bugbase` directory — searching the limbo project dir, then the +home dir, then the cwd (`bugbase.rs:132-158`) — and writes **one +directory per seed** containing `seed.txt`, `plan.sql`, +`shrunk.sql`, and `runs.json`. Each `BugRun` record +(`bugbase.rs:41-54`) carries the turso **commit hash**, a timestamp, +the error, the CLI options, and a `shrunk` flag. + +The commit hash is the interesting field: a seed alone does not +reproduce a bug, because the meaning of a seed changes whenever the +generator changes. Recording `(seed, commit, options)` is the +minimum tuple that reproduces. Our topic 15 sim tests hardcode seeds +42/7/11/13 — that is a bug base with one of the three fields, and +the `.bugbase` layout is what the other two look like. + +Why it matters: "we saved the seed" is a weaker claim than it +sounds. The seed indexes into a random stream whose *shape* is part +of the program. + +### Step 9 — outside the simulator: fuzzing and elle + +> **In:** the same repo, two sibling test harnesses. +> **Out:** an oracle that isn't turso-vs-turso, and a history format +> that isn't turso's at all. + +Two things live outside `testing/simulator/` and change what the +tree can find. + +`fuzz/fuzz_targets/expression.rs` (299 lines) is a **differential** +target, not a "doesn't crash" one. `do_fuzz` (lines 248-297) +evaluates the generated expression in in-memory SQLite (257-264) and +in turso (266-287) and `assert_eq!`s the two (289-294); expressions +deeper than 100 are rejected from the corpus (252-255). The +`fuzz_target!` macro invocation is the last line of the file, 299. +Sibling targets: `cast_real.rs`, `scalar_func.rs`, `schema.rs`. + +The corresponding restriction inside the simulator is worth knowing: +`Differential` mode disables fault injection outright +(`runner/env.rs:1341-1359` sets `profile.io.enable = false` with the +comment that faults can't be controlled on rusqlite), and also turns +off LIMIT and CREATE SEQUENCE generation. So in this tree, *faults +and a second implementation are mutually exclusive*. + +`testing/concurrent-simulator/elle.rs` (317 lines) emits histories +in **elle's list-append EDN format** for `elle-cli` to check: the +module doc (lines 1-7) names G0/G1/G2/G-Single from Adya's +formalism, `ElleOp` (18-30) is Append/Read/Write/RwRead, `to_edn` +(36-67) produces `[:append "key" v]` and `[:r "key" [1 2 3]]`, and +`ElleEventType` (72-79) is Invoke/Ok/Fail/Info. There is a +`.github/workflows/elle.yml` to run it. That is the exact format +Figure 2 of the Elle paper prints — see +[reading-jepsen.md](reading-jepsen.md) Step 4. + +And Antithesis is wired in for real: `Dockerfile.antithesis`, +`.github/workflows/antithesis.yml` (default **240-minute** +experiments, with an optional `diff_base` for targeted testing), +`scripts/antithesis/diff_to_targeted_coverage.py`, and workloads +under `testing/antithesis/` (`bank-test/`, `stress-composer/`, with +`anytime_validate.py` / `eventually_validate.py` / +`finally_validate.py`). + +Why it matters: the simulator is one of four harnesses in this repo, +and they cover different bug classes on purpose. Reading only +`testing/simulator/` will make you think turso has no ground-truth +oracle; it has two, they just live elsewhere. ## Where each step lives in the code @@ -159,35 +544,62 @@ The tree, top to bottom: ``` testing/simulator/ main.rs entry: seed → config → plan → execute → check + profiles/ + io.rs latency/tick/fault defaults (steps 2-3) runner/ clock.rs SimulatorClock — time is an RNG stream (step 2) io.rs SimulatorIO — fault injection switchboard (step 3) file.rs SimulatorFile — per-op faults + seeded latency (step 3) - execution.rs drive the plan, catch assertion failures (step 4) - doublecheck.rs run the same plan twice, diff outputs (step 5) - bugbase.rs known-bug corpus (regression seeds) (step 6) - generation/ plan/property/query generators (step 4) - model/ the in-memory oracle + interaction model (step 4) - shrink/ plan minimization (step 6) + env.rs SimulatorEnv — profiles, Differential mode (step 9) + doublecheck.rs two envs stepped in lockstep, then file diff (step 6) + bugbase.rs .bugbase — a directory per failing seed (step 8) + generation/ plan/property/query generators (steps 4-5) + model/ the in-memory oracle + Property enum (step 4) + shrink/ plan minimization (step 7) + fuzz/fuzz_targets/ differential fuzzing vs rusqlite (step 9) + testing/concurrent-simulator/elle.rs EDN histories for elle-cli (step 9) ``` | anchor | step | what it is | |---|---|---| -| runner/clock.rs:8-13 | 2 | `SimulatorClock { curr_time, rng: ChaCha8Rng, min_tick, max_tick }` | -| runner/clock.rs:25-34 | 2 | `now()` ADVANCES time by a seeded random tick — time is data | -| runner/io.rs:14 | 3 | `fault: Cell` — the injection master switch | -| runner/io.rs:64-77 | 3 | `inject_fault` / `inject_fault_selective` (per-file stem!) | -| runner/io.rs:135-138 | 3 | per-op fault counters: pread/pwrite/sync faults | -| runner/file.rs:40 | 3 | `latency_probability` — seeded IO delay | -| runner/file.rs:100-110 | 3 | `generate_latency_duration` — random_bool from the file's rng | -| runner/file.rs:149-233 | 3 | every op (read/write/sync) can be delayed into a `DelayedIo` queue | -| generation/property.rs:270 | 4 | `FsyncNoWait` / `FaultyQuery` — fault-flavored properties | -| generation/property.rs:276-282 | 4 | the metamorphic set: SelectSelectOptimizer, WhereTrueFalseNull, UnionAllPreservesCardinality, ReadYourUpdatesBack | -| fuzz/fuzz_targets/expression.rs:299 | — | `fuzz_target!(\|expr: Expr\|)` — STRUCTURED fuzzing via arbitrary (topic README §3, lives outside the simulator tree) | +| `runner/clock.rs:8-13` | 2 | `SimulatorClock { curr_time, rng: ChaCha8Rng, min_tick, max_tick }` | +| `runner/clock.rs:18` | 1 | `curr_time` seeded from the **real** `Utc::now()` | +| `runner/clock.rs:25-34` | 2 | `now()` advances by a seeded tick; line 31 is `from_micros` | +| `profiles/io.rs:45-53` | 2-3 | defaults: `latency_probability: 1`, `min_tick: 1`, `max_tick: 30` | +| `profiles/io.rs:70-78` | 3 | fault-enable defaults: read/write/sync all `true` | +| `runner/io.rs:14` | 3 | `pub(crate) fault: Cell` — the injection master switch | +| `runner/io.rs:64-80` | 3 | `inject_fault` / `inject_fault_selective` (per-file stem) | +| `runner/io.rs:114-122` | 1 | `impl Clock` — wall clock simulated, **monotonic clock real** | +| `runner/file.rs:19-34` | 3 | the six call/fault counters; `nr_sync_faults` at :34 | +| `runner/file.rs:40` | 3 | `pub latency_probability: u8` — a **percent** | +| `runner/file.rs:87-91` | 3 | `stats_table` prints a hard-coded `0` for sync faults | +| `runner/file.rs:99-109` | 3 | `generate_latency_duration` — `/100.0` at :103, `5..20` ms at :106 | +| `runner/file.rs:149-268` | 3 | the per-op delay blocks (pread 149, pwrite 175, sync 200, pwritev 233, truncate 257) | +| `runner/file.rs:192-199` | 3 | **sync faults are swallowed and cleared** at this revision | +| `runner/env.rs:1341-1359` | 9 | `Differential` mode disables fault injection | +| `model/property.rs:11-212` | 4 | the `Property` enum — 16 variants | +| `model/property.rs:39-53` | 4 | `ReadYourUpdatesBack` — success *and* rollback arms | +| `model/property.rs:142-148` | 4 | `SelectSelectOptimizer` — doc cites **NoREC** | +| `model/property.rs:153-160` | 4 | `WhereTrueFalseNull` — doc cites **TLP** | +| `generation/property.rs:1073-1083` | 5 | the three TLP partitions built as predicates | +| `generation/property.rs:1094-1115` | 5 | stitched with `UNION ALL` | +| `generation/property.rs:1138`, `1146-1177` | 5 | the assertion: size, then set containment both ways | +| `runner/doublecheck.rs:56-73` | 6 | final byte-for-byte database file comparison | +| `runner/doublecheck.rs:104-176` | 6 | the lockstep interaction loop | +| `runner/doublecheck.rs:178-201` | 6 | `compare_results`; mismatch reported at :193 | +| `main.rs:484-491` | 6 | `env.clone_as(SimulationType::Default)` — both sides are turso | +| `shrink/plan.rs:24-100` | 7 | phase 1: truncate + drop by table dependency, no re-runs | +| `shrink/plan.rs:103-173` | 7 | phase 2: one property at a time, reverse order, re-run each | +| `shrink/plan.rs:175-201` | 7 | `test_shrunk_plan` — "same failure" is `e1 == e2` at :198 | +| `runner/bugbase.rs:41-54` | 8 | `BugRun` — commit hash, timestamp, error, options, `shrunk` | +| `runner/bugbase.rs:132-158` | 8 | `.bugbase` directory discovery | +| `fuzz/fuzz_targets/expression.rs:248-297` | 9 | differential against in-memory SQLite | +| `fuzz/fuzz_targets/expression.rs:299` | 9 | `fuzz_target!(\|expr: Expr\| -> Corpus {...})` | +| `testing/concurrent-simulator/elle.rs:36-67` | 9 | `to_edn` — elle's list-append history format | Reading order: follow the anchor map top to bottom — clock, then IO -and file (the fault switchboard), then the properties, then -doublecheck and shrink. +and file (the fault switchboard), then the properties and the TLP +assertion, then doublecheck, shrink, and the bug base. ## Questions for notes.md @@ -200,23 +612,200 @@ doublecheck and shrink. 4. The shrink/ module: why is shrinking HARDER for stateful op sequences than for pure inputs (proptest's integrated shrinking vs delta debugging)? -5. For M16: which three properties from generation/property.rs port +5. For M16: which three properties from `model/property.rs` port directly to Cypher? Sketch the graph equivalents. ## Done when -- [ ] You can explain what it takes to make a program a pure function of a seed, and name the three sources of nondeterminism that must be captured. -- [ ] You can say why ChaCha8 rather than the default RNG, and what property DST needs from its generator. -- [ ] You can describe fault injection at the file layer and why targeting the WAL separately from the db file matters. -- [ ] You can explain the doublecheck oracle — determinism itself — and why it is the cheapest one available. -- [ ] You can say why shrinking stateful op sequences is harder than shrinking values, and connect it to this topic's `shrink.rs` exercise. +Answer each before unfolding it. + +- [ ] You can explain what it takes to make a program a pure function of a seed, and name the three sources of nondeterminism that must be captured — plus the two turso still leaks. + +
Answer + + Time, IO (results *and* timing), and scheduling. Each must be reached + only through a trait whose test implementation draws from one seeded + RNG (Step 1). turso does this for the wall clock (`clock.rs`) and for + files (`file.rs`). + + The two leaks are both visible in the code. `impl Clock for + SimulatorIO` (`runner/io.rs:114-122`) returns the real + `MonotonicInstant::now()` from `current_time_monotonic` — only + `current_time_wall_clock` is simulated. And `SimulatorClock::new` + seeds `curr_time` from the real `Utc::now()` (`clock.rs:18`), so + absolute timestamps differ run to run; only deltas reproduce. + +
+ +- [ ] You can say why ChaCha8 rather than the default RNG, and why reading the clock is not a side-effect-free operation. + +
Answer + + `rand`'s default generator is explicitly allowed to change between + releases. If it did, every archived seed in `.bugbase` would start + meaning a different execution — the regression corpus would silently + evaporate. ChaCha8 is named and versioned (`clock.rs:5, 10`), so a + seed keeps its meaning. + + `now()` (`clock.rs:25-34`) mutates `curr_time` *and* consumes a draw + from the RNG (line 30). So adding a clock read anywhere — even inside + a log line — shifts every subsequent random decision in the run. The + seed still reproduces, but it reproduces a different execution than + it did yesterday. + +
+ +- [ ] You can describe fault injection at the file layer, quantify the latency injection, and say which fault does *not* fire at `dd775bc`. + +
Answer + + Two mechanisms. A `fault: Cell` flag (`io.rs:14`) armed by + `inject_fault` / `inject_fault_selective` (`io.rs:64-80`), the latter + matching a file stem so the WAL can be faulted independently of the + database file. And a latency path: `generate_latency_duration` + (`file.rs:99-109`) fires with `latency_probability / 100.0` — the + default of `1` (`profiles/io.rs:45-53`) is therefore **1%** — and on + a hit defers the operation by `U[5, 20)` ms (line 106) into the + `DelayedIo` queue. + + Mean delay per operation is `0.01 × 12.5 ms = 125 µs`, against a mean + simulated clock tick of 15 µs — so one injected delay is worth about + 830 ticks of reordering. + + **`sync` faults do not fire.** `file.rs:192-199` checks the flag, + logs "ignoring sync fault because it causes false positives with + current simulator design", and clears it. `nr_sync_faults` + (`file.rs:34`) is never incremented and `stats_table` hard-codes a + zero (`file.rs:87-91`) — even though `profiles/io.rs:70-78` defaults + `sync: true`. + +
+ +- [ ] You can explain the doublecheck oracle, and say precisely what it compares and when. + +
Answer + + It runs two `SimulatorEnv`s — both turso, built by + `env.clone_as(SimulationType::Default)` at `main.rs:484-491` — in + **lockstep, interaction by interaction, inside one process** + (`doublecheck.rs:104-176`). Result values are compared as each + interaction completes (`compare_results`, `:178-201`, mismatch at + `:193`), and at the end the two database *files* are compared + byte-for-byte (`:56-73`). + + It needs no model of correctness — only the determinism promise from + Step 1 — and it catches the class of bug (`HashMap` order, hidden + clock reads, uninitialised memory, stray threads) that silently + invalidates every *other* seed-based result. Lockstep is the point: + it localises divergence to the interaction that caused it. + +
+ +- [ ] You can say why turso's TLP property is weaker than the published TLP oracle, and construct an input that slips through. + +
Answer + + The paper's identity uses `⊎`, multiset addition (Rigger & Su, + OOPSLA 2020, Table 1, WHERE row). turso builds the partitions + correctly (`generation/property.rs:1073-1083`, `UNION ALL` at + `1094-1115`) but then checks **equal length** (`:1138`) plus **set + containment in both directions** (`:1146-1177`). + + `{a, a, b}` versus `{a, b, b}`: both have length 3, both have set + `{a, b}`, every element of each appears in the other. It passes. A + duplicate-multiplicity bug is invisible. + + SQLancer's comparator has the same shape — size at + `ComparatorHelper.java:91`, then `HashSet` equality at `:108-112`. + Two independent implementations of the same paper, the same gap. + +
+ +- [ ] You can say why shrinking stateful op sequences is harder than shrinking values, and describe turso's actual algorithm and its cost. + +
Answer + + Later ops depend on state earlier ops created: delete the + `CREATE TABLE` and every following statement changes meaning, so you + cannot treat the plan as an unordered bag of independent elements. + + turso's shrinker is *not* delta debugging. Phase 1 + (`shrink/plan.rs:24-100`, comment at :25 calling itself "very + naive") truncates after the failing interaction and statically drops + properties that don't touch the depended-on tables, with **no + re-runs**. Phase 2 (`:103-173`) removes one whole property at a time + in reverse order, re-running each time, and keeps a removal only if + the error string still matches exactly (`e1 == e2`, `:198`). + + Cost: exactly `n` re-runs for `n` properties, in a single pass — + where ddmin would do ~O(n log n) and reach a 1-minimal subset. A + removal that only becomes possible after an earlier removal is never + found, and a shrink that changes the error *message* is rejected + even if it preserves the bug. + +
+ +- [ ] You can describe what `.bugbase` actually stores, and say why a seed alone is not a reproducer. + +
Answer + + One **directory per failing seed** (`runner/bugbase.rs:132-158` + finds `.bugbase` under the project dir, then home, then cwd), + holding `seed.txt`, `plan.sql`, `shrunk.sql`, and `runs.json`. Each + `BugRun` (`:41-54`) records the turso **commit hash**, a timestamp, + the error, the CLI options, and whether the plan was shrunk. + + A seed indexes into a random stream whose *shape* is defined by the + generator. Change the generator — add a property, reorder a match + arm, add a clock read — and the same seed produces a different plan. + The minimum reproducing tuple is `(seed, commit, options)`, which is + exactly what `BugRun` stores. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the three properties you will port. +
Answer + + No unfoldable answer — this one is the writing. The `Property` table + in Step 4 is the menu; the ones that port to a graph engine without + a SQL optimizer to lean on are the state-based ones + (`TableHasExpectedContent`, `DeleteSelect`, `DropSelect`, + `SavepointRollback`), while `WhereTrueFalseNull` needs a graph + answer to "what is NULL here" — a missing property on a node, per + [reading-sqlancer.md](reading-sqlancer.md) question 5. + +
+ ## References **Code** -- [turso](https://github.com/tursodatabase/turso) — +- [turso](https://github.com/tursodatabase/turso) @ `dd775bc` — `testing/simulator/` (clock/io/file fault injection, interaction - plans, properties, doublecheck, shrink) plus - `fuzz/fuzz_targets/expression.rs` for structured fuzzing via - `arbitrary` — clone it; the anchor map above is your reading order + plans, properties, doublecheck, shrink), `fuzz/fuzz_targets/` for + differential fuzzing against rusqlite, and + `testing/concurrent-simulator/elle.rs` for elle-format histories + +| File | Lines | What | +|---|---|---| +| `testing/simulator/runner/clock.rs` | 8-13, 18, 25-34 | the entire simulated clock (35 lines) | +| `testing/simulator/profiles/io.rs` | 45-53, 70-78 | tick range, latency probability, fault-enable defaults | +| `testing/simulator/runner/io.rs` | 14, 64-80, 114-122 | fault switch, injection, the partial `Clock` impl | +| `testing/simulator/runner/file.rs` | 19-34, 40, 87-91, 99-109, 149-268, 192-199 | counters, latency, per-op delay, the disabled sync fault | +| `testing/simulator/model/property.rs` | 11-212 | the 16-variant `Property` enum with its doc comments | +| `testing/simulator/generation/property.rs` | 1073-1083, 1094-1115, 1138-1177 | TLP partition construction and its weaker-than-`⊎` assertion | +| `testing/simulator/runner/doublecheck.rs` | 56-73, 104-176, 178-201 | file diff, lockstep loop, per-interaction compare | +| `testing/simulator/shrink/plan.rs` | 24-100, 103-173, 175-201 | two-phase greedy shrinker | +| `testing/simulator/runner/bugbase.rs` | 41-54, 132-158 | `BugRun` records and `.bugbase` layout | +| `testing/simulator/runner/env.rs` | 1341-1359 | `Differential` mode disables faults | +| `fuzz/fuzz_targets/expression.rs` | 248-297, 299 | differential fuzz target vs in-memory SQLite | +| `testing/concurrent-simulator/elle.rs` | 1-7, 18-30, 36-67, 72-79 | elle list-append EDN histories | + +**Papers** +- Rigger & Su — "Finding Bugs in Database Systems via Query + Partitioning" (OOPSLA 2020) — Table 1 is the identity turso's + `WhereTrueFalseNull` implements; see + [reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md) +- Kingsbury & Alvaro — "Elle: Inferring Isolation Anomalies from + Experimental Observations" (VLDB 2020) — Figure 2 is the format + `elle.rs` emits; see [reading-jepsen.md](reading-jepsen.md) diff --git a/topics/16-testing-correctness/reading-z3.md b/topics/16-testing-correctness/reading-z3.md index b7395b6..47a038d 100644 --- a/topics/16-testing-correctness/reading-z3.md +++ b/topics/16-testing-correctness/reading-z3.md @@ -9,6 +9,20 @@ Cosette-style to verify our topic-10 rewrite rules. Read Z3 the way PLAN.md says to: as a masterclass high-performance search engine over LOGIC whose architecture rhymes with a query engine. +**Scope, and the split with topic 21.** Z3's internals — the CDCL(T) +loop, Nelson–Oppen theory combination, the e-graph, E-matching — are +topic 21's subject and are walked at length in +[topics/21-formal/reading-z3-tacas08.md](../21-formal/reading-z3-tacas08.md). +This chapter is about *using* a solver as a test oracle: Step 1 +compresses the architecture to the parts you must hold to read an +`unsat`, and Steps 2 onward are all encoding, which topic 21 does not +cover. If a claim here about Z3's insides feels thin, that is +deliberate — the deep version is one link away. + +Every code anchor is `Z3Prover/z3` at commit **`1d425e5`**, the +revision this repo pins in `resources/codebases.md`, with the line +numbers it occupies there. + ## The problem in one sentence A fuzzer that runs 10 million random rows through two query plans @@ -19,71 +33,131 @@ counterexample row, usually in milliseconds. ## The concepts, step by step -### Step 1 — SAT: search over boolean assignments +### Step 1 — SAT, CDCL, SMT, tactics: the four ideas you need to read an answer + +> **In:** a formula over booleans and theory atoms like `x < 3`. +> **Out:** `sat` with a model, `unsat`, or `unknown` — and you need +> to know why each is possible. -SAT (boolean satisfiability) is the question "given a formula over -true/false variables, is there an assignment making it true?" A SAT -**solver** is a search engine over the 2^n assignments — and modern -ones routinely handle formulas with millions of variables because -the search is ruthlessly pruned. Flip the answer around and you get -verification: to prove a property P holds always, ask the solver for -a case where `NOT P` holds. **UNSAT** ("no satisfying assignment -exists") is then a proof; **SAT** hands you a concrete +**SAT** (boolean satisfiability): given a formula over true/false +variables, is there an assignment making it true? A SAT **solver** +is a search engine over the `2^n` assignments. Flip the answer around +and you get verification: to prove a property P holds always, ask +the solver for a case where `NOT P` holds. **UNSAT** ("no satisfying +assignment exists") is then a proof; **SAT** hands you a concrete counterexample. That inversion — prove by failing to find — is the whole chapter. -### Step 2 — CDCL: the search loop that learns from every dead end - -CDCL (conflict-driven clause learning) is the algorithm inside every -modern SAT solver: guess a variable (**decide**), push the logical -consequences (**propagate**), and when a contradiction appears -(**conflict**), *analyze why*, record the reason as a new learned -clause so the same dead end is never entered again, and jump back -(**backjump**) past the guesses the conflict proved irrelevant. The -learned clauses are why CDCL beats brute force by orders of -magnitude: every failure permanently shrinks the remaining search -space. The DB analogy runs deep — CDCL is adaptive execution with -feedback, and learned clauses are materialized negative results. - -### Step 3 — SMT: SAT proposes, theories veto - -Real verification needs more than booleans — integers, arrays, bit -patterns. SMT (satisfiability modulo theories) keeps the CDCL engine -and attaches **theory solvers**, each a decision procedure for one -domain (linear arithmetic, bitvectors, arrays, uninterpreted -functions, strings): - -``` - SAT solver: boolean skeleton (CDCL: decide → propagate → - conflict → learn clause → backjump) - + - theory solvers: linear arithmetic, bitvectors, arrays, - uninterpreted functions, strings... - = - SMT: SAT proposes boolean assignments; theories veto with - conflict explanations ("x<3 ∧ x>5 is impossible") that - become learned clauses -``` - -The SAT core treats `x < 3` as an opaque boolean; when it proposes -`x < 3 ∧ x > 5`, the arithmetic theory vetoes with an explanation -that becomes a learned clause. Theory propagation is predicate -pushdown into specialized engines — same shape as topic 10. - -### Step 4 — tactics: query plans for proofs - -Z3 doesn't run one fixed algorithm; it composes **tactics** — -transformers that rewrite a goal (simplify, eliminate equalities, -blast bitvectors to SAT) — into pipelines, chosen by **probes** that -inspect the formula first. `(then simplify solve-eqs bit-blast sat)` -is a pipeline of rewrites ending in an executor, and -`default_tactic.cpp` dispatches on the detected logic the way a -planner dispatches on statistics — probes are cardinality -estimation for proofs. This is why PLAN.md calls Z3 a query engine -for logic: the architecture is parse → rewrite → cost-informed -dispatch → execute. - -### Step 5 — symbolic rows: encoding a query plan as a formula +**CDCL** (conflict-driven clause learning) is the algorithm inside: +guess a variable (**decide**), push the consequences (**propagate**), +and on a contradiction (**conflict**) analyze why, record the reason +as a **learned clause** so the dead end is never re-entered, and +**backjump** past the guesses the conflict proved irrelevant. + +**SMT** (satisfiability modulo theories) keeps the CDCL engine and +attaches **theory solvers**, each a decision procedure for one domain +(linear arithmetic, bitvectors, arrays, uninterpreted functions, +strings). The SAT core treats `x < 3` as an opaque boolean; when it +proposes `x < 3 ∧ x > 5`, the arithmetic theory vetoes with an +explanation that becomes a learned clause. Theory propagation is +predicate pushdown into specialized engines — same shape as topic 10. + +**Tactics** are the fourth idea, and the one that most rewards a +look at the source, because it is where the query-engine analogy +stops being an analogy. Z3 doesn't run one fixed algorithm; it +composes goal transformers into pipelines, chosen by **probes** that +inspect the formula first: + +```c +// src/tactic/portfolio/default_tactic.cpp — mk_default_tactic, 36-55 (elided) + 36 tactic * mk_default_tactic(ast_manager & m, params_ref const & p) { + 37 tactic * st = using_params(and_then(mk_simplify_tactic(m, p), + 38 cond(mk_and(mk_is_propositional_probe(), mk_not(mk_produce_proofs_probe())), + 39 mk_lazy_tactic(m, p, [&](auto& m, auto const& p) { return mk_fd_tactic(m, p); }), + 40 cond(mk_is_qfbv_probe(), ... mk_qfbv_tactic ... + 42 cond(mk_is_qflia_probe(), ... mk_qflia_tactic ... +// ... 41, 43-50: qfaufbv, qfauflia, qflra, qfnra, qfnia, lira, nra, qffp, qffplra ... + 52 and_then(mk_preamble_tactic(m), mk_lazy_tactic(m, p, [&](auto& m, auto const& p) { return mk_smt_tactic(m, p);}))))))))))))))), + 53 p); + 54 return st; + 55 } +``` + +Fifty-six lines, and twelve `cond(probe, specialised_tactic)` +branches at `:38-50` before the general fallback at `:52`. Every +branch is "if the formula is in *this* fragment, use the engine built +for it". `mk_is_qflia_probe()` at line 42 detects quantifier-free +linear integer arithmetic — the fragment Step 3's encoding lands in — +and dispatches to a solver that will not pay for anything QF_LIA +doesn't need. `mk_lazy_tactic` means the specialised tactic isn't +even *constructed* unless its probe fires. + +That is parse → rewrite (`mk_simplify_tactic`, line 37) → +cost-informed dispatch (the probe cascade) → execute. A planner +dispatching on statistics has the same shape, and probes are its +cardinality estimation. + +Why it matters for *this* topic: the twelve branches are the reason +Step 3's encoding is fast. Land in a named fragment and you get a +specialist; land outside one and you get `mk_smt_tactic`, the general +engine, which is where `unknown` answers come from. + +For the CDCL(T) loop itself, the e-graph, Nelson–Oppen and +E-matching, go to +[topics/21-formal/reading-z3-tacas08.md](../21-formal/reading-z3-tacas08.md) +— that chapter reads `src/ast/euf/euf_egraph.h` and `euf_mam.h` line +by line. Don't duplicate the work. + +### Step 2 — the API surface is three calls + +> **In:** a formula you have built. +> **Out:** `l_true` / `l_false` / `l_undef`, and — on `l_true` — a +> model. + +You do not need to understand Z3 to use Z3. The entire testing +interface is visible in one header: + +```c +// src/solver/solver.h — the assert / check surface, 124 and 177-183 (elided) + 124 void assert_expr(expr* f); +// ... 126: assert_expr_core, the virtual each backend implements ... +// ... 128-130: the vector overload, a loop over the scalar one ... + 177 lbool check_sat(unsigned num_assumptions, expr * const * assumptions); +// ... 179-181: expr_ref_vector / app_ref_vector convenience overloads ... + 183 lbool check_sat() { return check_sat(0, nullptr); } +``` + +Push formulas with `assert_expr`, ask with `check_sat`. The return +type is `lbool`, a **three**-valued boolean — `l_true` (sat), +`l_false` (unsat), `l_undef` (unknown: resource limit, timeout, or a +fragment Z3 cannot decide). The doc comment at `:172-174` names the +other half of the contract: on unsat with core generation enabled, +"the unsat-core is a subset of these assumptions" — which is how you +find out *which* of your assumptions did the proving. + +Do not skip `l_undef`. Line it up against the topic's other oracles: + +``` + crash_matrix (this topic): bug found / not found this seed + "not found" ≠ "not present" + TLP: partitions reconcile / don't + Z3 check_sat: l_false = PROVEN for all inputs + l_true = counterexample in hand + l_undef = you learned nothing + + A harness that treats l_undef as l_false silently converts + "the solver gave up" into "the rewrite is correct." +``` + +Why it matters: this is the only failure mode in this chapter that +is *silent*, and it is one `if` away in every solver harness anyone +writes. + +### Step 3 — symbolic rows: encoding a query plan as a formula + +> **In:** two filter chains you believe are equivalent. +> **Out:** one formula whose unsatisfiability is a proof over every +> row that could ever exist. To verify a rewrite rule, replace concrete data with one **symbolic row** — a tuple of solver variables, one per column — and compile @@ -99,8 +173,10 @@ question: SAT → the model IS the counterexample row ``` -```rust -// verify a rewrite for ALL rows by asking for ONE disagreeing row +```text +// ILLUSTRATION — the shape of the harness you write in this topic's +// z3 rewrite exercise. Not quoted from Z3; the C++ calls it bottoms +// out in are src/solver/solver.h:124 (assert_expr) and :177 (check_sat). let a = Int::fresh("a"); let a_null = Bool::fresh("a_null"); let b = Int::fresh("b"); let b_null = Bool::fresh("b_null"); let row = Row { a, a_null, b, b_null }; @@ -111,54 +187,190 @@ let p2 = compile(plan_after, &row); match solver.check(p1.keeps_row().xor(p2.keeps_row())) { Unsat => Proven, // no row distinguishes the plans Sat(m) => Counterexample(m), // the model IS the failing row + Unknown => Inconclusive, // NOT Proven — see Step 2 } ``` -One subtlety keeps this tractable: filters are row-at-a-time pure -logic, so one symbolic row quantifies over all databases — -no quantifiers needed, and quantifier-free formulas are Z3's fast -path. +Now the arithmetic that justifies the whole approach. Compare +exhaustively testing a two-column filter against solving it: + +``` + columns: a, b — each a 64-bit nullable integer + rows to enumerate = (2^64 + 1)^2 ≈ 3.4 × 10^38 + + at crash_matrix's measured rate (this topic's notes.md baseline): + ≈ 200,000 harness runs per second + 3.4 × 10^38 / 2 × 10^5 ≈ 1.7 × 10^33 s ≈ 5 × 10^25 years -### Step 6 — the NULL trap: encode SQL's three-valued logic honestly + the same question as one QF_LIA query: one check_sat, milliseconds. + + and a fuzzer that samples 10^7 of those rows covers + 10^7 / 3.4 × 10^38 ≈ 3 × 10^-32 of the space — measure zero. +``` + +That is not a speedup, it is a change of category: the solver never +enumerates rows, it reasons about the *constraint* the rows satisfy. + +One subtlety keeps this tractable: filters are row-at-a-time pure +logic, so one symbolic row quantifies over all databases. There is +no "for all rows" quantifier in the formula — the universal +quantification is in the *interpretation* ("a free variable stands +for an arbitrary value"), not in the syntax. That matters because +quantifier-free formulas are Z3's fast path: they land in Step 1's +`mk_is_qflia_probe` branch (`default_tactic.cpp:42`) instead of the +general `mk_smt_tactic` fallback at `:52`, and they avoid E-matching +entirely — which is where topic 21's `euf_mam.h` and the +trigger-selection heuristics come in, and where `l_undef` starts +appearing. + +Why it matters: "no quantifiers needed" is the single design +decision that makes a solver a practical test oracle rather than a +research project. + +### Step 4 — the NULL trap: encode SQL's three-valued logic honestly + +> **In:** a nullable column. +> **Out:** two solver variables, not one — and an operator table +> you have to write out. SQL predicates evaluate to TRUE, FALSE, or NULL ("unknown"), and -WHERE keeps only TRUE — so a two-valued encoding proves rewrites +`WHERE` keeps only TRUE — so a two-valued encoding proves rewrites that are false in real SQL. The honest encoding: each nullable -column becomes a pair (value, is_null), and AND/OR/NOT/comparison -are defined per SQL's Kleene semantics (NULL AND FALSE = FALSE, NULL -AND TRUE = NULL, …). This is the trap AND the point: most real -optimizer bugs (TLP's bread and butter — reading-pqs-tlp-papers.md) -are exactly NULL-semantics violations, and Z3 finds them as SAT -models in milliseconds. +column becomes a pair `(value, is_null)`, and AND/OR/NOT/comparison +are defined per SQL's Kleene semantics. Write the tables out once; +they are the specification: -### Step 7 — Cosette: the full SQL-equivalence prover +``` + NOT: T→F F→T N→N + + AND T F N OR T F N + T T F N T T T T + F F F F F T F N + N N F N N T N N + + note the two asymmetries that catch people: + F AND N = F (falsity is absorbing — you need not know the other side) + T OR N = T (truth is absorbing) + a two-valued encoder gets both of these wrong in the direction + of "propagate the unknown", which is a STRICTER filter than SQL's — + so it proves rewrites SQL does not satisfy. +``` + +Encoding cost, so you know what you are buying: + +``` + n nullable columns + two-valued encoding: n solver variables + honest encoding: 2n variables + 1 is_null term per comparison + blow-up: 2× in variables, ~2× in formula size + + for the 2-column filter of Step 3 that is 4 variables instead of 2. + QF_LIA solves both in milliseconds. There is no reason to cheat. +``` + +This is the trap AND the point: most real optimizer bugs — TLP's +bread and butter, [reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md) +— are exactly NULL-semantics violations, and Z3 finds them as SAT +models in milliseconds. Note the pleasing symmetry with TLP: TLP +partitions on `p` / `NOT p` / `p IS NULL` because SQL has three +truth values; the solver encoding needs `(value, is_null)` for the +same reason. Same fact, two techniques. + +Why it matters: an encoder that is wrong here does not fail loudly. +It returns `unsat` — a *proof* — of something false. + +### Step 5 — Cosette: the full SQL-equivalence prover + +> **In:** two arbitrary SQL queries, not two filter chains. +> **Out:** a counterexample, a proof, or neither — from two +> different engines, because no single one can do both. Cosette answers "are Q1 and Q2 equivalent for ALL databases?" — the -general problem, beyond single-row filters. It compiles SQL to -**K-relations** (relations where each row carries a multiplicity, so -bag/duplicate semantics work — SQL tables are bags, not sets), then -splits: easy fragments → SMT for counterexamples, hard equivalences -→ Coq proof search over HoTT encodings. Our use is the SMT half: -filters and projections over symbolic rows, exactly Steps 5–6, which -is all topic 10's rewrite rules need. +general problem, beyond single-row filters. Start with the fact that +frames it, which the paper states outright: query equivalence for +arbitrary SQL is **undecidable**, "so an automated proof system for +SQL will never be complete." Everything about Cosette's architecture +follows from that. + +It compiles SQL to **K-relations**: a relation is a *function from +tuple to multiplicity*, so bag semantics (SQL tables are bags, not +sets) fall out of the algebra rather than being bolted on. Union is +addition of multiplicities, join is multiplication, selection +multiplies by a 0/1 indicator. + +Then it splits — and the split is **by outcome, not by difficulty**, +which is the part everyone gets backwards: + +``` + constraint solver (Rosette) can only ever DISPROVE + → finds counterexamples + proof assistant (Coq) can only ever PROVE + → establishes equivalences + + neither can do the other's job. Running both is not a + "fast path / slow path" — it is two half-oracles. +``` + +The solver half is bounded, which is the honest reading of this +chapter's title. §3.1: `Tuple := List`, `Relation := +List>`; strings are modelled as integers and +floats are unsupported; symbolic relations are **fixed-size** lists +of symbolic values (including symbolic multiplicities), grown by +incremental solving. That is bounded model checking: "testing every +input at once" holds *up to the current bound on relation size*, not +absolutely. Step 3's single-symbolic-row encoding is the degenerate +case where the bound is 1 and therefore exact — because a row filter +cannot see other rows. + +The implementation is small enough to be encouraging: about **3k +lines of Rosette and 2k lines of Coq** (Rosette 2.2, Coq 8.5pl1). +And the results are honest about the split: §6 reports that the +solver found counterexamples for every query in the Bugs, Exams and +XData benchmarks it was pointed at, while on the Rules benchmark of +23 known-equivalent rewrite rules Coq **automatically proved 17** +(7 of them via the `CQSolve` tactic) and the remaining 6 needed +human interaction. + +``` + Rules benchmark, §6: + 23 rewrite rules known to be equivalent + 17 proved automatically → 17 / 23 = 73.9% + 6 needed interactive proof → 6 / 23 = 26.1% + + read that as the price list for the general problem. Step 3's + filter rules are in the 73.9% — and in fact below it, since a + quantifier-free single-row encoding needs no proof assistant at all. +``` + +Our use is the SMT half: filters and projections over symbolic rows, +exactly Steps 3–4, which is all topic 10's rewrite rules need. + +Why it matters: knowing that the general problem is undecidable is +what stops you trying to build Cosette. Knowing that *your* fragment +is quantifier-free and single-row is what tells you your version is +a weekend. ## Where each step lives in the code | anchor | step | what it is | |---|---|---| -| src/solver/solver.h:58 | 1 | `class solver` — check_sat over assertions | -| src/smt/smt_context.h:89 | 2–3 | `smt::context` — the CDCL(T) core loop | -| src/tactic/tactic.h:34 | 4 | `class tactic` — composable transformers | -| src/tactic/portfolio/default_tactic.cpp | 4 | the default strategy: probe → dispatch by logic | -| src/tactic/portfolio/smt_strategic_solver.cpp | 4 | tactic → solver bridge | -| src/ast/ | — | hash-consed terms (one node per distinct expr — topic 2's interning) | -| src/smt/mam.cpp | — | matching abstract machine for quantifier triggers — a compiled pattern matcher (topic 19 vibes) | - -Reading order: `solver.h` for the public shape, `smt_context.h` for -the CDCL(T) loop (don't read it all — find decide/propagate/ -conflict), then the tactic machinery. The `src/ast/` hash-consing -and `mam.cpp` are optional side quests that rhyme with topics 2 -and 19. +| `src/solver/solver.h:58` | 1 | `class solver : public check_sat_result` — the public shape | +| `src/solver/solver.h:124` | 2 | `assert_expr` — push a formula onto the assertion stack | +| `src/solver/solver.h:172-174` | 2 | the unsat-core contract, in the doc comment | +| `src/solver/solver.h:177-183` | 2 | `check_sat` and its overloads; returns three-valued `lbool` | +| `src/smt/smt_context.h:89` | 1 | `class context` — the CDCL(T) core loop (1,980 lines; topic 21 reads it) | +| `src/tactic/tactic.h:34` | 1 | `class tactic` — composable goal transformers | +| `src/tactic/portfolio/default_tactic.cpp:36-55` | 1 | twelve `cond(probe, tactic)` branches, then the `smt` fallback | +| `src/tactic/portfolio/smt_strategic_solver.cpp` | 1 | tactic → solver bridge | +| `src/ast/` | — | hash-consed terms (one node per distinct expr — topic 2's interning) | +| `src/smt/mam.cpp` | — | the matching abstract machine for quantifier triggers — 4,042 lines you do **not** need for Step 3's quantifier-free encoding; topic 21 covers its modern sibling `src/ast/euf/euf_mam.h` | + +Reading order for *this* topic: `solver.h` for the three calls in +Step 2, then `default_tactic.cpp` in full — it is 56 lines and it is +the query-planner analogy made literal. Stop there. `smt_context.h` +and the e-graph belong to +[topics/21-formal/reading-z3-tacas08.md](../21-formal/reading-z3-tacas08.md); +reading them twice is not twice as useful. ## Questions for notes.md @@ -181,24 +393,204 @@ and 19. ## Done when +Answer each before unfolding it. + - [ ] You can explain CDCL as a search loop that learns, and say what a learned clause is. + +
Answer + + Decide (guess a variable's value) → propagate (push forced + consequences) → conflict (some clause is now falsified) → analyze + the conflict to find the subset of decisions responsible → record + their negation as a **learned clause** → backjump past every + decision the analysis proved irrelevant, not merely the last one. + + A learned clause is a fact derived from the input formula that was + not stated in it — logically redundant, operationally decisive, + because it prunes that entire region of the search space forever. + It is a materialized negative result, and the reason CDCL beats + brute force by orders of magnitude on structured formulas. + + The database rhyme: adaptive execution with feedback. The full + treatment, with Z3's actual data structures, is in + [topics/21-formal/reading-z3-tacas08.md](../21-formal/reading-z3-tacas08.md). + +
+ - [ ] You can state the SAT/theory division of labour: SAT proposes, theories veto. + +
Answer + + The CDCL core sees only a **boolean skeleton**: `x < 3` and `x > 5` + are two opaque propositional variables. It proposes an assignment + making the skeleton true. + + Each theory solver then checks its own atoms for consistency in its + domain. Linear arithmetic looks at `x < 3 ∧ x > 5`, declares it + infeasible, and hands back an **explanation** — a minimal + inconsistent subset — which becomes a learned clause in the core. + The core backjumps and proposes differently. + + So the theory never searches and the core never does arithmetic. + Theory propagation — a theory *deducing* an atom's value rather + than merely rejecting an assignment — is the same move as predicate + pushdown in topic 10: send the constraint to the engine that can + evaluate it cheaply, instead of filtering after the fact. + +
+ - [ ] You can encode a small query plan as symbolic rows and say what the formula asserts. + +
Answer + + One symbolic row: a fresh solver variable per column (plus an + `is_null` companion per nullable column, Step 4). Compile each + plan's filter chain into a boolean formula over those variables — + `P1(row)`, `P2(row)` — each meaning "this plan keeps this row". + Assert `P1 XOR P2` and call `check_sat`. + + The formula asserts *there exists a row on which the two plans + disagree*. `l_false` (unsat) means no such row exists in the entire + domain of the variables, which is a proof of equivalence for every + possible database. `l_true` means the model is literally the + counterexample row — print it and you have a bug report. `l_undef` + means you learned nothing (Step 2). + + The universal quantification is in the interpretation of a free + variable, not in the syntax, so the formula stays quantifier-free + and lands in `default_tactic.cpp:42`'s QF_LIA branch. + +
+ - [ ] You can encode `WHERE NOT (a = b)` against `WHERE a <> b` over nullable columns and show where they differ. + +
Answer + + They are equivalent — and that is the *interesting* answer, because + the naive worry is wrong for a reason worth naming. + + Under Kleene semantics with `a` NULL: `a = b` evaluates to NULL, + `NOT NULL` is NULL, and `WHERE` drops the row. `a <> b` also + evaluates to NULL, and `WHERE` drops the row. Same on both sides; + the row is dropped either way. With both non-null the two are + ordinary boolean negations of each other. So the encoding gives + `unsat`. + + Where the equivalence *does* break is the moment the predicate + stops being a top-level `WHERE`: put it inside `NOT EXISTS`, a + `CHECK` constraint, or a `CASE`, and the difference between "NULL" + and "not TRUE" becomes observable. Which is the general lesson: + Kleene expressions are only interchangeable relative to a *context* + that collapses NULL and FALSE the same way. `WHERE` does. Not + everything does. + + Run it and check rather than trusting this paragraph — that is what + the exercise is for. + +
+ - [ ] You can say why Cosette needs bags (K-relations) rather than sets, and which SQL feature forces it. + +
Answer + + Because SQL tables are bags: `SELECT` without `DISTINCT` preserves + duplicates, and `UNION ALL` adds multiplicities. A K-relation makes + that primitive — a relation is a function from tuple to + multiplicity, so union is addition and join is multiplication. + + The rewrite that forces it: pushing `DISTINCT` (or dropping it) is + valid under set semantics and invalid under bag semantics, as is + any rule that changes how many times a row is produced — + reassociating a join that duplicates rows, or eliminating a + self-join. A set-semantics prover would happily "prove" those + correct. + + This is the same gap this topic keeps finding from the other + direction: SQLancer's TLP comparator checks size then `HashSet` + equality (`ComparatorHelper.java:91, 108-112`) and turso's checks + size then two-way containment + (`generation/property.rs:1138, 1146-1177`) — both blind to + multiplicity. Cosette is the one tool in this topic that gets bags + right by construction, and that is precisely because it is doing + algebra rather than comparing outputs. + +
+ +- [ ] You can say what `l_undef` means and why a solver harness that ignores it is worse than no harness. + +
Answer + + `check_sat` returns `lbool`, three-valued (`src/solver/solver.h:177`): + `l_true` = sat (counterexample found), `l_false` = unsat (proof), + `l_undef` = the solver stopped without deciding — a timeout, a + resource limit, or a fragment outside any decision procedure (which + in practice means quantifiers, incomplete theory combinations, or + nonlinear arithmetic). + + A harness that writes `if result != Sat { report_proven() }` turns + every timeout into a proof. That is worse than no harness, because + it produces *false confidence* rather than no confidence — and + unlike a fuzzer that finds nothing, it emits a green check. + + The defence is Step 3's design, not a bigger timeout: keep the + encoding quantifier-free and in a named fragment so it hits the + probe cascade at `default_tactic.cpp:38-50` rather than the general + `mk_smt_tactic` fallback at `:52`. And treat `l_undef` as a test + failure that must be investigated, exactly like a flaky test. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the two topic-10 rewrite rules you would verify. +
Answer + + No unfoldable answer — this one is the writing. For question 5, the + useful observation is that *both* rules are single-row and therefore + need only Step 3's encoding, but they need different amounts of + Step 4. + + Commuting `σ_p σ_q` is `(p AND q)` versus `(q AND p)` on one row — + Kleene AND is commutative, so the honest encoding proves it and the + two-valued one would too. Filter-past-projection is the one that + bites: the projection may drop a column the filter reads, or change + a column's nullability, so the `(value, is_null)` pair is + load-bearing and the rule has a *side condition* the encoding has + to state. Encoding a side condition is the skill the exercise is + actually teaching. + +
+ ## References **Papers** - de Moura & Bjørner — "Z3: An Efficient SMT Solver" (TACAS 2008) - — 4 pages, read whole + — 4 pages, read whole; then read topic 21's chapter, which walks + it against the source - Chu, Wang, Weitz, Cheung, Suciu — "Cosette: An Automated Prover - for SQL" (CIDR 2017) — read for the K-relations encoding and the - SMT/Coq split; our use is the SMT half - -**Code** -- [z3](https://github.com/Z3Prover/z3) — `src/` — start from - `src/solver/solver.h` and `src/smt/smt_context.h`, then the - tactic machinery in `src/tactic/` (tactics ARE query plans for - proofs) + for SQL" (CIDR 2017) — §2 for undecidability and the + prove/disprove split, §3.1 for the bounded symbolic data model + (`Tuple := List`, fixed-size symbolic relations, no + floats), §6 for the 17-of-23 Rules result; our use is the SMT half + +**Cross-references** +- [topics/21-formal/reading-z3-tacas08.md](../21-formal/reading-z3-tacas08.md) + — CDCL(T), Nelson–Oppen, `src/ast/euf/` and E-matching in depth. + Everything this chapter compresses into Step 1 +- [reading-pqs-tlp-papers.md](reading-pqs-tlp-papers.md) — TLP's + three-way partition is Step 4's Kleene table, arrived at from the + testing side +- [reading-sqlancer.md](reading-sqlancer.md) — `ComparatorHelper`'s + size-plus-set comparison, the bag-semantics gap Cosette closes + +**Code** — [z3](https://github.com/Z3Prover/z3) @ `1d425e5` + +| File | Lines | What | +|---|---|---| +| `src/solver/solver.h` | 58 | `class solver : public check_sat_result` | +| `src/solver/solver.h` | 124, 177-183 | `assert_expr` and `check_sat` — the entire testing API | +| `src/solver/solver.h` | 172-174 | the unsat-core contract | +| `src/tactic/tactic.h` | 34 | `class tactic` — composable goal transformers | +| `src/tactic/portfolio/default_tactic.cpp` | 36-55 | twelve probe-guarded branches, then `and_then(preamble, smt)` | +| `src/tactic/portfolio/smt_strategic_solver.cpp` | — | tactic → solver bridge | +| `src/smt/smt_context.h` | 89 | `class context` — CDCL(T); topic 21's territory | +| `src/smt/mam.cpp` | — | matching abstract machine; not needed for a quantifier-free encoding | diff --git a/topics/17-simd/README.md b/topics/17-simd/README.md index 95ea7be..b7fee34 100644 --- a/topics/17-simd/README.md +++ b/topics/17-simd/README.md @@ -48,8 +48,9 @@ The DB kernel (SIGMOD '15's centerpiece). Three shapes: compress: mask = v .< t AVX-512: vpcompressd (polars filter/avx512.rs:59 — hardware) NEON: no compress! 4-bit mask → LUT of shuffle masks → - vqtbl1q (simdjson arm64/simd.h:267-276 does exactly - this for 8-byte compaction) + vqtbl1q (simdjson arm64/simd.h:283-299, + `compress_halves`, does exactly this for 8-byte + compaction; :267-276 is the 16-byte `compress`) ``` Selectivity decides the winner: branchy wins at ~0%/100% (predicted @@ -61,9 +62,11 @@ that's the experiments' centerpiece curve. x86 `movemask` (bitmask from lanes) has no NEON equivalent — the idiom is `vshrn` (shift-right-narrow) folding 16 lanes into a 64-bit "4 bits per lane" mask (hashbrown group/neon.rs, memchr's -Vector::movemask). SwissTable = SIMD probing: 16 control bytes per -group, one `vceqq`+narrow gives candidate slots in 2 instructions — -topic 2's hash table, now explained at lane level. +Vector::movemask). SwissTable = SIMD probing: **8** control bytes per +group on this host — `src/control/group/mod.rs:24-33` picks the NEON +backend on aarch64 and `neon.rs:16` is `Group(uint8x8_t)`; the familiar +16 is the SSE2 group. One `vceq`+narrow gives candidate slots in 2 +instructions — topic 2's hash table, now explained at lane level. ## 5. The masterclass codebases (per reading guide) @@ -78,7 +81,7 @@ topic 2's hash table, now explained at lane level. comments; multiple ISA files per kernel (haswell/skylake/neon/ sve...) dispatched at runtime. - **memchr**: `Vector` trait over ISAs; the 4×-unrolled search loop. -- **Mojo**: `SIMD[type, width]` as a first-class parametric type — +- **Mojo**: `SIMD[dtype, size]` as a first-class parametric type — what `std::simd` wants to be with a compiler behind it. ## 6. FastLanes (bit-packing at SIMD speed) @@ -118,7 +121,7 @@ stable stand-in): | [reading-simsimd.md](reading-simsimd.md) | SimSIMD: the port/latency table is the design doc | | [reading-sigmod15-vectorization.md](reading-sigmod15-vectorization.md) | SIMD for databases: two primitives, four operators | | [reading-fastlanes.md](reading-fastlanes.md) | FastLanes: bit-unpacking at memory bandwidth | -| [reading-mojo-simd.md](reading-mojo-simd.md) | Mojo's `SIMD[type, width]`: width as a type parameter | +| [reading-mojo-simd.md](reading-mojo-simd.md) | Mojo's `SIMD[dtype, size]`: width as a type parameter | ## Capstone M17 diff --git a/topics/17-simd/experiments/.gitignore b/topics/17-simd/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/17-simd/experiments/.gitignore +++ b/topics/17-simd/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/17-simd/notes.md b/topics/17-simd/notes.md index 8d72fca..6863012 100644 --- a/topics/17-simd/notes.md +++ b/topics/17-simd/notes.md @@ -2,6 +2,13 @@ ## Baseline (provided rungs, release, Apple Silicon, measured 2026-07-10) +> A different run of the same lane than +> [FINDINGS.md](../../FINDINGS.md) row 17 (dot **8.88 → 26.32 GB/s**, +> branchy at 50% **0.95 GB/s** against the 1.19 below). Where the two +> disagree, FINDINGS is canonical; cite one run or the other by name and +> never average them. Re-run `./verify.sh 17` before treating any cell +> below as current. + N = 4M f32 (16 MB per input — out of L2, into memory), 20 reps. ### dot product diff --git a/topics/17-simd/reading-fastlanes.md b/topics/17-simd/reading-fastlanes.md index 1f1fd2e..5a81ff7 100644 --- a/topics/17-simd/reading-fastlanes.md +++ b/topics/17-simd/reading-fastlanes.md @@ -1,194 +1,560 @@ # FastLanes: bit-unpacking at memory bandwidth -Topic 12 decoded bit-packed integers one value at a time; FastLanes -(Afroozeh & Boncz) redesigns the STORAGE LAYOUT so that decoding any -bit width is the same straight-line SIMD kernel — no shuffles, no -per-width special cases — and hits memory bandwidth on every ISA -from NEON to AVX-512, *including scalar code that autovectorizes*. -Before the paper, this chapter builds the argument step by step: -what bit-packing is, why the obvious layout can't vectorize, and how -transposing the data dissolves every obstacle. The punchline for -this whole topic: layout, not intrinsics, is the win. +Topic 12 decoded bit-packed integers one value at a time. FastLanes +(Afroozeh & Boncz, VLDB 2023) redesigns the **storage layout** so that +decoding any bit width is the same straight-line sequence of loads, +shifts, masks and ORs — no shuffles, no cross-lane traffic, no +per-width special cases — and so that scalar C compiled with +`-O3` auto-vectorises to match hand-written intrinsics. Before the +paper, this chapter builds the argument: what bit-packing is, why the +obvious layout cannot vectorise, and how interleaving the values +dissolves every obstacle. The punchline for the whole topic: layout, +not intrinsics, is the win. + +There is no pinned clone for FastLanes, so every claim below is +anchored to the paper — "The FastLanes Compression Layout: Decoding +> 100 Billion Integers per Second with Scalar Code", PVLDB 16(9), +pp. 2132-2144 — by section, listing or figure number, and the local +measurements come from this topic's own `notes.md`. Where the paper +and this machine disagree, both numbers are given with their source. ## The problem in one sentence Bit-packed columns are how analytical databases fit in RAM, but the -standard sequential layout decodes serially — one value's position -depends on all previous values — leaving a memory-bandwidth-class -job (100+ billion integers/second in the title) running at scalar -ALU speed. +standard sequential ("horizontal") layout decodes serially — value +*i*'s bit position depends on all the values before it — leaving a +memory-bandwidth-class job running at scalar ALU speed. ## The concepts, step by step ### Step 1 — bit-packing: pay only the bits you need -Bit-packing stores integers in exactly the bits their range requires: -values 0–7 need 3 bits each, so 1024 of them take 384 bytes instead -of 4096 as u32s — a 10.7× compression. For an analytical scan the -compression IS the performance (topic 12): a scan reads 10× fewer -bytes through the topic-0 memory ladder. The catch is decoding — -turning packed bits back into usable u32s — which now sits on the -hot path of every scan. +> **In:** 1024 integers known to fit in W bits, currently stored as +> `uint32`. +> **Out:** the same 1024 integers in `1024·W` bits, and a decode step +> that now sits on the hot path of every scan. -### Step 2 — why the sequential layout can't vectorize +Bit-packing stores each integer in exactly the bits its range +requires. Work the case the paper uses throughout (§2.1, Figure 1): +W = 3, so values in 0..=7. -The obvious layout packs values back-to-back: value 1's bits -immediately follow value 0's. Two things break. Values straddle word -boundaries (a 3-bit value can start at bit 62 of a u64 and end in -the next word), and — worse — value i's bit position is -`i × w mod 64`, so each decode step's shift amount depends on where -the previous one ended: +``` + 1024 values as uint32 : 1024 * 32 bits = 32768 bits = 4096 bytes + 1024 values at W = 3 : 1024 * 3 bits = 3072 bits = 384 bytes + compression ratio : 4096 / 384 = 10.67x +``` + +For an analytical scan the compression *is* the performance (topic +12): the scan pulls 10.67× fewer bytes through the topic-0 memory +ladder. The catch is that decoding — turning packed bits back into +usable `uint32`s — is now a per-value cost on the read path, and the +whole question is whether it can be made to cost nothing. + +Note the block size, because everything downstream depends on it. The +paper's footnote 2 (in §2.1) puts it exactly: "a chunk of 1024·W +(bit-width) encoded values fit in exactly W FLMM1024 registers", and +warns that larger chunks compress worse (the bit width is set by the +widest value in the chunk) and coarsen scan granularity. **1024 values +per block, at every bit width.** + +### Step 2 — why the sequential layout cannot vectorise + +> **In:** the "horizontal" layout — value 1's bits immediately follow +> value 0's, and so on. +> **Out:** a decode loop with a serial dependency and a data-dependent +> shift amount, i.e. both of README §2's autovectorisation failures at +> once. + +Two things break. Values straddle word boundaries (a 3-bit value can +start at bit 62 of a `uint64` and finish in the next word), and — the +fatal one — value *i* sits at bit position `i·W mod 64`, so each +decode step's shift amount is a function of where the previous one +ended: ``` - 3-bit values packed sequentially in a u64: - |v0 |v1 |v2 |v3 |v4 ... v20|v21⟨spans the word boundary⟩ - decode v21: load TWO words, shift both, OR, mask ← branchy, serial, - and lane i+1 depends on where lane i ended ← unvectorizable + 3-bit values packed horizontally in a uint64: + |v0 |v1 |v2 |v3 |v4 ... v20|v21 <- spans the word boundary + decode v21: load TWO words, shift both, OR, mask <- branchy, serial + and lane i+1's shift depends on lane i's end <- unvectorisable ``` -That's a serial dependency chain (each step needing the previous -step's result — the enemy from README §1) *plus* data-dependent -control flow (README §2's failure #2). SIMD lanes want to execute -the identical operation; sequential packing guarantees they can't. +The paper's §1.1 lists this under "Value-interleaving": the naive +layouts lead to "lack of parallel work and unused lanes or expensive +compensating actions such as PERMUTE and BITSHUFFLE". SIMD lanes want +to execute the identical operation on independent data; horizontal +packing guarantees neither. + +### Step 3 — the fix: round-robin the values over 1024/T lanes + +> **In:** 1024 W-bit values and a chosen lane width T ∈ {8,16,32,64}. +> **Out:** the same values distributed over S = 1024/T lanes so that +> every lane applies the *same* shift and mask at every step. + +FastLanes targets a **virtual** register. §1.1: "we preempt further +widening of SIMD registers and propose a layout optimized for a +virtual 1024-bits register FLMM1024 that gets the best performance out +of any existing ISA, and even from scalar code." + +The layout inside that register is the load-bearing idea, and it is +**not** bit-planes. §1.1 again: FastLanes "distributes all logically +subsequent e.g., 3-bit values round-robin over 128 separate 8-bit +lanes." So the unit that is spread is the **whole value**, not one of +its bits. §2.1 fixes the parameters: "To maximize decoding performance +we use the smallest lane-width that fits that, i.e. 8-bits (T = 8), +and therefore we have 128 (S = 1024/T = 128) lanes in our FLMM1024 +word." -### Step 3 — the fix: transpose into 1024 virtual bit-serial lanes +Derive the placement rule and then check it against the paper's own +figure: -FastLanes packs a block of 1024 values as if the machine had 1024 -one-bit-wide lanes ("the 1024-bit virtual ISA"): instead of value -after value, it stores bit-plane after bit-plane — plane b holds bit -b of many values, arranged so that every real SIMD lane always -applies the SAME shift and mask: +``` + T = 8 -> S = 1024 / T = 128 lanes per FLMM1024 word + -> each lane holds 1024 / S = 8 of the 1024 values + -> lane s holds values s, s+128, s+256, ..., s+896 + -> those 8 values need 8 * W = 24 bits, but a lane in ONE + word is only T = 8 bits, so a lane's 24 bits are spread + over B = 1024*W/1024 = W = 3 consecutive FLMM1024 words + + lane 0, concatenated across the 3 words (24 bits), W = 3: + bit offset 0 3 6 9 12 15 18 21 + value | 0|128|256|384|512|640|768|896| + word boundaries fall at bit 8 and bit 16 + -> the value at offset 6..8 (position 256) is SPLIT: 2 bits in + word 0, 1 bit in word 1 + -> the value at offset 15..17 (position 640) is SPLIT: 1 bit in + word 1, 2 bits in word 2 +``` +That is exactly Figure 1's caption: "In the first word, only the first +two bits (yellow,pink) of the value at position 256 fit, so it is +continued in the second word (blue bit). The value at position 640 is +also split. This happens in all lanes." If your arithmetic reproduces +256 and 640 you have the layout right; if it does not, re-read the +round-robin rule before going further. + +The crucial consequence: the split happens **in all lanes, at the same +bit offset**. Every lane is doing the identical work, so the fix-up is +one extra shift and one extra OR applied to the whole register — never +a permute, never a lane-crossing move. + +### Step 4 — the kernel: a pseudo-ISA of six operations + +> **In:** the interleaved layout of Step 3. +> **Out:** a straight-line kernel of loads, masked shifts and ORs, +> generated once per (W, T) pair, with no branches and no shuffles. + +Listing 1 (§2.2) defines the whole instruction set on FLMM1024: +`LOAD`, `STORE`, `AND_LSHIFT`, `AND_RSHIFT`, `AND`, +`OR`, `XOR`, `ADD`, `SET`. §2.2 explains the choice: +"FastLanes only uses simple operators, such as load/store, +left/right-shift, and/or/xor, addition and set instructions; supported +for all lane-widths, T ∈ {8, 16, 32, 64}… This instruction set can be +trivially mapped to intrinsics in all previously mentioned thinner +ISAs, just by using multiple identical instructions on independent +registers." + +Listing 2 is the W = 3, T = 8 unpack kernel in that pseudo-ISA. Read +it against the bit offsets you just derived: + +``` + Listing 2 (paper p. 2134), lines 1-15, abridged: + 1 uint<8> MASK1 = (1<<1)-1, MASK2 = (1<<2)-1, MASK3 = (1<<3)-1; + 3 r0 = LOAD<8>(in+0); + 4 r1 = AND_RSHIFT<8>(r0,0,MASK3); STORE<8>(out+0,r1); <- offset 0 + 5 r1 = AND_RSHIFT<8>(r0,3,MASK3); STORE<8>(out+1,r1); <- offset 3 + 6 r1 = AND_RSHIFT<8>(r0,6,MASK2); <- offset 6, 2 bits + 7 r0 = LOAD<8>(in+1); STORE(out+2,OR<8>(r1, + 8 AND_LSHIFT<8>(r0,2,MASK1))); <- + 1 bit, <<2 + 9 r1 = AND_RSHIFT<8>(r0,1,MASK3); STORE<8>(out+3,r1); <- offset 9 + 10 r1 = AND_RSHIFT<8>(r0,4,MASK3); STORE<8>(out+4,r1); <- offset 12 + 11 r1 = AND_RSHIFT<8>(r0,7,MASK1); <- offset 15, 1 bit + 12 r0 = LOAD<8>(in+2); STORE(out+5,OR<8>(r1, + 13 AND_LSHIFT<8>(r0,1,MASK2))); <- + 2 bits, <<1 + 14 r1 = AND_RSHIFT<8>(r0,2,MASK3); STORE<8>(out+6,r1); <- offset 18 + 15 r1 = AND_RSHIFT<8>(r0,5,MASK3); STORE<8>(out+7,r1); <- offset 21 ``` - 1024 values, width w → w × 128-byte "bit-planes": - word j of plane b holds bit b of values {j, j+64, j+128, ...} - (transposed order, via the "unified 04261537" permutation) - decode = for each output vector: - acc = (plane_word >> shift) & mask ← same shift for ALL lanes - no value ever crosses a lane boundary - no cross-lane shuffle, EVER +Every shift constant is an immediate. Lines 6-8 and 11-13 are the two +splits you predicted, stitched with one `AND_LSHIFT` and one `OR` +rather than by padding — and note that the two halves come from +*different loads of the same lane position*, so nothing crosses a lane. +The paper generates 116 such kernels statically, one for each +(W, T) with W < T and T ∈ {8,16,32,64} (§2.2, above Listing 2). + +Now count what the kernel costs, because this is the number the title +is made of: + +``` + Listing 2 per 1024 values: 3 LOAD + 8 STORE + 8 AND_RSHIFT + + 2 AND_LSHIFT + 2 OR = 23 FLMM1024 ops + + mapping FLMM1024 (1024 bits) onto real registers: + AVX-512 (512-bit): 1024/512 = 2 real instrs per FLMM1024 op + 23 * 2 = 46 instrs -> 1024/46 = 22.3 values/instr + NEON (128-bit): 1024/128 = 8 real instrs per FLMM1024 op + 23 * 8 = 184 instrs -> 1024/184 = 5.6 values/instr + uint64 scalar : 1024/64 = 16 real instrs per FLMM1024 op + 23 * 16 = 368 instrs -> 1024/368 = 2.8 values/instr ``` -Because every lane does the identical shift+mask at every step, the -kernel is the same for NEON's 128-bit vectors, AVX-512's 512-bit, or -a u64 scalar loop — the vector width just decides how many of the -1024 virtual lanes you process per instruction. Wider ISA = same -code, fewer iterations: +Compare with §1.1's headline: decoding "delivers a vector of 1024 +tuples at-a-time, in sometimes as little as 17 CPU cycles (an +astonishing 70 values per CPU core cycle)". 1024/17 = 60.2, so the "70 +values per cycle" is the best point of Figure 8 rather than the 17-cycle +case; both are W = 8 on the widest machine in Table 2. Your Mac is the +middle row of the arithmetic above. + +Here is the same skeleton in Rust, so you can see the shape the +autovectoriser is being handed: ```rust -// 1024 values as 16 u64 lanes advancing in LOCKSTEP — every lane runs the -// identical shift+mask, which is all autovectorization needs to see -fn unpack(planes: &[[u64; 16]], w: u32, out: &mut [[u64; 16]; 64]) { +// ILLUSTRATION — not quoted from the paper; the real kernel is Listing 2, +// p. 2134, whose control flow this reproduces. Compare with the topic's own +// scalar decoder at topics/17-simd/experiments/src/unpack.rs:7 — that one is +// horizontal, and this one is interleaved. +fn unpack_interleaved(words: &[[u64; 16]], w: u32, out: &mut [[u64; 16]]) { let mask = (1u64 << w) - 1; let (mut word, mut shift) = (0usize, 0u32); for group in out.iter_mut() { - for lane in 0..16 { // ← the vectorized dimension - group[lane] = (planes[word][lane] >> shift) & mask; + for lane in 0..16 { // <- the vectorised dimension: 16 u64 + group[lane] = (words[word][lane] >> shift) & mask; } shift += w; if shift + w > 64 { word += 1; shift = 0; } - // (real FastLanes stitches the boundary bits with one extra - // OR instead of padding — still the same shift for ALL lanes) + // the real kernel stitches the straddling value with one extra + // AND_LSHIFT + OR instead of restarting the shift; still no branch + // inside the lane loop, and still the same shift for ALL lanes } } ``` -The cost of the transpose: values are no longer stored in logical -order, so random access to value i needs reads from w separate -planes. Analytic scans, which decode whole blocks anyway, never -notice. +The inner loop has a compile-time trip count, no lane-dependent index, +no branch, and one shift amount for all 16 lanes. That is the entire +list of things an autovectoriser needs. §3.1's answer to Q4: "clang++ +can auto-vectorize our Scalar code, matching the performance of +explicit intrinsics — denoted SIMD", and the recommendation that +follows it: "when incorporating FastLanes in future systems, we +recommend just using the Scalar code paths." + +### Step 5 — the Unified Transposed Layout, and what it buys DELTA -### Step 4 — the unified transposed order: one permutation for every lane width +> **In:** a DELTA-encoded column, whose decode is a 1024-long serial +> prefix sum, and a table whose columns have different widths. +> **Out:** one tuple order that works for every lane width, and a +> per-lane chain of length T instead of 1023. -The permutation 04261537 reorders values so that ALL of -{8,16,32,64}-bit lane types see a consistent order — so you can -bit-unpack u8s, then delta-decode as u16s, without re-permuting -between kernels. And delta (each value stored as the difference from -its predecessor — a PREFIX dependency, inherently serial) is -computed *per-lane over the transposed order*: each lane keeps its -own running base, turning one 1024-long serial prefix-sum into -1024/W independent short chains. Same trick as multi-accumulator dot -(reading-simsimd.md): break the chain by restructuring the data. -Question: why does delta need the unified order at all? +§2.3 states the dependency problem and its fix. In the default layout +"adding the values at position 0 and position 1 correspond to different +lanes"; the transposed layout stores values out of order — "The order +for the first 16 values here is 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, +3, 7, 11, 15" — and §2.3 reports the payoff for 16 values in a 128-bit +register: "only 4 additions are needed." -### Step 5 — results worth remembering +Do the general version: -- Decode at RAM bandwidth: unpacking is FREE relative to the memory - it saves — the final word on topic 12's "compression IS - performance" table. -- Scalar Rust/C compiled with autovec reaches ~the intrinsic - version, BECAUSE the layout removed everything autovec chokes on - (README §2's four failures — all four absent by construction). -- The same layout accelerates delta, RLE, dictionary, and FOR — - it's a compression *layout*, not a codec. +``` + sequential DELTA over 1024 values: + chain = 1023 dependent adds, each waiting on the previous + transposed, S = 1024/T lanes each carrying its own base: + values per lane = 1024 / S = 1024 / (1024/T) = T + chain = T dependent adds, S of them running in parallel + + T = 64 -> chain 64, 1023/64 = 16.0x shorter + T = 32 -> chain 32, 1023/32 = 32.0x shorter + T = 8 -> chain 8, 1023/8 = 127.9x shorter +``` -### Step 6 — our baby version: `unpack.rs` +That is the same move as the eight-accumulator dot product this topic +measures (`reading-simsimd.md`, and `notes.md`'s 10.89 → 42.12 GB/s): +break one long dependency chain into many short independent ones by +restructuring the data rather than the instructions. -The experiments' 4-bit unpack keeps topic 12's sequential layout — -legitimately, because at w=4 values never straddle bytes (4 divides -8), the one family of widths where sequential is already -SIMD-friendly: +§2.4 then solves the problem that makes it usable in a real scan. +Different columns have different widths, "and different columns will +have different widths. However, when we reorder tuples, we should use +the same order for all columns, because a scan needs to create a +consistent stream of tuples." The construction: "The basic building +block are transposed tiles of 8x16 values. We have eight such tiles for +each vector of 1024 tuples", ordered **04261537**, with DELTA +processing order per width: ``` - 16 bytes = 32 nibbles: lo = bytes & 0x0F, hi = bytes >> 4 - → interleave/widen to u32 lanes. No LUT. Two ops + widening. + §2.4, verbatim processing orders: + 8-bit : bases -> 04261537 + 16-bit : bases -> 0426 -> 1537 + 32-bit : bases -> 04 -> 15 -> 26 -> 37 + 64-bit : bases -> 0 -> 1 -> .. -> 7 ``` -Question: at which widths does the sequential layout stop being -this easy (hint: w ∤ 8), and what does FastLanes' transposition buy -exactly there? +§2.4 proves 04261537 is the *only* order with the two required +properties (start at tile 0; successive SIMD operations touch directly +subsequent tile numbers in the same lane position). Read that proof — +it is nine lines and it is the answer to "why this permutation". One +caution when you do: the running text on p. 2137 writes "04261357" +once, while the abstract, §1.1 and the proof's conclusion all give +**04261537**. The proof is the authority; the single "1357" is a typo. + +### Step 6 — results, with their measurement conditions attached + +> **In:** the paper's Table 2 platforms and §3.1/§3.2 methodology. +> **Out:** which numbers describe a CPU kernel, which describe a query, +> and which of them can be expected on this Mac. + +Table 2 lists six machines. Two matter here: **Intel Ice Lake 8375C at +3.5 GHz with AVX-512**, which produces most of the headline figures, +and **Apple M1 at 3.2 GHz with 128-bit NEON** — the closest thing in +the paper to the machine you are reading on. §3.1 also notes that on +Graviton3 "SVE is slower than NEON", so every ARM number in the paper +is a NEON number. + +The methodology matters more than usual. §3.1: "These micro-benchmarks +aim to characterize pure CPU cost and decompress a single vector 30M +times; hence **all data is L1 resident**." The scalar baselines were +de-vectorised on purpose with `-O3 -mno-sse -fno-slp-vectorize +-fno-vectorize`. So the micro-benchmark speedups are *ALU* ratios, not +bandwidth ratios: + +| claim | value | where | +|---|---|---| +| SIMD vs de-vectorised Scalar | 40×–70× at T = 8, 3×–4× at T = 64 | §3.1 (Q1), Fig. 8 | +| `Scalar_T64` vs Scalar | 64/T ×, i.e. 8× at T = 8 | §3.1 (Q3), Fig. 8/9 | +| autovectorised Scalar vs intrinsics | matches | §3.1 (Q4) | +| peak decode rate | 70 values per core cycle at W = 8 | §3.1 (Q1) | +| interleaving's cost to plain scalar | none — "performance is equal to the naive horizontal layout" | §3.1, Fig. 9 | +| M1 specifically | "just 128-bit NEON, but clearly has more instruction level paralellism"; "In terms of scalar performance, M1 tops Ice Lake clock-for-clock" | §3.1 | + +The honest end-to-end claim is **not** "decoding is free". It is §3.2's +crossover, measured with `SELECT SUM(COL) FROM TAB` over `10 · 2^28` +uint32 values (10 GB, RAM-resident) on Ice Lake, from Figure 12's +caption: "The crossover point where decompressing scans (plots) +outperform plain array scans (horizontal lines), moves from a minimal +compression ratio of 4x (≈8bits) with Scalar decoding to just 25% +compression (≈24bits) with FastLanes… FastLanes can then improve +end-to-end performance up to 7x vs. uncompressed and 4x vs. scalar." + +Read that as the real result: decoding is not free, it is cheap enough +that *almost any* compression now pays for itself in a scan. Figure 11 +adds the last few percent by fusing bit-unpacking with the FOR / DELTA +/ DICT / RLE decode, which removes an intermediate STORE + LOAD. + +### Step 7 — our baby version: `unpack.rs` + +> **In:** this topic's 4-bit unpacking bench, which keeps the +> horizontal layout. +> **Out:** an understanding of exactly which width family lets you get +> away with that, and what FastLanes buys at every other width. + +```rust +// topics/17-simd/experiments/src/unpack.rs:7-14 — the provided scalar rung + 7 pub fn unpack4_scalar(packed: &[u8], out: &mut Vec) { + 8 out.clear(); + 9 out.reserve(packed.len() * 2); + 10 for &b in packed { + 11 out.push((b & 0x0F) as u32); + 12 out.push((b >> 4) as u32); + 13 } + 14 } +``` + +That is the horizontal layout of Step 2 — and it is fine, because W = 4 +divides 8. No value ever straddles a byte, so the shift amounts are the +constant pair (0, 4) rather than a running position. The whole family +`W ∈ {1, 2, 4, 8}` has this property; `W = 3, 5, 6, 7` does not, and +that is precisely the gap FastLanes' interleaving closes. + +`notes.md` records this rung at **10.20 GB/s of output** (provided +rungs, release, Apple Silicon, measured 2026-07-10). Before you write +the NEON version, predict it, then reconcile with the paper: if +clang has already autovectorised the loop above, the intrinsics rung +should win little or nothing — which is FastLanes' own Q4 result +arriving on your desk. ## How to read the paper (with the concepts in hand) -- **§3–4 — read carefully.** The interleaved layout (Step 3) and the - unified transposed order (Step 4). Draw the bit-planes for w=3, - 16 lanes, by hand — once you can place value 17's bits yourself, - the rest of the paper is bookkeeping. -- **Delta section** — check that the per-lane-base trick really is - Step 4's chain-breaking, then compute the chain-length ratio - (question 3 below). -- **Evaluation** — the claim to verify is the scalar-autovec one - (Step 5, second bullet): find the table where scalar compiled code - matches intrinsics, and note on which ISA the gap is largest. -- The FastLanes repo (CWI's reference implementation) is optional; - the paper's kernels are self-contained. +- **§2.1 + Figure 1 — read first, with a pencil.** Reproduce Step 3's + offset table for W = 3, T = 8 and confirm you get positions 256 and + 640 as the split values. Do not move on until you do; §2.2 onwards is + bookkeeping on top of this. +- **§2.2, Listing 1 and Listing 2.** Nine pseudo-instructions and one + 15-line kernel. Match every shift constant in Listing 2 to a bit + offset from your table. +- **§2.3 then §2.4.** §2.3 for why transposition breaks the DELTA + chain (and the "only 4 additions" figure), §2.4 for why one order has + to serve every column width, plus the nine-line uniqueness proof of + 04261537. +- **§3.1 — read the methodology paragraph before the results.** "All + data is L1 resident" and the `-fno-vectorize` flags on the baseline + decide what the 40×–70× means. +- **§3.2 and Figure 12** are the numbers to quote when someone asks + whether compression pays: the crossover moves from 4× compression to + 25 %. +- The [FastLanes repo](https://github.com/cwida/FastLanes) is optional; + it is not in this repo's pin table, and the paper's kernels are + self-contained. ## Questions for notes.md -1. Block = 1024 values regardless of width. What two constraints - pick 1024 (largest vector ISA lanes × smallest type, and - cacheline alignment of every plane)? -2. Interleaved decode touches w planes 128B apart — is that still - sequential enough for the prefetcher (topic 13's stride limits)? -3. Delta-decode with per-lane bases: what's the ratio of chain - length, 1024 sequential vs transposed on 128-bit NEON (16 u64 - lanes... derive it)? -4. Random access to value i now needs w bit-plane reads — what did - we trade away vs sequential packing, and why doesn't an analytic - scan care (topic 12's block-granularity access)? -5. For M17's checklist item "SIMD-ize one topic 12 decoder": ours is - w=4 sequential. Predict GB/s scalar vs NEON before running - simd_bench — then reconcile with FastLanes' claim that layout, - not intrinsics, is the win. +1. Redo Step 3's offset table for W = 5, T = 8. How many FLMM1024 + words does one lane's values span, how many of the 8 values in a + lane are split across a word boundary, and how many extra + `AND_LSHIFT` + `OR` pairs does the kernel therefore need compared + with the W = 3 case's two? +2. Step 1 fixes the block at 1024 values. Footnote 2 in §2.1 gives two + reasons against making it larger. State both, and say which one + would bite a Cypher property scan hardest. +3. Interleaved decode reads W words that are 128 bytes apart in the + T = 8 case. Is that stride still prefetcher-friendly (topic 13)? + Compute the number of distinct cache lines one 1024-value decode + touches at W = 3 and at W = 32. +4. Step 5 gives the chain length as T. On 128-bit NEON a `uint64` lane + width means 2 lanes per physical register. Work out how many + physical NEON registers one FLMM1024 DELTA step occupies at + T = 64, and whether that fits the 32 architectural `v` registers. +5. Random access to value *i* now needs the lane index `i mod S` and + the within-lane index `i / S`, plus up to two word reads if the + value is split. Write the formula, then say why an analytic scan + never pays it (topic 12's block-granularity access). +6. For M17's "SIMD-ize one topic 12 decoder": ours is W = 4, + horizontal, measured at 10.20 GB/s of output. Predict the NEON rung + *before* running the bench, then reconcile your result with §3.1's + Q4 claim that autovectorised scalar already matches intrinsics. ## Done when -- [ ] You can explain why the sequential bit-packed layout cannot vectorize, at the level of which value crosses which lane boundary. -- [ ] You can describe the transpose into 1024 virtual bit-serial lanes and say what makes one permutation work for every lane width. -- [ ] You can name the two constraints that fix the block at 1024 values regardless of bit width. -- [ ] You can say what random access to value i now costs, and what was traded to get the scan speed. -- [ ] You wrote answers to all five questions in notes.md, and can state how your `unpack.rs` differs from the paper's layout — this topic measures the scalar version at 7.99 GB/s of output. +Answer each before unfolding it. + +- [ ] You can explain why the horizontal bit-packed layout cannot vectorise, naming both failures. + +
Answer + + Value *i* starts at bit `i·W mod 64`, so the shift amount for lane + *i+1* depends on where lane *i* ended — a serial dependency — and + values that straddle a word boundary need a second load, a second + shift and an OR, which is a data-dependent branch. SIMD lanes must + execute the identical operation on independent data; horizontal + packing supplies neither. §1.1 lists the usual escape routes — + PERMUTE and BITSHUFFLE — as the expensive compensating actions the + layout is designed to avoid. + +
+ +- [ ] You can state what FastLanes interleaves, and place a specific value in a specific lane and word. + +
Answer + + It interleaves **whole values**, not bits: §1.1 says it "distributes + all logically subsequent e.g., 3-bit values round-robin over 128 + separate 8-bit lanes." With T = 8, S = 1024/T = 128 lanes, each lane + holds 1024/S = 8 values — lane *s* holds positions + s, s+128, …, s+896 — and those 8·W = 24 bits span W = 3 consecutive + FLMM1024 words. In lane 0 the values sit at bit offsets 0, 3, 6, 9, + 12, 15, 18, 21, so positions 256 (offset 6) and 640 (offset 15) + straddle the word boundaries at bits 8 and 16 — which is exactly what + Figure 1's caption says. + + Anyone who tells you plane *b* holds bit *b* of many values is + describing a bit-plane / BITSHUFFLE layout, which is not this. + +
+ +- [ ] You can name the block size and say what fixes it. + +
Answer + + 1024 values, at every bit width. §2.1 footnote 2: "a chunk of 1024·W + (bit-width) encoded values fit in exactly W FLMM1024 registers." + Larger chunks are rejected for two reasons given there — worse + compression, because the bit width is set by the value domain of the + whole chunk, and a coarser minimum vector size "imposed to the scan + subsystem". + +
+ +- [ ] You can explain what the Unified Transposed Layout is for, and why the order is 04261537 rather than anything else. + +
Answer + + A scan reads several columns of different widths and must emit one + consistent tuple stream, so all columns need the *same* reordering + (§2.4). The building block is eight transposed 8×16 tiles per + 1024-tuple vector. The order must start at tile 0 (the 64-bit case + processes one tile at a time and the header holds bases for tile 0) + and must make successive SIMD operations touch directly subsequent + tile numbers in the same lane position; §2.4's proof shows those two + requirements admit **04261537** alone. Processing orders: 8-bit + `04261537`; 16-bit `0426` then `1537`; 32-bit `04, 15, 26, 37`; + 64-bit `0…7`. + +
+ +- [ ] You can quote the paper's speedups *with* the conditions that produced them, and say which one to cite for "does compression pay?". + +
Answer + + §3.1's 40×–70× (T = 8) down to 3×–4× (T = 64) is a **pure-CPU, + L1-resident** micro-benchmark that decompresses one vector 30M times, + against a baseline compiled with `-mno-sse -fno-slp-vectorize + -fno-vectorize`. `Scalar_T64` is 64/T× faster than Scalar; clang++ + auto-vectorises the scalar path to intrinsic speed (Q4); peak is 70 + values per cycle at W = 8. Most figures are Ice Lake 8375C at + 3.5 GHz with AVX-512; the M1 row of Table 2 is 128-bit NEON at + 3.2 GHz. + + For "does compression pay?" cite **§3.2 / Figure 12** instead: on a + 10 GB RAM-resident `SELECT SUM(COL)`, the crossover where a + decompressing scan beats a plain array scan moves from 4× + compression (≈8 bits) with scalar decoding to just 25 % compression + (≈24 bits) with FastLanes, and up to 7× vs uncompressed / 4× vs + scalar at 8 threads. + +
+ +- [ ] You can say what random access to value *i* now costs, and why the scan does not care. + +
Answer + + Value *i* is in lane `i mod S` at within-lane index `i / S`, at bit + offset `(i / S) · W` inside that lane's `T·W`-bit run — so up to two + word reads plus a shift/mask/OR, versus one or two reads in the + horizontal layout, and the address arithmetic is no longer monotone + in *i*. An analytic scan decodes whole 1024-value blocks and consumes + them in whatever order they arrive; §2.3 argues the reordering is + free in the relational setting because "query operator semantics + typically do not depend on order", and where it does the order can be + restored or carried in a selection vector. + +
+ +- [ ] You wrote answers to all six questions in notes.md, and can state how `unpack.rs` differs from the paper's layout. + +
Answer + + `unpack4_scalar` (`experiments/src/unpack.rs:7-14`) is horizontal, not + interleaved — and gets away with it because W = 4 divides 8, so no + value straddles a byte and the shifts are the constants 0 and 4. The + whole family W ∈ {1,2,4,8} is like this; W ∈ {3,5,6,7} is where the + horizontal layout's running bit position reappears and where + FastLanes' interleaving earns its keep. `notes.md` records the scalar + rung at **10.20 GB/s of output** (Apple Silicon, 2026-07-10). + +
## References **Papers** -- Afroozeh & Boncz — "The FastLanes Compression Layout: Decoding - > 100 Billion Integers per Second with Scalar Code" (VLDB 2023) - — read §3-4 for the interleaved layout and the unified transposed - order; the eval confirms the autovectorization claim +- Azim Afroozeh, Peter Boncz — "The FastLanes Compression Layout: + Decoding > 100 Billion Integers per Second with Scalar Code", + *PVLDB* 16(9), 2023, pp. 2132-2144. + — §2.1 and + Figure 1 for the interleaved layout, §2.2 with Listings 1-2 for the + kernel, §2.3-§2.4 for the transposed and Unified Transposed layouts + (including the 04261537 uniqueness proof), §3.1 for the L1-resident + micro-benchmarks and Table 2 for the hardware, §3.2 and Figure 12 for + the end-to-end crossover. **Code** - [FastLanes](https://github.com/cwida/FastLanes) — CWI's reference - implementation of the layout (optional; the paper's kernels are - self-contained) + implementation. Not pinned in `resources/codebases.md`, so nothing in + this guide is anchored to it; the paper's listings are self-contained. +- This topic's own `experiments/src/unpack.rs` — the horizontal W = 4 + decoder to contrast against. diff --git a/topics/17-simd/reading-hashbrown-simd.md b/topics/17-simd/reading-hashbrown-simd.md index 5f079a5..7ea4ff0 100644 --- a/topics/17-simd/reading-hashbrown-simd.md +++ b/topics/17-simd/reading-hashbrown-simd.md @@ -1,220 +1,682 @@ # hashbrown & memchr: movemask without movemask Two crates, one question: how do you get an x86 `movemask` (one bit -per lane) on ISAs that don't have it — and when should you not even -try? Before the anchors, this chapter builds the pieces in order: -what movemask is for, the three ways to fake it, the SWAR fallback -that needs no SIMD at all, and the SwissTable probe loop that puts -it all to work. hashbrown answers the title question by shrinking -the group to 8 bytes so the comparison result already *is* the mask; -memchr answers with the `vshrn` nibble-mask idiom. Between them sits -the portability pattern every SIMD kernel layer copies. +per lane) on an ISA that does not have it — and when should you not +even try? This chapter builds the pieces in order: what movemask is +for, the three ways to fake it, the SWAR fallback that needs no SIMD +at all, and the SwissTable probe loop that puts it to work. hashbrown +answers the title question by shrinking its group to 8 bytes so the +comparison result already *is* the mask; memchr answers with the +`vshrn` nibble idiom, and then avoids paying for it on the miss path. +Between them sits the portability pattern every SIMD kernel layer +copies. + +Every anchor below is `rust-lang/hashbrown@d69025b` or +`BurntSushi/memchr@5fdb40c` (`resources/codebases.md`), quoted with the +line numbers the code occupies in those revisions. Both crates pick +their backend with `cfg!`, so the code your Mac compiles is the +aarch64 branch — and for hashbrown that branch is **not** the one the +crate's own design comment describes. ## The problem in one sentence Every SIMD search kernel ends the same way — "compare 16 bytes at -once, then tell me WHICH lanes matched, as an integer I can iterate" -— and ARM NEON simply has no instruction for that second half, so -every fast hash table and substring search on your Mac is built -around a workaround. +once, then tell me *which* lanes matched, as an integer I can iterate" +— and ARM NEON has no single instruction for that second half, so +every fast hash table and substring search on your Mac is built around +a workaround. ## The concepts, step by step ### Step 1 — movemask: from vector comparison to iterable integer -A SIMD comparison (e.g. `vceqq_u8` on NEON, ARM's 128-bit SIMD -instruction set) compares 16 byte lanes at once, producing a vector -where each matching lane is 0xFF and each non-match is 0x00. Useless -by itself — you can't loop over a vector. x86's `PMOVMSKB` -("movemask") fixes that: it extracts the top bit of each lane into a -16-bit integer, one bit per lane. Now ordinary scalar tools finish -the job: `mask != 0` (any match?), `trailing_zeros()` (index of the -first match), clear-lowest-bit (next match). Search = one vector -compare + one movemask + bit iteration. NEON has the compare but not -the movemask — hence this chapter. +> **In:** a vector of 16 comparison results, each lane `0xFF` or +> `0x00`. +> **Out:** an integer whose set bits name the matching lanes, so +> scalar bit tricks can finish the job. + +A SIMD comparison such as NEON's `vceqq_u8` compares 16 byte lanes at +once and writes `0xFF` into every matching lane and `0x00` into every +other. That is useless on its own, because you cannot loop over a +vector: to *use* the result you need to know which lanes matched. + +x86 has one instruction for the conversion, `PMOVMSKB`: take the top +bit of each of the 16 lanes and pack them into a 16-bit integer. After +that, ordinary scalar tools finish the search — `mask != 0` answers +"any match?", `trailing_zeros()` gives the first matching lane, +`mask & (mask - 1)` clears it and moves to the next. + +NEON has the compare and not the pack. Everything below is a +consequence. ### Step 2 — three answers to "one bit per lane" +> **In:** the comparison vector from Step 1, on a machine with no +> `PMOVMSKB`. +> **Out:** three different integers, with three different bits-per-lane +> conventions — and therefore three different index arithmetics. + +``` + (a) SSE2, 16-byte group native, 1 bit per lane + _mm_cmpeq_epi8 -> 16 lanes of 0xFF/0x00 + _mm_movemask_epi8-> u16, bit i = lane i + BITMASK_STRIDE = 1 (sse2.rs:12) + + (b) NEON, memchr style, 16-byte vector 4 bits per lane + vceqq_u8 -> 16 lanes of 0xFF/0x00 + vshrn_n_u16(_,4) -> narrow each u16 pair to 8 bits, keeping + the top nibble of each half + vget_lane_u64 -> u64, each lane owning a NIBBLE + & 0x8888... -> one bit per nibble (vector.rs:325-328) + lane = trailing_zeros() >> 2 (vector.rs:455) + + (c) NEON, hashbrown style, 8-byte group 8 bits per lane + vceq_u8 on uint8x8_t -> 8 lanes of 0xFF/0x00 = one u64 exactly + vget_lane_u64 -> done; no narrowing instruction at all + BITMASK_STRIDE = 8 (neon.rs:8) + lane = trailing_zeros() / 8 (bitmask.rs:58) +``` + +memchr keeps 16 lanes and pays one `vshrn_n_u16` (shift-right-narrow — +halve each 16-bit element to 8 bits; here abused so that each pair of +byte lanes contributes one nibble). hashbrown instead **shrinks its +unit of work to 8 bytes**, so the comparison result reinterpreted as a +`u64` already *is* the bitmask, at the cost of scanning half as many +slots per instruction. + +Verify which branch your machine takes, because it decides every +number in Steps 4 and 5: + +```rust +// hashbrown src/control/group/mod.rs:8-33 — backend selection + 8 cfg_if! { +// ... 9-12: SSE2 preferred; no AVX because the probability of finding a +// match drops off drastically after the first few buckets ... + 14 // I attempted an implementation on ARM using NEON instructions, but it + 15 // turns out that most NEON instructions have multi-cycle latency, which in + 16 // the end outweighs any gains over the generic implementation. + 17 if #[cfg(all( + 18 target_feature = "sse2", +// ... 19-21: x86 / x86_64, not miri ... + 22 mod sse2; + 23 use sse2 as imp; + 24 } else if #[cfg(all( + 25 target_arch = "aarch64", + 26 target_feature = "neon", +// ... 27-30: little-endian only, not miri ... + 32 mod neon; + 33 use neon as imp; +``` + +Read that carefully. The comment at lines 14-16 is **stale**: it +records a past experiment, but a NEON backend ships now and lines 24-33 +select it on your machine. It also does not say what the failed +experiment's group width was, does not cite a benchmark, and does not +mention narrowing — it blames "multi-cycle latency" generally. Any +claim that "a 16-byte NEON group lost to the u64 SWAR in benchmarks" +is not supported by anything in this repository; what the source +supports is only that an earlier attempt lost, and that the attempt +which finally shipped **shrank the group to 8 bytes so that no +narrowing instruction is needed at all**: + +```rust +// hashbrown src/control/group/neon.rs:6-21 — the shape of the group + 6 pub(crate) type BitMaskWord = u64; + 8 pub(crate) const BITMASK_STRIDE: usize = 8; + 9 pub(crate) const BITMASK_ITER_MASK: BitMaskWord = 0x8080_8080_8080_8080; +// ... 11-15: doc comment — "uses a 64-bit NEON value" ... + 16 pub(crate) struct Group(neon::uint8x8_t); +// ... 18-20 ... + 21 pub(crate) const WIDTH: usize = mem::size_of::(); ``` - SSE2 (16B group): vceqq → PMOVMSKB → u16, 1 bit/lane. Native. Done. - NEON, memchr style (16B): no PMOVMSKB. Idiom: - vceqq_u8 → 16 lanes of 0xFF/0x00 - vshrn_n_u16(,4) → narrow each u16 pair, keeping 4 bits per byte - vget_lane_u64 → u64 where each lane owns a NIBBLE - & 0x8888... → keep 1 bit per nibble (vector.rs:322-328) - position = trailing_zeros() >> 2 ← note the /4! +`uint8x8_t` is 8 bytes, so `WIDTH` is **8**. The SSE2 sibling +(`sse2.rs:20`) is `__m128i`, so its `WIDTH` is **16**. Same crate, +same algorithm, half the group. + +And the match itself is two instructions: - NEON, hashbrown style (8B group): don't narrow at all. - vceq_u8 on uint8x8_t → 8 lanes of 0xFF/0x00 = exactly one u64 - vget_lane_u64 → done. BitMask where each lane owns a BYTE. - position = trailing_zeros() >> 3 +```rust +// hashbrown src/control/group/neon.rs:68-73 — match_tag on aarch64 + 68 pub(crate) fn match_tag(self, tag: Tag) -> BitMask { + 69 unsafe { + 70 let cmp = neon::vceq_u8(self.0, neon::vdup_n_u8(tag.0)); + 71 BitMask(neon::vget_lane_u64(neon::vreinterpret_u64_u8(cmp), 0)) + 72 } + 73 } ``` -memchr keeps 16 lanes and pays one `vshrn` (shift-right-narrow: an -instruction that halves each 16-bit element to 8 bits, here abused -to fold 16 comparison lanes into a 64-bit mask with 4 bits per -lane). hashbrown instead SHRINKS its unit of work to 8 bytes so the -comparison result, reinterpreted as a u64, already *is* the bitmask -— zero extra instructions. Question: why does the right choice -differ? (Hint: hash probing expects to find its match in the first -group — mod.rs's comment: "the probability of finding a match drops -off drastically after the first few buckets" — while memchr scans -megabytes and amortizes.) +Line 70 compares, line 71 moves the 64-bit result to a general-purpose +register. `vreinterpret_u64_u8` is a type pun, not an instruction. +There is no movemask because there is nothing left to pack. + +Two details make the byte-per-lane convention usable. First, the +divide: `bitmask.rs:58` computes `self.0.trailing_zeros() / +BITMASK_STRIDE`, and `BITMASK_STRIDE` is 8 here and 1 on SSE2 — so the +*same* iterator code yields lane indices on both. Second, the mask at +`bitmask.rs:89`: + +```rust +// hashbrown src/control/bitmask.rs:86-90 — why iteration needs a second mask + 86 fn into_iter(self) -> BitMaskIter { + 87 // A BitMask only requires each element (group of bits) to be non-zero. + 88 // However for iteration we need each element to only contain 1 bit. + 89 BitMaskIter(BitMask(self.0 & BITMASK_ITER_MASK)) + 90 } +``` -### Step 3 — SWAR: a u64 is an 8-lane vector if you're careful +On NEON a matching lane is `0xFF` — eight set bits, so +`trailing_zeros` would land correctly but `mask & (mask-1)` would step +*within* a lane. ANDing with `0x8080_8080_8080_8080` (`neon.rs:9`) +keeps exactly the top bit of each lane. On SSE2 `BITMASK_ITER_MASK` is +`!0` (`sse2.rs:13`) — a no-op, because `PMOVMSKB` already produced one +bit per lane. -SWAR (SIMD within a register — doing lane-parallel work with plain -integer instructions on a u64) is the portable fallback when there's -no SIMD at all (hashbrown's generic.rs). Compare 8 bytes against a -tag in four ALU ops: +### Step 3 — SWAR: a u64 is an 8-lane vector if you are careful + +> **In:** eight control bytes packed in a plain `u64`, and a tag byte. +> **Out:** the same `BitMask` the SIMD backends produce, using four +> integer instructions and no vector unit at all. + +SWAR — SIMD Within A Register — does lane-parallel work with ordinary +integer instructions. It is hashbrown's fallback when no supported +vector ISA is present (`mod.rs:42-44`), and it is the reference +implementation the SIMD backends are checked against: ```rust -let cmp = self.0 ^ repeat(tag); // matching byte → 0x00 -BitMask((cmp.wrapping_sub(repeat(0x01)) & !cmp & repeat(0x80)).to_le()) +// hashbrown src/control/group/generic.rs:97-109 — SWAR match_tag + 97 /// This function may return a false positive in certain cases where + 98 /// the tag in the group differs from the searched value only in its + 99 /// lowest bit. This is fine because: + 100 /// - This never happens for `EMPTY` and `DELETED`, only full entries. + 101 /// - The check for key equality will catch these. + 102 /// - This only happens if there is at least 1 true match. + 103 /// - The chance of this happening is very low (< 1% chance per tag). +// ... 104-107: attribute and the bithacks citation ... + 108 let cmp = self.0 ^ repeat(tag); + 109 BitMask((cmp.wrapping_sub(repeat(Tag(0x01))) & !cmp & repeat(Tag::DELETED)).to_le()) ``` -XOR turns matches into zero bytes; then the classic zero-byte -detector — subtracting 1 borrows into bit 7 only where the byte was -0 — leaves bit 7 set per matching lane. The subtraction's borrow can -ripple across lane boundaries, so this can false-positive on -adjacent bytes. Question: why is that acceptable here (what does the -caller do with a candidate match)? Compare neon.rs which has no -false positives. Remarkably, hashbrown's mod.rs comment records that -a 16-byte NEON Group *lost* to this u64 SWAR in benchmarks — the -narrowing overhead wasn't worth it for probes that end early. +Line 108 turns every matching byte into `0x00`. Line 109 is the classic +zero-byte detector: subtracting `0x01` from each byte borrows into +bit 7 only where the byte was zero; `& !cmp` cancels bytes that already +had bit 7 set; `& 0x8080…` keeps one bit per lane. Four ALU ops for +eight lanes. + +Get the failure mode right, because it is narrower than "adjacent +bytes can false-positive". The doc comment at lines 97-103 says the +false positive happens when a byte differs from the searched tag +**only in its lowest bit**, that it happens **only when there is at +least one true match** (the borrow has to come from somewhere), and +that its probability is **under 1 % per tag**. Work an example: + +``` + tag = 0x51, group byte = 0x50 (differs only in bit 0) + cmp = 0x50 ^ 0x51 = 0x01 + cmp - 0x01 = 0x00 <- borrow did NOT leave the byte + & !cmp = 0x00 & 0xFE = 0x00 <- no false positive on its own + now put a true match (0x51) in the byte BELOW it: + that byte's cmp = 0x00, so its subtraction borrows out of the byte, + and the borrow lands in the 0x01 byte, turning it into 0xFF + -> bit 7 set -> a false positive, exactly as documented +``` + +`match_empty` needs no such trick, because `EMPTY` (`0b1111_1111`) and +`DELETED` (`0b1000_0000`) are the only tags with the top bit set +(`tag.rs:9` and `:12`), and only `EMPTY` also has bit 6 set: + +```rust +// hashbrown src/control/group/generic.rs:115-119 — no subtraction needed + 115 pub(crate) fn match_empty(self) -> BitMask { +// ... 116-118: comment — top two bits set means EMPTY ... + 119 BitMask((self.0 & (self.0 << 1) & repeat(Tag::DELETED)).to_le()) +``` + +Three ops, no borrow, no false positive. The encoding was designed to +make this possible. ### Step 4 — SwissTable: control bytes with the answer in the sign bit -SwissTable (the hash-table design behind hashbrown, hence Rust's -`HashMap`) keeps, alongside the key/value slots, one **control -byte** per slot in a dense array — and probes the control bytes a -**group** (8 or 16) at a time with exactly Step 2's machinery. Each -control byte is either EMPTY=0xFF, DELETED=0x80, or FULL=0..0x7f -(a 7-bit **tag** — the top 7 bits of the hash, stored so most -non-matching slots are rejected without touching the actual key): +> **In:** a 64-bit hash and a table of control bytes. +> **Out:** a group index, a 7-bit tag, and three single-instruction +> predicates over a whole group. + +SwissTable — the design behind hashbrown, and therefore behind Rust's +`HashMap` — stores one **control byte** per slot in a dense array +beside the key/value slots, and probes the control array a **group** +at a time using exactly Step 2's machinery. Each control byte is +`EMPTY = 0b1111_1111`, `DELETED = 0b1000_0000`, or a 7-bit **tag** in +`0x00..=0x7f` taken from the top of the hash: + +```rust +// hashbrown src/control/tag.rs:35-48 — h2, the 7-bit tag + 35 pub(crate) const fn full(hash: u64) -> Tag { +// ... 36-46: MIN_HASH_LEN handles hashers that only fill a usize ... + 47 let top7 = hash >> (MIN_HASH_LEN * 8 - 7); + 48 Tag((top7 & 0x7f) as u8) // truncation +``` + +On a 64-bit target line 47 is `hash >> 57`. Note that the *top* bits go +into the tag while the *bottom* bits pick the group — two disjoint +slices of the same hash, so a slot's tag carries information the group +index does not. + +The encoding is the trick: `EMPTY` and `DELETED` both have the sign bit +set and a full tag never does, so each predicate is one comparison: + +```rust +// hashbrown src/control/group/neon.rs:85-97 — the sign-bit predicates + 85 pub(crate) fn match_empty_or_deleted(self) -> BitMask { + 87 let cmp = neon::vcltz_s8(neon::vreinterpret_s8_u8(self.0)); + 88 BitMask(neon::vget_lane_u64(neon::vreinterpret_u64_u8(cmp), 0)) +// ... 92-93: doc comment for match_full ... + 94 pub(crate) fn match_full(self) -> BitMask { + 96 let cmp = neon::vcgez_s8(neon::vreinterpret_s8_u8(self.0)); + 97 BitMask(neon::vget_lane_u64(neon::vreinterpret_u64_u8(cmp), 0)) +``` + +`vcltz_s8` is "lanes less than zero as signed bytes" — that is the sign +bit, and it answers "empty or deleted" with no constant to load and no +comparison operand. `vcgez_s8` at line 96 is its complement, "full". + +### Step 5 — the probe loop, and what the group width costs + +> **In:** a hash, a table, and an equality closure. +> **Out:** the index of the matching slot, or `None` — visiting one +> whole group per iteration. + +The real loop is short enough to read whole: + +```rust +// hashbrown src/raw.rs:2009-2045 — RawTableInner::find_inner + 2009 unsafe fn find_inner(&self, hash: u64, eq: &mut dyn FnMut(usize) -> bool) -> Option { + 2010 let tag_hash = Tag::full(hash); + 2011 let mut probe_seq = self.probe_seq(hash); + 2013 loop { +// ... 2014-2027: safety argument for the unaligned load ... + 2028 let group = unsafe { Group::load(self.ctrl(probe_seq.pos)) }; + 2030 for bit in group.match_tag(tag_hash) { +// ... 2031-2032: the mask is cheaper than a modulo ... + 2033 let index = (probe_seq.pos + bit) & self.bucket_mask; + 2035 if likely(eq(index)) { + 2036 return Some(index); + 2037 } + 2038 } + 2040 if likely(group.match_empty().any_bit_set()) { + 2041 return None; + 2042 } + 2044 probe_seq.move_next(self.bucket_mask); + 2045 } +``` + +Line 2030 iterates only the candidate lanes; line 2035 is the only +place a real key is touched. Line 2040 is the termination rule: an +`EMPTY` slot in the group proves the key is absent, because insertion +would have used it. A tag false positive — two keys sharing 7 bits, or +Step 3's borrow noise — costs exactly one extra `eq` call. Correctness +never depended on the mask being exact, only on it never *missing* a +real match. + +Line 2044 is not a linear scan: + +```rust +// hashbrown src/raw.rs:83-92 — triangular probing + 83 fn move_next(&mut self, bucket_mask: usize) { +// ... 84-89: debug assertion that the sequence has not run off the end ... + 90 self.stride = self.stride.wrapping_add(Group::WIDTH); + 91 self.pos = self.pos.wrapping_add(self.stride) & bucket_mask; +``` + +The stride grows by one group each time, so the visited positions are +the triangular numbers times `WIDTH` — which, for a power-of-two table, +visits every group exactly once (the proof is linked at line 74). + +Now cost it, because this is where the group width shows up. +hashbrown's maximum load factor is 7/8: `raw.rs:182-190` reserves +"12.5 % of the slots as empty". So, ignoring `DELETED` and treating +slots as independent, the probability that a group contains **no** +empty slot — i.e. that the probe must continue — is `(7/8)^W`: ``` - h1(hash) → group index h2(hash) → 7-bit tag - ┌────────────────────── one group (8 or 16 control bytes) - │ 0x51 0x7f EMPTY 0x51 DEL 0x12 ... - └── match_tag(0x51) → candidates 0b...01001 → probe those slots - match_empty() → can this group absorb an insert? - match_empty_or_deleted() → insertion slot (vcltz: top bit set) + W = 8 (NEON, your machine): (7/8)^8 = 0.3436 + W = 16 (SSE2): (7/8)^16 = 0.1181 + + expected groups scanned = 1 / (1 - p) + W = 8 : 1 / 0.6564 = 1.524 groups -> 1.524 * 8 = 12.2 control bytes + W = 16: 1 / 0.8819 = 1.134 groups -> 1.134 * 16 = 18.1 control bytes + + instructions per group (Step 2): load + dup + compare + extract = 4 + W = 8 : 4 / 8 = 0.50 instructions per slot scanned + W = 16: 4 / 16 = 0.25 + a scalar per-slot probe: load + compare + branch >= 3 per slot +``` + +Two conclusions the width hazard is designed to hide. The 8-wide group +takes **more iterations** (1.52 vs 1.13) but touches **fewer control +bytes** (12.2 vs 18.1), because a wide group scans slots it did not +need. And even the "worse" NEON path is about 6× fewer instructions per +slot than a scalar probe. That is why halving the group is survivable; +it is also why `mod.rs:9-12` refuses to go *wider* than 16 with AVX — +"the probability of finding a match drops off drastically after the +first few buckets", so extra width buys slots you were never going to +look at. + +(Two caveats, stated because the model is a model: `DELETED` bytes do +not terminate a probe, which pushes both numbers up in a table that has +seen erases; and after the first iteration the triangular stride jumps +to an uncorrelated region, so the independence assumption is better +after the first group than within it.) + +### Step 6 — memchr's 4× unroll, and the movemask it does *not* pay + +> **In:** a haystack of megabytes and one needle byte. +> **Out:** the position of the first match, having executed zero +> `vshrn` sequences on any block that does not contain one. + +memchr has the opposite profile from hash probing: it expects to scan +enormous runs of non-matches, so the miss path is the one to optimise. +`arch/generic/memchr.rs:107` sets `LOOP_SIZE = 4 * V::BYTES` — 64 bytes +on NEON — and the loop loads four vectors, compares each, and combines: + +```rust +// memchr src/arch/generic/memchr.rs:172-206 — the unrolled search loop + 172 while cur <= end.sub(Self::LOOP_SIZE) { + 175 let a = V::load_aligned(cur); + 176 let b = V::load_aligned(cur.add(1 * V::BYTES)); + 177 let c = V::load_aligned(cur.add(2 * V::BYTES)); + 178 let d = V::load_aligned(cur.add(3 * V::BYTES)); + 179 let eqa = self.v1.cmpeq(a); +// ... 180-182: eqb, eqc, eqd ... + 183 let or1 = eqa.or(eqb); + 184 let or2 = eqc.or(eqd); + 185 let or3 = or1.or(or2); + 186 if or3.movemask_will_have_non_zero() { + 187 let mask = eqa.movemask(); + 188 if mask.has_non_zero() { + 189 return Some(cur.add(topos(mask))); + 190 } +// ... 192-204: the same for eqb, eqc, eqd; the last needs no test ... + 205 } + 206 cur = cur.add(Self::LOOP_SIZE); +``` + +The OR tree at 183-185 collapses four comparison vectors into one, so +the loop asks a single question per 64 bytes. That much is the same +idea as simdjson's block pipeline and polars' one-branch-per-block +filter: amortise the expensive extraction, then localise only on a hit. + +But look at line 186, and at what NEON does with it: + +```rust +// memchr src/vector.rs:358-368 — the NEON override + 358 /// This is the only interesting implementation of this routine. + 359 /// Basically, instead of doing the "shift right narrow" dance, we use + 360 /// adjacent folding max to determine whether there are any non-zero + 361 /// bytes in our mask. If there are, *then* we'll do the "shift right + 362 /// narrow" dance. In benchmarks, this does lead to slightly better + 363 /// throughput, but the win doesn't appear huge. + 365 unsafe fn movemask_will_have_non_zero(self) -> bool { + 366 let low = vreinterpretq_u64_u8(vpmaxq_u8(self, self)); + 367 vgetq_lane_u64(low, 0) != 0 + 368 } ``` -The encoding is the trick: EMPTY and DELETED both have the top -(sign) bit set, FULL never does — so all three predicates are -single-instruction (neon.rs:85,94 use `vcltz`/`vcgez`, "compare -less-than/greater-equal zero" on signed bytes). +`vpmaxq_u8` is a pairwise maximum: it folds the 16 lanes down so that +any non-zero byte survives into the low half, and one `vgetq_lane_u64` +plus a compare answers "is anything set?". So on the miss path — the +one that runs for essentially the whole haystack — memchr executes +**no** `vshrn`, no `& 0x8888…`, and no `movemask` at all. The nibble +dance at `vector.rs:323-329` runs only inside the `if` at line 186, +i.e. only in the 64-byte block that actually contains the match. -### Step 5 — the probe loop: ~3 instructions per 8–16 slots +That is a strictly better claim than "one movemask per 64 bytes", and +it is the shape to copy: the *cheapest possible* any-match test on the +hot path, the *precise* extraction only where the answer matters. -Assemble Steps 1–4 and a lookup probes a whole group per iteration: +Finally, note the index arithmetic that goes with convention (b). +Because each lane owns a nibble, `topos` must divide by 4: ```rust -// the probe loop at group granularity: ~3 instructions per 8-16 slots -fn find(&self, hash: u64, key: &K) -> Option { - let (mut g, tag) = (h1(hash) & self.mask, h2(hash)); // 7-bit tag - loop { - let group = Group::load(&self.ctrl[g]); - let mut m = group.match_tag(tag); // vceq + extract → BitMask - while let Some(i) = m.next_set() { // trailing_zeros() >> 3 - if self.slot(g + i).key == *key { return Some(g + i); } - } // false positive? just loop - if group.match_empty().any() { return None; } // EMPTY ends the probe - g = (g + GROUP_SIZE) & self.mask; // (triangular in real code) - } -} -``` - -Tag false positives (two keys sharing a 7-bit tag, or SWAR borrow -noise) just cost one extra key comparison — correctness never -depended on the mask being exact, only on it never missing a real -match. This is topic 2's hash table at lane level. Question: rewrite -your M2 probe loop's per-slot compare as a per-group `match_tag` and -count instructions per probed slot. - -### Step 6 — memchr's 4× unroll: one movemask per 64 bytes - -memchr (substring/byte search) has the opposite profile from probing -— it expects to scan megabytes of non-matches. Its main loop -(arch/generic/memchr.rs:171-206, `LOOP_SIZE = 4 * V::BYTES`) loads 4 -vectors, `cmpeq`s each, ORs the four results together, and pays the -(NEON-expensive) movemask ONCE per 64 bytes; only on a hit does it -re-movemask the individual vectors to localize the match. The miss -path — the overwhelmingly common one — stays minimal. Same shape as -polars' one-branch-per-block filter and simdjson's 64-byte stage 1: -amortize the expensive extraction over a block, localize only on -hit. Question: why OR-then-locate instead of 4 movemask+test — count -the instructions on the miss path. +// memchr src/vector.rs:453-456 — first_offset for the nibble mask + 453 #[inline(always)] + 454 fn first_offset(self) -> usize { + 455 (self.0.trailing_zeros() >> 2) as usize + 456 } +``` + +Forgetting the `>> 2` gives an offset 4× too large — which is exactly +why memchr wraps the value in a `NeonMoveMask` newtype +(`vector.rs:387`) instead of passing a bare `u64` around: the type is +what stops a nibble-mask being used where a bit-mask is expected. +hashbrown makes the same move with `BITMASK_STRIDE` (8 on NEON, +1 on SSE2), so its shared `BitMask` code divides by the right constant +without knowing which backend it is on. ### Step 7 — the portability pattern to copy -Both crates write the ALGORITHM once against a tiny interface — -memchr's `Vector` trait (splat/load/cmpeq/movemask), hashbrown's -`Group` struct with one file per backend — and each ISA implements -that interface in ~100 lines of intrinsics, selected by `cfg_if!` at -COMPILE time (vs polars' runtime dispatch — binding times again). -The abstraction cost is zero after monomorphization, and the -generic/SWAR backend doubles as the oracle for testing the SIMD -ones. This is the shape for M17's kernel layer. +> **In:** one algorithm you want on three ISAs. +> **Out:** one algorithm, three ~100-line backends, and zero +> abstraction cost after monomorphisation. + +Both crates write the algorithm once against a tiny interface and +implement that interface per ISA: + +- memchr defines `trait Vector` (`vector.rs:17`) with `BYTES` + (`:20`), `splat`, `load`, `cmpeq`, `movemask` (`:54`) and the + optional `movemask_will_have_non_zero` from Step 6; a second trait, + `MoveMask` (`vector.rs:82`), owns the bits-per-lane convention. +- hashbrown defines `struct Group` with `WIDTH`, `load`, `match_tag`, + `match_empty`, `match_empty_or_deleted`, `match_full` — one file per + backend, chosen by the `cfg_if!` of Step 2. + +Both bind at **compile** time. That is the cheapest of the three +binding times this topic shows you: hashbrown and memchr bind with +`cfg`, polars binds at **call** time with a runtime feature test +(`filter/primitive.rs:33`), and SimSIMD binds at **init** time with a +function-pointer table filled by a library constructor +(`c/numkong.c:917`). Compile-time binding is right here because a +`HashMap` probe is a handful of instructions — an indirect call would +cost more than the work — and because Rust ships source, so the user's +own `cargo build` is the dispatch. + +The generic/SWAR backend earns its keep twice: it is the portability +floor, and it is the oracle. Any new backend must agree with it on +every input, which is exactly the testing strategy for `filter.rs`'s +NEON compaction — write the scalar version first, then diff. ## Where each step lives in the code +hashbrown at `d69025b`, memchr at `5fdb40c`. + | anchor | step | what it is | |---|---|---| -| hashbrown group/mod.rs:8-30 | 2, 7 | the `cfg_if!` backend choice + the famous "NEON wasn't worth it" comment | -| hashbrown group/sse2.rs:20 | 2 | `Group(__m128i)` — 16 control bytes | -| hashbrown group/sse2.rs:73-84 | 1–2 | `match_tag` = `_mm_cmpeq_epi8` + `_mm_movemask_epi8` → `BitMask(u16)` | -| hashbrown group/neon.rs:16 | 2 | `Group(uint8x8_t)` — EIGHT bytes, not 16! | -| hashbrown group/neon.rs:68-75 | 2 | `match_tag` = `vceq_u8` + reinterpret as u64 — NO movemask at all | -| hashbrown group/neon.rs:85-99 | 4 | `match_empty_or_deleted` via `vcltz_s8` (sign bit test) | -| hashbrown group/generic.rs:41 | 3 | `Group(GroupWord)` — SWAR on a plain u64 | -| hashbrown group/generic.rs:105-109 | 3 | SWAR match_tag: `x ^ repeat(tag)`, then the zero-byte trick | -| memchr vector.rs:25-64 | 7 | the `Vector` trait: splat/load/cmpeq/movemask over 3 ISAs | -| memchr vector.rs:322-328 | 2 | NEON movemask: `vshrn_n_u16(_, 4)` → u64 with 4 bits/lane | -| memchr arch/generic/memchr.rs:107 | 6 | `LOOP_SIZE = 4 * V::BYTES` — the 4× unroll | -| memchr arch/generic/memchr.rs:171-206 | 6 | the unrolled search loop (OR-combine 4 cmpeqs, one movemask check) | - -Reading order: hashbrown's mod.rs comment first (the design doc), -then sse2.rs → neon.rs → generic.rs in that order (native → shrunk -→ SWAR), then memchr's vector.rs and the unrolled loop. +| `hashbrown src/control/group/mod.rs:8-45` | 2, 7 | the `cfg_if!` backend choice; the stale "NEON wasn't worth it" comment at 14-16 | +| `hashbrown src/control/group/sse2.rs:12-20` | 2 | `BITMASK_STRIDE = 1`, `Group(__m128i)` — 16 control bytes | +| `hashbrown src/control/group/sse2.rs:73-86` | 1-2 | `match_tag` = `_mm_cmpeq_epi8` + `_mm_movemask_epi8` | +| `hashbrown src/control/group/neon.rs:6-21` | 2 | `BitMaskWord = u64`, `BITMASK_STRIDE = 8`, `Group(uint8x8_t)` — EIGHT bytes | +| `hashbrown src/control/group/neon.rs:68-73` | 2 | `match_tag` = `vceq_u8` + `vget_lane_u64` — no narrowing at all | +| `hashbrown src/control/group/neon.rs:85-99` | 4 | `match_empty_or_deleted` via `vcltz_s8`, `match_full` via `vcgez_s8` | +| `hashbrown src/control/bitmask.rs:55-59` | 2 | the divide by `BITMASK_STRIDE` that makes both conventions share code | +| `hashbrown src/control/bitmask.rs:86-90` | 2 | `BITMASK_ITER_MASK` — one bit per lane before iterating | +| `hashbrown src/control/group/generic.rs:97-109` | 3 | SWAR `match_tag` and the exact false-positive condition | +| `hashbrown src/control/group/generic.rs:115-119` | 3 | `match_empty` with no subtraction — why the encoding matters | +| `hashbrown src/control/tag.rs:9,12,35-48` | 4 | `EMPTY`, `DELETED`, and the top-7-bits tag | +| `hashbrown src/raw.rs:2009-2045` | 5 | `find_inner` — the real probe loop | +| `hashbrown src/raw.rs:83-92` | 5 | `ProbeSeq::move_next` — triangular stride | +| `hashbrown src/raw.rs:182-190` | 5 | the 7/8 maximum load factor used in Step 5's arithmetic | +| `memchr src/vector.rs:17-82` | 7 | `trait Vector` and `trait MoveMask` | +| `memchr src/vector.rs:321-329` | 2 | NEON movemask: `vshrn_n_u16(_, 4)` then `& 0x8888…` | +| `memchr src/vector.rs:358-368` | 6 | `movemask_will_have_non_zero` via `vpmaxq_u8` — the miss path | +| `memchr src/vector.rs:453-456` | 2 | `first_offset` divides by 4 | +| `memchr src/arch/generic/memchr.rs:107` | 6 | `LOOP_SIZE = 4 * V::BYTES` | +| `memchr src/arch/generic/memchr.rs:172-206` | 6 | the unrolled loop: OR tree, one test, localise on hit | + +Reading order: `mod.rs` first (48 lines, and it tells you which file +matters), then `neon.rs` → `sse2.rs` → `generic.rs` in that order +(shrunk → native → SWAR), then `bitmask.rs` to see how the three +conventions share an iterator, then `raw.rs:2009` for the loop that +uses them. Then memchr's `vector.rs` NEON block and the unrolled loop. ## Questions for notes.md -1. hashbrown mod.rs says a 16-byte NEON Group lost to the generic - u64 SWAR. What cost model explains that (latency of narrow + - extract vs the SWAR's 4 ALU ops)? -2. The vshrn nibble-mask means `trailing_zeros()>>2`; hashbrown's - byte-mask means `>>3`. What breaks if you forget the shift? - (memchr wraps it in `NeonMoveMask` newtype — why?) -3. SWAR match_tag tolerates false positives; match_empty (bit - pattern 0b1111_1111) doesn't need the subtract trick — why - (generic.rs:119 uses `self.0 & (self.0<<1)`)? -4. For M2's table: your tags are full hashes. What do you lose by - truncating to 7 bits + sign-bit encoding, and what do you gain - per probe? -5. Compile-time cfg (here) vs runtime detect (polars) vs init-time - fn pointers (SimSIMD): which fits a Cypher engine that ships one - binary to unknown ARM servers? +1. Redo Step 5's probe arithmetic for a table at 50 % load instead of + 87.5 %. At what load factor does the expected number of control + bytes scanned by `W = 8` exceed that of `W = 16`, if ever? +2. Convention (b) gives 4 bits per lane and convention (c) gives 8. + Write the two `trailing_zeros` expressions, then say what goes wrong + if you swap them: which one silently returns a *valid but wrong* + index, and which one returns an out-of-range one? +3. `generic.rs:97-103` says the SWAR false positive needs at least one + true match. Construct an 8-byte group and a tag where it fires, and + one where a naive reading ("adjacent bytes false-positive") would + predict a hit but the code gives none. +4. For M2's table: your tags are currently full hashes. Compute what + truncating to 7 bits costs you — the probability that a + non-matching slot survives `match_tag` — and what it buys per probe + in bytes touched. +5. Compile-time `cfg` (here), runtime detect (polars + `filter/primitive.rs:33`), init-time function pointers (SimSIMD + `c/numkong.c:917`): which fits a Cypher engine shipping one binary + to unknown ARM servers, and what does that choice cost in the + `HashMap` probe specifically? ## Done when -- [ ] You can explain movemask and give the three different answers to "one bit per lane" across ISAs. -- [ ] You can explain SWAR: how a u64 acts as an 8-lane vector, and which operation is the dangerous one. -- [ ] You can narrate the SwissTable probe loop and count the instructions per 8-16 slots. -- [ ] You can explain why `match_tag` may tolerate false positives while `match_empty` may not. -- [ ] You can state the portability pattern — and say why hashbrown found a 16-byte NEON group *lost* to the generic path. -- [ ] You wrote answers to all five questions in notes.md. +Answer each before unfolding it. + +- [ ] You can explain what movemask is for, and give the three different answers to "one bit per lane" with their index arithmetic. + +
Answer + + A vector compare produces lanes of `0xFF`/`0x00`, which you cannot + loop over; movemask converts that to an integer whose bits name the + matching lanes, after which `trailing_zeros` and `mask & (mask-1)` + finish the search. + + (a) SSE2: `_mm_movemask_epi8` gives 1 bit per lane, stride 1 + (`sse2.rs:12`). (b) memchr on NEON: `vshrn_n_u16(_,4)` plus + `& 0x8888…` gives 4 bits per lane, so the index is + `trailing_zeros() >> 2` (`vector.rs:325-328`, `:455`). (c) hashbrown + on NEON: an 8-byte group means the compare result *is* the mask, + 8 bits per lane, index `trailing_zeros() / 8` + (`neon.rs:8`, `bitmask.rs:58`). + +
+ +- [ ] You can state hashbrown's group width on your machine, prove it from the source, and say what the crate's design comment does and does not claim. + +
Answer + + **8 bytes.** `mod.rs:24-33` selects `mod neon` when + `target_arch = "aarch64"`, `target_feature = "neon"` and + `target_endian = "little"`; `neon.rs:16` is + `struct Group(neon::uint8x8_t)` and `neon.rs:21` defines `WIDTH` as + its size — 8. SSE2's is `__m128i`, so 16 (`sse2.rs:20`). + + The comment at `mod.rs:14-16` says only that an earlier NEON attempt + lost to the generic implementation because "most NEON instructions + have multi-cycle latency". It names no width, cites no benchmark, and + is now stale, since a NEON backend ships. The design that *did* win + is visible in the code: shrink the group to 8 so that + `vceq_u8` + `vget_lane_u64` needs no narrowing step at all. + +
+ +- [ ] You can explain SWAR, and state the SWAR false positive's exact condition rather than a vague one. + +
Answer + + SWAR treats a `u64` as eight byte lanes and uses integer + instructions: `cmp = word ^ repeat(tag)` makes matching bytes zero, + then `(cmp - repeat(0x01)) & !cmp & repeat(0x80)` sets bit 7 in + exactly the zero bytes (`generic.rs:108-109`). Four ALU ops for eight + lanes. + + The documented failure (`generic.rs:97-103`) is narrower than "the + borrow can cross lanes": it fires only for a byte differing from the + tag **in its lowest bit only**, only when there is **at least one + true match** to originate the borrow, never for `EMPTY` or `DELETED`, + and with probability under 1 % per tag. It is safe because the key + comparison at `raw.rs:2035` rejects it. + +
+ +- [ ] You can narrate the probe loop and compute what the group width costs, rather than asserting that wider is better. + +
Answer + + `find_inner` (`raw.rs:2009-2045`): compute the 7-bit tag, load a + group, iterate `match_tag`'s candidate lanes calling `eq` on each, + stop with `None` if the group holds an `EMPTY`, else advance by the + triangular stride (`raw.rs:90-91`). + + At hashbrown's 7/8 load factor (`raw.rs:188-189`), a group continues + the probe with probability `(7/8)^W`: 0.3436 at W = 8 and 0.1181 at + W = 16, so the expected groups scanned are 1.52 and 1.13 — but the + expected *control bytes* scanned are 12.2 and 18.1. The narrow group + loops more and reads less. Per slot the group probe costs + 4/8 = 0.5 instructions (NEON) or 4/16 = 0.25 (SSE2), against ≥ 3 for + a scalar per-slot probe. + +
+ +- [ ] You can say what memchr pays on the miss path, and why "one movemask per 64 bytes" is not quite right. + +
Answer + + Zero movemasks. The loop ORs four comparison vectors together + (`arch/generic/memchr.rs:183-185`) and tests the result with + `movemask_will_have_non_zero`, which NEON overrides + (`vector.rs:365-368`) to be a single `vpmaxq_u8` plus a lane read — + the comment at 358-363 says so explicitly. The `vshrn` / + `& 0x8888…` sequence runs only inside the `if` at line 186, i.e. only + in a 64-byte block that actually contains the needle. + + So the miss path costs 4 loads, 4 compares, 3 ORs, 1 fold and 1 test + per 64 bytes, and no bitmask is ever materialised. + +
+ +- [ ] You can state the portability pattern and place all three binding times in this topic. + +
Answer + + Write the algorithm once against a minimal interface — memchr's + `Vector`/`MoveMask` traits (`vector.rs:17`, `:82`), hashbrown's + `Group` struct — and give each ISA a ~100-line implementation. The + generic backend is both the portability floor and the test oracle. + + Binding times: **compile** (hashbrown/memchr `cfg_if!`), **init** + (SimSIMD's `__attribute__((constructor))` at `c/numkong.c:917` + filling a function-pointer table), **call** (polars' + `is_avx512_enabled()` test in `filter/primitive.rs:33`). Compile-time + wins here because the dispatched unit — one group probe — is smaller + than an indirect call. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including the load-factor crossover and the 7-bit-tag cost for M2. + +
Answer + + Self-check. Question 1 has a checkable form: solve + `(1/(1-α^8))·8 > (1/(1-α^16))·16` for α, and notice that the left + side is smaller for every α in (0,1) — a narrow group never reads + *more* control bytes, it only loops more often. Question 4 has a + one-line answer: a 7-bit tag lets 1/128 of non-matching slots through + to the key comparison. + +
## References **Code** -- [hashbrown](https://github.com/rust-lang/hashbrown) — - `src/control/group/` — one file per backend (sse2/neon/generic); - the `cfg_if!` block in `mod.rs` and its "NEON wasn't worth it" - comment are the design doc -- [memchr](https://github.com/BurntSushi/memchr) — `src/vector.rs` - (the `Vector` trait + NEON movemask idiom) and - `src/arch/generic/memchr.rs` (the 4× unrolled search loop) +- [hashbrown](https://github.com/rust-lang/hashbrown) at `d69025b` — + `src/control/group/` (one file per backend), `src/control/bitmask.rs` + and `src/control/tag.rs` for the shared conventions, and + `src/raw.rs:2009` for the probe loop that uses them. Read `neon.rs` + before `sse2.rs`: it is the one your machine compiles. +- [memchr](https://github.com/BurntSushi/memchr) at `5fdb40c` — + `src/vector.rs` (the `Vector` trait, the NEON movemask idiom, and the + `vpmaxq_u8` override that avoids it) and + `src/arch/generic/memchr.rs` (the 4× unrolled search loop). diff --git a/topics/17-simd/reading-mojo-simd.md b/topics/17-simd/reading-mojo-simd.md index 5b7f0fa..b5ea105 100644 --- a/topics/17-simd/reading-mojo-simd.md +++ b/topics/17-simd/reading-mojo-simd.md @@ -1,4 +1,4 @@ -# Mojo's `SIMD[type, width]`: width as a type parameter +# Mojo's `SIMD[dtype, size]`: width as a type parameter What does SIMD look like when the TYPE SYSTEM, not a library, owns it? In Mojo, scalars are literally width-1 vectors, and vector width @@ -10,6 +10,26 @@ it solves. Read it for the language-design angle; it's the contrast that explains why our Rust experiments hand-write what Mojo's `vectorize` generates. +**Read the version number before anything else.** Mojo is a moving +target: the documentation moved from `docs.modular.com/mojo/…` to +`mojolang.org/docs/…` (the old stdlib URLs now 404), the stdlib +namespace is `std.…`, the compile-time keyword is `comptime`, and +`simdwidthof` has been renamed `simd_width_of`. Every quotation below +is from **Mojo 1.0.0b2** — the version string the docs site reports +in its own page class (`docs-version-1.0.0b2`) — and every doc page +cited is reachable as clean Markdown by appending `.md` to its URL, +which is how the quotes here were extracted. If you are reading this +against a later release, re-check the names before trusting the +translation table in Step 8. + +One naming caveat worth stating up front: many older write-ups — +including this repo, until the H1 above was corrected alongside its +`SUMMARY.md` link — spell the type `SIMD[type, width]`. At 1.0.0b2 the +declaration is `struct SIMD[dtype: DType, size: Int]`: the second +parameter is **`size`**, not `width`, and the first is a `DType` +*value* rather than a type. Expect the old spelling in any tutorial +written before the rename. + ## The problem in one sentence The same dot-product kernel must be written once for NEON's 4 f32 @@ -22,153 +42,685 @@ the *code you wrote*, not a *parameter the compiler fills in*. ### Step 1 — the ladder of SIMD ergonomics +> **In:** the same operation — a fused multiply-add across a vector of +> f32 — expressed three ways. +> **Out:** the axis those three ways vary along, which is *who chooses +> the width*. + SIMD (single instruction, multiple data — one instruction operating on a vector of W values, its **lanes**) can be programmed at three levels of abstraction, each trading control for portability: ``` - raw intrinsics vfmaq_f32(acc, a, b) per-ISA names, unsafe, - (core::arch) exact instruction control + raw intrinsics vfmaq_f32(acc, a, b) per-ISA names, unsafe, + (core::arch) exact instruction control │ - portable library acc = a.mul_add(b, acc) one vocabulary, library - (std::simd, wide) Simd / f32x4 picks instructions; width - │ is a const generic bolted on + portable library acc = a.mul_add(b, acc) one vocabulary, library + (std::simd, wide) Simd / f32x4 picks instructions; width + │ is a const generic bolted on │ - language type SIMD[DType.float32, 4] scalars ARE SIMD[T,1]; - (Mojo) fn foo[w: Int](x: SIMD[T,w]) width is a first-class - compile-time parameter + language type SIMD[DType.float32, 4] scalars ARE SIMD[T, 1]; + (Mojo 1.0.0b2) fn f[w: Int](x: SIMD[T, w]) width is a first-class + compile-time parameter +``` + +**Intrinsics** are compiler-provided functions that map 1:1 to machine +instructions — NEON's `vfmaq_f32` *is* the `FMLA` in +`include/numkong/dot/neon.h:14`. Maximal control, zero portability: +`experiments/src/dot.rs:62` can only exist behind +`#[cfg(target_arch = "aarch64")]` at line 61. + +A **portable library** gives one vocabulary and lets the library pick +instructions. `wide::f32x4` and `std::simd::Simd` both do +this. The width is still a number *you* typed, though: polars writes +`const STRIPE: usize = 16;` (`float_sum.rs:13`) and that 16 is 16 on +every machine polars will ever run on. + +Mojo moves the whole idea into the language. Here is the declaration, +verbatim: + +```mojo +# mojolang.org/docs/std/builtin/simd/SIMD — Mojo 1.0.0b2, verbatim +struct SIMD[dtype: DType, size: Int] ``` -Intrinsics (compiler-provided functions mapping 1:1 to machine -instructions, like NEON's `vfmaq_f32`) name the exact instruction — -maximal control, zero portability. Portable libraries give one -vocabulary and let the library pick instructions. Mojo moves the -whole idea into the language. +Both parameters are compile-time. The doc's own summary of the +consequences is worth quoting because it is the language's thesis +statement: + +> **Hardware-mapped**: Directly maps to CPU vector registers · +> **Type-safe**: Data types and vector sizes are checked at compile +> time · **Zero-cost**: No runtime overhead compared to hand-optimized +> intrinsics · **Portable**: Same code works across different CPU +> architectures (x86, ARM, etc.) + +And one hard constraint, stated in the same page under +**Constraints**: "The size of the SIMD vector must be positive and a +power of 2." That is not a stylistic rule — it is what lets the +compiler decompose an over-wide vector into whole registers, which is +Step 6's subject. ### Step 2 — scalars are width-1 vectors -Mojo's unification move: `Float32` is literally an alias for -`SIMD[DType.float32, 1]`. There is no separate scalar world — every -scalar function is already the width-1 instance of a width-generic -function, so vectorizing an algorithm means changing a parameter, -not rewriting the body. Two practical wins: no duplicate -scalar/vector implementations to keep in sync, and the width-1 -instance is a free test oracle — run the same body at w=1 and w=4 -and diff the outputs (steal this for dot.rs's tests). +> **In:** Mojo's `Float32`, `Int8`, and the rest of the ordinary-looking +> numeric types. +> **Out:** the observation that none of them are scalar types, and the +> two practical consequences. -### Step 3 — parametric width in action: `simdwidthof` + `vectorize` +Mojo's unification move is stated in the manual, in three lines of +`comptime` aliases: + +```mojo +# mojolang.org/docs/manual/types — "Scalar values", Mojo 1.0.0b2, verbatim +comptime Scalar = SIMD[size=1] +comptime Int8 = Scalar[DType.int8] +comptime Float32 = Scalar[DType.float32] +``` -With width as a compile-time parameter, the machine's natural width -becomes a *queryable constant* and the boilerplate becomes generated -code: +`Scalar` is `SIMD` with `size` bound to 1 and `dtype` still open; +`Float32` closes `dtype` too. There is no separate scalar world. The +manual draws the conclusion itself: "whether you're working with a +single `Float32` value or a vector of float32 values, the math +operations go through exactly the same code path." + +Two consequences you can use tomorrow, in Rust, without Mojo. + +**No duplicate implementations to keep in sync.** Vectorizing an +algorithm means changing a parameter, not rewriting a body. Contrast +`experiments/src/dot.rs`, where the same reduction exists four times — +`dot_naive` (scalar, lines 10-17), `dot_unrolled8` (scalar, 8 named +accumulators, 22-37), `dot_wide` (portable, 46-49), `dot_neon` +(intrinsics, 62-65) — with four separate bodies that must agree to +within float rounding. The `#[cfg(target_arch = "aarch64")]` at line +61 is the fourth body's admission that it is not portable. + +**The width-1 instance is a free test oracle.** If the scalar version +is literally the `size=1` instantiation of the vector version, then +running the same body at 1 and at 4 and diffing is a *tautologically +correct* test — you are testing the compiler's monomorphization, not +your algorithm. In Rust you have to build that oracle by hand, and +`dot.rs` does: its test module (`dot.rs:67` onward) has a `rel_err` +helper comparing each rung against `dot_naive`. Note that the +comparison there is *approximate* (`rel_err`), because in Rust the two +bodies really are different code with different summation orders. In +Mojo the reduce-order difference is still real — `reduce_add()` on a +4-lane vector is not the same order as four scalar adds — so the +oracle is exact only for order-independent kernels. Name that +distinction before you steal the idea: it holds for `filter`, not for +`dot`. + +### Step 3 — `simd_width_of`: the machine's width as a queryable constant + +> **In:** a target machine and an element dtype. +> **Out:** the natural lane count, as a compile-time `Int` — computed +> here for both this host and an AVX-512 server. + +With width as a parameter, the machine's natural width becomes a +compile-time query rather than a number you hardcode: ```mojo -fn kernel[w: Int](v: SIMD[DType.float32, w]) -> SIMD[DType.float32, w] -vectorize[kernel, simdwidthof[DType.float32]()](n) +# mojolang.org/docs/std/sys/info/simd_width_of — Mojo 1.0.0b2, verbatim +def simd_width_of[dtype: DType, target: __mlir_type.`!kgen.target` = _current_target()]() -> Int +``` + +"Returns the vector size of the type on the host system." The +`target` parameter defaults to `_current_target()`, so it is a +*compile-time* property of what you are building for — and it can be +overridden, which is how you cross-compile without editing the +kernel. (There is a second overload taking a `type: RegisterPassable` +rather than a `DType`, on the same page.) + +Do the arithmetic, because the whole argument is in the numbers. The +value is register bits divided by element bits: + +``` + this host (Apple M5, aarch64, NEON, 128-bit vector registers): + simd_width_of[DType.float32]() = 128 / 32 = 4 + simd_width_of[DType.float64]() = 128 / 64 = 2 + simd_width_of[DType.int8]() = 128 / 8 = 16 + + an AVX-512 x86 server (512-bit vector registers): + simd_width_of[DType.float32]() = 512 / 32 = 16 + simd_width_of[DType.float64]() = 512 / 64 = 8 + simd_width_of[DType.int8]() = 512 / 8 = 64 + + ratio: the SAME source text yields 4x more lanes per instruction ``` -`simdwidthof` is a compile-time query of the TARGET (4 on NEON, 16 -on AVX-512); `vectorize` instantiates the kernel at that width plus -a scalar remainder at width 1 — the same body, monomorphized twice. -The polars contrast makes the value concrete: polars hardcodes -STRIPE=16 and writes the AVX-512 filter twice (u8 and u32) because -Rust's const generics can't cleanly abstract "the best width for -this type on this target." Question: what stops `wide`/std::simd -from doing this today? (The width is a const generic, but there's no -portable "native width" query, and no autogenerated remainder loop — -you write both by hand in dot.rs.) +Now compare what this repo does. `dot.rs:41-45` instructs you to use +`wide::f32x4` — the 4 is baked into the type name, so the byte +`4` in `f32x4` is this machine's `simd_width_of[DType.float32]()` +frozen at authoring time. On an AVX-512 box that same code still +issues 128-bit instructions and leaves three quarters of each register +idle. There is no portable, stable-Rust way to write "the native width +for `f32` on the target"; you write 4 and you write a comment. + +The four is not automatically wrong, mind. Step 6 shows a case where +polars deliberately picks a width *larger* than any register, and +`reading-simsimd.md` Step 1 shows why: lanes and chains are different +resources, and only one of them is what `simd_width_of` returns. + +### Step 4 — `vectorize`: the remainder loop, generated + +> **In:** a length that is not a multiple of the vector width, and a +> width-generic closure. +> **Out:** the exact iteration schedule Mojo emits — including a +> surprise about the tail — and the cost of that tail, computed. + +`simd_width_of` gives you a constant; `vectorize` turns it into a +loop. The signature, reduced to its load-bearing parts: -### Step 4 — the matmul arc: four separable layers +```mojo +# mojolang.org/docs/std/algorithm/backend/vectorize/vectorize — Mojo 1.0.0b2 +def vectorize[func: ..., //, simd_width: Int, /, *, unroll_factor: Int = 1](size: Int, closure: func) +``` -The famous progression (numbers from Modular's blog, M-series-class -hardware, GFLOPS order-of-magnitude): +The one-line description is the whole contract: it maps "a function +across a range from 0 to `size`, incrementing by `simd_width` at each +step. The remainder of `size % simd_width` will run in separate +iterations." + +The doc's own worked example is better than anything this guide could +invent, because it prints the schedule. It sets `comptime size = 10` +and `comptime simd_width = simd_width_of[DType.int32]()`, "assumed to +be 4 in this example", with a closure that prints its own width and +position. The documented output: ``` - Python baseline ~0.002 ×1 - naive Mojo (same loops) ~5 ×2000 compiled, typed - + vectorize inner loop ~25 SIMD lanes - + parallelize outer loop ~100 cores (topic 14, not 17) - + tile + unroll ~200+ cache blocking (topic 13) +storing 4 els at pos 0 +storing 4 els at pos 4 +storing 1 els at pos 8 +storing 1 els at pos 9 +[0, 0, 0, 0, 4, 4, 4, 4, 8, 9] ``` -Note the ORDER: types/compilation first, lanes second, cores third, -cache blocking last — and each step is a decorator/parameter, not a -rewrite. The lesson isn't "Mojo is fast"; it's that the four layers -(scalar semantics → lanes → threads → tiles) are SEPARABLE when -width is parametric. Our experiments walk the same rungs by hand. +Work through it: + +``` + size = 10, simd_width = 4 + full iterations : floor(10 / 4) = 2, covering elements 0..7 + remainder : 10 % 4 = 2, elements 8 and 9 -### Step 5 — what parametric width does NOT solve + what got emitted for the remainder: + NOT one width-2 iteration (2 is a legal SIMD size — a power of 2) + BUT two width-1 iterations ("1 els at pos 8", "1 els at pos 9") -The type system dissolves boilerplate, not microarchitecture: + trip count: 2 vector iterations + 2 scalar = 4 iterations for 10 elements + 50 % of the trip count handles 20 % of the data +``` -- compress/gather-shaped problems (simdjson's LUT trick) still need - per-ISA thought — a parametric width doesn't conjure vpcompress - on NEON; you still write the LUT or take the branchless store. -- ports × latency (the accumulator count — reading-simsimd.md) is - still yours: the matmul blog manually unrolls 4 accumulator - vectors, exactly like SimSIMD's 4-state API. No compiler infers - "12 chains" for you — reassociation of floats stays illegal in - every language. +That the tail is width-1 and not width-2 is the detail worth carrying +away. Mojo *could* emit a descending ladder of widths (4, 2, 1) and +finish the tail in one step, but each distinct width is another +monomorphization of the closure — more code, more compile time — and +width 1 is the one instance Step 2 guarantees already exists for free. +The unification of scalars and vectors is what makes the cheap tail +strategy also the *only* tail strategy you need. + +`unroll_factor` is the other knob. The doc shows `unroll_factor=2` +producing, in pseudocode, `closure[4](0); closure[4](4);` straight-line +instead of a loop, and warns that "the remainder loop won't unroll +unless `size` is passed as a parameter". This is the same lever as +`dot.rs`'s hand-written four accumulator vectors, with one crucial +difference covered in Step 7: unrolling a loop is not the same as +creating independent accumulator chains, and only the second one buys +you `reading-simsimd.md`'s 12. + +Now price the tail on this repo's actual workload: -The dividing line: anything expressible as "same op, any width" the -language absorbs; anything about *which* instructions and *how many -independent chains* stays engineering. +``` + our benches: N = 4M f32 = 4,194,304 elements (notes.md), width 4 + 4,194,304 % 4 = 0 -> the remainder loop never runs -### Step 6 — the translation table for our stack + an odd length, e.g. a 1,537-dim embedding at width 4: + 1537 % 4 = 1 -> 1 scalar iteration out of 385 = 0.26 % of trips +``` -Every Mojo construct has a hand-written stable-Rust equivalent — -this table is the map from the ideal to what M17 actually ships: +Which is exactly why remainders are the classic source of SIMD bugs: +they cost nothing and they run almost never, so they are the code path +your benchmark never touches and your fuzzer finds first. Two ways out +appear in this topic: generate the tail (Mojo), or make the tail +impossible. simdjson takes the second route — see +`reading-simdjson.md` on its padded input buffer, which lets every +kernel read a full vector past the logical end without a tail at all. -| Mojo | stable Rust (our experiments) | +### Step 5 — the four separable layers + +> **In:** a slow loop and the four independent things you could do to +> it. +> **Out:** the order they must be applied in, and which of them this +> repo has actually measured. + +The structural claim behind Mojo's design is that four optimizations +are *separable* — each a parameter or a decorator rather than a +rewrite — and that they stack in a fixed order: + +``` + layer what changes which topic + 1 scalar semantics types + compilation (the baseline) + 2 lanes + chains SIMD width, accumulator count 17 (this one) + 3 threads cores 14 + 4 tiles cache blocking 13 +``` + +Types first, lanes second, cores third, cache blocking last. The +ordering is not arbitrary: you cannot vectorize a loop whose element +type is decided at runtime, you cannot usefully parallelize a loop +that is already memory-bound, and tiling only pays once the inner +kernel is fast enough that the cache is the constraint. + +The earlier version of this guide illustrated the ladder with a +GFLOPS progression from Modular's matmul blog post. **That source is +gone** — the notebook and blog URLs all return 404 as of this +revision, and the brief's rule is that an unverifiable number does not +go in. So the ladder above carries no borrowed numbers. What it +carries instead is this repo's own measurement of layer 2, which you +can rerun: + +``` + layer 2, measured here (notes.md, dot lane, N = 4M f32, + release, Apple Silicon, measured 2026-07-10): + dot_naive (1 accumulator chain) 10.89 GB/s + dot_unrolled8 (8 accumulator chains) 42.12 GB/s + ratio 42.12 / 10.89 = 3.9x + + the same lane in FINDINGS.md row 17, a different run: + 8.88 -> 26.32 GB/s = 3.0x + + layers 3 and 4 are NOT measured in topic 17 — they are topics + 14 and 13, and this topic's framing is deliberately "the last + 10x on a single core". +``` + +Cite whichever run you use by name; never average them. And note what +the 3.9× is *not*: it is not a lane-width effect. `dot_unrolled8` +(`dot.rs:22-37`) has eight `f32` accumulators, no vector types +anywhere, and no intrinsics — it is FINDINGS.md's "eight accumulators +and no intrinsics". Layer 2 is two things wearing one name, and this +repo's headline measures the chain half of it. Step 7 returns to why +the compiler could not have done that for you. + +### Step 6 — where the type system stops caring: the 2× register rule + +> **In:** Mojo's documented advice about over-wide vectors, and polars' +> deliberate violation of it. +> **Out:** the resolution — which is that the advice is right for maps +> and wrong for reduces, with the register counts to prove it. + +The `SIMD` page carries an explicit caution, verbatim: + +> **Caution:** If you declare a SIMD vector size larger than the +> vector registers of the target hardware, the compiler will break up +> the SIMD into multiple vector registers for compatibility. However, +> you should avoid using a vector that's more than 2x the hardware's +> vector register size because the resulting code will perform poorly. + +Now hold that against `reading-polars-compute.md` Step 2, where +`float_sum.rs:13` sets `const STRIPE: usize = 16;` and the reduce runs +on `Simd` (`float_sum.rs:82-83`): + +``` + NEON vector register (this host) = 128 bits + Mojo's ceiling: 2x the register = 256 bits + at that ceiling: f32 <= 8 lanes, f64 <= 4 lanes + + polars ships: + Simd = 16 x 32 = 512 bits = 4x the register + Simd = 16 x 64 = 1024 bits = 8x the register + + polars exceeds Mojo's stated ceiling by 2x (f32) and 4x (f64) +``` + +Both documents are right, about different kernels, and the difference +is what the extra registers are *for*. + +Mojo's caution is about **element-wise maps**: `c = a * b + 1.0` over +a `SIMD[DType.float32, 32]`. Splitting that into 8 physical registers +buys nothing — the 8 multiply-adds were already independent, the +hardware would have pipelined 4-lane versions of them just as well, +and you have spent 8 of ~32 architectural vector registers to say so. +The compiler-generated spills are the "perform poorly". + +polars' 16 is about **reduces**, where the extra registers *are* the +point. A reduction is a dependency chain, and +`reading-simsimd.md` Step 1 gives the rule: you need +`latency × ports` independent chains, which on this host is 3 × 4 = 12 +for f32 FMA (`include/numkong/dot/neon.h:14`, M5 column). Declaring an +over-wide type is a portable way to spell "give me N chains" in a +language that has no direct way to ask: + +``` + Simd on NEON = 4 physical float32x4_t + -> 4 independent add chains (against the 12 the core wants) + Simd on NEON = 8 physical float64x2_t + -> 8 independent add chains (against 16 for f64 FMA, 4cy x 4p) + + so even polars' "excessive" 4x reaches only 4/12 = 33 % of the + chains this core can keep in flight +``` + +The resolution: Mojo's rule is right when the operation is a map and +wrong when it is a reduce, and neither document says which it means. +This is a real limit of parametric width as a *language* feature — +`size` conflates "how much data per instruction" with "how many +independent chains", and the compiler cannot tell which one you wanted +from the type alone. Measure before you follow either. + +### Step 7 — what parametric width does NOT solve + +> **In:** the two hardest things in this topic — data-dependent +> compaction, and accumulator count. +> **Out:** why neither is a width problem, so no type system dissolves +> them. + +The type system dissolves boilerplate, not microarchitecture. Two +things stay yours. + +**Compress- and gather-shaped problems still need per-ISA thought.** +AVX-512 has `vpcompressd`, which packs the selected lanes of a vector +to the front in one instruction; NEON has no such instruction. A +parametric width cannot conjure one. On this host you write the +16-entry shuffle-mask lookup table instead — `notes.md`'s +implementation log records exactly that (`filter.rs: count_neon + +compact_neon (LUT built, all 16 masks pass)`), and +`reading-simdjson.md` walks the same trick in simdjson's own +`arm64/simd.h`. A hypothetical Mojo version of `compact_neon` would +still contain a table, because the problem is that the instruction +does not exist, not that the width was hard to spell. + +**Accumulator count stays an engineering decision.** Float addition is +not associative, so reassociating a reduction changes the answer, so +no compiler in any language may split your one accumulator into twelve +without permission. This is why `dot_naive` (`dot.rs:10-17`) stays at +10.89 GB/s: its doc comment at lines 8-9 says so — "LLVM cannot +vectorize this without `-ffast-math` (float reassociation changes the +answer)". The permission has to be in the source, and giving it is +what `dot_unrolled8` does with eight named accumulators. The +instruction is the same either way; the *chain count* changed. + +Note that Step 4's `unroll_factor` does not solve this either. +Unrolling replicates the loop body; if every replica still accumulates +into the same variable, you have one chain and a bigger loop body. +`dot.rs:41` therefore says "FOUR independent accumulator vectors", not +"unroll by four" — and even that is 4 chains against a machine that +wants 12, which is the first thing to try changing when your rung 3 +number disappoints. + +The dividing line: anything expressible as "same operation, any width" +the language absorbs; anything about *which* instructions exist and +*how many independent chains* you keep in flight stays engineering. + +### Step 8 — the translation table for our stack + +> **In:** each Mojo construct above. +> **Out:** the hand-written stable-Rust line in `experiments/` that +> stands in for it, with the anchor. + +Every Mojo construct has a stable-Rust equivalent somewhere in this +topic's crate. This table is the map from the ideal to what M17 +actually ships: + +| Mojo 1.0.0b2 | stable Rust in `topics/17-simd/experiments/` | |---|---| -| `SIMD[DType.float32, 4]` | `wide::f32x4` | -| `simdwidthof[T]()` | hardcode 4 on NEON (128-bit / 32) | -| `vectorize[kernel, w](n)` | hand-written chunks_exact(w) + remainder | -| `@unroll` / accumulator params | 4 named accumulator variables | -| width-1 fallback | your scalar fn, kept for dispatch | +| `SIMD[DType.float32, 4]` | `wide::f32x4` (`dot.rs:41`) | +| `Scalar[DType.float32]` = `SIMD[size=1]` | a separate `fn`: `dot_naive` (`dot.rs:10-17`), kept for the tail and for dispatch | +| `simd_width_of[DType.float32]()` | the literal `4` inside the type name `f32x4`; 128-bit NEON ÷ 32 | +| `vectorize[simd_width](size, closure)` | `chunks_exact(16)` plus a hand-written scalar remainder (`dot.rs:42-45`) | +| `unroll_factor = 2` | writing the accumulators out by hand (`dot.rs:41`, `dot_unrolled8` at `dot.rs:22-37`) | +| `reduce_add()` | `f32x4::reduce_add` (`dot.rs:45`) / `vaddvq_f32` (`dot.rs:56`) | +| target-conditional codegen | `#[cfg(target_arch = "aarch64")]` (`dot.rs:61`) | + +Read the table in both directions. Left to right it says how much +boilerplate the language absorbs. Right to left it says something less +flattering to Mojo: every one of those Rust lines is *visible*, so you +can see the width, the chunk size, the accumulator count and the +reduce instruction at a glance — and this topic's whole argument is +that those four numbers are the performance. The generated version is +easier to write and harder to audit, which is the trade every +abstraction in this course makes. ## How to read the docs (with the concepts in hand) -1. **The `SIMD` stdlib page** — skim the type's surface with Step 2 - in mind; the interesting part is what's a *parameter* (dtype, - width) versus what's a method. -2. **The matmul blog arc** — read in full against Step 4's table; - at each rung, name which topic of this course (11, 13, 14, 17) - the speedup belongs to. -3. Then go back to `dot.rs` and identify each hand-written line that - `vectorize` would have generated — that's Step 6 made concrete. +Everything below is a web page, not a clone — there is nothing to +check out and nothing in the pin table (`resources/codebases.md`) for +Mojo, which is exactly why the version caveat at the top of this guide +matters. Append `.md` to any URL to get clean Markdown, and start from +`https://mojolang.org/llms.txt` or `sitemap.xml` if a link has moved +again. + +1. **`/docs/std/builtin/simd/SIMD`** — skim the type's surface with + Step 2 in mind. The interesting part is what is a *parameter* + (`dtype`, `size`) versus what is a method (`reduce_add`, `select`, + `shuffle`, `cast`). Read the **Caution** block against Step 6 and + decide, for your own kernel, whether it is a map or a reduce. +2. **`/docs/manual/types` § "Scalar values"** — three lines of + `comptime` aliases and one sentence of consequence. This is Step 2 + in full; it is shorter than this paragraph. +3. **`/docs/std/algorithm/backend/vectorize/vectorize`** — read the + worked example and predict its printed output *before* looking, per + Step 4. If you predicted one width-2 tail iteration, reread Step 2 + and work out why width-1 is cheaper for the compiler. +4. **`/docs/std/sys/info/simd_width_of`** — one signature; note the + `target` parameter and what it means for cross-compilation. +5. Then go back to `experiments/src/dot.rs` and mark every line that + `vectorize` would have generated. That is Step 8 made concrete, and + it is the point of reading a language you are not going to use. ## Questions for notes.md -1. `Float32 = SIMD[f32,1]`: what does making scalars width-1 vectors - buy for TESTING kernels (hint: run the same body at w=1 as the - oracle for w=4 — steal this for dot.rs's tests)? -2. `vectorize` generates the remainder loop; where in our filter.rs - is the equivalent, and why is the remainder the classic source - of SIMD bugs (simdjson pads its input to 64 instead — compare)? -3. Rust's std::simd `Simd` has the parametric type but sits - on nightly for years. What's actually hard: the type, the - portable ops (compress!), or stabilizing the ISA mapping? -4. The matmul arc gains more from tiling than from SIMD. Reconcile - with this topic's "last 10× on a single core" framing — when is - topic 13 the bigger lever than topic 17? -5. For M17: our engine will hardcode NEON width 4. Write the one +1. `comptime Float32 = Scalar[DType.float32]` = `SIMD[DType.float32, 1]`: + what does making scalars width-1 vectors buy for TESTING kernels? + Then state the limit Step 2 names — for which of `dot` and `filter` + is the width-1 instance an *exact* oracle rather than an + approximate one, and why? +2. `vectorize` generates the remainder loop and runs `size % + simd_width` as that many **width-1** iterations. Where in + `filter.rs` is the equivalent code, what is the remainder for this + repo's N = 4M at width 4, and why is a path that runs 0 % of the + time in your benchmarks the classic source of SIMD bugs? Compare + simdjson's answer (pad the input so there is no tail). +3. Rust's `std::simd` has had the parametric type `Simd` for + years and is still nightly-only. Which piece is actually hard: the + type, the portable operations (`compress`!), or stabilizing the + ISA mapping? Use Step 7's NEON-has-no-`vpcompressd` example to + argue one side. +4. Step 6's contradiction: Mojo says never exceed 2× the register + width; polars ships 4× for `f32` and 8× for `f64`. Compute the + physical register count and the chain count for both, decide which + rule applies to `dot_wide` (`dot.rs:46`), and pick a `STRIPE` for + your own version. Then measure and see if you were right. +5. Step 5's ladder gains more from tiling (topic 13) and threads + (topic 14) than from lanes. Reconcile that with this topic's "last + 10× on a single core" framing: when is topic 13 the bigger lever + than topic 17, and what does `notes.md`'s 42 GB/s ceiling tell you + about which one you are up against at N = 4M? +6. For M17: our engine will hardcode NEON width 4. Write the one sentence justifying that (deployment target) and the one-line - escape hatch if SVE servers arrive. + escape hatch if SVE servers arrive — say which of Step 8's table + rows would have to change, and which would not. ## Done when -- [ ] You can explain what making scalars width-1 vectors buys in the type system. -- [ ] You can say what `simdwidthof` + `vectorize` generate that you currently write by hand, including the remainder loop. -- [ ] You can separate the four layers of the matmul arc and say which one contributes most (it is not SIMD). -- [ ] You can name what parametric width does *not* solve. -- [ ] You wrote answers to all five questions in notes.md, including the one-line justification for hardcoding NEON width 4 in M17. +Answer each before unfolding it. + +- [ ] You can state the actual 1.0.0b2 signature of `SIMD` and say which of its two parameters is not a type. + +
Answer + + `struct SIMD[dtype: DType, size: Int]`. Neither parameter is a type + in the usual sense: `dtype` is a **value** of the `DType` struct + (`DType.float32` is an identifier, not a type — the manual is + explicit that "you can't create a variable with the type + `DType.float64`"), and `size` is an `Int`. Both are compile-time + parameters, and `size` "must be positive and a power of 2". Note the + spelling: `size`, not `width`, despite this page's own title. + +
+ +- [ ] You can explain what making scalars width-1 vectors buys in the type system, and where the free test oracle stops being free. + +
Answer + + `comptime Scalar = SIMD[size=1]`, so `Float32` is + `SIMD[DType.float32, 1]` and, per the manual, "the math operations + go through exactly the same code path". Two wins: no duplicate + scalar/vector bodies to keep in sync (contrast `dot.rs`'s four + separate rungs), and the width-1 instantiation is a test oracle for + the width-4 one. The limit: it is an *exact* oracle only for + order-independent kernels. `filter` qualifies; `dot` does not, + because `reduce_add()` over 4 lanes sums in a different order than + four scalar adds and float addition is not associative — which is + why `dot.rs`'s tests use a relative-error helper rather than + equality. + +
+ +- [ ] You can compute `simd_width_of` for three dtypes on this host and on an AVX-512 server. + +
Answer + + Register bits ÷ element bits. On this host (Apple M5, NEON, + 128-bit): f32 → 128/32 = **4**, f64 → 128/64 = **2**, i8 → + 128/8 = **16**. On a 512-bit AVX-512 machine: f32 → **16**, f64 → + **8**, i8 → **64**. The same source text, 4× the lanes. In our Rust + the value is frozen into the type name `f32x4` (`dot.rs:41`), so on + an AVX-512 box that code leaves three quarters of every register + idle. `simd_width_of` also takes an explicit `target` parameter, + defaulting to `_current_target()`, so the query works for + cross-compilation too. + +
+ +- [ ] You can say what `vectorize` generates that you write by hand, and predict its remainder schedule exactly. + +
Answer + + It emits `floor(size / simd_width)` full-width iterations plus + `size % simd_width` iterations **at width 1** — not one narrower + vector iteration. The doc's example (size 10, width 4) prints + "storing 4 els at pos 0 / 4 els at pos 4 / 1 els at pos 8 / 1 els + at pos 9" and ends with `[0, 0, 0, 0, 4, 4, 4, 4, 8, 9]`. Width-1 + is chosen because Step 2 already guarantees that instantiation + exists, so no extra monomorphization is needed. In Rust you write + the `chunks_exact` loop and the scalar tail yourself + (`dot.rs:42-45`). For this repo's N = 4M at width 4 the remainder is + 4,194,304 % 4 = **0**, so the tail never executes in any benchmark + here — which is precisely why it is where bugs hide. + +
+ +- [ ] You can separate the four layers and say which one this repo has actually measured. + +
Answer + + Scalar semantics/compilation → lanes and chains (topic 17) → + threads (topic 14) → tiles and cache blocking (topic 13), in that + order, because each layer needs the one before it to be settled. + This repo has measured **only layer 2**: `notes.md`'s dot lane goes + 10.89 → 42.12 GB/s (3.9×) and `FINDINGS.md` row 17 records + 8.88 → 26.32 GB/s (3.0×) from a different run. Both come from + `dot_unrolled8`, which has eight scalar `f32` accumulators and no + intrinsics — so the headline is the *chain* half of layer 2, not the + lane half. The GFLOPS ladder that used to sit here was taken from a + Modular blog post that now 404s, and has been removed rather than + quoted from memory. + +
+ +- [ ] You can resolve the 2× rule against polars' 4×, with register counts. + +
Answer + + Mojo's `SIMD` page cautions against "a vector that's more than 2x + the hardware's vector register size because the resulting code will + perform poorly" — on 128-bit NEON that caps f32 at 8 lanes. polars + ships `Simd` (512 bits, 4×) and `Simd` (1024 bits, + 8×) from `float_sum.rs:13`. Both are right, for different kernels. + For an element-wise **map** the split into 4 or 8 physical registers + buys nothing and costs register pressure — Mojo's case. For a + **reduce** the physical registers *are* independent dependency + chains, and the core wants latency × ports = 3 × 4 = 12 of them for + f32 FMA (`dot/neon.h:14`) — polars' case, and its 4 chains are still + only a third of what the machine would take. `size` conflates + "data per instruction" with "chains in flight", and no type system + yet distinguishes them. + +
+ +- [ ] You can name what parametric width does *not* solve, with a concrete example of each. + +
Answer + + **Missing instructions.** AVX-512's `vpcompressd` has no NEON + equivalent, so data-dependent compaction needs a 16-entry + shuffle-mask table on this host regardless of language — + `notes.md`'s log records `compact_neon` with all 16 masks passing, + and simdjson does the same thing in `arm64/simd.h`. **Chain count.** + Float addition is not associative, so no compiler may turn one + accumulator into eight; `dot.rs:8-9` says LLVM cannot vectorize + `dot_naive` "without `-ffast-math`". The permission must be written + in the source, which is what `dot_unrolled8` does. `unroll_factor` + does not help either: replicating a body that accumulates into one + variable gives you one chain and a longer body. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the one-line justification for hardcoding NEON width 4 in M17. + +
Answer + + The justification is a deployment-target claim, not a performance + one: M17 ships to aarch64 hosts with 128-bit NEON and no SVE + exposed, so `simd_width_of[DType.float32]()` would return 4 on every + machine we run on, and a hardcoded 4 is that query constant-folded. + The escape hatch is the last row of Step 8's table — the + `#[cfg(target_arch = ...)]` boundary — behind which a second kernel + can appear without touching the callers; what would *not* change is + the accumulator count, because Step 7 shows that number is set by + latency × ports rather than by register width. + +
## References -**Papers & docs** -- Modular — Mojo stdlib docs for `SIMD` - ([docs.modular.com](https://docs.modular.com/mojo/stdlib/builtin/simd/)) - — the parametric type itself; no clone needed -- Modular — the "Matrix Multiplication in Mojo" blog arc - (matmul.mojo) — the ×2000-to-tiled progression walked in §4 +**Docs (Mojo 1.0.0b2 — check the version before trusting any name)** +- `https://mojolang.org/docs/std/builtin/simd/SIMD` — the + `struct SIMD[dtype: DType, size: Int]` declaration, the + power-of-two constraint, the reduce/select/shuffle API surface, and + the "avoid more than 2x the hardware's vector register size" + caution quoted in Step 6. +- `https://mojolang.org/docs/manual/types` § "Scalar values" — the + three `comptime` aliases of Step 2 and the "exactly the same code + path" sentence. +- `https://mojolang.org/docs/std/sys/info/simd_width_of` — Step 3's + signature. Note the rename: this was `simdwidthof` in earlier + releases, and the sibling queries are `simd_bit_width` and + `simd_byte_width`. +- `https://mojolang.org/docs/std/algorithm/backend/vectorize/vectorize` + — Step 4's signature, worked example and printed schedule; + `parallelize` lives one directory over + (`/docs/std/algorithm/backend/cpu/parallelize/`) and is topic 14's + business. +- Index pages: `https://mojolang.org/llms.txt` and + `https://mojolang.org/sitemap.xml`. Appending `.md` to any doc URL + returns Markdown. +- **Removed:** the earlier version of this guide illustrated Step 5 + with a GFLOPS ladder (Python → naive Mojo → vectorized → + parallelized → tiled) from Modular's "Matrix Multiplication in Mojo" + notebook. That page and its blog mirrors return 404 at the time of + writing, so the figures could not be checked against a source and + have been dropped rather than repeated. If it reappears, the numbers + belong here with their hardware. + +**This repo** +- `topics/17-simd/experiments/src/dot.rs` — the four rungs Step 8 + maps onto: `dot_naive` (10-17), `dot_unrolled8` (22-37), `dot_wide` + (39-49), `dot_neon` (51-65). +- `topics/17-simd/notes.md` and `FINDINGS.md` row 17 — the layer-2 + measurements of Step 5, from two different runs of the same lane. +- `reading-simsimd.md` — the latency × ports rule Step 6 and Step 7 + lean on, and the port/latency table it is derived from. +- `reading-polars-compute.md` Step 2 — `STRIPE = 16`, the deliberate + violation of Mojo's 2× advice. +- `reading-simdjson.md` — padding instead of a remainder loop + (Step 4), and the shuffle-table approach to compaction (Step 7). diff --git a/topics/17-simd/reading-polars-compute.md b/topics/17-simd/reading-polars-compute.md index 068cb7a..6d03331 100644 --- a/topics/17-simd/reading-polars-compute.md +++ b/topics/17-simd/reading-polars-compute.md @@ -1,180 +1,653 @@ # polars-compute: shipping SIMD in stable Rust The production-Rust answer to "how do I ship SIMD without a nightly -compiler or per-CPU binaries": autovec-friendly scalar bodies, -explicit `std::simd` where it pays, raw intrinsics only for the one -instruction Rust can't reach (vpcompress). Before the anchors, this -chapter builds the two kernels every engine needs — the reduction -and the filter — concept by concept, as polars actually ships them. +compiler or per-CPU binaries": autovectorisation-friendly scalar +bodies, `std::simd` where it pays, and raw intrinsics only for the one +instruction no portable abstraction can express. Before the anchors, +this chapter builds the two kernels every engine needs — the reduction +and the filter — as polars actually ships them, and then works out +what your machine gets from each. + +Every anchor below is `pola-rs/polars@f8bcc3d` (`resources/codebases.md`), +quoted with the line numbers the code occupies in that revision. Read +it with one fact in front of you: **on aarch64 the entire AVX-512 half +of the filter is `#[cfg]`-ed out of existence**, so the code your Mac +runs is the scalar path — which turns out to be the more interesting +one anyway. ## The problem in one sentence -`xs.iter().sum::()` on Apple Silicon leaves ~15/16 of the FPU -idle — one accumulator is one serial dependency chain — and the -naive filter loop mispredicts a branch on every unpredictable -element; polars-compute fixes both in stable Rust with two files you -can read in an afternoon. +`xs.iter().sum::()` leaves most of the FPU idle because one +accumulator is one serial dependency chain, and the naive filter loop +mispredicts a branch on every unpredictable element; polars-compute +fixes both in stable Rust, in two files you can read in an afternoon. ## The concepts, step by step ### Step 1 — the reduction problem: one accumulator, one chain -A reduction (folding n values into one, like a sum) has a hidden -serial bottleneck: `acc += x[i]` can't start until the previous -`acc += x[i-1]` finished, because each add needs the last add's -result. That's a **dependency chain** — with ~3-cycle add/FMA -latency and 4 vector ports on M-series, a single chain uses 1/12th -of the machine (README §1). And the compiler can't fix it: float -addition isn't associative (reordering changes rounding), so LLVM -won't reassociate `a+b+c+d` into `(a+b)+(c+d)` without -`-ffast-math`. Fast float sums must *explicitly* restructure the -order of additions. - -### Step 2 — STRIPE=16: many accumulators, reduced once at the end - -polars' `float_sum.rs` sums blocks of 128 elements as 16 parallel -lanes (`STRIPE = 16`, float_sum.rs:13): lane j accumulates elements -j, j+16, j+32, … — sixteen independent chains instead of one, kept -in `Simd` via `chunks_exact` (float_sum.rs:67-90). Lanes are -combined into a single scalar only at block end -(`vector_horizontal_sum`, float_sum.rs:44) — a horizontal reduce is -slow, so you do it once per 128 elements, not once per element. -STRIPE=16 for f32 is 512 bits = 4 NEON registers: *wider than the -vector* on purpose, because the accumulator count is set by -latency × ports, not by vector width. +> **In:** `n` floats and a loop that does `acc += x[i]`. +> **Out:** a serial dependency chain of length `n`, whose length — +> not the machine's throughput — sets the runtime. + +`acc += x[i]` cannot start until `acc += x[i-1]` has finished, because +the second add consumes the first add's result. That is a **dependency +chain**, and its cost is `n × latency`, independent of how many adders +the machine has. + +Put this topic's own numbers on it. `notes.md` (provided rungs, +release, Apple Silicon, 2026-07-10, N = 4M f32, 20 reps): + +| dot rung | GB/s | ms | vs naive | +|---|---|---|---| +| naive, 1 chain | 10.89 | 3.081 | 1.0× | +| unrolled-8, autovectorised | 42.12 | 0.797 | 3.9× | + +3.9× from zero intrinsics — just writing eight accumulators so LLVM is +permitted to reassociate. (`FINDINGS.md` row 17 records 8.88 → 26.32 +GB/s from a different run of the same bench; cite whichever run you are +quoting, and do not average them.) + +The compiler cannot do this for you unasked. Float addition is not +associative — `(a+b)+c` and `a+(b+c)` round differently — so LLVM will +not turn one chain into four without `-ffast-math`. Restructuring the +order of additions has to be written down. polars writes it down, and +its comment says exactly why: + +```rust +// polars crates/polars-compute/src/float_sum.rs:44-63 — the final reduce + 44 fn vector_horizontal_sum(mut v: V) -> T +// ... 45-48: bounds ... + 49 // We have to be careful about this reduction, floating + 50 // point math is NOT associative so we have to write this + 51 // in a form that maps to good shuffle instructions. + 52 // We fold the vector onto itself, halved, until we are down to + 53 // four elements which we add in a shuffle-friendly way. + 54 let mut width = STRIPE; + 55 while width > 4 { + 56 for j in 0..width / 2 { + 57 v[j] = v[j] + v[width / 2 + j]; + 58 } + 59 width /= 2; + 60 } + 62 (v[0] + v[2]) + (v[1] + v[3]) + 63 } +``` + +Line 62 is the tell: the last four elements are added in a fixed +*shuffle-friendly* pairing, not left to right. The summation order is +part of the API. + +### Step 2 — STRIPE = 16: many accumulators, reduced once at the end + +> **In:** a block of 128 elements. +> **Out:** 16 lanes of partial sums held across the whole block, folded +> to one scalar exactly once. + +```rust +// polars crates/polars-compute/src/float_sum.rs:13-14 — the two constants + 13 const STRIPE: usize = 16; + 14 const PAIRWISE_RECURSION_LIMIT: usize = 128; +``` + +```rust +// polars crates/polars-compute/src/float_sum.rs:79-85 — the SIMD block sum + 79 fn sum_block_vectorized(&self) -> F { + 80 let vsum = self + 81 .chunks_exact(STRIPE) + 82 .map(|a| Simd::::from_slice(a).cast_generic::()) + 83 .sum::>(); + 84 vector_horizontal_sum(vsum) + 85 } +``` + +Line 81 cuts the 128-element block into 8 chunks of `STRIPE = 16`; line +83 sums them *as vectors*, so lane *j* accumulates elements +j, j+16, j+32, … — 16 independent chains of length 8 instead of one +chain of 128. Line 84 collapses them, once per 128 elements. + +Now size it on real registers, because "16 accumulators" is not what +the hardware sees: + +``` + STRIPE = 16 lanes. Simd occupies 16 * size_of::() * 8 bits. + + f32 : 16 * 32 = 512 bits + NEON (128-bit regs): 512 / 128 = 4 physical registers + -> 4 independent add chains per lane group + AVX-512 (512-bit regs): 512 / 512 = 1 physical register + + f64 : 16 * 64 = 1024 bits + NEON: 1024 / 128 = 8 physical registers -> 8 chains + + how many chains do you need? chains = latency x issue ports + FMLA on an Apple M-class P-core: 3 cycles, 4 pipes (SimSIMD's own + table, include/numkong/dot/neon.h:13-24) -> 12 +``` + +So `Simd` on NEON is 4 chains against a model that wants ~12, +and `Simd` is 8. Deliberately **wider than the vector +register** — the accumulator count is set by latency × ports, not by +register width. Hold onto that number: Mojo's own documentation +(`reading-mojo-simd.md`) advises never exceeding 2× the hardware +register size, and polars ships 4× for `f32` and 8× for `f64`. Both +positions are defensible and they cannot both be right for your +kernel; measure. + +The same file contains this topic's headline trick with no SIMD at all: + +```rust +// polars crates/polars-compute/src/float_sum.rs:155-169 — the non-SIMD build + 155 #[cfg(not(feature = "simd"))] + 156 impl SumBlock for [T; PAIRWISE_RECURSION_LIMIT] +// ... 157-160: bounds ... + 161 fn sum_block_vectorized(&self) -> F { + 162 let mut vsum = [F::default(); STRIPE]; + 163 for chunk in self.chunks_exact(STRIPE) { + 164 for j in 0..STRIPE { + 165 vsum[j] = vsum[j] + chunk[j].as_(); + 166 } + 167 } + 168 vector_horizontal_sum(vsum) + 169 } +``` + +A plain `[F; 16]` array, indexed lane-wise, with a compile-time trip +count and no intrinsic anywhere — and it is still called +`sum_block_vectorized`, because that is exactly the shape LLVM +autovectorises. This is `notes.md`'s "3.9× from ZERO intrinsics" as a +library ships it: the win came from writing 16 accumulators, not from +naming a register. + +One caution before you quote the 3.9× as a compute result: at N = 4M +each input is 16 MB, well out of L2, and 42.12 GB/s may be a bandwidth +ceiling rather than an ALU one. `notes.md`'s own prediction worksheet +asks precisely this ("rerun at N=64K in-cache to see compute limit") — +run it before you decide. ### Step 3 — pairwise recursion: the same tree fixes accuracy -Naive left-to-right float summation accumulates O(n) rounding error -— each add rounds, and errors compound linearly. Pairwise summation -(add halves recursively, `PAIRWISE_RECURSION_LIMIT = 128`) gets -O(log n) error — and it's the same tree shape the SIMD blocking -already built. One design, both wins: below 128 elements, the -striped block; above, recursive halving. Question: why is the -null-masked variant (`_with_mask`) just `select(mask, x, 0)` + the -same sum, and what does that say about null handling in vectorized -engines generally (topic 11's validity-mask philosophy)? - -### Step 4 — the filter problem, and the bit-iteration fallback - -A filter (keep elements where a boolean mask is set, packed densely -into the output) is the other universal kernel. The naive -`if keep { out.push(x) }` mispredicts at mid selectivity (the -fraction of elements kept — at 50%, the branch is a coin flip, ~15 -cycles per miss). polars' scalar fallback (filter/scalar.rs:12) -sidesteps prediction entirely: process 64 elements per iteration -using their mask *word*, and iterate only the SET bits with -`trailing_zeros` (index of the lowest 1-bit): +> **In:** an array whose length is a multiple of 128. +> **Out:** a sum with `O(log n)` rounding error instead of `O(n)`, +> using the tree the SIMD blocking already built. ```rust -// one 64-element block per mask word; cost ∝ popcount, not 64 -fn filter_block(vals: &[T; 64], mut m: u64, out: &mut Vec) { - while m > 0 { - let i = m.trailing_zeros() as usize; // next surviving element - out.push(vals[i]); - m &= m - 1; // clear lowest set bit - } -} +// polars crates/polars-compute/src/float_sum.rs:196-209 — the recursion + 196 let block: Option<&[T; PAIRWISE_RECURSION_LIMIT]> = f.try_into().ok(); + 197 if let Some(block) = block { + 198 return block.sum_block_vectorized(); + 199 } +// ... 201-204: the safety argument for the split ... + 206 let blocks = f.len() / PAIRWISE_RECURSION_LIMIT; + 207 let left_len = (blocks / 2) * PAIRWISE_RECURSION_LIMIT; + 208 let (left, right) = (f.get_unchecked(..left_len), f.get_unchecked(left_len..)); + 209 pairwise_sum(left) + pairwise_sum(right) ``` -Cost is proportional to survivors (popcount — the number of set -bits), not to 64: selectivity-adaptive for free, and the loop branch -(`while m > 0`) is highly predictable. This is simdjson's -flatten-bits idiom, value edition. +Below 128 elements, the striped block of Step 2; above it, recursive +halving. One design, two wins. Work the error bound for a realistic +column, using ε for the unit roundoff and counting the longest chain of +dependent additions any single value passes through: -### Step 5 — the compress instruction: the one thing that needs intrinsics +``` + n = 1e8, PAIRWISE_RECURSION_LIMIT = 128, STRIPE = 16 -AVX-512 (x86's 512-bit SIMD extension) has **compress-store** -instructions that do the entire filter in hardware: take a vector -and a mask, write only the selected lanes, packed left. polars wraps -them behind one macro, `simd_filter!` (avx512.rs:7), which fixes the -loop skeleton — load 64 mask bits, loop vectors of the value type, -compress-store, advance the output pointer by popcount: + naive left-to-right worst case ~ n * eps = 1e8 * eps + polars, longest path for one value: + inside its lane, within a block : 128/16 = 8 adds + the horizontal fold : log2(16) = 4 adds + across blocks : log2(1e8/128) + = log2(781250) = 19.6 -> 20 adds + total ~ 32 * eps + + ratio: 1e8 / 32 = 3.1e6 times tighter +``` + +The masked variant is the same tree with a `select`: + +```rust +// polars crates/polars-compute/src/float_sum.rs:87-97 — nulls, without a branch + 87 fn sum_block_vectorized_with_mask(&self, mask: BitMask<'_>) -> F { + 88 let zero = Simd::default(); +// ... 89-92: the same chunks_exact(STRIPE) pipeline, enumerated ... + 93 let m: Mask = mask.get_simd(i * STRIPE); + 94 m.select(Simd::from_slice(a).cast_generic::(), zero) +// ... 96-97: same sum, same horizontal fold ... ``` - u8 → vbmi2 _mm512_maskz_compress_epi8 (needs Ice Lake+) - u32 → avx512f _mm512_maskz_compress_epi32 - scalar fallback → while m > 0 { tz = m.trailing_zeros(); ... } + +Line 94 turns "is this value null?" into "add zero", which is topic +11's validity-mask philosophy in one line: nulls never become control +flow. The non-SIMD sibling at line 175 says so out loud — +"Unconditional add with select for better branch-free opts." + +### Step 4 — the filter: two scalar kernels and a selectivity threshold + +> **In:** 64 values and their 64-bit mask word. +> **Out:** the survivors, packed, using one of two kernels chosen by +> the popcount — with no branch on any individual element. + +This is the file to read closely, because it is the one your machine +runs. `scalar.rs` has *two* inner kernels, not one. The sparse kernel +iterates set bits: + +```rust +// polars crates/polars-compute/src/filter/scalar.rs:9-25 — the sparse kernel + 9 unsafe fn scalar_sparse_filter64(v: &[T], mut m: u64, out: *mut T) { + 10 let mut written = 0usize; + 12 while m > 0 { + 13 // Unroll loop manually twice. + 14 let idx = m.trailing_zeros() as usize; + 15 *out.add(written) = *v.get_unchecked(idx); + 16 m &= m.wrapping_sub(1); // Clear least significant bit. + 17 written += 1; + 19 // tz % 64 otherwise we could go out of bounds + 20 let idx = (m.trailing_zeros() % 64) as usize; + 21 *out.add(written) = *v.get_unchecked(idx); + 22 m &= m.wrapping_sub(1); // Clear least significant bit. + 23 written += 1; + 24 } + 25 } ``` +Cost is proportional to the popcount, not to 64 — and note line 13's +manual ×2 unroll, which is why the safety contract at line 8 demands +room for "`m.count_ones() + 1` writes": the second half of an unrolled +iteration may run with `m == 0` and write one junk element. This is +simdjson's flatten-bits idiom (`json_structural_indexer.h:93-121`) +applied to values instead of indices. + +The dense kernel never branches at all: + ```rust -// AVX-512 replaces the whole scalar loop with one compress-store per vector: -// _mm512_maskz_compress_epi32(mask, v); out_ptr += mask.count_ones(); +// polars crates/polars-compute/src/filter/scalar.rs:30-46 — the dense kernel + 30 unsafe fn scalar_dense_filter64(v: &[T], mut m: u64, out: *mut T) { +// ... 31-33: comment — pointer form generates better code ... + 34 let mut written = 0usize; + 35 let mut src = v.as_ptr(); + 37 // We hope the outer loop doesn't get unrolled, but the inner loop does. + 38 for _ in 0..16 { + 39 for i in 0..4 { + 40 *out.add(written) = *src; + 41 written += ((m >> i) & 1) as usize; + 42 src = src.add(1); + 43 } + 44 m >>= 4; + 45 } + 46 } ``` -This is the only place polars drops to raw intrinsics — compress is -data-dependent lane *movement*, which no portable abstraction (and -no autovectorizer) can express. Question: at 99% selectivity, which -wins — bit-iteration or copy-everything-then-truncate? What does -polars do for the mostly-true case (look for the `is_simple` / -all-set fast path)? +Lines 40-41 are exactly this topic's `compact_branchless` +(`experiments/src/filter.rs:21-22`): store unconditionally, advance the +cursor by the predicate. 64 iterations regardless of the data. -### Step 6 — what NEON gets instead +And the caller chooses between them by counting bits: -No vpcompress on ARM. Options polars doesn't need but you do (M17): -simdjson's LUT-shuffle compress (a table of shuffle patterns indexed -by the mask, 8 lanes max per `vqtbl1q` — reading-simdjson.md step 7), -or the branchless scalar append (`out[k] = x; k += keep as usize` — -often wins; measure!). This is the experiments' `filter.rs` stub. +```rust +// polars crates/polars-compute/src/filter/scalar.rs:102-124 — the four paths + 102 // Fast-path: empty mask. + 103 if m == 0 { + 104 continue; + 105 } +// ... 107-110: safety comment ... + 111 // Fast-path: completely full mask. + 112 if m == u64::MAX { + 113 core::ptr::copy_nonoverlapping(value_chunk.as_ptr(), out, 64); + 114 out = out.add(64); + 115 continue; + 116 } + 118 let m_popcnt = m.count_ones(); + 119 if m_popcnt <= 16 { + 120 scalar_sparse_filter64(value_chunk, m, out) + 121 } else { + 122 scalar_dense_filter64(value_chunk, m, out) + 123 }; + 124 out = out.add(m_popcnt as usize); +``` + +Four paths, and the decision is made **once per 64 elements** on a +value (`m`) that is already in a register — never per element: -### Step 7 — dispatch: pay for the feature test once per block +``` + m == 0 (0% in this word) -> skip 64 elements entirely + m == u64::MAX (100%) -> one 64-element memcpy + popcnt <= 16 (<= 25%) -> sparse: cost ~ popcount + popcnt > 16 (> 25%) -> dense: fixed 64 iterations +``` -The AVX-512 path exists only on some CPUs, so filter/mod.rs does -runtime dispatch: `is_x86_feature_detected!` at the kernel boundary -— one check per 64+ elements, not per element. Compare the two other -binding times in this topic's codebases: hashbrown binds at COMPILE -time (`cfg_if!` per Group backend), SimSIMD at INIT time -(function-pointer tables filled once). Question: when is each of the -three binding times right (compile / init / call)? +Why 16 and not something else? Count operations and you will not get +16: sparse costs about 5 ops per survivor (tzcnt, load, store, blsr, +increment), dense about 4 per element regardless, so a pure op count +would put the crossover near 50 elements. The real cost is the shape, +not the count — sparse's load address depends on `tzcnt(m)`, so each +survivor sits behind a serial `m → tzcnt → address → load` chain, and +the `while m > 0` trip count is data-dependent. Dense's loads are a +sequential stream and its trip count is a constant. The code offers no +justification comment for 16, so read it as polars' measured choice +(25 % of 64) rather than a derivation. + +Note also what is *absent*: there is no `is_simple` helper in this +file. The fast paths are the two literal comparisons at lines 103 and +112. + +### Step 5 — the compress instruction: the one place intrinsics are needed + +> **In:** a 512-bit vector and a 64-bit mask, on x86 hardware that has +> AVX-512. +> **Out:** the selected lanes packed to the left in one instruction — +> and a comment explaining why polars still does not use the +> compress-*store* form. + +```rust +// polars crates/polars-compute/src/filter/avx512.rs:50-62 — the u8 kernel + 50 pub unsafe fn filter_u8_avx512vbmi2<'a>( +// ... 51-54: signature ... + 55 simd_filter!(values, mask_bytes, out, |vchunk, m: u64| { + 56 // We don't use compress-store instructions because they are very slow + 57 // on Zen. We are allowed to overshoot anyway. + 58 let v = _mm512_loadu_si512(vchunk.as_ptr().cast()); + 59 let filtered = _mm512_maskz_compress_epi8(m, v); + 60 _mm512_storeu_si512(out.cast(), filtered); + 61 out = out.add(m.count_ones() as usize); + 62 }) +``` + +Lines 56-57 are the whole lesson: the ISA offers a fused +compress-and-store, and polars declines it because it is slow on one +vendor's implementation, preferring compress-to-register (line 59) plus +a full-width store (line 60) and a pointer bump by popcount (line 61). +The overshoot is safe because `filter_values_generic` over-allocated — +`Vec::with_capacity(mask_bits_set + pad)` at `primitive.rs:75`, with +`pad` = 64/32/16/8 for u8/u16/u32/u64, i.e. one 512-bit vector's worth +of elements — and then truncates with `out.set_len(mask_bits_set)` at +`primitive.rs:80`. + +The `simd_filter!` macro (`avx512.rs:7-43`) is the shared skeleton: +`avx512.rs:12` chunks by 64 for the same `m64 == 0` sparse fast path as +the scalar file (line 20), then `avx512.rs:24` walks sub-chunks of +`MASK_BITS` elements. Three kernels instantiate it — `epi8` under +`avx512vbmi2` (line 59), `_mm512_maskz_compress_epi32` under `avx512f` +(line 94), and `_mm512_maskz_compress_epi64` (line 104+). + +This is the **only** place polars drops to raw intrinsics for filtering, +and the reason is structural: compress is data-dependent lane +*movement*. `std::simd` has no portable operation for it, and no +autovectoriser will invent one. + +### Step 6 — what your machine actually runs + +> **In:** `target_arch = "aarch64"`. +> **Out:** `nop_filter`, and 100 % of the work in `scalar_filter`. + +The dispatch is not in `mod.rs`. `mod.rs` only decides whether the +AVX-512 module *exists*: + +```rust +// polars crates/polars-compute/src/filter/mod.rs:2-7 — module gating + 2 mod boolean; + 3 mod primitive; + 4 mod scalar; + 6 #[cfg(all(target_arch = "x86_64", feature = "simd"))] + 7 mod avx512; +``` + +The runtime test lives one file over, and — correcting a claim that is +easy to make from the shape of the code — it happens **once per array**, +not once per 64-element block: + +```rust +// polars crates/polars-compute/src/filter/primitive.rs:49-56, 67-80 + 49 fn filter_values_u32(values: &[u32], mask: &Bitmap) -> Vec { + 50 #[cfg(all(target_arch = "x86_64", feature = "simd"))] + 51 if is_avx512_enabled() { + 52 return filter_values_generic(values, mask, 16, avx512::filter_u32_avx512f); + 53 } + 55 filter_values_generic(values, mask, 1, nop_filter) + 56 } +// ... 58-65: the u64 twin ... + 67 fn filter_values_generic( +// ... 68-74: signature and set_bits() ... + 75 let mut out = Vec::with_capacity(mask_bits_set + pad); + 77 let (values, mask_bytes, out_ptr) = scalar_filter_offset(values, mask, out.as_mut_ptr()); + 78 let (values, mask_bytes, out_ptr) = bulk_filter(values, mask_bytes, out_ptr); + 79 scalar_filter(values, mask_bytes, out_ptr); + 80 out.set_len(mask_bits_set); +``` + +On aarch64 lines 50-53 do not compile at all, so `filter_values_u32` is +line 55 and nothing else. Trace it through: `pad = 1`, +`bulk_filter = nop_filter`, and `nop_filter` (`primitive.rs:13-19`) +returns its three arguments unchanged. Line 78 is a no-op; line 79 does +everything. **Your Mac's polars filter is Step 4's two scalar kernels, +start to finish** — which is why Step 4 is the long one. + +If you want the vector version on NEON you have to write it, and this +topic's `experiments/src/filter.rs` is where. The options are +simdjson's LUT-shuffle compress (`arm64/simd.h:246-278`, an 8-lane +`vqtbl1q_u8` per half) or the branchless append polars already uses. +`notes.md` records branchless at 12.73 GB/s at 50 % selectivity against +branchy's 1.19 — so the bar your NEON kernel has to clear is a scalar +one, and it is not low. + +### Step 7 — dispatch, and the three binding times + +> **In:** one binary that must run on machines with different ISAs. +> **Out:** a choice between compile-time, init-time and call-time +> binding — and a cost model for picking one. + +This topic shows all three: + +| binding time | who | mechanism | cost per use | +|---|---|---|---| +| compile | hashbrown, memchr | `cfg_if!` / `#[cfg]` selects a backend file (`group/mod.rs:8-45`) | zero | +| init | SimSIMD | `__attribute__((constructor))` fills a function-pointer table (`c/numkong.c:917-919`) | one indirect call per kernel invocation | +| call | polars | `is_avx512_enabled() && is_x86_feature_detected!(…)` (`primitive.rs:33`) | one predictable branch per array | + +polars' choice is right for its shape: the dispatched unit is a whole +column, so a branch per array is unmeasurable, and shipping source +means the `#[cfg]` layer above it already removed the impossible +options. hashbrown's is right because its dispatched unit is a +handful of instructions. SimSIMD's is right because it ships a C +library as a binary and cannot recompile per host. ## Where each step lives in the code +polars at `f8bcc3d`, under `crates/polars-compute/src/`. + | anchor | step | what it is | |---|---|---| -| float_sum.rs:13-14 | 2–3 | `STRIPE = 16`, `PAIRWISE_RECURSION_LIMIT = 128` | -| float_sum.rs:44 | 2 | `vector_horizontal_sum` — reduce lanes at the END only | -| float_sum.rs:67-90 | 2 | `SumBlock`: sum 128 elems as 16-lane chunks (chunks_exact) | -| filter/scalar.rs:12 | 4 | mask-bit loop: `while m > 0` + trailing_zeros (simdjson's flatten!) | -| filter/scalar.rs:90 | 4 | 64-element blocks — process a whole mask word | -| filter/avx512.rs:7 | 5 | `simd_filter!` macro — the shared loop skeleton | -| filter/avx512.rs:50-60 | 5 | `filter_u8_avx512vbmi2`: `_mm512_maskz_compress_epi8` | -| filter/avx512.rs:87-95 | 5 | u32 via `_mm512_maskz_compress_epi32` (AVX-512F) | -| filter/mod.rs | 7 | dispatch: runtime feature detect → avx512 or scalar | -| min_max/ | — | same pattern for min/max kernels (a second lap, optional) | - -Reading order: `float_sum.rs` top to bottom (steps 1–3 in ~100 -lines), then `filter/scalar.rs`, then `avx512.rs` with the macro -expanded in your head, then `mod.rs` for the dispatch. +| `float_sum.rs:2-3` | 2 | `std::simd` imported only under `feature = "simd"` | +| `float_sum.rs:13-14` | 2-3 | `STRIPE = 16`, `PAIRWISE_RECURSION_LIMIT = 128` | +| `float_sum.rs:44-63` | 1-2 | `vector_horizontal_sum` — the once-per-block fold, and the non-associativity comment | +| `float_sum.rs:79-85` | 2 | `sum_block_vectorized` — `chunks_exact(STRIPE)` into `Simd` | +| `float_sum.rs:87-97` | 3 | the masked variant: `m.select(v, zero)`, nulls without branches | +| `float_sum.rs:155-186` | 2 | the **non-SIMD** fallback: a plain `[F; 16]` accumulator array | +| `float_sum.rs:189-211` | 3 | `pairwise_sum` — recursive halving down to 128 | +| `filter/mod.rs:6-7` | 6 | `mod avx512` exists only on `x86_64` | +| `filter/mod.rs:30-52` | 6 | `filter_with_bitmap` — leading/trailing-zero trim and all-empty/all-full paths | +| `filter/primitive.rs:11-19` | 6 | `FilterFn` and `nop_filter` — what aarch64 gets | +| `filter/primitive.rs:21-65` | 6-7 | dispatch by element size, then one feature test **per array** | +| `filter/primitive.rs:67-83` | 5-6 | `filter_values_generic`: over-allocate by `pad`, offset, bulk, scalar, `set_len` | +| `filter/scalar.rs:9-25` | 4 | `scalar_sparse_filter64` — `trailing_zeros` + `m &= m-1`, unrolled ×2 | +| `filter/scalar.rs:30-46` | 4 | `scalar_dense_filter64` — store-always, advance by bit | +| `filter/scalar.rs:102-124` | 4 | the four paths and the `popcnt <= 16` threshold | +| `filter/avx512.rs:7-43` | 5 | `simd_filter!` — the shared 64-element skeleton | +| `filter/avx512.rs:50-63` | 5 | `filter_u8_avx512vbmi2` and the "slow on Zen" comment | +| `filter/avx512.rs:87-98` | 5 | `_mm512_maskz_compress_epi32` under plain AVX-512F | +| `min_max/` | — | the same pattern for min/max (`scalar.rs`, `simd.rs`) — an optional second lap | + +Reading order: `float_sum.rs` top to bottom (Steps 1-3 in about 210 +lines, and read the `#[cfg(not(feature = "simd"))]` impl at 155 as +carefully as the SIMD one), then `filter/primitive.rs` to see where +your machine lands, then `filter/scalar.rs` because that is where it +lands, then `filter/avx512.rs` for the road not taken. ## Questions for notes.md -1. STRIPE=16 for f32 = 512 bits = 4 NEON registers. Why does a - WIDER stripe than the vector width still help on ARM (ports × - latency)? -2. Pairwise limit 128: derive the error bound difference vs - left-to-right for n = 10⁸ (hint: ~ε·log₂(n/128) vs ~ε·n). -3. The `simd_filter!` skeleton advances `out` by popcount without - zeroing skipped lanes. Why is the trailing garbage safe (who - truncates)? -4. Filter returns (values, validity) — how does the validity BITMAP - itself get filtered (bit-level compress — the harder problem)? -5. For M17: polars chose NOT to use `std::simd` for filter, only - intrinsics + scalar. Why does compress specifically defeat - portable SIMD abstractions? +1. Step 2 puts `Simd` at 4 NEON registers and `Simd` + at 8, against a latency × ports target of about 12. Work out the + STRIPE that would hit 12 chains for `f32` on NEON, then say what it + would cost on AVX-512 — and why one constant has to serve both. +2. Derive Step 3's error bound for your own N = 4M dot product rather + than n = 1e8, and compare it with `notes.md`'s open question about + "max f32 dot error vs naive at N=4M". +3. `scalar_sparse_filter64`'s contract asks for `count_ones() + 1` + writes and `filter_values_generic` allocates `mask_bits_set + pad`. + Trace one 64-element word at popcount 1 and say exactly which write + is the extra one. +4. The `popcnt <= 16` threshold does not fall out of an operation + count (Step 4 shows it would predict ~50). Design the experiment + that would find the real crossover on your machine, and predict + which way it moves for `u64` values versus `u8`. +5. `filter_with_bitmap` (`filter/mod.rs:30-52`) trims leading and + trailing zero runs before doing anything. For a Cypher `WHERE` over + a sorted-ish column, what fraction of the work does that remove, + and which of Step 4's four paths does it make redundant? +6. For M17: polars uses `std::simd` for the sum but *not* for the + filter. State the property of compress that defeats portable SIMD, + and name one other operation in this topic with the same property. ## Done when -- [ ] You can explain the reduction problem and why STRIPE=16 fixes it — then connect it to this topic's measured 8.88 -> 26.32 GB/s from eight accumulators alone. -- [ ] You can explain how pairwise recursion fixes accuracy and derive why the error bound differs from a single chain. -- [ ] You can describe the `simd_filter!` skeleton and how it advances the output pointer by popcount. -- [ ] You can say what NEON gets instead of a compress instruction, and what that costs. -- [ ] You can explain the dispatch strategy — one feature test per block — and why polars chose not to use `std::simd` for filter. -- [ ] You wrote answers to all five questions in notes.md. +Answer each before unfolding it. + +- [ ] You can explain the reduction problem, and connect STRIPE = 16 to this topic's own measured accumulator win. + +
Answer + + One accumulator is one dependency chain of length n, costing + `n × latency` no matter how many adders exist. `float_sum.rs:81-83` + accumulates into `Simd` so lane j sums elements j, j+16, … — + 16 lanes, folded to a scalar once per 128-element block by + `vector_horizontal_sum` (`:44-63`), which is careful about ordering + because float addition is not associative (`:49-51`). + + `notes.md` measures the same idea in this topic's dot product: + 10.89 → 42.12 GB/s (3.9×) from eight accumulators and no intrinsics. + (`FINDINGS.md` row 17 has 8.88 → 26.32 from another run.) Caveat: at + N = 4M the fast rung may be bandwidth-bound, which `notes.md`'s + worksheet asks you to check at N = 64K. + +
+ +- [ ] You can size STRIPE = 16 in physical registers on this machine, and state the tension it creates. + +
Answer + + `Simd` is 512 bits = **4** NEON registers; `Simd` + is 1024 bits = **8**. On AVX-512 the f32 case is a single register. + So polars deliberately runs 4× the hardware vector width for f32, + because accumulator count is set by latency × ports (about 3 × 4 = 12 + for FMLA on an Apple M-class P-core, per SimSIMD's table at + `dot/neon.h:13-24`), not by register width. + + Mojo's documentation gives the opposite advice — never exceed 2× the + register size, because the compiler splits the vector and "the + resulting code will perform poorly". Both are real positions; the + difference is that polars is accumulating, where extra registers buy + independent chains, and Mojo's warning is about straight-line + element-wise code, where they buy nothing. + +
+ +- [ ] You can explain how pairwise recursion fixes accuracy, with the arithmetic. + +
Answer + + `pairwise_sum` (`float_sum.rs:189-211`) halves the array down to + 128-element blocks (line 207) and sums each block with Step 2's + striped kernel. For n = 1e8 the longest chain of dependent additions + any one value passes through is 128/16 = 8 within its lane, plus + log2(16) = 4 for the horizontal fold, plus log2(1e8/128) ≈ 20 across + blocks — about 32ε against a naive left-to-right bound of 1e8·ε, a + factor of ~3.1e6. + + The SIMD blocking and the accuracy fix are the same tree. + +
+ +- [ ] You can describe both scalar filter kernels and the rule that picks between them. + +
Answer + + `scalar_sparse_filter64` (`filter/scalar.rs:9-25`) iterates set bits + with `trailing_zeros` and `m &= m.wrapping_sub(1)`, manually unrolled + ×2 — cost proportional to popcount, and the reason its contract + demands `count_ones() + 1` writes. `scalar_dense_filter64` (`:30-46`) + stores every element and advances the cursor by + `((m >> i) & 1)` — 64 fixed iterations, no branch. + + `scalar_filter` (`:102-124`) picks per 64-element word: skip if + `m == 0`, one 64-element `copy_nonoverlapping` if `m == u64::MAX`, + sparse if `popcnt <= 16` (25 %), dense otherwise. The threshold is + empirical — an operation count would predict about 50 — and reflects + sparse's data-dependent load address and trip count. + +
+ +- [ ] You can say what your machine actually executes, and prove it from the `#[cfg]`s. + +
Answer + + Only the scalar path. `filter/mod.rs:6-7` gates `mod avx512` on + `target_arch = "x86_64"`, so on aarch64 it does not exist; + `filter_values_u32` (`primitive.rs:49-56`) therefore compiles to just + `filter_values_generic(values, mask, 1, nop_filter)`, and `nop_filter` + (`primitive.rs:13-19`) returns its arguments unchanged. In + `filter_values_generic` (`:67-83`) line 78 is a no-op and line 79's + `scalar_filter` does all the work, with `pad = 1`. + + So there is no compress instruction, no runtime feature test that can + succeed, and no vector code in polars' primitive filter on this Mac. + +
+ +- [ ] You can explain why compress needs intrinsics, and place polars' dispatch among the three binding times. + +
Answer + + Compress is data-dependent lane *movement*: which source lane feeds + which destination lane is a function of the mask, not of the program. + `std::simd` has no portable operation for that and no autovectoriser + will synthesise one, so `avx512.rs` is the one file in + polars-compute that uses raw intrinsics. (simdjson answers the same + problem on NEON with a lookup table of shuffle patterns.) + + Binding times: compile-time in hashbrown/memchr (`cfg_if!`), + init-time in SimSIMD (a constructor filling a function-pointer table + at `c/numkong.c:917-919`), call-time in polars + (`primitive.rs:33`, once per array). polars' unit of dispatch is a + whole column, so a branch per array costs nothing measurable. + +
+ +- [ ] You wrote answers to all six questions in notes.md. + +
Answer + + Self-check. Question 3 has one right answer worth verifying: at + popcount 1 the ×2 unroll runs its second half with `m == 0`, so + `trailing_zeros() % 64` gives index 0 and one junk element is written + past the survivor — which is exactly the "+1" in both the contract at + `filter/scalar.rs:8` and the `+ pad` at `primitive.rs:75`. + +
## References **Code** -- [polars](https://github.com/pola-rs/polars) — - `crates/polars-compute/src/` — start with `float_sum.rs` and - `filter/` (scalar.rs, avx512.rs, mod.rs); `min_max/` repeats the - same pattern if you want a second lap +- [polars](https://github.com/pola-rs/polars) at `f8bcc3d` — + `crates/polars-compute/src/float_sum.rs` for the reduction (read the + `#[cfg(not(feature = "simd"))]` impl too), `filter/primitive.rs` for + the dispatch, `filter/scalar.rs` for the two kernels your machine + runs, `filter/avx512.rs` for the intrinsic path. `min_max/` repeats + the same structure if you want a second lap. + +**Cross-references in this topic** +- `reading-simdjson.md` — the NEON answer to compress + (`arm64/simd.h:246-278`) and the flatten-bits loop that + `scalar_sparse_filter64` mirrors. +- `reading-sigmod15-vectorization.md` — the selectivity curve these two + kernels are sitting on, and where the crossover comes from. +- `reading-mojo-simd.md` — the "never exceed 2× the register width" + advice that Step 2 contradicts on purpose. diff --git a/topics/17-simd/reading-sigmod15-vectorization.md b/topics/17-simd/reading-sigmod15-vectorization.md index 733a091..7786f89 100644 --- a/topics/17-simd/reading-sigmod15-vectorization.md +++ b/topics/17-simd/reading-sigmod15-vectorization.md @@ -1,204 +1,552 @@ # SIMD for databases: two primitives, four operators -Polychroniou, Raghavan & Ross's SIGMOD '15 paper turned "SIMD for -databases" from folklore into a catalog. It vectorizes the FOUR -fundamental operators — selection scan, hash probe, bloom filter, +Polychroniou, Raghavan & Ross's SIGMOD '15 paper, "Rethinking SIMD +Vectorization for In-Memory Databases", turned "SIMD for databases" +from folklore into a catalog. It vectorises four fundamental operators +— selection scan, hash table probe/build, Bloom filter probe, radix partition — and shows each is a composition of two primitives: -**selective store** (compress) and **selective load / gather**. -Before the paper, this chapter builds those primitives and each -operator's shape step by step. Read it as the spec for our +**selective store** (compress) and **selective load** (expand), plus +their indexed cousins gather and scatter. Before the paper, this +chapter builds those primitives and each operator's shape step by +step. Read it as the specification for this topic's `experiments/filter.rs` and for M17's engine kernels. +There is no pinned clone for the paper's code, so every claim is +anchored to the paper by section, algorithm or figure number, and +every speedup is reported **with the operator and the machine it was +measured on** — the paper's numbers vary from 1.05× to 10× depending +on both. Local measurements come from this topic's `notes.md` and are +labelled as such. + ## The problem in one sentence -Database operators branch on data (does this row pass? did this -probe hit?), and a mispredicted branch costs ~15 cycles — the paper -shows that recasting the four core operators as branch-free lane -operations wins up to 3–6× in cache, and tells you exactly when it -doesn't (out-of-cache gathers). +Database operators branch on data (does this row pass? did this probe +hit?), and a mispredicted branch costs tens of cycles; the paper +recasts the four core operators as branch-free lane operations, and — +just as usefully — tells you exactly where that stops paying, which is +wherever the kernel's cost is a random memory access rather than an +instruction. ## The concepts, step by step ### Step 1 — lanes, masks, and the operator question -SIMD (single instruction, multiple data — one instruction operating -on a vector of W values, its **lanes**; W=4 f32s on 128-bit NEON, -16 on 512-bit AVX-512) is trivial for `a[i] + b[i]`. Database -operators are harder because each lane wants to *do something -different*: keep this row, drop that one, probe another bucket. The -paper's framing: express every such divergence as a **mask** (a -bitmap with one bit per lane, produced by a vector comparison) and -find instructions that consume masks instead of branching on them. -The whole catalog reduces to two such instructions. - -### Step 2 — the two primitives: selective store and selective load/gather - -**Selective store** (also "compress") writes only the masked lanes -to memory, packed contiguously. **Selective load** ("expand") fills -only the masked lanes from memory, leaving the rest untouched. Add -their indexed cousins — **gather** (lanes = `mem[idx[i]]`, W loads -from arbitrary places) and **scatter** (the reverse) — and every -operator in the paper is a loop of: compute masks → compress -finished lanes out → refill from input: +> **In:** an operator whose per-row work diverges — keep this row, drop +> that one, probe another bucket. +> **Out:** the same operator expressed as a **mask** plus instructions +> that consume masks, with no branch anywhere. + +SIMD — single instruction, multiple data — applies one instruction to a +vector of W values, its **lanes**. On this Mac a 128-bit NEON register +holds W = 4 `f32` or `u32` lanes; the paper's Xeon Phi holds W = 16 +32-bit lanes in 512 bits. + +`a[i] + b[i]` is trivial to vectorise. Database operators are not, +because each lane wants to do something *different*. The paper's whole +framing is: express every divergence as a **mask** — a value with one +bit per lane, produced by a vector comparison — and then find +instructions that *consume* masks instead of branching on them. Four +such instructions carry the entire catalog. + +### Step 2 — the two primitives (§3), and how to fake them + +> **In:** a vector of values and a mask. +> **Out:** a memory operation that touches only the masked lanes — +> either natively, or emulated with a permutation lookup table. + +§3 defines four operations, each with its own figure: ``` - selective STORE (compress): selective LOAD (expand/gather): - lanes: a b c d e f g h memory: p q r s ... - mask: 1 0 1 1 0 0 1 0 mask: 1 0 1 1 ... - memory: a c d g ────────► lanes: p . q r ... ◄──────── - (filter output, partition out) (refill lanes after some finish) - - gather: lanes = mem[idx[0..W]] (hash probe, dictionary decode) - scatter: mem[idx[0..W]] = lanes (partition, hash build) + selective STORE (Fig. 1) selective LOAD (Fig. 2) + lanes: a b c d e f g h memory: p q r s ... + mask: 1 0 1 1 0 0 1 0 mask: 1 0 1 1 ... + memory: a c d g --------> lanes: p . q r ... <-------- + (filter output, partition out) (refill lanes after some finish) + + gather (Fig. 3): lanes = mem[idx[0..W]] (hash probe, dict decode) + scatter (Fig. 4): mem[idx[0..W]] = lanes (partition, hash build) ``` -AVX-512 has all four as instructions; NEON has none natively (hence -simdjson's LUT compress and the gather cost model below). That gap -is why the paper (written pre-AVX-512) emulates compress via -permutation LUTs — exactly the trick your NEON kernels will use. +§3's note on scatter semantics matters later: "If multiple vector lanes +point to the same location, we assume that the rightmost value will be +written." That single sentence is why Step 6 exists. -### Step 3 — selection scan (§4): three shapes, one selectivity sweep +Table 1 tells you which machine has which. Xeon Phi 7120P: gather yes, +scatter yes. Haswell E3-1275v3: gather yes, scatter **no**. Sandy +Bridge E5-4620: neither. Your NEON machine has none of the four as +instructions — so the paper's emulation is not a historical footnote +for you, it is the implementation: + +> "Selective loads and stores are also not supported on the latest +> mainstream CPUs, but can be emulated using vector permutations. The +> lane selection mask is extracted as a bitmask and is used as an array +> index to load a permutation mask from a pre-generated table. The data +> vector is then permuted in a way that splits the active lanes of the +> mask to the one side of the register and the inactive lanes to the +> other side." (§3) -The filter kernel — keep elements passing a predicate — comes in -three shapes, and **selectivity** (the fraction of elements kept) -decides the winner. Branchy code mispredicts worst at 50% -selectivity (the branch is a coin flip, ~15 cycles per miss); -branchless always-store code is flat; compress does W elements per -instruction: +For a selective store you then store the whole vector unaligned and +advance the pointer by the popcount; for a selective load you load a +new vector and blend. §3 credits the technique to the vectorized Bloom +filter work [27], "without defining the operations". +Size the table, because that is what decides whether you can use it: + +``` + permutation LUT = 2^W entries, each W lane-indices + + W = 4 (NEON, u32 lanes) : 2^4 = 16 x 16 B = 256 B <- this topic + W = 8 (Haswell, 32-bit) : 2^8 = 256 x 8 B = 2 KB + W = 16 (Phi / AVX-512) : 2^16 = 65536 x 16 B = 1 MB <- L2-sized ``` - branchy │ ns/elem peaks at ~50% sel (mispredict wall) - branchless │ flat line — always-store, data-independent - SIMD compress │ flat, lower — W elems per compress - └──────────────── selectivity → - crossover: branchy wins BELOW ~few % and ABOVE ~95% + +The W = 16 row is why simdjson splits its 16-byte compress into two +8-byte halves (`internal/simdprune_tables.h:11`, a 256 × 8 B = 2 KB +table) rather than building the 1 MB one. This topic's own kernel is +the top row: `notes.md`'s implementation log calls for a compact_neon +"(LUT built, all 16 masks pass)" — 16 masks because W = 4. + +### Step 3 — selection scan (§4): three shapes, one selectivity sweep + +> **In:** a key column, a range predicate, and a selectivity — the +> fraction of rows that pass. +> **Out:** the surviving rows (or their indexes), and a curve that +> shows which of three implementations wins at which selectivity. + +§4 gives three algorithms. Algorithm 1 is scalar with a branch; +Algorithm 2 is the scalar branchless trick — store unconditionally, +advance the cursor by the predicate; Algorithm 3 is the vector version. +This topic ships the first two verbatim: + +```rust +// topics/17-simd/experiments/src/filter.rs:5-25 — §4's Algorithms 1 and 2 + 5 pub fn compact_branchy(vals: &[f32], t: f32, out: &mut Vec) { + 6 out.clear(); + 7 for &v in vals { + 8 if v < t { + 9 out.push(v); + 10 } + 11 } + 12 } +// ... 14-15: doc comment ... + 16 pub fn compact_branchless(vals: &[f32], t: f32, out: &mut Vec) { + 17 out.clear(); + 18 out.resize(vals.len(), 0.0); + 19 let mut k = 0usize; + 20 for &v in vals { + 21 out[k] = v; + 22 k += (v < t) as usize; + 23 } + 24 out.truncate(k); + 25 } ``` -The paper's addition our README didn't have: with SIMD you compute -the mask for W lanes and use it to compress-store BOTH the values -and their RIDs (row ids — the positions of surviving rows, which is -what a real engine's filter actually outputs; topic 11's selection -vectors). Question: does compressing (value,rid) pairs double the -cost or can one mask drive two compresses? +Line 8 is a data-dependent branch; line 22 is the same decision as +arithmetic. That is the entire difference, and the sweep is dramatic. +From `notes.md` (provided rungs, release, Apple Silicon, 2026-07-10; +GB/s of *input*, N = 4M f32 = 16,777,216 B, 20 reps): -### Step 4 — hash probe (§5): W independent probes, refilled as they finish +| selectivity | branchy GB/s | branchless GB/s | +|---|---|---| +| 1 % | 10.95 | 12.70 | +| 25 % | 2.13 | 13.32 | +| 50 % | **1.19** | 12.73 | +| 75 % | 2.11 | 12.38 | +| 99 % | 6.65 | 11.98 | -The naive way to vectorize a hash-table probe vectorizes ONE probe's -steps (horizontal). The paper's way is **vertical vectorization**: -run W *independent* probes, one per lane. The complication is that -probes finish at different times (some hit immediately, some -collide and probe on) — solved by the done-mask + refill pattern, -built entirely from Step 2's primitives: +Now derive the mispredict cost from that, instead of quoting a folklore +number. The paper itself never states one — §1 says only that a +mispredicted branch costs "several cycles" — so the honest figure is +the one your own machine just produced: ``` - keys = selective_load(input, done_mask) ← refill finished lanes - hashes = hash(keys) - slots = gather(table, hashes) - done = (slots.key == keys) | (slots == EMPTY) - output = selective_store(matches) - bucket += 1 where !done ← collided lanes probe on + branchy at 50%: 16,777,216 B / 1.19e9 B/s = 14.098 ms + branchless at 50%: 16,777,216 B / 12.73e9 B/s = 1.318 ms + elements: 4,194,304 + + per element: branchy 3.361 ns, branchless 0.314 ns + gap = 3.047 ns/element + clock, derived not asserted (reading-simsimd.md Step 2): + naive dot 10.89 GB/s / 8 B per element-pair = 1.361 G pair/s + one scalar FMA chain, FMLA latency 3 cy => >= 4.08 GHz + 3.047 ns x 4.08 GHz = 12.43 cycles per element + at 50% selectivity a coin-flip branch misses ~half the time: + 12.43 / 0.5 = ~25 cycles per mispredict ``` +Treat 25 as an **upper bound**: the two kernels do not do identical +work apart from the branch (branchless always stores, so it moves more +bytes), and the clock is a lower bound derived from another lane. But +it is a measured bound from this machine, which is worth more than an +unattributed "~15 cycles". +(`FINDINGS.md`'s row 17 records 0.95 GB/s at 50 % from a different run +of the same bench; use whichever you are citing, do not average them.) + +Two things the README's version of this story leaves out. First, §4's +Algorithm 3 does not buffer the qualifying *values* — it selectively +stores their **indexes** into a cache-resident buffer and dereferences +keys and payloads only when the buffer is flushed, "which are used to +dereference the actual key and payload values during buffer flushing" +(§10.1). At low selectivity that skips the payload column almost +entirely, which is the real source of the low-selectivity win. Second, +the winner depends on the machine as much as the selectivity: §10.1 +reports that on Xeon Phi "scalar code is almost an order of magnitude +slower than vector code, whereas on Haswell, vector code is about twice +faster", and that "on Haswell, all vector versions are almost identical +by saturating the bandwidth, while the branchless scalar code catches +up on 10% selectivity." + +Your own sweep matches Haswell's shape, not Phi's — and it adds a +finding the paper does not have. `notes.md`: "branchy never actually +wins here even at 1%/99% — the paper's crossover needs even more +extreme selectivities (<1%) on this core." Branchless is flat within +±5 % across the whole sweep, which is the point: its control flow does +not depend on the data. + +### Step 4 — hash probe (§5.1): W independent probes, refilled as they finish + +> **In:** a probe column and a linear-probing hash table. +> **Out:** W probes in flight at once, each lane on its own bucket, +> with finished lanes refilled from the input mid-loop. + +The obvious way to vectorise a probe is **horizontally**: compare one +key against several table slots at once (which is what a bucketised +table does, and what hashbrown does with control bytes). §5.1's way is +**vertical**: run W *independent* probes, one per lane. + +The complication is that probes finish at different times — one hits +immediately, another collides and walks on — and Algorithm 5 solves it +with exactly Step 2's primitives: + ```rust -// vertical probing: W INDEPENDENT probes in flight, refilled as they finish +// ILLUSTRATION — the shape of §5.1's Algorithm 5, not quoted code. +// The primitives are §3's Figures 1-4; this topic's scalar analogue is +// topics/17-simd/experiments/src/filter.rs:16, whose "advance by the +// predicate" is the same idea one lane wide. loop { - keys = selective_load(keys, input, done); // finished lanes take new keys - let slot = gather(table, hash(keys) + bucket); // ~1 cache access PER LANE + keys = selective_load(keys, input, done); // refill finished lanes + let slot = gather(table, hash(keys) + offset); // ~1 cache access PER LANE let hit = slot.key.simd_eq(keys); - let empty = slot.simd_eq(EMPTY); - selective_store(out, hit, slot.val); // compress matched lanes out + let empty = slot.key.simd_eq(EMPTY); + selective_store(out, hit, slot.payload); // compress matches out done = hit | empty; - bucket = done.select(ZERO, bucket + 1); // collided lanes probe on + offset = done.select(ZERO, offset + 1); // collided lanes probe on } ``` -This is hashbrown's group probing turned 90°: hashbrown = SIMD -*within* one probe (compare 16 control bytes of one bucket at once), -SIGMOD15 = SIMD *across* probes (W separate lookups in flight). -Question: which does M11's hash join want, given batch sizes of 1024 -and a table that misses L2? +Every lane carries its own bucket offset, so the vector never waits for +its slowest lane; a lane that finishes is refilled on the next +iteration rather than idling. §5.1 states the price plainly: "By +reusing vector lanes dynamically, we are reading the probing input +'out-of-order'. Thus, the probing algorithm is no longer stable, i.e., +the order of the output does not match the order of the input." + +This is hashbrown's group probing turned 90°. hashbrown is SIMD +*within* one probe — 8 control bytes of one bucket compared at once on +aarch64 (`reading-hashbrown-simd.md`, Step 2) — while §5.1 is SIMD +*across* probes, W separate lookups in flight. They compose badly: +vertical probing wants one table slot per lane, which is exactly what a +control-byte group is not. ### Step 5 — the gather cost model (§3): parallel instructions, serial memory -The paper's most durable measurement: a gather costs ~1 cache access -PER LANE — the instruction is one opcode, but the memory system -still performs W independent loads. Gather parallelizes the -*instruction stream*, not the *memory system*. Consequences: gather -wins only when the computation around it vectorizes; it never fixes -topic 13's pointer-chasing tax. Corollary the paper proves: -vectorized probe ≈ scalar probe when the table exceeds cache, but is -3–6× faster in-cache. Question: FalkorDB's adjacency lookups are -gathers over CSR — in-cache or out? What does that predict for -SIMD-izing traversals (M24)? - -### Step 6 — partition (§6): scatter needs conflict detection - -Radix partition (distributing rows into buckets by some digits of -their key) scatters each lane to `out[hist[digit(k)]++]`. New -hazard: two lanes in the SAME vector with the same digit collide on -the histogram slot — both would read the same counter, write the -same address, and lose one row. The paper detects conflicts -(AVX-512 `vpconflictd`; emulated before that) and serializes only -the colliding lanes. Question: NEON has no conflict detect either — -sketch the scalar-fallback-inside-vector-loop shape, and note where -topic 13's software write-combining buffers make the scatter moot. +> **In:** a gather instruction over W arbitrary addresses. +> **Out:** one instruction, but W cache accesses — and therefore a rule +> about which parts of an operator vectorisation can help. + +This is the paper's most durable paragraph, and it is worth memorising +verbatim: + +> "Gathers and scatters are not really executed in parallel because the +> (L1) cache allows one or two distinct accesses per cycle. Executing W +> cache accesses per cycle is an impractical hardware design. Thus, +> random memory accesses have to be excluded from the O(f(n)/W) +> vectorization rule." (§3) + +Cost it: + +``` + W = 16 lanes, L1 serves 1-2 distinct accesses/cycle + -> a fully-scattered 16-lane gather takes >= 8-16 cycles + -> i.e. no better than 16 scalar loads that all hit L1 + the gather WINS only on the instruction stream around it: + address arithmetic, comparisons, masking, the compress-store + and it wins nothing at all if the lines miss L1 (topic 13) +``` + +The corollary shows up in the measured results. §10.2 on linear probing +and double hashing: the vertical vector code is "up to 6X faster than +everything else on Xeon Phi, and gain a smaller speedup for cache +resident hash tables on Haswell". The 6× is a 61-core in-order machine +with 512-bit vectors at 1.238 GHz, whose scalar code is unusually weak; +the Haswell number is small enough that the paper does not give it a +figure. Vectorisation never fixed the memory system — it removed the +instructions *around* the memory system. + +### Step 6 — partition (§7.3): scatter needs conflict serialisation + +> **In:** W tuples in a vector, each destined for +> `out[offset[digit(k)]++]`. +> **Out:** correct offsets even when two lanes share a digit — using +> only gather and scatter, because the machine has no conflict-detect +> instruction. + +Radix partitioning scatters each lane to its partition's current +offset. The hazard is new: if two lanes in the same vector have the +same digit, both read the same counter, both compute the same address, +and §3's "the rightmost value will be written" rule silently drops a +row. + +Get the fix right, because it is commonly mis-stated. AVX-512's +`vpconflictd` is mentioned in §5.1 only as **future** hardware: "Future +SIMD instruction sets include special instructions that can support +this functionality (`vpconflictd` in AVX 3), thus saving the need for +the extra scatter and gather to detect conflicts. Nevertheless, these +instructions are not supported on mainstream CPUs or the Xeon Phi as of +yet." What the paper actually implements is Algorithm 13 (§7.3), +hand-rolled from the two primitives: + +``` + §7.3, Algorithm 13 — conflict serialization(h, A): + 1. reverse the lane order (permute by {W-1, ..., 0}) + 2. repeat: + scatter the unique per-lane values l into A[h] + gather them back into l_back + a lane is CONFLICTING where l != l_back + increment that lane's offset c + until no lane conflicts + 3. un-reverse c + + cost: up to W iterations, but "the total number of accesses to + distinct memory locations is always W" — sum over iterations of + the distinct accesses = W. +``` + +Two details in §7.3 explain the reversal. Because the rightmost lane is +the one that survives a conflicting scatter, tuples of the same +partition inside a vector would be written in reverse order; and per +group of k conflicting lanes the rightmost lane would increment the +offset by 1 instead of k. Reversing first fixes both, which keeps the +partitioning **stable** — and §7.3 notes that "Stable partitioning is +essential for algorithms such as LSB radixsort." + +Your machine has no scatter either, so if you ever need this you are in +the same position as the paper's Sandy Bridge row: the practical answer +is topic 13's software write-combining buffers, which make the +per-tuple scatter disappear into a per-partition sequential store. ### Step 7 — what to steal for the experiments -- The selectivity sweep axes (their Fig: cycles/tuple vs sel%) = - `simd_bench`'s filter output. Plot branchy/branchless/compress. -- Rigged input trick: they control selectivity EXACTLY by - construction — our bench does the same (threshold = quantile). -- Report cycles/tuple, not GB/s, for the probe kernel (memory-bound - kernels hide instruction wins behind bandwidth). +> **In:** the paper's methodology. +> **Out:** three habits that make this topic's bench comparable to it. + +- **The sweep axes.** The paper plots throughput against selectivity + (§10.1, Fig. 5). `simd_bench`'s filter lane does the same; plot + branchy, branchless and compress on one chart and the crossover is + visible rather than argued. +- **Rigged input.** Selectivity is controlled exactly by construction, + not sampled — this topic's bench picks the threshold as a quantile so + 1 %, 25 %, 50 %, 75 %, 99 % are exact. +- **Report the right unit.** Cycles or ns per tuple for probe kernels, + GB/s for scan kernels. A memory-bound kernel reported in GB/s hides + every instruction-level win behind the bandwidth ceiling — which is + precisely what §10.1 saw on Haswell, where "all vector versions are + almost identical by saturating the bandwidth". ## How to read the paper (with the concepts in hand) -- **§3 — read carefully.** The gather cost model (Step 5) is the - section that ages best; everything else depends on whether you - believe it. -- **§4 (selection)** — Step 3; compare their crossover points with - the ones your `simd_bench` selectivity sweep produces. -- **§5 (hash probe)** — Step 4; keep the vertical-vs-horizontal - contrast (SIGMOD15 vs hashbrown) in mind throughout. -- **§6 (partition)** — Step 6; skim the conflict-detection emulation - unless you're implementing it. -- The bloom-filter kernel is probe-minus-refill — read it as a - simplified §5. -- Skim the AVX-512 forecast knowing it came true: the "future - hardware" they emulate with LUTs is now `vpcompressd` in polars - (reading-polars-compute.md). +- **§3 — read carefully, twice.** Figures 1-4 are the whole vocabulary, + and the gather-cost paragraph (Step 5) is the sentence the rest of the + paper's honesty rests on. Read the permutation-LUT emulation + paragraph as *your* implementation, not as history. +- **§4 with Algorithms 1-3** — Step 3. Note that Algorithm 3 buffers + indexes, not values, and work out why before reading §10.1. +- **§5.1, Algorithm 5** — Step 4, including the stability sentence. + §5.2 (double hashing) and §5.3 (cuckoo) are variations on it; skim + unless you are implementing them. +- **§6 (Bloom filters)** — the paper evaluates the design of [27] + rather than inventing one. The load-bearing line is that "aborting a + tuple as soon as one bit-test fails is essential"; that is why + §10.3's speedups are the largest of any operator. +- **§7.3, Algorithm 13** — Step 6. Read the two paragraphs after the + algorithm, which explain the reversal. +- **§10 — read Table 1 first.** Every speedup below it is + per-operator and per-machine, and the two machines differ by more + than SIMD width: 61 in-order cores at 1.238 GHz versus 4 + out-of-order cores at 3.5 GHz. + +The speedups, kept together so they cannot be quoted loose: + +| operator | Xeon Phi (512-bit) | Haswell (256-bit) | where | +|---|---|---|---| +| selection scan | ~10× ("almost an order of magnitude") | ~2× | §10.1, Fig. 5 | +| linear probing / double hashing probe | up to 6× | "smaller … for cache resident hash tables" | §10.2, Fig. 6 | +| cuckoo probe | 5× | 1.7× | §10.2, Fig. 7 | +| Bloom filter probe | 3.6–7.8× | 1.3–3.1× | §10.3, Fig. 10 | +| radix histogram (count replication) | 2.55× | bandwidth-saturated | §10.4, Fig. 11 | +| LSB radixsort | 2.2× | saturated | §10.5.1 | +| hash join, no-partition / min-partition / max-partition | 1.05× / 1.25× / 3.3× | — | §10.5.1, Fig. 15 | + +The abstract's headline — "up to an order of magnitude faster than the +state-of-the-art scalar and vector approaches" — is the top-left cell +of that table, not a general result. ## Questions for notes.md -1. Why does branchless lose to branchy at 99% selectivity in their - data (hint: store traffic — branchless writes EVERY element)? -2. Vertical probing needs W independent probes in flight. What does - that do to the ORDER of join output, and which downstream - operators care (topic 11's sort-sensitivity)? -3. Their bloom-filter kernel is probe-minus-refill — why do bloom - lookups vectorize even better than hash probes (fixed iteration - count)? -4. The paper predates AVX-512 on servers; they emulate compress via - permutation LUTs — exactly simdjson's arm64 trick. Compare table - sizes: 8-lane f32 LUT vs simdjson's 8-byte LUT. -5. For M17: rank the four operators by expected engine-level win in - our Cypher pipeline (filter, probe, partition, bloom) given M11's - profile — where does Amdahl bite first? +1. Recompute Step 3's mispredict bound from the 25 % and 75 % rows of + `notes.md` instead of the 50 % row. A random branch taken with + probability p mispredicts with probability 2p(1-p) under a simple + predictor; does the implied cycles-per-miss stay near 19, and what + does it mean if it does not? +2. Step 2's LUT table has three rows. Your NEON kernel needs the + 256 B one. Work out what changes if you compact `f64` (W = 2) or + `u8` (W = 16) lanes instead, and say at which W you would switch to + simdjson's split-the-mask-in-half trick. +3. §4's Algorithm 3 buffers indexes rather than values. At 1 % + selectivity with a 4-byte key and a 32-byte payload, compute the + bytes touched per input row for both designs, and check the answer + against §10.1's claim that "avoiding payload column accesses + dominates low selectivities". +4. Vertical probing is "no longer stable" (§5.1). Which downstream + operators in a Cypher pipeline care about tuple order, and which + can be told not to (topic 11's selection vectors)? +5. §10.3's Bloom filter speedup is the largest in the table. The + mechanism is that a Bloom probe aborts as soon as one bit-test + fails (§6). Explain why that makes vectorisation *easier* here and + harder for the hash probe of Step 4. +6. For M17: rank the four operators by expected engine-level win in + our Cypher pipeline, using the table above *and* Step 5's rule + about random memory access. Where does Amdahl bite first? ## Done when -- [ ] You can name the two primitives — selective store, selective load/gather — and say which operators need which. -- [ ] You can explain why branchless loses to branchy at extreme selectivities in the paper, and check that against this topic's measured sweep, where branchless never actually loses (0.95 vs 10.04 GB/s at 50%). -- [ ] You can describe vertical hash probing with W probes in flight and say what it requires of the hash table. -- [ ] You can state the gather cost model: parallel instructions, serial memory. -- [ ] You can say why scatter needs conflict detection during partitioning. -- [ ] You wrote answers to all five questions in notes.md, including your ranking of the four operators by expected engine-level win. +Answer each before unfolding it. + +- [ ] You can name the two primitives and their indexed cousins, and say how to emulate them on a machine that has none. + +
Answer + + Selective store (compress, §3 Fig. 1) and selective load (expand, + Fig. 2); gather (Fig. 3) and scatter (Fig. 4) are the indexed + versions. Emulation (§3): extract the lane mask as a bitmask, index a + pre-generated permutation table with it, permute the vector so the + active lanes are contiguous, then store unaligned (for a store) or + load-and-blend (for a load). The table is `2^W` entries of W lane + indices — 256 B at W = 4, 2 KB at W = 8, 1 MB at W = 16. + +
+ +- [ ] You can state what a mispredicted branch costs on *your* machine, and show the arithmetic. + +
Answer + + The paper gives no figure — §1 says only "several cycles" — so derive + it. From `notes.md`, at 50 % selectivity over N = 4M f32 + (16,777,216 B): branchy 1.19 GB/s = 14.098 ms, branchless + 12.73 GB/s = 1.318 ms. Per element that is 3.361 ns vs 0.314 ns, a + gap of 3.047 ns ≈ 12.43 cycles at the ≥ 4.08 GHz clock derived in + `reading-simsimd.md` Step 2; a coin-flip branch misses + about half the time, so **≲ 25 cycles per mispredict**. It is an + upper bound because branchless also stores every element. + +
+ +- [ ] You can explain the selection-scan result *with* its machine, and say how your own sweep differs from the paper's. + +
Answer + + §10.1: on Xeon Phi scalar is "almost an order of magnitude slower + than vector code"; on Haswell "vector code is about twice faster", + and all the vector variants tie by saturating bandwidth while the + branchless scalar catches up at 10 % selectivity. Algorithm 3's real + trick is buffering qualifier *indexes* so payload columns are skipped + at low selectivity. + + On this Mac (`notes.md`) branchy collapses from 10.95 GB/s at 1 % to + 1.19 GB/s at 50 % while branchless stays flat within ±5 % — and + branchy **never wins**, not even at 1 % or 99 %, because the + crossover needs selectivity below 1 % on this core. + +
+ +- [ ] You can describe vertical hash probing, and say what it costs you. + +
Answer + + §5.1's Algorithm 5 runs W independent probes, one per lane, each with + its own bucket offset: selectively load new keys into finished lanes, + gather one table slot per lane, compare for hit and for EMPTY, + selectively store the matched payloads, advance the offset only where + not done. It never stalls on its slowest lane. + + The price, stated in §5.1: lanes are reused dynamically, so the probe + input is read out of order and "the probing algorithm is no longer + stable". Contrast hashbrown, which is SIMD *within* a single probe + (8 control bytes per group on aarch64). + +
+ +- [ ] You can state the gather cost model and use it to predict where vectorisation will not help. + +
Answer + + §3: gathers and scatters are not really parallel, because the L1 + cache serves only one or two distinct accesses per cycle; "random + memory accesses have to be excluded from the O(f(n)/W) + vectorization rule". So a W-lane gather costs about W cache accesses + and the vector win comes only from the instructions around it. This + predicts exactly §10.2's result: up to 6× on Xeon Phi's weak in-order + cores, "a smaller speedup for cache resident hash tables on Haswell", + and nothing at all once the table exceeds cache. + +
+ +- [ ] You can explain why scatter needs conflict serialisation, and say what the paper actually implemented. + +
Answer + + Two lanes with the same partition digit read the same offset counter + and scatter to the same address; §3's semantics say the rightmost + wins, so a row is lost and the counter is short by k-1. + + The paper does **not** use `vpconflictd` — §5.1 names it as future + AVX-3 hardware unavailable on its machines. §7.3's Algorithm 13 does + it with the primitives: reverse the lanes, scatter unique per-lane + values, gather them back, treat mismatches as conflicts, bump those + lanes' offsets, repeat. Total distinct memory accesses is always W. + The reversal keeps the partition stable, which LSB radixsort needs. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including your ranking of the four operators. + +
Answer + + Self-check. Question 6 has a defensible shape rather than one right + answer: selection scan first (pure instruction work, this topic + measures a 10× branchy/branchless gap), Bloom filter second (§10.3's + largest speedups, and no gather if the filter is cache-resident), + hash probe third (Step 5's rule caps it once the table leaves cache), + partition last on a machine with no scatter — where topic 13's + write-combining buffers are the real answer. + +
## References **Papers** -- Polychroniou, Raghavan, Ross — "Rethinking SIMD Vectorization for - In-Memory Databases" (SIGMOD 2015) — §3 the gather cost model, - §4 selection, §5 probe, §6 partition; skim the AVX-512 forecast - knowing it came true +- Orestis Polychroniou, Arun Raghavan, Kenneth A. Ross — "Rethinking + SIMD Vectorization for In-Memory Databases", *SIGMOD 2015*. + — §3 for the four + primitives, their emulation and the gather cost model; §4 for + selection scans (Algorithms 1-3); §5.1 for vertical hash probing + (Algorithm 5) and its stability cost; §6 for Bloom filters; §7.3 for + conflict serialisation (Algorithm 13); Table 1 for the three machines + and §10.1-§10.5 for the per-operator speedups. + +**Code** +- This topic's `experiments/src/filter.rs` — Algorithms 1 and 2 of §4, + verbatim, plus the NEON rungs you are asked to write. +- `reading-polars-compute.md` — the AVX-512 `vpcompressd`/`vpcompressb` + the paper forecast, now shipping, and the scalar path your machine + actually runs instead. diff --git a/topics/17-simd/reading-simdjson.md b/topics/17-simd/reading-simdjson.md index f53f731..c917fc0 100644 --- a/topics/17-simd/reading-simdjson.md +++ b/topics/17-simd/reading-simdjson.md @@ -1,205 +1,767 @@ # simdjson: parsing without branches Parsing — the most branchy code imaginable — rebuilt as branch-free -bitmask algebra over 64-byte blocks, at gigabytes per second. Before -the paper and the headers, this chapter builds the tricks one at a -time: why branches kill parsers, how masks replace them, the nibble -lookup, the carry-less multiply, the backslash-parity dance, and the -over-write/under-advance flatten. Read the paper alongside -`include/simdjson/arm64/` (you're on ARM — the NEON implementation -is the one your machine runs). Every trick here transfers to a DB +bitmask algebra over 64-byte blocks. Before the paper and the headers, +this chapter builds the tricks one at a time: why branches kill +parsers, how masks replace them, the operator table, the prefix-XOR +ladder, the backslash-parity subtraction, the over-write/under-advance +flatten, and NEON's emulated compress. Every trick transfers to a DB engine: RESP framing, CSV ingest, LIKE prefilters. +Every anchor below is simdjson at the pinned revision +`simdjson/simdjson@c783809` (`resources/codebases.md`), quoted with the +line numbers the code occupies in that revision. Your machine is +aarch64, so the files that actually run are `include/simdjson/arm64/` +and `src/arm64.cpp` — and several of them do **not** work the way the +x86 files (or the 2019 paper) describe. Where they differ, this guide +gives you both and says which one your CPU executes. + ## The problem in one sentence -A conventional JSON parser branches on every input byte, and since -JSON bytes are unpredictable, each mispredicted branch costs -~15 cycles — capping parsers around a few hundred MB/s while the -memory system could deliver tens of GB/s; simdjson closes that ~10× -gap by removing the branches. +A conventional JSON parser asks a data-dependent question of every +input byte, so it takes a mispredicted branch on unpredictable data +roughly once per byte; simdjson replaces the per-byte question with +per-64-byte **bitmask** arithmetic that has no data-dependent branches +at all, and only lets the branchy code see the few bytes that matter. + +How expensive is one such branch? Rather than quoting a number, use +this topic's own measurement. `notes.md`'s filter sweep (Apple +Silicon, measured 2026-07-10, N = 4M f32 = 16,777,216 input bytes) has +the branchy compaction loop at **1.19 GB/s** at 50 % selectivity and +the branchless one at **12.73 GB/s**: + +``` + branchy 16,777,216 B / 1.19e9 B/s = 14.10 ms + branchless 16,777,216 B / 12.73e9 B/s = 1.32 ms + per element (4,194,304 of them): 3.361 ns vs 0.314 ns + gap = 3.047 ns/element + clock (derived, see reading-simsimd.md Step 2): >= 4.08 GHz + 3.047 ns x 4.08 GHz = 12.4 cycles per element + half the elements mispredict at 50 % sel ⇒ ≈ 25 cycles per miss + (an upper bound: it charges the entire gap to mispredicts) +``` + +(The clock is not asserted; it is solved for from this topic's own +naive dot rung — 10.89 GB/s over 8 bytes per element-pair on a +3-cycle FMA chain — in `reading-simsimd.md` Step 2. The host is an +Apple M5, `sysctl -n machdep.cpu.brand_string`.) + +The paper is deliberately vaguer — §1 says only "several cycles of +penalty due to a mispredicted branch" — and that vagueness is correct, +because the penalty is a property of the pipeline you are on. What is +not vague is the shape: an order of magnitude, and it lands squarely +in the middle of the selectivity range. ## The concepts, step by step ### Step 1 — why parsers are slow: one branch per unpredictable byte -SIMD (single instruction, multiple data — one CPU instruction -operating on a whole vector of values at once; NEON, ARM's SIMD -instruction set, uses 128-bit vectors = 16 bytes per instruction) is -useless for a textbook parser, because a parser is a chain of -data-dependent branches: `if byte == '"' … else if byte == '{' …`. -The CPU predicts every branch and speculates ahead; when input bytes -are effectively random, it mispredicts constantly at ~15 cycles a -miss. That serializes execution around unpredictable data — the same -disease as topic 17 README §2's failure #2, at one-byte granularity. +> **In:** a byte stream and a parser written as `if (c == '"') … else +> if (c == '{') …`. +> **Out:** an understanding of why that shape caps at hundreds of +> MB/s no matter how wide the machine's vectors are. + +**SIMD** (single instruction, multiple data) is one CPU instruction +operating on a whole vector of values at once; **NEON** is ARM's SIMD +instruction set, with 128-bit vectors — 16 bytes, or four f32, per +instruction. SIMD is useless for a textbook parser, because a textbook +parser is a chain of data-dependent branches. The CPU predicts each +one and speculates past it; JSON bytes are effectively random from the +predictor's point of view, so it is wrong constantly, and each miss +throws away the pipeline's speculative work. + +That is the same disease as this topic's README §2 failure #2 +("data-dependent control flow"), at one-byte granularity — and it is +the same curve `notes.md` measures for `filter`, where the branchy +lane collapses 9× at 50 % selectivity while the branchless lane stays +flat within ±5 % across the whole sweep. + +The paper frames the goal as a *cost model* rather than a speed: +§3 promises "a fixed number of instructions per input byte" for the +quoted-string detection, with no data-dependent branch at all. Fixed +cost per byte is what makes a parser's throughput a property of the +machine instead of a property of the document. ### Step 2 — the fix: classify 64 bytes into bitmasks, branch once per block -simdjson's core move: process input in 64-byte blocks and convert -every per-byte question into a **bitmask** — a u64 where bit i -answers the question for byte i ("is byte i a quote?" → one bit). -Questions about bytes become bit arithmetic on whole blocks, which -has no branches at all. The architecture splits in two: +> **In:** 64 input bytes. +> **Out:** a handful of `uint64_t` masks, bit *i* answering one yes/no +> question about byte *i*, computed with zero data-dependent branches. + +The core move: process input in 64-byte blocks and turn every per-byte +question into a **bitmask** — a `uint64_t` in which bit *i* answers the +question for byte *i*. "Is byte *i* a quote?" becomes one bit; the +answers for a whole block become one register; and questions about +bytes become bit arithmetic, which has no branches. + +The architecture splits in two: ``` stage 1: structural indexing (SIMD, branch-free) - 64 input bytes → classify → bitmasks (one bit per byte): - quotes, backslashes, whitespace, operators {}[]:, - → resolve strings (quote parity) → structural positions - → flatten bit positions into an index array - stage 2: tape building (branchy, but only touches ~1/8 of bytes) + 64 input bytes -> classify -> masks (one bit per byte): + op {}[]:, whitespace, backslash, quote + -> escape parity -> real quotes -> in-string regions + -> structural positions -> flatten bits into an index array + stage 2: tape building (branchy, but only visits flagged bytes) walk the structural indexes, parse numbers/strings, emit tape ``` -Stage 1 never branches on DATA — the only branches are the loop. -Stage 2 stays branchy but only visits the ~1/8 of bytes stage 1 -flagged as structural, so Amdahl's law works *for* you: the branchy -part shrank 8×. - -### Step 3 — classification by nibble LUT (`lookup_16`) - -The first mask to build: which of these 16 bytes are `{ } [ ] : ,` -or whitespace? The tool is `vqtbl1q_u8` — NEON's table-lookup -instruction, which uses each byte of one vector as an index into a -16-entry table (a LUT — lookup table) held in another vector: 16 -parallel table lookups in one instruction. One byte is too big for a -16-entry table, so split it into **nibbles** (4-bit halves, values -0–15): look up the high nibble in one table, the low nibble in -another, and AND the results. Any predicate expressible as -(hi-nibble class) ∧ (lo-nibble class) costs 2 shuffles + 1 AND for -16 bytes — versus 16 branchy comparisons. Question: build the two -tables that classify `{ } [ ] : ,` — why do hi and lo tables -disagree on false positives, and why does ANDing fix it? - -### Step 4 — quote parity by carry-less multiply (`prefix_xor`) - -Knowing where quotes are isn't enough — a byte is *inside a string* -if it's preceded by an odd number of quotes. That's a running -(prefix) parity over 64 positions: naively a serial loop, the exact -dependency chain SIMD can't do. The trick: `prefix_xor(m)` computes, -for each bit position, the XOR of all lower bits — exactly "odd -quote count so far" — in ONE `PMULL` instruction (carry-less -multiply: binary multiplication where additions are XORs, i.e. -multiplication in GF(2); multiplying by all-ones makes every output -bit the XOR of all inputs below it). One instruction turns the -quote mask into an in-string region mask. Question: why is -escaped-quote handling (backslash runs) done BEFORE this, and why -does odd/even backslash parity need its own trick (the -odd_sequence_starts dance)? - -### Step 5 — the escaped-backslash problem - -`\\\"` vs `\\\\"` — whether a quote is real depends on the PARITY of -the preceding backslash run: `\"` escapes the quote, `\\"` doesn't -(the backslash escaped itself). Run-length parity is again -inherently sequential-looking — and the scanner solves it with -add-carry propagation on masks: adding `backslash_starts` to the run -mask makes the carry ripple through each run of 1-bits and pop out -at the end, landing on an odd or even position depending on the -run's length. Branch-free parity-of-run-length via the adder's carry -chain. This is the paper's cleverest three lines — work the example -in §3.1.1 by hand. - -### Step 6 — flatten_bits: over-write, under-advance (`bit_indexer`) - -Stage 2 wants an array of positions, not a mask. Turning a 64-bit -mask into positions: `cnt = popcnt` (population count — how many -bits are set), then repeatedly `trailing_zeros` (index of the lowest -set bit) + clear that bit — unrolled by 8 with all 8 slots written -UNCONDITIONALLY, advancing the output cursor by the real count only: - -```rust -// bit_indexer: mask → positions. Over-write, under-advance. -fn flatten(out: &mut [u32], n: usize, start: u32, mut m: u64) -> usize { - let cnt = m.count_ones() as usize; - let mut k = 0; - while k < cnt { // ceil(cnt/8) iterations, branch-free body - for j in 0..8 { // write 8 UNCONDITIONALLY — - out[n + k + j] = start + m.trailing_zeros(); // garbage lanes are fine - m &= m.wrapping_sub(1); // clear lowest set bit - } - k += 8; - } - n + cnt // advance by the REAL count only -} -``` - -Writing garbage past the real count costs a few redundant stores; -branching on the exact count would cost a mispredict. Same shape as -our branchless filter append — over-write, under-advance is THE -selection-kernel idiom. Question: why is writing 8 always faster -than writing exactly cnt? +Why 64 and not NEON's 16? Because the mask is the unit of work, and a +mask is a general-purpose register. The classifier still runs at +16 bytes per `vqtbl1q_u8`; it just runs four times and glues the four +16-bit results into one 64-bit mask (Step 3). The block size is set by +the *mask* width, not the *vector* width — which is why the arm64 +kernel and the westmere (SSE) kernel both use 64 while haswell (AVX2) +and icelake (AVX-512) use 128: + +| kernel | block size | anchor | +|---|---|---| +| arm64 (your machine) | 64 | `src/arm64.cpp:126` — `stage1::json_structural_indexer::index<64>` | +| westmere (SSE4.2) | 64 | `src/westmere.cpp:140` | +| haswell (AVX2) | 128 | `src/haswell.cpp:135` | +| icelake (AVX-512) | 128 | `src/icelake.cpp:181` | + +The 128-byte variant is not a wider mask — it is *two* 64-byte blocks +pipelined together for instruction-level parallelism, which +`json_structural_indexer.h:220-229` (`step<128>`) spells out and the +PERF NOTES at `json_structural_indexer.h:176-191` justify. + +### Step 3 — classification: how arm64 really does it (not nibble tables) + +> **In:** four `uint8x16_t` chunks holding 64 bytes. +> **Out:** two `uint64_t` masks — `op` (is byte *i* one of `,:[]{}`?) +> and `whitespace` — costing a fixed 8 vector instructions each. + +The tool is `vqtbl1q_u8`: NEON's table-lookup instruction, which uses +each byte of one vector as an index into a **16-entry table** (a LUT) +held in another vector — sixteen parallel lookups in one instruction. +A byte's value is 0–255 and the table has 16 entries, so *something* +has to shrink the index. + +The famous textbook answer is "split the byte into **nibbles** (4-bit +halves) and AND two lookups." That is what the paper describes in +§3.1.2, and it is what the x86 kernel does (`src/haswell.cpp:43-72` +folds the high bits with `| 0x20` and looks up the low nibble). It is +**not** what your kernel does. `src/arm64.cpp:40-80` uses a single +table and an offset shift: + +```c +// src/arm64.cpp:41-56 — json_character_block::classify (op half only) + 41 const uint8x16_t op_table = simd8( + 42 0xff, 0, ',', ':', 0, '[', ']', '{', '}', 0, 0, 0, 0, 0, 0, 0 + 43 ); +// ... 44-52: ws_table and the four 16-byte chunks d0_0 .. d0_3 ... + 53 const uint8x16_t match_op_0 = vceqq_u8(vqtbl1q_u8(op_table, vshrq_n_u8(vaddq_u8(d0_0, vdupq_n_u8(3)), 4)), d0_0); +``` + +Read line 53 inside out: add 3 to every byte, shift right by 4 (so the +index is `(b + 3) >> 4`, a value in 0–15), look that up in `op_table`, +and compare the *result* back against the original byte. A lane +matches only if the table entry for its bucket **is** the byte itself. + +Work the arithmetic for all six structural characters, and for one +near miss: + +``` + byte hex b+3 (b+3)>>4 op_table[i] equals b? + ',' 0x2C 47 2 ',' yes + ':' 0x3A 61 3 ':' yes + '[' 0x5B 94 5 '[' yes + ']' 0x5D 96 6 ']' yes + '{' 0x7B 126 7 '{' yes + '}' 0x7D 128 8 '}' yes + '-' 0x2D 48 3 ':' (0x3A) NO <- rejected + 'a' 0x61 100 6 ']' (0x5D) NO <- rejected +``` + +The `+3` is the whole trick: without it `[` (0x5B) and `]` (0x5D) land +in the same bucket 5, and `{` (0x7B) and `}` (0x7D) both land in 7. +Adding 3 pushes `]` into bucket 6 and `}` into bucket 8, giving each of +the six characters a private bucket. The final `vceqq_u8` is what makes +the collisions harmless: every other byte that shares a bucket gets +compared against a character it is not. + +Whitespace uses a different instruction on line 58 — `vqtbx1q_u8`, +table-lookup-*with-fallback*: indices ≥ 16 leave the destination +untouched instead of writing zero. So `ws_table` (`arm64.cpp:44-46`, +`0xff` at indices 9, 10 and 13) catches `\t` (0x09), `\n` (0x0A) and +`\r` (0x0D) directly by byte value, and space (0x20, index 32 ≥ 16) +falls through to the destination operand — which is +`vceqq_u8(d, vdupq_n_u8(' '))`, computed for exactly that purpose. +One instruction, four whitespace characters, no nibbles. + +Then 64 lanes of `0x00`/`0xFF` have to become 64 bits. x86 has one +instruction for this (`PMOVMSKB`); NEON does not, so `arm64.cpp:63-77` +ANDs each lane with a repeating `0x01,0x02,…,0x80` pattern and folds +with a tree of `vpaddq_u8` (pairwise add): 4 ANDs + 4 pairwise adds +per mask, halving the lane count each time — 64 lanes → 32 → 16 → 8 +bytes = one `uint64_t`, extracted at line 76. Count it: **8 vector +instructions to produce one 64-bit mask**, against x86's four +`PMOVMSKB` plus three shift-or. This is the third distinct answer to +"one bit per lane" you will meet in this topic; hashbrown's and +memchr's are the other two, and neither matches this one. + +### Step 4 — quote parity: prefix-XOR, and why it is *not* PMULL here + +> **In:** `quote`, a mask with a 1 at every real (unescaped) quote. +> **Out:** `in_string`, a mask with a 1 at every byte that lies inside +> a string — computed without a loop over the 64 positions. + +Knowing where quotes are is not enough: a byte is inside a string if +an **odd** number of quotes precede it. That is a running parity over +64 positions — a serial scan, the exact shape SIMD cannot do. +`prefix_xor(m)` computes, for every bit position, the XOR of all lower +bits, which is precisely "odd number of quotes so far". + +The paper's answer (§3.1.1) is one carry-less multiply: multiplying the +mask by all-ones in GF(2) makes each output bit the XOR of all input +bits below it, and the paper quotes `pclmulqdq` at 7 cycles latency, +1 per cycle throughput on Skylake. That is a real instruction and the +x86 kernels use it — +`include/simdjson/westmere/bitmask.h:22` (and the haswell and icelake +copies) call `_mm_clmulepi64_si128`. + +Your kernel does something else, and says why: + +```c +// include/simdjson/arm64/bitmask.h:17-38 — prefix_xor on aarch64 + 17 simdjson_inline uint64_t prefix_xor(uint64_t bitmask) { + 19 // We could do this with PMULL, but it is apparently slow. +// ... 21-23: the vmull_p64 version, commented out ... + 24 // Analysis by @sebpop: +// ... 25-27: the eors interleave with vector code, so their latency hides ... + 28 // Also the PMULL requires two extra fmovs: GPR->FP (3 cycles in N1, 5 cycles in A72 ) + 29 // and FP->GPR (2 cycles on N1 and 5 cycles on A72.) + 31 bitmask ^= bitmask << 1; + 32 bitmask ^= bitmask << 2; + 33 bitmask ^= bitmask << 4; + 34 bitmask ^= bitmask << 8; + 35 bitmask ^= bitmask << 16; + 36 bitmask ^= bitmask << 32; + 37 return bitmask; + 38 } +``` + +Six shift-XOR steps, because log₂ 64 = 6: after the `<< 1` step each +bit holds the XOR of itself and its neighbour; after `<< 2`, of a span +of 4; after `<< 32`, of all 64. Check the doc comment's own example at +line 15 by hand — `prefix_xor(0b00100100) == 0b00011100` — and note +that the ladder runs in **general-purpose registers**, which is exactly +@sebpop's argument: the GPR units are idle while the FP side does the +classification, so six cheap integer ops on the idle side beat one +"fast" FP instruction that needs a 3-cycle `fmov` in and a 2-cycle +`fmov` out (N1 numbers; 5 and 5 on A72). + +The lesson is the one SimSIMD's FCMLA comment makes independently: a +specialised instruction has to beat the *whole sequence including the +data movement to reach it*, not just the arithmetic. + +Where it is used (`src/generic/stage1/json_string_scanner.h:62-85`): + +```c +// src/generic/stage1/json_string_scanner.h:62-78 — one block of string state + 62 simdjson_really_inline json_string_block json_string_scanner::next(const simd::simd8x64& in) { + 63 const uint64_t backslash = in.eq('\\'); + 64 const uint64_t escaped = escape_scanner.next(backslash).escaped; + 65 const uint64_t quote = in.eq('"') & ~escaped; +// ... 67-72: comment explaining the xor with the carry-in ... + 73 const uint64_t in_string = prefix_xor(quote) ^ prev_in_string; +// ... 75-77: comment ... + 78 prev_in_string = uint64_t(static_cast(in_string) >> 63); +``` + +Line 78 is the cross-block carry: an *arithmetic* right shift by 63 +smears the top bit across all 64, producing 0 or `~0`, which line 73 +XORs into the next block. Blocks stay independent except for one bit. + +### Step 5 — the escaped-backslash problem: a subtraction, not an addition + +> **In:** `backslash`, a mask of every `\` in the block. +> **Out:** `escaped`, a mask of every byte that a backslash escapes — +> so Step 4's `quote` can exclude `\"`. + +Whether a quote is real depends on the **parity of the backslash run** +before it: in `\"` the backslash escapes the quote, in `\\"` the two +backslashes escape each other and the quote is real. Run-length parity +looks inherently sequential. + +The paper's Fig. 3 (§3.1.1) resolves it with two *additions*, letting +an adder's carry ripple the length of each run and pop out at its end. +The shipped code does it with **one subtraction** and the constant +`ODD_BITS`: + +```c +// src/generic/stage1/json_escape_scanner.h:127-142 — next_escape_and_terminal_code + 127 uint64_t maybe_escaped = potential_escape << 1; +// ... 129-133: comment — bring in all odd bits, for speed ... + 134 uint64_t maybe_escaped_and_odd_bits = maybe_escaped | ODD_BITS; + 135 uint64_t even_series_codes_and_odd_bits = maybe_escaped_and_odd_bits - potential_escape; +// ... 137-141: comment — flip the odd bytes back ... + 142 return even_series_codes_and_odd_bits ^ ODD_BITS; +``` + +`ODD_BITS` is `0xAAAAAAAAAAAAAAAA` (`json_escape_scanner.h:74`) — every +odd-numbered bit. Line 135's borrow chain is the engine: subtracting +the run's own bits from a field of alternating 1s propagates a borrow +the length of the run, and where it stops encodes the run's parity. + +Do it on paper in 8 bits, with `ODD_BITS = 0b10101010 = 0xAA` and bit 0 +as the leftmost character: + +``` + input `\\\n` (odd run of 3, then 'n') input `\\n` (even run of 2) + potential_escape = 0b00000111 = 0x07 0b00000011 = 0x03 + maybe_escaped = 0b00001110 = 0x0E 0b00000110 = 0x06 + | ODD_BITS = 0b10101110 = 0xAE 0b10101110 = 0xAE + - potential_esc = 0b10100111 = 0xA7 0b10101011 = 0xAB + ^ ODD_BITS = 0b00001101 = 0x0D 0b00000001 = 0x01 +``` + +Then `escaped = escape_and_terminal_code ^ backslash` +(`json_escape_scanner.h:67`, with the carry-in bit folded in): + +``` + odd run: 0x0D ^ 0x07 = 0x0A = bits 1,3 -> the 2nd backslash and the 'n' + even run: 0x01 ^ 0x03 = 0x02 = bit 1 -> only the 2nd backslash; 'n' is FREE +``` + +That is the whole answer: in `\\\n` the `n` is escaped, in `\\n` it is +not, and the difference fell out of one subtract. `escape` (line 68) +is the complementary mask of backslashes that *do* escape something, +and line 69 carries its top bit into the next block, exactly as Step 4 +carries `in_string`. + +Note the short circuit at `json_escape_scanner.h:53`: if the whole +block has no backslash, all of this is skipped. That is a +data-dependent branch — a well-predicted one, since most JSON blocks +contain no backslash at all. + +### Step 6 — flatten: over-write, under-advance (and STEP is 4, not 8) + +> **In:** a 64-bit `structural` mask and the block's base index. +> **Out:** `cnt` 32-bit positions appended to an array, with more than +> `cnt` slots written and only `cnt` counted. + +Stage 2 wants positions, not a mask. The classic loop is +`while (bits) { *out++ = idx + ctz(bits); bits &= bits - 1; }` — and +the paper is explicit (§3.1.4) that this "introduces an unpredictable +branch; unless there is a regular pattern in our bitsets, we would +expect to have at least one branch miss for each word." + +The paper's fix (Fig. 6) is to extract **8** indexes unconditionally +and overwrite the excess on the next iteration: "as long as the +frequency of our set bits is below 8 bits out of 64 we expect few +unpredictable branches," and the paper calls 8 "a heuristic based on +our experience with JSON documents." + +The shipped code has moved on. It writes in steps of **4**, up to 24, +then falls back to a scalar loop: + +```c +// src/generic/stage1/json_structural_indexer.h:93-121 — bit_indexer::write + 93 simdjson_inline void write(uint32_t idx, uint64_t bits) { +// ... 94-96: comment — this branch is sometimes mispredicted, sometimes vital ... + 97 if (bits == 0) + 98 return; + 100 int cnt = static_cast(count_ones(bits)); + 103 bits = reverse_bits(bits); // #if SIMDJSON_PREFER_REVERSE_BITS + 108 static constexpr const int STEP = 4; + 110 static constexpr const int STEP_UNTIL = 24; + 112 write_indexes_stepped<0, STEP_UNTIL, STEP>(idx, bits, cnt); +// ... 113-119: scalar tail for cnt > 24, marked simdjson_unlikely ... + 121 this->tail += cnt; +``` + +Line 112 writes 4 slots at a time and only checks `cnt` every 4 +(`write_indexes_stepped` at lines 71-80 recurses while +`START+STEP < cnt`); line 121 advances the cursor by the **real** +count. Slots past `cnt` hold garbage that the next block silently +overwrites — the doc comment at lines 85-86 says the buffer must be +oversized for exactly this reason. **Over-write, under-advance.** + +Line 103 is another aarch64-specific choice. `write_index` has two +bodies: the ARM one at lines 44-48 uses `leading_zeroes` + +`zero_leading_bit`, the x86 one at lines 56-59 uses `trailing_zeroes` + +`clear_lowest_bit`. The comment at lines 31-43 gives the reason — +"ARM lacks a fast trailing zero instruction, but it has a fast bit +reversal instruction and a fast leading zero instruction" — so the +mask is reversed **once** (line 103) and then consumed from the top. +`include/simdjson/arm64/bitmanipulation.h:76` is what sets +`SIMDJSON_PREFER_REVERSE_BITS` to 1 for your build. + +Now compute whether STEP=4 is enough for a real document. The paper's +Table 5 gives bytes-per-structural for each test file; convert to bits +set per 64-byte mask by dividing 64 by it: + +``` + twitter 11.4 B/structural -> 64/11.4 = 5.6 set bits per 64-bit mask + gsoc-2018 43.9 -> 64/43.9 = 1.5 + citm_catalog 12.7 -> 64/12.7 = 5.0 + marine_ik 4.6 -> 64/4.6 = 13.9 <- exceeds 8, and 4, and needs 24 + canada 6.7 -> 64/6.7 = 9.6 +``` + +For twitter the paper's 8-wide unconditional write covers 5.6 on +average and the shipped 4-wide covers it in two rounds with one check; +for marine_ik neither does, and the `STEP_UNTIL = 24` ceiling is what +keeps the tail loop rare. The paper's own framing (§3.1.4) is that a +wider unconditional extraction is "more expensive due to having to use +more operations, but even less likely to cause a branch miss" — the +2026 code has simply re-tuned that trade-off downward, presumably +because the check at every 4 is cheaper than 4 extra writes. ### Step 7 — compress on NEON: the missing instruction, emulated -Sometimes stage 1 needs to *compact* the surviving bytes themselves -(keep the lanes where the mask is set, packed left). AVX-512 has an -instruction for this (`vpcompress`); NEON does not. simdjson's -emulation (arm64/simd.h:267-276): take the mask's 4-bit chunks, -index a precomputed LUT of shuffle patterns, and feed that pattern -to `vqtbl1q` (Step 3's table lookup, now used as a byte-shuffler) — -compaction as a lookup-then-shuffle, 8 lanes per shuffle. This is -NEON's missing vpcompress, and it's exactly what our `filter.rs` -NEON compact stub reimplements for f32. - -### Step 8 — what transfers to a DB engine - -- RESP protocol framing (M7) = structural indexing over `\r\n$*:+-` -- CSV/JSON bulk ingest = the whole pipeline -- string-escape scanning = LIKE/regex prefilters -- the meta-lesson: turn per-byte branches into per-block masks, - THEN branch once per block (topic 11's vectorization, byte - edition) +> **In:** 16 bytes in a vector plus a 16-bit mask. +> **Out:** the *unmasked* bytes packed to the left, one 16-byte store, +> with only `16 - popcount(mask)` of them meaningful. + +Sometimes stage 1 must compact surviving bytes, not just index them. +AVX-512 has `vpcompressb`/`vpcompressd` for this. NEON has nothing. +simdjson emulates it with two table lookups: + +```c +// include/simdjson/arm64/simd.h:246-277 — simd8::compress + 246 simdjson_inline void compress(uint16_t mask, L * output) const { +// ... 247-251: using-declarations and the two-halves comment ... + 252 uint8_t mask1 = uint8_t(mask); // least significant 8 bits + 253 uint8_t mask2 = uint8_t(mask >> 8); // most significant 8 bits + 257 uint64x2_t shufmask64 = {thintable_epi8[mask1], thintable_epi8[mask2]}; +// ... 258-265: reinterpret, and add 0x08 to the second half's indices ... + 267 uint8x16_t pruned = vqtbl1q_u8(*this, shufmask); + 270 int pop1 = BitsSetTable256mul2[mask1]; + 275 uint8x16_t compactmask = vld1q_u8(reinterpret_cast(pshufb_combine_table + pop1 * 8)); + 276 uint8x16_t answer = vqtbl1q_u8(pruned, compactmask); + 277 vst1q_u8(reinterpret_cast(output), answer); +``` + +Read the semantics off the doc comment at lines 238-241 before the +code, because they are inverted from what you expect: it "copies to +`output` all bytes corresponding to a **0** in the mask", and "only the +first `16 - count_ones(mask)` bytes of the result are significant but +16 bytes get written". A set bit means *drop this byte*, and the store +is unconditionally 16 wide — over-write, under-advance again, this time +in bytes rather than indexes. + +Why two halves? Because of the table size. Work it out: + +``` + one table for all 16 mask bits: 2^16 entries x 16-byte shuffle pattern + = 65,536 x 16 B = 1,048,576 B = 1 MB (blows every cache) + two tables of 8 bits each: 2^8 entries x 8-byte pattern + = 256 x 8 B = 2,048 B = 2 KB (internal/simdprune_tables.h:11) + plus the stitcher pshufb_combine_table[272] bytes (:13) + plus the doubled popcount BitsSetTable256mul2[256] (:11) +``` + +Two 8-bit halves cost 512× less table for one extra shuffle and one +extra lookup. Line 267 prunes each half independently (which leaves a +gap in the middle, because half 1 kept only `pop1/2` bytes); line 270 +reads the *doubled* popcount of the low half straight out of a table +rather than computing it; line 275 uses it to index a second shuffle +table that slides half 2 down onto half 1; line 276 applies it. Two +`vqtbl1q_u8`, three table reads, one store. + +`compress_halves` at `arm64/simd.h:283-299` is the 8-lane sibling, +using `vqtbl1_u8` (64-bit table) twice. It is the closest thing on +NEON to a per-8-lane compress, and it is what a `f32x4` compact in +`filter.rs` will end up shaped like — with the important difference +that for four 32-bit lanes the mask has only 16 possible values, so the +whole LUT is 16 × 16 = 256 bytes and no stitching is needed at all. + +### Step 8 — what stage 2 costs, and what transfers to a DB engine + +> **In:** the paper's regression model and dataset table. +> **Out:** a defensible answer to "does the branchy half kill the +> speedup?", computed rather than asserted. + +Stage 2 is still branchy. The Amdahl argument is usually waved at with +"it only touches 1/8 of bytes" — which is true for exactly one file. +Table 5 of the paper gives bytes-per-structural from **4.6** +(marine_ik) to **43.9** (gsoc-2018); the paper's own §3.1.4 phrasing is +"once every 40 characters or once every 4 characters." + +The paper's §4.3 regression (R² ≥ 0.99) lets you check the split +directly. On its Skylake machine, with `B` input bytes, `S` structural +characters and `F` floating-point numbers: + +``` + stage 1 = 1.7*S + 0.62*B cycles + stage 2 = 19*F + 8.7*S + 0.31*B + total = 19*F + 11*S + 0.92*B +``` + +Run it on twitter.json (Table 5: S = 55,264, F = 1; Table 6: +B = 631,514): + +``` + stage 1 = 1.7*55,264 + 0.62*631,514 = 93,948.8 + 391,538.7 = 485,487 cy + total = 19 + 11*55,264 + 0.92*631,514 + = 19 + 607,904 + 580,993 = = 1,188,916 cy + stage 1 share = 485,487 / 1,188,916 = 40.8 % + throughput = 631,514 B / (1,188,916 cy / 3.4e9 Hz) = 1.81 GB/s +``` + +Two things to take from that. First, 40.8 % matches §4.3's prose +("about half the CPU cycles per input byte — between 0.5 and 3 cycles +— are spent in stage 1"), so the branchy half is *not* a rounding +error; it is the larger half. Second, the model predicts 1.81 GB/s +where Table 10 measures **2.2 GB/s** for twitter — the regression is +fitted across all files and under-predicts this one by 18 %. Quote the +measurement, not the model. + +And now the headline, with its hardware attached, because it does not +transfer to your Mac. §4.1: **Intel i7-6700 Skylake at 3.4 GHz** +(3.7 GHz turbo), DDR4-2133, GCC 9.1 with `-O3 -march=native`, Linux. +§4.5: "our parser can achieve and even surpass 2 GB/s in six +instances, and for gsoc-2018, we reach 3 GB/s." Table 10, GB/s: + +| file | simdjson | RapidJSON | sajson | +|---|---|---|---| +| gsoc-2018 | 3.2 | 0.68 | 1.2 | +| citm_catalog | 2.5 | 0.72 | 1.1 | +| twitter | 2.2 | 0.55 | 0.83 | +| canada | 1.1 | 0.38 | 0.62 | +| marine_ik | 0.94 | 0.42 | 0.66 | + +The spread within simdjson's own column (0.94 to 3.2, a 3.4× range on +one CPU) is the real lesson: "gigabytes per second" is a property of +the *document's* structural density as much as of the parser. Your +machine has different vector widths, a different `prefix_xor`, a +different classifier and a different flatten step — treat every number +above as the paper's, not as a prediction. + +What transfers to the engine you are building: + +- RESP protocol framing (M7) = structural indexing over `\r\n$*:+-`; + the op-table trick of Step 3 generalises to any six-ish byte set. +- CSV / JSON bulk ingest = the whole pipeline. +- string-escape scanning = LIKE and regex prefilters. +- the meta-lesson: turn per-byte branches into per-block masks, then + branch once per block — topic 11's vectorization, byte edition. ## Where each step lives in the code +Every anchor is `simdjson/simdjson@c783809`. + | anchor | step | what it is | |---|---|---| -| arm64/simd.h:179 | 3 | `repeat_16` — build 16-byte LUTs | -| arm64/simd.h:226-229 | 3 | `lookup_16` = `vqtbl1q_u8` — the classification workhorse | -| arm64/bitmask.h:15-22 | 4 | `prefix_xor` — carry-less multiply (PMULL) turns quote bits into in-string regions | -| src/generic/stage1/json_string_scanner.h:16-30 | 4–5 | the string-state block: escaped/quote/in_string masks | -| src/generic/stage1/json_structural_indexer.h:24-28 | 6 | `bit_indexer` — flatten mask bits to positions | -| src/generic/stage1/json_structural_indexer.h:194 | 2 | the stage-1 driver loop | -| arm64/simd.h:267-276 | 7 | compress via pruned `vqtbl1q` + LUT — NEON's missing vpcompress, emulated | -| src/generic/stage1/utf8_lookup4_algorithm.h | 3 | UTF-8 validation as 3 table lookups (the nibble-LUT trick, applied thrice) | - -Reading route: the stage-1 driver loop first (see the block -pipeline whole), then simd.h's LUT machinery, then the string -scanner with the paper's §3.1.1 open beside it. In the paper, §3 is -stage 1 — the rest you can skim once the steps above are solid. +| `src/arm64.cpp:40-80` | 3 | `json_character_block::classify` — `(b+3)>>4` op table, `vqtbx1q_u8` whitespace, `vpaddq_u8` bit-gather | +| `src/haswell.cpp:43-72` | 3 | the x86 contrast: low-nibble table plus an OR with `0x20`, with its own false-positive note at 57-66 | +| `include/simdjson/arm64/simd.h:123-136` | 3 | `to_bitmask` — the AND-plus-`vpaddq` fold, as a reusable helper | +| `include/simdjson/arm64/bitmask.h:17-38` | 4 | `prefix_xor` — six shift-XOR steps, *not* PMULL, with the reason at 19-29 | +| `include/simdjson/westmere/bitmask.h:22` | 4 | the CLMUL version the paper describes (x86 only) | +| `src/generic/stage1/json_string_scanner.h:62-85` | 4 | `next()` — backslash → escaped → quote → `prefix_xor` → `in_string`, carry-out at 78 | +| `src/generic/stage1/json_escape_scanner.h:96-143` | 5 | `next_escape_and_terminal_code` — OR with `ODD_BITS`, subtract, XOR back | +| `src/generic/stage1/json_escape_scanner.h:50-71` | 5 | `next()` — the short circuit at 53 and the block carry at 69 | +| `src/generic/stage1/json_structural_indexer.h:93-121` | 6 | `bit_indexer::write` — `STEP = 4`, `STEP_UNTIL = 24`, `tail += cnt` | +| `src/generic/stage1/json_structural_indexer.h:44-48` | 6 | the ARM `write_index`: reverse once, then leading-zeroes | +| `src/generic/stage1/json_structural_indexer.h:209-247` | 2 | the driver loop, `step<128>` at 220 and `step<64>` at 231 | +| `src/arm64.cpp:126` | 2 | which one your CPU runs: `index<64>` | +| `include/simdjson/arm64/simd.h:246-278` | 7 | `compress` — two `thintable_epi8` halves + `pshufb_combine_table` stitch | +| `include/simdjson/arm64/simd.h:283-299` | 7 | `compress_halves` — the 8-lane sibling | +| `src/generic/stage1/utf8_lookup4_algorithm.h:44,60,88,104` | 3 | where the nibble-AND trick really lives: `byte_1_high & byte_1_low & byte_2_high` | + +Reading route: `src/arm64.cpp` first, because it is short (161 lines) +and shows which generic pieces your CPU instantiates. Then +`arm64/bitmask.h` (44 lines, one function). Then the string scanner +with the paper's §3.1.1 open beside it, then the escape scanner, then +the structural indexer. `arm64/simd.h` is a toolbox — read `compress` +and `to_bitmask`, skim the rest. ## Questions for notes.md -1. Why 64-byte blocks (one u64 mask = 64 lanes) rather than the - 16-byte NEON width? -2. The compress LUT at simd.h:267: how many entries, indexed by - what, and why does the same trick cap at 8 lanes per shuffle? -3. Stage 2 is still branchy. Why does Amdahl not kill the speedup - (what fraction of bytes reach stage 2)? -4. UTF-8 validation in 3 lookups: what property of UTF-8 error - patterns makes nibble tables sufficient? -5. For M7: sketch stage-1 masks for RESP (`*3\r\n$3\r\nSET...`) — - which characters are "structural"? +1. Step 2 shows arm64 and westmere both using 64-byte blocks while + haswell and icelake use 128. Since the mask is a `uint64_t` in every + case, what does `step<128>` actually buy (read the PERF NOTES at + `json_structural_indexer.h:176-191`), and why would that pay off on + AVX2 but not on NEON? +2. Redo Step 3's bucket table for a RESP framing classifier over + `\r \n $ * : + -` (0x0D, 0x0A, 0x24, 0x2A, 0x3A, 0x2B, 0x2D). Does + any constant offset `k` give all seven a private `(b+k)>>4` bucket? + If not, what is the smallest set of extra comparisons you need? +3. `prefix_xor` costs 6 dependent XOR-shift pairs on your machine. + That is a 12-instruction dependency chain per 64-byte block. At + 64 bytes per block, how many bytes/s does that chain alone permit if + each pair is 2 cycles and nothing overlaps — and why is the real + answer higher (re-read @sebpop's note at `bitmask.h:24-29`)? +4. The compress LUT arithmetic in Step 7 assumed byte lanes. Redo it + for `f32x4`: how many mask values, how big is the shuffle table, and + why does the two-halves stitching disappear? +5. For M7: sketch stage-1 masks for `*3\r\n$3\r\nSET\r\n...` — which + characters are structural, what is your bytes-per-structural, and + where does it sit in Step 6's table (closer to twitter or to + marine_ik)? ## Done when -- [ ] You can explain the stage-1 idea: classify 64 bytes into bitmasks, branch once per block instead of once per byte. -- [ ] You can explain quote parity by carry-less multiply and why prefix-xor is the right primitive. -- [ ] You can describe the escaped-backslash problem and why it cannot be solved with a single mask. -- [ ] You can explain `flatten_bits`'s over-write-and-under-advance trick and why it is safe. -- [ ] You can say why stage 2 stays branchy and give the Amdahl argument for why that is acceptable. -- [ ] You wrote answers to all five questions in notes.md, including stage-1 masks sketched for RESP. +Answer each before unfolding it. + +- [ ] You can explain the stage-1 idea — classify 64 bytes into bitmasks, branch once per block — and say what sets the block size. + +
Answer + + Every per-byte question becomes one bit of a `uint64_t`, so questions + about a whole block become branch-free bit arithmetic. The block size + is set by the **mask** width (a general-purpose register), not the + vector width: the arm64 classifier still works 16 bytes at a time and + runs four times per block. That is why arm64 (`src/arm64.cpp:126`) + and westmere (`src/westmere.cpp:140`) both use `index<64>` while + haswell (`:135`) and icelake (`:181`) use 128 — and the 128 variant + is two 64-byte blocks pipelined for ILP (`step<128>`, + `json_structural_indexer.h:220-229`), not a wider mask. + +
+ +- [ ] You can describe what the arm64 classifier actually does, and why it is *not* the paper's two-nibble AND. + +
Answer + + `src/arm64.cpp:53` indexes a single 16-entry `op_table` with + `(byte + 3) >> 4` and then compares the looked-up value back against + the original byte (`vceqq_u8`). The `+3` is what separates `[`/`]` + (0x5B/0x5D → buckets 5 and 6) and `{`/`}` (0x7B/0x7D → 7 and 8); the + final compare rejects every other byte that shares a bucket, e.g. + `-` (0x2D) lands in bucket 3 and is compared against `:`. + Whitespace uses `vqtbx1q_u8` at line 58 — table-lookup with fallback + — so `\t\n\r` come from `ws_table` and space falls through to a + `vceqq_u8(d, ' ')` destination. + + The two-nibble AND the paper describes in §3.1.2 is real, but it + lives in UTF-8 validation, and there it is **three** tables: + `utf8_lookup4_algorithm.h:104` returns + `byte_1_high & byte_1_low & byte_2_high`. The x86 classifier + (`src/haswell.cpp:43-72`) uses a low-nibble table plus `| 0x20`, and + admits in its own comment (57-66) that it also matches two control + characters, caught later in stage 2. + +
+ +- [ ] You can explain quote parity by prefix-XOR, and say which instruction computes it *on your machine*. + +
Answer + + A byte is inside a string iff an odd number of quotes precede it, so + the primitive needed is "XOR of all lower bits", per position. + On x86 that is one carry-less multiply by all-ones + (`westmere/bitmask.h:22`, `_mm_clmulepi64_si128`; the paper quotes + `pclmulqdq` at 7-cycle latency, 1/cycle throughput on Skylake). + + On aarch64 it is **not**: `arm64/bitmask.h:31-36` is a six-step + shift-XOR ladder (log₂ 64 = 6) in general-purpose registers, and + lines 19-29 give the reason — PMULL needs a GPR→FP `fmov` in + (3 cycles on N1, 5 on A72) and an FP→GPR `fmov` out (2 / 5), while + the GPR units are idle anyway because the critical path is on the FP + side doing classification. + +
+ +- [ ] You can describe the escaped-backslash problem and show, on numbers, why the shipped code subtracts rather than adds. + +
Answer + + A quote is real only if the backslash run before it has even length, + so the parser needs run-length parity without a loop. The paper's + Fig. 3 uses two additions and a carry ripple. The shipped code + (`json_escape_scanner.h:127-142`) does + `((potential_escape << 1) | ODD_BITS) - potential_escape) ^ ODD_BITS` + with `ODD_BITS = 0xAAAA…` (line 74): the *borrow* chain of the + subtraction ripples the length of each run, and XORing the odd bits + back off leaves 1s exactly on the escaped codes. + + In 8 bits, `\\\n` gives `0x07 → 0x0E → 0xAE → 0xA7 → 0x0D`, and + `0x0D ^ 0x07 = 0x0A` = the second backslash and the `n`. For `\\n` + it gives `0x03 → 0x06 → 0xAE → 0xAB → 0x01`, and + `0x01 ^ 0x03 = 0x02` = only the second backslash — the `n` is free. + +
+ +- [ ] You can explain over-write / under-advance in `bit_indexer::write`, and say what the step width is in the code versus in the paper. + +
Answer + + The loop writes a fixed number of index slots per round regardless of + how many bits are actually set, and advances the output cursor by the + true popcount (`json_structural_indexer.h:121`, `this->tail += cnt`). + Garbage past `cnt` is overwritten by the next block; the buffer is + oversized on purpose (doc comment, lines 85-86). The cost is a few + redundant stores; the saving is the unpredictable branch the paper + measures at "at least one branch miss for each word" (§3.1.4). + + The paper's Fig. 6 writes **8** at a time and calls it a heuristic. + The pinned code writes **4** (`STEP`, line 108) up to + `STEP_UNTIL = 24` (line 110), then a scalar tail. It also reverses + the mask once (line 103) and consumes it with leading-zeroes, because + ARM has no fast trailing-zero instruction (comment, lines 31-43). + +
+ +- [ ] You can say why stage 2 stays branchy, and give the Amdahl argument with a number you computed rather than one you were told. + +
Answer + + Stage 2 does the irreducibly data-dependent work — number parsing, + string unescaping, tape emission — on only the bytes stage 1 flagged. + "It only sees 1/8 of the bytes" is true for twitter and nothing else: + Table 5 spans 4.6 to 43.9 bytes per structural. + + Using §4.3's model on twitter (B = 631,514, S = 55,264, F = 1): + stage 1 = 1.7·S + 0.62·B = 485,487 cycles, total = 19·F + 11·S + + 0.92·B = 1,188,916 cycles, so stage 1 is **40.8 %** — the branchy + half is the larger one. It works anyway because stage 2's per-byte + term (0.31·B) is small; its cost is concentrated in the 8.7·S and + 19·F terms, which scale with *structure*, not with input size. + +
+ +- [ ] You can state the paper's throughput result together with the machine it was measured on, and say why it does not predict your Mac. + +
Answer + + §4.1: an **Intel i7-6700 (Skylake, 3.4 GHz, 3.7 GHz turbo)**, + DDR4-2133, GCC 9.1, `-O3 -march=native`, Linux. §4.5 claims 2 GB/s or + better on six files and 3 GB/s on gsoc-2018; Table 10 gives twitter + 2.2, citm_catalog 2.5, canada 1.1, marine_ik 0.94 GB/s — a 3.4× + spread *within one CPU*, driven by structural density. + + It does not predict an M-series Mac because the aarch64 kernel is a + different program: 64-byte blocks instead of 128, a shift-XOR + `prefix_xor` instead of CLMUL, an offset-table classifier instead of + nibble tables, a reverse-bits flatten instead of tzcnt, and an + 8-instruction bitmask fold instead of `PMOVMSKB`. The only honest + local number is one you measure. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including the RESP structural sketch and its bytes-per-structural. + +
Answer + + Self-check. The RESP one has a concrete target: count structural + characters in a real command frame such as + `*3\r\n$3\r\nSET\r\n$1\r\na\r\n$1\r\nb\r\n` (34 bytes), decide + whether `\r\n` counts as one structural position or two, and divide. + Compare against Step 6's table — a protocol with a marker every few + bytes sits at the marine_ik end, which is precisely where the paper's + 8-wide unconditional flatten stops helping. + +
## References **Papers** - Langdale & Lemire — "Parsing Gigabytes of JSON per Second" (VLDB Journal 2019, - [arXiv:1902.08318](https://arxiv.org/abs/1902.08318)) — §3 is - stage 1; work the escaped-backslash example in §3.1.1 by hand + [arXiv:1902.08318](https://arxiv.org/abs/1902.08318)). §3.1.1 escape + parity and prefix-XOR; §3.1.2 nibble classification; §3.1.4 index + extraction (Fig. 6, the 8-wide flatten); §4.1 hardware; §4.3 the + regression model used in Step 8; Table 5 dataset statistics; + Table 10 the throughput comparison. **Code** -- [simdjson](https://github.com/simdjson/simdjson) — - `include/simdjson/arm64/` (simd.h, bitmask.h) plus - `src/generic/stage1/` — read the NEON files, they're what your - machine runs +- [simdjson](https://github.com/simdjson/simdjson) at `c783809` — + `src/arm64.cpp` and `include/simdjson/arm64/` (simd.h, bitmask.h, + bitmanipulation.h) are what your machine runs; `src/generic/stage1/` + holds the ISA-independent algorithms they instantiate. Read the + arm64 files *first* — several of them contradict the paper, and the + comments explain why. diff --git a/topics/17-simd/reading-simsimd.md b/topics/17-simd/reading-simsimd.md index bbabca9..be45fb3 100644 --- a/topics/17-simd/reading-simsimd.md +++ b/topics/17-simd/reading-simsimd.md @@ -7,180 +7,980 @@ the headers, this chapter builds the microarchitecture vocabulary — ports, latency, dependency chains — then walks each design decision as a consequence of the table. The through-line: ports × latency decides everything, and fancy instructions lose to plain FMAs that -spread across ports. (Note: the headers live under -`include/numkong/`, the project's internal rename.) +spread across ports. + +Two things make this guide unusually concrete. First, the headers +carry a column literally labelled `M5`, and this repo's host is an +Apple M5 (`sysctl -n machdep.cpu.brand_string`) — so for once the +vendor-independent latency table you are reading *is your machine's*. +Second, that means you can use it to predict `notes.md`'s numbers +instead of admiring them, which Step 2 does. + +Every anchor below is SimSIMD at the pinned revision +`ashvardanian/SimSIMD@63a254f` (`resources/codebases.md`), quoted with +the line numbers the code occupies in that revision. The headers live +under `include/numkong/` — the project renamed its internal namespace +to `numkong`/`nk_`, so `SimSIMD` the repo ships `numkong` the C +library, and every symbol below is `nk_*`. ## The problem in one sentence -A dot-product loop with one accumulator runs at 1/12th of an Apple -M-series core's floating-point throughput — and SimSIMD's own +A dot-product loop with one accumulator runs at one twelfth of an +Apple M5 core's floating-point issue rate — and SimSIMD's own benchmark shows the "obvious" specialized instruction (FCMLA) losing 2.3× to plain FMAs (17.1 vs 39.7 GiB/s) — so every kernel here is -shaped by two numbers from the CPU manual, not by instruction -counts. +shaped by two numbers from the CPU manual, latency and port count, +not by instruction counts. ## The concepts, step by step -### Step 1 — ports and latency: the machine's real currency +### Step 1 — ports, latency, and why one accumulator is 1/12 of the machine + +> **In:** a loop that does one fused multiply-add per element into one +> running sum. +> **Out:** the number of *independent* such loops the core needs before +> it stops idling — computed from two numbers, latency and port count. + +Three definitions, because the rest of the guide is arithmetic on +them. + +An **execution port** (also "pipe") is an independent hardware unit +that can *start* one instruction per cycle. A core with 4 vector +floating-point pipes can begin 4 vector FMAs in the same cycle. Port +count sets **throughput**: how many operations can be *started* per +cycle. + +**Latency** is the number of cycles between an instruction starting +and its result being usable by a *dependent* instruction. An FMA +(**fused multiply-add**: `acc = acc + a*b`, computed as one +instruction with one rounding) has a latency of a few cycles. + +A **dependency chain** is a sequence where each operation consumes the +previous one's result. `acc += a[i]*b[i]` is a chain: iteration `i+1` +cannot start its FMA until iteration `i`'s FMA has retired its result +into `acc`. A chain therefore advances at exactly one operation per +`latency` cycles, no matter how many ports are idle. + +Little's law does the rest. To keep `ports` operations starting every +cycle when each takes `latency` cycles to complete, you need + +``` + in-flight operations = latency x ports +``` + +independent chains. Nothing about lane width appears in that formula. +Widening from 4 lanes to 16 lanes multiplies the work each chain does +but does not add a single chain. This is the sentence the whole topic +turns on, and `experiments/src/dot.rs` says it in the module doc: -A modern core doesn't execute one instruction at a time — it has -multiple **execution ports** (independent hardware units; M-series -has 4 that can each start a vector floating-point op every cycle), -and each instruction has a **latency** (cycles until its result is -usable — ~3 for an FMA, fused multiply-add: `acc = acc + a*b` in one -instruction). Peak throughput needs every port starting a new op -every cycle — but an op whose *input* is a previous op's *output* -must wait out the latency. A **dependency chain** (each op needing -the last one's result) therefore runs at 1 op per `latency` cycles, -using one port a third of the time. To saturate the machine you need -`latency × ports` = 3 × 4 = **12 independent chains** in flight. One -accumulator = one chain = 1/12 of the machine. Data-parallel loops -don't make you fast; independent chains do (README §1). +```rust +// topics/17-simd/experiments/src/dot.rs:1-5 + 1 //! Dot product: the reduction kernel. Four rungs. + 2 //! + 3 //! The lesson (README §1): lanes don't make you fast, independent + 4 //! dependency chains do. M-series wants ~12 FMA chains in flight; + 5 //! one accumulator uses 1/12 of the machine. +``` + +Step 2 turns "~12" into a number you can derive rather than quote. + +### Step 2 — the table is the design doc, and it predicts your own baseline -### Step 2 — the table is the design doc (spatial/neon.h:10-20) +> **In:** the comment block at the top of `include/numkong/dot/neon.h` +> and the `dot` row of this topic's `notes.md`. +> **Out:** the chain count this core wants, and the core's clock — +> solved for, not asserted. Every SimSIMD NEON header opens with the numbers Step 1 needs, -measured per microarchitecture: +measured per microarchitecture. The `dot` header's table is the +longest: + +```c +// include/numkong/dot/neon.h:11-27 (comment block, verbatim) + 11 * Key NEON instructions for dot products: + 12 * + 13 * Intrinsic Instruction A76 M5 + 14 * vfmaq_f32 FMLA (V.4S, V.4S, V.4S) 4cy @ 2p 3cy @ 4p + 15 * vfmaq_f64 FMLA (V.2D, V.2D, V.2D) 4cy @ 2p 4cy @ 4p + 16 * vfmsq_f64 FMLS (V.2D, V.2D, V.2D) 4cy @ 2p 4cy @ 4p + 17 * vmulq_f32 FMUL (V.4S, V.4S, V.4S) 3cy @ 2p 3cy @ 4p + 18 * vmulq_f64 FMUL (V.2D, V.2D, V.2D) 3cy @ 2p 3cy @ 4p + 19 * vaddvq_f32 FADDP+FADDP (reduce) 5cy @ 1p 8cy @ 1p + 20 * vaddvq_f64 FADDP (V.2D to scalar) 3cy @ 1p 3cy @ 1p + 21 * vpaddq_f32 FADDP (V.4S, V.4S, V.4S) 2cy @ 2p 3cy @ 4p + 22 * vpaddq_f64 FADDP (V.2D, V.2D, V.2D) 2cy @ 2p 3cy @ 4p + 23 * vcvt_f64_f32 FCVTL (V.2D, V.2S) 3cy @ 2p 3cy @ 2p + 24 * vld2_f32 LD2 ({Vt.2S, Vt2.2S}, [Xn]) 4cy @ 1p 4cy @ 1p + 25 * + 26 * FMA throughput doubles on cores with 4 SIMD pipes (Apple M4+, Graviton3+, Oryon), but + 27 * horizontal reductions remain at 1/cy on all cores and become the main bottleneck. +``` + +Read `3cy @ 4p` as "3 cycles of latency, 4 ports". Now apply Step 1's +formula to the two rows that matter, using the **M5** column, which is +this machine: + +``` + f32 FMA (line 14): latency 3 x ports 4 = 12 independent chains + f64 FMA (line 15): latency 4 x ports 4 = 16 independent chains + + peak f32 flops: 4 pipes x 4 lanes x 2 flops/FMA = 32 flops/cycle + peak f64 flops: 4 pipes x 2 lanes x 2 flops/FMA = 16 flops/cycle + + A76 for contrast (2 pipes): 4 x 2 = 8 chains, 16 f32 flops/cycle +``` + +So "M-series wants ~12 chains" is line 14's `3cy @ 4p` multiplied out, +and the A76 wants 8. The same source file, one column over, is a +different design. + +Now the part that makes the table yours. `notes.md` records the `dot` +lane at N = 4M f32 with a single accumulator (`dot_naive`, +`experiments/src/dot.rs:10-17`, one scalar `f32` chain) at +**10.89 GB/s**. That kernel is pure Step 1: one chain, so it must +advance at one element per FMA latency. Solve for the clock: + +``` + notes.md, dot lane: naive 10.89 GB/s, counting BOTH inputs + bytes per element-pair: 4 (a[i]) + 4 (b[i]) = 8 B + element-pairs per second: 10.89e9 / 8 = 1.361e9 /s + one chain at FMLA latency 3 cy (dot/neon.h:14, M5 col) + => 3 cycles per element-pair + => clock >= 1.361e9 x 3 = 4.08 GHz +``` + +4.08 GHz is a lower bound (it charges the whole 3-cycle latency to +useful work and none to loop overhead), and it is a plausible M5 +P-core boost clock. The table predicted the shape of a measurement +taken on a different day by a different tool, to within a napkin. Use +that 4.08 GHz figure wherever a cycle count is needed for this +machine; `reading-simdjson.md` and `reading-sigmod15-vectorization.md` +both cite it rather than guessing a clock. + +The second rung checks the other half of the model. `notes.md` has +`dot_unrolled8` (8 accumulators, `dot.rs:22-37`) at **42.12 GB/s**: ``` - Intrinsic Instruction A76 Apple M5 - vfmaq_f32 FMLA 4cy @ 2p 3cy @ 4p - vaddq_f32 FADD 2cy @ 2p 2cy @ 4p - vsqrtq_f32 FSQRT 12cy @ 1p 9cy @ 1p - vrsqrteq_f32 FRSQRTE 2cy @ 2p 3cy @ 1p + 8 chains vs 1 chain => model predicts up to 8x + measured 42.12 / 10.89 = 3.87x + fraction of the chain ceiling reached: 3.87 / 8 = 48 % + absolute rate: 42.12e9 / 8 B = 5.27e9 pairs/s + at 4.08 GHz = 1.29 pairs/cycle + chain ceiling with 8 chains at latency 3: 8/3 = 2.67 pairs/cycle ``` -Read it as: on M-series, FMA needs latency(3) × ports(4) = 12 -in-flight independent FMAs to saturate; on A76 only 8. And FSQRT is -a 1-port 9-cycle disaster — hence the kernels use `vrsqrteq` (a fast -~8-bit reciprocal-square-root *estimate*) refined by 3 Newton-Raphson -rounds (each round roughly doubles the correct bits: 8 → 16 → 32 → -~48, f64-grade) instead of ever issuing FSQRT. Every choice below is -a row of this table. +Halfway to the ceiling and no further — because at N = 4M the two +input arrays are 16.78 MB each and the loop is now moving 42 GB/s +through memory, which is single-core DRAM territory on this class of +part. That is the honest reading of `notes.md`'s 3.9×: the first +rung was latency-bound and the model explains it exactly; the second +rung escaped latency and hit bandwidth, so the model only bounds it. +Keep that distinction — Step 3 depends on it. + +### Step 3 — precision by wider accumulators, not by reordering -### Step 3 — precision by wider accumulators, not reordering (dot/neon.h:126) +> **In:** `nk_dot_f32_neon`, the f32 dot product every Python/Rust +> caller of this library actually reaches. +> **Out:** why it accumulates in f64 at a 16× cost in peak issue rate, +> and why that 16× is nearly free at the sizes it runs on. -Summing millions of f32 products accumulates rounding error. polars' -answer was pairwise recursion (restructure the ADDITION ORDER); -SimSIMD's answer is accumulate in f64 (restructure the PRECISION) — -upcast each half of the f32 vector with FCVTL and FMA into two f64 -accumulators: +Summing millions of f32 products accumulates rounding error: once the +running sum is large, small addends fall off the bottom of the +24-bit significand. polars' answer (`reading-polars-compute.md`, +Step 5) was pairwise recursion — restructure the *addition order*. +SimSIMD's answer is to restructure the *precision*: ```c -float64x2_t sum_low = vdupq_n_f64(0); // chain 1 -float64x2_t sum_high = vdupq_n_f64(0); // chain 2 -for (; i + 4 <= n; i += 4) { - a_f32x4 = vld1q_f32(a+i); b_f32x4 = vld1q_f32(b+i); - // FCVTL / FCVTL2: upcast each half to f64x2 - sum_low = vfmaq_f64(sum_low, a_low_f64, b_low_f64); - sum_high = vfmaq_f64(sum_high, a_high_f64, b_high_f64); -} -``` - -f32 inputs, f64 accumulators — half the lane width, deliberately: -the kernel trades throughput for error control that costs no extra -instructions per element. But notice: only TWO chains, when Step 1 -demanded 12. The missing parallelism comes from Step 4. Question: -for M14's l2 distance over 1536-dim embeddings, which error-control -strategy is cheaper on M-series, and does recall@10 even care? - -### Step 4 — batch candidates, don't unroll pairs (dot/neon.h:37-45) - -The streaming API's doc-comment shows where the other chains come -from: score ONE query against FOUR targets simultaneously, keeping -four independent accumulator states: - -``` - for idx: chains in flight: - q = load(query+idx) state1 += q·t1 ┐ - t1..t4 = load(4 targets) state2 += q·t2 │ 4 FMA chains, - state3 += q·t3 │ shared q load - state4 += q·t4 ┘ - finalize(4 states) → one f32x4 of results -``` - -The instruction-level parallelism comes from BATCHING CANDIDATES, -not from unrolling one pair — the query load is amortized 4×, and -4 states × 2 chains each (Step 3's low/high split) ≈ the 12 chains -the machine wants. This is exactly M14's HNSW inner loop shape -(score one query against a neighbor list). Question: why is this -better than 4 accumulators over a single pair for the short-vector -case (n=128 dims: how many iterations does each scheme get to -overlap)? - -### Step 5 — the FCMLA lesson: specialized instructions must beat the table - -ARMv8.3 added FCMLA, a complex-multiply instruction that looks -purpose-built for complex dot products. SimSIMD benchmarked it -(comment near dot/neon.h:150): 17.1 GiB/s vs 39.7 GiB/s for the -"dumb" alternative — deinterleave with `vld2` + 4 independent FMAs -on M4. The fancy instruction LOST 2.3× because it serializes work -that 4 plain FMAs spread over 4 ports (Step 1's arithmetic: fewer, -longer chains). The meta-lesson for M17: newer/specialized -instruction ≠ faster; ports × latency decides. Question: what's the -NEON analogue in our filter kernel (is `vqtbl1q` compress always -better than branchless stores)? - -### Step 6 — dispatch: one file per ISA, function pointers at init +// include/numkong/dot/neon.h:126-146 + 126 NK_PUBLIC void nk_dot_f32_neon(nk_f32_t const *a_scalars, nk_f32_t const *b_scalars, nk_size_t count_scalars, + 127 nk_f64_t *result) { + 128 // Upcast f32 to f64 via FCVTL/FCVTL2, two independent FMA chains for ILP + 129 float64x2_t sum_low_f64x2 = vdupq_n_f64(0); + 130 float64x2_t sum_high_f64x2 = vdupq_n_f64(0); + 131 nk_size_t idx_scalars = 0; + 132 for (; idx_scalars + 4 <= count_scalars; idx_scalars += 4) { + 133 float32x4_t a_f32x4 = vld1q_f32(a_scalars + idx_scalars); + 134 float32x4_t b_f32x4 = vld1q_f32(b_scalars + idx_scalars); + 135 float64x2_t a_low_f64x2 = vcvt_f64_f32(vget_low_f32(a_f32x4)); + 136 float64x2_t a_high_f64x2 = vcvt_high_f64_f32(a_f32x4); + 137 float64x2_t b_low_f64x2 = vcvt_f64_f32(vget_low_f32(b_f32x4)); + 138 float64x2_t b_high_f64x2 = vcvt_high_f64_f32(b_f32x4); + 139 sum_low_f64x2 = vfmaq_f64(sum_low_f64x2, a_low_f64x2, b_low_f64x2); + 140 sum_high_f64x2 = vfmaq_f64(sum_high_f64x2, a_high_f64x2, b_high_f64x2); + 141 } + 142 nk_f64_t sum_f64 = vaddvq_f64(vaddq_f64(sum_low_f64x2, sum_high_f64x2)); + 143 for (; idx_scalars < count_scalars; ++idx_scalars) + 144 sum_f64 += (nk_f64_t)a_scalars[idx_scalars] * (nk_f64_t)b_scalars[idx_scalars]; + 145 *result = sum_f64; + 146 } +``` + +Lines 129-130 are the two accumulators; 135-138 are the four FCVTL +upcasts (`vcvt_f64_f32` takes the low half, `vcvt_high_f64_f32` the +high half); 139-140 are the two FMAs, one per chain; 142 folds the +chains and reduces; 143-144 is the scalar tail for `count_scalars % 4`. +Note the signature: `nk_f64_t *result` at line 127. The f64-ness is +not an internal detail, it is in the API. + +Now price it, using the M5 column and the loop's own shape: + +``` + per iteration (dot/neon.h:132-141): 4 f32 element-pairs consumed + 2 loads, 4 FCVTL (line 23: 3cy @ 2p), 2 vfmaq_f64 (line 15: 4cy @ 4p) + + chains: 2 (lines 129-130), each advancing 1 FMA per 4 cy + => 1 iteration per 4 cycles => 4 pairs / 4 cy = 1.00 pair/cycle + + what the core could do with 12 f32 chains and no upcast: + 12 chains / 3 cy latency = 4 FMAs/cy x 4 lanes = 16 pairs/cycle + + peak-issue cost of this design: 16 / 1 = 16x +``` + +Sixteen times slower than the machine's f32 ceiling — and SimSIMD +ships it anyway. Step 2 explains why that is defensible: + +``` + 1.00 pair/cycle at 4.08 GHz x 8 B/pair = 32.6 GB/s + this topic's measured bandwidth ceiling (notes.md, + dot_unrolled8, 8 chains, no upcast) = 42.1 GB/s + shortfall at DRAM-resident sizes: 1 - 32.6/42.1 = 23 % +``` + +A 16× penalty in peak issue becomes a ~23 % penalty in delivered +bandwidth, because at these sizes nothing is issue-bound. That is the +whole argument for the f64 upcast: it costs almost nothing where the +library actually runs, and it removes an error mode that is very hard +to debug from Python. It would be a bad trade for L1-resident data, +which is exactly where Step 6's batching applies. + +One more detail worth naming: the FMAs are not the busiest +instruction here. Line 23 gives FCVTL 2 ports; the loop issues 4 of +them and only 2 FMAs, so the *converts* consume 2 cycles of issue +against the FMAs' 0.5. If you ever try to beat this kernel, the +converts are what you have to remove, not the FMAs. + +### Step 4 — the horizontal reduction is the other bottleneck + +> **In:** the two `vaddvq_*` rows of the table and line 142 of +> `nk_dot_f32_neon`. +> **Out:** the vector length below which the reduce dominates the +> kernel, computed. + +Line 27 of the header states the problem in one sentence: "horizontal +reductions remain at 1/cy on all cores and become the main +bottleneck." A **horizontal reduction** sums the lanes *within* one +vector register down to a scalar; unlike lane-wise arithmetic it +cannot be spread across ports, because each stage feeds the next. + +The table prices two of them, and the gap is startling: + +``` + vaddvq_f32 (line 19): FADDP+FADDP A76 5cy @ 1p M5 8cy @ 1p + vaddvq_f64 (line 20): FADDP A76 3cy @ 1p M5 3cy @ 1p +``` + +On M5 the f32 reduce is **8/3 = 2.7× more expensive** than the f64 +one, and it is the only row in the whole table that got *worse* from +A76 to M5 (5 → 8 cycles) while everything else got better. Line 142 +therefore reduces with `vaddvq_f64`, which it gets for free because +Step 3 already chose f64 accumulators. The precision decision paid a +performance dividend one line later — that is not a coincidence, it is +what reading your own table buys you. + +Now compute when it matters. Take the reduce at line 142 as one +`vaddq_f64` plus one `vaddvq_f64` ≈ 3 + 3 = 6 cycles, and the loop at +1 iteration per 4 cycles from Step 3: + +``` + n = 1536 (a typical embedding dimension): + iterations = 1536 / 4 = 384; loop = 384 x 4 cy = 1536 cy + reduce = 6 cy + overhead = 6 / 1542 = 0.39 % + + n = 8 (a tiny vector, e.g. a quantized codebook entry): + iterations = 2; loop = 2 x 4 cy = 8 cy + reduce = 6 cy + overhead = 6 / 14 = 43 % + + break-even at 5 % overhead: loop >= 6 / 0.05 = 120 cy + => 30 iterations => n >= 120 elements +``` + +Below ~120 dimensions the reduction is a first-order cost and the +kernel shape should change — which is precisely the case the streaming +API in Step 6 is built for, because it reduces four vectors at once. +Above it, reduce once at the end and forget about it. Note also what +this does to Step 8's dispatch argument: a kernel that costs 14 cycles +cannot afford a feature test. + +### Step 5 — FSQRT vs. estimate-and-refine, and the doubling rule + +> **In:** the `spatial` header's table and its two reciprocal-square-root +> helpers, one for f32 and one for f64. +> **Out:** why neither ever issues FSQRT, how many Newton-Raphson +> rounds each needs, and one place where SimSIMD's own comment +> overstates the case. + +The `spatial` header (L2, cosine/angular distances) opens with its own +table, and its bottom two rows are the story: + +```c +// include/numkong/spatial/neon.h:13-26 (comment block, verbatim) + 13 * Intrinsic Instruction A76 M5 + 14 * vfmaq_f32 FMLA (V.4S, V.4S, V.4S) 4cy @ 2p 3cy @ 4p + 15 * vmulq_f32 FMUL (V.4S, V.4S, V.4S) 3cy @ 2p 3cy @ 4p + 16 * vaddq_f32 FADD (V.4S, V.4S, V.4S) 2cy @ 2p 2cy @ 4p + 17 * vsubq_f32 FSUB (V.4S, V.4S, V.4S) 2cy @ 2p 2cy @ 4p + 18 * vrsqrteq_f32 FRSQRTE (V.4S, V.4S) 2cy @ 2p 3cy @ 1p + 19 * vsqrtq_f32 FSQRT (V.4S, V.4S) 12cy @ 1p 9cy @ 1p + 20 * vrecpeq_f32 FRECPE (V.4S, V.4S) 2cy @ 2p 3cy @ 1p + 21 * + 22 * FRSQRTE provides ~8-bit precision; two Newton-Raphson iterations via vrsqrtsq_f32 achieve + 23 * ~23-bit precision, sufficient for f32. This is much faster than FSQRT (0.25/cy). + 24 * + 25 * Distance computations (L2, angular) benefit from 2x throughput on 4-pipe cores (Apple M4+, + 26 * Graviton3+, Oryon), but FSQRT remains slow on all cores. Use rsqrt+NR when precision allows. +``` + +`FSQRT` at `9cy @ 1p` with a reciprocal throughput of 0.25/cy (line +23 — one result every 4 cycles) is the worst instruction in the file. +`FRSQRTE` is a **reciprocal square root estimate**: a table lookup +that returns an approximation of `1/sqrt(x)` in 3 cycles. It is not +accurate enough on its own, so it is refined: + +```c +// include/numkong/spatial/neon.h:50-61 + 50 * @brief Reciprocal square root of 4 floats with Newton-Raphson refinement. + 51 * + 52 * Uses `vrsqrteq_f32` (~8-bit initial estimate) followed by two Newton-Raphson iterations + 53 * via `vrsqrtsq_f32`, achieving ~23-bit precision — sufficient for f32. + 54 * Much faster than `vsqrtq_f32` (2 cy vs 9-12 cy latency, 2/cy vs 0.25/cy throughput). + 55 */ + 56 NK_INTERNAL float32x4_t nk_rsqrt_f32x4_neon_(float32x4_t x) { + 57 float32x4_t rsqrt_f32x4 = vrsqrteq_f32(x); + 58 rsqrt_f32x4 = vmulq_f32(rsqrt_f32x4, vrsqrtsq_f32(vmulq_f32(x, rsqrt_f32x4), rsqrt_f32x4)); + 59 rsqrt_f32x4 = vmulq_f32(rsqrt_f32x4, vrsqrtsq_f32(vmulq_f32(x, rsqrt_f32x4), rsqrt_f32x4)); + 60 return rsqrt_f32x4; + 61 } +``` + +**Two** refinement rounds (lines 58 and 59), not three. Newton-Raphson +on this function converges quadratically, so each round roughly +doubles the number of correct bits, and the significand width caps it: + +``` + f32 path (lines 57-59), 2 rounds: + 8 bits -> 16 -> 32, capped by f32's 24-bit significand + header line 53 records ~23 bits + + f64 path (lines 110-115), 3 rounds: + 8 bits -> 16 -> 32 -> 64, capped by f64's 53-bit significand + header line 67 records ~48 bits +``` + +The f64 sibling is a separate piece of code, inside the angular-distance +helper, and it is where the third round lives: + +```c +// include/numkong/spatial/neon.h:105-115 + 105 // Unlike x86, Arm NEON manuals don't explicitly mention the accuracy of their `rsqrt` approximation. + 106 // Third-party research suggests that it's less accurate than SSE instructions, having an error of 1.5×2⁻¹². + 107 // One or two rounds of Newton-Raphson refinement are recommended to improve the accuracy. + 108 // https://github.com/lighttransport/embree-aarch64/issues/24 + 109 // https://github.com/lighttransport/embree-aarch64/blob/3f75f8cb4e553d13dced941b5fefd4c826835a6b/common/math/math.h#L137-L145 + 110 float64x2_t rsqrts_f64x2 = vrsqrteq_f64(squares_f64x2); + 111 // Perform three rounds of Newton-Raphson refinement for f64 precision (~48 bits): + 112 rsqrts_f64x2 = vmulq_f64(rsqrts_f64x2, vrsqrtsq_f64(vmulq_f64(squares_f64x2, rsqrts_f64x2), rsqrts_f64x2)); + 113 rsqrts_f64x2 = vmulq_f64(rsqrts_f64x2, vrsqrtsq_f64(vmulq_f64(squares_f64x2, rsqrts_f64x2), rsqrts_f64x2)); + 114 rsqrts_f64x2 = vmulq_f64(rsqrts_f64x2, vrsqrtsq_f64(vmulq_f64(squares_f64x2, rsqrts_f64x2), rsqrts_f64x2)); +``` + +Line 106 is a second, better-sourced estimate of FRSQRTE's starting +accuracy, and it disagrees with line 22's "~8-bit": + +``` + error 1.5 x 2^-12 => correct bits = 12 - log2(1.5) = 12 - 0.585 = 11.4 bits + header line 22 claims ~8 bits +``` + +Both numbers are in the same file, 84 lines apart, and 11.4 is the one +with a citation attached (line 108). Nothing downstream breaks — +starting from 11.4 bits, two rounds still saturate f32 — but it is a +good habit to notice when a file's summary table is rounder than its +own footnotes. + +Two more places where the prose is looser than the table, both +checkable from what you have already read: + +1. Line 54's "2 cy vs 9-12 cy latency, 2/cy vs 0.25/cy throughput" is + an **A76** comparison. Line 18's M5 column says FRSQRTE is + `3cy @ 1p` — the *same single port* as FSQRT, and one cycle slower + than the A76 it is being contrasted with. On M5 the port advantage + the comment advertises does not exist; the win is the 3-vs-9 latency + plus the fact that the six refinement instructions (lines 58-59: + four FMUL, two FRSQRTS) are ordinary vector arithmetic that spreads + over 4 pipes. +2. That same "2 cy" compares a *bare* FRSQRTE against a *complete* + FSQRT. The honest comparison includes lines 58-59, and the + refinement is itself a dependency chain — each round's FMUL needs + the previous round's result. SimSIMD's table does not list a + latency for FRSQRTS, so the total cannot be computed from this + source, and this guide will not invent one. What you *can* say from + the table alone is the throughput claim, which is the one that + matters for a loop over many vectors. + +The kernels that use all this are compact. `nk_sqeuclidean_f32_neon` +is the L2-squared workhorse: + +```c +// include/numkong/spatial/neon.h:123-140 + 123 NK_PUBLIC void nk_sqeuclidean_f32_neon(nk_f32_t const *a, nk_f32_t const *b, nk_size_t n, nk_f64_t *result) { + 124 // Accumulate in f64 for numerical stability (2 f32s per iteration, avoids slow vget_low/high) + 125 float64x2_t sum_f64x2 = vdupq_n_f64(0); + 126 nk_size_t i = 0; + 127 for (; i + 2 <= n; i += 2) { + 128 float32x2_t a_f32x2 = vld1_f32(a + i); + 129 float32x2_t b_f32x2 = vld1_f32(b + i); + 130 float32x2_t diff_f32x2 = vsub_f32(a_f32x2, b_f32x2); + 131 float64x2_t diff_f64x2 = vcvt_f64_f32(diff_f32x2); + 132 sum_f64x2 = vfmaq_f64(sum_f64x2, diff_f64x2, diff_f64x2); + 133 } + 134 nk_f64_t sum_f64 = vaddvq_f64(sum_f64x2); + 135 for (; i < n; ++i) { + 136 nk_f64_t diff_f64 = (nk_f64_t)a[i] - (nk_f64_t)b[i]; + 137 sum_f64 += diff_f64 * diff_f64; + 138 } + 139 *result = sum_f64; + 140 } +``` + +Count the chains: **one** (line 125), consuming **two** f32 per +iteration (`float32x2_t` at 128-129, half a register). That is +2 elements per 4 cycles = 0.5 elements/cycle, against a 16-chain f64 +ceiling. It is the least aggressive kernel in the library, and the +comment at 124 explains the width choice ("avoids slow +vget_low/high") but not the chain count. Whether that is a real +oversight or a judgement that L2 is memory-bound anyway is question 2 +below — and after Step 2 you can predict the answer before you test +it. The Euclidean distance is then one line, reusing it: + +```c +// include/numkong/spatial/neon.h:142-145 + 142 NK_PUBLIC void nk_euclidean_f32_neon(nk_f32_t const *a, nk_f32_t const *b, nk_size_t n, nk_f64_t *result) { + 143 nk_sqeuclidean_f32_neon(a, b, n, result); + 144 *result = nk_f64_sqrt_neon(*result); + 145 } +``` + +One square root for the whole vector — which is why Step 5's FSQRT +analysis matters less than it looks for L2, and much more for +normalization, where you take one per *element*. + +### Step 6 — batch candidates, don't unroll pairs + +> **In:** the streaming-state API documented at the top of +> `dot/neon.h`, and the chain deficit left over from Step 3. +> **Out:** where the missing chains come from, and how much load +> traffic batching saves — both computed. + +Step 3 left `nk_dot_f32_neon` with 2 chains where the machine wants +16. The missing parallelism does not come from unrolling the loop +further over *one* pair of vectors — it comes from scoring one query +against *several* targets at once. SimSIMD documents the pattern in +the header, as runnable example code: + +```c +// include/numkong/dot/neon.h:40-60 (doc-comment example, verbatim) + 40 * @code{c} + 41 * nk_dot_f32x2_state_neon_t state_first, state_second, state_third, state_fourth; + 42 * float32x2_t query_f32x2, target_first_f32x2, target_second_f32x2, target_third_f32x2, target_fourth_f32x2; + 43 * nk_dot_f32x2_init_neon(&state_first); + // ... 44-46: three more init calls, one per state ... + 47 * for (nk_size_t idx = 0; idx + 2 <= depth; idx += 2) { + 48 * query_f32x2 = vld1_f32(query_ptr + idx); + 49 * target_first_f32x2 = vld1_f32(target_first_ptr + idx); + 50 * target_second_f32x2 = vld1_f32(target_second_ptr + idx); + 51 * target_third_f32x2 = vld1_f32(target_third_ptr + idx); + 52 * target_fourth_f32x2 = vld1_f32(target_fourth_ptr + idx); + 53 * nk_dot_f32x2_update_neon(&state_first, query_f32x2, target_first_f32x2, idx, 2); + // ... 54-56: three more updates, same query vector, different targets ... + 57 * } + 58 * float32x4_t results_f32x4; + 59 * nk_dot_f32x2_finalize_neon(&state_first, &state_second, &state_third, &state_fourth, depth, &results_f32x4); + 60 * @endcode +``` + +Line 48 loads the query **once**; lines 49-52 load four different +targets; lines 53-56 feed four independent states. Line 59 reduces all +four together into a single `float32x4_t` — four distances, one +register. That signature is the shape of an HNSW inner loop: score one +query against a neighbour list. + +The state is exactly what Step 1 predicts it should be — one +accumulator, so one chain per candidate: + +```c +// include/numkong/dot/neon.h:230-246 + 230 typedef struct nk_dot_f32x2_state_neon_t { + 231 float64x2_t sum_f64x2; + 232 } nk_dot_f32x2_state_neon_t; + // ... 234: init zeroes sum_f64x2 ... + 236 NK_INTERNAL void nk_dot_f32x2_update_neon(nk_dot_f32x2_state_neon_t *state, nk_b64_vec_t a, nk_b64_vec_t b, + 237 nk_size_t depth_offset, nk_size_t active_dimensions) { + // ... 238-239: unused-parameter shims ... + 240 // Upcast 2 f32s to f64s for high-precision accumulation + 241 float32x2_t a_f32x2 = vreinterpret_f32_u32(a.u32x2); + 242 float32x2_t b_f32x2 = vreinterpret_f32_u32(b.u32x2); + 243 float64x2_t a_f64x2 = vcvt_f64_f32(a_f32x2); + 244 float64x2_t b_f64x2 = vcvt_f64_f32(b_f32x2); + 245 state->sum_f64x2 = vfmaq_f64(state->sum_f64x2, a_f64x2, b_f64x2); + 246 } +``` + +Price the two arrangements over the same work — one query against four +targets, `depth` dimensions each: + +``` + A. four separate nk_dot_f32_neon calls (Step 3's kernel) + per call: 2 chains, 4 pairs per 4 cy = 1.00 pair/cycle + the query array is streamed 4 times = 4x query load traffic + chains in flight at any moment = 2 of 16 + + B. the batched streaming loop (lines 47-57) + loads per iteration: 1 query + 4 targets = 5 loads + element-pairs per iteration: 4 states x 2 = 8 pairs + chains: 4 states x 1 accumulator = 4 of 16 + each chain: 1 vfmaq_f64 per 4 cy + => 8 pairs per 4 cycles = 2.00 pairs/cycle + + speedup B/A = 2.0x + load traffic: naive would need 4 x (1 query + 1 target) = 8 loads + batched needs 5 = 37.5 % fewer +``` + +Twice the throughput and a third fewer loads, from restructuring the +*call*, not the kernel. And notice what the arithmetic also says: +4 chains of 16 is still 25 % of the machine, so batching **eight** +targets instead of four would double it again — the API's choice of +four is a register-pressure decision, not a ceiling. That is a +concrete thing to try in `experiments/`. + +Why batching rather than unrolling? Because unrolling needs +iterations to unroll *over*. At `depth = 128`, Step 3's kernel gets +128/4 = 32 iterations, so an 8-wide unroll leaves 4 iterations per +chain — barely enough to fill the pipeline before the loop ends, and +the Step 4 reduce then costs 6 of the ~134 cycles. Batching adds +chains without needing a single extra iteration, which is the only +option when `depth` is small and the candidate list is long. Short +vectors, many of them: that is the vector-search workload exactly. + +### Step 7 — the FCMLA lesson: a specialized instruction must beat the table + +> **In:** ARMv8.3's `FCMLA`, an instruction that computes a complex +> multiply-accumulate in one go, and SimSIMD's benchmark of it. +> **Out:** why fewer instructions lost to more instructions, by 2.3×, +> and what that means for your own kernel choices. + +ARMv8.3-A added `FCMLA` (fused complex multiply-add), which looks +purpose-built for complex dot products: it does the rotate-and-multiply +dance of `(a+bi)(c+di)` in hardware. SimSIMD measured it and rejected +it, recording the result in the code that replaced it: + +```c +// include/numkong/dot/neon.h:154-159 (inside nk_dot_f32c_neon, 148-187) + 154 // ARMv8.3-A FCMLA (`vcmlaq_rot0/rot90_f32`) was benchmarked as an alternative to the + 155 // deinterleave+4FMA pattern below. FCMLA processes only 2 complex pairs per iteration + 156 // (interleaved 128-bit operands, 2x `vcmlaq`), while `vld2_f32` deinterleaves 2 pairs + 157 // with 4 independent FMA instructions that fully utilize M4's 4 SIMD pipes. Result on + 158 // Apple M4 at n=4096: manual f32 39.7 GiB/s, FCMLA 17.1 GiB/s (2.3x slower). + 159 // The f64 upcast here trades throughput for precision — FCMLA offers neither advantage. +``` + +Read the instruction counts before the conclusion: + +``` + per 2 complex pairs: + FCMLA path : 2 x vcmlaq = 2 instructions + deinterleave path : 1 x vld2_f32 + 4 FMA = 5 instructions + + measured (line 158, Apple M4, n=4096): + deinterleave (manual f32) = 39.7 GiB/s + FCMLA = 17.1 GiB/s + ratio 39.7 / 17.1 = 2.32x +``` + +The path with **2.5× more instructions is 2.3× faster.** Instruction +count is simply not the metric. Two `vcmlaq` are two operations on the +critical path; four FMAs are four independent chains that Step 1's +formula says the 4-pipe core wants. Ports × latency decided it, as it +decides everything else in this file. + +Three cautions on the number itself, because it is easy to misquote: + +- **39.7 GiB/s is not what the shipped code does.** Line 159 says so + explicitly: the benchmark's fast variant accumulated in f32, and the + shipped `nk_dot_f32c_neon` upcasts to f64 (lines 165-168) for the + Step 3 reason. The 39.7 is the *rejected alternative's* speed, kept + as evidence about FCMLA, not as a performance claim for the library. +- **It is an M4 number, and this machine is an M5.** The port counts + are the same (4 pipes, `dot/neon.h:26` lists "Apple M4+"), so the + argument transfers; the absolute figure need not. +- **GiB/s, not GB/s** — 2^30, while `notes.md` and `FINDINGS.md` use + decimal 10^9. Never compare the two without converting + (39.7 GiB/s = 42.6 GB/s). + +The generalization for your own work: a new instruction earns its +place only if it improves `chains x lanes / latency`, and "it exists +and it is named after my problem" is not evidence. The same test +applies to the NEON table lookup in this topic's own filter kernel — +`notes.md`'s implementation log records `count_neon + compact_neon` +with all 16 mask cases passing, and question 5 asks whether `vqtbl1q` +compression actually beats the branchless store it replaces, or +whether it just looks more like SIMD. + +### Step 8 — dispatch: one file per ISA, function pointers filled at load + +> **In:** the same kernel written once per instruction set, and a +> caller that must not care. +> **Out:** the third of this topic's three binding times, and the cost +> model that picks between them. Each kernel family exists once per ISA — `dot/neon.h`, `dot/sve.h`, -`dot/haswell.h`, `dot/skylake.h` — and the directory layout IS the -dispatch table. At startup, capability detection fills a table of -function pointers once; call sites pay an indirect call, never a -feature test. That's the middle binding time of this topic's three: -hashbrown binds at compile time (`cfg_if!`), polars at call time -(runtime detect per kernel invocation), SimSIMD at init. Question: -an indirect call can't inline — when does THAT cost exceed the -runtime-check cost it saves (think n=8 dims vs n=4096)? +`dot/haswell.h`, `dot/skylake.h` — so the directory layout *is* the +dispatch table's shape. What connects them is a struct of function +pointers, one field per (kernel, dtype): + +```c +// include/numkong via c/dispatch.h:10-12, 23-27, 36-45 + 10 #define NK_DYNAMIC_DISPATCH 1 + // ... 14-22: NK_TARGET_* defines come from the build system ... + 23 * OS/compiler capabilities summary: + 24 * - Linux: everything available in GCC 12+ and Clang 16+. + // ... 25-26: FreeBSD and Windows/MSVC rows ... + 27 * - macOS - Apple Clang: only Arm NEON and x86 AVX2 Haswell extensions. + // ... 30-34: includes and extern "C" ... + 36 // Forward declaration of dispatch table type (same structure as in numkong.c) + 37 typedef struct { + 38 // Dot products + 39 nk_metric_dense_punned_t dot_f64c; + 40 nk_metric_dense_punned_t dot_f32c; + 41 nk_metric_dense_punned_t dot_bf16c; + 42 nk_metric_dense_punned_t dot_f16c; + 43 nk_metric_dense_punned_t dot_f64; + 44 nk_metric_dense_punned_t dot_f32; +``` + +Line 27 is worth pausing on: on this Mac, Apple Clang supports only +NEON, so the SVE and Skylake files are not merely unselected, they are +not compiled. The `#[cfg]`-style layer removes the impossible options +before the runtime layer chooses among the possible ones — the same +two-layer structure `reading-polars-compute.md` Step 6 finds in polars. + +The table is filled exactly once, at library load: + +```c +// include/numkong via c/numkong.c:832-843 and 915-919 + 832 NK_DYNAMIC nk_capability_t nk_capabilities(void) { + 833 //! The latency of the CPUID instruction can be over 100 cycles, so we cache the result. + 834 static nk_capability_t static_capabilities = nk_cap_any_k; + 835 if (static_capabilities != nk_cap_any_k) return static_capabilities; + 836 + 837 static_capabilities = nk_capabilities_(); + 838 + 839 // Initialize the central dispatch table with the detected capabilities + 840 nk_dispatch_table_init(); + 841 + 842 return static_capabilities; + 843 } + // ... 844-914: per-ISA capability probes ... + 915 // Auto-initialization for dynamic libraries - ensures dispatch table is populated on library load + 916 #if defined(__GNUC__) || defined(__clang__) + 917 __attribute__((constructor)) static void nk_auto_init(void) { + 918 nk_capabilities(); // Triggers dispatch table initialization + 919 } +``` + +Line 917's `__attribute__((constructor))` runs `nk_auto_init` before +`main`; line 835 memoizes so the second call is a load and a compare; +line 840 fills the table. Line 833 gives the reason in the file's own +words — CPUID can cost over 100 cycles. + +That comment is the whole cost model, and Step 4 already gave you the +other side of it: + +``` + detection cost, per c/numkong.c:833 > 100 cycles + a short-vector dot kernel (Step 4, n = 8) = 14 cycles + + detecting per call: 100 / 14 = 7x the kernel itself + detecting once at load, amortized over 1e6 calls + 100 / 1e6 = 0.0001 cycles/call + residual per-call cost of init-time binding: one indirect call + (unpredictable target on first use, then predicted; never inlined) +``` + +Which gives this topic's three binding times, and why each is right +for its own dispatched unit: + +| binding time | who | mechanism | cost per use | dispatched unit | +|---|---|---|---|---| +| compile | hashbrown, memchr | `cfg_if!` picks a backend module (`group/mod.rs:8-45`) | zero | a handful of instructions | +| init | SimSIMD | `__attribute__((constructor))` fills a fn-pointer table (`c/numkong.c:917-919`) | one indirect call | one whole vector distance | +| call | polars | `is_x86_feature_detected!` per array (`filter/primitive.rs:33`) | one predictable branch | one whole column | + +SimSIMD sits in the middle because it ships a C library whose +compilation unit cannot know the target, but whose dispatched unit — +a full distance over hundreds of dimensions — is large enough to +absorb an indirect call. Change either fact and the answer changes: +that is what makes this a cost model rather than a style preference. ## Where each step lives in the code | anchor | step | what it is | |---|---|---| -| numkong/spatial/neon.h:10-20 | 2 | THE table: per-instruction latency/ports on A76 vs Apple M-series | -| numkong/spatial/neon.h:~100 | 2 | rsqrt by `vrsqrteq` + 3 Newton-Raphson rounds (no FSQRT) | -| numkong/dot/neon.h:126-146 | 3 | `nk_dot_f32_neon` — FCVTL upcast, TWO independent FMA chains | -| numkong/spatial/neon.h:123-140 | 3 | `nk_sqeuclidean_f32_neon` — f64 accumulation, 2 f32/iter | -| numkong/dot/neon.h:37-45 | 4 | stateful streaming API: FOUR `nk_dot_f32x2_state_neon_t`s | -| numkong/dot/neon.h:~150 | 5 | the FCMLA comment: measured 39.7 vs 17.1 GiB/s — why they said no | -| include/numkong/*/ | 6 | one file per ISA per kernel family: neon, sve, haswell, skylake... | - -Reading order: the table at the top of `spatial/neon.h` first — it's -the real reading assignment — then `dot/neon.h` top to bottom (the -streaming-state doc-comment, the kernel, the FCMLA comment), then -skim one x86 sibling to see the same kernel re-derived from a -different table. +| `dot/neon.h:11-27` | 2 | THE table: latency and port counts, A76 vs Apple M5; line 27 names the reduce as the bottleneck | +| `experiments/src/dot.rs:1-5` | 1 | this topic's own statement of the chain rule | +| `dot/neon.h:126-146` | 3, 4 | `nk_dot_f32_neon` — FCVTL upcasts (135-138), TWO f64 chains (129-130), `vaddvq_f64` reduce (142), scalar tail (143-144) | +| `dot/neon.h:19-20` | 4 | `vaddvq_f32` 8cy@1p vs `vaddvq_f64` 3cy@1p on M5 — the reduce that got worse | +| `spatial/neon.h:13-26` | 5 | the distance table; FSQRT `9cy @ 1p`, FRSQRTE `3cy @ 1p` | +| `spatial/neon.h:50-61` | 5 | `nk_rsqrt_f32x4_neon_` — FRSQRTE + **two** NR rounds → ~23 bits | +| `spatial/neon.h:105-115` | 5 | the f64 path — **three** NR rounds → ~48 bits, and the `1.5×2⁻¹²` citation | +| `spatial/neon.h:123-140` | 5 | `nk_sqeuclidean_f32_neon` — ONE chain, 2 f32/iteration | +| `spatial/neon.h:142-145` | 5 | `nk_euclidean_f32_neon` — one sqrt for the whole vector | +| `dot/neon.h:40-60` | 6 | the streaming example: one query load, FOUR target loads, four states, one `float32x4_t` of results | +| `dot/neon.h:230-246` | 6 | the state struct (one `float64x2_t`) and its 2-element update | +| `dot/neon.h:154-159` | 7 | the FCMLA comment: 39.7 vs 17.1 GiB/s on M4 at n=4096 | +| `c/dispatch.h:10, 23-27, 36-45` | 8 | `NK_DYNAMIC_DISPATCH`, the macOS/Apple-Clang capability row, the fn-pointer struct | +| `c/numkong.c:832-843, 915-919` | 8 | memoized detection, the >100-cycle CPUID comment, the load-time constructor | + +Reading order: the table at the top of `dot/neon.h` first — it is the +real reading assignment — then `nk_dot_f32_neon` and the streaming +doc-comment above it, then `spatial/neon.h`'s table and its two rsqrt +helpers, then the FCMLA comment, then the two dispatch files. Finish +by opening one x86 sibling (`dot/haswell.h`) and finding the same +kernel re-derived from a different table; you should be able to +predict its accumulator count before you read it. ## Questions for notes.md -1. From the table: peak f32 FMA throughput on M-series = 4 ports × - 4 lanes × 2 flops = 32 flops/cy. What fraction does - nk_dot_f32_neon reach, given f64 accumulation halves lanes? -2. sqeuclidean_f32 uses ONE f64x2 chain (spatial/neon.h:123) — - sloppy, or is L2-distance latency-bound elsewhere? Predict, then - check with your dot.rs bench. -3. Newton-Raphson: why 3 rounds for f64 (~48 bits) — how many bits - does each round double from FRSQRTE's ~8-bit estimate? -4. The stateful API returns `float32x4_t` of 4 results — how does - this shape M14's candidate-scoring loop signature? -5. For M17 dispatch: sketch the fn-pointer table for - {dot, l2sq, filter} × {neon, scalar} and where - `is_aarch64_feature_detected!` runs exactly once. +1. From `dot/neon.h:14-15`, compute peak f32 and f64 FMA throughput on + M5 in flops/cycle, then compute what fraction of the f64 figure + `nk_dot_f32_neon` reaches with its two chains. Step 3 shows the + arithmetic; redo it for a hypothetical four-chain version and say + what stops SimSIMD from shipping that. +2. `nk_sqeuclidean_f32_neon` (`spatial/neon.h:123-140`) uses ONE f64 + chain over `float32x2_t` loads — 0.5 elements/cycle by Step 4's + method. Sloppy, or is L2 memory-bound before it is issue-bound at + the sizes it runs on? Predict from Step 2's 42 GB/s ceiling, then + check with `dot.rs`. +3. Newton-Raphson: `spatial/neon.h:57-59` uses two rounds and + `105-115` uses three. Write out the doubling ladder from both + claimed starting accuracies (~8 bits at line 22, 11.4 bits from + line 106's `1.5×2⁻¹²`) and say which round count each justifies for + f32 and f64. Then say why line 68's advice — "for full 52-bit + mantissa fidelity, prefer `vsqrtq_f64`" — is consistent with the + ladder rather than a contradiction of it. +4. The streaming API returns a `float32x4_t` of 4 results + (`dot/neon.h:59`). Sketch M14's candidate-scoring loop signature + around it: what does the caller do with a neighbour list of 32, + and where does the Step 4 reduce cost land in that loop? +5. Apply Step 7's test to this topic's own kernel: `notes.md`'s + implementation log records `count_neon + compact_neon` (LUT built, + all 16 masks pass). Does the `vqtbl1q` table lookup beat the + branchless store it replaces, by the ports-and-latency argument + rather than by instruction count? Predict, then measure. +6. For M17's dispatch: sketch the fn-pointer table for + {dot, l2sq, filter} × {neon, scalar}, say where + `is_aarch64_feature_detected!` runs exactly once, and compute the + break-even call count against Step 8's >100-cycle detection cost + for a kernel of your chosen size. ## Done when -- [ ] You can read the port/latency table and compute peak f32 FMA throughput for M-series from it. -- [ ] You can explain why precision is bought with wider accumulators rather than with reordering. -- [ ] You can say why batching candidates beats unrolling pairs. -- [ ] You can state the FCMLA lesson: a specialized instruction must beat the table, not merely exist. -- [ ] You can sketch the function-pointer dispatch table and say when the ISA choice is made. -- [ ] You wrote answers to all five questions in notes.md, including the Newton-Raphson round count for f64. +Answer each before unfolding it. + +- [ ] You can read the port/latency table and compute both the chain count and the peak f32 FMA throughput for M5 from it. + +
Answer + + From `dot/neon.h:14`, M5 column, `vfmaq_f32` is `3cy @ 4p`. Chains + needed = latency × ports = 3 × 4 = **12**. Peak f32 flops = 4 pipes + × 4 lanes × 2 flops per FMA = **32 flops/cycle**. For f64 + (line 15, `4cy @ 4p`): 4 × 4 = **16 chains**, and 4 × 2 × 2 = + **16 flops/cycle**. The A76 column gives 8 chains and 16 f32 + flops/cycle from the same rows — same file, different machine, + different design. + +
+ +- [ ] You can derive this machine's clock from `notes.md` rather than assuming one. + +
Answer + + `notes.md`'s naive dot rung is 10.89 GB/s over both inputs, i.e. + 8 bytes per element-pair, so 1.361e9 pairs/s. `dot_naive` + (`experiments/src/dot.rs:10-17`) is one scalar accumulator = one + chain, so it advances one pair per FMA latency = 3 cycles + (`dot/neon.h:14`, M5). Clock ≥ 1.361e9 × 3 = **4.08 GHz**. It is a + lower bound because loop overhead is charged to useful work. The + host really is an M5 (`sysctl -n machdep.cpu.brand_string`), so the + table's `M5` column applies directly. + +
+ +- [ ] You can explain why precision is bought with wider accumulators rather than with reordering, and price the choice. + +
Answer + + polars restructures the addition *order* (pairwise summation); + SimSIMD restructures the *precision*, upcasting f32 to f64 with + FCVTL (`dot/neon.h:135-138`) and accumulating in `float64x2_t` + (129-130). Cost: 2 chains × 4 f32 per iteration per 4 cycles = + 1 pair/cycle, against 16 pairs/cycle for a 12-chain pure-f32 kernel + — **16× in peak issue**. But 1 pair/cycle at 4.08 GHz × 8 B = + 32.6 GB/s, against this topic's measured 42.1 GB/s ceiling, so the + real cost at DRAM-resident sizes is about **23 %**. It buys error + control that no reordering can match, in a library whose callers + cannot debug float cancellation from Python. + +
+ +- [ ] You can say which instruction in the table got *worse* from A76 to M5, and what the kernels do about it. + +
Answer + + `vaddvq_f32`, the f32 horizontal reduce (`dot/neon.h:19`): 5cy@1p on + A76, **8cy@1p** on M5 — every other row improved. `vaddvq_f64` + (line 20) stayed at 3cy@1p, so it is 2.7× cheaper on M5, and + `nk_dot_f32_neon:142` uses it — a free dividend from the Step 3 + precision choice. Line 27 states the general rule: reductions stay + at 1/cy on all cores and become the main bottleneck. Concretely, at + n = 8 the ~6-cycle reduce is 43 % of a 14-cycle kernel; at n = 1536 + it is 0.39 %; break-even at 5 % overhead is around n = 120. + +
+ +- [ ] You can say why batching candidates beats unrolling pairs, with the load count and the chain count. + +
Answer + + The streaming loop (`dot/neon.h:47-57`) issues 5 loads per iteration + (1 query at line 48, 4 targets at 49-52) for 4 states × 2 elements = + 8 element-pairs, versus 8 loads for the same 8 pairs when the four + dots run separately — **37.5 % fewer loads**, because the query is + read once instead of four times. Chains go from 2 to **4** (one + `float64x2_t` per state, `dot/neon.h:230-232`), so throughput goes + from 1.0 to **2.0 pairs/cycle**. Unrolling cannot do this when + `depth` is small: at depth 128 there are only 32 iterations to + unroll over, while batching adds chains without needing any. Four + candidates is still 4 of 16 chains, so eight would help again. + +
+ +- [ ] You can state the FCMLA lesson precisely, including whose number 39.7 GiB/s actually is. + +
Answer + + `dot/neon.h:154-159`: on an **Apple M4 at n = 4096**, the manual + deinterleave path (`vld2_f32` + 4 independent FMAs, 5 instructions + per 2 complex pairs) hit **39.7 GiB/s** while FCMLA (2 `vcmlaq`, + 2 instructions) hit **17.1 GiB/s** — 2.32× slower with 2.5× fewer + instructions, because 4 FMAs are 4 chains that fill 4 pipes. + Crucially, 39.7 belongs to the *rejected f32 variant*, not to the + shipped kernel: line 159 says the shipped code upcasts to f64 and + "FCMLA offers neither advantage". Also GiB/s (2^30), not the + decimal GB/s that `notes.md` uses — 39.7 GiB/s = 42.6 GB/s. + +
+ +- [ ] You can sketch the function-pointer dispatch table, say when the ISA choice is made, and justify it against the alternatives. + +
Answer + + `c/dispatch.h:37-45` declares a struct with one + `nk_metric_dense_punned_t` per (kernel, dtype) — `dot_f32`, + `dot_f64`, `dot_bf16c`, and so on. `c/numkong.c:917-919` marks + `nk_auto_init` as `__attribute__((constructor))`, so it runs at + library load; it calls `nk_capabilities()` + (`c/numkong.c:832-843`), which memoizes in a `static` at line 834 + and calls `nk_dispatch_table_init()` at 840. Line 833 gives the + motive: CPUID can cost over 100 cycles. That is **init-time** + binding — between hashbrown's compile-time `cfg_if!` (zero cost, + but the dispatched unit is a few instructions) and polars' + per-call `is_x86_feature_detected!` (one branch, but the dispatched + unit is a whole column). SimSIMD's unit is one distance over + hundreds of dimensions, big enough to absorb an indirect call and + small enough that a >100-cycle probe per call would be 7× the + kernel at n = 8. + +
+ +- [ ] You found at least one place where SimSIMD's prose is looser than its own table, and can say what the code actually does. + +
Answer + + Three candidates, all checkable. (a) `spatial/neon.h:22` says + FRSQRTE gives "~8-bit precision" while line 106 cites third-party + measurement of `1.5×2⁻¹²` error = 11.4 bits, with the issue link at + 108. (b) Line 54's "2 cy vs 9-12 cy latency, 2/cy vs 0.25/cy + throughput" is the **A76** comparison; line 18's M5 column has + FRSQRTE at `3cy @ 1p`, the same single port as FSQRT. (c) The same + line compares a bare FRSQRTE against a complete FSQRT, when the + usable f32 result needs the two refinement rounds at lines 58-59; + FRSQRTS has no row in the table, so the honest total cannot be + computed from this file. None of these change a design decision — + the throughput argument survives all three — but the f32 path is + **two** rounds to ~23 bits (57-59), not three, and the three-round + ~48-bit version is the separate f64 helper at 110-115. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the Newton-Raphson round counts for f32 and f64. + +
Answer + + The round counts: **two** rounds for f32 (`spatial/neon.h:57-59`), + because 8 → 16 → 32 bits saturates the 24-bit f32 significand and + the header records ~23 (line 53); **three** rounds for f64 + (`spatial/neon.h:110-115`), because 8 → 16 → 32 → 64 is needed to + approach the 53-bit f64 significand and the header records ~48 + (line 67). Each round roughly doubles the correct bits because + Newton's method on this function converges quadratically. Line 68 + is consistent: if you need the full 52-bit mantissa rather than ~48, + the ladder cannot get you there cheaply and you should just issue + `vsqrtq_f64`. + +
## References **Code** -- [SimSIMD](https://github.com/ashvardanian/SimSIMD) — - `include/numkong/` — one file per ISA per kernel family - (`dot/neon.h`, `spatial/neon.h`, sve/haswell/skylake siblings); - the port/latency tables at the top of each NEON header are the - real reading assignment +- [SimSIMD](https://github.com/ashvardanian/SimSIMD) at the pinned + revision `63a254f` (`resources/codebases.md`) — headers under + `include/numkong/`, one file per ISA per kernel family + (`dot/neon.h`, `spatial/neon.h`, and their `sve`/`haswell`/`skylake` + siblings). The port/latency tables at the top of each NEON header + (`dot/neon.h:11-27`, `spatial/neon.h:11-26`) are the real reading + assignment; the dispatch machinery is `c/dispatch.h` and + `c/numkong.c:829-843, 915-919`. + +**This repo** +- `topics/17-simd/notes.md` — the `dot` lane (naive 10.89 GB/s, + unrolled-8 42.12 GB/s, N = 4M f32, Apple Silicon, measured + 2026-07-10) that Steps 2 and 3 compute against. `FINDINGS.md` row 17 + records a different run of the same bench (8.88 → 26.32 GB/s); cite + whichever you use by name and never average them. +- `topics/17-simd/experiments/src/dot.rs` — `dot_naive` (one chain) + and `dot_unrolled8` (eight) are the two rungs Step 2 uses. +- `reading-polars-compute.md` (pairwise summation; per-call dispatch), + `reading-hashbrown-simd.md` (compile-time dispatch), + `reading-sigmod15-vectorization.md` and `reading-simdjson.md` (both + cite Step 2's derived 4.08 GHz for their cycle counts). + +**Hardware** +- Host for every "this machine" claim above: Apple M5 + (`sysctl -n machdep.cpu.brand_string`), aarch64, 128-bit NEON, no + SVE exposed and no AVX-512 — so the `M5` column of SimSIMD's tables + is the one that applies, and `c/dispatch.h:27` explains why only the + NEON kernels are even compiled here. diff --git a/topics/18-gpu/README.md b/topics/18-gpu/README.md index 3fb0a55..b5ad162 100644 --- a/topics/18-gpu/README.md +++ b/topics/18-gpu/README.md @@ -72,8 +72,11 @@ happens. fill second. GPU code can't `Vec::push` — every output needs its size known or an atomic cursor. - Group-by: shared-memory aggregation per block when cardinality - fits (groupby/hash/compute_shared_memory_aggs.cu), spilling to - global-memory atomics when it doesn't — topic 11's two-phase + fits (groupby/hash/compute_shared_memory_aggs.cu). It does *not* + spill: `compute_single_pass_aggs.cuh:95-122` sets a device + `atomic_flag` when a block overflows, copies it to the host, + synchronizes, and re-runs the **whole** aggregation in global + memory — all-or-nothing, decided on the host. Topic 11's two-phase partial aggregation, forced by the memory hierarchy. ## 5. GPU graph processing (Gunrock) & ANN (CAGRA) diff --git a/topics/18-gpu/experiments/.gitignore b/topics/18-gpu/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/18-gpu/experiments/.gitignore +++ b/topics/18-gpu/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/18-gpu/reading-cagra.md b/topics/18-gpu/reading-cagra.md index ca73773..45b0145 100644 --- a/topics/18-gpu/reading-cagra.md +++ b/topics/18-gpu/reading-cagra.md @@ -1,202 +1,711 @@ # CAGRA: HNSW rebuilt for warps -Topic 14's HNSW rebuilt from GPU-first principles: what does a -graph-traversal index look like when the executor is 32-wide warps -instead of one pointer-chasing core? This chapter builds the answer -step by step — what a proximity-graph search is, what SIMT hates -about HNSW, and the three fixes (flatten the levels, fix the degree, -move the visited set into shared memory) — a case study in making an -irregular algorithm regular enough for SIMT. The cuVS implementation -is the code half of this chapter. +Topic 14's HNSW asked "what index makes a pointer-chasing core fast?" CAGRA asks +the same question of a machine that executes 32 lanes in lockstep and answers +differently at every level: no hierarchy, fixed out-degree, a visited set that +fits in shared memory and is periodically *thrown away* on purpose. This chapter +builds those choices in order and shows each one in the code that implements it. + +You cannot run any of it here. cuVS is CUDA-only and this machine has no CUDA +device — the topic's runnable lane is wgpu/Metal, and its measured numbers +(`notes.md:9-16`) are the transfer tax, not ANN throughput. Every performance +figure below is quoted from the paper with the hardware it was measured on, and +every structural claim is quoted from source. Nothing here is a local +measurement, and nothing here should be repeated as one. + +Code anchors are [rapidsai/cuvs@8b97b61](https://github.com/rapidsai/cuvs); +check any of them with `python3 tools/pinned-source.py show cuvs -r A:B`. +Paper citations are to [arXiv:2308.15136v2](https://arxiv.org/abs/2308.15136) +(9 Jul 2024) — the version that matches the ICDE 2024 paper. The v1 preprint +(Aug 2023) reports different numbers; quote the one you actually read. ## The problem in one sentence -HNSW's greedy walk is one long chain of dependent random loads with -variable-degree nodes — the exact shape a 32-lane lockstep warp -executes worst — yet CAGRA reaches the same recall with ~10× faster -index build and an order-of-magnitude more queries per second. +HNSW's greedy walk is a chain of dependent random loads over variable-degree +nodes — the exact shape a 32-lane lockstep warp executes worst — and CAGRA's +answer is to delete every irregularity at build time, then spend the search +budget on a shared-memory hash table it can afford to forget. ## The concepts, step by step -### Step 1 — proximity-graph ANN: search is a greedy walk - -ANN (approximate nearest neighbor) search finds the k vectors -closest to a query without scanning all of them. Graph-based indexes -like HNSW (topic 14) connect each vector to a few dozen near -neighbors; search starts at an entry vertex and **greedily walks**: -compute distances to the current vertex's neighbors, move to the -closest unvisited one, repeat until no neighbor beats what you have. -Two supporting structures make it work: a **candidate list** of the -best vertices seen so far (HNSW's beam of width `ef` — wider beam, -better recall, more work), and a **visited set** so the walk never -scores the same vertex twice. HNSW additionally stacks sparse upper -**levels** (a skip-list-like hierarchy) to find a good entry point -in O(log n) hops. - -### Step 2 — what SIMT hates about that walk - -A warp is 32 threads executing one instruction in lockstep; it is -efficient only when all lanes do identical work on adjacent data. -Score HNSW against that executor, feature by feature — and CAGRA's -index is HNSW with each offending feature deleted: - -``` - HNSW (topic 14) CAGRA - multi-level skip list SINGLE flat level - variable degree ≤ M FIXED degree (e.g. 32) — no ragged - adjacency, no load balancing needed - (Gunrock's whole problem, deleted - by construction) - greedy walk, 1 candidate parallel walk, itopk candidate list, - beam ef search_width parents expanded/iter - visited: hash set on heap visited: hashmap in SHARED MEMORY -``` - -Fixed degree is the load-bearing change: one warp loads a neighbor -list in exactly one coalesced pass (adjacent lanes, adjacent -addresses, one memory transaction), and lane i always has lane-i -work. Question: what does fixed degree cost in graph quality, and -how does build compensate (rank-based pruning + detour counting in -graph_core.cuh — keeping the edges that SHORTCUT most 2-hop paths)? - -### Step 3 — build by NN-descent, not insert-one-at-a-time - -HNSW builds incrementally, one insert at a time — inherently serial -(topic 14's build took minutes). CAGRA builds the whole graph at -once with **NN-descent**: start every vertex with random candidate -neighbors, then iterate "my neighbors' neighbors are probably my -neighbors too" — a fixpoint of local refinement where every vertex -improves its list independently, embarrassingly parallel. Then prune -each list to the fixed degree, preferring edges that shortcut many -2-hop paths (detour counting). Paper's headline: build is ~10× -faster than HNSW at equal recall. Question: NN-descent is itself a -graph algorithm with ragged intermediate state — how does the paper -make ITS memory usage bounded (fixed-size candidate lists again)? - -### Step 4 — search: one CTA per query, parallel within each step - -A **CTA** (cooperative thread array — CUDA's thread block, a few -hundred threads sharing ~100 KB of fast scratch **shared memory**) -cooperates on ONE query in search_single_cta_kernel.cuh: - -``` - shared memory holds: itopk candidate list + visited hashmap - + distance scratch (:127-143 budgets this) - loop until itopk stable: - pick search_width best unvisited parents (bitonic/radix topk) - ALL threads: load their fixed-degree neighbors, compute - distances in parallel (one lane ≈ one neighbor) - dedupe via shared hashmap, merge into itopk -``` - -The greedy walk is still SEQUENTIAL across iterations — parallelism -is WITHIN each step (32–64 distance computations at once, expanding -`search_width` parents per iteration instead of HNSW's one) plus -ACROSS queries (one CTA each, thousands resident): - -```rust -// one CTA per query; the walk is sequential, each STEP is parallel -while !itopk.stable() { - let parents = itopk.best_unvisited(SEARCH_WIDTH); // bitonic/radix topk - par_for lane in 0..(SEARCH_WIDTH * DEGREE) { // one lane ≈ one neighbor - let v = graph[parents[lane / DEGREE]][lane % DEGREE]; - // FIXED degree ⇒ this load is one coalesced pass, no load balancing - if visited.insert(v) { // shared-memory hashmap - dist[lane] = l2(query, data[v]); - } - } - itopk.merge(dist); // shared-memory topk -} -``` - -The itopk list is maintained by bitonic/radix top-k — sorting -networks with a fixed compare-exchange schedule, not a heap, because -data-dependent branching diverges the warp. Question: batch size 1 -uses a fraction of the device; batch 10K saturates it — how does -that reshape M14's "QPS at recall" curve axes (GPU ANN is a -THROUGHPUT device: latency per query barely improves, queries per -second explode)? - -### Step 5 — the visited set becomes a shared-memory hashmap - -The visited set is queried on every candidate, so it must live in -the fastest memory the CTA owns — but shared memory is ~100 KB, -shared with the itopk list and distance scratch, so a bitmap over a -million vertices (125 KB) doesn't fit. hashmap.hpp packs an -open-addressing hash table into that budget, sized by -hashmap_min_bitlen / max_fill_rate (search_single_cta.cuh:57-59). -Collisions → a false "already visited" is acceptable (skip a node, -lose a bit of recall) but the reverse isn't tracked... check: is it -lossy or exact? Question: compare topic 14's visited-set choices -(bitmap vs hash set per query) — why does shared-memory capacity -force the hash here, and what happens to recall when the table -saturates on a long search? - -### Step 6 — what transfers to M18 - -Vector distance scoring is our engine's most GPU-shaped op (dense, -regular, high arithmetic intensity — the l2_batch stub is its -kernel). CAGRA's lesson: if you also want the TRAVERSAL on device, -you must first make the graph regular — fixed degree, flat levels, -bounded scratch. FalkorDB's CSR adjacency is not regular — which is -why M18's flag gates distance scoring, not traversal. +### Step 1 — the greedy walk, and the two structures it needs + +> **In:** a query vector and a proximity graph over N vectors. +> **Out:** k approximate nearest neighbours, plus the two auxiliary structures +> every graph ANN search must maintain. + +Graph ANN search starts at some entry node, scores that node's neighbours, moves +to the best unvisited one, and repeats until nothing improves. Two structures +ride along: + +- a **candidate list / internal top-M** — the best M nodes seen so far, M ≥ k + (HNSW calls its width `ef`); wider means better recall and more work; +- a **visited set**, so a node reachable from three different directions is + scored once, not three times. + +CAGRA's buffer is both, adjacent in memory: *"a sequential memory buffer +consisting of an internal top-M list … and its candidate list … The length of +the internal top-M list is M (≥ k), and the candidate list is p × d"* (§IV-A), +where *p* is the number of parents expanded per iteration and *d* the graph's +fixed degree. That is exactly the code: + +```cpp +// cpp/src/neighbors/detail/cagra/search_single_cta.cuh:106-115, the head of +// set_params — the paper's M is itopk_size, p is search_width, d is +// graph_degree. + 106 inline void set_params(raft::resources const& res) + 107 { + 108 num_itopk_candidates = search_width * graph_degree; + 109 result_buffer_size = itopk_size + num_itopk_candidates; + 111 typedef raft::Pow2<32> AlignBytes; + 112 unsigned result_buffer_size_32 = AlignBytes::roundUp(result_buffer_size); + 114 constexpr unsigned max_itopk = 512; + 115 RAFT_EXPECTS(itopk_size <= max_itopk, "itopk_size cannot be larger than %u", max_itopk); +``` + +HNSW additionally stacks sparse upper levels to find a good entry point in +O(log N) hops. CAGRA does not, and says why: *"in the case of GPU, we can obtain +compatible initial nodes by randomly picking some nodes and comparing their +distances to the query, thus employing the high parallelism and memory bandwidth +of GPU"* (§III). A hierarchy is a way to spend few distance computations; a GPU +would rather spend many in parallel. + +### Step 2 — what SIMT hates, and what CAGRA deletes + +> **In:** HNSW's structure. +> **Out:** the same structure with every irregularity removed — and the defaults +> that say how far. + +A warp is 32 lanes issuing one instruction; it is efficient when all lanes do +identical work on adjacent addresses. Score HNSW feature by feature: + +``` + HNSW (topic 14) CAGRA + multi-level skip list one flat graph, random entry points (§III) + variable degree <= M FIXED out-degree d, every list identical + 1 candidate expanded/iter p parents expanded per iteration + visited: heap-allocated set visited: hash table in shared memory (Step 6) +``` + +Fixed degree is the load-bearing deletion, and it is worth naming what it buys +in this repo's vocabulary: it deletes Gunrock's entire research problem. There is +no ragged frontier, no `thread_mapped`-vs-`merge_path` choice, no prefix scan to +size the output — every parent contributes exactly *d* children, so the output +size is `p × d` before you look (`search_single_cta.cuh:108`). Regularity bought +at build time, spent at every search. + +The defaults are bigger than most descriptions of CAGRA (including this guide's +previous version, which said "e.g. 32") suggest: + +```cpp +// cpp/include/cuvs/neighbors/cagra.hpp:149-153 — build defaults. + 149 struct index_params : cuvs::neighbors::index_params { + 150 /** Degree of input graph for pruning. */ + 151 size_t intermediate_graph_degree = 128; + 152 /** Degree of output graph. */ + 153 size_t graph_degree = 64; +``` + +```cpp +// cpp/include/cuvs/neighbors/cagra.hpp:291-318 — search defaults, elided. + 291 size_t itopk_size = 64; + 294 size_t max_iterations = 0; + 300 search_algo algo = search_algo::AUTO; + 303 size_t team_size = 0; + 307 size_t search_width = 1; + 312 size_t thread_block_size = 0; + 314 hash_mode hashmap_mode = hash_mode::AUTO; + 316 size_t hashmap_min_bitlen = 0; + 318 float hashmap_max_fill_rate = 0.5; +``` + +The paper's experiments sweep d ∈ {32, 48, 64, 80} with the initial graph at +`d_init = 3d` (Fig. 3 caption), which is where the library's 64/128 pair comes +from. Note `search_width = 1`: the default expands **one** parent per iteration, +and the paper confirms the intent — *"we typically set p = 1 to maximize the +throughput of single-CTA"* (§IV-C2). The parallelism is across queries, not +across parents. + +### Step 3 — build: NN-descent, then two graph surgeries + +> **In:** N raw vectors. +> **Out:** a fixed-degree graph that is strongly connected — built 2.2–27× +> faster than HNSW's, on the hardware named below. + +HNSW builds by inserting one vector at a time, each insert a search into the +graph built so far: sequential by construction. CAGRA builds the whole k-NN +graph at once by **NN-descent** (*"my neighbours' neighbours are probably my +neighbours"* — a fixpoint of independent local refinements, §III), then +*optimises* it in two passes. + +**Pass 1, rank-based reordering.** An edge X→Y is redundant if some Z gives a +two-hop route that is short enough — NGT's criterion, quoted as Eq. 3 in §III-A. +Counting *detourable routes* per edge and keeping the least-detourable edges is +the pruning rule. CAGRA's twist is to rank by **position in the neighbour list** +rather than by distance: *"we approximate the distance by the initial rank. This +approximation allows us not to compute the impractical amount of distance +computations and not to store the large size of the distance table in memory"* +(§III-A). The cost, stated in the same section: distance-based reordering needs +`N × d_init × (d_init − 1)` distance computations or an `N × d_init` table; both +reorderings are O(N d³). Measured payoff: rank-based is *"faster than the +distance-based for all datasets by as much as 1.9×"*, and distance-based ran out +of memory on DEEP-100M where rank-based did not (§V-A, Q-A2, Fig. 4). + +The triple loop is right there in the kernel, and note what it compares: + +```cpp +// cpp/src/neighbors/detail/cagra/graph_core.cuh:259-277, inside +// kern_fused_prune — counting A->D->B detours. No distances are read. + 259 // count number of detours (A->D->B) + 260 for (uint32_t kAD = 0; kAD < knn_graph_degree - 1; kAD++) { + 261 const uint64_t iD = smem_indices[kAD]; + 262 if (iD >= graph_size) { continue; } + 263 for (uint32_t kDB = lane_id; kDB < knn_graph_degree; kDB += raft::WarpSize) { + 264 const uint64_t iB_candidate = knn_graph(iD, kDB); + 265 for (uint32_t kAB = kAD + 1; kAB < knn_graph_degree; kAB++) { + 267 { + 268 const uint64_t iB = smem_indices[kAB]; + 269 if (iB == iB_candidate) { + 270 atomicAdd(smem_num_detour + kAB, 1); + 271 break; + 272 } + 273 } + 274 } + 275 } + 276 warp.sync(); + 277 } +``` + +**Pass 2, reverse edges.** Pruning leaves a directed graph in which some nodes +are unreachable. CAGRA reverses the pruned graph — *"Someone who considers you +are more important is also more important to you"* — at O(N d), then merges: +*"we basically take d/2 children for each parent node from each graph and +interleave them"* (§III-A; `kern_merge_graph`, `graph_core.cuh:375`). Fig. 3 +measures which pass does what: the raw k-NN graphs have between 90 and 105,333 +strongly connected components, and adding reverse edges brings every dataset to +between 1 and 26 — reordering alone never does. Reordering raises the 2-hop +count; reverse edges make the graph strongly connected. Two surgeries, two +different jobs. + +**The build headline, correctly stated.** The abstract's number is *"2.2–27× +faster than HNSW, which is one of the CPU SOTA implementations"* — measured on a +DGX A100 with an AMD EPYC 7742 (64 cores) and an A100 80 GB, with dataset and +graph resident in device memory (§V-A). This guide previously said "~10×"; that +figure is in neither the abstract nor §V. And read the residency clause the way +`reading-crystal-sigmod20.md` teaches: a GPU number quoted without saying where +the data lives is not a number. + +### Step 4 — search: one CTA per query, and warps split into teams + +> **In:** a batch of queries and a fixed-degree graph in device memory. +> **Out:** one thread block per query, one team of lanes per distance +> computation, and a loop whose iterations are serial but whose steps are not. + +A **CTA** (cooperative thread array — CUDA's thread block, 64–1024 threads with +private shared memory) owns one query and runs the entire search in one kernel. +The paper tried the obvious alternative and rejected it: *"extensive testing +revealed that the overhead of launching multiple kernels outweighs any potential +performance gains"* (§IV-C1) — the same per-dispatch tax this topic measures at +1544 µs on its runnable lane (`notes.md:11-14`), paid per *iteration* instead of +per search. + +The loop body, elided to its four phases: + +```cpp +// cpp/src/neighbors/detail/cagra/jit_lto_kernels/search_single_cta_jit.cuh: +// 256-291 — one iteration: pick parents, maybe reset the hash, score children. + 256 // pick up next parents + 257 if (threadIdx.x < 32) { + 259 pickup_next_parents( + 260 terminate_flag, parent_list_buffer, result_indices_buffer, internal_topk, search_width); + 262 } + 264 // restore small-hash table by putting internal-topk indices in it + 265 if ((iter + 1) % small_hash_reset_interval == 0) { + 266 const unsigned first_tid = ((blockDim.x <= 32) ? 0 : 32); + 268 hashmap_restore( + 269 local_visited_hashmap_ptr, hash_bitlen, result_indices_buffer, internal_topk, first_tid); + 271 } + 272 __syncthreads(); + 274 if (*terminate_flag && iter >= min_iteration) { break; } + 279 compute_distance_to_child_nodes_jit( + 280 result_indices_buffer + internal_topk, + 281 result_distances_buffer + internal_topk, + 291 search_width); +``` + +Parent selection is done by **32 threads only** (`:257`) — one warp, because +`pickup_next_parents` is a warp-level operation and the rest of the block would +only contend. The walk is still sequential across iterations; the parallelism is +*within* an iteration (`p × d` distance computations at once) and *across* +queries (thousands of CTAs resident). + +**Warp splitting** is the trick that keeps lanes busy inside one distance +computation. §IV-B1 does the arithmetic explicitly for a 96-dimensional float +dataset; here it is with the inputs named: + +``` + one 128-bit load instruction per lane, 32 lanes in a warp + warp-wide load width = 32 x 128 bits = 4096 bits + + dataset vector, dim 96, float32 + vector width = 96 x 32 bits = 3072 bits + ---- + waste if 1 warp = 1 distance = 4096 - 3072 = 1024 bits idle (25%) + + team of 8 lanes (team_size = 8) + team load width = 8 x 128 bits = 1024 bits + loads per vector = 3072 / 1024 = 3 (exact) + teams per warp = 32 / 8 = 4 distances in flight +``` + +*"Although we split the warp into teams in software, we don't encounter warp +divergence since all of the teams in each warp still execute the same +instructions"* (§IV-B1). `team_size = 0` in the defaults means "choose for me" +(`cagra.hpp:302-303`, which documents the legal values as 4, 8, 16 or 32), and +Fig. 8 shows the choice is worth a large constant factor on both DEEP-1M +(dim 96) and GIST (dim 960). + +### Step 5 — top-M without a heap, and the 512 that appears twice + +> **In:** a buffer holding `itopk_size + p×d` (index, distance) pairs, partially +> sorted. +> **Out:** the new top-M — chosen by a sorting network, not a priority queue. + +A heap is the CPU's answer (topic 14) and the worst possible warp code: every +sift-down is a data-dependent branch, so lanes diverge and the warp serialises. +CAGRA uses **bitonic sort** — a fixed compare-exchange schedule, identical for +every lane, executable in registers — and only falls back to a radix top-k when +the buffer is too big for registers: + +> *"we first sort the candidate buffer and merge it with the internal top-M +> buffer through the merge process of the bitonic sort. We use the single +> warp-level bitonic sort when the candidate buffer size is less or equal to +> 512, while we use a radix-based sort using within a single CTA when it is +> larger than 512."* (§IV-B2) + +The pinned code switches at a different number. Twice, in the same function: + +```cpp +// cpp/src/neighbors/detail/cagra/search_single_cta.cuh:134 and 161-169 — the +// radix decision and its block-size consequence. The paper says 512. + 134 if (num_itopk_candidates > 256) { // radix sort + 161 if (num_itopk_candidates > 256) { // radix sort + 162 // radix-based topk is used. + 163 block_size = min_block_size_radix; + 165 // Internal topk values per thread must be equlal to or less than 4 + 166 // when radix-sort block_topk is used. + 167 while ((block_size < max_block_size) && (max_itopk / block_size > 4)) { + 168 block_size *= 2; + 169 } + 170 } +``` + +Report the discrepancy rather than resolving it: the paper's threshold is 512 on +the *candidate buffer*, the code's is 256 on `num_itopk_candidates = +search_width × graph_degree` (`:108`), and the code additionally forces a +256-thread minimum block when radix is chosen. With the defaults (`search_width += 1`, `graph_degree = 64`) `num_itopk_candidates` is 64, so the bitonic path is +what runs. + +512 shows up again as the itopk ceiling (`max_itopk`, `:114`) and again in the +implementation-choice rule — where the paper's recommendation and the code +differ by a factor of two: + +```cpp +// cpp/src/neighbors/detail/cagra/search_plan.cuh:122-130 — algo = AUTO. + 122 } else if (algo == search_algo::AUTO) { + 123 const size_t num_sm = raft::getMultiProcessorCount(); + 124 if (itopk_size <= 512 && search_params::max_queries >= num_sm * 2lu) { + 125 algo = search_algo::SINGLE_CTA; + 127 } else { + 128 algo = search_algo::MULTI_CTA; + 130 } + 131 } +``` + +The paper recommends *"MT = 512 and bT = 'the number of SMs on the GPU'"* +(§IV-C3); the code demands **twice** the SM count of queries before it will use +single-CTA. Both agree on the shape: too few queries to fill the device, or too +large an internal top-M, and you switch to multi-CTA, which splits one query +across many blocks and moves the hash table to device memory (Table II). + +The split is concrete, and it costs a second kernel. `search_multi_cta.cuh:117-126` +pins each CTA's internal list to a hard-coded 32 entries and derives the CTA count +from the itopk you asked for — `num_cta_per_query = max(search_width, +ceildiv(global_itopk_size, 32))`, so `itopk_size = 128` becomes 4 CTAs. Those CTAs +cannot merge their lists in place, because CUDA gives you no device-wide barrier +inside a kernel. So they write `num_cta_per_query * itopk_size` candidates to +device memory and a *separate* `_cuann_find_topk` launch reduces them after the +search kernel has fully returned (`search_multi_cta.cuh:246-265`). That extra +launch is exactly the "overhead of launching multiple kernels" the paper's §IV-C1 +spends the whole single-CTA design avoiding — multi-CTA pays it because with too +few queries the device would otherwise sit idle. + +### Step 6 — the forgettable hash table + +> **In:** a visited set that must be checked on every candidate. +> **Out:** an *exact* open-addressing table small enough for shared memory, +> periodically wiped — and a computed reset interval. + +The old version of this guide asked, next to this table, "check: is it lossy or +exact?" Here is the answer, from the code: + +```cpp +// cpp/src/neighbors/detail/cagra/hashmap.hpp:15 and 37-60 — insert(). Linear +// probing is unconditional: the #define at line 15 is not guarded. + 15 #define HASHMAP_LINEAR_PROBING +... + 41 // Open addressing is used for collision resolution + 42 const uint32_t size = get_size(bitlen); + 43 const uint32_t bit_mask = size - 1; + 44 #ifdef HASHMAP_LINEAR_PROBING + 45 // Linear probing + 46 IdxT index = (key ^ (key >> bitlen)) & bit_mask; + 47 constexpr uint32_t stride = 1; + 53 constexpr IdxT hashval_empty = ~static_cast(0); + 55 for (unsigned i = 0; i < size; i++) { + 56 const IdxT old = atomicCAS(&table[index], hashval_empty, key); + 57 if (old == hashval_empty) { + 58 return 1; + 59 } else if (old == key) { + 60 return 0; +``` + +**Exact.** Open addressing stores the full key and compares it, so a collision +costs a probe, never a wrong answer; `atomicCAS` makes concurrent inserts by +different lanes safe and returns 1 exactly once per key. The only failure mode +is a *full* table: the loop at `:55` gives up after `size` probes and returns 0 +— "already visited" — so a saturated table silently starts skipping nodes. +Recall degrades; correctness of the data structure does not. + +Which is precisely why the table is sized to stay unsaturated, and reset when it +cannot be: + +```cpp +// cpp/src/neighbors/detail/cagra/search_plan.cuh:298-305 and 324-330 — +// small-hash sizing and the reset interval, in calc_hashmap_params. + 298 const auto max_visited_nodes = itopk_size + (search_width * graph_degree * 1); + 299 unsigned min_bitlen = 8; // 256 + 300 unsigned max_bitlen = 13; // 8K + 302 hash_bitlen = min_bitlen; + 303 while (max_visited_nodes > hashmap::get_size(hash_bitlen) * max_fill_rate) { + 304 hash_bitlen += 1; + 305 } +... + 324 small_hash_reset_interval = 1; + 325 while (1) { + 326 const auto max_visited_nodes = + 327 itopk_size + (search_width * graph_degree * (small_hash_reset_interval + 1)); + 328 if (max_visited_nodes > hashmap::get_size(hash_bitlen) * max_fill_rate) { break; } + 329 small_hash_reset_interval += 1; + 330 } +``` + +Run it on the defaults with `graph_degree = 32`: + +``` + inputs: itopk_size 64, search_width 1, graph_degree 32, max_fill_rate 0.5 + + max_visited_nodes (1 iteration) = 64 + 1 x 32 x 1 = 96 + bitlen 8 -> 2^8 x 0.5 = 128 capacity >= 96 -> bitlen stays 8 + table bytes = 2^8 x 4 B (uint32) = 1024 B + + reset interval: + r = 1 -> 64 + 32 x 2 = 128 <= 128 keep going + r = 2 -> 64 + 32 x 3 = 160 > 128 stop + small_hash_reset_interval = 2 +``` + +So the table is wiped every second iteration and re-seeded with only the current +internal top-M (`hashmap_restore`, the jit kernel at `:265-271`). Nodes visited +three iterations ago are forgotten and may be re-scored — the paper's +**forgettable hash table management**: *"Although this process may increase the +number of distance computations, catastrophic recall degradation will not occur… +We set the number of entries of the hash table as 2⁸ ∼ 2¹³ and the reset +interval as typically 1 ∼ 4"* (§IV-B3). Compare topic 14's CPU choice, where a +per-query bitmap over 1 M vertices costs 125 kB and nobody minds. + +When even 2¹³ is not enough, `hash_bitlen` is set to 0 and the table moves to +**global** memory, sized for the whole search and allocated per query in the +batch: + +```cpp +// cpp/src/neighbors/detail/cagra/search_single_cta.cuh:200-204 — the fallback. + 200 hashmap_size = 0; + 201 if (small_hash_bitlen == 0 && !this->persistent) { + 202 hashmap_size = max_queries * hashmap::get_size(hash_bitlen); + 203 hashmap.resize(hashmap_size, raft::resource::get_cuda_stream(res)); + 204 } +``` + +`hash_bitlen ≤ 20` there (`search_plan.cuh:346-348`), i.e. up to 4 MB per query +— which is why that path is for small batches. + +### Step 7 — the shared-memory budget, computed + +> **In:** every structure Steps 4-6 introduced. +> **Out:** one number per CTA, and the loop that turns it into a block size — +> the fight Question 4 asks about, with real coefficients. + +Everything the CTA owns is summed in one expression: + +```cpp +// cpp/src/neighbors/detail/cagra/search_single_cta.cuh:126-131 and 175-178 — +// the budget, and the rule that converts it into a thread count. + 126 const std::uint32_t topk_ws_size = 3; + 127 const std::uint32_t base_smem_size = + 128 dataset_desc.smem_ws_size_in_bytes + + 129 (sizeof(INDEX_T) + sizeof(DISTANCE_T)) * result_buffer_size_32 + + 130 sizeof(INDEX_T) * hashmap::get_size(small_hash_bitlen) + sizeof(INDEX_T) * search_width + + 131 sizeof(std::uint32_t) * topk_ws_size + sizeof(std::uint32_t); +... + 175 constexpr unsigned ulimit_smem_size_cta32 = 4096; + 176 while (smem_size > ulimit_smem_size_cta32 / 32 * block_size) { + 177 block_size *= 2; + 178 } +``` + +Evaluate it for the Step 6 configuration, uncompressed float data of dimension +128, `INDEX_T = uint32`, `DISTANCE_T = float`: + +``` + result_buffer_size = 64 + 1 x 32 = 96 + result_buffer_size_32 = roundUp(96, 32) = 96 + buffer bytes = (4 + 4) x 96 = 768 B + hash table = 2^8 x 4 = 1024 B + parent list = 4 x search_width(1) = 4 B + topk workspace = 4 x 3 = 12 B + terminate flag = 4 = 4 B + dataset workspace = sizeof(desc) + 128 x 4 ~ 512 B + desc + ------ + base_smem_size ~ 2324 B (+ desc) + + block-size rule: smem must fit 4096/32 = 128 B per thread + 64 threads -> 8192 B budget >= 2324 B -> the 64-thread floor binds +``` + +Comfortably under the paper's *"typically ≤ 4 kB"* per query (§IV-B3), with +~5.8 kB of headroom before the block size is forced to double. Now spend that +headroom on Question 4's collision. Under PQ compression the codebook is +*also* in the workspace: + +```cpp +// cpp/src/neighbors/detail/cagra/compute_distance_vpq-impl.cuh:112-114 and +// 133-143 — the PQ codebook and query buffer live in the same shared-memory +// workspace the hash table is competing for. + 112 static constexpr std::uint32_t kSMemCodeBookSizeInBytes = + 113 (1 << PQ_BITS) * PQ_LEN * utils::size_of() / + 114 smem_val_config::num_packed_elements; +... + 135 /* SMEM workspace layout: + 136 1. The descriptor itself + 137 2. Codebook (kSMemCodeBookSizeInBytes bytes) + 138 3. Queries (smem_query_buffer_length elems) + 139 */ + 140 return sizeof(cagra_q_dataset_descriptor_t) + kSMemCodeBookSizeInBytes + +``` + +With the default F16 packing (`uint32_t` holding 2 elements, +`compute_distance_vpq-impl.cuh:26-29`) and `PQ_BITS = 8`: + +``` + codebook bytes = 2^8 x PQ_LEN x 4 / 2 = 512 x PQ_LEN + PQ_LEN 2 -> 1024 B PQ_LEN 4 -> 2048 B PQ_LEN 8 -> 4096 B + + budget at 64 threads = 8192 B + spent by Step 6's config (excl. dataset ws) = 1812 B + PQ_LEN 8 codebook = 4096 B + remaining for hash table + buffer + query staging = 2284 B +``` + +That is the fight: the codebook and the visited table draw on one 8192-byte +budget, and losing it does not fail — it doubles `block_size` (`:176-178`), which +halves how many queries an SM can hold, which shows up as throughput. Whoever +wins, you pay in occupancy. ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| cpp/src/neighbors/detail/cagra/cagra_build.cuh | build: NN-descent → rank-based pruning | 3 | -| detail/cagra/graph_core.cuh | graph optimization (detour counting, reverse edges) | 2–3 | -| detail/cagra/search_single_cta_kernel.cuh:30-34 | the search kernel params: itopk, hashmap ptr | 4 | -| detail/cagra/search_single_cta.cuh:127-143 | shared-memory budget assembly (dataset ws + topk scratch) | 4–5 | -| detail/cagra/hashmap.hpp | the visited set: open-addressing table IN SHARED MEMORY | 5 | -| detail/cagra/topk_by_radix.cuh + bitonic.hpp | k-select without sorting everything | 4 | -| detail/cagra/search_multi_cta.cuh | many CTAs per query for large k / low QPS | 4, Q3 | -| detail/cagra/compute_distance_vpq-impl.cuh | PQ-compressed distance (topic 14's ADC on device) | 6, Q4 | - -Start from `search_single_cta_kernel.cuh` (Step 4's loop), with -`search_single_cta.cuh:127-143` open beside it to see the -shared-memory budget being assembled; then `hashmap.hpp`, then the -build side. In the paper: §III is build (Steps 2–3), §IV is the -single-CTA search (Steps 4–5). +| `cpp/include/cuvs/neighbors/cagra.hpp:149-153` | build defaults: `graph_degree = 64`, `intermediate_graph_degree = 128` | 2 | +| `cpp/include/cuvs/neighbors/cagra.hpp:286-318` | search params: `itopk_size = 64`, `search_width = 1`, `team_size`, hash knobs | 2, 4-6 | +| `cpp/src/neighbors/detail/cagra/graph_core.cuh:206-330` | `kern_fused_prune`: detour counting over ranks, then keep-fewest-detours | 3 | +| `cpp/src/neighbors/detail/cagra/graph_core.cuh:178-196, 375` | reverse graph, then the d/2-each interleaved merge | 3 | +| `cpp/src/neighbors/detail/cagra/search_plan.cuh:122-130` | AUTO: single-CTA iff `itopk_size ≤ 512 && max_queries ≥ 2 × #SMs` | 5 | +| `cpp/src/neighbors/detail/cagra/search_plan.cuh:289-330` | small-hash bitlen and reset interval | 6 | +| `cpp/src/neighbors/detail/cagra/search_plan.cuh:333-349` | the device-memory fallback table, `hash_bitlen ≤ 20` | 6 | +| `cpp/src/neighbors/detail/cagra/search_single_cta.cuh:106-131` | buffer sizing, `max_itopk = 512`, the shared-memory sum | 1, 5, 7 | +| `cpp/src/neighbors/detail/cagra/search_single_cta.cuh:157-197` | block size: radix minimum, the 128 B/thread rule, occupancy bump | 5, 7 | +| `cpp/src/neighbors/detail/cagra/hashmap.hpp:15-73` | linear-probing open addressing, `atomicCAS` insert, full-table behaviour | 6 | +| `cpp/src/neighbors/detail/cagra/jit_lto_kernels/search_single_cta_jit.cuh:176-300` | the search loop: parents, hash reset, child distances | 4 | +| `cpp/src/neighbors/detail/cagra/search_multi_cta.cuh:117-138` | small-batch mode: `itopk_size` forced to 32 per CTA, `num_cta_per_query` derived from the requested itopk | 5 | +| `cpp/src/neighbors/detail/cagra/search_multi_cta.cuh:246-265` | the partial lists merged by a *separate* kernel over device memory — the no-device-barrier tax | 5 | +| `cpp/src/neighbors/detail/cagra/compute_distance_vpq-impl.cuh:112-143` | PQ codebook in the shared-memory workspace | 7 | + +Reading order: `cagra.hpp` for the vocabulary, then `search_single_cta.cuh`'s +`set_params` — it is 100 lines and it contains the whole resource argument — +then `hashmap.hpp` and `search_plan.cuh:289-349` as a pair, then the jit kernel +loop. `graph_core.cuh` last; it is 1811 lines and only Steps 3's two passes +matter. In the paper: §III is build, §IV-A the algorithm, §IV-B the four +elemental techniques, §IV-C the two implementations and Table II. ## Questions for notes.md -1. Fixed degree 32 vs HNSW's M=16-64 with levels: derive expected - hops for 1M vectors (paper reports ~same recall at similar - memory — where did the levels' log-factor go?). -2. itopk lives in shared memory and is maintained by - bitonic/radix-topk — why is a HEAP (topic 14's CPU choice) - wrong on a warp? -3. search_multi_cta splits one query across CTAs — when (large k, - small batch)? What synchronizes the partial itopks (global - memory + separate merge kernel — the no-device-barrier tax - again)? -4. compute_distance_vpq: PQ codes unpacked per lane — topic 14's - ADC table lives where (shared memory — budget collision with - the hashmap: find who wins)? -5. For M14+M18: our rescore pipeline is exact-f32 over PQ - candidates. Which half goes to GPU first, and what's the batch - size per the crossover table you'll measure with l2_batch? +1. HNSW's levels exist to reach the right neighbourhood in O(log N) hops. CAGRA + deletes them and picks random entry points (§III). At fixed degree, where did + the log factor go — and what in Step 3's build is paying for it? (Fig. 3's + 2-hop counts and strong-CC numbers are the evidence; say which pass buys + which.) +2. Why is a heap the wrong top-M structure on a warp, and what does bitonic sort + buy instead? Then find the threshold discrepancy: §IV-B2 says 512, and + `search_single_cta.cuh:134` says 256 — of *what*, in each case? +3. `multi_cta` splits one query across CTAs (`search_plan.cuh:124-129` says + when). There is no device-wide barrier, so how do the partial internal-top-M + lists merge, and where does that table have to live (Table II)? Check your + answer against `search_multi_cta.cuh:246-265` — count the kernel launches. +4. Do the Step 7 arithmetic yourself for `graph_degree = 64`, `itopk_size = 128`, + PQ_LEN = 4: does the 64-thread block still fit in 128 B/thread? If not, what + does the doubling cost you in resident queries per SM? +5. For M14+M18: our rescore pipeline is exact-f32 over PQ candidates. Which half + goes to the GPU first, and at what batch size — given that this topic measured + no crossover at all up to 2²⁴ elements on the local lane + (`FINDINGS.md:36`) and CAGRA's own rule needs ≥ 2 × #SMs queries before it + will even use its throughput mode? ## Done when -- [ ] You can say what SIMT hates about a greedy proximity-graph walk, and why fixed degree 32 is the answer rather than levels. -- [ ] You can explain why the build is NN-descent rather than insert-one-at-a-time. -- [ ] You can describe the one-CTA-per-query shape and what is parallel inside a single step. -- [ ] You can explain why the visited set becomes a shared-memory hashmap, and what that costs. -- [ ] You wrote answers to all five questions in notes.md, including where `search_multi_cta` becomes the right choice. +Answer each before unfolding it. + +- [ ] You can name the three HNSW features CAGRA deletes, and say which one deletes another topic's entire problem. + +
Answer + + The hierarchy (random entry points instead, §III), variable degree (fixed + out-degree, `graph_degree = 64` by default, `cagra.hpp:153`), and the + heap-allocated visited set (shared-memory hash, Step 6). + + Fixed degree deletes Gunrock's: no ragged frontier, so no load-balancing + strategy to choose and no prefix scan to size the output — the candidate count + is `search_width × graph_degree`, known before the iteration starts + (`search_single_cta.cuh:108`). + +
+ +- [ ] You can state the build speedup with its caveats, and say which paper version you are quoting. + +
Answer + + 2.2–27× faster graph construction than HNSW (abstract and §V-A of + [arXiv:2308.15136**v2**](https://arxiv.org/abs/2308.15136), 9 Jul 2024), on a + DGX A100 — AMD EPYC 7742, 64 cores, against an A100 80 GB — with *"both the + dataset and graph on the device memory of the GPU"* (§V-A). Large-batch search + is 33–77× at 90-95 % recall, and 3.8–8.8× against other GPU implementations. + + Not "~10×", which this guide previously claimed and which appears nowhere in + the paper. + +
+ +- [ ] You can explain warp splitting with the arithmetic that motivates it. + +
Answer + + 32 lanes × 128-bit loads = 4096 bits per warp instruction, but a 96-dim float + vector is 3072 bits, so one-warp-per-distance idles a quarter of the lanes. A + team of 8 lanes loads 1024 bits, covers the vector in exactly 3 loads, and lets + 4 teams per warp work on 4 different candidates (§IV-B1). No divergence: all + teams execute the same instructions. + + `team_size` defaults to 0 = auto, legal values 4/8/16/32 (`cagra.hpp:302-303`). + +
+ +- [ ] You can say whether the visited table is exact or lossy, and what actually degrades recall. + +
Answer + + Exact: open addressing with linear probing stores full keys and compares them + (`hashmap.hpp:15,44-47`), and `atomicCAS` returns "newly inserted" exactly once + (`:56-60`). A collision costs a probe, not an error. + + Two things degrade recall. A *full* table — the probe loop gives up after + `size` attempts and returns 0, i.e. "already visited" (`:55,72`). And the + deliberate one: the small table is wiped every `small_hash_reset_interval` + iterations and re-seeded from the internal top-M only + (`search_single_cta_jit.cuh:265-271`), so older nodes are re-scored. The paper + calls it forgettable hash table management and reports no catastrophic recall + loss (§IV-B3). + +
+ +- [ ] You can compute the reset interval from the search parameters. + +
Answer + + `max_visited_nodes = itopk_size + search_width × graph_degree × (r + 1)`, and + `r` grows while that stays under `2^bitlen × max_fill_rate` + (`search_plan.cuh:324-330`). With itopk 64, width 1, degree 32, fill rate 0.5, + bitlen 8: capacity 128; r=1 → 128 (fits), r=2 → 160 (does not) → interval 2. + + The table itself is 2⁸ × 4 B = 1 kB, within the paper's stated 2⁸–2¹³ range + (§IV-B3). + +
+ +- [ ] You can say what happens when the shared-memory budget is exceeded, and why it is a throughput cost rather than an error. + +
Answer + + `while (smem_size > 4096 / 32 * block_size) block_size *= 2` + (`search_single_cta.cuh:175-178`): the CTA gets more threads so that the + per-thread shared-memory quota is met. More threads per query means fewer + queries resident per SM, so batch throughput falls. Nothing fails. + + The competing tenants are the top-M buffer, the hash table, the parent list, + and — under PQ — a codebook of `2^PQ_BITS × PQ_LEN × 2` bytes in the same + workspace (`compute_distance_vpq-impl.cuh:112-114,140`). + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the Step 7 arithmetic redone for question 4's parameters. + +
Answer + + The slots are `notes.md:87-93`. Question 4 wants the sum evaluated, not + described: buffer + hash + parents + workspace against 128 B × block_size. + +
## References **Papers** -- Ootomo, Naruse, Nolet, Wang, Feher, Wang — "CAGRA: Highly Parallel - Graph Construction and Approximate Nearest Neighbor Search for - GPUs" (ICDE 2024, - [arXiv:2308.15136](https://arxiv.org/abs/2308.15136)) — §III for - build (NN-descent + pruning), §IV for the single-CTA search + +- Hiroyuki Ootomo, Akira Naruse, Corey Nolet, Ray Wang, Tamas Feher, Yong Wang — + *"CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor + Search for GPUs"*, ICDE 2024; + [arXiv:2308.15136](https://arxiv.org/abs/2308.15136). Cite **v2** (9 Jul 2024) + for every number above — §III build, §IV-A the algorithm, §IV-B1 warp + splitting, §IV-B2 bitonic/radix, §IV-B3 the forgettable hash, §IV-B4 the MSB + parent flag (and its 2³¹−1 dataset limit for `uint32`), §IV-C and Table II the + two implementations, §V the evaluation on DGX A100 / EPYC 7742. **Code** -- [cuvs](https://github.com/rapidsai/cuvs) — - `cpp/src/neighbors/detail/cagra/` — the anchor map above is the - reading order; start from `search_single_cta_kernel.cuh` + +- [cuvs](https://github.com/rapidsai/cuvs) @ `8b97b61` — `cpp/src/neighbors/ + detail/cagra/`. CUDA only: it does not build on this machine, and the guide + quotes it rather than running it. + +**Measurements in this repo** + +- `topics/18-gpu/notes.md:9-16` and `FINDINGS.md:36` — the local wgpu/Metal + transfer tax that Step 4's kernel-launch argument and question 5 lean on. They + are not ANN measurements and must not be quoted as any. + +**Related guides** + +- `reading-crystal-sigmod20.md` — why "GPU is N× faster" is meaningless without + the residency clause the CAGRA paper states in §V-A. +- `reading-gunrock.md` — the ragged-frontier problem CAGRA's fixed degree + deletes. +- `reading-faiss-gpu.md` — the other GPU k-select design, in registers rather + than shared memory. diff --git a/topics/18-gpu/reading-crystal-sigmod20.md b/topics/18-gpu/reading-crystal-sigmod20.md index 6006458..65fa452 100644 --- a/topics/18-gpu/reading-crystal-sigmod20.md +++ b/topics/18-gpu/reading-crystal-sigmod20.md @@ -1,243 +1,506 @@ # GPU vs CPU for analytics: two regimes, two verdicts -Shanbhag, Madden & Yu's Crystal paper settled a decade of "GPU -databases: hype?" papers by building the fairest possible comparison: -a tile-based GPU query library vs a state-of-the-art CPU baseline, on -Star Schema Benchmark, with the transfer question made explicit. Its -two-regime framing is the go/no-go lens for every operator M18 -considers offloading. Before you open the paper, this chapter builds -the seven concepts it assumes — how a GPU executes, what its memory -rules are, why the bus dominates, and how to predict a winner with -one max() formula — then hands you a section-by-section route. +Shanbhag, Madden and Yu's Crystal paper ended a decade of "GPU databases: hype?" +papers by building the fairest comparison anyone had: a tile-based GPU query +library against a CPU baseline good enough to beat Hyper, on the Star Schema +Benchmark, with the transfer question made explicit instead of buried in a +footnote. Its two-regime framing is the go/no-go lens for every operator M18 +might offload. + +It is also the paper this topic's own measurement most directly contradicts — +or rather, agrees with, once you read the caveat. Every headline number in +sections 4 and 5 is measured with the data **already resident in the device's +memory**. This topic measures what happens when it is not: 7197 µs of upload +against a 2723 µs CPU total (`FINDINGS.md:36`). Both are true. Keeping them +straight is the entire skill this guide is teaching. + +Citations below are to +[arXiv:2003.01178](https://arxiv.org/abs/2003.01178), the extended version of +the SIGMOD 2020 paper; section, figure and table numbers are the paper's own. +None of the CUDA in this chapter can be run here — this machine has no NVIDIA +device — so every number is read, not reproduced, and is labelled with where it +was read from. ## The problem in one sentence -A discrete GPU has ~9× the memory bandwidth of a CPU (~880 vs -~100 GB/s) but sits behind a PCIe bus that moves only ~16 GB/s — so -for a scan-shaped query the GPU is either **~16× faster** or -**~6× slower** depending on one question: which side of the bus does -the data live on? +On the paper's own hardware a V100 reads its own memory at **880 GBps** while +the CPU reads at **53 GBps** (Table 2) — a ratio of **16.2** (§4) — but the +measured PCIe link between them moved only **12.8 GBps** (§5), so the same query +is either a 25× win or a 1.4× loss against a good CPU engine depending on one +question: which side of the bus does the data already live on? ## The concepts, step by step ### Step 1 — SIMT: the GPU hides latency with thread count, not caches -A GPU is a processor that runs *tens of thousands* of threads at -once, grouped into **warps** (bundles of 32 threads that execute the -same instruction in lockstep — SIMT, "single instruction, multiple -threads"). Warps live on **SMs** (streaming multiprocessors — the -GPU's cores, a few dozen to ~100+ per chip), and each SM keeps many -warps *resident* simultaneously. - -The point of all that parallelism is latency hiding. A CPU core -hides memory latency *per thread* — out-of-order execution plus big -caches keep one instruction stream busy. A GPU does the opposite: -when a warp stalls on a memory load (~hundreds of cycles), the SM -just issues instructions from a *different* resident warp. Nothing -waits as long as enough warps are resident: +> **In:** a memory access that misses cache. +> **Out:** on a CPU, a stall; on a GPU, a warp swap — provided enough warps are +> resident. Everything else in the paper follows from that difference. + +A GPU runs tens of thousands of threads at once, grouped into **warps** (32 +threads that issue the same instruction in the same cycle — SIMT, "single +instruction, multiple threads"). Warps live on **SMs** (streaming +multiprocessors); a **thread block**, or CTA, is a group of threads scheduled +onto one SM together, able to share scratch memory and to synchronise with a +barrier. + +The paper's own description of what that buys, from §5.3 (its explanation of a +model that under-predicted CPU time by 2.7×): + +> "On the GPU, a single streaming multiprocessor (SM) usually has 64 cores that +> can execute 2 warps (64 threads) at any point. However, the SM can keep > 2 +> warps active at a time. On Nvidia V100, each SM can hold 64 warps in total +> with 2 executing at any point in time. Any time a warp makes a memory request, +> the warp is swapped out from execution into the active pool and another warp +> that is ready to execute ends up executing." + +The measurement that forced that paragraph is worth carrying around. For SSB +q2.1 the authors' bandwidth model predicted **47 ms on the CPU and 3.7 ms on the +GPU**; the actual runtimes were **125 ms and 3.86 ms** (§5.3). The GPU landed +within 4 % of a model that assumes no stalls at all. The CPU missed by 166 %, +because a join probe is an irregular access and prefetchers do not help with +those. Latency hiding by oversubscription is not a small effect; it is the +difference between a model that works and one that does not. + +**Occupancy** is the fraction of the maximum resident warps you actually +achieve, and it is bounded by how many registers and how much shared memory each +thread uses — which is why every later step counts bytes per thread. Low +occupancy means too few warps to swap to, and the GPU stalls exactly like a CPU. +The other SIMT rule is **branch divergence**: when lanes of a warp disagree on an +`if`, the warp executes both sides with lanes masked. That is topic 17's +predication, done by hardware whether you asked for it or not — and §4.2's +measurement of it is in Step 4, where it is not what you would guess. + +### Step 2 — coalescing and shared memory: the two rules that decide layout + +> **In:** 32 lanes issuing 32 loads in one instruction. +> **Out:** one memory transaction if the addresses are adjacent, up to 32 if +> they are not — before any code of yours has run. + +**Coalescing**: adjacent lanes touching adjacent addresses collapse into one +transaction. Lane *i* reading `col[i]` (a dense column) fetches one contiguous +block; lane *i* reading `rows[i].field` issues up to 32 transactions and discards +most of each. This is topic 12's columnar argument with a 32× multiplier +attached, and it is why Crystal's write path (Step 6) goes to such trouble to +make the *output* contiguous too. + +**Shared memory**: a software-managed scratchpad per SM, shared by a thread +block. The paper gives its size the way that matters for algorithm design rather +than in bytes — per thread, at full occupancy: + +> "On the Nvidia V100, each GPU thread can only store roughly 24 4-byte entries +> in shared memory at full occupancy, with 5000 threads running in parallel." +> (§3.2) + +Table 2 gives the other side of the same coin: 16 KB of L1 per SM, 6 MB of L2 in +total, and no L3 at all, against the CPU's 32 KB L1 and 256 KB L2 per core plus +a 20 MB shared L3. A GPU has less cache per thread by orders of magnitude; what +it has instead is Step 1. The consequence the paper draws — that a *thread* is +too small a unit to plan around but a *thread block* is not — is Step 5. + +### Step 3 — the bus decides: why the coprocessor model fails, in arithmetic + +> **In:** SSB Q1.1 over a lineorder table of L rows, four 4-byte columns. +> **Out:** two bounds, one on each processor, whose order does not depend on how +> good your kernel is. + +§3.1 states the model in two lines. Let *B_c* be CPU memory bandwidth and *B_p* +PCIe bandwidth. A CPU can answer Q1.1 in one pass over four 4-byte columns, so +its optimal runtime *R_C* is **upper** bounded by `16L / B_c`. In the coprocessor +model those same four columns must cross the bus, so the GPU runtime *R_G* is +**lower** bounded by `16L / B_p`, a bound reached only with perfect +transfer/compute overlap. Since `B_c > B_p` on every real machine, `R_C < R_G` +— the direction of the comparison is fixed before either implementation exists. + +Put SSB scale factor 20 through it, which is what §5.1 says the paper ran: +lineorder has **120 million tuples**. ``` - CPU core: wide OoO, ~5 GHz, caches hide latency PER THREAD - GPU SM: 32-lane warps (SIMT), latency hidden by OVERSUBSCRIPTION - — thousands of resident threads; when a warp stalls on - memory, another issues. + L = 120e6 rows + bytes per row = 4 columns x 4 B = 16 B + 16L = 1.92e9 B (1.92 GB) + + CPU upper bound = 1.92e9 / 53e9 = 36.2 ms (B_c, Table 2 read BW) + GPU lower bound = 1.92e9 / 12.8e9 = 150.0 ms (B_p, measured, §5) + ------- + the best case for the GPU is 4.1x the worst case for the CPU ``` -**Occupancy** is the fraction of the maximum resident warps you -actually achieve (limited by how many registers and how much -scratch memory each thread uses). Low occupancy = not enough warps -to hide latency = the GPU sits idle exactly like a CPU with a cache -miss. One more SIMT rule: **branch divergence** — when threads of a -warp disagree on an `if`, the warp executes *both* sides with lanes -masked off. That is topic 17's predication done by hardware, and it -is why branchy code is a GPU anti-pattern. - -### Step 2 — coalescing and shared memory: the two memory rules - -A warp's 32 simultaneous loads become **one** memory transaction if -and only if adjacent lanes touch adjacent addresses — this is -**memory coalescing**. If lane i reads `col[i]` (a dense column), -the hardware fetches one contiguous block; if lane i reads -`rows[i].field` (a strided row layout), it issues up to 32 separate -transactions and throws away most of every one. Coalescing is the -GPU word for topic 12's columnar-layout argument, with a 32× -multiplier attached. - -The second rule: each SM has ~100 KB of **shared memory** — a -software-managed scratchpad, as fast as L1 cache but under *your* -control, visible to all threads of one **block** (a group of a few -hundred threads scheduled onto one SM; CUDA also calls it a CTA). -Shared memory is the GPU word for topic 13's cache blocking: stage -a chunk there, work on it repeatedly, write results back once. - -Why it matters: every GPU-DB trick in this paper is one of — -coalesce (layout), stay resident (occupancy), amortize atomics -(reduce first), or avoid the bus (Step 3). - -### Step 3 — the bus decides everything: regime A vs regime B - -A discrete GPU's fast memory (**HBM** — high-bandwidth memory -soldered next to the GPU die, 400–3000 GB/s) is reachable from the -CPU only over PCIe at 16–64 GB/s. That one number splits all -GPU-database papers into two regimes: - -``` - regime A: data ships over PCIe per query (coprocessor model) - GPU time ≈ transfer time; PCIe ~16 GB/s vs CPU membw ~100 GB/s - → CPU WINS almost always. Full stop. - regime B: working set resident in GPU HBM (primary-store model) - HBM ~880 GB/s vs CPU ~100 GB/s - → GPU wins by ~ the bandwidth ratio (they measure ~16× on SSB) -``` +And that is what the measurement showed (§3.1, Figure 3): the GPU coprocessor +was **1.5× faster than MonetDB but 1.4× slower than Hyper**, and *"for all +queries, the query runtime in GPU coprocessor is bound by the PCIe transfer +time."* The paper's diagnosis of the prior literature — reported coprocessor +speedups from 2× to 100× — is that those papers were beating MonetDB, not +beating a good CPU engine. -Everything else in the literature is confusion between A and B. The -architectural corollary Crystal works out: GPU as -accelerator-of-operators (ship data per query) fails; GPU as -primary-store-with-CPU-fallback works. +Two numbers people quote loosely here, with their actual sources: §2.2 says +*"the PCIe bandwidth of a modern machine is up to 16 GBps"*, an upper figure for +the era; the machine the paper actually measured on delivered **12.8 GBps** +bidirectional (§5). Use 12.8 when you are reproducing their arithmetic, 16 only +when you are quoting their prose. -Our gpu_bench's no-crossover table is regime A in miniature — except -on unified memory (Apple Silicon: CPU and GPU share one LPDDR pool, -~150–400 GB/s) the "transfer" is a staging copy + ~1.5 ms dispatch -overhead, and the bandwidth RATIO is ~1, so even regime B wouldn't -save a memory-bound scan on this Mac. Question: what DOES unified -memory save, and which operator class exploits it (arithmetic -intensity — the l2_batch stub)? +Now the local translation. This Mac has no PCIe hop at all — CPU and GPU share +one LPDDR pool — and yet: -### Step 4 — tiles: vectorized execution rebuilt for blocks +``` + upload, 2^24 f32 = 67,108,864 B in 7384.7 us = 9.1 GB/s (notes.md:16) + CPU sum, same bytes = 67,108,864 B in 2257.7 us = 29.7 GB/s (notes.md:16) + -------- + effective transfer path is 3.3x SLOWER than the CPU's own read +``` -Crystal's core idea is to process a query as a sequence of -BLOCK-WIDE functions over **tiles** — a tile is `items per thread × -threads per block` elements (e.g. 4 × 256 = 1024), loaded from HBM -with coalesced accesses, staged through shared memory, and handed -from one block-wide primitive to the next: +Structurally identical to `B_c / B_p` = 53 / 12.8 = 4.1, for a completely +different reason: wgpu stages the upload through a private buffer, so "unified" +memory still costs a copy. Crystal's regime A is not a PCIe fact. It is a +boundary-crossing fact, and the boundary is still there. + +### Step 4 — the caveat that governs every number in §4 and §5 + +> **In:** any speedup you are about to quote from this paper. +> **Out:** the sentence you must quote next to it, or the number is misleading. + +§4 opens its operator comparisons with the setup line, and then this: + +> "For the micro-benchmarks, we use a setup where GPU memory bandwidth is +> 880GBps and CPU memory bandwidth is 54GBps, resulting in a bandwidth ratio of +> 16.2 (see Section 5 for system details). **In all cases, we assume that the +> data is already in the respective device's memory.**" + +§5 repeats it for the full-workload numbers: *"In our evaluation, we ensure that +data is already loaded into the respective device's memory before experiments +start."* So every ratio below is a regime-B number. None of them survives +contact with Step 3's bus. + +With that said out loud, here is what they measured, operator by operator: + +| operator | ratio CPU:GPU | where | note | +|---|---|---|---| +| projection, linear combination (Q1) | **16.56** | §4.1 | ≈ the 16.2 bandwidth ratio | +| projection with UDF (Q2) | **17.95** | §4.1 | | +| selection, averaged over selectivity 0→1 | **15.8** | §4.2 | input 2²⁹ entries | +| hash join probe, HT 32–128 KB | **≈5.5** | §4.3, Fig 13 | HT in L2 on both; CPU is DRAM-bound, GPU L2-bound | +| hash join probe, HT 1–4 MB | **14.5** | §4.3 | GPU L2 vs CPU L3 bandwidth ratio | +| hash join probe, HT > 128 MB | **10.5** | §4.3 | model says 8.1 — see below | +| radix sort, 2²⁸ entries | **17.13** | §4.4 | 464 ms CPU vs 27.08 ms GPU | +| full SSB, SF20, 13 queries | **25** | §5.2 | *exceeds* the bandwidth ratio — Step 7 | + +The join row is the one that is usually flattened into a single number and +should not be. It is regime-dependent by an order of magnitude across the table, +and the largest-table case has a hardware reason worth memorising: *"The +granularity of reads from global memory is 128B on GPU while on CPU it is 64B. +Hence, random accesses into the hash table read twice the data on GPU compared +to CPU"* — 16.2 / 2 = 8.1 expected, 10.5 observed, the surplus being Step 1's +latency hiding again. + +Two more things §4.2 measured that contradict the folklore. First, on the GPU +there is **no difference between the branching and the predicated selection** +(`GPU If` vs `GPU Pred`) — a single mispredicted branch does not cost a GPU +anything measurable; the branch-versus-predication curve everyone half-remembers +is the *CPU* result in the same figure (`CPU If` vs `CPU Pred` vs +`CPU SIMDPred`). Second, both `CPU SIMDPred` and the GPU variants track their +bandwidth models closely, which is the paper's real claim: a *competently +written* CPU selection also saturates memory bandwidth, so the gap between them +is the hardware ratio and nothing else. + +### Step 5 — tiles: the thread block is the unit, not the thread + +> **In:** `SELECT y FROM R WHERE y > v`, run on 5000 threads that share no +> cursor. +> **Out:** one kernel instead of three, one pass over the input instead of two, +> and coalesced writes instead of random ones. + +Step 2 said a GPU thread can hold ~24 4-byte entries in shared memory at full +occupancy. That is too small to be an execution unit. A thread *block* holding +those 24 entries per thread collectively is not — the paper calls that unit a +**tile**, of size `items per thread × threads per block`, and it is the +GPU-shaped answer to topic 11's ~1000-element vector (Figure 5 draws exactly +that correspondence). §5.2 says their SSB runs used a block size of 256 and a +tile of 2048 (8 items per thread). + +What the tile replaces is worth spelling out, because it is what GPU databases +did before (§3.2, Figure 4a): ``` - load tile → coalesced, all threads - BlockPred: each thread evaluates predicate on its items → flags - BlockScan: prefix-sum flags → output offsets (compaction!) - BlockShuffle / BlockAggregate / BlockProbe ... - write tile → coalesced + (a) what existing GPU databases did — three kernels + K1 each thread reads a strided slice, evaluates the predicate, + writes count[t] ← pass 1 over the input + K2 prefix sum over count -> pf ← e.g. a Thrust call + K3 each thread reads its slice AGAIN and writes matches at + pf[t] + local_counter ← pass 2, random writes + + costs: input read twice, count and pf materialised in global + memory, every thread writing to a different place + + (b) tile-based, one kernel (Figure 4b, Figure 6) + load tile into shared memory ← coalesced + evaluate predicate -> bitmap + per-thread histogram of matches + block-wide prefix sum over the histogram ← offsets within the tile + ONE atomic add on the global counter, by the block, of the tile total + block-wide shuffle -> contiguous run in shared memory + write the run to global memory ← coalesced ``` -This is topic 11's vectorized execution with tiles for batches and -shared memory for the L1-resident chunk. The batch size is dictated -by the hardware (threads per block × registers per thread), not -chosen by a tuning knob — same reason topic 11 picked ~1024-row -vectors to fit L1. +Everything after the load reads shared memory, so the input is touched once. +Crystal is a library of these block-wide steps (`BlockLoad`, `BlockPred`, +`BlockScan`, `BlockShuffle`, `BlockStore`…) that compose into one fused kernel +per query. This is topic 11's operator fusion — optional on a CPU, structural +here, because every kernel boundary is a round trip through global memory and +the whole point of being on the GPU was the bandwidth. + +### Step 6 — compaction: a prefix scan is the substitute for a cursor -### Step 5 — compaction: filters need a prefix scan, not a cursor +> **In:** a bitmap of survivors, spread across 5000 threads with no ordering +> between them. +> **Out:** a distinct output slot per survivor, computed rather than claimed. -A filter's output is smaller than its input, and on a CPU you'd -append survivors with a cursor (`out[k] = x; k += mask`). With -100,000 concurrent threads there is no shared cursor — no total -order exists among the threads, so nobody knows *where* to write. -The fix: an **exclusive prefix scan** (each element gets the sum of -all flags *before* it — which is exactly its output offset) -computed block-wide in shared memory, plus **one** atomic add per -block to claim a range of the global output: +On a CPU, filter output is a cursor: `out[k] = x; k += mask`. §3.2 explains why +that works there and not here — a CPU thread updates the shared counter *once +per ~1000-entry vector* and there are only ~32 threads, so the counter is not the +bottleneck. With 5000 threads each wanting a slot, it is. + +The replacement is an **exclusive prefix scan**: each element's output offset is +the count of survivors before it, which the block computes cooperatively in +shared memory. The block then does exactly one atomic add on the global counter +to claim a contiguous range. The paper's own summary: *"By treating the thread +block as an execution unit, we reduce the number of atomic updates of the global +counter by a factor of size of tile T."* ```rust -// tile-based filter: 100K threads share no cursor — the SCAN makes the order +// ILLUSTRATION — not Crystal source (Crystal is CUDA C++ and is not pinned in +// this repo). This is Figure 6's kernel written as Rust-ish pseudocode; the +// code you are meant to write from it is the filter_count stub at +// experiments/src/gpu.rs:154, whose doc comment asks for exactly this shape. par_for tile in input.tiles(ITEMS_PER_THREAD * THREADS_PER_BLOCK) { - let items = block_load(tile); // coalesced - let flags = items.map(|x| pred(x) as u32); // BlockPred - let (offsets, total) = block_exclusive_scan(flags); // BlockScan - let base = atomic_add(&global_cursor, total); // once per BLOCK - for i in 0..ITEMS_PER_THREAD { - if flags[i] == 1 { out[base + offsets[i]] = items[i]; } - } + let items = block_load(tile); // coalesced, into shared + let flags = items.map(|x| pred(x) as u32); // bitmap + let hist = per_thread_count(flags); // matches per thread + let (off, total) = block_exclusive_scan(hist); // offsets within the tile + let base = atomic_add(&global_cursor, total); // ONCE per block + let packed = block_shuffle(items, flags, off); // contiguous in shared + block_store(&mut out[base..], packed); // coalesced write } ``` -Question: why does GPU filter output need a prefix-scan where the -CPU used a cursor `k += mask`? (No total order across 100K threads -— the scan MAKES one.) The compaction step is topic 17's compress, -built from scan instead of vpcompress — and one atomic per block -instead of per element is the "amortize atomics" rule from Step 2. -Crystal also measures that selection via scan+compact beats -branch-per-thread at mid selectivities — the topic 17 selectivity -curve, GPU edition (Step 1's divergence rule, quantified). - -### Step 6 — fusion: one kernel per query, or the bandwidth win evaporates - -Each kernel (a GPU function launched over many blocks) reads its -input from HBM and writes its output to HBM. Run a query as five -separate operator kernels and every intermediate result makes a -round trip through HBM — at 880 GB/s that traffic eats the exact -bandwidth advantage you came for. Crystal therefore **fuses** the -whole SSB query into one kernel: tiles flow from primitive to -primitive through shared memory and registers, touching HBM only at -scan and final output. This is topic 11's operator fusion — optional -on a CPU, mandatory here. The cost: fused kernels are monolithic -(one giant kernel per query shape) and kill operator-at-a-time -profiling — see question 4. - -### Step 7 — the roofline: one formula predicts the winner - -For a scan-shaped operator, execution time is bounded by whichever -resource saturates first: +The same rule, three ways, in three files you can actually read: one `atomicAdd` +per workgroup in the WGSL exercise (`experiments/src/gpu.rs:150-153`); one +device-scope `fetch_add` per *warp* in libcudf's conditional join +(`cpp/src/join/conditional_join_kernels.cuh:74-77`); and Crystal's one per +*block*. Nobody who has measured it does one atomic per element. + +### Step 7 — the roofline, and why the full query beat the ratio + +> **In:** an operator's bytes moved and FLOPs performed. +> **Out:** a predicted winner — plus the one effect that makes the prediction +> too conservative. + +For a scan-shaped operator, time is bounded by whichever resource saturates +first: ``` - time = max( bytes / memory_bandwidth , flops / peak_flops ) + time = max( bytes / memory_bandwidth , flops / peak_flops ) ``` -The ratio `flops/byte` is the operator's **arithmetic intensity**, -and it decides which term wins. GPU wins iff data is resident -(Step 3) AND the op is bandwidth-bound (ratio ~9×) or compute-bound -with high intensity (ratio can be ~50×). Neither holds for -ship-per-query. Question: place these on the roofline: sum -(0.25 FLOP/byte), filter (0.25), hash probe (~1 + random access), -l2 dim=128 (~32), CAGRA search (~high + irregular). Which two belong -on a GPU at all? +`flops / byte` is the operator's **arithmetic intensity**. §4's models are this +formula with the constants filled in: selection, for instance, is modelled as +`4N/B_r + 4σN/B_w` (§4.2), read every entry, write the σ fraction that survives. +When an implementation tracks that model, it is bandwidth-saturated, and the +CPU:GPU ratio can only be the bandwidth ratio — which is exactly what §4.1-4.4 +found, everywhere except the join, where the 128 B vs 64 B access granularity +halves it. + +So an operator-level roofline predicts *at most* 16.2× in regime B and a loss in +regime A. Then §5.2 measured **25×** on the full 13-query benchmark. The paper's +explanation is that the ceiling applies to operators in isolation, not to +chains: on the CPU, vectorising a chain of operators leaves gaps the model does +not capture (the q2.1 miss in Step 1 — 47 ms modelled, 125 ms actual), while the +GPU's latency hiding keeps it near its own model even through irregular join +probes. The lesson for M18's go/no-go is uncomfortable and worth stating +plainly: the roofline is a reliable *lower* bound on the GPU's advantage in +regime B and gives no protection at all in regime A, where Step 3's bus decides +everything before the first FLOP. ## How to read the paper (with the concepts in hand) -- **§2–3 — the tile model.** Steps 4–5 in the authors' words: the - block-wide primitives, the shared-memory staging, and the fused - SSB queries (Step 6). Map each primitive to its topic 11/17 CPU - ancestor as you go. -- **§5–6 — the two-regime measurements.** Step 3 quantified: the - ~16× regime-B win on SSB, and the transfer-inclusive numbers that - kill regime A. This is the go/no-go table for M18. -- **CPU-baseline honesty** — read this discussion even if you never - touch a GPU: their CPU code is AVX-vectorized and multi-threaded; - most prior "100× GPU speedups" compared against scalar - single-thread CPU code (topic 0's fair-benchmarking paper, case - study #1). Any speedup claim you publish for M18 gets held to this - standard. +Read it in this order, not front to back: + +- **§3.1 first** (two pages). The coprocessor bound, the SSB SF20 measurement, + Figure 3. If you read nothing else, read this — it is Step 3, and it is the + section that makes this topic's own no-crossover result unsurprising. +- **§3.2 and Figure 4/5/6.** The three-kernel selection and its tile-based + replacement — Steps 5 and 6 in the authors' words. Map each block-wide + primitive onto its topic 11 / topic 17 CPU ancestor as you go. +- **§4, starting with the last sentence of its opening paragraph.** That is the + residency assumption (Step 4). Then the operator subsections; in §4.3 read the + three hash-table size regimes rather than taking an average. +- **§5.1-5.2** for the platform table and the 25×, then **§5.3** for why the + 25× exceeds 16.2 (Step 1's latency hiding, quantified). +- **The CPU-baseline discussion in §5.2**, even if you never touch a GPU: their + CPU implementation beats Hyper by 1.17× and MonetDB by 2.5×, which is what + earns them the right to publish a 25×. Topic 0's fair-benchmarking rules, + applied by someone with something to lose. Any speedup you publish for M18 + gets held to the same standard. ## Questions for notes.md -1. SSB is denormalized-star scans. Which topic 22 benchmark shape - would flip the verdict back to CPU even in regime B (hint: - point lookups, topic 3)? -2. Crystal predates Apple unified memory. Rewrite their regime - table for M-series: what replaces PCIe, what replaces HBM, and - why does the GPU still lose our sum bench? -3. Their group-by uses atomics into a hash table when groups are - few. At what group cardinality does that collapse, and what's - the fallback (cudf's shared-mem vs global split)? -4. Fusing the whole query into one kernel kills operator-at-a-time - profiling. What replaces topic 0's flamegraph on GPU (NSight / - Metal capture — occupancy + achieved bandwidth per kernel)? -5. For M18: our engine's hot paths are graph expand (random), - filter (streaming), distance scoring (dense). Apply Step 7's - roofline to each and write the one-line go/no-go. +1. SSB is denormalised-star scans. Which topic 22 benchmark shape would flip the + verdict back to the CPU even in regime B (hint: point lookups, topic 3 — what + is the arithmetic intensity of a single B-tree descent, and how many warps + does it keep busy)? +2. Crystal predates Apple unified memory. Rewrite Step 3's regime table for + M-series: what replaces PCIe, what replaces HBM, and why does the GPU still + lose our sum bench? Use the two measured numbers, 9.1 GB/s and 29.7 GB/s. +3. Their group-by uses atomics into a hash table when groups are few. At what + group cardinality does that collapse, and what is the fallback? Compare with + libcudf's actual answer — a shared-memory set of 366 slots per block and a + whole-input re-run in global memory past a cardinality of 128 + (`reading-libcudf.md`, Step 5). +4. Fusing a whole query into one kernel kills operator-at-a-time profiling. What + replaces topic 0's flamegraph on a GPU? (Name the two counters that matter: + achieved occupancy and achieved bandwidth per kernel.) +5. For M18 our hot paths are graph expand (random access), filter (streaming), + distance scoring (dense). Apply Step 7's roofline to each and write the + one-line go/no-go — then check it against the fact that on this machine the + bandwidth ratio is ~1. ## Done when -- [ ] You can state the two regimes and say which one the bus puts you in — then connect it to this topic's measured result: no crossover up to 2^24, with upload alone costing 7197 µs at 16 M elements. -- [ ] You can explain the two memory rules (coalescing, shared memory) and what violating each costs. -- [ ] You can explain why a filter needs a prefix scan rather than an output cursor. -- [ ] You can state why kernel fusion is mandatory rather than an optimization, and what it costs in modularity. -- [ ] You can use the roofline formula to predict a winner before measuring. -- [ ] You wrote answers to all five questions in notes.md, including the unified-memory rewrite of their regime analysis. +Answer each before unfolding it. + +- [ ] You can state the two regimes, name the number that separates them, and connect them to this topic's measured result. + +
Answer + + Regime A (coprocessor): data ships per query, so GPU time ≥ `16L/B_p`. With + the paper's measured PCIe of 12.8 GBps against a CPU read bandwidth of + 53 GBps (Table 2), the CPU's *upper* bound beats the GPU's *lower* bound — + 36.2 ms vs 150 ms on SSB SF20's 120 M rows. Measured: GPU coprocessor 1.4× + slower than Hyper, PCIe-bound on every query (§3.1, Fig 3). + + Regime B (primary store): data resident, ratio 880/53 = 16.2 (§4), and 25× on + the full benchmark (§5.2). + + This topic measured regime A on hardware with no PCIe at all: 7197 µs of + upload against a 2723 µs CPU total (`FINDINGS.md:36`), no crossover to 2²⁴. + The boundary crossing survives the removal of the bus. + +
+ +- [ ] You can quote the caveat that governs every speedup in §4 and §5, and say which of this topic's numbers it invalidates. + +
Answer + + §4: *"In all cases, we assume that the data is already in the respective + device's memory."* §5 repeats it for the workload runs. So 16.56×, 15.8×, + 17.13× and 25× are all regime-B numbers. + + None of them is comparable to `notes.md`'s table, which measures a per-call + upload. The comparable Crystal number is §3.1's coprocessor result, and that + one agrees with ours. + +
+ +- [ ] You can explain the two memory rules and what violating each costs. + +
Answer + + Coalescing: 32 lanes on adjacent addresses = 1 transaction; scattered = up to + 32, most of each discarded. Shared memory: a per-SM scratchpad shared by a + block, but tiny per thread — ~24 4-byte entries at full occupancy on a V100 + (§3.2), against 16 KB of L1 per SM (Table 2). Using more of it per thread + costs occupancy, and occupancy is the only latency-hiding mechanism the GPU + has (Step 1). + + The join's largest-table regime is the concrete price of violating the first: + 128 B GPU read granularity vs 64 B on the CPU halves the expected 16.2× to + 8.1× for random hash-table probes (§4.3). + +
+ +- [ ] You can explain why filter output needs a prefix scan rather than a cursor, and what the tile-based version saves over the three-kernel version. + +
Answer + + With ~32 CPU threads updating a counter once per 1000-entry vector, a shared + cursor is not a bottleneck; with 5000 GPU threads it is (§3.2). A block-wide + exclusive prefix sum computes each element's offset instead of claiming it, + and one atomic per block claims the range — atomics reduced *"by a factor of + size of tile T"*. + + Against the older three-kernel approach (Fig 4a) the fused version saves: the + second pass over the input column, the materialisation of `count` and `pf` in + global memory, and the random output writes, which the shuffle step turns + contiguous (Fig 4b, Fig 6). + +
+ +- [ ] You can say what §4.2 actually found about branch divergence in selection — and what it did not. + +
Answer + + It found **no difference** between `GPU If` and `GPU Pred`: *"A single branch + misprediction does not impact performance on the GPU."* The predication story + in that figure belongs to the CPU curves (`CPU If` < `CPU Pred` < + `CPU SIMDPred`). + + What it did *not* measure is scan-and-compact versus branch-per-thread on the + GPU; the tile-based design is argued from first principles in §3.2, not + benchmarked against the three-kernel alternative. If you want that number you + have to produce it yourself — which is what this topic's `filter_count` stub + is for. + +
+ +- [ ] You can use the roofline to predict a winner, and say why the full-query result beat what it predicts. + +
Answer + + `time = max(bytes / bandwidth, flops / peak_flops)`; the operator's arithmetic + intensity picks the term. Bandwidth-bound operators can win by at most the + bandwidth ratio (16.2), and §4.1-4.4 land at 15.8-17.95 for everything except + the join. + + The full SSB gave 25× (§5.2) because the ceiling is per operator, not per + chained query: the CPU loses time between vectorised operators that the model + does not predict (q2.1: 47 ms modelled, 125 ms actual) while the GPU stays + near its model even through irregular probes, thanks to 64 resident warps per + SM (§5.3). + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the unified-memory rewrite of the regime analysis. + +
Answer + + The slots are `notes.md:55-61`. Question 2's rewrite needs the two measured + numbers from `notes.md:16` — 9.1 GB/s effective upload, 29.7 GB/s CPU read — + not the paper's. + +
## References **Papers** -- Shanbhag, Madden, Yu — "A Study of the Fundamental Performance - Characteristics of GPUs and CPUs for Database Analytics" - (SIGMOD 2020) — §2-3 for the tile model, §5-6 for the two-regime - measurements; the CPU-baseline-honesty discussion is worth reading - even if you never touch a GPU + +- Anil Shanbhag, Samuel Madden, Xiangyao Yu — *"A Study of the Fundamental + Performance Characteristics of GPUs and CPUs for Database Analytics"*, SIGMOD + 2020. Extended version: [arXiv:2003.01178](https://arxiv.org/abs/2003.01178), + which is what every citation here points at. + Route: §3.1 (the coprocessor bound and Figure 3) → §3.2 with Figures 4, 5, 6 + (the tile model) → §4 opening paragraph (the residency caveat) → §4.1-4.4 + (operator ratios; read §4.3's three regimes separately) → §5.1-5.2 (platform + Table 2, the 25×) → §5.3 (why 25 > 16.2). + +**Measurements in this repo** + +- `FINDINGS.md:36` — this topic's headline: no crossover to 2²⁴; 7197 µs upload + against a 2723 µs CPU total at 16 M. +- `topics/18-gpu/notes.md:9-16` — the phase-split table Step 3's local + arithmetic uses. + +**Related guides** + +- `reading-wgpu-compute.md` — the same boundary-crossing cost, on hardware you + can actually run. +- `reading-libcudf.md` — Steps 5 and 6 as shipped production CUDA, with the + atomic-amortisation rule at warp granularity instead of block. diff --git a/topics/18-gpu/reading-faiss-gpu.md b/topics/18-gpu/reading-faiss-gpu.md index 5cd3ae0..2aa4514 100644 --- a/topics/18-gpu/reading-faiss-gpu.md +++ b/topics/18-gpu/reading-faiss-gpu.md @@ -1,196 +1,621 @@ # Faiss GPU: k-select that never leaves registers -Johnson, Douze & Jégou's 2017 paper made GPU ANN real: IVF-PQ -(topic 14's quantization ladder) at billion scale, built around one -algorithmic contribution — k-selection that never leaves registers — -and one systems discipline: keep the index resident, stream only -queries. It's Crystal's regime B practiced before Crystal named it. -This chapter builds the six concepts the paper assumes — the IVF-PQ -vocabulary, the residency rule, why top-k selection was the -bottleneck, and how a sorting network fixes it — then routes you to -the two sections that matter. +Johnson, Douze and Jégou's 2017 paper made billion-scale GPU ANN real, and it +did it with one algorithmic idea and one systems discipline. The idea: +*k*-selection whose entire state lives in registers, so the step everyone else +staged through memory becomes free. The discipline: the index is resident on the +device and only queries cross the bus — Crystal's regime B, practiced three +years before Crystal named it. + +This chapter builds the vocabulary the paper assumes (IVF, PQ, ADC), then the +two ideas, then the limits the implementation actually enforces — which are not +the ones the wiki lists. + +You cannot run any of it here: the GPU half of Faiss is CUDA, and this machine +has no CUDA device. Every number below is either quoted from the paper with the +2017 hardware it was measured on, or read out of source. The only measurements +this repo owns are on the wgpu/Metal lane (`notes.md:9-16`), and they measure a +sum, not a search. + +Code anchors are [facebookresearch/faiss](https://github.com/facebookresearch/faiss) +at tag **v1.15.0** — note that faiss is *not* in this repo's pin table +(`resources/codebases.md`), so the tag is the pin; reproduce any anchor with +`python3 tools/pinned-source.py --ref v1.15.0 show facebookresearch/faiss +-r A:B`. Paper citations are to +[arXiv:1702.08734](https://arxiv.org/abs/1702.08734) (IEEE Trans. Big Data +2019). ## The problem in one sentence -An IVF scan produces millions of candidate distances per query and -you need the 100 best; sorting them costs O(n log n) of memory -traffic and the CPU's answer — a heap — is serial and branchy, so on -a 32-lane lockstep GPU the *selection*, not the distance math, was -the bottleneck. +An IVF scan produces millions of candidate distances per query and you want the +best 100; the CPU's answer is a heap, which is serial and branchy and therefore +the worst possible warp code — so on a GPU the *selection*, not the distance +arithmetic, was the bottleneck, and the paper's fix is to keep the selector's +whole state in registers and never write a distance to memory at all. ## The concepts, step by step -### Step 1 — the IVF-PQ vocabulary (topic 14 in five terms) - -Faiss's billion-scale index compresses and partitions before it ever -computes a distance. **IVF** (inverted file): cluster all vectors -into ~√n buckets by a **coarse quantizer** (k-means centroids — -finding a query's nearest centroids is a small brute-force matrix -multiply); at query time scan only the `nprobe` nearest buckets. -**PQ** (product quantization): split each vector into m sub-vectors -(e.g. 8), quantize each to 1 byte against its own 256-entry -codebook — a 128-dim f32 vector (512 B) becomes an 8-byte code, so -1B vectors fit in 8 GB. **ADC** (asymmetric distance computation): -the query stays uncompressed; per query you precompute a 256-entry -distance table per sub-quantizer, and each candidate's distance is -just m table lookups. That turns "distance to 1M candidates" into a -memory-bandwidth problem — which is exactly what the GPU has to -sell. - -### Step 2 — the residency rule: the index lives on the device - -A discrete GPU's HBM (high-bandwidth memory, ~900 GB/s) is reachable -from the host only over PCIe (~16 GB/s), so Faiss places each piece -of data by its lifetime — permanent things on the fast side, the -per-query trickle over the bus: - -``` - what where why - PQ codes (1B×8B) GPU HBM (8-32 GB) scanned every query — needs bandwidth - coarse centroids HBM tiny - original vectors CPU RAM / disk only for optional rescore - queries PCIe per batch small — the ONLY per-query transfer -``` - -Crystal's regime B by design: the billion-scale index lives on -device; a query batch ships kilobytes, not gigabytes. Question: our -gpu_bench shipped the DATA per call and lost everywhere — restate -Faiss's layout rule as a rule about which side of the bus each -data-lifetime class belongs on. - -### Step 3 — why heaps fail on warps - -A warp is 32 threads executing one instruction in lockstep. A binary -heap's insert takes a *data-dependent* path — compare, maybe swap, -maybe recurse — so 32 lanes inserting 32 different values each want -a different instruction sequence, and the warp serializes -(divergence: both paths execute, lanes masked). A **sorting -network** does the opposite: a *fixed* schedule of compare-exchange -operations that is identical no matter what the data is — every lane -executes the same instruction always, and "maybe swap" becomes a -branch-free min/max. Data-independent schedule = zero divergence — -the same reason branchless filter won at 50% selectivity in -topic 17. - -### Step 4 — WarpSelect: the top-k machine (their §4, the real contribution) - -The contribution: keep the whole k-selection state in **registers** -(each thread's private, fastest storage — zero memory traffic) and -communicate only by **warp shuffles** (instructions that move values -lane-to-lane without touching memory): - -``` - each lane keeps a tiny sorted queue IN REGISTERS - insert: compare-exchange against lane's queue (predicated, no branch) - when any lane's queue overflows → odd-even merge network across - the warp (warp shuffles, no shared memory), rebuild thresholds - end: merge 32 lane-queues once → warp's top-k -``` - -```rust -// WarpSelect, one lane's view: a tiny sorted queue in REGISTERS -let mut queue = [f32::INFINITY; Q]; // lane-local register array -let mut threshold = f32::INFINITY; // the warp's current kth-best -for d in my_stripe_of_distances { - if d < threshold { // overwhelmingly false → no work - queue.insert_sorted(d); // predicated compare-exchange - } - if ballot_any_lane_full() { // warp vote, no shared memory - odd_even_merge_across_warp(); // fixed schedule = zero divergence - threshold = kth_best(); // queues drain, threshold tightens - } -} -// end: merge the 32 lane-queues once → the warp's top-k -``` - -One pass over the distances, k-select at register speed. The fast -path is the `if d < threshold` test: as the threshold tightens, -almost every distance fails it and costs one predicated compare. -This is topic 17's "sorting networks beat comparison sorts at small -fixed n" scaled to warps — and CAGRA's bitonic itopk is its -descendant. Question: why do sorting NETWORKS (fixed -compare-exchange schedule) fit SIMT while heaps don't -(data-independent schedule = no divergence — the same reason -branchless filter won at 50% selectivity)? - -### Step 5 — the full query pipeline on device - -With Steps 1–4 in hand, the whole IVF-PQ query is four stages, each -placed in the memory tier it needs: - -- coarse quantizer: query → nprobe nearest inverted lists (a small - brute-force matmul — cuBLAS) -- ADC lookup tables: per query × subquantizer, built in shared - memory (256 entries × m subquantizers) -- scan: each thread streams PQ codes, 8 table lookups per 8-byte - code, feeds WarpSelect — crucially *fused*: distances flow - straight into k-select, never materialized to HBM -- batch everything: queries × lists tiled to saturate SMs - -Question: the ADC tables are per-QUERY — at what batch size does -shared memory run out, and what's the fallback (smaller tiles, or -float16 tables)? Compare CAGRA's shared-memory budget fight. - -### Step 6 — the numbers that set expectations (2017 hardware, still directive) - -- brute-force k-NN on 1M×128d: ~20× over CPU (dense matmul — the - best case; this is our l2_batch stub's ceiling shape) -- billion-scale IVF-PQ: ~8.5× over prior GPU art; k-select was the - bottleneck they removed -- multi-GPU: shard lists (data parallel) or replicate (query - parallel) — topic 15's scaling menu, verbatim - -What transfers to M14/M18: our M14 pipeline (PQ scan → rescore) -maps 1:1 — PQ scan is the GPU-shaped half (regular, -bandwidth-bound, k-select), rescore is gather-heavy (CPU keeps it -unless candidates batch well). M18's distance-scoring flag should -implement the brute-force tile first — it's the ~20× case above and -needs no index redesign. - -## How to read the paper (with the concepts in hand) - -- **§4 (k-selection) — read carefully.** This is Steps 3–4, the - actual contribution: the thread-queue/warp-queue split, the - odd-even merge network, and the measured selection throughput. - Watch for the overflow threshold t (question 1). -- **§5 (the system) — the layout table.** Step 2's residency - discipline and Step 5's fused pipeline in the authors' words; - the multi-GPU sharding/replication menu ends the section. -- Skim the rest: §2–3 are Step 1's IVF-PQ background (topic 14 - covered it), and the evaluation's absolute numbers are 2017 - hardware — read them as ratios, not throughputs. +### Step 1 — IVF, PQ, ADC: what has to happen before a distance is computed + +> **In:** a billion d-dimensional vectors that do not fit anywhere fast. +> **Out:** an 8-byte code per vector and a query plan that touches ~1 % of them. + +Three compressions, each named: + +- **IVF** (inverted file) — cluster all vectors with a **coarse quantizer** + (k-means centroids, |C₁| of them); at query time compute the query's distance + to every centroid and scan only the `nprobe` nearest lists. The coarse step is + itself a small brute-force search, which is why Step 6's fused kernel matters + twice. +- **PQ** (product quantization) — split each vector into *m* sub-vectors, + quantize each against its own 2^nbits-entry codebook. At m = 8, nbits = 8, a + 128-dim float vector (512 B) becomes 8 bytes, and a billion of them fit in + 8 GB. +- **ADC** (asymmetric distance computation) — the query is *not* quantized. + Per query, build a table of distances from the query's sub-vectors to every + codebook entry; then each candidate's distance is *m* table lookups and *m* + adds. §5.2 develops the expansion; the practical shape is 8 lookups per 8 + bytes of code read. + +That last line is the whole reason ANN is a GPU workload: it converts "distance +to a million candidates" into a bandwidth problem with a tiny table, and +bandwidth is what a discrete GPU sells. + +### Step 2 — the residency rule, with the only transfer rate this repo measured + +> **In:** a memory hierarchy split by a bus. +> **Out:** a placement rule per data class — and an order-of-magnitude reason to +> obey it. + +``` + data class lives why + PQ codes (1B x 8B) device memory scanned every query; needs bandwidth + coarse centroids device memory small, touched by every query + full-precision host RAM / disk only for optional rescore + queries cross the bus, batched kilobytes, the only per-query traffic +``` + +The paper simply assumes this and reports numbers against it; `reading-crystal- +sigmod20.md` is where the assumption is made explicit and quantified (Crystal's +Table 2: 880 GBps device memory against 53 GBps for the CPU, with PCIe rated at +"up to 16 GBps" in §2.2 and measured at 12.8 GBps in §5). Do the placement +arithmetic with the only numbers this repo has measured itself — the wgpu/Metal +lane at 16 M elements, `notes.md:16`: + +``` + measured upload rate (16M elems, 67,108,864 B in 7384.7 us) = 9.09 GB/s + measured CPU sum rate over the same bytes (2257.7 us) = 29.7 GB/s + + index, 1e9 vectors x 8 B PQ code = 8.0 GB + one-time upload at 9.09 GB/s = 0.88 s + query batch, 1e4 queries x 128 dims x 4 B = 5.12 MB + per-batch upload at 9.09 GB/s = 0.56 ms + ------ + ratio, index bytes : query bytes = 1562 : 1 +``` + +Resident, you pay 0.88 s once. Non-resident, you pay it per batch — 1562× the +traffic of the thing you actually needed to send. This topic's own headline is +the same lesson without the index: at 16 M elements upload alone costs 7197 µs +against a 2723 µs CPU total (`FINDINGS.md:36`). + +### Step 3 — why a heap is the wrong data structure on a warp + +> **In:** 32 lanes issuing one instruction. +> **Out:** an argument for fixed-schedule compare-exchange networks over +> anything with a data-dependent control path. + +A binary heap's sift-down branches on comparisons, so 32 lanes inserting 32 +different values want 32 different instruction sequences. The warp executes both +sides of every divergent branch with the wrong lanes masked off; a k-selection +built from heaps runs at a fraction of issue rate no matter how good the memory +system is. + +A **sorting network** has a schedule fixed before the data exists: the same +compare-exchange pairs in the same order, "maybe swap" compiled to branch-free +min/max. Zero divergence by construction — the same property that made +branchless filtering win in topic 17, and the property CAGRA's bitonic top-M +relies on (`reading-cagra.md`, Step 5). + +Faiss's networks are *odd-size*, and the paper is precise about why that +mattered: Batcher's classic formulation *"would require that 32t = k and is a +power-of-2; thus if k = 1024, t must be 32. We found that the optimal t is way +smaller"* (§4.3). So the paper builds `merge-odd` and `sort-odd` (Algorithms 1 +and 2), which merge arrays of unequal, non-power-of-two lengths in +`⌈log₂(max(ℓL, ℓR))⌉ + 1` parallel steps. Calling them "odd-even merge networks" +— as an earlier version of this guide did — names the wrong thing: odd-*even* is +Batcher's, and avoiding its size constraint is the contribution. + +### Step 4 — WarpSelect: two queues, both in registers + +> **In:** a stream of ℓ distances per query. +> **Out:** the k smallest, in one pass, with no shared memory and no cross-warp +> synchronisation. + +The paper's own summary: *"Our k-selection implementation, WarpSelect, maintains +state entirely in registers, requires only a single pass over data and avoids +cross-warp synchronization… Since the register file provides much more storage +than shared memory, it supports k ≤ 1024"* (§4.2). + +Two levels. Each lane owns a **thread queue** of *t* elements in registers, and +the warp collectively owns a **warp queue** of k elements held as a *lane-stride +register array* — element i lives in lane `i % 32`, so a "shared" array costs no +memory at all. Lane j reads elements a_j, a_{32+j}, … so the reads are +*"contiguous and coalesced into a minimal number of memory transactions"* +(§4.2). + +The whole update rule is 45 lines of C++: + +```cpp +// faiss/gpu/utils/Select.cuh:439-447 and 469-482 — construction and the fast +// path. kLane is where the current kth-best lives in the lane-stride array. + 439 struct WarpSelect { + 440 static constexpr int kNumWarpQRegisters = NumWarpQ / kWarpSize; + 442 __device__ inline WarpSelect(K initKVal, V initVVal, int k) + 443 : initK(initKVal), + 444 initV(initVVal), + 445 numVals(0), + 446 warpKTop(initKVal), + 447 kLane((k - 1) % kWarpSize) { +... + 469 __device__ inline void addThreadQ(K k, V v) { + 470 if (Dir ? Comp::gt(k, warpKTop) : Comp::lt(k, warpKTop)) { + 471 // Rotate right + 472 #pragma unroll + 473 for (int i = NumThreadQ - 1; i > 0; --i) { + 474 threadK[i] = threadK[i - 1]; + 475 threadV[i] = threadV[i - 1]; + 476 } + 478 threadK[0] = k; + 479 threadV[0] = v; + 480 ++numVals; + 481 } + 482 } +``` + +Note what `addThreadQ` is not: there is no memory access, no `__syncthreads`, +and the loop is `#pragma unroll` over a compile-time constant so `threadK[]` +stays in registers. The expensive path runs only when some lane is full, and the +warp finds out with a single ballot: + +```cpp +// faiss/gpu/utils/Select.cuh:484-512 — the slow path, and how the threshold is +// republished afterwards. + 484 __device__ inline void checkThreadQ() { + 485 bool needSort = (numVals == NumThreadQ); + 490 needSort = __any_sync(0xffffffff, needSort); + 493 if (!needSort) { + 494 // no lanes have triggered a sort + 495 return; + 496 } + 498 mergeWarpQ(); + 500 // Any top-k elements have been merged into the warp queue; we're + 501 // free to reset the thread queues + 502 numVals = 0; + 510 // We have to beat at least this element + 511 warpKTop = shfl(warpK[kNumWarpQRegisters - 1], kLane); +``` + +Line 511 is the design in one statement: the new rejection threshold is the +current kth-best, fetched from another lane by a **shuffle** — a register-to- +register instruction. Nothing about this algorithm ever addresses memory. + +The tuning parameter *t* is chosen per k. §4.3: *"For k ≤ 32, we use t = 2, +k ≤ 128 uses t = 3, k ≤ 256 uses t = 4, and k ≤ 1024 uses t = 8, all +irrespective of ℓ."* The shipped instantiations agree, one file per k, with the +thread-queue length as the last macro argument: + +``` + faiss/gpu/utils/warpselect/WarpSelectFloat1.cu:13 WARP_SELECT_IMPL(float, true, 1, 1) + faiss/gpu/utils/warpselect/WarpSelectFloat32.cu:13 WARP_SELECT_IMPL(float, true, 32, 2) + faiss/gpu/utils/warpselect/WarpSelectFloat64.cu:13 WARP_SELECT_IMPL(float, true, 64, 3) + faiss/gpu/utils/warpselect/WarpSelectFloat128.cu:13 WARP_SELECT_IMPL(float, true, 128, 3) + faiss/gpu/utils/warpselect/WarpSelectFloat256.cu:13 WARP_SELECT_IMPL(float, true, 256, 4) + faiss/gpu/utils/warpselect/WarpSelectFloatF512.cu:13 WARP_SELECT_IMPL(float, false, 512, 8) + faiss/gpu/utils/warpselect/WarpSelectFloatF1024.cu:13 WARP_SELECT_IMPL(float, false, 1024, 8) + faiss/gpu/utils/warpselect/WarpSelectFloatF2048.cu:15 WARP_SELECT_IMPL(float, false, 2048, 8) +``` + +The last line is a correction to make: the paper's ceiling was 1024, and the +shipped ceiling is **2048**, conditional on the compiler: + +```cpp +// faiss/gpu/utils/DeviceDefs.cuh:61-68 — the selection ceiling is a register +// allocation question, and the comment says so. + 61 #if CUDA_VERSION > 9000 + 62 // Based on the CUDA version (we assume what version of nvcc/ptxas we were + 63 // compiled with), the register allocation algorithm is much better, so only + 64 // enable the 2048 selection code if we are above 9.0 (9.2 seems to be ok) + 65 #define GPU_MAX_SELECTION_K 2048 + 66 #else + 67 #define GPU_MAX_SELECTION_K 1024 + 68 #endif +``` + +### Step 5 — what the ceiling costs: registers, counted + +> **In:** `kNumWarpQRegisters = NumWarpQ / kWarpSize` (`Select.cuh:440`). +> **Out:** the per-lane register bill, and the reason there is a ceiling at all. + +The warp queue is spread across the warp, so k elements cost k/32 registers per +lane — for keys, and again for values. The thread queue costs t of each. Count +it for the two extremes actually shipped: + +``` + k = 128, t = 3 (WarpSelectFloat128.cu) + warp queue = 128 / 32 = 4 key regs + 4 value regs = 8 + thread queue = 3 = 3 key regs + 3 value regs = 6 + ---- + per lane = 14 regs + per warp = 14 x 32 lanes = 448 regs + + k = 2048, t = 8 (WarpSelectFloatF2048.cu) + warp queue = 2048 / 32 = 64 key regs + 64 value regs = 128 + thread queue = 8 = 8 key regs + 8 value regs = 16 + ---- + per lane = 144 regs + per warp = 144 x 32 lanes = 4608 regs +``` + +144 registers per lane for queue state alone, before the kernel's own working +set — which is exactly why `DeviceDefs.cuh:62-64` makes 2048 conditional on a +compiler with a better allocator, and why the paper stopped at 1024 in 2017. +Registers are not free storage; they are the scarcest storage, and spilling them +would defeat the entire design. + +The contrast is in the same file. `BlockSelect` — the whole-block variant used +when one warp per query is not enough parallelism — puts the warp queues in +shared memory instead: + +```cpp +// faiss/gpu/utils/Select.cuh:177-187 — BlockSelect's queues are slices of a +// shared-memory array, one per warp, not registers. + 177 int laneId = getLaneId(); + 178 int warpId = threadIdx.x / kWarpSize; + 179 warpK = sharedK + warpId * kTotalWarpSortSize; + 180 warpV = sharedV + warpId * kTotalWarpSortSize; + 182 // Fill warp queue (only the actual queue space is fine, not where + 183 // we write the per-thread queues for merging) + 184 for (int i = laneId; i < NumWarpQ; i += kWarpSize) { + 185 warpK[i] = initK; + 186 warpV[i] = initV; + 187 } +``` + +and the caller pays for it in shared memory that scales with warps × k +(`L2Select.cu:146-149`): at 128 threads (4 warps) and k = 1024 that is +4 × 1024 × (4 + 4) = 32 KB per block. Same algorithm, different storage class, +completely different occupancy. + +### Step 6 — fusion: the pass that never happens + +> **In:** a GEMM that produces an nq × ℓ partial distance matrix. +> **Out:** two passes over it instead of three — and a measured 25 % for the +> pass you skipped. + +Exact search is a matrix multiply: cuBLAS computes the −2⟨x_j, y_i⟩ term into a +partial matrix D′, and then the ‖y_i‖² term has to be added and the top-k taken. +The naive pipeline writes D′, reads it to add norms, writes it again, reads it +to select. Faiss adds and selects in the same kernel: + +> *"To complete the distance calculation, we use a fused k-selection kernel that +> adds the ‖y_i‖² term to each entry of the distance matrix and immediately +> submits the value to k-selection in registers… Kernel fusion thus allows for +> only 2 passes (GEMM write, k-select read) over D′, compared to other +> implementations that may require 3 or more."* (§5.1) + +The kernel is 50 lines and does exactly what the sentence says: + +```cpp +// faiss/gpu/impl/L2Select.cu:161-179 — the fused add-and-select loop. There is +// no intermediate distance array: `v` is computed and consumed in registers. + 161 IndexT row = blockIdx.x; + 163 // Whole warps must participate in the selection + 164 IndexT limit = utils::roundDown(productDistances.getSize(1), kWarpSize); + 165 IndexT i = threadIdx.x; + 167 for (; i < limit; i += blockDim.x) { + 168 T v = Math::add(centroidDistances[i], productDistances[row][i]); + 169 heap.add(v, IndexT(i)); + 170 } + 172 // Handle the remainder if any separately (warp is divergent) + 173 if (i < productDistances.getSize(1)) { + 174 T v = Math::add(centroidDistances[i], productDistances[row][i]); + 175 heap.addThreadQ(v, IndexT(i)); + 176 } + 178 // Merge all final results + 179 heap.reduce(); +``` + +Measured, on SIFT1M (ℓ = 10⁶, d = 128, nq = 10⁴) on one Maxwell Titan X: the +whole pipeline reaches *"85 % of the peak possible performance, assuming GEMM +usage and our tiling"*, and *"Our same exact algorithm without fusion (requiring +an additional pass through D′) is at least 25 % slower"* (§6.3). The paper also +notes the limit of the idea: *"Row-wise k-selection is likely not fusable with a +well-tuned GEMM kernel"* (§5.1) — you can fuse the cheap pass into the selector, +not the selector into the GEMM. + +Why the two halves of an IVF-PQ query behave differently is arithmetic. Take the +coarse quantizer and the PQ scan with the same batch: + +``` + coarse quantizer: GEMM, nq x d by d x |C1|, with nq = 1024, d = 128, |C1| = 8192 + flops = 2 x nq x |C1| x d = 2.15e9 + bytes = 4 x (nq x d + |C1| x d + nq x |C1|) + = 4 x (131,072 + 1,048,576 + 8,388,608) = 38.3 MB + intensity = 2.15e9 / 38.3e6 = 56 flop/byte + -> compute bound + + PQ scan (ADC), m = 8 bytes per vector + per candidate: 8 table lookups + 8 adds = ~16 ops + per candidate: 8 bytes of code read + intensity = 16 / 8 = 2 ops/byte + -> bandwidth bound +``` + +Two kernels, one query, an intensity gap of ~28×. That is why the paper's +performance story is a GEMM story on one side and a memory story on the other — +and why the selection step had to stop touching memory before either could +matter. + +### Step 7 — the limits the code actually enforces + +> **In:** the IVF-PQ parameters a user picks (m, nbits, d). +> **Out:** the exact rejection rules — several of which are shared-memory +> arithmetic in disguise. + +The ADC table is per query and lives in shared memory, so its size is a hard +constraint checked at index construction: + +```cpp +// faiss/gpu/GpuIndexIVFPQ.cu:594-608 — the ADC lookup table has to fit in +// shared memory, and the comment states the consequence. + 594 // We must have enough shared memory on the current device to store + 595 // our lookup distances + 596 int lookupTableSize = sizeof(float); + 597 if (ivfpqConfig_.useFloat16LookupTables) { + 598 lookupTableSize = sizeof(half); + 599 } + 601 // 64 bytes per code is only supported with usage of float16, at 2^8 + 602 // codes per subquantizer + 603 size_t requiredSmemSize = + 604 lookupTableSize * subQuantizers_ * utils::pow2(bitsPerCode_); + 605 size_t smemPerBlock = getMaxSharedMemPerBlock(config_.device); + 607 FAISS_THROW_IF_NOT_FMT( + 608 requiredSmemSize <= getMaxSharedMemPerBlock(config_.device), +``` + +Evaluate the formula and the comment explains itself: + +``` + requiredSmemSize = lookupTableSize x m x 2^nbits + + nbits = 8 (2^8 = 256 entries per sub-quantizer) + m = 8, float32 tables: 4 x 8 x 256 = 8,192 B fine + m = 32, float32 tables: 4 x 32 x 256 = 32,768 B large + m = 64, float32 tables: 4 x 64 x 256 = 65,536 B rejected on most devices + m = 64, float16 tables: 2 x 64 x 256 = 32,768 B the comment's case +``` + +The rest of `verifyPQSettings_` is a list of restrictions worth reading before +you believe any tutorial: + +- with cuVS enabled: `4 ≤ nbits ≤ 8` **and** `nbits × m` a multiple of 8 + (`:556-565`); +- classic path with `interleavedLayout`: nbits ∈ {4, 5, 6, 8} (`:568-572`); + without it, nbits **must be 8** (`:574-577`); +- `d % m == 0` (`:587-592`); +- without `interleavedLayout`, m must be in an explicit list — + {1, 2, 3, 4, 8, 12, 16, 20, 24, 28, 32, 40, 48, 56, 64, 96}, where 56, 64 and + 96 are *"only supported with float16"* (`faiss/gpu/impl/IVFPQ.cu:80-102`). + +Finally, multi-GPU. The menu is two booleans in a struct, and the distinction is +topic 15's read-scaling question wearing different clothes: + +```cpp +// faiss/gpu/GpuClonerOptions.h:56-67 — shard versus replicate, and the +// IVF-specific middle option. + 56 struct GpuMultipleClonerOptions : public GpuClonerOptions { + 57 /// Whether to shard the index across GPUs, versus replication + 58 /// across GPUs + 59 bool shard = false; + 62 int shard_type = 1; + 64 /// set to true if an IndexIVF is to be dispatched to multiple GPUs with a + 65 /// single common IVF quantizer, ie. only the inverted lists are sharded on + 66 /// the sub-indexes (uses an IndexShardsIVF) + 67 bool common_ivf_quantizer = false; +``` + +Replicate: every GPU holds the whole index, queries are divided, results are +independent — the paper measures *"3.16× for 4 GPUs with 4096 centroids"* on +k-means (§6.2), i.e. near-linear read scaling. Shard: each GPU holds part of the +index, every query goes to every GPU, and the partial top-k lists must be merged +— which is what DEEP1B needed, *"4 GPUs with S = 2, R = 2"* because 20 GB does +not fit one 2017 device (§6.4). + +## Where each step lives in the code + +| anchor | what it is | step | +|---|---|---| +| `faiss/gpu/utils/Select.cuh:439-447` | `WarpSelect` state: `kNumWarpQRegisters`, `kLane` | 4-5 | +| `faiss/gpu/utils/Select.cuh:469-482` | `addThreadQ` — the branch-predicated fast path | 4 | +| `faiss/gpu/utils/Select.cuh:484-512` | `checkThreadQ` — ballot, merge, republish the threshold by shuffle | 4 | +| `faiss/gpu/utils/Select.cuh:517-532` | `mergeWarpQ` — the odd-size sort/merge networks | 3-4 | +| `faiss/gpu/utils/Select.cuh:147-190` | `BlockSelect`: same algorithm, queues in shared memory | 5 | +| `faiss/gpu/utils/DeviceDefs.cuh:61-68` | `GPU_MAX_SELECTION_K` — 2048 above CUDA 9.0, else 1024 | 4-5 | +| `faiss/gpu/utils/warpselect/WarpSelectFloat*.cu:13` | one instantiation per k, last macro argument is *t* | 4 | +| `faiss/gpu/impl/L2Select.cu:137-186` | `l2SelectMinK` — the fused add-and-select kernel | 6 | +| `faiss/gpu/impl/L2Select.cu:24-70` | `l2SelectMin1` — the k = 1 special case (a block reduction, not WarpSelect) | 6 | +| `faiss/gpu/GpuIndexIVFPQ.cu:544-619` | `verifyPQSettings_` — every restriction, in one function | 7 | +| `faiss/gpu/impl/IVFPQ.cu:80-102` | `isSupportedPQCodeLength` — the legal m values | 7 | +| `faiss/gpu/GpuClonerOptions.h:56-67` | shard vs replicate vs common-quantizer sharding | 7 | + +Reading order: `Select.cuh` from line 426 (WarpSelect) — it is the paper's §4 +and it is short; then scroll *up* to `BlockSelect` at line 147 to see the same +algorithm with a different storage class. Then `L2Select.cu` for the fusion, and +`GpuIndexIVFPQ.cu:544` last, which reads like a changelog of everything the +kernels cannot do. In the paper: §4 is the contribution (§4.2 the algorithm, +§4.3 the choice of *t*), §5 is the system, §6.1/§6.3/§6.4 the measurements. ## Questions for notes.md -1. WarpSelect keeps k ≤ ~1024 in registers per warp. What breaks - at larger k, and what did they use before overflow (thread-queue - + warp-queue two-level — find the threshold t)? -2. Faiss streams distances INTO k-select fused (no materialized - distance array). Crystal made the same fusion argument — what's - the HBM traffic ratio, fused vs staged, for 1M distances/query? -3. The coarse quantizer is a matmul (batch queries × centroids) — - why does THIS piece hit near-peak FLOPs while the PQ scan is - bandwidth-bound (arithmetic intensity of each)? -4. Their multi-GPU sharding sends every query to every shard; - replication doesn't. Map to topic 15's read-scaling vs - partitioning — which does recall@k prefer (shard = exact merge, - replica = independent)? -5. For M18: l2_batch(1 query × 100K targets, dim 128) ≈ their - brute-force case at batch 1. Predict from the roofline whether - Metal wins BEFORE running your implementation — then check. +1. The paper caps WarpSelect at k ≤ 1024 (§4.2) and the shipped code at 2048 + (`DeviceDefs.cuh:61-68`). Redo Step 5's register count for the k you would + actually use, find *t* for it in `faiss/gpu/utils/warpselect/`, and say what + the thread queue is *for* — what does raising *t* buy and cost (§4.3's N₂C₂ + against N₃C₃)? +2. Fused versus staged: the paper says 2 passes over D′ instead of 3 or more + (§5.1) and measures ≥ 25 % (§6.3). For nq = 1, ℓ = 10⁶ float distances, + compute both traffic figures in bytes and the ratio. Why is the measured + penalty smaller than the traffic ratio suggests? +3. Redo Step 6's two intensity calculations with your own nq, d, |C₁| and m. + At what nq does the coarse GEMM stop being compute-bound? (Hint: the nq × |C₁| + output term is the one that grows.) +4. Shard sends every query to every GPU and merges partial top-k lists; + replicate divides the queries and merges nothing. Map both onto topic 15's + read-scaling vocabulary, and say which one recall@k is indifferent to and why + (§6.2's 3.16×/4 GPUs is the replicate data point). +5. For M18: `l2_batch` (1 query × 100 K targets, dim 128) is the paper's exact + search at batch 1. Predict from Step 6's intensity arithmetic whether Metal + wins end-to-end *before* implementing it, write the prediction in the table at + `notes.md:34-41`, then measure. This topic's baseline says no crossover to + 2²⁴ (`FINDINGS.md:36`) — does your prediction agree, and if it does, what + would have to change for it not to? ## Done when -- [ ] You can state the residency rule and what it implies about index size versus device memory. -- [ ] You can explain why heaps fail on warps and what WarpSelect does instead. -- [ ] You can say what breaks when k exceeds roughly 1024 per warp. -- [ ] You can explain why fusing distance computation into k-select matters more than either part alone. -- [ ] You wrote answers to all five questions in notes.md, including the `l2_batch` comparison against their reported figures. +Answer each before unfolding it. + +- [ ] You can state the residency rule and put a number on what breaking it costs. + +
Answer + + PQ codes and centroids on the device, full-precision vectors on the host, + queries the only per-query traffic. At m = 8 a billion-vector index is 8 GB + and a 10 K × 128-dim float query batch is 5.12 MB — 1562:1. On this repo's + measured upload rate (9.09 GB/s, `notes.md:16`) that is 0.88 s once versus + 0.56 ms per batch; re-uploading per batch multiplies query traffic by 1562. + +
+ +- [ ] You can explain why heaps fail on warps, and name the network family Faiss uses instead — precisely. + +
Answer + + Heap inserts branch on data, so 32 lanes want 32 instruction sequences and the + warp serialises the divergent paths. Sorting networks fix the schedule at + compile time; "maybe swap" becomes branch-free min/max. + + Faiss uses **odd-size** networks (`merge-odd`, `sort-odd`, Algorithms 1-2), not + Batcher's odd-even merge — because Batcher requires `32t = k` with k a power of + two, forcing t = 32 at k = 1024, and the measured optimum for t is far smaller + (§4.3). + +
+ +- [ ] You can describe WarpSelect's two queues and say where each lives. + +
Answer + + Per-lane **thread queue** of t elements in that lane's registers, a + first-level filter: reject anything worse than the warp's current kth-best + (`Select.cuh:469-470`). Warp-wide **warp queue** of k elements held as a + lane-stride register array — element i in lane `i % 32`, `kNumWarpQRegisters = + k / 32` registers per lane (`:440`). + + When any lane fills up, `__any_sync` detects it (`:490`), the queues are sorted + and merged, and the new threshold is broadcast with a shuffle + (`:511`). No shared memory, no `__syncthreads`, one pass. + +
+ +- [ ] You can say what sets the k ceiling, with the register arithmetic. + +
Answer + + Registers. At k = 2048, t = 8: 2048/32 = 64 key + 64 value registers for the + warp queue plus 8 + 8 for the thread queue = **144 per lane** before the + kernel's own working set. The paper stopped at 1024 (§4.2); + `DeviceDefs.cuh:61-68` raises it to 2048 only when compiled above CUDA 9.0, + attributing the change to a better register allocator. + + `BlockSelect` avoids the register bill by putting the queues in shared memory + (`Select.cuh:179-180`) — and then pays warps × k × 8 bytes of it. + +
+ +- [ ] You can explain what "fused" removes, and quote the measured cost of not fusing. + +
Answer + + Adding ‖y‖² and taking the top-k in the same kernel means the partial distance + matrix D′ is written once by the GEMM and read once by the selector — *"only 2 + passes… compared to other implementations that may require 3 or more"* + (§5.1). `L2Select.cu:167-170` shows the value computed and consumed in a + register. + + Unfused is *"at least 25 % slower"* on SIFT1M on one Titan X, and the fused + pipeline reaches 85 % of peak possible (§6.3). + +
+ +- [ ] You can name at least three IVF-PQ GPU restrictions from the source, not the wiki. + +
Answer + + From `GpuIndexIVFPQ.cu:544-619`: nbits must be exactly 8 without + `interleavedLayout` (`:574-577`) and is limited to {4,5,6,8} with it + (`:568-572`); `d % m == 0` (`:587-592`); the ADC table + `lookupTableSize × m × 2^nbits` must fit in shared memory (`:603-608`), which + is why m = 64 at nbits = 8 needs float16 tables — 65,536 B versus 32,768 B. + Plus `isSupportedPQCodeLength`'s explicit m list (`IVFPQ.cu:80-102`), and the + cuVS path's own rule that `nbits × m` be a multiple of 8 (`:556-565`). + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the prediction in question 5 *before* measuring. + +
Answer + + The question slots are `notes.md:95-101`; the `l2_batch` prediction rows are at + `notes.md:34-41` and are meant to be filled in before the stub is implemented. + A prediction written afterwards teaches nothing. + +
## References **Papers** -- Johnson, Douze, Jégou — "Billion-scale similarity search with - GPUs" ([arXiv:1702.08734](https://arxiv.org/abs/1702.08734), IEEE - Trans. on Big Data 2019) — §4 (k-selection) is the real - contribution; §5's layout table is the systems lesson + +- Jeff Johnson, Matthijs Douze, Hervé Jégou — *"Billion-scale similarity search + with GPUs"*, [arXiv:1702.08734](https://arxiv.org/abs/1702.08734); IEEE + Transactions on Big Data, 2019. §4.1 the odd-size networks, §4.2 WarpSelect, + §4.3 the choice of *t*, §5.1 exact search and fusion, §5.2 the PQ lookup + tables, §6 the experiments — all on 2×2.8 GHz Xeon E5-2680v2 with 4 Maxwell + Titan X GPUs on CUDA 8.0 (§6). Read the numbers as ratios on 2017 hardware: + 55 % of peak at k = 100 but 16 % at k = 1000 (§6.1); 1.62× and 2.01× over fgknn + at ℓ = 128000 (§6.1); 8.5× over Wieschollek et al. on SIFT1B at equal memory + (§6.4). + +**Code** + +- [faiss](https://github.com/facebookresearch/faiss) @ **v1.15.0** — not in this + repo's pin table; the tag is the pin. CUDA only, so this guide reads it rather + than running it. Route: `faiss/gpu/utils/Select.cuh` → + `faiss/gpu/impl/L2Select.cu` → `faiss/gpu/GpuIndexIVFPQ.cu`. + +**Measurements in this repo** + +- `topics/18-gpu/notes.md:9-16` — the 9.09 GB/s upload and 29.7 GB/s CPU rates + Step 2's residency arithmetic uses. A sum kernel on Apple unified memory, not + a search, and not PCIe. +- `FINDINGS.md:36` — no crossover to 2²⁴; 7197 µs upload against a 2723 µs CPU + total at 16 M. + +**Related guides** + +- `reading-crystal-sigmod20.md` — the residency argument, quantified, with the + bandwidth numbers Step 2 leans on. +- `reading-cagra.md` — the other GPU k-select design: bitonic, in shared memory, + inside the search loop rather than after it. +- `reading-libcudf.md` — the same two-pass-versus-one-pass argument in a + relational engine. diff --git a/topics/18-gpu/reading-gunrock.md b/topics/18-gpu/reading-gunrock.md index e0ea038..a312403 100644 --- a/topics/18-gpu/reading-gunrock.md +++ b/topics/18-gpu/reading-gunrock.md @@ -1,205 +1,552 @@ # Gunrock: advance, filter, and the ragged-frontier problem -The GPU graph framework that reduced every graph algorithm to two -data-parallel operators over frontiers — and then spent its research -budget on the problem hiding inside: adjacency lists are RAGGED, and -warps hate ragged. This chapter builds the ideas in order — frontier -traversal, the two-operator model, why power-law degrees wreck naive -work assignment, and the three load-balancing strategies that answer -it — then maps each to the modern "Essentials" codebase. Read that -code alongside the paper; the load-balancing menu in -`operators/advance/` is the chapter's core. +The GPU graph framework that reduced every graph algorithm to two data-parallel +operators over frontiers — and then spent its research budget on the problem +hiding inside: adjacency lists are ragged, and warps hate ragged. This chapter +builds the ideas in order (frontier traversal, the two-operator model, why +power-law degrees wreck naive work assignment, and the strategies that answer +it) and maps each to the modern "Essentials" codebase. + +Two warnings before you start. First, none of this runs here: Gunrock is CUDA +(or HIP) and this machine has no such device, so every claim below is a claim +about source, anchored to a line. Second, the API moved between Gunrock 1.x and +Essentials, and most secondary descriptions of Gunrock — including this guide's +previous version — describe the old one. Where the code and the paper disagree, +this guide says so and takes the code. + +Every code anchor is +[gunrock/gunrock@748f79e](https://github.com/gunrock/gunrock), the pinned +revision; check any of them with `python3 tools/pinned-source.py show gunrock + -r A:B`. Paper citations are to +[arXiv:1501.05387](https://arxiv.org/abs/1501.05387) (PPoPP 2016). ## The problem in one sentence -In one BFS frontier, vertex degrees range from 1 to 10⁷ — assign one -thread per vertex and a single hub keeps one thread busy for -~10⁶ edge visits while thousands of warps sit idle, so the real -research problem is not the algorithm but dividing ragged work -evenly. +In one BFS frontier the vertex degrees span three orders of magnitude — this +repo's own generated graph has a p50 degree of 11 and a maximum of 6565 +(`topics/13-graph-engines/README.md:10-12`) — so assigning one thread per vertex +means one lane runs 6565 serial iterations while its 31 warp-mates and the rest +of the device wait. ## The concepts, step by step ### Step 1 — frontier-based traversal: graph algorithms as rounds -A **frontier** is the set of vertices active in the current round of -a graph algorithm. BFS (breadth-first search — visit all vertices at -distance 1, then 2, then 3...) is the archetype: the frontier starts -as {source}, each round expands every frontier vertex's neighbors, -and the *unvisited* neighbors become the next frontier. This -round-at-a-time shape is what makes graph algorithms GPU-friendly at -all: within one round, every vertex can be processed in parallel — -the sequential dependency is only *between* rounds. The graph itself -is stored as **CSR** (compressed sparse row — one array of -concatenated adjacency lists plus an offsets array saying where each -vertex's list starts, topic 13's format). - -### Step 2 — the programming model: advance + filter, and a lambda - -Gunrock's claim: every frontier algorithm is a loop over just two -data-parallel operators, specialized by a user **lambda** (a small -per-edge function): - -``` - while frontier not empty: - ADVANCE: frontier → all neighbors, apply user lambda - (BFS lambda: CAS parent; return "keep?" per edge) - FILTER: drop invalids/duplicates → next frontier - - BFS, SSSP, PageRank, connected components = different lambdas, - SAME two operators. GraphBLAS says the same thing with matrices: - advance = SpMV/SpMSpV over the frontier vector, filter = the mask - (topic 20's push/pull duality, imperative edition). -``` - -In code, with BFS's lambda spelled out (**CAS** = compare-and-swap, -an atomic "write only if still unset"): - -```rust -// every graph algorithm = the same two operators + a different lambda -while !frontier.is_empty() { - let next = advance(csr, &frontier, |src, dst| { - // BFS lambda: a LOST race is benign — any parent is a valid tree - parent[dst].compare_exchange(INVALID, src).is_ok() - }); - frontier = filter(next, |v| is_valid(v)); // dedupe/compact -} -// SSSP, PageRank, CC: same loop, different lambda + frontier policy -``` - -bfs.hxx:139-145 is the whole loop: `advance::execute_runtime` then -optionally `filter::execute_runtime` to remove invalids. Question: -BFS works WITHOUT the filter (bfs.hxx:114's comment) — what grows -unbounded if you skip it, and why is that sometimes still faster -(redundant work vs a full extra pass — the "idempotent BFS" trick)? - -### Step 3 — why raggedness breaks warps - -A warp is 32 threads executing in lockstep, and it is fast only when -all 32 lanes have the same amount of work. Adjacency lists give them -wildly different amounts: real graphs are power-law (topic 13), so a -frontier mixes degree-1 leaves with degree-10⁷ hubs. Whatever unit -of work you assign — vertex per thread, vertex per block — some unit -gets a hub and everything else waits. This is Gunrock's actual hard -problem; the two-operator model of Step 2 is just the stage it plays -on. - -### Step 4 — the load-balancing menu: thread, block, merge_path - -Three ways to split a frontier's edges across the device, each dying -on a different degree distribution: - -``` - thread_mapped: thread i ← vertex i good: uniform degree - dies: one hub = one thread - block_mapped: block ← one vertex good: hubs - dies: 1-degree leaves waste 255/256 - merge_path: binary-search the CSR offsets so every thread gets - the same number of EDGES regardless of which vertex - they belong to — perfect balance, pays a search -``` - -merge_path works because CSR's offsets array is a sorted prefix-sum -of degrees: "which vertex does global edge number e belong to?" is -one binary search, so thread t can independently compute its slice -of exactly `total_edges / n_threads` edges. advance.hxx:111-123 -dispatches on a runtime enum — because no single strategy wins; -real frontiers mix hubs and leaves. (CAGRA sidesteps this whole -problem by CONSTRUCTION: fixed-degree graph ⇒ thread_mapped is -perfect. Worth noticing.) Question: merge_path is topic 11's -morsel-stealing idea done with arithmetic instead of a queue — what -property of CSR (sorted prefix offsets) makes the binary search -sufficient? - -### Step 5 — frontier representation: sparse vs dense = push vs pull +> **In:** a CSR graph and a set of active vertices. +> **Out:** the next set of active vertices — with all the parallelism inside a +> round and all the dependency between rounds. + +A **frontier** is the set of vertices active in the current round. BFS is the +archetype: start with `{source}`, expand every frontier vertex's neighbours, +keep the newly-reached ones, repeat. The graph is **CSR** (compressed sparse +row): one array of concatenated adjacency lists plus an offsets array saying +where each vertex's list begins — topic 13's format, and the property that makes +Step 4 possible. + +Gunrock's driver loop is four lines, and the convergence test is exactly what +you would guess: + +``` +// include/gunrock/framework/enactor.hxx:272-278 (the run loop) and 328-330 +// (the default convergence test), quoted separately. + 272 prepare_frontier(get_input_frontier(), *context); + 274 while (!is_converged(*context)) { + 275 loop(*context); + 276 ++iteration; + 277 } + 278 finalize(*context); +... + 328 virtual bool is_converged(gcuda::multi_context_t& context) { + 329 return active_frontier->is_empty(); + 330 } +``` + +`loop` is one kernel launch per round, at least — which is Step 6's problem. + +### Step 2 — the programming model: advance, filter, and a lambda + +> **In:** a frontier and a user lambda over `(source, neighbor, edge, weight)`. +> **Out:** a new frontier, with `false` from the lambda meaning "put an invalid +> sentinel here instead of this neighbour". + +The claim is that every frontier algorithm is a loop over two operators +specialised by a lambda. Here is BFS's, whole: + +``` +// include/gunrock/algorithms/bfs.hxx:105-128 — the search lambda. Lines 116-122 +// are commented out IN THE SOURCE; they are quoted here because they are the +// version most descriptions of Gunrock (including this guide's last one) claim +// is running. + 105 auto search = [distances, single_source, iteration] __host__ __device__( + 106 vertex_t const& source, // ... source + 107 vertex_t const& neighbor, // neighbor + 108 edge_t const& edge, // edge + 109 weight_t const& weight // weight (tuple). + 110 ) -> bool { + 111 // If the neighbor is not visited, update the distance. Returning false + 112 // here means that the neighbor is not added to the output frontier, and + 113 // instead an invalid vertex is added in its place. These invalides (-1 in + 114 // most cases) can be removed using a filter operator or uniquify. + 116 // if (distances[neighbor] != std::numeric_limits::max()) + 117 // return false; + 118 // else + 119 // return (math::atomic::cas( + 120 // &distances[neighbor], + 121 // std::numeric_limits::max(), iteration + 1) == + 122 // std::numeric_limits::max()); + 124 // Simpler logic for the above. + 125 auto old_distance = + 126 math::atomic::min(&distances[neighbor], iteration + 1); + 127 return (iteration + 1 < old_distance); + 128 }; +``` + +Read lines 125-127 carefully, because the correction matters. There is **no +compare-and-swap** and there is **no `parent[]` array**: the state is +`distances[]`, the update is an atomic *min*, and the "did I win?" answer is +derived from the *old* value the atomic returns. Two threads reaching the same +unvisited neighbour in the same round both call `min`; the first sees +`old_distance = INT_MAX` and returns true, the second sees `old_distance = +iteration + 1` and returns `false` because `iteration + 1 < iteration + 1` is +false. Exactly one of them puts the neighbour in the output frontier. + +That is a better primitive than CAS for the same reason `min` is a better +primitive than "test then set": it is commutative and idempotent, so any +interleaving of concurrent updates converges to the same array, and the same +lambda shape extends to SSSP where the winning condition is a genuinely smaller +distance rather than a first write. + +The round body is two calls: + +``` +// include/gunrock/algorithms/bfs.hxx:138-146 — the entire loop body. + 138 // Execute advance operator on the provided lambda + 139 auto advance_load_balance = P->param.options.advance_load_balance; + 140 operators::advance::execute_runtime(G, E, search, advance_load_balance, context); + 142 // Execute filter operator to remove the invalids (if enabled via options). + 143 if (P->param.options.enable_filter) { + 144 auto filter_algorithm = P->param.options.filter_algorithm; + 145 operators::filter::execute_runtime(G, E, remove_invalids, filter_algorithm, context); + 146 } +``` + +The filter is **optional** (`options.enable_filter`), and the comment at +bfs.hxx:111-114 says why it can be: a rejected neighbour is not removed, it is +replaced by an invalid sentinel, so the output frontier is correct but padded. +The paper calls this the *idempotent* advance and is explicit about what the +filter buys — it *"can perform a series of inexpensive heuristics to reduce, but +not eliminate, redundant entries"*, and the non-idempotent variant *"internally +uses atomic operations to guarantee each element appears only once"* (§4.5). +Skipping the filter trades a growing, sentinel-padded frontier against one fewer +full pass per round. + +### Step 3 — why raggedness breaks warps, in this repo's own numbers + +> **In:** a frontier whose vertices have power-law degrees. +> **Out:** a makespan set by the largest list, not by the average — quantified +> below. + +A warp is 32 lanes issuing one instruction together, so a warp finishes when its +*slowest* lane finishes. `thread_mapped` gives each thread one vertex and a +serial loop over its neighbours: + +``` +// include/gunrock/framework/operators/advance/thread_mapped.hxx:58-80 — +// one thread per frontier element, elided at the invalid-vertex guard. + 58 auto thread_mapped = [=] __device__(int const& tid, int const& bid) { + 59 auto v = (input_type == advance_io_type_t::graph) + 60 ? type_t(tid) + 61 : input.get_element_at(tid); + 66 auto total_edges = G.get_number_of_neighbors(v); + 68 for (auto i = 0; i < total_edges; ++i) { + 69 auto starting_edge = G.get_starting_edge(v); + 70 auto e = i + starting_edge; // edge id + 71 auto n = G.get_destination_vertex(e); // neighbor id + 73 bool cond = op(v, n, e, w); + 75 if (output_type != advance_io_type_t::none) { + 76 std::size_t out_idx = segments_ptr[tid] + i; + 77 type_t element = cond ? n : gunrock::numeric_limits::invalid(); + 78 output.set_element_at(element, out_idx); + 79 } + 80 } + 81 }; +``` + +`total_edges` is the loop bound, and it is the vertex's degree. Put this repo's +measured graph through it — 1 M nodes, 16 M directed edges, p50 degree 11, max +degree 6565 (`topics/13-graph-engines/README.md:9-12`) — with a frontier of +100,000 vertices that happens to contain the top-degree node: + +``` + mean degree = 16e6 / 1e6 = 16 edges + edges in the frontier E ~ 100,000 x 16 = 1.6e6 edge visits + perfectly balanced work = 1.6e6 / 100,000 thr = 16 steps per thread + thread_mapped makespan = max degree in frontier = 6565 steps + ---- + makespan / balanced = 6565 / 16 = 410x +``` + +410× is the *load-balance* term alone, before any memory effect. And topic 13 +measured the consequence end to end on the CPU: the same two-hop query is +**101× slower** from supernodes than from random nodes (4914 ns vs 495,378 ns, +`topics/13-graph-engines/README.md:15-19`). Skew is not a GPU problem that CPUs +avoid; it is a graph problem that the GPU's lockstep execution amplifies. + +### Step 4 — the load-balancing menu, as it is actually spelled + +> **In:** the choice of how to map a frontier's edges onto threads. +> **Out:** a `load_balance_t` enum value — of which fewer are usable than the +> enum suggests. + +The menu is seven entries and three of them are marked work-in-progress: + +``` +// include/gunrock/framework/operators/configs.hxx:52-60 — verbatim, comments +// included, because the comments are the point. + 52 enum load_balance_t { + 53 thread_mapped, ///< 1 element per thread + 54 warp_mapped, ///< (wip) Equal # of elements per warp + 55 block_mapped, ///< Equal # of elements per block + 56 bucketing, ///< (wip) Davidson et al. (SSSP) + 57 merge_path, ///< Merrill & Garland (SpMV):: DEPRECATED (use merge_path_v2) + 58 merge_path_v2, ///< Merrill & Garland (SpMV):: CUSTOM + 59 work_stealing, ///< (wip) + 60 }; +``` + +So the strategies you can actually select are `thread_mapped`, `block_mapped`, +`merge_path` (deprecated) and `merge_path_v2` — and the runtime dispatch +confirms it, with `merge_path_v2` additionally guarded by +`#if __HIP_PLATFORM_NVIDIA__` (`advance.hxx:111-127` for the compile-time +dispatch, `advance.hxx:254-274` for `execute_runtime`, which is what +`bfs.hxx:140` calls). + +What each does, and the degree distribution that kills it: + +``` + thread_mapped thread i <- frontier element i, serial loop over its edges + good: uniform degrees dies: one hub stalls a whole warp + thread_mapped.hxx:58-81, launch box dim3_t<256> at :90 + + block_mapped equal number of elements per block + good: hubs dies: degree-1 leaves waste a block + + merge_path every thread gets the same number of EDGES, found by a + diagonal search over (segments, atoms) + good: power laws dies: pays an O(n) scan per round +``` + +The merge-path file states its own trade-off, which is exactly the arithmetic of +Step 3 turned into a design note: + +``` +// include/gunrock/framework/operators/advance/merge_path.hxx:17-29 — the +// algorithm and trade-off comment at the top of the file. + 17 * ALGORITHM: + 18 * 1. Compute prefix sum of segment sizes (compute_output_offsets) + 19 * 2. For each tile, use merge-path search to find tile boundaries + 20 * 3. Load segment offsets into shared memory + 21 * 4. Each thread uses merge-path to find its starting position + 22 * 5. Serial merge: walk the merge path processing items + 24 * TRADE-OFFS vs block_mapped: + 25 * - Requires O(n) prefix scan per iteration (vs O(n) reduce for block_mapped) + 26 * - Better load balancing for power-law graphs with hub vertices + 27 * - Worse performance for uniform-degree graphs (road networks, meshes) + 28 * - Each tile processes exactly merge_tile_size work items + 29 * - Hub vertices spanning multiple tiles are handled gracefully +``` + +`merge_tile_size = threads_per_block * items_per_thread` (`merge_path.hxx:131`), +and the tile boundaries are cached in shared memory (`:136-137`). Put the Step 3 +frontier through it: every thread gets `E / T` = 16 edges whatever the degrees +are, so the 410× load-balance penalty goes to 1× and the price is one exclusive +scan per round (Step 5). + +The paper describes the same territory with different names and one number worth +keeping: its hybrid *"set[s] a static threshold. When the frontier size is +smaller than the threshold, we use coarse-grained load-balance over nodes, +otherwise coarse-grained load-balance over edges… setting this threshold to 4096 +yields consistent high performance"* (§4.4). The paper's per-warp/per-CTA +strategy sorts neighbour lists into three size classes — larger than a CTA; +larger than a warp but smaller than a CTA; smaller than a warp — and processes +each class with its own pass, *"at the cost of higher overhead due to the +sequential processing of the three different sizes"* (§4.4). None of those class +names appear in the Essentials enum. Cite the paper for the idea and the code +for the API. + +Worth noticing while you are here: CAGRA deletes this entire problem by +construction. A fixed out-degree graph makes `thread_mapped` optimal, because +every list is the same length (`reading-cagra.md`, Step 2). Regularity bought at +build time, spent at search time. + +### Step 5 — the unknown output size, and the scan that answers it + +> **In:** a frontier of vertices with different degrees, and an output buffer +> that must be allocated before the kernel launches. +> **Out:** a per-element write offset, computed by an exclusive scan over the +> degrees — the same two-phase shape as libcudf's size/retrieve. + +Advance cannot know how many neighbours it will emit until it looks. Gunrock's +answer is not an atomic cursor: it is a prefix scan over the input frontier's +degrees, taken before the kernel runs. + +``` +// include/gunrock/framework/operators/advance/helpers.hxx:58-79 — the degree +// functor and the scan, inside compute_output_offsets. + 58 auto segment_sizes = [=] __host__ __device__(std::size_t const& i) { + 59 if (i == total_elems) // XXX: this is a weird exc. scan. + 60 return edge_t(0); + 62 auto v = graph_as_frontier ? vertex_t(i) : input_data[i]; + 63 // if item is invalid, segment size is 0. + 64 if (!gunrock::util::limits::is_valid(v)) + 65 return edge_t(0); + 66 else + 67 return G.get_number_of_neighbors(v); + 68 }; + 70 auto new_length = thrust::transform_exclusive_scan( + 71 context.execution_policy(), // execution policy + 72 thrust::make_counting_iterator(0), // input iterator: first + 73 thrust::make_counting_iterator(total_elems + + 74 1), // input iterator: last + 75 segments.begin(), // output iterator + 76 segment_sizes, // unary operation + 77 edge_t(0), // initial value + 78 thrust::plus() // binary operation + 79 ); +``` + +The scan's last element is the total output size, and element *i* is the write +base for frontier element *i* — which is precisely what `thread_mapped` uses at +`thread_mapped.hxx:76` (`segments_ptr[tid] + i`). One scan buys both the +allocation size and a contention-free write plan; no thread ever contends with +another for an output slot, because each thread's range was computed before the +kernel started. + +That same `segments` array is what merge-path binary-searches (`merge_path.hxx: +139-140`, *"segments is the exclusive prefix scan of degrees"*). This is why CSR +makes merge-path possible at all: the offsets array is already a sorted +prefix-sum of degrees, so "which vertex owns global edge *e*?" is one binary +search, computable by each thread independently with no communication. + +### Step 6 — the frontier's shape, and one dispatch per round + +> **In:** the round structure of Step 1 and a GPU with no device-wide barrier. +> **Out:** one launch per level, a host-visible convergence test, and a choice +> of frontier representation that is really a choice of push vs pull. A frontier can be a **sparse** list of vertex ids -(vector_frontier) or a **dense** bitmap with one bit per vertex -(boolmap_frontier) — exactly topic 20's SpMSpV-vs-SpMV and -direction-optimizing BFS. Small frontier → sparse/push (work -proportional to frontier size); huge frontier → dense/pull (scan -everything, but no atomics and no filter needed — the bitmap -dedupes by construction, since setting a bit twice is harmless). -Question: the switch threshold on CPU is ~|frontier| > n/20; what -changes on GPU (atomics for sparse output vs full-array scans being -nearly free at 400 GB/s)? - -### Step 6 — the host loop: one dispatch per BFS level - -There is no device-wide barrier inside a kernel launch (the wgpu -guide's point), so each BFS level is its own dispatch, and the -"is the frontier empty?" convergence test needs the frontier size -on the host — either a round-trip copy per level or **indirect -dispatch** (the GPU writes the next launch's size into a buffer the -runtime reads). Find how Gunrock decides iteration convergence. -Three consequences for our milestones: - -- The advance lambda = FalkorDB's per-edge semiring op; Gunrock is - what GraphBLAS-on-GPU compiles down to (M20). -- Advance produces a next frontier of unknown size — the cudf - guide's no-push problem again; Gunrock scans the input frontier's - degrees first (same two-phase, different name). -- The stretch-goal WGSL BFS: use boolmap frontier + level array — - dense SpMV shape, no atomics needed except the "changed" flag - (M18/M24). +(`include/gunrock/framework/frontier/vector_frontier.hxx`) or a **dense** bitmap +with one bit per vertex +(`include/gunrock/framework/frontier/experimental/boolmap_frontier.hxx`). That +is topic 20's SpMSpV-vs-SpMV distinction and direction-optimising BFS's push-vs- +pull, and Gunrock names the correspondence itself: + +``` +// include/gunrock/framework/operators/configs.hxx:78-82 + 78 enum advance_direction_t { + 79 forward, ///< Push-based approach + 80 backward, ///< Pull-based approach + 81 optimized ///< Push-pull optimized + 82 }; +``` + +Small frontier → sparse and push, work proportional to the frontier. Huge +frontier → dense and pull, which scans everything but needs no atomics and +dedupes by construction, because setting a bit twice is harmless. + +The round boundary itself is not free. There is no device-wide barrier inside a +launch (`reading-wgpu-compute.md`, Step 6), so each round is at least one +dispatch, and `is_converged` reads a host-visible emptiness flag +(`enactor.hxx:328-330`). On the lane you can actually run, that boundary costs +1544 µs (`notes.md:11-14`): a 9-level traversal spends 13.9 ms in submission +before counting an edge. On CUDA it is microseconds — but the *structure* is the +same, and it is why the stretch-goal WGSL BFS in this topic should use a boolmap +frontier and a level array (dense SpMV shape, one atomic-free dispatch per +level) rather than a sparse frontier whose size the host must learn every round. ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| include/gunrock/algorithms/bfs.hxx:95-149 | the whole BFS loop: advance + optional filter | 2, 6 | -| include/gunrock/framework/operators/advance/advance.hxx:94-123 | load-balance dispatch: thread/block/merge_path | 4 | -| operators/advance/thread_mapped.hxx | 1 thread : 1 vertex — dies on power laws | 3–4 | -| operators/advance/block_mapped.hxx | 1 block : 1 vertex's edges — dies on leaves | 4 | -| operators/advance/merge_path.hxx | binary-search work split — even by EDGE count | 4 | -| framework/frontier/vector_frontier.hxx | sparse frontier (vertex list) | 5 | -| framework/frontier/experimental/boolmap_frontier.hxx | dense frontier (bitmap) | 5 | -| include/gunrock/framework/operators/filter/ | dedupe/compact the output frontier | 2, 5 | - -Reading order: `algorithms/bfs.hxx` first (Step 2's loop, ~50 -lines), then the three load-balance strategies in -`framework/operators/advance/` side by side (Step 4 — the diff -between them IS the research), then the two frontier -representations. In the paper: §3 is the operator model (Step 2), -§4 is load balancing (Steps 3–4). +| `include/gunrock/framework/enactor.hxx:272-278, 328-330` | the run loop and the convergence test | 1, 6 | +| `include/gunrock/algorithms/bfs.hxx:105-128` | the BFS lambda: atomic **min**, not CAS (the CAS is commented out at 116-122) | 2 | +| `include/gunrock/algorithms/bfs.hxx:138-146` | advance, then optional filter | 2 | +| `include/gunrock/framework/operators/advance/thread_mapped.hxx:58-93` | 1 thread : 1 element, serial edge loop, `dim3_t<256>` launch box | 3-4 | +| `include/gunrock/framework/operators/configs.hxx:52-60` | the seven-entry `load_balance_t`, three `(wip)`, `merge_path` DEPRECATED | 4 | +| `include/gunrock/framework/operators/advance/advance.hxx:111-127` | compile-time dispatch on the strategy | 4 | +| `include/gunrock/framework/operators/advance/advance.hxx:254-274` | `execute_runtime` — what `bfs.hxx:140` actually calls | 4 | +| `include/gunrock/framework/operators/advance/merge_path.hxx:9-29` | the algorithm and its stated trade-offs | 4 | +| `include/gunrock/framework/operators/advance/merge_path.hxx:120-140` | the kernel, `merge_tile_size`, shared-memory tile offsets | 4 | +| `include/gunrock/framework/operators/advance/helpers.hxx:41-79` | `compute_output_offsets` — the degree scan | 5 | +| `include/gunrock/framework/frontier/vector_frontier.hxx` | sparse frontier (vertex list) | 6 | +| `include/gunrock/framework/frontier/experimental/boolmap_frontier.hxx` | dense frontier (bitmap) | 6 | +| `include/gunrock/framework/operators/filter/` | `bypass`, `compact`, `predicated`, `remove` — the filter menu | 2, 6 | + +Reading order: `bfs.hxx` first (the lambda and the two-call loop are the whole +programming model), then `helpers.hxx` for the scan that makes advance possible, +then `thread_mapped.hxx` and `merge_path.hxx` side by side — the diff between +those two files *is* the research. `configs.hxx` whenever a name confuses you. +In the paper: §3 is the operator model, §4.4 is load balancing, §4.5 is +idempotence and push/pull. ## Questions for notes.md -1. Advance produces the NEXT frontier with unknown size — cudf - solved this with size/retrieve; what does Gunrock use (scan the - degrees of the input frontier first — same two-phase, different - name)? -2. BFS's lambda uses CAS on parent[] — why is a LOST race benign - here (any parent is a valid BFS tree — idempotence again)? -3. Direction-optimizing BFS needs the REVERSE graph for pull. What - does that double (memory), and when is it worth it (topic 13's - CSR+CSC question resurfacing)? -4. Estimate: hub vertex, degree 10⁶, thread_mapped — how many - microseconds does one thread take at ~10 edges/cycle/SM... vs - merge_path spreading it over the whole device? -5. For M24: LDBC power-law graphs on GPU — which advance strategy - per LDBC scale factor, and does the answer change with the - frontier's hub fraction per BFS level? +1. Advance produces the next frontier with unknown size; libcudf solved the same + problem with size-then-retrieve. Name Gunrock's answer, the function it lives + in, and the one property of CSR that makes it cheaper than cudf's second + probe. +2. This guide's previous version said BFS does a CAS on `parent[]`. The pinned + code does `math::atomic::min` on `distances[]` (`bfs.hxx:125-127`). Work out + why a lost race is benign in *either* formulation — and then why the `min` + version is the one that also works for SSSP. +3. Direction-optimising BFS needs the reverse graph (CSC) for its pull phase. + What does that double, and when is it worth it? (Topic 13's CSR+CSC question, + resurfacing.) +4. Hub vertex of degree 10⁶ in a frontier of 100,000 degree-10 vertices: compute + `thread_mapped`'s makespan against merge-path's `E/T`, the way Step 3 does it + for the measured graph. Then say what the scan in Step 5 costs you per round, + and at what frontier size it stops being worth paying. +5. For M24: LDBC power-law graphs on a GPU — which advance strategy per scale + factor, and does the answer change per BFS level as the frontier's hub + fraction changes? (Note that Gunrock picks this per *call*, not per level: + `bfs.hxx:139` reads it from options once.) ## Done when -- [ ] You can express a graph algorithm as rounds of advance and filter. -- [ ] You can explain why frontier raggedness breaks warps, using this repo's own measured degree skew (max degree 6565 against a median of 11 in topic 13). -- [ ] You can name the load-balancing strategies and say which one a hub vertex of degree 10^6 demands. -- [ ] You can explain why sparse versus dense frontier representation is the same choice as push versus pull. -- [ ] You can say why a lost CAS race on `parent[]` is benign in BFS. -- [ ] You wrote answers to all five questions in notes.md. +Answer each before unfolding it. + +- [ ] You can express a graph algorithm as rounds of advance and filter, and say what the filter is allowed *not* to do. + +
Answer + + `while (!is_converged) { advance; maybe filter; }` (`enactor.hxx:274-277`, + `bfs.hxx:138-146`). Advance applies the lambda to every edge out of the + frontier and writes the neighbour — or an invalid sentinel when the lambda + returns false (`bfs.hxx:111-114`, `thread_mapped.hxx:77`). + + The filter is optional (`bfs.hxx:143`) and, in the idempotent formulation, + only *reduces* duplicates: the paper says its heuristics *"reduce, but not + eliminate, redundant entries"*, and only the non-idempotent advance + guarantees uniqueness, using atomics to do it (§4.5). + +
+ +- [ ] You can explain why frontier raggedness breaks warps, using this repo's own measured degree skew. + +
Answer + + A warp retires when its slowest lane does, and `thread_mapped`'s loop bound is + the vertex's degree (`thread_mapped.hxx:66-68`). On the topic 13 graph — 1 M + nodes, 16 M edges, p50 degree 11, max 6565 + (`topics/13-graph-engines/README.md:9-12`) — a 100,000-vertex frontier + containing the top node has ~1.6 M edge visits, i.e. 16 per thread if + balanced, against a makespan of 6565: **410×**. + + The end-to-end consequence was measured on the CPU in that same topic: 101× + slower two-hop queries from supernodes than from random nodes. + +
+ +- [ ] You can name the load-balancing strategies that are actually selectable, and say which one a degree-10⁶ hub demands. + +
Answer + + `thread_mapped`, `block_mapped`, `merge_path` (marked DEPRECATED in favour of + `merge_path_v2`) and `merge_path_v2` (NVIDIA-only, `advance.hxx:114-118`). + `warp_mapped`, `bucketing` and `work_stealing` are all `(wip)` + (`configs.hxx:52-60`). + + A 10⁶-degree hub demands merge-path: it is the only strategy that splits by + *edge* count rather than by vertex, giving every thread `num_atoms / + num_threads` work regardless of which vertex those edges belong to + (`merge_path.hxx:9-12`). The price is stated in the file: an O(n) prefix scan + per iteration, and worse performance than `block_mapped` on uniform-degree + graphs (`merge_path.hxx:24-27`). + +
+ +- [ ] You can explain what a lost race costs in the BFS lambda — using the code that is actually compiled. + +
Answer + + `auto old_distance = math::atomic::min(&distances[neighbor], iteration + 1); + return (iteration + 1 < old_distance);` (`bfs.hxx:125-127`). Two threads + reaching the same unvisited neighbour in one round: the first gets + `old_distance = INT_MAX` and returns true; the second gets `iteration + 1` and + returns false. Exactly one emission, no CAS loop, and the array converges to + the same contents under any interleaving because `min` is commutative and + idempotent. + + Benign, because at a given level every candidate parent is equally valid — any + of them yields a correct BFS tree. The CAS version is right there at + `bfs.hxx:116-122`, commented out, labelled by the author as the more + complicated way to get the same result. + +
+ +- [ ] You can say how advance sizes its output, and why that is the same problem cudf solves differently. + +
Answer + + `compute_output_offsets` runs a `thrust::transform_exclusive_scan` over the + frontier's degrees (`helpers.hxx:58-79`); element *i* is frontier element + *i*'s write base and the last element is the total. `thread_mapped` writes at + `segments_ptr[tid] + i` (`:76`), so no thread ever contends for a slot. + + cudf faces the identical constraint (no `push`, pre-sized buffers) and answers + with a counting pass through cuco plus a retrieve pass + (`reading-libcudf.md`, Step 2). Gunrock can be cheaper because a vertex's + output count is its degree, which CSR already stores — no probe required. + +
+ +- [ ] You can explain why sparse-vs-dense frontier is the same choice as push-vs-pull, and what a round boundary costs. + +
Answer + + Sparse (`vector_frontier.hxx`) means work proportional to the frontier and + atomics on output — push. Dense (`boolmap_frontier.hxx`) means scanning every + vertex but no atomics and free deduplication — pull. Gunrock names them + directly: `forward` = push, `backward` = pull, `optimized` = both + (`configs.hxx:78-82`). + + A round is at least one dispatch, because there is no device-wide barrier, and + convergence is a host-visible emptiness test (`enactor.hxx:328-330`). On this + repo's runnable lane that boundary is 1544 µs (`notes.md:11-14`) — 13.9 ms for + a 9-level traversal before a single edge is examined. + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the hub arithmetic. + +
Answer + + The slots are `notes.md:79-85`. Question 4 wants the arithmetic done, not + described: `E / T` against `max degree`, for the numbers in the question. + +
## References **Papers** -- Wang, Davidson, Pan, Wu, Riffel, Owens — "Gunrock: A - High-Performance Graph Processing Library on the GPU" (PPoPP 2016, - [arXiv:1501.05387](https://arxiv.org/abs/1501.05387)) — §3 the - operator model, §4 load balancing + +- Yangzihao Wang, Andrew Davidson, Yuechao Pan, Yuduo Wu, Andy Riffel, John D. + Owens — *"Gunrock: A High-Performance Graph Processing Library on the GPU"*, + PPoPP 2016, [arXiv:1501.05387](https://arxiv.org/abs/1501.05387). §3 for the + operator model, §4.4 for the load-balancing strategies and the 4096 threshold, + §4.5 for idempotent vs non-idempotent advance and push vs pull. Read it after + the code, not before: the names in §4.4 are not the names in `configs.hxx`. **Code** -- [gunrock](https://github.com/gunrock/gunrock) — the modern - "Essentials" rewrite under `include/gunrock/` — read - `algorithms/bfs.hxx` first, then the three load-balance strategies - in `framework/operators/advance/` + +- [gunrock](https://github.com/gunrock/gunrock) @ `748f79e` — the "Essentials" + rewrite under `include/gunrock/`. Route: `algorithms/bfs.hxx` → + `framework/operators/advance/helpers.hxx` → `thread_mapped.hxx` and + `merge_path.hxx` → `framework/operators/configs.hxx` → + `framework/frontier/`. + +**Measurements in this repo** + +- `topics/13-graph-engines/README.md:9-19` — the degree distribution and the + 101× supernode penalty Step 3 computes with. +- `topics/18-gpu/notes.md:11-14` — the 1544 µs per-dispatch floor Step 6 uses. diff --git a/topics/18-gpu/reading-libcudf.md b/topics/18-gpu/reading-libcudf.md index 923ea0a..dc7df18 100644 --- a/topics/18-gpu/reading-libcudf.md +++ b/topics/18-gpu/reading-libcudf.md @@ -1,206 +1,553 @@ # libcudf: GPU kernels can't push -RAPIDS' GPU DataFrame engine — Arrow-layout columns (topic 12) with -every operator rewritten under GPU constraints: no resizable output, -atomics that must be amortized, and a memory hierarchy you manage by -hand. This chapter builds those constraints one at a time — why a -GPU kernel can't `push`, the two-phase pattern that replaces it, how -a warp probes a hash table together, and where aggregation spills — -then maps each idea to the exact `.cu` file that implements it. The -two-phase size/retrieve pattern and cooperative-group probing here -are the idioms every GPU-DB operator ends up using. +RAPIDS' GPU DataFrame engine: Arrow-layout columns (topic 12) with every +operator rewritten under GPU constraints — no resizable output, atomics that +must be amortised, and a memory hierarchy you manage by hand. This chapter +builds those constraints one at a time and then points at the exact file that +implements each. + +Read it as architecture, not as a lab. There is no NVIDIA device on this +machine, so nothing here can be built or run; every claim below is a claim about +source you can read, and it is anchored to a line you can check. That is also +why the anchors matter more than usual — a wrong `file:line` in a guide you +cannot compile is a wrong belief that never gets corrected. + +Every anchor is [rapidsai/cudf@2f082a7](https://github.com/rapidsai/cudf), the +revision pinned in `resources/codebases.md`. Verify with +`python3 tools/pinned-source.py show cudf -r A:B`. cudf's hash tables are +all instances of **cuco** (CUDA Collections, NVIDIA's GPU hash-table library), +which is *not* pinned here — so where cuco's own semantics matter this guide +quotes cudf's usage and cudf's comments, and says when the two disagree. ## The problem in one sentence -A join's output size is unknown until you compute it, but a GPU -kernel's output buffer must be allocated *before* launch and shared -by ~100,000 threads with no `Vec::push` and no cheap lock — so every -variable-output operator needs a plan for where each thread writes. +A join's output size is unknown until you compute it, but a GPU kernel's output +buffer must be allocated *before* launch and is shared by ~100,000 threads with +no `Vec::push` and no cheap lock — so every variable-output operator needs an +explicit plan for where each thread writes. ## The concepts, step by step -### Step 1 — the executor: warps, coalescing, and the no-push rule - -A GPU runs kernels (device functions) over tens of thousands of -threads grouped in **warps** — bundles of 32 threads executing in -lockstep — and a warp's 32 loads become **one** memory transaction -iff adjacent threads touch adjacent addresses (**coalescing**). -Each thread block also owns ~100 KB of **shared memory** (a -software-managed scratchpad as fast as L1). Three consequences -shape every cudf operator: - -- **Output must be pre-sized.** There is no allocator you'd want to - call from 100K threads mid-kernel, and no `Vec::push`: every - output needs its size known up front or an atomic cursor. -- **Atomics must be amortized.** A global atomic per element from - 100K threads serializes on the contended word; the idiom is - reduce-locally-first, one atomic per block. -- **Layout is destiny.** A row-store strands 31/32 of every warp - transaction; dense columns coalesce by construction (Step 5). - -Why it matters: Steps 2–4 are the three standard escapes from these -constraints, and they recur in every GPU database ever written. - -### Step 2 — two-phase everything: size, then retrieve - -When output size is unknown, run the kernel **twice**: pass 1 -computes only *how many* results each thread produces, a prefix scan -(running total of the per-thread counts — each thread's total-before -is exactly its write offset) turns counts into exact write -positions, the host allocates exactly, and pass 2 re-runs the same -probe and writes through the computed offsets: +### Step 1 — the executor, and the three rules it imposes + +> **In:** a kernel launched over a grid of blocks. +> **Out:** three constraints that decide the shape of every operator below — +> pre-sized output, amortised atomics, coalesced layout. + +Vocabulary. A **kernel** is a device function launched over a grid; a **block** +(CTA) is a group of threads scheduled onto one SM, sharing a scratchpad and able +to synchronise; a **warp** is the 32 threads inside a block that issue together. +cudf fixes its block sizes as constants you can read: `DEFAULT_JOIN_BLOCK_SIZE += 128` (`cpp/src/join/join_common_utils.hpp:21`) and `GROUPBY_BLOCK_SIZE = 128` +(`cpp/src/groupby/hash/helpers.cuh:25`). Four warps per block, everywhere. + +The three rules: + +- **Output must be pre-sized.** There is no allocator you would want to call + from 100 K threads mid-kernel and no `push`. Either the size is known before + launch (Step 2) or a device-scope cursor hands out ranges (Step 4). +- **Atomics must be amortised.** One global atomic per element serialises every + thread on one cache line. The idiom is to reduce locally first and take one + atomic per warp or per block (Step 4). +- **Layout is destiny.** A warp's 32 loads coalesce into one transaction only if + the addresses are adjacent; the read granularity is 128 B on a GPU against + 64 B on a CPU (Crystal §4.3), so a row-store wastes most of every fetch + (Step 6). + +### Step 2 — size, then retrieve — and the pass you can skip + +> **In:** a build-side hash table and a probe table of `left_table_num_rows` +> rows. +> **Out:** an exact output size, then two `device_uvector`s of exactly that +> length — at the cost of probing twice, unless the caller already knows the +> size. + +The two entry points are one-function files. `inner_join_size.cu` is 20 lines +and `inner_join_retrieve.cu` is 28; both just instantiate templates from +`size_impl.cuh` / `retrieve_impl.cuh`. The counting pass is not a hand-written +kernel — it is cuco's own `count`: + +```cpp +// cpp/src/join/hash_join/size_impl.cuh:52-61 — the whole size pass, inside +// dispatch_join_comparator's lambda. `hash_table` is the cuco::static_multiset. + 52 [&](auto equality, auto d_hasher) { + 53 auto const iter = cudf::detail::make_counting_transform_iterator(0, pair_fn{d_hasher}); + 54 if constexpr (Join == join_kind::LEFT_JOIN) { + 55 return hash_table.count_outer( + 56 iter, iter + left_table_num_rows, equality, hash_table.hash_function(), stream.value()); + 57 } else { + 58 return hash_table.count( + 59 iter, iter + left_table_num_rows, equality, hash_table.hash_function(), stream.value()); + 60 } + 61 }); +``` + +Note what is *not* there: no per-thread `count[]` array, no exclusive prefix scan +over it, no second kernel to turn counts into offsets. Older GPU joins (and this +guide's previous version) described exactly that three-step shape; at this pin it +lives inside cuco, and cudf only asks for a total. + +The retrieve pass then makes the size pass optional: + +```cpp +// cpp/src/join/hash_join/retrieve_impl.cuh:49-58 and 71-85, elided in the middle +// (the zero-size early return and the two allocations). + 49 std::size_t const join_size = output_size + 50 ? *output_size + 51 : compute_join_output_size(right_table, + 52 left_table, + ... + 71 auto const out_probe_begin = + 72 thrust::make_transform_output_iterator(left_indices->begin(), output_fn{}); + 73 auto const out_build_begin = + 74 thrust::make_transform_output_iterator(right_indices->begin(), output_fn{}); + 76 auto retrieve_results = [&](auto equality, auto d_hasher) { + 78 if constexpr (Join == join_kind::INNER_JOIN) { + 79 hash_table.retrieve(iter, + 80 iter + left_table_num_rows, + 81 equality, + 82 hash_table.hash_function(), + 83 out_probe_begin, + 84 out_build_begin, + 85 stream.value()); +``` + +So the double probe is a *default*, not a law: pass `output_size` and the count +disappears. That is the whole reason `hash_join::inner_join_size` is public API +— a caller who joins the same tables repeatedly, or who has a cardinality +estimate it is willing to be wrong about, pays for one probe instead of two. The +`thrust::transform_output_iterator` pair is the other half of the trick: cuco +writes pairs, and the iterator splits them into two columns as they land, so no +intermediate array of pairs is ever materialised. + +Now the arithmetic that makes this concrete, using the only dispatch floor this +repo has measured. One `inner_join` costs at least three device operations: +build (`insert_async`), count, retrieve. On a real CUDA device a launch costs a +few microseconds, so this is nothing — but our runnable lane's floor is +**1544 µs per dispatch** (`notes.md:11-14`), and the M18 question is what this +pattern would cost if you ported it to wgpu: ``` - pass 1 (size): each thread COUNTS its matches → total via reduce - allocate exactly total - pass 2 (retrieve): same probe again, write via computed offsets + 3 dispatches x 1544 us = 4632 us of pure submission + to keep that under 10% overhead: total work >= 46.3 ms + at the 12.5 GB/s our device achieved on a streaming kernel + (notes.md:16, floor subtracted): 46.3e-3 s x 12.5e9 B/s = 579 MB + at 8 B per probe row (4 B key + 4 B index): ~72 M rows ``` -`inner_join_size.cu` and `inner_join_retrieve.cu` are literally the -same probe loop with different epilogues: - -```rust -// pass 1: the probe loop with a COUNTING epilogue -par_for i in 0..n_probe { - count[thread_id] += table.matches(keys[i]); -} -let offsets = exclusive_scan(count); // per-thread write positions -let out = alloc_exact(offsets.total()); // GPU output must be pre-sized - -// pass 2: the SAME probe loop with a WRITING epilogue -par_for i in 0..n_probe { - for m in table.probe(keys[i]) { // recompute beats remembering: - out[offsets[thread_id]] = (i, m); // HBM traffic to materialize - offsets[thread_id] += 1; // match lists costs more than - } // probing the table twice -} +Seventy-two million rows before the join's *plumbing* falls below a tenth of its +runtime. That number is the honest reason M18 offloads dense distance scoring +and not joins. + +### Step 3 — how a probe is actually parallelised (and by how many threads) + +> **In:** one probe key and a hash table in global memory. +> **Out:** the number of threads that cooperate on it, and the number of slots +> they touch per step — both of which are template parameters you can read. + +Two numbers govern a cuco probe: the **cooperative-group size** (how many +threads work one key together) and the **bucket/storage size** (how many slots +sit contiguously and are examined per probing step). cudf sets them per table +type, and the values are small: + +| table | probing scheme | storage | source | +|---|---|---|---| +| hash join (`static_multiset`) | `double_hashing` = **2** | `cuco::storage<2>` | `cpp/src/join/hash_join/hash_join_impl.cuh:50-57`, `cpp/include/cudf/detail/join/join.hpp:12` | +| distinct join (`static_set`) | `linear_probing<1, hasher>` | `cuco::storage<1>` | `cpp/include/cudf/detail/join/distinct_hash_join.cuh:147-157` | +| group-by set | `GROUPBY_CG_SIZE = 1` | `GROUPBY_BUCKET_SIZE = 1` | `cpp/src/groupby/hash/helpers.cuh:19,22` | +| filtered join, primitive rows | `linear_probing<1, …>` | `bucket_storage` | `cpp/include/cudf/detail/join/filtered_join.cuh:165-175` | +| filtered join, nested rows | `linear_probing<4, …>` | as above | `filtered_join.cuh:182-183` | + +Read that table before you repeat the folklore. There is no 4-to-8-thread +cooperative window anywhere in cudf's join path at this pin: the equi-join +probes with **two** threads and a two-slot bucket, and the distinct join — +the newer, faster path — probes with **one**. The only `4` is +`nested_probing_scheme`, for rows with nested columns, whose comparisons are +expensive enough to be worth spreading. + +A naming trap worth knowing, because it will otherwise cost you an hour: cudf's +own comments call `linear_probing`'s first template parameter *"bucket size"* +(`filtered_join.cuh:174, 182, 184`) while `join.hpp:12` names the same position +`DEFAULT_JOIN_CG_SIZE`. cuco is not pinned in this repo, so this guide does not +adjudicate; what it can tell you is what the values are and where they come +from, and that the two files disagree about what to call them. + +The arithmetic for why the numbers are small anyway: + +``` + key stored by the hash join = cuco::pair + = 4 B + 4 B = 8 B + GPU global-memory read granularity = 128 B (Crystal §4.3) + slots delivered by one transaction = 16 + slots a storage<2> bucket examines = 2 + --- + useful fraction of a random probe's fetch = 16 B / 128 B = 12.5 % ``` -The cost is doubled probe work; the payoff is zero contention and -zero over-allocation. Alternatives they could have used and didn't: -atomic global cursor (contended), max-size over-allocation -(memory). Question: pass 2 recomputes all of pass 1's probes — why -is recompute cheaper than remembering (HBM bandwidth vs -materializing per-thread match lists)? Compare simdjson's -over-write-under-advance: same problem, opposite answer — why? +A wider bucket would use more of each fetched line — and would also make every +probe step compare more keys it does not want. That trade is the tuning knob the +question below asks you to find, and it is the same one hashbrown makes when it +picks a SIMD group width. + +### Step 4 — amortising atomics: one `fetch_add` per warp, not per row + +> **In:** a kernel whose threads each produce an unpredictable number of output +> rows. +> **Out:** a contiguous output range per warp, claimed with a single +> device-scope atomic — the pattern to copy when you cannot pre-size. + +The clean example of "no push" solved without a size pass is the *conditional* +join, where output size cannot be counted cheaply. Its host side allocates from +either a caller-supplied size or a counting kernel, then creates one device-wide +cursor: + +```cpp +// cpp/src/join/conditional_join.cu:74-97 — the size decision and the cursor, +// with the has_nulls branch of the launch elided. + 74 std::size_t join_size; + 75 if (output_size.has_value()) { + 76 join_size = *output_size; + 77 } else { + 79 cudf::detail::device_scalar size(0, stream, mr); + 86 compute_conditional_join_output_size + 87 <<>>( + 88 *left_table, *right_table, join_type, parser.device_expression_data, false, size.data()); + 91 join_size = size.value(stream); + 92 } + 94 cudf::detail::device_scalar write_index( + 95 0, stream, cudf::get_current_device_resource_ref()); + 97 auto left_indices = std::make_unique>(join_size, stream, mr); +``` + +The kernel then buffers matches in shared memory with **block-scope** atomics — +cheap, because they never leave the SM: + +```cpp +// cpp/src/join/conditional_join_kernels.cuh:41-45 — add_pair_to_cache, the +// per-warp staging step. + 41 cuda::atomic_ref ref{*(current_idx_shared + warp_id)}; + 42 std::size_t my_current_idx = ref.fetch_add(1, cuda::memory_order_relaxed); + 43 // It's guaranteed to fit into the shared cache + 44 joined_shared_l[my_current_idx] = first; + 45 joined_shared_r[my_current_idx] = second; +``` + +and flushes with exactly one **device-scope** atomic per warp, broadcast to the +other lanes by a shuffle: + +```cpp +// cpp/src/join/conditional_join_kernels.cuh:74-93 — flush_output_cache, elided +// between the shuffle and the copy loop. + 74 if (0 == lane_id) { + 75 cuda::atomic_ref ref{*current_idx}; + 76 output_offset = ref.fetch_add(current_idx_shared[warp_id], cuda::memory_order_relaxed); + 77 } + 84 output_offset = cub::ShuffleIndex(output_offset, 0, activemask); + 86 for (std::size_t shared_out_idx = static_cast(lane_id); + 87 shared_out_idx < current_idx_shared[warp_id]; + 88 shared_out_idx += num_threads) { + 89 std::size_t thread_offset = output_offset + shared_out_idx; + 90 if (thread_offset < max_size) { + 91 join_output_l[thread_offset] = join_shared_l[warp_id][shared_out_idx]; + 92 join_output_r[thread_offset] = join_shared_r[warp_id][shared_out_idx]; +``` + +Count the atomics. Per output row, the naive kernel takes one device-scope +`fetch_add`; this one takes one *block*-scope `fetch_add` (staying in the SM) +plus one device-scope `fetch_add` per warp-flush. If a warp stages `C` rows per +flush, device-scope traffic drops by a factor of `C`. It is the same rule as +Crystal's one-atomic-per-tile (`reading-crystal-sigmod20.md`, Step 6) and the +same rule the `filter_count` stub asks you to implement in WGSL +(`experiments/src/gpu.rs:150-153`) — three engines, one arithmetic. + +### Step 5 — group-by: two tiers, and a fallback that restarts everything -### Step 3 — cooperative-groups probing: the warp is the vector register +> **In:** N rows and an unknown number of distinct keys. +> **Out:** either a shared-memory aggregation, or — if any single block sees too +> many distinct keys — the *entire* input re-aggregated through global memory. -A hash-table probe chases a random slot, then maybe the next slot, -and so on — one uncoalesced load per step if each thread probes -alone. cudf (via **cuco**, RAPIDS' GPU hash-table library) probes -with a **cooperative group**: a warp fragment of 4–8 threads loads a -whole window of adjacent buckets in one coalesced transaction, votes -on matches with a **ballot** (a warp instruction producing a bitmask -of which lanes matched), and advances together: +The shared-memory tier is sized by four constants and one multiplication +(`cpp/src/groupby/hash/helpers.cuh:19-46`): ``` - thread-per-probe: t0→slot17, t1→slot93, t2→slot4 (3 transactions) - group-per-probe: t0..t3 → slots 17,18,19,20 (1 transaction, - ballot → who matched) hashbrown Group - at warp scale!) + GROUPBY_BLOCK_SIZE = 128 threads per block (:25) + GROUPBY_CARDINALITY_THRESHOLD = 128 distinct keys a block may hold (:29) + GROUPBY_SHM_MAX_ELEMENTS = 128 + 128 = 256 (:39-40) + (threshold + block_size: after crossing the threshold every thread + in the block can still land one more insert) + shmem_extent_t = 256 x 1.43 = 366 slots (:44-46) + load factor at the cap: 256 / 366 = 0.70 ← the comment's "0.7 occupancy" + GROUPBY_CG_SIZE = 1, GROUPBY_BUCKET_SIZE = 1 (:19,22) ``` -This is EXACTLY topic 17's SwissTable `match_tag` — 16 control -bytes per `vceq` — with the warp playing the vector register. -Question: hashbrown shrank its NEON group to 8B; what's the -analogous tuning knob in cuco (window size vs probe length)? - -### Step 4 — group-by: aggregate in shared memory until it spills - -Aggregation wants one accumulator per group, updated by every -thread — a contention magnet. cudf's answer is two tiers: -`compute_shared_memory_aggs.cu` sizes per-block scratch for the -output columns and aggregates there (fast, block-local atomics), -then merges blocks' partials; when groups × columns don't fit in -the ~100 KB shared-memory budget (~a few hundred groups), it BAILS -to `compute_global_memory_aggs.cu` — atomics straight into global -memory. Two levels of the same aggregation = topic 11's -partial/final split, imposed by the memory hierarchy instead of by -threads. Question: high-cardinality group-by (1M groups) — neither -tier fits. What's the classical answer (partition by group hash -first — topic 13's radix partition, now for occupancy)? - -### Step 5 — Arrow layout: coalescing and null-handling by construction - -cudf columns are Arrow-format: a dense value array plus a -**validity bitmap** (one bit per row marking null/not-null). Dense -arrays mean warp loads coalesce by construction — a row-store on -GPU would strand 31/32 of every 128-byte transaction, topic 12's -layout argument with a 32× multiplier. Nulls process as bitmask -kernels (`src/bitmask/`), not per-row branches — branches diverge -warps; bit-ops don't. Question: strings. Arrow offsets+bytes means -variable work per element — find how cudf balances it -(warp-per-string vs thread-per-char kernels in src/strings/) and -relate to Gunrock's ragged-frontier problem. - -### Step 6 — when the pattern doesn't fit: conditional joins and JIT'd predicates - -Non-equi joins (`a.x < b.y`) can't hash — there is no key to hash -on — so `conditional_join.cu` falls back to a nested loop with the -predicate shipped to the device as an AST it interprets per pair -(same reason topic 10's planner keeps NL join). And because -interpreting an AST per pair is this topic's cardinal sin, -`src/join/jit/` JIT-compiles the predicate into the kernel at -runtime — a preview of topic 19: shader/kernel specialization *is* -query compilation. +A first kernel, `mapping_indices_kernel`, holds that 366-slot cuco set in +`__shared__` storage and maps each row to a local slot +(`cpp/src/groupby/hash/compute_mapping_indices.cuh:101-113`). If a block's +distinct-key count crosses the threshold it stops using the shared set and sets a +flag. The aggregation kernels then split by tier: `compute_shared_memory_aggs.cu` +for blocks under the threshold, `compute_global_memory_aggs.cu` for the rest, +with the shared tier's dynamic allocation coming from +`cudaOccupancyAvailableDynamicSMemPerBlock` — asked for the grid's actual blocks +per SM — and then **halved**, `0.5 * dynamic_shmem_size`, before being rounded +down to `ALIGNMENT` (`compute_shared_memory_aggs.cu:268-277`). cudf asks the +occupancy API what is available and then takes half of the answer; the margin is +not explained in the code, and guessing at its reason is the kind of thing this +guide is trying to stop you doing. + +The part that is usually described wrongly — including by this guide's previous +version — is what happens when the flag is set. It is not a per-block spill: + +```cpp +// cpp/src/groupby/hash/compute_single_pass_aggs.cuh:111-122 — the host reads +// the flag back and, if any block set it, discards the shared-memory plan. + 111 auto const needs_fallback = [&] { + 112 cuda::std::atomic_flag h_needs_fallback; + 115 CUDF_CUDA_TRY(cudf::detail::memcpy_async(&h_needs_fallback, + 116 needs_global_memory_fallback.data(), + 117 sizeof(cuda::std::atomic_flag), + 118 stream)); + 119 stream.synchronize(); + 120 return h_needs_fallback.test(cuda::std::memory_order_relaxed); + 121 }(); + 122 if (needs_fallback) { return run_aggs_by_global_mem_kernel(); } +``` + +One block over 128 distinct keys makes the *whole* aggregation re-run in global +memory, after a host round trip and a `stream.synchronize()`. The decision is +all-or-nothing and it is taken on the host. For a high-cardinality group-by the +consequence is that the shared-memory pass is pure loss, which is why the +classical answer — partition by key hash first, so each partition's cardinality +fits — is what the question below asks you to work out. That is topic 13's radix +partitioning, applied to occupancy instead of to cache. + +### Step 6 — layout, strings, and the one thing that is JIT-compiled + +> **In:** an Arrow column. +> **Out:** coalesced loads for free, null handling as bit operations rather than +> branches, and — for one specific operator — a kernel compiled at runtime. + +A `column_view` is Arrow by contract: *"Because column_view is non-owning, and +its data layout conforms to the Arrow Physical Memory Layout specification…"* +(`cpp/include/cudf/column/column_view.hpp:33-35`), which is a dense value buffer +plus an optional **validity bitmap** (one bit per row). Two GPU consequences: a +warp reading `col[i]` for adjacent `i` coalesces by construction, and nulls are +processed with bit operations rather than per-row branches — branches diverge +warps (`reading-crystal-sigmod20.md`, Step 1), bit twiddling does not. Slicing is +free because the offset lives in the view, not in the data +(`column_view.hpp:40-42`). + +Strings are where the regularity ends: a `strings_column_view` is an offsets +child at index 0 plus a character buffer (`cpp/include/cudf/strings/ +strings_column_view.hpp:53, 73-113`), so work per element is *variable* — the +same ragged-frontier problem Gunrock solves with load-balancing strategies +(`reading-gunrock.md`, Step 4). + +Two anchor corrections worth carrying, because the plausible-sounding versions +are wrong: + +- Validity-bitmap kernels are **not** a directory of many files. `cpp/src/ + bitmask/` contains exactly two: `null_mask.cu` and `is_element_valid.cpp`. +- cudf does **not** JIT-compile join predicates in general. + `cpp/src/join/jit/` holds two files, `filter_join_kernel.cu` and + `filter_join_kernel.cuh`, used by the filter-join path + (`cpp/src/join/filter_join_indices/filter_join_indices_jit.cu:6-7`, which pulls + in `jit/cache.hpp`, `jit/parser.hpp`, `jit/row_ir.hpp`). The *conditional* join + — the non-equi nested loop — does not JIT at all: it parses the predicate into + an AST on the host and ships the parsed form to the device + (`cpp/src/join/conditional_join.cu:61-62`, + `ast::detail::expression_parser{binary_predicate, left, right, …}`), then + interprets it per pair, sizing shared memory from + `parser.shmem_per_thread × threads_per_block` (`conditional_join.cu:70`). + +Both halves of that are the topic 19 preview: an interpreted AST per pair is +this topic's cardinal sin, and the JIT path is what removing it looks like — for +one operator, so far. ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| src/join/hash_join/ | size/retrieve split: `inner_join_size.cu` THEN `inner_join_retrieve.cu` | 2 | -| src/join/hash_join/kernels_common.cuh | the probe kernel shapes | 2–3 | -| src/join/distinct_hash_join.cu | cuco-based build + cooperative-groups probe | 3 | -| src/groupby/hash/compute_shared_memory_aggs.cu | per-block shared-mem aggregation + spill test | 4 | -| src/groupby/hash/compute_global_memory_aggs.cu | the global-atomics fallback | 4 | -| src/groupby/hash/compute_mapping_indices.cu | key → group index pass | 4 | -| src/bitmask/ | validity bitmaps as first-class kernels (topic 11's null masks) | 5 | -| src/join/conditional_join.cu | non-equi joins: nested loop, AST predicate on device | 6 | -| src/join/jit/ | JIT'd join predicates (topic 19 preview) | 6 | - -Reading order: the size/retrieve pair first (diff the two files — -the epilogues are the whole difference), then -`distinct_hash_join.cu` for the cooperative-group probe, then the -`groupby/hash/` trio, and `bitmask/`/`conditional_join.cu` as the -questions demand. +| `cpp/src/join/join_common_utils.hpp:21` | `DEFAULT_JOIN_BLOCK_SIZE = 128` | 1 | +| `cpp/src/join/hash_join/size_impl.cuh:52-61` | the size pass = one cuco `count` | 2 | +| `cpp/src/join/hash_join/retrieve_impl.cuh:49-58, 71-85` | the retrieve pass, and the size pass being optional | 2 | +| `cpp/src/join/hash_join/inner_join_size.cu` / `inner_join_retrieve.cu` | 20 and 28 lines of instantiation — read them to see how little is here | 2 | +| `cpp/src/join/hash_join/hash_join_impl.cuh:50-57` | the multiset type: `double_hashing<2>` + `storage<2>` | 3 | +| `cpp/include/cudf/detail/join/join.hpp:12` | `DEFAULT_JOIN_CG_SIZE = 2` | 3 | +| `cpp/include/cudf/detail/join/distinct_hash_join.cuh:147-157` | the distinct path: `linear_probing<1>` + `storage<1>` | 3 | +| `cpp/include/cudf/detail/join/filtered_join.cuh:165-185` | bucket 1 for primitive rows, 4 for nested | 3 | +| `cpp/src/join/conditional_join_kernels.cuh:34-95` | shared-memory staging + one device atomic per warp | 4 | +| `cpp/src/join/conditional_join.cu:61-97` | AST parse, optional size kernel, `write_index` cursor | 4, 6 | +| `cpp/src/groupby/hash/helpers.cuh:19-46` | every group-by constant in Step 5 | 5 | +| `cpp/src/groupby/hash/compute_mapping_indices.cuh:50, 85-113` | the shared cuco set and the bail-out | 5 | +| `cpp/src/groupby/hash/compute_single_pass_aggs.cuh:111-122` | the host-side all-or-nothing fallback | 5 | +| `cpp/src/groupby/hash/compute_shared_memory_aggs.cu:268-277` | the shared-memory budget: occupancy API result, halved | 5 | +| `cpp/include/cudf/column/column_view.hpp:33-42` | Arrow conformance and the zero-copy slice offset | 6 | +| `cpp/src/bitmask/null_mask.cu` | the validity-bitmask kernels (there are only two files) | 6 | +| `cpp/src/join/jit/filter_join_kernel.cuh` | the one JIT'd join path | 6 | + +Reading order: `inner_join_size.cu` and `inner_join_retrieve.cu` first — they are +tiny, and their emptiness is the lesson — then their two `_impl.cuh` headers, +then `hash_join_impl.cuh` and `distinct_hash_join.cuh` side by side for Step 3's +table. Then `conditional_join_kernels.cuh` end to end, which is the most +instructive single file here. `groupby/hash/` last, starting from `helpers.cuh`. ## Questions for notes.md -1. Count kernel launches for one `inner_join`: build + size + - retrieve (+ mapping). At ~1.5 ms dispatch overhead each (our - measured floor on Metal), what's the minimum batch that - amortizes four launches? -2. The size/retrieve recompute doubles probe FLOPs. On the Crystal - roofline, when is that free (probe is bandwidth-bound; second - pass hits the same cache lines... does HBM have a "cache" that - helps — L2)? -3. Why does conditional_join fall back to nested-loop + device AST - instead of hashing (non-equi predicates can't hash — same reason - topic 10's planner keeps NL join)? -4. cudf JIT-compiles join predicates (src/join/jit/) at runtime. - What's the WGSL analogue for our engine (naga compiles WGSL - strings at pipeline creation — shader specialization = topic 19's - query compilation)? -5. For M18: our filter_count stub's one-atomic-per-workgroup is - pass-1-only of the cudf pattern. Sketch the pass-2 (compact - values, not count) using a workgroup prefix scan — Crystal's - BlockScan. +1. Count the device operations in one `inner_join`: build, size, retrieve (and + the mapping pass if you go through the group-by path). At the 1544 µs floor + this repo measured on Metal, what is the minimum probe-side row count that + amortises them to under 10 %? Step 2 does the arithmetic for three + dispatches — redo it for your count, and say what the same number would be on + a CUDA device where a launch costs ~5 µs. +2. The size/retrieve pair probes twice. On Crystal's roofline, when is the second + probe nearly free? (What does the V100's 6 MB L2 do for a hash table of + 1-4 MB — and what does §4.3's 14.5× regime tell you about that?) Then: what + does `output_size` being an `optional` let a caller do about it? +3. Why does `conditional_join` interpret a device AST rather than hash, and why + does the *filter* join get a JIT path when the conditional join does not? +4. cudf JIT-compiles that one kernel at runtime. What is the WGSL analogue for + our engine — where in the wgpu ladder does "compile a shader specialised to + this predicate" happen, and what does it cost per distinct predicate? +5. For M18: our `filter_count` stub's one-atomic-per-workgroup is only pass 1 of + this pattern. Sketch pass 2 (compact the values, not just count them) using a + workgroup prefix scan — and say which of Step 4's two atomic scopes WGSL can + express. ## Done when -- [ ] You can explain the no-push rule and why it forces two-phase (size, then retrieve) kernels. -- [ ] You can count the kernel launches in one `inner_join` and say which of them recomputes work. -- [ ] You can explain cooperative-groups probing: the warp as the vector register. -- [ ] You can say why group-by aggregates in shared memory until it spills, and what the spill costs. -- [ ] You can explain why conditional joins fall back to nested loops with a device AST. -- [ ] You wrote answers to all five questions in notes.md, including what is wrong with one atomic per workgroup in the `filter_count` stub. +Answer each before unfolding it. + +- [ ] You can explain the no-push rule and say precisely which pass cudf runs to get around it — including when it does not run. + +
Answer + + Output buffers are allocated before launch, so the size must be known. cudf + gets it from cuco's `count` / `count_outer` (`size_impl.cuh:52-61`) and then + calls `retrieve` into a pair of `thrust::transform_output_iterator`s + (`retrieve_impl.cuh:71-85`). + + But the count is skipped whenever the caller passes `output_size` + (`retrieve_impl.cuh:49-58`) — which is why `hash_join::inner_join_size` is + public. There is no hand-rolled per-thread count array or prefix scan at this + pin; that shape is what cuco does internally. + +
+ +- [ ] You can state how many threads cooperate on one equi-join probe, and how many slots they examine per step. + +
Answer + + Two and two: `cuco::double_hashing` with + `DEFAULT_JOIN_CG_SIZE = 2` (`join.hpp:12`) and `cuco::storage<2>` + (`hash_join_impl.cuh:50-57`). + + The `distinct_hash_join` path uses `linear_probing<1>` and `storage<1>` + (`distinct_hash_join.cuh:147-157`), and group-by uses 1 and 1 + (`helpers.cuh:19,22`). The only 4 in the join code is + `nested_probing_scheme` for nested-typed rows (`filtered_join.cuh:183`). + "A cooperative group of 4-8 threads" is not what this code does. + +
+ +- [ ] You can explain how `conditional_join` avoids one device atomic per output row, and quantify the saving. + +
Answer + + Each warp stages its matches into a shared-memory cache using a **block**-scope + `atomic_ref::fetch_add` (`conditional_join_kernels.cuh:41-45`), which never + leaves the SM. On flush, lane 0 alone does a **device**-scope `fetch_add` of + the whole staged count (`:74-77`) and shuffles the returned base to the other + lanes (`:84`), which then write their rows at `base + i` (`:86-93`). + + Device-scope atomics therefore drop by the number of rows staged per flush. + Same rule as Crystal's one-atomic-per-tile and the WGSL stub's + one-`atomicAdd`-per-workgroup. + +
+ +- [ ] You can say what happens when a group-by block exceeds its cardinality threshold — and what it does *not* do. + +
Answer + + A block that passes `GROUPBY_CARDINALITY_THRESHOLD = 128` distinct keys stops + using its shared 366-slot cuco set and sets a device flag + (`compute_mapping_indices.cuh:50`). The host copies that flag back, + synchronises, and — if it is set — throws the shared-memory plan away and + re-runs the *entire* aggregation with the global-memory kernel + (`compute_single_pass_aggs.cuh:111-122`). + + It does not spill that block only, and it does not fall back per key. One bad + block costs everyone, which is exactly why a high-cardinality group-by wants + hash partitioning first. + +
+ +- [ ] You can name what is actually JIT-compiled in cudf's join code, and what the conditional join does instead. + +
Answer + + JIT: the **filter** join only — `cpp/src/join/jit/filter_join_kernel.{cu,cuh}`, + driven from `filter_join_indices/filter_join_indices_jit.cu`, which pulls in + `jit/cache.hpp`, `jit/parser.hpp` and `jit/row_ir.hpp`. + + The conditional (non-equi) join parses its predicate into an AST on the host + (`conditional_join.cu:61-62`) and interprets that AST per candidate pair on the + device, sizing shared memory from `parser.shmem_per_thread × threads_per_block` + (`:70`). Interpreted, not compiled. + +
+ +- [ ] You can explain why Arrow layout is not a portability choice here but a performance one. + +
Answer + + `column_view` conforms to the Arrow Physical Memory Layout + (`column_view.hpp:33-35`): dense values plus a validity bitmap. Dense values + mean a warp's 32 loads land in one 128 B transaction; a row-store would strand + most of every fetch. The bitmap means nulls are bit operations rather than + per-row branches, and branches diverge warps. And the layout makes a slice + zero-copy — the offset lives in the view (`:40-42`). + + Strings are the exception that proves it: offsets + chars means variable work + per element, which is the ragged-work problem Gunrock's load-balancing + strategies exist to solve. + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the pass-2 compaction sketch. + +
Answer + + The slots are `notes.md:71-77`. Question 1's answer needs a launch count *you* + derived from the files, not the one in Step 2 — Step 2 counts three, and the + group-by path is different. + +
## References **Code** -- [cudf](https://github.com/rapidsai/cudf) — `cpp/src/` — the anchor - map above: `join/hash_join/` for size/retrieve, - `join/distinct_hash_join.cu` for cooperative-groups probing, - `groupby/hash/` for the shared-vs-global aggregation split, - `bitmask/` for validity-mask kernels + +- [cudf](https://github.com/rapidsai/cudf) @ `2f082a7` (the pin). Route: + `cpp/src/join/hash_join/` (`inner_join_size.cu`, `inner_join_retrieve.cu`, then + `size_impl.cuh` and `retrieve_impl.cuh`) → `hash_join_impl.cuh` and + `cpp/include/cudf/detail/join/distinct_hash_join.cuh` for the probing + parameters → `cpp/src/join/conditional_join_kernels.cuh` for atomic + amortisation → `cpp/src/groupby/hash/helpers.cuh` and its neighbours for the + two-tier aggregation → `cpp/include/cudf/column/column_view.hpp` for the + layout contract. +- cuco (CUDA Collections) is where `count`, `retrieve`, `insert_async` and the + probing schemes actually live. It is **not** pinned in this repo, so no line in + this guide points inside it; if you need its semantics, pin it first. + +**Papers** + +- Crystal (SIGMOD 2020) §3.2 and §4.3 for the two numbers this guide borrows: + one atomic per tile, and the 128 B vs 64 B read granularity that makes random + probes twice as expensive on a GPU. See `reading-crystal-sigmod20.md`. + +**Measurements in this repo** + +- `topics/18-gpu/notes.md:11-16` — the 1544 µs dispatch floor and the 12.5 GB/s + streaming figure Step 2's arithmetic uses. Both are wgpu/Metal numbers; a CUDA + launch is orders of magnitude cheaper, and the guide says so where it matters. diff --git a/topics/18-gpu/reading-wgpu-compute.md b/topics/18-gpu/reading-wgpu-compute.md index f2c2e15..83312ab 100644 --- a/topics/18-gpu/reading-wgpu-compute.md +++ b/topics/18-gpu/reading-wgpu-compute.md @@ -1,214 +1,500 @@ # wgpu compute: the 1.5 ms tax before your first FLOP -The portable GPU-compute stack our experiments use: WGSL shaders → -naga → Metal on this Mac, Vulkan/DX12 elsewhere. Before you open the -examples, this chapter builds the concepts one at a time — what a -dispatch actually consists of, where the fixed ~1.5 ms goes, what -WGSL can and cannot express, and which hard limits bite — each step -fixing one naivety of the previous. Then it maps every step to the -example directory that demonstrates it. +This is the only guide in topic 18 whose code you can actually run. The other +five read CUDA that needs an NVIDIA device; this machine is an Apple-silicon Mac +with none, so wgpu talking to Metal is the whole of the reader's hardware. That +turns out to be enough, because the thing topic 18 is about — the cost of +crossing the host/device boundary — is *worse* on a portable API, not absent +from it, and it is measurable here to the microsecond. + +The chapter builds four ideas in order: what a dispatch is made of, where the +fixed ~1.5 ms goes, what WGSL cannot express and what that forces on your kernel +shapes, and which hard limits bite at database-sized inputs. Each step fixes one +naivety of the previous one. + +Every wgpu anchor below is [gfx-rs/wgpu@f945c78](https://github.com/gfx-rs/wgpu) +— the revision pinned in `resources/codebases.md`. Check any of them with +`python3 tools/pinned-source.py show wgpu -r A:B`. One mismatch to know +about before you start: this topic's crate builds against the released +`wgpu = "23"` (`experiments/Cargo.toml`), so `src/gpu.rs:130` says +`device.poll(wgpu::Maintain::Wait)` while the pinned tree spells the same act +`device.poll(PollType::wait_indefinitely())`. Same rung of the ladder, renamed +between releases; the concepts are unchanged. ## The problem in one sentence -On this machine, summing 16K floats takes the CPU **2 µs** and the -GPU **1619 µs** — the GPU spends ~1.5 ms on plumbing before the -first FLOP, so the only interesting question is which operators ever -amortize that tax. +On this machine, summing 16K floats takes the CPU **2.3 µs** and the GPU +**1618.9 µs**, of which 1567.1 µs is encode/submit/poll with no data in it +(`notes.md:11`) — so the only interesting question in the whole topic is which +operators ever move enough work to amortize a fixed millisecond-and-a-half. ## The concepts, step by step ### Step 1 — a dispatch is a ladder of objects, not a function call -Running code on a GPU is not `f(x)` — it is building a chain of -objects that describe the device, the code, the data, and the -submission, then waiting for an asynchronous queue. wgpu's ladder, -top to bottom: +> **In:** a `&[f32]` on the host and a WGSL source string. +> **Out:** a `Vec` back on the host, plus roughly a dozen live objects you +> had to construct in a fixed order to get it. Nothing here is `f(x)`. + +Vocabulary, defined once and used for the rest of topic 18. A **shader** is the +program the GPU runs, written in **WGSL** (WebGPU's shading language). A +**pipeline** is that shader compiled together with the layout of the resources +it reads and writes. A **dispatch** launches the shader over a 3-D grid of +**workgroups**; a workgroup is a block of **invocations** (threads) that share +one fast scratch memory and can synchronise with each other — WebGPU's name for +what CUDA calls a thread block. Results reach the host only by copying into a +buffer created with `MAP_READ`, then mapping it. + +The ladder, as `01_hello_compute` climbs it. This is the shortest complete wgpu +compute program in the tree (254 lines, most of them comments), and every rung +appears exactly once: ``` - Instance — loads Metal/Vulkan/DX12 - └ Adapter — one physical GPU; limits + features live here - └ Device — the logical connection; creates ALL resources - Queue — where encoded work is submitted - Buffer(STORAGE) — GPU-side data - Buffer(MAP_READ|COPY_DST)— the ONLY way back to the host - ShaderModule (WGSL) → ComputePipeline (entry point + layout) - BindGroup — binds buffers to @group/@binding slots - CommandEncoder → ComputePass → dispatch_workgroups(x,y,z) - submit → poll → map_async → read +Instance main.rs:41 loads Metal/Vulkan/DX12 + └ Adapter main.rs:48 one physical GPU; limits and features live here + └ Device main.rs:69 logical connection — creates ALL resources + Queue main.rs:69 returned with it; where work is submitted +ShaderModule main.rs:83 WGSL parsed and validated +Buffer(STORAGE) main.rs:91 input, filled via create_buffer_init +Buffer(STORAGE|COPY_SRC) main.rs:98 output the kernel writes +Buffer(MAP_READ|COPY_DST) main.rs:108 the ONLY road back to the host +BindGroupLayout main.rs:118 what @group(0) @binding(n) will mean +BindGroup main.rs:152 the actual buffers bound to those slots +PipelineLayout main.rs:168 +ComputePipeline main.rs:176 shader + entry point + layout, compiled +CommandEncoder main.rs:186 + └ ComputePass main.rs:191 set_pipeline, set_bind_group, + dispatch_workgroups(...) main.rs:207 +copy_buffer_to_buffer main.rs:214 storage → mappable +encoder.finish() main.rs:223 → CommandBuffer +queue.submit() main.rs:230 +map_async + poll main.rs:238, 245 +get_mapped_range main.rs:248 bytes, at last ``` -Vocabulary for the rest of the chapter: a **shader** is the GPU -program (written in **WGSL**, WebGPU's shading language); a -**pipeline** is the compiled shader plus its resource layout; a -**dispatch** launches the shader over a 3D grid of **workgroups** -(blocks of threads that share fast scratch memory — the WebGPU name -for CUDA's thread block); results come back only by copying into a -mappable buffer and polling. Why it matters: every rung of this -ladder has a cost, and only some rungs can be paid once instead of -per call — that split is Steps 2 and 3. +Two details on that ladder are worth stopping on. The dispatch argument is a +*workgroup count*, not a thread count — `01_hello_compute` computes it as +`arguments.len().div_ceil(64)` (main.rs:206) because its shader declares +`@workgroup_size(64)`. And the device is requested with an explicit limits +struct (`required_limits: wgpu::Limits::downlevel_defaults()`, main.rs:72), +which is Step 7's subject: limits are a negotiation at device creation, not a +property of the hardware you discover later. ### Step 2 — the fixed tax: ~1.5 ms per dispatch before any work -Encoding commands, submitting to the queue, Metal's command-buffer -scheduling, and the completion poll together cost about 1.5 ms on -this Mac — *independent of data size*. The hello_compute doc-comment -says it outright: for trivial math "running on the gpu is slower -than doing the same calculation on the cpu... transfer/submission -overhead is quite a lot higher than the actual computation." Our -gpu_bench measured that sentence: +> **In:** the ladder of Step 1, run once per call on inputs from 2¹⁴ to 2²⁴ f32. +> **Out:** a phase-split timing table in which one column does not move. + +wgpu's own example says it out loud, in the file's doc comment: + +```rust +// examples/standalone/01_hello_compute/src/main.rs:8-12 — the module doc comment, +// quoted whole. This is the topic's thesis, written by the API's authors. + 8 /// If you time the recording and execution of this example you will certainly see that + 9 /// running on the gpu is slower than doing the same calculation on the cpu. This is because + 10 /// floating point multiplication is a very simple operation so the transfer/submission overhead + 11 /// is quite a lot higher than the actual computation. This is normal and shows that the GPU + 12 /// needs a lot higher work/transfer ratio to come out ahead. +``` + +`gpu_bench` measures that sentence. From `notes.md:11-16` (Apple M3 Pro, wgpu → +Metal, 2026-07-10, 5-rep averages, µs): + +``` + n CPU GPU total upload kernel+submit readback + 16K 2.3 1618.9 48.5 1567.1 3.2 + 64K 9.2 1633.5 69.6 1560.8 3.1 + 256K 36.8 1701.5 151.8 1547.1 2.6 + 1M 154.4 1985.5 437.3 1544.2 4.0 + 4M 588.6 4554.8 1654.8 2887.2 12.8 + 16M 2257.7 14332.9 7384.7 6929.1 19.1 +``` + +Read the `kernel+submit` column downwards for 16K → 1M: 1567.1, 1560.8, 1547.1, +1544.2. The input grew **64×** and that column *fell* by 23 µs. Nothing in it is +work; it is encode, submit, Metal's command-buffer scheduling, and the +completion poll — a floor of about **1544 µs** that every dispatch pays. + +Now subtract the floor to recover the actual kernel throughput at the top end. +64 MiB of f32 is 2²⁴ × 4 = 67,108,864 bytes: ``` - sum of n f32 — CPU 8-acc autovec vs GPU workgroup reduction: - n=16K CPU 2 µs GPU 1619 µs ← ~1.5 ms FIXED dispatch cost - n=4M CPU 589 µs GPU 4555 µs - n=16M CPU 2258 µs GPU 14333 µs ← no crossover, ever + work time at 16M = 6929.1 µs − 1544.2 µs floor = 5384.9 µs + effective read BW = 67,108,864 B / 5384.9e-6 s = 12.5 GB/s + CPU, same bytes = 67,108,864 B / 2257.7e-6 s = 29.7 GB/s ``` -A memory-bound operator on unified memory never wins: CPU and GPU -see the same ~150–400 GB/s pool, so the GPU's only edge is FLOPs a -sum doesn't need. The tax means any candidate operator needs either -high arithmetic intensity (FLOPs per byte) or a huge batch. -Question: break down the 1.5 ms — encode, submit, Metal -command-buffer scheduling, poll — which part would a persistent -command buffer (repeated_compute) remove? +So even with the tax removed the GPU reads *slower* than the CPU here. That is +the second, deeper reason for "no crossover": on unified memory both processors +pull from the same pool, so there is no bandwidth ratio to win — which is +exactly the premise Crystal's 16× speedups rest on and this machine does not +have (`reading-crystal-sigmod20.md`, Step 4). + +`FINDINGS.md:36` records a different run of the same lane — 7197 µs upload +against a 2723 µs CPU total at 16 M. Quote whichever file you took the number +from; they are separate runs, not a contradiction. ### Step 3 — amortize what you can: setup once, dispatch many -Of the Step 1 ladder, the expensive top rungs — instance/adapter/ -device creation and shader compilation into a pipeline — are -one-time costs, and our `GpuCtx` already hoists them: per-call cost -is only buffer create + bind + encode + submit. The -repeated_compute example goes further and reuses *buffers* across -iterations too — which is exactly Crystal's regime A → regime B -move (data resident on the device, only the dispatch per call). -Question: rewrite GpuCtx::sum to take pre-uploaded input (upload -once, dispatch many) — how does the crossover table change? This is -expressible in ~15 lines. +> **In:** the ladder of Step 1 and the floor of Step 2. +> **Out:** a partition of the ladder into rungs paid once per process and rungs +> paid once per call — and the knowledge that the per-call set is the one that +> costs 1.5 ms. + +`GpuCtx` already hoists the top of the ladder. `GpuCtx::try_new` +(`experiments/src/gpu.rs:52-83`) builds the instance, adapter, device, queue, +shader module and pipeline once and stores three of them; `GpuCtx::sum` +(`gpu.rs:88-144`) then does, per call: + +``` + upload phase gpu.rs:92-110 create_buffer_init(input) + 2 create_buffer + dispatch phase gpu.rs:112-131 bind group, encoder, pass, dispatch, copy, + submit, poll ← Step 2's ~1544 µs lives here + readback phase gpu.rs:133-141 map_async, poll, get_mapped_range +``` + +Bind group and pipeline layout are cheap; the floor is the submit-and-wait pair. +Which means the amortization that matters is not "build fewer objects" but +"submit fewer times, and upload once". The wgpu example that shows the first is +`features/src/repeated_compute/`; the second is what Crystal calls moving from +regime A (data streamed per query) to regime B (data resident) — and it is +Question 1 below, which you should predict before measuring. Arithmetic for the +prediction: hoisting upload out of the 16 M row removes 7384.7 µs from 14332.9, +leaving 6948.2 µs against the CPU's 2257.7 — still 3.1× behind, because Step 2 +showed the kernel itself is the slower reader. Hoisting alone cannot fix a +bandwidth deficit. ### Step 4 — WGSL is CUDA with the sharp edges filed off -Every CUDA concept from the papers has a WGSL name, and two have no -WGSL equivalent at all: +> **In:** the CUDA vocabulary the other five guides use. +> **Out:** the WGSL spelling of each, and the two constructs that have no +> spelling at all — which is what Steps 5 and 6 are about. | CUDA | WGSL | note | |---|---|---| -| `__global__` kernel | `@compute @workgroup_size(N) fn` | size fixed at pipeline creation | -| blockIdx/threadIdx | `@builtin(workgroup_id / local_invocation_id)` | | -| `__shared__` | `var` | our sum.wgsl scratch | -| `__syncthreads()` | `workgroupBarrier()` | workgroup-scope only | -| warp shuffles | subgroup ops (feature-gated) | portable fallback: shared memory | -| atomicAdd | `atomicAdd(&x, v)` on `atomic` | NO float atomics in core WGSL | - -The two DB-relevant gaps: **no float atomics** (you cannot -`atomicAdd` an f32 — aggregate via u32-bitcast CAS loops or -per-workgroup partials) and **no device-wide barrier** (threads in -different workgroups can never synchronize inside one dispatch). -Steps 5 and 6 show what each gap forces. +| `__global__` kernel | `@compute @workgroup_size(N) fn` | N is fixed when the pipeline is created, not at launch | +| `blockIdx` / `threadIdx` | `@builtin(workgroup_id)` / `@builtin(local_invocation_id)` | `sum.wgsl:15-16` takes both | +| `__shared__` | `var` | `sum.wgsl:9` | +| `__syncthreads()` | `workgroupBarrier()` | workgroup scope only — see Step 6 | +| warp shuffle (`__shfl_sync`) | subgroup operations | feature-gated; portable fallback is shared memory | +| `atomicAdd(float*)` | — | **no float atomics in core WGSL**; `atomic` / `atomic` only | +| grid-wide sync (cooperative groups) | — | **does not exist**; end the dispatch instead | -### Step 5 — the reduction shape forced by "no float atomics" +The two blanks are not portability pedantry — they change what a kernel can +look like. Step 5 is what the first blank forces, Step 6 the second. -To sum n floats without a float `atomicAdd`, each thread folds a -strided slice into a register, the workgroup tree-reduces those -partials in shared memory (`var`), and exactly one -thread writes one partial per workgroup — our sum.wgsl: +### Step 5 — the reduction shape forced by "no float atomics" -```rust -// sum.wgsl's shape: fold in registers, tree-reduce in shared memory, -// ONE partial per workgroup — because WGSL has no float atomicAdd -var scratch: array; - -@compute @workgroup_size(WG) -fn sum(gid: u32, lid: u32) { - var acc = 0.0; - for (var i = gid; i < n; i += stride) { acc += input[i]; } // coalesced - scratch[lid] = acc; - workgroupBarrier(); - for (var s = WG / 2u; s > 0u; s >>= 1u) { // tree reduction - if (lid < s) { scratch[lid] += scratch[lid + s]; } - workgroupBarrier(); - } - if (lid == 0u) { partials[workgroup_id] = scratch[0]; } -} // second dispatch (or CPU) folds the partials — no device barrier +> **In:** n f32 in a storage buffer, and a language with no float `atomicAdd`. +> **Out:** one f32 partial per workgroup, and a second pass (or the host) to +> fold the partials. + +Every invocation folds a strided slice into a register, the workgroup +tree-reduces those registers through `var` scratch, and exactly one +invocation writes one partial. Our whole kernel, with its real line numbers: + +```wgsl +// experiments/src/shaders/sum.wgsl:9-39 — declarations and both loops; the +// @group/@binding lines (6-7) and the closing brace are elided. + 9 var scratch: array; + 11 const WG: u32 = 256u; + 12 const PER_THREAD: u32 = 4u; + 14 @compute @workgroup_size(256) + 15 fn main(@builtin(local_invocation_id) lid: vec3, + 16 @builtin(workgroup_id) wid: vec3) { + 17 let n = arrayLength(&input); + 18 let base = wid.x * WG * PER_THREAD + lid.x; + 19 var v = 0.0; + 20 for (var k = 0u; k < PER_THREAD; k = k + 1u) { + 21 let i = base + k * WG; // stride WG — see below + 22 if (i < n) { + 23 v = v + input[i]; + 24 } + 25 } + 26 scratch[lid.x] = v; + 27 workgroupBarrier(); + 29 var stride = WG / 2u; // tree reduction + 30 while (stride > 0u) { + 31 if (lid.x < stride) { + 32 scratch[lid.x] = scratch[lid.x] + scratch[lid.x + stride]; + 33 } + 34 workgroupBarrier(); + 35 stride = stride / 2u; + 36 } + 37 if (lid.x == 0u) { + 38 partials[wid.x] = scratch[0u]; + 39 } ``` -The strided load (`i += stride` where stride = total thread count) -keeps adjacent threads on adjacent addresses — coalesced. The design -is not a style choice; it is the only shape the language permits, -and it is also the right shape (one atomic-free partial per group = -Crystal's "amortize atomics" rule). +Three things to notice, each of which is a rule the other guides will restate in +CUDA. **Coalescing**: at step k, invocation `lid.x` reads `base + k*256`, so the +256 invocations of a workgroup touch 256 *adjacent* f32 — one contiguous 1 KiB +run per step, which is the access pattern every GPU memory system is built for. +The obvious alternative (`lid.x * 4 + k`, each thread taking a contiguous quad) +would have 256 threads touching 256 addresses 16 bytes apart, and is the classic +way to lose most of your bandwidth. **Barrier discipline**: the barrier at line +34 is inside the loop but outside the `if` — every invocation in the workgroup +must reach it, so it can never sit under a divergent condition. **Atomic +amortization**: one write per workgroup, not one per element. That is the same +rule libcudf applies in `conditional_join_kernels.cuh` and CAGRA applies to its +visited table, and it is the rule the `filter_count` stub asks you to implement +with a real `atomicAdd` (`gpu.rs:146-156`). + +Check the shared-memory cost against the limit while you are here: +`array` is 256 × 4 = **1024 bytes** per workgroup, against a default +`max_compute_workgroup_storage_size` of **16384** bytes +(`wgpu-types/src/limits.rs:451`). And `@workgroup_size(256)` is not a round +number chosen for looks — it is exactly `max_compute_invocations_per_workgroup` +and `max_compute_workgroup_size_x` (limits.rs:452-453). The kernel sits on the +ceiling of the portable defaults in one dimension while using 1/16th of the +budget in the other. + +### Step 6 — no device-wide barrier: multi-pass means multiple dispatches + +> **In:** an algorithm with a global "everyone has finished phase 1" point. +> **Out:** one dispatch per phase, each paying Step 2's floor again. + +`workgroupBarrier()` synchronises the 256 invocations of one workgroup. Nothing +synchronises workgroup 0 with workgroup 16383: they may run concurrently, or +sequentially, or overlapped, and the language deliberately refuses to say. So a +phase boundary can only be expressed by ending the dispatch. + +This is why `sum` is not finished when the kernel returns: 16384 partials come +back to the host and are folded there (`gpu.rs:137-140`). A second dispatch +would be the alternative, and at 16384 f32 it would obviously lose — 1544 µs of +floor to add 64 KiB of numbers the host adds in microseconds. The rule +generalises badly for graphs: a BFS is one dispatch per level, so a +9-level traversal pays the floor nine times — 9 × 1544 µs = 13.9 ms of pure +submission before counting a single edge. Topic 13's LDBC-shaped graphs have +diameters in that range, which is why the stretch-goal BFS in this topic's +`notes.md:48` is a warning as much as an exercise. -### Step 6 — no device-wide barrier: multi-pass = multiple dispatches +### Step 7 — the limits that bite at real data sizes -Because workgroups cannot synchronize with each other, any algorithm -with a global "everyone finished phase 1" point must end the -dispatch and start another — the folding of sum.wgsl's partials is a -second dispatch (or the CPU), and a BFS runs one dispatch per level, -each paying Step 2's submission cost. Question: what does the -no-device-barrier rule do to the stretch-goal BFS (frontier per -dispatch — where does the frontier size live)? +> **In:** `wgpu::Limits::default()` (what `DeviceDescriptor::default()` requests +> at `gpu.rs:64`) and an input of n f32. +> **Out:** the two n at which this program stops working, and which fix each one +> needs. -### Step 7 — the limits that bite at real data sizes +Two defaults decide it (`wgpu-types/src/limits.rs`, values verified at the pin): -Two defaults matter for database-sized inputs: a single storage -buffer binding maxes out at **128 MB** -(`max_storage_buffer_binding_size`), and a dispatch allows at most -**65535 workgroups per dimension**. At n = 2²⁴ f32 (64 MB) with -256-thread workgroups you'd already need 65536 groups — our sum -kernel folds 4 elements per thread partly to stay under that limit. -Question: at what n does the 128 MB limit break GpuCtx::sum, and -what's the fix (request higher limits at device creation vs chunked -dispatches)? +| limit | default | line | +|---|---|---| +| `max_storage_buffer_binding_size` | 128 MiB (`128 << 20`) | limits.rs:441 | +| `max_buffer_size` | 256 MiB (`256 << 20`) | limits.rs:443 | +| `max_compute_workgroups_per_dimension` | 65535 | limits.rs:456 | +| `max_compute_invocations_per_workgroup` | 256 | limits.rs:452 | +| `max_compute_workgroup_storage_size` | 16384 B | limits.rs:451 | + +Work out where each one lands, with `ELEMS_PER_GROUP = WG × PER_THREAD = 256 × 4 += 1024` (`gpu.rs:8-10`) and `n_groups = ceil(n / 1024)` (`gpu.rs:90`): + +``` + storage binding: 128 MiB / 4 B = 33,554,432 = 2^25 elements + n = 2^24 (64 MiB) → fits, half the budget + n = 2^25 (128 MiB) → exactly at the limit + n = 2^26 (256 MiB) → rejected at bind time + + workgroup count: 65535 × 1024 = 67,107,840 elements ≈ 2^26 + n = 2^24 → 16,384 groups. 4× under the cap. + + the same kernel WITHOUT the 4-element fold (PER_THREAD = 1): + 65535 × 256 = 16,776,960 elements + n = 2^24 = 16,777,216 → 65,536 groups + short by 256 elements — one workgroup over the cap, + at exactly the size this topic's bench tops out at. +``` + +So `PER_THREAD = 4` is not a tuning knob, it is what keeps the largest measured +size legal; and the limit that actually ends the program is the 128 MiB storage +binding, at n > 2²⁵. The two fixes differ in kind: request a higher +`max_storage_buffer_binding_size` at device creation (portable only if the +adapter reports it — this is what `Limits` negotiation is *for*), or chunk the +input across several dispatches, which re-pays Step 2's floor per chunk. Note +that `downlevel_defaults()` — what `01_hello_compute` asks for at main.rs:72 — +keeps all of the above except workgroup storage, which drops to 16352 B +(limits.rs:521). Asking for less does not buy you a bigger buffer. ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| examples/standalone/01_hello_compute/ | the full plumbing, heavily commented — read FIRST | 1–2 | -| examples/standalone/01_hello_compute/src/shader.wgsl | minimal WGSL compute entry | 1, 4 | -| examples/features/src/repeated_compute/ | amortizing setup across dispatches (what our GpuCtx does) | 3 | -| examples/features/src/hello_workgroups/ | workgroup semantics + shared memory | 4–5 | -| examples/features/src/hello_synchronization/ | barriers + atomics | 4–6 | -| examples/features/src/big_compute_buffers/ | >128 MB data — chunking around limits | 7 | - -Read in that order: hello_compute end to end (every rung of Step 1's -ladder appears once, commented), then repeated_compute (diff it -against hello_compute — what moved out of the loop is exactly -Step 3's amortizable set), then the workgroups/synchronization pair -next to sum.wgsl, and big_compute_buffers only when Step 7 bites. +| `examples/standalone/01_hello_compute/src/main.rs:41-248` | the entire ladder, once, commented — read FIRST | 1 | +| `examples/standalone/01_hello_compute/src/main.rs:8-12` | the doc comment that admits the overhead | 2 | +| `examples/standalone/01_hello_compute/src/shader.wgsl` | minimal WGSL compute entry point | 1, 4 | +| `examples/features/src/repeated_compute/` | setup amortised across dispatches | 3 | +| `examples/features/src/hello_workgroups/` | workgroup id / shared memory semantics | 4-5 | +| `examples/features/src/hello_synchronization/` | barriers and atomics | 5-6 | +| `examples/features/src/big_compute_buffers/` | data past one binding — chunking | 7 | +| `wgpu-types/src/limits.rs:404-476` | `Limits::default()`, the numbers in Step 7 | 7 | +| this topic: `experiments/src/gpu.rs:88-144` | the measured `sum`, phase by phase | 2-3 | +| this topic: `experiments/src/shaders/sum.wgsl` | the kernel of Step 5 | 5 | + +Read in that order. `01_hello_compute` end to end first — it is short and every +rung of Step 1 appears once with a comment. Then diff `repeated_compute` against +it: what moved out of the loop is precisely Step 3's amortizable set. Then the +workgroups/synchronization pair alongside `sum.wgsl`, and `big_compute_buffers` +only once Step 7 bites you. ## Questions for notes.md -1. Measure: GpuCtx::sum with upload hoisted out (regime B). Does - the GPU beat 2258 µs CPU at n=16M now? Predict first. -2. Why does WGSL make workgroup_size a compile-time pipeline - constant while CUDA takes it at launch (hint: what can the - compiler do with a known size — our scratch array)? -3. The readback in our sum is 3-19 µs — tiny. Why is upload so much - worse (staging copy through a private buffer even on unified - memory — find the wgpu buffer-mapping discussion)? -4. Subgroup (warp) ops vs shared-memory reduction: rewrite - sum.wgsl's tree loop with subgroupAdd — how many barriers - disappear? -5. For M18: the feature flag should gate at the operator boundary. - Which signature do you expose: `sum(&[f32])` (per-call upload, - regime A) or `upload(&[f32]) -> GpuVec` + `sum(&GpuVec)` (regime - B)? Justify from this guide's measurements. +1. Measure: `GpuCtx::sum` with the upload hoisted out (regime B). Does the GPU + beat 2257.7 µs at n = 16M now? Predict first — Step 3 does the subtraction + for you, so commit to the answer before running it. +2. Why does WGSL make `workgroup_size` a compile-time pipeline constant while + CUDA takes the block size at launch? (Hint: `sum.wgsl:9` sizes `scratch` from + the same constant. What can the compiler do with a workgroup size it knows, + and what would it have to do without one?) +3. Readback at 16M is 19.1 µs against 7384.7 µs of upload — but compute both per + byte before concluding anything, remembering that readback moves 16384 + partials (64 KiB) and upload moves 64 MiB. Then explain the *real* asymmetry: + why does wgpu stage the upload through a private buffer even on unified + memory? +4. Subgroup (warp) operations vs the shared-memory tree: rewrite `sum.wgsl`'s + loop at lines 29-36 using `subgroupAdd`. How many `workgroupBarrier()` calls + disappear, and what does the shader now require of the adapter? +5. For M18 the feature flag should gate at the operator boundary. Which + signature do you expose — `sum(&[f32])` (per-call upload, regime A) or + `upload(&[f32]) -> GpuVec` plus `sum(&GpuVec)` (regime B)? Justify it from + Step 3's arithmetic, not from taste. ## Done when -- [ ] You can list the object ladder a dispatch requires and say which parts can be hoisted out of a loop. -- [ ] You can state the fixed per-dispatch tax and check it against this topic's measurement: ~1.4 ms of kernel time at every size from 2^14 to 2^20. -- [ ] You can explain why the absence of float atomics forces the tree-reduction shape. -- [ ] You can say why there is no device-wide barrier and what multi-pass therefore means. -- [ ] You can explain why upload dominates readback in the measured table (7197 µs against 25.6 µs at 16 M elements). -- [ ] You wrote answers to all five questions in notes.md, including the regime-B rerun with upload hoisted out. +Answer each before unfolding it. + +- [ ] You can list the object ladder a dispatch requires and say which rungs can be hoisted out of a loop — and which hoist does *not* help. + +
Answer + + Instance → Adapter → Device+Queue → ShaderModule → ComputePipeline are + per-process (main.rs:41-176; `GpuCtx::try_new` hoists exactly these, + gpu.rs:52-83). Buffers, bind group, encoder, pass, submit and map are + per-call (gpu.rs:92-141). + + The trap: hoisting the *objects* is not where the money is. Step 2's floor + lives in submit-and-poll, and Step 3's subtraction shows that removing the + entire 7384.7 µs upload at 16M still leaves 6948.2 µs against a 2257.7 µs + CPU. The only hoists that change the verdict are "submit fewer times" and + "keep the data resident", and on unified memory even both together are not + enough for a streaming reduction. + +
+ +- [ ] You can state the fixed per-dispatch tax and show the evidence that it is fixed rather than work. + +
Answer + + ~1544 µs. Evidence: the `kernel+submit` column of `notes.md:11-14` reads + 1567.1 → 1560.8 → 1547.1 → 1544.2 µs while n goes 16K → 64K → 256K → 1M. A + 64× increase in data with a 23 µs *decrease* in time is not work; it is + encode + submit + Metal command-buffer scheduling + the completion poll. + Above 1M the column finally starts to climb (2887.2 at 4M, 6929.1 at 16M) + because real work has at last become the larger term. + +
+ +- [ ] You can explain why the absence of float atomics forces the tree-reduction shape, and why that shape is the right one anyway. + +
Answer + + WGSL has `atomic` and `atomic` only, so there is no way for 4 M + invocations to accumulate into one f32. The portable route is: fold into a + register, tree-reduce through `var` scratch with a barrier per + halving (`sum.wgsl:26-36`), and have invocation 0 write one partial + (`sum.wgsl:37-39`). + + It is also what you would write if float atomics existed: 16384 conflicting + atomics beat 16.7 M of them by four orders of magnitude, and the tree costs + log₂(256) = 8 barrier-separated steps on data already in scratch. Amortising + atomics per workgroup is the same rule libcudf uses for its join output + (`conditional_join_kernels.cuh:74-77`, one device-scope `fetch_add` per + warp). + +
+ +- [ ] You can say why there is no device-wide barrier and what multi-pass therefore costs. + +
Answer + + Workgroups are scheduled independently and may not be resident at the same + time, so a grid-wide barrier could deadlock; WebGPU simply does not offer + one. A phase boundary therefore means ending the dispatch. Cost: Step 2's + ~1544 µs floor per phase. A 9-level BFS pays 9 × 1544 µs ≈ 13.9 ms of + submission before any edge is counted — which is why `sum` folds its 16384 + partials on the host (`gpu.rs:137-140`) rather than launching a second + kernel. + +
+ +- [ ] You can state, per byte, how upload and readback actually compare in the measured table — and resist the obvious misreading. + +
Answer + + At 16M (`notes.md:16`): upload 7384.7 µs for 67,108,864 B = **0.110 ns/B**; + readback 19.1 µs for 16384 × 4 = 65,536 B = **0.291 ns/B**. Per byte the + readback is 2.6× *worse*. "Upload dominates" is true of the totals only + because upload moves 1024× more bytes — the reduction's entire job is to + shrink the return trip. + + The upload number is itself the honest one to quote for transfer cost: + 67,108,864 B / 7384.7 µs = 9.1 GB/s on a machine whose CPU reads the same + bytes at 29.7 GB/s. `FINDINGS.md:36` records 7197 µs for the same phase on a + different run; cite whichever file you read. + +
+ +- [ ] You can name the two limits that end this program and the n at which each does it. + +
Answer + + `max_storage_buffer_binding_size` = 128 MiB (limits.rs:441) → 2²⁵ f32; the + next power of two, 2²⁶, is rejected when the bind group is created. + `max_compute_workgroups_per_dimension` = 65535 (limits.rs:456) → 65535 × 1024 + = 67,107,840 elements with the current `PER_THREAD = 4`, so it does not bind + first. It *would* have: at `PER_THREAD = 1` the cap is 65535 × 256 = + 16,776,960, and n = 2²⁴ = 16,777,216 needs 65,536 groups — over by one, at + exactly the largest size the bench runs. + + Fixes: negotiate a higher limit at `request_device` (adapter permitting), or + chunk and pay the floor per chunk. + +
+ +- [ ] You wrote answers to all five questions in `notes.md`, including the regime-B rerun with upload hoisted out. + +
Answer + + The slots are `notes.md:63-69`. Question 1's number also belongs in the + prediction table at `notes.md:39`, where the prediction column must be filled + in *before* the measurement. + +
## References **Code** -- [wgpu](https://github.com/gfx-rs/wgpu) — `examples/` — read in - order: `standalone/01_hello_compute/` (the full plumbing, heavily - commented — its doc-comment admits the overhead out loud), - `features/src/repeated_compute/` (amortizing setup — what our - GpuCtx does), then `hello_workgroups` / `hello_synchronization` / - `big_compute_buffers` as needed + +- [wgpu](https://github.com/gfx-rs/wgpu) @ `f945c78` (the pin) — read in order: + `examples/standalone/01_hello_compute/src/main.rs` (the full ladder, and the + doc comment at lines 8-12 that admits the overhead), + `examples/features/src/repeated_compute/` (what amortising looks like), + `examples/features/src/hello_workgroups/` and `hello_synchronization/` + (workgroup semantics, barriers, atomics), `examples/features/src/ + big_compute_buffers/` (life past one binding), and + `wgpu-types/src/limits.rs:404-476` for every number in Step 7. +- This topic: `experiments/src/gpu.rs` and `experiments/src/shaders/sum.wgsl` — + the code the measurements come from. + +**Spec** + +- [WebGPU](https://www.w3.org/TR/webgpu/) and + [WGSL](https://www.w3.org/TR/WGSL/) — the normative source for "no float + atomics" and for workgroup-scope-only barriers. wgpu's limits are the spec's + `supported limits` defaults; when an adapter and the spec disagree, the + adapter wins and `request_device` fails. + +**Measurements** + +- `notes.md:9-16` — the phase-split table every number in Steps 2, 3 and 7 is + taken from (Apple M3 Pro, wgpu → Metal, 2026-07-10). +- `FINDINGS.md:36` — the headline row, from a separate run of `./verify.sh 18`. diff --git a/topics/19-jit/README.md b/topics/19-jit/README.md index 804acc4..68958fe 100644 --- a/topics/19-jit/README.md +++ b/topics/19-jit/README.md @@ -74,9 +74,10 @@ queries. ## 2. SQLite's VDBE — the bytecode VM that refuses to die [`~/repos/sqlite/src/vdbe.c`](https://github.com/sqlite/sqlite) — one giant dispatch loop -(vdbe.c:1049 `switch( pOp->opcode )`), 199 `case OP_` opcodes, each -op a fixed struct (vdbeInt.h:55 `struct VdbeOp`: opcode + p1..p5 -operands). `EXPLAIN SELECT ...` prints the program. +(vdbe.c:1049 `switch( pOp->opcode )`), 190 top-level `case OP_` opcodes +(`grep -c 'case OP_'` says 199 — nine of those are inner-switch cases and +a doc comment), each op a fixed struct (src/vdbe.h:55 `struct VdbeOp`: +opcode + p1..p5 operands). `EXPLAIN SELECT ...` prints the program. ``` SELECT a+1 FROM t WHERE b < 10; @@ -139,21 +140,25 @@ JIT only (NOT whole-pipeline: the executor stays interpreted; llvmjit_expr.c:80 `llvm_compile_expr` compiles `ExprState` step arrays, emitting one basic block per step, llvmjit_expr.c:302-307). Two LLJIT instances at opt0/opt3 (llvmjit.c:100-101). Gated by -`jit_above_cost` (planner.c:699-700) — a *planner cost estimate* -threshold. Failure mode: estimate says expensive, query is short, -you pay 50 ms of LLVM for a 5 ms query. That's why every Postgres -ops guide says "try jit=off". Guide: -[reading-postgres-jit.md](reading-postgres-jit.md). +`jit_above_cost` = 100000, with `jit_optimize_above_cost` = 500000 +choosing the opt3 instance and `jit_inline_above_cost` = 500000 +enabling cross-module inlining (guc_parameters.dat:1458+) — all three +are *planner cost estimate* thresholds, not measured times. And at this +pin the `jit` GUC itself boots to **false** (guc_parameters.dat:1451-1456, +`variable => 'jit_enabled'`), so the cautionary tale ends with the +project agreeing: the failure mode is that the estimate says expensive, +the query is short, and you pay tens of ms of LLVM for a 5 ms query. +Guide: [reading-postgres-jit.md](reading-postgres-jit.md). ## 6. GraphBLAS's JIT — compile the KERNEL, cache it forever SuiteSparse takes a third road: the JIT unit is not a query but a *kernel specialization* (semiring × types × sparsity formats). `Source/jitifyer/GB_jitifyer.c` — encode the problem to a hash -(GB_encodify_mxm.c:55-59), look up an in-memory hash table -(GB_jitifyer.c:2119), fall back to an on-disk cache of compiled +(GB_encodify_mxm.c:58-61), look up an in-memory hash table +(GB_jitifyer.c:2122), fall back to an on-disk cache of compiled `.so` files, fall back to invoking THE C COMPILER at runtime and -`dlopen`ing the result (GB_jitifyer.c:1565,1937). Compile once per +`dlopen`ing the result (GB_jitifyer.c:1576,1937). Compile once per type-combo ever, not per query — amortization across the process lifetime, not across rows. FalkorDB inherits this whole machinery. Guide: [reading-graphblas-jit.md](reading-graphblas-jit.md). @@ -175,9 +180,12 @@ the eval.rs interpreter is the FalkorDB analogue of ExprState. [`~/repos/cranelift-jit-demo/src/jit.rs`](https://github.com/bytecodealliance/cranelift-jit-demo) is the whole recipe (461 lines): JITBuilder/JITModule (:39-41), FunctionBuilder translates -AST→CLIF IR (:135, :189), then declare→define→finalize→pointer -(:69-90). Cranelift sits at Umbra's design point: fast single-pass -compiles (~10-100× faster than LLVM), decent code, pure Rust. +AST→CLIF IR (:135, FunctionTranslator at :187-192), then +declare→define→finalize→pointer (`compile()` :53-93). Cranelift sits at +Umbra's design point: fast single-pass compiles, decent code, pure Rust. +For what "fast single-pass" is worth against LLVM, the sourced numbers +are Umbra's own (Table 3: 108× the compile speed of LLVM -O3 for code +1.2× slower) rather than a folk figure. Guide: [reading-cranelift-jit-demo.md](reading-cranelift-jit-demo.md). ## Experiments (`experiments/`) diff --git a/topics/19-jit/experiments/.gitignore b/topics/19-jit/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/19-jit/experiments/.gitignore +++ b/topics/19-jit/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/19-jit/reading-cranelift-jit-demo.md b/topics/19-jit/reading-cranelift-jit-demo.md index ac3f140..60bba23 100644 --- a/topics/19-jit/reading-cranelift-jit-demo.md +++ b/topics/19-jit/reading-cranelift-jit-demo.md @@ -10,6 +10,15 @@ then maps each step into jit.rs. Read it before touching experiments/src/jit.rs, because every ceremony the stub needs appears here first. +**Version.** Anchors are against `bytecodealliance/cranelift-jit-demo` +at the pin in `resources/codebases.md`, **`3e5e9b6`**, whose +`Cargo.toml` pins `cranelift`, `cranelift-module`, `cranelift-jit` +and `cranelift-native` all at **0.125.3**, edition 2024. This is not +pedantry: cranelift's builder API churns hard enough that most +tutorials on the web will not compile against 0.125. Step 6 lists +the three renames visible in this very file. Fetch any anchor with +`python3 tools/pinned-source.py show cranelift-jit-demo src/jit.rs -r 53:93`. + ## The problem in one sentence M19 needs to turn an `Expr` tree into a `fn(*const f64) -> f64` it @@ -21,20 +30,59 @@ ever letting the function pointer outlive the memory it points into. ### Step 1 — what a JIT library does: IR in, function pointer out +> **In:** an `Expr` tree, in memory, in our process. +> **Out:** the vocabulary for what the library takes (CLIF, in SSA +> form) and what it returns (`*const u8`) — the two ends of every +> later step. + Cranelift is a code generator: you hand it a function written in **CLIF** (Cranelift's intermediate representation — typed -instructions like `fadd`/`load` organized in basic blocks), and it +instructions like `iadd`/`load` organized in basic blocks), and it gives back native machine code placed in executable memory, plus a raw pointer you can call. CLIF is in **SSA** form (static single assignment — every value is defined exactly once; re-assignment becomes new values, and control-flow merges pass values as block parameters). You never write SSA by hand: a helper called `FunctionBuilder` maintains it while you emit instructions one at a -time. So the whole job of our stub is a recursive walk: Expr node -in, CLIF instruction out, then one call to compile. +time. + +The demo names all four pieces in one struct: + +```rust +// cranelift-jit-demo/src/jit.rs — the whole state of a JIT, 10-26 + 10 pub struct JIT { + 11 /// The function builder context, which is reused across multiple + 12 /// FunctionBuilder instances. + 13 builder_context: FunctionBuilderContext, + 14 + 15 /// The main Cranelift context, which holds the state for codegen. Cranelift + 16 /// separates this from `Module` to allow for parallel compilation, with a + 17 /// context per thread, though this isn't in the simple demo here. + 18 ctx: codegen::Context, + 19 + 20 /// The data description, which is to data objects what `ctx` is to functions. + 21 data_description: DataDescription, + 22 + 23 /// The module, with the jit backend, which manages the JIT'd + 24 /// functions. + 25 module: JITModule, + 26 } +``` + +Line 16's comment is the one to remember: `ctx` is separate from +`module` **so that you can have one context per thread**. That is +the API telling you what is cheap to duplicate and what is not — the +subject of Step 2. + +So the whole job of our stub is a recursive walk: Expr node in, CLIF +instruction out, then one call to compile. ### Step 2 — the object ladder (compare wgpu's, topic 18) +> **In:** Step 1's four objects. **Out:** which of them to create +> once per process and which per expression — i.e. what the constant +> term in your compile-time measurement is made of (question 3). + Like every runtime-code system, cranelift splits expensive long-lived containers from cheap per-function scratch: @@ -44,40 +92,129 @@ long-lived containers from cheap per-function scratch: ├─ builder_context: FunctionBuilderContext (reused scratch) └─ declare/define/finalize API - FunctionBuilder(&mut ctx.func) (SSA construction helper — + FunctionBuilder(&mut ctx.func, &mut builder_context) + (SSA construction helper — you emit ops, IT handles - block params/phi nodes) + block params) ``` +The construction is `impl Default for JIT`, `src/jit.rs:28-49`: + +```rust +// cranelift-jit-demo/src/jit.rs — building the module once, 30-42 + 30 let mut flag_builder = settings::builder(); + 31 flag_builder.set("use_colocated_libcalls", "false").unwrap(); + 32 flag_builder.set("is_pic", "false").unwrap(); +// ... 33-38: cranelift_native::builder() detects the host ISA; panics +// ... on an unsupported host ... + 39 let mut builder = JITBuilder::with_isa(isa, cranelift_module::default_libcall_names()); + 40 + 41 builder.symbol("hello", hello as *const u8); + 42 let module = JITModule::new(builder); +``` + +Line 33's `cranelift_native::builder()` is the expensive rung — it +queries the host CPU for available ISA features. Line 41 is the +mechanism you will need for anything the generated code must call +(here, a `hello` function): **symbols must be registered on the +builder before `JITModule::new`**, i.e. before you know what your +expression contains. Keep that in mind for M19's fallback boundary +(question 5). + Same shape as topic 18's Instance→Device→Pipeline: expensive long-lived containers (`JITModule` owns the executable memory), cheap per-function contexts (reused between compiles), and an -explicit "finalize" moment after which you hold a raw pointer. Why -it matters: the ladder tells you what to hoist — create the module -once, reuse the contexts per expression (question 3 measures the -difference). +explicit "finalize" moment after which you hold a raw pointer. +The ladder tells you what to hoist — create the module once, reuse +the contexts per expression. Question 3 measures the difference, and +Step 6 tells you what to expect. ### Step 3 — the compile ladder: declare, define, finalize (memorize this) -Compilation is a fixed seven-rung sequence — the split between -`define` (generate code) and `finalize` (patch relocations — -addresses of other functions/data unknown until everything is -placed) is the part that surprises: +> **In:** an AST and the `JIT` from Step 2. **Out:** a `*const u8` +> that points at executable machine code — and, critically, *not* +> a typed function; the cast is a separate act (Step 5). + +Compilation is a fixed sequence, and the split between `define` +(generate code) and `finalize` (patch relocations — addresses of +other functions/data unknown until everything is placed) is the part +that surprises. Here is the whole function, comments intact, because +the comments *are* the documentation for this API: + +```rust +// cranelift-jit-demo/src/jit.rs — compile(), the entire ladder, 53-93 + 53 pub fn compile(&mut self, input: &str) -> Result<*const u8, String> { + 54 // First, parse the string, producing AST nodes. + 55 let (name, params, the_return, stmts) = + 56 parser::function(input).map_err(|e| e.to_string())?; + 57 + 58 // Then, translate the AST nodes into Cranelift IR. + 59 self.translate(params, the_return, stmts)?; +// ... 61-66: comment — functions must be declared before defined ... + 67 let id = self + 68 .module + 69 .declare_function(&name, Linkage::Export, &self.ctx.func.signature) + 70 .map_err(|e| e.to_string())?; + 71 + 72 // Define the function to jit. This finishes compilation, although + 73 // there may be outstanding relocations to perform. Currently, jit + 74 // cannot finish relocations until all functions to be called are + 75 // defined. For this toy demo for now, we'll just finalize the + 76 // function below. + 77 self.module + 78 .define_function(id, &mut self.ctx) + 79 .map_err(|e| e.to_string())?; + 80 + 81 // Now that compilation is finished, we can clear out the context state. + 82 self.module.clear_context(&mut self.ctx); + 83 + 84 // Finalize the functions which we just defined, which resolves any + 85 // outstanding relocations (patching in addresses, now that they're + 86 // available). + 87 self.module.finalize_definitions().unwrap(); + 88 + 89 // We can now retrieve a pointer to the machine code. + 90 let code = self.module.get_finalized_function(id); + 91 + 92 Ok(code) + 93 } +``` + +**Corrected anchor.** `compile()` is `src/jit.rs:53-93`, not +"55-92". And read line 92 carefully: **`compile()` returns +`*const u8`. There is no `transmute` in this function.** The cast +lives somewhere else entirely — Step 5. + +The seven rungs, and what each buys: ``` - 1. translate AST → CLIF (FunctionTranslator walk) - 2. module.declare_function(name, Linkage::Export, &sig) → id - 3. module.define_function(id, &mut ctx) ← compilation happens - 4. module.clear_context(&mut ctx) ← reuse scratch - 5. module.finalize_definitions() ← relocations patched - 6. module.get_finalized_function(id) → *const u8 (:90) - 7. unsafe { mem::transmute::<_, fn(f64...)->f64>(ptr) } + 1. translate AST → CLIF :59 (Step 4) + 2. declare_function(name, Linkage::Export) :69 → a FuncId; the + name now exists in the module's symbol table, so other + functions may reference it before it has a body + 3. define_function(id, &mut ctx) :78 ← CODEGEN HAPPENS + machine code exists, but call targets are still placeholders + 4. clear_context(&mut ctx) :82 ← reuse the scratch + 5. finalize_definitions() :87 ← relocations patched; + the code becomes executable (W^X flip happens here) + 6. get_finalized_function(id) :90 → *const u8 + 7. transmute to a typed fn src/bin/toy.rs:51 ``` +Rung 5 is why rung 3 does not hand you something callable. A +**relocation** is a hole in the emitted code where an address +belongs — `call ` — that cannot be filled until the +target's final address is known. Line 87's comment says it exactly: +"resolves any outstanding relocations (patching in addresses, now +that they're available)". Question 1 asks which of *our* `Expr` +nodes creates one; Step 4 has the answer. + The same ladder as our stub will run it: ```rust -// CLIF in, callable pointer out — the whole recipe +// ILLUSTRATION — not quoted from cranelift-jit-demo. This is the +// jit.rs:53-93 ladder above, rewritten for the pure-expression signature +// the stub needs (experiments/src/jit.rs:42 is the function to fill in). fn compile(&mut self, expr: &Expr) -> fn(*const f64) -> f64 { let mut b = FunctionBuilder::new(&mut self.ctx.func, &mut self.b_ctx); let block = b.create_block(); @@ -85,7 +222,7 @@ fn compile(&mut self, expr: &Expr) -> fn(*const f64) -> f64 { b.switch_to_block(block); b.seal_block(block); // one block: seal immediately let row_ptr = b.block_params(block)[0]; - let v = translate(&mut b, expr, row_ptr); // the §Step-4 table, recursively + let v = translate(&mut b, expr, row_ptr); // the Step-4 table, recursively b.ins().return_(&[v]); b.finalize(); let id = self.module.declare_function("f", Linkage::Export, &sig)?; @@ -97,130 +234,541 @@ fn compile(&mut self, expr: &Expr) -> fn(*const f64) -> f64 { ``` The SSA ceremony (`create_block`, `append_block_params...`, -`switch_to_block`, `seal_block` — sealing tells the builder no more -predecessors will arrive, so it can resolve block params) collapses -to four lines because a pure expression needs exactly one block. +`switch_to_block`, `seal_block`) collapses to four lines because a +pure expression needs exactly one block. In the demo those same +calls are `src/jit.rs:138`, `:144`, `:147`, `:152`, and the +`FunctionBuilder::new` is `:135` with `builder.finalize()` at +`:180`. **Sealing** tells the builder no more predecessors will +arrive for that block, so it can resolve block parameters; with one +block and no jumps you can seal immediately. ### Step 4 — translating an expression: one CLIF op per Expr node -The demo's translator (jit.rs:189+) is statement-oriented; our -`Expr` is pure — simpler. The entire translation is this table, -applied by recursion: +> **In:** Step 3's rung 1 and a `FunctionBuilder` positioned in a +> sealed block. **Out:** a single CLIF `Value` for the whole +> expression, produced by one recursive match — the code M19 asks +> you to write. + +The demo's translator is `FunctionTranslator`, and the anchor here +needs fixing too: +```rust +// cranelift-jit-demo/src/jit.rs — the translator's state and its core match, 187-208 + 187 struct FunctionTranslator<'a> { + 188 int: types::Type, + 189 builder: FunctionBuilder<'a>, + 190 variables: HashMap, + 191 module: &'a mut JITModule, + 192 } +// ... 194-196: impl block + the doc comment "you get back `Value`s" ... + 197 fn translate_expr(&mut self, expr: Expr) -> Value { + 198 match expr { + 199 Expr::Literal(literal) => { + 200 let imm: i32 = literal.parse().unwrap(); + 201 self.builder.ins().iconst(self.int, i64::from(imm)) + 202 } + 203 + 204 Expr::Add(lhs, rhs) => { + 205 let lhs = self.translate_expr(*lhs); + 206 let rhs = self.translate_expr(*rhs); + 207 self.builder.ins().iadd(lhs, rhs) + 208 } ``` - Col(i) → load: builder.ins().load(F64, MemFlags::trusted(), - row_ptr, (i*8) as i32) - Const(c) → builder.ins().f64const(c) - Add(a,b) → builder.ins().fadd(va, vb) - Mul(a,b) → builder.ins().fmul(va, vb) - Lt(a,b) → cmp = builder.ins().fcmp(FloatCC::LessThan, va, vb) - → select(cmp, one, zero) (we keep f64 1.0/0.0) - And(a,b) → both sides as f64 0/1 → fmin or fmul (branch-free — - topic 17's predication instinct, now in codegen) + +`struct FunctionTranslator` is **`:187-192`** (the old "189-191" +lands on two of its four fields), and `translate_expr` is +**`:197-249`**. + +**A correction that changes what you can copy.** The demo's toy +language is **integer-only**. Line 188's field is `int`, seeded at +`:122-124`: + +> `// Our toy language currently only supports I64 values, though Cranelift` +> `// supports other types.` +> `let int = self.module.target_config().pointer_type();` + +There is no `fadd`, no `fmul`, no `f64const` and no `fcmp` anywhere +in this repository. The operations you will actually read are +`iconst` (`:201`), `iadd` (`:207`), `isub` (`:213`), `imul` +(`:219`), `udiv` (`:225`) and `icmp` (`:264`). So the f64 table +below is **our design for M19**, not something you can lift from the +demo — read it as a specification, and read the demo's integer +`translate_expr` for the *shape*. + +```rust +// ILLUSTRATION — NOT in cranelift-jit-demo, which is I64-only +// (see the comment at src/jit.rs:122-123). This is M19's own spec; the +// authoritative copy is the stub's module docs at experiments/src/jit.rs:11-16. +// The integer analogues you CAN read are iadd (jit.rs:207), imul (:219), +// icmp (:264). +Col(i) => b.ins().load(F64, MemFlags::trusted(), row_ptr, (i * 8) as i32), +Const(c) => b.ins().f64const(c), +Add(a,b) => b.ins().fadd(va, vb), +Mul(a,b) => b.ins().fmul(va, vb), +Lt(a,b) => { let cmp = b.ins().fcmp(FloatCC::LessThan, va, vb); + b.ins().select(cmp, one, zero) } // f64 1.0 / 0.0 +And(a,b) => // both sides already 0.0/1.0 → fmul is branch-free AND + b.ins().fmul(va, vb), ``` Signature: `fn(*const f64) -> f64` — one pointer param -(`AbiParam::new(types::I64)` or a real pointer type via -`module.target_config().pointer_type()`), one F64 return. Note the -comparisons stay branch-free (`fcmp` + `select`, values not jumps) — -generated straight-line code with no control flow is exactly what -Step 6's "quality gap vanishes" claim relies on. +(`AbiParam::new(module.target_config().pointer_type())`, exactly as +the demo does at `:127`), one F64 return (the demo pushes `int` at +`:132`; you push `types::F64`). The comparisons stay branch-free +(`fcmp` + `select`, values not jumps) — generated straight-line code +with no control flow is exactly what Step 6's "quality gap +vanishes" claim relies on. It is also topic 17's predication +instinct, now applied in codegen rather than in hand-written Rust. + +Two demo details worth stealing even though our `Expr` has no +control flow. First, cranelift has **no phi nodes** — merges use +block parameters, and the source says so: + +```rust +// cranelift-jit-demo/src/jit.rs — block params instead of phis, 279-284 and 323 + 279 // If-else constructs in the toy language have a return value. + 280 // In traditional SSA form, this would produce a PHI between + 281 // the then and else bodies. Cranelift uses block parameters, + 282 // so set up a parameter in the merge block, and we'll pass + 283 // the return values to it from the branches. + 284 self.builder.append_block_param(merge_block, self.int); +// ... 286-322: brif to then/else, each jumping to merge with its value ... + 323 let phi = self.builder.block_params(merge_block)[0]; +``` + +That is question 2's answer, verbatim from the source. Second, +**the only thing in this demo that creates a relocation** is a call: + +```rust +// cranelift-jit-demo/src/jit.rs — translate_call, the relocation source, 372-376 + 372 let callee = self + 373 .module + 374 .declare_function(&name, Linkage::Import, &sig) + 375 .expect("problem declaring function"); + 376 let local_callee = self.module.declare_func_in_func(callee, self.builder.func); +``` + +`Linkage::Import` at `:374` says "this lives elsewhere"; `:376` +records a reference from the current function to it, which becomes a +hole to patch at `finalize_definitions()`. Pure arithmetic never +reaches this code, which is exactly why our `Expr` produces zero +relocations — and why adding a single `pow()` would change that. ### Step 5 — the lifetime contract: the pointer is borrowed, not owned +> **In:** Step 3's `*const u8` at rung 6. **Out:** a typed function +> pointer plus the full list of invariants that makes the cast +> sound — the boundary where Rust stops helping. + `get_finalized_function` returns a raw pointer into memory the -`JITModule` owns; `transmute` erases that relationship, and Rust -can no longer save you. The pointer is valid exactly as long as the -JITModule lives — so our `CompiledExpr` must own the module -(`CompiledExpr { module, func }`; drop order = use-after-free -otherwise). postgres solves the same lifetime with per-context -resource trackers (llvmjit.c:288); the obligation is universal to -JITs, only the spelling differs. The other half of the unsafe -contract is the signature: the transmuted type must match the CLIF -signature and ABI exactly (question 4 spells out every -precondition). +`JITModule` owns. The cast is in `src/bin/toy.rs`, not in `jit.rs`: + +```rust +// cranelift-jit-demo/src/bin/toy.rs — the ONLY transmute in the demo, 44-54 + 44 /// input and output types. Using incorrect types at this point may corrupt the program's state. + 45 unsafe fn run_code(jit: &mut jit::JIT, code: &str, input: I) -> Result { unsafe { + 46 // Pass the string to the JIT, and it returns a raw pointer to machine code. + 47 let code_ptr = jit.compile(code)?; + 48 // Cast the raw pointer to a typed function pointer. This is unsafe, because + 49 // this is the critical point where you have to trust that the generated code + 50 // is safe to be called. + 51 let code_fn = mem::transmute::<_, fn(I) -> O>(code_ptr); + 52 // And now we can call it! + 53 Ok(code_fn(input)) + 54 }} +``` + +**Corrected claim.** The previous version of this guide said "the +demo transmutes to `fn(f64) -> f64`". It does not. Line 51 +transmutes to a *generic* `fn(I) -> O` inside an `unsafe fn`, and +because the toy language is I64-only (Step 4), every actual call +site in `main` uses integer types. The `f64` in the old sentence was +imported from our own stub's signature. + +Line 45's `unsafe fn` and line 44's doc comment are doing real work: +the demo pushes the entire type-correctness obligation onto the +caller, in the type system, by making `I` and `O` free parameters. +Our stub does the opposite — it fixes the signature at +`experiments/src/jit.rs:30` (`func: fn(*const f64) -> f64`) so the +obligation is discharged once, inside `compile()`. + +The pointer is valid exactly as long as the JITModule lives — so +`CompiledExpr` must own the module. The stub already encodes this: + +```rust +// database-learning-path/topics/19-jit/experiments/src/jit.rs — the ownership fix, 26-31 + 26 pub struct CompiledExpr { + 27 /// Keeps the executable memory alive. Never dropped before `func` + 28 /// stops being called. + 29 _module: JITModule, + 30 func: fn(*const f64) -> f64, + 31 } +``` + +Field order matters: Rust drops fields in declaration order, so +`_module` at `:29` is dropped *before* `func` at `:30` — which is +harmless because `func` is a plain pointer with no destructor, but +reverse the fields and you have written a footgun for the next +person. Postgres solves the same lifetime with per-context resource +trackers (`llvmjit.c:288-289`, see `reading-postgres-jit.md`); the +obligation is universal to JITs, only the spelling differs. ### Step 6 — the design point: cranelift vs LLVM, and the gotcha list +> **In:** everything above — a working compile path. **Out:** where +> this compiler sits on the compile-time/code-quality frontier, what +> that predicts for our lane, and the version hazards that will +> actually cost you an afternoon. + ``` cranelift LLVM -O3 - compile speed ~10-100× faster baseline - code quality ~ -O0..-O1 best - passes e-graph based ~100 passes - mid-end (aegraph) + compile speed much faster baseline + code quality roughly -O0..-O1 best written in Rust (no FFI) C++ (bindgen pain) designed for wasmtime JIT everything ``` -Cranelift ≈ Umbra's Flying Start as a design point (fast, -single-tier, good-enough). For straight-line f64 arithmetic the -quality gap vs LLVM nearly vanishes — no loops to optimize, and -OUR loop (over rows) stays in Rust and gets rustc -O. +**Claim removed as unverifiable.** The previous version of this +guide printed "~10-100× faster than LLVM" and "e-graph based mid-end +(aegraph)". Neither is checkable from anything in this repo's pin +table — no cranelift-vs-LLVM benchmark ships in `cranelift-jit-demo` +— so both are gone rather than repeated. What *is* measured, in a +peer-reviewed paper this topic already reads, is the same design +point for a different fast compiler: + +``` + Umbra's Flying Start vs LLVM -O3, geometric mean over TPC-H + (Kersten/Leis/Neumann, VLDBJ 2021, Table 3, SF=1, 20 threads): + compilation 108× faster + execution 1.2× slower → 1/1.2 = 83% of -O3's speed + + Copy-and-Patch vs LLVM (Xu & Kjolstad, OOPSLA 2021, Fig. 24): + compile up to 276× faster than -O0 + up to 1435× faster than -O1/-O2/-O3 + execution 14% FASTER than -O0, 24% slower than -O3 +``` + +Cranelift occupies the same region of that frontier: single-pass-ish +compilation, code roughly at `-O0`/`-O1`, no LLVM dependency. Use +those two rows as the *prior* for what to expect from our lane, then +measure — that is exactly what `notes.md`'s prediction worksheet is +for. + +For straight-line f64 arithmetic the quality gap vs LLVM should +nearly vanish — there are no loops to optimize, and OUR loop (over +rows) stays in Rust and gets `rustc -O`. Predict that before you +measure it, then check whether the JIT lane beats `vector`'s +measured 11.8 M rows/s at 511 nodes (`notes.md`). The topic's own +prediction is that it will *not* clearly win, because the vectorized +lane gets SIMD from autovectorization and scalar CLIF does not. + +**Gotchas for the stub, with the evidence for each.** + +- **Version lock.** cranelift crates move together — + `Cargo.toml:11-15` pins `cranelift`, `cranelift-module`, + `cranelift-jit`, `cranelift-native` all at `0.125.3`. Three API + changes are visible in this file alone, and each will break a + tutorial written a year ago: + - jump arguments are now `&[BlockArg::Value(v)]` (`:301`, `:313`; + the import is `use cranelift::codegen::ir::BlockArg;` at + `:2`), not a bare `&[Value]`; + - `builder.declare_var(int)` **returns** the `Variable` (`:460`), + replacing the older `Variable::new(idx)` + `declare_var(var, + ty)` pair; + - the conditional branch is `brif(cond, then, &[], else, &[])` + (`:289`, `:339`), replacing `brz`/`brnz`. +- `cranelift_native::builder()` (`:33`) detects the host ISA; the + demo sets `is_pic` to `"false"` at `:32`, which is right for a + JIT that never writes a shared object. +- `MemFlags::trusted()` = aligned + notrap: we promise `row_ptr` is + valid — the unsafe contract lives at the `eval()` call site + (`experiments/src/jit.rs:34-38` documents it as a caller + contract). +- Floats: use `fcmp` + `select`, NOT bitcast tricks — CLIF's + boolean handling has changed across versions and `select` on f64 + is the stable spelling. +- The module must not be dropped: `CompiledExpr { _module, func }` + (`experiments/src/jit.rs:26-31`), with `func` called through the + stored pointer. + +### Step 7 — the arithmetic: what compile time has to beat + +> **In:** Step 6's design point and `notes.md`'s measured per-row +> rates. **Out:** the number `compile()` must come in under for +> M19's JIT lane to be worth shipping — computed, not guessed. + +The bench harness runs a fixed number of rows per expression. Turn +that into a compile-time budget: -Gotchas for the stub: +``` + Measured (notes.md, Apple M3 Pro, 2026-07-10, N_COLS=4, + depth 8 = 511 nodes, best-of-3): + interp lane 0.95 M rows/s → 1.053 µs/row + vector lane 11.8 M rows/s → 0.0847 µs/row + + Suppose the JIT lane lands at the vector lane's rate (the topic's + own prediction). Then, per row, compiling saves: + vs interp: 1.053 − 0.0847 = 0.9683 µs + vs vector: 0.0847 − 0.0847 = 0 ← nothing to win + + Break-even against the INTERPRETER, rows = compile_µs / 0.9683: + compile in 100 µs → 103 rows + compile in 500 µs → 516 rows + compile in 5000 µs → 5,164 rows + + Break-even against the VECTORIZED lane: undefined — the + denominator is zero or negative unless the JIT is genuinely + faster per row than autovectorized Rust. THAT is the real + experiment M19 is asking you to run, and the reason notes.md + wants a prediction first. + + Now invert it. If jit_bench feeds 2,000,000 rows and you want the + JIT to pay for itself in under 1% of total runtime: + interp time for 2M rows = 2e6 × 1.053 µs = 2.106 s + 1% budget = 21.06 ms + So any compile under ~21 ms is invisible at this row count. + Cranelift on a 511-node expression will be far under that — which + means at 2M rows the compile fee is NOT the interesting variable. + Re-run the sum at 1,000 rows and it becomes the ONLY variable. +``` -- Version lock: cranelift crates move together — Cargo.toml pins - matching versions of cranelift-{jit,module,frontend,codegen,native}. -- `cranelift_native::builder()` detects the host ISA; enable - `is_pic` false default is fine for JIT. -- `MemFlags::trusted()` = aligned + notrap: we promise row_ptr is - valid — the unsafe contract lives at the `eval()` call site. -- Floats: use `fcmp`+`select`, NOT bint/bitcast tricks — CLIF's - bool handling changed across versions; select on f64 is stable. -- The module must not be dropped: `CompiledExpr { module, func }` - with func called through a stored raw pointer. +That last inversion is the whole reason this topic exists. Compile +time is not expensive or cheap in the abstract; it is expensive or +cheap *relative to a row count you must name*. Postgres names it +with an estimate and gets it wrong (`reading-postgres-jit.md`); +Umbra refuses to name it and switches tiers mid-query +(`reading-umbra-tidy-tuples.md`); SQLite names it as "about five" +and correctly never compiles at all (`reading-sqlite-vdbe.md`). ## Where each step lives in the code +All anchors are `cranelift-jit-demo` at `3e5e9b6` (461 lines total). + | anchor | what it is | step | |---|---|---| -| src/jit.rs:12-25 | the four state objects | 2 | -| src/jit.rs:39-41 | `JITBuilder::with_isa(...)` → `JITModule::new` | 2 | -| src/jit.rs:55-92 | `compile()` — the whole ladder, annotated above | 3 | -| src/jit.rs:135 | `FunctionBuilder::new(&mut ctx.func, &mut builder_context)` | 3 | -| src/jit.rs:180 | `builder.finalize()` — seals the CLIF function | 3 | -| src/jit.rs:189-191 | `FunctionTranslator` — AST→CLIF recursion lives here | 4 | -| src/jit.rs:400+ | helper emitters (calls, comparisons) | 4 | -| src/frontend.rs | the toy parser (87 lines — ignore, we have `Expr`) | — | - -Read jit.rs top to bottom once, then re-read `compile()` (:55-92) -against Step 3's seven rungs until each line maps to a rung; the -`FunctionTranslator` walk (:189+) is Step 4 with statements added -that our pure `Expr` doesn't need. +| `src/jit.rs:10-26` | the four state objects, with the "one context per thread" note at `:16` | 1, 2 | +| `src/jit.rs:28-49` | `impl Default for JIT` — ISA detection and module construction | 2 | +| `src/jit.rs:32-33` | `is_pic=false`; `cranelift_native::builder()` | 2 | +| `src/jit.rs:39-42` | `JITBuilder::with_isa(...)`, `builder.symbol(...)`, `JITModule::new` | 2 | +| `src/jit.rs:53-93` | `compile()` — the whole ladder; returns `*const u8`, **no transmute** | 3 | +| `src/jit.rs:67-70` / `:77-79` / `:82` / `:87` / `:90` | declare / define / clear / finalize / get | 3 | +| `src/jit.rs:116-182` | `translate()` — signature, entry block, seal, `return_` (`:177`) | 3 | +| `src/jit.rs:122-124` | **"Our toy language currently only supports I64 values"** | 4 | +| `src/jit.rs:135` | `FunctionBuilder::new(&mut ctx.func, &mut builder_context)` | 3 | +| `src/jit.rs:138` / `:144` / `:147` / `:152` | `create_block`, `append_block_params_for_function_params`, `switch_to_block`, `seal_block` | 3 | +| `src/jit.rs:180` | `builder.finalize()` — seals the CLIF function | 3 | +| `src/jit.rs:187-192` | `FunctionTranslator` — AST→CLIF recursion state | 4 | +| `src/jit.rs:197-249` | `translate_expr` — `iconst` `:201`, `iadd` `:207`, `imul` `:219` | 4 | +| `src/jit.rs:251-395` | the helper emitters: assign `:251`, icmp `:261`, if/else `:267`, while `:328`, call `:360`, global data `:386` | 4 | +| `src/jit.rs:279-284`, `:323` | block parameters instead of phi nodes | 4 | +| `src/jit.rs:372-376` | `Linkage::Import` + `declare_func_in_func` — the only relocation source | 3, 4 | +| `src/jit.rs:398-461` | `declare_variables*` — note `declare_var` returns the `Variable` at `:460` | 6 | +| `src/bin/toy.rs:44-54` | `unsafe fn run_code` — the **only** `transmute`, at `:51` | 5 | +| `Cargo.toml:11-15` | the version lock: cranelift 0.125.3 × 4 crates | 6 | +| `src/frontend.rs` | the toy parser (87 lines — ignore, we have `Expr`) | — | + +**Anchor corrections against the previous version of this guide:** +`compile()` is `:53-93` (was "55-92"); `struct FunctionTranslator` +is `:187-192` (was "189-191"); the `JIT` struct is `:10-26` (was +"12-25"); and the helper emitters are `:251-395`, **not `:400+`** — +`:398-461` is variable declaration, a different subject. + +Read `jit.rs` top to bottom once, then re-read `compile()` +(`:53-93`) against Step 3's seven rungs until each line maps to a +rung. Then read `translate_expr` (`:197-249`) as Step 4 with +statements added that our pure `Expr` doesn't need, and finish with +`translate_if_else` (`:267-326`) for the block-parameter idiom. ## Questions for notes.md -1. Why does define_function (:78) not yet give you a callable — - what do relocations still need (addresses of other functions/ - data), and which of our Expr nodes would introduce one (none — - pure arithmetic; a `pow()` call would)? -2. FunctionBuilder "handles SSA construction" — what does that - mean concretely for a `var` assigned in two branches (block - params instead of phi nodes — how do they differ)? +1. Why does `define_function` (`src/jit.rs:78`) not yet give you a + callable — what do relocations still need? Read the comment at + `:72-76` and `:84-86` for the library's own answer. Which of our + `Expr` nodes would introduce one? (Trace it: the only path to a + relocation in this demo is `translate_call` at `:372-376`, via + `Linkage::Import`. Pure arithmetic never gets there. What would + adding `pow()` cost, and where would you have to register the + symbol — see `:41`?) +2. `FunctionBuilder` "handles SSA construction" — what does that + mean concretely for a variable assigned in two branches? The + demo answers it in a comment at `:279-284` and uses the result + at `:323`. State the difference between a phi node and a block + parameter, and say which one cranelift has. 3. Time `compile()` in jit_bench across expr depths 2..12. Is it - linear in node count? Where does the constant term come from - (ISA setup? module init? — hoist GLOBAL vs per-expr state and - measure both ways)? -4. The demo transmutes to `fn(f64) -> f64`. Spell out every - precondition that makes our `fn(*const f64) -> f64` transmute - sound (ABI = System V default? signature match? module alive? - W^X handled by JITModule?). -5. M19: eval.rs values aren't all f64 (nodes, strings, nulls). + linear in node count? Where does the constant term come from? + Hoist GLOBAL vs per-expr state and measure both ways — the + candidates are the ISA detection at `:33`, `JITModule::new` at + `:42`, and the per-call `FunctionBuilder::new` at `:135`. Then + plug your measured `compile_µs` into Step 7's division and say + how many rows the lane needs. +4. `src/bin/toy.rs:51` transmutes to a generic `fn(I) -> O` inside + an `unsafe fn`. Spell out every precondition that makes our + `fn(*const f64) -> f64` transmute sound: ABI match, signature + match (params AND return type AND count), the module still + alive, `row_ptr` valid for `n_cols * 8` bytes and aligned + (because we used `MemFlags::trusted()`), and W^X already flipped + by `finalize_definitions()`. Which of those does the type system + check for you? (Answer: none.) +5. M19: FalkorDB's values aren't all f64 (nodes, strings, nulls). Which subset of Cypher expressions compiles to this f64 scheme - directly, and what's the fallback boundary (per-node fallback - vs whole-expression bailout — pick one and defend it)? + directly, and what's the fallback boundary — per-node fallback + (call back into the interpreter for one node) vs whole-expression + bailout? Pick one and defend it. Note the constraint from + `:41`: symbols the generated code may call must be registered on + the `JITBuilder` *before* `JITModule::new`, i.e. before you have + seen the expression. ## Done when +Answer each before unfolding it. + - [ ] You can recite the compile ladder: declare, define, finalize — and say why `define_function` alone does not give you a callable. + +
Answer + + translate → `declare_function` (`:69`) → `define_function` + (`:78`) → `clear_context` (`:82`) → `finalize_definitions` + (`:87`) → `get_finalized_function` (`:90`) → transmute + (`src/bin/toy.rs:51`). `define_function` runs codegen and emits + machine code, but any address the code needs — another function, + a data object — is still a **relocation**: a hole waiting for a + final address. The source says so at `:72-76`. Only + `finalize_definitions()` patches those holes and makes the memory + executable, which is why `compile()` calls it before touching + `get_finalized_function`. Note also that `compile()` returns + `*const u8` (`:53`, `:92`) — the typed cast is not part of the + ladder. + +
+ - [ ] You can explain what `FunctionBuilder` handling SSA construction saves you from doing. + +
Answer + + SSA requires every value to be defined once, so a variable + assigned on both sides of an `if` needs a merge construct. In + classical SSA that is a phi node; cranelift instead uses **block + parameters** — the merge block declares a parameter + (`append_block_param`, `:284`) and each incoming branch passes its + value as a jump argument (`jump(merge_block, + &[BlockArg::Value(then_return)])`, `:301`/`:313`), then the merged + value is read back with `block_params(merge_block)[0]` (`:323`). + The comment at `:279-283` states this explicitly. What + `FunctionBuilder` saves you from is doing the dominance analysis + yourself: you write `declare_var`/`def_var`/`use_var` and it + inserts the parameters and arguments. `seal_block` is the one + obligation it hands back — you must tell it when a block has no + further predecessors. + +
+ - [ ] You can state the lifetime contract on the returned pointer and every invariant the `transmute` is assuming. -- [ ] You can time `compile()` across expression depths and say whether it is linear in node count. + +
Answer + + `get_finalized_function` (`:90`) returns a borrow into memory the + `JITModule` owns; dropping the module unmaps the code. So the + typed pointer is valid exactly as long as the module lives, and + `CompiledExpr` must own it — `experiments/src/jit.rs:26-31` does + this, with the comment at `:27-28` recording why. The transmute at + `src/bin/toy.rs:51` additionally assumes: the target type's ABI + matches the CLIF signature's calling convention; the parameter + count, parameter types and return type all match exactly; the + memory has already been made executable (true after + `finalize_definitions()`); and — for our signature — that + `row_ptr` points to at least `n_cols` aligned `f64`s, because + Step 4 emits `MemFlags::trusted()` which promises aligned and + non-trapping. The compiler checks none of these; that is why + `run_code` is an `unsafe fn` and why its doc comment at `:44` + warns that wrong types "may corrupt the program's state". + +
+ +- [ ] You can time `compile()` across expression depths and say whether it is linear in node count, and convert that into a break-even row count. + +
Answer + + Expect near-linear in node count with a constant term, because + the translation is one recursive pass emitting one CLIF + instruction per node and cranelift's backend is single-pass-ish. + The constant comes from the per-call `FunctionBuilder::new` + (`:135`) and the declare/define/finalize ceremony, *not* from ISA + detection (`:33`) or `JITModule::new` (`:42`) if you hoisted + those — which is the point of measuring both ways. Then: + `rows = compile_µs / (µs_per_row_interp − µs_per_row_jit)`. With + `notes.md`'s depth-8 rates (interp 1.053 µs/row; assume the JIT + reaches the vector lane's 0.0847 µs/row), the denominator is + 0.9683 µs, so 100 µs of compile pays back in 103 rows and 500 µs + in 516. Against the *vectorized* lane the denominator may be zero + or negative — record that as a finding, not a failure. + +
+ +- [ ] You can name the three API changes in this pin that will break an older cranelift tutorial. + +
Answer + + At `3e5e9b6` / cranelift 0.125.3: (1) jump arguments are + `&[BlockArg::Value(v)]` (`src/jit.rs:301`, `:313`, with + `use cranelift::codegen::ir::BlockArg;` at `:2`), not `&[Value]`; + (2) `builder.declare_var(ty)` **returns** the `Variable` + (`:460`), replacing `Variable::new(idx)` followed by + `declare_var(var, ty)`; (3) conditional branches are + `brif(cond, then_block, &[], else_block, &[])` (`:289`, `:339`), + replacing the older `brz`/`brnz` pair. This is why the + `Cargo.toml:11-15` version lock is in Step 6's gotcha list rather + than a footnote. + +
+ - [ ] You wrote answers to all five questions in notes.md, including how you will handle non-f64 values in M19. +
Answer + + The constraint that decides question 5: `src/jit.rs:41` + (`builder.symbol("hello", hello as *const u8)`) registers callable + symbols on the `JITBuilder` **before** `JITModule::new` at `:42`. + If you choose per-node fallback — calling back into the + interpreter for the nodes you cannot compile — you must register + those callbacks up front, and each one becomes a relocation + (`:372-376`) and a real call in the hot loop, which is precisely + what Neumann's §4.1 "the hot path is pure LLVM" rule warns + against. Whole-expression bailout keeps the compiled path + call-free at the cost of compiling nothing when any node is + unsupported. Defend whichever you pick with the fraction of real + Cypher expressions that are pure numeric — count it, don't guess. + +
+ ## References -**Code** -- [cranelift-jit-demo](https://github.com/bytecodealliance/cranelift-jit-demo) - — `src/jit.rs` — read it top to bottom; `src/frontend.rs` (the toy - parser) can be skipped, we already have `Expr` +**Code** — all anchors verified at `cranelift-jit-demo` `3e5e9b6` + +| file | what to read | +|---|---| +| `src/jit.rs` | read top to bottom; `:53-93` is the ladder, `:197-249` the translator, `:267-326` the block-parameter idiom | +| `src/bin/toy.rs:44-54` | the only `transmute`, and the `unsafe fn` that documents its obligations | +| `Cargo.toml:11-15` | the version lock (cranelift 0.125.3) | +| `src/frontend.rs` | the toy parser, 87 lines — skippable, we already have `Expr` | + +Fetch without a clone: +`python3 tools/pinned-source.py show cranelift-jit-demo src/jit.rs -r 187:250`. + +**Elsewhere in this repo** +- `experiments/src/jit.rs:11-21` — the stub's own spec for the f64 + translation table and the ownership rule; `:26-31` the + `CompiledExpr` shape; `:42` the function to fill in +- `reading-postgres-jit.md` — the same lifetime problem solved with + ORC resource trackers (`llvmjit.c:288-289`) +- `reading-umbra-tidy-tuples.md` — the measured compile-time / + code-quality trade-off Step 6 borrows its numbers from +- `reading-neumann-vldb11.md` — §4.1's rule about the hot path not + crossing a function boundary, which decides question 5 + +**Papers cited for the design-point numbers** +- Kersten, Leis, Neumann — "Tidy Tuples and Flying Start" (VLDB + Journal 2021), **Table 3**: Flying Start vs LLVM -O3 — 108× + faster to compile, 1.2× slower to execute. +- Xu, Kjolstad — "Copy-and-Patch Compilation" (OOPSLA 2021), + **Fig. 24**: up to 276× faster than LLVM -O0 (1435× vs -O1..-O3), + producing code 14% faster than -O0 and 24% slower than -O3. diff --git a/topics/19-jit/reading-graphblas-jit.md b/topics/19-jit/reading-graphblas-jit.md index b70f35e..c5f51e4 100644 --- a/topics/19-jit/reading-graphblas-jit.md +++ b/topics/19-jit/reading-graphblas-jit.md @@ -1,207 +1,839 @@ # GraphBLAS JIT: compile once per semiring, cache forever The third grain of JIT. Postgres compiles per query; Umbra per -pipeline; GraphBLAS compiles per *kernel specialization* — a -(operation × semiring × types × sparsity formats) combination — -and caches it for the lifetime of the machine. FalkorDB runs on -this. Home turf. This chapter builds the design one decision at a -time — why the kernel space explodes, what the generic fallback -costs, the four-level cache ladder, and the cache key that makes it -all sound — then maps each decision into GB_jitifyer.c. +pipeline; GraphBLAS compiles per *kernel specialization* — an +(operation × semiring × types × sparsity formats) combination — and +caches it for the lifetime of the machine. FalkorDB runs on this. +Home turf. This chapter builds the design one decision at a time — +why the kernel space explodes, what the generic fallback actually +costs, the cache ladder, the cache key that makes it sound, and the +locking that makes it thread-safe — then maps each decision into +GB_jitifyer.c. + +**Version.** All anchors are `DrTimothyAldenDavis/GraphBLAS` at the +pin in `resources/codebases.md`, **`1fd5475`** (a v10-era tree, with +the 32/64-bit integer-width flags of Step 5 and a +`Source/jitifyer/GB_jitifyer.c` of 2,780 lines). The JIT is recent +and the tree moves; four anchors in the previous version of this +guide were off, and Step 7's table records each correction. Fetch +any of them with +`python3 tools/pinned-source.py show GraphBLAS Source/jitifyer/GB_jitifyer.c -r 1626:1674`. ## The problem in one sentence GraphBLAS lets users define their own types and operators, so the -space of possible kernels is *infinite* — thousands of precompiled -"factory" kernels still can't cover it — and the fallback -(function-pointer calls per matrix entry) is a per-element -interpreter ~10× slower than a specialized kernel. +space of possible kernels is *unbounded* — thousands of precompiled +"factory" kernels still cannot cover it — and the fallback calls +the user's add and multiply through function pointers **once per +matrix entry**, which is a per-element interpreter sitting in the +innermost loop of every graph algorithm. ## The concepts, step by step ### Step 1 — semirings make the kernel space combinatorial -GraphBLAS expresses graph algorithms as sparse linear algebra over a -**semiring** — a user-chosen pair of operations (an "add" monoid and -a "multiply" op) standing in for the usual +/× of matrix multiply: -BFS uses (min, first), shortest paths (min, +), plain reachability -(or, and). `GrB_mxm` (masked sparse matrix multiply, the workhorse) -must therefore run for *any* semiring over *any* types, in any -storage format: +> **In:** a `GrB_mxm` call — the masked sparse matrix multiply that +> is GraphBLAS's workhorse. **Out:** the size of the space that +> would have to be precompiled, and hence the reason a JIT is +> structurally necessary here rather than merely fast. + +A **semiring** is a user-chosen pair of operations — an "add" +monoid and a "multiply" op — standing in for the usual `+`/`×` of +matrix multiply. BFS uses (min, first); shortest paths (min, +); +plain reachability (or, and). A **monoid** is an associative +operator with an identity, which is what lets the "add" side be +reassociated across threads. `GrB_mxm` must run for *any* semiring +over *any* types, in any storage format: ``` GrB_mxm(C, M, accum, semiring, A, B, desc) - semiring = (add monoid × multiply op) over any types - × A/B/C/M sparsity ∈ {sparse, hypersparse, bitmap, full} - × masked/complemented, accum present/absent, ... - ⇒ pre-compiling every combination: thousands of kernels ALREADY - shipped (the "factory" kernels) and still nowhere near coverage - — user-defined types/operators make it infinite. + semiring = (add monoid × multiply op) over any operand types + × C,M,A,B sparsity ∈ {sparse, hypersparse, bitmap, full} + × mask structural/valued, complemented or not + × operand types iso or not + × integer index widths 32 or 64 bit, independently for + C->p, C->h, C->i ← new in the v10-era tree; see Step 5 + + Count just the format axis for four matrices: + 4 sparsity formats ^ 4 matrices = 256 combinations + …before a single semiring or type is chosen. GraphBLAS ships + thousands of precompiled "factory" kernels and that is still + nowhere near coverage — and user-defined types and operators make + the space unbounded, because a user can define a type the library + has never seen. ``` -Why it matters: no finite build can enumerate this space, so the -choice is between a slow generic path and generating code on demand. +No finite build can enumerate this space. So the choice is between +a slow generic path (Step 2) and generating code on demand +(Step 3). That is the whole topic, arriving from a completely +different direction than a query compiler does. ### Step 2 — the generic fallback is this topic's villain at scalar grain -Without JIT, any non-factory combination falls back to a *generic* -kernel that calls the add and multiply ops through **function -pointers per entry** — every `z += a*b` in the inner loop becomes an -indirect call (~20 cycles) wrapping ~1 cycle of arithmetic. That is -a per-ELEMENT interpreter — the exact overhead this whole topic is -about, at the finest possible grain, and over a matrix with 10⁸ -nonzeros it runs 10⁸ times. Question 1 quantifies the gap. +> **In:** Step 1's uncovered combination. **Out:** the measured +> shape of the penalty — an indirect call *plus* an untyped memory +> round-trip per matrix entry — which is the number the JIT has to +> beat. + +Without a JIT, a non-factory combination falls back to a **generic** +kernel. The previous version of this guide pointed at +`Source/generic/` for it; at this pin **`Source/generic/` contains +only `GB_generic.h`**. The real generic mxm path is a set of 23-line +shims in `Source/mxm/GB_AxB_saxpy3_generic_*.c` over one 285-line +body: + +```c +// GraphBLAS/Source/mxm/factory/GB_AxB_saxpy_generic_method.c — the +// function pointers pulled out of the semiring once, 95-110 + 95 GrB_BinaryOp mult = semiring->multiply ; + 96 GrB_Monoid add = semiring->add ; +// ... 97-99: asserts that the ztypes agree ... + 100 GxB_binary_function fmult = mult->binop_function ; // NULL if positional + 101 GxB_index_binary_function fmult_idx = mult->idxbinop_function ; + 102 GxB_binary_function fadd = add->op->binop_function ; + 103 GB_Opcode opcode = mult->opcode ; + 104 + 105 size_t csize = C->type->size ; + 106 size_t asize = A_is_pattern ? 0 : A->type->size ; + 107 size_t bsize = B_is_pattern ? 0 : B->type->size ; + 108 + 109 size_t xsize = mult->xtype->size ; + 110 size_t ysize = mult->ytype->size ; +``` + +Lines 100 and 102 are the villain: `fmult` and `fadd` are function +*pointers*, resolved from the semiring at run time. Lines 105-110 +are the second half of the villain — the sizes are runtime values +too, so operands move as untyped bytes. Then the inner loop is +built out of macros that call through those pointers: + +```c +// GraphBLAS/Source/mxm/factory/GB_AxB_saxpy_generic_method.c — the +// per-entry operations, as macros over function pointers, 207-217 and 250 + 207 // Cx [p] += Hx [i] +// ... 208: #undef ... + 209 #define GB_CIJ_GATHER_UPDATE(p,i) fadd (GB_CX (p), GB_CX (p), GB_HX (i)) + 210 + 211 // Cx [p] += t +// ... 212: #undef ... + 213 #define GB_CIJ_UPDATE(p,t) fadd (GB_CX (p), GB_CX (p), t) + 214 + 215 // Hx [i] += t +// ... 216: #undef ... + 217 #define GB_HX_UPDATE(i,t) fadd (GB_HX (i), GB_HX (i), t) +// ... 219-249: generic C/Z type macros, then the flipxy variants ... + 250 #define GB_MULT(t, aik, bkj, i, k, j) fmult (t, bkj, aik) +``` + +**Correction of the cost model.** The previous version said the +fallback is "an indirect call (~20 cycles) wrapping ~1 cycle of +arithmetic". Look at line 209 again: `fadd` takes three `void *` +arguments — destination, and two sources — because the types are +not known at compile time. So each multiply-add is: + +``` + Per entry, GENERIC path (lines 209 / 250): + 1 indirect call through fmult (unpredictable target, + no inlining possible) + + operands passed BY ADDRESS as void* → the value must be in + memory, not a register + 1 indirect call through fadd + + another void* round-trip + + a typecast step if xtype != A->type (lines 109-110 exist + precisely for this) + + Per entry, JIT/factory path: + the semiring is #define'd, so `fadd`/`fmult` become the literal + operators, the accumulator stays in a register across the loop, + and the whole thing is one FMA the compiler can vectorize. + + The gap is therefore NOT "call overhead vs one FLOP". It is + "call + forced memory round-trip + no vectorization" vs "one + register-resident FMA in an unrolled loop". Question 1 asks you + to measure it rather than accept either estimate. +``` + +Over a matrix with 10⁸ nonzeros that runs 10⁸ times. This is the +exact overhead the whole topic is about, at the finest possible +grain — and note the parallel with topic 19's own measured lanes: +`interp` walks a tree and dispatches per node +(`experiments/src/interp.rs:8-16`), the generic kernel dispatches +per matrix entry. Same shape, different unit. ### Step 3 — the JIT grain: compile the specialization, not the query +> **In:** Step 2's per-entry dispatch. **Out:** the *unit of +> compilation* — a decision that determines how much caching can +> buy, and the reason a whole C-compiler invocation is affordable +> here and nowhere else in this topic. + GraphBLAS's move: when an uncovered combination arrives, write a -small C file that instantiates a kernel *template* -(Source/jit_kernels/) with `#define`s pinning the semiring, types, -and formats — then compile it into a real kernel, indistinguishable -from a factory one. The unit of compilation is the *kernel shape*, -not the user's query: two different graph queries using the same +small C file that instantiates a kernel *template* with `#define`s +pinning the semiring, types and formats, then compile it into a +real kernel, indistinguishable from a factory one. You can read the +source generator doing exactly that: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — writing the kernel's +// C source into the cache directory, 1997-2016 +1997 snprintf (GB_jit_temp, GB_jit_temp_allocated, "%s/c/%02x/%s.%s", +1998 GB_jit_cache_path, bucket, kernel_name, kernel_filetype) ; +1999 FILE *fp = fopen (GB_jit_temp, "w") ; +// ... 2000-2002: if the open succeeded ... +2003 GB_macrofy_preface (fp, kernel_name, +2004 GB_jit_C_preface, GB_jit_CUDA_preface, kcode, +2005 encoding->major, encoding->minor) ; +2006 // macrofy the kernel operators, types, and matrix formats +2007 GB_macrofy_family (fp, family, encoding->code, encoding->kcode, +2008 semiring, monoid, op, type1, type2, type3) ; +2009 // #include the kernel, renaming it for the PreJIT +2010 fprintf (fp, "#ifndef GB_JIT_RUNTIME\n" +2011 "#define GB_jit_kernel %s\n" +2012 "#define GB_jit_query %s_query\n" +2013 "#endif\n" +2014 "#include \"template/GB_jit_kernel_%s.%s\"\n", +2015 kernel_name, kernel_name, kname, +2016 kernel_filetype) ; +``` + +Line 2007 is the whole idea — "macrofy the kernel operators, types, +and matrix formats" writes the `#define`s — and line 2014 +`#include`s the template that consumes them. The templates live in +**`Source/jit_kernels/template/`** (about sixty +`GB_jit_kernel_*.c` files; `Source/jit_kernels/include/` holds two +headers). The previous version of this guide cited +`Source/jit_kernels/` without the `template/`. + +The unit of compilation is the *kernel shape*, not the user's +query: two completely different graph algorithms using the same semiring on the same formats share one kernel. That choice is what makes an enormous one-time compile cost rational — Step 4's ladder amortizes it across the lifetime of the machine, because the key -space is *small and stable* (type combos, not query texts). - -### Step 4 — the load ladder: four caches, each with a longer lifetime +space is small and stable (type combinations, not query texts). + +### Step 4 — the load ladder: three runtime levels, each longer-lived + +> **In:** Step 3's "generate on demand". **Out:** the lookup path a +> kernel request actually walks, with the cost and lifetime of each +> level named — the thing you must know to reason about a cold +> start. + +**Structural correction.** The previous version described "four +levels: PreJIT table, in-memory hash table, on-disk `.so`, C +compiler". At this pin there are **three** levels at run time, +because **PreJIT kernels are inserted into the same in-memory hash +table at initialization**, not probed separately: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — PreJIT harvest at init: +// the AOT kernels go into the SAME table, 413 and 596-617 + 413 GB_prejit (&nkernels, &Kernels, &Queries, &Names) ; + 414 + 415 for (int k = 0 ; k < nkernels ; k++) +// ... 416-595: recover each kernel's name, encoding, hash and suffix ... + 597 //------------------------------------------------------------------ + 598 // make sure this kernel is not a duplicate + 599 //------------------------------------------------------------------ + 600 + 601 int64_t k1 = -1, kk = -1 ; + 602 if (GB_jitifyer_lookup (hash, encoding, suffix, &k1, &kk) != NULL) +// ... 603-606: duplicate: ignore it ... + 609 // insert the PreJIT kernel in the hash table +// ... 610-611 ... + 612 if (!GB_jitifyer_insert (hash, encoding, suffix, NULL, dl_function, k)) +``` -A kernel request walks down four levels, each slower and -longer-lived than the last — and every level's hit means the levels -below never run: +Line 612 is decisive: a PreJIT kernel is `GB_jitifyer_insert`ed +into `GB_jit_table` exactly like a JIT-compiled one, distinguished +only by a non-negative `prejit_index`. So one hash probe covers +both. The real ladder: ```mermaid flowchart TD - E[encodify: problem → 64-bit hash + encoding\nGB_encodify_mxm.c:55-59] --> P{PreJIT table?\ncompiled into lib} - P -->|hit| RUN[call fn pointer] - P -->|miss| H{in-memory hash table?\nGB_jitifyer_lookup :2119} - H -->|hit| RUN - H -->|miss| D{.so in cache dir?\n~/.SuiteSparse/GrB.../} - D -->|hit| DL[dlopen :1937 → insert in table] --> RUN - D -->|miss| CC[write C source from template,\ninvoke C compiler, link .so\n:1677-1710 critical section] --> DL -``` - -```rust -// the load ladder: four caches, each with a longer lifetime -fn get_kernel(problem: &Mxm) -> KernelFn { - let (hash, enc) = encodify(problem); // SHAPE only — no data values - if let Some(f) = PREJIT.get(hash, &enc) { return f; } // in the binary - if let Some(f) = TABLE.lookup(hash, &enc) { return f; } // this process - if let Some(so) = cache_dir_probe(hash) { return dlopen_insert(so); } - critical_section(|| { // first time EVER: pay the compiler - write_c_from_template(&enc); // #defines into jit_kernels/ - invoke_cc_and_link(); // ~100 ms - 1 s, once per combo - dlopen_insert(so_path(hash)) - }) -} -``` - -Amortization horizon: the first `mxm` with a new semiring pays a -C-compiler invocation (~100 ms - 1 s); every later call in ANY -process pays a hash probe. Compare: postgres re-pays per query, -Umbra per query (µs), copy-and-patch per query (ns). GraphBLAS can -afford a huge one-time cost because the key space is small and -stable. + E["encodify: problem to a 64-bit hash + encoding
GB_encodify_mxm.c:58-77"] --> H{"in-memory hash table?
GB_jitifyer_lookup :2122
(holds PreJIT AND loaded JIT kernels)"} + H -->|hit| RUN[call the function pointer] + H -->|miss| D{".so already in the cache dir?
GB_jit_cache_path/lib/NN/"} + D -->|hit| DL["GB_file_dlopen :1937
then insert in the table"] --> RUN + D -->|miss| CC["macrofy C source :1997-2016
then GB_jitifyer_direct_compile :2043"] --> DL2["GB_file_dlopen :2050
then insert"] --> RUN +``` + +The probe itself is open addressing with linear probing, and it is +worth reading because it shows exactly what the key compares: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — GB_jitifyer_lookup's probe +// loop; the function is 2122-2171, this is its body, 2146-2167 +2146 for (uint64_t k = hash ; ; k++) +2147 { +2148 k = k & GB_jit_table_bits ; +2149 GB_jit_entry *e = &(GB_jit_table [k]) ; +2150 if (e->dl_function == NULL) +2151 { +2152 // found an empty entry, so the entry is not in the table +2153 return (NULL) ; +2154 } +2155 else if (e->hash == hash && +2156 e->encoding.code == encoding->code && +2157 e->encoding.kcode == encoding->kcode && +2158 e->encoding.suffix_len == suffix_len && +2159 (builtin || (memcmp (e->suffix, suffix, suffix_len) == 0))) +2160 { +// ... 2161-2166: read prejit_index atomically, hand back k ... +2167 return (e->dl_function) ; +2168 } +``` + +Lines 2155-2159 are the correctness condition of the entire cache: +a hash match is not enough — `code`, `kcode`, `suffix_len` and (for +user-defined semirings) the actual name bytes must all agree. The +hash narrows; the encoding decides. Note there is no eviction +anywhere in this loop or this file: line 2150's empty slot is the +only miss condition, so the table only grows. + +The table itself: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — the process-global table, 24-42 + 24 // The hash table is static and shared by all threads of the user application. + 25 // It is only visible inside this file. It starts out empty (NULL). Its size + 26 // is either zero (at the beginning), or a power of two (of size + 27 // GB_JITIFIER_INITIAL_SIZE or more). +// ... 29-34: the strings build filenames and compile commands; a smaller +// ... table under GBCOVER ... + 35 #define GB_JITIFIER_INITIAL_SIZE (32*1024) +// ... 36-37 ... + 38 static GB_jit_entry *GB_jit_table = NULL ; + 39 static int64_t GB_jit_table_size = 0 ; // always a power of 2 + 40 static uint64_t GB_jit_table_bits = 0 ; // hash mask (0xFFFF if size is 2^16) + 41 static int64_t GB_jit_table_populated = 0 ; + 42 static size_t GB_jit_table_allocated = 0 ; +``` + +Line 35's 32×1024 = 32,768 initial slots, and line 40's power-of-two +mask is what makes line 2148's `k & GB_jit_table_bits` a single AND +instead of a modulo. Question 3 is about why never evicting is fine +here. + +Now the amortization, with the levels priced: + +``` + Cost per level, order of magnitude: + hash probe (2146-2167) ~10-100 ns — a few cache lines + dlopen an existing .so (:1937) ~100 µs-1 ms — an OS operation + write C + fork a compiler + link (:1997-2050) + ~100 ms - 1 s — a PROCESS + + Ratio between the extremes: 1 s / 50 ns = 2 × 10^7. + A twenty-million-fold cost difference is only survivable if the + expensive level is hit essentially never. Suppose an application + uses 20 distinct semiring/format combinations and makes 10^6 mxm + calls: + compiles = 20 × 0.5 s = 10 s (once, ever — + the .so persists + across processes) + probes = 10^6 × 50 ns = 0.05 s + amortized compile per call = 10 s / 10^6 = 10 µs + …and on the SECOND run of the program the compile term is zero, + because level 2 (the on-disk .so) survives process exit. + + Compare the other systems in this topic: + PostgreSQL re-pays LLVM per query (ms, every time) + Umbra re-pays Flying Start per query (0.21 ms geomean) + copy-and-patch re-pays a memcpy per query (178 µs for TPC-H Q5) + GraphBLAS pays a C compiler once per SHAPE, forever + GraphBLAS can afford the most expensive compiler in the topic + precisely because its denominator is the largest. +``` ### Step 5 — the cache key: shape in, values out -A cache of compiled code is only sound if the key captures -*everything the generated code depends on* and nothing else. The -key (`GB_jit_encoding`, GB_encodify_mxm.c) is a packed bit-field: -kernel code, then `GB_enumify_mxm` packs semiring ops, types, -sparsity formats, mask/accum flags into `encoding->code` (:55-59). -User-defined ops add a name *suffix* (:16-18) since their semantics -aren't enumerable; hash = the lookup key, suffix disambiguates. -Crucially the key is the SHAPE with all data-dependent values -excluded — matrix contents, dimensions, sparsity *counts* don't -enter. This answers postgres-guide Q5, and getting it wrong in -either direction costs: include a value → cache miss per literal → -compile storm; omit a semantic input → wrong code served. - -### Step 6 — the compiler is literally `cc`, and the cache feeds back into the build - -No LLVM, no cranelift: write a `.c` file, shell out to the same -compiler that built the library (GB_jitifyer.c:59-71 stores -compiler+flags), `dlopen` the resulting `.so` (:1937 — -`dlopen`/`dlsym` being the OS's standard load-a-shared-library -mechanism). Crude and perfect for the amortization horizon: the C -optimizer gives factory-equal code, and the cache makes latency -irrelevant. The endgame is **PreJIT**: kernels harvested from a -JIT cache get compiled *into* the next binary release -(GB_jitifyer.c:299) — the JIT doubling as a build-time kernel -harvester, closing the loop between Step 4's slowest level and its -fastest. +> **In:** Step 4's ladder, which is only as sound as its key. +> **Out:** the exact contents of that key, and the two ways of +> getting it wrong — the reusable rule for any code cache, +> including M19's. + +A cache of compiled code is sound only if the key captures +*everything the generated code depends on* and nothing else. Here +is the whole key construction: + +```c +// GraphBLAS/Source/jitifyer/GB_encodify_mxm.c — problem to (encoding, hash); +// the file is 79 lines, this is its body, 46-77 + 46 if (semiring->hash == UINT64_MAX) + 47 { + 48 // cannot JIT this semiring + 49 memset (encoding, 0, sizeof (GB_jit_encoding)) ; + 50 (*suffix) = NULL ; + 51 return (UINT64_MAX) ; + 52 } +// ... 54-57: banner — "primary encoding of the problem" ... + 58 GB_encodify_kcode (encoding, kcode) ; + 59 GB_enumify_mxm (&encoding->code, C_iso, C_in_iso, C_sparsity, ctype, + 60 Cp_is_32, Cj_is_32, Ci_is_32, M, Mask_struct, Mask_comp, semiring, + 61 flipxy, A, B) ; +// ... 63-66: banner — "determine the suffix and its length" ... + 67 // if hash is zero, it denotes a builtin semiring + 68 uint64_t hash = semiring->hash ; + 69 encoding->suffix_len = (hash == 0) ? 0 : semiring->name_len ; + 70 (*suffix) = (hash == 0) ? NULL : semiring->name ; +// ... 72-75: banner — "compute the hash of the entire problem" ... + 76 hash = hash ^ GB_jitifyer_hash_encoding (encoding) ; + 77 return ((hash == 0 || hash == UINT64_MAX) ? GB_MAGIC : hash) ; +``` + +**Corrections.** The previous version anchored this at `:55-59` and +`:16-18`; the real encoding call is **`:58-61`**, the suffix logic +is **`:69-70`**, and the final hash is **`:76-77`** (`:16-18` is a +parameter declaration in the signature). It also listed `accum` +among the enumified inputs — **there is no `accum` in this key**. +Read the argument list at lines 59-61 and take it literally: + +``` + What GB_enumify_mxm actually packs into encoding->code (59-61): + C_iso, C_in_iso — is C a single repeated value? + C_sparsity, ctype — output format and type + Cp_is_32, Cj_is_32, Ci_is_32 + — the INTEGER WIDTH of C's pointer, hyperlist + and index arrays, independently 32 or 64 bit + M, Mask_struct, Mask_comp + — mask presence, structural-vs-valued, + complemented + semiring, flipxy — the operators, and whether x/y are swapped + A, B — operand formats and types + + Plus, separately: kcode (which kernel family) at :58, and for + user-defined semirings a NAME SUFFIX at :69-70. + + Why the three integer-width flags belong in the key: a kernel + compiled for 32-bit C->i indexes a different array type than one + compiled for 64-bit. Omit them and you serve a kernel that + misreads memory. This is a v10-era addition and a perfect example + of the rule below — a NEW code-generation input had to become a + NEW key component in the same commit. +``` + +Two more mechanisms in those 30 lines: + +- **Line 46**: `semiring->hash == UINT64_MAX` marks a semiring that + *cannot* be JIT'd (e.g. a user operator with no stringified + definition), and the function returns `UINT64_MAX` — a sentinel + the loader checks before doing anything else, sending the call + straight to Step 2's generic path. The JIT's failure mode is + slowness, never wrongness. +- **Lines 68-70**: built-in semirings have `hash == 0` and need no + suffix; user-defined ones carry their name, because their + *semantics* are not enumerable in a bit-field. The hash locates a + bucket; the suffix disambiguates within it — which is why line + 2159 `memcmp`s the name. + +The rule, stated so it transfers: **the key is the SHAPE, with all +data-dependent values excluded.** Matrix contents, dimensions and +nonzero counts do not enter. Getting it wrong costs in both +directions: + +``` + include a VALUE in the key → a distinct key per literal + → a cache miss per literal + → a COMPILE STORM (100 ms each) + omit a SEMANTIC input → a kernel compiled for one shape + → served for another + → WRONG ANSWERS, silently + + The asymmetry matters: the first failure is a performance + disaster you will notice in a profile; the second is a + correctness disaster you may not notice at all. When unsure, + over-include — a spurious key component costs recompiles, a + missing one costs correctness. +``` + +This is also the answer to the postgres guide's question about what +a query-compilation cache should be keyed on. + +### Step 6 — the compiler is literally `cc`, and the locking is coarse + +> **In:** Step 4's slowest level. **Out:** how the compile actually +> happens, and — correcting the previous version — what +> concurrency guarantee surrounds it. + +No LLVM, no cranelift: write a `.c` file (Step 3's `:1997-2016`), +shell out to the same compiler that built the library, `dlopen` the +resulting `.so`. The compiler and its flags are held as strings: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — the toolchain, as strings, 59-72 + 59 // name of the C compiler: + 60 static char *GB_jit_C_compiler = NULL ; +// ... 61-62 ... + 63 // flags for the C compiler: + 64 static char *GB_jit_C_flags = NULL ; +// ... 65-66 ... + 67 // link flags for the C compiler: + 68 static char *GB_jit_C_link_flags = NULL ; +// ... 69-70 ... + 71 // libraries to link against when using the direct compile/link: + 72 static char *GB_jit_C_libraries = NULL ; +``` + +and there are three ways to invoke it, selected at `:2029-2044`: +`GB_jitifyer_nvcc_compile` for CUDA kernels (`:2032`), +`GB_jitifyer_cmake_compile` if `GB_jit_use_cmake` (`:2038`), else +`GB_jitifyer_direct_compile` (`:2043`). The cmake toggle is at +`:44-49`, and line 48's comment reads "otherwise, default is to +skip cmake and compile directly" — MSVC is the only platform that +requires cmake (`:45-46`). Then `GB_file_dlopen` at `:2050` loads +the result, mirroring the fast-path `dlopen` at `:1937`. + +Crude, and perfect for the amortization horizon: the C optimizer +gives factory-equal code, and Step 4's arithmetic makes the latency +irrelevant. + +**Concurrency correction.** The previous version claimed the +critical section wraps "compile+insert only, with lookup +lock-free-ish before it", and asked in question 2 about two threads +both compiling with one insert winning benignly. **That is false at +this pin.** Read the entry point: + +```c +// GraphBLAS/Source/jitifyer/GB_jitifyer.c — GB_jitifyer_load's locking +// discipline; the function is 1576-1674, this is its tail, 1630-1673 +1630 if ((GB_jit_control == GxB_JIT_RUN) && +1631 (family != GB_jit_user_op_family) && +1632 (family != GB_jit_user_type_family)) +1633 { +// ... 1635-1638: banner ... +1639 int64_t k1 = -1, kk = -1 ; +1640 (*dl_function) = GB_jitifyer_lookup (hash, encoding, suffix, &k1, &kk) ; +// ... 1641-1644: k1 >= 0 means an unchecked PreJIT kernel — fall through +// ... to the critical section to validate it ... +1645 else if ((*dl_function) != NULL) +1646 { +1647 // found the kernel in the hash table +1648 return (GrB_SUCCESS) ; +1649 } +// ... 1650-1659: JIT is set to 'run', so nothing may be compiled or +// ... loaded: fall back to the generic kernel ... +1660 } +// ... 1662-1665: banner — "do the rest inside a critical section" ... +1666 GB_OPENMP_LOCK_SET (1) +1667 { +1668 info = GB_jitifyer_load2_worker (dl_function, family, kname, hash, +1669 encoding, suffix, semiring, monoid, op, type1, type2, type3) ; +1670 } +1671 GB_OPENMP_LOCK_UNSET (1) +``` + +The lock-free probe at line 1640 happens **only** when +`GB_jit_control == GxB_JIT_RUN` (line 1630) — a mode in which +nothing may be compiled or loaded at all, so the fast path is +"already-loaded kernels only, everything else goes generic" +(lines 1650-1659). In the **default** `GxB_JIT_ON` mode, control +falls straight past line 1660 and *the entire load — hash lookup +included — runs inside the OpenMP lock at 1666-1671*. So two +threads cannot both compile: the second blocks, then finds the +kernel already in the table when `GB_jitifyer_load2_worker` probes +at `:1710`. Coarse, serialized, and correct — the opposite of the +benign-race story. + +One more failure-handling detail worth carrying away: if the +compile fails, `:2060` sets `GB_jit_control = GxB_JIT_LOAD`, i.e. +the JIT **disables its own compile level** rather than retrying a +broken toolchain on every subsequent call. A cache whose expensive +level can fail needs exactly this. + +Finally, the endgame is **PreJIT**: kernels harvested from a JIT +cache get compiled *into* the next build of the library, and at +startup `GB_prejit` (`:413`) inserts them into the same hash table +(Step 4). The JIT doubles as a build-time kernel harvester, closing +the loop between the ladder's slowest level and its fastest. The +old anchor for this, `:299`, is a comment inside an `#else /* +NJIT */` block. ### Step 7 — what transfers to M19/FalkorDB -- FalkorDB's Delta matrices + custom semirings ride exactly this - machinery — a cold start on a new semiring stalls the first - query; consider warming the JIT cache at startup. -- M19's Cypher-expression JIT should copy the *two-level cache* - (in-memory hash + persist compiled artifacts keyed by expression - shape) rather than postgres's compile-every-time. -- The generic-kernel fallback is M19's interpreter fallback: same - contract — never fail, only be slower. +> **In:** Steps 2-6. **Out:** three design decisions for our own +> JIT, each with the GraphBLAS mechanism it is copied from. + +- **Warm the cache at startup.** FalkorDB's Delta matrices and + custom semirings ride exactly this machinery, so a cold start on + a new semiring stalls the *first* query by a full compiler + invocation (Step 4's ~100 ms-1 s). Since the key space is small + and enumerable (Step 1), you can issue tiny dummy `mxm` calls at + startup for the combinations you know you use, and pay it before + anyone is watching. +- **Copy the two-level cache, not postgres's compile-every-time.** + In-memory hash keyed by expression *shape*, plus persisted + compiled artifacts so a restart does not re-pay. GraphBLAS's + on-disk `.so` level (`:1935-1937`) is what makes the second run + of a program free; postgres has no equivalent, which is Step 4's + cost table in one sentence. +- **The generic kernel is M19's interpreter fallback.** Same + contract: never fail, only be slower. GraphBLAS enforces it + structurally — `GB_encodify_mxm.c:46-52` returns `UINT64_MAX` for + anything un-JIT-able and the loader sends it generic, and + `GB_jitifyer.c:2060` self-disables the compiler after a failure. + Our `Expr` JIT needs the same: any node it cannot compile routes + the whole expression to `interp`, and a cranelift error is a + fallback, not a query error. ## Where each step lives in the code +All anchors verified at GraphBLAS `1fd5475`. + | anchor | what it is | step | |---|---|---| -| Source/generic/ | the function-pointer-per-entry fallback | 2 | -| Source/jit_kernels/ | the kernel templates the JIT instantiates | 3 | -| GB_jitifyer.c:1565-1576 | `GB_jitifyer_load` — the full ladder | 4 | -| Source/jitifyer/GB_jitifyer.c:21-40 | the static hash table of loaded kernels | 4 | -| GB_jitifyer.c:2119 | `GB_jitifyer_lookup` — hash-table probe | 4 | -| GB_jitifyer.c:1677-1710 | load2_worker — compile path under a critical section | 4 | -| GB_jitifyer.c:1937, 2050 | `GB_file_dlopen` — load the compiled .so | 4, 6 | -| Source/jitifyer/GB_encodify_mxm.c:16-59 | problem → `GB_jit_encoding` + hash | 5 | -| GB_jitifyer.c:48 | direct compile/link vs cmake toggle | 6 | -| GB_control.h + "PreJIT" | ahead-of-time compiled kernel table | 6 | - -Reading order: `GB_jitifyer_load` (:1565) top to bottom — it IS the -Step 4 ladder — then `GB_encodify_mxm.c` for the key (Step 5), then -one template under `Source/jit_kernels/` next to its generic -counterpart under `Source/generic/` to see Steps 2–3 as a diff. +| `Source/mxm/GB_AxB_saxpy3_generic_*.c` | the generic mxm shims (23 lines each) | 2 | +| `Source/mxm/factory/GB_AxB_saxpy_generic_method.c:100-102` | `fmult`/`fmult_idx`/`fadd` — the function pointers | 2 | +| `…GB_AxB_saxpy_generic_method.c:105-110` | the runtime type sizes that force the `void*` round-trip | 2 | +| `…GB_AxB_saxpy_generic_method.c:209,213,217,250` | the per-entry macros that call through them | 2 | +| `Source/jit_kernels/template/` | ~60 `GB_jit_kernel_*.c` templates the JIT instantiates | 3 | +| `Source/jitifyer/GB_jitifyer.c:1997-2016` | macrofy the `#define`s and `#include` the template | 3 | +| `Source/jitifyer/GB_jitifyer.c:24-42` | the process-global hash table; 32K initial slots at `:35` | 4 | +| `Source/jitifyer/GB_jitifyer.c:413,596-617` | PreJIT harvest — inserted into the *same* table | 4, 6 | +| `Source/jitifyer/GB_jitifyer.c:1576-1674` | `GB_jitifyer_load` — the ladder's entry point | 4 | +| `Source/jitifyer/GB_jitifyer.c:2122-2171` | `GB_jitifyer_lookup` — open-addressed probe; the key comparison is `:2155-2159` | 4, 5 | +| `Source/jitifyer/GB_jitifyer.c:1935-1937` | on-disk `.so` probe + `GB_file_dlopen` | 4 | +| `Source/jitifyer/GB_jitifyer.c:2029-2050` | nvcc / cmake / direct compile, then `dlopen` | 6 | +| `Source/jitifyer/GB_jitifyer.c:1630-1671` | the locking discipline — lock-free probe only under `GxB_JIT_RUN` | 6 | +| `Source/jitifyer/GB_jitifyer.c:1680-1897` | `GB_jitifyer_load2_worker` — lookup `:1710`, PreJIT validation `:1715-1786` | 4, 6 | +| `Source/jitifyer/GB_jitifyer.c:44-49` | `GB_jit_use_cmake` — MSVC needs cmake, everyone else compiles directly | 6 | +| `Source/jitifyer/GB_jitifyer.c:59-72` | compiler / flags / link flags / libraries, as strings | 6 | +| `Source/jitifyer/GB_jitifyer.c:2060` | on compile failure, set `GB_jit_control = GxB_JIT_LOAD` — self-disable | 6 | +| `Source/jitifyer/GB_encodify_mxm.c:46-52` | the un-JIT-able early-out returning `UINT64_MAX` | 5 | +| `Source/jitifyer/GB_encodify_mxm.c:58-61` | `GB_encodify_kcode` + `GB_enumify_mxm` — the key's contents | 5 | +| `Source/jitifyer/GB_encodify_mxm.c:69-70` | the user-defined-semiring name suffix | 5 | +| `Source/jitifyer/GB_encodify_mxm.c:76-77` | the final XORed hash | 5 | + +**Anchor corrections against the previous version of this guide:** +`Source/generic/` → `Source/mxm/factory/GB_AxB_saxpy_generic_method.c` +(at this pin `Source/generic/` holds only `GB_generic.h`); +`Source/jit_kernels/` → `Source/jit_kernels/template/`; +`GB_encodify_mxm.c:55-59` → `:58-61` (and `:16-18` → `:69-70`); +`GB_jitifyer.c:2119` → `:2122` (2119 is the banner comment); +`GB_jitifyer.c:1565` → `:1576` (same reason); +`GB_jitifyer.c:21-40` → `:24-42`; +`GB_jitifyer.c:1677-1710` → the critical section is `:1666-1671` +around `GB_jitifyer_load2_worker`, whose own lookup is `:1710`; +`GB_jitifyer.c:299` → PreJIT harvest is `:413` and `:612`. + +Reading order: `GB_jitifyer_load` (`:1576-1674`) top to bottom — it +IS the Step 4 ladder, including the locking of Step 6 — then +`GB_encodify_mxm.c` end to end for the key (Step 5, only 79 lines), +then one template under `Source/jit_kernels/template/` next to +`Source/mxm/factory/GB_AxB_saxpy_generic_method.c` to see +Steps 2-3 as a diff. ## Questions for notes.md -1. Find the generic mxm path (function-pointer per multiply-add, - Source/generic/). Estimate its per-entry cost vs a JITed - `z += a*b` on f64 (call + load fn ptr vs 1 FMA) — does the - ratio match this topic's interpreter/compiled gaps (~10×)? -2. Why is the critical section (:1677-1710) around compile+insert - only, with lookup lock-free-ish before it — and what duplicate - work can two threads still do (both compile; one insert wins — - benign, same as Gunrock's lost CAS)? -3. The hash table is process-global and never evicts - (GB_jitifyer.c:24-40). Why is unbounded growth fine here but - would not be for a query-text-keyed cache (bounded key space — - count it for FalkorDB's actual semiring usage)? -4. PreJIT (:299): kernels harvested from the JIT cache get compiled - into the library. What's the copy-and-patch analogy (stencils = - AOT-compiled parametrized kernels), and where do the two differ - (holes patched at runtime vs full specialization)? +1. Read the generic mxm path + (`Source/mxm/factory/GB_AxB_saxpy_generic_method.c:100-102` + for the function pointers, `:209`/`:250` for the per-entry + macros). Estimate its per-entry cost against a JIT'd `z += a*b` + on f64, and be careful to count *both* penalties: the indirect + call and the `void*` operand round-trip forced by lines 105-110. + Does the ratio match this topic's own measured interpreter gaps + (`notes.md`: 6× at 7 nodes, 12× at 511)? +2. In the default `GxB_JIT_ON` mode the whole load runs inside + `GB_OPENMP_LOCK_SET(1)` (`:1666-1671`); the lock-free probe at + `:1640` applies only under `GxB_JIT_RUN` (`:1630`). Why is + serializing the *lookup* acceptable here, when it obviously + would not be for a per-query cache? (Hint: multiply Step 4's + probe cost by the number of `mxm` calls, then by the number of + *distinct shapes* — the lock is contended only on the second + number.) What would you have to change to make the fast path + lock-free in `GxB_JIT_ON` mode too? +3. The hash table is process-global and never evicts — the only + miss condition in the probe loop is an empty slot + (`:2150-2154`), and the table starts at 32,768 entries + (`:35`). Why is unbounded growth fine here but not for a + query-text-keyed cache? Count it: how many distinct + (semiring × format × type) combinations does FalkorDB actually + use? Multiply by `sizeof(GB_jit_entry)` and compare to a + query-text cache in a system with unique literals per query. +4. PreJIT (`:413`, `:612`): kernels harvested from a JIT cache get + compiled into the library and inserted into the same table at + startup. What is the copy-and-patch analogy (stencils = + AOT-compiled parametrized fragments — + `reading-umbra-tidy-tuples.md` Step 6), and where do the two + differ? Be precise about the axis: copy-and-patch patches + *holes* (literals, jump addresses, stack offsets) at runtime; + PreJIT ships a *fully specialized* kernel with nothing left to + patch. Which one can cover a shape it never saw at build time? 5. For M19: design the Cypher expression cache key. Which parts of - `WHERE n.age > $p AND n.name = 'x'` are shape vs parameter, and - what does getting this wrong cost (constant folded in → cache - miss per literal value → compile storm)? + `WHERE n.age > $p AND n.name = 'x'` are shape and which are + parameter? Apply Step 5's rule in both directions and price both + errors. Then ask the harder version: is the literal `'x'` shape + or value — and does your answer change if the JIT constant-folds + it into the emitted code? ## Done when +Answer each before unfolding it. + - [ ] You can explain why semirings make the kernel space combinatorial, and why that makes a JIT structurally necessary rather than merely fast. -- [ ] You can state the JIT grain: the specialization, not the query — and say why that grain makes caching effective. -- [ ] You can describe the four-level load ladder and the lifetime of each cache. -- [ ] You can state the cache key rule — shape in, values out — and why including values would defeat it. -- [ ] You can explain what PreJIT does and why harvesting from the cache feeds back into the build. + +
Answer + + `GrB_mxm` is parameterized by a semiring (any add monoid × any + multiply op), the operand and output types, four sparsity formats + for each of four matrices, mask presence/structural/complemented, + iso flags, `flipxy`, and — at this pin — independent 32-vs-64-bit + index widths for `C->p`, `C->h` and `C->i`. The format axis alone + is 4⁴ = 256 combinations. GraphBLAS already ships thousands of + precompiled factory kernels and still cannot cover it, and + user-defined types and operators make the space genuinely + unbounded: a user can define a type the library has never seen. + "Structurally necessary" rather than "fast" because the + alternative is not a slower compiled kernel — it is Step 2's + per-entry function-pointer path, which is a different asymptotic + class of overhead. + +
+ +- [ ] You can describe the generic fallback's per-entry cost precisely, including the part that is not the call. + +
Answer + + `Source/mxm/factory/GB_AxB_saxpy_generic_method.c:100-102` + extracts `fmult` and `fadd` as function pointers from the + semiring at run time; `:209`, `:213`, `:217` and `:250` define + the per-entry macros that call through them. The call is only + half the cost: because the types are runtime values (`:105-110` + reads `csize`, `asize`, `bsize`, `xsize`, `ysize` from the type + descriptors), the operands are passed **by address as `void *`** — + so every multiply-add forces the accumulator out to memory and + back, and nothing can be inlined, kept in a register across + iterations, or vectorized. The JIT'd kernel `#define`s the + operators and types, so the same loop becomes a register-resident + FMA the C compiler can unroll and vectorize. Estimating this as + "a ~20-cycle call around ~1 cycle of arithmetic" undercounts it. + +
+ +- [ ] You can state the JIT grain — the specialization, not the query — and say why that grain makes caching effective. + +
Answer + + The unit of compilation is the kernel *shape*: one + (kernel family × semiring × types × sparsity formats × index + widths) combination, which is exactly what + `GB_enumify_mxm` packs at `GB_encodify_mxm.c:59-61`. Two + unrelated graph algorithms using the same semiring on the same + formats share one compiled kernel. That makes the key space + small, stable, and *enumerable in advance* — which is why a + ~100 ms-1 s C-compiler invocation is affordable here when the + same cost would be absurd in postgres. The denominator is every + `mxm` call ever made with that shape, in every process, forever; + postgres's denominator is one query. + +
+ +- [ ] You can describe the load ladder and the lifetime of each level — and say how many levels there actually are. + +
Answer + + **Three** at run time, not four. (1) The in-memory hash table + (`:24-42`, probed at `:2146-2167`), which holds *both* PreJIT + kernels — inserted at init by the loop at `:413-617`, via + `GB_jitifyer_insert` at `:612` — and previously loaded JIT + kernels; lifetime: the process; cost: tens of nanoseconds. (2) + The on-disk `.so` under `GB_jit_cache_path/lib/NN/` + (`:1935-1937`), loaded with `GB_file_dlopen`; lifetime: the + machine, across process restarts; cost: hundreds of microseconds + to milliseconds. (3) Writing C source (`:1997-2016`) and forking + a compiler (`:2043`), then `dlopen` (`:2050`); lifetime: forever, + because it populates level 2; cost: 100 ms-1 s. The PreJIT table + is not a separate probe — that is the correction. + +
+ +- [ ] You can state the cache-key rule — shape in, values out — say what is in this key, and price both ways of getting it wrong. + +
Answer + + The key is `(kcode, encoding->code, suffix)` hashed at + `GB_encodify_mxm.c:76-77`, where `encoding->code` packs, from the + argument list at `:59-61`: `C_iso`, `C_in_iso`, `C_sparsity`, + `ctype`, `Cp_is_32`, `Cj_is_32`, `Ci_is_32`, mask + presence/`Mask_struct`/`Mask_comp`, the semiring, `flipxy`, and + A's and B's formats and types. **No `accum`** — that was wrong in + the previous version. User-defined semirings add a name suffix + (`:69-70`) because their semantics are not enumerable, and the + probe `memcmp`s it (`:2159`). Excluded: matrix contents, + dimensions, nonzero counts — every data-dependent value. + Including a value gives a distinct key per literal, hence a cache + miss per literal, hence a compile storm at ~100 ms each; omitting + a semantic input serves a kernel compiled for a different shape, + which is silent wrong answers. The three index-width flags are + the cautionary tale: a new codegen input had to become a new key + component. + +
+ +- [ ] You can explain what PreJIT does, why harvesting from the cache feeds back into the build, and where the JIT self-disables. + +
Answer + + PreJIT compiles kernels harvested from a JIT cache directly into + the next build of the library; at startup `GB_prejit` (`:413`) + hands them to a loop that computes each one's hash and encoding + and `GB_jitifyer_insert`s it into the ordinary hash table + (`:612`), skipping duplicates found by `GB_jitifyer_lookup` + (`:602`). So yesterday's slowest level becomes today's fastest, + with no code path difference at the call site — only a + non-negative `prejit_index` (`:2164`) marking kernels that still + need validation. The self-disable is `:2060`: if a compile fails, + `GB_jit_control` drops to `GxB_JIT_LOAD`, so the library stops + trying to compile rather than forking a broken toolchain on every + subsequent call, and everything falls back to the generic kernel. + Together with the `UINT64_MAX` early-out at + `GB_encodify_mxm.c:46-52`, that is the "never fail, only be + slower" contract enforced structurally. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your Cypher expression cache key design. +
Answer + + The question-5 trap worth having noticed: `$p` is obviously a + parameter and `n.age` obviously shape, but the inline literal + `'x'` is ambiguous *until you decide whether the JIT folds it + into the emitted code*. If it does, the literal is part of the + generated machine code and therefore part of the shape — and a + workload with unique literals per query becomes a compile storm. + If it does not (the literal is loaded from a side table at run + time, as `$p` is), the key stays small and the generated code is + marginally worse. GraphBLAS makes the same choice explicitly: + matrix *contents* never enter the key, only formats and + operators. Write down which you chose and what it costs. + +
+ ## References -**Code** -- [GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) — - `Source/jitifyer/` (GB_jitifyer.c is the machine, - GB_encodify_mxm.c the cache key) and `Source/jit_kernels/` (the - templates the JIT instantiates); `GB_control.h` for the PreJIT - table +**Code** — all anchors verified at GraphBLAS `1fd5475` + +| file | what to read | +|---|---| +| `Source/jitifyer/GB_jitifyer.c` | 2,780 lines; read `GB_jitifyer_load` `:1576-1674` first — it is the whole ladder plus the locking — then `GB_jitifyer_load2_worker` `:1680-1897`, `GB_jitifyer_load_worker` `:1905+` for the compile path, `GB_jitifyer_lookup` `:2122-2171` for the probe | +| `Source/jitifyer/GB_encodify_mxm.c` | 79 lines, read all of it — the cache key in one function | +| `Source/mxm/factory/GB_AxB_saxpy_generic_method.c` | 285 lines — the generic fallback; `:100-110` and `:209-250` are the cost | +| `Source/jit_kernels/template/` | ~60 `GB_jit_kernel_*.c` — the templates instantiated at `GB_jitifyer.c:2014` | +| `Source/mxm/GB_AxB_saxpy3_generic_*.c` | 23-line shims that select a variant of the generic method | + +Fetch without a clone: +`python3 tools/pinned-source.py show GraphBLAS Source/jitifyer/GB_encodify_mxm.c`. + +**Elsewhere in this repo** +- `reading-postgres-jit.md` — compile-per-query with no cache at + all, and the estimate-based gate this design has no need for +- `reading-umbra-tidy-tuples.md` — Step 6's copy-and-patch + comparison (stencils, holes, supernodes) for question 4 +- `reading-neumann-vldb11.md` — why a specialized kernel beats a + dispatching one, argued for query pipelines rather than matrix + entries +- `experiments/src/interp.rs:8-16` — our own per-node dispatcher, + the same villain as Step 2 at a different grain +- `notes.md` — the measured interpreter/vectorized gaps question 1 + compares against diff --git a/topics/19-jit/reading-neumann-vldb11.md b/topics/19-jit/reading-neumann-vldb11.md index 2ce5af7..e2a219c 100644 --- a/topics/19-jit/reading-neumann-vldb11.md +++ b/topics/19-jit/reading-neumann-vldb11.md @@ -10,6 +10,14 @@ builds the paper's five concepts one at a time — where iterator overhead actually comes from, what a pipeline is, how a tree walk generates a flat loop — then hands you the reading route. +**Which paper, which numbers.** Thomas Neumann, "Efficiently +Compiling Efficient Query Plans for Modern Hardware", PVLDB 4(9), +pp. 539–550, 2011. Every figure quoted below carries the table it +came from. The hardware is a **dual Intel X5570 quad-core, 64 GB, +RHEL 5.4, gcc 4.5.2, LLVM 2.8, single-threaded** (§6, first +paragraph) — 2011 silicon and a *fifteen-year-old LLVM*. Ratios +survive; absolute milliseconds do not. + ## The problem in one sentence In the iterator model, producing ONE tuple costs a virtual call plus @@ -20,7 +28,13 @@ generating a fresh loop of machine code per query. ## The concepts, step by step -### Step 1 — why iterators lose (the paper's §2, topic 11 recap) +### Step 1 — why iterators lose (the paper's §1, topic 11 recap) + +> **In:** a query plan as a tree of operators, each exposing +> `next()`. **Out:** an instruction budget per tuple per operator, +> and the name for the thing that eats it — the indirect call. +> Nothing is generated yet; this step is the accounting that +> motivates everything after it. The Volcano/iterator model runs a query plan as a tree of operators, each exposing `next()` — "give me your next tuple" — so the plan @@ -41,27 +55,107 @@ because which operator is downstream is only known at runtime) costs Topic 11's vectorization divides that constant by 1024; this paper's move is to make it zero. +The paper's own accounting is in §1, and it is worth quoting the +structure because two of its three charges are *not* about the call +instruction: + +1. `next()` "will be called several million times" for a query. +2. The call is virtual or through a function pointer, so it "is + even more expensive than a regular call **and degrades the branch + prediction**". +3. Each operator keeps enough bookkeeping to *resume* mid-scan — + the paper's example is a compressed table scan that must + remember where it stopped — which is "bad code locality and + complex book-keeping". + +Charge 3 is the one this topic keeps circling back to. SQLite's VDBE +pays it explicitly and cheerfully (`reading-sqlite-vdbe.md`: one +integer in a register *is* a coroutine's whole resumption state, +`src/vdbe.c:1269-1272`). Compiled code does not pay it because a +generated loop does not have to be resumable — it runs to +completion. + +**Corrected pointer.** This material is in **§1 (Introduction)**, +not §2. §2 of the paper is RELATED WORK. The old version of this +guide sent you to the wrong section for its own headline argument. + ### Step 2 — the deeper cost: operator boundaries are DATA boundaries -The paper's Figure 1 point: in Volcano, a tuple physically travels — -each operator reads it from memory, works, and hands a pointer up, -so the tuple visits memory between every pair of operators. The -alternative: if the code for scan, filter, and join is fused into -one loop, the current tuple's fields live in **CPU registers** (the -~16 general-purpose + 32 vector slots that cost 0 cycles to access) -from the moment the scan loads them to the moment the pipeline ends. -No loads, no stores, no cache traffic for intermediate hops. That is -the performance prize the whole paper is engineered around — and its -limit is register count (question 4). +> **In:** Step 1's per-operator dispatch cost. **Out:** the reason +> dispatch is the *smaller* half of the bill — a tuple's physical +> trip through memory at every operator boundary — and the register +> budget that caps the fix. + +In Volcano, a tuple physically travels — each operator reads it from +memory, works, and hands a pointer up, so the tuple visits memory +between every pair of operators. The alternative: if the code for +scan, filter, and join is fused into one loop, the current tuple's +fields live in **CPU registers** (the ~16 general-purpose + 32 +vector slots that cost 0 cycles to access) from the moment the scan +loads them to the moment the pipeline ends. No loads, no stores, no +cache traffic for intermediate hops. That is the performance prize +the whole paper is engineered around — and its limit is register +count (question 4). + +**Figure 1, attributed honestly.** The paper opens with a figure +comparing hand-written C++ against execution engines on TPC-H Q1. +It is **reproduced from reference [16]** (Boncz/Zukowski/Nes, +MonetDB/X100, CIDR'05) — Neumann did not measure it. Cite it as +motivation, never as this paper's evidence. + +The evidence for the register claim that *is* this paper's own +measurement arrives in §6.2, Table 3 (callgrind 3.6.0, TPC-CH Q1, +HyPer+LLVM vs MonetDB): + +| counter, Q1 | HyPer + LLVM | MonetDB | ratio | +|---|---|---|---| +| instructions executed | 132 million | 1,184 million | **9.0×** | +| branches | 19,765,048 | 144,557,672 | 7.3× | +| L1 instruction-cache misses | 2,793 | 187,471 | **67×** | +| L1 data-cache misses | 1,764,937 | 7,545,432 | 4.3× | + +Nine times fewer instructions retired for the same answer is the +whole thesis in one number. The instruction-cache figure is the +"small code fragments working on large amounts of data in tight +loops" claim of §3.1 made visible. + +**Report the counter-example too.** Table 3's Q2 row has HyPer+LLVM +*losing* on branch mispredictions: **6,581,223 vs MonetDB's +3,891,827**. The paper prints it without comment. A guide that only +quotes the 67× is quoting a sales deck. ### Step 3 — pipelines and pipeline breakers (the core vocabulary) +> **In:** Step 2's "keep the tuple in registers" goal. **Out:** the +> two words that turn that goal into a plan-cutting rule, and the +> boundary Umbra later reuses for swapping code versions +> mid-query. + A **pipeline** is a maximal stretch of a query plan through which a -tuple can flow without being parked in a data structure. A -**pipeline breaker** is any operator that must *materialize* — see -all its input before emitting anything: a hash-join build, a sort, a -group-by table. Breakers cut the plan into pipelines, and each -pipeline becomes exactly one generated loop: +tuple can flow without being parked in a data structure. + +The paper defines the cut point twice, and the difference matters +(§3.1, first paragraph — the authors flag it themselves as "more +restrictive than in standard database systems"): + +- A **pipeline breaker** for a given input side is an operator that + "takes an incoming tuple **out of the CPU registers**." +- A **full pipeline breaker** is one that "**materializes all** + incoming tuples from this side before continuing processing." + +Standard database usage — and the previous version of this guide — +uses "pipeline breaker" to mean the *full* one: a hash-join build, a +sort, a group-by table. Neumann's weaker definition deliberately +also catches spilling a tuple to memory at all. That is why the +paper can say, in the same section, that "the block-oriented +execution models have fewer passes across function boundaries, but +they clearly also break the pipeline as they produce batches of +tuples beyond register capacity." **Vectorized execution is a +pipeline breaker under this definition.** Topic 11's whole model is +on the wrong side of the line — which is precisely the fight VLDB'18 +re-litigates (Step 6). + +Breakers cut the plan into pipelines: ``` ⋈ (hash) @@ -76,26 +170,54 @@ memory anyway — so it is the natural boundary of compilation, and (later, in Umbra) the natural boundary for swapping code versions mid-query. Question 1 below asks you to do this for a Cypher plan. +**One pipeline is not one function.** §4.2 states the limit +explicitly: "it is not possible or even desirable to compile a +complex query into a single function." Two reasons, both worth +carrying: (a) LLVM code calls back into C++ that takes over control +flow — an external sort produces runs in LLVM but drives the merge +from C++; (b) inlining everything is exponential — "outer joins +will call their consumers in two different situations", so a +cascade of outer joins doubles the emitted code per level. HyPer +therefore defines *functions within LLVM* and calls them, with one +rule: "the hot path does not cross a function boundary." + ### Step 4 — produce/consume: a tree walk that emits a flat loop +> **In:** Step 3's pipelines. **Out:** the two-method interface that +> converts a plan tree into flat control flow, and the crucial +> caveat that neither method exists at runtime. + The code generator gives every operator two methods: `produce()` — -"emit code that produces your rows" — and `consume()` — "emit the -code that receives one row from your child". The generator recurses -through the plan tree *once at compile time*; what it emits has no -tree left in it, just nested control flow: +"emit code that produces your rows" — and +`consume(attributes, source)` — "emit the code that receives one row +from `source`". The generator recurses through the plan tree *once +at compile time*; what it emits has no tree left in it, just nested +control flow: ``` produce(op): "generate code that produces op's rows" - consume(op, source): "generate code receiving one row from source" + consume(op, attributes, source): "generate code receiving one row" scan.produce() → emit: for row in table { filter.consume() } filter.consume() → emit: if p(row) { join.consume() } join.consume(build) → emit: ht.insert(row) ``` +The paper is emphatic on a point every reimplementation gets wrong +at least once (§3.2, final paragraph): "**this produce/consume +interface is only a mental model. These functions do not exist +explicitly, they are only used by the code generation.**" There is +no `produce` in the generated program and no vtable at runtime. The +recursion happens once, in the compiler, and its only output is +text/IR. + ```rust -// the codegen walk: each operator knows how to PRODUCE rows and how to -// CONSUME one row from its child — the emitted code is one flat loop +// ILLUSTRATION — not quoted from any pinned source. This is the mental +// model of §3.2 / Figure 5 written as Rust. The real tree walk you can +// read and run is the tree-walking interpreter at +// experiments/src/interp.rs:8-16, which is what this replaces: interp.rs +// dispatches once per node PER ROW; a produce/consume walk dispatches +// once per node, total, at compile time. fn produce(op: &Op, g: &mut Codegen) { match op { Scan(t) => { g.emit("for row in {t} {"); consume(parent(op), g); g.emit("}"); } @@ -128,48 +250,205 @@ flowchart LR end ``` +Figure 4 of the paper is the output of applying Figure 5's rules to +Figure 3's plan, and it is four flat loop nests with no operator +structure left — worth reading side by side until the mapping is +mechanical. + ### Step 5 — what they compile WITH: the LLVM cocktail, and the latency seed +> **In:** Step 4's emitted control flow, still abstract. **Out:** a +> concrete target language (LLVM IR), the rule for what is *not* +> generated, and the compile-time number that starts this topic's +> arms race. + HyPer emits **LLVM IR** (the intermediate representation of the LLVM compiler toolkit — typed, portable assembly that LLVM optimizes and -lowers to machine code) rather than C source — they measure C -compiler latency as *seconds* per query. Not everything is -generated: complex operator logic lives in precompiled C++, and the -generated IR calls into it — the "cocktail". The engineering rule: -generated code should be branch-predictable and keep attributes in -registers; complex logic goes in precompiled C++ called from IR. - -Even so, LLVM -O3 on big queries costs **10–100 ms** — the number -that spawns Umbra's Tidy Tuples (reading-umbra-tidy-tuples.md) and -the entire compile-latency arms race this topic tracks. +lowers to machine code) rather than C source. §4.1 gives three +reasons, in the authors' order: an optimizing C++ compiler is "really +slow, compiling a complex query could take multiple seconds"; C++ +"does not offer total control over the generated code — in +particular, overflow flags etc. are unavailable"; and LLVM IR is +strongly typed, which "caught many bugs that were hidden in our +original textual C++ code generation." + +Not everything is generated. §4.1's metaphor (Figure 6): the +precompiled C++ is the **cogwheels**, the generated LLVM is the +**chain** linking them. Complex data-structure management, spilling, +index traversal — C++. Tuple access, filtering, materialization into +a hash table — generated IR. The rule, stated as a rule: "**the hot +path, i.e., the code that is executed for 99% of the tuples, is pure +LLVM.**" Calling C++ occasionally (a new page, an allocation) is +fine; the cost is spilling registers to a cache-hot stack, negligible +once but "if this is done millions of times it becomes noticeable." + +**The compile-latency number, corrected.** §4.1 says LLVM "usually +requires only a few milliseconds for query compilation, while C or +C++ compilers would need seconds." The measurement backing that is +Table 2, and it is stronger than the prose: for TPC-CH Q1–Q5, +**LLVM compile time is 16, 41, 30, 16, and 34 ms**, against C++ +compile times of **1556, 2367, 1976, 2214, and 2592 ms** — 1.6 to +2.6 *seconds* per query. + +The previous version of this guide said "LLVM -O3 on big queries +costs 10–100 ms". Two things are wrong with it: the paper never runs +LLVM at `-O3` (HyPer used LLVM 2.8's JIT), and 10–100 ms is not a +figure in the paper. The honest sentence is: **on 2011 hardware with +LLVM 2.8, HyPer's LLVM compile times on TPC-CH Q1–Q5 were 16–41 ms +(Table 2), and C++ was 40–140× slower to compile.** The number that +grows into a crisis is not this one — it is what happens to LLVM +when queries get big, which `reading-umbra-tidy-tuples.md` measures +at 2000 joins (LLVM: 150 s). ### Step 6 — the numbers (2011 hardware, directionally durable) -- TPC-H vs Volcano-style: ~2-10× faster per query -- vs vectorized (VectorWise): usually faster but same ballpark — - the honest comparison arrives in VLDB '18 (README §7) -- compile time: tens of ms with LLVM even then +> **In:** Steps 1–5's mechanism. **Out:** the actual measured +> speedups with their baselines named, including the two places +> compilation does *not* win — which is the honest case for the +> rest of this topic. + +**Table 1, OLTP (TPC-C, 12 warehouses, single-threaded):** + +| | transactions/s | total compile time | +|---|---|---| +| HyPer + C++ | 161,794 | 16.53 s | +| HyPer + LLVM | 169,491 | **0.81 s** | + +Read that row carefully. On OLTP the generated *code* is only 4.8% +faster (169,491 / 161,794 = 1.048) — the paper explains why: most +TPC-C transactions touch fewer than 30 tuples, so there is no +per-tuple overhead to amortize. What LLVM buys on OLTP is not +throughput, it is **20× less compile time** (16.53 / 0.81 = 20.4). +Compilation strategy is a *latency* decision here, not a throughput +one. That is the whole seed of topics 19's second half. + +**Table 2, OLAP (TPC-CH Q1–Q5, milliseconds, warm prepared queries):** + +| | Q1 | Q2 | Q3 | Q4 | Q5 | +|---|---|---|---|---|---| +| HyPer + LLVM (exec) | **35** | **125** | **80** | **117** | **1105** | +| HyPer + LLVM (compile) | 16 | 41 | 30 | 16 | 34 | +| HyPer + C++ (exec) | 142 | 374 | 141 | 203 | 1416 | +| VectorWise 1.0 | 98 | – | 257 | 436 | 1107 | +| MonetDB 1.36.5 | 72 | 218 | 112 | 8168 | 12028 | +| "DB X" (commercial, disk-based) | 4221 | 6555 | 16410 | 3830 | 15212 | + +**Corrected headline.** The previous version of this guide said +"~2-10× faster per query" against "Volcano-style". There is no +Volcano-style baseline in this paper, and the range is wrong in both +directions. Do the division yourself: + +``` + vs VectorWise (vectorized, the real rival): + Q1 98 / 35 = 2.8× + Q3 257 / 80 = 3.2× + Q4 436 / 117 = 3.7× + Q5 1107/1105 = 1.002× ← a dead tie + vs MonetDB (column-at-a-time, full materialization): + Q1 2.1× Q2 1.7× Q3 1.4× Q4 69.8× Q5 10.9× + vs "DB X" (disk-based commercial): + Q3 16410 / 80 = 205× ← a different storage architecture, + not a code-generation result +``` + +The paper's own summary is "frequently another factor **2–4** +faster" than the fast in-memory systems. So the honest three-line +version: + +- **2–4× against a well-engineered vectorized engine** on scan- and + aggregation-heavy queries; +- **1.00× on Q5**, the join-dominated one — the win vanishes exactly + where the work becomes memory-bound rather than compute-bound; +- **~200× against a disk-based system**, which measures storage, not + compilation, and should never be quoted as a JIT number. + +Q5's tie is the single most useful number in the paper for this +topic, because it is the seven-years-early preview of VLDB'18 and of +why DuckDB ships no JIT (README §7). It also matches this topic's +own measurement from the other direction: the vectorized lane in +`notes.md` beats the interpreter by 6× at 7 nodes and 12× at 511, so +the interpretation overhead a JIT would remove is *already gone* +before the JIT arrives. The durable reading: compilation beats *tuple-at-a-time interpretation* by a lot, beats *vectorization* by a little or not at all — so the argument for a JIT must be made against topic 11, not against a strawman tree-walker. +### Step 7 — the arithmetic: when is compiling worth it? + +> **In:** Table 1's and Table 2's compile times, plus this topic's +> own measured per-row rates from `notes.md`. **Out:** a break-even +> row count you compute, and the reason the answer is different for +> OLTP and OLAP. + +Compilation buys a lower per-row cost and charges a fixed fee up +front. The break-even is one division: + +``` + rows_breakeven = compile_time / (per_row_slow − per_row_fast) + + Case A — Table 1's OLTP transaction (HyPer, LLVM 2.8, 2011): + compile_time = 0.81 s / (number of prepared statements) + The paper does not need this division at all: TPC-C statements + are PREPARED once and run millions of times, so compile time is + divided by ~10^6 and vanishes. This is why 20× less compile time + showed up as +4.8% throughput and nothing else. + + Case B — this topic's own bench (notes.md, Apple M3 Pro, depth 8, + 511 nodes, N_COLS=4): + interpreter = 0.95 M rows/s → 1 / 0.95e6 = 1.053 µs/row + vectorized = 11.8 M rows/s → 1 / 11.8e6 = 0.0847 µs/row + saving = 1.053 − 0.0847 = 0.968 µs/row + + If a cranelift compile of that 511-node tree costs 500 µs, then + against the INTERPRETER: + rows = 500 / 0.968 = 516 rows. + Against the VECTORIZED lane it is a different subtraction — you + need the JIT's own per-row rate, which is exactly what M19 asks + you to measure. Predict it in notes.md before you run it. +``` + +Case A and Case B differ by six orders of magnitude in break-even +rows, from the *same* mechanism, because of one variable the formula +hides: **how many times the compiled artifact is reused.** Prepared +statements reuse forever; ad-hoc analytics reuse once. Every system +in this topic is an answer to that variable — Postgres gates on a +cost estimate (`reading-postgres-jit.md`), Umbra makes the fee +almost zero (`reading-umbra-tidy-tuples.md`), GraphBLAS caches the +artifact on disk across process lifetimes +(`reading-graphblas-jit.md`). + ## How to read the paper (with the concepts in hand) -Read the whole thing — it's short. - -- **§2 — the argument.** Steps 1–2: the per-tuple cost accounting - and Figure 1's data-boundary point. You already have the - vocabulary; verify the claims against topic 11's measurements. -- **§3 — produce/consume.** Steps 3–4 in the authors' words. Trace - their worked example until you can predict, for each operator, - what its produce/consume emit — then do question 1's Cypher plan - from memory. -- **§4 — the LLVM "cocktail".** Step 5. Note which parts of the - engine stay precompiled and why the boundary is a function call — - the same boundary M19's stub draws between generated CLIF and - precompiled Rust. +Read the whole thing — it's twelve pages. + +- **§1 (Introduction) — the argument.** Steps 1–2: the per-tuple + cost accounting, the three charges against `next()`, and Figure + 1's data-boundary point. Remember Figure 1 is reproduced from + [16], not measured here. Verify the claims against topic 11's + measurements. (**Not §2** — that is Related Work.) +- **§3 (The Query Compiler) — produce/consume.** Steps 3–4 in the + authors' words. §3.1 is the pipeline-breaker definition; §3.2 is + the interface. Read Figures 2 → 3 → 5 → 4 in that order: query, + plan, translation rules, emitted code. Apply Figure 5's rules to + Figure 3 by hand until you reproduce Figure 4, then do question + 1's Cypher plan from memory. +- **§4 (Code Generation) — the LLVM "cocktail".** Step 5. §4.1 is + Figure 6's cogwheels-and-chain and the 99%-hot-path rule; §4.2 is + the "not one function" limit and the outer-join code-explosion + argument. Note which parts of the engine stay precompiled and why + the boundary is a function call — the same boundary M19's stub + draws between generated CLIF and precompiled Rust + (`experiments/src/jit.rs:20-21` makes the ownership half of that + boundary explicit). +- **§5 (Advanced Parallelization)** — skim, but notice that the + paper already anticipates SIMD *inside* the compiled pipeline + ("as long as we can keep the whole block in registers", using + LLVM's vector types). The compiled-vs-vectorized dichotomy was + never as clean as the slogan. +- **§6 (Evaluation)** — Tables 1, 2, 3 with Step 6 open beside you. + Do the divisions. Find Q5. Find Table 3's Q2 mispredict row. - Then skim Kersten et al. VLDB '18 (References) for the compiled-vs-vectorized rematch question 5 leans on. @@ -179,36 +458,199 @@ Read the whole thing — it's short. `MATCH (a)-[:R]->(b) WHERE a.x < 10 RETURN b.y, count(*)`. Which operators break the pipeline, and what does M19's *expression-only* JIT compile vs what produce/consume would? + Answer it twice — once with the standard "full pipeline breaker" + definition and once with §3.1's stricter register-eviction one. + Do the two answers differ? 2. Why does push-based codegen produce ONE loop where pull-based codegen can't — what forces materialization of control state in pull (the resumability the VDBE gets from bytecode, coroutines)? + Ground it: `src/vdbe.c:1264-1274` is 11 lines of resumption state + that a compiled loop does not need at all. 3. The "cocktail" rule: which parts of our jit_bench expression executor belong in precompiled Rust vs generated CLIF, and why - is the boundary a function call in both HyPer and our stub? + is the boundary a function call in both HyPer and our stub? Use + §4.1's 99% test as the criterion, not taste. 4. Registers vs L1: the paper claims tuple-in-registers across a pipeline. With 16 GP + 32 vector registers, how wide can a tuple - get before this claim quietly dies (spills)? + get before this claim quietly dies (spills)? §3.1 admits the + problem ("a single tuple might already be too large to fit into + the available CPU registers") and defers it to §4 — check + whether §4 actually answers it. 5. VLDB '18's result — vectorized wins hash-probe-heavy queries via memory parallelism. Explain with topic 13's MLP argument: why does one-tuple-at-a-time compiled code serialize cache misses, and what did HyPer add to fix it (group prefetching / SIMD probe - batching)? + batching)? Then connect it to Table 2's Q5 tie: was the 2018 + result already visible in 2011? ## Done when +Answer each before unfolding it. + - [ ] You can explain why operator boundaries are data boundaries, and why that is the deeper cost than dispatch. + +
Answer + + Dispatch costs an indirect call (~20+ cycles mispredicted) per + tuple per operator. The boundary costs a *store plus a load* of + every live attribute, because the calling convention cannot keep + a tuple in registers across an unknown callee — so the tuple is + spilled and refetched at each hop. Dispatch is a fixed constant + you can amortize (vectorization divides it by 1024). The memory + round-trip scales with tuple width and cannot be amortized by + batching — batching only moves the spill from L1 to a larger + buffer. Neumann's measurement of the combined effect is Table 3: + **132 million instructions vs MonetDB's 1,184 million on Q1 + (9.0×)**, with L1-I misses at 2,793 vs 187,471 (67×). + +
+ - [ ] You can define pipelines and pipeline breakers and identify both in a plan you draw yourself. + +
Answer + + A pipeline is a maximal run of the plan a tuple crosses without + being parked. §3.1 gives *two* breaker definitions: a **pipeline + breaker** takes a tuple out of the CPU registers; a **full + pipeline breaker** materializes all its input from that side + first. Standard usage means the second. Under the first, + vectorized execution is itself a breaker — the paper says so + directly about "block-oriented execution models". In the hash-join + plan above, the build side and the group-by table are full + breakers; a filter is not a breaker at all; a sort is. + +
+ - [ ] You can explain how produce/consume turns a tree walk into one flat loop, and why push-based codegen produces one loop where pull-based produces several. + +
Answer + + `produce()` recurses *down* asking each child to emit its own row + source; `consume(attributes, source)` emits the code that runs on + one row and then calls its own parent's `consume`. The recursion + is entirely at compile time, so the emitted program has the scan + loop outermost and every downstream operator inlined in its body + — no tree, no calls. Pull cannot do this because a `next()` must + be able to *return* and later resume, which forces the operator's + loop counters and cursor state into memory; push runs each tuple + to the end of the pipeline before touching the next, so no + resumption state exists. Caveat from §4.2: one *pipeline* is not + necessarily one *function* — outer joins would blow up + exponentially if fully inlined, so HyPer emits LLVM functions and + only guarantees that the 99% hot path stays inside one. + +
+ - [ ] You can state the LLVM cocktail rule and say which parts of an expression should stay interpreted. + +
Answer + + §4.1: precompiled C++ is the cogwheels, generated LLVM is the + chain; "the hot path, i.e., the code that is executed for 99% of + the tuples, is pure LLVM". So: anything executed once per page, + once per allocation, once per operator setup, or that needs + complex data-structure logic stays precompiled and is *called*. + Anything on the per-tuple path is generated. Applied to a JIT'd + expression: arithmetic, comparisons and column loads are + generated; string collation, regex, `NULL`-heavy generic + fallbacks, and anything requiring an allocator stay as calls into + precompiled code. Postgres draws exactly this line — + `llvmjit_expr.c` generates the common opcodes and falls back to + calling the interpreter's C function otherwise. + +
+ +- [ ] You can compute the break-even row count for a compiled expression from a compile time and two per-row rates, and explain why the answer differs by six orders of magnitude between OLTP and OLAP. + +
Answer + + `rows = compile_time / (per_row_slow − per_row_fast)`. With + `notes.md`'s depth-8 numbers, interpreter 0.95 M rows/s = 1.053 + µs/row and vectorized 11.8 M rows/s = 0.0847 µs/row, the saving is + 0.968 µs/row, so a 500 µs compile pays for itself at 500 / 0.968 = + **516 rows**. The formula hides the reuse count. In Table 1's + OLTP workload the statement is *prepared* and executed millions of + times, so the effective compile cost per execution is 0.81 s / + 10⁶ ≈ 0.8 µs — which is why 20× less compile time (16.53 s → 0.81 + s) bought only +4.8% throughput (161,794 → 169,491 tps): there was + never any per-tuple overhead to remove, because TPC-C transactions + touch under 30 tuples each. + +
+ +- [ ] You can state the paper's measured speedups with their baselines named, including the query where compilation wins nothing. + +
Answer + + Table 2, TPC-CH, 2011 hardware, LLVM 2.8: HyPer+LLVM vs + **VectorWise 1.0** is 2.8× (Q1), 3.2× (Q3), 3.7× (Q4) and + **1.002× (Q5)** — a tie on the join-dominated query. vs + **MonetDB 1.36.5**: 1.4×–2.1× on Q1–Q3, 69.8× on Q4, 10.9× on Q5. + vs the disk-based commercial "DB X": up to 205× on Q3, which + measures storage architecture and is not a code-generation + result. The paper's own phrasing is "frequently another factor + 2–4 faster". Q5's tie is the honest headline: where the query is + memory-bound rather than instruction-bound, removing + interpretation removes nothing. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the VLDB '18 counter-result on hash-probe-heavy queries. +
Answer + + The VLDB'18 result to reconcile: Kersten et al. built *both* + engines (Typer, compiling; Tectorwise, vectorized) and measured + §4.1 "the relative performance ranges from Typer being faster by + 74% (Q1) to Tectorwise being faster by 32% (Q9)" — i.e. the two + models are within 2× on all of TPC-H. Tectorwise wins Q3 and Q9 + because "vectorization is better at hiding cache miss latency": + its probe loop contains nothing but probes, so the out-of-order + window holds many outstanding loads, while Typer's fused loop + (scan + selection + probe + aggregate) fills the reorder buffer + and issues fewer concurrent misses. That is topic 13's MLP + argument applied to codegen — and Neumann's own Table 2 Q5 tie + already showed it in 2011. + +
+ ## References **Papers** - Neumann — "Efficiently Compiling Efficient Query Plans for Modern - Hardware" (VLDB 2011) — read whole; §2 the argument, §3 - produce/consume, §4 the LLVM "cocktail" + Hardware" (PVLDB 4(9):539–550, 2011). Read whole. **§1** the + argument and the three charges against `next()`; **§3.1** + pipeline-breaker definitions; **§3.2** produce/consume + "only a + mental model"; **§4.1** the LLVM cocktail and the 99% rule; + **§4.2** why a pipeline is not one function; **§6.1** Tables 1–2; + **§6.2** Table 3. +- Boncz, Zukowski, Nes — "MonetDB/X100: Hyper-Pipelining Query + Execution" (CIDR 2005) — reference [16], the actual source of + Neumann's Figure 1. - Kersten et al. — "Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask" - (VLDB 2018) — the honest compiled-vs-vectorized comparison Q5 - leans on (also cited in README §7) + (PVLDB 11(13), 2018) — the honest compiled-vs-vectorized + comparison Q5 leans on (also cited in README §7). §4.1 has the + ±74%/−32% range and the memory-stall explanation. + +**Numbers quoted here, and where they come from** + +| number | source | +|---|---| +| 132 M vs 1,184 M instructions (Q1) | Table 3 | +| 2,793 vs 187,471 L1-I misses (Q1) | Table 3 | +| 6,581,223 vs 3,891,827 mispredicts (Q2) | Table 3 | +| 161,794 vs 169,491 tps; 16.53 s vs 0.81 s | Table 1 | +| exec 35/125/80/117/1105 ms | Table 2 | +| LLVM compile 16/41/30/16/34 ms | Table 2 | +| C++ compile 1556–2592 ms | Table 2 | +| VectorWise 98/–/257/436/1107 ms | Table 2 | +| dual X5570, LLVM 2.8, single-threaded | §6, setup | + +**Code in this repo** + +| anchor | what | +|---|---| +| `experiments/src/interp.rs:8-16` | the tree-walk this paper deletes | +| `experiments/src/jit.rs:20-21` | the codegen/precompiled ownership boundary | +| `~/repos/sqlite/src/vdbe.c:1264-1274` | resumption state a compiled loop never needs | diff --git a/topics/19-jit/reading-postgres-jit.md b/topics/19-jit/reading-postgres-jit.md index ba29576..bcf19a3 100644 --- a/topics/19-jit/reading-postgres-jit.md +++ b/topics/19-jit/reading-postgres-jit.md @@ -9,33 +9,111 @@ half, and the cost-model gate whose four failure modes are the lesson — then maps every step to the ~3 files under `src/backend/jit/llvm/`. +**Version.** Anchors are against postgres at the pin in +`resources/codebases.md`, **`701f021`** — which is **PostgreSQL +20devel**, not a released branch. That matters for one headline +number: see Step 4. Retrieve any anchor with +`python3 tools/pinned-source.py show postgres src/backend/jit/jit.c -r 32:42`. +Note also that GUC definitions have moved out of `guc_tables.c` into +the generated `src/backend/utils/misc/guc_parameters.dat`, so older +walkthroughs that send you to `guc_tables.c` for the JIT defaults +now find nothing. + ## The problem in one sentence -Postgres decides whether to spend 10–100 ms of LLVM compilation -using a *planner cost estimate* made before a single row is read — -so when the estimate says "expensive" and the query takes 5 ms, you -pay 50 ms of compile for nothing, and enough users hit that to make -"try jit=off" standard ops advice. +Postgres decides whether to spend tens of milliseconds of LLVM +compilation using a *planner cost estimate* made before a single row +is read — so when the estimate says "expensive" and the query takes +5 ms, you pay the compile for nothing, and enough users hit that to +make "try jit=off" standard ops advice. ## The concepts, step by step ### Step 1 — ExprState: Postgres already has bytecode +> **In:** a parsed WHERE clause or projection. +> **Out:** an `ExprState` — a flat array of `ExprEvalStep`s — plus +> the threaded interpreter that runs it. This is the *baseline* the +> JIT must beat, and every later step is defined against it. + Before any JIT enters the picture, Postgres does not tree-walk expressions per row. At plan time a WHERE clause or projection is flattened into an **ExprState**: a contiguous array of small *steps* -(opcodes like EEOP_FUNCEXPR "call this function", EEOP_QUAL "test -and jump out if false"), executed by a threaded-dispatch interpreter -(execExprInterp.c — computed goto, one indirect branch per step). -This is the same design as SQLite's VDBE at expression grain: -flatten once, dispatch per step per row. So the JIT's opponent is -already a decent bytecode interpreter, not a strawman — the win on -offer is only the per-step dispatch plus what a compiler can see -across steps. +(opcodes like `EEOP_FUNCEXPR` "call this function", `EEOP_QUAL` +"test and jump out if false"), executed by a threaded-dispatch +interpreter. This is the same design as SQLite's VDBE at expression +grain: flatten once, dispatch per step per row. + +But it is a *better* interpreter than SQLite's, in exactly the way +`reading-sqlite-vdbe.md` says SQLite is not: + +```c +// postgres/src/backend/executor/execExprInterp.c — threaded dispatch, 113-137 + 113 /* to make dispatch_table accessible outside ExecInterpExpr() */ + 114 static const void **dispatch_table = NULL; +// ... 115-118: reverse_dispatch_table, for mapping a label back to an opcode ... + 119 #define EEO_SWITCH() + 120 #define EEO_CASE(name) CASE_##name: + 121 #define EEO_DISPATCH() goto *((void *) op->opcode) + 122 #define EEO_OPCODE(opcode) ((intptr_t) dispatch_table[opcode]) + 123 + 124 #else /* !EEO_USE_COMPUTED_GOTO */ + 125 + 126 #define EEO_SWITCH() starteval: switch ((ExprEvalOp) op->opcode) + 127 #define EEO_CASE(name) case name: + 128 #define EEO_DISPATCH() goto starteval + 129 #define EEO_OPCODE(opcode) (opcode) + 130 + 131 #endif /* EEO_USE_COMPUTED_GOTO */ + 132 + 133 #define EEO_NEXT() \ + 134 do { \ + 135 op++; \ + 136 EEO_DISPATCH(); \ + 137 } while (0) +``` + +Line 121 is the load-bearing one. **Threaded dispatch** (also called +computed goto): instead of one shared indirect branch at the top of +a loop, *every* step ends with its own `goto *op->opcode` — so +`op->opcode` is not an enum at all, it is a pre-resolved *label +address*, taken from `dispatch_table` at line 114. The branch +predictor then gets one indirect-branch site per opcode kind, each +with its own history, instead of one 121-target site that it cannot +possibly predict. Lines 126–129 are the portable fallback when the +compiler has no `&&label` extension. + +So the JIT's opponent is a genuinely good bytecode interpreter, not +a strawman. The win on offer is only the per-step dispatch plus what +a compiler can see *across* steps. + +How big is the step set? Count it two ways and get the same answer, +which is the tidiest structural fact in this chapter: + +``` + grep -c 'EEO_CASE(' execExprInterp.c → 123 + minus the 2 #define lines at :120, :127 → 121 real step kinds + + the dispatch_table initialiser (execExprInterp.c:484-612) + → 121 `&&CASE_…` entries, and the file asserts it: + execExprInterp.c:608 StaticAssertDecl(lengthof(dispatch_table) + == EEOP_LAST + 1, …) + + grep -c 'case EEOP_' llvmjit_expr.c → 121 +``` + +**121 interpreter step kinds, 121 JIT cases.** The JIT is a +one-to-one mirror of the interpreter, maintained by hand. That is +the maintenance bill for this feature, stated as a number. ### Step 2 — what the JIT compiles: one basic block per step -The JIT's scope is deliberately narrow: +> **In:** Step 1's `ExprState` step array. **Out:** one LLVM +> function containing one basic block per step, with the dispatch +> replaced by fallthrough. Nothing above the expression is touched. + +The JIT's scope is deliberately narrow, and this is the single most +misreported fact about it: ``` NOT compiled: executor nodes (SeqScan, HashJoin...) — still the @@ -46,22 +124,70 @@ The JIT's scope is deliberately narrow: specialized: known offsets, nullability) ``` -`llvm_compile_expr` (llvmjit_expr.c:80) translates each step of one -ExprState into one **basic block** (a straight-line chunk of code -with one entry and one exit — LLVM's unit of control flow), wires -the blocks together in step order, and lets LLVM fold the dispatch -away — the indirect branch the interpreter pays per step becomes a -fallthrough: +Postgres compiles **expressions and tuple deforming**, not query +plans. `planner.c:717-720` says it in the source: the only two +things the flag word can request are `PGJIT_EXPR` and +`PGJIT_DEFORM`. If someone tells you Postgres "compiles queries the +way HyPer does", they have not read `llvmjit_expr.c`. + +`llvm_compile_expr` (`llvmjit_expr.c:80`) translates each step of +one ExprState into one **basic block** (a straight-line chunk of +code with one entry and one exit — LLVM's unit of control flow), +wires the blocks together in step order, and lets LLVM fold the +dispatch away: + +```c +// postgres/src/backend/jit/llvm/llvmjit_expr.c — the block-per-step loop, 301-324 + 301 /* allocate blocks for each op upfront, so we can do jumps easily */ + 302 opblocks = palloc_array(LLVMBasicBlockRef, state->steps_len); + 303 for (int opno = 0; opno < state->steps_len; opno++) + 304 opblocks[opno] = l_bb_append_v(eval_fn, "b.op.%d.start", opno); + 305 + 306 /* jump from entry to first block */ + 307 LLVMBuildBr(b, opblocks[0]); + 308 + 309 for (int opno = 0; opno < state->steps_len; opno++) + 310 { +// ... 311-315: local LLVMValueRef declarations ... + 316 LLVMPositionBuilderAtEnd(b, opblocks[opno]); + 317 + 318 op = &state->steps[opno]; + 319 opcode = ExecEvalStepOp(state, op); + 320 + 321 v_resvaluep = l_ptr_const(op->resvalue, l_ptr(TypeDatum)); + 322 v_resnullp = l_ptr_const(op->resnull, l_ptr(TypeStorageBool)); + 323 + 324 switch (opcode) +``` + +Line 302 is why the whole thing works: **every block is created +before any is filled in**, so a step that jumps forward (`EEOP_QUAL` +skipping to the end on a false qual) already has its target +available and needs no patch-up pass. Same instinct as SQLite's +fixed-width `p2` jump operand (`reading-sqlite-vdbe.md`, Step 3), +solved with an array of block handles instead. + +Lines 321–322 are the quiet performance story. `op->resvalue` is a +pointer that is *known at compile time*, so it is emitted as an LLVM +constant — the interpreter loads it from the step struct every row; +the JIT bakes it in. Multiply that by 121 opcode kinds and you have +most of the win that is not dispatch. + +**Anchor correction.** The giant `switch (opcode)` is at +**`llvmjit_expr.c:324`** (first case `EEOP_DONE_RETURN` at `:326`), +not "326+"; and the FETCHSOME group is at **`:344-348`**, not +"354+". ```rust -// llvm_compile_expr's shape: one basic block per interpreter step — -// the dispatch the interpreter pays per step becomes a fallthrough +// ILLUSTRATION — not quoted from postgres. The C at +// llvmjit_expr.c:301-324 (above) written as Rust, to make the shape +// obvious; read the real thing, this elides the LLVM builder plumbing. let opblocks: Vec = state.steps.iter().map(|_| new_block()).collect(); for (i, step) in state.steps.iter().enumerate() { position_at(opblocks[i]); match step.opcode { - EEOP_QUAL => emit_cmp_and_branch(step, opblocks[step.jumpdone]), - EEOP_FUNCEXPR => emit_direct_call(step.fn_addr, step.args), + EEOP_QUAL => emit_cmp_and_branch(step, opblocks[step.jumpdone]), + EEOP_FUNCEXPR => emit_direct_call(step.fn_addr, step.args), EEOP_SCAN_FETCHSOME => emit_deform(tupledesc, step.last_attr), // ... the giant switch mirrors execExprInterp.c case by case } @@ -69,130 +195,485 @@ for (i, step) in state.steps.iter().enumerate() { } ``` -Structurally the SAME translation our stub does for `Expr` → CLIF — -postgres just starts from bytecode instead of an AST. It is NOT -Neumann's whole-pipeline compilation: operators still call each -other through interpreted indirection; only the leaves got fast. +Structurally the SAME translation our stub does for `Expr` → CLIF +(`experiments/src/jit.rs:11-19`) — postgres just starts from +bytecode instead of an AST. It is NOT Neumann's whole-pipeline +compilation: operators still call each other through interpreted +indirection; only the leaves got fast. ### Step 3 — tuple deforming: the underrated half +> **In:** Step 2's `EEOP_*_FETCHSOME` steps and the `TupleDesc` +> that describes the row layout. **Out:** a decoder specialized to +> one schema, and the reason it is often worth more than the +> expression JIT. + **Deforming** is extracting attribute values from Postgres's on-disk row format — variable-length fields, a null bitmap, and alignment padding mean that reaching column 19 requires walking columns 1–18, testing the null bitmap at each. The generic decoder (`slot_deform_heap_tuple`) re-discovers the schema per row. -llvmjit_deform.c instead generates a decoder *specialized to the -schema*: attribute offsets constant-folded, null-bitmap checks -skipped for NOT NULL columns, alignment known. This routinely beats -the expression JIT in profit because deforming is -per-ROW-per-ATTRIBUTE and pure branchy pointer math — the same + +The hook is right where you would want it: + +```c +// postgres/src/backend/jit/llvm/llvmjit_expr.c — the deform hook, 404-410 + 404 if (tts_ops && desc && (context->base.flags & PGJIT_DEFORM)) + 405 { + 406 INSTR_TIME_SET_CURRENT(deform_starttime); + 407 l_jit_deform = + 408 slot_compile_deform(context, desc, + 409 tts_ops, + 410 op->d.fetch.last_var); +``` + +Three things to notice at :404. First, `PGJIT_DEFORM` is a +*separate* flag from `PGJIT_EXPR` — you can have one without the +other, and `jit_tuple_deforming` (`jit.c:39`, default `true`) is the +GUC. Second, `desc` — the `TupleDesc` — is the specialization key: +`slot_compile_deform` in `llvmjit_deform.c` constant-folds each +attribute's offset, skips null-bitmap tests for `NOT NULL` columns, +and knows every alignment in advance. Third, `op->d.fetch.last_var` +bounds the work: the generated decoder stops at the highest column +the expression actually references. If `tts_ops` or `desc` is +missing, `:434-437` falls back to emitting a plain call to +`slot_getsomeattrs_int` — the generic path. + +This routinely beats the expression JIT in profit because deforming +is per-ROW-per-ATTRIBUTE and pure branchy pointer math — the same reason topic 12's PAX/columnar layouts win, arrived at from the -compiler side. +compiler side. And note the framing: a columnar layout makes the +whole problem *disappear* rather than compiling a faster solution to +it. Question 4. ### Step 4 — the gate: a cost estimate decides, and misfires four ways -Compilation triggers when the planner's estimated total cost — an -abstract unitless number built from row-count guesses (topic 10) — -crosses a GUC threshold: +> **In:** the finished plan's `total_cost` — a unitless planner +> estimate (topic 10) — and five GUCs. **Out:** a `jitFlags` word +> that is fixed for the whole query before any row is read. This is +> the step the chapter exists for. + +**The headline correction: at this pin, `jit` defaults to OFF.** + +```c +// postgres/src/backend/jit/jit.c — every JIT GUC's C default, 32-42 + 32 /* GUCs */ + 33 bool jit_enabled = false; +// ... 34-36: jit_provider, jit_debugging_support, jit_dump_bitcode ... + 37 bool jit_expressions = true; +// ... 38: jit_profiling_support ... + 39 bool jit_tuple_deforming = true; + 40 double jit_above_cost = 100000; + 41 double jit_inline_above_cost = 500000; + 42 double jit_optimize_above_cost = 500000; +``` +Line 33 is the news. Upstream Postgres has flipped the default: the +same value appears in the generated GUC table +(`src/backend/utils/misc/guc_parameters.dat:1451-1456`, +`boot_val => 'false'`), in the shipped config sample +(`src/backend/utils/misc/postgresql.conf.sample:492` — `#jit = off`) +and in the docs (`doc/src/sgml/config.sgml:6836` — "The default is +`off`."). The community's answer to this chapter's title was, in +the end, to take the advice. **Check this against whatever version +you actually run** — releases through PG 17 shipped `jit = on`. + +Now the gate itself. It is **three** cost thresholds and two +booleans, not one threshold: + +```c +// postgres/src/backend/optimizer/plan/planner.c — the whole gate, 698-721 + 698 result->jitFlags = PGJIT_NONE; + 699 if (jit_enabled && jit_above_cost >= 0 && + 700 top_plan->total_cost > jit_above_cost) + 701 { + 702 result->jitFlags |= PGJIT_PERFORM; +// ... 703-706: comment — "how much effort should be put into better code" ... + 707 if (jit_optimize_above_cost >= 0 && + 708 top_plan->total_cost > jit_optimize_above_cost) + 709 result->jitFlags |= PGJIT_OPT3; + 710 if (jit_inline_above_cost >= 0 && + 711 top_plan->total_cost > jit_inline_above_cost) + 712 result->jitFlags |= PGJIT_INLINE; +// ... 713-716: comment — "which operations should be JITed" ... + 717 if (jit_expressions) + 718 result->jitFlags |= PGJIT_EXPR; + 719 if (jit_tuple_deforming) + 720 result->jitFlags |= PGJIT_DEFORM; + 721 } ``` - planner.c:699: use JIT iff estimated total_cost > jit_above_cost - (default 100000) - failure 1: estimate high, reality short → pay ~10-100ms LLVM - for a fast query (the classic complaint) - failure 2: cost is in COST UNITS not ms — jit_above_cost has no - unit relationship with compile time on this machine +Read `>= 0` on lines 699, 707, 710: **a negative value disables that +tier**, which is the documented escape hatch (`config.sgml:6483` — +"Setting this to `-1` disables JIT compilation"). The whole decision +is five comparisons against one number, `top_plan->total_cost`, and +it happens in `standard_planner`, before the executor starts. + +``` + the four thresholds, in cost units, from jit.c:40-42 + + total_cost > 100000 → PGJIT_PERFORM compile at all + total_cost > 500000 → PGJIT_OPT3 run LLVM -O3 + total_cost > 500000 → PGJIT_INLINE inline pg internals + (jit_expressions / jit_tuple_deforming are booleans, not costs) + + failure 1: estimate high, reality short → pay the compile for a + fast query (the classic complaint) + failure 2: cost is in COST UNITS not ms — 100000 has no unit + relationship with compile time on this machine, and + the cost model was calibrated for I/O, not for LLVM failure 3: decision is per-QUERY, all-or-nothing, made BEFORE any row is seen — no adaptivity (contrast Umbra) - failure 4: opt3 is gated by ANOTHER estimate (jit_optimize_above_ - cost) — two thresholds to mistune + failure 4: opt3 and inlining are gated by ANOTHER estimate at the + SAME default (500000) — so in practice you cross into + the two most expensive LLVM modes simultaneously ``` -There's a partial mitigation: two LLJIT tiers (opt0/opt3, -llvmjit.c:100-101 — LLJIT is LLVM's JIT engine; opt0 compiles fast -and slow, opt3 slow and fast) — but tier choice is still -estimate-driven. This is the actual lesson of the chapter: the -compile-or-not decision is a bet, and Postgres places it with the -least reliable number in the system. +Failure 4 is worth the arithmetic. A plan whose cost lands anywhere +above 500000 gets `PGJIT_OPT3 | PGJIT_INLINE` *together* — the two +settings that dominate compile time — from a single estimate that +was never designed to predict compile time. And the cost model's own +units come from `seq_page_cost = 1.0`: 500000 cost units ≈ half a +million sequential page reads ≈ 4 GB of I/O at 8 KB pages. That is +the quantity Postgres uses to decide how hard to run a compiler. + +There is a partial mitigation — two LLJIT tiers: + +```c +// postgres/src/backend/jit/llvm/llvmjit.c — the two engines, 100-101 + 100 static LLVMOrcLLJITRef llvm_opt0_orc; + 101 static LLVMOrcLLJITRef llvm_opt3_orc; +``` + +and the selection, one flag test: + +```c +// postgres/src/backend/jit/llvm/llvmjit.c — tier selection in llvm_compile_module, 716-721 + 716 LLVMOrcLLJITRef compile_orc; + 717 + 718 if (context->base.flags & PGJIT_OPT3) + 719 compile_orc = llvm_opt3_orc; + 720 else + 721 compile_orc = llvm_opt0_orc; +``` + +(**LLJIT** is LLVM's ORC-based JIT engine; opt0 compiles fast and +produces slow code, opt3 the reverse.) But tier choice is still +estimate-driven — line 718 reads a flag that `planner.c:709` set +before execution began. This is the actual lesson of the chapter: +the compile-or-not decision is a bet, and Postgres places it with +the least reliable number in the system, once, before it can +possibly learn anything. `reading-umbra-tidy-tuples.md` is what +placing it *after* you have evidence looks like. ### Step 5 — lifecycle plumbing worth stealing -JIT-compiled code is memory that something must own. llvmjit.c:716+ -compiles modules into a dylib with a resource tracker per -compilation; llvmjit.c:288-299 shows teardown (remove tracker, -clear dead symbol-pool entries). Ownership is per-query-context: -when the query dies, the code dies — no dangling function pointers. -M19 note: cranelift's `JITModule` has the same `free_memory` -obligation — our stub keeps the module alive inside `CompiledExpr` -so the fn pointer can't dangle. +> **In:** a compiled LLVM module and a query that will eventually +> end. **Out:** a rule for who owns executable memory, and the +> teardown call that makes dangling function pointers impossible. + +JIT-compiled code is memory that something must own. +`llvm_compile_module` (`llvmjit.c:710`) adds each module to an LLJIT +dylib and takes a **resource tracker** for it +(`LLVMOrcJITDylibCreateResourceTracker`, `:781`), which is stored on +the context (`LLVMOrcResourceTrackerRef resource_tracker;` +`llvmjit.c:51`). Release is two calls: + +```c +// postgres/src/backend/jit/llvm/llvmjit.c — llvm_release_context teardown, 288-289 + 288 LLVMOrcResourceTrackerRemove(jit_handle->resource_tracker); + 289 LLVMOrcReleaseResourceTracker(jit_handle->resource_tracker); +``` + +`:288` unmaps the code; `:289` drops the handle. The surrounding +block (`:290-299`) then clears dead symbol-pool entries, because ORC +would otherwise leak the mangled names. Ownership is +per-query-context: `llvm_release_context` (`:253`) is registered as +the provider's `release_context` callback at `llvmjit.c:155`, right +above `cb->compile_expr = llvm_compile_expr;` at `:156`. When the +query dies, the code dies — no dangling function pointers. -### Step 6 — what transfers to M19 +M19 note: cranelift's `JITModule` has the same obligation, and our +stub already encodes it — +`experiments/src/jit.rs:26-31` keeps the module alive *inside* +`CompiledExpr` so the `fn(*const f64) -> f64` at `:30` cannot +outlive its code. `experiments/src/jit.rs:21` cites this very +Postgres line for the pattern. + +### Step 6 — what transfers to M19, and the break-even arithmetic + +> **In:** Steps 2–5's mechanism and Step 4's failure list. +> **Out:** the gate you should build instead, expressed as a +> division you can actually evaluate at runtime. - Compile the *expression*, keep the executor: exactly M19's scope. -- Gate on MEASURED cost (rows already processed × measured ns/row - vs measured compile µs), not an estimate — Umbra's lesson applied - to Postgres's failure. +- Gate on MEASURED cost, not an estimate. - Deforming lesson: FalkorDB's property access (attribute fetch from the property store) is the deform-analogue — likely more profit than arithmetic JIT. +The gate Postgres cannot write, written out with this topic's own +measured numbers (`notes.md`, Apple M3 Pro, depth 8 = 511 nodes): + +``` + Postgres's gate: + compile iff planner_estimate > 100000 [cost units] + — one number, two unknowns (is the estimate right? what does + LLVM cost on this box?), evaluated before any evidence exists. + + A measured gate, same decision: + rows_breakeven = compile_µs / (µs_per_row_interp − µs_per_row_jit) + + interp lane, 511 nodes: 0.95 M rows/s → 1.053 µs/row + vector lane, 511 nodes: 11.8 M rows/s → 0.0847 µs/row + assume the JIT lands at the vector lane's rate (this topic's own + prediction — see notes.md): + saving = 1.053 − 0.0847 = 0.968 µs/row + with a 500 µs cranelift compile: + rows_breakeven = 500 / 0.968 = 516 rows + + Both inputs are things you can MEASURE, not estimate: + compile_µs — time your own compile() and keep a moving + average per node count + µs_per_row_interp — you are already running the interpreter; + count rows and nanoseconds as you go + So the rule becomes: interpret first, and switch to compiled code + once rows_seen exceeds break-even and the query is still running. + That is exactly Umbra's adaptive execution, and it needs no + planner estimate at all. +``` + +Note what the arithmetic reveals about Postgres's specific pain: +because the gate fires *before* row one, a plan estimated at 600000 +cost units that returns 3 rows pays `PGJIT_OPT3 | PGJIT_INLINE` +compile time — the most expensive mode — against a saving of +3 × 0.968 µs ≈ 3 µs. There is no compile fast enough to win that +bet. The bug is not the threshold's value; it is that a threshold +on an estimate can never be right. + ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| llvmjit.c:156 | provider hook: `cb->compile_expr = llvm_compile_expr` | 2 | -| llvmjit_expr.c:80 | `llvm_compile_expr(ExprState*)` — the entry point | 2 | -| llvmjit_expr.c:302-307 | one LLVM basic block per ExprState step (`opblocks`) | 2 | -| llvmjit_expr.c:326+ | the giant `case EEOP_*` switch — mirror of the interpreter | 1–2 | -| llvmjit_expr.c:354+ | EEOP_*_FETCHSOME → JIT tuple deforming (llvmjit_deform.c) | 3 | -| planner.c:699-700 | the gate: `top_plan->total_cost > jit_above_cost` | 4 | -| llvmjit.c:85-101 | session state: two LLJITs — `llvm_opt0_orc` / `llvm_opt3_orc` | 4 | -| llvmjit.c:363 | `llvm_get_function` — lookup + (lazy) emission | 5 | -| llvmjit.c:716-781 | module → ThreadSafeModule → LLJIT dylib + resource tracker | 5 | - -Pair llvmjit_expr.c with `src/backend/executor/execExprInterp.c` -side by side — every `case EEOP_*` in the JIT mirrors a case in the -interpreter, and seeing what each block replaces is Step 1 and -Step 2 in one diff. Then read planner.c:699 for the gate and -llvmjit.c for the lifecycle. +| `execExprInterp.c:113-137` | threaded dispatch: `dispatch_table`, `EEO_DISPATCH()` | 1 | +| `execExprInterp.c:484-612` | the 121-entry `&&CASE_…` table + its `StaticAssertDecl` at `:608` | 1 | +| `execExpr.h:296` | `EEOP_LAST` — the step-kind count the table asserts against | 1 | +| `llvmjit.c:155-156` | provider hooks: `release_context`, `cb->compile_expr = llvm_compile_expr` | 2, 5 | +| `llvmjit_expr.c:80` | `llvm_compile_expr(ExprState *state)` — the entry point | 2 | +| `llvmjit_expr.c:301-307` | one LLVM basic block per step, all allocated upfront | 2 | +| `llvmjit_expr.c:324` | the giant `switch (opcode)` — 121 `case EEOP_`, mirror of the interpreter | 1–2 | +| `llvmjit_expr.c:344-348` | the five `EEOP_*_FETCHSOME` cases | 3 | +| `llvmjit_expr.c:404-410` | `slot_compile_deform` — the deform hook, gated on `PGJIT_DEFORM` | 3 | +| `llvmjit_expr.c:434-437` | the generic fallback: emit a call to `slot_getsomeattrs_int` | 3 | +| `llvmjit_expr.c:972-1007` | `EEOP_QUAL` — the clearest single opcode to diff against the interpreter | 1–2 | +| `jit.c:33` | **`bool jit_enabled = false;`** — the default flipped upstream | 4 | +| `jit.c:40-42` | `jit_above_cost` 100000, `jit_inline_above_cost` 500000, `jit_optimize_above_cost` 500000 | 4 | +| `guc_parameters.dat:1451-1456` | the generated GUC entry (`boot_val => 'false'`) | 4 | +| `postgresql.conf.sample:492` | `#jit = off` | 4 | +| `config.sgml:6836` / `:6483-6499` | docs: "The default is `off`"; `-1` disables a tier | 4 | +| `planner.c:698-721` | the whole gate — 3 cost tests, 2 booleans, 5 flags | 4 | +| `llvmjit.c:100-101` | two LLJITs: `llvm_opt0_orc` / `llvm_opt3_orc` | 4 | +| `llvmjit.c:716-721` | tier selection from `PGJIT_OPT3` | 4 | +| `llvmjit.c:363` | `llvm_get_function` — lookup, calling `llvm_compile_module` at `:375` | 5 | +| `llvmjit.c:710-781` | module → ThreadSafeModule (`:778`) → dylib + resource tracker (`:781`) | 5 | +| `llvmjit.c:288-289` | teardown: `ResourceTrackerRemove` + `ReleaseResourceTracker` | 5 | + +Paths are relative to `src/backend/` except `execExpr.h` +(`src/include/executor/`) and the docs. Fetch without a clone: +`python3 tools/pinned-source.py show postgres src/backend/jit/llvm/llvmjit_expr.c -r 301:330`. + +Pair `llvmjit_expr.c` with `src/backend/executor/execExprInterp.c` +side by side — every `case EEOP_*` in the JIT mirrors an +`EEO_CASE()` in the interpreter, 121 for 121, and seeing what each +block replaces is Step 1 and Step 2 in one diff. Then read +`planner.c:698` for the gate and `llvmjit.c` for the lifecycle. ## Questions for notes.md -1. Trace one EEOP through both executors: find EEOP_QUAL in - execExprInterp.c and in llvmjit_expr.c. What does LLVM get to - do that the interpreter can't (cross-step constant prop, dead - null-check elimination)? +1. Trace one EEOP through both executors: find `EEOP_QUAL` in + `execExprInterp.c` and at `llvmjit_expr.c:972-1007`. What does + LLVM get to do that the interpreter can't (cross-step constant + prop, dead null-check elimination)? Start from + `llvmjit_expr.c:321-322` — what exactly became a constant there, + and what does the interpreter do instead? 2. Why does the JIT emit ONE function per ExprState with a block per step, rather than one function per step (call overhead + - register state across steps — the copy-and-patch contrast)? -3. jit_above_cost is in planner cost units. Propose the fix - postgres upstream keeps debating: what would a *time-based* - gate need to know (compile-time model per step count + rows - estimate — and which half is still an estimate)? + register state across steps)? Then read the copy-and-patch + contrast in `reading-umbra-tidy-tuples.md`: that system emits + one *stencil* per node and gets away with it. What does it do + differently at the call boundary? +3. `jit_above_cost` is in planner cost units (`jit.c:40`). Propose + the fix upstream keeps debating: what would a *time-based* gate + need to know (a compile-time model per step count, plus a rows + estimate) — and which half is still an estimate? Use Step 6's + division and identify which of its two inputs Postgres could + measure today without any new infrastructure. 4. Deform JIT: for a 20-column table where the query touches column 19, what does the generated decoder skip vs the generic `slot_deform_heap_tuple`, and which topic 12 layout makes the - whole problem vanish? -5. For M19: postgres compiles per-query with no cache. GraphBLAS - caches per type-combo forever (reading-graphblas-jit.md). - Which is right for Cypher expressions, and what's the cache - key (expression shape with constants as parameters — count how - many distinct shapes a workload of 1000 queries has)? + whole problem vanish? `llvmjit_expr.c:410` passes + `op->d.fetch.last_var` — what does that bound, and what does it + *not* let you skip? +5. For M19: postgres compiles per-query with no cache (the resource + tracker at `llvmjit.c:288-289` destroys the code when the query + ends). GraphBLAS caches per type-combo forever + (`reading-graphblas-jit.md`). Which is right for Cypher + expressions, and what's the cache key (expression shape with + constants as parameters — count how many distinct shapes a + workload of 1000 queries has)? ## Done when +Answer each before unfolding it. + - [ ] You can explain that Postgres already had bytecode (`ExprState`) before it had a JIT, and what the JIT therefore actually replaces. + +
Answer + + `ExprState` is a flat array of `ExprEvalStep`s produced at plan + time, run by a **threaded-dispatch** interpreter: + `execExprInterp.c:121` defines `EEO_DISPATCH()` as + `goto *((void *) op->opcode)`, where `op->opcode` has been + rewritten to a label address from `dispatch_table` + (`:114`). Every step ends with its own indirect branch, so the + predictor gets per-opcode history — strictly better than SQLite's + single shared `switch`. The JIT therefore replaces only (a) the + remaining per-step indirect branch, and (b) the per-row loads of + values that are constant for the whole query — `op->resvalue` and + `op->resnull` become LLVM constants at `llvmjit_expr.c:321-322`. + It does not replace any executor node. There are **121** step + kinds and **121** `case EEOP_` in the JIT: a hand-maintained + one-to-one mirror. + +
+ - [ ] You can explain tuple deforming and why it is the underrated half of the win. -- [ ] You can name all four ways the `jit_above_cost` gate misfires, and propose a better gate. + +
Answer + + Deforming turns Postgres's on-disk row (null bitmap, varlena + fields, alignment padding) into `Datum`s. Because field offsets + are not fixed, reaching attribute *n* means walking attributes + 1..n−1 and testing the null bitmap at each — per row. The generic + `slot_deform_heap_tuple` rediscovers the schema every row; + `slot_compile_deform` (called at `llvmjit_expr.c:407-410`, gated + on `PGJIT_DEFORM` at `:404`) generates a decoder specialized to + one `TupleDesc`: offsets constant-folded, null tests omitted for + `NOT NULL` columns, alignment known, and work bounded by + `op->d.fetch.last_var`. It is underrated because its cost is + per-row *per-attribute* and it is pure branchy pointer math — + the exact shape a compiler is good at — whereas expression JIT + usually removes only a handful of dispatches per row. + +
+ +- [ ] You can name all four ways the `jit_above_cost` gate misfires, propose a better gate, and state the current default of `jit` itself. + +
Answer + + (1) The estimate can be high while the query is short, so you pay + compile time for nothing. (2) The threshold is in planner cost + units — 100000 (`jit.c:40`) — which have no unit relationship to + milliseconds of LLVM on your hardware. (3) The decision is + per-query, all-or-nothing, made in `standard_planner` + (`planner.c:698-721`) before any row is read, so it can never + adapt. (4) `jit_optimize_above_cost` and `jit_inline_above_cost` + both default to 500000 (`jit.c:41-42`), so crossing one line + turns on LLVM `-O3` *and* inlining together — the two most + expensive modes, chosen by the same unreliable number. Better + gate: interpret first, measure `µs_per_row` and `compile_µs` + directly, and switch when `rows_seen > compile_µs / (µs_interp − + µs_jit)`. And the current default: **`jit` is `off`** at pin + `701f021` (`jit.c:33`, `guc_parameters.dat:1451-1456`, + `postgresql.conf.sample:492`, `config.sgml:6836`) — upstream took + the advice in this chapter's title. Releases through PG 17 shipped + it `on`. + +
+ - [ ] You can say why the JIT emits one function per `ExprState` with a block per step. + +
Answer + + Because basic blocks are free and function calls are not. All + blocks are allocated upfront at `llvmjit_expr.c:302-304`, so + forward jumps resolve without a patch-up pass; then LLVM's own + simplify-CFG pass merges adjacent blocks that have a single + predecessor, which is most of them — the interpreter's per-step + dispatch literally becomes fallthrough. If each step were a + function, every step would pay a call, a return, and a full + register spill at the boundary, and no value could stay in a + register across steps — which is Neumann's §4.1 "the hot path does + not cross a function boundary" rule. Copy-and-patch gets away with + one stencil per node only because it uses the GHC calling + convention and tail calls, so the "calls" lower to jumps and + parameters stay in registers. + +
+ +- [ ] You can compute the row count at which compiling an expression pays for itself, and explain why Postgres structurally cannot use that number. + +
Answer + + `rows_breakeven = compile_µs / (µs_per_row_interp − + µs_per_row_jit)`. With this topic's depth-8 measurements + (`notes.md`): 1.053 µs/row interpreted, 0.0847 µs/row for the + vectorized lane the JIT is predicted to match, saving 0.968 + µs/row; a 500 µs compile pays back at **516 rows**. Postgres + cannot use it because both inputs are unavailable at the moment + it decides: `planner.c:698-721` runs before execution, so + `rows_seen` is zero and `µs_per_row` has never been observed — + all it has is `top_plan->total_cost`, an estimate in units + calibrated for page reads. The fix is not a better threshold, it + is deciding later; that is what adaptive execution means in + `reading-umbra-tidy-tuples.md`. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the per-query-no-cache versus GraphBLAS-cache-forever contrast. +
Answer + + The contrast to write down: Postgres's compiled code is owned by + the query — `llvm_release_context` calls + `LLVMOrcResourceTrackerRemove` at `llvmjit.c:288`, so the code is + unmapped when the query ends and the next identical query + recompiles from scratch. GraphBLAS keys its kernels on a + *semiring × types × sparsity* encoding and keeps them in an + in-memory hash table plus an on-disk `.so` cache that survives + process restarts (`reading-graphblas-jit.md`). Which is right + depends on how many distinct artifacts the workload has: Postgres + has effectively unbounded distinct expressions (every literal is + a new one unless parameterized), GraphBLAS has a small closed set. + For Cypher, parameterize the constants and key on expression + *shape* — then count the distinct shapes in a real workload + before choosing. + +
+ ## References -**Code** -- [postgres](https://github.com/postgres/postgres) — - `src/backend/jit/llvm/` — llvmjit.c (lifecycle), llvmjit_expr.c - (the EEOP switch), llvmjit_deform.c (the underrated half); pair - with `src/backend/executor/execExprInterp.c` to see what each - EEOP block replaces, and `planner.c:699` for the gate +**Code** — all anchors verified at postgres `701f021` (PG 20devel) + +| file | anchors | +|---|---| +| `src/backend/jit/jit.c` | `:33` `jit_enabled = false`; `:37-42` the other GUC defaults | +| `src/backend/optimizer/plan/planner.c` | `:698-721` the gate | +| `src/backend/jit/llvm/llvmjit_expr.c` | `:80` entry; `:301-307` blocks; `:324` the switch; `:344-348` FETCHSOME; `:404-410` deform hook; `:434-437` fallback; `:972-1007` `EEOP_QUAL` | +| `src/backend/jit/llvm/llvmjit.c` | `:51` tracker field; `:100-101` two LLJITs; `:155-156` provider hooks; `:253` release; `:288-289` teardown; `:363` `llvm_get_function`; `:710-781` compile + tracker | +| `src/backend/jit/llvm/llvmjit_deform.c` | `slot_compile_deform` — the specialized decoder | +| `src/backend/executor/execExprInterp.c` | `:113-137` threaded dispatch; `:484-612` dispatch table; `:608` the static assert | +| `src/include/executor/execExpr.h` | `:296` `EEOP_LAST` | +| `src/backend/utils/misc/guc_parameters.dat` | `:1451-1456` the generated `jit` GUC | +| `src/backend/utils/misc/postgresql.conf.sample` | `:467`, `:470`, `:472`, `:492` | +| `doc/src/sgml/config.sgml` | `:6483-6499` cost GUCs; `:6836` "The default is `off`" | + +**Elsewhere in this repo** +- `experiments/src/jit.rs:11-19` — the same block-per-node + translation for CLIF; `:26-31` the ownership pattern this + chapter's Step 5 is the source of +- `reading-sqlite-vdbe.md` — the interpreter that does *not* thread +- `reading-umbra-tidy-tuples.md` — deciding after the evidence + arrives instead of before +- `reading-neumann-vldb11.md` — §4.1's rule for what stays + precompiled, which is exactly why Postgres JITs leaves only diff --git a/topics/19-jit/reading-sqlite-vdbe.md b/topics/19-jit/reading-sqlite-vdbe.md index 0e7cd9c..18195f2 100644 --- a/topics/19-jit/reading-sqlite-vdbe.md +++ b/topics/19-jit/reading-sqlite-vdbe.md @@ -10,6 +10,14 @@ why bytecode beats a tree walk, what a register machine is, what dispatch costs, and the coroutine trick flattening gives you for free — then maps each step into vdbe.c. +**Version.** Every anchor below is against sqlite at the pin in +`resources/codebases.md`, **`951de30`**, where `src/vdbe.c` is 9456 +lines. Retrieve any of them with +`python3 tools/pinned-source.py show sqlite src/vdbe.c -r 1926:2010`. +SQLite's opcode numbering is generated at build time by +`mkopcodeh.tcl` scanning this file, so line numbers here move +between releases but the *structure* has been stable since 2004. + ## The problem in one sentence Walking an AST (abstract syntax tree — the parsed expression as @@ -23,6 +31,11 @@ five rows beats any JIT. ### Step 1 — flatten once: from AST to a bytecode program +> **In:** SQL text and, from topic 19's framing, a tree-walking +> evaluator that pays a dispatch per node per row. +> **Out:** a contiguous `VdbeOp[]` array and one `for(;;)` loop +> that walks it — the object every later step operates on. + **Bytecode** is a program encoded as an array of small fixed-format instructions for a software-defined machine (a "virtual machine" — here the VDBE, Virtual DataBase Engine). At `sqlite3_prepare` time @@ -31,16 +44,38 @@ flat `VdbeOp[]` array; execution never sees the tree: ``` prepare: SQL ──parse──► AST ──codegen──► VdbeOp[] program - execute: pc = 0 - for(;;){ pOp = &aOp[pc]; - switch(pOp->opcode){ ... } ← vdbe.c:1049 - pc++ or jump } + execute: for(pOp=&aOp[p->pc]; 1; pOp++){ ← vdbe.c:966 + switch(pOp->opcode){ ... } ← vdbe.c:1049 + ... break, or goto jump_to_p2 } ← vdbe.c:1221 + } ← vdbe.c:9357 state: array of Mem registers (typed values), array of cursors (open B-tree positions). A register machine, NOT a stack machine — p1/p2/p3 name registers directly, no push/pop traffic. ``` +The loop header carries the whole design in one line: + +```c +// sqlite/src/vdbe.c — the entire interpreter loop header, 966-972 + 966 for(pOp=&aOp[p->pc]; 1; pOp++){ + 967 /* Errors are detected by individual opcodes, with an immediate + 968 ** jumps to abort_due_to_error. */ + 969 assert( rc==SQLITE_OK ); + 970 + 971 assert( pOp>=aOp && pOp<&aOp[p->nOp]); + 972 nVmStep++; +``` + +Read line 966 twice. The program counter is not an index — it is a +**pointer walked with `pOp++`**, over an array. Sequential execution +is a pointer increment with no bounds check in a release build (971 +is an `assert`), and the hardware prefetcher sees a perfectly linear +instruction stream. That is the entire structural win over a tree +walk, before a single opcode has run. `nVmStep` at 972 is what +`sqlite3_stmt_status(SQLITE_STMTSTATUS_VM_STEP)` reports — the count +you will use in question 1. + What flattening buys immediately: instructions live contiguously (cache-linear, no pointer chasing), the interpreter is one loop instead of recursion, and the program is *inspectable* — run @@ -62,6 +97,11 @@ asks you to read one: ### Step 2 — registers, not a stack: the machine model +> **In:** Step 1's `VdbeOp[]` array and its `Mem` register file. +> **Out:** the operand-naming convention (p1/p2/p3 as register +> indices) and a *count* of dispatches per expression — the unit +> Step 4 prices. + A **stack machine** (JVM, Python) makes every instruction implicitly pop operands and push results — simple codegen, but `a*b + c*d` costs ~7 push/pop-shuffling dispatches. A **register machine** names @@ -70,128 +110,580 @@ index into an array of `Mem` registers (typed value slots), so `Add r1 r2 r3` is one instruction and intermediate values just *stay put*. Fewer instructions = fewer dispatches = less interpreter tax per row; the price is that the code generator must do register -allocation (decide which value lives in which slot). Alongside the -registers sits an array of **cursors** — open positions inside -B-trees (topic 1's structure; a cursor is "where I am in table t") — -which the opcode set manipulates directly (OpenRead, Rewind, Next, -Column). +allocation (decide which value lives in which slot). + +Do the count, because it is the whole argument: + +``` + a*b + c*d on a STACK machine (JVM-style): + load a; load b; mul; load c; load d; mul; add + = 7 dispatched instructions, and every intermediate goes + through the operand stack (a store + a load each) + + a*b + c*d on the VDBE (register machine): + Column 0 ia r1 ; a → r1 + Column 0 ib r2 ; b → r2 + Multiply r1 r2 r5 ; r5 = r1*r2 ← operands NAMED + Column 0 ic r3 + Column 0 id r4 + Multiply r3 r4 r6 + Add r5 r6 r7 ; r7 = r5+r6 + = 7 dispatched instructions too — BUT the four Column ops are + work the stack machine also needs. Compare arithmetic only: + stack: mul, mul, add + 3 implicit push/pop pairs + register: mul, mul, add + 0 shuffling ops + 3 dispatches vs 7 → 2.3× fewer, for the same three flops. +``` + +Alongside the registers sits an array of **cursors** — open +positions inside B-trees (topic 1's structure; a cursor is "where I +am in table t") — which the opcode set manipulates directly +(OpenRead `:4421`, Rewind `:6407`, Next `:6545`, Column `:3010`). ### Step 3 — the ISA: fixed-width ops, one convention per field -The instruction format (vdbeInt.h:55) is a fixed struct — the entire -instruction set is 199 opcodes over this one shape: +> **In:** Step 2's operand convention. **Out:** the exact byte +> layout of one instruction, the real size of the instruction set, +> and the reason forward jumps need only one codegen pass. + +**Corrected anchor.** The struct is in **`src/vdbe.h:55`**, not +`src/vdbeInt.h:55`. `vdbeInt.h` only carries `typedef struct VdbeOp +Op;` at its line 46. The old anchor pointed at the right line number +of the wrong file, which is the most expensive kind of wrong: it +looks checkable and isn't. ```c -struct VdbeOp { - u8 opcode; /* one byte, 199 used */ - signed char p4type; /* what the union holds */ - u16 p5; /* flags */ - int p1, p2, p3; /* register/cursor/jump operands */ - union p4 { int i; char *z; ... KeyInfo*, FuncDef* ... }; -} +// sqlite/src/vdbe.h — struct VdbeOp, 55-95, p4 union elided + 55 struct VdbeOp { + 56 u8 opcode; /* What operation to perform */ + 57 signed char p4type; /* One of the P4_xxx constants for p4 */ + 58 u16 p5; /* Fifth parameter is an unsigned 16-bit integer */ + 59 int p1; /* First operand */ + 60 int p2; /* Second parameter (often the jump destination) */ + 61 int p3; /* The third parameter */ + 62 union p4union { /* fourth parameter */ +// ... 63-81: 16 alternatives — int, char*, FuncDef*, CollSeq*, Mem*, +// ... KeyInfo*, SubProgram*, Table*, Index*, … ... + 82 } p4; + 83 #ifdef SQLITE_ENABLE_EXPLAIN_COMMENTS + 84 char *zComment; /* Comment to improve readability */ + 85 #endif +// ... 86-93: iSrcLine (SQLITE_VDBE_COVERAGE), nExec/nCycle +// ... (SQLITE_ENABLE_STMT_SCANSTATUS || VDBE_PROFILE) ... + 94 }; + 95 typedef struct VdbeOp VdbeOp; ``` -Fixed 24-ish-byte ops, arrays not linked lists — the program is -cache-linear. p2 is *always* the jump target by convention, so the -code generator can fix up forward jumps in one pass. Compare -Umbra's IR (also fixed-width, also single-pass-friendly): same -instinct, different target (interpretation vs fast native lowering). +Compute the size, because "fixed 24-ish bytes" is the kind of claim +that should never be approximate: + +``` + opcode u8 1 byte + p4type signed char 1 + p5 u16 2 + p1,p2,p3 int × 3 12 subtotal 16, already 8-aligned + p4 union 8 (largest member is a pointer) + ---- + plain 64-bit build: 24 bytes exactly + + with SQLITE_ENABLE_EXPLAIN_COMMENTS: +8 (zComment) = 32 + with SQLITE_VDBE_COVERAGE: +8 (u32 + pad) = 40 + with VDBE_PROFILE: +16 (nExec, nCycle) = 56 +``` + +24 bytes means **2.67 instructions per 64-byte cache line**; a +1000-op program is 24 KB and fits in L1d on most machines. It also +means the *debug* builds you measure with are 2.3× wider than the +shipping one — measure the plain build (Step 7). + +**Corrected count: the ISA is 190 opcodes, not 199.** Count it the +way SQLite's own build does. `src/vdbe.c:1060-1065` documents the +convention: + +> "The makefile for SQLite generates two C files "opcodes.h" and +> "opcodes.c" by scanning this file looking for lines that **begin +> with** "case OP_"." + +So the opcode count is the number of *flush-left* `case OP_` lines: + +``` + grep -cE '^case OP_' src/vdbe.c → 190 ← the ISA + grep -cE 'case OP_' src/vdbe.c → 199 ← the naive count + + the 9 extra are: + :1062 the documentation comment quoted above + :1948-1951 (4) the INNER switch inside OP_Add's body + :1976-1979 (4) a THIRD switch, for the floating-point path +``` + +The naive 199 is what the topic README still prints. Hold on to +those 8 inner cases — they are Step 4's punchline. + +p2 is *always* the jump target by convention (`vdbe.h:60`: "often +the jump destination"), and every jumping opcode funnels through one +shared label: + +```c +// sqlite/src/vdbe.c — the shared jump tail, 1219-1225 (inside OP_InitCoroutine) +1219 /* Most jump operations do a goto to this spot in order to update +1220 ** the pOp pointer. */ +1221 jump_to_p2: +1222 assert( pOp->p2>0 ); /* There are never any jumps to instruction 0 */ +1223 assert( pOp->p2nOp ); /* Jumps must be in range */ +1224 pOp = &aOp[pOp->p2 - 1]; +1225 break; +``` + +Line 1224 is why forward jumps need one codegen pass: the target is +an operand the generator can patch in place later, and `-1` +compensates for the `pOp++` in the loop header at :966. Nothing in +the program moves when a jump is resolved, because instructions are +fixed-width. + +Compare Umbra's IR — the instinct is the same (contiguous array, +integer offsets, single-pass friendly) but the layout is *not*: see +`reading-umbra-tidy-tuples.md`, where the instructions are +**variable-length** because Umbra never has to index into the middle +of one at runtime, only append. ### Step 4 — dispatch cost: what bytecode buys and what it doesn't -The interpreter core is one `switch` on the opcode — compiled to one -**indirect branch** (a jump whose target comes from data, so the CPU -must *predict* where it goes). The predictor sees ONE hot indirect -jump with 199 possible targets — mispredict-prone (topic 17's -branchy filter, interpreter edition). Threaded dispatch (a computed -goto at the end of *each* opcode) gives the predictor per-op -history; SQLite gains limited benefit and keeps the portable switch -by default (look for the OP_-macros and perf notes). +> **In:** Step 3's 190-case instruction set and Step 2's dispatch +> count per expression. **Out:** a cycle price per dispatch, and +> the work-per-op ratio that decides whether that price matters — +> the number that defines a JIT's entire opening. + +The interpreter core is one `switch` on the opcode, at +`src/vdbe.c:1049`, closing at `:9357` — 8,300 lines of cases. A +`switch` over a dense integer range compiles to one **indirect +branch** (a jump whose target comes from data, so the CPU must +*predict* where it goes) through a jump table. The predictor sees +ONE hot indirect jump with 190 possible targets — mispredict-prone +(topic 17's branchy filter, interpreter edition). + +**Corrected claim.** The previous version of this guide said +SQLite "gains limited benefit and keeps the portable switch by +default" from threaded dispatch. That implies a threaded option +exists in the source and is switched off. At `951de30` it does not: + +``` + grep -niE 'computed.goto|COMPUTED_GOTO|dispatch_table|&&CASE' src/vdbe.c + → no matches +``` + +There is exactly one `switch` in `vdbe.c` (`:1049`, plus the two +*inner* ones inside OP_Add), and exactly two ways out of an opcode +body: `break` (fall to the loop's `pOp++`) or `goto jump_to_p2` +(`:1221`). The honest sentence is: **SQLite's VDBE uses a single +portable `switch`; the threaded-dispatch alternative is not +implemented in this file.** Postgres's *expression* interpreter, by +contrast, really does thread — `execExprInterp.c:119-122` defines +`EEO_DISPATCH()` as `goto *((void *) op->opcode)` — so the contrast +this guide wants is available; it is just across projects, not +inside SQLite. See `reading-postgres-jit.md`. Either way you pay ~5–20 cycles of dispatch per op. The verdict -depends entirely on what an op *does*: dispatch is noise when the op -is a B-tree step (hundreds of cycles of real work), brutal when the -op is `Add r1 r2 r3` (1 cycle of work, 20 of dispatch) executed -millions of times. That per-op work ratio is the JIT's entire -opening — and SQLite's few-rows workload simply doesn't have it. +depends entirely on what an op *does*. Measure that with a ruler — +the line span of each case body: + +| opcode | body | lines | dispatch as a share of the op | +|---|---|---|---| +| `OP_Column` | `:3010`–`:3339` | ~330 | noise — record decode dominates | +| `OP_Add`/`Sub`/`Mul`/`Div`/`Rem` | `:1926`–`:2010` | ~85 | still not one flop | +| `OP_Goto` | `:1098`–`:1104` | ~7 | dispatch *is* the op | + +**The `OP_Add` correction is the important one.** The previous +version called it "~10 lines". It is 85, and its shape is the best +argument for a JIT anywhere in this topic: + +```c +// sqlite/src/vdbe.c — OP_Add's shared body, 1938-1948 and 1975-1976 +1938 pIn1 = &aMem[pOp->p1]; +1939 type1 = pIn1->flags; +1940 pIn2 = &aMem[pOp->p2]; +1941 type2 = pIn2->flags; +1942 pOut = &aMem[pOp->p3]; +1943 if( (type1 & type2 & MEM_Int)!=0 ){ +1944 int_math: +1945 iA = pIn1->u.i; +1946 iB = pIn2->u.i; +1947 switch( pOp->opcode ){ +1948 case OP_Add: if( sqlite3AddInt64(&iB,iA) ) goto fp_math; break; +// ... 1949-1974: the other four integer ops, the NULL path, numericType() ... +1975 switch( pOp->opcode ){ +1976 case OP_Add: rB += rA; break; +``` + +Count what one `a+b` on two doubles costs here: an outer dispatch at +:1049, three register-array loads (:1938–:1942), a dynamic type test +(:1943), a fall to the float path, two `sqlite3VdbeRealValue` calls, +**a second dispatch on the same opcode at :1975**, and then — at +:1976 — the single `addsd` that is the actual work. The interpreter +switches on `pOp->opcode` *twice* for one arithmetic instruction, +and re-derives the operand types on every row even though they were +knowable at prepare time. + +That is exactly the overhead a JIT deletes: types are resolved once +at compile time, so the emitted code for `a+b` is one `addsd` with +no dispatch, no flag test, and no second switch. This topic's own +lane measures the same effect from the other side — the tree-walking +interpreter in `experiments/src/interp.rs:8-16` falls from 89.4 to +0.95 M rows/s (**94×**) as an expression grows 7 → 511 nodes, while +the vectorized lane falls only 47×, because the vectorized lane pays +its dispatch once per node per *batch*. + +And SQLite's few-rows workload simply doesn't have the ratio that +would justify paying compile time. Do the division in Step 6. ### Step 5 — coroutines: the feature flattening gives you for free +> **In:** Step 1's "program counter is a pointer into an array". +> **Out:** a suspend/resume primitive that costs one integer — the +> concrete answer to Neumann's complaint about resumability state. + A **coroutine** is a function that can suspend mid-execution and be resumed later. For a tree-walking interpreter, suspension is hard — the "where was I" state is a native call stack. For flattened bytecode it is trivial: the entire position is one integer, the -program counter. OP_InitCoroutine/OP_Yield (vdbe.c:1209, :1264) -exploit this: a subquery becomes a coroutine whose pc lives in a -register, and Yield just swaps pc values — so +program counter. + +```c +// sqlite/src/vdbe.c — OP_Yield, the whole opcode, 1264-1274 +1264 case OP_Yield: { /* in1, jump0 */ +1265 int pcDest; +1266 pIn1 = &aMem[pOp->p1]; +1267 assert( VdbeMemDynamic(pIn1)==0 ); +1268 pIn1->flags = MEM_Int; +1269 pcDest = (int)pIn1->u.i; +1270 pIn1->u.i = (int)(pOp - aOp); +1271 REGISTER_TRACE(pOp->p1, pIn1); +1272 pOp = &aOp[pcDest]; +1273 break; +1274 } +``` + +Lines 1269–1272 are the entire context switch: read the other side's +pc out of register p1, write *my* pc into the same register, jump. +One `Mem` slot holds a whole coroutine's resumption state, and +`OP_InitCoroutine` seeds it at `:1215` with `pOut->u.i = pOp->p3 - +1` (the `-1` again compensating for `pOp++`). A subquery becomes a +coroutine whose pc lives in a register, so `INSERT INTO t SELECT ...` streams rows from the SELECT program -without materializing it. This is the same resumability argument as -topic 7's io_uring state machines and Neumann's Q2 pull-model pain. +without materializing it. + +This is the same resumability argument as topic 7's io_uring state +machines and — importantly — the *inverse* of Neumann's §1 complaint +that operator resumption bookkeeping is "bad code locality and +complex book-keeping". Neumann's escape is to make code that never +needs to resume. SQLite's is to make resumption cost one integer. +Both are valid; they optimize different workloads. Question 3. ### Step 6 — where the VDBE sits, and what transfers to M19 -Place it on the topic's spectrum: FalkorDB's eval.rs walks an -expression tree per row — it sits LEFT of SQLite. M19's cranelift -JIT jumps two steps right. The VDBE lesson: there is a defensible +> **In:** Steps 1–5's machine and its per-op cost. **Out:** a +> break-even row count you compute, and a placement on this topic's +> spectrum. + +Place it on the topic's spectrum: FalkorDB's expression evaluator +walks a tree per row — it sits LEFT of SQLite. M19's cranelift JIT +jumps two steps right. The VDBE lesson: there is a defensible middle (flatten to a register program, interpret that) that costs -zero compile time and already kills tree-walk overhead — worth -benching as a fourth lane in jit_bench if the JIT crossover -disappoints (question 5). And why SQLite never JITs: its queries -touch a handful of rows, so no compile cost — however small — can -amortize. +zero compile time and already kills tree-walk overhead. + +Now price SQLite's refusal to JIT, using this topic's own measured +rates (`notes.md`, Apple M3 Pro, 2026-07-10, depth 8 = 511 nodes): + +``` + rows_breakeven = compile_time / (per_row_interp − per_row_jit) + + interpreter, 511 nodes: 0.95 M rows/s → 1.053 µs/row + vectorized, 511 nodes: 11.8 M rows/s → 0.0847 µs/row + saving if a JIT merely matched the vectorized lane: + 1.053 − 0.0847 = 0.968 µs/row + + a 500 µs cranelift compile pays back after + 500 / 0.968 = 516 rows + + SQLite's median query touches far fewer than 516 rows. At 5 rows: + JIT total = 500 µs + 5 × 0.085 µs = 500.4 µs + interp = 5 × 1.053 µs = 5.3 µs + the JIT is 94× SLOWER end to end. +``` + +That is the whole answer to "why has SQLite never JIT'd". It is not +conservatism; it is the arithmetic of a workload where the compile +fee is never amortized. Change the workload — an OLAP scan of 10 +million rows — and the same division gives the opposite verdict: +compile 500 µs, save 9.68 s. + +Worth benching a fourth lane in jit_bench if the JIT crossover +disappoints (question 5): flattening our `Expr` to a `Vec` costs +no compile latency at all and should land between `interp` and +`vector`. + +### Step 7 — how to read this file: turn on VDBE_PROFILE + +> **In:** the source at the pin, and a build you control. +> **Out:** a per-opcode cycle count, so every claim in Steps 4–6 +> becomes something you measured rather than something you read. + +`vdbe.c` ships with the instrument already installed. Three sites: + +```c +// sqlite/src/vdbe.c — the VDBE_PROFILE cycle counter, 27-31 / 974-977 / 9320-9322 + 27 #if defined(VDBE_PROFILE) \ + 28 || defined(SQLITE_PERFORMANCE_TRACE) \ + 29 || defined(SQLITE_ENABLE_STMT_SCANSTATUS) + 30 # include "hwtime.h" + 31 #endif +// ... 32-973: everything up to the top of the dispatch loop ... + 974 #if defined(VDBE_PROFILE) + 975 pOp->nExec++; + 976 pnCycle = &pOp->nCycle; + 977 if( sqlite3NProfileCnt==0 ) *pnCycle -= sqlite3Hwtime(); +// ... 978-9319: the switch and all 190 opcode bodies ... +9320 #if defined(VDBE_PROFILE) +9321 *pnCycle += sqlite3NProfileCnt ? sqlite3NProfileCnt : sqlite3Hwtime(); +9322 pnCycle = 0; +``` + +Subtract the timestamp before the opcode, add it after: `nCycle` +accumulates real cycles per instruction *slot* (not per opcode kind +— per address in the program), and `nExec` counts executions. Build +with `-DVDBE_PROFILE`, run your query, and you get the measured +version of Step 4's table for your own workload. Remember Step 3: +this build's `VdbeOp` is 56 bytes, not 24, so absolute cache +behaviour differs — use it for *ratios* between opcodes. + +Two neighbouring hooks are worth knowing: `SQLITE_DEBUG` adds an +operand sanity pass over p1/p2/p3 at `:1009-1044` before every +dispatch (so never benchmark a `SQLITE_DEBUG` build), and +`SQLITE_ENABLE_STMT_SCANSTATUS` is the same counters exposed through +the public `sqlite3_stmt_scanstatus()` API without a custom build. ## Where each step lives in the code | anchor | what it is | step | |---|---|---| -| src/vdbe.c:1049 | THE loop: `switch( pOp->opcode )` | 1, 4 | -| src/vdbe.c:1062 | comment: file is ordered by `case OP_` convention | 1 | -| src/vdbeInt.h:55 | `struct VdbeOp` — opcode, p1,p2,p3 ints, p4 union, p5 flags | 3 | -| src/vdbe.c:1098 | OP_Goto — jump = set pOp, `break` re-enters switch | 4 | -| src/vdbe.c:1154 / :1187 | OP_Gosub / OP_Return — subroutines via a register | 2, 5 | -| src/vdbe.c:1209 / :1264 | OP_InitCoroutine / OP_Yield — coroutines! | 5 | -| src/vdbe.c:1284 | OP_HaltIfNull — constraint checks as opcodes | 3 | -| 199 `case OP_` total | the entire ISA | 3 | - -Start at the dispatch loop (:1049) and read opcodes in file order — -the `case OP_` comment convention makes the 199-case file navigable. -Keep an `EXPLAIN` output from Step 1 beside you and find each opcode -it uses; Step 5's coroutine pair is the detour worth taking whole. +| `src/vdbe.h:55-95` | `struct VdbeOp` — 24 bytes: opcode, p4type, p5, p1..p3, p4 union | 3 | +| `src/vdbeInt.h:46` | `typedef struct VdbeOp Op;` — the *only* thing here (old anchor's target) | 3 | +| `src/vdbe.c:966` | THE loop header: `for(pOp=&aOp[p->pc]; 1; pOp++)` | 1 | +| `src/vdbe.c:1049` | THE dispatch: `switch( pOp->opcode )`, closing at `:9357` | 1, 4 | +| `src/vdbe.c:1060-1065` | the flush-left `case OP_` convention mkopcodeh scans for | 3 | +| `src/vdbe.c:1098` | OP_Goto — 7 lines; dispatch *is* the op | 4 | +| `src/vdbe.c:1154` / `:1187` | OP_Gosub / OP_Return — subroutines via a register | 2, 5 | +| `src/vdbe.c:1209-1226` | OP_InitCoroutine (`:1215` seeds pc) + the shared `jump_to_p2` tail | 3, 5 | +| `src/vdbe.c:1238` / `:1264-1274` | OP_EndCoroutine / OP_Yield — the pc swap | 5 | +| `src/vdbe.c:1284` | OP_HaltIfNull — constraint checks as opcodes | 3 | +| `src/vdbe.c:1926-2010` | OP_Add and friends — 85 lines, with switches at `:1947` and `:1975` | 4 | +| `src/vdbe.c:3010-3339` | OP_Column — ~330 lines of record decode | 4 | +| `src/vdbe.c:27-31`, `:974-977`, `:9320-9322` | VDBE_PROFILE per-opcode cycle accounting | 7 | +| `grep -cE '^case OP_'` → **190** | the entire ISA | 3 | + +Start at the loop header (`:966`), then the dispatch (`:1049`), then +read opcodes in file order — the flush-left `case OP_` convention +makes the 190-case file navigable. Keep an `EXPLAIN` output from +Step 1 beside you and find each opcode it uses; Step 5's coroutine +pair is the detour worth taking whole. Read `OP_Add` (`:1926`) and +`OP_Column` (`:3010`) back to back — that contrast is Step 4. ## Questions for notes.md 1. Run `EXPLAIN SELECT a+1 FROM t WHERE b<10` (any SQLite). Paste the program; identify the loop (Rewind/Next), the filter (Ge/Lt with p2 jump), the expression ops. How many dispatched ops per - row? + row? Cross-check your hand count against + `sqlite3_stmt_status(stmt, SQLITE_STMTSTATUS_VM_STEP, 0)`, which + returns the `nVmStep` incremented at `src/vdbe.c:972`. 2. Register machine vs stack machine: count the ops `a*b + c*d` - needs on each. Why did SQLite pick registers (fewer dispatches, - at the cost of codegen doing register allocation)? -3. OP_Yield: trace pc swapping between coroutine and caller. What - exactly is saved/restored (ONE register holding pc — why is - that sufficient, i.e. where do the coroutine's locals live)? -4. Why is `case OP_Column` (the B-tree record decoder) enormous - while `case OP_Add` is ~10 lines — and what does that say about - where VDBE dispatch overhead actually matters? + needs on each (Step 2 does it — redo it without looking). Why + did SQLite pick registers, and what did it have to buy with that + choice (register allocation in the code generator)? +3. OP_Yield: trace the pc swap at `src/vdbe.c:1269-1272` between + coroutine and caller. What exactly is saved/restored (ONE + register holding pc — why is that sufficient, i.e. where do the + coroutine's locals live)? Then answer the inverse: what would + this cost in Neumann's compiled pipeline, where there is no + program counter to save? +4. `OP_Column` is ~330 lines (`:3010-3339`) and `OP_Add` is ~85 + (`:1926-2010`) — but only ~3 of `OP_Add`'s lines are arithmetic. + Compute the ratio of *useful work* to dispatch for each, and say + which one a JIT should target first. Bonus: what do the two + inner switches at `:1947` and `:1975` cost, and why can't the + compiler hoist them? 5. Sketch the fourth lane: a bytecode compiler for our `Expr` enum (flatten to `Vec` with register slots, interpret with one - match). Predict where it lands between interp and JIT in - rows/s, then (stretch) build it and check. + match). Predict where it lands between `interp` (0.95 M rows/s + at 511 nodes) and `vector` (11.8 M rows/s) — then (stretch) + build it and check. Justify the prediction with a dispatch + count, not a feeling. ## Done when -- [ ] You can explain what flattening an AST into bytecode buys before any compilation is involved — and check it against this topic's measured interpreter numbers, which fall 94x from 7 nodes to 511. +Answer each before unfolding it. + +- [ ] You can explain what flattening an AST into bytecode buys before any compilation is involved — and check it against this topic's measured interpreter numbers, which fall 94× from 7 nodes to 511. + +
Answer + + It buys three things, none of which require a compiler: (1) the + program is contiguous, so the pc is a pointer increment + (`vdbe.c:966`) and the prefetcher sees a linear stream, instead of + chasing `Box` pointers; (2) the recursion is gone — one loop + replaces a call per node, so there is no stack traffic per node; + (3) any work that can be resolved at prepare time (register + assignment, jump targets, affinities) is resolved once instead of + per row. What it does *not* buy is elimination of per-op dispatch + or of dynamic type tests — `OP_Add` still switches twice + (`:1947`, `:1975`) and still tests `MEM_Int` flags at `:1943` + every row. That residue is what this topic's 94× degradation + measures: `experiments/src/interp.rs` goes 89.4 → 0.95 M rows/s + from 7 to 511 nodes because the per-node dispatch cost is paid per + row and scales with node count. Flattening removes the pointer + chase; only compiling removes the dispatch. + +
+ - [ ] You can say why a register machine beats a stack machine here, and count the ops for `a*b + c*d` under each. -- [ ] You can explain what dispatch cost bytecode removes and what it leaves behind. + +
Answer + + A stack machine needs 7 dispatched instructions (`load a, load b, + mul, load c, load d, mul, add`) and every intermediate makes a + round trip through the operand stack. The VDBE needs 4 `Column` + ops (which the stack machine also needs) plus exactly 3 arithmetic + ops (`Multiply r1 r2 r5`, `Multiply r3 r4 r6`, `Add r5 r6 r7`) + because p1/p2/p3 name the operands directly — 3 dispatches vs 7 + for the same three flops, and zero shuffling. The price is that + the code generator must do register allocation. SQLite pays it + because dispatch, at ~5–20 cycles, is the expensive resource and + compile-time register allocation is free. + +
+ +- [ ] You can explain what dispatch cost bytecode removes and what it leaves behind, and state correctly how SQLite dispatches. + +
Answer + + Removed: the per-node recursive call, the pointer chase, and the + re-parse. Left behind: one indirect branch per opcode through a + 190-target jump table at `vdbe.c:1049`, ~5–20 cycles depending on + prediction, *plus* whatever dynamic type dispatch the opcode body + does itself. On the correction: at pin `951de30` SQLite has **no + computed-goto / threaded-dispatch option** — grepping + `computed goto|dispatch_table|&&CASE` in `vdbe.c` finds nothing. + It is one portable `switch`, exited by `break` or `goto + jump_to_p2` (`:1221`). The project that *does* thread its + expression interpreter is Postgres — + `execExprInterp.c:119-122`, `EEO_DISPATCH()` → `goto *((void *) + op->opcode)`. + +
+ - [ ] You can trace `OP_Yield` and explain how flattening gives coroutines for free. + +
Answer + + `vdbe.c:1264-1274`. `pIn1` is register p1. Line 1269 reads the + destination pc out of it; line 1270 writes the *current* pc (`pOp + - aOp`, an integer offset) back into the same register; line 1272 + sets `pOp = &aOp[pcDest]`. A single `Mem` slot is the entire + saved context of a coroutine. It is free because flattening made + the execution position a scalar: with a tree walk the position is + a native call stack, which you cannot store in a register. Locals + live in the coroutine's own VDBE registers, which are never + reused by the caller, so nothing else needs saving. + `OP_InitCoroutine:1215` seeds the register with `p3 - 1`. + +
+ +- [ ] You can compute why SQLite has never needed a JIT, from a compile time and two measured per-row rates. + +
Answer + + `rows = compile_time / (per_row_interp − per_row_jit)`. Using this + topic's depth-8 measurements (`notes.md`): interpreter 0.95 M + rows/s = 1.053 µs/row, vectorized 11.8 M rows/s = 0.0847 µs/row, + saving 0.968 µs/row. A 500 µs compile breaks even at **516 rows**. + SQLite's design centre is queries touching a handful of rows: at + 5 rows the JIT costs 500.4 µs against the interpreter's 5.3 µs — + 94× *slower*. The refusal is arithmetic, not conservatism. + Neumann's Table 1 shows the same effect from the OLTP side: TPC-C + transactions touch under 30 tuples, so switching HyPer's codegen + from C++ to LLVM bought +4.8% throughput and a 20× compile-time + reduction — the compile time was the only thing that mattered. + +
+ +- [ ] You can state the true size and layout of the instruction set, and how you counted. + +
Answer + + **190 opcodes**, counted the way SQLite's own build counts them: + `mkopcodeh` scans `vdbe.c` for lines that *begin with* `case OP_` + (the convention is documented at `:1060-1065`), so + `grep -cE '^case OP_' src/vdbe.c` → 190. The naive + `grep -cE 'case OP_'` gives 199 because it also catches the + documentation comment at `:1062` and the eight inner-switch cases + at `:1948-1951` and `:1976-1979`. Layout: `struct VdbeOp` at + **`src/vdbe.h:55`** (not `vdbeInt.h`) — `u8 opcode` + `signed char + p4type` + `u16 p5` + three `int` operands + an 8-byte union = 24 + bytes exactly in a plain 64-bit build, growing to 32/40/56 under + `SQLITE_ENABLE_EXPLAIN_COMMENTS`, `SQLITE_VDBE_COVERAGE` and + `VDBE_PROFILE`. + +
+ - [ ] You wrote answers to all five questions in notes.md, including a sketch of a bytecode lane for this topic's `Expr` enum. +
Answer + + The sketch that matches this chapter: lower `Expr` + (`experiments/src/expr.rs:11-21`) to `Vec` in postorder, with + a register counter assigning each node a slot — `Op::Col{col, + dst}`, `Op::Const{v, dst}`, `Op::Add{a, b, dst}`, and so on — then + interpret with one `for op in &prog { match op { … } }` over a + `Vec` register file. Prediction to justify with numbers: it + removes the pointer chase and the recursive call but keeps one + dispatch per node per row, so it should land close to `interp` + (0.95 M rows/s at 511 nodes) — perhaps 1.5–3× better — and far + below `vector` (11.8 M rows/s), which pays one dispatch per node + per *batch*. If it lands near `vector`, your interpreter lane was + measuring allocation, not dispatch. + +
+ ## References -**Code** -- [sqlite](https://github.com/sqlite/sqlite) `src/vdbe.c` — start at - the dispatch loop (:1049) and read opcodes in file order; the - `case OP_` comment convention makes it navigable -- [sqlite](https://github.com/sqlite/sqlite) `src/vdbeInt.h` — - `struct VdbeOp` and the register/cursor state +**Code** — all anchors verified at sqlite `951de30` + +| anchor | what | +|---|---| +| `src/vdbe.c:966` | the loop header (`pOp++` — the pc is a pointer) | +| `src/vdbe.c:1049`–`:9357` | the one `switch`, and its close | +| `src/vdbe.c:1060-1065` | the `case OP_` convention that defines the ISA size | +| `src/vdbe.c:1221-1225` | `jump_to_p2`, the shared jump tail | +| `src/vdbe.c:1264-1274` | `OP_Yield` — a coroutine switch in 4 lines | +| `src/vdbe.c:1926-2010` | `OP_Add`, with its two inner switches | +| `src/vdbe.c:3010-3339` | `OP_Column`, the other end of the work-per-op scale | +| `src/vdbe.c:27-31`, `:974-977`, `:9320-9322` | `VDBE_PROFILE` | +| `src/vdbe.h:55-95` | `struct VdbeOp` | +| `src/vdbeInt.h:46` | `typedef struct VdbeOp Op;` | + +Fetch any of these without a clone: +`python3 tools/pinned-source.py show sqlite src/vdbe.c -r 966:980`. + +**Elsewhere in this repo** +- `experiments/src/interp.rs:8-16` — the tree walk the VDBE replaces +- `reading-postgres-jit.md` — the threaded-dispatch interpreter + SQLite does *not* have (`execExprInterp.c:119-122`) +- `reading-umbra-tidy-tuples.md` — the contrasting IR layout + (contiguous but variable-length) +- `reading-neumann-vldb11.md` — §1's resumability complaint, of + which `OP_Yield` is the cheerful counter-example + +**Tools** - `EXPLAIN` in any sqlite3 shell — the fastest way to see programs +- `-DVDBE_PROFILE` + `hwtime.h` — per-opcode cycles (Step 7) +- `sqlite3_stmt_scanstatus()` — the same counters without a custom + build (`SQLITE_ENABLE_STMT_SCANSTATUS`) diff --git a/topics/19-jit/reading-umbra-tidy-tuples.md b/topics/19-jit/reading-umbra-tidy-tuples.md index 4ae544b..e3e37d5 100644 --- a/topics/19-jit/reading-umbra-tidy-tuples.md +++ b/topics/19-jit/reading-umbra-tidy-tuples.md @@ -1,220 +1,819 @@ # Umbra & copy-and-patch: the war on compile latency Two attacks on the same enemy: compile LATENCY. HyPer proved -compiled queries run fast; production taught that 100 ms of LLVM -before a 10 ms query is a loss. Umbra's answer is a bespoke IR and -a tiered backend; copy-and-patch's answer is to do the compiling -at BUILD time and only memcpy at runtime. This chapter builds the -ideas in order — why LLVM is structurally slow, what an IR designed -for single-pass lowering looks like, how adaptive execution makes -the interpret-vs-compile choice unnecessary, and how far the -stencil trick pushes the floor — then routes you through both -papers. +compiled queries run fast; production taught that tens of +milliseconds of LLVM before a one-millisecond query is a loss. +Umbra's answer is a bespoke IR and a tiered backend; +copy-and-patch's answer is to do the compiling at BUILD time and +only memcpy at runtime. This chapter builds the ideas in order — +why LLVM is structurally slow, what an IR designed for single-pass +lowering looks like, how adaptive execution makes the +interpret-vs-compile choice unnecessary, and how far the stencil +trick pushes the floor — then routes you through both papers. + +**Sources.** Every number below is quoted with the table or figure +it came from, because the previous version of this guide carried +three that turned out to be wrong (Steps 4, 5 and 7 flag each). +The two papers: + +- Kersten, Leis, Neumann, *"Tidy Tuples and Flying Start: Fast + Compilation and Fast Execution of Relational Queries in Umbra"*, + **VLDB Journal 2021**. +- Xu, Kjolstad, *"Copy-and-Patch Compilation"*, **OOPSLA 2021** + ([arXiv:2011.13127](https://arxiv.org/abs/2011.13127)). + +Umbra's setup (§5.1): a 10-core Intel Skylake X i9-7900X at 3.4 GHz +(4.5 GHz turbo). TPC-H at SF=0.01 for the latency tables, SF=1 for +the throughput ones. Note the scale factors — they are the whole +argument, and quoting an Umbra number without its SF is how these +claims go wrong. ## The problem in one sentence -A short OLTP query executes in under a millisecond but LLVM -O3 -needs tens of milliseconds to compile it — a compile-to-run ratio -that can exceed **100:1** — so the fastest generated code in the -world loses to an interpreter unless compilation itself gets ~100× -cheaper. +A short OLAP query at SF=0.01 executes in **0.50 ms** (Umbra, +Table 2 geometric mean) but generating fully optimized LLVM code +for a large query can take **150 seconds** (Fig. 13, 2000 joins) — +so the fastest generated code in the world loses to an interpreter +unless compilation itself gets two orders of magnitude cheaper. ## The concepts, step by step ### Step 1 — the latency budget: name the enemy in numbers -Query compilation (topic recap: generate machine code per query -instead of interpreting the plan) has one price tag that HyPer's -LLVM backend made visible: +> **In:** topic 11's compiled-vs-interpreted spectrum and topic +> 19's own measured interpreter penalty. **Out:** the compile:run +> ratio, stated with real numbers and a named scale factor — the +> quantity every later step is trying to shrink. +**Compile latency** is time spent generating code before the first +row is produced. It is paid on every query, hit or miss, and it +does not amortize across queries the way an index build does. Two +measured pairs, both from the Umbra paper: + +``` + Umbra, TPC-H SF=0.01, 1 thread, geometric mean over 22 queries + (Table 2, "Σ" = plan + code generation + x86 + execution): + + Umbra: plan 0.25 + cdg. 0.20 + x86 0.21 + exec 0.50 = 1.24 ms + HyPer: plan 0.26 + cdg. 0.60 + bc. 0.47 + exec 3.33 = 5.06 ms + DuckDB: plan 0.47 + exec 5.72 = 6.40 ms + MonetDB: plan 0.53 + exec 0.84 = 1.46 ms + PostgreSQL: plan 1.53 + exec 8.50 = 10.82 ms + + Preparation time (everything before exec), §5.3's own arithmetic: + Umbra 0.25 + 0.20 + 0.21 = 0.66 ms + HyPer = 1.33 ms ("Umbra starts faster") + DuckDB 0.47 (an INTERPRETER) + MonetDB 0.53 (an INTERPRETER) + + So: Umbra pays 0.66 ms to prepare and 0.50 ms to execute. + compile:run = 0.66 / 0.50 = 1.32 : 1 + The compile side is BIGGER than the run side, and Umbra is the + fastest system in the table. That is the enemy. ``` - HyPer, TPC-H Q1 scale: LLVM -O3 compile ≈ tens of ms - short OLTP query: execution ≈ sub-ms - ⇒ compile:run ratio can exceed 100:1 - Umbra target: compile in ~100 µs — "Flying Start" +Note what Table 2 does *not* include, and say it out loud whenever +you quote these numbers: the paper excludes LLVM compilation from +the Umbra and HyPer rows "as its compile times are too long for a +data set this small". If LLVM were in that table it would not be a +1.32:1 ratio; it would be off the page. Fig. 13 shows how far off: + ``` + Fig. 13 — self-join of TPC-H `nation`, 2000 joins, SF=1, 1 thread. + That query generates 108,000 Umbra IR instructions, "the vast + majority ... in a single function". -Why it matters: compile latency is paid *before the first row*, on -every query, hit or miss — so it sets the minimum query size for -which a JIT is rational at all. Everything in this chapter is a way -to shrink that minimum. + LLVM (default pipeline) 150 seconds + LLVM Fast ISel 4 seconds + Flying Start < 0.04 seconds + + Ratios (do the division; the paper gives only the three times): + LLVM / Flying Start > 150 / 0.04 = 3,750× + Fast ISel / Flying Start > 4 / 0.04 = 100× +``` + +Everything in this chapter is a way to shrink that ratio. ### Step 2 — why LLVM is slow: the cost is structural, not a flag -LLVM is a general-purpose optimizing compiler: it builds **SSA** +> **In:** Step 1's 150-second data point. **Out:** the *mechanism* +> behind it — and the reason `-O0` does not rescue you, which is +> what licenses building a whole new backend rather than tuning +> flags. + +LLVM is a general-purpose optimizing compiler. It builds **SSA** form (static single assignment — every value defined exactly once, -which makes optimization clean but construction expensive), runs -~100 IR passes, then does instruction selection and register -allocation — each a multi-pass traversal over pointer-linked graph -structures. No `-O0` flag removes the graph-building and -multi-pass skeleton. Umbra's observation: *query* code is -generated, regular, and short-lived — short straight-line blocks, -few live values, no human weirdness — so it doesn't need a general -optimizer. A compiler specialized to that shape can be linear. +which makes dataflow analysis clean but construction expensive), +runs a long pass pipeline, then does instruction selection and +register allocation, each a multi-pass traversal over pointer-linked +graph structures. No `-O0` flag removes the graph building or the +multi-pass skeleton — Fig. 13's middle panel is precisely that +experiment, and LLVM Fast ISel with optimizations off still takes +**4 seconds** where Flying Start takes 0.04. Copy-and-Patch §7.4 +diagnoses the same thing from outside: "the performance of LLVM +`-O0` bogs down in instruction selection." + +Umbra's observation is about the *input*, not the compiler: +generated query code is regular and short-lived — short +straight-line blocks, few live values, no human weirdness — so it +does not need a general optimizer. A compiler specialized to that +shape can be linear. Copy-and-Patch §7.4 states the linearity +claim explicitly for its own algorithm: it "runs in linear time, +requiring only two traversals of the AST and one traversal of the +CPS call graph," and Fig. 26 measures it — normalizing the time to +compile 10k statements to 1, perfect scaling at 800k statements +would be 80, and C&P ends at **98**, while every LLVM level is +worse. ### Step 3 — Tidy Tuples: the codegen layer that never loses track -The name is the *data-centric value tracking* in the code -generator: as it walks the plan (produce/consume, Neumann's model), -it tracks every attribute with its type and current location — -register or memory — so the generator emits loads lazily, exactly -once, and never re-materializes a value it already has. That -bookkeeping is what keeps the generated code register-clean without -an optimizer cleaning up after the fact — the optimization happened -*during* generation. The layer stack: +> **In:** Step 2's "specialize to the shape of generated code." +> **Out:** the five-layer code generator that produces Umbra IR — +> the thing whose *output* Step 4's backend consumes. + +The name refers to **data-centric value tracking** in the code +generator: as it walks the plan (produce/consume — Neumann's model, +`reading-neumann-vldb11.md`, and the paper cites it as [25]), it +tracks every attribute with its type and current location, so the +generator emits loads lazily, exactly once, and never +re-materializes a value it already has. That bookkeeping is what +keeps the generated code register-clean *without* an optimizer +cleaning up afterwards — the optimization happened *during* +generation. + +The five layers, quoted from §2.2 (Fig. 4), coarse-grained at the +top and fine-grained at the bottom, each emitting fewer +instructions per operation than the one above: ``` - relational algebra - └─ Tidy Tuples codegen (produce/consume, tracks values) - └─ Umbra IR (SSA-ish, fixed-width ops, ONE pass - per lowering — designed so every - lowering step is linear scan) - ├─ Flying Start: direct x86 emit (~µs, ~LLVM -O0+) - └─ LLVM -O3 (background, hot queries only) + relational algebra (query plan) + 1. Operator Translators — produce/consume style [25] + 2. Data Structures — components that GENERATE CODE to + act on hash tables, etc. + 3. Tuples — pack/unpack/hash over several values + 4. SQL Values — per-SQL-type ops with + standard-conform NULL semantics + 5. Codegen API — Int8/UInt64/Double/Ptr, a + STATICALLY TYPED interface: the + result of a:Int8 + b:Int8 is Int8 + │ + ▼ + Umbra IR ──┬── Flying Start: direct x86 emit (Step 4) + └── LLVM optimizing compiler (Step 5 picks) +``` + +The static typing at layer 5 is the load-bearing detail: §2.2 says +the Codegen layer "ensures that, e.g., the result of `a:Int8 + +b:Int8` is again of type `Int8`," so type errors in the *generator* +are caught by the host C++ compiler at build time rather than +surfacing as miscompiled queries. This is the same instinct as +Neumann's §4.1 preference for LLVM IR's strong typing over C++, +pushed one level higher — into the code that writes the code. + +Table 4 sizes the layers, and the shape of it is the argument for +the whole design: + +``` + Table 4 — lines of code (h / C++ / tests) + Operator translators 2,360 / 8,347 / 3,225 + Data structures 187 / 399 / 113 + Tuples 172 / 1,019 / 2,205 + SQL values 772 / 6,834 / 2,283 + Codegen 975 / 1,049 / 690 + Σ Tidy Tuples 4,466 / 17,648 / 8,516 + Umbra IR 812 / 2,348 / 476 + Flying Start 399 / 3,790 / 1,072 + Σ All 5,677 / 23,786 / 10,064 + + Flying Start — a complete x86 backend — is 3,790 lines of C++, + about 16% of the total, and less than half the size of the + operator translators alone (8,347). Do that division: + 3,790 / 23,786 = 16% + Replacing LLVM was NOT the expensive part of this project. ``` ### Step 4 — Umbra IR + Flying Start: everything single-pass -The IR (intermediate representation — the in-between language -compilers lower through) is designed backwards from the constraint -"every lowering step must be one linear scan": IR ops are -fixed-size in one contiguous array (no pointer graphs to chase); -types are simple scalars; control flow is basic blocks with -fall-through bias. Flying Start then walks that array once, -emitting x86 directly with a linear-scan register allocator. -Compare the VDBE's fixed 24-byte ops: same instinct — flat arrays -of fixed-width instructions — different target (interpretation vs -fast native lowering). The result: ~100× faster compiles than LLVM -at ~70–80% of LLVM -O3's execution speed, with LLVM kept as the top -tier for queries that earn it. What it gives up: the global -optimizations a multi-pass compiler could do — acceptable precisely -because generated query code has so little to globally optimize -(question 2). +> **In:** Step 3's Codegen API calls. **Out:** x86 machine code, +> in one pass, with the four optimizations that make it fast enough +> to be worth using unoptimized. + +**Correction — the IR is variable-length, not fixed-size.** The +previous version of this guide said Umbra IR ops are "fixed-size in +one contiguous array." §3.2 says the opposite: "The first ingredient +to Umbra IR's compact program representation is a **variable length +instruction format**. All instructions begin with an opcode which +identifies the instruction — and determines its length — followed +by a type identifier that specifies the result type." What is +contiguous is the *storage*, not the instruction width: + +``` + §3.2 — three properties of code generation that the layout exploits: + 1. codegen mostly APPENDS at the end of blocks; instructions + are never moved + 2. codegen has high LOCALITY — one block/function is completed + before moving to the next + 3. all instructions have the SAME LIFETIME as the program + + Therefore: + instructions → one dynamic array; appending needs no allocation + (most of the time); references are 4-BYTE OFFSETS + into that array + basic blocks → a dynamic array of instruction offsets + functions → the offset of their FIRST block; the rest are + discoverable through the terminating branches + + §3.2's own verdict: "The shown representation is less flexible + than intermediate representations used in optimizing compilers, + e.g., LLVM. However, we find that it yields good cache efficiency + and accelerates the generation of programs and executables." +``` + +Contrast the VDBE's genuinely fixed **24-byte** `VdbeOp` +(`reading-sqlite-vdbe.md`, `src/vdbe.h:55-95`): same instinct — +flat arrays, cheap addressing — but SQLite pays fixed width to make +*interpretation* branch-predictable, while Umbra pays variable +width to make the array *smaller*, because nothing ever interprets +it twice. + +Two more IR choices, both from §3.3–3.4, both aimed at doing work +once instead of in a pass: +- **Constant folding at append time**, plus constant + deduplication, plus one dead-code-elimination pass. §3.3 explains + why DCE earns its keep even in a latency-obsessed compiler: without + it, every layer above Codegen would have to prove in advance that + a value it is about to generate has a user, "which makes the + generator simpler." +- **DBMS-specific instructions.** Checked arithmetic is an + instruction, not a pattern: `%c = checkedsadd i32 %a, %b + %continue %overflow` branches on overflow, so the backend gets the + *intent* instead of having to re-derive it. Address calculation is + inlined into loads and stores; `isNull` needs no second operand. + This is Neumann §4.1's complaint about C++ ("no way to get at the + overflow flag") answered by owning the IR. + +Flying Start then walks that array once. §4.2's Algorithm 1 is the +minimal version — everything on the stack: + +``` + Algorithm 1 (§4.2), translating `add`: + scratch ← allocScratchRegister() + result ← allocStackSlotFor(i) + emit "copy firstArgSlot into scratch" + emit "add secondArgSlot onto scratch" + emit "copy scratch to result" + free(scratch) + + i.e. mov eax, [rsp+a] + add eax, [rsp+b] + mov [rsp+r], eax +``` + +Then four optimizations, §4.3–4.6, layered onto that skeleton +without adding a pass: **stack space reuse** (4.3), **machine +register allocation** (4.4), **lazy address calculation** (4.5), +and **fuse comparison and branch** (4.6). §4.7 shows how they fit: +the register decision lives in `resultReg()`, called at the moment +of translation; the fusion lives in `argumentReg()`, which +translates an operand on demand and can pass a *placement hint* +("put your result in the flags register") down to it; and resources +are freed in `~Reg()`. The emitter itself is the `asmJIT` library, +assembling x86 directly into a buffer. + +**Correction — Flying Start does not use linear scan.** The +previous version of this guide said "a linear-scan register +allocator," and question 2 was built on that premise. §4.4 uses a +**best-effort heuristic** instead: of the 16 x86 registers, 4 are +scratch and 1 is the stack pointer, leaving **11** to hold values +across instruction translations; a value is prioritized if it lives +only within its defining block or was created in the most deeply +nested loop (the `onlyLiveInCurrentBlock(v) || loopIsDeepestNest()` +test in Fig. 10, line 4-5). Linear scan was *measured and +rejected*: + +``` + §5.5 / Fig. 16 — adding Linear Scan to Flying Start: + execution: 1% faster + compilation: 14% MORE time + "in the interest of low compile time for now we chose not to add + Linear Scan to the Flying Start default optimizations." + + A rare published negative result about a technique that WORKS. + The trade is 14 units of compile time for 1 unit of runtime; + at Step 1's 1.32:1 compile:run ratio that is a clear loss. +``` + +Register allocation is nevertheless the optimization that matters +most — §5.5: "On average it provides a **32% reduction of execution +time**," the largest of the four (Fig. 15), and the same is true +inside Umbra's own LLVM backend (Fig. 17). The quality of the +result, Fig. 18, relative to fully optimized LLVM on TPC-H at SF=1: + +``` + Fig. 18, medians, Flying Start relative to LLVM-optimized code: + cycles 1.6× higher + instructions 2.3× higher + IPC 1.4× higher + + Sanity-check the three against each other: + cycles = instructions / IPC + 2.3 / 1.4 = 1.64 ✓ matches the 1.6× cycles figure. + So Flying Start emits ~2.3× the instructions but the + out-of-order engine retires them 1.4× more densely, and the + damage lands at 1.6× rather than 2.3×. Straight-line generated + code is exactly the shape that lets ILP absorb slop — + topic 12's point, arriving here as a compiler design licence. +``` ### Step 5 — adaptive execution: never choose wrong -With a ~µs tier and a ~ms tier, Umbra refuses to *predict* which a -query needs — it measures: +> **In:** Step 4's fast-compile/slower-code tier and an LLVM +> slow-compile/faster-code tier. **Out:** a policy that needs no +> cost estimate — the direct answer to postgres's failure mode. + +**Adaptive execution** is Kohn et al.'s method (ICDE 2018, +reference [18] in the Umbra paper), built for HyPer: switch between +execution backends at runtime, "even half-way through a query," to +profit from fast compilation on short queries and fast execution on +long ones. §4.1 says Umbra "also applies the adaptive execution +approach," with two backends — Flying Start and LLVM — where HyPer +had three (bytecode interpreter, LLVM with optimizations off, LLVM +optimized). ```mermaid flowchart LR - Q[query] --> B[compile Flying Start ~µs] - B --> R[start running] - R --> H{still running after\nbudget? } + Q[query] --> B[compile with Flying Start] + B --> R[start running immediately] + R --> H{still running after budget?} H -->|no| DONE[done — never paid LLVM] - H -->|yes| L[LLVM -O3 in background thread] - L --> S[swap function pointer at\nnext morsel boundary] - S --> DONE2[rest of query at full speed] + H -->|yes| L[LLVM -O3 on a background thread] + L --> S[swap function pointer at next morsel boundary] + S --> DONE2[rest of the query at optimized speed] ``` -The swap granularity is topic 11's morsel: execution is already -chunked, so "replace the function between morsels" is natural. -This kills the postgres failure mode (reading-postgres-jit.md) — -the decision uses *measured* runtime, not a planner estimate. Short -queries never pay LLVM; long queries pay it off the critical path. +The swap granularity is topic 11's **morsel** — execution is +already chunked into fixed-size row batches, so "replace the +function between morsels" is natural, and the state that survives +the swap is exactly the pipeline-breaker state (hash tables, +cursors, partial aggregates) that Neumann's §3.1 already forces you +to materialize. That is question 4. + +This kills the postgres failure mode +(`reading-postgres-jit.md`): postgres decides with a planner *cost +estimate* compared against `jit_above_cost` (`jit.c:40`, +`planner.c:699-700`), before a single row is read, and if the +estimate is wrong the query eats the compile fee for nothing. +Umbra decides with *measured elapsed time*, after the fact, off the +critical path. Short queries never pay LLVM; long queries pay it +while already running. + +What the tiers cost, measured — and here is the third correction: + +``` + Table 3 — TPC-H SF=1, 20 threads, geometric mean over all queries, + each row: compilation speed and execution speed vs LLVM O3. + + Umbra: Flying Start vs LLVM O3 108× faster compile, 1.2× slower exec + HyPer: Interpreter vs LLVM O3 91× faster compile, 4.1× slower exec + HyPer: LLVM O0 vs LLVM O3 6× faster compile, 1.3× slower exec + + CORRECTION: the previous version of this guide said Flying Start + runs at "~70-80% of LLVM -O3's execution speed." Table 3 says + 1.2× slower, and 1 / 1.2 = 0.833 → 83%. Quote the paper's + own form ("1.2× slower") rather than a derived percentage. + + CORRECTION: "compile in ~100 µs" was also not a paper number. + The closest measured figure is Table 2's x86 column: 0.21 ms + geometric mean at SF=0.01 — i.e. ~210 µs, and that is machine-code + generation only, on the smallest data set in the paper. + + The interesting row is HyPer's interpreter: 91× faster to compile + buys 4.1× slower execution, while Flying Start's 108× buys only + 1.2×. Flying Start is strictly better than an interpreter on BOTH + axes. That is the whole result of the paper in one comparison. +``` + +The abstract's summary is worth holding onto because it names both +ends: Umbra "on small data sets is even faster than interpreter +engines like DuckDB and PostgreSQL; on large data sets throughput +is on par with HyPer." ### Step 6 — copy-and-patch: compile time ≈ memcpy -The OOPSLA '21 paper pushes the floor further: move compilation to -*build* time. Precompile a library of **stencils** — machine-code -fragments, one per (operator × type) combination, with **holes** -(unresolved relocations — the linker concept: addresses/constants -left blank in object code) for constants, offsets, and branch -targets. At runtime, "compilation" is copying stencils and filling -holes: +> **In:** Step 4's "single pass over the IR" as the apparent floor. +> **Out:** a lower floor — precompiled machine-code fragments with +> holes — and the calling-convention trick that makes them +> composable. + +The OOPSLA '21 paper moves compilation to *build* time. §3's +definition, verbatim, because every word is load-bearing: + +> "A **binary stencil** is a binary code function that implements a +> computation logic fragment, where **literals, jump addresses, and +> stack offsets are missing**." + +Those three missing things are the **holes** — the linker's +relocation concept, reused. Each stencil implements one AST node or +bytecode, "or a commonly-used shape of an AST subtree or bytecode +sequence (we call such stencils **supernodes**)". Supernodes are +where the code quality comes from: they "allow optimizations across +node boundaries," because Clang got to optimize the whole shape at +build time. Library sizes (§5, footnote 1): ``` - build time: compile a library of STENCILS with clang — - object code for each (operator × type) with HOLES - (relocations) for constants/offsets/branch targets - run time: for each IR op: memcpy stencil, patch holes - → machine code in ~100s of ns per op + WebAssembly compiler: 1,666 stencils, 35 kB + high-level language: 98,831 stencils, 17.5 MB (supernodes) + + The high-level library is 59× more stencils for 500× the memory, + and §5 says the difference is supernodes: they generated "close + to 100,000" of them because memory was cheap in that setting, + and note you "can simply remove the supernode stencils" to get + back to 35 kB. Code quality is a DIAL here, priced in bytes. ``` The runtime "compiler" is barely a loop: ```rust +// ILLUSTRATION — not quoted from any source; this is the §4 algorithm +// in Rust shape. The measured version generates code for TPC-H Q5 in +// 178 µs (Copy-and-Patch §7.3), against 326 µs to build the AST. +// Compare the real single-pass emitter at umbra §4.2 Algorithm 1, and +// the interpreter this replaces at experiments/src/interp.rs:8-16. fn compile(ops: &[IrOp], stencils: &Stencils, out: &mut Code) { for op in ops { let s = &stencils[op.kind()]; // object code built at BUILD time let base = out.append(&s.bytes); // "compilation" is a memcpy - for hole in &s.holes { // relocations left unresolved + for hole in &s.holes { // literals / jump targets / stack offsets out.patch(base + hole.offset, op.operand(hole.which)); } - } // no IR, no passes, no regalloc + } // no IR passes, no regalloc pass } ``` -The trick making stencils composable: continuation-passing style + -tail calls (`musttail`) so each stencil ends by jumping to the next -— no prologue/epilogue, registers stay live across stencils -(GHC-ish calling convention). Result: compiles ~2 orders faster -than LLVM -O0 with *better* code than -O0. This is the natural -floor of the spectrum between bytecode and real JIT — and -PostgreSQL people have prototyped it for ExprState. +**Correction of emphasis.** The trick making stencils composable is +continuation-passing style *plus a calling convention*, not +`musttail` by itself. §3: control passes directly to the next +operation instead of returning to the parent; those calls are tail +calls, so "the Clang C++ compiler that the MetaVar system uses to +compile stencils lowers them to jump instructions." Then: + +> "Combined with the **GHC calling convention**, in which all +> registers are saved by the caller and all parameters are passed +> in registers, continuation-passing removes most of the calling +> overhead between stencils." -### Step 7 — what transfers to M19 +And the sharpest sentence in the paper: "we **repurpose the +function prototype and the calling convention as a register +allocation protocol**, where each function parameter implicitly +corresponds to some physical register determined by the calling +convention." Different stencil variants exist for different +register configurations, and the generator picks one at runtime. +Register allocation without a register allocator. + +The measured result, Fig. 24 and §7.2–7.3: + +``` + Fig. 24 caption (TPC-H, in their metaprogramming system): + compile up to 276× faster than LLVM -O0 + up to 1435× faster than -O1/-O2/-O3 (range 1083-1435×) + execute 14% FASTER than -O0 + 22% slower than -O1, 25% slower than -O2, 24% than -O3 + + Fig. 25: C&P's startup overhead is 2-3× the interpreter's, but + both are negligible — "in most cases it takes longer to construct + the AST." Concretely, TPC-H Q5: 178 µs to generate code, + 326 µs to build the AST from the query plan, and 1.17 s for the + interpreter to execute it. +``` + +Note that C&P beats `-O0` on *both* axes — same shape of result as +Flying Start beating HyPer's interpreter on both axes in Step 5. +When a specialized compiler dominates a general one on both +compile time and code quality, the general one has no remaining +argument at that tier. + +### Step 7 — the arithmetic: when does optimizing ever pay? + +> **In:** Steps 5 and 6's paired (compile-cost, run-quality) +> numbers. **Out:** the break-even runtime, computed — and the +> reason "always compile with -O3" stopped being defensible. + +§7.3 does the arithmetic on itself, and it is the cleanest worked +example in either paper: + +``` + Copy-and-Patch §7.3, TPC-H Q5, in their system: + + LLVM -O3 compile time = 0.25 s + ratio vs C&P = 1435× + ⇒ C&P compile time = 0.25 / 1435 = 174 µs (they report + 178 µs in §7.3) + -O3's code is = 1.2% faster than C&P's + + Let R = the query's execution time under C&P. + Optimizing pays when compile_saving < execution_saving + 0.25 s - 0.000174 s < 0.012 × R + 0.2498 < 0.012 R + R > 20.8 s ← the paper says 21 s + + Measured reality: "the query finished execution in less than + 0.1 s." So -O3 would have to run 210× longer than it does + before it broke even. +``` + +And the historical framing, same section, which is why this is a +*change* rather than a curiosity: + +``` + Then: 38% average speedup over -O0 had to amortize 9.2× more compile + break-even runtime multiple ≈ 9.2 / 0.38 ≈ 24× compile time + Now: 24% average speedup over C&P has to amortize 1286× more compile + break-even runtime multiple ≈ 1286 / 0.24 ≈ 5,358× compile time + + The break-even bar rose by 5358 / 24 ≈ 223×. + Optimizing did not get worse. The baseline got 100× cheaper, and + that alone moved the decision. +``` + +Now run the same formula on *our* topic, with our own measured +numbers so the shape is familiar: + +``` + M19, from notes.md (Apple M3 Pro, 2026-07-10, N_COLS=4, + depth 8 = 511 nodes, best-of-3): + interp lane 0.95 M rows/s → 1/0.95e6 = 1.053 µs/row + vector lane 11.8 M rows/s → 1/11.8e6 = 0.0847 µs/row + + rows_breakeven = compile_µs / (µs_per_row_slow − µs_per_row_fast) + = compile_µs / (1.053 − 0.0847) + = compile_µs / 0.9683 + + compile in 100 µs → 103 rows + compile in 500 µs → 516 rows + compile in 5000 µs → 5,164 rows + + Same formula, three systems, three answers: + Umbra : denominator measured per morsel, decision deferred + → break-even discovered, never predicted (Step 5) + PostgreSQL : denominator ESTIMATED by the planner before row 1 + → wrong estimate = pure loss (reading-postgres-jit.md) + C&P : numerator driven to ~0 at build time + → break-even collapses to almost any row count +``` + +### Step 8 — what transfers to M19 + +> **In:** Steps 5, 6 and 7. **Out:** the design decision for our +> own JIT lane, stated as a policy rather than a preference. M19's budget heuristic should be Umbra-shaped, not postgres-shaped: -interpret first, count rows/time actually spent, JIT when the -measured cost clears the (measured) cranelift compile cost from -jit_bench. Cranelift itself sits near Flying Start on the ladder: -single-tier, fast compile, decent code — a sane single choice when -you don't want two backends. +interpret first, count rows and time actually spent, JIT when the +*measured* cost clears the *measured* cranelift compile cost from +`jit_bench`. Both inputs are things you will have measured by then +— that is the entire difference from postgres, which has neither. + +Cranelift sits near Flying Start on the ladder: single-tier, fast +compile, decent code, no LLVM dependency — a sane single choice +when you don't want two backends and cannot afford Umbra's 3,790 +lines of hand-written x86 emission (Table 4). What you give up +relative to Umbra is the second tier: no query in our system will +ever get `-O3`-quality code. Step 4's Fig. 18 numbers say what that +costs — about 1.6× the cycles at the median — and Step 7's +arithmetic says how rarely that matters at the row counts we +actually run. ## How to read the papers (with the concepts in hand) -- **Tidy Tuples / Flying Start (VLDBJ '21)** — read the IR-design - section against Step 4's checklist (fixed-width ops, contiguous - arrays, restricted types/CFG) and the value-tracking section - against Step 3; the adaptive-execution material (with the - ICDE '18 companion) is Step 5 — note the morsel-boundary swap and - what state both code versions must agree on (question 4). The - evaluation's compile-time vs run-time scatter plots are the - chapter's thesis in one figure. -- **Copy-and-Patch (OOPSLA '21)** — §on stencils and holes is - Step 6; the musttail/continuation-passing mechanics deserve a - slow read (question 3). Read their comparison against LLVM -O0 - skeptically and note which benchmark shapes favor stencils - (short, cold code) vs a real JIT (hot loops). +- **Tidy Tuples / Flying Start (VLDBJ '21)** — read §2.2 (Fig. 4, + the five layers) against Step 3 and §2.3 for how one operator + becomes instructions. Then §3.1–3.4 against Step 4's checklist, + and read §3.2 carefully enough to notice "variable length + instruction format" — that sentence is why this guide has a + correction in it. §4.1 is Step 5's adaptive execution; §4.2's + Algorithm 1 and §4.7's Fig. 9/10 are the emitter, and Fig. 10 + lines 4-5 are the register heuristic that is *not* linear scan. + In the evaluation, read Table 2 (§5.3) for where the milliseconds + go, Fig. 13 for the 150 s vs 0.04 s, Table 3 (§5.4) for the + compile/execute trade, and Fig. 16 (§5.5) for the measured + rejection of linear scan. Fig. 14's compile-vs-execute scatter on + Q3 is the chapter's thesis in one picture. +- **Copy-and-Patch (OOPSLA '21)** — §3 defines stencils, holes and + supernodes; read it against Step 6 and note the two paragraphs on + CPS and the GHC calling convention (question 3). §4 is the + algorithm; §5 is the stencil library and where 98,831 comes from. + In the evaluation, read Fig. 24 with its caption, §7.3's 21-second + break-even (Step 7 rebuilds it), Fig. 25 for interpreter + comparison, and Fig. 26 for scalability. Read the -O0 comparison + skeptically and note which benchmark shapes favor stencils (short, + cold code) versus a real JIT (hot loops) — the paper's own + concession is that in "an industry-strength database, compilation + would take several times longer, but execution would likely be + faster." ## Questions for notes.md 1. Umbra IR vs LLVM IR: name three concrete representation choices - that make single-pass lowering possible (fixed-width ops, - contiguous arrays, restricted types/CFG) and what each gives up. -2. Flying Start does register allocation in one linear pass — what - property of *generated query code* (short straight-line blocks, - few live values — the Tidy Tuples tracking) makes that - acceptable where a C compiler couldn't? -3. Copy-and-patch: why does continuation-passing + musttail let - stencils compose without spilling registers at boundaries, and - what does that share with WGSL/wgpu's "pipeline fixed at - creation" specialization from topic 18? + that make single-pass lowering possible and say what each gives + up. Get them from §3.2–3.4, not from memory — the answer + includes *variable*-length instructions, which is the opposite + of what you might guess. +2. Flying Start's register allocation is a heuristic over 11 + available registers (§4.4), not linear scan; §5.5 measured + linear scan at +14% compile time for −1% execution time and + rejected it. What property of *generated query code* (short + straight-line blocks, few live values — the Tidy Tuples + tracking) makes a heuristic that cheap acceptable, where a C + compiler could not get away with it? Then: at what compile:run + ratio would the linear-scan trade flip? Compute it from Step 1's + 0.66 ms / 0.50 ms. +3. Copy-and-patch: why does continuation-passing plus the GHC + calling convention let stencils compose without spilling + registers at boundaries — and what exactly does "we repurpose + the function prototype and the calling convention as a register + allocation protocol" (§3) mean operationally? What does that + share with WGSL/wgpu's "pipeline fixed at creation" + specialization from topic 18? 4. The adaptive swap happens at morsel boundaries. What state must the compiled and interpreted versions AGREE on for the swap to - be sound (hash tables, cursors, partial aggregates — the - pipeline-breaker state, exactly)? -5. For M19: cranelift compile of a depth-8 expression costs X µs - (measure in jit_bench). Using the measured interp rows/s, write - the break-even row count formula and compute it. Does a - FalkorDB `WHERE` clause over a 1M-node scan clear it? + be sound? (Hash tables, cursors, partial aggregates — i.e. + exactly the pipeline-breaker state from Neumann §3.1. Say why + the register-resident values *between* breakers are precisely + the ones that need not be transferred.) +5. For M19: measure cranelift's compile time for a depth-8 + expression in `jit_bench`. Using `notes.md`'s measured interp + rate, write the break-even row count formula and compute it — + Step 7 gives the shape. Does a FalkorDB `WHERE` clause over a + 1M-node scan clear it? By what margin? And at what node count + would it stop clearing it? ## Done when -- [ ] You can state the compile-latency budget in numbers and explain why LLVM's cost is structural rather than a flag away. -- [ ] You can name three concrete ways Umbra IR differs from LLVM IR. -- [ ] You can explain what Flying Start's single-pass register allocation gives up. -- [ ] You can explain why continuation-passing plus `musttail` is what makes copy-and-patch work. -- [ ] You can say what state must be transferable for an adaptive swap at a morsel boundary. +Answer each before unfolding it. + +- [ ] You can state the compile-latency budget in numbers, with the scale factor attached, and explain why LLVM's cost is structural rather than a flag away. + +
Answer + + At TPC-H **SF=0.01**, 1 thread, geometric mean over 22 queries + (Table 2), Umbra spends 0.66 ms preparing (plan 0.25 + codegen + 0.20 + x86 0.21) and 0.50 ms executing — a **1.32:1** + compile:run ratio in the *fastest* system in the table, and that + is with LLVM compilation excluded because it is "too long for a + data set this small." The structural claim is measured in + Fig. 13: on a 2000-join query producing 108,000 Umbra IR + instructions, LLVM takes **150 s**, LLVM with Fast ISel and no + optimizations still takes **4 s**, and Flying Start takes + **<0.04 s**. Turning optimization off bought one order of + magnitude; it did not remove the SSA construction, the + pointer-linked graphs, or the multi-pass instruction selection — + Copy-and-Patch §7.4 independently observes that `-O0` "bogs down + in instruction selection." + +
+ +- [ ] You can name three concrete ways Umbra IR differs from LLVM IR, and correct the claim that its instructions are fixed-size. + +
Answer + + From §3.2–3.4: (1) a **variable length** instruction format — + opcode first, and the opcode determines the instruction's length + — stored in one dynamic array, with instructions referenced by + **4-byte offsets** rather than pointers; blocks are arrays of + those offsets and a function stores only its first block's + offset. (2) **Constant folding and deduplication at append + time**, plus a single dead-code-elimination pass — no general + optimization pipeline. (3) **DBMS-specific instructions**: + `checkedsadd` with an overflow branch target built in, address + calculation inlined into loads/stores, a one-operand `isNull`. + What it gives up, in the paper's own words: the representation + "is less flexible than intermediate representations used in + optimizing compilers," and it is "not well suited for complex + restructuring passes" — fine, because nothing restructures it. + The layout is justified by three properties of the generator: + it only appends, it has high locality, and every instruction has + the program's lifetime. + +
+ +- [ ] You can say what register allocation scheme Flying Start actually uses and what the paper measured when it tried a better one. + +
Answer + + A **best-effort heuristic**, §4.4: of x86's 16 registers, 4 are + scratch and 1 is the stack pointer, so **11** can hold values + across instruction translations; a value gets one if registers + are available AND it either lives only within its defining block + or was created in the most deeply nested loop (Fig. 10, lines + 3-5). Not linear scan. §5.5 added **linear scan** as an + experiment and measured it (Fig. 16): **1% faster execution for + 14% more compile time**, and the authors declined it "in the + interest of low compile time." Register allocation is still the + single most valuable of the four optimizations — a **32% + reduction in execution time** on average (§5.5, Fig. 15) — which + is why the heuristic exists at all rather than everything living + on the stack as in §4.2's Algorithm 1. + +
+ +- [ ] You can explain why continuation-passing plus the GHC calling convention is what makes copy-and-patch work. + +
Answer + + A stencil is machine code with holes for literals, jump + addresses and stack offsets (§3). If stencils called each other + normally, every boundary would cost a prologue/epilogue and + force temporaries to memory. Instead each stencil ends by passing + control *forward* to the next (CPS), and because those are tail + calls Clang lowers them to plain jumps. The **GHC calling + convention** then does the rest: all parameters are passed in + registers and all registers are caller-saved, so a value handed + to the continuation *is* a value left in a register. §3 states + the consequence directly: "we repurpose the function prototype + and the calling convention as a register allocation protocol, + where each function parameter implicitly corresponds to some + physical register." Variants of each stencil exist for different + register configurations, and the generator picks the matching one + — which is also why the high-level library reaches 98,831 + stencils / 17.5 MB while the WebAssembly one is 1,666 / 35 kB. + +
+ +- [ ] You can say what state must be transferable for an adaptive swap at a morsel boundary, and why the swap is safe there and nowhere else. + +
Answer + + Everything materialized at a **pipeline breaker** — hash tables, + sort runs, cursors/scan positions, partial aggregates — must have + an identical layout in both versions, because both must be able + to read and continue it. Everything *between* breakers need not + transfer at all, and that is the point: Neumann §3.1 defines a + pipeline as the span over which tuples stay in CPU registers, so + at a morsel boundary there are by construction no live + register-resident intermediates to hand over — only the + materialized state. That is why the swap is sound at a morsel + boundary and would be a nightmare mid-pipeline. Kohn et al. + (ICDE 2018, [18] in the Umbra paper) go further and switch "even + half-way through a query"; Umbra's two-tier version is the same + idea with Flying Start replacing the bytecode interpreter. + +
+ +- [ ] You can compute the break-even runtime that makes an optimizing compile worth it, and explain why that bar moved by two orders of magnitude. + +
Answer + + §7.3, TPC-H Q5: `-O3` compiles in 0.25 s and its code is 1.2% + faster than copy-and-patch's. Break-even needs + `0.25 s ≈ 0.012 × R`, so `R ≈ 20.8 s` — the paper rounds to + **21 s** — against a query that actually finishes in under + 0.1 s. Historically a 38% speedup over `-O0` had to amortize a + 9.2× compile increase (break-even ≈ 24× the compile time); now a + 24% speedup must amortize an average **1286×** increase + (break-even ≈ 5,358×), a bar roughly **223× higher**. Nothing + about optimization got worse — the *baseline* got two orders of + magnitude cheaper, and that alone flipped the decision. The same + formula with M19's numbers: `rows = compile_µs / (1.053 − + 0.0847)`, so 500 µs of cranelift compile pays back after 516 + rows. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your measured cranelift compile time for a depth-8 expression. +
Answer + + The point of measuring it yourself is that every number in this + chapter is someone else's machine. `notes.md`'s baseline is + measured on an Apple M3 Pro; Umbra's is a 10-core Skylake X at + 3.4 GHz (§5.1). Ratios travel between machines much better than + absolute times, which is why Step 7 works in ratios and why the + break-even formula takes *your* compile time as its numerator. + Record the measured `compile_µs`, the division, and the resulting + row count — and if the JIT lane does not beat the vectorized lane + per row, record that too: a negative denominator is a finding + about autovectorization, not a failed experiment. + +
+ ## References **Papers** -- Kersten, Leis, Neumann — "Tidy Tuples and Flying Start: Fast - Compilation and Fast Execution of Relational Queries in Umbra" - (VLDB Journal 2021) -- Xu & Kjolstad — "Copy-and-Patch Compilation" (OOPSLA 2021, - [arXiv:2011.13127](https://arxiv.org/abs/2011.13127)) + +| paper | what to read | which numbers | +|---|---|---| +| Kersten, Leis, Neumann — "Tidy Tuples and Flying Start: Fast Compilation and Fast Execution of Relational Queries in Umbra" (VLDB Journal 2021) | §2.2 layers; §3.1-3.4 the IR; §4.1-4.7 the backend; §5.1-5.5 evaluation | Table 2 (SF=0.01 breakdown, Σ 1.24 ms), Fig. 13 (150 s / 4 s / 0.04 s), Table 3 (108× compile, 1.2× exec), Fig. 15-16 (32% from regalloc; linear scan +14%/−1%), Fig. 18 (1.6× cycles, 2.3× instructions, 1.4× IPC), Table 4 (LoC) | +| Xu, Kjolstad — "Copy-and-Patch Compilation" (OOPSLA 2021, [arXiv:2011.13127](https://arxiv.org/abs/2011.13127)) | §3 stencils/holes/supernodes; §4 the algorithm; §5 the library; §7.2-7.5 evaluation | Fig. 24 (276× vs -O0, 1435× vs -O1..-O3; +14% vs -O0, −24% vs -O3), §7.3 (0.25 s, 21 s break-even, 178 µs vs 326 µs AST, 1.17 s interpreter), Fig. 26 (98 vs perfect 80) | +| Kohn, Leis, Neumann — adaptive execution (ICDE 2018; reference [18] in the Umbra paper) | the tier-switching method Step 5 rests on | switches backends mid-query | + +**Elsewhere in this repo** +- `reading-neumann-vldb11.md` — produce/consume (the paper's [25]), + pipeline breakers, and the compile-time tension this chapter + resolves +- `reading-postgres-jit.md` — the estimate-based policy Step 5 + contrasts against, with `jit.c:40-42` and `planner.c:698-721` +- `reading-sqlite-vdbe.md` — the fixed-24-byte-op counterpoint to + Umbra IR's variable-length encoding (`src/vdbe.h:55-95`) +- `reading-cranelift-jit-demo.md` — the single-tier fast backend + M19 actually ships; Step 8's design choice +- `notes.md` — the measured interp/vector rates every arithmetic + block here divides by diff --git a/topics/20-graphblas/README.md b/topics/20-graphblas/README.md index 69f26a3..2faa107 100644 --- a/topics/20-graphblas/README.md +++ b/topics/20-graphblas/README.md @@ -24,10 +24,14 @@ replace the M13 adjacency core. Switch heuristics are *numbers in the code*, not magic: sparse→bitmap when `nnz > bitmap_switch * nrows*ncols` -(GB_convert_sparse_to_bitmap_test.c:32-38, default per-op table); -hyper↔sparse via `hyper_switch` on the count of non-empty vectors +(GB_convert_sparse_to_bitmap_test.c:32-38), where `bitmap_switch` is +indexed by `min(vlen, vdim)`, **not by the operator** — 0.04 at +dimension 1 rising to **0.40 above 64** (GB_Global.c:181-189, selected +by GB_Global.c:486-497), so every graph-sized matrix reads the same +0.40; hyper↔sparse via `hyper_switch` on the count of non-empty vectors (GB_conform_hyper.c:52); all applied by `GB_conform` -(GB_conform.c:33-89) after every operation. Why hypersparse matters +(GB_conform.c:150, cases at :157-160, :166-169, :175-184, :190-193) +after every operation. Why hypersparse matters to FalkorDB: node IDs are a namespace, most rows of a relation matrix are empty — CSR's rowptr alone for 10M nodes = 80 MB *per relation type* without it. @@ -50,8 +54,12 @@ work ∝ flops — mask only FILTERS"] coarse tasks (own whole vectors) and fine tasks (teams share one vector); each task independently picks **Gustavson** (dense workspace of size m — the SPA) or **hash** (table sized 2×next-pow2 -of estimated flops) — hash wins when the workspace would be cold, -Gustavson when the hash would exceed m/16 (:57). A *flopcount* pass +of estimated flops) — hash wins when the workspace would be cold. +The comment at GB_AxB_saxpy3.c:57-58 still says "m/16"; the shipped +rule is Gustavson when `flmax >= cvlen/2` +(GB_AxB_saxpy3_slice_balanced.c:65) or `hash_size >= cvlen/12` (:94), +and since `hash_size ≥ 2·flmax` always, no resize path exists. A +*flopcount* pass (GB_AxB_saxpy3_flopcount.c) sizes everything first — cudf's size/retrieve two-phase (topic 18), five years earlier. @@ -71,9 +79,12 @@ _template.c) is direction-optimizing BFS written in linear algebra: work ∝ rows still unvisited × early-exit — SpMV dot engine, each unvisited vertex scans ITS in-edges, stops at first hit - switch push→pull: frontier growing AND (nq > n/β1 OR - pushwork > unexplored/α) α=8, β1=8 (:184-187, :261) - switch pull→push: frontier shrinking below n/β2, β2=512 + the whole heuristic is off while edges_unexplored < n (:248-251) + switch push→pull: pushwork > unexplored/α α=8 (:253-262) + switch pull→push: frontier shrinking below n/β2, β2=512 (:263-278) + — the two tests sit in mutually exclusive branches, not an OR, + and LAGraph's α=8, β1=8, β2=512 are not Beamer's α=14, β=24 + (SC'12 §VI-B) ``` The semiring is `ANY_SECONDI` (:140-143): ANY = "any parent will do" @@ -105,15 +116,18 @@ matrices are fast to read, slow to mutate one edge at a time": Delta_Matrix = M (settled GrB_Matrix, hypersparse CSR) + delta-plus DP (pending additions) + delta-minus DM (pending deletions) - + the same trio TRANSPOSED (delta_matrix.h:110-113) + + the same trio TRANSPOSED (delta_matrix.h:108-115, + state table :26-106) - read: A ≡ (M + DP) minus DM + read: A ≡ (M + DP) minus DM, probed DP → DM → M + (delta_isStored.c:26/32/39 — legal because DP ∩ DM = ∅) write: O(1)-ish into DP/DM (bitmap/hash-friendly, tiny) sync: Delta_Matrix_wait — M ←(M ∪ DP) \ DM, clear deltas - (delta_wait.c:13-46: deletions via GrB_transpose-as-copy - with GrB_DESC_RSCT0 mask trick, additions via assign) - mxm: (A*(M+DP)) — delta_mxm.c:44-86 folds pending state - into ONE masked multiply instead of forcing a sync + (delta_wait.c:36-57: deletions via GrB_transpose-as-copy + with GrB_DESC_RSCT0 mask trick, additions via assign; + thresholds at :89 and :97) + mxm: (A*M) + A*DP — delta_mxm.c:104 masks only the + settled multiply; the eWiseAdd of A*DP at :107 is unmasked ``` This is topic 3's LSM memtable+tombstones, rebuilt over matrices — diff --git a/topics/20-graphblas/experiments/.gitignore b/topics/20-graphblas/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/20-graphblas/experiments/.gitignore +++ b/topics/20-graphblas/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/20-graphblas/notes.md b/topics/20-graphblas/notes.md index 46829d7..63d6734 100644 --- a/topics/20-graphblas/notes.md +++ b/topics/20-graphblas/notes.md @@ -2,6 +2,12 @@ ## Baseline (provided kernels, Apple M3 Pro, measured 2026-07-10) +> These figures predate the re-run recorded in +> [FINDINGS.md](../../FINDINGS.md) row 20 (SpMV **20.7 → 12.3 GB/s**, +> sweep **175×**). Same machine, later run; where the two disagree, +> FINDINGS is canonical. Re-run `./verify.sh 20` before treating any +> cell below as current — and never blend the two runs in one sentence. + gb_bench, RMAT edge_factor 8, best-of-N. ### SpMV (PLUS,TIMES) — the bandwidth story diff --git a/topics/20-graphblas/reading-beamer-sc12.md b/topics/20-graphblas/reading-beamer-sc12.md index f48530f..fcd3f20 100644 --- a/topics/20-graphblas/reading-beamer-sc12.md +++ b/topics/20-graphblas/reading-beamer-sc12.md @@ -7,21 +7,36 @@ picks per level. This chapter builds the two algorithms and the switch from zero — waste argument, early exit, thresholds, then the linear-algebra translation — so you can read the paper with LAGraph's template open ([reading-lagraph.md](reading-lagraph.md)): -the 2012 idea ships verbatim in the 2025 library, thresholds and -all. +the 2012 idea ships in the 2025 library, though the constants moved +and the switch is not shaped the way most summaries claim. + +Every paper number below is quoted from **Beamer, Asanović & +Patterson, "Direction-Optimizing Breadth-First Search", SC '12**, +with the section or figure it came from; every code anchor is +**LAGraph @ `e2539e2`** (the pin in `resources/codebases.md`), +quoted with the line numbers it occupies at that commit. Where the +paper and LAGraph disagree — and they do, on both constants and the +shape of the test — this guide says which is which. ## The problem in one sentence -On a small-world graph, the middle levels of a BFS contain most of -the graph, and there the classic algorithm wastes nearly every edge -inspection on already-visited vertices — direction-optimizing BFS -cuts total edge inspections 3-8× on scale-free graphs by flipping -who scans whom. +On a small-world graph the middle levels of a BFS contain most of +the graph, and there the classic algorithm spends nearly every edge +inspection on an already-visited vertex: on the paper's `kron27` +sample search, a top-down BFS performs about **67×** the edge +examinations that the BFS tree strictly requires (§III), and +flipping who scans whom recovers an **average speedup of 3.9, never +below 2.4**, across ten graphs on a 16-core machine (§VI-C, +Fig. 10). ## The concepts, step by step ### Step 1 — BFS, frontiers, and levels +> **In:** a graph and a source vertex. +> **Out:** the vocabulary — frontier, visited, level — and the single +> cost unit (the edge inspection) every later step is priced in. + **BFS** (breadth-first search) explores a graph outward from a source vertex in waves: **level** 0 is the source, level k+1 is every not-yet-seen vertex adjacent to level k. The set of vertices @@ -29,187 +44,783 @@ discovered at the current level is the **frontier**; the set of everything discovered so far is **visited**. Each iteration consumes the frontier and produces the next one, and the outputs people actually want are per-vertex *level* (distance) or *parent* -(the BFS tree). All the work is edge inspections — "is this -neighbor new?" — so the cost model is simply: how many edge -inspections does producing each next frontier take? +(the BFS tree). + +An **edge inspection** (the paper calls it an *edge check*) is one +look at one endpoint of one edge to ask "has this been visited?" +That is the cost unit for the whole chapter. The paper's own +statement of the baseline (§III): + +> "The total number of edge checks in the conventional top-down +> algorithm is equal to the number of edges in the connected +> component containing the source vertex, as on each step every +> edge in the frontier is checked." + +Beamer classifies the *outcome* of each check into four categories +(§III, Fig. 3 and Fig. 4), and this taxonomy is the whole argument: + +| outcome | meaning | +|---|---| +| **claimed child** | the neighbour was unvisited; the check did work | +| **failed child** | neighbour at depth d+1, already claimed by a rival | +| **peer** | neighbour at the same depth d | +| **valid parent** | neighbour at depth d−1 | -### Step 2 — push (top-down), and why it dies mid-search +Only *claimed child* is productive. The other three are the waste. -The classic formulation is **push**: each frontier vertex scans its -out-edges and tries to claim unvisited neighbors. Its work per -level is the sum of the frontier's out-degrees — every out-edge of -every frontier vertex is inspected, whether or not it finds -anything new. On small-world graphs the frontier explodes — level -3-4 of an RMAT graph holds most of the graph: +Why it matters: the entire paper is one observation about the ratio +between those four categories as the frontier grows, so pricing +everything in edge inspections is not a simplification — it is the +paper's own metric. + +### Step 2 — the size of the waste, measured + +> **In:** the four outcome categories from Step 1. +> **Out:** the paper's own arithmetic on `kron27`, reproduced, giving +> the headroom that direction-optimization is trying to claim. + +The paper's §III does the subtraction for you, on a Kronecker +graph it calls **kron27** — 128M vertices, 2B undirected edges, +input degree 16 (Fig. 3 caption): + +> "For the example in Figure 3, only 63,036,116 vertices are in the +> BFS tree, so at least 63,036,115 edges need to be considered, +> which is about 1/67th of all the edge examinations that would +> happen during a top-down traversal." + +Work that factor of 67 yourself; the paper explains it in the next +sentence and the two effects multiply: + +``` + inputs (Beamer §III, Fig. 3 caption): + undirected edges m = 2.0e9 + vertices in the BFS tree t = 63,036,116 + tree edges strictly needed t−1 = 63,036,115 + + top-down inspections = every edge in the component, from BOTH ends: + 2 × m = 4.0e9 + + ratio = 4.0e9 / 63,036,115 = 63.5× (paper: "about 67") + + why 67 and not the input degree 16, in the paper's own two reasons: + ×2 each undirected edge is checked from both endpoints + ×2.0 zero-degree vertices are excluded from the component, which + raises the effective degree of the vertices that remain + 16 × 2 × 2.0 ≈ 64 +``` + +The 63.5 from the nominal 2B and the paper's 67 differ because the +generator's realised edge count is not exactly 2.0e9; the point +survives either way. **Roughly 98.5% of a top-down BFS's edge +inspections on this graph are, in principle, avoidable.** + +Fig. 4 says where in the search the waste sits: "During the first +few steps, the percentage of claimed children is high… As the +frontier reaches its largest size, the percentage of peer edges +dominates." Waste is not spread evenly — it is concentrated in +exactly the two or three middle levels that also hold most of the +runtime (§III: "The middle steps (2 and 3) consume the vast +majority of the runtime"). + +Why it matters: this is the number your own trace has to reproduce +(question 1), and it tells you the switch only has to be right in +the middle of the search — the tails cannot pay for themselves. + +### Step 3 — push (top-down), and why it dies mid-search + +> **In:** the frontier/visited pair (Step 1) and the waste profile +> (Step 2). +> **Out:** push's cost formula, and the reason its cost tracks the +> frontier rather than the discoveries. + +The classic formulation is **push** (the paper says *top-down*): +each frontier vertex scans its out-edges and tries to claim +unvisited neighbours. Its work per level is the sum of the +frontier's out-degrees — call that quantity **m_f**, the paper's +name for "the number of edges to check from the frontier" (§V) — +whether or not those checks find anything new. ``` level: 0 1 2 3 4 5 |frontier|: 1 d̄ d̄² ~n/2 ~n/4 tail push work: d̄ d̄² d̄³ HUGE … … - ↑ most edge checks FAIL: neighbor already visited + ↑ at the apex, Fig. 4 says PEER edges dominate: + both endpoints are already in the frontier ``` -At the peak, nearly every edge inspection hits an already-visited -vertex — wasted claims (and wasted CAS on parallel hardware: many -threads race to claim the same few remaining vertices). Push is -perfect when the frontier is small and hopeless when it's huge. +Two costs, not one. The wasted checks are the obvious one. The +paper names the second in §IV: "In the top-down approach, there +could be multiple parallel writers to the same child, so atomic +operations are needed to ensure mutual exclusion." Push's failed +children are also its contended CAS operations — the same +few remaining vertices, raced for by many threads. -### Step 3 — pull (bottom-up): invert who scans whom, and exit early +Why it matters: push is not badly implemented; it is asked to do +the wrong thing. Its cost is m_f, and m_f peaks exactly where the +discoveries stop. -Beamer's bet is to invert the roles: each *unvisited* vertex scans -its in-edges asking "is any parent of mine in the frontier?" — and -stops at the FIRST hit, because one frontier parent is all it needs -to join the next level. That early exit is the entire speedup: when -the frontier is most of the graph, an unvisited vertex finds a -frontier parent in O(1) expected probes. +### Step 4 — pull (bottom-up): invert who scans whom, and exit early -```rust -// pull: each UNVISITED vertex asks "is any of MY parents in the frontier?" -fn bfs_pull_level(at: &Csr, frontier: &Bitmap, visited: &Bitmap) -> Bitmap { - let mut next = Bitmap::new(at.nrows); - for v in 0..at.nrows { - if visited.get(v) { continue; } - for &u in at.row(v) { // v's in-edges (a row of AT) - if frontier.get(u) { - next.set(v); - break; // ANY monoid: first hit suffices — - } // the early exit IS the speedup - } - } - next -} -``` +> **In:** push's cost m_f (Step 3). +> **Out:** pull's cost, the early exit that produces it, and the +> paper's measured evidence that the exit — not the inversion — is +> where the speed comes from. -The work comparison: +Beamer's inversion: each *unvisited* vertex scans its in-edges +asking "is any parent of mine in the frontier?" — and stops at the +**first** hit, because one frontier parent is all it needs to join +the next level. Fig. 5 is the paper's pseudocode, and the `break` +is the whole idea: ``` - push work ≈ Σ_{v ∈ frontier} out_degree(v) (all of it) - pull work ≈ Σ_{v unvisited} (probes until first frontier hit) - ≈ nnz-touched shrinks as frontier grows + Fig. 5 — Single Step of Bottom-Up Approach (Beamer §IV, verbatim) + + function bottom-up-step(vertices, frontier, next, parents) + for v ∈ vertices do + if parents[v] = -1 then + for n ∈ neighbors[v] do + if n ∈ frontier then + parents[v] ← n + next ← next ∪ {v} + break ← the early exit + end if + end for + end if + end for ``` -Push's work grows with the frontier; pull's shrinks as the -unvisited set drains and hits come faster. They cross somewhere in -the middle levels — which is the whole paper. +The paper's own summary of the two consequences (§IV): + +> "The advantage of this approach is that once a vertex has found a +> parent, it does not need to check the rest of its neighbors." + +> "With the bottom-up approach, only the child writes to itself, +> removing any contention." + +Do not take the early exit on faith as *the* mechanism — Yang, +Buluç & Owens (ICPP '18) took Beamer's construction apart into +separable optimizations and measured each one on +`kron_g500-logn21` (their **Table 2**, in GTEPS, cumulative left to +right, speedups standalone): + +| optimization | GTEPS | standalone speedup | +|---|---|---| +| baseline | 0.874 | — | +| structure only | 1.411 | 1.62× | +| **change of direction** | 1.527 | **1.08×** | +| masking | 3.932 | 2.58× | +| **early exit** | 15.83 | **4.02×** | +| operand reuse | 42.44 | 2.68× | -### Step 4 — what pull needs: the reverse graph and a dense frontier +Read the two bold rows together. Flipping direction *by itself* +buys 1.08×. The early exit buys 4.02× — Yang's §5.3 says flatly +that it "yielded the greatest speed-up". If you implement pull +without the `break`, you have implemented the 8% and skipped the +302%. -Pull's two prerequisites, each a real cost: +The work comparison, then: -- **the reverse graph**: pull scans *in*-edges, so it needs the - transpose AT (equivalently, CSC — the same edges indexed by - destination instead of source). For a graph stored once in CSR, - that's a second copy: memory ×2. This is the memory-doubling - question from topic 13 and Gunrock, and it's why LAGraph makes - AT an *optional* input. -- **a dense frontier representation**: pull tests "is u in the +``` + push work per level ≈ m_f = Σ_{v ∈ frontier} out_degree(v) (all of it) + pull work per level ≈ Σ_{v unvisited} (probes until first frontier hit) + + as the frontier grows toward n: + m_f grows — more frontier vertices, all their edges + pull shrinks — fewer unvisited vertices, each finding a hit sooner +``` + +They cross somewhere in the middle levels, which is the whole +paper. + +Why it matters: the early exit is the load-bearing part, and it is +also the part with an algebraic precondition (Step 7) — you cannot +transplant it into a semiring that needs every contribution. + +### Step 5 — what pull needs: the reverse graph and a dense frontier + +> **In:** pull's inner loop (Step 4). +> **Out:** its two prerequisites priced in bytes, and the reason +> LAGraph makes one of them optional. + +- **The reverse graph.** Pull scans *in*-edges, so it needs the + transpose Aᵀ (equivalently CSC — the same edges indexed by + destination instead of source). The paper is explicit about the + bill (§IV): "If the graph is directed, the bottom-up step will + require the inverse graph, **which could nearly double the + graph's memory footprint**." (For an undirected graph, it says, + "performing the bottom-up approach requires no modification to + the graph data structures as both directions are already + represented.") This is why LAGraph takes `G->AT` as an *optional* + cached property and silently degrades to push-only without it — + `LG_BreadthFirstSearch_SSGrB_template.c:18-22`, and the flag it + computes at `:128`. +- **A dense frontier representation.** Pull tests "is u in the frontier?" once per probe, so membership must be O(1) — a - **bitmap** (one bit per vertex; n/8 bytes total) rather than the - sparse list of vertex IDs that push iterates. Converting between - the two representations at each switch is itself an O(n) cost the - switch heuristic must respect. + **bitmap** (one bit per vertex) rather than the sparse list of + vertex ids that push iterates. The paper (§V): "Different data + structures are used since the frontiers are of radically + different sizes, and the conversion costs are far less than the + penalty of using the wrong data structure." + +Price both on this topic's own RMAT scale-18 graph, whose figures +are in `topics/20-graphblas/notes.md:13` and `:35` +(n = 262,144, nnz = 2.0M): + +``` + inputs: n = 262,144 vertices, m = 2.0e6 edges, 4-byte indices + + CSR alone : 4·(n+1) + 4·m = 1.05 MB + 8.0 MB = 9.05 MB + CSR + CSC (Aᵀ) : 2 × 9.05 MB = 18.10 MB + → the paper's "nearly double", exactly + + sparse frontier at the apex (say 40% of n in it): + 4 bytes × 104,858 = 419 KB + bitmap frontier, always: + n / 8 = 32 KB + → and it fits in L2, so each membership test is a cache hit + rather than a random 419 KB gather (topic 13's lesson) + + conversion cost per switch: one O(n) pass = 262,144 writes. + At notes.md:35's measured 1.6 ns/edge for this graph's BFS, an + O(n) pass is ≈ 0.4 ms against a 3.3 ms whole-search budget — + about 13% of the search per switch. You can afford two switches. + You cannot afford one per level. +``` + +Why it matters: the conversion is not free, and the size of that +13% is precisely why the switch heuristic needs hysteresis rather +than a single threshold (Step 6). -### Step 5 — the switch: two thresholds, with hysteresis +### Step 6 — the switch, as the paper actually states it -Per level, pick the cheaper direction. Beamer's heuristic compares -estimated push work against a slice of the unexplored edges, with -*asymmetric* thresholds so the algorithm doesn't oscillate: +> **In:** push's m_f (Step 3), pull's shrinking cost (Step 4), the +> conversion tax (Step 5). +> **Out:** the paper's two thresholds with their exact constants and +> the section they come from — not the version most summaries give. + +The paper defines three quantities in §V: **m_f** (edges to check +from the frontier), **n_f** (vertices in the frontier), and **m_u** +(edges to check from unexplored vertices). Fig. 7 is the control +algorithm, and it is a two-state machine, not a formula: ``` - push → pull: m_frontier_out > m_unexplored / α (α = 14 paper, - or |frontier| > n/β1 8 in LAGraph) - pull → push: |frontier| < n / β2 (β = 24 paper, - 512 in LAGraph) + Beamer §V, Fig. 7 — Control algorithm for hybrid algorithm + + m_f > C_TB & growing + Top-Down ─────────────────────→ Bottom-Up + ←───────────────────── + n_f < C_BT & shrinking + + with, from §V: + C_TB = m_u / α (switch to bottom-up) + C_BT = n / β (switch back to top-down) + + and, from §VI-B (the tuning sweeps, Figs. 8 and 9): + α = 14 β = 24 ``` -Asymmetric thresholds = hysteresis (same instinct as SuiteSparse's -format switches). LAGraph adds a refinement: track -`edges_unexplored` incrementally by subtracting frontier degrees -(template :196, :261-277) — the heuristic input is maintained, not -recomputed. Result on scale-free graphs: 3-8× total edge -inspections saved; on high-diameter graphs (road networks) pull -never triggers and the machinery must cost ~nothing — a heuristic -is judged on the workload where it *doesn't* fire, too. +Three things in that block are routinely gotten wrong, so read +them off the paper directly: + +1. **α gates edges, β gates vertices.** m_f > m_u/α is a + comparison between two *edge* counts. n_f < n/β is a comparison + between two *vertex* counts. They are not the same test in + different units. +2. **There is no `n_f > n/β₁` push→pull condition in the paper.** + The push→pull test is the α test alone. (LAGraph adds a + vertex-count test — Step 8 — but it is LAGraph's, not Beamer's.) +3. **`growing`/`shrinking` are separate conjuncts and they are + worth measuring.** Fig. 7's caption: "Growing and shrinking + refer to the frontier size, and although they are typically + redundant, their inclusion yields a speedup of about 10%." + +On the constants, §VI-B is candid about how much they matter: + +> "Sweeping α across a wide range demonstrates that once α is +> sufficiently large (>12), BFS performance for many graphs is +> relatively insensitive to its value (Figure 8)… we select α = 14 +> since it maximizes the average and minimum… even if a +> less-than-optimal α is selected, the hybrid-heuristic algorithm +> still executes within 15–20% of its peak performance on most +> graphs." + +> "Tuning β is less important than tuning α. We select β = 24… +> The value of β has a smaller impact on overall performance +> because the majority of the runtime is taken by the middle steps +> when the frontier is at its largest." + +And on what the whole thing buys, §VI-C, on the 16-core machine of +Table II, against two top-down baselines (Fig. 10): + +> "The hybrid provides large speedups across all of the graphs, +> with an average speedup of 3.9 and a speedup no lower than 2.4. +> The on-line heuristic often obtains performance within 10% of the +> oracle." + +Note the gap between §VI-C's 3.9× *speedup* and Step 2's 67× +*edge-check headroom*. §VI-D explains it and quantifies the +conversion: Fig. 13 plots speedup against edge-check reduction and +"the slope of a best-fit line is approximately 0.3", because "while +the bottom-up approach skips edges, it reduces the spatial locality +of the remaining memory accesses". Skipping an edge is worth about +a third of an edge. + +Why it matters: 0.3 is the exchange rate between the algorithmic +win and the measured win, and it is the number that stops you +predicting a 67× speedup from a 67× work reduction. + +### Step 7 — the linear-algebra translation: push = vxm, pull = mxv + +> **In:** the two directions (Steps 3-4) and the switch (Step 6). +> **Out:** each direction as one GraphBLAS call, and the algebraic +> precondition the early exit needs to stay legal. + +Davis states the correspondence in one paragraph +("Parallel GraphBLAS with OpenMP", CSC '20, §4.3), for matrices in +the default CSR format: + +> "The basic operation of this algorithm computes Aᵀq where q is +> the queue of nodes in the current level. This can be done with +> `GrB_vxm(q,A)` = (qᵀA)ᵀ = Aᵀq, or by `GrB_mxv(B,q)` = Bq = Aᵀq, +> where B = Aᵀ is the explicit transpose of A. Both steps compute +> the same thing, just in a different way; **the first is a push +> step and the second is a pull step**." + +Yang §4.1 gives the masked form of push: `f' = Aᵀf .* ¬v` — the +product, then filtered by "not yet visited". Yang §4.2 gives pull: +start *from* ¬v and look at each node's parents. -### Step 6 — the linear-algebra translation: push=vxm, pull=mxv +``` + push = qᵀ · A sparse vector × CSR → SpMSpV, saxpy engine + pull = Aᵀ · q CSR(Aᵀ) × vector → masked SpMV, dot engine + visited mask = the COMPLEMENTED structural mask (GrB_DESC_RSC) + direction switch = which of the two calls you make this level +``` -Yang/Buluç/Owens (ICPP '18) showed the whole construction is two -GraphBLAS calls plus the switch: +Which engine SuiteSparse actually picks for each, and why, is +`GB_AxB_dot2_control.c` and `GB_AxB_saxpy3.c` — walked in +[reading-suitesparse-internals.md](reading-suitesparse-internals.md). +Davis CSC '20 §3.1 states the intent directly: "By default, +GraphBLAS selects the masked-dot-product method for triangle +counting, LCC, **the pull phase of the push/pull BFS**… The +saxpy-based Gustavson or heap-based methods are used in the +K-truss, **the push phase of the push/pull BFS**…" + +The algebraic precondition. Pull's `break` says: *having found one +witness, stop looking for more.* That is only sound if the additive +monoid is **idempotent and selective** — if combining more +witnesses cannot change the answer. LAGraph's level-only BFS uses +`LAGraph_any_one_bool`, which `LAGraph.h:825-829` documents as +"using the `GrB_MIN_MONOID_T` for non-boolean types or +`GrB_LOR_MONOID_BOOL` for boolean, and the `GrB_ONEB_T` +multiplicative op". For booleans that is (OR, true): once you have +a `true`, every further `true` is absorbed, so stopping is not an +approximation. Under PLUS you must visit every contribution, and +the same `break` silently computes the wrong number — which is why +PageRank (Step 8's `LAGr_PageRankGAP.c`) never gets an early exit. + +Why it matters: "stop at the first hit" is a *semiring* property, +not a coding trick. Getting this wrong is how a fast BFS becomes a +wrong PageRank. + +### Step 8 — what LAGraph actually shipped, and where it diverges + +> **In:** the paper's Fig. 7 machine and constants (Step 6). +> **Out:** the shipped constants, the shipped control flow — which +> is not Fig. 7 — and the arithmetic that explains both changes. + +Here is the whole switch as LAGraph ships it. Read it against +Fig. 7 and note where the shapes differ: + +```c +// LG_BreadthFirstSearch_SSGrB_template.c — the constants, 183-188 + 183 GrB_Index nq = 1 ; // number of nodes in the current level + 184 double alpha = 8.0 ; + 185 double beta1 = 8.0 ; + 186 double beta2 = 512.0 ; + 187 int64_t n_over_beta1 = (int64_t) (((double) n) / beta1) ; + 188 int64_t n_over_beta2 = (int64_t) (((double) n) / beta2) ; +``` +```c +// LG_BreadthFirstSearch_SSGrB_template.c — the switch, 243-294. +// Three mutually exclusive branches, not one disjunction. + 243 if (do_push) + 244 { + 245 // check for switch from push to pull + 246 bool growing = nq > last_nq ; + 247 bool switch_to_pull = false ; + 248 if (edges_unexplored < n) + 249 { + 250 // very little of the graph is left; disable the pull + 251 push_pull = false ; + 252 } + 253 else if (any_pull) + 254 { + ... // (comment 255-260: after a pull phase the edge count is + ... // no longer tracked, so fall back on frontier size) + 261 switch_to_pull = (growing && nq > n_over_beta1) ; + 262 } + 263 else + 264 { + ... // w = Degree ; then sum it (268-274) + 275 edges_unexplored -= edges_in_frontier ; + 276 switch_to_pull = growing && + 277 (edges_in_frontier > (edges_unexplored / alpha)) ; + 278 } + ... + 285 else + 286 { + 287 // check for switch from pull to push + 288 bool shrinking = nq < last_nq ; + 289 if (shrinking && (nq <= n_over_beta2)) + 290 { + 291 do_push = true ; + 292 } + 293 } ``` - push = q' * A sparse vector × CSR = SpMSpV (saxpy engine) - pull = AT * q CSR(AT) × vector w/ mask = masked SpMV (dot - engine, ANY monoid ⇒ early exit is LEGAL) - visited mask = the complemented structural mask (GrB_DESC_RSC) - direction switch = engine dispatch on frontier density + +Four divergences from Fig. 7, each verifiable in the block above: + +1. **Direction optimization is switched off entirely, not switched + over, when the graph is nearly exhausted** — `:248-251`. Fig. 7 + has no such state. +2. **The α test and the β₁ test are in mutually exclusive + branches** (`:253-262` versus `:263-278`), selected by whether a + pull phase has already happened. It is *not* + `α-test OR β₁-test`. The β₁ path exists only because + `edges_unexplored` stops being maintained once pull runs — the + comment at `:255-260` says exactly that. +3. **`edges_unexplored` is maintained, not recomputed.** `:268-269` + masks the degree vector by the frontier, `:273-274` reduces it, + `:275` subtracts. That is a masked assign plus a reduce per + level — the paper's own §VI-D notes "conversion and m_f + calculation take a non-negligible fraction of the runtime". + Note the ordering: the subtraction at `:275` happens *before* + the comparison at `:276-277`, so α is applied to the count that + already excludes this level. +4. **The constants all moved.** Work out what they mean on this + topic's RMAT scale-18 graph (n = 262,144, m = 2.0e6, from + `notes.md:13`): + ``` + threshold value at n = 262,144 + Beamer α = 14 m_f > m_u/14 = 7.1% of remaining edges + LAGraph α = 8 m_f > m_u/8 = 12.5% of remaining edges + → LAGraph sets a HIGHER bar, so it switches to pull LATER + + Beamer β = 24 n_f < n/24 = 10,922 vertices + LAGraph β₂= 512 n_q ≤ n/512 = 512 vertices + → LAGraph waits until the frontier is 21× smaller before + switching back to push + + LAGraph β₁= 8 n_q > n/8 = 32,768 vertices + → only consulted on a second push→pull switch, which the + comment at :258 calls "unlikely" +``` + +Why β₂ moved from 24 to 512 is question 5, and the arithmetic in +Step 5 is the hypothesis: switching back costs an O(n) frontier +rebuild (≈ 0.4 ms of a 3.3 ms search), so a switch that only saves +the last few hundred vertices' worth of pull scanning cannot pay +for itself. 512 buys about 21× more certainty that the tail is +genuinely over. -The ANY monoid (an accumulator that may keep *any one* of the -values combined into it, so the reduction may stop at the first) -is what makes pull's `break` algebraically sound — question 2. The -profound part: SuiteSparse's *format* switch (sparse↔bitmap -vector) and *engine* switch (saxpy↔dot) mirror the push↔pull -switch — the same decision at three abstraction levels. Our stub -implements all three explicitly in ~100 lines. +Why it matters: three published statements of "the" heuristic +(Beamer's Fig. 7, Yang's §6.3 with α = β = 0.01, LAGraph's +`:243-294`) disagree in shape as well as in constants. Any claim +about "the switch condition" has to name which one. ## How to read the paper (with the concepts in hand) -- **§3-4** — the two algorithms and the waste argument: steps 2-3 - in the authors' words, with measured per-level edge-inspection - counts. Compare their per-level plots with gb_bench's `--trace` - output on an RMAT graph. -- **§5** — the α/β tuning: step 5. This is the part LAGraph copied - (with different constants — question 5 asks why β2 moved from 24 - to 512). -- **Yang, Buluç, Owens, §3** — the push=vxm / pull=mxv translation - (step 6); read it after the Beamer paper, with the LAGraph - template open — the three texts are one idea at three levels of - abstraction. +Read in this order; each text is the next one's input. + +- **Beamer §III** — the waste argument and the kron27 arithmetic of + Step 2. Fig. 3 (absolute) and Fig. 4 (percentage) are the same + data; Fig. 4 is the one to internalise. Compare its per-level + shape with `gb_bench`'s `--trace` output on an RMAT graph. +- **Beamer §IV** — pull in one page, Fig. 5's `break` included, plus + the two costs (atomics removed, transpose required). +- **Beamer §V and Fig. 7** — the control machine of Step 6. Read + Fig. 7's *caption* as carefully as the box: the 10% for + growing/shrinking is in it. +- **Beamer §VI-B** (Figs. 8-9, the α/β sweeps) then **§VI-C** + (Fig. 10, the speedups) then **§VI-D** (Fig. 13, the 0.3 slope) — + in that order, because §VI-D is what stops you over-reading + §VI-C. +- **Yang, Buluç & Owens, ICPP '18, §4 and §5** — the linear-algebra + translation (§4.1 push, §4.2 pull) and the ablation of Step 4's + Table 2. §6.3 restates Beamer's heuristic and then replaces it + with α = β = 0.01 on a *vertex* ratio, which is a third distinct + formulation. +- **LAGraph `LG_BreadthFirstSearch_SSGrB_template.c:183-188` and + `:241-296`** — Step 8. Read it last, with Fig. 7 beside it, and + find the four divergences yourself before re-reading Step 8. ## Questions for notes.md -1. Reproduce Beamer's waste argument from gb_bench's per-level +1. Reproduce Beamer's waste argument from `gb_bench`'s per-level trace: at the peak level, what fraction of push's edge checks found an already-visited target (count them — add a counter to - the stub)? + the stub)? Split the failures into the paper's categories + (failed child / peer / valid parent, §III) and compare the shape + against Fig. 4. 2. Why does pull's early exit require the ANY (or OR) monoid algebraically — what property (idempotent, any-witness-suffices) makes stopping sound, and which semirings BREAK it (PLUS: you need every contribution — BFS parent vs PageRank)? 3. Road network vs RMAT: predict which levels (if any) go pull on each, from diameter and degree distribution alone. Then check - with gb_bench --trace. + with `gb_bench --trace`. Use `notes.md:35`'s path-100K figure + (2041 µs, ~20 ns/hop) as the road-like case. 4. The reverse graph doubles memory. FalkorDB keeps BOTH (the - transposed delta trio, delta_matrix.h:20-22) — for which query - shapes besides BFS pull is AT load-bearing (incoming-edge + transposed delta trio, `delta_matrix.h:20-22`) — for which query + shapes besides BFS pull is Aᵀ load-bearing (incoming-edge traversals `<-[]-`)? -5. LAGraph's β2=512 (vs paper's 24) makes pull→push switch-back - very late. Hypothesize why (switch-back cost includes rebuilding - a SPARSE frontier from a bitmap — O(n) scan), and design the - experiment that would confirm it. +5. LAGraph's β₂ = 512 (vs the paper's β = 24) makes the pull→push + switch-back very late. Hypothesize why, using Step 5's + conversion arithmetic (switch-back rebuilds a SPARSE frontier + from a bitmap — an O(n) scan), and design the experiment that + would confirm it. ## Done when +Answer each before unfolding it. + +- [ ] You can state, with the paper's own number, how much of a top-down BFS's work is avoidable in principle. + +
Answer + + About 98.5%. §III: on `kron27` (128M vertices, 2B undirected + edges, Fig. 3 caption) the BFS tree contains 63,036,116 vertices, + so at least 63,036,115 edges must be considered — "about 1/67th + of all the edge examinations that would happen during a top-down + traversal". The 67 decomposes as roughly degree 16 × 2 (each + undirected edge is checked from both endpoints) × ~2 (zero-degree + vertices are outside the component, raising the effective degree + of those inside). + + The measured speedup is far smaller — §VI-C's average 3.9 — and + §VI-D explains the gap: Fig. 13's best-fit slope of speedup + against edge-check reduction is "approximately 0.3", because + bottom-up "reduces the spatial locality of the remaining memory + accesses". Skipping an edge is worth about a third of an edge. + +
+ - [ ] You can explain why push dies mid-search, in terms of edges checked per useful discovery. -- [ ] You can explain what pull inverts and why its early exit needs the ANY/OR monoid. -- [ ] You can state what pull requires — the reverse graph and a dense frontier — and what that costs in memory. -- [ ] You can describe the two thresholds and why hysteresis is needed. -- [ ] You can write the algebraic translation: push is `vxm`, pull is `mxv`. + +
Answer + + Push's cost per level is m_f, the sum of the frontier's + out-degrees (§V), and it is paid whether or not a check + discovers anything. On a small-world graph the frontier grows + exponentially for two or three levels and then holds most of the + graph, so m_f peaks exactly when there is almost nothing left to + discover. Fig. 4 shows the composition at that peak: claimed + children collapse and **peer** edges dominate — both endpoints + are already in the frontier, so the check cannot succeed. + + There is a second cost §IV names: "there could be multiple + parallel writers to the same child, so atomic operations are + needed to ensure mutual exclusion". The failed children are also + the contended CAS operations. + +
+ +- [ ] You can explain what pull inverts, and say which of pull's two ingredients the measurements say actually pays. + +
Answer + + Pull inverts the direction of the scan: instead of each frontier + vertex scanning its out-edges for unvisited children, each + *unvisited* vertex scans its in-edges for a parent in the + frontier, and stops at the first one (§IV, Fig. 5's `break`). + + The inversion alone is nearly worthless. Yang, Buluç & Owens + Table 2, on `kron_g500-logn21`, ablates the construction: + "change of direction" is a **1.08×** standalone speedup, while + "early exit" is **4.02×** — and their §5.3 says early-exit + "yielded the greatest speed-up" of the five optimizations. The + `break` is the mechanism; the direction flip is what makes the + `break` possible. An implementation with the flip and without the + exit has kept 8% of the idea. + +
+ +- [ ] You can price pull's two prerequisites in bytes on this topic's scale-18 graph. + +
Answer + + n = 262,144, m = 2.0e6 (`notes.md:13`), 4-byte indices. + + *Reverse graph*: CSR alone is 4·(n+1) + 4·m = 1.05 + 8.0 = + 9.05 MB; keeping Aᵀ as well is 18.10 MB — the paper's §IV "could + nearly double the graph's memory footprint", exactly. LAGraph + therefore treats `G->AT` as optional and falls back to push-only + (`LG_BreadthFirstSearch_SSGrB_template.c:18-22`, flag at `:128`). + + *Dense frontier*: a bitmap is n/8 = 32 KB regardless of frontier + size, against 419 KB for a 4-byte id list at a 40%-of-n apex. + The bitmap's real win is not the 13× — it is that 32 KB stays in + L2, so each membership test is a cache hit. + + *Conversion*: one O(n) pass per switch. At `notes.md:35`'s + measured ~1.6 ns/edge on this graph, that is ≈ 0.4 ms against a + 3.3 ms search — about 13% per switch. Two switches are + affordable; one per level is not, which is what the hysteresis in + Fig. 7 exists to prevent. + +
+ +- [ ] You can write down Beamer's two thresholds exactly, with their constants and the section they come from — and say what the third conjunct in each is worth. + +
Answer + + §V and Fig. 7. Top-down → bottom-up when **m_f > C_TB and + growing**, where **C_TB = m_u/α**. Bottom-up → top-down when + **n_f < C_BT and shrinking**, where **C_BT = n/β**. m_f is the + edges to check from the frontier, m_u the edges to check from + unexplored vertices, n_f the vertices in the frontier. + + Constants from §VI-B's sweeps: **α = 14**, **β = 24**. §VI-B adds + that performance is "relatively insensitive" to α once α > 12, + and a mistuned α still runs "within 15–20% of its peak". + + The `growing`/`shrinking` conjuncts are worth ~10%: Fig. 7's + caption says they are "typically redundant" but "their inclusion + yields a speedup of about 10%". + + Note what is *not* there: the paper has no `n_f > n/β₁` push→pull + condition. α gates an edge-count comparison; β gates a + vertex-count comparison. + +
+ +- [ ] You can write the algebraic translation — push is `vxm`, pull is `mxv` — and name the property that makes the early exit legal. + +
Answer + + Davis, CSC '20 §4.3, for CSR matrices: `GrB_vxm(q,A)` = (qᵀA)ᵀ = + Aᵀq is the **push** step; `GrB_mxv(B,q)` with B = Aᵀ is the + **pull** step. Both compute Aᵀq. Yang §4.1 gives push's masked + form as `f' = Aᵀf .* ¬v`; the "not yet visited" filter is a + complemented structural mask, `GrB_DESC_RSC` in LAGraph + (template `:307` for push, `:313` for pull). + + The early exit is legal when the additive monoid is idempotent + and selective, so that additional witnesses cannot change the + result. LAGraph's level-only BFS uses `LAGraph_any_one_bool`, + documented at `LAGraph.h:825-829` as `GrB_LOR_MONOID_BOOL` with + `GrB_ONEB_BOOL` — (OR, true). Once a `true` has been produced, + every further `true` is absorbed. Under PLUS every contribution + matters and the same `break` computes a wrong number, which is + why `LAGr_PageRankGAP.c:135-136`'s `PLUS_SECOND` mxv is dense and + unmasked. + +
+ +- [ ] You can state how LAGraph's shipped switch differs in *shape* — not just in constants — from Fig. 7. + +
Answer + + Three shape differences, all in + `LG_BreadthFirstSearch_SSGrB_template.c:243-294`. + + First, a state Fig. 7 does not have: at `:248-251`, if + `edges_unexplored < n`, direction optimization is **disabled + outright** (`push_pull = false`) rather than switched. + + Second, the α test and the β₁ test are in **mutually exclusive + branches**, not a disjunction. `:253-262` (frontier-size test + `growing && nq > n_over_beta1`) runs only when `any_pull` is + already true; `:263-278` (the edge test + `growing && edges_in_frontier > edges_unexplored/alpha`) runs + only the first time. The comment at `:255-260` gives the reason: + after a pull phase, `edges_unexplored` is no longer tracked, so + the edge test has no valid input. + + Third, `edges_unexplored` is *maintained* — masked assign at + `:268-269`, reduce at `:273-274`, subtract at `:275` — and the + subtraction happens before the comparison at `:276-277`, so α is + applied to a remainder that already excludes the current level. + + Constants: α 14 → 8 (a higher bar, so pull starts later: 12.5% of + remaining edges rather than 7.1%), β 24 → β₂ 512 (n/512 = 512 + vertices at n = 262,144, against n/24 = 10,922 — a 21× later + switch-back), plus a new β₁ = 8 used only on the unlikely second + switch. + +
+ - [ ] You wrote answers to all five questions in notes.md, and can reproduce Beamer's waste argument from `gb_bench`'s per-level trace once `bfs_diropt` runs. +
Answer + + The trace has to show, per level: frontier size, edges inspected, + and the outcome split. The check on your implementation is that + the peak level's *claimed child* fraction collapses toward zero + while *peer* rises to dominate — Fig. 4's shape. If your trace + shows failed children dominating at the apex rather than peers, + the graph is not small-world enough for the paper's story, which + is itself the finding (question 3's road-network case). + + On the two flat predictions in `notes.md:52-55`: pull should not + fire at all on path-100K, because the frontier never grows past + a handful of vertices, so `growing && edges_in_frontier > + edges_unexplored/8` is never satisfied — and `:248-251` will + disable direction optimization outright as the path drains. + A pull-only run on that graph is the catastrophe case: n probes + per level × 100K levels. + +
+ ## References **Papers** -- Beamer, Asanović, Patterson — "Direction-Optimizing - Breadth-First Search" (SC 2012) — §3-4 are the two algorithms and - the waste argument; §5's α/β tuning is the part LAGraph copied -- Yang, Buluç, Owens — "Implementing Push-Pull Efficiently in - GraphBLAS" (ICPP 2018, - [arXiv:1804.03327](https://arxiv.org/abs/1804.03327)) — the - push=vxm / pull=mxv translation in §3 + +- Beamer, Asanović, Patterson — **"Direction-Optimizing + Breadth-First Search"**, SC '12. Read §III (waste, Figs. 3-4), + §IV (bottom-up, Fig. 5), §V (the hybrid, Fig. 7), §VI-B (α/β + sweeps, Figs. 8-9), §VI-C (speedups, Fig. 10), §VI-D (Fig. 13's + 0.3 slope). Every paper figure in this guide is from that PDF. +- Yang, Buluç, Owens — **"Implementing Push-Pull Efficiently in + GraphBLAS"**, ICPP '18, + [doi:10.1145/3225058.3225122](https://doi.org/10.1145/3225058.3225122) + (Article 89; preprint + [arXiv:1804.03327](https://arxiv.org/abs/1804.03327)). §4 is the + vxm/mxv translation, §5 the five-optimization ablation whose + Table 2 is quoted in Step 4, §6.3 their own α = β = 0.01 + heuristic. LAGraph's template cites the DOI form at its `:26-29`. +- Davis — **"Parallel GraphBLAS with OpenMP"**, CSC '20 (SIAM + Workshop on Combinatorial Scientific Computing). §4.3 is the + one-paragraph push = `vxm` / pull = `mxv` statement quoted in + Step 7; §3.1 says which engine SuiteSparse picks for each. **Code** -- [LAGraph](https://github.com/GraphBLAS/LAGraph) - `src/algorithm/template/LG_BreadthFirstSearch_SSGrB_template.c` — - the shipped thresholds (:184-187) and switch logic (:243-292); - walked in [reading-lagraph.md](reading-lagraph.md) + +- [LAGraph](https://github.com/GraphBLAS/LAGraph) at `e2539e2`, + `src/algorithm/template/LG_BreadthFirstSearch_SSGrB_template.c` + (355 lines) — the shipped algorithm, walked in + [reading-lagraph.md](reading-lagraph.md). + +| File | Lines | What | +|------|-------|------| +| `LG_BreadthFirstSearch_SSGrB_template.c` | 18-22 | `G->AT` and `G->out_degree` optional; degrade to push-only | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 128 | `push_pull` computed from what the graph offers | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 161 | `LAGraph_any_one_bool` — the level-only semiring | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 183-188 | α = 8, β₁ = 8, β₂ = 512 and the two derived bounds | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 248-251 | direction optimization disabled when `edges_unexplored < n` | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 253-262 | the β₁ branch, taken only after a pull phase | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 263-278 | the α branch: maintain `edges_unexplored`, then compare | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 288-289 | pull → push: `shrinking && nq <= n/β₂` | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 303-308 | push: `LG_SET_FORMAT_HINT(q, LG_SPARSE)` then `GrB_vxm` | +| `LG_BreadthFirstSearch_SSGrB_template.c` | 309-314 | pull: `LG_SET_FORMAT_HINT(q, LG_BITMAP)` then `GrB_mxv` | +| `include/LAGraph.h` | 825-829 | `LAGraph_any_one_bool` = (LOR, ONEB) for booleans | + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:13` — RMAT scale 18: n = 262,144, + nnz = 2.0M, the graph Steps 5 and 8 do arithmetic on. +- `topics/20-graphblas/notes.md:35` — BFS scalar oracle: rmat18 + 3308 µs (~1.6 ns/edge), path-100K 2041 µs (~20 ns/hop). diff --git a/topics/20-graphblas/reading-davis-toms19.md b/topics/20-graphblas/reading-davis-toms19.md index b6e4150..852af3a 100644 --- a/topics/20-graphblas/reading-davis-toms19.md +++ b/topics/20-graphblas/reading-davis-toms19.md @@ -8,6 +8,17 @@ clothing. Before you open it, this chapter builds the six concepts the paper assumes, one at a time — then hands you a reading route and the numbers worth retaining. +**Read the version numbers before you read anything else**, because +this chapter's single biggest hazard is quoting the 2019 paper about +a 2025 library: + +| source | describes | status here | +|---|---|---| +| TOMS '19, "Algorithm 1000" | version **2.3.3**, **single-threaded**, **two** sparsity structures | quoted from the author's accepted manuscript, titled "Algorithm 9xx" | +| Davis, CSC '20, "Parallel GraphBLAS with OpenMP" | version **3.0.1**, the first parallel release | the citable source for every parallelism claim | +| TOMS '23, "Algorithm 1037" | version 7.x: JIT, iso, 32/64-bit indices | **could not be downloaded**; no claim below is attributed to it | +| the pinned source | **version 10.3.1** (`Include/GraphBLAS.h:290-292`), **four** sparsity structures × 2 orientations | where every v2-era claim below is verified instead | + ## The problem in one sentence One `GrB_mxm` call must behave well whether the operand is a @@ -21,21 +32,54 @@ handle. ### Step 1 — a graph is a sparse matrix; store only what exists +> **In:** a graph, and the definition of a matrix. +> **Out:** the single cost driver for this whole topic, and the +> order of magnitude it saves. + A **sparse matrix** is one where almost every cell is zero/absent, so you store only the present entries — each one an `(row, col, value)` fact. A graph maps onto this directly: the -**adjacency matrix** A has A(i,j) present iff there's an edge +**adjacency matrix** A has A(i,j) present iff there is an edge i→j, so "the graph" and "the matrix" are the same object. The count of present entries is **nnz** ("number of nonzeros") — the number that drives every cost in this topic. -Concrete: 10M nodes stored densely as booleans is 10M × 10M = -100 *trillion* cells (~12.5 TB at a bit each). The same graph with -100M edges stored sparsely is ~100M entries — roughly 1 GB with -32-bit indices. Everything in this paper is machinery for -exploiting the zeros you didn't store. +Davis is careful about what an absent entry *means*, and it is not +"zero" (TOMS '19 §4.1): + +> "MATLAB drops its entries with a numerical value of zero, but +> this is never done in GraphBLAS. A 'zero' is simply an entry that +> is not stored in the data structure, and **the value of this +> implicit entry depends on the semiring**. If the matrix is used +> in the conventional plus-times semiring, the implicit value is +> zero. If used in max-plus, the implicit entry is −∞." + +Concrete cost, on this topic's own configuration: + +``` + inputs: 10M-node id space (notes.md:41) + + dense boolean: 10e6 × 10e6 = 1.0e14 cells + at 1 bit each = 12.5 TB + sparse, 100M edges, 32-bit indices + 100e6 × 4 B = 400 MB of column indices + + ratio = 31,000× + + and the SPARSE representation still is not free: see Step 2, where + the row-pointer array alone costs 80 MB before a single edge. +``` + +Why it matters: everything in this paper is machinery for +exploiting the zeros you did not store, and the interesting failures +are the places where a *structure* proportional to the dimension +sneaks the cost back in. + +### Step 2 — the format ladder, and the two counts of it -### Step 2 — the format ladder: one matrix, four representations +> **In:** nnz and the matrix dimensions. +> **Out:** the four sparsity structures, why the source says +> "eight formats", and the byte cost of each rung. The standard sparse format is **CSR** (compressed sparse row): a `rowptr` array with one offset per row marking where that row's @@ -54,22 +98,111 @@ a ladder as its density changes: their ptrs) nvals ≪ nrows nvals ~ O(nrows) nvals > every cell - (10M×10M with the graph default ~4-8% of present + (10M×10M with the graph default 40% of present 100K edges) n×m ``` -**Hypersparse** matters most to FalkorDB: node IDs are a shared +Two corrections before you carry that picture into the paper. + +**First: the paper does not describe this ladder.** TOMS '19 §4.1 +says, in full: + +> "In SuiteSparse:GraphBLAS (**version 2.3.3**), a GraphBLAS matrix +> (the `GrB_Matrix` object) is stored in one of **four different +> formats: compressed-sparse column (standard CSC), +> compressed-sparse row (standard CSR), and hypersparse versions of +> these two formats** (hyper CSR and hyper CSC)." + +So the paper's "four formats" are *two* sparsity structures times +two orientations. Bitmap and full do not exist in it. The pinned +source counts differently again: + +```c +// GB_Matrix_content.h — how the source counts formats, 52 and 76 + 52 // The matrix can be held in one of 8 formats, each one consisting of a set of + 53 // vectors. The vector "names" are in the range 0 to A->vdim-1. + ... + 76 // The 8 formats: (hypersparse, sparse, bitmap, full) x (CSR or CSC) +``` + +Four **sparsity structures** × two **orientations** = eight +formats. Say which you mean. And note the consequence TOMS '19 §4.2 +draws from its own count — "the four matrices for C⟨M⟩ = AB give +rise to **256 variants** of the sparse matrix-matrix multiply" — +which at the pin's eight formats would be 8⁴ = 4,096. That +combinatorial explosion is the reason the JIT of Step 6 exists. + +**Second: the density thresholds.** The bitmap rung is not "~4-8%". +The struct's own comment states the rule: + +```c +// GB_Matrix_content.h — the bitmap rule, in the struct's comment, 450-457 + 450 // A->vdim can have at most anz_dense = (A->vlen)*(A->vdim) entries. + 451 // If A is sparse/hypersparse with anz > A->bitmap_switch * anz_dense, + 452 // then it switches to bitmap. If A is bitmap and anz = + 453 // (A->bitmap_switch / 2) * anz_dense, it switches to sparse. In + 454 // between those two regions, the sparsity structure is unchanged. + ... + 456 float hyper_switch ; // controls conversion hyper to/from sparse + 457 float bitmap_switch ; // controls conversion sparse to/from bitmap +``` + +and `bitmap_switch` defaults from a table indexed by the matrix's +*minimum dimension*, not by the operation: `GB_Global.c:181-189` +runs 0.04 → 0.40, and `GB_Global.c:486-497` selects **0.40** for +any dimension above 64. A graph adjacency matrix never becomes +bitmap; the arithmetic is in +[reading-suitesparse-internals.md](reading-suitesparse-internals.md) +Step 2. `hyper_switch` defaults to 0.0625 = 1/16 +(`GB_defaults.h:20`), which matches the paper's §4.2.1: "SuiteSparse: +GraphBLAS stores its matrices in hypersparse format if n̄ < n/16." + +Now price the ladder against this topic's measurement, because the +numbers reproduce exactly: + +``` + inputs: 10M-node id space, 100K edges (notes.md:41-42) + this repo's CSR: rowptr u64, colidx u32 + this repo's hyper: h u32, p u64, colidx u32 + + CSR index bytes + rowptr (10,000,000 + 1) × 8 = 80,000,008 + colidx 100,000 × 4 = 400,000 + total = 80,400,008 B = 80.4 MB ← measured: 80.4 MB + + hypersparse index bytes, with k distinct non-empty rows + h k × 4 + p (k+1) × 8 + colidx 100,000 × 4 = 400,000 + at k = 100,000: 400,000 + 800,008 + 400,000 = 1.60 MB + ← measured: 1.59 MB + the 0.01 MB gap is k slightly below 100,000: a few hundred of + the 100,000 edges share a source row. + + ratio 80.4 / 1.59 = 50.6× ← FINDINGS.md row 20 says "50x" + + and the term that vanished is exactly the O(n) rowptr: 80.0 of the + 80.4 MB, i.e. 99.5% of the CSR index, is pointers for rows that + hold nothing. +``` + +**Hypersparse matters most to FalkorDB**: node IDs are a shared namespace across all relation types, so most rows of any one -relation matrix are empty — and plain CSR's `rowptr` alone for 10M -nodes is 80 MB *per relation type*, before storing a single edge. -Hypersparse stores only the list of non-empty rows and their -pointers, so an almost-empty 10M×10M matrix costs KBs, not tens of -MBs. The switches between rungs are decided by two per-matrix -knobs (`hyper_switch`, `bitmap_switch`) applied after every -operation — the internals chapter reads that code. +relation matrix are empty, and that 80 MB is *per relation type* +before storing a single edge. The switches between rungs are decided +by the two per-matrix knobs applied after every operation — the +internals chapter reads that code. + +Why it matters: the topic's 50× headline is not a benchmark +artifact, it is `(n+1)×8` disappearing, and you can predict it to +within 1% with a pocket calculator. ### Step 3 — semiring, mask, accum: the GraphBLAS ops are executor concepts +> **In:** the matrix object of Steps 1-2. +> **Out:** the four operation parameters, each mapped onto a +> database-executor concept. + GraphBLAS operations are parameterized matrix products, and each parameter maps onto a database-executor concept: @@ -87,153 +220,582 @@ parameter maps onto a database-executor concept: - A **descriptor** (flags: transpose an input, complement the mask, replace C) is the query-hint block. -Why it matters: the paper's §3 describes these as an API; you -should read them as an *operator algebra* — the same shape as a -relational executor's, which is question 1 below. +The paper's §3.1.9 lists exactly these four as one bundle: + +> "Most GraphBLAS operations can be modified via transposing input +> matrices, using an accumulator operator, applying a mask or its +> complement, and by clearing all entries the matrix C after using +> it in the accumulator operator but before the final results are +> written back into it. All of these steps are optional, and are +> controlled by a **descriptor** object." + +You can read one off the header. `GrB_DESC_RSC` +(`Include/GraphBLAS.h:666`) is `GrB_REPLACE + GrB_STRUCTURE + +GrB_COMP` — replace C, use only the mask's pattern and ignore its +values, and complement it. That single constant is what LAGraph's +BFS passes to mean "write only where I have not been yet", and +Step 8 of the internals chapter shows how it decides which engine +runs. + +Why it matters: §3 reads like an API reference. Read it as an +*operator algebra* and question 1 answers itself. ### Step 4 — lazy mutation: zombies and pending tuples +> **In:** the packed CSR arrays of Step 2, which are expensive to +> splice. +> **Out:** the two deferral mechanisms, their exact encodings, and +> the complexity they buy. + CSR's packed arrays make single-entry mutation expensive: deleting one edge means splicing `colidx` (O(nnz) memmove), inserting one -means the same. SuiteSparse's answer is to *not do it yet*: +means the same. SuiteSparse's answer is to *not do it yet*. The +source defines both precisely: + +```c +// GB_Matrix_content.h — the zombie, 367-373 and 391 + 367 // A "zombie" is the opposite of a pending tuple. It is an entry A(i,j) that + 368 // has been marked for deletion, but has not been deleted yet because it is + 369 // more efficient to delete all zombies all at once, rather than one (or a few) + 370 // at a time. An entry A(i,j) is marked as a zombie by 'zombifying' its index + 371 // via GB_ZOMBIE (i). A zombie index is negative, and the actual index can be + 372 // obtained by GB_UNZOMBIE (i). GB_ZOMBIE (i) is a function that is its own + 373 // inverse: GB_ZOMBIE (GB_ZOMBIE (x))=x for all x. + ... + 391 uint64_t nzombies ; // number of zombies marked for deletion +``` -- a **zombie** is a deleted-but-still-present entry — deletion just - flags it in place; -- a **pending tuple** is an inserted-but-unsorted entry — insertion - appends to a side list, never touching the CSR. +TOMS '19 §4.1 gives the arithmetic the modern macro hides: "its row +index i is changed to **(-i-2)**, to accommodate zero-based row +indices". And the property that makes it worth the trouble: +"Zombies allow for fast deletion, and they also permit binary +searches of a sparse vector to performed, even if it contains +zombies" — the index is negated but the *ordering* is preserved +under the transform, so the sorted-array invariant survives. + +The source adds a second reason the paper does not stress +(`:375-383`): "a zombie may be restored as a regular entry by a +subsequent update… Had the zombie not been there, the update would +have to be placed in the pending tuple list." Delete-then-reinsert +is O(1) in place, and never lengthens the pending list. + +The pending tuple is the mirror image — TOMS '19 §4.1: "an entry +that has not yet been added to the compressed-sparse vector part of +the data structure. Pending tuples are held in an **unsorted list** +of row indices, column indices, and values. Duplicates may appear +in this list. The matrix also keeps track of a single operator to be +used to combine duplicate entries. A matrix can have both zombies +and pending tuples." The whole mechanism, distilled: ```rust +// ILLUSTRATION — not SuiteSparse source. The structural claims are +// GB_Matrix_content.h:361 (the Pending list), :367-373 and :391 (zombies, +// GB_ZOMBIE, nzombies) and TOMS '19 §3.1.8 (the O(e log e) bound). fn set_element(a: &mut Matrix, i: u64, j: u64, v: f64) { a.pending.push((i, j, v)); // O(1): append, don't restructure CSR } fn delete_element(a: &mut Matrix, i: u64, j: u64) { if let Some(e) = a.find_mut(i, j) { - e.mark_zombie(); // flag in place — no O(nnz) splice - } + e.mark_zombie(); // negate the index in place — no splice + } // ordering survives, binary search still works } fn wait(a: &mut Matrix) { // the GrB_wait boundary a.prune_zombies(); // one sweep drops ALL zombies - a.pending.sort_unstable(); // n inserts → one sort + one merge, - a.merge_pending_into_csr(); // not n binary-searched splices + a.pending.sort_unstable(); // e inserts → one sort + one merge, + a.merge_pending_into_csr(); // not e binary-searched splices conform(a); // then maybe switch format } ``` -The cost shape: n single inserts done eagerly = n O(nnz) splices; -done lazily = n O(1) appends + one sort + one merge. This is the -LSM memtable move (topic 3) inside a matrix library — and it's the -library's OWN delta mechanism, which makes FalkorDB's delta -matrices (this topic §5) look redundant until you ask who controls -the flush. Question 2. +The cost shape is the paper's headline claim, §3.1.8: e single +`GrB_Matrix_setElement` calls take **O(e log e)** in +SuiteSparse, where "the equivalent method in MATLAB takes **O(e²)** +time". Price it: + +``` + inputs: e = 100,000 edges inserted one at a time (notes.md:41) + + eager (MATLAB shape): e² = 1.0e10 index-slot moves + lazy (SuiteSparse): e log₂ e = 1.0e5 × 16.6 = 1.66e6 + + ratio = 6,000× + + at 1 ns per moved slot that is 10 seconds versus 1.7 milliseconds. + + and the paper's second claim: GrB_Matrix_build (all at once) is + ALSO O(e log e) — "both methods below take O(e log e) time". So + lazy incremental insertion costs the same asymptotically as + batch construction. That equality is the point of the section. +``` + +This is the LSM memtable move (topic 3) inside a matrix library — +and it is the library's OWN delta mechanism, which makes FalkorDB's +delta matrices (this topic §5) look redundant until you ask who +controls the flush. Question 2. + +Why it matters: `O(e log e)` for incremental inserts *equalling* +batch build is the sentence that licenses a graph database to be +built on a matrix library at all. ### Step 5 — non-blocking mode: the object model assembled +> **In:** Steps 2 and 4 — formats, zombies, pending tuples. +> **Out:** the full opaque handle, and who decides when the +> deferred work runs. + The GraphBLAS spec allows every operation to return before doing work; the deferred state is reconciled at **`GrB_wait`** boundaries — or forced implicitly by any operation that needs to -*read* the matrix. Assembling steps 2-4, the opaque handle looks -like: +*read* the matrix. The source says so for zombies specifically +(`GB_Matrix_content.h:386-389`): "methods and operations in +GraphBLAS that cannot tolerate zombies in their input matrices can +check the condition (A->nzombies > 0), and then delete all of them +if they appear, via GB_wait." + +Assembling steps 2-4 against the pinned struct, the opaque handle +looks like this — every line is a real field: ``` - GrB_Matrix = opaque header - ├─ format: hypersparse | sparse | bitmap | full (×2: by row/col) - ├─ pending tuples + zombies (lazy mutation!) - ├─ hyper_switch / bitmap_switch (per-matrix knobs) - └─ iso flag (all values equal — store ONE value) ← step 6 + GrB_Matrix = opaque header (GB_Matrix_content.h) + ├─ p, h, i, x, b :223-228 the five arrays; h only if hypersparse, + │ b only if bitmap, x only if not full-iso + ├─ nvals, nvec, nvec_nonempty :213-229 + ├─ Y :241-274 hyper_hash: a hash over h[] replacing + │ the binary search, load factor 2-4 + ├─ Pending :361 the unsorted insert list + ├─ nzombies :391 deleted-in-place count + ├─ hyper_switch/bitmap_switch :456-457 the two per-matrix knobs + ├─ sparsity_control :462 which of the four structures are allowed + ├─ is_csc / jumbled :497-498 orientation; "may be unsorted" + ├─ iso :524 all values equal ⇒ store ONE ← step 6 + └─ p_is_32/j_is_32/i_is_32 :534-536 per-array index width ← step 6 ``` +Two of those are worth a second look. `jumbled` (`:498`) is a +*deferred sort*: the matrix admits it may be out of order, which is +a third deferral mechanism alongside zombies and pending tuples. +And `Y` (`:241-274`) is a hash table over the hypersparse row list, +because otherwise finding row j in `h[]` is a binary search — the +`lg h` term that appears in `GB_AxB_saxpy3_flopcount.c:44-48`'s +complexity. Its documented load factor: "the load factor is +normally in the range of 2 to 4, so ideally each bucket will +contain about 4 entries on average". FalkorDB turns it *off* for +its delta matrices (`delta_new.c:33`, `GxB_HYPER_HASH` false) — +worth asking why. + SuiteSparse uses non-blocking mode for *mutation batching* -(pending tuples get sorted+merged once), not full lazy fusion (the -v2 paper discusses the JIT changing this calculus). Compare topic -27's incremental view maintenance: same "amortize small updates" -shape. The cost to remember: the flush point is chosen by the -*library* (any read can trigger it), not by the application — the -single fact that motivates FalkorDB's own delta layer. - -### Step 6 — the v2 update (TOMS '23): JIT, small indices, iso values - -Three changes since 2019, each a concrete constant-factor win: - -- the **CPU JIT** (topic 19's jitifyer) — user-defined - types/semirings now compile to specialized kernels at runtime - and run at factory speed, instead of through a - function-pointer-per-element fallback; -- **32/64-bit integer indices chosen per matrix** (v10) — halves - index memory for graphs under 4B edges, i.e. all of ours; -- **iso-valued matrices** — an **iso** matrix is one whose entries - all hold the same value, so it stores the pattern plus ONE - scalar and ZERO bytes of per-entry values. An unweighted graph - (A(i,j)=true for all edges) is exactly this. - -Iso + the (ANY,PAIR) semiring is why BFS over an unweighted +(pending tuples get sorted and merged once), not full lazy fusion. +Compare topic 27's incremental view maintenance: same "amortize +small updates" shape. The cost to remember: **the flush point is +chosen by the library** — any read can trigger it — not by the +application. That single fact is what motivates FalkorDB's own +delta layer, which exists precisely to move the decision. + +Why it matters: every deferral in this design is invisible until +something forces it, and the thing that forces it is not under the +caller's control. Hold that thought through +[reading-falkordb-delta-matrix.md](reading-falkordb-delta-matrix.md). + +### Step 6 — what changed after the paper: parallelism, JIT, iso, 32-bit + +> **In:** the 2019 design. +> **Out:** four changes, each attributed to a source that can be +> checked, and the one the paper flatly contradicts. + +**Parallelism.** TOMS '19 does not have it. Its own §4.2.1 says +"while SuiteSparse:GraphBLAS is not yet multi-threaded, it is +thread-safe", and §7 calls the work "an efficient and highly +optimized **single-threaded** implementation". Davis's CSC '20 §3 +dates the change: "Version 2.3.3 is to appear as a Collected +Algorithm… it does not exploit any parallelism at all. **Version +3.0.1 has been released (July 31, 2019), with exploitation of +multi-threaded parallelism expressed through OpenMP.**" Any +speedup number you attribute to "the TOMS paper" is wrong by +construction; use CSC '20's Table 2, which +[reading-openmp-vs-rayon.md](reading-openmp-vs-rayon.md) reads. + +**The CPU JIT** (topic 19's jitifyer). Verified at the pin: the +source tree has `Source/jitifyer/` with one encoder per operation — +`GB_encodify_mxm.c`, `GB_encodify_ewise.c`, `GB_encodify_reduce.c`, +`GB_encodify_select.c`, and nine more. What it buys is the +combinatorial problem of Step 2: TOMS '19 §4.2 counted 256 mxm +variants over four formats, and the pin has eight formats. Compiling +the one variant you need beats shipping 4,096 of them or falling +back to a function pointer per element. + +**Iso-valued matrices.** Verified at the pin, and the struct +explains itself: + +```c +// GB_Matrix_content.h — why iso exists, 513-524 + 513 // Instead, the common practice is to assign all entries present in the matrix + 514 // to be equal to a single value, typically 1 or true. SuiteSparse:GraphBLAS + 515 // exploits this typical practice by allowing for iso matrices, where all + 516 // entries present have the same value, held as A->x [0]. The sparsity + 517 // structure is kept, so in an iso matrix, A(i,j) is either equal to A->x [0], + 518 // or not present in the sparsity pattern of A. + ... + 521 // If A is full, A->x is the only component present, and thus a full iso matrix + 522 // takes only O(1) memory, regardless of its dimension. + 524 bool iso ; // true if all entries have the same value and only a +``` + +An unweighted graph — A(i,j) = true for every edge — is exactly +this. `:520-521`'s corollary is startling: a full iso matrix is +O(1) memory *regardless of dimension*, so `GrB_Matrix` can +represent an all-ones 10M×10M matrix in a handful of bytes. + +**32/64-bit indices per array.** Verified at the pin, and finer +than "per matrix": + +```c +// GB_Matrix_content.h — three independent width flags, 531-536 + 531 // A->p, A->h, and A->i can be either 32-bit or 64-bit integers. + ... + 534 bool p_is_32 ; // true if A->p is 32-bit, false if 64 + 535 bool j_is_32 ; // true if A->h and A->Y->[pix] are 32-bit, false if 64 + 536 bool i_is_32 ; // true if A->i is 32-bit, false if 64 +``` + +Three flags, three arrays, chosen independently — plus +`p_control`/`j_control`/`i_control` at `:464-466` so an application +can force any of them. Note which array each governs: `p_is_32` +covers the *offsets* (bounded by nnz), `i_is_32` the *indices* +(bounded by the dimension). A graph with 5B edges but 100M nodes +wants 64-bit `p` and 32-bit `i`, and this design lets it have +both. Question 5's arithmetic: + +``` + inputs: 10M nodes, 100M edges, CSR + + all-64-bit: p (10e6+1)×8 = 80.0 MB + i 100e6×8 = 800.0 MB + total index = 880.0 MB + + all-32-bit: p (10e6+1)×4 = 40.0 MB + i 100e6×4 = 400.0 MB + total index = 440.0 MB → exactly 2× + + mixed (p 64-bit, i 32-bit), which is what a 5B-edge graph needs: + p = 80.0 MB + i = 400.0 MB + total = 480.0 MB → still 1.83× + + legality check: i needs to address 10e6 < 2³¹, and p needs to + reach 100e6 < 2³¹ — so all-32-bit is legal here. It stops being + legal for p at 2.1e9 edges. +``` + +Iso plus the (ANY,PAIR) semiring is why BFS over an unweighted FalkorDB relation matrix moves no value data at all — pattern in, pattern out. Question 4 traces that path. +Why it matters: three of these four are checkable in the source in +under a minute, and the fourth (parallelism) is the one the paper +gets *actively wrong* if you read it as current. + ## How to read the paper (with the concepts in hand) -- **TOMS '19, §3 (object model)** — read closely. It's steps 2, 4, - and 5 in the authors' words; the code counterpart is one header, - `Source/matrix/GB_matrix.h`. Keep asking "what executor concept - is this?" (step 3's mapping — question 1). -- **TOMS '19, non-blocking mode discussion** — read closely - against step 5; note every place an implicit wait can fire. -- **TOMS '23 (the v2 update)** — read for the three items in - step 6; the JIT sections connect directly to topic 19. +- **TOMS '19, §3 (basic concepts)** — read §3.1.8 (non-blocking + mode) and §3.1.9 (accumulator and mask) closely; they are Steps + 3 and 4 in the author's words. §3.1.8 is two pages and contains + the `O(e log e)` versus `O(e²)` comparison whole. +- **TOMS '19, §4.1 (data structure)** — read closely against Step + 2, and keep the version caveat in view the entire time: it + describes *two* sparsity structures, and its "four formats" are + CSR/CSC × standard/hyper. The code counterpart at the pin is + `Source/builtin/include/GB_Matrix_content.h`, which is 657 lines + of comment-heavy struct and is genuinely readable top to bottom. +- **TOMS '19, §4.2.1 (matrix multiply)** — the three methods, + Gustavson's complexity, the hypersparse `n̄ < n/16` rule, and the + masked variant's "discarded if they are computed". Walked in + [reading-gustavson-spgemm.md](reading-gustavson-spgemm.md). +- **TOMS '19, §6 (performance)** — Table 6 is 3-truss throughput in + 10⁶ edges/s against hand-written C: roadNet-TX 10.8 (GraphBLAS) + vs 15.1 (sequential C) vs 56.6 (parallel C); cit-Patents 0.9 vs + 1.4 vs 11.5; g-1073643522 10.1 vs 38.6 vs 199.9. The paper's own + summary — "rarely taking more than twice the time as the + highly-optimized, sequential versions in pure C" — is the honest + claim; the parallel column is the gap version 3.0.1 was written + to close. +- **TOMS '23 (Algorithm 1037)** — not available to this repo. Do + not cite it from memory. Everything you would want from it — + JIT, iso, 32/64-bit indices, the hyper_hash — is verifiable in + the pinned source, as Steps 5 and 6 do. Numbers to retain while you read: -- format switch defaults: bitmap when nnz > ~4-8% (op-dependent), - hyper when non-empty vectors < hyper_switch × nrows (~1/16) -- saxpy3 hash→Gustavson threshold: hash table > m/16 ⇒ Gustavson - (the internals chapter reads this code) -- mxm engines: dot3 work ∝ nnz(M); saxpy3 work ∝ flops — the mask - changes the complexity CLASS, not a constant +- format switch defaults: **`bitmap_switch` is a table indexed by + `min(vlen, vdim)`, not by operation** — 0.04 for a dimension of + 1, rising to **0.40 for anything above 64** + (`GB_Global.c:181-189`, `:486-497`), with hysteresis at b/2 + (`GB_Matrix_content.h:450-454`). `hyper_switch` is 0.0625 + (`GB_defaults.h:20`), matching the paper's n/16. +- saxpy3's Gustavson-vs-hash threshold: **`hash_size >= cvlen/12`** + (`GB_AxB_saxpy3_slice_balanced.c:94`), plus an undocumented + `flmax >= cvlen/2` shortcut at `:65`. The widely repeated "m/16" + is a stale comment at `GB_AxB_saxpy3.c:57-58`. +- mxm engines: dot3's work is ∝ nnz(M) — provably, since + `GB_AxB_dot3.c:126` and `:171` size C to exactly nnz(M) — + while saxpy3's is ∝ flops. The mask changes the complexity + class, not a constant. +- this topic's measured SpMV bandwidth: **19.1 GB/s at scale 14 + falling to 15.8 GB/s at scale 20** (`notes.md:9-14`), against a + ~30 GB/s streaming baseline from topic 0/13. (`FINDINGS.md` row + 20 states the same decay as 20.7 → 12.3 GB/s. The two were + measured on different runs; cite one and say which.) ## Questions for notes.md 1. Map GrB objects to executor concepts: semiring ↔ ?, mask ↔ ?, accum ↔ ?, descriptor ↔ ? (operator, semi-join filter, UPDATE - expression, query hints — defend each). -2. Zombies+pending vs FalkorDB's DP/DM: why does FalkorDB need its - OWN deltas when the library already has them (control over WHEN - wait happens; transposed pair kept in lockstep; readers must see - pre-wait state — which reason dominates)? -3. The iso optimization: which FalkorDB matrices are iso (adjacency - bool — yes; relation with edge IDs as values — no). What does - losing iso cost on mxm bandwidth (values move again — 8×?)? + expression, query hints — defend each, and check your answer + against the four-item list in §3.1.9). +2. Zombies + pending vs FalkorDB's DP/DM: why does FalkorDB need + its OWN deltas when the library already has them? Candidates: + control over *when* wait happens; keeping the transposed pair in + lockstep; readers must see pre-wait state. Decide which + dominates, using Step 5's "the library chooses the flush point". +3. The iso optimization (`GB_Matrix_content.h:513-524`): which + FalkorDB matrices are iso — adjacency bool, yes; a relation + matrix holding edge IDs as values, no. What does losing iso cost + on mxm bandwidth? Compute it: an iso bool matrix moves 0 value + bytes per entry, a `uint64` relation matrix moves 8. Put that + against this topic's measured 15.8-19.1 GB/s + (`notes.md:9-14`). 4. Trace one BFS step through the v2 machinery: iso bool matrix, - ANY_PAIR semiring, sparse frontier — which engine runs - (saxpy3/SpMSpV), and what does the JIT specialize away? -5. 32-bit indices (v10): for a 10M-node 100M-edge graph, compute - the CSR memory in v9 (64-bit) vs v10 — and where the same 2× - shows up in our Rust CSR if we switch usize→u32. + ANY_PAIR semiring, sparse frontier — which engine runs, and what + does the JIT specialize away? Use + [reading-suitesparse-internals.md](reading-suitesparse-internals.md) + Step 8's dispatch trace, and remember that the pull step's + descriptor complements the mask. +5. 32-bit indices: for a 10M-node 100M-edge graph, compute the CSR + index memory with all-64-bit versus all-32-bit arrays, then + redo it with `p_is_32 = false, i_is_32 = true`. Where does the + same factor show up in our Rust CSR if we switch `usize` → `u32`, + and at what edge count does 32-bit `p` become illegal? ## Done when -- [ ] You can name the four sparsity formats and the rule that switches between them. +Answer each before unfolding it. + +- [ ] You can say which version each of your claims about this library comes from. + +
Answer + + TOMS '19 describes **version 2.3.3** and is **single-threaded** + (§4.2.1: "while SuiteSparse:GraphBLAS is not yet multi-threaded, + it is thread-safe"; §7: "an efficient and highly optimized + single-threaded implementation"). Parallelism arrives in 3.0.1, + per Davis's CSC '20 §3. The pin this repo reads is **10.3.1** + (`Include/GraphBLAS.h:290-292`). TOMS '23 could not be obtained, + so nothing here is attributed to it. + + The practical rule: performance claims → CSC '20 or the source; + data-structure claims → check the source, because §4.1 is two + sparsity structures behind. + +
+ +- [ ] You can name the four sparsity structures, say how the source counts formats, and give both switch rules. + +
Answer + + hypersparse, sparse, bitmap, full. The source counts **eight + formats** — `GB_Matrix_content.h:76`: "(hypersparse, sparse, + bitmap, full) x (CSR or CSC)". TOMS '19 §4.1 counts four, but its + four are CSC/CSR × standard/hyper: bitmap and full postdate it. + + Bitmap: switch up when `nnz > b × m·n`, back down when + `nnz <= (b/2) × m·n`, unchanged in between + (`GB_Matrix_content.h:450-454`). `b` comes from a table indexed + by `min(vlen, vdim)` — **0.40 for any dimension above 64** + (`GB_Global.c:189`, `:486-497`) — so it is neither "4-8%" nor + operation-dependent. + + Hyper: up when `k <= n × h`, down when `k > n × h × 2`, with + `h` = 0.0625 (`GB_defaults.h:20`). The paper agrees: "hypersparse + format if n̄ < n/16" (§4.2.1). + +
+ +- [ ] You can predict this topic's 50× hypersparse index saving from the array sizes. + +
Answer + + CSR index for a 10M id space holding 100K edges: rowptr + (10,000,001 × 8) + colidx (100,000 × 4) = 80,400,008 B = + **80.4 MB**, which is `notes.md:41`'s measured figure exactly. + + Hypersparse with k ≈ 100,000 non-empty rows: h (k × 4) + p + ((k+1) × 8) + colidx (100,000 × 4) = 1.60 MB against a measured + **1.59 MB** — within 1%, the gap being a few hundred edges + sharing a source row. + + 80.4 / 1.59 = 50.6×. And the term that disappeared is the + `(n+1)×8` rowptr: 80.0 of the 80.4 MB, 99.5% of the CSR index, + is pointers for rows that hold nothing. + +
+ - [ ] You can map semiring, mask and accumulator onto executor concepts. -- [ ] You can explain zombies and pending tuples, and why lazy mutation is necessary at all. -- [ ] You can say what non-blocking mode defers and what forces completion. + +
Answer + + Semiring → the inner loop's two operators, i.e. a pluggable + aggregate + combine, so one kernel computes matmul, shortest-path + relaxation, or reachability. Mask → a semi-join filter, which in + a dot engine becomes the *driving* iteration and changes the + complexity class. Accum → an UPDATE expression, merging into the + existing C. Descriptor → the query-hint block. + + §3.1.9 bundles all four in one sentence, and one constant proves + it: `GrB_DESC_RSC` = `GrB_REPLACE + GrB_STRUCTURE + GrB_COMP` + (`Include/GraphBLAS.h:666`) — three hints in a single opaque + handle, which is exactly what LAGraph's BFS passes to mean "write + only where I have not been". + +
+ +- [ ] You can explain zombies and pending tuples with their encodings, and give the complexity lazy mutation buys. + +
Answer + + Zombie = an entry marked for deletion in place by negating its + index: TOMS '19 §4.1 gives the transform as **i → (−i−2)**, and + the source describes `GB_ZOMBIE`/`GB_UNZOMBIE` as mutually + inverse (`GB_Matrix_content.h:370-373`). The transform preserves + ordering, so binary search still works on a vector containing + zombies — and a re-insert can *de-zombify* in place rather than + lengthening the pending list (`:375-383`). + + Pending tuple = an insert appended to an **unsorted** side list + of (i, j, value) with duplicates allowed and a combining operator + recorded (§4.1, and `GB_Matrix_content.h:361`). + + The payoff, §3.1.8: e incremental `setElement` calls cost + **O(e log e)** in SuiteSparse against **O(e²)** in MATLAB. At + e = 100,000 that is 1.66e6 versus 1.0e10 — 6,000×. And + `GrB_Matrix_build` is *also* O(e log e), so incremental costs the + same as batch. That equality is why a graph database can sit on + top of a matrix library. + +
+ +- [ ] You can say what non-blocking mode defers, what forces completion, and who decides. + +
Answer + + Deferred: pending inserts, zombie deletions, and — a third one + the paper does not stress — sortedness, via `jumbled` + (`GB_Matrix_content.h:498`). Completion is forced by `GrB_wait` + or implicitly by any operation that cannot tolerate the deferred + state; the source spells out the zombie case at `:386-389` + ("check the condition (A->nzombies > 0), and then delete all of + them if they appear, via GB_wait"). + + Who decides is the load-bearing part: **the library**, because + any read can trigger it. The application cannot pin the flush to + a transaction boundary. That is the gap FalkorDB's delta matrices + exist to fill, and question 2 asks you to weigh it against the + other candidate reasons. + +
+ - [ ] You can explain the iso-value optimization and identify which FalkorDB matrices are iso. + +
Answer + + An iso matrix keeps its full sparsity pattern but stores **one** + value, `A->x[0]`, for every present entry + (`GB_Matrix_content.h:513-518`, `:524`). The struct's own + rationale is that GraphBLAS deliberately has no structure-only + type — that "would result in a mathematical mismatch with all + other objects" — so iso is the sanctioned way to express an + unweighted graph. Corollary at `:520-521`: a *full* iso matrix is + O(1) memory regardless of dimension. + + FalkorDB: the boolean adjacency and delta matrices are iso — and + `delta_new.c:40-44` makes DM always `GrB_BOOL`. A relation matrix + carrying edge IDs as values is not. Question 3 asks for the mxm + bandwidth cost of losing it: 8 bytes per entry that iso would + have moved zero of, against a measured 15.8-19.1 GB/s + (`notes.md:9-14`). + +
+ - [ ] You wrote answers to all five questions in notes.md, including the 32-bit index memory computation. +
Answer + + For 10M nodes and 100M edges: all-64-bit CSR index is + 80.0 + 800.0 = **880 MB**; all-32-bit is 40.0 + 400.0 = + **440 MB**, exactly 2×. + + The finer point is that the pin has **three independent flags**, + not one — `p_is_32`, `j_is_32`, `i_is_32` + (`GB_Matrix_content.h:534-536`), plus `p_control`/`j_control`/ + `i_control` at `:464-466` for forcing them. `p` indexes *offsets* + (bounded by nnz) and `i` indexes *rows* (bounded by the + dimension), so a 5B-edge graph over 100M nodes wants 64-bit `p` + and 32-bit `i` — 480 MB here, still 1.83× better than all-64. + 32-bit `p` becomes illegal at about 2.1e9 edges. + +
+ ## References **Papers** -- Davis — "Algorithm 1000: SuiteSparse:GraphBLAS: Graph Algorithms - in the Language of Sparse Linear Algebra" (ACM TOMS 2019) — the - system paper; read §3 (object model) and the non-blocking-mode - discussion closely -- Davis — "Algorithm 1037: SuiteSparse:GraphBLAS: Parallel Graph - Algorithms in the Language of Sparse Linear Algebra" (ACM TOMS - 2023) — the v2 update: JIT, 32/64-bit indices, iso matrices + +- Davis, T. A. — "Algorithm 1000: SuiteSparse:GraphBLAS: Graph + Algorithms in the Language of Sparse Linear Algebra", ACM TOMS + 45(4), Article 44, December 2019, + [doi:10.1145/3322125](https://doi.org/10.1145/3322125). Read + §3.1.8 (non-blocking mode), §3.1.9 (accumulator and mask), §4.1 + (data structure) and §4.2.1 (matrix multiply). Cited here from + the author's accepted manuscript, which is titled "Algorithm 9xx" + and describes **version 2.3.3, single-threaded**. +- Davis, T. A. — "Parallel GraphBLAS with OpenMP", CSC '20 (SIAM + Workshop on Combinatorial Scientific Computing). §3 dates the + arrival of parallelism to version 3.0.1; §3.1 names the engine + chosen per algorithm; Table 2 is the 40-thread speedup table. + The citable source for anything about threads. Read in + [reading-openmp-vs-rayon.md](reading-openmp-vs-rayon.md). +- Davis, T. A. — "Algorithm 1037: SuiteSparse:GraphBLAS: Parallel + Graph Algorithms in the Language of Sparse Linear Algebra", ACM + TOMS 49(3), 2023. The v2 update: JIT, 32/64-bit indices, iso + matrices. **Not obtainable for this repo** — no claim in this + chapter is attributed to it; the same features are verified in + the pinned source instead. **Code** + - [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) - `Source/matrix/GB_matrix.h` — the object model in one header; - the internals walk is - [reading-suitesparse-internals.md](reading-suitesparse-internals.md) + at `1fd5475`, version **10.3.1** (`Include/GraphBLAS.h:290-292`). + `Source/builtin/include/GB_Matrix_content.h` is the object model + in one 657-line file — `:52` and `:76` (eight formats), + `:223-228` (the five arrays), `:241-274` (the hyper_hash Y), + `:361` (Pending), `:367-391` (zombies), `:450-457` (the two + switches), `:462-467` (sparsity and index-width controls), + `:497-498` (`is_csc`, `jumbled`), `:513-524` (iso), `:531-536` + (the three width flags). `Source/jitifyer/` is the JIT. + `Include/GraphBLAS.h:666` is `GrB_DESC_RSC`. The `Source/mxm/` + and `Source/convert/` walk is + [reading-suitesparse-internals.md](reading-suitesparse-internals.md). +- [FalkorDB](https://github.com/FalkorDB/FalkorDB) at `ccb449a9a` — + `src/graph/delta_matrix/delta_new.c:24-44` pins the sparsity and + disables the hyper_hash. Read in + [reading-falkordb-delta-matrix.md](reading-falkordb-delta-matrix.md). + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:41-42` — 80.4 MB → 1.59 MB and + 11,312 µs → 66 µs. Step 2 reproduces the first from array sizes + to within 1%. +- `topics/20-graphblas/notes.md:9-18` — the SpMV ladder, + 19.1 → 15.8 GB/s, and why it sits below the ~30 GB/s streaming + baseline. Question 3's bandwidth arithmetic runs on it. diff --git a/topics/20-graphblas/reading-falkordb-delta-matrix.md b/topics/20-graphblas/reading-falkordb-delta-matrix.md index dbda269..d42f4d0 100644 --- a/topics/20-graphblas/reading-falkordb-delta-matrix.md +++ b/topics/20-graphblas/reading-falkordb-delta-matrix.md @@ -9,6 +9,11 @@ load-bearing "why not the library's own deltas" question, the masked-multiply fold, and the compaction — then hands you the anchors into `src/graph/delta_matrix/` to verify each piece. +Every anchor is **FalkorDB at commit `ccb449a9a`**, the pin in +`resources/codebases.md`. Read the code, not the blog posts: three +of the claims in the previous version of this chapter were wrong at +this commit, and each correction is called out where it lands. + ## The problem in one sentence Deleting or inserting one edge in a packed sparse matrix means @@ -21,204 +26,881 @@ multiply-speed reads. ### Step 1 — the mutation problem: packed arrays hate point writes +> **In:** a settled sparse matrix and a single-edge write. +> **Out:** the cost of doing it eagerly, in this repo's own units. + A settled `GrB_Matrix` in sparse/hypersparse form is CSR-like: contiguous row-pointer and column-index arrays, packed with no -slack. That's exactly what makes reads and multiplies fast — and +slack. That is exactly what makes reads and multiplies fast — and exactly what makes one-edge mutation expensive: inserting edge (i,j) means shifting every index after it, deleting means the -same splice in reverse. For a 100M-edge relation matrix, one edge -insert done eagerly is an O(100M) memmove. The generic fix, seen -in topic 3 (LSM) and in SuiteSparse's own zombies/pending tuples: -don't restructure — *record the change somewhere cheap and merge -later*. +same splice in reverse. + +``` + inputs: a 100M-edge relation matrix, colidx u32 + memmove bandwidth ~30 GB/s (topic 0/13 streaming baseline) + + one eager insert at the midpoint: + bytes moved = 50e6 entries × 4 B = 200 MB + time = 200e6 / 30e9 = 6.7 ms + + a modest write burst of 10,000 edges, done eagerly: + 10,000 × 6.7 ms = 67 seconds + + the same 10,000 edges appended to a side matrix: + 10,000 × O(1) = microseconds + plus ONE fold of 100M entries at the end (Step 6) +``` + +The generic fix, seen in topic 3 (LSM) and in SuiteSparse's own +zombies/pending tuples: do not restructure — *record the change +somewhere cheap and merge later*. + +Why it matters: 67 seconds versus one fold is the entire +justification for the rest of this chapter, and the ratio is set by +the flush threshold you pick in Step 6. ### Step 2 — the trio: settled matrix plus two delta matrices -FalkorDB's delta matrix keeps three GrB matrices (plus the same -three transposed): +> **In:** the "record it somewhere cheap" idea. +> **Out:** the actual struct, its six matrices, and the read +> identity. + +```c +// delta_matrix.h — the struct, 108-115 + 108 struct _Delta_Matrix { + 109 bool locked; + 110 GrB_Matrix matrix; // Underlying GrB_Matrix + 111 GrB_Matrix delta_plus; // Pending additions + 112 GrB_Matrix delta_minus; // Pending deletions + 113 Delta_Matrix transposed; // Transposed matrix + 114 pthread_mutex_t mutex; // Lock + 115 }; +``` +Note the recursion at `:113`: `transposed` is itself a +`Delta_Matrix`, so it carries its own trio. Six GrB matrices per +logical relation, reached through six macros: + +```c +// delta_matrix.h — the accessors, 17-24 + 17 #define DELTA_MATRIX_M(C) ((C)->matrix) + 18 #define DELTA_MATRIX_DELTA_PLUS(C) ((C)->delta_plus) + 19 #define DELTA_MATRIX_DELTA_MINUS(C) ((C)->delta_minus) + 20 #define DELTA_MATRIX_TM(C) ((C)->transposed->matrix) + 21 #define DELTA_MATRIX_TDELTA_PLUS(C) ((C)->transposed->delta_plus) + 22 #define DELTA_MATRIX_TDELTA_MINUS(C) ((C)->transposed->delta_minus) + 23 + 24 #define DELTA_MATRIX_MAINTAIN_TRANSPOSE(C) ((C)->transposed != NULL) ``` - Delta_Matrix = M (settled GrB_Matrix, hypersparse CSR) - + delta-plus DP (pending additions) - + delta-minus DM (pending deletions) - + the same trio TRANSPOSED (delta_matrix.h:110-113) + +A naming trap worth fixing now: the header's comment diagrams call +the settled matrix **`A`**, but the struct field is `matrix` and +the macro is `DELTA_MATRIX_M`. Throughout this chapter, **M** is +the settled matrix and **A** is the *logical* matrix a reader +sees. + +M is big and packed; DP and DM are tiny (bounded by the write batch +since the last sync — Step 6 gives the bound as 10,000), so +mutating them is cheap. The logical matrix the rest of the engine +sees is defined algebraically: **A ≡ (M ∪ DP) \ DM** — everything +settled or pending-added, minus everything pending-deleted. Same +read algebra as an LSM point-read (memtable ∪ sstables minus +tombstones — DM's entries are exactly **tombstones**, deletion +markers that suppress a still-physically-present entry). + +The three matrices are not configured alike, and `delta_new.c` +says so: + +```c +// delta_new.c — the sparsity pins, 21-44 (elided) + 22 // m, can be either hypersparse or sparse + 24 GrB_OK(GrB_Matrix_new (&A->matrix, type, nrows, ncols)); + 25 GrB_OK (GrB_set ( + 26 A->matrix, GxB_SPARSE | GxB_HYPERSPARSE, GxB_SPARSITY_CONTROL)); + ... + 29 // delta-plus, always hypersparse + 31 GrB_OK (GrB_Matrix_new (&A->delta_plus, type, nrows, ncols)); + 32 GrB_OK (GrB_set (A->delta_plus, GxB_HYPERSPARSE, GxB_SPARSITY_CONTROL)); + 33 GrB_OK (GrB_set (A->delta_plus, (int32_t) false, GxB_HYPER_HASH)); + 35 GrB_OK (GxB_set (A->delta_plus, GxB_HYPER_SWITCH, GxB_ALWAYS_HYPER)); + ... + 38 // delta-minus, always hypersparse + 40 GrB_OK (GrB_Matrix_new (&A->delta_minus, GrB_BOOL, nrows, ncols)); + 41 GrB_OK (GrB_set (A->delta_minus, GxB_HYPERSPARSE, GxB_SPARSITY_CONTROL)); + 42 GrB_OK (GrB_set (A->delta_minus, (int32_t) false, GxB_HYPER_HASH)); + 44 GrB_OK (GxB_set (A->delta_minus, GxB_HYPER_SWITCH, GxB_ALWAYS_HYPER)); ``` -M is big and packed; DP and DM are tiny (bounded by the write -batch since the last sync), so mutating them is cheap. The logical -matrix the rest of the engine sees is defined algebraically: -**A ≡ (M ∪ DP) \ DM** — everything settled or pending-added, -minus everything pending-deleted. Same read algebra as an LSM -point-read (memtable ∪ sstables minus tombstones — DM's entries -are exactly **tombstones**, deletion markers that suppress a -still-physically-present entry). +Four decisions in twenty lines, and each is answerable from +[reading-suitesparse-internals.md](reading-suitesparse-internals.md): + +1. **M allows sparse or hypersparse, never bitmap or full.** In + `GB_conform.c:150`'s switch that is case (3) at `:175-184`, the + `GxB_HYPERSPARSE + GxB_SPARSE` arm — where the bitmap test is + not merely unlikely, it *never executes*. Given Step 2 of the + internals chapter (a graph matrix is 13,700× below the 0.40 + bitmap threshold), this pin costs nothing and removes a branch. +2. **DP and DM are pinned hypersparse and forced there** by + `GxB_ALWAYS_HYPER`, which lands in `GB_conform.c:157-160` — + `GB_convert_any_to_hyper` unconditionally, no test at all. A + matrix holding 10,000 entries over a 10M id space *must* be + hypersparse; the 50× index saving of `notes.md:41` is the + reason. +3. **The hyper_hash is disabled** (`:33`, `:42`). That is the + `A->Y` structure of `GB_Matrix_content.h:241-274`; without it, + finding row j in `h[]` is a binary search costing `lg k`. With + k ≤ 10,000 that is 14 comparisons — cheaper than building and + maintaining a hash table that gets cleared at every flush. +4. **DM is always `GrB_BOOL`** (`:40`), whatever the matrix type + is. A tombstone carries no value, so DM is iso by construction + (`GB_Matrix_content.h:513-524`) and stores one byte total for + its values. + +And the type restriction, at `delta_new.c:64-65`: "supported +types: boolean and uint64", `ASSERT (type == GrB_BOOL || type == +GrB_UINT64)`. + +Why it matters: the trio is not three copies of the same thing. +Each matrix is configured for its own access pattern, and every +one of those four settings is a decision you can now trace into +SuiteSparse's own source. ### Step 3 — the invariants, and the read/write paths -The header comment (delta_matrix.h:34-108) walks every operation -through a worked example — it IS the design doc. Distilled: +> **In:** the trio and the identity A ≡ (M ∪ DP) \ DM. +> **Out:** the eight states the header enumerates, the actual read +> order, and the GrB-call cost of one mutation. + +The header comment at `delta_matrix.h:26-106` walks every state +through a worked 3×3 example — it IS the design doc. (The previous +version of this chapter cited `:34-108`; the block runs `:26-106`, +and the struct that follows is `:108-115`.) It enumerates **four +legal states and four impossible ones**: + +| state | lines | A | DP | DM | +|---|---|---|---|---| +| empty | `:32-38` | · | · | · | +| flushed, no pending changes | `:41-47` | 1 | · | · | +| single entry added | `:50-56` | · | 1 | · | +| single entry deleted | `:59-65` | 1 | · | 1 | +| **impossible** — "existing entry deleted and then added back" | `:67-74` | 1 | 1 | 1 | +| **impossible** — "marked none existing entry for deletion" | `:77-84` | · | · | 1 | +| **impossible** — "adding to an already existing entry" | `:87-95` | 1 | 1 | · | +| **impossible** — "deletion of pending entry should have cleared it DP[0,0]" | `:98-105` | · | 1 | 1 | + +Read the four impossible rows as the invariants they encode: ``` logical A ≡ (M ∪ DP) \ DM - invariants: DP ∩ M = ∅ (additions are NEW entries) - DM ⊆ M (you can only pending-delete settled entries) - delete of a DP entry clears DP directly (:99) — never - passes through DM - the transposed twin maintains the same trio, updated in lockstep -``` - -The invariants keep every entry in exactly one state, so reads -never need conflict resolution. The read and write paths, -distilled: - -```rust -// logical A ≡ (M ∪ DP) \ DM — an LSM point-read over matrices -fn contains(&self, i: u64, j: u64) -> bool { - if self.dm.contains(i, j) { return false; } // tombstone wins - self.dp.contains(i, j) || self.m.contains(i, j) -} - -fn set(&mut self, i: u64, j: u64) { - if self.m.contains(i, j) { - self.dm.remove(i, j); // resurrect a pending-deleted entry - } else { - self.dp.set(i, j); // NEW entry → DP (keeps DP ∩ M = ∅) - } - self.transposed.set(j, i); // the twin trio, in lockstep -} -``` - -What it costs: every read is a 3-way check (read amplification), -every write touches two trios (the twin doubles write work — -question 2). What it buys: O(1)-ish writes against a matrix that -stays multiply-ready. + + DP ∩ M = ∅ additions are NEW entries (:87-95 forbids the overlap) + DM ⊆ M only settled entries can be (:77-84 forbids DM without M) + pending-deleted + DP ∩ DM = ∅ delete of a DP entry clears DP (:98-105) + directly — never passes through DM + M ∩ DP ∩ DM = ∅ re-adding a deleted entry (:67-74) + resurrects it out of DM +``` + +Every entry is in exactly one state, so reads never need conflict +resolution. Now the actual read path — and **the order is not what +the LSM analogy predicts**: + +```c +// delta_isStored.c — the read path, 25-40. DP first, then DM, then M. + 25 // if dp[i,j] exists return it + 26 info = GxB_Matrix_isStoredElement (DP, i, j) ; + 27 if(info == GrB_SUCCESS) { + 28 return info ; + 29 } + 30 + 31 // if dm[i,j] exists, return no value + 32 info = GxB_Matrix_isStoredElement (DM, i, j) ; + 33 if (info == GrB_SUCCESS) { + 34 // entry marked for deletion + 35 return GrB_NO_VALUE ; + 36 } + 37 + 38 // entry isn't marked for deletion, see if it exists in 'm' + 39 info = GxB_Matrix_isStoredElement (M, i, j) ; + 40 return info ; +``` + +**DP is probed first, not DM.** That is legal precisely because of +the `DP ∩ DM = ∅` invariant at `:98-105` — a tombstone can never +shadow a pending addition, so the order does not change the answer, +and probing the newest layer first is the cheapest ordering. An LSM +would have to check tombstones first because its layers *do* +overlap. Note also `:28`: the early return means an entry found in +DP costs **one** probe, not three. + +The write path, both directions: + +```c +// delta_set_element_bool.c — insert, 27-40 + 27 if (DELTA_MATRIX_MAINTAIN_TRANSPOSE (C)) { + 28 GrB_OK (Delta_Matrix_setElement_BOOL (C->transposed, j, i)) ; + 29 } + 30 + 31 GrB_OK (info = GxB_Matrix_isStoredElement (m, i, j)) ; + 32 already_allocated = (info == GrB_SUCCESS); + 33 + 34 if (already_allocated) { + 35 // unset delta-minus + 36 GrB_OK (GrB_Matrix_removeElement (dm, i, j)) ; + 37 } else { + 38 // update entry to dp[i, j] + 39 GrB_OK (GrB_Matrix_setElement_BOOL (dp, true, i, j)) ; + 40 } +``` + +```c +// delta_remove_element.c — delete, 28-44 + 28 if (DELTA_MATRIX_MAINTAIN_TRANSPOSE (C)) { + 29 GrB_OK (Delta_Matrix_removeElement (C->transposed, j, i)) ; + 30 } + ... + 36 info = GxB_Matrix_isStoredElement (m, i, j) ; + 37 in_m = (info == GrB_SUCCESS) ; + 38 + 39 if (in_m) { + 40 // mark deletion in delta minus + 41 GrB_OK (GrB_Matrix_setElement_BOOL (dm, (bool) true, i, j)) ; + 42 } else { + 43 GrB_OK (GrB_Matrix_removeElement (dp, i, j)) ; + 44 } +``` + +Both are the same shape: recurse into the transposed twin **first** +(`:27-29` and `:28-30`), probe M once, then branch. Cost it: + +``` + inputs: one logical edge insert on a delta matrix WITH a transposed twin + + per orientation: + 1 × GxB_Matrix_isStoredElement (M) probe + 1 × GrB_Matrix_removeElement(DM) OR + GrB_Matrix_setElement_BOOL(DP) mutation + = 2 GrB calls + + two orientations (the recursion at :28) = 4 GrB calls + + read path, worst case (entry lives in M): + 3 × GxB_Matrix_isStoredElement = 3 GrB calls + read path, best case (entry lives in DP): + 1 × GxB_Matrix_isStoredElement = 1 GrB call (:28) + + so: 4 calls to write, 1-3 to read, against ONE array splice + avoided (Step 1: 6.7 ms on a 100M-edge matrix). +``` + +Note also which branch fires. Inserting a *brand-new* edge writes +to DP; re-inserting a *deleted* edge (`already_allocated` true) +removes from DM instead, which is the `:67-74` impossible state +being actively prevented rather than merely asserted. + +Why it matters: the write path costs four library calls and the +read path costs one to three, and both numbers come from counting +lines, not from a blog post. ### Step 4 — why not SuiteSparse's own pending tuples? (the load-bearing question) +> **In:** the trio, and SuiteSparse's zombies + pending tuples. +> **Out:** three candidate reasons, each grounded in a specific +> line of one library or the other. + SuiteSparse already defers mutations (zombies + pending tuples, -[reading-davis-toms19.md](reading-davis-toms19.md) step 4). The -delta layer exists because: - -1. **flush control**: ANY GrB read op can force internal wait; - FalkorDB needs reads that DON'T flush (readers under a write - lock, MVCC-ish semantics) — DP/DM are ordinary matrices the - library never touches implicitly. -2. **the transposed twin**: SuiteSparse maintains ONE matrix; - FalkorDB needs M and Mᵀ synced under the same deltas - (delta_matrix.h:20-22) — pull traversals are always available - (`<-[]-` patterns). -3. **bounded sync cost**: wait folds a SMALL DP/DM (bounded by - write-batch size) — library pending tuples can degrade into a - full rebuild inside an unrelated query. +[reading-davis-toms19.md](reading-davis-toms19.md) Step 4), and its +own bound is excellent: e incremental inserts in **O(e log e)** +against MATLAB's O(e²) (TOMS '19 §3.1.8). So why rebuild it? + +1. **Flush control.** Any GrB read op can force an internal wait — + the source says so for zombies at + `GB_Matrix_content.h:386-389`. FalkorDB needs reads that do + *not* flush. DP and DM are ordinary matrices the library never + touches implicitly, and the flush decision lives in FalkorDB's + own code at `delta_wait.c:89` and `:97` (Step 6). +2. **The transposed twin.** SuiteSparse maintains one matrix; + FalkorDB needs M and Mᵀ synced under the same deltas. The + recursion is right there in the struct (`delta_matrix.h:113`) + and in every mutation (`delta_set_element_bool.c:27-29`), so + pull traversals (`<-[]-` patterns) are always available without + a transpose. +3. **Bounded sync cost.** A wait folds a *small* DP/DM — bounded by + `DELTA_MAX_PENDING_CHANGES_DEFAULT` = **10,000** + (`src/configuration/config.h:19`). Library pending tuples have + no such bound and can degrade into a full rebuild inside an + unrelated query. + +Question 2 asks you to decide which dominates. A hint from the +code: reason 2 is the only one that is *impossible* to get from +the library at any threshold, because SuiteSparse has no concept of +a matrix pair kept in lockstep. The general lesson: a lower layer's deferred-work mechanism is -only reusable if you control *when* it fires and *what -invariants* it maintains — otherwise you rebuild it one level up, -which is exactly what happened here. +only reusable if you control *when* it fires and *what invariants* +it maintains — otherwise you rebuild it one level up, which is +exactly what happened here. + +Why it matters: this is the transferable design judgement in the +whole topic, and it recurs every time you sit a system on a library +that already has a cache, a log, or a scheduler. ### Step 5 — delta_mxm: algebra instead of a flush +> **In:** a multiply where one operand carries pending state. +> **Out:** the four GrB calls that avoid a flush, the exact +> masking the code performs, and where it differs from its own +> comment. + The expensive operation on a delta matrix is a multiply — must the -deltas be folded into M first? delta_mxm.c:44-86 says no: fold the -pending state into the *algebra* of one multiply. To compute -C = A*B where B carries pending state: +deltas be folded into M first? `delta_mxm.c` (121 lines) says no: +fold the pending state into the *algebra* of one multiply. Its own +statement of intent, and its preconditions: + +```c +// delta_mxm.c — the contract, 40-50 + 40 // where A is fully synced! + ... + 43 // this operation performs: A * B by computing: + 44 // (A * (M + 'delta-plus')) + 45 + 46 // validate A doesn't contains entries in either delta-plus or delta-minus + 47 ASSERT(Delta_Matrix_Synced(A)); + 48 + 49 // validate C doesn't contains entries in either delta-plus or delta-minus + 50 ASSERT(Delta_Matrix_Synced(C)); +``` + +**Only B carries deltas.** A and C must be synced, asserted at +`:47` and `:50`. That narrows the problem enormously: the pending +state appears on exactly one side. + +The four calls, in order: + +```c +// delta_mxm.c — what actually runs, 71-108 (elided) + 71 if (dm_nvals > 0) { + 72 // compute A * 'delta-minus' + 74 GrB_OK (GrB_mxm (mask, NULL, NULL, GxB_ANY_PAIR_BOOL, _A, dm, NULL)) ; + 78 } + 80 if (dp_nvals > 0) { + 81 // compute A * 'delta-plus' + 86 GrB_OK (GrB_mxm (accum, NULL, NULL, semiring, _A, dp, NULL)) ; + 90 } + ... + 96 if (deletions) { + 97 desc = GrB_DESC_RSC ; + ... + 103 // compute (A * B) + 104 GrB_OK (GrB_mxm (_C, mask, NULL, semiring, _A, _B, desc)) ; + 105 + 106 if (additions) { + 107 GrB_OK (GrB_eWiseAdd (_C, NULL, NULL, semiring, _C, accum, NULL)) ; + 108 } +``` + +Line by line: `:74` builds `mask = A·DM` over `GxB_ANY_PAIR_BOOL` +(pattern only, no values, and ANY lets it stop at the first hit); +`:86` builds `accum = A·DP` over the real semiring; `:104` +computes the main product with `GrB_DESC_RSC` — which is +`REPLACE + STRUCTURE + COMP` (`Include/GraphBLAS.h:666`), so the +mask is *complemented*: write only where `A·DM` has **no** entry; +`:107` adds the additions in. + +**Two things here are not what the comment at `:44` says**, and the +code is authoritative: + +- The comment masks by `'delta-minus'`; the code masks by + `A · DM` (`:74`), which is a different, coarser matrix. +- The comment applies the mask to `A*(M + DP)` — additions + included. The code applies it only to `A*M` at `:104`, then adds + `accum` **unmasked** at `:107`. So a cell killed by the mask can + be revived by an addition, but only if a DP edge happens to + produce it. + +The over-masking is real and constructible. Here is the +counterexample question 3 asks for, entirely readable off `:74`, +`:97` and `:104`: + +``` + A = one row, two live edges: A(0,0) = 1, A(0,1) = 1 + B's settled M: M(0,5) = 1, M(1,5) = 1 + delete edge (0,5) from B ⇒ DM(0,5) = 1, M unchanged + + truth: logical B = (M ∪ DP) \ DM has only B(1,5) + A·B ⇒ C(0,5) present, via A(0,1)·B(1,5) ← LIVE + + code: mask = A·DM ⇒ mask(0,5) present, via A(0,0)·DM(0,5) + :104 = (A·M) ⇒ C(0,5) SUPPRESSED + :107 accum = A·DP = empty ⇒ nothing restores it + + result: delta_mxm drops a live entry. The mask is structural — + it kills the whole output cell if ANY contributing path used a + deleted edge, even when other paths are alive. +``` + +How correctness is restored at the call sites is **not verified +here** — question 3 sends you to `graph/graph.c` for it. Do not +assume it is fixed inside `delta_mxm.c`; it is not. + +Price the overhead, so you know what the algebra costs: ``` - accum = A * DP (:86 — the additions' contribution) - mask = A * DM (ANY_PAIR bool, :74 — rows poisoned by deletions) - C = (A * M) + accum, masked by !mask — "(A*(M+DP))" + inputs: main product A·B on an RMAT-scale-16-sized relation + (topics/24-graph-algorithms/notes.md:5) — mean degree 27.8 + DP and DM each at the flush threshold, 10,000 entries + main-product flops ≈ 1.28e8 (extrapolated from notes.md:22-26) + + A·DM flops = Σ over DM's 10,000 entries of nnz(A(:,k)) + ≈ 10,000 × 27.8 = 278,000 + A·DP flops ≈ same = 278,000 + total extra = 556,000 + + overhead = 5.56e5 / 1.28e8 = 0.43% + + versus the alternative — flushing B first — which is + Step 6's O(nnz(M)) fold on the critical path of the query. ``` -Two extra *small* multiplies (DP and DM are tiny) instead of one -big compaction — the LSM read-amplification-vs-compaction trade, -chosen per multiply. Note the mask is *coarse*: A*DM marks any -output touched by a deleted edge, potentially over-masking; check -how the caller compensates (question 3). +Two extra *small* multiplies instead of one big compaction: the LSM +read-amplification-versus-compaction trade, chosen per multiply, at +under half a percent. + +Why it matters: 0.43% is why this design is worth its correctness +hazard — and the hazard is the price, which you should be able to +construct on demand. ### Step 6 — wait: the two-sided compaction +> **In:** DP and DM, grown since the last fold. +> **Out:** the two GrB calls that fold them, the threshold that +> triggers it, and the amortized cost per mutation. + `Delta_Matrix_wait` is the compaction that folds the deltas into M -and resets the trio. Deletions first: `GrB_transpose(m, dm, NULL, -m, GrB_DESC_RSCT0)` — a transpose of m into itself, masked by the -COMPLEMENT of dm, with T0 transposing the transpose away: one -library call that copies M minus its tombstoned entries. Then -additions (assign/eWiseAdd DP into M), then clear both deltas -(delta_wait.c:13-46). - -The policy decision — sync now vs stay lazy — consults nvals -thresholds (delta_will_wait.c: "would GrB_wait do work?"). That's -compaction triggering by size, topic 3 again: small thresholds = -low read amplification but frequent O(nnz(M)) folds; large = -cheap writes but every read/multiply pays the 3-way tax longer. +and resets the trio. Deletions first: + +```c +// delta_wait.c — sync_deletions, 13-33 (elided) + 13 static GrB_Info Delta_Matrix_sync_deletions + ... + 25 if (nvals > 0) { //shortcut if no vals + ... + 29 GrB_RETURN_IF_FAIL (GrB_transpose (m, dm, NULL, m, GrB_DESC_RSCT0)) ; + 30 } + 31 + 32 // clear delta minus + 33 return GrB_Matrix_clear (dm) ; +``` + +Unpack `:29` carefully, because it is the cleverest line in the +directory. `GrB_transpose(C, Mask, accum, A, desc)` computes +`C = Aᵀ`. Here C = m, Mask = dm, A = m, and the descriptor is +`GrB_DESC_RSCT0` — `REPLACE + STRUCTURE + COMP` plus `GrB_TRAN` on +input 0 (`Include/GraphBLAS.h:668`). The `T0` transposes the input, +so `Aᵀ` becomes `(mᵀ)ᵀ = m`; the `C` complements the mask; the `S` +uses only its pattern; the `R` replaces. Net effect: **copy m into +itself, keeping only the entries dm does not mark.** One library +call performs the whole tombstone sweep, and the transpose flags +cancel each other out. + +Then additions: + +```c +// delta_wait.c — sync_additions, 36-56 (elided) + 36 static GrB_Info Delta_Matrix_sync_additions + ... + 48 if (nvals > 0) { //shortcut if no vals + ... + 51 GrB_RETURN_IF_FAIL (GrB_Matrix_assign (m, dp, NULL, dp, GrB_ALL, 0, + 52 GrB_ALL, 0, GrB_DESC_S)) ; + 53 } + 54 + 55 // clear delta plus + 56 return GrB_Matrix_clear (dp) ; +``` + +(The previous version of this chapter cited `delta_wait.c:36-46+` +for this function; it runs to `:57`.) + +Now the trigger, and **the correction that matters most in this +chapter**. The policy is *not* in `delta_will_wait.c`: + +```c +// delta_wait.c — the thresholds, 89-99 + 89 if (dm_nvals >= delta_max_pending_changes) { + 90 GrB_RETURN_IF_FAIL (Delta_Matrix_sync_deletions (C)) ; + 91 } + ... + 97 if (dp_nvals >= delta_max_pending_changes) { + 98 GrB_RETURN_IF_FAIL (Delta_Matrix_sync_additions (C)) ; + 99 } +``` + +Two independent thresholds against the same constant, deletions +tested first. `delta_max_pending_changes` comes from +`Config_DELTA_MAX_PENDING_CHANGES` (`delta_wait.c:126-128`), +defaulting to **10,000** (`src/configuration/config.h:19`). And +`force_sync` at `:71-73` bypasses both. + +What `delta_will_wait.c` actually does is a different question +entirely — it asks *SuiteSparse* whether the library has its own +pending work: + +```c +// delta_will_wait.c — asking the LIBRARY, not the delta layer, 34-44 + 34 // check if M contains pending changes + 35 GrB_OK (GrB_Matrix_get_INT32 (M, &p, GxB_WILL_WAIT)) ; + 36 res = res || p == 1 ; + ... + 39 GrB_OK (GrB_Matrix_get_INT32 (DP, &p, GxB_WILL_WAIT)) ; + ... + 43 GrB_OK (GrB_Matrix_get_INT32 (DM, &p, GxB_WILL_WAIT)) ; +``` + +`GxB_WILL_WAIT` is SuiteSparse's own zombies-and-pending-tuples +probe from +[reading-davis-toms19.md](reading-davis-toms19.md) Step 4. It is +used as an **assertion**, at `delta_wait.c:108-110`, to check that +after the three `GrB_wait(…, GrB_MATERIALIZE)` calls at `:103-105` +nothing is left deferred at either level. Two deferral systems +stacked, and this is the line that proves they are both drained. + +One more ordering detail: `Delta_Matrix_wait` recurses into the +transposed twin **before** doing its own work (`:122-124`), the +same first-the-twin discipline as the mutation paths. + +Now the compaction trade, made arithmetic: + +``` + inputs: M holds N = 100e6 edges; threshold T = 10,000 (config.h:19) + + a fold touches O(N) entries (the GrB_transpose at :29 rewrites m) + amortized fold cost per mutation = N / T = 100e6 / 10,000 = 10,000 + entries rewritten per edge + + compare eager (Step 1): 50e6 entries moved per edge + improvement = 5,000× + + halve the threshold to T = 5,000: + amortized fold cost doubles = 20,000 entries/edge + but DP/DM stay half as big, so Step 5's mxm overhead halves + (0.43% → 0.21%) and a DP-miss read path shortens + + raise it to T = 100,000: + amortized fold cost = 1,000 entries/edge + but mxm overhead rises to ~4.3% and every read carries a + 10× bigger DP/DM to probe +``` + +That is compaction triggering by size, topic 3 again: small +thresholds mean low read amplification but frequent O(nnz(M)) +folds; large thresholds mean cheap writes but every read and +multiply pays the three-way tax longer. + +Why it matters: one configuration constant sets the position on +that curve, and now you can compute both ends of it. ## Where each step lives in the code | anchor | step | what it is | |---|---|---| -| delta_matrix.h:110-116 | 2 | the struct: M + delta_plus + delta_minus + transposed twin | -| delta_matrix.h:17-22 | 2 | accessor macros incl. the T* transposed trio | -| delta_matrix.h:34-108 | 3 | the state-transition comment table (A/DP/DM invariants per op) — the spec | -| delta_get_set.c / delta_isStored.c | 3 | the 3-way read path (check DM, DP, M) | -| delta_mxm.c:44-99 | 5 | `(A*(M+DP))` — multiply WITHOUT forcing a sync | -| delta_wait.c:13-33 | 6 | sync_deletions: `GrB_transpose(m, dm, NULL, m, GrB_DESC_RSCT0)` — transpose-as-masked-copy | -| delta_wait.c:36-46+ | 6 | sync_additions: fold DP into M, clear DP | -| delta_will_wait.c | 6 | "would GrB_wait do work?" — the flush-decision probe | - -Navigation advice: start with the state-transition comment table -in `delta_matrix.h:34-108` (it IS the design doc), then -`delta_wait.c`, `delta_mxm.c`, `delta_get_set.c`, -`delta_will_wait.c` — read each against topics 3 (LSM), 6 (buffer -mgmt), and this topic's zombies/pending-tuples machinery, asking -at each step "why not just let SuiteSparse's own deltas do this?" +| `delta_matrix.h:17-24` | 2 | the six accessor macros plus `MAINTAIN_TRANSPOSE` | +| `delta_matrix.h:26-106` | 3 | the state table: 4 legal states, 4 impossible ones — the spec | +| `delta_matrix.h:108-115` | 2 | the struct; `:113` is the recursive transposed twin | +| `delta_new.c:21-44` | 2 | sparsity pins: M sparse+hyper, DP/DM always-hyper, no hyper_hash | +| `delta_new.c:64-65` | 2 | `GrB_BOOL` or `GrB_UINT64` only | +| `delta_isStored.c:25-40` | 3 | the read path — **DP, then DM, then M** | +| `delta_set_element_bool.c:27-40` | 3 | insert: twin first, probe M, then DM-remove or DP-set | +| `delta_remove_element.c:28-44` | 3 | delete: twin first, probe M, then DM-set or DP-remove | +| `delta_remove_element.c:50-81` | 3 | the bulk form, via `eWiseMult` + `assign` | +| `delta_mxm.c:40-50` | 5 | the contract: **only B may carry deltas** | +| `delta_mxm.c:74`, `:86` | 5 | `mask = A·DM` (ANY_PAIR bool), `accum = A·DP` | +| `delta_mxm.c:97`, `:104`, `:107` | 5 | `GrB_DESC_RSC`, the masked product, the **unmasked** add | +| `delta_wait.c:13-33` | 6 | sync_deletions: `GrB_transpose(m, dm, NULL, m, GrB_DESC_RSCT0)` | +| `delta_wait.c:36-56` | 6 | sync_additions: assign DP into M, clear DP | +| `delta_wait.c:89`, `:97` | 6 | **the flush thresholds live here** | +| `delta_wait.c:103-110` | 6 | `GrB_wait(…, GrB_MATERIALIZE)` ×3, then the willWait assertion | +| `delta_wait.c:122-128` | 6 | twin first; read `Config_DELTA_MAX_PENDING_CHANGES` | +| `delta_will_wait.c:34-44` | 6 | asks SuiteSparse `GxB_WILL_WAIT` — **not** the delta threshold | +| `src/configuration/config.h:19` | 6 | `DELTA_MAX_PENDING_CHANGES_DEFAULT 10000` | + +Navigation advice: start with the state table in +`delta_matrix.h:26-106` (it IS the design doc) and check it against +`delta_set_element_bool.c` and `delta_remove_element.c` — that is +question 1, and it takes ten minutes. Then `delta_wait.c` top to +bottom (218 lines), then `delta_mxm.c` (121 lines), then +`delta_isStored.c`. Read each against topics 3 (LSM), 6 (buffer +management) and this topic's zombies/pending-tuples machinery, +asking at each step "why not just let SuiteSparse's own deltas do +this?" ### What transfers to M20 -M20 rebuilds this over OUR kernels: the trio + transposed twin, -the read algebra in get/extract, the mxm fold, threshold-driven -wait. The reference is the spec; the interesting freedom is -choosing DP/DM's format (hash-of-pairs? small COO? bitmap?) now -that we own the representation — measure against `GrB_Matrix` DP -via the LDBC update workloads. +M20 rebuilds this over OUR kernels: the trio plus the transposed +twin, the read algebra in get/extract, the mxm fold, and +threshold-driven wait. The reference is the spec; the interesting +freedom is choosing DP/DM's representation (hash of pairs? small +COO? bitmap?) now that we own it. Three numbers to design against: +the write path is 4 calls (Step 3), the flush threshold is 10,000 +entries (Step 6), and the mxm overhead at that threshold is 0.43% +(Step 5). ## Questions for notes.md -1. Verify the invariants against delta_set_element_bool.c and - delta_remove_element.c: enumerate the 4 cases (entry in M, in - DP, in DM, absent) × (set, remove) — which transitions does the - header table at :34-108 show, and are any missing? +1. Verify the invariants against `delta_set_element_bool.c` and + `delta_remove_element.c`: enumerate the 4 cases (entry in M, in + DP, in DM, absent) × (set, remove), and match each to one of the + eight rows in the header table at `delta_matrix.h:26-106`. Which + transitions does the table show, and are any reachable + transitions missing from it? 2. The transposed twin doubles write work on every mutation. Cost - it: per set_element, how many GrB calls hit each trio — and - what would break if the transpose were rebuilt lazily at wait - instead (pull traversals see stale AT between waits)? -3. delta_mxm's mask A*DM over-masks (kills a full output entry if - ANY contributing edge is deleted — but other, live edges might - also produce it). Find how correctness is restored (recompute - masked region against (M+DP)\DM? restrict when delta_mxm is - used at all — check callers in graph/graph.c) — and write the - counterexample matrix that exposes it. -4. delta_will_wait / the sync thresholds: what nvals bounds - trigger a flush, and how do they map to LSM L0 file-count - triggers (write-visible latency vs read amplification)? + it: Step 3 counts 4 GrB calls per logical edge — say which two + belong to the twin, and work out what would break if the + transpose were rebuilt lazily at wait instead (pull traversals + see a stale Mᵀ between waits — for how long, given the 10,000 + threshold?). +3. `delta_mxm`'s mask over-masks. Step 5 gives a counterexample + readable off `:74`, `:97` and `:104`; reproduce it, then find + how correctness is restored — recompute the masked region + against (M ∪ DP) \ DM? restrict when `delta_mxm` is called at + all? Check the callers in `graph/graph.c`. This chapter did + **not** verify the answer. +4. The sync thresholds: `delta_wait.c:89` and `:97` both compare + against `DELTA_MAX_PENDING_CHANGES_DEFAULT` = 10,000 + (`config.h:19`). Map that onto LSM L0 file-count triggers + (topic 3) — write-visible latency versus read amplification — + and redo Step 6's amortized-cost arithmetic for the threshold + your own M20 workload would want. 5. For M20: pick DP/DM's representation in Rust. COO - `Vec<(u32,u32)>` + sort at wait (LSM-flavored) vs HashMap - (point-read-flavored) — which do the LDBC interactive - update+read mixes prefer? Predict, then bench both under - gb_bench's update workload. + `Vec<(u32,u32)>` + sort at wait (LSM-flavoured) versus HashMap + (point-read-flavoured) — which do the LDBC interactive + update+read mixes prefer? Predict from Step 3's call counts + (4 writes, 1-3 reads per operation) first, then bench both under + `gb_bench`'s update workload. ## Done when -- [ ] You can state the trio and write the read identity `(M ∪ DP) ∖ DM` from memory. +Answer each before unfolding it. + +- [ ] You can state the trio and write the read identity `(M ∪ DP) ∖ DM` from memory, and name all four invariants. + +
Answer + + M (`matrix`), DP (`delta_plus`), DM (`delta_minus`), plus the + recursive `transposed` twin carrying its own three + (`delta_matrix.h:108-115`). Logical A ≡ (M ∪ DP) \ DM. + + The four invariants are the four **impossible** states in the + header table: `DP ∩ M = ∅` (`:87-95`, "adding to an already + existing entry"), `DM ⊆ M` (`:77-84`, "marked none existing entry + for deletion"), `DP ∩ DM = ∅` (`:98-105`, "deletion of pending + entry should have cleared it"), and `M ∩ DP ∩ DM = ∅` (`:67-74`, + "existing entry deleted and then added back"). + + Careful with names: the header's diagrams call the settled matrix + `A`, but the struct field is `matrix` and the macro is + `DELTA_MATRIX_M`. + +
+ +- [ ] You can give the read order and explain why it is legal. + +
Answer + + **DP → DM → M** (`delta_isStored.c:26`, `:32`, `:39`) — not DM + first, which is what the LSM analogy would suggest. + + It is legal because of the `DP ∩ DM = ∅` invariant + (`delta_matrix.h:98-105`): a tombstone can never shadow a pending + addition, so the order cannot change the answer. Probing the + newest, smallest layer first is then simply the cheapest ordering, + and the early return at `:28` makes a DP hit cost one probe + instead of three. An LSM must check tombstones first precisely + because its layers do overlap. + +
+ +- [ ] You can cost one mutation and one read in GrB calls. + +
Answer + + Write: recurse into the twin first + (`delta_set_element_bool.c:27-29`), then one + `GxB_Matrix_isStoredElement` on M (`:31`), then one mutation — + `GrB_Matrix_removeElement(dm, …)` if the entry is in M (`:36`), + else `GrB_Matrix_setElement_BOOL(dp, …)` (`:39`). Two calls per + orientation, **four in total**. `delta_remove_element.c:28-44` is + the mirror image. + + Read: **1 to 3** `isStoredElement` probes — one if the entry is + in DP (`delta_isStored.c:28` returns early), three if it is in M. + + Against Step 1's alternative: one eager splice on a 100M-edge + matrix moves ~200 MB, about 6.7 ms at 30 GB/s. + +
+ +- [ ] You can say why each of the three matrices is pinned the way it is. + +
Answer + + From `delta_new.c:21-44`. M allows `GxB_SPARSE | GxB_HYPERSPARSE` + (`:25-26`), which lands in `GB_conform.c:175-184` — the bitmap + test never runs, and per the internals chapter a graph matrix is + 13,700× below the 0.40 threshold anyway, so nothing is lost. + + DP and DM are pinned `GxB_HYPERSPARSE` and forced with + `GxB_ALWAYS_HYPER` (`:32`, `:35`, `:41`, `:44`), landing in + `GB_conform.c:157-160` — `GB_convert_any_to_hyper` + unconditionally. Ten thousand entries over a 10M id space must be + hypersparse; that is `notes.md:41`'s 50×. + + The hyper_hash is disabled (`:33`, `:42`), so row lookup in `h[]` + is a binary search — 14 comparisons at k = 10,000, cheaper than + building and re-clearing SuiteSparse's `A->Y` + (`GB_Matrix_content.h:241-274`) at every flush. + + DM is always `GrB_BOOL` (`:40`) whatever the matrix type: a + tombstone carries no value, so DM is iso by construction. + +
+ - [ ] You can answer the load-bearing question: why FalkorDB needs its own deltas rather than SuiteSparse's pending tuples. -- [ ] You can explain `delta_mxm` as algebra instead of a flush, and identify where it over-masks. -- [ ] You can describe the two-sided compaction that `wait` performs and what triggers it. -- [ ] You can cost the transposed twin's extra write work per mutation. + +
Answer + + Three candidates, and each is grounded: (1) flush control — any + GrB read can force a wait (`GB_Matrix_content.h:386-389`), + whereas FalkorDB's thresholds are its own + (`delta_wait.c:89`, `:97`); (2) the transposed twin — + `delta_matrix.h:113` plus the recursion at every mutation, so + `<-[]-` traversals never need a transpose; (3) a bounded fold — + 10,000 pending changes (`config.h:19`), where library pending + tuples have no bound. + + Reason 2 is the only one SuiteSparse cannot supply at *any* + setting, because it has no concept of a matrix pair kept in + lockstep. Reasons 1 and 3 are about control of a mechanism that + exists; reason 2 is about a mechanism that does not. + + The transferable lesson: a lower layer's deferred-work mechanism + is reusable only if you control when it fires and what invariants + it maintains. + +
+ +- [ ] You can explain `delta_mxm` as algebra instead of a flush, and construct the case where it over-masks. + +
Answer + + Preconditions first: **only B may carry deltas**, asserted at + `delta_mxm.c:47` and `:50`. Then four calls — `mask = A·DM` over + `GxB_ANY_PAIR_BOOL` (`:74`), `accum = A·DP` over the real + semiring (`:86`), the main product `C = A·M` with + `GrB_DESC_RSC` (`:97`, `:104`), and an **unmasked** + `GrB_eWiseAdd` of `accum` (`:107`). + + Two departures from the comment at `:44`: the mask is `A·DM`, not + DM; and additions are added after the mask rather than under it. + + Over-masking: let A have two live edges out of row 0, to 0 and 1; + let B's settled M have M(0,5) and M(1,5); delete (0,5), so + DM(0,5) is set. Then `mask = A·DM` has an entry at (0,5), and + `:104` suppresses C(0,5) entirely — even though A(0,1)·B(1,5) is + a live path that should produce it, and `accum` is empty so + nothing restores it. The mask is structural: one dead path kills + the whole cell. + + How the callers compensate is question 3, and this chapter did + not verify it — it is not fixed inside `delta_mxm.c`. + + Cost of the algebra: at the 10,000 threshold and a mean degree of + 27.8, the two extra multiplies are about 5.6e5 flops against a + main product of ~1.28e8 — **0.43%**. + +
+ +- [ ] You can describe the two-sided compaction that `wait` performs, and say exactly what triggers it. + +
Answer + + Deletions first: `GrB_transpose(m, dm, NULL, m, GrB_DESC_RSCT0)` + at `delta_wait.c:29`, then `GrB_Matrix_clear(dm)` at `:33`. The + descriptor is `REPLACE + STRUCTURE + COMP + TRAN(input 0)`, so + the two transposes cancel and the net effect is "copy m into + itself, keeping only what dm does not mark" — the whole tombstone + sweep in one call. Then additions: + `GrB_Matrix_assign(m, dp, NULL, dp, GrB_ALL, 0, GrB_ALL, 0, + GrB_DESC_S)` at `:51-52`, then clear at `:56`. + + The trigger is **`delta_wait.c:89` and `:97`** — `dm_nvals >= + delta_max_pending_changes` and `dp_nvals >= …`, two independent + tests against `DELTA_MAX_PENDING_CHANGES_DEFAULT` = 10,000 + (`config.h:19`), with `force_sync` bypassing both at `:71-73`. + + It is **not** `delta_will_wait.c`. That file asks SuiteSparse + `GxB_WILL_WAIT` on all three matrices (`:35`, `:39`, `:43`) — + whether the *library* has zombies or pending tuples — and is used + as an assertion at `delta_wait.c:108-110` that both deferral + systems are drained after the three `GrB_wait` calls at + `:103-105`. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your Rust representation choice for DP/DM. +
Answer + + Design against three measured numbers rather than taste: the + write path is 4 GrB calls (Step 3), the fold threshold is 10,000 + entries (`config.h:19`), and at that threshold the mxm algebra + costs 0.43% (Step 5). + + The threshold curve is the real decision. Amortized fold cost per + mutation is N/T — at N = 100e6 and T = 10,000 that is 10,000 + entries rewritten per edge, against ~50e6 for an eager splice, a + 5,000× improvement. Halving T doubles the fold cost but halves + the mxm overhead and shortens the read path; raising it to + 100,000 cuts the fold to 1,000 entries per edge but pushes mxm + overhead to ~4.3%. Pick a point on that curve and say why. + +
+ ## References **Code** -- [FalkorDB](https://github.com/FalkorDB/FalkorDB) - `src/graph/delta_matrix/` — start with the state-transition - comment table in `delta_matrix.h:34-108` (it IS the design doc), - then `delta_wait.c`, `delta_mxm.c`, `delta_get_set.c`, - `delta_will_wait.c` + +- [FalkorDB](https://github.com/FalkorDB/FalkorDB) at `ccb449a9a` — + `src/graph/delta_matrix/` (23 files). Start with the state table + in `delta_matrix.h:26-106` (it IS the design doc), then + `delta_wait.c` (218 lines), `delta_mxm.c` (121 lines), + `delta_isStored.c`, `delta_set_element_bool.c`, + `delta_remove_element.c`, `delta_will_wait.c`. The flush constant + is `src/configuration/config.h:19`. Full anchor table above. +- [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) + at `1fd5475` — the layer underneath. + `Source/convert/GB_conform.c:157-160` and `:175-184` are the two + cases `delta_new.c`'s pins select; + `Source/builtin/include/GB_Matrix_content.h:241-274` is the + hyper_hash it disables, `:361` and `:367-391` are the pending + tuples and zombies it declines to rely on, `:386-389` is the + implicit-wait behaviour that motivates the whole layer. + `Include/GraphBLAS.h:666` and `:668` are `GrB_DESC_RSC` and + `GrB_DESC_RSCT0`. Walked in + [reading-suitesparse-internals.md](reading-suitesparse-internals.md). + +**Papers** + +- Davis, T. A. — "Algorithm 1000: SuiteSparse:GraphBLAS", ACM TOMS + 45(4), 2019. §3.1.8 is the O(e log e) bound this layer declines + to depend on; §4.1 defines zombies and pending tuples. Read in + [reading-davis-toms19.md](reading-davis-toms19.md). + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:41-42` — 80.4 MB → 1.59 MB and + 11,312 µs → 66 µs. The reason DP and DM are pinned hypersparse. +- `topics/24-graph-algorithms/notes.md:5-7` — RMAT scale 16, mean + degree 27.8. Step 5's overhead arithmetic runs on it. diff --git a/topics/20-graphblas/reading-gustavson-spgemm.md b/topics/20-graphblas/reading-gustavson-spgemm.md index 4f2fc9d..95325fe 100644 --- a/topics/20-graphblas/reading-gustavson-spgemm.md +++ b/topics/20-graphblas/reading-gustavson-spgemm.md @@ -8,18 +8,32 @@ multiply, what the SPA is — then uses Buluç & Gilbert's survey to map the whole design space onto one question: what data structure is the SPA? +A note on sourcing before anything else. Gustavson's 1978 paper is +behind the ACM paywall and could not be fetched for this repo, so +**nothing below is attributed to it directly**. The complexity +statement comes from Buluç & Gilbert 2012 §3, which restates it; +the workspace and masking behaviour come from Davis's TOMS '19 +§4.2.1; the shipped code comes from SuiteSparse:GraphBLAS at +`1fd5475`. Where those three disagree, the disagreement is the +interesting part and is called out. + ## The problem in one sentence Multiply two sparse matrices when the output's size is unknown -until you compute it — for a 10M×10M matrix, the naive -one-dot-product-per-output-cell view is 100 *trillion* dot -products, while the multiplications that actually exist may number -a few hundred million. +until you compute it — for the 10M-id-space matrix this topic +measures (`notes.md:41-42`), the naive one-dot-product-per-output- +cell view is 100 *trillion* candidate cells, while the +multiplications that actually exist number in the millions. ## The concepts, step by step ### Step 1 — SpGEMM, and why the dense loops die +> **In:** two sparse matrices A and B, and the textbook definition +> of matrix multiply. +> **Out:** the definition of *flops* for a sparse multiply, and the +> ratio between it and the dense candidate count. + **SpGEMM** (sparse general matrix-matrix multiply, C = A*B where both inputs are sparse) is dense matmul's three nested loops with two of them killed by sparsity. The **inner-product view** — for @@ -27,62 +41,174 @@ every output cell, C(i,j) = A(i,:)·B(:,j), a dot product of a row of A with a column of B — is how the math is written, but computed literally it does an intersection test per *output cell*: n² candidate cells, and for a graph matrix nearly all of those dot -products intersect to nothing. You'd spend almost all your time +products intersect to nothing. You would spend almost all your time proving zeros are zero. -The number to hold: the useful work is the **flops** — the scalar -multiplications between pairs of entries that both exist, -Σ over (i,k) ∈ A of nnz(B(k,:)) (nnz = number of stored entries). -For A² on a 100M-edge graph that's typically 10⁸–10⁹ flops — -against 10¹⁴ candidate output cells. An algorithm's quality is how -close its total work gets to the flops. +The number to hold is **flops**. Davis defines it precisely for a +semiring (TOMS '19 §4.2.1): "f is the number of 'multiply-adds' +computed (in the semiring)". Concretely, for C = A*B in the +column-wise form, it is Σ over (k,j) ∈ B of nnz(A(:,k)) — one +multiply-add per pair of entries that both exist. Note what it is +*not*: it is not nnz(C). Several flops can land in the same output +cell and be summed. + +This repo measures both, so the gap is not theoretical: + +``` + inputs: RMAT scale 14, edge factor 8 (notes.md:26) + n = 16,384, nnz(A) = 120K, flops = 17.1M, nnz(C) = 8.9M + + dense candidate cells = n² = 16,384² = 2.68e8 + flops that actually exist = = 1.71e7 + ratio = 15.7× wasted, at scale 14 -### Step 2 — the row-wise formulation: one output row at a time + now scale 20 (notes.md:14): n = 1.05e6 + dense candidate cells = 1.10e12 + flops (extrapolating the ×8 growth per +2 scale) ≈ 2.7e8 + ratio ≈ 4,100× wasted -Gustavson's move: compute C one *row* at a time, driven by A's -pattern instead of by output coordinates. For row i, each entry -A(i,k) contributes A(i,k) × (row k of B) into the output row — -scaled row additions instead of dot products: + the waste grows linearly in n. That is the whole reason the + inner-product view is unusable, and it gets worse with scale. + flops / nnz(C) = 1.71e7 / 8.9e6 = 1.92 (notes.md:28-31) + ⇒ the average output cell receives ~2 contributions. The + accumulator is doing real work on about half its updates and + pure insert on the other half. ``` - for i in rows(A): # one output row at a time - for k where A(i,k) ≠ 0: # A's row pattern - for j where B(k,j) ≠ 0: # B's row k - SPA[j] += A(i,k) * B(k,j) # scatter-accumulate - C(i,:) = gather nonzeros of SPA # then reset SPA + +Why it matters: an SpGEMM algorithm's quality is how close its +total work gets to flops. Everything below is measured against that +one number. + +### Step 2 — the row-wise (or column-wise) formulation + +> **In:** the flop count from Step 1. +> **Out:** the loop that achieves it, and the two names the +> literature gives the same loop. + +Gustavson's move: compute C one *vector* at a time, driven by an +input's pattern instead of by output coordinates. Buluç & Gilbert +§3, Algorithm 1, give it column-wise: + ``` + Algorithm 1 Column-wise formulation of serial matrix multiplication + 1: procedure Columnwise-SpGEMM(A, B, C) + 2: for j ← 1 to n do + 3: for k where B(k,j) ≠ 0 do + 4: C(:,j) ← C(:,j) + A(:,k) · B(k,j) +``` + +That is a *saxpy* — scalar times a vector, added into an +accumulator — which is where SuiteSparse's engine gets its name. +The row-wise form in this chapter's title is the same algorithm +read through a transpose: for row i, each entry A(i,k) contributes +A(i,k) × (row k of B). SuiteSparse says so outright: + +```c +// GB_AxB_saxpy3.c — the duality, in one line, 20 + 20 // all matrices are in CSC format, but the algorithm is CSR/CSC agnostic. +``` + +Keep both names, and always say which orientation you mean, because +the *storage* has to match the loop: the column-wise form wants +CSC for A and B, the row-wise form wants CSR. SuiteSparse's default +is CSR (`GB_Global.c:203` sets `.is_csc = false`), so its saxpy3 +comment's "vectors of B" are rows. -Work = flops = Σᵢₖ nnz(B(k,:)) over A's entries — *optimal*: every -multiplication performed is a term that exists in the answer, and -none can be skipped (unless the semiring short-circuits — question -1). No zero is ever inspected. Bonus: both A and B are consumed -row-by-row, so CSR (row-major sparse storage) serves both inputs -sequentially. This is why 1978's algorithm is still the one in -every library. +The complexity, from Buluç & Gilbert §3: + +> "That algorithm, shown in Figure 3.1, runs in **O(flops + nnz + n)** +> time, which is **optimal for flops ≥ max{nnz, n}**. It uses the +> popular compressed sparse column (CSC) format for representing +> its sparse matrices." + +Three terms, and each earns its place: `flops` is the useful work, +`nnz` is reading the inputs once, and `n` is *constructing the +output's pointer array* `C.p` of length n+1 — you pay for a slot +per column whether or not that column has anything in it. Davis +makes the same point in TOMS '19 §4.2.1: "Constructing C takes +Ω(n) time and space if it is stored in standard compressed +sparse-column form with a pointer array C.p of size n + 1." + +Why it matters: the optimality condition `flops ≥ max{nnz, n}` is +not decoration. Step 7 shows this repo's own measured case where it +fails by two orders of magnitude. ### Step 3 — the SPA: the accumulator that makes scattering O(1) -The one data structure the loop needs is somewhere to accumulate a -row's scattered contributions — entries for the same output column -j arrive from different k's, in no order. Gustavson's **SPA** -(sparse accumulator) is a dense array of size m (one slot per -possible output column) plus a list of which slots are occupied: +> **In:** the loop of Step 2, which produces contributions to the +> same output cell from different k, in no order. +> **Out:** the SPA, its three operations, its cost in bytes, and +> the trick that makes clearing it free. + +Gustavson's **SPA** (sparse accumulator) is a dense array of size m +— one slot per possible output index — plus a record of which slots +are occupied. Buluç & Gilbert's Figure 3.1 caption: + +> "Columns of A are accumulated as specified by the non-zero +> entries in a column of B using a **sparse accumulator or SPA**. +> The contents of the SPA are stored into a column of C once all +> required columns are accumulated." -- scatter: `SPA[j] += v` is one array write — O(1), no probing; -- a marker array (or generation counter) records first touches and - appends j to the occupied list; -- gather: walk the occupied list to emit the finished row, then - reset only those slots. +Three operations: -The cost profile: O(1) per flop, but m slots of memory per thread -— for m = 10M that's an 80 MB array touched at random points, -i.e. cold DRAM per row (question 2 makes you compute the -crossover). The SPA is dense-workspace thinking: pay memory for -zero per-element search cost. +- **scatter**: `SPA[j] += v` is one array write — O(1), no probing; +- **mark**: a parallel array records first touches and appends j to + the occupied list; +- **gather**: walk the occupied list to emit the finished vector, + then reset only those slots. + +The clever part is the reset. Davis describes SuiteSparse's version +in TOMS '19 §4.2.1, and it is the design our own stub is asked to +copy (`notes.md:59` — "stamp-marked SPA"): + +> "The other is an initialized integer array, `mark`… When +> initialized, `mark[i] true, `mark[i]=flag` is done, and clearing the entire `mark` array +> simply requires `flag` to be incremented. **The entire space is +> cleared in constant time.**" + +A generation counter, not a memset. Without it, each output vector +would cost O(m) to clear and the whole `O(n + f)` claim would +collapse to `O(nm)`. + +Price the array, using this topic's own configuration +(`notes.md:50-51` sizes our stub's SPA at 12 bytes per slot — an +f64 value plus a 4-byte stamp): + +``` + inputs: SPA slot = 12 bytes (f64 value + u32 stamp), notes.md:50 + M3 Pro, L2 ≈ 16 MB shared (notes.md:3, topic 13) + + scale 14: m = 16,384 → 16,384 × 12 B = 196,608 B = 192 KB fits L2 easily + scale 16: m = 65,536 → 786 KB fits + scale 18: m = 262,144 → 3.1 MB fits, but 8 threads want 25 MB + scale 20: m = 1,048,576→ 12.6 MB one thread nearly fills L2 + + so single-threaded, the SPA survives to scale 20; at 8 threads it + falls out of cache somewhere around scale 17-18, which is the + prediction notes.md:50-51 asks you to make. + + the crossover argument, in one line: the SPA costs 12·m bytes + regardless of how many entries the vector has, while a hash table + costs ~16 bytes × 2·flmax. They cost the same when + 12·m = 32·flmax ⇒ flmax = 0.375·m + — which is why every implementation's real threshold (SuiteSparse + uses m/12 at GB_AxB_saxpy3_slice_balanced.c:94) sits far BELOW + that break-even: cache residency, not byte count, is the criterion. +``` + +Why it matters: the SPA is dense-workspace thinking — pay memory to +make per-element search cost zero. Every argument about when to +abandon it is an argument about cache, not about asymptotics. ### Step 4 — the design space is "what data structure is the SPA" -Everything since 1978 keeps the row-wise loop and swaps the +> **In:** the SPA of Step 3. +> **Out:** the three accumulators the literature and the code +> actually ship, with the shipped selection rule. + +Everything since 1978 keeps the vector-at-a-time loop and swaps the accumulator: ``` @@ -90,119 +216,570 @@ accumulator: O(1) scatter, O(m) alloc, gather via occupied list SPA = hash table (saxpy3 hash task) O(1)-ish scatter, O(flops) alloc — wins for huge m - SPA = heap / sorted-list merge (merge k sorted rows of B) + SPA = heap / sorted-list merge (merge k sorted vectors of B) output comes out SORTED — no gather/sort pass ``` -The selection logic: dense SPA wins when the output row fills -enough of m to amortize the cold array; hash wins when m is huge -and the row sparse (the table is sized by flops, so it stays in -cache); heap/merge wins when you need sorted output for free. -saxpy3's m/16 rule (previous chapter) is exactly this decision, -automated per task. +All three are in SuiteSparse's history, and Davis names them in +TOMS '19 §4.2.1 as the library's three methods at version 2.3.3: +"(1) a variant of Gustavson's algorithm, (2) a heap-based method, +and (3) a dot-product formulation." The selection rule in that +paper is a sentence: "**If m is large compared with |A| + |B|, +Gustavson's method is not used, and the heap-based method is used +instead.**" + +The modern library's rule is different in every respect — the heap +method is gone, replaced by a hash, and the threshold is a number: + +```c +// GB_AxB_saxpy3_slice_balanced.c — the shipped rule, 83-95 (elided) + 82 // hash_size = 2 * (smallest power of 2 >= flmax) + 83 hash_size = ((uint64_t) 2) << (GB_FLOOR_LOG2 (flmax) + 1) ; + ... + 92 // default: auto selection: + 93 // use Gustavson's method if hash_size is too big + 94 use_Gustavson = (hash_size >= cvlen/12) ; +``` + +Note the direction: *bigger* hash table means *use Gustavson*. A +vector with many flops needs a big table; once that table +approaches the size of the dense SPA, the dense SPA's O(1) scatter +is strictly better. The crossover is worked exactly in +[reading-suitesparse-internals.md](reading-suitesparse-internals.md) +Step 6 — at m = 2²⁰ it lands at flmax = 32,768. + +Why it matters: "which accumulator" is not a research question, it +is a runtime branch on one number, and both the number and the +comparison direction are checkable in fifteen lines of source. ### Step 5 — the unknown-output-size problem: symbolic then numeric +> **In:** the loop and an accumulator. +> **Out:** why the pattern gets walked twice, and what the second +> walk buys. + nnz(C) is unknown before you compute C, so how big do you allocate -the output arrays? Gustavson's answer is two phases: a **symbolic -phase** runs the same loop on patterns only (no values, no -arithmetic) to compute each output row's nnz and allocate exactly, -then a **numeric phase** fills the values. Every system in this -curriculum that meets sparse output rediscovers this: saxpy3's -flopcount pre-pass, cudf's size/retrieve (topic 18), Gunrock's -degree scan. The alternative is guess-and-grow (topic 17's -simdjson over-allocate answer) — cheaper when rows are small and -uniform, disastrous under skew. Our stub does symbolic+numeric; -the HashMap reference does guess-free accumulation and pays for it -in allocator traffic. - -### Step 6 — Buluç & Gilbert's axes: the survey's map - -The survey organizes every SpGEMM as a point in a small space: - -- **formulation**: row-wise (Gustavson) / outer-product (column of - A × row of B → rank-1 updates, needs merging) / inner-product -- **accumulator**: SPA / hash / heap / merge — pick by density of - the output row and size of m -- **parallelism**: rows are independent (row-wise ⇒ embarrassingly - parallel over i) BUT power-law graphs make row costs wildly - unequal ⇒ saxpy3's coarse/fine split, Gunrock's merge_path — the - same load-balance problem at every layer of this curriculum -- **compression**: masked SpGEMM (`C=A*B`) can skip work only in - dot formulation; Gustavson's mask only prunes writes - -The last axis is worth dwelling on: in row-wise, the flops happen -before the mask can reject them; in dot (inner-product driven *by -the mask*), masked-out cells cost nothing — which is why LAGraph's -triangle counting ships both formulations (question 5). - -### Step 7 — skew: the cost intuition to carry - -For RMAT/power-law A², flops concentrate in hub rows: row i's cost -is Σ of the degrees of i's neighbors — a degree-squared weighting. -A few rows are 1000× the median, so static row partitioning dies -(7 threads finish, 1 grinds a hub row), which is why every real -implementation has the fine-task path. Whatever accumulator you -pick, the load balancer must be designed for the tail, not the -median. +the output arrays? The answer is two phases: a **symbolic phase** +runs the same loop on patterns only (no values, no arithmetic) to +compute each output vector's nnz and allocate exactly, then a +**numeric phase** fills the values. Davis, TOMS '19 §4.2.1: + +> "In the first method, when no mask is present, the work is split +> into a symbolic analysis phase that finds the pattern of C and a +> numerical phase that computes its values… **both phases take +> only O(n + f) time**, assuming all matrices are in CSC format, and +> assuming the O(m) workspace is already allocated and +> initialized." + +Read the assumption clause twice. The `O(n + f)` bound — better +than Buluç & Gilbert's `O(flops + nnz + n)` because it drops the +input-reading term into f — holds **only if the O(m) workspace is +already there**. Davis spends the next paragraph on why that matters +for BFS: "Assuming the workspace of size O(m) has already been +allocated and initialized, the time to compute this set is simply +O(f)… The O(m) work appears just once in the entire breadth-first +search algorithm." + +Every system in this curriculum that meets sparse output +rediscovers the two-phase shape: saxpy3's flopcount pre-pass +(`GB_AxB_saxpy3_flopcount.c:44-48`, `O(nnz(B)+n)`), cudf's +size/retrieve (topic 18), Gunrock's degree scan. The alternative is +guess-and-grow (topic 17's simdjson over-allocate answer) — cheaper +when vectors are small and uniform, disastrous under skew. Our stub +does symbolic+numeric; the HashMap reference does guess-free +accumulation and pays for it in allocator traffic, which is +measurable: + +``` + inputs: notes.md:22-26, HashMap reference (guess-and-grow, per-row alloc) + + scale 10: 298K flops / 3.9 ms = 76 Mflop/s = 13.1 ns/flop + scale 12: 2.27M flops / 33.0 ms = 69 Mflop/s = 14.5 ns/flop + scale 14: 17.10M flops / 279.4 ms = 61 Mflop/s = 16.3 ns/flop + + a multiply-add is ~1 ns of arithmetic at best. So 13-16 ns/flop + means ~93% of the time is NOT arithmetic — it is hashing, probing, + per-row allocation and the final sort. + + and the per-flop cost DEGRADES 24% from scale 10 to 14 even though + the flop count grew 57×: the accumulator is falling out of cache, + exactly the effect Step 3's byte arithmetic predicts. +``` + +Why it matters: the two-phase design is not fastidiousness. At +16 ns/flop the arithmetic is invisible; what you are optimizing is +allocation and memory traffic, and knowing the size up front is how +you remove both. + +### Step 6 — masking: where the mask can and cannot save work + +> **In:** the two formulations (Step 2's saxpy, Step 1's discarded +> inner product). +> **Out:** which one a mask actually prunes, in the source's own +> words. + +In row/column-wise saxpy, the flops happen *before* the mask can +reject them. Davis is unusually blunt about this, TOMS '19 §4.2.1: + +> "If the mask is present (and not complemented), only the subset +> of entries appearing in the mask are computed. This greatly +> reduces the time and memory usage. In this method, the symbolic +> analysis is skipped. A matrix T = AB is computed whose pattern is +> assumed to be a subset of the mask matrix M. **Entries in AB +> outside the mask need not be computed, and are discarded if they +> are computed.**" + +"Discarded if they are computed" is the honest half of the +sentence. The mask does buy something real — the symbolic phase is +skipped entirely, and `GB_AxB_saxpy3_flopcount.c:53` skips whole +vectors whose mask vector is empty — but within a vector that the +mask keeps, the multiply-adds still run and the losers are thrown +away. + +In the inner-product formulation driven *by the mask*, masked-out +cells cost nothing at all, because the mask is the loop bound. The +proof is allocation: `GB_AxB_dot3.c:126` computes `mnz = GB_nnz(M)` +and `:171` sets `cnz = mnz`. + +Davis's triangle-counting example, §4.2.1, is the canonical case: + +> "if L is the strictly lower triangular part of an unweighted +> graph A, then C⟨L⟩ = L² finds the number of triangles in the +> graph… Not all of L² is computed or stored, but only the entries +> corresponding to entries in the mask, L. This greatly reduces the +> time and memory complexity of the masked matrix multiply, as +> compared with computing all of L² first and then applying the +> mask, as would be done in the MATLAB expression `C=(L^2).*L`." + +Which is why LAGraph's triangle counting ships six formulations +(`LAGr_TriangleCount.c:31-37`) and its own performance note +(`:43-47`) refuses to name one winner — it says the dot-based +Sandia_LUT is usually fastest on the largest graphs *except* on +GAP-urand, where the saxpy-based LL wins. Question 5. + +Why it matters: "masks are free performance" is a half-truth whose +other half is a discarded multiply-add, and knowing which half you +are getting requires knowing which engine ran. + +### Step 7 — where O(flops + nnz + n) fails: hypersparsity + +> **In:** the complexity of Step 2 and its optimality condition. +> **Out:** the measured case in this repo where the `n` term wins, +> and the data structure that removes it. + +Buluç & Gilbert's condition was `optimal for flops ≥ max{nnz, n}`. +Check it against this topic's own measurements: + +``` + inputs: notes.md:22-26 (RMAT ladder) and notes.md:41-42 (hypersparse) + + RMAT scale 14: flops = 1.71e7, nnz = 1.20e5, n = 1.64e4 + max{nnz, n} = 1.20e5 + flops / max = 143× → comfortably optimal, the n term is 0.1% + + RMAT scale 10: flops = 2.98e5, nnz = 6.7e3, n = 1.02e3 + flops / max = 44× → still optimal + + hypersparse case: 10M-node id space, 100K edges + n = 1.0e7, nnz = 1.0e5, flops for A² ≈ nnz × mean-degree ≈ 1e6 + max{nnz, n} = 1.0e7 + flops / max = 0.1× → the condition FAILS by 10× + + Gustavson would spend O(n) = 1.0e7 just building C.p, against + 1e6 flops of real work: 91% of the runtime is allocating and + walking pointer slots for columns that are empty. +``` + +That is not a hypothetical — it is the measurement in `FINDINGS.md` +row 20 seen from the algorithm's side. The same 10M-id-space graph +costs **80.4 MB of CSR index versus 1.59 MB hypersparse (50×)**, +and a full sweep takes **11,312 µs versus 66 µs (171×)** +(`notes.md:41-42`). The 171× is the `n` term of +`O(flops + nnz + n)` being deleted. + +Buluç & Gilbert say exactly this, §3.1: + +> "any algorithm whose complexity depends on matrix dimension, such +> as Gustavson's serial SpGEMM algorithm, is **asymptotically too +> wasteful** to be used as a computational kernel for multiplying +> the hypersparse submatrices. Our HyperSparseGEMM, on the other +> hand, operates on the strictly O(nnz) doubly compressed sparse +> column (DCSC) data structure, and its time complexity does not +> depend on the matrix dimension." + +Their replacement is an outer-product formulation with complexity +`O(nzc(A) + nzr(B) + flops·lg nᵢ)` and memory +`O(nnz(A) + nnz(B) + nnz(C))` — note the `lg nᵢ` factor, which +they attribute to "the priority queue that is used to merge nᵢ +outer products on the fly". You trade the dimension term for a log +factor on the flops. **DCSC** is CSC with the repetitions in the +column-pointer array removed: "Only columns that have at least one +nonzero are represented, together with their column indices" (§3.2) +— which is the same idea as SuiteSparse's hypersparse `A.h`. + +Davis's version of the same argument, TOMS '19 §4.2.1: "A +hypersparse format need only operate on the non-empty columns of B +and C, however, so the time complexity drops to O(n̄_B + f) where +n̄_B < n is the number of non-empty columns of B." And the trigger: +"SuiteSparse:GraphBLAS stores its matrices in hypersparse format if +n̄ < n/16" — which matches `GB_defaults.h:20`'s 0.0625 exactly. + +Why it matters: this is the one place where an asymptotic term that +looks like bookkeeping turns into the entire runtime, and this +repo has the measurement. + +### Step 8 — skew: the cost intuition to carry + +> **In:** the row-wise loop, which looks embarrassingly parallel. +> **Out:** why it is not, with this repo's measured degree +> distribution. + +For RMAT/power-law A², flops concentrate in hub vectors: vector i's +cost is Σ of the degrees of i's neighbours — a degree-squared +weighting. Measured, on the graph generator this curriculum uses: + +``` + inputs: RMAT scale 16, topics/24-graph-algorithms/notes.md:5-7 + n = 65,536, m = 1,819,338, max degree = 9,751 + uniform graph, same n and m: max degree = 59 + + mean degree = 1,819,338 / 65,536 = 27.8 + max / mean = 9,751 / 27.8 = 351× + uniform max/mean = 59 / 27.8 = 2.1× + + now the SpGEMM cost, which is degree-SQUARED-ish: + a mean row's flops ≈ 27.8 × 27.8 = 773 + the hub row's flops ≈ 9,751 × 27.8 = 271,078 + ratio = 351× + + total flops at scale 16 are not in notes.md (the ladder stops at + 14), so extrapolate: the measured ladder grows 298K → 2.27M → + 17.1M, i.e. ×7.6 then ×7.5 per +2 scale, so scale 16 ≈ 1.28e8. + + static partition over 8 threads, 65,536 rows, 8,192 rows each: + fair share = 1.28e8 / 8 = 1.60e7 flops + one hub row = 271,078 flops = 1.7% of a thread's ENTIRE fair + share, in a single indivisible unit if you only split by row — + and a hub's neighbours are hubs too, so the 8,192-row block + containing it can easily carry a multiple of that. +``` + +A few rows are hundreds of times the median, so static row +partitioning dies — seven threads finish and one grinds a hub row — +which is why every real implementation has the fine-task path +(`GB_AxB_saxpy3.c:22-27`). Whatever accumulator you pick, the load +balancer must be designed for the tail, not the median. + +Why it matters: the 351× is measured on the same generator our +benchmarks use, so the load-imbalance argument in this curriculum is +not borrowed from a paper about someone else's graph. ## How to read the paper (with the concepts in hand) -- **Gustavson '78** — short and readable; read it whole. The - row-wise algorithm is step 2, the SPA is step 3, and the - symbolic/numeric two-phase is step 5 (it's also where the - "permuted transposition" half of the title lives — the same - two-phase builds a transpose). Notice how little the 1978 prose - differs from saxpy3's header comment. -- **Buluç & Gilbert 2012** — read §1-3 for the design-space - framing (step 6): formulation × accumulator × parallelism. Skim - the distributed-memory experiments; the axes are the payload. - Map each system you've met (saxpy3 Gustavson task, saxpy3 hash - task, our stub, the HashMap reference) onto a point in their - space as you read. +- **Gustavson '78** — the ACM version is paywalled; if you have + institutional access, read it whole (it is short). The row-wise + algorithm is Step 2, the SPA is Step 3, and the symbolic/numeric + two-phase is Step 5 — it is also where the "permuted + transposition" half of the title lives, since the same two-phase + shape builds a transpose. If you cannot get it, read its two + restatements instead: Buluç & Gilbert §3 (Algorithm 1, Figure + 3.1) and Davis TOMS '19 §4.2.1, both of which are open. +- **Buluç & Gilbert 2012** — read §3 first: it is two pages and + contains the complexity claim, Algorithm 1, and the SPA figure. + Then §3.1-3.2 for the hypersparse argument (Step 7) and DCSC. + The distributed-memory experiments in §4 onward are skimmable; + the axes are the payload. Map each system you have met — saxpy3's + Gustavson task, saxpy3's hash task, our stub, the HashMap + reference — onto a point in their space as you read. +- **Davis TOMS '19 §4.2.1** — the third statement of the same + algorithm, and the only one that tells you what the *workspace* + does. Read the four paragraphs from "In the first method" to + "cleared in constant time"; they are the specification for our + stamp-marked SPA stub. Remember its version caveat: everything in + that paper is single-threaded. + +### What transfers to M20 + +- `spgemm_spa` (`notes.md:59`) is Step 3 verbatim: dense array, + stamp array, occupied list, generation counter. The stamp trick + is what makes the reset O(nnz of this vector) instead of O(m). +- The symbolic/numeric split of Step 5 is what lets M20 allocate + `C.i`/`C.x` exactly once. The HashMap reference deliberately does + not, which is what the 13-16 ns/flop in Step 5's arithmetic is + measuring. +- Step 7's arithmetic is the argument for M20's hypersparse index: + the `n` term is the whole cost at 10M ids, and deleting it is the + 171× in `notes.md:42`. ## Questions for notes.md 1. Derive: why is Gustavson's total work exactly - Σ_{(i,k)∈A} nnz(B(k,:)) and why can no SpGEMM do fewer - multiplications (each is a necessary term — unless the SEMIRING - short-circuits: ANY_PAIR reachability can stop early — where?). -2. The dense SPA costs m bytes×2 (value + mark) per thread. For - m = 10M that's cold DRAM per row. Compute the crossover row - density where hash beats SPA using topic 13's cache numbers - (SPA touches nnz_out random cells of an 80 MB array; hash - touches nnz_out cells of a 2×flops table that fits L2). + Σ_{(k,j)∈B} nnz(A(:,k)) (equivalently Σ_{(i,k)∈A} nnz(B(k,:)) + in the row-wise orientation), and why can no SpGEMM do fewer + multiplications — each is a necessary term, *unless* the + semiring short-circuits. ANY_PAIR reachability can stop early: + find where, and say what property of the monoid licenses it. +2. The dense SPA costs 12 bytes per slot (`notes.md:50`) per + thread. Compute the crossover vector density where hash beats + SPA using topic 13's cache numbers — SPA touches nnz_out random + cells of a 12·m array; hash touches nnz_out cells of a + 2×flops table that fits L2 — and compare your answer to + SuiteSparse's shipped `cvlen/12` + (`GB_AxB_saxpy3_slice_balanced.c:94`). 3. Symbolic+numeric does the pattern walk TWICE. When is - guess-and-grow cheaper (flops/row small and uniform — the - variance argument; connect to topic 16's ddmin determinism - requirement... no wait, to cudf's retrieve-skip answer)? + guess-and-grow cheaper? Make the variance argument — flops per + vector small and uniform — and connect it to cudf's + retrieve-skip answer (topic 18). 4. Outer-product SpGEMM produces k rank-1 updates that must be merged — which topic 3 structure is that (LSM: sorted runs + merge), and why does it win out-of-core / distributed - (sequential I/O, no random SPA)? -5. Masked Gustavson can't skip work; masked dot can. Show it on - triangle counting `C=L*L`: what does each formulation compute - per wedge, and reconcile with LAGraph shipping BOTH Sandia_LL - (saxpy) and Sandia_LUT (dot) as the fastest per-graph choices. + (sequential I/O, no random SPA)? Buluç & Gilbert §3.1's + `flops·lg nᵢ` is the priority-queue cost of that merge; price it + against the SPA's O(1) scatter at scale 14's flop count. +5. Masked saxpy discards; masked dot never computes. Show it on + triangle counting `C=L*L`: what does each formulation do per + wedge, and reconcile that with LAGraph shipping both Sandia_LL + (saxpy) and Sandia_LUT (dot), with `LAGr_TriangleCount.c:43-47` + naming different winners on different graphs. ## Done when -- [ ] You can derive Gustavson's total work and say why it equals the flop count rather than the output size. -- [ ] You can explain what the SPA does and why it makes scattering O(1). -- [ ] You can state the design space in one sentence: what data structure the SPA is — and connect it to this topic's measured SpGEMM (356.9 ms hash at scale 14, 17.1 M flops). +Answer each before unfolding it. + +- [ ] You can define flops for a sparse multiply, and say why it is not nnz(C). + +
Answer + + Davis, TOMS '19 §4.2.1: "f is the number of 'multiply-adds' + computed (in the semiring)" — one per pair of input entries that + both exist, Σ over (k,j) ∈ B of nnz(A(:,k)). + + It is not nnz(C) because several flops can land in the same + output cell and be summed. This topic measures the ratio: + 17.1M flops against 8.9M output entries at scale 14 + (`notes.md:26`), so flops/nnz(C) = 1.92. `notes.md:28-31` reads + that as RMAT A² producing mostly-distinct pairs — the accumulator + "rarely accumulates", which is the hash's worst case because + almost every update is an insert rather than a merge. + +
+ +- [ ] You can state Gustavson's complexity with all three terms and its optimality condition, and say what each term pays for. + +
Answer + + Buluç & Gilbert §3: "**O(flops + nnz + n)** time, which is + **optimal for flops ≥ max{nnz, n}**." + + `flops` is the useful work; `nnz` is reading the inputs once; `n` + is building the output's pointer array `C.p` of length n+1 — one + slot per column whether or not the column is empty. Davis makes + the same point independently: "Constructing C takes Ω(n) time and + space if it is stored in standard compressed sparse-column form + with a pointer array C.p of size n + 1" (TOMS '19 §4.2.1). + + Davis's tighter `O(n + f)` for both phases is not a contradiction + — it assumes the O(m) workspace is already allocated and + initialised, which folds the input-reading term into f. State the + assumption whenever you quote it. + +
+ +- [ ] You can explain what the SPA does, why scattering is O(1), and how it is cleared. + +
Answer + + A dense array of size m — one slot per possible output index — + plus a marker array and an occupied list. Scatter is `SPA[j] += v`, + a single array write with no probing. Gather walks the occupied + list to emit the vector. + + Clearing is the interesting part. Davis, TOMS '19 §4.2.1: the + marker array `mark` satisfies `mark[i] < flag` when clear; setting + entry i is `mark[i] = flag`; and "clearing the entire mark array + simply requires flag to be incremented", so "the entire space is + cleared in constant time". Without the generation counter, each + output vector would cost O(m) to reset and the O(n + f) bound + would become O(nm). + + Buluç & Gilbert's Figure 3.1 caption is the other half: + "The contents of the SPA are stored into a column of C once all + required columns are accumulated." + +
+ +- [ ] You can state the design space in one sentence, and connect it to this topic's measured SpGEMM. + +
Answer + + Keep the vector-at-a-time loop; change what the SPA is — dense + array, hash table, or heap/merge. Davis's TOMS '19 §4.2.1 lists + the library's three methods of that era as Gustavson, heap-based, + and dot-product, with the rule "If m is large compared with + |A| + |B|, Gustavson's method is not used, and the heap-based + method is used instead." The modern code replaced the heap with a + hash and the prose rule with a number: + `use_Gustavson = (hash_size >= cvlen/12)` + (`GB_AxB_saxpy3_slice_balanced.c:94`). + + Measured here (`notes.md:22-26`), the HashMap reference runs + **279.4 ms on 17.1M flops at scale 14** — 61 Mflop/s, or + 16.3 ns/flop, degrading 24% from the 13.1 ns/flop it manages at + scale 10. That degradation with no change of algorithm is the + accumulator falling out of cache, which is the whole argument for + having two accumulators. + +
+ - [ ] You can explain the unknown-output-size problem and when symbolic-then-numeric beats guessing. -- [ ] You can show, on an example, why masked Gustavson cannot skip work but masked dot can. + +
Answer + + nnz(C) is unknowable before computing C, so either walk the + patterns twice (symbolic sizes the allocation exactly, numeric + fills it) or guess and grow. Symbolic+numeric wins when growth is + expensive or vector sizes vary wildly — which is every power-law + graph, per Step 8's 351× max/mean. + + Guess-and-grow wins when flops per vector are small and uniform, + because the second pattern walk is pure overhead you cannot + amortise. The measured price of guessing is in `notes.md:24-26`: + 13-16 ns/flop against roughly 1 ns of actual arithmetic, so about + 93% of the HashMap reference's time is hashing, probing, + per-row allocation and sorting. + +
+ +- [ ] You can show, on an example, why masked saxpy cannot skip work but masked dot can. + +
Answer + + Davis, TOMS '19 §4.2.1, on the masked saxpy: the symbolic phase is + skipped and "a matrix T = AB is computed whose pattern is assumed + to be a subset of the mask matrix M. Entries in AB outside the + mask **need not be computed, and are discarded if they are + computed**." The flops happen; the losers are thrown away. The + only whole-vector saving is at + `GB_AxB_saxpy3_flopcount.c:53`, which skips a column whose mask + column is empty. + + In the dot formulation the mask is the loop bound, so masked-out + cells cost literally nothing: `GB_AxB_dot3.c:126` takes + `mnz = GB_nnz(M)` and `:171` sets `cnz = mnz`, sizing C to the + mask exactly. + + Davis's own example is triangle counting: C⟨L⟩ = L² where "Not + all of L² is computed or stored, but only the entries + corresponding to entries in the mask, L", against MATLAB's + `C=(L^2).*L`, which materialises L² first. LAGraph ships both + formulations for the same reason and `LAGr_TriangleCount.c:43-47` + names different winners on different graphs. + +
+ +- [ ] You can name the case where O(flops + nnz + n) is *not* optimal, and quote this repo's measurement of it. + +
Answer + + When `flops < max{nnz, n}` — i.e. hypersparsity, where the matrix + dimension dwarfs the entry count. This repo's case: a 10M-node id + space with 100K edges (`notes.md:41-42`), where n = 1.0e7 against + nnz = 1.0e5. The `n` term is 100× the entry count, so almost all + the work is walking pointer slots for empty columns. + + Measured: **80.4 MB of CSR index versus 1.59 MB hypersparse + (50×)**, and a full sweep of **11,312 µs versus 66 µs (171×)**. + (`FINDINGS.md` row 20 quotes 175× for the sweep; `notes.md:42` + quotes 171×. Cite one, and say which — this answer uses + `notes.md`, the measured baseline for this topic.) + + Buluç & Gilbert §3.1 predict it: Gustavson's algorithm is + "asymptotically too wasteful" for hypersparse blocks, and their + DCSC-based HyperSparseGEMM runs in + `O(nzc(A) + nzr(B) + flops·lg nᵢ)` — dimension-free, at the cost + of a `lg nᵢ` factor from the merge priority queue. Davis's + equivalent is `O(n̄_B + f)` with the trigger "hypersparse format if + n̄ < n/16" (TOMS '19 §4.2.1), matching `GB_defaults.h:20`'s + 0.0625. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the dense-SPA memory cost per thread. +
Answer + + The number to have: 12 bytes per slot (`notes.md:50` — f64 value + plus a 4-byte stamp), so 12·m bytes per thread, independent of how + many entries any vector holds. At scale 14 that is 192 KB; at + scale 20 it is 12.6 MB; at 8 threads scale 18 already wants + 25 MB against an L2 of about 16 MB. + + For SuiteSparse's own coarse Gustavson task the figure is 16 + bytes per row (`GB_AxB_saxpy3.c:68` — `uint64_t Hf[m]` plus + `double Hx[m]`), and its fine variant is 9 (`:66` uses + `int8_t Hf`). Do not mix the three numbers; say which + implementation each belongs to. + +
+ ## References **Papers** -- Gustavson — "Two Fast Algorithms for Sparse Matrices: - Multiplication and Permuted Transposition" (ACM TOMS 1978) — the - row-wise algorithm + the symbolic/numeric two-phase; short and - readable -- Buluç, Gilbert — "Parallel Sparse Matrix-Matrix Multiplication - and Indexing: Implementation and Experiments" (SIAM J. Sci. - Comput. 2012, [arXiv:1109.3739](https://arxiv.org/abs/1109.3739)) - — the design-space framing: formulation × accumulator × - parallelism + +- Gustavson, F. G. — "Two Fast Algorithms for Sparse Matrices: + Multiplication and Permuted Transposition", ACM TOMS 4(3), 1978, + 250-269, + [doi:10.1145/355791.355796](https://doi.org/10.1145/355791.355796). + The row-wise algorithm, the SPA, and the symbolic/numeric + two-phase. **Paywalled**; nothing in this chapter is quoted from + it. It is reference [1] in `GB_AxB_saxpy3.c:78-80` and reference + [7] in Davis's CSC '20 paper. +- Buluç, A. & Gilbert, J. R. — "Parallel Sparse Matrix-Matrix + Multiplication and Indexing: Implementation and Experiments", + SIAM J. Sci. Comput. 34(4), 2012, + [arXiv:1109.3739](https://arxiv.org/abs/1109.3739). §3 is the + complexity claim, Algorithm 1 (column-wise) and Figure 3.1 (the + SPA); §3.1 is the hypersparse argument and HyperSparseGEMM's + `O(nzc(A) + nzr(B) + flops·lg nᵢ)`; §3.2 is DCSC. +- Davis, T. A. — "Algorithm 1000: SuiteSparse:GraphBLAS", ACM TOMS + 45(4), 2019. §4.2.1 is the third statement of Gustavson's + algorithm, and the only one that specifies the workspace, the + generation-counter clear, and the masked variant's + discard-if-computed behaviour. Cited from the author's accepted + manuscript (titled "Algorithm 9xx", describing version 2.3.3, and + **single-threaded**). Walked in + [reading-davis-toms19.md](reading-davis-toms19.md). +- Nagasaka, Y., Matsuoka, S., Azad, A., Buluç, A. — + "High-Performance Sparse Matrix-Matrix Products on Intel KNL and + Multicore Architectures", ICPP '18, Article 34, + [doi:10.1145/3229710.3229720](https://doi.org/10.1145/3229710.3229720) + — the hash accumulator that replaced the heap; reference [2] in + `GB_AxB_saxpy3.c:82-86`. + +**Code** + +- [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) + at `1fd5475` — `Source/mxm/GB_AxB_saxpy3.c:20` (the CSR/CSC + duality), `:62-70` (workspace), + `Source/mxm/GB_AxB_saxpy3_slice_balanced.c:56-99` (the shipped + accumulator choice), `Source/mxm/GB_AxB_saxpy3_flopcount.c:44-69` + (the symbolic phase), `Source/mxm/GB_AxB_dot3.c:126`, `:171` + (C sized to the mask), `Source/include/GB_defaults.h:20` + (`hyper_switch` = 1/16). Walked in + [reading-suitesparse-internals.md](reading-suitesparse-internals.md). +- [LAGraph](https://github.com/GraphBLAS/LAGraph) at `e2539e2` — + `src/algorithm/LAGr_TriangleCount.c:31-37` (the six + formulations), `:43-47` (which wins on which graph). + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:22-31` — the SpGEMM ladder: + 17.1M flops in 279.4 ms at scale 14, 61 Mflop/s, flops/nnz(C) + ≈ 2. Steps 1, 5 and the crossover arithmetic all run on these. +- `topics/20-graphblas/notes.md:41-42` — the hypersparse headline: + 80.4 MB → 1.59 MB, 11,312 µs → 66 µs. Step 7's failure of the + optimality condition, measured. +- `topics/20-graphblas/notes.md:50-51` — the two SPA prediction + rows, sized at 12 bytes per slot. Fill them before you implement. +- `topics/24-graph-algorithms/notes.md:5-7` — RMAT scale 16 max + degree 9,751 against a mean of 27.8. Step 8's 351×. diff --git a/topics/20-graphblas/reading-lagraph.md b/topics/20-graphblas/reading-lagraph.md index e7ea7df..a2c8b67 100644 --- a/topics/20-graphblas/reading-lagraph.md +++ b/topics/20-graphblas/reading-lagraph.md @@ -10,203 +10,687 @@ you the file:line anchors to watch the previous chapters' machinery get exercised end to end. This is also where M20's parity targets come from. +Every anchor below is **LAGraph at commit `e2539e2`** (the pin in +`resources/codebases.md`), quoted with the line numbers the code +occupies at that commit. The pinned tree's `src/algorithm/` +directory contains exactly these algorithms: `LAGr_Betweenness`, +`LAGr_BreadthFirstSearch`, `LAGr_ConnectedComponents`, +`LAGr_PageRank`, `LAGr_PageRankGAP`, +`LAGr_SingleSourceShortestPath`, `LAGr_TriangleCount`, +`LAGraph_TriangleCount`, `LG_BreadthFirstSearch_SSGrB`, +`LG_BreadthFirstSearch_vanilla`, `LG_CC_Boruvka`, `LG_CC_FastSV6` +and `LG_CC_FastSV7` — plus the two BFS templates. If you go looking +for k-truss or Louvain at this pin, they are not there. + ## The problem in one sentence Once traversal is matrix algebra, a whole graph algorithm collapses to a handful of library calls — LAGraph's direction-optimizing BFS -is ~70 lines, and its performance is decided entirely by *which* -call, *which* semiring, and *which* mask each line picks. +does its per-level work in exactly **two** lines of algebra +(`:307` and `:313`) — and its performance is decided entirely by +*which* call, *which* semiring, and *which* mask each line picks. ## The concepts, step by step ### Step 1 — an algorithm is a loop around one line of algebra +> **In:** the GraphBLAS operations from +> [reading-davis-toms19.md](reading-davis-toms19.md). +> **Out:** the four dials — call, semiring, mask, descriptor — that +> every later step turns, and what each one means. + LAGraph's algorithms all have the same skeleton: some scalar bookkeeping, then one GrB call per iteration that does all the -real work. The two workhorse calls are `GrB_vxm` (sparse -row-vector × matrix — a **SpMSpV**, sparse-vector times sparse -matrix, when the vector is sparse) and `GrB_mxv` (matrix × column -vector — a **SpMV**, sparse-matrix times vector). Add a mask to -scope the output and a descriptor flag like `GrB_DESC_RSC` -(Replace the output + use the Structural Complement of the mask — -i.e. "write only where the mask has NO entry, discarding what was -there") and one line expresses "advance the frontier, excluding -visited". Everything the previous chapters built — engine -dispatch, format switching, masks as outer loops — fires inside -that one line, invisibly. Reading LAGraph is learning to see it. - -### Step 2 — the BFS loop: the whole Beamer paper in 40 lines - -The template's loop (template :243-313) reads as a decision -procedure wrapped around one line of algebra per level: - +real work. Four things are chosen at each such call, and they are +the whole vocabulary of this chapter: + +- **The call.** `GrB_vxm` is sparse row-vector × matrix — an + **SpMSpV** (sparse-vector times sparse-matrix) when the vector is + sparse. `GrB_mxv` is matrix × column vector — an **SpMV** + (sparse-matrix times vector). They are the same product in + different index orders, so which one you write decides which + engine runs. +- **The semiring.** A pair (additive monoid, multiplicative + operator). It decides *what data moves*: `PLUS_TIMES` moves + values, `ANY_SECONDI` moves indices, `LAGraph_any_one_bool` moves + nothing but structure. +- **The mask.** A vector or matrix whose *structure* scopes the + output. With the complement flag it means "write only where the + mask has no entry". +- **The descriptor.** `GrB_DESC_RSC` = **R**eplace the output + + use the **S**tructure of the mask (ignore its values) + **C** + omplement it. `Include/GraphBLAS.h:666` in SuiteSparse defines it + as exactly `GrB_REPLACE + GrB_STRUCTURE + GrB_COMP`. So + `GrB_DESC_RSC` on a BFS frontier reads: *write only where the + vertex has not yet been visited, and discard whatever was in the + frontier before.* + +Everything the previous chapters built — engine dispatch, format +switching, masks as outer loops — fires inside that one line, +invisibly. Reading LAGraph is learning to see it. + +Why it matters: the four dials are the entire performance surface. +Nothing else in an LAGraph algorithm costs anything. + +### Step 2 — the BFS level, in two lines + +> **In:** the four dials (Step 1) and the direction switch from +> [reading-beamer-sc12.md](reading-beamer-sc12.md). +> **Out:** the two lines that do all the work, and the format hint +> that decides which engine each one reaches. + +Here is the payload of the whole algorithm — everything else in the +343-line template is bookkeeping around it: + +```c +// LG_BreadthFirstSearch_SSGrB_template.c — the level, 302-314. +// The comment on 305 and 311 is LAGraph's own naming of the engines. + 302 // mask is pi if computing parent, v if computing just level + 303 if (do_push) + 304 { + 305 // push (saxpy-based vxm): q'{!mask} = q'*A + 306 GRB_TRY (LG_SET_FORMAT_HINT (q, LG_SPARSE)) ; + 307 GRB_TRY (GrB_vxm (q, mask, NULL, semiring, q, A, GrB_DESC_RSC)) ; + 308 } + 309 else + 310 { + 311 // pull (dot-product-based mxv): q{!mask} = AT*q + 312 GRB_TRY (LG_SET_FORMAT_HINT (q, LG_BITMAP)) ; + 313 GRB_TRY (GrB_mxv (q, mask, NULL, semiring, AT, q, GrB_DESC_RSC)) ; + 314 } ``` - if push: switching to pull if frontier growing AND - (nq > n/8 OR push-work estimate > unexplored/8) - if pull: switching back if frontier < n/512 - then ONE line does the level: vxm (push) or mxv-on-AT (pull), - mask = complemented visited (DESC_RSC: replace + structural - complement), assign frontier into parent/level vectors (:335-340) + +Read `:306` and `:312` carefully, because this is where a common +claim is wrong. **The frontier's format is not left to +SuiteSparse's auto-conform heuristic — LAGraph sets it +explicitly**, sparse before every push, bitmap before every pull. +That hint is not decoration: it is what makes the engine dispatch +come out right. `GB_AxB_dot2_control.c:26-30` returns "use dot" +immediately if either operand is bitmap or full, so `LG_BITMAP` at +`:312` is what steers the pull into the dot engine. + +And note what the pull does *not* reach. +`GB_AxB_dot3_control` (`GB_mxm.h:235-243`) requires +`M != NULL && !Mask_comp && (M sparse or hypersparse)`. The pull's +descriptor is `GrB_DESC_RSC`, so `Mask_comp` is **true** — dot3 is +structurally ineligible here. A complemented-mask product on a +bitmap operand is dot2's case, which is what +`GB_AxB_dot2_control.c:10` documents. (Guides that say "pull uses +dot3" have skipped the descriptor.) + +The mask itself is chosen once, at `:200`: +`GrB_Vector mask = (compute_parent) ? pi : v` — the output vector +*is* the visited set. There is no separate `visited` bitmap, +because "has a parent assigned" and "has been visited" are the same +predicate. + +Why it matters: two lines, four dials each, and the format hint on +the line *before* the call is what decides which of SuiteSparse's +engines the call lands in. + +### Step 3 — the semiring trick: ANY_SECONDI, and its cheaper sibling + +> **In:** the semiring dial (Step 1). +> **Out:** two semirings, what each moves per level, and the +> algebraic property that lets both skip a comparison. + +`GxB_ANY_SECONDI_INT32/64` computes the parent vector with zero +comparisons: + +```c +// LG_BreadthFirstSearch_SSGrB_template.c — semiring selection, 130-165 + 130 // determine the semiring type + 131 GrB_Type int_type = (n > INT32_MAX) ? GrB_INT64 : GrB_INT32 ; + ... + 135 bool many_expected = (nvals >= n) ; + ... + 138 if (compute_parent) + 139 { + 140 // use the ANY_SECONDI_INT* semiring: either 32 or 64-bit depending on + 141 // the # of nodes in the graph. + 142 semiring = (n > INT32_MAX) ? + 143 GxB_ANY_SECONDI_INT64 : GxB_ANY_SECONDI_INT32 ; + ... + 147 if (many_expected) + 148 { + 149 GRB_TRY (LG_SET_FORMAT_HINT (pi, LG_BITMAP + LG_FULL)) ; + 150 } + ... + 158 else + 159 { + 160 // only the level is needed, use the LAGraph_any_one_bool semiring + 161 semiring = LAGraph_any_one_bool ; ``` -Everything in [reading-beamer-sc12.md](reading-beamer-sc12.md) is -these ~70 lines. The whole loop, transcribed: - -```rust -loop { - // the direction switch wraps ONE line of algebra per level - if push && growing && (nq > n / 8 || push_work > unexplored / 8) { - push = false; // frontier huge → pull - } else if !push && nq < n / 512 { - push = true; // tail → back to push - } - q = if push { - vxm(&q, &visited, AnySecondi, a) // q' = q' * A - } else { - mxv(at, &q, &visited, AnySecondi) // q = AT * q - }; - parent.assign_where(&q); // ANY: any parent will do - nq = q.nvals(); - if nq == 0 { break; } -} +The multiply op **SECONDI** returns the *index* of the second +operand's entry — the parent's id — rather than any stored value. +The additive monoid **ANY** is a reduction allowed to keep +whichever value arrives first; since any parent is a valid BFS +tree, no min, no compare, no tie-break is needed. This is +Gunrock's benign CAS race (topic 18) expressed as algebra rather +than as a data race you argue is harmless: ANY is associative, +commutative and idempotent, so the nondeterminism is +*definitionally* fine. + +The level-only path at `:161` uses a different semiring, and the +name is worth getting right. `LAGraph_any_one_bool` is not +"ANY_PAIR". `include/LAGraph.h:825-829` documents the family: + +> "`LAGraph_any_one_T`: using the `GrB_MIN_MONOID_T` for +> non-boolean types or `GrB_LOR_MONOID_BOOL` for boolean, and the +> `GrB_ONEB_T` multiplicative op. These semirings are very useful +> for unweighted graphs, or for algorithms that operate only on the +> sparsity structure of unweighted graphs." + +So for booleans it is **(OR, true)**. The multiply produces the +constant `true` regardless of operands, which means the matrix's +*values are never read* — only its pattern. That is the +"structure-only" optimization Yang, Buluç & Owens measured at +**1.62×** standalone (ICPP '18, Table 2). + +Price what each semiring moves per level, on this topic's RMAT +scale-18 graph (`notes.md:13`: n = 262,144, nnz = 2.0M): + +``` + inputs: n = 262,144 < INT32_MAX, so :131 and :142 pick the 32-bit forms + a level with nq = 40,000 frontier vertices, mean degree + 2.0e6 / 262,144 = 7.63 + + parent BFS, ANY_SECONDI_INT32: + values written per level = nq_next entries × 4 bytes + values READ from A = 0 (SECONDI reads the index, not A.x) + → A.x need never be touched; only A.p and A.i are streamed + + level BFS, LAGraph_any_one_bool (LOR, ONEB): + values written per level = nq_next entries × 1 byte + values read from A = 0 + → the same saving, and a 4× smaller output vector + + the edges streamed are the same either way: + 40,000 × 7.63 × 4 bytes of column indices = 1.22 MB per level ``` -What it costs to notice: the heuristic's inputs (`push_work`, -`unexplored`) must themselves be cheap — the template maintains -`edges_unexplored` incrementally by subtracting frontier degrees -(:196, :261-277) rather than recomputing a reduction per level. -Question 1 makes you audit every input. - -### Step 3 — the semiring trick: ANY_SECONDI - -`ANY_SECONDI` (:140-143) computes the parent vector with zero -comparisons. The multiply op SECONDI returns the *index of the -second operand's entry* — i.e. the parent's id; the monoid ANY (a -reduction allowed to keep whichever value arrives — any witness is -acceptable) keeps one of them. No min, no compare, no tie-break — -any parent is a valid BFS tree. This is Gunrock's benign CAS race -(topic 18), expressed as algebra instead of as a data race you -argue is harmless. The 32- vs 64-bit variant is chosen by -n > INT32_MAX: the v10 index-size story at the algorithm level. - -Why it matters: the semiring choice moved a correctness argument -(is the race benign?) into the algebra (ANY is associative and -idempotent — the race is *definitionally* fine), and it decides -what data moves — indices, not values. +The semiring choice does not change how many edges you touch. It +changes whether you touch `A.x` at all — and on an unweighted graph +`A.x` can be *iso* (one stored value for the whole matrix), so +touching it is pure waste. + +Why it matters: the semiring moved a correctness argument ("is the +race benign?") into the algebra, and it decides what data moves: +indices, not values. ### Step 4 — triangle counting: six spellings of one mask -LAGr_TriangleCount.c:31-46 — Burkhardt `sum((A²).*A)/6`, Cohen -`sum((L*U).*A)/2`, Sandia_LL `sum((L*L).*L)`, … Sandia_LUT -`sum((L*U').*L)` (L and U are the lower/upper triangles of A). All -compute the same count; they differ ONLY in which mxm engine runs -and how much the mask prunes: - -- `.*L` masks the OUTPUT to the lower triangle — dot3 iterates - only candidate wedges -- L*L vs L*U': saxpy vs dot formulation — the comment (:43-46) - says LUT (dot) usually wins, but LL (saxpy) wins on GAP-urand: - uniform-random degrees flatten the hub problem, exactly the - Gustavson-vs-hash tradeoff -- there's also a presort by degree (relabeling!) that bounds wedge - work — topic 13's "renumber for locality," used for algorithmic - pruning - -The lesson: at this level, "algorithm choice" has become "which -algebraic spelling triggers the best engine for this graph's -degree distribution" — six mathematically equal expressions, six -different cost profiles. +> **In:** the mask dial (Step 1) and the engine split (Step 2). +> **Out:** six algebraically identical expressions with six cost +> profiles, and LAGraph's own statement of which wins where. + +```c +// LAGr_TriangleCount.c — the six formulations, 27-47 (comment block) + 27 // One of 6 methods are used, defined below where L and U are the strictly + 28 // lower and strictly upper triangular parts of the symmetrix matrix A, + 29 // respectively. Each method computes the same result, ntri: + 30 // + 31 // 0: default: use the default method (currently method Sandia_LUT) + 32 // 1: Burkhardt: ntri = sum (sum ((A^2) .* A)) / 6 + 33 // 2: Cohen: ntri = sum (sum ((L * U) .* A)) / 2 + 34 // 3: Sandia_LL: ntri = sum (sum ((L * L) .* L)) + 35 // 4: Sandia_UU: ntri = sum (sum ((U * U) .* U)) + 36 // 5: Sandia_LUT: ntri = sum (sum ((L * U') .* L)). Note that L=U'. + 37 // 6: Sandia_ULT: ntri = sum (sum ((U * L') .* U)). Note that U=L'. + ... + 43 // The Sandia_* methods all tend to be faster than the Burkhardt or Cohen + 44 // methods. For the largest graphs, Sandia_LUT tends to be fastest, except for + 45 // the GAP-urand matrix, where the saxpy-based Sandia_LL method (L*L.*L) is + 46 // fastest. For many small graphs, the saxpy-based Sandia_LL and Sandia_UU + 47 // methods are often faster that the dot-product-based methods. +``` + +All six compute the same count. They differ only in which mxm +engine runs and how much the mask prunes: + +- **`.* L` masks the OUTPUT to the lower triangle.** In + SuiteSparse, a non-complemented sparse mask on `C = A*B'` is + precisely dot3's case (`GB_mxm.h:235-243`), and dot3 allocates + `C` with exactly `nnz(M)` entries — + `GB_AxB_dot3.c:126` computes `mnz = GB_nnz(M)` and `:171` sets + `cnz = mnz`. The mask is not a filter applied afterwards; it is + the loop bound. +- **`L*L` versus `L*U'`** is saxpy versus dot, and the comment at + `:43-47` gives LAGraph's own measured verdict rather than a rule: + LUT (dot) usually wins on the largest graphs, LL (saxpy) wins on + GAP-urand, and LL/UU win on many small graphs. Uniform-random + degrees flatten the hub problem — exactly the Gustavson-vs-hash + tradeoff of + [reading-gustavson-spgemm.md](reading-gustavson-spgemm.md). +- **Burkhardt divides by 6 and Cohen by 2**, because they count + each triangle from every orientation; the Sandia forms divide by + nothing, because the triangular masks already fix an orientation. + That factor is the whole reason the Sandia forms are faster. + +Note what the pinned source does *not* say. The comment claims a +performance ordering; it does not quantify it, and this repo has +not measured triangle counting through GraphBLAS. Do not carry a +"dot3 is 3× faster" number out of this step — carry the ordering +and go measure. (This topic's `notes.md:62` lists masked SpGEMM as +a *stretch* stub for exactly that reason.) + +Why it matters: at this level "algorithm choice" has become "which +algebraic spelling triggers the best engine for this graph's degree +distribution" — six mathematically equal expressions, six cost +profiles, and the library ships all six because none of them wins +everywhere. ### Step 5 — PageRank (GAP variant): no mask, all bandwidth -LAGr_PageRankGAP.c:99-135 — prescale out-degrees by damping once -(:112), then each iteration is `r = teleport; r += AT'*(t/d)` via -one mxv with PLUS_SECOND (:135) + eWise ops. Dense vectors, full -sweep, no early exit: unlike BFS, *every* vertex contributes every -iteration, so there is nothing for masks or sparsity to skip — -PageRank is the SpMV bandwidth benchmark (gb_bench's spmv lane IS -this). It's the algorithm that measures your memory system, not -your cleverness. Note what's absent: GAP PR skips proper -dangling-node handling for speed (:comment near top) — a -benchmark-vs-correctness tension to remember for topic 22. +> **In:** the mask dial (Step 1) and the six-spellings lesson +> (Step 4). +> **Out:** the algorithm with *nothing* to prune, and this topic's +> own measured bandwidth ladder as its cost model. + +```c +// LAGr_PageRankGAP.c — prescale then iterate, 109-142 + 109 // prescale with damping factor, so it isn't done each iteration + 110 // d = d_out / damping ; + 111 GRB_TRY (GrB_Vector_new (&d, GrB_FP32, n)) ; + 112 GRB_TRY (GrB_apply (d, NULL, NULL, GrB_DIV_FP32, d_out, damping, NULL)) ; + ... + 119 GRB_TRY (GrB_eWiseAdd (d, NULL, NULL, GrB_MAX_FP32, d1, d, NULL)) ; + ... + 126 for ((*iters) = 0 ; (*iters) < itermax && rdiff > tol ; (*iters)++) + 127 { + 128 // swap t and r ; now t is the old score + 129 GrB_Vector temp = t ; t = r ; r = temp ; + 130 // w = t ./ d + 131 GRB_TRY (GrB_eWiseMult (w, NULL, NULL, GrB_DIV_FP32, t, d, NULL)) ; + 132 // r = teleport + 133 GRB_TRY (GrB_assign (r, NULL, NULL, teleport, GrB_ALL, n, NULL)) ; + 134 // r += A'*w + 135 GRB_TRY (GrB_mxv (r, NULL, GrB_PLUS_FP32, LAGraph_plus_second_fp32, + 136 AT, w, NULL)) ; + ... + 142 GRB_TRY (GrB_reduce (&rdiff, NULL, GrB_PLUS_MONOID_FP32, t, NULL)) ; + 143 } +``` + +Count the `NULL`s on line 135. The mask argument is `NULL`. The +descriptor is `NULL`. Every vertex contributes every iteration, so +there is nothing for masks or sparsity to skip. PageRank is the +SpMV bandwidth benchmark — `gb_bench`'s spmv lane *is* this +algorithm's inner loop. It measures your memory system, not your +cleverness. + +Which makes this topic's own measured ladder its cost model. Use +`notes.md:9-14` for the per-scale numbers: + +| scale | n | nnz | µs | GB/s | +|---|---|---|---|---| +| 14 | 16K | 120K | 146 | 19.1 | +| 16 | 65K | 495K | 617 | 18.6 | +| 18 | 262K | 2.0M | 2547 | 18.3 | +| 20 | 1.05M | 8.2M | 11958 | 15.8 | + +Now predict a PageRank iteration from first principles and check +it against that table: + +``` + inputs: scale 18 — n = 262,144, nnz = 2.0e6 (notes.md:13) + CSR with 4-byte column indices, 4-byte f32 values + + bytes touched by ONE mxv at :135: + A.p 4 × (n+1) = 1.05 MB + A.i 4 × nnz = 8.00 MB + A.x 4 × nnz = 8.00 MB + w gathers: nnz random 4-byte reads = 8.00 MB (worst case, no reuse) + r writes 4 × n = 1.05 MB + ---------- + compulsory (A.p + A.i + A.x + r) = 18.10 MB + + predicted time at notes.md:13's measured 18.3 GB/s: + 18.10 MB / 18.3 GB/s = 0.99 ms + + measured for the SpMV lane at scale 18: 2547 µs = 2.55 ms + → 2.6× the compulsory-traffic prediction + + the gap IS the gather: notes.md:16-18 attributes the ~16-19 GB/s + (against topic 0/13's ~30 GB/s streaming baseline) to the random + x-gathers, "RMAT colidx sprays across the vector" +``` + +That 2.6× is the thing to remember. A PageRank iteration is not +"one pass over the matrix"; it is one pass over the matrix plus a +random walk through the vector, and on a scale-free graph the +second term dominates. + +One honesty note before you quote a headline number. This topic +reports the SpMV decay two ways: `FINDINGS.md` row 20 says +**20.7 → 12.3 GB/s**, and `notes.md:11-14`'s table says +**19.1 → 15.8 GB/s** over the same scale 14 → 20 span. They are not +the same measurement and this guide does not blend them: the +per-scale arithmetic above uses `notes.md`, and any headline +citation should say `FINDINGS.md:38`. Reconciling the two is real +work someone should do; see the note in the report at the end of +this topic. + +Finally, what the GAP variant deliberately gets wrong — read the +header before you trust the output: + +```c +// LAGr_PageRankGAP.c — the disclaimer, 20-29 + 20 // PageRank for the GAP benchmark (only). Do not use in production. + ... + 24 // ... The GAP specification + 25 // ignores dangling nodes (nodes with no outgoing edges, also called sinks), + 26 // and thus shouldn't be used in production. This method is for the GAP + 27 // benchmark only. See LAGr_PageRank for a method that + 28 // handles sinks correctly. This method does not return a centrality metric + 29 // such that sum(centrality) is 1, if sinks are present. +``` + +A benchmark-vs-correctness tension to remember for topic 22: the +fastest published implementation of an algorithm is sometimes +computing a slightly different function. + +Why it matters: the algorithm with no mask is the one whose runtime +you can predict from bytes, which makes it the only one in this +chapter you can hold the memory system accountable for. ### Step 6 — API design: pull is the caller's bill to pay -The out_degree vector and AT (the transpose, needed for pull — -step 4 of the Beamer chapter) are *optional inputs* to the BFS -template — without them it silently degrades to push-only -(:18-22). The library refuses to decide whether pull's memory -doubling is worth it; the caller does. This transfers directly: -FalkorDB always HAS the transpose (the delta trio keeps M and Mᵀ -in lockstep), so pull is always on the menu — a storage-layer -decision made once, unlocking an algorithm-layer option forever. +> **In:** pull's transpose requirement +> ([reading-beamer-sc12.md](reading-beamer-sc12.md), Step 5). +> **Out:** where LAGraph puts that decision, and what changes when +> the storage layer has already paid it. + +```c +// LG_BreadthFirstSearch_SSGrB_template.c — the optional inputs, 18-22 and 128 + 18 // This is an Advanced algorithm. G->AT and G->out_degree are required for + 19 // this method to use push-pull optimization. If not provided, this method + 20 // defaults to a push-only algorithm, which can be slower. This is not + 21 // user-callable (see LAGr_BreadthFirstSearch instead). G->AT and + 22 // G->out_degree are not computed if not present. + ... + 128 bool push_pull = (Degree != NULL && AT != NULL) ; +``` + +Read `:22` twice: "**are not computed if not present**". The +library will not quietly build a transpose to make your call +faster. One boolean at `:128` and the whole Beamer machinery is +either armed or gone, decided entirely by what the caller handed +in. + +(A footnote on provenance: the file's own reference block at +`:24-32` cites Yang, Buluç & Owens ICPP '18 and *"The GAP Benchmark +Suite", arXiv:1508.03619, 2015* — the latter is a **different** +Beamer/Asanović/Patterson paper from the SC '12 one that +[reading-beamer-sc12.md](reading-beamer-sc12.md) reads. The α/β +machinery comes from SC '12; the graph suite comes from the 2015 +report.) + +`LAGr_PageRankGAP.c:31-33` states the same policy from the other +side, with a shortcut: "The `G->AT` and `G->out_degree` cached +properties must be defined for this method. If G is undirected or +`G->A` is known to have a symmetric structure, then `G->A` is used +instead of `G->AT`." On an undirected graph the transpose is free +because it is the same matrix — which is exactly Beamer §IV's +"performing the bottom-up approach requires no modification to the +graph data structures". + +This transfers directly. FalkorDB always *has* the transpose — the +delta trio keeps a transposed twin in lockstep +(`delta_matrix.h:17-24`, walked in +[reading-falkordb-delta-matrix.md](reading-falkordb-delta-matrix.md)) +— so pull is always on the menu. That is a storage-layer decision +made once, paid for on every write, and it unlocks an +algorithm-layer option forever. + +Why it matters: "optional input" is an architectural choice, not an +API convenience. It puts the memory-doubling decision at the only +layer that knows the workload. ## Where each step lives in the code | anchor | step | what it is | |---|---|---| -| template/LG_BreadthFirstSearch_SSGrB_template.c:184-187 | 2 | α=8, β1=8, β2=512 — the Beamer thresholds | -| …template.c:243-292 | 2 | the push↔pull switch logic (growing/shrinking + thresholds) | -| …template.c:307 | 2 | push: `GrB_vxm(q, mask, …, q, A, GrB_DESC_RSC)` | -| …template.c:313 | 2 | pull: `GrB_mxv(q, mask, …, AT, q, GrB_DESC_RSC)` | -| …template.c:196, 261-277 | 2 | `edges_unexplored` maintained incrementally | -| …template.c:140-143 | 3 | `GxB_ANY_SECONDI_INT{32,64}` — parent BFS with zero comparisons | -| LAGr_TriangleCount.c:31-46 | 4 | all SIX masked-mxm triangle formulations + which wins where | -| LAGr_PageRankGAP.c:99-135 | 5 | GAP-style PR: prescaled degrees, `mxv` + PLUS_SECOND at :135 | -| …template.c:18-22 | 6 | optional AT / out_degree — silent degrade to push-only | -| LG_CC_FastSV7.c | — | connected components via hooking/shortcutting (min-semiring); M24 material | +| `…SSGrB_template.c:18-22`, `:128` | 6 | optional `AT`/`out_degree`; `push_pull` armed by one boolean | +| `…SSGrB_template.c:131`, `:142-143` | 3 | 32- vs 64-bit index type and semiring, by `n > INT32_MAX` | +| `…SSGrB_template.c:135`, `:147-150`, `:173-176` | 3 | `many_expected`; output vectors hinted `LG_BITMAP + LG_FULL` | +| `…SSGrB_template.c:161` | 3 | level-only semiring is `LAGraph_any_one_bool`, **not** ANY_PAIR | +| `…SSGrB_template.c:183-188` | 2 | α = 8, β₁ = 8, β₂ = 512 and the two derived bounds | +| `…SSGrB_template.c:200` | 2 | the mask *is* the output vector: `pi` or `v` | +| `…SSGrB_template.c:243-294` | 2 | the push↔pull switch — three exclusive branches, not one test | +| `…SSGrB_template.c:268-275` | 2 | `edges_unexplored` maintained: masked assign, reduce, subtract | +| `…SSGrB_template.c:306-307` | 2 | push: hint `LG_SPARSE`, then `GrB_vxm(…, GrB_DESC_RSC)` | +| `…SSGrB_template.c:312-313` | 2 | pull: hint `LG_BITMAP`, then `GrB_mxv(…, GrB_DESC_RSC)` | +| `…SSGrB_template.c:335`, `:340` | 2 | `pi{q} = q` and `v{q} = k`, both with `GrB_DESC_S` | +| `LAGr_TriangleCount.c:27-47` | 4 | the six masked-mxm formulations and which wins where | +| `LAGr_PageRankGAP.c:20-29` | 5 | "Do not use in production" — sinks are ignored | +| `LAGr_PageRankGAP.c:112`, `:119` | 5 | prescale `d = d_out/damping`, then `d = max(1/damping, d)` | +| `LAGr_PageRankGAP.c:126-143` | 5 | the iteration: eWiseMult, assign, unmasked `mxv`, reduce | +| `include/LAGraph.h:825-829` | 3 | the `LAGraph_any_one_T` family, documented | +| `LG_CC_FastSV7.c` | — | components via hooking/shortcutting; M24 material | Navigation advice: read the BFS template first, top to bottom — -it's ~70 lines of payload and every line is now familiar. Then -read just the comment block of LAGr_TriangleCount.c (:31-46), then -LAGr_PageRankGAP.c's loop. Leave LG_CC_FastSV7.c until M24. +355 lines of which maybe 70 are payload, and every line is now +familiar. Then read just the comment block of +`LAGr_TriangleCount.c` (`:27-47`), then `LAGr_PageRankGAP.c`'s +loop. Leave `LG_CC_FastSV7.c` until M24. Note the pin ships both +`LG_CC_FastSV6.c` and `LG_CC_FastSV7.c`; read 7. ### What transfers to M20/M24 -- M20's BFS parity target: match the template's switch behavior - with our α/β on LDBC graphs; the per-level trace in gb_bench is +- M20's BFS parity target: match the template's switch behaviour + with our α/β on LDBC graphs; the per-level trace in `gb_bench` is the debugging tool. -- FastSV (LG_CC_FastSV7.c) is M24 material: components via +- FastSV (`LG_CC_FastSV7.c`) is M24 material: components via min-semiring hooking — read after this topic settles. - The "optional AT" API design transfers directly: FalkorDB always - HAS the transpose (delta trio) — so pull is always on the menu, + HAS the transpose (delta trio), so pull is always on the menu, unlike LAGraph's caller-supplied AT. ## Questions for notes.md -1. Read the switch block (:243-292) and list every input the - heuristic consumes. Which are O(1) to maintain and which need - a reduction over the frontier (degree sum — GrB_reduce on a - masked degree vector)? -2. Why does the template keep BOTH `q` sparse and the visited - `mask` as a full vector — what format does q take at the peak - level (SuiteSparse auto-switches it to bitmap — verify via - GxB_print in a scratch C program, or reason from the conform - rules)? -3. Sandia_LUT uses L*U' with U'=L — so it's L*L with the SECOND - operand transposed, turning saxpy into dot. Spell out why dot3 - + lower-triangular mask visits each wedge exactly once. -4. PageRankGAP vs textbook PR: what does prescaling d/damping save - per iteration (one eWise divide over n), and why is the - important-teleport handled as scalar assign not vector add? -5. For M20: our engine's BFS needs parent AND level variants. - Which semiring per variant (ANY_SECONDI vs ANY_PAIR + level - assign), and what does each move per level (indices vs nothing - — iso!)? +1. Read the switch block (`:243-294`) and list every input the + heuristic consumes. Which are O(1) to maintain and which need a + reduction over the frontier (the masked degree assign at + `:268-269` plus the `GrB_reduce` at `:273-274`)? Note that the + three branches consume *different* inputs. +2. What format does `q` take at the peak level — and who decided? + Check `:306` and `:312` before reasoning about SuiteSparse's + conform rules, then say what `GB_conform.c:150`'s switch would + have done if the hint were absent. +3. Sandia_LUT uses `L*U'` with `U' = L` — so it is `L*L` with the + second operand transposed, turning saxpy into dot. Spell out why + dot3 plus a lower-triangular mask visits each wedge exactly once, + using `GB_AxB_dot3.c:126` and `:171` as the evidence. +4. PageRankGAP vs textbook PR: what does prescaling `d/damping` + (`:112`) save per iteration, and why is the teleport handled as + a scalar assign (`:133`) rather than a vector add? +5. For M20: our engine's BFS needs parent AND level variants. Which + semiring per variant (`GxB_ANY_SECONDI_INT32` vs + `LAGraph_any_one_bool`), and what does each move per level + (indices vs nothing — iso!)? ## Done when -- [ ] You can write the BFS loop as one line of algebra inside a while loop. -- [ ] You can explain what `ANY_SECONDI` computes and why that semiring gives you parents for free. -- [ ] You can give more than one masked spelling of triangle counting and say which one LAGraph picks. -- [ ] You can explain why PageRank needs no mask and is therefore pure bandwidth — check against this topic's measured SpMV ladder (20.72 GB/s at scale 14 falling to 12.26 at scale 20). +Answer each before unfolding it. + +- [ ] You can write the BFS level as two lines of algebra and say which dial differs between them. + +
Answer + + `:307` `GrB_vxm(q, mask, NULL, semiring, q, A, GrB_DESC_RSC)` and + `:313` `GrB_mxv(q, mask, NULL, semiring, AT, q, GrB_DESC_RSC)`. + + Same mask, same semiring, same descriptor. Two dials differ: the + **call** (`vxm` vs `mxv`) and the **operand** (`A` vs `AT`). A + third thing differs on the line before — the format hint, + `LG_SPARSE` at `:306` against `LG_BITMAP` at `:312` — and that is + what steers the two calls into different engines. + +
+ +- [ ] You can say who chooses the frontier's format, and name the engine each choice reaches. + +
Answer + + LAGraph chooses, explicitly, on the line before each product: + `LG_SET_FORMAT_HINT(q, LG_SPARSE)` at `:306` and + `LG_SET_FORMAT_HINT(q, LG_BITMAP)` at `:312`. It is not left to + SuiteSparse's auto-conform. + + Bitmap steers pull into the **dot2** engine: + `GB_AxB_dot2_control.c:26-30` returns true immediately when + either operand is bitmap or full. It cannot be dot3 — dot3 + requires a *non*-complemented sparse mask + (`GB_mxm.h:235-243`), and `GrB_DESC_RSC` sets `Mask_comp`. + Sparse steers push into **saxpy3** (`GB_AxB_saxpy3.c`), which is + what LAGraph's own comment at `:305` calls it. + +
+ +- [ ] You can explain what `ANY_SECONDI` computes, name the level-only semiring correctly, and say what each moves. + +
Answer + + `SECONDI` returns the *index* of the second operand's entry — the + parent id — so the product moves indices and never reads `A.x`. + `ANY` keeps whichever witness arrives; since any parent gives a + valid BFS tree, no comparison or tie-break is needed. ANY is + associative, commutative and idempotent, so the nondeterminism is + algebraic rather than a race you have to argue about. `:131` and + `:142-143` pick the 32- or 64-bit form by `n > INT32_MAX`. + + The level-only semiring at `:161` is **`LAGraph_any_one_bool`**, + not ANY_PAIR. `include/LAGraph.h:825-829` documents it as + `GrB_LOR_MONOID_BOOL` with `GrB_ONEB_BOOL` — (OR, true). The + multiply is a constant, so the matrix's values are never read; + only its pattern is. That is Yang's "structure only" optimization, + measured standalone at 1.62× (ICPP '18, Table 2). + +
+ +- [ ] You can give more than one masked spelling of triangle counting, say which LAGraph defaults to, and say what the source does *not* claim. + +
Answer + + Six, at `LAGr_TriangleCount.c:31-37`: Burkhardt + `sum(sum((A²).*A))/6`, Cohen `sum(sum((L*U).*A))/2`, Sandia_LL + `sum(sum((L*L).*L))`, Sandia_UU, Sandia_LUT + `sum(sum((L*U').*L))`, Sandia_ULT. `:31` says the default is + **Sandia_LUT**. + + `:43-47` gives the ordering: Sandia_* beat Burkhardt and Cohen; + LUT (dot) is usually fastest on the largest graphs *except* + GAP-urand, where saxpy-based LL wins; LL and UU are often faster + on many small graphs. + + What the source does *not* give is a ratio. There is no measured + speedup in that comment and this repo has not benchmarked masked + SpGEMM through GraphBLAS — `notes.md:62` lists it as a stretch + stub. Carry the ordering, not a number. + +
+ +- [ ] You can explain why PageRank needs no mask, and predict one iteration's time from bytes. + +
Answer + + Because every vertex contributes every iteration. `:135`'s + `GrB_mxv` passes `NULL` for both the mask and the descriptor — + there is nothing to skip, so there is nothing a mask could scope. + + At scale 18 (`notes.md:13`: n = 262,144, nnz = 2.0e6, 4-byte + indices and f32 values) the compulsory traffic is + A.p 1.05 MB + A.i 8.00 MB + A.x 8.00 MB + r 1.05 MB = 18.10 MB. + At `notes.md:13`'s measured 18.3 GB/s that predicts 0.99 ms; the + lane actually takes 2547 µs — **2.6×** the prediction. The gap is + the gather: `notes.md:16-18` attributes the shortfall against + topic 0/13's ~30 GB/s streaming baseline to the random x-gathers, + because RMAT column indices spray across the vector. + + Note the topic reports the decay two ways — `FINDINGS.md:38` says + 20.7 → 12.3 GB/s, `notes.md:11-14` says 19.1 → 15.8 — and they + should not be blended. Cite whichever you used. + +
+ +- [ ] You can say what LAGraph makes the caller decide, and what changes when the storage layer has already decided it. + +
Answer + + Whether pull exists at all. `:18-22` documents `G->AT` and + `G->out_degree` as required "for this method to use push-pull + optimization", says that without them "this method defaults to a + push-only algorithm, which can be slower", and — the load-bearing + clause — that they "are not computed if not present". `:128` is + the one boolean that arms the machinery. The library will not + spend the memory doubling on the caller's behalf. + + `LAGr_PageRankGAP.c:31-33` adds the shortcut: on an undirected or + structurally symmetric graph, `G->A` is used instead of `G->AT`, + so the transpose costs nothing. + + FalkorDB has already paid: the delta trio maintains a transposed + twin on every write (`delta_matrix.h:17-24`), so pull, and + incoming-edge traversal generally, is always available. The + decision moved from the algorithm's caller to the storage + engine's designer, and became a per-write cost instead of a + per-query one. + +
+ - [ ] You wrote answers to all five questions in notes.md, including which inputs the direction switch actually reads. +
Answer + + The switch's inputs are not one set — they differ per branch, and + that is question 1's real content. `:246` reads `nq` and + `last_nq` (both O(1), maintained at `:320-321`). `:248` reads + `edges_unexplored`, which is only valid until the first pull. + `:261` reads `nq` against the precomputed `n_over_beta1` (`:187`, + computed once). `:268-275` is the expensive one: a masked + `GrB_assign` of the degree vector, then a `GrB_reduce` — O(nq) + work per level, which is why `:275` maintains a running total + rather than recomputing from scratch, and why `:255-260` explains + that after a pull the total is abandoned rather than repaired. + +
+ ## References **Code** -- [LAGraph](https://github.com/GraphBLAS/LAGraph) `src/algorithm/` - — `template/LG_BreadthFirstSearch_SSGrB_template.c` (the whole - Beamer paper in ~70 lines), `LAGr_TriangleCount.c` (:31-46 lists - all six masked formulations), `LAGr_PageRankGAP.c`, - `LG_CC_FastSV7.c` (M24 material — read later) + +- [LAGraph](https://github.com/GraphBLAS/LAGraph) at `e2539e2`, + `src/algorithm/` — `template/LG_BreadthFirstSearch_SSGrB_template.c` + (355 lines; the whole Beamer paper in ~70 of them), + `LAGr_TriangleCount.c` (362 lines; `:27-47` lists all six masked + formulations), `LAGr_PageRankGAP.c` (152 lines), + `LG_CC_FastSV7.c` (M24 material — read later), and + `include/LAGraph.h` for the semiring families. +- [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) + at `1fd5475` — the engine-dispatch anchors this guide points at + (`Source/mxm/GB_AxB_dot2_control.c`, `GB_AxB_dot3.c`, + `GB_mxm.h`) are walked in + [reading-suitesparse-internals.md](reading-suitesparse-internals.md). + +**Papers** + +- Davis — "Parallel GraphBLAS with OpenMP", CSC '20. §4.3 states + the push = `vxm` / pull = `mxv` correspondence these two lines + implement; §3.1 says which engine the library picks for each. +- Yang, Buluç, Owens — "Implementing Push-Pull Efficiently in + GraphBLAS", ICPP '18, + [doi:10.1145/3225058.3225122](https://doi.org/10.1145/3225058.3225122). + Table 2's ablation is the source of the 1.62× attributed to + structure-only in Step 3. + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:9-18` — the SpMV ladder Step 5 does + arithmetic on, and the gather explanation for the shortfall. +- `topics/20-graphblas/notes.md:35` — the BFS scalar oracle, + rmat18 3308 µs. +- `FINDINGS.md:38` — the topic headline. Note it and `notes.md` + report the SpMV decay differently (20.7 → 12.3 versus + 19.1 → 15.8); cite one, do not average them. diff --git a/topics/20-graphblas/reading-openmp-vs-rayon.md b/topics/20-graphblas/reading-openmp-vs-rayon.md index 523a78c..83c5633 100644 --- a/topics/20-graphblas/reading-openmp-vs-rayon.md +++ b/topics/20-graphblas/reading-openmp-vs-rayon.md @@ -9,222 +9,950 @@ concepts first — the skew problem, static slicing, task teams, work-stealing deques — then reads saxpy3's slicing code and rayon's `join` as two answers to the same question. +This is the chapter most at risk of unsourced folklore, so a +standing rule applies below: **every performance number carries its +source**, either a line of pinned code, a table in a paper, or a +measurement in this repo's own `notes.md`. Anything that is a +teaching sketch rather than a measurement is fenced and labelled +`ILLUSTRATION`. Three claims in the previous version of this +chapter were folklore; each is corrected where it lands. + +Anchors are **SuiteSparse:GraphBLAS at `1fd5475`** (version 10.3.1) +and **rayon at `6d9e94b`**, the pins in `resources/codebases.md`. + ## The problem in one sentence -Split one sparse multiply across 8 cores when a power-law graph -puts 1000× more work in some rows than others — divide the rows -into 8 equal-count slices and 7 cores finish early while 1 grinds a -hub row, so "parallel" delivers ~1× speedup. +You cannot know a sparse row's cost without computing something — +so you either measure first and freeze a plan, or split blind and +rebalance by theft, and the two choices have different failure +modes. ## The concepts, step by step -### Step 1 — the skew problem: equal slices aren't equal work +### Step 1 — the skew problem, and how big it actually is + +> **In:** a power-law graph and a row-parallel kernel. +> **Out:** the measured degree skew, and the granularity at which +> it stops mattering. Row-parallel sparse kernels look embarrassingly parallel — every -output row is independent. But a row's cost is its flops (the -count of multiply-adds it actually performs), and on power-law -graphs flops concentrate in **hub** rows: a few rows cost 1000× -the median. Slicing by row *count* therefore produces wildly -unequal slices, and the multiply finishes when the unluckiest -thread does. Every parallel scheduler is an answer to "who does -which slice?" under that skew — and there are exactly two families -of answer: measure the work and slice by cost, or slice lazily and -let idle threads take from busy ones. +output row is independent. But a row's cost is its **flops** (here: +the number of semiring multiply-add pairs it performs, not +floating-point operations — the semiring may be integer or +boolean), and on power-law graphs flops concentrate in **hub** +rows. + +This repo measured the skew rather than assuming it: + +``` + source: topics/24-graph-algorithms/notes.md:5-7 + RMAT scale 16 — n = 65,536, m = 1,819,338 + + max degree 9,751 + mean degree 27.8 + ratio 351× ← the skew, measured + uniform graph max degree 59 ← the control: no skew at all +``` + +Now the part the folklore gets wrong. A 351× skew does *not* mean +"7 cores idle while 1 grinds" — that only follows if the slices are +about one row wide. Cost it at a realistic granularity: + +``` + inputs: SpMV over that RMAT-16 matrix; work per row = its degree + 16 slices (Step 5 shows why 16) + 65,536 / 16 = 4,096 rows per slice + + mean slice work = 4,096 × 27.8 = 113,869 flops + hub-bearing slice = 9,751 + 4,095 × 27.8 = 123,592 flops + imbalance = 123,592 / 113,869 = 1.085 + + ⇒ 8.5% straggler, not 8×. One hub row drowns in 4,095 ordinary + ones. +``` + +So when *does* the skew bite? Two cases, both real: + +1. **Fine granularity.** Slice to 4,096 tasks of 16 rows each and + the hub slice costs 9,751 + 15×27.8 = 10,168 against a mean of + 445 — a **23× straggler**. Skew severity is a function of slice + width, not of the graph alone. +2. **Superlinear per-row work.** In SpGEMM, row i of A·A costs + Σ over i's neighbours of their degree, so the hub's cost is + quadratic-ish in degree. Even so, at 16 slices over the + ~1.28e8 flops extrapolated from `notes.md:22-26`, a hub row + contributing ≥ 9,751 × 27.8 = 271,078 flops is 3.4% of an + 8.0e6-flop slice. Still not fatal. + +Report the negative result honestly: **at the granularities both +schedulers actually choose, degree skew alone is a single-digit-% +effect.** What genuinely destroys parallelism is Step 4's case — +too few columns to slice at all — and that is a *shape* problem, +not a skew problem. + +Why it matters: every parallel scheduler is an answer to "who does +which slice?", and you now know the size of the question at each +granularity instead of repeating a slogan. ### Step 2 — the static answer: cost the work, then freeze the plan -SuiteSparse measures first. The flopcount pre-pass walks the -patterns and produces total and per-column flops; the thread count -and the task list are then *derived* from those numbers and frozen -before any multiply happens: +> **In:** the pattern of A, B, and the mask. +> **Out:** an exact per-column flop vector, a thread count, and a +> frozen task list. +SuiteSparse measures first. The pre-pass walks only *patterns* — +never values — so it is much cheaper than the multiply it plans: + +```c +// GB_AxB_saxpy3_flopcount.c — the complexity claim, 44-48 + 44 // The algorithm scans all nonzeros in B. It only scans at most the min and + 45 // max (first and last) row indices in A and M (if M is present). If A and M + 46 // are not hypersparse, the time taken is O(nnz(B)+n). If all matrices are + 47 // hypersparse, the time is O(nnz(B)*log(h)) where h = max # of vectors present + 48 // in A and M. Assuming B is in standard (not hypersparse) form: +``` + +The pseudocode that follows (`:50-69`) is the whole pre-pass, and +two lines of it carry the design: + +```c +// GB_AxB_saxpy3_flopcount.c — the pre-pass, 54-67 (elided) + 54 for each column j in B: + 55 if (B (:,j) is empty) continue ; + 56 mjnz = nnz (M (:,j)) + 57 if (M is present, not complemented, and M (:,j) is empty) continue ; + ... + 60 for each k where B (k,j) is nonzero: + 61 aknz = nnz (A (:,k)) + 62 if (aknz == 0) continue ; + ... + 66 Bflops (j) += aknz + 67 end ``` - SuiteSparse (plan first, execute statically): - flopcount pass ──► total_flops, per-column flops - │ (GB_AxB_saxpy3_flopcount.c:80; itself - │ parallel: omp schedule(dynamic,1) at :219) - ▼ - nthreads = GB_nthreads(total_flops, chunk, nthreads_max) - │ (slice_balanced.c:418 — tiny job ⇒ 1 thread) - ▼ - slice B into tasks, balanced by flops (:434, :456) - ▼ - #pragma omp parallel — every thread grabs its task list +`:57` is the masked shortcut — an empty mask column means the whole +column of C is skipped, so the mask prunes the *plan*, not just the +output. `:66` is the flop definition: column j's cost is the sum of +`nnz(A(:,k))` over its nonzeros k. Exactly Gustavson's f from +[reading-gustavson-spgemm.md](reading-gustavson-spgemm.md). + +The pre-pass is itself parallel, and note *which* OpenMP schedule: + +```c +// GB_AxB_saxpy3_flopcount.c — the pre-pass parallelises itself, 219-221 + 219 #pragma omp parallel for num_threads(B_nthreads) schedule(dynamic,1) \ + 220 reduction(+:total_Mwork) + 221 for (taskid = 0 ; taskid < B_ntasks ; taskid++) ``` -Note `GB_nthreads`: a tiny multiply gets ONE thread — the -parallelism is *costed* like a query plan, using the same -flopcount that sizes the hash tables. What this buys: zero -scheduling overhead at execution time and a deterministic -schedule. What it costs: the O(nnz)-ish pre-pass runs before every -multiply, profitable only when the multiply is big enough to -repay it. +`schedule(dynamic,1)` over **pre-sliced tasks** — one task handed +out at a time, on demand. That is a mild form of dynamic balancing, +and it is here rather than in the multiply because at pre-pass time +the costs are precisely what is not yet known. Worth remembering +when someone claims SuiteSparse "has no dynamic scheduling". + +Then the plan is derived from the measurement: + +```c +// GB_AxB_saxpy3_slice_balanced.c — the pre-pass and its outputs, 308-311 + 308 GB_OK (GB_AxB_saxpy3_flopcount (&Mwork, Bflops, M, Mask_comp, A, B, Werk)) ; + 309 double total_flops = (double) Bflops [bnvec] ; + 310 double axbflops = total_flops - Mwork ; + 311 GBURBLE ("axbwork %g ", axbflops) ; +``` + +(The previous version of this chapter labelled `:309` "entry". +`:309` reads `total_flops` out of the last cell of `Bflops`; the +call is `:308`, and the flopcount function's own entry point is +`GB_AxB_saxpy3_flopcount.c:80`.) + +```c +// GB_AxB_saxpy3_slice_balanced.c — thread count and task count, 418-420 + 418 (*nthreads) = GB_nthreads (total_flops, chunk, nthreads_max) ; + 419 int ntasks_initial = ((*nthreads) == 1) ? 1 : + 420 (GB_NTASKS_PER_THREAD * (*nthreads)) ; +``` + +```c +// GB_AxB_saxpy3_slice_balanced.c — target task size, 456-459 + 456 double target_task_size = total_flops / ((double) ntasks_initial) ; + 457 target_task_size = GB_IMAX (target_task_size, chunk) ; + 458 double target_fine_size = target_task_size / GB_FINE_WORK ; + 459 target_fine_size = GB_IMAX (target_fine_size, chunk) ; +``` + +Read `:456` carefully: the target is **flops per task**, not +columns per task. Tasks are sized by cost. That is the whole static +philosophy in one line. + +One more line, because the previous version of this chapter cited +it wrongly. `:434` is *not* where B is sliced: + +```c +// GB_AxB_saxpy3_slice_balanced.c — the intensity heuristic, 432-438 + 432 double abnz = GB_nnz (A) + GB_nnz (B) + 1 ; + 433 double workspace = (double) ntasks_initial * (double) cvlen ; + 434 double intensity = total_flops / abnz ; + ... + 437 if (((*nthreads) <= 8 && intensity >= 8 && workspace < abnz) + 438 || ( intensity >= 16 && workspace < abnz)) +``` + +`:434` computes *arithmetic intensity* — flops per input nonzero — +and `:437-438` uses it to force Gustavson for every task when the +multiply is dense enough that dense-accumulator workspace is repaid. +Note `nthreads` appears in the condition at `:437`: **the thread +count feeds back into the algorithm choice**, because +`ntasks_initial` copies of an m-length accumulator is the workspace +bill. Fewer threads, cheaper Gustavson. + +Why it matters: parallelism here is *costed like a query plan*, +using the same flopcount that sizes the hash tables in +[reading-suitesparse-internals.md](reading-suitesparse-internals.md). +It buys a deterministic schedule and zero runtime scheduling +overhead; it costs an O(nnz(B)+n) pass before every multiply. ### Step 3 — coarse and fine tasks: ownership vs teams -Cost-balanced slicing still hits a wall when ONE column's flops -exceed a whole fair share — you can't give half a column to -another thread by slicing columns. saxpy3's escape hatch is a -second task kind: +> **In:** a flop-balanced target task size. +> **Out:** the two task kinds, what each owns, and the memory bill +> for each. + +Cost-balanced slicing hits a wall when ONE column's flops exceed a +whole fair share — you cannot hand half a column to another thread +by slicing columns. saxpy3's escape hatch is a second task kind, +and its header comment is the primary source: + +```c +// GB_AxB_saxpy3.c — the task taxonomy, 22-35 (elided) + 22 // The matrix B is split into two kinds of tasks: coarse and fine. A coarse + 23 // task computes C(:,j1:j2) = A*B(:,j1:j2), for a unique set of vectors j1:j2. + 24 // Those vectors are not shared with any other tasks. A fine task works with a + 25 // team of other fine tasks to compute C(:,j) for a single vector j. Each fine + 26 // task computes A*B(k1:k2,j) for a unique range k1:k2, and sums its results + 27 // into C(:,j) via atomic operations. + ... + 32 // fine Gustavson task + 33 // fine hash task + 34 // coarse Gustason task + 35 // coarse hash task +``` +Four kinds, "then subdivided into 3 variants, for `C=A*B`, `C=A*B`, +and `C=A*B`, giving a total of 12 different types of tasks" +(`:37-38`). And the preference order, stated by the source itself: + +```c +// GB_AxB_saxpy3.c — when fine tasks are the ONLY option, 40-48 + 40 // Fine tasks are used when there would otherwise be too much work for a single + 41 // task to compute the single vector C(:,j). Fine tasks share all of their + 42 // workspace with the team of fine tasks computing C(:,j). Coarse tasks are + 43 // prefered since they require less synchronization, but fine tasks allow for + 44 // better parallelization when B has only a few vectors. If B consists of a + 45 // single vector (for GrB_mxv if A is in CSC format and not transposed, or + 46 // for GrB_vxm if A is in CSR format and not transpose), then the only way to + 47 // get parallelism is via fine tasks. If a single thread is used for this + 48 // case, a single-vector coarse task is used. ``` - B's vectors (columns) → tasks: - coarse task: one thread OWNS whole columns of B - (private workspace, no coordination) - fine task: a TEAM splits one fat column (a hub); - Gustavson workspace shared, atomics coordinate + +`:44-47` is the load-bearing sentence of this whole chapter, and +Step 6 cashes it out into a measured speedup ceiling. + +The memory bill is tabulated in the same comment: + +```c +// GB_AxB_saxpy3.c — the workspace table, 66-70 + 66 // fine Gustavson task (shared): int8_t Hf [m] ; ctype Hx [m] ; + 67 // fine hash task (shared): uint64_t Hf [s] ; ctype Hx [s] ; + 68 // coarse Gustavson task: uint64_t Hf [m] ; ctype Hx [m] ; + 69 // coarse hash task: uint64_t Hf [s] ; ctype Hx [s] ; + 70 // uint64_t Hi [s] ; +``` + +Price it, because this is what `:433`'s `workspace` variable is +counting: + ``` + inputs: C is m × n with m = 1,048,576 (scale 20); ctype = f64 (8 B) + ntasks_initial = GB_NTASKS_PER_THREAD × nthreads (:419-420) + GB_NTASKS_PER_THREAD = 2 (:18) -Coarse tasks are the cheap common case: private workspace, no -atomics. Fine tasks buy load balance on hubs at the price of -atomic operations on the shared accumulator — coordination cost -paid only where skew forces it. This is the static world's -version of "help the overloaded thread": decided up front, from -the flopcount. + coarse Gustavson, per task: 8 B (uint64 Hf) + 8 B (Hx) = 16 B/row + one task : 1,048,576 × 16 B = 16.8 MB + 8 threads × 2 tasks/thread = 16 tasks = 268 MB + + fine Gustavson, SHARED by the team: + 1 B (int8 Hf) + 8 B (Hx) = 9 B/row + one shared copy = 9.4 MB + + ⇒ fine tasks are 28× cheaper in memory here, because the team + shares one accumulator instead of each thread owning one — + and pay for it with the atomics at :27. +``` + +That inversion is the point. Coarse = private workspace, no +coordination, memory × ntasks. Fine = one shared workspace, atomic +updates, memory × 1. Skew and shape decide which you can afford. + +Why it matters: "who owns the workspace?" is the question M20 has +to answer for every kernel, and it is a memory question as much as +a synchronization one. ### Step 4 — the dynamic answer: work stealing -rayon inverts the philosophy: measure nothing, split lazily, and -rebalance by theft. Each worker thread owns a **deque** -(double-ended queue) of pending work; an idle thread **steals** -from the other end of a busy thread's deque. The primitive is -`join(a, b)` — "these two closures *may* run in parallel": +> **In:** no cost model at all. +> **Out:** rayon's `join`, the deque, the steal loop, and what +> "potential parallelism" actually means. + +rayon inverts the philosophy: measure nothing, split lazily, +rebalance by theft. Its own doc states the contract: + +```rust +// rayon-core/src/join/mod.rs — the contract, 17-32 (elided) + 17 /// implementation is quite different and incurs very low + 18 /// overhead. The underlying technique is called "work stealing": the + 19 /// Rayon runtime uses a fixed pool of worker threads and attempts to + 20 /// only execute code in parallel when there are idle CPUs to handle + 21 /// it. + ... + 26 /// participates in the thread pool. It will begin by executing closure + 27 /// A (on the current thread). While it is doing that, it will advertise + 28 /// closure B as being available for other threads to execute. Once closure A + 29 /// has completed, the current thread will try to execute closure B; + 30 /// if however closure B has been stolen, then it will look for other work + 31 /// while waiting for the thief to fully execute closure B. (This is the + 32 /// typical work-stealing strategy). +``` + +"Attempts to only execute code in parallel when there are idle +CPUs" (`:19-21`) is the precise meaning of *potential* parallelism: +`join` is not a spawn. `join` itself is a two-line forwarder: + +```rust +// rayon-core/src/join/mod.rs — join, 93-106 (elided) + 93 pub fn join(oper_a: A, oper_b: B) -> (RA, RB) + ... + 105 join_context(call(oper_a), call(oper_b)) + 106 } +``` + +The scheduler is `join_context`, `:115-173`. Its core: + +```rust +// rayon-core/src/join/mod.rs — the real mechanism, 136-169 (elided) + 136 let job_b = StackJob::new(call_b(oper_b), SpinLatch::new(worker_thread)); + 137 let job_b_ref = job_b.as_job_ref(); + 138 let job_b_id = job_b_ref.id(); + 139 worker_thread.push(job_b_ref); + 140 + 141 // Execute task a; hopefully b gets stolen in the meantime. + 142 let status_a = unwind::halt_unwinding(call_a(oper_a, injected)); + ... + 153 while !job_b.latch.probe() { + 154 let Some(job) = worker_thread.take_local_job() else { + ... + 157 worker_thread.wait_until(&job_b.latch); + 159 break; + 160 }; + 161 if job_b_id == job.id() { + ... + 165 let result_b = job_b.run_inline(injected); + 166 return (result_a, result_b); + 167 } + 168 worker_thread.execute(job); + 169 } +``` + +Four behaviours, each on a line: **push** B onto the local deque +(`:139` — the previous version of this chapter cited `:115` for the +push; `:115` is the function signature), **run** A inline (`:142`), +**pop and run B inline** if nobody took it (`:154`, `:165`), and — +the part people forget — **do other people's work** while waiting +(`:168`), rather than blocking. A `join` where nothing is stolen +costs one deque push and one pop. + +The deque and the thief: +```rust +// rayon-core/src/registry.rs — one deque + one Stealer per worker, 248-257 + 248 let (workers, stealers): (Vec<_>, Vec<_>) = (0..n_threads) + 249 .map(|_| { + 250 let worker = if breadth_first { + 251 Worker::new_fifo() + 252 } else { + 253 Worker::new_lifo() + 254 }; + 255 + 256 let stealer = worker.stealer(); + 257 (worker, stealer) ``` - rayon (split lazily, steal dynamically): - par_iter over rows ──► join(left, right) (join/mod.rs:93) - caller runs left inline, pushes right onto ITS deque (:115) - idle worker steals right (registry.rs:248, Stealer) - each stolen half splits again — recursion IS the scheduler +LIFO by default (`:253`) — the owner pops the most recent push, +which is the depth-first order that keeps the working set hot; +thieves take from the other end, which is the *oldest* and +therefore biggest job. And the theft itself: + +```rust +// rayon-core/src/registry.rs — the steal loop, 886-895 (elided) + 886 loop { + 887 let mut retry = false; + 888 let start = self.rng.next_usize(num_threads); + 889 let job = (start..num_threads) + 890 .chain(0..start) + 891 .filter(move |&i| i != self.index) + 892 .find_map(|victim_index| { + 893 let victim = &thread_infos[victim_index]; + 894 match victim.stealer.steal() { + 895 Steal::Success(job) => Some(job), ``` -rayon's entire scheduler contract fits in one function: +A **random** starting victim (`:888`) then a round-robin sweep +(`:889-891`) — randomization is what stops all idle threads +hammering the same victim. + +Two hazards the docs state outright and which matter for a database +kernel: blocking I/O inside a `join` closure "may be poor" and can +**deadlock** (`join/mod.rs:76-84`), and panics propagate but both +closures always run (`:86-92`). + +Why it matters: no cost model, no pre-pass, and skew handled by +whoever is idle. But now the question is how far it splits — which +is where the third piece of folklore dies. + +### Step 5 — the small-job guard: thief-splitting, not split-to-one + +> **In:** a `par_iter` over rows. +> **Out:** the actual number of leaves rayon creates, and +> SuiteSparse's equivalent floor. + +**The correction.** The previous version of this chapter claimed +that without `with_min_len`, a small multiply "shatters into +thousands of deque pushes". That is false at this pin. rayon does +not split to one element; it uses **thief-splitting**, and the +source says so: ```rust -// join: run `left` inline, PUBLISH `right` for theft — recursion is the scheduler -fn join(left: A, right: B) { - let pending = my_deque.push(right); // ~free if no thread is idle - left(); // the caller does real work NOW - match my_deque.pop(pending) { - Some(right) => right(), // nobody stole it — run it inline - None => { - // an idle worker took `right`; don't block — steal OTHER - // work until it finishes (skewed halves rebalance themselves) - steal_until_done(pending); - } - } -} -// vs saxpy3: nthreads = f(total_flops, chunk); tasks pre-sliced by flops — -// the schedule is COSTED like a query plan, then frozen -``` - -Skew handles itself: a hub row's half gets split again and stolen -again until the work spreads. The flopcount pre-pass becomes -*optional* — RMAT's heavy tail rebalances dynamically. The price: -every potential split pays a deque push, and theft is -nondeterministic (two runs assign rows to threads differently — -question 4 asks when that shows in the *output*). - -### Step 5 — the small-job guard exists in both worlds - -Parallelism has a floor cost, and both schedulers refuse tiny -jobs — they just spell it differently. SuiteSparse: -`GB_nthreads(work, chunk, nthreads_max)` returns 1 when -total_flops is below a chunk — one thread, zero overhead. rayon: -`with_min_len(k)` stops the recursive splitting below k elements — -without it, a 1000×1000 multiply with 5K nonzeros shatters into -thousands of deque pushes that each cost more than the work they -carry (question 3 makes you write it). Same decision — "is this -worth parallelizing?" — made from a cost estimate in one world and -from a per-split granularity floor in the other. - -### Step 6 — the trade in one table +// src/iter/plumbing/mod.rs — the Splitter, 247-283 (elided) + 247 /// Thief-splitting is an adaptive policy that starts by splitting into + 248 /// enough jobs for every worker thread, and then resets itself whenever a + 249 /// job is actually stolen into a different thread. + ... + 252 /// The `splits` tell us approximately how many remaining times we'd + 253 /// like to split this job. We always just divide it by two though, so + 254 /// the effective number of pieces will be `next_power_of_two()`. + ... + 262 splits: crate::current_num_threads(), + ... + 267 fn try_split(&mut self, stolen: bool) -> bool { + ... + 270 if stolen { + 273 self.splits = Ord::max(crate::current_num_threads(), self.splits / 2); + 274 true + 275 } else if splits > 0 { + 277 self.splits /= 2; + 278 true + 279 } else { + 281 false + 282 } +``` + +Trace it, which is the arithmetic this step exists for: + +``` + inputs: current_num_threads() = 8; nothing gets stolen + Splitter::new() ⇒ splits = 8 (:262) + + depth 0: splits 8 > 0 → splits = 4, SPLIT (:277) + depth 1: splits 4 > 0 → splits = 2, SPLIT + depth 2: splits 2 > 0 → splits = 1, SPLIT + depth 3: splits 1 > 0 → splits = 0, SPLIT + depth 4: splits 0 → STOP (:281) + + 4 successful splits along every path ⇒ 2^4 = 16 leaves + + over n = 65,536 rows: 65,536 / 16 = 4,096 rows per leaf + (which is exactly the granularity Step 1 costed) + + versus "split all the way down": 65,536 leaves — 4,096× more + deque traffic than actually happens. +``` + +Theft *resets* the budget (`:273`), which is the adaptive part: a +job that was stolen is evidently in demand, so it is worth +splitting again. Sixteen leaves when nothing is stolen; more only +when the machine proves it needs them. + +`min_len` is a separate, harder floor, and its own doc rebuts the +folklore too: + +```rust +// src/iter/plumbing/mod.rs — min_len defaults to 1, 68-79 + 68 /// The minimum number of items that we will process + 69 /// sequentially. Defaults to 1, which means that we will split + 70 /// all the way down to a single item. This can be raised higher + 71 /// using the [`with_min_len`] method, which will force us to + 72 /// create sequential tasks at a larger granularity. Note that + 73 /// Rayon automatically normally attempts to adjust the size of + 74 /// parallel splits to reduce overhead, so this should not be + 75 /// needed. + ... + 78 fn min_len(&self) -> usize { + 79 1 + 80 } +``` + +`:72-75` — "this should not be needed" — because the Splitter +already caps the leaf count. `LengthSplitter` combines the two: + +```rust +// src/iter/plumbing/mod.rs — LengthSplitter, 308-331 (elided) + 308 fn new(min: usize, max: usize, len: usize) -> LengthSplitter { + 309 let mut splitter = LengthSplitter { + 310 inner: Splitter::new(), + 311 min: Ord::max(min, 1), + 312 }; + ... + 318 let min_splits = len / Ord::max(max, 1); + ... + 329 fn try_split(&mut self, len: usize, stolen: bool) -> bool { + 330 // If splitting wouldn't make us too small, try the inner splitter. + 331 len / 2 >= self.min && self.inner.try_split(stolen) + 332 } +``` + +`:331` is a conjunction: `with_min_len` can only make rayon split +*less*, never more; `with_max_len` raises the floor on splits via +`:318`. + +SuiteSparse's equivalent floor is a single function: + +```c +// Source/omp/include/GB_nthreads.h — the small-job guard, 17-32 (elided) + 17 // If work < 2*chunk, then only one thread is used. + 18 // else if work < 3*chunk, then two threads are used, and so on. + ... + 27 work = GB_IMAX (work, 1) ; + 28 chunk = GB_IMAX (chunk, 1) ; + 29 int64_t nthreads = (int64_t) floor (work / chunk) ; + 30 nthreads = GB_IMIN (nthreads, nthreads_max) ; + 31 nthreads = GB_IMAX (nthreads, 1) ; +``` + +`chunk` defaults to `GB_CHUNK_DEFAULT (64*1024)` = 65,536 +(`GB_defaults.h:24`). Run this repo's own measurements through it: + +``` + inputs: notes.md:22-26 SpGEMM flop counts; chunk = 65,536 + assume nthreads_max = 8 (substitute your core count) + + scale 10: 298,000 flops / 65,536 = 4.5 → floor 4 → min(4,8) = 4 threads + scale 12: 2,270,000 flops / 65,536 = 34.6 → floor 34 → min(34,8) = 8 threads + scale 14: 17,100,000 flops / 65,536 = 261 → floor 261 → min(261,8) = 8 threads + + the single-thread frontier: work < 2 × 65,536 = 131,072 flops + at notes.md:28-31's ~15 ns/flop that is ~1.97 ms of work + before GraphBLAS will even use a SECOND thread. +``` + +Two readings. First, the topic's own smallest bench (scale 10) gets +**half the machine**, by design — the guard is deliberately +conservative. Second, `GB_nthreads` saturates at scale 12; from +there on the thread count is pinned at `nthreads_max` and all the +remaining tuning happens in task *sizing* (`:456`), not in thread +count. + +Why it matters: both worlds refuse tiny jobs — one from a cost +estimate, one from an adaptive split budget — and both floors are +higher than intuition suggests. + +### Step 6 — the trade, and the one measured ceiling + +> **In:** both schedulers, understood. +> **Out:** the axis-by-axis comparison, plus the one published +> measurement that tells you which kernels will disappoint. | axis | static (SuiteSparse) | stealing (rayon) | |---|---|---| -| needs a cost model | yes (flopcount) | no | -| skew response | pre-balanced or fine-task atomics | automatic | -| per-task overhead | ~zero at runtime | deque push + potential steal | -| determinism of schedule | high | none | -| lines of scheduler code you own | many | zero (but tune min_len) | - -Two operational wrinkles to carry into M20. Determinism: with -floating-point ⊕ (addition isn't associative in floats), -schedule-dependent combination order means run-to-run output -wobble — static schedules sidestep the question, stealing must -answer it (question 4). And the FFI trap: no native-Rust GraphBLAS -exists — `rustgraphblas` and `graphblas_sparse_linear_algebra` are -FFI bindings over SuiteSparse, so your process ends up with TWO -thread pools (SuiteSparse's OpenMP + your rayon), both sized to -num_cpus — question 5. A pure-Rust kernel core — M20 — -parallelizes with rayon and must answer saxpy3's questions itself: -when is one thread right, and who owns the workspace? +| needs a cost model | yes — `flopcount`, O(nnz(B)+n) (`flopcount.c:44-48`) | no | +| skew response | pre-balanced by flops (`:456`), fine-task atomics for hubs (`saxpy3.c:22-27`) | theft resets the split budget (`plumbing/mod.rs:273`) | +| leaf count | `GB_NTASKS_PER_THREAD × nthreads` = 2 × nthreads (`:18`, `:419-420`) | ~`next_power_of_two(nthreads)` (`:254`, `:262`) | +| per-task overhead | ~zero at runtime; one pre-pass per multiply | one deque push + pop per `join` (`join/mod.rs:139`, `:154`) | +| small-job guard | `GB_nthreads`, work < 2·chunk ⇒ 1 thread | Splitter budget; `with_min_len` as a hard floor | +| determinism of schedule | high — same inputs, same tasks | none — random victim (`registry.rs:888`) | +| scheduler code you own | a lot | none (but tune `min_len`) | + +Now the number that matters, and it is published rather than +folklore. Davis, "Parallel GraphBLAS with OpenMP" (CSC '20), §5, +Table 2 — Intel Xeon E5-2698 v4, 20 cores / 40 hardware threads, +**speedup at 40 threads relative to 1 thread**: + +| kernel | datagen-8_9-fb | cit-Patents | g-1073643522 | graph500-scale25-ef16 | MAWI | +|---|---|---|---|---|---| +| Triangle Counting | 26.6 | 16.1 | 11.2 | 30.5 | 5.8 | +| 4-Truss | 27.7 | 19.7 | 16.6 | * | 13.4 | +| LCC | 25.8 | 11.7 | 8.4 | 30.2 | 5.7 | +| Bellman-Ford | 11.6 | 9.1 | 5.2 | 9.5 | 2.4 | +| **BFS** | **3.5** | **2.6** | **3.6** | **3.9** | **9.7** | + +(`*` = not reported for that pair.) The paper's own explanation, +§5: "Breadth-first search and Bellman-Ford both show modest +parallelism; they both rely on a matrix-vector or vector-matrix +multiply, which is harder to parallelize." + +Join that to Step 3's `saxpy3.c:44-47` and you have mechanism plus +measurement: + +``` + a matrix-VECTOR multiply means B has ONE column. + + coarse tasks own "a unique set of vectors j1:j2" (saxpy3.c:23) + with n = 1 there is exactly one vector + ⇒ at most ONE coarse task ⇒ speedup 1× + + "If B consists of a single vector ... then the only way to get + parallelism is via fine tasks." (saxpy3.c:44-47) + fine tasks sum into C(:,j) "via atomic operations" (:27) + + ⇒ the entire parallel speedup of an SpMV is bought with atomics + on one shared accumulator, which is why CSC'20 Table 2 shows + BFS at 2.6-3.9× where triangle counting reaches 11-30×. +``` + +That is a 4-10× gap between kernels *in the same library on the +same machine*, caused by operand shape rather than by scheduler +quality. M20's kernel list is dominated by SpMV and SpMSpV, so plan +for the low end of that range. + +Two more wrinkles to carry forward. + +**Determinism.** With a floating-point ⊕, addition is not +associative, so a schedule-dependent combination order gives +run-to-run output wobble. A frozen schedule sidesteps the question; +theft cannot. Note this is a property of the *monoid*, not of +rayon: `GxB_ANY_*` and `GrB_LOR` are indifferent to order, and the +`ANY_SECONDI` semiring of +[reading-lagraph.md](reading-lagraph.md) is nondeterministic *by +design*. Question 4 is about which of your semirings care. + +**The two-pool trap.** If your Rust process links a GraphBLAS +through FFI, you get SuiteSparse's OpenMP pool *and* your rayon +pool, both sized to the core count, and a rayon task calling +`GrB_mxm` oversubscribes. This chapter did **not** verify the +current state of the Rust GraphBLAS crates — question 5 asks you to +check the crate you actually intend to use before relying on any of +this. A pure-Rust kernel core — M20 — dodges the trap entirely and +inherits saxpy3's questions instead: when is one thread right, and +who owns the workspace? + +Why it matters: the scheduler is not the bottleneck you think it +is. Operand shape is. ## Where each step lives in the code -What to read, in order: - -1. `GB_AxB_saxpy3.c:22-48` (steps 2-3) — the header comment is a - scheduling essay: coarse/fine taxonomy, Gustavson-vs-hash per - task. -2. `GB_AxB_saxpy3_slice_balanced.c:309` (entry), :418 (nthreads - from flops — steps 2, 5), :456 (target task size). Note what is - *not* here: no dynamic load balancing at execution time. -3. `GB_AxB_saxpy3_flopcount.c:80` (step 2) — exact flops per column - of B, cheap because it only walks pattern, not values (and - itself parallel: `omp schedule(dynamic,1)` at :219). -4. rayon `join/mod.rs:93-140` (step 4) — `join_context`: inline + - push + steal-back. The "potential parallelism" framing: `join` - costs ~nothing when no thread is idle. -5. `registry.rs:10-60, :248` (step 4) — one `Worker` deque per - thread, `Stealer` handles crossed between them; the sleep/wake - protocol is why idle rayon threads don't spin. - -## Questions - -1. saxpy3's flopcount pass costs O(nnz(B) + flops-pattern-walk) - before any multiply happens. For which matrix shapes is that - pre-pass a bad deal, and what does rayon do instead of paying it? -2. Fine tasks share one Gustavson workspace with atomics. What is - the rayon-idiomatic equivalent for one fat row — and why does - "split the row, each half gets its own SPA, merge after" change - the memory bill? +| anchor | step | what it is | +|---|---|---| +| `GB_AxB_saxpy3_flopcount.c:44-48` | 2 | the pre-pass complexity: O(nnz(B)+n), or O(nnz(B)·log h) hypersparse | +| `GB_AxB_saxpy3_flopcount.c:50-69` | 2 | the pre-pass in pseudocode; `:57` mask pruning, `:66` the flop definition | +| `GB_AxB_saxpy3_flopcount.c:80` | 2 | the function's actual entry point | +| `GB_AxB_saxpy3_flopcount.c:219-221` | 2 | `schedule(dynamic,1)` — the one dynamic schedule in the path | +| `GB_AxB_saxpy3_slice_balanced.c:308-310` | 2 | the flopcount call; `:309` is `total_flops`, `:310` is `axbflops` | +| `GB_AxB_saxpy3_slice_balanced.c:18` | 3, 5 | `GB_NTASKS_PER_THREAD 2` — two tasks per thread, not 32 as in `GB_AxB_dot2.c:233` | +| `GB_AxB_saxpy3_slice_balanced.c:418-420` | 2, 5 | `GB_nthreads(total_flops, …)` and `ntasks_initial` | +| `GB_AxB_saxpy3_slice_balanced.c:432-438` | 2 | `intensity = total_flops/abnz` — **not** where B is sliced | +| `GB_AxB_saxpy3_slice_balanced.c:456-459` | 2 | `target_task_size` in **flops**, and `target_fine_size` | +| `GB_AxB_saxpy3.c:22-38` | 3 | coarse vs fine, 4 kinds × 3 variants = 12 task types | +| `GB_AxB_saxpy3.c:40-48` | 3, 6 | **why a single-vector B can only parallelize via fine tasks** | +| `GB_AxB_saxpy3.c:62-70` | 3 | the workspace table — 9 B/row shared vs 16 B/row per task | +| `Source/omp/include/GB_nthreads.h:17-32` | 5 | `clamp(floor(work/chunk), 1, nthreads_max)` | +| `Source/include/GB_defaults.h:24` | 5 | `GB_CHUNK_DEFAULT (64*1024)` = 65,536 | +| `rayon-core/src/join/mod.rs:17-32` | 4 | the work-stealing contract in rayon's own words | +| `rayon-core/src/join/mod.rs:76-92` | 4 | blocking-I/O deadlock warning; panic semantics | +| `rayon-core/src/join/mod.rs:93-106` | 4 | `join` — a forwarder to `join_context` at `:105` | +| `rayon-core/src/join/mod.rs:115-173` | 4 | `join_context`: **push at `:139`**, run A `:142`, steal-back `:153-169` | +| `rayon-core/src/registry.rs:248-257` | 4 | one LIFO `Worker` deque + one `Stealer` per thread | +| `rayon-core/src/registry.rs:875-905` | 4 | the steal loop: random victim `:888`, round-robin `:889-891` | +| `src/iter/plumbing/mod.rs:68-80` | 5 | `min_len` defaults to 1; `:72-75` says you normally should not need it | +| `src/iter/plumbing/mod.rs:246-284` | 5 | **thief-splitting** — the split budget that caps the leaf count | +| `src/iter/plumbing/mod.rs:286-333` | 5 | `LengthSplitter`; `:331` is a conjunction — `min_len` only reduces splitting | +| `src/iter/mod.rs:3152-3176` | 5 | `with_min_len` public doc | + +Read in this order: `GB_AxB_saxpy3.c:20-86` first (the header +comment is a scheduling essay and it is the best-written thing in +the directory), then `GB_AxB_saxpy3_flopcount.c:40-69`, then +`GB_AxB_saxpy3_slice_balanced.c:300-470`. Then rayon's +`join/mod.rs` end to end (186 lines), then `plumbing/mod.rs:246-333` +— which is the file that tells you what rayon *actually* does, and +where the folklore usually goes wrong. + +### What transfers to M20 + +Four kernels, four decisions, and Step 6 says which will +disappoint. SpMV and SpMSpV are the single-vector case +(`saxpy3.c:44-47`) — expect the CSC'20 BFS row, 2.6-3.9× at 40 +threads, and design the accumulator with that in mind. Masked +dot-SpGEMM and the `delta_mxm` fold have many output columns and +should behave like the triangle-counting row. For each: what axis +does `par_iter` run over, is a flopcount-style pre-pass worth its +O(nnz(B)+n), and who owns the workspace — a private 16 B/row +accumulator per task, or one shared 9 B/row accumulator with +atomics? + +## Questions to answer in notes.md + +1. saxpy3's flopcount pass costs O(nnz(B) + n) + (`GB_AxB_saxpy3_flopcount.c:44-48`) before any multiply happens. + For which matrix shapes is that pre-pass a bad deal, and what + does rayon do instead of paying it? Use Step 5's arithmetic: a + multiply below 131,072 flops gets one thread anyway, so what is + the pre-pass buying there? +2. Fine tasks share one Gustavson workspace with atomics + (`GB_AxB_saxpy3.c:27`, `:66`). What is the rayon-idiomatic + equivalent for one fat row — and why does "split the row, each + half gets its own SPA, merge after" change the memory bill? + Step 3 prices both: 9 B/row shared against 16 B/row × ntasks. 3. `GB_nthreads(work, chunk, nthreads_max)` returns 1 for small - work. Write the rayon equivalent — where does `with_min_len` - go, and what happens if you omit it on a 1000×1000 multiply - with 5K nonzeros? + work. Write the rayon equivalent — where does `with_min_len` go, + and what actually happens if you omit it on a 1000×1000 multiply + with 5K nonzeros? Predict the leaf count from + `plumbing/mod.rs:262` and `:277` *before* you measure it, then + check whether `with_min_len` changes anything at all + (`:331` is a conjunction). 4. Work-stealing is nondeterministic: two runs assign rows to - threads differently. Which GraphBLAS semirings make that - visible in the OUTPUT (hint: floating-point ⊕), and how does - SuiteSparse's static schedule sidestep the question? -5. rustgraphblas-style FFI bindings inherit SuiteSparse's OpenMP - pool; your Rust process also has a rayon pool. What goes wrong - when both are sized to num_cpus and a rayon task calls GrB_mxm? + threads differently (`registry.rs:888`). Which GraphBLAS + semirings make that visible in the OUTPUT (hint: floating-point + ⊕ is not associative; `GxB_ANY_*` and `GrB_LOR` are indifferent), + and how does SuiteSparse's static schedule sidestep the question? +5. FFI bindings to SuiteSparse inherit its OpenMP pool; your Rust + process also has a rayon pool. Check the crate you would + actually use — is it FFI or pure Rust? — and work out what goes + wrong when both pools are sized to `num_cpus` and a rayon task + calls `GrB_mxm`. This chapter deliberately asserts nothing about + the current crate ecosystem. 6. **M20 mapping**: pick the M20 kernel list (SpMV, SpMSpV, masked dot-SpGEMM, delta_mxm fold). For each, decide: par_iter over what axis, does it need a flopcount-style pre-pass, and who owns - the workspace? Write the four decisions in notes.md — that's the + the workspace? Write the four decisions in notes.md, with the + speedup you expect from CSC'20 Table 2's rows — that is the checklist item. ## Done when -- [ ] You can state the skew problem: equal slices are not equal work, and connect it to the RMAT max degree of 9751 measured in topic 24. -- [ ] You can explain the static answer (cost the work, freeze the plan) and what the flopcount pre-pass costs. -- [ ] You can explain work stealing and name what it gives up (determinism). -- [ ] You can say why both worlds need a small-job guard and what `GB_nthreads` does with a small `work`. -- [ ] You can fill in the trade table from memory. -- [ ] You wrote answers to all five questions in notes.md, including your M20 kernel list. +Answer each before unfolding it. + +- [ ] You can state the skew problem *and* say at what granularity it stops mattering, using the RMAT numbers measured in topic 24. + +
Answer + + `topics/24-graph-algorithms/notes.md:5-7`: RMAT scale 16 has + n = 65,536, m = 1,819,338, max degree **9,751**, mean degree + **27.8** — a 351× ratio. The uniform control graph's max degree is + 59, i.e. no skew. + + But at the granularity schedulers actually use, that skew is + small. Sixteen slices of 4,096 rows: the hub-bearing slice costs + 9,751 + 4,095×27.8 = 123,592 against a mean of 113,869 — an + **8.5%** straggler, not 8×. Cut to 4,096 slices of 16 rows and the + same hub gives 10,168 against 445, a **23×** straggler. + + So skew severity is a function of slice width. The thing that + really kills SpMV parallelism is operand shape (Step 6), not + degree skew. + +
+ +- [ ] You can explain the static answer and say what the flopcount pre-pass costs and what it returns. + +
Answer + + `GB_AxB_saxpy3_flopcount.c:44-48`: O(nnz(B)+n) when A and M are + not hypersparse, O(nnz(B)·log h) when they are. It walks patterns + only, never values, and it prunes on the mask (`:57`) so an empty + mask column skips the whole output column. + + It returns `Bflops`, the exact per-column flop vector, whose last + cell is `total_flops` (`slice_balanced.c:308-309`). Everything + downstream is derived: thread count via `GB_nthreads` at `:418`, + `ntasks_initial` at `:419-420`, the Gustavson-vs-hash intensity + test at `:432-438`, and `target_task_size = total_flops / + ntasks_initial` at `:456` — **flops per task, not columns per + task**. + + The pre-pass is itself parallel, with `schedule(dynamic,1)` at + `:219-221` — the one dynamic OpenMP schedule in the whole path, + and it is there precisely because at that moment the costs are + what is not yet known. + +
+ +- [ ] You can explain coarse vs fine tasks and price their workspace. + +
Answer + + `GB_AxB_saxpy3.c:22-27`: a coarse task owns a unique set of + columns of B outright; a fine task joins a **team** computing one + column, each member taking a range k1:k2 and summing into C(:,j) + "via atomic operations". Four kinds (coarse/fine × Gustavson/hash) + × 3 mask variants = 12 task types (`:30-38`). Coarse is preferred + "since they require less synchronization" (`:42-43`). + + Workspace, from the table at `:66-70`: coarse Gustavson is + `uint64_t Hf[m] + ctype Hx[m]` = 16 B/row **per task**; fine + Gustavson is `int8_t Hf[m] + ctype Hx[m]` = 9 B/row **shared by + the team**. At m = 1,048,576 with f64 values and 16 tasks, that is + 268 MB against 9.4 MB — a 28× memory inversion, paid for with + atomics. + + That workspace total is what `slice_balanced.c:433` computes and + `:437-438` tests against `nnz(A)+nnz(B)`. + +
+ +- [ ] You can explain work stealing from rayon's code, naming the four things `join_context` does and what it gives up. + +
Answer + + From `rayon-core/src/join/mod.rs:115-173`: **push** job B onto the + local deque (`:139`), **run** A inline on the calling thread + (`:142`), then either **pop and run B inline** if nobody took it + (`:154`, `:165`) or, if it was stolen, **execute other people's + jobs** while waiting (`:168`) rather than blocking. `join` itself + (`:93-106`) is a forwarder to `join_context` at `:105`. + + Backing it: one LIFO `Worker` deque plus one `Stealer` per thread + (`registry.rs:248-257`), and a steal loop that picks a **random** + starting victim (`:888`) then sweeps round-robin (`:889-891`). + + What it gives up: schedule determinism. What it demands: no + blocking I/O inside a closure — rayon's own doc says that can + deadlock (`join/mod.rs:76-84`). + +
+ +- [ ] You can say how many leaves rayon actually creates, and why the "splits all the way down" story is wrong. + +
Answer + + It is **thief-splitting** (`src/iter/plumbing/mod.rs:246-284`). + `Splitter::new()` sets `splits = current_num_threads()` (`:262`); + each unstolen split halves the budget (`:277`) and stops at zero + (`:281`); a **stolen** job resets the budget to the thread count + (`:273`), which is the adaptive part. + + With 8 threads and no theft: 8 → 4 → 2 → 1 → 0, four successful + splits per path, **2⁴ = 16 leaves** — 4,096 rows each over a + 65,536-row matrix. Not thousands of deque pushes; sixteen. The + comment at `:252-254` says it directly: "the effective number of + pieces will be `next_power_of_two()`". + + `min_len` defaults to 1 (`:78-80`) but its own doc says raising it + "should not be needed" because rayon already adjusts split size + (`:72-75`), and `LengthSplitter::try_split` is a conjunction + (`:331`) — `with_min_len` can only make rayon split *less*. + +
+ +- [ ] You can say what `GB_nthreads` does with each of this topic's three measured SpGEMM workloads. + +
Answer + + `Source/omp/include/GB_nthreads.h:27-31` is + `clamp(floor(work/chunk), 1, nthreads_max)` with chunk = 65,536 + (`GB_defaults.h:24`). Against `notes.md:22-26`, assuming 8 cores: + + - scale 10, 298K flops → floor(4.5) = **4 threads** (half the + machine, by design) + - scale 12, 2.27M flops → floor(34.6) = 34, clamped to **8** + - scale 14, 17.1M flops → floor(261) = 261, clamped to **8** + + The one-thread frontier is work < 2·chunk = 131,072 flops + (`:17-18`), which at `notes.md:28-31`'s ~15 ns/flop is about + **1.97 ms** of work before a second thread is allowed. From scale + 12 on, the thread count is pinned and all remaining tuning is task + *sizing* at `slice_balanced.c:456`. + +
+ +- [ ] You can fill in the trade table, and name the one *measured* reason an SpMV-heavy kernel list will disappoint. + +
Answer + + The table is in Step 6; the key rows are cost model (yes/no), leaf + count (`GB_NTASKS_PER_THREAD × nthreads` vs + `next_power_of_two(nthreads)`), and determinism (high vs none). + + The measured reason: Davis, CSC '20, §5 Table 2 (Xeon E5-2698 v4, + 40 threads vs 1) gives **BFS 2.6-3.9×** across four of five + datasets while triangle counting reaches **11.2-30.5×** and + 4-Truss **13.4-27.7×**. §5: "Breadth-first search and Bellman-Ford + both show modest parallelism; they both rely on a matrix-vector or + vector-matrix multiply, which is harder to parallelize." + + The mechanism is `GB_AxB_saxpy3.c:44-47`: a matrix-vector multiply + means B has one column, coarse tasks own whole columns (`:23`), so + there is at most one coarse task — "the only way to get + parallelism is via fine tasks", which sum into a shared + accumulator with atomics (`:27`). Shape, not scheduler quality. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including your M20 kernel list with an expected speedup per kernel. + +
Answer + + Four kernels, four decisions — axis, pre-pass, workspace owner — + and one expected speedup each, taken from the CSC'20 row that + matches the kernel's shape: SpMV and SpMSpV are single-vector, so + the BFS row (2.6-3.9× at 40 threads); masked dot-SpGEMM and the + `delta_mxm` fold have many output columns, so the + triangle-counting row (11-30×). + + Be explicit about which numbers are yours and which are Davis's: + the CSC'20 figures are 40-thread speedups on a 20-core Xeon and do + **not** transfer to the M3 Pro in `notes.md`. Use them as a + ranking of kernels, not as a prediction of your own wall-clock. + +
## References +**Papers** + +- Davis, T. A. — "Parallel GraphBLAS with OpenMP", CSC '20 + (SIAM Workshop on Combinatorial Scientific Computing). The + citable source for every parallel-speedup claim in this chapter: + §3 dates the parallel version ("Version 3.0.1 has been released + (July 31, 2019), with exploitation of multi-threaded parallelism + expressed through OpenMP"), §3.1 explains the + one-task-per-thread slicing, §5 and Table 2 give the per-kernel + 40-thread speedups and the sentence explaining why BFS lags. +- Davis, T. A. — "Algorithm 1000: SuiteSparse:GraphBLAS", ACM TOMS + 45(4), Article 44, 2019, doi:10.1145/3322125. **Do not cite this + one for parallelism**: it describes version 2.3.3, which §4.2.1 + and §7 state is "not yet multi-threaded" and "an efficient and + highly optimized single-threaded implementation". Read in + [reading-davis-toms19.md](reading-davis-toms19.md). + **Code** + - [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) - `Source/mxm/GB_AxB_saxpy3.c` (the header comment is a scheduling - essay), `GB_AxB_saxpy3_slice_balanced.c`, - `GB_AxB_saxpy3_flopcount.c` -- [rayon](https://github.com/rayon-rs/rayon) - `rayon-core/src/join/mod.rs` (:93 `join_context`), - `rayon-core/src/registry.rs` (:248 — one deque + `Stealer` per - worker) + at `1fd5475` — `Source/mxm/GB_AxB_saxpy3.c:20-86` (the header + comment is a scheduling essay and the best entry point), + `Source/mxm/GB_AxB_saxpy3_slice_balanced.c:300-470`, + `Source/mxm/GB_AxB_saxpy3_flopcount.c:40-69` and `:219-221`, + `Source/omp/include/GB_nthreads.h`, + `Source/include/GB_defaults.h:24`. Full anchor table above. +- [rayon](https://github.com/rayon-rs/rayon) at `6d9e94b` — + `rayon-core/src/join/mod.rs` (186 lines; `join` `:93`, + `join_context` `:115-173`, the push at `:139`), + `rayon-core/src/registry.rs:248-257` and `:875-905`, + `src/iter/plumbing/mod.rs:246-333` (thief-splitting — read this + before believing anything about how far rayon splits), + `src/iter/mod.rs:3152-3176`. + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:22-31` — SpGEMM flop counts at + scales 10/12/14 and the ~15 ns/flop rate. Step 5's `GB_nthreads` + arithmetic runs on them. +- `topics/24-graph-algorithms/notes.md:5-7` — RMAT scale 16 degree + distribution. Step 1's skew arithmetic runs on it. diff --git a/topics/20-graphblas/reading-suitesparse-internals.md b/topics/20-graphblas/reading-suitesparse-internals.md index 6ecd2c4..774854a 100644 --- a/topics/20-graphblas/reading-suitesparse-internals.md +++ b/topics/20-graphblas/reading-suitesparse-internals.md @@ -8,224 +8,1000 @@ builds the concepts each file implements, step by step, then hands you the anchors into `Source/` of the SuiteSparse:GraphBLAS repo to watch them happen. +Every anchor below is **SuiteSparse:GraphBLAS at commit +`1fd5475`** (the pin in `resources/codebases.md`), quoted with the +line numbers the code occupies at that commit. Two of the numbers +most often repeated about this library — the bitmap threshold and +the `m/16` hash rule — are *stale documentation* at this pin, and +this chapter says where the shipped code actually decides. + ## The problem in one sentence -Inside one `GrB_mxm`, per-column costs can vary 1000× (power-law -hub columns), the right accumulator depends on output density, and -the right *format* for the result depends on how dense it came out -— so the library must make cost-based decisions per matrix, per -multiply, and per task, and every one of those decisions is a -readable number in the code. +Inside one `GrB_mxm`, per-column costs can vary by three orders of +magnitude (this repo's own RMAT scale-16 graph has a max degree of +**9,751** against a mean of **27.8** — +`topics/24-graph-algorithms/notes.md:5-7`), the right accumulator +depends on output density, and the right *format* for the result +depends on how dense it came out — so the library must make +cost-based decisions per matrix, per multiply, and per task, and +every one of those decisions is a readable number in the code. ## The concepts, step by step -### Step 1 — format switching is a bitmask + two floats +### Step 1 — the four formats, and the bitmask that gates them + +> **In:** nothing; this fixes the vocabulary the rest of the +> chapter switches between. +> **Out:** the four data formats, and where a matrix records which +> of them it is allowed to become. + +SuiteSparse stores a matrix in one of **four** formats, not two: + +| format | what it is | index space | +|---|---|---| +| **full** | every entry present; just the values array | none needed | +| **bitmap** | values array of size m·n plus a byte-per-cell present flag | none needed | +| **sparse** | CSR/CSC: a pointer array `A.p` of length n+1, plus `A.i`, `A.x` | O(n + nnz) | +| **hypersparse** | `A.p` itself becomes sparse: `A.h` lists the non-empty vectors | O(nnz) | + +Davis's TOMS '19 paper (§4.1) describes only the last two — at +version 2.3.3 "a GraphBLAS matrix is stored in one of four +different formats: compressed-sparse column (standard CSC), +compressed-sparse row (standard CSR), and hypersparse versions of +these two". **Bitmap and full postdate that paper.** Any claim +about them has to come from the source or from the later parallel +paper; a guide that cites TOMS '19 for the bitmap format is citing +the wrong document. + +Each matrix carries a `sparsity_control` bitmask saying which of +the four it is *allowed* to be, plus two floats (`hyper_switch`, +`bitmap_switch`) saying *when* to move. `GB_conform` runs at the +end of every operation and dispatches on the bitmask — +`GB_conform.c:150` switches over +`GB_sparsity_control(A->sparsity_control, A->vdim)` into fifteen +cases, of which these three are the ones you will meet: + +| case | anchor | what it does | +|---|---|---| +| `GxB_HYPERSPARSE` alone | `GB_conform.c:157-160` | `GB_convert_any_to_hyper` unconditionally — no test is run | +| `GxB_SPARSE` alone | `GB_conform.c:166-169` | convert to sparse unconditionally | +| `GxB_HYPERSPARSE + GxB_SPARSE` | `GB_conform.c:175-184` | run the hyper test; the bitmap test never executes | + +That table is the answer to question 1 before you go looking: +pinning a matrix's `sparsity_control` does not bias a heuristic, it +*removes the branch that would have run it*. + +Why it matters: "which format is this matrix in" is a question with +four answers and a per-matrix policy, and the policy is enforced by +which case of a switch statement you land in. + +### Step 2 — the two switches, and the hysteresis in each + +> **In:** the four formats and the two float thresholds (Step 1). +> **Out:** four one-line predicates with their exact constants, and +> the width of the band in which nothing happens. + +The bitmap boundary is two functions, and the asymmetry between +them is deliberate. `GB_convert_bitmap_to_sparse_test.c:13-16` +states the policy in the library's own words: + +```c +// GB_convert_bitmap_to_sparse_test.c — the policy comment, 13-16 + 13 // If A is m-by-n and A->sparsity_control is GxB_ANY_SPARSITY with b = + 14 // A->bitmap_switch, the matrix switches to bitmap if nnz(A)/(m*n) > b. A + 15 // bitmap matrix switches to sparse if nnz(A)/(m*n) <= b/2. A matrix whose + 16 // density is between b/2 and b remains in its current state. +``` + +And the two predicates that implement it: + +```c +// GB_convert_sparse_to_bitmap_test.c — sparse → bitmap, 31-38 + 31 // current number of entries in the matrix or vector + 32 float nnz = (float) anz ; + 33 + 34 // maximum number of entries in the matrix or vector + 35 float nnz_dense = ((float) vlen) * ((float) vdim) ; + 36 + 37 // A should switch to bitmap if the following condition is true: + 38 return (nnz > bitmap_switch * nnz_dense && nnz_dense < (float) GB_NMAX) ; +``` + +```c +// GB_convert_bitmap_to_sparse_test.c — bitmap → sparse, 43-44 + 43 // A should switch to sparse if the following condition is true: + 44 return (nnz <= (bitmap_switch/2) * nnz_dense) ; +``` + +**Hysteresis** — the switch-up and switch-down thresholds differ, +so a matrix hovering near the boundary does not convert back and +forth on every operation — is exactly the `b` versus `b/2` gap. The +same instinct as topic 3's LSM compaction triggers, and the cost of +getting it wrong is that each conversion is an O(nnz) rebuild, so +ping-ponging turns every operation into a copy. + +The hyper boundary has the same shape with a factor of 2 on the +other side, where `k` is the number of non-empty vectors: + +```c +// GB_convert_sparse_to_hyper_test.c — sparse → hyper, 33 + 33 return (n > 1 && (((float) k) <= n * hyper_switch)) ; +``` + +```c +// GB_convert_hyper_to_sparse_test.c — hyper → sparse, 33 + 33 return (n <= 1 || (((float) k) > n * hyper_switch * 2)) ; +``` + +Now the constants, and the first correction. `hyper_switch` +defaults to `GB_HYPER_SWITCH_DEFAULT` = **0.0625** = 1/16 +(`Source/include/GB_defaults.h:20`) — which matches TOMS '19 +§4.2.1 exactly: "SuiteSparse:GraphBLAS stores its matrices in +hypersparse format if n̄ < n/16." + +`bitmap_switch` does **not** default to a small percentage, and it +does not depend on the operation. It is a table indexed by the +matrix's *minimum dimension*: + +```c +// GB_Global.c — the bitmap_switch table, 181-189 + 181 // min dimension density + 182 #define GB_BITMAP_SWITCH_1 ((float) 0.04) + 183 #define GB_BITMAP_SWITCH_2 ((float) 0.05) + 184 #define GB_BITMAP_SWITCH_3_to_4 ((float) 0.06) + 185 #define GB_BITMAP_SWITCH_5_to_8 ((float) 0.08) + 186 #define GB_BITMAP_SWITCH_9_to_16 ((float) 0.10) + 187 #define GB_BITMAP_SWITCH_17_to_32 ((float) 0.20) + 188 #define GB_BITMAP_SWITCH_33_to_64 ((float) 0.30) + 189 #define GB_BITMAP_SWITCH_gt_than_64 ((float) 0.40) +``` -Every matrix carries a `sparsity_control` bitmask saying which -formats (hypersparse/sparse/bitmap/full — the ladder from the -previous chapter) are *allowed*, plus two floats -(`hyper_switch`, `bitmap_switch`) saying *when* to move between -them. `GB_conform` runs at the end of every operation and applies -the tests: +```c +// GB_Global.c — which row of the table a matrix gets, 486-497 + 486 float GB_Global_bitmap_switch_matrix_get (int64_t vlen, int64_t vdim) + 487 { + 488 int64_t d = GB_IMIN (vlen, vdim) ; + 489 if (d <= 1) return (GB_Global.bitmap_switch [0]) ; + ... + 495 if (d <= 64) return (GB_Global.bitmap_switch [6]) ; + 496 return (GB_Global.bitmap_switch [7]) ; + 497 } +``` + +For any graph matrix — anything with more than 64 rows and columns +— the answer is row 7: **b = 0.40**. Work out what that means: ``` - allowed? test go to - bitmap nnz > bitmap_switch × n×m (:32-38) bitmap - sparse bitmap_to_sparse_test (reverse, with sparse - hysteresis — thresholds differ so it - doesn't ping-pong) - hyper #non-empty vectors vs hyper_switch hyper/sparse - (GB_conform_hyper.c:52) + inputs: a graph adjacency matrix, n × n, n = 262,144 (notes.md:13) + nnz = 2.0e6 + b = 0.40 (GB_Global.c:189, since min(vlen,vdim) = 262,144 > 64) + + density = 2.0e6 / (262,144)² = 2.0e6 / 6.87e10 = 0.0000291 = 0.0029% + + bitmap needs density > 0.40 → 40% + the graph is 13,700× below the threshold + nnz that WOULD trigger bitmap = 0.40 × 6.87e10 = 2.75e10 entries + ... which at 1 byte of flag each is 27.5 GB of presence bits alone + + hysteresis band: density between b/2 = 20% and b = 40% keeps the + current format. On this matrix the band is 1.37e10 to 2.75e10 + entries wide — utterly unreachable. ``` -**Hysteresis** (the switch-up and switch-down thresholds differ, -so a matrix hovering near the boundary doesn't convert back and -forth on every operation) is the lesson — the same instinct as -topic 3's LSM compaction triggers. FalkorDB pins relation matrices -hypersparse+sparse via GxB_set — find where (question 1). Cost of -getting this wrong: each conversion is an O(nnz) rebuild, so -ping-ponging turns every op into a copy. +**A real graph adjacency matrix will never become bitmap.** The +bitmap format exists for the *vectors* and small dense blocks in a +computation, which is why LAGraph has to ask for it explicitly +(`LG_SET_FORMAT_HINT(q, LG_BITMAP)` at the BFS template's `:312`) +rather than waiting for the heuristic. Anyone who tells you the +threshold is "about 4–8%, operation-dependent" is reading +`bitmap_switch[0..3]`, which apply only to matrices with a +dimension of 8 or less. + +Why it matters: the number that gets quoted for this switch is +wrong by an order of magnitude and wrong in kind (it is not +per-operation), and the correct number explains why the pull BFS +must set its format by hand. + +### Step 3 — count the work before allocating: the flopcount pre-pass -### Step 2 — count the work before allocating: the flopcount pre-pass +> **In:** a multiply that has not started yet. +> **Out:** two numbers — total flops and per-vector flops — that +> size every subsequent decision, and their cost. For sparse matrix multiply, **flops** means the number of scalar -multiply-add operations that actually exist — for C = A*B, that's -one per (A(i,k), B(k,j)) pair of present entries. Unlike dense -matmul, you can *count* them cheaply before doing any of them: walk -the patterns (the index structure, no values) and sum -`nnz(B(k,:))` over A's entries. - -saxpy3 runs exactly this pre-pass (`GB_AxB_saxpy3_flopcount.c`) -before allocating anything, producing total flops and per-column -flops. Those two numbers then size *everything*: how many threads, -how to slice the work, and how big each hash table should be. The -same two-phase shape recurs across the curriculum — cudf's +multiply-add operations that actually exist: for C = A*B, one per +(A(i,k), B(k,j)) pair of present entries. Davis's own definition +(TOMS '19 §4.2.1) is "f is the number of 'multiply-adds' computed +(in the semiring)". Unlike dense matmul you can *count* them +cheaply before doing any of them, because the count depends only on +the patterns: + +```c +// GB_AxB_saxpy3_flopcount.c — the algorithm, in the header comment, 50-69 + 50 // Bflops = zeros (1,n) % (set to zero in the caller) + 51 // for each column j in B: + 52 // if (B (:,j) is empty) continue + 53 // if (M is present and M (:,j) is empty and not Mask_comp) continue + 54 // for each k where (B (k,j) != 0): + 55 // aknz = nnz (A (:,k)) + 56 // if (aknz == 0) continue + 57 // Bflops (j) += aknz % A(:,k)*B(k,j) requires aknz flops +``` + +And its complexity, stated in the same file: + +```c +// GB_AxB_saxpy3_flopcount.c — the complexity claim, 44-48 + 44 // The complexity of this function is O(nnz(B)+n) if A and M are not + 45 // hypersparse. If A and/or M are hypersparse, then the complexity can + 46 // increase to O(nnz(B)*log(h)) where h is the # of non-empty vectors in + 47 // A (or M). The log(h) factor is due to the binary search of A->h or M->h + 48 // for each entry in B. +``` + +Read `:53` twice — the pre-pass already applies the mask, skipping +columns whose mask column is empty. The mask is priced in before a +single multiply happens. + +Those two numbers then size *everything*: how many threads, how to +slice the work, and how big each hash table should be. The same +two-phase shape recurs across the curriculum — cudf's size/retrieve (topic 18), Gunrock's degree-scan — because sparse -output size is the recurring villain: you can't allocate the -output until you've measured the work. +output size is the recurring villain: you cannot allocate the +output until you have measured the work. -### Step 3 — saxpy3's task taxonomy: coarse and fine tasks +Why it matters: this pre-pass is not an optimization, it is the +precondition for every decision in Steps 4 through 6. Everything +downstream is arithmetic on `total_flops` and `Bflops`. -With per-column flops in hand, saxpy3 slices the multiply into -tasks of two kinds. The header comment (GB_AxB_saxpy3.c:22-60) -describes a two-level work division that IS morsel-driven -parallelism (topic 11): +### Step 4 — the thread count, computed from flops +> **In:** `total_flops` from Step 3. +> **Out:** the actual number of threads this multiply will use, on +> this repo's own measured workloads. + +```c +// GB_AxB_saxpy3_slice_balanced.c — flopcount, then thread count, 308-310 and 418 + 308 GB_OK (GB_AxB_saxpy3_flopcount (&Mwork, Bflops, M, Mask_comp, A, B, + 309 &total_flops, &axbflops, Werk)) ; + ... + 418 (*nthreads) = GB_nthreads (total_flops, chunk, nthreads_max) ; ``` - B's vectors (columns) → tasks: - coarse task: owns ≥1 whole vectors, private workspace - fine task: teams up on ONE big vector (a hub column), - shares workspace, needs atomics + +```c +// GB_nthreads.h — the whole rule, 17-32 + 17 // If work < 2*chunk, then only one thread is used. + 18 // else if work < 3*chunk, then two threads are used, and so on. + ... + 27 work = GB_IMAX (work, 1) ; + 28 chunk = GB_IMAX (chunk, 1) ; + 29 int64_t nthreads = (int64_t) floor (work / chunk) ; + 30 nthreads = GB_IMIN (nthreads, nthreads_max) ; + 31 nthreads = GB_IMAX (nthreads, 1) ; ``` -A **coarse task** is one thread owning whole columns — no -coordination, its workspace is private. A **fine task** exists -because power-law graphs have hub columns whose flops exceed an -entire fair share: a *team* of threads splits one fat column, and -because they share one output workspace, they pay for atomics. -The cost gradient: coarse = zero coordination overhead; fine = -atomics on every scatter, bought only where skew forces it. +`chunk` defaults to `GB_CHUNK_DEFAULT` = **64·1024 = 65,536** +(`Source/include/GB_defaults.h:24`). Now run this repo's own +measured SpGEMM flop counts (`notes.md:22-26`) through it, on an +8-performance-core machine: -### Step 4 — each task picks its accumulator: Gustavson vs hash +``` + rule: nthreads = clamp(floor(total_flops / 65536), 1, nthreads_max) + with nthreads_max = 8 -Each task accumulates one output column's worth of scattered -contributions, and it independently picks the data structure to do -it in: + scale 10: 298,000 flops / 65,536 = floor(4.5) = 4 threads + scale 12: 2,270,000 flops / 65,536 = floor(34.6) = 34 → clamped to 8 + scale 14: 17,100,000 flops / 65,536 = floor(260.9) = 260 → clamped to 8 + so: the smallest of the three benchmarks does not even use the + whole machine, and the other two are clamped long before the + divide matters. ``` - each task independently picks its accumulator: - Gustavson: dense f64[m] + pattern marker ("SPA") — O(1) scatter, - wins when the column's flops fill enough of m - hash: open-addressing table 2×pow2(flops-estimate) — wins - when m is huge and the column is sparse - rule: hash size would exceed m/16 ⇒ just use Gustavson (:57) + +That is the whole story of `chunk`: it is a floor on how much work +justifies waking a thread, not a slicing parameter. Below +2 × 65,536 = 131,072 flops a multiply is single-threaded no matter +how many cores you own. + +Why it matters: the first two rows explain why a small SpGEMM does +not scale — it was never asked to. Before blaming the scheduler, +check whether `GB_nthreads` handed it one thread. + +### Step 5 — saxpy3's task taxonomy: coarse and fine + +> **In:** `Bflops` per vector (Step 3) and a thread budget (Step 4). +> **Out:** the four task kinds, the workspace each costs in bytes, +> and the reason one of them needs atomics. + +The header comment is the scheduler spec, and it is worth reading +in full: + +```c +// GB_AxB_saxpy3.c — the task taxonomy, 22-48 + 22 // The matrix B is split into two kinds of tasks: coarse and fine. A coarse + 23 // task computes C(:,j1:j2) = A*B(:,j1:j2), for a unique set of vectors j1:j2. + 24 // Those vectors are not shared with any other tasks. A fine task works with a + 25 // team of other fine tasks to compute C(:,j) for a single vector j. Each fine + 26 // task computes A*B(k1:k2,j) for a unique range k1:k2, and sums its results + 27 // into C(:,j) via atomic operations. + 28 + 29 // Each coarse or fine task uses either Gustavson's method [1] or the Hash + 30 // method [2]. There are 4 kinds of tasks: + 31 + 32 // fine Gustavson task + 33 // fine hash task + 34 // coarse Gustason task + 35 // coarse hash task + 36 + 37 // Each of the 4 kinds tasks are then subdivided into 3 variants, for C=A*B, + 38 // C=A*B, and C=A*B, giving a total of 12 different types of tasks. + ... + 42 // ... Coarse tasks are + 43 // prefered since they require less synchronization, but fine tasks allow for + 44 // better parallelization when B has only a few vectors. If B consists of a + 45 // single vector (for GrB_mxv if A is in CSC format and not transposed, or + 46 // for GrB_vxm if A is in CSR format and not transpose), then the only way to + 47 // get parallelism is via fine tasks. ``` +Note `:44-47`. A **matrix-vector** product — the BFS pull step — +has B with exactly one vector, so *the only available parallelism +is fine tasks*, with their atomics. That is a structural fact about +SpMV, and it is the mechanism behind the parallel-scaling gap in +[reading-openmp-vs-rayon.md](reading-openmp-vs-rayon.md). + +The workspace bill, also from the header: + +```c +// GB_AxB_saxpy3.c — workspace per task kind, 62-70 + 62 // The workspace allocated depends on the type of task. Let s be the hash + 63 // table size for the task, and C is m-by-n (assuming all matrices are CSC; if + 64 // CSR, then m is replaced with n). + 65 // + 66 // fine Gustavson task (shared): int8_t Hf [m] ; ctype Hx [m] ; + 67 // fine hash task (shared): uint64_t Hf [s] ; ctype Hx [s] ; + 68 // coarse Gustavson task: uint64_t Hf [m] ; ctype Hx [m] ; + 69 // coarse hash task: uint64_t Hf [s] ; ctype Hx [s] ; + 70 // uint64_t Hi [s] ; +``` + +Price those four rows for an f64 output at scale 20 (m = 2²⁰ = +1,048,576, `notes.md:14`): + +``` + inputs: m = 1,048,576 ; ctype = double (8 bytes) + a hash task with flmax = 4,096 gets s = 16,384 (Step 6) + + fine Gustavson (shared) : 1·m + 8·m = 9 bytes/row × 1,048,576 = 9.4 MB + coarse Gustavson : 8·m + 8·m = 16 bytes/row × 1,048,576 = 16.8 MB + fine hash : 8·s + 8·s = 16 bytes/slot × 16,384 = 262 KB + coarse hash : 8·s ×3 = 24 bytes/slot × 16,384 = 393 KB + + per-thread, on 8 threads, coarse Gustavson: + 8 × 16.8 MB = 134 MB of workspace, none of it in any cache + the M3 Pro's L2 is ~16 MB shared; ONE coarse Gustavson task + already exceeds it, and eight of them thrash. + + the same 8 threads on coarse hash tasks: 8 × 393 KB = 3.1 MB, + which fits in L2 with room to spare. +``` + +Note the fine Gustavson row is 9 bytes per row, not 16: `:66` uses +`int8_t Hf` rather than `uint64_t`, because a shared flag only +needs to hold a state, while a coarse task's `Hf` doubles as a +64-bit stamp. Nine versus sixteen is a 44% saving on the biggest +array in the library. + +Why it matters: this is where the hash-versus-Gustavson decision +comes from. It is not about collisions — it is about 16.8 MB versus +393 KB, and topic 13's blocking argument. + +### Step 6 — each task picks its accumulator, by a rule that is not m/16 + +> **In:** the per-vector flop maxima (Step 3) and the workspace +> bills (Step 5). +> **Out:** the shipped selection rule, the flop threshold it +> implies, and the stale comment it contradicts. + **Gustavson** here means the classic dense-workspace method: a -**SPA** (sparse accumulator — a dense array of size m, one slot -per possible output row, plus a marker of which slots are -occupied) gives O(1) scatter but costs m slots of (possibly cold) -memory per task. The **hash** alternative sizes a table by the -flops estimate instead of by m — small and cache-resident when the -column is sparse, no matter how big m is. The shipped rule at :57: -if the hash table would exceed m/16, the dense SPA is cheaper — -just use Gustavson. This is topic 8's hash-vs-sort aggregation -choice, made per task from step 2's numbers. - -### Step 5 — dot3: the mask as the outer loop - -The other engine inverts control entirely. dot3 computes -`C = A'*B` and *requires* the mask M (the "only produce outputs -here" matrix): it iterates over M's entries, and for each -(i,j) ∈ M computes one sparse dot product A(:,i)'·B(:,j). Work is -nnz(M) dot products — the mask isn't a filter applied afterwards, -it's the OUTER LOOP: +**SPA** (sparse accumulator — a dense array of size m, one slot per +possible output row, plus a marker of which slots are occupied) +gives O(1) scatter but costs m slots of possibly-cold memory per +task. The **hash** alternative sizes a table by the flop estimate +instead of by m — small and cache-resident when the vector is +sparse, no matter how big m is. + +The header comment states the rule as `m/16`: + +```c +// GB_AxB_saxpy3.c — the DOCUMENTED rule, 50-60. Read it, then read the code. + 50 // To select between the Hash method or Gustavson's method for each task, the + 51 // hash table size is first found. ... + 53 // ... It is set to twice the smallest power of 2 that + 54 // is greater than the flop count to compute that vector (plus the # of entries + 55 // in M(:,j) for tasks that compute C=A*B or C=A*B). This size ensures + 56 // the results will fit in the hash table, and with ideally only a modest + 57 // number of collisions. If the hash table size exceeds a threshold (currently + 58 // m/16 if C is m-by-n), then Gustavson's method is used instead, and the hash + 59 // table size is set to m, to serve as the gather/scatter workspace for + 60 // Gustavson's method. +``` + +The shipped code says something else: + +```c +// GB_AxB_saxpy3_slice_balanced.c — GB_hash_table_size, the real rule, 56-99 + 56 static inline uint64_t GB_hash_table_size + 57 ( + 58 int64_t flmax, // max flop count for any vector computed by this task + 59 int64_t cvlen, // vector length of C + 60 const int AxB_method // Default, Gustavson, or Hash + 61 ) + 62 { + 63 uint64_t hash_size ; + 64 + 65 if (AxB_method == GxB_AxB_GUSTAVSON || flmax >= cvlen/2) + ... + 72 hash_size = cvlen ; + ... + 82 // hash_size = 2 * (smallest power of 2 >= flmax) + 83 hash_size = ((uint64_t) 2) << (GB_FLOOR_LOG2 (flmax) + 1) ; + 84 bool use_Gustavson ; + 85 if (AxB_method == GxB_AxB_HASH) + 86 { + 87 // always use Hash method, unless the hash_size >= cvlen + 88 use_Gustavson = (hash_size >= cvlen) ; + 89 } + 90 else + 91 { + 92 // default: auto selection: + 93 // use Gustavson's method if hash_size is too big + 94 use_Gustavson = (hash_size >= cvlen/12) ; + 95 } + 96 if (use_Gustavson) + 97 { + 98 hash_size = cvlen ; + 99 } +``` + +**The threshold is `cvlen/12`, at `:94` — not `m/16`.** The +comment at `GB_AxB_saxpy3.c:57-58` has not been updated. There is +also an earlier cutoff the comment never mentions: `:65` sends any +vector with `flmax >= cvlen/2` straight to Gustavson without +computing a hash size at all. + +Work out where the crossover actually falls, at scale 20: + +``` + inputs: cvlen = m = 2^20 = 1,048,576 (notes.md:14, scale 20) + default auto-selection, so :94 applies + + Gustavson is chosen when hash_size >= cvlen/12 = 87,381 + + hash_size = 2 << (floor_log2(flmax) + 1) = 2^(floor_log2(flmax) + 2) + + smallest power of two >= 87,381 is 2^17 = 131,072 + → need floor_log2(flmax) + 2 >= 17 + → floor_log2(flmax) >= 15 + → flmax >= 2^15 = 32,768 + + so: a vector with 32,768 or more flops gets Gustavson; + fewer than that gets a hash table. + + as a fraction of m: 32,768 / 1,048,576 = 3.1% (i.e. m/32) + + sanity-check against the DOCUMENTED m/16 rule: + hash_size >= m/16 = 65,536 → 2^(k+2) >= 65,536 → k >= 14 + → flmax >= 16,384 = m/64 + the comment's rule would switch to Gustavson at HALF the flop + count the code does. The code is more willing to hash. +``` + +And a corollary worth its own line: because `hash_size` is at least +2× `flmax` (`:83` gives `2^(k+2)` where `2^k ≤ flmax`), and `flmax` +is an *exact* upper bound on the number of distinct entries the +vector can produce, **the hash table cannot overflow**. There is no +resize path in saxpy3 because there is nothing to resize for — the +pre-pass of Step 3 makes the size provable. That is question 3's +answer, and it is a genuinely different design from SwissTable's +grow-on-load-factor (topic 8): SuiteSparse buys exactness with a +whole extra pass over the patterns. + +Why it matters: two of the three numbers you would have quoted from +the header comment are wrong at this pin. The `m/16` rule is stale; +the `flmax >= cvlen/2` cutoff is undocumented; only the "twice the +smallest power of 2" sizing survives. + +### Step 7 — dot3: the mask as the outer loop + +> **In:** the mask, so far only a filter (Steps 3, 5). +> **Out:** the engine where the mask is the loop bound, and the +> line of code that proves it. + +The other engine inverts control entirely: + +```c +// GB_AxB_dot3.c — the contract, 10-13 + 10 // This function only computes C=A'*B. The mask must be present, and not + 11 // complemented, and can be either valued or structural. The mask is always + 12 // applied. C and M are both sparse or hypersparse, and have the same sparsity + 13 // structure. +``` + +Four preconditions in three lines: mask present, mask not +complemented, C and M both sparse-or-hypersparse, **same sparsity +structure**. The last one is not a hint — it is a promise the code +keeps literally: + +```c +// GB_AxB_dot3.c — C is allocated with exactly nnz(M) entries, 126 and 171 + 126 const int64_t mnz = GB_nnz (M) ; + ... + 171 int64_t cnz = mnz ; +``` + +`C` gets `nnz(M)` slots because it will have at most `nnz(M)` +entries, one candidate per mask entry. Work is nnz(M) sparse dot +products; the mask is not a filter applied afterwards, it is the +outer loop. Compare `GB_AxB_dot3.c:244`, where the thread count is +`GB_nthreads(cnz, chunk, nthreads_max)` — even the parallelism is +sized by the mask. + +The shape of the loop, since the real one is templated across +twelve type combinations: ```rust -// dot3: the mask M is the outer loop — work ∝ nnz(M), a complexity -// CLASS below computing A'*B and filtering afterward +// ILLUSTRATION — not quoted from SuiteSparse. The real loop is generated +// from templates; the structural claims are GB_AxB_dot3.c:10-13 (the +// contract), :126 and :171 (C sized to nnz(M)), and :244 (threads sized +// to cnz). fn dot3(m: &Pattern, a_t: &Csr, b: &Csc, semiring: &Semiring) -> Coo { - let mut c = Coo::new(); - for (i, j) in m.entries() { // one dot per MASK entry + let mut c = Coo::with_capacity(m.nnz()); // :171 — cnz = mnz + for (i, j) in m.entries() { // one dot per MASK entry // sparse dot = two-pointer intersect of the two patterns if let Some(v) = sparse_dot(a_t.row(i), b.col(j), semiring) { - c.push(i, j, v); // (ANY monoid ⇒ sparse_dot - } // may stop at first hit) + c.push(i, j, v); // ANY monoid ⇒ sparse_dot + } // may stop at the first hit } c } ``` -If M is triangle counting's lower-triangular L, that's one dot per +If M is triangle counting's lower-triangular L, that is one dot per candidate wedge — nothing is computed for output cells the mask -excludes. Contrast saxpy3, where the mask only prunes *writes*: -the flops still happen. This asymmetry is why "masks are free -performance" in FalkorDB — but only when the dispatcher picks dot3. - -### Step 6 — dispatch: a cost-based optimizer decision per multiply - -`GB_AxB_meta.c` chooses the engine for each multiply: dot3 when a -mask is present and C is sparse (work ∝ nnz(M)), saxpy3 for the -general case (work ∝ flops), bitmap/full variants (saxbit, dot2, -dot4) when operands or output are dense. The choice weighs nnz(M) -against predicted saxpy flops — a cost-based optimizer decision -(topic 10) made per multiply, using step 2's estimates as the cost -model. The consequence for API users: the *same* GrB_mxm line runs -a different algorithm depending on your mask's density — which is -exactly how the BFS push/pull switch will be implemented in the -Beamer and LAGraph chapters. +excludes. Contrast saxpy3, where the mask only prunes *writes*: the +flops still happen. Davis says the same thing about the masked case +in TOMS '19 §4.2.1: "If the mask is present (and not complemented), +only the subset of entries appearing in the mask are computed… +In this method, the symbolic analysis is skipped." + +Price the asymmetry on a masked triangle-count product: + +``` + inputs: C = L*L on RMAT scale 16 (topics/24-graph-algorithms/notes.md:5) + n = 65,536, m_directed = 1,819,338, so nnz(L) ≈ 909,669 + mean degree d̄ = 1,819,338 / 65,536 = 27.8 + + dot3 work = nnz(L) dot products + = 909,669 × (intersect two rows of avg length 27.8) + ≈ 909,669 × 55.6 pointer steps = 5.06e7 steps + + saxpy work = Σ over L's entries of nnz(L(k,:)) + ≈ 909,669 × 27.8 = 2.53e7 multiply-adds + ... of which only the entries landing inside L survive + + so saxpy does FEWER raw operations here, and dot3 still often wins, + because dot3's 5.06e7 steps are two sequential streams while + saxpy's 2.53e7 are scatters into a 65,536-slot SPA plus a + discard pass. Which is exactly why LAGr_TriangleCount.c:43-47 + refuses to pick a winner and ships both. +``` + +Why it matters: "masks are free performance" is true only when the +dispatcher picks a mask-as-outer-loop engine, and the four +preconditions at `:10-13` are what decide that. + +### Step 8 — dispatch: a cost-based optimizer decision per multiply + +> **In:** the two engines (Steps 5-7) and their preconditions. +> **Out:** the exact control function that chooses, and the reason +> the BFS pull step lands where it does. + +For the `C = A'*B` shape, the decision lives in one function: + +```c +// GB_AxB_meta_adotb_control.c — the auto-selection, 60-88 (elided) + 60 else if (AxB_method == GxB_DEFAULT) + 61 { + 62 // auto selection for A'*B + ... + 72 if (GB_AxB_dot4_control (C_out_iso, can_do_in_place ? C_in : NULL, + 73 M, Mask_comp, accum, semiring)) + 75 // C+=A'*B can be done with dot4 + 76 (*axb_method) = GB_USE_DOT ; + 78 else if (GB_AxB_dot3_control (M, Mask_comp)) + 80 // C=A'*B uses the masked dot product method (dot3) + 81 (*axb_method) = GB_USE_DOT ; + 83 else if (GB_AxB_dot2_control (A, B)) + 85 // C=A'*B or C=A'B* can efficiently use the dot2 method + 86 (*axb_method) = GB_USE_DOT ; + 88 } +``` + +Three `else if`s in order, and no `else` — falling off the end at +`:88` leaves `*axb_method` at the default set at `:36`, which is +`GB_USE_SAXPY`. **Saxpy is what you get when nothing else claims +the multiply.** + +with the two predicates it consults: + +```c +// GB_mxm.h — when dot3 is eligible, 235-243 + 235 static inline bool GB_AxB_dot3_control + ... + 241 return (M != NULL && !Mask_comp && + 242 (GB_IS_SPARSE (M) || GB_IS_HYPERSPARSE (M))) ; +``` + +```c +// GB_AxB_dot2_control.c — the first and decisive test, 23-30 + 23 // C = A'*B is very efficient if A and/or B are full or bitmap + ... + 26 if (GB_IS_FULL (A) || GB_IS_BITMAP (A) || + 27 GB_IS_FULL (B) || GB_IS_BITMAP (B)) + 28 { + 29 return (true) ; + 30 } +``` + +Trace the BFS pull step through it and you get a result that +contradicts the usual summary. LAGraph's pull is +`GrB_mxv(q, mask, NULL, semiring, AT, q, GrB_DESC_RSC)` +(`LG_BreadthFirstSearch_SSGrB_template.c:313`). `GrB_DESC_RSC` +includes `GrB_COMP` (`Include/GraphBLAS.h:666`), so `Mask_comp` is +true, so `GB_AxB_dot3_control` at `GB_mxm.h:241` returns **false**. +**Pull cannot use dot3.** What makes a dot engine eligible is the +line before the call — `LG_SET_FORMAT_HINT(q, LG_BITMAP)` at +`:312` — which makes `GB_AxB_dot2_control.c:26-30` return true on +its first test. Pull is **dot2**, and Davis's CSC '20 §3.1 agrees +in words: "By default, GraphBLAS selects the masked-dot-product +method for… the pull phase of the push/pull BFS." + +The consequence for API users: the *same* `GrB_mxm` line runs a +different algorithm depending on your mask's density and your +operands' formats — which is exactly how the BFS push/pull switch +is implemented in +[reading-beamer-sc12.md](reading-beamer-sc12.md) and +[reading-lagraph.md](reading-lagraph.md). + +Why it matters: the dispatcher reads *four* things — mask presence, +mask complement, operand format, and output format — and getting a +prediction right means checking all four, not just "is there a +mask". ## Where each step lives in the code | anchor | step | what it is | |---|---|---| -| Source/convert/GB_convert_sparse_to_bitmap_test.c:32-38 | 1 | THE bitmap heuristic: `nnz > bitmap_switch * nnz_dense` | -| Source/convert/GB_conform_hyper.c:52 | 1 | hyper→sparse test via `hyper_switch` | -| Source/convert/GB_conform.c:33-89 | 1 | conform runs after every op; sparsity_control bitmask | -| Source/mxm/GB_AxB_saxpy3_flopcount.c | 2 | the sizing pre-pass | -| Source/mxm/GB_AxB_saxpy3.c:22-60 | 3-4 | coarse/fine tasks × Gustavson/hash — read this header comment twice | -| Source/mxm/GB_AxB_saxpy3.c:57 | 4 | hash > m/16 ⇒ fall back to Gustavson | -| Source/mxm/GB_AxB_dot3.c:2-10 | 5 | `C=A'*B` — mask REQUIRED, work ∝ nnz(M) | -| Source/mxm/GB_AxB_dot2.c / dot4.c | 6 | unmasked / C+=A'*B dense-output variants | -| Source/mxm/GB_AxB_meta.c | 6 | engine dispatch (dot vs saxpy vs saxbit) | +| `Source/convert/GB_conform.c:150` | 1 | the 15-case switch on `sparsity_control` | +| `Source/convert/GB_conform.c:157-160`, `:166-169`, `:175-184` | 1 | hyper-only, sparse-only, hyper+sparse cases | +| `Source/convert/GB_conform_hyper.c:44-57` | 1-2 | `nvec_nonempty`, then the two hyper tests | +| `Source/convert/GB_convert_bitmap_to_sparse_test.c:13-16` | 2 | the b / b/2 hysteresis policy, in the library's words | +| `Source/convert/GB_convert_sparse_to_bitmap_test.c:32-38` | 2 | `nnz > bitmap_switch × nnz_dense` | +| `Source/convert/GB_convert_bitmap_to_sparse_test.c:44` | 2 | `nnz <= (bitmap_switch/2) × nnz_dense` | +| `Source/convert/GB_convert_sparse_to_hyper_test.c:33` | 2 | `n > 1 && k <= n × hyper_switch` | +| `Source/convert/GB_convert_hyper_to_sparse_test.c:33` | 2 | `n <= 1 \|\| k > n × hyper_switch × 2` | +| `Source/global/GB_Global.c:181-189`, `:486-497` | 2 | the bitmap_switch table, **indexed by min dimension** — 0.40 for graphs | +| `Source/include/GB_defaults.h:20`, `:24` | 2, 4 | `hyper_switch` 0.0625, `chunk` 65,536 | +| `Source/mxm/GB_AxB_saxpy3_flopcount.c:44-48`, `:50-69` | 3 | complexity and the pre-pass algorithm | +| `Source/mxm/GB_AxB_saxpy3_flopcount.c:219-221` | 3 | `schedule(dynamic,1)` over pre-sliced tasks | +| `Source/mxm/GB_AxB_saxpy3_slice_balanced.c:308-310`, `:418` | 3-4 | flopcount call, then `GB_nthreads(total_flops, …)` | +| `Source/omp/include/GB_nthreads.h:17-32` | 4 | `clamp(floor(work/chunk), 1, nthreads_max)` | +| `Source/mxm/GB_AxB_saxpy3.c:22-48` | 5 | coarse/fine, the 4 kinds, the 12 variants | +| `Source/mxm/GB_AxB_saxpy3.c:62-70` | 5 | the workspace table — 9 vs 16 bytes per row | +| `Source/mxm/GB_AxB_saxpy3_slice_balanced.c:56-99` | 6 | `GB_hash_table_size` — the **`cvlen/12`** rule | +| `Source/mxm/GB_AxB_saxpy3.c:57-58` | 6 | the stale `m/16` comment — do not quote it | +| `Source/mxm/GB_AxB_dot3.c:10-13`, `:126`, `:171`, `:244` | 7 | mask required and uncomplemented; C sized to nnz(M) | +| `Source/mxm/GB_AxB_meta_adotb_control.c:36`, `:60-93` | 8 | default saxpy, then dot4 / dot3 / dot2 in order | +| `Source/mxm/GB_mxm.h:235-243` | 8 | `GB_AxB_dot3_control` — the four-term predicate | +| `Source/mxm/GB_AxB_dot2_control.c:26-30`, `:68-79` | 8 | bitmap/full operand ⇒ dot2; the degree heuristic | +| `Include/GraphBLAS.h:666` | 8 | `GrB_DESC_RSC = REPLACE + STRUCTURE + COMP` | Navigation advice: start with the saxpy3 header comment -(GB_AxB_saxpy3.c:22-60) — it is the scheduler spec, and everything -else in `Source/mxm/` is implementation of that comment. Then read -`GB_conform.c` top to bottom (it's short), then skim -`GB_AxB_meta.c` for the dispatch conditions. +(`GB_AxB_saxpy3.c:22-86`) — it is the scheduler spec, and +everything else in `Source/mxm/` is an implementation of that +comment — but read it *next to* +`GB_AxB_saxpy3_slice_balanced.c:56-99`, because the comment's +selection rule is out of date. Then read `GB_conform.c` top to +bottom (391 lines, mostly a switch), then +`GB_AxB_meta_adotb_control.c` for the dispatch conditions. ### What transfers to M20 - Our stub SpGEMM = one coarse Gustavson task (dense SPA). The - HashMap reference = the hash task. gb_bench measures the m/16 - intuition directly. -- Masked-SpMV pull BFS = dot3's idea specialized: iterate the - UNVISITED set (the mask), early-exit each dot at first frontier - hit (ANY monoid ⇒ short-circuit legal). + HashMap reference = the hash task. `gb_bench` measures the + crossover directly — and Step 6 says where to expect it. +- Masked-SpMV pull BFS = a dot engine's idea specialized: iterate + the UNVISITED set (the complemented mask), early-exit each dot at + the first frontier hit (ANY monoid ⇒ short-circuit legal). - M20's kernel core needs only: saxpy-SpMSpV (push), masked - dot-SpMV (pull), one SPA SpGEMM, conform-lite (hyper↔sparse). + dot-SpMV (pull), one SPA SpGEMM, conform-lite (hyper↔sparse). The + bitmap arithmetic in Step 2 is the argument for *not* building a + bitmap matrix format at all. ## Questions for notes.md -1. Find FalkorDB's GxB_set calls pinning formats (grep GxB_SPARSITY - in [~/repos/FalkorDB](https://github.com/FalkorDB/FalkorDB)/src). Which matrices allow bitmap and why - not the adjacency ones? +1. Find FalkorDB's `GxB_set` calls pinning formats + (`src/graph/delta_matrix/delta_new.c` at the pin). Which matrices + allow bitmap, and why not the adjacency ones? Use Step 2's + arithmetic and Step 1's `GB_conform.c` case table for the + answer — the interesting part is that pinning removes the + branch, not that it biases it. 2. Why does a fine Gustavson task need atomics on the SPA but a - coarse one doesn't — and what's the topic 11 analogue - (shared hash aggregation vs per-thread pre-aggregation)? -3. The hash task's table is sized 2× next-pow2(estimated flops). - What happens on underestimate (collision pile-up — degrade, - or rebuild? find it in GB_AxB_saxpy3.c) — compare SwissTable's - resize story (topic 8). -4. dot3 vs saxpy3 crossover: for `C=L*U'` triangle counting on an - RMAT graph, estimate both costs (nnz(L) dots of avg length d̄ vs - Σ flops) — which wins and why does LAGraph still offer both - (LAGr_TriangleCount.c:31-46)? -5. Run gb_bench: at what RMAT scale does our dense-SPA Gustavson - lose to the HashMap version (SPA = m×8B cold bytes per row team - — when m outgrows L2, topic 13's blocking argument bites)? + coarse one does not — and what is the topic 11 analogue (shared + hash aggregation vs per-thread pre-aggregation)? Use + `GB_AxB_saxpy3.c:24-27` and the two `Hf` widths at `:66` and + `:68`. +3. The hash task's table is sized at `:83`. What happens on an + underestimate — collision pile-up, degrade, or rebuild? Find the + resize path in `GB_AxB_saxpy3*.c`, then explain why what you find + is what it is, and compare SwissTable's resize story (topic 8). +4. dot3 vs saxpy3 crossover: for `C = L*U'` triangle counting on + an RMAT graph, estimate both costs (nnz(L) dots of average length + d̄ versus Σ flops) using Step 7's arithmetic — which wins, and + why does LAGraph still ship both + (`LAGr_TriangleCount.c:43-47`)? +5. Run `gb_bench`: at what RMAT scale does our dense-SPA Gustavson + lose to the HashMap version? Predict it first from Step 5's + workspace table (coarse Gustavson = 16 bytes/row) and your + machine's L2 size, then measure. ## Done when -- [ ] You can explain format switching as a bitmask plus two floats, and name the two thresholds. -- [ ] You can explain why the flopcount pre-pass exists before any allocation. -- [ ] You can describe the coarse/fine task taxonomy and why a fine Gustavson task needs atomics while a coarse one does not. -- [ ] You can say how each task picks between Gustavson and hash accumulation, and how the hash table is sized. -- [ ] You can explain dot3's mask-as-outer-loop and where it crosses over against saxpy3. +Answer each before unfolding it. + +- [ ] You can name all four formats and say which two the TOMS '19 paper does *not* describe. + +
Answer + + full, bitmap, sparse, hypersparse. TOMS '19 §4.1 describes only + the sparse pair — at version 2.3.3 the four formats it names are + "standard CSC, standard CSR, and hypersparse versions of these + two". Bitmap and full arrived later, so any claim about them must + be sourced to the code or a later paper. + + A matrix's `sparsity_control` bitmask says which of the four it + may become, and `GB_conform.c:150` switches on it — fifteen + cases. Pinning a matrix to `GxB_HYPERSPARSE` sends it to + `:157-160`, which converts unconditionally: the test never runs. + +
+ +- [ ] You can state both format switches with their hysteresis, and give the default constants. + +
Answer + + Bitmap: sparse → bitmap when `nnz > b × m·n` + (`GB_convert_sparse_to_bitmap_test.c:38`); bitmap → sparse when + `nnz <= (b/2) × m·n` (`GB_convert_bitmap_to_sparse_test.c:44`). + The library's own summary is at `:13-16`: "A matrix whose density + is between b/2 and b remains in its current state." + + Hyper: sparse → hyper when `n > 1 && k <= n × h` + (`GB_convert_sparse_to_hyper_test.c:33`); hyper → sparse when + `n <= 1 || k > n × h × 2` + (`GB_convert_hyper_to_sparse_test.c:33`), where k is the number + of non-empty vectors. So h versus 2h. + + Constants: `h` = `GB_HYPER_SWITCH_DEFAULT` = 0.0625 = 1/16 + (`GB_defaults.h:20`), matching TOMS '19 §4.2.1's "hypersparse + format if n̄ < n/16". `b` is a table indexed by + `min(vlen, vdim)` (`GB_Global.c:486-497`) running 0.04 → 0.40; + for any dimension above 64 it is **0.40**, and it does not depend + on the operation. + +
+ +- [ ] You can say whether a graph adjacency matrix ever becomes bitmap, with the arithmetic. + +
Answer + + No. At scale 18 (`notes.md:13`) the matrix is 262,144 × 262,144 + with 2.0e6 entries, so its density is + 2.0e6 / 6.87e10 = 0.0029%. The threshold for a matrix with + min dimension > 64 is b = 0.40 = 40% (`GB_Global.c:189`), which + is 13,700× away. It would take 2.75e10 entries — 27.5 GB of + presence flags alone — to trigger. + + This is why LAGraph sets `LG_SET_FORMAT_HINT(q, LG_BITMAP)` by + hand at the BFS template's `:312` instead of letting the + heuristic decide: bitmap is for the *vector*, and the heuristic + would never choose it for the matrix. + +
+ +- [ ] You can explain why the flopcount pre-pass exists, and what its output sizes. + +
Answer + + Because the output's size is not knowable without it, and every + downstream decision needs a number. The pre-pass walks patterns + only — `GB_AxB_saxpy3_flopcount.c:50-57`'s + `Bflops(j) += nnz(A(:,k))` — at O(nnz(B)+n) when A and M are not + hypersparse, or O(nnz(B)·log h) when they are (`:44-48`). It also + applies the mask while counting (`:53`). + + Its two outputs then size: the thread count + (`GB_nthreads(total_flops, chunk, …)` at + `GB_AxB_saxpy3_slice_balanced.c:418`), the task slicing + (`target_task_size = total_flops / ntasks_initial` at `:456`), + and each task's hash table (`GB_hash_table_size(flmax, …)` at + `:56-99`). Same two-phase shape as cudf's size/retrieve + (topic 18). + +
+ +- [ ] You can compute how many threads a multiply gets, on this topic's own SpGEMM flop counts. + +
Answer + + `GB_nthreads.h:29-31`: + `clamp(floor(work/chunk), 1, nthreads_max)`, with `chunk` = 65,536 + (`GB_defaults.h:24`). From `notes.md:24-26`, on 8 cores: + + scale 10, 298K flops → floor(4.5) = **4 threads** — half the + machine idle. Scale 12, 2.27M → 34, clamped to 8. Scale 14, + 17.1M → 260, clamped to 8. + + The comment at `:17-18` gives the rule in words: "If work < + 2*chunk, then only one thread is used." So below 131,072 flops a + multiply is single-threaded regardless of core count. Check this + before blaming the scheduler for poor scaling. + +
+ +- [ ] You can describe the four task kinds, price their workspace, and say which one SpMV is forced into. + +
Answer + + `GB_AxB_saxpy3.c:22-38`: coarse (owns whole vectors of B, private + workspace) × fine (a team splits one vector, shares workspace, + sums via atomics), each × Gustavson or hash — four kinds, further + × 3 mask variants = twelve. + + Workspace, from `:66-70`, for an f64 output at m = 2²⁰: + fine Gustavson `int8_t Hf[m] + ctype Hx[m]` = 9 bytes/row = + 9.4 MB; coarse Gustavson `uint64_t Hf[m] + ctype Hx[m]` = + 16 bytes/row = 16.8 MB; a hash task with s = 16,384 is 262 KB + (fine) or 393 KB (coarse, because `:70` adds `Hi[s]`). Eight + coarse Gustavson tasks want 134 MB and will not fit any cache; + eight coarse hash tasks want 3.1 MB and will. + + SpMV is forced into fine tasks. `:44-47`: "If B consists of a + single vector… then the only way to get parallelism is via fine + tasks." A matrix-vector product therefore pays atomics on every + scatter, which is the structural reason BFS scales worse than + SpGEMM. + +
+ +- [ ] You can state the shipped Gustavson-vs-hash rule and say why the header comment disagrees. + +
Answer + + Shipped, at `GB_AxB_saxpy3_slice_balanced.c:56-99`: first, if + `flmax >= cvlen/2`, Gustavson immediately (`:65`) — undocumented + in the header. Otherwise size the table as + `2 << (floor_log2(flmax) + 1)` (`:83`) and take Gustavson if + `hash_size >= cvlen/12` (`:94`). + + `GB_AxB_saxpy3.c:57-58` still says the threshold is "m/16". It is + stale. At m = 2²⁰ the shipped `cvlen/12` rule puts the crossover + at flmax = 32,768 (= m/32, 3.1% of m); the documented m/16 rule + would put it at flmax = 16,384. The code is twice as willing to + hash as its own comment claims. + + The corollary at `:83`: `hash_size` is always at least 2× `flmax`, + and `flmax` is an exact bound on the vector's distinct outputs, so + the table cannot overflow. There is no resize path because there + is nothing to resize for — SuiteSparse buys that with the extra + pattern pass of Step 3, where SwissTable (topic 8) buys + amortisation with a growth policy. + +
+ +- [ ] You can explain dot3's mask-as-outer-loop, cite the line that proves it, and say why the BFS pull step is *not* dot3. + +
Answer + + dot3 computes `C = A'*B` with one sparse dot product per mask + entry. The proof is allocation, not documentation: + `GB_AxB_dot3.c:126` computes `mnz = GB_nnz(M)` and `:171` sets + `cnz = mnz`, so C is sized to the mask exactly; `:244` sizes the + thread count from `cnz` too. Contrast saxpy3, where a mask prunes + writes but the flops still happen. + + Pull is not dot3 because `GB_AxB_dot3.c:10-11` requires the mask + to be "present, and not complemented", and `GB_mxm.h:241` encodes + that as `M != NULL && !Mask_comp && (M sparse or hypersparse)`. + LAGraph's pull passes `GrB_DESC_RSC` + (`…SSGrB_template.c:313`), and `GrB_DESC_RSC` includes + `GrB_COMP` (`Include/GraphBLAS.h:666`), so `Mask_comp` is true + and the predicate fails. + + What pull actually reaches is **dot2**, because + `LG_SET_FORMAT_HINT(q, LG_BITMAP)` on the line before (`:312`) + makes `GB_AxB_dot2_control.c:26-30` return true on its first + test. Davis's CSC '20 §3.1 says the same in prose: the library + "selects the masked-dot-product method for… the pull phase of the + push/pull BFS". + +
+ - [ ] You wrote answers to all five questions in notes.md, including the RMAT scale at which the dense SPA stops fitting. +
Answer + + Predict before you measure. A coarse Gustavson task's workspace is + 16 bytes per row of C (`GB_AxB_saxpy3.c:68`, `uint64_t Hf[m]` + + `double Hx[m]`). On a machine with an L2 of size S, the SPA stops + fitting at m ≈ S/16 per thread — and with T threads each holding + its own, at m ≈ S/(16·T). + + On the M3 Pro of `notes.md:3`, with a shared L2 around 16 MB and + 8 threads, that is m ≈ 16 MB / (16 × 8) = 131,072 rows — RMAT + scale 17. Above that, every SPA scatter is a DRAM round trip and + the hash version, whose table is sized by flops rather than by m, + should overtake. `notes.md:50-51`'s two prediction rows are asking + for exactly this number at scales 14 and 20, which bracket it. + +
+ ## References **Papers** -- Davis — "Algorithm 1000: SuiteSparse:GraphBLAS" (ACM TOMS 2019) - — the companion paper; see - [reading-davis-toms19.md](reading-davis-toms19.md) + +- Davis — "Algorithm 1000: SuiteSparse:GraphBLAS", ACM TOMS 45(4), + 2019. §4.1 is the data structure (two sparse formats, zombies, + pending tuples); §4.2.1 is the multiply, including the "n̄ < n/16" + hyper rule this chapter checks against `GB_defaults.h:20`. Cited + here from the author's accepted manuscript, which is titled + "Algorithm 9xx". Walked in + [reading-davis-toms19.md](reading-davis-toms19.md). +- Davis — "Parallel GraphBLAS with OpenMP", CSC '20. §3.1 names the + engine chosen for each algorithm, including the pull BFS. The + parallelism this chapter reads does not exist in the TOMS '19 + paper's version of the library. +- Gustavson, F. G. — "Two Fast Algorithms for Sparse Matrices: + Multiplication and Permuted Transposition", ACM TOMS 4(3), 1978, + 250-269, + [doi:10.1145/355791.355796](https://doi.org/10.1145/355791.355796) + — reference [1] in `GB_AxB_saxpy3.c:78-80`. Read in + [reading-gustavson-spgemm.md](reading-gustavson-spgemm.md). +- Nagasaka, Matsuoka, Azad, Buluç — "High-Performance Sparse + Matrix-Matrix Products on Intel KNL and Multicore Architectures", + ICPP '18, Article 34, + [doi:10.1145/3229710.3229720](https://doi.org/10.1145/3229710.3229720) + — reference [2] in `GB_AxB_saxpy3.c:82-86`; the hash method + saxpy3 implements. **Code** + - [SuiteSparse:GraphBLAS](https://github.com/DrTimothyAldenDavis/GraphBLAS) - `Source/convert/GB_conform.c`, `GB_conform_hyper.c`, - `GB_convert_sparse_to_bitmap_test.c`; `Source/mxm/GB_AxB_meta.c`, - `GB_AxB_saxpy3.c`, `GB_AxB_saxpy3_flopcount.c`, `GB_AxB_dot3.c` — - read the saxpy3 header comment (:22-60) twice; it's the scheduler - spec + at `1fd5475`. Read `Source/mxm/GB_AxB_saxpy3.c:22-86` first (the + scheduler spec), with + `Source/mxm/GB_AxB_saxpy3_slice_balanced.c:56-99` open beside it + because the spec's selection rule is out of date. The full anchor + table is above. +- [LAGraph](https://github.com/GraphBLAS/LAGraph) at `e2539e2` — + `LG_BreadthFirstSearch_SSGrB_template.c:312-313` and + `LAGr_TriangleCount.c:43-47` are the two callers this chapter + traces through the dispatcher. + +**Measured, in this repo** + +- `topics/20-graphblas/notes.md:13-14`, `:22-26` — the SpMV and + SpGEMM ladders Steps 4, 5 and 6 do arithmetic on. +- `topics/24-graph-algorithms/notes.md:5-7` — RMAT scale 16: max + degree 9,751 against a mean of 27.8. The skew that fine tasks + exist for. diff --git a/topics/21-formal/README.md b/topics/21-formal/README.md index 61e35a5..4ee57fa 100644 --- a/topics/21-formal/README.md +++ b/topics/21-formal/README.md @@ -118,7 +118,7 @@ interleaving of a 3-replica, 3-entry model: | config | states (distinct) | result | |---|---|---| | `SyncCommit = TRUE` | 2583 (1080), depth 14 | **Durability holds** | -| `SyncCommit = FALSE` | 123 checked | **violated at depth 5** | +| `SyncCommit = FALSE` | 183 (123), depth 5 | **violated at depth 5** | The counterexample TLC prints is the exact PostgreSQL `synchronous_commit = off` data-loss story: Append → Commit (no @@ -131,20 +131,31 @@ WalReplication.tla` (flip `SyncCommit` in the .cfg to see the trace). ## 4. Z3 — SMT in one paragraph -CDCL SAT core + theory solvers (linear arithmetic, arrays, -uninterpreted functions) cooperating via DPLL(T); quantifiers via -e-matching **over a congruence-closure e-graph** — the same structure -as egg, built for search instead of rewriting. Z3's modern e-graph -(`src/ast/euf/euf_egraph.h:23`) literally cites egg's deferred -congruence repair. Databases meet Z3 in query equivalence checking -(Cosette, topic 16) and symbolic execution of UDFs. +A SAT core with two-watch literals, lemma learning from conflict +clauses, phase caching and non-chronological backtracking (TACAS'08's +own list) plus theory solvers (linear arithmetic, arrays, uninterpreted +functions); quantifiers via e-matching **over a congruence-closure +e-graph** — the same structure as egg, built for search instead of +rewriting. Note what the paper does *not* say: it names Nelson–Oppen +only as the traditional method Z3 **avoids**, in favour of model-based +theory combination (its ref [5]), and "DPLL(T)" appears exactly once, +under relevancy propagation. Z3's modern e-graph +(`src/ast/euf/euf_egraph.h:22-23`) mentions egg, but carefully: the +worklist "is in reality inherited from the legacy SMT solver. It is +*claimed* to have the same effect as delayed congruence table +reconstruction from egg." Databases meet Z3 in query equivalence +checking (Cosette, topic 16) and symbolic execution of UDFs. ## 5. Lean 4 — proofs, and a runtime worth reading Proofs are unbounded (no MaxLog=3), but cost weeks not days. Lean's -own runtime is a systems story: Perceus reference counting with -reuse tokens gives functional-but-in-place updates — an RC design -directly relevant to any Rust engine tempted by `Arc` everywhere. +own runtime is a systems story: **Counting Immutable Beans** (Ullrich & +de Moura, IFL 2019) reference counting with reuse tokens gives +functional-but-in-place updates — an RC design directly relevant to any +Rust engine tempted by `Arc` everywhere. Perceus is Koka's descendant of +it, and says so: its §5 is "closely based on the reference counting +algorithm in the Lean theorem prover as described by Ullrich and +de Moura". M21 taste: prove one delta-matrix invariant (`DP ∩ M = ∅` preserved by set/remove) in Lean, and compare with the same property as a proptest (topic 16). diff --git a/topics/21-formal/experiments/.gitignore b/topics/21-formal/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/21-formal/experiments/.gitignore +++ b/topics/21-formal/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/21-formal/notes.md b/topics/21-formal/notes.md index 0d02ec6..d8db076 100644 --- a/topics/21-formal/notes.md +++ b/topics/21-formal/notes.md @@ -67,45 +67,18 @@ Surprises / dead ends: ## Questions from the reading guides -### AWS CACM'15 (reading-aws-cacm15.md) - -1. Which capstone protocol clears the spec cost/benefit bar: -2. 35-step vs our 5-step trace — reachable-but-rare: -3. TLA+ Next action vs proptest state-machine transition: -4. Small-scope hypothesis: protocols vs B+tree edge cases: -5. Keeping spec and code honest in CI: - -### egg POPL'21 (reading-egg-popl21.md) - -1. Hand-trace (a*2)/2 unions; where (/ 2 2) meets 1: -2. Why memo re-canonicalization loops to fixpoint: -3. machine.rs Scan cost; classes_by_op index: -4. Assoc+comm growth per iteration; which limit trips: -5. Cascades memo vs e-graph — what each has the other lacks: - -### Z3 TACAS'08 (reading-z3-tacas08.md) - -1. Why Z3's e-graph needs justifications, egg doesn't: -2. Deferred rebuild vs backtracking trail: -3. x/x→1 soundness as SMT query (ints vs reals): -4. Nelson-Oppen equality exchange ↔ join-key exchange: -5. Trigger selection = index choice of SMT: - -### Specifying Systems + raft.tla (reading-tlaplus-raft.md) - -1. Rejoin(r) → what invariant breaks → why terms exist: -2. Longest-log failover: quorum-intersection argument + bad trace: -3. What "ship everything atomically" would hide: -4. MVCC visibility as TLA+ sketch (M21 outline): -5. Why stuttering is essential for refinement: - -### Beans + Perceus (reading-lean-perceus.md) - -1. Arc costs that borrow inference eliminates: -2. When the RC==1 reuse check costs more than it saves: -3. Garbage-free peak memory ↔ buffer pool budgets: -4. Proof vs TLC vs proptest ranking for DP∩M=∅: -5. Rust's equivalent of Koka's no-hidden-aliasing: +Each guide now ends with its own `## Questions (answer in notes.md)` +list — six per guide, and they changed when the guides were rewritten +against the depth rules. Answer them against the guide you just read +rather than against a copy that can drift out of sync; several of the +old entries here encoded claims the rewrite corrected (Z3 does *not* use +Nelson–Oppen, and Lean's runtime is Beans, not Perceus). + +- [reading-aws-cacm15.md](reading-aws-cacm15.md) — answers: +- [reading-egg-popl21.md](reading-egg-popl21.md) — answers: +- [reading-z3-tacas08.md](reading-z3-tacas08.md) — answers: +- [reading-tlaplus-raft.md](reading-tlaplus-raft.md) — answers: +- [reading-lean-perceus.md](reading-lean-perceus.md) — answers: ## Cross-topic threads diff --git a/topics/21-formal/reading-aws-cacm15.md b/topics/21-formal/reading-aws-cacm15.md index 549f426..c032c20 100644 --- a/topics/21-formal/reading-aws-cacm15.md +++ b/topics/21-formal/reading-aws-cacm15.md @@ -1,165 +1,492 @@ # Why AWS writes TLA+: exhaustively testable pseudo-code -The CACM 2015 experience report that moved TLA+ from academia to -industrial default for distributed protocols. Read it for the -*economics*, not the math: what class of bug justifies days of -spec-writing — and what a spec still can't do for you. Before the -paper, this chapter builds the concepts its argument rests on — -what a spec is, what model checking actually does, why testing -can't reach the bugs it finds — one step at a time. It frames every -other chapter in this topic. +The experience report that moved TLA+ from academia to industrial default for +distributed protocols. Read it for the *economics*, not the math: what class of +bug justifies days of spec-writing, what the specs actually cost in lines, and +what the method still cannot do — a boundary the authors state more bluntly than +most vendors would. This chapter builds the concepts the argument rests on +(what a spec is, what a model checker does, why testing cannot reach the bugs it +finds), then works the paper's own table into a cost-per-bug figure, then quotes +the limitations verbatim, because they are the half most summaries drop. + +Every figure below comes from the paper's table *Applying TLA+ to some of our +more complex systems* or from the named narrative section; the paper is +Newcombe, Rath, Zhang, Munteanu, Brooker and Deardeuff, dated 29 September 2014 +and published as *How Amazon Web Services Uses Formal Methods*, CACM 58(4), +April 2015. It frames every other chapter in this topic. ## The problem in one sentence -S3's replication protocol had a data-loss bug that required a -specific 35-step interleaving of events to trigger — design review, -code review, and testing all missed it, because no human or test -generator reliably explores 35 steps deep. +DynamoDB's replication and group-membership design had a data-loss bug whose +**shortest** error trace was **35 high-level steps**, and it "had passed +unnoticed through extensive design reviews, code reviews, and testing" — because +no human and no test generator reliably explores interleavings that deep, while +a breadth-first search does not get bored. ## The concepts, step by step ### Step 1 — a specification is the design, written so a machine can explore it -A **specification** (spec) is a description of a system as a state -machine: the variables that make up a **state**, the initial -states, and the allowed transitions between states. Nothing about -threads, packets, or code — just "from this state, these next -states are legal." **TLA+** is a language for writing exactly -that, and it deliberately reads like pseudo-code with math -instead of control flow. The point of the formality is not rigor -for its own sake: a design written this way can be *executed -exhaustively* by a tool, while a design written in prose can only -be reviewed by tired humans. (The companion chapter, -[reading-tlaplus-raft.md](reading-tlaplus-raft.md), teaches the -language itself.) - -### Step 2 — model checking: enumerate every reachable state - -A **model checker** (TLC, for TLA+) takes a spec plus fixed small -parameters — 3 replicas, 3 log entries — and does breadth-first -search over the *entire* reachable state graph, checking a stated -**invariant** (a property that must hold in every reachable state, -e.g. "committed data survives failover") at each state. Contrast -the testing spectrum (topic 16): a test — even a -property-based-test generator — *samples* behaviors; TLC -*enumerates* them. Our WalReplication model is ~1080 distinct -states, checked in under a second; when the invariant fails, TLC -prints the exact step-by-step trace that breaks it. The -limitation is equally crisp: it checked 3 replicas × 3 entries, -nothing more — that gap is step 6. - -### Step 3 — the core claim: human intuition fails at ~35 steps - -S3's replication bug needed a 35-step interleaving to trigger. -Design reviews, code review, and testing all missed it. TLC found -it, because exhaustive breadth-first search doesn't get bored: -depth 35 is just another BFS frontier. The paper's engineers -report the same experience repeatedly — humans reason reliably -about interleavings a handful of steps deep, and distributed -protocol bugs live well past that horizon. This is the paper's -answer to "we already review our designs carefully": review -quality is not the bottleneck; the state space is. - -### Step 4 — the economics: spec size vs payoff - -The trade the paper is actually selling: 2-3 weeks to a first -useful spec, against design bugs found *before implementation*: - -``` - spec size vs payoff (paper's table, paraphrased) - S3 repl. ~800 lines 2 design bugs, one 35-step - DynamoDB ~1000 lines 3 design bugs pre-impl - EBS ~450 lines design confirmed (also a win) -``` - -DynamoDB's ~1000-line spec found 3 design bugs, one requiring a -fundamental change — the cheapest possible time to find it. Note -the EBS row: finding *no* bugs is also a payoff (confidence in the -design), which matters when deciding whether specs are worth it -for protocols that turn out fine. - -### Step 5 — the pitch that worked: "exhaustively testable pseudo-code" - -AWS did not sell "formal verification" internally — that phrase -promises proofs and demands mathematicians. The pitch that worked: -engineers write the spec *as the design doc* (it reads like -pseudo-code), and model checking comes free. This reframing is -load-bearing: the spec has a reason to exist even before checking -(it forces precision about the design), and checking is then a -button, not a research project. Steal the framing for any tool -adoption argument: attach the new cost to an artifact people -already need. - -### Step 6 — model small, learn big: the small-scope hypothesis - -Checking 3 replicas × 3 entries (like our WalReplication) is not -a proof — the bug could in principle appear only at N=7. The -**small-scope hypothesis** is the empirical observation that -protocol design bugs almost never work that way: a broken quorum -or ordering argument breaks at the smallest size where the -concepts exist (usually 2-3 processes). So a model TLC can finish -in seconds still finds the real bugs. Know when the hypothesis -*fails*, though: bugs triggered by resource-boundary edge cases -(a B+tree page becoming exactly full — topic 3) are about -magnitudes, not protocol logic, and small models never reach them -— question 4. - -### Step 7 — what TLA+ did NOT do for them - -The honest half of the report, and the boundary of the tool: - -- No liveness in practice (they check safety; liveness is expensive - and fairness assumptions are subtle). -- No code conformance — the spec and the C++ can drift. (MongoDB - later attacked this with spec-driven test generation.) -- No performance modeling. - -The drift point deserves the most respect: TLC verified the -*design*, and nothing keeps the implementation honest against it -afterwards. Question 5 asks what our capstone CI could do about -that. +> **In:** a design that currently exists as prose, a whiteboard, or state +> machine diagrams. +> **Out:** the same design as a state machine a tool can execute, and the two +> languages AWS actually used to write them. + +A **specification** describes a system as a state machine: the variables that +constitute a **state**, the initial states, and the allowed transitions. Nothing +about threads, packets or code — just "from this state, these next states are +legal." **TLA+** is a language for writing exactly that, and it deliberately +reads like pseudo-code with mathematics instead of control flow. + +The paper is careful about a distinction most summaries lose. TLA+ ships with a +**second language, PlusCal**: + +> "TLA+ is accompanied by a second language called PlusCal which is closer to a +> C-style programming language, but much more expressive as it uses TLA+ for +> expressions and values. In fact, PlusCal is intended to be a direct +> replacement for pseudo-code. … PlusCal is automatically translated to TLA+ +> with a single key press. … Also, tools such as the TLC model checker work at +> the TLA+ level." + +This matters for reading the table in Step 4: **four of the six specs listed are +PlusCal, not TLA+**. "AWS writes TLA+" is true only in the sense that PlusCal +becomes TLA+ before anything checks it. + +The formality is not rigour for its own sake. A design written this way can be +executed exhaustively by a tool; a design written in prose can only be reviewed +by tired humans. The companion chapter, +[reading-tlaplus-raft.md](reading-tlaplus-raft.md), teaches the language itself. + +### Step 2 — model checking: enumerate, don't sample + +> **In:** a spec plus fixed finite parameters — 3 replicas, 3 log entries. +> **Out:** either "no reachable state violates the invariant" or a concrete +> counterexample trace, and a precise statement of what "enumerate" buys over +> "sample". + +A **model checker** — TLC, for TLA+ — takes a spec and a **model** (concrete +values for the constants) and searches the *entire* reachable state graph +breadth-first, checking a stated **invariant** (a predicate that must hold in +every reachable state, e.g. "committed data survives failover") at each state. + +Contrast the testing spectrum of topic 16: a test — even a property-based +generator — *samples* behaviours from a distribution you do not control; TLC +*enumerates* them. When the invariant fails, TLC prints the exact step-by-step +trace that breaks it, which is a debugging artefact a fuzzer's seed is not. + +This topic's own model, `specs/WalReplication.tla`, is the miniature: **1080 +distinct states** from **2583 generated**, search depth **14**, `Durability` +holds — measured in `notes.md`. Flip `SyncCommit` to `FALSE` and TLC reports +**123 distinct states**, depth **5**, and the invariant **VIOLATED** with a +trace. Two numbers, one button. + +The limitation is equally crisp: it checked 3 replicas × 3 entries and nothing +more. That gap is Step 6. + +### Step 3 — the 35-step claim, attributed correctly + +> **In:** Step 2's exhaustive search. +> **Out:** the paper's central evidence, with the right system attached to it — +> this is the fact most often mis-cited. + +The 35-step bug is **DynamoDB's**, not S3's. From the paper's *First Big Success +at Amazon* section: author T.R. (Tim Rath) built DynamoDB's replication and +fault-tolerance mechanisms, did extensive fault-injection testing with a +simulated network layer, stress-tested on real hardware, *and* wrote detailed +informal proofs — which "did indeed find several bugs in early versions of the +design." Then: + +> "This time the model checker found a bug that could lead to losing data if a +> particular sequence of failures and recovery steps was interleaved with other +> processing. This was a very subtle bug; the shortest error trace exhibiting +> the bug contained 35 high level steps." + +And the sentence that answers "but our reviews are good": + +> "The bug had passed unnoticed through extensive design reviews, code reviews, +> and testing, and T.R. is convinced that we would not have found it by doing +> more work in those conventional areas." + +Note the shape of the argument. It is not that AWS's engineers reason badly; it +is that review quality is not the bottleneck, the state space is. The paper +pre-empts the "but that combination is improbable" objection directly: +"historically, AWS has observed many combinations of events at least as +complicated as those that could trigger this bug." + +The checking itself was not free: the DynamoDB spec was checked with the +**distributed TLC model checker on a cluster of ten `cc1.4xlarge` EC2 instances, +each with 8 cores plus hyperthreads and 23 GB of RAM**. Hold that number against +the one-second local run of Step 2 — the cost of exhaustive search is entirely +set by the model size, and Step 6 is about choosing it. + +### Step 4 — the table, and the cost per bug you can compute from it + +> **In:** the paper's table *Applying TLA+ to some of our more complex systems* +> — six rows, each a system, a component, a line count excluding comments, and +> a benefit. +> **Out:** an arithmetic answer to "is this worth it", and a result that +> contradicts the intuition that bigger specs find more bugs. + +Here is the table as printed, with the language column made explicit: + +| System | Component | Lines | Language | Benefit (paper's words) | +|---|---|---|---|---| +| S3 | Fault-tolerant low-level network algorithm | 804 | PlusCal | Found 2 bugs. Found further bugs in proposed optimizations. | +| S3 | Background redistribution of data | 645 | PlusCal | Found 1 bug, and found a bug in the first proposed fix. | +| DynamoDB | Replication & group-membership system | 939 | TLA+ | Found 3 bugs, some requiring traces of 35 steps | +| EBS | Volume management | 102 | PlusCal | Found 3 bugs. | +| Internal distributed lock manager | Lock-free data structure | 223 | PlusCal | Improved confidence. Failed to find a liveness bug as we did not check liveness. | +| Internal distributed lock manager | Fault tolerant replication and reconfiguration algorithm | 318 | TLA+ | Found 1 bug. Verified an aggressive optimization. | + +**Work it.** Total spec lines: `804 + 645 + 939 + 102 + 223 + 318 = 3031`. +Explicitly counted bugs: `2 + 1 + 3 + 3 + 0 + 1 = 10`. That is **303 lines of +spec per design bug found**, across six specs and four systems. + +Now break it down per row and the average stops being the interesting number: + +- **EBS volume management: 102 lines, 3 bugs — 34 lines per bug.** The smallest + spec in the table has the best return by a factor of nine. +- **DynamoDB: 939 lines, 3 bugs — 313 lines per bug**, and it needed a ten-node + EC2 cluster to check. +- **The 223-line lock-free data structure found zero bugs**, and the paper says + why in the benefit column: "Failed to find a liveness bug as we did not check + liveness." That row is not a "confidence win" — it is a **miss**, and the + authors printed it. + +So the relationship between spec size and bugs found is, on this data, weakly +*negative*. What predicts a find is not spec length; it is whether the algorithm +had a subtle concurrency argument in it. Two of the rows also record bugs found +in *fixes and optimizations* — the S3 rows both do — which is the recurring +payoff nobody budgets for: the spec keeps earning after the first bug. + +Against those lines, the cost the paper quotes: engineers "from junior to +Principal have been able to learn TLA+ from scratch and get useful results in +**2 to 3 weeks**"; T.R. wrote the 939-line DynamoDB spec "in a couple of weeks"; +B.M. "spent two weeks learning TLA+ and writing the spec" and TLC "found the bug +in a few seconds." Adoption at the time of writing: TLA+ used on **10 large +complex real-world systems**, **7 teams** using it. + +### Step 5 — the pitch that worked, and why the wording is load-bearing + +> **In:** a proven technique and an engineering organisation that has heard +> "formal methods" before and did not like it. +> **Out:** the specific rhetorical moves the paper credits with adoption, and +> the transferable lesson. + +AWS did not sell "formal verification". The paper is explicit about the framing: + +> "Engineers think in terms of debugging rather than 'verification', so we +> called the presentation 'Debugging Designs'. Continuing that metaphor, we have +> found that software engineers more readily grasp the concept and practical +> value of TLA+ if we dub it: **Exhaustively testable pseudo-code**." + +And the omissions were deliberate too: + +> "We initially avoid the words 'formal', 'verification', and 'proof', due to +> the widespread view that formal methods are impractical. We also initially +> avoid mentioning what the acronym 'TLA' stands for, as doing so would give an +> incorrect impression of complexity." + +The reframing is load-bearing rather than cosmetic: PlusCal "is intended to be a +direct replacement for pseudo-code" (Step 1), so the spec has a reason to exist +*before* anyone checks it — it is the design document, written precisely — and +checking is then a button rather than a research project. The paper reports the +practice that followed: "first writing a conventional prose design document, +then incrementally refining parts of it into PlusCal or TLA+. Often this gives +important insights without ever going as far as a full specification or model +checking." + +Steal the structure for any tool-adoption argument: attach the new cost to an +artefact people already have to produce. + +### Step 6 — model small, learn big — and who actually said so + +> **In:** the observation that TLC only checked 3 replicas × 3 entries. +> **Out:** the justification for believing a small model anyway, attributed to +> the right source, plus the case where it fails. + +Checking 3 replicas × 3 entries is not a proof: the bug could in principle +appear only at N = 7. The empirical claim that protocol design bugs almost never +work that way — a broken quorum or ordering argument breaks at the smallest size +where the concepts exist, usually 2–3 processes — is the **small-scope +hypothesis**, and it is **Daniel Jackson's**, from the Alloy line of work +(*Software Abstractions*). It does not appear in this paper; do not cite it to +Newcombe et al. + +What the paper actually claims is narrower and worth quoting for its hedging: +the model checker verified a part of the DynamoDB algorithm "for a **sufficiently +large instance** of the system to give very high confidence that it is correct." +"Sufficiently large" is an engineering judgement about a particular model, not a +general hypothesis, and "very high confidence" is not "proof". + +The paper's own caveat, in its closing section, is blunter than either: "All +models are wrong, some are useful." + +Know where the hypothesis fails. Bugs triggered by resource-boundary edge cases +— a B+tree page becoming exactly full, topic 3 — are about magnitudes, not +protocol logic, and a model small enough for TLC to finish never reaches them +(question 4). + +### Step 7 — what formal specification is *not* good for + +> **In:** the successes of Steps 3 and 4. +> **Out:** the three boundaries the authors state themselves, each with the +> section it comes from — this is the honest half of the report. + +**Performance.** The section is titled *What Formal Specification Is Not Good +For*, and it names the failure mode precisely: "sustained emergent performance +degradation" — a momentary slowdown (say a Java GC pause) breaches client +timeouts, clients retry, retries add load, the server slows further. "In such +scenarios the system will eventually make progress; it is not stuck in a logical +deadlock, livelock, or other cycle. But from the customer's perspective it is +effectively unavailable." They considered specifying an upper bound on response +time as a real-time safety property and rejected it, because the underlying +disks, OS and network "do not support hard real-time scheduling or guarantees, +so real-time safety properties would not be realistic." The conclusion: "We +don't yet know of a feasible way to model a real system that would enable tools +to predict such emergent behavior." + +**Code conformance.** The section is titled *The Most Frequently Asked +Question*, and the answer is one sentence: "On learning about TLA+, engineers +usually ask, 'How do we know that the executable code correctly implements the +verified design?' **The answer is that we don't.**" They add that they know of +no tools "that can handle distributed systems as large and complex as those we +are building", and that conventional static analysis is "largely limited to +finding 'local' issues in the code, and cannot verify compliance with a +high-level specification." The paper's constructive answer is indirect: formal +methods help engineers find strong system invariants, and those become +assertions in the code. + +**Liveness.** Note that the paper does not make a general claim here. The only +evidence in the report is a single table cell — the 223-line lock-free data +structure, "Failed to find a liveness bug as we did not check liveness." That is +one team's choice on one spec, not a stated policy. Our own +`specs/WalReplication.tla` makes the same choice: its `.cfg` lists `TypeOK` and +`Durability` as `INVARIANTS`, both safety properties, and no `PROPERTIES` at all. + +The drift point deserves the most respect: TLC verified the *design*, and nothing +keeps the implementation honest against it afterwards. Question 5 asks what our +capstone CI could do about that. + +### Step 8 — the backstory, and why it is evidence rather than colour + +> **In:** the paper's *First Steps To Formal Methods* section. +> **Out:** the reason a tool choice was made, which is the part you can actually +> reuse when choosing one yourself. + +C.N. (Chris Newcombe) did not start with TLA+. He started dissatisfied with +systems that were "considered very successful, and yet bugs and operational +problems still remained", and observed that reactive mechanisms — pervasive +assertions, recovery-oriented computing — "cannot recover from the class of bugs +that cause permanent damage to customer data". + +He was moved off the bias against formal methods by **Pamela Zave's** Alloy work +finding serious bugs in the membership protocol of **Chord**, a design from "a +strong group at MIT" that had won a 10-year test-of-time award at SIGCOMM 2011. +He then evaluated Alloy himself and rejected it on expressiveness: "we could not +find a practical way in Alloy to represent rich data structures such as dynamic +sequences containing nested records with multiple fields." + +That is a reusable evaluation criterion, and it is why this topic's spec is TLA+ +rather than Alloy: a WAL is a *sequence*, and `specs/WalReplication.tla:40` +appends to one. ## How to read the paper (with the concepts in hand) -It's a short CACM piece — read all of it, in order. The sidebar -tables carry the economics (step 4); compare each project row -against the two-to-three-week spec cost as you go. Watch for the -S3 35-step story (step 3), the "exhaustively testable pseudo-code" -framing (step 5 — note *where* in the adoption story it appears), -and the closing candor about what the method doesn't cover -(step 7). Read it with our `specs/WalReplication.tla` in mind: -every claim the paper makes at S3 scale has a miniature -counterpart in that 94-line model. +It is a short CACM piece — read all of it, in order, in one sitting. + +- The **table** carries the economics (Step 4). Read the language column, not + just the line counts, and read the *Benefit* column as prose: two rows record + bugs found in proposed *fixes*, and one records a miss. +- ***The Value of Formal Methods for 'Real-world Systems'*** has the adoption + numbers (10 systems, 7 teams, 2–3 weeks). +- ***First Big Success at Amazon*** is the 35-step story (Step 3). Note how much + conventional verification T.R. had already done before TLA+ found it — that + ordering is the argument. +- ***Persuading More Engineers…*** has the pitch (Step 5) and the S3, EBS and + lock-manager stories that populate the table's other rows. +- ***What Formal Specification Is Not Good For*** and ***The Most Frequently + Asked Question*** are Step 7. Read them before you quote the successes. +- ***First Steps To Formal Methods*** is the Alloy-versus-TLA+ evaluation + (Step 8). + +Read it with `specs/WalReplication.tla` open: every claim the paper makes at S3 +scale has a miniature counterpart in that 92-line model, including the choice not +to check liveness. ## Questions (answer in notes.md) -1. Which capstone protocol clears the paper's cost/benefit bar for a - spec — MVCC visibility, delta-matrix `wait` concurrency, or WAL - replication — and which is fine with proptest alone (topic 16)? -2. The 35-step bug: what makes an interleaving reachable-but-rare? - Relate to why our SyncCommit=FALSE trace is only 5 steps (the - model has no noise to wade through). -3. "Exhaustively testable pseudo-code": how is a TLA+ `Next` action - different from a proptest state-machine transition (topic 16)? - What does TLC explore that proptest samples? -4. Why does the small-scope hypothesis hold for protocols but NOT - for, say, B+tree split bugs (topic 3) that need page-full edge - cases? +1. Compute lines-per-bug for each row of the paper's table, then rank the six. + What property of the *algorithm* — not the spec — predicts the rows at the + top? Which of the capstone's protocols has that property? +2. Which capstone protocol clears the paper's cost/benefit bar for a spec — + MVCC visibility, delta-matrix `wait` concurrency, or WAL replication — and + which is fine with proptest alone (topic 16)? Justify with a line estimate + and the 2–3 week figure. +3. The 35-step bug: what makes an interleaving reachable but rare? Relate it to + why our `SyncCommit = FALSE` counterexample is only **5 steps** deep — what + does the toy model not have that DynamoDB's did? +4. Why does the small-scope hypothesis hold for protocols but not for a B+tree + split bug (topic 3) that needs a page-full edge case? State the property of + the bug, not of the tool. 5. Spec-code drift: sketch how the capstone's CI could keep - WalReplication.tla honest against the real replication code. + `WalReplication.tla` honest against the real replication code. Use the + paper's own constructive answer (invariants become assertions) as the + baseline and say what it does and does not catch. +6. The paper reports 10 systems and 7 teams but only 6 specs in the table. What + would you want to know about the 4 unreported specs before treating the + 303-lines-per-bug figure as a planning number? ## Done when -- [ ] You can explain what model checking does and why exhaustive state enumeration is different in kind from testing. -- [ ] You can state the 35-step claim and say what makes an interleaving reachable but rare. -- [ ] You can explain the small-scope hypothesis and say where it holds (protocols) and where it does not. -- [ ] You can state the pitch — exhaustively testable pseudo-code — and say what a TLA+ `Next` action is that pseudo-code is not. -- [ ] You can name what TLA+ did not do for AWS. -- [ ] You wrote answers to all five questions in notes.md, including which capstone protocol clears the cost/benefit bar. +Answer each before unfolding it. + +- [ ] You can explain what model checking does and why exhaustive enumeration differs in kind from testing — with this topic's two measured numbers. + +
Answer + + TLC takes a spec plus a finite model and searches the *entire* reachable state + graph breadth-first, checking the invariant at every state; a test samples + behaviours from a distribution nobody fully controls. The difference is not + thoroughness, it is coverage semantics: TLC's "no violation" is a statement + about all reachable states of that model, and its failure output is a concrete + minimal-depth trace. + + Measured here (`notes.md`): `WalReplication.tla` with `SyncCommit = TRUE` gives + **2583 states generated, 1080 distinct, depth 14, `Durability` holds**; with + `SyncCommit = FALSE`, **123 distinct states, depth 5, VIOLATED** with a trace. + +
+ +- [ ] You can state the 35-step claim with the right system attached, and say what had already been tried. + +
Answer + + It is **DynamoDB's** replication and group-membership system — the 939-line + TLA+ spec — not S3's. Before TLA+, author T.R. had done extensive + fault-injection testing with a simulated network layer, long stress tests on + real hardware, *and* detailed informal proofs (which found several earlier + bugs). TLC then found a data-loss bug whose **shortest** trace was **35 high + level steps**, and which "had passed unnoticed through extensive design + reviews, code reviews, and testing." + + Checking it took the distributed TLC on **ten `cc1.4xlarge` EC2 instances, + 8 cores plus hyperthreads and 23 GB each**. + +
+ +- [ ] You can compute the paper's cost per bug and say why the average is the least useful number in the table. + +
Answer + + Lines: `804 + 645 + 939 + 102 + 223 + 318 = 3031`. Bugs: `2 + 1 + 3 + 3 + 0 + 1 + = 10`. Average: **303 lines per bug**. + + The average hides the spread. EBS volume management is **102 lines / 3 bugs = + 34 per bug**; DynamoDB is **939 / 3 = 313**; the 223-line lock-free data + structure found **none** — and the paper says why: "Failed to find a liveness + bug as we did not check liveness." Spec size does not predict finds; the + presence of a subtle concurrency argument does. Two rows also record bugs found + in proposed *fixes and optimizations*, a payoff that arrives after the spec is + written and is missing from any per-bug figure. + +
+ +- [ ] You can say how many of the six specs were PlusCal rather than TLA+, and why the distinction matters. + +
Answer + + **Four of six are PlusCal** — 804 + 645 + 102 + 223 = **1774 lines** — and two + are TLA+ — 939 + 318 = **1257 lines**. PlusCal is "closer to a C-style + programming language" and "intended to be a direct replacement for + pseudo-code"; it is translated to TLA+ "with a single key press", and "tools + such as the TLC model checker work at the TLA+ level." + + It matters twice: for the adoption argument (Step 5's pitch is literally about + pseudo-code, and the language that looks like pseudo-code is the one most of + the specs are written in), and for reading line counts (a PlusCal line and a + TLA+ line are not the same unit of work). + +
+ +- [ ] You can attribute the small-scope hypothesis correctly and quote what the paper itself claims instead. + +
Answer + + The **small-scope hypothesis is Daniel Jackson's**, from the Alloy line of work + (*Software Abstractions*). It is not in this paper. Newcombe et al. claim + something narrower: the model checker verified part of the DynamoDB algorithm + "for a **sufficiently large instance** of the system to give very high + confidence that it is correct" — an engineering judgement about one model, and + "very high confidence", not proof. Their own caveat is "All models are wrong, + some are useful." + + Where it fails: bugs about magnitudes rather than protocol logic — a page + becoming exactly full, a counter wrapping — because those need a scale the + model deliberately does not have. + +
+ +- [ ] You can name the three things the paper says formal specification did not do for AWS, and cite the section for each. + +
Answer + + 1. **Emergent performance degradation** — section *What Formal Specification Is + Not Good For*. Real-time safety properties were rejected as unrealistic on + infrastructure without hard real-time guarantees: "We don't yet know of a + feasible way to model a real system that would enable tools to predict such + emergent behavior." + 2. **Code conformance** — section *The Most Frequently Asked Question*: "How do + we know that the executable code correctly implements the verified design? + The answer is that we don't." No tools at that scale; static analysis finds + only local issues. + 3. **Liveness** — but carefully: the only evidence is one table cell, the + 223-line lock-free spec, "Failed to find a liveness bug as we did not check + liveness." That is a choice on one spec, not a stated policy. Our + `WalReplication.cfg` makes the same choice: two `INVARIANTS`, no + `PROPERTIES`. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the lines-per-bug ranking and which capstone protocol clears the bar. + +
Answer + + The ranking to check yours against: EBS 34, S3-network 402, S3-redistribution + 645, DynamoDB 313, lock-manager replication 318, lock-manager lock-free ∞ (zero + bugs). What the top rows share is a fault-tolerance or concurrency argument + whose correctness depends on interleaving, which is exactly what enumeration + buys you and sampling does not. + + Record the *reasoning* for the capstone choice, not just the pick: a protocol + earns a spec when its correctness argument is about orderings of concurrent + events across failure, and when getting it wrong loses data rather than + degrades performance. WAL replication under failover is that; a single-node + page-split boundary is not. + +
## References **Papers** -- Newcombe, Rath, Zhang, Munteanu, Brooker, Deroche — "How Amazon - Web Services Uses Formal Methods" (CACM 2015) — short; read all - of it, the sidebar tables carry the economics +- Chris Newcombe, Tim Rath, Fan Zhang, Bogdan Munteanu, Marc Brooker, Michael + Deardeuff — *How Amazon Web Services Uses Formal Methods*, CACM 58(4), April + 2015 (the preprint is titled *Use of Formal Methods at Amazon Web Services*, + dated 29 September 2014). Short; read all of it. The table *Applying TLA+ to + some of our more complex systems* carries the economics of Step 4; + *What Formal Specification Is Not Good For* and *The Most Frequently Asked + Question* carry Step 7. +- Pamela Zave — *Using Lightweight Modeling to Understand Chord* — the Alloy + work the paper credits with overcoming its own bias against formal methods + (Step 8). +- Daniel Jackson — *Software Abstractions: Logic, Language, and Analysis* — the + actual source of the small-scope hypothesis of Step 6, and of Alloy, the tool + AWS evaluated and rejected on expressiveness. + +**In this topic** +- `specs/WalReplication.tla` (92 lines) and `specs/WalReplication.cfg` — the + miniature: 3 replicas, `MaxLog = 3`, `Quorum = 2`, invariants `TypeOK` and + `Durability`, no liveness properties. +- `notes.md` — the measured TLC runs quoted in Steps 2 and 3. +- [reading-tlaplus-raft.md](reading-tlaplus-raft.md) — the language itself, and + what happens to the state space when the protocol gets real. diff --git a/topics/21-formal/reading-egg-popl21.md b/topics/21-formal/reading-egg-popl21.md index a7ba684..2c55a62 100644 --- a/topics/21-formal/reading-egg-popl21.md +++ b/topics/21-formal/reading-egg-popl21.md @@ -1,225 +1,799 @@ # egg: equality saturation with deferred rebuilding -egg is the e-graph library behind a wave of optimizer research — -and behind our `eqsat.rs` stub. Its POPL 2021 paper makes two -contributions worth reading the source for: **deferred rebuilding** -(batch congruence repair instead of fixing invariants after every -union) and **e-class analyses** (attach lattice facts like constant -values to classes). This chapter builds the data structure from its -parts — union-find, hashcons, congruence — then the saturation loop -and egg's two contributions, before pointing you at the code: the -`src/` tree is ~10K lines and half of that is `explain.rs`/tests — -you can read the core tonight. +The paper that fixes this topic's measured failure — using this topic's exact +expression. egg's POPL 2021 paper opens §2.2 by applying `(𝑎 × 2)/2 → (𝑎 ≪ 1)/2` +and observing that "applying strength reduction at this point prevents us from +canceling out 2/2", which is character-for-character what our `hand.rs` lane +does. This chapter builds the e-graph from its three parts — a union-find, a +map from ids to e-classes, and a hashcons — states the two invariants that +distinguish it from a plain union-find, then gets to the contribution: +**deferred rebuilding**, letting congruence go stale on purpose and repairing it +in one batch. Then it says exactly what egg's headline speedup was measured +against, because that number is quoted wrongly more often than not. + +Every code anchor below is `egraphs-good/egg` at the commit this repo pins, +**`f94c346`** (`resources/codebases.md` pin table), quoted with the line numbers +the code occupies at that commit. Every paper claim names the section, figure or +definition it came from in the POPL 2021 paper (arXiv:2004.03082). ## The problem in one sentence -A rewrite optimizer that applies rules in a fixed order, -destructively, can rewrite itself into a corner — `(a*2)/2` gets -stuck at cost 5 when strength-reduction fires first — and the fix -(keep *every* equivalent form, pick the best at the end) needs a -data structure that stores exponentially many terms in linear -space and repairs its invariants fast (egg's batching alone is -worth up to 88×). +Our hand-ordered rewriter answers `(a*2)/2` with `(a << 1) / 2` and stops at +cost 5 after **one** rule firing, because rewriting *destructively* means the +best local move — strength reduction — deletes the `*2` that `x/x → 1` needed; +the repair is to stop deleting, which needs a data structure that holds every +equivalent form at once and can restore its congruence invariant fast enough +that holding them is affordable. ## The concepts, step by step -### Step 1 — union-find: merging sets in near-constant time +### Step 1 — union-find: what egg's actually is, not what the textbook says -A **union-find** (disjoint-set) structure maintains a partition of -ids into groups, under two operations: `find(x)` returns the -group's canonical representative id, and `union(x, y)` merges two -groups. With path compression (each `find` re-points ids directly -at the root) both run in effectively O(1) — amortized inverse -Ackermann, written O(α). It's the standard answer to "these two -things just became equal; remember that, cheaply, forever." egg's -entire union-find is 60 lines (`unionfind.rs`). +> **In:** ids that become equal over time, one pair at a time. +> **Out:** a canonical id per equivalence class, and an honest cost for `find` +> — which is *not* the O(α) you were taught, because egg's union-find is not +> the textbook one. -### Step 2 — the e-graph: a set of terms closed under equivalence +A **union-find** (disjoint-set) structure maintains a partition of ids under +two operations: `find(x)` returns the partition's canonical representative, and +`union(x, y)` merges two partitions. It is the standard answer to "these two +things just became equal; remember that, cheaply, forever." -An **e-graph** stores terms (expression trees like `(a*2)/2`) -compactly under an equivalence relation. Its parts: an **e-node** -is an operator whose children are *ids of equivalence classes* -rather than subterms (`*` with children [class 3, class 7]); an -**e-class** is a set of e-nodes that are all equal (identified by -a union-find id); a **hashcons** (a hash map from e-node to its -e-class id — topic 8's hash table, again) guarantees each distinct -e-node is stored once. +egg's entire union-find is `src/unionfind.rs`, **93 lines** — of which the +implementation is lines 1–51 and the rest is a `#[cfg(test)]` module. Here it +is, all of it: +```rust +// egg src/unionfind.rs, lines 30-50 — the whole implementation, verbatim + 30 pub fn find(&self, mut current: Id) -> Id { + 31 while current != self.parent(current) { + 32 current = self.parent(current) + 33 } + 34 current + 35 } + 36 + 37 pub fn find_mut(&mut self, mut current: Id) -> Id { + 38 while current != self.parent(current) { + 39 let grandparent = self.parent(self.parent(current)); + 40 *self.parent_mut(current) = grandparent; + 41 current = grandparent; + 42 } + 43 current + 44 } + 45 + 46 /// Given two leader ids, unions the two eclasses making root1 the leader. + 47 pub fn union(&mut self, root1: Id, root2: Id) -> Id { + 48 *self.parent_mut(root2) = root1; + 49 root1 + 50 } +``` + +Three things the code says that the textbook does not: + +- `find` (30–35) does **no path compression at all**. It walks to the root and + leaves the tree exactly as it found it. Only `find_mut` (37–44) compresses, + and it does **path halving** — each visited node is re-pointed at its + *grandparent*, not at the root. Halving is one pointer write per step instead + of a second pass; it gets the same asymptotic bound as full compression, but + "re-points ids directly at the root" is not what line 40 does. +- `union` (47–50) is **unconditional**: `root1` always wins. There is no + union-by-rank and no rank array in the struct. +- The balancing decision therefore lives one level up. `EGraph::perform_union` + picks which root survives by **parent-list length**, not by tree height: + +```rust +// egg src/egraph.rs, lines 1170-1175 and 1182 — the leader choice, elided between + 1170 // make sure class2 has fewer parents + 1171 let class1_parents = self.classes[&id1].parents.len(); + 1172 let class2_parents = self.classes[&id2].parents.len(); + 1173 if class1_parents < class2_parents { + 1174 core::mem::swap(&mut id1, &mut id2); + 1175 } + 1182 self.unionfind.union(id1, id2); +``` + +So the honest cost statement is: `find` is a pointer walk whose length is +bounded by how unbalanced the forest got; `find_mut` halves paths as it goes, +so repeated canonicalization of the same ids is cheap; and the heuristic that +keeps trees shallow optimises for *fewer parents to re-canonicalize later* +(Step 6's cost), not for tree height. The textbook O(α(n)) amortized bound +needs union-by-rank **and** compression on every find; egg has neither exactly. +It is fast for the reason the comment on line 1170 gives, and that reason is +about congruence work, not about the union-find. + +### Step 2 — the e-graph: three maps, and what "represents" means + +> **In:** the union-find of Step 1. +> **Out:** a structure that stores a *set of terms*, and a precise definition +> of which terms it stores — which you will need in Step 3 to count them. + +The paper's Definition 2.1 is a tuple `(U, M, H)`: + +- **U**, the union-find over e-class ids (Step 1). +- **M**, the **e-class map**, from e-class id to **e-class** — a set of e-nodes. +- **H**, the **hashcons**, a map from e-node to e-class id. (The paper's + footnote 3: the name evokes memoization, "since both avoid creating new + duplicates of existing objects." This is topic 8's hash table doing topic 8's + job.) + +An **e-node** is a function symbol paired with a list of **children e-class +ids** — not child subterms. `*` with children `[c3, c7]`, where `c3` and `c7` +are whole classes. The paper's footnote 4 flags what is unusual here: "making +e-classes but not e-nodes identifiable is unique to our definition" — an +e-class has an identity, an e-node does not. + +In egg the three maps are struct fields, and their doc comments already tell +you the whole staleness story of Step 6: + +```rust +// egg src/egraph.rs, lines 62-88 — EGraph's fields, non-essential attributes elided + 62 /// Stores each enode's `Id`, not the `Id` of the eclass. + 63 /// Enodes in the memo are canonicalized at each rebuild, but after rebuilding new + 64 /// unions can cause them to become out of date. + 66 memo: HashMap, + 67 /// Nodes which need to be processed for rebuilding. The `Id` is the `Id` of the enode, + 68 /// not the canonical id of the eclass. + 69 pending: Vec, + 70 analysis_pending: UniqueQueue, + 78 pub(crate) classes: HashMap>, + 81 classes_by_op: HashMap>, + 82 /// Whether or not reading operation are allowed on this e-graph. + 83 /// Mutating operations will set this to `false`, and + 84 /// [`EGraph::rebuild`] will set it to true. + 88 pub clean: bool, +``` + +`memo` is H, `classes` is M, `unionfind` is U, and `clean` (88) is the flag that +says whether the invariants currently hold. Note line 69: `pending` holds +**e-node ids, not e-class ids** — a detail that matters when you read +`process_unions` and expect the paper's pseudocode. + +**Representation** (Definition 2.3) is recursive: an e-node `f(a₁, a₂, …)` +represents the term `f(t₁, t₂, …)` when `M[aᵢ]` represents `tᵢ`; an e-class +represents a term if any of its e-nodes do; the e-graph represents a term if +any of its e-classes do. The consequence is the compression: because children +are *classes*, one `/` e-node with children `[c_mul, c_two]` represents `(a*2)/2` +and `(a<<1)/2` simultaneously, as soon as `a*2` and `a<<1` are in `c_mul`. + +### Step 3 — count the terms an e-graph represents + +> **In:** Definition 2.3 from Step 2, and the paper's Figure 2, which is +> literally our trap expression. +> **Out:** a number, computed — the reason "exponentially many terms in linear +> space" is not a slogan. + +Take the paper's Figure 2a: the e-graph containing just `(a×2)/2`. Four +e-classes, one e-node each: `{a}`, `{2}`, `{*}` with children `[{a},{2}]`, +`{/}` with children `[{*},{2}]`. Terms represented: **1**. + +Now count with the multiplicative rule that Definition 2.3 gives you: an +e-class represents `Σ over its e-nodes` of `Π over that e-node's children` of +(terms the child class represents). + +**Figure 2b**, after `x×2 → x≪1`: the `*` class now holds two e-nodes, `*` and +`<<`, and a new class `{1}` appears. Class `{a}` = 1 term, `{2}` = 1, `{1}` = 1. +The mul class = `(*: 1×1) + (<<: 1×1)` = **2**. The root `/` class = +`/: 2 × 1` = **2** terms — `(a*2)/2` and `(a<<1)/2` — from **5** e-nodes. + +**Figure 2c**, after `(x×y)/z → x×(y/z)`: the root class gains a `*` e-node +whose children are `{a}` and a new `/` class holding `2/2`. Root = +`(/: 2×1) + (*: 1 × 1)` = **3** terms from 7 e-nodes. + +**Figure 2d**, after `x/x → 1` and `1×x → x`: the paper's own caption says it — +"The resulting e-graph has a cycle, representing infinitely many expressions: +`a`, `a×1`, `a×1×1`, and so on." The class containing `a` now also contains a +`*` e-node one of whose children *is that same class*. Count with the rule +above and the sum diverges. Nine e-nodes; **infinitely many** terms. + +That is the whole bargain in four pictures: **1 → 2 → 3 → ∞ terms, from 4 → 5 → +7 → 9 e-nodes.** And the answer we want, `a`, is now in the same e-class as the +input, so extraction (Step 8) finds it at cost 1 — against our hand rewriter's +measured cost 5. + +### Step 4 — the two invariants, stated exactly + +> **In:** the e-graph of Step 2. +> **Out:** the two properties `rebuild()` exists to restore, in the paper's own +> terms — you cannot reason about "letting them go stale" without them. + +**Congruence** (paper Definition 2.6): the equivalence over e-nodes must be +closed under congruence, `(≡node) = (≅*)`. Concretely: if `x ≡ y` then +`f(x) ≡ f(y)`. The paper adds the corollary people forget — "since identical +e-nodes are trivially congruent, this implies that an e-node must be uniquely +contained in a single e-class." **Deduplication is a consequence of congruence**, +not a separate rule. + +**The hashcons invariant** (Definition 2.7): `H` maps all *canonical* e-nodes to +their e-class ids — + + e-node n ∈ M[a] ⟺ H[canonicalize(n)] = find(a) + +where `canonicalize(f(a₁,a₂,…)) = f(find(a₁), find(a₂), …)` (Definition 2.2). +Its purpose is one line: when the invariant holds, `lookup(n) = H[canonicalize(n)]` +answers "is there already an e-class with an e-node congruent to `n`?" in one +hash probe. + +The cascade these force is the expensive part. After `union(a, b)`, every parent +e-node of the merged classes has a stale child id; re-canonicalizing it may make +it collide in `H` with another e-node, which means *those two are congruent*, so +their classes must be unioned too, which invalidates *their* parents — repeat to +fixpoint. This upward cascade is **congruence closure**, and it is precisely the +invariant egg chooses to let go stale. + +### Step 5 — equality saturation, and the trap it repairs + +> **In:** an e-graph plus a set of rewrite rules. +> **Out:** the read/write/rebuild loop, and the explicit connection to this +> topic's measured `hand.rs` failure. + +**Equality saturation** replaces ordered destructive rewriting with: seed an +e-graph with the input; **e-match** every rule's left-hand side against the whole +e-graph, collecting `(σ, c)` pairs where class `c` represents `ℓ[σ]`; apply each +by `merge(c, add(r[σ]))` — adding the right-hand side and *unioning* it with the +match, never deleting; repeat until **saturated** (no rule adds a node or +performs a merge) or a budget trips; then run an **extractor** to pick the +cheapest represented term. + +The paper's §2.2 names our failure for us: + +> "Consider applying a simple strength reduction rewrite: `(𝑎 × 2)/2 → (𝑎 ≪ 1)/2`. +> The new term carries no information about the initial term. Applying strength +> reduction at this point prevents us from canceling out `2/2`. In the compilers +> community, this classically tricky question of when to apply which rewrite is +> called the **phase ordering problem**." + +That is `topics/21-formal/experiments/src/hand.rs` in one paragraph. Our +rewriter's rule R2 (`x*2 → x<<1`) is tried before R4 (`(x*y)/z → x*(y/z)`), it +fires once, and R4 can never match again because the `*` node is gone. Measured +result, from this topic's `notes.md` baseline: output `(a << 1) / 2`, **cost 5, +1 rule firing**. An e-graph keeps `a*2` *and* `a<<1` in one class, so R4 still +matches, and the chain in Step 3's Figure 2c–2d runs to `a`, cost 1. + +egg's loop is one function, and its three phases are visible as three timers: + +```rust +// egg src/run.rs, lines 556-595 inside Runner::run_one — read, write, rebuild + 556 result = result.and_then(|_| { + 557 matches = self + 558 .scheduler + 559 .search_rewrites(i, &self.egraph, rules, &self.limits)?; + 568 let search_time = start_time.elapsed().as_secs_f64(); + 573 result = result.and_then(|_| { + 574 rules.iter().zip(matches).try_for_each(|(rw, ms)| { + 578 let actually_matched = self.scheduler.apply_rewrite(i, &mut self.egraph, rw, ms); + 587 self.check_limits() + 588 }) + 589 }); + 591 let apply_time = apply_time.elapsed().as_secs_f64(); + 594 let rebuild_time = Instant::now(); + 595 let n_rebuilds = self.egraph.rebuild(); +``` + +Search (556–566) is read-only over the *whole* e-graph — every rule sees the +same snapshot, so no rule can preempt another. Apply (573–589) only mutates. +`rebuild()` is called **once** (595), after all rules have applied. That +separation is the paper's Figure 5b, and Step 6 is why it is allowed. + +### Step 6 — deferred rebuilding: the contribution + +> **In:** the invariants of Step 4 and the phase-split loop of Step 5. +> **Out:** why batching congruence repair is asymptotically better, worked on +> concrete numbers, and where the deferral happens in egg's source. + +Traditional congruence closure restores congruence after **every** merge. egg +defers: `merge` records the work and returns; `rebuild()` does it all at once. +The deferral is one line — + +```rust +// egg src/egraph.rs, line 1159 and line 1190, inside perform_union + 1159 self.clean = false; + 1190 self.pending.extend(class2.parents.iter().copied()); ``` - e-class {a*2, a<<1} union-find: id → canonical id - / \ hashcons: e-node → e-class id - e-class {a} e-class {2} + +— and the batch drain is `process_unions`: + +```rust +// egg src/egraph.rs, lines 1346-1358 — the pending drain (analysis loop elided) + 1346 fn process_unions(&mut self) -> usize { + 1347 let mut n_unions = 0; + 1348 + 1349 while !self.pending.is_empty() || !self.analysis_pending.is_empty() { + 1350 while let Some(class) = self.pending.pop() { + 1351 let mut node = self.nodes[usize::from(class)].clone(); + 1352 node.update_children(|id| self.find_mut(id)); + 1353 if let Some(memo_class) = self.memo.insert(node, class) { + 1354 let did_something = + 1355 self.perform_union(memo_class, class, Some(Justification::Congruence)); + 1356 n_unions += did_something as usize; + 1357 } + 1358 } ``` -The compression is the point: because children are *classes*, one -e-node represents every combination of its children's forms — -`(a*2)/2` and `(a<<1)/2` share one `/` node. n e-nodes can -represent exponentially many distinct terms. - -### Step 3 — congruence: equal children make equal parents - -The invariant that makes an e-graph more than a union-find is -**congruence**: if x ≡ y, then f(x) ≡ f(y) — merging two classes -must also merge every pair of parent e-nodes that now have -identical (canonicalized) children. Mechanically: after -union(a, b), re-canonicalize every parent e-node of the merged -class; if two parents collide in the hashcons — they became the -same node — their classes are equal too, so union *them*, and -repeat to fixpoint. This upward cascade (**congruence closure**) -is the expensive part of every e-graph operation, and it's exactly -the invariant egg chooses to let go stale (step 5). - -### Step 4 — equality saturation: rewrite all ways, then pick - -**Equality saturation** replaces ordered, destructive rewriting -with: seed an e-graph with the input term; match every rewrite -rule against the whole e-graph; *apply* each match by `add`-ing -the right-hand side and `union`-ing it with the left — never -deleting anything; repeat until **saturated** (no rule adds -anything new) or a budget trips; then run an **extractor** with a -cost function to pick the cheapest term the graph now represents. -The trap it fixes: +Read line 1353 carefully, because it is the whole mechanism: re-canonicalize the +node (1352), re-insert into the hashcons, and **a returned old value means two +e-nodes now hash to the same key** — they are congruent, so union them (1355), +which pushes *their* parents back onto `pending`, which is why 1349 is a loop. +The recursion of Step 4's cascade is a worklist here. +`rebuild()` is the public entry point and does exactly two things plus logging: + +```rust +// egg src/egraph.rs, lines 1416-1444 — rebuild, logging elided + 1416 pub fn rebuild(&mut self) -> usize { + 1422 let n_unions = self.process_unions(); + 1423 let trimmed_nodes = self.rebuild_classes(); + 1443 debug_assert!(self.check_memo()); + 1444 self.clean = true; ``` - (a*2)/2 - hand (ordered): strength-reduce FIRST → (a<<1)/2 … stuck, cost 5 - egg (saturate): keep BOTH forms; (x*y)/z→x*(y/z) still matches - → a*(2/2) → a*1 → a, cost 1 + +**Work the asymptotics on numbers.** The paper's §3.2.1 gives two workloads. +Take the second: `w` terms each nested under `d` function symbols, +`f₁(f₂(…f_d(x₁)))` … `f₁(f₂(…f_d(x_w)))`, and a workload of `w−1` merges that +merge all the `x`s together. + +- *Eager.* Each `merge(xᵢ, xⱼ)` needs `O(d)` `repair` calls, one per layer of + `f`s. Over `w−1` merges: `O(wd)`. +- *Deferred.* All `w−1` merges happen first. The `x`s are now one e-class `c_x`, + so the **deduplicated** worklist has exactly one element. Repairing `c_x` + merges the `f_d` layer into one class; the worklist deduplicates to one + element again; repeat per layer. Total: `O(d)`. + +With `w = 100`, `d = 10`: eager does on the order of `100 × 10 = 1000` repair +calls; deferred does on the order of `10`. **A factor of `w` — the width — +disappears**, and it disappears because the worklist deduplicates. The paper's +first workload gives the same shape for hashcons updates: `O(n²)` eager against +`O(n)` deferred. + +This is the same move as topic 20's delta-matrix `wait` and topic 4's LSM +memtable flush: make the mutation O(1) by batching the expensive invariant +restoration, and pay once per batch instead of once per mutation. The price is +that between rebuilds the hashcons is *stale* — `memo` holds non-canonical keys +(the doc comment at `egraph.rs:63-64` says so) — which is safe only because +Step 5's phase split guarantees nobody reads the e-graph during the write phase. + +The paper's footnote 5 is worth the honesty: Z3's e-graph already separated read +and write phases "as an implementation detail"; egg is "the first algorithm to +take advantage of this by deferring invariant maintenance." + +### Step 7 — what the 88× actually measures + +> **In:** the deferred algorithm of Step 6. +> **Out:** the paper's number, with its baseline, its benchmark and its +> machine — because "egg is 88× faster" is false as usually stated. + +The figure is §3.4 and **Figure 6**. Read it precisely: + +- **The baseline is egg itself**, modified so that `rebuild` is invoked after + every merge. It is *not* another tool, not Z3, not a prior eqsat engine. The + paper is measuring one algorithmic change inside one codebase. +- **The benchmark is egg's own test suite** — the `math` (computer algebra) and + `lambda` (untyped-λ partial evaluator) test sets, **32 tests**. Eight of the + 32 hit the iteration limit of 100; the rest saturated. +- **Two numbers, not one.** Aggregated as a **geometric mean over the 32 tests**: + **88×** on *congruence closure alone*, and **21×** on the whole equality + saturation algorithm. Quoting 88× as the end-to-end speedup overstates it by + roughly 4×. +- **The machine** is a 2020 MacBook Pro, 2 GHz quad-core Intel Core i5, 16 GB. + +Two supporting figures matter more than the headline. **Figure 7** shows the +speedup is *asymptotic* — it grows with the cumulative number of rewrites +applied, which is what Step 6's `O(wd) → O(d)` predicts and a constant-factor +win would not. **Figure 8** correlates time spent in congruence maintenance with +the number of `repair` calls: Spearman **r = 0.98, p = 3.6e-47**. That is the +evidence that the count Step 6 reasoned about is the thing that costs time. + +One divergence to hold while reading the source: the paper's Figure 4 pseudocode +has a `repair(eclass)` method and a worklist of e-classes. Current egg has +neither — `process_unions` (`egraph.rs:1346`) works from a `Vec` of **e-node** +ids (`egraph.rs:67-69`) and there is no method named `repair` in the tree. The +algorithm is the same; the names are not. + +### Step 8 — e-matching is where the time goes when it is not congruence + +> **In:** the read phase of Step 5. +> **Out:** the cost model for finding matches, and the index that keeps it from +> being quadratic. + +egg compiles each pattern to a tiny virtual machine. Four instructions, not +three: + +```rust +// egg src/machine.rs, lines 24-29 — the complete instruction set + 24 enum Instruction { + 25 Bind { node: L, i: Reg, out: Reg }, + 26 Compare { i: Reg, j: Reg }, + 27 Lookup { term: Vec>, i: Reg }, + 28 Scan { out: Reg }, + 29 } ``` -The catch: the e-graph can blow up (associativity+commutativity -rules alone are exponential), so egg's `Runner` carries -node/iteration/time limits and reports a `StopReason` — saturation -is best-effort, a *search budget* like topic 10's join-order DP -cutoff. +`Bind` walks into an e-class's matching e-nodes and pushes their children into +registers; `Compare` checks two registers canonicalize to the same class (this +is how a pattern like `x/x` enforces that both `x`s are the *same* class); +`Lookup` looks a whole ground subterm up in the hashcons in one probe; `Scan` +is the expensive one: -### Step 5 — deferred rebuilding: the headline contribution +```rust +// egg src/machine.rs, lines 66-74 — Scan iterates every e-class in the e-graph + 66 Instruction::Scan { out } => { + 67 let remaining_instructions = instructions.as_slice(); + 68 for class in egraph.classes() { + 69 self.reg.truncate(out.0 as usize); + 70 self.reg.push(class.id); + 71 self.run(egraph, remaining_instructions, subst, yield_fn)? + 72 } + 73 return Ok(()); + 74 } +``` -Classic congruence closure (and old eqsat engines) restores the -congruence invariant after EVERY union — the full upward cascade -of step 3, every time. egg lets the e-graph go stale during a -batch of rule applications, then `rebuild()` repairs once: +`Scan` is `O(number of e-classes)` **per invocation**, and it recurses into the +rest of the program for each. That is why `classes_by_op` (`egraph.rs:81`) exists: +it indexes e-classes by the discriminant of the operators they contain, and the +pattern searcher consults it before falling back to a scan — +```rust +// egg src/pattern.rs, lines 300-304 — the op index short-circuits the scan + 300 fn search_with_limit(&self, egraph: &EGraph, limit: usize) -> Vec> { + 304 if let Some(ids) = egraph.classes_for_op(&key) { ``` - per-union repair: union → fix parents → fix grandparents → … - egg: union, union, union, … → rebuild (dedup work: - a class touched 10× is repaired once) + +Work it: a pattern rooted at `/` in an e-graph with 10,000 e-classes of which 40 +contain a `/` e-node costs 40 starting points through the index instead of +10,000 through `Scan` — **250× fewer** entries into the rest of the program. A +pattern whose root is a bare variable has no discriminant to index on and must +scan. + +The budget that stops the search from running forever: + +```rust +// egg src/run.rs, lines 343-345 — RunnerLimits defaults + 343 iter_limit: 30, + 344 node_limit: 10_000, + 345 time_limit: Duration::from_secs(5), ``` +`check_limits` (`run.rs:170`) tests them in the order time (176), nodes (181), +iterations (185), and the loop terminates with a `StopReason` (`run.rs:237`): +`Saturated`, `IterationLimit`, `NodeLimit`, `TimeLimit`, `Other`. Saturation is +best-effort — a *search budget*, exactly like topic 10's join-order DP cutoff. +A run that stops at `NodeLimit` has not proved anything; it has run out of money. + +### Step 9 — e-class analyses: a semilattice riding along with each class + +> **In:** an e-graph whose classes merge unpredictably. +> **Out:** the interface for attaching derived facts, and the algebraic +> condition that makes it well-defined under merging. + +An **e-class analysis** attaches a value `d_c` to every e-class. Paper §4.1 +gives three operations: `make(n)` produces the value for a new e-node, +`join(d₁, d₂)` combines the values of two classes being merged, and `modify(c)` +may optionally mutate the class. The domain and `join` must form a +**join-semilattice** — `join` associative, commutative and idempotent — because +merges happen in an order the analysis author does not control, and the result +must not depend on that order. + +The analysis invariant is `∀c. d_c = ⨅_{n∈c} make(n)` and `modify(c) = c`. + +In egg, `analysis_pending` (`egraph.rs:70`) is a `UniqueQueue` — deduplicating, +like the congruence worklist — and the second loop of `process_unions` +(`egraph.rs:1360-1371`) drains it, calling `N::remake`, `analysis.merge`, and +`N::modify` on any class whose data actually changed. + +The canonical instance is constant folding: the value is `Option`, `join` +is "agree or panic", and `modify` adds the literal e-node to the class when the +value becomes `Some`. Our `eqsat.rs` stub sidesteps this — `(/ 2 2)` folds via +the `div-same` rewrite instead — but the M21 planner stage is the interesting +version: carry **cardinality estimates** as the analysis and topic 10's +`estimate()` becomes a lattice value that merges when two plan alternatives are +proved equivalent. + +### Step 10 — extraction is where the guarantees stop + +> **In:** a saturated (or budget-stopped) e-graph. +> **Out:** one term, and a clear statement of which cost functions this can and +> cannot optimise. + ```rust -fn union(&mut self, a: Id, b: Id) { - let root = self.unionfind.union(a, b); // O(α) — and STOP: - self.pending.extend(self.classes[&root].parents()); // repair deferred -} - -fn rebuild(&mut self) { - while let Some((node, class)) = self.pending.pop() { - let node = node.canonicalize(&self.unionfind); // re-canon children - if let Some(old) = self.memo.insert(node, class) { - // hashcons collision = two nodes became equal children-wise: - // a DISCOVERED congruence — union them, which refills pending - self.union(old, class); // hence: loop to fixpoint - } - } -} +// egg src/extract.rs, lines 157-166 — AstSize, the default cost function + 157 pub struct AstSize; + 164 enode.fold(1, |sum, id| sum.saturating_add(costs(id))) ``` -Paper reports up to 88× from this alone. It is exactly the -delta-matrix `wait` (topic 20) / LSM memtable flush (topic 4) move: -make mutation O(1) by batching the expensive invariant restoration. -Z3's new e-graph adopted it (`euf_egraph.h:23` cites egg). The -subtlety worth holding: between rebuilds the hashcons is *stale* -(non-canonical keys), which is fine during rule application because -matching tolerates it — the invariant is needed at iteration -boundaries, not continuously. - -### Step 6 — e-class analyses: facts that ride along with classes - -An **e-class analysis** attaches a lattice value (a fact with a -defined way to merge two facts, like `Option` for "known -constant") to every e-class, maintained through merges -(`analysis_pending`, `egraph.rs:70`; `N::remake`/`merge` in -`process_unions`). The canonical one: constant folding — a class -carries `Option`; when it becomes `Some`, `modify` adds the -literal node, and extraction gets it for free. Our stub sidesteps -this (`(/ 2 2)` folds via the `div-same` rule), but M21's planner -stage would carry *cardinality estimates* as the analysis — topic -10's `estimate()` as a lattice. - -### Step 7 — extraction is the weak spot - -`find_best` is greedy per e-class — fixpoint of per-class -best-cost, optimal for tree cost like AstSize (count the nodes), -NOT optimal with sharing (DAG cost: a subterm used twice should be -priced once). `lp_extract.rs` does ILP extraction for that. -Planner analogy: greedy extraction ≈ picking the cheapest subplan -per group in a memo — which is exactly what a Cascades optimizer -does, and e-graph ≈ Cascades **memo** discovered independently -(question 5 pushes on what each side has that the other lacks). +`Extractor::find_best` (`extract.rs:225`) reads a table built by `find_costs` +(`extract.rs:254`), which is a **fixpoint**: repeatedly recompute each class's +best cost from its e-nodes' children's current best costs until nothing changes. + +This is optimal for a **local, tree-shaped** cost function — one where a term's +cost is a function of its node and its children's costs, and shared subterms are +paid for once per use. `AstSize` is exactly that: `1 + Σ children`. It is +**not** optimal under a DAG cost, where a subterm used twice should be priced +once; the paper's §4.3 notes that extraction with a local cost function can +itself be phrased as an e-class analysis, and points at other work (Wang et al. +2020; Wu et al. 2019) for the harder objectives. egg ships an ILP-based +extractor in `src/lp_extract.rs` for that case. + +The planner analogy is the reason this topic sits where it does: greedy +extraction is picking the cheapest subplan per group in a memo, which is what a +Cascades optimizer does. An e-graph *is* a Cascades memo discovered +independently — with congruence, which Cascades lacks, and without physical +properties and enforcers, which Cascades has (question 5). ## How to read the paper (with the concepts in hand) -- **§2** is the best e-graph intro in print — steps 1-4 with - pictures; skim if the steps above landed, read closely if not. -- **§3** is deferred rebuilding (step 5) — the invariant-staleness - argument and the 88× measurement; check their figure against the - `rebuild` pseudocode above. -- **§4** is e-class analyses (step 6) — read with the M21 - cardinality-lattice idea in mind. +Willsey et al., *egg: Fast and Extensible Equality Saturation*, POPL 2021, +arXiv:2004.03082. + +- **§2.1** — Definitions 2.1–2.7. Read Definition 2.6 and 2.7 slowly; they are + Step 4 and every later argument depends on them. +- **§2.2** — one page, and it contains our trap verbatim plus Figure 2, the + four-panel walkthrough Step 3 counted. Read Figure 2's caption; the phrase + "representing infinitely many expressions" is the point of the whole topic. +- **§3** — the contribution. §3.2.1's two worked examples are Step 6; §3.2.2 + proves termination by a lexicographic decrease of `(|I|, |W|)`; §3.4 with + Figures 6–8 is Step 7. Check Figure 6's caption for what the baseline is + before you quote the number. +- **§4** — e-class analyses (Step 9). §4.3's extraction paragraph is Step 10. +- **§5** — the implementation. Note the paper's own size claim: egg is "~5000 + lines of Rust, including code, tests, and documentation." +- **§6** — three case studies (6.1 Herbie, 6.2 Spores, 6.3 Szalinski). Skim + unless you want evidence that the library carried real work. ## Where each step lives in the code +Pinned at `egraphs-good/egg@f94c346`. Sizes are that commit's. + | file:line | step | what | |---|---|---| -| `unionfind.rs:30/:37/:47` | 1 | `find` (path-compressing in `find_mut`), `union` — 60 lines, the whole thing | -| `egraph.rs:66` | 2 | `memo: HashMap` — the hashcons; canonical only *after* rebuild | +| `unionfind.rs:30` | 1 | `find` — walks to root, **no** compression | +| `unionfind.rs:37` | 1 | `find_mut` — path **halving** (grandparent, line 40) | +| `unionfind.rs:47` | 1 | `union` — unconditional, `root1` wins. 93-line file, 51 lines of impl | +| `egraph.rs:66` | 2 | `memo: HashMap` — the hashcons H; stale between rebuilds (63-64) | +| `egraph.rs:69` | 2, 6 | `pending: Vec` — **e-node** ids, not class ids | +| `egraph.rs:78` | 2 | `classes` — the e-class map M | +| `egraph.rs:88` | 4 | `clean` — do the invariants currently hold? | | `egraph.rs:970` | 2 | `EGraph::add` — canonicalize children, memo lookup-or-insert | -| `egraph.rs:1147` | 3, 5 | `EGraph::union` — merge classes, push parents onto `pending` (:69) | -| `egraph.rs:1346` | 3, 5 | `process_unions` — drain `pending`: re-canonicalize node, re-insert into memo; a collision *is* a discovered congruence → recursive union | -| `egraph.rs:1416` | 5 | `rebuild` — the public batched-repair entry point | -| `machine.rs:8/:24` | 4 | pattern matching compiled to a tiny VM: `Bind`/`Scan`/`Compare` instructions over the e-graph | -| `run.rs:138/:161/:237` | 4 | `Runner`, `RunnerLimits` (iter/node/time), `StopReason` | -| `extract.rs:41/:116/:157/:225` | 7 | `Extractor`, `CostFunction`, `AstSize`, `find_best` (fixpoint of per-class best-cost) | - -Navigation advice: read `unionfind.rs` fully (it's 60 lines), then -`egraph.rs` by the anchors above, then `run.rs`'s loop, then -`extract.rs`. Skip `explain.rs` on the first pass — it's half the -tree and orthogonal. +| `egraph.rs:1147` | 5 | `EGraph::union` — the public merge | +| `egraph.rs:1170-1175` | 1 | the leader choice: fewer parents wins | +| `egraph.rs:1190` | 6 | `pending.extend(class2.parents…)` — the deferral, one line | +| `egraph.rs:1346` | 6 | `process_unions` — drain, re-canonicalize, memo collision ⇒ congruence | +| `egraph.rs:1416` | 6 | `rebuild` — `process_unions` + `rebuild_classes`, then `clean = true` | +| `egraph.rs:184` | 7 | `total_size()` = `memo.len()`, the node count `node_limit` watches | +| `machine.rs:24-28` | 8 | `Bind`, `Compare`, `Lookup`, `Scan` — the complete instruction set | +| `machine.rs:66-74` | 8 | `Scan` — `for class in egraph.classes()`, the quadratic risk | +| `pattern.rs:300-304` | 8 | `classes_for_op` short-circuit | +| `run.rs:525` | 5 | `run_one` — search (556), apply (573), one `rebuild` (595) | +| `run.rs:343-345` | 8 | defaults: 30 iterations, 10,000 nodes, 5 s | +| `run.rs:237` | 8 | `StopReason` | +| `extract.rs:157` | 10 | `AstSize` — `1 + Σ children` | +| `extract.rs:254` | 10 | `find_costs` — the fixpoint | + +Navigation advice: read `unionfind.rs` fully (it is 93 lines and you have +already seen 21 of them above), then `egraph.rs` by the anchors, then `run.rs`'s +`run_one`, then `extract.rs`. Skip `explain.rs` on the first pass — at 1962 +lines it is the largest file in the tree and it is orthogonal to everything +here. ## Questions (answer in notes.md) -1. Trace `(a*2)/2` by hand: which unions happen in iteration 1, and - in which e-class do `(/ 2 2)` and `1` meet? -2. Why must `memo` re-canonicalization happen in a loop (a repair - can create a new collision)? Find the fixpoint in - `process_unions` (:1346). -3. `machine.rs`: what does `Scan` cost when a pattern's root op has - thousands of e-nodes? Relate to `classes_by_op` (:81). -4. Assoc+comm on `+` alone: estimate e-graph growth per iteration on - a depth-8 sum. Which `RunnerLimit` trips first (predict, then - measure in the stub)? -5. Cascades memo vs e-graph: what does Cascades have that egg lacks - (physical properties, promises), and vice versa (congruence)? +1. Trace `(a*2)/2` by hand through iteration 1 of a saturating run: which + `merge` calls happen, in which e-class do `(/ 2 2)` and `1` meet, and how + many terms does the root class represent after each of Figure 2's panels? + (Step 3 did b–d; do 2a and check.) +2. `process_unions` (`egraph.rs:1349`) loops until `pending` is empty. Give a + concrete four-e-node example where one repair creates a *new* hashcons + collision, so a single pass would leave congruence broken. +3. Take an e-graph with 10,000 e-classes, 40 of which contain a `/` e-node. + Compute the starting points for a `/`-rooted pattern with and without + `classes_by_op`. Now do it for a pattern rooted at a bare pattern variable — + what changes, and why? +4. Associativity plus commutativity on `+` alone, applied to a depth-8 sum: + estimate e-node growth per iteration, then predict which of the three + `RunnerLimits` (30 iterations, 10,000 nodes, 5 s) trips first. Then measure + it in the stub. +5. Cascades memo against e-graph: name one thing Cascades has that egg lacks + (start with physical properties and enforcers) and one thing egg has that + Cascades lacks (start with congruence). Which of the two would our M21 + planner need first? +6. Step 7 says the 88× baseline is egg-with-eager-rebuild on egg's own 32-test + suite. Name one way that choice of baseline flatters the result, and one way + it is *more* honest than comparing against a different tool. ## Done when -- [ ] You can explain the e-graph as a set of terms closed under equivalence, and say what congruence adds to union-find. -- [ ] You can trace `(a*2)/2` by hand through iteration 1 and say which unions happen — then compare with the measured hand-rewriter result here, which gets stuck at `(a << 1) / 2`, cost 5. -- [ ] You can explain deferred rebuilding and why `memo` re-canonicalization needs a repair loop. -- [ ] You can explain what an e-class analysis is and give one useful example. -- [ ] You can say why extraction is the weak spot, and what a cost function cannot express. -- [ ] You wrote answers to all five questions in notes.md, including the growth estimate under associativity plus commutativity. +Answer each before unfolding it. + +- [ ] You can state both e-graph invariants precisely, and say which one implies that an e-node lives in exactly one e-class. + +
Answer + + **Congruence** (Definition 2.6): the equivalence over e-nodes is closed under + congruence, `(≡node) = (≅*)` — if `x ≡ y` then `f(x) ≡ f(y)`. **Hashcons** + (Definition 2.7): `n ∈ M[a] ⟺ H[canonicalize(n)] = find(a)`, where + `canonicalize(f(a₁,…)) = f(find(a₁),…)`. + + Uniqueness follows from **congruence**, and the paper spells out why: + identical e-nodes are trivially congruent, so if congruence holds they must be + in the same e-class. Deduplication is not an extra rule. + +
+ +- [ ] You can count the terms an e-graph represents, and produce the number for each panel of the paper's Figure 2. + +
Answer + + The rule from Definition 2.3: a class represents `Σ over its e-nodes` of + `Π over that e-node's children` of the child class's count. + + Figure 2a: 4 e-nodes, **1** term. 2b (after `x×2 → x≪1`): the mul class holds + `*` and `<<`, each contributing `1×1`, so the root `/` class is `2×1` = **2** + terms from 5 e-nodes. 2c (after `(x×y)/z → x×(y/z)`): the root gains a `*` + e-node, `(/: 2×1) + (*: 1×1)` = **3** terms from 7 e-nodes. 2d (after + `x/x → 1`, `1×x → x`): the class holding `a` now contains a `*` e-node with + itself as a child — a cycle — so the sum diverges: **infinitely many** terms + from 9 e-nodes. The paper's Figure 2d caption says exactly this. + +
+ +- [ ] You can explain deferred rebuilding, and show on numbers why it is asymptotically better rather than a constant-factor win. + +
Answer + + `perform_union` sets `clean = false` (`egraph.rs:1159`) and pushes the merged + class's parents onto `pending` (`egraph.rs:1190`), then returns. Repair happens + later, in `process_unions` (`egraph.rs:1346`), where each pending e-node is + re-canonicalized (1352) and re-inserted into `memo` (1353); a returned old + value means two e-nodes became congruent, so they are unioned (1355) — which + refills `pending`, hence the loop at 1349. + + Paper §3.2.1's second workload: `w` terms nested `d` deep, `w−1` merges of the + leaves. Eager: `O(d)` repairs per merge, `O(wd)` total. Deferred: all merges + land first, the deduplicated worklist holds one class per *layer*, so `O(d)` + total. At `w = 100, d = 10` that is ~1000 repairs against ~10. The saved + factor is `w`, the *width* — it grows with the workload, which is why + Figure 7 shows the speedup growing with cumulative rewrites instead of + flattening. + + The correctness condition is Step 5's phase split: the hashcons is stale + between rebuilds (`egraph.rs:63-64`), and that is safe only because search is + read-only and apply is write-only, with one `rebuild()` between them + (`run.rs:595`). + +
+ +- [ ] You can quote egg's headline speedup with its baseline, benchmark and caveat, and say what the *end-to-end* number is. + +
Answer + + §3.4, Figure 6: a **geometric mean over 32 tests** of egg's own `math` and + `lambda` test suites, on a 2020 MacBook Pro (2 GHz quad-core i5, 16 GB). + **88× on congruence closure**, **21× on the whole equality-saturation + algorithm** — the end-to-end figure is 21×, not 88×. The baseline is **egg + itself, modified to call `rebuild` after every merge**, not a competing tool. + Eight of the 32 tests hit the iteration limit of 100 rather than saturating. + + Figure 7 shows the speedup is asymptotic in cumulative rewrites; Figure 8 + correlates congruence time with `repair` call count at Spearman r = 0.98, + p = 3.6e-47. + +
+ +- [ ] You can name the four e-matching instructions and compute what the `classes_by_op` index saves. + +
Answer + + `Bind`, `Compare`, `Lookup`, `Scan` (`machine.rs:24-28`). `Bind` enters an + e-class's matching e-nodes; `Compare` (75-78) checks two registers have the + same canonical class, which is how `x/x` requires the *same* `x`; `Lookup` + probes the hashcons for a whole ground subterm; `Scan` (66-74) iterates + **every** e-class via `for class in egraph.classes()`. + + With 10,000 e-classes of which 40 hold a `/` e-node, `classes_for_op` + (`pattern.rs:304`) gives 40 starting points instead of 10,000 — 250× fewer + entries into the rest of the program. A pattern rooted at a bare variable has + no operator discriminant, so the index cannot help and `Scan` runs. + +
+ +- [ ] You can say what an e-class analysis requires algebraically, and why that requirement exists. + +
Answer + + §4.1: `make(n)`, `join(d₁,d₂)`, `modify(c)`, with the domain and `join` + forming a **join-semilattice** — associative, commutative, idempotent. The + reason is that merges arrive in an order the analysis author does not control + and can repeat (a class may be merged many times before a rebuild), so the + accumulated value must be independent of order and of repetition. The + invariant is `d_c = ⨅_{n∈c} make(n)` and `modify(c) = c`. egg drains + `analysis_pending` — a `UniqueQueue` (`egraph.rs:70`) — in the second loop of + `process_unions` (1360-1371). + +
+ +- [ ] You can say which cost functions extraction optimises correctly and which it does not. + +
Answer + + `find_costs` (`extract.rs:254`) is a fixpoint over per-class best cost, and + `find_best` (`extract.rs:225`) reads the resulting table. This is optimal for a + **local tree cost** — cost of a node is a function of the node and its + children's costs, sharing paid per use. `AstSize` (`extract.rs:157`, cost at + 164) is `1 + Σ children`, exactly that shape. + + It is **not** optimal for a DAG cost, where a subterm referenced twice should + be charged once; nor for any objective that is not decomposable per class + (e.g. a global register-pressure or code-size budget). §4.3 notes extraction + can be expressed as an e-class analysis and cites other work for the harder + objectives; egg ships `src/lp_extract.rs` for an ILP formulation. + +
+ +- [ ] You can connect the paper's §2.2 to this topic's measured lane, in both directions. + +
Answer + + §2.2's sentence — applying `(𝑎 × 2)/2 → (𝑎 ≪ 1)/2` "prevents us from canceling + out `2/2`" — is the specification of our failure. `hand.rs` tries R2 + (`x*2 → x<<1`) before R4 (`(x*y)/z → x*(y/z)`), R2 fires, and R4 can never + match again. Measured: `(a << 1) / 2`, **cost 5, 1 rule firing**. + + In the other direction, the e-graph run of Figure 2b–2d shows why saturation + escapes it: `a*2` and `a<<1` sit in one e-class, so R4 still matches the `*` + e-node, `2/2` folds to `1`, `1×a` folds to `a`, and `a` ends up in the root + class at cost 1. The whole difference is that `merge` adds an alternative + where `hand.rs` performs a replacement. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the associativity-plus-commutativity growth estimate and which `RunnerLimit` you predicted. + +
Answer + + The prediction worth writing down before measuring: commutativity alone at + most doubles the e-nodes at each `+`; associativity on a depth-8 right-nested + sum generates the distinct parenthesisations, which is Catalan-like growth in + the number of leaves. With defaults of 30 iterations, 10,000 nodes and 5 s + (`run.rs:343-345`), `NodeLimit` is the plausible first trip for a depth-8 sum, + because node count grows per-iteration while iterations are capped at 30 and + the per-iteration work is still small enough to stay inside 5 s early on. + Whatever you predict, record it before running — the point of the worksheet in + `notes.md` is the gap between the prediction and the measurement, and a run + that stops at `NodeLimit` has proved nothing about saturation. + +
## References **Papers** -- Willsey, Nandi, Wang, Flatt, Tatlock, Panchekha — "egg: Fast and - Extensible Equality Saturation" (POPL 2021, - [arXiv:2004.03082](https://arxiv.org/abs/2004.03082)) — §2 is the - best e-graph intro in print; §3 deferred rebuilding, §4 analyses - -**Code** -- [egg](https://github.com/egraphs-good/egg) `src/unionfind.rs`, - `src/egraph.rs` (add :970, union :1147, process_unions :1346, - rebuild :1416), `src/machine.rs`, `src/run.rs`, `src/extract.rs` - — read fully; skip `explain.rs` on the first pass +- Max Willsey, Chandrakana Nandi, Yisu Remy Wang, Oliver Flatt, Zachary Tatlock, + Pavel Panchekha — *egg: Fast and Extensible Equality Saturation*, POPL 2021 + ([arXiv:2004.03082](https://arxiv.org/abs/2004.03082)). §2.1 Definitions + 2.1–2.7; §2.2 the phase-ordering paragraph and Figure 2; §3 rebuilding, with + §3.2.1's two worked workloads and §3.2.2's termination proof; §3.4 and + Figures 6–8 the measurement; §4 e-class analyses; §4.3 extraction. +- Greg Nelson — *Techniques for Program Verification*, Stanford PhD thesis, + 1980, Chapter 7 — the upward-merging congruence closure that §3.2.2's proof + reduces to. +- Leonardo de Moura, Nikolaj Bjørner — *Efficient E-Matching for SMT Solvers*, + CADE 2007 — the e-matching procedure `machine.rs` implements. Read alongside + `reading-z3-tacas08.md`. + +**Code** — `egraphs-good/egg` at `f94c346` + +| File | Lines | What | +|------|-------|------| +| `src/unionfind.rs` | 93 | U. Impl is lines 1–51; read it whole | +| `src/egraph.rs` | 1511 | M, H, `add`, `union`, `process_unions`, `rebuild` | +| `src/machine.rs` | 345 | the e-matching VM | +| `src/pattern.rs` | 536 | patterns, and the `classes_for_op` short-circuit | +| `src/run.rs` | 994 | `Runner`, the three-phase loop, limits, `StopReason` | +| `src/extract.rs` | 315 | `Extractor`, `CostFunction`, `AstSize` | +| `src/rewrite.rs` | 703 | `Rewrite`, `Searcher`, `Applier` | +| `src/language.rs` | 1001 | `Language`, `define_language!` — what `eqsat.rs` uses | +| `src/explain.rs` | 1962 | proof production. Skip on the first pass | + +**In this topic** +- `experiments/src/hand.rs` — the ordered rewriter whose rule order is the trap; + the test `the_ordering_trap` asserts the cost-5 answer. +- `experiments/src/eqsat.rs` — the stub you fill in, with the rewrite list in + its module docs. +- `notes.md` — the measured baseline these guides quote. diff --git a/topics/21-formal/reading-lean-perceus.md b/topics/21-formal/reading-lean-perceus.md index 5a2a503..6651cfc 100644 --- a/topics/21-formal/reading-lean-perceus.md +++ b/topics/21-formal/reading-lean-perceus.md @@ -1,197 +1,749 @@ # Perceus: reference counting precise enough to reuse memory -How does a pure functional language (Lean 4, Koka) get in-place -update performance? Two compiler passes — borrow inference and -reuse tokens — make reference counting precise enough that copying -mostly disappears. This chapter builds the problem and both ideas -step by step, then routes you through the two runtime papers as a -*systems* story: they explain why Lean 4 is fast enough to be the -M21 proof target, and what `Arc`-everywhere Rust engines leave on -the table. +How does a pure functional language get in-place-update performance? Not by +giving up purity — by making the reference count precise enough that the runtime +can *see*, at each constructor, that nobody else is looking. This chapter reads +the two papers behind that idea as a systems story, because they explain why +Lean 4 is fast enough to be this topic's proof target, and what an +`Arc`-everywhere Rust engine is leaving on the table. + +**Two papers, two languages, and the attribution matters.** + +- **Ullrich & de Moura, "Counting Immutable Beans" (IFL 2019)** is **Lean 4's** + runtime. It contributes ownership-based RC, **borrow inference**, and the first + `reset`/`reuse` story. +- **Reinking, Xie, de Moura & Leijen, "Perceus: Garbage Free Reference Counting + with Reuse" (PLDI 2021 / MSR-TR-2020-42)** is **Koka's**. Its own §5 says: "Our + work is closely based on the reference counting algorithm in the Lean theorem + prover as described by Ullrich and de Moura [46] … We extend their work with + drop- and reuse specialization." + +So Perceus is downstream of Lean, not the other way round, and — the correction +this chapter turns on — **Perceus deliberately does *not* have borrow +inference.** Its conclusion lists integrating "selective borrowing" as future +work, and says doing so "would make certain programs **no longer be garbage +free**". Borrowing and garbage-freedom pull against each other; the two papers +sit on opposite sides of that trade. + +Code anchors are `leanprover/lean4` at **`v4.24.0`**. This repo's pin table +(`resources/codebases.md`) has no lean4 entry, so fetch with the explicit ref: + +``` +python3 tools/pinned-source.py --ref v4.24.0 show leanprover/lean4 src/include/lean/lean.h -r 112:136 +``` ## The problem in one sentence -Pure functional semantics say every update copies the structure, -and the naive fix — reference counting — adds an inc/dec (often an -*atomic* one, ~10-40 cycles contended) to every pointer move; two -compiler passes eliminate most of the counting and turn the copies -into in-place loops with zero allocation. +Pure functional semantics say every update copies the structure, and the obvious +fix — reference counting — adds an increment or decrement to every pointer move, +atomic if the object might be shared across threads; the question is how much of +that counting a compiler can delete, and how often the count that survives is +exactly 1 at the moment it would license mutating in place instead of copying. ## The concepts, step by step -### Step 1 — immutability means copying - -In a pure functional language, values are never mutated: "update -element 3 of the list" *means* "build a new list that differs at -element 3." Semantically clean — old readers keep a consistent -value, no aliasing bugs — but taken literally it turns O(1) -mutations into O(n) copies plus allocator traffic. The whole game -of a functional-language runtime is to keep the semantics while -making the copies not happen. The known escapes each cost -something: a GC (garbage collector) buys allocation throughput -but adds latency and — the subtle loss — can never mutate in -place, because it doesn't know how many references a value has -*right now*. - -### Step 2 — reference counting, and its tax - -**Reference counting** (RC) tracks, per heap object, how many -pointers refer to it; copy a pointer → increment, drop one → -decrement, count hits zero → free. RC knows something a tracing GC -doesn't: the count *right now* — and RC == 1 means "I am the only -owner," which is a license to mutate in place. The tax is that the -counting itself is chatty: naive RC emits inc/dec on every pointer -move, and in a multithreaded runtime those are atomic operations -on shared cache lines — the `Arc` tax from topics 2/9 -(contended atomics: ~10-40+ cycles each, plus the coherency -ping-pong). A hot loop that clones an `Arc` per element can spend -more time counting than computing. - -### Step 3 — borrow inference (Immutable Beans): don't count what you only look at - -Most inc/dec pairs bracket a function call that merely *reads* its -argument. Lean's compiler pass infers, per parameter, whether the -function **borrows** it (only inspects — caller keeps ownership, no -RC ops emitted at all) or **owns** it (consumes — the caller -transfers its reference, and the callee is responsible for the -eventual dec). Exactly Rust's `&T` vs `T` distinction, *inferred* -instead of written. Result: most inc/dec pairs simply vanish from -the emitted code — the read path of the program stops paying the -RC tax entirely, without the programmer annotating anything. - -### Step 4 — reuse tokens: functional-but-in-place - -Step 2's license gets cashed here. When a value's count is 1 at its -*last use*, the compiler hands its memory to the constructor about -to be allocated — a **reuse token**: +### Step 1 — immutability means copying, and what that costs + +> **In:** `map f xs` over a one-million-element list. +> **Out:** the byte count a literal implementation allocates, from Lean's actual +> object layout. + +In a pure functional language, "update element 3" *means* "build a new value that +differs at element 3". The semantics are clean — old readers keep a consistent +snapshot, no aliasing bugs — but taken literally, O(1) mutations become O(n) +copies plus allocator traffic. + +Put a number on it. Lean's object header is four fields: +```c +// leanprover/lean4@v4.24.0 src/include/lean/lean.h, lines 131-136 — the header + 131 typedef struct { + 132 int m_rc; + 133 unsigned m_cs_sz:16; + 134 unsigned m_other:8; + 135 unsigned m_tag:8; + 136 } lean_object; ``` - match xs with - | Cons x rest => Cons (f x) (map f rest) - │ │ - └─ if RC(xs)==1 ─┘ reuse xs's cell in place: map becomes - an in-place loop, zero allocation + +That is `4 + 2 + 1 + 1 = 8` bytes, and a constructor is the header followed by +its pointer fields: + +```c +// leanprover/lean4@v4.24.0 src/include/lean/lean.h, lines 170-173 — a constructor + 170 typedef struct { + 171 lean_object m_header; + 172 lean_object * m_objs[]; + 173 } lean_ctor_object; ``` -What the compiler actually emits for `map`, in Rust-ish form: - -```rust -fn map(f: &Closure, xs: Ptr) -> Ptr { - if rc(xs) == 1 { - // reuse token: we are the only owner — xs's cell is handed - // to the Cons about to be built. map becomes an in-place loop. - xs.head = f.call(xs.head); - xs.tail = map(f, xs.tail); - xs // zero allocation - } else { - let out = alloc(Cons { head: f.call(xs.head), tail: map(f, xs.tail) }); - dec(xs); // dropped at exact last use — - out // peak memory = live data +So a `List.cons` is **8 + 8 + 8 = 24 bytes** today. **Work the copy cost.** A +`map` over `N = 1,000,000` cells, implemented literally, allocates +`1,000,000 × 24 B = 24 MB` for the result while the 24 MB input is still live — +**48 MB peak** to transform a list that is 24 MB. + +**A verified drift worth noticing.** The Beans paper (§7.1) says "In a 64-bit +machine, the ctor value header is **16 bytes** long, twice the size of the header +used in OCaml", giving "**32 bytes** to implement a `List Cons` value: 16 bytes +for the header, and 16 bytes for storing the list head and tail." At `v4.24.0` +the header is 8 bytes and the cell 24. The paper is not wrong; it is six years +old, and the runtime got tighter. Cite the paper for 2019 and `lean.h:131-136` +for now. + +The escapes each cost something. A tracing GC buys allocation throughput but adds +latency, and — the subtle loss — **can never mutate in place**, because it does +not know how many references a value has *right now*. + +### Step 2 — the count is the license: ownership, borrowing, and the calling convention + +> **In:** a function that takes a heap value. +> **Out:** the two conventions Lean's runtime defines, and which one permits +> destructive update. + +**Reference counting** tracks, per heap object, how many pointers refer to it: +copy a pointer → increment, drop one → decrement, hits zero → free. What RC knows +that a tracing collector does not is the count *at this instant*, and **count == 1 +is a license to mutate in place**. + +Lean's runtime writes the two conventions down, in a comment, before any of the +analysis exists: + +```c +// leanprover/lean4@v4.24.0 src/include/lean/lean.h, lines 142-150 — verbatim + 142 1- "standard" calling convention if it consumes/decrements the RC. + 143 In this calling convention each argument should be viewed as a resource that is consumed by the function. + 144 This is roughly equivalent to `S && a` in C++, where `S` is a smart pointer, and `a` is the argument. + 145 When this calling convention is used for an argument `x`, then it is safe to perform destructive updates to + 146 `x` if its RC is 1. + 147 + 148 2- "borrowed" calling convention if it doesn't consume/decrement the RC, and it is the responsibility of the caller + 149 to decrement the RC. + 150 This is roughly equivalent to `S const & a` in C++, where `S` is a smart pointer, and `a` is the argument. +``` + +Line 145–146 is the whole thesis in two lines: **owned + RC == 1 ⇒ destructive +update is safe.** This is Rust's `T` versus `&T`, spelled out as a runtime ABI — +and Step 7 is about the fact that Lean *infers* which one each parameter gets, +while Rust makes you write it. + +### Step 3 — precise (ownership) RC: drop at the last use, not at scope exit + +> **In:** the naive scoped discipline — inc on entry, dec at end of scope. +> **Out:** Perceus §2.2's transfer-of-ownership rule and its measured effect on +> `map`. + +Perceus §2.2 replaces "retain until scope exit" with "transfer ownership". In the +`Cons` branch of `map`, the head and tail are `dup`ped (a `dup(x)` increments and +returns `x`) and then `drop(xs)` **frees the input cell immediately**, before the +recursive call builds the output. The paper's Figure 1b: + +``` +fun map( xs, f ) { -- Perceus Fig. 1b, §2.2 + match(xs) { + Cons(x,xx) { + dup(x); dup(xx); drop(xs) + Cons( dup(f)(x), map(xx, f)) } + Nil { drop(xs); drop(f); Nil } + } } ``` -The programmer wrote a pure `map`; unshared inputs run it as an -in-place loop with zero allocation, shared inputs transparently -copy. Copy-on-write, decided per cell at runtime, by a branch the -compiler inserted. The cost: that RC==1 check is a branch per -constructor — question 2 asks when it stops paying. +The paper is candid that this looks worse: "At first blush, this seems more +expensive than the scoped approach but, as we will see, this change enables many +further optimizations. More importantly, transferring ownership, rather than +retaining it, means we can free an object immediately when no more references +remain. This both increases cache locality and decreases memory usage. **For +`map`, the memory usage is halved**: the list `xs` is deallocated while the new +list `ys` is being allocated." + +**Work it.** Step 1's 48 MB peak for `N = 1,000,000` becomes **24 MB** — the +input dies one cell at a time as the output is built, so only one list is ever +fully live. That is the "halved" claim, in bytes, for Lean's current cell size. -### Step 5 — Perceus: garbage-free, drop at the exact last use +### Step 4 — drop specialization: inline the branch, then delete the dead half -Perceus (Koka's refinement) makes the counting *precise*: a -reference is dec'd at its exact last use (precise liveness -analysis), not at scope exit. Two consequences: more values hit -RC==1 in time for step 4's reuse (a reference lingering to end of -scope blocks reuse), and — the headline claim — the program is -**garbage-free**: at every point, peak memory equals live data, -with no GC headroom and no deferred frees. The ladder so far: +> **In:** the generic `drop`. +> **Out:** why inlining it is what makes Step 5 possible. + +Perceus §2.3 gives the basic operation as pseudocode: ``` - naive RC: inc on copy, dec on scope exit (chatty, atomic) - Beans: borrow inference kills most pairs - Perceus: drop-at-last-use + reuse ⇒ uniqueness typing effect - without the type system +fun drop( x ) { -- Perceus §2.3 + if (is-unique(x)) then drop children of x; free(x) + else decref(x) +} ``` -"Uniqueness typing effect without the type system": languages like -Clean prove uniqueness statically and demand annotations; Perceus -gets the same in-place behavior from a runtime count plus -compile-time precision. What a memory-budgeted system buys from -the garbage-free property is question 3. - -### Step 6 — why this is in a database curriculum - -- **The RC(1) fast path is delta-matrix thinking**: mutate in place - when you're the only owner, copy-on-write otherwise — it's Redis's - shared objects, FalkorDB's tensor sharing, and `Arc::make_mut` as - a compiler pass. -- **Borrowed params = zero-cost read path**: an executor passing - `&Value` down a pipeline (topic 11) is doing manual Beans. -- **Proof relevance**: Lean's kernel checks proofs by *running* - terms; a fast runtime is why mathlib-scale proof search is viable, - which is why Lean 4 (not Coq) is the M21 proof target. - -The transferable design rule: ownership information precise enough -to act on turns "immutable" and "in-place" from opposites into a -runtime branch. - -## How to read the paper (with the concepts in hand) - -- **Ullrich & de Moura, "Counting Immutable Beans"** — read first: - the problem framing (steps 1-2), borrow inference (step 3), and - the first reuse story (step 4). This is Lean 4's actual runtime; - read the benchmark section asking "which wins come from borrows, - which from reuse?" -- **Reinking, Xie, de Moura, Leijen, "Perceus"** — read second: - drop-at-last-use and the garbage-free claim (step 5), plus the - sharper reuse analysis. The formal core is skimmable; the - examples and the "functional but in place" section are the - payload. Keep asking the systems question: what would each pass - do to a Rust engine that currently clones an `Arc` in a hot loop? +Drop specialization inlines that at each call site, producing Figure 1c: -## M21 taste: the proof-vs-test trade-off +``` +Cons(x,xx) { -- Perceus Fig. 1c, §2.3 + dup(x); dup(xx) + if (is-unique(xs)) + then drop(x); drop(xx); free(xs) + else decref(xs) + Cons( dup(f)(x), map(xx, f)) +} +``` + +Now `dup(x)` immediately followed by `drop(x)` is visible to the optimiser on the +unique path and cancels. Inlining a runtime check to expose algebraic +cancellation is exactly the compiler move you would make in any hot path; the +novelty is doing it to memory-management code. + +### Step 5 — reuse analysis: hand the freed cell to the next constructor + +> **In:** Step 4's `free(xs)` immediately followed by an allocation of the same +> size. +> **Out:** the reuse token, and the allocation count it removes from a red-black +> insert. + +Perceus §2.4: "Instead of freeing `xs` and immediately allocating a fresh `Cons` +node, we can try to reuse `xs` directly as first described by Ullrich and de +Moura. Reuse analysis … analyses each match branch, and tries to pair each +matched pattern to allocated constructors **of the same size** in the branch." + +The pairing produces a **reuse token**: + +``` +fun map( xs, f ) { -- Perceus §2.4 + match(xs) { + Cons(x,xx) { + val ru = drop-reuse(xs) + Cons@ru( f(x), map(xx, f)) + } + Nil -> Nil + } +} +``` + +and `Cons@ru` compiles to a branch (§2.5): `if (ru != NULL) then { ru->head := x; +ru->tail := xx; ru } else Cons(x,xx)` — in-place when the token is live, `malloc` +otherwise. + +**Reuse specialization** (§2.5) sharpens this: "we only specialize constructors +if at least one of the fields stays the same." For red-black insert that is +almost every field, so a rebalance becomes `if (ru!=NULL) then { ru->left := y; +ru }` — one store instead of five. + +**Work the arithmetic the paper sets up.** §2.5, on Okasaki's rebalancing after +inlining `bal-left`: "every matched `Node` constructor has a corresponding `Node` +allocation – if we consider all branches we can see that we either match one +`Node` and allocate one, or we match three nodes deep and allocate three. With +reuse analysis this means that **every `Node` is reused in the fast path without +doing any allocations**." + +So take the `rbtree` benchmark's **42 million insertions** (§4). If a +rebalance-heavy insert rebuilds a path of `d` nodes, the no-reuse version does +`42 × 10⁶ × d` allocations and the same number of frees; the reuse version does +**zero** on the unique path. At `d = 3` — the deepest rebalance case the paper +names — that is `1.26 × 10⁸` allocate/free pairs deleted from one benchmark run, +before counting the path nodes above the rebalance. The measured consequence is +in §4: the "no-opt" build, with drop/reuse specialization and reuse analysis +disabled, is "**more than 2× slower**". + +Two runtime helpers make the fast path visible in C: + +```c +// leanprover/lean4@v4.24.0 src/include/lean/lean.h, lines 543-549 and 863-874 + 543 static inline bool lean_is_exclusive(lean_object * o) { + 544 if (LEAN_LIKELY(lean_is_st(o))) { + 545 return o->m_rc == 1; + 546 } else { + 547 return false; + 548 } + 549 } + 863 static inline lean_obj_res lean_ensure_exclusive_array(lean_obj_arg a) { + 864 if (lean_is_exclusive(a)) return a; + 865 return lean_copy_array(a); + 866 } + 868 static inline lean_object * lean_array_uset(lean_obj_arg a, size_t i, lean_obj_arg v) { + 869 lean_object * r = lean_ensure_exclusive_array(a); + 870 lean_object ** it = lean_array_cptr(r) + i; + 871 lean_dec(*it); + 872 *it = v; + 873 return r; + 874 } +``` + +`lean_ensure_exclusive_array` (863–866) is `Arc::make_mut`, letter for letter, +and `lean_array_uset` (868–874) is a *functional* array write that mutates when +unshared. Note line 547: for a multi-threaded object `lean_is_exclusive` returns +**false unconditionally**, even at count 1 — Step 8's point. + +### Step 6 — FBIP: reuse as a programming discipline + +> **In:** Step 5's reuse, applied deliberately rather than opportunistically. +> **Out:** what Perceus §2.6 claims you can now write without allocating. + +The paper's framing: "Just like tail-call optimization lets us describe loops in +terms of regular function calls, **reuse analysis lets us describe in-place +mutating imperative algorithms in a purely functional way** (and get persistence +as well)." That is **FBIP**, "functional but in place". + +The worked example is Knuth's 1968 problem — traverse a tree in order with no +extra stack or heap. Morris's classic answer (Fig. 2, in C) threads pointers +through the tree itself; the paper's verdict is "The algorithm is subtle, though. +Since it transforms the tree into an intermediate graph, we need to state +invariants over the so-called Morris loops to prove its correctness." + +The FBIP version (Fig. 3) instead defines an explicit `visitor` type — "our +visitor data type can be generically derived as a list of the *derivative* of the +tree data type" — and walks `Up`/`Down`. The payoff sentence: "**each `Bin` +matches up with a `BinR`, each `BinR` with a `BinL`, and finally each `BinL` with +a `Bin`. Since they all have the same size**, if the tree is unique, each branch +updates the tree nodes in-place at runtime without any allocation, where the +visitor structure is effectively overlaid over the tree nodes." All calls are +tail calls, so it is also a loop. + +The transferable rule: **make the constructors on each side of a match the same +arity**, and reuse fires. That is a design constraint you can apply in Rust too, +even without the compiler pass — it is why an in-place `Vec` transform beats +collect-into-new when the element sizes match. + +### Step 7 — borrow inference is Lean's, and it is worth far less than reuse + +> **In:** Beans §5.2's analysis, and Figure 6's ablation columns. +> **Out:** three ratios you compute yourself, which reorder the passes by value. + +Beans §5.2 infers, per parameter, whether it is **owned** or **borrowed**. The +algorithm is a fixpoint: start optimistically with every parameter borrowed +(`β(c) = Bⁿ`) and promote a parameter to owned when it is consumed — used in a +`reset`, or passed to a function that takes it owned. The paper states the +trade-off explicitly: "when we mark a parameter as borrowed, we reduce the number +of RC operations needed, but we also **prevent reset and reuse**." Never mark `x` +borrowed if the body contains `let y = reset x`. + +That tension is measurable, and this is where the usual story gets the ordering +wrong. Beans **Figure 6** (arithmetic mean of 50 runs via `temci`, on an i7-3770 +with 16 GB running Ubuntu 18.04, Clang 9.0.0; each column normalized to the base +run time, `rbmap` for the `rbmap_*` rows): + +| benchmark | base | `-reuse` | `-borrow` | `-ST` | +|---|---|---|---|---| +| binarytrees | 1.00 | 0.98 | 1.14 | 1.22 | +| deriv | 1.00 | 1.00 | 1.16 | 1.42 | +| const_fold | 1.00 | 1.64 | **0.90** | 1.23 | +| parser | 1.00 | 1.00 | 1.00 | 1.68 | +| qsort | 1.00 | 1.00 | 1.00 | 1.13 | +| rbmap | 1.00 | **3.23** | 1.07 | 1.71 | +| rbmap_10 | 1.49 | 3.62 | 1.52 | 2.43 | +| rbmap_1 | 4.72 | 5.42 | 4.47 | 8.02 | +| unionfind | 1.00 | 1.41 | 1.00 | 2.31 | +| **geom. mean** | **1.24** | **1.74** | **1.27** | **1.89** | + +`-reuse` disables `reset`/`reuse`; `-borrow` assumes all parameters owned; `-ST` +uses atomic RC for all values. The base column is not 1.00 because `rbmap_10` and +`rbmap_1` are normalized to `rbmap`, not to themselves — so **compare columns to +the base column, not to 1**. + +**Work the three ratios.** + +- reuse: `1.74 / 1.24 = 1.403` → turning reuse off costs **40%** overall, and + **3.23×** on `rbmap`. +- borrow inference: `1.27 / 1.24 = 1.024` → **2.4%** overall. +- atomic RC: `1.89 / 1.24 = 1.524` → **52%** overall. + +Two things fall out. First, **reuse is worth roughly sixteen times what borrow +inference is worth** on this suite, which inverts the order these passes are +usually presented in. Second, on `const_fold` the `-borrow` build is **0.90 — +10% *faster* without borrow inference**, exactly the §5.2 trade-off firing: +marking parameters borrowed suppressed reuse that was worth more than the RC ops +it saved. Read Figure 6's typographic convention before quoting it further: +"Digits whose order of magnitude is no larger than that of twice the standard +deviation are marked by squiggly lines" — and the last digit of the `-borrow` +geometric mean carries one. A 2.4% overall effect on this suite is at the noise +floor. + +The paper's own summary is correspondingly narrow: reset/reuse "significantly +improve performance in the benchmarks `const_fold`, `rbmap`, and `unionfind`", +while "the borrowed inference heuristic provides significant speedups in +benchmarks `binarytrees` and `deriv`" — two of nine. + +### Step 8 — the atomic tax, and why sharing a value across threads kills reuse + +> **In:** an object that might be reachable from another thread. +> **Out:** two measurements of what atomics cost, and the reuse that disappears +> with them. + +Lean encodes thread-sharedness in the sign of the count: + +```c +// leanprover/lean4@v4.24.0 src/include/lean/lean.h, lines 115-117 and 487-497 + 115 The reference counter `m_rc` field also encodes whether the object is single threaded (> 0), multi threaded (< 0), or + 116 reference counting is not needed (== 0). We don't use reference counting for objects stored in compact regions, or + 117 marked as persistent. + 487 static inline void lean_inc_ref_n(lean_object * o, size_t n) { + 488 if (LEAN_LIKELY(lean_is_st(o))) { + 489 o->m_rc += n; + 490 } else if (o->m_rc != 0) { + 492 std::atomic_fetch_sub_explicit(lean_get_rc_mt_addr(o), n, std::memory_order_relaxed); + 496 } + 497 } +``` -Property (topic 20): delta-matrix invariant `DP ∩ M = ∅ ∧ DM ⊆ M` -preserved by set/remove/wait. +Line 489 is a **non-atomic** add on the single-threaded fast path — no lock +prefix, no fence, no cache-line ping-pong. Line 492 subtracts rather than adds +because multi-threaded counts are negative (line 115). Beans §7.2 adds that +single-threaded values need **no memory fence at all**, while MT uses a relaxed +fetch-add on `inc` and release/acquire on `dec`. + +**Two independent measurements of what you pay to give that up:** + +- Beans Figure 6, `-ST` column: `1.89 / 1.24 = 1.524`, a **52%** geometric-mean + slowdown from using atomic RC for all values, and `2.31×` on `unionfind`. +- Perceus §4, last paragraph: "we also ran our benchmarks using just atomic + operations for our reference counts to see the impact of the thread-shared + flag. We observed a slowdown from **5% (rbtree) up to 59% (nqueens)** across our + benchmarks." + +But the second-order effect is larger than either number. `lean_is_exclusive` +(543–549) returns **false for any multi-threaded object**, whatever its count. So +the moment a value becomes thread-shared it does not merely pay atomics — **it +loses every in-place update in Step 5**, and every `map` over it reverts to the +allocating path. That is the exact cost structure of `Arc` in a Rust engine: +`Arc::make_mut` on a value you handed to a background thread copies, forever +after, even when the other reference is long gone. + +### Step 9 — garbage-free, and what Perceus deliberately gives up + +> **In:** the headline claim. +> **Out:** its precise definition, its one measured counterexample, and the +> feature Perceus refuses in order to keep it. + +The abstract: "Perceus emits precise reference counting instructions such that +(cycle-free) programs are **garbage free, where only live references are +retained**." §1 restates it as a theorem obligation: Perceus is proved "both sound +(i.e. never drops a live reference), and garbage free (i.e. only retains +reachable references)". Note the parenthetical in the abstract — **cycle-free**. +Perceus does not collect cycles; §6 lists cycle collection as open. + +Three things sharpen the claim into something you can argue with. + +**It is not always a memory win.** §4, on `cfold`: "The 'no-opt' version of Koka +also uses **11% less memory**; this is because the reuse analysis essentially +holds on to memory for later reuse. Just like with scoped based reference +counting that may lead to increased memory usage in some situations." Holding a +cell to reuse it *is* retaining a dead object — garbage-freedom is a property of +the emitted `drop`s, not a guarantee that peak RSS is minimal. On `deriv`, OCaml +uses slightly *less* memory than Koka, which the paper attributes to +case-of-case inlining that Koka does not do. + +**It is the reason Perceus has no borrow inference.** §6: "We would like to +integrate selective 'borrowing' into Perceus – this would **make certain programs +no longer be garbage free**, but we believe it could deliver further performance +improvements if judiciously applied." A borrowed parameter means the callee holds +a reference it is not accounted for and cannot drop early; that is precisely a +retained-but-dead reference. Step 7's 2.4% is the price Perceus declines to pay +for. + +**It is not uniqueness typing.** §5: linear types "like linear Haskell, or the +uniqueness typing of Clean, can offer static guarantees that the corresponding +objects are unique at runtime… However, this usually also requires writing +multiple versions of a function for each case (unique- versus shared argument). +**By contrast, reuse analysis relies on dynamic runtime information**, and thus +reuse can be performed generally. This is also what enables FBIP to use a single +function that can be used for both unique or shared objects (since the uniqueness +property is not part of the type)." One `map`, two behaviours, chosen by a branch +— the trade is a runtime check for not having to write the function twice. + +### Step 10 — why this is in a database curriculum + +> **In:** an engine written in Rust with `Arc` in the hot path. +> **Out:** three transfers, and the one design rule behind them. + +- **The RC == 1 fast path is delta-matrix thinking.** Mutate in place when you + are the only owner, copy-on-write otherwise — Redis's shared objects, + FalkorDB's tensor sharing, and `Arc::make_mut` are the same branch. + `lean_ensure_exclusive_array` (`lean.h:863-866`) is that branch as a runtime + primitive rather than a library call. +- **Borrowed parameters are the zero-cost read path.** An executor passing + `&Value` down a pipeline (topic 11) is doing by hand what Beans §5.2 infers — + and Step 7 says exactly how much that is worth on a suite of symbolic + workloads, which is less than you would guess. +- **Thread-sharing is the cliff, not the atomics.** Step 8: crossing into + multi-threaded ownership costs 52% in counting *and* disables in-place update + entirely. An engine that wraps everything in `Arc` "just in case" has paid both + halves before measuring either. + +The transferable design rule: **ownership information precise enough to act on +turns "immutable" and "in-place" from opposites into a runtime branch** — and the +measured lesson of Step 7 is that the branch (reuse) is worth far more than the +static analysis that avoids counting (borrowing). -- proptest (topic 16): minutes to write, samples the space. -- TLC: model M/DP/DM as small sets, exhaustive at n=4. -- Lean: `theorem set_preserves_inv : inv m → inv (set m i j)` — - unbounded, but you'll spend a day on set-theory lemmas. Do it once - to calibrate which properties deserve which tool. +## M21 taste: the proof-vs-test trade-off + +Property (topic 20): the delta-matrix invariant `DP ∩ M = ∅ ∧ DM ⊆ M`, preserved +by `set`/`remove`/`wait`. + +- **proptest** (topic 16): minutes to write, samples the space, finds shallow + counterexamples fast, says nothing about the cases it did not draw. +- **TLC**: model `M`/`DP`/`DM` as small sets, exhaustive at `n = 4` — see + [reading-tlaplus-raft.md](reading-tlaplus-raft.md) for how fast that state + space grows and why 4 is where you stop. +- **Lean**: `theorem set_preserves_inv : inv m → inv (set m i j)` — unbounded, no + `MaxLog = 3`, and you will spend a day on set-theory lemmas. + +Do it once, in all three, to calibrate which properties deserve which tool. The +answer is usually: proptest for everything, TLC for concurrency, Lean for the one +invariant the whole design rests on. + +## How to read the papers (with the concepts in hand) + +**Read Beans (IFL 2019) first** — it is Lean 4's runtime and the shorter paper. + +- §2–4 for the IR and the `reset`/`reuse` instructions (Step 5). +- **§5.2 borrow inference** — the fixpoint and the "prevents reset and reuse" + sentence (Step 7). +- §7.1 value representation and §7.2 the ST/MT/persistent tags (Steps 1, 8). +- **§8 and Figure 6** — read the ablation columns *before* the cross-language + comparison in Figure 7, and compute the three ratios of Step 7 yourself. Also + in §8: on `const_fold` Lean spends **17%** of runtime deallocating where OCaml + spends **90%** in GC, and Lean is **5× as fast as OCaml** on that benchmark. + +**Then Perceus (PLDI 2021 / MSR-TR-2020-42).** + +- **§2.2–2.6 are the payload** and are written as a worked example on `map` — + Figure 1a–1g is the whole algorithm as a sequence of program transformations. + Read it with Steps 3–6 open. +- §3's linear resource calculus `λ₁` and §4's soundness/garbage-free theorems are + skimmable on a first pass. Note that "borrowing" in §3 is a *typing + environment*, not Beans' calling convention — do not conflate them. +- **§4 Benchmarks**: read the system list first (Koka 2.0.3 + gcc 9.3.0 + + customized mimalloc; OCaml 4.08.1; GHC 8.6.5; Swift 5.3; Java SE 15.0.1 with + G1; C++ gcc 9.3.0 + libc allocator), then the caveat the authors put in front of + it — "we view these results therefore mostly as evidence that the Perceus + reference counting technique is viable … **not as a direct comparison of + absolute performance between systems**". Then the per-benchmark paragraphs, + which is where every number lives: `rbtree` is 42M insertions and Koka lands + "within 10% of the C++ performance" using `std::map`, while Java is close on + time but uses "almost 10× the memory of Koka (1.7 GiB vs. 170 MiB)" — check + that ratio: `1.7 × 1024 / 170 = 10.24`. +- **§5 Related Work and §6 Conclusion** — the attribution to Lean and the + borrowing-versus-garbage-free admission (Step 9). Do not skip them; they are + where the framing in the first half of this guide comes from. + +Then read `src/include/lean/lean.h` at `v4.24.0` in this order: 112–136 (Step 1), +138–160 (Step 2), 466–511 (Step 8), 543–561 and 863–874 (Step 5). ## Questions (answer in notes.md) -1. Where exactly does `Arc` in a Rust engine pay costs that - Beans-style borrow inference eliminates? (Think: clone in a hot - loop vs `&` reborrow — topic 9's contended counter.) -2. Reuse tokens require RC==1 checks at runtime. When does that - branch cost more than it saves (small cells? shared-by-design - structures like interned strings)? -3. Perceus "garbage-free" claim: what does peak-memory = live-data - buy a memory-budgeted buffer pool (topic 6) design? -4. Lean proof vs TLC vs proptest for `DP ∩ M = ∅`: rank by (cost to - write, strength of guarantee, maintenance under refactor). -5. Koka's effect types let Perceus assume no hidden aliasing. What's - the moral equivalent in Rust that makes `Arc::make_mut` sound? +1. Recompute Step 7's three ratios from Figure 6 and rank the passes by measured + value. Then explain `const_fold`'s `-borrow` entry of **0.90** using the + sentence from Beans §5.2 about what borrowing prevents. +2. Redo Step 1's arithmetic for a tree rather than a list: Lean's `Node(color, + left, key, value, right)`. How many bytes per node at `lean.h:131-136`'s + layout, and what does a 1M-node `tmap` allocate with reuse and without? +3. Where exactly does `Arc` in a Rust engine pay what Beans-style borrow + inference removes, and where does it pay the *larger* cost from Step 8? Be + specific about which one `Arc::clone` in a hot loop is. +4. Reuse tokens need an `is-unique` check per constructor. Name a data structure + where that branch is pure loss, and say why — then check your answer against + the `deriv` paragraph in Perceus §4. +5. "Garbage free" and `cfold` using 11% *more* memory than "no-opt" are both true. + Reconcile them in two sentences, and say what that means for a + memory-budgeted buffer pool (topic 6). +6. Rank Lean, TLC and proptest for `DP ∩ M = ∅` by (cost to write, strength of + guarantee, maintenance cost under refactor). Which column decides it for a + codebase that changes weekly? ## Done when -- [ ] You can explain why immutability means copying and what reference counting costs to avoid it. -- [ ] You can explain borrow inference: not counting what you only look at. -- [ ] You can explain reuse tokens and the runtime RC==1 check they depend on. -- [ ] You can state what "garbage-free" means precisely (peak memory equals live data) and what it assumes. -- [ ] You can say why this belongs in a database curriculum, in terms of where `Arc` costs a Rust engine. -- [ ] You wrote answers to all five questions in notes.md, including your ranking of Lean, TLC and proptest for the `DP ∩ M = ∅` invariant. +Answer each before unfolding it. + +- [ ] You can say which paper belongs to which language, and what Perceus does *not* have. + +
Answer + + **Beans (IFL 2019, Ullrich & de Moura) is Lean 4's** runtime: ownership-based + RC, **borrow inference**, `reset`/`reuse`. **Perceus (PLDI 2021, Reinking, Xie, + de Moura, Leijen) is Koka's**, and its §5 says it is "closely based on the + reference counting algorithm in the Lean theorem prover", extending it with + drop- and reuse specialization plus the `λ₁` formalization. + + **Perceus has no borrow inference.** §6: integrating "selective 'borrowing'" + is future work and "would make certain programs no longer be garbage free". + The "borrowing" that does appear in Perceus §3 is a typing environment in the + linear resource calculus, not a calling convention. + +
+ +- [ ] You can compute what `map` costs at each stage of the pipeline, in bytes. + +
Answer + + A Lean `List.cons` at `v4.24.0` is **24 bytes**: an 8-byte header + (`lean.h:131-136` — `int` + 16 + 8 + 8 bits) plus two pointers + (`lean_ctor_object`, `lean.h:170-173`). For `N = 1,000,000`: + + - literal copying / scoped RC — both lists live at peak: `2 × 24 MB = 48 MB`; + - precise ownership RC (Perceus §2.2, "for `map`, the memory usage is + **halved**") — the input dies cell by cell: **24 MB**; + - plus reuse analysis (§2.4) — the freed cell becomes the token for the next + `Cons`: still 24 MB live, but **zero allocations**. + + The Beans paper (§7.1) says 32 bytes per `Cons` (16-byte header); that is the + 2019 layout, and the runtime has since tightened it. Cite the paper for 2019 + and `lean.h` for now. + +
+ +- [ ] You can explain the reuse token and reuse specialization, and quote the red-black arithmetic. + +
Answer + + Reuse analysis (§2.4) pairs each matched pattern with a constructor **of the + same size** allocated in the same branch, replaces `drop` with `drop-reuse` + returning a token `ru`, and attaches it: `Cons@ru(f(x), map(xx,f))`. That + compiles to `if (ru != NULL) then { ru->head := x; ru->tail := xx; ru } else + Cons(x,xx)`. + + Reuse specialization (§2.5) applies "only … if at least one of the fields stays + the same", so a red-black rebalance becomes `if (ru!=NULL) then { ru->left := + y; ru }` — one store, not five. + + The arithmetic: after inlining `bal-left`, "we either match one `Node` and + allocate one, or we match three nodes deep and allocate three… every `Node` is + reused in the fast path **without doing any allocations**." Over the `rbtree` + benchmark's 42M insertions, a 3-node rebalance path is `1.26 × 10⁸` + allocate/free pairs removed; measured, the "no-opt" build is "more than 2× + slower". + +
+ +- [ ] You can state, with numbers, why reuse matters far more than borrow inference. + +
Answer + + From Beans **Figure 6** geometric means (base column **1.24**, because + `rbmap_10`/`rbmap_1` are normalized to `rbmap`): + + - `-reuse` `1.74 / 1.24 = 1.403` → **40%**, and **3.23×** on `rbmap`; + - `-borrow` `1.27 / 1.24 = 1.024` → **2.4%**; + - `-ST` `1.89 / 1.24 = 1.524` → **52%**. + + Reuse is worth about **16× what borrow inference is worth** on this suite. On + `const_fold`, `-borrow` is **0.90** — 10% *faster* without it — because + Beans §5.2's trade-off fires: "when we mark a parameter as borrowed, we reduce + the number of RC operations needed, but we also **prevent reset and reuse**". + And Figure 6's caption marks digits within twice the standard deviation with a + squiggle; the `-borrow` geomean's last digit carries one, so 2.4% is at the + noise floor. + +
+ +- [ ] You can explain the atomic-RC cost and the larger second-order cost of thread sharing. + +
Answer + + Lean encodes thread-sharedness in the *sign* of `m_rc` (`lean.h:115-117`), so + `lean_inc_ref_n` (`:487-497`) takes a **non-atomic** `o->m_rc += n` on the + single-threaded path and a relaxed `atomic_fetch_sub_explicit` otherwise + (subtract, because MT counts are negative). Beans §7.2: no memory fence at all + for ST values. + + Measured cost of losing that: Beans Figure 6 `-ST` is `1.89/1.24 = 1.52` + (**52%** geomean, `2.31×` on `unionfind`); Perceus §4 independently reports + "a slowdown from **5% (rbtree) up to 59% (nqueens)**" from using atomics + everywhere. + + The bigger cost is not the atomics. `lean_is_exclusive` (`:543-549`) returns + **false for any MT object regardless of its count**, so a thread-shared value + loses *every* in-place update. `Arc` in a Rust engine pays exactly this: once + a value has been handed to another thread, `Arc::make_mut` copies forever after. + +
+ +- [ ] You can state "garbage free" precisely and name its one measured counterexample. + +
Answer + + Perceus emits precise RC instructions such that **(cycle-free)** programs are + garbage free — "only retains reachable references" (§1); proved sound (never + drops a live reference) and garbage free. Cycles are not collected; §6 lists + cycle collection as open. + + The counterexample is `cfold` in §4: the **"no-opt" build uses 11% *less* + memory**, "because the reuse analysis essentially holds on to memory for later + reuse. Just like with scoped based reference counting that may lead to + increased memory usage in some situations." Holding a cell for reuse *is* + retaining a dead object, so garbage-freedom is a property of where the `drop`s + are emitted, not a promise about peak RSS. (On `deriv`, OCaml also uses + slightly less memory than Koka, which §4 attributes to case-of-case inlining.) + +
+ +- [ ] You can explain why reuse analysis is not uniqueness typing, and what that buys. + +
Answer + + Perceus §5: linear types "like linear Haskell, or the uniqueness typing of + Clean, can offer static guarantees that the corresponding objects are unique at + runtime… However, this usually also requires **writing multiple versions of a + function** for each case (unique- versus shared argument). By contrast, reuse + analysis relies on **dynamic runtime information**… This is also what enables + FBIP to use a single function that can be used for both unique or shared + objects (since the uniqueness property is **not part of the type**)." + + So the trade is a runtime `is-unique` branch in exchange for writing `map` + once. The same value used persistently degrades gracefully: §2.5 notes the + red-black algorithm "adapts to copying exactly the shared spine of the tree + (and no more), while still rebalancing in place for any unshared parts". + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the recomputed Figure 6 ratios and the tree-node byte count. + +
Answer + + The shape to check yours against for question 2: `Node(color, left, key, value, + right)` in Lean is an 8-byte header plus five fields; with all five boxed that + is `8 + 5×8 = 48` bytes, and Lean can unbox the scalars (`m_other` holds "the + number of fields in a constructor object", `lean.h:129`), so a tuned layout is + smaller. A 1M-node `tmap` allocates `~48 MB` without reuse and **0 bytes** with + it, because §2.6's `Bin`/`BinR`/`BinL` all have the same size — which is the + precondition, not a coincidence. + +
## References **Papers** -- Ullrich, de Moura — "Counting Immutable Beans: Reference Counting - Optimized for Purely Functional Programming" (IFL 2019, - [arXiv:1908.05647](https://arxiv.org/abs/1908.05647)) — borrow - inference + the first reuse story; this is Lean 4's runtime -- Reinking, Xie, de Moura, Leijen — "Perceus: Garbage Free - Reference Counting with Reuse" (PLDI 2021) — drop-at-last-use, - the garbage-free claim, and the sharper reuse analysis + +- Sebastian Ullrich, Leonardo de Moura — *Counting Immutable Beans: Reference + Counting Optimized for Purely Functional Programming*, IFL 2019 + ([arXiv:1908.05647](https://arxiv.org/abs/1908.05647)). **Lean 4's runtime.** + §5.2 borrow inference; §7.1 value layout (16-byte ctor header, 32-byte `Cons` + in 2019); §7.2 the ST/MT/persistent tags; §8 and **Figure 6** the ablation + table of Step 7 — i7-3770, 16 GB, Ubuntu 18.04, Clang 9.0.0, arithmetic mean of + 50 runs via `temci`. +- Alex Reinking, Ningning Xie, Leonardo de Moura, Daan Leijen — *Perceus: Garbage + Free Reference Counting with Reuse*, PLDI 2021 (MSR-TR-2020-42, Nov 22 2020). + **Koka's runtime.** §2.2 precise RC and the halved `map`; §2.3 drop + specialization; §2.4 reuse analysis; §2.5 reuse specialization and the + red-black arithmetic; §2.6 FBIP and Morris; §4 benchmarks (Figure 9, median of + 10 runs normalized to Koka) and the atomic-RC experiment; §5 the attribution to + Lean and the uniqueness-typing comparison; §6 the borrowing-versus-garbage-free + admission. +- Joseph M. Morris — *Traversing binary trees simply and cheaply*, IPL 1979 — the + algorithm Perceus Figure 2 shows in C and Figure 3 replaces. +- Chris Okasaki — *Purely Functional Data Structures* — the red-black insertion + §2.5 uses as its reuse-specialization example. + +**Code** — `leanprover/lean4` at `v4.24.0` (no pin-table entry; fetch with +`tools/pinned-source.py --ref v4.24.0`) + +| Anchor | What | +|---|---| +| `src/include/lean/lean.h:112-136` | the object header; `m_rc`'s sign encodes ST (> 0) / MT (< 0) / no-RC (== 0); 8 bytes on 64-bit | +| `src/include/lean/lean.h:138-160` | the "standard" vs "borrowed" calling conventions, and the RC == 1 licence at 145–146 | +| `src/include/lean/lean.h:170-173` | `lean_ctor_object` — header plus a flexible array of fields | +| `src/include/lean/lean.h:466-481` | `lean_is_mt`, `lean_is_st`, `lean_is_persistent`, `lean_has_rc` | +| `src/include/lean/lean.h:487-511` | `lean_inc_ref_n` (non-atomic ST fast path) and `lean_dec_ref` | +| `src/include/lean/lean.h:543-561` | `lean_is_exclusive` — false for MT objects at any count — and `lean_is_shared` | +| `src/include/lean/lean.h:863-874` | `lean_ensure_exclusive_array` (= `Arc::make_mut`) and `lean_array_uset` | + +**In this topic** +- [reading-tlaplus-raft.md](reading-tlaplus-raft.md) — the model-checking side of + the proof-vs-test trade-off in the M21 taste. +- `topics/21-formal/README.md` §5 — where Lean sits on this topic's cost ladder. diff --git a/topics/21-formal/reading-tlaplus-raft.md b/topics/21-formal/reading-tlaplus-raft.md index 98937fd..7f41bcd 100644 --- a/topics/21-formal/reading-tlaplus-raft.md +++ b/topics/21-formal/reading-tlaplus-raft.md @@ -1,214 +1,622 @@ # A spec is a state machine: TLA+ through raft.tla -TLA+ has one idea — describe your protocol as "which next-states -are allowed" and let TLC enumerate every interleaving. Lamport's -*Specifying Systems* part I (chapters 1-7) teaches the language; -Ongaro's published Raft spec (471 lines) shows what a real protocol -spec looks like. This chapter builds the mental model step by step -— states, actions, the Next disjunction, invariants, model-size -discipline — using our `specs/WalReplication.tla` (94 lines) as the -running example, so both texts read as instances of one idea. +TLA+ has one idea — describe your protocol as "which next-states are allowed" +and let TLC enumerate every interleaving. Lamport's *Specifying Systems* part I +teaches the language; Ongaro's published Raft spec shows what a real protocol +spec looks like. This chapter builds the mental model from the ground up — +states, actions, the `Next` disjunction, invariants, and the state-space +arithmetic that decides whether TLC finishes at all — using this topic's +`specs/WalReplication.tla` as the running example, so both texts read as +instances of one idea. + +Two sources, both quoted with real line numbers. `specs/WalReplication.tla` is +**92 lines**, in this repo, and its model lives in `specs/WalReplication.cfg`. +`raft.tla` is **471 lines**, pinned at `ongardie/raft.tla@6ecbdbc` +(`resources/codebases.md` pin table); that repository contains exactly two files, +`raft.tla` and a 9-line `README.md`, and — a fact Step 8 turns on — **no `.cfg` +and no invariants**. ## The problem in one sentence -Even a toy 3-replica, 3-entry WAL-shipping protocol has over a -thousand distinct reachable states across all interleavings of -ship/commit/crash/failover — far too many for a human to reason -through, and exactly the right size for a machine to enumerate in -under a second. +A toy 3-replica, 3-entry WAL-shipping protocol has **1080 distinct reachable +states** across every interleaving of ship / commit / crash / failover — already +past what a human reviews reliably, and small enough for TLC to enumerate in +under a second; the same protocol written the way Raft actually needs it has on +the order of **10¹¹** type-correct states before you count messages, and the gap +between those two numbers is the entire practical skill. ## The concepts, step by step ### Step 1 — a state is a snapshot of the variables -A TLA+ spec picks a handful of **variables**, and a **state** is -one assignment of values to them. Our WalReplication uses four: -`wal` (how many entries each replica has — a log that's always a -prefix can be modeled as just its length), `primary`, `crashed` -(the set of dead replicas), `committed` (how many entries the -protocol has acknowledged). A **behavior** is a sequence of states -— one possible execution. The whole protocol universe at MaxLog=3, -3 replicas is small: `wal ∈ [Replicas → 0..3]` alone is 4³ = 64 -combinations. Deliberately small — step 5 is about keeping it so. +> **In:** a protocol you can describe in prose. +> **Out:** a finite set of variables, and the size of the space they span — +> computed, because Step 6 is going to spend it. + +A TLA+ spec declares a handful of **variables**; a **state** is one assignment of +values to all of them, and a **behavior** is an infinite sequence of states — one +possible execution. There are no threads, no objects, no heap. Just variables. + +Ours declares four: + +```tla +-- specs/WalReplication.tla, lines 17-23 — the entire state of the protocol + 17 VARIABLES + 18 primary, \* current primary + 19 crashed, \* set of crashed replicas (crashes are permanent here) + 20 wal, \* [Replicas -> 0..MaxLog], length of each prefix log + 21 committed \* client-visible commit point + 22 + 23 vars == <> +``` + +Note what `wal` is *not*: it is not a sequence of entries. Line 20 stores a +single natural number per replica, and the module header (lines 3–4) says why — +"Entries are sequential 1..MaxLog and ship in order, so each log is a prefix of +the primary's — one natural number per replica." That is a **modelling +decision**, and Step 6 shows exactly what it costs and what it buys. + +**Count the space.** With `Replicas = {r1,r2,r3}` and `MaxLog = 3` +(`WalReplication.cfg`): + +- `primary ∈ Replicas` → 3 +- `crashed ⊆ Replicas` → 2³ = 8 +- `wal ∈ [Replicas → 0..3]` → 4³ = 64 +- `committed ∈ 0..3` → 4 + +Product: `3 × 8 × 64 × 4 = 6144` **type-correct** states. Not all are reachable: +`Crash(r)` (line 65) is guarded by `Cardinality(Alive \ {r}) >= Quorum` with +`Quorum = 2`, so at most one replica ever crashes and only 4 of the 8 subsets +occur — `3 × 4 × 64 × 4 = 3072`. TLC's measured answer is **1080 distinct** +states (`notes.md`). The remaining factor of 2.8 is what the *action guards* +prune. This gap — type-correct, guard-reachable, actually reachable — is the +thing to keep in your head for Step 6. ### Step 2 — an action is a predicate relating now to next -An **action** describes one atomic step as a boolean predicate -over two states: unprimed variables (`wal`) mean the current -state, **primed** ones (`wal'`) mean the next. No assignment, no -control flow — the action is simply *true* of exactly the -(current, next) pairs it allows. A real one, from our spec: +> **In:** the variables of Step 1. +> **Out:** the only construct in the language that does any work, and the three +> layers every one of them has. + +An **action** describes one atomic step as a boolean predicate over *two* +states: unprimed variables (`wal`) denote the current state, **primed** ones +(`wal'`) the next. There is no assignment and no control flow — the action is +simply *true* of exactly the (current, next) pairs it permits. ```tla -\* WAL shipping: backup r pulls the next entry it is missing. -Ship(r) == - /\ r # primary /\ r \notin crashed /\ primary \notin crashed - /\ wal[r] < wal[primary] \* enabled only when behind - /\ wal' = [wal EXCEPT ![r] = @ + 1] \* ONE entry per action — - /\ UNCHANGED <> \* atomicity IS the model +-- specs/WalReplication.tla, lines 46-51 — WAL shipping, one action, verbatim + 46 \* WAL shipping: backup r pulls the next entry it is missing. + 47 Ship(r) == + 48 /\ r # primary /\ r \notin crashed /\ primary \notin crashed + 49 /\ wal[r] < wal[primary] + 50 /\ wal' = [wal EXCEPT ![r] = @ + 1] + 51 /\ UNCHANGED <> ``` -Read it in three layers: the first two lines are the **enabling -condition** (in which states can this happen at all), the third -says what changes, the fourth pins everything else (omit -`UNCHANGED` and you've allowed those variables to change to -*anything*). The comment "atomicity IS the model" is load-bearing: -whatever one action changes is what the model treats as -indivisible — question 3 turns on it. +Read it in three layers: -### Step 3 — Next is a disjunction: concurrency falls out for free +1. **Enabling condition** (48–49): in which states can this happen at all? Line + 48 is liveness of the participants, line 49 is "`r` is actually behind". If + no conjunct on these lines holds, the action is **disabled** in that state and + contributes no successor. +2. **The change** (50). `[wal EXCEPT ![r] = @ + 1]` is a function identical to + `wal` except at `r`, where `@` denotes the old value. Exactly **one** entry + moves. +3. **The frame** (51). `UNCHANGED` pins everything else. Omit it and the action + permits those variables to change to *anything* — the single most common + beginner bug, and it produces a spec that checks nothing while looking fine. -The full spec is one formula: +The granularity of layer 2 is not a detail, it is the model. Whatever one action +changes is what the specification treats as indivisible; a `Ship` that moved the +whole log at once would be asserting that shipping is atomic, and TLC would +never explore the interleavings in which it is not. +### Step 3 — `Next` is a disjunction, and that is where concurrency comes from + +> **In:** a set of actions. +> **Out:** the whole spec as a single formula, and the reason you never write a +> scheduler. + +```tla +-- specs/WalReplication.tla, lines 81-86 — the whole protocol, five lines + 81 Next == + 82 \/ Append + 83 \/ Commit + 84 \/ \E r \in Replicas : Ship(r) \/ Crash(r) \/ Failover(r) + 85 + 86 Spec == Init /\ [][Next]_vars ``` - Spec == Init /\ [][Next]_vars - │ │ - │ └─ every step satisfies Next (or stutters) - └─ initial-state predicate - Next == A1 \/ A2 \/ ∃ r ∈ S : A3(r) ← actions, primed vars +Each step of a behavior is *any one* enabled disjunct. There are no processes and +no scheduler: **every interleaving of enabled actions is a behavior, +automatically**, because the disjunction does not say which one happens. The `\E` +on line 84 quantifies over replicas, so `Ship(r1)`, `Ship(r2)` and `Ship(r3)` are +three separate disjuncts generated from one line. + +`Spec` (86) reads: start in a state satisfying `Init`, and **always** (`[]`) +every step satisfies `Next` — *or* leaves `vars` unchanged. That last clause is +what the `_vars` subscript means, and it is called **stuttering**. + +Raft's `Next` has the identical shape at 10× the width: + +```tla +-- raft.tla, lines 454-465 (ongardie/raft.tla@6ecbdbc) — same construct, nine actions + 454 Next == /\ \/ \E i \in Server : Restart(i) + 455 \/ \E i \in Server : Timeout(i) + 456 \/ \E i,j \in Server : RequestVote(i, j) + 457 \/ \E i \in Server : BecomeLeader(i) + 458 \/ \E i \in Server, v \in Value : ClientRequest(i, v) + 459 \/ \E i \in Server : AdvanceCommitIndex(i) + 460 \/ \E i,j \in Server : AppendEntries(i, j) + 461 \/ \E m \in DOMAIN messages : Receive(m) + 462 \/ \E m \in DOMAIN messages : DuplicateMessage(m) + 463 \/ \E m \in DOMAIN messages : DropMessage(m) + 464 \* History variable that tracks every log ever: + 465 /\ allLogs' = allLogs \cup {log[i] : i \in Server} ``` -In ours: +Two things to notice. First, lines 462–463: message **duplication and loss are +actions**, so an unreliable network is not an assumption bolted on the side — it +is two more disjuncts. Second, the outer `/\` at 454 with the conjunct at 465: +`allLogs` is updated on *every* step, so it is not one of the disjuncts, it rides +along with all of them. + +### Step 4 — stuttering, and why it is not a technicality + +> **In:** the `[][Next]_vars` of Step 3. +> **Out:** why a specification must allow steps in which nothing happens. + +`[][Next]_vars` is shorthand for `[](Next \/ UNCHANGED vars)`. A behavior may +contain steps where nothing changes at all. This looks like a loophole and is +the opposite. + +The reason is **refinement**. If a detailed spec `D` implements an abstract spec +`A`, you show it by mapping each state of `D` to a state of `A` and proving every +`D` step maps to an `A` step. But `D` has more steps than `A` — internal actions +that `A` does not model at all. Those must map to *something*, and what they map +to is a stuttering step of `A`. Without stuttering, `D` could never implement +`A`, and refinement — the reason TLA+ has a temporal logic rather than just a +state machine — would not exist. + +The practical consequence: a spec's behaviors are closed under inserting and +deleting finite runs of repeated states, so "how many steps did it take" is never +a meaningful property, and TLC's `-deadlock` flag exists because a state with no +enabled action is normally *fine* under stuttering and only sometimes a bug. + +### Step 5 — invariants, and what TLC actually does + +> **In:** a spec and a model. +> **Out:** a property, a search, and a counterexample trace — with this topic's +> two measured runs. + +An **invariant** is a predicate on single states that must hold in every +reachable one. ```tla -Next == - \/ Append - \/ Commit - \/ \E r \in Replicas : Ship(r) \/ Crash(r) \/ Failover(r) +-- specs/WalReplication.tla, lines 88-90 — the property under test + 88 \* THE invariant: a live primary's WAL contains every committed entry. + 89 \* (Logs are prefixes, so "contains entry k" is just wal >= k.) + 90 Durability == primary \notin crashed => committed <= wal[primary] ``` -No processes, no threads: each step of a behavior is *any one* -enabled disjunct. Concurrency falls out of the disjunction — every -interleaving of enabled actions is a behavior, automatically. The -`[]` means "always", and the `_vars` subscript permits -**stuttering** steps (states where nothing changes) — a technical -allowance that's essential for refinement (question 5). That's the -whole language, conceptually; everything else is notation. +The `.cfg` names the model and the properties: -### Step 4 — invariants, and TLC's exhaustive breadth-first search +``` +CONSTANTS Replicas = {r1, r2, r3} MaxLog = 3 Quorum = 2 SyncCommit = TRUE +INIT Init NEXT Next INVARIANTS TypeOK, Durability +``` -An **invariant** is a predicate on single states that must hold in -every reachable one. Ours: +**TLC** does breadth-first search from the initial states: at each state, fire +every enabled action, deduplicate successors against a set of seen states, check +every invariant on each new state. Because the search is breadth-first, the first +violation found is at minimum depth — the trace TLC prints is a **shortest** +counterexample, which is what makes it a debugging artefact rather than a +failure report. + +Measured, from `notes.md`: + +| `SyncCommit` | generated | distinct | depth | `Durability` | +|---|---|---|---|---| +| `TRUE` | 2583 | 1080 | 14 | holds | +| `FALSE` | 183 | 123 | 5 | **VIOLATED** | + +The 5-step trace is the interesting one: `Append` → `Commit` without a quorum ack +→ `Crash(primary)` → `Failover` to a replica that never saw entry 1 → the +invariant fails. That is PostgreSQL's `synchronous_commit = off` data-loss story, +found by a machine in a fraction of a second, guaranteed rather than sampled. + +Why the flip breaks it is one conjunct — line 60, `SyncCommit => +Cardinality(AckedBy(committed + 1)) >= Quorum`. With `SyncCommit = FALSE` the +implication is vacuously true, the quorum gate disappears, and quorum +intersection — the argument that `Failover`'s "longest surviving log" (line 77) +must hold every committed entry — loses its premise. + +### Step 6 — model-size discipline, computed + +> **In:** the observation that TLC enumerates everything. +> **Out:** the arithmetic that decides whether a spec is checkable, done on both +> models, with the modelling knobs identified by their cost. + +TLC's budget is states, so every modelling choice is a purchase. Here is the +purchase, priced. + +**Our model** (Step 1): `3 × 8 × 64 × 4 = 6144` type-correct states, 1080 +reachable. Now price the alternative. Suppose `wal` were a real **sequence** of +entries rather than a length, with entries drawn from 3 possible values. Logs of +length 0..3 over 3 values: `1 + 3 + 9 + 27 = 40` distinct logs per replica, so +`wal` alone becomes `40³ = 64,000` instead of `64` — a **1000× multiplier** on +the whole state space, for a protocol in which logs are prefixes by construction +and therefore carry no information beyond their length. That one modelling +decision, at line 20, is the difference between a one-second run and an +overnight one. + +**raft.tla** cannot make that simplification, because Raft's whole difficulty is +logs that *diverge*. It declares **13 variables** (`raft.tla:32-85`): `messages`, +`elections`, `allLogs`, `currentTerm`, `state`, `votedFor`, `log`, `commitIndex`, +`votesResponded`, `votesGranted`, `voterLog`, `nextIndex`, `matchIndex`. Take a +deliberately tiny hypothetical model — 3 servers, 1 client value, terms bounded +at 3, logs bounded at length 3 — and price just five of them: + +| variable | domain at this model | size | +|---|---|---| +| `currentTerm` | `[Server → 1..3]` | 3³ = 27 | +| `state` | `[Server → {Follower,Candidate,Leader}]` | 3³ = 27 | +| `votedFor` | `[Server → Server ∪ {Nil}]` | 4³ = 64 | +| `log` | `[Server → Seq(entry)]`, 40 logs each | 40³ = 64,000 | +| `commitIndex` | `[Server → 0..3]` | 4³ = 64 | + +Product: `27 × 27 × 64 × 64,000 × 64 = 191,102,976,000` ≈ **1.9 × 10¹¹** +type-correct states — from five of thirteen variables. The remaining eight are +worse, not better: `nextIndex` and `matchIndex` are each `[Server → [Server → +0..3]]` = `64³ = 262,144`; `messages` is a *bag* of records with no finite bound +at all; and `allLogs` is a **set of logs**, so its domain is the powerset of the +40 possible logs — `2⁴⁰ ≈ 1.1 × 10¹²` values, single-handedly larger than the +five variables above combined. + +That last one is worth sitting with. `allLogs` is declared with the comment "A +history variable used in the proof. This would not be present in an +implementation" (`raft.tla:41-44`), and it is updated on every step +(`raft.tla:465`). A variable added purely to make a proof expressible is the +largest term in the model checker's state space. Proof convenience and checking +cost pull in opposite directions. + +So the three knobs, each now with a price attached: + +- **Abstract the data.** Logs-as-lengths saved us 1000×. Only legal because the + prefix property is guaranteed by construction — an assumption Step 7 removes. +- **Keep atomic regions small.** Raft is explicit about this in a comment, and + it is the same reasoning as our one-entry `Ship`: ```tla -\* THE invariant TLC checks on every reachable state: -Durability == primary \notin crashed => committed <= wal[primary] +-- raft.tla, lines 201-204 — the model-size argument, in the spec's own words + 201 \* Leader i sends j an AppendEntries request containing up to 1 entry. + 202 \* While implementations may want to send more than 1 at a time, this spec uses + 203 \* just 1 because it minimizes atomic regions without loss of generality. + 204 AppendEntries(i, j) == ``` -**TLC**, the model checker, does breadth-first search from the -initial states, firing every enabled action at every state, -deduplicating, and checking the invariant on each state found. BFS -means the first violation found is a *shortest* counterexample — -the trace TLC prints is the minimal story of the bug. Our measured -runs: SyncCommit=TRUE → 1080 distinct states, depth 14, holds in -under a second. SyncCommit=FALSE → violated at depth 5 after 123 -states, and the 5-step trace (Append → Commit without quorum → -Crash(primary) → Failover to an empty log) is exactly the -PostgreSQL `synchronous_commit = off` data-loss story. - -### Step 5 — model-size discipline (why TLC finishes) - -TLC enumerates *everything*, so state-count is the budget and -modeling choices are what spend it: - -- Logs-as-lengths: our `wal ∈ [Replicas → 0..MaxLog]` gives 4³ log - states; raft.tla with real sequences and terms explodes — Ongaro - notes it's checked only for tiny bounds. -- One entry per Ship/AppendEntries action: granularity of atomicity - IS the model — batching would hide interleavings. -- Small constants (3 replicas, 3 entries) on the small-scope bet - from [reading-aws-cacm15.md](reading-aws-cacm15.md): protocol - bugs almost never need N=7. - -Small models, real bugs: 123 states were enough to catch the -async-commit data loss no test generator finds *guaranteed*. - -### Step 6 — what Raft needs that our toy doesn't: terms - -Reading raft.tla after WalReplication, the striking additions are -**terms** (a monotonically increasing epoch number attached to -every leader and log entry) and the **log-matching check** -(followers reject entries whose predecessor doesn't match). Our -model gets away without them because (a) entries are sequential -integers shipped in order, so logs are prefixes by construction, -and (b) crashes are permanent, so there is never a *stale -ex-primary* that can come back and diverge the log. Un-model -either assumption and you re-derive Raft piece by piece — a great -exercise: allow crashed replicas to rejoin and watch TLC show you -why terms exist (question 1). This is the general skill: every -mechanism in a real protocol answers a behavior some simpler model -excluded. - -### Step 7 — safety vs liveness - -Everything above is **safety** ("nothing bad ever happens" — an -invariant can be violated by a finite trace). **Liveness** -("something good eventually happens") is a different kind of -property: it's violated only by *infinite* behaviors, e.g. one -where shipping simply never runs. Checking it requires **fairness** -assumptions — `WF_vars(Ship(r))` says Ship can't stay enabled -forever without firing — otherwise TLC accepts the do-nothing -behavior. Raft's spec famously checks safety only; so does ours. -Start there; liveness doubles the conceptual load for a different + Note the direction: sending one entry at a time makes the *spec* explore more + interleavings, not fewer. It costs states and buys coverage. "Minimizes atomic + regions" is the goal; "without loss of generality" is the claim that batching + adds no behaviors a sequence of single sends cannot produce. +- **Use small constants.** 3 replicas, 3 entries, on the bet that protocol bugs + do not first appear at N = 7. That bet is the small-scope hypothesis discussed + in [reading-aws-cacm15.md](reading-aws-cacm15.md) — and note there that it is + Daniel Jackson's hypothesis, not something the AWS paper claims. + +### Step 7 — what Raft needs that our toy does not + +> **In:** two specs, one 92 lines and one 471. +> **Out:** the two mechanisms the difference is made of, and the assumption each +> one pays for. + +Read `raft.tla` after `WalReplication.tla` and the additions that jump out are +**terms** (a monotonically increasing epoch attached to every leader and every +log entry) and the **log-matching check**: + +```tla +-- raft.tla, lines 327-337 — logOk is the log-matching check, reject branch shown + 327 HandleAppendEntriesRequest(i, j, m) == + 328 LET logOk == \/ m.mprevLogIndex = 0 + 329 \/ /\ m.mprevLogIndex > 0 + 330 /\ m.mprevLogIndex <= Len(log[i]) + 331 /\ m.mprevLogTerm = log[i][m.mprevLogIndex].term + 332 IN /\ m.mterm <= currentTerm[i] + 333 /\ \/ /\ \* reject request + 334 \/ m.mterm < currentTerm[i] + 335 \/ /\ m.mterm = currentTerm[i] + 336 /\ state[i] = Follower + 337 /\ \lnot logOk +``` + +`logOk` (328–331) says: the entry before the one you are sending me must exist in +my log **at the same term**. Index agreement is not enough; term agreement is the +point, because two leaders in different terms can write different entries at the +same index. + +Our model gets away with neither mechanism because of two assumptions it makes +without saying so loudly: + +- Entries are sequential and ship in order (module header, lines 3–4), so logs + are prefixes **by construction**. There is no index at which two replicas can + disagree, so there is nothing for a log-matching check to check. +- `Crash(r)` is permanent (line 19's comment: "crashes are permanent here"), so + there is never a stale ex-primary that comes back and writes. There is no + second leader, so there is no need for terms to order them. + +Remove either assumption and you re-derive Raft piece by piece. Adding a +`Rejoin(r)` action is question 1 for exactly this reason: TLC will hand you the +trace that proves you now need terms. The general skill this teaches is the one +worth taking away — **every mechanism in a real protocol answers a behavior some +simpler model excluded**, and a model checker will tell you which one if you let +the behavior back in. + +### Step 8 — safety, liveness, and what raft.tla does not contain + +> **In:** the invariant of Step 5. +> **Out:** the second class of property, and an honest reading of what the +> published Raft spec does and does not check. + +**Safety** is "nothing bad ever happens" — violated by a finite trace, which is +why an invariant check works and why TLC can print a counterexample. `Durability` +is safety. **Liveness** is "something good eventually happens" — violated only by +an *infinite* behavior, e.g. one in which `Ship` is enabled forever and never +fires. Checking liveness needs **fairness** assumptions: `WF_vars(Ship(r))` (weak +fairness) says `Ship(r)` cannot remain continuously enabled forever without +occurring. Without a fairness conjunct, the behavior that stutters forever +satisfies any spec, so every liveness property fails trivially. + +`WalReplication.tla` contains no fairness conjuncts and its `.cfg` lists two +`INVARIANTS` and no `PROPERTIES`. Safety only, deliberately. + +Now the correction that matters for reading `raft.tla`. **It does not check +anything at all.** Search the 471 lines for `THEOREM`, `Inv`, `Invariant`, +`PROPERTY`, `WF_` or `SF_` and there are no matches. The file defines constants, +13 variables, helper operators, nine actions, `Init`, `Next` and `Spec` — and +stops at line 471. There is no `.cfg` in the repository; the repository contains +`raft.tla` and a 9-line `README.md`, nothing else. The README's own guidance is: + +> "If you're trying to run the TLA+ model checker on this specification, check +> out Jin Li's changes in Pull Request #4." + +— i.e. the published spec is not TLC-ready as distributed. The safety argument +lives elsewhere: the README points at "Chapter 8 (Correctness) and Appendix B +(Safety proof and formal specification)" of Ongaro's dissertation. + +So the honest statement is not "Raft's spec checks safety only". It is: **the +published Raft spec states the protocol and asserts no properties**; the +properties and their proof are in the dissertation, and running TLC on it is +something you have to set up yourself. Start with safety in your own specs +anyway — liveness roughly doubles the conceptual load and targets a different class of bug (stuck protocols, not corrupt ones). ## How to read the paper (with the concepts in hand) -- **Lamport, *Specifying Systems*, part I (chapters 1-7)** — the - language behind steps 1-4 and 7, in Lamport's own order (he - builds from a one-bit clock up to a FIFO). With the steps above - as scaffolding, these chapters are a fast read; the rest of the - book is reference material. -- **Our `specs/WalReplication.tla` (94 lines)** — read it in full - before Raft; every construct in it now has a step number. Run it: - `java -cp ~/repos/tla2tools.jar tlc2.TLC -deadlock - WalReplication.tla` (flip `SyncCommit` in the .cfg to see the - depth-5 trace from step 4 yourself). -- **raft.tla (471 lines)** — read by these anchors: - -| line | step | what | +- **Lamport, *Specifying Systems*, part I (chapters 1–7)** — the language behind + Steps 1–5 and 8, in Lamport's own order: he builds from a one-bit clock to an + asynchronous FIFO. With the steps above as scaffolding these chapters are a + fast read; the rest of the book is reference material. Chapter 8 is where + liveness and fairness get their proper treatment. +- **`specs/WalReplication.tla` (92 lines)** — read it in full before Raft; every + construct in it now has a step number. Then run it: + `java -cp tlc2.TLC -deadlock WalReplication.tla`, and flip + `SyncCommit` to `FALSE` in the `.cfg` to get the depth-5 trace of Step 5 + yourself. +- **`raft.tla` (471 lines, pinned at `6ecbdbc`)** — read by the anchors below, + in this order: variables first, then `Init`, then the actions in `Next`'s + order, then `Next` itself. Do not look for invariants; Step 8 explains why. + +| raft.tla:line | step | what | |---|---|---| -| :24 | 1 | message types incl. `AppendEntriesRequest/Response` | -| :155 | 1 | `Init` — everything empty, all followers | -| :204 | 2, 5 | `AppendEntries(i, j)` — leader ships **up to 1 entry** per action (model-size discipline; same reason our `Ship` moves one entry) | -| :229 | 6 | `BecomeLeader(i)` — quorum of votes ⇒ leader | -| :327 | 6 | `HandleAppendEntriesRequest` — the consistency check: term + prevLogIndex/prevLogTerm match, else reject | +| `:23-24` | 3 | message-type constants — `RequestVote*`, `AppendEntries*` | +| `:32-85` | 6 | the 13 `VARIABLE` declarations. `:41-44` is `allLogs`, the history variable, with its own disclaimer | +| `:99` | 7 | `Quorum` — "every quorum overlaps with every other", the property `Failover` needs | +| `:102` | 7 | `LastTerm` — the election-restriction helper | +| `:155` | 5 | `Init` — six conjuncts, one per variable group | +| `:167` / `:178` | 7 | `Restart` (loses everything but `currentTerm`, `votedFor`, `log`) and `Timeout` — the actions our permanent-crash model has no analogue for | +| `:201-204` | 6 | `AppendEntries` — "up to 1 entry … minimizes atomic regions" | +| `:229` | 7 | `BecomeLeader` — a quorum of votes ⇒ leader | +| `:259` | 5 | `AdvanceCommitIndex` — Raft's `Commit`, gated on `matchIndex` | +| `:327-331` | 7 | `HandleAppendEntriesRequest` / `logOk` — the log-matching check | +| `:443` / `:448` | 3 | `DuplicateMessage`, `DropMessage` — the unreliable network, as actions | +| `:454-465` | 3 | `Next`, and `allLogs'` riding along on every step | +| `:469` | 3 | `Spec == Init /\ [][Next]_vars` — and then the file ends | ## Questions (answer in notes.md) -1. Add `Rejoin(r)` (crashed → alive, keeping its stale wal) to - WalReplication. What new invariant is needed, and what trace does - TLC find without it? (This re-derives Raft's term check.) -2. Why does `Failover` need "longest log among survivors" — exhibit - the quorum-intersection argument for Quorum=2, |Replicas|=3, and - the trace when failover picks an arbitrary survivor instead. -3. raft.tla:204 ships ≤1 entry per action. What bug class would a - "ship everything atomically" model hide in OUR spec? -4. Express topic 8's MVCC snapshot-visibility as a TLA+ invariant - sketch (what are the variables? what's an action?) — this is the - M21 deliverable's outline. -5. `[][Next]_vars` allows stuttering. Why is that essential for - refinement (mapping a detailed spec onto an abstract one)? +1. Add `Rejoin(r)` (crashed → alive, keeping its stale `wal`) to + `WalReplication.tla`. What trace does TLC find, and what new mechanism is + needed to rule it out? (You are re-deriving Raft's term check; compare your + answer with `logOk` at `raft.tla:328-331`.) +2. Exhibit the quorum-intersection argument for `Quorum = 2`, `|Replicas| = 3`: + why must the longest surviving log (line 77) hold every committed entry, and + what exactly fails when `SyncCommit = FALSE` removes the premise? Then write + the counterexample for a `Failover` that picks an *arbitrary* survivor. +3. `raft.tla:201-203` ships ≤1 entry per action "without loss of generality". + Work out what bug class a "ship everything atomically" version of *our* + `Ship` would hide, and say whether the WLOG claim would still hold. +4. Recompute Step 6's raft.tla estimate for 5 servers instead of 3, with the same + term and log bounds. By what factor does it grow, and which single variable + dominates? Now do it for `allLogs`. +5. Express topic 8's MVCC snapshot visibility as a TLA+ spec sketch: what are the + variables, what is one action, what is the invariant? This is the M21 + deliverable's outline. +6. `[][Next]_vars` allows stuttering. Construct a two-spec refinement example + (an abstract queue and a detailed one with an internal buffer) and identify + which detailed steps must map to abstract stuttering steps. ## Done when -- [ ] You can explain a state as a variable snapshot and an action as a predicate relating now to next. -- [ ] You can explain why `Next` being a disjunction gives you concurrency for free. -- [ ] You can state the model-size discipline that lets TLC finish, and name the knobs. -- [ ] You can explain why `[][Next]_vars` allows stuttering and why that is essential. -- [ ] You can state the difference between safety and liveness and which one TLC checks cheaply. -- [ ] You wrote answers to all five questions in notes.md, including the `Rejoin` action and the longest-log-among-survivors counterexample. +Answer each before unfolding it. + +- [ ] You can explain a state as a variable snapshot and an action as a predicate over two states, and name the three layers of an action. + +
Answer + + A state assigns values to all declared variables + (`WalReplication.tla:17-21`, four of them); an action is a boolean predicate + over unprimed (now) and primed (next) variables that is simply true of the + transitions it permits — no assignment, no control flow. + + The three layers, on `Ship(r)` (lines 47–51): **enabling condition** (48–49) — + in which states can it happen; **the change** (50) — `[wal EXCEPT ![r] = @+1]`, + exactly one entry; **the frame** (51) — `UNCHANGED` pins every other variable. + Dropping the frame permits those variables to take any value, producing a spec + that checks nothing while still parsing. + +
+ +- [ ] You can compute the state space of `WalReplication.tla` at its `.cfg` model and account for the gap to TLC's measured figure. + +
Answer + + `primary` 3 × `crashed` 2³ = 8 × `wal` 4³ = 64 × `committed` 4 = **6144** + type-correct states. `Crash(r)` (line 65) requires `Cardinality(Alive \ {r}) >= + Quorum` = 2, so with three replicas at most one ever crashes and only 4 of the + 8 `crashed` subsets occur: **3072**. TLC reports **1080 distinct** (`notes.md`). + + The remaining ~2.8× is what the other action guards prune — e.g. `committed > + wal[primary]` is type-correct but unreachable because `Commit` (line 59) + requires `committed < wal[primary]`, and `Failover` (line 77) requires the new + primary to have the longest surviving log. + +
+ +- [ ] You can explain why `Next` being a disjunction gives concurrency for free, and point at where Raft's unreliable network lives. + +
Answer + + Each step of a behavior is *any one* enabled disjunct, and nothing says which, + so every interleaving of enabled actions is a behavior. `\E r \in Replicas` + (`WalReplication.tla:84`) expands one line into one disjunct per replica. No + scheduler is written because none is needed. + + Raft's network unreliability is two disjuncts: `DuplicateMessage(m)` + (`raft.tla:462`) and `DropMessage(m)` (`raft.tla:463`). Loss and duplication + are transitions, not assumptions. + +
+ +- [ ] You can explain why `[][Next]_vars` permits stuttering, and why removing it would break something real. + +
Answer + + `[][Next]_vars` abbreviates `[](Next \/ UNCHANGED vars)`, so behaviors may + contain steps in which nothing changes. + + It is required for **refinement**. To show a detailed spec `D` implements an + abstract spec `A`, you map `D`'s states onto `A`'s and show each `D` step is an + `A` step. `D` has internal actions `A` does not model — they must map to + *something*, and what they map to is an `A` stuttering step. Without stuttering + no implementation could refine any abstraction, and refinement is the reason + TLA+ is a temporal logic rather than a state-machine notation. A side + consequence: step counts are never a meaningful property of a behavior. + +
+ +- [ ] You can state the model-size discipline with a price on each knob, including what logs-as-lengths saved. + +
Answer + + **Abstract the data**: `wal` as a length (`WalReplication.tla:20`) gives 4³ = 64 + values; as a sequence over 3 entry values bounded at length 3 it would be + `(1+3+9+27)³ = 40³ = 64,000` — a **1000×** multiplier, legal only because + the module header's "entries ship in order" makes logs prefixes by + construction. + + **Keep atomic regions small**: `raft.tla:201-203` sends ≤1 entry "because it + minimizes atomic regions without loss of generality". This *costs* states and + buys interleaving coverage — the opposite direction from the first knob. + + **Small constants**: 3 replicas, 3 entries, on the small-scope bet (Daniel + Jackson's, discussed in `reading-aws-cacm15.md`). + + For scale: five of raft.tla's thirteen variables at 3 servers / 3 terms / + length-3 logs already give `27 × 27 × 64 × 64,000 × 64 ≈ 1.9 × 10¹¹`, and + `allLogs` alone — a *set* of the 40 possible logs — has `2⁴⁰ ≈ 1.1 × 10¹²` + values. + +
+ +- [ ] You can state the difference between safety and liveness, and say exactly what properties `raft.tla` checks. + +
Answer + + Safety is violated by a finite trace ("nothing bad happens"), so an invariant + check finds it and TLC prints a shortest counterexample. Liveness is violated + only by an infinite behavior ("something good eventually happens") and requires + **fairness** conjuncts such as `WF_vars(Ship(r))`, because without them the + forever-stuttering behavior satisfies every spec and defeats every liveness + property. + + `raft.tla` checks **nothing**: there is no `THEOREM`, no invariant definition, + no `PROPERTY`, no `WF_`/`SF_` in its 471 lines, and no `.cfg` in the + repository — which contains only `raft.tla` and a 9-line `README.md`. The + README says to use "Jin Li's changes in Pull Request #4" to run TLC, and points + at Chapter 8 and Appendix B of Ongaro's dissertation for the safety proof. Our + own spec checks `TypeOK` and `Durability` — safety only, no `PROPERTIES`. + +
+ +- [ ] You can explain the depth-5 counterexample and which single conjunct causes it. + +
Answer + + Line 60: `SyncCommit => Cardinality(AckedBy(committed + 1)) >= Quorum`. With + `SyncCommit = FALSE` the implication is vacuously true, so `Commit` no longer + waits for a quorum ack. + + Trace: `Append` (primary's `wal` → 1) → `Commit` (`committed` → 1 with no + replica having the entry) → `Crash(primary)` → `Failover(r)` to a survivor + whose `wal` is 0 → `Durability` (`committed <= wal[primary]`) fails, since + `1 > 0`. Measured: **123 distinct states, depth 5, VIOLATED**. Breadth-first + search guarantees this is a *shortest* such trace. It is PostgreSQL's + `synchronous_commit = off` data-loss story in five steps. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the `Rejoin` trace and the quorum-intersection argument. + +
Answer + + The shape to check yours against on question 1: once a crashed replica can + rejoin with a stale `wal`, `Failover`'s "longest surviving log" (line 77) can + select a replica that was primary in an *earlier* epoch and has entries the + current primary never had — or, more simply, a rejoined stale replica can + become primary and `Ship` can now move entries *backwards* relative to the + committed point. The mechanism that rules it out is an epoch number attached to + both leaders and entries, plus a check that the predecessor entry agrees on + that epoch — which is exactly `logOk` at `raft.tla:328-331`. + + Question 2's argument in one line: with `|Replicas| = 3` and `Quorum = 2`, any + two quorums share at least `2 + 2 − 3 = 1` replica, so a committed entry (on a + quorum) is on at least one member of any surviving quorum, hence on the longest + surviving log. `SyncCommit = FALSE` removes the premise "committed ⇒ on a + quorum", and the whole argument collapses. + +
## References -**Papers** -- Lamport — *Specifying Systems* (Addison-Wesley 2002) — part I, - chapters 1-7; free PDF from Lamport's site — the rest of the book - is reference material +**Books** +- Leslie Lamport — *Specifying Systems: The TLA+ Language and Tools for Hardware + and Software Engineers* (Addison-Wesley, 2002; free PDF from Lamport's site). + Part I, chapters 1–7, is the language of Steps 1–5; chapter 8 is liveness and + fairness (Step 8). The rest is reference material. **Code** -- [raft.tla](https://github.com/ongardie/raft.tla) `raft.tla` — - Ongaro's published spec, 471 lines; anchors above -- `specs/WalReplication.tla` (this topic's experiments) — the - 94-line toy to read first +- [raft.tla](https://github.com/ongardie/raft.tla) at `6ecbdbc` — Diego Ongaro's + published Raft specification, **471 lines**. The repository contains only + `raft.tla` and a 9-line `README.md`: **no `.cfg`, no invariants, no + theorems**. The README directs would-be model checkers to Pull Request #4, and + the safety proof to Chapter 8 (Correctness) and Appendix B of + [Ongaro's dissertation](https://github.com/ongardie/dissertation). +- `specs/WalReplication.tla` (**92 lines**) and `specs/WalReplication.cfg` in + this topic — the toy to read first: 4 variables, 5 actions, 2 invariants, + 1080 reachable states. + +**In this topic** +- `notes.md` — the measured TLC runs quoted in Steps 5 and 6. +- [reading-aws-cacm15.md](reading-aws-cacm15.md) — why an organisation pays for + this, what it costs per bug, and the correct attribution of the small-scope + hypothesis behind Step 6's third knob. diff --git a/topics/21-formal/reading-z3-tacas08.md b/topics/21-formal/reading-z3-tacas08.md index dfe23a8..497edb3 100644 --- a/topics/21-formal/reading-z3-tacas08.md +++ b/topics/21-formal/reading-z3-tacas08.md @@ -1,208 +1,715 @@ # Z3: SAT plus theories, with an e-graph at the core -SMT is what turns "is this rewrite rule sound?" into a solver -query. This chapter reads de Moura & Bjørner's 4-page TACAS 2008 -tool paper — the architecture is the point — alongside Z3's modern -e-graph in `src/ast/euf/`, which turns out to be egg's data -structure ([reading-egg-popl21.md](reading-egg-popl21.md)) built -for search instead of rewriting. Before either, this chapter builds -the stack from the bottom: SAT, theories, the DPLL(T) loop, then -the e-graph's role in it. +SMT is what turns "is this rewrite rule sound?" into a solver query. This +chapter reads de Moura and Bjørner's TACAS 2008 tool paper alongside Z3's modern +e-graph in `src/ast/euf/` — which turns out to be egg's data structure +([reading-egg-popl21.md](reading-egg-popl21.md)) built for *search* rather than +rewriting, and which cites egg by name in a source comment. + +**Read the paper knowing what it is.** *Z3: An Efficient SMT Solver* is a +**four-page tool paper** — LNCS 4963, pages 337–340 — announcing a first external +release. It has an architecture figure and one short paragraph per component. It +does **not** contain DPLL(T) internals, the theory-combination algorithm, the +e-matching algorithm, or the relevancy algorithm; each of those is a separate +cited paper. Attributing them to this paper is the standard mistake, and this +chapter flags each one at the point where the temptation arises. + +Code anchors are `Z3Prover/z3` at the commit this repo pins, **`1d425e5`** +(`resources/codebases.md` pin table). Note the date gap before you read them: the +paper is 2008 and `src/ast/euf/euf_egraph.h` is headed "Copyright (c) 2020 … +Nikolaj Bjorner (nbjorner) 2020-08-23". You are reading a twelve-year-younger +rewrite next to the announcement of the original. ## The problem in one sentence -Decide whether a formula mixing booleans, integer arithmetic, -arrays, and uninterpreted functions has a satisfying assignment — -"does any input make this rewrite change the result?" is one such -formula, and Z3 answers it in milliseconds where enumeration would -take longer than the universe. +Decide whether a formula mixing booleans, integer arithmetic, arrays and +uninterpreted functions has a satisfying assignment — "does any input make this +rewrite change the result?" is one such formula, and the whole engineering +problem is that the boolean part wants exhaustive case splitting while the +arithmetic part wants a decision procedure, and the two must exchange +information without either giving up its specialised representation. ## The concepts, step by step -### Step 1 — SAT and CDCL: the boolean engine - -**SAT** is the problem of finding a true/false assignment to -boolean variables that satisfies a formula (conventionally a -conjunction of **clauses**, each a disjunction of literals like -`p ∨ ¬q`). It's NP-complete, yet modern solvers routinely handle -millions of clauses, because of **CDCL** (conflict-driven clause -learning): guess a variable (*decide*), *propagate* forced -consequences, and when a contradiction appears, analyze it into a -new **learned clause** — a compact "never go down this road again" -— then backtrack and keep it forever. Each conflict permanently -prunes an exponential slice of the search space, which is the whole -reason SAT solving works in practice. - -### Step 2 — atoms that mean something: SMT = SAT + theories - -**SMT** (satisfiability modulo theories) lifts SAT to formulas -whose atomic propositions have *meaning*: `x + y ≤ 3` is not an -opaque boolean p — it's a claim in the **theory** of linear -arithmetic. A theory solver is a decision procedure for -conjunctions of such atoms: simplex for linear arithmetic, a -congruence engine for **EUF** (equality with uninterpreted -functions — you know nothing about f except x = y ⇒ f(x) = f(y)), -plus arrays, bit-vectors. The SAT core sees only the **boolean -skeleton** — each theory atom replaced by a fresh boolean — so it -can happily assert `x ≤ 3` and `x ≥ 7` together; only the theory -solver knows those conflict. - -### Step 3 — DPLL(T): the SAT core proposes, the theories dispose - -**DPLL(T)** is the loop that couples them: the SAT core proposes -an assignment to the skeleton; the theory solvers check whether -the implied conjunction of atoms is consistent; if not, they hand -back a **theory lemma** — a clause like `¬(x≤3) ∨ ¬(x≥7)` that -encodes the inconsistency in the SAT core's language, pruning the -search exactly like a learned clause: +### Step 1 — SAT and CDCL, in the paper's own vocabulary + +> **In:** a conjunction of clauses over boolean variables. +> **Out:** the four search-pruning techniques the paper names, and the one term +> it never uses. + +**SAT** asks for a true/false assignment to boolean variables satisfying a +formula, conventionally a conjunction of **clauses**, each a disjunction of +literals such as `p ∨ ¬q`. It is NP-complete, and modern solvers handle millions +of clauses anyway. + +The mechanism is usually called **CDCL** — conflict-driven clause learning: +*decide* a variable, *propagate* forced consequences, and on contradiction +analyse the conflict into a **learned clause** ("never go down this road again"), +then backtrack and keep the clause. Each conflict permanently prunes a slice of +the search space. + +The paper never writes "CDCL". Its *SAT Solver* paragraph says: + +> "Boolean case splits are controlled using a state-of-the art SAT solver. The +> SAT solver integrates standard search pruning methods, such as **two-watch +> literals** for efficient Boolean constraint propagation, **lemma learning using +> conflict clauses**, **phase caching** for guiding case splits, and performs +> **non-chronological backtracking**." + +Four named techniques, no algorithm. That is the level the whole paper operates +at, and knowing it saves you looking for detail that is not there. + +Two of those four matter downstream in this chapter. **Non-chronological +backtracking** means the solver may jump back several decision levels at once — +Step 6's e-graph must be able to undo that many merges. **Lemma learning** is the +currency Step 3 uses to get theory knowledge into the boolean search. + +### Step 2 — SMT: atoms that mean something + +> **In:** a formula whose atoms are not opaque. +> **Out:** the boolean skeleton, and why the SAT core cannot see the +> contradiction in `x ≤ 3 ∧ x ≥ 7`. + +**SMT** (satisfiability modulo theories) lifts SAT to formulas whose atomic +propositions have meaning. `x + y ≤ 3` is not an opaque `p`; it is a claim in the +**theory** of linear arithmetic. A **theory solver** is a decision procedure for +*conjunctions* of such atoms. + +The SAT core is handed only the **boolean skeleton** — each theory atom replaced +by a fresh boolean variable. That is why it will cheerfully assert `x ≤ 3` and +`x ≥ 7` together: at the skeleton level those are two unrelated variables, both +set true, no clause violated. Only the arithmetic solver knows they conflict. + +The paper's abstract fixes the theory list: "arithmetic, bit-vectors, arrays, and +uninterpreted functions", and the introduction adds quantifiers. **EUF** — +equality with uninterpreted functions — is the theory where you know nothing +about `f` except the congruence axiom `x = y ⇒ f(x) = f(y)`, and it is the one +Step 5 is about. + +### Step 3 — the architecture figure, read as a dataflow + +> **In:** Step 1's boolean engine and Step 2's theory solvers. +> **Out:** the paper's actual figure, with its actual edge labels — this is the +> payload of the whole four pages. + +The paper's one-sentence summary: "Z3 integrates a modern DPLL-based SAT solver, +a core theory solver that handles equalities and uninterpreted functions, +satellite solvers (for arithmetic, arrays, etc.), and an **E-matching abstract +machine** (for quantifiers). Z3 is implemented in C++." + +The figure, redrawn with the edge labels the paper prints: ``` - formula (QF or quantified) - │ simplify / tactics - ▼ - ┌──────── CDCL SAT core ────────┐ boolean skeleton: - │ decide / propagate / learn │ p ∨ ¬q, p ≡ "x+y ≤ 3" … - └──────┬─────────────▲──────────┘ - │ partial │ theory lemma - │ assignment │ (conflict clause) - ▼ │ - theory solvers: EUF (congruence closure e-graph), - linear arith (simplex), arrays, bit-vectors … + SMT-LIB Simplify Native text C .NET OCaml + └────────┬──────────┘ └────┬────┘ + ▼ ▼ + Simplifier ← contextual simplification, x=4 ∧ q(x) ↦ x=4 ∧ q(4) + ▼ + Compiler ← AST becomes clauses + congruence-closure nodes + ▼ + ┌──── Congruence closure core ────┐ ◄── literal assignments ── SAT solver + │ (the E-graph) │ ─── new atoms, clauses ──► + └──┬───────────────────────▲──────┘ + │ equalities │ equalities + ▼ │ + Theory Solvers: Linear arithmetic · Bit-vectors · Arrays · Tuples + ▲ + └── E-matching engine (quantifier instantiation) ``` +Each box gets one paragraph in the paper, and each paragraph contains one +specific, quotable fact: + +- **Simplifier** — "incomplete, but efficient". Does contextual simplification: + `x = 4 ∧ q(x) ↦ x = 4 ∧ q(4)`. The trivially satisfiable conjunct `x = 4` is + *not* compiled into the core, but "kept aside in the case the client requires a + model to evaluate `x`". +- **Compiler** — converts the simplified AST into "a set of clauses and + congruence-closure nodes". This is where the boolean skeleton of Step 2 is + actually built. +- **Congruence closure core** — Step 5. +- **Deleting clauses** — quantifier instantiation produces new clauses and atoms; + Z3 garbage collects the ones "that were useless in closing branches". But: + "Conflict clauses, and literals used in them, are on the other hand not + deleted, so quantifier instantiations that were useful in producing conflicts + are retained as a side-effect." A cache-eviction policy in a theorem prover. +- **Relevancy propagation** — "**DPLL(T) based solvers** assign a Boolean value + to potentially all atoms appearing in a goal. In practice, several of these + atoms are don't cares. Z3 ignores these atoms for expensive theories, such as + bit-vectors, and inference rules, such as quantifier instantiation." That + sentence is the *only* occurrence of "DPLL(T)" in the paper, and the algorithm + is in a separate technical report (MSR-TR-2007-140). +- **Theory Solvers** — linear arithmetic "based on the algorithm used in + **Yices**"; arrays use "**lazy instantiation of array axioms**"; bit-vectors + apply "**bit-blasting to all bit-vector operations, but equality**". +- **Model generation** — models assign values to constants and "generate partial + function graphs for predicates and function symbols". + +The **DPLL(T)** loop those pieces implement — SAT core proposes a partial +assignment, theory solvers check consistency, an inconsistency comes back as a +clause the SAT core can learn — is worth having in your head, but it is *not* +described in this paper. Here it is as pseudocode, so you can hold it while +reading the figure: + ```rust -// DPLL(T): the SAT core proposes, the theory solvers dispose -fn smt_solve(mut clauses: Vec, theories: &Theories) -> Result { - loop { - match sat_cdcl(&clauses) { - Unsat => return Unsat, // even the skeleton is out - Sat(assignment) => { - // the boolean skeleton says: these theory atoms hold - match theories.check(assignment.atoms()) { - Consistent(model) => return Sat(model), - Conflict(lemma) => clauses.push(lemma), - // the lemma ("¬(x≤3) ∨ ¬(x≥7)") prunes the SAT - // search — theory knowledge flows back as clauses - } - } +// ILLUSTRATION — not Z3 code. The contract the figure implies; the real loop +// interleaves theory checks with propagation. For the code, read the worklist +// drain at src/ast/euf/euf_egraph.cpp:654 and the merge at :511. +loop { + match sat_core.next_assignment() { + Unsat => return Unsat, // even the skeleton is out + Sat(assignment) => match theories.check(assignment.atoms()) { + Consistent(model) => return Sat(model), + Conflict(lemma) => sat_core.learn(lemma), // e.g. ¬(x≤3) ∨ ¬(x≥7) } } } ``` -(Real solvers interleave theory checks *during* propagation rather -than waiting for full assignments — but the contract is this loop.) -The division of labor is the design's genius: boolean case -splitting is CDCL's specialty, theory reasoning stays inside -specialized procedures, and clauses are the only currency between -them. - -### Step 4 — theories must also talk to each other: Nelson-Oppen - -A formula like `f(x) = f(y) ∧ x + 1 ≤ y ∧ y ≤ x + 1` splits atoms -between EUF and arithmetic — but arithmetic knows x = y and only -EUF can conclude f(x) = f(y). The **Nelson-Oppen** combination -scheme has theories cooperate by exchanging exactly one kind of -fact: *equalities between shared terms*. Each theory propagates -the equalities it can derive; the others consume them. Equalities -are the narrow-waist interface — the analogy to operators -exchanging join keys (topic 11) is question 4. - -### Step 5 — the e-graph, again: EUF's engine, with two extra duties - -EUF's decision procedure is congruence closure over an e-graph — -the very structure from the egg chapter (e-classes of equal terms, -hashcons, congruence: equal children ⇒ equal parents). Z3's -modern rewrite of it lives in `src/ast/euf/`: - -| anchor | what | -|---|---| -| `euf_egraph.h:23` | comment: "same effect as delayed congruence table reconstruction **from egg**" — the 2021 paper flowing back into the 2008 solver | -| `euf_egraph.h:85` | `class egraph` | -| `euf_egraph.h:91-96` | `to_merge` queue (plain / commutativity / justified) — the pending-unions worklist, egg's `pending` | -| `euf_enode.h` | e-node: term + parents + root pointer | -| `euf_etable.h` | the congruence table (hashcons keyed on canonicalized children) | -| `euf_justification.h` | proof-producing unions — egg's `explain.rs` counterpart; Z3 needs it for conflict lemmas | - -Key difference from egg: Z3's e-graph must support **backtracking** -(the SAT core undoes decisions, so unions must be undoable via a -trail — a log of mutations replayed in reverse) and -**justifications** (every merge must be explainable, because a -theory conflict must be handed back as a *specific* lemma naming -the guilty atoms). egg only needs monotone growth + optional -explanations. Same structure, different contract — and the -deferred-repair idea still transferred (the :23 comment), 13 years -from solver to library and back. - -### Step 6 — quantifiers: e-matching, heuristic by necessity - -A quantified axiom like `∀x. f(g(x)) = x` can't be handed to CDCL -— there are infinitely many instances. Z3 picks a **trigger** (a -subterm pattern, here `f(g(x))`) and instantiates the axiom for -every term in the e-graph matching the trigger *modulo the known -equalities* — that matching is **e-matching**, implemented as an -abstract machine (`euf_mam.h` — egg's `machine.rs`, industrial -strength). This is why quantified SMT is incomplete-but-useful: -instantiation is heuristic — too general a trigger floods the -solver with instances, too specific misses the needed one -(question 5 calls this the "index choice" problem of SMT). - -### Step 7 — where a database meets Z3 - -- **Query equivalence** (Cosette, topic 16): compile two SQL plans - to formulas, ask Z3 if outputs can differ. UNSAT = equivalent. -- **Constraint-based test generation**: "give me a row that makes - this WHERE clause true" is a SAT query. -- **Optimizer rule soundness**: our `x/x → 1` caveat is checkable — - `assert x=0 ∧ rewrite-changes-result`, SAT means unsound rule. - -The usage pattern is always the same inversion: encode "a -counterexample exists" and hope for UNSAT — the solver's failure -to satisfy is your proof. +The division of labour is the design: boolean case splitting stays in CDCL, +theory reasoning stays inside specialised procedures, and **clauses are the only +currency between them**. + +### Step 4 — theory combination: what Z3 does *instead of* Nelson-Oppen + +> **In:** two theory solvers that each know part of a formula. +> **Out:** the classical answer, and the paper's explicit statement that Z3 does +> something else — the correction that matters most in this chapter. + +The problem is real. In `f(x) = f(y) ∧ x + 1 ≤ y ∧ y ≤ x + 1`, arithmetic can +derive `x = y` but knows nothing about `f`; EUF can conclude `f(x) = f(y)` but +only if someone tells it `x = y`. The classical solution is **Nelson–Oppen** +combination: theories cooperate by exchanging exactly one kind of fact — +*equalities between shared terms* — and each must be able to produce all the +equalities it implies. + +**Z3 does not do this, and the paper says so in its own section:** + +> "Traditional methods for combining theory solvers rely on capabilities of the +> solvers to produce all implied equalities or a pre-processing step that +> introduces additional literals into the search space. Z3 uses a new theory +> combination method that **incrementally reconciles models maintained by each +> theory** [5]." + +Reference [5] is de Moura and Bjørner, *Model-based Theory Combination*, SMT +2007. The idea named there is different in kind: rather than deriving and +exchanging implied equalities, each theory keeps a candidate **model**, and the +combination procedure looks at those models for variables that happen to be +assigned equal values, guesses the corresponding equality, and repairs when the +guess fails. It is a search-with-backtracking strategy where Nelson–Oppen is a +deduction strategy — which is why the paper calls out the two costs it avoids +(producing *all* implied equalities; introducing extra literals). + +Do not write "Z3 uses Nelson–Oppen" and cite this paper. The paper's only mention +of the traditional method is to say it is not what Z3 does. + +The equality-exchange picture is still useful, because it is what the e-graph +actually implements at the interface: "Nodes in the E-graph may point to one or +more theory solvers. When two nodes are merged, the set of theory solver +references are merged, and the merge is propagated as an equality to the theory +solvers **in the intersection** of the two sets of solver references." That +sentence describes a real field on a real struct — Step 5. + +### Step 5 — the e-graph: same structure, different contract + +> **In:** the egg chapter's e-graph. +> **Out:** the fields Z3 adds, and the two requirements — backtracking and +> justification — that make them necessary. + +The paper is explicit that the structure is borrowed and even that the name is: +"Equalities asserted by the SAT solver are propagated by the congruence closure +core using a data structure that we will call an **E-graph following [8]**" — +[8] being Detlefs, Nelson and Saxe's *Simplify* (JACM 52(3), 2005). + +Open `euf_enode.h` and the paper's architecture paragraph turns into fields: + +```cpp +// z3 src/ast/euf/euf_enode.h, lines 40-65 — the enode, boolean flags elided + 40 class enode { + 41 expr* m_expr = nullptr; + 50 bool m_is_relevant = false; + 51 lbool m_is_shared = l_undef; + 52 lbool m_value = l_undef; // Assignment by SAT solver for Boolean node + 53 sat::bool_var m_bool_var = sat::null_bool_var; // SAT solver variable associated with Boolean node + 54 unsigned m_class_size = 1; // Size of the equivalence class if the enode is the root. + 56 unsigned m_generation = 0; // Tracks how many quantifier instantiation rounds were needed to generate this enode. + 57 enode_vector m_parents; + 58 enode* m_next = nullptr; + 59 enode* m_root = nullptr; + 62 th_var_list m_th_vars; + 63 justification m_justification; +``` + +Read it against the paper: `m_th_vars` (62) is "nodes in the E-graph may point to +one or more theory solvers"; `m_bool_var` and `m_value` (52–53) are the wire from +the SAT solver; `m_is_relevant` (50) is the relevancy-propagation paragraph; +`m_generation` (56) is the e-matching paragraph's instantiation rounds; +`m_justification` (63) is what makes conflicts explainable. egg's `EClass` has +none of these — because egg answers a different question. + +**Difference 1: Z3 has no union-find path to walk.** `m_root` (59) points +directly at the class root, always, and `merge` maintains that eagerly: + +```cpp +// z3 src/ast/euf/euf_egraph.cpp, lines 536-551 — union by class size, eager roots + 536 if (!r2->interpreted() && + 537 (r1->class_size() > r2->class_size() || r1->interpreted() || r1->value() != l_undef)) { + 538 std::swap(r1, r2); + 539 std::swap(n1, n2); + 540 } + 542 remove_parents(r1); + 543 push_eq(r1, n1, r2->num_parents()); + 545 for (enode* c : enode_class(n1)) + 546 c->m_root = r2; + 548 r2->inc_class_size(r1->class_size()); + 551 reinsert_parents(r1, r2); +``` + +Line 537 is **union by class size** (smaller class becomes `r1`), and 545–546 +rewrites `m_root` for **every node in the smaller class**. The header comment +says as much: `euf_egraph.h:20`, "it still uses eager path compression." + +**Work the cost.** Union by size means a node's root is rewritten only when the +class containing it at least doubles, so each node is rewritten at most `log₂ n` +times. Merging `n = 1000` singleton nodes into one class therefore costs at most +`1000 × log₂ 1000 ≈ 1000 × 9.97 ≈ 9,966` root writes **in total** — and every +`get_root()` (`euf_enode.h:203`) is a single load, forever, with no pointer chain +to walk. + +Compare egg (`unionfind.rs:47-50`): `union` writes **one** pointer, and `find` +(`:30-35`) pays by walking a chain at read time. The two libraries chose opposite +sides of the same trade because their access patterns are opposite: Z3's SAT core +canonicalizes constantly during propagation and merges relatively rarely per +query, while egg merges in enormous batches and then reads in bulk at rebuild. + +**Difference 2: everything must be undoable.** Step 1's non-chronological +backtracking means merges get retracted, in bulk. `push()` and `pop(unsigned)` +(`euf_egraph.h:277-278`) bracket scopes, `update_record` (`euf_egraph.h:112`) is +the trail entry, and `push_eq` at line 543 above records the pre-merge parent +count. Undo is the mirror image of merge: + +```cpp +// z3 src/ast/euf/euf_egraph.cpp, lines 627-650 — undo_eq, traces elided + 627 void egraph::undo_eq(enode* r1, enode* n1, unsigned r2_num_parents) { + 628 enode* r2 = r1->get_root(); + 630 r2->dec_class_size(r1->class_size()); + 632 std::swap(r1->m_next, r2->m_next); + 633 auto begin = r2->begin_parents() + r2_num_parents, end = r2->end_parents(); + 634 for (auto it = begin; it != end; ++it) { + 639 if (p->cgc_enabled()) + 640 erase_from_table(p); + 641 } + 643 for (enode* c : enode_class(r1)) + 644 c->m_root = r1; + 649 r2->m_parents.shrink(r2_num_parents); + 650 unmerge_justification(n1); + 651 } +``` + +Line 643–644 is line 545–546 run backwards; line 649 truncates the parent vector +to the length recorded at merge time. **This is why Z3 cannot use egg's +union-find.** `find_mut`'s path halving (`unionfind.rs:37-44`) rewrites parent +pointers as a side effect of *reading*, and undoing those would require logging +every compressed pointer. Eager roots make undo a matter of re-walking one class +list; lazy roots with compression would make it a general-purpose trail of every +read. The backtracking requirement chose the data structure. + +**Difference 3: justifications.** Every merge records *why*: + +```cpp +// z3 src/ast/euf/euf_justification.h, lines 41-47 — the five reasons to merge + 41 enum class kind_t { + 42 axiom_t, + 43 congruence_t, + 44 external_t, + 45 dependent_t, + 46 equality_t + 47 }; +``` + +A theory conflict has to be handed back as a *specific* clause naming the guilty +atoms — a lemma over `x ≤ 3` and `x ≥ 7`, not "something is wrong". So the +e-graph must be able to explain any derived equality in terms of asserted ones, +which is what `push_congruence` (`euf_egraph.cpp:765`) does by walking to the +least common ancestor of each argument pair. egg's equivalent, `explain.rs`, is +**optional** and off by default; in Z3 it is load-bearing. + +### Step 6 — congruence repair: eager table, deferred merges + +> **In:** Step 5's merge. +> **Out:** what Z3's worklist actually defers, and an honest reading of the +> source comment that cites egg. + +The comment is real, and it is the first thing in the file: + +```cpp +// z3 src/ast/euf/euf_egraph.h, lines 16-24 — the header's Notes block, verbatim + 16 Notes: + 17 + 18 It relies on + 19 - data structures form the (legacy) SMT solver. + 20 - it still uses eager path compression. + 21 + 22 NB. The worklist is in reality inherited from the legacy SMT solver. + 23 It is claimed to have the same effect as delayed congruence table reconstruction from egg. + 24 Similar to the legacy solver, parents are partially deduplicated. +``` + +Read line 22 before line 23. The worklist is **inherited from the legacy SMT +solver** — it predates egg — and line 23 says it "is **claimed** to have the same +effect", which is a careful hedge, not an adoption notice. (egg's own paper, +footnote 5, makes the matching claim from the other side: Z3's e-graph separates +read and write phases "as an implementation detail", and egg is "the first +algorithm to take advantage of this by deferring invariant maintenance.") + +And in the code, what is deferred is narrower than egg's rebuilding. +`egraph::merge` repairs the congruence table **inline, in the same call** — +`remove_parents(r1)` at line 542 pulls the parents out of the hash table and +`reinsert_parents(r1, r2)` at 551 puts them back canonicalized. What gets queued +is the *consequences*: + +```cpp +// z3 src/ast/euf/euf_egraph.cpp, lines 592-599 — a table collision becomes a queued merge + 592 if (p->cgc_enabled()) { + 593 auto [p_other, comm] = insert_table(p); + 596 if (p_other != p) + 597 m_to_merge.push_back(to_merge(p_other, p, comm)); + 598 else + 599 r2->m_parents.push_back(p); + 600 if (p->is_equality()) +``` + +Line 596–597 is the same discovery egg makes at `egraph.rs:1353` — a hash +collision *is* a congruence — but instead of recursing it appends to +`m_to_merge`, drained by a fixpoint loop: + +```cpp +// z3 src/ast/euf/euf_egraph.cpp, lines 654-677 — the propagate fixpoint + 654 bool egraph::propagate() { + 656 unsigned i = 0; + 657 bool change = true; + 658 while (change) { + 659 change = false; + 660 propagate_plugins(); + 661 for (; i < m_to_merge.size() && m.limit().inc() && !inconsistent(); ++i) { + 662 auto const& w = m_to_merge[i]; + 666 merge(w.a, w.b, justification::congruence(w.commutativity(), m_congruence_timestamp++)); + 675 } + 676 } + 677 m_to_merge.reset(); +``` + +So the accurate statement is: **Z3 defers the cascading merges, egg defers the +table repair as well.** Both replace recursion with a worklist; egg additionally +lets the hashcons hold non-canonical keys between rebuilds +(`egraph.rs:63-64`), which Z3 does not do — its table is canonical at the end of +every `merge`, because a solver that must answer `get_root()` and produce a +conflict at any moment cannot afford a window in which its index is wrong. + +### Step 7 — quantifiers: e-matching, and where triggers actually come from + +> **In:** an axiom like `∀x. f(g(x)) = x`. +> **Out:** what the tool paper claims, what it does not, and the mechanism's +> real fragility. + +A quantified axiom cannot be handed to CDCL: there are infinitely many instances. +The standard approach instantiates the axiom only for terms already present, +matched **modulo the equalities the e-graph currently knows** — that is +**e-matching**. The subterm pattern used to find candidates is a **trigger** +(here `f(g(x))`). + +The paper's entire quantifier paragraph is three sentences: + +> "Z3 uses a well known approach for quantifier reasoning that works over an +> E-graph to instantiate quantified variables. Z3 uses new algorithms that +> identify matches on E-graphs incrementally and efficiently. Experimental +> results show substantial performance improvements over existing +> state-of-the-art SMT solvers [4]." + +Note what is absent. **The word "trigger" never appears in this paper.** Neither +does model-based quantifier instantiation, nor any description of how patterns +are selected or how the matching machine works. The "well known approach" is +Simplify's [8]; the "new algorithms" are reference [4], Bjørner and de Moura, +*Efficient E-Matching for SMT Solvers*, CADE 2007. Cite those, not this. + +What you *can* verify is that the machine exists and predates the paper: + +```cpp +// z3 src/ast/euf/euf_mam.h, lines 8-15 and 50 — the Matching Abstract Machine + 8 Abstract: + 9 + 10 Matching Abstract Machine + 12 Author: + 14 Leonardo de Moura (leonardo) 2007-02-13. + 15 Nikolaj Bjorner (nbjorner) 2021-01-22. + 50 class mam { +``` + +The 2007 date on line 14 matches the CADE'07 paper; the 2021 date on line 15 is +the port into the new `euf` layer. This is egg's `machine.rs`, at industrial +scale and fourteen years older. + +The fragility is worth stating plainly because it is the practical face of +"incomplete but useful": instantiation is heuristic. Too general a trigger floods +the solver with useless instances — and by the *Deleting clauses* paragraph of +Step 3, the useless ones get garbage-collected while the ones that produced +conflicts are kept, which is a mitigation, not a cure. Too specific a trigger +never fires and the needed fact is never derived, so the solver returns `unknown` +on a valid formula. Question 5 calls this the index-choice problem of SMT: the +same shape as picking which index to build, with the same failure modes at both +extremes. + +### Step 8 — where a database meets Z3 + +> **In:** a solver that decides formulas. +> **Out:** three concrete uses, and the one inversion they all share. + +- **Query equivalence** (Cosette, topic 16): compile two SQL plans to formulas + and ask whether their outputs can differ. `unsat` means equivalent. +- **Constraint-based test generation**: "give me a row that makes this `WHERE` + clause true" is literally a satisfiability query, and the paper's *Pex* client + (§2) is exactly this pattern for unit tests — "Z3 is used to produce new test + cases with different behavior." +- **Optimizer rule soundness**: the `div-same` rewrite `(/ ?x ?x) => 1`, which + this topic's stub suggests you add to the saturating lane + (`experiments/src/eqsat.rs:87`, in the doc comment on `egg_optimize`), is + checkable. Assert `x = 0 ∧ (x/x ≠ 1)` and ask whether it is satisfiable. Over + the integers with SMT-LIB's *total* `div`, `(div 0 0)` is an + arbitrary-but-fixed value rather than an error, so the query is satisfiable and + the rule is unsound as written; over the reals with the same totalisation + convention, likewise. The rule is still fine for the trap expression, where it + only ever fires on the literal `(/ 2 2)` — but that is a fact about the input, + not about the rule, and a solver is how you find out which one you have. + +The usage pattern is always the same inversion: encode "a counterexample exists" +and hope for `unsat`. **The solver's failure to satisfy is your proof** — which +is also why an `unknown` from a quantified query (Step 7) is not a proof of +anything. ## How to read the paper (with the concepts in hand) -It's 4 pages — read all of it. The architecture diagram is the -payload: it's step 3's picture with Z3's actual component names. -Map each named component to a step as you read (SAT core → step 1, -theory solvers and their combination → steps 2-4, congruence -closure → step 5, e-matching/quantifiers → step 6). Then read the -`src/ast/euf/` headers in the order of step 5's anchor table — -starting with the comment at `euf_egraph.h:23`, the 2021 idea -cited inside the 2008 solver. +Four pages, LNCS 4963 pp. 337–340. Read all of it in twenty minutes, then spend +the afternoon in `src/ast/euf/`. + +- **§1 Introduction** — the adoption facts, which are the point of a tool paper: + a prototype won **4 first places and 7 second places at SMT-COMP'07**; first + external release **September 2007**; in use at Microsoft since **February 2007** + in Spec#/Boogie, Pex, HAVOC, Vigilante, VCC and Yogi. Note the sentence "Z3 + uses novel algorithms for quantifier instantiation [4] and theory combination + [5]" — that is the paper telling you where its own content is not. +- **§2 Clients** — Spec#/Boogie and Pex. Three textual input formats (SMT-LIB, + Simplify, a native DIMACS-like one) and three APIs (ANSI C, .NET, OCaml). +- **§3 System Architecture** — the figure and the per-component paragraphs of + Step 3. Read *Theory Combination* twice; it is the paragraph most often + misremembered (Step 4). +- **§4 Conclusion** — four sentences. + +Then read the code in this order: the `Notes:` block at `euf_egraph.h:16-24` +(Step 6), `euf_enode.h:40-69` (Step 5's field-by-field map to the paper), +`egraph::merge` at `euf_egraph.cpp:511`, `undo_eq` at `:627`, and +`propagate` at `:654`. ## Questions (answer in notes.md) -1. Why must Z3's e-graph carry justifications while egg's can skip - them? What would proof-producing unions cost egg's rebuild? -2. The trail/backtracking requirement: why does deferred rebuilding - interact badly with undo, and how does `to_merge_t` (:91) hint at - the resolution? -3. Encode the `x/x → 1` soundness check as an SMT query (ints, then - reals). Which theory answers each? -4. Nelson-Oppen needs theories to agree on equalities of shared - terms — spot the analogy to exchanging join keys between operators - (topic 11). -5. E-matching triggers: why is trigger selection the "index choice" - problem of SMT (too general = blowup, too specific = incomplete)? +1. Why must Z3's e-graph carry justifications while egg's `explain.rs` is + optional and off by default? Name the specific output that needs them, and + estimate what always-on proof production would cost egg's `rebuild`. +2. Path compression versus eager roots: state the read/write cost of each + (`unionfind.rs:30-50` against `euf_egraph.cpp:536-551`), then explain in one + sentence why `undo_eq` (`:627-651`) would be impractical if Z3 used egg's + `find_mut`. +3. Recompute Step 5's arithmetic for `n = 10⁶` nodes. How many root writes, worst + case, and how many pointer dereferences does a `get_root()` cost after them? + Do the same for egg's `find` on a maximally unbalanced forest. +4. Encode the `x/x → 1` soundness check as an SMT query, first over `Int` and + then over `Real`. Which theory answers each, and what does SMT-LIB's + totalisation of division do to your answer? +5. Trigger selection is the index-choice problem of SMT. Write out both failure + modes with a concrete axiom, and say which one the *Deleting clauses* + paragraph partially mitigates and which it does not. +6. The paper says Z3 "incrementally reconciles models maintained by each theory" + rather than exchanging all implied equalities. Name one cost Nelson–Oppen pays + that model-based combination avoids, and one risk model-based combination + takes that Nelson–Oppen does not. ## Done when -- [ ] You can explain DPLL(T): the SAT core proposes, theories dispose. -- [ ] You can explain why Nelson-Oppen requires theories to agree on shared equalities. -- [ ] You can say why Z3's e-graph must carry justifications while egg's need not, and connect it to backtracking. -- [ ] You can explain e-matching and why trigger selection is the index-choice problem of theorem proving. -- [ ] You can encode `x/x -> 1` as an SMT query and say what changes between integers and reals. -- [ ] You wrote answers to all five questions in notes.md. +Answer each before unfolding it. + +- [ ] You can state what the TACAS'08 paper does and does not contain, and name three things commonly misattributed to it. + +
Answer + + It is a **four-page tool paper** (LNCS 4963, 337–340) announcing Z3's first + external release: clients, an architecture figure, one paragraph per component, + adoption facts. It contains no algorithms. + + Commonly misattributed: **DPLL(T) internals** (the term appears once, inside + the *Relevancy propagation* paragraph, describing a class of solvers); + **e-matching / quantifier instantiation** (reference [4], Bjørner and de Moura, + CADE 2007 — and the word "trigger" never appears); **theory combination** + (reference [5], *Model-based Theory Combination*, SMT 2007). Also frequently + mis-said: "CDCL", which the paper never writes — it names two-watch literals, + lemma learning using conflict clauses, phase caching and non-chronological + backtracking. + +
+ +- [ ] You can explain what Z3 does instead of Nelson–Oppen, and quote the paper on it. + +
Answer + + Nelson–Oppen has theories exchange **equalities between shared terms**, and + requires each solver to produce all the equalities it implies. The paper's + *Theory Combination* paragraph rejects that: "Traditional methods for combining + theory solvers rely on capabilities of the solvers to produce all implied + equalities or a pre-processing step that introduces additional literals into + the search space. Z3 uses a new theory combination method that **incrementally + reconciles models maintained by each theory** [5]." + + Reference [5] is *Model-based Theory Combination* (SMT 2007): each theory keeps + a candidate model, equalities are *guessed* from variables that happen to be + assigned equal values, and wrong guesses are repaired. Search where + Nelson–Oppen deduces. The two costs it names as avoided are producing all + implied equalities and introducing extra literals. + +
+ +- [ ] You can map at least four fields of `euf::enode` onto sentences of the paper's architecture section. + +
Answer + + From `euf_enode.h:40-65`: `m_th_vars` (62) ↔ "Nodes in the E-graph may point to + one or more theory solvers … the merge is propagated as an equality to the + theory solvers in the intersection of the two sets"; `m_bool_var` (53) and + `m_value` (52) ↔ "The congruence closure core receives truth assignments to + atoms from the SAT solver"; `m_is_relevant` (50) ↔ the *Relevancy propagation* + paragraph; `m_generation` (56, "how many quantifier instantiation rounds were + needed to generate this enode") ↔ the E-matching paragraph; `m_justification` + (63) ↔ the conflict clauses that flow back to the SAT solver. + + egg's `EClass` has none of these, which is the compact statement of "same data + structure, different contract". + +
+ +- [ ] You can compute the cost of Z3's eager root maintenance and say why it, rather than egg's, is the right choice here. + +
Answer + + `merge` swaps so the smaller class is `r1` (`euf_egraph.cpp:536-540`, union by + `class_size`) and then rewrites `m_root` for every node in it (545–546). Union + by size means a node is rewritten only when its class at least doubles, so at + most `log₂ n` times: merging `n = 1000` singletons costs at most + `1000 × log₂ 1000 ≈ 9,966` root writes **in total**, and every `get_root()` + is one load with no chain to walk. + + egg does the opposite: `union` writes one pointer (`unionfind.rs:47-50`) and + `find` walks (`:30-35`). Z3 needs O(1) reads because the SAT core canonicalizes + constantly during propagation, and — decisively — because `undo_eq` + (`euf_egraph.cpp:643-644`) restores roots by re-walking one class list. With + egg's `find_mut` path halving (`unionfind.rs:37-44`), *reading* mutates the + forest, so undo would need a trail entry per compressed pointer. Backtracking + chose the data structure. + +
+ +- [ ] You can say precisely what Z3's worklist defers and how that differs from egg's rebuilding, and read the egg-citing comment correctly. + +
Answer + + Z3's `merge` repairs the congruence table **inline**: `remove_parents(r1)` + (`euf_egraph.cpp:542`) then `reinsert_parents(r1, r2)` (`:551`). What it queues + is the *consequential merges* — a table collision at `:596-597` pushes onto + `m_to_merge`, drained by the fixpoint at `:654-677`. egg defers the table + repair as well, so its hashcons holds non-canonical keys between rebuilds + (`egraph.rs:63-64`); Z3's table is canonical when `merge` returns, because a + solver must be able to answer `get_root()` and produce a conflict at any + instant. + + The comment: `euf_egraph.h:22` says the worklist "is in reality **inherited + from the legacy SMT solver**" and `:23` that it "is **claimed** to have the same + effect as delayed congruence table reconstruction from egg." That is a hedged + note of resemblance, not "Z3 adopted egg's algorithm". egg's paper footnote 5 + makes the mirror-image claim from its side. + +
+ +- [ ] You can explain e-matching and trigger selection, and say which paper to cite for it. + +
Answer + + A quantified axiom has infinitely many instances, so instead of case-splitting + it, instantiate it only for terms already in the e-graph that match a **trigger** + — a subterm pattern — **modulo the equalities currently known**, which is what + makes it *e*-matching rather than plain matching. + + Cite **Bjørner and de Moura, *Efficient E-Matching for SMT Solvers*, CADE 2007** + (the TACAS paper's reference [4]) and Simplify (reference [8]) for the "well + known approach". The tool paper says only that Z3 "uses new algorithms that + identify matches on E-graphs incrementally and efficiently" and never uses the + word "trigger". The machine is `src/ast/euf/euf_mam.h` — "Matching Abstract + Machine", authored 2007-02-13, ported into `euf` in 2021. + + Failure modes: too general a trigger floods the solver with instances (partly + mitigated by the clause GC of the *Deleting clauses* paragraph, which keeps + only instantiations that produced conflicts); too specific never fires, and the + solver returns `unknown` on a valid formula — which is not a proof of anything. + +
+ +- [ ] You wrote answers to all six questions in notes.md, including the `x/x → 1` encoding over both sorts. + +
Answer + + The shape to check yours against: assert `(and (= x 0) (not (= (div x x) 1)))` + over `Int`. SMT-LIB makes division **total** — `(div a 0)` is an + uninterpreted-but-fixed value rather than an error — so the solver can satisfy + this, and the rewrite `x/x → 1` is unsound at `x = 0` unless your language + guarantees a non-zero divisor. `Real` behaves the same way for the same reason. + The theory answering it is linear arithmetic plus EUF for the totalised + division symbol. + + Record what this tells you about the `div-same` rule the stub suggests + (`experiments/src/eqsat.rs:87`): it is fine for the trap expression, where the + only redex is the literal `(/ 2 2)`, and it is not a rule you could ship in a + real optimiser without a non-zero side condition. + +
## References **Papers** -- de Moura, Bjørner — "Z3: An Efficient SMT Solver" (TACAS 2008) — - 4 pages; read all of it for the architecture diagram - -**Code** -- [z3](https://github.com/Z3Prover/z3) `src/ast/euf/` — - `euf_egraph.h` (:23 cites egg's deferred repair, :91-96 the - `to_merge` worklist), `euf_enode.h`, `euf_etable.h`, - `euf_justification.h`, `euf_mam.h` (e-matching abstract machine) +- Leonardo de Moura, Nikolaj Bjørner — *Z3: An Efficient SMT Solver*, TACAS 2008, + LNCS 4963, pp. 337–340. Four pages. The architecture figure of Step 3 is the + payload; the *Theory Combination* paragraph is Step 4. +- Leonardo de Moura, Nikolaj Bjørner — *Model-based Theory Combination*, SMT 2007 + (the tool paper's [5]) — what Z3 does **instead of** Nelson–Oppen. +- Nikolaj Bjørner, Leonardo de Moura — *Efficient E-Matching for SMT Solvers*, + CADE 2007, LNCS 4603, pp. 183–198 (the tool paper's [4]) — Step 7's actual + source. +- David Detlefs, Greg Nelson, James B. Saxe — *Simplify: a theorem prover for + program checking*, JACM 52(3), 2005 (the tool paper's [8]) — where the term + "E-graph" and the trigger-based approach come from. +- Bruno Dutertre, Leonardo de Moura — *A Fast Linear-Arithmetic Solver for + DPLL(T)*, CAV 2006 (the tool paper's [9]) — the Yices algorithm Z3's arithmetic + solver is based on. +- de Moura, Bjørner — *Relevancy Propagation*, MSR-TR-2007-140 — the don't-care + algorithm the *Relevancy propagation* paragraph points at. + +**Code** — `Z3Prover/z3` at `1d425e5` + +| File | Lines | What | +|------|-------|------| +| `src/ast/euf/euf_egraph.h` | 382 | `class egraph` (85), the `to_merge` queue (91–100), scopes (277–278), the `Notes:` block citing egg (16–24) | +| `src/ast/euf/euf_egraph.cpp` | 1120 | `merge` (511), `remove_parents` (563), `reinsert_parents` (585), `undo_eq` (627), `propagate` (654), `push_congruence` (765) | +| `src/ast/euf/euf_enode.h` | 310 | `class enode` (40) — the field-by-field map to the paper's architecture section | +| `src/ast/euf/euf_etable.h` | — | the congruence table: `cg_hash`/`cg_eq` (109, 113), `insert` (166) with its commutativity note | +| `src/ast/euf/euf_justification.h` | 143 | `kind_t` (41–47) — the five reasons two nodes are equal | +| `src/ast/euf/euf_mam.h` | 85 | the Matching Abstract Machine, `class mam` (50) | + +**In this topic** +- [reading-egg-popl21.md](reading-egg-popl21.md) — the same data structure with + the opposite contract; read it first. +- `experiments/src/eqsat.rs:82-98` — the stub's suggested rewrite list, including + the `div-same` rule Step 8 puts under a solver. diff --git a/topics/22-benchmarks/README.md b/topics/22-benchmarks/README.md index 54283fd..071962e 100644 --- a/topics/22-benchmarks/README.md +++ b/topics/22-benchmarks/README.md @@ -22,7 +22,7 @@ M22's standing regression suite. ```mermaid graph TD Q["a benchmark number"] --> W["workload: mix + distribution
(YCSB: A-F × zipfian/uniform)"] - Q --> D["data: scale factor + skew +
correlation (dbgen: none!)"] + Q --> D["data: scale factor + skew +
correlation (dbgen: dates only)"] Q --> H["harness: open vs closed loop,
think times, warmup, driver cost"] Q --> M["metric: tpmC? geomean?
p999? GB/s? recall@10?"] W & D & H & M --> V{"change ANY one
⇒ different number"} @@ -30,8 +30,9 @@ graph TD ## Choke points, one line each -- **TPC-H Q1**: tiny group domain ⇒ hash table free ⇒ pure - expression eval + fused agg (our `q1_flat` makes it explicit). +- **TPC-H Q1**: tiny group domain (Boncz's CP1.3, Small Group-By Keys) + ⇒ hash table free ⇒ pure expression eval + fused agg (our `q1_flat` + makes it explicit). - **TPC-H Q6**: 2%-selective scan ⇒ SIMD predicates, the "GB/s" headline query (our `q6_branchless`, topic 17's filter shapes). - **TPC-H Q9**: 6-way join order + LIKE '%green%' + skew — the diff --git a/topics/22-benchmarks/experiments/.gitignore b/topics/22-benchmarks/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/22-benchmarks/experiments/.gitignore +++ b/topics/22-benchmarks/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/22-benchmarks/experiments/Cargo.lock b/topics/22-benchmarks/experiments/Cargo.lock new file mode 100644 index 0000000..8235df4 --- /dev/null +++ b/topics/22-benchmarks/experiments/Cargo.lock @@ -0,0 +1,134 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bench-experiments" +version = "0.1.0" +dependencies = [ + "rand", + "rand_chacha", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rand" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22f6172bdec972074665ed81ed53b71da00bfc44b65a753cfde883ec4c702a1a" +dependencies = [ + "libc", + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom", +] + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "zerocopy" +version = "0.8.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] diff --git a/topics/22-benchmarks/notes.md b/topics/22-benchmarks/notes.md index d149b31..bdfe061 100644 --- a/topics/22-benchmarks/notes.md +++ b/topics/22-benchmarks/notes.md @@ -2,6 +2,12 @@ ## Baseline (provided code, Apple M3 Pro, measured 2026-07-10) +> These figures predate the 2026-07-28 re-run recorded in +> [FINDINGS.md](../../FINDINGS.md) row 22 (Q1 5.2–5.7 GB/s, Q6 +> 9.0–14.4 GB/s, YCSB-E p999 12.9 µs against read-only's 4.0 µs). Same +> machine, later run; where the two disagree, FINDINGS is canonical. +> Re-run `./verify.sh 22` before treating any cell below as current. + ### TPC-H choke points (bench_suite, dbgen-lite) | SF | rows | Q1 oracle ms | Q1 GB/s | Q6 oracle ms | Q6 GB/s | @@ -10,7 +16,8 @@ | 0.25 | 1.5M | 10.2 | 5.6 | 2.7 | 15.7 | Q1 oracle: HashMap entry per row (even with only 6 groups) + 4 f64 -FMAs — hashing dominates, exactly the CP1.2 story: the group domain +FMAs — hashing dominates, exactly the CP1.3 story (Small Group-By +Keys, Boncz TPCTC'13 Table 1): the group domain is tiny, so a real engine replaces the hash with an array and turns Q1 into an expression benchmark. Q6 branchy oracle already at 15.7 GB/s — the 2% selectivity means the branch is nearly diff --git a/topics/22-benchmarks/reading-boncz-tpch.md b/topics/22-benchmarks/reading-boncz-tpch.md index 06508f5..7e6d8ed 100644 --- a/topics/22-benchmarks/reading-boncz-tpch.md +++ b/topics/22-benchmarks/reading-boncz-tpch.md @@ -11,6 +11,15 @@ slide — then hands you a reading plan. Read the paper WITH the queries open (DuckDB vendors them — see References and [reading-duckdb-tpch.md](reading-duckdb-tpch.md)). +Every clause number below is from **TPC-H Standard Specification +revision 3.0.1**; every choke-point number is from the TPCTC 2013 +paper's Table 1; every SQL line number belongs to +**duckdb/duckdb@6c0c1a68**, whose `extension/tpch/dbgen/queries/` +holds the 22 queries with the spec's validation parameters already +substituted. Rust line numbers are this topic's `experiments/`. +Where the paper, the spec and the shipped data disagree — and they +do, twice — the disagreement is the lesson, so all three are quoted. + ## The problem in one sentence "Engine A runs TPC-H 3× faster than engine B" is meaningless until @@ -20,216 +29,896 @@ be "fast" while the optimizer that Q9 needs is broken. ## The concepts, step by step -### Step 1 — what TPC-H is: one schema, one generator, 22 queries - -TPC-H is the industry-standard analytical benchmark (a fixed, -published test everyone runs so numbers are comparable): a fixed -8-table schema modeling orders and suppliers, a data generator -(**dbgen**) that produces deterministic data at a chosen **scale -factor** (SF — SF1 ≈ 1 GB, SF100 ≈ 100 GB; the biggest table, -`lineitem`, has SF × 6M rows), and 22 read-only SELECT queries plus -two refresh (insert/delete) streams. Because the generator is seeded -and spec-exact, an SF1 run in 2013 and an SF1 run today scan -byte-identical data — that determinism is the whole value. +### Step 1 — what TPC-H is: one schema, one generator, 22 queries, 2 refresh functions + +> **In:** nothing but the name. +> **Out:** the four moving parts (schema, generator, query set, refresh +> functions), the exact SF-1 row counts every later step computes with, and +> the one sentence about "SF1 = 1 GB" that you must never repeat carelessly. + +TPC-H is the industry-standard analytical benchmark — a fixed, +published test everyone runs so numbers are comparable. It has four +parts: + +1. A fixed **8-table schema** modelling parts, suppliers, customers + and orders (Clause 1.2, and the diagram in Clause 1.4). +2. **dbgen**, a data generator producing deterministic data at a + chosen **scale factor** (SF). +3. **22 read-only SELECT queries** (Clause 2.4), each with + *substitution parameters* — the constants in the WHERE clauses are + drawn from spec-defined ranges, so no engine can precompute a + literal answer. +4. **Two refresh functions**, RF1 (insert new orders and their + lineitems) and RF2 (delete the same), which Step 7 shows are half + the reason published numbers are hard to compare. + +The scale factor is not a free dial. Clause 4.1.3.1 lists the only +permitted values — 1, 10, 30, 100, 300, 1000, 3000, 10000, 30000, +100000 — and says "SF = 1; approximately 1GB as per Clause 4.2.5", +where GB is 2^30 bytes. SF 3 is not a legal TPC-H scale factor. + +Clause 4.2.5.1's Table 3 gives the SF-1 cardinalities. These are the +numbers every derivation in this guide uses: -Cost of that fixedness: everyone optimizes *for these 22 queries*, -which is exactly why a decoder ring for what each one stresses -matters. +``` + table rows @ SF1 scales with SF? + SUPPLIER 10,000 yes + PART 200,000 yes + PARTSUPP 800,000 yes + CUSTOMER 150,000 yes + ORDERS 1,500,000 yes + LINEITEM 6,001,215 yes, but not exactly (see below) + NATION 25 NO — fixed + REGION 5 NO — fixed + total 8,661,245 + — Clause 4.2.5.1, Table 3 +``` -### Step 2 — the choke point: naming what a query actually stresses +Two things in that table are load-bearing. + +**LINEITEM is not SF × 6,000,000.** Table 3's footnote 3 says the +cardinality "is not a strict multiple of SF since the number of +lineitems in an order is chosen at random with an average of four". +Check it: 6,001,215 lineitems ÷ 1,500,000 orders = **4.0008 lineitems +per order**, and Clause 4.2.3 sets the per-order count to a random +value in [1..7], whose mean is 4. Clause 4.2.5.2's Table 4 spells out +what that does at scale — SF 10 is **59,986,052** rows, not +60,012,150. If your loader asserts `rows == sf * 6001215`, it will +fail at SF 10. + +**"SF1 ≈ 1 GB" is about generated data volume, not your database.** +Table 3's footnote 2: "Typical lengths and sizes given here are +examples, not requirements, of what could result from an +implementation (sizes do not include storage/access overheads)." +The 641 MB it lists for LINEITEM is one illustrative row layout, not +a promise. A column store with dictionary-encoded flags and +delta-encoded dates will hold SF1 in a fraction of it; a row store +with per-row headers and indexes will hold it in several times it. +"DuckDB stores SF1 in X MB" is a fact about DuckDB, never a fact +about TPC-H. + +Why the fixedness is worth the cost: the generator is seeded and +spec-exact, so an SF1 run in 2013 and an SF1 run today scan the same +data — that determinism is the whole value. The price is that +everyone optimizes *for these 22 queries*, which is exactly why a +decoder ring for what each one stresses matters. + +### Step 2 — the choke point: 28 of them, six groups, three layers + +> **In:** Step 1's four parts. +> **Out:** the paper's actual Table 1 — the six group names, the numbering +> scheme, and the QOPT/QEXE/STORAGE tag on each entry — plus an arithmetic +> check that you have the whole catalog and not a paraphrase of it. A **choke point** is a named engine capability that dominates a query's runtime — the thing the query is *really* measuring, beneath -the SQL. The paper's contribution is a 28-entry catalog mapping every -TPC-H query to its choke points, in six families: +the SQL. The paper's abstract states the catalog's shape exactly: + +> "We identify **28** different such choke points, grouped into **six** +> categories: Aggregation Performance, Join Performance, Data Access +> Locality, Expression Calculation, Correlated Subqueries and Parallel +> Execution." + +Table 1 lists all 28 with a three-way tag saying *which layer of the +system* must implement the capability — **QOPT** (query optimizer), +**QEXE** (execution engine), **STORAGE** (physical layout): + +``` + CP1 Aggregation Performance + CP1.1 QEXE Ordered Aggregation + CP1.2 QOPT Interesting Orders + CP1.3 QOPT Small Group-by Keys (array lookup) ← Q1 + CP1.4 QEXE Dependent Group-By Keys (removal of) ← Q10 + CP2 Join Performance + CP2.1 QEXE Large Joins (out-of-core) + CP2.2 QEXE Sparse Foreign Key Joins (bloom filters) + CP2.3 QOPT Rich Join Order Optimization ← Q9 + CP2.4 QOPT Late Projection (column stores) + CP3 Data Access Locality + CP3.1 STORAGE Columnar Locality + CP3.2 STORAGE Physical Locality by Key (clustered index, partitioning) + CP3.3 QOPT Detecting Correlation (ZoneMap, MinMax, multi-attr histograms) + CP4 Expression Calculation + CP4.1 Raw Expression Arithmetic + CP4.1a QEXE Arithmetic Operation Performance ← Q1 + CP4.1b QEXE Overflow Handling + CP4.1c QEXE Compressed Execution + CP4.1d QEXE Interpreter Overhead (vectorization, JIT) ← Q1 + CP4.2 Complex Boolean Expressions in Joins and Selections + CP4.2a QOPT Common Subexpression Elimination + CP4.2b QOPT Join-Dependent Expression Filter Pushdown + CP4.2c QOPT Large IN Clauses (invisible join) + CP4.2d QEXE Evaluation Order in Conjunctions/Disjunctions + CP4.3 String Matching Performance + CP4.3a QOPT Rewrite LIKE(X%) into a Range Query ← Q9 can't + CP4.3b QEXE Raw String Matching Performance (SSE4.2) + CP4.3c QEXE Regular Expression Compilation (JIT/FSA) + CP5 Correlated Subqueries + CP5.1 QOPT Flattening Subqueries (into join plans) + CP5.2 QOPT Moving Predicates into a Subquery + CP5.3 QEXE Overlap between Outer- and Subquery + CP6 Parallelism and Concurrency + CP6.1 QOPT Query Plan Parallelization + CP6.2 QEXE Workload Management + CP6.3 QEXE Result Re-use + — TPCTC 2013, Table 1, verbatim names +``` + +Count them and you have a cheap check that you copied the catalog +and not a summary of it: ``` - CP1 aggregation dominated by GROUP BY machinery - CP1.1 ordered agg / CP1.2 small group-by keys (Q1!) / - CP1.4 dependent group-by (Q18) - CP2 joins order (Q5,Q7-Q9), semijoin (Q4,Q21,Q22), - large vs selective probes - CP3 locality materialized views would help (Q14/Q15), - physical column order - CP4 expressions arithmetic-heavy (Q1 again), string match - LIKE '%green%' (Q9), date logic everywhere - CP5 correlated subq Q2, Q11, Q17, Q20-Q22 - CP6 parallelism all of them, but skew hits Q9/Q18 hardest + by group: CP1 4 + CP2 4 + CP3 3 + CP4 (4 + 4 + 3) + CP5 3 + CP6 3 = 28 ✓ + by layer: QOPT 12 + QEXE 14 + STORAGE 2 = 28 ✓ ``` +Two things fall out of that arithmetic that a prose summary hides. +Almost **40% of the catalog (11 of 28) is CP4, expression +calculation** — a benchmark reputed to be about joins spends most of +its named capabilities on arithmetic, booleans and strings. And +**only two entries are STORAGE**: everything else is code you write, +not a layout you choose. + Why the framing matters: a benchmark result becomes a *diagnosis*. "Slow on Q4/Q21/Q22" doesn't mean "slow engine" — it means "no -semijoin rewrite". The choke-point method was so useful it was -reused to design LDBC SNB (topic 13) from scratch. +semijoin rewrite" (CP2.2) or "no subquery flattening" (CP5.1). The +choke-point method was so useful it was reused to design LDBC SNB +(topic 13) from scratch. -### Step 3 — aggregation, and why Q1's hash table is free +### Step 3 — CP1.3: why Q1's hash table is free, and what "four groups" costs to state + +> **In:** Step 2's catalog, and Step 1's 6,001,215 SF-1 lineitem rows. +> **Out:** the derivation of Q1's group count and its selectivity from the +> spec's population rules, checked twice against DuckDB's shipped SF-1 +> answer file — and the discovery that the spec's own stated intent for +> Q1's selectivity is not what the spec's own rules produce. **Aggregation** (GROUP BY) means partitioning rows by a key and -computing sums/counts per partition — normally via a **hash table** -(a structure mapping each distinct key to its running totals). The -cost of aggregation is usually that hash table: hashing, probing, -resizing, cache misses on millions of distinct groups. - -Q1 is the deliberate degenerate case: it groups 6M × SF rows by -`(returnflag, linestatus)` — which has only **~4–6 distinct values -total**. The hash table degenerates into a flat 6-slot array that -lives in registers, so what's left to measure is pure **expression -evaluation** (the per-row arithmetic) and fused accumulation. That -is exactly what our `q1_flat` stub implements: +computing sums and counts per partition — normally via a **hash +table** (a structure mapping each distinct key to its running +totals). The cost of aggregation is usually that hash table: +hashing, probing, resizing, cache misses across millions of distinct +groups. + +CP1.3 is the deliberate degenerate case, and the paper states it in +one sentence: + +> "CP1.3: Small Group-By Keys. **Q1 computes eight aggregates: a +> count, four sums and three averages.** Group-by keys are +> `l_returnflag`, `l_linestatus`, with **just four occurring value +> combinations.** … if all group-by expressions can be represented as +> integers in a small range, one can use an array to keep the +> aggregate totals by position, rather than keeping them in a +> hash-table." + +That is CP**1.3**, not CP1.2 — CP1.2 is *Interesting Orders*, about +reusing sort orders a clustered index already provides. And the +paper's headline query for CP1.4 (Dependent Group-By Keys) is +**Q10**, not Q18: "Q10 has a group-by on `c_custkey` and the columns +`c_comment, c_address, n_name, c_phone, c_acctbal, c_name`", which +`c_custkey` functionally determines. + +Here is the query itself, with the eight aggregates and the two group +keys where the paper says they are: + +```sql +-- duckdb extension/tpch/dbgen/queries/q01.sql, all 21 lines + 1 SELECT + 2 l_returnflag, + 3 l_linestatus, + 4 sum(l_quantity) AS sum_qty, + 5 sum(l_extendedprice) AS sum_base_price, + 6 sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, + 7 sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, + 8 avg(l_quantity) AS avg_qty, + 9 avg(l_extendedprice) AS avg_price, + 10 avg(l_discount) AS avg_disc, + 11 count(*) AS count_order + 12 FROM + 13 lineitem + 14 WHERE + 15 l_shipdate <= CAST('1998-09-02' AS date) + 16 GROUP BY + 17 l_returnflag, + 18 l_linestatus + 19 ORDER BY + 20 l_returnflag, + 21 l_linestatus; +``` + +Lines 4-11 are the eight aggregates (four sums, three averages, one +count). Line 15's constant is not arbitrary: Clause 2.4.1.3 defines +Q1's only substitution parameter as `DELTA`, "randomly selected +within [60. 120]", subtracted from 1998-12-01; DuckDB ships the +validation value DELTA = 90, and 1998-12-01 − 90 days = 1998-09-02. + +**Why exactly four groups.** `l_returnflag` takes values R, A or N +and `l_linestatus` takes O or F, so the Cartesian product is six — +but Clause 4.2.3's population rules make two of them impossible: + +``` + L_LINESTATUS = "O" if L_SHIPDATE > CURRENTDATE else "F" + L_RETURNFLAG = "R" or "A" if L_RECEIPTDATE <= CURRENTDATE else "N" + L_RECEIPTDATE = L_SHIPDATE + random[1..30] (so receipt > ship) + CURRENTDATE = 1995-06-17 (Clause 4.2.2.12) + + receiptdate <= CURRENTDATE ⇒ shipdate < CURRENTDATE ⇒ linestatus = F + ⇒ R and A can only ever pair with F; (R,O) and (A,O) cannot exist. + surviving combinations: (A,F) (R,F) (N,F) (N,O) = 4 ✓ +``` + +That is a **functional dependency between the two group keys** — CP1.4's +subject — hiding inside CP1.3's four groups. And it is checkable +without running anything, because DuckDB ships the answer: + +``` + duckdb extension/tpch/dbgen/answers/sf1/q01.csv — 4 data rows + A|F| … |1478493 + N|F| … | 38854 + N|O| … |2920374 + R|F| … |1478870 +``` + +**Q1's selectivity, derived and then checked.** Sum that last column +— the `count_order` aggregate — and divide by Step 1's lineitem +cardinality: + +``` + rows Q1 aggregates = 1,478,493 + 38,854 + 2,920,374 + 1,478,870 + = 5,916,591 + SF-1 LINEITEM = 6,001,215 (Clause 4.2.5.1) + measured selectivity = 5,916,591 / 6,001,215 = 0.98590 = 98.59% +``` + +Now derive the same figure from the population rules alone, with no +data. Clause 4.2.3 sets `L_SHIPDATE = O_ORDERDATE + random[1..121]` +and Clause 4.2.2.12 makes `O_ORDERDATE` uniform over +[STARTDATE, ENDDATE − 151 days] = [1992-01-01, 1998-08-02], which is +2406 days. Let `d` be the number of days from a row's orderdate back +from 1998-08-02 (uniform on 0..2405) and `k` its shipdate offset +(uniform on 1..121). The cutoff 1998-09-02 is 31 days after +1998-08-02, so a row is *excluded* exactly when `k > d + 31`: + +``` + d = 0 → k ∈ [32..121] → 90 of 121 offsets excluded + d = 1 → k ∈ [33..121] → 89 + … + d = 89 → k ∈ [121..121] → 1 + d ≥ 90 → none + excluded pairs = 90 + 89 + … + 1 = 90·91/2 = 4,095 + total pairs = 2406 · 121 = 291,126 + P(excluded) = 4,095 / 291,126 = 0.014066 + P(scanned) = 1 − 0.014066 = 0.98593 = 98.593% +``` + +Derived 98.593%, measured 98.590% — 0.003 percentage points apart on +a finite generated instance. The paper rounds this to "the large +amount of tuples to go through in Q1, **which selects 99% of +LINEITEM**" (§2.4). + +**And now the disagreement.** Clause 2.4.1.3's Comment says: "The +intent is to choose DELTA so that **between 95% and 97%** of the rows +in the table are scanned." Run the same derivation across the whole +legal DELTA range and the intent is never met: + +``` + DELTA=60 (cutoff 1998-10-02, 61 days out): 60·61/2 = 1,830 excluded → 99.37% scanned + DELTA=90 (cutoff 1998-09-02, 31 days out): 90·91/2 = 4,095 excluded → 98.59% scanned + DELTA=120 (cutoff 1998-08-03, 1 day out): 120·121/2 = 7,260 excluded → 97.51% scanned +``` + +The spec's population rules and the spec's stated intent for the same +query do not agree, and the shipped SF-1 answer file sides with the +rules. This is the single most useful habit this topic can give you: +when a spec, a paper and a data file all describe the same number, do +the arithmetic and find out which two are wrong. + +Our own generator does *not* reproduce the functional dependency — +it draws the two flags independently, so it has six live groups where +real TPC-H has four: ```rust -// Q1: "GROUP BY returnflag, linestatus" has ~6 groups TOTAL, so the -// hash table degenerates into a flat array — all that's left to -// measure is expression evaluation and fused accumulation. -fn q1_flat(c: &LineItemColumns) -> [Agg; 6] { - let mut g = [Agg::default(); 6]; - for i in 0..c.len { - if c.shipdate[i] > CUTOFF { continue; } - let k = group_code(c.returnflag[i], c.linestatus[i]); // 0..5 - let disc_price = c.extendedprice[i] * (1.0 - c.discount[i]); - g[k].sum_qty += c.quantity[i]; - g[k].sum_disc_price += disc_price; - g[k].sum_charge += disc_price * (1.0 + c.tax[i]); - g[k].count += 1; - } - g -} -``` - -Cost of not knowing this: a benchmark win on Q1 (CP1.2 + CP4) says -*nothing* about high-cardinality GROUP BY — that's why ClickBench -and TPC-DS exist. Our measured baseline: the row-at-a-time HashMap -oracle does SF 0.25 in 10.2 ms; `q1_flat` shows how much of that was -the map. - -### Step 4 — selectivity, and why Q6 is the "GB/s" headline query +// experiments/src/lineitem.rs — the two Q1 group keys, 44-45 + 44 t.returnflag.push(*[b'A', b'N', b'R'].get(rng.gen_range(0..3)).unwrap()); + 45 t.linestatus.push(if rng.gen_bool(0.5) { b'O' } else { b'F' }); +``` + +```rust +// experiments/src/tpch.rs — q1_oracle's hash-per-row, 25-39 (body elided at 31-35) + 25 pub fn q1_oracle(t: &LineItem) -> HashMap { + 26 let mut groups: HashMap = HashMap::new(); + 27 for i in 0..t.len() { + 28 if t.shipdate[i] <= 2450 { + 29 let g = groups.entry((t.returnflag[i], t.linestatus[i])).or_default(); + 30 let disc_price = t.extendedprice[i] * (1.0 - t.discount[i]); + // ... 31-35: four accumulations and a count ... + 36 } + 37 } + 38 groups + 39 } +``` + +Line 29 is the whole point of CP1.3: a hash of a two-byte key, +computed 6 million times, to reach one of **six** slots. Replacing it +with `g[rf_idx * 2 + ls_idx]` is the array-lookup optimization the +paper describes, and it is what the `q1_flat` stub asks you to build. +Line 28's cutoff of 2450 out of the generator's 0..=2526 shipdate +range gives our Q1 a selectivity of 2451/2527 = **97.0%**, close +enough to real Q1's 98.59% that the comparison is fair. + +Cost of not knowing this: a benchmark win on Q1 (CP1.3 + CP4.1) says +*nothing* about high-cardinality GROUP BY — that's why ClickBench and +TPC-DS exist. Our measured baseline (notes.md, M3 Pro, 2026-07-10): +the row-at-a-time HashMap oracle does SF 0.25 in 10.2 ms; `q1_flat` +shows how much of that was the map. + +### Step 4 — CP4.1: selectivity, and why Q6 is the "GB/s" headline query + +> **In:** Step 1's cardinalities and Step 3's habit of deriving a selectivity +> from the population rules. +> **Out:** Q6's ~1.9% selectivity, derived from three spec clauses and +> cross-checked against DuckDB's shipped answer to within 0.14% — plus why +> that number, not the SQL, decides whether branchy or branchless wins. **Selectivity** is the fraction of rows a filter keeps. Q6 is a -single-table scan with three range predicates that keep **~2%** of -`lineitem` — no join, no meaningful aggregation, just "how fast can -you evaluate predicates over columns". That makes it: +single-table scan with three range predicates, no join and no +GROUP BY — just "how fast can you evaluate predicates over columns": + +```sql +-- duckdb extension/tpch/dbgen/queries/q06.sql, all 10 lines + 1 SELECT + 2 sum(l_extendedprice * l_discount) AS revenue + 3 FROM + 4 lineitem + 5 WHERE + 6 l_shipdate >= CAST('1994-01-01' AS date) + 7 AND l_shipdate < CAST('1995-01-01' AS date) + 8 AND l_discount BETWEEN 0.05 + 9 AND 0.07 + 10 AND l_quantity < 24; +``` + +Clause 2.4.6.3 defines the three substitution parameters — DATE is +1 January of a year in [1993..1997], DISCOUNT is in [0.02..0.09], +QUANTITY is 24 or 25 — and Clause 2.4.6.4 gives the validation values +DuckDB shipped: 1994-01-01, 0.06, 24. The `BETWEEN` window on lines +8-9 is DISCOUNT ± 0.01. + +**Deriving 1.9%.** Three independent predicates, three spec clauses: + +``` + shipdate: O_ORDERDATE uniform over 2406 days, + random[1..121] + ⇒ the interior of the shipdate range is flat at 121/291,126 + = 1/2406 per day, and 1994 lies wholly inside it + P(one calendar year) = 365 / 2406 = 0.151704 + + discount: L_DISCOUNT is "random value [0.00 .. 0.10]" in steps of + 0.01 ⇒ 11 distinct values; the window {0.05,0.06,0.07} is 3 + P(discount in window) = 3 / 11 = 0.272727 + + quantity: L_QUANTITY is "random value [1..50]" ⇒ 50 values; + "< 24" keeps 1..23 + P(quantity < 24) = 23 / 50 = 0.460000 + + independent, so multiply: + 0.151704 × 0.272727 × 0.460000 = 0.019032 = 1.90% + rows kept at SF1 = 6,001,215 × 0.019032 = 114,215 +``` + +**Cross-checking it against the shipped answer.** DuckDB's +`answers/sf1/q06.csv` holds one number, `revenue = 123141078.2283`. +Clause 4.2.3 says `L_EXTENDEDPRICE = L_QUANTITY * P_RETAILPRICE` and +`P_RETAILPRICE = (90000 + ((P_PARTKEY/10) modulo 20001) + 100 * +(P_PARTKEY modulo 1000))/100`, whose mean over the 200,000 SF-1 parts +is (90000 + 10000 + 49950)/100 = 1499.50. So: + +``` + E[quantity | quantity < 24] = mean(1..23) = 12 + E[extendedprice | that] = 12 × 1499.50 = 17,994 + E[discount | 0.05..0.07] = 0.06 + E[revenue per qualifying row] = 17,994 × 0.06 = 1,079.64 + implied row count = 123,141,078.2283 / 1,079.64 = 114,058 + derived above = 114,215 + agreement: 0.14% +``` + +Two routes to the same number, neither of which required running a +database. That makes Q6: - the SIMD/vectorization showcase (topic 17's filter shapes), and - the source of every "our engine scans N GB/s" headline number. -At 2% selectivity a *branchy* scalar loop is competitive — the branch -predictor guesses "skip" and is right 98% of the time. Branchless -mask-multiply evaluation (`q6_branchless`) wins near 50% selectivity, -where branches mispredict constantly (topic 17's crater). Our branchy -oracle already hits 15.7 GB/s at SF 0.25 — half of memory bandwidth — -so predict what branchless adds *at this selectivity* before -implementing (maybe nothing!). +At ~2% selectivity a *branchy* scalar loop is competitive — the +branch predictor guesses "skip" and is right 98% of the time. +Branchless mask-multiply evaluation (`q6_branchless`) wins near 50% +selectivity, where branches mispredict constantly (topic 17's +crater). Our branchy oracle: + +```rust +// experiments/src/tpch.rs — q6_oracle, the branchy scalar scan, 43-56 + 43 pub fn q6_oracle(t: &LineItem) -> f64 { + 44 let mut rev = 0.0; + 45 for i in 0..t.len() { + 46 if t.shipdate[i] >= 730 + 47 && t.shipdate[i] < 1095 + 48 && t.discount[i] >= 0.05 + 49 && t.discount[i] <= 0.07 + 50 && t.quantity[i] < 24.0 + 51 { + 52 rev += t.extendedprice[i] * t.discount[i]; + 53 } + 54 } + 55 rev + 56 } +``` + +Lines 46-50 are the same three predicates with the same constants; +only the date encoding differs (days since 1992-01-01). Our +generator's shipdate is uniform over 0..=2526 rather than the real +convolution, so our Q6 selectivity is +(365/2527) × (3/11) × (23/50) = **1.81%** against real TPC-H's 1.90% — +close enough that the branch-prediction story transfers, and +different enough that our absolute row counts are ours alone. + +The topic's headline (FINDINGS.md row 22) is **Q1 at 5.2–5.7 GB/s and +Q6 at 9.0–14.4 GB/s effective**; notes.md's baseline table records the +SF-0.25 end of those ranges at 5.6 and 15.7 GB/s on an M3 Pro. Q6's +branchy oracle is already a large fraction of memory bandwidth *at +this selectivity* — so predict what branchless adds before +implementing it (topic 17's answer: at 2%, possibly nothing). + +### Step 5 — CP2.3, CP4.3a and CP6.1: why Q9 punishes optimizers -### Step 5 — join order, and why Q9 punishes optimizers +> **In:** Steps 3 and 4's single-table queries. +> **Out:** the three separate choke points Q9 stacks, each anchored to the +> line of `q09.sql` that triggers it, and the group-cardinality contrast with +> Q1 that explains why Q9's aggregation is a different problem. A **join** matches rows across tables; with N tables there are exponentially many orders to do it in, and the optimizer picks one -using **cardinality estimates** (predicted result sizes). A bad -order can materialize billions of intermediate rows where a good one -touches thousands — orders of magnitude, not percent. +using **cardinality estimates** (predicted result sizes). The paper's +CP2.3 says why that matters here: "TPC-H has queries which join up to +**eight** tables … the execution times of different join orders differ +by orders of magnitude." + +Q9 is the punisher, and the FROM clause settles the arity: + +```sql +-- duckdb extension/tpch/dbgen/queries/q09.sql, the join and its predicates, 10-27 + 10 FROM + 11 part, + 12 supplier, + 13 lineitem, + 14 partsupp, + 15 orders, + 16 nation + 17 WHERE + 18 s_suppkey = l_suppkey + 19 AND ps_suppkey = l_suppkey + 20 AND ps_partkey = l_partkey + 21 AND p_partkey = l_partkey + 22 AND o_orderkey = l_orderkey + 23 AND s_nationkey = n_nationkey + 24 AND p_name LIKE '%green%') AS profit + 25 GROUP BY + 26 nation, + 27 o_year +``` -Q9 is the punisher: a **6-way join**, plus `LIKE '%green%'` (a -substring match whose selectivity is nearly impossible to estimate), -plus per-nation **skew** (some groups far bigger than others, which -also breaks parallel load balance — CP6). Get any of the three wrong -and Q9's runtime explodes. The three queries everyone profiles, -side by side: +Six tables (11-16), six equi-join predicates (18-23) — a **6-way +join**, which is where the "6-way" in every summary of Q9 comes from. +Three choke points stack on it: + +- **CP2.3 (QOPT), join order.** Six tables joined through LINEITEM, + the 6-million-row table; an order that materializes + `part × partsupp` before filtering is orders of magnitude worse + than one that starts from the `%green%` selection on PART. +- **CP4.3a (QOPT), string matching.** The paper lists Q9 among + "Q2,9,13,14,16,20 contain expensive LIKE predicates", and says the + optimizable special case is *prefix* search, `LIKE 'xxx%'`, which + "occurs in Q14,16,20" and can be prefiltered by a range comparison. + Line 24's `'%green%'` is not a prefix, so that rewrite is + unavailable — the engine must actually scan strings (CP4.3b) and + the optimizer must estimate the selectivity of a substring match, + which it cannot do well. +- **CP6.1 (QOPT), parallelization.** Line 26-27's GROUP BY is over + `nation` × `o_year`. NATION is fixed at 25 rows (Step 1) and + O_ORDERDATE spans 1992-01-01 to 1998-08-02, so o_year takes 7 + values: **at most 25 × 7 = 175 groups**. Compare Step 3's four. + 175 groups is small enough that the array trick still applies and + large enough that per-nation size differences make partitioned + parallel aggregation imbalanced. + +The three queries everyone profiles, side by side: | query | choke points | what it really measures | |---|---|---| -| Q1 | CP1.2 + CP4 | expression evaluation + tiny-domain aggregation: ~4 groups, so the hash table is FREE and fused arithmetic dominates — our `q1_flat` stub makes this explicit | -| Q6 | CP4 + scan | pure selection: ~2% selectivity, SIMD-able predicates — DBMS "GB/s scanned" headline numbers are usually Q6 | -| Q9 | CP2 + CP4 (LIKE) + CP6 skew | 6-way join order + `%green%` string matching + per-nation skew — the query that punishes optimizers | +| Q1 | CP1.3 (small group-by keys) + CP4.1a/d (arithmetic, interpreter overhead) | expression evaluation over 98.6% of LINEITEM into **4** groups — the hash table is free, so fused arithmetic dominates; our `q1_flat` stub makes this explicit | +| Q6 | CP4.1a + CP4.2d (evaluation order in conjunctions) | pure selection at **1.9%** — SIMD-able predicates; "GB/s scanned" headlines are usually Q6 | +| Q9 | CP2.3 (join order) + CP4.3a/b (`%green%`, no prefix rewrite) + CP6.1 (175 groups, skewed) | 6-way join order + substring matching + parallel load balance — the query that punishes optimizers | A fourth family hides in CP5: **correlated subqueries** (a subquery -that re-runs per outer row unless the optimizer decorrelates it into -a join) — Q2, Q11, Q17, Q20–Q22. An engine without decorrelation -runs Q17 thousands of times slower. Different capability than -join order, same "optimizer or bust" flavor. - -### Step 6 — dbgen's dirty secret: uniform, independent data - -dbgen generates values **uniformly** (every value equally likely) -and **independently** (no correlation between columns — shipdate -doesn't predict discount). Real data is neither. Two consequences: - -- **Cardinality estimation is EASY on TPC-H** — multiply independent - selectivities and you're right. The JOB benchmark (topic 10) was - built on real IMDB data precisely because TPC-H lets naive - estimators look good. -- **Uniformity is a lie you can exploit**: an engine tuned on TPC-H - may have never seen skewed group sizes or correlated filters. - -Our dbgen-lite is uniform and independent like the real thing — -question 2 asks which correlations would break `q1_flat`. - -### Step 7 — reading published numbers: refresh streams and scale factors - -Two more hidden messages that change how you read any "TPC-H" claim: - -- **Refresh functions (RF1/RF2) are always skipped** in informal - runs — published numbers are usually just the power test's 22 - SELECTs, i.e. read-only. Official audited results require the - refresh streams; say "TPC-H-derived" for anything else (the spec - police are real, and so is the Fair Benchmarking paper — topic 0 - guide). -- **Scale factor changes the winner**: SF1 (~1 GB) fits in cache, - SF100 doesn't — engine rankings flip between them, exactly topic - 0's memory ladder. A comparison at one SF is a data point, not a - ranking. +whose result depends on the current outer row, so it re-runs per row +unless the optimizer *decorrelates* it into a join). CP5.1 +("Flattening Subqueries") is the capability; CP5.3 names +"Q2,11,15,17 and Q20" as the queries where outer and subquery overlap +so much that the shared work should be computed once. An engine +without decorrelation runs Q17 thousands of times slower. Different +capability than join order, same "optimizer or bust" flavour. + +### Step 6 — dbgen's uniformity, and the correlation the paper insists IS there + +> **In:** Steps 3-5's derivations, every one of which multiplied independent +> probabilities. +> **Out:** the precise statement of dbgen's independence — including the +> place where the paper says the opposite, which is the part most summaries +> get wrong. + +dbgen draws most column values **uniformly** (every value equally +likely) from spec-fixed ranges: `L_QUANTITY` random [1..50], +`L_DISCOUNT` random [0.00..0.10], `L_TAX` random [0.00..0.08], +`O_ORDERDATE` uniform across 2406 days (Clause 4.2.3). Steps 3 and 4 +exploited exactly that: every derivation there was a product of +independent probabilities, and each landed within a fraction of a +percent of the shipped answer. That *is* the demonstration — +**cardinality estimation is easy on TPC-H**, and the JOB benchmark +(topic 10) was built on real IMDB data precisely because TPC-H lets +naive estimators look good. + +But "dbgen has no correlation between columns" is too strong, and the +paper says so directly. CP3.3, *Detecting Correlation*, is an entire +choke point about the correlations dbgen **does** create: + +> "in case of LINEITEM the question then is which of the three date +> columns to use as key … in fact it should not matter which column is +> used, as **range-propagation between correlated attributes of the +> same table is relatively easy** … even if the LINEITEM is clustered +> on `l_receiptdate`, this will still find tight tuple position ranges +> for predicates on `l_shipdate` (and vice versa)." + +The mechanism is in Step 3's population rules: all three LINEITEM +dates are derived from the same `O_ORDERDATE` by small random offsets +(`L_SHIPDATE = O_ORDERDATE + [1..121]`, `L_COMMITDATE = O_ORDERDATE + +[30..90]`, `L_RECEIPTDATE = L_SHIPDATE + [1..30]`), so they are +tightly correlated with each other and with tuple position. That is +why zone maps and MinMax indexes work so well on TPC-H — a choke +point in their own right (CP3.3, QOPT). Step 3 also showed +`l_returnflag` and `l_linestatus` are *functionally* dependent through +those same dates. + +The accurate statement is therefore narrower and more useful: + +> dbgen's *value distributions* are uniform and its *unrelated* +> columns are independent — but its date columns are strongly +> correlated with each other, and its two Q1 group keys are +> functionally dependent. Uniformity is what flatters cardinality +> estimators; the date correlation is what flatters zone maps. + +Our dbgen-lite is uniform *and* fully independent, including the +dates and flags — a stronger simplification than the real generator's +(`lineitem.rs:39-47` draws every column from its own `rng.gen_range`). +Question 2 asks which correlations you would have to add back to +break `q1_flat`'s perfect-group-code trick. + +### Step 7 — reading a published number: the two refresh functions and the metric + +> **In:** everything above, plus a vendor's press release. +> **Out:** the exact definition of Power@Size from Clause 5.4.1.1, and the +> arithmetic showing what dropping RF1/RF2 does to it — the difference +> between "TPC-H" and "TPC-H-derived". + +Two hidden messages change how you read any "TPC-H" claim. + +**The refresh functions are part of the metric, not an extra.** +Clause 5.3.3.2 defines the power test as RF1, then the 22 queries, +then RF2 — and Clause 5.3.3.3 requires all 24 intervals to be timed. +Clause 5.4.1.1 then defines: + +``` + 3600 × SF + TPC-H Power@Size = ─────────────────────────────────────────── + geomean of the 24 timing intervals (22 + RF1 + RF2) + + TPC-H Throughput@Size = (S × 22 × 3600) / Ts × SF (Clause 5.4.2) + QphH@Size = sqrt(Power@Size × Throughput@Size) +``` + +A geometric mean over 24 terms is not a geometric mean over 22. Drop +the two refresh intervals and every remaining term's weight rises +from 1/24 to 1/22 — and the two you dropped were the write-heavy +ones, which on a column store are usually the slowest. The published +number goes up, and it is no longer Power@Size. Informal runs skip +RF1/RF2 almost universally; say "TPC-H-derived" for anything that +isn't audited (topic 0's Fair Benchmarking guide is the methodology +companion here). + +**Scale factor changes the winner.** SF1 (~1 GB of generated data) +fits in a modern server's cache hierarchy; SF100 does not. Engine +rankings flip between them, exactly topic 0's memory ladder. A +comparison at one SF is a data point, not a ranking — and per Step 1, +"SF1 ≈ 1 GB" describes generated volume, not what any engine stores. ## How to read the paper (with the concepts in hand) -TPCTC 2013, ~20 pages, one evening — but only with the queries open -(`extension/tpch/dbgen/queries/q01.sql…q22.sql` in DuckDB): - -- **§1–2** History and benchmark-design philosophy — skim; the "a - benchmark shapes a decade of engine development" argument is the - keeper. -- **§3–4 — read carefully.** The 28 choke points (Step 2's taxonomy - expanded), each with the queries that hit it and what an engine - must implement to pass. For each CP, ask: does FalkorDB have this - capability? That turns the catalog into an audit list. -- For every choke point, open the actual query text and find the - clause that triggers it — Q1's tiny GROUP BY domain (Step 3), - Q6's three range predicates (Step 4), Q9's join graph and LIKE - (Step 5) are the three to do first. -- **§5 (lessons/hidden messages)** — this is where Step 6 and - Step 7 live: uniformity, refresh-stream skipping, scale-factor - sensitivity. +TPCTC 2013, 16 pages, one evening — but only with the queries open +(`extension/tpch/dbgen/queries/q01.sql…q22.sql` in DuckDB). The paper +has **three sections**, not five, and almost all of it is §2: + +- **§1 Introduction** — skim. The keeper is the argument that a + benchmark shapes a decade of engine development, and the framing of + TPC-H as a design document rather than a scoreboard. +- **§2 TPC-H Choke Point Analysis** — the whole paper. Table 1 first + (Step 2's catalog, one page), then the six subsections in order: + §2.1 Aggregation, §2.2 Join, §2.3 Data Access Locality, + §2.4 Expression Calculation, §2.5 Correlated Subqueries, + §2.6 Parallelism and Concurrency. §2.4 is the longest — 11 of the 28 + choke points live there. +- **§3 Conclusion** — short. + +Read it in this order: + +1. Table 1, and check the two counts from Step 2 (28 by group, 28 by + layer). If your copy doesn't add up, you're reading a summary. +2. §2.1 CP1.3 with `q01.sql` and `answers/sf1/q01.csv` open — Step 3's + four groups and 98.59% are both visible in those two files. +3. §2.4 CP4.1a-d with `q06.sql` open. Note footnote 5 ("Some notes on + Q1"): Q1 has more computation per tuple than Q6, parallelizes + trivially, and is the only query where cross-system + back-of-the-envelope compute estimates are meaningful. +4. §2.2 CP2.3 and §2.4 CP4.3a with `q09.sql` open — Step 5's three + stacked choke points. +5. §2.3 CP3.3 last, because it is the one that contradicts the folk + version of dbgen (Step 6). + +For each CP, ask: does FalkorDB have this capability? That turns the +catalog into an audit list. ## Questions (answer in notes.md) 1. Map Q1/Q6/Q9 onto FalkorDB-relevant analogues: which Cypher query - shapes hit the same choke points (small-domain agg, scan+filter, - join-order + skew)? -2. Our dbgen-lite is uniform AND independent like the real dbgen. - Which columns would need correlation to break `q1_flat`'s - perfect-group-code trick? -3. Why does Q6's ~2% selectivity favor branchy evaluation while 50% - would favor branchless (topic 17's crater)? Predict the measured - crossover for `q6_branchless`. -4. Choke point CP3 (materialization): which of the 22 queries would - an incremental-view engine (topic 27 preview) answer in O(1)? -5. TPC-H says nothing about updates. What does TPC-C's NewOrder mix - test that no TPC-H query can (see reading-oltpbench-tpcc.md)? + shapes hit the same choke points (CP1.3 small-domain aggregation, + CP4.1 scan+filter arithmetic, CP2.3+CP6.1 join-order and skew)? +2. Our dbgen-lite draws `returnflag` and `linestatus` independently + (`lineitem.rs:44-45`), so it has six live groups where real TPC-H + has four. Which *other* correlations from Step 6 would you have to + add back to break `q1_flat`'s perfect-group-code trick — and which + ones would leave it untouched? +3. Step 4 derived Q6 at 1.90% for real dbgen and 1.81% for ours. + Rework the derivation for a predicate that keeps 50% of rows, and + predict the branchy/branchless crossover for `q6_branchless` + against topic 17's measured sweep. +4. Choke point CP3.1/CP3.2 (data access locality): which of the 22 + queries would an incremental-view engine (topic 27 preview) answer + in O(1), and which of the 28 choke points does that make irrelevant? +5. TPC-H says nothing about updates beyond RF1/RF2. What does TPC-C's + NewOrder mix test that no TPC-H query can (see + [reading-oltpbench-tpcc.md](reading-oltpbench-tpcc.md))? ## Done when -- [ ] You can define a choke point and name what Q1, Q6 and Q9 each stress. -- [ ] You can explain why Q6's low selectivity favours branchy evaluation while 50% does not — and check it against topic 17's measured sweep. -- [ ] You can state dbgen's dirty secret (uniform, independent columns) and what it flatters. -- [ ] You can read a published TPC-H number and say what refresh streams and scale factor do to its meaning. +Answer each before unfolding it. + +- [ ] You can define a choke point, state how many there are, and name the six groups and the three implementation layers Table 1 tags them with. + +
Answer + + A choke point is a named engine capability that dominates a query's + runtime — what the query is really measuring beneath the SQL. There are + **28**, in six groups: Aggregation Performance, Join Performance, Data + Access Locality, Expression Calculation, Correlated Subqueries, and + Parallelism and Concurrency (the abstract names the sixth "Parallel + Execution"; Table 1 calls it "Parallelism and Concurrency"). + + Table 1 tags each entry **QOPT**, **QEXE** or **STORAGE**. The counts + check out two ways: 4 + 4 + 3 + (4+4+3) + 3 + 3 = 28 by group, and + 12 QOPT + 14 QEXE + 2 STORAGE = 28 by layer. Two consequences worth + carrying: CP4 (expression calculation) is 11 of the 28, and only two + entries are about physical layout — the rest is code. + +
+ +- [ ] You can name Q1's choke point by its correct number, say how many groups it has and why, and derive its selectivity without running anything. + +
Answer + + **CP1.3, Small Group-By Keys** (QOPT) — not CP1.2, which is *Interesting + Orders*. The paper: "Q1 computes eight aggregates: a count, four sums and + three averages. Group-by keys are `l_returnflag`, `l_linestatus`, with just + four occurring value combinations." + + Four, not six, because Clause 4.2.3 makes the keys functionally dependent: + `L_RECEIPTDATE = L_SHIPDATE + [1..30]`, so `receiptdate <= CURRENTDATE` + implies `shipdate < CURRENTDATE` implies `linestatus = 'F'`. R and A + therefore only ever pair with F, killing (R,O) and (A,O). Survivors: + (A,F), (R,F), (N,F), (N,O). + + Selectivity from the rules alone: orderdate is uniform over 2406 days + ending 1998-08-02, shipdate adds [1..121], and DELTA=90 puts the cutoff 31 + days past the last orderdate. A row is excluded when `k > d + 31`, giving + 90·91/2 = 4,095 excluded pairs out of 2406 × 121 = 291,126, so 98.593% is + scanned. DuckDB's `answers/sf1/q01.csv` sums to 5,916,591 of 6,001,215 rows + = 98.590%. The paper rounds to "99%". Clause 2.4.1.3's Comment claims the + intent is 95–97%, which no legal DELTA in [60..120] achieves — the range is + 97.51% to 99.37%. + +
+ +- [ ] You can derive Q6's selectivity from three spec clauses and check it against a file DuckDB ships. + +
Answer + + Three independent predicates. Shipdate: the interior of the shipdate + distribution is flat at 1/2406 per day (orderdate uniform over 2406 days + convolved with a uniform [1..121] offset), and 1994 lies inside it, so + 365/2406 = 0.151704. Discount: `L_DISCOUNT` is random [0.00..0.10] in 0.01 + steps = 11 values, and `BETWEEN 0.05 AND 0.07` keeps 3, so 3/11 = 0.272727. + Quantity: random [1..50], `< 24` keeps 23, so 0.46. Product = 0.019032, and + 6,001,215 × 0.019032 = **114,215 rows, 1.90%**. + + The check: `answers/sf1/q06.csv` says revenue = 123,141,078.2283. Since + `L_EXTENDEDPRICE = L_QUANTITY × P_RETAILPRICE` and P_RETAILPRICE averages + 1499.50 over the SF-1 parts, a qualifying row contributes on average + 12 × 1499.50 × 0.06 = 1,079.64, implying 114,058 rows. The two routes agree + to 0.14%. + + Ours differs slightly: our shipdate is uniform over 0..=2526 rather than a + convolution, giving (365/2527) × (3/11) × 0.46 = 1.81%. + +
+ +- [ ] You can name the three choke points Q9 stacks and point at the line of `q09.sql` that triggers each. + +
Answer + + **CP2.3, Rich Join Order Optimization** (QOPT) — `q09.sql:11-16` lists six + tables and `:18-23` six equi-join predicates, all routed through the + 6-million-row LINEITEM. The paper notes TPC-H joins "up to eight tables" and + that join orders "differ by orders of magnitude". + + **CP4.3a/b, string matching** — `q09.sql:24`, `p_name LIKE '%green%'`. The + paper's optimizable case is *prefix* search (`LIKE 'xxx%'`, in Q14/16/20), + rewritable to a range comparison. `%green%` is not a prefix, so the engine + must scan strings (CP4.3b) and the optimizer must estimate a substring + match's selectivity, which it cannot do well. + + **CP6.1, Query Plan Parallelization** — `q09.sql:25-27` groups by nation × + o_year. NATION is fixed at 25 rows and orderdates span 1992–1998, so at most + 175 groups; unequal per-nation sizes make partitioned parallel aggregation + imbalanced. + +
+ +- [ ] You can state what dbgen's uniformity does and does not imply — including the correlation the paper devotes a choke point to. + +
Answer + + Uniform value distributions plus independence between *unrelated* columns + make cardinality estimation easy: multiply selectivities and you are right, + which Steps 3 and 4 demonstrated to within 0.14% twice. That is why JOB + (topic 10) exists on real IMDB data. + + But "no correlation between columns" is wrong. CP3.3, *Detecting + Correlation* (QOPT), is entirely about the correlation dbgen does create: + all three LINEITEM dates derive from the same `O_ORDERDATE` by small random + offsets, so they are correlated with each other and with tuple position. + The paper: "it should not matter which column is used, as range-propagation + between correlated attributes of the same table is relatively easy … even if + the LINEITEM is clustered on `l_receiptdate`, this will still find tight + tuple position ranges for predicates on `l_shipdate`." That correlation is + why zone maps and MinMax indexes look so good on TPC-H. And the two Q1 group + keys are functionally dependent through those same dates. + + Our dbgen-lite is *more* independent than the real thing — it draws dates + and flags from separate generators (`lineitem.rs:39-47`). + +
+ +- [ ] You can read a published TPC-H number and say exactly what skipping the refresh functions does to it. + +
Answer + + Clause 5.3.3.2 defines the power test as RF1 → the 22 queries → RF2, and + Clause 5.3.3.3 times all 24 intervals. Clause 5.4.1.1 sets + `Power@Size = 3600 × SF / geomean(those 24 intervals)`; Clause 5.4.2 defines + Throughput@Size, and QphH is their geometric mean. + + Skipping RF1/RF2 turns a 24-term geometric mean into a 22-term one, raising + every surviving term's weight from 1/24 to 1/22 — and the two dropped terms + are the write-heavy ones, typically the slowest on a column store. The + reported figure rises and is no longer Power@Size. Call it "TPC-H-derived". + + Scale factor matters as much: SF1's ~1 GB of generated data fits in cache + where SF100 does not, and rankings flip. And "SF1 ≈ 1 GB" is generated + volume — Table 3's footnote 2 says its byte sizes are "examples, not + requirements", so any on-disk figure is a fact about one engine. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the Cypher analogues of Q1/Q6/Q9. +
Answer + + Self-check — the answers belong in `notes.md`, not here. The one worth + arguing about is question 1: Q1's analogue is an aggregation over a + small label/type domain (`MATCH (n:Person) RETURN n.country, count(*)`), + Q6's is a scan with independent property range filters and no traversal, + and Q9's is a multi-hop pattern where the traversal order is the optimizer's + choice and the per-node degree distribution is skewed — which in a graph is + worse than TPC-H's, because real degree distributions are power-law where + NATION is uniform at 25. + +
+ ## References **Papers** - Boncz, Neumann, Erling — "TPC-H Analyzed: Hidden Messages and - Lessons Learned from an Influential Benchmark" (TPCTC 2013) — the - choke-point catalog; read with the queries open + Lessons Learned from an Influential Benchmark", TPCTC 2013, + LNCS 8391, pp. 61-76 (16 pages). Structure: §1 Introduction, + §2 TPC-H Choke Point Analysis (§2.1–§2.6), §3 Conclusion. Table 1 + is the 28-entry catalog; there is no §4 or §5. + [PDF](https://www.cwi.nl/~boncz/snb-challenge/chokepoints-tpctc.pdf) + +**Specification** +- TPC BenchmarkTM H Standard Specification, revision 3.0.1 + ([tpc.org](https://www.tpc.org/tpch/)). Clauses used above: + +| Clause | What | +|---|---| +| 2.4.1.1–2.4.1.4 | Q1's text, `DELTA ∈ [60..120]`, the 95–97% intent Comment, and the validation value 90 | +| 2.4.6.2–2.4.6.4 | Q6's text, substitution parameter ranges, and validation values 1994-01-01 / 0.06 / 24 | +| 4.1.3.1 | the ten legal scale factors; "SF = 1; approximately 1GB" | +| 4.2.2.12 | STARTDATE 1992-01-01, CURRENTDATE 1995-06-17, ENDDATE 1998-12-31 | +| 4.2.3 | population rules: quantity [1..50], discount [0.00..0.10], the three date offsets, RETURNFLAG/LINESTATUS derivation, `P_RETAILPRICE`, `L_EXTENDEDPRICE` | +| 4.2.5.1 Table 3 | SF-1 cardinalities; footnote 2 (sizes are examples, not requirements) | +| 4.2.5.2 Table 4 | LINEITEM cardinality per SF; footnote 3 (not a strict multiple of SF) | +| 5.3.3.2–5.3.3.3 | the power test is RF1 → 22 queries → RF2, all 24 intervals timed | +| 5.4.1.1 / 5.4.2 | Power@Size, Throughput@Size, QphH definitions | **Code** -- [duckdb](https://github.com/duckdb/duckdb) - `extension/tpch/dbgen/queries/q01.sql … q22.sql` — the 22 - queries; reference answers in `dbgen/answers/` + +| File | Lines | What | +|---|---|---| +| duckdb `extension/tpch/dbgen/queries/q01.sql` | 1-21 | Q1: eight aggregates (4-11), two group keys (17-18), DELTA=90 cutoff (15) | +| duckdb `extension/tpch/dbgen/queries/q06.sql` | 1-10 | Q6: three range predicates with the validation parameters | +| duckdb `extension/tpch/dbgen/queries/q09.sql` | 10-27 | Q9: six tables (11-16), six join predicates (18-23), `%green%` (24), 175-group aggregation (25-27) | +| duckdb `extension/tpch/dbgen/answers/sf1/q01.csv` | 1-5 | four result rows; `count_order` sums to 5,916,591 | +| duckdb `extension/tpch/dbgen/answers/sf1/q06.csv` | 2 | `revenue = 123141078.2283` — Step 4's cross-check | +| `experiments/src/lineitem.rs` | 39-47 | dbgen-lite: every column from its own `gen_range`, no correlation at all | +| `experiments/src/tpch.rs` | 25-39 | `q1_oracle` — the per-row hash into six slots that CP1.3 says to delete | +| `experiments/src/tpch.rs` | 43-56 | `q6_oracle` — the branchy scan at 1.81% selectivity | + +Pinned revisions: duckdb/duckdb@6c0c1a68 (regenerate the pin table +with `python3 tools/pin-table.py`). + +**Cross-topic** +- topic 0 `reading-fair-benchmarking.md` — the methodology companion + to Step 7's "TPC-H-derived" distinction. +- topic 10 — JOB, built because Step 6's uniformity flatters + cardinality estimators. +- topic 13 `reading-ldbc-snb.md` — the choke-point method reused to + design a benchmark from scratch. +- topic 17 — the branchy/branchless filter crater that Step 4's 1.9% + selectivity sits on the safe side of. +- topic 34 — coordinated omission, the OLTP-side measurement error + this guide's read-only queries never encounter. diff --git a/topics/22-benchmarks/reading-duckdb-tpch.md b/topics/22-benchmarks/reading-duckdb-tpch.md index 0ca2ddd..10c8b74 100644 --- a/topics/22-benchmarks/reading-duckdb-tpch.md +++ b/topics/22-benchmarks/reading-duckdb-tpch.md @@ -7,10 +7,17 @@ also a correctness test. It's also the fastest way to get real TPC-H numbers on this machine (no CLI install needed; `pip install duckdb` or the Rust crate both carry the extension). Before pointing at the code, this chapter builds the four design -ideas in order — table functions, vendored determinism, streaming +ideas in order — table functions, vendored determinism, chunked generation, and answers-as-oracle — then hands you the file anchors and the exact SQL to run. +Every C++ line number below belongs to **duckdb/duckdb@6c0c1a68** +(`python3 tools/pinned-source.py show duckdb extension/tpch/…` prints +the same gutters). Rust line numbers are this topic's `experiments/`. +Spec clause numbers are TPC-H revision 3.0.1. Where the file layout +differs from what a summary of it would say — and in two places it +does — the code wins. + ## The problem in one sentence The classic TPC-H workflow — download dbgen, fight its 1990s @@ -22,103 +29,487 @@ answers. ## The concepts, step by step -### Step 1 — the table function: a generator that pretends to be a table +### Step 1 — the table function: a generator wearing an operator's interface + +> **In:** SQL, and the idea that a query scans stored pages. +> **Out:** what `CALL dbgen(sf=1)` actually is — the bind/init/execute +> lifecycle, the seven named parameters, and the surprise that this table +> function returns no table. A **table function** is a function the query engine treats as a table: instead of scanning stored pages, the engine repeatedly asks -the function "give me the next batch of rows". Anything that can -produce rows on demand — a CSV reader, a range generator, a -benchmark data generator — plugs into the query machinery this way. -DuckDB exposes dbgen exactly so: `CALL dbgen(sf=1)` invokes a table -function that *generates* TPC-H data and feeds it straight into -table creation. The plumbing is small: `tpch_extension.cpp:17-30` -declares `DBGenFunctionData`; :49-95 is the standard bind (parse -the `sf` argument, :63) → init → execute lifecycle every DuckDB -operator follows. - -Why it matters: once the generator is an operator, it inherits the -engine's whole execution stack — parallelism, batching, pipelining — -for free. +the function "give me the next chunk of rows". Anything that can +produce rows on demand — a CSV reader, a range generator, a benchmark +data generator — plugs into the query machinery this way. DuckDB +registers three of them plus a pragma for TPC-H: + +```cpp +// duckdb extension/tpch/tpch_extension.cpp — LoadInternal, 244-267 (elided at 246-252) + 244 static void LoadInternal(ExtensionLoader &loader) { + 245 TableFunction dbgen_func("dbgen", {}, DbgenFunction, DbgenBind, DbgenInit); + // ... 246-252: named_parameters sf, overwrite, catalog, schema, suffix, children, step ... + 253 dbgen_func.call_return_type = StatementReturnType::NOTHING; + 254 dbgen_func.table_scan_progress = DbgenProgress; + 255 loader.RegisterFunction(dbgen_func); + 256 + 257 // create the TPCH pragma that allows us to run the query + 258 auto tpch_func = PragmaFunction::PragmaCall("tpch", PragmaTpchQuery, {LogicalType::BIGINT}); + 259 loader.RegisterFunction(tpch_func); + 260 + 261 // create the TPCH_QUERIES function that returns the query + 262 TableFunction tpch_query_func("tpch_queries", {}, TPCHQueryFunction, TPCHQueryBind, TPCHInit); + 263 loader.RegisterFunction(tpch_query_func); + 264 + 265 // create the TPCH_ANSWERS that returns the query result + 266 TableFunction tpch_query_answer_func("tpch_answers", {}, TPCHQueryAnswerFunction, TPCHQueryAnswerBind, TPCHInit); + 267 loader.RegisterFunction(tpch_query_answer_func); + 268 } +``` + +Line 245 is the three-callback shape every DuckDB table function has: + +- **bind** — `DbgenBind` (49-93) runs once at plan time. It parses the + named parameters (59-78), defaults catalog and schema from the + session (54-57), rejects `children` without `step` (79-81), + registers that the statement modifies the database (82-89), and + declares the output columns (90-91). +- **init** — `DbgenInit` (95-97) creates the per-execution state, here + a `DBGenGlobalState` (30-35) holding the generator behind a mutex. +- **execute** — `DbgenFunction` (99-133) is called repeatedly until it + reports finished. + +The struct bind fills in is small enough to read whole: + +```cpp +// duckdb extension/tpch/tpch_extension.cpp — DBGenFunctionData, 17-28 + 17 struct DBGenFunctionData : public TableFunctionData { + 18 DBGenFunctionData() { + 19 } + 20 + 21 double sf = 0; + 22 Identifier catalog = INVALID_CATALOG; + 23 Identifier schema = DEFAULT_SCHEMA; + 24 string suffix; + 25 bool overwrite = false; + 26 uint32_t children = 1; + 27 int step = -1; + 28 }; +``` + +`sf` is a `double` (21), so `CALL dbgen(sf=0.01)` is legal even though +Clause 4.1.3.1 admits only ten scale factors — DuckDB's small SFs are +useful and are not TPC-H. `suffix` (24) lets you generate a second +copy of the schema alongside the first; `children`/`step` (26-27) are +Step 3's explicit partitioning. + +**The surprise.** Lines 90-91 of `DbgenBind` declare a single BOOLEAN +column named `Success`, and line 253 sets +`call_return_type = StatementReturnType::NOTHING`. `dbgen` is a table +function that **returns no rows of TPC-H data at all**: it *creates +and populates the eight tables as a side effect* +(`CreateTPCHSchema` at 107) and reports only that it finished. So the +accurate sentence is "the generator is an operator, so it inherits +the engine's scheduling, progress reporting and interrupt handling" +(`DbgenProgress` at 135-148, `context.InterruptCheck()` at 1127 of +`dbgen.cpp`) — not "the generated rows flow through the query as this +function's output". They do not. + +`tpch_queries` (262) and `tpch_answers` (266) *are* ordinary +row-returning table functions, and `PRAGMA tpch(6)` is neither: line +258 registers it as a **query-rewrite pragma**, whose handler simply +returns the query's text: + +```cpp +// duckdb extension/tpch/tpch_extension.cpp — PragmaTpchQuery, 239-242 + 239 static string PragmaTpchQuery(ClientContext &context, const FunctionParameters ¶meters) { + 240 auto index = parameters.values[0].GetValue(); + 241 return tpch::DBGenWrapper::GetQuery(index); + 242 } +``` + +`PRAGMA tpch(6)` does not "run Q6 specially" — it expands to the text +of `q06.sql` and DuckDB then plans and executes that SQL like any +other. Which is why timing `PRAGMA tpch(6)` is timing Q6. ### Step 2 — vendoring the real dbgen: determinism is the product +> **In:** Step 1's registration. +> **Out:** which files are the TPC-official C, which file is a build manifest +> (not, as is often claimed, a code generator), and where the queries and +> answers physically live once compiled. + DuckDB does not reimplement the generator — it **vendors** (copies -into its own tree) the TPC-official dbgen C code (`dbgen/` — -`bm_utils.cpp`, `build.cpp`, `permute.cpp`: 1990s C, seeded, -spec-exact). The seeds and value distributions are the ones every -published TPC-H result used, so DuckDB's SF1 `lineitem` is +into its own tree) the TPC-official dbgen C code. The build manifest +names it exactly: + +```python +# duckdb extension/tpch/tpch_config.py — the whole file is 22 lines, 8-22 + 8 source_files = [ + 9 os.path.sep.join(x.split('/')) + 10 for x in [ + 11 'extension/tpch/tpch_extension.cpp', + 12 'extension/tpch/dbgen/bm_utils.cpp', + 13 'extension/tpch/dbgen/build.cpp', + 14 'extension/tpch/dbgen/dbgen.cpp', + 15 'extension/tpch/dbgen/dbgen_gunk.cpp', + 16 'extension/tpch/dbgen/permute.cpp', + 17 'extension/tpch/dbgen/rnd.cpp', + 18 'extension/tpch/dbgen/rng64.cpp', + 19 'extension/tpch/dbgen/speed_seed.cpp', + 20 'extension/tpch/dbgen/text.cpp', + 21 ] + 22 ] +``` + +Nine vendored translation units (12-20) plus the DuckDB glue (11). +`rnd.cpp`, `rng64.cpp` and `speed_seed.cpp` are the seeded random +number machinery; `permute.cpp` is dbgen's deterministic shuffle; +`text.cpp` builds the comment strings that Q9's `%green%` and Q13's +`l_comment` search. The seeds and value distributions are the ones +every published TPC-H result used, so DuckDB's SF1 `lineitem` is row-for-row the same data as everyone else's SF1 `lineitem`. -That's the non-negotiable property: a benchmark generator's output +That is the non-negotiable property: a benchmark generator's output must be *deterministic and shared*, or cross-paper comparison dies. Rewriting dbgen "cleanly" and drifting by one distribution would be worse than the ugly C. -### Step 3 — streaming chunks: the generator never touches disk +**`tpch_config.py` is a build manifest, not a code generator.** It is +22 lines and contains only two Python lists — include directories +(4-6) and the source files above. The queries and answers are baked +in by a different script entirely, and the generated file says so on +its first line: + +``` + extension/tpch/dbgen/include/tpch_constants.hpp — 189 lines + 1 /* THIS FILE WAS AUTOMATICALLY GENERATED BY generate_csv_header.py */ + 5 const int TPCH_QUERIES_COUNT = 22; + 28 const char *TPCH_QUERIES[] = { + 74 const char *TPCH_ANSWERS_SF0_01[] = { + 120 const char *TPCH_ANSWERS_SF0_1[] = { + 166 const char *TPCH_ANSWERS_SF1[] = { +``` + +The generator is `scripts/generate_csv_header.py` at the DuckDB repo +root; it encodes each `.sql` and `.csv` file as a `uint8_t` array so +the extension is a single self-contained binary with no data files to +lose. That is the same determinism argument one level up: if the +queries were read from disk at runtime, two installs could disagree. + +### Step 3 — chunked generation: 2,048 rows at a time, and never a .tbl file + +> **In:** Step 1's execute callback, Step 2's vendored C. +> **Out:** the constant that sets the chunk size, how many chunks an SF-1 +> LINEITEM takes, and the difference between "parallel because chunks are +> independent" and "parallel because you asked for it". + +Classic dbgen writes `.tbl` flat files that you then parse and load — +**materializing** (writing out in full) the entire dataset once on +disk and again in the database. DuckDB's generator instead appends +into the engine's own chunk format and flushes a chunk whenever it +fills: + +```cpp +// duckdb extension/tpch/dbgen/dbgen.cpp — append_begin_row, 184-192 + 184 static void append_begin_row(tpch_append_information &info) { + 185 D_ASSERT(info.appender || info.optimistic_collection); + 186 D_ASSERT(info.active_row == DConstants::INVALID_INDEX); + 187 if (info.row >= STANDARD_VECTOR_SIZE) { + 188 info.FlushChunk(); + 189 } + 190 info.active_row = info.row; + 191 info.active_col = 0; + 192 } +``` + +Line 187's `STANDARD_VECTOR_SIZE` is not folklore — it is +`DEFAULT_STANDARD_VECTOR_SIZE`, defined as `2048U`: + +```cpp +// duckdb src/include/duckdb/common/vector_size.hpp — the vector size, 15-21 + 15 //! The default standard vector size + 16 #define DEFAULT_STANDARD_VECTOR_SIZE 2048U + 17 + 18 //! The vector size used in the execution engine + 19 #ifndef STANDARD_VECTOR_SIZE + 20 #define STANDARD_VECTOR_SIZE DEFAULT_STANDARD_VECTOR_SIZE + 21 #endif +``` + +So the arithmetic, using Step 1's link to Clause 4.2.5.1's SF-1 +cardinality: + +``` + SF-1 LINEITEM rows = 6,001,215 (Clause 4.2.5.1) + rows per chunk = 2,048 (vector_size.hpp:16) + chunks for LINEITEM = 6,001,215 / 2,048 = 2,930.7 → 2,931 + all eight tables, SF 1 = 8,661,245 / 2,048 = 4,229.1 → ~4,230 chunks + peak intermediate on disk = 0 bytes +``` + +Nothing in there is a 641 MB `.tbl` file, no file format to version, +no parser to disagree. + +Parallelism is a separate decision, not a consequence. `GenerateNext` +dispatches to one of two modes: + +```cpp +// duckdb extension/tpch/dbgen/dbgen.cpp — GenerateNext, 1126-1143 + 1126 bool GenerateNext() override { + 1127 context.InterruptCheck(); + 1128 if (finished.load()) { + 1129 return true; + 1130 } + 1131 if (total_work == 0) { + 1132 Finish(); + 1133 return true; + 1134 } + 1135 switch (mode) { + 1136 case DBGenMode::PARALLEL: + 1137 return GenerateParallel(); + 1138 case DBGenMode::SEQUENTIAL: + 1139 return GenerateSequential(); + 1140 default: + 1141 throw InternalException("Unexpected TPC-H dbgen mode"); + 1142 } + 1143 } +``` + +and `DbgenBind` exposes dbgen's own partitioning scheme so you can +generate one slice per process: + +```cpp +// duckdb extension/tpch/tpch_extension.cpp — the children/step parameters, 73-81 + 73 } else if (kv.first == "children") { + 74 result->children = UIntegerValue::Get(kv.second); + 75 } else if (kv.first == "step") { + 76 result->step = UIntegerValue::Get(kv.second); + 77 } + 78 } + 79 if (result->children != 1 && result->step == -1) { + 80 throw InvalidInputException("Step must be defined when children are defined"); + 81 } +``` -Classic dbgen writes `.tbl` flat files that you then parse and load -— materializing (writing out in full) the entire dataset once on -disk and again in the database. DuckDB's `DbgenFunction` (:99) -instead **streams**: it produces data one vectorized chunk (~2K -rows) at a time, directly into the engine's ingest path. Because -each chunk is independent, SF100 generation parallelizes across -threads and never creates a 100 GB intermediate file — no file -format to version, no parser to disagree. +`CALL dbgen(sf=100, children=8, step=3)` generates the fourth eighth +of SF-100 — the same `-C`/`-S` flags the original dbgen has, because +the vendored C is doing the work. Line 1127's `InterruptCheck` is the +other half of "the generator is an operator": Ctrl-C works during a +20-minute SF-100 generation, and `DbgenProgress` (135-148) drives the +progress bar, because the generator lives inside the engine's task +machinery rather than beside it. This is topic 11's operator-vs-materialization lesson wearing a benchmark costume: expose work as an iterator over chunks, and -composition plus parallelism come for free. +scheduling, cancellation and progress come for free. ### Step 4 — shipping answers: every benchmark run is a correctness test +> **In:** Step 2's generated header. +> **Out:** the exact set of scale factors whose answers are usable, the +> shape of the `tpch_answers` table, and the row-count arithmetic that +> tells you the answers stop where the header stops. + Next to the 22 parameter-substituted queries -(`dbgen/queries/q01.sql…q22.sql`), DuckDB ships the **reference -answers** per scale factor (`dbgen/answers/`) — the exact result -rows a correct engine must produce (deterministic data ⇒ -deterministic answers). `tpch_config.py` embeds both into a -generated header. So `PRAGMA tpch(1)` can be *diffed*, not just +(`dbgen/queries/q01.sql…q22.sql`) DuckDB ships the **reference +answers** — the exact result rows a correct engine must produce. +Deterministic data plus fixed substitution parameters means +deterministic answers, so `PRAGMA tpch(1)` can be *diffed*, not just timed. -This closes the loop on Fair Benchmarking's pitfall 3.8 (topic 0: -"incorrect code wins" — a fast wrong answer beats every correct -system unless someone checks): the correctness oracle rides along -with the benchmark, and a speed regression and a wrongness -regression are caught by the same run. Topic 16's oracle habit, -institutionalized. - -### Step 5 — scoping your own generator: why dbgen-lite is NOT dbgen - -Our dbgen-lite (`lineitem.rs`) generates uniform, independent -values for three columns' worth of fidelity — enough for Q1/Q6 -choke-point work, and deliberately nothing more. Real dbgen adds -what those queries don't need: correlated text fields (`comment` -with pattern-planted `%green%` for Q9's LIKE), spec-exact value -distributions, refresh streams, and the shared seeds of Step 2. -Consequence: our numbers are comparable only to ourselves, and Q9 -or any optimizer study is out of scope by construction. The -principle: **scope your generator to your question**, and say out -loud which questions it cannot answer. +But only up to a point, and the point is in the code: + +```cpp +// duckdb extension/tpch/dbgen/dbgen.cpp — GetAnswer, 1451-1466 + 1451 string DBGenWrapper::GetAnswer(double sf, int query) { + 1452 if (query <= 0 || query > TPCH_QUERIES_COUNT) { + 1453 throw SyntaxException("Out of range TPC-H query number %d", query); + 1454 } + 1455 const char *answer; + 1456 if (sf == 0.01) { + 1457 answer = TPCH_ANSWERS_SF0_01[query - 1]; + 1458 } else if (sf == 0.1) { + 1459 answer = TPCH_ANSWERS_SF0_1[query - 1]; + 1460 } else if (sf == 1) { + 1461 answer = TPCH_ANSWERS_SF1[query - 1]; + 1462 } else { + 1463 throw NotImplementedException("Don't have TPC-H answers for SF %llf!", sf); + 1464 } + 1465 return answer; + 1466 } +``` -## Where each step lives in the code +Three scale factors: **0.01, 0.1 and 1** (1456-1461), and anything +else throws (1462-1464). The repository contains +`dbgen/answers/sf10/` and `dbgen/answers/sf100/` directories too, but +Step 2's generated header only carries three arrays +(`TPCH_ANSWERS_SF0_01`, `_SF0_1`, `_SF1`), so those two SFs are +present as files and absent from the binary. Verifying an SF-10 run +against shipped answers is not something this extension can do. + +The `tpch_answers` table function makes the same statement in SQL: + +```cpp +// duckdb extension/tpch/tpch_extension.cpp — TPCHQueryAnswerFunction, 209-218 + 209 static void TPCHQueryAnswerFunction(ClientContext &context, TableFunctionInput &data_p, DataChunk &output) { + 210 auto &data = data_p.global_state->Cast(); + 211 idx_t tpch_queries = 22; + 212 vector scale_factors {0.01, 0.1, 1}; + 213 idx_t total_answers = tpch_queries * scale_factors.size(); + 214 if (data.offset >= total_answers) { + 215 // finished returning values + 216 return; + 217 } +``` + +``` + SELECT count(*) FROM tpch_answers(); + = tpch_queries × scale_factors.size() (line 213) + = 22 × 3 + = 66 rows, with columns (query_nr, scale_factor, answer) (195-207) +``` -Layout of [`~/repos/duckdb/extension/tpch/`](https://github.com/duckdb/duckdb): +So the correctness loop you can actually close is: -| path | what (step) | -|---|---| -| `tpch_extension.cpp:17-30` | `DBGenFunctionData` — dbgen exposed as a **table function**: `CALL dbgen(sf=1)` (1) | -| `tpch_extension.cpp:49-95` | bind (parse `sf`, :63) → init → `DbgenFunction` (:99) streaming chunks — the generator IS an operator, so SF100 generation parallelizes and never materializes a .tbl file (1, 3) | -| `dbgen/` | the actual TPC-official dbgen C code, vendored (`bm_utils.cpp`, `build.cpp`, `permute.cpp` — 1990s C, seeded, spec-exact) (2) | -| `dbgen/queries/q01.sql…q22.sql` | the 22 queries, parameter-substituted (4) | -| `dbgen/answers/` | reference results per SF — correctness oracle, not just speed (4) | -| `tpch_config.py` | generates the header embedding queries/answers (4) | +```sql +-- run every query and diff it against the shipped answer, at SF 1 +CALL dbgen(sf=1); +SELECT query_nr, answer FROM tpch_answers() WHERE scale_factor = 1; +-- then, per query_nr, execute PRAGMA tpch(query_nr) and compare +``` + +This closes the loop on Fair Benchmarking's "incorrect code wins" +pitfall (topic 0: a fast wrong answer beats every correct system +unless someone checks): the correctness oracle rides along with the +benchmark, and a speed regression and a wrongness regression are +caught by the same run. Topic 16's oracle habit, institutionalized — +and bounded, at SF ≤ 1. + +### Step 5 — scoping your own generator: why dbgen-lite is not dbgen, in bytes + +> **In:** Steps 1-4's full-fidelity machinery. +> **Out:** the exact byte accounting behind this topic's GB/s headline, +> reproduced by hand from the spec's own sizing convention — and the list of +> questions our generator cannot answer. + +Our dbgen-lite (`lineitem.rs`) generates seven columns of uniform, +independent values — enough for Q1/Q6 choke-point work, and +deliberately nothing more: + +```rust +// experiments/src/lineitem.rs — the seven columns Q1 and Q6 touch, 8-16 + 8 pub struct LineItem { + 9 pub quantity: Vec, // 1..=50 + 10 pub extendedprice: Vec, // ~ 900..=105000 + 11 pub discount: Vec, // 0.00..=0.10 + 12 pub tax: Vec, // 0.00..=0.08 + 13 pub returnflag: Vec, // 'A' | 'N' | 'R' + 14 pub linestatus: Vec, // 'O' | 'F' + 15 pub shipdate: Vec, // days since 1992-01-01, 0..=2526 + 16 } +``` + +That layout is where this topic's headline GB/s comes from, and the +bench states its byte accounting inline rather than asking you to +trust it: + +```rust +// experiments/src/bin/bench_suite.rs — the effective-bandwidth report, 74-79 + 74 let bytes = t.len() * (8 * 4 + 2 + 4); // cols Q1 touches + 75 println!( + 76 " Q1 effective {:.1} GB/s | Q6 scans {:.1} GB/s (oracle lanes)", + 77 bytes as f64 / q1 / 1e6, + 78 (t.len() * (8 * 3 + 4)) as f64 / q6 / 1e6 + 79 ); +``` + +Work line 74 out, and note that it agrees with the TPC-H spec's own +sizing convention — the Comment under Clause 4.2.5.1's Table 3 says +"4-byte integers, 8-byte decimals, 4-byte dates": + +``` + Q1 touches (line 74): 4 decimals × 8 B = 32 quantity, extendedprice, discount, tax + + 2 chars × 1 B = 2 returnflag, linestatus + + 1 date × 4 B = 4 shipdate + ─── + 38 B per row + + Q6 touches (line 78): 3 decimals × 8 B = 24 quantity, extendedprice, discount + + 1 date × 4 B = 4 shipdate + ─── + 28 B per row +``` -Reading order: `tpch_extension.cpp` top-to-bottom (it's short), -then skim one file of the vendored C (`build.cpp`) just to see the -seeded generation, then open `queries/q06.sql` and its answer file -side by side. +Then the GB/s, using notes.md's baseline table (M3 Pro, measured +2026-07-10) at SF 0.25 = 1,500,000 rows. `q1` and `q6` are +milliseconds, so `bytes / ms / 1e6` is GB/s: + +``` + Q1: 1,500,000 × 38 B = 57,000,000 B over 10.2 ms + 57,000,000 / 10.2 / 1e6 = 5.59 GB/s (notes.md prints 5.6) + Q6: 1,500,000 × 28 B = 42,000,000 B over 2.7 ms + 42,000,000 / 2.7 / 1e6 = 15.6 GB/s (notes.md prints 15.7) +``` + +The canonical headline is FINDINGS.md row 22, from the later +2026-07-28 run: **Q1 at 5.2–5.7 GB/s and Q6 at 9.0–14.4 GB/s +effective**. Both are "effective" bandwidth — bytes the query +logically consumed divided by wall time — not DRAM traffic; at these +working-set sizes much of it is served from cache. + +For calibration on the same machine, FINDINGS.md row 17 measures a +tuned eight-accumulator sum at **26.32 GB/s** and a branchless filter +flat at **~10 GB/s** while the branchy version collapses to +**0.95 GB/s** at 50% selectivity. Q6's 1.9% selectivity (see +[reading-boncz-tpch.md](reading-boncz-tpch.md) Step 4) sits on the +safe side of that crater, which is why the branchy oracle is already +fast. + +What real dbgen adds that ours does not: correlated text fields +(`text.cpp`, whose comments carry the `%green%` Q9 searches for), +spec-exact value distributions and the functional dependency between +`returnflag` and `linestatus`, the correlated date columns of CP3.3, +the other seven tables, refresh streams RF1/RF2, and Step 2's shared +seeds. Consequence: our numbers are comparable only to ourselves, and +Q9, cardinality estimation, join order and anything requiring +Power@Size are out of scope by construction. The principle: **scope +your generator to your question**, and say out loud which questions +it cannot answer. + +## Where each step lives in the code + +Layout of [`duckdb/extension/tpch/`](https://github.com/duckdb/duckdb) +at `6c0c1a68` — 159 files, of which 110 are answer CSVs and 22 are +query `.sql` files, leaving 27 files of actual code and build glue: + +| path | lines | what (step) | +|---|---|---| +| `tpch_extension.cpp` | 17-28 | `DBGenFunctionData` — the seven bind parameters (1) | +| `tpch_extension.cpp` | 49-93 | `DbgenBind` — parse (59-78), validate `children`/`step` (79-81), declare one BOOLEAN column (90-91) (1) | +| `tpch_extension.cpp` | 95-97 | `DbgenInit` — per-execution `DBGenGlobalState` (1) | +| `tpch_extension.cpp` | 99-133 | `DbgenFunction` — create schema (107), loop `GenerateNext` (117-132) (3) | +| `tpch_extension.cpp` | 135-148 | `DbgenProgress` — why the progress bar works (3) | +| `tpch_extension.cpp` | 209-237 | `TPCHQueryAnswerFunction` — 22 × 3 = 66 answer rows (4) | +| `tpch_extension.cpp` | 239-242 | `PragmaTpchQuery` — `PRAGMA tpch(n)` is a query rewrite (1) | +| `tpch_extension.cpp` | 244-268 | `LoadInternal` — the four registrations; `NOTHING` return type at 253 (1) | +| `dbgen/dbgen.cpp` | 184-192 | `append_begin_row` — flush at `STANDARD_VECTOR_SIZE` (3) | +| `dbgen/dbgen.cpp` | 1126-1143 | `GenerateNext` — PARALLEL/SEQUENTIAL dispatch, interrupt check (3) | +| `dbgen/dbgen.cpp` | 1451-1466 | `GetAnswer` — answers exist for SF 0.01/0.1/1 only (4) | +| `src/include/duckdb/common/vector_size.hpp` | 15-21 | `2048` — the chunk size Step 3 divides by (3) | +| `dbgen/bm_utils.cpp`, `build.cpp`, `permute.cpp`, `rnd.cpp`, `rng64.cpp`, `speed_seed.cpp`, `text.cpp`, `dbgen_gunk.cpp` | — | the vendored TPC-official C, listed at `tpch_config.py:12-20` (2) | +| `dbgen/queries/q01.sql … q22.sql` | — | the 22 queries with validation parameters substituted (4) | +| `dbgen/answers/sf{0.01,0.1,1,10,100}/qNN.csv` | — | 110 answer files on disk; only the first three SFs are compiled in (4) | +| `dbgen/include/tpch_constants.hpp` | 1, 5, 28, 74, 120, 166 | the generated header: `TPCH_QUERIES_COUNT = 22`, one query array, three answer arrays (2, 4) | +| `tpch_config.py` | 4-22 | build manifest: include dirs and the ten source files (2) | + +Reading order: `tpch_extension.cpp` top to bottom (301 lines), then +`dbgen.cpp:184-192` and `1126-1143` for the chunking, then open +`queries/q06.sql` and `answers/sf1/q06.csv` side by side — the second +is one number, and [reading-boncz-tpch.md](reading-boncz-tpch.md) +Step 4 derives it. The lesson for M22: **benchmark data generators belong inside the -engine as table functions** — deterministic, parallel, no +engine as table functions** — deterministic, chunked, cancellable, no file-format drift, and answers ship next to queries so every run is also a correctness test. @@ -127,46 +518,214 @@ also a correctness test. ```sql -- python: import duckdb; con = duckdb.connect() INSTALL tpch; LOAD tpch; -CALL dbgen(sf=1); -PRAGMA tpch(1); -- Q1 -PRAGMA tpch(6); -- Q6 -PRAGMA tpch(9); -- Q9 --- .timer on / %timeit around them; compare against our --- dbgen-lite oracle numbers (bench_suite) at matched row counts +CALL dbgen(sf=1); -- creates and fills the 8 tables; returns "Success" +PRAGMA tpch(1); -- expands to q01.sql and runs it +PRAGMA tpch(6); -- Q6 +PRAGMA tpch(9); -- Q9 +SELECT * FROM tpch_answers() WHERE scale_factor = 1 AND query_nr IN (1, 6); +``` + +Then, for the comparison that matters: + +```sql +SET threads = 1; -- match our single-threaded lanes +SET disabled_optimizers = 'join_order'; -- Q9's horror version ``` -Expected shape (verify): Q6 saturates memory bandwidth (topic 0's -30 GB/s baseline), Q1 is compute-bound in expression eval + fused -aggregation, Q9 is join-order sensitive (try -`SET disabled_optimizers='join_order'` for the horror version). +Expected shape, to check rather than assume: Q6 should be the fastest +of the three per byte read and should scale close to linearly with +threads; Q1 should be compute-bound in expression evaluation and +fused aggregation over 98.6% of LINEITEM; Q9 should be the one that +moves by a large factor when the join-order optimizer is disabled. +Compute effective GB/s the way Step 5 does — Q1 reads 38 B/row and Q6 +28 B/row of the *spec's* column widths, so SF 1 is 228 MB and 168 MB +respectively — and compare against FINDINGS.md row 22's 5.2–5.7 and +9.0–14.4 GB/s. ## Questions (answer in notes.md) -1. Measure DuckDB Q1 and Q6 at SF1 on this machine; compute effective - GB/s and compare with our oracle lanes AND topic 0's streaming - baseline. Where does the gap come from (vectorization? fewer - passes? parallelism — check with `SET threads=1`)? +1. Measure DuckDB Q1 and Q6 at SF 1 on this machine, with + `SET threads = 1` and without. Compute effective GB/s using Step 5's + 38 and 28 bytes per row, and compare against our oracle lanes and + topic 17's 26.32 GB/s eight-accumulator ceiling. Where does the gap + come from — vectorization, fewer passes, or parallelism? 2. Why does shipping `answers/` matter more than shipping `queries/`? - Relate to topic 16's oracle taxonomy. -3. `DbgenFunction` streams chunks instead of writing .tbl files — - which topic-11 concept is that (operator vs materialization)? -4. Q9 with join order disabled: how much slower, and which topic-10 - lesson does the number reproduce? -5. Sketch M22's `CALL ldbc_datagen(sf=1)` equivalent for the - capstone: what determinism/answer-shipping properties must it keep? + Relate to topic 16's oracle taxonomy — and say what the SF ≤ 1 + limit in `GetAnswer` (dbgen.cpp:1462-1464) costs you at SF 10. +3. `dbgen` is registered with `call_return_type = NOTHING` + (tpch_extension.cpp:253) and generates into tables rather than into + its own output chunk. Which topic-11 concept is the chunked + `append_begin_row`/`FlushChunk` loop, and which one is it *not*? +4. Q9 with `disabled_optimizers = 'join_order'`: how much slower, and + which topic-10 lesson does the number reproduce? +5. Sketch M22's `CALL ldbc_datagen(sf=1)` equivalent for the capstone. + Which of Steps 1-4's properties must it keep — and what is the + graph analogue of shipping `answers/`, given that SNB's answers + depend on the substitution parameters? ## Done when -- [ ] You can explain what a table function is and why shipping the generator inside the engine makes determinism the product. -- [ ] You can say why shipping `answers/` matters more than shipping `queries/`. -- [ ] You can explain what streaming chunks avoids that writing `.tbl` files does not. -- [ ] You can measure DuckDB Q1 and Q6 at SF1 on this machine and compare effective GB/s against this topic's own measured lane (5.2-5.7 GB/s for Q1, 9.0-14.4 for Q6). +Answer each before unfolding it. + +- [ ] You can explain what a table function is, name the three callbacks, and say what `CALL dbgen(sf=1)` actually returns. + +
Answer + + A table function is a function the engine treats as a table: it plugs into + the scan interface and is asked for chunks. DuckDB's shape is + **bind** (`DbgenBind`, 49-93 — plan time: parse named parameters, declare + output columns), **init** (`DbgenInit`, 95-97 — per-execution state), and + **execute** (`DbgenFunction`, 99-133 — called until finished). + + `CALL dbgen(sf=1)` returns a single BOOLEAN column called `Success` + (declared at 90-91), and line 253 sets + `call_return_type = StatementReturnType::NOTHING`. The TPC-H rows are a + *side effect*: `CreateTPCHSchema` at 107 creates the eight tables and the + generator fills them. The payoff of being an operator is scheduling, + progress (`DbgenProgress`, 135-148) and cancellation + (`InterruptCheck`, dbgen.cpp:1127) — not that the data flows through the + query. + +
+ +- [ ] You can say which files are the vendored TPC code, and which file generates the compiled-in queries and answers. + +
Answer + + `tpch_config.py:12-20` lists the nine vendored translation units: + `bm_utils.cpp`, `build.cpp`, `dbgen.cpp`, `dbgen_gunk.cpp`, `permute.cpp`, + `rnd.cpp`, `rng64.cpp`, `speed_seed.cpp`, `text.cpp`. That file is a + **build manifest** — 22 lines, two Python lists — and generates nothing. + + The queries and answers are baked into + `dbgen/include/tpch_constants.hpp`, whose line 1 reads "THIS FILE WAS + AUTOMATICALLY GENERATED BY generate_csv_header.py". It holds + `TPCH_QUERIES_COUNT = 22` (5), one `TPCH_QUERIES[]` array (28), and three + answer arrays (74, 120, 166). Encoding them as byte arrays means the + extension is one self-contained binary with no data files to lose or + diverge. + +
+ +- [ ] You can state the chunk size, where it is defined, and how many chunks an SF-1 LINEITEM takes. + +
Answer + + `append_begin_row` (dbgen.cpp:184-192) flushes when `info.row >= + STANDARD_VECTOR_SIZE` (187-189), and `STANDARD_VECTOR_SIZE` is + `DEFAULT_STANDARD_VECTOR_SIZE = 2048U` in + `src/include/duckdb/common/vector_size.hpp:16`. + + 6,001,215 / 2,048 = 2,930.7, so 2,931 chunks for LINEITEM and about 4,230 + for all 8,661,245 SF-1 rows. Peak intermediate storage: zero — there is no + `.tbl` file, no file format to version and no parser to disagree. + + Parallelism is separate: `GenerateNext` (1126-1143) dispatches to + `GenerateParallel` or `GenerateSequential`, and `children`/`step` + (tpch_extension.cpp:73-81) expose dbgen's own `-C`/`-S` partitioning for + generating one slice per process. + +
+ +- [ ] You can say which scale factors DuckDB can actually verify against, and how many rows `tpch_answers()` returns. + +
Answer + + `GetAnswer` (dbgen.cpp:1451-1466) handles `sf == 0.01`, `0.1` and `1` + (1456-1461) and throws `NotImplementedException` for anything else + (1462-1464). The repo contains `answers/sf10/` and `answers/sf100/` + directories, but the generated header carries only three answer arrays, so + those two are files on disk and absent from the binary — an SF-10 run + cannot be diffed against a shipped answer. + + `tpch_answers()` returns `tpch_queries × scale_factors.size()` = + 22 × 3 = **66 rows** (tpch_extension.cpp:211-213), with columns + `query_nr`, `scale_factor`, `answer` (195-207). + +
+ +- [ ] You can reproduce this topic's effective-GB/s figure by hand from the byte accounting, and say what "effective" excludes. + +
Answer + + `bench_suite.rs:74` charges Q1 `8*4 + 2 + 4 = 38` bytes per row — four + 8-byte decimals, two 1-byte chars, one 4-byte date — which is exactly the + sizing convention in the Comment under TPC-H Clause 4.2.5.1's Table 3. + Line 78 charges Q6 `8*3 + 4 = 28` bytes. + + At SF 0.25 (1,500,000 rows) with notes.md's 2026-07-10 baseline: + 1,500,000 × 38 = 57,000,000 B over 10.2 ms = 5.59 GB/s for Q1, and + 1,500,000 × 28 = 42,000,000 B over 2.7 ms = 15.6 GB/s for Q6. FINDINGS.md + row 22's canonical figures from the later 2026-07-28 run are 5.2–5.7 and + 9.0–14.4 GB/s. + + "Effective" means bytes the query logically consumed ÷ wall time. It is + **not** DRAM traffic: at these working-set sizes much of the data is served + from cache, and it excludes the write side, the allocator, and everything + the HashMap does. Comparing it to a DRAM bandwidth figure is a category + error; comparing it to topic 17's 26.32 GB/s accumulate lane, measured the + same way on the same machine, is not. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your `CALL ldbc_datagen` sketch for M22. +
Answer + + Self-check — the answers belong in `notes.md`. The interesting half of + question 5 is that SNB's queries take substitution parameters drawn from the + generated graph, so "ship the answers" cannot mean a static CSV per query: + it has to be answers *per parameter set*, generated alongside the data and + pinned by the same seed. That is a stronger determinism requirement than + TPC-H's, and it is why the generator and the answer generator have to be the + same program. + +
+ ## References **Code** - [duckdb](https://github.com/duckdb/duckdb) `extension/tpch/` — - `tpch_extension.cpp` (the table-function plumbing), `dbgen/` (the - vendored TPC-official C code), `dbgen/queries/`, - `dbgen/answers/`, `tpch_config.py` + pinned at `6c0c1a68`. `tpch_extension.cpp` is 301 lines and reads + top to bottom in one sitting; the vendored `dbgen/` C is 1990s + TPC-official code and is worth skimming, not reading. + +| File | Lines | What | +|---|---|---| +| `extension/tpch/tpch_extension.cpp` | 17-28 | `DBGenFunctionData` — sf, catalog, schema, suffix, overwrite, children, step | +| `extension/tpch/tpch_extension.cpp` | 49-93 | `DbgenBind`; named-parameter parse at 59-78, output column at 90-91 | +| `extension/tpch/tpch_extension.cpp` | 63-64 | where `sf` is read out of the named parameters | +| `extension/tpch/tpch_extension.cpp` | 73-81 | `children`/`step`, and the check that they come as a pair | +| `extension/tpch/tpch_extension.cpp` | 95-97 | `DbgenInit` | +| `extension/tpch/tpch_extension.cpp` | 99-133 | `DbgenFunction`; `CreateTPCHSchema` at 107, generate loop at 117-132 | +| `extension/tpch/tpch_extension.cpp` | 135-148 | `DbgenProgress` | +| `extension/tpch/tpch_extension.cpp` | 172-193 | `TPCHQueryFunction` — the 22 query texts as a table | +| `extension/tpch/tpch_extension.cpp` | 209-237 | `TPCHQueryAnswerFunction` — 66 rows, three scale factors | +| `extension/tpch/tpch_extension.cpp` | 239-242 | `PragmaTpchQuery` — `PRAGMA tpch(n)` returns SQL text | +| `extension/tpch/tpch_extension.cpp` | 244-268 | `LoadInternal`; `StatementReturnType::NOTHING` at 253 | +| `extension/tpch/dbgen/dbgen.cpp` | 184-192 | `append_begin_row` — the chunk flush | +| `extension/tpch/dbgen/dbgen.cpp` | 1126-1143 | `GenerateNext` — parallel/sequential dispatch | +| `extension/tpch/dbgen/dbgen.cpp` | 1444-1449 | `GetQuery` — bounds-checked lookup into `TPCH_QUERIES` | +| `extension/tpch/dbgen/dbgen.cpp` | 1451-1466 | `GetAnswer` — SF 0.01 / 0.1 / 1 only | +| `extension/tpch/dbgen/include/tpch_constants.hpp` | 1, 5, 28, 74, 120, 166 | generated by `scripts/generate_csv_header.py` | +| `extension/tpch/tpch_config.py` | 4-22 | build manifest | +| `src/include/duckdb/common/vector_size.hpp` | 15-21 | `STANDARD_VECTOR_SIZE = 2048` | +| `experiments/src/lineitem.rs` | 8-16 | dbgen-lite's seven columns | +| `experiments/src/bin/bench_suite.rs` | 74-79 | the 38 B / 28 B per row accounting behind the GB/s headline | + +Pinned revisions: duckdb/duckdb@6c0c1a68 (regenerate the pin table +with `python3 tools/pin-table.py`). + +**Specification** +- TPC-H revision 3.0.1, Clause 4.1.3.1 (the ten legal scale factors — + DuckDB's `sf` is a `double` and accepts many more), Clause 4.2.5.1 + Table 3 and its Comment (SF-1 cardinalities; "4-byte integers, + 8-byte decimals, 4-byte dates", which Step 5's byte accounting + follows). + +**Cross-topic** +- topic 11 — operator-vs-materialization; Step 3's chunk loop. +- topic 16 — oracle taxonomy; Step 4's shipped answers. +- topic 17 — 26.32 GB/s accumulate and the branchless filter floor, + the calibration for Step 5's effective bandwidth. +- topic 0 `reading-fair-benchmarking.md` — "incorrect code wins". diff --git a/topics/22-benchmarks/reading-oltpbench-tpcc.md b/topics/22-benchmarks/reading-oltpbench-tpcc.md index 9c5aa1a..11dd388 100644 --- a/topics/22-benchmarks/reading-oltpbench-tpcc.md +++ b/topics/22-benchmarks/reading-oltpbench-tpcc.md @@ -2,208 +2,843 @@ TPC-C doesn't measure throughput — it measures how an engine behaves when the workload deliberately funnels transactions through -hot rows. This chapter builds that idea step by step — what an OLTP -benchmark even measures, where TPC-C's contention is planted, the -two spec devices that stop you from cheating around it, and why -almost nobody runs it honestly — then reads the OLTP-Bench paper -(VLDB 2013) for what a fair OLTP harness must do (rate control -above all), with the code anchors in the maintained fork, CMU's -BenchBase: one harness, ~20 benchmarks, one config format, -per-phase rate control. +hot rows. This chapter builds that idea step by step: what an OLTP +benchmark even measures, where TPC-C's contention is planted, what +the spec *actually* mandates about the transaction mix (less than +everyone says), the two devices that stop you cheating around the +skew, and why almost nobody runs it honestly — then reads the +OLTP-Bench paper for what a fair OLTP harness must do, with the code +anchors in the maintained successor, CMU's BenchBase. + +Clause numbers are **TPC-C Standard Specification revision 5.11.0**. +The paper is **Difallah, Pavlo, Curino, Cudré-Mauroux, "OLTP-Bench: +An Extensible Testbed for Benchmarking Relational Databases", PVLDB +Vol. 7 No. 4 (copyright 2013), presented at the 40th VLDB, September +2014** — cite the volume, not a year, because both years are +defensible and neither is unambiguous. Java and XML line numbers +belong to **cmu-db/benchbase@33c0047**; `OLTP-Bench` and `BenchBase` +are named separately below wherever they differ. ## The problem in one sentence Strip TPC-C's mandated think times and run 4 warehouses instead of the thousands the spec forces, and your "tpmC" number measures one -contended counter's latch, not the engine — which is exactly what -most informal "TPC-C" results do (spec-compliant, one warehouse -supports only **~12.86 tpmC**). +contended counter's row lock, not the engine — which is exactly what +most informal "TPC-C" results do, because a spec-compliant warehouse +supports only **12.86 tpmC** and the spec says so itself. ## The concepts, step by step ### Step 1 — what an OLTP benchmark measures: contention, not speed -OLTP (online transaction processing — many small concurrent -read-write transactions, opposite of TPC-H's big read-only scans) -performance is limited by **contention**: multiple transactions -needing the *same rows* at the same time, forcing the engine to -serialize them via locks or abort-and-retry (topic 8's concurrency -control). An OLTP benchmark with no contention just measures how -fast you can hash keys — YCSB territory. TPC-C's design question -is: *given* deliberately contended rows, multi-statement -transactions, and mandatory aborts, how much throughput survives? -That is a property of the concurrency-control design, not the -per-op code path. - -### Step 2 — TPC-C's anatomy: the hot counter is the benchmark - -TPC-C models order entry: a hierarchy of warehouses, districts, and -customers, with five transaction types weighted 45/43/4/4/4 — -NewOrder, Payment, OrderStatus, Delivery, StockLevel -(`TPCCConfig.java` holds the weights). Contention is BY DESIGN: - -``` - warehouse (W of them) ← every NewOrder updates its W_YTD row-ish - └─ district (10/W) ← D_NEXT_O_ID: THE hot counter, serializes - └─ orders NewOrders within a district - ~1% NewOrders touch a REMOTE warehouse ⇒ cross-shard txns exist - ~1% NewOrders ABORT by spec (rollback path must be exercised) -``` - -Every NewOrder in a district must read-increment-write that -district's `D_NEXT_O_ID` counter — so NewOrders within a district -are *forcibly serialized*, and with 10 districts per warehouse, -warehouse count directly caps parallelism. The 1% remote-warehouse -orders exist so that partitioning by warehouse can't make -cross-partition transactions disappear; the 1% mandated aborts -force the rollback path to be real code, not dead code. - -### Step 3 — NURand: skew you can't preload away - -**NURand** is TPC-C's non-uniform random function: it ORs two -uniform random numbers, which biases bits toward 1 and concentrates -selections (customer names, item ids) in a hot region — skew, like -YCSB's Zipfian, but with a spec-mandated twist: the constant `C` -that positions the hot region **must differ between load time and -run time** (the delta is constrained — e.g. 157 or 223 work, others -don't). Otherwise a vendor could pre-sort or pre-cache exactly the -rows the run will hammer. - -```rust -// NURand: TPC-C's non-uniform random — OR of two uniforms biases bits -// toward 1, concentrating hits in a hot region you can't cheat away -fn nurand(a: u64, x: u64, y: u64, c: u64, rng: &mut Rng) -> u64 { - // c MUST differ between load time and run time (TPCCUtil:94) — - // otherwise the loader could pre-sort the hot region into cache - (((rng.range(0, a) | rng.range(x, y)) + c) % (y - x + 1)) + x -} -``` - -In code: `TPCCUtil.java:94-116` — note :94's constraint on -`C_LAST_LOAD_C` vs `C_LAST_RUN_C` (157/223). The lesson generalizes: -a benchmark's data loader and its runtime driver must not share the -knowledge that lets one flatter the other. - -### Step 4 — think times: the human simulator nobody runs - -The spec simulates a human terminal operator: after each -transaction, the emulated user "keys in" the next one and "thinks" -— a capped exponential wait (`TPCCWorker.java:85-100`): - -```rust -// keying + think time: the simulated human nobody runs — capped -// exponential wait between transactions (TPCCWorker:85-100) -fn think_time(mean: f64, rng: &mut Rng) -> f64 { - (-rng.f64().ln() * mean).min(10.0 * mean) // spec caps at 10× mean -} -``` - -Consequence: with think times, one warehouse supports **~12.86 tpmC -max** — so a spec-compliant run posting millions of tpmC needs -hundreds of thousands of warehouses, i.e. terabytes of data, and -the metric secretly becomes "how much hardware can you scale to". -Everyone strips think times and runs 4 warehouses instead ⇒ they're -benchmarking the D_NEXT_O_ID latch (Step 2), not the engine. -"tpmC" without an audit is a vibe. - -| anchor | what | -|---|---| -| `TPCCWorker.java:85-100` | keying + think times: `-log(c)·mean`, capped at 10× — the spec's human simulator | -| `TPCCUtil.java:94-116` | `NURand` non-uniform randoms; note :94's constraint on `C_LAST_LOAD_C` vs `C_LAST_RUN_C` (157/223) — load-time and run-time skew must DIFFER by spec | -| `TPCCConfig.java` | the 45/43/4/4/4 weights | - -### Step 5 — rate control: what an honest harness must add - -A **closed-loop** driver (each thread waits for a response before -sending the next request) measures maximum throughput but hides -queueing — when the system stalls, the load politely stops, and the -tail latencies a real **open-loop** client (requests arrive on a -fixed schedule regardless of responses) would suffer are never -recorded — the coordinated-omission problem from reading-ycsb.md. -The OLTP-Bench paper's three contributions are exactly the fixes: - -1. **Rate control as a first-class knob** — closed-loop (max speed), - open-loop (fixed rate, honest tails), and phases that change the - rate/mix mid-run (diurnal patterns). Most homegrown harnesses - have only closed-loop. -2. **Benchmark = workload descriptor, not code fork** — transaction - weights in XML (`config/postgres/sample_tpcc_config.xml`), so - "TPC-C but 100% NewOrder" is a config edit, not a patched driver. -3. **Everything is measured the same way** — one histogram, one - sampling story across ~20 benchmarks; comparisons are apples to - apples. - -The cost of skipping this: any tail-latency claim from a -closed-loop run understates real p999, sometimes by orders of -magnitude — the higher (open-loop) number is the honest one. - -### Step 6 — TPC-C vs YCSB-A: two different contentions +> **In:** nothing but the acronym. +> **Out:** the definition of contention, and the reason a benchmark without +> it measures a different machine than the one you are buying. + +**OLTP** (online transaction processing) is many small concurrent +read-write transactions — the opposite of TPC-H's few big read-only +scans. Its performance is limited by **contention**: multiple +transactions needing the *same rows* at the same time, forcing the +engine to serialize them behind locks or to abort and retry (topic +8's concurrency control). + +An OLTP benchmark with no contention measures how fast you can hash +keys — YCSB territory (Step 7). TPC-C's design question is the +opposite: *given* deliberately contended rows, multi-statement +transactions and mandatory aborts, how much throughput survives? That +is a property of the concurrency-control design, not of the per-op +code path — which is why the same engine can win YCSB and lose +TPC-C, and why the two numbers are not comparable in either +direction. + +### Step 2 — the anatomy: where the contention is planted + +> **In:** Step 1's definition. +> **Out:** the exact table cardinalities from Clause 4.2.2, the SQL statement +> that serializes a district, and the derived fraction of transactions that +> actually cross a warehouse boundary — which is not the number you have +> heard. + +TPC-C models order entry over a fixed hierarchy. Clause 4.2.2 sets +the cardinalities **per warehouse**, and BenchBase mirrors them: + +```java +// benchbase src/main/java/com/oltpbenchmark/benchmarks/tpcc/TPCCConfig.java — the cardinalities, 32-38 + 32 public static final int configWhseCount = 1; + 33 public static final int configItemCount = 100000; // tpc-c std = 100,000 + 34 public static final int configDistPerWhse = 10; // tpc-c std = 10 + 35 public static final int configCustPerDist = 3000; // tpc-c std = 3,000 + 36 + 37 /** An invalid item id used to rollback a new order transaction. */ + 38 public static final int INVALID_ITEM_ID = -12345; +``` + +``` + per warehouse: 10 districts × 3,000 customers = 30,000 customers + 100,000 stock rows + 10 terminals (Clause 4.2.2) + fixed: 100,000 items, shared by all warehouses +``` + +Contention is by design, and it has a specific SQL shape. Every +New-Order must claim the next order id from its district's counter, +and BenchBase takes an explicit row lock to do it: + +```java +// benchbase .../tpcc/procedures/NewOrder.java — the district counter, 55-62 + 55 public final SQLStmt stmtGetDistSQL = + 56 new SQLStmt( + 57 """ + 58 SELECT D_NEXT_O_ID, D_TAX + 59 FROM %s + 60 WHERE D_W_ID = ? AND D_ID = ? FOR UPDATE + 61 """ + 62 .formatted(TPCCConstants.TABLENAME_DISTRICT)); +``` + +`FOR UPDATE` on line 60 is the whole benchmark in one clause: every +New-Order in a district takes an exclusive lock on that district's +single row, holds it for the rest of the transaction, and increments +`D_NEXT_O_ID` before committing. New-Orders within a district are +*forcibly serialized*. With 10 districts per warehouse, **warehouse +count directly caps New-Order parallelism at 10 × W**. + +Two more devices stop you from optimizing the contention away: + +```java +// benchbase .../tpcc/procedures/NewOrder.java — remote lines and the mandated rollback, 147-168 (elided) + 147 int numItems = TPCCUtil.randomNumber(5, 15, gen); + // ... 148-152: per-item arrays, allLocal = 1 ... + 153 for (int i = 0; i < numItems; i++) { + 154 itemIDs[i] = TPCCUtil.getItemID(gen); + 155 if (TPCCUtil.randomNumber(1, 100, gen) > 1) { + 156 supplierWarehouseIDs[i] = terminalWarehouseID; + 157 } else { + // ... 158-160: pick a different warehouse ... + 161 allLocal = 0; + 162 } + // ... 163-164: order quantity ... + 165 } + 166 // we need to cause 1% of the new orders to be rolled back. + 167 if (TPCCUtil.randomNumber(1, 100, gen) == 1) { + 168 itemIDs[numItems - 1] = TPCCConfig.INVALID_ITEM_ID; + 169 } +``` + +Line 147 is Clause 2.4.1.3 ("The number of items in the order +(ol_cnt) is randomly selected within [5 .. 15] (an average of 10)"). +Line 155 is Clause 2.4.1.5 item 2 ("A supplying warehouse number is +selected as the home warehouse 99% of the time and as a remote +warehouse 1% of the time … generating a random number x within +[1 .. 100]"). Lines 166-168 are Clause 2.4.1.4 ("A fixed 1% of the +New-Order transactions are chosen at random to simulate user data +entry errors and exercise the performance of rolling back update +transactions"), and Clause 5.2.5.x requires the observed rate to land +in 0.9%–1.1%. + +**Now the arithmetic everyone skips.** "1% remote" is a property of +*order lines*, not of transactions. An order has 10 lines on average, +so: + +``` + P(a New-Order is entirely local) = 0.99^10 = 0.9044 + P(a New-Order crosses a warehouse) = 1 − 0.9044 = 0.0956 = 9.56% +``` + +Clause 2.4.1.5's own Comment 1 confirms it: "With an average of 10 +items per order, approximately **90%** of all orders can be supplied +in full by stocks from the home warehouse." + +And New-Order is not even the main source of cross-warehouse traffic. +Clause 2.5.1.2 puts Payment's customer in a remote warehouse **15%** +of the time (validated to 14–16% by Clause 5.2.5.x), and Payment is +43% of the mix. So, using Step 3's weights: + +``` + distributed New-Order: 0.45 × 0.0956 = 0.0430 = 4.30% of all transactions + distributed Payment: 0.43 × 0.15 = 0.0645 = 6.45% of all transactions + ───── + total distributed: 10.75% +``` + +**Payment contributes half again as much cross-partition traffic as +New-Order does.** Any "we partition by warehouse, so only 1% of +transactions are distributed" claim is wrong by an order of +magnitude, and wrong about which transaction to look at. + +### Step 3 — the transaction mix: what the spec mandates, and what it doesn't + +> **In:** Step 2's transaction types. +> **Out:** Clause 5.2.3's table — which has a blank where everyone quotes a +> 45 — the 23-card deck that produces the folk numbers, and the file in +> BenchBase where the weights actually live. + +Everyone writes TPC-C's mix as "45/43/4/4/4". The spec does not. +Clause 5.2.3 gives **minimum percentages**, and New-Order's entry is +empty: + +``` + transaction minimum % of mix (Clause 5.2.3) + New-Order n/a ← footnote 1: "There is no minimum for the New-Order + transaction as its measured rate is the reported + throughput" + Payment 43.0 + Order-Status 4.0 + Delivery 4.0 + Stock-Level 4.0 + ───── + mandated floor 55.0 ⇒ New-Order is the residual, at most 45% +``` + +New-Order is a *residual*, not a mandate: you must run at least 43% +Payment and at least 4% of each of the other three, and New-Order is +whatever is left — at most 45%. The familiar figure comes from +Clause 5.2.4.2's alternative selection method, a shuffled deck of +cards: + +``` + a deck of 23 cards: 10 New-Order, 10 Payment, 1 Order-Status, + 1 Delivery, 1 Stock-Level (Clause 5.2.4.2) + + New-Order = 10/23 = 43.478% + Payment = 10/23 = 43.478% + each other = 1/23 = 4.348% + ─────── + 99.998% (rounding) +``` + +So a compliant run's New-Order share is between 43.478% (deck) and +45% (residual ceiling), and Step 5's tpmC derivation uses 45% +because the spec's own worked example does. + +The weights are **not** in `TPCCConfig.java` — that file is 39 lines +and holds only Step 2's cardinalities. They live in the workload +descriptor, which is OLTP-Bench's second contribution (Step 6) made +concrete: + +```xml + + 14 + 15 1 + 16 + 17 + 18 1 + 19 + 20 + 21 + 22 10000 + 23 45,43,4,4,4 + 24 + 25 +``` + +Line 15's comment is the one to internalize: in BenchBase, +**`scalefactor` is the warehouse count**, and the shipped sample is +**one** warehouse with **one** terminal, a 60-second run and a target +rate of 10,000 tps (line 22 — effectively "as fast as possible"). +Step 5 shows what those three numbers do to the metric. + +### Step 4 — NURand: skew you cannot preload away + +> **In:** Step 2's item and customer lookups. +> **Out:** the exact NURand formula from Clause 2.1.6, why its `A` constants +> are all one less than a power of two, a computed measure of how much skew +> it actually produces, and the constraint on the load-vs-run constants that +> the existing folklore states backwards. + +**NURand** is TPC-C's non-uniform random function. Clause 2.1.6: + +``` + NURand(A, x, y) = (((random(0, A) | random(x, y)) + C) % (y − x + 1)) + x + + used as: NURand(1023, 1, 3000) for C_ID (customer id) + NURand(255, 0, 999) for C_LAST (customer last name) + NURand(8191, 1, 100000) for OL_I_ID (item id) +``` + +BenchBase implements it in one line: + +```java +// benchbase src/main/java/com/oltpbenchmark/benchmarks/tpcc/TPCCUtil.java — NURand, 119-125 + 119 public static int randomNumber(int min, int max, Random r) { + 120 return (int) (r.nextDouble() * (max - min + 1) + min); + 121 } + 122 + 123 public static int nonUniformRandom(int A, int C, int min, int max, Random r) { + 124 return (((randomNumber(0, A, r) | randomNumber(min, max, r)) + C) % (max - min + 1)) + min; + 125 } +``` + +**Why bitwise OR, and how much skew it buys.** Every `A` is one less +than a power of two — 255 = 2⁸−1, 1023 = 2¹⁰−1, 8191 = 2¹³−1 — so +`random(0, A)` is a uniform bit pattern of exactly k bits. ORing it +into the draw forces those k low bits toward 1: each is 1 unless +*both* sources have a 0 there. For item ids, k = 13: + +``` + P(a given low bit is 1) uniform: 1/2 = 0.5 + NURand: 3/4 = 0.75 + + E[number of the 13 low bits set] + uniform: 13 × 0.5 = 6.5 + NURand: 13 × 0.75 = 9.75 + + P(all 13 low bits set) uniform: (1/2)^13 = 0.000122 + NURand: (3/4)^13 = 0.023763 + ratio = 195× +``` + +Item ids whose low 13 bits are mostly ones are drawn up to two orders +of magnitude more often than a uniform generator would. That is the +hot region — spread through the id space by the `+ C` offset and the +modulo, so it is not a contiguous range you can pin in cache by +sorting. + +**The constraint the folklore gets backwards.** Clause 2.1.6.1 +requires the C used at load time and the C used at run time to +*differ*, but within a window — and BenchBase quotes the clause in +the source: + +```java +// benchbase src/main/java/com/oltpbenchmark/benchmarks/tpcc/TPCCUtil.java — the four constants, 92-97 + 92 private static final int OL_I_ID_C = 7911; // in range [0, 8191] + 93 private static final int C_ID_C = 259; // in range [0, 1023] + 94 // NOTE: TPC-C 2.1.6.1 specifies that abs(C_LAST_LOAD_C - C_LAST_RUN_C) must + 95 // be within [65, 119] + 96 private static final int C_LAST_LOAD_C = 157; // in range [0, 255] + 97 private static final int C_LAST_RUN_C = 223; // in range [0, 255] +``` + +The full clause is slightly stricter than the comment: the delta must +be **in [65..119] and must not equal 96 or 112**. So 157 and 223 are +not "values that work while others don't" — they are one arbitrary +admissible pair, and what makes them admissible is the *difference*: + +``` + C-Delta = |C_LAST_RUN_C − C_LAST_LOAD_C| = |223 − 157| = 66 + 66 ∈ [65, 119] ✓ 66 ≠ 96 ✓ 66 ≠ 112 ✓ + + (0, 66), (100, 200) [delta 100], (255, 190) [delta 65] are equally legal; + (157, 253) [delta 96] and (157, 45) [delta 112] are not. +``` + +The exclusions of 96 and 112 exist because those two deltas make the +run-time hot set overlap the load-time hot set more than the spec +tolerates. The point of the rule is that the loader must not know +which rows the run will hammer — otherwise a vendor could physically +cluster exactly the hot customers, and measure a cache instead of a +database. Lines 86-91 of the same file admit BenchBase does not do +this properly: "TODO: … the constants … are supposed to be selected +ONCE and reused. We just hardcode one selection of parameters here, +but we should generate these each time." + +The lesson generalizes past TPC-C: **a benchmark's data loader and +its runtime driver must not share the knowledge that lets one flatter +the other.** + +### Step 5 — keying and think times, and where 12.86 tpmC comes from + +> **In:** Step 3's mix and Step 2's 10 terminals per warehouse. +> **Out:** the per-transaction wait table from Clause 5.2.5.7, the derivation +> of the spec's own 12.86 tpmC per warehouse, and the checked-in file that +> proves nobody runs it. + +The spec does not simulate a client library; it simulates a human at +a terminal. Clause 5.2.5.7's table gives, per transaction type, a +**minimum keying time** (typing the input before the transaction +starts) and a **minimum mean think time** (staring at the result +before starting the next one): + +``` + transaction mix % keying (s) 90th-%ile RT (s) mean think (s) + New-Order n/a 18.0 5.0 12.0 + Payment 43.0 3.0 5.0 12.0 + Order-Status 4.0 2.0 5.0 10.0 + Delivery 4.0 2.0 5.0 5.0 + Stock-Level 4.0 2.0 20.0 5.0 + — Clause 5.2.5.7 +``` + +Think time is exponential and capped, per Clause 5.2.5.4: "Tt = +−log(r) × μ", natural log, r uniform on (0,1), and "each distribution +may be truncated at 10 times its mean value". BenchBase implements +exactly that: + +```java +// benchbase src/main/java/com/oltpbenchmark/benchmarks/tpcc/TPCCWorker.java — the human simulator, 83-101 + 83 @Override + 84 protected long getPreExecutionWaitInMillis(TransactionType type) { + 85 // TPC-C 5.2.5.2: For keying times for each type of transaction. + 86 return type.getPreExecutionWait(); + 87 } + 88 + 89 @Override + 90 protected long getPostExecutionWaitInMillis(TransactionType type) { + 91 // TPC-C 5.2.5.4: For think times for each type of transaction. + 92 long mean = type.getPostExecutionWait(); + 93 + 94 float c = this.getBenchmark().rng().nextFloat(); + 95 long thinkTime = (long) (-1 * Math.log(c) * mean); + 96 if (thinkTime > 10 * mean) { + 97 thinkTime = 10 * mean; + 98 } + 99 + 100 return thinkTime; + 101 } +``` + +Line 95 is the spec's formula and 96-98 its 10× truncation. + +**Deriving 12.86 tpmC per warehouse.** Every input is in the table +above plus Clause 4.2.2's ten terminals. A terminal's average cycle +is keying + think, weighted by the mix (using 45% New-Order, as the +spec's own comment does): + +``` + New-Order 0.45 × (18 + 12) = 0.45 × 30 = 13.50 s + Payment 0.43 × ( 3 + 12) = 0.43 × 15 = 6.45 s + Order-Status 0.04 × ( 2 + 10) = 0.04 × 12 = 0.48 s + Delivery 0.04 × ( 2 + 5) = 0.04 × 7 = 0.28 s + Stock-Level 0.04 × ( 2 + 5) = 0.04 × 7 = 0.28 s + ─────── + mean cycle per terminal 20.99 s (response time ≈ 0) + + transactions per minute per terminal = 60 / 20.99 = 2.8585 + New-Orders per minute per terminal = 2.8585 × 0.45 = 1.28633 + terminals per warehouse = 10 (Clause 4.2.2) + New-Orders per minute per warehouse = 12.8633 → 12.86 tpmC +``` + +Clause 4.1.3's Comment states the same figure — "computed to be +**12.86 tpmC** per warehouse" — and adds the floor: a reported +throughput may not fall below **9 tpmC per warehouse**, which is 70% +of the maximum. That floor is what forces scale: a 1,000,000 tpmC +result needs at least 1,000,000 / 12.86 ≈ **77,760 warehouses**, and +at ~100 MB of data per warehouse that is several terabytes before the +first transaction runs. The metric secretly includes "how much +hardware and data can you bring". + +**And here is the proof that nobody runs it.** BenchBase's shipped +sample config contains the spec's keying and think times — commented +out: + +```xml + + 28 + 29 + 30 NewOrder + 31 + 32 + 33 + 34 + 35 Payment + 36 + 37 + 38 + 39 + 40 OrderStatus + 41 + 42 + 43 +``` + +18000 / 12000 / 3000 / 12000 / 2000 / 10000 milliseconds — the exact +Clause 5.2.5.7 values, present, correct, and inside XML comments (the +same pattern continues for Delivery and Stock-Level at 44-53). With +them enabled, the sample's single warehouse and single terminal would +produce about 1.29 New-Orders per minute. With them commented out and +`10000`, the same config runs a closed loop as fast as +one thread can, against one district-counter row per district. + +That is why informal TPC-C numbers are a measurement of the +`FOR UPDATE` on `NewOrder.java:60` and its surrounding lock queue, +not of the engine. Which is a perfectly good thing to measure — as +long as you say that is what you measured. + +### Step 6 — tpmC: New-Order transactions only + +> **In:** Steps 3 and 5. +> **Out:** the exact definition of the reported metric, and the two ways +> people compute it wrongly. + +Clause 5.4.2 defines the reported throughput as the **total number of +completed New-Order transactions** during the measurement interval, +divided by that interval's elapsed time in minutes. Clause 5.4.3 +names the unit tpmC and Clause 5.4.4 truncates it to zero decimals. + +Two consequences that catch people: + +- **The other four transaction types contribute nothing to the + number.** They are mandatory — the minimums in Step 3 exist so you + cannot skip the expensive ones — but Payment, Order-Status, + Delivery and Stock-Level are *load*, not *score*. A harness that + reports "total transactions per minute" is reporting roughly + 1/0.45 ≈ 2.22× the tpmC. This is the single most common error in + informal results. +- **Rolled-back New-Orders still count.** Step 2's mandated 1% + rollback (Clause 2.4.1.4) produces transactions that end in an + abort by design, and Clause 5.4.2 counts them as completed. An + implementation that quietly retried them until they succeeded would + be inflating its own denominator *and* skipping the rollback path + the clause exists to exercise. + +### Step 7 — rate control: what an honest harness adds + +> **In:** Step 5's closed-loop sample config. +> **Out:** the OLTP-Bench paper's three system models, the experiment that +> shows why rate control matters, and the pointer to where this repo measures +> the size of the lie. + +A **closed-loop** driver has each thread wait for a response before +sending the next request. It measures maximum throughput honestly and +hides queueing dishonestly: when the system stalls, the load politely +stops, so the tail latencies a real **open-loop** client would suffer +are never recorded. The OLTP-Bench paper's §3.2 names all three +options: + +> "OLTP-Bench supports three different system models for Workers to +> invoke transactions: (1) closed-loop, (2) open-loop, and (3) +> semi-open-loop. … In closed-loop testing, OLTP-Bench initializes a +> fixed number of Workers that repeatedly issue transactions with a +> random think time between each request. With the open-loop execution +> setting, the rate at which requests are invoked follows a stochastic +> process. Lastly, under a semi-open policy, the system acts +> essentially as an open-system with the difference that the Worker +> pauses for a random think time before submitting a new transaction." + +That is requirement **R2** of the eight the paper lists in §2, and +**R3** is "Fine-Grained Rate Control: the ability to control request +rates with great precision (since even small oscillations of the +throughput can make the interpretation of results difficult)". §6.1 +demonstrates it: MySQL running Wikipedia over 100k articles, starting +at 25 transactions per second and increasing by 25 tps every 10 +seconds. Delivered throughput tracks the target exactly until roughly +680 seconds in, when the DBMS saturates — and at that point the +95th-percentile latency crosses one second. A closed-loop run would +have shown the throughput plateau and nothing else. + +Do not restate the mechanism of coordinated omission here: **topic 34 +owns it and measured it**. FINDINGS.md row 34 — a closed-loop +benchmark reports **p99 = 1.0 µs** where an open-loop one reports +**90 ms** on identical work, a **90,000× lie**. That is the number to +cite whenever someone quotes a closed-loop p999, including the one +this topic's own YCSB driver produces +([reading-ycsb.md](reading-ycsb.md) Step 5). + +The paper's other durable contributions: + +1. **Benchmark = workload descriptor, not code fork.** Step 3's + `45,43,4,4,4` means "TPC-C but 100% New-Order" + is a config edit. BenchBase ships 19 such sample configs in + `config/postgres/` — auctionmark, chbenchmark, epinions, hyadapt, + noop, otmetrics, resourcestresser, seats, sibench, smallbank, + tatp, templated, tpcc, tpcds, tpch, twitter, voter, wikipedia, + ycsb. The paper itself describes **15** implemented benchmarks + (§1); the extras arrived after it. +2. **Phases.** §3.1 specifies workload parameters *per phase*, so + rate, mix and worker count can change mid-run — diurnal patterns + and spikes without restarting. +3. **One measurement path for everything.** §6.2 runs TPC-C at + saturation and reports per-transaction-type breakdowns; the + finding is that "although the NewOrder and Payment transactions + represent the majority of the transaction in the TPC-C workload, + the **Delivery** transaction has the most significant impact on + the overall system response time" — a result you cannot even see + without per-class reporting, and one that Step 3's 4% minimum for + Delivery exists to preserve. + +### Step 8 — TPC-C vs YCSB-A: two different contentions + +> **In:** Steps 2 and 7. +> **Out:** the reason a concurrency-control claim backed by YCSB is backed +> by nothing. Both are "write-heavy contended workloads", but they exercise different machinery: -- YCSB-A zipfian: skewed READS+UPDATES on independent keys — no - transaction spans keys; MVCC barely matters. -- TPC-C NewOrder: multi-statement transaction, read-modify-write on - a hot counter + ~10 item updates — THIS is what write-skew, - 2PL queues, and MVCC abort rates (topic 8) are about. +- **YCSB-A zipfian**: skewed reads and updates on *independent* keys. + No operation spans two keys, so there is no transaction to + serialize; the contention is cache-line and lock-striping + contention, and MVCC barely matters. +- **TPC-C New-Order**: one multi-statement transaction containing a + `SELECT … FOR UPDATE` on a hot counter, an update of it, and 5-15 + stock updates — with a mandated 1% abort and a ~9.6% chance of + touching a second warehouse. This is what write-skew, lock queues + and MVCC abort rates (topic 8) are actually about. -If your concurrency-control claim (isolation levels, abort rates, -lock queues) is backed only by YCSB, it's backed by nothing — -YCSB-A can't even express the anomaly the claim is about. +If your isolation-level, abort-rate or lock-queue claim is backed +only by YCSB, it is backed by nothing: YCSB-A cannot express the +anomaly the claim is about. ## How to read the paper (with the concepts in hand) -VLDB 2013, ~12 pages; §3 is the part that aged well: - -- **§1–2** Motivation + benchmark taxonomy — skim; the "everyone - hand-rolls a broken harness" complaint is still true. -- **§3 — read carefully.** Harness architecture: the worker/driver - split, rate control and phases (Step 5), the workload-descriptor - design. This is a checklist for M22's own driver. -- **§4–5** The benchmark catalog and demo experiments — skim; note - which of the ~20 benchmarks map to which contention pattern - (TPC-C = Step 2's designed hot rows, YCSB = Step 6's independent - keys, TATP/SmallBank in between). -- Then open BenchBase (the maintained fork) with Step 4's anchor - table: `TPCCWorker.java` for think times, `TPCCUtil.java` for - NURand, `TPCCConfig.java` for the weights, and the XML config to - see contribution 2 in the flesh. +PVLDB 7(4), 12 pages. §3 is the part that aged well; §6 is where the +methodology arguments are demonstrated rather than asserted. + +- **§1-2** Motivation and the ten requirements R1-R10 — read §2's + requirement list carefully and skim the rest. R2 (open/closed/ + semi-open), R3 (fine-grained rate control) and R4 (mixed and + evolving workloads) are Step 7; R10 (repeatability and + verification) is the reason the config file is the benchmark. +- **§3 — read carefully.** §3.1 Workload Manager (per-phase + parameters; the reported ceiling of "12.5k transactions per second + per Worker thread" on a main-memory DBMS is the driver-cost figure + to remember), §3.2 Workload Generation (the three system models), + §3.3 the SQL-dialect manager, §3.4 distributed clients, §3.5 + statistics collection. This is a checklist for M22's own driver. +- **§4** The benchmark catalog — skim, but note §4.1.2's CH-benCHmark + (TPC-C plus 22 analytical queries), which is the bridge between + this guide and [reading-boncz-tpch.md](reading-boncz-tpch.md). +- **§6 — read 6.1 and 6.2.** 6.1 is the rate-control experiment + (Step 7); 6.2 is multi-class reporting and the Delivery finding. +- Then open BenchBase with Step 5's anchor table: + `TPCCWorker.java:83-101` for the human simulator, + `TPCCUtil.java:92-125` for NURand and its constants, + `procedures/NewOrder.java:55-62` for the `FOR UPDATE`, and + `config/postgres/sample_tpcc_config.xml` for the weights and the + commented-out think times. ## Questions (answer in notes.md) -1. D_NEXT_O_ID: under MVCC-OCC (topic 8's stub), what abort rate do - you expect at 4 warehouses × 16 threads, closed loop? What - changes with per-district queues (topic 9)? -2. Why must load-time and run-time C_LAST constants differ - (TPCCUtil:94)? What cheat does the constraint block? -3. Design "TPC-C for graphs": what's the hot-counter analogue in a - social-network write workload (hint: supernode edge appends, - topic 13)? -4. Open vs closed loop on workload E (our scan-heavy mix): which - reports the higher p999, and why is that the honest one? -5. OLTP-Bench's phased rates: sketch the config that reproduces a - cache-warmup-then-spike incident (topic 6's eviction storm). +1. `D_NEXT_O_ID`: under MVCC-OCC (topic 8's stub), what abort rate do + you expect at 4 warehouses × 16 threads, closed loop? Step 2 caps + concurrent New-Orders at 10 × W = 40 districts — does 16 threads + even reach the ceiling? What changes with per-district queues + (topic 9)? +2. Step 4 showed `|223 − 157| = 66` is admissible under Clause + 2.1.6.1's [65..119] window. Pick a *different* legal pair, and + explain what cheat the constraint blocks — and what + `TPCCUtil.java:86-91`'s TODO means for BenchBase's compliance. +3. Design "TPC-C for graphs": what is the `D_NEXT_O_ID` analogue in a + social-network write workload (hint: supernode edge appends, topic + 13)? What plays the role of the 1% remote warehouse, and what + fraction of transactions would actually cross a partition? +4. Step 2 derived 10.75% of TPC-C transactions as cross-warehouse, + with Payment contributing more than New-Order. Redo that + derivation for a partitioning scheme that shards by *district* + instead of warehouse. Does it get better or worse? +5. OLTP-Bench's phased rates (§3.1): sketch the config that + reproduces a cache-warmup-then-spike incident (topic 6's eviction + storm), and say which of R1-R4 each phase exercises. ## Done when -- [ ] You can explain that an OLTP benchmark measures contention, not speed, and identify TPC-C's hot counter. -- [ ] You can explain NURand and why the skew cannot be preloaded away. -- [ ] You can say what think times are for and what removing them changes. -- [ ] You can state what an honest harness must add (rate control) and why — connect it to topic 34's coordinated-omission lane. -- [ ] You can contrast TPC-C's contention with YCSB-A's; they are not the same shape. +Answer each before unfolding it. + +- [ ] You can explain that an OLTP benchmark measures contention, identify TPC-C's hot row, and name the SQL clause that serializes it. + +
Answer + + Contention is multiple transactions needing the same rows at once, forcing + the engine to serialize behind locks or abort and retry. A benchmark without + it measures the per-op code path, not the concurrency-control design. + + TPC-C's hot row is the district's `D_NEXT_O_ID` counter — one row per + district, 10 districts per warehouse (Clause 4.2.2). BenchBase serializes it + explicitly: `NewOrder.java:60` ends `WHERE D_W_ID = ? AND D_ID = ? FOR + UPDATE`, an exclusive row lock held for the rest of the transaction. So + New-Order parallelism is capped at 10 × W, and warehouse count *is* the + parallelism dial. + +
+ +- [ ] You can state what the spec actually mandates about the transaction mix, and where the 45/43/4/4/4 figures come from. + +
Answer + + Clause 5.2.3 gives **minimums**, and New-Order's cell is "n/a" — footnote 1: + "There is no minimum for the New-Order transaction as its measured rate is + the reported throughput". The mandated floors are Payment 43.0%, + Order-Status 4.0%, Delivery 4.0%, Stock-Level 4.0% — 55% total, leaving + New-Order as a residual of at most 45%. + + The familiar numbers come from Clause 5.2.4.2's alternative: a shuffled deck + of **23 cards** — 10 New-Order, 10 Payment, 1 each of the rest — giving + 10/23 = 43.478% for the first two and 1/23 = 4.348% for the others. A + compliant New-Order share is therefore between 43.478% and 45%. + + In BenchBase the weights are in `config/postgres/sample_tpcc_config.xml:23`, + not in `TPCCConfig.java`, which is 39 lines of table cardinalities. + +
+ +- [ ] You can explain NURand, quantify the skew it produces, and state the constraint on its load-time and run-time constants correctly. + +
Answer + + Clause 2.1.6: `NURand(A, x, y) = (((random(0,A) | random(x,y)) + C) % + (y−x+1)) + x`, with A = 255 for C_LAST, 1023 for C_ID and 8191 for OL_I_ID. + Every A is 2^k − 1, so `random(0,A)` is a uniform k-bit pattern and the OR + forces the low k bits toward 1: each is 1 with probability 3/4 instead of + 1/2. For items (k=13), all-13-bits-set goes from (1/2)^13 = 0.000122 to + (3/4)^13 = 0.023763 — **195× more likely**. `+ C` and the modulo scatter that + hot region so it is not a contiguous range you can pin in cache. + + Clause 2.1.6.1 requires `|C_load − C_run|` to be **in [65..119], excluding 96 + and 112**. BenchBase's 157 and 223 (`TPCCUtil.java:96-97`) satisfy it because + their difference is 66 — the values are arbitrary, the *delta* is what the + spec constrains. The rule stops the loader from physically clustering the + rows the run will hammer. `TPCCUtil.java:86-91` admits the constants should + be re-drawn per run and are hardcoded instead. + +
+ +- [ ] You can derive 12.86 tpmC per warehouse from the spec's own tables, and say what removing think times changes. + +
Answer + + Inputs: Clause 5.2.5.7's keying and mean-think times, Step 3's 45/43/4/4/4 + mix, and Clause 4.2.2's 10 terminals per warehouse. The mean terminal cycle + is 0.45×(18+12) + 0.43×(3+12) + 0.04×(2+10) + 0.04×(2+5) + 0.04×(2+5) = + 13.50 + 6.45 + 0.48 + 0.28 + 0.28 = **20.99 s**. So 60/20.99 = 2.8585 + transactions per minute per terminal, × 0.45 = 1.28633 New-Orders, × 10 + terminals = **12.86 tpmC per warehouse** — the figure Clause 4.1.3's Comment + states. Clause 4.1.3 also sets a floor of 9 tpmC/warehouse (70% of maximum), + so a 1,000,000 tpmC result needs ≥ 77,760 warehouses. + + Removing think times removes the only thing bounding request rate, so the + run becomes a closed loop at full speed against 10 × W district rows. You + are then measuring the `FOR UPDATE` lock queue. `sample_tpcc_config.xml:31-52` + makes this visible: the spec's 18000/12000/3000/12000/2000/10000/2000/5000/ + 2000/5000 ms values are present and commented out. + +
+ +- [ ] You can state exactly what tpmC counts, and name the two common ways of computing it wrongly. + +
Answer + + Clause 5.4.2: the reported throughput is the **number of completed New-Order + transactions** in the measurement interval divided by its length in minutes. + Clause 5.4.3 names it tpmC; 5.4.4 truncates to zero decimals. + + Wrong way one: counting all five transaction types. That inflates the figure + by about 1/0.45 = 2.22×. The other four are mandatory load, not score. + Wrong way two: excluding the 1% of New-Orders that roll back by design + (Clause 2.4.1.4). Clause 5.4.2 counts them as completed, and retrying them + until they commit both inflates the count and skips the rollback path the + clause exists to exercise. + +
+ +- [ ] You can name the three system models OLTP-Bench supports, and cite this repo's measurement of what choosing wrongly costs. + +
Answer + + §3.2: **closed-loop** (a fixed number of Workers, each issuing the next + transaction after the previous reply, with a random think time), + **open-loop** (arrivals follow a stochastic process regardless of replies), + and **semi-open-loop** (open arrivals, but the Worker pauses for a think + time before submitting). Requirement R2 is that the user gets to choose; R3 + is fine-grained rate control, demonstrated in §6.1, where a Wikipedia + workload ramped 25 tps every 10 s tracks its target until the DBMS saturates + at ~680 s and 95th-percentile latency crosses one second. + + The cost of choosing wrongly is measured in topic 34, not restated here: + FINDINGS.md row 34 reports **p99 = 1.0 µs closed-loop against 90 ms + open-loop on identical work — a 90,000× lie**. + +
+ +- [ ] You can contrast TPC-C's contention with YCSB-A's, and say what each cannot measure. + +
Answer + + YCSB-A is skewed reads and updates on independent keys: no operation spans + two keys, so there is nothing to serialize and no anomaly to prevent. It + measures the per-op path, lock striping and cache behaviour under skew. + + TPC-C New-Order is a multi-statement transaction: `SELECT … FOR UPDATE` on + the district counter, an increment, and 5-15 stock updates, with a mandated + 1% abort and (Step 2) a 9.56% chance of touching a second warehouse — plus + Payment's 15% remote customers, which together make 10.75% of all + transactions distributed. It measures isolation, abort rates and lock + queues. + + So: an isolation-level or abort-rate claim backed by YCSB is backed by + nothing, and a "our per-op path is fast" claim backed by TPC-C is buried + under lock waiting. + +
+ - [ ] You wrote answers to all five questions in notes.md, including your design for a graph analogue of the hot counter. +
Answer + + Self-check — the answers belong in `notes.md`. The one worth arguing about + is question 3: the honest graph analogue of `D_NEXT_O_ID` is not "a hot + node" but "a hot *counter on* a node" — an append to a supernode's adjacency + list plus an update of its degree, which serializes every writer touching + that node exactly the way a district serializes New-Orders. The remote- + warehouse analogue is an edge whose endpoints live in different partitions, + and its rate is set by the partitioner's edge-cut, which for a power-law + graph is far worse than TPC-C's 1%. + +
+ ## References **Papers** -- Difallah, Pavlo, Curino, Cudré-Mauroux — "OLTP-Bench: An - Extensible Testbed for Benchmarking Relational Databases" (VLDB - 2013) — §3 (harness architecture, rate control) is the part that - aged well +- Difallah, Pavlo, Curino, Cudré-Mauroux — "OLTP-Bench: An Extensible + Testbed for Benchmarking Relational Databases", **PVLDB Vol. 7, + No. 4** (copyright 2013), presented at the 40th International + Conference on Very Large Data Bases, September 2014, Hangzhou. + [PDF](https://www.vldb.org/pvldb/vol7/p277-difallah.pdf). Sections + used above: §1 (15 implemented benchmarks), §2 (requirements + R1-R10), §3.1 (per-phase parameters; 12.5k txn/s per Worker thread), + §3.2 (the three system models), §4.1.2 (CH-benCHmark), §6.1 (rate + control), §6.2 (multi-class reporting; the Delivery finding). + **OLTP-Bench** is the artifact the paper describes; **BenchBase** + (cmu-db) is its maintained successor and the code anchored below. + +**Specification** +- TPC BenchmarkTM C Standard Specification, revision 5.11.0 + ([tpc.org](https://www.tpc.org/tpcc/)). Clauses used above: + +| Clause | What | +|---|---| +| 2.1.6 | the NURand formula and its A constants (255, 1023, 8191) | +| 2.1.6.1 | C-Delta must be in [65..119] and not 96 or 112 | +| 2.4.1.3 | `ol_cnt` random in [5..15], average 10 | +| 2.4.1.4 | a fixed 1% of New-Orders roll back by design | +| 2.4.1.5 | 1% of order *lines* are remote; Comment 1's "approximately 90% of all orders" are fully local | +| 2.5.1.2 | Payment's customer is remote 15% of the time; 60% selected by last name | +| 4.1.3 | Comment: maximum "computed to be 12.86 tpmC per warehouse"; floor of 9 | +| 4.2.2 | 10 terminals per warehouse; 10 districts, 30,000 customers, 100,000 stock per warehouse; 100,000 items fixed | +| 5.2.3 | mix minimums — New-Order "n/a", Payment 43.0, others 4.0 | +| 5.2.4.2 | the 23-card deck: 10 / 10 / 1 / 1 / 1 | +| 5.2.5.4 | think time `Tt = −log(r) × μ`, truncated at 10 × μ | +| 5.2.5.7 | the keying / response-time / think-time table | +| 5.4.2–5.4.4 | tpmC = completed New-Order transactions per minute, truncated | **Code** -- [benchbase](https://github.com/cmu-db/benchbase) — the maintained - fork; `src/main/java/com/oltpbenchmark/benchmarks/tpcc/` - (`TPCCWorker.java`, `TPCCUtil.java`, `TPCCConfig.java`) and - `config/postgres/sample_tpcc_config.xml` + +| File | Lines | What | +|---|---|---| +| benchbase `.../tpcc/TPCCConfig.java` | 32-38 | the per-warehouse cardinalities; `INVALID_ITEM_ID` — **not** the weights | +| benchbase `.../tpcc/TPCCUtil.java` | 86-91 | the TODO admitting the NURand constants are hardcoded | +| benchbase `.../tpcc/TPCCUtil.java` | 92-97 | `OL_I_ID_C`, `C_ID_C`, `C_LAST_LOAD_C` 157, `C_LAST_RUN_C` 223, with Clause 2.1.6.1 quoted at 94-95 | +| benchbase `.../tpcc/TPCCUtil.java` | 99-117 | `getItemID`, `getCustomerID`, load-vs-run last names | +| benchbase `.../tpcc/TPCCUtil.java` | 119-125 | `randomNumber` and `nonUniformRandom` | +| benchbase `.../tpcc/TPCCWorker.java` | 83-101 | keying wait, and think time with the 10× cap | +| benchbase `.../tpcc/procedures/NewOrder.java` | 55-62 | `SELECT D_NEXT_O_ID, D_TAX … FOR UPDATE` — the serialization point | +| benchbase `.../tpcc/procedures/NewOrder.java` | 73-81 | `UPDATE … SET D_NEXT_O_ID = D_NEXT_O_ID + 1` | +| benchbase `.../tpcc/procedures/NewOrder.java` | 147-168 | 5-15 items, the 1% remote branch, the 1% forced rollback | +| benchbase `config/postgres/sample_tpcc_config.xml` | 11 | `TRANSACTION_SERIALIZABLE` | +| benchbase `config/postgres/sample_tpcc_config.xml` | 14-25 | scalefactor = warehouses, 1 terminal, 60 s, rate 10000, weights 45,43,4,4,4 | +| benchbase `config/postgres/sample_tpcc_config.xml` | 28-53 | the spec's keying and think times, commented out | +| benchbase `config/postgres/` | — | 19 sample configs, one per bundled benchmark | + +Pinned revision: cmu-db/benchbase@33c0047 (regenerate the pin table +with `python3 tools/pin-table.py`). + +**Cross-topic** +- topic 34 — coordinated omission, measured: closed-loop p99 = 1.0 µs + against open-loop 90 ms, a 90,000× lie (FINDINGS.md row 34). Cite + it rather than re-deriving the mechanism. +- topic 8 — MVCC and abort rates, which Step 2's hot counter is the + canonical workload for. +- topic 9 — contended counters and per-district queueing. +- [reading-ycsb.md](reading-ycsb.md) — Step 8's other contention. +- [reading-boncz-tpch.md](reading-boncz-tpch.md) — the analytical + half, and CH-benCHmark's other end. diff --git a/topics/22-benchmarks/reading-ycsb.md b/topics/22-benchmarks/reading-ycsb.md index f96090f..9c1d34a 100644 --- a/topics/22-benchmarks/reading-ycsb.md +++ b/topics/22-benchmarks/reading-ycsb.md @@ -1,213 +1,967 @@ # YCSB: six mixes, five distributions, one Zipfian generator -Cooper et al.'s SoCC 2010 paper standardized KV benchmarking by -factoring a workload into an operation mix times a key -distribution — and its θ=0.99 Zipfian generator is the skew behind -nearly every KV paper since (our `zipf.rs` stub reimplements it -from the go-ycsb port). Before pointing you at the paper and the -generator source, this chapter builds the ideas one at a time: the -factoring, Zipf's law, how you actually *sample* from it in O(1), -why the hot keys must be scattered, and the trap (coordinated -omission) to know before citing any YCSB number. +YCSB is the most-cited and most-misquoted benchmark in key-value +storage. This chapter takes it apart in the order that matters: what +it was designed to measure (and the two tiers it never covers), what +the six workloads *actually* are in the paper's own Table 2, how the +Zipfian generator produces skew cheaply enough to keep up with a +million ops per second, why the popular keys have to be scattered by +a hash, and where the shipped harness — and this topic's own driver — +throw away the timestamp that would make the tail latencies honest. + +Paper: **Cooper, Silberstein, Tam, Ramakrishnan, Sears, +"Benchmarking Cloud Serving Systems with YCSB", SoCC 2010**, 8 pages. +Code line numbers are **pingcap/go-ycsb@f030f99**, the Go port; the +original is brianfrankcooper/YCSB in Java, and the two agree on the +algorithm and the property names. Repo line numbers are this topic's +`experiments/src/`. + +The title says *five* distributions because that is how the folklore +counts them. The paper names **four** (§4.1) and go-ycsb accepts +**six** for `requestdistribution` — Step 3 reconciles this. Chapter +titles are load-bearing links in SUMMARY.md, so the heading stays; +the body is where the count gets fixed. ## The problem in one sentence -Two serving systems can each claim "1M ops/s" while one was measured -on uniform reads and the other on a 50%-write workload where 10% of -the keys absorb most of the traffic — YCSB exists so that a KV -benchmark names its operation mix and its key distribution, the two -knobs that change the number by 4× (our own A–F runs span 1.11 to -4.40 Mops/s on the same store). +A key-value benchmark has to produce realistic skew — a few keys +getting most of the traffic — millions of times per second without +the generator itself becoming the bottleneck, and without leaking the +hot set's *identity* into the physical layout it is supposed to be +testing. ## The concepts, step by step -### Step 1 — the factoring: workload = op mix × key distribution - -A serving workload (the traffic a key-value store — a database -exposing only get/put/scan on keys — actually sees) is characterized -by two independent choices: *what* operations arrive (the **mix**) -and *which* keys they touch (the **distribution**). YCSB's core -contribution is treating these as orthogonal axes: - -``` - op mix (what) key distribution (where) - A 50r/50u update-heavy uniform — every key equal - B 95r/5u read-mostly zipfian .99 — hot head, θ=0.99 - C 100r read-only latest — zipf over newest - D 95r/5i read-latest scrambled — zipf rank, fnv- - E 95scan/5i short ranges hashed into space - F 50r/50rmw read-modify-write hotspot — x% ops on y% keys -``` - -Property files: `workloads/workloada:31-36` (proportions + -`requestdistribution`). The genius is the factoring — 6 mixes × 5 -distributions covers most serving systems' realities, and any -published "workload B, zipfian" is exactly reproducible. Our -measured spread on one BTreeMap store (uniform keys): A 2.88, B -4.15, C 3.72, D 4.40, E 1.11, F 2.85 Mops/s — the *mix alone* is a -4× lever before skew even enters. - -### Step 2 — Zipf's law: what θ=0.99 means - -A **Zipfian distribution** models popularity skew: the k-th most -popular key is requested with probability proportional to `1/k^θ`, -where θ (theta) tunes how brutal the skew is — θ=0 is uniform, -θ→1 concentrates traffic in a tiny head. Real access logs (web -pages, videos, social profiles) fit this shape, which is why YCSB -defaults to it. - -The probabilities must sum to 1, so each is divided by the -normalizing constant **zetan** = Σ 1/i^θ over all n keys (the -generalized harmonic sum). Concretely at θ=0.99, n=1M: the single -hottest key gets ~7% of all requests, and a few hundred keys absorb -the majority of traffic. YCSB pinned θ=**0.99** — just under 1, -where the math stays finite-friendly — and because everyone copied -the generator, "zipfian 0.99" is now the de-facto meaning of -"skewed" in every KV paper. Cost of ignoring it: a cache-friendly -hot head makes skewed reads *faster* than uniform, so -uniform-vs-zipfian is not a fairness detail, it's the experiment. - -### Step 3 — sampling in O(1): the inverse-CDF trick - -Drawing a Zipf sample naively means walking the cumulative -probabilities until they exceed a random `u` — O(n) per draw. -YCSB's generator instead inverts the cumulative distribution -analytically (an **inverse-CDF** sampler: map uniform `u ∈ [0,1)` -straight to a rank with one `pow()`), paying the O(n) cost once, -at construction, to compute zetan. The most-copied benchmark code -in existence — our `zipf.rs` stub — lives in -`pkg/generator/zipfian.go`: - -| anchor | what | -|---|---| -| :43 | `ZipfianConstant = 0.99` — why every paper says θ=0.99 | -| :92-118 | constructor: `zetan` (harmonic-ish sum, O(n)!), `eta`, `alpha = 1/(1-θ)` | -| :125-132 | `zetaStatic` — the O(n) sum; incremental recompute when item count GROWS (:135-147), full recompute (slow, warned) when it shrinks | -| :150-163 | the sampler: two fast paths (`uz < 1` → rank 0, `< 1+0.5^θ` → rank 1), else `n·(ηu − η + 1)^α` | -| `scrambled_zipfian.go` | fnv64(rank) % n — same skew, scattered hot keys | - -The sampler, transcribed (zipfian.go:150-163): +### Step 1 — what YCSB measures, and the two tiers nobody runs + +> **In:** nothing but the acronym. +> **Out:** the paper's tier structure, and the reason "YCSB numbers" in the +> wild almost always mean tier 1 only. + +YCSB — Yahoo! Cloud Serving Benchmark — was written to compare +"cloud serving" stores (Cassandra, HBase, PNUTS, sharded MySQL in the +paper's §6) on **online serving** work: single-record reads, updates, +inserts and short scans, no joins, no multi-record transactions. The +paper structures it as tiers: + +``` + Tier 1 Performance §3.1 latency vs throughput as offered load rises + Tier 2 Scaling §3.2 scaleup (grow servers with data) and + elastic speedup (add servers mid-run) + Tier 3 Availability §7.1 performance while a server is killed + Tier 4 Replication §7.2 consistency/performance of replicas +``` + +Tiers 3 and 4 are §7's *proposed future* tiers — the paper describes +them but does not report results for them, and neither the Java YCSB +nor go-ycsb ships them. So a "YCSB result" you read anywhere is +almost certainly tier 1, one workload, one client count, and it says +nothing about failure behaviour. + +Also worth knowing before you cite the paper's own numbers: §6 states +its own precision limit — "The 95th and 99th percentile latencies are +not reported" — and its runs were 30 minutes each after 10-20 hour +loads (§4.2). The paper is about *shapes of curves*, not about tails. + +### Step 2 — the six mixes, exactly as Table 2 defines them + +> **In:** Step 1's operation types. +> **Out:** the paper's Table 2 verbatim, the reason F is not in it, and the +> mapping onto this repo's `WORKLOADS` array. + +The classic error is swapping D and E, or claiming F is one of the +core five. Table 2 (§4.2) defines **A through E only**: + +``` + workload operations record selection application example + A update-heavy Read 50% / Update 50% Zipfian session store recording recent + actions in a user session + B read-heavy Read 95% / Update 5% Zipfian photo tagging; add a tag is an + update, most operations read tags + C read-only Read 100% Zipfian user profile cache, profiles + constructed elsewhere (e.g. Hadoop) + D read-latest Read 95% / Insert 5% Latest user status updates; people want + to read the latest statuses + E short-ranges Scan 95% / Insert 5% Zipfian/Uniform* threaded conversations, each scan + for the posts in a given thread + — Table 2, YCSB SoCC 2010 + * Table 2's footnote: "Workload E uses the Zipfian distribution to choose + the first key in the range, and the Uniform distribution to choose the + number of records to scan." +``` + +Two things to fix in your memory: + +- **D is "read latest" and uses the Latest distribution; E is "short + ranges" and is the only one with scans.** D inserts and reads the + *newest* records; E inserts and *scans* from a Zipfian-chosen start. + They are the two insert workloads, which is why they get confused — + but only E scans, and only D is skewed toward recency. +- **F is not in Table 2.** It appears once, in §6.5: "a + 'read-modify-write' workload … similar to workload A (50/50) except + that the updates are 'read-modify-write' rather than blind writes. + The results (not shown) showed the same trends as workload A." F is + a later addition to the tool, and the paper explicitly did not plot + it. If your result for F differs wildly from A, that is a finding + about your read-modify-write path, not a reproduction of anything + in the paper. + +This repo encodes all six, with F's provenance in mind: ```rust -fn next(&mut self, rng: &mut Rng) -> u64 { - let u = rng.f64(); - let uz = u * self.zetan; // zetan = Σ 1/i^θ — O(n), computed ONCE - if uz < 1.0 { return 0; } // fast path: THE hottest key - if uz < 1.0 + 0.5f64.powf(self.theta) { return 1; } - // general case: inverse-CDF approximation, rank from one pow() - let rank = (self.n as f64 - * (self.eta * u - self.eta + 1.0).powf(self.alpha)) as u64; - rank // alpha = 1/(1-θ) -} -``` - -The two fast paths exist because ranks 0 and 1 are so hot they're -worth special-casing before the `pow()` (question 2 asks what -fraction of draws they absorb). The costs to notice: zetan is -**O(n) at startup** — at 1B keys the constructor takes minutes, so -ports cache zetan constants for common sizes — and a *growing* -keyspace (workload D) makes zetan stale (question 5). - -### Step 4 — scrambling: same skew, scattered hot keys - -Plain Zipfian's hot keys are ranks 0, 1, 2, … — which the generator -returns *as key ids*, so the hottest keys are **adjacent**. Adjacent -hot keys share cache lines, pages, and shards, and suddenly you're -benchmarking spatial locality instead of skew. The fix is one hash: +// topics/22-benchmarks/experiments/src/ycsb.rs — the six mixes, 61-68 + 61 pub const WORKLOADS: [Mix; 6] = [ + 62 Mix { name: "A update-heavy", read: 0.5, update: 0.5, insert: 0.0, scan: 0.0, rmw: 0.0 }, + 63 Mix { name: "B read-mostly", read: 0.95, update: 0.05, insert: 0.0, scan: 0.0, rmw: 0.0 }, + 64 Mix { name: "C read-only", read: 1.0, update: 0.0, insert: 0.0, scan: 0.0, rmw: 0.0 }, + 65 Mix { name: "D read-latest", read: 0.95, update: 0.0, insert: 0.05, scan: 0.0, rmw: 0.0 }, + 66 Mix { name: "E short-ranges", read: 0.0, update: 0.0, insert: 0.05, scan: 0.95, rmw: 0.0 }, + 67 Mix { name: "F read-mod-write", read: 0.5, update: 0.0, insert: 0.0, scan: 0.0, rmw: 0.5 }, + 68 ]; +``` + +The proportions match Table 2 for A-E and §6.5 for F. What the array +*cannot* express is the record-selection column — that is the +`KeyGen` passed in, which is why D's "latest" is approximated by +whatever generator you hand it (`ycsb.rs:59-60`) and E's two +distributions collapse to one. + +### Step 3 — the distributions: four in the paper, six in the tool + +> **In:** Step 2's "record selection" column. +> **Out:** an exact count of what exists where, and the one that is not a +> record-selection distribution at all. + +§4.1 says "YCSB has several built-in distributions" and lists +**four**: + +``` + Uniform every record equally likely + Zipfian a few records extremely popular, most unpopular + Latest like Zipfian, but the head is the most recently inserted + Multinomial explicit per-item probabilities + — §4.1, YCSB SoCC 2010 +``` + +**Multinomial is not a record-selection distribution.** §4.1's own +example is choosing the *operation*: "we might assign a probability +of 0.95 to the Read operation, a probability of 0.05 to the Update +operation, and a probability of 0 to Scan and Insert." That is Step +2's mix, not Step 2's key choice. So the paper offers **three** +record-selection distributions, not five. + +The tool has grown since. go-ycsb's `requestdistribution` property +accepts exactly six values: + +```go +// go-ycsb pkg/workload/core.go — the requestdistribution switch, 655-678 (elided) + 655 switch requestDistrib { + 656 case "uniform": + 657 c.keyChooser = generator.NewUniform(keyrangeLowerBound, keyrangeUpperBound) + 658 case "sequential": + 659 c.keyChooser = generator.NewSequential(keyrangeLowerBound, keyrangeUpperBound) + 660 case "zipfian": + // ... 661-664: expand the keyrange for expected inserts — see Step 6 ... + 665 c.keyChooser = generator.NewScrambledZipfian(keyrangeLowerBound, keyrangeUpperBound, generator.ZipfianConstant) + 666 case "latest": + 667 c.keyChooser = generator.NewSkewedLatest(c.transactionInsertKeySequence) + 668 case "hotspot": + // ... 669-670: read hotset/hotop fractions ... + 671 c.keyChooser = generator.NewHotspot(keyrangeLowerBound, keyrangeUpperBound, hotsetFraction, hotopnFraction) + 672 case "exponential": + // ... 673-674: read percentile/frac ... + 675 c.keyChooser = generator.NewExponential(percentile, float64(c.recordCount)*frac) + 676 default: + 677 util.Fatalf("unknown request distribution %s", requestDistrib) + 678 } +``` + +Note line 665: **asking for `zipfian` gets you `NewScrambledZipfian`, +not `NewZipfian`.** The plain Zipfian generator is never a key chooser +— it is always wrapped. Step 5 is why. + +So the honest count: three record-selection distributions in the +paper, six selectable in go-ycsb (uniform, sequential, zipfian, +latest, hotspot, exponential), and the chapter title's "five" is +folklore that split the difference. Use the numbers, not the title. + +### Step 4 — the Zipfian generator: constant time, precomputed zeta + +> **In:** Step 3's `zipfian` case. +> **Out:** the three precomputed constants, the two fast paths, and a +> computed probability for the hottest key that you can check against the +> code's own hardcoded constant. + +A Zipfian draw over n items assigns rank i (1-based) probability +proportional to `1/i^θ`, normalized by the generalized harmonic +number: + +``` + ζ(n, θ) = Σ 1/i^θ for i = 1 .. n + P(rank i) = (1/i^θ) / ζ(n, θ) +``` + +Computing that sum per draw is O(n) — hopeless. YCSB uses the +inversion method from **Gray, Sundaresan, Englert, Baclawski, Weinberger, +"Quickly Generating Billion-Record Synthetic Databases", SIGMOD +1994** (§5.3 cites it), which precomputes three constants at +construction and then does one `Float64()` and one `math.Pow` per +draw: + +```go +// go-ycsb pkg/generator/zipfian.go — precomputation, 97-118 (elided) + 97 func NewZipfian(min int64, max int64, zipfianConstant float64, zetan float64) *Zipfian { + 98 items := max - min + 1 + // ... 99-107: z.items, z.base, z.theta ... + 108 z.zeta2Theta = z.zeta(0, 2, theta, 0) + 109 + 110 z.alpha = 1.0 / (1.0 - theta) + 111 z.zetan = zetan + 112 z.countForZeta = items + 113 z.eta = (1 - math.Pow(2.0/float64(items), 1-theta)) / (1 - z.zeta2Theta/z.zetan) + // ... 114-117: seed, prime the generator ... + 118 } +``` + +```go +// go-ycsb pkg/generator/zipfian.go — the O(n) sum, done once, 125-133 + 125 func zetaStatic(st int64, n int64, theta float64, initialSum float64) float64 { + 126 sum := initialSum + 127 + 128 for i := st; i < n; i++ { + 129 sum += 1 / math.Pow(float64(i+1), theta) + 130 } + 131 + 132 return sum + 133 } +``` + +- `zetan` = ζ(n, θ), the normalizer — the only O(n) work, done once. +- `zeta2Theta` = ζ(2, θ) = 1 + 2^−θ, the mass of the top two ranks. +- `alpha` = 1/(1−θ), the inversion exponent (line 110). +- `eta` (line 113) is the interpolation constant that maps a uniform + u onto the rank scale; it is the piece that lets the draw be one + `Pow` instead of a search. + +The draw itself: + +```go +// go-ycsb pkg/generator/zipfian.go — one draw, 151-164 + 151 u := r.Float64() + 152 uz := u * z.zetan + 153 + 154 if uz < 1.0 { + 155 return z.base + 156 } + 157 + 158 if uz < 1.0+math.Pow(0.5, z.theta) { + 159 return z.base + 1 + 160 } + 161 + 162 ret := z.base + int64(float64(itemCount)*math.Pow(z.eta*u-z.eta+1, z.alpha)) + 163 z.SetLastValue(ret) + 164 return ret +``` + +Lines 154-160 are the two special cases of the inversion, and they +are also the arithmetic worth doing. Since `uz = u × ζn` with u +uniform on [0,1): + +``` + P(fast path 1, rank 0) = P(u·ζn < 1) = 1 / ζn + P(fast path 2, rank 1) = P(1 ≤ u·ζn < 1+2^−θ) = 2^−θ / ζn + 2^−0.99 = 0.503478 +``` + +Evaluate at θ = 0.99 for a 1,000,000-key store — sum +`Σ 1/i^0.99` for i = 1..10⁶ and you get **ζ = 15.3918497460**: + +``` + P(hottest key) = 1 / 15.39185 = 0.064969 = 6.50% + P(second key) = 0.503478 / 15.39185 = 0.032711 = 3.27% + P(either fast path) = 0.097680 = 9.77% + + top 100 keys (ζ(100)/ζ(10⁶) = 5.29457/15.39185) = 34.40% of all draws + top 1,000 keys (ζ(1000)/ζ(10⁶) = 7.72895/15.39185) = 50.21% of all draws +``` + +**Half of all traffic lands on 0.1% of the keys, and one draw in ten +never reaches the `math.Pow` on line 162.** That is what "Zipfian +θ=0.99" buys, and it is why cache-hit-rate results are so sensitive +to it. + +You can check the code's arithmetic without running it. Step 5's +scrambled generator hardcodes `zetan = 26.46902820178302` for +n = 10¹⁰, θ = 0.99. Summing 10 billion terms directly is impractical, +but Euler–Maclaurin on the tail reproduces it: + +``` + ζ(10¹⁰, 0.99) ≈ Σ_{i≤10⁵} i^−0.99 + + (10¹⁰·⁰¹ − 10⁵·⁰¹)/0.01 [∫ x^−0.99 dx] + − ½·10⁵^−0.99 + ½·10¹⁰^−0.99 + + 0.99·(10⁵^−1.99 − 10¹⁰^−1.99)/12 + = 26.46902820175… + hardcoded 26.46902820178302 → agrees to 10 significant figures +``` + +At that keyspace, `P(hottest) = 1/26.469 = 3.78%` and the two fast +paths absorb 5.68% of draws — skew *falls* as the keyspace grows, +which is exactly what a fixed θ means. + +**A defect worth noticing while you read**: `SetLastValue(ret)` is +called only on line 163, on the general path. Both fast paths return +at 155 and 159 without updating it. Any consumer that reads +`LastValue()` — the `SkewedLatest` generator does — sees a stale +value ~9.8% of the time at n = 10⁶. + +### Step 5 — scrambling: the hot set must not be lexicographically first + +> **In:** Step 4's generator, which makes rank 0 the hottest key. +> **Out:** the layout leak that creates, the paper's two failed fixes and its +> accepted one, and the exact constants go-ycsb uses. + +Gray's algorithm returns *ranks*: item 0 is hottest, item 1 next, and +so on. §5.3 states the problem plainly: + +> "The first problem is that the popular items are clustered together +> in the keyspace. In particular, the most popular item is item 0; the +> second most popular item is item 1, and so on. For the Zipfian +> distribution, the popular items should be scattered across the +> keyspace. In real web applications, the most popular user or blog +> topic is not necessarily the lexicographically first item." + +This is a *methodology* bug, not an aesthetic one. If the hot keys +are `user0, user1, user2, …`, then in any range-partitioned or +clustered store they land in the same leaf pages, the same shard and +the same cache lines. You would be measuring one hot page, and a +B-tree would look artificially wonderful against a hash index. The +benchmark must not leak the hot set's identity into the physical +layout it is testing — the same principle as TPC-C's NURand +load-vs-run constants ([reading-oltpbench-tpcc.md](reading-oltpbench-tpcc.md) +Step 4). + +§5.3 records two attempts and the fix: + +1. **Hash with `String.hashCode()`** — "tended to leave the popular + items clustered". Rejected. +2. **Hash with anything, 1:1** — "after hashing, collisions meant + that only about **80 percent** of the keyspace would be generated + in the sequence. This was true even as we tried a variety of hash + functions (FNV, Jenkins, etc.)." Perfect hashing was considered + and rejected for setup cost — "multiple minutes for hundreds of + millions of records". +3. **Accepted**: "construct a Zipfian generator for a much larger + keyspace than we actually needed; apply the FNV hash to each + generated value; and then take mod N … The result was that + **99.97%** of the keyspace is generated, and the generated keys + continued to have a Zipfian distribution." + +"Much larger" is a specific number in go-ycsb: + +```go +// go-ycsb pkg/generator/scrambled_zipfian.go — the oversized inner keyspace, 50-67 (elided) + 50 func NewScrambledZipfian(min int64, max int64, zipfianConstant float64) *ScrambledZipfian { + 51 const ( + 52 zetan = float64(26.46902820178302) + 53 usedZipfianConstant = float64(0.99) + 54 itemCount = int64(10000000000) + 55 ) + // ... 56-60: s.min, s.max, s.itemCount = max - min + 1 ... + 61 if zipfianConstant == usedZipfianConstant { + 62 s.gen = NewZipfian(0, itemCount, zipfianConstant, zetan) + 63 } else { + 64 s.gen = NewZipfianWithRange(0, itemCount, zipfianConstant) + 65 } + 66 return s + 67 } +``` + +```go +// go-ycsb pkg/generator/scrambled_zipfian.go — scatter, 70-76 + 70 func (s *ScrambledZipfian) Next(r *rand.Rand) int64 { + 71 n := s.gen.Next(r) + 72 + 73 n = s.min + util.Hash64(n)%s.itemCount + 74 s.SetLastValue(n) + 75 return n + 76 } +``` + +The inner generator always draws over **10 billion** items regardless +of your `recordcount`; line 73 folds that down with FNV into the real +range. Line 52's hardcoded zetan is why: ζ over 10¹⁰ terms is the one +sum you cannot afford at startup, so it is precomputed for the +default θ = 0.99, and line 64's fallback pays the O(n) loop only if +you change θ (which the doc comment at `zipfian.go:47-64` warns takes +"over a minute for 100 million objects"). + +The hash is FNV-1a over the big-endian 8 bytes: + +```go +// go-ycsb pkg/util/hash.go — FNV-1a 64, 21-32 + 21 // Hash64 returns a fnv Hash of the integer. + 22 func Hash64(n int64) int64 { + 23 var b [8]byte + 24 binary.BigEndian.PutUint64(b[0:8], uint64(n)) + 25 hash := fnv.New64a() + 26 hash.Write(b[0:8]) + 27 result := int64(hash.Sum64()) + 28 if result < 0 { + 29 return -result + 30 } + 31 return result + 32 } +``` + +**Why 10¹⁰ and not, say, 2×N?** The 99.97% figure depends on the +ratio. If the inner keyspace M equals the outer N, FNV mod N behaves +like a random function and the expected coverage is + +``` + N · (1 − (1 − 1/N)^M) → N · (1 − 1/e) = 63.2% when M = N +``` + +— a third of your keys never drawn at all. With M = 10¹⁰ ≫ N, every +residue class is hit many times over and the shortfall collapses to +the paper's 0.03%. This is the calculation to redo before you shrink +the inner keyspace in `zipf.rs`, whose stub currently pairs a +1,000,000-item inner generator with a 1,000,000-key store. + +### Step 6 — the growing keyspace: N + T×I + ε + +> **In:** Step 5's fixed inner keyspace and Step 2's insert workloads. +> **Out:** why a Zipfian generator cannot simply be resized mid-run, and the +> two different fixes for Zipfian and Latest. + +Workloads D and E insert. If the generator's keyspace is fixed at +load size N, it can never draw the records the run inserts. §5.3: + +> "For Zipfian, we expanded the initial keyspace to the expected size +> after inserts. If a data set had N records, and the workload had T +> total operations, with an expected fraction I of inserts, then we +> constructed the Zipfian generator to draw from a space of size +> **N + T × I + ε**. We added an additional factor ε since the actual +> number of inserts depends on the random choice of operations during +> the workload according to a multinomial distribution. While running +> the workload, if the generator produced an item which had not been +> inserted yet, we [skipped it and redrew]." + +go-ycsb implements ε as a factor of two on the insert term: + +```go +// go-ycsb pkg/workload/core.go — N + T×I×2, 660-665 + 660 case "zipfian": + 661 insertProportion := p.GetFloat64(prop.InsertProportion, prop.InsertProportionDefault) + 662 opCount := p.GetInt64(prop.OperationCount, 0) + 663 expectedNewKeys := int64(float64(opCount) * insertProportion * 2.0) + 664 keyrangeUpperBound = insertStart + insertCount + expectedNewKeys + 665 c.keyChooser = generator.NewScrambledZipfian(keyrangeLowerBound, keyrangeUpperBound, generator.ZipfianConstant) +``` + +For workload E at the shipped defaults (`recordcount=1000`, +`operationcount=1000`, `insertproportion=0.05`): + +``` + expectedNewKeys = 1000 × 0.05 × 2.0 = 100 + keyrange = [0, 0 + 1000 + 100 − 1] = 1,099 keys for a 1,000-record load +``` + +Latest gets the *opposite* treatment — its head must move to the new +keys, so §5.3 recomputes the distribution on every insert, "to do +this cheaply we modified the Gray algorithm of [23] to compute its +constants incrementally". go-ycsb does the same for Zipfian when the +item count grows: + +```go +// go-ycsb pkg/generator/zipfian.go — incremental zeta on growth, 136-149 (elided) + 136 if itemCount != z.countForZeta { + 137 z.lock.Lock() + 138 if itemCount > z.countForZeta { + 139 //we have added more items. can compute zetan incrementally, which is cheaper + 140 z.zetan = z.zeta(z.countForZeta, itemCount, z.theta, z.zetan) + 141 z.eta = (1 - math.Pow(2.0/float64(z.items), 1-z.theta)) / (1 - z.zeta2Theta/z.zetan) + 142 } else if itemCount < z.countForZeta && z.allowItemCountDecrease { + 143 //note : for large itemsets, this is very slow. so don't do it! + 144 fmt.Printf("recomputing Zipfian distribution, should be avoided,item count %v, count for zeta %v\n", itemCount, z.countForZeta) + // ... 145-146: full O(n) recompute ... + 147 } + 148 z.lock.Unlock() + 149 } +``` + +Growth is a partial sum resumed from `countForZeta` (line 140) — +O(Δ). *Shrinking* is O(n) and prints a warning to stderr (144), which +is a rare and admirable piece of honesty: a benchmark generator +telling you it is about to distort your measurement. + +### Step 7 — where the tail latency goes missing + +> **In:** Step 6's driver loop. +> **Out:** the two lines that create coordinated omission in go-ycsb, the one +> line that creates it in this repo, and the pointer to where it is measured. + +go-ycsb *does* have a rate knob — the paper's Fig. 2 lists "Target +throughput" among the client's command-line properties — and it +computes each operation's **intended** start time correctly: + +```go +// go-ycsb pkg/client/client.go — the intended schedule, 97-111 + 97 func (w *worker) throttle(ctx context.Context, startTime time.Time) { + 98 if w.targetOpsPerMs <= 0 { + 99 return + 100 } + 101 + 102 d := time.Duration(w.opsDone * w.targetOpsTickNs) + 103 d = startTime.Add(d).Sub(time.Now()) + 104 if d < 0 { + 105 return + 106 } + 107 select { + 108 case <-ctx.Done(): + 109 case <-time.After(d): + 110 } + 111 } +``` + +Line 102 is an *absolute* schedule — operation k was supposed to +begin at `startTime + k × tick`, not "tick nanoseconds after the last +one finished". That is the right way to build an open-loop driver, +and it means the intended time genuinely exists in the process. + +Then line 104 throws it away. If the worker is already late +(`d < 0`), it returns and immediately issues the next operation — and +the measurement clock starts at *issue* time, not intended time: + +```go +// go-ycsb pkg/client/dbwrapper.go — where the clock starts, 53-57 + 53 func (db DbWrapper) Read(ctx context.Context, table string, key string, fields []string) (_ map[string][]byte, err error) { + 54 start := time.Now() + 55 defer func() { + 56 measure(start, "READ", err) + 57 }() +``` + +`start := time.Now()` on line 54 is taken after the stall has already +happened. Every microsecond spent waiting to *get to* the operation +is invisible; only service time is recorded. The intended time is +computed at `client.go:102` and never passed to `measure`. The +one-line fix is to plumb it through — which tells you this is a +design decision, not an oversight. + +The throttle also runs *after* the operation, not before: + +```go +// go-ycsb pkg/client/client.go — throttle placement, 144-147 + 144 if measurement.IsWarmUpFinished() { + 145 w.opsDone += int64(opsCount) + 146 w.throttle(ctx, startTime) + 147 } +``` + +So with no target set (`targetOpsPerMs <= 0`, line 98), the loop is a +pure closed loop: the next request is only issued once the previous +reply arrives. When the store stalls, the load politely stops. + +**This repo's driver has the same omission, deliberately and +declaredly.** `ycsb.rs:1-7` says so — "closed-loop, single thread … +no threads, no target rate" — and the clock placement makes it +concrete: ```rust -fn next_scrambled(&mut self, rng: &mut Rng) -> u64 { - fnv64(self.next(rng)) % self.n // same skew, hot keys NOT ids 0,1,2… -} -``` - -Rank → fnv64 hash → key id: the popularity *distribution* is -unchanged, but the hot keys land anywhere in the keyspace. This is -YCSB's default "zipfian". (Our test pins the property: the hottest -key must not be id 0.) - -### Step 5 — coordinated omission: the closed-loop lie - -YCSB's driver is **closed-loop**: each client thread issues an -operation, waits for it to finish, then issues the next. When one -operation stalls for 100 ms, the thread sends *nothing* during the -stall — so the stall appears **once** in the histogram, while a -real open-loop client (requests arriving on a schedule, regardless -of responses) would have had dozens of requests queue up behind it, -each experiencing most of that 100 ms. Tene named this -**coordinated omission** (topic 0): the benchmark and the system -coordinate to omit the worst latencies. Recorded p999 under load is -fiction unless you use a target rate plus intended-start-time -correction (measure each op from when it *should* have started). -Our driver records service time only — question 4 asks you to -sketch the fix. - -### Step 6 — what YCSB deliberately doesn't test - -YCSB benchmarks the KV layer and nothing above it: **no -transactions** (no operation spans two keys), **no -scans-with-filter**, and values are opaque blobs (no schema, no -secondary indexes). That's fine for M22's graph micro-benches — a -graph engine's adjacency reads are close to KV reads — and wrong -for any MVCC/isolation claim (topic 8's territory; see TPC-C in -reading-oltpbench-tpcc.md for contention that spans keys). Citing -YCSB for a transactional system measures its cheapest path. - -## How to read the paper (with the concepts in hand) - -SoCC 2010, ~12 pages; the design half aged well, the eval half didn't: - -- **§1–2** Motivation (cloud serving stores circa 2010) — skim; - the system list (Cassandra, HBase, PNUTS, "sharded MySQL") is a - time capsule. -- **§3–4 — read carefully.** The workload-design sections: the - mix × distribution factoring (Step 1), the distribution zoo - (Step 2), and the tiered "performance vs scaling" methodology. - This is the reusable part. -- **§5–6** The 2010 measurements — skim for method, ignore the - numbers; every system in them has been rewritten since. -- Then read the generator source with Step 3's anchor table: - constructor first (zetan, eta, alpha), then the sampler's two - fast paths, then `scrambled_zipfian.go` (Step 4). It's ~200 - lines total. -- Nothing in the paper covers coordinated omission (Step 5) — it - predates Tene's talk; carry that critique with you. +// topics/22-benchmarks/experiments/src/ycsb.rs — the clock starts after key selection, 95-117 (elided) + 95 for _ in 0..ops { + 96 let k = keygen.next(next_id as usize) as u64; + 97 let r: f64 = rng.gen(); + 98 let t = Instant::now(); + 99 if r < mix.read { + 100 std::hint::black_box(store.read(k)); + // ... 101-116: the update / insert / scan / read-modify-write arms ... + 117 hist.record(t.elapsed().as_nanos() as u64); +``` + +Line 98 starts the timer *after* the key generator and RNG have run, +so `zipf.rs`'s own cost is excluded — good, that is what you want +when comparing mixes — but there is no intended time anywhere, so +every percentile this driver prints is a **service-time** percentile. + +Do not re-derive what that costs: **topic 34 measured it**. +FINDINGS.md row 34 records a closed-loop p99 of **1.0 µs** against an +open-loop **90 ms** on identical work — a **90,000×** lie. Cite that +number; the mechanism is topic 34's to explain. + +The practical rule: a YCSB percentile is comparable to another YCSB +percentile on the same driver, and to nothing else. This topic's +headline (FINDINGS.md row 22) is stated as a *ratio* for exactly that +reason — "YCSB-E's p999 is 12.9 µs against read-only's 4.0 µs". + +### Step 8 — read the property files, not the property names + +> **In:** Steps 2 and 3. +> **Out:** three concrete discrepancies in the shipped workload files, and +> the habit they should install. + +The shipped `workloads/workload*` files are the most-copied +configuration in the field, and at `f030f99` they contradict their own +documentation: + +``` +# go-ycsb workloads/workloada — the comment and the property disagree, 18-36 (elided) + 18 # Workload A: Update heavy workload + # ... 19-22: application example, ratio, record size ... + 23 # Request distribution: zipfian + 24 + 25 recordcount=1000 + 26 operationcount=1000 + # ... 27-30: workload=core, readallfields ... + 31 readproportion=0.5 + 32 updateproportion=0.5 + 33 scanproportion=0 + 34 insertproportion=0 + 35 + 36 requestdistribution=uniform +``` + +Line 23 says zipfian. Line 36 says **uniform**. The same +contradiction is in `workloadb` (comment :22, property :35), +`workloadc` (:22, :35), `workloade` (:22, :40) and `workloadf` +(:22, :36). Only `workloadd` is consistent, because Latest is what +it sets (`requestdistribution=latest`, :40). **Running the shipped +files unmodified gives you a uniform key distribution for five of the +six workloads** — no skew, no hot set, and a cache-hit rate that has +nothing to do with the workload you think you ran. + +Workload E has a second problem: + +``` +# go-ycsb workloads/workloade — "short ranges" of length one, 35-44 + 35 readproportion=0 + 36 updateproportion=0 + 37 scanproportion=0.95 + 38 insertproportion=0.05 + 39 + 40 requestdistribution=uniform + 41 + 42 maxscanlength=1 + 43 + 44 scanlengthdistribution=uniform +``` + +`maxscanlength=1` (line 42) makes every "scan" return a single +record. Table 2's whole point for E is *short ranges* — sequential +access that a hash index cannot serve and a B-tree can. With +`maxscanlength=1`, E degenerates into workload C with 5% inserts, and +the structural difference the workload exists to expose disappears. + +And `recordcount=1000` / `operationcount=1000` (lines 29-30) is a +smoke-test size: 1,000 records of ~1 KB is a megabyte, which fits in +L2. Any storage result from the unmodified files is a measurement of +your CPU cache. + +This repo's driver is, on this narrow point, *more* faithful than the +shipped files: `ycsb.rs:110` scans 100 records +(`store.scan(k, 100)`), which is a short range in Table 2's sense. + +**The habit:** before quoting any benchmark result, open the config +and read the properties, not the comments. The Boncz guide's version +of this is checking which query variant was run; TPC-C's is checking +whether think times were enabled ([reading-oltpbench-tpcc.md](reading-oltpbench-tpcc.md) +Step 5). Same failure, three benchmarks. + +## Where each step lives in the sources + +The paper is 8 pages; read §4 and §5, skim the rest. + +- **§1-2** — the motivation and the tradeoffs (read/write, latency/ + durability, synchronous/asynchronous replication). Skim. +- **§3** — the tier model (Step 1). Read §3.1 and §3.2; §7's tiers 3 + and 4 are proposals, so read them last and do not cite them as + implemented. +- **§4 — read carefully.** §4.1 the four distributions (Step 3), + §4.2 and **Table 2** the five core workloads (Step 2). Table 2 is + the single most-misquoted table in the paper; copy it out by hand. +- **§5 — read carefully.** §5.1 architecture, §5.2 the extension + points, and **§5.3** the generator engineering: Gray's algorithm, + the clustering problem, the 80%-coverage failure, the oversized- + keyspace fix and its 99.97%, and the `N + T×I + ε` growing keyspace + (Steps 4-6). §5.3 is the densest page in the paper. +- **§6** — the results. Read 6.4 (workload E) and 6.5 (where F is + mentioned, and only mentioned). Remember §6's own caveat that 95th + and 99th percentiles are not reported. +- Then open go-ycsb in this order: `pkg/generator/zipfian.go` + (97-133 precompute, 135-165 draw), `scrambled_zipfian.go` (50-76), + `pkg/util/hash.go` (21-32), `pkg/workload/core.go` (655-678), + `pkg/client/client.go` (97-111, 144-147), `pkg/client/dbwrapper.go` + (30-39, 53-60), and finally `workloads/workload{a..f}` with Step 8 + in mind. ## Questions (answer in notes.md) -1. Derive why P(rank 0) = 1/ζ(n,θ). Then: at n=1M, θ=0.99, what - fraction of ops hit the top 100 keys? (Compute, then verify with - the stub.) -2. Why do the two fast paths in `next()` exist — what fraction of - draws do they absorb at θ=0.99? -3. Predict uniform → zipfian effect per workload on OUR BTreeMap - store: A-F, which speeds UP (cache-hot head) and which barely - moves (E's scans)? Fill the prediction table before implementing. -4. Coordinated omission: our driver records service time. Sketch the - fix (intended arrival times at a target rate) and what p999 would - show for workload E. -5. Workload D's "latest" distribution: why is passing a plain - zipfian to a growing keyspace subtly wrong (hint: zetan - staleness, go-ycsb :135)? +1. Step 4 computed P(hottest key) = 6.50% at n = 10⁶, θ = 0.99, and + 3.78% at go-ycsb's inner n = 10¹⁰. Our `zipf.rs` stub builds its + inner generator at 1,000,000 items. Which of those two skews will + `Scrambled` actually produce, and does it matter for the + measurement — or only for the comparison to published numbers? +2. Step 5's coverage formula gives 63.2% when the inner keyspace + equals the outer. Work out what inner size you need for 99% + coverage of a 1,000,000-key store, and decide whether `zipf.rs` + should adopt it or document the divergence. +3. `hash.go:24` uses **big-endian** bytes; `zipf.rs`'s stub doc says + little-endian. Does endianness change the *distribution* of + `hash % N`, or only which specific keys are hot? Justify, then say + which property the benchmark actually needs. +4. Step 4 found `SetLastValue` is skipped on both fast paths + (`zipfian.go:155`, `:159`). `SkewedLatest` consumes `LastValue()`. + Estimate how often that stale read happens at n = 10⁶ and at + n = 10¹⁰, and say whether it biases workload D toward or away from + recency. +5. Step 8: reproduce the *intended* workload A by fixing the property + files. Predict, before running, how the Mops/s and p999 move when + `requestdistribution` goes uniform → zipfian at 1M keys, and what + fraction of reads should hit the top 1,000 keys (Step 4 gives you + the number). +6. Design a workload G that would expose something A-F cannot. What + is the operation mix, what is the record-selection distribution, + and what structural property of the store does it separate? ## Done when -- [ ] You can state the factoring: workload equals op mix times key distribution, and name all six mixes. -- [ ] You can derive `P(rank 0) = 1/ζ(n,θ)` and compute the hot-key probability at n=1M, θ=0.99. -- [ ] You can explain the O(1) inverse-CDF sampling trick and why the two fast paths in `next()` exist. -- [ ] You can explain what scrambling changes and what it deliberately preserves. -- [ ] You can predict the uniform-to-Zipf effect per workload against this topic's measured uniform baseline (3.16 Mops/s on B, 0.86 on E) before implementing `zipf.rs`. -- [ ] You wrote answers to all five questions in notes.md, including the coordinated-omission fix for the driver. +Answer each before unfolding it. + +- [ ] You can write out Table 2 from memory — all five core workloads with their operation mix and record-selection distribution — and say where F comes from. + +
Answer + + A update-heavy: Read 50% / Update 50%, Zipfian, session store. + B read-heavy: Read 95% / Update 5%, Zipfian, photo tagging. + C read-only: Read 100%, Zipfian, user profile cache. + D read-latest: Read 95% / **Insert** 5%, **Latest**, user status updates. + E short-ranges: **Scan** 95% / Insert 5%, **Zipfian/Uniform** — Zipfian + picks the first key, Uniform picks the scan length (Table 2's footnote) — + threaded conversations. + + F is **not** in Table 2. §6.5 mentions it once: a read-modify-write variant + of A, "The results (not shown) showed the same trends as workload A." + +
+ +- [ ] You can say how many record-selection distributions the paper actually defines, and which of its four is not one. + +
Answer + + §4.1 lists four: Uniform, Zipfian, Latest, Multinomial. **Multinomial is not + a record-selection distribution** — §4.1's own example uses it to choose the + *operation* (0.95 Read / 0.05 Update / 0 Scan / 0 Insert), which is the mix, + not the key. So the paper defines **three** record-selection distributions. + + go-ycsb accepts six for `requestdistribution` (`core.go:655-678`): uniform, + sequential, zipfian, latest, hotspot, exponential — and `zipfian` constructs + a **Scrambled**Zipfian (line 665), never a bare one. The chapter title's + "five" is folklore. + +
+ +- [ ] You can name the three constants the Zipfian generator precomputes, compute the probability of the hottest key at n = 10⁶ and θ = 0.99, and say what fraction of draws never reach `math.Pow`. + +
Answer + + `zetan` = ζ(n,θ) (the O(n) normalizer, `zipfian.go:111` — passed in, or + summed by `zetaStatic` at 125-133), `zeta2Theta` = ζ(2,θ) = 1 + 2^−θ (:108), + `alpha` = 1/(1−θ) (:110), and the derived `eta` (:113). + + ζ(10⁶, 0.99) = 15.3918497460, so P(rank 0) = 1/15.39185 = **6.50%** and + P(rank 1) = 2^−0.99/15.39185 = 0.503478/15.39185 = **3.27%**. The two fast + paths at `zipfian.go:154-156` and `:158-160` therefore absorb **9.77%** of + draws before line 162's `math.Pow`. Top-1,000 of 1,000,000 keys take + ζ(1000)/ζ(10⁶) = 7.72895/15.39185 = **50.2%** of all traffic. + + Bonus check: the hardcoded `zetan = 26.46902820178302` in + `scrambled_zipfian.go:52` is ζ(10¹⁰, 0.99), reproducible to 10 significant + figures by Euler–Maclaurin. + +
+ +- [ ] You can explain why the Zipfian output must be hashed, what the paper tried first, and why the inner keyspace is 10 billion. + +
Answer + + Gray's algorithm returns ranks, so the hottest keys are 0, 1, 2, … — §5.3: + "the popular items should be scattered across the keyspace. In real web + applications, the most popular user or blog topic is not necessarily the + lexicographically first item." Unscattered, the hot set lands in the same + leaf pages and shard, so you measure one hot page and flatter range- + partitioned stores. + + Attempt 1: `String.hashCode()` — "tended to leave the popular items + clustered". Attempt 2: any 1:1 hash — collisions meant "only about **80 + percent** of the keyspace would be generated", true for FNV, Jenkins and + others; perfect hashing was rejected at "multiple minutes for hundreds of + millions of records". Accepted fix: draw from a much larger keyspace, FNV + the value, take mod N — **99.97%** coverage with the Zipfian shape intact. + + go-ycsb makes "much larger" = **10,000,000,000** + (`scrambled_zipfian.go:54`), folded down by FNV-1a over big-endian bytes + (`hash.go:22-32`) at `scrambled_zipfian.go:73`. The ratio is what matters: + at inner = outer the expected coverage is only N(1 − 1/e) = **63.2%**. + +
+ +- [ ] You can state the growing-keyspace rule and compute the keyrange for workload E at the shipped defaults. + +
Answer + + §5.3: for Zipfian, expand the keyspace to **N + T × I + ε**, where N is the + loaded record count, T the total operations and I the expected insert + fraction; ε covers the variance from choosing operations multinomially. + Items not yet inserted are skipped and redrawn. Latest instead recomputes + its constants incrementally on every insert. + + go-ycsb sets ε by doubling the insert term (`core.go:663`: + `opCount × insertProportion × 2.0`). At workloade's shipped + `recordcount=1000`, `operationcount=1000`, `insertproportion=0.05`: + expectedNewKeys = 1000 × 0.05 × 2.0 = **100**, giving a keyrange of + [0, 1099] — 1,100 keys for a 1,000-record load. + + Growth is handled incrementally (`zipfian.go:138-141`, resuming the partial + sum from `countForZeta`); shrinking costs O(n) and prints a warning + (`:142-147`). + +
+ +- [ ] You can point at the exact lines where go-ycsb and this repo's driver discard the intended start time, and cite what that costs. + +
Answer + + `client.go:102-103` computes the correct absolute schedule — + `startTime + opsDone × targetOpsTickNs` — and `:104-106` discards it with + `if d < 0 { return }` when the worker is late. The measurement clock then + starts at issue time: `dbwrapper.go:54`, `start := time.Now()`, inside + `Read` after the stall. `measure` (`:30-39`) never sees an intended time. + The throttle also runs *after* the operation (`client.go:144-147`), so with + no target it is a pure closed loop. + + This repo's `ycsb.rs:98` does the same: `let t = Instant::now();` after + key generation, with no intended time anywhere — declaredly, per the module + header at `ycsb.rs:1-7`. So every percentile it prints is service time. + + The cost is measured in **topic 34**, not here: FINDINGS.md row 34 reports + closed-loop p99 = 1.0 µs against open-loop 90 ms on identical work — a + **90,000×** lie. That is why this topic's headline is a ratio: FINDINGS.md + row 22, "YCSB-E's p999 is 12.9 µs against read-only's 4.0 µs". + +
+ +- [ ] You can name three things wrong with the shipped `workloads/` files, and say what each one silently changes. + +
Answer + + 1. **The comment contradicts the property.** `workloada:23` documents + "Request distribution: zipfian"; `workloada:36` sets + `requestdistribution=uniform`. Same in b (:22/:35), c (:22/:35), + e (:22/:40) and f (:22/:36); only d is consistent. Five of six shipped + workloads run with **no skew at all** — no hot set, and a cache-hit rate + unrelated to the workload's premise. + 2. **`workloade:42` sets `maxscanlength=1`.** Every "scan" returns one + record, so E collapses into C plus 5% inserts and stops testing ordered + access — the one structural property it exists to test. + 3. **`recordcount=1000` / `operationcount=1000`** in every file. At ~1 KB + per record that is one megabyte — an L2-resident smoke test, not a + storage benchmark. + + This repo's `ycsb.rs:110` scans 100 records, which is closer to Table 2's + intent than the shipped file is. + +
+ +- [ ] You have predictions in notes.md for the uniform → Zipfian move on every workload, written before you implement `zipf.rs`. + +
Answer + + Self-check — the predictions belong in `notes.md`, written before the run. + Anchor them to the measured baseline rather than to intuition. The canonical + headline is FINDINGS.md row 22 (measured 2026-07-28): **YCSB-E's p999 is + 12.9 µs against read-only's 4.0 µs**, a ratio of 3.2×. The `notes.md` + baseline records an earlier run (M3 Pro, 2026-07-10) with uniform keys at + A 2.88, B 4.15, C 3.72, D 4.40, E 1.11, F 2.85 Mops/s and p999 of 2,041 ns + (E) against 958 ns (C) — a ratio of 2.1×. The absolute numbers are + machine- and run-dependent; **the ratio is the invariant**, so predict + ratios. + + A model for E-vs-C: a read is one descent of a `BTreeMap` holding 10⁶ keys, + and a 100-record scan is that descent plus a leaf walk. With Rust's node + capacity of 11 keys (`B = 6` in your toolchain's + `library/alloc/src/collections/btree/node.rs` — check it), the descent is + ⌈log₁₁ 10⁶⌉ ≈ 6 node visits and the walk is ⌈100/11⌉ ≈ 9 more, so + (6 + 9)/6 ≈ **2.5×** the work. That brackets both measured ratios, which is + the point: a prediction you can defend beats a number you remembered. + + Going uniform → Zipfian at 1M keys, expect *throughput to rise* on the + read-mostly mixes, because Step 4's top 1,000 keys take 50.2% of the + traffic and fit trivially in cache — and expect the p999 to move much less, + because the tail is set by the cold 49.8%. + +
## References **Papers** -- Cooper, Silberstein, Tam, Ramakrishnan, Sears — "Benchmarking - Cloud Serving Systems with YCSB" (SoCC 2010) — §3-4 (the - mix×distribution factoring); the eval section is dated +- Cooper, Silberstein, Tam, Ramakrishnan, Sears — "Benchmarking Cloud + Serving Systems with YCSB", **SoCC 2010** + ([PDF](https://www.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf)). + Sections used above: §3.1-3.2 (tiers 1-2), §4.1 (the four + distributions), §4.2 and **Table 2** (workloads A-E), §5.3 (Gray's + algorithm, the clustering problem, 80% coverage, the oversized- + keyspace fix and 99.97%, `N + T × I + ε`), §6.4-6.5 (workload E; + F's only appearance), §7.1-7.2 (the unimplemented tiers 3-4). +- Gray, Sundaresan, Englert, Baclawski, Weinberger — "Quickly + Generating Billion-Record Synthetic Databases", **SIGMOD 1994**. + Reference [23] of the YCSB paper; the source of the constant-time + Zipfian inversion in Step 4. **Code** -- [go-ycsb](https://github.com/pingcap/go-ycsb) - `pkg/generator/zipfian.go`, `scrambled_zipfian.go`, - `workloads/workloada` — the Go port; structure mirrors the Java - original + +| File | Lines | What | +|---|---|---| +| go-ycsb `pkg/generator/zipfian.go` | 42-45 | `ZipfianConstant = 0.99` (the value is on 44) | +| go-ycsb `pkg/generator/zipfian.go` | 47-64 | doc comment; cites Gray et al.; "over a minute for 100 million objects" | +| go-ycsb `pkg/generator/zipfian.go` | 97-118 | precompute `zeta2Theta` (108), `alpha` (110), `zetan` (111), `eta` (113) | +| go-ycsb `pkg/generator/zipfian.go` | 125-133 | `zetaStatic` — the one O(n) sum | +| go-ycsb `pkg/generator/zipfian.go` | 136-149 | incremental zeta on growth; the warning on shrink | +| go-ycsb `pkg/generator/zipfian.go` | 154-160 | the two fast paths — and where `SetLastValue` is skipped | +| go-ycsb `pkg/generator/zipfian.go` | 162-163 | the general inversion, and the only `SetLastValue` | +| go-ycsb `pkg/generator/scrambled_zipfian.go` | 50-67 | hardcoded `zetan`, θ = 0.99, inner `itemCount = 10¹⁰` | +| go-ycsb `pkg/generator/scrambled_zipfian.go` | 70-76 | `min + Hash64(n) % itemCount` — the scatter | +| go-ycsb `pkg/util/hash.go` | 21-32 | FNV-1a 64 over big-endian bytes | +| go-ycsb `pkg/workload/core.go` | 655-678 | the six `requestdistribution` values; `zipfian` → Scrambled | +| go-ycsb `pkg/workload/core.go` | 660-665 | `N + T × I × 2` keyspace expansion | +| go-ycsb `pkg/client/client.go` | 97-111 | the intended schedule (102-103) and its discard (104-106) | +| go-ycsb `pkg/client/client.go` | 144-147 | throttle called *after* the operation | +| go-ycsb `pkg/client/dbwrapper.go` | 30-39 | `measure` — takes a start time, never an intended time | +| go-ycsb `pkg/client/dbwrapper.go` | 53-60 | `start := time.Now()` at issue time (54) | +| go-ycsb `workloads/workloada` | 23, 36 | comment says zipfian, property says uniform | +| go-ycsb `workloads/workloade` | 22, 40, 42 | same contradiction, plus `maxscanlength=1` | +| this repo `experiments/src/ycsb.rs` | 1-7 | the declared simplifications | +| this repo `experiments/src/ycsb.rs` | 61-68 | the six mixes | +| this repo `experiments/src/ycsb.rs` | 95-117 | the driver loop; the clock at 98; the 100-record scan at 110 | +| this repo `experiments/src/zipf.rs` | 1-12 | the stub's spec and its go-ycsb anchors | + +Pinned revision: pingcap/go-ycsb@f030f99 (regenerate the pin table +with `python3 tools/pin-table.py`). + +**Measurements** +- FINDINGS.md row 22 — the canonical headline: "YCSB-E's p999 is + **12.9 µs** against read-only's **4.0 µs**". Reproduce with + `./verify.sh 22`. +- `notes.md` — an earlier baseline run (M3 Pro): uniform-key + throughputs A 2.88, B 4.15, C 3.72, D 4.40, E 1.11, F 2.85 Mops/s. + Use ratios, not absolutes, when comparing across runs or machines. + +**Cross-topic** +- topic 34 — coordinated omission, measured: closed-loop p99 = 1.0 µs + against open-loop 90 ms, a 90,000× lie (FINDINGS.md row 34). Step 7 + cites it rather than re-deriving it. +- [reading-oltpbench-tpcc.md](reading-oltpbench-tpcc.md) — the other + contention shape, and the same "read the config, not the comments" + failure in TPC-C's think times. +- [reading-boncz-tpch.md](reading-boncz-tpch.md) — the analytical + counterpart: what a benchmark's *queries* encode, rather than what + its *keys* do. diff --git a/topics/23-fulltext/experiments/.gitignore b/topics/23-fulltext/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/23-fulltext/experiments/.gitignore +++ b/topics/23-fulltext/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/23-fulltext/notes.md b/topics/23-fulltext/notes.md index 27e86e8..d930843 100644 --- a/topics/23-fulltext/notes.md +++ b/topics/23-fulltext/notes.md @@ -8,6 +8,11 @@ df(t0)=99888 (in 99.9% of docs — "the"), df(t100)=8259, df(t10000)=83. ### BM25 top-10, exhaustive TAAT oracle +> An earlier run than [FINDINGS.md](../../FINDINGS.md) row 23, which +> spans **0.009 ms to 10.378 ms** over the same four queries. FINDINGS is +> canonical for the timings; the posting counts (272,310 and 159) are +> generator-determined and agree. Cite one run by name, never average. + | query | ms | postings walked | top1 score | |---|---|---|---| | common∧common [t0 t1 t5] | 8.75 | 272,310 | 0.612 | @@ -16,7 +21,7 @@ df(t0)=99888 (in 99.9% of docs — "the"), df(t100)=8259, df(t10000)=83. | rare∧rare [t9000 t15000] | 0.008 | 159 | 9.208 | The common∧rare row is the WAND poster child: 99,964 postings walked -but the rare term (df=83, idf≈9.0) contributes ~93% of the top-1 +but the rare term (df=83, idf≈7.1) contributes ~93% of the top-1 score — nearly all of t0's 99,888 postings are provably hopeless once the heap holds 10 docs that contain t12000. ~32 ns/posting for the oracle (hash accumulate dominates — topic 22's Q1 story again). diff --git a/topics/23-fulltext/reading-blockmax-wand.md b/topics/23-fulltext/reading-blockmax-wand.md index 32fc92e..9fc1c26 100644 --- a/topics/23-fulltext/reading-blockmax-wand.md +++ b/topics/23-fulltext/reading-blockmax-wand.md @@ -2,27 +2,36 @@ Top-k retrieval doesn't need to score every document — only the ones whose score *upper bound* beats the current k-th best. That -one observation (WAND, CIKM 2003) plus per-block score ceilings -(Ding & Suel, SIGIR 2011) is what our `wand::wand_topk` stub +one observation (WAND, Broder et al. CIKM 2003) plus per-block score +ceilings (Ding & Suel, SIGIR 2011) is what our `wand::wand_topk` stub implements. This chapter builds the algorithm in five steps — the threshold, the bounds, the pivot, the block refinement, and the -traps — so the two papers read as confirmations. Prereq: the BM25 -chapter's saturation ceiling; then read the original WAND paper's -§2 — it's 3 pages. +traps — so the two papers read as confirmations. + +Prereq: the BM25 chapter's saturation ceiling. Source pins: the code +anchors are tantivy at `7152d53`; the repo stub is +`experiments/src/wand.rs`. The original WAND paper (Broder et al., +CIKM 2003) is paywalled — if you cannot get it, Ding & Suel's §2 +Background re-derives its pivoting, and all algorithm anchors below +are to Ding & Suel, which I verified against the PDF. ## The problem in one sentence Our exhaustive scorer spends 6.34 ms on a common∧rare query -(100K postings) even though the rare term's idf ≈ 9 guarantees -almost none of the common term's postings can reach the top-10 — -block-max WAND returns the *identical* top-10 while fully scoring -under 25% of the docs (the stub's test demands it), and the papers -report 2.5–4× at TREC scale. +([t0, t12000], 99,964 postings; notes.md) even though the rare +term's idf ≈ 7.1 guarantees almost none of the common term's +postings can reach the top-10 — block-max WAND returns the +*identical* top-10 while fully scoring under 25% of the docs (the +stub's test demands it), and Ding & Suel report 2.8–3.0× over plain +WAND at TREC scale (Table 1, below). ## The concepts, step by step ### Step 1 — the threshold θ: top-k means most docs don't matter +> **In:** a stream of candidate docs and a request for the top k by BM25. +> **Out:** a rising threshold θ (the k-th best score so far) below which every doc can be discarded unscored — turning "what does d score?" into "can d beat θ?". + A top-k query keeps a min-heap of the k best scores seen so far (a heap whose root is the *smallest* of the k — the score to beat), and θ (theta) names that k-th best score. Once the heap is full, a @@ -30,31 +39,49 @@ doc scoring ≤ θ changes nothing — it is discarded on arrival. So the real question is never "what does doc d score?" but "can doc d possibly beat θ?" — and θ only rises as better docs arrive, so docs get *easier* to rule out as the query progresses. Exhaustive -scoring (the TAAT oracle) answers the first question 100K times; -everything below answers the second, usually without scoring. +scoring (the TAAT — term-at-a-time — oracle) answers the first +question 100K times; everything below answers the second, usually +without scoring. ### Step 2 — upper bounds make skipping safe +> **In:** BM25's per-term score ceiling from the previous chapter (idf·(K1+1)). +> **Out:** a whole-doc upper bound = sum of query terms' ceilings; any doc whose bound ≤ θ is skippable with zero risk to the exact top-k. + If you know a **ceiling** for each term — a value its BM25 contribution can never exceed for any doc — then the sum of the query terms' ceilings bounds any doc's total score, and a doc whose bound is ≤ θ can be skipped *with zero risk to correctness*. BM25 hands us the ceiling for free (previous chapter): tf saturates, so -score(t, d) ≤ idf(t)·(K1+1), computable at index time. Concretely -for our `[t0, t12000]` query: t0's ceiling ≈ 0.7·2.2 ≈ 1.5, t12000's -≈ 9·2.2 ≈ 20 — once θ passes 1.5, *no doc containing only t0 can -ever win*, and 100K postings become skippable in principle. The -magic word is **safe-to-k**: WAND returns the EXACT top-k, not an -approximation — correctness needs only that the bounds are true, -not tight. +`score(t, d) ≤ idf(t)·(K1+1)`, computable at index time. Worked for +our `[t0, t12000]` query using the repo's own idf (bm25.rs:15-16, +N=100000): + +``` +term df idf = ln(1+(N-df+0.5)/(df+0.5)) ceiling = idf·(K1+1), K1=1.2 +t0 99888 ln(1+0.00113) = 0.0011 0.0011·2.2 = 0.0025 +t12000 83 ln(1+1196.6) = 7.088 7.088·2.2 = 15.59 +``` + +So t0's *entire* contribution to any doc is at most 0.0025 — once +θ climbs past that (which the very first rare-term doc does), *no +doc containing only t0 can ever win*, and its 99,888 postings become +skippable in principle. The magic word is **safe**: WAND returns the +EXACT top-k, not an approximation — correctness needs only that the +bounds are true, not tight. (This is why the "idf ≈ 0.7 vs 9" that +an earlier draft of this guide quoted was not just imprecise but +understated the effect: t0's ceiling is ~0.0025, not ~1.5.) ### Step 3 — the pivot: turning bounds into a jump target -WAND runs doc-at-a-time: one cursor per term over its doc-sorted -posting list. Each round, sort cursors by their current doc id and -accumulate ceilings down the list until they exceed θ; the cursor -where that happens marks the **pivot** — the smallest doc id that -could possibly beat θ: +> **In:** one doc-sorted cursor per query term, and the current θ. +> **Out:** the pivot doc — the smallest doc id whose leading terms' ceilings sum past θ — and a decision: score it, or seek the trailing cursor forward over the provable-loser gap. + +WAND runs doc-at-a-time (DAAT): one cursor per term over its +doc-sorted posting list. Each round, sort cursors by their current +doc id and accumulate ceilings down the list until they exceed θ; +the cursor where that happens marks the **pivot** — the smallest doc +id that could possibly beat θ: ``` cursors sorted by current doc id; θ = current k-th best score @@ -79,7 +106,9 @@ happens) or one of them leaps forward over the dead zone. One round of the loop, in code: ```rust -// θ = current k-th best score; upper bounds make skipping SAFE +// ILLUSTRATION — one WAND round. You implement this in +// experiments/src/wand.rs:52 (wand_topk); tantivy's real, +// trap-fixed version is find_pivot_doc at block_wand_union.rs:16-43. fn wand_round(cursors: &mut [Cursor], theta: f32) -> Option { cursors.sort_by_key(|c| c.doc()); // by current doc id let mut ub = 0.0; @@ -106,31 +135,44 @@ round, versus decoding and scoring thousands of postings. ### Step 4 — block-max: per-block ceilings fix the pessimistic bound +> **In:** a term-level ceiling set by that term's single best doc — wildly loose everywhere else. +> **Out:** per-128-doc block ceilings that let a false-positive pivot be skipped by *moving* a block cursor (shallow) without *decoding* its 128 postings (deep). + Term-level max_score is one global ceiling — for a common term it's set by its single best doc, wildly pessimistic everywhere else (t0's one lucky tf=30 doc inflates the bound for all 100K -postings). Ding & Suel's 2011 fix: postings are already stored in -128-doc compressed blocks, so store max_score per block too, as -uncompressed metadata next to the compressed postings: +postings). Ding & Suel's 2011 fix (their **Block-Max Index**, §3): +postings are already stored in 128-doc compressed blocks, so keep a +per-block score ceiling next to each compressed block as +uncompressed metadata: - pivot found with term maxima as before (cheap, monotone); - then REFINE with the current blocks' maxima: if Σ block-max ≤ θ, - the pivot is a false positive — skip to - `min(block boundary) + 1` without decompressing anything (§4's - "shallow" vs "deep" pointer movement: moving a block cursor doesn't - decode the block). -- §5's numbers: ~2.5-4× over WAND at TREC scale, more at deeper k. + the pivot is a false positive — skip past the nearest block + boundary without decompressing anything. Ding & Suel §5 names the + two motions: a **deep pointer movement** decodes a block; a + **shallow** one (their `NextShallow`, §5) only reads block-boundary + metadata. The algorithm does shallow moves "instead of deep pointer + movements whenever possible" (§5). +- the payoff, measured (Table 2, TREC 2006): WAND evaluates 178,391 + docIDs per query; BMW evaluates 21,921 (≈8× fewer), at the cost of + 0.42M deep + 0.76M shallow pointer moves. The shallow/deep distinction is the engineering payload: a block cursor can *move* (shallow — just read skip metadata) without *decoding* (deep — decompress 128 postings), so false-positive pivots cost almost nothing. Our `BlockMeta { last_doc, max_score }` -in `index.rs` is exactly their metadata; tantivy's is -`postings/skip.rs:175` (`block_max_score`) + `:186` -(`last_doc_in_block`). +in `experiments/src/index.rs:26-30` stores the block ceiling +directly; tantivy instead stores the block's `(fieldnorm_id, +quantized tf)` and *recomputes* the ceiling on demand — +`SkipReader::block_max_score` (skip.rs:175-181) calls +`bm25_weight.score(...)`, and `last_doc_in_block()` is skip.rs:186-187. ### Step 5 — the traps (learned by others, cheaply) +> **In:** a working pivot loop and the wish to match the oracle's exact top-k. +> **Out:** four failure modes (empty-heap θ, livelock on false-positive pivots, score ties, and the metric mismatch) your `wand_topk` must avoid. + Four failure modes every WAND implementation rediscovers — check your `wand_topk` against each: @@ -140,32 +182,39 @@ your `wand_topk` against each: 2. When the block-max check fails, advance past `min(last_doc of the cursors' current blocks)` — advancing only to pivot_doc re-finds the same dead pivot forever (livelock). + tantivy's `block_max_was_too_low_advance_one_scorer` + (block_wand_union.rs:49) is exactly this fix. 3. Ties at the k-boundary: WAND may return a different doc with an EQUAL score — compare scores, not doc ids (our test does). 4. `docs_scored` counts full evaluations; postings_skipped counts - what you jumped — the paper's Table 4 metric is "docs evaluated", - make sure yours matches for comparability. + what you jumped — Ding & Suel's comparability metric is "evaluated + docIDs" (Table 2), make sure yours matches theirs. ## How to read the papers (with the concepts in hand) -Two papers, one evening, in order: +Two papers, one evening, in order (section numbers verified against +the Ding & Suel PDF): -- **Broder et al. (CIKM 2003), §2 first — 3 pages.** The pivot idea +- **Broder et al. (CIKM 2003) — the original WAND.** The pivot idea (Step 3) in its original two-level form: a cheap bound pass over - cursors, then full evaluation only at pivots. The rest of the - paper (their production context, approximate variants) is - optional. -- **Ding & Suel (SIGIR 2011).** §4 is the payload — block metadata - and shallow vs deep pointer movement (Step 4); §5's numbers set - your expectations (2.5–4× over WAND, more at deeper k). Skim - their list-caching and layout discussion. + cursors, then full evaluation only at pivots. Paywalled; if you + can't get it, read Ding & Suel **§2 Background**, which re-derives + the pivot mechanism before extending it. +- **Ding & Suel (SIGIR 2011).** **§3** proposes the Block-Max Index + (per-block ceilings); **§5 Block-Max WAND Algorithm** is the + payload — Algorithm 1, and shallow vs deep pointer movement + (`NextShallow`, `CheckBlockMax`) (Step 4); **§6 Experiments** has + the numbers (Table 1: BMW 27.9 ms vs WAND 77.6 ms on TREC 2006 = + 2.8×; 21.2 vs 64.4 on TREC 2005 = 3.0×; Table 2: evaluated docIDs). + (§4 is Related Work, not the algorithm — skip on a first pass.) - Then the shipped version, mapped: | paper concept | tantivy anchor | |---|---| -| pivot selection | `query/boolean_query/block_wand_union.rs:8-24` `find_pivot_doc` — walks scorers sorted by doc, accumulates max_weight until > threshold | -| block metadata | `postings/skip.rs:93` `SkipReader`, `:175/:186` | -| term upper bound | `Scorer::max_score` per term weight | +| pivot selection | `query/boolean_query/block_wand_union.rs:16-43` `find_pivot_doc` — walks scorers sorted by doc, accumulates `max_score` until `> threshold` | +| block metadata | `postings/skip.rs:93` `SkipReader`; block ceiling recomputed at `:175-181`, `last_doc_in_block` at `:186-187` | +| term upper bound | `Bm25Weight::max_score` per term (bm25.rs:184) | +| false-positive skip | `block_wand_union.rs:49` `block_max_was_too_low_advance_one_scorer` | | union top-k | `block_wand_union.rs` (OR queries), `block_wand_intersection.rs` (AND) | Compare tantivy's `find_pivot_doc` with your stub only *after* @@ -174,15 +223,16 @@ already fixed. ## Questions (answer in notes.md) -1. For our `[t0, t12000]` query (df 99888 vs 83, idf ≈ 0.7 vs 9): - after the heap fills with rare∧common docs, θ ≈ ? Can t0 alone - ever cross it? Predict wand's docs_scored (the test demands <25% - of 99964). +1. For our `[t0, t12000]` query (df 99888 vs 83, idf ≈ 0.0011 vs + 7.09): after the heap fills with rare∧common docs, θ ≈ ? Can t0 + alone ever cross it? Predict wand's docs_scored (the test demands + <25% of 99964). 2. Why does block-max help MOST on common terms? Relate to the variance of per-block maxima under Zipf tf distributions. 3. The paper stores block maxima uncompressed. At 128 docs/block, what's the metadata overhead per posting, and why is quantizing - maxima to u8 safe but quantizing DOWN unsafe? + maxima UP (tantivy's `encode_block_wand_max_tf`) safe but + quantizing DOWN unsafe? 4. Block-max WAND is exact top-k. What changes if the scorer adds M14's vector similarity (no static bound)? Sketch M23's hybrid: WAND for BM25 candidates + RRF, vs a fused traversal. @@ -192,26 +242,81 @@ already fixed. ## Done when +Answer each before unfolding it. + - [ ] You can explain the threshold θ and why top-k means most documents are provably irrelevant. +
θ and the discard rule + + θ is the k-th best score seen so far (root of a size-k min-heap). + Once the heap is full, any doc scoring ≤ θ cannot enter it and is + discarded unscored. θ only rises, so more docs become skippable as + the query proceeds. + +
- [ ] You can compute a pivot from term upper bounds and say what makes the jump safe. +
the pivot and its guarantee + + Sort cursors by doc id; accumulate per-term ceilings; the first + cursor whose running sum exceeds θ marks pivot_doc. Every doc below + it can contain only the preceding terms, whose ceilings sum to ≤ θ + — so skipping to pivot_doc cannot drop a true top-k doc. Safe + because the bounds are true, not because they are tight. + +
- [ ] You can explain what per-block ceilings fix about the global upper bound. -- [ ] You can say why block-max helps most on common terms, and check it against this topic's measured oracle: 10.378 ms and 272 310 postings for `t0∧t1∧t5` against 0.009 ms and 159 postings for two rare terms. +
looseness of the global max + + A term's global ceiling is set by its single best doc and is wildly + loose elsewhere. Per-128-doc block ceilings are tight locally, so + a pivot whose block maxima sum ≤ θ is exposed as a false positive + and skipped via a shallow (metadata-only) move — no 128-posting + decode. + +
+- [ ] You can say why block-max helps most on common terms, and check it against this topic's measured oracle: 10.378 ms and 272,310 postings for `t0∧t1∧t5` against 0.009 ms and 159 postings for two rare terms. +
variance of block maxima + + Common terms have huge, mostly-hopeless posting lists whose + per-block maxima vary a lot under Zipf tf; block-max prunes the + low-max blocks that the single global ceiling could never rule out. + The oracle figures (FINDINGS row 23) show the cost is all in the + dense lists: 272,310 postings / 10.378 ms vs 159 / 0.009 ms — term + rarity, not query complexity. + +
- [ ] You can state what breaks if the scorer stops having a ceiling. +
no bound, no skip + + Every skip in WAND rests on a true static upper bound. A scorer + without one (a neural/vector similarity, M14) can't be bounded, so + no doc can be safely skipped — it must run as a second stage over + WAND's BM25 candidates (M23's hybrid). + +
- [ ] You wrote answers to all five questions in notes.md. +
check + + Five answers in notes.md, each tied to a measured repo number or a + Ding & Suel section/table — not to the vague "2.5–4×" folklore. + +
## References **Papers** - Broder, Carmel, Herscovici, Soffer, Zien — "Efficient Query - Evaluation using a Two-Level Retrieval Process" (CIKM 2003) — - read §2 first (3 pages): the pivot idea + Evaluation using a Two-Level Retrieval Process" (CIKM 2003) — the + original WAND pivot idea (paywalled; re-derived in Ding & Suel §2) - Ding, Suel — "Faster Top-k Document Retrieval Using Block-Max - Indexes" (SIGIR 2011) — §4 (shallow vs deep pointer movement) and - §5's numbers + Indexes" (SIGIR 2011) — §3 the Block-Max Index, §5 the BMW + algorithm (shallow vs deep pointer movement), §6 Tables 1–2 the + numbers **Code** -- [tantivy](https://github.com/quickwit-oss/tantivy) - `src/query/boolean_query/block_wand_union.rs` (:8-24 - `find_pivot_doc`), `block_wand_intersection.rs`, - `src/postings/skip.rs` (:175 `block_max_score`, :186 - `last_doc_in_block`) — the paper, shipped +- [tantivy](https://github.com/quickwit-oss/tantivy) `@7152d53` — + `src/query/boolean_query/block_wand_union.rs` (`find_pivot_doc` + :16-43, `block_max_was_too_low_advance_one_scorer` :49), + `src/postings/skip.rs` (`SkipReader` :93, `block_max_score` + :175-181, `last_doc_in_block` :186-187) — the paper, shipped +- This repo — `experiments/src/wand.rs` (`wand_topk` stub :52), + `experiments/src/index.rs` (`BlockMeta` :26-30) diff --git a/topics/23-fulltext/reading-bm25.md b/topics/23-fulltext/reading-bm25.md index a336447..a7847d8 100644 --- a/topics/23-fulltext/reading-bm25.md +++ b/topics/23-fulltext/reading-bm25.md @@ -11,6 +11,11 @@ saturation, length normalization — then maps every piece to a line of tantivy and explains why the next chapter (WAND) depends on one property of this formula. +Source pins: the monograph is Robertson & Zaragoza, *The +Probabilistic Relevance Framework: BM25 and Beyond*, Foundations and +Trends in IR 3(4), 2009 (equation and section numbers below are from +it); code anchors are tantivy at `7152d53`. + ## The problem in one sentence Given a query and 100K candidate documents, produce one number per @@ -22,10 +27,13 @@ inside the tightest loop the search engine has. ### Step 1 — ranking as probability: the one principled starting point -The **probabilistic ranking principle** says: sort documents by -P(relevant | document) — the probability a user with this query -would judge the document relevant — and no other ordering does -better on average. Since only the *order* matters, any monotonic +> **In:** a query, a document, and the wish to order documents by usefulness. +> **Out:** the odds ratio P(rel|doc)/P(irrel|doc) as the thing to sort by, and the fact that any monotonic transform (logs, sums) ranks identically. + +The **probability ranking principle** (monograph §2.3) says: sort +documents by P(relevant | document) — the probability a user with +this query would judge the document relevant — and no other ordering +does better on average. Since only the *order* matters, any monotonic transform is equally good, and the odds ratio P(relevant|doc)/P(irrelevant|doc) turns products of per-term probabilities into sums of logs. Everything in BM25 is this ratio @@ -33,78 +41,116 @@ under successively weaker simplifying assumptions. The whole ladder, which Steps 2–4 climb rung by rung: ``` - binary independence model (§3) - terms present/absent, independent ⇒ score = Σ log odds per term - └─ with no relevance info ⇒ the idf shape: log (N - df + 0.5)/(df + 0.5) - + term frequency via 2-Poisson "eliteness" (§3.3) + binary independence model (§3.1) + terms present/absent, independent => score = Σ log-odds per term + └─ no relevance info => the idf shape (Eq 3.3): log (N - df + 0.5)/(df + 0.5) + + term frequency via 2-Poisson "eliteness" (§3.4.1) docs are elite/non-elite for a term; tf is a noisy signal of eliteness - ⇒ tf weight must SATURATE: tf·(k1+1)/(tf + k1) ← not log(tf), not raw tf - + document length (§3.4) - long docs: more of everything ⇒ normalize tf by len/avg_len, - but only partially (verbosity vs scope hypothesis) ⇒ the B knob - = BM25 (§3.5): - Σ idf(t) · tf·(k1+1) / (tf + k1·(1 - b + b·len/avg_len)) + => tf weight must SATURATE: tf / (tf + k1) ← not log(tf), not raw tf + + document length (§3.4.5) + long docs: more of everything => normalize tf by len/avg_len, + but only partially (verbosity vs scope hypothesis) => the B knob (Eq 3.12) + = BM25 (Eq 3.15): + Σ idf(t) · tf / (tf + k1·(1 - b + b·len/avg_len)) + (tantivy/Lucene multiply the numerator by (k1+1); §3.5.1 — see Step 3) ``` ### Step 2 — the binary independence model, and where idf comes from +> **In:** the log-odds sum from Step 1 and the usual case of *no* relevance judgments. +> **Out:** the per-term weight collapses to a function of one statistic, df, giving the idf shape — big for rare terms, ~0 for ubiquitous ones. + Assume each term is merely present or absent in a doc (binary), and -terms are independent of each other. Then the log-odds ratio -decomposes into a per-term weight summed over query terms present -in the doc. With *no* relevance judgments available (the usual -case), the weight collapses to a function of one statistic — -**df** (document frequency: how many of the N docs contain the -term): +terms are independent of each other — the **binary independence +model** (monograph §3.1). Then the log-odds ratio decomposes into a +per-term weight summed over query terms present in the doc. With *no* +relevance judgments available (the usual case), setting R = r = 0 in +the Robertson-Spärck Jones weight collapses it to a function of one +statistic — **df** (document frequency: how many of the N docs +contain the term) — which the monograph calls a close approximation +to classical idf (Eq 3.3): ``` -idf(t) = log (N − df + 0.5) / (df + 0.5) +idf(t) = log (N − df + 0.5) / (df + 0.5) (monograph Eq 3.3) ``` This is **idf** (inverse document frequency): rare terms get big -weights, terms in half the corpus get ~0. Concretely, in our 100K -corpus: df=159 → idf ≈ 6.4; df=100K → idf ≈ 0. The +0.5s are -smoothing (a Jeffreys prior) so df=0 and df=N don't produce -infinities (question 2). Cost of the model's honesty: binary -presence ignores that a doc mentioning `fox` 12 times is more -about foxes than one mentioning it once — Step 3's job. +weights, terms in half the corpus get ~0. Worked on our 100K corpus: + +``` +df = 159: (100000 − 159 + 0.5)/(159 + 0.5) = 99841.5/159.5 = 625.97 + log(625.97) = 6.44 → big weight +df = 100000: (100000 − 100000 + 0.5)/(100000 + 0.5) = 0.5/100000.5 + log(5.0e-6) = −12.2 → NEGATIVE (see below) +``` + +The +0.5s are smoothing (a Jeffreys prior) so df=0 and df=N don't +produce infinities (question 2). Note the plain Eq 3.3 goes *negative* +for a term in nearly every document — which is why Lucene/tantivy add +a +1 inside the log (Step 5): `ln(1 + 625.97) = 6.44` for df=159 (same +to two places), but `ln(1 + 5.0e-6) ≈ 0.000005` for df=100000, never +below zero. Cost of the model's honesty: binary presence ignores that +a doc mentioning `fox` 12 times is more about foxes than one +mentioning it once — Step 3's job. ### Step 3 — term frequency must saturate: the 2-Poisson argument +> **In:** the observation that tf should raise the score, but a doc repeating one word 500× is not 500× more relevant. +> **Out:** a saturating tf weight tf/(tf+k1) that approaches a ceiling, with k1 tuning how fast — the ceiling is what WAND later exploits. + **tf** (term frequency: occurrences of the term in this doc) should raise the score — but not linearly. The 2-Poisson **eliteness** -model says a doc either *is about* the term ("elite") or isn't, and -tf is only a noisy signal of that hidden bit: going 0→3 occurrences -is strong evidence of eliteness, 50→53 is nothing. Working the -model through yields a weight that **saturates**: +model (monograph §3.4.1) says a doc either *is about* the term +("elite") or isn't, and tf is only a noisy signal of that hidden bit: +going 0→3 occurrences is strong evidence of eliteness, 50→53 is +nothing. Working the model through (Eq 3.11) yields a weight that +**saturates**: ``` -tf·(k1+1) / (tf + k1) → k1+1 as tf → ∞ +raw: tf / (tf + k1) → 1 as tf → ∞ (Eq 3.11, times wRSJ) +tantivy/Lucene: tf·(k1+1) / (tf + k1) → k1+1 (the §3.5.1 variant) ``` -`K1` (≈1.2 by default) sets how fast the ceiling is approached: at -K1=1.2, tf=1 already gives 1.0 of the max 2.2; tf=11 gives ~90% -(question 1). What breaks without it: a doc repeating `quick` 500× -beats a doc with `quick fox` — the spam magnet. Neither raw tf nor -log(tf) has the bounded ceiling; the *bound* is what WAND will -exploit (Step 6). +The `(k1+1)` numerator is the monograph's §3.5.1 variant: "the same +for all terms, and therefore does not affect the ranking" — it just +makes a single-occurrence term score the same as under the bare RSJ +weight. tantivy uses it (Step 5). **K1** (=1.2 in tantivy) sets how +fast the ceiling is approached. Worked at K1=1.2, len=avg (so the +denominator's length term is just k1): + +``` +tf-weight = tf·(k1+1)/(tf+k1), ceiling = k1+1 = 2.2 + tf = 1: 1·2.2/(1+1.2) = 2.2/2.2 = 1.00 → 1.00 of 2.2 = 45% + tf = 11: 11·2.2/(11+1.2)= 24.2/12.2= 1.98 → 1.98 of 2.2 = 90% +``` + +So the first occurrence already buys 45% of the ceiling and the +eleventh only reaches 90% (question 1). What breaks without +saturation: a doc repeating `quick` 500× beats a doc with `quick fox` +— the spam magnet. Neither raw tf nor log(tf) has the bounded +ceiling; the *bound* is what WAND will exploit (Step 6). ### Step 4 — document length: normalize, but only partly +> **In:** long documents carry more of every term, inflating tf regardless of relevance. +> **Out:** a soft length-normalization knob b ∈ [0,1] (Eq 3.12) that divides tf by (1−b+b·len/avg) before saturation — b=0 off, b=1 full. + Long documents have more of every term, so tf must be discounted by doc length — but *how much* depends on *why* the doc is long: pure verbosity (same content, more words → fully normalize) or wider scope (genuinely more topics → don't). Truth is in between, so BM25 -interpolates with knob `b ∈ [0,1]`, replacing K1 in the denominator -with: +interpolates with knob `b ∈ [0,1]` (monograph §3.4.5, Eq 3.12), +scaling k1 in the denominator by the soft-normalization factor: ``` -k1 · (1 − b + b · len/avg_len) b = 0.75 by default +B = (1 − b + b · len/avg_len) (Eq 3.12; b = 0.75 in tantivy) +denominator uses k1·B in place of k1 ``` What breaks at the extremes: b=0 (no normalization) → encyclopedic docs win everything; b=1 (full) → long docs can never win, even legitimately comprehensive ones. Assembling Steps 2–4 gives BM25 -(§3.5 in the ladder above) — and note each piece failed *toward* a +(monograph Eq 3.15) — and note each piece failed *toward* a concrete pathology: - no saturation → keyword-stuffing spam wins; @@ -113,42 +159,58 @@ concrete pathology: ### Step 5 — in code: precompute everything, one multiply-add per posting +> **In:** the assembled BM25 formula and a query-time budget of one arithmetic op per posting. +> **Out:** idf is per-term (from the dictionary), the length term is a 256-entry table keyed by a 1-byte fieldnorm, and (k1+1) is folded into a per-term `weight` — leaving `weight · tf/(tf+norm)` per posting. + At query time, idf is per-term (known from the dictionary before any posting is read) and the length-norm denominator is per-doc — both precomputable, leaving one multiply-add per posting. tantivy's -whole scorer (`query/bm25.rs`): +scorer, quoted (note it splits the textbook single fraction: `(1+K1)` +is folded into `weight` at construction, and the hot path multiplies +that by `tf/(tf+norm)`): ```rust -const K1: f32 = 1.2; -const B: f32 = 0.75; - -fn idf(n_docs: f32, df: f32) -> f32 { - ((n_docs - df + 0.5) / (df + 0.5) + 1.0).ln() // +1: Lucene's tweak, -} // never negative at df > N/2 - -fn bm25(idf: f32, tf: f32, len: f32, avg_len: f32) -> f32 { - let norm = K1 * (1.0 - B + B * len / avg_len); // Lucene: a 256-entry - idf * (tf * (K1 + 1.0)) / (tf + norm) // table, len as u8 -} -// tf → ∞ ⇒ score → idf·(K1+1): the saturation ceiling that makes -// WAND's per-term upper bounds possible (next chapter) +// tantivy src/query/bm25.rs:8-9, 52-60 and 158-192 (elided) + 8 const K1: Score = 1.2; + 9 const B: Score = 0.75; + 52 pub(crate) fn idf(doc_freq: u64, doc_count: u64) -> Score { + 53 assert!(doc_count >= doc_freq, "{doc_count} >= {doc_freq}"); + 54 let x = ((doc_count - doc_freq) as Score + 0.5) / (doc_freq as Score + 0.5); + 55 (1.0 + x).ln() // +1: Lucene tweak, never < 0 + 56 } + 58 fn cached_tf_component(fieldnorm: u32, average_fieldnorm: Score) -> Score { + 59 K1 * (1.0 - B + B * fieldnorm as Score / average_fieldnorm) // per-fieldnorm norm + 60 } + 159 let weight = idf_explain.value() * (1.0 + K1); // fold (k1+1) into weight + 179 pub fn score(&self, fieldnorm_id: u8, term_freq: u32) -> Score { + 180 self.weight * self.tf_factor(fieldnorm_id, term_freq) // one mul per posting + 181 } + 189 pub(crate) fn tf_factor(&self, fieldnorm_id: u8, term_freq: u32) -> Score { + 190 let term_freq = term_freq as Score; + 191 let norm = self.cache[fieldnorm_id as usize]; // 256-entry table lookup + 192 term_freq / (term_freq + norm) + 193 } ``` | formula piece | anchor | |---|---| -| K1=1.2, B=0.75 (the paper's "reasonable defaults", §4.2) | bm25.rs:8-9 | -| idf with +1 under the ln (Lucene tweak: never negative when df > N/2) | bm25.rs:52 | -| `K1 * (1 - B + B * fieldnorm / average_fieldnorm)` precomputed per fieldnorm byte | bm25.rs:59 | -| fieldnorm quantized to 1 byte, 256-entry cache table | fieldnorm/ + the `cache` in bm25.rs | +| K1=1.2, B=0.75 — tantivy/Lucene convention, *not* stated by the monograph (which gives no defaults, only the §3.5 range 1.2 N/2); the bare monograph form is Eq 3.3 | bm25.rs:52-56 | +| `K1·(1 − B + B·fieldnorm/average_fieldnorm)` precomputed per fieldnorm byte | bm25.rs:58-60 | +| fieldnorm quantized to 1 byte → a 256-entry cache table (`compute_tf_cache`) | bm25.rs:62-68 | +| `(k1+1)` folded into `weight`; hot path is `weight · tf/(tf+norm)` | bm25.rs:159, :180, :189-192 | Lucene's extra trick: doc length (**fieldnorm**) is quantized to a u8 (lossy!), so the entire length-normalization term becomes a -256-entry lookup table. Our `bm25.rs` keeps exact lengths; the -experiments' block maxima would be *slightly* different under -quantization (question 4). +256-entry lookup table (`compute_tf_cache`, bm25.rs:62-68). Our +`bm25.rs` keeps exact lengths; the experiments' block maxima would be +*slightly* different under quantization (question 4). ### Step 6 — why WAND loves BM25: the score has a ceiling +> **In:** the saturating tf weight (bounded by k1+1) and a per-term idf. +> **Out:** a static, index-time per-term ceiling idf·(k1+1) — the monotone upper bound the next chapter's skipping depends on. + Because tf saturates at (K1+1) and fieldnorm has a minimum, every term's contribution is bounded for ALL docs: @@ -156,62 +218,126 @@ term's contribution is bounded for ALL docs: score(t, d) ≤ idf(t) · (K1 + 1) ``` -— a static per-term ceiling, computable at index time, refinable -per 128-doc block. The next chapter's entire algorithm (skip every -doc whose summed ceilings can't beat the current top-k) rests on -this monotone bound existing. Learned/neural scorers without such -bounds lose it — which is why neural rerankers run AFTER a -BM25/WAND first stage, never instead of it. +tantivy computes exactly this ceiling in `max_score` (bm25.rs:184-186: +`self.score(255u8, 2_013_265_944)` — max fieldnorm id, saturating tf) +— a static per-term ceiling, computable at index time, refinable per +128-doc block. The next chapter's entire algorithm (skip every doc +whose summed ceilings can't beat the current top-k) rests on this +monotone bound existing. Learned/neural scorers without such bounds +lose it — which is why neural rerankers run AFTER a BM25/WAND first +stage, never instead of it. ## How to read the paper (with the concepts in hand) -The 2009 monograph is ~90 pages; you need two sections: - -- **§2** Background/notation — skim to anchor the probabilistic - ranking principle (Step 1). -- **§3 — read carefully.** The derivation ladder: §3.2 - Robertson-Spärck Jones weights and the no-relevance-info idf - (Step 2), §3.3 eliteness and saturation (Step 3), §3.4 length - normalization (Step 4), §3.5 the assembled BM25. At each rung ask - the guide's question: what breaks if this rung is dropped? -- **§4.2** — where K1=1.2, b=0.75 come from (grid search over TREC - collections; "reasonable defaults", not laws). -- The rest (BM25F for fields, relevance feedback) — skim; return - for M23's per-field weighting if needed. +The 2009 monograph is ~60 pages; you need three sections, and it is +worth knowing where each rung actually lives (the subsection numbers +below are verified against the monograph's table of contents): + +- **§2.3** The Probability Ranking Principle — the probabilistic + starting point (Step 1). +- **§3.1 The Binary Independence Model** — read carefully: the RSJ + weight and, with no relevance info, the idf of Eq 3.3 (Step 2). + (§3.2 is Relevance Feedback and §3.3 Blind Feedback — skip on a + first pass; they are *not* the tf/length rungs.) +- **§3.4 The Eliteness Model and BM25** — the heart. §3.4.1 the + 2-Poisson/eliteness argument and saturation (Step 3), §3.4.5 + Document Length and the B factor (Step 4), Eq 3.15 the assembled + classic BM25. §3.5.1 lists the `(k1+1)`-numerator variant tantivy + uses. +- **§3.5 / §5** — parameters: the monograph gives *no* prescribed + defaults ("the model provides no guidance"), only the empirical + range "0.5 < b < 0.8 and 1.2 < k1 < 2 are reasonably good" (§3.5); + §5 is parameter optimisation. The specific 1.2/0.75 are Lucene's + choice, not the paper's. (§4.2 is "The Unified Model" — a + comparison, *not* where the constants come from.) +- The rest (BM25F for fields §3.6, relevance feedback §3.2) — skim; + return for M23's per-field weighting if needed. ## Questions (answer in notes.md) 1. Derive the tf-saturation limit: as tf→∞ the weight → K1+1. At - K1=1.2, what tf reaches 90% of the ceiling (len=avg)? What does - that say about keyword stuffing? + K1=1.2, len=avg, what tf reaches 90% of the ceiling (the text + worked it to ≈11)? What does that say about keyword stuffing? 2. The +0.5s in idf are a smoothing (Jeffreys prior). What happens - at df=0 and df=N without them? + at df=0 and df=N without them, and separately, why does Lucene's + +1-under-the-ln matter only near df=N? 3. b=0.75: our corpus has uniform lengths 50-150. Predict how much scores change b=0.75 → b=0 here vs on a corpus of tweets+books. 4. Lucene's 1-byte fieldnorm: worst-case relative score error vs exact lengths? Why is this fine for ranking but would corrupt our oracle-equality test? -5. RSJ weights need relevance judgments (§3.2); idf is the +5. RSJ weights need relevance judgments (§3.1/§3.2); idf is the no-information special case. Where would M23 get click/edge feedback to use the full RSJ weight, and is it worth it? ## Done when +Answer each before unfolding it. + - [ ] You can explain where idf comes from, rather than asserting it. +
the derivation, not the formula + + From the binary independence model (§3.1): sum per-term log-odds of + relevance; with no relevance judgments, set R=r=0, and the RSJ + weight reduces to the idf of Eq 3.3, `log((N−df+0.5)/(df+0.5))`. It + is the no-relevance-information special case of a probabilistic term + weight, not an axiom. + +
- [ ] You can derive the tf saturation limit and say what it approaches as tf grows. +
the ceiling + + `tf·(k1+1)/(tf+k1) → k1+1` as tf→∞ (the tantivy/Lucene §3.5.1 form; + the bare Eq 3.11 form `tf/(tf+k1) → 1`). At k1=1.2 the ceiling is + 2.2; tf=1 gives 1.0 (45%), tf=11 ≈ 1.98 (90%). + +
- [ ] You can explain what b controls and predict its effect on a corpus with near-uniform lengths. +
soft length normalization + + b scales how much tf is divided by len/avg_len (Eq 3.12: factor + `1−b+b·len/avg_len`). On near-uniform lengths (our 50–150 corpus) + len/avg≈1 so the factor ≈1 for any b — b barely moves scores. On + tweets+books the same b swings scores hard. + +
- [ ] You can say why WAND needs BM25's score ceiling and what a scorer without one costs. +
the monotone bound + + WAND skips any doc whose summed per-term ceilings can't beat θ. + BM25's ceiling is `idf·(k1+1)` (tantivy `max_score`, bm25.rs:184). + A scorer without a static upper bound (a neural reranker) can't be + skipped safely, so it must run as a second stage over WAND's + candidates. + +
- [ ] You can state what the +0.5 smoothing terms are doing. +
Jeffreys prior + + They keep `(N−df+0.5)/(df+0.5)` finite and defined at df=0 and df=N + — a Jeffreys (½-count) prior on the presence/absence probabilities, + so no term produces ±∞. + +
- [ ] You wrote answers to all five questions in notes.md. +
check + + Five answers in notes.md, each tied to an equation/section of the + monograph or a line of bm25.rs — not folklore. + +
## References **Papers** - Robertson, Zaragoza — "The Probabilistic Relevance Framework: - BM25 and Beyond" (Foundations and Trends in IR 2009) — §3 is the - derivation ladder; §4.2 the default constants + BM25 and Beyond" (Foundations and Trends in IR 3(4), 2009) — §2.3 + the ranking principle, §3.1 the BIM and idf (Eq 3.3), §3.4 the + eliteness model, saturation and length (Eq 3.15), §3.5.1 the + `(k1+1)` variant, §3.5 the empirical k1/b range **Code** -- [tantivy](https://github.com/quickwit-oss/tantivy) - `src/query/bm25.rs` — K1/B at :8-9, idf at :52, the precomputed - fieldnorm table at :59 +- [tantivy](https://github.com/quickwit-oss/tantivy) `@7152d53` + `src/query/bm25.rs` — K1/B at :8-9, idf at :52-56, the precomputed + fieldnorm table at :58-68, `weight`/`score`/`tf_factor` at + :159/:180/:189-192, the WAND ceiling `max_score` at :184-186 diff --git a/topics/23-fulltext/reading-redisearch.md b/topics/23-fulltext/reading-redisearch.md index adf0d53..698eab4 100644 --- a/topics/23-fulltext/reading-redisearch.md +++ b/topics/23-fulltext/reading-redisearch.md @@ -11,6 +11,9 @@ code, this chapter builds the design one constraint at a time — every delta from tantivy falls out of "updates must be cheap NOW" — then hands you the anchors. +Source pin: every anchor is RediSearch at `87276ca`, re-verified with +`tools/pinned-source.py`; all paths are under `src/redisearch_rs/`. + ## The problem in one sentence tantivy absorbs one new document by buffering it and eventually @@ -24,6 +27,9 @@ the fallout of that requirement. ### Step 1 — the constraint: mutable NOW, or nothing +> **In:** a write command inside Redis's single-threaded command loop. +> **Out:** the document is queryable when the command returns — which rules out immutable-segments + background merge, and forces one mutable posting list per term. + A Redis module runs inside Redis's (mostly) single-threaded command loop: no fleet of merge threads, no "visible after the next flush" — a write command returns and the data is queryable. That kills the @@ -36,82 +42,113 @@ metadata, cursor-invalidation protocols — is the running theme. ### Step 2 — the structure: chained growable blocks per term -Each term's index (`core.rs:30`, `InvertedIndex`) is a -`ThinVec` plus counters (`n_unique_docs`), flags, a -`gc_marker: AtomicU32`, and a `unique_id`. An `IndexBlock` -(`core.rs:75`) is `{ first_doc_id, last_doc_id, num_entries: u16, -buffer: Vec }` — a growable byte buffer of varint-encoded -entries, chained one after another. Contrast tantivy: blocks here -are **variable-length and append-tail-mutable**, not fixed 128-wide -bitpacked — because an append must be O(1) bytes written, not a -block re-pack. The block chain still gives coarse skipping -(`first_doc_id`/`last_doc_id` per block), which is what a mutable -index can afford instead of skip files. +> **In:** the "append must be cheap" constraint from Step 1. +> **Out:** each term is a `ThinVec` of variable-length `IndexBlock`s, each an append-tail-mutable byte buffer — coarse per-block skipping, no fixed-width repack. + +Each term's index (`inverted_index/src/index/core.rs:30`, +`InvertedIndex`) is a `ThinVec` plus counters +(`n_unique_docs`), flags, a `gc_marker: AtomicU32`, and a +`unique_id`. An `IndexBlock` (`core.rs:75`) is `{ first_doc_id, +last_doc_id, num_entries: u16, buffer: Vec }` — a growable byte +buffer of varint-encoded entries, chained one after another. +Contrast tantivy: blocks here are **variable-length and +append-tail-mutable**, not fixed 128-wide bitpacked — because an +append must be O(1) bytes written, not a block re-pack. The block +chain still gives coarse skipping (`first_doc_id`/`last_doc_id` per +block), which is what a mutable index can afford instead of skip +files. ### Step 3 — the write path: varint deltas, new block on overflow -Appending a posting means varint-encoding (a byte-at-a-time +> **In:** a new `(doc_id, record)` for a term whose last block ends at some doc. +> **Out:** the delta `doc_id − delta_base` varint-appended to that block — unless the codec can't represent the delta, which chains a fresh block starting at delta 0. + +Appending a posting means **varint**-encoding (a byte-at-a-time variable-length integer encoding — small deltas take 1 byte) the -delta from the block's last doc id into the last block's buffer. -One edge case drives the block-chaining: a delta too large for the -codec's representable range starts a fresh block at delta 0 -(`core.rs:229`, the `IdDelta::from_u64` → None path, -codec/mod.rs:28-44): +delta from the block's `delta_base` (usually its last doc id) into +the last block's buffer. One edge case drives the block-chaining: a +delta too large for the codec's representable range starts a fresh +block at delta 0. The real path, quoted: ```rust -// append one posting: varint-encode the delta into the last block; -// a delta the codec can't represent starts a NEW block at delta 0 -fn add(&mut self, doc_id: u64, rec: &Record) { - let block = self.blocks.last_mut().unwrap(); - match E::delta(doc_id, block) { // None ⇒ overflow for this codec - Some(delta) => { - E::write(&mut block.buffer, rec, delta); // byte-at-a-time varint - block.last_doc_id = doc_id; - block.num_entries += 1; - } - None => { - self.blocks.push(IndexBlock::new(doc_id)); // chain a fresh block - self.add::(doc_id, rec); // — simple, robust - } - } - self.n_unique_docs += 1; -} +// RediSearch src/redisearch_rs/inverted_index/src/index/core.rs:195-243 (elided) + 195 pub fn add_record(&mut self, record: &RSIndexResult) -> std::io::Result { + 196 let doc_id = record.doc_id; + 216 let mut block = self.take_block(doc_id, same_doc); // last block, or a fresh one + 219 let delta_base = E::delta_base(&block); // defaults to block.last_doc_id + 224 let delta = doc_id.wrapping_sub(delta_base); + 226 let delta = match E::Delta::from_u64(delta) { // None ⇒ too big for this codec + 227 Some(delta) => delta, + 228 None => { // start a NEW block at delta 0 + 231 let new_block = IndexBlock::new(doc_id); + 234 mem_growth += self.add_block(block); + 235 block = new_block; + 237 E::Delta::zero() + 238 } + 239 }; + 242 let writer = block.writer(); + 243 let _bytes_written = E::encode(writer, delta, record)?; // codec writes the varint ``` -Simple and robust — and the cost is exactly topic 17's lesson: the -branchy per-byte varint decode loop caps read-side GB/s, versus -tantivy's branchless 128-at-a-time SIMD unpack. Cheap writes were -bought with slower scans. +The `from_u64 → None` overflow arm lives in the codec's `IdDelta` +trait (`codec/mod.rs:36`, trait at :29-40; the comment at :33-35 +notes None means "new block per doc"). Simple and robust — and the +cost is exactly topic 17's lesson worked on our densest list +(`t0`, delta ≈ 1): + +``` +varint: delta 1 → 1 byte = 8 bits / posting +bitpack: block max delta 1 → 1 bit / posting (tantivy) + → varint is ~8× larger on a dense list, and its per-byte + decode loop is branchy where bitpacking is branch-free SIMD +``` -### Step 4 — the codec ladder: one trait, eleven encoders, chosen at compile time +Cheap writes were bought with slower, bulkier scans (question 4). -What a posting *carries* (Zobel-Moffat's granularity ladder: ids → -frequencies → fields → positions) is a codec choice: `trait -Encoder` (`codec/mod.rs:53` — `write(record, delta)`, -`delta_base(block)`) has eleven implementations in `codec/` — -`doc_ids_only` / `raw_doc_ids_only` / `freqs_only` / `freqs_fields` -/ `fields_offsets` / `full` / `numeric` … — the granularity ladder -as a directory listing, over one shared varint wire format -(`varint/src/lib.rs:98`, `VarintEncode`). +### Step 4 — the codec ladder: one trait, ten codecs, chosen at compile time + +> **In:** the question "what does a posting carry — just ids? +freqs? +fields? +positions?". +> **Out:** that granularity is a codec module implementing one `Encoder` trait, selected as a *type parameter* so the per-record branch is monomorphized away. + +What a posting *carries* (Zobel & Moffat's granularity ladder: ids → +frequencies → fields → positions) is a codec choice: `trait Encoder` +(`codec/mod.rs:53`) declares `encode(writer, delta, record)` (:74) +and `delta_base(block)` (:81, defaulting to the block's last doc id +:82), with `RECOMMENDED_BLOCK_ENTRIES: u16 = 100` (:70). There are +**ten codec modules** in `codec/` (`codec/mod.rs:10-19`): +`doc_ids_only`, `raw_doc_ids_only`, `freqs_only`, `freqs_fields`, +`freqs_offsets`, `fields_only`, `fields_offsets`, `offsets_only`, +`full`, `numeric` — the granularity ladder as a directory listing, +over one shared varint wire format (`varint/src/lib.rs:98`, +`VarintEncode`; `write_as_varint` :101). Each codec may override the +block-size policy: `doc_ids_only` sets `RECOMMENDED_BLOCK_ENTRIES = +1000` (`doc_ids_only.rs:26`) because id-only postings are tiny +(question 1). The encoder is a *type parameter* (`InvertedIndex`, `PhantomData`), so codec choice is compile-time. This is the Rust rewrite earning its keep: the C original dispatched on -`IndexFlags` at runtime *per record*; the Rust one monomorphizes -eleven codecs and lets FFI pick the concrete type once -(`c_entrypoint/inverted_index_ffi`) — the per-posting branch simply -no longer exists. +`IndexFlags` at runtime *per record*; the Rust one monomorphizes the +codecs and lets FFI pick the concrete type once — +`NewInvertedIndex_Ex` (`c_entrypoint/inverted_index_ffi/src/lib.rs:105`) +matches the flags to a concrete arm like `InvertedIndex::Full(...)` +(:116) — so the per-posting branch simply no longer exists. ### Step 5 — deletes and readers: GC, gc_marker, unique_id -A mutable index can't do tantivy's "alive-bitmap now, purge at -merge" — there is no merge. Instead a **GC pass** (`gc.rs`) -rewrites blocks in place to purge deleted docs — compaction for a -mutable index — which invalidates any cursor mid-list. Two -validation devices protect readers: +> **In:** deleted docs accumulating in a mutable, in-place index that no merge ever rewrites. +> **Out:** a GC pass that compacts blocks in place, plus two integers (`gc_marker`, `unique_id`) that let concurrent readers detect that their cursor went stale. -- `gc_marker` (an atomic counter bumped by GC) — a cursor compares - its saved marker and knows its position is stale; +A mutable index can't do tantivy's "alive-bitmap now, purge at +merge" — there is no merge. Instead a **GC pass** (`gc.rs`) rewrites +blocks in place to purge deleted docs — compaction for a mutable +index (`repair` :139, `scan_gc` :214, `apply_gc` :242) — which +invalidates any cursor mid-list. Two validation devices protect +readers: + +- `gc_marker` (an atomic counter bumped by GC — `apply_gc` calls + `gc_marker_inc`, gc.rs:340) — a cursor compares its saved marker + and knows its position is stale; - `unique_id` — ABA detection (the "freed, then something new allocated at the same address" hazard): if the whole index was dropped and reallocated at the same pointer, cursors notice via @@ -124,6 +161,9 @@ maps this onto FalkorDB's delta-matrix `wait`/version story). ### Step 6 — the deltas vs tantivy, and what M23 should copy +> **In:** the two indexes side by side — immutable-batch vs mutable-now. +> **Out:** a per-axis diff, and a copy/avoid list for M23's own index. + The whole comparison, one line per axis: ``` @@ -132,7 +172,7 @@ The whole comparison, one line per axis: encoding 128-block bitpack (SIMD) varint per entry (byte-at-a-time) deletes alive-bitmap, purge on merge GC pass rewrites blocks in place concurrency segment = snapshot gc_marker + unique_id cursor validation - granularity postings files per field codec picked per index flags (11 variants) + granularity postings files per field codec picked at compile time (10 modules) why batch search workloads a Redis module: single-threaded-ish, updates must be cheap NOW, no background merge infrastructure @@ -159,12 +199,13 @@ unless noted: |---|---| | `core.rs:30` `InvertedIndex` | `blocks: ThinVec`, `n_unique_docs`, `flags: IndexFlags`, `gc_marker: AtomicU32`, `unique_id` — encoder is a type parameter (`PhantomData`), so codec choice is compile-time (2, 4) | | `core.rs:75` `IndexBlock` | `{ first_doc_id, last_doc_id, num_entries: u16, buffer: Vec }` — a growable byte buffer of varint-encoded entries, chained, NOT fixed 128-wide bitpacked (2) | -| `core.rs:229` | a delta too large for the codec ⇒ start a new block with delta 0 (`IdDelta::from_u64` → None path, codec/mod.rs:28-44) (3) | -| `codec/mod.rs:53` `trait Encoder` | `write(record, delta)`, `delta_base(block)` — one trait, eleven codecs (4) | -| `codec/` | `doc_ids_only` / `raw_doc_ids_only` / `freqs_only` / `freqs_fields` / `fields_offsets` / `full` / `numeric` … — the granularity ladder from Zobel-Moffat §3 as a directory listing (4) | -| `varint/src/lib.rs:98` `VarintEncode` | the wire format under most codecs (3, 4) | -| `gc.rs` | garbage collection rewrites blocks to purge deleted docs — compaction for a mutable index; `gc_marker` tells live readers their cursor is stale (5) | -| `unique_id` (core.rs comment) | ABA detection: index freed + reallocated at same address ⇒ cursors notice via id mismatch — a very Redis-module concern (5) | +| `core.rs:195` `add_record`; delta logic :219-243 | `delta_base` (:219) → `wrapping_sub` (:224) → `from_u64` None ⇒ new block at delta 0 (:226-238) → `E::encode` (:243) (3) | +| `codec/mod.rs:53` `trait Encoder` | `encode(writer, delta, record)` (:74), `delta_base(block)` (:81), `RECOMMENDED_BLOCK_ENTRIES = 100` (:70) (4) | +| `codec/mod.rs:10-19` | ten codec modules: `doc_ids_only` / `raw_doc_ids_only` / `freqs_only` / `freqs_fields` / `freqs_offsets` / `fields_only` / `fields_offsets` / `offsets_only` / `full` / `numeric` — Zobel & Moffat's granularity ladder as a directory listing (4) | +| `codec/doc_ids_only.rs:26` | `RECOMMENDED_BLOCK_ENTRIES = 1000` override — id-only postings pack more per block (1, 4) | +| `varint/src/lib.rs:98` `VarintEncode` | the wire format under most codecs; `write_as_varint` :101 (3, 4) | +| `inverted_index/src/gc.rs` | `repair` :139, `scan_gc` :214, `apply_gc` :242 rewrite blocks to purge deleted docs; `gc_marker_inc` :340 tells live readers their cursor is stale (5) | +| `c_entrypoint/inverted_index_ffi/src/lib.rs:105` `NewInvertedIndex_Ex` | C picks the monomorphized type once, e.g. `InvertedIndex::Full(...)` :116 (4, 5) | Read order: `core.rs` top-to-bottom (it's the smallest core file in this topic), then `codec/mod.rs` + one concrete codec @@ -175,12 +216,14 @@ monomorphized type. ## Questions (answer in notes.md) 1. `num_entries: u16` and buffer-growth: what's the effective block - size policy, and why does variable block length make block-max - metadata harder to bolt on than tantivy's fixed 128? + size policy (default `RECOMMENDED_BLOCK_ENTRIES = 100`, but + `doc_ids_only` overrides to 1000), and why does variable block + length make block-max metadata harder to bolt on than tantivy's + fixed 128? 2. The `gc_marker`/`unique_id` cursor-validation dance: map it onto FalkorDB's delta-matrix `wait` + version story. What does each protect against, and which is stricter? -3. Eleven codecs vs tantivy's one postings format + fast fields: +3. Ten codecs vs tantivy's one postings format + fast fields: which RediSearch codecs correspond to "positions" and "doc values" in the Lucene taxonomy? 4. Varint vs bitpacked at df=99888/100K docs (delta≈1, one byte @@ -193,17 +236,62 @@ monomorphized type. ## Done when +Answer each before unfolding it. + - [ ] You can state the constraint that shapes the whole design: mutable now, or nothing. +
the single-threaded command loop + + A Redis write command must leave the document queryable when it + returns, inside a (mostly) single-threaded loop with no merge + threads. That rules out immutable-segments + background merge and + forces one in-place-mutable posting list per term. + +
- [ ] You can describe the chained growable block structure per term and the write path through it. +
ThinVec of byte-buffer blocks + + `InvertedIndex` (core.rs:30) holds a `ThinVec`; each + `IndexBlock` (core.rs:75) is a growable `Vec` of varint entries. + `add_record` (core.rs:195) varint-appends `doc_id − delta_base` + (:219-224) to the last block, chaining a fresh block at delta 0 + when `from_u64` returns None (:226-238). + +
- [ ] You can explain the codec ladder — one trait, many encoders, chosen at compile time — and why that is a codegen decision rather than a runtime one. +
Encoder as a type parameter + + Ten codec modules (codec/mod.rs:10-19) each `impl Encoder` + (:53, `encode` :74). `InvertedIndex` carries the codec as a type + parameter (`PhantomData`), so the compiler monomorphizes it and + the per-record `IndexFlags` branch the C code paid disappears; FFI + picks the concrete type once (inverted_index_ffi/src/lib.rs:105). + +
- [ ] You can explain how GC, `gc_marker` and `unique_id` let readers survive concurrent deletes. +
compact-in-place + two guards + + GC (gc.rs: `scan_gc` :214, `apply_gc` :242) rewrites blocks to drop + deleted docs, invalidating cursors. `gc_marker` (bumped at :340) is + compared by a cursor to detect a stale position; `unique_id` + catches ABA (index dropped and reallocated at the same address). + +
- [ ] You wrote answers to all questions in notes.md. +
check + + Five answers in notes.md, including the varint-vs-bitpack + bytes/posting computation (Q4) and the FalkorDB cursor-validation + mapping (Q2). + +
## References **Code** -- [RediSearch](https://github.com/RediSearch/RediSearch) - `src/redisearch_rs/` — `inverted_index/src/index/core.rs` (the - structure), `inverted_index/src/codec/` (eleven codecs, one - trait), `varint/src/lib.rs`, `inverted_index/src/gc.rs`, and the - FFI seam in `c_entrypoint/inverted_index_ffi` +- [RediSearch](https://github.com/RediSearch/RediSearch) `@87276ca` + `src/redisearch_rs/` — `inverted_index/src/index/core.rs` + (`InvertedIndex` :30, `IndexBlock` :75, `add_record` :195-243), + `inverted_index/src/codec/mod.rs` (`trait Encoder` :53, ten modules + :10-19), `inverted_index/src/codec/doc_ids_only.rs:26`, + `varint/src/lib.rs:98`, `inverted_index/src/gc.rs:139,214,242`, and + the FFI seam `c_entrypoint/inverted_index_ffi/src/lib.rs:105` diff --git a/topics/23-fulltext/reading-roaring.md b/topics/23-fulltext/reading-roaring.md index ae49dce..5d34436 100644 --- a/topics/23-fulltext/reading-roaring.md +++ b/topics/23-fulltext/reading-roaring.md @@ -10,6 +10,11 @@ their break-even point, the two-level partition, the per-pair kernel matrix — and ends with why posting lists (the filter lane of a search engine) care. +Source pins: two papers, cited by arXiv id below; the code you +implement is this repo's `experiments/src/postings.rs` stub; the +production cousin quoted at the end is RediSearch at `87276ca`. A +**posting list** here means the set of doc ids that contain a term. + ## The problem in one sentence Store "the set of doc ids matching a filter" so that both a @@ -22,6 +27,9 @@ small AND intersect fast — a sorted `Vec` makes the dense one ### Step 1 — two ways to store a set of integers, and the break-even +> **In:** a set of integers drawn from a 65,536-value universe. +> **Out:** two representations (sorted array, bitmap) and the density where their sizes cross — 4096 elements — below which the array is smaller, above which the bitmap is. + A set of integers has two classic representations. A **sorted array** stores each member explicitly — cost proportional to *how many* members (2 bytes each if values fit u16). A **bitmap** stores @@ -29,12 +37,25 @@ one bit per *possible* value — cost proportional to the *universe size*, membership is one bit test, and intersection is a word-wise AND running at 64 members per instruction. Over a 65,536-value universe the bitmap costs a flat 8 KiB; the array costs -2·|set| bytes. Equating them: 8192 bytes / 2 bytes = **4096 -elements** — below that the array is smaller, above it the bitmap -is. Density decides, and real data mixes both regimes in one set. +2·|set| bytes. Equating them (worked): + +``` +bitmap: 65536 bits / 8 = 8192 bytes (flat, any density) +array: 2 bytes/value · |set| +equal: 8192 / 2 = 4096 elements ← the crossover + |set| = 100: array 200 B vs bitmap 8192 B → array wins + |set| = 4096: array 8192 B vs bitmap 8192 B → tie + |set| =50000: array 100000 B vs bitmap 8192 B → bitmap wins (12×) +``` + +Below 4096 the array is smaller, above it the bitmap is. Density +decides, and real data mixes both regimes in one set. ### Step 2 — the partition: choose a representation every 64K values +> **In:** a full 32-bit value space where density varies across ranges. +> **Out:** split each u32 into a 16-bit chunk key and 16-bit low half; each chunk stores its low bits in a **container** whose type is chosen by that chunk's local density — capping size at 8 KiB and at 2 bytes/value simultaneously. + Roaring splits each 32-bit value into high and low halves: the high 16 bits select a **chunk** (one of up to 64K aligned ranges of 65,536 values), and each chunk stores its members' low 16 bits in @@ -60,17 +81,25 @@ density: The guarantee that falls out: every container is at most 8 KiB *and* at most 2 bytes per stored value — the adaptive choice caps both failure modes. The **run container** ((start, length) pairs — -run-length encoding, the 2016 paper's addition) handles the third -regime the first paper missed: long consecutive runs of ids, where -even a bitmap wastes bits (question 1 asks which posting-list -shapes produce runs). +run-length encoding, the 2016 Lemire paper's addition) handles the +third regime the first paper missed: long consecutive runs of ids, +where even a bitmap wastes bits. The 2016 paper (§4) only converts +to a run container when it would be smaller than *both* alternatives +and there are ≤ 2047 runs — and *only* on an explicit `runOptimize` +call, never automatically (question 1 asks which posting-list +shapes produce runs). This repo's Rust stub implements array + +bitmap only (`postings::Container`), the two the CRoaring reference +starts from. ### Step 3 — the kernel matrix: one algorithm per container pair +> **In:** two roaring bitmaps, each a sorted list of typed containers. +> **Out:** the set operation decomposes into per-chunk kernels, dispatched by the *pair* of container types — each kernel the textbook-optimal algorithm for that shape. + With two (or three) container types, a set operation between two roaring bitmaps decomposes into per-chunk operations, each dispatched to a specialized **kernel** by the pair of container -types (§3 of the paper — what the stub implements): +types (the kernel matrix the stub implements): | A ∩/∪ B | array | bitmap | |---|---|---| @@ -83,9 +112,13 @@ two sorted arrays → two-pointer merge, escalating to **galloping** one) when one side is ≥64× smaller; array vs bitmap → probe each array element (one word test each), never touching the bitmap's other 65K bits; bitmap vs bitmap → 1024 unconditional word ANDs. +The stub you fill in stores containers exactly this way and returns +a plain `Vec` to compare against the two-pointer oracle: ```rust -// the whole design in one match: kernel AND output type chosen per chunk +// ILLUSTRATION — the design the reader implements in the stub at +// experiments/src/postings.rs:56-86 (Container enum, ARRAY_MAX = 4096, +// and()/or() stubs). Real bodies are `todo!()`; fill them to match this. fn and(a: &Container, b: &Container) -> Container { match (a, b) { (Array(x), Array(y)) => two_pointer(x, y), // gallop if ≥64× skew @@ -107,6 +140,9 @@ fn and(a: &Container, b: &Container) -> Container { ### Step 4 — the two details that carry the performance +> **In:** the kernel matrix, which looks like a mechanical case-split. +> **Out:** the two decisions that actually decide performance — choosing the *output* container type by popcount, and tracking cardinality as a byproduct rather than recomputing it. + The match arms are obvious; two less-obvious decisions do the real work: @@ -125,40 +161,51 @@ The general lesson: an adaptive data structure lives or dies by its ### Step 5 — why posting lists care: the filter lane -Measured in fts_bench: `t0 ∧ t5000` (99888 ∩ 172 docs) costs 52 µs -with two-pointer — it walks all 99888. Roaring: t0 at df≈100K over -100K docs is ~1.5 dense chunks → bitmap containers; the 172-element -side probes 172 times → ~1 µs. Same asymmetry galloping fixes for -arrays, but roaring ALSO compresses t0 to 8 KiB·2 instead of 400 KB -— 25× less memory traffic on the dense side, which is where the -time actually goes (question 3). +> **In:** the two measured intersections from this topic — dense∩sparse and dense∩dense. +> **Out:** why roaring turns both the memory and the time of the dense side down by ~25×, and why this is the FILTER lane, not the RANKING lane BM25/WAND own. + +Measured in fts_bench (this topic's `notes.md`): `t0 ∧ t5000` +(99,888 ∩ 172 docs) costs 52 µs with two-pointer — it walks all +99,888. Roaring: t0 at df≈100K over 100K docs is ~1.5 dense chunks +→ bitmap containers; the 172-element side probes 172 times → ~1 µs. +Same asymmetry galloping fixes for arrays, but roaring ALSO +compresses t0 to 8 KiB·2 instead of 400 KB — 25× less memory +traffic on the dense side, which is where the time actually goes +(question 3). Lucene's `RoaringDocIdSet` and RediSearch's doc tables use exactly -this for filters (the `docs_ids_only` codec in -`redisearch_rs/inverted_index/src/codec/doc_ids_only.rs` is the -varint cousin). Note what roaring does NOT store: tf, positions, -scores — it's the FILTER lane (Cypher `WHERE n.name CONTAINS ...` -feeding a graph traversal), not the RANKING lane; BM25/WAND (the -previous chapters) own that one. And a bitmap container is exactly -a dense GraphBLAS vector chunk (question 4) — the M20/M23 bridge. +this for filters (the `doc_ids_only` codec at +`src/redisearch_rs/inverted_index/src/codec/doc_ids_only.rs` is the +varint cousin — `RECOMMENDED_BLOCK_ENTRIES = 1000` there, doc_ids_only.rs:26). +Note what roaring does NOT store: tf, positions, scores — it's the +FILTER lane (Cypher `WHERE n.name CONTAINS ...` feeding a graph +traversal), not the RANKING lane; BM25/WAND (the previous chapters) +own that one. And a bitmap container is exactly a dense GraphBLAS +vector chunk (question 4) — the M20/M23 bridge. ## How to read the papers (with the concepts in hand) Two short papers, both readable in one sitting: -- **Chambi et al. 2014/2016 (arXiv:1402.6407).** §2 is Steps 1–2 - (the partition and the 4096 crossover); §3 is Step 3's kernel - matrix — read it against the `match` above and check every arm. - The experiments compare against WAH/Concise (older compressed +- **Chambi et al. (Software: Practice & Experience 2016, + [arXiv:1402.6407](https://arxiv.org/abs/1402.6407)).** §2 is + Steps 1–2 (the partition and the 4096 crossover); §3 is Step 3's + kernel matrix — read it against the `match` above and check every + arm. This paper has TWO container types only (array + bitmap), and + the experiments compare against WAH/Concise (older compressed bitmaps that lack random access) — skim, the lesson is that chunked-and-adaptive beats stream-compressed. -- **Lemire et al. 2016 (arXiv:1603.06549).** Adds the run container - (Step 2's third regime) and SIMD kernels; read the run-container - conversion rules ("convert only when smaller") — the same +- **Lemire et al. (SPE 2016, + [arXiv:1603.06549](https://arxiv.org/abs/1603.06549)).** Adds the + run container (Step 2's third regime) and SIMD kernels, and + describes the CRoaring C reference implementation; read the + run-container conversion rules (§4: convert only when smaller than + both, ≤2047 runs, only on `runOptimize`) — the same transition-logic discipline as Step 4. -- Then implement the `postings::Roaring` stub — array/bitmap - containers with AND/OR against the two-pointer vec oracle — - before answering the questions. +- Then implement the `postings::Roaring` stub + (`experiments/src/postings.rs`) — array/bitmap containers with + AND/OR against the two-pointer vec oracle — before answering the + questions. ## Questions (answer in notes.md) @@ -183,11 +230,54 @@ Two short papers, both readable in one sitting: ## Done when +Answer each before unfolding it. + - [ ] You can derive the 4096 crossover from bytes per value. +
the arithmetic + + Bitmap is flat 65536/8 = 8192 bytes. Array is 2 bytes/value. + 8192/2 = 4096 values is where they cost the same; below it array + is smaller, above it bitmap is. A container therefore never + exceeds 8 KiB nor 2 bytes/value. + +
- [ ] You can explain why the representation is chosen per 64K range rather than per set. +
local vs global density + + One set can be sparse in some 64K ranges and dense in others. + Choosing per chunk (high 16 bits) lets each range pick the smaller + representation, so the structure adapts to *local* density instead + of paying one global choice. + +
- [ ] You can name the kernel matrix idea: one algorithm per container pair. -- [ ] You can say what a 99.9%-dense posting list like this topic's `t0` (df 99 888 of 100 000 docs) should become, and what that costs against the sorted-vec baseline measured here (0.1178 ms for dense∧dense AND). +
dispatch by type pair + + Each set op dispatches on (typeA, typeB): array∩array → two-pointer + (gallop on skew), array∩bitmap → probe the array into the bitmap, + bitmap∩bitmap → 1024 word ANDs + popcount to choose the output + type. Each is optimal for that shape. + +
+- [ ] You can say what a 99.9%-dense list like `t0` (df 99,888 of 100,000) should become, and its cost against the sorted-vec baseline. +
bitmap containers + + ~1.5 dense chunks → bitmap containers (8 KiB each, ~16 KiB total vs + 400 KB as a `Vec`). Intersection with a sparse side probes the + small side (~1 µs); dense∩dense is ~1024 word ANDs per chunk — + against the sorted-vec baseline measured here (0.1178 ms for + dense∧dense, notes.md). + +
- [ ] You wrote answers to all five questions in notes.md, including the M20 bitmap-container tie-in. +
check + + Five answers in notes.md; question 4 explicitly maps a bitmap + container to a dense GraphBLAS vector chunk and an array container + to a sparse one, comparing the 4096/65536 thresholds to GraphBLAS's + format lattice. + +
## References @@ -199,4 +289,13 @@ Two short papers, both readable in one sitting: - Lemire, Ssi-Yan-Kai, Kaser — "Consistently faster and smaller compressed bitmaps with Roaring" (SPE 2016, [arXiv:1603.06549](https://arxiv.org/abs/1603.06549)) — adds the - run container and the SIMD kernels + run container (§4, the ≤2047-run / runOptimize rule) and SIMD + kernels; describes the CRoaring reference implementation + +**Code** +- This repo — `experiments/src/postings.rs`: `Container` enum + (:58-61), `ARRAY_MAX = 4096` (:56), `Roaring::and`/`or` stubs + (:79-86), the `vec_and`/`vec_or` oracle (:11, :29) +- [RediSearch](https://github.com/RediSearch/RediSearch) `@87276ca` + `src/redisearch_rs/inverted_index/src/codec/doc_ids_only.rs` — the + varint doc-id codec, `RECOMMENDED_BLOCK_ENTRIES = 1000` (:26) diff --git a/topics/23-fulltext/reading-tantivy.md b/topics/23-fulltext/reading-tantivy.md index 02dcefa..f86eb76 100644 --- a/topics/23-fulltext/reading-tantivy.md +++ b/topics/23-fulltext/reading-tantivy.md @@ -9,6 +9,11 @@ dictionary, the posting blocks, the skip data, the scoring/WAND wiring, and the segment write path — then hands you every file:line anchor and a 90-minute read order. +Source pin: every anchor below is tantivy at `7152d53` and was +re-verified with `tools/pinned-source.py` at that SHA. A **segment** +is one immutable, self-contained mini-index (its own dictionary + +postings); a query fans out over all of them. + ## The problem in one sentence Turn "quick fox" into a ranked top-10 over millions of documents in @@ -24,20 +29,27 @@ only immutable files, four lookups deep: ### Step 1 — analysis: text becomes terms before anything is indexed -An analyzer is the pipeline that converts raw text into the terms +> **In:** raw document text (and, at query time, raw query text). +> **Out:** a stream of normalized terms produced by the *same* pipeline for both, so query terms and indexed terms meet in one space. + +An **analyzer** is the pipeline that converts raw text into the terms the index actually stores: tokenize (split on word boundaries) → lowercase → stem ("running" → "run") → drop stopwords. Query text runs through the *same* pipeline, so query terms and indexed terms meet in the same normalized space — mismatched analyzers are the classic "search finds nothing" bug. tantivy models this as -composition: `TextAnalyzer` (`tokenizer/tokenizer.rs`) is a boxed -`Tokenizer` plus a filter chain (lower_caser, stemmer, -stop_word_filter, ngram…), with one dyn-dispatch per *stream*, not -per token — pipeline flexibility without a virtual call in the -per-token hot loop. +composition: `TextAnalyzer` (`tokenizer/tokenizer.rs:9-11`) is a +single boxed `Box` (:10) wrapping a +tokenizer plus its filter chain (lower_caser, stemmer, +stop_word_filter, ngram…). `token_stream` does one dyn-dispatch per +*stream* (:19-20), not per token — pipeline flexibility without a +virtual call in the per-token hot loop. ### Step 2 — the term dictionary: an FST from term bytes to postings +> **In:** a term's bytes (e.g. `b"fox"`). +> **Out:** a `TermInfo` — where the postings live and how many docs contain the term — reached through an ordered automaton, not a hash. + The term dictionary maps each term's bytes to where its postings live. tantivy uses an **FST** (finite state transducer — a minimized automaton over sorted keys that shares common prefixes @@ -46,26 +58,50 @@ bytes → term ordinal → a `TermInfoStore` entry. Versus a hash map, the FST is smaller (shared structure) and *ordered* — enabling prefix, range, and regex queries by automaton intersection, which a hash can never do. The price: an FST is built from sorted keys and -is immutable (`MapBuilder`, `termdict/fst_termdict/termdict.rs:25`, -insert at `:46`) — hence per-segment build + merge (Step 6), and -opening is mmap-friendly (`:92 open_fst_index`, `Fst::new(bytes)` — -no deserialization). - -The value side is `TermInfo { doc_freq, postings_range }` -(`postings/term_info.rs:9-13`): **df rides in the dictionary**, so -idf — and WAND's per-term ceiling — is known before a single -posting is read. +is immutable (`MapBuilder` field at +`termdict/fst_termdict/termdict.rs:25`, `insert(term, &TermInfo)` at +`:46`) — hence per-segment build + merge (Step 6), and opening is +mmap-friendly (`:92 open_fst_index`, `Fst::new(bytes)` at `:94` — no +deserialization). + +The value side is `TermInfo { doc_freq, postings_range, +positions_range }` (`postings/term_info.rs:9-16` — `doc_freq` at +:11, the `.idx` byte range at :13, the `.pos` byte range at :15): +**df rides in the dictionary**, so idf — and WAND's per-term ceiling +— is known before a single posting is read. ### Step 3 — posting blocks: 128 deltas, one bit width, SIMD unpack +> **In:** a term's doc-sorted posting list. +> **Out:** fixed 128-doc blocks, each delta-encoded and bit-packed to the width of its largest delta, so all 128 unpack branch-free with SIMD. + Posting lists store doc ids as deltas (previous chapter's Zipf -argument) in fixed blocks of 128 (`postings/compression/mod.rs:3`, -`COMPRESSION_BLOCK_SIZE = BitPacker4x::BLOCK_LEN`), where each -block is bit-packed to the width of its *largest* delta (`:61`, -delta-encoded against `block_minus_one`): +argument) in fixed blocks of 128 +(`postings/compression/mod.rs:3`, `COMPRESSION_BLOCK_SIZE = +BitPacker4x::BLOCK_LEN` = 128). Doc-id blocks are *strictly sorted*, +so they go through `compress_block_sorted(block, offset)` +(`compression/mod.rs:36-46`): it delta-encodes against `offset` — +the previous block's last doc id (`None` for the first block, :39) — +then bitpacks with `compress_strictly_sorted` (:43-44) to the width +of the block's largest delta. (The separate `block_minus_one` path +at `:61` is inside `compress_block_unsorted` (:54-69) for +*term-frequency* values, which are ≥1 and unsorted — comment +:50-53; it is NOT the doc-id delta path, a point an earlier draft of +this guide got wrong.) Worked, for a block whose largest gap is 100: + +``` +bits = 32 − leading_zeros(100) = 32 − 25 = 7 bits per delta +block = 1 width byte + ⌈7·128 / 8⌉ = 1 + 112 = 113 bytes for 128 docs + ≈ 7.06 bits/posting — vs 32 bits raw, and one branch-free SIMD unpack +``` + +The design, reconstructed (the real code delegates the pack to +`BitPacker4x`): ```rust -// 128 doc-id deltas, bit-packed to the WHOLE block's max width +// ILLUSTRATION — the real doc-id path is compress_block_sorted at +// src/postings/compression/mod.rs:36-46, which calls the SIMD +// BitPacker4x::compress_strictly_sorted (offset = prev block's last doc). fn write_block(docs: &[u32; 128], prev_last: u32, out: &mut Vec) { let mut deltas = [0u32; 128]; for i in 0..128 { @@ -75,8 +111,6 @@ fn write_block(docs: &[u32; 128], prev_last: u32, out: &mut Vec) { out.push(bits); // ONE width per block → SIMD unpacks all bitpack(&deltas, bits, out); // 128 at once, no per-posting branches } -// next to it, a skip entry: { last_doc, block_max_score } — WAND moves -// across blocks without ever decoding the losers ``` One width per block wastes a few bits on outlier deltas but buys @@ -86,50 +120,66 @@ question 3 covers the <128-tail vint fallback). ### Step 4 — skip data: block metadata that answers questions without decoding -Next to each compressed block lives an uncompressed skip entry: -`last_doc_in_block` (`postings/skip.rs:186`) and -`block_max_score` (`:175`, via `bm25_weight`), read through -`SkipReader` (`:93`). This is the block-max WAND chapter's -"shallow pointer movement" made concrete: a cursor can answer "does -this block contain doc ≥ d?" and "can this block possibly beat θ?" -from metadata alone, decompressing only blocks that survive both -tests. The design rule: keep the metadata that *steers* uncompressed -and tiny, and the payload it steers compressed and bulky. +> **In:** a compressed posting list and a WAND cursor asking "any doc ≥ d here? could this block beat θ?". +> **Out:** both answers from a tiny uncompressed skip entry, decoding only blocks that survive both tests. + +Next to each compressed block lives an uncompressed skip entry, read +through `SkipReader` (`postings/skip.rs:93`). It stores the block's +`last_doc_in_block` (`:186-187`) and the block-WAND inputs +`(block_wand_fieldnorm_id, block_wand_term_freq)` (:113-114) — not +the score itself; `block_max_score(bm25_weight)` (`:175-181`) +*recomputes* the ceiling on demand via `bm25_weight.score(...)`, +returning `None` for the last incomplete block. This is the +block-max WAND chapter's "shallow pointer movement" made concrete: a +cursor can answer "does this block contain doc ≥ d?" and "can this +block possibly beat θ?" from metadata alone, decompressing only +blocks that survive both tests. The design rule: keep the metadata +that *steers* uncompressed and tiny, and the payload it steers +compressed and bulky. ### Step 5 — scoring and WAND: the previous chapters, wired together +> **In:** a `TermInfo` per query term and a posting stream. +> **Out:** a BM25 score per surviving doc (a table lookup + one multiply-add) and a block-max WAND top-k loop over them — the two previous chapters, shipped. + Scoring is BM25 exactly as derived: K1/B at `query/bm25.rs:8-9`, -idf at `:52`, and the length-norm term precomputed per 1-byte -fieldnorm into a 256-entry table (`:59`) — scoring is a table -lookup plus one multiply-add per posting. Top-k evaluation is -block-max WAND: `find_pivot_doc` -(`query/boolean_query/block_wand_union.rs:8-24`) walks scorers -sorted by doc id accumulating `max_weight` until it crosses the -threshold — the SIGIR'11 paper, shipped — with the sibling -`block_wand_intersection.rs` for AND queries. Nothing in this step -is new if you read the two previous chapters; that's the point — -tantivy is those papers with error handling. +idf at `:52-56`, and the length-norm term precomputed per 1-byte +fieldnorm into a 256-entry table (`cached_tf_component` :58-60, +`compute_tf_cache` :62-68) — scoring is a table lookup plus one +multiply-add per posting. Top-k evaluation is block-max WAND: +`find_pivot_doc` (`query/boolean_query/block_wand_union.rs:16-43`) +walks scorers sorted by doc id accumulating `max_score` (:20, :25) +until it crosses `threshold` (:26) — the SIGIR'11 paper, shipped — +with the sibling `block_wand_intersection.rs` for AND queries. +Nothing in this step is new if you read the two previous chapters; +that's the point — tantivy is those papers with error handling. ### Step 6 — the write path: topic 4's LSM wearing a hat +> **In:** a stream of new documents into an immutable-file engine. +> **Out:** RAM-buffered segments flushed (never updated) to disk, then merged in log-size tiers — an LSM without a key range to prune. + Everything above is immutable — so writes go to an in-RAM segment that is *flushed*, never updated: ```mermaid graph LR A["IndexWriter
(RAM budget)"] -->|flush| S1["segment (immutable):
.term .idx .pos .fieldnorm .fast"] - S1 --> MP["LogMergePolicy
indexer/log_merge_policy.rs:20-24:
min_num_segments,
max_docs_before_merge,
level_log_size"] + S1 --> MP["LogMergePolicy
indexer/log_merge_policy.rs:20-26:
min_num_segments,
max_docs_before_merge,
level_log_size"] MP -->|merge ~same-size tier| S2["bigger segment"] D["deletes"] --> DB["alive bitset per segment
(tombstones)"] ``` -`LogMergePolicy` groups segments into log-size levels and merges -within a level — Lucene's tiered compaction, not leveled: full-text -tolerates overlapping "levels" because every query fans out over all -segments anyway (there's no key range to prune, unlike topic 4's -SSTable ranges). Deletes are an alive-bitmap per segment, purged at -merge. The cost of more segments isn't wrong answers — it's -per-query fan-out and duplicated dictionary lookups (question 4). +`LogMergePolicy` (struct :20-26; defaults :8-11 — +`level_log_size=0.75`, `min_layer_size=10_000`, +`min_num_segments=8`, `max_docs_before_merge=10_000_000`) groups +segments into log-size levels and merges within a level — Lucene's +tiered compaction, not leveled: full-text tolerates overlapping +"levels" because every query fans out over all segments anyway +(there's no key range to prune, unlike topic 4's SSTable ranges). +Deletes are an alive-bitmap per segment, purged at merge. The cost +of more segments isn't wrong answers — it's per-query fan-out and +duplicated dictionary lookups (question 4). Fast fields (`fastfield/`) are the columnar side — doc values for sorting/faceting — literally topic 12 embedded in a text index. @@ -138,14 +188,14 @@ sorting/faceting — literally topic 12 embedded in a text index. | subsystem (step) | anchor | what to see | |---|---|---| -| analysis (1) | `tokenizer/tokenizer.rs` `TextAnalyzer` — boxed `Tokenizer` + filter chain (lower_caser, stemmer, stop_word_filter, ngram…) | pipelines as composition, one dyn-dispatch per stream not per token | -| term dict (2) | `termdict/fst_termdict/termdict.rs:25` builder wraps `tantivy_fst::MapBuilder`; `:46 insert(term, &TermInfo)`; `:92 open_fst_index` (mmap-friendly `Fst::new(bytes)`) | FST maps term bytes → term ordinal → `TermInfoStore` — prefix+suffix sharing beats a hash dict AND gives range/regex queries | -| term info (2) | `postings/term_info.rs:9-13` `TermInfo { doc_freq, postings_range }` | df rides in the dictionary — idf is known before touching postings | -| postings (3) | `postings/compression/mod.rs:3` `COMPRESSION_BLOCK_SIZE = BitPacker4x::BLOCK_LEN` (=128); `:61` delta-encode against `block_minus_one` | 128 deltas bit-packed to the block's max width; SIMD unpack | -| skip data (4) | `postings/skip.rs:93` `SkipReader`; `:175 block_max_score(bm25_weight)`; `:186 last_doc_in_block` | block-max metadata lives in skip entries — moving blocks never decodes postings | -| scoring (5) | `query/bm25.rs:8-9` K1/B; `:52` idf; `:59` tf-norm via 1-byte fieldnorm table | scoring = table lookup + multiply-add | -| WAND (5) | `query/boolean_query/block_wand_union.rs:8-24` `find_pivot_doc`; sibling `block_wand_intersection.rs` | the SIGIR'11 paper, shipped | -| merge (6) | `indexer/log_merge_policy.rs:20-24` | tiered, not leveled, compaction | +| analysis (1) | `tokenizer/tokenizer.rs:9-11` `TextAnalyzer` (boxed `BoxableTokenizer`); `token_stream` :19-20 | pipelines as composition, one dyn-dispatch per stream not per token | +| term dict (2) | `termdict/fst_termdict/termdict.rs:25` `MapBuilder` field; `:46 insert(term, &TermInfo)`; `:92 open_fst_index` (mmap `Fst::new` :94) | FST maps term bytes → term ordinal → `TermInfoStore` — prefix+suffix sharing beats a hash dict AND gives range/regex queries | +| term info (2) | `postings/term_info.rs:9-16` `TermInfo { doc_freq, postings_range, positions_range }` | df rides in the dictionary — idf is known before touching postings | +| postings (3) | `postings/compression/mod.rs:3` `COMPRESSION_BLOCK_SIZE` (=128); doc-id path `compress_block_sorted` :36-46 (delta vs prev block's last, then bitpack) | 128 deltas bit-packed to the block's max width; SIMD unpack | +| skip data (4) | `postings/skip.rs:93` `SkipReader`; `:175-181 block_max_score(bm25_weight)` (recomputes); `:186-187 last_doc_in_block` | block-max metadata lives in skip entries — moving blocks never decodes postings | +| scoring (5) | `query/bm25.rs:8-9` K1/B; `:52-56` idf; `:58-68` tf-norm 256-entry fieldnorm table | scoring = table lookup + multiply-add | +| WAND (5) | `query/boolean_query/block_wand_union.rs:16-43` `find_pivot_doc`; sibling `block_wand_intersection.rs` | the SIGIR'11 paper, shipped | +| merge (6) | `indexer/log_merge_policy.rs:20-26` | tiered, not leveled, compaction | Suggested 90-minute read order: @@ -177,20 +227,66 @@ Suggested 90-minute read order: ## Done when +Answer each before unfolding it. + - [ ] You can say why the term dictionary is an FST rather than a hash map, and list what the FST gives you that a hash cannot. +
ordered automaton vs hash + + An FST shares prefixes AND suffixes (smaller than a hash dict) and + keeps keys *ordered*, so prefix, range, and regex queries run by + automaton intersection — a hash offers none of these. The cost: + build-from-sorted-keys + immutability, hence per-segment build and + merge. + +
- [ ] You can describe 128-delta posting blocks with one bit width, and what happens to a final partial block. +
fixed blocks + vint tail + + Each 128-doc block is delta-encoded (`compress_block_sorted`, + compression/mod.rs:36-46) and bit-packed to the block's largest + delta width — one width byte + packed deltas, SIMD-unpacked. The + trailing <128 postings can't fill a BitPacker4x block, so they fall + back to variable-length ints (question 3). + +
- [ ] You can explain what skip data answers without decoding. +
steer without decode + + A skip entry (skip.rs:93) gives `last_doc_in_block` (:186-187) and + the block-WAND inputs to recompute `block_max_score` (:175-181), so + a cursor answers "any doc ≥ d here?" and "can this block beat θ?" + from metadata — decoding only blocks that survive both. + +
- [ ] You can explain why the write path is topic 4's LSM wearing a hat, and contrast LogMergePolicy with leveled compaction. +
tiered, no key range + + Writes buffer in RAM and flush to immutable segments; merges happen + in log-size tiers (LogMergePolicy :20-26), like Lucene's tiered + compaction. Unlike a leveled LSM there's no key range to prune — + every query fans out over all segments — so overlapping tiers cost + fan-out, not correctness. + +
- [ ] You wrote answers to all five questions in notes.md, including which of WAND's needs `TermInfo.doc_freq` serves. +
check + + Five answers in notes.md; the doc_freq one names idf (and hence + WAND's per-term ceiling `idf·(K1+1)`) as the input made free by + df riding in the dictionary. + +
## References **Code** -- [tantivy](https://github.com/quickwit-oss/tantivy) — the anchors - above: `src/tokenizer/tokenizer.rs`, - `src/termdict/fst_termdict/termdict.rs`, - `src/postings/term_info.rs`, `src/postings/compression/mod.rs`, - `src/postings/skip.rs`, `src/query/bm25.rs`, - `src/query/boolean_query/block_wand_union.rs`, - `src/indexer/log_merge_policy.rs` — the 90-minute order above is - the recommended pass +- [tantivy](https://github.com/quickwit-oss/tantivy) `@7152d53` — + the anchors above: `src/tokenizer/tokenizer.rs:9-11`, + `src/termdict/fst_termdict/termdict.rs:25,46,92`, + `src/postings/term_info.rs:9-16`, + `src/postings/compression/mod.rs:3,36-46`, + `src/postings/skip.rs:93,175-181,186-187`, + `src/query/bm25.rs:8-9,52-68`, + `src/query/boolean_query/block_wand_union.rs:16-43`, + `src/indexer/log_merge_policy.rs:20-26` — the 90-minute order above + is the recommended pass diff --git a/topics/23-fulltext/reading-zobel-moffat.md b/topics/23-fulltext/reading-zobel-moffat.md index 53d8eb6..8d1b272 100644 --- a/topics/23-fulltext/reading-zobel-moffat.md +++ b/topics/23-fulltext/reading-zobel-moffat.md @@ -1,14 +1,21 @@ # Inverted indexes: the whole design space in one survey -Zobel & Moffat's CSUR 2006 survey compresses 30 years of IR -engineering into 50 coherent pages. Read it as "the B-tree paper" -of text indexing: everything since (Lucene, tantivy, RediSearch) is -an implementation of choices this paper enumerates — which makes it -the right first chapter of this topic. Before you open it, this -chapter builds each axis of the design space from zero — what an -inverted index even is, what a posting carries, why deltas compress, -why construction is a merge, and how queries actually walk the -lists — so the survey reads as a map instead of a wall. +Zobel & Moffat's ACM Computing Surveys (2006) survey compresses 30 +years of IR engineering into ~50 coherent pages. Read it as "the +B-tree paper" of text indexing: everything since (Lucene, tantivy, +RediSearch) is an implementation of choices this paper enumerates — +which makes it the right first chapter of this topic. Before you open +it, this chapter builds each axis of the design space from zero — +what an inverted index even is, what a posting carries, why deltas +compress, why construction is a merge, and how queries actually walk +the lists — so the survey reads as a map instead of a wall. + +A note on citing it: this survey is paywalled (ACM Digital Library), +and I could not open a verified copy while writing this guide, so the +map below labels the survey's parts by *theme* rather than by section +number — confirm the exact §N against your own copy. Everything +attributed to a measured number or a line of code, by contrast, is +verified against this repo. ## The problem in one sentence @@ -22,6 +29,9 @@ size/speed/updatability trade. ### Step 1 — the inverted index: flip document→words into word→documents +> **In:** a corpus that maps each document to the words it contains. +> **Out:** the inverse map — each term to its sorted list of documents (its posting list) — so a query fetches lists instead of scanning text. + An **inverted index** stores, for every **term** (a normalized word produced by an analyzer: tokenize → lowercase → stem → drop stopwords), the sorted list of documents containing it — the @@ -39,27 +49,30 @@ this one structure: posting order: doc-sorted ─── supports AND/WAND skipping (everyone) frequency-sorted / impact-sorted ─── early termination - (§8; block-max WAND got the best of both) + (block-max WAND later got the best of both) compression: Golomb/Rice → variable-byte → word-aligned (Simple-9) (2006's menu; today: PForDelta / bitpacking / roaring) construction: in-memory inversion → sort-based → MERGE-BASED - (§5: build runs, merge them = Lucene segments = LSM) + (build runs, merge them = Lucene segments = LSM) update: rebuild / merge / in-place - (§7 concludes merge wins — Lucene's whole architecture) + (the survey concludes merge wins — Lucene's architecture) ``` Steps 2–6 take these axes one at a time. ### Step 2 — granularity: what each posting carries +> **In:** the choice of what to record per (term, document) pair. +> **Out:** a ladder — ids → +frequencies → +positions → +fields — where each rung buys query types with index bytes. + A posting can be just a doc id, or a doc id plus payload — and each addition buys query types with index bytes: - **doc ids only** — boolean AND/OR/NOT; the filter lane. -- **+ frequencies** (how often the term occurs in that doc) — +- **+ frequencies** (**tf**: how often the term occurs in that doc) — enables ranking (BM25 needs tf; next chapter). - **+ positions** (word offsets within the doc) — enables phrase ("quick fox" adjacent) and proximity queries, at ~3× the index @@ -68,67 +81,91 @@ addition buys query types with index bytes: weighting and filtering. This ladder is literally a directory listing in RediSearch's Rust -crate (eleven codecs from `doc_ids_only` to `full` — +crate (ten codec modules from `doc_ids_only` to `full` — reading-redisearch.md). The cost rule: pay for the payload only where a query type needs it. ### Step 3 — posting order: doc-sorted vs impact-sorted +> **In:** the freedom to store a posting list in any order. +> **Out:** doc-sorted (cheap intersection + skipping) vs impact-sorted (trivial early termination but no ordered merge) — a fork block-max WAND later reconciles. + Doc-sorted lists (postings ordered by doc id) make intersection cheap — two sorted lists merge in one pass, and a cursor can *skip ahead* to any doc id. **Impact-sorted** lists (postings ordered by -score contribution, best first) make top-k trivially early-terminating -— read from the front until the tail can't matter — but wreck AND: -neither list is in id order, so intersection needs a hash. 2006 -presents them as a fork in the road; the resolution came later — -block-max WAND (this topic's third chapter) keeps doc-sorted lists -and bolts per-block impact metadata on top, getting both. +score contribution, best first) make top-k trivially +early-terminating — read from the front until the tail can't matter +— but wreck AND: neither list is in id order, so intersection needs +a hash. 2006 presents them as a fork in the road; the resolution +came later — block-max WAND (this topic's third chapter) keeps +doc-sorted lists and bolts per-block impact metadata on top, getting +both. ### Step 4 — compression: store the gaps, not the ids +> **In:** a doc-sorted list of 32-bit ids, mostly redundant because they're increasing. +> **Out:** delta (gap) coding whose values are small exactly where lists are long (Zipf) — a few bits per posting instead of 32. + Doc-sorted ids compress because you store **deltas** (gaps between consecutive ids) instead of raw 32-bit ids — and Zipf's law makes -the gaps small exactly where the lists are long: a term appearing -in half the docs has average gap 2, fitting in 2–3 bits instead of -32. The 2006 menu is Golomb/Rice (bit-optimal, slow), -variable-byte (byte-aligned, fast), word-aligned Simple-9; today's -answers are 128-block bitpacking (tantivy), PForDelta, and roaring -(this topic's fourth chapter). Why it matters: postings dominate -index size, and decompression speed is the scan speed of the whole -query engine — pick wrong and topic 17's GB/s ceiling drops by 10×. +the gaps small exactly where the lists are long. The gap is inverse +to frequency; worked for a term in half the corpus: + +``` +avg gap ≈ N / df (uniform placement) +df = N/2: gap ≈ 2 → the delta '2' needs 2 bits, not 32 (16× win) +df = N/100: gap ≈ 100 → ~7 bits +df = 1: gap = doc id → the full 32 bits (rare terms don't compress) +``` + +The 2006 menu is Golomb/Rice (bit-optimal, slow), variable-byte +(byte-aligned, fast), word-aligned Simple-9; today's answers are +128-block bitpacking (tantivy), PForDelta, and roaring (this topic's +fourth chapter). Why it matters: postings dominate index size, and +decompression speed is the scan speed of the whole query engine — +pick wrong and topic 17's GB/s ceiling drops by 10×. ### Step 5 — construction and update: it's an LSM +> **In:** a corpus too big to invert into one in-memory map. +> **Out:** invert what fits, flush a sorted run, repeat, merge runs — and keep updates as new runs merged in the background: topic 4's LSM, independently rediscovered. + You can't build a big inverted index by inserting into one giant -in-memory map — it doesn't fit. §5's merge-based construction: -invert as much as fits in RAM, flush the sorted **run** to disk, -repeat, then merge runs into the final index. §7 reaches the -matching update conclusion: of rebuild / in-place / merge, **merge -wins** — keep new documents in a RAM index, flush as immutable -runs, merge in the background. +in-memory map — it doesn't fit. The survey's merge-based +construction: invert as much as fits in RAM, flush the sorted **run** +to disk, repeat, then merge runs into the final index. Its +maintenance discussion reaches the matching update conclusion: of +rebuild / in-place / merge, **merge wins** — keep new documents in a +RAM index, flush as immutable runs, merge in the background. That is topic 4's LSM tree, rediscovered independently: run = memtable flush, merge pass = compaction, immutable segments + tombstoned deletes. Lucene's entire architecture (and tantivy's — -this topic's fifth chapter) is §5 + §7 productionized. Inverted -indexes are cheap to build and expensive to update in place — -exactly the LSM bet. +this topic's fifth chapter) is the survey's construction + +maintenance sections productionized. Inverted indexes are cheap to +build and expensive to update in place — exactly the LSM bet. ### Step 6 — query evaluation: TAAT vs DAAT +> **In:** a multi-term query and two posting lists to combine. +> **Out:** two traversal orders — term-at-a-time (simple, no skipping, our oracle) and doc-at-a-time (needs doc-sorted lists, enables WAND's skipping). + Two ways to walk multiple posting lists: - **TAAT** (term-at-a-time): process one term's *entire* list before the next, accumulating partial scores per doc in a map of **accumulators**. Simple, sequential, cache-friendly — and no skipping is possible, since you don't know a doc's full score - until every term has been walked. Our `oracle_topk`, and the - baseline every later chapter tries to beat: + until every term has been walked. This is our provided oracle + (`bm25::oracle_topk`), and the baseline every later chapter tries + to beat: ```rust -// term-at-a-time: walk each term's WHOLE list, accumulate per doc +// ILLUSTRATION — the repo's real TAAT oracle is bm25::oracle_topk at +// experiments/src/bm25.rs:28-48 (walk every posting, accumulate in a +// HashMap, sort desc, truncate to k). This is a faithful sketch of it. fn taat_topk(terms: &[PostingList], k: usize) -> Vec<(DocId, f32)> { - let mut acc: HashMap = HashMap::new(); // §6's accumulators + let mut acc: HashMap = HashMap::new(); // the accumulators for t in terms { for p in t.postings() { // every posting, every term — *acc.entry(p.doc).or_default() // no skipping possible @@ -136,7 +173,7 @@ fn taat_topk(terms: &[PostingList], k: usize) -> Vec<(DocId, f32)> { } } top_k(acc, k) - // §6's insight: CAP the accumulator map (~1% of docs) and lose + // the survey's insight: CAP the accumulator map (~1% of docs) and lose // almost nothing — the 2006 answer to what WAND later solved exactly } ``` @@ -146,45 +183,49 @@ fn taat_topk(terms: &[PostingList], k: usize) -> Vec<(DocId, f32)> { needs doc-sorted lists (Step 3), and enables skipping: that's WAND's home. -§6's accumulator-limiting trick — allow only ~1% of docs to hold -accumulators, lose almost no ranking quality — is the heuristic -2006 answer to bounding work; WAND (Step 3's lineage, §8) is the -exact answer. Measured stakes from fts_bench: TAAT on -common∧rare (100K postings) takes 6.34 ms even though the rare -term's idf ≈ 9 means almost none of the common term's postings can -reach the top-10 — all that work is provably skippable. +The survey's accumulator-limiting trick — allow only ~1% of docs to +hold accumulators, lose almost no ranking quality — is the heuristic +2006 answer to bounding work; WAND (Step 3's lineage) is the exact +answer. Measured stakes from fts_bench (notes.md): TAAT on +common∧rare ([t0, t12000], 99,964 postings) takes 6.34 ms even +though the rare term's idf ≈ 7.1 (repo bm25.rs formula, df=83) means +almost none of the common term's postings can reach the top-10 — +all that work is provably skippable. ## How to read the paper (with the concepts in hand) -50 pages, but it's a survey — the section map, with the step each -one expands: +~50 pages, but it's a survey — read it by theme, mapping each part to +the step it expands. (The survey numbers these sections; I've labeled +them descriptively rather than assert §N I couldn't verify against an +open copy — match them to your printout as you go.) -| section | why (step) | +| survey theme | why (step) | |---|---| -| §2-3 | vocabulary + postings anatomy; the doc-id vs word-position granularity trade (1, 2) | -| §4 | compression: deltas are what make postings compressible at all — Zipf gives small gaps for common terms (4) | -| §5 | merge-based construction — recognize topic 4's LSM before Lucene made it famous (5) | -| §6 | query eval: term-at-a-time vs doc-at-a-time (our oracle is TAAT, WAND is DAAT); the accumulator-limiting trick (6) | -| §7 | index maintenance — why everyone chose immutable segments + merge (5) | -| §8 | ranked retrieval + early termination — the WAND lineage starts here (3, 6) | - -Read §2-3 fast, slow down for §5-§7 (the architecture payload), and -treat §8 as the setup for the block-max WAND chapter. The -compression specifics in §4 are 2006's menu — read for the *why* +| vocabulary + postings anatomy | the doc-id vs frequency vs word-position granularity trade (1, 2) | +| index compression | deltas are what make postings compressible at all — Zipf gives small gaps for common terms (4) | +| merge-based construction | recognize topic 4's LSM before Lucene made it famous (5) | +| query evaluation | term-at-a-time vs doc-at-a-time (our oracle is TAAT, WAND is DAAT); the accumulator-limiting trick (6) | +| index maintenance | why everyone chose immutable segments + merge (5) | +| ranked retrieval + early termination | the WAND lineage starts here (3, 6) | + +Read the vocabulary/postings part fast, slow down for +construction + maintenance (the architecture payload), and treat the +ranked-retrieval part as the setup for the block-max WAND chapter. +The compression specifics are 2006's menu — read for the *why* (deltas + Zipf), not the codec details. ## Questions (answer in notes.md) 1. Delta+compress works because Zipf makes common-term gaps small. - What's the expected gap for a term with df = n/2, and why does - bitpacking 128-blocks (tantivy) beat per-posting varint - (RediSearch) on exactly those terms? -2. §6's capped accumulators vs WAND: both bound work; which gives an - exactness guarantee and what does the other buy instead? -3. Merge-based construction (§5) vs topic 4's LSM: map runs/merge - passes onto memtable/flush/compaction. Where does Lucene's - tiered merge policy differ from leveled compaction and why does - full-text tolerate it? + What's the expected gap for a term with df = n/2 (worked above to + ≈2), and why does bitpacking 128-blocks (tantivy) beat per-posting + varint (RediSearch) on exactly those terms? +2. The survey's capped accumulators vs WAND: both bound work; which + gives an exactness guarantee and what does the other buy instead? +3. Merge-based construction vs topic 4's LSM: map runs/merge passes + onto memtable/flush/compaction. Where does Lucene's tiered merge + policy differ from leveled compaction and why does full-text + tolerate it? 4. Positions multiply index size ~3×. For M23's node/edge property search, when do you actually need them (phrase queries on `description` props?) and what's the cheaper substitute? @@ -194,16 +235,72 @@ compression specifics in §4 are 2006's menu — read for the *why* ## Done when +Answer each before unfolding it. + - [ ] You can explain granularity: what each posting carries and what that costs. +
the ladder + + doc ids (boolean) → +frequencies (BM25 ranking) → +positions + (phrase/proximity, ~3× size) → +fields (per-field weighting). Each + rung buys query types with index bytes; pay only where a query type + needs it. RediSearch encodes this as ten codec modules. + +
- [ ] You can state the difference between doc-sorted and impact-sorted postings and which query strategy each enables. +
the fork + + Doc-sorted → cheap ordered intersection + skipping (DAAT/WAND). + Impact-sorted → trivial top-k early termination but no ordered + merge (intersection needs a hash). Block-max WAND keeps doc-sorted + order and adds per-block impact metadata to get both. + +
- [ ] You can explain why storing gaps works, and how Zipf makes it work. +
gaps + Zipf + + Store deltas between consecutive ids, not the ids. Average gap ≈ + N/df, so the *long* lists (common terms, big df) have the *smallest* + gaps — a term in half the corpus has gap ≈2, ~2 bits vs 32. Zipf + concentrates postings in a few common terms, so this wins on the + bytes that dominate. + +
- [ ] You can explain the TAAT/DAAT distinction and which one this topic's oracle lane implements. +
traversal order + + TAAT walks each term's whole list into per-doc accumulators — no + skipping. DAAT advances one cursor per term in doc-id lockstep, + finishing each doc — enables skipping. The repo oracle + (`bm25::oracle_topk`, bm25.rs:32) is TAAT; WAND is DAAT. + +
- [ ] You can say what capped accumulators bound and how that differs from WAND's guarantee. +
heuristic vs exact + + Capping accumulators (~1% of docs) bounds work by dropping most + docs' partial scores — a heuristic that loses a little ranking + quality. WAND bounds work via true score ceilings and returns the + *exact* top-k. Same goal, one approximate and one safe. + +
- [ ] You wrote answers to all five questions in notes.md. +
check + + Five answers in notes.md, each tied to a concept above or a measured + repo number — the df=N/2 gap (Q1) worked, the accumulator-vs-WAND + exactness contrast (Q2) stated. + +
## References **Papers** - Zobel, Moffat — "Inverted Files for Text Search Engines" (ACM - Computing Surveys 2006) — read §2-8 with the section map above; - §5 and §7 are where Lucene's architecture comes from + Computing Surveys, 2006) — the whole design-space map above; + construction + maintenance are where Lucene's architecture comes + from. Paywalled (ACM Digital Library); read it against the thematic + map above. + +**Code** +- This repo — `experiments/src/bm25.rs:28-48` `oracle_topk`, the + term-at-a-time baseline the later chapters beat diff --git a/topics/24-graph-algorithms/README.md b/topics/24-graph-algorithms/README.md index 72c26e7..b77ce5a 100644 --- a/topics/24-graph-algorithms/README.md +++ b/topics/24-graph-algorithms/README.md @@ -49,7 +49,7 @@ RMAT scale 16 (n=65,536, m=1.82M directed, max deg 9,751) vs uniform | Dijkstra ×3 sources | 33.7 ms, 343K pops | — | | CC union-find | 18,844 comps, 4.2 ms, all m edges | — | -The TC row is the whole "skew matters" lecture: same n and m, 2883× +The TC row is the whole "skew matters" lecture: same n and m, 2882× more triangles — hub neighborhoods intersect. Any TC benchmark on uniform data measures a different algorithm. diff --git a/topics/24-graph-algorithms/experiments/.gitignore b/topics/24-graph-algorithms/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/24-graph-algorithms/experiments/.gitignore +++ b/topics/24-graph-algorithms/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/24-graph-algorithms/notes.md b/topics/24-graph-algorithms/notes.md index f1fd903..4adf5e6 100644 --- a/topics/24-graph-algorithms/notes.md +++ b/topics/24-graph-algorithms/notes.md @@ -2,6 +2,13 @@ ## Baseline (provided code, Apple M3 Pro, measured 2026-07-10) +> [FINDINGS.md](../../FINDINGS.md) row 24 reports a later run of the +> same lanes: **447 ms** RMAT and **195 ms** uniform for triangle count. +> The counts are identical (15,645,988 and 5,428), so the two runs +> differ only in timing — cite one run or the other by name and never +> average them. Re-run `./verify.sh 24` before treating a millisecond +> figure below as current. + Graphs: RMAT scale 16 (n=65,536, m=1,819,338 directed after symmetrize+dedup, max deg 9,751) vs uniform (same n, m=2,096,564, max deg 59). Build 258 ms. @@ -13,7 +20,7 @@ max deg 59). Build 258 ms. | Dijkstra ×3 sources | 33.7 ms, 342,909 pops | | | CC union-find | 18,844 components, 4.2 ms, all m inspected | | -- TC: same n, comparable m, **2,883× more triangles** on RMAT — hub +- TC: same n, comparable m, **2,882× more triangles** on RMAT — hub neighborhoods intersect; uniform graphs have nothing to count. Per-triangle cost is what the skew hides: rmat does 24 ns/triangle only because intersections are fat; uniform pays 29 µs/triangle. diff --git a/topics/24-graph-algorithms/reading-brandes.md b/topics/24-graph-algorithms/reading-brandes.md index 10909a0..e45ca23 100644 --- a/topics/24-graph-algorithms/reading-brandes.md +++ b/topics/24-graph-algorithms/reading-brandes.md @@ -10,18 +10,29 @@ collapses the whole thing. Our `bc::brandes` stub implements it against the O(n³) definitional oracle; gapbs's `bc.cc` and LAGraph's `LAGr_Betweenness.c` show the two production shapes. +*Sources pinned (resources/codebases.md): gapbs @`b5e3e19`, LAGraph +@`e2539e2`. Line anchors below were re-checked against those trees with +`tools/pinned-source.py`. The complexity claims are Brandes 2001 +(J. Math. Sociology); the operation counts are computed from this +topic's `notes.md` graph (n = 65,536, m = 1,819,338 directed edges).* + ## The problem in one sentence Betweenness as defined sums over all (source, target) pairs — on our 65,536-vertex RMAT that is **~2.8 × 10¹⁴ elementary operations -(n³)**, and Brandes gets the identical numbers for roughly -n × m ≈ 1.2 × 10¹¹, three orders of magnitude less, without -approximating anything. +(n³ = 65,536³ = 2.81 × 10¹⁴)**, and Brandes gets the identical numbers +for roughly n × m = 65,536 × 1,819,338 ≈ 1.19 × 10¹¹ — about 2,360× +(≈ three orders of magnitude) less, without approximating anything. ## The concepts, step by step ### Step 1 — what betweenness measures: traffic through a vertex +> **In:** a graph, and a vertex v to score. +> **Out:** two counts that make "traffic through v" precise — σ_st (the +> number of shortest paths s→t) and σ_st(v) (how many of those pass +> through v). + Betweenness centrality scores a vertex by how many shortest paths pass through it — a bridge vertex connecting two clusters lies on *every* cross-cluster shortest path and scores enormously; a leaf lies on none @@ -43,6 +54,10 @@ n² vertex pairs. ### Step 2 — the definitional cost: O(n³), and why we keep it anyway +> **In:** the definition from Step 1. +> **Out:** its literal cost — O(n³) time and O(n²) memory — and the +> reason we keep the slow version anyway: it is the correctness oracle. + Computing bc directly means: for every pair (s, t), find all shortest paths, attribute fractions to every interior vertex — an all-pairs computation with a triple loop, O(n³) time and O(n²) memory for the @@ -55,6 +70,11 @@ the right to sample. ### Step 3 — counting paths with one BFS: σ flows along the BFS DAG +> **In:** one source s. +> **Out:** depths and shortest-path counts σ_s(v) for *every* vertex, +> from a single O(E) BFS — which kills the "for every t" half of the +> pair sum. + The number of shortest paths from a fixed source s to every vertex comes out of a single BFS (breadth-first traversal that labels each vertex with its **depth** — hop distance from s). The edges that go @@ -81,6 +101,11 @@ is Step 4's job. ### Step 4 — the dependency: fold the sum over targets +> **In:** the per-source path counts σ_s from Step 3. +> **Out:** the *dependency* δ_s(v) = Σ_t σ_st(v)/σ_st — a pure +> regrouping of the pair sum that names the inner sum over targets so it +> can be computed without enumerating targets. + Brandes' move is to fix the source s and give a name to the entire inner sum over targets — the **dependency** of s on v: @@ -97,6 +122,10 @@ order of summation moved. ### Step 5 — the recurrence: one backward sweep per source +> **In:** the dependency δ_s from Step 4 and the BFS DAG from Step 3. +> **Out:** the recurrence that computes δ_s(v) for *all* v in one +> deepest-first backward sweep — the entire speedup, O(V·E) total. + Every shortest path from s through v continues into exactly one DAG successor w of v — so partition the paths-through-v by that successor, and δ_s(v) becomes a sum over v's successors of already-computed @@ -116,14 +145,35 @@ quantities: bc(v) = Σ_s δ_s(v). n sources × O(E) each = O(V·E). ``` -The recurrence is the entire paper — derive it once by hand -(partition shortest s→t paths through v by v's DAG successor w; the -1 accounts for t=w itself: paths *ending at* w also pass through v). -The factor σ_sv/σ_sw is v's share of the traffic entering w. Because -δ of a vertex needs δ of its successors (which are deeper), the sweep -must run deepest-first. Transcribed: +The recurrence is the entire paper (Brandes 2001, Theorem 6) — derive +it once by hand (partition shortest s→t paths through v by v's DAG +successor w; the 1 accounts for t=w itself: paths *ending at* w also +pass through v). The factor σ_sv/σ_sw is v's share of the traffic +entering w. Because δ of a vertex needs δ of its successors (which are +deeper), the sweep must run deepest-first. + +Work it on a concrete graph — source s, edges s–a, s–b, a–c, b–c, c–d: + +``` + s BFS depths: s=0, a=b=1, c=2, d=3 + / \ σ: σ(s)=1; σ(a)=σ(b)=1; σ(c)=σ(a)+σ(b)=2; σ(d)=σ(c)=2 + a b + \ / backward sweep, deepest first (δ starts at 0): + c δ(d) = 0 (leaf) + | δ(c) = σ(c)/σ(d)·(1+δ(d)) = 2/2·(1+0) = 1 (w=d) + d δ(a) = σ(a)/σ(c)·(1+δ(c)) = 1/2·(1+1) = 1.0 (w=c) + δ(b) = σ(b)/σ(c)·(1+δ(c)) = 1/2·(1+1) = 1.0 + this single source s adds to bc: a+=1.0, b+=1.0, c+=1, d+=0 +``` + +The 1/2 is the σ-fraction (a carries half the traffic into c); the +`(1+δ(c))` is the `+1` for t=c itself plus c's onward dependency +(δ(c)=1, from the c→d traffic). Transcribed as pseudocode: ```rust +// ILLUSTRATION — not quoted; the reader's own stub. The real +// deepest-first sweep is gapbs bc.cc:123-134 (recurrence at :130); +// our version lands in experiments/src/bc.rs:83 (brandes). // after a forward BFS from s: depth[], sigma[] (path counts), // and order = vertices sorted by depth fn accumulate(bc: &mut [f64], order: &[u32], g: &Csr, @@ -144,10 +194,16 @@ fn accumulate(bc: &mut [f64], order: &[u32], g: &Csr, Per source: one forward BFS + one backward sweep, both O(E). Over all n sources: O(V·E) time, O(V) extra memory per source — the n² all-pairs tables of Step 2 never exist. When even n sources is too -many, sample k of them and scale — gapbs defaults to 16. +many, sample k of them and scale. (gapbs's `bc.cc` runs **1** source +by default — `CLIterApp(..., 1)` at bc.cc:234; the GAP *spec* is the +one that samples, 16 trials of 4 sources each — see reading-gap.md.) ### Step 6 — the two production shapes: a bitmap vs a batch +> **In:** Steps 3–5 as a single-source algorithm. +> **Out:** the two production shapes — gapbs's per-source successor +> *bitmap* and LAGraph's *batched* ns×n source matrix. + Both production codes implement Steps 3–5; they diverge on how the backward sweep answers "is (v, w) a DAG edge" and on how many sources run at once: @@ -155,7 +211,7 @@ run at once: | | gapbs bc.cc | LAGraph LAGr_Betweenness.c | |---|---|---| | forward | `PBFS` (:51): CAS on depths, records `succ` BITMAP (:76) — "is (u,v) a DAG edge" = one bit | `frontier`/`paths` are ns×n MATRICES (:110-164) — a BATCH of sources advances as one masked mxm | -| σ | `path_counts` accumulated at depth boundaries (`depth_index` slices the BFS queue by level) | `paths += frontier` per level, FP64 semiring | +| σ | `path_counts` accumulated at depth boundaries (`depth_index` slices the BFS queue by level) | `paths += frontier` per level, `plus_first_fp64` semiring (:168) | | backward | deepest-first over `depth_index`, reads `succ` | transposed mxm per level with `bc_update` matrix | | sampling | k sources, scores scaled | `sources` array — batch size = ns | | wins | per-edge constants, one bitmap read per edge | no atomics; 4-32 sources amortize each matrix pass | @@ -167,6 +223,11 @@ code cannot (it would need 32 separate BFS queues). ### Step 7 — what breaks in practice: the four traps +> **In:** a working single-source Brandes. +> **Out:** the four boundary conditions (DAG-edge test, σ overflow, +> unreachable sources, directed/undirected convention) that make the +> stub *wrong* before they make it slow. + The stub's failure modes are all boundary conditions of Steps 3–5: 1. σ must be accumulated ONLY along depth+1 edges (BFS DAG), and @@ -217,12 +278,69 @@ The stub's failure modes are all boundary conditions of Steps 3–5: ## Done when +Answer each before unfolding it. + - [ ] You can explain what betweenness measures and why the definitional cost is O(n³). -- [ ] You can derive the dependency recurrence from the definition, using the partition-by-predecessor argument. +
Answer + + It scores a vertex by its share of shortest-path traffic: + bc(v) = Σ_{s≠v≠t} σ_st(v)/σ_st. The definition ranges over all n² + ordered pairs (s,t) and each pair needs its shortest paths and interior + attributions — an all-pairs computation, O(n³) time (≈2.8×10¹⁴ ops on + this topic's n=65,536 graph). + +
+- [ ] You can derive the dependency recurrence from the definition, using the partition-by-successor argument. +
Answer + + Fix s and define δ_s(v) = Σ_t σ_st(v)/σ_st. Partition the shortest s→t + paths through v by the DAG successor w of v they use; each contributes + v's share of w's traffic, σ_sv/σ_sw, times (1+δ_s(w)). So + δ_s(v) = Σ_{w : v∈pred_s(w)} (σ_sv/σ_sw)·(1+δ_s(w)) (Brandes 2001, + Theorem 6). The **+1** is t=w itself: paths ending at w also pass + through v. + +
- [ ] You can explain how one BFS counts paths along the level DAG. +
Answer + + BFS from s labels depths; the depth-d→depth-(d+1) edges are the BFS + DAG. With σ_s(s)=1 and σ_s(v)=Σ σ_s(u) over DAG predecessors u, + a single O(E) sweep gives σ for all targets at once — killing the + "for every t" loop. + +
- [ ] You can say why the brute-force version is O(n²) in *memory* as well as O(n³) in time. +
Answer + + `bc_brute` materializes all-pairs depths and σ tables — n×n each — to + attribute fractions across every pair, so O(n²) storage. Brandes keeps + only per-source arrays (depth, σ, δ), O(V) extra beyond the graph. + For a *correctness* oracle the O(n²) memory is acceptable because it + only ever runs on small test graphs. + +
- [ ] You can name the four practical traps and say which one bites on a scale-free graph like this topic's RMAT (max degree 9751). +
Answer + + (1) accumulate σ only along depth+1 edges and back-propagate strictly + deepest-first; (2) σ overflows integers on dense diamonds — use floats + for the counts, the ratio stays exact; (3) unreachable vertices have + depth −1, contribute nothing, never divide by σ=0; (4) directed-sum on + a symmetric graph double-counts — halve to match undirected tools. On + RMAT the σ-overflow trap bites: the degree-9,751 hub creates deep + diamonds where σ multiplies fast. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md has your worked recurrence derivation (Q1), the + LLC/oracle-fit reasoning (Q2), the bitmap memory-touch count (Q3), the + ns batch-size sweet spot (Q4), and the FalkorDB stale-matrix decision + (Q5). + +
## References diff --git a/topics/24-graph-algorithms/reading-delta-stepping.md b/topics/24-graph-algorithms/reading-delta-stepping.md index 79e27ad..2d6eaea 100644 --- a/topics/24-graph-algorithms/reading-delta-stepping.md +++ b/topics/24-graph-algorithms/reading-delta-stepping.md @@ -9,17 +9,28 @@ reading — then compares the two production implementations (gapbs's frontier version and LAGraph's algebraic one) that our `sssp::delta_stepping` stub sits between. +*Sources pinned (resources/codebases.md): gapbs @`b5e3e19`, LAGraph +@`e2539e2`; anchors re-checked with `tools/pinned-source.py`. The bound +and its preconditions are Meyer & Sanders (J. Algorithms 2003); the +343K-pop and timing figures are this topic's `notes.md`.* + ## The problem in one sentence Dijkstra settles exactly one vertex at a time, so on our 65,536-vertex -RMAT it is a strictly sequential chain of **343K heap pops** — the -answer is perfectly work-efficient and perfectly unparallelizable, and -Δ-stepping asks how much wasted work buys how much parallelism. +RMAT it is a strictly sequential chain of **343K heap pops** (measured: +342,909 pops across its 3 source runs, ~114K per source, notes.md) — +the answer is perfectly work-efficient and perfectly unparallelizable, +and Δ-stepping asks how much wasted work buys how much parallelism. ## The concepts, step by step ### Step 1 — SSSP and relaxation: the one move every algorithm shares +> **In:** a weighted graph and a source vertex. +> **Out:** the single primitive every SSSP algorithm is built from — +> *relaxing* an edge — and the definition of *wasted work* (a +> relaxation later overwritten by a better one). + Single-source shortest paths (SSSP) computes, for every vertex, the minimum total edge weight of any path from one source vertex. Every SSSP algorithm maintains a **tentative distance** per vertex — the @@ -33,6 +44,11 @@ whole chapter trades in. ### Step 2 — Dijkstra: perfect order, zero parallelism +> **In:** relaxation from Step 1. +> **Out:** Dijkstra's rule (always relax out of the global-minimum +> vertex) — zero wasted relaxations, but a strictly sequential chain of +> priority-queue pops. + Dijkstra's rule is: always relax out of the unfinished vertex with the smallest tentative distance. That vertex's distance can never improve again (all other paths to it must pass through vertices at least as @@ -46,6 +62,11 @@ vertex — which Dijkstra's correctness argument forbids. ### Step 3 — Bellman-Ford: zero order, perfect parallelism +> **In:** relaxation from Step 1, with the ordering dropped. +> **Out:** Bellman-Ford's opposite trade — every edge relaxed in any +> order, |V| rounds, embarrassing parallelism bought with embarrassing +> re-relaxation waste. + Bellman-Ford drops the ordering entirely: relax *every* edge, in any order, and repeat until no distance changes (at most |V| rounds). Every relaxation within a round is independent — embarrassingly @@ -57,6 +78,11 @@ waste, bought for the right to use every core. ### Step 4 — the dial: buckets of width Δ +> **In:** Dijkstra (Step 2) and Bellman-Ford (Step 3) as the two ends. +> **Out:** the dial between them — buckets of width Δ, Dijkstra order +> *between* buckets and Bellman-Ford freedom *within* one — plus its +> two degenerate endpoints (Δ→min = Dijkstra, Δ→∞ = Bellman-Ford). + Δ-stepping interpolates: group vertices into **buckets** by tentative distance in bands of width Δ, process buckets in order, and inside a bucket relax freely in parallel — accepting Bellman-Ford-style waste @@ -86,12 +112,20 @@ amount of parallel width — our stub's stats expose exactly that trade. ### Step 5 — the bucket loop, and the three traps inside it +> **In:** the dial from Step 4. +> **Out:** the drain-until-empty bucket loop and its three +> implementation traps (stale entries, lazy bin growth, refill during +> drain). + The machinery fits on one screen; the subtleties are the loop conditions. An edge of weight w < Δ (a "light" edge) can re-insert its target *into the bucket currently being drained*, so a bucket must be drained until empty, not iterated once: ```rust +// ILLUSTRATION — not quoted; the reader's own stub. The production +// frontier version is gapbs sssp.cc:87-116 (RelaxEdges at :69-85); +// ours lands in experiments/src/sssp.rs:45 (delta_stepping). fn delta_stepping(g: &Csr, src: u32, delta: u64) -> Vec { let mut dist = vec![u64::MAX; g.n]; dist[src as usize] = 0; let mut bins: Vec> = vec![vec![src]]; // bins[i] = [iΔ, (i+1)Δ) @@ -135,18 +169,47 @@ binary-heap Dijkstra that never does decrease-key. ### Step 6 — where the dial fails: diameter +> **In:** the dial from Step 4 and the paper's analysis. +> **Out:** the exact bound Δ=Θ(1/d) achieves, the three preconditions +> it rests on, and the graph family (high diameter) that voids it. + The paper's analysis promises that for random edge weights on low-diameter graphs (diameter = the longest shortest-path distance in hops) there is a Δ giving near-linear total work AND polylog depth — -both ends of the trade at once. The promise dies on road networks: -with a huge diameter, few vertices share any given distance band, so -the buckets are nearly empty at *every* Δ — there is no parallelism -for the dial to buy. That is exactly why the GAP suite includes a road -graph (see reading-gap.md, Step 3): one graph family flips the SSSP -ranking. +both ends of the trade at once. Precisely (Meyer & Sanders 2003, +Theorem 1 and Conclusions): + +``` + Given max degree d (or a random graph G(n, d/n)) and RANDOM edge + weights, the Δ = Θ(1/d)-stepping scheme runs in + O(d·n) expected sequential work (linear if d = O(1)) + O(log³n / log log n) time (phases) whp + + It rests on THREE preconditions, all stated in the paper: + 1. weights uniform in [0,1] — used ONLY in Lemma 3: a length-ℓ + path has total weight ≤ Δ with probability ≤ Δ^ℓ/ℓ!, so long + light paths (the sequential-chain risk) are unlikely. + 2. bounded max degree d — Δ is tuned as Θ(1/d). + 3. low diameter — the longest settling chain is ℓ_max = + O(log n / log log n) whp (Lemma 4); a big diameter makes ℓ_max + large and the phase count blows up. +``` + +The promise dies on road networks: with a huge diameter, few vertices +share any given distance band, so the buckets are nearly empty at +*every* Δ — there is no parallelism for the dial to buy. That is +exactly why the GAP suite includes a road graph (see reading-gap.md, +Step 3): one graph family flips the SSSP ranking. (Our RMAT weights are +uniform 1..=255, not [0,1], so the paper's Δ=Θ(1/d) rescales by the +weight range — Q1 has you predict the actual relaxation curve.) ### Step 7 — the algebraic reading: SSSP is MIN_PLUS matrix multiplication +> **In:** the bucket loop from Step 5. +> **Out:** the algebraic reading — one relaxation round *is* one +> (min,+) matrix-vector product — and the two production shapes it +> maps to (gapbs's frontier bins, LAGraph's semiring vxm). + Replace ordinary (+, ×) arithmetic with (min, +) — the "tropical semiring", where matrix multiply computes minimum path sums instead of dot products — and one relaxation round over the whole graph becomes @@ -158,16 +221,20 @@ The two production implementations are the two readings of this: | | gapbs sssp.cc | LAGraph LAGr_SingleSourceShortestPath.c | |---|---|---| | bucket | thread-local `vector` bins (:32-44), merged at sync points | `tmasked` sparse vector = current bucket (:100-142) | -| relax | explicit `RelaxEdges` with CAS-free benign races (:69-79) | one `GrB_vxm` with MIN_PLUS semiring (:151-185) per inner iteration | -| stale entries | left in old bins; skipped when drained (:44 — redundancy beats bookkeeping) | mask + select prune them algebraically | +| relax | explicit `RelaxEdges` with a lock-free CAS retry loop (:69-85) | one `GrB_vxm` with MIN_PLUS semiring (:151-185) per inner iteration | +| stale entries | left in old bins; skipped when drained — a vertex is processed only if `dist[u] >= delta*curr_bin` (:110); the :41-44 header argues redundancy beats bookkeeping | mask + select prune them algebraically | | light/heavy split | skipped entirely (re-relax instead) | skipped too; `Delta` is a GrB_Scalar knob | Same lesson as topic 20's BFS: the algebraic version is ~15 lines of semiring calls and inherits parallelism from the runtime; the -frontier version owns its memory layout and wins constants. The races -in gapbs are safe because `min` is idempotent and monotone — writing a -worse value twice or losing a race just means one more re-relaxation, -never a wrong answer. +frontier version owns its memory layout and wins constants. gapbs is +not race-*free*: each write goes through `compare_and_swap`, and a +thread that loses the race rechecks `dist` and retries the loop +(sssp.cc:74-83). The algebraic reason that loop is safe — and that +re-relaxation across threads never corrupts a distance — is that `min` +is an idempotent, monotone monoid: writing a worse value, losing a +race, or relaxing twice only ever triggers one more `min`, never a +wrong answer. ## How to read the paper (with the concepts in hand) @@ -194,9 +261,10 @@ never a wrong answer. 2. Δ=1 with integer weights: exactly which Dijkstra do you get, and why is it still cheaper than a binary heap (hint: Dial's algorithm, O(1) bucket ops)? -3. Why do thread-local bins + benign write races (gapbs :32) not - corrupt distances? What property of `min` makes the race safe — - and which GraphBLAS concept is that (idempotent monoid)? +3. Why do thread-local bins + the lock-free CAS retry loop (gapbs + sssp.cc:74-83) not corrupt distances? What property of `min` makes + a lost or repeated relaxation safe — and which GraphBLAS concept is + that (idempotent monoid)? 4. LAGraph does one vxm per INNER iteration — how does the number of vxm calls relate to (max_dist/Δ + reinsertions)? Where does the algebraic version pay that gapbs doesn't? @@ -207,12 +275,65 @@ never a wrong answer. ## Done when +Answer each before unfolding it. + - [ ] You can state relaxation as the one move all SSSP algorithms share. +
Answer + + For an edge u→v of weight w, if dist[u]+w < dist[v] then lower dist[v] + to dist[u]+w. Every SSSP algorithm is just a policy for *which* edges + to relax and *in what order*; a relaxation later overwritten by a + better one is wasted work. + +
- [ ] You can explain the Dijkstra/Bellman-Ford trade as order against parallelism, and where Δ sits on that dial. +
Answer + + Dijkstra imposes a total order (relax out of the global minimum) — zero + waste, zero parallelism. Bellman-Ford drops order — full parallelism, + up to |V|× re-relaxation. Δ-stepping keeps Dijkstra order *between* + Δ-width buckets and Bellman-Ford freedom *within* one, trading a + measurable amount of re-relaxation for parallel width. + +
- [ ] You can say exactly which algorithm Δ=1 gives you with integer weights. -- [ ] You can name the three traps inside the bucket loop and why benign write races are acceptable. +
Answer + + Δ=1 puts each integer distance in its own bucket, so buckets are + settled in strict increasing order — that is Dial's-bucket Dijkstra. + It is cheaper than a binary heap because bucket insert/extract is O(1) + (array indexing) rather than O(log n). + +
+- [ ] You can name the three traps inside the bucket loop and why a lost CAS is acceptable. +
Answer + + (1) skip stale entries whose dist has dropped below the current + bucket's floor; (2) grow the bins vector lazily — you don't know + max_dist/Δ in advance; (3) drain a bucket to empty, since light edges + (w<Δ) can refill it. A lost CAS in gapbs (sssp.cc:74-83) just rechecks + dist and retries; because `min` is an idempotent monotone monoid, the + extra relaxation can never write a wrong final value. + +
- [ ] You can write SSSP as MIN_PLUS matrix multiplication. -- [ ] You wrote answers to all five questions in notes.md, and predicted a Δ for weights uniform in 1..=255 before running the lane against the measured Dijkstra oracle (42.5 ms, 342 909 heap pops). +
Answer + + Over the tropical (min,+) semiring, one relaxation round is + dist' = dist min.+ (dist ⊗ A): every vertex takes the min over its + in-neighbours of dist[u]+w(u,v). Δ-buckets are a sparsity filter on + which entries of dist participate in each vxm. + +
+- [ ] You wrote answers to all five questions in notes.md, and predicted a Δ for weights uniform in 1..=255 before running the lane against the measured Dijkstra oracle (notes.md: 33.7 ms for 3 sources, 342,909 heap pops). +
Answer + + Done when notes.md has your relaxations-vs-Δ prediction filled in + *before* the run, plus the answers to Q2–Q5, and your prediction is + compared against the measured Dijkstra baseline (33.7 ms / 342,909 + pops across 3 sources). + +
## References diff --git a/topics/24-graph-algorithms/reading-gap.md b/topics/24-graph-algorithms/reading-gap.md index b2e2b19..e7a8dc3 100644 --- a/topics/24-graph-algorithms/reading-gap.md +++ b/topics/24-graph-algorithms/reading-gap.md @@ -10,19 +10,29 @@ kernel is, then the three graph properties — degree skew, diameter, and source luck — that let a single benchmark graph crown the wrong winner. +*Source pinned (resources/codebases.md): gapbs @`b5e3e19`; anchors +re-checked with `tools/pinned-source.py`. Trial counts are the GAP +paper (arXiv:1508.03619, Table 1); the triangle figures are this +topic's `notes.md`.* + ## The problem in one sentence Graph-algorithm performance depends on the input graph's *shape* so strongly that one benchmark graph ranks implementations backwards — -on our own bench, two graphs with identical n=65,536 and m=1.82M -contain **15,645,988 vs 5,428 triangles (a 2,883× difference)**, so a -triangle counter tuned on one is being measured on a different job on -the other. +on our own bench, two graphs with identical n=65,536 and m=1,819,338 +contain **15,645,988 vs 5,428 triangles — a ≈2,882× difference** +(15,645,988 / 5,428; notes.md), so a triangle counter tuned on one is +being measured on a different job on the other. ## The concepts, step by step ### Step 1 — a kernel: the unit of fair comparison +> **In:** the goal of comparing graph implementations fairly. +> **Out:** the *kernel* — an algorithm pinned by input/output only — and +> the 6-kernel × 5-graph matrix GAP runs so no data structure is +> privileged. + A **kernel** is a self-contained algorithm with a precisely specified input and output — specified tightly enough that any implementation, in any language over any data structure, can be timed on the same @@ -34,7 +44,8 @@ task. GAP picks six kernels, five graphs, and runs the full matrix graphs: twitter (skew) web (locality) road (diameter!) kron (RMAT synthetic) urand (uniform synthetic) │ - every kernel × every graph, 64 trials from random sources — + every kernel × every graph, many trials from random sources + (Table 1: BFS/SSSP 64, PR/CC 16, BC 16×4 sources, TC 3) — because ONE graph shape crowns the wrong winner: road kills delta-stepping's parallelism (long diameter), urand kills direction-optimizing BFS (no hubs), @@ -55,6 +66,11 @@ over-fit to the rest. ### Step 2 — degree skew: hubs change the work, not just the clock +> **In:** two graphs with identical n and m. +> **Out:** why *degree skew* (a power-law hub distribution) changes +> which algorithm you are effectively running, not just how fast — and +> why GAP ships both skewed and uniform graphs. + A vertex's **degree** is its edge count, and real-world graphs are **skewed**: degree follows a power law, so a handful of hub vertices carry a huge fraction of the edges while most vertices have a few. @@ -79,6 +95,12 @@ winner. ### Step 3 — diameter: how many rounds the algorithm must take +> **In:** a frontier algorithm that advances one distance-level per +> round. +> **Out:** why *diameter* sets the round count (and per-round frontier +> size the parallelism), so a road graph starves the parallelism that +> twitter/kron hand out. + The **diameter** is the longest shortest-path distance in the graph, measured in hops — and for any algorithm that advances a **frontier** (the set of vertices discovered in the current round of a traversal) @@ -99,31 +121,53 @@ implementation that looks great on twitter can crawl on road. One graph family flips the SSSP ranking — that's the suite's argument in one row. -### Step 4 — source luck: why 64 trials from random sources +### Step 4 — source luck: why many trials from random sources + +> **In:** the per-source kernels (BFS, SSSP, BC) on a skewed graph. +> **Out:** why source choice can swamp most optimizations, and GAP's +> defense — many trials from random non-zero-degree sources, all +> reported (Table 1: BFS/SSSP 64 trials/64 sources; BC 16 trials +> averaging 4 sources; whole-graph PR/CC 16, TC 3). Per-source kernels (BFS, SSSP, BC) start from a chosen vertex, and on a skewed graph the choice is worth more than most optimizations: -starting at a hub reaches the giant component in 2 hops; starting at -a degree-1 leaf adds rounds and shrinks early frontiers. Source -choice changes BFS/SSSP/BC time by **more than 10×** on skewed -graphs. GAP's rule: 64 trials from random sources, report ALL of -them — not the mean, not the best. Our bench uses 3 fixed sources — -upgrade when it matters. The cost of skipping this: a lucky source is -a silent 10× overstatement in your headline number. +starting at a hub reaches the giant component in ~2 hops; starting at +a degree-1 leaf adds rounds and shrinks early frontiers — the same +work, wildly different clock. GAP's rule (paper Table 1): BFS and SSSP +run **64 trials from 64 random sources**, BC runs **16 trials, each +averaging 4 sources**, and the whole-graph kernels run enough trials to +catch non-determinism (PR 16, CC 16, TC 3). Report ALL trials — not +the mean, not the best. Our bench uses 3 fixed sources — upgrade when +it matters. The cost of skipping this: one lucky hub source silently +overstates a per-source headline number. ### Step 5 — the spec binds: kernel specification ≠ implementation +> **In:** a kernel specified by input/output only. +> **Out:** how the spec *forks* implementations — e.g. GAP's PR spec +> (L1-error stop, ignore dangling vertices) is why `LAGr_PageRankGAP` +> exists as a separate function from textbook PR. + GAP specifies each kernel by input and output only, so algebraic codes (LAGraph runs GAP too) and frontier codes compete honestly — no data structure is privileged. But a spec is an interface, and interfaces bind implementations: `LAGr_PageRankGAP` exists as a -separate function because GAP's PR spec (stop on L1 error, handle -dangling vertices the way gapbs does) differs from textbook PR. -Benchmark specs fork implementations — remember that when you write -M22's lanes: whatever you specify is what everyone will build. +separate function because GAP's PR spec differs from textbook PR — it +stops when the summed score change drops below 10⁻⁴, and it **ignores +dangling (zero-out-degree) vertices** rather than redistributing their +rank, exactly as gapbs's `pr.cc` does. Textbook PR (and LAGraph's other +entry, `LAGr_PageRank`) instead redistributes sink rank each iteration +to keep the scores summing to 1. Benchmark specs fork implementations — +remember that when you write M22's lanes: whatever you specify is what +everyone will build. ### Step 6 — the baseline problem: reference code that is itself state of the art +> **In:** the classic benchmarking sin (beating a strawman baseline). +> **Out:** how GAP forecloses it — shipping gapbs, whose reference +> kernels are themselves state-of-the-art, each opening with a +> mini-paper header comment. + The classic benchmarking sin (topic 22) is beating a strawman baseline. GAP forecloses it by shipping gapbs: reference implementations that are themselves state-of-the-art single-node @@ -142,7 +186,7 @@ Each file's header comment = required reading (Step 6): | `src/bfs.cc` | direction-optimizing | topic 20's guide covers it — α=15, β=18 here | | `src/sssp.cc:87` | `DeltaStep` | thread-local bins (`:32` comment); `:44`: redundant relaxation is CHEAPER than removing stale entries — same lazy-deletion bet as our Dijkstra oracle | | `src/bc.cc:51` | Brandes | `PBFS` records a `succ` BITMAP (:76) so backprop tests "is w my BFS successor" in one bit — no depth recheck | -| `src/cc.cc:95` | Afforest | `:106` neighbor_rounds=2 link sweeps, `:69` SampleFrequentElement (1024 samples), `:129` final sweep skips the giant component | +| `src/cc.cc:95` | Afforest | `:106` neighbor_rounds=2 link sweeps, `:69` SampleFrequentElement (1024 samples), `:127` final sweep skips the giant component (`if (comp[u]==c) continue;`) | | `src/pr.cc:31-57` | pull PR | kDamp .85, L1-error stop; `pr_spmv.cc` is the same as one SpMV per iter — the algebraic identity made explicit | | `src/tc.cc:52-99` | ordered TC | `OrderedCount` after `RelabelByDegree` if `WorthRelabelling` (:75 samples degree skew) | @@ -151,8 +195,9 @@ Each file's header comment = required reading (Step 6): - The graph-selection discussion is Steps 2–3: for each of the five graphs, name the property (skew, locality, diameter) and which kernel ranking it exists to flip. -- The methodology section is Step 4 — 64 trials from random sources, - all reported. Steal it verbatim for M22/M24's lanes. +- The methodology section is Step 4 — many trials from random sources, + all reported (Table 1: 64 for BFS/SSSP, 16 for PR/CC/BC, 3 for TC). + Steal it for M22/M24's lanes. - The kernel specifications are Step 5 — notice how tightly PR's stopping condition is pinned, and why (specs bind implementations). - Then go to `src/` with the table above; the header comments (Step @@ -165,9 +210,11 @@ Each file's header comment = required reading (Step 6): and what property (diameter, degree variance) drives each flip? 2. sssp.cc:44 argues redundant relaxations beat precise bucket removal. Under what edge-weight distribution does that bet fail? -3. bc.cc approximates with 16 sources by default. On our RMAT - (18,844 components!), what systematic error does source sampling - introduce and how would you stratify? +3. gapbs's `bc.cc` runs 1 source by default (`CLIterApp(..., 1)` at + bc.cc:234); the GAP spec approximates BC from 4 sources per trial, + 16 trials (paper Table 1). On our RMAT (18,844 components!), what + systematic error does source sampling introduce and how would you + stratify? 4. pr.cc vs pr_spmv.cc: same math, different memory access. Which wins on kron and why (hint: pull = gather = topic 20's SpMV 16-19 GB/s lane)? @@ -177,12 +224,66 @@ Each file's header comment = required reading (Step 6): ## Done when +Answer each before unfolding it. + - [ ] You can explain what a kernel is and why the spec binds the kernel rather than the implementation. +
Answer + + A kernel is an algorithm pinned by input and output only, so any + implementation over any data structure can be timed on the identical + task. The spec is an interface: it forks implementations (GAP's + L1-stop, ignore-dangling PR spec is why `LAGr_PageRankGAP` is a + separate function), so whatever you specify is what everyone builds. + +
- [ ] You can explain why degree skew changes the work and not just the clock — this topic measures max degree 9751 on RMAT against 59 on uniform, with triangle counts of 15.6 M against 5428. +
Answer + + Same n and m, but a degree-9,751 hub concentrates edges, so + neighbour-list intersections (triangles) and any O(degree²) per-vertex + cost explode where they were trivial on uniform data (15,645,988 vs + 5,428 triangles). The counter is running a different job, not the same + job slower. + +
- [ ] You can explain why diameter sets the round count and why road networks are therefore in the suite. -- [ ] You can say why 64 trials from random sources are required, and what source luck does to a single measurement. +
Answer + + A frontier algorithm advances one distance-level per round, so the + diameter *is* the round count and per-round frontier size is the only + parallelism. Twitter/kron (diameter ~10-20) give million-vertex + frontiers; road (diameter ~1000s) gives thousands of tiny sequential + rounds — flipping the SSSP ranking. + +
+- [ ] You can say why many trials from random sources are required, and what source luck does to a single measurement. +
Answer + + On a skewed graph, a hub source reaches everything in ~2 hops while a + leaf source adds rounds — same work, very different clock. GAP runs 64 + trials/64 sources for BFS/SSSP and 16 trials of 4 sources for BC + (Table 1) and reports all of them, so one lucky source can't silently + inflate the headline number. + +
- [ ] You can state the baseline problem: reference code that is itself state of the art. +
Answer + + The classic sin is beating a strawman. GAP ships gapbs — reference + kernels that are themselves state-of-the-art (direction-optimizing + BFS, delta-stepping, Brandes with a successor bitmap, Afforest) — so a + claimed win over gapbs actually means something. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md answers Q1 (which kernels flip on road vs twitter), + Q2 (when redundant-relaxation loses), Q3 (BC source-sampling error and + stratification), Q4 (pr.cc vs pr_spmv.cc on kron), and Q5 (why + community detection is benchmark-hostile). + +
## References diff --git a/topics/24-graph-algorithms/reading-lagraph-algos.md b/topics/24-graph-algorithms/reading-lagraph-algos.md index e4e2c28..545f45d 100644 --- a/topics/24-graph-algorithms/reading-lagraph-algos.md +++ b/topics/24-graph-algorithms/reading-lagraph-algos.md @@ -10,18 +10,29 @@ pattern that exists, not inventing one. Before the code, this chapter rebuilds the four verbs, then walks the shelf one algorithm at a time. +*Sources pinned (resources/codebases.md): LAGraph @`e2539e2`, +GraphBLAS @`1fd5475`, FalkorDB @`ccb449a`; anchors re-checked with +`tools/pinned-source.py`. The triangle timings are this topic's +`notes.md` baseline (M3 Pro).* + ## The problem in one sentence Can every whole-graph analytic be written with four bulk operations and no per-vertex control flow — and what does that cost against hand-tuned frontier code (concretely: our scalar triangle count does -**15.6M triangles in 376 ms**, and LAGraph's masked-multiply -formulation of the same count must pay for its generality somewhere)? +**15.6M triangles in 376 ms** and **5,428 in 158 ms**, notes.md, and +LAGraph's masked-multiply formulation of the same count must pay for +its generality somewhere)? ## The concepts, step by step ### Step 1 — the four verbs: mxv, mxm, semiring, mask +> **In:** a graph as its adjacency matrix A. +> **Out:** the four verbs — mxv, mxm, semiring, mask — that express +> every algorithm on the shelf, so parallelism and direction belong to +> the runtime, not the algorithm. + GraphBLAS expresses graph algorithms as sparse linear algebra: the graph is its adjacency matrix A (A[u][v] nonzero iff edge u→v), and exactly four verbs do all the work. **mxv** (matrix-vector multiply) @@ -39,6 +50,11 @@ runtime, and the algorithm is a handful of verb calls. ### Step 2 — connected components as algebra: FastSV +> **In:** connected components, classically a per-edge union-find. +> **Out:** FastSV's bulk reformulation — a parent array improved each +> round by hooking (one mxv, MIN semiring) and shortcutting (jump to +> grandparent), O(log n) rounds. + Connected components (CC — label every vertex with its reachable island) is classically solved with union-find, a per-edge pointer-chasing structure; FastSV instead keeps a **parent** array @@ -51,6 +67,9 @@ halving all chains at once). One round, de-algebra'd — three bulk ops where union-find does per-edge pointer chases: ```rust +// ILLUSTRATION — not quoted; the reader's mental model. The real +// algebra is LG_CC_FastSV7.c:102 (hooking mxv) and :145-161 +// (shortcutting); our stub is experiments/src/cc.rs:57 (afforest). fn fastsv_round(a: &SparseMat, parent: &mut [u32], gp: &mut [u32]) -> bool { // hooking: every vertex reads all neighbors' grandparents AT ONCE let mngp = a.mxv_min_2nd(gp); // mngp[v] = min gp[u] over u∈N(v) @@ -72,6 +91,12 @@ weight" — question 1 asks what breaks without it. ### Step 3 — sampling in two worlds: FastSV vs Afforest +> **In:** the observation that most edges are inside the giant +> component and teach nothing. +> **Out:** how FastSV expresses "skip edges" as in-matrix column +> sampling (bulk-synchronous) while Afforest samples neighbor offsets +> with union-find (asynchronous) — same idea, two vocabularies. + Both modern CC codes exploit the same observation — most edges are inside the giant component and inspecting them teaches you nothing — but each expresses "skip edges" in its own world's vocabulary. FastSV @@ -87,6 +112,10 @@ makes you count both. ### Step 4 — triangle counting as masked multiplication +> **In:** a triangle = a wedge whose endpoints are also adjacent. +> **Out:** the masked-SpGEMM count `(L*U').*L` where the mask *prunes* +> the multiply, and the six LAGraph spellings of that one count. + A triangle is a wedge (path u–v–w) whose endpoints are also directly connected — so counting triangles is "multiply the adjacency by itself (enumerate wedges), then keep only entries where A also has an @@ -98,32 +127,40 @@ one count (L and U are the lower/upper triangles of A, which deduplicate the 6 orderings of each triangle): ``` - :33-37 0 default (currently Sandia_LUT) + :31-37 0 default (currently Sandia_LUT) + 1 Burkhardt: ntri = sum((A^2) .* A) / 6 2 Cohen: ntri = sum((L*U) .* A) / 2 3 Sandia_LL: ntri = sum((L*L) .* L) 4 Sandia_UU: ntri = sum((U*U) .* U) 5 Sandia_LUT: ntri = sum((L*U') .* L) ← dot product form 6 Sandia_ULT: ntri = sum((U*L') .* U) - :44-47 LUT fastest on large graphs EXCEPT GAP-urand, where + :43-47 LUT fastest on large graphs EXCEPT GAP-urand, where saxpy-based LL wins — the dot-vs-saxpy split (topic 20) decided by TRIANGLE DENSITY, not just matrix shape ``` Our scalar `triangle_count` is Sandia's formulation with rank-ordered adjacency instead of tril: "orient by degree, intersect forward -lists" IS `(L*L).*L` read row-wise. One measured point: rmat 15.6M -triangles in 376 ms vs uniform 5.4K in 158 ms — method choice (:44) -flips exactly because urand has ~no triangles to prune with: a mask -with nothing in it saves nothing. +lists" IS `(L*L).*L` read row-wise. One measured point (notes.md): +rmat 15.6M triangles in 376 ms vs uniform 5.4K in 158 ms — method +choice (:43) flips exactly because urand has ~no triangles to prune +with: a mask with nothing in it saves nothing. ### Step 5 — the rest of the shelf, rapid-fire +> **In:** the four verbs and the two CC/TC worked examples. +> **Out:** the rest of the shelf as the same move (pick a semiring, +> pick a mask, iterate), plus the dangling-PR benchmarking lesson. + The remaining algorithms are the same move — pick a semiring, pick a mask, iterate — plus one benchmarking lesson: -- `LAGr_PageRankGAP.c` vs `LAGr_PageRank.c`: GAP-spec PR (dangling - handled gapbs-style, L1 stop) vs textbook. Benchmark specs fork - implementations — topic 22's lesson in filenames. +- `LAGr_PageRankGAP.c` vs `LAGr_PageRank.c`: `LAGr_PageRankGAP` + **ignores** dangling (zero-out-degree) vertices to match the GAP + spec and gapbs (LAGr_PageRankGAP.c:20-29), L1-error stop; + `LAGr_PageRank` instead **redistributes** sink rank each iteration to + keep scores summing to 1 (LAGr_PageRank.c:20-26). Benchmark specs + fork implementations — topic 22's lesson in filenames. - `LAGr_SingleSourceShortestPath.c:151-185`: MIN_PLUS delta-stepping (see reading-delta-stepping.md) — the bucket is a masked sparse vector, one vxm per inner iteration. @@ -135,6 +172,11 @@ mask, iterate — plus one benchmarking lesson: ### Step 6 — the FalkorDB tie-in: the flush boundary is the cost +> **In:** the shelf behind a Cypher procedure surface. +> **Out:** M24's real design question — the delta-matrix flush/export +> boundary and result materialization, not the (already solved) +> algorithms. + FalkorDB's procedure layer is exactly this shelf behind a Cypher surface, and its shape names M24's real design question. `proc_pagerank.c`: parse args → get the delta-matrix-backed A → @@ -155,13 +197,13 @@ read it first. | anchor | what | |---|---| | `:69-71` | the state: `mngp` (min neighbor grandparent), `gp`, `gp_new` — SV's hooking/shortcutting as three vectors | -| `:102` | hooking = ONE mxv: `mngp = min_2nd(A, gp)` — every vertex reads its neighbors' grandparents in one masked matrix op | -| `:145-158` | shortcutting: `parent = min(parent, mngp)` via mxv on a PARENT MATRIX + `gp_new = parent(parent)` (extract = pointer chase as assign) | +| `:102` | hooking = ONE mxv: `mngp = min(mngp, A·gp)` with the MIN_SECOND semiring (no mask) — every vertex reads its neighbors' grandparents in one matrix op | +| `:145-161` | shortcutting: `parent2 = Parent·mngp` (mxv, min_2nd, :145) then `parent = min(parent, parent2)` (GrB_assign, :155) + `gp_new = parent(parent)` (GrB_extract at :161 = pointer chase as assign) | | `:335-338` | sampling: `FASTSV_SAMPLES` per row, `sampling = nvals > n*samples*2 && n > 1024` — Afforest's idea imported (Step 3) | | `:231-235` | built-in timing printfs: sample phase vs hash phase vs final mxv — SuiteSparse's authors profile like topic 0 | -- **Step 4 — `LAGr_TriangleCount.c`**: the six methods at `:33-37`, - the LUT-vs-LL crossover note at `:44-47`. +- **Step 4 — `LAGr_TriangleCount.c`**: the seven methods (0-6) at + `:31-37`, the LUT-vs-LL crossover note at `:43-47`. - **Step 5 — the shelf**: `LAGr_PageRankGAP.c`, `LAGr_PageRank.c`, `LAGr_SingleSourceShortestPath.c:151-185`, `LAGr_Betweenness.c:110-164`, `LG_CC_Boruvka.c`. @@ -180,8 +222,9 @@ read it first. 3. Sandia_LUT (dot) vs Sandia_LL (saxpy) — connect :44-47's urand exception to topic 20's dot3-vs-saxpy3 rule. What property of urand (no hubs, no triangles) starves the dot-form's mask? -4. LAGr_PageRankGAP handles dangling vertices with an extra - reduction per iteration. Our pull PR ignores them — quantify the +4. `LAGr_PageRank` handles sinks with an extra reduction per iteration + (redistributing dangling rank to keep scores summing to 1); + `LAGr_PageRankGAP` and our pull PR both ignore them — quantify the error on a graph with 18K single-node components. 5. M24 API: `CALL algo.wcc()` on a graph with pending deltas — enumerate the three options (flush first / run on main / run on @@ -190,12 +233,66 @@ read it first. ## Done when +Answer each before unfolding it. + - [ ] You can name the four verbs and express connected components as algebra. +
Answer + + The verbs are mxv, mxm, semiring, and mask. CC as algebra is FastSV: + a parent array improved each round by hooking (mngp = min(mngp, A·gp) + via mxv with MIN_SECOND) and shortcutting (gp_new = parent(parent)), + converging in O(log n) bulk rounds instead of per-edge union-find. + +
- [ ] You can explain FastSV's `min_2nd` semiring and why it takes the neighbour's grandparent. +
Answer + + MIN_SECOND multiplies by taking the *second* operand (the neighbour's + grandparent value gp[u]) and ignoring the matrix entry (the edge + value), then reduces with MIN. Hooking wants the smallest grandparent + label among neighbours; a weighted MIN_TIMES would fold edge weights + into the label and corrupt it (Q1). + +
- [ ] You can contrast sampling in FastSV against Afforest and count matrix ops against pointer chases. +
Answer + + FastSV samples columns *inside* the matrix ops (FASTSV_SAMPLES per + row, bulk-synchronous), staying algebraic; Afforest samples per-vertex + neighbour offsets with an asynchronous union-find and a final sweep + that skips the identified giant component. Same "skip inside-GC edges" + idea; the test is rounds×bulk-pass (FastSV) vs edges-inspected + (Afforest's <50% of m). + +
- [ ] You can write triangle counting as a masked multiplication. +
Answer + + ntri = sum((L*U').*L): multiply the triangles of A to enumerate + wedges, and use A itself as a mask so only wedges that close are + computed. The mask prunes the SpGEMM (never materializing L*U'), which + is why triangle *density* — not matrix shape — picks dot vs saxpy. + +
- [ ] You can explain why the flush boundary is the cost in the FalkorDB tie-in. +
Answer + + The algorithms are solved; FalkorDB's cost is around them — + exporting/flushing the delta-matrix-backed A into a GrB_Matrix before + the call, and materializing full result vectors after. M24's design + question is whether algorithms can run masked over the DM/DP directly + and stream top-k instead. + +
- [ ] You wrote answers to all five questions in notes.md, including what `CALL algo.wcc()` must do about pending deltas. +
Answer + + Done when notes.md answers Q1 (min_2nd), Q2 (ops vs chases and why + Afforest wins wall-clock), Q3 (urand starves the dot mask), Q4 + (sink-rank error), and Q5 (the three delta-handling options and their + consistency semantics). + +
## References diff --git a/topics/24-graph-algorithms/reading-ligra.md b/topics/24-graph-algorithms/reading-ligra.md index 909b9ce..b5e7afc 100644 --- a/topics/24-graph-algorithms/reading-ligra.md +++ b/topics/24-graph-algorithms/reading-ligra.md @@ -9,19 +9,27 @@ frontier is, its two physical representations, the push/pull choice, the one threshold that automates it, and what a whole algorithm looks like when it's reduced to a single edge function. +*Source pinned (resources/codebases.md): ligra @`8763202`; anchors +re-checked with `tools/pinned-source.py`.* + ## The problem in one sentence Every frontier algorithm faces the same per-round choice — push from the frontier's out-edges or pull over all vertices' in-edges — and -getting it wrong costs up to 10× per round (topic 20's BFS numbers); -Ligra moves that choice out of every algorithm and into one framework -function with one threshold: **|frontier| + its out-degree sum vs -m/20**. +getting it wrong can make a single round traverse the graph's *entire* +edge set when a small fraction of it would have done; Ligra moves that +choice out of every algorithm and into one framework function with one +threshold: **|frontier| + its out-degree sum vs m/20**. ## The concepts, step by step ### Step 1 — the frontier: the set of vertices that matter this round +> **In:** a graph algorithm that proceeds in rounds. +> **Out:** the *frontier* — the active vertex set this round — and the +> single quantity (its size relative to n) that everything in Ligra +> keys off. + A **frontier** is the set of active vertices in the current round of a graph algorithm — in BFS, the vertices discovered last round whose edges must be explored next. Frontier algorithms proceed in rounds: @@ -34,6 +42,11 @@ graph, is the single quantity everything in Ligra keys off. ### Step 2 — two physical representations: id array vs bitmap +> **In:** a frontier that can be tiny or huge. +> **Out:** its two physical forms — a sparse id array (small +> frontiers) and a dense boolean array of size n (big frontiers) — and +> the cost trade Ligra switches between automatically. + A set of vertices can be stored two ways, and the right one depends on its size. Ligra's `vertexSubset` is physically EITHER: @@ -52,6 +65,11 @@ converts between them automatically as the wave grows and shrinks. ### Step 3 — push vs pull: whose edges do you traverse? +> **In:** one round of updates to run. +> **Out:** the two directions — push (iterate the frontier's +> out-edges, good when small) and pull (iterate every vertex's +> in-edges with early exit, good when large) — and which wins when. + There are two ways to run one round of updates, with different cost shapes. **Push** iterates the frontier and follows each member's out-edges — work proportional to the frontier's out-degree sum, ideal @@ -63,10 +81,16 @@ trick: once one in-neighbor claims the vertex, stop scanning an early in-edge, so pull touches far fewer than m edges — while push would faithfully traverse the frontier's entire (huge) out-degree sum and fight write contention doing it. Small frontier: push wins. Big -frontier: pull wins. Same asymptotics, ~10× apart in constants. +frontier: pull wins. Same asymptotics; which direction wins can swing +the per-round cost by a large constant factor. ### Step 4 — the switch: edgeMap and the m/20 threshold +> **In:** the push/pull choice from Step 3. +> **Out:** `edgeMap`'s one comparison — |frontier| + its out-degree +> sum vs (graph edges)/20 — that automates the choice for every +> algorithm. + Ligra's `edgeMap` packages Step 3's choice behind one comparison, so every algorithm inherits direction switching without asking: @@ -81,9 +105,25 @@ every algorithm inherits direction switching without asking: whether v joins the next frontier ``` -The switch, as code — everything else in Ligra is plumbing around it: +That comparison is real code — note that in `ligra.h` the local `m` is +the *frontier* size (`vs.numNonzeros()`), while the threshold divides +the *graph's* edge count (`numEdges = GA.m`); the guide's "m/20" means +graph edges / 20: + +```c +// ligra/ligra.h — edgeMapData, jshun/ligra@8763202 + 237 long numVertices = GA.n, numEdges = GA.m, m = vs.numNonzeros(); + 238 if(threshold == -1) threshold = numEdges/20; //default threshold + ... + 261 if (!(fl & no_dense) && m + outDegrees > threshold) { +``` + +The switch, as pseudocode — everything else in Ligra is plumbing +around it: ```rust +// ILLUSTRATION — not quoted; the real switch is ligra/ligra.h:261 +// (threshold :238), dense/pull at ligra.h:59, sparse/push at :111. fn edge_map(g: &Graph, front: &VertexSubset, f: &impl Fn(u32, u32) -> bool) -> VertexSubset { if front.len() + front.out_degree_sum(g) > g.m / 20 { @@ -112,6 +152,12 @@ reads in-edges, so the graph AND its transpose must both be resident. ### Step 5 — an algorithm is just F: reading the apps +> **In:** frontier, representation, and switch all owned by the +> framework. +> **Out:** what's left of an algorithm — a single per-edge function +> F(u,v) — read across five apps, ending at PR where the frontier is +> always everything and Ligra ≡ SpMV. + With frontier, representation, and switch all owned by the framework, an algorithm shrinks to its per-edge update function F(u, v) — which does the algorithm-specific write and returns whether v joins the @@ -135,6 +181,11 @@ Ligra generalizes the case where they do. ### Step 6 — Ligra vs GraphBLAS, honestly +> **In:** Ligra's edgeMap model and GraphBLAS's semiring model. +> **Out:** the honest trade — edgeMap's arbitrary-F expressiveness vs +> semiring fusability — and the three names (m/20, α/β, dot-vs-saxpy) +> for the one direction decision. + The two frameworks in this topic's dichotomy trade expressiveness for fusability, and neither dominates: @@ -185,12 +236,66 @@ fusability, and neither dominates: ## Done when +Answer each before unfolding it. + - [ ] You can define the frontier and both physical representations. +
Answer + + The frontier is the active vertex set this round (what changed last + round). Ligra's vertexSubset stores it either sparse (an array of + vertex ids, cheap for small frontiers) or dense (a size-n boolean + array, cheap membership tests for big frontiers), converting between + them as the wave grows and shrinks. + +
- [ ] You can explain push against pull as whose edges you traverse. +
Answer + + Push iterates the frontier and follows out-edges — cost ∝ frontier + out-degree sum, best when small. Pull iterates every vertex and scans + in-edges asking "is a neighbour in the frontier?", stopping at the + first claim — cost bounded by m but far less when the frontier is + dense. Small: push. Big: pull. + +
- [ ] You can construct a frontier where the m/20 threshold is the wrong call. +
Answer + + Pick a frontier whose out-degree sum is just under (graph edges)/20 — + so edgeMap chooses push — but whose *next* frontier is nearly empty, + so pull's early exit would almost never fire and pull would scan close + to m anyway. The threshold can't see next-frontier fullness, so it + mis-picks (Q1). + +
- [ ] You can explain why `edgeMapDenseForward` pushes from all vertices and when that is cheaper. +
Answer + + `edgeMapDenseForward` (ligra.h:85) scans out-edges of all vertices + without early exit — used when the update isn't "claim once" so + early-exit can't apply (e.g. PageRank-style accumulation). It beats + pull-with-break when every in-edge must be visited anyway, so the + break never saves work (Q2). + +
- [ ] You can compare Ligra's model honestly against GraphBLAS's and say what each makes awkward. +
Answer + + Ligra's F is an arbitrary CAS-using function (expressive, but not + fusable); GraphBLAS semirings are (monoid, binop) pairs (fusable, but + can't express strided sampling like Afforest). Both need G and Gᵀ + resident for the pull/dense direction. Neither dominates. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md answers Q1 (the wrong-threshold frontier), Q2 + (denseForward vs dense), Q3 (mapping BC.C's transpose passes onto + LAGr_Betweenness), Q4 (label-prop vs Afforest edges touched), and Q5 + (edgeMap-callback vs fixed-menu API for a safe embedding). + +
## References diff --git a/topics/24-graph-algorithms/reading-louvain-leiden.md b/topics/24-graph-algorithms/reading-louvain-leiden.md index 0fccd83..abbd3fe 100644 --- a/topics/24-graph-algorithms/reading-louvain-leiden.md +++ b/topics/24-graph-algorithms/reading-louvain-leiden.md @@ -10,17 +10,30 @@ how Louvain climbs it, exactly where the greedy climb breaks connectivity, and how Leiden's refinement phase repairs the guarantee for free. +*Paper: Traag, Waltman & van Eck, "From Louvain to Leiden" +(Scientific Reports 2019, arXiv:1810.08473). The disconnection +frequencies, guarantees, and refinement rule below are quoted from it; +this topic has no pinned Louvain/Leiden code, so the snippets are +illustrative pseudocode.* + ## The problem in one sentence Louvain greedily optimizes a global score with moves that never check -connectivity, and on real graphs **up to 25% of the communities it -outputs are internally disconnected** — two islands wearing one -label — and iterating the algorithm makes it worse, not better. +connectivity, and on real graphs **up to 16% of the communities it +outputs are internally disconnected** (and up to 25% are *badly* +connected) — two islands wearing one label — and iterating the +algorithm makes it worse, not better (Traag et al. 2019, abstract). ## The concepts, step by step ### Step 1 — communities, and a score for them: modularity +> **In:** the goal "find groups with many internal, few crossing +> edges". +> **Out:** modularity Q — internal edges minus a degree-preserving +> random baseline — worked on a concrete graph, and the fact that Q +> never mentions connectivity. + A **community** is a set of vertices with many edges inside the set and few crossing its boundary — and to optimize for that, you need a number. **Modularity** (Q) compares each community's internal edge @@ -28,23 +41,52 @@ count against what a random graph with the same vertex degrees would put there: ``` - Q = (1/2m) Σ_ij [ A_ij − k_i·k_j/2m ] · δ(c_i, c_j) + Q = (1/2m) Σ_ij [ A_ij − γ·k_i·k_j/2m ] · δ(c_i, c_j) "edges inside communities, minus what a degree-preserving - random graph would put there" (γ = resolution knob) + random graph would put there" + + The paper writes it per community (Eq. 1): + H = (1/2m) Σ_c ( e_c − γ·K_c²/2m ) + with e_c the internal edge weight of community c (both directions) + and K_c the sum of its vertices' degrees; γ is the resolution knob. ``` Here A_ij is the (weighted) adjacency entry, k_i is vertex i's degree, m the total edge weight, and δ(c_i, c_j) is 1 when i and j -share a community. The subtraction is the insight: two hubs sharing -an edge is unremarkable (random graphs do that too — k_i·k_j/2m is -high), two leaves sharing an edge is signal. γ (the resolution -parameter) scales the null-model term to tune community size. Why it -matters: modularity turns "find communities" into "maximize Q" — an -optimization problem a greedy algorithm can attack. Note what Q does -*not* mention: connectivity. That omission is the whole paper. +share a community. Work it on two triangles {a,b,c} and {d,e,f} joined +by one bridge edge c–d (m = 7 edges, 2m = 14; degrees a,b,e,f = 2 and +c,d = 3): + +``` + partition {{a,b,c},{d,e,f}}, γ = 1: + community {a,b,c}: internal edge weight e_c = 2×3 = 6 (3 edges, + both directions); K_c = 2+2+3 = 7 + term = 6 − 7²/14 = 6 − 3.5 = 2.5 + community {d,e,f}: identical by symmetry = 2.5 + Q = (1/14)(2.5 + 2.5) = 0.357 + + everything in ONE community: + e = 2×7 = 14 (all edges internal); K = 14 + term = 14 − 14²/14 = 0 → Q = 0 +``` + +Splitting into the two triangles scores Q = 0.357 against 0 for one +blob — modularity rewards the real structure. The subtraction is the +insight: two hubs sharing an edge is unremarkable (random graphs do +that too — k_i·k_j/2m is high), two leaves sharing an edge is signal. +γ (the resolution parameter) scales the null-model term to tune +community size. Why it matters: modularity turns "find communities" +into "maximize Q" — an optimization a greedy algorithm can attack. +Note what Q does *not* mention: connectivity. That omission is the +whole paper. ### Step 2 — Louvain: greedy local moves plus aggregation +> **In:** the objective "maximize Q". +> **Out:** Louvain's two alternating phases (greedy single-vertex +> moves, then aggregate-and-recurse) and the O(deg) ΔQ that makes a +> move cheap — plus the fact that aggregation is irreversible. + Louvain climbs Q with two alternating phases — move single vertices greedily, then shrink the graph and repeat: @@ -60,7 +102,8 @@ The local-move kernel — the part both algorithms share and Leiden speeds up with a queue — is cheap because moving one vertex changes Q by an amount (ΔQ) computable from just that vertex's edges: -```rust +``` +// ILLUSTRATION — pseudocode; this topic pins no Louvain/Leiden source. fn local_move(v: u32, g: &Csr, comm: &mut [u32], tot: &mut [f64]) -> bool { let mut w_to = HashMap::new(); // topic 20's SPA, again for (u, w) in g.edges(v) { *w_to.entry(comm[u]).or_insert(0.0) += w; } @@ -84,6 +127,11 @@ is frozen into the super-vertices. ### Step 3 — the bug: a bridge vertex walks away +> **In:** Louvain's local move plus irreversible aggregation. +> **Out:** the concrete failure — a bridge vertex leaves, its old +> community splits into two islands that keep one label, and +> aggregation freezes them — traced on Fig. 1's graph. + A vertex v can be the BRIDGE holding community C together — remove v and C falls into two pieces. Louvain's local move relocates v anyway: ΔQ is evaluated against v's current neighbors' communities (look at @@ -104,12 +152,18 @@ then FREEZES them into a single super-vertex forever: ``` This is the paper's §2 and Fig. 1 — internalize this figure. On real -graphs, up to 25% of Louvain communities end up disconnected -(§Results); iterating Louvain makes it WORSE, not better, because -each iteration adds more frozen mistakes. +graphs, up to 16% of Louvain communities end up *disconnected* and up +to 25% are *badly connected* (Traag et al. 2019, abstract & Fig. 2); +iterating Louvain makes it WORSE, not better, because each iteration +adds more frozen mistakes. ### Step 4 — the root cause generalizes: greedy + irreversible = unfixable +> **In:** the bridge-vertex failure from Step 3. +> **Out:** why it's structural, not a slip — greedy local scoring plus +> irreversible aggregation accumulate errors monotonically — and the +> two possible fixes (per-move connectivity check vs. an undo path). + The failure is not a coding slip; it is a structural property of the algorithm class: greedy local search makes locally-scored decisions, and irreversible aggregation removes the ability to undo them — so @@ -122,6 +176,11 @@ undo path before aggregation freezes things. Leiden picks the second. ### Step 5 — Leiden's fix: refine before you freeze +> **In:** the "keep an undo path" fix direction. +> **Out:** Leiden's refinement phase — re-cluster within each +> community from singletons, randomized ∝ ΔH, before aggregating — so +> aggregation only ever fuses connected pieces. + Leiden inserts a refinement phase between moving and aggregating — re-cluster each community from scratch, *within* the community, so aggregation only ever fuses pieces that are actually connected: @@ -137,18 +196,25 @@ aggregation only ever fuses pieces that are actually connected: ``` Refinement is the undo mechanism: aggregation now operates on pieces -that are guaranteed γ-connected (Theorem: Leiden communities are -connected; iterated Leiden converges to subset-optimal partitions). -The randomization in step 2 (merge proportional to exp(ΔQ/θ), not -greedy-max) is load-bearing — it lets refinement explore partitions -the greedy climb would never visit; §Methods explains what breaks if -you make it deterministic (question 3). And empirically Leiden is -also FASTER than Louvain — the queue in phase 1 (only revisit -vertices whose neighborhood changed) more than pays for refinement. -The fix costs nothing. +that are guaranteed γ-connected (Traag et al. 2019 prove Leiden +communities are γ-connected and, iterated, converge to subset-optimal +partitions — their Table I of guarantees). The randomization in step 2 +(merge with probability ∝ exp((1/θ)·ΔH) when ΔH ≥ 0, else 0 — the +paper's Eq. 38, with H the modularity from Step 1), not greedy-max, is +load-bearing — it lets refinement explore partitions the greedy climb +would never visit; §Methods explains what breaks if you make it +deterministic (question 3). And empirically Leiden is also FASTER than +Louvain (Traag et al. 2019, abstract & Fig. 3) — the queue in phase 1 +(only revisit vertices whose neighborhood changed) more than pays for +refinement. The fix costs nothing. ### Step 6 — what this costs an engine: SPA, SpGEMM, and seeds +> **In:** the Louvain/Leiden algorithm as described. +> **Out:** where each phase lands on the M20 sparse core — ΔQ on a SPA +> accumulator, aggregation as S·A·Sᵀ SpGEMM, and why randomized +> refinement makes seeding mandatory for a reproducible engine. + Mapping the algorithm onto the M20 sparse core, each phase lands on machinery that already exists: @@ -168,12 +234,13 @@ machinery that already exists: - §2 and Fig. 1 are Step 3 — read them first and reconstruct the bridge-vertex failure on paper (question 1) before continuing. -- The results on disconnected-community frequency (the 25% number, - and the it-gets-worse-with-iteration result) justify Step 4's +- The results on disconnected-community frequency (the 16% + disconnected / 25% badly-connected numbers, and the + it-gets-worse-with-iteration result) justify Step 4's framing: this is accumulation, not bad luck. - §Methods carries Step 5: the queue-based fast local move, the - randomized refinement (find the exp(ΔQ/θ) rule and the argument - for why greedy refinement fails), and the connectivity / + randomized refinement (find the exp((1/θ)·ΔH) rule, Eq. 38, and the + argument for why greedy refinement fails), and the connectivity / subset-optimality theorems. - Read the guarantees the way topic 16 reads invariants: "communities are γ-connected" is a property you can test — question 5 turns it @@ -185,11 +252,13 @@ machinery that already exists: 5-vertex example: which move disconnects the community and why was its ΔQ positive? 2. The resolution limit: modularity at γ=1 can't see communities - smaller than ~√(2m). Where does that bite a fraud-ring query on - a payments graph, and which knob (γ, or CPM as the paper hints) - fixes it? -3. Leiden's refinement merges randomly ∝ exp(ΔQ/θ). What breaks if - you make it greedy-deterministic (the paper tells you — §Methods)? + smaller than ~√(2m) (Fortunato & Barthélemy, "Resolution limit in + community detection", PNAS 2007). Where does that bite a fraud-ring + query on a payments graph, and which knob (γ, or CPM as the paper + hints) fixes it? +3. Leiden's refinement merges randomly ∝ exp((1/θ)·ΔH) (Eq. 38). What + breaks if you make it greedy-deterministic (the paper tells you — + §Methods)? 4. Map one Leiden iteration onto the M20 sparse core: which steps are SpGEMM, which are the SPA-style local kernel, and where do delta matrices interact with aggregation? @@ -200,12 +269,64 @@ machinery that already exists: ## Done when +Answer each before unfolding it. + - [ ] You can define modularity and say what it scores. +
Answer + + Q = (1/2m) Σ_ij [A_ij − γ·k_i·k_j/2m]·δ(c_i,c_j): the fraction of + edges inside communities minus what a degree-preserving random graph + would put there. It scores "are there more internal edges than chance + predicts?" — and mentions connectivity nowhere. + +
- [ ] You can reproduce Figure 1's failure: how a bridge vertex ends up disconnected from its own community. +
Answer + + A vertex v holds community C connected. Louvain evaluates v's ΔQ + against its neighbours' communities and moves it out (positive gain + there), never checking that removing v splits C. C's two halves keep + C's label as disconnected islands, then aggregation fuses them into + one super-vertex permanently (paper §2, Fig. 1). + +
- [ ] You can state the general root cause — greedy plus irreversible is unfixable — and what Leiden's refinement changes. +
Answer + + Greedy local moves score decisions locally; irreversible aggregation + removes the undo path, so errors accumulate monotonically. Leiden adds + a refinement phase that re-clusters within each community from + singletons before aggregating, so aggregation only fuses genuinely + connected pieces — restoring the undo path. + +
- [ ] You can explain the resolution limit and what γ does about it. +
Answer + + At γ=1 modularity can't resolve communities smaller than ~√(2m) + (Fortunato & Barthélemy 2007): small real groups get merged. Raising + γ scales up the null-model penalty so smaller communities survive; + the paper's CPM objective removes the size-dependence entirely. + +
- [ ] You can map one Leiden iteration onto SPA and SpGEMM steps. +
Answer + + ΔQ evaluation is a SPA-style accumulator keyed by community id (topic + 20's SPA). Aggregation is the quotient graph S·A·Sᵀ — two masked + SpGEMMs. So one iteration ≈ two SpGEMMs plus the local-move kernel, + with a seed fixed for reproducibility. + +
- [ ] You wrote answers to all five questions in notes.md, including the topic-16 property test for community connectivity. +
Answer + + Done when notes.md answers Q1–Q5, and Q5 is a concrete property test: + for each output community, run one BFS (or FastSV on the induced + subgraph) and assert it reaches every member — the test Louvain fails + and Leiden passes. + +
## References diff --git a/topics/25-graph-ml/README.md b/topics/25-graph-ml/README.md index 4f587be..65d288d 100644 --- a/topics/25-graph-ml/README.md +++ b/topics/25-graph-ml/README.md @@ -67,11 +67,13 @@ with a cost model could *choose*. |---|---| | SBM build (64 blocks x 256) | 34.4 ms | | uniform walks 65,536 x 40 steps | 61.2 ms, 42.8 Msteps/s | -| SpMM (D^-1 A) x X[16384x64] | 3.42 ms/iter, **21.2 GFLOP/s** | -| dense matmul [16384x64]x[64x64] | 5.12 ms/iter, 26.2 GFLOP/s | +| SpMM (D^-1 A) x X[16384x64] | 4.31 ms/iter, **16.82 GFLOP/s** | +| dense matmul [16384x64]x[64x64] | 5.65 ms/iter, 23.75 GFLOP/s | -The headline: naive scalar SpMM reaches **81% of dense matmul's -throughput** on this graph. Sparse's irregular gather is amortized by the +The headline: naive scalar SpMM reaches **~71% of dense matmul's +throughput** on this graph (FINDINGS row 25; 2·566,564·64 flops in +4.31 ms against 2·16384·64·64 in 5.65 ms). Sparse's irregular gather is +amortized by the 64-float dense rows it drags along — a GNN's SpMM is memory-friendly in exactly the way topic 20's SpMV (1-wide) is not. Fat right-hand sides forgive sparsity. diff --git a/topics/25-graph-ml/experiments/.gitignore b/topics/25-graph-ml/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/25-graph-ml/experiments/.gitignore +++ b/topics/25-graph-ml/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/25-graph-ml/experiments/src/embed.rs b/topics/25-graph-ml/experiments/src/embed.rs index 3c7db35..5c67be9 100644 --- a/topics/25-graph-ml/experiments/src/embed.rs +++ b/topics/25-graph-ml/experiments/src/embed.rs @@ -34,7 +34,9 @@ pub fn sigmoid(x: f32) -> f32 { /// s = sigmoid(z_u . c_c'); g = lr * (0 - s) /// symmetric update with c_c'. /// (That's the gradient of log sigma(z.c) + sum_neg log sigma(-z.c') — -/// PyG's Node2Vec.loss at node2vec.py:135 is this exact expression.) +/// the classic SGNS two-table form. PyG's Node2Vec.loss +/// (node2vec.py:140,:142) optimizes the same objective but shares ONE +/// embedding table for both roles.) /// /// Init both matrices U(-0.5/dim, 0.5/dim) from ChaCha8Rng::seed_from_u64(seed). /// `epochs` full passes over the walks; linear LR decay per epoch is fine diff --git a/topics/25-graph-ml/notes.md b/topics/25-graph-ml/notes.md index e445939..bc64c15 100644 --- a/topics/25-graph-ml/notes.md +++ b/topics/25-graph-ml/notes.md @@ -2,6 +2,14 @@ ## Baseline (provided code, Apple M3 Pro, measured 2026-07-10) +> An earlier run of the same lanes than +> [FINDINGS.md](../../FINDINGS.md) row 25 (**SpMM 4.31 ms at +> 16.82 GFLOP/s** against a **5.65 ms** dense transform, i.e. ~71% of +> dense throughput, not the 81% below). Where the two disagree, FINDINGS +> is canonical; cite one run or the other by name and never average +> them. Re-run `./verify.sh 25` before treating any cell below as +> current. + SBM: 64 blocks x 256 = 16,384 vertices, m=566,564 directed (avg_deg 34.6), p_in=0.12, p_out=0.00025, build 34.4 ms. diff --git a/topics/25-graph-ml/reading-gat.md b/topics/25-graph-ml/reading-gat.md index f02e6a3..0959113 100644 --- a/topics/25-graph-ml/reading-gat.md +++ b/topics/25-graph-ml/reading-gat.md @@ -21,6 +21,10 @@ the model *learn* which of the 100 to listen to, at the price of ### Step 1 — the limitation: GCN's weights are structural constants +> **In:** GCN's aggregation weight on each edge (reading-gcn.md). +> **Out:** the expressiveness gap — a degree constant cannot prefer one +> neighbour over another. Step 2 replaces the constant with a learned score. + In GCN (reading-gcn.md), the weight on edge (u, v) during aggregation is 1/√(d_u·d_v) — computed from degrees alone, identical for every layer, every epoch, every input. That makes A_hat precomputable @@ -32,14 +36,19 @@ routine ones — is beyond what a structural constant can express. ### Step 2 — attention: score each edge from its endpoints' features +> **In:** the endpoint features `h_u`, `h_v` and the shared transform `W`. +> **Out:** a raw per-edge score `e_uv`. Step 3 normalizes these into weights. + GAT's move: compute a per-edge score from the *current features* of the edge's two endpoints, using a small learned vector `a`. Transform both endpoint features with the shared weight matrix W, concatenate, -dot with `a`, and pass through LeakyReLU (a ReLU variant that leaks a -small slope for negative inputs, keeping gradients alive): +dot with `a`, and pass through **LeakyReLU** (a ReLU variant that leaks a +small slope for negative inputs, keeping gradients alive; GAT fixes the +negative slope at **0.2**, §2.1, matched by PyG's +`negative_slope=0.2` default at gat_conv.py:136): ``` - e_uv = LeakyReLU( a^T [ W h_u || W h_v ] ) per EDGE (u,v) ∈ A + e_uv = LeakyReLU_0.2( a^T [ W h_u || W h_v ] ) per EDGE (u,v) ∈ A ``` The score says "how much should v listen to u, given what both look @@ -51,10 +60,15 @@ systems story — Step 6. ### Step 3 — softmax: turning scores into weights that sum to one +> **In:** the raw per-edge scores `e_uv` from Step 2. +> **Out:** normalized attention weights `alpha_uv` summing to 1 over each +> vertex's in-neighbourhood, and the weighted aggregate `h'_v`. Step 4 maps +> all three formulas to kernels. + Raw scores have arbitrary scale, so each vertex normalizes the scores on its incoming edges with a softmax (exponentiate, divide by the sum), yielding attention weights alpha that sum to 1 over each -vertex's in-neighborhood: +vertex's in-neighborhood (§2.1, eq. for α_ij): ``` alpha_uv = softmax over v's in-edges ( e_uv ) @@ -70,6 +84,9 @@ again). ### Step 4 — the kernel view: SDDMM + segmented softmax + SpMM +> **In:** the three formulas from Steps 2–3 (score, softmax, weighted sum). +> **Out:** the three engine kernels they compile to. Step 5 prices them. + Now translate the three formulas into engine kernels. The score computation is an **SDDMM** (sampled dense-dense matrix multiply: a dense computation over pairs of rows, evaluated ONLY at positions @@ -90,6 +107,9 @@ that were computed microseconds ago: The three kernels for one destination row, spelled out: ```rust +// ILLUSTRATION — not quoted; PyG's real path is gat_conv.py:392 +// (alpha_j + alpha_i), :403 (leaky_relu), :404 (segmented softmax), +// :408-409 (message = alpha * x_j). There is no message_and_aggregate. fn gat_row(a_t: &Csr, v: u32, wh: &Mat, a_src: &[f32], a_dst: &[f32]) -> Vec { // SDDMM: dense scores, computed ONLY at A's nonzeros (in-edges of v) let e: Vec = a_t.row(v) @@ -107,32 +127,44 @@ fn gat_row(a_t: &Csr, v: u32, wh: &Mat, a_src: &[f32], a_dst: &[f32]) -> Vec **In:** the three-kernel pipeline from Step 4, and the multi-head count K. +> **Out:** the per-layer edge-pass count against GCN, and the K-fold cost of +> multi-head. Step 6 draws the materialize-vs-compute line. + Counting edge passes per layer: GCN does one (the SpMM). GAT does the SDDMM, the softmax's max pass, its exp-sum pass, and the SpMM — the sparse-softmax is a segmented reduction over CSR rows, same shape as topic 20's row-wise SpMV, run twice. Call it ~3 extra passes over the edges per layer (question 2 turns this into a forward-time estimate -against our 21 GFLOP/s SpMM lane). **Multi-head attention** (K -independent attention weightings whose outputs are concatenated — the -standard variance-reduction trick) multiplies everything by K — it's -K SpMMs with shared structure, different values. A delta-matrix -engine would store one structure + K value arrays (FalkorDB's -multi-value matrix problem, again). +against the **16.82 GFLOP/s** SpMM lane, FINDINGS.md row 25). +**Multi-head attention** (K independent attention weightings — the +standard variance-reduction trick) multiplies everything by K: it's +K SpMMs with shared structure, different values. The paper concatenates +the K heads on intermediate layers (`h'_i = ‖_{k=1}^K σ(Σ α_ij^k W^k h_j)`, +§2.1) but **averages** them on the final prediction layer, where concat +"is no longer sensible" (§2.1); on Cora it uses K=8 heads of 8 features +each. A delta-matrix engine would store one structure + K value arrays +(FalkorDB's multi-value matrix problem, again). ### Step 6 — the line this pair of papers draws: materialize vs compute +> **In:** GCN's precomputable `A_hat` and GAT's feature-dependent attention. +> **Out:** the materialized-view vs computed-view distinction that splits the +> two papers' systems profiles. + GCN's A_hat is a **materialized view**: computed once from the graph, reused by every query, invalidated only by graph changes. GAT's attention matrix is a **computed view**: its values depend on the @@ -166,7 +198,7 @@ is the explanation (question 4: what Cypher surface exposes it). transpose tax)? 2. Count edge passes per GAT layer vs GCN layer. On our 566K-edge SBM at 21 GFLOP/s SpMM, estimate the forward-time ratio. -3. The a_src/a_dst per-node split at gat_conv.py:332 turns O(m·d) score +3. The a_src/a_dst per-node split at gat_conv.py:330-332 turns O(m·d) score work into O(n·d) + O(m). Which database trick is this (hint: factor computation out of a join)? 4. GAT attention weights are data — a fraud analyst asks "WHY did this @@ -177,12 +209,74 @@ is the explanation (question 4: what Cypher surface exposes it). ## Done when +Answer each before unfolding it. + - [ ] You can say what GCN's structural constant weights cannot express. + +
Answer + + A GCN edge weight is `1/√(d_u·d_v)` — degree arithmetic, fixed before + training and identical every layer and epoch. It is content-blind: it + cannot prefer one neighbour over another based on their features, so any + task where *which* neighbour matters (one incriminating transaction among a + hundred routine ones) is beyond it. GAT makes the weight a learned function + of the endpoints' current features instead. + +
+ - [ ] You can explain why the softmax is over in-edges of v, and what normalizing over out-edges would mean. + +
Answer + + Normalization is per-destination: it is v deciding how to divide its + attention among its sources, so the softmax runs over v's in-edges and the + weights sum to 1 there. Normalizing over u's out-edges would instead make u + ration how much of itself it sends out — a different (and not what the paper + wants) semantics. The in-edge choice forces the kernel to iterate + in-neighbourhoods, which wants the transposed adjacency A^T resident + (topic 20's transpose tax). + +
+ - [ ] You can decompose a GAT layer into SDDMM, segmented softmax and SpMM. + +
Answer + + SDDMM computes the per-edge scores `e_uv` only at A's nonzeros (a masked + dense-dense product); a segmented softmax normalizes each CSR row's scores + into `alpha`; the SpMM aggregates `Σ_u alpha_uv · W h_u`. The middle kernel + is two passes (a max pass and an exp-sum pass), so a GAT layer is ~3 extra + edge passes on top of GCN's single SpMM. PyG runs them at gat_conv.py:392 + (score), :404 (softmax), :408 (message). + +
+ - [ ] You can count edge passes per GAT layer against per GCN layer on this topic's 566 K-edge SBM, whose SpMM measures 4.31 ms at 16.82 GFLOP/s. + +
Answer + + GCN is one edge pass (the SpMM). GAT is roughly four: SDDMM, softmax-max, + softmax-expsum, SpMM — about 4× the sparse traffic per layer, before the + ×K for multi-head. Anchoring to the measured SpMM (4.31 ms at 16.82 + GFLOP/s, FINDINGS.md row 25) as the unit edge pass, a single-head GAT layer + lands near 4×4.31 ≈ 17 ms of sparse work; do the full estimate against your + own bench in notes.md, and remember the SDDMM/softmax passes move less data + per edge than the 64-wide SpMM, so the true ratio is below 4. + +
+ - [ ] You wrote answers to all five questions in notes.md, including whether GAT is worth engine support at all. +
Answer + + All five `## Questions` answered in notes.md — the in-edge/transpose + question, the edge-pass forward-time estimate, the a_src/a_dst + factor-out-of-the-join trick, the Cypher surface for `attention > t`, and + the M25 argument from each variant's kernel inventory (whether GCN/SAGE + + the vector index already covers the 95% case). + +
+ ## References **Papers** @@ -193,5 +287,7 @@ is the explanation (question 4: what Cypher surface exposes it). **Code** - [pytorch_geometric](https://github.com/pyg-team/pytorch_geometric) - `torch_geometric/nn/conv/gat_conv.py` — score split :392, segmented - softmax :404, message :408; note the absent `message_and_aggregate` + `torch_geometric/nn/conv/gat_conv.py` — per-node score halves :330-331, + edge score `alpha_j + alpha_i` :392, `leaky_relu` (slope 0.2) :403, + segmented softmax :404, message :408-409; note the absent + `message_and_aggregate`, and `negative_slope=0.2` default at :136 diff --git a/topics/25-graph-ml/reading-gcn.md b/topics/25-graph-ml/reading-gcn.md index 9dcaab8..8e787d4 100644 --- a/topics/25-graph-ml/reading-gcn.md +++ b/topics/25-graph-ml/reading-gcn.md @@ -20,6 +20,12 @@ treating rows as independent. ### Step 1 — the task: semi-supervised node classification +> **In:** a graph — vertices with feature vectors, edges, and labels on a +> small fraction of vertices. +> **Out:** the requirement that drives every later step — each vertex's +> representation must depend on its neighbours', not just its own row. Step 2 +> is the mechanism that delivers it. + Each vertex carries a feature vector (for Cora: a 1,433-wide bag-of-words per paper) and a few vertices carry labels; the job is to predict labels for all the rest. A plain classifier over the @@ -33,6 +39,10 @@ neighbors'. ### Step 2 — the idea: average your neighbors, then transform +> **In:** the adjacency and the feature rows from Step 1. +> **Out:** one layer's rule — a neighbour-averaged, linearly transformed +> representation per vertex. Step 3 fixes the two bugs in the plain average. + One GCN layer sets each vertex's new representation to (roughly) the average of its neighbors' current representations, pushed through a small learned linear map and a nonlinearity. That's it — the @@ -46,8 +56,13 @@ spatial work, the weights only re-mix feature channels. ### Step 3 — A_hat: self-loops and symmetric normalization +> **In:** the raw adjacency `A` and the **degree matrix** `D` (the diagonal +> matrix whose entry `D_ii` is vertex *i*'s number of edges). +> **Out:** `A_hat`, the fixed propagation matrix computed once from the graph +> alone. Step 4 multiplies it against the features. + Raw neighbor-averaging has two bugs, and A_hat is the two-line fix -baked into a single matrix. The layer is: +baked into a single matrix. The layer is (Kipf & Welling eq. 2): ``` H(l+1) = sigma( D^-1/2 (A + I) D^-1/2 · H(l) · W(l) ) @@ -56,29 +71,55 @@ baked into a single matrix. The layer is: precomputed ONCE dense tiny dense ``` -- `A + I`: self-loops so a vertex keeps its own features (the - renormalization trick, §2.2). Without it, a vertex's own signal is - discarded each layer and deep stacking oscillates. -- Symmetric normalization `D^-1/2 · D^-1/2` (D = the diagonal degree - matrix): averages neighborhoods without letting hub degrees explode - activations — each edge (u, v) is weighted 1/√(d_u · d_v). Compare +- `A + I`: **self-loops** — adding the identity `I` so a vertex keeps its + own features (the renormalization trick, §2.2). Kipf & Welling write the + self-looped adjacency `Ã = A + I_N` and its degree `D̃_ii = Σ_j Ã_ij` + (§2.2). Without it, a vertex's own signal is discarded each layer and deep + stacking oscillates. +- Symmetric normalization `D̃^-1/2 · D̃^-1/2` (D̃ = the diagonal degree + matrix of `Ã`): averages neighborhoods without letting hub degrees explode + activations — each edge (u, v) is weighted `1/√(d̃_u · d̃_v)`. Compare topic 24's PageRank pull matrix (row-normalized `D^-1 A`) — same - idea, symmetric so the operator stays PSD-friendly, which is what + idea, but symmetric so the operator stays PSD-friendly, which is what keeps its eigenvalues in [-1, 1] (question 1: that bound is the whole point). +Work it by hand on a 3-vertex path `1—2—3` to see why symmetric ≠ +row-normalized. `A` has edges (1,2) and (2,3); `Ã = A + I`, so the +self-looped degrees are `D̃ = diag(2, 3, 2)` (vertex 2 has two neighbours +plus itself). With `Â_ij = Ã_ij / √(D̃_ii·D̃_jj)`: + +```text + 1 2 3 +1 [ 1/2 1/√6 0 ] row sum 0.908 +2 [ 1/√6 1/3 1/√6 ] row sum 1.149 +3 [ 0 1/√6 1/2 ] row sum 0.908 +``` + +The rows do **not** sum to 1 — the symmetric form scales every edge by +*both* endpoints' degrees, so it is not row-stochastic. The random-walk +matrix `D^-1 Ã` *would* be row-stochastic (rows `[1/2,1/2,0]`, +`[1/3,1/3,1/3]`, `[0,1/2,1/2]`). That distinction matters in Step 4: the +measured lane uses the row-stochastic form, not this symmetric one. + The critical systems fact: A_hat depends only on the graph, not the features or weights — compute it ONCE, reuse it every layer, every -epoch, every inference. PyG's `gcn_norm` (gcn_conv.py:45-71) is the -reference implementation: fill_diag with 1, deg^-0.5 masked at inf, -scale rows then columns. Our `gcn::gcn_norm` stub reproduces it in -CSR; the dense oracle `gcn_norm_dense` is the definitional check. +epoch, every inference. PyG's `gcn_norm` (gcn_conv.py:45, the dense/edge +branch) is the reference implementation: `add_self_loops` for `A + I`, then +`deg.pow_(-0.5)` with the infinities from isolated vertices masked to 0 +(gcn_conv.py:67-68), then scale rows and columns (gcn_conv.py:69-70). Our +`gcn::gcn_norm` stub reproduces it in CSR; the dense oracle +`gcn_norm_dense` is the definitional check. Two layers, softmax, cross-entropy on the few labeled nodes. That's the whole model: `Z = softmax(A_hat · relu(A_hat X W1) · W2)` (eq. 9). ### Step 4 — the kernel view: one SpMM plus one tiny matmul +> **In:** `A_hat` from Step 3, the `n×d` feature matrix, and the `d×h` weight. +> **Out:** one layer's output, expressed as two kernels — a sparse aggregation +> and a dense transform. Step 5 chooses the order to run them in. + Strip the ML vocabulary and one layer is two matrix products: a **SpMM** (sparse-times-dense matrix multiply — A_hat in CSR against the n×h dense feature matrix; the aggregation) and a small dense @@ -86,6 +127,9 @@ matmul (the n×d features against the d×h weights; the transform). One layer, no framework — a query plan with two operators: ```rust +// ILLUSTRATION — not quoted; the measured kernels are spmm.rs:18 +// (row_norm_adj SpMM) and gcn.rs:51 (gcn_layer), and PyG's fused form is +// gcn_conv.py:273 (message_and_aggregate -> spmm(adj_t, x, reduce=aggr)). fn gcn_layer(a_hat: &Csr, h: &Mat, w: &Dense) -> Mat { let t = h.matmul(w); // transform FIRST: n×d · d×h — because // h < d, this shrinks what SpMM drags @@ -100,12 +144,26 @@ fn gcn_layer(a_hat: &Csr, h: &Mat, w: &Dense) -> Mat { ``` Per layer: one SpMM (`2·nnz·h` FLOPs) + one small dense matmul -(`2·n·d·h`). On our SBM bench the SpMM runs at 21.2 GFLOP/s — 81% of -dense matmul's throughput — because the 64-float dense rows amortize -the sparse gather. Fat right-hand sides forgive sparsity. +(`2·n·d·h`). On this topic's SBM bench the message-passing SpMM runs at +**16.82 GFLOP/s in 4.31 ms, against 5.65 ms for the dense transform beside +it** (FINDINGS.md row 25) — roughly 71% of the dense kernel's throughput +(16.82 / 23.75 GFLOP/s, the dense figure being `2·n·d·h = 2·16384·64·64 = +134 MFLOP` over 5.65 ms). The 64-float dense rows amortize the sparse +gather: fat right-hand sides forgive sparsity. One honesty note (rule 6): +the *measured* lane normalizes with the row-stochastic `D^-1 A` +(`spmm.rs:38 row_norm_adj`, driven from `bin/gnn_bench.rs`), not the +symmetric `A_hat` of Step 3 — `gcn::gcn_norm` is still a stub. Both are the +same SpMM shape (`2·nnz·64`), so the timing is a faithful proxy for the +symmetric kernel; the number is the aggregation cost, not a claim about +which normalization ran. ### Step 5 — associativity is a query plan +> **In:** the three-factor product `A_hat · X · W` from Step 4. +> **Out:** the cheaper of the two evaluation orders, chosen by which +> dimension the sparse multiply has to drag. Step 6 reuses this plan at +> inference time. + `A_hat · X · W` can be evaluated `(A_hat X) W` or `A_hat (X W)`, and the choice swaps which term carries the big dimension — exactly a join-ordering decision (topic 10). The SpMM costs `2·nnz·(width of @@ -113,12 +171,17 @@ its dense operand)`: aggregate-first drags d-wide rows through the sparse multiply, transform-first drags h-wide rows. On Cora (n=2708, nnz=13K, d=1433, h=16) transform-first makes the sparse side 90x cheaper, and the DENSE transform dominates; on our SBM (nnz=566K, -d=64) they're comparable — measured 3.42 ms SpMM vs 5.12 ms dense at -64-wide. Transform-first wins whenever h < d. Frameworks hardcode -this; a database would COST it (topic 10). +d=64) they're comparable — measured **4.31 ms SpMM against 5.65 ms dense** +at 64-wide (FINDINGS.md row 25). Transform-first wins whenever h < d. +Frameworks hardcode this; a database would COST it (topic 10). ### Step 6 — inference is a query +> **In:** a trained model — the materialized `A_hat` and the two weight +> matrices `W1`, `W2`. +> **Out:** a forward pass as a fixed two-operator plan over stored data. +> Step 7 names where that plan hits its ceilings. + Training needs gradients and a framework; *inference* on a static graph needs neither. A_hat is a materialized matrix, W1 and W2 are two small constants, and a GCN forward pass is: SpMM, small matmul, @@ -131,6 +194,10 @@ materialized it — staleness semantics land on you, not the framework ### Step 7 — the limits, and why the next two papers exist +> **In:** the working two-layer GCN from Steps 3–6. +> **Out:** its three structural ceilings, each naming the successor paper +> that lifts it. + Three built-in ceilings, each motivating a successor: - Full-batch: every layer touches every vertex — memory O(n·d) per @@ -173,12 +240,77 @@ Three built-in ceilings, each motivating a successor: ## Done when +Answer each before unfolding it. + - [ ] You can write `A_hat = D^-1/2 (A+I) D^-1/2` and say why its eigenvalues lie in [-1, 1]. + +
Answer + + `Ã = A + I` adds self-loops; `D̃` is its diagonal degree matrix; the + symmetric scaling `D̃^-1/2 Ã D̃^-1/2` weights each edge by + `1/√(d̃_u·d̃_v)`. It is similar to the random-walk matrix `D̃^-1 Ã` + (share a spectrum via `D̃^1/2`), whose eigenvalues lie in [-1, 1] because + it is stochastic; the renormalization trick (§2.2) shifts the raw + `I + D^-1/2 A D^-1/2` — whose spectrum reaches into [0, 2] and blows up + under repeated application — into that stable band. Bounded eigenvalues + are what let you stack layers without activations exploding or vanishing. + +
+ - [ ] You can decompose a layer into one SpMM plus one small dense matmul. + +
Answer + + A layer is `σ(A_hat · X · W)`. `X · W` is a dense `n×d · d×h` matmul (the + transform, `2·n·d·h` FLOPs); `A_hat · (XW)` is a SpMM of the CSR matrix + against the `n×h` dense result (the aggregation, `2·nnz·h` FLOPs); `σ` is + a free elementwise relu. Two operators, one sparse and one dense — the + same shape as `spmm.rs:18` beside a dense `matmul`. + +
+ - [ ] You can explain why associativity is a query plan, and count FLOPs both ways on this topic's SBM — the measured SpMM is 4.31 ms against 5.65 ms for the dense transform. + +
Answer + + `(A_hat X) W` vs `A_hat (X W)` are equal by associativity but cost + differently: the SpMM's cost is `2·nnz·(width of its dense operand)`, so + aggregate-first drags the `d`-wide feature rows through the sparse + multiply and transform-first drags the `h`-wide rows. Transform-first + wins whenever `h < d`. On the SBM (nnz≈566K, d=h=64) the two widths match, + so the measured kernels are comparable: 4.31 ms for the SpMM at 16.82 + GFLOP/s against 5.65 ms for the dense transform (FINDINGS.md row 25). + Picking the order is exactly join-ordering (topic 10). + +
+ - [ ] You can say what being baked into `A_hat` at training time costs when a node arrives. + +
Answer + + `A_hat` is materialized from the graph as it stood when you built it, so a + new node or edge is invisible until you recompute the affected rows — + `add_self_loops` and the `1/√(d̃_u·d̃_v)` scaling both shift when a + neighbour's degree changes. `W1`/`W2` are unaffected (they re-mix feature + channels, not structure) and need no refresh. So the cheap fix is to + re-normalize the touched rows of `A_hat`; retraining `W` is the expensive + path and usually unnecessary for a small structural delta. + +
+ - [ ] You wrote answers to all five questions in notes.md, including what pending deltas mean for a forward pass over the M20 graph. +
Answer + + The five questions live in `## Questions`; the M25 one asks whether the + M20 delta-matrix's pending (un-merged) edges participate in `A_hat`. They + do only if you fold the delta into the degree counts and re-scale the + affected rows before the SpMM — the same three-way "read committed / read + pending / merge-then-read" choice as topic 24's `CALL algo.wcc`. Write the + reasoning out in notes.md. + +
+ ## References **Papers** @@ -189,5 +321,7 @@ Three built-in ceilings, each motivating a successor: **Code** - [pytorch_geometric](https://github.com/pyg-team/pytorch_geometric) - `torch_geometric/nn/conv/gcn_conv.py` — `gcn_norm` (:45-71) is the - reference A_hat construction our `gcn::gcn_norm` stub reproduces + `torch_geometric/nn/conv/gcn_conv.py` — `gcn_norm` (def at :45, the + `deg.pow_(-0.5)` scaling at :67-70) is the reference A_hat construction our + `gcn::gcn_norm` stub reproduces; `GCNConv.message_and_aggregate` (:273) + is the fused `spmm(adj_t, x, reduce='add')` form of Step 4. diff --git a/topics/25-graph-ml/reading-graphrag-sdk.md b/topics/25-graph-ml/reading-graphrag-sdk.md index e227f26..e40824f 100644 --- a/topics/25-graph-ml/reading-graphrag-sdk.md +++ b/topics/25-graph-ml/reading-graphrag-sdk.md @@ -21,6 +21,11 @@ plan. ### Step 1 — RAG, and why a graph gets involved +> **In:** a private document corpus and a natural-language question. +> **Out:** the evidence the LLM needs — fetched by vector similarity and, in +> GraphRAG, by explicit graph structure. Step 2 is the pipeline that builds +> and reads both. + **RAG** (retrieval-augmented generation) answers questions by retrieving relevant evidence from a private corpus and stuffing it into an LLM's prompt — the LLM supplies fluency, retrieval supplies @@ -38,6 +43,11 @@ neighbors. ### Step 2 — the pipeline as a dataflow +> **In:** the corpus (ingestion) and a question (query) from Step 1. +> **Out:** two indexes written once and read together — a graph store and a +> vector store on one FalkorDB instance. Step 3 is the storage contract they +> imply. + The SDK's whole shape is one ingestion path that writes two indexes, and one query path that reads them both: @@ -60,16 +70,21 @@ client-side fiction, which is exactly why the joins in Step 4 hurt. ### Step 3 — the storage contract: what the SDK asks the database for +> **In:** the two-index dataflow from Step 2. +> **Out:** the concrete set of database features the SDK depends on — three +> index types queried through Cypher, plus an external write path. Step 4 is +> the joins layered on top. + `storage/vector_store.py` is the SDK's entire database contract, and reading it tells you which features carry the workload: | anchor | what | |---|---| -| `:344` | `CALL db.idx.vector.queryNodes('{label}', 'embedding', $top_k, vecf32($vector))` — chunk ANN | +| `:344` | `CALL db.idx.vector.queryNodes('{safe_label}', 'embedding', $top_k, vecf32($vector))` — chunk ANN | | `:378` | same over `__Entity__` — entity ANN | -| `:426` | `queryRelationships('RELATES', ...)` — EDGE vectors, with a Cypher cosine-scan fallback (:414) if unsupported | +| `:426` | `queryRelationships('RELATES', ...)` — EDGE vectors, with a Cypher cosine-scan fallback (`vecf32.distance.cosine`, :454-458) if unsupported | | `:219,:234,:312` | `SET c.embedding = vecf32($vector)` — embeddings computed OUTSIDE, written back as properties | -| `:133` | full-text index too — hybrid = vector + FT + graph, three indexes on one store | +| `:133` | `create_fulltext_index` too — hybrid = vector + FT + graph, three indexes on one store | Note the asymmetry: the read path is database-native (three index types queried through Cypher), but the WRITE path — embedding @@ -80,6 +95,10 @@ leave the database — only text embeddings need the round-trip. ### Step 4 — retrieval strategies: joins, hand-rolled in the client +> **In:** the storage contract from Step 3 (ANN + Cypher on one store). +> **Out:** each retrieval strategy as a query plan the Python client executes +> by hand — the k+1 round trips Step 6 files as smell #1. + Each retrieval strategy is a query plan executed by Python instead of the database: @@ -89,11 +108,16 @@ the database: and the graph: k queries where one Cypher query with a vector predicate should do — the exact hybrid query M25's capstone must serve in ONE plan. -- `multi_path.py:48` runs chunk-ANN, entity-ANN, edge-ANN - concurrently, reranks with client-side `_cosine_sim` (:362) — a - scatter-gather union of three indexes with score fusion done in - Python. Compare topic 23's WAND: score fusion is what the engine's - top-k machinery is FOR. +- `multi_path.py:48` `MultiPathRetrieval` runs a 9-phase pipeline + (`_execute`, :182) that fans out across chunk, entity and edge indexes + plus Text-to-Cypher, then reranks with a client-side `_cosine_sim` + (:362). Its explicit concurrency is one `asyncio.gather` (:198) running + the RELATES-edge vector search and the Text-to-Cypher retrieval in + parallel; the entity-discovery and chunk-retrieval phases are separate + awaited steps, not a single three-way gather. Either way it is a + scatter-gather union of several indexes with score fusion done in Python + — compare topic 23's WAND: score fusion is what the engine's top-k + machinery is FOR. The cost is structural, not incidental: every strategy pays k+1 round trips and recomputes distances the index already knew. The asyncio @@ -101,18 +125,30 @@ sophistication is compensation for a missing query surface. ### Step 5 — the router: a planner with no cost model +> **In:** the several retrieval strategies from Step 4. +> **Out:** a per-question strategy choice — made by first-matching predicate, +> with no statistics. Step 6 collects this and the other systems smells. + `router.py:19` `SemanticRouter` picks a retrieval strategy per -question by embedding the question and matching it against strategy -descriptions — a query PLANNER driven by embeddings instead of -statistics (topic 9 with vibes). The structure is right (multiple -plans, a chooser); what's missing is everything topic 9 built: -cardinality estimates, cost per plan, feedback from execution. -Question 4 asks what statistic would turn "graph expansion vs pure -ANN" into a costed choice — the router names M25's planner-shaped -hole. +question, but read what the pinned version actually does (rule 6): despite +the name, it is **not** embedding-driven. Strategies register a +`condition(query) -> bool` callable (`register`, :46 — the docstring's +example is `lambda q: "how" in q.lower()`), and `_select` (:84) returns the +first strategy whose predicate fires, falling back to a default (:98). The +class docstring says so outright — "In v1, this is a simple rule-based +router." So it is a query PLANNER driven by keyword rules, not by +cardinality (topic 9): the structure is right (multiple plans, a chooser), +but everything topic 9 built is missing — cost per plan, selectivity +estimates, feedback from execution. Question 4 asks what statistic would +turn "graph expansion vs pure ANN" into a costed choice — the router names +M25's planner-shaped hole. ### Step 6 — the four systems smells: M25's worklist +> **In:** everything Steps 3–5 found the client doing by hand. +> **Out:** four named engine deficiencies, each mapped to a topic already +> covered — the worklist M25 closes. + Reading the whole SDK as a bug report against the engine yields four named deficiencies: @@ -143,8 +179,9 @@ database. write halves of the contract. - **Step 4 — the joins**: `retrieval/strategies/relationship_expansion.py` (:12, :35, :62) - and `retrieval/strategies/multi_path.py` (:48, :362). -- **Step 5 — the router**: `retrieval/router.py:19`. + and `retrieval/strategies/multi_path.py` (:48 class, :182 `_execute`, + :198 `asyncio.gather`, :362 `_cosine_sim`). +- **Step 5 — the router**: `retrieval/router.py:19` (`_select` :84). - Navigation advice: read each file as a feature request against the engine, not as Python to review — the question is never "is this code good" but "which missing engine feature made this code @@ -169,11 +206,69 @@ database. ## Done when +Answer each before unfolding it. + - [ ] You can draw the pipeline as a dataflow and name what the SDK asks the database for. +
Answer + + Ingestion writes two indexes on one FalkorDB store — a property graph + (`__Entity__` nodes, `RELATES` edges) and vector indexes over chunks, + entities and edges (`vector_store.py` `:344/:378/:426`), plus a + full-text index (`:133`). Query reads both together. What the SDK asks + the database for: ANN over node/edge vectors, Cypher pattern matching, + full-text search, and a write path that stores externally-computed + embeddings back as `vecf32` properties (`:219/:234/:312`). Three index + types, one store. + +
- [ ] You can write the single Cypher query that replaces the client-side relationship expansion. +
Answer + + `expand_relationships` (`relationship_expansion.py:12`) does ANN, then + one `MATCH (a:__Entity__ {id: eid})-[r:RELATES]->(b)` per hit (`:35`). + The one-query form pushes the vector predicate INTO the pattern: + `CALL db.idx.vector.queryNodes('__Entity__','embedding',$k,vecf32($v)) + YIELD node MATCH (node)-[r:RELATES]->(b) RETURN node, r, b`. For the + planner not to run it as k+1 lookups anyway it must treat the vector + index as a leaf operator feeding the expand, not a black box called k + times — i.e. a join order that keeps the ANN result set streaming into + the pattern match. + +
- [ ] You can name all four systems smells and say which one `SET c.embedding = vecf32(...)` inside a loop is. +
Answer + + The four: (1) k+1 round trips, (2) client-side rerank, (3) + non-transactional embedding writes, (4) no incremental re-embed. + `SET c.embedding = vecf32(...)` after ingest is smell #3: the embedding + write is a separate batch from entity creation (`vector_store.py` + `:219/:234/:312`), so there is a staleness window with no + read-your-writes guarantee (topic 8). + +
- [ ] You can say what statistic would give the router a cost model. +
Answer + + The router (`router.py:19`) is first-match predicate selection + (`_select` `:84`), no statistics. To cost "graph expansion vs pure + ANN" you need the pattern's selectivity (how many `RELATES` neighbours + a matched entity expands to — expansion fan-out) and the ANN index's + recall@k, so a plan that expands can be priced against one that does + not. That is exactly topic 9's cardinality-estimation machinery, absent + here. + +
- [ ] You wrote answers to all five questions in notes.md, including the M25 acceptance test that puts pattern and similarity in one query. +
Answer + + The acceptance test: a single query that both matches a graph pattern + and ranks by vector similarity — e.g. entities within 2 hops of a seed + ranked by embedding distance to the question — executed as ONE engine + plan (vector predicate pushed into the pattern match, distances + returned by the index, no Python rerank), and verified to return the + same answers this SDK produces by its k+1 round trips on the same data. + +
## References diff --git a/topics/25-graph-ml/reading-graphsage.md b/topics/25-graph-ml/reading-graphsage.md index a4b6786..5488eab 100644 --- a/topics/25-graph-ml/reading-graphsage.md +++ b/topics/25-graph-ml/reading-graphsage.md @@ -20,6 +20,10 @@ so either you bound the fan-in or you don't train at all. ### Step 1 — transductive vs inductive: a lookup table vs a function +> **In:** the modelling choice — what a trained GNN actually stores. +> **Out:** the transductive/inductive split that decides whether a new vertex +> is serveable. Step 2 is the aggregator that makes the inductive side work. + A **transductive** method learns one vector per vertex that existed at training time — the model *is* a lookup table (node2vec, GCN as usually trained). An **inductive** method learns a *function* from a @@ -34,6 +38,11 @@ inductive version work: don't learn *where each node goes*, learn ### Step 2 — the aggregator layer: summarize neighbors, keep yourself +> **In:** the sampled neighbourhood (Step 4's fan-in) and each vertex's +> previous-layer representation. +> **Out:** the vertex's next-layer representation. Step 3 shows why the +> neighbourhood must be sampled at all. + Each GraphSAGE layer computes a vertex's new representation from two inputs kept deliberately separate — a summary of its (sampled) neighbors, and its own previous representation (Alg. 1): @@ -46,11 +55,16 @@ neighbors, and its own previous representation (Alg. 1): ``` - AGG ∈ {mean, LSTM, max-pool} — any order-insensitive summary of a - set of vectors. Mean-SAGE ≈ GCN without the symmetric - normalization; PyG's SAGEConv fuses it as - `spmm(adj_t, x, reduce=mean)` (sage_conv.py:149-152) with the self - path as a separate `lin_r` (sage_conv.py:108,139) — concat - implemented as sum of two linears. + set of vectors. Careful (rule 6): the paper's default **mean aggregator** + (Alg. 1) concatenates the vertex's own vector with the neighbour mean, + whereas its separate *convolutional/GCN variant* (Eq. 2) folds self + *into* the mean and drops the concat — so "mean aggregator" is subtly + **not** the GCN rule, and Hamilton et al. say so in §3.3. PyG's `SAGEConv` + (default `aggr="mean"`, sage_conv.py:70) implements the concat form: it + fuses the neighbour mean as `spmm(adj_t, x[0], reduce="mean")` + (sage_conv.py:152) and adds the self path as a separate `lin_r` + (sage_conv.py:108, used at :139) — a concat expressed as the sum of two + linears. - The concat `[h_v || h_N(v)]` (rather than adding self into the average) preserves "what I am" and "what surrounds me" as separate learnable channels — question 1 asks what the two-linears trick @@ -60,6 +74,9 @@ neighbors, and its own previous representation (Alg. 1): One mean-SAGE layer for one node, sampling included: ```rust +// ILLUSTRATION — not quoted; PyG's fused mean aggregator is +// sage_conv.py:149-152 (message_and_aggregate -> spmm(adj_t, x[0], +// reduce="mean")) with the self path at sage_conv.py:139. fn sage_layer(g: &Csr, h: &Mat, v: u32, s: usize, w_self: &Dense, w_nbr: &Dense, rng: &mut Rng) -> Vec { let mut agg = vec![0.0; h.d]; @@ -75,6 +92,11 @@ fn sage_layer(g: &Csr, h: &Mat, v: u32, s: usize, ### Step 3 — the fan-out explosion: why full neighborhoods can't ship +> **In:** the K-layer aggregator from Step 2 and a minibatch of B seed +> vertices. +> **Out:** the size of the K-hop neighbourhood that batch must load — the +> quantity Step 4 bounds. + Stacking K layers means a vertex's output depends on its K-hop neighborhood — so a minibatch of B seeds must *load* the union of their K-hop neighborhoods, and that union multiplies per layer: @@ -94,12 +116,18 @@ means unbounded memory means no training loop — hence Step 4. ### Step 4 — neighbor sampling: a page budget for graph access +> **In:** Step 3's unbounded K-hop neighbourhood. +> **Out:** a fixed per-batch cost `B·S1·S2` from a uniform fan-in cap. Step 5 +> is the accuracy price of that cap. + GraphSAGE's fix is blunt: at each layer, use only a fixed-size uniform sample of each vertex's neighbors — S1=25 at layer 1, S2=10 at layer 2 — making every batch cost B·S1·S2 regardless of what the -degree distribution does. This is a query optimizer problem stated in -ML clothes: the full neighborhood is the correct answer, the sample -is an approximation with a resource bound. PyG's `NeighborLoader` +degree distribution does. Those two numbers are the paper's own: "we set +K=2 with neighborhood sample sizes S1=25 and S2=10" with the budget +`S1·S2 ≤ 500` (Hamilton et al. §4.1). This is a query optimizer problem +stated in ML clothes: the full neighborhood is the correct answer, the +sample is an approximation with a resource bound. PyG's `NeighborLoader` (loader/neighbor_loader.py:10) industrializes it; the sampled subgraph handed to the model is exactly a database *view* — materialized per batch, biased by design. Mechanically it's cheap: @@ -111,6 +139,10 @@ sample answers well enough. ### Step 5 — what the sample costs: bias you must measure +> **In:** the sampled aggregation from Step 4. +> **Out:** the bias and run-to-run variance it introduces — a number you +> measure, not assume. Step 6 is why the whole scheme is worth it. + The bound isn't free. Sampled mean-aggregation is an unbiased estimator of the true mean only *before* the nonlinearity — after sigma, the estimate is biased, and the per-epoch re-sampling variance @@ -124,6 +156,10 @@ two meet. ### Step 6 — why inductive is the database-compatible variant +> **In:** the inductive aggregator (Step 1) and the bounded fan-in (Step 4). +> **Out:** the reason SAGE is the only variant here that survives a +> write-heavy database — plus the staleness question it leaves open. + Put Steps 1 and 4 together and GraphSAGE is the only GNN variant in this topic that composes with a write-heavy database. Transductive embeddings (node2vec, GCN-as-trained) go stale on insert — the vertex @@ -171,13 +207,83 @@ stale is acceptable — question 4 makes this precise. ## Done when +Answer each before unfolding it. + - [ ] You can state the transductive/inductive distinction as a lookup table against a function. + +
Answer + + Transductive learns one vector per training-time vertex — the model *is* a + lookup table (node2vec, GCN as usually trained), and a vertex absent at + training has no row. Inductive learns a *function* from features and + neighbourhood to embedding, so any vertex — including one inserted after + training — gets an embedding from a forward pass. The gap is invisible on a + frozen benchmark and decisive under writes. + +
+ - [ ] You can explain the fan-out explosion and compute nodes touched for B=512, S=(25,10) against a full 2-hop on this topic's SBM (avg degree 34.6). + +
Answer + + A K-layer model needs each seed's K-hop neighbourhood, and the union + multiplies per layer. With sampling, B=512 and (S1,S2)=(25,10) touches + `B·S2·S1 = 512·10·25 = 128,000` nodes — a fixed budget. Full 2-hop on the + SBM (avg deg 34.6) touches ≈ `B·34.6² = 512·1197 ≈ 613,000` on average, and + far more on any batch containing a hub, since the worst case is `B·d_hub²`. + Sampling flattens that skew to a constant. + +
+ - [ ] You can explain why neighbour sampling is a page budget for graph access. + +
Answer + + Fixing S1, S2 caps how much adjacency a batch reads regardless of degree — + exactly a page/I-O budget. The full neighbourhood is the correct answer; + the fixed-size uniform sample is an approximation with a hard resource + bound, and over CSR it is O(S) contiguous reads per row. It is the same + refusal-to-pay-for-everything as Afforest's r-neighbour sample (topic 24). + +
+ - [ ] You can say what bias the sample introduces and how you would measure it. + +
Answer + + The sampled mean is unbiased for the true mean *before* the nonlinearity; + after σ the estimate is biased, and re-sampling each epoch turns into + run-to-run accuracy variance. Measure it the topic-22 way: same model, same + data, five seeds, report the spread — do not quote a single accuracy as if + it were a constant. + +
+ - [ ] You can explain why inductive is the database-compatible variant, in terms of which embeddings an insert invalidates. + +
Answer + + A transductive table has no entry for a newly inserted vertex — you must + retrain or serve garbage. A SAGE aggregator is a stored function: the new + vertex gets an embedding from one forward pass over its sampled + neighbourhood, at a bounded `S1·S2` reads. What remains is staleness — an + embedding computed at snapshot T and read at T+k is a stale materialized + result, and topic 8's read-your-writes / monotonic-reads vocabulary is how + you state the tolerance. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + All five `## Questions` answered in notes.md — including the two-linears vs + true-concat expressiveness question, the nodes-touched computation across + the SBM and an RMAT hub, and the transductive-vs-inductive shipping + decision for M25's `algo.embed()`. + +
+ ## References **Papers** diff --git a/topics/25-graph-ml/reading-node2vec.md b/topics/25-graph-ml/reading-node2vec.md index 4e59b4f..8b419a1 100644 --- a/topics/25-graph-ml/reading-node2vec.md +++ b/topics/25-graph-ml/reading-node2vec.md @@ -22,6 +22,10 @@ not the learner. ### Step 1 — node embeddings: geometry as a stand-in for structure +> **In:** a graph — vertices and edges, no coordinates. +> **Out:** one dense vector per vertex, where distance in vector space stands +> in for structural closeness. Step 2 is how those vectors get trained. + A **node embedding** assigns each vertex a dense vector (say 128 floats) such that geometric closeness in vector space stands in for structural closeness in the graph. Once vertices are points, the @@ -35,40 +39,62 @@ can change after the snapshot is taken. ### Step 2 — walks as sentences: borrow word2vec wholesale +> **In:** the graph's adjacency (a CSR neighbour list). +> **Out:** a corpus of walk "sentences" — vertex-id sequences. Step 3 feeds +> them to word2vec; Step 4 biases how they are generated. + A **random walk** — start at a vertex, repeatedly hop to a random neighbor, record the sequence — turns a graph into a corpus of "sentences" whose "words" are vertex ids. That is DeepWalk's entire insight: word2vec (the standard word-embedding trainer) only needs a stream of tokens where co-occurrence implies relatedness, and vertices that co-occur on short walks are exactly the related ones. -Generate, say, 10 walks of length 80 per vertex, and the learning +Generate, say, 10 walks of length 80 per vertex (the paper's settings: +`r = 10` walks per node, `l = 80` per walk, §4.1), and the learning half of the problem is *finished* — solved by an NLP tool that never -knows it's looking at a graph. Cheap, too: our scalar Rust walker -does 42.8 million steps/second on an M3 Pro. +knows it's looking at a graph. Cheap, too: this topic's scalar Rust +walker does **42.8 million steps/second** on an M3 Pro (notes.md, +`uniform walks 65,536 × 40`). ### Step 3 — skip-gram with negative sampling: the training objective -Skip-gram trains two vectors per vertex (an embedding z and a context -vector c) so that pairs that co-occur within a window on some walk -get high dot products, and random pairs get low ones. Maximize -`log sigma(z_u . c_v)` for co-visited pairs (sigma = the sigmoid -squashing a dot product into a probability), and `log sigma(-z_u . c_n)` -for k random "negative" vertices n — the negatives are what stop the -trivial solution where every vector is identical. PyG's -`Node2Vec.loss` (node2vec.py:135-160) is a direct transcription — -read it as the reference: two embedding lookups, inner product, -`-log(sigmoid)`, positive + negative terms summed. Walk generation -there is `torch.ops.pyg.random_walk` (node2vec.py:64) — a custom -C++/CUDA op, because Python-level walking would dominate runtime. -The lesson in that anchor: in this whole pipeline, the *walker* is -the systems bottleneck, not the SGD. +> **In:** the walk corpus from Step 2. +> **Out:** trained embedding vectors — the Step 1 output. Step 4 changes how +> the corpus is sampled, not this objective. + +**Skip-gram with negative sampling (SGNS)** trains vectors so that pairs +that co-occur within a window on some walk get high dot products, and +random pairs get low ones. Maximize `log sigma(z_u . c_v)` for co-visited +pairs (sigma = the sigmoid squashing a dot product into a probability), +and `log sigma(-z_u . c_n)` for k random "negative" vertices n — the +negatives are what stop the trivial solution where every vector is +identical. Classic SGNS keeps *two* tables — an embedding `z` and a +separate context `c` per vertex (this repo's `embed.rs:27` stub does +exactly that). PyG's `Node2Vec.loss` (node2vec.py:135) takes a shortcut +worth noting (rule 6): it looks the start node and the context node up in +**the same `self.embedding` table** (node2vec.py:140 and :142), so in that +implementation there is one vector per vertex, not two. Read it as the +reference for the *shape* — two lookups, inner product, `-log(sigmoid)` +for the positive term (:146), `-log(1 - sigmoid)` for the negatives +(:157), summed (:159). Walk generation there is `torch.ops.pyg.random_walk` +(node2vec.py:64) — a custom C++/CUDA op, because Python-level walking +would dominate runtime. The lesson in that anchor: in this whole +pipeline, the *walker* is the systems bottleneck, not the SGD. ### Step 4 — the p/q bias: a second-order walk +> **In:** Step 2's uniform walk, plus the vertex `t` you arrived from. +> **Out:** a biased next-hop distribution over `v`'s neighbours — three +> weight classes set by `p` and `q`. Step 5 reads off what they buy. + node2vec's contribution is to bias Step 2's uniform walk with two knobs, evaluated against the *previous* vertex t — making it a **second-order walk** (the next-hop distribution depends on the edge -you arrived by, not just where you stand): +you arrived by, not just where you stand). The paper's unnormalized +transition weight is `π_vx = α_pq(t,x) · w_vx`, where the search bias +`α_pq(t,x)` is `1/p` if `d_tx = 0`, `1` if `d_tx = 1`, and `1/q` if +`d_tx = 2` (§3.2, eq. for α), and `d_tx` is the shortest-path distance +from the previous vertex t to the candidate x: ``` came from t, now at v — where next? @@ -81,17 +107,22 @@ you arrived by, not just where you stand): ``` Every neighbor of v falls into exactly three classes by its distance -from t: t itself (weight 1/p — the backtrack knob), mutual neighbors -of t and v (weight 1 — sideways), everything else (weight 1/q — the -outward knob). This figure is §3.2, and §3.2 is the whole paper. The -second-order property is what costs: any preprocessing must be -per-EDGE (t, v), not per-node — which is where Step 6's trap comes -from. +from t: t itself (`d_tx = 0`, weight `1/p` — **p is the return +parameter**, the backtrack knob), mutual neighbors of t and v +(`d_tx = 1`, weight 1 — sideways), everything else (`d_tx = 2`, weight +`1/q` — **q is the in-out parameter**, the outward knob). This figure is +§3.2, and §3.2 is the whole paper. The second-order property is what +costs: any preprocessing must be per-EDGE (t, v), not per-node — which is +where Step 6's trap comes from. ### Step 5 — what the knobs buy: roles vs communities +> **In:** the `p`/`q` bias from Step 4. +> **Out:** which notion of similarity the embedding encodes — structural +> roles or communities. Step 6 is what this costs to sample. + The q knob selects which *kind* of similarity the embedding encodes, -by shaping what a walk's co-occurrence window contains: +by shaping what a walk's co-occurrence window contains (§3.1): - q > 1: stay near t — BFS-flavored samples → embeddings encode *structural roles* (hubs look like hubs, bridges like bridges, @@ -110,15 +141,19 @@ query knob. ### Step 6 — the systems trap: alias tables vs rejection sampling +> **In:** Step 4's per-edge weighted distribution. +> **Out:** an O(1) sampler and its memory bill — the reason node2vec earned +> its "doesn't scale" reputation. Step 7 maps the fix onto engine machinery. + Sampling from Step 4's weighted distribution in O(1) is a solved problem — an **alias table** (a precomputed pair of arrays that turns a biased die roll into one uniform draw plus one comparison) — but because the walk is second-order, the original implementation builds one alias table per directed edge over the destination's neighbors: O(1) sampling but **O(m · avg_deg) memory** — on our 16K-vertex SBM -that's 566K x 34.6 ≈ 20M table entries for a toy graph. This is the -documented reason node2vec "doesn't scale"; it's the sampling that -doesn't. Fixes: +that's `m = 566,564` directed edges × `avg_deg 34.6 ≈ 19.6M ≈ 20M` table +entries for a toy graph (notes.md graph stats). This is the documented +reason node2vec "doesn't scale"; it's the sampling that doesn't. Fixes: - rejection sampling (KnightKing, our stub's prescription): draw uniform from N(v), accept with w/w_max, w_max = max(1, 1/p, 1/q). @@ -129,6 +164,8 @@ doesn't. Fixes: One biased step via rejection, the whole mechanism: ```rust +// ILLUSTRATION — not quoted; the measured node2vec step is walks.rs:56, +// with the 1/p, 1, 1/q weights at walks.rs:37-40. fn step(g: &Csr, t: u32, v: u32, p: f64, q: f64, rng: &mut Rng) -> u32 { let w_max = 1f64.max(1.0 / p).max(1.0 / q); loop { @@ -143,6 +180,10 @@ fn step(g: &Csr, t: u32, v: u32, p: f64, q: f64, rng: &mut Rng) -> u32 { ### Step 7 — what this looks like from inside a database +> **In:** the walk + sampler machinery from Steps 2–6. +> **Out:** the mapping onto structures an engine already owns — CSR rows, +> binary search, seeded RNG. + Everything above maps onto machinery an engine already owns: - Walks are embarrassingly parallel and CSR-native (CSR = compressed @@ -190,12 +231,90 @@ Everything above maps onto machinery an engine already owns: ## Done when +Answer each before unfolding it. + - [ ] You can explain why the walk bias must be second-order to interpolate between BFS-ish and DFS-ish neighbourhoods. + +
Answer + + The bias `α_pq(t,x)` is a function of the *previous* vertex t as well as + the current vertex v — it classifies each candidate x by `d_tx ∈ {0,1,2}`. + A first-order rule (one that only sees v) cannot tell "back toward t" from + "onward, away from t", so it cannot dial between staying local (BFS-ish, + `q>1`) and exploring outward (DFS-ish, `q<1`). The memory of where you came + from is the whole mechanism, and it is what forces per-edge preprocessing. + +
+ - [ ] You can say what p and q buy — roles against communities — and predict the effect before running the lane. + +
Answer + + `q>1` keeps walks near the origin (BFS-flavoured), so co-occurrence + captures *structural roles* — hubs resemble hubs even far apart (structural + equivalence, §3.1). `q<1` pushes outward (DFS-flavoured), so co-occurrence + captures *communities* (homophily). `p` (the return parameter) tunes + backtracking: large `p` discourages returning to t, small `p` keeps the + walk glued locally. On the ring-of-cliques lane, `q=0.25` should visit + >1.15× more distinct vertices per walk than `q=4`. + +
+ - [ ] You can state the skip-gram-with-negative-sampling objective. + +
Answer + + Maximize `log σ(z_u · c_v)` over pairs (u,v) that co-occur within a window + on a walk, and `Σ_n log σ(-z_u · c_n)` over k random negatives n. The + positive term pulls co-visited vectors together; the negatives push random + pairs apart and prevent the degenerate all-equal solution. In PyG's + `Node2Vec.loss` the two roles read from one shared table (node2vec.py:140, + :142); classic SGNS and this repo's `embed.rs` use separate embedding and + context tables. + +
+ - [ ] You can explain the alias-table against rejection-sampling trade and compute the expected draw count at p=1, q=0.25. + +
Answer + + Alias tables give O(1) draws but need one table per directed edge for a + second-order walk — O(m·avg_deg) ≈ 20M entries on the SBM. Rejection + sampling needs O(1) memory: propose uniformly from N(v), accept with + probability `w/w_max`, `w_max = max(1, 1/p, 1/q)`. Expected draws = the + reciprocal of the mean acceptance probability. At `p=1, q=0.25`, + `w_max = max(1,1,4) = 4`; a candidate's weight is 1 for return/mutual and + `1/q = 4` for outward, so acceptance depends on the local mix — at a bridge + vertex where most neighbours are "away", mean acceptance ≈ (weighted mean + of w)/4, giving ≈ 4/(fraction near 4) draws. Derive the exact figure from + the bridge's degree split in notes.md. + +
+ - [ ] You can explain embeddings as a materialized view and say which ones an edge insert invalidates. -- [ ] You wrote answers to all five questions in notes.md, and compared your walk rate against the measured uniform-walk baseline of 35.1 Msteps/s. + +
Answer + + Frozen embeddings are a materialized view over the walk corpus. One edge + insert changes the transition distribution at both endpoints and therefore + any walk that *could* have passed through them — unboundedly many, since a + walk reaching either endpoint later is affected too. That is why the view + is effectively non-incremental (topic 27): you cannot cheaply patch a + bounded set of walks, so re-embedding is periodic, not per-write. + +
+ +- [ ] You wrote answers to all five questions in notes.md, and compared your walk rate against the measured uniform-walk baseline of 42.8 Msteps/s. + +
Answer + + The baseline is `42.8 Msteps/s` (notes.md, `uniform walks 65,536 × 40` = + 2.62M steps in 61.2 ms ≈ 23 ns/step). Record your own biased-walk rate + beside it: rejection sampling adds a `has_edge` binary search per candidate + and repeats on rejection, so expect it below the uniform figure, more so as + p, q move away from 1. All five `## Questions` answered in notes.md. + +
## References @@ -207,5 +326,6 @@ Everything above maps onto machinery an engine already owns: **Code** - [pytorch_geometric](https://github.com/pyg-team/pytorch_geometric) - `torch_geometric/nn/models/node2vec.py` — `loss` (:135-160) is a - direct SGNS transcription; walks are a custom op (:64) + `torch_geometric/nn/models/node2vec.py` — `loss` (:135, positive term + :146, negative :157, sum :159) reads start and context from one shared + `self.embedding` (:140, :142); walks are a custom op (:64) diff --git a/topics/25-graph-ml/reading-pyg-message-passing.md b/topics/25-graph-ml/reading-pyg-message-passing.md index a18eea9..4720091 100644 --- a/topics/25-graph-ml/reading-pyg-message-passing.md +++ b/topics/25-graph-ml/reading-pyg-message-passing.md @@ -20,6 +20,12 @@ that the fused path replaces with zero bytes. ### Step 1 — message passing: three overridable functions +> **In:** the observation that every GNN layer is "combine each vertex's +> neighbours' vectors somehow". +> **Out:** the `MessagePassing` skeleton — `message`, `aggregate`, `update` +> (plus the fused `message_and_aggregate`) behind one `propagate` dispatcher. +> Step 2 executes it the literal way. + **Message passing** is the GNN literature's common skeleton: for each edge, compute a **message** from the source vertex's vector; at each vertex, **aggregate** the incoming messages with an order-insensitive @@ -35,11 +41,16 @@ goes in `message`, and can it fuse? ### Step 2 — the COO path: gather, message, scatter — and the m×d temp +> **In:** the `MessagePassing` skeleton from Step 1 and a COO `edge_index`. +> **Out:** the literal gather/message/scatter execution — and the `m×d` +> temporary it materializes. Step 3 fuses that temporary away. + The general execution strategy stores edges as a COO list (coordinate format — a 2 × m array of (source, destination) pairs) and runs the skeleton literally: gather each source's vector, apply -`message` per edge, scatter-reduce the results by destination. The -literal reading has a cost — the per-edge messages exist all at once: +`message` per edge (message_passing.py:523), scatter-reduce the results by +destination (`aggregate`, :541), then `update` (:550). The literal reading +has a cost — the per-edge messages exist all at once: ``` edge_index (COO 2 x m) adj_t (CSR/SparseTensor) @@ -53,6 +64,8 @@ literal reading has a cost — the per-edge messages exist all at once: The COO path, de-tensored — see the m×d temp being born: ```rust +// ILLUSTRATION — not quoted; PyG's COO branch is message_passing.py:499 +// (the `else`), with message at :523 and scatter aggregate at :541. fn propagate_coo(edges: &[(u32, u32)], x: &Mat, msg: impl Fn(&[f32]) -> Vec) -> Mat { let mut tmp = Vec::with_capacity(edges.len()); // m×d — THE temporary @@ -72,19 +85,30 @@ database person recognizes the shape instantly: this is a join ### Step 3 — the fused path: message_and_aggregate is one SpMM +> **In:** the `m×d` temporary from Step 2 and a CSR/`SparseTensor` adjacency. +> **Out:** a single fused SpMM that produces the same aggregate with zero +> temporaries. Step 4 is the layer that *cannot* take this path. + When the message is simple enough — a copy or a scalar multiple of the source row — the gather/message/scatter triple collapses into a single **SpMM** (sparse-times-dense matrix multiply: the adjacency in CSR against the n × d feature matrix), streaming messages into their destinations with zero temporaries. PyG's hook for this is -`message_and_aggregate`: if a layer defines it, `propagate` skips the -COO path entirely (the fuse check at :469-470). What the big three -put there: +`message_and_aggregate`. The dispatch is two-part in the pinned version +(message_passing.py): a subclass sets `self.fuse` true only if it *overrides* +`message_and_aggregate` (checked once in `__init__` at :154 via +`inspector.implements`), and even then `propagate` fuses only when the +`edge_index` handed in is sparse — `fuse = False` at :469, flipped to true +at :471-472 (`if is_sparse(edge_index)`) inside the `if self.fuse` guard at +:470. So the same GCNConv fuses on a `SparseTensor` and falls back to the +COO gather/scatter on a plain `edge_index`. When it does fuse, `propagate` +calls `message_and_aggregate` (:489) and skips `message`/`aggregate` +entirely. What the big three put there: | layer | message_and_aggregate | anchor | |---|---|---| -| GCNConv | `spmm(adj_t, x, reduce=sum)` | gcn_conv.py:270-274 | -| SAGEConv | `spmm(adj_t, x[0], reduce=mean)` | sage_conv.py:146-152 | +| GCNConv | `spmm(adj_t, x, reduce=self.aggr)` (`aggr='add'`) | gcn_conv.py:273-274 | +| SAGEConv | `spmm(adj_t, x[0], reduce=self.aggr)` (`aggr='mean'`) | sage_conv.py:149-152 | | GATConv | — (can't fuse: per-edge softmax weights) | gat_conv.py:392-408 | GCN and SAGE are literally one SpMM per layer. PyG docs call @@ -96,6 +120,10 @@ a shim over torch.sparse CSR, torch_sparse, or EdgeIndex backends. ### Step 4 — SDDMM: the second primitive, forced by attention +> **In:** GAT's feature-dependent edge weights, which block Step 3's fusion. +> **Out:** the second kernel — SDDMM — and the finding that SpMM + SDDMM span +> the whole GNN inventory. Step 5 is what the generality costs. + GAT breaks the fusion because its edge weights depend on the current features — a per-edge score plus a per-row softmax must run *before* the SpMM can. The kernel that computes those scores is **SDDMM** @@ -111,6 +139,11 @@ pattern from topic 24's triangle counting). ### Step 5 — what PyG pays for generality +> **In:** the `MessagePassing` abstraction from Steps 1–4. +> **Out:** the bill for its flexibility — an unfusable Python callable, a +> template-generated hot path — plus the two supporting pieces (walker, +> loader) that round out the map. + The abstraction's flexibility has a bill, and it reads like Ligra's (topic 24): @@ -126,10 +159,10 @@ The abstraction's flexibility has a bill, and it reads like Ligra's Beyond the layers, two more pieces round out the map: `node2vec.py` implements walks as a custom C++/CUDA op (:64) with the SGNS loss in -plain PyTorch (:101-160) — the walker, not the learner, is the hot -path (reading-node2vec.md); and `neighbor_loader.py:10` is GraphSAGE's -sampling industrialized into a minibatch loader -(reading-graphsage.md). +plain PyTorch (`loss` at :135, sampling helpers :101-129) — the walker, +not the learner, is the hot path (reading-node2vec.md); and +`neighbor_loader.py:10` is GraphSAGE's sampling industrialized into a +minibatch loader (reading-graphsage.md). ## Where each step lives in the code @@ -140,12 +173,12 @@ Read in this order — the table is the 90-minute route: | 1 | `torch_geometric/nn/conv/message_passing.py:39` | the base class — every conv layer subclasses this | 1 | | 2 | `:421` `propagate()` | the dispatcher: fused path check at :469-470 (`if self.fuse`) | 1, 3 | | 3 | `:565/:577/:598/:609` | the four overridables: `message`, `aggregate`, `message_and_aggregate`, `update` | 1 | -| 4 | `nn/conv/gcn_conv.py:45-71` | `gcn_norm` — A_hat construction (our stub's reference) | 3 | -| 5 | `gcn_conv.py:270-274` | GCN's two personalities: per-edge `message` (COO gather-scatter) vs fused `spmm(adj_t, x)` | 2, 3 | -| 6 | `nn/conv/sage_conv.py:146-152` | SAGE: same fusion, `reduce=mean` | 3 | +| 4 | `nn/conv/gcn_conv.py:45-72` | `gcn_norm` — A_hat construction (our stub's reference) | 3 | +| 5 | `gcn_conv.py:270-274` | GCN's two personalities: per-edge `message` (:270, COO gather-scatter) vs fused `spmm(adj_t, x)` (:273-274) | 2, 3 | +| 6 | `nn/conv/sage_conv.py:146-152` | SAGE: same fusion, `reduce=mean` (`message_and_aggregate` :149-152) | 3 | | 7 | `nn/conv/gat_conv.py:392-408` | GAT: why fusion is impossible (per-edge softmax) | 4 | | 8 | `utils/_spmm.py:12` | the `spmm` shim — dispatches to torch.sparse CSR, torch_sparse, or EdgeIndex backends | 3 | -| 9 | `nn/models/node2vec.py:64,101-160` | walks as a custom op + SGNS loss | 5 | +| 9 | `nn/models/node2vec.py:64,101-159` | walks as a custom op (:64) + SGNS loss (:135) | 5 | | 10 | `loader/neighbor_loader.py:10` | minibatch sampling (GraphSAGE industrialized) | 5 | Navigation advice: stops 1–3 are the skeleton — don't leave them @@ -171,13 +204,85 @@ are the supporting cast. ## Done when +Answer each before unfolding it. + - [ ] You can name the three overridable functions and trace one `GCNConv.forward`. + +
Answer + + `message` (per-edge, message_passing.py:565), `aggregate` (order-insensitive + reduce, :577), `update` (per-vertex, :609) — plus the fused + `message_and_aggregate` (:598). `GCNConv.forward` runs `gcn_norm` + (gcn_conv.py:45) to build the normalized adjacency, calls `propagate` + (:263), which — on a `SparseTensor` — takes the fused branch and calls + `message_and_aggregate` → `spmm(adj_t, x, reduce='add')` (gcn_conv.py:274); + the bias adds afterward in `forward`. The cached `adj_t` is a materialized + view (question 1). + +
+ - [ ] You can compute the memory footprint of the COO path's m×d temporary against a CSR SpMM on this topic's graph. + +
Answer + + The COO path materializes one message per edge: `m × d` floats. On the SBM + (m=566,564, d=64, 4 bytes) that is `566564·64·4 ≈ 145 MB` per layer. The + fused CSR SpMM writes only the `n × d` output (`16384·64·4 ≈ 4 MB`) and + streams edges without a per-edge temporary — the m×d allocation drops to + zero. Redo it for RMAT scale-16 at d=128 in notes.md. + +
+ - [ ] You can explain what `message_and_aggregate` fuses and why that is exactly an SpMM. + +
Answer + + It fuses the gather + `message` + scatter-reduce into one pass: for a copy + or scalar-multiple message, aggregating source rows into destinations + weighted by the adjacency *is* `A · X` — a sparse-times-dense multiply. + PyG writes it literally as `spmm(adj_t, x, reduce=...)` (gcn_conv.py:274, + sage_conv.py:152), so no `m × d` messages are ever built. + +
+ - [ ] You can explain why attention forces SDDMM as a second primitive. + +
Answer + + GAT's edge weight depends on both endpoints' current features, so a per-edge + score must be computed *before* the SpMM can run — and computing a dense + function over row pairs only at the adjacency's nonzeros is exactly SDDMM + (sampled dense-dense matmul, the mask does the sampling). SpMM alone can't + express it, so the inventory needs both; SpMM + SDDMM then span every + mainstream GNN. + +
+ - [ ] You can say why `reduce='max'` is not a semiring on floats-with-gradients. + +
Answer + + `max` has no additive inverse and its "sum" (max) is not differentiable + where two inputs tie — backward has to route the gradient to a single + argmax, which is a subgradient choice, not a ring operation. So the clean + (⊕,⊗)-semiring story that makes SpMM composable breaks for max-pooling + aggregation, which constrains any "GNN over GraphBLAS" plan (M20's semiring + menu) to the sum/mean reductions. + +
+ - [ ] You wrote answers to all five questions in notes.md, including which single kernel you would expose to Cypher. +
Answer + + All five `## Questions` answered in notes.md — the GCNConv.forward trace and + `self._cached_adj_t` (a materialized view), the COO-vs-CSR memory + computation, the `reduce='max'` gradient break, NeighborLoader's working set + vs a buffer pool, and which single kernel (SpMM) is the right Cypher surface + with SDDMM/softmax staying engine-internal. + +
+ ## References **Code** diff --git a/topics/25-graph-ml/reading-transe.md b/topics/25-graph-ml/reading-transe.md index 5d0c0eb..9ca104f 100644 --- a/topics/25-graph-ml/reading-transe.md +++ b/topics/25-graph-ml/reading-transe.md @@ -20,6 +20,10 @@ Freebase-scale data: millions of entities, a few thousand relations). ### Step 1 — the knowledge graph: facts as typed triples +> **In:** a store of known facts — typed edges. +> **Out:** the link-prediction task: rank candidate tails for a query (h, r, +> ?). Step 2 is the model that scores them. + A knowledge graph (KG) stores facts as **triples** (h, r, t) — "head entity, relation, tail entity": (Alice, works_at, Acme), (Acme, based_in, Berlin). It's a graph whose edges carry types, which means @@ -33,6 +37,10 @@ in Freebase lack a birthplace fact), so completion is the workload. ### Step 2 — the model: relations are translations in vector space +> **In:** the triples from Step 1. +> **Out:** one `d`-vector per entity and per relation, with a scalar +> `score(h,r,t)` per candidate fact. Step 3 is how those vectors are trained. + Embed every entity AND every relation as a point in R^d, and demand that a true fact line up as vector addition — head plus relation lands near tail: @@ -47,6 +55,14 @@ lands near tail: z_Bob ●────z_works_at────▶● z_BobCorp shared by all its edges ``` +This is exactly the paper's dissimilarity `d(h + ℓ, t)`, "which we take +to be either the L1 or the L2-norm" (Bordes et al. §2). Work one score by +hand in R²: let `z_Alice = (0,0)`, `z_works_at = (1,0)`, `z_Acme = +(0.9, 0.1)`. Then `z_h + z_r − z_t = (0.1, −0.1)` and the L2 score is +`√(0.1² + 0.1²) = √0.02 ≈ 0.14` — small, so the model believes it. A +corrupted tail `z_Berlin = (−1, 2)` scores `‖(1,0) − (−1,2)‖ = ‖(2,−2)‖ = +2.83` — far, so it does not. + The one arrow per relation is the model's entire capacity: every works_at edge in the graph must be (approximately) the *same* displacement vector. That's an aggressive compression — a relation @@ -57,21 +73,35 @@ model believes this fact". ### Step 3 — training: push true triples together, corrupted ones apart +> **In:** the entity and relation vectors from Step 2. +> **Out:** trained vectors, via a margin ranking loss over true-vs-corrupted +> triples. Step 4 reads off what this model structurally cannot learn. + Distances only mean something relative to alternatives, so TransE trains with a **margin ranking loss**: for each true triple, make a deliberately-broken one — a **corrupted triple**, the true triple with head OR tail swapped for a random entity — and require the true score to beat the corrupted score by a margin γ: -`max(0, γ + score(h,r,t) − score(h',r,t'))`. Plus the detail everyone -forgets: entity embeddings are re-normalized to the unit ball every -batch — otherwise the loss is trivially minimized by inflating all -norms (make every vector huge and every margin is satisfied without -learning anything). The whole training step: +`[γ + score(h,r,t) − score(h',r,t')]_+` (Bordes et al. eq. 1, where +`[x]_+` is the positive part and γ > 0). Plus the detail everyone +forgets: entity embeddings are re-normalized to **unit L2 norm** +(‖z‖ = 1) at the start of every batch — Algorithm 1 line 5, +`e ← e/‖e‖ for each entity e` — otherwise the loss is trivially minimized +by inflating all norms (make every vector huge and every margin is +satisfied without learning anything). Note the asymmetry: the paper +normalizes *entities* every batch but the *relation* vectors only once at +init (Algorithm 1 line 2), which is why `train_step` below renormalizes +`ent` and leaves `rel` alone. The whole training step, as the paper's +Algorithm 1 spells it (this repo has no TransE lane — the experiments +crate implements node2vec/SGNS, GCN and SpMM only, so this is pseudocode, +not a quote): -```rust +```text +// PSEUDOCODE — transcribed from Bordes et al. Algorithm 1; no repo/pinned +// source implements TransE, so there is no file:line to anchor here. fn train_step(ent: &mut Mat, rel: &Mat, (h, r, t): Triple, gamma: f32, lr: f32, rng: &mut Rng) { - ent.renormalize_unit_ball(); // the detail everyone forgets + ent.renormalize_unit_norm(); // Algorithm 1 line 5 (entities only) let (hc, tc) = corrupt(h, t, rng); // swap head OR tail, random entity let pos = l2(ent.row(h) + rel.row(r) - ent.row(t)); let neg = l2(ent.row(hc) + rel.row(r) - ent.row(tc)); @@ -89,6 +119,10 @@ cardinality statistics. ### Step 4 — the failure modes: what one arrow per relation can't say +> **In:** the single-translation-per-relation model from Step 2. +> **Out:** its relation algebra — which relation shapes it can and cannot +> represent. Step 5 turns the trained vectors into a query. + The compression of Step 2 has a relation algebra, and knowing it is knowing when to use the model: @@ -107,6 +141,11 @@ knowing when to use the model: ### Step 5 — serving is a nearest-neighbor query: why a database cares +> **In:** the trained entity/relation vectors and a query (h, r, ?). +> **Out:** the missing tail as `argmin_t ‖z_h + z_r − z_t‖` — a +> nearest-neighbour query over the entity index, filtered to exclude known +> tails. + Here is why this topic includes a 2013 ML paper: the *serving* path lands squarely on database machinery. "Predict the missing tail" = argmin over all entities t of ‖z_h + z_r − z_t‖ = a nearest-neighbor @@ -156,13 +195,83 @@ problem wearing KG clothes (question 3). ## Done when +Answer each before unfolding it. + - [ ] You can state the model in one equation and say what it assumes about relations. + +
Answer + + `score(h,r,t) = ‖z_h + z_r − z_t‖` (L1 or L2), trained so true triples + score low. The assumption is that every instance of a relation is the + *same* translation vector `z_r` — one arrow per relation type, shared by + all its edges. That is a strong compression: it forces `z_h + z_r ≈ z_t` + to hold simultaneously for every head/tail pair the relation connects. + +
+ - [ ] You can prove the symmetric-relation collapse. + +
Answer + + If r is symmetric, `(h,r,t)` and `(t,r,h)` are both true, so the model + wants `z_h + z_r ≈ z_t` and `z_t + z_r ≈ z_h`. Adding the two gives + `2 z_r ≈ 0`, i.e. `z_r ≈ 0`, so a symmetric relation degenerates to the + zero translation — `married_to` becomes "same embedding" and cannot be + distinguished from identity. Translation simply cannot express symmetry. + +
+ - [ ] You can name the failure modes one arrow per relation cannot express. + +
Answer + + 1-to-N relations (`works_at` maps many heads to one tail) collapse all + those heads toward `z_t − z_r`; symmetric relations force `z_r ≈ 0`; + reflexive/N-to-N relations are similarly crushed. What it *can* do is + composition — `z_born_in + z_city_of ≈ z_born_in_country` — because + translations add. TransH/TransR (per-relation projections) and RotatE + (rotation instead of translation) exist to lift these limits. + +
+ - [ ] You can explain why serving is a nearest-neighbour query and what filter the vector index needs. + +
Answer + + Predicting the tail is `argmin_t ‖z_h + z_r − z_t‖` — a nearest-neighbour + search for the query point `z_h + z_r` in the entity embedding index, which + an HNSW index (topic 14) answers in milliseconds over millions of entities. + The filter is the "filtered ranking" protocol: exclude tails already known + true for (h, r), so it becomes a filtered ANN query — topic 14's + filtered-search problem in KG clothing. + +
+ - [ ] You can say what degenerates when TransE is applied to an untyped single-relation graph like this topic's SBM. + +
Answer + + With one relation type, there is a single translation `z_r` shared by every + edge, so the model reduces to "`z_h + z_r ≈ z_t` for all edges" — which for + an undirected/symmetric graph drives `z_r ≈ 0` (the collapse above), leaving + only `z_h ≈ z_t` across edges: a plain proximity embedding with none of the + relational structure TransE was built for. That is exactly when node2vec's + untyped proximity objective is the better tool, and when typed KG + embeddings start to pay off. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + All five `## Questions` answered in notes.md — the symmetric-collapse proof, + the false-negative sampler fix via cardinality statistics (topic 9), the + filtered-ANN interaction with HNSW, the single-relation degeneracy, and + where per-relation vectors live under `CALL algo.transe(...)`. + +
+ ## References **Papers** diff --git a/topics/26-probabilistic/README.md b/topics/26-probabilistic/README.md index 8a6d973..01d67c7 100644 --- a/topics/26-probabilistic/README.md +++ b/topics/26-probabilistic/README.md @@ -8,6 +8,13 @@ production math, not exotica. ## Our motivation numbers first (Apple M3 Pro, 10M sorted u64, 2026-07-10) +> An earlier run than [FINDINGS.md](../../FINDINGS.md) row 26, which +> measures the same three lanes at **246 ns** (binary search), **299 ns** +> (BTreeMap) and **28 ns** (HashSet). Where the two disagree, FINDINGS is +> canonical; cite one run or the other by name and never average them. +> Re-run `./verify.sh 26` before treating a nanosecond figure below as +> current — the ratios, which are what the topic is about, hold in both. + | point-miss lookup | ns | memory | |---|---|---| | binary search over sorted vec | 167 | 76 MB (the data) | @@ -43,8 +50,9 @@ predictable. ``` k probes, b bits/key: FPR ≈ (1 − e^(−k/b))^k optimal k = b·ln2 → at 10 bits/key: k≈7, FPR ≈ 0.82% - rule of thumb: every +4.8 bits/key HALVES... no — ×10 needs +4.8 bits? - memorize instead: 10 bits/key ≈ 1%, 16 ≈ 0.04%, each bit/key is ~2× FPR + rule of thumb: +1.44 bits/key HALVES the FPR; +4.79 bits/key cuts it 10x + (log10(1/FPR) = b·ln2²/ln10 = 0.209·b, so one bit/key is ~1.6x) + spot-check: 10 bits/key ≈ 0.82%, 16 ≈ 0.046% ``` Blocked bloom (RocksDB `FastLocalBloomImpl`, util/bloom_impl.h:144) puts diff --git a/topics/26-probabilistic/experiments/.gitignore b/topics/26-probabilistic/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/26-probabilistic/experiments/.gitignore +++ b/topics/26-probabilistic/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/26-probabilistic/notes.md b/topics/26-probabilistic/notes.md index 515526c..508ea63 100644 --- a/topics/26-probabilistic/notes.md +++ b/topics/26-probabilistic/notes.md @@ -5,6 +5,10 @@ Machine: Apple M3 Pro, macOS. `cargo run --release --bin filter_bench` ## Measured baselines (provided lanes) +> An earlier run than [FINDINGS.md](../../FINDINGS.md) row 26 (**246 / +> 299 / 28 ns** for the three lanes below). FINDINGS is canonical where +> they disagree; cite one run by name and never average them. + | lane | ns/lookup | note | |---|---|---| | binary search (miss) | 167 | ~23 dependent cache misses over 76 MB | @@ -49,7 +53,7 @@ they're the record.) - [ ] HLL Q2: show sigma() term ⇒ linear counting for n ≪ m. - [ ] HLL Q4: ZERO/XZERO/VAL vs roaring containers — the density metric each switches on. - [ ] Learned Q1: 4 points where the cone splits but optimal PLA doesn't. -- [ ] Learned Q4: ALEX under adversarial (clustered) inserts — predict, then paper §5.5. +- [ ] Learned Q4: ALEX under adversarial (clustered) inserts — predict, then paper §6.2.6. - [ ] Roaring Q1: workload where per-chunk adaptivity beats per-matrix (topic 20). - [ ] Postgres Q3: BRIN pruning condition; place timestamp / UUIDv4 / monotone ID. diff --git a/topics/26-probabilistic/reading-bloom-to-ribbon.md b/topics/26-probabilistic/reading-bloom-to-ribbon.md index 184312b..cef71fe 100644 --- a/topics/26-probabilistic/reading-bloom-to-ribbon.md +++ b/topics/26-probabilistic/reading-bloom-to-ribbon.md @@ -9,19 +9,31 @@ one-sided answer buys, the bloom math you must own, the two sins, and the two very different fixes — then hands you the file anchors to watch each one in production code. +Every code anchor below is RocksDB at commit `7c80a5a`, the revision this +repo pins (`util/bloom_impl.h` is 489 lines; `util/ribbon_impl.h` is 1137 +lines), quoted with the line numbers the code occupies in that version. +The bloom-math figures are worked out on the spot from the formulas the +code uses. + ## The problem in one sentence Answering "is key X in this set?" exactly for 10M u64 keys costs a -HashSet — **224 MB** at 24 ns/lookup on the motivation bench — while a -structure allowed to be wrong 1% of the time, in one direction only, does -it in **12 MB** at the same speed; the fifty-year question is how close to -the information-theoretic minimum that 12 MB can get without paying extra -cache misses. +HashSet — **224 MB** at **28 ns/lookup** ([FINDINGS.md](../../FINDINGS.md) +row 26) — while a structure allowed to be wrong 1% of the time, in one +direction only, does it in **12 MB** (10M keys × 10 bits) at roughly the +same speed; the fifty-year question is how close to the +information-theoretic minimum that 12 MB can get without paying extra cache +misses. ## The concepts, step by step ### Step 1 — the filter contract: one-sided error +> **In:** nothing yet — this step fixes the contract every later step +> leans on. +> **Out:** the one-sided guarantee, and the going exchange rate (~10 bits +> per key ≈ 1% FPR) that Steps 2–3 derive and Steps 4–6 defend. + A filter is a compact set-membership structure that may answer "maybe present" for a key that is absent, but must never answer "absent" for a key that is present. The rate of the first mistake is the **false positive @@ -36,6 +48,10 @@ merely slower, never wrong. ### Step 2 — the bloom filter: k shared bits per key +> **In:** the one-sided contract from Step 1. +> **Out:** the false-positive formula and the optimal probe count `k`, +> both of which Step 3 measures against the theoretical floor. + Bloom's 1970 design is an m-bit array plus k hash functions: to insert a key, set the k bits its hashes pick; to query, check them — all k set means "maybe", any zero means "definitely absent" (a present key's bits were all @@ -43,27 +59,53 @@ set at insert time and bits are never cleared, so no false negatives). The bits are *shared* between keys, which is where false positives come from — and the math is worth deriving once, not memorizing. -Derive (don't memorize) `FPR ≈ (1 − e^(−kn/m))^k`: -- One insert with one probe leaves a given bit 0 with prob (1 − 1/m). -- After kn probes: (1 − 1/m)^kn ≈ e^(−kn/m) — fraction of bits still 0. -- A miss query needs all k of its probe bits set: (1 − e^(−kn/m))^k. -- Minimize over k: optimal k = (m/n)·ln2 ≈ 0.69·bits_per_key. At 10 bpk → k≈7. - -Rules of thumb that fall out: 10 bits/key ≈ 1% FPR, 16 ≈ 0.04%, and each -added bit/key cuts FPR roughly in half. The cost baked into the design: -shared bits mean you can never delete (clearing a bit may lie about other -keys), and every query touches k scattered bits. +Derive (don't memorize) the false-positive rate. Name the symbols: **m** +is the number of bits in the array, **n** the number of keys inserted, +**k** the number of hash probes per key, so **b = m/n** is the *bits per +key* budget: + +- One insert with one probe leaves a given bit 0 with probability + (1 − 1/m). +- After all kn probes: (1 − 1/m)^kn ≈ e^(−kn/m) — the fraction of bits + still 0. +- A miss query reports "maybe" only if all k of its probe bits are set: + **FPR ≈ (1 − e^(−k/b))^k** (using kn/m = k/b). This is exactly RocksDB's + `BloomMath::StandardFpRate` (`util/bloom_impl.h:35`, + `pow(1.0 - exp(-num_probes / bits_per_key), num_probes)`). +- Minimize over k: **optimal k = (m/n)·ln2 = b·ln2**. + +Worked once at **b = 10 bits/key**: optimal k = 10 × 0.6931 = **6.93 ≈ 7** +probes. Plug back in: k/b = 7/10 = 0.7, e^(−0.7) = 0.4966, so +FPR = (1 − 0.4966)^7 = 0.5034^7 = **0.0082 = 0.82%**. At b = 16 and its +optimal k = 11, the same formula gives (1 − e^(−11/16))^11 = **0.046%**. +Those are the two rules of thumb — **10 bits/key ≈ 0.8% FPR, 16 ≈ 0.05%**, +each extra bit/key cutting FPR by roughly half. The cost baked into the +design: shared bits mean you can never delete (clearing a bit may lie +about other keys), and every query touches k scattered bits. ### Step 3 — the two sins: 1.44× space and k cache misses -Measured against the theoretical floor, bloom wastes space: storing a set -with FPR f needs at least log2(1/f) bits per key (the information-theoretic -lower bound), and bloom needs 1.44·log2(1/f) — **44% overhead**, forever, -by construction. And it wastes time: the k probe bits land in k random -words of a large array, so a query costs up to **k cache misses** (~7 at -10 bpk) — on a machine where one miss is ~80–100 ns, the filter meant to -*save* a probe costs seven. Fifty years of fixes attack exactly those two -sins: +> **In:** the working bloom filter and its FPR formula from Step 2. +> **Out:** the two named defects — a 44% space tax and up to k cache +> misses — that Step 4 attacks one of and Steps 5–6 the other. + +Measured against the theoretical floor, bloom wastes space. Storing a set +so that a non-member reports "maybe" with probability f needs at least +**log₂(1/f)** bits per key — the **information-theoretic lower bound**, the +fewest bits any approximate-membership structure can use for that FPR. +Bloom instead spends **(1/ln2)·log₂(1/f) = 1.4427·log₂(1/f)** bits — **44% +overhead**, forever, by construction. Worked at the Step 2 operating point +(b = 10 bits/key, f = 0.82%): the floor is log₂(1/0.0082) = log₂(122) = +**6.93 bits/key**, and 10 / 6.93 = **1.44** — the 44% is exactly the factor +1/ln2. The Ribbon paper opens on this same number: Bloom uses "at least +44% more space than the information-theoretic" minimum +([arXiv:2103.02515](https://arxiv.org/abs/2103.02515), §1). + +And bloom wastes time: the k probe bits land in k random words of a large +array, so a query costs up to **k cache misses** (7 at 10 bpk). Taking a +DRAM miss at an assumed ~80–100 ns, that is 7 × ~90 ns ≈ **630 ns** of +stall — a filter meant to *save* one SST probe costing the equivalent of +several. Fifty years of fixes attack exactly those two sins: ``` sin #1: k cache misses sin #2: 1.44x space @@ -78,13 +120,41 @@ for the space fix. Steps 4–6 walk them in turn. ### Step 4 — blocked bloom: all k probes in one cache line +> **In:** sin #2 from Step 3 — the k scattered cache misses. +> **Out:** a query that touches exactly one cache line, and the FPR tax +> it pays for that (a currency Step 5's fix does *not* spend). + A blocked bloom filter first hashes the key to one cache-line-sized -**block** (512 bits in RocksDB's `FastLocalBloomImpl`), then runs a -miniature bloom filter entirely inside that block — so a query costs -exactly **one** memory access instead of k. The entire query path, -de-SIMD'd (this is `HashMayMatchPrepared`): +**block** (512 bits in RocksDB's `FastLocalBloomImpl`, `util/bloom_impl.h:144`), +then runs a miniature bloom filter entirely inside that block — so a query +costs exactly **one** memory access instead of k. The real probe loop is +nine lines; this is the *insert* side, `AddHashPrepared`, and the query +side at :231 is the same loop with an early-exit `return false`: + +```c +// util/bloom_impl.h — FastLocalBloomImpl::AddHashPrepared, 206-214 + 206 static inline void AddHashPrepared(uint32_t h2, int num_probes, + 207 char* data_at_cache_line) { + 208 uint32_t h = h2; + 209 for (int i = 0; i < num_probes; ++i, h *= uint32_t{0x9e3779b9}) { + 210 // 9-bit address within 512 bit cache line + 211 int bitpos = h >> (32 - 9); + 212 data_at_cache_line[bitpos >> 3] |= (uint8_t{1} << (bitpos & 7)); + 213 } + 214 } +``` + +Line 211 is the one to watch: `h >> (32 - 9)` keeps the **top 9 bits** of +the 32-bit hash — a value 0..511, one bit in the 512-bit line — and line +209 re-multiplies `h` by the golden-ratio constant `0x9e3779b9` between +probes so each probe reads different bits. The block itself was already +chosen by `h1` in `AddHash` (:200-204, `FastRange32(h1, len_bytes >> 6)`). +The same six-probe loop, de-SIMD'd into one function so the control flow +is visible at a glance: ```rust +// ILLUSTRATION — not quoted from RocksDB; the query path is the AVX2 loop +// in util/bloom_impl.h:231 (HashMayMatchPrepared), block choice at :225-228. const PROBES: u32 = 6; fn may_contain(bits: &[u64], num_blocks: u32, h1: u32, mut h2: u32) -> bool { @@ -100,24 +170,45 @@ fn may_contain(bits: &[u64], num_blocks: u32, h1: u32, mut h2: u32) -> bool { } ``` -The price is **Poisson crowding**: keys per block follow a Poisson -distribution (the statistics of throwing n balls into n/512-bit bins), so -some blocks get twice the average load — and a block that got 2× the keys -has much worse FPR than the formula in Step 2 predicts. RocksDB is honest -about it: `CacheLocalFpRate` (bloom_impl.h:42) computes the real blocked -FPR as the *expectation over the Poisson distribution of keys-per-block* — -the whole blocked-bloom trade in 10 lines, and worse than the naive -`StandardFpRate` at the same bits/key. Measured: **~1.5–2× the standard -FPR** at the same bpk, in exchange for k× fewer misses. That ratio is -exactly what our stub's `fpr < 4× theory` test bounds. +The price is **Poisson crowding**: keys land in blocks independently, so +the count per block follows a Poisson distribution (the statistics of +throwing n balls into n/512 bins), and a block that happens to hold twice +the average load has a much worse local FPR than Step 2's formula predicts. +RocksDB prices it in `CacheLocalFpRate` (`util/bloom_impl.h:42`), and the +model is cruder than "sum the Poisson distribution": it takes the **average +of two `StandardFpRate` values**, one at one standard deviation *above* the +mean block occupancy (the crowded case, :53-54) and one *below* (the +uncrowded case, :55-56), where the mean is `keys_per_cache_line = +cache_line_bits / bits_per_key` (:48) and the spread is `sqrt` of that +(:52). + +Worked at b = 10, the stub's 6 probes, a 512-bit line: mean = +512/10 = 51.2 keys, stddev = √51.2 = 7.16, so the crowded arm evaluates +`StandardFpRate(512/58.36, 6)` = `StandardFpRate(8.77, 6)` = **1.48%** and +the uncrowded arm `StandardFpRate(512/44.04, 6)` = `StandardFpRate(11.62, 6)` += **0.43%**; their average is **0.95%**, versus the un-blocked **0.84%** — +a **1.13× tax** for the k× fewer misses. The tax is small *because the +block is a whole cache line*; it grows fast as the block shrinks — the same +formula gives 1.32× at 256-bit blocks, 1.64× at 128-bit, and 2.27× at +64-bit. The "1.5–2×" figure that folklore attaches to blocked bloom is a +small-block number; at a 512-bit line it is closer to 1.1–1.2×. (This +ratio is unmeasured in our repo — the stub is not yet implemented; notes.md +*predicts* 1.2–1.8×, and the `fpr < 4× theory` test only bounds it below +2.5% at 10 bpk.) ### Step 5 — filters as linear algebra: solve for bits, don't set them +> **In:** sin #1 from Step 3 — the 44% space overhead. (This step and +> Step 4 fork off Step 3: Step 4 spent FPR to fix the misses; this one +> spends updatability to fix the space.) +> **Out:** the "solve for bits" reframe, the exact 2^−r false-positive +> rate, and the space law r·(1+overhead) that Step 6 makes cheap to build. + The conceptual jump behind the space fix: a bloom filter *sets* bits; a xor/ribbon filter *solves for* bits. Give every key an r-bit **fingerprint** (a short hash of the key), and find an array S of r-bit -slots such that each key's equation holds over GF(2) (arithmetic on bits -where addition is XOR): +slots such that each key's equation holds over **GF(2)** (arithmetic on +single bits, where addition is XOR and multiplication is AND): ``` row(key) · S = fingerprint(key) ← S is the filter, r fingerprint bits @@ -126,34 +217,56 @@ where addition is XOR): `row(key)` is a hash-derived coefficient vector saying which slots of S to XOR together. Query = recompute `row·S`, compare against the key's fingerprint. For inserted keys the equation holds by construction (no false -negatives); a false positive is a non-key whose equation *happens* to hold — -probability exactly 2^−r. So space ≈ r·(1+overhead) bits/key, where -overhead is the fraction of unusable slots the solver needs — **~10% for -ribbon vs bloom's 44%**. The catch: you must solve a linear system over -all keys at once, which is why this family is *static* — build once, never -insert again. +negatives); a false positive is a non-key whose equation *happens* to hold, +with probability exactly **2^−r**. + +Worked at r = 12: FP = 2^−12 = 1/4096 = **0.024%**. Space is +**r·(1+overhead)** bits/key, where *overhead* is the fraction of extra +slots the solver needs beyond one per key. Line the three up at that same +0.024% target, whose information floor is log₂(1/f) = r = 12 bits/key: +bloom spends 1.44 × 12 = **17.3 bits/key**, an xor filter (Step 5's family, +overhead 23%) spends 1.23 × 12 = **14.8**, and a ribbon filter (overhead +~10%) spends ~1.10 × 12 = **13.2** — about **24% less than bloom** for the +identical FPR. The catch: you must solve a linear system over all keys at +once, which is why this family is *static* — build once, never insert +again. ### Step 6 — the ribbon band: locality makes the solve O(n), and builds can fail -The "ribbon" trick makes the linear solve cheap enough for production: -`StandardHasher` (ribbon_impl.h:165) gives each key a coefficient vector -that is nonzero only in a `kCoeffBits`-wide (:114, = 64 or 128) *band* -starting at a hashed position. A system where every row's nonzeros sit in -a narrow diagonal band admits **banded Gaussian elimination** — O(n) with -tiny constants — and `StandardBanding` (:471, `num_starts_ = num_slots - -kCoeffBits + 1` at :504) does it *incrementally*, back-substituting as -keys stream in (`BandingAddRange` :577). Streaming build is ribbon's edge -over xor filters, which need all keys up front. - -Two costs to hold onto. First, construction can *fail* (the random system -comes out singular), and RocksDB retries with a different hash seed -(`StandardRehasherAdapter` :416) — unlike blocked bloom, whose monotone -"set bits" build can never fail. Second, both build and query burn more -CPU than bloom's bit probes. RocksDB's deployment follows directly: +> **In:** the "solve for bits" system from Step 5. +> **Out:** the banding trick that makes the solve O(n) and incremental, +> the failure mode it introduces, and the LSM-level deployment that trades +> the two filters off against each other. + +The name is the trick: **Ribbon** stands for "Rapid Incremental Boolean +Banding ON the fly" ([arXiv:2103.02515](https://arxiv.org/abs/2103.02515), +§1). `StandardHasher` (`util/ribbon_impl.h:165`) gives each key a +coefficient vector that is nonzero only in a `kCoeffBits`-wide *band* +starting at a hashed position — and `kCoeffBits` is just `sizeof(CoeffRow) +* 8`, i.e. **64 or 128** (:114-115). A system where every row's nonzeros +sit in a narrow diagonal band admits **banded Gaussian elimination** — +O(n) with tiny constants instead of the O(n³) of a dense solve. +`StandardBanding` (:471) runs it *incrementally*: as each key arrives its +banded row is reduced against the rows already placed and dropped into an +empty pivot slot (the on-the-fly insertion the paper describes at §4). The +number of band start positions is `num_starts_ = num_slots - kCoeffBits + 1` +(:504) — for, say, 1000 slots and 64-bit bands, 937 places a band can +begin. The actual back-substitution lives one file over, in +`BandingAddRange` (`util/ribbon_alg.h:611`), which `AddRange` +(`util/ribbon_impl.h:570-577`) calls. Streaming build is ribbon's edge over +xor filters, which need all keys up front. + +Two costs to hold onto. First, construction can *fail* — the random banded +system can come out singular — and RocksDB retries with a different hash +seed (`StandardRehasherAdapter` :416), unlike blocked bloom, whose monotone +"set bits" build can never fail. Second, both build and query burn more CPU +than bloom's bit probes. RocksDB's own deployment note quantifies the +trade: across large LSM deployments, blocked Bloom filters use "roughly 10% +of memory and roughly 1% of CPU" (paper §1, footnote 5) — so ribbon buys +back that ~10% memory at a CPU cost. The policy follows directly: **ribbon for the cold bottom LSM levels** (most keys live there — space -dominates) and **blocked bloom for the hot top levels** (queried -constantly — speed dominates), via `RibbonFilterPolicy`'s -`bloom_before_level`. +dominates) and **blocked bloom for the hot top levels** (queried constantly +— speed dominates), via `RibbonFilterPolicy`'s `bloom_before_level` knob. ## Where each step lives in the code @@ -164,19 +277,20 @@ docs — read code and comments together. | anchor | what it is | |---|---| -| `LegacyBloomImpl` (:364-476) | old format: one cache line per key (`AddHash` :432 picks `num_lines`), but probes derived by weak shift-rotate — measurable FPR bias | +| `LegacyLocalityBloomImpl` (:404) | old "one cache line per key" format: `AddHash` (:432) picks a line via `GetLine` (:406), but probes are derived by a weak shift-rotate (`delta = (h>>17)\|(h<<15)` at :473) — measurable FPR bias (the comment at :107 clocks it at 1.138% vs `FastLocalBloomImpl`'s 0.957% at the same setting) | | `FastLocalBloomImpl` (:144) | current "format_version=5" bloom: 512-bit (64-byte) blocks, probes from `h *= 0x9e3779b9` golden-ratio remix (Step 4) | -| `AddHashPrepared` (:206) | the probe loop: each probe uses bits (h >> 27) & 511 of a *re-multiplied* h — 9 bits per probe, all inside one line | -| `HashMayMatchPrepared` (:231) | query = same loop, early-exit on first zero bit — Step 4's code sample | -| `CacheLocalFpRate` (:42) | the honesty function: blocked-bloom FPR as the *expectation over the Poisson distribution of keys-per-block* (Step 4's tax, quantified) | +| `AddHashPrepared` (:206) | the probe loop: each probe takes `h >> (32 - 9)` — the top 9 bits of a *re-multiplied* h, one bit inside the line (Step 4's quoted block) | +| `HashMayMatchPrepared` (:231) | query = same loop, early-exit on the first zero bit (the AVX2 path; Step 4's illustration is its scalar shape) | +| `StandardFpRate` (:32) | the un-blocked bloom FPR, `pow(1 - exp(-k/b), k)` — Step 2's formula, verbatim | +| `CacheLocalFpRate` (:42) | the blocked-bloom tax: the **average of two `StandardFpRate` values**, at one std-dev above (:53-54) and below (:55-56) the mean keys-per-line — Step 4's ±1σ model, *not* a Poisson sum | -`util/ribbon_impl.h` — Steps 5–6: +`util/ribbon_impl.h` (+ `util/ribbon_alg.h`) — Steps 5–6: | anchor | what it is | |---|---| -| `StandardHasher` (:165) | coefficient vectors nonzero only in a `kCoeffBits`-wide band (:114) | -| `StandardBanding` (:471) | incremental banded elimination; `num_starts_` at :504 | -| `BandingAddRange` (:577) | streaming back-substitution as keys arrive | +| `StandardHasher` (:165) | coefficient vectors nonzero only in a `kCoeffBits`-wide band; `kCoeffBits = sizeof(CoeffRow)*8` = 64 or 128 (:114-115) | +| `StandardBanding` (:471) | incremental banded elimination; `num_starts_ = num_slots - kCoeffBits + 1` at :504 | +| `AddRange` (:570-577) → `BandingAddRange` (`ribbon_alg.h:611`) | on-the-fly back-substitution as keys arrive | | `StandardRehasherAdapter` (:416) | the build-failure retry with a fresh seed | ## Tie back to the stub @@ -209,12 +323,115 @@ for your keys-per-block Poisson mean. ## Done when +Answer each before unfolding it. + - [ ] You can state the filter contract: one-sided error, and which side. + +
Answer + + A filter may answer "maybe present" for a key that is absent (a false + positive) but must never answer "absent" for a key that is present (a + false negative is forbidden by construction, because a present key's bits + were all set at insert and bits are never cleared). The one-sidedness is + what a lookup path needs: "definitely absent" is trustworthy, so you skip + the expensive SST read with certainty; a false positive costs only one + wasted probe. At the going rate of ~10 bits per key the FPR is ~0.8% + (Step 2's `StandardFpRate(10, 7)`), so 99.2% of misses are caught for + 5% of a HashSet's memory. + +
+ - [ ] You can explain why exactly half the bits are set at optimal k, and why that is intuitive. + +
Answer + + The fraction of bits still 0 after all inserts is e^(−k/b) (Step 2). At + the optimal probe count k = b·ln2, that exponent is −(b·ln2)/b = −ln2, so + e^(−ln2) = **exactly 1/2** — half the bits are 0, half are 1. Worked at + b = 10, k = 7: e^(−0.7) = 0.497, essentially half. + + It is intuitive as an entropy argument: a single bit carries the most + information when it is 0 with probability 1/2 (maximal entropy, 1 bit). + Push more keys in and too many bits are 1, so every query's probes match + and the FPR climbs; use fewer probes and you waste array capacity. The + minimum FPR sits exactly where each bit is a fair coin. + +
+ - [ ] You can name the two sins — 1.44x space and k cache misses — and say which one blocked bloom fixes. + +
Answer + + Sin #1 is space: bloom spends 1.4427·log₂(1/f) bits per key against an + information floor of log₂(1/f) — a 44% overhead, the factor 1/ln2 (Step 3; + Ribbon paper §1). Sin #2 is time: the k probe bits are in k random words, + so a query costs up to k cache misses (7 at 10 bpk). + + Blocked bloom fixes **sin #2 only** — `FastLocalBloomImpl` + (`bloom_impl.h:144`) puts all probes in one 512-bit cache line, so a + query touches one line instead of k. It pays for that fix in FPR, not + space: `CacheLocalFpRate` (:42) prices the crowding tax at ~1.13× the + un-blocked rate for a 512-bit line. Sin #1 (space) is what the ribbon/xor + family attacks instead. + +
+ - [ ] You can explain filters as a linear solve, and why the ribbon band makes it O(n). + +
Answer + + A xor/ribbon filter *solves for* an array S of r-bit slots such that + `row(key)·S = fingerprint(key)` over GF(2) for every key, where `row(key)` + is a hash-derived coefficient vector. Query recomputes `row·S` and + compares; a non-key collides with probability exactly 2^−r. A dense + Gaussian solve over n such equations is O(n³). + + Ribbon makes `row(key)` nonzero only inside a `kCoeffBits`-wide (64 or + 128) diagonal band starting at a hashed position (`StandardHasher`, + `ribbon_impl.h:165`). Because every row's nonzeros are confined to that + band, banded Gaussian elimination reduces each new row against only the + O(kCoeffBits) rows it overlaps — O(n) total, done incrementally by + `StandardBanding` (:471), back-substituting in `BandingAddRange` + (`ribbon_alg.h:611`). + +
+ - [ ] You can say what happens when ribbon construction fails and what RocksDB does about it. -- [ ] You wrote answers to all five questions in notes.md, including why RocksDB picks ribbon for the bottom level — and you have this topic's measured miss costs to compare against: 246 ns binary search, 299 ns BTreeMap, 28 ns HashSet at 224 MB. + +
Answer + + The random banded system can come out singular — no assignment of S + satisfies all the equations. Construction then *fails*, and RocksDB + retries the whole build with a different hash seed via + `StandardRehasherAdapter` (`ribbon_impl.h:416`). This is the price of + "solve" over "set": blocked bloom's build is monotone (only ever sets + bits) and can never fail, whereas ribbon and cuckoo can, so both need a + retry/fallback path. It is a build-time cost, not a query-time one — once + a seed succeeds, queries are deterministic. + +
+ +- [ ] You wrote answers to all five questions in notes.md, including why RocksDB picks ribbon for the bottom level — and you have this topic's measured miss costs to compare against. + +
Answer + + RocksDB's split — ribbon on the bottom LSM levels, blocked bloom on the + hot top levels (`RibbonFilterPolicy`'s `bloom_before_level`) — follows + from ribbon being ~10% smaller than bloom but several× slower to build and + query. The bottom level holds the overwhelming majority of keys, so its + filters dominate memory, yet it is probed rarely; there, the ~10% space + win is worth the CPU. The top levels are tiny but queried constantly, so + blocked bloom's one-cache-line speed wins. The paper's own measurement + frames the stakes: blocked Bloom filters are "roughly 10% of memory and + roughly 1% of CPU" in large deployments (§1, footnote 5). + + Put the win beside this topic's baseline: a point miss costs **246 ns** + (binary search) or **299 ns** (BTreeMap), while a 224 MB HashSet does it + in **28 ns** ([FINDINGS.md](../../FINDINGS.md) row 26). A blocked bloom + aims for HashSet-class miss-skipping at ~12 MB (10M × 10 bits) — the + space that ribbon then shaves another ~24% off on the cold levels. + +
## References diff --git a/topics/26-probabilistic/reading-cuckoo-xor.md b/topics/26-probabilistic/reading-cuckoo-xor.md index e831192..a719cb0 100644 --- a/topics/26-probabilistic/reading-cuckoo-xor.md +++ b/topics/26-probabilistic/reading-cuckoo-xor.md @@ -10,82 +10,218 @@ can't delete, what a fingerprint is, the XOR involution that makes kicking possible, and the peeling construction that makes static filters smaller. +Every code anchor below is RedisBloom at commit `ab734fa`, the revision +this repo pins (`src/cuckoo.c` is 439 lines), quoted with the line +numbers the code occupies there. The formulas are Fan, Andersen, +Kaminsky & Mitzenmacher, *Cuckoo Filter* (CoNEXT 2014) and Graf & +Lemire, *Xor Filters* (ACM JEA 2020); every figure is worked on the spot +from the equation it cites. Two fingerprint sizes appear on purpose: +production `cuckoo.c` uses an **8-bit** fingerprint (`fp = hash % 255 + 1`, +:127), while this topic's stub (`experiments/src/cuckoo.rs`) uses a +**12-bit** one — the guide labels which is which every time it quotes a +number. + ## The problem in one sentence Delete one key from a bloom filter and you corrupt others — clearing any of its k shared bits can create a **false negative** (the filter says "absent" for a key that is present, breaking the one contract a filter has) for every key that shares those bits — yet caches, routing tables, -and any filter over churning data need membership *with* deletion. +and any filter over churning data need membership *with* deletion, and +they need it cheap: a point miss over 10M keys costs **246 ns** (binary +search) or **299 ns** (BTreeMap) while a 224 MB HashSet answers in +**28 ns** ([FINDINGS.md](../../FINDINGS.md) row 26) — that gap is what a +filter is bidding for. ## The concepts, step by step ### Step 1 — why bloom can't delete: the bits are shared +> **In:** nothing yet — this step fixes the failure every later step is +> built to avoid. +> **Out:** the requirement — *discrete, identifiable residence* per key — +> that Step 2 satisfies with fingerprints. + In a bloom filter, one bit typically serves many keys, so removing a key has no safe implementation. Concretely: insert A sets bits {3, 17, 40}; insert B sets bits {17, 52, 88}. Delete A by clearing {3, 17, 40} and B — still present — now fails its probe on bit 17: a false negative, the -forbidden error. (Counting blooms replace each bit with a counter, but -that multiplies space by 4–8× and still can't say *which* key a counter +forbidden error. (**Counting bloom filters** — bloom with a small counter +per bit instead of a single bit — support decrement-on-delete, but that +multiplies space by 4–8× and a counter still can't say *which* key it belongs to.) The fix requires keys to occupy *discrete, identifiable* residence — which is Step 2. ### Step 2 — fingerprints in buckets: membership as a tiny hash table +> **In:** the discrete-residence requirement from Step 1. +> **Out:** the fingerprint, the false-positive formula (Eq 5) and the +> minimal fingerprint size (Eq 6) — plus the two-candidate-bucket table +> whose alternate-bucket problem Step 3 has to solve. + Instead of smearing a key across shared bits, store one **fingerprint** -per key — a short hash of the key, e.g. 12 bits — as a discrete resident -in a slot of a hash-table bucket. Query = "does my fingerprint appear in -my bucket?"; delete = find it and zero the slot. A false positive is now -a fingerprint *collision*: some other key in the same bucket happens to -carry your 12 bits — probability ≈ `2 × slots × 2^−f` (two candidate -buckets, `slots` fingerprints compared in each, each matching with -2^−f) — at f=12 and 4 slots, ~0.2%. The open problem this creates: -hash-table buckets fill up, and a plain table stalls at ~50% occupancy. -**Cuckoo hashing** fixes occupancy by giving every key *two* candidate -buckets and, when both are full, evicting ("kicking") a resident to *its* -other bucket, recursively — that discipline pushes usable load to ~95% -with 4-slot buckets (paper Table 2: 1 slot tops out ~50%, 4 slots ~95%). +per key — a short hash of the key — as a discrete resident in a slot of a +hash-table **bucket** (a fixed group of slots; 4 in both `cuckoo.c` and +the stub). In production `cuckoo.c` the fingerprint is +`fp = hash % 255 + 1` (`getLookupParams`, :127): an 8-bit value in the +range 1..255, with 0 (`CUCKOO_NULLFP`) reserved to mean "empty slot". +Query = "is my fingerprint in either of my candidate buckets?" +(`Filter_Find` :146 scans both with `Bucket_Find` :137); delete = find it +and zero the slot (`Filter_Delete` :164 via `Bucket_Delete` :154). + +A **false positive** here is a fingerprint *collision*: some other key in +a candidate bucket happens to carry your fingerprint. The paper computes +its rate in Eq (5), line 771: + +``` + false-positive rate = 1 - (1 - 1/2^f)^(2b) ≈ 2b / 2^f +``` + +Name every symbol: **f** is the fingerprint length in bits; **b** is the +bucket size (slots per bucket). The exponent **2b** is the number of +comparisons a lookup makes — *two* candidate buckets × *b* slots each — +and each comparison hits your fingerprint with probability 1/2^f. Worked +on three concrete configurations (arithmetic verified): + +- **production** f=8, b=4: exact 1−(1−1/2^8)^8 = **3.08%**, approximation + 2b/2^f = 8/256 = **3.125%**. (`cuckoo.c`'s fingerprint has 255 values, + not 256, so its per-comparison rate is 1/255 and the figure is + 8/255 = **3.14%** — essentially the same.) +- **stub** f=12, b=4: exact 1−(1−1/2^12)^8 = **0.195%**, approximation + 8/4096 = **0.195%**. +- f=16, b=4: 8/65536 = **0.0122%**. + +Each extra fingerprint bit halves the FPR. The inverse question — how many +fingerprint bits does a target rate demand? — is Eq (6), line 775: + +``` + f ≥ ⌈log2(2b / ϵ)⌉ = ⌈log2(1/ϵ) + log2(2b)⌉ bits +``` + +where **ϵ** is the target false-positive rate. Worked (verified): + +- ϵ = 1/64 (≈1.56%), b=4: 2b/ϵ = 8·64 = 512, log₂512 = 9 → **f ≥ 9 bits**. +- ϵ = 0.2% (0.002), b=4: 8/0.002 = 4000, log₂4000 = 11.97 → + **f ≥ 12 bits** — which is exactly why the stub picks 12. +- ϵ = 3.125% (production's operating point), b=4: 8/0.03125 = 256, + log₂256 = 8 → **f ≥ 8 bits** — which is why 8 bits suffice for + `cuckoo.c`. + +The open problem this creates: hash-table buckets fill up, and a plain +table stalls at ~50% **occupancy** (the fraction of slots in use, also +called **load factor** α). **Cuckoo hashing** — give every key *two* +candidate buckets and, when both are full, evict ("kick") a resident to +*its* other bucket, recursively — pushes usable load far higher. Paper +Figure 2 / §4: with b=1 the load factor tops out at **50%** (line 731), +with b=4 it reaches **95%** (line 663), with b=8 **98%** (line 664). That +b in the denominator is why fingerprints stay short: the minimum size +grows only as f = Ω(log n / b) bits (line 641). ### Step 3 — the one trick that makes cuckoo *filters* possible +> **In:** the two-candidate-bucket table from Step 2 — where a kick must +> compute a victim's *other* bucket, but the victim's original key is gone. +> **Out:** the `getAltHash` involution, the single operation Step 4's +> kicking loop calls to move a fingerprint. + Cuckoo *hashing* moves keys between two candidate buckets — but a filter stores only fingerprints; after insertion the original key is gone, so how do you compute a victim's alternate bucket to kick it? -**Partial-key cuckoo hashing** (paper §3.1; `getAltHash`, cuckoo.c:122): +**Partial-key cuckoo hashing** (paper §3.1, Eq 1–2). The whole trick is +one line of `cuckoo.c`: -``` - i1 = hash(key) - i2 = i1 XOR hash(fingerprint) ← involution: i1 = i2 XOR hash(fp) +```c +// src/cuckoo.c — getAltHash, 122-124 + 122 static CuckooHash getAltHash(CuckooFingerprint fp, CuckooHash index) { + 123 return ((CuckooHash)(index ^ ((CuckooHash)fp * 0x5bd1e995))); + 124 } ``` -Because XOR is its own inverse, the alternate bucket is computable from -*(current bucket, fingerprint)* alone — apply the same XOR from either -side and you get the other. Two costs come with the trick: the bucket -count is forced to a power of two (XOR must stay in range — RedisBloom -asserts it at filter creation), and the two buckets aren't independent — -a fingerprint's candidate pair is determined by only -`log2(buckets) + fp_bits` bits, which caps how large the table can get -before FPR degrades (paper §4). +Line 123 carries the argument: the alternate bucket is +`index XOR (fp × 0x5bd1e995)`, where `0x5bd1e995` is the MurmurHash2 +mixing constant — a cheap multiplicative hash of the fingerprint. The +paper writes it as Eq (1) `h1(x) = hash(x)`, `h2(x) = h1(x) ⊕ hash(fp)` +and Eq (2) `j = i ⊕ hash(fp)`. Because XOR is its own inverse, applying +the same operation from either bucket returns the other — an +**involution** (a function that is its own inverse). So the alternate +bucket is computable from *(current bucket, fingerprint)* alone, and an +insertion "only uses information in the table, and never has to retrieve +the original item x" (§3.1). + +Worked with real numbers (numBuckets = 2^16, so an index is reduced mod +65536; for fp=200 the low 16 bits of `fp × 0x5bd1e995` are 31848): + +- i1 = 12345 → i2 = 12345 XOR 31848 = **19537**. +- from i2: 19537 XOR 31848 = **12345** = i1 — it round-trips exactly. + +That round-trip holds only because reducing mod a power of two *is* +masking the low bits, and XOR commutes with masking. That is why +`cuckoo.c` forces the bucket count to a power of two: +`numBuckets = getNextN2(capacity / bucketSize)` (`CuckooFilter_Init` :50) +and then `assert(isPower2(filter->numBuckets))` (:54). + +Why hash the fingerprint at all instead of the simpler `i XOR fp`? Paper +§3.1: with an 8-bit fingerprint, unhashed `i XOR fp` flips only the low 8 +bits, so a kicked key "will be placed to buckets that are at most 256 +buckets away from bucket i" and clumps; multiplying by `0x5bd1e995` first +spreads the flip across all bits, "relocating to buckets in an entirely +different part of the hash table". Two costs come with the trick: the +power-of-two sizing above, and the fact that a fingerprint's candidate +pair is determined by only `log₂(buckets) + f` bits, so as the table +grows those pairs repeat and the load analysis degrades (paper §4). ### Step 4 — the kicking loop, mechanically -With Steps 2–3 in hand, insertion is: try both candidate buckets; if both -are full, evict a random resident, move it to *its* other bucket -(computable by the involution), and repeat up to a bound. The insert path -with the kicking loop, in one screen: +> **In:** the involution from Step 3 and the two-bucket table from Step 2. +> **Out:** a working insert — plus the failure mode (kick chains that +> cycle) that the stub returns `false` on and production `cuckoo.c` +> absorbs with a subfilter chain. + +With Steps 2–3 in hand, insertion is: try both candidate buckets +(`Filter_FindAvailable` :241, first empty slot in either); if both are +full, evict a random resident, move it to *its* other bucket (the +involution), and repeat up to a bound. The real loop is `Filter_KOInsert` +:307; its heart: + +```c +// src/cuckoo.c — Filter_KOInsert kick loop, 318-332 + 318 while (counter++ < maxIterations) { + 319 uint8_t *bucket = &curFilter->data[ii * bucketSize]; + 320 swapFPs(bucket + victimIx, &fp); + 321 ii = getAltHash(fp, ii) % numBuckets; + 322 // Insert the new item in potentially the same bucket + 323 uint8_t *empty = Bucket_FindAvailable(&curFilter->data[ii * bucketSize], bucketSize); + 324 if (empty) { + // ... 325-327: three debug printf lines elided ... + 328 *empty = fp; + 329 return CuckooInsert_Inserted; + 330 } + 331 victimIx = (victimIx + 1) % bucketSize; + 332 } +``` + +Line 320 `swapFPs` swaps our fingerprint with the resident at `victimIx`, +so we now carry the evicted one; line 321 computes that evicted +fingerprint's *other* bucket via `getAltHash` (Step 3's involution); line +323 tries to seat it there. If the loop runs `maxIterations` times without +a free slot (the stub and the paper's empirical "full" threshold both use +**500**, line 656), insertion fails. The same conceptual flow, de-C'd so +the whole insert path is on one screen: ```rust -fn insert(&mut self, key: &[u8]) -> bool { - let (mut fp, i1) = self.fp_and_index(key); // fp: 12 bits, never 0 - let i2 = (i1 ^ self.hash_fp(fp)) & self.mask; // partial-key involution +// ILLUSTRATION — not quoted from RedisBloom; the real paths are +// Filter_FindAvailable at src/cuckoo.c:241 and Filter_KOInsert at src/cuckoo.c:307. +fn insert(&mut self, key: u64) -> bool { + let mut fp = fingerprint(key); // 12 bits in the stub, never 0 + let i1 = hash(key) & self.mask; + let i2 = (i1 ^ hash_fp(fp)) & self.mask; // partial-key involution (Step 3) if self.put_if_free(i1, fp) || self.put_if_free(i2, fp) { return true; } let mut i = if coin_flip() { i1 } else { i2 }; - for _ in 0..MAX_KICKS { // 500 - fp = self.swap_with_random_resident(i, fp); // evict someone - i = (i ^ self.hash_fp(fp)) & self.mask; // victim's OTHER bucket + for _ in 0..MAX_KICKS { // 500 + fp = self.swap_with_random_resident(i, fp); // evict someone + i = (i ^ hash_fp(fp)) & self.mask; // victim's OTHER bucket if self.put_if_free(i, fp) { return true; } } false // paper behavior; RedisBloom grows a subfilter instead @@ -95,35 +231,73 @@ fn insert(&mut self, key: &[u8]) -> bool { The cost that bloom never has: **insertion can fail** — at high load the kick chain can cycle for 500 hops without finding a free slot. The paper says return "full"; RedisBloom instead keeps a *chain of subfilters* (like -an LSM of filters): when kicking fails at MAX_KICKS it allocates a new -subfilter and inserts there (`CuckooFilter_InsertFP`, cuckoo.c:256 — try -all subfilters' empty slots first, kick only in the newest). Our stub -returns `false` (the paper behavior) — the graceful-failure test pins -that. Deletion (`CuckooFilter_Delete` :216) is find + zero the slot — but -it is only *safe* for keys actually inserted; deleting a false-positive -fingerprint removes someone else's resident. +an LSM of filters). `CuckooFilter_InsertFP` (:256) tries every existing +subfilter's empty slots first (:257–264, newest first), kicks only in the +newest (:268), and when even kicking fails it grows a fresh subfilter +(`CuckooFilter_Grow` :278) and retries (:283). Our stub returns `false` +(the paper behavior) — the `insert_fails_gracefully_when_full` test pins +that. Deletion (`CuckooFilter_Delete` :216, newest subfilter first) is +find + zero the slot — but it is only *safe* for keys actually inserted; +deleting a false-positive fingerprint removes someone else's resident. ### Step 5 — XOR filters: drop updates, win space +> **In:** the fingerprint idea from Step 2, now applied to a set known to +> be *static*. +> **Out:** the peeling construction and the 1.23 space factor that Step 6 +> ranks against bloom, cuckoo and ribbon. + The xor filter takes cuckoo's fingerprint idea and asks: if the set is -*static*, why pay for empty slots and kicking at all? Store an array B of -fingerprints such that for every key: +*static*, why pay for empty slots and kicking at all? Store an array **B** +of k-bit fingerprints such that for every key x: ``` - B[h0(x)] XOR B[h1(x)] XOR B[h2(x)] = fingerprint(x) + B[h0(x)] XOR B[h1(x)] XOR B[h2(x)] = fingerprint(x) ``` -Query = XOR three slots, compare — exactly 3 memory accesses, flat. -Construction "peels" a random 3-uniform hypergraph (each key is an edge -touching its 3 slots): repeatedly find a key that is the *only* one -touching some slot, assign that slot last (stack), pop and back-fill. -Peeling succeeds w.h.p. when slots ≥ 1.23 × keys — hence -**1.23 × f bits/key**, beating both bloom (1.44×) and cuckoo (~1.05/α× -but α≤0.95 plus empty-slot overhead). The price: build-once, forever — -adding one key invalidates the peeling order, so there is no insert, ever. +where **h0, h1, h2** are three independent hash functions, each mapping +into a disjoint third of B (Graf & Lemire, Table 1 / §3, line 153). Query += XOR three slots, compare — exactly 3 memory accesses, flat. The +false-positive rate is **ϵ = 1/2^k** for a k-bit fingerprint (line 127). + +Construction "peels" an **acyclic 3-partite random hypergraph** — a graph +whose edges each touch 3 vertices, here one edge per key touching its 3 +slots (§3.2, line 168): repeatedly find a slot touched by exactly one +key, push that (slot, key) onto a stack and remove the key from its three +slots; when the stack holds every key, pop it in reverse and back-fill +each slot so the key's XOR equation holds. The array must be a little +bigger than the key set for peeling to succeed — quote the size formula +(Table 1, line 139): + +``` + c = ⌊1.23 · |S|⌋ + 32 (c ≈ 1.23 · |S|) +``` + +Name the symbols: **|S|** is the number of keys, **c** is the array +length in slots. Peeling succeeds with probability > 0.8 at +c = 1.23·|S| + 32 for small sets and → 1 for large ones (line 193); the +**1.23** is the peelability threshold of a random 3-uniform hypergraph. +Worked on k=8-bit fingerprints (verified): + +- space = k · 1.23 = 8 × 1.23 = **9.84 bits/key** (line 272); the + compressed xor+ variant strips the ~19% empty slots (23 empty of every + 123, line 268) to 8 + 1.23 = **9.23 bits/key** (line 273). +- a bloom at the same ϵ = 1/2^8 spends 1.44 × 8 = **11.52 bits/key**, so + xor is ~15% smaller (9.84 / 11.52 = **0.854**). +- xor's overhead over the 8-bit floor is 9.84 − 8 = **1.84 bits/key**, + below standard cuckoo's ~3 and the semi-sorted variant's ~2 (line 111). +- concrete array size: |S| = 1,000,000 keys → c = ⌊1.23·10^6⌋ + 32 = + **1,230,032 slots**. + +The price: build-once, forever — adding one key invalidates the peeling +order, so there is no insert, ever. ### Step 6 — the lineage, with the trade each hop makes +> **In:** cuckoo (Steps 2–4) and xor (Step 5), each with its cost. +> **Out:** the workload→filter mapping — the practical output of the whole +> chapter. + Every hop in the fifty-year lineage buys one property by selling another — updatability, space, cache misses, and build reliability rotate through the designs: @@ -132,7 +306,7 @@ the designs: flowchart TD B["bloom: k smeared bits/key
1.44x space, k misses, no delete"] BB["blocked bloom: 1 miss
pays ~1.5-2x FPR"] - CK["cuckoo: discrete fingerprints
delete + ~0.18% FPR at 12 bits
pays: build can fail, pow2 sizing"] + CK["cuckoo: discrete fingerprints
delete + FPR 2b/2^f (3% at f=8, 0.2% at f=12)
pays: build can fail, pow2 sizing"] X["xor: static peeling
1.23x, 3 flat misses
pays: no updates ever"] RB["ribbon: banded GF(2) solve
~1.10x, streaming build
pays: slower build/query CPU"] B --> BB @@ -147,44 +321,51 @@ one cache miss matters more than FPR → blocked bloom. ## Where each step lives in the code -`cuckoo.c` — the production shape: +`cuckoo.c` — the production shape (RedisBloom @ `ab734fa`): | anchor | step | what it does | |---|---|---| -| `getAltHash` :122 | 3 | the involution: `i XOR hash(fp)` | -| `Filter_Find` :146 | 2 | check fp in both candidate buckets | +| `getLookupParams` :126 | 2 | `fp = hash % 255 + 1` (:127, 8-bit, 0 = empty), `h1 = hash`, `h2 = getAltHash(fp, h1)` | +| `getAltHash` :122 | 3 | the involution: `index ^ (fp * 0x5bd1e995)` (:123) | +| `CuckooFilter_Init` :44 | 3 | `numBuckets = getNextN2(...)` (:50), `assert(isPower2(...))` (:54) — power-of-two sizing the XOR needs | +| `Filter_Find` :146 | 2 | check fp in both candidate buckets (`Bucket_Find` :137) | | `Filter_FindAvailable` :241 | 4 | first empty slot in either bucket | -| `Filter_KOInsert` :307 | 4 | the kicking loop: evict a resident (`ii = getAltHash(fp, ii)` :321), swap, retry up to maxIterations | -| `CuckooFilter_InsertFP` :256 | 4 | try all subfilters' empty slots first, kick only in the newest, **grow a new subfilter** when kicking fails | +| `Filter_KOInsert` :307 | 4 | the kicking loop: evict a resident, `ii = getAltHash(fp, ii) % numBuckets` (:321), retry up to `maxIterations` | +| `CuckooFilter_InsertFP` :256 | 4 | try all subfilters' empty slots first, kick only in the newest (:268), **grow a new subfilter** (:278) when kicking fails | | `CuckooFilter_Delete` :216 | 4 | delete = find + zero the slot, newest subfilter first | Note what RedisBloom adds over the paper: the subfilter chain. When -kicking fails at MAX_KICKS it doesn't return "full" — it allocates a new -subfilter and inserts there. The xor filter (Step 5) has no reference +kicking fails at `maxIterations` it doesn't return "full" — it allocates a +new subfilter and inserts there. The xor filter (Step 5) has no reference implementation here — read the Graf & Lemire paper §2–3 with the peeling picture in hand. ## Tie back to the stub -`cuckoo::CuckooFilter` is cuckoo.c minus subfilter chaining: pow-2 buckets -of 4 × u16, 12-bit fp (never 0 = empty), random-victim kicking to -MAX_KICKS=500. The `delete_actually_removes` test is the point of the whole +`cuckoo::CuckooFilter` (`experiments/src/cuckoo.rs`) is `cuckoo.c` minus +subfilter chaining, and it swaps the fingerprint width: pow-2 buckets of +4 × `u16`, a **12-bit** fp (never 0 = empty), random-victim kicking to +`MAX_KICKS = 500`. Production `cuckoo.c` uses an 8-bit fp instead, so the +two disagree on FPR by design — 2b/2^f is 3.125% at f=8 and 0.195% at +f=12. The `delete_actually_removes` test is the point of the whole exercise — it's the test a bloom filter *cannot* pass. ## Questions to answer in notes.md 1. Why hash the fingerprint in `i1 XOR hash(fp)` instead of the simpler - `i1 XOR fp`? (Paper §3.1: with small fp values, unhashed XOR only - perturbs the low bits — kicked keys land nearby and clump.) + `i1 XOR fp`? (Paper §3.1: with an 8-bit fp, unhashed XOR flips only the + low 8 bits, so kicked keys land at most 256 buckets away and clump; + `× 0x5bd1e995` spreads the flip across all bits.) 2. Deletion is only safe if the key was actually inserted (deleting a false-positive fingerprint removes *someone else's* resident, creating a false negative for them). Redis documents this contract. How would you misuse `CF.DEL` to silently corrupt a filter, and why can't bloom have this failure mode (nor deletion at all)? -3. Why 4 slots per bucket? Paper Table 2: with 1 slot, load factor tops - out ~50%; with 4, ~95%. But more slots = more fingerprints compared per - query = higher FPR (`2 × slots × 2^−f`). Where's our stub's FPR bound - (12-bit fp, 4 slots, ~0.9 load) relative to the `< 1%` test? +3. Why 4 slots per bucket? Paper Figure 2 / §4: with b=1 the load factor + tops out ~50% (line 731); with b=4, ~95% (line 663). But more slots = + more fingerprints compared per query = higher FPR (2b/2^f). Where's our + stub's FPR bound (12-bit fp, 4 slots, ~0.9 load → 2·4/4096·0.9 ≈ + 0.176%) relative to the `< 1%` test? 4. The peeling stack is why xor filters are build-once: adding one key invalidates the topological order. Ribbon (see [reading-bloom-to-ribbon.md](reading-bloom-to-ribbon.md)) gets the same @@ -194,25 +375,129 @@ exercise — it's the test a bloom filter *cannot* pass. ## Done when +Answer each before unfolding it. + - [ ] You can explain why bloom cannot delete, in terms of shared bits. + +
Answer + + A bloom bit is shared across many keys, so clearing one key's bits can + lie about another. Insert A → bits {3, 17, 40}; insert B → {17, 52, 88}. + Delete A by clearing {3, 17, 40}; now B probes bit 17, finds 0, and is + reported **absent** though it was inserted — a false negative, the one + error a filter must never make (Step 1). Counting bloom filters swap each + bit for a 4–8× larger counter so a delete can decrement, but a counter + still cannot name *which* key it counts, so it cannot support the + discrete, per-key residence Step 2 needs. + +
+ - [ ] You can state the one trick that makes cuckoo *filters* possible: partial-key cuckoo hashing. + +
Answer + + Cuckoo *hashing* relocates keys between two candidate buckets, but a + filter has thrown the key away and kept only the fingerprint — so it + cannot recompute a victim's alternate bucket the normal way. Partial-key + cuckoo hashing computes it from the fingerprint instead: + `getAltHash` (`src/cuckoo.c:123`) returns `index ^ (fp * 0x5bd1e995)`, + the paper's Eq (2) `j = i ⊕ hash(fp)`. Since XOR is an involution, + applying it from either bucket yields the other, so a kick needs nothing + but the current bucket and the fingerprint sitting in it. + +
+ - [ ] You can explain why the alternate bucket is `i1 XOR hash(fp)` rather than something simpler. + +
Answer + + You need an involution so that a stored fingerprint can find its way + back — `i XOR hash(fp)` gives that from either side. The *hash* matters: + paper §3.1 shows that unhashed `i XOR fp`, with an 8-bit fingerprint, + flips only the low 8 bits, so a kicked key "will be placed to buckets + that are at most 256 buckets away" and everything clumps in one region, + wrecking occupancy. Multiplying by the MurmurHash2 constant + `0x5bd1e995` (`getAltHash`, :123) scatters the flip across all bits so + victims relocate to an entirely different part of the table. The trick + also forces the bucket count to a power of two (`assert(isPower2(...))`, + :54) so that XOR and the mod-numBuckets reduction commute. + +
+ - [ ] You can say why deletion is only safe for keys actually inserted. + +
Answer + + `CuckooFilter_Delete` (:216) finds a slot holding your fingerprint and + zeroes it (`Bucket_Delete` :154). But a fingerprint is only 8 bits in + production (12 in the stub), so two keys can share one; if you delete a + key that was never inserted but *collides* with a resident's + fingerprint, you remove that resident's only copy, and the resident now + probes both its candidate buckets, finds nothing, and is reported + **absent** — a false negative you manufactured. So `CF.DEL` is only + defined for keys you know were added. Bloom cannot even reach this + failure mode because it has no delete at all (Step 1). + +
+ - [ ] You can explain why 4 slots per bucket, with the load-factor numbers. + +
Answer + + Bucket size trades occupancy against FPR. Paper Figure 2 / §4: with b=1 + a cuckoo table tops out at ~50% load (line 731), with b=4 it fills to + ~95% (line 663), with b=8 to ~98% (line 664) — more slots per bucket + means fewer dead-end kicks. But every extra slot is another fingerprint + compared per query, so the FPR is 2b/2^f (Eq 5) and climbs linearly with + b. Four is the knee: near-full tables without paying much FPR. For the + stub (f=12, b=4, ~0.9 load) that bound is 2·4/4096·0.9 ≈ **0.176%**, + comfortably under the `< 1%` test. + +
+ - [ ] You can explain why the peeling stack makes XOR filters build-once. -- [ ] You wrote answers to all questions in notes.md. + +
Answer + + Construction peels the 3-uniform hypergraph by repeatedly removing a key + that is the sole occupant of some slot, pushing it on a stack, then + popping in reverse to back-fill each slot so `B[h0] XOR B[h1] XOR B[h2] + = fingerprint` holds (Step 5, §3.2). That order is a global property of + the whole key set: inserting one new key changes which slots are + singly-occupied and invalidates the topological order, so there is no + incremental insert — you rebuild from scratch. It is also why the array + must carry slack: c = ⌊1.23·|S|⌋ + 32 (line 139), the 1.23 being the + peelability threshold. Ribbon recovers a streaming build by solving a + banded GF(2) system instead of peeling. + +
+ +- [ ] You wrote answers to all four questions in notes.md. + +
Answer + + The four questions are the ones above: why the fingerprint is hashed + before the XOR (§3.1, the 256-bucket clumping argument), how `CF.DEL` + can corrupt a filter and why bloom can't have that mode, why 4 slots and + where the stub's 0.176% FPR sits under the `< 1%` test, and the + bloom/cuckoo/xor/ribbon ranking mapped onto memtable / churny routing + table / immutable SST. Each answer must cite either a `cuckoo.c` anchor + or a paper line, not just assert the shape — that is the exercise. + +
## References **Papers** - Fan, Andersen, Kaminsky, Mitzenmacher — "Cuckoo Filter: Practically - Better Than Bloom" (CoNEXT 2014) — §3 algorithm, §4 why partial-key - works, §5 space analysis; skim the eval + Better Than Bloom" (CoNEXT 2014) — §3 algorithm (Eq 1–2 partial-key), + §4 why partial-key works (Figure 2 load factors), §5 space (Eq 5–7) - Graf & Lemire — "Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters" (ACM JEA 2020, - [arXiv:1912.08258](https://arxiv.org/abs/1912.08258)) — §2-3 + [arXiv:1912.08258](https://arxiv.org/abs/1912.08258)) — §3 (Table 1, + Algorithms 1–3, the 1.23 factor and xor+ compression) **Code** - [RedisBloom](https://github.com/RedisBloom/RedisBloom) `src/cuckoo.c` - — the production shape, including the subfilter-chain growth the - paper doesn't have + @ `ab734fa` — the production shape, including the subfilter-chain growth + the paper doesn't have diff --git a/topics/26-probabilistic/reading-geo-indexes.md b/topics/26-probabilistic/reading-geo-indexes.md index 90c13ed..c7f976c 100644 --- a/topics/26-probabilistic/reading-geo-indexes.md +++ b/topics/26-probabilistic/reading-geo-indexes.md @@ -8,6 +8,12 @@ seams — then surveys the families that *do* build real spatial structures (R-tree, S2, H3), with the valkey source as the running example. +Every code anchor below is valkey at commit `8891441ab`, the revision +this repo pins (`src/geohash.c`, `src/geohash_helper.c`, `src/geo.c`), +quoted with the line numbers the code occupies in that version. The S2 +and H3 figures are quoted from their project docs; the geohash precision +is worked out on the spot from `GEO_STEP_MAX`. + ## The problem in one sentence "Every member within 200 m of this point" over millions of stored @@ -19,6 +25,11 @@ queries and a distance check on the few candidates** they return. ### Step 1 — the reframe: make 2D nearness look like key order +> **In:** nothing yet — this step fixes the one question a sorted index +> answers fast. +> **Out:** the plan to turn "near in 2D" into "a few 1D key ranges" via a +> single interleaved key. + A sorted index (zset, B-tree, anything) answers exactly one question fast: "give me all keys in range [a, b]". Spatial search needs a different question — "all points near (x, y)" — where nearness lives in @@ -31,46 +42,78 @@ function, one range computation. ### Step 2 — quantize: coordinates become fixed-width integers +> **In:** the "one integer key" plan from Step 1. +> **Out:** each of lat/lon as a 26-bit cell number, and why 26 (not 27) +> is the ceiling. + Bit tricks need integers, so each coordinate is first mapped from its continuous range to a fixed-width integer: valkey quantizes latitude and longitude each to **26 bits** within their range (lat −90..90, lon -−180..180) — cell number = `(value − min) / range × 2^26`. Two costs to -note: quantization is lossy (everything inside one cell is -indistinguishable until the final exact check), and 26 was not picked -casually — the combined 52 bits must survive storage in a zset score, -which is an IEEE double with a 52-bit mantissa (question 1 below makes -you work out both the precision and what breaks at 27 bits). +−180..180) — cell number = `(value − min) / range × 2^26` +(`GEO_STEP_MAX = 26` at geohash.h:46, commented "26*2 = 52 bits"). Worked +at the equator: `2^26 = 67,108,864` cells per axis; longitude spans 360° +≈ 40,075,017 m, so one cell is `40,075,017 / 2^26 ≈ 0.60 m` wide, and +latitude's 180° ≈ 20,003,931 m gives `20,003,931 / 2^26 ≈ 0.30 m` tall — +a sub-metre cell. Two costs to note: quantization is lossy (everything +inside one cell is indistinguishable until the final exact check), and 26 +was not picked casually — the combined 52 bits must survive storage in a +zset score, an IEEE double whose exact-integer ceiling is `2^53`; a 52-bit +code clears it, but 27 bits/axis (54 bits) would not (question 1 below +makes you work out both the precision and what breaks at 27 bits). ### Step 3 — interleave the bits: the Morton / Z-order code +> **In:** the two 26-bit cell numbers from Step 2. +> **Out:** one 52-bit Morton code whose prefixes name square cells — so a +> cell equals a contiguous key range. + A **Morton code** (Z-order code) interleaves the bits of the two quantized coordinates — y's bit i and x's bit i alternate — producing one 52-bit integer whose *prefixes* mean something: the top 2k bits identify a square cell at level k, so **two codes sharing a prefix are in the same cell** — prefix-similar codes = spatially-near points. The -interleave is five magic-mask rounds (geohash.c:52 does exactly this): - -```rust -fn interleave64(xlo: u32, ylo: u32) -> u64 { - let spread = |mut v: u64| { // 26 bits → every other bit - v = (v | (v << 16)) & 0x0000FFFF0000FFFF; - v = (v | (v << 8)) & 0x00FF00FF00FF00FF; - v = (v | (v << 4)) & 0x0F0F0F0F0F0F0F0F; - v = (v | (v << 2)) & 0x3333333333333333; - v = (v | (v << 1)) & 0x5555555555555555; - v - }; - spread(xlo as u64) | (spread(ylo as u64) << 1) // y25 x25 ... y0 x0 -} +interleave is five magic-mask rounds, quoted verbatim: + +```c +// src/geohash.c:52-76 (interleave64, valkey@8891441ab) + 52 static inline uint64_t interleave64(uint32_t xlo, uint32_t ylo) { + 53 static const uint64_t B[] = {0x5555555555555555ULL, 0x3333333333333333ULL, 0x0F0F0F0F0F0F0F0FULL, + 54 0x00FF00FF00FF00FFULL, 0x0000FFFF0000FFFFULL}; + 55 static const unsigned int S[] = {1, 2, 4, 8, 16}; + 56 + 57 uint64_t x = xlo; + 58 uint64_t y = ylo; + 59 + 60 x = (x | (x << S[4])) & B[4]; + 61 y = (y | (y << S[4])) & B[4]; + 62 + 63 x = (x | (x << S[3])) & B[3]; + 64 y = (y | (y << S[3])) & B[3]; + 65 + 66 x = (x | (x << S[2])) & B[2]; + 67 y = (y | (y << S[2])) & B[2]; + 68 + 69 x = (x | (x << S[1])) & B[1]; + 70 y = (y | (y << S[1])) & B[1]; + 71 + 72 x = (x | (x << S[0])) & B[0]; + 73 y = (y | (y << S[0])) & B[0]; + 74 + 75 return x | (y << 1); + 76 } ``` -(The same bit-twiddling as HAKMEM / Bit Twiddling Hacks.) The consequence -that makes everything work: a level-k cell is exactly the set of codes in -one contiguous range `[prefix << shift, (prefix+1) << shift)` — a cell IS -a key range. +(The same bit-twiddling as HAKMEM / Bit Twiddling Hacks — `y << 1` puts +latitude in the odd bit positions.) The consequence that makes everything +work: a level-k cell is exactly the set of codes in one contiguous range +`[prefix << shift, (prefix+1) << shift)` — a cell IS a key range. ### Step 4 — the search: candidate cells, range scans, exact verify +> **In:** the Morton-coded zset from Step 3. +> **Out:** the 9-cell candidate scan + haversine verify — one-sided +> over-fetch, then exact filter. + A radius query now decomposes into three moves: pick a cell size roughly matching the radius, scan that cell plus its 8 neighbors as zset score ranges, then filter the candidates with the exact **haversine** distance @@ -117,6 +160,10 @@ circle), and the exact filter fixes that. Two ideas worth stealing: ### Step 5 — the curve's seams: Z-order vs Hilbert +> **In:** the Z-order code from Step 3 and the 9-cell scan from Step 4. +> **Out:** why Z-order's jumps force many ranges per box, and what +> Hilbert trades to fix them. + A **space-filling curve** is the 1D visiting order a code imposes on the 2D grid, and Z-order's has seams — adjacent cells can be far apart on the curve: @@ -140,6 +187,10 @@ rotations instead of one mask cascade). That trade is the one S2 takes. ### Step 6 — the families that do build real spatial structures +> **In:** the curve-plus-verify approach from Steps 4–5 and its limits. +> **Out:** three real spatial families (R-tree, S2, H3) and the exact +> price each pays. + When candidate-then-verify over a curve isn't enough — exact containment, arbitrary polygons, spherical correctness — three families take over: @@ -150,14 +201,19 @@ arbitrary polygons, spherical correctness — three families take over: R-tree implemented *as a GiST extension* — read [reading-postgres-indexam.md](reading-postgres-indexam.md) with this in mind: GiST is the AM that lets `picksplit`/`penalty` be plugins. -- **S2 (Google)**: sphere → 6 cube faces → quadtree per face → - Hilbert-ordered 64-bit cell IDs. Hierarchy = prefix relation, so - containment tests are integer ops; coverings of a region are - sets of cells at mixed levels. +- **S2 (Google)**: sphere → 6 cube faces (level 0 is exactly **6 cells**) + → quadtree per face (×4 cells per level, **levels 0–30**) → + Hilbert-ordered **64-bit** cell IDs, with sub-cm² leaf cells at level + 30. Hierarchy = prefix relation, so containment tests are integer ops; + coverings of a region are sets of cells at mixed levels + ([s2geometry.io cell statistics](https://s2geometry.io/resources/s2cell_statistics)). - **H3 (Uber)**: hexagons (equidistant neighbors — nicer for - gradients/flows), icosahedron-based, but hexes don't nest - cleanly — the hierarchy is approximate. Great for - sharding/aggregation, weaker for exact containment. + gradients/flows), icosahedron-based across **16 resolutions (0–15)**; + resolution 0 has **122 base cells (110 hexagons + 12 pentagons)**, and + there are *exactly* 12 pentagons at every resolution (aperture-7, so a + cell has ≈7 children). Hexes don't nest cleanly — the hierarchy is + approximate. Great for sharding/aggregation, weaker for exact + containment ([h3geo.org resolution table](https://h3geo.org/docs/core-library/restable/)). The through-line: geohash-in-a-zset spends zero new structures and pays in over-fetch; the R-tree spends a whole tree and pays in overlap-driven @@ -168,11 +224,11 @@ discrete-cell-only answers. | anchor | step | what it does | |---|---|---| -| `geohash.c:52` `interleave64` | 3 | the Morton interleave, five magic-mask rounds | +| `geohash.c:52-76` `interleave64` | 3 | the Morton interleave, five magic-mask rounds | | `geohash_helper.c:64` `geohashEstimateStepsByRadius` | 4 | pick the cell level covering the radius; latitude-dependent, clamped near the poles | | `geo.c:338` `scoresOfGeoHashBox` | 4 | cell → zset score range: `hash << shift` to `(hash+1) << shift` | -| `geo.c:367` | 4 | the ZRANGEBYSCORE candidate fetch | -| `geo.c:375` `membersOfAllNeighbors` | 4 | the 3×3 neighborhood scan + haversine post-filter | +| `geo.c:367` `membersOfGeoHashBox` | 4 | one box's score range → ZRANGEBYSCORE candidate fetch | +| `geo.c:375` `membersOfAllNeighbors` | 4 | the 3×3 neighborhood scan (calls `membersOfGeoHashBox` per box at :424) + haversine post-filter | Read them in pipeline order (encode → step estimate → ranges → neighbors) — it is one straight-line data path, ~400 lines total. @@ -204,13 +260,84 @@ Read them in pipeline order (encode → step estimate → ranges → neighbors) ## Done when +Answer each before unfolding it. + - [ ] You can explain the reframe: making 2D nearness look like key order. + +
Answer + + A sorted index answers "all keys in [a, b]" fast. Interleaving (x, y) + into one Morton code makes code *prefixes* name square cells, so "near + in 2D" becomes a handful of 1D key ranges — the zset you already have + becomes a spatial index for one encode function plus one range + computation, zero new structures. + +
+ - [ ] You can compute a Morton code by hand and say why interleaving works. + +
Answer + + Alternate the bits so y-bit-i and x-bit-i interleave (`interleave64`, + geohash.c:52-76: five `(v | (v << S)) & B` mask rounds, then `x | (y << + 1)`). The top 2k bits then identify a level-k square cell, so a shared + prefix ⇒ same cell ⇒ numerically close codes ⇒ spatially near points, + and a level-k cell is one contiguous code range. + +
+ - [ ] You can explain why 26 bits per axis, connected to the zset score's precision. + +
Answer + + 26 bits/axis = a 52-bit code (`GEO_STEP_MAX = 26`, geohash.h:46). A zset + score is an IEEE double whose largest exact integer is 2^53, so 52 bits + store losslessly while 54 (27/axis) would round. At the equator a cell + is ~0.60 m wide (40,075,017 m / 2^26) × ~0.30 m tall (20,003,931 m / + 2^26). + +
+ - [ ] You can describe the candidate-cells, range-scan, exact-verify search and estimate the over-fetch factor. + +
Answer + + Pick a cell level ≈ the radius (`geohashEstimateStepsByRadius`, + geohash_helper.c:64), scan the cell + 8 neighbors as zset score ranges + (`scoresOfGeoHashBox` geo.c:338 → `membersOfGeoHashBox` geo.c:367, + looped over the 9 in `membersOfAllNeighbors` geo.c:375), then + haversine-filter. The 3×3 block is 9 cells against a query circle of + area ≈ π r²; with one cell ≳ the radius that is roughly a 3–10× + over-fetch, which the exact filter removes. + +
+ - [ ] You can explain the curve's seams and what Hilbert fixes. + +
Answer + + Z-order (Morton) jumps far along the curve when it crosses a quadrant + boundary (3 → 4 in the diagram), so one bounding box shatters into many + score ranges; valkey caps the damage with a fixed 3×3 neighborhood scan. + Hilbert rotates its pattern per quadrant so spatial neighbors stay + adjacent on the curve — fewer, longer ranges — at the cost of per-level + rotations instead of one mask cascade. S2 takes that trade. + +
+ - [ ] You wrote answers to all questions in notes.md, including the `GEO.ADD`/`GEO.SEARCH` sketch for M26. +
Answer + + Self-check: the six questions cover the 26-bit/double-precision link, + the latitude-dependent step estimate and pole clamps, the 3×3 + over-fetch factor vs precise Z-range decomposition, R-tree multi-path + descent, S2 prefix-containment vs H3 hexagons, and the M26 + `GEO.ADD`/`GEO.SEARCH` mapping — where encode + 9-cell range + haversine + are the only new code and the sorted property index is reused verbatim. + +
+ ## References **Papers** @@ -220,6 +347,9 @@ Read them in pipeline order (encode → step estimate → ranges → neighbors) **Code & docs** - [valkey](https://github.com/valkey-io/valkey) `src/geohash.c`, - `src/geohash_helper.c`, `src/geo.c` -- [s2geometry.io](https://s2geometry.io) — S2 cell hierarchy docs -- [h3geo.org](https://h3geo.org) — H3 hex grid docs + `src/geohash_helper.c`, `src/geo.c` (pinned at `8891441ab`; + `GEO_STEP_MAX = 26` in `src/geohash.h`) +- [s2geometry.io cell statistics](https://s2geometry.io/resources/s2cell_statistics) + — S2 cell hierarchy: 6 faces at level 0, levels 0–30 +- [h3geo.org resolution table](https://h3geo.org/docs/core-library/restable/) + — H3 hex grid: 16 resolutions, 122 base cells (110 hex + 12 pentagons) diff --git a/topics/26-probabilistic/reading-hyperloglog.md b/topics/26-probabilistic/reading-hyperloglog.md index ba0fea8..b3697f0 100644 --- a/topics/26-probabilistic/reading-hyperloglog.md +++ b/topics/26-probabilistic/reading-hyperloglog.md @@ -7,6 +7,13 @@ estimator step by step from that observation, then walks redis's production implementation, which adds a sparse encoding and a better count formula on top. +Every code anchor below is redis at commit `a176d1225`, the revision this +repo pins (`src/hyperloglog.c`), quoted with the line numbers the code +occupies in that version. The α constants and error figures are quoted +from Flajolet et al. 2007 and worked out on the spot; where redis diverges +from the original paper (a 64-bit hash, Ertl's estimator) the guide says +which paper each piece comes from. + ## The problem in one sentence Counting *distinct* elements exactly means remembering every element @@ -18,6 +25,11 @@ recognizing a duplicate requires the full history; HLL answers within ### Step 1 — why exact counting is expensive: duplicates need memory +> **In:** nothing yet — this step frames why a counter and a hash set are +> the two bad extremes. +> **Out:** the requirement for a *small, duplicate-blind* observable of +> the stream. + Cardinality (the number of *distinct* elements in a stream) can't be computed with a counter, because a counter can't tell a new element from a repeat — the only exact answer is a set, and a set's memory grows with @@ -28,10 +40,16 @@ the number of distinct elements but not with repeats? ### Step 2 — the observation: rare hash patterns imply many elements +> **In:** the "small, duplicate-blind observable" requirement from Step 1. +> **Out:** `rank` = leading-zero count + 1, whose running maximum tracks +> log₂(distinct count) — and the reason one lone max is too noisy to use. + Hash every element to uniform random bits; the probability that a given hash starts with j zero bits is 2^−(j+1), so if the *maximum* run of leading zeros you ever saw is j, you've plausibly seen ~2^(j+1) distinct -elements. Call `rank` = (leading-zero count + 1). Two properties make +elements. Call `rank` = (leading-zero count + 1). Worked: P(≥3 leading +zeros) = 2^−3 = 1/8, so seeing a rank of 4 (three zeros then a one) +suggests on the order of 2^4 = 16 distinct draws. Two properties make this the right observable: it's tiny (a max fits in 6 bits, since ranks top out near 64), and it's **duplicate-blind** — hashing the same element twice produces the same rank, and `max()` of a repeat changes nothing. @@ -40,11 +58,17 @@ off by 2–4×. ### Step 3 — registers: average away the noise +> **In:** the single, noisy `rank` maximum from Step 2. +> **Out:** m = 2^P registers and the ~1.04/√m error law — 0.81% at P=14. + Split the stream into m = 2^P substreams by the hash's low P bits, keep one 6-bit max ("register") per substream, and combine m noisy estimates into one — averaging cuts the relative error to ~1.04/√m, which at P=14 (m = 16,384 registers) is **0.81%** for 12 KB of state (16,384 × 6 bits). -One hashed key contributes only to one register: +Worked: √16,384 = 128, so 1.04/128 = 0.008125 = 0.81%; and 16,384 × 6 = +98,304 bits = 12,288 bytes = 12 KB. The 1.04/√m law is Flajolet et al. +2007 (§, "typical relative error ±1.04/√m"). One hashed key contributes +only to one register: ``` hash(x) = |...... 50 bits pattern ......|.. 14 bits ..| @@ -56,6 +80,8 @@ One hashed key contributes only to one register: The whole write path is five lines, and the merge is one: ```rust +// ILLUSTRATION — not quoted from redis; the real write path is hllPatLen +// (hyperloglog.c:467) computing rank, then hllDenseSet (:502) packing it. const P: u32 = 14; const M: usize = 1 << P; // 16384 registers, 1 byte each here @@ -79,20 +105,53 @@ you've committed 12 KB per counted thing even when it holds 3 elements ### Step 4 — the estimator: harmonic means and Ertl's formula +> **In:** the m register maxima from Step 3. +> **Out:** one cardinality number — via Flajolet's `α_m·m²/Σ` estimator +> (with its two range fixes), or redis's Ertl `σ`/`τ` re-derivation. + Turning 16,384 maxima into one number is the delicate part: the naive arithmetic mean of 2^rank is wrecked by outliers, so HLL uses a **harmonic mean** (the reciprocal of the average of reciprocals — it damps large outliers instead of amplifying them), plus corrections at -both extremes. Historically this was patched piecewise: Google's -"HLL in Practice" added an empirical bias table and a switch to linear -counting for small n; Ertl then *re-derived* the estimator so one formula -— two analytic series, `sigma` for the many-empty-registers low end and -`tau` for the saturation high end — is unbiased across the whole range. -Redis shipped Google's version for years, then switched (see the comment -above `hllCount`). The estimator, transcribed (this is `hllCount` minus -the caching): +both extremes. + +Three names to keep straight (rule 3 — every constant here is quoted from +its paper, not remembered): + +- **Original HLL (Flajolet et al. 2007).** Estimate `E = α_m · m² / Σⱼ + 2^−M[j]`, where `M[j]` is register *j*'s stored max rank and `α_m` + corrects the harmonic mean's bias. Flajolet §3 *defines* `α₁₆ = 0.673, + α₃₂ = 0.697, α₆₄ = 0.709`, and `α_m = 0.7213 / (1 + 1.079/m)` for `m ≥ + 128`. Worked at m = 16,384: `0.7213 / (1 + 1.079/16384) = 0.72125`. + Beware — the small-*m* entries are tabulated, not from that formula: + `0.7213/(1 + 1.079/16) = 0.676`, which is *not* the listed `0.673`, so + 16/32/64 get their own constants. Two range fixes bolt on: **small + range** — when `E ≤ 5m/2` (= 40,960 at m = 16,384) and `V` registers are + still zero, switch to linear counting `m·ln(m/V)`; **large range** — + when `E > 2³²/30 ≈ 1.43×10⁸`, undo 32-bit hash collisions with + `−2³²·ln(1 − E/2³²)` (Flajolet §4, small/large-range corrections). +- **HLL++ (Heule et al. 2013, "HyperLogLog in Practice").** Replaces the + 32-bit hash with a **64-bit** one (§5.1), which retires the large-range + correction outright; swaps Flajolet's small-range switch for an + **empirically-tabulated bias correction** (§5.2); and adds the sparse + representation of Step 5. These are HLL++'s fixes, *not* Flajolet's and + *not* Ertl's. +- **Ertl 2017 — what redis ships now.** Re-derives one estimator with no + piecewise switch: `α_∞ · m² / z`, where `α_∞ = 1/(2 ln 2) = 0.7213475` + (redis `HLL_ALPHA_INF` = `0.721347520444481703680` at + `hyperloglog.c:404`, commented "constant for 0.5/ln(2)") and `z` folds + in two analytic series — `σ` (`hllSigma` :1016) for the + many-empty-registers low end and `τ` (`hllTau` :1033) for the saturated + high end. + +Redis shipped Google's HLL++ estimator for years, then switched to Ertl's; +the comment above `hllCount` records the change. The Ertl estimator, +transcribed (this is `hllCount` minus the caching): ```rust +// ILLUSTRATION — not quoted from redis; the real estimator is hllCount +// (hyperloglog.c:1058), with sigma/tau at :1016/:1033 and the reghisto +// fold at :1084-:1090. fn count(regs: &[u8; M]) -> f64 { let mut histo = [0u32; 64]; for &r in regs { histo[r as usize] += 1; } // count() reads the HISTOGRAM @@ -111,9 +170,13 @@ Notice `count()` consumes the *histogram* of register values ### Step 5 — the sparse encoding: why PFCOUNT keys start at 30 bytes +> **In:** the fixed 12 KB dense array from Step 3. +> **Out:** a run-length sparse encoding that starts at ~30 bytes and +> promotes to dense on demand. + Dense = 12 KB always, even for 3 elements — so redis adds a second, run-length-encoded representation for the mostly-zero early life of a -sketch (the opcode table at hyperloglog.c:380-383): +sketch (the opcode macros at hyperloglog.c:380-392): ``` ZERO: 00xxxxxx → 1..64 zero registers in ONE byte @@ -126,10 +189,15 @@ elements costs ~30 bytes, not 12 KB. The price is write complexity: `hllSparseSet` (:675) is a 150-line opcode splice — an *insert into a compressed stream* — and the encoding promotes to dense (`hllSparseToDense` :593) when it exceeds `hll-sparse-max-bytes` (3 KB -default) or any rank > 32 arrives (VAL has only 5 value bits). +default) or any rank > 32 arrives (`HLL_SPARSE_VAL_MAX_VALUE = 32` at +:389 — VAL has only 5 value bits). ### Step 6 — merge = max: the killer feature is algebraic +> **In:** the register array from Step 3 (dense, or promoted from sparse). +> **Out:** the merge = per-register max identity, and what that buys a +> distributed distinct-count. + Because a register is a max and max is associative, commutative, and idempotent, `merge(A,B).regs == union(A∪B).regs` *exactly* (our test demands register equality, not approximate counts) — HLLs form a @@ -154,7 +222,7 @@ encodings; read it before the functions. | `hllDenseRegHisto` :528 | 4 | builds `reghisto[rank]` — count() consumes the *histogram*, not the registers | | `hllSigma` :1016, `hllTau` :1033 | 4 | Ertl's two series (linear-counting-like correction at the low end, saturation correction at the high end) | | `hllCount` :1058 | 4 | the estimator: `m·tau(...)`, fold histogram with repeated halving, `+ m·sigma(reghisto[0]/m)`, then `alpha_inf·m²/z` | -| :380-383 opcode table, `hllSparseSet` :675, `hllSparseToDense` :593 | 5 | the sparse encoding and its promotion | +| :380-392 opcode macros, `hllSparseSet` :675, `hllSparseToDense` :593 | 5 | the sparse encoding and its promotion (`HLL_SPARSE_VAL_MAX_VALUE 32` at :389) | | `hllMergeDense` :1279 (AVX2 :1116, NEON :1218) | 6 | merge = per-register max, vectorized | ## Tie back to the stub @@ -187,22 +255,102 @@ the ranges the old estimator needed three different formulas for. ## Done when +Answer each before unfolding it. + - [ ] You can explain why rare hash patterns imply many distinct elements. + +
Answer + + For a uniform hash, P(a value has ≥ j leading zeros) = 2^−j, so the + *maximum* rank (leading-zero-run + 1) observed over a stream grows like + log₂(distinct count). A repeat hashes to the same value and can't push a + max higher, so the observable tracks cardinality, not traffic. Redis + computes the rank in `hllPatLen` (hyperloglog.c:467). + +
+ - [ ] You can say why index bits and pattern bits must not overlap. + +
Answer + + The low P bits pick the register `j`; the remaining bits form the + pattern whose leading-zero rank is stored. If the two sets overlapped, + `j` and `rank` would be correlated, violating the assumption that the m + substreams are independent — and the 1.04/√m error law only holds for m + *independent* estimators. (Question 1.) + +
+ - [ ] You can explain what the registers average away and why the harmonic mean. + +
Answer + + A single max is off by 2–4×; splitting into m = 2^P registers and + combining drops the relative error to 1.04/√m (0.81% at P = 14, since + 1.04/128 = 0.008125). The arithmetic mean of 2^rank is dominated by one + lucky outlier, so HLL uses a harmonic mean scaled by a bias constant: + Flajolet's `α₁₆ = 0.673 … α_m = 0.7213/(1+1.079/m)`, or in redis Ertl's + `α_∞ = 1/(2 ln 2) = 0.72135` (`HLL_ALPHA_INF` at :404) with `σ`/`τ` + (`hllSigma` :1016, `hllTau` :1033). + +
+ - [ ] You can explain the sparse encoding and why a PFCOUNT key starts at 30 bytes. + +
Answer + + Dense is 12 KB regardless of load. Sparse run-length-encodes the + mostly-zero early sketch: ZERO/XZERO pack up to 16,384 zero registers in + 1–2 bytes, VAL packs a rank 1..32 repeated 1..4 times (opcode macros + hyperloglog.c:380-392). An empty HLL is `XZERO(16384)` ≈ 2 bytes + + header; ~100 elements ≈ 30 bytes. It promotes to dense + (`hllSparseToDense` :593) past `hll-sparse-max-bytes` (3 KB) or when a + rank > 32 arrives (`HLL_SPARSE_VAL_MAX_VALUE = 32` at :389). + +
+ - [ ] You can state the killer feature — merge is max, therefore algebraic — and say what that buys a distributed count. + +
Answer + + A register is a max, and max is associative, commutative, and + idempotent, so `merge(A,B)` equals the HLL of `A ∪ B` exactly (register + equality, not approximate counts) — HLLs form a semilattice. Per-shard, + per-hour, per-node sketches therefore merge losslessly in any order, + with repeats and overlaps free, so a distinct-count needs no + coordination. `hllMergeDense` (:1279) is a per-register max, vectorized + (AVX2 :1116, Aarch64/NEON :1218). Cost asymmetry: PFADD touches 1 + register, PFMERGE touches all 16,384. + +
+ - [ ] You wrote answers to all five questions in notes.md, including the ZERO/XZERO/VAL comparison against roaring's containers. +
Answer + + Self-check: the five questions cover index/pattern independence, the + low-range degeneracy to linear counting `m·ln(m/V)`, the rank ≤ 32 + sparse ceiling, the ZERO/XZERO/VAL-vs-roaring-container density metric, + and the per-label HLL write-path sketch. All five belong in notes.md + before this box is checked. + +
+ ## References **Papers** +- Flajolet, Fusy, Gandouet, Meunier — "HyperLogLog: the analysis of a + near-optimal cardinality estimation algorithm" (AofA 2007) — §3 defines + the bias constants (`α₁₆ = 0.673, α₃₂ = 0.697, α₆₄ = 0.709, α_m = + 0.7213/(1+1.079/m)` for `m ≥ 128`) and the `1.04/√m` standard error; §4 + the small/large-range corrections. This is the *original* HLL, not the + version redis runs today - Heule, Nunkesser, Hall — "HyperLogLog in Practice" (Google, EDBT 2013) - — §3-5 are the practical fixes; the original Flajolet '07 analysis is - optional + — §5.1 the 64-bit hash, §5.2 the empirical bias table, and the sparse + representation; these are HLL++'s additions on top of Flajolet - Ertl — "New cardinality estimation algorithms for HyperLogLog sketches" ([arXiv:1702.01284](https://arxiv.org/abs/1702.01284), 2017) - — §2-3; the estimator redis uses now + — §2-3; the `σ`/`τ` estimator (`α_∞ = 1/(2 ln 2)`) redis uses now **Code** - [redis](https://github.com/redis/redis) `src/hyperloglog.c` — the diff --git a/topics/26-probabilistic/reading-learned-indexes.md b/topics/26-probabilistic/reading-learned-indexes.md index 1692b3e..3c44104 100644 --- a/topics/26-probabilistic/reading-learned-indexes.md +++ b/topics/26-probabilistic/reading-learned-indexes.md @@ -1,39 +1,68 @@ # Learned indexes: the index is a model of the CDF -An index maps key → position. If the key distribution is smooth, a -handful of linear models approximates that map with a bounded error you -binary-search away — replacing a tree walk's cache misses with two -multiply-adds. Three designs mark the territory: RMI (the provocation), -PGM (the guarantee — our stub), and ALEX (the one that takes writes). -This chapter builds the idea from the reframe up — index as function, -error bounds, segment construction, updatability — then anchors each -piece in the PGM and ALEX sources. +An index maps a key to a position in a sorted array. If the key +distribution is smooth, a handful of linear models approximates that map +with a **bounded error** (a hard cap ε on how far a prediction can miss) +you binary-search away — replacing a tree walk's dependent memory +accesses with two multiply-adds. Three designs mark the territory: RMI +(the provocation — fast, no guarantee), PGM (the guarantee — our stub), +and ALEX (the one that takes writes). This chapter builds the idea from +the reframe up — index as function, error bounds, segment construction, +updatability — then anchors each piece in the PGM and ALEX sources. + +Every code anchor below is PGM-index at commit `c6fcf3d` +(`include/pgm/pgm_index.hpp` is 266 lines, `piecewise_linear_model.hpp` +is 365) and ALEX at commit `4370da6` (`src/core/alex_nodes.h` is 2330 +lines), quoted with the line numbers each occupies in that revision. +Paper facts come from Kraska et al. 2018 ("The Case for Learned Index +Structures"), the PGM-index paper (Ferragina & Vinciguerra, VLDB 2020) +and the ALEX paper (Ding et al., SIGMOD 2020); every number names the +section or figure it came from. ## The problem in one sentence -On the motivation bench, a point-miss binary search over 10M sorted u64 -keys costs **167 ns ≈ 23 dependent cache misses** — and if the key -distribution is smooth, most of those 23 hops land exactly where a +On this topic's motivation bench a point-miss over 10M sorted u64 keys +costs **246 ns** by binary search ([FINDINGS.md](../../FINDINGS.md) row +26) — about **⌈log₂(10,000,000)⌉ = 24** dependent comparisons, each a +branch mispredict into a different cache line — and if the key +distribution is smooth, most of those 24 hops land where a two-multiply-add linear model would have predicted for free. ## The concepts, step by step ### Step 1 — the reframe: an index is a function, and a B-tree is already a model +> **In:** nothing yet — this step fixes the reframe every later step +> leans on. +> **Out:** the identity `pos = n · CDF(key)`, and the "predict, then +> search a small window" template that Steps 2, 3 and 5 each implement +> with a different accuracy guarantee. + An index is a function from key to position in a sorted array — and that function is precisely the **CDF** (cumulative distribution function: the -fraction of keys ≤ x) of the key distribution, scaled by n. This is -Kraska's opening move: a B-tree computes `pos ≈ n · CDF(key)` as a -piecewise-constant approximation with worst-case-everything guarantees; -if the CDF is *smooth*, a few linear models predict the position in O(1) -with a small error to binary-search away: +fraction of keys ≤ x, a number in [0, 1]) of the key distribution, scaled +by n. Kraska et al. open on exactly this — "a B-Tree-Index can be seen as +a model to map a key to the position of a record within a sorted array" +(2018, Abstract). Written out: ``` - pos ≈ n · CDF(key) + pos(key) = n · CDF(key) n = number of keys, CDF(key) ∈ [0, 1] +``` - B-tree: log_B(n) node hops, each a cache miss (167 ns measured, ~23 misses) - learned: 1-2 model evals + binary search of 2ε (the bet: most of the - window tree walk is predictable) +Name the symbols: **n** is the key count, **CDF(key)** is the fraction of +keys ≤ key, and **pos(key)** is that key's index in the sorted array. +Worked on 10M keys: a key at the 37th percentile (CDF = 0.37) sits at +pos = 10,000,000 × 0.37 = **3,700,000**; the median (CDF = 0.5) at +**5,000,000**. A B-tree computes this same `pos ≈ n · CDF(key)` as a +**piecewise-constant** approximation — one constant per leaf — with +worst-case-everything guarantees; if the CDF is *smooth*, a few **linear** +models predict the position in O(1) with a small residual to +binary-search away: + +``` + B-tree: ~log_B(n) node hops, each a dependent miss (246 ns, ~24 comparisons) + learned: 1-2 model evals + binary search of a 2ε (the bet: most of the + window tree walk is predictable) ``` The bet, stated honestly: trade guaranteed log-time on any distribution @@ -42,120 +71,259 @@ auto-increment IDs, steady-ingest timestamps. ### Step 2 — RMI: the provocation, without a safety net -The RMI (recursive model index, Kraska §3) is a fixed 2-stage hierarchy -of models where stage 1's model doesn't predict the position — it *picks* -which stage-2 model does. The stage-2 model then predicts a position, and -the search corrects the residual error. The flaw that motivates -everything after it: **no error bound**. A model that fits badly on some -key region gives predictions off by thousands of slots, the correcting -search becomes long and unpredictable, and there's no principled way to -size the stages. RMI proved the reframe was fast; it didn't make it safe. +> **In:** the reframe from Step 1 — index as `n · CDF(key)`. +> **Out:** a fast but *unbounded* design; the missing error guarantee is +> exactly what Step 3 (PGM) supplies. + +The **RMI** (recursive-model index, Kraska §3.2) is a hierarchy of models +— inspired by the mixture-of-experts idea — where an upper-stage model +does not predict the position but *picks* which lower-stage model does. +Formally, "at stage ℓ there are Mℓ models"; the stage-0 model f0(x) ≈ y +takes the key and selects a model in the next stage, "until the final +stage predicts the position" (Kraska §3.2, Figure 3). There is **no +search between stages** — each stage is a bare model evaluation. Their +experiments use two stages. + +Why a hierarchy at all: one model over 100M keys cannot get the residual +small, but "reducing the error to 10k from 100M ... a precision gain of +100 ∗ 100 = 10000 to replace the first 2 layers of a B-Tree ... is much +easier" (Kraska §3.2). Worked: stage 0 narrows 100,000,000 → 10,000 (a +10⁴ cut), and the picked stage-1 model then narrows 10,000 → 100 — the +same 10⁴ overall, split across two easy models instead of one impossible +one. + +The flaw that motivates everything after it: **no error bound**. A +stage-2 model that fits badly on some key region gives a prediction off +by thousands of slots, the correcting last-mile search becomes long and +unpredictable, and there is no principled ε to size it. (The naïve +single-network version made the point by counter-example: it took +"≈ 80,000 ns" per lookup in TensorFlow versus "≈ 300ns" for a B-tree +traversal over the same data, Kraska §2.3 — accuracy, not raw model +speed, is the game.) RMI proved the reframe was fast; it did not make it +safe. ### Step 3 — PGM: fix the error first, then minimize the model -PGM inverts the design: choose a hard error bound ε *up front*, then -compute the **minimum number of linear segments** such that every key's -predicted position is within ε of the truth — lookup = evaluate the -segment's line, then binary-search a window of just 2ε+2 slots. To find -the right segment among (say) 2,000 of them, index the segments' first -keys with... another PGM, recursively, until one segment remains — each -level is itself ε-bounded, so each hop is a *constant-size* search, not a -binary search over all segments. Why it matters: the segments (a few KB) -live in cache where a B-tree's top levels don't even, and the ε guarantee -holds on *any* distribution — hostile keys cost more *segments* (space), -never a longer lookup. Our `epsilon_holds_on_hostile_distribution` test -pins exactly that. +> **In:** Step 2's missing guarantee. +> **Out:** the ε-bounded segment and the recursive lookup, whose one-pass +> construction Step 4 details. + +The **PGM-index** (Piecewise Geometric Model) inverts the design: choose a +hard error bound **ε** (the maximum a prediction may differ from the true +position) *up front*, then compute the **minimum number of linear +segments** such that every key's predicted position is within ε of the +truth. That is the paper's Definition 2: "computing the PLA-model which +minimises the number of its segments ... provided that each segment is +ε-approximate for its covered range of keys." A lookup evaluates the +segment's line, then binary-searches a window of just **2ε + 2** slots. + +That window is not folklore — it is two macros in the header: + +```cpp +// pgm_index.hpp — the search-window macros, 32-33 + 32 #define PGM_SUB_EPS(x, epsilon) ((x) <= (epsilon) ? 0 : ((x) - (epsilon))) + 33 #define PGM_ADD_EPS(x, epsilon, size) ((x) + (epsilon) + 2 >= (size) ? (size) : (x) + (epsilon) + 2) +``` + +`search` (:192-198) predicts `pos`, then returns `[PGM_SUB_EPS(pos, ε), +PGM_ADD_EPS(pos, ε, n))`. Name the symbols: **x** is the predicted +position, **epsilon** the bound, **size** = n the key count; the low edge +clamps at 0 and the high edge at n, and the trailing `+ 2` covers the +segment boundary. Worked at ε = 64, n = 10M, predicted pos = 5,000,000: +the window is [5,000,000 − 64, 5,000,000 + 64 + 2) = **[4,999,936, +5,000,066)**, width **130 = 2ε + 2**, so the final binary search is +⌈log₂130⌉ = **8** comparisons instead of the full 24. At ε = 16 the width +is 34 (⌈log₂34⌉ = 6); at ε = 256 it is 514 (⌈log₂514⌉ = 10). + +To find the right segment among (say) 2,000 of them, PGM indexes the +segments' first keys with... another PGM, recursively, until one segment +remains (PGM paper §3.2: "proceed recursively by building another optimal +PLA-model ... until the PLA-model consists of one" segment). Each level is +itself ε-bounded, so `segment_for_key` (:134) descends level to level and +each hop is a **constant-size** search over a window of `EpsilonRecursive` +slots (:143-153, default EpsilonRecursive = 4), not a binary search over +all segments. Why it matters: the segments (a few KB) live in cache where +a B-tree's upper levels may not, and the ε guarantee holds on *any* +distribution — hostile keys cost more *segments* (space), never a longer +lookup. Our `epsilon_holds_on_hostile_distribution` test pins exactly +that. ### Step 4 — building segments in one pass: the shrinking cone -Computing the minimal ε-bounded piecewise-linear fit sounds expensive but -is a streaming, O(n) pass: maintain the set of lines that could still fit -every point seen so far within ε, and emit a segment the moment that set -goes empty. PGM's `OptimalPiecewiseLinearModel` uses O'Rourke '81's -streaming convex-hull method (provably *fewest* segments for a given ε); -our stub uses the simpler **shrinking cone**: keep an interval [lo, hi] -of feasible slopes through the segment's first point; each new point -narrows it; emit when empty. Same ε guarantee, ≥ as many segments, and -O(1) state instead of two hulls: +> **In:** the ε target from Step 3. +> **Out:** an O(n) streaming build, and the *static* limitation (one +> insert invalidates every later position) that Step 5 removes. + +Computing an ε-bounded piecewise-linear fit sounds expensive but is a +streaming, O(n) pass: maintain the set of lines that could still fit every +point seen so far within ε, and emit a segment the moment that set goes +empty. PGM's `OptimalPiecewiseLinearModel` (`piecewise_linear_model.hpp:45`) +runs the *optimal* version — the streaming convex-hull method the PGM +paper proves yields the fewest segments for a given ε (Lemma 1: it +"computes the minimum number of segments"). It keeps upper and lower hulls +in a `rectangle[4]` (`add_point` :96; the point-outside test that closes a +segment at :130-136; hull maintenance :154-158; `get_segment` :190). + +Our stub uses the simpler heuristic the PGM paper attributes to the +FITing-tree and calls the **shrinking cone** — "linear in time but does +not guarantee to find the optimal PLA-model" (PGM paper §3.1). Keep an +interval `[lo, hi]` of feasible slopes through the segment's *first* +point; each new point narrows it; emit when it empties. The narrowing is +the whole algorithm, and it is the stub's own doc comment: ```rust -struct Cone { x0: u64, y0: f64, lo: f64, hi: f64 } // slopes through (x0,y0) - -fn add_point(c: &mut Cone, x: u64, y: usize, eps: f64) -> bool { - let (dx, dy) = ((x - c.x0) as f64, y as f64 - c.y0); - c.lo = c.lo.max((dy - eps) / dx); // each point NARROWS the feasible - c.hi = c.hi.min((dy + eps) / dx); // slope interval... - c.lo <= c.hi // ...empty ⇒ emit segment, start fresh -} +// topics/26-probabilistic/experiments/src/pgm.rs — LearnedIndex::build stub, 22-32 + 22 /// STUB — shrinking-cone greedy PLA over sorted, deduped keys: + 23 /// open a segment at (k0, pos0) with slope cone (lo, hi) = (0, inf); + 24 /// for each next point (k, pos), the segment can keep it iff some + 25 /// slope in the cone predicts pos within eps — narrow the cone to + 26 /// lo = max(lo, (pos - eps - pos0) / (k - k0)) + 27 /// hi = min(hi, (pos + eps - pos0) / (k - k0)) + 28 /// and close the segment (emit slope = (lo+hi)/2) when the cone + 29 /// empties, starting a fresh one at (k, pos). + 30 pub fn build(_keys: &[u64], _epsilon: usize) -> LearnedIndex { + 31 todo!("greedy shrinking-cone segmentation") + 32 } ``` -The cost profile that falls out: build is O(n) single-pass (vs a B-tree's -O(n log n) of page splits), on 1M uniform keys under 2K segments suffice -(the `uniform_data_compresses_hard` test), and the structure is *static* — -one insert invalidates every position after it. Which is Step 5's -problem. +Name the symbols: **(k0, pos0)** is the segment's anchor point, **(k, +pos)** the incoming key and its true rank, **eps** the bound, and +**[lo, hi]** the still-feasible slopes for a line *through the anchor*. +Worked on four points at ε = 1, anchor (k0, pos0) = (0, 0), cone starting +(0, ∞): + +``` + pt (1, 0): lo=max(0, (0-1-0)/1)=0.000 hi=min(∞, (0+1-0)/1)=1.000 [0.000, 1.000] open + pt (2, 2): lo=max(0, (2-1-0)/2)=0.500 hi=min(1, (2+1-0)/2)=1.000 [0.500, 1.000] open + pt (3, 5): lo=max(.5, (5-1-0)/3)=1.333 hi=min(1, (5+1-0)/3)=1.000 [1.333, 1.000] EMPTY +``` + +At the third point lo (1.333) exceeds hi (1.000): no single slope through +(0, 0) keeps all four within ε = 1, so the segment closes after three +points and a fresh one opens at (3, 5). The catch that the cone pays for +being cheap is visible here — a line *not* forced through the anchor, +`pos = 1.5·k − 0.5`, fits all four (residuals −0.5, +1.0, +0.5, −1.0, all +≤ 1), so the optimal convex-hull method would have kept the segment open. +That is why the stub emits ≥ as many segments as PGM, never fewer. + +The cost profile that falls out: build is O(n) single-pass (versus a +B-tree's O(n log n) of page splits), and on uniform keys segments ≪ n — +PGM reserves only `n / (epsilon * epsilon)` of them (`pgm_index.hpp:97`), +which at n = 1M, ε = 64 is 1,000,000 / 4096 = **244**, comfortably under +the `uniform_data_compresses_hard` test's ceiling of n/500 = 2,000. But +the structure is *static*: one insert shifts every rank after it, so every +downstream segment is invalidated. That is Step 5's problem. ### Step 5 — ALEX: gapped arrays make the model updatable -A static PGM re-builds on change; ALEX makes the *data layout* absorb -updates instead. Its nodes are **gapped arrays** — sorted arrays with -~50% empty slots left deliberately interspersed — and the model is used -not only to search but to *place*: model-based insertion puts a new key -at its predicted slot (shifting only to the closest gap), so the data -keeps matching the model as it arrives. Lookups use **exponential -search** from the predicted slot (probe at distance 1, 2, 4, 8... then -binary-search the bracketed range): cost is O(log of the model's actual -error), so it adapts — usually 0–2 slots — without needing PGM's hard-ε -accounting. When a node overflows its density bound it splits and -retrains: the B-tree skeleton reappears, but with models as node search -and gaps as write absorbers. The cost: hostile insert patterns pile keys -onto one predicted slot and trigger shift/retrain storms — write -amplification is where ALEX degrades. +> **In:** Step 4's static limitation — a PGM must rebuild on insert. +> **Out:** a design that absorbs writes in the *data layout*, at a +> write-amplification cost Step 6 puts on the scoreboard. + +A static PGM rebuilds on change; **ALEX** makes the data layout absorb +updates instead. Its data nodes are **gapped arrays** — sorted arrays +with empty slots deliberately interspersed — and the model is used not +only to search but to *place*. ALEX does not pack the array full: it holds +density between a lower and an upper limit, "dl = 0.6 and du = 0.8 to +achieve average data storage utilization of 0.7" (ALEX §4.3.1), so on +average **~30% of slots are gaps**, not the half a first guess suggests. +Gaps are filled "with the closest key to the right of the gap, which helps +maintain exponential search performance" (§3.1). + +The per-node model is a clamped linear predictor: + +```cpp +// alex_nodes.h — AlexDataNode::predict_position, 1448-1452 + 1448 inline int predict_position(const T& key) const { + 1449 int position = this->model_.predict(key); + 1450 position = std::max(std::min(position, data_capacity_ - 1), 0); + 1451 return position; + 1452 } +``` + +Line 1450 is the one to watch: the raw model output is clamped to +`[0, data_capacity_ - 1]`, so a wild prediction can never index out of the +array. Worked at `data_capacity_ = 1,000,000`: a model that outputs −3 is +clamped to **0**, one that outputs 1,050,000 to **999,999**, and a sane +500,000 passes through unchanged. Lookups then use **exponential search** +from that slot — `find_key` (:1456) calls `exponential_search_upper_bound` +(:1557), probing at distance 1, 2, 4, 8, … until the key is bracketed, +then binary-searching the bracket. Cost is O(log d) in the model's +*actual* error d, so it adapts without PGM's hard-ε accounting: if the +model is off by d = 100 slots the search takes about ⌈log₂100⌉ = **7** +doublings plus a short binary search; if it is spot-on, 0–1 probes. The +paper relies on exactly this — "exponential search without bounds is +faster than binary search with bounds ... because if the models are good, +their prediction is close enough to the correct position" (§3.1). + +Insertion places a key near its predicted slot and shifts toward the +**closest gap** rather than the array end (`insert_element_at` calls +`closest_gap` :1935; the shift count is tallied in `num_shifts_` +:1915/:1927), so the data keeps matching the model as it arrives. When a +node's fill would exceed du it expands — allocating `n / dl` new slots and +re-inserting every element under a retrained model (§4.3.2). Those shifts +and expansions are ALEX's cost currency: **write amplification**. The +honest correction to the folklore that ALEX falls over on adversarial +inserts — the paper measures the opposite for the classic adversary: +initialized with the 50M smallest keys and fed the rest in ascending +sorted order, "ALEX has up to 3.6× higher throughput than B+Tree" +(§6.2.6). Where it *can* degrade is a model so mismatched to the arriving +keys that shifts run long and expansions come often; constructing that +case is the exercise, precisely because the sorted-insert one does not. ### Step 6 — the honest scoreboard: how each design degrades -The deep difference between the three is not speed on friendly data — -it's *which resource* gives out on hostile data: +> **In:** the three designs from Steps 2, 3 and 5. +> **Out:** the one axis that separates them — *which resource* gives out +> on hostile data. + +The deep difference between the three is not speed on friendly data — it +is *which resource* gives out on hostile data: ``` build lookup (smooth keys) lookup (hostile) inserts - B-tree O(n log n) ~log_B(n) misses same native + B-tree O(n log n) ~log_B(n) hops same native RMI train fast, NO bound can be terrible no - PGM O(n) 1-3 hops + 2ε window MORE segments, PGM-dynamic: - bound still holds LSM-of-PGMs - ALEX O(n) predict + exp search retrain storms native, gapped + PGM O(n) 1-3 hops + 2ε window MORE segments, static (rebuild); + bound still holds PGM-dynamic: LSM-of-PGMs + ALEX O(n) predict + exp search more shifts / native, gapped + expansions (robust to sorted, §6.2.6) ``` The ε guarantee is the dividing line: PGM degrades in *space* (more -segments) while lookup stays bounded; RMI degrades in *time*; ALEX -degrades in *write amplification*. The B-tree degrades in nothing and -wins on nothing — which is exactly why it's the incumbent. +segments) while its lookup stays bounded at 2ε + 2; RMI degrades in *time* +(no bound on the last-mile search); ALEX degrades in *write amplification* +(shifts and node expansions), though the paper shows that stays modest +even under sorted-order inserts (§6.2.6). The B-tree degrades in nothing +and wins on nothing — which is exactly why it is the incumbent. ## Where each step lives in the code PGM — Steps 3–4 -([`~/repos/PGM-index/include/pgm/`](https://github.com/gvinciguerra/PGM-index)): +([`~/repos/PGM-index/include/pgm/`](https://github.com/gvinciguerra/PGM-index), +commit `c6fcf3d`): | anchor | what it is | |---|---| -| `pgm_index.hpp:32-33` | `PGM_SUB_EPS`/`PGM_ADD_EPS` — the window is [pos−ε, pos+ε+2), clamped; the +2 matters (segment boundaries) | -| `pgm_index.hpp:67` | `class PGMIndex`; `build` :88 loops `make_segmentation` per level | -| `segment_for_key` :134 | the recursive descent: each level is itself ε-bounded, so each hop is a *constant-size* search (:143-152), not a binary search over all segments | -| `search` :192 | predict, widen by ε, return the window — our `search_window` | -| `piecewise_linear_model.hpp:45` | `OptimalPiecewiseLinearModel` — O'Rourke '81 streaming convex-hull method | -| `add_point` :96, hull updates :154-190 | maintains upper/lower convex hulls of the feasible-slope region; segment closes when hulls cross | -| `make_segmentation` :276 | the greedy driver: `if (!opt.add_point(x,y)) { out(segment); start fresh }` | +| `pgm_index.hpp:32-33` | `PGM_SUB_EPS`/`PGM_ADD_EPS` — the window is [pos−ε, pos+ε+2), clamped to [0, n); the trailing `+2` covers the segment boundary (Step 3's quoted block) | +| `pgm_index.hpp:66-67` | `template<..., size_t Epsilon = 64, size_t EpsilonRecursive = 4, ...> class PGMIndex`; `build` :88 loops `make_segmentation` per level | +| `segment_for_key` :134 | the recursive descent (:143-153): each level is itself ε-bounded, so each hop is a *constant-size* search over `EpsilonRecursive` slots, not a binary search over all segments | +| `search` :192 | predict, widen by ε, return the `ApproxPos` window (approximate position + [lo, hi)) — our `search_window` | +| `piecewise_linear_model.hpp:45` | `OptimalPiecewiseLinearModel` — the optimal streaming convex-hull PLA (PGM paper Lemma 1: minimum segments in O(n)) | +| `add_point` :96; outside-test :130-136; hull update :154-158; `get_segment` :190 | maintains upper/lower hulls in `rectangle[4]`; the segment closes when a new point falls outside the feasible parallelogram (:133-136 returns `false`) | +| `make_segmentation` :276 | the greedy driver: `if (!opt.add_point(x,y)) { out(get_segment()); re-add }` (:280-286) | ALEX — Step 5 -([`~/repos/ALEX/src/core/alex_nodes.h`](https://github.com/microsoft/ALEX)): +([`~/repos/ALEX/src/core/alex_nodes.h`](https://github.com/microsoft/ALEX), +commit `4370da6`): | anchor | what it is | |---|---| -| `class AlexDataNode` :293 | gapped array + per-node linear model; `num_keys_` :325 vs slots = the gap budget | -| `predict_position` :1448 | the model eval | -| `find_key` :1456 | predict, then `exponential_search_upper_bound` :1462 from the predicted slot — cost is O(log distance-of-model-error), no ε needed | -| `find_insert_position` :1497 | same predict-then-search on the insert path | -| :28, :474, :1513 | the gap machinery: bitmap marks gap vs key; inserts shift toward the *closest gap*, not the array end | +| `class AlexDataNode` :293 | gapped array + per-node linear model; `data_capacity_` :324 (slots) vs `num_keys_` :325 (filled) is the gap budget — default du = 0.8 fullness, ~0.7 utilization (§4.3.1) | +| `predict_position` :1448 | the model eval; :1450 clamps the output to `[0, data_capacity_-1]` (Step 5's quoted block) | +| `find_key` :1456 | predict, then `exponential_search_upper_bound` :1462 (defined :1557) from the predicted slot — cost O(log distance-of-model-error), no ε needed | +| `find_insert_position` :1497 | the same predict-then-search on the insert path | +| `check_exists` :474; `get_next_filled_position` :1513; `closest_gap` :1935 | the gap machinery: the bitmap marks gap vs key (:474), inserts shift toward the *closest gap* not the array end (:1935; shifts tallied in `num_shifts_` :1915/:1927) | ## Questions to answer in notes.md @@ -164,8 +332,9 @@ ALEX — Step 5 *first* point; optimal PLA doesn't.) 2. ε trades segment count against final-search width. Segments live in cache; the 2ε window is one or two line fetches into the data. Given - the motivation numbers (167 ns ≈ 23 misses), predict the ns/lookup - curve for ε ∈ {16, 64, 256} on 10M uniform keys *before* running + the motivation numbers (246 ns ≈ 24 comparisons, + [FINDINGS.md](../../FINDINGS.md) row 26), predict the ns/lookup curve + for ε ∈ {16, 64, 256} on 10M uniform keys *before* running filter_bench. 3. `uniform_data_compresses_hard` demands < 2K segments for 1M random u64. Why is a *uniform* CDF the easy case, and what real key patterns @@ -177,7 +346,8 @@ ALEX — Step 5 ALEX's shifts-per-insert, and which classical structure degrades the same way under sorted-order inserts? (This is the "does ALEX survive adversarial inserts?" question in notes.md — predict, then read the - paper's §5.5.) + paper's §6.2.6, which measures ALEX at up to 3.6× B+Tree throughput on + sorted-order inserts, and §5.1's worst-case RMI-depth bound.) 5. **(cross-topic)** ALEX's gapped array + model placement vs a B-tree leaf with slotted-page free space (topic 2): both reserve slack to make inserts local. What does ALEX's *model* buy over the B-tree's @@ -186,28 +356,174 @@ ALEX — Step 5 ## Done when +Answer each before unfolding it. + - [ ] You can state the reframe: an index is a model of the CDF, and a B-tree is already one. + +
Answer + + An index is a function from key to position in a sorted array, and that + function is `pos(key) = n · CDF(key)` — n the key count, CDF(key) the + fraction of keys ≤ key, so pos is the key's rank (Step 1; Kraska 2018, + Abstract: "a B-Tree-Index can be seen as a model to map a key to the + position of a record within a sorted array"). Worked: 10M keys, a key at + CDF = 0.37 sits at 10,000,000 × 0.37 = 3,700,000. + + A B-tree is already a model of that CDF — a *piecewise-constant* one, one + constant per leaf, with worst-case guarantees. The learned bet is to + replace it with a *piecewise-linear* model where the CDF is smooth, so a + prediction plus a tiny corrective search beats the ~24 dependent + comparisons a 10M-key binary search costs (246 ns, + [FINDINGS.md](../../FINDINGS.md) row 26). + +
+ - [ ] You can explain what RMI provokes and what safety net it lacks. + +
Answer + + The RMI (recursive-model index, Kraska §3.2, Figure 3) is a hierarchy of + models where an upper stage *picks* the lower-stage model rather than + predicting a position — "at stage ℓ there are Mℓ models ... until the + final stage predicts the position", with no search between stages. It + provokes by proving the reframe is fast: splitting a hard 100M→100 fit + into two easy stages (100M→10k then 10k→100, "a precision gain of + 100 ∗ 100 = 10000", Kraska §3.2). + + The safety net it lacks is an **error bound**. A stage model that fits + badly gives a prediction off by thousands of slots, and the last-mile + search that corrects it has no principled ε to size it — so the + worst-case lookup is unbounded. That missing guarantee is exactly what + PGM supplies (Step 3). + +
+ - [ ] You can explain PGM's inversion: fix the error bound first, then minimize the model. + +
Answer + + PGM fixes a hard ε up front, then computes the *minimum* number of + ε-approximate linear segments (PGM paper Definition 2: "the PLA-model + which minimises the number of its segments ... provided that each + segment is ε-approximate"). A lookup evaluates the segment's line and + binary-searches a window of exactly 2ε + 2 slots, spelled out by + `PGM_SUB_EPS`/`PGM_ADD_EPS` (`pgm_index.hpp:32-33`) and returned by + `search` (:192). + + Worked at ε = 64, pos = 5,000,000, n = 10M: window [4,999,936, + 5,000,066), width 130 = 2ε + 2, final search ⌈log₂130⌉ = 8 comparisons + versus the full 24. Segments are found by another PGM recursively + (§3.2), each level ε-bounded so each hop is a constant-size search + (`segment_for_key` :134, :143-153). The payoff: the guarantee holds on + *any* distribution — hostile keys cost more segments (space), never a + longer lookup. + +
+ - [ ] You can construct four points where the shrinking cone closes a segment. + +
Answer + + Take ε = 1, anchor (k0, pos0) = (0, 0), cone starting (0, ∞), and the + points (1, 0), (2, 2), (3, 5). The cone narrows with + `lo = max(lo, (pos−eps−pos0)/(k−k0))`, `hi = min(hi, (pos+eps−pos0)/(k−k0))` + (the stub doc, `pgm.rs:22-32`): + + - (1, 0): lo = max(0, −1) = 0.000, hi = min(∞, 1) = 1.000 → [0.000, 1.000] + - (2, 2): lo = max(0, 0.5) = 0.500, hi = min(1, 1.5) = 1.000 → [0.500, 1.000] + - (3, 5): lo = max(0.5, 1.333) = 1.333, hi = min(1, 2.0) = 1.000 → **empty** + + At the third point lo > hi, so the segment closes after three points and + a new one opens at (3, 5). This is also the answer to notes.md Q1: the + line `pos = 1.5·k − 0.5` (not through the anchor) fits all four within + ε = 1 (residuals −0.5, +1.0, +0.5, −1.0), so PGM's optimal convex-hull + method keeps the segment open where the cone splits — the cone forces + every candidate line through the *first* point, which optimal PLA does + not. + +
+ - [ ] You can explain how ε trades segment count against final search width. + +
Answer + + ε is the only knob. A larger ε lets one line cover a wider key range, so + there are *fewer* segments (PGM reserves `n/(epsilon*epsilon)`, + `pgm_index.hpp:97` — at n = 1M, ε = 64 that is 244), but the final search + window is *wider*: 2ε + 2 slots, `⌈log₂(2ε+2)⌉` comparisons in the data. + + Worked on 10M keys: ε = 16 → window 34, ⌈log₂34⌉ = 6; ε = 64 → window + 130, 8; ε = 256 → window 514, 10. So doubling-and-then-some of ε adds + ~2 comparisons to the last-mile search while cutting segment count + roughly with ε². Segments live in cache (cheap to touch); the 2ε window + is one or two line fetches into the 76 MB data array (expensive), which + is why the sweet spot is workload-dependent — the subject of notes.md + Q2. + +
+ - [ ] You can describe an adversarial insert sequence and how ALEX's gapped arrays respond. + +
Answer + + A gapped array is a sorted array kept below a density limit — default + du = 0.8, ~0.7 average utilization, so ~30% gaps (ALEX §4.3.1). Inserts + place a key at its model-predicted slot and shift toward the *closest + gap* (`closest_gap` :1935, shifts tallied in `num_shifts_` + :1915/:1927); when a node would exceed du it expands to `n/dl` slots and + re-inserts under a retrained model (§4.3.2). The adversary is a stream + whose keys the node's linear model cannot fit — every insert then shifts + far and expansions come often, driving up write amplification. + + The honest result, though: for the *classic* adversary — sorted-order + inserts, every new key larger than all present — ALEX is measured at "up + to 3.6× higher throughput than B+Tree" (§6.2.6), the same pattern that + degrades a naïve B-tree's rightmost leaf. So the degrading case has to be + a genuine model mismatch, not merely sorted input; that is the + construction notes.md Q4 asks for, checked against §6.2.6 and the + §5.1 RMI-depth bound. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five (Step references in parentheses): Q1 — four points where the + cone splits but optimal PLA continues (Step 4; the (0,0),(1,0),(2,2),(3,5) + construction above). Q2 — the predicted ns/lookup curve for + ε ∈ {16, 64, 256}, reasoned from window widths 34/130/514 and the 246 ns + ≈ 24-comparison baseline ([FINDINGS.md](../../FINDINGS.md) row 26). + Q3 — why a uniform CDF compresses to < 2,000 segments for 1M keys and + what breaks it (Step 4; near-uniform = auto-increment IDs, steady-ingest + timestamps; broken by hot/cold tenants, hash-distributed keys). Q4 — + the adversarial-insert construction and ALEX's shift/expansion response + (Step 5, §6.2.6). Q5 — ALEX's model-placement vs a slotted B-tree leaf + (topic 2). + + Write them as predictions *before* running `filter_bench`; the point of + notes.md is to keep the wrong predictions next to the measured numbers. + +
+ ## References **Papers** - Kraska, Beutel, Chi, Dean, Polyzotis — "The Case for Learned Index Structures" (SIGMOD 2018, - [arXiv:1712.01208](https://arxiv.org/abs/1712.01208)) — §1-3 (RMI), - skim the rest + [arXiv:1712.01208](https://arxiv.org/abs/1712.01208)) — §2.3 (the naïve + learned index), §3.2 (RMI, Figure 3), skim the rest - Ferragina & Vinciguerra — "The PGM-index" (VLDB 2020, - [pgm.di.unipi.it](https://pgm.di.unipi.it)) + [pgm.di.unipi.it](https://pgm.di.unipi.it)) — §3.1 Definition 2 + (ε-approximate PLA), §3.1 the shrinking cone vs optimal streaming + convex hull (Lemma 1), §3.2 the recursive construction - Ding et al. — "ALEX: An Updatable Adaptive Learned Index" (SIGMOD - 2020, [arXiv:1905.08898](https://arxiv.org/abs/1905.08898)) + 2020, [arXiv:1905.08898](https://arxiv.org/abs/1905.08898)) — §3.1 + (gapped array + exponential search), §4.3 (density limits, node + expansion), §6.2.6 (robustness to sorted-order inserts) **Code** -- [PGM-index](https://github.com/gvinciguerra/PGM-index) +- [PGM-index](https://github.com/gvinciguerra/PGM-index) @ `c6fcf3d` `include/pgm/` — `pgm_index.hpp` + `piecewise_linear_model.hpp` -- [ALEX](https://github.com/microsoft/ALEX) `src/core/` — +- [ALEX](https://github.com/microsoft/ALEX) @ `4370da6` `src/core/` — `alex_nodes.h` is where the gapped-array machinery lives diff --git a/topics/26-probabilistic/reading-postgres-indexam.md b/topics/26-probabilistic/reading-postgres-indexam.md index 0840778..87a7b4f 100644 --- a/topics/26-probabilistic/reading-postgres-indexam.md +++ b/topics/26-probabilistic/reading-postgres-indexam.md @@ -8,18 +8,32 @@ AM step by step — what an access method is, what an exact tree probe costs, how an inverted index compresses, and what the smallest possible index looks like — then points you at the postgres sources. +Every code anchor below is postgres at commit `701f021`, the revision +this repo pins — PostgreSQL **20devel** (`meson.build` `version: +'20devel'`) — quoted with the line numbers the code occupies in that +version. The `IndexAmRoutine` struct and its callback list have grown +across releases, so the anchors in Step 1 are stated against this tree +specifically. + ## The problem in one sentence Exactness has a price list: a postgres btree probe is 3–4 *page* reads -cold (218 ns for an in-memory BTreeMap on the motivation bench), every -insert dirties a leaf page plus WAL, and the tree itself costs ~50–100 -bits per key — each probabilistic structure in this topic undercuts -exactly one line of that bill. +cold (299 ns for an in-memory BTreeMap on the motivation bench, +[FINDINGS.md](../../FINDINGS.md) row 26), every insert dirties a leaf page +plus WAL, and the tree itself costs ~50–100 bits per key — each +probabilistic structure in this topic undercuts exactly one line of that +bill. ## The concepts, step by step ### Step 1 — what an index AM is: three price points behind one API +> **In:** nothing yet — this step fixes the common interface the three +> AMs plug into. +> **Out:** the `IndexAmRoutine` callback vtable, and the three AMs +> (nbtree/GIN/BRIN) that fill it at different points on the exactness +> spectrum. + An index **access method** (AM) is a pluggable index implementation behind a common postgres interface — build, insert, and "give me candidate row locations (**TIDs** — tuple identifiers, physical @@ -30,29 +44,75 @@ key, amortized writes), and BRIN (a one-sided "maybe in this page range" — barely an index at all). Same question as a bloom filter — "where might X be?" — three different bills. -### Step 2 — nbtree descent: what 23 cache misses buys you +The "common interface" is literally a struct of function pointers: +`IndexAmRoutine` (`src/include/access/amapi.h:233-326` in this tree), +whose fields are the callbacks every AM must supply — `aminsert` (:298), +and the two scan entry points `amgettuple` (:312, "can be NULL") and +`amgetbitmap` (:313, "can be NULL"). Each AM exports one handler function +that allocates and fills this struct: `brinhandler` (`brin.c:254`) sets +`.amgetbitmap = bringetbitmap` at `brin.c:301`, while nbtree's `bthandler` +supplies `amgettuple` instead. That NULL-vs-set choice — tuple-at-a-time +ordered scan (nbtree) versus bitmap-only (BRIN, GIN) — is the first +visible fork between "exact position" and "candidate set." + +### Step 2 — nbtree descent: what 3–4 page reads buy you + +> **In:** the `amgettuple` scan callback from Step 1. +> **Out:** the root→leaf descent (`_bt_search` → per-page `_bt_binsrch`) +> and the *exact position* it returns, priced at 3–4 page reads. A btree probe walks root→leaf: read a page, binary-search *within* the page to find the child pointer, follow it, repeat — `_bt_search` -(nbtsearch.c:100) calling `_bt_binsrch` (:33, called at :153) per level. +(nbtsearch.c:100) calling `_bt_binsrch` (defined at :343, called at :153) +per level. The in-page search is an ordinary invariant-carrying binary +search, worth reading in the original because every filter in this topic +is trying to avoid it: + +```c +// src/backend/access/nbtree/nbtsearch.c:388-404 (_bt_binsrch, postgres@701f021) + 388 high++; /* establish the loop invariant for high */ + 389 + 390 cmpval = key->nextkey ? 0 : 1; /* select comparison value */ + 391 + 392 while (high > low) + 393 { + 394 OffsetNumber mid = low + ((high - low) / 2); + 395 + 396 /* We have low <= mid < high, so mid points at a real slot */ + 397 + 398 result = _bt_compare(rel, key, page, mid); + 399 + 400 if (result >= cmpval) + 401 low = mid + 1; + 402 else + 403 high = mid; + 404 } +``` + On a 10M-key index that's 3–4 page visits, each a cache-or-disk miss -chain — the in-memory analogue measured 218 ns. What the misses buy is -the strongest possible answer: the *exact* position of the key, plus +chain — the in-memory analogue (a `BTreeMap` point miss) measured 299 ns +([FINDINGS.md](../../FINDINGS.md) row 26). What the misses buy is the +strongest possible answer: the *exact* position of the key, plus ordered iteration from it (range scans), on *any* key distribution, with no error to verify. That "no verification needed" property is exactly what every structure in this topic gives up first. ### Step 3 — what exactness costs under concurrency and on writes +> **In:** the working descent from Step 2. +> **Out:** the three bills a filter never pays — lock-free concurrency, +> suffix truncation/dedup, and per-insert write amplification. + Three things bloom/PGM never have to deal with, all visible in nbtree: -- **Concurrency**: `_bt_moveright` (:211) — a reader racing a page split +- **Concurrency**: `_bt_moveright` (defined at :242; the doc comment + explaining the move is at :211) — a reader racing a page split recovers by walking right-links (Lehman & Yao); no lock coupling on the descent. The README's L&Y section is the payoff read. - **Suffix truncation & deduplication**: internal keys are truncated separators, duplicate leaf keys share a posting list - (`_bt_binsrch_posting` :34) — nbtree has been absorbing - compressed-postings ideas from the GIN/roaring world. + (`_bt_binsrch_posting` defined at :603, called at :573) — nbtree has + been absorbing compressed-postings ideas from the GIN/roaring world. - **Write path**: every insert dirties a leaf (WAL, FPIs, topic 3) — the write amplification that makes "just add another index" a real bet. @@ -63,6 +123,10 @@ alternatives worth wanting. ### Step 4 — GIN: an inverted index is topic 23 wearing a trench coat +> **In:** nbtree's "one key, one TID" model from Steps 2–3. +> **Out:** GIN's key → sorted-TID posting list, its varbyte delta +> compression, and the pending-list write buffer. + GIN maps key → **posting list** of TIDs — exactly a search engine's term → docIDs — for the "many keys per row" cases (arrays, JSONB, full-text). Because posting lists are sorted TIDs, they compress: @@ -79,16 +143,27 @@ list, and a neglected vacuum lets it grow. ### Step 5 — BRIN: the zone map that admits it's a filter -BRIN stores per-block-range summaries: min/max per 128-page range -(`brininsert` brin.c:349 unions new values into the range's `BrinMemTuple` -:157-170; `bringetbitmap` :301 returns *candidate page ranges*, never -rows). It is exactly topic 12's zone map, and it is *already* -probabilistic in the useful direction: *one-sided* — it can say "range -definitely has no qualifying rows," never "row definitely exists." +> **In:** the exact AMs of Steps 2–4. +> **Out:** BRIN's per-range min/max summary and its *one-sided* +> `amgetbitmap` that prunes page ranges, never confirms rows. + +BRIN stores per-block-range summaries: min/max per 128-page range. +`brininsert` (brin.c:349) folds a new heap tuple's values into the +covering range's summary — an in-memory `BrinMemTuple` +(`src/include/access/brin_tuple.h:44-56`; two summaries merge via +`union_tuples`, brin.c:225; the build-time holder `BrinBuildState` is +brin.c:159-172). `bringetbitmap` (registered as the `amgetbitmap` +callback at brin.c:301, defined at brin.c:572) returns *candidate page +ranges*, never rows. It is exactly topic 12's zone map, and it is +*already* probabilistic in the useful direction: *one-sided* — it can say +"range definitely has no qualifying rows," never "row definitely exists." The entire query-side logic fits in a filter: ```rust +// ILLUSTRATION — not quoted from postgres; the real callback is +// bringetbitmap (src/backend/access/brin/brin.c:572), which ANDs the +// scankeys against each range summary and emits candidate page ranges. fn bringetbitmap(ranges: &[MinMax], q: (Val, Val)) -> Vec { ranges.iter().enumerate() .filter(|(_, r)| r.min <= q.1 && q.0 <= r.max) // overlap ⇒ MAYBE @@ -111,13 +186,17 @@ postgres is also the most workload-dependent. ### Step 6 — the price list, side by side +> **In:** the three AMs from Steps 2–5. +> **Out:** the one table that names, per AM, the exact-cost column each +> probabilistic structure in this topic undercuts. + Line the three AMs up and the whole topic's thesis appears: each probabilistic structure shadows one exact AM and undercuts one column of its bill. | AM | granularity | answer type | write cost | shadow in this topic | |---|---|---|---|---| -| nbtree | row (TID) | exact | leaf dirty + WAL per insert | the 167/218 ns baseline lanes | +| nbtree | row (TID) | exact | leaf dirty + WAL per insert | the 246/299 ns baseline lanes | | GIN | key → TID set | exact set | pending-list amortized | roaring/postings (topic 23) | | BRIN | 128-page range | one-sided maybe | update range summary | zone maps (topic 12), bloom's cousin | @@ -132,10 +211,11 @@ All under [postgres](https://github.com/postgres/postgres) | anchor | step | what it is | |---|---|---| +| `amapi.h` `IndexAmRoutine` :233-326 | 1 | the callback vtable every AM fills (`aminsert` :298, `amgettuple` :312, `amgetbitmap` :313) | | `nbtree/README` | 2–3 | genuinely one of the best docs in any codebase; read it fully (the Lehman & Yao section is the payoff) | -| `nbtree/nbtsearch.c` | 2–3 | the descent: `_bt_search` :100, `_bt_binsrch` :33, `_bt_moveright` :211, `_bt_binsrch_posting` :34 | +| `nbtree/nbtsearch.c` | 2–3 | the descent: `_bt_search` :100, `_bt_binsrch` def :343 (called :153), `_bt_moveright` def :242, `_bt_binsrch_posting` def :603 | | `gin/ginpostinglist.c` + `gin/README` | 4 | varbyte posting lists: `ginCompressPostingList` :196, decode :284/:297 | -| `brin/brin.c` + `brin/README` | 5 | block-range summaries: `brininsert` :349, `BrinMemTuple` :157-170, `bringetbitmap` :301 | +| `brin/brin.c` + `brin/README` | 5 | block-range summaries: `brinhandler` :254, `brininsert` :349, `bringetbitmap` def :572 (registered :301), `BrinBuildState` :159-172 | Read the READMEs before the .c files — postgres's in-tree docs are the rare case where that order pays. @@ -167,13 +247,88 @@ rare case where that order pays. ## Done when +Answer each before unfolding it. + - [ ] You can explain what an index AM is and name the three price points behind the one API. + +
Answer + + An AM is a pluggable index behind the `IndexAmRoutine` callback vtable + (`amapi.h:233-326`): build, `aminsert` (:298), and a scan callback — + `amgettuple` (:312) for ordered tuple-at-a-time or `amgetbitmap` (:313) + for a candidate bitmap. The three price points are nbtree (exact + position, most expensive), GIN (exact TID *set* per key, amortized + writes), and BRIN (one-sided "maybe in this page range," nearly free). + +
+ - [ ] You can say what nbtree's cache misses buy you that a filter cannot. + +
Answer + + The 3–4 page reads of `_bt_search` (nbtsearch.c:100) / `_bt_binsrch` + (:343) return the *exact* position — no false positives to verify — plus + ordered iteration for range scans, on any key distribution. The + in-memory analogue is a 299 ns `BTreeMap` point miss + ([FINDINGS.md](../../FINDINGS.md) row 26). A filter only ever says + "definitely absent / maybe present"; it can neither locate the row nor + scan in order. + +
+ - [ ] You can explain what exactness costs under concurrency and on writes. + +
Answer + + Concurrency: `_bt_moveright` (def :242) walks right-links so a reader + racing a split needs no lock coupling (Lehman & Yao). Writes: every + insert dirties a leaf page plus WAL (and possible full-page images), so + N btree indexes on a table cost N dirtied leaves per row insert — the + write amplification a probabilistic structure avoids. Dedup shares + duplicate leaf keys via posting lists (`_bt_binsrch_posting` :603). + +
+ - [ ] You can explain why GIN is an inverted index and BRIN is an admitted filter. + +
Answer + + GIN maps key → sorted posting list of TIDs (a search engine's term → + docIDs), delta-compressed with varbyte (`ginCompressPostingList` + ginpostinglist.c:196), buffered through a pending list — the inverted + index of topic 23. BRIN keeps only a min/max summary per 128-page range + (`brininsert` brin.c:349) and its `amgetbitmap` (`bringetbitmap` :572) + returns candidate page ranges only, never rows — a one-sided filter, the + same "definitely not here" shape as a bloom filter. + +
+ - [ ] You can state the precise condition under which BRIN on a column is useful. + +
Answer + + BRIN prunes a range iff that range's `[min, max]` does not overlap the + query interval, so it only helps when the column is correlated with + physical (heap) order — append-only timestamps prune well; a random UUID + v4 makes every range's `[min, max]` span the whole domain, so nothing + prunes. That is why BRIN is ~10,000× smaller than a bloom filter + (128 pages per entry) yet entirely workload-dependent. + +
+ - [ ] You wrote answers to all questions in notes.md, including the M26 synthesis — and you have this topic's in-memory baseline (BTreeMap miss at 299 ns) to put beside postgres's page-based numbers. +
Answer + + Self-check: map each M26 piece onto the AM it shadows — range index → + nbtree, LSM blooms → none (postgres has no LSM to hang them on), roaring + label filters → GIN, HLL count-distinct → none (postgres computes + `count(DISTINCT)` exactly). The absence that hurts a graph workload most + is the missing per-file bloom, priced by point-miss cost (299 ns) × + MATCH miss rate — the product topic 4 already measured. + +
+ ## References **Code** ([postgres](https://github.com/postgres/postgres), `src/backend/access/`) diff --git a/topics/26-probabilistic/reading-roaring-internals.md b/topics/26-probabilistic/reading-roaring-internals.md index 08073bc..45b4cfc 100644 --- a/topics/26-probabilistic/reading-roaring-internals.md +++ b/topics/26-probabilistic/reading-roaring-internals.md @@ -3,145 +3,259 @@ The workhorse of every "set of row/node IDs" problem: chop the u32 space into 64K chunks and store each chunk in whichever of three encodings is smallest for its density. This chapter extends topic 23's -guide (`topics/23-search/reading-postings.md`) and its `postings.rs` -stub — array/bitmap containers exist there already; here we build the -full machine step by step — the density crossover, the chunking, the run -container, the pairwise kernels, the SIMD story — following the -roaring-rs port. +Roaring guide (`topics/23-fulltext/reading-roaring.md`) and its +`topics/23-fulltext/experiments/src/postings.rs` stub — array/bitmap +containers exist there already; here we build the full machine step by +step — the density crossover, the chunking, the run container, the +pairwise kernels, the SIMD story — reading the production code. + +Every code anchor below is **roaring-rs** — the Rust port +`RoaringBitmap/roaring-rs` at commit `83caaca`, *not* the C library +CRoaring — quoted with the line numbers the code occupies in that +revision (all paths sit under `roaring/src/bitmap/`). Paper figures come +from Chambi, Lemire, Ssi-Yan-Kai & Kaser, "Roaring Bitmaps: +Implementation of an Optimized Software Library" (Software: Practice & +Experience 2018, [arXiv:1709.07821](https://arxiv.org/abs/1709.07821)), +cited by section or page. Where roaring-rs diverges from the paper or +CRoaring — it does in three places below — this guide says so rather than +describing a technique the source does not use. ## The problem in one sentence -Store and intersect sets of u32 IDs: a sorted `Vec` costs 4 bytes -per element and slow intersections, a flat bitmap over the whole u32 -space costs **512 MB no matter how few elements it holds** — and no +Store and intersect sets of u32 IDs cheaply: a sorted `Vec` costs 4 +bytes/element and answers membership by binary search — the topic +headline's **246 ns** point miss ([FINDINGS.md](../../FINDINGS.md) row +26) — while a flat bitmap over the whole u32 space answers in O(1) like +that row's **28 ns** 224 MB HashSet but costs **512 MB no matter how few +elements it holds** (2^32 bits ÷ 8 = 536,870,912 bytes = 512 MiB); no single encoding wins, because density varies wildly across the key space -of any real ID set. +of any real ID set, so Roaring keeps both and picks per 64K chunk. ## The concepts, step by step ### Step 1 — two encodings, one crossover: density decides -For a set of small integers there are two natural representations: a -**sorted array** of the values (cost proportional to how many you store) -and a **bitmap** (one bit per *possible* value — fixed cost, regardless -of how many are present). Over a 16-bit universe (65,536 possible -values), the arithmetic is exact: an array of 16-bit entries costs -2 bytes/element; the bitmap costs a flat 65,536 bits = **8 KB**. They -cross at 8 KB / 2 B = **4,096 elements** — below that the array is -smaller (and the bitmap mostly zeros); above it the bitmap is smaller -(and gives O(1) membership and word-at-a-time set operations for free). -No threshold tuning, pure arithmetic — the same density crossover -GraphBLAS meets at whole-matrix granularity (topic 20). +> **In:** nothing yet — this step fixes the density arithmetic every +> later step reuses. +> **Out:** the array↔bitmap crossover at **4,096 elements**, derived from +> the 8 KB bitmap size; Step 2 applies it per chunk and Step 3 reuses the +> same "which encoding is smaller" test for runs. + +A **container** here is the storage for one 16-bit key range, and it has +two natural encodings. A **sorted array** stores the present values +themselves, so its cost is proportional to how many you store; a +**bitmap** stores one bit per *possible* value, a fixed cost regardless +of how many are present. Over a 16-bit universe (2^16 = 65,536 possible +values) the arithmetic is exact and worth doing once: + +- The bitmap is 2^16 bits = **65,536 bits = 8,192 bytes = 8 KB**, laid + out as **1024 × 64-bit words** (65,536 ÷ 64 = 1024). That is exactly + `BITMAP_LENGTH = 1024` in roaring-rs (`store/bitmap_store.rs:15`), and + the paper's "1024 64-bit words (using 8 kB)" (§ containers, p. 5). +- An array entry is a `u16` = **2 bytes**. +- They cross where the array grows to the bitmap's size: + 8,192 bytes ÷ 2 bytes/element = **4,096 elements**. That is exactly + `ARRAY_LIMIT = 4096` (`container.rs:9`). + +Below 4,096 the array is smaller (and a bitmap would be mostly zeros); at +or above it the bitmap is smaller — and it also buys **O(1) membership** +and word-at-a-time set algebra, the 28 ns HashSet side of the topic +headline, where the array is the 246 ns binary-search side +([FINDINGS.md](../../FINDINGS.md) row 26). No threshold tuning, pure +arithmetic — the same density crossover GraphBLAS meets at whole-matrix +granularity (topic 20). ### Step 2 — chunking: apply the crossover per 64K range +> **In:** the array↔bitmap crossover from Step 1. +> **Out:** the per-chunk container assignment — up to 65,536 chunks, each +> an `Array`, `Bitmap` or `Run` store (`store/mod.rs:28-31`) — that Steps +> 3-4's kernels dispatch on. + Roaring makes the crossover *local*: split the u32 space by the high 16 bits into up to 65,536 chunks, and give each chunk its own **container** holding the members' low 16 bits in whichever encoding is smallest *for -that chunk's density*: +that chunk's density* (paper § containers, p. 2): | container | roaring-rs type | when | size | |---|---|---|---| | array | `ArrayStore` (sorted `Vec`) | card ≤ 4096 | 2 bytes/element | -| bitmap | `BitmapStore` (1024 × u64) | card > 4096 | 8 KB flat | -| run | `IntervalStore` (sorted (start, end) pairs) | few runs | 4 bytes/run | - -Anchors: `store/mod.rs:28-31` (`enum Store { Array, Bitmap, Run }`), -`container.rs:9-11` (`ARRAY_LIMIT = 4096`, `RUN_MAX_SIZE = 2048`), -`container.rs:70` (`ensure_correct_store` — every mutation may -demote/promote). The payoff: a graph with one dense community (bitmap -containers) and a long sparse tail of node IDs (array containers) pays -the right price in *each region* — empty chunks cost nothing at all. +| bitmap | `BitmapStore` (1024 × `u64`) | card > 4096 | 8 KB flat | +| run | `IntervalStore` (sorted `{start, end}` pairs) | few runs | 4 bytes/run | + +The `Store` enum is three variants (`store/mod.rs:28-31`). The threshold +lives in `ARRAY_LIMIT = 4096` (`container.rs:9`). Promotion and demotion +are re-checked on every mutation by `ensure_correct_store` (defined at +`container.rs:225`, called from `insert` at `:70`): its two arms promote +`Array → Bitmap` when `vec.len() > ARRAY_LIMIT` (`:230-231`) and demote +`Bitmap → Array` when `bits.len() <= ARRAY_LIMIT` (`:227-228`). The +payoff: a graph with one dense community (bitmap containers) and a long +sparse tail of node IDs (array containers) pays the right price in *each +region* — and empty chunks cost nothing at all. ### Step 3 — the third container: runs, for clustered data -A **run container** stores maximal intervals as (start, length) pairs — -4 bytes per run — and wins when the data arrives *clustered*: sequential -IDs, time ranges, "all rows in partition". The threshold is the same -arithmetic as Step 1: a run container beats the 8 KB bitmap iff -runs × 4 bytes < 8 KB → `RUN_MAX_SIZE = 2048`. A chunk holding one run -of 60,000 consecutive IDs costs 4 bytes instead of 8 KB. The operational -wrinkle: checking run-worthiness on every insert would be wasteful, so -roaring formats have an explicit `optimize()`/run-conversion pass after -bulk load instead — `insert_range` (`store/mod.rs:107-109`) into a Run -is O(runs); into a Bitmap it's word-fill; into an Array it's a splice. +> **In:** the array/bitmap chunks from Step 2. +> **Out:** the third encoding — the **run container** — and the +> serialized-size rule (not a fixed threshold) that `optimize()` uses to +> choose it, which Step 4's kernels dispatch on as a third shape. + +A **run container** stores maximal runs of consecutive values and wins +when the data arrives *clustered*: sequential IDs, time ranges, "all rows +in a partition". Here roaring-rs **diverges from the paper**: the paper +stores each run as a `(start, length)` pair (§ containers, p. 6), but +roaring-rs's `IntervalStore` stores `Interval { start: u16, end: u16 }` — +an inclusive `{start, end}` pair (`store/interval_store.rs:900-902`). +Either way a run is **4 bytes** (two `u16`s, `RUN_ELEMENT_BYTES = 4` at +`store/interval_store.rs:14`), with a 2-byte run-count header +(`serialized_byte_size = 2 + 4 × runs`, `:39-40`). + +The break-even against the bitmap is the same kind of arithmetic as +Step 1: a run container beats the 8 KB bitmap while its serialized bytes +stay under 8,192 — `2 + 4 × runs < 8192`, i.e. **runs ≤ 2047** (solving +`runs < 2047.5`). A chunk holding one run of 60,000 consecutive IDs is a +single `Interval` — **4 bytes instead of 8 KB**, a 2048× saving. + +Two corrections to the folklore here. First, `RUN_MAX_SIZE = 2048` +(`container.rs:11`) — the round number `8192 ÷ 4` — is **`#[cfg(test)]` +only** (the attribute sits on `:10`); it does *not* drive production +conversion. Second, the real decision lives in `optimize()` +(`container.rs:243`), which compares *actual serialized sizes*: +`BITMAP_BYTES` (8192) against `IntervalStore::serialized_byte_size(num_runs)` +for a bitmap (`:246-251`), and `array.byte_size()` against the run size +for an array (`:254-262`). Checking run-worthiness on every insert would +be wasteful, so this is an explicit post-bulk-load pass, not a per-insert +test — `insert_range` itself (`store/mod.rs:107-109`) just dispatches: +into a `Run` it is O(runs), into a `Bitmap` word-fill, into an `Array` a +splice. ### Step 4 — the density algebra: ops pick kernels pairwise -Every binary set operation dispatches on the container *pair* — 3×3 -kernels, each the natural algorithm for that shape (`store/mod.rs:207-224` -shows the is_disjoint/is_subset matrix; the BitAnd/BitOr impls follow the -same pattern): +> **In:** the three container shapes from Steps 2-3. +> **Out:** the 3×3 kernel dispatch (`is_disjoint` at `store/mod.rs:200`, +> `is_subset` at `:215`), the *linear-merge* array∩array kernel roaring-rs +> actually uses, and the `insert_range` promotion check +> (`container.rs:102-110`) that Step 5 vectorizes. + +Every binary set operation dispatches on the container *pair* — a 3×3 +grid of kernels, each the natural algorithm for that shape. +`store/mod.rs:200-213` (`is_disjoint`) and `:215-227` (`is_subset`) show +the full matrix; the `BitAnd`/`BitOr` impls follow the same pattern: ``` ∩ array ∩ bitmap ∩ run - array merge or GALLOP probe bits per elem probe intervals - bitmap (symmetric) 1024 x (a & b) mask interval spans + array linear merge probe each u16 probe intervals + bitmap (symmetric) 1024 × (a & b) mask interval spans run (symmetric) (symmetric) interval intersection ``` -The galloping case is the one topic 23 met as skip-lists/WAND: -**galloping** (exponential search — probe at strides 1, 2, 4, 8... then -binary-search the bracketed range) exploits size asymmetry: when -|A| ≪ |B|, walk A and gallop through B — O(|A|·log|B|) beats the linear -merge. Same asymmetry-exploiting move as ALEX's exponential search -([reading-learned-indexes.md](reading-learned-indexes.md)) and topic 23's -galloping in `MAXSCORE`. +Here roaring-rs **diverges from CRoaring**, and it is worth being honest +about. CRoaring's array∩array has a *galloping* variant (**galloping** = +exponential search — probe at strides 1, 2, 4, 8… then binary-search the +bracketed range, which beats a linear scan when one side is far smaller). +roaring-rs does **not** gallop: its scalar array∩array is a flat +two-pointer merge that advances one index at a time: ```rust -fn intersect_gallop(small: &[u16], big: &[u16], out: &mut Vec) { - let mut lo = 0; - for &x in small { // |small| ≪ |big| - let mut step = 1; // gallop: 1, 2, 4, 8, ... - while lo + step < big.len() && big[lo + step] < x { step <<= 1; } - let hi = (lo + step + 1).min(big.len()); - match big[lo..hi].binary_search(&x) { // then binary in the bracket - Ok(i) => { out.push(x); lo += i + 1; } - Err(i) => { lo += i; } - } - } // O(|small| · log|big|) -} +// store/array_store/scalar.rs — and(), the array∩array kernel, 37-54 + 37 pub fn and(lhs: &[u16], rhs: &[u16], visitor: &mut impl BinaryOperationVisitor) { + 38 // Traverse both arrays + 39 let mut i = 0; + 40 let mut j = 0; + 41 while i < lhs.len() && j < rhs.len() { + 42 let a = unsafe { lhs.get_unchecked(i) }; + 43 let b = unsafe { rhs.get_unchecked(j) }; + 44 match a.cmp(b) { + 45 Less => i += 1, + 46 Greater => j += 1, + 47 Equal => { + 48 visitor.visit_scalar(*a); + 49 i += 1; + 50 j += 1; + 51 } + 52 } + 53 } + 54 } ``` -One subtlety worth noticing: union of two arrays can overflow -ARRAY_LIMIT, so `container.rs:106` checks -`union_cardinality <= ARRAY_LIMIT` *before* choosing the output -container — question 2 asks why counting first beats build-then-promote. +Line 45 is the one to watch: on `Less` it steps `i` by **one**, not by a +gallop stride. The size-asymmetry win galloping chases still exists in +roaring-rs, but it comes from a *different* kernel — the array-vs-bitmap +case (`store/mod.rs:204-206`), which iterates the small array and does an +**O(1) bit-test per element** into the big bitmap, so the cost is +O(|array|) regardless of how dense the bitmap is (the paper's O(|B₁|) +intersection argument, p. 2). That is the honest mapping of topic 23's +skip-list/WAND galloping onto this code: the same "walk the small side" +idea, realised by container choice rather than by exponential search. + +One subtlety in the *insert* path feeds question 2. Adding a range to an +array chunk can overflow `ARRAY_LIMIT`, so `insert_range`'s array arm +counts the union first: it computes `union_cardinality = array.len() + +added_amount` (`container.rs:102`) and only *then* branches — `== 1<<16` +becomes a full-range `Run` (`:103-104`), `<= ARRAY_LIMIT` stays an array +(`:106-107`), otherwise it promotes to a bitmap before inserting +(`:108-110`). Counting first beats build-then-promote because it never +materialises an over-limit array it would immediately throw away. ### Step 5 — the SIMD story: same kernels, vector width -`array_store/` splits into `scalar.rs` and `vector.rs` — the same -kernels twice, and the module picks at compile time (paper §3). The -paper's two famous kernels: - -- **Array ∩ array**: compare a block of A against a block of B with a - shuffle network; SPE'18 §3.2's `_mm_cmpistrm`-style or the simpler - broadcast-compare. `vector.rs` uses portable `std::simd` — read its - intersect and note the *tail fallback to scalar*. -- **Bitmap card**: population count over 1024 words; the paper's Harley-Seal - AVX2 popcount is why `intersection_len` (`array_store/mod.rs:258`) style - cardinality-only ops never materialize a result container. +> **In:** the pairwise kernels from Step 4. +> **Out:** how roaring-rs vectorizes them — `core::simd` in `vector.rs` +> with a scalar tail fallback — and the two spots where its kernels are +> *not* the paper's. + +`array_store/` splits into `scalar.rs` and `vector.rs` (`mod scalar; mod +vector;` at `array_store/mod.rs:1-2`) — the same kernels twice, picked at +compile time. `vector.rs` is gated `#![cfg(feature = "simd")]` (`:11`) +and built on portable `core::simd` (`:14-15`) with 8-wide `u16x8` lanes; +its `and` is at `:119`. Two honest divergences from the paper's kernels: + +- **Array ∩ array**: the paper (and CRoaring) use the x86 `PCMPESTRM` + string-compare instruction. roaring-rs's own header says it "replaced + [PCMPESTRM] with a simple vector or-shift … what is available through + LLVM intrinsics and is portable" (`vector.rs:6-9`). Read `and` + (`:119`) and note the **tail fallback to `scalar`** for the leftover + elements. +- **Bitmap cardinality**: the paper describes a vectorized Harley-Seal + popcount; roaring-rs does **not** ship one — it sums + `u64::count_ones()` per word (`store/bitmap_store.rs:34`, `:143`), + leaning on the hardware `popcnt` intrinsic the compiler emits. That + per-word popcount is why `intersection_len` (`array_store/mod.rs:258`) + can count matches without materializing a result: it drives a + `CardinalityCounter` visitor (`:259-264`) through `vector::and` / + `scalar::and` and keeps only the tally. Cardinality-only ops (`intersection_len`, `is_disjoint`) are -zero-allocation on purpose — they're the hot path in query *planning* +zero-allocation on purpose — they are the hot path in query *planning* (estimate selectivity before executing, topic 9), where allocating a -result you'll throw away would dominate the cost. +result you will throw away would dominate the cost. ### Step 6 — one idea, three systems: adaptive encodings everywhere -Roaring's promote-on-density-threshold move is not a bitmap trick — it's +> **In:** roaring's promote-on-density move from Steps 1-3. +> **Out:** the same adaptive-encoding pattern in two other systems, and +> the *demotion* question you answer in notes.md. + +Roaring's promote-on-density-threshold move is not a bitmap trick — it is a recurring systems pattern: | | roaring | redis HLL sparse | postgres GIN posting | |---|---|---|---| -| unit | 64K chunk | register stream | TID list segment | -| encodings | array/bitmap/run | ZERO/XZERO/VAL | varbyte deltas | -| promote when | card > 4096 | bytes > 3 KB or rank > 32 | page overflow → posting tree | - -Fill in the *demotion* column yourself: which of the three ever converts -back down, and why is demotion rarer than promotion everywhere? Topic -20's GraphBLAS sparse↔bitmap switch is the same crossover at per-matrix -granularity — the same density arithmetic, measured twice. +| unit | 64K chunk | register run | TID list segment | +| encodings | array/bitmap/run | ZERO/XZERO/VAL opcodes | varbyte deltas | +| promote when | card > 4096 (`container.rs:9`, `:230`) | rank > 32 or bytes > max | page overflow → posting tree | + +The redis numbers are checkable too: HLL's sparse encoding promotes to +dense when a register value exceeds `HLL_SPARSE_VAL_MAX_VALUE = 32` +(`redis src/hyperloglog.c:389`; `if (count > HLL_SPARSE_VAL_MAX_VALUE) +goto promote;` at `:683`) or when the sparse blob would exceed +`server.hll_sparse_max_bytes` (default 3000 ≈ 3 KB; `:863`). Fill in the +*demotion* column yourself: which of the three ever converts back down, +and why is demotion rarer than promotion everywhere? (Roaring's own +`Bitmap → Array` demotion at `container.rs:227-228` is the exception that +makes the rule interesting.) Topic 20's GraphBLAS sparse↔bitmap switch is +the same crossover at per-matrix granularity — the same density +arithmetic, measured twice. ## Where each step lives in the code @@ -152,22 +266,29 @@ containers and the pairwise kernels. | anchor | step | what it is | |---|---|---| | `store/mod.rs:28-31` | 2 | `enum Store { Array, Bitmap, Run }` | -| `container.rs:9-11` | 2–3 | `ARRAY_LIMIT = 4096`, `RUN_MAX_SIZE = 2048` — the two crossovers | -| `container.rs:70` | 2 | `ensure_correct_store` — every mutation may demote/promote | -| `container.rs:106` | 4 | union cardinality checked *before* choosing the output container | +| `container.rs:9` | 1-2 | `ARRAY_LIMIT = 4096` — the array↔bitmap crossover | +| `store/bitmap_store.rs:15` | 1 | `BITMAP_LENGTH = 1024` — the 1024×`u64` = 8 KB bitmap | +| `container.rs:10-11` | 3 | `RUN_MAX_SIZE = 2048` — **`#[cfg(test)]` only**, not the production rule | +| `container.rs:225` | 2 | `ensure_correct_store` — promote `Array→Bitmap` (`:230`), demote `Bitmap→Array` (`:227`) | +| `container.rs:243` | 3 | `optimize` — run conversion by comparing serialized byte sizes | +| `container.rs:102-110` | 4 | `insert_range` array arm: count `union_cardinality` *before* choosing the output container | | `store/mod.rs:107-109` | 3 | `insert_range` per container: O(runs) / word-fill / splice | -| `store/mod.rs:207-224` | 4 | the pairwise dispatch matrix (is_disjoint/is_subset) | -| `store/array_store/scalar.rs` + `vector.rs` | 5 | the same kernels twice; `std::simd` with scalar tail fallback | +| `store/mod.rs:200-227` | 4 | the pairwise dispatch matrix (`is_disjoint` `:200`, `is_subset` `:215`) | +| `store/array_store/scalar.rs:37` | 4 | array∩array = flat two-pointer merge (no galloping) | +| `store/array_store/vector.rs:11` | 5 | `#![cfg(feature="simd")]` `core::simd` port; `and` at `:119`, scalar tail fallback | | `array_store/mod.rs:258` | 5 | `intersection_len` — cardinality-only, zero-allocation | ## Tie back to the stubs -Topic 23's `postings.rs` stub already fixes array↔bitmap promotion at 4096. -After this guide: (a) add the galloping intersect to your mental model of -why FalkorDB label filters should be roaring, not `Vec`; (b) M26's plan -(roaring for label/type filtering) inherits the run container for -"all nodes created in bulk-load order" — measure whether your ID allocator -produces runs. +Topic 23's `postings.rs` stub +(`topics/23-fulltext/experiments/src/postings.rs`) already fixes +array↔bitmap promotion at `ARRAY_MAX = 4096` and uses the same +two-pointer array∩array kernel this guide found in `scalar.rs:37`. After +this guide: (a) hold onto why FalkorDB label filters should be roaring, +not `Vec` — the array-vs-bitmap probe kernel is O(|small side|), not +O(|filter|); (b) M26's plan (roaring for label/type filtering) inherits +the run container for "all nodes created in bulk-load order" — measure +whether your ID allocator actually produces runs before assuming it does. ## Questions to answer in notes.md @@ -191,13 +312,132 @@ produces runs. ## Done when +Answer each before unfolding it. + - [ ] You can state the density crossover and why it is applied per 64K chunk. + +
Answer + + Over a 16-bit key range the bitmap is a fixed 2^16 bits = 8,192 bytes = + 8 KB (`BITMAP_LENGTH = 1024` × `u64`, `store/bitmap_store.rs:15`), and a + sorted array costs 2 bytes per `u16`. They cross at 8,192 ÷ 2 = **4,096 + elements** — exactly `ARRAY_LIMIT = 4096` (`container.rs:9`): below it + the array is smaller, at/above it the bitmap is smaller and O(1) to + probe. + + Roaring applies this *per chunk* — splitting the u32 space by the high + 16 bits into up to 65,536 containers (`Store::{Array,Bitmap,Run}`, + `store/mod.rs:28-31`), each choosing its own encoding via + `ensure_correct_store` (`container.rs:225`, promote at `:230`, demote at + `:227`). Per-chunk adaptivity wins because density varies across the key + space: one dense community lands in a bitmap while a sparse tail stays + in arrays, and empty chunks cost nothing — a single global bitmap would + pay 512 MB regardless. + +
+ - [ ] You can explain what run containers add and which data shape wants them. + +
Answer + + A run container stores maximal runs of consecutive values. In roaring-rs + each run is an `Interval { start: u16, end: u16 }` (an inclusive + `{start, end}` pair, `store/interval_store.rs:900-902`) at 4 bytes — + *diverging from the paper's `(start, length)` encoding*. It wins on + *clustered* data: sequential IDs, time ranges, bulk-loaded partitions. A + single run of 60,000 consecutive IDs is one 4-byte `Interval` instead of + an 8 KB bitmap — a 2048× saving. + + The break-even is `2 + 4 × runs < 8192`, i.e. **runs ≤ 2047**. The + `RUN_MAX_SIZE = 2048` constant (`container.rs:11`) is only its round + approximation, and it is `#[cfg(test)]`-only (`:10`); production instead + picks runs in `optimize()` (`container.rs:243`) by comparing actual + serialized byte sizes — `BITMAP_BYTES` (8192) against `2 + 4 × num_runs` + (`:246-251`). + +
+ - [ ] You can explain the density algebra: kernels chosen pairwise per container type. + +
Answer + + Every binary op dispatches on the container *pair* — a 3×3 grid, shown + by `is_disjoint` (`store/mod.rs:200-213`) and `is_subset` (`:215-227`). + Array∩array is a flat two-pointer merge (`scalar.rs:37-54` — line 45 + steps by one, **not** the galloping/exponential search CRoaring has); + array∩bitmap iterates the small array and does an O(1) bit-test per + element (`store/mod.rs:204-206`), so its cost is O(|array|) whatever the + bitmap's density; bitmap∩bitmap is 1024 word-wise `&`s. + + So the size-asymmetry win in roaring-rs comes from *container choice* + (probe the small side into the big bitmap), not from galloping. Under + the `simd` feature these kernels run 8 `u16` lanes wide in `vector.rs` + (`:11`, `and` at `:119`) with a scalar tail fallback. + +
+ - [ ] You can say what happens when a union overflows the array limit. + +
Answer + + In `insert_range`'s array arm the code counts the union *before* + choosing the output: `union_cardinality = array.len() + added_amount` + (`container.rs:102`), then branches — `== 1<<16` becomes a full-range + `Run` (`:103-104`), `<= ARRAY_LIMIT` stays an array (`:106-107`), + otherwise `self.store = self.store.to_bitmap()` promotes before + inserting (`:108-110`). + + Counting first beats "build an array, promote if it turns out too big" + because the exact union cardinality is cheap (an `intersection_len` + count), whereas building an over-limit array means allocating and + filling storage you would immediately discard on the demote. The steady + state is re-asserted by `ensure_correct_store` (`:225`), which promotes + any array whose `len() > ARRAY_LIMIT` to a bitmap. + +
+ - [ ] You can explain why cardinality-only operations are the hot path and what they skip. + +
Answer + + Query planning estimates selectivity — "how many rows survive this + intersection?" — before deciding a plan (topic 9), so it needs the + *count* of a set operation, not its elements. `intersection_len` + (`array_store/mod.rs:258`) serves that by driving a `CardinalityCounter` + visitor (`:259-264`) through `vector::and` / `scalar::and`: it tallies + matches and returns a `u64`, never allocating a result container. + + What it skips is the output buffer. A full `BitAnd` must materialize the + intersected set; a cardinality-only op keeps a running counter, and for + bitmaps that counter is just `u64::count_ones()` summed per word + (`store/bitmap_store.rs:34`, `:143`) — the hardware `popcnt`, not the + paper's Harley-Seal AVX2 kernel. Allocating a result you would throw + away after reading its length would dominate the cost on this path. + +
+ - [ ] You wrote answers to all questions in notes.md, including the three-adaptive-encodings cross-topic thread with GraphBLAS and HLL. +
Answer + + The four notes.md questions cover: per-chunk vs per-matrix adaptivity + (roaring switches per 64K chunk, GraphBLAS per whole matrix, topic 20); + why counting `union_cardinality` first is cheaper than build-then-promote + (`container.rs:102`); why cardinality-only ops are zero-allocation + (`intersection_len`, `array_store/mod.rs:258`); and the cross-topic + *demotion* column of Step 6's table. + + For that last thread: roaring demotes `Bitmap → Array` when a container + drops to `len() <= ARRAY_LIMIT` (`container.rs:227-228`); redis HLL + never demotes dense → sparse (promotion at rank > 32 or bytes > max, + `src/hyperloglog.c:389`, `:683`, `:863` is one-way); postgres GIN posting + trees do not collapse back to inline lists. Demotion is rarer than + promotion because promotion is forced by a size/precision bound while + demotion is only an optional space reclaim after deletes — which most of + these workloads never do. + +
+ ## References **Papers** @@ -207,6 +447,7 @@ produces runs. containers, §3 SIMD kernels, skim benchmarks **Code** -- [roaring-rs](https://github.com/RoaringBitmap/roaring-rs) - `roaring/src/bitmap/` — the Rust port; `store/` holds the three - containers and the pairwise kernels +- [roaring-rs](https://github.com/RoaringBitmap/roaring-rs) at commit + `83caaca` — the Rust port (**not** CRoaring); `roaring/src/bitmap/` + holds the containers, `store/` the three encodings and the pairwise + kernels diff --git a/topics/27-streaming/README.md b/topics/27-streaming/README.md index 90e7239..000fb69 100644 --- a/topics/27-streaming/README.md +++ b/topics/27-streaming/README.md @@ -35,13 +35,13 @@ Predict the speedup for each of the three before implementing, and predict which one's incremental version is *slower* to initialize than a full recompute — that one is the interesting result, not the 100× one. -## Our motivation numbers first (Apple M3 Pro, 50K nodes / 500K edges, batches of 100 changes, 2026-07-10) +## Our motivation numbers first (Apple M3 Pro, 50K nodes / 500K edges, batches of 100 changes) | standing query | full recompute / batch | incremental target | |---|---|---| -| triangle count | 97.2 ms | ~µs (stub) — batch·d̄ probes, not m·d̄ | -| 2-hop wedge join | 894.3 ms | ~µs-ms (stub) — bilinear delta rule | -| reachability from src | 24.7 ms (re-BFS) | semi-naive: each edge relaxed O(1) times *ever* | +| triangle count | 141.6 ms | ~µs (stub) — batch·d̄ probes, not m·d̄ | +| 2-hop wedge join | 1111.0 ms | ~µs-ms (stub) — bilinear delta rule | +| reachability from src | 31.2 ms (re-BFS) | semi-naive: each edge relaxed O(1) times *ever* | The gap is 3-5 orders of magnitude, and none of it requires cleverness — just refusing to touch data that didn't change. diff --git a/topics/27-streaming/experiments/.gitignore b/topics/27-streaming/experiments/.gitignore index 96ef6c0..ea8c4bf 100644 --- a/topics/27-streaming/experiments/.gitignore +++ b/topics/27-streaming/experiments/.gitignore @@ -1,2 +1 @@ /target -Cargo.lock diff --git a/topics/27-streaming/notes.md b/topics/27-streaming/notes.md index 3817f98..0162f09 100644 --- a/topics/27-streaming/notes.md +++ b/topics/27-streaming/notes.md @@ -6,6 +6,12 @@ Machine: Apple M3 Pro, macOS. `cargo run --release --bin ivm_bench` ## Measured baselines (provided full-recompute lanes — the enemy, priced) +> An earlier run than the one the README's opening lane and +> [FINDINGS.md](../../FINDINGS.md) row 27 report (**triangle 141.6 ms, +> wedge 1111.0 ms, re-BFS 31.2 ms**). Where the two disagree, FINDINGS is +> canonical; cite one run or the other by name and never average them. +> Re-run `./verify.sh 27` before treating any cell below as current. + | standing query | full recompute / batch | notes | |---|---|---| | triangle count | 97.2 ms | O(m·d̄) sorted-intersect sweep; count 1366 after last batch | diff --git a/topics/27-streaming/reading-dbsp.md b/topics/27-streaming/reading-dbsp.md index 7276026..4a75970 100644 --- a/topics/27-streaming/reading-dbsp.md +++ b/topics/27-streaming/reading-dbsp.md @@ -11,15 +11,22 @@ Rust implementation, where each operator of the calculus is a file. ## The problem in one sentence The topic bench recomputes a 2-hop wedge join from scratch in -**894.3 ms per 100-change batch**; DBSP's claim is that for *any* query -built from its operators, the version that costs per-change instead of -per-database is not designed but *derived* — by one definition and a -handful of rewrite rules. +**1111.0 ms per 100-change batch** (this topic's measured headline — +`../../FINDINGS.md` row 27, reproduced in `README.md`'s "The problem, +measured" lane); DBSP's claim is that for *any* query built from its +operators, the version that costs per-change instead of per-database is +not designed but *derived* — by one definition and a handful of rewrite +rules. ## The concepts, step by step ### Step 1 — Z-sets: make deletion a first-class value +> **In:** a collection plus a batch of table changes (rows inserted, +> rows deleted). **Out:** one *value* — a Z-set — in which the sign of +> each element's weight encodes insert (+) vs delete (−), and which is +> closed under addition and negation (an abelian group). + A **Z-set** is a collection where every element carries an integer weight — weight +2 means "present twice," weight −1 means "one copy removed" — so a batch of table changes is itself a value: inserts are @@ -34,6 +41,11 @@ zset.rs `distinct_is_not_linear` test pokes at — is question 2.) ### Step 2 — streams and the four operators +> **In:** a stream — one Z-set per logical clock tick (a transaction's +> changes, or a full snapshot). **Out:** the four circuit primitives +> (z⁻¹, I, D, and a lifted query Q) plus the inversion identity +> D(I(s)) = I(D(s)) = s that makes I and D mutually inverse. + A **stream** is an infinite sequence of values, one per logical clock tick — a function ℕ→group, where each tick's value is a Z-set (one transaction's worth of changes, or one snapshot). Circuits are built @@ -42,20 +54,40 @@ from exactly four operators: ``` z^-1 delay (one-tick memory) operator/z1.rs:221 Z1 I integrate: running sum operator/integrate.rs:85 - D differentiate: x[t] - x[t-1] operator/differentiate.rs:38 + D differentiate: a - z^-1(a) operator/differentiate.rs:38 Q any query, lifted pointwise ``` -`z^-1` outputs its input one tick late (the only stateful primitive — -one value of memory). `I` turns a stream of deltas into a stream of -accumulated states (running sum). `D` turns states back into deltas. -"Lifted" means an ordinary query Q applied independently at every tick. -The two identities that everything hangs on: **D(I(x)) = x and -I(D(x)) = x** — integrate and differentiate are mutually inverse, which -only works because Step 1 gave us subtraction. +The paper pins each down precisely (§2). **Delay** (Def 2.5): +z⁻¹(s)[0] = 0 and z⁻¹(s)[t] = s[t−1] for t ≥ 1 — output the input one +tick late (the only stateful primitive, one value of memory). +**Differentiation** (Def 2.17): D(s) := s − z⁻¹(s), so D(s)[t] = +s[t] − s[t−1] — feldera writes this comment verbatim at +`differentiate.rs:31` (`differentiate(a) = a - z^-1(a)`): + +```rust +// feldera crates/dbsp/src/operator/differentiate.rs +30 /// Computes the difference between current and previous value +31 /// of `self`: `differentiate(a) = a - z^-1(a)`. +38 pub fn differentiate(&self) -> Stream { +``` + +**Integration** (Def 2.19, Prop 2.20): I(s)[t] = Σ_{i≤t} s[i] — the +running sum; feldera's `integrate.rs:80` documents it with the same +example, `input 1,1,1,1,1… → output 1,2,3,4,5…`. Worked on s = id = +[0 1 2 3 4 …]: D(id) = [0 1 1 1 1 …] (each tick minus the last) and +I(id) = [0 1 3 6 10 …] (partial sums). "Lifted" means an ordinary query +Q applied independently at every tick. The identity everything hangs on +is the paper's **Theorem 2.22 (inversion): I(D(s)) = D(I(s)) = s** — +integrate and differentiate are mutually inverse, which only works +because Step 1 gave us subtraction. ### Step 3 — incrementalization, defined in one line +> **In:** any query Q that maps a full state to a full view. **Out:** its +> change-to-change version, *defined* as Q^Δ := D ∘ Q ∘ I (Def 3.1) — +> correct by construction, ruinous if run literally. + The incremental version of any query is *defined* as **Q^Δ = D ∘ Q ∘ I**: integrate the input deltas back into full states, run the ordinary query on each state, differentiate the outputs back @@ -67,24 +99,49 @@ Q^Δ until the Is and Ds vanish or shrink — Step 4. ### Step 4 — the rewrite rules: push I and D through the query -Three rules do almost all the work: +> **In:** Q^Δ = D ∘ Q ∘ I, with the expensive I and D wrapped around Q. +> **Out:** an equivalent circuit in which I and D are pushed inward until +> only small per-operator state survives — nothing for linear operators, +> two delayed integrals for a join, one integral for a nonlinear operator. + +Three theorems do almost all the work; quote them as the paper states +them (§3): ``` - linear Q: Q^Δ = Q (deltas stream through) - bilinear join: (A⋈B)^Δ = ΔA⋈I(B) + I(A)⋈ΔB + ΔA⋈ΔB - ^ the z^-1-delayed integrals = arrangements - chain rule: (Q1∘Q2)^Δ = Q1^Δ ∘ Q2^Δ (incrementalize COMPOSITIONALLY) + linear (Thm 3.3): Q^Δ = Q for LTI Q + bilinear (Thm 3.4): (a×b)^Δ = a×b + z^-1(I(a))×b + a×z^-1(I(b)) + chain (Prop 3.2): (Q1∘Q2)^Δ = Q1^Δ ∘ Q2^Δ incrementalize COMPOSITIONALLY ``` **Linear** operators (map, filter, flat_map, union — those that -distribute over addition) are their own incremental versions: deltas -stream straight through, zero state. The **bilinear** join (linear in -each input separately) needs exactly two pieces of state — the integrals -of its inputs, one of them delayed — which are precisely differential's -arrangements. As an operator — note the state is exactly two integrals, -one of them delayed (`z^-1`): +distribute over addition) are their own incremental versions (Thm 3.3): +deltas stream straight through, zero state. The **bilinear** join is +linear in each input separately, and Theorem 3.4 is exact — note it uses +the *delayed* integrals `z⁻¹(I(a))` and `z⁻¹(I(b))`, i.e. each input's +accumulated state *as of the previous tick*. The paper then rewrites it +into "the familiar formula for incremental equi-joins," Δ(a×b) = +Δa×Δb + a×Δb + Δa×b, where `a`,`b` are the accumulated relations. Those +delayed integrals are precisely differential's arrangements, which is +why `djoin.rs:43` insists "A, B are the states BEFORE the deltas." + +Worked example (scalar × as the bilinear op, to keep the arithmetic in +view). Let the change streams be a = [2 3 1 …], b = [5 1 4 …]. Then +I(a) = [2 5 6 …], z⁻¹(I(a)) = [0 2 5 …], z⁻¹(I(b)) = [0 5 6 …]. Theorem +3.4 per tick: + +- t=0: 2·5 + 0·5 + 2·0 = **10** +- t=1: 3·1 + 2·1 + 3·5 = **20** +- t=2: 1·4 + 5·4 + 1·6 = **30** + +Cross-check against the definition Q^Δ = D∘Q∘I: I(a)·I(b) = +[10 30 60], and D of that = [10 20 30]. Same stream — the theorem is the +definition with the I/D pushed through the multiply. As an operator, the +state is exactly two integrals, one delayed (`z⁻¹`): ```rust +// ILLUSTRATION — shape of Thm 3.4 as a stepping operator; +// the real bilinear delta lives in experiments/src/djoin.rs:43 +// and feldera crates/dbsp/src/operator/join.rs:350 (join_generic). struct IncJoin { ia: ZSet, ib_delayed: ZSet } // I(A), z^-1(I(B)) fn step(&mut self, da: &ZSet, db: &ZSet) -> ZSet { @@ -97,11 +154,27 @@ fn step(&mut self, da: &ZSet, db: &ZSet) -> ZSet { } ``` -Nonlinear operators (distinct, count, sum, top-k, min/max) keep their -integral — that stored I(input) is the state, and it's *all* the state. +The genuinely **nonlinear** operators keep their integral — that stored +I(input) is the state, and it's *all* the state. `distinct` is the +paper's canonical example (Def 4.3; its incremental form is derived +specially in Prop 4.7, still O(|change|)). The order-sensitive +aggregates `min`/`max`/`top-k` are nonlinear because deleting the current +survivor (the current maximum) forces the operator to consult the +runner-up. Even SQL `GROUP BY` `SUM`/`COUNT` keep state: the underlying +summation is linear and "automatically incremental" on its own (§7.4), +but emitting the grouped *relation* `(key, aggregate)` composes it with +the nonlinear `makeset` step, so a group's output row must be retracted +and re-emitted when its aggregate changes — "the count function… is not +linear since it uses the makeset non-linear function" (§7.4). The +`zset.rs` `distinct_is_not_linear` test pins the operator the whole +scheme turns on (question 2). ### Step 5 — the chain rule: why this covers a whole SQL dialect +> **In:** a composite query Q1 ∘ Q2. **Out:** (Q1 ∘ Q2)^Δ = Q1^Δ ∘ Q2^Δ +> (Prop 3.2) — incrementalize each primitive once, and a whole dialect +> falls out for free. + The chain rule — (Q1∘Q2)^Δ = Q1^Δ ∘ Q2^Δ — is the paper's practical bombshell: incrementalization is **compositional**, so you incrementalize operator-by-operator, and a whole SQL dialect (joins, @@ -115,25 +188,38 @@ operator, DBSP gets as a theorem. ### Step 6 — recursion: nested circuits instead of lattice times +> **In:** a recursive (fixpoint) query — e.g. transitive closure. +> **Out:** a nested circuit: δ₀ introduces an inner stream, an inner loop +> iterates the query to fixpoint within one outer tick, and ∫ reads the +> fixpoint back out — no partially-ordered timestamps required. + DBSP handles recursion by nesting: an inner circuit with its own clock runs to fixpoint *within* each outer tick (`DelayedFeedback`, z1.rs:37, -wires the cycle; `delta0.rs` injects an outer-clock stream into the -inner circuit — the paper's δ₀). Same expressive result as -differential's lattice timestamps, but staged — outer tick, then inner -fixpoint — rather than a general product order. The trade (question 3): -DBSP gives up mixing epochs mid-iteration and out-of-order input within -a tick; it gains engineering simplicity and clean per-tick transactional -semantics — Feldera's "synchronous circuit" story. +wires the cycle; `delta0.rs:22` is feldera's counterpart of the paper's +**δ₀** stream-introduction operator — "the delta function… δ₀(v)[t] = v +for t=0, else 0" (§5), which imports a parent-circuit value as the +inner stream at inner-time 0; the dual **∫** reads the fixpoint back +out). Same expressive result as differential's lattice timestamps, but +staged — outer tick, then inner fixpoint — rather than a general product +order. The trade (question 3): DBSP gives up mixing epochs mid-iteration +and out-of-order input within a tick; it gains engineering simplicity +and clean per-tick transactional semantics — Feldera's "synchronous +circuit" story. ### Step 7 — what the calculus buys a database +> **In:** the finished calculus (operators, Q^Δ, the rewrite theorems). +> **Out:** three database-grade payoffs — per-tick transactions, state +> that is nothing but integrals (so checkpointing is trivial), and the +> FalkorDB/M27 delta-matrix mapping. + - **Per-tick transactions**: each input Z-set batch = one transaction; outputs are exactly the view deltas for that transaction. This is the contract M27's standing Cypher queries want: mutation batch in, result delta out, push to subscribers. - **State = integrals**: every stateful operator's memory is I(something), spillable to storage (feldera's `storage/` crate) — checkpointing is - checkpointing integrals, nothing else (z1.rs's `CommittedZ1` :241). + checkpointing integrals, nothing else (z1.rs's `CommittedZ1` :231). - **The FalkorDB mapping (M27)**: delta matrix DP−DM is ΔA for one tick; `wait` = I. A standing pattern query is Q; what M27 must build is Q^Δ — masked SpGEMM terms ΔA·A + A·ΔA + ΔA·ΔA instead of recomputing A² @@ -152,7 +238,7 @@ semantics — Feldera's "synchronous circuit" story. | `operator/join.rs:123/:283/:350` | 4 | `join`, `stream_join_generic`, `join_generic` — the ^Δ forms specialized | | `operator/distinct.rs`, `aggregate.rs` | 4 | the nonlinear ops, each carrying its integral | | `operator/delta0.rs` | 6 | injects an outer-clock stream into a nested circuit — the paper's δ₀ | -| `operator/z1.rs:241` `CommittedZ1` | 7 | checkpointing integrals | +| `operator/z1.rs:231` `CommittedZ1` | 7 | checkpointing integrals | Paper route: read §1–4 (the algebra — Steps 1–5) with the operator table open; read §5 (recursion — Step 6) if the differential guide left @@ -183,12 +269,62 @@ questions about what nesting trades against lattice times. ## Done when -- [ ] You can explain why Z-sets make deletion a first-class value and why sets cannot. -- [ ] You can name the four operators and write incrementalization in one line. -- [ ] You can prove the bilinear rule by expanding `Q^Δ = D∘Q∘I`. -- [ ] You can explain the chain rule and why it covers a whole dialect rather than one query. -- [ ] You can say how recursion is handled by nested circuits. -- [ ] You wrote answers to all questions in notes.md, including the wedge count — which this topic measures at 1111.0 ms per batch under full recompute. +Answer each before unfolding it. + +- [ ] Why do Z-sets make deletion a first-class value, and why can't plain sets? +
answer + + Z-sets attach an integer weight to every element and form an abelian + group, so a deletion is just adding the element with weight −1 and every + value has a negation. Plain sets have no subtraction — there is no set + that "removes" another — so a change and a collection cannot share a + type. Group structure is the precondition for I and D (Step 1). + +
+- [ ] Name the four operators and write incrementalization in one line. +
answer + + z⁻¹ (delay), I (integrate, running sum), D (differentiate, a − z⁻¹(a)), + and a lifted query Q. Incrementalization is Q^Δ := D ∘ Q ∘ I (Def 3.1). + +
+- [ ] Prove the bilinear rule by expanding `Q^Δ = D∘Q∘I`. +
answer + + Expand D(I(a)×I(b))[t] = I(a)[t]×I(b)[t] − I(a)[t−1]×I(b)[t−1]. Write + I(a)[t] = I(a)[t−1] + a[t] and likewise for b, multiply out, and the + cross terms collect into Theorem 3.4: a×b + z⁻¹(I(a))×b + a×z⁻¹(I(b)). + The delayed integrals z⁻¹(I(·)) are why the code keeps *delayed* traces + (the arrangements / states-before-the-deltas). + +
+- [ ] Explain the chain rule and why it covers a whole dialect, not one query. +
answer + + (Q1∘Q2)^Δ = Q1^Δ ∘ Q2^Δ (Prop 3.2), proved by inserting I∘D = id + between the two stages. So you give each primitive its ^Δ form once and + compose; no per-query delta derivation is ever needed — that is what + Feldera's SQL-to-circuit compiler exploits. + +
+- [ ] Say how recursion is handled by nested circuits. +
answer + + δ₀ introduces an inner stream from an outer value (§5); an inner loop + with its own clock and a z⁻¹ back-edge iterates the query to fixpoint + within one outer tick; ∫ reads the fixpoint back out. No + partially-ordered timestamps — the nesting is staged, outer then inner. + +
+- [ ] You wrote answers to all questions in notes.md, including the wedge count. +
answer + + This topic measures the full-recompute wedge join at 1111.0 ms per + 100-change batch (`../../FINDINGS.md` row 27 / README measured lane). + Your notes should carry the DBSP circuit for the wedge count and mark + which arrows carry deltas vs integrals. + +
## References diff --git a/topics/27-streaming/reading-differential-dataflow.md b/topics/27-streaming/reading-differential-dataflow.md index 4a0a970..6b1d299 100644 --- a/topics/27-streaming/reading-differential-dataflow.md +++ b/topics/27-streaming/reading-differential-dataflow.md @@ -14,13 +14,19 @@ iterate) that our topic-27 stubs are simplified excerpts of. Delete one edge from a 500K-edge graph and a maintained reachability view must retract every fact derived *through* that edge — across however many BFS rounds derived them, while other facts re-derive via -surviving paths — without falling back to the 24.7 ms full re-BFS our -insert-only stub would need. +surviving paths — without falling back to the 31.2 ms full re-BFS our +insert-only stub would need (this topic's measured reachability lane — +`../../FINDINGS.md` row 27 / `README.md` "The problem, measured"). ## The concepts, step by step ### Step 1 — the delta discipline: streams of weighted, timestamped updates +> **In:** a changing collection (a table under inserts, deletes, +> updates). **Out:** a stream of `(data, time, diff)` updates whose +> implicit collection at time t is the sum of all updates at times ≤ t — +> kept canonical by consolidation (sort, sum diffs, drop zeros). + A differential **Collection** is not a table — it is a stream of `(data, time, diff)` updates: the record, the logical timestamp it changed at (Naiad's lattice time, from the timely guide), and an integer @@ -37,6 +43,11 @@ inserts, deletes, and updates — no per-operator retraction logic. ### Step 2 — arrangements: the indexed update log, shared and compacted +> **In:** a stream of updates that operators need to look up by key. +> **Out:** an `Arranged` collection whose trace is an LSM-of-batches +> index of `(key, val, time, diff)`, shared by reference across every +> operator that reads that key, and compacted against the frontier. + Operators like join need to look up "all updates for key k" — so differential builds **arrangements**: `arrange` (operators/arrange/arrangement.rs:311, core at :336) turns an update @@ -61,6 +72,11 @@ collapse and their diffs consolidate, bounding state. ### Step 3 — the incremental join: the bilinear rule on traces, with fuel +> **In:** two arranged, changing inputs A and B. **Out:** the output +> delta ΔA⋈B + A⋈ΔB + ΔA⋈ΔB, computed by joining each new batch against +> the *other* input's trace — work metered by a fuel loop so a large +> delta never stalls the worker. + The join of two changing inputs updates by the product rule — new output = ΔA⋈B + A⋈ΔB + ΔA⋈ΔB — and `join_traces` (operators/join.rs:69) is that rule executed against arrangements: each input is arranged; when a @@ -75,6 +91,11 @@ Cooperative scheduling at the operator level: topic 7's lesson, again. ### Step 4 — iteration: lattice timestamps make recursion retractable +> **In:** a recursive loop body (e.g. BFS relaxation) over changing +> input. **Out:** every derived fact stamped with an **(outer, round)** +> lattice time, so deleting an input edge retracts exactly the +> round-and-epoch-dependent facts it produced — no support counting. + This is where differential earns its name. `iterate` (operators/iterate.rs:192 `Variable`, `set` :262) runs a loop body inside a nested scope where every update carries an **(outer, round)** @@ -88,18 +109,29 @@ counting, no over-deletion bug — the two failure modes every hand-rolled incremental-recursion scheme hits. This is the machinery our insert-only `reach.rs` deliberately lacks (the topic README's scope cut). -`examples/bfs.rs:101-107` is the whole algorithm: +`examples/bfs.rs:98-109` is the whole algorithm (real code, pinned at +`3f279da` — the closure takes `(scope, inner)`, and the final `reduce` +keeps the minimum distance, it is not a `...min...` placeholder): ```rust -nodes.iterate(|inner| { - inner.join_map(&edges, |_k, l, d| (*d, l + 1)) // relax - .concat(&nodes) // keep roots - .reduce(...min...) // keep shortest -}) +// differential-dataflow/examples/bfs.rs @3f279da +98 let nodes = roots.map(|x| (x, 0)); +101 nodes.clone().iterate(|scope, inner| { +103 let nodes = nodes.enter(scope); +104 let edges = edges.enter(scope); +106 inner.join_map(edges, |_k,l,d| (*d, l+1)) // relax: one hop +107 .concat(nodes) // keep roots +108 .reduce(|_, s, t| t.push((*s[0].0, 1))) // keep shortest +109 }) ``` ### Step 5 — semi-naive evaluation falls out for free +> **In:** a recursive query run round by round. **Out:** semi-naive +> behavior — each round joins only the *newly derived* diffs against the +> full relation — with no special code, because unchanged facts emit no +> updates. + Semi-naive evaluation — the classic Datalog optimization of joining only the *newly derived* facts against the full relation each round, instead of re-joining everything — is not implemented anywhere in differential; @@ -112,6 +144,12 @@ representation itself — question 3 asks you to line the two up. ### Step 6 — what the generality costs, and what it buys +> **In:** the topic's three stubs (delta_join, IncrementalTriangles, +> SemiNaiveReach) — differential with the general machinery deleted. +> **Out:** a clear ledger of what the real system pays (arrangements, +> lattice times, compaction) and what that buys (retractions inside +> recursion — the one thing the stubs cannot do). + Our three stubs are differential with the general machinery deleted: `delta_join` = join_traces without times/fuel; `IncrementalTriangles` = a 3-way delta join specialized by hand; `SemiNaiveReach` = iterate for @@ -133,7 +171,7 @@ problem here and an open one in most hand-built IVM systems. | `operators/arrange/arrangement.rs:311` (core :336), `Arranged` :45 | 2 | update stream → shared trace (LSM of batches) | | `operators/join.rs:69` `join_traces`; `Deferred` :311; fuel :348, :355-395 | 3 | the bilinear rule against traces, work-metered | | `operators/iterate.rs:192` `Variable`, `set` :262 | 4 | nested scope, (outer, round) timestamps | -| `examples/bfs.rs:101-107` | 4–5 | 40 lines that do what our reach.rs stub cannot | +| `examples/bfs.rs:98-109` | 4–5 | 12 lines that do what our reach.rs stub cannot | Paper route: the CIDR '13 paper is short — read all of it, twice. First pass after Steps 1–3 (collections, arrangements as "indexed @@ -162,12 +200,62 @@ bug you can now name. ## Done when -- [ ] You can explain the delta discipline: weighted, timestamped updates. -- [ ] You can explain what an arrangement is and why sharing one across queries matters. -- [ ] You can explain the incremental join as the bilinear rule on traces, and what "fuel" is for. -- [ ] You can explain why lattice timestamps are required for retractable recursion, not merely convenient. -- [ ] You can show how semi-naive evaluation falls out for free. +Answer each before unfolding it. + +- [ ] Explain the delta discipline: weighted, timestamped updates. +
answer + + A collection is a stream of `(data, time, diff)` updates; the collection + at time t is the sum of all updates at times ≤ t and never materializes + except inside arrangements. One consolidation path (sort, sum diffs, + drop zeros) handles inserts, deletes, and updates uniformly. + +
+- [ ] What is an arrangement, and why does sharing one across queries matter? +
answer + + An arrangement is an indexed, compacted trace (an LSM of `(key, val, + time, diff)` batches) built by `arrange`. It is shared by reference, so + two queries joining the same collection on the same key reuse one + index instead of each building its own — Materialize's main memory win. + +
+- [ ] Explain the incremental join as the bilinear rule on traces, and what "fuel" is for. +
answer + + `join_traces` computes ΔA⋈B + A⋈ΔB + ΔA⋈ΔB by joining each new batch + against the other input's trace up to the frontier. Fuel meters the + work (`Deferred` state, effort accounting) so a huge delta yields + cooperatively instead of stalling the worker. + +
+- [ ] Why are lattice timestamps *required* for retractable recursion, not merely convenient? +
answer + + Each derived fact carries an (outer-epoch, round) time. Deleting an + input edge must retract exactly the facts derived through it at each + round while facts re-derived by surviving paths persist. A total order + can't keep a mid-flight iteration from epoch 1 separate from a new + change at epoch 2; the product order can, so retractions stay exact. + +
+- [ ] Show how semi-naive evaluation falls out for free. +
answer + + At round r+1 the join's inputs are exactly the diffs produced at round + r, because unchanged facts emit no updates. So the "join only the new + facts" discipline is a consequence of the update representation, not a + hand-written optimization. + +
- [ ] You wrote answers to all questions in notes.md, including the ordering issue in `IncrementalJoin::step`. +
answer + + The batch of A must join B's trace as of the frontier *before* B's + matching delta is folded in (and vice versa); fold both first and the + ΔA⋈ΔB cross term is counted twice. Record the correct order and why. + +
## References @@ -179,5 +267,5 @@ bug you can now name. - [differential-dataflow](https://github.com/TimelyDataflow/differential-dataflow) `differential-dataflow/src/` — `consolidation.rs`, `operators/arrange/arrangement.rs`, `operators/join.rs`, - `operators/iterate.rs`; plus `examples/bfs.rs` — 40 lines that do + `operators/iterate.rs`; plus `examples/bfs.rs` — a dozen lines that do what our reach.rs stub cannot diff --git a/topics/27-streaming/reading-kafka-log.md b/topics/27-streaming/reading-kafka-log.md index 06bf198..c585d40 100644 --- a/topics/27-streaming/reading-kafka-log.md +++ b/topics/27-streaming/reading-kafka-log.md @@ -22,6 +22,11 @@ changed since. ### Step 1 — the log: an append-only sequence where position is identity +> **In:** a stream of records to persist for many readers. **Out:** an +> append-only partition file in which each record's identity is its +> **offset** (its position) — no per-message id, no broker-side index, +> no mutation. + A log is a file (conceptually) that is only ever appended to, where each record's identity is simply its **offset** — its position in the sequence. No per-message IDs, no broker-side index, no mutation: @@ -41,6 +46,11 @@ which Step 2 turns into the whole consumer model. ### Step 2 — dumb broker, smart consumer +> **In:** many independent consumers reading the same partition at +> different rates. **Out:** a broker that stores no per-consumer state — a +> consumer *is* a `(partition, offset)` pair it holds itself; rewind and +> replay are just resetting that integer. + The broker keeps NO per-consumer state: a consumer *is* a (partition, offset) pair, stored by the consumer itself, and "consume" means "read forward from my offset." Rewind = set the integer back; @@ -54,18 +64,30 @@ broker doesn't know or care how many there are. ### Step 3 — the mechanical bet: sequential IO and the OS page cache +> **In:** the need to serve high-volume reads and writes off disk +> cheaply. **Out:** sequential appends, no in-process message cache (lean +> on the OS page cache), and `sendfile` zero-copy delivery — cheap enough +> to retain days of history so replay stays economical. + Kafka's performance design is to *not have one*: writes are sequential appends (the fastest thing a disk does — topic 0's ~100× sequential vs random gap), there is no in-process message cache (the OS page cache already caches the segment files — topic 6's "don't fight the OS" lesson, chosen deliberately), and delivery to consumers uses -**sendfile** (a zero-copy syscall that moves bytes from file to socket -without passing through userspace). The consequence that matters -downstream: a log this cheap can retain days of history, which is what -makes Step 2's "replay from anywhere" economical rather than theoretical. +**sendfile** (a zero-copy syscall that, the paper notes in §3.1, "avoids +2 of the copies and 1 system call" of the four copies and two syscalls a +naive send would make). The consequence that matters downstream: a log +this cheap can retain days of history — the paper's retention SLA is +"typically 7 days" (§3.1) — which is what makes Step 2's "replay from +anywhere" economical rather than theoretical. ### Step 4 — ordering per partition only +> **In:** the question of how much ordering a maintained view needs. +> **Out:** order guaranteed *within* a partition and nothing across +> partitions — route each key to a fixed partition, so per-partition +> order is per-key order, which is all correctness requires. + Kafka guarantees order *within* a partition and nothing across partitions — because a total order across partitions would cost coordination (topic 15), and state maintenance doesn't need it: what @@ -78,18 +100,31 @@ question topic 15 asks of replication, answered minimally. ### Step 5 — delivery semantics: it's all about where the offset lives +> **In:** a consumer that can crash mid-processing. **Out:** the delivery +> guarantee — at-most-once, at-least-once, or exactly-once — determined +> solely by where the consumed offset is stored and whether that store +> commits atomically with the output. + With a dumb broker, delivery guarantees degrade to one question: **where do you store your consumed offset, and is that store transactional with your output?** Offset stored before processing → at-most-once (crash -loses a message); after → at-least-once (crash duplicates). The only -real "exactly-once" is consumer-side: commit the offset *atomically -with* the derived output — an idempotent or transactional sink. -RisingWave's barrier checkpoint is exactly this recipe (offsets stored -IN the same checkpoint as operator state — question 1); so is every -"exactly-once" system you'll meet. +loses a message); after → at-least-once (crash duplicates). Kafka's own +default is the middle one — the paper states plainly that "Kafka only +guarantees at-least-once delivery," since "exactly-once delivery +typically requires two-phase commits" (§3.3). The only real +"exactly-once" is consumer-side: commit the offset *atomically with* the +derived output — an idempotent or transactional sink. RisingWave's +barrier checkpoint is exactly this recipe (offsets stored IN the same +checkpoint as operator state — question 1); so is every "exactly-once" +system you'll meet. ### Step 6 — log compaction: the log becomes a table changelog +> **In:** a topic whose time-based retention would discard history a new +> consumer still needs. **Out:** compaction that keeps the *latest record +> per key* (plus tombstones for deletes), turning the topic into a table +> changelog a late-joining consumer can bootstrap a full table from. + Retention by time throws away history a new consumer needs; **log compaction** instead retains *the latest record per key*, turning a topic into a table changelog that a late-joining consumer can bootstrap @@ -104,6 +139,10 @@ about the deletion at all (question 2). ### Step 7 — the ideology: turn the database inside out +> **In:** the classic stack, app → DB → CDC → caches. **Out:** the log is +> the database and tables are caches of log prefixes — write to the log +> first and derive *everything*, the DB included, as consumers. + Kreps' thesis, distilling the paper: **the log is the database; tables are caches of log prefixes.** Instead of app → DB → CDC → caches, write to the log first and derive *everything* — the DB included — as @@ -128,16 +167,23 @@ reading-materialize-risingwave.md fixes that with timestamps). The paper is 7 pages — read the whole thing, watching for the four bets: -- **§3 (architecture + storage)** — Steps 1–3: segment files, - offset-as-identity, page cache + sendfile. Notice what is *absent*: no +- **§3.1 (efficiency on a single partition)** — Steps 1, 3: segment + files, offset-as-identity, page cache + `sendfile`, the "stateless + broker" decision (the offset is held by the consumer, not the broker), + and the "typically 7 days" retention SLA. Notice what is *absent*: no broker index, no message cache, no ack bookkeeping. -- **§3.2 (consumer)** — Step 2: the pull model and consumer-held - offsets; §4's delivery-semantics discussion is Step 5 in 2011 - vocabulary (compaction, Step 6, came later — read its design in the - Kreps blog). -- **§4–5 (coordination + numbers)** — Step 4's per-partition ordering - and the throughput comparisons; the numbers are dated, the ratios - (sequential append vs per-message ack) aren't. +- **§3.2 (distributed coordination)** — Step 2/Step 4: consumer groups, + ZooKeeper-mediated offset ownership across many consumers, and why + ordering is per-partition. +- **§3.3 (delivery guarantees)** — Step 5: the paper commits only to + at-least-once and explains why exactly-once is left to the consumer + side. (Compaction, Step 6, is *not* in this paper — it came later; + read its design in the Kreps blog.) +- **§5 (experimental results)** — the throughput comparison (e.g. a + producer sustaining ~50,000 msg/s at batch size 1 and ~400,000 msg/s + at batch size 50 against ActiveMQ); the numbers are dated, the ratios + (sequential append vs per-message ack) aren't. §4 is LinkedIn + deployment context, not the mechanics. Then the Kreps blog ("The Log", 2013) — the ideology of Step 7, read after the paper so the architecture claims have mechanics under them. @@ -169,13 +215,72 @@ after the paper so the architecture claims have mechanics under them. ## Done when -- [ ] You can explain why position is identity in an append-only log. -- [ ] You can explain the dumb-broker/smart-consumer split and what it moves to the client. -- [ ] You can state the mechanical bet: sequential IO plus the OS page cache. -- [ ] You can explain why ordering is per partition only and what that forbids. -- [ ] You can say where the offset lives for each delivery semantic. -- [ ] You can explain log compaction as turning a topic into a table changelog. +Answer each before unfolding it. + +- [ ] Why is position identity in an append-only log? +
answer + + Records are only ever appended, so a record's offset (its position in + the partition) never changes and uniquely names it. No separate + per-message id or broker-side index is needed — "where was I?" is one + integer. + +
+- [ ] Explain the dumb-broker/smart-consumer split and what it moves to the client. +
answer + + The broker stores no per-consumer state (§3.1 "stateless broker"); the + consumer holds its own `(partition, offset)`. This moves progress + tracking, rewind, and replay to the client and lets any number of + independent consumers read the same log without broker bookkeeping. + +
+- [ ] State the mechanical bet: sequential IO plus the OS page cache. +
answer + + Writes are sequential appends; there is no in-process message cache + (the OS page cache serves segment files); delivery uses `sendfile`, + which the paper says avoids 2 of 4 copies and 1 of 2 syscalls (§3.1). + Cheap enough to retain ~7 days of history. + +
+- [ ] Why is ordering per partition only, and what does that forbid? +
answer + + A total order across partitions would need coordination and isn't + required: correctness only needs same-key updates not to reorder. Route + each key to a fixed partition and per-partition order is per-key order. + It forbids relying on a global order across keys/partitions. + +
+- [ ] Where does the offset live for each delivery semantic? +
answer + + Store the offset before processing → at-most-once; after → at-least-once + (Kafka's own guarantee, §3.3). Exactly-once requires committing the + offset atomically with the output (idempotent/transactional sink) — a + consumer-side property, not a broker one. + +
+- [ ] Explain log compaction as turning a topic into a table changelog. +
answer + + Compaction retains the latest record per key (with tombstones for + deletes for a grace period), so a late consumer can read the compacted + prefix to reconstruct the current table, then follow the live tail. + (This is a post-2011 feature — see the Kreps blog, not the paper.) + +
- [ ] You wrote answers to all questions in notes.md, including what FalkorDB's existing log already gives M27. +
answer + + FalkorDB already has a log (Redis replication / AOF, topic 5). The open + choice for M27 is whether standing-query subscribers consume the raw + mutation log (Kafka-style rebuild) or per-query result deltas + (Materialize SUBSCRIBE-style), and what the server must persist for a + disconnected subscriber — the Step 6 retention-window trade. + +
## References diff --git a/topics/27-streaming/reading-materialize-risingwave.md b/topics/27-streaming/reading-materialize-risingwave.md index c27a318..efb2312 100644 --- a/topics/27-streaming/reading-materialize-risingwave.md +++ b/topics/27-streaming/reading-materialize-risingwave.md @@ -22,6 +22,11 @@ three in opposite directions. ### Step 1 — what production adds to the theory +> **In:** the calculus's rule "keep an integral per nonlinear operator." +> **Out:** the three systems decisions it leaves open — state placement +> (RAM vs object storage), the consistency unit (which input prefix an +> output reflects), and index sharing plus recovery. + An IVM engine in production is the delta algebra plus three systems decisions. **State placement**: every nonlinear operator's integral (join state, aggregate counts) must live somewhere with a cost — @@ -35,6 +40,11 @@ Everything in the two codebases below is one of these three, answered. ### Step 2 — Materialize: indexes are arrangements are memory +> **In:** SQL view definitions. **Out:** differential dataflows whose +> "indexes" *are* differential arrangements pinned in RAM and shared by +> every query that can use them — durability delegated to persist, +> consistency inherited from timely's frontiers. + Materialize's bet is to change as little theory as possible: the compute layer (`src/compute/src/render/`) compiles SQL plans into differential dataflows, and its signature idea is **indexes are arrangements are @@ -51,6 +61,12 @@ serializable. ### Step 3 — delta joins: the bilinear rule scaled to n inputs +> **In:** an n-way incremental join. **Out:** n dataflows, each starting +> from one input's changes and looking up the other n−1 inputs' *already +> existing* arrangements — the bilinear rule generalized so no +> intermediate arrangements are built, with per-path timestamping to stop +> double-counting. + An n-way incremental join done as a binary tree needs an arrangement for every *intermediate* result — state that exists only to serve the join. Materialize's "dogs^3" **delta joins** @@ -59,15 +75,22 @@ dataflows, each starting from one input's changes and looking up the other n−1 inputs' *existing* arrangements — the bilinear rule generalized so NO intermediate arrangements are built. The correctness subtlety is double-counting: the n paths must not each claim the same -joint update, so `half_join` (:315, and the newer `half_join2` :402) -time-stamps lookups — ΔA joins B's arrangement *as of the time just -before* the delta — our stub's "state BEFORE the delta" rule, industrial -edition. The cost: delta joins need an arrangement per input per join -key, so they're chosen when those arrangements already exist (question 1 -maps this onto topic 10's "interesting orders"). +joint update, so `build_halfjoin` (:325, dispatching to the newer +default `build_halfjoin2` :380) time-stamps lookups — ΔA joins B's +arrangement *as of the time just before* the delta — our stub's "state +BEFORE the delta" rule, industrial edition. The cost: delta joins need +an arrangement per input per join key, so they're chosen when those +arrangements already exist (question 1 maps this onto topic 10's +"interesting orders"). ### Step 4 — RisingWave: hand-written executors, state in an LSM on S3 +> **In:** the same relational operators, but no differential core. +> **Out:** hand-written incremental executors, each managing explicit +> schema'd state tables in Hummock (a shared LSM over object storage); +> Z-set weights ride as an `Op` enum, and retraction is hand-rolled per +> operator via degree tables. + RisingWave's bet is the opposite: no differential core, no general delta algebra — each relational operator is a hand-written incremental executor (`src/stream/src/executor/`) that manages *explicit, schema'd @@ -79,14 +102,20 @@ with Update split into paired Delete+Insert so downstream operators never need "modify". Where differential gets retraction from diff arithmetic, RisingWave hand-rolls it per operator: `HashJoinExecutor` (hash_join.rs:158) keeps both sides' rows in state -tables plus **degree tables** (:117 `need_degree_table`, :269) tracking -match counts, so outer joins can retract their NULL rows when the last -match leaves. What the per-operator schemas buy: state that is legible -to S3 spill, per-key TTL, and elastic scaling of a *single* operator -(question 2). +tables plus **degree tables** (:118 `need_degree_table`, +`degree_state_table_l` :269) tracking match counts, so outer joins can +retract their NULL rows when the last match leaves. What the per-operator +schemas buy: state that is legible to S3 spill, per-key TTL, and elastic +scaling of a *single* operator (question 2). ### Step 5 — barriers: consistency and recovery by checkpoint +> **In:** a distributed dataflow that must stay consistent and recover +> after a crash. **Out:** Chandy-Lamport **barriers** per epoch — +> two-input operators align on them, each operator flushes its state +> tables to Hummock at a barrier, and recovery reloads the last +> checkpoint and replays the source log since it. + RisingWave's consistency unit is the **barrier** — a Chandy-Lamport-style marker injected at sources that flows through the dataflow with the data. Two-input operators align on barriers before emitting @@ -101,6 +130,11 @@ subsumed by "know which input prefix your output reflects." ### Step 6 — the comparison that matters for M27 +> **In:** the two systems' opposite bets on state, consistency, and +> sharing. **Out:** the axis-by-axis table below, and the observation +> that a single-writer graph engine gets the hard parts free — no barrier +> alignment (one clock), no distributed frontier (one writer). + | axis | Materialize | RisingWave | M27 (FalkorDB standing queries) | |---|---|---|---| | delta algebra | diffs everywhere (differential) | Op enum per chunk | delta matrices (DP/DM) | @@ -123,7 +157,7 @@ Materialize — Steps 2–3 | anchor | what it is | |---|---| | `render/join/delta_join.rs:47` | "dogs^3" delta-query joins: an n-way join becomes n dataflows, each starting from one input's changes — the bilinear rule generalized so NO intermediate arrangements are built | -| `delta_join.rs:315/:402` | `half_join` construction (and the newer `half_join2`): ΔA against B's arrangement, time-stamped so the n paths don't double-count — our stub's "state BEFORE the delta" rule, industrial edition | +| `delta_join.rs:325/:380` | `build_halfjoin` construction (dispatching to the newer default `build_halfjoin2`): ΔA against B's arrangement, time-stamped so the n paths don't double-count — our stub's "state BEFORE the delta" rule, industrial edition | | `render/reduce.rs` | the nonlinear ops, each with its arrangement | | `src/compute/src/arrangement/` | arrangement sharing across dataflows — one index, many standing queries | | `src/persist-client/` | the durable shard log: compute is stateless-ish; state rehydrates from persist (topic 28's disaggregation, applied to IVM) | @@ -137,9 +171,9 @@ RisingWave — Steps 4–5 | anchor | what it is | |---|---| | `common/src/array/stream_chunk.rs:45` | `enum Op { Insert, Delete, UpdateDelete, UpdateInsert }` — Z-set weights as a protocol; Update split into paired Delete+Insert so downstream operators never need "modify" | -| `stream/src/executor/hash_join.rs:158` | `HashJoinExecutor`: both sides' rows in state tables; `need_degree_table` :117 + degree tables :269 track match counts so outer joins can retract NULLs correctly — hand-rolled weight bookkeeping | +| `stream/src/executor/hash_join.rs:158` | `HashJoinExecutor`: both sides' rows in state tables; `need_degree_table` :118 + degree state table `degree_state_table_l` :269 track match counts so outer joins can retract NULLs correctly — hand-rolled weight bookkeeping | | `executor/barrier_align.rs` | two-input operators align on barriers before emitting — the consistency unit | -| `executor/aggregate/`, `top_k/` | each nonlinear op = explicit state table schema in Hummock | +| `executor/aggregate/`, `top_n/` | each nonlinear op = explicit state table schema in Hummock | ## Questions to answer in notes.md @@ -162,12 +196,66 @@ RisingWave — Steps 4–5 ## Done when -- [ ] You can say what production adds to the theory, in failure modes rather than features. -- [ ] You can explain Materialize's identity: indexes are arrangements are memory. -- [ ] You can explain delta joins and why they need an arrangement per input per key. -- [ ] You can contrast RisingWave's hand-written executors plus LSM-on-S3 state with that. -- [ ] You can explain what barriers give you for consistency and recovery. +Answer each before unfolding it. + +- [ ] What does production add to the theory, in failure modes rather than features? +
answer + + Three decisions the calculus leaves open: state placement (RAM + evaporates on crash, object storage survives but costs ms), the + consistency unit (outputs must reflect the same input prefix), and + sharing/recovery (don't keep 1000 copies of an index; rebuild state on + restart). Each is a way the system can go wrong, not a feature. + +
+- [ ] Explain Materialize's identity: indexes are arrangements are memory. +
answer + + A Materialize index is a differential arrangement pinned in RAM and + shared by every query that can use it, so capacity planning is + arrangement accounting. Durability is delegated to persist; consistency + comes from timely frontiers. + +
+- [ ] Explain delta joins and why they need an arrangement per input per key. +
answer + + An n-way join becomes n dataflows, each driven by one input's changes + and looking up the other inputs' existing arrangements — no intermediate + state. That requires each input to already be arranged on the join key, + and per-path timestamping so the n paths don't double-count a joint + update. + +
+- [ ] Contrast RisingWave's hand-written executors plus LSM-on-S3 state with that. +
answer + + No differential core: each operator is hand-written with explicit + schema'd state tables in Hummock (LSM on S3). Weights ride as the `Op` + enum; retraction is hand-rolled (e.g. degree tables for outer joins). + The schemas buy S3 spill, per-key TTL, and single-operator elastic + scaling. + +
+- [ ] Explain what barriers give you for consistency and recovery. +
answer + + A barrier is a per-epoch Chandy-Lamport marker: two-input operators + align on it, and each flushes state to Hummock when it has the barrier + from all inputs — a globally consistent checkpoint. Recovery reloads the + checkpoint and replays the source since it; the checkpoint interval is + the replay window. + +
- [ ] You wrote answers to all questions in notes.md, including the degree-table against diff-arithmetic comparison. +
answer + + Both retract the outer-join NULL when the last match leaves: RisingWave + with a per-operator degree table and code, differential with one + consolidation rule for every operator. RisingWave trades generality for + state that is legible to spill, TTL, and per-operator scaling. + +
## References @@ -179,5 +267,5 @@ RisingWave — Steps 4–5 (`doc/developer/` — skim "formalism" and "platform") - [risingwave](https://github.com/risingwavelabs/risingwave) `src/` — stream executors: `src/stream/src/executor/` (hash_join.rs, - barrier_align.rs, aggregate/, top_k/); the Op enum: + barrier_align.rs, aggregate/, top_n/); the Op enum: `common/src/array/stream_chunk.rs:45`; Hummock state store diff --git a/topics/27-streaming/reading-naiad-timely.md b/topics/27-streaming/reading-naiad-timely.md index 53bb8de..09aa00c 100644 --- a/topics/27-streaming/reading-naiad-timely.md +++ b/topics/27-streaming/reading-naiad-timely.md @@ -21,6 +21,11 @@ ordered clock. ### Step 1 — dataflow: the program is a graph, data does the moving +> **In:** a computation you want to run in parallel and incrementally. +> **Out:** a directed graph of stateful operators connected by channels — +> records flow in at sources and out at sinks, with no global controller, +> so parallelism and incremental re-execution both fall out of the shape. + A dataflow system represents a computation as a directed graph of **operators** (small stateful functions: map, join, count) connected by channels; input records flow in at sources and results flow out at @@ -36,6 +41,11 @@ covers all three, if the messages carry the right notion of time. ### Step 2 — logical timestamps: every message says which batch it belongs to +> **In:** messages flowing through the graph, processed out of order for +> speed. **Out:** each message tagged with a **logical timestamp** — the +> epoch it derives from (§2.1) — so "results for epoch 7" stays a +> well-defined set even while epochs 8 and 9 are in flight. + In timely dataflow every message carries a **logical timestamp** — not a wall-clock time but a coordinate naming the unit of input it derives from, starting with the **epoch** (which round of input the external @@ -48,6 +58,11 @@ in flight. ### Step 3 — the completeness problem: frontiers +> **In:** an operator (count, min) that must not emit epoch t's final +> answer while a message with timestamp ≤ t might still arrive. +> **Out:** the **frontier** — the proven statement "no message with +> timestamp ≤ t will ever arrive at this input" — computed, not guessed. + An operator like count or min cannot emit a *final* answer for time t while any message with timestamp ≤ t might still arrive — emitting early means emitting wrong. The system-wide statement "no message with @@ -60,20 +75,45 @@ output (emit per closed timestamp). Contrast the industry norm: Flink / MillWheel-style watermarks are *heuristics* ("probably no events older than t−5s") that can be violated by stragglers; timely frontiers cannot. -### Step 4 — the protocol: could-result-in, and progress as a refcount - -The frontier is computed by counting, per (location, timestamp), the -outstanding **pointstamps** — evidence that a message *at* that time and -place exists or could still be produced. Naiad §3.2: a pointstamp is in -the frontier when no other outstanding pointstamp **could-result-in** it -(reachability through the graph combined with timestamp order — operator -A at time t could-result-in B at t' if a message at (A, t) could cause -one at (B, t')). Every produced message increments a count, every -consumed one decrements — progress is just a distributed refcount over -the lattice. The frontier advance, mechanically — progress is count -arithmetic: +### Step 4 — the protocol: could-result-in, and two counts per pointstamp + +> **In:** the set of unprocessed events in the running dataflow. +> **Out:** the frontier, computed by tracking, per active **pointstamp**, +> an *occurrence count* (outstanding events) and a *precursor count* +> (active pointstamps that could-result-in it); a pointstamp is in the +> frontier exactly when its precursor count is zero (§2.3). + +The frontier is computed from **pointstamps** — a pointstamp is a +`(timestamp t, location l ∈ Edge ∪ Vertex)` pair naming a place-and-time +where an event exists or could still be produced (Naiad §2.3). The graph +induces an order: pointstamp `(t1,l1)` **could-result-in** `(t2,l2)` iff +some path ψ through the graph adjusts t1 (via ingress/egress/feedback) +so that `Ψ[l1,l2](t1) ≤ t2` — i.e. a message at `(t1,l1)` could cause one +at `(t2,l2)`. The scheduler keeps a set of **active** pointstamps and, +for each, **two** counts (this is the part a plain refcount misses): + +- an **occurrence count** — how many outstanding events bear the + pointstamp. Updated by the vertex methods: `SENDBY` +1, `ONRECV` −1, + `NOTIFYAT` +1, `ONNOTIFY` −1. A pointstamp leaves the active set when + its occurrence count hits zero. +- a **precursor count** — how many active pointstamps precede it in the + could-result-in order. When p becomes active its precursor count is + initialized to the number of active pointstamps that could-result-in + it, and it increments the precursor count of everything it + could-result-in; when p leaves, it decrements them. + +A pointstamp is **in the frontier** precisely when its *precursor count +is zero* — "there is no other pointstamp in the active set that +could-result-in p" (§2.3) — and only then may its notification be +delivered. The occurrence count decides when a pointstamp is done; the +precursor count decides when it is *safe*. The `MutableAntichain` +(`frontier.rs:380`, `update_iter` :533) is the code that maintains this +and reports which minimal times appeared or vanished: ```rust +// ILLUSTRATION — the frontier-advance step, sketched; the real +// count arithmetic is timely progress/frontier.rs:533 (update_iter) +// over the (time, ±count) buffer progress/change_batch.rs:16. fn apply(counts: &mut BTreeMap, changes: &[(Time, i64)]) -> Vec { - Empty, - TCell1(EventTime, A), // one event: no allocation - TCellCap(SVM), // few: small-vector map - TCellN(BTreeMap), // many: real tree -} +// raphtory-core/src/storage/timeindex.rs:12 @ 5d0d286 — WHEN an entity existed +12 #[derive(Default, Debug, Clone, Serialize, Deserialize, PartialEq)] +13 pub enum TimeIndex { +14 #[default] +15 Empty, +16 One(T), +17 Set(BTreeSet), +18 } ``` -The in-source comment says it plainly: "TCells represent a value in -time that can be set at multiple times and keeps a history" — a -property is a timeline, not a value; reading it *requires* saying at -what time. Why it matters: per-entity time indexes are what make a -window a probe instead of a replay — `BETWEEN t1 AND t2` on one node -is a range query on its TimeIndex, not a scan of the global log. +Then property values over time — note the in-source comment on line 9 +states the contract outright: + +```rust +// raphtory-core/src/entities/properties/tcell.rs:8 @ 5d0d286 + 8 #[derive(Debug, PartialEq, Default, Clone, Serialize, Deserialize)] + 9 // TCells represent a value in time that can be set at multiple times and keeps a history +10 pub enum TCell { +11 #[default] +12 Empty, +13 TCell1(EventTime, A), // one event: stored inline, no allocation +14 TCellCap(SVM), // a few: SVM = small-vector map, inline array +15 TCellN(BTreeMap), // many: a real balanced tree +16 } +``` + +A property is a timeline, not a value; reading it *requires* saying at +what time. **SVM** here is a *small-vector map* — an association list +kept in a stack-inlined array while it stays short, so a property that +changes a handful of times pays no heap allocation and no tree +overhead. Worked ladder: a node with **1** existence event is +`TimeIndex::One` (16 bytes, no alloc); a property set **once** is +`TCell1` (inline); one edited **~8** times is `TCellCap` (one small +inline array); one edited **thousands** of times spills to `TCellN`'s +`BTreeMap`, paying `O(log n)` lookup only where the history actually +warrants it. + +Why it matters: per-entity time indexes are what make a window a probe +instead of a replay — `BETWEEN t1 AND t2` on one node is a range query +on its TimeIndex, not a scan of the global log. ### Step 4 — properties: columnar log + time→offset index -Property *values* don't live inside the TCell — they live in a -columnar log, and the TCell maps time to an offset into it: +> **In:** the `TCell` timeline from Step 3. +> **Out:** the split between the *time* index (small, per-entity) and the +> *value* log (dense, columnar) that Step 5's scans stream over. + +Property *values* don't live inside the TCell — they live in a columnar +log, and the TCell stores an *offset* into it, so the timeline indexes +`Option` (an offset) rather than the value itself: ```rust -// raphtory-core/src/entities/properties/tprop.rs:22 -pub struct TPropCell<'a> { - t_cell: Option<&'a TCell>>, // time → offset - // ... new(t_cell, log: Option<&PropColumn>) at :28 -} +// raphtory-core/src/entities/properties/tprop.rs:21 @ 5d0d286 +21 #[derive(Copy, Clone, Debug, Default)] +22 pub struct TPropCell<'a> { +23 t_cell: Option<&'a TCell>>, // time → offset +24 log: Option<&'a PropColumn>, // the columnar value store +25 } +26 // new(t_cell, log: Option<&PropColumn>) at tprop.rs:28 wires the two together ``` -So one property read is: probe the TCell for the newest EventTime ≤ t, -get a `usize`, index the PropColumn. Why it matters: this splits the -two access patterns cleanly — temporal navigation stays in small -per-entity indexes (cache-friendly, Step 3's enums), while values sit -in dense columns (topic 12's layout) that scans and analytics can -stream. It's the same time-vs-payload separation AeonG gets from KV -key-vs-value, done in-memory and columnar. +Both fields are borrows and the struct is `Copy`: a `TPropCell` is a +*pair of pointers* — the per-entity time index (`t_cell`) and the shared +column (`log`) — not owned data. So one property read is: probe the +`t_cell` for the newest EventTime ≤ t, get a `usize` offset, index the +`PropColumn`. + +Why it matters: this splits the two access patterns cleanly — temporal +navigation stays in small per-entity indexes (cache-friendly, Step 3's +enums), while values sit in dense columns (topic 12's layout) that +scans and analytics can stream. It's the same time-vs-payload +separation AeonG gets from KV key-vs-value, done in-memory and +columnar. ### Step 5 — WindowedGraph + TimeOps: views as composable zero-copy lenses +> **In:** the per-entity time indexes of Steps 3–4, which make time +> filtering cheap per node. +> **Out:** a graph-level `WindowedGraph` view and the `TimeOps` algebra +> that turns M33's `FOR TT` clauses into ordinary constructors. + A **view** is a struct that wraps a graph and reinterprets every read -through a filter — here, a time filter: +through a filter — here, a time filter — and it is `Copy`: ```rust -// raphtory/src/db/graph/views/window_graph.rs:87 — derives Copy, Clone -pub struct WindowedGraph { - pub graph: G, - pub start: Option, - pub end: Option, -} +// raphtory/src/db/graph/views/window_graph.rs:85 @ 5d0d286 +85 /// A struct that represents a windowed view of a `Graph`. +86 #[derive(Copy, Clone)] +87 pub struct WindowedGraph { +88 /// The underlying `Graph` object. +89 pub graph: G, +90 /// The inclusive start time of the window. +91 pub start: Option, +92 /// The exclusive end time of the window. +93 pub end: Option, +94 } ``` -It derives `Copy`: a BETWEEN view is two optional timestamps wrapping -the graph, nothing copied. The `TimeOps` trait -(`raphtory/src/db/api/view/time.rs:116`) declares -`fn window(&self, start, end) -> -Self::WindowedViewType` with default impls (~:245), and every view -type — graph, node, edge — implements it, so views *compose*: a window -of a window intersects the ranges; `at(t)` is a degenerate window. -Downstream, even existence is windowed — an edge's presence is an -iterator over its addition times per layer, not a boolean -(`additions_iter`/`additions`, -`raphtory-storage/src/graph/edges/edge_storage_ops.rs:110,:140`). Why -it matters: this is AT TIME/BETWEEN done as *algebra* — M33's `FOR -TT`-style clauses become constructors of a view type the whole query -engine already runs on, instead of a special mode threaded through -every operator. +It derives `Copy`: a BETWEEN view is the wrapped graph handle plus two +`Option` fields (start inclusive, end exclusive), nothing in +the log copied. The `TimeOps` trait declares `window` and `at` — the +former takes two `IntoTime` bounds, the latter one, and `at(t)` is a +degenerate window: + +```rust +// raphtory/src/db/api/view/time.rs:115 @ 5d0d286 +115 /// Create a view including all events between `start` (inclusive) and `end` (exclusive) +116 fn window(&self, start: T1, end: T2) -> Self::WindowedViewType; +117 +118 /// Create a view that only includes events at `time` +119 fn at(&self, time: T) -> Self::WindowedViewType; +``` + +Every view type — graph, node, edge — implements `TimeOps` (default +impls around `time.rs:245`), so views *compose*: a window of a window +intersects the ranges. Downstream, even existence is windowed — an +edge's presence is an iterator over its addition times per layer, not a +boolean (`additions_iter` at +`raphtory-storage/src/graph/edges/edge_storage_ops.rs:110`, `additions` +at `:140`). + +Why it matters: this is AT TIME/BETWEEN done as *algebra* — M33's +`FOR TT`-style clauses become constructors of a view type the whole +query engine already runs on, instead of a special mode threaded +through every operator. ### Step 6 — where it's going: db4 segments and Cypher -The workspace tells you the roadmap: `raphtory-cypher` runs Cypher -over these temporal views (the same "bolt a query language onto a time -model" move as AeonG's `FOR TT`), and `db4-graph` + `db4-storage` are -a newer segmented storage engine — see `pub struct MemEdgeSegment` -(`db4-storage/src/segments/edge/segment.rs:58`) — replacing per-entity -allocations with segment-grained storage. Why it matters: the pure -event-log model is allocation-heavy at scale for the same reason -memgraph is (Step 3's per-entity enums are still per-entity objects); -segments are the "batch it into arrays" correction — the recurring -arc of this whole learning path. +> **In:** the per-entity model of Steps 3–5, which is elegant but +> allocation-heavy at scale. +> **Out:** the roadmap crates (`raphtory-cypher`, `db4-*`) that batch it +> into segments — the recurring "arrays beat objects" arc of this path. + +The workspace tells you the roadmap: `raphtory-cypher` runs Cypher over +these temporal views (the same "bolt a query language onto a time +model" move as AeonG's `FOR TT`), and `db4-graph` + `db4-storage` are a +newer segmented storage engine: + +```rust +// db4-storage/src/segments/edge/segment.rs:57 @ 5d0d286 +57 #[derive(Debug)] +58 pub struct MemEdgeSegment { +59 layers: Vec>, +60 est_size: usize, +``` + +It replaces per-entity allocations with segment-grained storage. Why it +matters: the pure event-log model is allocation-heavy at scale for the +same reason Memgraph is (Step 3's per-entity enums are still per-entity +objects); segments are the "batch it into arrays" correction — the +recurring arc of this whole learning path. ## Where each step lives in the code -All paths relative to `~/repos/raphtory`. Workspace crates: `raphtory` -(main API), `raphtory-api`, `raphtory-core`, `raphtory-storage`, -`raphtory-cypher`, `raphtory-graphql`, `db4-graph` + `db4-storage`. +All paths relative to `~/repos/raphtory` at `5d0d286`. Workspace crates: +`raphtory` (main API), `raphtory-api`, `raphtory-core`, +`raphtory-storage`, `raphtory-cypher`, `raphtory-graphql`, `db4-graph` ++ `db4-storage`. | Step | Anchor | What to see | |---|---|---| -| 2 | `raphtory-api/src/core/storage/timeindex.rs:28` | `EventTime(pub i64, pub usize)` — the universal key | +| 2 | `raphtory-api/src/core/storage/timeindex.rs:28` | `EventTime(pub i64, pub usize)` — the universal 16-byte key (derive at :27) | | 3 | `raphtory-core/src/storage/timeindex.rs:13` | `TimeIndex { Empty, One, Set }` — when an entity existed | -| 3 | `raphtory-core/src/entities/properties/tcell.rs:10` | `TCell` enum ladder + the "value in time" comment | -| 4 | `raphtory-core/src/entities/properties/tprop.rs:22` | `TPropCell` — TCell holds time→offset into a `PropColumn` (`new` at :28) | -| 5 | `raphtory/src/db/graph/views/window_graph.rs:87` | `WindowedGraph{graph, start, end}`, derives `Copy` | -| 5 | `raphtory/src/db/api/view/time.rs:116` | `TimeOps::window` declaration; default impls ~:245 | +| 3 | `raphtory-core/src/entities/properties/tcell.rs:10` | `TCell` enum ladder + the "value in time" comment at :9 | +| 4 | `raphtory-core/src/entities/properties/tprop.rs:22` | `TPropCell { t_cell, log }` — time→offset index + `PropColumn` (`new` at :28) | +| 5 | `raphtory/src/db/graph/views/window_graph.rs:87` | `WindowedGraph{graph, start, end}`, derives `Copy` (at :86) | +| 5 | `raphtory/src/db/api/view/time.rs:116` | `TimeOps::window` decl; `at` at :118; default impls ~:245 | | 5 | `raphtory-storage/src/graph/edges/edge_storage_ops.rs:110,:140` | `additions_iter` / `additions` — edge existence as an iterator | | 6 | `db4-storage/src/segments/edge/segment.rs:58` | `MemEdgeSegment` — the newer segmented engine | -Read order: EventTime → TCell (read the whole enum and its comment) → -TPropCell → WindowedGraph → TimeOps (trace `window` from declaration -to one default impl) → skim a db4 segment. Resist reading more; these -eight anchors are the design. +Read order: EventTime → TimeIndex → TCell (read the whole enum and its +comment) → TPropCell → WindowedGraph → TimeOps (trace `window` from +declaration to one default impl) → skim a db4 segment. Resist reading +more; these eight anchors are the design. ## Questions (answer in notes.md) -1. M33: what would a `WindowedGraph` over FalkorDB's GraphBLAS - matrices be? Two timestamps can't lazily filter a dense SpMV — do - BETWEEN views become masks (topic 20), materialized submatrices, or +1. M33: what would a `WindowedGraph` over FalkorDB's GraphBLAS matrices + be? Two timestamps can't lazily filter a dense SpMV — do BETWEEN + views become masks (topic 20), materialized submatrices, or per-operation time predicates, and what does each cost? 2. The experiments crate's `events.rs::replay_at_time` answers AT TIME - by replaying the log from t=0. Which Raphtory structures replace + by replaying the log from t = 0. Which Raphtory structures replace the replay, and what is the probe cost per node in their terms (Step 3)? -3. EventTime's `usize` tiebreaker vs the λ=0 tie-order stream from Wu +3. EventTime's `usize` tiebreaker vs the λ = 0 tie-order stream from Wu et al. (exercise 2 in the README): show how a total order on events makes the one-pass earliest-arrival deterministic where bare i64 timestamps aren't. -4. Contrast with memgraph (topic 13): both end up with per-entity +4. Contrast with Memgraph (topic 13): both end up with per-entity small-then-spill collections (small_vector vs TCell's ladder), yet - one is object-first and one log-first. What query does each answer - in O(1) that costs the other a scan? + one is object-first and one log-first. What query does each answer in + O(1) that costs the other a scan? 5. Raphtory has no GC question — nothing is ever superseded — but that means the log only grows. Steal AeonG's vocabulary: what would an - "anchor" be in an event-log-first engine, and where would you put - it? (Hint: your `snapshot.rs` is exactly this hybrid.) + "anchor" be in an event-log-first engine, and where would you put it? + (Hint: your `snapshot.rs` is exactly this hybrid.) ## Done when -You can trace, naming the concrete types at each hop, what -`g.window(t1, t2).node(n).properties()` touches — WindowedGraph → -TimeOps → TimeIndex range → TCell probe → PropColumn offset — and say -which single hop `at(t)` on a never-updated node skips (TCell1: no -tree, no allocation). +Answer each before unfolding it. + +- [ ] You can trace `g.window(t1, t2).node(n).properties()` naming the concrete type at each hop. + +
Answer + + `window(t1, t2)` constructs a **`WindowedGraph`** (window_graph.rs:87) + — the graph handle plus `start`/`end: Option`, `Copy`, nothing + copied. `.node(n)` yields a node view still carrying those bounds. + `.properties()` reads through a **`TPropCell`** (tprop.rs:22): it probes + the node's **`TCell`** timeline (tcell.rs:10) for the newest + `EventTime ≤ t2` (and `≥ t1`), which is a range query on a per-entity + index — `TimeIndex`/`TCell` from Step 3 — yielding an `Option` + offset, then indexes the shared **`PropColumn`** (`log`) at that offset. + Time navigation stays in small per-entity structures; only the value + read touches the dense column. + +
+ +- [ ] You can say which hop `at(t)` on a never-updated node skips, and why it costs nothing. + +
Answer + + A node whose property was set exactly once stores it as + **`TCell1(EventTime, A)`** (tcell.rs:13) — the value is inline in the + enum, no `SVM` array and no `BTreeMap`. So `at(t)` skips the + `TCellN::BTreeMap` `O(log n)` seek entirely: there is one event, it is + either `≤ t` or not, an `O(1)` inline check with zero allocation. The + size-adaptive ladder means the common case (power-law: most entities + have one event) pays nothing. + +
+ +- [ ] You can explain why a BETWEEN view copies zero bytes of the graph. + +
Answer + + `WindowedGraph` (window_graph.rs:86–93) derives `Copy` and holds only + the wrapped `graph: G` (itself a handle/reference in practice) plus two + `Option` fields — `start` inclusive, `end` exclusive. A window + is therefore ~two 16-byte timestamps beside the handle, constructed by + value; the append-only event log is never duplicated. Filtering happens + lazily at read time by comparing each event's `EventTime` against the + bounds (Step 5), so `BETWEEN` is an algebra of cheap wrappers, not a + materialization. + +
+ +- [ ] You can name the tiebreaker that makes equal-timestamp events reproducible, and where it lives. + +
Answer + + `EventTime(pub i64, pub usize)` (timeindex.rs:28): the second field is a + per-event `usize` sequence id, and because `Ord` is derived on the tuple + struct after the i64, comparison is lexicographic — equal timestamps fall + back to the event id. Every `BTreeSet`/`BTreeMap` in the engine + (`TimeIndex::Set`, `TCellN`) keys on `EventTime`, so all of them order + colliding events identically. That is the type-system answer to Wu et + al.'s λ = 0 tie-order requirement (reading-temporal-paths.md): the + one-pass algorithms need a deterministic order among same-`t` contacts, + and Raphtory makes bare i64 collisions impossible to observe. + +
## References **Code** - [Raphtory](https://github.com/Pometry/Raphtory) — cloned at - `~/repos/raphtory`; the eight anchors above are the read -- [memgraph](https://github.com/memgraph/memgraph) — topic 13 clone; - the object-first pole to hold this against + `~/repos/raphtory`, pinned `5d0d286` (resources/codebases.md); the eight + anchors above are the read. Re-verify any anchor with + `python3 tools/pinned-source.py show raphtory -r A:B`. +- [memgraph](https://github.com/memgraph/memgraph) — topic 13 clone; the + object-first pole to hold this against. - This topic's `experiments/src/events.rs` (`replay_at_time`) and - `snapshot.rs` — the naive and anchor+delta baselines Raphtory's - indexes replace + `snapshot.rs` — the naive and anchor+delta baselines Raphtory's indexes + replace. **Related guides** - [reading-aeong.md](reading-aeong.md) — the object-first counterpoint - built on memgraph + built on Memgraph. - [reading-temporal-paths.md](reading-temporal-paths.md) — Wu et al.'s - one-pass algorithms, whose tie-order subtlety EventTime solves in - the type system + one-pass algorithms, whose tie-order subtlety EventTime solves in the + type system. diff --git a/topics/33-temporal-graphs/reading-temporal-motifs.md b/topics/33-temporal-graphs/reading-temporal-motifs.md index c20eb66..ac65b7f 100644 --- a/topics/33-temporal-graphs/reading-temporal-motifs.md +++ b/topics/33-temporal-graphs/reading-temporal-motifs.md @@ -1,28 +1,50 @@ # δ-temporal motifs: counting ordered patterns inside a time window -Topic 24 counted static triangles with a masked matrix multiply; the -previous guide showed that once edges carry timestamps, *order* is -information. This paper fuses the two: a pattern is no longer a subgraph -but an ordered sequence of edges that must all land inside a window δ — -and counting those is a new algorithmic problem. This chapter builds the -six concepts, ending at the exact window-scan operator M33's WITHIN δ -needs. +Topic 24 counted static triangles with a masked matrix multiply; the previous +guide showed that once edges carry timestamps, *order* is information. This +paper fuses the two: a pattern is no longer a subgraph but an ordered sequence +of edges that must all land inside a window δ — and counting those is a new +algorithmic problem. This chapter builds the six concepts, ending at the exact +window-scan operator M33's WITHIN δ needs. + +This is a guide to a **paper**, so its anchors are the paper's own section, +figure, algorithm and theorem numbers: **Paranjape, Benson, Leskovec, "Motifs +in Temporal Networks," WSDM 2017** +([arXiv:1612.09259](https://arxiv.org/abs/1612.09259)). Every count and +complexity below was checked against that PDF; the one Rust block is an +illustration, marked as such. One notation warning up front: the paper writes +**`k` for the number of *nodes*** and **`l` for the number of *edges*** in a +motif — this guide uses those letters the paper's way throughout. ## The problem in one sentence -In a trace like Stack Overflow's ~63 million timestamped edges, "A messaged -B, then B messaged C, then C messaged A — all within one hour" is a single -pattern out of **36** possible 3-edge orderings on at most 3 nodes, and -counting its instances by enumerating triples of edges is hopeless. +In a trace like the SNAP `sx-stackoverflow` dataset the paper released +(2,601,977 nodes, **63,497,050 timestamped edges**), "A messaged B, then B +messaged C, then C messaged A — all within one hour" is a single pattern out +of the **36** possible 3-edge orderings on at most 3 nodes, and counting its +instances by enumerating triples of edges is hopeless. ## The concepts, step by step ### Step 1 — timestamped edges, and why order is information -A **temporal network** here is just a multiset of directed, timestamped -edges `(u, v, t)` — who contacted whom, when (no duration λ this time; an -edge is an instantaneous event). A static motif (a small subgraph pattern, -e.g. a triangle) treats these two histories as identical: +> **In:** nothing yet — this step fixes what a temporal edge is here (no +> duration λ, unlike the paths guide) and why a static count throws away the +> answer. +> **Out:** the multiset of `(u, v, t)` events every later step consumes, and +> the observation that reordering them changes their meaning. + +A **temporal network** here is a multiset of directed, timestamped edges +`(u, v, t)` — who contacted whom, when. The paper's own definition: a temporal +graph `T` on node set `V` is a collection of tuples `(ui, vi, ti)`, +`i = 1..m`, each `ti` a timestamp in ℝ (§2). Note there is **no traversal time +λ** — an edge is an *instantaneous event*, not a road you spend time on (this +is the sharpest difference from the temporal-paths guide's `(u, v, t, λ)`). The +paper assumes the `ti` are **unique**, so the edges are strictly ordered +(§2 — an assumption for clean presentation, adaptable to ties). + +A **static motif** (a small subgraph pattern, e.g. a triangle) treats these two +histories as identical: ``` history 1: A→B at 9:00, B→C at 9:05 plausible information flow @@ -32,73 +54,99 @@ history 2: B→C at 9:00, A→B at 9:05 B "forwarded" before receiving Both condense to the static path A→B→C, but only one is a possible relay. Why it matters: every behavioral question — forwarding, reciprocation, -who-answers-whom — lives in the *ordering*, which the static count -destroys. This is Step 2 of the previous guide (condensing lies) applied -to patterns instead of paths. +who-answers-whom — lives in the *ordering*, which the static count destroys. +This is the previous guide's "condensing lies" applied to patterns instead of +paths. ### Step 2 — the δ-temporal motif: sequence + order + window -A **δ-temporal motif** is an ordered sequence of k edge patterns on l node -placeholders — say `M = (A→B, B→A, A→B)` — and an **instance** of it is a -set of k actual edges that (a) map onto the placeholders consistently, -(b) occur in exactly the specified order, and (c) all fit in a window of -duration δ: last timestamp − first timestamp ≤ δ. Example with δ = 1 h: +> **In:** the `(u, v, t)` events of Step 1. +> **Out:** the formal object being counted — a `k`-node, `l`-edge motif and its +> **instances** — plus the number 36 that indexes the whole empirical paper. + +The paper's definition, quoted (§2): a **`k`-node, `l`-edge δ-temporal motif** +is a sequence of `l` edges `M = (u1, v1, t1), …, (ul, vl, tl)` that are +**time-ordered within a δ duration**, i.e. `t1 < t2 < … < tl` and +`tl − t1 ≤ δ`. Here **`δ`** is the window: the span from first to last edge may +not exceed it. An **instance** of `M` is a set of `l` actual edges that +(a) map onto the placeholders consistently, (b) occur in exactly the specified +order, and (c) satisfy `tl − t1 ≤ δ`. Example with δ = 1 h: ``` edges between ann and bob: ann→bob 9:00 bob→ann 9:20 ann→bob 9:50 ann→bob 11:00 +M = (A→B, B→A, A→B): (9:00, 9:20, 9:50) ✓ instance of M — right order, spans 50 min ≤ δ (9:00, 9:20, 11:00) ✗ spans 2 h > δ (9:20, 9:50, 11:00) ✗ order is B→A, A→B, A→B — a different motif ``` -The paper fixes k = 3 edges and l ≤ 3 nodes and shows there are exactly -**36** such motifs (their grid figure — a 6 × 6 layout: the first two edges -determine a row, the third a column). +The paper fixes attention on `l = 3` edges and `k ≤ 3` nodes and shows there +are exactly **36** such motifs (Fig 3). The count decomposes cleanly (Fig 3's +own colours): **4** two-node motifs + **8** triangle motifs + **24** star +motifs = 36. Fig 3 lays them in a 6 × 6 grid: the first edge is fixed (green +node → orange node), the **second edge indexes the row**, the **third edge the +column**. (Why 6 per axis: with the first edge fixed as node 1 → node 2, a +following edge is either *between the two existing nodes* — 2 directions — or +*between an existing node and a new third node* — 2 existing nodes × 2 +directions = 4, for 2 + 4 = 6 choices. Q1 asks you to finish this argument.) -Why it matters: δ is doing real semantic work — it encodes "these events -belong to one interaction," and every count is meaningless without stating -it. This is precisely the WITHIN δ clause of capstone M33. +Why it matters: δ is doing real semantic work — it encodes "these events belong +to one interaction," and every count is meaningless without stating it. This is +precisely the WITHIN δ clause of capstone M33. ### Step 3 — why counting is hard: one static shape, many temporal instances +> **In:** the motif `M` and window δ from Step 2. +> **Out:** the cost model that rules out naive enumeration and demands the +> per-event state of Step 4. + Because the same pair can carry many timestamped edges, a *single* static -subgraph instance can host an enormous number of temporal instances — and -they overlap. If ann and bob exchanged just 20 messages, there are -C(20, 3) = 1,140 3-edge subsequences to test against M for order and -window; a static triangle of three chatty nodes multiplies three such +subgraph instance can host an enormous number of temporal instances — and they +overlap. If ann and bob exchanged just 20 messages, there are +**C(20, 3) = 20·19·18 / 6 = 1,140** 3-edge subsequences to test against `M` for +order and window; a static triangle of three chatty nodes multiplies three such counts. Naive enumeration of edge triples over the whole trace is -O(m³)-shaped; even per-subgraph enumeration explodes with activity. +`O(m³)`-shaped; even per-subgraph enumeration explodes with activity. -Two structural facts rescue us: instances of a k-edge motif are -*subsequences* (not sets) of the time-sorted edge list, and the window -constraint means an edge only ever combines with edges at most δ away — -a sliding window. +Two structural facts rescue us: instances of an `l`-edge motif are +*subsequences* (not arbitrary sets) of the time-sorted edge list, and the +window constraint means an edge only ever combines with edges at most δ away — +a **sliding window**. -Why it matters: this is a classic streaming-aggregation shape — the cost -model is "per-edge work × m", not "candidate tuples" — if you can find the +Why it matters: this is a classic streaming-aggregation shape — the cost model +becomes "per-edge work × m," not "candidate tuples" — *if* you can find the right per-edge state. Step 4 is that state. ### Step 4 — the general algorithm: gather, then one window scan -The paper's general algorithm has two phases: (1) enumerate instances of -the motif's underlying *static* subgraph H (subgraph matching, topic 24 -machinery); (2) for each instance, gather the timestamped edges among its -nodes, sort by time, and count matching subsequences with one pass of a -sliding window, maintaining counts of *partial* matches. The paper's -Algorithm 1 counts all motifs at once by keying counters on label strings; -specialized to one motif, the state is counts of the motif's contiguous -fragments: +> **In:** the time-sorted edges among one static subgraph's nodes (Step 3). +> **Out:** a single motif-instance count, produced by one sliding-window pass +> that maintains counts of *partial* matches. + +The paper's general algorithm (§4.1, Algorithm 1) has two phases: (1) enumerate +instances of the motif's underlying *static* subgraph `H` (subgraph matching, +topic 24 machinery); (2) for each instance, gather the timestamped edges among +its nodes, sort by time, and count matching subsequences with one pass of a +sliding window, maintaining counts of *partial* matches. Algorithm 1 counts +**all** motifs at once by keying counters on label strings; the paper notes +there are `O(l²)` contiguous subsequences of an `l`-edge motif (§4.1). +Specialized to one motif, the state is exactly those `O(l²)` counters of the +motif's contiguous fragments: ```rust -/// Count instances of one k-edge motif in a single pass over the -/// time-sorted edges of ONE static instance's node set. -/// event = (t, lab); lab says which ordered node-pair the edge uses -/// (for M = (A→B, B→A, A→B): A→B ⇒ 0, B→A ⇒ 1, so motif = [0, 1, 0]). +// ILLUSTRATION — not quoted from Paranjape et al. This specializes the paper's +// Algorithm 1 (§4.1) to a single motif; the paper's own version keys counters +// on label strings to count all 36 at once (its Fig 2 traces the counters). +// The nearest single-pass-over-time-sorted-events code in this repo is +// experiments/src/temporal_reach.rs:20 (same streaming shape, different state). +// +// event = (t, lab); lab says which ordered node-pair the edge uses +// (for M = (A→B, B→A, A→B): A→B ⇒ 0, B→A ⇒ 1, so motif = [0, 1, 0]). fn count_delta_motif(events: &[(u64, u8)], motif: &[u8], delta: u64) -> u64 { - let k = motif.len(); - let mut cnt = vec![vec![0u64; k]; k]; // cnt[i][j]: matches of motif[i..=j] + let l = motif.len(); + let mut cnt = vec![vec![0u64; l]; l]; // cnt[i][j]: matches of motif[i..=j] let (mut total, mut head) = (0u64, 0usize); for &(t, lab) in events { // 1. expire events older than t − δ. The expiring event is the @@ -107,8 +155,8 @@ fn count_delta_motif(events: &[(u64, u8)], motif: &[u8], delta: u64) -> u64 { // the inner count cnt[i+1][j] is already old-free when used. while events[head].0 + delta < t { let old = events[head].1; - for len in 1..k { - for i in 0..=k - len { + for len in 1..l { + for i in 0..=l - len { let j = i + len - 1; if motif[i] == old { cnt[i][j] -= if len == 1 { 1 } else { cnt[i + 1][j] }; @@ -119,13 +167,13 @@ fn count_delta_motif(events: &[(u64, u8)], motif: &[u8], delta: u64) -> u64 { } // 2. bank completions BEFORE inserting: the new event can only // ever be the LAST edge of a full match. `total` never expires. - if lab == motif[k - 1] { - total += if k == 1 { 1 } else { cnt[0][k - 2] }; + if lab == motif[l - 1] { + total += if l == 1 { 1 } else { cnt[0][l - 2] }; } // 3. insert: extend fragments, LONGEST first, so the new event is // counted at most once per match. - for len in (1..k).rev() { - for j in len - 1..k { + for len in (1..l).rev() { + for j in len - 1..l { let i = j + 1 - len; if motif[j] == lab { cnt[i][j] += if len == 1 { 1 } else { cnt[i][j - 1] }; @@ -137,111 +185,224 @@ fn count_delta_motif(events: &[(u64, u8)], motif: &[u8], delta: u64) -> u64 { } ``` -Per event the work is O(k²) counter updates — constant for k = 3 — so the -scan is linear in the instance's edge count and never materializes a -candidate triple. +Per event the work is the `O(l²)` counter updates — for `l = 3` that is a +fixed 3×3 grid, a constant — so the scan is linear in the instance's edge count +and never materializes a candidate triple. The paper states the matching +2-node bound as `O(2lm)`, linear in `m` and optimal up to constants (§4.1). -Why it matters: correctness lives entirely in the two update orders -(expire shortest-first, insert longest-first) — get either wrong and you -double-count. The cost that remains is phase (1): static subgraph -enumeration dominates, which motivates Step 5. +Why it matters: correctness lives entirely in the two update orders (expire +shortest-first, insert longest-first) — get either wrong and you double-count. +The cost that remains is phase (1): static subgraph enumeration dominates, +which motivates Step 5. ### Step 5 — fast paths: 2-node and star motifs are easy, triangles are the fight -For motifs whose static shape is trivial, phase (1) collapses. **2-node -motifs**: group edges by unordered pair, run Step 4's scan per pair — -linear overall. **Star motifs** (all three edges touch one center node): -one pass over each center's incident edges with per-neighbor, -per-direction counters — again near-linear, with a correction for the -degenerate case where the two "spoke" neighbors coincide (that instance is -really a 2-node motif). **Triangle motifs** are the hard case: an edge -between u and v participates in every triangle through that pair, so -per-triangle scanning re-reads hot edges over and over. The paper adapts -the classic static trick — treat high-degree ("heavy") pairs specially and -assign each edge to the triangles it can complete — landing in the same -O(m√m) territory as static triangle listing, instead of paying "edges × -triangles through them." +> **In:** phase (1)'s subgraph-enumeration cost from Step 4. +> **Out:** three specialized bounds — the reason the 63M-edge traces are +> feasible at all. + +For motifs whose static shape is trivial, phase (1) collapses: + +- **2-node motifs**: group edges by unordered pair, run Step 4's scan per pair. + Linear overall, `O(m)` — the paper calls this optimal up to constants (§4.1). +- **Star motifs** (all three edges touch one center node): a dynamic program + over each center's incident edges, keyed by neighbor and direction, with a + correction that *subtracts* the 2-node counts for the degenerate case where + the two "spoke" neighbors coincide (that instance is really a 2-node motif). + Also **`O(m)`**, linear in the input (§4.2). +- **Triangle motifs** are the hard case: an edge between `u` and `v` + participates in every triangle through that pair, so per-triangle scanning + re-reads hot edges. The paper's fast algorithm (§4.2, Alg 5) assigns each edge + to the triangles it can complete and runs in **`O(TriEnum + m√τ)`**, where + `TriEnum` is the time to list all static triangles, `m` is the number of + temporal edges, and **`τ` is the number of static triangles** in the induced + graph. That is a genuine reduction from the naive per-triangle `O(mτ)` down to + `O(m√τ)` (§4.2, Theorem). + +Worked, to feel the reduction: on a graph with `m = 10⁶` temporal edges and +`τ = 10⁴` static triangles, the naive `O(mτ) = 10¹⁰`; the fast `O(m√τ)` +replaces `τ = 10⁴` with `√τ = 10²`, giving `10⁸` — a **100×** cut, which is why +the paper reports its fast temporal-triangle counter is **up to 56.5×** faster +than a competitive baseline in practice (abstract / §5). Why it matters: this mirrors topic 24 exactly — stars are the cheap -degree-local counts, triangles are where algorithmic care pays — and the -paper's scalability experiments (pull the exact speedups into notes.md) -show the specialized algorithms are what make the 63M-edge and larger -traces feasible at all. +degree-local counts, triangles are where algorithmic care pays. The paper's +scalability experiments (§5; pull the exact per-dataset speedups into notes.md) +show the specialized algorithms are what make the 63M-edge and larger traces +feasible. ### Step 6 — what the counts reveal: motif fingerprints of communication -A network's vector of 36 motif counts (usually normalized to fractions) is -a behavioral fingerprint. The paper's flagship contrast is **blocking** -vs **non-blocking** communication: on a phone call you cannot talk to two -people at once, so motifs where a node fires a second outgoing edge before -receiving a reply are rare in call networks — while email, which queues, -shows them freely. Reciprocation chains like `(A→B, B→A, A→B)` dominate -messaging data; on-off Q&A rhythms show up in the Stack Exchange traces. -And sweeping δ turns one count into a curve whose knees expose the natural -timescales of an interaction (seconds for SMS ping-pong, days for email -threads). - -Why it matters: these analyses are exactly the query shapes a temporal -graph database gets asked — MATCH an ordered pattern WITHIN δ, GROUP BY -motif, sweep δ — so the counting operators of Steps 4–5 are not paper -curiosities; they are M33's aggregate path. +> **In:** the per-motif counts produced by Steps 4–5. +> **Out:** the 36-vector "fingerprint" and the query shapes M33 must serve. + +A network's vector of 36 motif counts (usually normalized to fractions) is a +behavioral fingerprint. The paper's flagship contrast is **blocking** vs +**non-blocking** communication: on a phone call you cannot talk to two people +at once, so motifs where a node fires a second outgoing edge before receiving a +reply are rare in call networks — while email, which queues, shows them freely +(§5, the switching analysis of Fig 7 finds switching *least* common on Stack +Overflow, *most* common in email). Reciprocation chains like `(A→B, B→A, A→B)` +dominate messaging data; on-off Q&A rhythms show up in the Stack Exchange +traces. And sweeping δ turns one count into a curve whose knees expose the +natural timescales of an interaction (the paper finds certain Stack Overflow +Q&A patterns need ≥ 30 minutes to develop, §5). + +Why it matters: these analyses are exactly the query shapes a temporal graph +database gets asked — MATCH an ordered pattern WITHIN δ, GROUP BY motif, sweep +δ — so the counting operators of Steps 4–5 are not paper curiosities; they are +M33's aggregate path. ## How to read the paper (with the concepts in hand) +~10 pages, budget ~2.5 h. + - **§1 (intro) — read carefully.** The motivating example and the blocking/non-blocking teaser; this is Steps 1 and 6 in miniature. -- **§2 (definitions) — read carefully.** The formal δ-temporal motif and - instance definitions (Step 2) and the 36-motif grid figure. Spend real - time on the grid — the empirical sections index everything by its rows - and columns, and you want to be able to point at any cell and name the - behavior it encodes. -- **§3 (algorithms) — the core.** Read the general algorithm (Step 4) - first and check its counter-update orders against the Rust above; then - the star section (cheap), then the triangle section slowly (Step 5) — - the edge-to-triangle assignment argument is the paper's main algorithmic - contribution. Skim complexity proofs on first pass. -- **§4 (experiments/analysis) — read the heatmaps carefully, skim the - rest.** The per-dataset motif-fraction heatmaps carry the findings of - Step 6; extract the blocking-vs-non-blocking evidence and two concrete - speedup numbers (general vs fast algorithms) into notes.md. +- **§2 (definitions) — read carefully.** The formal `k`-node, `l`-edge + δ-temporal motif and instance definitions (Step 2) and the 36-motif grid + (Fig 3). Spend real time on Fig 3 — the empirical sections index everything by + its rows and columns, and you want to point at any cell and name the behavior + it encodes. +- **§4 (algorithms) — the core.** Read the general algorithm §4.1 (Step 4) + first and check its counter-update orders against the Rust above; then the + star section (cheap, §4.2), then the triangle section slowly (Step 5) — the + edge-to-triangle assignment argument and its `O(m√τ)` bound are the paper's + main algorithmic contribution. Skim complexity proofs on first pass. +- **§5 (experiments/analysis) — read the heatmaps carefully, skim the rest.** + The per-dataset motif-fraction heatmaps (Fig 5) carry the findings of Step 6; + extract the blocking-vs-non-blocking evidence and two concrete speedup numbers + (general vs fast algorithms) into notes.md. - **Related work — skim**, noting how δ-motifs differ from earlier - "time-respecting subgraph" definitions that require paths rather than - ordered windows. + "time-respecting subgraph" definitions that require paths rather than ordered + windows. ## Questions to answer in notes.md -1. Derive the 36: why exactly that many motifs with 3 edges on at most - 3 nodes and a total order? Show the counting argument. +1. Derive the 36: why exactly that many motifs with `l = 3` edges on `k ≤ 3` + nodes and a total order? Show the counting argument (the 4 + 8 + 24 + decomposition, or the 6 × 6 grid). 2. In Step 4's code, why must expiry update shortest fragments first and - insertion longest first? Construct a 3-event sequence that gets - miscounted if either order is flipped. -3. From the heatmaps: which motif cells separate the phone/SMS (blocking) - datasets from email (non-blocking)? Record the actual fractions the - paper reports. -4. Capstone M33: write motif `M = (A→B, B→A, A→B)` as a time-respecting - MATCH with WITHIN δ. Which parts does the planner get free from the - δ constraint, and where must Step 4's window-scan operator replace - enumerate-then-filter to avoid Step 3's C(n, 3) blowup? + insertion longest first? Construct a 3-event sequence that gets miscounted if + either order is flipped. +3. From the heatmaps (Fig 5): which motif cells separate the phone/SMS + (blocking) datasets from email (non-blocking)? Record the actual fractions + the paper reports. +4. Capstone M33: write motif `M = (A→B, B→A, A→B)` as a time-respecting MATCH + with WITHIN δ. Which parts does the planner get free from the δ constraint, + and where must Step 4's window-scan operator replace enumerate-then-filter to + avoid Step 3's C(n, 3) blowup? 5. Topic 24 tie: static triangle counting is a masked matrix multiply in GraphBLAS. Exactly where does the *temporal* triangle count stop being expressible as a matrix product, and what per-triangle state survives? -6. δ-sweep as a workload: if a user recomputes counts at 20 values of δ, - what does topic 30's time-bucketed storage (M30) let you reuse across - sweeps, and what must be recomputed? +6. δ-sweep as a workload: if a user recomputes counts at 20 values of δ, what + does topic 30's time-bucketed storage (M30) let you reuse across sweeps, and + what must be recomputed? ## Done when -You can derive the 36-motif count, hand-trace Step 4's window scan over a -five-event sequence without miscounting, explain in one sentence each why -stars are cheap and triangles are hard, and state which M33 query shape -each of the paper's two algorithm families (general vs specialized) maps -onto. +Answer each before unfolding it. + +- [ ] You can derive the 36-motif count. + +
Answer + + Count `l = 3`-edge motifs on `k ≤ 3` nodes, up to relabeling nodes by order + of first appearance, and decompose by node count (Fig 3's colours): + + - **2-node** (nodes A, B only): the first edge is fixed A→B by the + first-appearance convention, leaving 2 directions each for edges 2 and 3 → + `2 × 2 = 4` motifs. + - **Triangle** (3 distinct nodes, each of the 3 edges on a different pair): + `8` motifs. + - **Star** (a center plus two spokes, all 3 edges incident to the center): + the paper groups these into 3 classes (pre / post / mid) of `2³ = 8` each → + `24` motifs. + + `4 + 8 + 24 = 36`, exactly the 6 × 6 grid the paper indexes by second edge + (row) and third edge (column) with the first edge fixed (§2, Fig 3). + +
+ +- [ ] You can hand-trace Step 4's window scan over a five-event sequence without miscounting. + +
Answer + + Take `M = [0, 1, 0]` (i.e. A→B, B→A, A→B), δ large enough that nothing + expires, and events `(t, lab)` = `(1,0), (2,1), (3,0), (4,1), (5,0)`. Process + left to right, banking completions *before* inserting (Step 4, phase 2): + + - `(1,0)`: `lab 0 == motif[2] 0`, but `cnt[0][1] = 0`, so bank 0. Insert + extends fragment `[0]`: `cnt[0][0] = 1`. + - `(2,1)`: `lab 1 ≠ motif[2] 0`, bank nothing. Insert: `motif[1] = 1`, so + `cnt[0][1] += cnt[0][0] = 1`. + - `(3,0)`: `lab 0 == motif[2]`, bank `cnt[0][1] = 1` → `total = 1`. Insert: + `cnt[0][0] += 1 = 2`. + - `(4,1)`: bank nothing. Insert: `cnt[0][1] += cnt[0][0] = 2` → `cnt[0][1] = 3`. + - `(5,0)`: bank `cnt[0][1] = 3` → `total = 4`. + + Four instances: the A→B at position 5 completes with each earlier + (A→B, B→A) prefix, and the A→B at position 3 completed one earlier. Banking + before inserting is what stops event `(5,0)` from pairing with itself; the + shortest-first expiry (unused here) is what keeps the subtraction consistent + when δ is finite. + +
+ +- [ ] You can explain in one sentence each why stars are cheap and triangles are hard. + +
Answer + + **Stars are cheap** because every edge of a 3-node star touches the center, so + a single dynamic-programming pass over each center's incident edges — keyed by + neighbor and direction, minus the 2-node-motif correction for coincident + spokes — counts them in `O(m)`, linear in the input (§4.2). + + **Triangles are hard** because an edge between `u` and `v` lies on every + triangle through that pair, so scanning per triangle re-reads hot edges; the + paper's fix assigns each edge to the triangles it can complete, cutting the + naive `O(mτ)` to `O(TriEnum + m√τ)` where `τ` is the static-triangle count + (§4.2) — the `√τ` is the whole reason a 63M-edge trace finishes. + +
+ +- [ ] You can state which M33 query shape each of the paper's two algorithm families maps onto. + +
Answer + + The **general algorithm** (§4.1, Step 4) counts any single `M` by gathering + the edges among a matched static subgraph and running the `O(l²)`-state + sliding-window scan — it maps onto M33's *"MATCH this specific ordered pattern + WITHIN δ"*, where the planner has already pinned the shape and only the + window scan remains. + + The **specialized family** (§4.2, Step 5 — 2-node, star, triangle) counts + *all* motifs of a class at once with per-class bounds (`O(m)` for 2-node and + stars, `O(m√τ)` for triangles) — it maps onto M33's *"GROUP BY motif over the + whole trace"* aggregate, where enumerate-then-filter would pay Step 3's + `C(n,3)` blow-up and the specialized counters avoid it. + +
## References **Papers** -- Paranjape, Benson, Leskovec — "Motifs in Temporal Networks" (WSDM - 2017) — [arXiv](https://arxiv.org/abs/1612.09259) / - [PDF](https://arxiv.org/pdf/1612.09259) — ~10 pages, ~2.5 h: definitions - and the 36-grid carefully, the general algorithm against Step 4's code, - the triangle section slowly, heatmaps for the findings; skim proofs +- Paranjape, Benson, Leskovec — "Motifs in Temporal Networks" (WSDM 2017) — + [arXiv](https://arxiv.org/abs/1612.09259) / + [PDF](https://arxiv.org/pdf/1612.09259) — ~10 pages, ~2.5 h: read the §2 + definitions and Fig 3 carefully, the §4.1 general algorithm against Step 4's + code, the §4.2 triangle section slowly, and the §5 heatmaps for the findings; + skim proofs. Anchors used above: §2 + Fig 3 (definition, the 36 motifs), + §4.1 (Algorithm 1, `O(l²)` counters, `O(2lm)` 2-node bound), §4.2 (star `O(m)`, + triangle `O(TriEnum + m√τ)`, up to 56.5× speedup), §5 (blocking vs + non-blocking, δ-sweep). +- Dataset: the SNAP `sx-stackoverflow` temporal network the paper released — + 2,601,977 nodes, 63,497,050 temporal edges, 2,774-day span + ([snap.stanford.edu/data/sx-stackoverflow](https://snap.stanford.edu/data/sx-stackoverflow.html)). + +**Related guides** +- [reading-temporal-paths.md](reading-temporal-paths.md) — where *reachability* + (not ordering) is the information; the two guides are the two ways timestamps + change a graph question. +- [README.md](README.md) — the topic's measured false-positive headline; the + motif count is the *aggregate* companion to that path query. diff --git a/topics/33-temporal-graphs/reading-temporal-paths.md b/topics/33-temporal-graphs/reading-temporal-paths.md index 9c5a36a..b8c5293 100644 --- a/topics/33-temporal-graphs/reading-temporal-paths.md +++ b/topics/33-temporal-graphs/reading-temporal-paths.md @@ -3,9 +3,17 @@ Topic 8 gave every record a `begin_ts`/`end_ts` interval; topic 30's capstone stored a graph you can time-travel. This paper asks the question both left open: once the *edges themselves* carry timestamps, what is a path? The -answer breaks static-graph intuition twice — reachability changes and -Dijkstra's greedy invariant dies — and this chapter builds the seven -concepts one at a time before handing you a section-by-section reading lens. +answer breaks static-graph intuition twice — reachability stops being +transitive and Dijkstra's greedy invariant dies — and this chapter builds the +seven concepts one at a time before handing you a section-by-section reading +lens. + +This is a guide to a **paper**, not a repo, so its anchors are the paper's own +section, definition, theorem and table numbers: **Wu, Cheng, Huang, Ke, Lu, +Xu, "Path Problems in Temporal Graphs," PVLDB 7(9), 2014** +([PDF](http://www.vldb.org/pvldb/vol7/p721-wu.pdf)). Every definition and +complexity below was checked against that PDF; the one Rust block is an +illustration pinned to the topic's own `experiments/` crate, marked as such. ## The problem in one sentence @@ -19,52 +27,102 @@ that need four different algorithms. ### Step 1 — the temporal edge: a road that exists at one moment +> **In:** nothing yet — this step fixes the notation (`t`, `λ`, `π`, `M`) +> every later step's complexity bound is written in. +> **Out:** the quadruple `(u, v, t, λ)` and the input size `M`, the units +> Steps 2–7 consume. + A **temporal edge** is a quadruple `(u, v, t, λ)`: you may leave `u` toward -`v` only at **start time** `t`, and the crossing takes **traversal time** -`λ`, so you arrive at `v` at `t + λ`. Think of a flight: SFO→JFK departing -09:00 with λ = 5 h — the "edge" is useless at 09:01. The same vertex pair -can carry many temporal edges (the 14:00 flight, the 19:00 flight); the -paper writes `π(u, v)` for that multiplicity, and `M` for the total number -of temporal edges — the real size of the input. +`v` only at **start time** `t` (the instant the edge is usable), and the +crossing takes **traversal time** `λ` (how long the hop itself lasts), so you +arrive at `v` at `t + λ`. Think of a flight: SFO→JFK departing 09:00 with +λ = 5 h — the "edge" is useless at 09:01. The paper writes this exactly: +a temporal edge is `(vi, vi+1, ti, λi) ∈ E` (§2, Definition of a temporal +graph). + +The same vertex pair can carry many temporal edges (the 14:00 flight, the +19:00 flight); the paper writes **`π(u, v)`** for that **multiplicity** — the +number of temporal edges from `u` to `v` — and **`M = |E|`** for the total +number of temporal edges, versus **`m = |Es|`** for the edges of the +*condensed* static graph (Step 2) and **`n = |V|`** for the vertices (§2). Why it matters: `M` counts *events*, not relationships. A social network with 1M static edges and daily interactions for a year has `M ≈ 365M` -temporal edges. Every complexity bound below is in `M`, and every storage -decision in M33 is about where those `(t, λ)` pairs live. +temporal edges. Every complexity bound below is in `M` (never `m`), and every +storage decision in M33 is about where those `(t, λ)` pairs live. + +### Step 2 — condensing lies: reachability is not transitive -### Step 2 — condensing lies: the static view over-reports reachability +> **In:** the temporal edges of Step 1. +> **Out:** the **condensed graph** and the proof that the reachability it +> reports is wrong — the fact the whole topic's measured headline counts. The obvious move — drop the timestamps, keep one static edge per connected -pair (the **condensed graph**) — gives wrong answers, not just imprecise -ones. Watch a 3-node graph (all λ = 1): +pair — gives the **condensed graph** `Gs` (the paper's term: all temporal +edges between the same pair collapse to one static edge, §2). It gives *wrong* +answers, not just imprecise ones, and the cleanest way to see why is that +**temporal reachability is not transitive**: write `u ⇝ w` for "a +time-respecting path runs from `u` to `w`"; then `a ⇝ b` and `b ⇝ c` do **not** +imply `a ⇝ c`. A concrete three-node counterexample (all λ = 1): ``` temporal: a ──(t=2)──► b ──(t=1)──► c -condensed: a ────────► b ────────► c "c reachable from a" — FALSE + a ⇝ b : take (a,b,2,1), arrive b at time 3. TRUE + b ⇝ c : from b, take (b,c,1,1), arrive c at time 2. TRUE + a ⇝ c : to chain them you must be at b BEFORE t=1 to board + b→c, but the only way into b arrives at time 3 > 1. FALSE -reality: you arrive at b at time 3, but the only b→c edge - departed at time 1. It's gone. c is unreachable from a. +condensed: a ────────► b ────────► c → says "c reachable from a" — a LIE ``` -The paper's Fig 1 makes exactly this point on a slightly larger example, -and adds a second lie: even when the destination *is* reachable, the -condensed graph's hop-count or weight-sum "shortest path" can name a route -that no time-respecting traversal can follow. - -Why it matters: an `AT TIME t` snapshot view (capstone M33) is a condensed -graph of the edges alive at `t`. It is the right tool for "what did the -graph look like" and provably the *wrong* tool for "what could flow through -it" — no single snapshot can answer a cross-time reachability question. +`a ⇝ b` holds, `b ⇝ c` holds, `a ⇝ c` fails: reachability composed across `b` +does not survive, because `b`'s onward contact departed at t = 1 but the path +into `b` does not arrive until t = 3. Static condensation silently assumes the +transitivity that temporal graphs lack, so it counts `(a, c)` as reachable. + +The paper's Fig 1 makes the same point on a larger example and adds a second +lie: even when the destination *is* reachable, the condensed graph's +hop-count or weight-sum "shortest path" can name a route no time-respecting +traversal can follow. + +Why it matters: this non-transitivity is exactly what this topic *measured*. +On the sparse contact graph, static reachability reports **25,031** reachable +pairs where time-respecting paths number **137** — **99.5% false positives** +([FINDINGS.md](../../FINDINGS.md) row 33; the full density sweep is in +[README.md](README.md)). An `AT TIME t` snapshot view (capstone M33) is a +condensed graph of the edges alive at `t`: the right tool for "what did the +graph look like," and provably the *wrong* tool for "what could flow through +it." ### Step 3 — the temporal path and its four minima -A **temporal path** (also: time-respecting path) is a sequence of temporal -edges where each edge departs no earlier than the previous one arrives: -`tᵢ + λᵢ ≤ tᵢ₊₁` — timestamps non-decreasing along the path, exactly M33's -MATCH semantics. Queries fix a time window `[tα, tω]` (depart no earlier -than `tα`, arrive no later than `tω`). Now "best" splits four ways. One -graph, source `a`, target `c`, window `[0, 10]`: +> **In:** the temporal edges (Step 1), now to be *chained* legally. +> **Out:** four scalar objectives — earliest-arrival, latest-departure, +> fastest, shortest — each of which Steps 5–7 compute with a different +> algorithm. + +A **temporal path** (also **time-respecting path**) is a sequence of temporal +edges where each edge departs no earlier than the previous one arrives. The +paper states it as: for consecutive edges on the path, +`(ti + λi) ≤ ti+1` (§2). Two quantities are read off any path `P`: +its **starting time** `start(P) = t1`, its **ending time** +`end(P) = tk + λk` (departure plus traversal of the last edge), and from them +its **duration** `dura(P) = end(P) − start(P)` and its **distance** +`dist(P) = Σ λi` (the sum of traversal times) (§2, Definition 1 preamble). + +A query fixes a **time window** `[tα, tω]`: consider only paths with +`start(P) ≥ tα` and `end(P) ≤ tω`. Within that set the paper defines +**four minimum temporal paths** (§3, Definition 1), quoted as it states them: + +- **Earliest-arrival path** — minimizes `end(P)`. (Called the **foremost** + path in the earlier Bui-Xuan–Ferreira–Jarry lineage the paper cites, ref + [21]; "earliest-arrival" and "foremost" name the same objective.) +- **Latest-departure path** — maximizes `start(P)` subject to arriving by `tω`. +- **Fastest path** — minimizes `dura(P) = end(P) − start(P)`. +- **Shortest path** — minimizes `dist(P) = Σ λi`. + +Worked on one graph — source `a`, target `c`, window `[0, 10]`: ``` edges (u, v, t, λ): (a, b, 1, 4) depart 1, arrive 5 @@ -72,26 +130,42 @@ edges (u, v, t, λ): (a, b, 1, 4) depart 1, arrive 5 (a, c, 8, 1) depart 8, arrive 9 ``` -| minimum | optimizes | winner here | value | +There are two temporal paths from `a` to `c`. Compute each objective on both, +by hand, then read off the winner: + +| path | start | end | dura = end − start | dist = Σλ | +|---|---|---|---|---| +| `a→b→c` = ⟨(a,b,1,4),(b,c,6,1)⟩ | 1 | 6 + 1 = 7 | 7 − 1 = 6 | 4 + 1 = 5 | +| `a→c` = ⟨(a,c,8,1)⟩ | 8 | 8 + 1 = 9 | 9 − 8 = 1 | 1 | + +| minimum | objective | winner | value | |---|---|---|---| -| **earliest-arrival** | min arrival time | a→b→c | arrives 7 | -| **latest-departure** | max start time (arriving by tω) | a→c | departs 8 | -| **fastest** | min duration (arrival − departure) | a→c | 9 − 8 = 1 | -| **shortest** | min Σλ (total traversal time) | a→c | Σλ = 1 | +| **earliest-arrival** | min `end(P)` | `a→b→c` | end = 7 (< 9) | +| **latest-departure** | max `start(P)` | `a→c` | start = 8 (> 1) | +| **fastest** | min `dura(P)` | `a→c` | 9 − 8 = 1 (< 6) | +| **shortest** | min `dist(P)` | `a→c` | Σλ = 1 (< 5) | -In a static graph all four collapse into "shortest". Here the -earliest-arrival route is *neither* fastest nor shortest — waiting for the -late direct edge wins three of the four criteria. +In a static graph all four collapse into "shortest." Here the +earliest-arrival route (`a→b→c`, arriving 7) is *neither* fastest nor +shortest, and waiting for the late direct edge `a→c` wins the other three +criteria. Three of the four "best paths" disagree with the fourth on one tiny +graph. -Why it matters: these are four distinct path *functions* for M33; a query -planner must know which one the user asked for, because no single answer -serves all four. +Why it matters: these are four distinct path *functions* for M33. A query +planner must know which one the user asked for, because — as the table proves +— no single answer serves all four. ### Step 4 — greedy dies: a subpath of a shortest path isn't shortest -Dijkstra's algorithm rests on one invariant: any prefix of a shortest path -is itself a shortest path, so a vertex can be "settled" once. Temporal -edges break it — a cheap prefix can arrive *too late* to catch the +> **In:** the shortest-path objective from Step 3. +> **Out:** the counterexample that forbids Dijkstra's "settle once" step, and +> so forces either the dominance lists of Step 6 or the transformation of +> Step 7. + +Dijkstra's algorithm rests on the **subpath optimality** invariant: any prefix +of a shortest path is itself a shortest path to its endpoint, so a vertex can +be **settled** (fixed at its best-known distance, never revisited) once. +Temporal edges break it — a cheap prefix can arrive *too late* to catch the connecting edge: ``` @@ -99,30 +173,42 @@ connecting edge: (a, b, 8, 1) the cheap prefix: arrive b at 9, cost Σλ = 1 ← shortest to b (b, c, 6, 1) departs b at 6 -shortest a→c = (a,b,0,5)+(b,c,6,1), Σλ = 6 — its prefix to b costs 5, -even though a Σλ = 1 route to b exists. The cheap route misses the bus. +shortest a→c = (a,b,0,5)+(b,c,6,1), Σλ = 5 + 1 = 6 — its prefix to b costs 5, +even though a Σλ = 1 route to b exists. The cheap route arrives b at 9, +after b→c has departed at 6, so it cannot extend to c at all. ``` -So you cannot settle `b` with its best-known distance; a *dominated-looking* -label (higher cost, earlier arrival) must be kept alive. The fix is either -Pareto frontiers per vertex (Step 6) or restructuring the input so greedy -works again (Steps 5 and 7). +So you cannot settle `b` at its best-known distance (1): the *dominated-looking* +label (cost 5, but arriving at 5 instead of 9) is the one that extends to `c`. +A label must be kept alive when it is worse in cost but better in arrival time. +The fix is either Pareto frontiers per vertex (Step 6) or restructuring the +input so greedy works again (Step 7). The paper flags this — "subpaths of a +shortest path may not be shortest" — in its abstract and §3. Why it matters: this is the single theorem-shaped fact to carry out of the -paper — it is why you can't bolt a timestamp filter onto topic 24's -frontier BFS/Dijkstra and call it done. +paper. It is why you cannot bolt a timestamp filter onto topic 24's frontier +BFS/Dijkstra and call it done. ### Step 5 — the one-pass scan: earliest arrival in O(n + M) +> **In:** the earliest-arrival objective (Step 3), plus the assumption that +> edges are pre-sorted by start time. +> **Out:** one number per vertex — `arr[v]`, its earliest arrival time — from +> a single sequential pass, no priority queue. + If edges are pre-sorted by start time `t` (the paper's **edge stream** -representation), earliest-arrival needs no priority queue at all — one -sequential pass, each edge examined exactly once: +representation, §2), earliest-arrival needs no priority queue: one sequential +pass, each edge examined exactly once (Algorithm 1). The paper proves this +runs in **O(n + M) time and O(n) space** (§4.2, Theorem for Algorithm 1). The +shape, illustrated on the topic's own stub: ```rust -/// One-pass earliest-arrival over a time-sorted edge stream. -/// Returns the earliest arrival time at every vertex within [tα, tω]. +// ILLUSTRATION — not quoted from Wu et al.; this is the paper's Algorithm 1 +// (§4.2, earliest-arrival) with the [tα, tω] window written out. The real +// code you implement to this contract is experiments/src/temporal_reach.rs:20 +// (its relax rule is stated at temporal_reach.rs:14). fn earliest_arrival( - stream: &[(u32, u32, u64, u64)], // (u, v, t, λ), sorted by t + stream: &[(u32, u32, u64, u64)], // (u, v, t, λ), sorted by t ascending src: usize, n: usize, t_alpha: u64, t_omega: u64, ) -> Vec { @@ -141,41 +227,67 @@ fn earliest_arrival( ``` Why it works: by the time the stream reaches start time `t`, every way of -arriving anywhere before `t` has already been recorded — time order *is* -the topological order. Latest-departure is the mirror image: scan the -stream backwards, maintaining the latest possible departure from each -vertex. For a single target, stop as soon as `t ≥ arr[target]`. - -Why it matters: a single forward scan over a sorted array is the -best-behaved access pattern topic 0 knows — prefetch-friendly, no pointer -chasing — and it's the shape M33's earliest-arrival path function wants. -The price is the precondition: storage must hand you edges in time order -(question 3). +arriving anywhere before `t` has already been recorded — **time order is the +topological order**, so one relaxation per edge suffices (this is the exact +claim README exercise 2 asks you to prove, and the `zero_lambda_chains_...` +test at `experiments/src/temporal_reach.rs:59` pins the λ = 0 tie case). +Latest-departure is the mirror image: scan the stream backwards, maintaining +the latest possible departure from each vertex (§4.3, also O(n + M)). For a +single target, stop as soon as `t ≥ arr[target]`. + +Why it matters: a single forward scan over a sorted array is the best-behaved +access pattern topic 0 knows — prefetch-friendly, no pointer chasing — and it +is the shape M33's earliest-arrival path function wants. The price is the +precondition: storage must hand you edges in time order (question 3). ### Step 6 — dominance lists: fastest and shortest in one pass, plus a log -Fastest and shortest can't be summarized by one number per vertex (Step 4), -so the one-pass framework keeps a small **dominance list** (Pareto -frontier — set of candidates none of which is better in both coordinates) -at each vertex: for fastest, pairs of (departure-from-source `s`, arrival -`a`); for shortest, pairs of (distance `d`, arrival `a`). A new pair is -inserted only if nothing in the list dominates it, and pairs it dominates -are evicted; the lists stay sorted, so each edge costs a binary search — a -log factor over Step 5, still a single time-ordered pass. - -Why it matters: the memory cost moved from O(1) to O(frontier size) per -vertex — bounded in practice by the number of distinct useful departure -times. This is the same labels-not-scalars move that multi-criteria route -planning makes, and it's what your M33 executor must carry per node when a -query asks for fastest rather than earliest. +> **In:** the fastest and shortest objectives (Step 3), which Step 4 proved +> cannot be summarized by one scalar per vertex. +> **Out:** a per-vertex **dominance list** and the extra `log` factor keeping +> it costs, still over a single time-ordered pass. + +Fastest and shortest need more than one number per vertex (Step 4), so the +one-pass framework keeps a **dominance list** at each vertex — a **Pareto +frontier**, the set of candidate labels none of which is better than another in +*both* coordinates. For fastest the coordinates are (departure-from-source `s`, +arrival `a`); for shortest they are (distance `d`, arrival `a`). A new label is +inserted only if nothing already in the list **dominates** it (beats or ties it +on both coordinates), and every label it dominates is evicted. The lists stay +sorted, so each edge costs a binary search. + +That binary search is the whole price. The paper's bounds (§4.4, §4.5): + +- **Fastest**: `O(n + M log c)` time, `O(min{n|S|, n + M})` space, where `S` + is the set of distinct out-edge start times. +- **Shortest**: `O(n + M log dmax)` time, `O(n + M)` space, where + `dmax = max{din(v)}` is the largest in-degree. + +Both are a single time-ordered pass with a `log`-sized list operation per +edge, versus Step 5's `O(1)` per edge. Worked, to see the `log` is not free +but is small: on the topic's lane-1 graph, `n = 2000`, `M = 3999`; a shortest +run costs on the order of `n + M·log(dmax)` — with an average in-degree near +`M/n ≈ 2`, `log dmax` is a single-digit factor, so the pass stays within a +small constant of Step 5's `n + M = 5999` edge-touches rather than blowing up. + +Why it matters: the memory cost moved from `O(1)` to `O(frontier size)` per +vertex — bounded in practice by the number of distinct useful departure times. +This is the same labels-not-scalars move multi-criteria route planning makes, +and it is what your M33 executor must carry per node when a query asks for +fastest rather than earliest. ### Step 7 — the transformed graph: pay O(M) space, get statics back (§5) -The alternative to new algorithms is a **time-expanded graph**: replace -each vertex `v` by copies `(v, time-point)` — one per distinct time an edge -arrives at or departs from `v` — chain the copies forward with 0-weight -"wait here" edges, and turn each temporal edge `(u, v, t, λ)` into a static -edge from copy `(u, t)` to copy `(v, t + λ)`: +> **In:** the original temporal edges (Step 1) and the four objectives +> (Step 3). +> **Out:** a static **DAG** on which plain BFS/Dijkstra recompute all four +> minima — the materialized-view alternative to Steps 5–6. + +The alternative to new algorithms is a **time-expanded graph** (the paper's +graph transformation, §5): replace each vertex `v` by copies `(v, t)` — one +per distinct time an edge arrives at or departs from `v` — chain the copies +forward in time with 0-weight "wait here" edges, and turn each temporal edge +`(u, v, t, λ)` into a static edge from copy `(u, t)` to copy `(v, t + λ)`: ``` (b,3) ──wait──► (b,6) vertex b's timeline @@ -184,83 +296,202 @@ edge from copy `(u, t)` to copy `(v, t + λ)`: arrive b at 3 depart b at 6 static DAG edges ``` -Because chaining (not all-pairs wiring) connects the copies, the result has -O(M) vertices *and* O(M) edges, and it is a DAG (edges only go forward in -time) — so plain BFS/Dijkstra/topological-order algorithms compute all four -minima correctly again. +Because chaining (not all-pairs wiring) connects the copies, the result is a +**directed acyclic graph** — every edge points forward in time, so no cycle +exists — and §5.5 proves that, assuming `n < M`, **both its vertex count and +its edge count are O(M)**. So plain BFS / Dijkstra / topological-order +algorithms compute all four minima correctly again: single-source +earliest-arrival and latest-departure each cost one BFS, i.e. `O(M)` (§5.5). +The paper's Table 4 makes the blow-up concrete — for the `arxiv` trace the +transformed `|Ṽ| = 433K` and `|Ẽ| = 9759K`, both the same order as `M`. Why it matters: this is the materialized-view option — precompute a bigger static structure so the classic toolbox (and topic 24's frontier engines) -applies unchanged. The paper's experiments measure exactly this trade: -transformation pays construction time and a blown-up working set per query -window; the one-pass algorithms stream the original data. Read the -experiment tables as a build-vs-scan price list. +applies unchanged. The paper's experiments (§6) measure exactly this trade: +transformation pays construction time and a per-query window's blown-up working +set; the one-pass algorithms stream the original data. Read the experiment +tables as a build-vs-scan price list. ## How to read the paper (with the concepts in hand) The paper is ~12 pages; the definitions and the one-pass algorithms are the -payload. +payload. Budget ~2 h. - **§1 (intro) + Fig 1 — read carefully.** Fig 1 is Step 2 in the authors' example; reproduce its reachability lie in your notes before moving on. -- **§2 (definitions) — read carefully.** Temporal graph, `π(u, v)`, `M`, - the edge-stream representation, temporal paths, and the formal statements - of the four minima (Step 3). Nail the notation table — everything later - leans on it. -- **§3–§4 (one-pass algorithms) — the core; read carefully.** The - earliest-arrival pseudocode should match Step 5's Rust nearly line for - line; latest-departure is its mirror. For fastest and shortest, focus on - the dominance-list bookkeeping (Step 6) — read the invariants, skim the - proofs on first pass. Note where the subpath-property failure (Step 4) - is invoked to justify the lists. +- **§2 (definitions) — read carefully.** Temporal graph, `π(u, v)`, `M`, `m`, + `n`, the edge-stream representation, temporal paths, `start`/`end`/`dura`/ + `dist`, and the formal four minima (Step 3, Definition 1). Nail the notation + table — everything later leans on it. +- **§4 (one-pass algorithms) — the core; read carefully.** Algorithm 1 + (earliest-arrival) should match Step 5's Rust nearly line for line; + latest-departure (§4.3) is its mirror. For fastest (§4.4) and shortest + (§4.5), focus on the dominance-list bookkeeping (Step 6) — read the + invariants, skim the proofs on first pass. Note where the subpath-property + failure (Step 4) is invoked to justify the lists. - **§5 (graph transformation) — read the construction, skim the proofs.** - Check the figure against Step 7's sketch; the thing to verify is *why* - the size stays O(M) (chaining, not complete wiring). -- **§6 (experiments) — skim with two questions:** how much faster is - one-pass than transformation per query, and how does transformation's - cost scale with window size? Pull two concrete numbers from the tables - into notes.md. -- **Related work / conclusion — skim.** Note the lineage they cite for the - transformation idea; it predates the one-pass framework. + Check Fig 2 against Step 7's sketch; the thing to verify is *why* the size + stays O(M) (chaining, not complete wiring — §5.5). +- **§6 (experiments) — skim with two questions:** how much faster is one-pass + than transformation per query, and how does transformation's cost scale with + window size? Pull two concrete numbers from the tables into notes.md. +- **Related work / conclusion — skim.** Note the lineage they cite (ref [21], + Bui-Xuan et al.) for shortest/fastest/foremost journeys; it predates the + one-pass framework. ## Questions to answer in notes.md 1. From Fig 1: which pairs does the condensed graph claim are reachable but - temporally are not — and even for a truly reachable pair, which of the - four minima does the static graph compute wrongly? -2. State precisely which invariant of Dijkstra's correctness proof the - Step 4 counterexample violates, and why keeping (distance, arrival) - Pareto pairs restores correctness. + temporally are not — and even for a truly reachable pair, which of the four + minima does the static graph compute wrongly? +2. State precisely which invariant of Dijkstra's correctness proof the Step 4 + counterexample violates, and why keeping (distance, arrival) Pareto pairs + restores correctness. 3. Step 5's precondition is a time-sorted edge stream. FalkorDB stores - adjacency as GraphBLAS matrices (topic 13): what is the cheapest layout - that yields per-window time-ordered edges — timestamped edge-list - sidecar, per-time-bucket delta matrices (topic 30's M30), or sorting at - query time? Sketch the cost of each for a `[tα, tω]` query. + adjacency as GraphBLAS matrices (topic 13): what is the cheapest layout that + yields per-window time-ordered edges — timestamped edge-list sidecar, + per-time-bucket delta matrices (topic 30's M30), or sorting at query time? + Sketch the cost of each for a `[tα, tω]` query. 4. Capstone M33: earliest-arrival as a path function. Rewrite Step 5's - relaxation condition for (a) a WITHIN δ constraint (path duration ≤ δ) - and (b) MATCH with non-decreasing timestamps but no λ — which of the - four minima does each correspond to? + relaxation condition for (a) a WITHIN δ constraint (path duration ≤ δ) and + (b) MATCH with non-decreasing timestamps but no λ — which of the four minima + does each correspond to? 5. MVCC tie-back (topic 8): `begin_ts`/`end_ts` version intervals are - *transaction time*; `(u, v, t, λ)` is *valid time*. Which queries from - this paper can an AT TIME snapshot answer exactly, and which are - unanswerable by any single snapshot no matter how it's chosen? -6. Treat §5's transformation as a materialized view of size O(M): given - average multiplicity π and a query mix, when does building it beat - running one-pass scans per query? Where's the break-even? + *transaction time*; `(u, v, t, λ)` is *valid time*. Which queries from this + paper can an AT TIME snapshot answer exactly, and which are unanswerable by + any single snapshot no matter how it is chosen? +6. Treat §5's transformation as a materialized view of size O(M): given average + multiplicity `π` and a query mix, when does building it beat running + one-pass scans per query? Where is the break-even? ## Done when -You can state the four minima and produce a graph where all four differ; -you can reproduce the greedy counterexample from memory and say which -Dijkstra invariant it kills; you can write the one-pass earliest-arrival -scan without looking; and you can say, in one sentence each, what storage -order it demands from FalkorDB and why an AT TIME view can never answer it. +Answer each before unfolding it. + +- [ ] You can state the four minima and produce a graph where all four differ. + +
Answer + + The four are Wu et al.'s Definition 1 (§3): **earliest-arrival** minimizes + `end(P)`, **latest-departure** maximizes `start(P)`, **fastest** minimizes + `dura(P) = end − start`, **shortest** minimizes `dist(P) = Σλ`. + (Earliest-arrival is the objective the older literature calls *foremost*, + ref [21].) + + Step 3's graph separates them: edges `(a,b,1,4)`, `(b,c,6,1)`, `(a,c,8,1)`, + window `[0,10]`. The path `a→b→c` has `end = 7`, `dura = 6`, `dist = 5`; the + direct `a→c` has `end = 9`, `dura = 1`, `dist = 1`, `start = 8`. + Earliest-arrival picks `a→b→c` (end 7 < 9); latest-departure, fastest and + shortest all pick `a→c` (start 8, dura 1, dist 1). One graph, the + earliest-arrival winner losing every other criterion — the proof that four + algorithms are genuinely needed. + +
+ +- [ ] You can reproduce the greedy counterexample from memory and say which Dijkstra invariant it kills. + +
Answer + + The invariant is **subpath optimality**: a prefix of a shortest path is a + shortest path, which licenses settling a vertex once and never revisiting it. + + Step 4's edges kill it: `(a,b,0,5)`, `(a,b,8,1)`, `(b,c,6,1)`. The cheapest + route to `b` is `(a,b,8,1)` at `dist = 1`, but it arrives at time 9, after + `b→c` departs at 6, so it extends nowhere. The shortest `a→c` is + `(a,b,0,5)+(b,c,6,1)` with `dist = 6`, whose prefix to `b` costs 5 — a + *non-shortest* prefix. So `b` cannot be settled at cost 1; the (cost 5, + arrival 5) label, dominated on cost, is the one that reaches `c`. Keeping + both coordinates as a Pareto pair (Step 6) is what preserves the label greedy + would have discarded, restoring correctness at a `log` factor. + +
+ +- [ ] You can write the one-pass earliest-arrival scan without looking, and say why one relaxation per edge suffices. + +
Answer + + The loop is Step 5's `earliest_arrival`: initialize `arr[src] = tα`, then for + each `(u, v, t, λ)` in start-time order, if `t ≥ arr[u]` and + `t + λ < arr[v]`, set `arr[v] = t + λ`. Skip edges arriving after `tω`; break + once `t > tω`. Wu et al. prove this is `O(n + M)` time, `O(n)` space + (§4.2, Algorithm 1). + + One relaxation per edge is enough because the stream is sorted by `t`, so + **time order is topological order**: when the scan reaches an edge departing + at `t`, every arrival earlier than `t` has already been written into `arr`, + so `arr[u]` is final for the purpose of departing at `t`. There is no way a + later edge improves an arrival that an earlier departure needed — the exact + claim README exercise 2 asks you to prove, and the reason the crate's contract + (`experiments/src/temporal_reach.rs:14`) forbids a fixpoint loop. The λ = 0 + case (`temporal_reach.rs:59`) works because the relax test uses `t ≥ arr[u]`, + non-strict, so an arrival at `t` can board a departure at the same `t`. + +
+ +- [ ] You can say what storage order the one-pass scan demands from FalkorDB, and why an AT TIME view can never answer it. + +
Answer + + It demands **edges delivered in non-decreasing start-time order** within the + query window — the edge-stream representation (§2). FalkorDB's GraphBLAS + adjacency is not time-ordered, so M33 must supply the order some other way: a + timestamped edge-list sidecar, per-time-bucket delta matrices (M30), or a + sort at query time (question 3 weighs the three). + + An `AT TIME t` view cannot answer earliest-arrival because it is a *condensed + graph of one instant* — it discards the very `(t, λ)` ordering the scan needs, + and Step 2 proved reachability across such a condensation is not even + transitive: `a ⇝ b` and `b ⇝ c` there do not imply `a ⇝ c`. This is the + topic's measured 99.5% false-positive result ([FINDINGS.md](../../FINDINGS.md) + row 33): no single snapshot, however chosen, holds the cross-time information + a time-respecting path is made of. + +
+ +- [ ] You can say when materializing §5's transformed graph beats streaming the one-pass scans. + +
Answer + + Building the time-expanded graph costs `O(M)` space and construction time + (§5.5) but then answers each single-source query with one plain BFS/Dijkstra + over a static DAG, reusing the structure across queries. The one-pass scans + (Steps 5–6) touch the original stream once *per query* and keep only `O(n)` + to `O(n + M)` transient state. + + So the transformation wins when the same window is queried many times — the + `O(M)` build amortizes over a query batch — and loses on a single ad-hoc + query, where you pay the full build to run one BFS. The break-even is roughly + "build cost / per-query scan cost" queries against the same window; question 6 + works it against the multiplicity `π` and a query mix. It is precisely + topic 5's checkpoint-vs-redo trade again: materialize once and replay cheaply, + or stream every time. + +
## References **Papers** - Wu, Cheng, Huang, Ke, Lu, Xu — "Path Problems in Temporal Graphs" (PVLDB Vol 7, No 9, 2014) — - [PDF](http://www.vldb.org/pvldb/vol7/p721-wu.pdf) — ~12 pages, ~2 h: - read §1–§2 and the one-pass algorithms carefully, the §5 construction - once, and skim the experiments for the one-pass vs transformation gap + [PDF](http://www.vldb.org/pvldb/vol7/p721-wu.pdf) — ~12 pages, ~2 h: read + §1–§2 and the §4 one-pass algorithms carefully, the §5 construction once, and + skim the §6 experiments for the one-pass vs transformation gap. Anchors used + above: §2 (notation, temporal path, edge stream), §3 Definition 1 (four + minima), §4.2/§4.3 (earliest-arrival / latest-departure, O(n+M)), + §4.4/§4.5 (fastest / shortest, the `log` factor), §5.5 + Table 4 + (transformation size O(M)). +- Bui-Xuan, Ferreira, Jarry — "Computing shortest, fastest, and foremost + journeys in dynamic networks" (Int. J. Found. Comput. Sci. 14(2), 2003) — + ref [21], the source of the "foremost" = earliest-arrival naming. + +**Code** +- This topic's `experiments/src/temporal_reach.rs` (`earliest_arrival` stub, + contract at `:14`, tests at `:33`) — the one-pass scan you implement; bench + lane 2 times it against the fixpoint oracle in `events.rs`. + +**Related guides** +- [reading-temporal-motifs.md](reading-temporal-motifs.md) — δ-temporal + motifs, where ordering (not reachability) is the information. +- [reading-aeong.md](reading-aeong.md) and + [reading-raphtory.md](reading-raphtory.md) — the storage engines that must + hand these algorithms edges in time order. diff --git a/topics/34-debugging/README.md b/topics/34-debugging/README.md index 5ed6957..57d20d4 100644 --- a/topics/34-debugging/README.md +++ b/topics/34-debugging/README.md @@ -143,7 +143,7 @@ actually zero. | redis | `src/latency.c:182` | `createLatencyReport` — an advice engine over 160-sample rings | | redis | `src/debug.c:2643` | `sigalrmSignalHandler` — the watchdog that stack-traces a stuck server | | rocksdb | `include/rocksdb/perf_context.h:305` | `PerfContext` — per-query counters, thread-local | -| rocksdb | `monitoring/perf_context_imp.h:27` | `PERF_TIMER_GUARD` — level-gated, compile-out-able | +| rocksdb | `monitoring/perf_context_imp.h:45` (live), `:27` (`NPERF_CONTEXT` no-op) | `PERF_TIMER_GUARD` — compile-out-able; the level check is in `PerfStepTimer`'s ctor (`perf_step_timer.h:19`), not the macro | | rocksdb | `monitoring/histogram.h:21` | `HistogramBucketMapper` — all of u64 in 109 buckets | | FalkorDB | `src/slow_log/slow_log.c` | `SlowLog_Add` — the C surface M34 ports to the Rust engine | diff --git a/topics/34-debugging/notes.md b/topics/34-debugging/notes.md index eeb09f8..4ffdd13 100644 --- a/topics/34-debugging/notes.md +++ b/topics/34-debugging/notes.md @@ -63,7 +63,8 @@ numbers, not noise. perf_context_imp.h:27/:45/:81/:88, perf_step_timer.h:13/:29, histogram.h:21/:46/:84/:110, statistics_impl.h:42, statistics.cc:549; FalkorDB src/slow_log/slow_log.{c,h}. -- rr facts verified from the ATC'17 PDF (arXiv:1705.05937), pp. 1–8: +- rr facts verified from the extended technical report (arXiv:1705.05937 — the + ~21-page edition, not the shorter ATC'17 conference paper), pp. 1–8: RCB counter, seccomp-bpf in-process interception, RR page, < 2× slowdown, one-thread-at-a-time limitation. - Crate: 3 provided tests green (workload.rs), 6 stub tests fix diff --git a/topics/34-debugging/reading-flamegraphs.md b/topics/34-debugging/reading-flamegraphs.md index f62b791..3a47680 100644 --- a/topics/34-debugging/reading-flamegraphs.md +++ b/topics/34-debugging/reading-flamegraphs.md @@ -1,35 +1,61 @@ # The Flame Graph: folding a million stacks into one picture -Brendan Gregg's CACM article (Vol 59 No 6, June 2016; also ACM Queue) -is the canonical write-up of the visualization you will stare at for -the rest of your performance career. In topic 34's diagnosis triad — -wrong answers / too slow / crashed — this is the "too slow" leg: -profiler samples are perishable evidence, and the flame graph -aggregates them into one durable artifact. This chapter builds the one -structural idea (merging identical stacks) and the reading discipline -(width, not height) before you open the article — with the database -question in view: for a query engine under load, the off-CPU variant -is usually where the story is. +Brendan Gregg's "The Flame Graph" is the canonical write-up of the +visualization you will stare at for the rest of your performance +career. In topic 34's diagnosis triad — wrong answers / too slow / +crashed — this is the "too slow" leg: profiler samples are perishable +evidence, and the flame graph aggregates them into one durable +artifact. This chapter builds the one structural idea (merging +identical stacks) and the reading discipline (width, not height) +before you open the article — with the database question in view: for a +query engine under load, the off-CPU variant is usually where the story +is. + +This is an *article*, not code, so every claim below is anchored to a +**section** of Gregg's paper rather than a `file:line`. The version +cited is **ACM Queue, Vol 14, Issue 2 (April 2016)** — republished as +**Communications of the ACM, Vol 59, No 6 (June 2016)**, DOI +`10.1145/2927299.2927301`. Numbers quoted are the ones the article +itself prints, in the section named. ## The problem in one sentence -A CPU profiler at 99 Hz across many cores emits hundreds of thousands -of stack traces per minute — far too many to read as text — and the -flame graph compresses them into one interactive picture by merging -identical stacks and drawing width proportional to sample count. +A CPU profiler at 99 Hz across many cores emits tens of thousands of +stack traces per minute — far too many to read as text — and the flame +graph compresses them into one interactive picture by merging identical +stacks and drawing width proportional to sample count. ## The concepts, step by step ### Step 1 — the raw material: sampled stacks, not traced calls -Everything starts with a sampling profiler: interrupt the CPU at a -fixed rate (e.g. `perf record -g -F 99`) and record the full call -stack of whatever is running. Two properties matter. First, sampling -is statistical: a function's share of samples estimates its share of -CPU time. Second, it is *zero code change*: you attach to the live -database when the pager fires, detach when done — the deliberate -complement to bench lane 3, which prices the observability tax of -always-on instrumentation. The output, though, is a wall of stacks. +> **In:** nothing yet — this step names the raw evidence every later +> step consumes. +> **Out:** a wall of **stack traces**, one per timer interrupt, that +> Step 2 will merge. + +A **sampling profiler** interrupts the CPU at a fixed rate and records +what is running; it does *not* instrument every call. A **stack trace** +(call stack) is the list of nested function calls active at that +instant — the leaf (on-CPU function) at the top, its caller beneath, +down to the thread entry at the root. A single frame is one function in +that list. + +Two properties make sampling the right raw material. First, it is +**statistical**: a function's *share of samples* estimates its *share +of CPU time*, so you never need to catch every call, only enough +samples to be representative. Second, it is **zero code change** — you +attach to the live database when the pager fires and detach when done. +That is the deliberate complement to this topic's bench lane 3, which +prices the observability tax of *always-on* instrumentation: sampling +costs nothing until you attach. + +Gregg's §"CPU Profiling" gives the canonical rate: stack traces are +sampled at **99 times per second** — "not 100, to avoid lock-step +sampling", i.e. to avoid beating against any 100 Hz periodic activity. +Over 30 seconds on a 16-CPU box that is `16 × 99 × 30 = 47,520` +samples; "as text, this would be hundreds of thousands of lines." The +output is unreadable precisely because it is complete: ``` worker-3 worker-3 worker-3 @@ -38,15 +64,24 @@ always-on instrumentation. The output, though, is a wall of stacks. executor::pull executor::pull parser::parse expand_op filter_op GrB_mxm eval_predicate - ─ sample 184,001 ─ ─ sample 184,002 ─ ─ sample 184,003 ─ ...×10⁵ + ─ sample 184,001 ─ ─ sample 184,002 ─ ─ sample 184,003 ─ ...×10⁴ ``` +Why it matters: any tool that dumps raw samples buries the signal. The +next step is the one idea that makes 47,520 stacks legible. + ### Step 2 — the merge: identical stacks become one column -The whole trick of the flame graph is one aggregation step: two -samples with byte-identical stacks are the same evidence, so merge -them and keep a count. In the classic pipeline, `stackcollapse-perf.pl` -folds `perf script` output into one line per unique stack: +> **In:** the wall of stack traces from Step 1. +> **Out:** a **folded profile** — one line per *unique* stack with a +> count — which Step 3 renders as geometry. + +The whole trick of the flame graph is one aggregation step: two samples +with byte-identical stacks are the same evidence, so merge them and +keep a count. In Gregg's §"Instructions", the pipeline's middle stage, +`stackcollapse-perf.pl`, folds `perf script` output into the **folded +format** — each stack on one line, functions separated by semicolons, +then a space and a count: ``` main;run_query;executor::pull;expand_op;GrB_mxm 41203 @@ -54,20 +89,59 @@ main;run_query;executor::pull;filter_op;eval_predicate 8931 main;run_query;parser::parse 1204 ``` -Hundreds of thousands of samples collapse into a few hundred unique -stacks. `flamegraph.pl` renders these as an SVG: shared stack prefixes -become shared boxes, unique suffixes branch off above the last common -frame. Why it matters: the folded file is a greppable, diffable text -artifact — the SVG is a view; the folded file is the data. +The article's own case study is the scale argument: the MySQL profile +in §"The Problem" was `591,622` lines of DTrace output holding `27,053` +unique stacks — collapsing merges the samples down to those unique +paths, and the flame graph makes the whole thing readable on one +screen. The renderer, `flamegraph.pl`, turns the folded file into an +SVG: shared stack *prefixes* become shared boxes, unique suffixes +branch off above the last common frame. + +The full three-step pipeline (§"Instructions"), worth memorizing +because you will type it: + +``` +# perf record -F 99 -a -g -- sleep 60 +# perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > out.svg +``` + +Why it matters: the folded file is a greppable, diffable *text* +artifact — the SVG is only a view of it. Step 5's regression diff and +your own `grep`/`awk` post-processing both operate on the folded file, +not the picture. ### Step 3 — reading the geometry: width is everything, height is nothing -In the rendered graph, the y-axis is stack depth (root at the bottom, -leaves at the top) and each box is one function frame. Box **width** -is the fraction of samples in which that frame appeared — relative -time, inclusive of everything called above it. Crucially, the x-axis -is **not** time: after merging, sibling frames sort alphabetically, so -left-to-right order carries no temporal meaning. The samples above: +> **In:** the folded profile from Step 2, rendered as an SVG. +> **Out:** the reading discipline — which boxes to trust and which to +> ignore — that the rest of the chapter applies. + +Gregg's §"Flame Graphs Explained" fixes the semantics precisely. The +**y-axis** is stack depth, root at the bottom and leaf at the top; each +box is one function frame, and the box beneath a box is its caller. The +**x-axis** "does *not* show the passage of time" — after merging, +sibling frames are sorted **alphabetically** by function name, "which +maximizes box merging" (identical adjacent frames fuse), so +left-to-right order carries no temporal meaning at all. + +Box **width** is the load-bearing dimension. It is the fraction of +samples in which that frame appeared, counting every sample where the +frame was anywhere in the stack — its own time *plus* everything it +called. That "plus everything above it" is why width is called +**inclusive**: a caller is always at least as wide as its widest child. +As a formula, for frame `f`: + +``` +width(f) = (samples whose stack contains f) / (total samples) +``` + +Worked on the article's own numbers. Total samples for a 16-CPU, 30 s, +99 Hz profile is `16 × 99 × 30 = 47,520`. A frame that shows up in +`11,880` of those stacks renders at `11,880 / 47,520 = 25%` of the full +width. And the MySQL red herring from §"The Problem": its `status` +command appeared in `5,530` of `348,427` samples, so +`5,530 / 348,427 = 1.59% ≈ 1.6%` — visibly a sliver, which is exactly +why the eye skips it and lands on the real culprit (`join`). ``` ┌──────────┐┌──┐ @@ -84,23 +158,39 @@ left-to-right order carries no temporal meaning. The samples above: width ∝ samples ──────────▶ (x-order alphabetical, NOT time) ``` -The reading discipline: hunt for **wide plateaus** — flat-topped boxes -where samples terminate, i.e. functions actually on-CPU — not tall -spikes. Depth is just call-path length; a 60-frame tower 2 pixels wide -costs nothing, while a squat 40%-wide `memcpy` plateau is your bug. -The top edge, left to right, is the histogram of on-CPU leaf functions. +The reading discipline (§"Flame Graph Interpretation"): hunt for **wide +plateaus** — flat-topped boxes along the top edge where samples +*terminate*, i.e. functions actually on-CPU — not tall spikes. A +**plateau** is a wide box whose top edge is exposed; that width is +on-CPU time. Depth is just call-path length: a 60-frame tower two +pixels wide costs nothing, while a squat 40%-wide `memcpy` plateau is +your bug. + +Why it matters: every beginner reads height (deep = important) and gets +it backwards. The top edge, left to right, is the histogram of on-CPU +leaf functions; that is the whole profile. ### Step 4 — on-CPU vs off-CPU: two graphs, two questions -The on-CPU flame graph answers "where is CPU time spent." Its dual, -the **off-CPU flame graph**, samples the stacks of threads that are -*blocked* — waiting on locks, disk I/O, page faults, the scheduler — -weighted by time spent off-CPU. For databases this is often the -interesting one: a write-heavy engine's on-CPU graph can look -innocently flat while every worker spends most of wall time parked in -`futex_wait` under a latch or `fdatasync` behind the WAL. Those stacks -never run, so the on-CPU graph *cannot* show them — the two graphs -partition wall-clock time: +> **In:** the reading discipline from Step 3, applied to *two different* +> sample sources. +> **Out:** the split — which of two flame graphs answers "where is CPU +> spent" and which answers "where is wall-clock spent waiting." + +The **on-CPU flame graph** (everything so far) answers "where is CPU +time spent." Its dual, the **off-CPU flame graph** (§"Other Targets" → +Off-CPU), samples the stacks of threads that are *blocked* — not +running — with the box width proportional to time spent blocked rather +than to sample count. Gregg lists the reasons a thread goes off-CPU: +"waiting on I/O, locks, timers, a turn on-CPU, and waiting for paging +or swapping." Because those stacks were captured *when the thread was +descheduled*, the width is blocked time. + +For databases this is often the interesting graph. A write-heavy engine +can show an innocently flat on-CPU profile while every worker spends +most of wall-clock time parked in `futex_wait` under a latch or +`fdatasync` behind the WAL. Those stacks never run, so the on-CPU graph +*cannot* show them — the two graphs partition wall-clock time: ``` wall time of one worker thread @@ -109,92 +199,210 @@ partition wall-clock time: eval_pred read() (page miss) ``` -Why it matters: "the database is slow but CPUs are idle" is exactly -where an on-CPU profile shrugs; off-CPU names the lock and the fsync. +Why it matters: "the database is slow but the CPUs are idle" is exactly +where an on-CPU profile shrugs. Off-CPU names the lock and the fsync — +and off-CPU is only possible *because* Step 3's rule "width can measure +anything, not just sample count" is baked into the format +(§"Flame Graphs Explained": "widths can reflect measures other than +sample counts"). ### Step 5 — differential (red/blue) flame graphs for regressions -Given two folded profiles — before/after a commit, or a good node and -a bad node — a differential flame graph draws the second profile's -shape and colors each frame by its change in sample count: red grew, -blue shrank. The regression is literally the red patch. Capture a -folded file per release, and "what got slower?" is a one-command diff. +> **In:** *two* folded profiles from Step 2 — a "before" (A) and an +> "after" (B). +> **Out:** one flame graph colored by the per-frame delta, so the +> regression is a visible patch. + +Given two folded profiles — before/after a commit, or a healthy node +and a sick one — a **differential flame graph** (§"Differential Flame +Graphs") draws the **B** profile's shape and colors each frame by its +change in sample count from A to B: "red colors indicate functions that +increased, and blue colors indicate those that decreased." The +regression is literally the red patch. + +The article is honest about the failure mode, and you should carry it: +because the drawing uses B's shape, "some code paths present in the A +profile may be missing entirely in the B profile, and so will be +missing from the final visualization" — a path that vanished shows up +as *nothing*, not as blue. Gregg's `flamegraphdiff` fixes this by +drawing three graphs (A, B, and the delta). Capture a folded file per +release and "what got slower?" becomes a one-command diff — the reason +Netflix generates these nightly. + +Why it matters: this is the payoff of Step 2's insistence that the +folded file *is* the data. You cannot diff two SVGs; you diff two text +files and render the result. ### Step 6 — pitfalls: broken stacks make lying graphs -The graph is only as good as the stack walks. If the workload is -built with `-fomit-frame-pointer` (long the compiler default), the -frame walker cannot follow the chain and you get truncated one- or -two-frame stacks — a wide lawn of "grass" at the bottom, or towers -floating on `[unknown]`. Fixes: rebuild with frame pointers -(`-fno-omit-frame-pointer`), or use DWARF or LBR-based unwinding. -Similarly, JIT and interpreted frames show as anonymous addresses -unless the runtime exports a symbol map (e.g. `/tmp/perf-PID.map`). -Rule of thumb: before believing any plateau, check that stacks reach -a plausible root (`main`, a thread entry) — a broken-stack graph -misattributes time with total confidence. - -## How to read the paper (with the concepts in hand) - -CACM 59(6) / ACM Queue, ~10 pages of prose and figures; budget ~1h. - -- **Opening problem statement** (10 min) — the MySQL mystery that raw - profiler text couldn't crack; this motivates Step 2's merge. -- **The visualization definition** (15 min) — box/width/ordering - semantics; make sure "x-axis is not time" survives the figures. -- **Implementation / pipeline** (10 min) — stackcollapse + flamegraph - SVG generation (Step 2); note how many profilers have collapsers. -- **Variants** (15 min) — off-CPU, memory, differential (Steps 4–5); - read the off-CPU part twice with your WAL-fsync hat on. -- **Challenges** (10 min) — Step 6's broken stacks and symbol - problems, straight from the source. -- Then do, don't just read (~15 min): `perf record -g` against - FalkorDB under a benchmark, render the SVG, find the widest plateau. +> **In:** any flame graph from Steps 3–5. +> **Out:** the two ways the picture can be confidently wrong, and how to +> spot them before you trust a plateau. + +The graph is only as good as the stack walks, and §"Challenges" names +the two ways they break. First, **incomplete stack traces**. A **frame +pointer** is the register (`%rbp` on x86-64) that chains each stack +frame to its caller; when "the software compiler reuses the frame +pointer register as a compiler optimization" — the historical default +under `-fomit-frame-pointer` — the frame walker cannot follow the chain +and you get truncated one- or two-frame stacks: a wide lawn of "grass" +at the bottom, or towers floating on `[unknown]`. The fix is "a +different compiled binary (e.g., using gcc's `-fno-omit-frame-pointer`) +or a different stack-walking technique" (DWARF or LBR unwinding). At +Netflix the Java fix was the JVM's `-XX:+PreserveFramePointer`. + +Second, **missing function names**. Here the stack is complete but many +frames "are represented as hexadecimal addresses" — the JIT/interpreted +case, "which may not create a standard symbol table for profilers." The +fix is a supplemental symbol file (Linux `perf_events` reads one; the +Java fix is `perf-map-agent`). + +Rule of thumb: before believing any plateau, check that stacks reach a +plausible root (`main`, a thread entry). A broken-stack graph +misattributes time with total confidence — it is the flame-graph +equivalent of a benchmark that measured the wrong thing. + +Why it matters: a lying flame graph looks exactly like a truthful one. +The only defense is checking the roots before you read the widths. + +## How to read the article (with the concepts in hand) + +ACM Queue 14(2) / CACM 59(6), ~10 pages of prose and figures; budget +~1h. Read it section by section, mapping each to a step above: + +- **§"CPU Profiling"** (10 min) — sampling, the 99 Hz convention, the + wall-of-text problem (Step 1). +- **§"The Problem"** (10 min) — the MySQL 40%-CPU mystery on Joyent that + raw profiler text couldn't crack; the numbers (591,622 lines, 27,053 + stacks, `join` was the culprit) motivate Step 2's merge. +- **§"Flame Graphs Explained" + "Flame Graph Interpretation"** (15 min) + — box/width/ordering semantics; make sure "x-axis is not time" + survives the figures (Step 3). +- **§"Other Targets" → Off-CPU** (15 min) — read it twice with your + WAL-fsync hat on (Step 4). +- **§"Differential Flame Graphs"** (10 min) — red/blue, and the missing- + path caveat (Step 5). +- **§"Challenges"** (10 min) — broken stacks and missing symbols + (Step 6), straight from the source. +- Then do, don't just read (~15 min): `perf record -g` against FalkorDB + under a benchmark, render the SVG, find the widest plateau. ## Questions to answer in notes.md 1. Why is "the x-axis is not time" the load-bearing design decision? - Name one question a time-ordered stack chart answers that a flame - graph cannot, and the far more common converse. + Name one question a time-ordered stack chart (Gregg's "flame chart") + answers that a flame graph cannot, and the far more common converse. 2. Predict FalkorDB's flame graphs under a read-heavy workload: which - plateaus dominate on-CPU (GraphBLAS matrix ops? filter eval? - result serialization? allocator?), and what appears off-CPU that - the on-CPU graph hides entirely? Then measure and score yourself. -3. A frame occupying 30% width at mid-height but with almost no - samples terminating in it — what does that tell you, and where do - you look next? + plateaus dominate on-CPU (GraphBLAS matrix ops? filter eval? result + serialization? allocator?), and what appears off-CPU that the on-CPU + graph hides entirely? Then measure and score yourself. +3. A frame occupying 30% width at mid-height but with almost no samples + terminating in it — what does that tell you, and where do you look + next? 4. Bench lane 3 prices always-on instrumentation; sampling costs ~nothing until attached. What can built-in counters tell you that a flame graph cannot (per-query attribution, tail latencies, rare events between samples)? 5. Your nightly profile of a Rust binary shows a wide floor of - two-frame "grass" stacks. Give the diagnosis steps and the two - fixes from Step 6; which fits a production FalkorDB build, and why? + two-frame "grass" stacks. Give the diagnosis steps and the two fixes + from Step 6; which fits a production FalkorDB build, and why? ## Done when -- [ ] You can explain why width = fraction of samples containing the - frame, and why alphabetical order (not time) enables the merge. -- [ ] You have generated one on-CPU flame graph of FalkorDB under load - via perf → stackcollapse → flamegraph.pl and named its widest - plateau. -- [ ] You can state which class of database slowness (locks, fsync, - page faults) is invisible on-CPU and needs the off-CPU variant. -- [ ] You can spot broken stack walks (grass, floating towers) before - trusting any plateau. +Answer each before unfolding it. + +- [ ] You can explain why box width is the fraction of samples containing the frame, and why alphabetical (not time) ordering is what enables the merge. + +
Answer + + Width is inclusive sample share: `width(f) = (samples whose stack + contains f) / (total samples)` (§"Flame Graphs Explained"), so a + caller is always at least as wide as its widest child, and the + visible top edge is the histogram of on-CPU leaf functions. On the + article's own 16-CPU/30 s/99 Hz profile that denominator is + `16 × 99 × 30 = 47,520`; a frame in 11,880 of them draws at 25%. + + The merge only works because sibling frames are sorted + *alphabetically* by name, not by time. Alphabetical order guarantees + that two identical frames which belong side by side end up + horizontally adjacent, so they fuse into one wide box; a time-ordered + x-axis would scatter the same function across the width and destroy + the merge (that is exactly why Gregg's time-ordered "flame chart" + merges poorly, especially across threads). + +
+ +- [ ] You have generated one on-CPU flame graph of FalkorDB under load via perf → stackcollapse → flamegraph.pl and named its widest plateau. + +
Answer + + The three-step pipeline is `perf record -F 99 -a -g -- sleep 60`, then + `perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > out.svg` + (§"Instructions"). `perf record` samples stacks at 99 Hz across all + CPUs; `stackcollapse-perf.pl` folds them to one line per unique stack + with a count; `flamegraph.pl` renders that folded file to SVG. + + The widest plateau is whatever flat-topped box owns the largest slice + of the top edge — for a read-heavy FalkorDB run, likely a GraphBLAS + kernel (`GrB_mxm`/`GrB_mxv`) or filter evaluation. Its width, read off + the SVG's mouse-over as "N samples, P percent", is that function's + share of on-CPU time. + +
+ +- [ ] You can state which class of database slowness is invisible on-CPU and needs the off-CPU variant. + +
Answer + + Anything the thread does *while descheduled*: waiting on I/O, on + locks, on timers, for a turn on-CPU, or for paging/swapping + (§"Other Targets" → Off-CPU). For a database that is lock contention + (`futex_wait` under a latch) and durable-write stalls (`fdatasync` + behind the WAL). These stacks never execute, so an on-CPU profile — + which only samples running threads — cannot contain them. The off-CPU + flame graph captures the stack at the moment the thread blocks and + weights each box by blocked time, so the two graphs together partition + wall-clock time. + +
+ +- [ ] You can spot broken stack walks (grass, floating towers) before trusting any plateau. + +
Answer + + Two symptoms from §"Challenges". Truncated stacks — a wide floor of + one- or two-frame "grass", or towers floating on `[unknown]` — mean + the frame pointer was omitted (`-fomit-frame-pointer`) so the walker + lost the chain; the fix is `-fno-omit-frame-pointer` (or DWARF/LBR + unwinding, or `-XX:+PreserveFramePointer` for the JVM). Frames shown + as bare hex addresses mean missing symbols, typically from JIT'd code; + the fix is a supplemental symbol map (`perf-map-agent` for Java). + + The check is mechanical: before believing a plateau, confirm its stack + reaches a plausible root (`main` or a thread entry). A broken-stack + graph misattributes time with total confidence, so a plateau on top of + grass is worthless until the walk is fixed. + +
## References **Article** -- Brendan Gregg — "The Flame Graph" (Communications of the ACM 59(6), - June 2016; also ACM Queue) — - [queue.acm.org/detail.cfm?id=2927301](https://queue.acm.org/detail.cfm?id=2927301) +- Brendan Gregg — "The Flame Graph" (ACM Queue 14(2), April 2016; + republished in Communications of the ACM 59(6), June 2016; DOI + `10.1145/2927299.2927301`) — + [queue.acm.org/detail.cfm?id=2927301](https://queue.acm.org/detail.cfm?id=2927301). + Sections cited: CPU Profiling; The Problem; Flame Graphs Explained; + Instructions; Flame Graph Interpretation; Other Targets (Off-CPU); + Differential Flame Graphs; Challenges. **Code & further material** - [FlameGraph repo](https://github.com/brendangregg/FlameGraph) — - `stackcollapse-*.pl` and `flamegraph.pl`; the pipeline in Step 2 + `stackcollapse-*.pl` and `flamegraph.pl`; the pipeline in Step 2. + (Not in this repo's pin table; treat script names as stable + interface, not pinned line numbers.) - [Gregg's flame graphs page](https://www.brendangregg.com/flamegraphs.html) — index of variants (off-CPU, memory, differential) and per-profiler - instructions + instructions. - This topic's bench lane 3 — the always-on instrumentation cost that - sampling profilers complement + sampling profilers complement. diff --git a/topics/34-debugging/reading-redis-doctors.md b/topics/34-debugging/reading-redis-doctors.md index 9af82fa..9a3cf60 100644 --- a/topics/34-debugging/reading-redis-doctors.md +++ b/topics/34-debugging/reading-redis-doctors.md @@ -7,9 +7,10 @@ history into prose advice, and the software watchdog interrupts a stuck server to log where it is standing. For a FalkorDB developer this is not analogy — this code IS the host process your module runs inside, and `slowlogPushEntryIfNeeded` fires after every -`GRAPH.QUERY`. The repo is cloned at `~/repos/redis`; this is a -code-read, ~1.5h, across four small files. The anchor table below -maps each step to an exact file:line. +`GRAPH.QUERY`. The repo is cloned at `~/repos/redis`, pinned at +`redis@a176d1225`; this is a code-read, ~1.5h, across four small +files. The anchor table below maps each step to an exact file:line, +and every snippet is quoted from that pinned SHA with real gutters. ## The problem in one sentence @@ -23,8 +24,15 @@ be turned off. ### Step 1 — the theorem: tier the diagnosis surface by cost -Redis solves the observer-cost problem by splitting diagnosis into -three tiers, each with a hard budget it can always afford: +> **In:** the problem statement above — "zero overhead on the fast +> path, useful answers on demand." +> **Out:** a three-tier cost model that the next five steps each fill +> in with one concrete Redis surface. + +Redis solves the **observer-cost problem** (the act of measuring must +cost less than what it measures, or it gets disabled) by splitting +diagnosis into three tiers, each with a hard budget it can always +afford: ``` tier surface cost per command when it runs @@ -45,196 +53,354 @@ budget is only meetable because of this tiering. ### Step 2 — tier 1: SLOWLOG's one-compare fast path -`slowlogPushEntryIfNeeded` (`src/slowlog.c:103`) is called after -EVERY command completes, with the measured duration in microseconds: +> **In:** the "always-on" tier named in Step 1. +> **Out:** the exact fast-path contract (`>=` logs, negative disables, +> oldest evicted) that Step 3 then shows must also bound the *record* +> it creates. + +`slowlogPushEntryIfNeeded` is called after EVERY command completes, +with the measured `duration` in microseconds. The entire always-on +tier is the two comparisons at `:104`–`:105`: ```c -void slowlogPushEntryIfNeeded(client *c, robj **argv, int argc, long long duration) { - if (server.slowlog_log_slower_than < 0 || server.slowlog_max_len == 0) return; - if (duration >= server.slowlog_log_slower_than) - listAddNodeHead(server.slowlog, slowlogCreateEntry(c,argv,argc,duration)); - /* trim list down to slowlog-max-len: oldest evicted */ +// src/slowlog.c:103 +103 void slowlogPushEntryIfNeeded(client *c, robj **argv, int argc, long long duration) { +104 if (server.slowlog_log_slower_than < 0 || server.slowlog_max_len == 0) return; /* disabled */ +105 if (duration >= server.slowlog_log_slower_than) +106 listAddNodeHead(server.slowlog, +107 slowlogCreateEntry(c,argv,argc,duration)); + // ... :109 remove old entries, trimming to slowlog-max-len ... +110 while (listLength(server.slowlog) > server.slowlog_max_len) +111 listDelNode(server.slowlog,listLast(server.slowlog)); // listLast == oldest +112 } ``` -The whole always-on tier is those two comparisons (`:104`–`:105`). -The exact semantics are load-bearing and easy to get subtly wrong: -`>=` not `>` (a command equal to the threshold logs); a *negative* -threshold disables entirely (0 logs everything); the list is trimmed -to `slowlog-max-len` with the OLDEST evicted; and entry ids -(`server.slowlog_entry_id`, stamped in `slowlogCreateEntry`) -increase monotonically and are NOT reset by SLOWLOG RESET, so a -poller never confuses a new entry with one it already saw. Topic -34's `SlowLog` Rust stub is exactly this function reshaped: its -contract tests encode >=-logs, negative-disables, -fixed-ring-oldest-evicted, ids-monotonic-across-reset — pass them -and you have reimplemented `:103`–`:111`. +The semantics are load-bearing and easy to get subtly wrong. `>=` not +`>`: a command *equal* to the threshold logs (`:105`). A *negative* +threshold disables the log entirely, and `slowlog-log-slower-than` +defaults to `10000` µs = 10 ms (`src/config.c:3270`); setting it to +`0` logs everything. The list is a head-insert (`:106` +`listAddNodeHead`) and trims from the *tail* (`:111` +`listDelNode(..., listLast(...))`), so the **oldest** entry is evicted +at `slowlog-max-len`. Entry ids (`se->id = server.slowlog_entry_id++` +in `slowlogCreateEntry`, `src/slowlog.c:70`) increase monotonically +and are NOT reset by `SLOWLOG RESET`, so a poller never confuses a new +entry with one it already saw. + +Topic 34's `SlowLog` Rust stub is exactly this function reshaped: its +contract tests encode `>=`-logs, negative-disables, +fixed-ring-oldest-evicted, ids-monotonic-across-reset — pass them and +you have reimplemented `:103`–`:111`. Why it matters: this is the one +tier that runs unconditionally, so its cost — one branch on a server +global — is the floor of the whole design. ### Step 3 — the evidence must not become the problem +> **In:** the entry that Step 2's `:106` creates via +> `slowlogCreateEntry`. +> **Out:** the two caps (argc, string length) and the dup-not-share +> rule that keep one logged command from pinning megabytes of keyspace. + `slowlogCreateEntry` (`src/slowlog.c:28`) shows the second-order -discipline: the diagnostic record itself is bounded. Arguments are -capped at `SLOWLOG_ENTRY_MAX_ARGC` (the last slot becomes -`"... (N more arguments)"`), and any string longer than -`SLOWLOG_ENTRY_MAX_STRING` is truncated with a -`"... (N more bytes)"` suffix: +discipline: the diagnostic record itself is bounded, because a slow +command is often slow *because* its arguments are huge. Two caps, +defined in `src/slowlog.h`: + +- `SLOWLOG_ENTRY_MAX_ARGC` = `32` (`:13`): argument slots beyond 31 are + collapsed into a single `"... (%d more arguments)"` marker + (`src/slowlog.c:39`–`:42`). +- `SLOWLOG_ENTRY_MAX_STRING` = `128` (`:14`): any argument longer than + 128 bytes is truncated and a `"... (%lu more bytes)"` suffix records + how many bytes were dropped (`src/slowlog.c:45`–`:54`). + +Worked on a concrete command. `SET giant-key <10 MB value>` has a value +of `10 × 1024 × 1024 = 10,485,760` bytes. The entry keeps the first +`128` bytes, and the suffix reports `10,485,760 − 128 = 10,485,632` +more bytes: ``` - SET giant-key <10 MB value> slowlog entry: - ──────────────────────────► [SET][giant-key][first 128 B "... (10485632 more bytes)"] + SET giant-key <10 MB value> slowlog entry (bounded): + ──────────────────────────► [SET][giant-key][first 128 B + "... (10485632 more bytes)"] ``` -Non-shared argument objects are duplicated, not refcounted — the -comment at `:58` explains why: sharing an robj between the slowlog -and the keyspace means FLUSHALL ASYNC could free it under the log's -feet. Why it matters: a slow command is often slow *because* its -arguments are huge; the log observes values, it must never own them. +Non-shared argument objects are *duplicated* (`dupStringObject`, +`src/slowlog.c:64`), not refcounted — the comment at `:58` explains +why: sharing an `robj` between the slowlog and the keyspace means +`FLUSHALL ASYNC` could free the object on a background thread while the +log still points at it. Why it matters: the log observes values, it +must never own them, and it must never let a pathological command turn +a diagnostic into an OOM. ### Step 4 — tier 2: latency rings, zero-cost when disarmed +> **In:** the "armed" tier from Step 1 — the middle budget. +> **Out:** the fixed-ring data structure and the macro pair whose cost +> is one compare when disarmed, which Step 5's doctors then read. + The latency monitor tracks named *events* (fork, expire-cycle, -command, aof-write...), each in a fixed ring of 160 one-second -samples — `#define LATENCY_TS_LEN 160` (`src/latency.h:17`): memory -per event is `160 * 8` bytes, forever. Instrumentation is macros: +command, aof-write…), each in a fixed ring of 160 one-second samples — +`#define LATENCY_TS_LEN 160` (`src/latency.h:17`). A sample is a +`{time, latency}` pair, so per-event memory is `160 × sizeof(sample)`, +bounded forever regardless of load. Instrumentation is two macros +whose whole body is guarded by a single server global: ```c -#define latencyStartMonitor(var) if (server.latency_monitor_threshold) { \ - var = mstime(); } else { var = 0; } /* latency.h:50 */ -#define latencyAddSampleIfNeeded(event,var) \ - if (server.latency_monitor_threshold && \ - (var) >= server.latency_monitor_threshold) \ - latencyAddSample((event),(var)); /* latency.h:63 */ +// src/latency.h:50 +50 #define latencyStartMonitor(var) if (server.latency_monitor_threshold) { \ +51 var = mstime(); \ +52 } else { \ +53 var = 0; \ +54 } + // ... :58 latencyEndMonitor computes (mstime() - var) under the same guard ... +63 #define latencyAddSampleIfNeeded(event,var) \ +64 if (server.latency_monitor_threshold && \ +65 (var) >= server.latency_monitor_threshold) \ +66 latencyAddSample((event),(var)); ``` -When `latency-monitor-threshold` is 0 — the default, set in -`src/config.c:3271` — both macros collapse to a single compare on a -server global: no `mstime()` call, no sample. This is -zero-cost-when-off as macro discipline; you can sprinkle these pairs -through the codebase without budgeting for them. When armed, -`latencyAddSample` (`src/latency.c:63`) fetches the event's ring -from a dict, updates the max, coalesces same-second samples (keeping -the worse latency), and advances `idx` modulo 160. Why it matters: -the armed tier's cost is proportional to how *sick* the server is, -not how busy — a healthy server pays the compare and nothing else. +When `latency-monitor-threshold` is `0` — the default, set in +`src/config.c:3271` — every one of these macros collapses to a single +`if` on a server global: no `mstime()` syscall, no sample write. This +is **zero-cost-when-off** as macro discipline; you can sprinkle the +`Start`/`End`/`AddSampleIfNeeded` triple through the codebase without +budgeting for it. When armed, `latencyAddSample` (`src/latency.c:63`) +does the ring bookkeeping: + +- fetch the event's ring from a dict, creating it on first sight + (`:64`, `:69`–`:75`); +- update the all-time max (`:77`); +- **coalesce same-second samples**: if the previous slot's timestamp + equals `now`, keep only the worse latency and return (`:81`–`:86`); +- otherwise write the new slot and advance `idx` modulo `LATENCY_TS_LEN` + (`:88`–`:92`). + +Why it matters: the armed tier's cost is proportional to how *sick* the +server is, not how *busy* — a healthy server pays the compare and +nothing else, and even a spiking server writes at most one ring slot +per event per second. ### Step 5 — tier 3a: the doctors are expert systems over the rings -`createLatencyReport` (`src/latency.c:182`) is LATENCY DOCTOR: it -walks every event's 160-sample ring, computes stats (min/max/avg/ -mean-absolute-deviation), then runs rule-based checks — slow fork? -expire-cycle spikes? appendfsync misconfigured? — accumulating an -`advices` counter (from around `:200`) and emitting human-readable -paragraphs. `getMemoryDoctorReport` (`src/object.c:1421`) is the -same pattern for memory: fragmentation ratio, allocator stats, -eviction policy, printed as advice sentences. +> **In:** the bounded rings Step 4 deposits (and the memory-overhead +> data Redis already tracks). +> **Out:** two on-demand text reports whose *only* inputs are those +> cheap tiers, so asking is always safe on a struggling server. + +`createLatencyReport` (`src/latency.c:182`) is LATENCY DOCTOR: it walks +every event's 160-sample ring, computes stats (min/max/avg/mean- +absolute-deviation), then runs rule-based checks — slow fork? +expire-cycle spikes? `appendfsync` misconfigured? — accumulating an +`advices` counter (`int advices = 0;`, `src/latency.c:200`) and +emitting human-readable paragraphs. Its output strings are *literal +source*, and are the signature you will grep for; quoted verbatim: + +```c +// src/latency.c:207 — monitoring disabled branch +207 report = sdscat(report,"I'm sorry, Dave, I can't do that. Latency monitoring is disabled in this Redis instance. [...]"); + // ... :226 first spike seen ... +226 report = sdscat(report,"Dave, I have observed latency spikes in this Redis instance. You don't mind talking about it, do you Dave?\n\n"); + // ... :355 nothing wrong ... +355 report = sdscat(report,"Dave, no latency spike was observed during the lifetime of this Redis instance, not in the slightest bit. I honestly think you ought to sit down calmly, take a stress pill, and think things over.\n"); + // ... :362 advice header ... +362 report = sdscat(report,"\nI have a few advices for you:\n\n"); +``` + +`getMemoryDoctorReport` (`src/object.c:1421`) is the same pattern for +memory. Its thresholds are explicit: "empty" if +`total_allocated < 5 MB` (`:1434`, `1024*1024*5`); big-peak if +`peak_allocated / total_allocated > 1.5` (`:1439`); high-frag if +`total_frag > 1.4 && total_frag_bytes > 10 MB` (`:1445`, `10<<20`). +Its literal strings, verbatim: + +```c +// src/object.c:1491 — no issue found +1491 s = sdsnew("Hi Sam, I can't find any memory issue in your instance. I can only account for what occurs on this base.\n"); + // ... :1494 empty instance ... +1494 s = sdsnew("Hi Sam, this instance is empty or is using very little memory, my issues detector can't be used in these conditions. [...]"); + // ... :1502 issues detected header ... +1502 s = sdsnew("Sam, I detected a few issues in this Redis instance memory implants:\n\n"); +``` ``` tier-2 rings (cheap, always bounded) tier-3 doctor (expensive, on demand) [fork: 160 samples] ──┐ - [expire-cycle:160 samples] ──┼──► walk + stats + IF/THEN rules ──► prose + [expire-cycle:160 samples] ──┼──► walk + stats + IF/THEN rules ──► "Dave, ..." [command: 160 samples] ──┘ (runs only when you type LATENCY DOCTOR) ``` Why it matters: the doctors do string formatting, allocation, and -O(events × 160) analysis — costs the fast path could never absorb — -but they read only bounded evidence tiers 1–2 already deposited, so -asking the question is always safe on a struggling server. +O(events × 160) analysis — costs the fast path could never absorb — but +they read only the bounded evidence tiers 1–2 already deposited, so +asking the question is always safe on a server that is already in +trouble. ### Step 6 — tier 3b: the watchdog, when the loop can't confess -All previous tiers assume the event loop is running. When it isn't — -a command stuck in a loop, a module (FalkorDB!) blocking the main -thread — redis can't log anything, so the last tier interrupts from -outside the loop. `watchdogScheduleSignal` (`src/debug.c:2673`) arms -a one-shot `setitimer(ITIMER_REAL, ...)` for `watchdog-period` -milliseconds; the serverCron re-arms it each tick, so the SIGALRM -only actually fires if cron *stops running*. The handler, +> **In:** the failure case *all* prior tiers assume away — an event +> loop that has stopped running, so no tier can log anything. +> **Out:** an out-of-band SIGALRM stack dump, the last-resort artifact +> that for FalkorDB points straight into your wedged module. + +All previous tiers assume the event loop is running. When it isn't — a +command stuck in a loop, a module (FalkorDB!) blocking the main thread +— Redis can't log anything, so the last tier interrupts from *outside* +the loop. `watchdogScheduleSignal` (`src/debug.c:2673`) arms a one-shot +`setitimer(ITIMER_REAL, ...)` for `watchdog-period` milliseconds (a +one-shot: `it_interval` is zero). `serverCron` re-arms it every tick +(`src/server.c:1491`), so the SIGALRM only actually fires if cron +*stops running* — i.e. if the loop is genuinely wedged. The handler, `sigalrmSignalHandler` (`src/debug.c:2643`), logs -`--- WATCHDOG TIMER EXPIRED ---` and calls `logStackTrace` -(`src/debug.c:2115`) to dump where the main thread is stuck — from +`--- WATCHDOG TIMER EXPIRED ---` (`:2657`) and calls `logStackTrace` +(`src/debug.c:2115`) to dump where the main thread is standing — from inside a signal handler, using only async-signal-safe raw logging. + Disabled by default (`watchdog-period 0`): this tier's price is a -signal handler racing your code, so it is paid only when a human has +signal handler racing your code, so it is paid only once a human has already decided the server is sick. For FalkorDB: when a graph query -wedges the main thread, this stack trace is the first artifact you -will ever see — and it points into your module. +wedges the main thread, this stack trace is the first artifact you will +ever see — and it points into your module. Why it matters: it closes +the coverage gap — every other tier needs a working loop to file its +report; this one is the report the loop cannot make itself. ## Where each step lives in the code -All paths relative to `~/repos/redis`. FalkorDB's existing C -counterpart is `~/repos/FalkorDB/src/slow_log/slow_log.c` -(`SlowLog_Add`) — hold it against Step 2 while reading. +All paths relative to `~/repos/redis` at `redis@a176d1225`. FalkorDB's +existing C counterpart is `~/repos/FalkorDB/src/slow_log/slow_log.c` +(`SlowLog_Add`, `:190`; note the per-log `pthread_mutex_t lock` at +`:37` and the `pthread_mutex_lock` at `:222`) — hold it against Step 2 +while reading, because FalkorDB pays a mutex per query where +single-threaded Redis pays none. | Step | Anchor | What to see | |---|---|---| -| 2 | `src/slowlog.c:103` | `slowlogPushEntryIfNeeded` — `:104` negative disables, `:105` `>=` logs, then trim to max-len | -| 3 | `src/slowlog.c:28` | `slowlogCreateEntry` — arg-count cap, string truncation, dup-not-share | +| 2 | `src/slowlog.c:103` | `slowlogPushEntryIfNeeded` — `:104` negative disables, `:105` `>=` logs, `:110`–`:111` trim to max-len (oldest evicted) | +| 2 | `src/slowlog.c:70` | `se->id = server.slowlog_entry_id++` — monotonic ids, survive RESET | +| 3 | `src/slowlog.c:28` | `slowlogCreateEntry` — argc cap (`slowlog.h:13`), string truncation (`slowlog.h:14`), `dupStringObject` (`:64`), FLUSHALL race (`:58`) | | 4 | `src/latency.h:17` | `LATENCY_TS_LEN 160` — fixed ring per event | | 4 | `src/latency.h:50`, `:63` | `latencyStartMonitor` / `latencyAddSampleIfNeeded` — one compare when off (default 0 at `src/config.c:3271`) | -| 4 | `src/latency.c:63` | `latencyAddSample` — ring insert, max update, same-second coalescing | -| 5 | `src/latency.c:182` | `createLatencyReport` — LATENCY DOCTOR's rule engine (advice counter from ~`:200`) | -| 5 | `src/object.c:1421` | `getMemoryDoctorReport` — MEMORY DOCTOR, same pattern | -| 6 | `src/debug.c:2673`, `:2643` | `watchdogScheduleSignal` + `sigalrmSignalHandler` → `logStackTrace` (`:2115`) | +| 4 | `src/latency.c:63` | `latencyAddSample` — ring insert (`:88`), max update (`:77`), same-second coalescing (`:82`–`:84`) | +| 5 | `src/latency.c:182`, `:200`, `:207` | `createLatencyReport` — advice counter and literal "Dave" strings | +| 5 | `src/object.c:1421`, `:1491` | `getMemoryDoctorReport` — thresholds (`:1434`/`:1439`/`:1445`), literal "Sam" strings | +| 6 | `src/debug.c:2673`, `:2643` | `watchdogScheduleSignal` (re-armed by `server.c:1491`) + `sigalrmSignalHandler` → `logStackTrace` (`:2115`) | -Read order: slowlogPushEntryIfNeeded → slowlogCreateEntry → the two -latency.h macros → latencyAddSample → skim createLatencyReport for -its shape (don't read every rule) → the watchdog pair. Six anchors -carry the design; the doctors' rule bodies are trivia. +Read order: `slowlogPushEntryIfNeeded` → `slowlogCreateEntry` → the two +`latency.h` macros → `latencyAddSample` → skim `createLatencyReport` +for its shape (don't read every rule) → the watchdog pair. The anchors +carry the design; the doctors' individual rule bodies are trivia. ## Questions to answer in notes.md -1. Enumerate the exact SLOWLOG contract the topic-34 Rust stub's - tests encode (threshold comparison, disable value, eviction - order, id behavior across reset) and point to the line in - `slowlogPushEntryIfNeeded` / `slowlogCreateEntry` implementing - each clause. Which would you have gotten wrong from memory? -2. FalkorDB's `SlowLog_Add` runs per query, per graph, from - concurrent threads — redis's slowlog is single-threaded main-loop - code. What may the always-on tier cost under contention, and what - does that imply for the M34 Rust port (lock, sharded ring, - per-thread buffers)? +1. Enumerate the exact SLOWLOG contract the topic-34 Rust stub's tests + encode (threshold comparison, disable value, eviction order, id + behavior across reset) and point to the line in + `slowlogPushEntryIfNeeded` / `slowlogCreateEntry` implementing each + clause. Which would you have gotten wrong from memory? +2. FalkorDB's `SlowLog_Add` runs per query, per graph, from concurrent + threads (`slow_log.c:222` takes a mutex) — Redis's slowlog is + single-threaded main-loop code with no lock. What may the always-on + tier cost under contention, and what does that imply for the M34 + Rust port (lock, sharded ring, per-thread buffers)? 3. The latency monitor coalesces samples landing in the same second, - keeping only the max (`latency.c:82`). What information does this - deliberately throw away, and why is that the right trade for a + keeping only the max (`latency.c:82`–`:84`). What information does + this deliberately throw away, and why is that the right trade for a 160-slot ring whose consumer is a rule engine rather than a percentile dashboard? -4. Design GRAPH.DOCTOR: a `createLatencyReport`-style advice engine - for a graph database. List at least four rules and the cheap - evidence each needs deposited in advance (e.g., hot label scanned - without an index, result serialization dominating execution time, - matrix resize storms, BFS frontier repeatedly spilling). -5. The watchdog fires SIGALRM and walks the stack of whatever the - main thread is doing — including FalkorDB module code mid- - GraphBLAS-call. What must be true of the handler's code for this - to be safe, and what could a module-aware watchdog additionally - report (query text? graph key?) within those constraints? +4. Design GRAPH.DOCTOR: a `createLatencyReport`-style advice engine for + a graph database. List at least four rules and the cheap evidence + each needs deposited in advance (e.g., hot label scanned without an + index, result serialization dominating execution time, matrix resize + storms, BFS frontier repeatedly spilling). +5. The watchdog fires SIGALRM and walks the stack of whatever the main + thread is doing — including FalkorDB module code mid-GraphBLAS-call. + What must be true of the handler's code for this to be safe + (async-signal-safety), and what could a module-aware watchdog + additionally report (query text? graph key?) within those + constraints? ## Done when -- [ ] You can state the three-tier cost theorem and assign each of - the five surfaces (SLOWLOG, latency rings, two doctors, - watchdog) to its tier with its per-command cost. -- [ ] You can recite the SLOWLOG contract precisely — `>=` logs, - negative disables, oldest evicted at max-len, ids monotonic - across RESET — and your Rust stub passes its contract tests. -- [ ] You can explain why `slowlogCreateEntry` truncates and - duplicates arguments, including the FLUSHALL ASYNC race the - `:58` comment describes. -- [ ] You can trace what happens, function by function, when a - FalkorDB query blocks the main thread for 2× watchdog-period. +Answer each before unfolding it. + +- [ ] You can state the three-tier cost theorem and assign each of the five surfaces (SLOWLOG, latency rings, two doctors, watchdog) to its tier with its per-command cost. + +
Answer + + Tier 1, always-on: **SLOWLOG**, ~one integer compare on every command + (`slowlog.c:104`–`:105`). Tier 2, armed: the **latency monitor** + rings — one compare when `latency-monitor-threshold == 0` (the + default), and at most one ring-slot write per event per second when + armed (`latency.h:50`/`:63`, `latency.c:63`). Tier 3, on-demand: + **LATENCY DOCTOR** and **MEMORY DOCTOR** (walk bounded evidence, build + text — `latency.c:182`, `object.c:1421`) and the **software + watchdog** (a SIGALRM stack dump — `debug.c:2673`/`:2643`), all paid + only when a human asks. The invariant: expensive analysis never rides + the hot path; the hot path only deposits cheap, bounded evidence. + +
+ +- [ ] You can recite the SLOWLOG contract precisely — `>=` logs, negative disables, oldest evicted at max-len, ids monotonic across RESET — and your Rust stub passes its contract tests. + +
Answer + + From `slowlogPushEntryIfNeeded`: a negative `slowlog-log-slower-than` + (or `slowlog-max-len == 0`) disables logging and returns (`:104`); a + command whose `duration >= threshold` is logged, so equality logs, not + just strictly-greater (`:105`); entries are head-inserted (`:106`) and + trimmed from the tail (`:110`–`:111`), so the **oldest** is evicted + once `slowlog-max-len` is exceeded. Ids come from + `server.slowlog_entry_id++` in `slowlogCreateEntry` (`:70`): strictly + increasing and never reset by `SLOWLOG RESET` (which only clears the + list), so a poller can dedupe by id. + +
+ +- [ ] You can explain why `slowlogCreateEntry` truncates and duplicates arguments, including the FLUSHALL ASYNC race the `:58` comment describes. + +
Answer + + A slow command is frequently slow *because* its arguments are huge, so + logging them verbatim would let one entry pin megabytes. The entry + caps argument count at `SLOWLOG_ENTRY_MAX_ARGC = 32` (`slowlog.h:13`) + with a `"... (N more arguments)"` marker, and truncates any argument + over `SLOWLOG_ENTRY_MAX_STRING = 128` bytes (`slowlog.h:14`) with a + `"... (N more bytes)"` suffix — a 10 MB value keeps 128 bytes and + reports `10,485,632` more. It `dupStringObject`s non-shared args + (`:64`) rather than bumping a refcount because sharing an `robj` with + the keyspace means `FLUSHALL ASYNC` could free it on a background + thread while the log still references it (the `:58` comment). + +
+ +- [ ] You can trace what happens, function by function, when a FalkorDB query blocks the main thread for 2× watchdog-period. + +
Answer + + With `watchdog-period` set (non-zero), `watchdogScheduleSignal` + (`debug.c:2673`) has armed a one-shot `setitimer(ITIMER_REAL, ...)`. + Normally `serverCron` re-arms it every tick (`server.c:1491`), so it + never fires. When a FalkorDB query wedges the main thread, `serverCron` + stops running, the timer is not re-armed, and after `watchdog-period` + ms the kernel delivers SIGALRM to `sigalrmSignalHandler` + (`debug.c:2643`). The handler logs `--- WATCHDOG TIMER EXPIRED ---` + (`:2657`) and calls `logStackTrace` (`debug.c:2115`), dumping the + stack of the wedged main thread — which is inside your module's + GraphBLAS call — using only async-signal-safe logging. + +
## References -**Code** -- [redis](https://github.com/redis/redis) — cloned at - `~/repos/redis`; the anchors above are the read +**Code** (pinned at `redis@a176d1225`) +- [redis](https://github.com/redis/redis) — cloned at `~/repos/redis`; + the anchors above are the read. - FalkorDB's existing surface: - `~/repos/FalkorDB/src/slow_log/slow_log.c` (`SlowLog_Add`) — the C - implementation M34 ports to Rust + `~/repos/FalkorDB/src/slow_log/slow_log.c` (`SlowLog_Add`, `:190`) — + the C implementation M34 ports to Rust; note its per-log mutex. - This topic's `SlowLog` Rust stub and its contract tests — Step 2 - reshaped + reshaped. **Docs** -- [SLOWLOG GET](https://redis.io/docs/latest/commands/slowlog-get/) - — the observable contract (entry fields, ids, reset semantics) +- [SLOWLOG GET](https://redis.io/docs/latest/commands/slowlog-get/) — + the observable contract (entry fields, ids, reset semantics). - [Latency monitor](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency-monitor/) - — events, threshold, LATENCY DOCTOR usage + — events, threshold, LATENCY DOCTOR usage. diff --git a/topics/34-debugging/reading-rocksdb-perfcontext.md b/topics/34-debugging/reading-rocksdb-perfcontext.md index 7e5dc68..84c18a9 100644 --- a/topics/34-debugging/reading-rocksdb-perfcontext.md +++ b/topics/34-debugging/reading-rocksdb-perfcontext.md @@ -6,10 +6,11 @@ every query thread carries a thread-local struct of ~100 counters clock depending on a runtime dial (PerfLevel), and the cross-query picture lives in a second, global tier (Statistics) whose latency histograms squeeze all of u64 into 109 fixed buckets. The repo is -cloned at `~/repos/rocksdb`; this is a code-read, ~1.5h, aimed at -capstone M34 — a per-query perf context for the Rust engine whose -level-0 cost is provably zero. Build the ideas in order first; the -anchor table maps each to an exact file:line. +cloned at `~/repos/rocksdb`, pinned at `rocksdb@7c80a5a`; this is a +code-read, ~1.5h, aimed at capstone M34 — a per-query perf context for +the Rust engine whose level-0 cost is provably zero. Build the ideas in +order first; the anchor table maps each to an exact file:line, and +every snippet below is quoted from that pinned SHA with real gutters. ## The problem in one sentence @@ -22,6 +23,12 @@ literally nothing — not one branch more than an uninstrumented build. ### Step 1 — two tiers: per-query thread-local vs per-DB global +> **In:** the problem statement — "explain one query" *and* "watch the +> whole DB" are different questions. +> **Out:** the two structures (PerfContext, StatisticsImpl) that the +> rest of the chapter dissects; Steps 2–5 are the per-query tier, Step 6 +> is the global one. + RocksDB keeps two parallel metric systems, one per question: ``` @@ -33,24 +40,35 @@ RocksDB keeps two parallel metric systems, one per question: doing overall?" (tickers + histos) all queries aggregation ``` -PerfContext (`include/rocksdb/perf_context.h:305`, counters declared -in `PerfContextBase` at `:73`) is the per-query tier: block cache -hits, block read counts/bytes/nanos, `internal_key_skipped_count` -(iterator tombstone skips), memtable/WAL write times, mutex wait -nanos — `Reset()` before your query, run, read after. `StatisticsImpl` +PerfContext (`include/rocksdb/perf_context.h:305`, counters declared in +`PerfContextBase` at `:73`) is the per-query tier: block cache hits, +block read counts/bytes/nanos, `internal_key_skipped_count` (the count +of internal keys skipped during iteration — previous-key entries and +*updates hidden by tombstones*, but explicitly **not** the tombstones +themselves; the comment at `:136` says "the tombstones are not included +in this counter", they land in `internal_delete_skipped_count` at +`:149`), memtable/WAL write times, mutex wait nanos — `Reset()` before +your query, run, read after. `StatisticsImpl` (`monitoring/statistics_impl.h:42`) is the global tier: `recordTick` (`monitoring/statistics.cc:549`) folds increments from every thread into per-DB tickers and histograms. Why it matters: a p99 spike in the -global histogram tells you *that* something is slow; PerfContext on -one repro tells you *where its nanoseconds went*. M34 needs both -tiers, and conflating them (a mutex-protected global per-query map) -buys the worst of each. +global histogram tells you *that* something is slow; PerfContext on one +repro tells you *where its nanoseconds went*. M34 needs both tiers, and +conflating them (a mutex-protected global per-query map) buys the worst +of each. ### Step 2 — thread-local access: no synchronization, by construction +> **In:** the per-query tier named in Step 1 (PerfContext). +> **Out:** the reason it needs zero synchronization — each thread owns +> its struct — which Step 3 then makes *conditionally* cheap via the +> level dial. + `PerfContext* get_perf_context()` (`include/rocksdb/perf_context.h:342`) returns a pointer to a -`thread_local PerfContext` (`monitoring/perf_context_imp.h:19`): +`thread_local PerfContext` (`monitoring/perf_context_imp.h:19`). A +**thread-local** is a variable with one independent instance per +thread, so no two threads ever touch the same bytes: ``` thread A ──▶ its own PerfContext ──▶ plain `metric += value` @@ -59,93 +77,164 @@ buys the worst of each. A counter bump is a non-atomic add to memory only this thread ever touches; reading the results is the same thread inspecting its own -struct after the query returns. Why it matters: the per-query tier is -cheap *because* it never aggregates — aggregation is deferred to the -moment you copy the struct out. The Rust equivalent is a context owned -by the query task (or a `thread_local!` slot), not an `Arc>`. +struct after the query returns. (When RocksDB is built with +`NPERF_CONTEXT`, `get_perf_context()` instead returns one shared global +no-op object — see the contract comment at `:333`–`:341`.) Why it +matters: the per-query tier is cheap *because* it never aggregates — +aggregation is deferred to the moment you copy the struct out. The Rust +equivalent is a context owned by the query task (or a `thread_local!` +slot), not an `Arc>`. ### Step 3 — PerfLevel: the ladder, ordered by clock reads -Instrumentation cost is not uniform — a counter bump is ~1ns, a +> **In:** the thread-local counters from Step 2, which are cheap to +> bump but expensive to *time*. +> **Out:** the runtime `perf_level` dial whose rungs Step 4's macros and +> Step 5's timer branch on to decide whether to touch a clock at all. + +Instrumentation cost is not uniform — a counter bump is ~1 ns, a timestamp is a `clock_gettime`/`rdtsc` pair per scope, CPU-time clocks are syscalls — so the enable knob is a *ladder* (`include/rocksdb/perf_level.h:27`), each rung admitting a more expensive class of measurement: ``` - kDisable = 1 nothing - kEnableCount = 2 counters only — no clock reads - kEnableWait = 3 + time blocked inside RocksDB - kEnableTimeExceptForMutex = 4 + wall-clock timers everywhere - kEnableTimeAndCPUTimeExceptForMutex = 5 + CPU-time clocks - kEnableTime = 6 + mutex/condvar wait timing + kUninitialized = 0 (unset) + kDisable = 1 nothing + kEnableCount = 2 counters only — no clock reads + kEnableWait = 3 + time blocked inside RocksDB + kEnableTimeExceptForMutex = 4 + wall-clock timers everywhere + kEnableTimeAndCPUTimeExceptForMutex = 5 + CPU-time clocks + kEnableTime = 6 + mutex/condvar wait timing ``` -Mutex timing is last: it adds clock reads *inside critical sections*, -lengthening the very contention it measures. Why it matters: this is -the tax topic 34's bench lane 3 measures (bare loop → +clock pair → -+histogram.record → +slowlog check) — PerfLevel exists so production -sits at level 2 (counts are nearly free) and a repro session dials to -4+ without a rebuild. M34's dial should copy the ordering principle: -rungs sorted by cost class, not by feature. +The rung *names* encode the naming convention `perf_level.h` documents: +a `*_count`/`*_byte` counter is live at `kEnableCount` (2); a +`*_[wait|delay]_*` metric needs `kEnableWait` (3); a plain `*_time`/ +`*_nanos` needs `kEnableTimeExceptForMutex` (4); a `*_cpu_*_time` needs +`kEnableTimeAndCPUTimeExceptForMutex` (5); and a +`*_[mutex|condition]_*` metric needs the top rung `kEnableTime` (6). +Mutex timing is last deliberately: it adds clock reads *inside critical +sections*, lengthening the very contention it measures. Why it matters: +this is the tax topic 34's bench lane 3 measures (bare loop → +clock +pair → +histogram.record → +slowlog check) — PerfLevel exists so +production sits at level 2 (counts are nearly free) and a repro session +dials to 4+ without a rebuild. M34's dial should copy the ordering +principle: rungs sorted by cost class, not by feature. ### Step 4 — macros: the zero position is provably zero +> **In:** the `perf_level` dial from Step 3. +> **Out:** the two-layer gate — a compile-time `#if` for absence and a +> runtime `if (perf_level >= …)` for the cheap counters — that Step 5's +> timer completes for the expensive clock reads. + Instrumentation is written as macros, not calls, so absence can be -compiled (`monitoring/perf_context_imp.h`): +*compiled out* entirely: ```c -#if defined(NPERF_CONTEXT) -#define PERF_TIMER_GUARD(metric) // nothing. at all. -#else -#define PERF_TIMER_GUARD(metric) \ - PerfStepTimer perf_step_timer_##metric(&(perf_context.metric)); \ - perf_step_timer_##metric.Start(); -#endif +// monitoring/perf_context_imp.h:23 +23 #if defined(NPERF_CONTEXT) + // ... :25 every guard/counter macro expands to nothing ... +27 #define PERF_TIMER_GUARD(metric) + // ... +34 #define PERF_COUNTER_ADD(metric, value) +35 #define PERF_COUNTER_BY_LEVEL_ADD(metric, value, level) +37 #else + // real definitions follow: +45 #define PERF_TIMER_GUARD(metric) \ +46 PerfStepTimer perf_step_timer_##metric(&(perf_context.metric)); \ +47 perf_step_timer_##metric.Start(); + // ... :80 the counter add is the runtime-gated one ... +80 #define PERF_COUNTER_ADD(metric, value) \ +81 if (perf_level >= PerfLevel::kEnableCount) { \ +82 perf_context.metric += value; \ +83 } ``` -With `NPERF_CONTEXT` every macro (`:27` onward) expands to empty — -zero cost is a preprocessor fact, not a benchmark claim. Without it, -cost is gated at runtime: `PERF_COUNTER_ADD` (`:81`) is -`if (perf_level >= kEnableCount) perf_context.metric += value;`, and -the timer guards (`:45` wall-clock, `:88` per-LSM-level variant) check -the level before ever touching a clock. Why it matters: M34's bar is -"level 0 provably free" — in Rust, a `#[cfg(feature = "perf")]` macro -for the compile-out, plus a runtime branch on a thread-local level for -everything the feature flag keeps. +Two layers of gating, and it is worth keeping them straight. First, +compile-time: with `NPERF_CONTEXT` defined, *every* macro (`:25`–`:35`) +expands to nothing — zero cost is a preprocessor fact, not a benchmark +claim. Second, runtime, for the builds that keep instrumentation: +`PERF_COUNTER_ADD` (`:80`) wraps its add in +`if (perf_level >= kEnableCount)`, and `PERF_COUNTER_BY_LEVEL_ADD` +(`:87`) is the per-LSM-level *counter* variant (a map keyed by level, +also gated at `>= kEnableCount`) — note it is a **counter**, not a +timer. The **timer** guard `PERF_TIMER_GUARD` (`:45`) does something +subtler: it unconditionally constructs a `PerfStepTimer` and calls +`Start()`, and the level check that decides whether a clock is read +lives *inside* that object (Step 5), not in this macro. Why it matters: +M34's bar is "level 0 provably free" — in Rust, a +`#[cfg(feature = "perf")]` macro for the compile-out, plus a runtime +branch on a thread-local level for everything the feature flag keeps. ### Step 5 — PerfStepTimer: RAII so a scope can't leak its time +> **In:** the `PERF_TIMER_GUARD` macro from Step 4, which drops a +> `PerfStepTimer` on the stack. +> **Out:** the RAII object whose constructor caches the level check and +> whose destructor guarantees the interval is recorded — the mechanism +> behind Step 4's claim that no clock is read below level 4. + `PerfStepTimer` (`monitoring/perf_step_timer.h:13`) is what the guard -macros drop on the stack: the constructor evaluates -`perf_level >= enable_level` once and caches it; `Start()` reads the -clock only if enabled; the destructor (`:29`) calls `Stop()`, adding -`now - start_` to the target `uint64_t*` — optionally `RecordTick`ing -the same duration into Statistics: two tiers, one clock read. +macros drop on the stack. The constructor evaluates +`perf_level >= enable_level` exactly once and caches it; `Start()` reads +the clock only if that cache is set (or a `Statistics*` sink is +attached); the destructor calls `Stop()`, adding `now - start_` to the +target `uint64_t*`: -``` - { PERF_TIMER_GUARD(block_read_time); ┐ ctor: level check - ... read the block ... │ Start(): clock #1 - } ◀── destructor fires on ANY exit ──────┘ Stop(): clock #2, += +```c +// monitoring/perf_step_timer.h:15 +15 explicit PerfStepTimer( +16 uint64_t* metric, SystemClock* clock = nullptr, bool use_cpu_time = false, +17 PerfLevel enable_level = PerfLevel::kEnableTimeExceptForMutex, ...) +19 : perf_counter_enabled_(perf_level >= enable_level), // level check, cached once + // ... +29 ~PerfStepTimer() { Stop(); } // fires on ANY scope exit +31 void Start() { +32 if (perf_counter_enabled_ || statistics_ != nullptr) { +33 start_ = time_now(); // clock read, only if enabled +34 } +35 } ``` +The default `enable_level` is `kEnableTimeExceptForMutex` (4), so a +plain `PERF_TIMER_GUARD` reads *no clock* until the dial reaches 4 — +below that, `perf_counter_enabled_` is false and `Start()`/`Stop()` are +branches over nothing. `Stop()` optionally `RecordTick`s the same +duration into `Statistics`, so one clock-read pair can feed both tiers. `Measure()` (`:37`) restarts the interval mid-scope for multi-step -timing. Why it matters: early return, `?`, exception — the scope -closes, the time lands; a branch that skips the stop call is -unrepresentable. In Rust this is `Drop`, the natural shape for M34's +timing. + +``` + { PERF_TIMER_GUARD(block_read_time); ┐ ctor: cache perf_level >= 4 + ... read the block ... │ Start(): clock #1 (iff enabled) + } ◀── destructor fires on ANY exit ──────┘ Stop(): clock #2, += into struct +``` + +Why it matters: early return, `?`, exception — the scope closes, the +time lands; a branch that skips the stop call is unrepresentable. In +Rust this is `Drop`, the natural shape for M34's parse/plan/execute/serialize step timers. ### Step 6 — 109 buckets for all of u64: O(1) percentiles, exact merges +> **In:** the global tier (`StatisticsImpl`) named in Step 1, which must +> summarize latencies from all threads. +> **Out:** the fixed-bucket histogram that makes per-thread merges exact +> and percentiles O(1) — the structure M34's `LogHistogram` stub +> reimplements. + The global tier's latency histograms never store samples. `HistogramBucketMapper` (`monitoring/histogram.h:21`) precomputes a fixed set of geometrically growing bucket limits covering all of u64; `HistogramStat` (`:46`) is min/max/count/sum/sum-of-squares plus `std::atomic_uint_fast64_t buckets_[109]` (`:84`) — a fixed array, the -comment explains, so the struct needs no dynamic allocation and can -live in thread-local storage. `Add` is: map value to bucket index, one -relaxed atomic increment. `Percentile(p)` walks 109 buckets and -interpolates inside the one holding the p-th sample. +comment at `:76`–`:78` explains, so the struct needs no dynamic +allocation and can live in thread-local storage. `Add` maps the value +to a bucket index and does one relaxed atomic increment; `Percentile(p)` +walks the 109 buckets and interpolates inside the one holding the p-th +sample. ``` value ──IndexForValue──▶ bucket i ──fetch_add──▶ buckets_[i] @@ -153,30 +242,40 @@ interpolates inside the one holding the p-th sample. p50/p99/p999 ◀── walk 109 counters, interpolate ────┘ ``` -Because bucket boundaries are fixed at compile time, merging two -histograms (`HistogramImpl::Merge`, class at `:110`) is 109 exact -additions — per-thread histograms combine losslessly; the price is -relative error bounded by the geometric growth ratio. Why it matters: -this is precisely topic 34's `LogHistogram` stub — HdrHistogram -generalizes it with a `sub_bits` error knob — and M34 should record -step latencies this way, never as samples. +The bucket ratio is worth deriving, because it *is* the accuracy story +(and Question 5). To cover u64 (≈ `2^64`) in 109 geometric buckets +starting near 1, the ratio `r` satisfies `r^109 ≈ 2^64`, i.e. +`r ≈ 2^(64/109) ≈ 2^0.587 ≈ 1.5` — which is exactly the `1.5 × previous` +growth `histogram.cc:28` uses (rounded to nice values). A value lands in +some bucket `[L, ~1.5L)`, so representing it by the bucket limit is up +to `(1.5 − 1) = 50%` high at the low edge; using the geometric midpoint +(`≈ 1.22 L`) bounds the relative error near `±22%`, and the linear +interpolation `Percentile` does within the bucket brings the typical +error well below that. Because bucket boundaries are fixed at compile +time, merging two histograms (`HistogramImpl::Merge`, class at `:110`) +is 109 exact additions — per-thread histograms combine losslessly; the +price is that bounded relative error. Why it matters: this is precisely +topic 34's `LogHistogram` stub — HdrHistogram generalizes it with a +`sub_bits` knob that subdivides each power-of-two band into `2^sub_bits` +linear sub-buckets to drive the error down — and M34 should record step +latencies this way, never as samples. ## Where each step lives in the code -All paths relative to `~/repos/rocksdb`. +All paths relative to `~/repos/rocksdb` at `rocksdb@7c80a5a`. | Step | Anchor | What to see | |---|---|---| -| 1 | `include/rocksdb/perf_context.h:73` | `PerfContextBase` — read 30 counters' comments; note count/byte/time naming | +| 1 | `include/rocksdb/perf_context.h:73` | `PerfContextBase` — read the counters' comments; note count/byte/time naming and `:136` (tombstones excluded from `internal_key_skipped_count`) | | 1 | `include/rocksdb/perf_context.h:305` | `PerfContext` — `Reset()`, `ToString(exclude_zero_counters)`, per-level map | -| 1 | `monitoring/statistics_impl.h:42` + `monitoring/statistics.cc:549` | `StatisticsImpl` / `recordTick` — the global tier | -| 2 | `include/rocksdb/perf_context.h:342` | `get_perf_context()` — thread-local contract in the comment above it | +| 1 | `monitoring/statistics_impl.h:42` + `monitoring/statistics.cc:549` | `StatisticsImpl` / `recordTick` — the global tier, per-core aggregation | +| 2 | `include/rocksdb/perf_context.h:342` | `get_perf_context()` — thread-local contract in the comment above it (`:333`–`:341`) | | 3 | `include/rocksdb/perf_level.h:27` | the `PerfLevel` ladder + naming-convention comments per rung | -| 4 | `monitoring/perf_context_imp.h:27,:45,:81,:88` | `NPERF_CONTEXT` empty expansions; `PERF_TIMER_GUARD`; level-gated counter adds | -| 5 | `monitoring/perf_step_timer.h:13,:29` | `PerfStepTimer` ctor's cached level check; destructor → `Stop()` | -| 6 | `monitoring/histogram.h:21,:46,:84,:110` | `HistogramBucketMapper`; `HistogramStat` with `buckets_[109]`; `HistogramImpl` | +| 4 | `monitoring/perf_context_imp.h:25,:45,:80,:87` | `NPERF_CONTEXT` empty expansions; `PERF_TIMER_GUARD` (timer); `PERF_COUNTER_ADD` / `PERF_COUNTER_BY_LEVEL_ADD` (level-gated counters) | +| 5 | `monitoring/perf_step_timer.h:15,:19,:29` | `PerfStepTimer` ctor's cached `perf_level >= enable_level`; destructor → `Stop()` | +| 6 | `monitoring/histogram.h:21,:46,:84,:110` | `HistogramBucketMapper`; `HistogramStat` with `buckets_[109]`; `HistogramImpl`; ratio at `histogram.cc:28` | -Read order: perf_level.h (whole file, 60 lines) → perf_context_imp.h +Read order: perf_level.h (whole file, ~60 lines) → perf_context_imp.h (whole file — the macros ARE the design) → perf_step_timer.h (whole file) → skim PerfContextBase's counters → histogram.h. Then grep `PERF_TIMER_GUARD(get_from_memtable_time)` for a live call site. @@ -189,52 +288,109 @@ file) → skim PerfContextBase's counters → histogram.h. Then grep (count + bytes), per-operator intermediate-record counts, result serialization bytes — marking level-2 counts vs level-4 timers. 2. `PERF_COUNTER_ADD` branches on `perf_level >= kEnableCount` for a - ~1ns add — is the branch cheaper than the unconditional add? Use + ~1 ns add — is the branch cheaper than the unconditional add? Use bench lane 3's numbers to argue when the level check itself is the tax, and what a Rust `const LEVEL: u8` generic would change. 3. `PerfStepTimer::Stop()` can feed both tiers (the `uint64_t*` metric - and `RecordTick` on a `Statistics*`) from one clock-read pair. - Where in M34's step timers would you replicate this, and where must - the tiers stay decoupled? -4. Mutex wait timing is the top rung (`kEnableTime = 6`) because - clocks inside critical sections perturb contention. Which M34 - measurement has the same observer effect (hint: per-operator timers - inside a tight matrix loop), and what is your rung 6? -5. RocksDB's 109 buckets vs HdrHistogram's `sub_bits`: compute the - worst-case relative error of a geometric ladder spanning u64 in 109 - buckets, and pick the bucket count your `LogHistogram` needs to - keep p99 error under 5% for latencies between 1us and 10s. + and `RecordTick` on a `Statistics*`) from one clock-read pair. Where + in M34's step timers would you replicate this, and where must the + tiers stay decoupled? +4. Mutex wait timing is the top rung (`kEnableTime = 6`) because clocks + inside critical sections perturb contention. Which M34 measurement + has the same observer effect (hint: per-operator timers inside a + tight matrix loop), and what is your rung 6? +5. RocksDB's 109 buckets vs HdrHistogram's `sub_bits`: starting from + `r ≈ 2^(64/109) ≈ 1.5`, compute the worst-case relative error of the + geometric ladder, and pick the bucket count (or `sub_bits`) your + `LogHistogram` needs to keep p99 error under 5% for latencies between + 1 µs and 10 s. ## Done when -- [ ] You can trace `PERF_TIMER_GUARD(block_read_time)` end to end — - macro expansion → ctor's cached level check → `Start()` clock - read → destructor `Stop()` adding into the thread-local struct — - and name the two settings under which no clock is ever read - (NPERF_CONTEXT; level < 4). -- [ ] Given a counter name (`*_count`, `*_time`, `*_cpu_*`, mutex/wait - metrics), you can say from perf_level.h's naming conventions at - which rung it becomes live. -- [ ] You can explain why `HistogramStat` merges are exact while its - percentiles are approximate, in one sentence each. -- [ ] M34's design doc names its two tiers, its ladder rungs in cost - order, and its provably-zero mechanism. +Answer each before unfolding it. + +- [ ] You can trace `PERF_TIMER_GUARD(block_read_time)` end to end — macro expansion → ctor's cached level check → `Start()` clock read → destructor `Stop()` adding into the thread-local struct — and name the two settings under which no clock is ever read (NPERF_CONTEXT; level < 4). + +
Answer + + `PERF_TIMER_GUARD(block_read_time)` expands (`perf_context_imp.h:45`) + to a stack `PerfStepTimer perf_step_timer_block_read_time(&perf_context.block_read_time)` + followed by `.Start()`. The constructor (`perf_step_timer.h:15`) + caches `perf_counter_enabled_ = (perf_level >= enable_level)`, whose + default `enable_level` is `kEnableTimeExceptForMutex` (4). `Start()` + (`:31`) reads the clock only if `perf_counter_enabled_ || + statistics_ != nullptr`. At scope exit the destructor (`:29`) runs + `Stop()`, which adds `now - start_` into `perf_context.block_read_time` + (the thread-local struct). No clock is read when the build defines + `NPERF_CONTEXT` (the macro is empty) or when `perf_level < 4` (the + cached flag is false and no `Statistics` sink is attached). + +
+ +- [ ] Given a counter name (`*_count`, `*_time`, `*_cpu_*`, mutex/wait metrics), you can say from perf_level.h's naming conventions at which rung it becomes live. + +
Answer + + From `perf_level.h`'s per-rung naming comments: `*_count`/`*_byte` + counters are live at `kEnableCount` (2); `*_wait_*` / `*_delay_*` + metrics need `kEnableWait` (3); plain `*_time` / `*_nanos` timers need + `kEnableTimeExceptForMutex` (4); `*_cpu_*_time` / `*_cpu_*_nanos` need + `kEnableTimeAndCPUTimeExceptForMutex` (5); and + `*_mutex_*` / `*_condition_*` wait metrics need the top rung + `kEnableTime` (6). The ladder is ordered by cost class — count, wait, + wall-time, CPU-time, mutex-time — so each rung admits a strictly more + expensive measurement. + +
+ +- [ ] You can explain why `HistogramStat` merges are exact while its percentiles are approximate, in one sentence each. + +
Answer + + Merges are exact because every histogram shares the same 109 + compile-time-fixed bucket boundaries, so combining two is just 109 + element-wise `uint64` additions with no re-bucketing and no lost + counts (`HistogramImpl::Merge`). Percentiles are approximate because a + bucket only records *how many* samples fell in `[L, ~1.5L)`, not their + values, so `Percentile` must interpolate within the bucket — leaving a + bounded relative error set by the ~1.5× geometric ratio (≈20% at + midpoint before interpolation). + +
+ +- [ ] M34's design doc names its two tiers, its ladder rungs in cost order, and its provably-zero mechanism. + +
Answer + + Two tiers: a per-query, thread-local **PerfContext** analogue (plain + non-atomic counters, owned by the query task) and a per-DB global + **Statistics** analogue (per-core tickers + fixed-bucket histograms). + Ladder in cost order, mirroring PerfLevel: counts (≈free) → in-engine + wait time → wall-clock timers → CPU-time → mutex/critical-section + timing (last, because it perturbs contention). Provably-zero + mechanism: a compile-time gate (`#[cfg(feature = "perf")]` in Rust, + RocksDB's `NPERF_CONTEXT`) that expands all instrumentation to nothing, + plus a runtime `perf_level` branch (or a `const LEVEL` generic) for the + builds that keep it — so level 0 costs not one branch more than an + uninstrumented build. + +
## References -**Code** +**Code** (pinned at `rocksdb@7c80a5a`) - [RocksDB](https://github.com/facebook/rocksdb) — cloned at - `~/repos/rocksdb`; the anchors above are the read -- [HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) — Step - 6 generalized, with an explicit precision knob (`sub_bits`) + `~/repos/rocksdb`; the anchors above are the read. +- [HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) — Step 6 + generalized, with an explicit precision knob (`sub_bits`). **Docs** - [RocksDB wiki: Perf Context and IO Stats Context](https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context) - — the usage pattern: SetPerfLevel → Reset → query → ToString + — the usage pattern: SetPerfLevel → Reset → query → ToString. **Related** - This topic's `experiments/` — the `LogHistogram` stub (Step 6 with - `sub_bits`) and bench lane 3 (the observability tax Step 3 controls) + `sub_bits`) and bench lane 3 (the observability tax Step 3 controls). - Capstone M34 — per-query perf context for the Rust engine: step timers + operator counters behind a PerfLevel-style dial; level 0 - provably free, full level < 5% overhead on the bench suite + provably free, full level < 5% overhead on the bench suite. diff --git a/topics/34-debugging/reading-rr.md b/topics/34-debugging/reading-rr.md index c7c63be..0e25150 100644 --- a/topics/34-debugging/reading-rr.md +++ b/topics/34-debugging/reading-rr.md @@ -2,15 +2,23 @@ Topic 34's premise is that production evidence is perishable — a heap corruption you saw once at 3am is gone by the time you attach a -debugger. rr (Mozilla, USENIX ATC 2017) is the sharpest answer to -that: record a failing run once, on stock Linux with no kernel module, -no root, no special hardware, and replay it deterministically as many -times as it takes — including *backwards*, with gdb watchpoints -running in reverse. If topic 16's deterministic simulation testing -controls nondeterminism *before* you ship, rr captures it *after* — -the same move (put all nondeterminism behind one interface and log -it), applied at the opposite end of the lifecycle. This chapter builds -the six engineering ideas that make it deployable; then a section map. +debugger. rr (Mozilla, USENIX ATC 2017) is the sharpest answer to that: +record a failing run once, on stock Linux with no kernel module, no +root, no special hardware, and replay it deterministically as many +times as it takes — including *backwards*, with gdb watchpoints running +in reverse. If topic 16's deterministic simulation testing controls +nondeterminism *before* you ship, rr captures it *after* — the same +move (put all nondeterminism behind one interface and log it), applied +at the opposite end of the lifecycle. This chapter builds the six +engineering ideas that make it deployable; then a section map. + +This is a *paper*, not a repo in this course's pin table, so every claim +below cites a **section or figure** of the paper rather than a +`file:line`. Cite the **arXiv version, 1705.05937** — note that this is +the ~21-page **Extended Technical Report** (its title page says so, and +§1 states "this extended technical report elaborates on published +[work]"); the ATC'17 conference paper is the shorter ~14-page version. +Section numbers below are the extended report's. ## The problem in one sentence @@ -24,9 +32,15 @@ one deterministic hardware counter. ### Step 1 — the record boundary is the user/kernel interface +> **In:** nothing yet — this step draws the line everything else is +> defined relative to. +> **Out:** the two nondeterminism sources (syscall results, async-event +> timing) that Steps 2–5 each pin down. + Everything below the syscall line is the environment; everything above -it is the program. For a user-space program confined to a single core, -exactly two kinds of nondeterminism cross that line: +it is the program. A **tracee** is the process rr records. For a +user-space tracee confined to a single core, exactly two kinds of +nondeterminism cross that line (paper §2.1): ``` ┌─────────────────────────────────────┐ @@ -43,18 +57,26 @@ exactly two kinds of nondeterminism cross that line: Record (a) the result and side effects of every syscall and (b) the exact point at which every async event was delivered, and replay is -just "run the same code, feed it the same answers at the same -points." Why it matters: this is the identical abstraction bet as -topic 16's simulation harness — DST mocks the interface so tests are -deterministic by construction; rr logs the interface so one real -execution becomes deterministic in retrospect. +just "run the same code, feed it the same answers at the same points." +Why it matters: this is the identical abstraction bet as topic 16's +simulation harness — DST mocks the interface so tests are deterministic +by construction; rr logs the interface so one real execution becomes +deterministic in retrospect. ### Step 2 — one core, one thread at a time +> **In:** the "async events" source (b) from Step 1 — specifically +> context switches, the hardest to reproduce. +> **Out:** the single-core scheduling decision that demotes a context +> switch to just another recordable async event, and the bug classes it +> costs you. + rr pins all tracee threads to a single core (`sched_setaffinity`) and -runs them one at a time. This is what makes (b) tractable: a context -switch is just another async event with a recordable delivery point, -not a source of true parallelism. The trade is stark: +runs them one at a time (§2.2). This is what makes (b) tractable: a +context switch is just another async event with a recordable delivery +point, not a source of true parallelism. The paper is explicit that +this makes weak-memory reordering unobservable and penalizes +high-parallelism workloads. The trade is stark: | Bug class | Under rr | |------------------------------|-----------------------------------| @@ -65,35 +87,57 @@ not a source of true parallelism. The trade is stark: | Parallel-only perf pathology | invisible — you serialized it | Why it matters for a database engine: your executor is a thread pool -hammering shared version chains and matrices. rr will still catch -many races (as interleavings at switch points), but a bug that only -exists because two cores genuinely raced on a cache line will never -fire under rr. Know which class you're hunting before you reach for -the tool. +hammering shared version chains and matrices. rr will still catch many +races (as interleavings at switch points), but a bug that only exists +because two cores genuinely raced on a cache line will never fire under +rr. Know which class you're hunting before you reach for the tool. ### Step 3 — RCB + registers = an execution point +> **In:** the requirement from Step 1(b) — replay must re-deliver each +> async event at the *exact* instruction it originally hit. +> **Out:** the (RCB, registers) coordinate that names that instruction, +> and the hardware constraint (Step 5 relies on it) that this coordinate +> demands. + To replay "SIGSEGV was delivered *here*," rr needs a coordinate system -for points in an execution. Instruction counters on real CPUs are -mostly nondeterministic noise, but one counter — **retired conditional -branches (RCB)** — is deterministic on modern Intel CPUs. rr -identifies an execution point as the pair **(RCB count, full register -state)**: run replay forward until the counter says you're in the -right neighborhood, then match registers to land on the exact -instruction, and re-deliver the async event precisely there. Two -corollaries the paper is honest about: RDTSC must be trapped (via -`prctl`) so the program can't read a nondeterministic clock behind -rr's back, and at the time of the paper rr could not work on ARM — no -suitable deterministic counter existed there. Why it matters: the -entire replay guarantee hangs on one line in a CPU errata sheet; +for points in an execution. An **execution point** is a specific +dynamic instruction instance, not just a code address (the same address +recurs every loop iteration). Instruction counters on real CPUs are +mostly nondeterministic — the paper notes "instructions retired" is +unusable because a page-faulting instruction is restarted and counted +twice (§2.4.1) — but "modern Intel CPUs have exactly one deterministic +performance counter: retired conditional branches (RCB)" (§2.4.1). RCB +alone does not uniquely identify a point, so rr pairs it with "the +complete state of general-purpose registers (including the program +counter)": the execution point is the pair **(RCB count, full register +state)**. Run replay forward until the counter reaches the target +neighborhood, then match registers to land on the exact instruction, +and re-deliver the async event precisely there. + +Two corollaries the paper is honest about. RDTSC (the timestamp +instruction) must be trapped and emulated so the program can't read a +nondeterministic clock behind rr's back (§2.6). And the whole +(RCB, registers) scheme rests on RCB being deterministic — see Step 5's +ARM note for where that footing gives way. Why it matters: the entire +replay guarantee hangs on one line in a CPU errata sheet; "deployability" includes being at the mercy of silicon you don't control. ### Step 4 — in-process syscall buffering (the performance trick) -A ptrace stop costs 4 context switches per syscall (tracee→kernel→ -rr→kernel→tracee). At database or browser syscall rates that's fatal. -rr's fix: intercept common syscalls *in-process*, without any stop. +> **In:** the "record every syscall" requirement from Step 1(a), which +> is ruinously slow if each syscall traps to rr. +> **Out:** the seccomp-bpf + RR-page + instruction-rewrite fast path +> that makes recording sub-2×, and the scratch buffers Step 5 injects +> from. + +A ptrace stop costs 4 context switches per syscall (§3, Fig 1: +tracee→kernel→rr→kernel→tracee — two blocking ptrace notifications). At +database or browser syscall rates that's fatal; the paper notes that +for common syscalls this one context-switch cost "dwarfs" the syscall +itself. rr's fix: intercept common syscalls *in-process*, without any +stop (§3.1–3.2). ```mermaid graph TD @@ -104,41 +148,62 @@ graph TD ``` The mechanism: a seccomp-bpf filter makes every syscall trap *unless* -it is issued from a designated "RR page"; rr then patches the hot call -sites — rewriting 2-byte `syscall` instructions into 5-byte `call` -instructions into a stub — so common syscalls run the real syscall -from the RR page and log their results into a **syscall buffer**, -never waking rr at all. Syscall outputs are redirected into **scratch -buffers** during recording, so that during replay the recorded bytes -can be injected in their place. Why it matters: this is classic +it is issued from a designated "RR page" (§3.2); rr then patches the hot +call sites — the x86 `syscall` instruction is 2 bytes, rewritten into a +5-byte `call` into an injected stub (§3.1) — so common syscalls run the +real syscall from the RR page and log their results into a **syscall +buffer**, never waking rr at all. Syscall outputs are redirected into +scratch buffers during recording, so that during replay the recorded +bytes can be injected in their place. Why it matters: this is classic hot-path engineering — keep the slow, general ptrace path as the correctness fallback, carve a monitored fast path for the common case. -It's the same shape as your syscall-heavy Redis module I/O: the -overhead story is decided entirely by what happens per-event on the -hot path. +It's the same shape as your syscall-heavy Redis module I/O: the overhead +story is decided entirely by what happens per-event on the hot path. ### Step 5 — replay: inject, don't re-execute -During replay, syscalls are not re-executed against the kernel — -their recorded results are injected from the log, and async events are +> **In:** the recorded log from Step 4 (syscall results) and the +> (RCB, registers) points from Step 3 (async events). +> **Out:** a fully deterministic re-execution — the "perishable evidence +> made permanent" payoff, and the input Step 6's reverse debugging +> replays over. + +During replay, syscalls are not re-executed against the kernel — their +recorded results are injected from the log (§3.8), and async events are re-delivered at their recorded (RCB, registers) points. The program's -own computation, being deterministic given those inputs (Step 1), -takes care of itself. Consequence: replay is *fully* deterministic — -the same bug fires at the same instruction every single time, no -matter how flaky the original repro was. A once-in-a-thousand-runs -crash, captured once, becomes a 100%-reproducible artifact you can -attach to a bug report. Why it matters: this converts debugging from -statistics back into logic — the perishable production evidence of -this topic's framing, made permanent. +own computation, being deterministic given those inputs (Step 1), takes +care of itself. Consequence: replay is *fully* deterministic — the same +bug fires at the same instruction every single time, no matter how flaky +the original repro was. A once-in-a-thousand-runs crash, captured once, +becomes a 100%-reproducible artifact you can attach to a bug report. + +The load-bearing caveat, and the correction to a common misconception: +rr's ARM port *failed* not because ARM lacks an RCB-like counter, but +because "all ARM atomic memory operations use the load-linked/ +store-conditional approach, which is inherently nondeterministic" — the +conditional store can fail due to non-user-space-observable activity +(e.g. a hardware interrupt), so retired-branch/instruction counts for +code doing atomics are nondeterministic (§5.1). On x86(-64), atomics +like compare-and-swap are deterministic in user-space state, so RCB +holds. Why it matters: this converts debugging from statistics back into +logic — the perishable production evidence of this topic's framing, made +permanent — but only on hardware where the counter keeps its promise. ### Step 6 — reverse execution: replay + checkpoints under gdb +> **In:** the deterministic replay from Step 5. +> **Out:** the time-travel workflow (backwards watchpoints) that turns a +> multi-day "who wrote this?" bisect into one `reverse-continue` — this +> topic's exercise 5. + "Deterministic replay" upgrades gdb from a state inspector into a time machine. rr serves the gdb remote protocol; `reverse-continue` and -backwards watchpoints are implemented on top of replay plus -checkpoints: restore an earlier checkpoint, replay forward, and use -determinism to stop just before the point you came from. The killer -workflow — exercise 5 in this topic's README — is: +backwards watchpoints are implemented on top of replay plus checkpoints. +A **checkpoint** is a cheap snapshot: the paper takes them by `fork`ing +the replay process to clone its address space, typically in under ~10 ms +(§6.2). Reverse execution then restores the nearest earlier checkpoint, +replays forward, and uses determinism to stop just before the point you +came from. The killer workflow — exercise 5 in this topic's README — is: ``` 1. rr record ./test --seed 42 # capture the failing run once @@ -150,81 +215,145 @@ workflow — exercise 5 in this topic's README — is: Why it matters: "who scribbled on this version chain?" is normally a days-long bisect; under rr it is one watchpoint and one -reverse-continue, because the write that corrupted the value must -happen at the same execution point in every replay. +reverse-continue, because the write that corrupted the value must happen +at the same execution point in every replay. ## How to read the paper (with the concepts in hand) -USENIX ATC 2017 (arXiv:1705.05937), ~14 pages; budget ~1.5h. +arXiv:1705.05937 (the extended technical report, ~21 pp; the ATC'17 +conference version is ~14 pp); budget ~1.5h. -- **Abstract + intro** (10 min) — the deployability thesis: ptrace, +- **Abstract + §1 intro** (10 min) — the deployability thesis: ptrace, no kernel modules, no root, no special hardware (Step 1's boundary, and why every prior system failed to spread). Read it as a systems paper about *constraints*, not features. -- **Design/approach sections** (20 min) — single-core scheduling - (Step 2), the two nondeterminism sources, and the (RCB, registers) - execution-point scheme (Step 3). This is the conceptual core; the - rest is engineering to make it fast. -- **Syscall buffering / performance engineering** (25 min) — the - seccomp-bpf + RR-page + instruction-rewriting machinery, scratch - buffers, RDTSC trapping (Step 4). Slowest, densest, most valuable +- **§2 Design** (20 min) — single-core scheduling (§2.2, Step 2), the + two nondeterminism sources (§2.1), and the (RCB, registers) + execution-point scheme (§2.4, Step 3). This is the conceptual core; + the rest is engineering to make it fast. +- **§3 In-process system-call interception** (25 min) — the seccomp-bpf + + RR-page + instruction-rewriting machinery, scratch buffers, and + (from §2.6) RDTSC trapping (Step 4). Slowest, densest, most valuable part for you — map it onto "what would intercepting FalkorDB's hot syscalls cost." -- **Replay and reverse execution** (15 min) — injection instead of - re-execution, gdb integration, checkpoints (Steps 5-6). -- **Evaluation** (15 min) — where "under 2× for the workloads Mozilla - cared about" comes from (Firefox test suites; cheap enough for CI). - Check which workloads before generalizing to a database server. -- **Limitations** (5 min) — weak memory, ARM, parallelism. These are - Step 2 and Step 3's trades stated by the authors themselves. +- **§3.8 + reverse execution / §6.2** (15 min) — injection instead of + re-execution, gdb integration, fork-based checkpoints (Steps 5–6). +- **§4 Results (Fig 5)** (15 min) — where "under 2× for the workloads + Mozilla cared about" comes from (Firefox test suites; cheap enough for + CI). Check which workloads before generalizing to a database server. +- **§5 Constraints** (5 min) — weak memory, ARM (§5.1), shared memory + (§5.2). These are Step 2 and Step 5's trades stated by the authors + themselves. ## Questions to answer in notes.md 1. rr and topic 16's deterministic simulation testing both put nondeterminism behind an interface — but DST *replaces* the - environment and rr *records* it. For FalkorDB, which failure - classes does each end of the lifecycle catch that the other - structurally cannot? + environment and rr *records* it. For FalkorDB, which failure classes + does each end of the lifecycle catch that the other structurally + cannot? 2. One-thread-at-a-time on one core is both rr's superpower and its - blind spot for a database engine: which FalkorDB bug classes - (MVCC version-chain races, lock-free index ops, GraphBLAS - parallel kernels) stay observable as switch-timing bugs, and which - become invisible because they need true multi-core interleaving or - weak-memory reordering? + blind spot for a database engine: which FalkorDB bug classes (MVCC + version-chain races, lock-free index ops, GraphBLAS parallel kernels) + stay observable as switch-timing bugs, and which become invisible + because they need true multi-core interleaving or weak-memory + reordering? 3. Reconstruct the per-syscall cost argument: what exactly do the 4 - context switches of a ptrace stop cost, and how do seccomp-bpf + - the RR page + the 2-byte-to-5-byte rewrite eliminate them for the - common case? What limits which syscalls can take the fast path? + context switches of a ptrace stop cost (§3, Fig 1), and how do + seccomp-bpf + the RR page + the 2-byte-to-5-byte rewrite eliminate + them for the common case? What limits which syscalls can take the + fast path? 4. Why is (RCB count, register state) sufficient to identify a unique - execution point, and what would break if the counter overcounted - by even one on rare occasions? Connect to why ARM was unsupported. + execution point in practice, and what would break if the counter + overcounted by even one on rare occasions? Connect to why the ARM + port failed (§5.1). 5. After doing exercise 5 (`rr record` a failing seeded test, then - watchpoint + reverse-continue to the corrupting write): how long - did the same hunt take you last time without rr, and what recording + watchpoint + reverse-continue to the corrupting write): how long did + the same hunt take you last time without rr, and what recording overhead would you accept to run rr on FalkorDB's CI flakes? ## Done when -- [ ] You can name the two nondeterminism sources rr records and state - why nothing else crosses the boundary on a single core. -- [ ] You can explain how an async signal gets re-delivered at exactly - the recorded instruction during replay, using RCB + registers. -- [ ] You can sketch the syscall fast path (seccomp-bpf, RR page, - rewritten stub, syscall buffer) and say which context switches - it removes. -- [ ] You have completed exercise 5: recorded a failing seeded test - and found the corrupting write with a watchpoint plus - reverse-continue. +Answer each before unfolding it. + +- [ ] You can name the two nondeterminism sources rr records and state why nothing else crosses the boundary on a single core. + +
Answer + + The two sources (§2.1) are (a) the results and side effects of + syscalls — what the kernel hands back across the interface — and (b) + the *timing* of asynchronous events: signals and context switches, + i.e. *when* they are delivered. On a single core with threads run one + at a time (§2.2), there is no true parallelism, so a context switch is + itself just another async event with a recordable delivery point; the + tracee's own user-space computation is deterministic given its inputs, + so once (a) and (b) are pinned, nothing else can differ between + record and replay. + +
+ +- [ ] You can explain how an async signal gets re-delivered at exactly the recorded instruction during replay, using RCB + registers. + +
Answer + + rr records the execution point at which the signal was delivered as + the pair (retired-conditional-branch count, full general-purpose + register state including the program counter) (§2.4.1). On replay it + programs the RCB performance counter to fire an interrupt as the count + approaches the recorded value, single-steps into the neighborhood, and + compares registers until they match the recorded state exactly — that + uniquely identifies the dynamic instruction instance (RCB alone does + not, e.g. an `inc [x]; jmp` loop repeats the same PC). It then injects + the signal there, reproducing the original delivery point precisely. + +
+ +- [ ] You can sketch the syscall fast path (seccomp-bpf, RR page, rewritten stub, syscall buffer) and say which context switches it removes. + +
Answer + + A ptrace-trapped syscall costs 4 context switches (tracee→kernel→rr→ + kernel→tracee — two blocking ptrace notifications; §3, Fig 1). The + fast path removes all of them for common syscalls: a seccomp-bpf + filter traps every syscall *except* those issued from a designated "RR + page" (§3.2); rr rewrites the hot 2-byte `syscall` instructions into + 5-byte `call`s into an injected stub (§3.1) that performs the real + syscall from the RR page and logs its result into an in-process + syscall buffer — so the common case never wakes rr (0 extra switches). + Uncommon or unsafe syscalls fall back to the slow 4-switch ptrace + path. + +
+ +- [ ] You have completed exercise 5: recorded a failing seeded test and found the corrupting write with a watchpoint plus reverse-continue. + +
Answer + + Concretely: `rr record ./test --seed 42` captures the flaky run once; + `rr replay` starts a deterministic replay with gdb attached; a + hardware watchpoint `watch -l corrupted_field` arms on the bad memory; + `reverse-continue` runs the replay *backwards* to the store that last + wrote it, stopping with full stack and register state. It works + because replay is fully deterministic (Step 5) and reverse execution + is replay-from-a-fork-checkpoint plus determinism (§6.2), so the + corrupting write sits at the same execution point in every replay. + +
## References **Papers** - O'Callahan, Jones, Froyd, Huey, Noll, Partush — "Engineering Record - and Replay for Deployability" (USENIX ATC 2017) — - [arXiv:1705.05937](https://arxiv.org/abs/1705.05937) + and Replay for Deployability" (USENIX ATC 2017; extended technical + report) — [arXiv:1705.05937](https://arxiv.org/abs/1705.05937). + Sections cited: §2.1 (nondeterminism sources), §2.2 (single-core + scheduling), §2.4.1 (RCB + registers), §2.6 (RDTSC), §3/§3.1/§3.2 + (syscall interception, Fig 1), §3.8 (replay), §4 (results, Fig 5), + §5.1 (ARM/hardware), §6.2 (fork checkpoints). **Code** - [rr](https://github.com/rr-debugger/rr) — the debugger itself; - `rr record` / `rr replay` are all exercise 5 needs + `rr record` / `rr replay` are all exercise 5 needs. (Not in this + course's pin table; no `file:line` anchors are cited above.) - Topic 16 (deterministic simulation testing) — the same - capture-nondeterminism-behind-an-interface move, before ship + capture-nondeterminism-behind-an-interface move, before ship. diff --git a/topics/35-overload/README.md b/topics/35-overload/README.md index 1171c9a..5d0b535 100644 --- a/topics/35-overload/README.md +++ b/topics/35-overload/README.md @@ -5,8 +5,9 @@ about the day it's slow because everyone is asking at once. Overload is not just "more load than capacity": retries, timeouts, and failover create feedback loops in which a *temporary* trigger flips the system into a *permanent* zero-goodput state that outlives the trigger — -Bronson et al. call these **metastable failures**, and they "account -for many of the largest outages at major web companies." The defenses — +Bronson et al. call these **metastable failures**, which have "caused +widespread outages at large internet companies, lasting from minutes to +hours" (§1). The defenses — retry budgets, priority admission control, backpressure — are policies a database must carry before the incident, for exactly topic 34's reason: you cannot deploy a load shedder to last Tuesday. @@ -150,12 +151,12 @@ nothing at all). CockroachDB's admission package states the reframe in its package doc: the goal is to **shift queueing out of the goroutine scheduler** — where the runtime picks what runs next — **into admission queues that -can reorder by priority and tenant** (`admission.go:1`). Slots +can reorder by priority and tenant** (`admission.go:21-24`). Slots (concurrency, occupied-while-running) govern CPU; tokens (rate, consumed-at-admission) govern IO, because LSM overload isn't a point-in-time queue but debt — L0 read amplification that compactions must pay down (topic 4's write stalls, promoted from per-store reflex -to node-wide policy). The slot count itself is AIMD-adjusted from a +to node-wide policy). The slot count itself is AIAD-adjusted (additive both ways) from a 1 ms-sampled `runnable goroutines per CPU` signal — queuing-time detection in scheduler clothing. @@ -186,7 +187,7 @@ one node instead of 3,000 services. | redis | `src/networking.c:5151` | `checkClientOutputBufferLimits` — backpressure on slow readers | | cockroach | `pkg/util/admission/admission.go:1` | the package doc — the whole design in one comment | | cockroach | `pkg/util/admission/work_queue.go:813` | `Admit` — where requests wait, ordered by (tenant, priority, ts) | -| cockroach | `pkg/util/admission/kv_slot_adjuster.go:46` | `CPULoad` — AIMD slots from runnable-goroutine counts | +| cockroach | `pkg/util/admission/kv_slot_adjuster.go:46` | `CPULoad` — AIAD (additive-both-ways) slots from runnable-goroutine counts | | cockroach | `pkg/util/admission/io_load_listener.go:69` | L0 thresholds — LSM debt as an admission signal | ## Reading guides diff --git a/topics/35-overload/reading-cockroach-admission.md b/topics/35-overload/reading-cockroach-admission.md index fd0263d..d1903be 100644 --- a/topics/35-overload/reading-cockroach-admission.md +++ b/topics/35-overload/reading-cockroach-admission.md @@ -5,11 +5,11 @@ this topic's papers: where DAGOR sheds across services and redis rejects on one thread, cockroach builds a user-space scheduler inside each node — work is intercepted before it becomes a runnable goroutine and queued where it can be reordered by tenant and -priority. The repo is cloned at `~/repos/cockroach`; this is a -code-read, ~1.5h, focused on two interfaces, one queue, and two -overload signals. Before opening files, this chapter builds the ideas -in order; the anchor table below maps each step to an exact -file:line. +priority. The repo is cloned at `~/repos/cockroach` (pinned at +`cockroach@a7e11788`); this is a code-read, ~1.5 h, focused on two +interfaces, one queue, and two overload signals. Before opening files, +this chapter builds the ideas in order; the anchor table below maps +each step to an exact file:line. ## The problem in one sentence @@ -20,18 +20,38 @@ the queue into its own code.** Admission control doesn't eliminate the wait; it relocates it to a place that can reorder it while keeping the CPU and disks busy. +Terms of art used below: + +- **Runnable goroutine** — a goroutine ready to run but not currently + on a CPU: work already waiting in the Go scheduler's queue. Cockroach + samples *runnable-per-CPU* as its CPU overload signal (Step 3). +- **Slot** — a unit of concurrency, held while work runs and returned + when it finishes (Step 2). +- **Token** — a unit of rate, consumed at admission and never returned, + used where the true cost lands later (Step 2). +- **L0 files / sub-levels** — Pebble's LSM write-stall signals (topic + 4): the depth of unpaid compaction debt, cockroach's IO overload + signal (Step 4). + ## The concepts, step by step ### Step 1 — the reframe: overload control as a user-space scheduler -The package doc comment (`admission.go:1-120`) states the two goals — -limit node overload, and provide performance isolation between -priorities and tenants — and the central move: "shift queueing from -system-provided resource allocation abstractions that we do not -control, like the goroutine scheduler, to queueing in admission -control, where we can reorder." Scope is deliberately node-local, not -cluster-level: in a system with strong work affinity, only the node -itself can protect itself in time. +> **In:** nothing yet — this step establishes the central design move +> every later step implements. +> **Out:** the reframe (move the queue out of the Go scheduler into +> code you can reorder) and its deliberately node-local scope. + +The package doc comment states the two goals — limit node overload +(`admission.go:11-13`) and provide performance isolation between +priorities and tenants (`admission.go:14-19`) — and the central move +(`admission.go:21-24`): "shift queueing from system-provided resource +allocation abstractions that we do not control, like the goroutine +scheduler, to queueing in admission control, where we can reorder." +Scope is deliberately node-local, not cluster-level (`admission.go:26-33`): +in a system with strong work affinity, only the node itself can protect +itself in time — cluster-level admission "can complement node level +admission control" but not replace it. ``` without admission control: with admission control: @@ -49,6 +69,12 @@ visible, owned, and reorderable. ### Step 2 — slots vs tokens: concurrency vs rate +> **In:** the reorderable queue from Step 1 needs a currency for "is a +> resource free?" +> **Out:** the two currencies — slots (returnable, for CPU) and tokens +> (non-returnable, for IO) — and why the choice encodes closed- vs +> open-loop control. + The package doc (`admission.go:54`, "Tokens and slots are the two ways admission is granted") splits resources by whether work completion is observable. A **slot** models concurrency: occupied @@ -72,7 +98,12 @@ Why it matters: the slot/token split is the type system of overload — it encodes whether backpressure can be closed-loop (slots: measure occupancy) or must be open-loop (tokens: refill on a capacity model). -### Step 3 — the CPU signal: runnable goroutines per CPU, AIMD slots +### Step 3 — the CPU signal: runnable goroutines per CPU, additive slots + +> **In:** the slot currency from Step 2, which needs a target count. +> **Out:** the CPU overload signal (runnable-per-CPU, not utilization) +> and the additive-increase/additive-decrease loop that hunts the slot +> count the machine can sustain. The overload signal for CPU is not utilization — it is **runnable goroutines per CPU, sampled every 1 ms** @@ -86,11 +117,28 @@ with a deep queue is overload. `kvSlotAdjuster.CPULoad(runnable, procs, samplePeriod)` (`kv_slot_adjuster.go:29` for the type, `:46` for the method) turns -the signal into an adaptive concurrency limit: at -`runnable >= threshold*procs` it decreases total slots (`:99`); at or -below half that (`:103`) it increases them — additive up, additive -down, every millisecond, an AIMD-style controller hunting the -concurrency the machine can actually sustain. +the signal into an adaptive concurrency limit. The adjustment is +**additive both ways** — the code's own comments say so — one slot per +1 ms tick: + +```go +// kv_slot_adjuster.go — CPULoad: additive adjust (71–72, 84, 91), triggers (99, 103) +71 if usedSlots > 0 && total > kvsa.minCPUSlots && usedSlots <= total { +72 total-- // comment :65/:81: "additive decrease", 1 slot per 1 ms tick +84 if usedSlots >= total && total < kvsa.maxCPUSlots { +91 total++ // comment :81/:90: "additive increase", 1 slot per 1 ms tick +99 if runnable >= threshold*procs { // overloaded → decrease +103 } else if float64(runnable) <= float64((threshold*procs)/2) { // underloaded → increase +``` + +So at `runnable >= threshold*procs` it decreases total slots by one +(`:72`); at or below half that (`:103`) it increases them by one +(`:91`) — additive-increase/additive-decrease (AIAD), every +millisecond, hunting the concurrency the machine can actually sustain. +(This is *not* AIMD: the decrease is `total--`, not a multiplicative +`total *= (1-α)`; the comment at `:65` calls it "additive decrease" +explicitly. DAGOR's admission controller is the AIMD one — do not +conflate them.) ``` runnable/CPU @@ -103,8 +151,17 @@ concurrency the machine can actually sustain. └────────────────────────────▶ sampled every 1 ms ``` +The dead band between `threshold/2` and `threshold` is what keeps the +controller from oscillating on every tick: it only acts at the extremes. + ### Step 4 — the IO signal: L0 debt, tokens as compaction budget +> **In:** the token currency from Step 2, which is open-loop and needs a +> feedback signal to size refills. +> **Out:** the LSM-derived IO overload signal (L0 file and sub-level +> counts) and how it turns token refill into a compaction budget spent +> by priority. + For stores, overload is read straight off the LSM: **L0 file count and L0 sub-level count** (`io_load_listener.go:69` and `:77`). You know these numbers from topic 4 — they are Pebble's write-stall @@ -131,10 +188,15 @@ accumulated consequence of past grants and throttles future ones. ### Step 5 — the priority ladder: below zero means "yield to users" +> **In:** the WorkQueue that spends slots (Step 3) and tokens (Step 4). +> **Out:** the concrete `int8` priority ladder that decides which work +> waits under overload, and how priority and tenancy compose. + `WorkPriority` is an `int8` (`admissionpb/admissionpb.go:23`) and the -ladder is deliberate: `LowPri` = MinInt8, `BulkLowPri` = -100, -`UserLowPri` = -50, `BulkNormalPri` = -30, `NormalPri` = 0, -`LockingNormalPri` = 10, `UserHighPri` = 50. Everything below zero is +ladder is deliberate: `LowPri` = MinInt8 (−128), `BulkLowPri` = −100, +`UserLowPri` = −50, `BulkNormalPri` = −30, `NormalPri` = 0, +`LockingNormalPri` = 10, `UserHighPri` = 50 +(`admissionpb/admissionpb.go:29-48`). Everything below zero is bulk/background — backups, rebalancing, changefeed catch-up — so under overload it is precisely the elastic work that waits while user foreground traffic keeps its latency. Within one priority, the @@ -143,22 +205,35 @@ of work, tenancy divides capacity inside a class. ### Step 6 — the grant loop: requester and granter +> **In:** the signals (Steps 3–4) and the priority policy (Step 5). +> **Out:** the two interfaces that turn "who wants to run" and "what is +> free" into grants, so adding a resource is writing a granter, not a +> scheduler. + Two small interfaces decouple "who wants to run" from "what resource is free": `requester` (`admission.go:178`) answers `hasWaitingRequests` and accepts `granted`, while `granter` (`admission.go:198`) offers `tryGet` (the uncontended fast path) and `returnGrant`. The concrete requester is `WorkQueue` -(`work_queue.go:303`), which orders waiting work by (tenant, -WorkPriority, FIFO arrival time). A request enters at -`WorkQueue.Admit` (`work_queue.go:813`) — try the fast path, else -queue and block — and CPU-bound KV work reports completion via -`AdmittedWorkDone` (`work_queue.go:1196`), returning its slot and -closing the loop of Step 2. Because signal (Steps 3-4), policy (Step -5), and mechanism (this loop) are separate interfaces, adding a -resource means writing a granter, not a scheduler. +(`work_queue.go:303`), whose doc comment (`work_queue.go:277`) spells +out the ordering: a group heap orders tenants by used slots/tokens +(fairness), and within each tenant, work is ordered by priority and +create time — i.e. (tenant fairness, WorkPriority, FIFO arrival). A +request enters at `WorkQueue.Admit` (`work_queue.go:813`) — try the +fast path, else queue and block — and CPU-bound KV work reports +completion via `AdmittedWorkDone` (`work_queue.go:1196`, which panics +if called for non-KV work), returning its slot and closing the loop of +Step 2. Because signal (Steps 3-4), policy (Step 5), and mechanism +(this loop) are separate interfaces, adding a resource means writing a +granter, not a scheduler. ### Step 7 — contrast: redis rejects, DAGOR spans services, cockroach reorders +> **In:** the full cockroach mechanism from Steps 1–6. +> **Out:** where cockroach sits against this topic's other two +> code-reads on the reject-vs-reorder and intra-node-vs-cross-service +> axes. + Hold this topic's three code-reads side by side. Redis (reading-redis-backpressure.md) is single-threaded: it cannot reorder admitted work, so its only move is a fast error at the door (OOM @@ -175,16 +250,17 @@ All paths relative to `~/repos/cockroach/pkg/util/admission`. | Step | Anchor | What to see | |---|---|---| -| 1 | `admission.go:1-120` | Package doc: goals, "shift queueing... where we can reorder", node-level scope | +| 1 | `admission.go:11-33` | Package doc: goals (11-19), "shift queueing... where we can reorder" (21-24), node-level scope (26-33) | | 2 | `admission.go:54` | Package-doc line naming tokens and slots as the two grant kinds | -| 3 | `kv_slot_adjuster.go:16` | `KVSlotAdjusterOverloadThreshold` — runnable goroutines per CPU | -| 3 | `kv_slot_adjuster.go:29`, `:46` | `kvSlotAdjuster` and `CPULoad`; decrease at `:99`, increase at `:103` | +| 3 | `kv_slot_adjuster.go:16` | `KVSlotAdjusterOverloadThreshold` — runnable goroutines per CPU, default 32 | +| 3 | `kv_slot_adjuster.go:29`, `:46` | `kvSlotAdjuster` and `CPULoad`; `total--` at `:72`, `total++` at `:91` (additive both ways) | +| 3 | `kv_slot_adjuster.go:99`, `:103` | decrease at `runnable ≥ threshold·procs`, increase at `≤ half` | | 4 | `io_load_listener.go:69`, `:77` | `L0FileCountOverloadThreshold`, `L0SubLevelCountOverloadThreshold` | -| 5 | `admissionpb/admissionpb.go:23` | `WorkPriority int8` and the full ladder of constants | +| 5 | `admissionpb/admissionpb.go:23`, `:29-48` | `WorkPriority int8` and the full ladder of constants | | 6 | `admission.go:178`, `:198` | `requester` / `granter` — the two halves of the grant loop | -| 6 | `work_queue.go:303` | `WorkQueue` — ordering by (tenant, priority, arrival) | +| 6 | `work_queue.go:277`, `:303` | `WorkQueue` doc + type — ordering by (tenant fairness, priority, create time) | | 6 | `work_queue.go:813` | `WorkQueue.Admit` — fast path, else wait | -| 6 | `work_queue.go:1196` | `AdmittedWorkDone` — slot return for KV work | +| 6 | `work_queue.go:1196` | `AdmittedWorkDone` — slot return for KV work (panics if not KV) | Read order: the package doc top to bottom (it is the design document) → `requester`/`granter` → `WorkQueue.Admit` → `kvSlotAdjuster.CPULoad` @@ -200,7 +276,7 @@ Resist reading the rest of work_queue.go; these anchors are the skeleton. 2. Why can a slot be returned but a token cannot? Trace one KV read and one write: at what moment is each resource's true cost fully known, and what does that imply for closed- vs open-loop control? -3. The AIMD slot adjuster decreases at `threshold*procs` but only +3. The AIAD slot adjuster decreases at `threshold*procs` but only increases at or below half that. What failure mode does the dead band prevent, and what would equal thresholds do? 4. Topic 4's Pebble stalls writes when L0 gets deep — every writer, @@ -215,21 +291,90 @@ Resist reading the rest of work_queue.go; these anchors are the skeleton. ## Done when +Answer each before unfolding it. + - [ ] You can narrate one KV request end to end — `Admit` fast path vs queue, grant by (tenant, priority, arrival), run, `AdmittedWorkDone` slot return — naming each hop's interface. + +
Answer + + A KV request calls `WorkQueue.Admit` (`work_queue.go:813`). Admit asks + the `granter` (`admission.go:198`) for `tryGet` — the uncontended fast + path; if a slot is free it runs immediately. Otherwise it enqueues in + the `WorkQueue` (the concrete `requester`, `work_queue.go:303`), which + orders waiting work by tenant fairness (group heap on used + slots/tokens), then `WorkPriority`, then create time + (`work_queue.go:277`). When a slot frees, the granter calls `granted` + on the requester, which dequeues the next winner. The work runs, then + reports completion via `AdmittedWorkDone` (`work_queue.go:1196`), + returning its slot — closing the concurrency loop of Step 2. + `AdmittedWorkDone` panics if called for non-KV work, because only + slots (not tokens) are returnable. + +
+ - [ ] You can state both overload signals (runnable per CPU; L0 files/sub-levels) and say why neither is utilization. + +
Answer + + CPU overload is **runnable goroutines per CPU**, sampled every 1 ms + against `KVSlotAdjusterOverloadThreshold` (default 32, + `kv_slot_adjuster.go:16`). IO overload is **L0 file count and L0 + sub-level count** (`io_load_listener.go:69`, `:77`) — Pebble's + write-stall signals from topic 4, the depth of unpaid compaction debt. + Neither is utilization because utilization cannot tell a healthy busy + server from an overloaded one: 100% CPU with an empty runnable queue + is fine, 100% CPU with a deep runnable queue is overload. Both signals + measure *waiting* — runnable-but-not-running work, and accumulated + write debt — which is exactly DAGOR's queuing-time instinct in a + different vocabulary. + +
+ - [ ] You can explain, in two sentences, why writes get tokens and CPU work gets slots — and why the two are not interchangeable. + +
Answer + + CPU-bound KV work has an observable completion — the goroutine + finishes — so a **slot** (held while running, returned on completion, + `AdmittedWorkDone`) models it as concurrency and closes the loop. + A write's true cost lands *later*, when compactions rewrite its bytes + out of L0, so there is nothing to return at admission time; a + **token** (consumed once, never returned, refilled on a capacity model + driven by the L0 signal) models it as a rate. They are not + interchangeable because a returnable slot assumes you know when the + cost is paid, and for IO you do not until compaction happens. + +
+ - [ ] You can place cockroach, redis, and DAGOR on the axes reject-vs-reorder and intra-node-vs-cross-service unaided. +
Answer + + **Reject vs reorder:** redis *rejects* (fast `-OOM`/`-BUSY`/disconnect + at the door — one thread, nothing to reorder); cockroach *reorders* + (work waits in a WorkQueue and is granted by priority/tenant, it does + not fail); DAGOR *sheds by priority* (drops low-priority whole tasks). + **Intra-node vs cross-service:** redis and cockroach are both + intra-node (one process/one node), while DAGOR is cross-service + (priorities ride RPC headers, upstreams throttle for downstreams). + Cockroach is the multi-core, priority-aware midpoint: intra-node like + redis, priority-aware like DAGOR, and it explicitly leaves + cluster-level admission as a complement, not a replacement + (`admission.go:26-33`). + +
+ ## References **Code** - [CockroachDB](https://github.com/cockroachdb/cockroach) — - `pkg/util/admission`, cloned at `~/repos/cockroach` + `pkg/util/admission`, cloned at `~/repos/cockroach`, pinned at + `cockroach@a7e11788` **Related guides** - [README.md](README.md) — topic 35 overview and the capstone gate diff --git a/topics/35-overload/reading-dagor.md b/topics/35-overload/reading-dagor.md index a35fcf8..9da7f67 100644 --- a/topics/35-overload/reading-dagor.md +++ b/topics/35-overload/reading-dagor.md @@ -1,14 +1,15 @@ # DAGOR: overload control when every task is a fan-out DAGOR (Zhou et al., SoCC 2018) is the overload-control system inside -WeChat's microservice platform: 3000+ services on 20000+ machines -absorbing 10^10–10^11 requests per day, with a daily peak around 3× -the average and Chinese New Year pushing the request rate to roughly -10× the daily peak. You cannot provision for that; you must shed — -and the paper's contribution is *which* load to shed, *where* to -detect the need, and *who* pays for the rejection, when a user-visible -task fans out and partial success is worth nothing. This chapter -builds the seven ideas first, then maps the ~12-page paper. +WeChat's microservice platform: more than 3000 services on over 20000 +machines (§2.2) absorbing 10^10–10^11 requests per day (§2.2), with a +daily peak around 3× the daily average and Chinese New Year pushing the +peak to roughly 10× the daily average (§2.3). You cannot provision for +that; you must shed — and the paper's contribution is *which* load to +shed, *where* to detect the need, and *who* pays for the rejection, +when a user-visible task fans out and partial success is worth nothing. +This chapter builds the seven ideas first, then maps the ~12-page +paper. ## The problem in one sentence @@ -16,16 +17,38 @@ builds the seven ideas first, then maps the ~12-page paper. service invocations succeed, so naive random load shedding at the overloaded service wastes the work of every partially-completed task — overload control must shed consistently, by priority, across the whole -call tree.** The paper calls this *subsequent overload* (Definition -1) — the failure mode single-server admission controllers like CoDel -and SEDA were never designed to see. +call tree.** The paper calls this *subsequent overload* (§3.1, +Definition 1) — the failure mode single-server admission controllers +like CoDel and SEDA were never designed to see. + +Terms of art, used with the paper's definitions: + +- **Entry task** — one user-visible request (open a chat, send a + payment). It fans out into many **service invocations** down a call + tree, and succeeds only if *all* of them succeed. +- **Subsequent overload** (§3.1, Definition 1) — overload in which more + than one service is overloaded along a task's path, *or* a single + overloaded service is invoked multiple times by one task. This is + what makes random shedding collapse. +- **Admission level** — the priority cursor a server is currently + admitting down to. A compound (business, user) value; a server admits + a request iff its priority is at least as high as the current level. +- **Business / user priority** — the two halves of a request's + priority: a coarse per-action rank (login > pay > message) and a fine + per-user tiebreak. ## The concepts, step by step ### Step 1 — subsequent overload: why random shedding collapses +> **In:** nothing yet — this step is the motivation, the failure mode +> single-server admission controllers cannot see. +> **Out:** the reason shedding must key on a *task-wide priority* rather +> than a per-request coin flip — the constraint every later step obeys. + Suppose service M is at 2× capacity and sheds 50% of requests at -random, and each task must call M twice: +random, and each task must call M twice (§3.1, the paper's own worked +example, Figure 2.b / Form 2): ```mermaid graph LR @@ -36,18 +59,33 @@ graph LR ``` Success probability is 0.5 × 0.5 = 25% — yet M did 50% of its normal -useful work admitting first calls whose sibling call then died. The -served half-tasks are pure waste; with k invocations, random shedding -admits `0.5^k` of tasks while burning full capacity. The fix is -*consistency*: admit or kill whole tasks, which forces shedding to key -on a priority that travels with the task, not a per-request coin flip. +useful work admitting first calls whose sibling call then died. §3.1 +works it in throughput terms: feed rate 2C at service M with capacity +C, random shedding admits half, so each M-invocation succeeds with +p=0.5; a task that calls M twice survives with p=0.25, so of C tasks +issued (2C requests to M) only `0.25C` tasks survive while M burns its +full C of capacity. The served half-tasks are pure waste; with k +invocations, random shedding admits `0.5^k` of tasks while burning full +capacity — for k=3 that is `0.5^3 = 12.5%`, for k=4, `6.25%`. The fix +is *consistency*: admit or kill whole tasks, which forces shedding to +key on a priority that travels with the task, not a per-request coin +flip. (§3.1 also notes the flip side: if the offered load is only +0.5C, service M is *just* saturated and `0.5C` tasks survive — the +seed of Step 7's f_sat/f yardstick.) ### Step 2 — the signal: queuing time, not response time or CPU +> **In:** Step 1's requirement to shed the right load; a controller +> first needs to know it is overloaded. +> **Out:** the local, demand-sensitive overload signal (queuing time) +> and its window (1 s or 2000 requests, 20 ms threshold) that Steps 5–6 +> feed on. + DAGOR declares overload from the **average request queuing time** — arrival to start of processing — over a window of 1 second or 2000 requests, whichever comes first, against a 20 ms threshold (task -timeout: 500 ms). It explicitly rejects the two obvious alternatives: +timeout: 500 ms) (§4.1). It explicitly rejects the two obvious +alternatives: ``` response time = queuing + processing, and processing is RECURSIVE: @@ -61,18 +99,28 @@ timeout: 500 ms). It explicitly rejects the two obvious alternatives: ``` CPU utilization fails the other way: high CPU-busy is normal on a -well-utilized server — busy is not overloaded. Queuing time alone is -both local and demand-sensitive: topic 34's lesson again, the queue is -where the truth lives. +well-utilized server — busy is not overloaded (§4.1). Queuing time +alone is both local and demand-sensitive: topic 34's lesson again, the +queue is where the truth lives. The window bounds — 1 s or 2000 +requests — matter: a shorter window would react to bursts, a longer one +would lag; and the 20 ms threshold sits well under the 500 ms task +timeout so the controller acts before deadlines start firing. ### Step 3 — business priority: assigned once, copied everywhere -Priorities come from a replicated hash table with a few tens of -entries; smaller value means higher priority. Login is the highest, -and WeChat Pay sits above Instant Messaging because users complain -about failed payments roughly 100× more than about failed messages. -The crucial mechanic: the priority is decided at the **entry task** -and **copied to every subsequent request in the task's call tree**: +> **In:** Step 1's demand for a task-wide priority and Step 2's overload +> signal. +> **Out:** the coarse half of that priority (business level) and the +> mechanic — decided at the entry task, copied to every child — that +> makes shedding consistent across a call tree. + +Priorities come from a replicated hash table with a few tens of entries +(§4.2.1, Figure 3); smaller value means higher priority. Login is the +highest, and WeChat Pay sits above Instant Messaging because users +complain about failed payments roughly 100× more than about failed +messages. The crucial mechanic: the **business priority** is decided at +the **entry task** and **copied to every subsequent request in the +task's call tree**: ```mermaid graph TD @@ -84,16 +132,25 @@ graph TD Every server shedding at admission level τ therefore makes the *same* decision for all pieces of one task — exactly the consistency Step 1 -demanded: whole tasks admitted or killed, never fragments. +demanded: whole tasks admitted or killed, never fragments. This is why +a per-request coin flip (Step 1) is replaced by a per-*task* label: the +label is set once, at the tree root, and inherited unchanged. ### Step 4 — user priority: 128 sublevels so the cursor can settle +> **In:** the business level from Step 3, which is too coarse to tune +> against. +> **Out:** the fine half of the priority (128 user sublevels) that gives +> the admission cursor enough resolution not to oscillate, plus two +> anti-gaming design choices. + Business levels alone are too coarse: with a few tens of levels the load gap between neighbors is huge, so admission level τ sheds too much, τ−1 is overloaded again, and the controller oscillates forever. DAGOR splits each business level into 128 **user levels** — a hash of -the user ID — giving a compound (business, user) priority with ~10^4 -fine-grained levels, enough resolution for the cursor to settle: +the user ID (§4.2.2, Figure 4) — giving a compound (business, user) +priority with ~10^4 fine-grained levels, enough resolution for the +cursor to settle: ``` business level: ... │ B=5 │ B=6 │ ... tens of levels @@ -103,35 +160,52 @@ fine-grained levels, enough resolution for the cursor to settle: flapping between whole levels ``` -Two design notes. The hash is **rotated hourly**, so no user is -permanently the sacrificial low-priority one. And a session-oriented -priority was considered and **rejected because users figured it out**: -logging out and back in re-rolled the priority, so people relogged to -escape shedding. Hourly user-ID hashing removes the incentive. +Two design notes from §4.2.2. The hash is **rotated hourly**, so no +user is permanently the sacrificial low-priority one. And a +session-oriented priority was considered and **rejected because users +figured it out**: logging out and back in re-rolled the priority, so +people relogged to escape shedding. Hourly user-ID hashing removes the +incentive while keeping the property that all of one user's requests +sort together, so a user tends to see whole tasks succeed or whole +tasks fail rather than half-broken results. ### Step 5 — adaptive admission: histogram plus prefix sums -The cursor is not adjusted by fixed steps. DAGOR's Algorithm 1 adapts -an *expected admitted count*: when a window is overloaded (average -queuing time over 20 ms), the next window's expected admissions shrink -multiplicatively to (1−α)·N_adm with α = 5%; when healthy, they grow -additively by β·N with β = 1% — AIMD's cousin, pointed at admission. -To turn "admit roughly N requests" back into a cursor, each server -keeps a histogram of request counts per compound (B, U) level; a -prefix-sum walk finds the lowest-priority level whose cumulative count -still fits under the expected total — that level is the new cursor. -This is exactly the contract of this topic's stub in -`experiments/src/admission.rs` — priority histogram, 5%/1% adaptation, -O(1) `admit(priority)` gate — minus the user sublevels. +> **In:** the overload verdict from Step 2 and the fine (B, U) levels +> from Steps 3–4. +> **Out:** the AIMD-style rule that moves the admission cursor each +> window, and the O(1) histogram + prefix-sum trick that turns "admit N +> requests" back into a concrete (B, U) cursor. + +The cursor is not adjusted by fixed steps. DAGOR's **Algorithm 1** +(§4.2.3) adapts an *expected admitted count*: when a window is +overloaded (average queuing time over 20 ms), the next window's +expected admissions shrink multiplicatively to (1−α)·N_adm with α = 5%; +when healthy, they grow additively by β·N with β = 1% — this is a +genuine **AIMD** (additive-increase/multiplicative-decrease) rule, the +classic TCP-congestion shape pointed at admission. To turn "admit +roughly N requests" back into a cursor, each server keeps a histogram +of request counts per compound (B, U) level; a prefix-sum walk finds +the lowest-priority level whose cumulative count still fits under the +expected total — that level is the new cursor. This is exactly the +contract of this topic's stub in `experiments/src/admission.rs` — +priority histogram, 5%/1% adaptation, O(1) `admit(priority)` gate — +minus the user sublevels. ### Step 6 — collaborative shedding: reject before you send +> **In:** the per-server admission level maintained by Step 5. +> **Out:** how enforcement migrates one hop upstream so the overloaded +> server never pays for rejections — while detection and adaptation +> stay local. + Local admission control still charges the overloaded server for every rejection: the request crossed the network and sat in the queue before being refused. DAGOR makes rejection free for the victim by **piggybacking** the server's current admission level (B, U) on every -response; each upstream stores the freshest level per downstream and -sheds doomed requests *before* sending them: +response (§4.2.4, Figure 5's workflow); each upstream stores the +freshest level per downstream and sheds doomed requests *before* +sending them: ```mermaid graph LR @@ -140,52 +214,74 @@ graph LR ``` The overloaded server spends its cycles only on requests it will -actually serve. Detection and adaptation stay purely local (Steps 2 -and 5), but enforcement migrates upstream one hop at a time — no -central coordinator, no config push. +actually serve. Detection and adaptation stay purely local (Steps 2 and +5), but enforcement migrates upstream one hop at a time — no central +coordinator, no config push. This is the "collective, not per-service" +feedback the paper stresses: no single component sees global state, yet +the composition converges because each server advertises its own cursor +and each upstream respects the freshest cursor it has seen. ### Step 7 — the yardstick: optimal success rate is f_sat/f +> **In:** all the machinery of Steps 1–6. +> **Out:** the single curve the evaluation measures everything against +> (f_sat/f), and the two headline results (630 vs 750 QPS; ~50% over +> CoDel/SEDA) that show DAGOR tracking it. + Under subsequent overload, the best any controller can do is serve whole tasks up to saturation: with offered load f and saturation -throughput f_sat, the optimal task success rate is **f_sat/f** — the -line the evaluation plots everything against. In the stress tests, -service M saturates at ~750 QPS on 3 servers. DAGOR_q (the real thing: +throughput f_sat, the optimal task success rate is **f_sat/f** (§5.3, +defined exactly: f_sat is "the maximum feed rate that makes the +downstream service just saturated," f is "the actual feed rate when the +downstream service is overloaded") — the line the evaluation plots +everything against. In the stress tests (§5.1), service M is deployed +over 3 servers and saturates at ~750 QPS. DAGOR_q (the real thing: queuing-time signal, 20 ms) sheds correctly all the way to saturation, -sustaining ~750 QPS; DAGOR_r (a variant on a 250 ms response-time -threshold) begins shedding at ~630 QPS — Step 2's recursive false -positives, measured. On the M² workload (each task makes 2 calls into -the overloaded service) DAGOR beats CoDel and SEDA by about 50% in -task success rate, and across M¹–M⁴ (tasks with 1–4 subsequent calls) -its success rate stays uniform while CoDel favors the simple-overload -case. One workload detail: upstreams resend rejected invocations up to -3 times, so shedding also multiplies offered load — another reason -rejection must be cheap (Step 6). +postponing shedding to ~750 QPS; DAGOR_r (a variant on a 250 ms +response-time threshold) begins shedding at ~630 QPS — Step 2's +recursive false positives, measured (§5.2, Figure 6). On the M² +workload (each task makes 2 calls into the overloaded service) DAGOR +beats CoDel and SEDA by about 50% in task success rate (§5.3, Figure +7.b), and across M¹–M⁴ at a fixed 1500 QPS feed rate (Figure 8) its +advantage grows with subsequent-overload depth while CoDel and SEDA, +tuned for simple overload (M¹, Figure 7.a where all are roughly equal), +fall away. One workload detail (§5.1, footnote 8): upstreams resend +rejected invocations up to 3 times, so shedding also multiplies offered +load — another reason rejection must be cheap (Step 6). ## How to read the paper (with the concepts in hand) -SoCC 2018, ~12 pages (arXiv:1806.04075); budget ~1.5h. - -- **§1, intro** (10 min) — the scale numbers (3000+ services, 20000+ - machines, 10^10–10^11 requests/day) and the burstiness (~3× daily - peak, ~10× at Chinese New Year) that makes provisioning hopeless. -- **§2, WeChat background** (10 min) — the service DAG and entry - tasks: just enough topology to see why a priority must be copied - down a call tree (Step 3). -- **§3, overload in microservices** (15 min) — Definition 1 and the - subsequent-overload arithmetic (Step 1). Do the 0.25 computation - yourself before reading theirs. -- **§4, DAGOR design** (30 min) — **the core**. Queuing-time detection - with the 20 ms / 1 s-or-2000-requests window (Step 2); business and - user priorities, including the rejected session priority (Steps - 3–4); Algorithm 1 with α = 5%, β = 1% (Step 5); collaborative - shedding (Step 6). -- **§5, implementation** (5 min) — where the hooks live in the RPC - framework; note how little each service must change. -- **§6, evaluation** (20 min) — find every number from Step 7 in its - figure: 750 vs 630 QPS for DAGOR_q/DAGOR_r, the ~50% win over - CoDel/SEDA on M², the M¹–M⁴ fairness plot. Ask of each graph: how - far below f_sat/f is each line, and why? +SoCC 2018, ~12 pages (arXiv:1806.04075); budget ~1.5 h. The paper is +seven sections; §5 is the **Evaluation** and §6 is **Related Work** — +there is no standalone implementation section (the wiring is §4.3 +Workflow). + +- **§1 Introduction** (10 min) — the problem and DAGOR's design + principles (service-agnostic, decentralized, no central quorum). +- **§2 Background** (10 min) — §2.1 service architecture and entry + tasks (Figure 1: just enough topology to see why a priority must be + copied down a call tree, Step 3); §2.2 the scale numbers (3000+ + services, 20000+ machines, 10^10–10^11 requests/day); §2.3 the + dynamic workload — ~3× daily average at peak, ~10× the daily average + at Chinese New Year — that makes provisioning hopeless. +- **§3 Overload in WeChat** (15 min) — §3.1 Definition 1 and the + subsequent-overload arithmetic with Figure 2's three forms (Step 1); + do the 0.25 computation yourself before reading theirs. §3.2 lists the + scaling challenges DAGOR's decentralization answers. +- **§4 DAGOR Overload Control** (30 min) — **the core**. §4.1 + queuing-time detection with the 20 ms / 1 s-or-2000-requests window + (Step 2); §4.2.1 business priority (Figure 3) and §4.2.2 user priority + with the rejected session priority (Figure 4, Steps 3–4); §4.2.3 + Algorithm 1 with α = 5%, β = 1% (Step 5); §4.2.4 collaborative + shedding (Step 6); §4.3 the end-to-end workflow (Figure 5). +- **§5 Evaluation** (20 min) — find every number from Step 7 in its + figure: §5.2 Figure 6 gives 750 vs 630 QPS for DAGOR_q/DAGOR_r; §5.3 + Figures 7–8 give the ~50% win over CoDel/SEDA on M² and the M¹–M⁴ + progression at 1500 QPS; §5.4 Figure 9 is fairness. Ask of each graph: + how far below f_sat/f is each line, and why? +- **§6 Related Work** (5 min) — where CoDel and SEDA sit relative to + DAGOR; skim. +- **§7 Conclusion** (5 min) — skim. ## Questions to answer in notes.md @@ -213,28 +309,97 @@ SoCC 2018, ~12 pages (arXiv:1806.04075); budget ~1.5h. ## Done when +Answer each before unfolding it. + - [ ] You can state from memory why queuing time beats response time and CPU as the overload signal, with the recursive-inflation argument. + +
Answer + + Response time is `queuing + processing`, and processing is recursive: + an overloaded leaf C inflates the response time of every ancestor + (`resp(A) = q(A) + p(A) + resp(B)`, and so on), so a + response-time signal fires false positives at servers that are + themselves healthy (§4.1). CPU utilization fails the other way — a + well-utilized server runs at high CPU-busy without being overloaded, + so it is not a distinguishing signal. **Queuing time** (arrival → + start of processing) is *local* — `q(B)` only grows when B itself + cannot keep up — and demand-sensitive. DAGOR averages it over a 1 s / + 2000-request window against a 20 ms threshold (task timeout 500 ms). + +
+ - [ ] You can explain subsequent overload with the 0.5 × 0.5 = 25% example and say what "consistent shedding" buys instead. + +
Answer + + A task that calls overloaded service M twice, where M sheds 50% at + random, succeeds with `0.5 × 0.5 = 25%` (§3.1) — yet M spent its full + capacity admitting first calls whose siblings then died, so half its + useful work is waste. With k calls the survival rate is `0.5^k` + (12.5% at k=3) while M stays saturated. **Consistent shedding** keys + the decision on a task-wide priority copied down the whole call tree + (Steps 3–4), so every server admits or kills the *same* tasks: + whole tasks survive up to saturation instead of `0.5^k` fragments, + which is what lets DAGOR approach the f_sat/f optimum. + +
+ - [ ] You can run Algorithm 1 on paper: window verdict → (1−α)·N_adm or +β·N → prefix-sum over the (B, U) histogram → new cursor. + +
Answer + + Each window, DAGOR compares average queuing time to 20 ms. If + overloaded, the expected admitted count shrinks *multiplicatively* to + `(1−α)·N_adm` with α = 5%; if healthy, it grows *additively* by `β·N` + with β = 1% (§4.2.3, Algorithm 1) — additive-increase/ + multiplicative-decrease. To convert that count into a cursor, the + server keeps a histogram of request counts per compound (B, U) level + and walks a prefix sum from highest priority down, stopping at the + lowest-priority level whose cumulative count still fits under the + expected total — that level becomes the new admission level. The 128 + user sublevels (Step 4) give the prefix-sum enough resolution to + settle instead of oscillating between whole business levels. + +
+ - [ ] You have implemented the gate in `experiments/src/admission.rs` far enough that its tests exercise the 5%/1% adaptation. +
Answer + + The stub mirrors DAGOR's core minus user sublevels: a priority + histogram, an O(1) `admit(priority)` gate that compares against the + current cursor, and the Algorithm-1 adaptation — `(1−0.05)·N_adm` on + an overloaded window, `+0.01·N` on a healthy one — that moves the + cursor via a prefix-sum walk. The tests are the specification: they + drive overloaded and healthy windows and assert the cursor tightens + and loosens by those factors, and that `admit` is consistent for a + given priority within a window (the Step-1 consistency property). The + reference numbers live in `notes.md`. + +
+ ## References **Papers** - Zhou et al. — "Overload Control for Scaling WeChat Microservices" - (SoCC 2018) — [arXiv:1806.04075](https://arxiv.org/abs/1806.04075) + (SoCC 2018) — [arXiv:1806.04075](https://arxiv.org/abs/1806.04075). + Definition 1 and the 0.25C arithmetic are §3.1; the queuing-time + signal is §4.1; business/user priority are §4.2.1–4.2.2 (Figures 3–4); + Algorithm 1 is §4.2.3; collaborative shedding is §4.2.4; the f_sat/f + yardstick and the 630/750 QPS and ~50% results are §5.2–5.3 + (Figures 6–8). **This learning path** - [Topic 35 README](README.md) — the overload topic this guide belongs - to, and the bench lanes that price shedding strategies + to, and the bench lanes that price shedding strategies. - [Topic 34 — debugging and production diagnosis](../34-debugging/README.md) — coordinated omission and slow logs; the measurement discipline - DAGOR's queuing-time signal comes from + DAGOR's queuing-time signal comes from. - This topic's `experiments/src/admission.rs` — DAGOR-lite stub: queuing-time windows, priority cursor, 5%/1% adaptation of - Algorithm 1, minus user sublevels + Algorithm 1, minus user sublevels. diff --git a/topics/35-overload/reading-metastable.md b/topics/35-overload/reading-metastable.md index 6aeb41a..faca9e4 100644 --- a/topics/35-overload/reading-metastable.md +++ b/topics/35-overload/reading-metastable.md @@ -4,12 +4,13 @@ Bronson, Aghayev, Charapko, and Zhu (HotOS 2021) name a failure class you have seen in an incident channel: something bad happens for ten seconds, the bad thing goes away, and the system stays down anyway — until a human sheds load or restarts everything. The paper's claim is -that these *metastable failures* account for many of the largest -outages at major web companies, and that they are systematically -misdiagnosed because everyone hunts the trigger while the real culprit -is a feedback loop. Read this 7-page position paper as the theory -chapter for this topic's simulator: every number in its Figure 2 is -reproduced exactly in `experiments/` (lane 1). +that these *metastable failures* have caused widespread outages at +large internet companies (§1: "lasting from minutes to hours") and have +"a disproportionate impact on hyperscale distributed systems," and that +they are systematically misdiagnosed because everyone hunts the trigger +while the real culprit is a feedback loop. Read this 7-page position +paper as the theory chapter for this topic's simulator: every number in +its Figure 2 is reproduced exactly in `experiments/` (lane 1). ## The problem in one sentence @@ -22,12 +23,34 @@ blip. The paper contributes a vocabulary (stable / vulnerable / metastable), a minimal worked example, and a catalog of sustaining loops and mitigations. +A few terms of art, used with the paper's definitions (§1) throughout: + +- **Trigger** — a temporary disturbance (a load spike, a brief outage, + a deploy) that pushes a *vulnerable* system over the edge. It need + not still be present for the failure to continue. +- **Sustaining effect** — the feedback loop, "often involving work + amplification or decreased overall efficiency" (§1), that keeps the + system in the bad state after the trigger is gone. This is the root + cause. +- **Goodput** — throughput of *useful* work, i.e. requests that + complete before their deadline. Distinct from throughput: a system + can be busy at 100% CPU with goodput near zero. +- **Hidden capacity** (defined §4) — the load a system can actually + sustain *once the sustaining loop is active*, as opposed to its + **advertised capacity** measured in the healthy state. + ## The concepts, step by step ### Step 1 — three states, one arrow that does not reverse itself -A system moves between three states. The trigger arrow is temporary; -the trap is that removing the trigger does not walk you back: +> **In:** nothing yet — this step establishes the paper's Figure 1 +> state machine and the vocabulary every later step leans on. +> **Out:** the reason "remove the trigger" is not a recovery plan — +> the arrow into *metastable* has no passive way back. + +A system moves between three states (paper Figure 1, §1). The trigger +arrow is temporary; the trap is that removing the trigger does not walk +you back: ```mermaid graph LR @@ -39,28 +62,46 @@ graph LR ``` The self-loop on *metastable* is the whole paper. Recovery never -happens passively; it requires a deliberate push — shed load below the -hidden capacity, or break the retry loop directly. +happens passively; §1 says leaving the state "requires a strong +corrective push, such as rebooting the system or dramatically reducing +the load" — shed load below the hidden capacity, or break the retry +loop directly. Contrast this with failures that *do* self-heal when the +trigger leaves: the paper explicitly excludes a denial-of-service +attack, limplock, and livelock, "are not metastable" (§1). The +distinguishing test is the self-loop, not the severity. ### Step 2 — vulnerable is not a defect -The vulnerable state is where efficient systems live on purpose: -higher utilization means fewer machines, so staying out of the -vulnerable region wastes most of your capacity most of the time. -Organizational incentives push the same direction: the paper's example -is a better cache eviction algorithm that raises the hit rate, which -lets you serve more load from the same database — and thereby raises -the hidden work amplification if the cache is ever lost. A false +> **In:** the three states from Step 1. +> **Out:** why healthy production systems sit in *vulnerable* on +> purpose, so "just run in the stable region" is not a real answer. + +The vulnerable state is where efficient systems live on purpose. §1 is +blunt about it: "many production systems choose to run in the +vulnerable state all the time because it has much higher efficiency +than the stable state." Higher utilization means fewer machines, so +staying out of the vulnerable region wastes most of your capacity most +of the time. Organizational incentives push the same direction: §3's +example is a better cache eviction algorithm that raises the hit rate, +which lets you serve more load from the same database — and thereby +raises the hidden work amplification if the cache is ever lost. A false economy, invisible until the trigger arrives. So do not read "vulnerable" as "buggy"; read it as "operating with a hidden debt that -a trigger can call in." +a trigger can call in." The paper is careful here: "The vulnerable +state is not an overloaded state; a system can run for months or years +in the vulnerable state" (§1). ### Step 3 — Figure 2, the minimal metastable system -The paper's worked example needs only two components: a database that -handles at most 300 QPS (requests complete in under 100 ms below -that), and a web app that sends 1 query per request with 1 retry -after a 1 s timeout. +> **In:** the vulnerable/metastable distinction from Steps 1–2. +> **Out:** a fully worked numerical example — 280 QPS offered → 560 QPS +> sustained against 300 QPS capacity — that this topic's lane 1 +> reproduces exactly, plus the two recovery thresholds (150 and 20 QPS). + +The paper's worked example (§2.1, plotted in **Figure 2**) needs only +two components: a database that handles at most 300 QPS (requests +complete in under 100 ms below that), and a web app that sends 1 query +per request with 1 retry after a 1 s timeout. ``` offered load: 280 QPS (vulnerable: inside 150-300) @@ -76,29 +117,53 @@ after a 1 s timeout. goodput: 0, forever ``` -At 280 QPS offered, a 10 s outage queues enough requests that every -one of them times out and is retried: the server now faces a sustained +At 280 QPS offered, a 10 s outage queues enough requests that every one +of them times out and is retried: the server now faces a sustained 560 QPS against a 300 QPS capacity, and goodput drops to 0 -*permanently*. The stable region is load below 150 QPS; between 150 -and 300 QPS the system is vulnerable. Recovery requires dropping -offered load below 150 QPS or the retry rate below 20 QPS. Note the -gap: *advertised* capacity is 300 QPS, but the *hidden* capacity — -what survives the retry amplification — is 150. +*permanently*. Work the arithmetic the way §2.1 does, because it is the +same arithmetic lane 1 runs: + +- **Sustained load under the loop.** Each request that overloads is + retried once, so a stuck server sees offered + retries. At 280 QPS + offered every request is retried: `280 + 280 = 560 QPS`, well past + the 300 QPS ceiling, so the queue never drains. +- **Load recovery threshold (150 QPS).** With a 1-retry policy the + sustained load is `2 × offered`. Stability needs `2 × offered ≤ 300`, + i.e. `offered ≤ 150`. So the *hidden* capacity is `300 / (1 + retries) + = 300 / 2 = 150 QPS`, half the advertised 300. Below 150 QPS offered + the loop cannot sustain itself; between 150 and 300 the system is + vulnerable; above 300 it is over capacity even when healthy. +- **Retry recovery threshold (20 QPS), at the 280 QPS operating point.** + Hold offered load at 280 and instead cap the retry rate. Total load is + `280 + retries`; stability needs `280 + retries ≤ 300`, i.e. + `retries ≤ 20 QPS`. That is where §2.1's "limit retries to below + 20 QPS" comes from: it is the 300 − 280 = 20 QPS of headroom left at + that load. + +Note the gap the paper keeps returning to: *advertised* capacity is +300 QPS, but the *hidden* capacity — what survives the retry +amplification — is 150. That gap is the whole danger. ### Step 4 — work amplification is the fuel -Every sustaining loop runs on work amplification: the failure mode +> **In:** the retry loop (2×) from Step 3. +> **Out:** the general form — a sustaining loop is any mechanism that +> makes each unit of user demand cost more once you are overloaded — +> with the cache case's 10× as the scarier instance. + +Every sustaining loop runs on **work amplification**: the failure mode makes each unit of user demand cost more than in the healthy state. -Retries are the simplest amplifier (2× in Fig 2); the paper mentions a -100× anecdote in the wild. The look-aside cache is the scarier common -case: a 90% hit rate lets a 3,000 QPS application run on a 300 QPS -database, so losing the cache is a 10× work amplification — and the -cold cache cannot refill, because refilling requires database reads -and the database is saturated. Hidden capacity 300 QPS, advertised -capacity 3,000 QPS. A third amplifier hides in error handling itself: -if the error path costs more than the success path (e.g., logging that -takes locks), the system does its most expensive work exactly when -capacity is gone. +Retries are the simplest amplifier (2× in Figure 2, §2.1); §3 mentions +a 100× amplification anecdote in the wild. The look-aside cache (§2.2) +is the scarier common case: a 90% hit rate lets a 3,000 QPS application +run on a 300 QPS database (only 1 in 10 requests reaches the DB), so +losing the cache is a 10× work amplification — and the cold cache +cannot refill, because refilling requires database reads and the +database is saturated. Hidden capacity 300 QPS, advertised capacity +3,000 QPS: a 10× gap versus the retry loop's 2×. A third amplifier +hides in error handling itself (§2.3): if the error path costs more +than the success path (e.g., logging that takes locks), the system does +its most expensive work exactly when capacity is gone. ```mermaid graph TD @@ -110,38 +175,57 @@ graph TD ### Step 5 — the loop spans systems that are individually fine -The paper's flagship case study is Facebook's link-imbalance outage: -an MRU connection pool interacted with hash-assigned aggregated -network links to form a sustaining loop — congestion on one link -slowed its connections, and the MRU policy then concentrated traffic -onto exactly those connections, keeping the link congested. It went -undiagnosed for over 2 years; the eventual fix was a one-line change -to the connection-pool policy. The lesson: no single component was -broken — the feedback loop only exists in the composition, which is -why the paper calls metastable failures "emergent behavior rather than -a logic bug — one cannot write a unit or integration test to trigger -them." +> **In:** the idea (Step 4) that amplification lives in a mechanism. +> **Out:** the production lesson — the loop can live in the +> *composition* of two correct components, which is why you cannot +> unit-test for it. + +The paper's flagship case study is Facebook's link-imbalance outage +(§2.4): an MRU (most-recently-used) connection pool interacted with +hash-assigned aggregated network links to form a sustaining loop — +congestion on one link slowed its connections, and the MRU policy then +concentrated traffic onto exactly those connections, keeping the link +congested. It went undiagnosed for over two years; the eventual fix was +a one-line change to the connection-pool policy. The lesson: no single +component was broken — the feedback loop only exists in the +composition. The paper's conclusion (§5) generalizes it: metastable +failures "are an emergent behavior rather than a logic bug — one cannot +write a unit or integration test to trigger them." ### Step 6 — breaking the loop: change policy under overload -The mitigations share one shape: detect persistent overload, then +> **In:** the sustaining loops catalogued in Steps 4–5. +> **Out:** the shared shape of the mitigations (§3) — detect persistent +> overload, then switch policy — and the CoDel-style detection signal +> that tells a burst apart from real overload. + +The mitigations (§3) share one shape: detect persistent overload, then *switch policies* rather than trying harder at the normal one — LIFO queues, retry budgets, circuit breakers, smaller queues, disabling failover. Detection matters because bursts are normal: the paper -endorses a CoDel-style signal — the *minimum* queueing latency over a -sliding window; a burst leaves the minimum low, persistent overload -raises it. Other levers: give retries lower priority; make error paths -fast (a bounded lock-free queue feeding a logging thread, sampled -stack traces); and define a "characteristic metric" per known feedback -loop — retry rate, cache hit rate — since goodput alone tells you that -you are dying, not which loop is killing you. +endorses a **CoDel**-style signal — the *minimum* queueing latency over +a sliding window; a burst leaves the minimum low, persistent overload +raises it. Other levers from §3: give retries lower priority; make +error paths fast (a bounded lock-free queue feeding a logging thread, +sampled stack traces); run live-traffic stress tests (Facebook's +Kraken); and define a **characteristic metric** per known feedback loop +— retry rate, cache hit rate — since goodput alone tells you that you +are dying, not which loop is killing you. §3 also names the +organizational-incentives trap directly: the cache-eviction improvement +that widens the hidden-capacity gap is a false economy. ### Step 7 — trigger intensity, distance from the cliff +> **In:** the hidden-capacity gap quantified in Step 3. +> **Out:** why "vulnerable" is a spectrum — how large a trigger you +> survive is the margin between offered load and hidden capacity — and +> why that makes small-scale stress tests weak. + How big a trigger you survive depends on how deep in the vulnerable -region you sit. A system at 151 QPS recovers from a much bigger spike -than one at 299 QPS — both are "vulnerable," but the margin between -offered load and hidden capacity is the real safety budget: +region you sit. §4 makes this a number: a system at 151 QPS recovers +from a much bigger spike than one at 299 QPS — both are "vulnerable," +but the margin between offered load and hidden capacity is the real +safety budget (the paper calls this **trigger intensity**): ``` QPS @@ -159,38 +243,46 @@ offered load and hidden capacity is the real safety budget: This is also why testing is hard: stress tests at small scale are weak at finding metastable failures, because the loop's gain depends on scale and traffic shape — Facebook's Kraken does live-traffic testing -instead. And reproducing one requires a load generator free of -coordinated omission — the paper cites Gil Tene here, exactly topic -34's lane 1: a generator that backs off when the server slows down -silently erases the sustained arrivals that make Fig 2 lock up. +instead (§3). And reproducing one requires a load generator free of +**coordinated omission** — §4 cites Gil Tene here, exactly topic 34's +lane 1: a closed-loop generator that backs off when the server slows +down silently erases the sustained arrivals that make Figure 2 lock up. ## How to read the paper (with the concepts in hand) -7 pages, HotOS position-paper style; budget ~1h. - -- **§1** (10 min) — the definition and the trigger-vs-root-cause - claim (Step 1). Read carefully; every later section leans on the - "root cause = sustaining loop" framing. -- **§2** (15 min) — the state machine and **Figure 2**. This is the - figure to stare at: reproduce the 280→560 QPS arithmetic (Step 3) on - paper, then verify it against this topic's simulator output. -- **§3** (10 min) — the vulnerability discussion (Steps 2 and 7): - why systems run vulnerable deliberately, and why trigger intensity - interacts with distance from the cliff. -- **§4** (10 min) — the catalog of sustaining loops (Step 4): retries, - look-aside cache, slow error handling. Skim the list but slow down - on the cache arithmetic — it is the 10× version of Fig 2's 2×. -- **§5** (10 min) — the Facebook link-imbalance case study (Step 5). - Read fully; it is the only production narrative in the paper. -- **§6** (10 min) — approaches to handling (Step 6): policy switches, - CoDel-style detection, retry priority, fast error paths, Kraken, - characteristic metrics, the organizational-incentives point. -- **§7** (5 min) — skim the research agenda; note which items your - simulator already touches. +7 pages, HotOS position-paper style; budget ~1 h. The paper has five +sections — do not expect a §6 or §7. + +- **§1 Introduction** (15 min) — the definition, **Figure 1**'s state + machine, and the trigger-vs-root-cause claim (Steps 1–2). Read + carefully; every later section leans on "the true root cause is the + sustaining effect." This is also where *vulnerable-on-purpose* lives. +- **§2 Case Studies** (20 min) — the four sustaining loops. + - **§2.1 Request Retries** holds **Figure 2**; this is the figure to + stare at. Reproduce the 280 → 560 QPS arithmetic and the 150/20 QPS + thresholds (Step 3) on paper, then verify against this topic's lane + 1. + - **§2.2 Look-aside Cache** — the 90% hit / 3,000 QPS / 10× cold-cache + case (Step 4). + - **§2.3 Slow Error Handling** — the amplifier hiding in the error + path (Step 4). + - **§2.4 Link Imbalance** — the Facebook MRU-pool case study (Step 5); + the only production narrative in the paper. Read it fully. +- **§3 Approaches to Handling Metastability** (15 min) — the mitigations + (Step 6): policy switches, CoDel-style detection, retry priority, + fast error paths, Kraken, characteristic metrics, and the + organizational-incentives point. +- **§4 Discussion and Research Directions** (10 min) — work + amplification, hidden vs advertised capacity, and **trigger intensity + (151 vs 299 QPS)** (Steps 4 and 7), plus the coordinated-omission note + that ties back to topic 34. +- **§5 Conclusion** (5 min) — short; note the "emergent behavior … one + cannot write a unit or integration test" line (Step 5) and which + research items your simulator already touches. ## Questions to answer in notes.md -1. In Fig 2's system, why is recovery possible at load below 150 QPS +1. In Figure 2's system, why is recovery possible at load below 150 QPS or retry rate below 20 QPS, but not at 200 QPS offered? Derive both thresholds from the 300 QPS capacity and the 1-retry policy. 2. The look-aside cache gives 10× amplification vs the retry loop's @@ -206,35 +298,103 @@ silently erases the sustained arrivals that make Fig 2 lock up. sliding window distinguish persistent overload from a burst, where average or p99 queueing latency does not? 5. Explain, using topic 34's coordinated-omission argument, why a - closed-loop load generator cannot reproduce Fig 2 — what does it do - during the 10 s outage that an open-loop generator does not? + closed-loop load generator cannot reproduce Figure 2 — what does it + do during the 10 s outage that an open-loop generator does not? ## Done when +Answer each before unfolding it. + - [ ] You can draw the stable → vulnerable → metastable state machine from memory and state why the bad state persists after the trigger is removed. -- [ ] You can reproduce Fig 2's arithmetic (280 offered → 560 + +
Answer + + The three states are Figure 1 (§1): **stable** (below the hidden load + threshold), **vulnerable** (healthy but one trigger away from the + trap), **metastable** (the bad state). Rising load moves stable → + vulnerable; a temporary **trigger** moves vulnerable → metastable; and + the arrow that matters has no passive reverse — the **sustaining + effect** (a work-amplifying feedback loop) holds the system in the bad + state after the trigger is gone. §1: leaving it "requires a strong + corrective push, such as rebooting the system or dramatically reducing + the load." That self-loop is the difference from a DoS, limplock, or + livelock, which §1 explicitly says "are not metastable" because they + clear when the trigger clears. + +
+ +- [ ] You can reproduce Figure 2's arithmetic (280 offered → 560 sustained vs 300 capacity; recovery below 150 QPS load or 20 QPS - retries) and match lane 1 of `experiments/`: 280 QPS never - recovers (offered locks at 560 QPS, goodput 0 at t=199 s though - the outage ended at t=40 s), 140 QPS heals at t=161 s. + retries) and match lane 1 of `experiments/`. + +
Answer + + With a 1-retry-after-1s policy, a stuck server sees offered + retries. + At 280 QPS offered every request is retried: `280 + 280 = 560 QPS` + against a 300 QPS ceiling, so the queue never drains and goodput sits + at 0. The load threshold is `300 / (1 + retries) = 150 QPS` (need + `2 × offered ≤ 300`); the retry threshold at 280 QPS offered is the + leftover headroom `300 − 280 = 20 QPS`. Lane 1 reproduces this + exactly: at 280 QPS it never recovers — offered locks at 560 QPS and + goodput is still 0 at t=161 s+ though the 10 s outage ended at t=40 s + — while at 140 QPS (below 150) goodput returns, healing at t=161 s. + +
+ - [ ] You can name three sustaining loops (retries, cold look-aside cache, slow error paths) with their amplification factors and one policy switch that breaks each. + +
Answer + + **Retries** (§2.1): 2× amplification (one retry per request); break it + with a retry budget, circuit breaker, or lower-priority retries. + **Cold look-aside cache** (§2.2): up to 10× (a 90% hit rate means the + cold DB sees 10× its healthy load, and it cannot refill because + refilling needs the saturated DB); break it by shedding load so the + cache can rebuild, or serving stale. **Slow error handling** (§2.3): + amplification equal to the error-path/success-path cost ratio; break + it by making the error path *cheaper* than success (bounded lock-free + logging queue, sampled stack traces). All three share §3's shape: + detect persistent overload with a CoDel-style minimum-queueing-latency + signal, then switch policy. + +
+ - [ ] You can explain hidden vs advertised capacity and why a better cache eviction algorithm can *widen* that gap. +
Answer + + **Advertised capacity** is what a system sustains in the healthy state + (300 QPS DB, or 3,000 QPS app behind a warm 90% cache). **Hidden + capacity** (§4) is what it sustains once the sustaining loop is active + — 150 QPS for the retry loop, 300 QPS for the cache app with a cold + cache. The gap is the danger, and §3's cache-eviction example shows + the perverse incentive: a better eviction algorithm *raises the hit + rate*, which lets you serve more load from the same DB (advertised + capacity goes up) while the DB's real ceiling is unchanged — so the + cold-cache amplification, and thus the advertised-minus-hidden gap, + gets *wider*. The efficiency win is a hidden debt a trigger calls in. + +
+ ## References **Papers** - Bronson, Aghayev, Charapko, Zhu — "Metastable Failures in Distributed Systems" (HotOS 2021) — - [PDF](https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf) + [PDF](https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf). + Figure 1 (state machine) and the definitions are in §1; Figure 2 and + the 280/560/150/20 QPS arithmetic are in §2.1; hidden capacity and the + 151-vs-299 trigger intensity are in §4; the "emergent behavior" quote + is in §5. **Cross-links** - [Topic 34 — debugging & production diagnosis](../34-debugging/README.md) - — Gil Tene's coordinated omission; the paper cites Tene for why - reproducing a metastable failure needs an open-loop load generator. + — Gil Tene's coordinated omission; §4 cites Tene for why reproducing a + metastable failure needs an open-loop load generator. - This topic's [README](README.md) and [`experiments/`](experiments/) - — the deterministic simulator whose lane 1 reproduces Fig 2 exactly. + — the deterministic simulator whose lane 1 reproduces Figure 2 exactly. diff --git a/topics/35-overload/reading-redis-backpressure.md b/topics/35-overload/reading-redis-backpressure.md index 2609ebc..c649d14 100644 --- a/topics/35-overload/reading-redis-backpressure.md +++ b/topics/35-overload/reading-redis-backpressure.md @@ -6,9 +6,9 @@ accepted command runs to completion on the one thread — so DAGOR-style priority admission is structurally unavailable. Instead, redis's overload control lives at the *edges* of the event loop, converting each unbounded queue (memory, reply buffers, time behind a script) -into a bounded, fast error. The repo is cloned at `~/repos/redis`; -this is a code-read, ~1.5h, across `evict.c`, `server.c`, and -`networking.c` (all anchors under `src/`). +into a bounded, fast error. The repo is cloned at `~/repos/redis` +(pinned at `redis@a176d1225`); this is a code-read, ~1.5 h, across +`evict.c`, `server.c`, and `networking.c` (all anchors under `src/`). ## The problem in one sentence @@ -19,10 +19,28 @@ README showed how queueing plus retries sustains zero goodput after the trigger ends; redis's stance is that no queue ever grows without a bound and a cheap error on the far side of it. +Terms of art used below: + +- **`CMD_DENYOOM`** — a per-command flag marking commands that may grow + memory (writes, mostly). Only these are rejected when over budget; a + read still runs. +- **Approximated LRU** — redis does not keep a true global + least-recently-used list; it *samples* keys and keeps a small sorted + pool of eviction candidates (Step 3). +- **Hard vs soft output-buffer limit** — a hard limit disconnects a + client the instant its pending replies exceed N bytes; a soft limit + disconnects only if replies stay above M bytes for T seconds (Step 5). + ## The concepts, step by step ### Step 1 — one thread means the edges are all you have +> **In:** nothing yet — this step frames the architectural constraint +> that shapes every later step. +> **Out:** the two moments (left edge, right edge) where a +> single-threaded server can act, and why priority shedding is off the +> table. + DAGOR sheds by priority: under pressure, drop low-priority work, keep high. That requires a queue the scheduler can reorder. Redis's event loop has none — once `processCommand` dispatches, the command owns the @@ -45,6 +63,11 @@ shaped by the scheduling freedom the architecture leaves you. ### Step 2 — the memory budget: getMaxmemoryState +> **In:** the left-edge gate from Step 1 needs a cheap "am I over +> budget?" test. +> **Out:** `getMaxmemoryState`'s budget math and its fast path — the +> precondition for the reject-before-work gate in Step 4. + `maxmemory` turns RAM into an explicit budget. `getMaxmemoryState` (`evict.c:384`) computes usage, bytes over budget (`mem_tofree`), and a `level` ratio that may exceed 1.0; it returns `C_ERR` when over the @@ -58,6 +81,11 @@ not a policy. ### Step 3 — approximated LRU: a 16-entry pool fed by 5-key samples +> **In:** Step 2's verdict that redis is over budget and must free +> memory. +> **Out:** how redis chooses victims cheaply (sample 5, keep a sorted +> 16-slot pool) so the defense itself never becomes steady-state load. + Redis keeps no global LRU list — linking every key into one would tax *every* command to fund the rare eviction. Instead `evictionPoolPopulate` (`evict.c:134`) grabs `maxmemory-samples` @@ -80,28 +108,43 @@ defense with heavy steady-state cost is work amplification. ### Step 4 — the OOM gate: reject-before-work with `-OOM` +> **In:** the budget check (Step 2) and the eviction loop (Step 3). +> **Out:** the exact reject-before-work sequence in `processCommand`, +> and why the error path is strictly cheaper than the work it replaces +> — the anti-metastability property. + Before each command, `processCommand` runs the eviction loop: `performEvictions` (`evict.c:532`) frees keys until under budget — -returning `EVICT_OK`, `EVICT_RUNNING`, or `EVICT_FAIL` — then the gate -(`server.c:4485`): +returning `EVICT_OK`, `EVICT_RUNNING`, or `EVICT_FAIL` — then the gate: ```c -int out_of_memory = (performEvictions() == EVICT_FAIL); -... -if (out_of_memory && is_denyoom_command) { - rejectCommand(c, shared.oomerr); /* server.c:4498 */ +// src/server.c:4484–4499 — the OOM gate in processCommand (4487–4494 comments elided) +4484 if (server.maxmemory && !isInsideYieldingLongCommand()) { +4485 int out_of_memory = (performEvictions() == EVICT_FAIL); +4495 if (server.current_client == NULL) return C_ERR; +4497 if (out_of_memory && is_denyoom_command) { +4498 rejectCommand(c, shared.oomerr); +4499 return C_OK; ``` `is_denyoom_command` (`server.c:4391`) is any command flagged `CMD_DENYOOM` — writes, mostly; reads still pass, because they don't -grow the queue. `-OOM` is a shared preallocated string: zero -allocation, zero keyspace work, the cheapest possible shed. Why it -matters: the error path costs strictly less than the work path, so the -more overloaded redis gets, the less each request costs it — a -stabilizing loop, not a sustaining one: anti-metastability. +grow the queue. The guard on line 4484 is important: eviction is +skipped when a yielding long command is running (Step 6), so its +replication stream is not interleaved with eviction DELs. `-OOM` is a +shared preallocated string (`shared.oomerr`): zero allocation, zero +keyspace work, the cheapest possible shed. Why it matters: the error +path costs strictly less than the work path, so the more overloaded +redis gets, the less each request costs it — a stabilizing loop, not a +sustaining one: anti-metastability. ### Step 5 — output-buffer limits: backpressure on slow readers +> **In:** the memory budget from Steps 2–4, which writers grow. +> **Out:** the right-edge limit that bounds memory grown by *slow +> readers* — and why, as producer, redis's only bounded outcome is to +> evict the consumer. + Memory pressure doesn't only come from writers. A client that issues big reads but never drains its socket forces redis to hold the replies — memory grows with no bound the *client* controls: @@ -116,9 +159,23 @@ graph LR ``` `checkClientOutputBufferLimits` (`networking.c:5151`) enforces a -**hard** limit (over N bytes: gone now) and a **soft** limit (over M -bytes for T seconds: gone), per client class — normal, replica, -pubsub; even unauthenticated clients are capped at 1KB. +**hard** limit (`used_mem >= hard_limit_bytes`: gone now, line 5175) +and a **soft** limit (over `soft_limit_bytes` continuously for +`soft_limit_seconds`: gone, lines 5178–5197), per client class. The +defaults are literal in the source, and normal clients are *unlimited* +by default — only replicas and pubsub subscribers are capped: + +```c +// src/config.c:171–175 — clientBufferLimitsDefaults {hard, soft, soft_seconds} +171 clientBufferLimitsConfig clientBufferLimitsDefaults[CLIENT_TYPE_OBUF_COUNT] = { +172 {0, 0, 0}, /* normal — no limit */ +173 {1024*1024*256, 1024*1024*64, 60}, /* slave — 256MB hard, 64MB soft/60s */ +174 {1024*1024*32, 1024*1024*8, 60} /* pubsub — 32MB hard, 8MB soft/60s */ +175 }; +``` + +Even unauthenticated clients are capped at 1 KB +(`networking.c:5157`: `used_mem > 1024 && authRequired(c)`). `closeClientOnOutputBufferLimitReached` (`networking.c:5215`) disconnects *asynchronously* (`freeClientAsync`): it is called deep inside reply-writing code, where freeing the client under your own @@ -128,20 +185,31 @@ evict the consumer. ### Step 6 — `-BUSY`: bounding the time axis +> **In:** the memory and reply queues bounded in Steps 4–5. +> **Out:** the third queue — time behind a long command — and how +> `-BUSY` keeps redis answerable instead of banking a storm. + Memory and reply bytes are two queues; *time behind a long command* is the third. While a Lua script or module command runs past -`busy_reply_threshold` (checked in `script.c:150`, default 5s), redis -re-enters the event loop just enough to answer new commands with -`-BUSY` — the shared errors at `server.c:2130` — instead of banking a -storm it must later drain. `isInsideYieldingLongCommand` -(`server.c:825`) marks this mode; Step 4's gate consults it to skip -eviction while yielded (eviction DELs must not interleave with the -script's replication stream). Why it matters: without `-BUSY`, a 30s -script under 280 QPS banks 8,400 commands — lane 1's backlog on the -time axis. +`server.busy_reply_threshold` (checked in `script.c:150`; default +5000 ms, set at `config.c:3264` under the name `busy-reply-threshold`, +alias `lua-time-limit`), redis re-enters the event loop just enough to +answer new commands with `-BUSY` — the shared errors at +`server.c:2130` — instead of banking a storm it must later drain. +`isInsideYieldingLongCommand` (`server.c:825`) marks this mode; Step +4's gate consults it to skip eviction while yielded (eviction DELs must +not interleave with the script's replication stream). Why it matters: +without `-BUSY`, a 30 s script under 280 QPS banks 30 × 280 = 8,400 +commands — lane 1's backlog on the time axis, exactly the queued mass +that sustains a metastable stall once the script returns. ### Step 7 — CLIENT PAUSE: intake suspension as choreography +> **In:** the per-axis bounds of Steps 4–6, which shed individual +> requests. +> **Out:** the blunt whole-intake suspension used for the brief windows +> (shutdown, failover) where accepting a write would lose it. + The bluntest instrument: stop accepting (some) work entirely. `pauseActions(PAUSE_DURING_SHUTDOWN, ...)` (`server.c:4850`) reuses the CLIENT PAUSE machinery during shutdown — writes are suspended so @@ -155,6 +223,10 @@ lose it: pause the herd, don't let it stampede the survivor. ### Step 8 — the pattern: every queue bounded, every error cheap +> **In:** all four shed points from Steps 4–7. +> **Out:** the single invariant that unifies them, and where redis sits +> on the scheduling-freedom axis against DAGOR and cockroach. + ``` axis queue bound cheap error ───────── ────────────────── ──────────────── ───────────────── @@ -167,9 +239,9 @@ lose it: pause the herd, don't let it stampede the survivor. Redis rejects *everything equally* — no priorities, because one thread has no scheduling freedom to honor them (Step 1). DAGOR sits mid-pole (a priority cursor over a queue it controls); cockroach's admission -package is the far pole (a full scheduler: tenant fairness, AIMD -slots, LSM-debt tokens). Same invariant everywhere: the response to -overload must cost less than the work declined. +package is the far pole (a full scheduler: tenant fairness, additive +CPU-slot control, LSM-debt tokens). Same invariant everywhere: the +response to overload must cost less than the work declined. ## Where each step lives in the code @@ -181,11 +253,13 @@ overload must cost less than the work declined. | 3 | `config.c:3223` | `maxmemory-samples` default 5 | | 4 | `evict.c:532` | `performEvictions` — the three-value return contract in the comment | | 4 | `server.c:4391` | `is_denyoom_command` — CMD_DENYOOM plus the MULTI/EXEC case | -| 4 | `server.c:4485`, `server.c:4498` | the gate: `EVICT_FAIL` → `rejectCommand(c, shared.oomerr)` | +| 4 | `server.c:4484`–`4499` | the gate: `EVICT_FAIL` → `rejectCommand(c, shared.oomerr)` | | 5 | `networking.c:5151` | `checkClientOutputBufferLimits` — soft (limit + time) vs hard; 1KB unauthenticated cap | +| 5 | `config.c:171` | `clientBufferLimitsDefaults` — normal `{0,0,0}`, replica 256/64MB/60s, pubsub 32/8MB/60s | | 5 | `networking.c:5215` | `closeClientOnOutputBufferLimitReached` — why async | | 6 | `server.c:2130` | the `-BUSY` shared error strings | -| 6 | `script.c:150`, `server.c:825` | the threshold check in a running script; `isInsideYieldingLongCommand` | +| 6 | `script.c:150`, `config.c:3264` | the threshold check in a running script; `busy-reply-threshold` default 5000 ms | +| 6 | `server.c:825` | `isInsideYieldingLongCommand` | | 7 | `server.c:4850` | `pauseActions(PAUSE_DURING_SHUTDOWN, ...)` in the shutdown path | | 7 | `networking.c:4482` | `unpauseActions(PAUSE_BY_CLIENT_COMMAND)` | @@ -215,20 +289,87 @@ Read order: `getMaxmemoryState` → `evictionPoolPopulate` → ## Done when +Answer each before unfolding it. + - [ ] You can narrate the pre-command sequence at `server.c:4485` — evict, gate, reject — and say why eviction runs *before* the check rather than lazily on allocation failure. + +
Answer + + In `processCommand`, guarded by `if (server.maxmemory && + !isInsideYieldingLongCommand())` (`server.c:4484`), redis first runs + `performEvictions()` and records `out_of_memory = (… == EVICT_FAIL)` + (`:4485`). Only if eviction failed *and* the command is + `is_denyoom_command` does it `rejectCommand(c, shared.oomerr)` and + `return C_OK` (`:4497–4499`). Eviction runs *first*, and *before* the + command executes, because reject-before-work is the whole point: a + budget discovered by malloc failing mid-command is a crash, not a + policy, and `shared.oomerr` is a preallocated string so the shed costs + no allocation and no keyspace work — strictly less than the write it + declines. + +
+ - [ ] You can explain approximated LRU (5 samples, 16-slot sorted pool, evict rightmost) and its cost when under budget (zero). + +
Answer + + A true global LRU list would tax *every* command to maintain, funding + a rare eviction from the common path. Instead `evictionPoolPopulate` + (`evict.c:134`) samples `maxmemory-samples` random keys (default 5, + `config.c:3223`) and merges the better candidates into a persistent + sorted pool of `EVPOOL_SIZE` = 16 entries (`evict.c:36`), best victim + at the right end. Over many passes this converges on nearly the same + victims as exact LRU at O(samples) cost — and that cost is paid only + when actually over budget, so the steady-state overhead is zero. The + governance mechanism must not itself become load; a defense with heavy + steady-state cost would be its own work amplification. + +
+ - [ ] You can name the three queues (memory, replies, time), the bound on each, and the cheap error on the far side of each bound. + +
Answer + + **Memory** — keyspace bytes, bounded by `maxmemory`, cheap error + `-OOM` (`shared.oomerr`, Step 4). **Replies** — per-client output + buffer, bounded by hard/soft `client-output-buffer-limit` + (`networking.c:5151`; defaults `{0,0,0}` normal, 256/64 MB·60 s + replica, 32/8 MB·60 s pubsub at `config.c:171`), cheap error an async + disconnect (Step 5). **Time** — commands banked behind a long + script/module, bounded by `busy-reply-threshold` (default 5000 ms, + `config.c:3264`), cheap error `-BUSY` (`server.c:2130`, Step 6). A + fourth, blunt lever — `CLIENT PAUSE` — suspends intake entirely for + short windows (Step 7). Every queue has a bound and a far-side error + cheaper than the work it replaces. + +
+ - [ ] You can place redis, DAGOR, and cockroach admission on the scheduling-freedom axis and say which surface redis cannot build. +
Answer + + Redis is the low-freedom pole: one thread, no reorderable queue, so it + can only gate at the left edge (before dispatch) and cut at the right + edge (reply buffers), and it rejects *everything equally* — it cannot + build a priority admission surface at all (Step 1). DAGOR is the + mid-pole: a priority cursor over a queue it controls, shedding + low-priority whole tasks first. Cockroach's `pkg/util/admission` is + the far pole: a full user-space scheduler with tenant fairness, + additive CPU-slot control, and LSM-debt tokens, reordering work by + priority and tenant. The invariant is shared across all three: the + overload response must cost less than the declined work. + +
+ ## References -- [redis](https://github.com/redis/redis) — cloned at `~/repos/redis`; - all anchors under `src/` +- [redis](https://github.com/redis/redis) — cloned at `~/repos/redis`, + pinned at `redis@a176d1225`; all anchors under `src/` - [README.md](README.md) — topic 35: metastability, hidden capacity, the code-anchor table this guide expands - [reading-cockroach-admission.md](reading-cockroach-admission.md) — diff --git a/topics/36-sharding/README.md b/topics/36-sharding/README.md index 3d74988..947072b 100644 --- a/topics/36-sharding/README.md +++ b/topics/36-sharding/README.md @@ -125,7 +125,8 @@ load. ``` PowerGraph's argument (OSDI'12): natural graphs are power-law -(P(d) ∝ d^−α, α≈2; Twitter's follower graph has in-degree α=1.7 and 1% +(P(d) ∝ d^−α, α≈2; Twitter's follower graph has a heavier in-degree +tail than out-degree (Fig 1) and 1% of vertices adjacent to nearly half the edges), so *balanced edge-cuts barely beat random* — Theorem 5.1: random vertex placement cuts 1−1/p of edges. Cutting **vertices** instead works: assign each *edge* diff --git a/topics/36-sharding/notes.md b/topics/36-sharding/notes.md index 520bc1d..f13d804 100644 --- a/topics/36-sharding/notes.md +++ b/topics/36-sharding/notes.md @@ -54,7 +54,7 @@ can spread the rank-0 key's ~10% of traffic. Splitting *between* keys - No new clones: redis and cockroach already under ~/repos. - Redis anchors verified by grep this session: cluster.h:23 (CLUSTER_SLOTS = 1<<14), :59 (keyHashSlot, hash-tag carve-out); - cluster.c:36 (patternHashSlot), :1191 (getNodeByQuery), :1397 + cluster.c:35 (patternHashSlot), :1191 (getNodeByQuery), :1397 (CLUSTER_REDIR_ASK), :1432 (CLUSTER_REDIR_MOVED), :1443 (clusterRedirectClient), :1680 (askingCommand); cluster_legacy.h:343/:344 (migrating_slots_to / importing_slots_from); cluster_legacy.c:6072-6075 @@ -66,14 +66,17 @@ can spread the rank-0 key's ~10% of traffic. Splitting *between* keys (shouldQueue); split/decider.go:155 (Decider), :222 (Record), :329 (RecordMax); allocatorimpl/allocator.go:125-127 (AllocatorAction); store_rebalancer.go:114 (StoreRebalancer), :218 (RebalanceMode). -- Papers verified from PDFs: Dynamo (SOSP'07, /tmp/dynamo.pdf, §4 + +- Papers verified from PDFs: Dynamo (SOSP'07, `.cache/papers/dynamo-sosp07.txt`, §4 + §6.1-6.3) — MD5→128-bit ring, tokens/vnodes, N/R/W with common (3,2,2), sloppy quorum + hinted handoff, per-range Merkle trees, strategies 1/2/3 (strategy-1 bootstrap "almost a day", strategy 3 = Q/S tokens, metadata 3 orders smaller, partition-as-file), imbalance ratio 20% low load vs 10% high (15% threshold), 99.94% of reads see - one version; PowerGraph (OSDI'12, /tmp/powergraph.pdf, pp. 1-8) — - GAS, α≈2 natural graphs, Twitter in-degree α=1.7, 1% of vertices ~ + one version; PowerGraph (OSDI'12, `.cache/papers/powergraph-osdi12.txt`, + pp. 1-8) — GAS, α≈2 natural graphs (the paper assigns no numeric α + to any real graph; 1.65/1.7/1.8/2.0 are Fig 6 synthetic-curve + labels), Twitter's in-degree tail heavier than its out-degree + (Fig 1), 1% of vertices ~ half the edges, Thm 5.1 (random cut 1−1/p), Thm 5.2 (replication from degree distribution), Thm 5.3 (vertex-cut ≤ ghosts of any edge-cut), greedy Cases 1-4, coordinated vs oblivious, Table 1 diff --git a/topics/36-sharding/reading-cockroach-rebalancing.md b/topics/36-sharding/reading-cockroach-rebalancing.md index f653cf1..6a14394 100644 --- a/topics/36-sharding/reading-cockroach-rebalancing.md +++ b/topics/36-sharding/reading-cockroach-rebalancing.md @@ -1,40 +1,49 @@ # CockroachDB rebalancing: ranges that split where the load is CockroachDB is the production counter-design to Redis Cluster's fixed 16384 hash slots -(see `reading-redis-cluster.md`): the keyspace is one big sorted map, chopped into -contiguous **ranges** (~512 MB each, every range a Raft group — topics 15/21), and the -range boundaries *move*. Ranges split when they get too big **or too hot**, merge back -when they get small and cold, and get shuffled between stores by a two-level placement -machine (per-range allocator + store-level rebalancer). This guide walks the code that -makes those four decisions. - -The payoff for topic 36 is the hot-key story. Lane 1's Zipf experiment showed hashing's -dead end: with Zipf(1.0) on 16 hash shards, the hottest shard takes 14.7% of traffic -(2.4x its fair share) and no re-hashing can split a single key's load. Range -partitioning *can* split between keys — and CockroachDB's `Decider` even keeps honest -counters for the moment that stops working, when one key IS the load. +(see [reading-redis-cluster.md](reading-redis-cluster.md)): the keyspace is one big sorted map, +chopped into contiguous **ranges** (~512 MB each, every range a Raft group — topics 15/21), and +the range boundaries *move*. Ranges split when they get too big **or too hot**, merge back when +they get small and cold, and get shuffled between stores by a two-level placement machine +(per-range **allocator** + store-level **rebalancer**). This guide walks the code that makes +those four decisions, in `~/repos/cockroach/pkg` at the SHA in the topic's pin table; every +`file:line` below was checked against that pinned tree this session. + +The payoff for topic 36 is the hot-key story. Lane 1's Zipf experiment showed hashing's dead +end: with Zipf(1.0) on 16 hash shards, the hottest shard takes 14.7% of traffic (2.4× its fair +share) and no re-hashing can split a single key's load. Range partitioning *can* split between +keys — and CockroachDB's `Decider` even keeps honest counters for the moment that stops working, +when one key IS the load. This complements the topic headline (FINDINGS row 36): mod-N's 94.1% +remap on a 16→17 grow is the *placement* disaster; the Zipf hot key is the *load* disaster, and +ranges address both. ## The problem in one sentence -**Static partitioning of a skewed, shifting workload leaves some shards overloaded and -others idle; CockroachDB instead treats partition boundaries and replica placement as -continuously re-optimized outputs of observed size and load.** +**Static partitioning of a skewed, shifting workload leaves some shards overloaded and others +idle; CockroachDB instead treats partition boundaries and replica placement as continuously +re-optimized outputs of observed size and load.** -Every mechanism below is a feedback loop: measure a range (bytes, QPS, CPU) or a store -(aggregate load), compare to a threshold, and emit a cheap corrective action — split, -merge, lease transfer, replica move — ordered from cheapest to most expensive. +Every mechanism below is a feedback loop: measure a range (bytes, QPS, CPU) or a store (aggregate +load), compare to a threshold, and emit a cheap corrective action — split, merge, lease transfer, +replica move — ordered from cheapest to most expensive. ## The concepts, step by step ### Step 1 — Range partitioning: contiguous spans you can cut anywhere -The keyspace is ordered; a range is the span between two boundary keys. Hashing scatters -adjacent keys to fixed buckets; ranges keep them together, so a split is just "pick a -key in the middle, write two descriptors." The default size cap is in -`pkg/config/zonepb/zone.go`: +> **In:** the mod-N / hash-slot world where a key's bucket is a fixed function of its hash. +> **Out:** the **range** — a span between two boundary keys, cuttable at *any* key — which is the +> structural precondition for load-based splits (Steps 3–4) that hashing cannot do. + +A **range** is the contiguous span of the sorted keyspace between two boundary keys; it is the +unit of replication (one Raft group) and placement. Hashing scatters adjacent keys to fixed +buckets; ranges keep them together, so a split is just "pick a key in the middle, write two +descriptors." The default size band is in `pkg/config/zonepb/zone.go`: ```go -RangeMaxBytes: proto.Int64(512 << 20), // 512 MB +// pkg/config/zonepb/zone.go:256-257 — default range size band (DefaultZoneConfig) +256 RangeMinBytes: proto.Int64(128 << 20), // 128 MB +257 RangeMaxBytes: proto.Int64(512 << 20), // 512 MB ``` ```text @@ -47,44 +56,60 @@ RangeMaxBytes: proto.Int64(512 << 20), // 512 MB ^ hot? split r3 at any key between k and s ``` -Because a range is a Raft group, a split is metadata-cheap: allocate a new range ID, -new descriptor, new Raft group at the boundary key. No data is copied at split time — -that expense is deferred to rebalancing (step 7). +Because a range is a Raft group, a split is metadata-cheap: allocate a new range ID, new +descriptor, new Raft group at the boundary key. No data is copied at split time — that expense is +deferred to rebalancing (Step 7). ### Step 2 — Size-based splits: the split queue -Each store runs a `splitQueue`. Its `shouldQueue` (`split_queue.go:194`) asks -`shouldSplitRange` (`split_queue.go:145`): does the range exceed `RangeMaxBytes` for -its zone, or does the load-split machinery (steps 3-4) say split? Size splits keep -snapshots, backups, and Raft log truncation bounded; they are the boring baseline that -runs regardless of load. +> **In:** the range from Step 1, growing as writes accumulate. +> **Out:** the `splitQueue` size trigger — the boring baseline that keeps snapshots and Raft logs +> bounded regardless of load, and the frame into which Steps 3–4 bolt the *load* trigger. + +The **split queue** is a per-store background queue that scans replicas and decides which to +split. Its `shouldQueue` (`split_queue.go:194`) asks `shouldSplitRange` (`split_queue.go:145`): +does the range exceed `RangeMaxBytes` for its zone, or does the load-split machinery (Steps 3–4) +say split? Size splits keep snapshots, backups, and Raft log truncation bounded; they are the +baseline that runs regardless of load. ### Step 3 — Load-based splits: from QPS to CPU as the signal -Size says nothing about heat: a 64 MB range can eat a core. Two cluster settings in -`replica_split_load.go` define "too hot": +> **In:** the size-only trigger of Step 2, blind to heat (a 64 MB range can saturate a core). +> **Out:** the two "too hot" thresholds — QPS (legacy) and CPU (default) — that turn observed +> *load*, not bytes, into a split candidate; Step 4 then decides *where* to cut. + +Size says nothing about heat. Two cluster settings in `replica_split_load.go` define "too hot" +— **QPS** (queries per second, the legacy signal) and **CPU** (attributed CPU-seconds per wall +second, the current default): ```go -// replica_split_load.go:34 -SplitByLoadQPSThreshold // "kv.range_split.load_qps_threshold", default 2500 -// replica_split_load.go:52 -SplitByLoadCPUThreshold // "kv.range_split.load_cpu_threshold", default 500ms +// pkg/kv/kvserver/replica_split_load.go:34-56 — the two "too hot" thresholds +34 var SplitByLoadQPSThreshold = settings.RegisterIntSetting( +36 "kv.range_split.load_qps_threshold", +38 2500, // 2500 req/s +52 var SplitByLoadCPUThreshold = settings.RegisterDurationSetting( +54 "kv.range_split.load_cpu_threshold", +56 500 * time.Millisecond, ``` -CPU became the default objective: 500ms of attributed CPU per second of wall time, -i.e. half a core per range. The comment block explains why 500ms: attributed CPU is -roughly one third of real usage, so at most ~cores/1.5 load splits happen per node, and -the value was tuned with kv(0|95) workloads and allocbench (issue #96869). QPS is a -poor proxy — one query can be a point read or a full scan — while CPU is the resource -that actually saturates. +CPU became the default objective: 500 ms of attributed CPU per second of wall time, i.e. half a +core per range. The comment block (`replica_split_load.go:41-51`) explains the number: attributed +CPU is roughly one third of "real" usage (real ≈ 3× attributed), so in steady state at most +~cores/1.5 load splits happen per node; the value was tuned by running kv(0|95) and allocbench +and picking the best-performing threshold (issue #96869). QPS is a poor proxy — one query can be +a point read or a full scan — while CPU is the resource that actually saturates. ### Step 4 — The Decider: WHERE to split, and when no key helps -Crossing the threshold answers "should we split?" but not "where?" — the median of the -*key distribution* is useless if 99% of requests hit one end. `Decider` -(`split/decider.go:155`) holds per-replica load-split state: every request span is fed -into a windowed per-key load sketch via `Record` (`decider.go:222`, plus `RecordMax` -at `:329`), and the Decider searches for a key with ~half the observed load on each side. +> **In:** the "this range is too hot" verdict from Step 3. +> **Out:** either a balanced split *key*, or one of three honest failure counters — the explicit +> detection of the Zipf single-hot-key case that partitioning provably cannot fix. + +Crossing the threshold answers "should we split?" but not "where?" — the median of the *key +distribution* is useless if 99% of requests hit one end. **`Decider`** (`split/decider.go:155`) +holds per-replica load-split state: every request span is fed into a windowed per-key load sketch +via `Record` (`decider.go:222`, plus `RecordMax` at `:329`), and the Decider searches for a key +with ~half the observed load on each side. ```mermaid flowchart TD @@ -97,29 +122,39 @@ flowchart TD PopularKeyCount / NoSplitKeyCount / ClearDirectionCount] ``` -The failure counters are the honest part (`LoadSplitterMetrics`): +The failure counters are the honest part — the `LoadSplitterMetrics` struct at +`decider.go:146-149`, incremented deep in the finder: -- **PopularKeyCount** — a single key dominates; every candidate boundary leaves the - load on one side. Splitting *cannot* help. This is the Zipf hot-key lesson surviving - into range partitioning: when one key stands alone, the remaining tools are - replication (spread reads) and admission control (topic 35), not partitioning. -- **NoSplitKeyCount** — no balanced boundary found (e.g. spans straddle every candidate). -- **ClearDirectionCount** — load is a moving scan front (sequential ingest); the "hot - half" keeps shifting, so a split would go stale immediately. +- **PopularKeyCount** (`decider.go:147`, incremented at `:293`) — a single key dominates; every + candidate boundary leaves the load on one side. Splitting *cannot* help. This is the Zipf + hot-key lesson surviving into range partitioning: when one key stands alone, the remaining tools + are replication (spread reads) and admission control (topic 35), not partitioning. +- **NoSplitKeyCount** (`decider.go:148`, incremented at `:308`) — no balanced boundary found + (e.g. spans straddle every candidate). +- **ClearDirectionCount** (`decider.go:149`, incremented at `:300`) — load is a moving scan front + (sequential ingest); the "hot half" keeps shifting, so a split would go stale immediately. ### Step 5 — The merge queue: splits must be undoable -Without merges, splits are a one-way ratchet: a table that shrank, or a load spike -that passed, leaves behind tiny ranges whose fixed overhead (Raft heartbeats, replica -metadata, queue scanning) accumulates forever. `mergeQueue.shouldQueue` -(`merge_queue.go:138`) finds a range that is small and cold, checks its *right -neighbor*, and merges the pair back if the combined range would not immediately -re-split on size or load. Note the symmetry: merge criteria mirror split criteria, so -the two queues don't fight. +> **In:** the accumulated splits from Steps 2–4, some now stale (table shrank, spike passed). +> **Out:** the `mergeQueue` reverse gear, whose criteria mirror the split thresholds so the two +> queues can't oscillate. + +Without merges, splits are a one-way ratchet: a table that shrank, or a load spike that passed, +leaves behind tiny ranges whose fixed overhead (Raft heartbeats, replica metadata, queue +scanning) accumulates forever. **`mergeQueue.shouldQueue`** (`merge_queue.go:138`) finds a range +that is small and cold, checks its *right neighbor*, and merges the pair back if the combined +range would not immediately re-split on size or load. Note the symmetry: merge criteria mirror +split criteria (with hysteresis), so the two queues don't fight. ### Step 6 — Allocator vs store rebalancer: two levels of placement -Placement decisions are split across two components with different scopes: +> **In:** ranges that exist and are sized (Steps 1–5) but may be mis-*placed* across stores. +> **Out:** the two-scope placement machine — a per-range **allocator** and a per-store +> **rebalancer** — and its cheapest-first ordering (lease transfer before replica move). + +Placement decisions are split across two components with different scopes. The **allocator** asks +"is THIS range healthy?"; the **store rebalancer** asks "is THIS store overloaded?": ```text per-RANGE view per-STORE view @@ -136,30 +171,44 @@ Placement decisions are split across two components with different scopes: ``` The allocator's decision enum is `AllocatorAction` -(`allocator/allocatorimpl/allocator.go:125-127`); it repairs individual ranges -(under-replicated, wrong locality, dead store). `StoreRebalancer` -(`store_rebalancer.go:114`, mode at `:218`) is a separate store-level loop — its doc -comment says it is "motivated by store-level load imbalances" — that sheds load from -hot stores to cold ones, preferring **lease transfers first**: moving a lease shifts -read and coordination load instantly without copying a byte. Only when leases aren't -enough does it move replicas. +(`allocator/allocatorimpl/allocator.go:125-127`); it repairs individual ranges (under-replicated, +wrong locality, dead store). **`StoreRebalancer`** (`store_rebalancer.go:114`; `RebalanceMode` at +`:218`) is a separate store-level loop. Its struct doc comment (`store_rebalancer.go:104-113`) +makes the key point — it is deliberately *not* a Queue, because Queues decide one replica at a +time and can't see how a replica compares to others on the store, whereas the goal here is +*store-level* balance. The phrase "motivated by store-level load imbalances" that names this loop's +work is the `Help` string on its two metrics — `rebalancing.lease.transfers` +(`store_rebalancer.go:32`) and `rebalancing.range.rebalances` (`store_rebalancer.go:41`). + +It sheds load from hot stores to cold ones **lease-first**: the `rebalanceStore` doc comment +(`store_rebalancer.go:373-377`) and its numbered phases (`store_rebalancer.go:389-395`) spell out +the order — Phase (1) search for lease-transfer targets for the hottest leases; only after it runs +out of leases to transfer (Phase (2)) does it move replicas. Moving a lease shifts read and +coordination load instantly without copying a byte; moving a replica streams a snapshot. ### Step 7 — Rebalancing is itself load -Moving a replica means streaming a snapshot of up to ~512 MB into the target store — -real disk and network work competing with foreground queries. Snapshots are paced and -throttled, and snapshot *ingestion* is governed by the same admission-control machinery -(topic 35's `io_load_listener`) that protects foreground writes from LSM overload. The -general lesson: a rebalancer without pacing converts "imbalance" into "outage" — -migration traffic must be a background-priority tenant of the system it is healing. +> **In:** the "move a replica" action of Step 6, the expensive branch. +> **Out:** the pacing/admission-control constraint that keeps migration traffic a +> background-priority tenant — the reason a rebalancer doesn't turn imbalance into an outage. + +Moving a replica means streaming a snapshot of up to ~512 MB into the target store — real disk and +network work competing with foreground queries. Snapshots are paced and throttled, and snapshot +*ingestion* is governed by the same admission-control machinery (topic 35's `io_load_listener`) +that protects foreground writes from LSM overload. The general lesson: a rebalancer without pacing +converts "imbalance" into "outage" — migration traffic must be a background-priority tenant of the +system it is healing. ### Step 8 — What M36 copies -The capstone milestone borrows the shape, not the code: (1) split on a measured signal -with a threshold + hysteresis, not on key count; (2) pick the split point from observed -request load, and detect the one-hot-key case explicitly instead of splitting uselessly; -(3) make merges mirror splits; (4) do the cheap rebalancing action (ownership/lease -move) before the expensive one (data move), and throttle the expensive one. +> **In:** the four decisions traced in Steps 2–7. +> **Out:** the four design rules the M36 capstone borrows — the shape, not the Go code. + +The capstone milestone borrows the shape, not the code: (1) split on a measured signal with a +threshold + hysteresis, not on key count; (2) pick the split point from observed request load, and +detect the one-hot-key case explicitly (a `PopularKeyCount`-style counter) instead of splitting +uselessly; (3) make merges mirror splits so they don't oscillate; (4) do the cheap rebalancing +action (ownership/lease move) before the expensive one (data move), and throttle the expensive one. ## Where each step lives in the code @@ -167,48 +216,107 @@ All paths relative to `~/repos/cockroach/pkg`. | Step | Anchor | What to read | |---|---|---| -| 1 | `config/zonepb/zone.go:257` | `RangeMaxBytes` default: `512 << 20` | -| 2 | `kv/kvserver/split_queue.go:145,:194` | `shouldSplitRange`, `shouldQueue` | -| 3 | `kv/kvserver/replica_split_load.go:34,:52` | QPS (2500) and CPU (500ms) thresholds; comment on why 500ms | -| 4 | `kv/kvserver/split/decider.go:155,:222,:329` | `Decider` struct, `Record`, `RecordMax`; `LoadSplitterMetrics` counters | +| 1 | `config/zonepb/zone.go:256-257` | `RangeMinBytes` (128 MB) / `RangeMaxBytes` (512 MB) defaults | +| 2 | `kv/kvserver/split_queue.go:145,:194` | `shouldSplitRange`, `splitQueue.shouldQueue` | +| 3 | `kv/kvserver/replica_split_load.go:34,:52` | QPS (2500) and CPU (500 ms) thresholds; comment `:41-51` on why 500 ms | +| 4 | `kv/kvserver/split/decider.go:146-149,:155,:222,:329` | `LoadSplitterMetrics`, `Decider`, `Record`, `RecordMax`; increments at `:293/:300/:308` | | 5 | `kv/kvserver/merge_queue.go:138` | `mergeQueue.shouldQueue` | | 6 | `kv/kvserver/allocator/allocatorimpl/allocator.go:125-127` | `AllocatorAction` enum | -| 6-7 | `kv/kvserver/store_rebalancer.go:114,:218` | `StoreRebalancer`, `RebalanceMode`; lease-first doc comment | +| 6 | `kv/kvserver/store_rebalancer.go:104-113,:114,:218` | `StoreRebalancer` doc + struct, `RebalanceMode` | +| 6 | `kv/kvserver/store_rebalancer.go:32,:41,:373-377,:389-395` | metric `Help` strings; lease-first phase ordering in `rebalanceStore` | ## Questions to answer in notes.md -1. `SplitByLoadCPUThreshold` defaults to 500ms of attributed CPU per second. Per the - comment at `replica_split_load.go:52`, why does that imply at most ~cores/1.5 load - splits per node, and what workloads was the value tuned against? -2. Walk `splitQueue.shouldQueue` (`split_queue.go:194`): how are the size trigger and - the load trigger combined, and which produces the split *key* in each case? -3. The Decider increments `PopularKeyCount` vs `NoSplitKeyCount` vs - `ClearDirectionCount` in different situations. Give a concrete workload that - triggers each, and say what the operator's correct response is for each. -4. In `mergeQueue.shouldQueue` (`merge_queue.go:138`), what conditions must hold on the - range and its neighbor before a merge is attempted, and how do they mirror the split - thresholds so split/merge don't oscillate? -5. Why does the `StoreRebalancer` try lease transfers before replica moves? List what a - lease transfer shifts versus what a replica rebalance costs, citing the doc comment - at `store_rebalancer.go:114`. +1. `SplitByLoadCPUThreshold` defaults to 500 ms of attributed CPU per second. Per the comment at + `replica_split_load.go:41-51`, why does that imply at most ~cores/1.5 load splits per node, and + what workloads was the value tuned against? +2. Walk `splitQueue.shouldQueue` (`split_queue.go:194`): how are the size trigger and the load + trigger combined, and which produces the split *key* in each case? +3. The Decider increments `PopularKeyCount` (`decider.go:293`) vs `NoSplitKeyCount` (`:308`) vs + `ClearDirectionCount` (`:300`) in different situations. Give a concrete workload that triggers + each, and say what the operator's correct response is for each. +4. In `mergeQueue.shouldQueue` (`merge_queue.go:138`), what conditions must hold on the range and + its right neighbor before a merge is attempted, and how do they mirror the split thresholds so + split/merge don't oscillate? +5. Why does the `StoreRebalancer` try lease transfers before replica moves? Using the phase list + at `store_rebalancer.go:389-395`, list what a lease transfer shifts versus what a replica + rebalance costs. ## Done when -- [ ] You can explain why range partitioning can absorb a Zipf hot *span* but not a - single hot key, and name the Decider counter that reports the latter. -- [ ] You can trace one split end to end: threshold check in `shouldQueue` → split key - from the Decider → new range descriptor / Raft group. -- [ ] You can state the division of labor between `AllocatorAction` and - `StoreRebalancer`, and why leases move before replicas. +Answer each before unfolding it. + +- [ ] You can explain why range partitioning can absorb a Zipf hot *span* but not a single hot + key, and name the Decider counter that reports the latter. + +
Answer + + Range partitioning cuts the keyspace at *any* key, so a hot *span* (many adjacent hot keys) can + be divided until each piece fits a store — the Decider finds a boundary with ~half the load on + each side and emits it. A single hot *key* has no such boundary: every candidate split leaves the + key (and thus almost all the load) wholly on one side, so splitting cannot reduce the hottest + range's load. The Decider reports exactly this as **`PopularKeyCount`** (`decider.go:147`, + incremented at `:293`). The remaining tools are then replication (spread reads across replicas) + and admission control (topic 35), not partitioning — the Zipf(1.0) 14.7%-hottest-shard result + from lane 1 surviving into range-land. + +
+ +- [ ] You can trace one split end to end: threshold check in `shouldQueue` → split key from the + Decider → new range descriptor / Raft group. + +
Answer + + `splitQueue.shouldQueue` (`split_queue.go:194`) calls `shouldSplitRange` (`:145`), which returns + true if the range exceeds its zone's `RangeMaxBytes` (`zone.go:257`) *or* the load-split + machinery says so. For a load split, the `Decider` (`decider.go:155`) has been fed request spans + via `Record` (`:222`); once sustained load clears `SplitByLoadCPUThreshold` + (`replica_split_load.go:52`) it searches its per-key sketch for a balanced boundary and, if one + exists, emits that key. The split queue then executes an `AdminSplit` at that key: allocate a new + range ID and descriptor and start a new Raft group at the boundary — metadata only, no data copy + (that cost is deferred to rebalancing, Step 7). + +
+ +- [ ] You can state the division of labor between `AllocatorAction` and `StoreRebalancer`, and why + leases move before replicas. + +
Answer + + `AllocatorAction` (`allocator.go:125-127`) is the *per-range* view: for one range it recommends + add / remove / replace / rebalance a replica or move the lease, fixing under-replication and + zone-config violations. `StoreRebalancer` (`store_rebalancer.go:114`) is the *per-store* view: + it compares aggregate store loads and sheds load from overloaded stores — a scope the per-range + allocator cannot see, which is why it is deliberately not a Queue (`:104-113`). + + Leases move before replicas because a lease transfer shifts read and coordination load *instantly + and byte-free*, whereas a replica move streams a snapshot (up to ~512 MB). `rebalanceStore` + (`store_rebalancer.go:373-377`, phases at `:389-395`) does Phase (1) lease transfers for the + hottest leases and only falls through to Phase (2) replica rebalances when leases can no longer + rebalance the store — cheapest corrective action first. + +
+ - [ ] All 5 questions above are answered in `notes.md` with file:line citations. +
Answer + + Done when `notes.md` records your worked answers to all five questions, each anchored to a real + `file:line` from the "Where each step lives" table (the thresholds in `replica_split_load.go`, + the three Decider counters and their increment sites, the merge symmetry in `merge_queue.go`, and + the lease-first phases in `store_rebalancer.go`), cross-checked against the pinned tree with + `tools/pinned-source.py show cockroach `. + +
+ ## References - Source: `~/repos/cockroach/pkg/kv/kvserver/` — `split_queue.go`, `merge_queue.go`, `replica_split_load.go`, `split/decider.go`, `store_rebalancer.go`, - `allocator/allocatorimpl/allocator.go`; `~/repos/cockroach/pkg/config/zonepb/zone.go` -- [Topic 36 README](README.md) — lane-1 Zipf numbers, milestone M36 -- [reading-redis-cluster.md](reading-redis-cluster.md) — the contrasting design: fixed - hash slots with manual resharding vs dynamic ranges -- Topic 35 — admission control (`io_load_listener`) governing snapshot ingestion; - topics 15/21 — Raft: every range is a Raft group + `allocator/allocatorimpl/allocator.go`; `~/repos/cockroach/pkg/config/zonepb/zone.go` (pinned + SHA in the topic's `resources/codebases.md` pin table). +- [Topic 36 README](README.md) — lane-1 Zipf numbers, milestone M36. +- [reading-redis-cluster.md](reading-redis-cluster.md) — the contrasting design: fixed hash slots + with manual resharding vs dynamic ranges. +- Topic 35 — admission control (`io_load_listener`) governing snapshot ingestion; topics 15/21 — + Raft: every range is a Raft group. diff --git a/topics/36-sharding/reading-dynamo.md b/topics/36-sharding/reading-dynamo.md index 8997990..ffac23f 100644 --- a/topics/36-sharding/reading-dynamo.md +++ b/topics/36-sharding/reading-dynamo.md @@ -1,8 +1,10 @@ # Dynamo: the ring that taught everyone consistent hashing — and then outgrew it -Dynamo (DeCandia et al., SOSP 2007) is the paper that made consistent hashing the default answer to "how do I shard a key-value store?" — and, less famously, the paper that documented why the textbook version of consistent hashing failed in production and had to be replaced. Amazon built Dynamo around 99.9th-percentile SLAs (a typical one: 99.9% of requests within 300 ms), which forced a "zero-hop DHT" design: every node knows enough routing state to reach the right node directly, because multi-hop routing à la Chord/Pastry adds latency variability exactly at the percentiles the SLA measures. +Dynamo (DeCandia et al., SOSP 2007) is the paper that made consistent hashing the default answer to "how do I shard a key-value store?" — and, less famously, the paper that documented why the textbook version of consistent hashing failed in production and had to be replaced. Amazon built Dynamo around 99.9th-percentile SLAs (a typical one: 99.9% of requests within 300 ms, §2.2), which forced a "zero-hop DHT" design: every node knows enough routing state to reach the right node directly, because multi-hop routing à la Chord/Pastry adds latency variability exactly at the percentiles the SLA measures. -For this topic, read the paper as a partitioning-and-rebalancing story. The ring, virtual nodes, and especially the strategy 1 → 2 → 3 evolution in §6.2 are the main plot — that evolution ends at "fixed equal partitions, moved as whole files", which is the same destination Redis Cluster hard-coded as 16384 slots (see reading-redis-cluster.md) and the destination the capstone's M36 milestone copies. Vector clocks, quorums, and hinted handoff are supporting cast here; each gets one step, with topic 21 as the deeper home for replication. +For this topic, read the paper as a partitioning-and-rebalancing story. The ring, virtual nodes, and especially the strategy 1 → 2 → 3 evolution in §6.2 are the main plot — that evolution ends at "fixed equal partitions, moved as whole files", which is the same destination Redis Cluster hard-coded as 16384 slots (see [reading-redis-cluster.md](reading-redis-cluster.md)) and the destination the capstone's M36 milestone copies. Vector clocks, quorums, and hinted handoff are supporting cast here; each gets one step, with topic 21 as the deeper home for replication. + +Every section, figure and table cited below is from the SOSP 2007 paper as text-extracted this session; the numbers were checked against it rather than repeated from memory. ## The problem in one sentence @@ -14,9 +16,37 @@ Dynamo's answer has two layers that the paper initially conflated and later sepa ### Step 1 — Hash mod N, and why "keys moved" is the metric -The naive scheme — `node = hash(key) mod N` — balances load perfectly and routes in zero hops. Its fatal flaw is rebalancing cost: changing N remaps almost everything. Growing from N to N+1 nodes moves N/(N+1) of all keys; the experiments' lane 1 measured exactly that — 80.0% of keys moved going from 4 to 5 nodes, versus consistent hashing's 1/(N+1). +> **In:** nothing yet — this step fixes the metric (keys moved on growth) that judges every scheme in the paper. +> **Out:** the exact movement fraction `N/(N+1)`, worked on real N, and the reason it, not balance, is what killed the naive scheme. Step 2 spends the rest of the paper beating it. + +A **shard** (or partition) is the subset of the key space one node is responsible for. The naive scheme assigns each key to a node with `node = hash(key) mod N`, where `N` is the node count: it balances load perfectly and routes in **zero hops** (the client computes the owner directly, no lookup). Its fatal flaw is **rebalancing cost** — the data that must physically move when the cluster changes size. Change `N` and almost every key is remapped. + +Work the fraction exactly, because it is this topic's headline number ([FINDINGS.md](../../FINDINGS.md) row 36) and the reason the rest of the field exists. Growing from `N` to `N+1` nodes, a key `k` *stays put* only if it hashes to the same node under both moduli: + +``` +key k stays ⇔ (k mod N) == (k mod (N+1)) + +N and N+1 are coprime, so by the Chinese Remainder Theorem the pair +(k mod N, k mod N+1) is fixed by k mod N(N+1). Equality forces both +residues to a common value r, and a residue mod N lives in 0..N−1, +so r ∈ {0, 1, …, N−1}: exactly N of the N(N+1) residues keep the key. + +fraction that STAY = N / (N(N+1)) = 1/(N+1) +fraction that MOVE = 1 − 1/(N+1) = N/(N+1) +``` + +Run it on the sizes the bench measures (lane 1), and against consistent hashing's mirror-image cost of `1/(N+1)` (only the arcs the new node claims change hands — Step 2): + +``` +N → N+1 mod-N moves = N/(N+1) ring moves = 1/(N+1) + 4 → 5 4/5 = 80.0% 1/5 = 20.0% + 8 → 9 8/9 = 88.9% 1/9 = 11.1% +16 → 17 16/17 = 94.117…% ≈ 94.1% 1/17 = 5.882…% ≈ 5.9% +``` -That single number defines the design space. Every key moved is a disk read, a network transfer, a cache invalidation, and — per topic 35 — load added to a cluster that is probably being expanded *because* it is already overloaded. Dynamo's §2.3 design principles make the requirement explicit: incremental scalability (add one node at a time with minimal impact), symmetry (no distinguished nodes), decentralization, and heterogeneity (work proportional to node capacity). Hash mod N fails the first principle outright. +Two facts fall out of the arithmetic. First, the two schemes are exact mirror images: mod-N moves `N/(N+1)`, the ring moves `1/(N+1)`. Second, mod-N gets *worse* as you grow — `N/(N+1) → 1` — so the scheme is most wasteful exactly when a large, busy cluster reshards. The experiments measured 80.0% at 4→5 and 94.1% at 16→17, matching the closed form to the digit. + +Why it matters: every key moved is a disk read, a network transfer, a cache invalidation, and — per topic 35 — load added to a cluster that is probably being expanded *because* it is already overloaded. Dynamo's §2.3 design principles make the requirement explicit: incremental scalability (add one node with minimal impact), symmetry (no distinguished nodes), decentralization, and heterogeneity (work proportional to node capacity). Hash mod N fails the first outright. | Scheme | Keys moved growing N→N+1 | Balance | Heterogeneity-aware | |---|---|---|---| @@ -28,7 +58,10 @@ Keep Table 1 of the paper at hand while reading — it maps each problem to its ### Step 2 — The consistent-hashing ring and virtual nodes -Dynamo (§4.2) treats the output range of the hash function as a fixed circular ring. Each node picks a random position on the ring; a key (MD5-hashed to a 128-bit identifier, §4.1) is stored at the first node clockwise from its hash. Now a node's arrival or departure affects only its immediate neighbors — the 1/(N+1) movement cost from Karger et al. (STOC'97). +> **In:** the movement metric from Step 1, and the `1/(N+1)` target it set. +> **Out:** the ring that hits that target, plus the **token/virtual-node** vocabulary Steps 6–7 later split apart. This is the data structure everything else sits on. + +**Consistent hashing** (Karger et al., STOC'97) treats the output range of the hash function as a fixed circular ring. Dynamo (§4.2) hashes each key to a 128-bit identifier with MD5 (§4.1) and stores it at the first node encountered walking **clockwise** from that point. Each node sits at a random ring position. A node's arrival or departure now affects only its immediate neighbor's arc — the `1/(N+1)` movement Step 1 wanted. ``` hash ring (128-bit MD5 space, wraps around) @@ -44,21 +77,27 @@ Dynamo (§4.2) treats the output range of the hash function as a fixed circular remove B → only keys in (A, B] move (to C); A, C, D keep everything else ``` -Two problems remain with one-token-per-node: random positions produce arcs of very different sizes (non-uniform load), and the scheme ignores heterogeneity — a big machine and a small one get statistically identical arcs. The fix is **virtual nodes**: each physical node claims multiple ring positions ("tokens"). Benefits per §4.2: when a node dies, its load disperses evenly across all survivors (each survivor inherits a few small arcs, not one big one); a new or rejoining node accepts roughly equal load from every existing node; and the token count per node can be set proportional to capacity. +Two problems remain with one position per node. Random positions produce arcs of very different sizes, so load is non-uniform; and a big machine and a small one get statistically identical arcs, ignoring heterogeneity. The fix is **virtual nodes**: each physical node claims *multiple* ring positions. Each such position is a **token** — a single point a node owns on the ring. Benefits per §4.2: when a node dies its load disperses evenly across all survivors (each inherits a few small arcs, not one big one); a joining node accepts roughly equal load from every existing node; and a node's token count can be set proportional to its capacity. -Note what a token is at this stage, because it changes later: in the original design a token is *both* a partition boundary and an ownership claim. Holding that dual role in mind now makes the strategy 1 post-mortem (Step 6) read as inevitable rather than surprising. +Note what a token *is* at this stage, because it changes later: in the original design a token is **both** a partition boundary (where one range ends and the next begins) **and** an ownership claim (who holds that range). Holding that dual role in mind now makes the strategy 1 post-mortem (Step 6) read as inevitable rather than surprising. ### Step 3 — Preference lists and N/R/W quorums (supporting cast) -Each key is replicated at N hosts (§4.3): the coordinator stores it locally and replicates to the N−1 clockwise successors. The resulting node list is the key's **preference list** — built by *skipping* ring positions so that it contains N distinct *physical* nodes (two vnodes of the same machine must not count as two replicas), and holding more than N entries to cover failures. The interface above all this is minimal (§4.1): get(key) and put(key, context, object), where the context carries version metadata opaque to the caller. +> **In:** the ring and tokens from Step 2. +> **Out:** the **preference list** (which N nodes replicate a key) and the R/W numbers that read/write it. Step 4 makes this fault-tolerant; Step 5 repairs it after the fact. + +Each key is replicated at **N** hosts (§4.3): the **coordinator** (the node handling the request) stores it locally and forwards to the N−1 clockwise successors. That ordered node list is the key's **preference list**. It is built by *skipping* ring positions so that it names N distinct *physical* nodes — two virtual nodes of the same machine must not count as two replicas (§4.2, "distinct physical nodes") — and it holds more than N entries so failures can be walked past. The interface above all this is minimal (§4.1): `get(key)` and `put(key, context, object)`, where the opaque `context` carries version metadata. -Reads and writes use quorums (§4.5): R nodes must participate in a read, W in a write; R + W > N gives quorum-like behavior. Operation latency is dictated by the *slowest* of the R (or W) replicas contacted, so both are usually set below N. The common production configuration (§6) is (N, R, W) = (3, 2, 2), chosen to balance performance, durability, consistency, and availability. A write coordinator writes locally and sends to the N highest-ranked reachable nodes, succeeding after W−1 responses. +Reads and writes use **quorums** (§4.5) — the rule that an operation must touch enough replicas to overlap with other operations. **R** is how many replicas must answer a read, **W** how many must acknowledge a write; setting **R + W > N** guarantees any read's replica set overlaps any write's, giving quorum-like consistency. Operation latency is set by the *slowest* of the R (or W) replicas contacted, so both are usually kept below N. The common production configuration (§6) is **(N, R, W) = (3, 2, 2)**, chosen to balance performance, durability, consistency, and availability; a write coordinator writes locally and sends to the N highest-ranked reachable nodes, succeeding after W−1 further responses. -Concurrent versions are tracked with vector clocks (§4.4) — lists of (node, counter) pairs per object version. When one clock dominates another, reconciliation is syntactic and automatic; when clocks are concurrent, the application merges semantically (the shopping cart takes the union of divergent carts — "add to cart" must never be rejected, and the known cost is that deleted items can occasionally resurface after a merge). Clocks are truncated at a threshold (say 10 pairs) by evicting the pair with the oldest timestamp; the paper reports this never caused a production issue. Depth on quorums and versioning lives in topic 21. +Concurrent versions are tracked with **vector clocks** (§4.4) — a list of `(node, counter)` pairs stamped on each object version, recording which nodes have updated it. When one clock dominates another (every counter ≥), reconciliation is syntactic and automatic; when two clocks are *concurrent* (neither dominates), the application merges semantically. The shopping cart takes the union of divergent carts, because "add to cart" must never be rejected — the known cost is that a deleted item can occasionally resurface after a merge. Clocks are truncated once they exceed a **threshold** (say 10 pairs, §4.4) by evicting the pair with the oldest timestamp; the paper reports this never caused a production issue. Depth on quorums and versioning lives in topic 21. ### Step 4 — Sloppy quorum and hinted handoff (supporting cast) -A strict quorum over the key's home replicas would be unavailable whenever those specific nodes fail — unacceptable for a store where "add to cart" must never be rejected. Dynamo's sloppy quorum (§4.6) instead operates on the first N *healthy* nodes found walking the ring. If home node A is down, the write lands on the next node D carrying a **hint** naming A; D stores hinted replicas in a separate local database, scans periodically, delivers them back to A on recovery, and deletes its copy. +> **In:** the preference list and R/W quorum from Step 3. +> **Out:** the rule that a *temporary* node failure never moves data — only membership changes do. That invariant is what Steps 6–7 are protecting. + +A strict quorum over a key's home replicas would be unavailable whenever those specific nodes are down — unacceptable for a store where "add to cart" must never be rejected. A **sloppy quorum** (§4.6) instead takes the first N *healthy* nodes found walking the ring, which need not be the key's usual home. If home node A is down, the write lands on the next healthy node D carrying a **hint** — a tag on the replica naming its true owner A. This is **hinted handoff**: D stores the hinted replica in a separate local database, scans it periodically, delivers it back to A when A recovers, and then deletes its copy. ```mermaid sequenceDiagram @@ -72,13 +111,16 @@ sequenceDiagram D->>D: delete hinted copy ``` -The partitioning-relevant point: hints keep the durability count at N without changing the ring, so *temporary* failures never trigger rebalancing. Only membership changes move data. +The partitioning-relevant point: hints keep the durability count at N without changing the ring, so *temporary* failures never trigger rebalancing. Only membership changes move data — the whole cost model of Step 1 applies only to real joins and leaves, not to transient outages. -Two knobs deserve a note. Setting W=1 gives maximum write availability — a write succeeds as long as any single node in the walk is up. And because preference lists span multiple data centers, the same walk-the-ring rule that handles a dead node also handles a dead data center, with no separate mechanism. +Two knobs deserve a note. Setting **W = 1** gives maximum write availability — a write succeeds as long as any single node in the walk is up (at the cost of durability). And because preference lists are constructed to span multiple data centers, the same walk-the-ring rule that survives a dead node also survives a dead data center, with no separate mechanism. ### Step 5 — Merkle anti-entropy, and its coupling to partitioning -Hinted replicas can be lost before delivery, so Dynamo also runs anti-entropy (§4.7): each node keeps one Merkle tree **per key range** (per virtual node), with leaves hashing individual keys' values. Two replicas exchange the root for a shared range and descend only into subtrees whose hashes differ — minimizing both bytes transferred and disk reads. +> **In:** the replicas from Steps 3–4, which can silently diverge (a hint lost before delivery, a missed write). +> **Out:** the repair mechanism — and the observation that it is *keyed to ranges*, which is the first of three threads pushing toward fixed partitions (Step 7). + +Hinted replicas can be lost before delivery, so Dynamo also runs **anti-entropy** (§4.7) — a background process that compares two replicas and repairs divergence. It does so with a **Merkle tree**: a tree of hashes whose leaves hash individual keys' values and whose internal nodes hash their children, so two replicas can find their differences by exchanging only a few hashes. Each node keeps one Merkle tree **per key range** (per virtual node). Two replicas exchange the root for a shared range and descend only into subtrees whose hashes differ — minimizing both bytes transferred and disk reads. ``` range root compare roots: differ → descend @@ -90,25 +132,31 @@ Hinted replicas can be lost before delivery, so Dynamo also runs anti-entropy ( mismatch ``` -The stated disadvantage matters for this topic: a node join or leave *changes the key ranges*, invalidating and forcing recalculation of Merkle trees on many nodes. This is the first thread of the argument that ranges should be fixed — strategy 3 (Step 7) resolves it. Membership itself is gossip-based (§4.8–4.9): every node reconciles membership history with a random peer each second, seed nodes prevent logical ring partitions, and failure detection is purely local (A considers B failed if B stops answering A). +The stated disadvantage matters for this topic: a node join or leave *changes the key ranges*, which invalidates and forces recalculation of the Merkle trees on many nodes. This is the first thread of the argument that ranges should be fixed — strategy 3 (Step 7) resolves it. Membership itself is **gossip-based** (§4.8–4.9): every node reconciles its membership history with a random peer each second, seed nodes prevent a logically partitioned ring, and failure detection is purely local (A considers B failed if B stops answering A, no distributed agreement). ### Step 6 — Strategy 1: tokens as boundaries, and what broke in production -Strategy 1 is the original design: T random tokens per node, and the partition boundaries *are* the token values. Ranges therefore vary in size and change whenever any node joins or leaves. §6.2 lists what this did in production: +> **In:** the original ring of Step 2, where a token is both boundary and ownership. +> **Out:** three named production failures and their single root cause — "partitioning and placement are intertwined". Step 7 is the fix for exactly this. + +**Strategy 1** is the original design (§6.2, and §4.2): T random tokens per node, and the partition boundaries *are* the token values. Ranges therefore vary in size and change whenever any node joins or leaves. §6.2 lists what this did in production: | Problem | Mechanism | |---|---| -| Bootstrapping a node took "almost a day" during the busy holiday season | A new node "steals" key ranges; donors must *scan their local persistence store* to extract the right keys — a heavyweight background task competing with live traffic | +| Bootstrapping a node took "almost a day" during the busy holiday season | A new node "steals" key ranges; donors must *scan their local persistence store* to extract the right keys — a resource-intensive background task run at lowest priority so it does not hurt live traffic, which makes it slow | | Merkle-tree recalculation storms | Join/leave changes many ranges (Step 5), so many nodes rebuild trees | -| No whole-keyspace snapshot/archival | Ranges are random per node; there is no clean unit to archive | +| No whole-keyspace snapshot/archival | Ranges are random per node; there is no clean unit to archive, so archival must retrieve keys from every node separately | -The root diagnosis, in the paper's own framing: strategy 1 **intertwines data partitioning and data placement**. The tokens simultaneously decide where the range boundaries fall and which node owns them, so you cannot add capacity (placement) without redrawing boundaries (partitioning). +The root diagnosis, in the paper's own framing: strategy 1 **intertwines data partitioning and data placement**. The tokens simultaneously decide where the range boundaries fall (partitioning) and which node owns them (placement), so you cannot add capacity without redrawing boundaries. §6.2: "in this scenario, it is not possible to add nodes without affecting data partitioning." ### Step 7 — Strategies 2 and 3: decouple partitioning from placement -Strategy 2 (the interim step) divides the hash space into Q *equal-size* partitions, with Q much larger than N and Q much larger than S·T (S nodes, T tokens each). Nodes still hold T random tokens, but tokens now decide only *placement*: a partition lives on the first N distinct nodes clockwise from its end. Boundaries never move. This achieves the decoupling — but Figure 8 shows it has the *worst* load-balancing efficiency (mean load / max load) of the three at the evaluated setup (S=30 nodes, N=3, equal metadata budgets). +> **In:** the intertwining diagnosis from Step 6. +> **Out:** the final design — Q fixed equal partitions, Q/S tokens per node, partition-as-file — and Figure 8's verdict that it balances best. This is the destination Redis Cluster and M36 copy. -Strategy 3 (the final design) keeps the Q equal partitions and drops random tokens: each node holds exactly Q/S tokens. When a node leaves, its tokens are randomly redistributed to the survivors preserving the Q/S invariant; a joining node steals tokens likewise. Node addition itself (§4.9) is confirmation-based in every strategy: the nodes that lose ranges to the newcomer offer the keys and transfer them with a confirmation round, which avoids duplicate transfers. +**Strategy 2** (the interim step) divides the hash space into **Q** equal-size partitions, with Q ≫ N and Q ≫ S·T (S nodes, T tokens each). Nodes still hold T random tokens, but tokens now decide only *placement*: a partition lives on the first N distinct nodes clockwise from the partition's end. Boundaries never move. This achieves the decoupling — but Figure 8 shows strategy 2 has the **worst** load-balancing efficiency of the three at the evaluated setup. + +**Strategy 3** (the final design) keeps the Q equal partitions and drops random tokens: each node holds exactly **Q/S** tokens. When a node leaves, its tokens are redistributed to survivors preserving the Q/S invariant; a joining node steals tokens likewise. Node addition itself (§4.9) is confirmation-based in every strategy: the nodes that lose ranges to the newcomer offer the keys and transfer them with a confirmation round, avoiding duplicate transfers. ``` strategy 1: boundaries = random tokens strategy 3: Q fixed equal partitions @@ -118,7 +166,7 @@ strategy 1: boundaries = random tokens strategy 3: Q fixed equal partitio each partition = one file ``` -The payoffs listed in §6.2: best load-balancing efficiency of the three; membership metadata reduced by **three orders of magnitude** versus strategy 1; faster bootstrapping and recovery because partitions are fixed ranges stored as *separate files* that transfer as a unit (no scanning, no random I/O); and trivial archival (copy the partition files). The one disadvantage: join/leave now needs coordination to preserve the token invariant — the symmetric, coordination-free ring is gone. +The **load-balancing efficiency** the paper plots in Figure 8 is defined precisely (§6.2): the ratio of the *average* number of requests a node serves to the *maximum* served by the hottest node — 1.0 is perfect. Strategy 3's payoffs, all listed in §6.2: best efficiency of the three; **membership metadata reduced by three orders of magnitude** versus strategy 1 (each node stores partition-to-node assignments, not every node's token positions); faster bootstrap and recovery because partitions are fixed ranges stored as *separate files* that transfer as a unit (no scanning, no random I/O); and trivial archival (copy the partition files). The one disadvantage: join/leave now needs coordination to preserve the token invariant — the symmetric, coordination-free ring is gone. | | Strategy 1 | Strategy 2 | Strategy 3 | |---|---|---|---| @@ -129,17 +177,20 @@ The payoffs listed in §6.2: best load-balancing efficiency of the three; member | Membership metadata | largest | between | ~1000× smaller than strategy 1 | | Bootstrap unit | scanned key ranges | fixed ranges | fixed ranges as whole files | -Redis Cluster is strategy 3 with the last step taken: Q fixed at 16384 and slot assignment made fully explicit and operator-visible (see reading-redis-cluster.md); the capstone's M36 milestone does the same. +Redis Cluster is strategy 3 with the last step taken: Q fixed at 16384 and slot assignment made fully explicit and operator-visible (see [reading-redis-cluster.md](reading-redis-cluster.md)); the capstone's M36 milestone does the same. ### Step 8 — Measuring imbalance: the 15% rule and the load paradox -§6.2 defines a node as "in balance" if its request load is within 15% of the fleet average, and tracks the *imbalance ratio* — the fraction of nodes out of balance. The counterintuitive result: imbalance is about 20% during *low* load and drops close to 10% at *high* load. Under high load, many popular keys are active and the hash spreads them evenly; at low load (around 1/8th of peak), only a few hot keys are in play, so their random placement dominates. +> **In:** any of the strategies from Steps 6–7, now running under real traffic. +> **Out:** a cheap, comparable *balance metric* (the imbalance ratio) and the counter-intuitive rule about *when* to measure it. + +§6.2 defines a node as **in balance** if its request load is within **15%** of the fleet average, and tracks the **imbalance ratio** — the fraction of nodes out of balance, sampled over a 24-hour trace in 30-minute windows. The counter-intuitive result: the imbalance ratio is about **20% during low load** and drops close to **10% at high load**. Under high load many popular keys are active at once and the hash spreads them evenly; at low load — around 1/8th of peak — only a few hot keys are in play, so their random placement dominates and the fleet looks lumpy. -The same section's latency numbers explain why balance matters at the tail (§6.1): 99.9th-percentile latencies ran around 200 ms — an order of magnitude above the average — and write buffering (an in-memory object buffer drained by a writer thread) cut the 99.9th percentile by a factor of 5 at peak. The "durable write" variant recovers durability cheaply: the coordinator picks one of the N replicas to perform a durable write, since W responses are needed before acknowledging anyway. +The same section's latency numbers explain why balance matters at the tail (§6.1): 99.9th-percentile latencies ran around **200 ms** — an order of magnitude above the average — and **write buffering** (an in-memory object buffer drained by a writer thread) cut the 99.9th percentile by a **factor of 5** at peak, for a buffer of only a thousand objects (Figure 5). The "durable write" variant recovers durability cheaply: the coordinator picks one of the N replicas to perform a synchronous durable write, since W responses are needed before acknowledging anyway. Two takeaways for any sharded system. First, define balance as a measurable ratio against the mean and monitor the *fraction of violating nodes*, not just a max. Second, benchmark balance at low traffic, not just peak — that is where placement randomness shows. -For calibration on how rare divergence was in this design (§6.3), over a 24-hour trace: +For calibration on how rare divergence was in this design (§6.3), over a 24-hour trace of the shopping-cart service: | Versions returned | Fraction of requests | |---|---| @@ -148,18 +199,21 @@ For calibration on how rare divergence was in this design (§6.3), over a 24-hou | 3 | 0.00047% | | 4 | 0.00009% | -Divergence was driven by concurrent writers — busy robots — not by failures. +Divergence was driven by concurrent writers — busy robots (automated clients) — not by failures. ### Step 9 — What a graph engine should copy -For the Rust graph-engine capstone, the durable lessons are almost all from Steps 6–8: +> **In:** the durable lessons of Steps 6–8. +> **Out:** the specific decisions that survive into the M36 capstone, and the one that does not (hash partitioning itself). + +For the Rust graph-engine capstone, the transferable lessons are almost all from Steps 6–8: - **Fix Q up front.** M36 uses redis-style 16384 slots, so partitioning is decided once and only placement ever changes — strategy 3's core move, taken to its logical end. - **Make the partition the unit of everything.** Transfer, Merkle tree, archival: one file (or file set) per partition, moved as a unit, so a donor never scans its store during rebalancing. - **Treat rebalancing traffic as load to be governed.** A partition transfer competes with live queries for disk and network; topic 35's admission-control lens applies directly — strategy 1's day-long bootstrap happened *during the busy holiday season*. - **Measure balance the Dynamo way.** The 15%-of-mean rule and the imbalance ratio are cheap to compute and directly comparable across load levels. -What a graph store cannot copy blindly is hash partitioning itself — hashing vertex IDs destroys locality for traversals, so the slot-assignment layer (strategy 3's placement freedom) matters even more: it is the knob that lets you co-locate related partitions later without re-hashing. Client-driven coordination (§6.4 — instead of a per-request state machine on a server picked by a load balancer, the client library polls a random node for membership every so often and routes directly) also transfers to a smart-client graph protocol, removing the extra hop from every query. +What a graph store cannot copy blindly is hash partitioning itself — hashing vertex IDs destroys locality for traversals, so the slot-assignment layer (strategy 3's placement freedom) matters even more: it is the knob that lets you co-locate related partitions later without re-hashing. Client-driven coordination (§6.4 — the client library polls a random node for membership every 10 seconds and routes directly, instead of a per-request state machine on a load-balancer-chosen server) also transfers to a smart-client graph protocol, removing the extra hop from every query. ## How to read the paper (with the concepts in hand) @@ -180,21 +234,66 @@ Read §6.2 twice: once for the mechanics of each strategy, once asking "which pr ## Questions to answer in notes.md 1. Strategy 1 caused three concrete production problems (day-long bootstrap, Merkle recalculation, no archival). Trace each one back to the single root cause the paper names — what exactly does "intertwining partitioning and placement" mean mechanically? -2. In strategy 2, what do the two conditions "Q much larger than N" and "Q much larger than S·T" each buy you? What goes wrong if either fails? +2. In strategy 2, what do the two conditions "Q ≫ N" and "Q ≫ S·T" each buy you? What goes wrong if either fails? 3. Why does a preference list skip ring positions to guarantee N distinct physical nodes, and what failure would occur if it naively took the next N vnodes? 4. Why is the imbalance ratio higher at low load (~20%) than at high load (~10%)? What does this imply about when to measure balance in your own system? 5. Strategy 3 wins on efficiency, metadata size, and transfer speed but loses the coordination-free join/leave of strategy 1. What coordination does it now require, and how does Redis Cluster's fixed-16384-slot design answer the same question? ## Done when +Answer each before unfolding it. + - [ ] You can explain, with the lane-1 numbers (80.0% vs 1/(N+1) at 4→5 nodes), why movement cost — not balance — killed hash mod N. + +
Answer + + `node = hash(key) mod N` balances load perfectly and routes in zero hops, so balance was never its problem. Its problem is that changing N changes the *function*: a key stays only if `k mod N == k mod (N+1)`, which by CRT holds for exactly N of every N(N+1) keys, i.e. a fraction `1/(N+1)` stays and `N/(N+1)` moves. At 4→5 that is 80.0% of all keys moving (the bench measured exactly this), against consistent hashing's `1/(N+1) = 20.0%`. + + The fraction gets *worse* as the cluster grows — `N/(N+1) → 1`, 94.1% at 16→17 — so the scheme is most wasteful precisely when a large cluster reshards, which per Dynamo's §2.3 "incremental scalability" principle and topic 35's overload lens is the worst possible time to move data. Balance is free; movement is the cost that made it unusable. + +
+ - [ ] You can state the three production failures of strategy 1 and derive each from "tokens are boundaries". + +
Answer + + In strategy 1 (§6.2) a node's T random tokens *are* the partition boundaries, so any join or leave redraws ranges. The three failures all follow: (1) a new node must *scan the donors' persistence stores* to extract the keys of its new, arbitrary ranges — a lowest-priority background scan that took "almost a day" in the holiday season; (2) because many ranges shift, many nodes must *recompute their per-range Merkle trees* (Step 5); (3) there is *no clean archival unit* because ranges are random per node, so a full snapshot means retrieving keys from every node separately. + + The single root cause the paper names is that data partitioning and data placement are *intertwined*: the same tokens decide both where boundaries fall and who owns them, so capacity (placement) cannot change without redrawing boundaries (partitioning). + +
+ - [ ] You can describe strategy 3 precisely (Q equal partitions, Q/S tokens per node, partitions as files) and name its one disadvantage. + +
Answer + + Strategy 3 (§6.2) fixes the hash space into **Q equal-size partitions** and gives each of the S nodes exactly **Q/S tokens**; tokens now decide only placement, boundaries never move. A leaving node's tokens are redistributed to survivors preserving the Q/S invariant; a joiner steals them likewise. Because each partition is a fixed range it is stored as a *separate file* that transfers as a unit — no donor scan — which makes bootstrap, recovery and archival cheap, and shrinks membership metadata by three orders of magnitude versus strategy 1. Figure 8 shows it has the best load-balancing efficiency (average/max requests per node) of the three. + + Its one disadvantage: changing node membership now *requires coordination* to preserve the Q/S assignment invariant — the symmetric, coordination-free join/leave of the original ring is gone. + +
+ - [ ] You can define the 15% imbalance ratio and explain the low-load/high-load paradox. + +
Answer + + A node is "in balance" (§6.2) if its request load is within **15%** of the fleet average; the **imbalance ratio** is the fraction of nodes out of balance, measured over 24 hours in 30-minute windows. The paradox: the ratio is ~20% at low load and drops to ~10% at high load — imbalance *falls* as traffic rises. + + The mechanism is that hashing spreads *many* keys well but *few* keys poorly. At high load a large set of popular keys is active simultaneously, and their uniform hashing evens the load out; at low load (about 1/8th of peak) only a handful of hot keys are in play, and the randomness of where those few keys landed dominates the per-node totals. The practical lesson: measure balance at low traffic, where placement randomness is exposed, not only at peak. + +
+ - [ ] Questions 1–5 are answered in notes.md. +
Answer + + The five questions target the paper's load story, not its trivia: the mechanical meaning of "intertwining" (Q1, Step 6); what each of strategy 2's two `Q ≫ …` conditions buys — Q ≫ N so every node holds several partitions for balance, Q ≫ S·T so tokens don't collide inside partitions (Q2, Step 7); why the preference list skips vnodes of the same physical node, else a "3-replica" key could sit on one machine and lose all copies to one failure (Q3, Step 3); the low/high-load imbalance paradox (Q4, Step 8); and the coordination strategy 3 trades for its balance, which Redis Cluster answers by fixing Q at 16384 and making slot ownership an explicit, gossiped table (Q5, Step 7). Each answer should cite the section it rests on. + +
+ ## References -- G. DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store", SOSP 2007. -- D. Karger et al., "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web", STOC 1997. -- [Topic 36 README](./README.md) — sharding, partitioning & rebalancing (lane 1: hash mod N vs consistent hashing, measured). +- G. DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store", SOSP 2007. Sections cited above: §2.2 (SLAs, 300 ms), §2.3 (design principles), §3 (zero-hop DHT), §4.1–4.9 (interface, MD5 ring, vnodes, preference lists, quorums, sloppy quorum, Merkle anti-entropy, gossip), §6–§6.4 ((3,2,2), 200 ms p99.9, factor-of-5 write buffering, strategies 1–3, Figure 8, 15% imbalance rule, divergent-version table, client-driven coordination). +- D. Karger et al., "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web", STOC 1997 — the `1/(N+1)` movement result. +- [Topic 36 README](README.md) — sharding, partitioning & rebalancing (lane 1: hash mod N vs consistent hashing, measured). +- [FINDINGS.md](../../FINDINGS.md) row 36 — "Growing 16 shards to 17 moves 94.1% of all keys (ideal: 5.9%)." diff --git a/topics/36-sharding/reading-powergraph.md b/topics/36-sharding/reading-powergraph.md index e3ff27a..134a4ad 100644 --- a/topics/36-sharding/reading-powergraph.md +++ b/topics/36-sharding/reading-powergraph.md @@ -12,6 +12,11 @@ For this learning path the paper matters twice: it is the theoretical backbone f design input for the M36 capstone, where a sharded Rust graph engine must decide what to do with high-degree vertices. Read it as a sharding paper first, computation second. +Every theorem, figure and table cited below is from the OSDI 2012 paper as text-extracted +this session; the numbers were checked against it, and where the earlier draft of this +guide had attached power-law exponents to the paper's real-world graphs that the paper +does not state, that has been corrected (Step 9). + ## The problem in one sentence **On power-law graphs, random (hashed) vertex placement cuts an expected 1 − 1/p @@ -26,20 +31,33 @@ per vertex, and on power-law graphs that number is small and greedily improvable ### Step 1 — Natural graphs are power-law, and α is the whole story -The degree distribution of a natural graph follows P(d) ∝ d^−α, typically with α ≈ 2 -(Twitter follower graph: in-degree α = 1.7, out-degree α = 2); lower α means a heavier -tail. The consequence that drives the entire paper: under α ≈ 2 a tiny fraction of -vertices is adjacent to a large fraction of edges — one percent of the vertices in the -Twitter web graph are adjacent to nearly half of the edges — so any per-vertex quantity -(work, storage, messages) is wildly unbalanced. +> **In:** nothing yet — this step fixes the one graph property (the exponent α) that +> every later step depends on. +> **Out:** the fact that a tiny vertex set touches most edges, which is what makes +> per-vertex placement (Step 4) fail and per-edge placement (Step 5) win. + +A **power-law degree distribution** means the probability a vertex has degree `d` falls +off as a power of `d`: `P(d) ∝ d^−α`, where the **exponent α** is a positive constant +controlling the skew (§3.1). Higher α ⇒ a lighter tail (most vertices low-degree, few +hubs); lower α ⇒ a heavier tail (denser graph, more and bigger hubs). Most natural graphs +sit around **α ≈ 2** (§3); Faloutsos et al. measured the Internet's inter-domain graph at +α ≈ 2.2. The paper's own illustrative curves (Fig 6) sweep α ∈ {1.65, 1.7, 1.8, 2.0}. + +The consequence that drives the entire paper: under α ≈ 2 a tiny fraction of vertices is +adjacent to a large fraction of edges. The paper's headline example (§3): **one percent +of the vertices in the Twitter follower graph are adjacent to nearly half of the edges.** +Figure 1 plots that graph's in- and out-degree distributions in log-log scale; both are +power-law, and the in-degree tail is the heavier of the two — a few celebrities have +millions of followers. So any *per-vertex* quantity (work, storage, messages) is wildly +unbalanced. ``` count of vertices with degree d (log-log) |* | * slope = −α - | ** α = 1.7 Twitter in-degree (heavy tail) - | *** α = 2.2 Hollywood (lighter tail) + | ** smaller α (≈1.7) → heavier tail, more/bigger hubs + | *** larger α (≈2.2) → lighter tail (e.g. Faloutsos's Internet) | ***** | ******** +---------------------------→ degree d @@ -49,55 +67,104 @@ count of vertices with degree d (log-log) ### Step 2 — Five ways high-degree vertices break Pregel and GraphLab -The paper lists five challenges, all downstream of degree skew: +> **In:** the degree skew from Step 1. +> **Out:** the five named challenges; keep challenge 2 (partitioning) in view, because +> Steps 4–7 are entirely about replacing the hashed placement it falls back to. + +The paper lists five challenges, all downstream of degree skew (§3): 1. **Work balance** — per-vertex work is degree-dependent, so vertex-balanced partitions are work-imbalanced. -2. **Partitioning** — good edge-cuts are unavailable in practice; both systems fall back - to hashed random vertex placement on natural graphs. +2. **Partitioning** — good edge-cuts are unavailable in practice on natural graphs, so + both systems fall back to hashed random vertex placement. 3. **Communication** — a high-degree vertex floods messages to millions of neighbors. 4. **Storage** — the full adjacency list of a high-degree vertex must fit on one machine. 5. **Computation** — a sequential per-vertex program over a huge neighborhood cannot itself be parallelized. -Keep the second in front of you: Steps 4-6 are entirely about what hashed random +Keep the second in front of you: Steps 4–6 are entirely about what hashed random placement costs and what replaces it. ### Step 3 — The GAS decomposition (just enough to motivate the partitioning) -PowerGraph splits a vertex program into three phases so work on one vertex can spread -over machines: +> **In:** a vertex program that would otherwise run on one machine (challenge 5). +> **Out:** the three-phase shape (Gather/Apply/Scatter) that lets one vertex's work +> split across machines — the precondition that makes vertex replication (Step 5) +> *semantically free*. -- **Gather**: for each adjacent edge compute g(D_u, D_(u,v), D_v) and combine the results - with a commutative, associative sum ⊕ into an accumulator Σ. -- **Apply**: D_u_new ← a(D_u, Σ). The apply function must be sub-linear (ideally - constant) in degree, and vertex data must be small. +**GAS** is PowerGraph's decomposition of a vertex program into three phases (§4.1) so +that work on one vertex can spread over machines: + +- **Gather**: for each adjacent edge compute `g(D_u, D_(u,v), D_v)` and combine the + results with a **commutative, associative sum** `⊕` — one whose result is + order-independent — into an accumulator `Σ`. +- **Apply**: `D_u_new ← a(D_u, Σ)`. The accumulator's size and the apply function's cost + "should be sub-linear and ideally constant in the degree" (§4.1), and vertex data must + be small. - **Scatter**: over adjacent edges, update edge data and activate neighbors. -Because ⊕ is commutative and associative, the gather can run in parallel over the -replicas of a vertex, each producing a partial accumulator summed at one designated -replica. This makes vertex replication *semantically free*: the program never sees that -its neighborhood was split. GAS exists so vertex-cuts can exist. +Because `⊕` is commutative and associative, the gather can run in parallel over the +replicas of a vertex, each producing a partial accumulator that is summed at one +designated replica. This makes vertex replication *semantically free*: the program never +sees that its neighborhood was split. GAS exists so vertex-cuts can exist. ### Step 4 — Edge-cuts and Theorem 5.1: random placement cuts almost everything -Edge-cut systems place vertices and pay (ghosts, storage, network) for every edge whose -endpoints land on different machines. Theorem 5.1: randomly placing |V| vertices on p -machines cuts an expected fraction 1 − 1/p of the edges — at p = 2 half the edges, at -large p nearly all. This is the hash-ring placement celebrated in reading-dynamo.md, -perfect for independent keys, indicted here because edges make keys dependent. -Ghost-based systems store and communicate along every cut edge, so the "boundary" is -effectively the whole graph — in GraphBLAS terms (topics 18/26), random placement makes -almost the whole distributed-SpMV matrix off-diagonal. +> **In:** the hashed vertex placement that challenge 2 (Step 2) falls back to. +> **Out:** Theorem 5.1's expected cut fraction `1 − 1/p`, worked on real p — the number +> that indicts edge-cuts and motivates the whole inversion of Step 5. + +An **edge-cut** places *vertices* on machines and pays (ghosts, storage, network) for +every edge whose endpoints land on different machines. A **ghost** is a local read-only +copy of a remote endpoint kept so a machine can evaluate a cut edge. **Theorem 5.1**: if +vertices are assigned to `p` machines uniformly at random, the expected fraction of edges +cut is + +``` +E[ |Edges Cut| / |E| ] = 1 − 1/p (paper Eq. 5.1) + +proof sketch: an edge is cut iff its two endpoints land on different machines, +which happens with probability 1 − 1/p. +``` + +Worked on the machine counts that matter here: + +``` +p = 2 : 1 − 1/2 = 0.500 → half the edges cross +p = 8 : 1 − 1/8 = 0.875 → 87.5% cross (lane 3's k = 8 random baseline) +p = 16 : 1 − 1/16 = 0.9375 → 93.75% +p → ∞ : → 1 → nearly every edge crosses +``` + +At `p = 8` that is exactly the `(k−1)/k = 7/8 = 87.5%` random-cut baseline lane 3 +measures on its generated graphs. Ghost-based systems store and communicate along *every* +cut edge, so the "boundary" is effectively the whole graph. This is the hash-ring +placement celebrated in [reading-dynamo.md](reading-dynamo.md) — perfect for independent +keys, indicted here because edges make keys dependent. In GraphBLAS terms (topics 18/26), +random placement makes almost the whole distributed-SpMV matrix off-diagonal. ### Step 5 — Vertex-cuts: place edges, replicate vertices, masters and mirrors -Invert the assignment. A balanced p-way vertex-cut assigns each **edge** to exactly one -machine. A vertex v then spans the set of machines A(v) that hold at least one of its -edges. The objective: minimize the average replication (1/|V|) Σ_v |A(v)| subject to the -balance constraint that no machine holds more than λ|E|/p edges. One replica of each -vertex is randomly nominated the **master** (canonical vertex data); the rest are -read-only **mirrors** that receive updates from the master after apply. +> **In:** Theorem 5.1's indictment of placing vertices (Step 4). +> **Out:** the inverted scheme — place *edges*, replicate the vertices they span — and +> the new cost metric (replicas per vertex) that Steps 6–7 minimize. + +Invert the assignment. A **balanced p-way vertex-cut** assigns each **edge** to exactly +one machine `A(e) ∈ {1,…,p}` (§5.1). A vertex `v` then spans the set of machines `A(v)` +that hold at least one of its edges; the copies of `v` on those machines are its +**replicas**. The objective and its balance constraint, verbatim from the paper (Eq. 5.3–5.4): + +``` +minimize (1/|V|) Σ_v |A(v)| ← average replication factor +subject to max_m |{ e ∈ E : A(e) = m }| < λ|E|/p + + |A(v)| = number of machines vertex v spans (its replica count) + λ ≥ 1 = the imbalance factor, a small constant capping the hottest machine's edges +``` + +One replica of each vertex is randomly nominated the **master** (it holds the canonical +vertex data); the rest are read-only **mirrors** that receive the updated value from the +master after the apply phase (§5.1). ``` edge-cut (place vertices) vertex-cut (place edges) @@ -113,92 +180,150 @@ edge-cut (place vertices) vertex-cut (place edges) ``` The cost model changes from "how many edges cross" to "how many replicas per vertex" — -communication becomes one accumulator and one update per mirror, not one per edge. +communication becomes one accumulator and one update per mirror, not one message per +cut edge. ### Step 6 — Theorems 5.2 and 5.3: replication is a function of the degree distribution -Theorem 5.2 gives the expected replication factor of *random* edge placement: +> **In:** the vertex-cut objective from Step 5. +> **Out:** Theorem 5.2's closed-form expected replication (worked on a hub and a leaf) +> and Theorem 5.3's existence guarantee — together the proof that the inversion pays off, +> and pays off *more* the more skewed the graph. + +**Theorem 5.2** gives the expected replication factor of *random* edge placement (Eq. 5.5): ``` E[ (1/|V|) Σ_v |A(v)| ] = (p/|V|) Σ_v ( 1 − (1 − 1/p)^D[v] ) + + D[v] = degree of vertex v + per-vertex term: E[|A(v)|] = p ( 1 − (1 − 1/p)^D[v] ) (Eq. 5.10) ``` -where D[v] is the degree of v. For power-law graphs this is determined entirely by α, -and the punchline is directional: the reduction in replication from vertex-cuts over -edge-cuts *increases as α decreases* — heavier skew, bigger win, an order-of-magnitude -gain in the paper's Figure 6. The pathology of Step 1 becomes the opportunity: a hub's -replication saturates at p while its edge-cut cost would have grown with degree. +Worked per vertex on `p = 8` machines, to see the point — a hub's replication is +*bounded by p* no matter its degree, while its edge-cut cost would have grown with degree: -Theorem 5.3 closes the argument: for any edge-cut with g ghosts there is a vertex-cut -along the same boundary with strictly fewer than g mirrors — a good vertex-cut exists -wherever a good edge-cut does; the converse is false. Percolation theory adds that -power-law graphs have good vertex-cuts to find. +``` +leaf, D = 1 : 8·(1 − (7/8)^1) = 8·0.125 = 1.00 replica + D = 2 : 8·(1 − (7/8)^2) = 8·0.2344 = 1.88 replicas + D = 4 : 8·(1 − (7/8)^4) = 8·0.4138 = 3.31 replicas +hub, D = 1000 : 8·(1 − (7/8)^1000) = 8·(1−~0) ≈ 8.00 replicas (saturates at p) +``` + +For a power-law graph the whole average is "determined entirely by the power-law constant +α" (§5, following Eq. 5.5), and the punchline is directional: the reduction in replication +from vertex-cuts over edge-cuts *increases as α decreases* — heavier skew, bigger win, up +to an order-of-magnitude improvement in Figure 6(b). The pathology of Step 1 becomes the +opportunity: a hub's replication saturates at `p` while its edge-cut cost grows with degree. + +**Theorem 5.3** closes the argument: for any edge-cut with `g` ghosts, any vertex-cut +along the same partition boundary has **strictly fewer than g mirrors** — a good +vertex-cut exists wherever a good edge-cut does; the converse is not claimed. Percolation +theory adds that power-law graphs have good vertex-cuts to find. ### Step 7 — Greedy streaming placement: four cases, two implementations -Even random edge placement beats edge-cuts, but a one-pass greedy heuristic does better: -place each edge (u, v) conditioned on the machine sets A(u), A(v) built so far. +> **In:** the random edge placement of Theorem 5.2 (Step 6), which already beats edge-cuts. +> **Out:** the one-pass greedy rule (four cases) that beats *it*, and the coordinated vs +> oblivious trade — the shape lane 3's partitioner and M36 both copy. + +Even random edge placement beats edge-cuts, but a **greedy** de-randomization does better +(§5.2): place each edge `(u, v)` on the machine that minimizes the *conditional* expected +replication, given the machine sets `A(u), A(v)` built so far. That reduces to four cases: ```mermaid flowchart TD E["edge (u, v) arrives"] --> C1{"A(u) ∩ A(v) non-empty?"} C1 -- yes --> P1["Case 1: place in the intersection"] C1 -- no --> C2{"both A(u), A(v) non-empty?"} - C2 -- yes --> P2["Case 2: machines of the vertex with\nmore unassigned edges remaining"] + C2 -- yes --> P2["Case 2: machines of the vertex with
more unassigned edges remaining"] C2 -- no --> C3{"exactly one assigned?"} C3 -- yes --> P3["Case 3: one of its machines"] C3 -- no --> P4["Case 4: least loaded machine"] ``` -The intuition: never create a new replica when an existing one can absorb the edge; when -forced to choose, spend the replica on the vertex likely to need it again (more -unassigned edges left). Two implementations trade cut quality against speed: -**coordinated** keeps A(v) in a distributed table (slower, better cuts); **oblivious** -runs the heuristic independently per machine with no communication (slightly worse -cuts). Compare lane 3 of the experiments: LDG-style streaming greedily places *vertices* -(edge-cut world); PowerGraph's greedy places *edges* (vertex-cut world) — same one-pass -shape, opposite variable. +The intuition: never create a new replica when an existing one can absorb the edge (Case +1); when forced to choose, spend the replica on the vertex likely to need it again — the +one with more unassigned edges left (Case 2). Two implementations trade cut quality +against speed (§5.2): **coordinated** keeps `A(v)` in a distributed table, periodically +synced (slower, better cuts); **oblivious** runs the heuristic independently per machine +with no communication, each keeping its own estimate of `A` (slightly worse cuts). +Figure 7(a) shows both beating random on every real graph, coordinated best. + +Compare lane 3 of the experiments: an LDG-style streaming partitioner greedily places +*vertices* (edge-cut world); PowerGraph's greedy places *edges* (vertex-cut world) — same +one-pass shape, opposite variable. ### Step 8 — Delta caching and sync/async execution (brief) -Two refinements ride on the abstraction. **Delta caching**: the accumulator Σ is cached -per vertex; a scatter returns a delta Δa atomically added to the neighbor's cached -accumulator, skipping redundant gathers — valid when the ⊕ sum forms an Abelian group -(commutative with an inverse; sums qualify, max does not). **Execution modes**: -synchronous runs deterministic bulk-synchronous supersteps; asynchronous executes -vertices as resources free up, with optional serializability via vertex locking. Both -pay the replication factor in network traffic every round. +> **In:** the GAS engine of Step 3 running over the vertex-cut of Steps 5–7. +> **Out:** two refinements that change the per-round cost but not the placement story; +> here so the paper's §4.2–4.3 read as an aside rather than a gap. + +Two refinements ride on the abstraction. **Delta caching** (§4.2): the accumulator `Σ` is +cached per vertex; a scatter returns a delta `Δa` atomically added to the neighbor's +cached accumulator, skipping redundant gathers. It is valid only when `⊕` forms an +**Abelian group** — commutative and associative *with an inverse* — so a change can be +subtracted out again: sums qualify (PageRank), set union does not (graph coloring), and +`max` does not either (no inverse). **Execution modes** (§4.3): synchronous runs +deterministic bulk-synchronous supersteps; asynchronous executes vertices as resources +free up, with optional serializability via vertex locking. Both pay the replication factor +in network traffic every round — which is why Steps 5–7 spend it carefully. ### Step 9 — What a graph database should copy +> **In:** the design decisions accumulated across Steps 5–7. +> **Out:** the specific choices M36 adopts, and the scale anchors from Table 1 — read +> exactly as the paper states them. + For the M36 capstone (sharding a Rust graph engine) the transferable decisions are: - Store edges with their source vertex, but treat *edge placement* as the primary sharding decision — a hub's edge list may be split. - Replicate high-degree vertices as master + mirrors; route writes to the master and propagate to mirrors after apply. -- Use a streaming greedy placer (the four cases) at ingest time; one pass, only A(v) +- Use a streaming greedy placer (the four cases) at ingest time; one pass, only `A(v)` bookkeeping, beats hashing without an offline partitioner like METIS. -- Measure replication factor, not edge-cut, as the quality metric — Theorem 5.2 gives - the random-placement baseline, computable from the degree sequence alone (exercise 5 - in the experiments does exactly this). -- Table 1 is a scale anchor: Twitter 41M vertices / 1.4B edges (α = 1.8), UK web - 132.8M / 5.5B (α = 1.9), LiveJournal 5.4M / 79M (α = 2.1). +- Measure **replication factor**, not edge-cut, as the quality metric — Theorem 5.2 gives + the random-placement baseline, computable from the degree sequence alone (exercise 5 in + the experiments does exactly this). + +Table 1 is the scale anchor, and it is two separate tables — do not merge them, as an +earlier draft of this guide did. Table 1(a) lists **real-world graphs by size only**; the +paper attaches **no per-graph α** to them: + +| Real-world graph (Table 1a) | \|V\| | \|E\| | +|---|---|---| +| Twitter | 41M | 1.4B | +| UK web | 132.8M | 5.5B | +| Amazon | 0.7M | 5.2M | +| LiveJournal | 5.4M | 79M | +| Hollywood | 2.2M | 229M | + +Table 1(b) is the *synthetic* generator: ten-million-vertex power-law graphs whose α is +the input and whose edge count is the output — "smaller α produces denser graphs": + +| Synthetic α (Table 1b) | # Edges (on 10M vertices) | +|---|---| +| 1.8 | 641,383,778 | +| 1.9 | 245,040,680 | +| 2.0 | 102,838,432 | +| 2.1 | 57,134,471 | +| 2.2 | 35,001,696 | ## How to read the paper (with the concepts in hand) -- **Sections 1-2 (intro, graph-parallel background)** — Steps 1-2. Get the α ≈ 2 claim +- **Sections 1–2 (intro, graph-parallel background)** — Steps 1–2. Get the α ≈ 2 claim and the five challenges; skim the Pregel/GraphLab recaps if you know them. -- **Section 3 (PowerGraph abstraction: GAS, delta caching, sync/async)** — Steps 3 and 8. +- **Section 3 (challenges)** — Step 1's "1% of vertices → half the edges" lives here (§3), + with Fig 1 (Twitter in/out degree). +- **Section 4 (PowerGraph abstraction: GAS, delta caching, sync/async)** — Steps 3 and 8. Read GAS carefully enough to see why gather parallelizes over replicas; skim the rest. -- **Section 4 (vertex programs as examples)** — optional; PageRank in GAS form is worth - 30 seconds. -- **Section 5 (distributed graph placement)** — the heart, Steps 4-7. Read fully: +- **Section 5 (distributed graph placement)** — the heart, Steps 4–7. Read fully: Theorem 5.1 (edge-cut indictment), the vertex-cut objective and master/mirror design, - Theorems 5.2-5.3, Figure 6 (replication gap vs α), and the greedy heuristic with + Theorems 5.2–5.3, Figure 6 (replication gap vs α), and the greedy heuristic with coordinated vs oblivious variants. -- **Sections 6-7 (implementation, evaluation)** — read for Table 1 (the five graphs and - their α values) and how replication factor tracks runtime; skim the rest. +- **Sections 6–7 (implementation, evaluation)** — read for Table 1 (the real graphs and + the synthetic α sweep) and how replication factor tracks runtime; skim the rest. - **Section 8 (related work)** — skim; note the streaming-partitioning lineage of Stanton & Kliot and FENNEL (references below). @@ -217,21 +342,110 @@ For the M36 capstone (sharding a Rust graph engine) the transferable decisions a ## Done when +Answer each before unfolding it. + - [ ] You can state Theorem 5.1 and reproduce the 1 − 1/p expectation from scratch, including the p = 2 sanity check. + +
Answer + + Theorem 5.1 (Eq. 5.1): placing vertices on `p` machines uniformly at random cuts an + expected `E[|Edges Cut|/|E|] = 1 − 1/p` of the edges. The derivation is one line: an + edge is cut iff its two endpoints land on different machines; for a fixed first + endpoint the second lands elsewhere with probability `1 − 1/p`, and expectation is + linear over edges. + + Sanity checks: `p = 2 → 1/2` (half the edges cross, obviously right for a coin flip per + endpoint); `p = 8 → 7/8 = 87.5%` (lane 3's random baseline); `p → ∞ → 1` (almost every + edge crosses). The same argument does *not* doom random *edge* placement, because there + each edge sits wholly on one machine by construction — nothing is "cut"; the cost + reappears only as vertex replicas, which Theorem 5.2 bounds. + +
+ - [ ] You can define a balanced p-way vertex-cut (objective + constraint) and explain masters vs mirrors without looking at the paper. + +
Answer + + A balanced p-way vertex-cut (§5.1, Eq. 5.3–5.4) assigns each edge to one machine + `A(e) ∈ {1,…,p}`; each vertex `v` then spans `A(v)`, the machines holding its edges. + It **minimizes** the average replication factor `(1/|V|) Σ_v |A(v)|` **subject to** no + machine holding more than `λ|E|/p` edges, where `λ ≥ 1` is a small imbalance constant. + + Of the `|A(v)|` replicas of a vertex, one is randomly nominated the **master** and holds + the canonical vertex data; the rest are read-only **mirrors**. Each round, mirrors send + partial gather accumulators to the master, the master runs apply, then pushes the new + value back to the mirrors — so communication per vertex is one accumulator and one + update per mirror, i.e. proportional to `|A(v)|`, which is exactly what the objective + minimizes. + +
+ - [ ] You can list the four greedy placement cases in order and say why Case 2 prefers the vertex with more unassigned edges. + +
Answer + + For edge `(u, v)` (§5.2): **Case 1** — if `A(u) ∩ A(v)` is non-empty, place in the + intersection (adds no replica); **Case 2** — if both are non-empty but disjoint, place + on a machine of the endpoint with more *unassigned* edges remaining; **Case 3** — if + only one endpoint is assigned, use one of its machines; **Case 4** — if neither is + assigned, use the least-loaded machine. + + Case 2 prefers the vertex with more unassigned edges because that vertex will force more + future placement decisions; keeping the *new* edge near it raises the chance those + future edges land on an already-used machine (a Case 1 hit), whereas the low-degree + endpoint is unlikely to be seen again, so pinning it costs little. It is a bet that + concentrating a busy vertex's edges now avoids replicas later. + +
+ - [ ] You have computed Theorem 5.2's replication factor on the experiments' generated degree sequence (lane 3, exercise 5) and compared it to the (k−1)/k baseline. + +
Answer + + Theorem 5.2 (Eq. 5.5): expected replication `= (p/|V|) Σ_v (1 − (1 − 1/p)^D[v])`, i.e. + sum the per-vertex `p(1 − (1 − 1/p)^D[v])` over the generated degree sequence and divide + by `|V|`. On `p = 8`, a degree-1 vertex costs 1.00 replica, degree-2 costs 1.88, and a + degree-1000 hub costs ≈ 8.00 — replication is *bounded by p* however large the degree. + + The comparison is against the edge-cut world's `(k−1)/k = 7/8 = 87.5%` random-cut + fraction from Theorem 5.1 at `k = 8`: the vertex-cut spends a handful of replicas per + vertex where the edge-cut cuts seven of every eight edges, and the gap widens as α falls + (Fig 6b). Reporting the vertex-cut's replication factor and the edge-cut's cut fraction + side by side is the point of exercise 5. + +
+ - [ ] You can name the one sharding decision from Step 9 you will adopt in the M36 capstone, and the metric to judge it. +
Answer + + Adopt **edge placement with a one-pass greedy placer** (the four cases) and + **master/mirror replication of high-degree vertices**: store each edge with its source + vertex, but let a hub's edge list split across shards, replicating the hub as master + + mirrors and routing writes to the master. This is the vertex-cut of Steps 5–7 applied to + the graph engine. + + Judge it by **replication factor** `(1/|V|) Σ_v |A(v)|`, not edge-cut, because that is + what GAS communication and storage are proportional to. Theorem 5.2 gives the + random-placement baseline from the degree sequence alone, and the greedy placer must + beat it — with the gap over a random *edge-cut* (Theorem 5.1) widening as the generated + graph's α falls. + +
+ ## References - Gonzalez, Low, Gu, Bickson, Guestrin — *PowerGraph: Distributed Graph-Parallel - Computation on Natural Graphs*, OSDI 2012. + Computation on Natural Graphs*, OSDI 2012. Cited above: §3 (α ≈ 2, "1% of vertices → + half the edges", Fig 1), §4.1 (GAS, sub-linear apply), §4.2–4.3 (delta caching/Abelian + group, sync/async), §5.1 (vertex-cut objective Eq. 5.3–5.4, masters/mirrors), Theorem + 5.1 (Eq. 5.1), Theorem 5.2 (Eq. 5.5, 5.10), Theorem 5.3, Fig 6 (replication vs α), + Fig 7 (random/oblivious/coordinated), Table 1 (real graphs a; synthetic α sweep b). - Stanton, Kliot — *Streaming Graph Partitioning for Large Distributed Graphs*, KDD 2012 (the LDG heuristic used in lane 3 — greedy *vertex* placement, the edge-cut counterpart of PowerGraph's greedy *edge* placement). diff --git a/topics/36-sharding/reading-redis-cluster.md b/topics/36-sharding/reading-redis-cluster.md index 593ec0f..29487f3 100644 --- a/topics/36-sharding/reading-redis-cluster.md +++ b/topics/36-sharding/reading-redis-cluster.md @@ -8,10 +8,11 @@ can tell a client where a key really lives. The entire live-migration story is b two error replies (`-MOVED` and `-ASK`), one client command (`ASKING`), and four admin verbs (`CLUSTER SETSLOT ... MIGRATING/IMPORTING/STABLE/NODE`). -This guide walks the C source in `~/repos/redis/src`. The interesting split: `cluster.h` / -`cluster.c` hold the generic slot math and redirect logic, while `cluster_legacy.h` / -`cluster_legacy.c` hold the concrete node state (per-slot migrating/importing pointers) and the -`SETSLOT` admin machinery. +This guide walks the C source in `~/repos/redis/src`, pinned at the SHA in the topic's pin +table. The interesting split: `cluster.h` / `cluster.c` hold the generic slot math and redirect +logic, while `cluster_legacy.h` / `cluster_legacy.c` hold the concrete node state (per-slot +migrating/importing pointers) and the `SETSLOT` admin machinery. Every `file:line` below was +checked against the pinned tree this session. ## The problem in one sentence @@ -19,17 +20,24 @@ This guide walks the C source in `~/repos/redis/src`. The interesting split: `cl must get a correct answer — served, or redirected with enough information to retry — without any central router and without ever blocking the keyspace.** -Mod-N hashing fails this test before migration even starts: the topic README's lane-1 numbers -show growing 4→5 shards remaps about 80% of keys. Redis Cluster's answer is to hash keys into a -fixed universe of 16384 slots and move *slot ownership*, one slot at a time, with a per-slot -state machine that keeps both nodes answering correctly mid-move. +Mod-N hashing fails this test before migration even starts: the topic's measured headline +(FINDINGS row 36) is that growing 16 shards to 17 moves 94.1% of all keys against an ideal of +5.9%, and lane 1 shows the smaller 4→5 case still remaps about 80%. Redis Cluster's answer is to +hash keys into a fixed universe of 16384 slots and move *slot ownership*, one slot at a time, +with a per-slot state machine that keeps both nodes answering correctly mid-move. ## The concepts, step by step ### Step 1 — Fixed slots decouple partitioning from placement -`cluster.h:23` defines the universe: `CLUSTER_SLOTS` is 2^14 = 16384 (`CLUSTER_SLOT_MASK_BITS` -is 14). A key maps to a slot with `crc16(key) & 0x3FFF` — masking to the low 14 bits. The +> **In:** the mod-N remap disaster from the topic headline (94.1% of keys move on a 16→17 grow). +> **Out:** the two-stage `key → slot → node` indirection, where the first arrow is frozen +> forever and only the second moves — the structural reason every later step is possible. + +A **slot** is one of a fixed number of hash buckets; a key is assigned to a slot by hashing, and +each slot is *owned* by exactly one node. `cluster.h:23` defines the universe: `CLUSTER_SLOTS` is +`1 << CLUSTER_SLOT_MASK_BITS = 2^14 = 16384` (`CLUSTER_SLOT_MASK_BITS` is 14, `cluster.h:22`). A +key maps to a slot with `crc16(key) & 0x3FFF` — masking a 16-bit CRC to its low 14 bits. The slot→node assignment is a separate, mutable table that every node gossips. ``` @@ -37,34 +45,49 @@ slot→node assignment is a separate, mutable table that every node gossips. ▲ fixed forever ▲ movable ``` -Contrast with mod-N: there, adding a shard changes the *function* and remaps most keys. Here the -function never changes; only rows of the slot map change. Rebalancing 4→5 nodes means handing -off roughly 16384/5 ≈ 3276 slots — about 20% of the data, the theoretical minimum — instead of 80%. +Contrast with mod-N: there, adding a shard changes the *function* `hash % N` and remaps most +keys (the headline's 94.1%). Here the function never changes; only rows of the slot map change. +Rebalancing 4→5 nodes means handing off roughly 16384/5 ≈ 3276 slots — about 20% of the data, +close to the theoretical minimum — instead of 80%. ### Step 2 — Hash tags: carving the hash input for co-location -`keyHashSlot()` (`cluster.h:59`) doesn't always hash the whole key. If the key contains a `{` -followed by a non-empty section closed by `}`, ONLY the substring between the first `{` and the -next `}` is hashed. An empty `{}` falls back to hashing the whole key. +> **In:** the `key → slot` hash from Step 1, which by default scatters related keys. +> **Out:** the hash-tag rule that lets a user *force* chosen keys into one slot — the +> precondition for the multi-key commands that Step 3's `-CROSSSLOT` check otherwise forbids. + +A **hash tag** is a substring of the key, delimited by `{` … `}`, that is hashed *instead of* the +whole key. `keyHashSlot()` (`cluster.h:59`) implements it: if the key contains a `{` followed by +a non-empty section closed by `}`, ONLY the substring between the first `{` and the next `}` is +hashed. An empty `{}` (nothing between the braces) falls back to hashing the whole key, and so +does a `{` with no following `}`. ``` - "user:{42}:cart" ──▶ hash("42") ─┐ - "user:{42}:profile" ──▶ hash("42") ─┤─▶ same slot, same node - "user:{42}:orders" ──▶ hash("42") ─┘ - "user:42:cart" ──▶ hash("user:42:cart") ──▶ some other slot + "user:{42}:cart" ──▶ crc16("42") & 0x3FFF ─┐ + "user:{42}:profile" ──▶ crc16("42") & 0x3FFF ─┤─▶ same slot, same node + "user:{42}:orders" ──▶ crc16("42") & 0x3FFF ─┘ + "user:42:cart" ──▶ crc16(whole key) & 0x3FFF ──▶ some other slot ``` This is the user-facing co-location tool: keys sharing a tag land in one slot, so multi-key -commands, MULTI/EXEC transactions, and Lua scripts over them are legal. There's also a -pattern-matching sibling, `patternHashSlot` (`cluster.c:36`), used when the "key" is a glob -pattern (e.g. pubsub patterns) — it must decide whether a pattern pins to a single slot at all. +commands, MULTI/EXEC transactions, and Lua scripts over them are legal (Step 3 rejects +cross-slot multi-key commands with `-CROSSSLOT`). There's also a pattern-matching sibling, +`patternHashSlot` (`cluster.c:35`), used when the "key" is a glob pattern (e.g. pubsub patterns): +it decides whether a pattern pins to a single slot at all by finding a `{`…`}` tag inside the +pattern. ### Step 3 — Request routing: getNodeByQuery decides serve / MOVED / ASK -`getNodeByQuery()` (`cluster.c:1191`) is the router. For each arriving command it extracts the -keys, computes their slot, and checks three things: do all keys share one slot (else -`-CROSSSLOT` error), does this node own the slot, and is the slot currently migrating or -importing. The outcome is either "serve locally" or an error code that drives a redirect. +> **In:** a command carrying one or more keys, each resolved to a slot by Steps 1–2. +> **Out:** exactly one of {serve locally, `-CROSSSLOT`, `-MOVED`, `-ASK`, `-TRYAGAIN`} — the +> decision that Steps 4–6 each pick up one branch of. + +`getNodeByQuery()` (`cluster.c:1191`) is the router. +For each arriving command it extracts the keys, computes their slot, and checks three things: do +all keys share one slot (else `-CROSSSLOT` error), does this node own the slot, and is the slot +currently migrating or importing. The outcome is either "serve locally" or an error code that +drives a redirect. **MOVED** means the slot's home has permanently changed; **ASK** means only +this one request should hop, because a live migration is mid-flight (Steps 4–5 draw the line). ```mermaid flowchart TD @@ -82,40 +105,64 @@ flowchart TD ``` Note the multi-key subtlety mid-migration: if a command touches several keys in a MIGRATING -slot and only *some* have already moved, neither node can serve it — the client gets -`-TRYAGAIN` and must back off and retry. +slot and only *some* have already moved (`multiple_keys && missing_keys` at `cluster.c:1409`), +neither node can serve it — the client gets `-TRYAGAIN` (`CLUSTER_REDIR_UNSTABLE`) and must back +off and retry. ### Step 4 — MOVED: the durable redirect -When the slot simply belongs to another node, `clusterRedirectClient()` (`cluster.c:1443`) -formats `-MOVED slot host:port` (the MOVED branch of the decision logic is at `cluster.c:1432`). -MOVED means: *the slot's home has permanently changed — update your slot map*. A well-behaved -client rewrites its cached slot→node entry (or refreshes the whole map with `CLUSTER SHARDS`) -and never asks the wrong node for that slot again. MOVED is how a cold client with an empty or -stale map converges: worst case one extra hop per slot, then steady-state direct routing. +> **In:** the "I don't own this slot, and no migration is in flight" branch from Step 3. +> **Out:** the `-MOVED` reply and the *permanent* client-map update it demands — the mechanism +> by which a cold or stale client converges to direct routing. + +When the slot simply belongs to another node, the base case at `cluster.c:1432` sets +`CLUSTER_REDIR_MOVED`, and `clusterRedirectClient()` (`cluster.c:1443`) formats +`-MOVED slot host:port`. **MOVED** means: *the slot's home has permanently changed — update your +slot map.* A well-behaved client rewrites its cached slot→node entry (or refreshes the whole map +with `CLUSTER SHARDS` / `CLUSTER SLOTS`) and never asks the wrong node for that slot again. MOVED +is how a cold client with an empty or stale map converges: worst case one extra hop per slot, +then steady-state direct routing. ### Step 5 — ASK + ASKING: the one-shot redirect during migration -ASK (branch at `cluster.c:1397`, same formatter at `cluster.c:1443`) is the temporary cousin: -*just this once, ask over there — do NOT update your map*. The source node emits it while a slot -is MIGRATING and the requested key has already been transferred. The client must then send two -commands to the target: `ASKING`, then the retried command. `askingCommand()` (`cluster.c:1680`) -sets a client flag permitting exactly ONE subsequent command against an IMPORTING slot. Without -the ASKING flag the target replies `-MOVED` *back to the source* — correct, because ownership -hasn't flipped yet. The one-shot design keeps the invariant: at any instant exactly one node is -the authoritative owner of a slot, and only explicitly-flagged requests may jump the gun. +> **In:** the "I own this slot but the key already migrated away" branch from Step 3. +> **Out:** the `-ASK`/`ASKING` two-command dance and the single invariant it protects — that at +> every instant exactly one node authoritatively owns a slot, MOVED's permanence notwithstanding. + +**ASK** is the temporary cousin of MOVED: *just this once, ask over there — do NOT update your +map.* The ASK branch is at `cluster.c:1397` (`CLUSTER_REDIR_ASK`, returning +`getMigratingSlotDest(slot)`), formatted by the same `clusterRedirectClient()` at +`cluster.c:1443`. The source node emits it while a slot is MIGRATING and the requested key has +already been transferred. The client must then send two commands to the target: `ASKING`, then +the retried command. `askingCommand()` (`cluster.c:1680`) sets `CLIENT_ASKING` (`cluster.c:1685`), +a flag permitting exactly ONE subsequent command against an IMPORTING slot; the flag is cleared +right after that command runs, in `commandProcessed()` at `networking.c:2891-2896`. Without the +ASKING flag the target replies `-MOVED` *back to the source* — correct, because ownership hasn't +flipped yet (the importing-slot serve path at `cluster.c:1406-1414` requires the flag). The +one-shot design keeps the invariant: at any instant exactly one node is the authoritative owner +of a slot, and only explicitly-flagged requests may jump the gun. + +The distinction in one line each: **MOVED = permanent, update your map, applies to every future +request; ASK = transient, do not update your map, applies to exactly this one request.** ### Step 6 — The SETSLOT state machine: moving a slot live -Per-node state lives in `cluster_legacy.h:343-344`: +> **In:** the ASK/MOVED replies of Steps 4–5, which are only *correct* if backed by per-slot +> migration state. +> **Out:** the two state arrays and four admin verbs that produce that state, and the proof that +> no instant leaves a key unanswerable. + +Per-node state lives in the `clusterState` struct at `cluster_legacy.h:343-344`: ```c -clusterNode *migrating_slots_to[CLUSTER_SLOTS]; /* source side: slot leaving, to whom */ -clusterNode *importing_slots_from[CLUSTER_SLOTS]; /* target side: slot arriving, from whom */ +// src/cluster_legacy.h:343-344 — per-slot migration pointers inside clusterState +343 clusterNode *migrating_slots_to[CLUSTER_SLOTS]; /* source side: slot leaving, to whom */ +344 clusterNode *importing_slots_from[CLUSTER_SLOTS]; /* target side: slot arriving, from whom */ ``` -The four `CLUSTER SETSLOT` verbs (`cluster_legacy.c:6072-6075`) drive the protocol — -MIGRATING, IMPORTING, STABLE (clear migration state), NODE (the final ownership flip): +The four `CLUSTER SETSLOT` verbs (documented at `cluster_legacy.c:6072-6075`, dispatched from +`cluster_legacy.c:6071`) drive the protocol — MIGRATING, IMPORTING, STABLE (clear migration +state), NODE (the final ownership flip): ```mermaid stateDiagram-v2 @@ -128,14 +175,18 @@ stateDiagram-v2 ``` The operator (e.g. `redis-cli --cluster reshard`) sets IMPORTING on the target first, then -MIGRATING on the source; keys move batch by batch with `MIGRATE` commands. Throughout, the -source serves keys it STILL HAS and ASK-redirects for keys already gone (Step 3's decision -tree). When the slot is empty, `SETSLOT slot NODE target-id` flips ownership; from then on -queries to the old owner get `-MOVED`. No moment exists where a key is unanswerable — the worst -outcomes are one extra network hop or a `-TRYAGAIN` retry. +MIGRATING on the source; keys move batch by batch with `MIGRATE` commands. Throughout, the source +serves keys it STILL HAS and ASK-redirects for keys already gone (Step 3's decision tree). When +the slot is empty, `SETSLOT slot NODE target-id` flips ownership; from then on queries to the old +owner get `-MOVED`. No moment exists where a key is unanswerable — the worst outcomes are one +extra network hop or a `-TRYAGAIN` retry. ### Step 7 — What the client library must implement +> **In:** the server-side contract of Steps 3–6, deliberately kept cheap. +> **Out:** the four client obligations that pay for that cheapness — read this as the spec the +> M36 capstone's Rust client must satisfy. + The server keeps its half of the contract cheap by pushing four obligations onto clients: 1. Maintain a slot→node map (16384 entries) and route directly on the fast path. @@ -145,29 +196,34 @@ The server keeps its half of the contract cheap by pushing four obligations onto atomically together, and treat `-TRYAGAIN` as retryable backoff. This is exactly the redirect contract planned for the M36 capstone's Rust graph engine -(slot = hash and 0x3FFF, MOVED/ASK-equivalent replies), so read this step as a spec. +(slot = `crc16 & 0x3FFF`, MOVED/ASK-equivalent replies), so read this step as a spec. ### Step 8 — Why 16384? +> **In:** the fixed slot count `2^14` asserted in Step 1. +> **Out:** the two opposing cost pressures that pick that exact number — so the constant reads as +> an engineering trade, not a magic value. + Two order-of-magnitude pressures meet in the middle. Each node advertises its owned slots as a bitmap in every gossip heartbeat, and the full slot map is serialized into node config — so -slot-count cost is paid per message and per node: 16384 slots is a 2 KiB bitmap, while 65536 -slots would quadruple every heartbeat's slot payload. Pulling the other way, more slots means -finer rebalancing granularity and more headroom for cluster size. At the intended scale (order -of a thousand masters), 16384 still leaves double-digit slots per node, so 2^14 is the sweet -spot: gossip stays small, granularity stays fine. Keep this qualitative — the exact message -layout lives in `cluster_legacy.h` if you want byte-level numbers. +slot-count cost is paid per message and per node: 16384 slots is a 2 KiB bitmap +(16384 / 8 = 2048 bytes), while 65536 slots would quadruple every heartbeat's slot payload. +Pulling the other way, more slots means finer rebalancing granularity and more headroom for +cluster size. At the intended scale (order of a thousand masters), 16384 still leaves double-digit +slots per node, so `2^14` is the sweet spot: gossip stays small, granularity stays fine. Keep +this qualitative — the exact message layout lives in `cluster_legacy.h` if you want byte-level +numbers. ## Where each step lives in the code | Step | What | Where | |---|---|---| -| 1 | `CLUSTER_SLOTS` = 2^14 via `CLUSTER_SLOT_MASK_BITS` | `cluster.h:23` | +| 1 | `CLUSTER_SLOTS` = 2^14 via `CLUSTER_SLOT_MASK_BITS` | `cluster.h:22-23` | | 1, 2 | `keyHashSlot()`: `crc16 & 0x3FFF`, hash-tag extraction, empty-`{}` fallback | `cluster.h:59` | -| 2 | `patternHashSlot()` for glob patterns | `cluster.c:36` | +| 2 | `patternHashSlot()` for glob patterns | `cluster.c:35` | | 3 | `getNodeByQuery()`: slot check, ownership, CROSSSLOT/TRYAGAIN | `cluster.c:1191` | -| 4 | MOVED decision branch; error formatting in `clusterRedirectClient()` | `cluster.c:1432`, `cluster.c:1443` | -| 5 | ASK decision branch; `askingCommand()` one-shot flag | `cluster.c:1397`, `cluster.c:1680` | +| 4 | MOVED base-case branch; error formatting in `clusterRedirectClient()` | `cluster.c:1432`, `cluster.c:1443` | +| 5 | ASK decision branch; `askingCommand()` one-shot flag set / cleared | `cluster.c:1397`, `cluster.c:1685`, `networking.c:2891-2896` | | 6 | `migrating_slots_to[]` / `importing_slots_from[]` per-slot state | `cluster_legacy.h:343-344` | | 6 | `CLUSTER SETSLOT` MIGRATING / IMPORTING / STABLE / NODE verbs | `cluster_legacy.c:6072-6075` | @@ -176,37 +232,110 @@ layout lives in `cluster_legacy.h` if you want byte-level numbers. 1. In `keyHashSlot()` (`cluster.h:59`), trace the exact behavior for the keys `"{}"`, `"{user}"`, and `"a{b}c{d}e"` — which bytes get hashed in each case, and why does the empty-tag fallback exist? -2. In `getNodeByQuery()` (`cluster.c:1191`), under precisely what combination of conditions - does a client get `-TRYAGAIN` instead of `-ASK`? Why can't the source just forward or serve? +2. In `getNodeByQuery()` (`cluster.c:1191`), under precisely what combination of + conditions does a client get `-TRYAGAIN` instead of `-ASK`? Why can't the source just forward + or serve? (Look at `multiple_keys && missing_keys`, `cluster.c:1409`.) 3. Follow `askingCommand()` (`cluster.c:1680`): where is the client's ASKING flag consumed and - cleared so that it permits exactly one command? What happens if the client sends ASKING to a - node whose slot is not importing? + cleared so that it permits exactly one command (`networking.c:2891-2896`)? What happens if the + client sends ASKING to a node whose slot is not importing? 4. Walk the SETSLOT verbs (`cluster_legacy.c:6072-6075`): what does `SETSLOT ... NODE` check - before flipping ownership, and how does the new owner make sure the rest of the cluster - learns about the flip rather than trusting stale gossip? + before flipping ownership, and how does the new owner make sure the rest of the cluster learns + about the flip rather than trusting stale gossip? 5. During a slot migration, list every reply a client can receive for a single-key GET on that slot (from source and from target, with and without ASKING) and confirm each against the decision paths in `cluster.c:1397-1443`. ## Done when +Answer each before unfolding it. + - [ ] You can compute a key's slot by hand (crc16, mask, hash-tag rules) and predict which keys co-locate. + +
Answer + + Slot = `crc16(H) & 0x3FFF`, where `H` is the *hash input* chosen by `keyHashSlot()` + (`cluster.h:59`): for a key with a `{`…`}` tag whose contents are non-empty, `H` is the bytes + between the first `{` and the next `}`; otherwise `H` is the whole key. Worked on the three + probe keys: + + - `"{}"` → the braces are empty, so the fallback fires and `H = "{}"` (the whole key is hashed). + - `"{user}"` → non-empty tag, `H = "user"`. + - `"a{b}c{d}e"` → first `{` then next `}` bracket just `"b"`, so `H = "b"`; the later `{d}` is + ignored. + + So keys co-locate iff their chosen `H` is byte-identical: `user:{42}:cart` and + `user:{42}:orders` both hash `"42"` → same slot; `user:42:cart` hashes the whole key → a + different slot. The empty-`{}` fallback exists so a literal `{}` in a key can't collapse every + such key onto one slot. + +
+ - [ ] You can state the MOVED vs ASK distinction in one sentence each, including what the client does to its slot map in each case. + +
Answer + + **MOVED** (`cluster.c:1432`, formatted at `:1443`): the slot's home has *permanently* changed — + the client updates its slot→node map (or refreshes it wholesale) and routes all future requests + for that slot to the new owner. **ASK** (`cluster.c:1397`): a migration is in flight and *this + one key* has already moved — the client sends `ASKING` + the command to the named target for + *this request only* and leaves its slot map untouched, because ownership has not flipped yet. + The map change is the whole difference: MOVED mutates it, ASK must not. + +
+ - [ ] You can draw the SETSLOT state machine from memory and explain why no request is ever unanswerable mid-migration. + +
Answer + + States: Stable (A owns) → Moving (operator runs `SETSLOT IMPORTING` on B, then + `SETSLOT MIGRATING` on A, setting `importing_slots_from[]` / `migrating_slots_to[]`, + `cluster_legacy.h:343-344`) → Flipped (`SETSLOT ... NODE B` once the slot is empty), with an + abort edge `SETSLOT STABLE` back to Stable. The four verbs live at `cluster_legacy.c:6072-6075`. + + No request is unanswerable because during Moving the source serves every key it still holds and + `-ASK`-redirects only keys already transferred; the target serves an ASK-flagged request and + otherwise `-MOVED`s back to the source; a partially-migrated multi-key command gets a retryable + `-TRYAGAIN`. Every case yields either an answer or a redirect carrying enough information to + retry — never a dropped or silently-wrong reply. + +
+ - [ ] You traced one full redirect in the source: `getNodeByQuery` → error code → `clusterRedirectClient` → client obligation. + +
Answer + + A cold client GETs a key whose slot B now owns. `getNodeByQuery()` (`cluster.c:1191`) + computes the slot, finds this node (A) is not the owner and no ASKING applies, and hits the base + case at `cluster.c:1432`, setting `CLUSTER_REDIR_MOVED` and returning node B. + `clusterRedirectClient()` (`cluster.c:1443`) formats `-MOVED `. The client's + obligation (Step 7 rule 2) is to update its slot→node map to point that slot at B and retry + there — after which it routes directly, one extra hop amortized away. The ASK path is the same + chain but via `cluster.c:1397` and leaves the map untouched. + +
+ - [ ] Questions 1-5 are answered in [notes.md](notes.md). +
Answer + + Done when `notes.md` contains your worked answers to all five questions above, each grounded in + a real `file:line` from this guide's "Where each step lives" table (not paraphrased from + memory), and cross-checked against the source with `tools/pinned-source.py show redis `. + +
+ ## References - Source: `~/repos/redis/src/cluster.h`, `~/repos/redis/src/cluster.c`, - `~/repos/redis/src/cluster_legacy.h`, `~/repos/redis/src/cluster_legacy.c` + `~/repos/redis/src/cluster_legacy.h`, `~/repos/redis/src/cluster_legacy.c` (pinned SHA in the + topic's `resources/codebases.md` pin table). - The Redis Cluster specification (the official protocol document; the source above is its - reference implementation) -- [Topic README](README.md) — lane 1 (mod-N vs fixed slots numbers) and lane context -- [reading-dynamo.md](reading-dynamo.md) — strategy 3: fixed partitions, movable ownership + reference implementation). +- [Topic README](README.md) — lane 1 (mod-N vs fixed slots numbers) and lane context. +- [reading-dynamo.md](reading-dynamo.md) — strategy 3: fixed partitions, movable ownership. - Topic 35's [reading-redis-backpressure.md](../35-overload/reading-redis-backpressure.md) - — same codebase, different subsystem + — same codebase, different subsystem. diff --git a/topics/37-distributed-query/notes.md b/topics/37-distributed-query/notes.md index d0fbbe8..5d92cc3 100644 --- a/topics/37-distributed-query/notes.md +++ b/topics/37-distributed-query/notes.md @@ -54,16 +54,21 @@ exposure (canary requests, micro-partitions). physical-plan/src/repartition/mod.rs:1150 (RepartitionExec), :1160 (preserve_order), :398-538 (merge mode, per-(input,output) spill channels), :560 (BatchPartitioner), :592 (REPARTITION_RANDOM_STATE, - seed 0), :667/:689 (hash ctor), :699 (round-robin ctor), :825 - (partition_iter), :854 (create_hashes), :675 (hash % strength- - reduced), :1329 (execute), :1742 (pull_from_input); + seed 0), :679 (new_hash_partitioner; :667 is its doc comment, + :689 the Hash state literal, :691 StrengthReducedU64::new), :710 + (new_round_robin_partitioner; :699 is its doc comment), :825 + (partition_iter), :854 (create_hashes), :862 + (partition_reducer.partition_indices — the strength-reduced + modulo; :675 is only a doc comment), :1329 (execute), :1742 + (pull_from_input); distributor_channels.rs:55 (channels()), :62 (Gate empty_channels), :121 (DistributionSender), :131 (send — parks when ALL buffers non-empty); physical-expr/src/partitioning.rs:117 (Partitioning), :119/:122 (RoundRobinBatch/Hash). Real finding: EnforceDistribution was retired into EnsureRequirements - (physical-optimizer/src/ensure_requirements/mod.rs:159; - enforce_distribution.rs:18/:76 are helpers + the retirement note). + (physical-optimizer/src/ensure_requirements/mod.rs:166, doc comment + :157-164; ensure_requirements/enforce_distribution.rs:18/:76 are + helpers + the retirement note). - Cockroach anchors verified: distsql_check.go:214 (checkSupportForPlanNode); distsql_physical_planner.go:312 (mustWrapNode — "no DistSQL-processor equivalent"), :971 diff --git a/topics/37-distributed-query/reading-cockroach-distsql.md b/topics/37-distributed-query/reading-cockroach-distsql.md index bf247f3..ee0ae37 100644 --- a/topics/37-distributed-query/reading-cockroach-distsql.md +++ b/topics/37-distributed-query/reading-cockroach-distsql.md @@ -20,6 +20,10 @@ looks like an ordinary iterator. ### Step 1 — First, decide whether the plan can distribute at all +> **In:** a logical plan — the tree of planNodes the optimizer produced. +> **Out:** a per-node verdict (distributable, local-only, or wrapped) so physical +> planning knows what can fan out. + Not every logical operator has a distributed-processor equivalent. `checkSupportForPlanNode` walks the logical plan and votes on each node: distributable, local-only, or somewhere in between. Nodes with no DistSQL processor equivalent are not a dead end — `mustWrapNode` wraps @@ -37,6 +41,11 @@ graph LR ### Step 2 — Data placement becomes the parallelism plan +> **In:** the table spans a scan needs, plus the range→leaseholder placement map +> from topic 36. +> **Out:** per-node scan work — each node assigned exactly the spans whose ranges +> it already leads, so placement *is* the parallelism. + This is the bridge from topic 36. `PartitionSpans` takes the table spans a scan needs, consults range ownership — the placement map you built in the sharding topic — and partitions the spans by the node holding each range's leaseholder. There is no separate scheduling decision: each @@ -56,6 +65,10 @@ fan-out width now equals the number of nodes owning relevant ranges — Step 7 c ### Step 3 — The physical plan: processors connected by streams +> **In:** the logical plan plus Step 2's per-node span assignment. +> **Out:** a `PhysicalPlan` — processors (typed by spec) wired by location-agnostic +> `StreamEndpointSpec` streams that may be a local queue or a remote gRPC hop. + `createPhysPlan` and `createPhysPlanForPlanNode` recursively turn the logical plan into a `PhysicalPlan` — the under-construction distributed plan, which is nothing more than a set of processors (typed by spec) plus the streams wiring their outputs to inputs. A stream endpoint is @@ -80,6 +93,10 @@ distributed and runs a copy on every node that already has a flow. ### Step 4 — Routers: Volcano's partitioning policies as protobuf enums +> **In:** a processor's output rows and a required distribution. +> **Out:** an `OutputRouterSpec` policy — PASS_THROUGH, MIRROR, BY_HASH, or +> BY_RANGE — that decides which output stream each row takes. + Where Volcano's exchange took C support functions to decide which consumer gets each row, DistSQL declares the policy in `OutputRouterSpec` and the wire format enumerates exactly the classic options: @@ -100,6 +117,11 @@ matching keys onto the same node without any join-side awareness. ### Step 5 — Flows: one fragment per node replaces fork() +> **In:** the `PhysicalPlan` from Step 3, sliced into the processors and streams +> that belong to one node. +> **Out:** a running `Flow` per node — processors instantiated from spec, +> goroutines launched, the last processor run inline in the caller's goroutine. + A `Flow` is the set of processors and streams scheduled on ONE node for one query — the unit the gateway ships out instead of forking worker processes. `Setup` instantiates processors from the spec, `StartInternal` launches the internal goroutines, and `Run` executes the *last* processor @@ -117,12 +139,26 @@ gateway node remote node 2 remote node 3 ### Step 6 — The exchange's two halves: Outbox and Inbox over gRPC +> **In:** a router's output on the producer node and a consumer waiting on another +> node. +> **Out:** an `Outbox` → gRPC `FlowStream` → `Inbox` pipe whose consumer end, +> `Inbox.Next`, is an ordinary operator iterator — the network hidden. + The vectorized engine splits exchange across the wire. The producer half is `Outbox`: its `Run` dials the consumer node and opens a FlowStream RPC, then `sendBatches` serializes record batches onto the stream. The consumer half is `Inbox`: `RunWithStream` is where the gRPC handler hands the incoming stream to the reader, and `Next` is a plain operator iterator — the downstream join or aggregator pulls batches from the Inbox exactly as it would from a local scan. Volcano's -encapsulation survives the network hop intact. +encapsulation survives the network hop intact. The signature is the whole point — no stream, no +node, just a batch: + +```go +// pkg/sql/colflow/colrpc/inbox.go — Inbox.Next, the consumer half of the exchange +333 func (i *Inbox) Next() (coldata.Batch, *execinfrapb.ProducerMetadata) { +334 if i.done { +335 return coldata.ZeroBatch, nil +336 } +``` Read the two halves in this order, and the symmetry becomes obvious: @@ -158,6 +194,10 @@ graph TD ### Step 7 — The price: fan-out width is tail-latency exposure +> **In:** the fan-out width `PartitionSpans` derived from placement (Step 2). +> **Out:** the tail-latency bill — a query over ranges on N nodes waits for its +> slowest flow, exactly topic 37's fan-out math. + Because `PartitionSpans` derives fan-out from placement, a query over ranges on N nodes waits for its slowest flow. This is exactly topic 37's fanout lane from "The Tail at Scale": if each node is slow 1 time in 100, a 100-way fan-out query is slow 63% of the time — the per-node p99 @@ -200,14 +240,73 @@ Paths are relative to `~/repos/cockroach`. ## Done when +Answer each before unfolding it. + - [ ] You can narrate the full path — logical plan → `checkSupportForPlanNode` → `PartitionSpans` → `PhysicalPlan` → per-node `Flow` — without looking at the code. + +
Answer + + `checkSupportForPlanNode` (distsql_check.go:214) votes each planNode + distributable / local-only / wrapped, and `mustWrapNode` + (distsql_physical_planner.go:312) embeds the ones with no processor equivalent. + `PartitionSpans` (:971) splits the scan's spans by leaseholder node, so placement + sets the fan-out. `createPhysPlan` / `createPhysPlanForPlanNode` (:3604 / :3632) + build the `PhysicalPlan` (physicalplan/physical_plan.go:125) — processors joined + by `StreamEndpointSpec` streams (execinfrapb/data.proto:72). The gateway then ships + each node its slice as a `Flow` (flowinfra/flow.go:72), set up and started per node. + Placement → fragments → streams → flows. + +
+ - [ ] You can point at the line where the consumer side of a network exchange becomes an ordinary iterator, and explain why that keeps every other operator network-oblivious. + +
Answer + + `Inbox.Next` (colflow/colrpc/inbox.go:333) returns + `(coldata.Batch, *execinfrapb.ProducerMetadata)` — the exact signature of any + vectorized operator, with no stream or node in it. The downstream join or + aggregator calls `Next` and cannot tell whether the batch arrived from a local + queue or a gRPC `FlowStream` (opened by `Outbox.Run`, outbox.go:218; fed by + `sendBatches` :323; handed to the reader in `Inbox.RunWithStream` :212). Because the + network hides behind the same iterator contract, every other operator is unchanged + from the single-node case — Volcano's anonymous input, now remote. + +
+ - [ ] You can map all four `OutputRouterSpec` policies to their Volcano exchange ancestors and name which one a distributed hash join uses and why. + +
Answer + + From `execinfrapb/data.proto`: PASS_THROUGH (:152) = single consumer, no routing; + MIRROR (:154) = broadcast to all consumers (Volcano's broadcast-by-pinning); + BY_HASH (:157) = hash of key columns picks the stream; BY_RANGE (:160) = preset key + boundaries pick the stream. They are Volcano's support-function policies + (round-robin / range / hash, plus broadcast) written as a protobuf enum. A + distributed hash join uses **BY_HASH**: hashing the join keys routes matching keys + from both inputs to the same node, so each node joins its own partition with no + join-side awareness — the runtime router is `hashRouter` (rowflow/routers.go:538) + or `HashRouter` (colflow/routers.go:443). + +
+ - [ ] You have answered all five questions above in `notes.md`, with file:line evidence. +
Answer + + Each answer should carry the anchor a reader can check against `~/repos/cockroach`: + (1) `checkSupportForPlanNode`:214 + `mustWrapNode`:312 — the wrapped planNode runs + locally on the gateway, losing distribution; (2) `PartitionSpans`:971 and the + stale-leaseholder re-route; (3) `Flow.Run`:566 runs the last processor inline, + saving one goroutine per flow per node; (4) `sendBatches`:323 — backpressure is + gRPC stream flow control, not an explicit window; (5) the Step 7 model, + 1 − 0.99^20 ≈ 18.2% at 20-way fan-out versus 63.4% at 100-way, so keeping hot tables + on fewer nodes shrinks the tail exposure. + +
+ ## References - **Code**: `~/repos/cockroach` — all anchors above are relative to the repo root. diff --git a/topics/37-distributed-query/reading-datafusion-repartition.md b/topics/37-distributed-query/reading-datafusion-repartition.md index 9576b34..1fe3246 100644 --- a/topics/37-distributed-query/reading-datafusion-repartition.md +++ b/topics/37-distributed-query/reading-datafusion-repartition.md @@ -17,6 +17,11 @@ came from eight different producers. RepartitionExec is where that pretense is m ### Step 1 — The N-to-M fan: what the operator actually is +> **In:** N input partitions of `RecordBatch`es (from the child plan) and a target +> output-partition count M. +> **Out:** M output partitions, each a stream a downstream operator drains as if it +> were the only consumer of a single-threaded child. + `RepartitionExec` (mod.rs:1150) maps N input partitions to M output partitions. `execute` (:1329) is called once per *output* partition; on first call it spawns one `pull_from_input` task (:1742) per *input* partition. Each task drains its input stream, routes every batch, and pushes into per-output channels. Consumers just poll their channel. @@ -38,6 +43,10 @@ in-flight batches. `consume_input_streams` (:393) is the entry point on the prod ### Step 2 — Routing policy is a plan-time property +> **In:** a plan node's required input distribution (what Step 7's rule computes). +> **Out:** a `Partitioning` enum value — `RoundRobinBatch(M)` or `Hash(exprs, M)` — +> that fixes routing before a single batch moves. + The `Partitioning` enum (partitioning.rs:117) declares the contract: `RoundRobinBatch(usize)` (:119) says "any batch anywhere, just balance the load"; `Hash(Vec<Arc<dyn PhysicalExpr>>, usize)` (:122) says "rows with equal key expressions must land in the same output partition." Round-robin is what you use to widen @@ -56,25 +65,40 @@ graph LR ### Step 3 — Deterministic hashing: the seeds are zero on purpose -`BatchPartitioner` (mod.rs:560) is constructed for hash mode at :667 and :689 using -`REPARTITION_RANDOM_STATE` (:592) — a `RandomState` built with FIXED seeds of 0. This is not laziness; it is a +> **In:** the `Hash(exprs, M)` policy from Step 2 and a batch's key columns. +> **Out:** a deterministic per-row partition index — identical across runs, +> operators, and (in a distributed setting) nodes. + +`BatchPartitioner` (mod.rs:560) is built for hash mode by `new_hash_partitioner` (:679), whose `Hash` +state carries a strength-reduced reducer, `StrengthReducedU64::new(num_partitions)` (:691), and hashes +with `REPARTITION_RANDOM_STATE` (:592) — not a per-instance `RandomState` but a +`SeededRandomState::with_seed(0)`, i.e. FIXED seeds of 0. This is not laziness; it is a correctness contract. A hash join repartitions *both* inputs by the join keys: if the build side and the probe side hashed with different per-instance random seeds, key `42` could go to partition 1 on one side and partition 5 on the other, and the join would silently drop matches. Fixed seeds make routing deterministic across runs, across -operators, and (in a distributed setting) across nodes. The partition index is then `hash % partition_count` -(:675 — a strength-reduced modulo, since M is a runtime value, not a power of two you can mask with). +operators, and (in a distributed setting) across nodes. The partition index is not a visible `hash % M`: the +routing loop calls `partition_reducer.partition_indices(hash_buffer, indices)` (:862), and the reducer built at +:691 turns each hash into `hash % M` without a division in the hot loop (the reason is documented at :594 — M is a +runtime value, not a power of two you can mask with). The flip side of determinism: adversarial or pathological key sets that collide will collide *every* run — skew is reproducible, which is good for debugging and bad if your workload hits it. ### Step 4 — Batch economics: whole batches round-robin, per-row scatter for hash +> **In:** one input `RecordBatch` and the routing policy from Step 2. +> **Out:** for round-robin, that whole batch handed to the next output in rotation; +> for hash, M smaller batches, one per output partition. + `partition_iter` (mod.rs:825) is the routing loop, and it treats the two policies asymmetrically: -- **Round-robin** forwards the *entire* RecordBatch to the next output in rotation. Zero per-row work, zero copies — - the batch is just a bundle of Arc'd arrays changing queues. -- **Hash** must look at every row: `create_hashes` (:854) computes one hash per row over the key columns, the loop - builds an index list per target partition, and arrow's `take` kernel gathers one new batch per partition. +- **Round-robin** forwards the *entire* RecordBatch to the next output in rotation + (`*next_idx = (*next_idx + 1) % *num_partitions`, :836; the whole batch is yielded at :837). Zero per-row work, + zero copies — the batch is just a bundle of Arc'd arrays changing queues. +- **Hash** must look at every row: `create_hashes` (:854), seeded by `REPARTITION_RANDOM_STATE.random_state()` + (:856), computes one hash per row over the key columns; the reducer buckets rows into a per-partition index list + (`partition_reducer.partition_indices`, :862); then `Self::partition_grouped_take` (:868) runs arrow's `take` per + bucket to gather one new batch per partition. This is Volcano's packet-economics lesson restated. Graefe measured 171 s at 1 record per exchange packet versus 13.7 s at 83 records per packet — a 12x swing purely from amortizing per-transfer cost. DataFusion's RecordBatch @@ -95,10 +119,27 @@ re-materializes results as full columnar batches so every downstream operator st ### Step 5 — Flow control: unbounded channels behind one global gate -The channels are built by `channels()` (distributor_channels.rs:55) — N linked per-output buffers sharing one -`Gate` with an `empty_channels` counter (:62). `DistributionSender::send` (:121, :131) implements the unusual -semantics: each per-output buffer is individually *unbounded*, and a sender parks only when *all* M output buffers -are non-empty. If any single channel is empty, every send proceeds. +> **In:** routed `(partition, batch)` pairs from the N producer tasks. +> **Out:** M per-output channels plus one shared `Gate` that applies backpressure +> only when *every* output buffer is non-empty. + +The channels are built by `channels()` (distributor_channels.rs:55) — M linked per-output buffers sharing one +`Gate` whose `empty_channels` counter starts at M (`AtomicUsize::new(n)`, :62). `DistributionSender::send` +(:121, :131) implements the unusual semantics: each per-output buffer is individually *unbounded*, and a sender +parks only when *all* M output buffers are non-empty. If any single channel is empty, every send proceeds. The +rule is one branch in `SendFuture::poll` — when the counter of empty channels is zero, the sender registers a +waker and yields: + +```rust +// datafusion/physical-plan/src/repartition/distributor_channels.rs — SendFuture::poll +226 if this.gate.empty_channels.load(Ordering::SeqCst) == 0 { +227 let mut guard = this.gate.send_wakers.lock(); +228 if let Some(send_wakers) = guard.deref_mut() { +229 send_wakers.push((cx.waker().clone(), this.channel.id)); +230 return Poll::Pending; +231 } +232 } +``` Why not M bounded queues? Deadlock and starvation in join plans. With per-channel bounds, one slow consumer (say, a join output partition blocked on its other input) would block the producer, which would then stop feeding @@ -122,6 +163,11 @@ graph TD ### Step 6 — preserve_order: the merging exchange +> **In:** N input partitions that are each already sorted, and a plan that needs +> that order kept. +> **Out:** M output streams, each a k-way merge over its N per-input channels — +> Volcano's merging exchange, one process at a time. + When each input partition is already sorted and the plan needs that order, the `preserve_order` flag (mod.rs:1160) switches RepartitionExec into merge mode — Volcano's MERGING exchange. Records from different producers must be kept *distinct* per producer and merged by sort key, never dumped into one bag. The machinery at :398-538 keeps a @@ -140,12 +186,22 @@ its next head), which is exactly why the flag is opt-in rather than default. ### Step 7 — Who inserts RepartitionExec: EnforceDistribution is retired +> **In:** a plan whose nodes declare required input distributions (hash on join +> keys, or a single partition). +> **Out:** the same plan with a `RepartitionExec` inserted wherever a child's +> distribution does not already satisfy the requirement. + You will read blog posts saying "the EnforceDistribution rule inserts RepartitionExec." That rule was RETIRED and -folded into `EnsureRequirements` (physical-optimizer/src/ensure_requirements/mod.rs:159). The old file -`enforce_distribution.rs` still exists but now holds helper functions; its doc comments (:18, :76) say so -explicitly. When a plan node declares a required input distribution — hash-partitioned on join keys, or a single -partition — EnsureRequirements inserts the RepartitionExec that satisfies it. Do not be confused when you grep for -the rule the guides mention and find only helpers. +folded into `EnsureRequirements`, a struct at `physical-optimizer/src/ensure_requirements/mod.rs:166` (the doc +comment at :157 says it "combines the functionality of `EnforceDistribution` and `EnforceSorting`"). The old +standalone file did not merely lose its rule — it *moved*: the helpers now live at +`physical-optimizer/src/ensure_requirements/enforce_distribution.rs`, whose retirement note is spelled out in a +doc comment (:18) and again at :76. `EnsureRequirements::optimize` (mod.rs:176) calls `ensure_distribution` +(enforce_distribution.rs:1053) bottom-up; when a plan node declares a required input distribution that a child does +not meet, that pass inserts the RepartitionExec — round-robin via `add_roundrobin_on_top` (:674, building +`RepartitionExec::try_new` at :688) or hash via the `should_add_hash_repartition` branch (:1281) building +`RepartitionExec::try_new` at :1291. Do not be confused when you grep for the rule the old guides mention and find +only helpers behind a moved path. ## Where each step lives in the code @@ -156,21 +212,27 @@ the rule the guides mention and find only helpers. | 1 | Per-input pull task | datafusion/physical-plan/src/repartition/mod.rs:1742 | | 2 | Partitioning enum | datafusion/physical-expr/src/partitioning.rs:117 | | 2 | RoundRobinBatch / Hash variants | datafusion/physical-expr/src/partitioning.rs:119, :122 | -| 3 | Fixed-seed RandomState | datafusion/physical-plan/src/repartition/mod.rs:592 | -| 3 | Hash constructor paths | datafusion/physical-plan/src/repartition/mod.rs:667, :689 | -| 3 | hash % partition_count | datafusion/physical-plan/src/repartition/mod.rs:675 | -| 4 | Routing core struct | datafusion/physical-plan/src/repartition/mod.rs:560 | -| 4 | Round-robin constructor | datafusion/physical-plan/src/repartition/mod.rs:699 | -| 4 | partition_iter routing loop | datafusion/physical-plan/src/repartition/mod.rs:825 | -| 4 | Per-row create_hashes | datafusion/physical-plan/src/repartition/mod.rs:854 | +| 3 | Fixed-seed SeededRandomState::with_seed(0) | datafusion/physical-plan/src/repartition/mod.rs:592 | +| 3 | Hash constructor (`new_hash_partitioner`) | datafusion/physical-plan/src/repartition/mod.rs:679 | +| 3 | Strength-reduced reducer built (`StrengthReducedU64::new`) | datafusion/physical-plan/src/repartition/mod.rs:691 | +| 3 | `hash % M` applied (`partition_reducer.partition_indices`) | datafusion/physical-plan/src/repartition/mod.rs:862 | +| 4 | Routing core struct (`BatchPartitioner`) | datafusion/physical-plan/src/repartition/mod.rs:560 | +| 4 | Round-robin constructor (`new_round_robin_partitioner`) | datafusion/physical-plan/src/repartition/mod.rs:710 | +| 4 | partition_iter routing loop; round-robin next_idx at :836 | datafusion/physical-plan/src/repartition/mod.rs:825 | +| 4 | Per-row create_hashes (with `.random_state()` :856) | datafusion/physical-plan/src/repartition/mod.rs:854 | +| 4 | Per-partition gather (`partition_grouped_take`, arrow `take`) | datafusion/physical-plan/src/repartition/mod.rs:868 | | 5 | channels constructor | datafusion/physical-plan/src/repartition/distributor_channels.rs:55 | | 5 | Gate with empty_channels | datafusion/physical-plan/src/repartition/distributor_channels.rs:62 | | 5 | DistributionSender::send | datafusion/physical-plan/src/repartition/distributor_channels.rs:121, :131 | +| 5 | Gate park branch (`SendFuture::poll`) | datafusion/physical-plan/src/repartition/distributor_channels.rs:226 | | 6 | preserve_order flag | datafusion/physical-plan/src/repartition/mod.rs:1160 | | 6 | Merge-mode channel machinery | datafusion/physical-plan/src/repartition/mod.rs:398-538 | | 6 | consume_input_streams | datafusion/physical-plan/src/repartition/mod.rs:393 | -| 7 | EnsureRequirements rule | datafusion/physical-optimizer/src/ensure_requirements/mod.rs:159 | -| 7 | Retirement notes | datafusion/physical-optimizer/src/enforce_distribution.rs:18, :76 | +| 7 | EnsureRequirements struct (doc :157) | datafusion/physical-optimizer/src/ensure_requirements/mod.rs:166 | +| 7 | optimize → ensure_distribution | datafusion/physical-optimizer/src/ensure_requirements/mod.rs:176 | +| 7 | Retirement notes (moved path) | datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:18, :76 | +| 7 | Hash RepartitionExec inserted | datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:1291 | +| 7 | Round-robin RepartitionExec inserted | datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs:688 | ## Questions to answer in notes.md @@ -189,14 +251,84 @@ the rule the guides mention and find only helpers. ## Done when +Answer each before unfolding it. + - [ ] You can sketch the N×M channel matrix from memory and explain why it is not one shared MPMC queue. + +
Answer + + `execute` (mod.rs:1329) is called once per *output* partition; its first call spawns + one `pull_from_input` task (:1742) per *input* partition. So there are N producer + tasks and M consumer streams, wired by N×M logical channels (`channels()`, + distributor_channels.rs:55). It is not one shared MPMC queue for two reasons: + order preservation (Step 6) needs each (input, output) pair kept distinct so the + k-way merge can tell producers apart; and a single global lock over all in-flight + batches would re-serialize every producer and consumer, throwing away the + parallelism — the matrix confines contention to one channel plus the shared gate. + +
+ - [ ] You can state the deterministic-hashing contract and name two operators whose correctness depends on it. + +
Answer + + `REPARTITION_RANDOM_STATE` is a `SeededRandomState::with_seed(0)` (mod.rs:592) — + fixed zero seeds — so `create_hashes` (:854) followed by the strength-reduced + `partition_reducer.partition_indices` (:862) maps a given key to the *same* + partition on every instance, run, and node. A **hash join** depends on it (build + and probe sides must send equal join keys to the same partition, or matches are + silently dropped) and so does a **hash aggregate** (equal group keys must co-locate, + or one group is split across partitions and never merged). + +
+ - [ ] You can explain the Gate flow-control rule ("park only when all buffers are non-empty") and its failure mode trade-off versus per-channel bounds. + +
Answer + + Each per-output buffer is individually unbounded; `SendFuture::poll` parks the + sender only when `gate.empty_channels == 0` — every output already has data waiting + (distributor_channels.rs:226). If any channel is empty the send proceeds, so a fast + consumer is never starved by backpressure aimed at a slow one. Per-channel bounds + would let one blocked consumer (say a join output waiting on its other input) stall + the producer, which then stops feeding the fast consumers too — a distribution + deadlock. The trade-off is honest: in the worst case (every consumer stalled except + one that never fills) buffering can grow past what a bounded design would allow; + DataFusion accepts that memory risk in exchange for liveness. + +
+ - [ ] You can explain when preserve_order forces merge mode and what it costs relative to interleaving. + +
Answer + + When each input partition is already sorted and the plan requires that order, the + `preserve_order` flag (mod.rs:1160) switches `consume_input_streams` (:393; the + preserve-order branch spans :398–538) to a dedicated channel per (input, output) + pair, and each output runs a streaming k-way merge over its N per-input channels + instead of interleaving in arrival order. The cost: a per-output merge heap plus + stricter buffering — the merge cannot emit until *every* input channel has shown its + next head — so first-row latency rises. That is why the flag is opt-in, not default. + +
+ - [ ] You have run the local exchange stub and compared its routing throughput numbers against your own reasoning about DataFusion's batch-level costs. +
Answer + + The stub (`experiments/src/exchange.rs`) routes row-at-a-time, so at k=8 hash + (543.0 M rows/s) *beats* round-robin (229.6 M rows/s): hashing is cheap and + round-robin's per-row bookkeeping dominates when there is no batch to amortize. + DataFusion inverts this because it works on whole batches — round-robin forwards a + `RecordBatch` with zero per-row work (Step 4, :836–837), while hash pays + `create_hashes` + `partition_grouped_take` per row before re-materializing batches. + Same operation, opposite verdict: what looks cheap depends on whether you measure + per row or per batch — the packet-economics lesson from Volcano's §5. + +
+ ## References - Repo: `~/repos/datafusion` — `datafusion/physical-plan/src/repartition/` (operator + channels), diff --git a/topics/37-distributed-query/reading-tail-at-scale.md b/topics/37-distributed-query/reading-tail-at-scale.md index 2d8e406..fccfd5c 100644 --- a/topics/37-distributed-query/reading-tail-at-scale.md +++ b/topics/37-distributed-query/reading-tail-at-scale.md @@ -17,6 +17,10 @@ so the system must tolerate the tail rather than try to eliminate it. ### Step 1 — Individual machines are irreducibly variable +> **In:** one server, one request. +> **Out:** the premise every later step rests on — a single machine's latency +> distribution has a long tail you cannot engineer to zero. + Before any distributed effect, a single server's latency already has a long tail, caused by: - **Shared resources**: CPU cores, processor caches, memory and network bandwidth contended by @@ -31,19 +35,32 @@ The paper's stance: you can trim these, but you cannot remove them all. Plan acc ### Step 2 — The fan-out arithmetic -The headline calculation. Suppose a server answers slowly 1 time in 100. Alone, that is fine. -Now fan a query out to 100 such servers and wait for **all** of them: +> **In:** a per-leaf slow probability `p` and a fan-out width `n` — Step 1 said +> `p` is irreducible, not that it is large. +> **Out:** `P(at least one slow) = 1 − (1 − p)^n`, the probability a wait-for-all +> query is slow. + +The headline calculation. **Fan-out** means one request splits into `n` sub-requests, one per leaf +server; **wait-for-all** (a scatter-gather) means the query cannot answer until every leaf has. If a +leaf is slow independently with probability `p`, the query is fast only when *all* `n` leaves are +fast — probability `(1 − p)^n` — so it is slow with probability `1 − (1 − p)^n`. Work it on the +pairs the topic's crate pins, and the shape appears: ``` -P(at least one leaf is slow) = 1 - 0.99^100 ≈ 63% +P(at least one leaf is slow) = 1 − (1 − p)^n p = per-leaf slow prob, n = fan-out - 1-in-100 slow, 100 leaves, wait-all → 63% of queries slow - 1-in-10,000 slow, 2,000 leaves → 1 - 0.9999^2000 ≈ 18% slow + p = 1/100, n = 100: 1 − 0.99^100 = 1 − 0.3660 = 0.6340 → 63.4% + p = 1/1000, n = 100: 1 − 0.999^100 = 1 − 0.9048 = 0.0952 → 9.5% + p = 1/1000, n = 1000: 1 − 0.999^1000 = 1 − 0.3677 = 0.6323 → 63.2% + p = 1/10000, n = 2000: 1 − 0.9999^2000 = 1 − 0.8187 = 0.1813 → 18.1% ``` -Fan-out converts rare slowness into common slowness: **the component's tail becomes the -service's median**. Even heroic per-machine engineering (1-in-10,000) does not save a -2,000-leaf query. The topic's crate pins these exact numbers in tests: 63.4% and 18.1%. +Read the pairs against each other: driving `p` down 10× (1/100 → 1/1000) buys back the fan-out you +lost, but only until `n` climbs to match — 1/1000 slowness at 1,000 leaves is the *same* 63% as +1/100 at 100. Fan-out converts rare slowness into common slowness: **the component's tail becomes +the service's median**. Even heroic per-machine engineering (1-in-10,000) does not save a +2,000-leaf query. The topic's crate pins two of these exactly: `p_any_slow(0.01, 100) = 0.633968` +and `p_any_slow(0.0001, 2000) = 0.1813` (`experiments/src/fanout.rs`). ```mermaid graph LR @@ -57,6 +74,10 @@ graph LR ### Step 3 — Table 1: what this looks like in a real Google service +> **In:** one leaf's latency distribution (the p50/p95/p99 of a single random leaf). +> **Out:** the end-to-end distribution of a 100-leaf fan-out under two gather +> policies — wait for 95% of leaves, or wait for 100%. + The paper measures a real service, per-leaf versus end-to-end: ``` @@ -69,11 +90,33 @@ The paper measures a real service, per-leaf versus end-to-end: Read the last row against the middle row: **the slowest 5% of leaf requests account for half of the 99th-percentile end-to-end latency** (140 ms vs 70 ms). This single table motivates every technique that follows — and the "95% row" is itself a technique (Step 7). The local -simulation reproduces the shape: one-leaf p50 5.6 ms vs 100-leaf-wait-all p50 1000 ms, while -waiting for only 95% of leaves gives p99 9.9 ms. +simulation reproduces the shape, and its full gather table is worth staring at because it also +shows the *cost* of good-enough (all three rows measured, `experiments/src/fanout.rs`): + +``` + wait for p50 p95 p99 + one leaf 5.6 ms 9.6 ms 10.0 ms + 95% of 100 9.6 ms 9.9 ms 9.9 ms + all 100 1000.0 ms 1000.0 ms 1000.0 ms +``` + +Three readings, and the third is the honest one: +- **Wait for all 100 is catastrophic**: every percentile is a full 1000 ms stall, because 63.4% + of queries hit at least one slow leaf (Step 2) — even the *median* query stalls. +- **Waiting for 95% rescues the tail**: dropping the slowest 5% removes the stall, so p99 falls + to 9.9 ms — essentially the single-leaf p99 (10.0 ms). +- **But 95% is not free at the median**: p50 rises from 5.6 ms (one leaf) to 9.6 ms, because you + now always wait for the 95th-fastest of 100 fast leaves instead of one random leaf. This is the + measured headline (`FINDINGS.md`): p99 10.0 → 9.9 ms, **p50 5.6 → 9.6 ms**. Partial response + trades median latency for tail latency; it is a choice, not a win. ### Step 4 — Hedged requests: pay a little extra load to cut the tail +> **In:** one outstanding request plus a delay budget — the paper's budget is the +> **95th-percentile expected latency** for that request class. +> **Out:** at most one extra ("hedged") copy and the first answer to return, with +> the added load bounded to ~5% because only the slowest requests ever hedge. + The simplest within-request technique. Send the request to one replica. If no reply arrives within a delay — they use the **95th-percentile expected latency** — send a secondary request to another replica, take the first answer, and cancel the loser. Deferring the hedge until p95 @@ -94,6 +137,11 @@ local `hedge.rs` stub targets the same effect: its reference solution takes p99. ### Step 5 — Tied requests: cancel at queue-entry, not at completion +> **In:** one request, enqueued on **two** servers at once, each copy tagged with +> the identity of its twin. +> **Out:** exactly one execution plus one cross-server cancellation — the duplicate +> is dequeued before it runs, so the extra work is queue slots, not CPU. + Hedging still waits out the delay. Tied requests go further: enqueue the request on **two** servers immediately, each copy tagged with the identity of its twin. When one server **starts executing**, it sends a cross-server cancellation to the other, which dequeues the still-queued @@ -124,6 +172,11 @@ interference. Disk-read overhead from duplicate dequeues stays under 1%. ### Step 6 — Why not just probe queue lengths and pick the shorter queue? +> **In:** the tied-request design from Step 5, and its obvious-looking rival — +> ask both servers how busy they are, then send once to the shorter queue. +> **Out:** three reasons the probe loses, so tying (commit to both, cancel one) +> wins. + The obvious alternative — ask both servers how busy they are, then send once — is worse, for three reasons: @@ -137,6 +190,11 @@ decide. ### Step 7 — Cross-request, longer-term techniques +> **In:** slower-moving imbalance — skew and hot spots that persist across many +> requests, not the per-request jitter Steps 4–6 attack. +> **Out:** five cross-request tools (micro-partitions, selective replication, +> latency-induced probation, good-enough results, canary requests). + Within-request tricks handle transient variability; these handle slower-moving imbalance: - **Micro-partitions**: many more partitions than machines (about 20 per machine; BigTable @@ -147,13 +205,20 @@ Within-request tricks handle transient variability; these handle slower-moving i issuing shadow requests to it, reinstate when it recovers. Counterintuitively, **removing capacity improves latency** during overload. - **Good-enough results**: once enough leaves respond, answer with what you have — Table 1's - 95% row shows the payoff (p99 70 ms instead of 140 ms). + 95% row shows the payoff (p99 70 ms instead of 140 ms; in the paper's smooth distribution the + p50 also improves, 12 ms vs 40 ms). But the payoff is distribution-dependent: on the local + two-mode model (Step 3), 95% rescues the p99 yet *raises* the p50 (5.6 → 9.6 ms). Reach for it + when the alternative is waiting on stragglers, not as a universal speed-up. - **Canary requests**: on every large fan-out, send to 1-2 leaves first; fan out fully only if the canaries succeed in reasonable time. This guards against an untested code path crashing thousands of servers at once. Google applies canaries to every large fan-out query. ### Step 8 — Mutations are the easy case, and the thesis restated +> **In:** the write (mutation) path, plus every read-side technique from Steps 4–7. +> **Out:** why writes are the easy case, and the paper's thesis restated — build +> tail-*tolerant* systems, do not try to erase the tail. + Writes are much easier than reads: they can be taken off the critical path (acknowledge after a durable log write, apply asynchronously), and quorum-based systems such as Paxos with 3-5 replicas are **inherently tail-tolerant** — they only need the fastest majority, so the @@ -194,16 +259,88 @@ tail-tolerant systems that mask it, exactly as fault-tolerant systems mask failu ## Done when +Answer each before unfolding it. + - [ ] You can derive 1 − 0.99^100 ≈ 63% and explain "the tail becomes the median" without looking at the paper. + +
Answer + + Each leaf is fast with probability 0.99, so all 100 are fast with probability + 0.99^100 = 0.3660, and a wait-for-all query is slow with probability + 1 − 0.3660 = 0.6340 — 63.4%. "The tail becomes the median": the crate's leaf is + a two-mode mixture — fast (uniform 1–10 ms) or, with probability 0.01, a 1000 ms + stall. One leaf's p50 is ~5.6 ms, but the *max* over 100 leaves has a p50 of a + full 1000 ms stall (`fanout.rs::the_leaf_tail_becomes_the_service_median`), so + the leaf's 1-in-100 tail event is the *median* outcome of the fan-out. The + closed form is pinned at `p_any_slow(0.01, 100) = 0.633968` + (`experiments/src/fanout.rs`). + +
+ - [ ] You can state Table 1's half-of-p99 observation and the BigTable hedging result (1,800 ms → 74 ms at +2% requests) from memory. + +
Answer + + Table 1 (a real Google service): one random leaf p99 = 10 ms; wait for 95% of + leaves p99 = 70 ms; wait for 100% p99 = 140 ms. The slowest 5% of leaf requests + are responsible for half the 99th-percentile end-to-end latency (140 − 70 = 70, + half of 140). Hedging benchmark: reading 1,000 keys spread over 100 BigTable + servers, sending a secondary request after a 10 ms delay cut the 99.9th-percentile + from 1,800 ms to 74 ms while sending just 2% more requests — because 98% of + requests finished before the hedge fired (paper, "Hedged requests"). + +
+ - [ ] You can explain tied-request cancellation and why it beats queue-length probing. + +
Answer + + Tied requests enqueue on two servers, each copy tagged with its twin's identity; + the moment one server *dequeues to start executing*, it sends a cancellation to + the twin, which drops the still-queued copy. Sends are staggered by ~2× the + average network message delay (≤1 ms) so both do not start at once. It beats + probe-then-send because probing suffers staleness (load moves between probe and + arrival), hard service-time estimation from queue length alone, and herding (every + client piles onto the momentarily-idle server). Tying commits to both queues and + lets execution order decide. Table 2: idle-cluster p99.9 98 → 61 ms (−38%); with a + concurrent terasort 159 → 108 ms (−32%); tied-under-terasort ≈ unhedged-on-idle. + +
+ - [ ] The local fan-out simulation's pinned numbers (63.4%, 18.1%, wait-95% p99 9.9 ms) match your hand-derived expectations. + +
Answer + + `p_any_slow(0.01, 100) = 0.633968` and `p_any_slow(0.0001, 2000) = 0.1813` + (`fanout.rs` tests); the seed-7, 20k-trial simulation lands within 0.02 of 0.634. + The gather table: one-leaf p50 5.6 / p99 10.0 ms; 95%-of-100 p50 9.6 / p99 9.9 ms; + all-100 p50 = p99 = 1000 ms. Note the trade the numbers force: wait-95% holds the + tail at the single-leaf level (p99 10.0 → 9.9) but nearly doubles the median + (p50 5.6 → 9.6). Partial response buys tail latency with median latency — do not + sell it as free. + +
+ - [ ] You have completed the hedging stub and observed a tail reduction comparable to the reference solution (p99.9 1000 ms → 18.3 ms at +0.5% requests). +
Answer + + Implement `request_with_hedge` in `experiments/src/hedge.rs`: draw the primary + latency; with `hedge_delay = Some(d)`, if the primary exceeds `d` fire a second + draw and return `(min(primary, d + secondary), 2)`, otherwise `(primary, 1)`; with + `None` return `(primary, 1)`. At `P_SLOW = 0.005` the unhedged p99.9 *is* the + 1000 ms stall; a 10 ms hedge replaces it with "delay + a second draw" — both draws + must stall to stay slow and `p_slow²` is negligible — so p99.9 falls to ~18.3 ms at + about +0.5% requests. The pinned contracts: a 10 ms hedge cuts p99.9 by ≥10×; the + extra-request fraction stays under 10%; a zero-delay hedge degenerates into sending + every request twice. + +
+ ## References - Jeffrey Dean and Luiz André Barroso. "The Tail at Scale." *Communications of the ACM*, diff --git a/topics/37-distributed-query/reading-volcano-exchange.md b/topics/37-distributed-query/reading-volcano-exchange.md index e28a1cb..a2a79d2 100644 --- a/topics/37-distributed-query/reading-volcano-exchange.md +++ b/topics/37-distributed-query/reading-volcano-exchange.md @@ -19,6 +19,10 @@ every operator had to know about processes, queues, and partitions. ### Step 1 — Anonymous inputs: the iterator contract does the heavy lifting +> **In:** an operator (scan, join, sort) and whatever feeds it. +> **Out:** the discipline that an operator never knows *what* produces its input — +> the one precondition that lets Step 2 encapsulate parallelism in a new operator. + Volcano makes every operator an iterator with `open`/`next`/`close`. The crucial discipline is that inputs are **anonymous**: an operator never knows or cares what produces its input — it just calls `next` on an opaque handle. A join pulling from a scan is indistinguishable, from the join's point @@ -28,6 +32,10 @@ operators but *one more operator*. ### Step 2 — Exchange: drop-in parallelism +> **In:** a working single-threaded plan of anonymous-input iterators (Step 1). +> **Out:** the same plan with one `exchange` operator spliced between two operators +> — now parallel, with every scan/join/sort's code unchanged. + Because inputs are anonymous, you can splice an exchange operator between any two operators in a plan. Scan, join, and sort code runs unchanged, single-threaded, inside each process; exchange forks processes, routes records between them, and hides all synchronization behind the same @@ -45,6 +53,10 @@ The optimizer reasons about query semantics; exchange placement is a separate, m ### Step 3 — What happens on open: forks, packets, queues (§4.2) +> **In:** an `exchange` operator's `open` call, driven from its consumer side. +> **Out:** a forked producer process (or group) shipping **packets** — arrays of +> records, 1–32,000 per packet — through shared-memory queues to the consumer. + Exchange's consumer side is an ordinary iterator. On `open`, it forks a producer process (or a group of them). Producer and consumer exchange data as **packets** — batches of records — through shared-memory queues. `next` on the consumer side just unpacks the current packet and blocks on @@ -57,6 +69,11 @@ GAMMA) so query start does not pay fork latency. ### Step 4 — Three kinds of parallelism from one operator +> **In:** the fork-and-route machinery from Step 3, plus a per-record *support +> function* that picks an output queue. +> **Out:** all three classic parallel forms — vertical (pipelining), bushy, and +> intra-operator — from that single operator. + Exchange gives all three classic forms: - **Vertical parallelism** — pipelining: producer and consumer subtrees run concurrently in @@ -83,6 +100,11 @@ termination interesting. ### Step 5 — Counted end-of-stream, not assumed (§4.3) +> **In:** the j-producers × k-consumers mesh from Step 4, each producer finishing +> at its own time. +> **Out:** correct termination — every consumer counts one end-of-stream packet +> from each producer before it reports end-of-stream upward. + End-of-stream is **counted, not assumed**. Each producer, when done, sends a flagged end-of-stream packet to *every* consumer; each consumer must count one from every producer before it reports end-of-stream upward. The paper's example: 3 producers × 4 consumers = 12 end-of-stream packets. @@ -92,6 +114,10 @@ coordinator watches the pipeline. ### Step 6 — The §4.4 variants: broadcast, merging, exchange-in-the-middle +> **In:** the basic exchange from Steps 3–5. +> **Out:** four refinements — broadcast-by-pinning, the merging exchange, +> exchange-in-the-middle (the paper's *interchange*), and run-time fork-vs-reuse. + Four refinements, each with a lasting lesson: 1. **Broadcast by pinning, not copying.** To send one packet to multiple consumers, exchange pins @@ -101,8 +127,8 @@ Four refinements, each with a lasting lesson: draining queues into one big bag. Mixing streams destroys the sort order each producer worked to establish. This is the lesson the topic's stub test pins. 3. **Exchange-in-the-middle.** An exchange that does not fork at all but re-routes partitions - between processes created by other exchanges. This variant makes flow control obsolete — and - makes vertical parallelism optional. + between processes created by other exchanges — the paper calls this variant **interchange**. + This variant makes flow control obsolete — and makes vertical parallelism optional. 4. **Fork vs reuse is a run-time switch**, not a compile-time decision. ```text @@ -117,6 +143,10 @@ Four refinements, each with a lasting lesson: ### Step 7 — A buffer manager built for many processes (§4.5) +> **In:** many producer and consumer processes contending on one shared buffer pool. +> **Out:** a deadlock-free, two-level-locking buffer manager that never becomes the +> serialization bottleneck the parallelism was meant to remove. + Shared-memory parallelism needs a buffer manager that will not become the bottleneck or deadlock. Volcano uses two-level locking: a pool lock that is **never held during I/O**, plus per-descriptor locks; a restart scheme removes hold-and-wait, making the buffer manager deadlock-free by design. @@ -126,6 +156,11 @@ QUIT requests, decoupling I/O from the query processes. ### Step 8 — The numbers: exchange overhead and the 12× batching swing (§5) +> **In:** the exchange implementation of Steps 3–7, benchmarked on a Sequent +> Symmetry over 100K-record inputs. +> **Out:** the measured price of an exchange (25.73 µs/record) and a ~12× batching +> swing — the economics that drove vectorized execution two decades later. + Measured on a Sequent Symmetry — 12 CPUs, 16 MHz Intel 80386, 100K-record inputs: | Configuration | Time | @@ -182,16 +217,86 @@ top-down vs a scheduler pushing bottom-up) sharpens why exchange needs no schedu ## Done when +Answer each before unfolding it. + - [ ] You can sketch the j × k producer/consumer mesh and state the end-of-stream packet count for any j and k without looking it up. + +
Answer + + Each of the `j` producers fills packets destined for any of the `k` consumers, + chosen per record by the support function, so every producer can reach every + consumer — a full `j × k` mesh. On termination each producer sends one flagged + end-of-stream packet to *every* consumer, so there are `j × k` end-of-stream + packets in total (the paper's example: 3 × 4 = 12). Each consumer must count one + from every producer — `j` flags each — before it reports end-of-stream upward. + Stop after the first flag and you truncate (a still-working producer's rows are + dropped); the symmetric mistake — never reaching the count — hangs forever + (§4.3). + +
+ - [ ] You can explain, in two sentences, why anonymous inputs are the precondition for encapsulating parallelism in one operator. + +
Answer + + Because every operator pulls from an opaque input handle and never learns what + produces it, an exchange can be spliced between any two operators and neither + notices — a join draining a shared-memory queue fed by another process is + indistinguishable from a join draining a local scan. If inputs were typed or + known, every operator would need process-, queue-, and partition-awareness (as in + GAMMA's bracket model), and parallelism could not be confined to one new operator. + +
+ - [ ] You can name all four §4.4 variants and the failure mode the merging exchange avoids. + +
Answer + + (1) **Broadcast by pinning** — send one packet to many consumers by pinning it in + the shared buffer for each, no copy; (2) the **merging exchange** — fuse `k` + sorted producer streams, kept grouped by producer; (3) **exchange-in-the-middle / + interchange** — no fork, just re-route partitions among existing processes, which + makes flow control obsolete; (4) **fork-vs-reuse** as a run-time switch. The + merging exchange avoids *order loss*: dumping every producer's packets into one bag + and "merging" destroys the sort order each producer established. The merge iterator + must distinguish records by their producer — the paper: "it is crucial to + distinguish the input records by their producer in order to merge multiple sorted + streams correctly." + +
+ - [ ] You can quote the batching swing (1 vs 83 records/packet) and connect it to vectorized execution economics. + +
Answer + + Packet-size sweep: 1 record/packet → 171 s; 2 → 94 s; 50 → 15.0 s; 83 (one page's + worth, the default) → 13.7 s — roughly a 12× swing purely from amortizing the + per-packet fixed costs (a semaphore signal, one linked-list insertion into the + port, buffer fix/unfix) over more records. That is the vectorized-execution + argument two decades early: do one synchronization and dispatch per *batch*, not + per row. DataFusion's `RecordBatch` is exactly this packet, and round-robin + forwards it intact (§5). + +
+ - [ ] The merging-exchange test against the stub in `experiments/src/exchange.rs` passes with a per-producer-stream merge and fails with a single-bag merge. +
Answer + + The stub's merging-exchange contract pins Step 6's lesson. Implement the merge by + keeping one cursor per producer stream and repeatedly emitting the smallest current + head (a streaming k-way merge) and the globally sorted output is correct. + Implement it by concatenating every producer's packets into one buffer and emitting + arrival order, and the per-producer sort order is lost and the test fails. It is the + same distinction DataFusion draws between `preserve_order` (its merging + `RepartitionExec`) and the default arrival-order interleave. + +
+ ## References - Goetz Graefe. *Encapsulation of Parallelism in the Volcano Query Processing System.* SIGMOD 1990. diff --git a/topics/38-graphrag-agent-memory/README.md b/topics/38-graphrag-agent-memory/README.md index 09591ce..79ce0ac 100644 --- a/topics/38-graphrag-agent-memory/README.md +++ b/topics/38-graphrag-agent-memory/README.md @@ -85,7 +85,7 @@ edges; news: 15,754 / 19,520): community-level answers win 72-83% on comprehensiveness and 62-82% on diversity vs vector RAG, and the root level C0 needs 9-43× fewer query tokens than map-reducing source texts — 26,657 tokens ≈ 2.6% of the corpus. Indexing cost: 281 min of -gpt-4-turbo. The trade is explicit: pay once at index time for +gpt-4-turbo on the Podcast dataset, at a 600-token chunk window. The trade is explicit: pay once at index time for structure so every global query is cheap. ## Zep/Graphiti: the bi-temporal agent memory diff --git a/topics/38-graphrag-agent-memory/notes.md b/topics/38-graphrag-agent-memory/notes.md index f5b7171..0baad30 100644 --- a/topics/38-graphrag-agent-memory/notes.md +++ b/topics/38-graphrag-agent-memory/notes.md @@ -51,9 +51,9 @@ sources. At damping 0.5 the meet node gets ~2× a dead-end's mass. ## Infra notes -- Papers read in full from PDFs: /tmp/hipporag.pdf (arXiv 2405.14831v3), - /tmp/graphrag.pdf (arXiv 2404.16130v2), /tmp/zep.pdf (arXiv - 2501.13956). +- Papers read in full: arXiv 2405.14831v3 (HippoRAG), 2404.16130v2 + (GraphRAG), 2501.13956 (Zep). Extract them into `.cache/papers/` with + `tools/pinned-source.py` rather than a scratch path. - HippoRAG facts: neocortex=LLM / parahippocampal=retrieval encoders (synonymy at cosine τ=0.8) / hippocampus=KG+PPR (damping 0.5). Two-step OpenIE (NER, then triples). Node specificity sᵢ=|Pᵢ|⁻¹ @@ -72,7 +72,8 @@ sources. At damping 0.5 the meet node gets ~2× a dead-end's mass. substitute child summaries on overflow). Query: shuffle+chunk summaries, map partial answers scored 0-100 (0 filtered), reduce by helpfulness. Podcast corpus 1669 chunks (~1M tokens) → 8,564/20,691; - News 3197 (~1.7M) → 15,754/19,520; indexing 281 min gpt-4-turbo. + News 3197 (~1.7M) → 15,754/19,520; indexing 281 min gpt-4-turbo on + Podcast at a 600-token window (the 8k window is generation-side). Win rates vs vector RAG: comprehensiveness 72-83%, diversity 62-82%. C0 query cost 26,657 tokens ≈ 2.6% of TS max; root 9-43× fewer tokens; C3 26-33% fewer. Claim experiment: 47,075 claims, avg diff --git a/topics/38-graphrag-agent-memory/reading-graphrag-paper.md b/topics/38-graphrag-agent-memory/reading-graphrag-paper.md index bf7f773..534a803 100644 --- a/topics/38-graphrag-agent-memory/reading-graphrag-paper.md +++ b/topics/38-graphrag-agent-memory/reading-graphrag-paper.md @@ -1,14 +1,18 @@ # Microsoft GraphRAG: pay at index time so global questions get cheap -Vector RAG answers "what does the corpus say about X" by pulling the top-k most -similar passages. That works when the evidence lives in a few passages — it -fails structurally when the question is about the corpus as a whole ("what are -the main themes across this dataset?"), because no top-k retrieval can see -everything. Edge et al. propose GraphRAG: extract an entity-relationship graph -from the corpus once, partition it hierarchically with Leiden, pre-summarize -every community, and answer global queries by map-reduce over those summaries. -For a database engineer this is a familiar shape: a materialized view built -offline so that an expensive analytical query becomes a cheap scan. +Vector RAG answers "what does the corpus say about X" by pulling the top-k most similar +passages. That works when the evidence lives in a few passages — it fails structurally when the +question is about the corpus as a whole ("what are the main themes across this dataset?"), +because no top-k retrieval can see everything. Edge et al. propose GraphRAG (arXiv 2404.16130): +extract an entity-relationship graph from the corpus once, partition it hierarchically with +Leiden, pre-summarize every community, and answer global queries by map-reduce over those +summaries. For a database engineer this is a familiar shape: a materialized view built offline +so that an expensive analytical query becomes a cheap scan. + +Every number below is quoted with the section, table or figure it comes from in arXiv +2404.16130. The distinct **local search** algorithm (entity-anchored, for specific-fact +questions) is a different algorithm with a different cost profile; this guide is about the +paper's headline **global search** path, and says so wherever the two could be confused. ## The problem in one sentence @@ -17,23 +21,28 @@ in top-k retrieved passages, while global sensemaking questions require aggregating over the entire corpus — which query-focused summarization can do, but not at RAG-scale corpus sizes.** -Vector RAG ("semantic search" in the paper) is local by construction. -Query-focused summarization (QFS) methods produce the right kind of answer but -do not scale to corpora of a million tokens or more. GraphRAG bridges the two -by moving the expensive aggregation work to index time. +Vector RAG ("semantic search" — abbreviated **SS** in the paper) is local by construction: it +ranks passages by similarity to the query and reads the top few. **Query-focused +summarization (QFS)** — summarizing a whole corpus through the lens of a specific query — +produces the right kind of answer but does not scale to corpora of a million tokens or more. +GraphRAG bridges the two by moving the expensive aggregation work to index time. ## The concepts, step by step ### Step 1 — Chunk, extract, and re-extract ("gleanings") -Source documents are split into 600-token chunks with 100-token overlap. An -LLM extracts entities, relationships, and claims from each chunk — and -crucially, each extracted element carries a free-text description, not just a -name and type. Because a single pass over a large chunk misses things, the -pipeline runs "gleanings": the LLM is asked whether it missed anything and -re-extracts, up to a maximum number of passes. This is what lets larger chunk -sizes recover entities they would otherwise drop. Few-shot examples in the -extraction prompt tailor it to the domain. +> **In:** the source documents. +> **Out:** per-chunk entities, relationships and claims — each carrying a free-text +> description — which Step 2 merges into a graph. + +Source documents are split into **600-token chunks with 100-token overlap** (§A.2). An LLM +extracts entities, relationships, and claims from each chunk — and crucially, each extracted +element carries a **free-text description**, not just a name and type, because those +descriptions are what Step 4 later summarizes. Because a single pass over a large chunk misses +things, the pipeline runs **gleanings**: after the first pass the LLM is asked whether it missed +anything and re-extracts, up to a maximum number of passes (§2.3). That is what lets larger +chunk sizes recover entities they would otherwise drop. Few-shot examples in the extraction +prompt tailor it to the domain. ``` documents @@ -50,19 +59,28 @@ extraction prompt tailor it to the domain. ### Step 2 — Dedup builds the graph; duplicates become edge weights -Entity instances are merged by exact string match on the entity name — a -deliberately crude resolution step. The interesting move is on edges: when the -same relationship is detected multiple times across chunks, the duplicate -count becomes the edge weight. Frequency of independent extraction is treated -as a signal of importance, for free, with no extra LLM calls. +> **In:** the per-chunk extractions from Step 1. +> **Out:** one weighted graph — the input Leiden partitions in Step 3. + +Entity instances are merged by **exact string match on the entity name** (§2.3) — a deliberately +crude resolution step (the paper notes it is robust because later community summarization is +resilient to duplicate variants). The interesting move is on edges: when the same relationship +is detected multiple times across chunks, the **duplicate count becomes the edge weight**. +Frequency of independent extraction is treated as a signal of importance, for free, with no +extra LLM calls — and that weight is what Leiden optimizes over next. ### Step 3 — Hierarchical community detection with Leiden -The weighted graph is partitioned with the Leiden algorithm (graspologic -implementation), recursively, producing a multi-level hierarchy: level C0 is -the root partition (coarsest, fewest communities), then C1, C2, C3 -progressively finer. Every node belongs to exactly one community per level, so -each level is a complete, mutually exclusive cover of the graph. +> **In:** the weighted graph from Step 2. +> **Out:** a multi-level community hierarchy C0…C3 — the units Step 4 summarizes. + +A **community** is a group of nodes more densely connected to each other than to the rest of the +graph. The **Leiden algorithm** (Traag et al. 2019, via the graspologic implementation; §2.4) +finds them and guarantees connected communities; run recursively it produces a hierarchy. Level +**C0** is the root partition — coarsest, fewest communities (**34 units** for Podcast, **55** for +News; Table 2) — then C1, C2, **C3** progressively finer (**1310** and **2142** units). Every +node belongs to exactly one community per level, so each level is a complete, mutually exclusive +cover of the graph. ``` C0 (root): [==== community A ====][==== community B ====] @@ -76,23 +94,31 @@ each level is a complete, mutually exclusive cover of the graph. ### Step 4 — Bottom-up community summaries under a token budget -Each community gets an LLM-written summary, built bottom-up. For leaf -communities, element summaries (node, edge, and claim descriptions) are added -in order of decreasing combined source+target node degree until the context -window fills — high-degree elements first, so the most connected facts win -budget. For higher-level communities: if all child element summaries fit, use -them directly; otherwise substitute the shorter sub-community summaries, -prioritizing the sub-communities that account for the most element summaries. -This is recursive compression with a degree-based eviction policy. +> **In:** the community hierarchy from Step 3 and the element descriptions from Step 1. +> **Out:** one LLM-written summary per community per level — the artifacts Step 5 answers +> from. + +Each community gets an LLM-written summary, built **bottom-up** (§2.5). For **leaf** communities, +element summaries (node, edge, and claim descriptions) are added in order of **decreasing +combined source+target node degree** until the context window fills — high-degree elements +first, so the most connected facts win budget. For **higher-level** communities: if all child +element summaries fit, use them directly; otherwise substitute the shorter **sub-community +summaries**, prioritizing the sub-communities that account for the most element summaries. This +is recursive compression with a degree-based eviction policy — the summaries themselves are +generated once, at index time, within the fixed 8k generation window (§3.3). ### Step 5 — Query time: map-reduce over summaries -At query time, pick a hierarchy level. The community summaries at that level -are shuffled and packed into chunks. The map step has the LLM answer the query -from each chunk independently, producing a partial answer plus a 0-100 -helpfulness score. Partials scoring 0 are filtered out. The reduce step -assembles the surviving partials in descending helpfulness order into the -final answer. +> **In:** a global query and the community summaries at a chosen level (from Step 4). +> **Out:** one final answer — assembled without touching the source corpus. + +At query time, pick a hierarchy level. The community summaries at that level are **shuffled** and +packed into chunks (§2.6). The **map** step has the LLM answer the query from each chunk +independently, producing a partial answer plus a **0–100 helpfulness score**. Partials scoring 0 +are filtered out. The **reduce** step assembles the surviving partials in descending helpfulness +order into the final answer. The shuffle matters: it spreads relevant facts across chunks +"rather than concentrated (and potentially lost) in a single context window" (§2.6, line 281 of +the extracted text). ``` community summaries @ level Ck @@ -112,90 +138,186 @@ final answer. ### Step 6 — The experimental grid: C0-C3 vs TS vs SS -Six conditions: C0, C1, C2, C3 (map-reduce over community summaries, root -down to the lowest level), TS (map-reduce directly over the source texts — the -no-index upper bound), and SS (vanilla vector RAG). Two corpora: podcast -transcripts (1669 chunks of 600 tokens, ~1M tokens → 8,564 nodes, 20,691 -edges) and news articles (3197 chunks, ~1.7M tokens → 15,754 nodes, 19,520 -edges). Indexing took 281 minutes with gpt-4-turbo at an 8k context window — -that number is the price of the materialized view. +> **In:** the built index (Steps 1–4) and the query path (Step 5). +> **Out:** the six comparison conditions and the two corpora Step 7 judges. + +Six conditions: **C0, C1, C2, C3** (map-reduce over community summaries, root down to the lowest +level), **TS** (map-reduce directly over the source texts — the no-index upper bound), and **SS** +(vanilla vector RAG). Two corpora: podcast transcripts (**1669** × 600-token chunks, +≈ 1,014,611 tokens → **8,564 nodes, 20,691 edges**; §4.1) and news articles (**3197** chunks, +≈ 1,707,694 tokens → **15,754 nodes, 19,520 edges**). + +Get the cost anchor exactly right, because the paper states two different window sizes and they +are easy to conflate: + +``` + 600-token window -> used to CHUNK for graph indexing (§A.2); + indexing took 281 minutes for the PODCAST dataset + on gpt-4-turbo (§3.3, "2M TPM, 10k RPM") + 8k-token window -> used to GENERATE community summaries, community + answers, and global answers (§3.3, Appendix C) +``` + +The 281 minutes is the price of the materialized view, and it is the *Podcast* index built +under a *600-token* chunk window — not an 8k-window number. ### Step 7 — Evaluating global answers without gold labels -Global questions have no reference answers, so both question generation and -judging use LLMs. Questions: K=5 personas × M=5 tasks × N=5 questions = 125 -per dataset, generated from only a corpus description (no specific texts), so -they stay global by construction. Judging: head-to-head LLM-as-judge on -comprehensiveness, diversity, and empowerment, plus directness as a control — -directness is expected to favor vector RAG, and it does, which sanity-checks -the judge. A second experiment corroborates the judge: an LLM claim extractor -(Claimify) pulled 47,075 factual claims across all answers (≈31 per answer on -average); C0 on News yielded 34.18 claims per answer vs 25.23 for SS. +> **In:** the six conditions from Step 6. +> **Out:** the win-rate and claim-count evidence Step 8 reads for cost. + +Global questions have no reference answers, so both question generation and judging use LLMs. +**Questions:** K=5 personas × M=5 tasks × N=5 questions = **125 per dataset** (§3.2), generated +from only a corpus description (no specific texts), so they stay global by construction. +**Judging:** head-to-head **LLM-as-judge** on **comprehensiveness, diversity, and empowerment**, +plus **directness** as a *control* — directness rewards concision, so it is "effectively in +opposition to comprehensiveness and diversity" (§3.4) and is expected to favor vector RAG. It +does, which sanity-checks the judge. A second experiment corroborates the judge (§5.2): an LLM +claim extractor (**Claimify**) pulled **47,075 unique claims** across all answers (**≈ 31 per +answer**); on News, C0 yielded **34.18 claims/answer** vs **25.23** for SS (Table 3). ### Step 8 — The token economics (why C0 is the headline) -Graph conditions beat SS on comprehensiveness with win rates of 72-83% on -Podcast and 72-80% on News (diversity: 75-82% and 62-71%). The cost side is -Table 2: C0 answers a Podcast query with 26,657 tokens, roughly 2.6% of what -TS needs at its maximum (News C0: 39,770 tokens ≈ 2.3%). Root-level summaries -need 9-43× fewer query tokens than TS — over 97% fewer — and even the finest -level C3 uses 26-33% fewer tokens than TS. The engineering trade: spend the -281 minutes once, then every global query is an order of magnitude (or two) -cheaper. Same amortization argument as a materialized view or an analytical -replica. +> **In:** the win-rates and Table 2 token counts from Step 7. +> **Out:** the amortization argument — the whole reason to build the index. + +Global (graph) conditions beat SS on **comprehensiveness** with win rates of **72–83%** on +Podcast and **72–80%** on News; **diversity** win rates are **75–82%** and **62–71%** (§5.1). The +cost side is **Table 2**: C0 answers a Podcast query with **26,657 tokens**, and TS needs +**1,014,611**; C0 on News is **39,770** vs TS **1,707,694**. Worked as percentages: + +``` + Podcast C0 / TS = 26,657 / 1,014,611 = 2.63% (Table 2 "% Max" row: 2.6) + News C0 / TS = 39,770 / 1,707,694 = 2.33% (Table 2 "% Max" row: 2.3) + + per-query tokens SAVED on News by using C0 instead of TS: + 1,707,694 - 39,770 = 1,667,924 tokens/query +``` + +Across levels, root summaries need **9×–43×** fewer query tokens than TS — **over 97% fewer** — +and even the finest level **C3** uses **26–33% fewer** tokens than TS (§5.1). The engineering +trade: spend the one-time indexing cost, then each of those 1.67M-token-per-query savings +accrues on every global query. The break-even is the materialized-view calculation — index cost +÷ per-query saving = the query volume at which the view pays for itself — with the caveat that +the paper reports indexing *time* (281 min, Podcast) rather than an indexing *token* count, so +the token break-even needs one stated assumption about index cost. ## How to read the paper (with the concepts in hand) -1. Read the abstract and introduction for the local-vs-global framing — the - claim that vector RAG cannot, structurally, answer whole-corpus questions - (the problem sentence above). -2. Find the pipeline description and match it to Steps 1-4: chunking and - gleanings, then the dedup/edge-weight construction, then Leiden levels, - then the bottom-up summary packing rules. The degree-ordered element - packing is easy to skim past — slow down there. -3. Read the query-time map-reduce description against Step 5, paying - attention to the shuffle, the 0-100 scoring, and the score-0 filter. -4. Move to the evaluation setup: the six conditions (Step 6), the two - datasets with their node/edge counts, and the persona-based question - generation (Step 7). Note that TS is the expensive upper bound, SS the - cheap baseline, and C0-C3 the dial between them. -5. Read the results with Step 8 in hand: win rates first, then Table 2 for - token costs. Check the directness control behaves as predicted. -6. Finish with the Claimify claim-count experiment — the authors' answer to +1. Read the abstract and introduction for the local-vs-global framing — the claim that vector + RAG cannot, structurally, answer whole-corpus questions (the problem sentence above). +2. Read §2 and match it to Steps 1–4: chunking and gleanings (§2.3), the dedup/edge-weight + construction (§2.3), Leiden levels (§2.4), then the bottom-up summary packing rules (§2.5). + The degree-ordered element packing is easy to skim past — slow down there. +3. Read the query-time map-reduce (§2.6) against Step 5, paying attention to the shuffle, the + 0–100 scoring, and the score-0 filter. +4. Move to the evaluation setup (§3): the six conditions (Step 6), the two datasets with their + node/edge counts (§4.1), the persona-based question generation (§3.2), and — carefully — + the two window sizes in §3.3 (600-token indexing vs 8k generation). TS is the expensive + upper bound, SS the cheap baseline, C0–C3 the dial between them. +5. Read the results (§5.1) with Step 8 in hand: win rates first, then Table 2 for token costs. + Check the directness control behaves as predicted. +6. Finish with the Claimify claim-count experiment (§5.2, Table 3) — the authors' answer to "isn't LLM-as-judge circular?" ## Questions to answer in notes.md -1. Why does the gleanings mechanism specifically enable larger chunk sizes, - and what is the cost model trade-off (extraction calls vs chunks) it buys? -2. Entity dedup is exact string match on the name. Where does that break, and - what would a graph database bring to the resolution step instead? -3. Why must community summaries be built bottom-up with degree-ordered - packing rather than summarizing each community's raw text independently? -4. In the map step, why shuffle the community summaries before chunking, and - what failure mode would an unshuffled ordering create? -5. Using Table 2, at what query volume does the 281-minute indexing cost - break even against TS for the News corpus? Sketch the arithmetic. +1. Why does the gleanings mechanism specifically enable larger chunk sizes, and what is the + cost model trade-off (extraction calls vs chunks) it buys? +2. Entity dedup is exact string match on the name. Where does that break, and what would a + graph database bring to the resolution step instead? +3. Why must community summaries be built bottom-up with degree-ordered packing rather than + summarizing each community's raw text independently? +4. In the map step, why shuffle the community summaries before chunking, and what failure mode + would an unshuffled ordering create? +5. Using Table 2, sketch the break-even: News C0 saves 1,667,924 tokens/query vs TS. What + index cost (in the same tokens) would make the crossover happen at, say, 100 queries — and + what does the paper actually report instead (§3.3)? ## Done when -- [ ] You can draw the full indexing pipeline (chunks → extraction with - gleanings → weighted graph → Leiden hierarchy → summaries) from memory. -- [ ] You can explain why C0 wins on token cost and when you would pick C3 - or TS instead. -- [ ] You can state the three judged metrics plus the directness control and - why the control matters. -- [ ] You have answered the five questions above in notes.md, including the - break-even arithmetic. +Answer each before unfolding it. + +- [ ] You can draw the full indexing pipeline (chunks → extraction with gleanings → weighted graph → Leiden hierarchy → summaries) from memory. + +
Answer + + Documents → **600-token chunks / 100-token overlap** (Step 1) → per-chunk LLM extraction of + entities/relationships/claims *with free-text descriptions*, repeated as **gleanings** until + the LLM finds nothing new → merge by **exact-string entity name**, with **duplicate + relationship counts becoming edge weights** (Step 2) → **Leiden** recursive partition into a + **C0…C3 hierarchy** where every node sits in exactly one community per level (Step 3) → + **bottom-up** per-community summaries that pack element descriptions in **decreasing + source+target node degree**, substituting shorter sub-community summaries on overflow (Step 4). + All of it is index-time work. + +
+ +- [ ] You can explain why C0 wins on token cost and when you would pick C3 or TS instead. + +
Answer + + C0 is the **root** level — the fewest communities (34 Podcast / 55 News units, Table 2), so a + query touches the fewest, shortest summaries: **26,657 tokens on Podcast (2.6% of TS), 39,770 + on News (2.3%)**, i.e. **9–43× fewer** than TS and **over 97% fewer** at root (§5.1). It still + wins comprehensiveness (72% win rate) and diversity (62%) over vector RAG. Pick a **finer + level (C1–C3)** when you need more specific coverage and can pay 26–33% (C3) up to most-of-TS + (C1/C2) token cost for a modest quality gain. Pick **TS** only as the no-index upper bound — + it reads the whole corpus per query (1.0M/1.7M tokens) and is what the index exists to avoid. + +
+ +- [ ] You can state the three judged metrics plus the directness control and why the control matters. + +
Answer + + The three judged metrics are **comprehensiveness, diversity, and empowerment** (§3.4). + **Directness** is the **control**: it rewards concise, narrowly-on-point answers, so it is "in + opposition to comprehensiveness and diversity" and is *expected* to favor vector RAG (SS). It + does — and that expected result is the point: an LLM judge that also handed GraphRAG the + directness win would be suspect, so directness going the other way is evidence the judge + discriminates rather than rubber-stamps the graph method. The Claimify claim-count experiment + (§5.2, Table 3: C0 News 34.18 vs SS 25.23 claims/answer) is the second, non-LLM-judge check. + +
+ +- [ ] You have answered the five questions above in notes.md, including the break-even arithmetic. + +
Answer + + notes.md records all five with their anchors: gleanings vs chunk size and the extraction-call + trade (§2.3); where exact-string dedup breaks and what real entity resolution would add + (§2.3, and this repo's topic 25 SDK resolution ladder); why bottom-up degree-ordered packing + beats independent per-community summarization (§2.5); why the map step shuffles summaries + (§2.6, to avoid concentrating relevant facts in one dropped chunk); and the break-even sketch + — News C0 saves **1,667,924 tokens/query** vs TS, so with an assumed index cost of X tokens + the crossover is X ÷ 1,667,924 queries, noting the paper reports **281 min (Podcast)** wall + clock rather than an index token count (§3.3). + +
## References -- Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused - Summarization", arXiv 2404.16130v2 — https://arxiv.org/abs/2404.16130 -- Local PDF: /tmp/graphrag.pdf -- Companion notes in this topic: [README.md](README.md) — HippoRAG covers the - local/associative axis and Zep the temporal axis; GraphRAG covers the - global-question axis. -- [experiments/](experiments/) — this repo's crate demonstrates the local - path-finding side, not community summarization. -- Leiden implementation used by the paper: graspologic. +- Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," + arXiv 2404.16130 — https://arxiv.org/abs/2404.16130. Section, table and figure numbers in + this chapter are from that version. + +| Where | What it settles | +|---|---| +| §2.3 | 600-token / 100-overlap chunks; gleanings; exact-string entity merge; duplicate count → edge weight | +| §2.4 | Leiden hierarchical communities (graspologic) | +| §2.5 | bottom-up summaries; decreasing source+target degree packing; sub-community substitution | +| §2.6 | map-reduce; shuffle; 0–100 helpfulness; score-0 filter; descending-helpfulness reduce | +| §3.2 | K=M=N=5 → 125 questions from a corpus description | +| §3.3, App. C | 600-token indexing window vs 8k generation window; 281 min for the **Podcast** index; gpt-4-turbo | +| §3.4 | comprehensiveness / diversity / empowerment + directness control | +| §4.1 | Podcast 1669 chunks → 8,564 nodes / 20,691 edges; News 3197 → 15,754 / 19,520 | +| §5.1, Table 2 | tokens: Podcast C0 26,657 (2.6%), News C0 39,770 (2.3%); 9–43× / >97% fewer; C3 26–33% fewer; win rates 72–83% / 72–80% comprehensiveness, 75–82% / 62–71% diversity | +| §5.2, Table 3 | Claimify 47,075 unique claims (≈31/answer); C0 News 34.18 vs SS 25.23 | + +- Companion notes in this topic: [README.md](README.md) — HippoRAG covers the local/associative + axis and Zep the temporal axis; GraphRAG covers the global-question axis. +- [experiments/](experiments/) — this repo's crate demonstrates the local path-finding side, not + community summarization; its measured headline is in [FINDINGS.md](../../FINDINGS.md) row 38. +- Leiden implementation used by the paper: graspologic. Local search (entity-anchored) is a + separate algorithm in the same system and is not the subject of this guide. diff --git a/topics/38-graphrag-agent-memory/reading-graphrag-sdk.md b/topics/38-graphrag-agent-memory/reading-graphrag-sdk.md index c81a9e1..7a83b64 100644 --- a/topics/38-graphrag-agent-memory/reading-graphrag-sdk.md +++ b/topics/38-graphrag-agent-memory/reading-graphrag-sdk.md @@ -1,12 +1,15 @@ # GraphRAG-SDK: the whole GraphRAG stack inside one FalkorDB round trip -Most GraphRAG stacks are Frankenstein architectures: a graph database for entities, a -separate vector database for embeddings, maybe an Elasticsearch for keyword search, and -application glue that pays a network round trip to each. FalkorDB's GraphRAG-SDK takes the -opposite bet — the knowledge graph, the vector indices, and the fulltext indices all live -inside FalkorDB itself, so entity discovery, vector search, and graph expansion are queries -against one database. This guide walks the code at commit `f42ab3d` (clone at -`~/repos/GraphRAG-SDK`), from the fixed ingestion pipeline to the multi-path retriever. +Most GraphRAG stacks are Frankenstein architectures: a graph database for entities, a separate +vector database for embeddings, maybe an Elasticsearch for keyword search, and application glue +that pays a network round trip to each. FalkorDB's GraphRAG-SDK takes the opposite bet — the +knowledge graph, the vector indices, and the fulltext indices all live inside FalkorDB itself, +so entity discovery, vector search, and graph expansion are queries against one database. This +guide walks the pinned source `FalkorDB/GraphRAG-SDK@f42ab3d`, package root +`graphrag_sdk/src/graphrag_sdk/`, from the fixed ingestion pipeline to the multi-path retriever. + +Every `file:line` below is quoted from that pinned commit; the anchors resolve against it +exactly. ## The problem in one sentence @@ -22,17 +25,39 @@ the entire RAG substrate — every design choice here is a workload you serve. ### Step 1 — One database, three index types -The architectural thesis lives in `storage/`. `GraphStore` (graph_store.py:21) wraps a -`FalkorDBConnection` and handles node/relationship upserts and traversals. `vector_store.py` -(:35) creates vector indices on entities (:104) and on relationships (:108), plus a fulltext -index (:485) — all as FalkorDB indices, not external services. +> **In:** a FalkorDB connection. +> **Out:** one instance exposing a property graph, vector indices, and a fulltext index — +> the substrate every later step queries. + +**FalkorDB** is a graph database that also serves vector and fulltext (RediSearch) indices, so +"three stores" become three index types in one engine. `GraphStore` +(`storage/graph_store.py:21`) wraps a `FalkorDBConnection` and handles node/relationship +upserts (`:56`, `:129`) and traversals (`:214`). `VectorStore` (`storage/vector_store.py`) +creates the vector index on entities (`:104`) and on relationships (`:108`) and the **fulltext** +index (`:133`) — all FalkorDB indices, not external services. The fulltext *creation* call is +`create_fulltext_index`, and it emits a native index-creation Cypher: + +```python +# storage/vector_store.py:133–148 — fulltext INDEX CREATION (not the query path) +133 async def create_fulltext_index( +134 self, +135 label: str = "Chunk", +136 *properties: str, +137 ) -> None: +144 if not properties: +145 properties = ("text",) +148 query = f"CALL db.idx.fulltext.createNodeIndex('{safe_label}', {props})" +``` + +Do not confuse this with `fulltext_search` at `:485`, which is the *query* side that calls +`db.idx.fulltext.queryNodes`. Creation lives at `:133`; search lives at `:485`. ``` +--------------------------- FalkorDB ---------------------------+ | | query --> | property graph vector indices fulltext index | - | (entities, chunks, (entity embeddings, (entity names, | - | MENTIONED_IN, relationship/RELATES chunk text) | + | (entities, chunks, (entity embeddings, (chunk text, | + | MENTIONED_IN, relationship/RELATES entity names) | | RELATES edges) embeddings) | | | +----------------------------------------------------------------+ @@ -41,15 +66,17 @@ index (:485) — all as FalkorDB indices, not external services. vs. the usual stack: app --> graph DB --> app --> vector DB --> app --> ES ``` -Contrast with stacks where a query fans out to three systems and reassembles in the -application tier. Here the join happens where the data is. +The join happens where the data is, not in the application tier. ### Step 2 — A fixed 9-step ingestion pipeline -`IngestionPipeline` (ingestion/pipeline.py:35, `run` at :94) hard-codes the step order — -it is not a configurable DAG. The sequence: Load → Chunk → lexical graph → Extract → -quality filter (:329) → Prune (:282) → Resolve → Write → then Mentions and Index run -concurrently via `asyncio.gather` (:175). +> **In:** raw documents. +> **Out:** a populated FalkorDB graph — the corpus every retrieval step reads. + +`IngestionPipeline` (`ingestion/pipeline.py:35`, `run` at `:94`) hard-codes the step order — it +is **not a configurable DAG**. The sequence: Load → Chunk → lexical graph → Extract → quality +filter (`:329`) → Prune (`:282`) → Resolve → Write → then Mentions and Index run **concurrently** +via `asyncio.gather` (`:175`). ``` Load -> Chunk -> LexicalGraph -> Extract -> QualityFilter -> Prune -> Resolve -> Write @@ -59,70 +86,146 @@ concurrently via `asyncio.gather` (:175). Mentions Index ``` -The lexical-graph step is mandatory: chunk and document nodes plus MENTIONED_IN edges are -always built, even when entity extraction is disabled. You always get a retrievable corpus -graph; entities are the optional enrichment, not the foundation. +A **lexical graph** here means chunk and document nodes joined by `MENTIONED_IN` edges — the +plain text-retrieval layer, separate from the extracted entity graph. It is mandatory: those +nodes and edges are always built, even when entity extraction is disabled. You always get a +retrievable corpus graph; entities are the optional enrichment, not the foundation. ### Step 3 — Two-step extraction: local NER first, LLM second -`GraphExtraction` (extraction_strategies/graph_extraction.py:89) splits extraction into a -pluggable NER pass (default `GLiNERExtractor`, a local model — no API call) and an LLM pass -that verifies the entities and extracts relations. This is the same shape as HippoRAG's -2-step OpenIE (see [reading-hipporag.md](reading-hipporag.md)): a cheap recall-oriented -first pass, an expensive precision-oriented second pass. +> **In:** text chunks from Step 2. +> **Out:** entities and relations, merged across chunks — the raw material Step 4 resolves. + +`GraphExtraction` (`ingestion/extraction_strategies/graph_extraction.py:89`) splits extraction +into a pluggable **NER** (named-entity recognition) pass and an LLM pass — its own docstring +states the contract: + +```python +# ingestion/extraction_strategies/graph_extraction.py:89–98 — the 2-step contract +89 class GraphExtraction(ExtractionStrategy): +90 """Composable 2-step extraction with pluggable entity NER. +92 **Step 1** — Entity extraction via a pluggable ``EntityExtractor``. +93 Default: ``GLiNERExtractor`` (local, no API calls). +96 **Step 2** — LLM verification + relationship extraction. The LLM +97 receives the pre-extracted entities and original text, verifies +98 entities, and extracts relationships. +``` -Two details worth reading closely. First, budget awareness: `ctx.budget_exceeded` is -checked at :148, so extraction degrades gracefully when the token budget runs out instead -of failing the pipeline. Second, cross-chunk merging: `_aggregate_entities` (:438) and -`_aggregate_relations` (:479) fold duplicates found in different chunks before resolution -even starts. Coreference resolution is optional. +This is the same shape as HippoRAG's two-step OpenIE (see +[reading-hipporag.md](reading-hipporag.md)): a cheap recall-oriented first pass (here a *local* +GLiNER model, no API call), an expensive precision-oriented second pass. Two details worth +reading closely. First, **budget awareness**: `ctx.budget_exceeded` is checked at `:148`, so +extraction degrades gracefully when the token budget runs out instead of failing the pipeline. +Second, **cross-chunk merging**: `_aggregate_entities` (`:438`) and `_aggregate_relations` +(`:479`) fold duplicates found in different chunks before resolution even starts. ### Step 4 — A ladder of resolution strategies, cheap to expensive -`resolution_strategies/` is an escalation ladder — each rung costs more and catches more: +> **In:** the aggregated entities from Step 3. +> **Out:** merged entities — fewer, canonical nodes for Step 5 to dedup and write. + +**Entity resolution** decides when two mentions are the same entity. `ingestion/ +resolution_strategies/` is an escalation ladder — each rung costs more and catches more: ``` cost - ^ llm_verified_resolution.py:75 embedding candidates -> LLM-verified merge (:192) - | semantic_resolution.py:33 embedding similarity, _fuzzy_merge (:122) - | description_merge.py:30 merge descriptions of same-name entities - | exact_match.py:21 string equality + ^ llm_verified_resolution.py:75 embedding candidates -> LLM-verified merge (:192) + | semantic_resolution.py:33 embedding similarity, _fuzzy_merge (:122) + | description_merge.py:30 merge descriptions of same-name entities + | exact_match.py:21 string equality +------------------------------------------------------------------> recall ``` -The top rung — `_embedding_and_llm_merge` (llm_verified_resolution.py:192) — searches for -merge candidates by embedding, then asks an LLM to confirm each merge. That is precisely -the escalation pattern Zep/Graphiti uses for entity resolution -(see [reading-zep-graphiti.md](reading-zep-graphiti.md)); it shows up independently in -every serious KG-construction system because embeddings alone over-merge. +The top rung — `_embedding_and_llm_merge` (`ingestion/resolution_strategies/ +llm_verified_resolution.py:192`, inside `LLMVerifiedResolution` at `:75`) — searches for merge +candidates by embedding, then asks an LLM to confirm each merge. That is precisely the +escalation pattern Zep/Graphiti uses for entity resolution (see +[reading-zep-graphiti.md](reading-zep-graphiti.md)); it recurs in every serious KG-construction +system because embeddings alone over-merge (near-duplicate embeddings for genuinely distinct +entities). ### Step 5 — Deduplication with the survivor pattern -`EntityDeduplicator` (storage/deduplicator.py:35) runs exact dedup (:74) and fuzzy dedup -(:115). The interesting graph-database mechanics are in `_remap_entity_edges` (:228): when -two nodes merge, the loser's edges are re-pointed to the surviving node *before* the loser -is deleted. Miss that ordering and every merge silently drops relationships — this is the -kind of invariant your database's users depend on getting right in application code. +> **In:** resolved entity groups from Step 4. +> **Out:** a graph with duplicates removed and no dangling edges — Step 2's Write made safe. + +`EntityDeduplicator` (`storage/deduplicator.py`) runs exact dedup (`:74`) and fuzzy dedup +(`:115`). The load-bearing graph mechanic is the **survivor pattern**: re-point the loser's +edges onto the survivor *before* deleting the loser, and only delete if the remap succeeded. + +```python +# storage/deduplicator.py:97–105 — remap-then-delete, guarded +97 for dup in duplicates: +98 if not await self._remap_entity_edges(dup["id"], survivor["id"]): +99 logger.warning(f"Skipping deletion of {dup['id']} — edge remap incomplete") +100 continue +101 try: +102 await self._graph.query_raw( +103 "MATCH (e:__Entity__ {id: $dup_id}) DETACH DELETE e", +104 {"dup_id": dup["id"]}, +105 ) +``` + +`_remap_entity_edges` (`:228`) re-points the duplicate's `RELATES` and `MENTIONED_IN` edges to +the survivor and returns `False` on any failure; the guard at `:98–100` then *skips* the +`DETACH DELETE`. Miss that ordering (or the guard) and a merge silently drops relationships — +exactly the invariant a graph database's users depend on getting right. ### Step 6 — Rule-based routing: no LLM in the hot path -`SemanticRouter` (retrieval/router.py:19) picks a retrieval strategy with plain rules — -first match wins (`_select`, :84), falling back to a default strategy. No LLM call means -routing costs microseconds, is deterministic, and is debuggable. Supporting classifiers are -also rule-based: `is_enumeration_query` (retrieval/entity_discovery.py:20) and -`detect_question_type` (retrieval/result_assembly.py:105). +> **In:** a query string. +> **Out:** the chosen retrieval strategy — selected without an LLM call. + +`SemanticRouter` (`retrieval/router.py:19`) picks a retrieval strategy with plain rules — +first match wins, default fallback. Its own code says so; it is **not** an embedding-driven +router: + +```python +# retrieval/router.py:19–23 and 84–98 — rule-based FIRST-MATCH, not embeddings +19 class SemanticRouter: +20 """Route queries to the best retrieval strategy based on intent. +22 In v1, this is a simple rule-based router. Users register strategies +23 with keywords or conditions, and the router picks the best match. +84 def _select(self, query: str) -> tuple[str, RetrievalStrategy]: +90 for name, (strategy, condition) in self._strategies.items(): +92 if callable(condition) and condition(query): +93 return name, strategy +98 return "default", self._default +``` + +No LLM call means routing costs microseconds, is deterministic, and is debuggable. The +supporting classifiers are rule-based too: `is_enumeration_query` +(`retrieval/strategies/entity_discovery.py:20`) and `detect_question_type` +(`retrieval/strategies/result_assembly.py:105`). ### Step 7 — Multi-path retrieval: four chunk paths, one reranker -`MultiPathRetrieval` (retrieval/strategies/multi_path.py:48) is the flagship, itself a -9-step sequence: (1) keyword extraction (stopword filter + LLM), (2) embed the query ONCE -and reuse it everywhere, (3) vector search over RELATES relationship embeddings, (4) entity -discovery via Cypher CONTAINS + fulltext, (5) 1-hop and 2-hop graph expansion from found -entities, (6) chunk retrieval over four parallel paths, (7) source-document fetch, -(8) cosine rerank, (9) context assembly. +> **In:** a routed query (from Step 6) and the FalkorDB graph. +> **Out:** an assembled context block for the LLM. + +`MultiPathRetrieval` (`retrieval/strategies/multi_path.py:48`) is the flagship, itself a 9-step +sequence — read it from the docstring: + +```python +# retrieval/strategies/multi_path.py:48–63 — the retrieval pipeline docstring +48 class MultiPathRetrieval(RetrievalStrategy): +52 Retrieval pipeline: +53 1. Keyword extraction (stopword filter + LLM proper nouns) +54 2. Embed question only (single API call) +55 3. RELATES edge vector search -> fact strings + entity entry points +56 4. Entity discovery (2 paths: Cypher CONTAINS, fulltext) +58 5. Relationship expansion (1-hop + 2-hop from top entities) +59 6. Chunk retrieval (4 paths: fulltext, vector, MENTIONED_IN, 2-hop) +61 8. Cosine reranking of all candidate chunks +62 9. Context assembly into structured sections (...) +``` + +The **four chunk paths** (step 6 of the docstring) — fulltext, vector on the chunk index, +`MENTIONED_IN` from found entities, and 2-hop expansion chunks — are all queries against the +same FalkorDB, which is Step 1's thesis cashing out: ``` - query (embedded once) + query (embedded ONCE at docstring step 2) | +----------+--------+---------+------------------+ | | | | @@ -136,84 +239,164 @@ entities, (6) chunk retrieval over four parallel paths, (7) source-document fetc context assembly ``` -Defaults: chunk_top_k=15, max_entities=30, max_relationships=20, rel_top_k=15, -keyword_limit=10. All four chunk paths are queries against the same FalkorDB — Step 1's -thesis cashing out. The reranker is `CosineReranker` (reranking_strategies/cosine.py:18). - -Note what is absent: there is no community-summary layer, no global axis in the Microsoft -GraphRAG sense (see [reading-graphrag-paper.md](reading-graphrag-paper.md)). The SDK's -retrieval is the local/associative kind — start from entities, expand, rerank. +Defaults (the `__init__` signature, `multi_path.py:163–167`): `chunk_top_k=15`, +`max_entities=30`, `max_relationships=20`, `rel_top_k=15`, `keyword_limit=10`. The query +embedding is computed once (docstring step 2) and reused across the vector paths. The reranker is `CosineReranker` +(`retrieval/reranking_strategies/cosine.py:18`, `top_k` default 15). Note what is absent: there +is no community-summary layer, no global axis in the Microsoft GraphRAG sense (see +[reading-graphrag-paper.md](reading-graphrag-paper.md)). The SDK's retrieval is the +local/associative kind — start from entities, expand, rerank. ### Step 8 — Text-to-Cypher exists, but behind a guard rail -`enable_cypher=False` by default: the text-to-Cypher path is experimental and off. When -enabled, `retrieval/cypher_generation.py` runs generated queries through `extract_cypher` -(:145), `_sanitize_cypher` (:162), and `validate_cypher` (:187) before execution. LLM-written -Cypher hitting a production graph gets extracted, sanitized, and validated — a sensible -posture for anyone who has watched an LLM hallucinate a Cartesian product. +> **In:** a natural-language query, when `enable_cypher=True`. +> **Out:** a validated read-only Cypher query — or rejection before execution. + +`enable_cypher=False` by default: the text-to-Cypher path is experimental and off. When enabled, +`retrieval/strategies/cypher_generation.py` runs generated queries through three guards before +execution: + +```python +# retrieval/strategies/cypher_generation.py:145,162,187 — extract, sanitize, validate +145 def extract_cypher(text: str) -> str: +146 """Extract Cypher from LLM response, handling markdown code blocks.""" +162 def _sanitize_cypher(cypher: str) -> str: +178 if not re.search(r"\bLIMIT\b", cypher, re.IGNORECASE): +179 cypher = cypher.rstrip().rstrip(";") + "\nLIMIT 25" +187 def validate_cypher(cypher: str) -> list[str]: +190 """Uses an allowlist approach: the query must start with a read-only +191 keyword, and dangerous constructs are explicitly rejected.""" +``` + +LLM-written Cypher hitting a production graph gets extracted from the markdown, sanitized (a +`LIMIT 25` is injected if missing, to prevent runaway scans; `:178`), and validated against a +read-only allowlist (`:190`) before execution — a sensible posture for anyone who has watched an +LLM hallucinate a Cartesian product. ## Where each step lives in the code -Paths relative to `graphrag_sdk/src/graphrag_sdk/`. +Paths relative to `graphrag_sdk/src/graphrag_sdk/`, at `FalkorDB/GraphRAG-SDK@f42ab3d`. | Step | Anchor (file:line) | What to see | |---|---|---| | 1 | storage/graph_store.py:21 | `GraphStore` wrapping FalkorDBConnection | | 1 | storage/graph_store.py:56, :129, :214 | `upsert_nodes`, `upsert_relationships`, `get_connected_entities` | -| 1 | storage/vector_store.py:104, :108, :485 | entity/relationship vector indices + fulltext, all in FalkorDB | -| 1 | storage/vector_store.py:169, :244 | `index_chunks`, `embed_relationships` | +| 1 | storage/vector_store.py:104, :108, :133 | entity/relationship vector indices + `create_fulltext_index` | +| 1 | storage/vector_store.py:169, :244, :485 | `index_chunks`, `embed_relationships`, `fulltext_search` (query, not creation) | | 2 | ingestion/pipeline.py:35, :94 | `IngestionPipeline` and `run` — the fixed 9 steps | -| 2 | ingestion/pipeline.py:329, :282, :175 | quality filter, prune, concurrent Mentions+Index via gather | -| 3 | extraction_strategies/graph_extraction.py:89 | `GraphExtraction` — NER pass then LLM pass | -| 3 | extraction_strategies/graph_extraction.py:148 | `ctx.budget_exceeded` graceful degradation | -| 3 | extraction_strategies/graph_extraction.py:438, :479 | `_aggregate_entities`, `_aggregate_relations` | -| 4 | resolution_strategies/exact_match.py:21 | cheapest rung: string equality | -| 4 | resolution_strategies/semantic_resolution.py:33, :122 | embedding similarity, `_fuzzy_merge` | -| 4 | resolution_strategies/llm_verified_resolution.py:75, :192 | `_embedding_and_llm_merge` escalation | +| 2 | ingestion/pipeline.py:329, :282, :175 | quality filter, prune, concurrent Mentions+Index via `asyncio.gather` | +| 3 | ingestion/extraction_strategies/graph_extraction.py:89 | `GraphExtraction` — NER pass then LLM pass | +| 3 | ingestion/extraction_strategies/graph_extraction.py:148 | `ctx.budget_exceeded` graceful degradation | +| 3 | ingestion/extraction_strategies/graph_extraction.py:438, :479 | `_aggregate_entities`, `_aggregate_relations` | +| 4 | ingestion/resolution_strategies/exact_match.py:21 | cheapest rung: string equality | +| 4 | ingestion/resolution_strategies/description_merge.py:30 | same-name description merge | +| 4 | ingestion/resolution_strategies/semantic_resolution.py:33, :122 | embedding similarity, `_fuzzy_merge` | +| 4 | ingestion/resolution_strategies/llm_verified_resolution.py:75, :192 | `LLMVerifiedResolution`, `_embedding_and_llm_merge` | | 5 | storage/deduplicator.py:74, :115, :228 | exact/fuzzy dedup, `_remap_entity_edges` survivor pattern | | 6 | retrieval/router.py:19, :84 | `SemanticRouter`, first-match `_select` | +| 6 | retrieval/strategies/entity_discovery.py:20 | `is_enumeration_query` (rule-based) | +| 6 | retrieval/strategies/result_assembly.py:105 | `detect_question_type` (rule-based) | | 7 | retrieval/strategies/multi_path.py:48 | `MultiPathRetrieval` — the 9-step flagship | | 7 | storage/vector_store.py:371, :406 | `search_entities`, `search_relationships` | -| 7 | reranking_strategies/cosine.py:18 | `CosineReranker` (top_k=15) | -| 8 | retrieval/cypher_generation.py:145, :162, :187 | `extract_cypher`, `_sanitize_cypher`, `validate_cypher` | +| 7 | retrieval/reranking_strategies/cosine.py:18 | `CosineReranker` (top_k=15) | +| 8 | retrieval/strategies/cypher_generation.py:145, :162, :187 | `extract_cypher`, `_sanitize_cypher`, `validate_cypher` | ## Questions to answer in notes.md -1. The multi-path retriever issues fulltext, vector, MENTIONED_IN, and 2-hop chunk queries - against one FalkorDB. Which of the four is the latency bottleneck at scale, and could - they be fused into fewer Cypher round trips? -2. The lexical graph (chunk/document nodes + MENTIONED_IN) is built even with entity - extraction disabled. What retrieval quality do you keep in that degenerate mode, and - what breaks? -3. `_remap_entity_edges` re-points the loser's edges before deleting the node. What are the - atomicity guarantees during that window, and how would you implement merge-with-remap as - a single server-side operation in FalkorDB? -4. The resolution ladder runs exact → description-merge → semantic → LLM-verified. Which - rung dominates wall-clock time on a large ingest, and where would caching or batching - help most? -5. Routing is rule-based first-match with a default fallback. What query shapes fall - through to the default today, and would an LLM router earn its latency cost for any of - them? +1. The multi-path retriever issues fulltext, vector, `MENTIONED_IN`, and 2-hop chunk queries + against one FalkorDB. Which of the four is the latency bottleneck at scale, and could they be + fused into fewer Cypher round trips? +2. The lexical graph (chunk/document nodes + `MENTIONED_IN`) is built even with entity + extraction disabled. What retrieval quality do you keep in that degenerate mode, and what + breaks? +3. `_remap_entity_edges` re-points the loser's edges before deleting the node, and the delete is + skipped if the remap fails. What atomicity gap remains between the remap and the delete, and + how would you make merge-with-remap a single server-side FalkorDB operation? +4. The resolution ladder runs exact → description-merge → semantic → LLM-verified. Which rung + dominates wall-clock time on a large ingest, and where would caching or batching help most? +5. Routing is rule-based first-match with a default fallback. What query shapes fall through to + the default today, and would an LLM router earn its latency cost for any of them? ## Done when -- [ ] You can sketch the fixed 9-step ingestion pipeline from memory, including which step - is mandatory and which two run concurrently at the tail. -- [ ] You have traced one document through `IngestionPipeline.run` (pipeline.py:94) in the - clone and watched the lexical graph land in FalkorDB. -- [ ] You can explain the four chunk-retrieval paths in `MultiPathRetrieval` and where the - query embedding is computed and reused. -- [ ] You can state why the resolution ladder escalates to `_embedding_and_llm_merge` and - how it relates to Zep/Graphiti's approach. +Answer each before unfolding it. + +- [ ] You can sketch the fixed 9-step ingestion pipeline from memory, including which step is mandatory and which two run concurrently at the tail. + +
Answer + + Load → Chunk → **lexical graph (mandatory)** → Extract → Quality filter (`pipeline.py:329`) → + Prune (`:282`) → Resolve → Write → then **Mentions and Index run concurrently** via + `asyncio.gather` (`:175`). The order is hard-coded in `IngestionPipeline.run` (`:94`), not a + configurable DAG. The lexical-graph step (chunk/document nodes + `MENTIONED_IN`) always runs, + so even with entity extraction off you get a retrievable corpus graph. + +
+ +- [ ] You have traced one document through `IngestionPipeline.run` (pipeline.py:94) and watched the lexical graph land in FalkorDB. + +
Answer + + Following `run` (`ingestion/pipeline.py:94`): the document is loaded and chunked, the lexical + graph writes chunk/document nodes and `MENTIONED_IN` edges via `GraphStore.upsert_nodes` + (`graph_store.py:56`) and `upsert_relationships` (`:129`), extraction (Step 3) optionally adds + entities/relations, quality filter (`:329`) and prune (`:282`) trim them, resolution merges + duplicates, Write persists, and finally Mentions + Index run together (`:175`). The chunk and + document nodes are queryable in FalkorDB regardless of whether extraction ran. + +
+ +- [ ] You can explain the four chunk-retrieval paths in `MultiPathRetrieval` and where the query embedding is computed and reused. + +
Answer + + From the docstring (`multi_path.py:48–63`): the query is embedded **once** (docstring step 2, + "Embed question only (single API call)") and reused across the vector paths. Chunk retrieval + (docstring step 6) runs **four paths**: **fulltext** (RediSearch), **vector** on the chunk + index, **`MENTIONED_IN`** from discovered entities, and **2-hop expansion** chunks. All four + are queries against the same FalkorDB. Candidates are then cosine-reranked (`CosineReranker`, + `retrieval/reranking_strategies/cosine.py:18`, top_k=15) and assembled into a context block. + +
+ +- [ ] You can state why the resolution ladder escalates to `_embedding_and_llm_merge` and how it relates to Zep/Graphiti's approach. + +
Answer + + The ladder runs exact-match (`exact_match.py:21`) → description-merge (`description_merge.py:30`) + → semantic/embedding (`semantic_resolution.py:33`) → LLM-verified + (`llm_verified_resolution.py:75`). It escalates to `_embedding_and_llm_merge` (`:192`) because + **embeddings alone over-merge**: near-identical vectors for genuinely distinct entities would + collapse them, so the LLM is used as a precision gate on the embedding-found candidates. That + is exactly Zep/Graphiti's entity resolution (embedding candidate search, then LLM verify — see + [reading-zep-graphiti.md](reading-zep-graphiti.md)); the pattern recurs across serious KG + builders. + +
+ - [ ] You have answered the 5 questions above in notes.md. +
Answer + + notes.md records all five with code anchors: the four-path latency bottleneck and whether the + paths can be fused into fewer Cypher round trips (`multi_path.py:48`); what survives in the + extraction-disabled lexical-graph mode and what breaks (Step 2); the residual atomicity gap + between `_remap_entity_edges` and the guarded `DETACH DELETE` (`deduplicator.py:98–105`); + which resolution rung dominates ingest wall-clock and where batching/caching helps (Step 4); + and which query shapes fall through to the router default and whether an LLM router would pay + for itself (`router.py:84–98`). + +
+ ## References -- Clone: `~/repos/GraphRAG-SDK` at commit `f42ab3d`; package under - `graphrag_sdk/src/graphrag_sdk/`. -- [reading-hipporag.md](reading-hipporag.md) — the 2-step OpenIE that mirrors the SDK's +- Pinned source: `FalkorDB/GraphRAG-SDK@f42ab3d`; package under + `graphrag_sdk/src/graphrag_sdk/`. Every `file:line` in this chapter resolves against that + commit (verify with `tools/pinned-source.py show GraphRAG-SDK -r A:B`). +- [reading-hipporag.md](reading-hipporag.md) — the two-step OpenIE that mirrors the SDK's NER-then-LLM extraction. -- [reading-zep-graphiti.md](reading-zep-graphiti.md) — the embedding-then-LLM entity - resolution escalation the SDK's top rung reuses. -- [reading-graphrag-paper.md](reading-graphrag-paper.md) — Microsoft GraphRAG's - community-summary global axis, which this SDK deliberately does not have. +- [reading-zep-graphiti.md](reading-zep-graphiti.md) — the embedding-then-LLM entity resolution + escalation the SDK's top rung reuses. +- [reading-graphrag-paper.md](reading-graphrag-paper.md) — Microsoft GraphRAG's community-summary + global axis, which this SDK deliberately does not have. +- This topic's measured headline is in [FINDINGS.md](../../FINDINGS.md) row 38. diff --git a/topics/38-graphrag-agent-memory/reading-hipporag.md b/topics/38-graphrag-agent-memory/reading-hipporag.md index 885a193..91e3066 100644 --- a/topics/38-graphrag-agent-memory/reading-hipporag.md +++ b/topics/38-graphrag-agent-memory/reading-hipporag.md @@ -9,6 +9,10 @@ graph-database developer the payoff is concrete: multi-hop retrieval becomes one query instead of N LLM calls, and the paper measures the difference in recall, dollars, and latency. +Every number below is quoted with the table, figure or section it comes from in the NeurIPS +2024 version of the paper (arXiv 2405.14831). Where a figure is this repo's own measurement it +says so and links [FINDINGS.md](../../FINDINGS.md). + ## The problem in one sentence **Single-step retrievers and iterative LLM retrievers both fail path-finding multi-hop @@ -16,15 +20,27 @@ questions — questions whose answer is the one entity associated with several q when no single passage mentions them together — because they chain lookups instead of aggregating association strength across a corpus-wide index.** -Path-following questions can be solved hop by hop (each hop's answer names the next clue). -Path-finding questions cannot: the connection only exists as a pattern across passages, so -retrieval must complete the pattern, not follow a chain. +Define the two shapes the paper contrasts (§1, Figure 1): + +- A **path-following** multi-hop question can be solved hop by hop — each hop's answer *names* + the next clue, so a retriever that finds passage 1 finds the bridge entity that leads to + passage 2. +- A **path-finding** multi-hop question cannot: the answer is the entity that several query + entities *both* point at, and no single passage names them together, so there is no chain + to follow — retrieval must complete the pattern across passages instead. ## The concepts, step by step ### Step 1 — The hippocampal memory indexing analogy -The theory separates where memories live from how they are found. HippoRAG maps each brain +> **In:** nothing yet — this step is the motivation and fixes the vocabulary. +> **Out:** the split — a content store versus a sparse association index — that every later +> step implements. + +The **hippocampal memory indexing theory** (§2.1) is a model of human memory in which the +neocortex stores the actual content of memories while the hippocampus stores only a sparse +*index* of associations between them; recall is **pattern completion** — retrieving a whole +memory from a partial cue by spreading activation through that index. HippoRAG maps each brain region to a system component: ``` @@ -37,126 +53,239 @@ region to a system component: stores NO content, only links ``` -The key claim: the hippocampus stores no content, only the index. Pattern completion — -retrieving a whole memory from a partial cue — happens by spreading activation through the -index. HippoRAG implements exactly that split. +The load-bearing claim (§2.1): the hippocampus stores no content, only the index. That is why +retrieval can be one graph operation rather than a scan of the corpus — the index is small and +the association is explicit. HippoRAG implements exactly that split, and Steps 2–7 are its two +halves: build the index offline, complete the pattern online. ### Step 2 — Offline indexing: two-step OpenIE into a schemaless graph -For each passage, an LLM (GPT-3.5-turbo-1106, temperature 0) performs one-shot OpenIE in two -steps: NER first, then triple extraction with the extracted entities included in the prompt. -The result is a schemaless knowledge graph whose nodes are noun phrases — no fixed ontology, -no entity resolution pipeline. Two artifacts matter downstream: +> **In:** the passage corpus P. +> **Out:** the knowledge graph — nodes N (noun phrases), triple edges E, and synonymy edges +> E′ — which Step 6's PPR walks. (Step 3 derives the other offline artifact from the same +> extraction.) + +**OpenIE** (open information extraction) means pulling `(subject, relation, object)` triples +out of free text without a fixed schema. HippoRAG runs it with an LLM (GPT-3.5-turbo-1106, +temperature 0; §3.4) in **two prompted steps** (§2.3, 1-shot): + +1. **NER** (**named-entity recognition** — find the entity mentions in the passage) extracts a + set of named entities. +2. those entities are pasted into a second prompt that extracts the final triples, which may + also contain concepts (noun phrases) beyond the named entities. + +The paper's stated reason for the two-step order: it "leads to an appropriate balance between +generality and bias towards named entities" (§2.3). The result is a **schemaless** graph — +nodes are raw noun phrases, with no fixed ontology and no entity-resolution pipeline. + +One more edge type is welded on. A **synonymy edge** connects two nodes whose embedding cosine +similarity exceeds a threshold **τ = 0.8** (§3.4) — the parahippocampal analogue of Step 1, +cheap fuzzy matching turned into graph structure so the walk can cross surface-form differences +("JFK" ↔ "John F. Kennedy"). Table A in the appendix reports tens of thousands of these E′ +edges per corpus; they are ablated in Step 9. -- Synonymy edges: connect nodes whose embedding cosine similarity exceeds τ = 0.8. This is - the parahippocampal analogue — cheap fuzzy matching welded into graph structure. -- Matrix P (|N|×|P|): counts node-in-passage occurrences, linking index nodes back to the - content they came from. +### Step 3 — The fork: matrix P, the node→passage index -### Step 3 — Online retrieval: one PPR query, no LLM loop +> **In:** the same OpenIE output from Step 2 — this is the *other* thing built from it. +> **Out:** a `|N| × |P|` count matrix P that Step 7 multiplies against the PPR distribution to +> score passages. Nothing else uses it. + +Extraction produces two artifacts, and they feed different downstream steps — so the matrix +gets its own step. **Matrix P** (§2.3) is `|N| × |P|` (nodes × passages) and holds "the number +of times each noun phrase in the KG appears in each original passage." It is the only thing +that links an index node back to the *content* it was extracted from; the KG of Step 2 has no +passage text in it at all (Step 1's "index stores no content"). ``` - query --LLM NER--> named entities --embedding cosine--> query nodes R_q - | - restart mass ONLY on R_q - v - Personalized PageRank (damping 0.5) - | - stationary π⃗ - v - passage score = π⃗ · P --> top-k passages + passage₁ passage₂ passage₃ ... + node_a 2 0 1 + node_b 0 3 0 P[i][j] = # times node i appears in passage j + node_c 1 1 0 ``` -Extract named entities from the query with the LLM, map each to graph nodes by embedding -cosine, then run PPR with restart mass only on those query nodes and damping factor 0.5. -Passage score is π⃗·P — PPR mass summed over the nodes each passage mentions. A single graph -query replaces the per-hop LLM calls of iterative retrievers. +Keep the fork in mind: the **KG (Step 2)** is what PPR walks; **P (this step)** is what turns a +distribution over nodes into a ranking over passages. Step 7 is where they rejoin. + +### Step 4 — Online retrieval: from query to seed nodes R_q + +> **In:** a query string, plus the KG from Step 2. +> **Out:** the seed set R_q — the graph nodes PPR will restart from — which Step 5 then +> re-weights. + +Online retrieval starts with one LLM call: the same NER prompt extracts the query's named +entities Cq = {c₁, …, cₙ} (§2.3 — "Stanford" and "Alzheimer's" in the paper's Figure 2 +example). Each cᵢ is embedded and matched to its nearest graph node by cosine similarity, giving +the **query nodes** (the seeds) R_q = {r₁, …, rₙ}, where rᵢ = arg max over graph nodes eⱼ of +`cosine(M(cᵢ), M(eⱼ))`. These are the only nodes PPR will inject restart mass into — nothing +else is seeded. + +This is the whole "single-step" claim in miniature: one LLM call to name the query's entities, +then a single graph computation. There is no second LLM call to decide the next hop. + +### Step 5 — Node specificity: local IDF weighting of the seeds + +> **In:** the seed set R_q from Step 4, and the passage counts behind P from Step 3. +> **Out:** the restart vector n⃗ — the seeds' probabilities re-weighted so common entities get +> less mass — which Step 6 restarts from. -### Step 4 — Why PPR solves path-finding: mass sums at the meet node +Left alone, a seed like "USA" would flood the graph with restart mass because it connects to +everything. **Inverse document frequency (IDF)** is the classic fix — weight a term by the +inverse of how many documents contain it, so common terms count for less — but true IDF needs a +global corpus count. HippoRAG uses a local stand-in. **Node specificity** of node i is -Consider the paper's case study: "Which Stanford professor works on the neuroscience of -Alzheimer's?" — no passage mentions both Stanford and Alzheimer's together with the answer. +``` + sᵢ = |Pᵢ|⁻¹ (§2.3) + |Pᵢ| = the number of passages node i was extracted from — a count already + stored at the node, so no global corpus statistic is needed. ``` - [Stanford] ----seed----. .----seed---- [Alzheimer's] - \ \ / / - (edges to mass flows mass flows (edges to - other nodes) \ / other nodes) - v v - [Thomas Südhof] <-- PPR mass from BOTH seeds SUMS here - | - passages mentioning him rank first + +It is used by multiplying each query node's restart probability n⃗ by sᵢ before PPR (§2.3). +Worked on two seeds, one common and one rare: + ``` + seed "Alzheimer's" in |P| = 20 passages → s = 1/20 = 0.05 + seed "Stanford" in |P| = 5 passages → s = 1/5 = 0.20 + + raw restart (equal split): n(Alz) = 0.50 n(Stan) = 0.50 + after × sᵢ: 0.50·0.05 = 0.025 0.50·0.20 = 0.100 + renormalize (÷ 0.125): n(Alz) = 0.20 n(Stan) = 0.80 +``` + +The rarer, more discriminating seed keeps 4× the restart mass of the common one — exactly the +Figure 2 illustration where "the Stanford logo grows larger than the Alzheimer's symbol since +it appears in fewer documents" (§2.3). The database-friendly property: adding a passage changes +only the counts of the nodes it mentions, so specificity is maintainable incrementally, which +true IDF is not. + +### Step 6 — PPR: why mass sums at the meet node -Mass restarts at both query entities and diffuses along edges; the one node connected to both -accumulates the sum and outranks every node reachable from only one seed. ColBERTv2 and IRCoT -both fail this example; HippoRAG ranks Thomas Südhof first. This is association, not chaining -— the structural reason iterative retrieval cannot fix the problem by looping harder. +> **In:** the weighted restart vector n⃗ from Step 5, walking over the KG from Step 2. +> **Out:** a stationary distribution π⃗ (the paper's n⃗′) over all nodes — high on nodes near +> *several* seeds — which Step 7 scores passages with. -### Step 5 — Node specificity: local IDF without global statistics +**PageRank** is the stationary distribution of a random walk that, at each step, either follows +a random out-edge or teleports; **Personalized PageRank (PPR)** replaces the uniform teleport +with a fixed **restart vector** so the walk keeps returning to a chosen set of seeds. The paper +sets the **damping factor** to 0.5, which it defines (§3.4) as "the probability that PPR will +restart a random walk from the query nodes instead of continuing to explore the graph" — i.e. +restart probability 0.5, continue probability 0.5. The seeds are R_q, weighted by Step 5. -Common nodes ("USA") would flood the graph with restart mass. HippoRAG multiplies each query -node's restart probability by sᵢ = |Pᵢ|⁻¹ — the inverse of the number of passages mentioning -node i. It behaves like IDF but is computable per node without global corpus statistics, which -the authors argue is neurobiologically plausible and which a database developer will recognize -as an update-friendly property: adding a passage touches only the counts of the nodes it -mentions. +Why does this solve a path-finding question? Because the one node reachable from *both* seeds +collects restart-driven mass along *two* inflows, while every dead-end collects from one. +Worked on the README's instance shape — two seeds u, w, each of degree 9 (one edge to the +answer a, eight to distractor dead-ends), restart 0.5 split equally over the two seeds: -### Step 6 — The two-pipeline view: index once, query cheap +``` + restart vector r: r(u) = 0.5, r(w) = 0.5, everything else 0 + one PPR step (restart prob 0.5, continue prob 0.5, seed degree 9): + + walk-mass into a from u = 0.5 · r(u)/deg(u) = 0.5 · 0.5/9 = 0.0278 + walk-mass into a from w = 0.5 · r(w)/deg(w) = 0.5 · 0.5/9 = 0.0278 + π(a) walk-mass = 0.0278 + 0.0278 = 0.0556 ← sums from BOTH + π(dead-end from u only) = 0.5 · 0.5/9 = 0.0278 ← ONE source + + ratio a : dead-end = 0.0556 / 0.0278 = 2.0 +``` + +The answer node ends one iteration ahead at ≈ 2× a dead-end's mass — the 2× gap README exercise +2 asks you to derive. The full stationary values are larger (the walk keeps circulating), but +the ordering holds because a is the only node collecting from both seeds at *every* iteration. +This is association, not chaining: on the paper's real example — "Which Stanford professor works +on the neuroscience of Alzheimer's?" (Table 7, §5.3) — ColBERTv2 and IRCoT return the wrong +people, while HippoRAG ranks the correct answer, **Thomas Südhof**, first. (The Figure 1 / §2.3 +walkthrough abbreviates that same entity as "Professor Thomas".) Looping an iterative retriever +harder cannot fix it, because the connection never lives in a single passage for a hop to land +on. + +### Step 7 — Passage scoring: π⃗ · P + +> **In:** the PPR distribution π⃗ from Step 6 and the matrix P from Step 3 — the fork rejoins +> here. +> **Out:** one score per passage, ranked; the top-k are returned. + +The final step multiplies the node distribution by the node→passage matrix: **passage score +p⃗ = π⃗ · P** (§2.3). Concretely, each passage's score is the PPR mass summed over the nodes it +mentions, weighted by how often it mentions them. A passage that mentions "Professor Thomas" — +the node Step 6 pushed mass onto — rises to the top even though it never mentions both query +entities. One matrix-vector product turns "which nodes matter" into "which passages to read". + +### Step 8 — The two-pipeline view: index once, query cheap + +> **In:** the offline and online halves as built in Steps 2–7. +> **Out:** the cost argument — LLM work at index time, graph work at query time. ``` OFFLINE (per passage, LLM-priced) ONLINE (per query, graph-priced) --------------------------------- -------------------------------- passage query - | LLM NER | LLM NER (one call) - | LLM triple extraction | cosine match -> R_q - v | weight by specificity s_i + | LLM NER | LLM NER (one call, Step 4) + | LLM triple extraction (Step 2) | cosine match -> R_q + v | weight by specificity sᵢ (Step 5) triples -> KG nodes/edges v - | cosine > 0.8 -> synonymy edges PPR (damping 0.5) -> π⃗ + | cosine > 0.8 -> synonymy edges PPR (restart 0.5) -> π⃗ (Step 6) v v - matrix P (node × passage counts) score = π⃗ · P -> ranked passages + matrix P (Step 3) score = π⃗ · P -> ranked passages (Step 7) ``` -All LLM-heavy work moves offline. Online retrieval is 10-30× cheaper and 6-13× faster than -IRCoT, which loops an LLM per hop. This is the classic database trade: pay at write/index -time to make reads a single index probe. +All LLM-heavy work is offline. Online retrieval is **10–30× cheaper and 6–13× faster** than +IRCoT, which loops an LLM per hop (§4, measured in Appendix G). This is the classic database +trade: pay at index time to make reads one index probe. The offline bill dominates when the +corpus is large and the query volume is low — the break-even is the same amortization argument +as a materialized view. -### Step 7 — What the numbers say +### Step 9 — What the numbers say, and where the design earns its keep -Retrieval (recall@2 / recall@5): MuSiQue 40.9/51.9, 2WikiMultihopQA 70.7/89.1, HotpotQA -60.5/77.7 — against ColBERTv2 on 2Wiki that is roughly +11% R@2 and +20% R@5. The all-recall -metric (fraction of questions where ALL supporting passages are retrieved) jumps 37.1 → 75.7 -AR@5 on 2Wiki. QA improves up to +3 F1 (MuSiQue), +17 F1 (2Wiki), +1 F1 (HotpotQA). HippoRAG -also composes: plugging it in as IRCoT's retriever adds about +4% (MuSiQue), +18% (2Wiki), -+1% (HotpotQA) R@5 over IRCoT alone. +> **In:** the built system from Steps 2–7. +> **Out:** the measured recall/QA gains (Tables 2–4, 6) and the ablations that locate them +> (Table 5). -### Step 8 — Ablations: where the design earns its keep +Single-step retrieval, recall@2 / recall@5 (**Table 2**), HippoRAG on the ColBERTv2 backbone: -- Extractor quality dominates: replacing LLM OpenIE with REBEL (a small fine-tuned extraction - model) drops recall sharply — GPT-3.5 produces about 2× as many triples as REBEL. Recall of - the index bounds recall of retrieval. -- Llama-3.1-70B as extractor outperforms GPT-3.5 on 2 of 3 datasets — open models suffice. -- PPR strongly beats using query nodes only, or query nodes plus their direct neighbors: - multi-hop diffusion is doing real work, not just neighborhood lookup. -- Node specificity helps MuSiQue and HotpotQA; synonymy edges help 2Wiki most. +``` + MuSiQue 2Wiki HotpotQA + R@2 / R@5 40.9 / 51.9 70.7 / 89.1 60.5 / 77.7 (Table 2) +``` + +The §4 text reads the 2Wiki column as "an impressive improvement of 11 and 20% for R@2 and R@5" +over ColBERTv2 (which scores 59.2 / 68.2 in the same table) and "around 3% on MuSiQue". The +**all-recall** metric — the fraction of questions for which *every* supporting passage is +retrieved (**Table 6**) — shows an even larger gap on 2Wiki: ColBERTv2 AR@5 37.1 → HippoRAG +75.7. QA F1 improves by up to 3 (MuSiQue), 17 (2Wiki) and 1 (HotpotQA) point (**Table 4**). +HippoRAG also *composes*: as IRCoT's retriever it adds about +4 / +18 / +1% R@5 over IRCoT alone +(**Table 3**). + +The ablations (**Table 5**) locate the gains: + +- **Extractor quality dominates.** Swapping the LLM OpenIE for REBEL — a small fine-tuned + end-to-end extraction model — drops the average R@5 from 72.9 to 58.4; "GPT-3.5 produces + twice as many triples" as REBEL (§5.1). Recall of the index bounds recall of retrieval. +- **Open models suffice.** Llama-3.1-70B as the extractor beats GPT-3.5 on 2 of the 3 datasets + (MuSiQue and HotpotQA; it trails on 2Wiki) — Table 5, rows for the OpenIE alternatives. +- **PPR is doing real work.** "Rq Nodes Only" (score only the seeds) and "Rq Nodes & Neighbors" + (seeds plus their one-hop neighbors) both fall far below full PPR — Table 5, PPR-alternative + rows — so the multi-hop diffusion, not a neighborhood lookup, is what pays. +- **Specificity and synonymy split by dataset.** Node specificity helps MuSiQue and HotpotQA + and barely moves 2Wiki; synonymy edges help 2Wiki most (§5.1) — 2Wiki is entity-centric, so + standardizing surface forms matters more there than term weighting. ## How to read the paper (with the concepts in hand) -1. §1 — read the introduction for the path-finding motivation figure; hold Step 1 and - Step 4 in mind: the paper's whole bet is that association beats chaining. -2. §2.1 — the neurobiological framing; check the component mapping against the Step 1 - diagram and note the "index stores no content" claim. -3. §2.2 — offline indexing; verify the two-step OpenIE order (NER, then triples with - entities in the prompt), τ = 0.8 synonymy edges, and matrix P (Step 2). -4. §2.3 — online retrieval; trace the Step 3 pipeline and find where node specificity - (Step 5) multiplies restart probabilities. -5. §3 — experimental setup: MuSiQue, 2WikiMultihopQA, HotpotQA; baselines including - ColBERTv2 and IRCoT; note Contriever/ColBERTv2 double as HippoRAG's encoders. -6. §4 — results; check the Step 7 numbers, especially the all-recall jump and the - IRCoT + HippoRAG combination. -7. §5 — discussion: the ablations of Step 8 and the Stanford/Alzheimer's path-finding - case study of Step 4; the appendices hold the prompts and further ablations if you - want to reproduce the extraction. +1. **§1** — the introduction and Figure 1: the path-finding motivation. Hold Steps 1 and 6 in + mind; the paper's whole bet is that association beats chaining. +2. **§2.1** — the neurobiological framing; check the component mapping against Step 1 and note + the "index stores no content" claim. +3. **§2.2–§2.3** — indexing and retrieval; verify the two-step OpenIE order (NER, then triples + with the entities pasted in), τ = 0.8 synonymy edges, matrix P (Steps 2–3), and where node + specificity multiplies the restart probabilities (Step 5). +4. **§3** — setup: MuSiQue, 2WikiMultihopQA, HotpotQA; baselines including ColBERTv2 and IRCoT; + note Contriever/ColBERTv2 double as HippoRAG's encoders. Confirm τ = 0.8 and damping 0.5 in + §3.4. +5. **§4, Tables 2–4, 6** — results; check the Step 9 numbers, especially the Table 6 all-recall + jump and the Table 3 IRCoT + HippoRAG combination. +6. **§5.1, Table 5** — the ablations of Step 9; then **§5.3, Table 7** — the path-finding vs + path-following case study (answer Thomas Südhof); the appendices hold the prompts and the + Appendix G cost/latency measurements. ## Questions to answer in notes.md @@ -166,37 +295,118 @@ also composes: plugging it in as IRCoT's retriever adds about +4% (MuSiQue), +18 incremental index maintenance compared to true corpus-level IDF? 3. The REBEL ablation shows extraction recall bounds retrieval recall. How would you monitor index recall in production without gold triples? -4. Online retrieval is 10-30× cheaper and 6-13× faster than IRCoT. Which costs moved - offline to make that possible, and when does the offline bill dominate? +4. Online retrieval is 10–30× cheaper and 6–13× faster than IRCoT (§4, Appendix G). Which costs + moved offline to make that possible, and when does the offline bill dominate? 5. Synonymy edges (cosine above τ = 0.8) help 2Wiki most while specificity helps MuSiQue and HotpotQA. What does that split suggest about the datasets' entity-surface variety? ## Done when -- [ ] You can draw the brain-to-component mapping from memory and state what the - hippocampus analogue does and does not store. -- [ ] You can explain path-finding vs path-following and why the Südhof example defeats - ColBERTv2 and IRCoT but not PPR. -- [ ] You can write the online scoring pipeline end to end: query NER → R_q → specificity - weights → PPR (damping 0.5) → π⃗·P. -- [ ] You have run the companion experiment and reproduced mention-count ranking degrading - to 9.21 mean rank at 2 hops while PPR stays at 1.00. +Answer each before unfolding it. + +- [ ] You can draw the brain-to-component mapping from memory and state what the hippocampus analogue does and does not store. + +
Answer + + Three regions map to three components (§2.1, Step 1): the **neocortex** to the LLM plus the + passage store (actual content), the **parahippocampal regions** to the retrieval encoders + (Contriever / ColBERTv2) that detect synonymy, and the **hippocampus** to the knowledge graph + plus PPR. The hippocampal analogue stores **no content — only the index of associations** + (nodes N, triple edges E, synonymy edges E′). The passage text lives only in the passage + store, reachable from index nodes through matrix P (Step 3). That separation is what makes + retrieval one graph computation rather than a corpus scan. + +
+ +- [ ] You can explain path-finding vs path-following and why the Figure 1 example defeats ColBERTv2 and IRCoT but not PPR. + +
Answer + + A **path-following** question can be solved hop by hop because each hop's answer names the + next clue; a **path-finding** question cannot, because the answer is the entity several query + entities *both* point at and no single passage names them together (§1, Figure 1). In the + paper's example — "Which Stanford professor works on the neuroscience of Alzheimer's?" — no + passage mentions Stanford, Alzheimer's, and the answer together, so a passage encoder + (ColBERTv2) scores every candidate near zero and an iterative retriever (IRCoT) has no chain + to follow. + + PPR wins because it does not look for a passage containing the pattern; it lets the pattern + emerge in the graph. Restart mass at both seeds diffuses, and the one node reachable from both + — **Thomas Südhof** (Table 7, §5.3; abbreviated "Professor Thomas" in the §2.3 walkthrough) — + sums both inflows and outranks every dead-end that collects from one seed. Step 6's arithmetic + makes it ≈ 2× a dead-end after one iteration. + +
+ +- [ ] You can write the online scoring pipeline end to end: query NER → R_q → specificity weights → PPR (restart 0.5) → π⃗·P. + +
Answer + + One LLM NER call extracts the query entities Cq (Step 4); each is embedded and matched to its + nearest graph node by cosine, giving the seeds R_q (§2.3). Each seed's restart probability is + multiplied by its node specificity sᵢ = |Pᵢ|⁻¹ and renormalized, so common entities get less + mass (Step 5). PPR runs over the KG (nodes N, edges E + E′) with restart probability 0.5 from + that weighted vector, yielding a stationary distribution π⃗ high on nodes near several seeds + (Step 6). Finally p⃗ = π⃗ · P scores each passage by the PPR mass of the nodes it mentions, + weighted by mention counts, and the top-k passages are returned (Step 7). Exactly one LLM call + and one graph computation — no per-hop loop. + +
+ +- [ ] You have run the companion experiment and reproduced mention-count ranking degrading to 9.21 mean rank at 2 hops while PPR stays at 1.00. + +
Answer + + The companion crate ([experiments/src/kg.rs](experiments/src/kg.rs), + [experiments/src/ppr.rs](experiments/src/ppr.rs)) builds synthetic path-finding instances: two + seeds, eight distractor chains each, one shared answer, one passage per fact, and no passage + mentioning both seeds. **Mention-count** ranking (vector RAG's shape — score each passage + against the query independently) finds the answer at mean rank **1.00** at one hop, because + the answer is named next to both seeds, but collapses to **9.21** at two hops — chance among + 17 candidates is (1+17)/2 = 9 — because every candidate's interior passages mention no query + entity and all score zero ([FINDINGS.md](../../FINDINGS.md) row 38; notes.md). PPR restores + rank 1.00 at one, two and three hops, because mass still sums at the meet node regardless of + chain length. One PPR query on a 100k-node / ~400k-edge graph runs 30 power iterations in + ≈ 56.6 ms (notes.md reference). + +
+ - [ ] notes.md answers all 5 questions. +
Answer + + The five questions above are recorded and answered in notes.md, each tied to the paper section + or table that settles it: the meet-node argument and the graph shape that defeats it (Step 6); + the incremental-maintenance property of local specificity versus global IDF (Step 5, §2.3); + index-recall monitoring without gold triples (the REBEL ablation, Table 5 / §5.1); which costs + moved offline and when the offline bill dominates (Step 8, §4 / Appendix G); and what the + specificity-vs-synonymy dataset split implies about entity-surface variety (§5.1). + +
+ ## References -- Gutiérrez et al., "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large - Language Models," NeurIPS 2024 — https://arxiv.org/abs/2405.14831 -- Local copy of the PDF: /tmp/hipporag.pdf -- Companion experiment in this repo: [experiments/src/ppr.rs](experiments/src/ppr.rs) - (PPR stub) and [experiments/src/kg.rs](experiments/src/kg.rs) (synthetic path-finding - instances). -- Companion measurements: mention-count ranking (vector RAG's shape) gets mean rank 1.00 - at 1 hop but 9.21 at 2 hops (chance = 9.0 among 17 candidates); the PPR reference scores - 1.00 at 1, 2, and 3 hops; one PPR query on a 100k-node / ~400k-edge graph with 30 power - iterations ≈ 57 ms. -- Retrieval encoders used by the paper: Contriever and ColBERTv2. -- Extraction model: GPT-3.5-turbo-1106 at temperature 0; Llama-3.1-70B tested in - ablations, outperforming GPT-3.5 on 2 of 3 datasets. -- Benchmarks used: MuSiQue, 2WikiMultihopQA, HotpotQA; baselines include ColBERTv2 - and IRCoT. +- Gutiérrez et al., "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language + Models," NeurIPS 2024 — https://arxiv.org/abs/2405.14831. Section, table and figure numbers + in this chapter are from that version. + +| Where | What it settles | +|---|---| +| §2.1, Figure 1 | the hippocampal analogy; the path-finding example (answer "Professor Thomas" in the walkthrough) | +| §2.3 | two-step OpenIE; matrix P (`|N|×|P|`); query nodes R_q; PPR; node specificity sᵢ = |Pᵢ|⁻¹; scoring p⃗ = π⃗·P | +| §3.4 | GPT-3.5-turbo-1106, temperature 0; synonymy threshold τ = 0.8; PPR damping (restart) 0.5 | +| §4, Appendix G | 11/20% R@2/R@5 gain on 2Wiki; 10–30× cheaper, 6–13× faster than IRCoT | +| Table 2 | single-step R@2/R@5: MuSiQue 40.9/51.9, 2Wiki 70.7/89.1, HotpotQA 60.5/77.7 | +| Table 3 | IRCoT + HippoRAG: +4/+18/+1% R@5 | +| Table 4 | QA F1: up to +3 / +17 / +1 | +| Table 5 | ablations: REBEL collapse, Llama-3.1-70B, Rq-only, w/o specificity, w/o synonymy | +| Table 6 | all-recall: ColBERTv2 2Wiki AR@5 37.1 → HippoRAG 75.7 | +| §5.3, Table 7 | the path-finding case study; answer **Thomas Südhof** ranked 1st by HippoRAG; ColBERTv2/IRCoT fail | + +- Companion experiment: [experiments/src/ppr.rs](experiments/src/ppr.rs) (PPR stub) and + [experiments/src/kg.rs](experiments/src/kg.rs) (synthetic path-finding instances). Measured + shape in [FINDINGS.md](../../FINDINGS.md) row 38: mention-count mean rank 1.00 at 1 hop, 9.21 + at 2 hops (chance = 9 among 17); the PPR reference scores 1.00 at 1/2/3 hops; one PPR query on + a 100k-node / ~400k-edge graph with 30 power iterations ≈ 56.6 ms (notes.md). +- Encoders used by the paper: Contriever and ColBERTv2. Benchmarks: MuSiQue, 2WikiMultihopQA, + HotpotQA; baselines include ColBERTv2 and IRCoT. diff --git a/topics/38-graphrag-agent-memory/reading-zep-graphiti.md b/topics/38-graphrag-agent-memory/reading-zep-graphiti.md index 2e3a59f..b694c10 100644 --- a/topics/38-graphrag-agent-memory/reading-zep-graphiti.md +++ b/topics/38-graphrag-agent-memory/reading-zep-graphiti.md @@ -1,13 +1,17 @@ # Zep/Graphiti: agent memory as a bi-temporal knowledge graph -Zep (arXiv 2501.13956, Rasmussen et al.) is the paper behind Graphiti, an engine that -turns an agent's ever-growing chat history into an incrementally-built temporal knowledge -graph. Instead of stuffing the full conversation into the prompt or doing flat RAG over -message chunks, Graphiti extracts entities and facts as episodes arrive, stamps every edge -with four timestamps (valid time and transaction time), and answers queries with a hybrid -search-rerank-construct pipeline. For a graph-database developer, the interesting part is -that this is bitemporal MVCC applied to graph edges — version chains, time-travel reads, -and invalidation-instead-of-deletion, wearing an LLM costume. +Zep (arXiv 2501.13956, Rasmussen et al.) is the paper behind Graphiti, an engine that turns an +agent's ever-growing chat history into an incrementally-built temporal knowledge graph. Instead +of stuffing the full conversation into the prompt or doing flat RAG over message chunks, +Graphiti extracts entities and facts as episodes arrive, stamps every edge with four timestamps +(valid time and transaction time), and answers queries with a hybrid search-rerank-construct +pipeline. For a graph-database developer, the interesting part is that this is **bi-temporal +versioning applied to graph edges** — version chains, time-travel reads, and +invalidation-instead-of-deletion, wearing an LLM costume. This repo's **topic 33 (temporal +graphs)** builds the same valid-time × transaction-time model from the storage side; read this +paper as that model shipped as a product. + +Every number below is quoted with the section or table it comes from in arXiv 2501.13956. ## The problem in one sentence @@ -24,187 +28,327 @@ are invalidated (never deleted) and every edge carries its own history. ### Step 1 — Three subgraphs: episodes, entities, communities -Graphiti layers three subgraphs in one graph, from raw to abstract: +> **In:** raw conversation data (messages, text, or JSON). +> **Out:** a three-layer graph — the substrate every later step reads and writes. + +Graphiti layers three subgraphs in one graph, from raw to abstract (§2): ``` +--------------------------------------------------+ - | Community subgraph | + | Community subgraph (Gc, §2.3) | | clusters of related entities + LLM summaries | +------------------------^-------------------------+ | label propagation +------------------------+-------------------------+ - | Semantic entity subgraph | + | Semantic entity subgraph (Gs, §2.2) | | entities + relations (facts), with embeddings | +------------------------^-------------------------+ | LLM extraction +------------------------+-------------------------+ - | Episode subgraph (data layer, lossless) | + | Episode subgraph (Ge, §2.1, non-lossy) | | raw messages, text, JSON — never rewritten | +--------------------------------------------------+ ``` -The episode layer is the lossless source of truth; entities and facts are derived from it; -communities summarize clusters of entities. Think base table → index → materialized view. +An **episode** is one ingested data unit — a message, a text blob, or a JSON record (§2.1). The +episode layer is the **non-lossy** source of truth (the paper's word); entities and facts are +derived from it; communities summarize clusters of entities. Bidirectional **episodic edges** +(Ge) tie every derived fact back to the episode it came from, so "semantic artifacts can be +traced to their sources for citation or quotation" (§2.1). Think base table → index → +materialized view, with the episode log as the layer the others can always be rebuilt from. ### Step 2 — Extraction and entity resolution -Entities are extracted per-episode with the last few messages as context, so coreference -("she", "the company") resolves correctly. Each entity gets an embedding and a summary; -each edge (fact) carries a relation name plus a fact string. Entity resolution is a -two-stage merge: embedding candidate search finds plausible existing nodes, then an LLM -verifies whether the new mention is the same entity before merging. This is fuzzy -upsert-by-similarity — the write path's dedup logic, done with an LLM as the comparator. +> **In:** a new episode plus the last few messages as context (from Step 1's episode layer). +> **Out:** new/merged entity nodes and fact edges in the semantic layer — the things Steps 3–4 +> timestamp and invalidate. + +Entities are extracted per-episode with the last few messages as context, so **coreference** +("she", "the company" — resolving a pronoun or description to the entity it refers to) resolves +correctly (§2.2.1). Each entity gets an embedding and a summary; each edge (fact) carries a +relation name plus a fact string (§2.2.2). **Entity resolution** — deciding whether a new +mention is an entity the graph already knows — is a two-stage merge: an embedding candidate +search finds plausible existing nodes, then an **LLM verifies** whether the new mention is the +same entity before merging (§2.2.1, and the "is_duplicate" prompt in §6.1). This is fuzzy +upsert-by-similarity: the write path's dedup logic, with an LLM as the comparator. + +### Step 3 — The bi-temporal model: two timelines, four timestamps -### Step 3 — The bi-temporal model (§2.1): two timelines, four timestamps +> **In:** an extracted fact edge from Step 2. +> **Out:** an edge stamped on two independent timelines — the state Step 4 mutates on +> contradiction and Step 6 filters on at query time. -Every edge carries timestamps on two independent timelines. Timeline T is event time — -when the fact was true in the world. Timeline T' is ingestion (transactional) time — when -the system learned it. Four timestamps per edge: +Zep implements a **bi-temporal model** (§2.1): **timeline T** is *event time* — the chronological +ordering of events in the world — and **timeline T′** is *transaction time* — the order of Zep's +data ingestion (T′ "serves the traditional purpose of database auditing," §2.1). Every fact edge +carries **four timestamps** (§2.2): `t'_created, t'_expired ∈ T′` record when the fact was +created or invalidated *in the system*, while `t_valid, t_invalid ∈ T` record the range during +which the fact *held true in the world*. ``` - Timeline T (event time): t_valid ......... t_invalid - Timeline T' (transaction time): t'_created ...... t'_expired + Timeline T (event/valid time): t_valid ......... t_invalid + Timeline T' (transaction time): t'_created ...... t'_expired edge: (Alice) -[WORKS_AT]-> (Acme) - t_valid=Jan t_invalid=Jun "true in the world Jan..Jun" - t'_created=Feb t'_expired=Jul "known to the system Feb..Jul" + t_valid = Jan t_invalid = Jun "true in the world Jan..Jun" + t'_created= Feb t'_expired = Jul "known to the system Feb..Jul" ``` -This is classic valid-time × transaction-time bitemporality from the database literature, -applied to knowledge-graph edges. The two timelines are independent: a fact can be learned -long after it became true, and unlearned (superseded) long after it stopped being true. +The two timelines are independent: a fact can be learned long after it became true (T lags T′), +and superseded in the system long after it stopped being true. Relative mentions ("I started my +new job two weeks ago") are resolved against the episode's reference timestamp `t_ref` at +extraction time (§2.2), so the stored `t_valid` is an absolute datetime. ### Step 4 — Edge invalidation: contradiction as versioning -At ingest, an LLM compares each new fact against existing semantically-related edges. When -a new fact contradicts an old one, the old edge is invalidated — never deleted: +> **In:** a new fact edge from Step 3 and the semantically-related edges already in the graph. +> **Out:** old edges marked invalid (not deleted) — the version chain Step 6's as-of reads walk. + +At ingest, an LLM compares each new fact against existing semantically-related edges. When a new +fact **temporally overlaps and contradicts** an old one, the old edge is *invalidated*, never +deleted. The precise contract (§2.2): the system sets the old edge's **`t_invalid` to the +`t_valid` of the invalidating edge**, and expires it on the transaction timeline. ``` ingest: "Alice works at Beta" (valid from Jun) old edge (Alice)-[WORKS_AT]->(Acme) - t_invalid := Jun (new fact's validity start) + t_invalid := Jun (:= the NEW fact's t_valid, per §2.2) t'_expired := now - edge KEPT — audit trail + edge KEPT — audit trail, nothing deleted new edge (Alice)-[WORKS_AT]->(Beta) t_valid := Jun t'_created := now ``` -New information wins by default. Because nothing is deleted, "what was true in March?" -(filter on T) and "what did we know in March?" (filter on T') are each one predicate away. -This is exactly a version chain: invalidation writes tombstone timestamps instead of -removing tuples, and as-of queries are time-travel reads over that chain. +"Graphiti consistently prioritizes new information when determining edge invalidation" (§2.2) — +new information wins by default. Because nothing is deleted, both temporal questions are one +predicate away. Worked as an **as-of** filter — "what did we believe was true in March, using +only what the system knew by end-of-June?": + +``` + keep edge e iff + e.t_valid <= Mar-31 < e.t_invalid (event-time slice: true in March) + AND + e.t'_created <= Jun-30 < e.t'_expired (as-known-at: system's June view) + + Alice/Acme edge: t_valid=Jan <= Mar < t_invalid=Jun -> TRUE (event) + t'_created=Feb <= Jun < t'_expired=Jul -> TRUE (transaction) + => returned: in March the system's June-view had Alice at Acme +``` + +Drop the second predicate and you get "what was true in March" regardless of when learned; drop +the first and you get "what did we know as of June." This is a version chain: invalidation +writes tombstone timestamps instead of removing tuples, and as-of queries are time-travel reads +over that chain — exactly topic 33's model. ### Step 5 — Communities via dynamic label propagation -Communities cluster related entities, each with an LLM-written summary. Graphiti uses -label propagation rather than Leiden for one engineering reason: label propagation has a -cheap dynamic extension. When a new node arrives, it simply adopts the majority label of -its neighbors — full recomputation is postponed rather than triggered per write. That is -the incremental-maintenance trade every streaming system makes: accept slightly stale -partitions in exchange for O(degree) update cost. +> **In:** the semantic entity subgraph from Step 2. +> **Out:** community nodes with summaries (Gc) — a coarse retrieval target and one φ signal in +> Step 6. + +Communities cluster related entities, each with an LLM-written summary. Graphiti uses **label +propagation** rather than **Leiden** (the algorithm GraphRAG uses) for one engineering reason +(§2.3): label propagation has a cheap **dynamic extension**. When a new node arrives, it adopts +"the community held by the **plurality** of its neighbors" (§2.3 — plurality, i.e. the most +common neighbor label, not necessarily a majority), then updates that community's summary — +full recomputation is postponed rather than triggered per write. The paper is candid that "the +resulting communities gradually diverge from those that would be generated by a complete label +propagation" (§2.3): that is the incremental-maintenance trade every streaming system makes — +accept slightly stale partitions in exchange for O(degree) update cost. ### Step 6 — Retrieval: the φ → ρ → χ funnel -Query time is a three-phase pipeline: +> **In:** a text query α and the graph built by Steps 1–5. +> **Out:** a compact context string β for the agent's prompt. + +Zep's search API is a function `f(α) = χ(ρ(φ(α))) = β` (§3): a query string in, a formatted +context string out. Three phases: ``` - query + query α | v - φ search (parallel): cosine similarity on embeddings - + BM25 fulltext - + graph BFS from recently-mentioned nodes - | + φ search (§3.1, three functions run in parallel): + φ_cos : cosine similarity on embeddings + φ_bm25 : Okapi BM25 full-text (Neo4j/Lucene) + φ_bfs : breadth-first search over n-hops, SEEDED from + recently-mentioned nodes + | returns a 3-tuple: (semantic edges, entity nodes, community nodes) v - ρ rerank: RRF, MMR, episode-mentions reranker, - node-distance reranker, cross-encoder + ρ rerank (§3.2): RRF (fusion) · MMR (diversity) + · episode-mentions reranker (frequency) + · node-distance reranker (locality to a centroid) + · cross-encoder (precision, highest cost) | v - χ construct: assemble facts / entities / summaries - into a compact context string for the prompt + χ construct (§3): for each edge return fact + t_valid,t_invalid; + for each entity the name + summary; for each + community the summary -> context string β ``` -φ casts a wide net across three signal types; ρ fuses and reorders (RRF for fusion, MMR -for diversity, graph-aware rerankers for locality, a cross-encoder for precision); χ -serializes the survivors into a small context block. It is a query executor: scan -operators feeding a rank-merge feeding a projection. +φ casts a wide net across three signal types for **recall**; ρ fuses and reorders for +**precision** (RRF fuses the ranked lists, MMR trades relevance for diversity, the two graph +rerankers add locality, the cross-encoder is the most accurate and most expensive); χ serializes +the survivors — crucially, it emits each edge's **`t_valid`/`t_invalid`** alongside the fact +(§3), so the temporal model reaches the prompt. It is a query executor: scan operators feeding a +rank-merge feeding a projection. ### Step 7 — Results: accuracy up, latency way down -On DMR (Deep Memory Retrieval), Zep scores 94.8% vs MemGPT's 93.4% (gpt-4-turbo). The -stronger evidence is LongMemEval with gpt-4o: accuracy 60.2% → 71.2%, and response latency -28.9 s → 2.58 s — about a 90% cut — because the prompt shrinks from ~115k tokens (full -conversation) to ~1.6k tokens (retrieved facts). Biggest category gains: -single-session-preference +184% and temporal reasoning +38.4%. One honest regression: -single-session-assistant −17.7% — when the answer needs verbatim recall of one recent -session, full context beats retrieval. +> **In:** the full system from Steps 1–6. +> **Out:** the measured accuracy/latency evidence Step 8 reads architecturally. + +On **DMR** (Deep Memory Retrieval, MemGPT's benchmark), Zep scores **94.8% vs MemGPT's 93.4%**, +run on **gpt-4-turbo** for comparability (§4.2). The stronger evidence is **LongMemEval** with +gpt-4o (§4.3, **Table 2**): accuracy **60.2% → 71.2%** and response latency **28.9 s → 2.58 s** +— because the prompt shrinks from **≈ 115k tokens** (full conversation) to **≈ 1.6k tokens** +(retrieved facts). The abstract frames the accuracy gain as "up to 18.5%." Per-category (§4.3, +**Table 3**, gpt-4o, Zep vs full-context): **single-session-preference +184%** (20.0% → 56.7%), +**temporal-reasoning +38.4%** (45.1% → 62.4%), **multi-session +30.7%**. One honest regression: +**single-session-assistant −17.7%** (94.6% → 80.4%) — when the answer needs verbatim recall of +one recent session, full context beats retrieval. ### Step 8 — The database-internals reading -Strip the LLM machinery and Graphiti is bitemporal MVCC on a graph. Invalidation instead -of deletion is a version chain; t'_expired is a tombstone; as-of queries are snapshot -reads; the episode layer is the WAL-like lossless log the derived layers can always be -rebuilt from. Topic 33 in this path (temporal graphs) covers the same model from the -storage side — this paper is that model deployed as a product, with the extraction and -retrieval stages bolted on where a database would have parsers and planners. +> **In:** everything above. +> **Out:** the one-sentence mental model to carry to topic 33. + +Strip the LLM machinery and Graphiti is **bi-temporal versioning on a graph**. Invalidation +instead of deletion is a version chain; `t'_expired` is a tombstone; as-of queries are snapshot +reads; the episode layer is the WAL-like non-lossy log the derived layers can always be rebuilt +from. Topic 33 in this path (temporal graphs) covers the same valid-time × transaction-time +model from the storage side — this paper is that model deployed as a product, with the +extraction and retrieval stages bolted on where a database would have parsers and planners. ## How to read the paper (with the concepts in hand) -1. Read the abstract and introduction for the problem framing (Step 1's motivation): why - context windows and flat RAG fail for agent memory. -2. Read the architecture description of the three subgraphs — episodes, entities, - communities — and map each to the stack diagram in Step 1. Note the label-propagation - choice and its dynamic extension (Step 5). -3. Slow down at §2.1, the bi-temporal model. Draw the two timelines yourself and confirm - the four timestamps match Step 3. This is the core of the paper. -4. Read the extraction and entity-resolution passages with Step 2 in hand; then the edge - invalidation contract against Step 4 — check that old edges are kept, not deleted. -5. Read the retrieval section mapping each named component (RRF, MMR, cross-encoder, BFS) - into the φ/ρ/χ funnel of Step 6. -6. Finish with the evaluation: DMR, then LongMemEval per-category numbers from Step 7. - Look for the token-count explanation of the latency drop, and find the - single-session-assistant regression — ask yourself why retrieval loses there. -7. Then run the companion experiment (see References) and compare its four-timestamp - contract to the paper's. +1. Read the abstract and §1 for the problem framing (Step 1's motivation): why context windows + and flat RAG fail for agent memory. +2. Read §2 architecture — episodes (§2.1), entities/facts (§2.2), communities (§2.3) — and map + each to the stack diagram in Step 1. Note the label-propagation choice and its plurality + dynamic extension (Step 5). +3. Slow down at the bi-temporal passages: the timeline T/T′ definition in §2.1 and the + four-timestamp + invalidation contract in §2.2. Draw the two timelines yourself and confirm + they match Steps 3–4. This is the core of the paper. +4. Read the extraction and entity-resolution passages (§2.2.1) with Step 2 in hand; then the + edge-invalidation contract in §2.2 against Step 4 — check that old edges are kept, not + deleted, and that `t_invalid := t_valid` of the invalidating edge. +5. Read §3 retrieval, mapping each named component (RRF, MMR, episode-mentions, node-distance, + cross-encoder, BFS) into the φ/ρ/χ funnel of Step 6, and note that χ emits `t_valid`/ + `t_invalid`. +6. Finish with §4: DMR (§4.2), then LongMemEval per-category numbers (§4.3, Tables 2–3). Find + the token-count explanation of the latency drop, and the single-session-assistant regression + — ask yourself why retrieval loses there. +7. Then run the companion experiment (see References) and compare its four-timestamp contract to + the paper's §2.2. ## Questions to answer in notes.md -1. For each of the four timestamps (t_valid, t_invalid, t'_created, t'_expired), which - timeline does it live on, and who sets it — the world, the LLM, or the ingest clock? -2. When a new fact contradicts an old edge, exactly which timestamps change on the old - edge and why is the edge kept rather than deleted? What queries would break if it were - deleted? -3. Why does Graphiti choose label propagation over Leiden for communities, and what - staleness does the dynamic extension accept in exchange? +1. For each of the four timestamps (t_valid, t_invalid, t'_created, t'_expired), which timeline + does it live on, and who sets it — the world, the LLM, or the ingest clock? +2. When a new fact contradicts an old edge, exactly which timestamps change on the old edge and + why is the edge kept rather than deleted? What queries would break if it were deleted? +3. Why does Graphiti choose label propagation over Leiden for communities, and what staleness + does the plurality dynamic extension accept in exchange? 4. In the φ → ρ → χ pipeline, which reranker would you expect to matter most for temporal - reasoning questions, and what graph-database operator does each φ search method - correspond to? + reasoning questions, and what graph-database operator does each φ search method correspond to? 5. Why does single-session-assistant regress −17.7% while single-session-preference gains +184%? What does that say about when retrieval beats full context? ## Done when +Answer each before unfolding it. + - [ ] You can draw the two timelines and place all four edge timestamps without looking. -- [ ] You can state the invalidation contract (which timestamps are set, nothing deleted) - and phrase both "true in March" and "known in March" as single filters. + +
Answer + + Two timelines (§2.1): **T = event/valid time** (when the fact was true in the world) and + **T′ = transaction time** (when the system learned it; database-audit purpose). Four + timestamps (§2.2): on **T**, `t_valid` (fact became true) and `t_invalid` (fact stopped being + true); on **T′**, `t'_created` (edge written to the store) and `t'_expired` (edge expired in + the store). For the Alice/Acme edge: t_valid=Jan, t_invalid=Jun on T; t'_created=Feb, + t'_expired=Jul on T′. The two are independent — a fact learned in Feb can have been true since + Jan. + +
+ +- [ ] You can state the invalidation contract (which timestamps are set, nothing deleted) and phrase both "true in March" and "known in March" as single filters. + +
Answer + + On a contradicting, temporally-overlapping new fact, the old edge's **`t_invalid` is set to the + new edge's `t_valid`** and its `t'_expired` is set to now; the edge is **kept, not deleted** + (§2.2). "True in March" is the event-time filter `t_valid ≤ Mar < t_invalid`; "known in March" + is the transaction-time filter `t'_created ≤ Mar < t'_expired`. An as-of query ANDs both. If + the edge were deleted instead of tombstoned, every historical/as-of query — "what did we + believe last quarter?" — would lose its answer, because the superseded version would be gone. + +
+ - [ ] You can name the three φ search methods and at least three ρ rerankers. -- [ ] You ran the companion temporal.rs experiment and reproduced the reference shape: - 100,000 edges kept from 10,000 entities × 10 job changes, 10,000 current, as-of - scan ≈ 0.09 ms. -- [ ] You can explain the LongMemEval latency drop in tokens (~115k → ~1.6k) and name the - one category that regressed. + +
Answer + + φ (§3.1): **φ_cos** cosine similarity on embeddings, **φ_bm25** Okapi BM25 full-text (Neo4j/ + Lucene), and **φ_bfs** breadth-first search over n-hops seeded from recently-mentioned nodes; + φ returns a 3-tuple of semantic edges, entity nodes, and community nodes. ρ (§3.2), any three + of: **RRF** (Reciprocal Rank Fusion), **MMR** (Maximal Marginal Relevance), the + **episode-mentions** reranker (frequency of mention), the **node-distance** reranker (locality + to a centroid node), and the **cross-encoder** (LLM relevance scoring, highest cost). χ then + builds the context string, emitting `t_valid`/`t_invalid` per edge. + +
+ +- [ ] You ran the companion temporal.rs experiment and reproduced the reference shape. + +
Answer + + The companion crate ([experiments/src/temporal.rs](experiments/src/temporal.rs)) is a + miniature bi-temporal edge store with the same four timestamps and the same + contradiction-invalidation contract (`t_invalid := t_valid` of the superseding edge, nothing + deleted). Its reference lane builds **100,000 edges from 10,000 entities × 10 job changes**, + of which **10,000 are current**, and an **as-of scan runs in ≈ 0.09 ms** (notes.md baseline; + [FINDINGS.md](../../FINDINGS.md) row 38). Reproducing it confirms that as-of reads are cheap + filters over a version chain, not reconstructions. + +
+ +- [ ] You can explain the LongMemEval latency drop in tokens (~115k → ~1.6k) and name the one category that regressed. + +
Answer + + Full-context feeds the whole conversation — **≈ 115k tokens on average** (§4.3) — to the LLM + per query, costing **28.9 s** at gpt-4o. Zep retrieves only the relevant facts, shrinking the + prompt to **≈ 1.6k tokens** and the latency to **2.58 s** (Table 2) — the token count *is* the + latency story. The one regressing category is **single-session-assistant, −17.7%** at gpt-4o + (Table 3): when the answer requires verbatim recall of a single recent session, full context + still beats retrieval because retrieval can drop the exact wording. + +
## References -- Paper: "Zep: A Temporal Knowledge Graph Architecture for Agent Memory", Rasmussen et - al. — https://arxiv.org/abs/2501.13956 (local copy: /tmp/zep.pdf) +- Rasmussen et al., "Zep: A Temporal Knowledge Graph Architecture for Agent Memory," arXiv + 2501.13956 — https://arxiv.org/abs/2501.13956. Section and table numbers in this chapter are + from that version. + +| Where | What it settles | +|---|---| +| §2.1 | episodes (message/text/JSON), non-lossy episode subgraph; bi-temporal model — timeline T (event) vs T′ (transaction) | +| §2.2 | four timestamps (t_valid/t_invalid ∈ T, t'_created/t'_expired ∈ T′); invalidation sets old t_invalid := new t_valid, nothing deleted; t_ref-based relative-date resolution | +| §2.2.1 | entity extraction with coreference; embedding-candidate + LLM-verified resolution | +| §2.3 | communities via **label propagation** (not Leiden); dynamic extension adopts **plurality** neighbor label | +| §3, §3.1, §3.2 | f(α)=χ(ρ(φ(α))); φ_cos/φ_bm25/φ_bfs; RRF, MMR, episode-mentions, node-distance, cross-encoder; χ emits t_valid/t_invalid | +| §4.2 | DMR 94.8% vs MemGPT 93.4% (gpt-4-turbo) | +| §4.3, Table 2 | LongMemEval gpt-4o: 60.2%→71.2%, 28.9s→2.58s, ~115k→~1.6k tokens; abstract "up to 18.5%" | +| §4.3, Table 3 | gpt-4o per-category: single-session-preference +184%, temporal-reasoning +38.4%, multi-session +30.7%, single-session-assistant −17.7% | + - Companion experiment in this repo: [experiments/src/temporal.rs](experiments/src/temporal.rs) - — miniature bi-temporal edge store with the same four timestamps and - contradiction-invalidation contract. -- Topic 33 of this learning path — temporal graphs; same valid-time × transaction-time - model from the storage-engine side. -- Classic background: valid-time and transaction-time bitemporality in the temporal - database literature (the model §2.1 instantiates on KG edges). + — miniature bi-temporal edge store; reference shape 100k edges / 10k current, as-of scan + ≈ 0.09 ms (notes.md; [FINDINGS.md](../../FINDINGS.md) row 38). +- Topic 33 of this learning path — temporal graphs; the same valid-time × transaction-time model + from the storage-engine side. diff --git a/topics/39-fraud-identity-graphs/README.md b/topics/39-fraud-identity-graphs/README.md index 92c2453..31401c3 100644 --- a/topics/39-fraud-identity-graphs/README.md +++ b/topics/39-fraud-identity-graphs/README.md @@ -107,9 +107,10 @@ a middle account contributes min(inflow, outflow) and is penalized λ × max(...) for imbalance, so parking money or camouflage transfers *hurt* the score. Same near-greedy peeling, same style of guarantee. On CBank's real 6.13M-account / 43.98M-transfer data with a labeled -ring (4 sources, 12 mules, 2 destinations, ~452M yuan) it scores +ring (4 sources, 12 mules, 2 destinations; the central mule v5 alone +passes ≈452.1M yuan, in ≈ out) it scores FAUC 0.761/0.843 vs FRAUDAR's 0.529/0.704, and holds F1 ≥ 0.9 down to -injected volumes of 76M vs FRAUDAR's 180M. Covered as a reading guide +injected volumes of $76M vs FRAUDAR's $180M. Covered as a reading guide plus exercise 5 — the peeling machinery is the same as fraudar.rs. ## Production shape: splink (cloned under ~/repos/splink @ 04189f5) @@ -119,14 +120,14 @@ plus exercise 5 — the peeling machinery is the same as fraudar.rs. | `linker.py:66` | `Linker` — the API façade; settings = comparisons + blocking rules | | `linker_components/training.py:163` | `estimate_u_using_random_sampling` — u without labels | | `linker_components/training.py:231` | `estimate_parameters_using_expectation_maximisation(blocking_rule)` — one session per pass | -| `expectation_maximisation.py:225` | the EM core; E-step `:18`, M-step `maximisation_step:193` | +| `expectation_maximisation.py:225` | the EM core; E-step `predict_from_comparison_vectors_sqls` at `:268`, M-step `compute_new_parameters_sql` `:45`/`:278` inside `maximisation_step:193` | | `comparison_level.py:148` | `ComparisonLevel` — m at `:190`, u at `:191`; match weight log2(m/u) at `:426` | | `comparison_level.py:667` | `_tf_adjustment_sql` — term-frequency: "Smith" agreement is worth less | | `comparison_level_library.py:406/:458/:493` | Levenshtein / Jaro-Winkler / Jaro levels — agreement is graded, not boolean | | `predict.py:203` | prior + match weights → probability 1/(1+2^(−mw)); pairwise scoring SQL at `:42` | | `blocking.py:747` | `block_using_rules_sqls` — blocking passes as SQL self-joins | | `linker_components/clustering.py:43` | threshold → `connected_components.py:121` — clusters, exactly lane 3's union-find | -| `dialects.py:24` | one model, four engines: DuckDB `:270`, Spark `:402`, SQLite `:532`, PostgreSQL `:674` | +| `dialects.py:24` | one model, four engines: DuckDB `:270`, Spark `:402`, SQLite `:532`, PostgreSQL `:573` | ## Reading guides diff --git a/topics/39-fraud-identity-graphs/notes.md b/topics/39-fraud-identity-graphs/notes.md index 9717c19..5c9e00e 100644 --- a/topics/39-fraud-identity-graphs/notes.md +++ b/topics/39-fraud-identity-graphs/notes.md @@ -67,8 +67,8 @@ camo 2 — Theorem 3, verified by exercise 2's hand-derivation. /tmp/flowscope.pdf (AAAI'20), /tmp/winkler-rl.pdf (Winkler 2006 survey, pp. 1–22). - FRAUDAR facts: metric family g(S) = f(S)/|S|, f sums in-block edge - weights; column weight 1/log(d_j + c), c = 5. Axioms 1–3 (node - suspiciousness, edge suspiciousness, concentration). Greedy peel = + weights; column weight 1/log(d_j + c), c = 5. Axioms 1–4 (node + suspiciousness, edge suspiciousness, size, concentration). Greedy peel = "exonerate the least suspicious," O(|E| log |V|) with a priority tree. Theorem 2: g(returned) ≥ g_OPT/2. Theorem 3: column weights are camouflage-resistant — camo edges land on honest columns, the @@ -82,8 +82,9 @@ camo 2 — Theorem 3, verified by exercise 2's hand-derivation. q_i = max(in, out); g = (1/|S|) Σ [(1+λ)f_i − λq_i], λ = 4 — parking money or camouflage transfers *lower* the score. Same near-greedy peel and guarantee style. CBank 6.13M accounts / 43.98M transfers, - labeled ring 4 sources / 12 mules / 2 destinations ≈ 452M yuan: - FAUC 0.761/0.843 vs FRAUDAR 0.529/0.704; holds F1 ≥ 0.9 down to 76M + labeled ring 4 sources / 12 mules / 2 destinations, central mule v5 + alone ≈ 452.1M yuan: + FAUC 0.761/0.843 vs FRAUDAR 0.529/0.704; holds F1 ≥ 0.9 down to $76M injected volume vs FRAUDAR's 180M. Covered as guide + exercise 5 (no stub module — the peel machinery is fraudar.rs's). - Winkler/FS facts: R = P(γ|M)/P(γ|U), thresholds T_λ/T_μ with a @@ -106,12 +107,13 @@ camo 2 — Theorem 3, verified by exercise 2's hand-derivation. - splink anchors (cloned ~/repos/splink @ 04189f5) verified by grep/read this session — full table in README, headline set: linker.py:66; training.py:163 (u by random sampling) / :231 (EM per - blocking rule); expectation_maximisation.py:225 (E `:18`, M `:193`); + blocking rule); expectation_maximisation.py:225 (E `:268`, M `:45`/`:278` in + `maximisation_step:193`); comparison_level.py:148 (m `:190`, u `:191`, weight `:426`, tf-adjustment `:667`); predict.py:203 (prior + weights → 1/(1+2^(−mw))); blocking.py:747; clustering.py:43 → connected_components.py:121; dialects.py:24 (DuckDB/Spark/SQLite/ - PostgreSQL at :270/:402/:532/:674). + PostgreSQL at :270/:402/:532/:573). - Crate: 3 provided tests green (review_graph.rs — instance shape 20×80 block / 1600 fraud edges / camo ratio 0.8–1.05; degree-rank camo-0 precision < 0.3; obscurity 0.75 at camo 0 → < 0.3 at camo 2). diff --git a/topics/39-fraud-identity-graphs/reading-fellegi-sunter.md b/topics/39-fraud-identity-graphs/reading-fellegi-sunter.md index 7b3db99..a6fd990 100644 --- a/topics/39-fraud-identity-graphs/reading-fellegi-sunter.md +++ b/topics/39-fraud-identity-graphs/reading-fellegi-sunter.md @@ -16,6 +16,9 @@ with better tooling. Read it as the spec for the `er.rs` experiment you will imp ### Step 1 — A record pair is an agreement pattern, and matching is hypothesis testing +> **In:** nothing yet — a single pair of records, compared field by field. +> **Out:** the agreement pattern γ (a bit vector of agree/disagree) and the likelihood ratio R = P(γ|M)/P(γ|U). + Forget records for a moment; look at a *pair* of records. Compare them field by field (last name, first name, dob, city, phone) and reduce the pair to an agreement pattern γ — essentially a bit vector of agree/disagree outcomes. Fellegi and Sunter (JASA 1969) framed @@ -30,9 +33,12 @@ EM, comparators — is machinery for computing and thresholding R at scale. ### Step 2 — Two thresholds, and why the rule is optimal -Pick an upper cutoff T_μ and a lower cutoff T_λ. If R is at or above T_μ, designate the -pair a match; if R is at or below T_λ, designate it a nonmatch; in between, send it to -clerical review. +> **In:** the ratio R (or its log) from Step 1, for one candidate pair. +> **Out:** a three-way label — match, clerical review, or nonmatch — via two thresholds, provably minimizing the review region. + +Pick an upper cutoff T_μ and a lower cutoff T_λ. Following the paper's rule (Eq. 2): if +R > T_μ, designate the pair a match; if R < T_λ, designate it a nonmatch; if +T_λ ≤ R ≤ T_μ — boundaries included — send it to clerical review. ``` nonmatch clerical review match @@ -48,25 +54,46 @@ theorem that justified automation — the 1990 US Census matching went from an e ### Step 3 — Match weights: log2 R decomposes into per-field bits +> **In:** the ratio R from Step 1 and the fitted (m_i, u_i) per field (Step 5 fits them; here they are given). +> **Out:** a per-field weight table in bits whose signed sum over an agreement pattern equals log2 R. + Under conditional independence of fields given the class (exactly the naive Bayes assumption), the log-likelihood ratio splits into a sum of per-field contributions. With -`m_i = P(agree on field i | M)` and `u_i = P(agree on field i | U)`: +**m_i = P(agree on field i | M)** and **u_i = P(agree on field i | U)**, a field that +agrees contributes **+log2(m_i / u_i)** bits and a field that disagrees contributes +**+log2((1 − m_i) / (1 − u_i))** bits (a negative number). Using the experiment's fitted +parameters (m = [0.80 0.86 0.94 0.78 0.90], u = [0.0052 0.0021 0.0003 0.0051 0.0006]): ``` - field agrees: w_i = +log2( m_i / u_i ) (positive bits) - field disagrees: w_i = +log2( (1 - m_i) / (1 - u_i) ) (negative bits) + field m u agree +log2(m/u) disagree +log2((1-m)/(1-u)) + last 0.80 0.0052 +7.27 -2.31 + first 0.86 0.0021 +8.68 -2.83 + dob 0.94 0.0003 +11.61 -4.06 + city 0.78 0.0051 +7.26 -2.18 + phone 0.90 0.0006 +10.55 -3.32 +``` +Work one pattern — last, first, dob agree, city disagrees, phone agrees: + +``` γ = [ last=agree, first=agree, dob=agree, city=disagree, phone=agree ] - +7.3 +8.7 +11.6 -3.4 +10.6 - └──────────────────── sum ────────────────────┘ log2 R = 34.8 bits + +7.27 +8.68 +11.61 -2.18 +10.55 + └──────────────────── sum ────────────────────┘ log2 R = 35.93 bits ``` -Intuition: u_i for a random pair is roughly 1/pool-size of the field, so rare values are -worth more bits; m_i is dominated by the field's typo rate. Score a pair by summing bits -and compare against a threshold in bits. This is exactly what splink calls match weights. +All five fields agreeing sums to 45.36 bits; all five disagreeing to −14.70. Intuition: +u_i for a random pair is roughly 1/pool-size of the field, so rare values are worth more +bits (dob, with a 3650-value pool, pays +11.61 on agreement); m_i is dominated by the +field's typo rate. Score a pair by summing bits and compare against a threshold in bits — +exactly what splink calls match weights. (The 1969 paper writes the weight as any +monotone function of R, e.g. the natural log; bits — that is, log2 — is the splink +convention this topic uses throughout.) ### Step 4 — String comparators: exact equality throws away a quarter of your matches +> **In:** the binary agree/disagree pattern γ from Step 3, which typos corrupt. +> **Out:** a richer γ where a string comparator (Jaro–Winkler) discounts the agreement weight by similarity. + Exact character-by-character comparison misses more than 25% of true matches in census data, purely from typos. Jaro's comparator counts common characters within a sliding window plus transpositions; Winkler's variant boosts agreement when the strings share a @@ -78,6 +105,9 @@ is unchanged; only the γ alphabet gets richer. ### Step 5 — EM: fitting m, u, and p with zero labeled data +> **In:** the observed agreement-pattern counts over candidate pairs — no labels. +> **Out:** the fitted parameter vector (p, m, u) that Steps 2–3 consume, via EM on the latent class. + You never have labeled match/nonmatch pairs at census scale. Treat the class (M or U) of each pair as a latent variable and run EM over the observed agreement-pattern counts, fitting the parameter vector (p, m, u) where p is the proportion of matches among @@ -102,6 +132,9 @@ are needed — the weights stop being a clean per-field sum, but the decision ru ### Step 6 — Blocking: never score n² pairs +> **In:** the two files of records (n² pairs is infeasible). +> **Out:** a candidate-pair set — the union of several blocking passes on different keys. + Scoring every pair is quadratic death. Only generate candidate pairs that agree on a cheap blocking key (same postcode, same surname soundex), and run several passes with *different* keys so a typo in one key cannot hide a duplicate — the union of passes is @@ -116,13 +149,17 @@ the candidate set. pairs = Σ_buckets C(bucket,2) instead of C(n,2) ``` -Winkler's 2004 example: two files of roughly 10^8.5 records each imply ~10^17 raw pairs; -11 blocking criteria cut that to ~10^12 pairs while retaining 99.5% of true matches. -Database hook: a blocking key is a hash-partition key (topic 36), and multi-pass blocking -is just multiple shuffles over the same data. +Winkler's 2004 example self-matches the 2000 Decennial Census of 300 million records — +10^17 pairs (300M × 300M) — and shows that 11 blocking criteria cut that to a subset of +~10^12 pairs while retaining 99.5% of the true matches. Database hook: a blocking key is a +hash-partition key (topic 36), and multi-pass blocking is just multiple shuffles over the +same data. ### Step 7 — Production scale: BigMatch and the census pipeline +> **In:** the multi-pass blocking + scoring pipeline from Steps 3–6, at census scale. +> **Out:** BigMatch — all 10 passes evaluated in one streaming pass over the big file, at ~100k pairs/sec. + BigMatch is the Census Bureau's production blocking-and-matching engine: it handles workloads on the order of 100M × 4B record comparisons at roughly 100k pairs/sec, and — the performance-engineering punchline — evaluates all 10 blocking passes *simultaneously @@ -133,6 +170,9 @@ error rates, comparator microbenchmarks, and an engine that streams the big file ### Step 8 — The local experiment: er.rs, and one EM per blocking pass +> **In:** the whole pipeline (Steps 3–6) as the er.rs experiment on 15,000 synthetic records. +> **Out:** measured u, EM-fitted m and p, a 415× blocking cut, and precision/recall 0.989/0.992 at a 12-bit threshold. + The stub in `experiments/src/er.rs` generates 15,000 records over 5 fields with value pools [200, 500, 3650, 200, 2000] and typo rates [0.10, 0.07, 0.03, 0.12, 0.05]. You estimate u from random pairs (measured [0.0052 0.0021 0.0003 0.0051 0.0006], i.e. about @@ -187,15 +227,84 @@ weights, threshold + connected components); see the separate splink code guide. ## Done when +Answer each before unfolding it. + - [ ] You can write the two-threshold decision rule from memory and explain, in one paragraph, what Fellegi–Sunter proved optimal about it. + +
Answer + + Rule (Eq. 2): `R > T_μ` → match; `R < T_λ` → nonmatch; `T_λ ≤ R ≤ T_μ` → + clerical review (the boundaries themselves fall in the review band). Fellegi and + Sunter proved that among all decision rules holding the false-match rate ≤ μ and + the false-nonmatch rate ≤ λ, this likelihood-ratio rule *minimizes the + probability of the clerical (no-decision) region*. + + The band exists because a single threshold cannot hold both error rates under + their targets at once — the middle is where the evidence is genuinely + ambiguous, and routing only that band to humans is what keeps both automated + error rates bounded. That theorem is what justified automation: the 1990 Census + matching dropped from an estimated 3000 clerks over 3 months to 200 over 6 + weeks. + +
+ - [ ] You have hand-computed per-field match weights in bits from (m, u) and matched them to the experiment's measured values. + +
Answer + + Agreement weight is `+log2(m/u)`, disagreement is `+log2((1−m)/(1−u))`. From the + fitted `m = [0.80 0.86 0.94 0.78 0.90]`, `u = [0.0052 0.0021 0.0003 0.0051 + 0.0006]`, the agreement weights are `[+7.27 +8.68 +11.61 +7.26 +10.55]` and the + disagreement weights `[−2.31 −2.83 −4.06 −2.18 −3.32]`. + + dob pays the most on agreement (+11.61) because its 3650-value pool makes u tiny, + and it also punishes disagreement hardest (−4.06). A full five-field match sums + to 45.36 bits — far above the 12-bit link threshold — while the pattern last, + first, dob agree / city disagree / phone agree sums to 35.93 bits. + +
+ - [ ] Your er.rs run reproduces blocking (~271k pairs from ~112.5M), EM-fitted m within a point or two of (1−typo)², and precision/recall ≈ 0.989/0.992 at 12 bits. + +
Answer + + Blocking on last name and dob (unioned) turns 112,492,500 naive pairs into + 271,012 — a 415× cut. EM (one fixed-u session per pass, the blocked field + excluded) fits `m = [0.80 0.86 0.94 0.78 0.90]` against the analytic `(1−typo)² = + [0.81 0.87 0.94 0.77 0.90]` (within a point or two) and `p = 0.184`. Linking at a + 12-bit threshold plus union-find gives pair precision 0.989, recall 0.992 in + ~48 ms. + + Including the blocked field in its own session degenerates the fit: every blocked + pair agrees on the key by construction, so the field looks perfectly + discriminating and the fitted prior p is driven to 1.0 — which is why each pass + excludes its own blocking column. + +
+ - [ ] You can explain why exact string comparison loses over 25% of census matches and how Jaro–Winkler similarity is discounted into the weights. +
Answer + + On census data more than 25% of true matches disagree on a field's exact string, + purely from typos and scanning error — the hardest missed matches in Winkler's + Table 9 were children whose two records shared no name 3-grams at all. Exact + equality therefore throws those matches away before scoring even starts. + + Jaro's comparator scores partial similarity (common characters in a sliding + window, minus transpositions); Winkler's variant boosts it when the strings + share a common prefix, since typos cluster toward the ends of names. That + similarity in [0, 1] is folded into the weight by interpolating between the full + agreement weight `+log2(m/u)` and the disagreement weight, so a near-miss earns + partial positive bits instead of the full negative penalty. The likelihood-ratio + skeleton is unchanged; only the γ alphabet gets richer. + +
+ ## References - W. E. Winkler, "Overview of Record Linkage and Current Research Directions," US Census diff --git a/topics/39-fraud-identity-graphs/reading-flowscope.md b/topics/39-fraud-identity-graphs/reading-flowscope.md index 64a2725..4042691 100644 --- a/topics/39-fraud-identity-graphs/reading-flowscope.md +++ b/topics/39-fraud-identity-graphs/reading-flowscope.md @@ -16,6 +16,9 @@ and then reuse the same near-greedy peeling machinery FRAUDAR made famous. ### Step 1 — Layering: why laundering is shaped like a flow +> **In:** nothing yet — a stream of account-to-account transfers under per-account and per-pair reporting thresholds. +> **Out:** the observation that evasion forces a high-volume, balanced, multi-step flow: few sources, pass-through mules, few destinations. + Regulators impose per-account and per-pair reporting thresholds. Launderers respond with layering: split the dirty amount into many transfers and route them through middle ("mule") accounts that retain almost nothing. The result is a high-volume, @@ -34,6 +37,9 @@ in aggregate, but only visible when you require it to pass *through* the middle. ### Step 2 — Why dense-block methods miss it +> **In:** the layered-flow shape from Step 1. +> **Out:** why any per-hop dense-block score is blind — the anomaly is conjunctive, coupled through the mules. + FRAUDAR-style detectors score one bipartite block: rows X against columns Y, edges weighted by column degree. In a laundering ring, the X→W hop alone is not dense — each mule receives from only a few sources; the W→Y hop alone is equally bland. @@ -50,6 +56,9 @@ the two (or more) hops jointly, coupled through the mules. ### Step 3 — The k-partite transfer graph +> **In:** the conjunctive-signal requirement from Step 2. +> **Out:** the k-partite transfer graph (X → W → Y, k=3 in the paper) and the "pick a subset S, score it, optimize" template. + Model transfers as a k-partite graph: sources X in the first partite, one or more middle layers W, destinations Y in the last. The paper works out k=3 (X → W → Y) in full and generalizes to more middle layers for deeper laundering chains. An @@ -66,6 +75,9 @@ the score changes. ### Step 4 — Throughput f, imbalance q, and the score g(S) +> **In:** a candidate subgraph S spanning all partites (Step 3). +> **Out:** the score g(S), built from each mule's throughput f_i = min(in, out) and imbalance q_i = max(in, out). + For each middle account i inside a candidate subgraph S, define f_i = min(inflow, outflow) — the money that genuinely flows through — and q_i = max(inflow, outflow). The subgraph score is the size-normalized sum @@ -86,13 +98,21 @@ the metric rather than bolted on. ### Step 5 — Near-greedy peeling with a flow-aware heap key +> **In:** the metric g(S) from Step 4 to maximize. +> **Out:** the near-greedy peel with the Eq. (5) priority key, returning ˆS under Theorem 1's bound g(ˆS) ≥ (|M'|/|S'|)·(g(S*) − λε). + The optimizer is FRAUDAR's near-greedy peel: start from the full graph, repeatedly remove the node whose removal hurts g least, remember the best S seen, return it. -A priority queue keyed on each node's marginal contribution to g makes the whole -loop near-linear in edges, and the paper carries over a FRAUDAR-style approximation -guarantee for the returned subgraph. The delta from FRAUDAR: a middle node's key is -its (1+lambda) f_i − lambda q_i term, and peeling a source or destination changes -the inflow/outflow — hence f and q, hence the keys — of its middle-layer neighbors. +A priority tree keyed by Eq. (5) makes the whole loop near-linear in edges. The key is +role-dependent: a middle node v_i ∈ M is keyed by `w_i = f_i − (λ/(1+λ)) q_i` — +proportional to its g-contribution `(1+λ) f_i − λ q_i`, so the argmin peel order is +identical — while a source or destination node is keyed by its plain degree d_i. +Peeling a source or destination changes the inflow/outflow — hence f and q, hence the +keys — of its middle-layer neighbors. The paper proves an approximation bound +(Theorem 1): `g(ˆS) ≥ (|M'|/|S'|)·(g(S*) − λε)`, where ε is the largest camouflage +volume a laundering account exchanges with honest accounts. It is FRAUDAR's "first +optimal node removed" proof technique, but the constant is |M'|/|S'| (bounded below by +the mule count) and the slack is λε — not a flat ½. ``` while nodes remain: @@ -105,6 +125,9 @@ the inflow/outflow — hence f and q, hence the keys — of its middle-layer nei ### Step 6 — Why camouflage is self-defeating here +> **In:** the metric and peel from Steps 4–5. +> **Out:** the argument that any camouflage transfer raises some mule's q without raising f, so it lowers g — no column weights needed. + FRAUDAR resists camouflage via column weighting: camo edges land on honest high-degree columns and earn little. FlowScope needs no column weights at all. Any extra transfer a launderer adds to look normal lands on one side of some @@ -116,17 +139,26 @@ robustness goal, two very different mechanisms (reweighting vs metric shape). ### Step 7 — Evidence: CBank and CFD +> **In:** FlowScope run on the CBank and CFD datasets. +> **Out:** FAUC 0.761/0.843 on CBank (vs FRAUDAR 0.529/0.704) and F1 ≥ 0.9 down to $76M injected vs FRAUDAR's $180M. + CBank is a real bank dataset: 6.13M accounts and 43.98M transfer records, with a -labeled real laundering ring — 4 sources, 12 mules, 2 destinations moving about -452M yuan. On the two CBank injection settings FlowScope scores FAUC 0.761 and -0.843 versus FRAUDAR's 0.529 and 0.704. In injection experiments FlowScope holds -F1 at 0.9 or above down to injected laundering volumes of 76M yuan, where FRAUDAR -needs 180M — FlowScope detects laundering at less than half the volume. On the -Czech Financial Dataset (CFD) it reaches FAUC 0.970 and 0.900. The practical -reading: the flow objective buys sensitivity, not just elegance. +labeled real laundering ring of 4 sources, 12 mules, and 2 destinations (Fig. 1). Its +central mule v5 alone passes ≈452.1M yuan through — inflow ≈ outflow, so q5 − f5 ≈ 0 +and almost nothing is left in balance (Example 1); that near-zero residue is exactly +what g(S) rewards. On the two CBank injection settings FlowScope scores FAUC 0.761 and +0.843 versus FRAUDAR's 0.529 and 0.704 (7:5:3 A:M:C ratio). In the injection +experiments FlowScope holds F1 at 0.9 or above down to injected volumes of $76 million, +where FRAUDAR needs $180 million (paper units: million $) — FlowScope detects +laundering at less than half the volume. On the Czech Financial Dataset (CFD) it +reaches FAUC 0.970 and 0.900. The practical reading: the flow objective buys +sensitivity, not just elegance. ### Step 8 — Database-engineer lens: peeling, k-core, and streaming +> **In:** the static-snapshot peel from Steps 5–7. +> **Out:** the mapping to k-core peeling machinery and the open streaming/temporal-window gap for production AML. + The peel is the same degree-ordered elimination family as k-core decomposition (topic 18) — a lazy min-heap over a per-node key, with neighbor updates on each removal; everything you know about making k-core fast (bucketed keys, cache-aware @@ -177,15 +209,82 @@ It is a short AAAI paper; one careful pass suffices if you enter with the metric ## Done when +Answer each before unfolding it. + - [ ] You can write g(S) from memory and explain why parking and camouflage both lower it via q without touching f. + +
Answer + + `g(S) = (1/|S|) Σ_{mules i} [(1+λ) f_i − λ q_i]`, with `f_i = min(inflow, outflow)`, + `q_i = max(inflow, outflow)`, and λ = 4 (Eq. 4). Each mule contributes + `(1+λ) f_i − λ q_i`. A balanced mule (in 100, out 98) contributes + `5·98 − 4·100 = +90`. + + Parking money (in 100, out 10) gives f = 10, q = 100 → + `5·10 − 4·100 = −350`. Camouflage (in 130, out 98) gives f = 98, q = 130 → + `5·98 − 4·130 = −30`. Both raise q while f stays capped by the smaller side, so + the penalty `−λ(q − f)` drags the contribution down — the metric is maximized + exactly by dedicated, perfectly balanced mules, which is what a real ring looks + like. + +
+ - [ ] You can state the peel loop and the exact heap-key delta from fraudar.rs's bipartite version to the k=3 flow version (exercise 5). -- [ ] You can quote the CBank sensitivity result (F1 at 0.9 or above down to 76M - yuan vs FRAUDAR's 180M) and say what the injection protocol measures. + +
Answer + + The loop is FRAUDAR's near-greedy peel: start from all nodes, pop the + minimum-key node, remove it, update its neighbors' keys, and track the best g + seen. The delta is the key (Eq. 5): fraudar.rs keys every node by weighted + degree, whereas the flow version keys a *middle* node by + `f_i − (λ/(1+λ)) q_i` (proportional to its g-contribution, so the same peel + order) and a *source/destination* node by its plain degree d_i. + + Removals now propagate through the coupling: peeling a source changes its + mules' inflow, hence their f and q, hence their keys — a two-hop update the + bipartite version never performs. That is the whole engineering delta exercise + 5 asks you to write as pseudocode. + +
+ +- [ ] You can quote the CBank sensitivity result (F1 at 0.9 or above down to $76M + injected vs FRAUDAR's $180M) and say what the injection protocol measures. + +
Answer + + FlowScope holds F1 ≥ 0.9 down to $76 million of injected laundering volume, + where FRAUDAR needs $180 million (paper table, million $) — under half the + volume. The injection protocol plants a synthetic ring of known A:M:C ratio + (e.g. 7:5:3) and volume into the real transfer graph, then sweeps either the + injected money volume or the injected account count and records the lowest + setting at which F1 stays ≥ 0.9. + + It measures the faintest ring a detector can still recover against real + background banking traffic — a sensitivity floor, not a headline accuracy. + +
+ - [ ] You have a written position on the static-snapshot vs streaming-window gap for production AML. +
Answer + + The paper scores one static snapshot; production AML is streaming and + temporal — transfers arrive in windows, rings persist for weeks. A streaming + variant would maintain each mule's inflow/outflow (hence f_i and q_i) + incrementally under a sliding window, re-keying only the mules touched by an + arriving or expiring transfer, and re-peel incrementally instead of from + scratch. + + The open question is bounding how far a single transfer can move g, so you know + when a re-peel is actually needed. Your notes should take a position on window + length versus ring lifetime — too short a window and a slow ring never + accumulates detectable throughput. + +
+ ## References - Li, X., Liu, S., Li, Z., Han, X., Shi, C., Hooi, B., Huang, H., Cheng, X. diff --git a/topics/39-fraud-identity-graphs/reading-fraudar.md b/topics/39-fraud-identity-graphs/reading-fraudar.md index b0878ec..dc141c5 100644 --- a/topics/39-fraud-identity-graphs/reading-fraudar.md +++ b/topics/39-fraud-identity-graphs/reading-fraudar.md @@ -17,6 +17,9 @@ accounts deliberately add edges to popular honest objects to look normal.** ### Step 1 — Fraud rings are dense bipartite blocks +> **In:** nothing yet — a users × objects bipartite graph (followers × followees, reviewers × products). +> **Out:** the framing that a fraud ring is a near-biclique, so detection reduces to dense-subgraph mining. + The setting is a bipartite graph: users on one side, objects on the other (followers × followees, reviewers × products). A fraud ring is economically constrained — the operator owns a finite pool of accounts and sells engagement to a finite set of customers — @@ -27,6 +30,9 @@ anywhere, which is what the next step exploits against naive detectors. ### Step 2 — Camouflage: four ways to hide a block +> **In:** the dense-block framing from Step 1, plus the fact that the fraudster owns the user rows. +> **Out:** four camouflage attacks (random, biased, hijacked, reverse) that defeat any averaging or popularity-trusting metric. + Fraud accounts add extra edges to honest, popular objects so their degree and neighborhood statistics look organic. The paper studies four attack variants: @@ -49,19 +55,27 @@ incoming popularity gets fooled by 4. ### Step 3 — Axioms: what a suspiciousness metric must satisfy -Section 3 of the paper pins down three axioms for a block-suspiciousness metric g: +> **In:** the camouflage failure modes from Step 2 — the metric must survive them. +> **Out:** four axioms a suspiciousness metric must satisfy, and the admitted family g(S) = f(S)/|S|. + +Section 3 of the paper pins down **four axioms** for a block-suspiciousness metric g: -- **node suspiciousness** — a bigger or denser block is more suspicious than a smaller/sparser one; -- **edge suspiciousness** — adding an edge inside the block must increase suspicion; -- **concentration** — the same edge mass on fewer nodes is more suspicious. +- **node suspiciousness** (Axiom 1) — with size and edge weight fixed, a block of higher-suspiciousness nodes beats one of lower; +- **edge suspiciousness** (Axiom 2) — adding an edge inside the block must increase suspicion; +- **size** (Axiom 3) — with equal node/edge weights and equal edge density, a larger block is more suspicious; +- **concentration** (Axiom 4) — the same total suspiciousness on fewer nodes is more suspicious. -These rule out surprisingly many intuitive metrics (raw edge count ignores concentration; -edge fraction/density violates edge or node suspiciousness in various regimes). They admit the -family FRAUDAR uses: `g(S) = f(S) / |S|`, where S spans both sides and f(S) sums the weights +These rule out surprisingly many intuitive metrics: edge density ρ(S) violates **Size** (Axiom 3) +— it does not grow with |S| — and the total edge weight `Σ c_ij` violates **Concentration** +(Axiom 4), since it ignores how concentrated the mass is. Theorem 1 proves the family FRAUDAR +uses satisfies all four: `g(S) = f(S) / |S|`, where S spans both sides and f(S) sums the weights of edges with both endpoints inside S. Unweighted, g is average degree up to a factor of 2. ### Step 4 — Why unweighted average degree finds the wrong block +> **In:** the metric g(S)=f(S)/|S| from Step 3 with unit edge weights. +> **Out:** the failure — unweighted average degree ranks the organic power-user × hit-product core above the fraud block, and worse as camouflage grows. + On real graphs with skewed (Zipf-like) degree distributions, the densest set under unweighted average degree is not the fraud block — it is the power-users × hit-products core: the heaviest reviewers crossed with the most-reviewed products. That core is organically dense. Worse, @@ -73,6 +87,9 @@ not a smarter search — it is a smarter edge weight. ### Step 5 — Column weighting: agreement on a popular column is cheap +> **In:** the unweighted-metric failure from Step 4. +> **Out:** the camouflage-resistant edge weight c_ij = 1/log(d_j + 5), keyed on each column's global degree. + Weight each edge (i, j) into object j by its global column degree: ``` @@ -89,6 +106,9 @@ zero. Down-weighting is logarithmic, not a hard threshold, so mid-popularity col ### Step 6 — Theorem 3: camouflage provably cannot lower g(block) +> **In:** the weighted metric g with c_ij from Step 5. +> **Out:** Theorem 3 — camouflage never changes the block's in-edges or its column degrees, so g(block) is identical before and after. + This is the paper's core guarantee, and the argument is one picture: ``` @@ -110,6 +130,9 @@ which the column weighting already arranged by deflating the power-user core. ### Step 7 — Greedy peeling: exonerate the least suspicious, O(|E| log |V|) +> **In:** the weighted metric g to maximize over all subsets — an intractable search. +> **Out:** the greedy peel returning a set with g ≥ g_OPT/2 (Theorem 2) in O(|E| log |V|). + Maximizing g exactly is hopeless; FRAUDAR uses the classic peel. Start with all nodes, and repeatedly delete the node (either side) with minimum weighted degree — "exonerate the least suspicious" — recording g of every intermediate set; return the best prefix. @@ -131,14 +154,18 @@ node's degree also upper-bounds contributions in S*; averaging closes the factor ### Step 8 — What it finds in the wild +> **In:** the weighted peel from Step 7 run on injected-block and real graphs. +> **Out:** F above 0.95 (block densities ≥ 0.04) under all four attacks, and a 4031×4313 Twitter block at 68% density. + On real review graphs with injected 200×200 fraud blocks, FRAUDAR scores F above 0.95 under -*all four* camouflage attacks. On the Twitter follower graph (41.7M users, 1.47B edges) it -surfaced a 4031 × 4313 block at 68% edge density. Hand-labeling sampled block accounts found -57% fraudulent (a second sample gave 40%) versus 12–25% in degree-matched control samples; -many block accounts were created within the same short time window and tweeted follower-buying -links. Cross-topic hook: the peel is the same degree-ordered vertex elimination as k-core -decomposition — topic 18's GPU graph analytics implements exactly this loop with a bucketed -frontier instead of a heap. +*all four* camouflage attacks for block densities of at least 0.04 (§5.2). On the Twitter follower +graph (41.7M users, 1.47B edges) it surfaced a 4031 × 4313 block at 68% edge density. Hand-labeling +found 57% of the detected *followers* and 40% of the detected *followees* were fraudulent, deleted, +or suspended, against 25% in a degree-matched control and 12% in an unconditioned control (§5.3); +many block accounts were created within the same short time window and used the follower-buying +services TweepMe and TweeterGetter. Cross-topic hook: the peel is the same degree-ordered vertex +elimination as k-core decomposition — topic 18's GPU graph analytics implements exactly this loop +with a bucketed frontier instead of a heap. ## How to read the paper (with the concepts in hand) @@ -146,9 +173,9 @@ frontier instead of a heap. approximation guarantee, and the Twitter case study. You have the map from Steps 1–2. - **Section 2 (Related work).** One pass. Note which prior dense-block methods lack camouflage guarantees — this motivates the axiomatic reset in Step 3. -- **Section 3 (Problem / axioms).** Read carefully against Step 3. For each axiom, test it - mentally on raw edge count and on edge density; seeing them fail is the point. Confirm the - metric family `g = f/|S|` and that S mixes rows and columns. +- **Section 3 (Problem / axioms).** Read carefully against Step 3. Test edge density ρ(S) + (fails **Size**, Axiom 3) and the total edge weight `Σ c_ij` (fails **Concentration**, Axiom 4); + seeing them fail is the point. Confirm the metric family `g = f/|S|` and that S mixes rows and columns. - **Section 4.1–4.2 (Algorithm, Theorem 2).** Step 7 is your companion. Walk the peel proof: find the sentence fixing "the first time an optimal node is removed" and check the averaging argument. Map the data-structure claim to the lazy-heap variant you will implement. @@ -156,15 +183,16 @@ frontier instead of a heap. proof only uses two invariants: camouflage lands on honest columns, and block column degrees never change. Note c = 5 in `1/log(d_j + 5)` and the global-degree choice. - **Section 5 (Experiments).** Match the injection setup to Step 4's failure numbers and Step 8's - F above 0.95 across attacks. Then the Twitter results: block size, 68% density, 57% vs 12–25% - labeling, account-creation timing. Ask what a degree-matched control actually controls for. + F above 0.95 across attacks (densities ≥ 0.04). Then the Twitter results: block size, 68% density, + 57%/40% follower/followee labeling vs 25% and 12% controls, account-creation timing. Ask what a + degree-matched control actually controls for. - **After the paper.** Do the local experiment: implement `fraudar.rs` (lazy min-heap peel) over `review_graph.rs`'s generator, reproduce the unweighted-vs-weighted F table, and time the peel on the 100k × 50k-node / ~1.02M-edge graph (~0.2 s with the reference solution). ## Questions to answer in notes.md -1. Which of the three axioms does plain edge density `f(S)/(|S_rows| * |S_cols|)` violate, and +1. Which of the four axioms does plain edge density `f(S)/(|S_rows| * |S_cols|)` violate, and with what concrete counterexample block? 2. Theorem 3's proof needs column weights to depend on GLOBAL degree, fixed up front. What breaks — both in the guarantee and in peel complexity — if you recompute d_j inside the @@ -181,14 +209,91 @@ frontier instead of a heap. ## Done when -- [ ] You can state the three axioms and give one metric that fails each. +Answer each before unfolding it. + +- [ ] You can state the four axioms and give one metric that fails each. + +
Answer + + The four (§3) are Axiom 1 node suspiciousness, Axiom 2 edge suspiciousness, + Axiom 3 size, and Axiom 4 concentration. Edge density ρ(S) fails **Size**: it + does not grow with |S|, so a larger block at the same density scores no higher. + The total edge weight `Σ c_ij` fails **Concentration**: it ignores how the mass + is spread, so smearing the same weight over more nodes scores the same. + + FRAUDAR's `g(S) = f(S)/|S|` satisfies all four (Theorem 1). Note that unweighted + average degree also satisfies the axioms — the axioms are necessary, not + sufficient; camouflage resistance is a *separate* property that needs the column + weight of Step 5. + +
+ - [ ] You can reproduce Theorem 3's argument from memory as the two-line "block edges and block column degrees are untouched" invariant. + +
Answer + + `f(block)` counts only edges with *both* endpoints inside the block. Camouflage + edges run from fraud rows to honest columns, so they end outside the block and + are never counted. And `c_ij = 1/log(d_j + 5)` depends only on the column's + *global* degree `d_j`; camouflage aimed at honest columns never changes the + fraud columns' degrees. + + Both quantities that determine `g(block) = f(block)/|S|` are therefore identical + before and after camouflage (§4.3), so the fraudster cannot push the block's own + score down — only raise the score of honest-looking sets, which the column + weighting has already deflated. + +
+ - [ ] You can sketch the ½-approximation proof shape (first optimal node peeled + averaging). + +
Answer + + Theorem 2 gives `g(returned) ≥ g_OPT/2`. Look at the first moment the peel + removes a node belonging to the optimal set S*. At that instant every surviving + node has weighted degree at least `g(current set)` — otherwise it, not this one, + would have been peeled first. + + That per-node degree bound also caps each node's edge contribution within S*. + Since each edge is charged to at most two endpoints, summing the bound over S* + and dividing by |S*| gives `g(S*) ≤ 2·g(current) ≤ 2·g(returned)`. The peel + itself is `O(|E| log|V|)` with a priority tree keyed on weighted degree. + +
+ - [ ] Your `fraudar.rs` reproduces log-weighted F = 1.00 at camo 0/0.5/1/2 while unweighted degrades to 0.65, and peels the ~1.02M-edge graph in about 0.2 s. + +
Answer + + The lane shows unweighted peeling's F degrading 1.00 / 0.95 / 0.69 / 0.65 as + camouflage goes 0 / 0.5 / 1 / 2 edges per fraud edge, while the log-weighted + peel holds F = 1.00 across all four. The ~1.02M-edge graph (100k users × 50k + objects) peels in ≈0.2 s with the reference solution. + + The weighting `c_ij = 1/log(d_j + 5)` is the *only* change between the two runs; + the search — greedy minimum-weighted-degree peeling — is identical, which is + the point of Theorem 3: the fix is the edge weight, not a smarter algorithm. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + This is a worklog check. The five questions test: which axiom edge density + violates (Size, Axiom 3); why global-degree weights are load-bearing for *both* + Theorem 3 and the `O(|E| log|V|)` bound; why reverse camouflage still fails + despite changing fraud-column degrees; the opposite monotonicity of the two + naive rankers (the repo's headline result); and how to port the weighted peel + to a bucketed GPU frontier. + + Confirm each has a written answer in notes.md before ticking — the point is to + force the reasoning, not to look up an answer here. + +
+ ## References - Hooi, Song, Beutel, Shah, Shin, Faloutsos. *FRAUDAR: Bounding Graph Fraud in the Face of diff --git a/topics/39-fraud-identity-graphs/reading-splink.md b/topics/39-fraud-identity-graphs/reading-splink.md index fe49097..7c228c1 100644 --- a/topics/39-fraud-identity-graphs/reading-splink.md +++ b/topics/39-fraud-identity-graphs/reading-splink.md @@ -16,6 +16,9 @@ and ties each stage back to the miniature Rust reimplementation in this topic's ### Step 1 — The Linker façade: a model is comparisons + blocking rules +> **In:** nothing yet — a settings object (comparison ladders + blocking rules) and a chosen `db_api` dialect. +> **Out:** a `Linker` whose every downstream method emits generated SQL for one of the five stages below. + `Linker` (`linker.py:66`) is the API surface. You construct it with a settings object — a list of comparisons (how to compare each field, as a ladder of agreement levels) and a list of blocking rules (which candidate pairs to even look at) — plus a `db_api` that @@ -36,6 +39,9 @@ The five-stage pipeline: ### Step 2 — Blocking: SQL self-joins instead of n² pairs +> **In:** the raw record table and the blocking rules from Step 1's settings. +> **Out:** a deduplicated candidate-pair table — 271,012 pairs from the 112,492,500 full space (415×, experiment). + `block_using_rules_sqls` (`blocking.py:747`) compiles each blocking rule into a SQL self-join (e.g. equal `dob`), and unions multiple rules with deduplication so a pair matched by two rules is scored once. Before committing to a rule, the pre-flight analysis @@ -57,6 +63,9 @@ surname) becomes a hot partition exactly like a skewed shard. ### Step 3 — u probabilities: random pairs are almost all nonmatches +> **In:** the raw records — no labels, no blocking. +> **Out:** per-level u = [0.0052 0.0021 0.0003 0.0051 0.0006], the nonmatch agreement rates. + `estimate_u_using_random_sampling` (`linker_components/training.py:163`) estimates u — the probability that a *nonmatching* pair agrees on a comparison level — by sampling random record pairs and requiring no labels at all: in the full n² space, almost every @@ -69,16 +78,22 @@ sampling almost never draws a match, which is why m needs EM on blocked pairs (S ### Step 4 — m and the prior via EM, one session per blocking rule +> **In:** the blocked candidate pairs from Step 2, with the u's from Step 3 held fixed. +> **Out:** per-level m = [0.80 0.86 0.94 0.78 0.90] and the prior p = 0.184. + `estimate_parameters_using_expectation_maximisation` (`linker_components/training.py:231`) runs ONE EM session per blocking rule, and — the crucial trick — *excludes* the comparisons on the blocking-rule columns from that session: every candidate pair agrees on them by construction, so including them degenerates the fit. m values estimated by multiple sessions are averaged, and the prior p is fitted alongside. The core loop is -`expectation_maximisation` (`expectation_maximisation.py:225`); the E-step SQL is built at -`expectation_maximisation.py:18` (`compute_new_parameters_sql`), the M-step -(`maximisation_step`) at `:193`. E-step: score every blocked pair with current m/u/p to -get a match probability. M-step: recompute m, u, p as probability-weighted agreement -rates. Repeat to convergence — each iteration is one SQL round trip. +`expectation_maximisation` (`expectation_maximisation.py:225`), iterating from `:253`. +E-step (labelled `# Expectation step` at `:260`): the prediction SQL is built by +`predict_from_comparison_vectors_sqls` (called at `:268`) — score every blocked pair with +current m/u/p to get a match probability. M-step: `compute_new_parameters_sql` (defined +`:45`, "compute m and u counts from the results of predict", called `:278`) aggregates +those probabilities into new m/u counts, and `maximisation_step` (`:193`, called `:293`) +writes them back as the new parameters. Repeat to convergence — each iteration is one SQL +round trip. ``` session A: block on last_name session B: block on dob @@ -95,6 +110,9 @@ p races to 1.0; mask it and you get m = [0.80 0.86 0.94 0.78 0.90], p = 0.184. ### Step 5 — Comparison ladders: graded agreement, per-level m/u +> **In:** the trained m and u from Steps 3–4, one pair attached to each ladder rung. +> **Out:** a per-level match weight of log2(m/u) bits. + A comparison is not boolean. `ComparisonLevel` (`comparison_level.py:148`) represents one rung of an ordered ladder, each rung carrying its own m (`comparison_level.py:190`) and u (`:191`); the library ships graded levels like `LevenshteinLevel` @@ -112,6 +130,9 @@ so the "else" rung soaks up disagreement and carries a negative weight. ### Step 6 — Term frequency: "Smith" is worth fewer bits +> **In:** a scored ladder rung from Step 5 plus a per-column token-frequency table. +> **Out:** the same weight scaled by the token's rarity — an additive correction in bits. + `_tf_adjustment_sql` (`comparison_level.py:667`) implements term-frequency adjustment: agreeing on the surname "Smith" carries less evidence than agreeing on a rare surname, so the adjustment scales the level's weight by the token's frequency relative to the @@ -124,6 +145,9 @@ reused across predictions, and the adjustment is a multiplicative factor on the ### Step 7 — Predict: sum the bits, squash to a probability +> **In:** the candidate pairs (Step 2), the trained per-level weights (Steps 5–6), and the prior (Step 4). +> **Out:** a match probability per pair, 1/(1+2^(-mw)). + `predict()` (`linker_components/inference.py:294`) is the user-facing scoring entry point: block, compute the comparison-vector level per field per pair, then sum. The pairwise scoring SQL is assembled in `predict.py:42`, and `_combine_prior_and_mws` (`predict.py:203`) @@ -145,14 +169,17 @@ richer dependency model, because independence is what keeps scoring a single SQL ### Step 8 — Clustering: connected components in SQL +> **In:** the scored pairs from Step 7, thresholded at 12 bits. +> **Out:** one cluster id per record — the connected components of the thresholded graph. + `cluster_pairwise_predictions_at_threshold` (`linker_components/clustering.py:43`) -thresholds the pairwise scores and calls the iterative connected-components algorithm in -`graph_operations/connected_components.py:121` — a union-find-equivalent computed as -repeated SQL passes until representatives stabilize, yielding one cluster id per record. -The experiment does the same with an in-memory union-find: linking at 12 bits gives -precision 0.989, recall 0.992, in 48 ms for 15k records — same algorithm, different +thresholds the pairwise scores and calls the iterative connected-components solver +`solve_connected_components` (`connected_components.py:121`) — a union-find-equivalent +computed as repeated SQL passes until representatives stabilize, yielding one cluster id +per record. The experiment does the same with an in-memory union-find: linking at 12 bits +gives precision 0.989, recall 0.992, in 48 ms for 15k records — same algorithm, different substrate. The dialect layer (`dialects.py:24`, `SplinkDialect`) is why all of this ports: -DuckDB at `:270`, Spark at `:402`, SQLite at `:532`, PostgreSQL at `:674` — one model, +DuckDB at `:270`, Spark at `:402`, SQLite at `:532`, PostgreSQL at `:573` — one model, four engines. Every stage in this guide — blocking, EM, prediction, clustering — is generated SQL, which is the whole reason the same settings object runs single-node DuckDB during development and a Spark cluster in production. @@ -168,13 +195,13 @@ All paths under `splink/internals/` in `~/repos/splink` @ `04189f5`. | 2 | Pre-flight comparison counts per blocking rule | `blocking_analysis.py:349` | | 3 | u from random sampling, no labels | `linker_components/training.py:163` | | 4 | One EM session per blocking rule, blocked columns excluded | `linker_components/training.py:231` | -| 4 | EM core loop; E-step SQL; M-step | `expectation_maximisation.py:225`, `:18`, `:193` | +| 4 | EM core loop; E-step SQL; M-step | `expectation_maximisation.py:225`, E-step `:268`, M-step `:45`/`:193` | | 5 | `ComparisonLevel`; m at `:190`, u at `:191`; weight `log2(m/u)` | `comparison_level.py:148`, `:426` | | 5 | Graded levels: Levenshtein / JaroWinkler / Jaro | `comparison_level_library.py:406`, `:458`, `:493` | | 6 | Term-frequency adjustment SQL | `comparison_level.py:667` | | 7 | `predict()` entry point; scoring SQL; prior + weights → probability | `linker_components/inference.py:294`, `predict.py:42`, `:203` | -| 8 | Threshold + clustering; iterative connected components in SQL | `linker_components/clustering.py:43`, `graph_operations/connected_components.py:121` | -| all | `SplinkDialect`: DuckDB/Spark/SQLite/PostgreSQL | `dialects.py:24`, `:270`, `:402`, `:532`, `:674` | +| 8 | Threshold + clustering; iterative connected components in SQL | `linker_components/clustering.py:43`, `connected_components.py:121` | +| all | `SplinkDialect`: DuckDB/Spark/SQLite/PostgreSQL | `dialects.py:24`, `:270`, `:402`, `:532`, `:573` | ## Questions to answer in notes.md @@ -186,11 +213,80 @@ All paths under `splink/internals/` in `~/repos/splink` @ `04189f5`. ## Done when +Answer each before unfolding it. + - [ ] You can sketch the five-stage pipeline from memory and name the file that owns each stage. + +
Answer + + Blocking compiles rules to SQL self-joins in `block_using_rules_sqls` + (`blocking.py:747`); u comes from `estimate_u_using_random_sampling` + (`training.py:163`); m and the prior p come from + `estimate_parameters_using_expectation_maximisation` (`training.py:231`), + whose loop is `expectation_maximisation` (`expectation_maximisation.py:225`); + scoring is `predict()` (`inference.py:294`) assembling SQL in `predict.py:42`; + clustering is `cluster_pairwise_predictions_at_threshold` (`clustering.py:43`) + calling `solve_connected_components` (`connected_components.py:121`). + + The `Linker` façade (`linker.py:66`) owns them all, and none of them touches a + record in Python — every stage is a generated SQL string handed to the chosen + `SplinkDialect` (`dialects.py:24`). + +
+ - [ ] You can explain, in bits, how a pair's match weight is assembled (prior + per-level `log2(m/u)` + TF adjustment) and squashed via `1/(1+2^(-mw))`. + +
Answer + + Each field lands on the first ladder rung it satisfies, contributing that + rung's `log2(m/u)` bits (`comparison_level.py:426`). Term frequency adds a + per-token correction (`_tf_adjustment_sql`, `comparison_level.py:667`): an + agreement on a rare surname is worth more bits than one on "Smith". The prior + enters as `log2(p/(1-p))` in `_combine_prior_and_mws` (`predict.py:203`); with + the experiment's p = 0.184 that is `log2(0.184/0.816) ≈ -2.15` bits. + + All the per-field weights simply add because Fellegi–Sunter assumes + conditional independence (naive Bayes), so the total match weight `mw` is a + sum, and the probability is the logistic squash `1/(1+2^(-mw))` — a match + weight of 0 bits means an even-odds pair at p = 0.5. + +
+ - [ ] You can state why each EM session masks its own blocking columns, and reproduce the degeneracy in the local experiment. + +
Answer + + A session blocked on `dob` only ever sees pairs that already agree on `dob`, + so the observed agreement rate on that field is 1.0 for matches and nonmatches + alike. If EM is allowed to fit `m(dob)` from that, the field looks perfectly + discriminating and the fitted prior p races to 1.0 to explain the universal + agreement. Excluding the blocking-rule comparisons from the session + (`training.py:231`) removes the degenerate column. + + The experiment reproduces both branches: keep the blocked field in and p → 1.0; + mask it and you recover m = [0.80 0.86 0.94 0.78 0.90], p = 0.184. m values + from several sessions are then averaged, which is why one masked column per + session still yields an estimate for every field. + +
+ - [ ] You have run or re-read `experiments/src/er.rs` and matched each of its phases (u sampling, masked EM, blocking, 12-bit threshold, union-find) to its splink counterpart. +
Answer + + `er.rs` mirrors the pipeline: 200k random pairs give + u = [0.0052 0.0021 0.0003 0.0051 0.0006] (Step 3); masked EM yields + m = [0.80 0.86 0.94 0.78 0.90], p = 0.184 (Step 4); blocking cuts + 112,492,500 pairs to 271,012 (415×, Step 2); a 12-bit threshold plus an + in-memory union-find gives precision 0.989, recall 0.992 in 48 ms for 15k + records (Step 8). + + Each phase is the same algorithm as its splink counterpart on a different + substrate: the Rust reimplementation runs arrays in memory, while splink emits + the identical logic as SQL for DuckDB, Spark, SQLite, or PostgreSQL. + +
+ ## References - Repo: `~/repos/splink` @ commit `04189f5` (moj-analytical-services/splink) — read under `splink/internals/`. diff --git a/topics/40-security-attack-graphs/README.md b/topics/40-security-attack-graphs/README.md index 8bdbcc0..520253e 100644 --- a/topics/40-security-attack-graphs/README.md +++ b/topics/40-security-attack-graphs/README.md @@ -51,7 +51,7 @@ the collector**, not of how much privilege exists. Mean shortest path: An edge means *control of the source yields control of the target*. That single rule makes 104 Active Directory concepts into one graph -with 63 traversable edge kinds, and it makes the pentest report a +with 64 traversable edge kinds, and it makes the pentest report a shortest-path query. `HasSession` is the edge that ruins everything: a privileged token sitting on a workstation makes every local admin of that box a domain admin, transitively — measured above, two misplaced @@ -192,7 +192,7 @@ Leopard itself at 1.56M QPS and a **150 µs** median. | anchor (`packages/go/`) | what to see | |---|---| | `graphschema/ad/ad.go:28` | 104 `StringKind` node and edge kinds — the whole ontology as constants | -| `graphschema/ad/ad.go:1160` | `PathfindingRelationships` — the 63 kinds an attacker may traverse | +| `graphschema/ad/ad.go:1160` | `PathfindingRelationships` — the 64 kinds an attacker may traverse | | `graphschema/ad/ad.go:1172` | `PostProcessedRelationships` — 31 kinds that are *derived*, not collected | | `analysis/analysis.go:346` | `newPipeline` — AD post-processing → Azure → tagging → data quality | | `analysis/analysis.go:104` | `ExpandGroupMembershipPaths` — nesting expansion as a path query | diff --git a/topics/40-security-attack-graphs/notes.md b/topics/40-security-attack-graphs/notes.md index 9d85aa5..8ba10f4 100644 --- a/topics/40-security-attack-graphs/notes.md +++ b/topics/40-security-attack-graphs/notes.md @@ -70,7 +70,7 @@ analysis. | split code/data t-tags: **1305×** vs 4.68× for a single t-tag (forward analysis) | SLEUTH Table 11 | | 174 entities correctly identified, **0 incorrectly, 2 missed** across 8 campaigns | SLEUTH Table 7 | | >99.9% of audit events were benign activity | SLEUTH §6.3 | -| BloodHound: **104** node/edge kinds, **63** traversable, **31** derived by post-processing | `graphschema/ad/ad.go` | +| BloodHound: **104** node/edge kinds, **64** traversable, **31** derived by post-processing | `graphschema/ad/ad.go` | ## Cross-topic threads (worked) diff --git a/topics/40-security-attack-graphs/reading-attack-graph-monotonicity.md b/topics/40-security-attack-graphs/reading-attack-graph-monotonicity.md index 0a57703..347a042 100644 --- a/topics/40-security-attack-graphs/reading-attack-graph-monotonicity.md +++ b/topics/40-security-attack-graphs/reading-attack-graph-monotonicity.md @@ -21,6 +21,9 @@ ever un-establishes a fact, the two questions have the same answer.** ### Step 1 — The state-based attack graph, and its size +> **In:** the network as boolean state variables, exploits as state transitions. +> **Out:** why the state-based attack graph is exponential — 229 bits → 2²²⁹ reachable states at five hosts — and the one-line escape the rest of the chapter earns. + Sheyner et al. (Oakland'02) model the network as a collection of boolean variables — which service runs where, which host trusts which, what privilege the attacker holds on each machine — and an exploit as a state transition. The attack graph is then the set of reachable states with @@ -41,6 +44,9 @@ off. ### Step 2 — Attributes and exploits +> **In:** the escape hint from Step 1 — count the *facts* an attacker can establish, not the states. +> **Out:** the flat model — attributes (the graph's nodes) and exploits (pre/postcondition transforms) — and why per-host instantiation makes the exploit count quadratic (two-host) or cubic (three-host). + The replacement model is deliberately flat. Let `A = {a₀ … a_N}` be **attributes**: atomic facts about the system. An attribute can be a vulnerability ("host 3 runs a vulnerable sshd"), a connectivity fact ("host 1 can reach host 2 on the ftp port"), a trust fact ("host 2's .rhosts @@ -67,6 +73,9 @@ vulnerabilities × 3 hosts = 18 attributes, 3 connectivity relations × 9 host p ### Step 3 — Monotonicity, stated precisely +> **In:** attributes and exploits from Step 2. +> **Out:** the exact assumption — a fact, once satisfied, is never un-satisfied — with its three technical consequences and the disjointness corollary the polynomial bound rests on. + > "The precondition of a given exploit is never invalidated by the successful application of > another exploit. In other words, the attacker never needs to backtrack." @@ -84,6 +93,9 @@ never has an attribute as both input and output. ### Step 4 — Where the assumption bends, and why it survives +> **In:** the monotonicity assumption from Step 3. +> **Out:** the three canonical non-monotone exploits — port forward, code green, the sshd-crash postcondition — and the argument that modelling each monotonically loses nothing an attacker could not recover. + The paper is honest about this and the examples are worth remembering: - **`port forward`** genuinely consumes a port on the middleman, so that port is now unavailable @@ -102,6 +114,9 @@ are choosing a model, not proving a theorem about reality. ### Step 5 — `markAttributes`: BFS over facts, in layers +> **In:** a monotone exploit set and the initially satisfied attributes. +> **Out:** the layered fixpoint that marks every reachable attribute with the round it first became satisfied, and its **O(|A|²·|E|)** cost. + With monotonicity, forward reachability is a fixpoint computed layer by layer. Layer 1 is everything one exploit can establish from the initial state; layer n is everything reachable in n chained exploits. @@ -125,6 +140,9 @@ exploits to establish the attribute, which is what `findShort` later uses. ### Step 6 — The three analyses you get for free +> **In:** the marked attribute/exploit graph from Step 5 and a goal attribute. +> **Out:** findMinimal / findAll / findShort with their three correctness results, and where NP-completeness actually sits (minimum-cardinality, not minimal). + Once the marked attribute/exploit graph exists, you do not need to materialize an attack tree: - **`findMinimal(S, att)`** — one minimal attack: recursively pick a minimal exploit set covering @@ -142,6 +160,9 @@ difference before you promise an optimizer. ### Step 7 — Cut sets: §2.3, one paragraph, the whole defensive story +> **In:** the marked graph and a goal attribute. +> **Out:** the defensive question — which nodes or edges to remove to disconnect the goal from the initial state — reduced to standard graph algorithms, and this repo's lane-2 dominator-tree realization of it. + > "It is also useful to think in terms of 'cut sets' of either exploits or attributes. These > approaches ask the question: what set of exploits (edges) or attributes (nodes) in our graph > must be removed to disconnect the goal state from the initial state? Standard graph analysis @@ -155,6 +176,9 @@ reachability re-runs, agreeing on every node. ### Step 8 — MulVAL: the fixpoint is a Datalog derivation +> **In:** the same monotone model, re-expressed as Datalog interaction rules. +> **Out:** the derivation graph (AND derivation nodes, OR fact nodes) that tabled evaluation produces, and the three complexity theorems — O(N²) steps, O(N²) graph size, O(N² log N) to build. + Ou, Boyer & McQueen (CCS'06) make the same move as a logic program. A MulVAL interaction rule: ```prolog @@ -195,6 +219,9 @@ MulVAL is at ~1 second at 50. ### Step 9 — Cycles and "useless edges" +> **In:** a derivation graph that tabling already kept from looping during evaluation. +> **Out:** why the *recorded trace* can still contain meaningless back edges, and the paper's derivability-based definition of a useless edge (not a DFS heuristic, which Fig 8 shows is wrong). + Tabling stops the *evaluation* from looping, but the recorded trace can still contain cycles, because two rules can be mutually satisfiable: @@ -264,15 +291,103 @@ you have read topic 27, this is stratified negation and provenance-tracking terr ## Done when +Answer each before unfolding it. + - [ ] You can state monotonicity in one sentence and list its three technical consequences. + +
Answer + + One sentence: *the precondition of an exploit is never invalidated by another + exploit's success — the attacker never has to backtrack.* Three consequences + (Ammann §2): (1) attributes go *unsatisfied → satisfied* and never the + reverse; (2) **no negation in preconditions**, since an unsatisfied attribute + can still become satisfied later; (3) pre- and postconditions are + **conjunctions** — a disjunctive precondition is modelled by splitting the + exploit in two. Plus the corollary the bound uses: `preConds(e) ∩ postConds(e) + = ∅`. + +
+ - [ ] You can derive `O(|A|²·|E|)` from the two facts the paper gives. + +
Answer + + Fact one: `Uₙ` only grows and is bounded by `A`, so there are at most `|A|` + layers. Fact two: because `preConds(e) ∩ postConds(e) = ∅`, each layer applies + every exploit at most once against the newly satisfied attributes, i.e. at + most `|A|·|E|` work. `|A|` layers × `|A|·|E|` per layer = **O(|A|²·|E|)**. The + layer index is not bookkeeping — it is the minimum number of chained exploits + to reach the attribute, which is exactly what `findShort` consumes. + +
+ - [ ] You can explain the difference between minimal and minimum attacks, and which is NP-complete. + +
Answer + + A **minimal** attack is locally irreducible: remove any one exploit and it no + longer reaches the goal. `findMinimal` returns one in `O(|E|²)`. A + **minimum-cardinality** attack is the globally smallest such set; finding it + is **NP-complete** (Sheyner et al.). A minimal attack can be far larger than + the minimum — "minimal" promises only that nothing in *this* set is redundant, + not that no smaller set exists. + +
+ - [ ] You can draw a logical attack graph with both node types and say which is AND and which OR. + +
Answer + + Two node types (MulVAL Figs 4–5): a **derivation node** ▭ is one rule + application and is an **AND** — every child fact must hold. A **fact node** ◯ + is one attribute and is an **OR** — any incoming derivation suffices. + **Primitive facts** ● are leaves supplied by the scanner. So a fact is true if + *any* derivation of it fires; a derivation fires only if *all* its + precondition facts are true. + +
+ - [ ] You can quote §2.3's cut-set paragraph and connect it to the dominator tree in `chokepoint.rs`. -- [ ] Your `chokepoint.rs` reproduces lane 2: exact agreement with the naive oracle on every node, - and the tiered/flat contrast. + +
Answer + + §2.3: "what set of exploits (edges) or attributes (nodes) … must be removed to + disconnect the goal state from the initial state? Standard graph analysis + algorithms can be applied." In the reverse graph rooted at the goal, node `d` + dominates `u` iff every path from `u` to the goal crosses `d`, so a single + dominator-tree pass prices every single-node cut exactly. Lane 2 measures it + at **0.8 ms** versus **543 ms** for 3400 individual reachability re-runs, + agreeing with the naive oracle on every node. + +
+ +- [ ] Your `chokepoint.rs` reproduces lane 2: exact agreement with the naive oracle on every node, and the tiered/flat contrast. + +
Answer + + Tiered directory: the top choke point covers **1992 / 2000 = 99.6%** of + exposure, and the greedy cut collapses the reachable set `2000 → 8 → 5`. Flat + directory: **no single-node cut frees anyone** — only the gateway cut + (2000×5 → 8) is structural. The dominator pass and the per-node naive oracle + return the *same* verdict on every node; dominators just deliver it in 0.8 ms + instead of 543 ms. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five: (1) monotonicity of `MemberOf` / `HasSession` / `GenericAll` and + what monotone `HasSession` over-reports; (2) where the O(N⁶)-vs-O(N²) + accounting difference goes; (3) reconstructing the §3 layer numbers and why + `findShort` needs them but `findMinimal` does not; (4) incremental maintenance + of the derivation graph under config changes (topic-27 IVM, and what a + retraction means when the analysis is monotone); (5) the flat-directory no-cut + result restated as a statement about the minimum cut set. + +
+ ## References - Ammann, Wijesekera, Kaushik. *Scalable, Graph-Based Network Vulnerability Analysis.* CCS 2002, diff --git a/topics/40-security-attack-graphs/reading-bloodhound.md b/topics/40-security-attack-graphs/reading-bloodhound.md index 7753c95..e093673 100644 --- a/topics/40-security-attack-graphs/reading-bloodhound.md +++ b/topics/40-security-attack-graphs/reading-bloodhound.md @@ -18,6 +18,9 @@ company — and no per-object permission review can see the difference.** ### Step 1 — An edge means "control of the source yields control of the target" +> **In:** the directory as a list of objects, each carrying an ACL. +> **Out:** the graph reframe — nodes are principals or resources, edges are rights oriented so that traversal *is* privilege escalation — and why "can Alice become Domain Admin?" is a shortest-path query the AD console cannot ask. + That one sentence is the entire data model. A *node* is a principal or a resource: a user, a group, a computer, a GPO, a certificate template. An *edge* is a right, oriented so that traversal means privilege escalation. `Alice -MemberOf-> Engineering` means controlling Alice @@ -27,37 +30,47 @@ Admin?" is `MATCH p = shortestPath((alice)-[*1..]->(da))` and nothing else. The felt like a revelation in 2016 is that the directory's own tooling has no way to ask it: the console shows you one object's ACL at a time, and composition is invisible one object at a time. -### Step 2 — The ontology: 104 kinds, 63 of them traversable +### Step 2 — The ontology: 104 kinds, 64 of them traversable + +> **In:** the edge-means-control model from Step 1. +> **Out:** the kind namespace (104 `StringKind` constants) and its four purposeful partitions — `Relationships` (88 edge kinds), `ACLRelationships` (30), `PathfindingRelationships` (64), `PostProcessedRelationships` (31) — and why the pathfinding subset is a query-time edge-kind filter. `graphschema/ad/ad.go:28` onward is a wall of constants — `graph.StringKind("User")`, `StringKind("GenericAll")`, `StringKind("ADCSESC1")` — 104 of them, node kinds and edge kinds in one namespace. What matters is that the file then partitions them into *purposeful* sets: ```go -func Relationships() []graph.Kind // ad.go:1151 — everything -func ACLRelationships() []graph.Kind // ad.go:1154 — rights that come from a DACL -func PathfindingRelationships() []graph.Kind // ad.go:1160 — the 63 an attacker may walk -func PostProcessedRelationships() []graph.Kind // ad.go:1172 — the 31 that are DERIVED +// graphschema/ad/ad.go — the four partitions (non-adjacent in the file; each line keeps its real number) +1151 func Relationships() []graph.Kind // everything: 88 edge kinds +1154 func ACLRelationships() []graph.Kind // 30 rights that come from a DACL +1160 func PathfindingRelationships() []graph.Kind // the 64 an attacker may walk +1172 func PostProcessedRelationships() []graph.Kind // the 31 that are DERIVED ``` -`PathfindingRelationships` is the attacker's alphabet, and it is smaller than the full set: some -edges (`Contains`, structural containment) exist for display or for post-processing but do not -by themselves grant control. A traversal that ignores this distinction reports paths that are not -attacks. This is a *query-time edge-kind filter*, and it is exactly the mask your Cypher engine -has to push down into the CSR scan — see the capstone. +`PathfindingRelationships` is the attacker's alphabet, and it is smaller than the full 88-kind edge +set: some collected rights are deliberately *excluded* because they do not, on their own, grant +control. `GetChanges` and `GetChangesAll` — the two directory-replication rights — are a clean +example: both appear in `Relationships` and `ACLRelationships` (they are real, collected ACEs), but +neither is in `PathfindingRelationships`, because only their *conjunction* is dangerous, and +post-processing synthesizes that conjunction into a single `DCSync` edge (Step 3). A traversal that +walked `GetChanges` alone would report a path that is not an attack. This is a *query-time edge-kind +filter*, and it is exactly the mask your Cypher engine has to push down into the CSR scan — see the +capstone. ### Step 3 — 31 edge kinds are materialized views +> **In:** the four partitions, and the 31-kind `PostProcessedRelationships` set. +> **Out:** why edges like `AdminTo` / `DCSync` / `ADCSESC*` are *derived* by a post-ingest pass and written back as real edges — a batch materialized view, which is topic 27's question in disguise. + `AdminTo` is not collected. Nor is `CanRDP`, nor `DCSync`, nor the ADCS certificate-abuse family `ADCSESC1..ADCSESC13`. They are *derived* by a post-processing pass and written back into the graph as real edges. `analysis/ad/post.go:84`: ```go -// PostDCSync: an attacker who holds both GetChanges and GetChangesAll on the domain -// can replicate secrets — so synthesize one DCSync edge instead of making every -// query re-derive the conjunction. -func PostDCSync(ctx context.Context, db graph.Database, localGroupData *LocalGroupData) - (*post.AtomicPostProcessingStats, error) +// analysis/ad/post.go — an attacker holding both GetChanges and GetChangesAll on the domain +// can replicate secrets, so post-processing synthesizes one DCSync edge instead of making +// every query re-derive the conjunction. +84 func PostDCSync(ctx context.Context, db graph.Database, localGroupData *LocalGroupData) (*post.AtomicPostProcessingStats, error) { ``` The trade is the one topic 1 calls RUM and topic 27 calls incremental view maintenance: pay once @@ -68,14 +81,28 @@ misconfiguration into a single edge. The whole pipeline is declared in one place `analysis/analysis.go:346`: ```go -func newPipeline() analysisPipeline { - return analysisPipeline{ - {analysisStep: model.AnalysisStepADPostProcessing(), operation: adPostProcessingOperation}, - {analysisStep: model.AnalysisStepAzurePostProcessing(), operation: azurePostProcessingOperation}, - {analysisStep: model.AnalysisStepTagging(), operation: taggingOperation}, - {name: DataQuality, operation: dataQualityOperation}, - } -} +// analysis/analysis.go — the post-ingest pipeline, four ordered stages +345 // The definition of our analysis pipeline +346 func newPipeline() analysisPipeline { +347 return analysisPipeline{ +348 { +349 analysisStep: model.AnalysisStepADPostProcessing(), +350 operation: adPostProcessingOperation, +351 }, +352 { +353 analysisStep: model.AnalysisStepAzurePostProcessing(), +354 operation: azurePostProcessingOperation, +355 }, +356 { +357 analysisStep: model.AnalysisStepTagging(), +358 operation: taggingOperation, +359 }, +360 { +361 name: DataQuality, +362 operation: dataQualityOperation, +363 }, +364 } +365 } ``` Four ordered stages, run after every ingest. That is a batch view-maintenance schedule, and the @@ -84,15 +111,19 @@ question topic 27 spends a whole topic on. ### Step 4 — Principal sets are roaring bitmaps +> **In:** the derived-edge pass, which is set algebra over node ids. +> **Out:** roaring bitmaps (`cardinality.Duplex[uint64]`) as the principal-set representation, and the recognition that this is topic 23's postings lists holding principals instead of document ids. + Every interesting operation here is set algebra over node ids: "principals with `GetChanges`" intersected with "principals with `GetChangesAll`", "everything reachable from these seeds" minus "everything already tagged". BloodHound stores those sets as roaring bitmaps — `cardinality.Duplex[uint64]` — and `analysis/ad/post.go:244` is the workhorse: ```go -// FetchNodeIDsByKind fetches a bitmap of node IDs where each node has at least one -// kind assignment that matches the given kind. -func FetchNodeIDsByKind(tx graph.Transaction, targetKind graph.Kind) (cardinality.Duplex[uint64], error) +// analysis/ad/post.go — roaring bitmaps as principal sets +242 // FetchNodeIDsByKind fetches a bitmap of node IDs where each node has at least one kind assignment +243 // that matches the given kind. +244 func FetchNodeIDsByKind(tx graph.Transaction, targetKind graph.Kind) (cardinality.Duplex[uint64], error) { ``` This is topic 23's postings-list structure doing identity management. The intersection that @@ -102,23 +133,23 @@ principals instead of document ids. ### Step 5 — Traversal: parallel BFS with a shared bitmap as the visited set +> **In:** roaring-bitmap principal sets and the derived graph. +> **Out:** the parallel BFS whose visited set is a thread-safe roaring bitmap, with `CheckedAdd` as the atomic test-and-set, and `direction` as the forward/backward switch that defense needs. + `analysis/ad/membership.go:81`: ```go -func FetchPathMembers(ctx context.Context, db graph.Database, root graph.ID, - direction graph.Direction, queryCriteria ...graph.Criteria) - (cardinality.Duplex[uint64], error) { - traversalMap := cardinality.ThreadSafeDuplex(cardinality.NewBitmap64()) - return traversalMap, traversal.New(db, post.MaximumDatabaseParallelWorkers).BreadthFirst(ctx, traversal.Plan{ - Root: graph.NewNode(root, graph.NewProperties()), - Driver: func(...) ([]*graph.PathSegment, error) { - // ... for each neighbour: - if traversalMap.CheckedAdd(next.Node.ID.Uint64()) { - nextSegments = append(nextSegments, nextSegment) - } - }, - }) -} +// analysis/ad/membership.go — parallel BFS; the thread-safe roaring bitmap is the visited set + 81 func FetchPathMembers(ctx context.Context, db graph.Database, root graph.ID, direction graph.Direction, queryCriteria ...graph.Criteria) (cardinality.Duplex[uint64], error) { + 82 traversalMap := cardinality.ThreadSafeDuplex(cardinality.NewBitmap64()) + 84 return traversalMap, traversal.New(db, post.MaximumDatabaseParallelWorkers).BreadthFirst(ctx, traversal.Plan{ + // … Driver visits each neighbour of the current segment: + 95 for next := range cursor.Chan() { + 96 nextSegment := segment.Descend(next.Node, next.Relationship) + 98 if traversalMap.CheckedAdd(next.Node.ID.Uint64()) { + 99 nextSegments = append(nextSegments, nextSegment) +100 } +101 } ``` Three things to notice. The traversal is *parallel* over a worker pool. The visited set is a @@ -130,19 +161,23 @@ matters for defense, and it is the one lane 2's dominator analysis builds on. ### Step 6 — Tier Zero: the label the product is organised around +> **In:** cheap forward/backward reachability from Step 5. +> **Out:** the two labeled node sets — Tier Zero and Owned — that every product question reduces to, and lane 2's measured tiered-vs-flat contrast. + `analysis/tiering/tiering.go:37`: ```go -const ( - StrTagTierZero = "Tag_Tier_Zero" - StrTagOwned = "Tag_Owned" -) - -func IsTierZero(node *graph.Node) bool { - if node.Kinds.ContainsOneOf(KindTagTierZero) { return true } - startSystemTags, _ := node.Properties.Get(common.SystemTags.String()).String() - return strings.Contains(startSystemTags, ad.AdminTierZero) -} +// analysis/tiering/tiering.go — the Tier Zero predicate (string tags at :28, kind tags at :33) +28 StrTagTierZero = "Tag_Tier_Zero" +29 StrTagOwned = "Tag_Owned" +37 func IsTierZero(node *graph.Node) bool { +38 if node.Kinds.ContainsOneOf(KindTagTierZero) { +39 return true +40 } else { +42 startSystemTags, _ := node.Properties.Get(common.SystemTags.String()).String() +43 return strings.Contains(startSystemTags, ad.AdminTierZero) +44 } +45 } ``` Tier Zero is "assets whose compromise is game over"; `Owned` is "assets the attacker already @@ -160,6 +195,9 @@ and the second one has no remediation you can rank. ### Step 7 — Asset-group selectors: user-defined node sets, diffed +> **In:** labeled node sets, and a way to declare more of them. +> **Out:** analyst-declared selectors (by object id or arbitrary Cypher), expanded along known parent/child paths and kept current as a *diff* rather than a rewrite — with two operational details worth stealing. + `analysis/agt.go:137` (`FetchNodesFromSeeds`) and `:562` (`SelectNodes`) implement "an analyst declares a set of nodes — by object id, or by an arbitrary Cypher selector — and the system expands it along known parent/child paths and keeps it current". Two details worth stealing: @@ -189,9 +227,9 @@ Repo: [`~/repos/bloodhound`](https://github.com/SpecterOps/BloodHound) @ `196838 ## Questions to answer in notes.md -1. `PathfindingRelationships` (63 kinds) is a strict subset of `Relationships` (104). Pick two - kinds that are excluded and explain, in attacker terms, why walking them would produce a path - that is not an attack. +1. `PathfindingRelationships` (64 kinds) is a strict subset of the 104-kind ontology (and of the + 88-kind `Relationships` edge set). Pick two kinds that are excluded and explain, in attacker + terms, why walking them would produce a path that is not an attack. 2. The 31 post-processed edges are a materialized view refreshed after every ingest. Sketch what incremental maintenance would cost instead, for `DCSync` specifically: which writes invalidate it, and how would you index for that? @@ -207,14 +245,82 @@ Repo: [`~/repos/bloodhound`](https://github.com/SpecterOps/BloodHound) @ `196838 ## Done when +Answer each before unfolding it. + - [ ] You can state the edge semantics in one sentence and derive the shortest-path formulation from it. + +
Answer + + Edge semantics: `A → B` means *control of A yields control of B*, oriented so that traversal is + privilege escalation. Because control composes transitively, "can A become Domain Admin?" is + exactly "is there a directed path `A →* DA`?" — `MATCH p = shortestPath((a)-[*1..]->(da))`. A + per-object ACL review sees one hop at a time; the query sees the composition, which is why the + reachable set is most of the company while the membership answer is five names. + +
+ - [ ] You can name the four kind partitions and explain what each is for. + +
Answer + + From `graphschema/ad/ad.go`: `Relationships()` (:1151, **88** edge kinds) is the full edge + alphabet; `ACLRelationships()` (:1154, **30**) are the rights that come from a DACL; + `PathfindingRelationships()` (:1160, **64**) is the subset an attacker may actually walk — the + traversal mask; `PostProcessedRelationships()` (:1172, **31**) are the derived edges written back + after ingest. The whole namespace is **104** `StringKind` constants (16 node kinds + 88 edge + kinds). + +
+ - [ ] You can explain why 31 edge kinds are derived, and connect that to topic 27. + +
Answer + + `AdminTo`, `DCSync`, `CanRDP`, the `ADCSESC*` family, etc. are conjunctions or closures over + collected rights — e.g. `DCSync` = `GetChanges` ∧ `GetChangesAll` on the domain. Materializing + them once per ingest (`newPipeline`, `analysis.go:346`) turns every later path query into a plain + traversal instead of a per-hop predicate evaluation: RUM's read/update trade (topic 1), i.e. a + batch materialized view. Topic 27's question is why refresh in bulk rather than incrementally on + write. + +
+ - [ ] You can point at the roaring bitmap in the traversal and say what it replaces. + +
Answer + + `FetchPathMembers` (`membership.go:81`) uses `traversalMap`, a `cardinality.ThreadSafeDuplex` + roaring bitmap, as its visited set; `CheckedAdd` — an atomic test-and-set — replaces a per-node + lock, so the parallel frontier expansion is race-free without one. It is topic 23's postings-list + membership test doing identity management: a roaring AND derives `DCSync`, a roaring membership + check dedupes the BFS frontier. + +
+ - [ ] Your `chokepoint.rs` reproduces the tiered/flat contrast: 1992-user blast radius vs none. + +
Answer + + Tiered directory: one group has a **1992 / 2000 = 99.6%** blast radius and a single cut collapses + exposure `2000 → 8`. Flat directory (three unmanaged service-account groups plus two misplaced + Domain Admin tokens): the *same* exposure number, but **no single-node cut frees a single user** — + there is no remediation you can rank. Same headline, only one is actionable. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five: (1) two pathfinding-excluded kinds explained in attacker terms (e.g. `GetChanges` / + `GetChangesAll`); (2) the incremental-maintenance cost of `DCSync` and how to index for it; (3) + why `CheckedAdd` beats `Contains`-then-`Add` in a parallel BFS; (4) what "% of users with a path + to Tier Zero" measures given that sessions are *sampled* live; (5) three misconfigurations that + create a no-choke-point flat directory, and which to fix first. + +
+ ## References - Code: [SpecterOps/BloodHound](https://github.com/SpecterOps/BloodHound) — `packages/go/graphschema/ad/`, diff --git a/topics/40-security-attack-graphs/reading-sleuth.md b/topics/40-security-attack-graphs/reading-sleuth.md index 17efb4c..6fae6b2 100644 --- a/topics/40-security-attack-graphs/reading-sleuth.md +++ b/topics/40-security-attack-graphs/reading-sleuth.md @@ -7,7 +7,8 @@ subgraph. The obstacle is not subtlety, it is volume and connectivity. Enterpris billions of events a day, more than 99.9% of them benign, and naive backward tracing from an alert reaches almost everything — the *dependency explosion* problem. SLEUTH's answer is worth reading as a database paper: a purpose-built main-memory graph at under 10 bytes per event -(against ~250 for a general graph database and ~3 KB for STINGER/NetworkX), plus a tag system +(against ~250 bytes/edge for STINGER and ~3 KB for NetworkX — the two *main-memory-optimized* +graph stores the paper actually measures), plus a tag system that turns pruning into a shortest-path problem with tag-derived edge costs. The combination gets 79 hours of audit data analysed in 14 seconds. @@ -21,6 +22,9 @@ prune while it searches, not after.** ### Step 1 — The provenance graph +> **In:** a host's raw audit log — Windows event logs, Linux audit, or FreeBSD DTrace. +> **Out:** the OS-neutral provenance graph (subjects = processes, objects = files/pipes/sockets, edges = timestamped information-flow events), and what "an attack" is as a connected subgraph of it. + Two vertex types and one edge type: ``` @@ -36,11 +40,16 @@ SLEUTH normalises Windows event logs, Linux audit and FreeBSD DTrace into the sa ### Step 2 — Why a general graph database is the wrong tool here +> **In:** the provenance graph and enterprise event volumes (billions to tens of billions/day). +> **Out:** the memory argument — general graph stores cost too much per edge — and SLEUTH's <10-bytes-per-event encoding (a 6-byte bidirectional edge), the same domain-specific-encoding move as topic 12. + §2 is unusually direct about this, and it is the part a database engineer should read twice. -Neo4j-class stores use roughly **250 bytes per graph edge**; STINGER and NetworkX about **3 KB**. +General graph databases (Neo4J, Titan) it dismisses qualitatively — their memory use is simply +"too high", with no figure. The numbers it *does* give are for the two stores optimized for +main-memory performance: **STINGER ≈ 250 bytes per graph edge**, **NetworkX ≈ 3 KB per edge**. At "billions to tens of billions of events per day" that is terabytes of RAM. SLEUTH's design -gets to **under 10 bytes per event** — a **25× to 300× reduction** — and the techniques are the -same ones this book applies to columnar and index storage: +gets to **under 10 bytes per event** — a **25× (vs STINGER) to 300× (vs NetworkX) reduction** — and +the techniques are the same ones this book applies to columnar and index storage: - **32-bit identifiers instead of 64-bit pointers.** Enough for 4 billion objects/subjects per host; the largest data set had orders of magnitude fewer. @@ -68,6 +77,9 @@ conclusion from the security side. ### Step 3 — Tags: two dimensions, and the split that matters +> **In:** the compact provenance graph, and the need to prune traffic that is >99.9% benign. +> **Out:** the two tag dimensions (trustworthiness t-tags, confidentiality c-tags), the code-vs-data t-tag split, and the conservative propagation rule — plus Table 10's measured payoff for the split. + Every subject and object carries tags summarising *provenance-derived* trust and sensitivity. **Trustworthiness tags (t-tags)**, three levels: @@ -103,6 +115,9 @@ but will not cause attacks to go undetected". ### Step 4 — Detection: four policies about means and motive +> **In:** tagged subjects and objects from Step 3. +> **Out:** the four objective-based detection policies (means = an untrusted source, recorded by the `unknown` t-tag; motive = a goal-advancing event) attached to trigger points, and how external detectors compose by setting a code t-tag. + SLEUTH deliberately avoids application-specific knowledge and detects on attacker *objectives* instead. The reasoning: an attacker needs both motive (an event advances a goal) and means (the data or code came from an untrusted source, which is what the `unknown` t-tag records). The four @@ -122,6 +137,9 @@ suspicion. ### Step 5 — Backward analysis as shortest path with tag-derived costs +> **In:** alarms (flagged subjects) and the tagged graph. +> **Out:** backward analysis reframed as Dijkstra with tag-derived edge costs (0 / high / 1), why it can stop the moment an entry point joins the shortest-path tree, and how it resolves multiple candidate entry points. + This is the algorithmic core. Backward analysis starts from alarms and walks the graph in reverse to find entry points (in-degree zero, untrusted — typically network connections). Two problems: the graph has hundreds of millions of edges, and many entry points are backward-reachable from @@ -143,6 +161,9 @@ closest by path cost. ### Step 6 — Forward analysis and simplification +> **In:** the entry point found by backward analysis. +> **Out:** forward impact analysis (same cost metric, plus a distance threshold `d_th`) reduced 100×–500×, and the three simplifications that make the graph human-readable. + Forward analysis from the entry point assesses impact, and has the mirror-image size problem: "a naive analysis produced impact graphs with millions of edges, whereas our refined algorithm reduces this number by **100x to 500x**". Same cost metric, plus a tunable distance threshold @@ -156,6 +177,9 @@ Three simplifications then produce something a human can read: ### Step 7 — The reduction, measured end to end +> **In:** all of the above, run on the DARPA Transparent Computing campaigns. +> **Out:** Table 11 read by column — single t-tag 4.68×, split t-tags 1305×, simplification 41.8× — plus the runtime (79 h in 14 s) and accuracy (174 correct / 0 wrong / 2 missed) figures. + Table 11 is the summary the whole paper builds to. For each DARPA Transparent Computing campaign: initial event count, final scenario-graph event count, and the reduction attributable to each stage. @@ -194,9 +218,9 @@ That single rule is the difference between a usable tool and an alert firehose. - **§1.1 Approach overview + Fig 1.** The four-stage pipeline. The headline numbers (79 hours in 14 s at 84 MB; 38.5M events → 130) are here. - **§2 Main-memory dependency graph.** Read this as a storage-engine section, because it is one. - The comparison against Neo4j/Titan (250 B/edge) and STINGER/NetworkX (3 KB) is the motivation; - then work through the encoding bullet by bullet against Step 2 and convince yourself the 6-byte - bidirectional edge is real. + The motivation is the memory comparison: Neo4J/Titan dismissed as "too high" with no figure, + STINGER quoted at ~250 B/edge and NetworkX at ~3 KB/edge; then work through the encoding bullet + by bullet against Step 2 and convince yourself the 6-byte bidirectional edge is real. - **§3 Tags and attack detection.** §3.1 for the tag lattices; the paragraph on splitting code and data t-tags is the one to mark. §3.2 for the four policies and the motive/means argument. - **§4 Policy framework + Table 2.** Trigger points as a level of indirection over events. Note @@ -234,14 +258,87 @@ That single rule is the difference between a usable tool and an alert firehose. ## Done when +Answer each before unfolding it. + - [ ] You can state the dependency-explosion problem and why post-hoc filtering does not solve it. + +
Answer + + Dependency explosion: naive backward tracing from a single alert follows information-flow edges + until it reaches almost every node — an enterprise host emits billions of events/day, >99.9% of + them benign, and everything is transitively connected. Post-hoc filtering does not help because + you would first have to *materialize* the millions-of-edges graph you are trying to avoid; + SLEUTH instead prunes *during* the search — Dijkstra stops the moment an entry point joins the + shortest-path tree (Step 5). + +
+ - [ ] You can explain the <10 bytes/event encoding well enough to sketch the record layout. + +
Answer + + 32-bit ids (4 billion entities/host) not 64-bit pointers; events stored *inside* subjects, which + removes subject→event pointers and event ids entirely (events outnumber objects/subjects ~100×, + so event compactness is what matters); variable-length subject-event records — **4 bytes** + typical, up to 16 — with **3-bit** event names for frequent syscalls and **≤8-bit** per-subject + object references "like file descriptors"; **delta timestamps** at ms resolution relative to the + subject's last event (**16 bits**, with a `timegap` pseudo-event for long gaps); object-event + records only for `read`/`write`, stored as a **12-bit** relative index. Net: a bidirectional edge + in ~**6 bytes** (4 + 2); 38M events in 329 MB. + +
+ - [ ] You can name the two tag dimensions, the three t-tag levels, and why code and data t-tags are separate. + +
Answer + + Dimensions: **trustworthiness** (t-tags) and **confidentiality** (c-tags). Three t-tag levels: + *benign authentic* → *benign* → *unknown*. c-tags: *secret* → *sensitive* → *private* → + *public*. A subject carries **two** t-tags — one for its **code**, one for its **data** — because + a process that reads an untrusted file has untrusted data but still-trusted code; conflating them + over-taints everything downstream. Table 11 measures the split at **1305×** against **4.68×** for + a single tag. + +
+ - [ ] You can explain backward analysis as Dijkstra and give the three edge costs. + +
Answer + + Backward analysis walks the graph in reverse from alarms toward entry points (in-degree zero, + untrusted — typically outside network connections). Reuse the tags as edge costs: + `unknown → benign` = **0** (the malicious/benign boundary — must be on the path); + `benign → benign` = **HIGH** (trusted flows — exclude); `unknown → unknown` = **1** (inside the + suspicious region). Dijkstra discovers paths in increasing cost order, so it can **stop as soon + as an entry point enters the shortest-path tree**, and it naturally prefers the lowest-cost entry + point when several are reachable. + +
+ - [ ] You can read Table 11 by column and say which stage contributes what. + +
Answer + + The columns are the finding, not the totals. Forward analysis with a *single* t-tag: **4.68×** + average. Splitting code and data t-tags: **1305×** — two and a half orders of magnitude from one + modelling decision. Simplification (prune / merge / filter): **41.8×**. On L-2 the chain is + 38.5M events → 130 (297,100× total), of which the split column alone is 2971×. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five: (1) which encoding decisions survive if the graph must be updatable and Cypher-queryable; + (2) the mechanism behind the 1305× code/data split, with a concrete two-process example; (3) why + **0** (not 1) is the right cost for boundary edges under Dijkstra; (4) the propagation rule stated + as a soundness-vs-completeness claim and which is sacrificed; (5) two things the event graph makes + harder and one it makes easier, versus a permission graph. + +
+ ## References - Hossain, Milajerdi, Wang, Eshete, Gjomemo, Sekar, Stoller, Venkatakrishnan. *SLEUTH: Real-time diff --git a/topics/40-security-attack-graphs/reading-zanzibar.md b/topics/40-security-attack-graphs/reading-zanzibar.md index 957576f..ed1fc4f 100644 --- a/topics/40-security-attack-graphs/reading-zanzibar.md +++ b/topics/40-security-attack-graphs/reading-zanzibar.md @@ -22,6 +22,9 @@ request, without ever using a stale ACL on new content.** ### Step 1 — Relation tuples: one row shape for everything +> **In:** the authorization question "can user U do R to object O?". +> **Out:** the single relation-tuple row shape — `object#relation@user`, where `user` may itself be a `userset` — that unifies ACLs, groups and inheritance, plus the versioned primary key that makes snapshot reads possible. + ``` ⟨tuple⟩ ::= ⟨object⟩ '#' ⟨relation⟩ '@' ⟨user⟩ ⟨object⟩ ::= ⟨namespace⟩ ':' ⟨object_id⟩ @@ -48,6 +51,9 @@ is what makes snapshot reads at any timestamp within the GC window possible. ### Step 2 — Userset rewrite rules: three leaf types +> **In:** stored relation tuples from Step 1. +> **Out:** the three rewrite leaf kinds (`_this`, `computed_userset`, `tuple_to_userset`) that define a relation without storing every tuple, combined by union / intersection / exclusion — and where SpiceDB evaluates each. + Storing a tuple per object per relation would be wasteful and rigid, so relations are defined by *rewrite rules* in a namespace config (Figure 1 in the paper): @@ -79,6 +85,9 @@ Leaves combine with union, intersection and exclusion. In SpiceDB the whole tree ### Step 3 — Check is a graph traversal, stated as one recursion +> **In:** relation tuples and rewrite rules. +> **Out:** Check as one recursion (direct membership ∨ recursive userset membership), its concurrent-with-cancellation evaluation and read pooling, and lane 3's measured linear-in-depth×width cost. + §3.2.3, and it is worth memorising: ``` @@ -113,6 +122,9 @@ flatten. ### Step 4 — Leopard: the transitive closure as two sorted integer lists +> **In:** the deep or wide group structures that make Check's pointer-chasing expensive. +> **Out:** the Leopard index — `GROUP2GROUP` and `MEMBER2GROUP` named sets stored as ordered integer lists — that turns a membership check into an `O(min(|A|,|B|))` skip-list intersection (topic 23's galloping intersect). + §3.2.4. For namespaces with deep or wide group structure, Zanzibar precomputes membership into a specialised index over "named sets" of `(T, s, e)` tuples — set type, set id, element id. Group membership uses two set types: @@ -154,6 +166,9 @@ Topic 1's RUM conjecture with a security label. ### Step 5 — What the index costs you: freshness +> **In:** the offline-built Leopard index, stale by construction. +> **Out:** the incremental layer (Watch API → `(T,s,e,t,d)` updates merged at query time), the tens-of-thousands-of-events fan-out a single tuple change can cause, and Leopard's measured latency and QPS. + An offline pipeline reads periodic snapshots of the tuples, recursively expands the ACL graph, and ships shards that Leopard servers hot-swap. Which means the index is *stale by construction* and cannot serve a consistent read on its own. The fix is an incremental layer: Leopard's indexer @@ -172,6 +187,9 @@ median and under 1 ms at p99**. Shards are usually served entirely from memory. ### Step 6 — Zookies and the new enemy problem +> **In:** a cache that, in authorization, is a correctness decision rather than a latency one. +> **Out:** the two "new enemy" failures (§2.2 Examples A and B), and the zookie protocol — an opaque token encoding a timestamp ≥ every prior ACL write — whose `≥` guarantee lets most checks be served from a local replica. + A cache is normally a latency decision. In authorization it is a correctness decision, and the paper gives two concrete failures (§2.2): @@ -212,6 +230,9 @@ Same median, 4× the p95, because `Recent` often needs the leader replica. ### Step 7 — Hot spots: the frontier +> **In:** popular objects concentrating bursty read traffic on single storage servers. +> **Out:** the four hot-spot mechanisms (consistent-hashing cache trees, timestamp quantization, a lock table, prefetch + delayed eager cancellation), and why a 10% hit rate is worth it when it protects the tail — it prevents 500K internal RPC/s. + §3.2.5 opens with "We found the handling of hot spots to be the most critical frontier in our pursuit of low latency and high availability." Popular objects concentrate reads on one storage server, and authorization traffic is bursty by nature (one search results page fires hundreds of @@ -238,6 +259,9 @@ building at a 10% hit rate if what it protects is a tail, not a mean. ### Step 8 — SpiceDB: which parts are essential +> **In:** the paper's mechanisms and the SpiceDB source at pin `8422483`. +> **Out:** which mechanisms are inherent to the problem (recursion, rewrite tree, set algebra, reverse traversal, canonicalized cache key, lock table, fan-out bound) versus Google-shaped (Spanner/TrueTime, Leopard as a separate service, Slicer). + Reading the implementation tells you which of the above is Google-specific and which is inherent. Inherent, all present in `internal/`: @@ -299,15 +323,86 @@ Repo: [`~/repos/spicedb`](https://github.com/authzed/spicedb) @ `8422483`, paths ## Done when +Answer each before unfolding it. + - [ ] You can write the relation tuple grammar from memory and explain why the user slot holds a userset. + +
Answer + + `⟨tuple⟩ ::= ⟨object⟩'#'⟨relation⟩'@'⟨user⟩`, `⟨object⟩ ::= ⟨namespace⟩':'⟨object_id⟩`, + `⟨user⟩ ::= ⟨user_id⟩ | ⟨userset⟩`, `⟨userset⟩ ::= ⟨object⟩'#'⟨relation⟩`. The user slot holds a + userset so that groups, nesting and ACL inheritance are all the *same* row shape — there is no + separate group table — which "unifies the concepts of ACLs and groups and supports efficient + reads and incremental updates". The primary key `(shard ID, object ID, relation, user, commit + timestamp)` carries the commit timestamp, so versions coexist for snapshot reads. + +
+ - [ ] You can state the Check recursion and name the two leaf kinds that make it recursive. + +
Answer + + `CHECK(U, object#relation) = ∃ tuple object#relation@U ∨ ∃ tuple object#relation@U' where + U' = object'#relation' s.t. CHECK(U, U')`. It bottoms out on `_this` (the stored tuples). The two + leaf kinds that introduce recursion are `computed_userset` (same object, different relation) and + `tuple_to_userset` (the *arrow* — follow a tupleset, then evaluate a relation on each returned + object). The paper calls it "pointer chasing", expensive when groups are deep or wide. + +
+ - [ ] You can explain Leopard's two set types and why the intersection is `O(min(|A|,|B|))`. + +
Answer + + `GROUP2GROUP(s)` = groups directly or indirectly under `s`; `MEMBER2GROUP(u)` = groups `u` is a + *direct* member of. Then `U ∈ G ⟺ MEMBER2GROUP(U) ∩ GROUP2GROUP(G) ≠ ∅`. Index tuples are + ordered integer lists in a skip list, so the intersection iterates the *smaller* set and *seeks* + into the larger — `O(min(|A|,|B|))` seeks, not `O(|A|+|B|)`. That is why "user in 3 groups" vs + "group with 100,000 descendants" costs almost nothing; lane 3 finds a needle in a 500,000-element + list in tens of probes. + +
+ - [ ] You can give both new-enemy examples and explain how a zookie prevents each. + +
Answer + + Example A (neglecting ACL update order): Alice removes Bob from a folder ACL, then has Charlie + move docs into the folder; Bob must not see them. Example B (old ACL on new content): Alice + removes Bob from a doc's ACL, then has Charlie add content; Bob must not see it. On each content + change the client requests a **zookie** — an opaque token encoding a global timestamp ≥ every + prior ACL write — stored atomically with the content. Later checks pass the zookie and evaluate at + any snapshot **≥** it, so a check can never run against an ACL older than the content it guards. + +
+ - [ ] Your `authz.rs` reproduces lane 3: 19→559 tuple reads against 4→12 index probes across nesting depth 2→32, with the index agreeing with pointer chasing on every pair. + +
Answer + + Pointer chasing (`check_pointer`) reads **19** tuples at depth 2 rising to **559** at depth 32 + (0.46 → 11.28 µs) — linear in depth × width, as the recursion predicts. The Leopard index + (`LeopardIndex::build` + `intersect_galloping`) stays flat: **4 → 12** probes, ~0.01 µs, ~1000× + cheaper at depth 32, for a 1.7× space cost (6672 stored tuples → 11393 index entries). Both must + return the identical membership verdict on every pair. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five: (1) one policy per rewrite leaf kind and which makes Check's cost depend on data not + schema; (2) reconciling the 10% delegate cache hit rate with the "80% or it's not worth it" + instinct; (3) why rounding evaluation timestamps *up* is safe under the zookie `≥` guarantee while + rounding down is a bug; (4) the break-even write rate for incremental closure maintenance, set up + with ~500 updates/s and 1.56M QPS; (5) whether Leopard's precomputed closure survives conditional + (caveated) membership. + +
+ ## References - Pang, Cáceres, Burrows, Chen, Dave, Germer, Golynski, Graney, Kang, Kissner, Korn, Parmar, diff --git a/topics/41-onchain-analytics/reading-bitcoin-redux.md b/topics/41-onchain-analytics/reading-bitcoin-redux.md index 2ab580e..69a634a 100644 --- a/topics/41-onchain-analytics/reading-bitcoin-redux.md +++ b/topics/41-onchain-analytics/reading-bitcoin-redux.md @@ -9,6 +9,13 @@ the Master of the Rolls set down, first-in-first-out, turns out to be not just t but the *computationally* right one, because it is lossless. This guide reads the paper alongside RustyTaintChain, the authors' Rust implementation, whose core is fifteen lines. +Every code anchor below is RustyTaintChain at commit `4e12fd0` (the revision this repo pins), +all of it in one file, `src/callbacks/bootstrap_taint_fifo.rs`, quoted with the line numbers the +code occupies in that version. Every paper number cites the section or figure of *Bitcoin Redux* +(Anderson, Shumailov, Ahmed & Rietmann, WEIS 2018) it came from. Where a figure is one this +repo measured, it is labelled as a bench lane and traces to +[`../../FINDINGS.md`](../../FINDINGS.md) row 41 and this topic's `notes.md`. + ## The problem in one sentence **Trace a stolen coin forward through a transaction graph where money is constantly split and @@ -19,27 +26,48 @@ everybody.** ### Step 1 — Why the question is legally live: `nemo dat` -`Nemo dat quod non habet` — no one gives what they do not own — is a principle of nearly every -legal system. "If Alice steals Bob's horse and sells it to Charlie, Charlie doesn't end up owning -it; when Bob sees him riding it, he can simply demand it back." - -The exception that used to matter, *market overt* (buy openly in a public market and you get good -title), was abolished in Britain in 1995. Two exceptions remain, for **money** and for **bills of -exchange** — and the USA has designated bitcoin a *commodity*, not money. So: "Unless +> **In:** nothing yet — this step is the legal motivation for building a tracer at all. +> **Out:** the reason a *forward* taint trace has teeth: if a stolen coin can be followed, it +> can be reclaimed. Step 2 asks how to follow it. + +**Nemo dat quod non habet** — "no one gives what they do not own" — is the principle that you +cannot pass better title to property than you yourself hold. It is part of nearly every legal +system. "If Alice steals Bob's horse and sells it to Charlie, Charlie doesn't end up owning it; +when Bob sees him riding it, he can simply demand it back." + +The rule has exceptions. **Market overt** — an old rule that buying openly in a recognised public +market gives you good title even to stolen goods — was the one that used to matter; Britain +abolished it in 1995 after thieves abused it to launder stolen antiques (§2). Two exceptions +remain, for **money** and for **bills of exchange** (a bill of exchange is a transferable written +order to pay, like a cheque). A **commodity**, by contrast, is an ordinary tradable good with no +such exception — and the USA has designated bitcoin a commodity, not money. So: "Unless cryptocurrencies acquire this privileged status, there is no general exception to the nemo dat rule — so a theft victim can pursue and retrieve his stolen property." That is why taint tracking is not an academic exercise. If a stolen coin can be followed, it can -be reclaimed, and every exchange that touched it has a problem. +be reclaimed, and every exchange that touched it has a problem. Note the framing is careful: the +paper *assumes* bitcoin is a commodity (§2, "In what immediately follows, we will assume that +bitcoin is a commodity") and reasons from there — it does not claim FIFO tracing is settled law. +The conditional is the honest version, and this guide keeps it. ### Step 2 — Poison and haircut, and what the default does -Möser, Böhme and Breuker named the two policies the industry actually uses. +> **In:** the graph of transactions, and one output flagged stolen (Step 1's premise). +> **Out:** for every UTXO, a *taint fraction* between 0 and 1 — under two rival rules. This is +> the dataset Step 3 replaces with a lossless alternative. -**Poison**: if any input to a transaction is tainted, *every* output is entirely tainted. -**Haircut**: each output is tainted by the fraction of input value that was tainted. +A **taint policy** is a rule for propagating "this money is stolen" across a transaction that +mixes tainted and clean inputs. Möser, Böhme and Breuker (their 2014 risk-scoring paper, [MBB14], +cited by *Bitcoin Redux* §3.1) named the two the industry actually uses: -Haircut became the default. Here is what it does, traced forward over real thefts to 2016: +- **Poison**: if *any* input to a transaction is tainted, *every* output is entirely tainted + (taint fraction 1.0). Taint spreads like a contagion and never dilutes. +- **Haircut**: each output is tainted by the *fraction* of input value that was tainted. A + **taint fraction** is the share of an output's value that traces to the theft, a real number in + [0, 1]. + +Haircut became the default. Here is what it does, traced forward over real thefts to 2016 +(*Bitcoin Redux* §3.3): ``` 2012 Linode theft, 46,653 BTC @@ -51,19 +79,52 @@ Haircut became the default. Here is what it does, traced forward over real theft FIFO ............. 15,265 accounts ``` +**Why haircut smears everything — worked by hand.** Haircut's fraction is +`out_taint = in_taint / in_total`, where `in_taint` is the tainted satoshis flowing in and +`in_total` is all satoshis flowing in. Follow one stolen coin through merges that each add nine +times as much clean money (a **merge** is a transaction whose inputs include both tainted and +clean UTXOs): + +``` + stolen coinbase: 1,000,000 sat, fraction 1.000 (100% tainted) + + hop 1: 1,000,000 tainted + 9,000,000 clean = 10,000,000 in + fraction = 1,000,000 / 10,000,000 = 0.100 (10%) + hop 2: same 10x dilution = 0.010 (1%) + hop 3: same 10x dilution = 0.001 (0.1%) + + total tainted value, summed over ALL outputs at every hop = 1,000,000 sat (conserved) +``` + +Three merges of ten-fold dilution and a descendant coin is **0.1% tainted** — below any threshold +worth acting on — yet the theft has now touched every output on all three hops. The total tainted +value never changes (haircut conserves it to the satoshi; it does not invent money), it is just +smeared thinner at each hop until "is this coin tainted?" stops meaning anything. + +Lane 1 of this topic's crate reproduces exactly that mechanism on a synthetic chain: one stolen +coinbase worth 0.25% of all the money ends up tainting **97.9% of the UTXO set (3657 of 3734) and +98.0% of addresses (3553 of 3627)**, and of those tainted UTXOs **658 carry less than 0.1% taint, +2,997 carry between 0.1% and 5%, and exactly two carry more than 5%** ([FINDINGS.md](../../FINDINGS.md) +row 41). The 658 sub-0.1% UTXOs are the coins three-or-more dilutions downstream in the worked +example above. That is the headline of this topic: 98% of everybody holds a trace, and two UTXOs +in the whole chain hold a share worth arguing about. + The paper's summing up: "'haircut' tainting smears the taint over the actively traded bitcoin stock. Bitcoin laundries are designed to make this even worse." And the consequence, stated without hedging: "the effect of aggressive asset recovery via regulated exchanges might be more akin to a tax on all users." -Lane 1 of this topic's crate reproduces the mechanism on a synthetic chain: one stolen coinbase -worth 0.25% of all the money ends up tainting **97.9% of the UTXO set and 98.0% of addresses**, -with 2,997 of those UTXOs carrying between 0.1% and 5% taint and exactly two carrying more than -5%. The total is conserved to the satoshi — haircut does not invent money — it is just no longer -information. - ### Step 3 — Clayton's Case, 1816 +> **In:** the same merging transactions Step 2 fed to haircut. +> **Out:** for every output, a *queue of satoshi runs* tagged stolen-or-clean instead of a single +> fraction. Step 4 shows why that representation is the whole argument. + +**FIFO (first-in-first-out)** is the rule that the earliest money in is the earliest money out. +Applied to taint it is a two-hundred-year-old legal precedent, not an algorithm the authors +invented — **Clayton's Case** (formally *Devaynes v Noble*, 1816), which is a rule of English +equity, adopted here as an accounting convention rather than as settled cryptocurrency law: + > In English law, there is a long-standing legal precedent on tracing stolen funds. It was > established in 1816, when a court had to tackle the problem of mixing funds after a bank went > bust and its obligations relating to one customer account depended on what sums had been @@ -82,13 +143,22 @@ off the front of that queue in output order. └──────┘ └──────┘ ``` -The paper's Figures 1–3 draw exactly this for poison, haircut and FIFO with four colours of -tainted input. Note what FIFO does that the others cannot: "the taint does not go across in -percentages, but to individual components (indeed, individual Satoshis) of each output." +Read the diagram against Step 2's arithmetic: haircut gives all three outputs the same +`2/9 = 0.222` fraction, so the taint is *everywhere and dilute*; FIFO puts the whole 2 stolen +satoshis on output E and leaves D and F provably clean, so the taint is *somewhere and exact*. +The paper's Figures 1–3 draw this for poison, haircut and FIFO with four colours of tainted +input. Note what FIFO does that the others cannot: "the taint does not go across in percentages, +but to individual components (indeed, individual Satoshis) of each output." ### Step 4 — Lossless, and therefore reversible -This is the property that matters and it is easy to skate past: +> **In:** the per-output satoshi-run queues from Step 3. +> **Out:** the property — losslessness — that makes those queues worth the extra storage. Step 5 +> is the code that maintains them. + +**Lossless** here means no information is destroyed at a merge: a satoshi stays labelled stolen or +clean, and nothing is rounded or averaged away. That is the property that matters and it is easy +to skate past: > As the taint does not spread or diffuse, the transaction processes it in a lossless way. This > means that we can trace a bitcoin's heritage backwards as well as tracing taint forwards, and @@ -97,64 +167,86 @@ This is the property that matters and it is easy to skate past: A satoshi under FIFO is stolen or it is not; there is no fractional state to accumulate rounding in, and no information is destroyed at a merge. So provenance survives arbitrarily many hops, and you can ask "where did *this particular* satoshi come from" as well as "where did the theft go". -Haircut destroys that on the first merge: 2/9 of 3/7 of 5/11 is a number, not a history. +Haircut destroys that on the first merge: 2/9 of 3/7 of 5/11 is a number, not a history — you +cannot invert a product of fractions back into which coin came from where. -It also makes the tracing **deterministic**, which matters legally as much as technically: two -investigators running FIFO on the same chain get the same answer, and can be cross-examined on +**Deterministic** means two runs on the same chain produce the same answer bit-for-bit. FIFO is +deterministic (given a fixed input/output ordering), which matters legally as much as technically: +two investigators running FIFO on the same chain get the same answer, and can be cross-examined on it. ### Step 5 — `extract_taint`: Clayton's Case in fifteen lines -RustyTaintChain represents an output's provenance as a queue of runs: +> **In:** an output's provenance as a `VecDeque` (Step 3's queue of runs), plus the +> number of satoshis this output claims. +> **Out:** a new queue holding exactly those satoshis, cut off the front of the input queue. +> Step 6 wraps two more operations around this one. + +RustyTaintChain represents an output's provenance as a queue of runs. A **run** is a contiguous +block of satoshis that share one provenance — `name` identifies the source (0 = clean, other +values name distinct crimes), `value` counts the satoshis in the block: ```rust -// src/callbacks/bootstrap_taint_fifo.rs:52 -pub struct TaintPart { - name : u16, // 0 = clean; other values name distinct crime sources - value: u64, // satoshis in this contiguous run -} +// src/callbacks/bootstrap_taint_fifo.rs:51-55 — the run type +51 #[derive(PartialEq, Eq, Hash, Default, Debug, RustcDecodable, RustcEncodable, Clone)] +52 pub struct TaintPart { +53 name : u16, // 0 = clean; other values name distinct crime sources +54 value: u64 // satoshis in this contiguous run +55 } ``` -and the whole of the FIFO rule is one function that cuts `value` satoshis off the front of a -queue, splitting the run that straddles the boundary: +Line 53's `name: u16` is the whole reason FIFO is lossless (Step 4): it is a *label*, not a +fraction. The whole of the FIFO rule is then one function that cuts `value` satoshis off the +front of a queue, splitting the run that straddles the boundary: ```rust -// :142, lightly trimmed -fn extract_taint(given_taints: &mut VecDeque, value: u64) -> VecDeque { - let mut remaining = value; - let mut new_tainted_balance = VecDeque::new(); - while remaining > 0 { - if given_taints.is_empty() { - new_tainted_balance.push_back(TaintPart { name: 0, value: remaining }); - remaining = 0; - } else { - let mut ctaint = given_taints.pop_front().unwrap(); - if remaining >= ctaint.value { - remaining -= ctaint.value; - new_tainted_balance.push_back(ctaint); // whole run fits - } else { - ctaint.value -= remaining; // run straddles the cut - new_tainted_balance.push_back(TaintPart { name: ctaint.name, value: remaining }); - given_taints.push_front(ctaint); // put the remainder back - remaining = 0; - } - } - } - new_tainted_balance -} +// src/callbacks/bootstrap_taint_fifo.rs:142-172 — extract_taint (asserts on 149/154/159 elided) +142 fn extract_taint(given_taints: &mut VecDeque, value: u64)->VecDeque{ +143 let mut remaining = value; +144 let mut new_tainted_balance = VecDeque::new(); +146 while remaining > 0{ +147 if given_taints.is_empty(){ // branch 1: queue ran dry +148 new_tainted_balance.push_back(TaintPart{name: 0, value:remaining}); // rest is clean +150 remaining = 0; +151 }else{ +152 let mut ctaint = given_taints.pop_front().unwrap(); +153 if remaining >= ctaint.value{ // branch 2: whole run fits +155 remaining -= ctaint.value; +156 new_tainted_balance.push_back(ctaint); +157 }else{ // branch 3: run straddles the cut +158 ctaint.value -= remaining; +160 new_tainted_balance.push_back(TaintPart{name:ctaint.name, value:remaining}); +161 given_taints.push_front(ctaint); // put the remainder back +162 remaining = 0; +163 } +164 } +165 } +167 if remaining > 0{ // belt-and-braces: any leftover is clean +168 new_tainted_balance.push_back(TaintPart{name:0, value: remaining}); +169 } +171 return new_tainted_balance; +172 } ``` -Three branches: the queue ran dry (the rest is clean), the run fits entirely, the run straddles -the cut and must be split. Getting the third one right is the whole exercise — the crate's -`extract_taint_splits_runs_at_the_boundary` test asks for 4 satoshis out of a 10-satoshi stolen -run and insists you get back a 4 and leave a 6. +The line that carries the argument is **158–161**, branch 3: when the requested `value` lands in +the middle of a run, it splits the run, keeps the front piece with the *same* `name`, and pushes +the remainder back on the queue for the next output. Branch 1 (147–150) covers a queue that ran +dry — the rest is clean — and branch 2 (153–156) is the run that fits entirely. Getting branch 3 +right is the whole exercise: the crate's `extract_taint_splits_runs_at_the_boundary` test asks for +4 satoshis out of a 10-satoshi stolen run and insists you get back a 4 and leave a 6. Processing a transaction is then: concatenate the input queues in input order, call -`extract_taint` once per output in output order. Measured in lane 2: **3.1 million transactions -per second**, because that is all it is. +`extract_taint` once per output in output order. Measured in lane 2: **20,400 transactions in +6.6 ms = 3.1 million transactions per second** (this topic's `notes.md`), because that is all it +is. ### Step 6 — The two operations `extract_taint` needs around it +> **In:** the output queues `extract_taint` produces (Step 5), fed back in as inputs to later +> transactions. +> **Out:** merged, coalesced queues that stay bounded in size. Step 7 leaves the code and returns +> to the paper's argument. + Real chains need two more pieces, both in the same file: - **`combine_taints:174`** — when two provenance queues meet, runs with different `name`s collide, @@ -168,8 +260,15 @@ Real chains need two more pieces, both in the same file: ### Step 7 — Why mixers make it worse, not better -The received wisdom is that a mixer launders coins: put one black coin in with nine white ones, -get ten white ones out. The paper inverts it, and the argument is legal rather than technical. +> **In:** the FIFO tracer of Steps 3–6 plus the legal frame of Step 1. +> **Out:** the paper's counter-intuitive claim about **mixers** — services that pool many users' +> coins to break the on-chain link between input and output. Step 8 is the caveat that undermines +> the whole method, honestly. + +The received wisdom is that a **mixer** (also *tumbler* or *laundry* — a service that pools coins +from many users and pays out unrelated coins, to break the on-chain link) launders coins: put one +black coin in with nine white ones, get ten white ones out. The paper inverts it, and the argument +is legal rather than technical. Because getting good title requires acquiring in **good faith**, and because every transaction is public, "the act of passing a bitcoin through a laundry should put all its subsequent owners on @@ -182,13 +281,17 @@ been using quite the wrong metrics of quality." ### Step 8 — And then the paper undermines itself, honestly +> **In:** the whole tracing method of Steps 1–7. +> **Out:** the boundary of what any chain analysis can see — coins that never move on-chain. This +> is the honest weaker claim the method has to live with. + The last third is the part most summaries skip, and it is the most valuable. Having built the tracing machinery and gone looking for theft victims, the authors found that "with one exception, -the victims we talked to were using **hosted wallets**" — the exchange holds the keys, the -customer sees a balance, and increasingly the exchange does not actually move coins on-chain at -all: "many bitcoin exchanges do not now give their customers actual bitcoin, but rather do -off-chain transactions with other exchange customers or transact on customers' behalf with -outsiders." +the victims we talked to were using **hosted wallets**" — a hosted wallet is one where the +exchange holds the keys, the customer sees a balance, and increasingly the exchange does not +actually move coins on-chain at all: "many bitcoin exchanges do not now give their customers +actual bitcoin, but rather do off-chain transactions with other exchange customers or transact on +customers' behalf with outsiders." If the transaction never reaches the chain, no amount of chain analysis will see it. "In no case could we find any clear documentation of the actual ownership of the missing cryptocurrency." The @@ -234,13 +337,65 @@ all in `src/callbacks/bootstrap_taint_fifo.rs`. ## Done when +Answer each before unfolding it. + - [ ] You can state `nemo dat` and why bitcoin being a commodity rather than money matters. +
Answer + + `Nemo dat quod non habet` — you cannot pass better title than you hold, so a thief's buyer does + not own the goods and the victim can reclaim them (Step 1). The exceptions are money and bills of + exchange; **market overt** was abolished in Britain in 1995. Because the USA classes bitcoin a + *commodity*, not money, no exception applies and stolen coins remain reclaimable — which is the + entire reason a forward tracer has legal teeth. The paper *assumes* commodity status rather than + asserting it as settled law. + +
- [ ] You can give the Linode and Flexcoin haircut-vs-FIFO numbers from memory. +
Answer + + Linode (46,653 BTC, 2012): haircut taints 16,855,619 addresses ("just over 93%"), FIFO 245,120 + ("just over 1.35%"). Flexcoin (2014): haircut 10,421,112 addresses ("over 57%"), FIFO 15,265 + accounts (*Bitcoin Redux* §3.3). The repo's synthetic lane 1 echoes it: 98.0% of addresses + tainted, 658 of them under 0.1% ([FINDINGS.md](../../FINDINGS.md) row 41). + +
- [ ] You can explain "lossless" and why it makes backwards tracing possible. +
Answer + + Under FIFO a satoshi keeps a stolen-or-clean *label* (`TaintPart.name`, line 53), never a + fraction, so no information is destroyed at a merge (Step 4). Provenance therefore survives any + number of hops and you can trace a coin's heritage backwards, not just taint forwards. Haircut + destroys it on the first merge: a product of fractions like 2/9 × 3/7 is a number, not a history, + and cannot be inverted. + +
- [ ] You can write `extract_taint`'s three branches without looking. +
Answer + + (1) Queue empty (`bootstrap_taint_fifo.rs:147`) — the rest of the requested value is clean, push + a `name: 0` run. (2) Whole run fits (`:153`) — pop it, subtract its value, keep it. (3) Run + straddles the cut (`:157–162`) — split it, keep the front piece with the same `name`, push the + remainder back on the front of the queue. Branch 3 is the load-bearing one. + +
- [ ] Your `taint.rs` reproduces lane 2: poison 394.67×, haircut 1.00× over 97.9% of UTXOs, FIFO 1.00× over 0.9%. +
Answer + + Lane 2 (this topic's `notes.md`): poison inflates tainted value to 394.67× the theft (it invents + taint at every merge); haircut conserves it (1.00×) but smears it across 97.9% of UTXOs (3657 of + 3734); FIFO conserves it (1.00×) and confines it to 0.9% (32 of 3734 UTXOs, the largest holding + 22.5%). Throughput: 20,400 tx in 6.6 ms ≈ 3.1M tx/s. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md holds your five written answers — the arbitrariness-of-ordering argument, the + source of poison's extra money, an unbounded-queue-despite-coalescing sequence, the mixer claim + restated in terms of sets and notice, and the on-chain-coverage blind spot shared with topic 40. + +
## References diff --git a/topics/41-onchain-analytics/reading-blocksci.md b/topics/41-onchain-analytics/reading-blocksci.md index ea696c5..a6c1e30 100644 --- a/topics/41-onchain-analytics/reading-blocksci.md +++ b/topics/41-onchain-analytics/reading-blocksci.md @@ -10,6 +10,11 @@ for sequential access. They then benchmark that claim against Neo4j, Memgraph an which makes Table 3 the one published measurement in this book where FalkorDB's own ancestor is a baseline. Read it as a specification for what a graph engine has to do to win these queries back. +Every number below cites the section, figure or table of *BlockSci* (Kalodner et al., USENIX +Security 2020) it came from; the figures are on the December 2019 chain the paper measured. Where a +figure is one this repo measured instead, it is labelled as a bench lane and traces to this topic's +`notes.md` and [`../../FINDINGS.md`](../../FINDINGS.md). + ## The problem in one sentence **The Bitcoin blockchain is 260 GB of append-only graph-structured data that researchers want to @@ -20,14 +25,22 @@ two to three orders of magnitude.** ### Step 1 — The chain of design decisions +> **In:** the raw Bitcoin blockchain — 260 GB of blocks as of December 2019 (§1). +> **Out:** a justification for one architectural choice (an in-memory analytical database), reached +> by deleting requirements one at a time. Step 2 is the record layout that choice implies. + > BlockSci's design starts with the observation that blockchains are append-only databases; > further, the snapshots used for research are static. Thus, the ACID properties of transactional > databases are unnecessary. This makes an in-memory analytical database the natural choice. -Each link is doing work. *Append-only* means no in-place updates except length-preserving ones -(an existing output being marked spent). *Static snapshots* means no concurrent writers during -analysis. *No ACID* means no write-ahead log, no MVCC versions, no lock manager — all the -machinery topics 5, 8 and 9 build, deleted because the workload does not need it. +Each link is doing work. **Append-only** means rows are only ever added, never updated in place, +except for length-preserving edits (an existing output being marked spent). **Static snapshots** +means no concurrent writers during analysis. **ACID** (atomicity, consistency, isolation, +durability) is the set of guarantees a transactional database gives concurrent writers; "no ACID" +means no write-ahead log, no MVCC versions, no lock manager — all the machinery topics 5, 8 and 9 +build, deleted because the workload does not need it. An **in-memory analytical database** is one +that holds the whole working set in RAM and is tuned for scans and aggregates rather than for +point updates — the opposite end of the design space from the OLTP engines those topics build. And a claim you should push back on before accepting: @@ -35,13 +48,19 @@ And a claim you should push back on before accepting: > analysis has infinite COST (Configuration that Outperforms a Single Thread), in the sense that > no level of parallelism can outperform an optimized single-threaded implementation. -COST is McSherry, Isard and Murray's metric, and "infinite COST" is a strong claim: not "slower" -but "cannot be fixed by adding machines". The justification is that blockchain data is -graph-structured and therefore hard to partition — which is topic 36's vertex-cut problem -arriving from the other side. +**COST** — *Configuration that Outperforms a Single Thread* — is McSherry, Isard and Murray's +metric (HotOS 2015): the hardware parallelism a system needs before it beats one good +single-threaded program. "Infinite COST" is a strong claim: not "slower" but "cannot be fixed by +adding machines". The justification is that blockchain data is graph-structured and therefore hard +to partition — which is topic 36's vertex-cut problem arriving from the other side. ### Step 2 — The transaction record, and 19% deliberate duplication +> **In:** the design choice from Step 1 (in-memory, scan-optimised) plus the raw chain. +> **Out:** a fixed-layout transaction record (Figure 2) with inputs and outputs stored *inline*, +> and a priced-out table of four alternative layouts (Table 4). Step 3 explains how a growing file +> still presents a fixed snapshot. + Figure 2's layout, with everything sized to the byte: ``` @@ -58,38 +77,65 @@ Figure 2's layout, with everything sized to the byte: ``` Note the 60-bit value and 4-bit address type packed into one 64-bit word, and the 32-bit ids — -which is where Table 4's fourth row comes from: moving to 64-bit ids would take the Bitcoin -transaction graph from **50.09 GB to 69.26 GB**. +which is where Table 4's fourth row comes from: widening every id to 64 bits (an extra 8 bytes per +input and per output) would take the Bitcoin transaction graph from **50.09 GB to 69.26 GB**. The important decision is that inputs and outputs are stored **inline with the transaction**, not -in normalized side tables: +in normalized side tables. **Normalization** here is the database sense: storing each fact once and +referencing it, rather than duplicating it. BlockSci deliberately does the opposite: > The layout stores both inputs and outputs as part of a transaction, resulting in a small amount > of duplication (a space cost of about 19%), but resulting in a significant speedup for > sequential iteration compared to a normalized layout. -Table 4 prices the alternatives on the Dec 2019 chain (489M transactions, 1.198B inputs, 1.302B -outputs): current 50.09 GB, normalized 40.50 GB, fee-cached 54.00 GB. Normalizing saves 19% "but -it leads to a steep drop in performance for typical queries such as max-fee." That is topic 12's -argument, in a security paper. +**Where the 19% comes from — worked from Table 4.** Each memory layout costs +`bytes = 24·N_tx + 16·N_in + 16·N_out`, where `N_tx`, `N_in`, `N_out` are the transaction, input +and output counts and the coefficients are the per-record byte widths. On the December 2019 chain +(489M transactions, 1.198B inputs, 1.302B outputs) that is the "Current" row, 50.09 GB. The +"Normalized" row stores each input as a single 8-byte reference instead of a 16-byte inline copy — +so it saves 8 bytes per input: + +``` + saving = 8 bytes × N_in + = 8 × 1.198e9 = 9.58 GB + 50.09 GB − 9.58 GB = 40.51 GB (Table 4 "Normalized" = 40.50 GB) + 9.58 / 50.09 = 19.1% → the paper's "about 19%" +``` + +So the 19% is the price of storing every input's data twice (once as the spending tx's input, once +as the spent tx's output). BlockSci pays it on purpose: normalizing "leads to a steep drop in +performance for typical queries such as max-fee", because a normalized layout turns one sequential +read into a pointer chase. That is topic 12's columnar-locality argument, arriving in a security +paper. ### Step 3 — The snapshot illusion +> **In:** the memory-mapped transaction file from Step 2, which the parser keeps appending to. +> **Out:** a fixed *snapshot* view for each analysis process — a consistent past-state read out of +> a file that is still growing. Step 4 is why that view costs nothing to share. + Three properties that look contradictory: 1. The transactions table is updated on disk as new blocks arrive. 2. The table is memory-mapped and shared between all running instances. 3. Each instance sees a snapshot that never changes unless it explicitly reloads. -They coexist because the append-only property means the state at any past block height is -**reconstructible from the current state**: a `chain` object records the height at +**Memory-mapping** (`mmap`) maps a file directly into a process's address space so reads hit the +page cache with no copy and no parse. A **snapshot** is a consistent view of the data as of one +point in time. The three coexist because the append-only property means the state at any past +block height is **reconstructible from the current state**: a `chain` object records the height at initialization, the analysis library intercepts accesses to outputs spent in later blocks and -rewrites them as unspent, and accesses past the recorded height are prevented. Cheap MVCC for a -data structure that only grows — worth comparing to what topic 8 has to build when updates are -arbitrary. +rewrites them as unspent, and accesses past the recorded height are prevented. This is a cheap +form of **MVCC** (multi-version concurrency control — giving each reader a consistent version +without blocking the writer), available for free *only* because the data structure grows and never +mutates in place — worth comparing to what topic 8 has to build when updates are arbitrary. ### Step 4 — Memory mapping buys parallelism for free +> **In:** the shared memory-mapped file from Step 3, and the fact that only the parser writes it. +> **Out:** lock-free multi-reader parallelism — many analysis threads over one physical copy. Step +> 5 is the parser that produces the file in the first place. + > Memory mapping also allows multithreaded parallel processing with no additional effort. Recall > that if a file is mapped into memory by multiple processes, they use the same physical memory > for the file. The file has only one writer (the parser); it is not modified by the analysis @@ -107,6 +153,10 @@ performance scales roughly linearly with the number of virtual CPUs." ### Step 5 — The parser, and two distribution facts worth stealing +> **In:** the raw serialized blocks from Bitcoin Core. +> **Out:** the memory-mapped transaction file of Step 2, with every input linked to the output it +> spends and every output linked to an address id. Step 6 clusters those addresses. + Parsing is the hard part, because it is inherently sequential and stateful — you must link each input to the output it spends, and each output to an address id. Two measured facts about Bitcoin drive the optimisation, and both are the kind of thing you should look for in your own @@ -117,13 +167,23 @@ workload: occurrences.** A heavy tail, so caching *multi-use* addresses specifically captures half the traffic in a fraction of the space. -The resulting three-tier structure: a **bloom filter** of all seen addresses (negatives are -always correct, so it eliminates lookups for addresses that cannot exist), a **multi-use address -cache** that never evicts, and a **RocksDB** key-value store with LRU for the rest. Parsing to -block 610,695 takes **5.5 hours**, and "incremental updates are essentially instantaneous". +The resulting three-tier structure: a **bloom filter** of all seen addresses (a bloom filter is a +compact probabilistic set that never reports a false negative, so it can rule out an +address-not-seen lookup without touching disk, at the cost of occasional false positives), a +**multi-use address cache** that never evicts, and a **RocksDB** key-value store with LRU for the +rest. Parsing to block 610,695 takes **5.5 hours**, and "incremental updates are essentially +instantaneous". ### Step 6 — Address linking is union-find, and it takes minutes +> **In:** the transaction file of Step 5, plus a chosen clustering heuristic (co-spend, change). +> **Out:** a partition of addresses into *clusters* — the disjoint sets a union-find builds. Step 7 +> benchmarks queries over the whole structure. + +**Union-find** (a.k.a. disjoint-set) is the near-linear algorithm for maintaining a partition under +"merge the sets containing x and y" operations; each heuristic edge is one such merge, and the +final sets are the clusters. + > These heuristics create links (edges) in a graph of addresses. By iterating over all > transactions and applying the union-find algorithm on the contained addresses we generate > clusters of addresses... Clustering takes only a few minutes, allowing the analyst to recompute @@ -146,6 +206,12 @@ result of such a collapse." ### Step 7 — Table 3, and how to read it +> **In:** a 25M-transaction snapshot (block height 262,176) loaded into BlockSci and into Neo4j, +> RedisGraph and Memgraph. +> **Out:** per-query wall-clock times (seconds, average of five runs) that say *which query shapes* +> a scan-optimised layout wins and which a graph engine wins. Step 8 measures the cost of the +> query *interface* rather than the engine. + 25 million transactions, block height 262,176, average of five runs, in **seconds**: | query | BlockSci C++ (ST) | (MT) | Fluent (ST) | Neo4j w/o idx | Neo4j w/ idx | RedisGraph | Memgraph | @@ -179,6 +245,11 @@ question with a different label. ### Step 8 — The interface tax +> **In:** one anomalous-fee query, written three ways (pure Python, a C++ builtin helper, the +> fluent DSL) against the same engine. +> **Out:** the cost of the *interface* alone — how much of the runtime is the query language rather +> than the storage. This closes the guide's argument about layout versus abstraction. + Table 2 measures the same anomalous-fee query through three Python paradigms: ``` @@ -238,12 +309,67 @@ writes and what the machine should run is worth closing automatically. ## Done when +Answer each before unfolding it. + - [ ] You can recite the four-link design chain and say what each link deletes. +
Answer + + Append-only ⟹ static snapshots ⟹ ACID unnecessary ⟹ in-memory analytical database (Step 1, + quoting the paper). Append-only deletes in-place updates; static snapshots delete concurrent + writers; no ACID deletes the write-ahead log, MVCC and lock manager; the analytical engine keeps + the working set in RAM in a scan-friendly layout. The paper pushes further to the "infinite COST" + conjecture — no amount of distribution beats one good thread on this workload. + +
- [ ] You can draw the Figure 2 transaction record from memory, with bit widths. +
Answer + + Header: Real size 32, Base size 32, Locktime 32, Input count 16, Output count 16 — then the + inputs and outputs inline. Each input/output entry is 128 bits: Spent/spending tx 32, Address ID + 32, Value 60, Address type 4 (Step 2). Inputs and outputs are stored inline, not normalized — + a deliberate ~19% duplication. + +
- [ ] You can explain the snapshot illusion in three sentences. +
Answer + + The file is appended to on disk, memory-mapped and shared by all instances, yet each instance + sees an unchanging snapshot (Step 3). It works because append-only data lets any past state be + reconstructed from the current one: a `chain` records its height, and the library rewrites + later-spent outputs as unspent and blocks reads past that height. It is free MVCC that exists + only because the structure grows and never mutates in place. + +
- [ ] You can read Table 3 by row and say which queries a graph engine legitimately loses and why. +
Answer + + Point-lookup-plus-local-expansion queries are close: Neo4j-with-index does `Satoshi Dice address` + in 0.95 s vs BlockSci's 0.49 s, and *beats* single-threaded BlockSci on `Tx locktime > 0` + (0.05 vs 0.31 s) because an index turns a scan into a lookup. Full-scan-with-arithmetic queries + (`Calculate fee`, `Locktime change`) lose by 300–500×, because a property graph pays a pointer + chase per input where BlockSci pays a sequential read (Step 7). It is a benchmark of storage + layout on scan-shaped queries, not "graph databases are slow". + +
- [ ] You can name the two Bitcoin distribution facts the parser exploits. +
Answer + + (1) 88% of inputs spend outputs created in the last 4000 blocks → recency, so a small cache wins. + (2) Only 8.6% of addresses are used more than once, but those account for 51% of all occurrences + → a heavy tail, so a never-evicting multi-use cache captures half the traffic cheaply (Step 5). + Together they justify the bloom-filter / multi-use-cache / RocksDB three-tier parser. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md holds your five written answers — which design link breaks first for a + live-write graph database, the strongest form of the infinite-COST claim and a workload that + refutes it, the query-plan predicate that splits Table 3's winners from losers, the + break-even for the 19% inline-layout cost, and a measurement distinguishing the three + explanations for supercluster growth. + +
## References diff --git a/topics/41-onchain-analytics/reading-elliptic-aml.md b/topics/41-onchain-analytics/reading-elliptic-aml.md index a135e23..a965f31 100644 --- a/topics/41-onchain-analytics/reading-elliptic-aml.md +++ b/topics/41-onchain-analytics/reading-elliptic-aml.md @@ -10,6 +10,11 @@ worth more to a database engineer than a win would have been, because it says so about when graph structure pays for itself — and because a single real-world event, the shutdown of a dark market in the middle of the test period, breaks every model in the table. +Every number below is quoted from *Anti-Money Laundering in Bitcoin* (Weber et al., KDD 2019 AML +workshop, [arXiv:1908.02591](https://arxiv.org/abs/1908.02591)): the counts from §2, the scores +from Table 1 and Table 2. Nothing here is a repo-measured lane — this guide is a reading companion, +not a bench. + ## The problem in one sentence **Classify each Bitcoin transaction as licit or illicit from a labelled graph where only 23% of @@ -20,6 +25,11 @@ under you.** ### Step 1 — The data set +> **In:** the public Elliptic data set — a labelled sub-graph of the Bitcoin transaction network. +> **Out:** the raw material every later step consumes: 203,769 nodes, 234,355 edges, 166 features +> per node, 49 time steps, and a 2%/21%/77% illicit/licit/unlabelled label split. Step 2 dissects +> the 166 features. + ``` 203,769 node transactions 234,355 directed edge payment flows (the full Bitcoin network: ~438M nodes, 1.1B edges — this is a subgraph) @@ -40,6 +50,11 @@ guides in this topic. Circularity worth noticing. ### Step 2 — Two kinds of feature, and what the split measures +> **In:** the 166-feature node vectors from Step 1. +> **Out:** a split of those features into 94 *local* (about the transaction itself) and 72 +> *aggregated* (one-hop neighbourhood statistics) — the split that makes the later GCN comparison +> fair. Step 3 looks at the time dimension. + The 166 features divide sharply: - **94 local features**: time step, number of inputs/outputs, transaction fee, output volume, and @@ -48,9 +63,12 @@ The 166 features divide sharply: coefficients of the *same* local features taken over the node's one-hop neighbours, forward and backward. -So the 72 are a hand-rolled, single-layer, fixed-aggregation message pass. This is the comparison -the paper is really running: **hand-built one-hop aggregation vs a learned multi-hop one**. Keep -that in mind when you read the results, because it is a fairer fight than "features vs graphs". +So the 72 are a hand-rolled, single-layer, fixed-aggregation message pass. (A **message pass** is +the operation at the heart of a graph neural network: each node updates its own vector from an +aggregate of its neighbours' vectors; doing it once reaches one-hop neighbours, twice reaches +two-hop, and so on.) This is the comparison the paper is really running: **hand-built one-hop +aggregation vs a learned multi-hop one**. Keep that in mind when you read the results, because it +is a fairer fight than "features vs graphs". The paper flags the limitation itself: "In building the 72 aggregated features, the problem of heterogeneous neighborhoods is addressed by naively constructing statistical aggregates @@ -59,6 +77,10 @@ solution is sub-optimal because it carries a significant loss of information." ### Step 3 — The temporal structure, and the choice that deletes it +> **In:** the 49 time steps from Step 1, and the edge set from Step 2's neighbourhoods. +> **Out:** the key structural fact — the 49 steps are *disjoint* graphs with no edges between them +> — and the train/test split built on it. Step 4 reads the scores this split produces. + Each of the 49 time steps is "a single connected component of transactions that appeared on the blockchain within less than three hours between each other; **there are no edges connecting different time steps**." @@ -70,11 +92,27 @@ detect. Topic 33 spends a whole topic on time-respecting paths; this data set ma impossible by construction. That is a modelling decision with consequences, and it is the subject of one of the questions below. -The evaluation uses a **70:30 temporal split**: train on steps 1–34, test on 35–49. Temporal, not -random — correct, and much harder. +The evaluation uses a **70:30 temporal split**: train on the earliest 34 steps, test on the last +15 (steps 35–49). A **temporal split** trains only on data from before a cut-off time and tests +only on data after it, unlike a random split that shuffles all rows together — it is the honest +way to measure a model that must predict the future, and much harder, because the test +distribution is genuinely unseen. ### Step 4 — The results table, read carefully +> **In:** the train/test split from Step 3, the features from Step 2, and seven classifiers +> (Logistic Regression, Random Forest, MLP, GCN, Skip-GCN, EvolveGCN). +> **Out:** Table 1 — illicit-class precision, recall and F1 per model — the headline result that +> the Random Forest beats the GCN. Step 5 is the event that breaks all of them. + +First, the metrics, because the whole table is read through them. For the illicit class, +**precision** is the fraction of transactions *flagged illicit* that really are (`TP / (TP + FP)`); +**recall** is the fraction of *actually illicit* transactions that were flagged (`TP / (TP + FN)`); +**F1** is their harmonic mean, `2·P·R / (P + R)`, which is low unless *both* are high. A **GCN** +(graph convolutional network) is a neural network whose layers are learned message passes (Step 2) +over the graph; a **node embedding** (`NE`) is the vector a GCN learns for each node, which can be +fed to another model as extra features. + Illicit-class precision / recall / F1, plus micro-averaged F1. `AF` = all 166 features, `LF` = the 94 local ones only, `NE` = node embeddings from a GCN concatenated on. @@ -112,6 +150,10 @@ Four readings, in order of usefulness: ### Step 5 — The dark market shutdown: the finding that actually matters +> **In:** the per-time-step predictions of every model from Step 4, plotted over the 15 test steps. +> **Out:** Figure 2's finding — at time step 43 a real-world event collapses every model's illicit +> F1, and re-training does not recover it. Step 6 explains why the aggregate metric hides this. + At time step 43 — inside the test period — a dark market closed. Figure 2 plots illicit F1 per time step, and every method falls off a cliff there and does not recover. @@ -133,11 +175,33 @@ equivalent move here, which is the honest state of the art. ### Step 6 — Why 2% illicit changes the whole evaluation -Micro-averaged F1 is above 0.92 for *every* method in the table, including the worst one. It is -meaningless: with 2% illicit, a classifier that says "licit" always scores well. The paper trains -the GCN "using a weighted cross entropy loss to provide higher importance to the illicit samples" -at a 0.3/0.7 ratio and reports illicit-class metrics separately — do the same in any comparable -setting, and be suspicious of any AML result quoted as accuracy. +> **In:** the `micro F1` column of Step 4's table and the 2% illicit base rate from Step 1. +> **Out:** the reason that column is worthless and the illicit-class columns are not — a +> class-imbalance argument, worked below. Step 7 generalises the lessons. + +**Micro-averaged F1** pools true positives, false positives and false negatives across *all* +classes before computing one score; for single-label classification that makes micro-precision, +micro-recall and micro-F1 all equal to plain accuracy. So it is above 0.92 for *every* method in +the table, including the worst one, and it is meaningless here. Work it: + +``` + predict "licit" for every node. + let p = fraction illicit. + + full node set, p = 0.02: + accuracy = 1 − p = 0.98 → micro-F1 = 0.98 + labelled test set actually scored, illicit = 4,545 of (4,545 + 42,019) = 46,564: + p = 4,545 / 46,564 = 0.098 + accuracy = 1 − 0.098 = 0.902 → micro-F1 ≈ 0.90 +``` + +Either way the do-nothing classifier scores ~0.90–0.98 micro-F1 — higher than the GCN's *illicit* +F1 of 0.628. A metric a trivial classifier wins is no metric. That is why the paper reports +illicit-class precision/recall/F1 separately, and trains the GCN "using a weighted cross entropy +loss to provide higher importance to the illicit samples" at a 0.3/0.7 ratio. **Weighted cross +entropy** simply multiplies each class's contribution to the loss by a weight, so mislabelling a +rare illicit node costs the optimiser more than mislabelling a common licit one — the training-time +counterpart to reporting the minority class separately. The paper also frames the business constraint precisely: "Industry standard high false positive rates of upwards of 90% inhibit this effort. We want to reduce false positive rates without @@ -146,6 +210,11 @@ increasing false negative rates." Random Forest's 0.956 precision against Logist ### Step 7 — What this says about graph ML generally +> **In:** everything above — the fair-fight feature split (Step 2), the results (Step 4), the +> distribution shift (Step 5), the imbalance (Step 6). +> **Out:** three transferable rules for deciding when graph structure earns its complexity. This is +> the guide's takeaway. + Three transferable lessons: 1. **A learned aggregation must beat a hand-built one to be worth it.** Here the hand-built @@ -198,13 +267,66 @@ Three transferable lessons: ## Done when +Answer each before unfolding it. + - [ ] You can state the data set's size, label balance and temporal structure from memory. +
Answer + + 203,769 nodes (transactions), 234,355 directed edges (payment flows), 166 features each; 4,545 + illicit (2%), 42,019 licit (21%), the rest unlabelled (77%); 49 time steps ~2 weeks apart, + 1,000–8,000 nodes each, and crucially **no edges between time steps** — 49 disjoint graphs + (Steps 1 and 3). + +
- [ ] You can explain the 94/72 feature split and why it makes the GCN comparison a fair fight. +
Answer + + 94 local features describe the transaction itself; 72 aggregated features are min/max/std/corr of + those local features over one-hop neighbours (Step 2). The 72 are therefore a hand-built, + single-layer, fixed-aggregation message pass — so Table 1 is really "hand-built one-hop + aggregation vs a learned multi-hop GCN", a fairer contest than "features vs graphs". + +
- [ ] You can give the headline result (RF 0.788/0.796 vs GCN 0.628, Skip-GCN 0.705, EvolveGCN 0.720) and the paper's explanation for it. +
Answer + + Random Forest on all features scores illicit-F1 0.788, and 0.796 with GCN embeddings added — the + best row; the plain GCN scores 0.628, Skip-GCN 0.705, EvolveGCN 0.720 (Step 4). The paper's + explanation: RF ensembles many decision trees by voting, whereas a GCN ends in a logistic- + regression output layer and is "a nontrivial generalization of Logistic Regression" — which sits + at the bottom of the table. + +
- [ ] You can describe the dark market shutdown and why re-training does not fix it. +
Answer + + At test time step 43 a dark market closed; Figure 2 shows every model's illicit F1 collapses and + never recovers, and even a Random Forest re-trained after every step with fresh ground truth + cannot capture the new illicit transactions (Step 5). It is a distribution-shift problem, not a + stale-model one: the post-shutdown illicit behaviour is genuinely different, so fitting the old + behaviour cannot help. + +
- [ ] You can say why micro-F1 is the wrong metric here. +
Answer + + Micro-F1 pools all classes and equals accuracy for single-label prediction, so a + predict-licit-always classifier scores 1 − p: about 0.98 on the full 2%-illicit node set, or + ≈0.90 on the labelled test set where illicit is 4,545/46,564 = 9.8% (Step 6). Both beat the GCN's + *illicit* F1 of 0.628, so micro-F1 rewards doing nothing — which is why the paper reports + illicit-class precision/recall/F1 separately. + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md holds your five written answers — Table 1 recast as learned-vs-hand-built + aggregation, two laundering behaviours the missing cross-time edges make undetectable, the + worked predict-licit-always micro-F1, concept drift vs covariate shift for the dark-market + collapse, and what a licit/illicit super-cluster merge would do to labels, features and F1. + +
## References diff --git a/topics/41-onchain-analytics/reading-fistful-of-bitcoins.md b/topics/41-onchain-analytics/reading-fistful-of-bitcoins.md index 88aca25..3f290b2 100644 --- a/topics/41-onchain-analytics/reading-fistful-of-bitcoins.md +++ b/topics/41-onchain-analytics/reading-fistful-of-bitcoins.md @@ -9,6 +9,11 @@ profiles, and understanding *why* is a lesson about any system that merges recor One keys on a property of the protocol and cannot be wrong. The other keys on a habit, and being wrong once welds two strangers together forever. +Every number below is quoted from *A Fistful of Bitcoins* (Meiklejohn et al., IMC 2013): the parse +counts and cluster counts from §2–§4, the false-positive ladder from §4.5, the service figures from +§5. Where a figure is one this repo measured instead, it is labelled as lane 3 and traces to this +topic's `notes.md` and [`../../FINDINGS.md`](../../FINDINGS.md). + ## The problem in one sentence **Twelve million public keys are not twelve million people — but deciding which of them are one @@ -18,31 +23,45 @@ person is an inference from behaviour, and a merge you get wrong can never be un ### Step 1 — Address is not identity, and neither is a cluster -A Bitcoin address is a public key; anyone can make as many as they like, for free, and wallet -software does exactly that. The paper's parse of the chain to 13 April 2013 found **231,207 +> **In:** the public Bitcoin blockchain parsed to 13 April 2013. +> **Out:** 12,056,684 distinct public keys and a working definition of *control* — the relation the +> two heuristics will cluster on. Step 2 is the first heuristic. + +A Bitcoin **address** is a public key; anyone can make as many as they like, for free, and wallet +software does exactly that. (A **public key** is one half of a cryptographic keypair; spending +money sent to it requires the matching private key, so possession of that private key is what +"control" ultimately means.) The paper's parse of the chain to 13 April 2013 found **231,207 blocks, 16,086,073 transactions and 12,056,684 distinct public keys** — for a user base orders of magnitude smaller. -The paper is careful about what a cluster means, and you should be too. It defines *control*, not -ownership: "the controller of an address is the entity that is expected to participate in -transactions involving that address." If you buy a physical bitcoin from a vendor who knows the -private key, and then redeem it at Mt. Gox, three parties have known that key. Clustering answers -"who transacts with this", which is what an investigator wants and is not the same as "who owns -this". +The paper is careful about what a cluster means, and you should be too. It defines **control**, not +ownership — the entity that can sign for an address, which is not necessarily its economic owner: +"the controller of an address is the entity (or in exceptional cases multiple entities) that is +expected to participate in transactions involving that address." If you buy a physical bitcoin from +a vendor who knows the private key, and then redeem it at Mt. Gox, three parties have known that +key. Clustering answers "who transacts with this", which is what an investigator wants and is not +the same as "who owns this". ### Step 2 — Heuristic 1: co-spending is a protocol property +> **In:** the transaction set from Step 1, treated as a hypergraph whose edges are transactions. +> **Out:** a partition of the 12M public keys into 5,579,176 co-spend clusters — the *safe* half of +> the method. Step 3 sets up the second, riskier heuristic. + > **Heuristic 1.** If two (or more) addresses are inputs to the same transaction, they are > controlled by the same user. Because spending an output requires a signature from its key, whoever assembled a transaction with inputs A and B held both private keys. The relation is transitive — if one transaction joins {A, B} and another joins {B, C}, then A, B and C are one user — so the whole computation is -a **union-find over the co-spend hypergraph**, one linear pass over the transactions. +a **union-find over the co-spend hypergraph**, one linear pass over the transactions. (**Union-find** +is the near-linear disjoint-set algorithm for merging groups under "these two belong together"; a +**hypergraph** is a graph whose edges can join more than two vertices at once, which is exactly +what a multi-input transaction is — one edge over all its input addresses.) The paper's phrasing of why it is safe is worth keeping: "it is also quite safe: the sender in -the transaction must know the private key belonging to each public key used as an input, so it is -unlikely that the collection of public keys are controlled by multiple entities (as these +the transaction must know the private signing key belonging to each public key used as an input, +so it is unlikely that the collection of public keys are controlled by multiple entities (as these entities would need to reveal their private keys to each other)." Result on the 2013 chain: **12,056,684 public keys → 5,579,176 clusters**. Accounting for "sink" @@ -55,11 +74,17 @@ entities co-spend. That is not the crate being kind — it is the heuristic bein ### Step 3 — Change addresses, and why they leak +> **In:** the co-spend clusters from Step 2, still missing the links that co-spending never +> reveals. +> **Out:** the *change address* — the fresh address a wallet sends surplus to — as the leak a +> second heuristic can exploit. Step 4 states that heuristic precisely. + A payment rarely matches a UTXO exactly. Spending a 10 BTC output to pay 3 BTC means creating two -outputs: 3 to the payee and 7 back to yourself, at a *fresh* address the wallet generated. The -paper's Definition 4.2 makes the underlying fact precise — "a public key can therefore spend -money only as many times as it has received money (again, because each time it spends money it -must spend all of it at once)." +outputs: 3 to the payee and 7 back to yourself, at a *fresh* address the wallet generated. That +fresh self-directed output is the **change address**: a brand-new address, made by the sender's own +wallet, holding the leftover of a spend. The paper's Definition 4.2 makes the underlying fact +precise — "a public key can therefore spend money only as many times as it has received money +(again, because each time it spends money it must spend all of it at once)." If you can pick out which output is the change, you have linked the sender's brand-new address to the addresses they just spent from, and you can keep doing it forever. That is the prize; the @@ -67,6 +92,11 @@ next step is the trap. ### Step 4 — Heuristic 2: Definition 4.3, all four conditions +> **In:** each transaction `t`, and the co-spend clusters from Step 2. +> **Out:** at most one output of `t` labelled its *one-time change address*, adding a new edge to +> the cluster graph — or a refusal when the transaction is ambiguous. Step 5 measures how often +> that label is wrong. + A public key `pk` is a *one-time change address* for a transaction `t` when: ``` @@ -91,6 +121,10 @@ lack robustness in the face of changing (or adversarial) patterns in the network ### Step 5 — The false-positive ladder: precision bought with latency +> **In:** the change labels Heuristic 2 assigns at each block height (Step 4). +> **Out:** a false-positive rate per refinement — 13% down to 0.17% — measured behaviourally, and +> the lesson that latency buys precision. Step 6 shows why even 0.17% is dangerous. + The authors had no ground truth, so they measured the false-positive rate *behaviourally*: if an address met Definition 4.3 at block height h — meaning it looked like a one-time change address — and was then used again later, the label was wrong. @@ -109,6 +143,10 @@ Exercise 5 of this topic asks you to reproduce the curve. ### Step 6 — Cluster collapse: why a 0.17% error rate is still dangerous +> **In:** the 0.17% mislabelled change addresses from Step 5, fed into the union-find of Step 2. +> **Out:** the reason a tiny *label* error rate becomes a catastrophic *partition* error — worked +> as pair-precision arithmetic below. Step 7 weighs the payoff against this risk. + Even after all of that, the refined run produced "a giant super-cluster containing the public keys of Mt. Gox, Instawallet, BitPay, and Silk Road, among others; in total, this super-cluster contained **1.6 million public keys**." @@ -121,11 +159,28 @@ in the definition: 2. Self-change addresses (allowed by advanced wallets like Armory and My Wallet) later used separately with a new address, so the new address is falsely labelled. -The deep problem is that union-find is transitive and has no undo. A 0.17% error rate on *labels* -is not a 0.17% error rate on the *partition*: each false merge fuses two whole components, so -errors compound multiplicatively while correct merges only add. This is the arithmetic that makes -a heuristic with recall 0.04 and precision 1.000 more useful than one with recall 0.45 and -precision 0.09. +The deep problem is that union-find is transitive and has no undo. Measure cluster quality by +**pair precision** — of all address *pairs* placed in the same cluster, the fraction that really +are the same user — and **pair recall** — of all pairs that really are the same user, the fraction +grouped together. A false merge and a missed merge damage these asymmetrically: + +``` + Two real users, 1,000 addresses each. Correct same-user pairs = 2 × C(1000,2) = 999,000. + + ONE false merge unites them into one 2,000-address cluster: + same-cluster pairs = C(2000,2) = 1,999,000 + of which cross-user (all wrong) = 1000×1000 = 1,000,000 + pair precision = 999,000 / 1,999,000 ≈ 0.50 ← one mistake halves precision + + ONE missed merge instead splits a true 1,000-cluster into 500+500: + lost true pairs = 500×500 = 250,000 + pair recall = (999,000−250,000) / 999,000 ≈ 0.75 ← precision stays 1.000 +``` + +A false merge creates `a×b` wrong pairs (multiplicative in the cluster sizes); a missed merge only +withholds pairs (it costs recall, never precision). That asymmetry is why a heuristic with recall +0.04 and precision 1.000 is more useful than one with recall 0.45 and precision 0.09 — you can +always union more clusters later, but you can never un-merge a wrong one. Lane 3 of the crate plants exactly mechanism 1 and sweeps it: @@ -143,6 +198,10 @@ million addresses** and says it is "likely a result of such a collapse." ### Step 7 — Why the clusters were worth it anyway +> **In:** the co-spend + change clusters (Steps 2–6) seeded with 1,070 hand-tagged addresses. +> **Out:** 2,197 named clusters over 1.8M addresses, and the structural finding that services are +> chokepoints. This closes the paper's argument that pseudonymity leaks at the exchange. + The payoff is leverage. Hand-tagging 1,070 addresses through 344 transactions, then clustering, let the authors name **2,197 clusters accounting for over 1.8 million addresses** — "Heuristic 2 allowed us to name 1,600 times more addresses than our own manual observation provided." @@ -151,8 +210,8 @@ And the structural finding, §5: services are chokepoints. Satoshi Dice alone ac **60% of all Bitcoin activity** at the time, and **21% of all bets (896,864 of 4,127,979)** were exactly the 0.01 BTC minimum. Exchanges are chokepoints too, which is what makes the whole enterprise matter: "the demonstrated centrality of these services makes it difficult for even -highly motivated individuals — e.g., thieves or others strongly attracted to the anonymity -properties of Bitcoin — to stay completely anonymous, if they are interested in cashing out." +highly motivated individuals — e.g., thieves or others attracted to the anonymity properties of +Bitcoin — to stay completely anonymous, provided they are interested in cashing out." ## How to read the paper (with the concepts in hand) @@ -195,13 +254,63 @@ properties of Bitcoin — to stay completely anonymous, if they are interested i ## Done when +Answer each before unfolding it. + - [ ] You can state both heuristics and explain why one is a protocol property and one is not. +
Answer + + Heuristic 1: addresses that are inputs to the same transaction share a controller — safe, because + co-spending requires holding every input's private *signing* key, so faking it means strangers + swapping private keys (Step 2). Heuristic 2: the one-time change address of Definition 4.3 — a + *usage* pattern, not a protocol rule, so it "lack[s] robustness in the face of changing (or + adversarial) patterns" (Step 4). + +
- [ ] You can recite Definition 4.3's four conditions and say what each one rules out. +
Answer + + (1) `pk` appears for the first time — change is a fresh address. (2) `t` is not a coin generation + — coinbase outputs are not change. (3) No output address is also an input — rules out self-change + (23% of transactions). (4) No *other* output is also brand-new — forces the heuristic to decline + when it cannot tell payment from change (Step 4). Condition 4 is the one people forget. + +
- [ ] You can explain why union-find makes a low label-error rate into a high partition-error rate. +
Answer + + Merges are transitive and cannot be undone, so one false merge of clusters sized a and b creates + a×b wrong same-user pairs. Two 1,000-address clusters wrongly merged make 1,000,000 false pairs + and drop pair precision from 1.000 to ≈0.50 in a single mistake, while a missed merge only costs + recall (Step 6). Errors compound multiplicatively; corrections only add. + +
- [ ] You can quote the false-positive ladder and name what buys each step. +
Answer + + Naive Definition 4.3: 555,348 false positives = 13%. Ignore the Satoshi Dice payout pattern → 1%. + Wait a day before labelling → 0.28%. Wait a week → 0.17% (7,382 addresses) (Step 5). Precision is + bought with latency — the same heuristic is ~76× more precise if you are willing to wait a week. + +
- [ ] Your `clustering.rs` reproduces lane 3: precision 1.000 for co-spend at every reuse rate, and the 1.000 → 0.089 → 0.009 collapse for the change heuristic. +
Answer + + Co-spend (Heuristic 1) precision stays 1.000 at every change-reuse rate because the generator + never lets two entities co-spend. The change heuristic collapses as reuse rises: precision 1.000 + at 0.00 (largest cluster 93), 0.661 at 0.01 (366), 0.089 at 0.05 (1894), 0.009 at 0.10 (7991 = + 71% of addresses) (Step 6, this topic's `notes.md`). + +
- [ ] You wrote answers to all five questions in notes.md. +
Answer + + Done when notes.md holds your five written answers — a real construction that breaks Heuristic 1 + and BlockSci's response, the effect of dropping condition 4, the pair-precision-vs-recall + arithmetic for one false vs one missed merge, the operational cost of the week's delay for three + actors, and which single heuristic you would ship at an exchange. + +
## References diff --git a/topics/42-recommendations-social/README.md b/topics/42-recommendations-social/README.md index 52ad37c..86039b1 100644 --- a/topics/42-recommendations-social/README.md +++ b/topics/42-recommendations-social/README.md @@ -30,8 +30,8 @@ nonzero at all because we filter out items each user already has — everybody is being handed the same list. Pixie's unmodified Algorithm 1 does personalize, but **45% of what it returns is the bestseller list again**, because an unbiased walk's visit distribution drifts toward -degree. That is Pixie's own complaint, stated in §3.1: "In classical -random walk low degree nodes with fewer edges contribute less signal. +degree. That is Pixie's own complaint, stated in §1: "In classical +random walks low degree nodes with fewer edges contribute less signal. This is undesirable because smaller boards ... are more likely to produce highly relevant recommendations." diff --git a/topics/42-recommendations-social/reading-graphjet.md b/topics/42-recommendations-social/reading-graphjet.md index 806a013..1ced0b9 100644 --- a/topics/42-recommendations-social/reading-graphjet.md +++ b/topics/42-recommendations-social/reading-graphjet.md @@ -19,6 +19,11 @@ milliseconds, and never let it grow without bound.** ### Step 1 — The unconventional bet: one machine +> **In:** nothing yet — this step fixes the design premise the whole engine is built on. +> **Out:** the single-server bet and its 80 GB arithmetic (10 billion edges × 8 bytes), the +> constraint every later step honours: the graph lives in one machine's RAM, so the work is *fitting +> it there* (Steps 4–6) rather than partitioning it. + Twitter's recommendation work began with WTF ("Who To Follow") in 2010, and the first decision was the one everybody questions: @@ -39,6 +44,11 @@ real-world problems." ### Step 2 — Four generations, and what each one got wrong +> **In:** the single-server constraint from Step 1. +> **Out:** four systems in sequence (Cassovary → Hadoop → MagicRecs → GraphJet) and the failure +> that killed each — culminating in MagicRecs' reformulation of "B→C edges in a time window" as an +> **intersection of adjacency lists**, which is the primitive GraphJet turns into a storage engine. + - **Cassovary (2010)** — in-memory, single server, snapshots of the follow graph from HDFS, computed *circle of trust* (an egocentric random walk = personalized PageRank) and SALSA. It worked. Its limit: snapshots could only be refreshed about once a day, so new users got nothing @@ -56,6 +66,11 @@ real-world problems." ### Step 3 — The API is five methods, and the omissions are the design +> **In:** MagicRecs' single hard-coded push rule from Step 2. +> **Out:** GraphJet's five-method interface — insert one edge, iterate a vertex's edges, sample `k` +> of them (both left and right) — and the three omissions (no delete, no timestamp, sampling *with +> replacement*) that make Steps 4–8 cheap. + ``` insertEdge(u, t, r) insert user→tweet edge of type r getLeftVertexEdges(u) iterator over (t, r) incident to u @@ -83,6 +98,12 @@ is enough for a random walk and much cheaper than a snapshot. ### Step 4 — Temporal index segments +> **In:** the write-and-sample API from Step 3, plus the "never grow without bound" requirement from +> the problem sentence. +> **Out:** the graph split into **temporally-ordered index segments** — one active (writable), the +> rest immutable, the oldest dropped whole — the structure Steps 5 (id narrowing), 6 (write-side +> allocator) and 8 (read-side relayout) each exploit. + The graph is partitioned into **temporally-ordered index segments**. Only the newest accepts writes; the rest are immutable. A segment older than *n* hours is discarded whole. @@ -108,6 +129,11 @@ adjacency lists, again the same problem. ### Step 5 — Id mapping and bit-packing +> **In:** a single segment's bounded vertex set from Step 4. +> **Out:** a segment-internal id — 64-bit external ids hashed into a small per-segment id, then +> **bit-packed** with the edge type into one 32-bit integer, so an adjacency list is just a +> `u32` array. Step 6 allocates space for those arrays. + External vertex ids are 64-bit. Within a segment, they are hashed to a segment-internal id using double hashing in an open-addressed table — and crucially "we use the hash value as our internal vertex id", so the table cannot be rehashed to grow. The workaround is a chain of power-of-two @@ -122,6 +148,11 @@ array of 32-bit integers. ### Step 6 — Edge pools: the allocator as a model of the data +> **In:** the 32-bit edge entries from Step 5, arriving one at a time into the active segment. +> **Out:** the **doubling edge-pool allocator** — each adjacency list stored as a chain of +> power-of-two slices (`P_r` holds slices of `2^r` edges) — whose growth curve *is* a bet that the +> data follows a power law. Step 8 tears this down once the segment seals. + This is the part to steal. Adjacency lists cannot be kept contiguous as the graph grows (you would relocate constantly), so GraphJet stores each list as a chain of **slices** — and the slice sizes **double**: @@ -144,6 +175,19 @@ The justification is a statement about the data, not about memory: > an edge incident to a vertex, the more likely that more edges will follow. Hence, it makes > sense to exponentially increase the amount of allocated space each time. +**Preferential attachment** is the "rich get richer" process — a vertex that already has many edges +is disproportionately likely to gain more — and it is the standard generative story for a +**power-law** degree distribution (a handful of vertices with enormous degree, a very long tail of +tiny ones). The allocator bakes that assumption into its growth curve. + +The general rule for an arbitrary degree `d`: the edges fill pools in order, one slice per pool, +where pool `P_r`'s slice holds `2^r` edges. A vertex of degree `d` therefore occupies slices in +`P_1 … P_k`, where `k` is the smallest integer with cumulative capacity +`2^{k+1} − 2 = 2 + 4 + … + 2^k ≥ d`, and its top slice has `2^{k+1} − 2 − d` unused slots. Check it +against the degree-25 case: cumulative capacities are `P1→2, P1..P2→6, P1..P3→14, P1..P4→30`, and +`30 ≥ 25` first at `k = 4`, so the vertex spans `P1..P4` with `30 − 25 = 5` free slots — exactly the +figure above. + Because the slice sizes are fixed and known, "we know from the vertex degree where to insert the next edge and how much space is left in the current slice" — no per-vertex metadata beyond the degree and the slot indices. Exercise 5 asks you to build it and measure bytes-per-edge against a @@ -151,6 +195,11 @@ doubling `Vec`. ### Step 7 — One writer, no locks +> **In:** the active segment's edge pools from Step 6, written by one thread and read by many. +> **Out:** the **single-writer, multi-reader** concurrency model — no write–write conflicts to +> guard, only memory-visibility handled with memory barriers — which is why the entire latch +> hierarchy topic 9 builds is simply absent here. + "Since we adopt a single-writer, multi-reader design, there is no need to worry about write–write conflicts." Edge insertions all come from one thread reading a Kafka queue; reads are served by many threads; and "judicious use of memory barriers is sufficient to address memory visibility @@ -163,6 +212,11 @@ make the writer singular, the entire latch hierarchy disappears. ### Step 8 — Sealed-segment relayout, and O(1) sampling +> **In:** a segment that has just stopped accepting edges (sealed), still in the write-optimized +> edge-pool layout of Step 6. +> **Out:** a compacted, gap-free, read-optimized relayout of that segment, plus the **alias method** +> that makes cross-segment sampling O(1) — the primitive Step 9's random walks call. + Once a segment stops accepting edges, a background thread rebuilds it: > since the graph partition is now immutable, we no longer need the edge pool structure to store @@ -184,7 +238,14 @@ added after the API call are not visible." ### Step 9 — SALSA, full and subgraph -The recommendation algorithms are random walks on the bipartite graph. **Full SALSA**: start from +> **In:** the O(1) edge-sampling primitive from Step 8. +> **Out:** the two recommendation algorithms that ride on it — full **SALSA** and subgraph SALSA — +> and the memory-versus-quality trade between them (roughly half the index, at the cost of +> second-order paths). + +The recommendation algorithms are random walks on the bipartite graph. **SALSA** (*Stochastic +Approach for Link-Structure Analysis*) is a bipartite random walk that alternates sides and ranks +vertices by how often the walk visits them. **Full SALSA**: start from the user (or a *seed set* — the circle of trust, which handles users with no interactions), alternate left→right→left, restart with probability α, and rank right-hand vertices by visit distribution. @@ -201,6 +262,10 @@ Fitting in cache and halving the index, at the cost of second-order paths. Both ### Step 10 — The numbers +> **In:** the complete engine of Steps 4–9, deployed. +> **Out:** the measured envelope — insertion throughput, per-request latency percentiles, capacity +> per machine, availability — the figures your own build should be judged against. + Two Intel Xeon 6-core E5-2620 v2 at 2.10 GHz: ``` @@ -220,6 +285,11 @@ with increasingly stale data in memory)." ### Step 11 — §7.3, the paragraph for anyone building on Redis +> **In:** the allocator (Step 6) and temporal pruning (Step 4) as the two things that distinguish +> GraphJet from a generic list store. +> **Out:** §7.3's verdict — the two specific, named reasons Redis's `LPUSH` cannot be the graph +> store, which is exactly the gap capstone M42 asks you to close. + > It is possible, of course, to use any key–value store to hold the adjacency lists that comprise > a graph, thus serving as a real-time graph store ... but Redis in particular supports a command > (LPUSH) that inserts specified values at the head of the list stored at a key. The @@ -274,14 +344,103 @@ is exactly what capstone M42 asks you to build. ## Done when +Answer each before unfolding it. + - [ ] You can state the single-server argument and its 80 GB arithmetic. + +
Answer + + Twitter bet the whole graph fits in one server's RAM rather than partitioning it (§2.1): "we took + exactly the opposite approach of scaling up on individual large-memory (but still commodity) + servers." The arithmetic: a graph of ten billion edges, stored naïvely as an edge list at 8 bytes + per edge, is "a mere 80 GB, which is well in the range of memory available on commodity servers." + The payoff is that the hard problems become *fitting it in memory* (id narrowing, the doubling + allocator) instead of distributed coordination — and the paper openly doubts distributed graph + stores are as important as the literature treats them. + +
+ - [ ] You can name all four generations and what killed each. + +
Answer + + **Cassovary (2010)** — in-memory single server, HDFS snapshots, circle of trust + SALSA; killed by + once-a-day snapshot freshness, so new users got nothing (cold start). **RealGraph on Hadoop + (2012)** — richer behavioural signals, no longer memory-resident; killed by being batch, "roughly + daily," dissonant with the live Twitter experience. **MagicRecs (2013)** — real-time push on + edge arrival, its key move recasting "B→C edges in a time window" as an *intersection of adjacency + lists*; limited to one hard-coded rule, ~7 s median latency dominated by message propagation. + **GraphJet (2014)** — MagicRecs generalized into a real storage engine with a five-method API. + +
+ - [ ] You can draw the index-segment picture and say what immutability buys. + +
Answer + + A row of temporally-ordered segments; only the newest takes writes (single writer), the rest are + read-only, and a segment older than *n* hours is discarded whole. Immutability buys three things + (§3.3): pruning is coarse-grained and free (drop a segment, no per-edge expiry — "does not have a + noticeable impact on recommendation quality"); only the one active segment needs write-optimized + structures, so sealed segments can be relaid out for reads (Step 8); and each segment's bounded + vertex set lets 64-bit ids collapse to small segment-internal ids (Step 5). The idea is borrowed + from Earlybird. + +
+ - [ ] You can write the edge-pool layout for an arbitrary degree. + +
Answer + + Pool `P_r` holds slices of `2^r` edges; a vertex fills one slice per pool in order. A degree-`d` + vertex spans `P_1 … P_k`, where `k` is the smallest integer with `2^{k+1} − 2 ≥ d` (cumulative + capacity `2 + 4 + … + 2^k`), and the top slice has `2^{k+1} − 2 − d` free slots. Degree 25 → + cumulative `2, 6, 14, 30`; `30 ≥ 25` at `k = 4`, so `P1..P4` with `30 − 25 = 5` free. The doubling + is deliberate: it "implicitly assumes some type of preferential attachment effect," so more edges + are expected precisely where edges already exist. + +
+ - [ ] You can explain why single-writer removes the need for locks entirely. + +
Answer + + All edge insertions come from one thread draining a Kafka queue, so there are no write–write + conflicts to serialize — the only remaining concern is that readers see writes, which "judicious + use of memory barriers is sufficient to address," and barriers are cheap enough that the penalty + is acceptable. It works because a single writer sustains ~1,000,000 edges/s, well above the + steady-state engagement rate, so one writer is never the bottleneck. Contrast topic 9: when you + can make the writer singular, the entire latch hierarchy disappears. + +
+ - [ ] You can quote §7.3's two gaps and connect them to capstone M42. + +
Answer + + §7.3: Redis's `LPUSH` could hold adjacency lists, but "the implementation of the command ... lacks + the memory allocation optimizations in GraphJet," and "Redis lacks a mechanism for pruning these + lists; although it would be possible to implement temporal partitioning, it would basically be + replicating some of the main design features in GraphJet." The two gaps are the doubling + **allocator** (Step 6) and **temporal pruning via segments** (Step 4) — which is precisely the + feature list capstone M42 asks you to add to a Redis-module graph store. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions live in `notes.md`'s guide-question checklist. The load-bearing ones: Q1 (no + deletes is safe because interactions are point events that can't be undone; no timestamps is safe + because window membership carries almost all the signal — both break on a workload with revocable + edges or timestamp-sensitive scoring); Q3 (the id table can't rehash because the hash *is* the id, + so it grows by a power-of-two chain and lookup probes each table in turn); Q4 (degree-proportional + segment selection then uniform within-segment sampling equals uniform over all edges — sampling + segments uniformly over-weights low-degree segments). Q2 and Q5 are measurements you run yourself. + +
+ ## References - Sharma, Jiang, Bommannavar, Larson, Lin. *GraphJet: Real-Time Content Recommendations at diff --git a/topics/42-recommendations-social/reading-link-prediction.md b/topics/42-recommendations-social/reading-link-prediction.md index 4c9cf01..2a74c08 100644 --- a/topics/42-recommendations-social/reading-link-prediction.md +++ b/topics/42-recommendations-social/reading-link-prediction.md @@ -18,6 +18,12 @@ are to be joined next — using nothing but the topology.** ### Step 1 — The experimental setup, and why it is honest +> **In:** nothing yet — a network snapshot and the question "who connects next?". +> **Out:** an honest evaluation design — a training interval that sees only the past, a test interval +> that hides the future, predictions restricted to **Core** (nodes with ≥ κ = 3 edges in *both* +> intervals), and top-`n` scoring of a ranked list — plus the five arXiv datasets Steps 2–5 measure +> on. §2. + Two time intervals: a training interval `[t₀, t₀′]` and a test interval `[t₁, t₁′]`. The predictor sees only the training graph. For the arXiv data these are 1994–1996 and 1997–1999. @@ -36,6 +42,11 @@ training edges, of which Core is 1,561 authors with 6,178 old edges and 5,751 ne ### Step 2 — Why "factor improvement over random" is the only sane metric +> **In:** the ranked top-`n` predictions from Step 1. +> **Out:** the one interpretable score — **factor improvement over random** — because raw precision +> is a few percent *by design*; the random baseline is 0.15–0.48% across the datasets (this crate: +> 0.314%), so every later number is a multiple of that. §4. + > As discussed in Section 1, many collaborations form (or fail to form) for reasons outside the > scope of the network; thus the raw performance of our predictors is relatively low. To more > meaningfully represent predictor quality, we use as our baseline a *random predictor* which @@ -46,11 +57,22 @@ Raw accuracy of a few percent sounds terrible and is actually excellent; only th interpretable. This topic's crate reproduces the setup exactly, with a random accuracy of **0.314%**, and `graphs::evaluate` returns the factor. +Worked example — the factor is `(fraction of the top-n predictions that are real) / (random +accuracy)`. On `astro-ph`, random is correct 0.475% of the time, and common neighbours scores +**18.0×** (Figure 3). Invert it: common neighbours' raw precision is `18.0 × 0.00475 = 0.0855`, i.e. +**8.55%** of its top-n pairs actually collaborate. An 8.55% hit rate reads as a failure and is in +fact 18× better than chance — which is exactly why the raw number is useless without its baseline. + Keep this metric in mind whenever you see a recommender quoted at "5% precision@10" with no baseline attached. ### Step 3 — Neighbourhood measures +> **In:** the factor-over-random metric from Step 2. +> **Out:** the four one-line **neighbourhood measures** — common neighbours, Jaccard, Adamic/Adar, +> preferential attachment — each a function of the two nodes' neighbour sets, and the hub-discount +> idea Adamic/Adar contributes. §3. + For a node `x`, `Γ(x)` is its neighbour set. - **Common neighbours**: `|Γ(x) ∩ Γ(y)|`. The direct implementation of "friends of friends become @@ -67,8 +89,27 @@ Adamic/Adar's hub discount is the same idea as topic 23's inverse document frequ 39's FRAUDAR column weights `1/log(d+5)`: **evidence everybody shares is worth less**. Three fields, one line of arithmetic. +Worked example — nodes `x` and `y` share two neighbours: `z₁`, a hub with `|Γ(z₁)| = 1000`, and +`z₂`, a specialist with `|Γ(z₂)| = 4`. Common neighbours scores both the same — it counts 2, one +per shared neighbour. Adamic/Adar discounts each by `1/log|Γ(z)|`: + +``` +Adamic/Adar = 1/ln(1000) + 1/ln(4) = 1/6.9078 + 1/1.3863 = 0.1448 + 0.7213 = 0.8661 + specialist z₂ contributes 0.7213, hub z₁ contributes 0.1448 + ratio = ln(1000)/ln(4) = 4.98 -> the specialist is worth ~5x the hub +``` + +The ratio is independent of the log's base, so "≈5×" holds whether you use natural log or log₁₀. Two +people who both know the same rarely-connected specialist is strong evidence they belong together; +two people who both follow the same celebrity is almost none. + ### Step 4 — Path-ensemble measures +> **In:** the neighbourhood scores from Step 3, which see only *shared direct* neighbours. +> **Out:** the **path-ensemble measures** — Katz, hitting/commute time, rooted PageRank, SimRank — +> which sum over *all* paths between the two nodes, and the popularity trap that reappears inside +> hitting time. §3. + Shortest-path distance is a weak measure — "For all of our graphs, there are well more than n pairs at shortest-path distance two, so our shortest-path predictor simply selects a random subset of these distance-two pairs." The better measures sum over *all* paths: @@ -77,9 +118,10 @@ of these distance-two pairs." The better measures sum over *all* paths: `(I − βM)^{-1} − I`. "A very small β yields predictions much like common neighbors, since paths of length three or more contribute very little." - **Hitting / commute time**: expected steps for a random walk from `x` to reach `y`. Both need - normalizing by the stationary distribution, "because `H_{x,y}` is quite small whenever `y` is a - node with a large stationary probability, regardless of the identity of `x`" — the popularity - trap again, arriving from a third direction. + normalizing by the **stationary distribution** (the long-run fraction of time an unconstrained + random walk spends at each node — large for popular hubs), "because `H_{x,y}` is quite small + whenever `y` is a node with a large stationary probability, regardless of the identity of `x`" — + the popularity trap again, arriving from a third direction. - **Rooted PageRank**: restart at `x` with probability α each step. The reset is there to stop the measure depending on "parts of the graph far away from x and y". This is Pixie's walk, and HippoRAG's, in its 2003 clothes. @@ -88,6 +130,11 @@ of these distance-two pairs." The better measures sum over *all* paths: ### Step 5 — Figure 3, and the row that matters +> **In:** every measure defined in Steps 3–4. +> **Out:** Figure 3's factor-over-random table and its three readings — no single winner, +> preferential attachment loses badly, Adamic/Adar's discount earns its line — the ordering lane 3 +> reproduces. §4, Figure 3. + Factor improvement over random: | predictor | astro-ph | cond-mat | gr-qc | hep-ph | hep-th | @@ -116,7 +163,8 @@ Three readings: networks, and by a lot on `cond-mat` (54.8 vs 41.1). Lane 3 of this crate reproduces the ordering on a synthetic collaboration graph grown with -preferential attachment and triadic closure: +preferential attachment and **triadic closure** (if `x` knows `y` and `y` knows `z`, the edge `x–z` +becomes more likely — the very mechanism common neighbours exploits): ``` predictor hits / n factor over random @@ -131,6 +179,11 @@ property of the generator — worth investigating rather than explaining away.) ### Step 6 — The meta-approaches +> **In:** any base measure from Steps 3–5, written in matrix form. +> **Out:** three techniques that *compose* with any of them — low-rank approximation, unseen +> bigrams, clustering — and the lines they draw to matrix-factorization recommenders, smoothing, and +> Pixie-style graph pruning. §3 (higher-level approaches). + Three techniques that compose with any measure above, and are the bridge to modern methods: - **Low-rank approximation.** Every measure has a matrix formulation, so replace the adjacency @@ -146,6 +199,11 @@ Three techniques that compose with any measure above, and are the bridge to mode ### Step 7 — What this does and does not license +> **In:** the whole catalogue's results from Steps 5–6. +> **Out:** the correct scope of the finding — topology carries *useful*, not *sufficient*, +> information, so a graph traversal is a legitimate first-stage *candidate generator*, not the final +> ranker. This is the architecture all three systems papers in this topic use. §4. + The honest framing, from §4: "a number of methods significantly outperform the random predictor, suggesting that there is indeed useful information contained in the network topology alone." @@ -193,13 +251,85 @@ first-stage retrieval, which is exactly the architecture all three systems use. ## Done when +Answer each before unfolding it. + - [ ] You can write all four neighbourhood measures from memory. + +
Answer + + With `Γ(x)` the neighbour set of `x` (§3): **common neighbours** `|Γ(x) ∩ Γ(y)|`; **Jaccard** + `|Γ(x) ∩ Γ(y)| / |Γ(x) ∪ Γ(y)|` (common neighbours normalized by union size); **Adamic/Adar** + `Σ_{z ∈ Γ(x) ∩ Γ(y)} 1/log|Γ(z)|` (each shared neighbour discounted by how many people it knows); + **preferential attachment** `|Γ(x)| · |Γ(y)|` (pure degree product). The first three look at what + the two nodes have *in common*; preferential attachment does not, which is why it loses. + +
+ - [ ] You can explain why factor-over-random is the metric and raw accuracy is not. + +
Answer + + Many collaborations form for reasons the graph never sees, so raw precision is a few percent even + for a good predictor — the paper uses a random predictor (correct 0.15–0.48% of the time; this + crate 0.314%) as the baseline and reports the *ratio*. Worked: on `astro-ph`, common neighbours + scores 18.0×, i.e. raw precision `18.0 × 0.00475 = 8.55%`. An 8.55% hit rate reads as failure but + is 18× chance. Only the ratio is interpretable — distrust any recommender quoted at "5% + precision@10" with no baseline attached. + +
+ - [ ] You can say why preferential attachment loses, in one sentence connecting it to lane 1. + +
Answer + + Preferential attachment `|Γ(x)|·|Γ(y)|` is the only measure that never checks whether `x` and `y` + have anything in common — it just multiplies their degrees, so it ranks pairs of *famous* nodes + highly — which is lane 1's popularity baseline in a link-prediction costume, and it fails for the + same reason: knowing who is popular is not knowing who will connect (Figure 3: 4.7–15.2× versus + common neighbours' 18.0–47.2×; lane 3: 1.9× versus 20.7×). + +
+ - [ ] You can explain Adamic/Adar's discount and name its two cousins in other topics. + +
Answer + + Adamic/Adar weights each shared neighbour `z` by `1/log|Γ(z)|`, so a rarely-connected specialist + counts far more than a hub — worked in Step 3, a degree-4 specialist is worth ~5× a degree-1000 + hub (`ln 1000 / ln 4 ≈ 4.98`). Its cousins: topic 23's **inverse document frequency** (rare terms + weigh more) and topic 39's FRAUDAR column weight `1/log(d+5)` (edges to high-degree nodes count + less). The shared statement: evidence that many nodes share is worth less than evidence that few + share. + +
+ - [ ] Your `linkpred.rs` reproduces lane 3's ordering with PA far behind the rest. + +
Answer + + Lane 3's reference ordering on the synthetic graph: preferential attachment 1.9× (6/985 hits), + common neighbours 20.7× (64/985), Jaccard 25.6× (79/985), Adamic/Adar 22.3× (69/985) — + preferential attachment an order of magnitude behind the neighbourhood measures, matching Figure + 3's shape. (Jaccard edging Adamic/Adar here, the reverse of the paper's usual order, is a property + of the generator — investigate it rather than explain it away.) If your PA lands near the others, + your candidate enumeration is probably leaking degree information. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions live in `notes.md`'s guide-question checklist. The load-bearing ones: Q1 + (preferential attachment correctly predicts *how many* edges a node gains, not *which* pairs + connect — right marginal, wrong joint); Q2 (Adamic/Adar, IDF and `1/log(d+5)` all instantiate + "down-weight evidence shared by high-degree/high-frequency entities"; discounting hubs is wrong + when the hub *is* the signal, e.g. a shared rare disease gene); Q3 (hitting time's popularity bias + is lane 1's popularity trap, addressed by Pixie's biasing innovation). Q4 and Q5 are experiments + and arguments you write yourself. + +
+ ## References - Liben-Nowell & Kleinberg. *The Link-Prediction Problem for Social Networks.* CIKM 2003 / JASIST diff --git a/topics/42-recommendations-social/reading-pixie.md b/topics/42-recommendations-social/reading-pixie.md index 421331b..fc2ede1 100644 --- a/topics/42-recommendations-social/reading-pixie.md +++ b/topics/42-recommendations-social/reading-pixie.md @@ -6,7 +6,15 @@ states almost in passing: **a random walk's cost depends on the number of steps, of the graph**. Everything else in Pixie — the biased edge selection, the weighted query set, the multi-hit booster, the early stopping — is engineering on top of that one fact. It is worth reading as a case study in choosing an algorithm whose cost model matches your latency budget, -and then spending all your cleverness inside it. +and then spending all your cleverness inside it. It is *also* the topic's worked example of a +published trick that does not reproduce: the multi-hit booster buys nothing on this repo's +generator, and the honest half of the guide is explaining exactly which premise is missing. + +Every section, figure and equation number below is from the published Pixie paper — Eksombatchai +et al., *Pixie*, WWW 2018 ([arXiv:1711.07601](https://arxiv.org/abs/1711.07601)). The measured +numbers labelled "lane 1" and "lane 2" are this repo's own, from +[`../../FINDINGS.md`](../../FINDINGS.md) row 42, this topic's [`README.md`](README.md) and +[`notes.md`](notes.md); they were produced by the crate in `experiments/`, not by the paper. ## The problem in one sentence @@ -18,244 +26,567 @@ and reacting to intent in real time is worth 30–50% more engagement.** ### Step 1 — Why real-time, and why not precompute -The industry default was batch: run a pipeline nightly, write recommendations to a key-value -store, serve them. Pixie's argument against it is economic, not aesthetic: "if providing -recommendations would take just 1 second, then the user would have to wait too long ... In such -cases recommendations would have to be precomputed (say once a day) and then served out of a -key-value store. However, old recommendations are stale and not engaging." And the measured -consequence: "reacting to user's intent in real-time leads to 30-50% higher engagement than -needing to wait days or hours for recommendations to refresh." +> **In:** nothing yet — this step fixes the budget and the reason the whole system exists. +> **Out:** two numbers every later step obeys — a **60 ms** per-request latency budget and the +> **30–50%** engagement lift that pays for hitting it in real time rather than from a cache. + +A **recommender** here answers one query: given a *query pin* — an item the user just engaged with +— return a ranked list of other items they are likely to want. The industry default was **batch +precomputation**: run a pipeline nightly, write each user's recommendations to a key–value store, +and serve them as static lookups. Pixie's argument against batch is economic, not aesthetic (§1): + +> if providing recommendations would take just 1 second, then the user would have to wait too +> long ... In such cases recommendations would have to be precomputed (say once a day) and then +> served out of a key-value store. However, old recommendations are stale and not engaging. + +And the measured consequence, also §1: "reacting to user's intent in real-time leads to 30-50% +higher engagement than needing to wait days or hours for recommendations to refresh." + +That 30–50% is the entire justification for the system: it is what buys back the cost of computing +a fresh answer on every request. Hold on to it — everything after this is what you must build once +you have decided 60 ms is the budget and that a day-old cache will not do. -That number is the entire justification for the system. Hold on to it — the rest of the paper is -what you have to build once you have decided 60 ms is the budget. +Why it matters: a recommender's design is downstream of its latency budget. Pixie picks an +algorithm whose cost it can bound *before* choosing any of the four refinements below. ### Step 2 — The graph, and Algorithm 1 -Pinterest is a bipartite graph `G = (P, B, E)`: pins `P`, boards `B`, an edge when a user saved -a pin to a board. Recommending from a query pin `q` is a random walk that alternates sides: +> **In:** the 60 ms budget from Step 1. +> **Out:** `BasicRandomWalk` (Algorithm 1) and its one load-bearing property — cost is `O(N)` +> steps, independent of graph size — plus the measured way it fails (lane 1). Steps 3–6 each fix +> one of its failures; Step 8 makes its inner loop free. + +Pinterest is a **bipartite graph** `G = (P, B, E)` — a graph whose vertices fall into two sets with +edges only *between* the sets, never within one. Here the two sets are pins `P` and boards `B`, +and an edge `(p, b) ∈ E` means a user saved pin `p` to board `b`. A **random walk** from a query +pin `q` is a process that starts at `q` and repeatedly steps to a randomly chosen neighbour; +because the graph is bipartite each pair of steps goes pin → board → pin. A **visit count** `V[p]` +is how many times the walk landed on pin `p`; the pins with the highest visit counts are the +recommendation. + +Algorithm 1 is eleven lines, transcribed here exactly as the paper prints it: ``` - BasicRandomWalk(q, E, α, N): - totSteps = 0; V = 0 - repeat - currPin = q - currSteps = SampleWalkLength(α) - for i in 1..currSteps: - currBoard = E(currPin)[rand()] # pin -> board - currPin = E(currBoard)[rand()] # board -> pin - V[currPin]++ - totSteps += currSteps - until totSteps ≥ N - return V +Algorithm 1 — Basic Random Walk (Pixie §3), transcribed verbatim +BasicRandomWalk(q: Query pin, E: Set of edges, α: Real, N: Int) + 1: totSteps = 0, V = 0 + 2: repeat + 3: currPin = q + 4: currSteps = SampleWalkLength(α) # α sets the (random) length of one walk + 5: for i = [1 : currSteps] do + 6: currBoard = E(currPin)[rand()] # pin -> board, uniform neighbour + 7: currPin = E(currBoard)[randNeighbor()] # board -> pin, uniform neighbour + 8: V[currPin]++ + 9: totSteps += currSteps +10: until totSteps >= N +11: return V ``` -Twenty lines, and it is already a personalized recommender: the top visit counts are the pins -most reachable from `q`. Note there is no matrix, no factorization, no model. Note also that `N` -— the step budget — is the only thing that determines runtime. +`α` is the **restart probability's cousin**: `SampleWalkLength(α)` draws how many steps one walk +runs before it snaps back to `q`, so a smaller expected length keeps visits close to the query. +There is no matrix, no factorization, no learned model. And the only thing that determines runtime +is `N`, the total step budget on line 10 — "the time taken by this procedure is constant and +independent of graph size (determined by parameter N)" (§3). That sentence is why 60 ms is +reachable from 17 billion edges: you pay for steps, not for the graph. + +Now the failure, measured. Lane 1 of this topic's crate runs exactly this walk +(`graphs::basic_random_walk`) alongside a plain popularity recommender, and reports them side by +side: + +``` +lane 1 (provided) — 3000 users x 6000 items, 30 communities, 60000 training edges + recommender hit-rate@50 personalization overlap w/ bestsellers + popularity 0.340 0.155 0.923 + basic walk 0.403 0.820 0.451 +``` -Lane 1 of this topic's crate runs exactly this and measures what is wrong with it: a hit rate of -0.403, but **45% of every returned list is the global bestseller list**, because an unbiased -walk's visit distribution drifts toward degree. +The `popularity` row is this topic's headline: recommending the global bestseller list to everyone +scores **0.340 hit-rate@50** with **0.923 overlap** with the bestseller list. That is the result +[`FINDINGS.md`](../../FINDINGS.md) row 42 headlines as "35.3% hit-rate@50 with 92.2% overlap" — the +crate lane here reports 0.340 / 0.923, agreeing to within about a percentage point. Popularity is +not a weak baseline; the walk has to *beat* it. + +Two definitions are needed to read that block, and both are computed by `experiments/src/graphs.rs`: + +- **hit-rate@k** (`hit_rate`, k = 50): the fraction of users for whom *at least one* held-out item + appears in the top-k list. For each user you hold out some of their real future engagements, + build their top-50, and score 1 if any held-out item is in it, 0 otherwise; the metric is the + mean of those over all users. Worked micro-example, k = 3, five users with one held-out item each: + if the held-out item lands in the top-3 for two of the five, hit-rate@3 = 2/5 = 0.40. +- **overlap w/ bestsellers** (`popularity_overlap`): the mean over users of + `|list ∩ bestsellers| / |list|`, where `bestsellers` is the global top-k by degree. 1.0 means + "you rebuilt the bestseller list"; 0.0 means "nothing you returned was globally popular". Worked + micro-example, k = 5, bestsellers `{A,B,C,D,E}`: a user who owns none of them gets the list + `{A,B,C,D,E}` → overlap 5/5 = 1.0; a user who already owns `E` has it filtered out and gets + `{A,B,C,D,F}` (F the 6th bestseller) → overlap 4/5 = 0.80. Average a population where most users + own zero or one top item and you land near 0.92 — which is why popularity's overlap is 0.923, not + a flat 1.0. + +So the basic walk gets 40.3% of users right — but **45.1% of every list it returns is the global +bestseller list**, because an unbiased walk's long-run visit distribution drifts toward high +degree (a hub is reachable from everywhere). It personalizes (0.820) and still leaks popularity. +Steps 3–6 are the four fixes; Step 3 attacks exactly this drift. + +Why it matters: a walk this simple is already a personalized recommender, and its cost is a knob +you set. Everything else is buying back quality the plain walk gives away. ### Step 3 — Biasing the walk (innovation 1) -The fix for personalization-beyond-the-query is to make edge selection depend on the *user*, not -just the graph. `PersonalizedNeighbor(E, U)` prefers edges matching the user's features — -language, topic — so the same query set gives different results to different people. The paper -is careful about why this is cheap: "one could think of this method as using a different graph -for each user where edge weights are tailored to that user (but without the need to store a -different graph for each of the 200+ million users)." In practice the weights take values from a -small discrete set and edges for similar languages are stored consecutively in memory, so -`PersonalizedNeighbor` is a *subrange operator* — a slice, not a filter. +> **In:** Algorithm 1's uniform neighbour choice from Step 2 (line 6–7's `rand()`), which drifts +> toward degree. +> **Out:** `PersonalizedNeighbor(E, U)` — the same walk with a *user-biased* edge choice — and the +> paper's sharpest measurement, Table 3. This is the one innovation the crate leaves to you +> (exercise 2). + +The fix for "the walk returns the same popular pins to everyone" is to make edge selection depend +on the *user*, not just the graph. **`PersonalizedNeighbor(E, U)`** takes the walking pin's edges +`E` and the user's feature vector `U` (language, topic) and prefers edges that match `U`, so the +same query gives different results to different people. The paper is careful about why this is +cheap (§3.1, innovation 1): + +> one could think of this method as using a different graph for each user where edge weights are +> tailored to that user (but without the need to store a different graph for each of the 200+ +> million users). + +The implementation trick that keeps it O(1): "we currently limit the weights to only take values +from a discrete set of possible values ... by storing edges for similar languages and topics +consecutively in memory ... `PersonalizedNeighbor(E, U)` is a **subrange operator**" (§3.1). A +subrange operator is a slice, not a filter: matching edges are already adjacent in the array, so +you sample from a contiguous window `[lo, hi)` of the adjacency list instead of scanning it. -Table 3 is the sharpest measurement in the paper. Percentage of results in the target language, -basic walk vs Pixie: +Table 3 is the payoff — the percentage of returned pins actually in the user's target language, +basic walk versus Pixie: ``` - En→Japanese Japanese→Japanese +Table 3 (Pixie §4.2) — % of results in the target language + En->Japanese Japanese->Japanese BasicRandomWalk 16.35% 52.95% PixieRandomWalk 80.33% 100.00% - En→Slovak Slovak→Slovak + En->Slovak Slovak->Slovak BasicRandomWalk 2.13% 16.06% PixieRandomWalk 42.55% 100.00% ``` -Slovak goes from 2.13% to 42.55%. For a small-language user the basic walk was returning -essentially nothing usable. +Read the Slovak row: an English-speaking user querying with a Slovak interest went from **2.13% to +42.55%** target-language content — the basic walk was returning essentially nothing usable for a +small-language interest, because the walk drowned in the majority language's high-degree pins. The +crate does not implement biasing; exercise 2 asks you to add a language attribute, make neighbour +selection prefer matching edges, reproduce the *shape* of Table 3, and measure the per-step cost. + +Why it matters: this is degree-drift's direct cure, and the paper's largest single measured +effect. If you build only one of the four innovations, build this one. ### Step 4 — Multiple query pins, and the step-allocation problem (innovation 2) -A user is not one pin. Pixie takes a weighted query set `Q = {(q, w_q)}` built from recent -interactions, weighted by recency and interaction type, and runs a separate walk per query pin. +> **In:** the single-query walk of Steps 2–3. +> **Out:** a weighted query *set* `Q = {(q, w_q)}`, and Equations 1–2 — a sub-linear rule that +> hands every query pin a step budget `N_q`. Each pin's walk produces its own counter `V_q`, which +> Step 5 combines. + +A user is not one pin. Pixie takes a **weighted query set** `Q = {(q, w_q)}` — recent interactions +`q`, each with a weight `w_q` set by recency and interaction type — and runs a separate walk per +query pin. That raises a budgeting question with a non-obvious answer. A high-degree query pin +needs *more* steps, because its walk diffuses across many neighbours before its top stabilises. But +allocate steps *linearly* in degree and a low-degree pin gets **less than one step** — its whole +interest is silently dropped (§3.1): + +> the challenge remains that if we assign the number of steps in linear proportion to the degree +> then we can end up allocating not even a single step to pins with low degrees. + +Equation 1 builds a scaling factor that grows **sub-linearly** in degree, and Equation 2 turns the +scaling factors into a normalized step budget: -Which raises a budgeting question with a non-obvious answer. High-degree query pins need *more* -steps, because their walks diffuse across many neighbours. But allocate steps linearly in degree -and a low-degree pin gets **less than one step** — its interest is silently dropped. The paper: -"the challenge remains that if we assign the number of steps in linear proportion to the degree -then we can end up allocating not even a single step to pins with low degrees." +``` +Eq. 1: s_q = |E(q)| · (C - log|E(q)|) +Eq. 2: N_q = w_q · s_q / Σ_{r∈Q} s_r · N +``` -Equation 1's scaling factor grows sub-linearly: +Naming the symbols: `|E(q)|` is the degree of query pin `q`; `w_q` is its weight in the set; +`Σ_{r∈Q} s_r` normalizes across the query set; `N` is the total step budget; and `C` is a +graph-wide maximum. **A subtlety the paper states loosely and the crate pins exactly.** Pixie's §3.1 +prints "`C = max_{p∈P} |E(p)|` ... the maximum pin degree", but taken literally that makes +`C − log|E(q)| ≈ C` for every pin, so `s_q ≈ |E(q)|·C` is *linear* in degree — the exact behaviour +Equation 1 exists to avoid, and it contradicts the paper's own next sentence ("does not give +disproportionately high weights to popular pins"). For the sub-linearity to exist, `C` must be on +the log scale, and that is what this repo implements: `C = max_{p∈P} log|E(p)| = log(max pin +degree)`, taken over **all** pins in the graph, not the query set (`experiments/src/pixie.rs`, +`allocate_steps`: "`C = ln(max item degree in the WHOLE graph)`"). + +Why "the whole graph" matters: if you computed `C` over only the query set `Q`, then the +highest-degree *query* pin has `log|E(q)| = C`, so `s_q = |E(q)|·(C − C) = 0` — it gets zero steps. +The crate's `every_query_pin_gets_at_least_one_step` test catches exactly that, and also asserts +the step ratio is strictly below the degree ratio (the sub-linearity). + +Worked example — two query pins, degrees `|E(q₁)| = 1` and `|E(q₂)| = 10,000`, equal weights +`w = 1`, budget `N = 10,000`, and a graph whose maximum pin degree is `100,000` so +`C = ln(100,000) = 11.5129`: ``` - s_q = |E(q)| · (C − log|E(q)|) C = max_p log|E(p)| (over ALL pins) - N_q = w_q · s_q / Σ_{r∈Q} s_r +linear allocation N_q ∝ |E(q)|: + N_1 = 10,000 · 1/(1+10,000) = 0.9999 -> floor to 0 steps (interest dropped) + +Equation 1 (sub-linear): + s_1 = 1 · (11.5129 - ln 1) = 1 · 11.5129 = 11.5129 + s_2 = 10,000· (11.5129 - ln 10,000)= 10,000 · 2.30259 = 23,025.9 + Σ s = 23,037.4 + N_1 = 10,000 · 11.5129 / 23,037.4 = 5.00 steps (interest kept) + N_2 = 10,000 · 23,025.9 / 23,037.4 = 9,995 steps + + step ratio N_2/N_1 = 1,999 vs degree ratio = 10,000 -> strictly sub-linear ``` -Note `C` is the maximum over the *whole graph*, not over the query set. Get that wrong and the -highest-degree query pin gets `s_q = 0`; the crate's `every_query_pin_gets_at_least_one_step` -test catches it, and the test also asserts the step ratio is strictly below the degree ratio, -which is the sub-linearity. +The low-degree pin gets ~5 steps under Equation 1 and 0 under linear allocation; the high-degree +pin still gets the lion's share, but the ratio (1,999) is a fifth of the degree ratio (10,000). + +Why it matters: this is the least-glamorous innovation and the biggest measured win. Lane 2 moves +hit-rate@50 from the basic walk's 0.403 to **0.823** on eight query pins with this allocation — +the largest single jump in the topic ([notes.md](notes.md)). ### Step 5 — The multi-hit booster (innovation 3) -Equation 3: +> **In:** the per-query visit counters `V_q[p]` produced by Step 4's walks. +> **Out:** one combined score `V[p]` per pin, via Equation 3 (implemented as line 5 of Algorithm +> 3) — and the topic's headline negative result, which you must not overstate. + +With one counter `V_q` per query pin, you still have to combine them into a single ranking. +Summing them is the obvious choice. Pixie instead uses Equation 3, which the paper implements as +line 5 of **Algorithm 3** (`PixieRandomWalkMultiple`): ``` +Algorithm 3 line 5 = Eq. 3 (Pixie §3.1), transcribed verbatim V[p] = ( Σ_{q ∈ Q} sqrt( V_q[p] ) )² ``` -A pin visited 4 times from one query pin scores 4. A pin visited twice from each of two query -pins scores `(√2+√2)² = 8`. Same total visits, twice the score. The intuition: "candidates with -high visit counts from multiple query pins are more relevant to the query than for example -candidates having equally high total visit count but all coming from a single query pin." +Naming the symbols: `V_q[p]` is how many times the walk *from query pin q* visited pin `p`; the +inner sum is over the query set; and `V[p]` is the combined score. Worked on the paper's own +intuition: + +``` +single source: p visited 4 times from one query pin -> ( sqrt(4) )² = 4 +multi source: p visited twice from each of two query pins -> ( sqrt(2)+sqrt(2) )² = 8 +``` + +Same **four total visits**, twice the score, when they came from two interests instead of one. And +a single-source pin is unchanged — `(√4)² = 4` — so Equation 3 is strictly a *bonus* for +cross-interest pins, never a re-weighting. The paper's justification (§3.1, innovation 3): + +> candidates with high visit counts from multiple query pins are more relevant to the query than +> for example candidates having equally high total visit count but all coming from a single query +> pin. + +**And now the honest part — this is the topic's worked example of "report the negative result".** +Lane 2 measures the booster against plain summed visit counts on the crate's synthetic graph, at +one interest per user *and* at three, and finds **no gain either way** ([README.md](README.md), +[notes.md](notes.md)): -Note the boost leaves single-source scores unchanged — `(√4)² = 4` — so it is strictly a bonus, -not a re-weighting. +``` +lane 2 — multi-hit booster ablation + 1 interest/user : 0.823 unboosted vs 0.803 boosted + 3 interests/user: 0.563 unboosted vs 0.547 boosted +``` -**And now the honest part.** Lane 2 of this crate measures the booster against plain summed visit -counts on a synthetic graph, at one interest per user and at three, and finds **no gain either -time** — 0.823 unboosted against 0.803 boosted, 0.563 against 0.547. The arithmetic is right (the -unit test pins it); the *premise* is missing. Equation 3 is a bet that a pin sitting at the -intersection of several of your interests is more engaging than one deep inside a single -interest. That is a claim about people, and a generator that draws its held-out item from the -same distribution as its training items does not contain it. +Summing raw visit counts scores *slightly better* than Equation 3 in both regimes. This is **not** +an implementation bug: the crate's unit test pins the arithmetic exactly — `(√2+√2)² = 8` against a +single-source `4` — so the formula is right. What is missing is the *premise*. Equation 3 is a bet +that a pin sitting at the intersection of several of your interests is more engaging than one deep +inside a single interest — a claim about **people**, not about graphs. This generator draws each +user's held-out item from the same distribution as its training items, so being reachable from +several query pins carries no extra information about which item the user actually takes next; the +boost is a no-op at best. -This is the most useful thing in the topic. A published trick encodes a domain assumption. Before -you ship it, measure whether your data has the assumption in it — exercise 4 asks you to build a -graph where it does, and to find how strong the effect must be before the boost pays. +That is the transferable lesson, and the reason this topic exists: **a published trick encodes a +domain assumption, and you owe it a measurement on your own data before you ship it.** Exercise 4 +asks you to add a `cross_interest_bias` that makes held-out items favour the overlap of a user's +interests, sweep it, and find the crossover where the boost finally pays. + +Why it matters: the elegant idea is the one that fails here, and diagnosing *why* — a missing +premise, not a bug — is a skill worth more than the booster itself. ### Step 6 — Early stopping (innovation 4) -The walks run for a fixed `N_q`. But you do not need convergence, you need a *stable top*. Pixie -terminates once at least `n_p` candidate pins have each been visited at least `n_v` times — -monitored with a single counter, incremented when a pin's count crosses `n_v` exactly, so the -check is O(1) per step: +> **In:** the fixed per-query budgets `N_q` from Step 4 — walks that run to completion. +> **Out:** Algorithm 2's `n_p`/`n_v` termination condition, which stops each walk once its top is +> stable, and the measured speedup (lane 2 and §4.2). + +The walks run for a fixed `N_q` steps. But you do not need the walk to *converge*; you need its +**top** — the highest-visited pins — to stop moving. Pixie terminates a walk once at least `n_p` +candidate pins have each been visited at least `n_v` times. Two integers name the rule: + +- **`n_v`** — the visit threshold a pin must reach to be "high-visited". +- **`n_p`** — how many pins must cross `n_v` before the walk stops. + +Monitoring this could cost more than the walk, so Pixie keeps a single counter, incremented the one +step a pin's count crosses `n_v` *exactly* — Algorithm 2, transcribed: ``` - Algorithm 2, lines 10–13: - V[currPin]++ - if V[currPin] == n_v: nHighVisited++ - until totSteps ≥ N or nHighVisited > n_p +Algorithm 2 — Pixie Random Walk with early stopping (Pixie §3.1), transcribed verbatim + 9: V[currPin]++ +10: if V[currPin] == n_v then +11: nHighVisited++ +12: totSteps += currSteps +13: until totSteps >= N or nHighVisited > n_p +14: return V ``` -The counter is *per walk* — Algorithm 2 is invoked once per query pin, and each pin decides for -itself. (Sharing one counter across the query set starves the later pins; the crate's -implementation note says so, because it is an easy mistake.) +Line 10's `== n_v` (not `>= n_v`) is what makes the counter O(1): a pin bumps `nHighVisited` on the +single step it *reaches* the threshold, never again, so no per-step scan of the candidate set is +needed. The counter is **per walk** — Algorithm 2 is invoked once per query pin (Algorithm 3, line +3), and each pin decides for itself when to stop. Sharing one counter across the query set would +let early, high-degree pins trip it and starve the later ones; the crate's implementation note +(`experiments/src/pixie.rs`) says so, because it is an easy mistake. -The paper's measurement: at `n_p = 2000, n_v = 4`, results overlap the gold-standard long walk by -**84%** at **one third** of the runtime; at `n_v = 6` the runtime halves. Lane 2 reproduces the -shape almost exactly at `n_p = 100, n_v = 3` on a smaller graph: **35% of the steps, 2.2× faster, -0.793 top-50 overlap, hit rate unchanged**. +The paper's measurement (§4.2): at `n_p = 2000, n_v = 4`, results overlap the gold-standard long +walk by **84%** at **one third** of the runtime; raising to `n_v = 6` halves the runtime again. +Lane 2 reproduces the shape almost exactly at `n_p = 100, n_v = 3` on the smaller crate graph: + +``` +lane 2 — early stopping + full walk: 9,000,004 steps, 2.12 ms/query + early stop: 3,170,675 steps (35% of full), 0.97 ms/query (2.2x faster) + early-stopped top-50 overlaps the full walk by 0.793, hit rate unchanged +``` + +35% of the steps, 2.2× faster, top-50 overlap 0.79, hit-rate@50 unchanged — the same trade the +paper reports at a third of the runtime for 84% overlap. + +Why it matters: unlike Step 5's booster, early stopping *does* reproduce here, and it is free +quality-for-speed — the one refinement you can adopt without checking a domain premise first. ### Step 7 — Pruning improves quality *and* shrinks the graph -The original Pinterest graph is 7 billion nodes and over 100 billion edges. Pixie prunes it two -ways: boards whose LDA topic distribution has high entropy (diverse boards "diffuse the walk in -too many directions") are removed entirely, and for high-degree pins, edges to boards whose topic -vector has low cosine similarity to the pin's are discarded, controlled by a pruning factor δ. +> **In:** the raw Pinterest graph — 7 billion nodes, over 100 billion edges (§3.2). +> **Out:** a pruned graph of 1 B boards, 2 B pins, 17 B edges in ~120 GB that recommends *better*, +> and the single most instinct-changing number in the paper. -The result is the one that should change your instincts about data cleaning: +Pixie prunes the graph two ways before serving from it (§3.2). **Board entropy**: a board's topic +mix is scored by the entropy of its **LDA** topic distribution — *Latent Dirichlet Allocation*, a +model that assigns each board a distribution over latent topics — and boards with high entropy +("diverse boards diffuse the walk in too many directions") are removed entirely. **Edge cosine +similarity**: for high-degree pins, edges to boards whose topic vector has low **cosine +similarity** (the cosine of the angle between two vectors — 1.0 when identical, 0 when orthogonal) +to the pin's are discarded, controlled by a **pruning factor** `δ`. -> when δ = 0.91, the F1 score peaks at 58% above the unpruned graph F1 and the graph contains -> only 20% the original number of edges. +The result should change your instincts about data cleaning (§4.3): -A graph a fifth the size that recommends 58% better — and, because it now fits, one that does not -have to be distributed at all. The pruned graph is 1B boards, 2B pins, 17B edges, about **120 GB** -on an r3.8xlarge with 244 GB of RAM. +> when δ = 0.91, the F1 score peaks at 58% above the unpruned graph F1 and the graph contains only +> 20% the original number of edges. + +**F1** is the harmonic mean of precision and recall — the standard single-number quality score, so +"58% above" is a real quality gain, not a size trade. A graph a fifth the size that recommends 58% +*better* — and, because it is now 1 B boards, 2 B pins, 17 B edges in about **120 GB**, one that +fits on a single AWS r3.8xlarge (244 GB RAM) and never has to be distributed at all (§1). + +Why it matters: the reflex is that throwing data away costs quality. Here it *buys* quality and a +cheaper machine at once, because the discarded edges were noise the walk would otherwise diffuse +into. Exercise 3 in [README.md](README.md) has you find the analogous `δ` for a workload you know. ### Step 8 — The two data structures that make the inner loop free -Lines 6–13 of Algorithm 2 are the whole runtime, so two structures get hand-built: +> **In:** Algorithm 2's inner loop (Step 6) — a neighbour sample and a visit-count increment, run +> billions of times. +> **Out:** two hand-built structures that make each of those O(1), plus the OS trick that removes +> the page-table tax. This is the section a systems engineer reads twice (§3.3). -**`edgeVec`** — every adjacency list concatenated into one contiguous array, with an offset per -node, allocated from an object pool. Sampling a neighbour of node `i` is then: +Lines 6–11 of Algorithm 2 are the entire runtime, so two structures are hand-built. + +**`edgeVec`** — every adjacency list concatenated into one contiguous array, with an `offset` per +node, allocated once from an object pool (so no per-node allocation and no fragmentation). Sampling +a neighbour of node `i` is Equation 4: ``` - F[ offset_i + (rand() % (offset_{i+1} − offset_i)) ] +Eq. 4 (Pixie §3.3): F[ offset_i + ( rand() % (offset_{i+1} - offset_i) ) ] ``` -One multiply, one modulo, one load. No per-node allocation, no fragmentation, no pointer chasing. +`offset_{i+1} − offset_i` is node `i`'s degree; one multiply-free modulo picks a slot; one load +returns the neighbour. "The accesses on lines 5, 8, and 10 of Algorithm 2 can be performed +efficiently in constant time" (§3.3). + +**The visit counter** — an open-addressing hash table. **Open addressing** stores entries directly +in one array and, on a collision, probes nearby slots rather than chasing a linked list; Pixie uses +**linear probing** (try the next slot, then the next) "to maintain good cache locality", and a +**multiplicative hash** (multiply the key by a fixed prime, modulo the array size) because it must +be fast (§3.3). The table is sized to `N` up front and never resizes, because "the number of steps +N provides an upper bound on the number of keys" — a walk of `N` steps can visit at most `N` +distinct pins. -**The visit counter** — an open-addressing hash table with linear probing and a multiplicative -hash, sized to `N` up front because "the number of pins with non-zero visit counts can never -exceed the number of steps", so it never resizes. Linear probing is chosen explicitly for cache -locality. +And an operational detail worth stealing (§3.3): Pixie uses **Linux HugePages** to raise the page +size from 4 KB to 2 MB, "thus decreasing the number of page table entries needed by a factor 512. +Too many page table entries is especially problematic on virtual machines; the HugePages option +enabled Pixie on virtual machines to serve twice as many requests at half the runtime." A page +table is the map from virtual to physical addresses; 512× fewer entries means 512× fewer TLB +misses walking a graph that is almost entirely random access. -And an operational detail worth stealing: Pixie uses **Linux HugePages** to raise the page size -from 4 KB to 2 MB, "thus decreasing the number of page table entries needed by a factor 512. Too -many page table entries is especially problematic on virtual machines; the HugePages option -enabled Pixie on virtual machines to serve twice as many requests at half the runtime." +Why it matters: the O(N) cost model from Step 2 is only *achievable* if each of the N steps is +genuinely O(1). These three structures are what make the constant factor small enough for 60 ms. ## How to read the paper (with the concepts in hand) -- **§1.** The scale claims and the 30–50% engagement number. This is the budget the rest obeys. +- **§1.** The scale claims (17 B edges, 120 GB, r3.8xlarge, p99 < 60 ms, ~1,200 req/s per server, + ~100,000 cluster-wide) and the 30–50% engagement number. This is the budget the rest obeys. - **§2 Related work.** One pass. The paragraph on Twitter's WTF is the connection to the GraphJet guide; the paragraph on collaborative filtering explains why factorization was rejected (complexity linear in nodes, and Pinterest has billions). -- **§3 + Algorithm 1.** Read the twenty lines and convince yourself the cost is `N`, full stop. -- **§3.1 innovation (1).** Biasing, and the "different graph per user without storing one" trick. +- **§3 + Algorithm 1.** Read the eleven lines and convince yourself the cost is `N`, full stop. +- **§3.1 innovation (1).** Biasing, `PersonalizedNeighbor`, and the "different graph per user + without storing one" subrange trick. - **§3.1 innovation (2) + Equations 1–2.** The step-allocation problem. Derive for yourself why - linear allocation starves low-degree pins, then check `C` is the graph-wide maximum. -- **§3.1 innovation (3) + Equation 3.** The booster. Then read Step 5 above and hold the claim - loosely. -- **§3.1 innovation (4) + Algorithm 2 lines 10–13.** Early stopping, and note the counter is per - walk. + linear allocation starves low-degree pins (Step 4's worked example), then confirm `C` must be the + graph-wide maximum on the *log* scale. +- **§3.1 innovation (3) + Equation 3 / Algorithm 3 line 5.** The booster. Then read Step 5 above and + hold the claim loosely — it does not reproduce on this generator. +- **§3.1 innovation (4) + Algorithm 2 lines 9–13.** Early stopping; note the counter is per walk and + the `== n_v` test is what keeps it O(1). - **§3.2 Graph pruning.** The 58%-better-at-20%-of-the-edges result. -- **§3.3 Implementation.** `edgeVec`, the visit counter, HugePages, the once-a-day graph build. - This is the section a systems engineer should read twice. +- **§3.3 Implementation.** `edgeVec` (Eq. 4), the open-addressing visit counter, HugePages, the + once-a-day graph build. Read twice. - **§4.1 + Tables 1–2.** Hit rate against content-based baselines (6.3% / 23.1% / 52.2% at top - 10/100/1000) and the A/B lifts (homefeed +48%, localization +48–75%). + 10/100/1000, against content-combined 2.1 / 4.6 / 10.5%) and the A/B lifts (homefeed +48%, + related pins +13%, localization +48–75%). - **§4.2 + Figures 1–3 + Table 3.** Runtime linear in steps; stability against step count; the - language-biasing table; the early-stopping parameter sweeps. + language-biasing table; the early-stopping sweeps. - **After the paper.** Implement `pixie.rs` and reproduce lane 2. Then do exercise 2 — biasing — - which is the one innovation the crate leaves to you, and the one with the biggest measured - effect in the paper. + the one innovation the crate leaves to you, and the one with the biggest measured effect. ## Questions to answer in notes.md -1. Pixie's cost is `O(N)` steps, independent of graph size. Name the two things that *are* - affected by graph size, and say what each costs at Pinterest's scale (hint: §3.3 and §4.2's - cache-miss remark). +1. Pixie's cost is `O(N)` steps, independent of graph size. Name the two things that *are* affected + by graph size, and say what each costs at Pinterest's scale (hint: §3.3's `edgeVec` build and + HugePages, and §4.2's cache-miss remark). 2. Derive the failure of linear step allocation: with query pins of degree 1 and 10,000 and a budget of 10,000 steps, how many steps does the low-degree pin get under `N_q ∝ |E(q)|`, and - under Equation 1? Then explain why `C` must be the graph-wide maximum. + under Equation 1? Then explain why `C` must be the graph-wide maximum on the log scale. 3. Lane 2 finds the multi-hit booster gives no gain. Write down the precise property a data set must have for Equation 3 to pay, as a statement about the joint distribution of (query pins, held-out item). Then say how you would test for it in one query, before implementing anything. 4. Pruning improves F1 by 58% while removing 80% of edges. That is a statement about the *graph*, not the algorithm. What is the equivalent move for a workload you know, and what would you measure to find your δ? -5. Early stopping monitors `n_p` pins reaching `n_v` visits. Why is that a good proxy for "the - top of the ranking has stopped moving", and construct a graph where it is a bad one. +5. Early stopping monitors `n_p` pins reaching `n_v` visits. Why is that a good proxy for "the top + of the ranking has stopped moving", and construct a graph where it is a bad one. + +## Takeaway + +Pixie is one algorithm — a random walk whose cost is `O(N)` steps, not `O(graph)` — with four +refinements bolted on: bias the edge choice, split the budget sub-linearly across a weighted query +set, boost cross-interest pins, and stop each walk when its top is stable. Three of the four +reproduce on this repo's generator; the fourth, the multi-hit booster, does not, because the +generator lacks the behavioural premise Equation 3 bets on. That gap is the lesson: **measure a +borrowed trick's assumption on your own data before shipping it.** ## Done when +Answer each before unfolding it. + - [ ] You can state the cost model in one sentence and explain why it makes 60 ms possible. + +
Answer + + A random walk of `N` steps from the query pins costs `O(N)`, and `N` is a parameter set on line + 10 of Algorithm 1 — "the time taken by this procedure is constant and independent of graph size" + (§3). It makes 60 ms possible because you pay for steps, not for the 17 billion edges: the graph + can grow arbitrarily and the per-request cost does not move, as long as each step stays O(1). + Step 8's `edgeVec` (Eq. 4) and the open-addressing visit counter are what keep each step O(1), so + the constant factor is small enough that ~3 M steps fit in 2.12 ms (lane 2), and the paper's + full-scale walk fits in a p99 under 60 ms (§1). + +
+ - [ ] You can write Algorithm 1 from memory. + +
Answer + + `totSteps = 0, V = 0`; then `repeat`: set `currPin = q`, draw `currSteps = SampleWalkLength(α)`, + and for that many iterations step `currBoard = E(currPin)[rand()]` then + `currPin = E(currBoard)[randNeighbor()]` and `V[currPin]++`; add `currSteps` to `totSteps`; + `until totSteps >= N`; `return V`. Eleven lines (Pixie §3). The only runtime knob is `N`; `α` + controls how long each sub-walk runs before restarting at `q`, which keeps visits local. The + recommendation is the pins with the highest `V`. + +
+ - [ ] You can explain all four innovations and what each one fixes. + +
Answer + + (1) **Biasing** (`PersonalizedNeighbor`, §3.1) fixes the basic walk returning the same popular + pins to everyone: it slices edges matching the user's language/topic, a subrange operator, and + moves Slovak-target content from 2.13% to 42.55% (Table 3). (2) **Weighted query set + + sub-linear allocation** (Eqs. 1–2) fixes both "a user is not one pin" and "linear allocation + starves low-degree pins" — it is lane 2's largest win, 0.403 → 0.823. (3) **Multi-hit booster** + (Eq. 3 / Algorithm 3 line 5) is meant to reward cross-interest pins, `(√2+√2)² = 8` vs a + single-source `4`, but reproduces no gain here because the generator lacks the premise. (4) + **Early stopping** (Algorithm 2, `n_p`/`n_v`) fixes over-walking: stop when the top is stable, + measured at 35% of the steps and 2.2× faster with unchanged hit rate. + +
+ - [ ] You can give the language-biasing numbers and the pruning result. -- [ ] Your `pixie.rs` reproduces lane 2's early stopping (~35% of steps, ~0.79 overlap) and you - have measured the multi-hit booster yourself rather than assuming it helps. + +
Answer + + Language biasing (Table 3, §4.2): for an English user with a Japanese interest, target-language + content rises from 16.35% (basic walk) to 80.33% (Pixie); for a Slovak interest, 2.13% → 42.55%. + Same-language queries go to ~100%. Pruning (§4.3): at `δ = 0.91` the F1 score peaks 58% above the + unpruned graph while keeping only 20% of the edges — a smaller graph that recommends better, + because the discarded edges were high-entropy boards and low-cosine-similarity edges the walk + would have diffused into. The pruned graph is 1 B boards, 2 B pins, 17 B edges in ~120 GB (§1). + +
+ +- [ ] Your `pixie.rs` reproduces lane 2's early stopping (~35% of steps, ~0.79 overlap) and you have + measured the multi-hit booster yourself rather than assuming it helps. + +
Answer + + Early stopping should land near lane 2's reference: ~3.17 M of 9.0 M steps (35%), ~0.97 ms/query + against 2.12 ms (2.2× faster), top-50 overlap ~0.793, hit-rate@50 unchanged + ([notes.md](notes.md)). The booster is the test of the lesson: run it against plain summed visit + counts at one interest per user and at three, and you should see **no gain** (0.823 vs 0.803, and + 0.563 vs 0.547) — not because the arithmetic is wrong (the unit test pins `(√2+√2)² = 8`) but + because the generator's held-out item is drawn independently of interest overlap. If your run + shows a gain, check whether your generator accidentally introduced cross-interest correlation. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions are in `notes.md`'s guide-question checklist. The load-bearing ones: Q1 (graph + size affects the `edgeVec` build and cache-miss rate, not the per-request step count); Q2 (linear + gives the degree-1 pin `< 1` step → 0, Equation 1 gives it ~5, and `C` must be the graph-wide + log-max or the top query pin gets 0); Q3 (Equation 3 pays only when reachability from multiple + query pins is correlated with the held-out item — test by measuring that correlation in the + training data first). Q4 and Q5 are open-ended; answer them against a workload you actually know. + +
+ ## References - Eksombatchai, Jindal, Liu, Liu, Sharma, Sugnet, Ulrich, Leskovec. *Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time.* WWW 2018 — - [arXiv:1711.07601](https://arxiv.org/abs/1711.07601). + [arXiv:1711.07601](https://arxiv.org/abs/1711.07601). Every section, equation, algorithm and + table number in this chapter is from that paper. + +| Where | What | +|-------|------| +| §1 | scale (17 B edges, 120 GB, r3.8xlarge, p99 < 60 ms, ~1,200 req/s per server); 30–50% engagement lift | +| §3, Algorithm 1 | `BasicRandomWalk`, cost `O(N)` steps | +| §3.1 innovation (1) | biasing, `PersonalizedNeighbor` as a subrange operator | +| §3.1 innovation (2), Eqs. 1–2 | sub-linear step allocation; `C = max log-degree` over all pins | +| §3.1 innovation (3), Eq. 3 / Algorithm 3 line 5 | multi-hit booster | +| §3.1 innovation (4), Algorithm 2 lines 9–13 | early stopping, `n_p`/`n_v` | +| §3.2 / §4.3 | pruning; `δ = 0.91`, F1 +58% at 20% of edges | +| §3.3, Eq. 4 | `edgeVec`, open-addressing visit counter, HugePages (factor 512) | +| §4.1, Tables 1–2 | hit rate 6.3 / 23.1 / 52.2%; A/B lifts | +| §4.2, Table 3 | early-stopping sweep (84% at 1/3 runtime); language biasing | + - Leskovec & Sosič. *SNAP: A General-Purpose Network Analysis and Graph-Mining Library* — the library Pixie is built on. -- Local exercise stub: `topics/42-recommendations-social/experiments/pixie.rs`. +- Repo sources: [`../../FINDINGS.md`](../../FINDINGS.md) row 42; this topic's [`README.md`](README.md), + [`notes.md`](notes.md), and the exercise stub + `topics/42-recommendations-social/experiments/pixie.rs` (`allocate_steps`, `walk_per_query`, + `multi_hit_boost`, `pixie_walk`). - Topic 38 (GraphRAG) — HippoRAG's personalized PageRank is the same primitive with different seeds; topic 18 (GPU) — `edgeVec` is a CSR by another name. diff --git a/topics/42-recommendations-social/reading-tao.md b/topics/42-recommendations-social/reading-tao.md index f882953..88e01c5 100644 --- a/topics/42-recommendations-social/reading-tao.md +++ b/topics/42-recommendations-social/reading-tao.md @@ -18,6 +18,11 @@ hundreds of objects and associations.** ### Step 1 — Why not memcache over MySQL +> **In:** nothing yet — this step is the autopsy of the system TAO replaced. +> **Out:** the three failures of a memcache-over-MySQL **lookaside cache** (a cache the client +> checks first, filling it on a miss) and the thesis they prove — a *narrower* data model buys +> consistency and control a generic key–value cache cannot. §2.1. + Facebook already had a lookaside cache. TAO exists because three specific things went wrong with it, and each is a general lesson: @@ -39,6 +44,12 @@ The pattern: **a narrower data model buys you consistency and control you cannot ### Step 2 — Objects and associations +> **In:** the "narrower data model" mandate from Step 1. +> **Out:** the two shapes the whole store is built from — **objects** (typed, keyed nodes; good for +> repeatable actions) and **associations** (typed, directed, at-most-once edges carrying a 32-bit +> `time` field) — plus inverse types and the deliberately non-atomic "hanging association" repair. +> §3.1. + ``` Object: (id) → (otype, (key → value)*) Assoc: (id1, atype, id2) → (time, (key → value)*) @@ -62,6 +73,11 @@ system choosing eventual repair over distributed transactions, explicitly. ### Step 3 — The query API, in full +> **In:** the objects-and-associations model from Step 2. +> **Out:** the entire query surface — four association reads, three association writes, object CRUD — +> and, as load-bearing as what is present, the omissions: no multi-hop traversal, no pattern +> matching, no path queries, and a per-atype limit "typically 6,000". §3.4. + ``` assoc_get(id1, atype, id2set, high?, low?) assoc_count(id1, atype) @@ -80,6 +96,12 @@ with `pos` or `high`. ### Step 4 — Creation-time locality, and why lists are newest-first +> **In:** the 32-bit `time` field from Step 2 and the `assoc_range`/`assoc_time_range` calls from +> Step 3. +> **Out:** **creation-time locality** (most data is old, most queries want the newest) → association +> lists stored newest-first, and the cache holding *contiguous prefixes* of them — the one +> representation choice Steps 5–8 all serve. §3.4. + The organising principle: > **Association List**: `(id1, atype) → [a_new … a_old]` @@ -98,6 +120,11 @@ prefixes* of them. Everything downstream follows from that one representation ch ### Step 5 — The caching hierarchy +> **In:** the newest-first prefix lists from Step 4. +> **Out:** the three-tier path a query takes — follower → leader → MySQL shard — the sharding rule +> that puts every association on the shard of its `id1` (one hop = one server, which is *why* the API +> has no traversal), and the master/slave geo split forced by reads outnumbering writes 25×. §4.1–4.5. + Three layers, each solving a different problem: ``` @@ -129,7 +156,25 @@ round trip, plus the work. ### Step 6 — Refill, not invalidate -The subtle consequence of caching prefixes: +> **In:** the contiguous-prefix association cache from Steps 4–5. +> **Out:** TAO's consistency model — globally **eventually consistent**, **read-after-write within a +> single tier**, maintained by a cache-coherence protocol that *refills* rather than invalidates — +> and why the prefix representation forces exactly that choice. §6.1, §4.4. + +TAO's consistency, stated plainly (§6.1). Normally "objects and associations in TAO are eventually +consistent; after a write, TAO guarantees the eventual delivery of an invalidation or refill to all +tiers" — replication lag is usually under a second, and once inputs quiesce all copies converge. +**Eventual consistency** means replicas may briefly disagree but a write is guaranteed to propagate +to every tier eventually. On top of that, "TAO provides read-after-write consistency within a single +tier" — **read-after-write** meaning a client that just wrote is guaranteed to see its own write: +the master leader returns a *changeset* synchronously when the write succeeds, and that changeset is +pushed down through the slave leader to the follower tier that originated the write, so a re-read in +that tier reflects it immediately. A version number in both the store and the cache breaks the race +when a second follower's update has not yet arrived. It is, in the paper's own framing, "eventual +consistency with a cache invalidation protocol" — not strong consistency, chosen deliberately for +availability and efficiency. + +The subtle consequence of caching prefixes (§4.4): > Since we cache only contiguous prefixes of association lists, invalidating an association might > truncate the list and discard many edges. Instead, the leader sends a *refill* message to notify @@ -140,8 +185,24 @@ Invalidate a prefix and the follower loses everything after the invalidated edge re-fetch it all. Refill sends the update instead. A cache-coherence protocol designed around the *shape* of the cached value — worth remembering next time you reach for a blanket invalidate. +Worked example — a list of 1,000 cached edges, a write that touches the edge at position 500: + +``` +invalidate-the-list: drop positions 0..999 -> 1,000 edges discarded, all re-fetched on next read +invalidate-at-500: prefix truncates at 500 -> positions 500..999 (500 edges) discarded and re-fetched +refill (what TAO does): fetch the one changed edge, splice it in -> 0 edges needlessly discarded +``` + +Refill re-fetches one edge; the truncating invalidate would have thrown away up to 500. That gap is +the whole argument for building the coherence protocol around the cached value's shape. + ### Step 7 — Hot spots: cloning and client-side caching +> **In:** the consistent-hashed shard→follower mapping from Step 5, which spreads load unevenly. +> **Out:** two hot-spot remedies — **shard cloning** (a hot shard served by several followers) and +> **access-rate-gated client-side caching** (the client caches only items the follower flags as +> hot). §5.3. + Shards map to cache servers by consistent hashing, which "can lead to load imbalance: some followers will shoulder a larger portion of the request load". Two mechanisms: @@ -157,6 +218,11 @@ table and timestamp quantization. Same disease, different medicine. ### Step 8 — Memory engineering +> **In:** the follower cache from Steps 5–7, whose per-entry overhead decides how much of the graph +> fits in RAM. +> **Out:** two space tricks — type-partitioned **arenas** over a slab allocator (so one type can't +> evict another), and the pointerless **14-byte association count** that holds 20% more items. §5.1. + TAO's cache is a slab allocator with LRU and a dynamic slab rebalancer, partitioned into **arenas** by object or association type — "This allows us to extend the cache lifetime of important types, or to prevent poor cache citizens from evicting the data of better-behaved @@ -175,6 +241,11 @@ overhead in a cache holding a social graph. ### Step 9 — What the workload actually looks like +> **In:** the production trace. +> **Out:** the two distributions any honest benchmark of this system must reproduce — 1% of counts +> ≥ 512K (high-degree nodes are real, §5.4) and 64% of non-empty ranges return exactly one edge +> against limits usually ≥ 1000 — plus the payload sizes. §7. + Two distributions from the production trace, both with long tails you must design for: - `assoc_count`: **1% of returned counts were ≥512K**. High-degree nodes are not hypothetical. @@ -193,6 +264,11 @@ average object payload 673 bytes; **39.5% of associations queried contained no d ### Step 10 — The performance envelope +> **In:** everything Steps 5–9 built, deployed on 144 GB Xeon boxes. +> **Out:** the measured envelope — the hit/miss latency table, the 96.4% read hit rate, throughput +> rising with hit rate, five-nines-plus availability — and the reading that the tail is made +> entirely of misses. §8. + 144 GB RAM, 2× 8-core Xeon E5-2660 at 2.2 GHz, 10 GbE. Client-observed, including network and the PHP client stack: @@ -252,13 +328,89 @@ mechanism in Steps 6–8 is about protecting the hit rate rather than making mis ## Done when +Answer each before unfolding it. + - [ ] You can write the two data shapes and the four association queries from memory. + +
Answer + + Shapes (§3.1): an **object** is `(id) → (otype, (key→value)*)`; an **association** is + `(id1, atype, id2) → (time, (key→value)*)`, at most one per (id1, atype, id2), carrying a 32-bit + time field. The four association reads (§3.4): `assoc_get(id1, atype, id2set, high?, low?)`, + `assoc_count(id1, atype)`, `assoc_range(id1, atype, pos, limit)`, and + `assoc_time_range(id1, atype, high, low, limit)` — plus writes `assoc_add`, `assoc_delete`, + `assoc_change_type` and object CRUD. Repeatable actions become objects (a comment); at-most-once + actions become associations (a like). + +
+ - [ ] You can define creation-time locality and say what it forces about list ordering and caching. + +
Answer + + **Creation-time locality** (§3.4): "most of the data is old, but many of the queries are for the + newest subset." It forces two things: association lists are stored in descending time order + (newest first), and the cache holds *contiguous prefixes* of them — so `assoc_range(id1, atype, + 0, 50)` ("50 most recent comments") is a cheap prefix read, and the optional time bounds on + `assoc_get` exist "to improve cacheability for large association lists." It also forces the + invalidation policy to change (Step 6): you can't blindly invalidate a prefix. + +
+ - [ ] You can explain refill-versus-invalidate and why prefix caching demands it. + +
Answer + + Because the cache holds a contiguous prefix, "invalidating an association might truncate the list + and discard many edges" (§4.4) — drop the edge at position 500 of a 1,000-edge list and you lose + positions 500–999 and must re-fetch all of them. Instead the leader sends a **refill**: it fetches + the one changed edge and splices it into the follower's cached list, discarding nothing. The + coherence protocol is built around the *shape* of the cached value (a prefix), not the value's key + — the general lesson against blanket invalidation. + +
+ - [ ] You can quote the 25×-reads-to-writes ratio and say which design decisions follow from it. + +
Answer + + "Read misses by followers are 25 times as frequent as writes in our workloads" (§4.5). From it: + the master/slave *region* architecture (reads must be served locally even when writes cross an + ocean — in-region write latency 12.1 ms versus 74.4 ms = 58.1 + 16.3 from a region 58 ms away); + the leader/follower tiering that keeps reads off the database; and the entire emphasis of Steps + 6–8 on *protecting the hit rate* rather than making misses faster — because at a 96.4% hit rate + the tail latency is made almost entirely of the rare misses. + +
+ - [ ] You can give the hit/miss latency gap and explain why the tail is made of misses. + +
Answer + + From §8's table: a hit is ~1.0–1.3 ms at p50; a miss is 5.0–8.2 ms at p50 and up to 143–187 ms at + p99 (e.g. `assoc_count` hit p50 1.1 / miss p99 186.8; `obj_get` hit p50 1.0 / miss p99 186.4). + Overall read hit rate is 96.4%. Because 96.4% of requests take ~1 ms and only the 3.6% misses take + 5–187 ms, the tail of the latency distribution is composed of misses — which is exactly why every + mechanism in Steps 6–8 targets the hit rate. Throughput per follower rises from ~350K to ~600K + req/s as hit rate goes 85% → 99%; availability over 90 days had a failed-query fraction of + 4.9 × 10⁻⁶. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions are in `notes.md`'s guide-question checklist. The load-bearing ones: Q1 (no + multi-hop is a feature because sharding by `id1` makes every association query single-server; two + hops would need cross-shard fan-out or denormalization); Q2 (the refill/invalidate edge-count + worked in Step 6 — up to 500 edges lost under truncating invalidate, zero under refill); Q4 (the + 14-byte pointerless count buys ~20% more entries — worth a special case only when counts dominate + the working set, as they do here). Q3 and Q5 are analyses you write against the workload and + against topic 40's Zanzibar. + +
+ ## References - Bronson, Amsden, Cabrera, Chakka, Dimov, Ding, Ferris, Giardullo, Kulkarni, Li, Marchukov, diff --git a/topics/43-ops-dependency-graphs/README.md b/topics/43-ops-dependency-graphs/README.md index d88b78e..a90325b 100644 --- a/topics/43-ops-dependency-graphs/README.md +++ b/topics/43-ops-dependency-graphs/README.md @@ -243,9 +243,14 @@ And then the part that will look familiar from topic 10: Table 3 is a set of **query rewrite rules** that push projection, selection and aggregation as close as possible to the source tracepoints — `Π_{p,q}(P ⋈ Q) → Π_p(P) ⋈ Π_q(Q)`, `σ_q(P ⋈ Q) → P ⋈ σ_q(Q)`, and so -on. Measured effect: one query goes from ~600 tuples/s to **6 tuples/s -per DataNode**. Predicate pushdown and join placement, in a tracing -system, for a hundredfold reduction in tuple traffic. +on. Table 3's own target is the number of tuples **packed into the +baggage** — happened-before joins carried in-band. The separately +measured ~600 tuples/s → **6 tuples/s per DataNode** headline is §4's +*process-level (intermediate) aggregation*, which aggregates emitted +tuples within each process and reports globally once per second. Two +optimizations, two metrics: keep them apart. Predicate pushdown and +join placement, in a tracing system, plus in-process pre-aggregation +for a hundredfold reduction in emitted-tuple traffic. ## Reading guides @@ -310,7 +315,9 @@ rare-path recall 1.000 → 0.001, p99 error 0% → 25.6% as the rate goes given two tracepoint predicates, return the pairs where one causally precedes the other within a request. Then implement Pivot Tracing's Table 3 pushdown rules and measure the reduction in tuples that have - to cross the join — their reported figure is ~600/s down to 6/s. + to cross the join — that is Table 3's metric (packed tuples). The + paper's ~600/s → 6/s figure is a different one: emitted tuples under + process-level aggregation (§4). Measure both if you can. ## Cross-topic threads @@ -327,8 +334,9 @@ rare-path recall 1.000 → 0.001, p99 error 0% → 25.6% as the rate goes alerts, and hedged requests are one of the few mitigations that work against a gray failure. - **Topic 10 (query planning) ↔ 43**: Pivot Tracing's Table 3 is - predicate pushdown and join placement, and the 600 → 6 tuples/s result - is exactly the win an optimizer exists to produce. + predicate pushdown and join placement; the 600 → 6 tuples/s result is + §4's process-level pre-aggregation. Both are wins an optimizer exists + to produce — pushdown and partial aggregation. - **Topic 26 (probabilistic structures) ↔ 43**: a real trace pipeline cannot store per-edge latency raw. Histograms, t-digests and count-min sketches are what M43's edge weights have to be. @@ -356,4 +364,5 @@ rare-path recall 1.000 → 0.001, p99 error 0% → 25.6% as the rate goes Dapper's 426 bytes; localization latency on a 10,000-service graph vs `ops_bench` lane 2; **top-1 accuracy under sampling** — the number the whole topic converges on; and happened-before join cost with and - without pushdown, against Pivot Tracing's 600 → 6 tuples/s. + without pushdown, against Pivot Tracing's 600 → 6 tuples/s (which is + its emitted-tuple, aggregation-side figure). diff --git a/topics/43-ops-dependency-graphs/notes.md b/topics/43-ops-dependency-graphs/notes.md index a6da986..f5481ed 100644 --- a/topics/43-ops-dependency-graphs/notes.md +++ b/topics/43-ops-dependency-graphs/notes.md @@ -76,7 +76,7 @@ aggregate, rare-event, or tail — they do not have the same answer. | deployment: 40 servers / 34 routers / 54 links / 3 weeks; agent report <40 KB, 10⁵ agents ≈ **10 Mbps** | Sherlock §5–6 | | Pivot Tracing: `Q1 ⋈ Q2` on Lamport's →, evaluated **in-band via baggage** | Pivot §3–4 | | advice primitives: **OBSERVE, UNPACK, FILTER, PACK, EMIT**; no jumps, no recursion, guaranteed to terminate | Pivot §3 | -| pushdown rewrites reduce one query from **~600 tuples/s to 6 tuples/s** per DataNode | Pivot §4 | +| **process-level (intermediate) aggregation** reduces one query's *emitted* tuples from **~600/s to 6/s** per DataNode; Table 3's rewrites target *packed* (baggage) tuples instead | Pivot §4 | | "all users of HBase pay the **10% performance overhead**" of SchemaMetrics | Pivot §2.3 | | gray failure = **differential observability**: the app sees a problem, the observer does not | Gray Failure §2 | @@ -99,8 +99,9 @@ aggregate, rare-event, or tail — they do not have the same answer. retry storm the *sustaining loop* of a metastable failure. Lane 1's timeout-generated errors are that loop in miniature. - **Topic 10 ↔ 43**: Pivot Tracing's Table 3 is predicate and aggregate - pushdown, and the 600 → 6 tuples/s result is exactly the win an - optimizer exists to produce. The novelty is only *where* it is applied. + pushdown; the 600 → 6 tuples/s result is §4's process-level + pre-aggregation. Both are wins an optimizer exists to produce — the + novelty is only *where* they are applied. - **Topic 26 ↔ 43**: a real trace pipeline cannot store per-edge latency raw. Histograms, t-digests and count-min sketches are what M43's edge weights have to be, and the p99 row of lane 3 is the reason. diff --git a/topics/43-ops-dependency-graphs/reading-dapper.md b/topics/43-ops-dependency-graphs/reading-dapper.md index ae80347..7c31c81 100644 --- a/topics/43-ops-dependency-graphs/reading-dapper.md +++ b/topics/43-ops-dependency-graphs/reading-dapper.md @@ -9,6 +9,12 @@ each justified by a nanosecond count or a percentage, and the biggest of them request in a thousand — comes with an argument about what you are actually looking for that is still the clearest thing written on the subject. +This is a paper, not a codebase, so every number below is anchored to the section, table or figure +of the **Dapper technical report** (Sigelman et al., *dapper-2010-1*, Google, April 2010) that +states it; each was re-checked against the PDF while writing this chapter. Where a figure comes +from this repo's own crate instead, it is marked as a lane of `ops_bench` and traced to +[FINDINGS.md](../../FINDINGS.md) or the topic's `notes.md`. + ## The problem in one sentence **Reconstruct the causal structure of every request across thousands of services, with an overhead @@ -18,93 +24,156 @@ small enough that nobody notices and nobody turns it off.** ### Step 1 — Two requirements, and they fight -Dapper names them up front: **ubiquitous deployment** and **continuous monitoring**. A tracing -system that covers 90% of your services will fail to explain the incident that involves the other -10%, and one you turn on during incidents will not be on when the incident starts. - -Both requirements push toward the same constraint: the overhead must be negligible *always*, not -just acceptable *sometimes*. That is what the rest of the design is optimising. +> **In:** nothing yet — this step is the motivation. +> **Out:** the single constraint (overhead negligible *always*, not just *sometimes*) that every +> later step is optimising, plus the three design goals it decomposes into. -Three derived design goals: +Dapper names two requirements up front (§1): **ubiquitous deployment** — the tracing must cover +essentially every service, because "the usefulness of a tracing infrastructure can be severely +impacted if even small parts of the system are not being monitored" — and **continuous +monitoring**, because "unusual or otherwise noteworthy system behavior is difficult or impossible +to reproduce", so it must already be on when the incident starts. -- **Low overhead** — "a valuable tracing infrastructure could be worth a performance penalty, [but] - we believe that initial adoption would be greatly facilitated if the baseline overheads could be - demonstrably negligible." -- **Application-level transparency** — programmers should not have to be aware of it. -- **Scalability**. +Both requirements push toward the same constraint: the overhead must be negligible *always*, not +just acceptable *sometimes*. A tracing system that covers 90% of your services will fail to explain +the incident that involves the other 10%, and one you turn on during incidents will not be on when +the incident starts. That is what the rest of the design is optimising. + +§1 decomposes the constraint into three **design goals**: + +- **Low overhead** — "the tracing system should have negligible performance impact on running + services… even small monitoring overheads are easily noticeable, and might compel the deployment + teams to turn the tracing system off." The overhead section later reinforces the framing: "one + can argue that a valuable tracing infrastructure could be worth a performance penalty, [but] we + believed that initial adoption would be greatly facilitated if the baseline overheads could be + demonstrably negligible" (§4). +- **Application-level transparency** — programmers should not need to be aware of the tracing + system, because "a tracing infrastructure that relies on active collaboration from + application-level developers… becomes extremely fragile." +- **Scalability** — it must handle Google's size "for at least the next few years." + +Why it matters: transparency and ubiquity are the same requirement seen from two sides, and low +overhead is the price of admission for both. ### Step 2 — The trace tree, and where instrumentation actually goes -A trace is a tree of **spans**; each span has a name, a span id, a parent span id, and belongs to a -trace id. Spans usually correspond to RPCs, with client-send/server-receive/server-send/client-recv -timestamps — and since those live on different machines, "in our analysis tools, we take advantage -of the fact that an RPC client always sends a request before a server receives one, and vice versa -for the server response. In this way we have a lower and upper bound for the span timestamps on -the server side of RPCs." Clock skew handled by causality rather than by NTP. - -Transparency comes from instrumenting three things and nothing else: - -1. **Thread-local trace context** when a thread handles a traced control path. +> **In:** the transparency goal from Step 1. +> **Out:** the data model — a **trace** made of **spans** carrying **annotations** — and the three +> instrumentation points that emit spans without the application's help. Step 3 collects what they +> emit. + +A **trace** is the record of one request's path across services, structured as a tree. A **span** +is one node of that tree: it has a span name, a span id, a parent span id, and belongs to a single +trace id; spans usually correspond to RPCs (§2.1). An **annotation** is application-attached data +on a span — a timestamp or a key/value pair the developer chose to record. **Context propagation** +is the mechanism that carries the trace id and current span id along the request so that every +service attaches its spans to the same tree. + +Each span usually carries client-send / server-receive / server-send / client-receive timestamps. +Those live on different machines with unsynchronised clocks, so Dapper avoids trusting NTP: "an RPC +client always sends a request before a server receives one, and vice versa for the server response. +In this way we have a lower and upper bound for the span timestamps on the server side of RPCs" +(§2.1). Clock skew handled by **causality** — the happened-before ordering of send and receive — +rather than by synchronised clocks. (Topic 43's other papers, Pivot Tracing and Lamport's ordering, +are this same idea generalised.) + +Transparency comes from instrumenting three things and nothing else (§2.2): + +1. **Thread-local trace context** — when a thread handles a traced control path, the trace context + is attached to thread-local storage. 2. **The common control-flow library**, so deferred and asynchronous work carries the trace context of its creator into the callback. 3. **The RPC framework** — "nearly all of Google's inter-process communication is built around a - single RPC framework". + single RPC framework." That is the whole trick, and it is why it worked: Google had one RPC framework and one threading -library. The core instrumentation is **under 1000 lines of C++ and under 800 of Java**. Where the -assumption fails, so does the tracing — "40 C++ applications and 33 Java applications required -some manual trace propagation", and programs using raw TCP sockets or SOAP get nothing. +library. The core instrumentation is **under 1000 lines of C++ and under 800 of Java** (§3.1). +Where the single-framework assumption fails, so does the tracing — "40 C++ applications and 33 Java +applications required some manual trace propagation" (§3.2), and programs using raw TCP sockets or +SOAP RPCs get nothing. + +Why it matters: the data model is inherited unchanged by every modern tracer, but the transparency +that made it deployable is a property of a monoculture few organisations have. ### Step 3 — Out-of-band collection, for two non-obvious reasons -Spans are written to local log files, pulled by a per-machine daemon, and written to a Bigtable -where **a trace is one row and each span is a column** — sparse rows being exactly right for -traces with an arbitrary number of spans. +> **In:** the spans emitted by Step 2's instrumentation. +> **Out:** a Bigtable of traces (**one row per trace, one column per span**), plus the price paid +> for collecting them off the request path — a collection latency that is usually seconds and +> occasionally hours. + +**Out-of-band collection** means the trace data travels through a side channel — written to local +logs, pulled by a daemon, written to a store — instead of riding back inside the RPC responses. The +pipeline is a **three-stage process** (§2.5): span data is (1) written to local log files, (2) +pulled from every production host by per-machine Dapper daemons, and (3) written to a regional +Bigtable, where **a trace is one row and each span is a column**. Sparse Bigtable rows are exactly +right for traces with an arbitrary number of spans. Why not just return trace data in the RPC response? §2.5.1 gives two reasons and the second is the one people miss: - **It would dwarf the application data.** "RPC responses — even near the root of such large - distributed traces — can still be comparatively small: often less than ten kilobytes... the - in-band Dapper trace data would dwarf the application data and bias the results of subsequent - analyses." + distributed traces — can still be comparatively small: often less than ten kilobytes… the in-band + Dapper trace data would dwarf the application data and bias the results of subsequent analyses." - **It assumes perfect nesting.** "in-band collection schemes assume that all RPCs are perfectly nested. We find that there are many middleware systems which return a result to their caller - before all of their own backends have returned a final result." + before all of their own backends have returned a final result." An in-band scheme cannot account + for that non-nested execution. Cost of the choice: collection is not instantaneous. Median latency from log to repository is **under 15 seconds**, but the 98th percentile is bimodal — "approximately 75% of the time, 98th -percentile collection latency is less than two minutes, but the other approximately 25% of the -time it can grow to be many hours." +percentile collection latency is less than two minutes, but the other approximately 25% of the time +it can grow to be many hours" (§2.5). If you build on Dapper-style traces, freshness is a +distribution, not a number. + +Why it matters: the design choice that keeps tracing off the hot path is also the one that means a +trace you need during an incident may not have landed yet. ### Step 4 — The overhead budget, itemised -This is the section that got Dapper deployed, and the numbers are worth memorising because they -set the bar for anything you build: +> **In:** the runtime library and daemon of Steps 2–3. +> **Out:** a table of per-operation costs — the numbers that got Dapper deployed, and the bar for +> anything you build. + +This is the section that got Dapper deployed (§4.1–4.2), and the numbers are worth memorising +because they set the bar for anything you build: ``` - root span creation + destruction ....... 204 ns (extra cost: allocating a global trace id) - non-root span .......................... 176 ns - annotation, span NOT sampled ............. 9 ns (a thread-local lookup) - annotation, span sampled ................ 40 ns - collection daemon .................... < 0.3% of one core, negligible memory - per span on the wire ................... 426 bytes - share of production network traffic ... < 0.01% + root span creation + destruction ....... 204 ns (§4.1; extra cost: allocating a global trace id) + non-root span .......................... 176 ns (§4.1) + annotation, span NOT sampled ............. 9 ns (§4.1; a thread-local lookup) + annotation, span sampled ................ 40 ns (§4.1) + collection daemon .................... < 0.3% of one core, negligible memory (§4.2, Table 1) + per span on the wire ................... 426 bytes (§4.2) + share of production network traffic ... < 0.01% (§4.2) ``` -(Measured on a 2.2 GHz x86 server.) The daemon is also "restricted to the lowest possible priority -in the kernel scheduler in case CPU contention arises." +(Measured on a 2.2 GHz x86 server, §4.1.) The difference between the 204 ns root span and the +176 ns non-root span **is** the cost of allocating a globally unique trace id, and nothing else. The +daemon is also "restricted to the lowest possible priority in the kernel scheduler in case CPU +contention arises" (§4.2). -Note the 9-vs-40 ns split for annotations. Making the unsampled path a thread-local read is what -lets Dapper tell developers to annotate freely — and they did: **70% of all spans and 90% of all -traces have at least one application-specific annotation.** +Note the 9-vs-40 ns split for annotations. The unsampled path is a **thread-local read** — no lock, +no allocation — which is what lets Dapper tell developers to annotate freely; and they did: **70% of +all spans and 90% of all traces have at least one application-specific annotation** (§3.3). The +cheap path is what bought the coverage. + +Why it matters: a per-request nanosecond budget, published operation by operation, is how you argue +a monitoring system into a latency-sensitive fleet. ### Step 5 — Sampling, and the two questions it answers differently -Table 2, on a web-search cluster (experimental error 2.5% for latency, 0.15% for throughput): +> **In:** the overhead numbers from Step 4 — in particular the 16.3% latency cost of tracing every +> request. +> **Out:** the decision to sample, the rate, and the distinction between an **aggregate** question +> and a **rare-event** question that decides whether a rate is safe. + +**Sampling** means recording only a fraction of requests. Table 2 (§4.3), measured on a web-search +cluster with experimental error 2.5% for latency and 0.15% for throughput, is the reason it is not +optional: ``` - sampling avg latency avg throughput + sampling avg latency avg throughput (Dapper Table 2, §4.3) 1/1 +16.3% −1.48% 1/2 +9.40% −0.73% 1/4 +6.38% −0.30% @@ -113,8 +182,11 @@ Table 2, on a web-search cluster (experimental error 2.5% for latency, 0.15% for 1/1024 −0.20% −0.06% ← inside experimental error ``` -Tracing everything costs 16% of your latency. So Dapper sampled **one trace in 1024**, uniformly, -and justified it with an argument about the *kind* of question being asked: +Tracing everything costs 16.3% of your latency. So the **first production version of Dapper** +sampled **one trace in 1024**, a uniform probability applied to Google's high-throughput services +(§4.4) — this is where the famous "1 in 1024" belongs: a fleet-wide default for services doing tens +of thousands of requests per second, not a universal law. It justified the rate with an argument +about the *kind* of question being asked (§4.5): > for high-throughput services, aggressive sampling does not hinder most important analyses. If a > notable execution pattern surfaces once in such systems, it will surface thousands of times. @@ -122,70 +194,107 @@ and justified it with an argument about the *kind* of question being asked: with the caveat in the very next sentence: > Services with lower volume — perhaps dozens rather than tens of thousands of requests per second -> — can afford to trace every request; this is what motivated our decision to move towards -> adaptive sampling rates. +> — can afford to trace every request; this is what motivated our decision to move towards adaptive +> sampling rates. -Lane 3 of this topic's crate measures exactly the gap between those two sentences: +Lane 3 of this topic's crate measures exactly the gap between those two sentences (reference values +in `notes.md`; full table in the topic README): ``` - rate traces edge recall rare-path recall mean-latency err p99 err + rate traces edge recall rare-path recall mean-latency err p99 err (ops_bench lane 3) 1/1 40000 1.000 1.000 0.0% 0.0% 1/16 2470 1.000 0.062 0.0% 1.5% 1/1024 39 1.000 0.001 5.8% 25.6% ``` -Thirty-nine traces recover **all** of the dependency edges, and **one thousandth** of the rare -paths. The mean latency is unbiased; the p99 is 25.6% off. Same sample, three completely different -verdicts, depending on whether your question is aggregate, rare-event, or tail. +Two terms name the two questions. **Edge recall** is the fraction of the true dependency edges that +appear at all in the sampled set — an *aggregate* question, since every edge is exercised constantly. +**Rare-path recall** is the fraction of rarely-taken execution paths that survive — a *rare-event* +question. Work the arithmetic: 40,000 requests at 1/1024 keep `40000 / 1024 = 39.06`, so **39 +traces**. Every dependency edge still shows up somewhere in those 39 traces, so edge recall is +1.000. But a path taken by one request in 40,000 appears in the sample with probability only +`39 / 40000 ≈ 0.001` — which is exactly the measured rare-path recall. Same 39 traces, and the +answer to "what is the dependency graph?" is *complete* while the answer to "what happened on that +one weird path?" is *gone*. + +And note which *metrics* survive (last two columns). The **mean** latency is unbiased under uniform +sampling — averaging a tenth of a percent of the requests still estimates the average — so it stays +within 5.8% even at 39 traces. The **p99** (the latency 99% of requests beat) is made of the tail, +and at 39 traces the tail is a handful of samples, so it is 25.6% off. Aggregate, rare-event, tail: +three different verdicts from one sample. + +Why it matters: "is this sampling rate safe?" has no answer until you say which of the three kinds +of question you are asking of the data. ### Step 6 — Two more sampling layers, and one crucial detail -**Adaptive sampling** (§4.4) replaces the uniform probability with "a desired rate of sampled -traces per unit time", so low-traffic workloads sample themselves up automatically. The actual -probability used is recorded *with the trace*, so analytical tools can weight correctly — do not -skip that detail if you build this. +> **In:** the uniform 1/1024 rate from Step 5, and its two failure modes (low-traffic blind spots; +> a repository write limit). +> **Out:** adaptive sampling and collection-time sampling — and the invariant that makes any of it +> safe: sample **whole traces, never individual spans**. + +**Adaptive sampling** (§4.4) replaces the uniform probability with "a desired rate of sampled traces +per unit time", so low-traffic workloads sample themselves up automatically while very high-traffic +ones lower their rate to keep overhead bounded. One detail decides whether the resulting data is +usable: "the actual sampling probability used is recorded along with the trace itself; this +facilitates accurate accounting of trace frequencies in analytical tools" — without it, you cannot +weight a low-volume trace against a high-volume one, and every aggregate is wrong. **Collection-time sampling** (§4.6) is a second, independent round, needed because the repository -has its own write-throughput limit (Google generated **over 1 TB of sampled trace data per day**, -retained at least two weeks). And here is the detail that matters most: +has its own write-throughput limit: Google's clusters "presently generate more than 1 terabyte of +sampled trace data per day", which users want retained "for at least two weeks." And here is the +detail that matters most: -> We leverage the fact that all spans for a given trace — though they may be spread across -> thousands of distinct host machines — share a common trace id. For each span seen in the -> collection system, we hash the associated trace id as a scalar `z`, where `0 ≤ z ≤ 1`. If `z` is -> less than our collection sampling coefficient, we keep the span and write it to the Bigtable. -> Otherwise, we discard it. By depending on the trace id for our sampling decision, we either -> sample or discard entire traces rather than individual spans within traces. +> We leverage the fact that all spans for a given trace — though they may be spread across thousands +> of distinct host machines — share a common trace id. For each span seen in the collection system, +> we hash the associated trace id as a scalar `z`, where `0 ≤ z ≤ 1`. If `z` is less than our +> collection sampling coefficient, we keep the span… By depending on the trace id for our sampling +> decision, we either sample or discard entire traces rather than individual spans within traces. -**Whole traces, not spans.** Sampling spans independently would leave you with disconnected -fragments — you would keep the data and destroy the causality, which is the only thing you were -collecting it for. The crate's `sampling_keeps_whole_traces` test exists to make you implement -this correctly. +**Whole traces, not spans.** Hashing the *trace* id means every span of a kept trace is kept and +every span of a dropped trace is dropped. Sampling spans independently would leave you with +disconnected fragments — you would keep the data and destroy the causality, which is the only thing +you were collecting it for. The crate's `sampling_keeps_whole_traces` test exists to make you +implement this correctly. + +Why it matters: the causal structure is fragile in exactly one way, and this one hashing decision is +what protects it across two independent sampling stages. ### Step 7 — What it was actually used for -Worth knowing, because it shapes what a trace store must support. The Depot API (DAPI) offers three -access patterns: **by trace id**, **bulk** (MapReduce over billions of traces), and **indexed** — -and the indexing note is a nice storage-engineering aside: "the compressed storage required for an -index into the trace data is only 26% less than for the actual trace data itself", so you cannot -index everything. +> **In:** the trace store filled by Steps 3–6. +> **Out:** the three access patterns a trace store must support, and the storage fact that stops you +> indexing everything. + +Worth knowing, because it shapes what a trace store must support. The Depot API (DAPI, §5.1) offers +three access patterns: **by trace id**, **bulk** (a MapReduce over billions of traces), and +**indexed**. The indexing note is a nice storage-engineering aside: "the compressed storage required +for an index into the trace data is only 26% less than for the actual trace data itself" (§5.1) — an +index that is only a quarter smaller than the data is an index you cannot afford to build over +everything. Section 6 (worth skimming for the war stories) covers inferring service dependencies, tracking -network usage, tracing shared storage, and — the surprise — using trace data to verify that -security policies hold, "which provide greater assurance than source code audits". +network usage, tracing shared storage, and — the surprise — using trace data to verify that security +policies hold: such measurements "provide greater assurance than source code audits" (§2.6). Section +6.2, inferring service dependencies, is this topic's lane 1 in production form. + +Why it matters: a trace store is a database, and the access patterns above are its query workload; +design the storage for them or you will index yourself out of your disk budget. ## How to read the paper (with the concepts in hand) -- **§1.** The two requirements and three design goals. Note the framing that adoption depends on - demonstrable negligibility. +- **§1.** The two requirements (ubiquitous deployment, continuous monitoring) and three design + goals. Note the framing that adoption depends on demonstrable negligibility. - **§2 + Figures 2–3.** Trace trees and spans. The clock-skew-by-causality remark is in §2.1. - **§2.2.** The three instrumentation points. Ask yourself which of the three your own stack has. - **§2.3 + Figure 4.** Annotations, and the configurable upper bound on annotation volume — a guardrail against your own users. +- **§2.5 + Figure 5.** The three-stage collection pipeline and the bimodal 98th-percentile latency. - **§2.5.1.** Out-of-band collection. Both reasons; the perfect-nesting one is the subtle one. - **§4.1–4.3 + Tables 1–2.** The overhead numbers. Memorise the 204 / 176 / 9 / 40 ns figures and Table 2's sampling costs. - **§4.4–4.6.** Adaptive sampling, the "if it surfaces once it will surface thousands of times" - argument, and trace-id-hash collection sampling. + argument (§4.5), and trace-id-hash collection sampling (§4.6). - **§5.1.** The Depot API and the 26% index remark. - **§6.** Skim the experience reports; §6.2 (inferring service dependencies) is this topic's lane 1 in production form. @@ -196,35 +305,148 @@ security policies hold, "which provide greater assurance than source code audits 1. Dapper's transparency rests on Google having one RPC framework and one control-flow library. List what your own stack would need instrumented, and estimate how many of your services would - fall into the "manual propagation required" bucket. -2. Explain both reasons for out-of-band collection. Then say what out-of-band costs, using the - bimodal 98th-percentile collection latency. -3. Annotations cost 9 ns unsampled and 40 ns sampled. Work out why that gap is the design decision - that made 70%-of-spans annotation coverage possible. + fall into the "manual propagation required" bucket (§3.2 puts Google's count at 40 C++ and 33 + Java out of thousands). +2. Explain both reasons for out-of-band collection (§2.5.1). Then say what out-of-band costs, using + the bimodal 98th-percentile collection latency (§2.5). +3. Annotations cost 9 ns unsampled and 40 ns sampled (§4.1). Work out why that gap is the design + decision that made 70%-of-spans annotation coverage (§3.3) possible. 4. Lane 3 shows edge recall at 1.000 with 39 traces and rare-path recall at 0.001. State the property of a question that determines which curve it follows, and classify five questions you have actually asked of a tracing system. -5. Sampling per trace rather than per span is presented almost in passing. Construct the failure: - what does a per-span-sampled data set let you compute, and what does it silently get wrong? +5. Sampling per trace rather than per span (§4.6) is presented almost in passing. Construct the + failure: what does a per-span-sampled data set let you compute, and what does it silently get + wrong? ## Done when +Answer each before unfolding it. + - [ ] You can draw a trace tree and say what a span carries. + +
Answer + + A trace is a tree of spans representing one request's path across services (§2.1). Each span + carries a span name, a span id, a parent span id, and the trace id it belongs to — the parent id + and trace id are what make the collection of spans a tree rather than a bag. A span usually + corresponds to one RPC and holds four timestamps: client-send, server-receive, server-send, + client-receive. + + Because those timestamps come from unsynchronised machine clocks, Dapper does not trust their + absolute values across the client/server boundary. It uses the causal fact that a client sends + before a server receives, and a server sends its response before the client receives it, to bound + the server-side timestamps (§2.1). Developers may also attach annotations — timestamps or + key/value pairs — and 70% of spans carry at least one (§3.3). + +
+ - [ ] You can name the three instrumentation points and the assumption each depends on. + +
Answer + + Thread-local trace context (§2.2) assumes a request is handled by threads that read a thread-local + store; the common control-flow library assumes deferred and async work goes through that one + library, so the callback inherits its creator's context; and the RPC framework assumes "nearly all + of Google's inter-process communication is built around a single RPC framework" (§2.2). + + All three are the same bet: a monoculture. When it fails, tracing fails — §3.2 records 40 C++ and + 33 Java applications that needed manual propagation, and programs on raw TCP sockets or SOAP get + nothing. The whole instrumentation is under 1000 lines of C++ and 800 of Java (§3.1) *because* it + only has to touch three shared libraries. + +
+ - [ ] You can give both reasons for out-of-band collection. + +
Answer + + First, size and bias: traces can have thousands of spans, but RPC responses "even near the root… + can still be comparatively small: often less than ten kilobytes", so in-band trace data "would + dwarf the application data and bias the results of subsequent analyses" (§2.5.1). Returning the + trace inside the response would change the network behaviour you are trying to measure. + + Second, and the one people miss: "in-band collection schemes assume that all RPCs are perfectly + nested" (§2.5.1), and many middleware systems return to their caller before their own backends + finish. An in-band scheme cannot represent that non-nested execution at all. The cost of going + out-of-band is collection latency: median under 15 s, but a bimodal 98th percentile that is under + two minutes 75% of the time and up to many hours the rest (§2.5). + +
+ - [ ] You can quote the overhead numbers and Table 2's sampling costs. + +
Answer + + From §4.1, on a 2.2 GHz x86 server: root span create+destroy 204 ns, non-root 176 ns (the 28 ns + gap is allocating a globally unique trace id), an unsampled annotation 9 ns (a thread-local + lookup), a sampled one 40 ns. From §4.2: the collection daemon never exceeds 0.3% of one core, + each span is 426 bytes on the wire, and trace collection is under 0.01% of production network + traffic. + + Table 2 (§4.3), on a web-search cluster with 2.5%/0.15% experimental error: tracing every request + (1/1) costs +16.3% latency and −1.48% throughput; 1/16 costs +2.12% / −0.08%; 1/1024 is −0.20% / + −0.06%, inside the error bars. The 16.3% latency hit at 1/1 is the number that forced sampling. + +
+ - [ ] You can explain why sampling must be per trace, not per span. + +
Answer + + Because the thing you are collecting is the *causal structure* — the tree that links a request's + spans — and that structure only exists if you keep all of a trace's spans or none. Dapper hashes + the **trace id** to a scalar `z ∈ [0,1]` and keeps the span iff `z` is below the coefficient + (§4.6); since every span of a trace shares that id, the decision is identical for all of them, so + entire traces are kept or dropped. + + Sample spans independently and you keep, say, a thousandth of the spans of every trace — a pile of + disconnected fragments with no parent whose span survived. You would pay the full collection cost + and be left unable to reconstruct a single request. The `sampling_keeps_whole_traces` test in the + crate encodes exactly this invariant. + +
+ - [ ] Your `sampling.rs` reproduces lane 3's two curves. + +
Answer + + Two curves, because there are two questions. Edge recall stays at 1.000 all the way down to + 1/1024 (39 traces from 40,000): every dependency edge is exercised on nearly every request in this + topology, so a thousandth of the traffic still touches every edge. Rare-path recall falls + roughly linearly with the rate — 1.000, 0.062 at 1/16, 0.001 at 1/1024 — because a path taken by + one request in 40,000 appears with probability `39/40000 ≈ 0.001`. + + The metric columns behave differently again: mean-latency error stays ≤ 5.8% because the mean is + unbiased under uniform sampling, while p99 error reaches 25.6% at 39 traces because the tail is + built from a handful of samples. Reproducing all three columns is the point of the lane — it makes + concrete that "aggregate", "rare-event" and "tail" are not the same question. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions push on the parts of Dapper that transfer to your own stack: what instrumenting + three shared libraries would cost you (§2.2, §3.2), why out-of-band collection is worth its latency + (§2.5, §2.5.1), why the 9-ns unsampled annotation path bought 70% coverage (§3.3, §4.1), how to + classify a question as aggregate/rare-event/tail before choosing a rate (lane 3), and how per-span + sampling silently destroys causality (§4.6). + + Write the answers against the anchors above, not from memory — the point of the exercise is that + every claim traces to a section, a table, or a lane of `ops_bench`. + +
+ ## References - Sigelman, Barroso, Burrows, Stephenson, Plakal, Beaver, Jaspan, Shanbhag. *Dapper, a Large-Scale Distributed Systems Tracing Infrastructure.* Google Technical Report dapper-2010-1, April 2010 — [PDF](https://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf). + Section, table and figure citations in this chapter refer to that report. - Fonseca, Porter, Katz, Shenker, Stoica. *X-Trace: A Pervasive Network Tracing Framework.* NSDI 2007 — the metadata-propagation ancestor. - The OpenTelemetry specification, if you want to see this paper's vocabulary standardised. -- Local exercise stub: `topics/43-ops-dependency-graphs/experiments/sampling.rs`. +- Local exercise stub: `topics/43-ops-dependency-graphs/experiments/src/sampling.rs`. - Topic 34 (debugging & production diagnosis) — sampling bias and coordinated omission, the single-machine version of this problem. diff --git a/topics/43-ops-dependency-graphs/reading-gray-failure.md b/topics/43-ops-dependency-graphs/reading-gray-failure.md index 7b06c2a..b226a70 100644 --- a/topics/43-ops-dependency-graphs/reading-gray-failure.md +++ b/topics/43-ops-dependency-graphs/reading-gray-failure.md @@ -5,8 +5,13 @@ the assumption that failures are detectable: a component is up or it is down, a which, and the redundancy machinery does the rest. Huang et al.'s observation is that the failures that actually cause long outages are not like that. The component is *degraded* — slow, intermittently wrong, dropping a fraction of packets — and, critically, **its own failure detector -does not notice while its users are suffering**. They name the general condition *differential -observability*, and once you have the term you start seeing it in every postmortem you read. +does not notice while its users are suffering**. They name the general condition **differential +observability** (§3.2), and once you have the term you start seeing it in every postmortem you read. + +This is a paper, not a codebase, so every claim below is anchored to the section, table or figure of +*Gray Failure: The Achilles' Heel of Cloud-Scale Systems* (Huang et al., HotOS 2017) that states it; +each was re-checked against the PDF while writing this chapter. Where a figure comes from this +repo's own crate instead, it is marked as a lane of `ops_bench` and traced to `notes.md`. ## The problem in one sentence @@ -15,36 +20,85 @@ because the detector is what triggers recovery, nothing happens.** ## The concepts, step by step -### Step 1 — The model: an observer, an app, and a disagreement +### Step 1 — The model: a system, an observer, a reactor, and an app that disagrees + +> **In:** the informal intuition that "degraded but up" breaks health checks. +> **Out:** the paper's four-entity model (§3.1) and the precise definition of gray failure as one +> quadrant of a two-by-two table (§3.2). Every later step is commentary on this table. -The framing is deliberately minimal. A system has some *ground truth* about its own health. Two -parties form a view of it: +The framing is deliberately minimal (§3.1, Figure 2). There is a **system** that provides a service +(a storage service, a data-center network, an IaaS platform) and an **app** that uses it (a web +application, a user, an operator — and "one system may be an app for another system"). Inside the +system live two more entities, and the paper is careful to keep them distinct: ``` - the OBSERVER — the failure detector, the health check, the monitoring - probe. Its view triggers recovery. - the APP — everything that actually uses the component. Its view - is what users experience. + the OBSERVER — "actively or passively gathers information about + whether the system is failing or not": the failure + detector, the health check, the monitoring probe. + the REACTOR — "based on the observations, takes actions to recover + the system": the failover logic, the load-balancer + eviction, the quorum reconfiguration. + the APP — makes its OWN observations, "typically based on + application-specific, end-to-end metrics such as + query latency and remote I/O status". ``` -Four combinations. Three are unremarkable: both healthy (fine), both unhealthy (a *fail-stop* -failure, and the recovery machinery works), observer unhealthy but app fine (a false positive, -annoying but safe). +The observer observes; the reactor acts on what the observer reported. Keeping them separate is the +whole point: the reactor is only ever as good as the observer's view, so if the observer is blind, +the reactor is inert no matter how well it is built. Both observer and reactor "are considered part +of the system" (§3.1). -The fourth is the one with a name. **Differential observability**: the app sees a problem, the -observer does not. That is a gray failure, and it is dangerous precisely *because* the recovery -machinery is inert — the system believes it is fine, so nothing fails over, nothing sheds, nothing -pages, and the degradation persists until a human works it out. +Now the two-by-two (Table 1, §3.2). Rows are the observer's verdict (`Sgood` / `Sbad`); columns are +the app's (`Agood` / `Abad`): -### Step 2 — Why redundancy does not save you +``` + app good (Agood) app bad (Abad) + observer good ➊ no failure ➋ GRAY FAILURE + observer bad ➌ "good kind" ➍ fail-stop / crash +``` -Every fault-tolerance mechanism you have is keyed on the observer's view. Failover triggers when -the detector says the primary is down. Load balancers eject a backend when a probe fails. A quorum -excludes a replica when it stops responding. +- **➊** neither sees a problem — no failure. +- **➋** the app observes a failure but the observer does not. **This is gray failure**, "since users + are suffering but the reactor will not be invoked to help fix the problem" (§3.2). +- **➌** the observer sees a problem the app has not felt yet. This is *also* differential + observability, "but of the good kind": the observer "will take proactive steps to repair it" + before the app is affected. The paper flags it as problematic *only if it is a false positive* — + "but that is a different kind of problem than gray failure" (§3.2). It is not simply "annoying but + safe"; used well it is the system fixing itself early. +- **➍** both agree the system is failing — "crash and fail-stop failures fall under this case", and + the recovery machinery works. + +The exact definition, worth quoting because people paraphrase it wrongly: a system experiences gray +failure "when at least one app makes the observation that system is unhealthy, but observer observes +that system is healthy" (§3.2). Cell ➋, and only cell ➋. + +### Step 2 — Why redundancy does not save you, with the fan-out arithmetic + +> **In:** the model from Step 1, plus the industry reflex that redundancy buys availability. +> **Out:** the §2.1 result that redundancy can *lower* availability under gray failure, and the +> one formula that makes it quantitative. + +Every fault-tolerance mechanism you own is keyed on the *observer's* view, because the reactor acts +only on what the observer reports. Failover triggers when the detector says the primary is down. A +load balancer ejects a backend when a probe fails. A quorum excludes a replica when it stops +responding. Under gray failure (cell ➋) none of that fires — and worse, the degraded component keeps +*accepting* work, because it is up, so a round-robin or least-connections balancer may send it +*more*. + +§2.1 ("High redundancy hurts") turns this into arithmetic. Consider a front-end that must fan out a +request to many back-ends and wait for almost all to respond. With `n` core switches and fan-out +factor `m`, the probability that a *given* core switch is traversed by a request is: -Under differential observability none of that fires. Worse, the degraded component keeps -*accepting* work — it is up, after all — so it keeps absorbing traffic it will serve slowly, and a -load balancer using round-robin or least-connections may even send it *more*. +``` + P(switch on path) = 1 − ((n − 1) / n)^m (Gray Failure §2.1) +``` + +Read the limit: as `m` grows, `((n−1)/n)^m → 0`, so `P → 100%` — "each such request has a high +probability of involving every core switch." So a gray failure at *any one* switch delays *nearly +every* front-end request. And now the counter-intuitive part: "the more core switches there are, the +more likely at least one of them will experience a gray failure." Adding redundancy adds surfaces +that can silently degrade, and the fan-out guarantees each one touches almost every request. The +mechanism that was supposed to mask failures is the mechanism that spreads this one. This is the connective tissue with topic 37: a fan-out to N backends takes the maximum of N latencies, so one degraded backend in a hundred contaminates a large fraction of requests. Hedged @@ -53,51 +107,92 @@ not require anybody to declare the slow component dead. ### Step 3 — Why detection is genuinely hard, not merely neglected -The paper is careful not to make this a story about lazy monitoring. Three structural reasons: +> **In:** the fact from Step 2 that recovery depends on the observer noticing. +> **Out:** three structural reasons the observer misses the problem — drawn from the §2.2 "under the +> radar" incident — so you stop treating gray failure as a monitoring-team oversight. + +The paper is careful not to make this a story about lazy monitoring; its §2.2 example is a driver +bug where "the failure detector, a remote compute manager, does not observe any problems because it +does not exercise the VM's external network" — it reads heartbeats over a local RPC path the bug +does not touch. Generalise that into three structural reasons: - **The observer is usually cheap and shallow**, by necessity: a health check that exercised every code path would cost as much as the workload. So it checks liveness, not correctness, and certainly not latency under contention. -- **The observer's workload differs from the app's.** A probe that reads a fixed small key will not - see a problem that appears only under a particular access pattern, at a particular size, on a - particular device. -- **Degradation is often partial and intermittent.** A disk that is slow on 1% of writes, a NIC - dropping a small fraction of packets, a memory leak that only matters after eight hours. Any - single probe is likely to miss it. +- **The observer's workload differs from the app's** (this is the §2.2 incident exactly). A probe on + a path the fault does not touch cannot see the fault; app observations are "based on + application-specific, end-to-end metrics" (§3.2) that exercise different paths. +- **Degradation is often partial and intermittent.** A disk slow on 1% of writes, a NIC dropping a + small fraction of packets, a memory leak that only matters after eight hours. Any single probe is + likely to miss it. + +The paper's word for the underlying gap is *observational differences* (§2.2): the app and the +observer are looking at different things, so of course they can disagree. ### Step 4 — Gray failures escalate, which is why they end up in postmortems -The observation that makes this operationally urgent: gray failure is frequently not the end state -but the *prologue*. The degradation persists, work backs up behind it, queues grow, retries pile -on, and eventually something crosses a threshold and fails hard — at which point the failure is -detectable, but you are now diagnosing the crash rather than the degradation that caused it, and -the trail is cold. +> **In:** the persistent, undetected degradation of Step 3. +> **Out:** the §3.3 temporal model — latent → gray → complete failure, a ➊→➋→➍ walk across Table 1 +> — and why the postmortem always arrives after the trail has gone cold. + +§3.3 ("Temporal evolution") gives the lifecycle explicitly: "initially, the system experiences minor +faults (latent failure) that it tends to suppress. Gradually, the system transits into a degraded +mode (gray failure) that is externally visible but which the observer does not see. Eventually, the +degradation may reach a point that takes the system down (complete failure), at which point the +observer also realizes the problem." In Table 1's coordinates this "manifests as a transition from +➊ to ➋ to ➍." The canonical example the paper gives is a memory leak. + +Operationally that ordering is the trap: by the time the failure is detectable (cell ➍), you are +diagnosing the crash rather than the degradation that caused it, and the trail is cold. §2.3 +("Recovery that kills") is the worked horror story — a storage manager keeps routing writes to a +capacity-degraded server it cannot see is degraded, crashing and rebooting it in a loop until a +cascading failure takes down the cluster. If you have read topic 35, this is a metastable failure with a gray failure as its trigger: the sustaining feedback loop (retries against a slow dependency) outlives whatever started it. And in -topic 43's lane 1 you can see the mechanism in miniature — the broken service is slow, its callers +this topic's lane 1 you can see the mechanism in miniature — the broken service is slow, its callers time out, and the *timeouts* are what generate the error storm. -### Step 5 — What to do about it, and what this topic does about it - -The paper's direction is to close the gap between the two views: make the observer's view -approximate the app's, by deriving health signals from what applications actually experience -rather than from synthetic probes. Aggregate client-side latency, error rates observed *by callers*, -and cross-check components against their peers, since a degraded component usually looks different -from its replicas even when it looks fine on its own. - -That is exactly what the two localization methods in this topic do, and it is worth seeing them as -answers to this paper: +### Step 5 — What to do about it, and the ranking arithmetic that proves you must + +> **In:** the escalation from Step 4 and the definition from Step 1. +> **Out:** the paper's four solution directions (§4), the way the two localization methods in this +> topic instantiate them, and the lane-1 numbers — worked out — that show a per-node health check +> ranking the broken service *below* the median. + +The paper's §4 outlines four directions, and it is worth being precise about them because the naive +summary ("just watch the app side") is not quite what the paper says: + +- **§4.1 Multi-dimensional health monitoring.** Move "from singular failure detection (e.g., with + heartbeats) to multi-dimensional health monitoring" — the vital-signs analogy: not just a + heartbeat but temperature and blood pressure too. +- **§4.2 Approximating application views.** Eliminating differential observability entirely is + "practically infeasible", so instead the system should "measure metrics that approximate the + observations of its apps." Note carefully: the paper's own example *is* a probe — "send probes to + measure server-to-server latency and reachability to emulate observations of the network… as in + Pingmesh." So the fix is not "stop using probes"; it is "use probes/metrics that approximate the + app's end-to-end experience," with the caveat that "overly active probing may further burden an + already degraded system." +- **§4.3 Leveraging the power of scale.** Because gray failure "is often due to isolated + observations of an observer," aggregate observations "from a large number of different components + that are complementary to each other" and apply statistical inference. This is the direction the + two localization methods in this topic live in. +- **§4.4 Harnessing temporal patterns.** Find the temporal precursors (the latent-failure prelude) + to warn before apps are affected. + +The two localization methods in this topic are answers to §4.3 — cross-component inference from +app-side signals: - **Sherlock** (2007) refuses a binary health model outright. Its *troubled* state — "servers or links continue to function but users perceive poor performance" — is differential observability encoded in the data model, and its **observation nodes are client-side measurements**, never server-side health checks. - **The random walk** never asks any component whether it is healthy. It only uses the topology and - the correlation between a component being on a path and that request failing — a purely - app-side signal. + the correlation between a component being on a path and that request failing — a purely app-side + signal. -Lane 1 measures what happens when you do not do this: +Now lane 1 measures what a per-node health check does *without* any of this — and the ranking +arithmetic is the point, so work it through rather than just reading the numbers: ``` the broken service is infra-0 — SLOW on 55% of calls, not failing @@ -109,72 +204,190 @@ Lane 1 measures what happens when you do not do this: and all five infra leaves sit at 0.0040-0.0041 — indistinguishable ``` -Thirty-four alerts, and the cause is in the bottom half of both rankings with a health check that -is green. Lane 2 shows both graph methods finding it at mean rank 1.0. +Why 41st by error rate? Because being slow never sets infra-0's *own* error flag. In the generator +its own failures come only from the `baseline_error` = 0.004 coin, exactly like every other healthy +leaf, so its error rate is 0.0040 — pinned to the baseline. Fifty-four services, and 40 of them +happen to have a slightly higher error rate by chance or by manufacturing timeouts, so the *actually +broken* one lands 41st. A per-error-rate ranking does worse than random on it. + +Why then is it 35th by failure *count*, a little higher? Because infra-0 is the infra leaf with the +most callers (20 of them), so it receives the most calls; the same baseline 0.4% rate applied to a +larger call volume produces more failures in *absolute* count, nudging it up from 41st to 35th. But +still bottom-half: the 34 services above it are its *callers*, which time out on its slowness +(`timeout_prob` = 0.7) and "report an error of its own — so the errors appear one hop above the thing +that is actually broken" (services.rs). Those 34 callers are exactly the "34 of 55 alerting", and +the broken service sits at rank 35, just underneath the storm it caused. That gap — cause below the +median, symptoms above the alert line — is differential observability made numeric. Lane 2 shows +both graph methods recovering it at mean rank 1.0. ### Step 6 — The transferable habit +> **In:** everything above — the model, the escalation, the numbers. +> **Out:** two questions to ask of any system you operate, and one design principle that generalises +> past infrastructure. + Two questions to ask of any system you operate: 1. **Whose view triggers my recovery?** If the answer is a health endpoint the component serves - about itself, you have a differential-observability gap by construction. + about itself, you have a differential-observability gap by construction: the observer and the + thing it observes are the same component (§3.1's observer-inside-the-system). 2. **What would a degraded-but-up component look like in my telemetry?** If the honest answer is - "like a healthy one", you will find out about it from a user. + "like a healthy one", you will find out about it from a user — cell ➋, every time. -And a design note that generalises past infrastructure: any time a system's self-assessment drives -its own remediation, ask what happens when the self-assessment is the thing that is broken. +And a design note that generalises: any time a system's self-assessment drives its own remediation, +ask what happens when the self-assessment is the thing that is broken. That is §2.3's storage +manager, and it is the reactor acting on a blind observer. ## How to read the paper (with the concepts in hand) It is six pages; read all of it. But read it in this order: -- **§2 (the model)** first — the observer/app/ground-truth triangle and the four-cell table. The - term *differential observability* is defined here and the rest of the paper is commentary. -- **§1 and §3 (the examples)** second, now that you have the frame. The value of the examples is - recognising the shape, not memorising the incidents. -- **§4–5 (implications and directions)** last. Read the escalation argument against topic 35's - metastable-failure paper and note that they are describing the same lifecycle from two ends. -- **After the paper.** Re-read lane 1's output and identify, for each row, which of the two views - it represents. Then do exercise 4 of this topic — localize under sampling — because the question - "how much observability do I actually need to close the gap?" is the practical form of this - paper's argument. +- **§3 (the model) first** — §3.1 Terminology (the system / observer / reactor / app quartet and + Figure 2) and §3.2 Differential observability (the four-cell Table 1). The term is *defined* in + §3.2 and the rest of the paper is commentary; §3.3 adds the temporal ➊→➋→➍ walk. +- **§2 (the examples) second**, now that you have the frame: §2.1 High redundancy hurts (the fan-out + formula), §2.2 Under the radar (the driver-bug detector gap), §2.3 Recovery that kills (the + cascading storage loop), §2.4 The blame game. The value of the examples is recognising the shape, + not memorising the incidents. +- **§4 (directions) last.** Read the four directions and note which of this topic's methods each one + predicts. Read the escalation argument against topic 35's metastable-failure paper and note that + they describe the same lifecycle from two ends. +- **After the paper.** Re-read lane 1's output and identify, for each row, which of the two views it + represents. Then do exercise 4 of this topic — localize under sampling — because "how much + observability do I actually need to close the gap?" is the practical form of §4.2. ## Questions to answer in notes.md -1. Draw the four-cell observer × app table and put a real incident you have seen in each cell. +1. Draw the four-cell observer × app table (§3.2) and put a real incident you have seen in each cell. Which cell was hardest to diagnose, and did the model predict that? 2. Lane 1's broken service has an error rate exactly at baseline. Write the health check that would have caught it, then estimate what that health check costs to run continuously against every - component. Is it affordable? -3. The paper argues gray failures escalate into fail-stop ones. Connect that to topic 35's + component. Is it affordable? Tie your answer to §4.2's warning that "overly active probing may + further burden an already degraded system." +3. The paper argues gray failures escalate into fail-stop ones (§3.3). Connect that to topic 35's metastable failures: which is the trigger and which is the sustaining loop, and where would you cut? -4. Sherlock's *troubled* state predates this paper by ten years. Why do you think the industry - still ships binary health checks — and what would have to change in a load balancer's interface - to express three states? -5. Both localization methods in lane 2 use only app-side signals. Construct a gray failure that - defeats them both, and say what additional observation would be needed. +4. Sherlock's *troubled* state predates this paper by ten years. Why do you think the industry still + ships binary health checks — and what would have to change in a load balancer's interface to + express three states? +5. Both localization methods in lane 2 use only app-side signals — §4.3's "leveraging scale." + Construct a gray failure that defeats them both, and say what additional observation would be + needed. ## Done when +Answer each before unfolding it. + - [ ] You can define differential observability and draw the four-cell table. + +
Answer + + Gray failure is differential observability: "at least one app makes the observation that system is + unhealthy, but observer observes that system is healthy" (§3.2). The four-cell table has the + observer's verdict on the rows (`Sgood`/`Sbad`) and the app's on the columns (`Agood`/`Abad`): ➊ + both good = no failure; ➋ `Sgood`/`Abad` = **gray failure** (users suffer, reactor never invoked); + ➌ `Sbad`/`Agood` = differential observability "of the good kind", where the observer repairs + proactively before the app feels it (bad only if it is a false positive); ➍ both bad = fail-stop, + where recovery works. + + The trap is ➋ and only ➋. ➌ is the *same* asymmetry pointed the other way and is usually benign — + do not lump it in with gray failure. + +
+ - [ ] You can explain why redundancy mechanisms are inert under gray failure. + +
Answer + + Because every fault-tolerance mechanism is driven by the *reactor*, and the reactor acts only on + the *observer's* view (§3.1). Under gray failure the observer sees health (cell ➋), so nothing + fails over, nothing is evicted, no quorum reconfigures. Worse, the degraded component keeps + accepting work because it is "up", so a round-robin or least-connections balancer may route it + *more* traffic. + + §2.1 makes it quantitative: with `n` core switches and fan-out `m`, a given switch is on a request + with probability `1 − ((n−1)/n)^m`, which tends to 100% as `m` grows — so one gray-failing switch + delays nearly every request, and adding switches only adds more surfaces that can silently + degrade. Redundancy raises the chance that *at least one* component is gray-failing. + +
+ - [ ] You can give three structural reasons detection is hard. + +
Answer + + (1) The observer is cheap and shallow by necessity — a check that exercised every path would cost + as much as the workload, so it checks liveness, not latency-under-contention. (2) The observer's + workload differs from the app's: the §2.2 incident is a driver bug the detector never sees because + its heartbeat travels a local RPC path the bug does not touch, while the app's traffic takes the + broken external path. (3) Degradation is partial and intermittent — 1% of writes slow, a fraction + of packets dropped — so any single probe likely misses it. + + The paper's umbrella term is *observational differences* (§2.2): the observer and the app measure + different things, so disagreement is structural, not negligent. + +
+ - [ ] You can connect gray failure to metastable failure as trigger and sustaining loop. + +
Answer + + §3.3 gives gray failure a lifecycle: latent → gray → complete, a ➊→➋→➍ walk. The gray phase is a + persistent, undetected degradation. In topic 35's terms that degradation is the *trigger*, and the + *sustaining loop* is the retry/timeout traffic it induces — callers giving up on the slow + dependency and retrying, which keeps the pressure on even after the original fault would have + cleared. §2.3's storage manager rebooting a degraded server in a loop is the worked example. + + Where to cut: break the sustaining loop (bound retries, shed load, hedge instead of retry) rather + than only chasing the trigger, because by the time you reach cell ➍ the trail to the trigger is + cold. + +
+ - [ ] You can point at lane 1's output and say which numbers are the observer's view and which are the app's. + +
Answer + + The observer's view is infra-0's own error rate, 0.0040 — pinned to the baseline because being + slow never sets its own error flag (its failures come only from the `baseline_error` = 0.004 coin, + like every healthy leaf). So a per-node health check ranks it 41st of 55 by error rate and 35th by + failure count; the failure-count rank is a little higher only because infra-0 has the most callers + (20), so the same baseline rate over more calls yields more absolute failures. + + The app's view is the "34 of 55 alerting" — those 34 are infra-0's *callers*, which time out on + its slowness (`timeout_prob` = 0.7) and manufacture errors "one hop above the thing that is + actually broken." Cause at rank 35, symptoms at ranks 1–34: differential observability made + numeric. Lane 2's graph methods put it back at mean rank 1.0. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions push the model onto systems you operate: placing real incidents in the four + cells (§3.2), pricing the health check that would catch lane 1's fault against §4.2's + probing-burden caveat, mapping the gray→fail-stop escalation onto topic 35's trigger/loop + distinction, asking why binary health checks persist despite Sherlock's three-state model, and + constructing a gray failure that defeats app-side localization. + + Write the answers against the anchors above — §3.1's reactor-vs-observer split, §2.1's fan-out + formula, §3.3's lifecycle, and lane 1's ranking arithmetic — not from the summary. + +
+ ## References - Huang, Guo, Lou, Liu, Bragstad, Bhatti, Chandra, Kumar, Maltz, Zhang. *Gray Failure: The Achilles' Heel of Cloud-Scale Systems.* HotOS 2017 — - [PDF](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf). + [PDF](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf). Section and + table citations in this chapter refer to this paper. - Bahl et al. *Towards Highly Reliable Enterprise Network Services via Inference of Multi-level Dependencies.* SIGCOMM 2007 — the *troubled* state, ten years earlier. - Dean & Barroso. *The Tail at Scale.* CACM 2013 (topic 37) — why one degraded backend contaminates a fan-out, and why hedging works when failure detection does not. - Bronson, Aghayev, Charapko, Zhu. *Metastable Failures in Distributed Systems.* HotOS 2021 (topic 35) — the lifecycle a gray failure often triggers. -- Local experiment: `topics/43-ops-dependency-graphs/experiments/services.rs` — the gray failure, +- Local experiment: `topics/43-ops-dependency-graphs/experiments/src/services.rs` — the gray failure, planted. diff --git a/topics/43-ops-dependency-graphs/reading-pivot-tracing.md b/topics/43-ops-dependency-graphs/reading-pivot-tracing.md index abfa440..305133b 100644 --- a/topics/43-ops-dependency-graphs/reading-pivot-tracing.md +++ b/topics/43-ops-dependency-graphs/reading-pivot-tracing.md @@ -4,42 +4,59 @@ This is the database paper in an operations topic, and it should be read as one. contribution is a **relational operator** — the happened-before join — plus an evaluation strategy and a set of query rewrite rules. Swap the vocabulary and you are reading about a distributed join whose predicate is Lamport's `→`, evaluated by pushing state along the request instead of shipping -tuples to a coordinator, with projection, selection and aggregation pushed down to the sources. -The measured effect of the pushdown is a hundredfold reduction in tuple traffic. Topic 10 would +tuples to a coordinator, with projection, selection and aggregation pushed down to the sources. The +measured effect of the pushdown is a hundredfold reduction in tuple traffic. Topic 10 would recognise every move. +This is a paper, not a codebase, so every claim below is anchored to the section, table or figure of +*Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems* (Mace, Roelke, Fonseca, SOSP 2015, +Best Paper) that states it; each was re-checked against the PDF while writing this chapter. + ## The problem in one sentence -**The metric you need was not the one anybody thought to record, and the fields you want to group -it by are measured in a different process on a different machine.** +**The metric you need was not the one anybody thought to record, and the fields you want to group it +by are measured in a different process on a different machine.** ## The concepts, step by step ### Step 1 — Two failures of ordinary monitoring -**"One size does not fit all."** What gets logged is decided a priori, by developers, and "there is -a mismatch between the expectations and incentives of the developer and the needs of operators and -users." The paper's evidence is a wall of Apache issue-tracker citations: users asking for new -metrics, new aggregations, new breakdowns of existing metrics — and being refused. And when -metrics *are* added, everybody pays: "HBase SchemaMetrics were introduced to aid developers, but -all users of HBase pay the 10% performance overhead they incur." +> **In:** nothing yet — this step is the motivation. +> **Out:** the two problems (§2.3) that every later step attacks: *what* gets recorded is fixed too +> early, and the *cause* lives across a boundary the record cannot cross. + +**"One size does not fit all"** (§2 heading). What gets logged is decided a priori, by developers, +and "there is a mismatch between the expectations and incentives of the developer and the needs of +operators and users." The paper's evidence is a wall of Apache issue-tracker citations: users asking +for new metrics, new aggregations, new breakdowns of existing metrics — and being refused. And when +metrics *are* added, everybody pays: "HBase SchemaMetrics were introduced to aid developers, but all +users of HBase pay the 10% performance overhead they incur" (§2.3). -**Crossing boundaries.** The root cause and the symptom live in different processes, different -tiers, and are visible to different people. The paper quotes a Mesos issue: "The actually +**Crossing boundaries.** The root cause and the symptom live in different processes, different tiers, +and are visible to different people. The paper quotes a Mesos issue (§2.3): "The actually interesting / useful information is hidden in one of four or five different places, potentially -spread across as many different machines. This leads to unpleasant and repetitive searching -through logs looking for a clue to what went wrong. (…) There's a lot of information that is -hidden in log files and is very hard to correlate." +spread across as many different machines. This leads to unpleasant and repetitive searching through +logs looking for a clue to what went wrong. (…) There's a lot of information that is hidden in log +files and is very hard to correlate." + +Dynamic instrumentation (DTrace, Fay, SystemTap) fixes the first problem and not the second. The +paper's own framing (§1): the limitation "is fundamental" — those probes are side-effect-free by +design, so "neither Fay nor DTrace can affect the monitored system to propagate the monitoring +context" across an address-space or OS boundary. Which is exactly what the second problem needs. -Dynamic instrumentation (DTrace, Fay, SystemTap) fixes the first problem and not the second: those -probes are side-effect-free by design, so they cannot share information across boundaries. +Why it matters: the two problems are orthogonal, and Pivot Tracing is the first system that answers +both — dynamic queries (problem one) whose operator spans boundaries (problem two). ### Step 2 — Tracepoints and a query language +> **In:** the "record it dynamically" half of Step 1. +> **Out:** the data model (**tracepoint invocations are streaming datasets**) and the relational +> query language over them (§3, Table 1) — so that Step 3's join has operands. + A **tracepoint** is a location in the code where instrumentation can be installed; when execution reaches it, it emits a tuple of exported variables plus host, timestamp, process id and name. Tracepoint invocations are therefore *streaming datasets*, and Pivot Tracing queries are relational -queries over them: +queries over them (§3, Table 1): ``` From use tuples from a set of tracepoints @@ -51,30 +68,47 @@ queries over them: ⋈ Join d In Disk On d -> e the happened-before join ``` -plus temporal filters `MostRecent`, `MostRecentN`, `First`, `FirstN`. +plus temporal filters `MostRecent`, `MostRecentN`, `First`, `FirstN` (Table 1). + +Why it matters: framing instrumentation output as a *relation* is what lets a query optimizer touch +it at all — every rewrite in Step 6 is legal only because these are relational operators. ### Step 3 — The happened-before join +> **In:** two tracepoint queries `Q1` and `Q2` from Step 2. +> **Out:** the paper's one novel operator, defined exactly (§3), plus the scoping note that stops +> you mistaking it for a general join. + ``` Q1 ⋈ Q2 produces t1t2 for all t1 ∈ Q1, t2 ∈ Q2 such that t1 → t2 ``` where `a → b` means "the occurrence of `a` causally preceded the occurrence of `b`, and they -occurred as part of the execution of the same request". If `a` and `b` are in different requests, -or in parallel threads that never communicate, there is no join. +occurred as part of the execution of the same request" — Lamport's happened-before, restricted to a +single request. If `a` and `b` are in different requests, or in parallel threads that never +communicate, there is no join. -Figure 3 of the paper is the one to internalise: one execution triggering tracepoints A, B and C -several times, and the tuples produced by `A`, `A ⋈ B`, `B ⋈ C`, and `(A ⋈ B) ⋈ C`. Work it by -hand once and the operator stops being mysterious. +Figure 3 is the one to internalise: one execution triggering tracepoints A, B and C several times, +and the tuples produced by `A`, `A ⋈ B`, `B ⋈ C`, and `(A ⋈ B) ⋈ C`. Work it by hand once and the +operator stops being mysterious. -The claim for it is precise: "Happened-before join substantially improves our ability to perform -root cause analysis by giving us visibility into the relationships *between* events in the -system." And the honest scoping note: "Pivot Tracing is designed to efficiently support -happened-before joins, but does not optimize more general joins such as equijoins." +The claim for it is precise (§3): "Happened-before join substantially improves our ability to perform +root cause analysis by giving us visibility into the relationships *between* events in the system." +And the honest scoping note, which is the reason the whole system can be efficient: "Pivot Tracing is +designed to efficiently support happened-before joins, but does not optimize more general joins such +as equijoins." The operator is narrow on purpose. + +Why it matters: this operator — not "dynamic instrumentation", which predates the paper — is the +contribution. It joins on *causal reachability within one request*, and everything downstream exists +to evaluate it cheaply. ### Step 4 — Advice, and how a query becomes instrumentation -Queries compile to **advice**, woven into tracepoints at runtime. Five primitives: +> **In:** the query and its `⋈` from Step 3. +> **Out:** the five-primitive intermediate form (**advice**, §3, Table 2) that a query compiles to +> and that gets woven into tracepoints at runtime — the executable form the join takes. + +Queries compile to **advice**, woven into tracepoints at runtime. Five primitives (§3, Table 2): ``` OBSERVE construct a tuple from the tracepoint's exported variables @@ -84,38 +118,56 @@ Queries compile to **advice**, woven into tracepoints at runtime. Five primitive EMIT output a tuple for global aggregation ``` -The compilation is mechanical: a `From` clause becomes `OBSERVE`; each `Join` becomes an `UNPACK` -in the downstream advice and a `PACK` in the upstream one; `Where` becomes `FILTER`; `Select` -becomes `EMIT`. `PACK` has the special cases `FIRST` and `RECENT` (and their `N` variants) that -implement the temporal filters. +The compilation is mechanical: a `From` clause becomes `OBSERVE`; each `Join` becomes an `UNPACK` in +the downstream advice and a `PACK` in the upstream one; `Where` becomes `FILTER`; `Select` becomes +`EMIT`. `PACK` has the special cases `FIRST` and `RECENT` (and their `N` variants) that implement the +temporal filters from Table 1. + +The advice API is deliberately restricted (§3): "advice code has no jumps or recursion, and is +guaranteed to terminate." A safety property you would want from anything you weave into a production +system at runtime. -The advice API is deliberately restricted: "advice code has no jumps or recursion, and is -guaranteed to terminate." A safety property you would want from anything you weave into a -production system at runtime. +Why it matters: `PACK`/`UNPACK` are where the join is realised — the upstream side stashes its tuples +and the downstream side retrieves them, which is only possible because of the channel in Step 5. ### Step 5 — Baggage, and why the join is evaluated in-band -The naive way to evaluate `⋈` is the way Magpie did: ship all tuples to a coordinator and join -them there. Figure 6a. It works and it is expensive. +> **In:** the `PACK`/`UNPACK` pair from Step 4, which need a channel between them. +> **Out:** **baggage** (§4) — the per-request container that carries packed tuples along the +> execution path — and the reason the join runs in-situ instead of at a coordinator (Figure 6). -Pivot Tracing instead uses **baggage**: "a per-request container for tuples that is propagated +The naive way to evaluate `⋈` is the way Magpie did: ship all tuples to a coordinator and join them +there (Figure 6a). It works and it is expensive. + +Pivot Tracing instead uses **baggage** (§4): "a per-request container for tuples that is propagated alongside a request as it traverses thread, application and machine boundaries. `PACK` and `UNPACK` store and retrieve tuples from the current request's baggage. Tuples follow the request's execution path and therefore explicitly capture the happened-before relationship." -So the join happens *in situ*, during execution, at the downstream tracepoint. No coordinator, no -cross-cluster tuple shuffle for the join itself — only the final aggregates are emitted. Figure 6b. +So the join happens *in situ*, during execution, at the downstream tracepoint (Figure 6b). No +coordinator, no cross-cluster tuple shuffle for the join itself — only the final aggregates are +emitted. + +Baggage is a generalisation of X-Trace's and Dapper's metadata propagation (§4). If you have met the +W3C `baggage` header in OpenTelemetry, this is where it comes from. -(Baggage is a generalisation of X-Trace's and Dapper's metadata propagation. If you have met the -W3C `baggage` header in OpenTelemetry, this is where it comes from.) +The risk is named rather than hidden (§4): "Pivot Tracing does not inherently bound the number of +packed tuples and potentially accumulates a new tuple for every tracepoint invocation. However, we +liken this to database queries that inherently risk a full table scan — our optimizations mean that +in practice, this is an unlikely event." -The risk is named rather than hidden: "Pivot Tracing does not inherently bound the number of packed -tuples and potentially accumulates a new tuple for every tracepoint invocation. However, we liken -this to database queries that inherently risk a full table scan — our optimizations mean that in -practice, this is an unlikely event." +Why it matters: evaluating the join *in-band* is what makes the whole system cheap enough to leave on +— but it requires a propagation channel through every boundary, which is the price and the deployment +constraint. ### Step 6 — Pushdown, and the hundredfold +> **In:** the in-band join of Step 5, whose two costs are *tuples emitted for aggregation* and +> *tuples packed into baggage*. +> **Out:** the §4 optimizations that cut each cost separately — process-level aggregation for the +> first (the 600 → 6 result), Table 3 rewrites for the second — and why keeping them distinct +> matters. + Table 3 is a set of query rewrite rules, and if you have read topic 10 you already know them: ``` @@ -127,33 +179,48 @@ Table 3 is a set of query rewrite rules, and if you have read topic 10 you alrea ``` "Pivot Tracing rewrites queries to minimize the number of tuples packed... push projection, -selection, and aggregation terms as close as possible to source tracepoints." `Combine` is the -aggregator's combiner function — `Sum` for `Count` — which is the same partial-aggregation trick as -a map-side combiner. +selection, and aggregation terms as close as possible to source tracepoints" (§4). `Combine` is the +aggregator's combiner function — `Sum` for `Count` — the same partial-aggregation trick as a map-side +combiner. -Two measured effects. Intermediate aggregation within each process: "Q2 from §2 is reduced from -approximately **600 tuples per second to 6 tuples per second** from each DataNode." And a reduction -in tuples carried in the baggage, from the join rewrites. +There are **two distinct costs, cut by two distinct mechanisms**, and the paper is careful to keep +them apart — so keep them apart too: -A hundredfold reduction in data movement, from predicate and aggregate pushdown, in a monitoring -system. That is the topic-10 lesson arriving from an unexpected direction: **the gap between what -a user writes and what should actually run is worth closing automatically, wherever the query -happens to live.** +1. **Tuples emitted for global aggregation.** Reduced by *process-level (intermediate) aggregation*, + not by the Table 3 rewrites: Pivot Tracing "aggregates the emitted tuples within each process and + reports results globally at a regular interval, e.g., once per second. Process-level aggregation + substantially reduces traffic for emitted tuples; Q2 from §2 is reduced from approximately **600 + tuples per second to 6 tuples per second** from each DataNode" (§4). *That* is the hundredfold. +2. **Tuples packed into the baggage.** Reduced by the Table 3 rewrites, which push projection, + selection and aggregation toward the sources so fewer tuples ride along the request. + +Conflating the two — attributing the 600 → 6 to the join rewrites — is a common misreading; the paper +credits it to intermediate aggregation. Both are pushdown in spirit; they act on different cost +metrics. + +The topic-10 lesson arriving from an unexpected direction: **the gap between what a user writes and +what should actually run is worth closing automatically, wherever the query happens to live** — even +when "where it lives" is woven into a running production system. ### Step 7 — What this means for a graph engine -Two things worth carrying into capstone M43. +> **In:** the join (Step 3), its in-band evaluation (Step 5) and its rewrites (Step 6). +> **Out:** two things to carry into capstone M43 — the join is a graph reachability operator, and the +> real design axis is *where* it runs, not how. + +The happened-before join is a *graph* operator wearing relational clothes: `t1 → t2` is reachability +in the causal DAG of a request. Implemented over a trace store, it is a variable-length pattern match +with a temporal constraint — precisely what a Cypher engine already knows how to plan. The interesting +question is not how to evaluate it but **where**: in-band during execution (Pivot Tracing's answer, +cheap but requires instrumentation everywhere) or post-hoc over stored traces (the answer available +to a database, expensive but requires nothing of the application). -The happened-before join is a *graph* operator wearing relational clothes: `t1 → t2` is -reachability in the causal DAG of a request. Implemented over a trace store, it is a -variable-length pattern match with a temporal constraint — precisely what a Cypher engine already -knows how to plan. The interesting question is not how to evaluate it but **where**: in-band -during execution (Pivot Tracing's answer, cheap but requires instrumentation everywhere) or -post-hoc over stored traces (the answer available to a database, expensive but requires nothing of -the application). +And the pushdown rules apply to the post-hoc version unchanged. Exercise 7 asks you to implement both +the join and the rewrites over this topic's trace set and measure the reduction. -And the pushdown rules apply to the post-hoc version unchanged. Exercise 7 asks you to implement -both the join and the rewrites over this topic's trace set and measure the reduction. +Why it matters: Pivot Tracing and a trace database are the same query evaluated at two ends of a +spectrum; knowing that is what lets you choose the point on it that your deployment can actually pay +for. ## How to read the paper (with the concepts in hand) @@ -164,47 +231,142 @@ both the join and the rewrites over this topic's trace set and measure the reduc - **§3 + Table 1.** The query language. Then §3's happened-before join definition and **Figure 3** — work the example by hand. - **§3 Advice + Table 2 + Figures 4–5.** The five primitives and how a query compiles to them. -- **§4 Baggage + Figure 6.** In-situ versus centralised evaluation. Figure 6 is the whole argument - in one picture. -- **§4 + Table 3.** The rewrite rules and the 600 → 6 tuples/s result. +- **§4 Baggage + Figure 6.** In-situ versus centralised evaluation. Figure 6 is the whole argument in + one picture. +- **§4 + Table 3.** The rewrite rules, and the two cost metrics: emitted tuples (600 → 6 via + intermediate aggregation) versus packed tuples (Table 3 rewrites). Keep them straight. - **§5.** Implementation: runtime weaving, the agent in every process, one-second publish interval. -- **After the paper.** Do exercise 7 — implement `Q1 ⋈ Q2` over this topic's traces, then the - Table 3 rewrites, and measure the tuple reduction. +- **After the paper.** Do exercise 7 — implement `Q1 ⋈ Q2` over this topic's traces, then the Table 3 + rewrites, and measure the tuple reduction. ## Questions to answer in notes.md 1. State the happened-before join as a graph query rather than a relational one. What is the graph, what is the path predicate, and which topic-11 execution model would you use for it? -2. Baggage evaluates the join in-band, during execution. List what that buys and what it costs, - and name the situation in which post-hoc evaluation over stored traces is strictly better. -3. Table 3's rewrites are textbook pushdown. For each of the five rules, say what would go wrong - if you applied it without checking a precondition. +2. Baggage evaluates the join in-band, during execution. List what that buys and what it costs, and + name the situation in which post-hoc evaluation over stored traces is strictly better. +3. Table 3's rewrites are textbook pushdown. For each of the five rules, say what would go wrong if + you applied it without checking a precondition. 4. The paper likens unbounded baggage growth to "database queries that inherently risk a full table - scan". Extend the analogy: what is the equivalent of a query planner's cost estimate here, and - what would an admission-control policy (topic 35) look like? + scan" (§4). Extend the analogy: what is the equivalent of a query planner's cost estimate here, + and what would an admission-control policy (topic 35) look like? 5. Pivot Tracing needs tracepoints everywhere and a baggage channel through every boundary. Given OpenTelemetry's `baggage` header exists, what is stopping you from running Q2 on your own stack tomorrow? Be specific. ## Done when +Answer each before unfolding it. + - [ ] You can write the happened-before join's definition and work Figure 3's example. + +
Answer + + `Q1 ⋈ Q2` produces the concatenated tuple `t1t2` for every `t1 ∈ Q1` and `t2 ∈ Q2` with `t1 → t2` + (§3), where `→` is Lamport happened-before *restricted to a single request*: `a → b` iff `a` + causally precedes `b` and both are part of the same request's execution. Tuples in different + requests, or in parallel non-communicating threads, do not join. + + Figure 3 shows one execution hitting A, B, C repeatedly; working it by hand you should be able to + produce the tuple sets for `A`, `A ⋈ B`, `B ⋈ C`, and `(A ⋈ B) ⋈ C`, and see that each downstream + tuple pairs only with the upstream tuples that causally preceded it on that request. The scoping + note matters: the system optimizes *this* join and "does not optimize more general joins such as + equijoins" (§3). + +
+ - [ ] You can name the five advice primitives and say how a query compiles to them. + +
Answer + + OBSERVE, UNPACK, FILTER, PACK, EMIT (§3, Table 2). Compilation is mechanical: `From` → OBSERVE; + each `Join` → an UNPACK in the downstream advice paired with a PACK in the upstream advice; `Where` + → FILTER; `Select` → EMIT. PACK's `FIRST`/`RECENT` special cases (and their `N` variants) implement + Table 1's temporal filters. + + The API is restricted on purpose: "advice code has no jumps or recursion, and is guaranteed to + terminate" (§3) — the safety property you need before weaving code into a live system. The PACK on + one side and UNPACK on the other are the two halves of the happened-before join, connected by the + baggage channel. + +
+ - [ ] You can explain baggage and draw Figure 6's two evaluation strategies. + +
Answer + + Baggage is "a per-request container for tuples that is propagated alongside a request as it + traverses thread, application and machine boundaries" (§4); PACK/UNPACK write and read it, so tuples + follow the execution path and "explicitly capture the happened-before relationship." It generalises + X-Trace/Dapper metadata propagation and is the ancestor of OpenTelemetry's `baggage` header. + + Figure 6a is the Magpie-style strategy: ship every tuple to a coordinator and join there — correct + but a cross-cluster shuffle. Figure 6b is Pivot Tracing's: the join runs in-situ at the downstream + tracepoint using baggage, so only final aggregates leave the process. The trade is a propagation + channel through every boundary in exchange for eliminating the shuffle. + +
+ - [ ] You can state three of Table 3's rewrite rules and the 600 → 6 result. + +
Answer + + Three rules (§4, Table 3): projection distributes over the join, `Π_{p,q}(P ⋈ Q) → Π_p(P) ⋈ + Π_q(Q)`; a selection on `P`'s columns pushes to `P`, `σ_p(P ⋈ Q) → σ_p(P) ⋈ Q`; aggregation pushes + through with a combiner, `A_p(P ⋈ Q) → Combine_p(A_p(P)) ⋈ Q`. They push projection, selection and + aggregation toward the source tracepoints, cutting the number of tuples *packed* into baggage. + + The 600 → 6 result is a *different* cost metric and a *different* mechanism: it is the number of + tuples *emitted for global aggregation*, cut by process-level (intermediate) aggregation — "Q2 from + §2 is reduced from approximately 600 tuples per second to 6 tuples per second from each DataNode" + (§4). Do not attribute the hundredfold to the Table 3 rewrites; the paper credits intermediate + aggregation. + +
+ - [ ] You can argue in-band versus post-hoc evaluation for a graph engine. + +
Answer + + The happened-before join is reachability in a request's causal DAG, so it can run two ways. + *In-band* (Pivot Tracing) evaluates it during execution via baggage: cheap at query time, only + aggregates leave each process, but it requires tracepoints and a propagation channel everywhere and + can only answer queries installed before the request ran. *Post-hoc* over a stored trace set asks + nothing of the application and can answer questions you thought of after the fact, but pays to + store and scan the traces and re-derive causality. + + Crucially the Table 3 pushdown rules apply to the post-hoc version unchanged, so the two are the + same query at two ends of a spectrum — you pick the point your deployment can pay for. Exercise 7 + builds the post-hoc version and measures the reduction. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions restate the join as a graph query (topic 11 execution models), weigh in-band vs + post-hoc evaluation, probe the precondition behind each Table 3 rewrite, extend the "full table + scan" analogy toward admission control (topic 35), and ask what actually blocks running Q2 on your + own stack given OpenTelemetry baggage exists. + + Answer them against the anchors above — §3 for the operator and advice, §4 for baggage and the two + cost metrics — not from memory. The recurring lesson is topic 10's: automatically close the gap + between the written query and what runs, wherever the query lives. + +
+ ## References -- Mace, Roelke, Fonseca. *Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems.* - SOSP 2015 (Best Paper) — [PDF](https://cs.brown.edu/~rfonseca/pubs/mace15pivot.pdf). +- Mace, Roelke, Fonseca. *Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems.* SOSP + 2015 (Best Paper) — [PDF](https://cs.brown.edu/~rfonseca/pubs/mace15pivot.pdf). Section, table and + figure citations in this chapter refer to this paper. - Lamport. *Time, Clocks, and the Ordering of Events in a Distributed System.* CACM 1978 — the `→` the join is built on. - Erlingsson, Peinado, Peter, Budiu. *Fay: Extensible Distributed Tracing from Kernels to Clusters.* SOSP 2011 — the dynamic-instrumentation ancestor, and the source of the pushdown optimizations. - Barham, Donnelly, Isaacs, Mortier. *Using Magpie for Request Extraction and Workload Modelling.* OSDI 2004 — the centralised join strategy Figure 6a describes. -- Topic 10 (query planning) — the rewrite rules; topic 11 (execution models) — how you would - actually run the join; topic 35 (overload control) — what an admission policy for a monitoring - query would need. +- Topic 10 (query planning) — the rewrite rules; topic 11 (execution models) — how you would actually + run the join; topic 35 (overload control) — what an admission policy for a monitoring query would + need. diff --git a/topics/43-ops-dependency-graphs/reading-sherlock.md b/topics/43-ops-dependency-graphs/reading-sherlock.md index ca70998..360cc82 100644 --- a/topics/43-ops-dependency-graphs/reading-sherlock.md +++ b/topics/43-ops-dependency-graphs/reading-sherlock.md @@ -8,33 +8,51 @@ paper, and two of its ideas have not been improved on since: modelling a compone *three* states rather than two, and pruning an exponential search with a single empirical observation about how incidents actually happen. +This is a paper, not a codebase, so every claim below is anchored to the section, figure or table of +*Towards Highly Reliable Enterprise Network Services via Inference of Multi-level Dependencies* +(Bahl et al., SIGCOMM 2007) that states it; each was re-checked against the PDF while writing this +chapter. Where a figure comes from this repo's own crate instead, it is marked as a lane of +`ops_bench` and traced to `notes.md`. + ## The problem in one sentence **Users report that a service is slow; hundreds of components are involved and any of them could be -responsible; find the one that is, using only client-side response times and a dependency graph -you had to infer from network traffic.** +responsible; find the one that is, using only client-side response times and a dependency graph you +had to infer from network traffic.** ## The concepts, step by step ### Step 1 — Three states, not two -Every node in Sherlock's model carries a three-tuple: +> **In:** the binary up/down health model everything else assumes. +> **Out:** Sherlock's three-state node (§3.1) and the reason the third state exists — it is the one +> that defeats a health check, and the one this topic's lane 1 plants. + +Every node in Sherlock's model carries a three-tuple (§3.1): ``` (P_up, P_troubled, P_down) summing to 1 ``` -`P_down` is a fail-stop failure — a server is off, a link is cut. `P_troubled` is the state the -whole paper exists for: **"servers or links continue to function but users perceive poor -performance."** +`P_down` is a fail-stop failure — a server is off, a link is cut. `P_troubled` is the state the whole +paper exists for: **"servers or links continue to function but users perceive poor performance"** +(§3.1). -This is *gray failure*, ten years before the HotOS paper named it, and it is the state that -defeats binary health checks. This topic's lane 1 plants exactly it: a shared dependency that is -slow on 55% of calls, whose own error rate never leaves the baseline (0.0040) while 34 of 55 -services alert. A model with only up and down cannot represent the thing that is wrong. +This is *gray failure*, ten years before the HotOS paper named it, and it is the state that defeats +binary health checks. This topic's lane 1 plants exactly it: a shared dependency that is slow on 55% +of calls, whose own error rate never leaves the baseline (0.0040) while 34 of 55 services alert. A +model with only up and down cannot represent the thing that is wrong. + +Why it matters: the whole localization machinery below only earns its keep because it can distinguish +*troubled* from *down* and from *up* — collapse those to two states and Sherlock degenerates into the +per-node ranking that lane 1 shows failing. ### Step 2 — Three kinds of node +> **In:** the three-state node from Step 1. +> **Out:** the three node types of the **Inference Graph** (§3.1) — what is a hidden cause, what is +> measurable, and the layer of "glue" nodes where all the modelling happens. + ``` root-cause nodes physical components whose failure can cause an end-user experience failure: a computer (an IP address), a service @@ -44,74 +62,106 @@ services alert. A model with only up and down cannot represent the thing that is meta-nodes the glue between the two, and where all the modelling is ``` -The state of root-cause nodes is independent; the state of observation nodes is "uniquely -determined from the state of its ancestors". Edges are labelled with a **dependency probability**: -a client may not need DNS on every file fetch, because the name may already be in its local cache, -so the edge is real but weaker than 1.0. +The state of root-cause nodes is independent; the state of an observation node is "uniquely +determined from the state of its ancestors" (§3.1). Edges are labelled with a **dependency +probability**: a client may not need DNS on every file fetch, because the name may already be in its +local cache, so the edge is real but weaker than 1.0. + +Why it matters: the split between what you can *measure* (observation nodes) and what you want to +*blame* (root-cause nodes) is the entire problem statement — localization is inference from the first +layer to the third across the meta-node glue. ### Step 3 — Meta-nodes: three ways for a parent to affect a child +> **In:** the meta-node layer from Step 2. +> **Out:** the three propagation semantics (§3.1.1, Figures 3–5) — noisy-max, selector, failover — +> and the worked argument for why one rule cannot cover all three. + The whole art is here, and the paper is explicit that no single rule works. -**Noisy-max.** *Max*: if any parent is down, the child is down; if none is down and any is -troubled, the child is troubled. *Noisy*: "unless a parent's dependency probability is 1.0, there -is some chance that the child will be up even if the parent is down. Formally, if the weight of a -parent's edge is `d`, then with probability `(1−d)` the child is not affected by that parent." -Figure 3's truth table works this out for two parents — e.g. +**Noisy-max** (§3.1.1, Figure 3). *Max*: if any parent is down, the child is down; if none is down +and any is troubled, the child is troubled. *Noisy*: "unless a parent's dependency probability is +1.0, there is some chance that the child will be up even if the parent is down. Formally, if the +weight of a parent's edge is `d`, then with probability `(1−d)` the child is not affected by that +parent." Figure 3's truth table works this out for two parents — e.g. `P(Child=Troubled | Parent1=Down, Parent2=Troubled) = (1 − d₁) · d₂`, because the child escapes -parent1's state with probability `1−d₁` and then inherits parent2's. +parent1's down state with probability `1−d₁` and then inherits parent2's troubled state with +probability `d₂`. + +**Selector** (§3.1.1, Figure 4). Load balancing. A network load balancer in front of two servers +hashes requests and sends each client to one of them. "An NLB cannot be modeled as a noisy-max +meta-node because the client cannot depend on each server with a probability of 0.5. Using a +noisy-max meta-node will assign the client a 25% chance of being up even when both the servers are +down, which is obviously incorrect." The selector's truth table forces `P(up) = 0` when both parents +are down. Exercise 3 of this topic asks you to build this and show the noisy-max version getting it +wrong. -**Selector.** Load balancing. A network load balancer in front of two servers hashes requests and -sends each client to one of them. "An NLB cannot be modeled as a noisy-max meta-node because the -client cannot depend on each server with a probability of 0.5. Using a noisy-max meta-node will -assign the client a 25% chance of being up even when both the servers are down, which is obviously -incorrect." The selector's truth table forces `P(up) = 0` when both parents are down. Exercise 3 -of this topic asks you to build this and show the noisy-max version getting it wrong. +**Failover** (§3.1.1, Figure 5). Primary/secondary — DNS, WINS, authentication, DHCP. "As long as +the primary server is up or troubled, the child is not affected by the state of the secondary server. +When the primary server is in the down state, the child is still up if the secondary server is up." -**Failover.** Primary/secondary — DNS, WINS, authentication, DHCP. "As long as the primary server -is up or troubled, the child is not affected by the state of the secondary server. When the -primary server is in the down state, the child is still up if the secondary server is up." +Why it matters: get the meta-node wrong and every probability downstream of it is wrong — the NLB/25% +example is the paper showing you that the "obvious" noisy-max default silently mismodels one of the +most common topologies in a data centre. ### Step 4 — The escape hatch, priced -Every Inference Graph gets two extra root causes: **always troubled** at `(0,1,0)` and **always -down** at `(0,0,1)`, wired to *every* observation node. They model "external factors not part of -our model that might cause a user-perceived failure". +> **In:** the Inference Graph from Steps 2–3, which is necessarily incomplete. +> **Out:** the two pseudo-root-causes (§4.2) that absorb everything the model left out, and the exact +> probabilities that make the choice defensible rather than a fudge. -The probabilities are stated, not hand-waved: edges from AT/AD to observation nodes get **0.001**, -"which implies that 1 in 1000 failures are caused by a component not in our model", and router or -path meta-node edges get **0.9999**, "a 1-in-10,000 chance that our network topology or traceroutes -are incorrect or the router is not actually on the path." +Every Inference Graph gets two extra root causes (§3.1 / §4.2): **always troubled** at `(0,1,0)` and +**always down** at `(0,0,1)`, wired to *every* observation node. They model "external factors not +part of our model that might cause a user-perceived failure." -Two things worth taking from this. Every model is incomplete, and the honest response is a term -that absorbs the incompleteness rather than pretending it away. And the paper immediately adds -that "Sherlock's results are not sensitive to the precise setting of these parameters" — which is -the sentence that makes the choice defensible. +The probabilities are stated, not hand-waved (§4.2): edges from AT/AD to observation nodes get +**0.001**, "which implies that 1 in 1000 failures are caused by a component not in our model", and +router or path meta-node edges get **0.9999**, "a 1-in-10,000 chance that our network topology or +traceroutes are incorrect or the router is not actually on the path." + +Two things worth taking from this. Every model is incomplete, and the honest response is a term that +absorbs the incompleteness rather than pretending it away. And the paper immediately adds that +"Sherlock's results are not sensitive to the precise setting of these parameters (Section 6.2)" — +which is the sentence that makes the choice defensible. + +Why it matters: an escape-hatch term with a tiny, sensitivity-tested weight is how a probabilistic +model stays honest about what it does not know without letting that ignorance dominate the ranking. ### Step 5 — The cost of propagation, and the way out +> **In:** the noisy-max semantics of Step 3, applied to a node with `n` parents. +> **Out:** why the naive computation is `O(3ⁿ)` and how noisy-max collapses it to `O(n)` (§3.1.2) — +> the three closed-form products, read as English. + Computing a child's state distribution from `n` parents is `O(3ⁿ)` in general for a three-state -model — you sum over the full truth table. For noisy-max nodes, which are the majority, that -collapses to **`O(n)`**: +model — you sum over the full truth table. For noisy-max nodes, which are the majority, that collapses +to **`O(n)`** (§3.1.2): ``` - P(child up) = Π_j ( (1 − d_j) · (p_j^troubled + p_j^down) + p_j^up ) + P(child up) = Π_j ( (1 − d_j) · (p_j^troubled + p_j^down) + p_j^up ) 1 − P(child down) = Π_j ( 1 − p_j^down + (1 − d_j) · p_j^down ) P(child troubled) = 1 − ( P(child up) + P(child down) ) ``` -Read the first line as: the child is up only if, for every parent, either it does not depend on -that parent (probability `1−d_j`) or that parent is up. Selector and failover stay exponential, -but "these two types of meta-nodes have no more than 6 parents, and hence do not add a significant -computation burden." +Read the first line as: the child is up only if, for every parent, either it does not depend on that +parent (probability `1−d_j`) or that parent is up. Selector and failover stay exponential, but "these +two types of meta-nodes have no more than 6 parents, and hence do not add a significant computation +burden" (§3.1.2). + +Why it matters: the `O(3ⁿ)→O(n)` collapse is what lets Sherlock evaluate a single candidate quickly; +the *number* of candidates is a separate explosion, handled next. ### Step 6 — Ferret, and Observation 3.1 -An **assignment-vector** assigns a state to every root-cause node — "link₁ is down and server₂ is -down and all the other root-cause nodes are up". Fault localization is finding the assignment -vector that best explains the observations. With `r` root causes there are `3^r` of them, and -"existing solutions to this problem in machine learning literature, such as loopy belief -propagation, do not scale to the Inference Graph sizes encountered in enterprise networks." +> **In:** a fast way to score one candidate (Step 5), and `3^r` candidates to score. +> **Out:** the empirical observation (§3.2) that prunes `3^r` to `(2r)^k`, the worked size of that +> reduction, and the second observation that cuts the constant by two orders of magnitude. + +An **assignment-vector** assigns a state to every root-cause node — "link₁ is down and server₂ is down +and all the other root-cause nodes are up". Fault localization is finding the assignment vector that +best explains the observations. With `r` root causes there are `3^r` of them, and "existing solutions +to this problem in machine learning literature, such as loopy belief propagation, do not scale to the +Inference Graph sizes encountered in enterprise networks" (§3.2). The way out is not a better algorithm. It is a fact about incidents: @@ -120,10 +170,15 @@ The way out is not a better algorithm. It is a fact about incidents: > > In large enterprises, there are problems all the time, but they are usually not ubiquitous. -So Ferret evaluates only the assignment vectors with at most `k` abnormal nodes: `2r` vectors with -one abnormal, `2²·C(r,2)` with two, and so on — **at most `(2r)^k`**. And the error is bounded: -"the probability that Ferret does not arrive at the correct solution ... decreases exponentially -with `k` and becomes vanishingly small for `k = 4` onwards." +So Ferret evaluates only the assignment vectors with at most `k` abnormal nodes: `2r` vectors with one +abnormal, `2²·C(r,2)` with two, and so on — **at most `(2r)^k`** (§3.2). Work the reduction for a +realistic graph, `r = 200` root causes and `k = 2`: the brute-force space is `3^200 ≈ 10^95`, while +Ferret evaluates the one-abnormal vectors (`2·200 = 400`) plus the two-abnormal vectors +(`2²·C(200,2) = 4·19,900 = 79,600`) — about **80,000** vectors, under the bound `(2·200)² = 160,000`. +That is `10^95` down to `10^5`. And the error is bounded: "the probability that Ferret does not arrive +at the correct solution ... decreases exponentially with `k` and becomes vanishingly small for +`k = 4` onwards" (§3.2). The one caveat is in a footnote: the observation can fail "in important cases +such as rapid malware infection and propagation" — the regime where many components go bad at once. A second observation halves the constant: @@ -131,77 +186,96 @@ A second observation halves the constant: > evaluation of an assignment vector only requires evaluation of states at the descendants of > root-cause nodes that are not *up*. -Ferret preprocesses by setting everything up and propagating once; each candidate then only -recomputes the descendants of its abnormal nodes and rolls back afterwards. "As there are never -more than `k` nodes that change state out of the hundreds of root-cause nodes in our Inference -Graphs, this reduces Ferret's time to localize by roughly two orders of magnitude." +Ferret preprocesses by setting everything up and propagating once; each candidate then only recomputes +the descendants of its abnormal nodes and rolls back afterwards. "As there are never more than `k` +nodes that change state out of the hundreds of root-cause nodes in our Inference Graphs, this reduces +Ferret's time to localize by roughly two orders of magnitude" (§3.2). -Both observations are worth internalising as a technique: when a search space is exponential, look +Why it matters: both observations are the same technique — when a search space is exponential, look for a fact about the *distribution of real inputs* before you look for a cleverer algorithm. ### Step 7 — Scoring against real measurements +> **In:** a candidate assignment vector and the actual client measurements. +> **Out:** the two-Gaussian response-time score (§4.3) and the significance test that decides whether +> the top-ranked candidate deserves attention at all. + For each observation node, Ferret needs a score in `[0,1]` for how well the predicted state distribution matches what was actually measured. -When the observation is an error or a timeout, the score is just the predicted probability of -being down. When it is a **response time**, Sherlock fits two Gaussians to the historical data — -`Gaussian_up` and `Gaussian_troubled` (the paper's example: mean 200 ms and mean 2 s) — and scores -a measured time `t` as: +When the observation is an error or a timeout, the score is just the predicted probability of being +down. When it is a **response time**, Sherlock fits two Gaussians to the historical data — +`Gaussian_up` and `Gaussian_troubled` (the paper's example, from Figure 1: mean 200 ms and mean 2 s) +— and scores a measured time `t` as (§4.3): ``` p_up · Prob(t | Gaussian_up) + p_troubled · Prob(t | Gaussian_troubled) ``` The score for an assignment vector is the product over observations. And then a significance test, -because a ranked list is worthless without one: Ferret computes the score of the *null hypothesis* -(all root causes up) and requires the best prediction to beat it "by more than one standard -deviation" of the score distribution. - -This topic's crate implements the k = 1 case with a simpler scoring function — least-squares -residual on predicted front-end failure rates — and the detail that makes it work is worth -noticing: **clamping the fitted severity to `[0,1]`**. A severity is a probability, so a candidate -that is simply not on enough requests would need one above 1 to explain the observed rates, and -the clamp is what makes it pay for that. Without the clamp, all five infrastructure leaves score -alike; with it, the right one wins 5/5. +because a ranked list is worthless without one (§4.3). Ferret computes the score of the *null +hypothesis* (all root causes up), and over time obtains the distribution of +`Score(best prediction) − Score(null hypothesis)`. For a new set of observations the prediction is +declared significant only if that score difference **exceeds the median of that distribution by more +than one standard deviation** — not merely "beats the null by one standard deviation." The bar is the +median of the historical best-minus-null gap, plus one standard deviation. + +This topic's crate implements the `k = 1` case with a simpler scoring function — least-squares +residual on predicted front-end failure rates — and the detail that makes it work is worth noticing: +**clamping the fitted severity to `[0,1]`**. A severity is a probability, so a candidate that is +simply not on enough requests would need one above 1 to explain the observed rates, and the clamp is +what makes it pay for that. Without the clamp, all five infrastructure leaves score alike; with it, +the right one wins 5/5 (lane 2). + +Why it matters: the significance test is what separates "here is the most likely cause" from "there +is a cause worth paging someone about" — and getting its definition right (median + one std dev of +the *difference* distribution) is the difference between a calibrated alert and a noise generator. ### Step 8 — Discovering the graph in the first place -Sherlock has no service registry, so it infers dependencies from timing: "if accessing service B -depends on service A, then packets exchanged with A and B are likely to co-occur." The dependency -probability of A when accessing B is approximated as the conditional probability of accessing A -within a **dependency interval** — fixed at **10 ms** — before accessing B. +> **In:** everything above assumed an Inference Graph existed. +> **Out:** how Sherlock infers the dependency edges from packet timing (§4.1), the 10 ms interval +> trade-off, the chance-co-occurrence correction, and the deployment numbers that show it scales. + +Sherlock has no service registry, so it infers dependencies from timing (§4.1): "if accessing service +B depends on service A, then packets exchanged with A and B are likely to co-occur." The dependency +probability of A when accessing B is approximated as the conditional probability of accessing A within +a **dependency interval** — fixed at **10 ms** — before accessing B. -The trade is stated plainly: "Too large an interval will introduce false dependencies on services -that are accessed with a high frequency, while too small an interval will miss some true +The trade is stated plainly (§4.1): "Too large an interval will introduce false dependencies on +services that are accessed with a high frequency, while too small an interval will miss some true dependencies." And there is a chance-co-occurrence correction: with average interval `I` between -accesses to a service, the likelihood of accidental co-occurrence is estimated as `(10ms)/I`, and -only dependencies far above that are kept. +accesses to a service, the likelihood of accidental co-occurrence is estimated as `(10ms)/I`, and only +dependencies far above that are kept. -Deployment: 40 servers, 34 routers, 54 IP links, 2 LANs, three weeks, ~1,500 clients with agents on -23 of them. Agents report every 300 s; a per-host dependency graph is under 40 KB, so **10⁵ agents -would need about 10 Mbps** in aggregate. Localization complexity is "proportional to the number of -root causes in the inference graph × the graph depth", and depth is "less than 10 for all the -applications we have studied". +Deployment (§5–6): 40 servers, 34 routers, 54 IP links, 2 LANs, three weeks, ~1,500 clients with +agents on 23 of them. Agents report every 300 s; a per-host dependency graph is under 40 KB, so +**10⁵ agents would need about 10 Mbps** in aggregate. Localization complexity is "proportional to the +number of root causes in the inference graph × the graph depth", and depth is "less than 10 for all +the applications we have studied." + +Why it matters: the graph is the input to everything else, and Sherlock's willingness to *infer* it +from traffic — rather than demand a hand-maintained registry — is what made it deployable, and is +exactly the move this topic's lane 1 generator reverses to test the localizers. ## How to read the paper (with the concepts in hand) - **§1 + Figure 1.** The motivating incident and the *troubled* state. Read the definition twice. - **§3.1 + Figure 2.** The three node types on a worked example (a client fetching a file from a - network share, via Kerberos, via DNS, via routers). Trace one path from observation to root - cause yourself. -- **§3.1.1 + Figures 3–5.** The three meta-nodes and their truth tables. Derive one entry of - Figure 3 by hand; then read the NLB/25% argument for why selector must exist. + network share, via Kerberos, via DNS, via routers). Trace one path from observation to root cause + yourself. +- **§3.1.1 + Figures 3–5.** The three meta-nodes and their truth tables. Derive one entry of Figure 3 + by hand; then read the NLB/25% argument for why selector must exist. - **§3.1.2.** The `O(3ⁿ) → O(n)` reduction. Read the first product formula as a sentence in English. - **§3.2 + Algorithm 1.** Ferret. Observations 3.1 and 3.2 are the paper's real contribution; everything else is bookkeeping. -- **§4.1–4.1.2.** Dependency discovery, the 10 ms interval, chance co-occurrence, and aggregating - across similar clients. +- **§4.1.** Dependency discovery, the 10 ms interval, chance co-occurrence, and aggregating across + similar clients. - **§4.2–4.3.** Graph construction, the AT/AD escape hatch and its 0.001, and the two-Gaussian response-time scoring plus the significance test. - **§5–6.** Implementation and the production deployment (Figure 8's topology). - **After the paper.** Implement `sherlock_single_fault` in `rca.rs` and reproduce lane 2, then do - exercises 2 and 3 — k = 2 for simultaneous faults, and the selector meta-node. + exercises 2 and 3 — `k = 2` for simultaneous faults, and the selector meta-node. ## Questions to answer in notes.md @@ -209,34 +283,141 @@ applications we have studied". up/down model cannot express, using lane 1's numbers as the example. 2. Derive `P(Child = Troubled | Parent1 = Down, Parent2 = Troubled) = (1 − d₁) · d₂` from the noisy-max definition, in words. -3. Show concretely that a noisy-max node models a load balancer incorrectly: two servers, both - down, dependency probability 0.5 each. What does noisy-max give, and what should it be? +3. Show concretely that a noisy-max node models a load balancer incorrectly: two servers, both down, + dependency probability 0.5 each. What does noisy-max give, and what should it be? 4. Observation 3.1 turns `3^r` into `(2r)^k`. Compute both for `r = 200` and `k = 2`, and say what - assumption about incidents you are buying with that reduction — then name a failure mode where - the assumption is false (the paper names one). -5. The AT/AD pseudo-causes absorb model error at probability 0.001. Argue for and against making - that a tunable, given the paper's claim that results are insensitive to it. + assumption about incidents you are buying with that reduction — then name a failure mode where the + assumption is false (the paper names one). +5. The AT/AD pseudo-causes absorb model error at probability 0.001. Argue for and against making that + a tunable, given the paper's claim that results are insensitive to it. ## Done when -- [ ] You can name the three node types and the three meta-nodes, and say what each meta-node is - for. +Answer each before unfolding it. + +- [ ] You can name the three node types and the three meta-nodes, and say what each meta-node is for. + +
Answer + + Node types (§3.1): root-cause nodes (physical components that can fail — a computer, a service, a + router, an IP link), observation nodes (one per client-per-service measurement, the only thing + Sherlock actually sees), and meta-nodes (the glue that propagates state from causes to + observations). + + Meta-nodes (§3.1.1): noisy-max is the default AND-of-dependencies with a per-edge escape probability + `1−d`; selector models a load balancer, forcing `P(up)=0` when all backends are down (which + noisy-max gets wrong, assigning 25% up for two 0.5-weight down parents); failover models + primary/secondary, where the secondary only matters once the primary is fully down. + +
+ - [ ] You can define *troubled* and explain why two states are not enough. + +
Answer + + *Troubled* is when "servers or links continue to function but users perceive poor performance" + (§3.1) — degraded, not dead. A binary up/down model has nowhere to put it: the component answers + health checks (so it is "up") while users suffer, so a two-state model records it as healthy. + + Lane 1 is the numeric proof: the broken infra leaf is slow on 55% of calls but its own error rate + stays at the 0.0040 baseline, so a per-node up/down view ranks it 41st of 55 by error rate while + 34 of its callers alert. The third state is exactly what a model needs to represent "working but + hurting." + +
+ - [ ] You can state Observations 3.1 and 3.2 and the complexity each one buys. + +
Answer + + Observation 3.1 (§3.2): at any moment only a few root causes are abnormal, so Ferret evaluates only + assignment vectors with at most `k` abnormal nodes — `2r` with one, `2²·C(r,2)` with two, at most + `(2r)^k` overall. For `r=200, k=2` that is ~80,000 vectors (bound 160,000) instead of `3^200 ≈ + 10^95`, with error "vanishingly small for `k=4` onwards." The assumption fails under mass events + like rapid malware propagation (the paper's footnote). + + Observation 3.2 (§3.2): since most root causes are *up* in most vectors, only the descendants of the + abnormal nodes need re-evaluating. Ferret propagates the all-up state once, then recomputes only the + affected subtree per candidate — "roughly two orders of magnitude" faster. + +
+ - [ ] You can explain the two-Gaussian response-time score and the significance test. + +
Answer + + For a response-time observation `t`, Sherlock fits `Gaussian_up` and `Gaussian_troubled` to history + (Figure 1's example: means 200 ms and 2 s) and scores a candidate predicting `(p_up, p_troubled, + p_down)` as `p_up·Prob(t|Gaussian_up) + p_troubled·Prob(t|Gaussian_troubled)` (§4.3); the vector's + score is the product over all observations. + + The significance test: Ferret computes the null-hypothesis score (all root causes up) and, over + time, the distribution of `Score(best) − Score(null)`. A new prediction counts as significant only + if its best-minus-null difference **exceeds the median of that distribution by more than one + standard deviation** (§4.3) — the bar is median + 1 std dev of the historical gap, not simply "one + std dev above the null." + +
+ - [ ] You can describe how the dependency graph is discovered, and the 10 ms trade-off. + +
Answer + + With no registry, Sherlock infers edges from packet timing: if B depends on A, packets to A and B + co-occur, so the dependency probability of A given B is the conditional probability of an A access + within a fixed **10 ms** dependency interval before a B access (§4.1). Too large an interval invents + false dependencies on high-frequency services; too small a one misses real ones — hence a fixed, + tuned middle value. + + A chance-co-occurrence correction guards against coincidence: with mean inter-access interval `I`, + accidental co-occurrence is ~`(10ms)/I`, and only dependencies well above that survive (§4.1). At + deployment scale (§5–6) this stayed cheap: per-host graphs under 40 KB, ~10 Mbps for 10⁵ agents, + graph depth under 10. + +
+ - [ ] Your `rca.rs` reproduces lane 2: mean rank 1.0 against the baselines' 36.4 and 44.0. + +
Answer + + Lane 2 runs the graph-aware localizer against the two per-node baselines. The baselines put the true + cause at mean rank 36.4 (rank-by-failure-count) and 44.0 (rank-by-error-rate) — bottom half, exactly + the differential-observability failure. The Sherlock-style single-fault localizer recovers it at + mean rank 1.0 across the seeds. + + The load-bearing detail is clamping the fitted severity to `[0,1]`: a candidate that is not on + enough requests would need a severity above 1 to explain the observed failure rates, and the clamp + forces it to pay for that mismatch. Without the clamp all five infra leaves score alike; with it the + true cause wins 5/5. + +
+ - [ ] You wrote answers to all five questions in notes.md. +
Answer + + The five questions cover the transferable core: why the *troubled* state is irreducible (lane 1), + the noisy-max conditional derivation, the load-balancer counter-example that forces the selector + meta-node, the `3^r → (2r)^k` pruning and the malware-propagation regime where it fails, and whether + the AT/AD escape-hatch weight should be tunable given the paper's insensitivity claim. + + Answer them against the anchors above — §3.1 for the model, §3.1.1 for the meta-nodes, §3.2 for + Ferret's observations, §4.1–4.3 for discovery and scoring — not from memory. The recurring lesson is + Observation 3.1's: beat an exponential search with a fact about real inputs before reaching for a + cleverer algorithm. + +
+ ## References - Bahl, Chandra, Greenberg, Kandula, Maltz, Zhang. *Towards Highly Reliable Enterprise Network Services via Inference of Multi-level Dependencies.* SIGCOMM 2007 — [PDF](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/sherlock_sigcomm_07.pdf). + Section, figure and table citations in this chapter refer to this paper. - Kandula, Katabi, Vasseur. *Shrink: A Tool for Failure Diagnosis in IP Networks.* SIGCOMM MineNet 2005 — the two-level, two-state predecessor Ferret's approximation builds on. - Kim, Sumbaly, Shah. *Root Cause Detection in a Service-Oriented Architecture.* SIGMETRICS 2013 — MonitorRank, the random-walk alternative the crate's other stub implements. -- Local exercise stub: `topics/43-ops-dependency-graphs/experiments/rca.rs`. -- Topic 40 (attack graphs) — the same reasoning with the arrows reversed; topic 21 (formal methods) - — what it would take to verify a model like this rather than tune it. +- Local exercise stub: `topics/43-ops-dependency-graphs/experiments/src/rca.rs`. +- Topic 40 (attack graphs) — the same reasoning with the arrows reversed; topic 21 (formal methods) — + what it would take to verify a model like this rather than tune it.