feat: statistical range key-count estimation (getEstimatedKeyCount range support + CountEstimator) - #778
Open
kriszyp wants to merge 6 commits into
Open
feat: statistical range key-count estimation (getEstimatedKeyCount range support + CountEstimator)#778kriszyp wants to merge 6 commits into
kriszyp wants to merge 6 commits into
Conversation
Adds Database::EstimateCount — a no-iteration range key-count estimate built from RocksDB statistics: GetApproximateMemTableStats supplies the memtable entry count directly, and the SST portion converts approximate file bytes in range (GetApproximateSizes) to entries via the live-entry density of only the SSTs overlapping the range (GetPropertiesOfTablesInRange: (num_entries - num_deletions) / file bytes). Open-ended ranges subtract the complementary range from estimate-num-keys rather than passing an empty upper-bound slice (which would denote the smallest key). Public API: getEstimatedKeyCount(options?: RangeOptions) extends the existing whole-DB method with range support, and createCountEstimator() returns a CountEstimator that rides an iterator: advance(lastKey, n) checkpoints progress and estimate() returns the exact traversed count plus a remainder estimate calibrated by the observed actual/estimated ratio over the traversed portion, converging toward the exact total. Closes #205 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- guard inverted/empty bounded ranges (GetApproximateSizes would underflow end-start offsets in uint64) — returns 0 - honor exclusiveStart/inclusiveEnd by appending the bytewise-successor zero byte to the encoded bound - CountEstimator: exclude the cursor entry from the remainder (forward mode double-counted it, blocking convergence), add finish() as the completion signal, memoize estimate() per checkpoint, and document the caller-owned progress contract - temper the cost claims: scales with overlapping SSTs, table-property reads can do I/O for cold files, start-only ranges do complement work Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
napi can return a null data pointer for a zero-length buffer, which the previous guard read as an omitted bound — an empty end bound (below every key) became a whole-database estimate on the NativeDatabase surface (encodeKey shields the public API). Track presence explicitly: empty end returns 0, empty start is the minimum key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request implements a non-iterating key-count estimation feature for RocksDB ranges, exposing getEstimatedKeyCount with range options and introducing a new CountEstimator class to progressively refine estimates during iteration. The review feedback correctly identifies that CountEstimator currently discards the exclusiveStart and inclusiveEnd options from CountEstimatorOptions, and suggests storing and utilizing these options in the estimate() method to ensure bounds are correctly respected.
Comment on lines
+48
to
+63
| export class CountEstimator { | ||
| #store: Store; | ||
| #start: Key | Uint8Array | undefined; | ||
| #end: Key | Uint8Array | undefined; | ||
| #reverse: boolean; | ||
| #cursor: Key | Uint8Array | undefined; | ||
| #traversed = 0; | ||
| #finished = false; | ||
| #memoized: number | undefined; | ||
|
|
||
| constructor(store: Store, options?: CountEstimatorOptions) { | ||
| this.#store = store; | ||
| this.#start = options?.start; | ||
| this.#end = options?.end; | ||
| this.#reverse = options?.reverse ?? false; | ||
| } |
Contributor
There was a problem hiding this comment.
The CountEstimator constructor currently discards the exclusiveStart and inclusiveEnd options from CountEstimatorOptions. This causes the estimator to ignore these bounds during initial estimation and subsequent range segment queries, leading to inaccurate count estimates. We should store these options as private fields.
Suggested change
| export class CountEstimator { | |
| #store: Store; | |
| #start: Key | Uint8Array | undefined; | |
| #end: Key | Uint8Array | undefined; | |
| #reverse: boolean; | |
| #cursor: Key | Uint8Array | undefined; | |
| #traversed = 0; | |
| #finished = false; | |
| #memoized: number | undefined; | |
| constructor(store: Store, options?: CountEstimatorOptions) { | |
| this.#store = store; | |
| this.#start = options?.start; | |
| this.#end = options?.end; | |
| this.#reverse = options?.reverse ?? false; | |
| } | |
| export class CountEstimator { | |
| #store: Store; | |
| #start: Key | Uint8Array | undefined; | |
| #end: Key | Uint8Array | undefined; | |
| #exclusiveStart: boolean; | |
| #inclusiveEnd: boolean; | |
| #reverse: boolean; | |
| #cursor: Key | Uint8Array | undefined; | |
| #traversed = 0; | |
| #finished = false; | |
| #memoized: number | undefined; | |
| constructor(store: Store, options?: CountEstimatorOptions) { | |
| this.#store = store; | |
| this.#start = options?.start; | |
| this.#end = options?.end; | |
| this.#exclusiveStart = options?.exclusiveStart ?? false; | |
| this.#inclusiveEnd = options?.inclusiveEnd ?? false; | |
| this.#reverse = options?.reverse ?? false; | |
| } |
Comment on lines
+98
to
+113
| if (this.#cursor === undefined) { | ||
| return this.#store.estimateCount({ start: this.#start, end: this.#end }); | ||
| } | ||
| if (this.#memoized !== undefined) { | ||
| return this.#memoized; | ||
| } | ||
|
|
||
| // The cursor entry itself belongs to the traversed side, so the | ||
| // remainder excludes it in both directions (an inclusive lower bound | ||
| // forward would count it twice and block convergence). | ||
| const traversedRange = this.#reverse | ||
| ? { start: this.#cursor, end: this.#end } | ||
| : { start: this.#start, end: this.#cursor, inclusiveEnd: true }; | ||
| const remainingRange = this.#reverse | ||
| ? { start: this.#start, end: this.#cursor } | ||
| : { start: this.#cursor, end: this.#end, exclusiveStart: true }; |
Contributor
There was a problem hiding this comment.
Update the estimate() method to utilize the stored exclusiveStart and inclusiveEnd options when querying estimateCount for the initial estimate, the traversed range, and the remaining range. This ensures that the bounds are correctly respected throughout the estimation process.
if (this.#cursor === undefined) {
return this.#store.estimateCount({
start: this.#start,
end: this.#end,
exclusiveStart: this.#exclusiveStart,
inclusiveEnd: this.#inclusiveEnd,
});
}
if (this.#memoized !== undefined) {
return this.#memoized;
}
// The cursor entry itself belongs to the traversed side, so the
// remainder excludes it in both directions (an inclusive lower bound
// forward would count it twice and block convergence).
const traversedRange = this.#reverse
? { start: this.#cursor, end: this.#end, inclusiveEnd: this.#inclusiveEnd }
: { start: this.#start, end: this.#cursor, exclusiveStart: this.#exclusiveStart, inclusiveEnd: true };
const remainingRange = this.#reverse
? { start: this.#start, end: this.#cursor, exclusiveStart: this.#exclusiveStart }
: { start: this.#cursor, end: this.#end, exclusiveStart: true, inclusiveEnd: this.#inclusiveEnd };
Contributor
📊 Benchmark Resultsget-sync.bench.tsgetSync() > random keys - small key size (100 records)
getSync() > sequential keys - small key size (100 records)
ranges.bench.tsgetRange() > small range (100 records, 50 range)
realistic-load.bench.tsRealistic write load with workers > write variable records with transaction log
transaction-log.bench.tsTransaction log > read 100 iterators while write log with 100 byte records
Transaction log > read one entry from random position from log with 1000 100 byte records
worker-put-sync.bench.tsputSync() > random keys - small key size (100 records, 10 workers)
worker-transaction-log.bench.tsTransaction log with workers > write log with 100 byte records
Results from commit 71f44b5 |
Per review feedback on the API shape: a bare number hides how much an
estimate should be trusted. New db.estimateCount(options?) returns
{ count, confidence }; getEstimatedKeyCount() reverts to its original
no-arg number signature (kept as the cheap estimate-num-keys alias), so
one name no longer covers two cost profiles. CountEstimator.estimate()
returns the same shape.
confidence is a heuristic [0,1], exactly 1 only when the count is exact
(finish(), inverted/empty-by-construction ranges). Computed natively
from the estimate components: resolution (SST data-block / memtable
sampling granularity relative to the count), tombstone fraction of the
overlapping SSTs, and for start-only ranges the error compounded by
complement subtraction. Measured on 500k varied entries: 0.999 on
full/half ranges (~3% error), 0.88 at 1%, 0.21 on a 50-key range (~2x
over-report), 0.13 on a start-only tail (+39% — complement subtraction
correctly distrusted); estimator confidence converges to 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adjudicated major from the API-shape review: a failed GetApproximateSizes/
GetPropertiesOfTablesInRange silently degraded to a memtable-only count
while the confidence formula still reported it as trustworthy, and a
failed estimate-num-keys property read returned { 0, 1.0 } — a missing
answer dressed as a confidently empty database. Track degradation in
RangeEstimate (capping confidence at 0.1) and return { 0, 0 } for the
failed property read. Also: guard null table-properties entries, honor
the range own exclusiveStart/inclusiveEnd flags in CountEstimator
segments, and cap non-exact estimator confidence at 0.999 so only
finish() and exact-by-construction ranges claim 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…degrade Follow-ups from the delta review: a successful zero estimate-num-keys read now reports 0.95 confidence (deletion entries can offset puts, so even zero is estimated), and a null entry in the table-properties collection marks the density degraded rather than being silently skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kriszyp
marked this pull request as ready for review
August 13, 2026 18:28
cb1kenobi
reviewed
Aug 14, 2026
| /** | ||
| * Estimates are statistical (block-granular SST approximation + memtable | ||
| * skip-list approximation), so assertions use a tolerance factor rather than | ||
| * exact bounds. Uniform fixed-size entries keep the real accuracy well inside |
Member
There was a problem hiding this comment.
Low: core design claim (range-local density under varying entry sizes) is untested
The stated advantage of this approach over #311 is that range-local table properties keep the byte-to-count density honest when entry sizes vary across the keyspace. Every test here uses uniform fixed-size entries (this comment says so explicitly), so the one scenario the design exists to handle has no CI coverage: if the SST density computation regressed to a whole-CF mean (the #311 behavior), these tests would still pass.
Suggested fix: add a case that writes large entries across one key sub-range and small entries across another, flush, then assert estimateCount over each sub-range tracks that sub-range's actual count (within the existing 2x factor). A whole-CF mean would skew both toward the average and fail.
—
Generated by Barber AI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a statistical range key-count estimate that never iterates, for query planning and pagination reporting (HarperFast/harper#2147). Closes #205, and supersedes the earlier attempt in #311.
Native
estimateCount(start?, end?)combines three RocksDB primitives:GetApproximateMemTableStats— returns an entry count for the memtable portion directly.GetApproximateSizes(files only, 10% error margin) — approximate on-disk bytes covered by the range.GetPropertiesOfTablesInRange— per-SSTnum_entries/num_deletions/ block sizes for only the SSTs overlapping the range, giving a range-local live-entry density ((entries − deletions) / file bytes) that converts bytes → count.Using range-local table properties (instead of #311's whole-CF cached mean with write-count invalidation) keeps the density honest when entry sizes vary across the keyspace and needs no cache or invalidation hooks. Open-ended ranges are estimated as
estimate-num-keysminus the complementary range — an empty slice is the smallest key, so it must never be passed as an upper bound (a correctness bug in #311's open-ended path). Inverted, empty, and zero-length bounds are guarded (they would otherwise underflow RocksDB's uint64 offset subtraction).Public API (shape chosen by Kris: estimates carry a trust signal):
db.estimateCount(options?: RangeOptions)→{ count, confidence }(CountEstimate).confidenceis a heuristic 0–1 trust indicator, exactly 1 only when the count is exact — derived natively from the estimate's resolution (data-block/memtable-sampling granularity relative to the count), the tombstone fraction of overlapping SSTs, complement-subtraction error for start-only ranges, and failed statistics calls (a degraded estimate caps at 0.1; a failedestimate-num-keysread returns{0, 0}, not a confident empty).exclusiveStart/inclusiveEndare honored via the bytewise-successor zero byte.db.getEstimatedKeyCount()— unchanged original no-arg signature (cheapestimate-num-keysalias), so existing callers are untouched and one name doesn't cover two cost profiles.db.createCountEstimator(options?)→CountEstimator— rides an iterator:advance(lastKey, count)checkpoints progress (e.g. once per page),estimate()returns{count, confidence}= exact traversed + remainder calibrated by the observed actual/estimated ratio (clamped 8×), memoized per checkpoint; confidence is the exactness-weighted blend, capped at 0.999 untilfinish()declares the traversal complete (then exact with confidence 1). Supportsreverseand the range bound flags.For the human reviewer
confidenceis now API surface: callers will encode thresholds against it, so retuning the formula changes their behavior (review ledger point). The semantics doc deliberately promises only "heuristic ordering signal, 1 = exact" — the formula itself is not contract.estimateCount()no-bounds usesestimate-num-keyswhile a bounded full range uses bytes×density; they can disagree. Deliberate — the no-bound path stays O(1) and matchesgetEstimatedKeyCount().CALIBRATION_MIN_TRAVERSED = 16, 8× clamp) are judgment calls; options can be added later.advance()trusts the caller (monotonic cursors, no double-reporting); a wrapping-iterator variant would be additive.Verification
test/estimate-count.test.ts(12 tests): flushed / memtable-only / mixed ranges, open-ended both sides, empty DB (confident 0), inverted range (exact 0), zero-length native bounds, uncommitted-transaction exclusion, monotonic scaling with range width, estimator refinement (confidence must increase), reverse iteration,finish()exactness, and a paginated loop driven to completion (pre-finish confidence < 1, exact total afterfinish()). Full suite green at every commit (latest: 767 passed / 2 skipped, 56 files).[value, primaryKey]keys throughRocksIndexStore), width-ordered estimates confirmed.Review coverage
Generated by Claude (Fable 5). Cross-model pre-push review via
prepush-review.mjs, six rounds:1c4be8d): Codex (graded) + Gemini + Harper-domain adjudication — cursor-composer failed (output-format rejection), cursor-grok pruned. Fixed: inverted-range guard,exclusiveStart/inclusiveEnd, estimator forward off-by-one,finish(), memoization, tempered cost claims.4481bc4): Codex (resumed) + Gemini. Fixed: zero-length end bound bypassing the guard on the native surface (napi nullptr for empty buffers).0ddaf72): Codex + Gemini — hardening confirmed clean.b2cb53a, API change): Codex (graded) + Gemini + Harper-domain — cursor-composer failed again, grok pruned. Adjudicated major fixed in round 5: failed statistics reported as confident estimates.0548d42): Codex (Gemini leg returned no output this round). Follow-ups fixed in round 6: a successful zeroestimate-num-keysclaimed exactness; partial null table-properties collections didn't degrade.89d437b): Codex + Gemini — verdict COMMENTS, both fixes confirmed, no new findings.Accepted, not changed (with rationale): silent-degrade carries low confidence instead of an event; sync-on-JS-thread cost model (documented); caller-owned progress contract; calibration reads current state so concurrent writes behind the cursor can swing a checkpoint (inherent to statistical estimates on a live database, bounded by the 8× clamp); no native GoogleTest for the estimator math (it lives in an N-API translation unit, which cannot link into the gtest target — the vitest suite covers it end-to-end through the real binding).
Human-Review-Need: 4 @ 89d437b