docs(analytics): document transaction-commit-time metric by kriszyp · Pull Request #572 · HarperFast/documentation

kriszyp · 2026-07-07T15:47:38Z

Summary

Adds a Storage Metrics section documenting transaction-commit-time (write commit submit→durable duration distribution) and how its upper percentiles serve as the leading indicator for the storage overload (503) rejection.

Purpose

Companion docs for HarperFast/harper#1688 (harper#592).

Generated by Claude Opus 4.8.

Add a Storage Metrics section documenting transaction-commit-time (write commit submit->durable duration distribution) and how its upper percentiles serve as the leading indicator for the storage overload (503) rejection. Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-07T15:50:58Z

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-572

This preview will update automatically when you push new commits.

gemini-code-assist

Code Review

This pull request adds documentation for a new transaction-commit-time storage metric in reference/analytics/overview.md, detailing its distribution and its relationship with the overload guard configuration. The feedback suggests correcting a grammatical typo in the documented error message and adding backticks around a configuration parameter for markdown consistency.

github-actions · 2026-07-07T16:32:16Z

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-572

This preview will update automatically when you push new commits.

- Correct transaction-commit-time description: the recorder fires on both fulfilment and rejection (submit -> settle), not only on durable commit — a failed/retried commit still consumes queue time and should not be undercounted. - Restore the storage. prefix on maxTransactionQueueTime for consistency with its earlier mention. Co-Authored-By: Claude Opus <noreply@anthropic.com>

…tle-only Cross-model review (codex) flagged the original framing as overstating the metric's reliability as a leading indicator: it is RocksDB-only, records one sample per commit attempt (not per logical transaction — conflict retries each get their own sample), and only records once an attempt settles, so a wedged commit contributes nothing while still tripping the 503 guard. Co-Authored-By: Claude Opus <noreply@anthropic.com>

…ding - median/p90/p95/p99/p999 are only present on the per-minute aggregate (hdb_analytics); raw per-second entries only carry mean/distribution/count. - The overload guard tracks a commit it happened to arm on, not reliably the oldest outstanding one (retries/chained commits can wedge unarmed). - ASCII "to" instead of the arrow glyph per the content style guide. Co-Authored-By: Claude Opus <noreply@anthropic.com>

… signal - Add a transaction-commit-time entry to the v5.2 release notes so it's discoverable from the changelog. - Replace the vague "queue-depth or stuck-commit observability" pointer (no such metric is documented yet) with the actual overload-guard log line operators can watch today. Co-Authored-By: Claude Opus <noreply@anthropic.com>

…cy-592 # Conflicts: # release-notes/v5-lincoln/5.2.md

Cross-model review (codex + grok + harper-domain adjudication) found the initial framing overstated this metric's reliability as a leading indicator. Verified each claim against harper's DatabaseTransaction.ts and analytics/write.ts before applying: - RocksDB-only; never emitted for LMDB-backed databases. - One sample per commit attempt, not per logical transaction (conflict retries each record their own sample). - Only records once an attempt settles — a still-outstanding commit contributes nothing. - Raw entries (hdb_raw_analytics) carry mean/distribution/count only; percentiles are computed on the per-minute aggregate (hdb_analytics). Fixed the page's own raw-entry JSON example, which showed percentile fields that don't exist on raw rows. - The overload guard rejects record updates and publishes only — deletes and canonical-source writes (replication, caching sources) bypass it. - The guard tracks at most one outstanding commit per thread: a retry issued while the prior attempt still holds that slot is never armed (so a wedge there won't trip the 503); a later backoff-delayed retry re-arms fresh. - Added a v5.2 release-note entry and VersionBadge. Co-Authored-By: Claude Opus <noreply@anthropic.com>

recordCommitLatency wraps the async transaction.commit() path only; replayLogs.ts's directCommitSync() calls commitSync() directly and isn't instrumented, so replay/startup commits emit no sample either. Co-Authored-By: Claude Opus <noreply@anthropic.com>

github-actions · 2026-08-01T01:58:19Z

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-572

This preview will update automatically when you push new commits.

Ethan-Arrowood

Reviewed against the harper source at v5.2.0 (c11e0976) and origin/main. The metric name, emission path, units, the raw/aggregate field split, the 503 error string, and the bypass rules all check out — I verified each rather than assuming. Four things to fix, plus one sequencing decision shared with #573.

Sequencing with #573 — read this first. #573 documents the same transaction-commit-time metric in the same file, in the Resource Usage Metrics table, with a contradictory attribute list (mean, median, p90, p95, p99, p999 — i.e. exactly the raw/aggregate confusion this PR correctly fixes). I ran the trial merge: git merges the two cleanly, with zero conflicts, so nothing warns you. The merged page documents the metric in two different tables with contradictory guidance, and 5.2.md ends up with two ## Analytics H2s.

Merge order: this PR (#572) first, then #573. This PR should own transaction-commit-time — it has the purpose-built ### Storage Metrics section, its table shape (path/method/type) matches how the metric is actually emitted via recordAction() with no byThread flag, and it fixes the stale raw-entry example. #573 then rebases onto this, drops its transaction-commit-time row and duplicated prose, cross-links #storage-metrics, and folds its release note under the ## Analytics heading this PR creates. I've left the matching comments on #573.

One thing that is not a problem, so it doesn't get re-litigated: the Rejecting writes on this thread log string at line 201 is real. I grepped it at both v5.2.0 (DatabaseTransaction.ts:394) and origin/main (:485), and the source comment there confirms it fires once per stuck commit. Keep it.

Aggregate example left stale while the raw one was fixed (can't suggest inline — unchanged, so not in the diff). You correctly rewrote the raw example, but the aggregate example just below it (reference/analytics/overview.md lines ~107-120) still shows only median/mean/p95/p90, omitting the p99/p999 this PR's new prose tells operators to alert on. A reader who follows the guidance and looks at the example won't find the fields they were just told to query. Same drive-by class as the fix already in here.

Two smaller ones:

The alerting advice doesn't mention that the aggregate has a 60s floor (analytics.aggregatePeriod, default 60), which bounds how fast a percentile alert can fire. One clause would make the guidance actionable.
storage-tuning.md's storage.maxTransactionQueueTime section says "Harper rejects new writes with HTTP 503" and now has an observable signal to point at — worth adding the reverse link to #storage-metrics.

_{sent with Claude Opus 5}

Ethan-Arrowood · 2026-08-04T16:52:58Z

+`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on
+the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table
+for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard,


The percentile list reads as exhaustive, but the aggregator emits nine, not five (resources/analytics/write.ts:983-984: p1, p10, p25, median, p75, p90, p95, p99, p999). Since this sentence is specifically teaching the raw-vs-aggregate split, an incomplete list undercuts the point it's making.

Suggested change

`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on

the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table

for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard,

`distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`,

`p999`) are only available on the per-minute aggregate (`hdb_analytics`) once raw entries are rolled

up — query the aggregate table for percentile-based alerting. It shares a timebase with the RocksDB

storage engine's overload guard,

Ethan-Arrowood · 2026-08-04T16:52:58Z

+which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds
+`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on


Two things here. storage.maxTransactionQueueTime appears three times in this section as bare inline code with no link; it's documented in storage-tuning.md, and #573 already links it in exactly this relative form — worth matching on first mention.

Separately, "times only the commit attempt it arms" carries the same single-slot assumption as the paragraph below (see my other comment) — on main the guard times the oldest of all tracked outstanding commits. Neutral phrasing here is true on both.

Suggested change

which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds

`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on

which times outstanding commit attempts: when a tracked outstanding commit on a thread exceeds

[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime)

(default 45s), Harper rejects new record updates and publishes on

Ethan-Arrowood · 2026-08-04T16:52:58Z

+source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry
+issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff
+retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a
+later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt.


This documents a bug that harper has already fixed.

The single-slot description is an accurate read of v5.2.0 (let outstandingCommit, outstandingCommitStart; plus if (!outstandingCommit) arming). But 319714cf1 ("Track every outstanding write commit, not just one per thread", harper#2009) replaced the slot with a linked list: origin/main now holds oldestOutstandingCommit/newestOutstandingCommit, and the source comment names the single-slot behaviour as the defect — a wedged retry was invisible to checkOverloaded() forever. I confirmed 319714cf1 is not an ancestor of v5.2.0, so it ships in the next minor and this paragraph becomes false then.

I'd delete it rather than version-qualify it. <VersionBadge> means since, not only in, so there's no way to scope it — and it isn't operator-actionable either way, since nobody can observe or act on per-thread arming slots. Everything actionable in this section survives the cut.

Suggested change

source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry

issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff

retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a

later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt.

source) bypass this check.

Ethan-Arrowood · 2026-08-04T16:52:58Z

+load. But the metric shares only a timebase with the guard, not its population: a wedged commit
+contributes no sample until it settles, and an early retry that wedges can go untracked by the guard
+entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the
+server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and
+don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the
+threshold against a baseline for your workload.


Follow-on from the retry-slot paragraph above: this clause depends on it and is false on main for the same reason. Reflowed to drop it. The rest of the guidance is unchanged and holds on both v5.2.0 and main, including the Rejecting writes on this thread reference, which I verified exists.

Suggested change

load. But the metric shares only a timebase with the guard, not its population: a wedged commit

contributes no sample until it settles, and an early retry that wedges can go untracked by the guard

entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the

server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and

don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the

threshold against a baseline for your workload.

load. But the metric shares only a timebase with the guard, not its population: a wedged commit

contributes no sample until it settles. Don't treat this distribution as a leading indicator on its

own — watch the server log for the "Rejecting writes on this thread" error the guard emits when it

does trip, and don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`.

Tune the threshold against a baseline for your workload.

kriszyp mentioned this pull request Jul 7, 2026

feat(analytics): record write-transaction commit latency HarperFast/harper#1688

Merged

github-actions Bot temporarily deployed to pr-572 July 7, 2026 15:50 Inactive

gemini-code-assist Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread reference/analytics/overview.md Outdated

Comment thread reference/analytics/overview.md Outdated

fix(analytics): satisfy prettier table formatting

3408176

github-actions Bot temporarily deployed to pr-572 July 7, 2026 16:32 Inactive

kriszyp marked this pull request as ready for review August 1, 2026 00:09

kriszyp requested a review from a team as a code owner August 1, 2026 00:09

cb1kenobi reviewed Aug 1, 2026

View reviewed changes

Comment thread reference/analytics/overview.md Outdated

kriszyp and others added 7 commits July 31, 2026 18:38

Merge remote-tracking branch 'origin/main' into kris/txn-commit-laten…

05d974b

…cy-592 # Conflicts: # release-notes/v5-lincoln/5.2.md

github-actions Bot deployed to pr-572 August 1, 2026 01:58 View deployment

cb1kenobi approved these changes Aug 3, 2026

View reviewed changes

Ethan-Arrowood requested changes Aug 4, 2026

View reviewed changes

Ethan-Arrowood mentioned this pull request Aug 4, 2026

docs(analytics): document transaction queue depth metrics #573

Open

Navigation Menu

Uh oh!

docs(analytics): document transaction-commit-time metric - #572

docs(analytics): document transaction-commit-time metric#572
kriszyp wants to merge 9 commits into
mainfrom
kris/txn-commit-latency-592

kriszyp commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 1, 2026

Uh oh!

Ethan-Arrowood left a comment

Uh oh!

Ethan-Arrowood Aug 4, 2026

Uh oh!

Ethan-Arrowood Aug 4, 2026

Uh oh!

Ethan-Arrowood Aug 4, 2026

Uh oh!

Ethan-Arrowood Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds
		`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on

Uh oh!

Conversation

kriszyp commented Jul 7, 2026

Summary

Purpose

Uh oh!

github-actions Bot commented Jul 7, 2026

🚀 Preview Deployment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

🚀 Preview Deployment

Uh oh!

Uh oh!

github-actions Bot commented Aug 1, 2026

🚀 Preview Deployment

Uh oh!

Ethan-Arrowood left a comment

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants