Skip to content

docs(analytics): document transaction-commit-time metric #572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kriszyp wants to merge 9 commits into
base: main
Choose a base branch
from
+42 −3
Open

docs(analytics): document transaction-commit-time metric #572

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
38b14bb
docs(analytics): document transaction-commit-time metric
kriszyp Jul 7, 2026
3408176
fix(analytics): satisfy prettier table formatting
kriszyp Jul 7, 2026
daf8ab9
docs(analytics): address PR #572 review feedback
kriszyp Aug 1, 2026
1771f0a
docs(analytics): qualify transaction-commit-time as RocksDB-only, set…
kriszyp Aug 1, 2026
f2fd67a
docs(analytics): fix raw/aggregate field split and overload-guard wor…
kriszyp Aug 1, 2026
c2abfaf
docs(analytics): add 5.2 release note, point to the real stuck-commit…
kriszyp Aug 1, 2026
05d974b
Merge remote-tracking branch 'origin/main' into kris/txn-commit-laten…
kriszyp Aug 1, 2026
d90edac
docs(analytics): fix accuracy issues found by independent review
kriszyp Aug 1, 2026
a4cca36
docs(analytics): note the metric is absent during transaction-log replay
kriszyp Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions reference/analytics/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,8 @@ Example raw entry:
"metric": "bytes-sent",
"path": "search_by_conditions",
"type": "operation",
"median": 202,
"mean": 202,
"p95": 202,
"p90": 202,
"distribution": [202],
"count": 1
},
{
Expand Down Expand Up @@ -169,6 +167,41 @@ Harper automatically tracks the following metrics for all services. Applications
| `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication |
| `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time |

### Storage Metrics

<VersionBadge version="v5.2.0" />

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- |
| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt |

`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not
emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during
transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a
transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the
number of logical writes. A sample is only recorded once an attempt settles, so a commit that is
still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`,
`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on
the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table
for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard,
Comment on lines +184 to +186

@Ethan-Arrowood Ethan-Arrowood Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The percentile list reads as exhaustive, but the aggregator emits nine, not five (resources/analytics/write.ts:983-984: p1, p10, p25, median, p75, p90, p95, p99, p999). Since this sentence is specifically teaching the raw-vs-aggregate split, an incomplete list undercuts the point it's making.

Suggested change
`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on
the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table
for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard,
`distribution`, and `count`; percentiles (`p1`, `p10`, `p25`, `median`, `p75`, `p90`, `p95`, `p99`,
`p999`) are only available on the per-minute aggregate (`hdb_analytics`) once raw entries are rolled
up — query the aggregate table for percentile-based alerting. It shares a timebase with the RocksDB
storage engine's overload guard,

which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds
`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on
Comment on lines +187 to +188

@Ethan-Arrowood Ethan-Arrowood Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here. storage.maxTransactionQueueTime appears three times in this section as bare inline code with no link; it's documented in storage-tuning.md, and #573 already links it in exactly this relative form — worth matching on first mention.

Separately, "times only the commit attempt it arms" carries the same single-slot assumption as the paragraph below (see my other comment) — on main the guard times the oldest of all tracked outstanding commits. Neutral phrasing here is true on both.

Suggested change
which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds
`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on
which times outstanding commit attempts: when a tracked outstanding commit on a thread exceeds
[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime)
(default 45s), Harper rejects new record updates and publishes on

that thread with `Outstanding write transactions have too long of queue, please try again later`
(HTTP 503) — deletes and writes applied from a canonical source (e.g. replication or a caching
source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry
issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff
retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a
later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt.
Comment on lines +191 to +194

@Ethan-Arrowood Ethan-Arrowood Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This documents a bug that harper has already fixed.

The single-slot description is an accurate read of v5.2.0 (let outstandingCommit, outstandingCommitStart; plus if (!outstandingCommit) arming). But 319714cf1 ("Track every outstanding write commit, not just one per thread", harper#2009) replaced the slot with a linked list: origin/main now holds oldestOutstandingCommit/newestOutstandingCommit, and the source comment names the single-slot behaviour as the defect — a wedged retry was invisible to checkOverloaded() forever. I confirmed 319714cf1 is not an ancestor of v5.2.0, so it ships in the next minor and this paragraph becomes false then.

I'd delete it rather than version-qualify it. <VersionBadge> means since, not only in, so there's no way to scope it — and it isn't operator-actionable either way, since nobody can observe or act on per-thread arming slots. Everything actionable in this section survives the cut.

Suggested change
source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry
issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff
retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a
later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt.
source) bypass this check.


A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large
transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write
load. But the metric shares only a timebase with the guard, not its population: a wedged commit
contributes no sample until it settles, and an early retry that wedges can go untracked by the guard
entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the
server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and
don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the
threshold against a baseline for your workload.
Comment on lines +198 to +203

@Ethan-Arrowood Ethan-Arrowood Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-on from the retry-slot paragraph above: this clause depends on it and is false on main for the same reason. Reflowed to drop it. The rest of the guidance is unchanged and holds on both v5.2.0 and main, including the Rejecting writes on this thread reference, which I verified exists.

Suggested change
load. But the metric shares only a timebase with the guard, not its population: a wedged commit
contributes no sample until it settles, and an early retry that wedges can go untracked by the guard
entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the
server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and
don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the
threshold against a baseline for your workload.
load. But the metric shares only a timebase with the guard, not its population: a wedged commit
contributes no sample until it settles. Don't treat this distribution as a leading indicator on its
own — watch the server log for the "Rejecting writes on this thread" error the guard emits when it
does trip, and don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`.
Tune the threshold against a baseline for your workload.


### Resource Usage Metrics

| `metric` | Key attributes | Other | Unit | Description |
Expand Down
6 changes: 6 additions & 0 deletions release-notes/v5-lincoln/5.2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,9 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t
### Web Application Firewall

Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview).

## Analytics

### `transaction-commit-time` metric

A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics).