perf(crdt): calibrate the memory estimate instead of re-exporting per write

[farhan-syah]

Summary

estimated_memory_bytes answers "how big is this document?" by serialising the whole document. nodedb-lite calls it from update_memory_stats, which runs at the end of every write path — document.rs, document_batch.rs, batch.rs (×4), vector.rs (×2), graph.rs — and on the health route. So every operation re-encodes the entire CRDT document to update a pressure counter.

Measured on a 25k-row document (4.1 MB encoded):

one measurement, cold	26.2 ms
one measurement, document unchanged	1.05 µs
one measurement, after a single write	101 ms

The middle row is the memoisation added in #231, which is why reads on an idle document are no longer affected. The last row is what remains: a write moves the version, the memo correctly misses, and the full export runs again. At 25k writes that is roughly 42 minutes of pure re-encoding, and it scales with document size — the store in #230 is 77 MB.

This is the second fault in #230. That issue reports a document which, once the import ceiling is raised, opens but then pegs a core on every read with flat RSS and zero I/O, and reads it as "a tight loop, not lazy index construction". Flat RSS is equally consistent with this: the export's Vec is allocated and dropped per call, so a recycled transient allocation is indistinguishable from no allocation at that resolution. The reporter's own throughput figure — 25,120 records in ~4 hours, 0.57 s/record — is this cost, not storage.

Change

The export is now used to calibrate rather than to answer.

len_ops is an inlined oplog counter, and it counts operations still sitting in an open transaction, so it tracks a write the moment it happens rather than when it commits (pinned by a test, since the estimate is wrong the day that stops holding). The answer is that counter scaled by a measured bytes-per-operation ratio. A real export runs only when the operation count has halved or doubled since the ratio was taken, which bounds the export cost to O(1) amortised per write instead of O(document).

The return value is still bytes, so the governor thresholds built on it keep their meaning.

Same workload, after:

	before	after
25k writes, each followed by a measurement	~42 min	373 ms (14.9 µs/write)
full exports performed	25,000	14
final estimate vs real encoded size	exact	0.995×

Exactness is preserved where it is free — a document that has not changed since it was measured returns the measured bytes — and interpolated otherwise, which is what a pressure signal needs.

Tests

• estimated_memory_does_not_re_encode_on_every_write — 2000 writes, each followed by a measurement, must cost fewer than 32 real exports. This is the property the change exists for, so it is asserted rather than left to a benchmark.
• estimated_memory_stays_close_to_the_real_encoded_size — interpolation may drift, but must stay within 2× of the truth or the thresholds built on it mean nothing.
• oplog_version_counts_uncommitted_operations (existing) — pins the Loro property the counter depends on.
• estimated_memory_follows_compaction / estimated_memory_grows_with_data (existing) — still exact after compaction, still monotonic with data.

Validation

• cargo nextest run -p nodedb-crdt — 163/163
• cargo nextest run -p nodedb --all-features — 7938/7938
• cargo clippy --all-targets --all-features -p nodedb-crdt -- -D warnings — clean
• cargo fmt --all — clean

On #230

This addresses the Origin half of fault 2. It has not been verified against the reporter's preserved 32 GB store, and "explains every symptom recorded" is not the same as "is the sole cause" — the reproduction in #230 has not been run here. The reporter offered to run instrumented builds against that store, which is the acceptance path before #230 closes.

The lite side still calls the capped CrdtState::import on restore, and NodeDB-Lab/nodedb-lite#10 covers the flush tick, which is the same root cause at a different call site.