feat(crdt): separate local reload from peer admission

[mkhairi]

Summary

A document this library writes can grow past what it will read back.

export_snapshot has no size bound. Import has three, and until now every import went through them — including the ones where the bytes came from this process rather than from a peer: restoring a document from durable storage, folding a shallow snapshot back in during compaction, rolling a transaction back to a pre-image, and seeding the validation candidate on every authenticated apply. Once a document's snapshot exceeds DEFAULT_MAX_IMPORT_BYTES or its history exceeds DEFAULT_MAX_IMPORT_OPS, that document can no longer be opened. It also can no longer be compacted, because compact_history and compact_at_version re-admit their own output through the same ceiling — so the one operation that would bring it back under the limit is gated by the limit. There is no recovery path from inside the library; raising max_bytes only moves the wall to max_encoded_operations.

The limits themselves are correct for what they were written for: bounding how much work an untrusted peer can make a document do. They just answer a different question when applied to bytes we produced ourselves. This splits the two so each keeps its own answer — the peer path stays capped, and the local path keeps every structural check while dropping the size ceilings.

Context in #230. The embedded-side half of that investigation — flush rewriting a full snapshot per tick — is a nodedb-lite change and is open as NodeDB-Lab/nodedb-lite#10; the two PRs share no files.

Changes

feat(crdt): admit locally-generated imports without the peer ceilings

admit_import splits into admit_structure — the authenticated metadata decode and the rejection of regressing per-peer ranges — and the three ceiling checks layered on top of it. import_local runs the structure half and keeps the pending-dependency check; the compaction paths use it. The peer path is unchanged, and import_with_limits remains the knob for tuning it.

It is a separate entry point rather than an unbounded CrdtImportLimits constructor, so bytes off the wire cannot reach the exempt path by passing a value.

perf(crdt): stop materialising the document to answer cheap questions

collection_names called get_deep_value, materialising every row and field of every collection to read the top-level keys — O(document) for a list whose length is the number of collections, called per query during name resolution. The shallow value carries the same keys.

estimated_memory_bytes exports a full snapshot as its size proxy. That is the honest proxy, but callers poll it to decide when to compact and an unwritten document returns the same number every time, so it is now computed once per version.

fix(crdt): admit local-snapshot reloads without the peer ceilings

Two things the first two commits left unfinished.

The Origin callers were still capped. import_local reached only the two compaction paths, so the failure this PR describes stayed live where it does the most damage. restore_collection_snapshot reloads a pre-image this process exported when the transaction opened: past the ceiling a collection could be written but never rolled back, and the transaction driver was handed a failure it has no way to act on. apply_committed_delta_authenticated seeds its validation candidate from current.export_snapshot() on the line above, on every authenticated apply: past the ceiling every incoming delta returns Malformed, leaving the collection silently unwritable and blaming the sender for it.

Both now go through CrdtState::from_local_snapshot, a constructor pairing new with import_local. Both sites were assembling those two steps by hand, and hand-assembly is exactly where the wrong import gets picked — a caller reaching for "load a state from our own snapshot" now finds an API that is already correct.

The memory estimate outlived the document it measured. Compaction does not mutate a document, it replaces one — and a shallow snapshot preserves the version vector, because peers have to keep delta-syncing across it. A cache keyed on the version alone therefore still looked current after the bytes it measured were gone. Measured on a document with 512 operations: 1791 bytes reported against 329 actual, and it never moves again. Since the documented use is "poll this to decide when to trigger compact_history()", a caller would compact, see no change, and compact forever.

Fixed by ownership rather than by a reset line. DocumentCell holds the document together with everything derived from it; reads go through Deref so no call site changes, but self.doc = new_doc no longer type-checks and replace — which drops the cache with the document — is the only way to swap it. A future compaction path cannot forget. A failed export is also no longer cached, which would otherwise have pinned a document at "empty" until its next write.

compact_history and compact_at_version were duplicating the whole of compaction; they now share compact_to_frontiers.

Tests

• local_import_admits_what_the_peer_ceilings_would_reject — the peer path still rejects, the local path admits the same bytes
• local_import_still_rejects_malformed_metadata — dropping the ceilings does not drop the structural checks
• from_local_snapshot_loads_without_the_peer_ceilings / from_local_snapshot_rejects_malformed_bytes — the constructor reloads our own snapshot and keeps every structural check
• collection_names_lists_top_level_keys_only — pins what the shallow read must return
• estimated_memory_follows_compaction — fails on the pre-fix code with the 1791-vs-329 stale value
• oplog_version_counts_uncommitted_operations — pins the Loro property the estimate's cache key depends on: the version advances when an operation is written, not when its transaction commits. The cache is wrong the day this stops holding, so it is asserted rather than assumed.
• estimated_memory_grows_with_data (existing) — covers cache invalidation on the write path

The behaviour past the default ceilings is not covered. Reaching them needs a document over a gigabyte or over a million operations, and a fixture that size does not belong in the suite. A round-trip test against explicit tight limits was written and dropped: it passed with import_local wired to the capped path, so it asserted nothing about the change.

Validation

• cargo nextest run -p nodedb-crdt — 161/161
• cargo nextest run -p nodedb --all-features — 7938/7938
• cargo nextest run -p nodedb-cluster-tests --all-features --retries 0 — 322/323. The one failure, a4_placement_convergence::placement_converges_to_min_rf_when_node_count_exceeds_rf, is vShard placement with no CRDT involvement; it took 31.7s before failing under full-suite load and passes standalone in 2.4s, with the suite's usual single retry disabled.
• cargo clippy --all-targets -- -D warnings — clean across the workspace
• cargo fmt --all — clean
• cargo check --workspace --all-targets — clean

Follow-up

The embedded side does not benefit until its restore path calls import_local — it currently goes through CrdtState::import and stays capped. That is a one-line change in nodedb-lite, blocked on a release carrying import_local. The store preserved in #230 is the acceptance case for both and will be exercised once that lands; it is kept until then.