feat(payload): positional-cursor /check paging; remove the spool

[andinux]

Summary

Replace the server-side download spool with a positional cursor on
cloudsync_payload_chunks, so the /check job can page a tenant's change window
one chunk per round-trip with no temporary table. This is the extension half of the
change; the companion server PR is sqliteai/cloudsync#52.

This branch also carries three client send-path fixes found while benchmarking
the change — see Send-path fixes.

Why

The spool stages a whole /check window into a cloudsync_payload_spool table on
the tenant, then pages it out. That has three operational costs this PR removes:

1. Privilege — spool_fill issues CREATE TABLE, which the tenant's default
   readwrite apikey is not authorized for → failures.check: database_permission_denied
   (the original bug; the bench polled forever). Positional needs no DDL.
2. Disk — the spool temporarily ~doubles tenant DB space per requesting client.
   Positional writes nothing to the tenant.
3. Ops — no provisioning / migration / dbadmin-fallback to keep the spool table
   creatable.

How it works

cloudsync_payload_chunks gains an optional positional cursor:

• inputs resume_db_version, resume_seq, resume_frag_offset — start the scan
  at (db_version, seq) inclusive and re-enter a mid-value fragment at a byte offset;
• outputs next_db_version, next_seq, next_frag_offset, is_final — where the
  emitted chunk stopped.

The server pages with … LIMIT 1, threading each chunk's next_* back as the next
resume_* under a pinned until watermark, until is_final. Tiling is exact —
(db_version, seq) is a unique total order, so each change is emitted in exactly one
chunk with no overlap and no reliance on changes-level idempotency. The first fetch
uses the legacy exclusive since (drain-start "exclusive-after" rule).

The /check client wire protocol is unchanged — the positional cursor is
server-internal (job ↔ node); the client still pages by an integer artifact cursor.
The positional cursor itself touches no client code; the send-path fixes below are a
separate, unrelated change that does modify network.c.

Feature comparison

	positional cursor (new)	spool (old)
Tenant CREATE TABLE privilege	none	required (the bug)
Tenant disk during /check	none	~doubles per client
Provisioning / migration	none	spool table must exist
Node-side state	none (cursor is the job's loop var)	spool rows per stream
Client wire protocol	unchanged	unchanged
Drain cost	O(N²) (see below)	O(N)

Performance (test/chunk_bench.c, local SQLite, end-to-end paging)

window (multi-version)	spool O(N)	positional O(N²)	ratio
100 chunks	164 ms	872 ms	5.3×
200 chunks	350 ms	3,439 ms	9.8×

Chunk counts/bytes match exactly (positional is correct, just slower); the ratio
doubles as the window doubles ⇒ positional is O(N²). Root cause: each resume runs
… FROM cloudsync_changes WHERE (db_version,seq) ≥ cursor ORDER BY db_version, seq,
and cloudsync_changes has no (db_version, seq) index, so it re-scans + re-sorts
the window every call.

Accepted trade-off: /check is an async background job, so the O(N²) is node CPU
on cold-start/large windows — fine for typical windows and mitigatable. The real O(N)
fix is a follow-up (indexed (db_version, seq) seek on cloudsync_changes);
test/chunk_bench.c is kept as the guard to confirm it flattens the curve.

Changes

• cloudsync_payload_chunks positional cursor — SQLite vtab (cloudsync_sqlite.c)
  and PostgreSQL SRF (cloudsync_postgresql.c + SQL signatures). Legacy callers
  (send path) unchanged; columns appended, positional branch only active when
  resume_db_version is bound.
• Removed cloudsync_payload_spool table + spool_fill/spool_drop/drop_chunk
  on both engines, the SQLite Payload Download Spool unit test, and the spool
  sections of the PG chunk test.
• New tests on both engines: byte-for-byte tiling incl. a mid-fragment resume, plus
  an end-to-end drain→apply→content-hash round-trip.
• test/chunk_bench.c — local positional-drain benchmark / O(N²)→O(N) guard.

Send-path fixes (independent of the positional cursor)

Three pre-existing bugs in the unreleased 1.1 chunked send path, surfaced while
benchmarking. No CHANGELOG entries (introduced and fixed within the same unreleased
version).

1. Send announces the covered window, not the raw change range.
   cloudsync_network_send_changes announced each chunk's actual [db_version_min, db_version_max] to /apply. When the local db_version clock has run ahead of the
   site's own changes — after merging applied remote changes, or any transaction that
   produced no local-site change — the skipped versions are absent from the sent
   ranges, so the server's per-site coverage (ComputeGaps walks it from version 1)
   reports them as gaps and never advances lastOptimisticVersion to the local
   version; the send then churns and never reaches synced. Now it announces the
   covered window [min(running_lo, chunk_min) … chunk_max], running_lo chained
   from send_checkpoint+1, tiling [send_checkpoint+1 … watermark] contiguously.
   The min(…, chunk_min) clamp keeps min ≤ max for consecutive chunks that share
   a db_version (a value fragmented across chunks); a partial-fragment failure stays
   safe because the receiver stages incomplete fragments (0 applied rows), so coverage
   is recorded only once a value is whole.

2. lastOptimisticVersion taken latest-valid, not monotonic-max.
   network_sync_state_update_from_response kept the maximum optimistic/confirmed
   version across a multi-chunk send. The server moves the optimistic version
   backward when a later chunk fails (rollback), and that value becomes the durable
   send checkpoint — so the max masked the rollback and advanced the checkpoint past
   rolled-back changes (silent data loss). Now it takes the latest valid (>= 0)
   value, matching how gaps is already handled.

3. Client fail-fast on a non-retryable failures.check — surfaces a permanent
   error immediately instead of polling until timeout.

New network-layer unit-test harness (make network-unittest, test/network_unit.c),
built with networking compiled in so network.c's internal functions run on in-memory
buffers with no server — the layer previously had none. Plus an e2e repro
test_send_gap_from_clock_hole that forces a clock hole with cloudsync_db_version_next()
and asserts serverVersion reaches localVersion (red before fix 1, green after).

Verification

• SQLite unit suite passes; PG chunk tests (52/53/54/55) pass on PostgreSQL 17 with
  the spool removed.
• make network-unittest passes (version fold, announce-window tiling — leading
  hole / inter-chunk hole / same-version fragment — and status computation).
• Full basic e2e suite green against a live server, including the new
  test_send_gap_from_clock_hole repro.

Follow-ups (not blocking)

• Indexed (db_version, seq) seek on cloudsync_changes → positional drain O(N).
• /check job-duration metric/alert for pathological large windows.

🤖 Generated with Claude Code