perf(rivetkit): reduce internal sqlite round trips

[NathanFlurry]

• Batch remote SQLite statements through one versioned envoy request while preserving transaction rollback semantics.
• Remove internal KV pagination and batch due one-shot claims to reduce actor-to-engine round trips.
• Use bounded history pruning and a partial running-history index to reduce SQLite page reads and storage.
• Cover query plans, cold-cache page fetches, batch protocol compatibility, migration recovery, and schedule claim boundaries.

[NathanFlurry]

Stack for rivet-dev/rivet

• #5444 perf(rivetkit): reduce internal sqlite round trips 👈
• #5443 test(rivetkit): verify core SQL query efficiency
• #5427 fix(rivetkit): clean up cron history efficiently
• #5439 docs(rivetkit): remove redundant overdue schedule note
• #5440 docs(rivetkit): use scheduling code snippets
• #5442 fix(rivetkit): configure inspector protocol versions
• #5441 feat(rivetkit): add actor scheduling
• main

Get stack: forklift get 5444
Push local edits: forklift submit
Merge when ready: forklift merge 5444

_{change upkwotmo}

[NathanFlurry]

Folded into #5443.

[claude]

Review

Reviewed the diff across the three areas this PR touches: envoy-protocol v6 versioning, rivetkit-core SQLite/schedule logic, and pegboard-envoy batched execution.

Overview

Adds a batched remote-SQLite request/response to the envoy protocol (v5 to v6), removes internal KV-list pagination in favor of caller-supplied limits, batches due one-shot schedule claims, and adds a partial index (WHERE result = 0) plus bounded pruning for schedule history. Good breadth of new tests (query plans, cold-cache fetches, protocol compat, migration recovery, claim boundaries).

Findings

1. Possible in-place edit of an already-shipped migration (rivetkit-rust/packages/rivetkit-core/src/actor/internal_schema.rs)
   The new _rivet_schedule_history_running partial index is added by editing MIGRATIONS[1] (the v1 to v2 step) directly, rather than appending a new migration entry, and INTERNAL_SCHEMA_VERSION stays at 6 (unchanged by this diff). The comment directly above MIGRATIONS in this same file states: "Rewriting these entries in place is safe only while no internal schema version has shipped; after release, all changes must be appended as new migrations and INTERNAL_SCHEMA_VERSION must advance." Since ensure_internal_schema short-circuits when current_version equals INTERNAL_SCHEMA_VERSION (around line 174), any actor whose DB is already at version 6 will never replay migration index 1 and will keep the old non-partial index forever, silently missing the storage/page-read win this PR is meant to deliver, with no test covering upgrade from an existing version 6. If schema version 6 has already been released to any deployed actor, this should be a new migration entry with INTERNAL_SCHEMA_VERSION bumped to 7 instead.

2. Batched-transaction handle can be left poisoned after a double failure (engine/packages/pegboard-envoy/src/ws_to_tunnel_task.rs, around lines 1759 through 1786)
   The new batch path wraps statements in BEGIN/COMMIT/ROLLBACK. If COMMIT fails and the follow-up ROLLBACK also fails, the error is correctly surfaced, but the cached NativeDatabaseHandle for that (actor_id, generation) is never evicted; remove_remote_sqlite_executor_generation and clear_remote_sqlite_executors are cfg(test)-only (around lines 1944 through 1963). All subsequent remote-SQL requests for that actor generation would then reuse the same possibly-mid-transaction connection until the generation changes. This is a new failure mode introduced by moving from per-statement autocommit to an explicit multi-statement transaction. Also, no test in pegboard-envoy or engine/packages/engine/tests/envoy exercises the mid-batch-failure/rollback path itself; the rollback-semantics claim in the PR description isn't directly verified by a test.

3. Minor: dropped scaling regression test (rivetkit-rust/packages/rivetkit-core/tests/sql_efficiency.rs)
   The previous schedule.prune_global_under_cap case (asserting no wasted work when under the retention cap) appears removed with no replacement. The query plan is still covered by the catalog explain-plan assertion, so the "every modified SQL needs a plan test" rule is technically satisfied, but the specific scaling-property regression check is gone.

4. Minor: unbounded KV list without limit (rivetkit-rust/packages/rivetkit-core/src/actor/internal_storage.rs)
   Removing internal pagination means ListOpts.limit = None now runs a single unbounded query instead of paging internally. This looks like an intentional, documented tradeoff (per the updated sql_efficiency.rs comment), but there's no hard cap protecting callers that omit limit on a very large KV namespace. Worth confirming this is the intended contract.

Checks that came back clean

• v5.bare untouched; v6.bare added as a genuinely new schema (byte-identical copies in engine/sdks/rust/envoy-protocol/schemas and engine/sdks/schemas/envoy-protocol), consistent with the "never edit a published .bare schema" rule.
• v5_to_v6.rs and v6_to_v5.rs do explicit field-by-field conversion for every type; no serde_bare::to_vec/from_slice round-trip shortcuts. The v6 to v5 downgrade path correctly errors with incompatible(...) for the new batch variants instead of silently dropping data, and this is exercised by remote_sql_batch_requires_v6.
• No raw f64 in the new v6.bare (uses fixed 8-byte SqliteValueFloat).
• Batched one-shot schedule claims execute all chunked deletes inside one transaction, preserving prior all-or-nothing semantics.
• RECOVER_HISTORY_SQL hardcodes result = 0 as a literal (not a bound parameter) so the planner can pick the new partial index, a good, easy-to-miss detail done correctly.
• No Mutex-wrapped HashMap or RwLock-wrapped HashMap, and no std::sync violations introduced; no forbidden wildcard fallthrough arm on the new protocol enums; new logging uses structured fields.
• Batch size (256 statements) and aggregate bind-param bytes (128 KiB) are bounded at the pegboard-envoy trust boundary, consistent with treating envoy to pegboard-envoy traffic as untrusted; per-statement/total SQL text length is unbounded, but that is pre-existing, not new in this PR.
• No new lease/lock/fencing mechanism added; batching reuses the existing per-(actor_id, generation) cache, consistent with the single-writer invariant.

Suggestions

• Confirm whether schema version 6 has shipped; if so, convert the schedule-history index change into a new migration (version 7) rather than editing MIGRATIONS[1] in place.
• Add an eviction path (even non-test-gated) for the cached NativeDatabaseHandle when both COMMIT and the recovery ROLLBACK fail, so a generation is not permanently stuck.
• Consider a regression test for the mid-batch-failure rollback path in pegboard-envoy to directly verify the "rollback semantics preserved" claim from the PR description.