TypeScript SDK: WebSocket frames are delivered out of order under compression, silently corrupting the client cache

[adlaika]

Describe the bug

WebsocketDecompressAdapter.onmessage binds an async handler directly to ws.onmessage, and nothing sequences concurrent invocations:

https://github.com/clockworklabs/SpacetimeDB/blob/ba8c8c2269533e5e96a5730ea9274d2a6b0b8007/crates/bindings-typescript/src/sdk/websocket_decompress_adapter.ts

set onmessage(handler: (msg: { data: Uint8Array }) => void) {
  this.#ws.onmessage = async (msg: MessageEvent) => {
    let data;
    try {
      data = await this.#decompress(new Uint8Array(msg.data));
    } catch (e) { /* ... close ... */ return; }
    handler({ data });   // <-- delivery order = decompression completion order
  };
}

When two frames are in flight, both invocations enter the handler and both await #decompress. Whichever finishes first is delivered first. Decompression time scales with payload size, so a small frame routinely overtakes a larger one that arrived before it, and DbConnection applies transaction updates out of order.

Row updates are delete+insert on a primary key and do not commute. Applying TX2 before TX1 leaves the row holding TX1's older value. Nothing repairs it: the subscription is not re-evaluated, so the row stays wrong until some unrelated later write happens to touch it. The client cache silently diverges from the server with no error, no onDisconnect, and no reconnect.

The user-visible symptom in our app is a chart frozen on a stale forecast while the server holds the correct value the whole time.

Two notes on why this is not already fixed:

• The compression: 'none' path is unaffected. #decompress is awaited unconditionally, but for tag 0 it returns without performing async work, so its continuation runs before the next message event can fire and delivery stays ordered. gzip is the default, so the defect is on by default.
• The #inboundQueue drain loop added to DbConnectionImpl in 2.8.x does not fix this. That queue preserves the order in which frames arrive at the handler, and the race is upstream of it — frames reach it already reordered. It addresses reentrancy during synchronous processing, which is a different problem.

This is the same function as #5667, which fixed the rejection path (the try/catch + close now in 2.8.1) and left the ordering race in place.

It is also distinct from the older closed reports it resembles: this is purely client-side (no reconnect is involved, unlike #4944), and the server's send order is correct — verified by a fresh subscription returning the right value immediately after each miss — so it is not #996. #1973 is the same class of symptom in the Rust SDK, not this code path.

This violates a documented guarantee

I want to be precise that this is not a request for a new guarantee. Subscription semantics states:

Each database transaction generates exactly zero or one update message sent to clients. These updates are atomic and reflect the exact order of committed transactions.

The client cache always maintains a consistent and correct subset of the committed database state.

The second sentence is the one being broken. After a reordered pair the cache holds a value that is not the committed state and never becomes it, so it is neither consistent with nor a correct subset of the committed database state — indefinitely, and silently.

One clarification to head off a reasonable objection. The same page says:

No relative ordering guarantees are made regarding the invocation order of these callbacks.

That disclaims the order of on_insert / on_delete / on_update invocations within a single transaction update. This report is about the order in which whole transactions are applied to the cache, which the page explicitly does guarantee ("the exact order of committed transactions") and which the disclaimer does not cover. The bug is not that callbacks fire in a surprising order; it is that the resulting cache state is wrong.

For what it is worth, the underlying mechanism is a known JavaScript hazard rather than anything exotic — an async handler assigned to ws.onmessage reorders messages by their resolution time even though TCP delivered them in order. Sitong Peng's "WebSockets guarantee order — so why are my messages scrambled?" (May 2025) describes the identical shape with await blob.arrayBuffer() in place of await this.#decompress(...).

Evidence

I replaced the factory via withWSFn with a byte-faithful replica of the adapter (same URL construction, same v2.bsatn.spacetimedb protocol, same decompress implementation) plus one counter. An inversion = a frame delivered to the SDK after a frame that arrived on the socket later than it did.

Workload is identical in every row: one module, a burst of 7 un-awaited reducer calls that each cause a delete+insert of the same projection_result row, 20 or 40 iterations. A miss = the client never converges to the value the server already holds (verified independently with spacetime sql) within a 4s budget.

arm	frames	inversions	misses
gzip, quiet machine	163	20	0/20
none, quiet machine	163	0	0/20
none + injected 0–25ms per-frame delay	166	55	3/20
gzip, 8 CPU burners (SDK 2.0.3)	330	58	7/40
none, 8 CPU burners (SDK 2.0.3)	323	0	0/40

Three independent claims:

1. gzip reorders; none does not. 163 frames in both quiet arms — identical workload, one flag changed, 20 inversions vs 0.
2. Reordering alone is sufficient to corrupt the cache. The jitter arm keeps compression: 'none' (provably ordered) and injects only a variable per-frame delay — precisely what await decompress contributes. Misses appear.
3. Load is not the cause. Under identical CPU load, gzip misses 7/40 while none misses 0/40 with zero inversions.

Every miss had the same signature: the client stuck on the pre-update value, the server correct throughout, and any subsequent write to that row repairing the client in 25–209ms.

Upgrading 2.0.3 → 2.8.1 does not change the rate: 8/40 on 2.8.1 vs 7/40 on 2.0.3 under identical load.

Chaining delivery on a promise queue takes the same configuration — gzip, same load, same burst — to 0/40. (That figure is from the simpler variant that chains decode and delivery; the exact variant suggested below, which keeps decompression concurrent, is the one verified by the standalone repro.)

The standalone repro linked below reproduces all of this without any load trick, and takes the fix from 15/20 stale to 0/20 by patching the published package in place.

Steps to reproduce

Runnable repro: https://github.com/adlaika/stdb-ts-frame-ordering-repro — one public table, one delete+insert reducer, ~100 lines of client. No CPU-load trick needed.

cd module
spacetime build
spacetime publish ordering-repro --server local
spacetime generate --lang typescript --out-dir ../client/module_bindings --module-path .

cd ../client
npm install
npx tsx repro.ts        # gzip (the default) -> 15/20 stale
npx tsx repro.ts none   # control            -> 0/20 stale

node apply-suggested-fix.mjs && npx tsx repro.ts   # fix applied -> 0/20 stale

All three are bare commands at stock defaults (BURST=8, PADDING_BYTES=2000000, ITERATIONS=20); nothing extra needs setting to get those numbers.

Each iteration writes to one primary key 8 times, concurrently and un-awaited. Writes 1–7 carry ~2 MB of highly compressible padding — tiny on the wire, expensive to inflate; write 8 carries empty padding and is trivial to inflate. The server commits in order so the row always ends at 8, but the client settles on 7 and stays there:

iter 0: MISS  client=7  fresh-subscription=8  <- server holds 8, the client's stream lost the update

client=7 is the tell: the cheap final frame was applied before the heavy one preceding it. The run uses the stock SDK — no withWSFn, no instrumentation — so it is not a harness artefact, and the fresh subscription on each miss shows the server was correct throughout.

apply-suggested-fix.mjs patches node_modules/spacetimedb/dist/index.mjs in place with the fix below, so it can be verified against the published package rather than a reimplementation. --revert restores it.

One note if it does not reproduce for you first try: the padding must be compressible. Decompression cost scales with output size, not wire size — incompressible padding makes a large frame that gzip stores in raw blocks and that inflates at nearly memcpy speed, so the timing gap never opens. Rate also rises with anything widening timing variance between frames (CPU load, payload spread, latency); on a quiet machine with a small burst it can sit at 0 for many iterations, which makes this easy to mistake for absent.

Expected behavior

Frames are delivered to DbConnection in the order they were received on the socket, so that transactions are applied in commit order and the cache stays "a consistent and correct subset of the committed database state" as documented. Out-of-order delivery of non-commuting deltas produces a wrong state rather than an early one, and the frame cannot be applied before it is decompressed either way.

Suggested fix

Serialise delivery without serialising the decompression itself — start each frame's inflate on arrival, and chain only the hand-off:

set onmessage(handler: (msg: { data: Uint8Array }) => void) {
  let tail: Promise<void> = Promise.resolve();
  this.#ws.onmessage = (msg: MessageEvent) => {
    const pending = this.#decompress(new Uint8Array(msg.data));
    // Mark the rejection handled now: the chain may not reach this frame for
    // several ticks, and without this an inflate failure surfaces as an
    // unhandledrejection in the meantime. The real error still reaches .catch.
    pending.catch(() => {});
    tail = tail
      .then(async () => handler({ data: await pending }))
      .catch((e) => {
        console.error('[SpacetimeDB] WebSocket decompress failed, closing socket:', e);
        this.#ws.close();
      });
  };
}

Frames still inflate concurrently — including off-thread, where DecompressionStream does that work — so this costs no decompression parallelism; only the order results are handed to the SDK changes. The .catch keeps one bad frame from poisoning the chain and stalling the stream, and preserves #5667's close-on-failure behaviour (the close now fires at that frame's position in the chain rather than immediately).

A chain does retain each frame's buffer until its turn, but the current code has no backpressure either, and the retained compressed buffers are smaller than the concurrent inflations they replace.

Environment

• spacetimedb npm 2.0.3 and 2.8.1 — both reproduce, at the same rate
• Server: standalone 2.8.0, localhost
• Node 22.23.1 (global WebSocket). The browser bundles (dist/index.browser.mjs, dist/sdk/index.browser.mjs) contain the identical handler, the identical awaited decompress, and the same gzip default, so the browser is exposed by the same code path; all the numbers above were measured under Node.