Skip to content

adapter: add internal subscribes for reading a mutation's input - #37922

Open
aljoscha wants to merge 1 commit into
aljoscha/occ-04-group-commitfrom
aljoscha/occ-05-internal-subscribes
Open

adapter: add internal subscribes for reading a mutation's input#37922
aljoscha wants to merge 1 commit into
aljoscha/occ-04-group-commitfrom
aljoscha/occ-05-internal-subscribes

Conversation

@aljoscha

@aljoscha aljoscha commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Motivation

Part 5 of 7 in a stack that moves DELETE, UPDATE and INSERT ... SELECT
off the coordinator onto the session task, using optimistic concurrency
control. Design doc:
20260210_incremental_occ_read_then_write.md
(lands in part 6).

A read-then-write sequenced off the coordinator needs to observe its
selection as it changes, and stream the mutation's diffs rather than peek
every matched row and recompute them. A subscribe already does exactly that.
What it must not do is appear to be one.

Closes SQL-591

Description

An internal subscribe writes no mz_subscriptions row and does not move the
active-subscribes gauge, so introspection keeps showing only the subscribes a
user started.

Creation is fallible, and that is the substantive difference from the
coordinator's own read-then-write path. Here the dataflow is optimized
against a catalog snapshot on the session task and shipped when the
coordinator handles the message, and a DROP of a dependency can land in
that window. The coordinator optimizes and ships in one turn of its loop and
has no such window, which is why it can treat creation as infallible. So this
returns ConcurrentDependencyDrop for the session to report, the same answer
the frontend peek path already gives. Shipping precedes registering the sink,
so a failure has nothing to unwind, which costs no concurrency because an
internal subscribe's registration writes no introspection row.

Read holds are passed through the stages rather than kept in the
connection-keyed txn_read_holds map, so concurrent operations on one
connection cannot interfere with each other's holds, and they are released
only once the dataflow is installed.

Temporary scaffolding. Nothing calls these handlers until part 6, so a
dead_code allowance keeps the crate warning-free. Part 6 removes it in the
same commit that adds the callers.

Verification

No caller yet, so this part is carried by the tests in parts 6 and 7. Part 6
includes a test that a dependency dropped underneath a running read-then-write
produces a retryable error rather than a coordinator panic.

@linear-code

linear-code Bot commented Jul 29, 2026

Copy link
Copy Markdown

SQL-591

@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch from 4a3266b to 628c455 Compare July 29, 2026 12:19
@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch 2 times, most recently from 5acb075 to 988d42f Compare July 30, 2026 06:02
@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch 2 times, most recently from 07fe1b8 to 4027fbc Compare July 30, 2026 06:30
@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch from 4027fbc to 71eb277 Compare August 5, 2026 06:16
@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch from 71eb277 to 033c9d0 Compare August 5, 2026 12:13
A read-then-write sequenced off the coordinator needs to observe its
selection as it changes, which is what a subscribe already does. What it
must not do is appear to be one: an internal subscribe writes no
`mz_subscriptions` row and does not move the active-subscribes gauge, so
introspection keeps showing only the subscribes a user started.

Creation is fallible. The dataflow is optimized against a catalog snapshot
on the session task and shipped when the coordinator handles the message,
and a `DROP` of a dependency can land in that window. The coordinator's
own read-then-write path optimizes and ships in one turn of its loop and
has no such window, which is why it can treat creation as infallible. Here
it is a `ConcurrentDependencyDrop` for the session to report, the same
answer the frontend peek path gives. Shipping precedes registering the
sink so a failure has nothing to unwind, which costs nothing because an
internal subscribe's registration writes no introspection row.

Read holds are passed through the stages rather than kept in the
connection-keyed map, so concurrent operations on one connection cannot
interfere with each other's holds, and they are released only once the
dataflow is installed.

Callers arrive with the next commit. Until they do, a `dead_code`
allowance on the handlers keeps the crate warning-free, and it goes away
with them.
@aljoscha
aljoscha force-pushed the aljoscha/occ-05-internal-subscribes branch from 033c9d0 to ba6bd61 Compare August 5, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant