adapter: add internal subscribes for reading a mutation's input - #37922
Open
aljoscha wants to merge 1 commit into
Open
adapter: add internal subscribes for reading a mutation's input#37922aljoscha wants to merge 1 commit into
aljoscha wants to merge 1 commit into
Conversation
This was referenced Jul 29, 2026
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 07:14
85ff499 to
864d924
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 09:35
864d924 to
a722968
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 09:46
a722968 to
9358755
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 10:57
9358755 to
87d0459
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 11:44
87d0459 to
4a3266b
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
July 29, 2026 12:19
4a3266b to
628c455
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
2 times, most recently
from
July 30, 2026 06:02
5acb075 to
988d42f
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
2 times, most recently
from
July 30, 2026 06:30
07fe1b8 to
4027fbc
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
August 5, 2026 06:16
4027fbc to
71eb277
Compare
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
August 5, 2026 12:13
71eb277 to
033c9d0
Compare
A read-then-write sequenced off the coordinator needs to observe its selection as it changes, which is what a subscribe already does. What it must not do is appear to be one: an internal subscribe writes no `mz_subscriptions` row and does not move the active-subscribes gauge, so introspection keeps showing only the subscribes a user started. Creation is fallible. The dataflow is optimized against a catalog snapshot on the session task and shipped when the coordinator handles the message, and a `DROP` of a dependency can land in that window. The coordinator's own read-then-write path optimizes and ships in one turn of its loop and has no such window, which is why it can treat creation as infallible. Here it is a `ConcurrentDependencyDrop` for the session to report, the same answer the frontend peek path gives. Shipping precedes registering the sink so a failure has nothing to unwind, which costs nothing because an internal subscribe's registration writes no introspection row. Read holds are passed through the stages rather than kept in the connection-keyed map, so concurrent operations on one connection cannot interfere with each other's holds, and they are released only once the dataflow is installed. Callers arrive with the next commit. Until they do, a `dead_code` allowance on the handlers keeps the crate warning-free, and it goes away with them.
aljoscha
force-pushed
the
aljoscha/occ-05-internal-subscribes
branch
from
August 5, 2026 13:05
033c9d0 to
ba6bd61
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part 5 of 7 in a stack that moves
DELETE,UPDATEandINSERT ... SELECToff the coordinator onto the session task, using optimistic concurrency
control. Design doc:
20260210_incremental_occ_read_then_write.md(lands in part 6).
A read-then-write sequenced off the coordinator needs to observe its
selection as it changes, and stream the mutation's diffs rather than peek
every matched row and recompute them. A subscribe already does exactly that.
What it must not do is appear to be one.
Closes SQL-591
Description
An internal subscribe writes no
mz_subscriptionsrow and does not move theactive-subscribes gauge, so introspection keeps showing only the subscribes a
user started.
Creation is fallible, and that is the substantive difference from the
coordinator's own read-then-write path. Here the dataflow is optimized
against a catalog snapshot on the session task and shipped when the
coordinator handles the message, and a
DROPof a dependency can land inthat window. The coordinator optimizes and ships in one turn of its loop and
has no such window, which is why it can treat creation as infallible. So this
returns
ConcurrentDependencyDropfor the session to report, the same answerthe frontend peek path already gives. Shipping precedes registering the sink,
so a failure has nothing to unwind, which costs no concurrency because an
internal subscribe's registration writes no introspection row.
Read holds are passed through the stages rather than kept in the
connection-keyed
txn_read_holdsmap, so concurrent operations on oneconnection cannot interfere with each other's holds, and they are released
only once the dataflow is installed.
Temporary scaffolding. Nothing calls these handlers until part 6, so a
dead_codeallowance keeps the crate warning-free. Part 6 removes it in thesame commit that adds the callers.
Verification
No caller yet, so this part is carried by the tests in parts 6 and 7. Part 6
includes a test that a dependency dropped underneath a running read-then-write
produces a retryable error rather than a coordinator panic.