From e0236ea7271ffd1eddc26aee663de3c70ba8618b Mon Sep 17 00:00:00 2001 From: rob Date: Fri, 15 May 2026 14:54:27 +0300 Subject: [PATCH 01/40] Add initial prompt --- .../sync-service/subquery-index-prompt.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 packages/sync-service/subquery-index-prompt.md diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md new file mode 100644 index 0000000000..2d7d9b6cdb --- /dev/null +++ b/packages/sync-service/subquery-index-prompt.md @@ -0,0 +1,152 @@ +# Shared Subquery Views with Logical-Time Reads + +## Summary + +Electric v1.6 introduced per-shape subquery indexing so consumers can keep +boolean subquery shapes live while dependency rows move across `WHERE` +boundaries. That solved correctness, but it made memory scale with the number of +shape consumers. Each consumer can keep its own materialized dependency view in +the `SubqueryIndex`, and while move-ins are buffered it can also hold both a +before and after view. + +This RFC proposes replacing per-consumer materialized subquery views with one +shared, versioned view per subquery. Consumers do not copy the view. +Instead, they read the shared view at a logical time. + + +## Background + +Related implementation work: + +- Commit: https://github.com/electric-sql/electric/commit/a04b25962cdb7ca86c4434585b6f74c758e1a31b +- PR: https://github.com/electric-sql/electric/pull/4051 +- issue: https://github.com/electric-sql/electric/issues/4279 +- Current index: `packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex` + +The v1.6 work lets shapes with boolean combinations around subqueries remain +live when dependency rows move. The key correctness problem is that consumers can +temporarily disagree about a subquery's membership while one consumer has +processed a move and another has not. + +The current implementation handles that by letting each shape consumer seed and +update exact per-shape membership rows. That keeps each consumer correct, but it +duplicates the same view across many shapes. During move-in buffering, the +consumer also carries before and after views so it can convert buffered +transactions and build the move-in query. + +## Problem + +The memory problem is broader than value-keyed routing rows in `SubqueryIndex`. +There are at least two duplicated memory pools: + +1. `SubqueryIndex` membership and routing rows, currently keyed by shape. +2. Consumer/materializer views, including before and after views during active + move-in buffering. + +Adding a reverse index such as `shape_handle -> all values` would make removal +faster, but it would increase memory. + +## Goals + +- Reduce memory footprint of subqueries significantly while remaining consitant and performant +- have near O(1) performance for: + - subquery addition and removal + - row processing by the where clause filter (so for afffected_shapes in the SubqueryIndex) +- Store one shared materialized view per subquery. +- Support exact membership reads at separate logical times. +- Preserve positive, negated, AND, OR, and NOT subquery correctness from v1.6. + +## Non-Goals + +- Do not change the client wire protocol. + +## Proposal + +### Components + +#### SubqueryIndex.MultiTimeView + +The MultiTimeView is an Materialized view of a subquery, queryable at multiple points in time. + +It's implimented as an ETS table (one ETS table per stack_id) + +subquery_id, value -> list(times) + +the meaning of the result: + doesn't exist - the value is not a member of the subquery for all logical times + [] - the value is a member of the subquery for all logical times + [:out, 9] - the value was out of the set before 9 and in the set from time 9 and above + [:out, 9, 11] - the value was out of the set before 9 and in the set from 9 to 10 and out the set again from time 11 and above + [:in, 9] - the value was in of the set before 9 and out the set from time 9 and above + [:in, 9, 11] - the value was in of the set before 9 and out the set from 9 to 10 and in the set again from time 11 and above + +note: the list(times) structure above has been chosen for memory efficientcy, but if you can think of a smaller structure let me know. for example if `[]` takes up more space than `true` then we should use `true` since this will be the most common case and we want to be memory efficient. + +so for subquery_id, value - [:in, 9, 11] + +member?(subquery_id, value, time: 8) = true +member?(subquery_id, value, time: 9) = false +member?(subquery_id, value, time: 10) = false +member?(subquery_id, value, time: 11) = true + + +rather than specifying a time you can also ask for membership across all times: + +member_of_union?(subquery_id, value) = true +member_of_intersection?(subquery_id, value) = false + +These are useful for the where clause filter which needs to keep the filter broad enough so that all consumers get all the changes they need while they may be at any of the logical times. + +For each subquery there will be a minimum logical time needed (the minimum in-flight logical time for the subquery) which the SubqueryProgressMonitor will set on the MultiTimeView. This allows the MultiTimeViewETS table to be compacted for memory and performace efficientcy. For any given list(times) it can be compacted by removing times from before the minimum in-flight logical time, making sure to update the :in/:out marker at the beginning of the list appropriately or removing it if there are no times left. + +Compacting should happen: +- when the list is read (e.g. when member? for the value is called) +- when the list is written to (e.g. when a value is moved in or out) +- when an async compaction routine is run (the design of this will need to be discussed) + +Removing a subquery should not involve a full ETS table scan as this will be too slow with lots of subqueries. If the ETS table is orderd we should be able to find the first item for the subqery, delete that, then find the next, and continue until the whole subquery is gone. That means it scales with the number of values (which is acceptable) rather than the number of subqueries. + + +#### SubqueryIndex + +This is a complete re-write of the existing SubqueryIndex that delegates most of it's resposibility to the the MultiTimeView. +It initialised the MultiTimeView. + +The Subquery index will need to know if a shapes use of a subquery is positive or negative (as it does now) and use: +- member_of_union? for positive +- member_of_intersection? for negative +This will ensure that the rows are included for all available times + +If the MultiTimeView has not been populated by the Materializer yet, the SubqueryIndex should just widen as much as possible. + +#### Materializer + +This is the existing Materializer. It will just need to be updated to: +- populate the MultiTimeView when the Materializer has initialised (it has a full materialized view). This should be at logical time 0. +- increment logical time for each `{:materializer_changes` message it sends to outer consumers, and include the new logical time in that message +- before the `{:materializer_changes` message is sent, the MultiTimeView should be updated with the changes giving the new logical time as the time of the change + +#### Logical Time + +Logical Time is monotonically incrementing counter per subquery. + +This needs to be a memory efficient data staructure that can be incremented indefinately. If it needs to wrap we need to make sure we use appropriate conparison functions when comparing times. Wrapping is an acceptable solution since there will only ever be so many moves in flight for any given subquery and memory would explode due to that before wrapping would cause comparison failures. + +#### SubqueryProgressMonitor + +Pin worked out from acks +-LRU algorithm + +#### Consumer EventProcessors + +These should be updated so that rather than holding views of the subquery, they just hold the logical time. so the before and after views should instead just be the before and after logical times. +- `convert_change` should have a function passed to it that access MultiTimeView.member? at the specified time +- the move-in query needs entire views at specific times and so should call MultiTimeView.get(time) and care should be made to not keep this in memory for too long, perhaps we should GC the consumer process afterwards, or perhaps the task process that runs the query should call MultiTimeView.get(time) so that the memory is freed when the process ends + + + +Please: +- read the codebase to check this design would work +- ask any clarifying questions about anything you are unsure of about my proposal +- give me your evaluation of the design +- make suggestions to improve the design for memory, performance or simplicity reasons From 95e74c53d917e71604a9fcd778edff2fef65240d Mon Sep 17 00:00:00 2001 From: rob Date: Fri, 15 May 2026 15:33:08 +0100 Subject: [PATCH 02/40] Update subqueryIndex's role --- .../sync-service/subquery-index-prompt.md | 35 ++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md index 2d7d9b6cdb..4f812161ed 100644 --- a/packages/sync-service/subquery-index-prompt.md +++ b/packages/sync-service/subquery-index-prompt.md @@ -89,11 +89,10 @@ member?(subquery_id, value, time: 9) = false member?(subquery_id, value, time: 10) = false member?(subquery_id, value, time: 11) = true - rather than specifying a time you can also ask for membership across all times: -member_of_union?(subquery_id, value) = true -member_of_intersection?(subquery_id, value) = false +member_at_some_time?(subquery_id, value) = true +member_at_all_times?(subquery_id, value) = false These are useful for the where clause filter which needs to keep the filter broad enough so that all consumers get all the changes they need while they may be at any of the logical times. @@ -110,14 +109,34 @@ Removing a subquery should not involve a full ETS table scan as this will be too #### SubqueryIndex This is a complete re-write of the existing SubqueryIndex that delegates most of it's resposibility to the the MultiTimeView. -It initialised the MultiTimeView. -The Subquery index will need to know if a shapes use of a subquery is positive or negative (as it does now) and use: -- member_of_union? for positive -- member_of_intersection? for negative +When a shape is added to the SubqueryIndex at a particular node in the filter tree, SubqueryIndex will need to keep something like: + +node_id, subquery_id, polarity -> child_node_id + +and add the shape to the child WhereCondition node + +When the SubqueryIndex is asked the affected_shapes for a given value, it will need to iterate through all {subquery_id, polarity} pairs it has and MapSet.union the affected shapes for each. + +For a given {subquery_id, :positive} pair the affected shaped will be: + +if MultiTimeView.member_at_some_time?(subquery_id, value) do + WhereCondition.affected_shapes(node_id) +else + MapSet.new() +end + +For a given {subquery_id, :negative} pair the affected shaped will be: + +if MultiTimeView.member_at_all_times?(subquery_id, value) do + MapSet.new() +else + WhereCondition.affected_shapes(node_id) +end + This will ensure that the rows are included for all available times -If the MultiTimeView has not been populated by the Materializer yet, the SubqueryIndex should just widen as much as possible. +If the MultiTimeView has not been populated by the Materializer yet, the SubqueryIndex should return WhereCondition.affected_shapes(node_id) #### Materializer From 3af8d6b1ba62b6fa061f4a31ab9538f40f375566 Mon Sep 17 00:00:00 2001 From: rob Date: Sat, 16 May 2026 10:11:58 +0100 Subject: [PATCH 03/40] Add problem --- .../sync-service/subquery-index-prompt.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md index 4f812161ed..bf8634d769 100644 --- a/packages/sync-service/subquery-index-prompt.md +++ b/packages/sync-service/subquery-index-prompt.md @@ -163,9 +163,27 @@ These should be updated so that rather than holding views of the subquery, they - the move-in query needs entire views at specific times and so should call MultiTimeView.get(time) and care should be made to not keep this in memory for too long, perhaps we should GC the consumer process afterwards, or perhaps the task process that runs the query should call MultiTimeView.get(time) so that the memory is freed when the process ends +# The Problem With The Above Design -Please: -- read the codebase to check this design would work -- ask any clarifying questions about anything you are unsure of about my proposal -- give me your evaluation of the design -- make suggestions to improve the design for memory, performance or simplicity reasons +Subqueries have different subquery_ids even if they only differ in a constant, so: +- SELECT id FROM users WHERE company_id=7 +- SELECT id FROM users WHERE company_id=8 +are two different subqueries. If the SubqueryIndex iterates through {subquery_id, :positive} pairs that may be thousands of pairs and be too slow since it's in the replication stream hot path. + +Instead we should, at each node, for each {field, polarity} pair, keep a reverse index for all the subqueries for that pair. So: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE user_id IN (SELECT id FROM users WHERE company_id=8) + +would be in the same reverse index because they have the same field (user_id) and polarity (:positive). + +Perhaps the index could have the form: +subquery_cohort_id, value -> list({child_node_id, list(times)}) + +where: +subquery_cohort_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: +subquery_cohort_id -> {node_id, field, polarity} and +{node_id, field, polarity} -> subquery_cohort_id + +and there's one child_node_id per subquery_id for the cohort + +Shape removal can be quick because we can keep track of subquery_id -> child_node_id and remove the shape from the child node, but removing nodes becomes slow since they're scattered throughout the ETS table. I suggest the cleaning up of nodes should be done asynchronously by a process that walks through the ETS table for nodes with no shapes and removes them. Race conditions can be avoided by doing an atomic conditional replace in the ETS table. From 034d74a7c6c48cc7f7693a3f0d4a85c6d7688ec6 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 10:34:53 +0100 Subject: [PATCH 04/40] Update problem --- .../sync-service/subquery-index-prompt.md | 49 ++++++++++++++----- 1 file changed, 37 insertions(+), 12 deletions(-) diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md index bf8634d769..a12a711e3e 100644 --- a/packages/sync-service/subquery-index-prompt.md +++ b/packages/sync-service/subquery-index-prompt.md @@ -94,9 +94,9 @@ rather than specifying a time you can also ask for membership across all times: member_at_some_time?(subquery_id, value) = true member_at_all_times?(subquery_id, value) = false -These are useful for the where clause filter which needs to keep the filter broad enough so that all consumers get all the changes they need while they may be at any of the logical times. +These are useful for the where clause filter which needs to keep the filter broad enough so that all consumers get all the changes they need while they may be at any of the logical times. -For each subquery there will be a minimum logical time needed (the minimum in-flight logical time for the subquery) which the SubqueryProgressMonitor will set on the MultiTimeView. This allows the MultiTimeViewETS table to be compacted for memory and performace efficientcy. For any given list(times) it can be compacted by removing times from before the minimum in-flight logical time, making sure to update the :in/:out marker at the beginning of the list appropriately or removing it if there are no times left. +For each subquery there will be a minimum logical time needed (the minimum in-flight logical time for the subquery) which the SubqueryProgressMonitor will set on the MultiTimeView. This allows the MultiTimeViewETS table to be compacted for memory and performace efficientcy. For any given list(times) it can be compacted by removing times from before the minimum in-flight logical time, making sure to update the :in/:out marker at the beginning of the list appropriately or removing it if there are no times left. Compacting should happen: - when the list is read (e.g. when member? for the value is called) @@ -147,19 +147,20 @@ This is the existing Materializer. It will just need to be updated to: #### Logical Time -Logical Time is monotonically incrementing counter per subquery. +Logical Time is monotonically incrementing counter per subquery. This needs to be a memory efficient data staructure that can be incremented indefinately. If it needs to wrap we need to make sure we use appropriate conparison functions when comparing times. Wrapping is an acceptable solution since there will only ever be so many moves in flight for any given subquery and memory would explode due to that before wrapping would cause comparison failures. #### SubqueryProgressMonitor -Pin worked out from acks --LRU algorithm +This can be a separate process that the outer consumer calls to acknoledge that it's finished with a logical time for a subquery. The SubqueryProgressMonitor can then keep track of the minimum in-flight logical time for each subquery and set that on the MultiTimeView so that the MultiTimeView can compact it's ETS table for memory and performance efficientcy. + +The SubqueryProgressMonitor can be implimented as an ETS table ordered by subquery_id then logical time with an index to where an outer shape_id entry is so that when an outer consumer acks a logical time for a subquery, the outer shape can be found in the the ordered list and removed and replaced with the acked time. The minimum of theses times is the minimum in-flight logical time for the subquery. This should mean that updating a outer shape's logical time is O(1) and reading the minimum in-flight logical time is O(1). The SubqueryProgressMonitor should notify the MultiTimeView when the minimum in-flight logical time for a subquery changes so that the MultiTimeView can compact it's ETS table. #### Consumer EventProcessors -These should be updated so that rather than holding views of the subquery, they just hold the logical time. so the before and after views should instead just be the before and after logical times. -- `convert_change` should have a function passed to it that access MultiTimeView.member? at the specified time +These should be updated so that rather than holding views of the subquery, they just hold the logical time. so the before and after views should instead just be the before and after logical times. +- `convert_change` should have a function passed to it that access MultiTimeView.member? at the specified time - the move-in query needs entire views at specific times and so should call MultiTimeView.get(time) and care should be made to not keep this in memory for too long, perhaps we should GC the consumer process afterwards, or perhaps the task process that runs the query should call MultiTimeView.get(time) so that the memory is freed when the process ends @@ -177,13 +178,37 @@ WHERE user_id IN (SELECT id FROM users WHERE company_id=8) would be in the same reverse index because they have the same field (user_id) and polarity (:positive). Perhaps the index could have the form: -subquery_cohort_id, value -> list({child_node_id, list(times)}) +subquery_group_id, value -> list({child_node_id, list(times)}) where: -subquery_cohort_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: -subquery_cohort_id -> {node_id, field, polarity} and -{node_id, field, polarity} -> subquery_cohort_id +subquery_group_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: +subquery_group_id -> {node_id, field, polarity} and +{node_id, field, polarity} -> subquery_group_id -and there's one child_node_id per subquery_id for the cohort +and there's one child_node_id per subquery_id for the group Shape removal can be quick because we can keep track of subquery_id -> child_node_id and remove the shape from the child node, but removing nodes becomes slow since they're scattered throughout the ETS table. I suggest the cleaning up of nodes should be done asynchronously by a process that walks through the ETS table for nodes with no shapes and removes them. Race conditions can be avoided by doing an atomic conditional replace in the ETS table. + +Perhaps this will replace the MultiTimeView proposes above, we're still multi-time but we work at a group level rather than the subquery level. This would mean that the Materializer calls to add/remove values from the subquery must update all groups that the subquery is in. + + +### Definitions + +#### Subquery + +Each subquery gets it's own shape. If the select statement differs at all we count it as a different subquery, even if the difference is just in a constant. So: +- SELECT id FROM users WHERE company_id=7 +- SELECT id FROM users WHERE company_id=8 +are two different subqueries and each get their own subquery_id (the handle for the subquery shape) + +#### Subquery Group + +A subquery group is a set of subqueries that have the same field and polarity at a particular node in the filter tree. + +So for example the two subqueries in the two shapes below are differnt subqueries (because they differ by the company_id constant) but they are in the same subquery group because they have the same field (user_id) and polarity (:positive) at the same node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE user_id IN (SELECT id FROM users WHERE company_id=8) + +A subquery_id may appear in multiple subquery groups if it appears at multiple nodes in the filter tree. For the subquery is the same (has the same subquery_id) in the two shapes below but falls into different subquery groups because it appears at a differnt node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE project_id=4 AND user_id IN (SELECT id FROM users WHERE company_id=7) From 523071a2e1963f66a45501cf272b51941fffdae2 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 17:48:51 +0100 Subject: [PATCH 05/40] Add concurrency models an O(1) removal with groups --- .../sync-service/subquery-index-prompt.md | 111 ++++++++---------- 1 file changed, 50 insertions(+), 61 deletions(-) diff --git a/packages/sync-service/subquery-index-prompt.md b/packages/sync-service/subquery-index-prompt.md index a12a711e3e..a75af9a287 100644 --- a/packages/sync-service/subquery-index-prompt.md +++ b/packages/sync-service/subquery-index-prompt.md @@ -46,12 +46,33 @@ There are at least two duplicated memory pools: Adding a reverse index such as `shape_handle -> all values` would make removal faster, but it would increase memory. +## Definitions + +### Subquery + +Each subquery gets it's own shape. If the select statement differs at all we count it as a different subquery, even if the difference is just in a constant. So: +- SELECT id FROM users WHERE company_id=7 +- SELECT id FROM users WHERE company_id=8 +are two different subqueries and each get their own subquery_id (the handle for the subquery shape) + +### Subquery Group + +A subquery group is a set of subqueries that have the same field and polarity at a particular node in the filter tree. + +So for example the two subqueries in the two shapes below are differnt subqueries (because they differ by the company_id constant) but they are in the same subquery group because they have the same field (user_id) and polarity (:positive) at the same node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE user_id IN (SELECT id FROM users WHERE company_id=8) + +A subquery_id may appear in multiple subquery groups if it appears at multiple nodes in the filter tree. For the subquery is the same (has the same subquery_id) in the two shapes below but falls into different subquery groups because it appears at a differnt node in the filter tree: +WHERE user_id IN (SELECT id FROM users WHERE company_id=7) +WHERE project_id=4 AND user_id IN (SELECT id FROM users WHERE company_id=7) + ## Goals - Reduce memory footprint of subqueries significantly while remaining consitant and performant - have near O(1) performance for: - - subquery addition and removal - - row processing by the where clause filter (so for afffected_shapes in the SubqueryIndex) + - subquery addition and removal, including subquery group addition and removal where needed + - row processing by the where clause filter (so for afffected_shapes in the SubqueryIndex) even when there are thousands of subqueries in a subquery group - Store one shared materialized view per subquery. - Support exact membership reads at separate logical times. - Preserve positive, negated, AND, OR, and NOT subquery correctness from v1.6. @@ -108,20 +129,24 @@ Removing a subquery should not involve a full ETS table scan as this will be too #### SubqueryIndex -This is a complete re-write of the existing SubqueryIndex that delegates most of it's resposibility to the the MultiTimeView. +This is a complete re-write of the existing SubqueryIndex that delegates some of it's resposibility to the the MultiTimeView. + +Since there may be many subqueries in a subquery group, the SubqueryIndex should keep: -When a shape is added to the SubqueryIndex at a particular node in the filter tree, SubqueryIndex will need to keep something like: +subquery_group_id, value -> list(child_node_id) -node_id, subquery_id, polarity -> child_node_id +where: +subquery_group_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: +subquery_group_id -> {node_id, field, polarity} and +{node_id, field, polarity} -> subquery_group_id -and add the shape to the child WhereCondition node +and there's one child_node_id per subquery_id for the group. child_node_id is smaller in memory so we keep that in places where it's going to be repeated lots in the ETS table (e.g. in `subquery_group_id, value -> list(child_node_id)`) -When the SubqueryIndex is asked the affected_shapes for a given value, it will need to iterate through all {subquery_id, polarity} pairs it has and MapSet.union the affected shapes for each. -For a given {subquery_id, :positive} pair the affected shaped will be: +So for `afffected_shapes` for a particular value, we'd look up the list of child_node_ids from the subquery_group_id, value pair then lookup the subquery_ids from the child_node_ids then for each subquery_id: if MultiTimeView.member_at_some_time?(subquery_id, value) do - WhereCondition.affected_shapes(node_id) + WhereCondition.affected_shapes(child_node_id) else MapSet.new() end @@ -131,19 +156,21 @@ For a given {subquery_id, :negative} pair the affected shaped will be: if MultiTimeView.member_at_all_times?(subquery_id, value) do MapSet.new() else - WhereCondition.affected_shapes(node_id) + WhereCondition.affected_shapes(child_node_id) end -This will ensure that the rows are included for all available times +This will ensure that the rows are included for all available times. -If the MultiTimeView has not been populated by the Materializer yet, the SubqueryIndex should return WhereCondition.affected_shapes(node_id) +If the MultiTimeView has not been marked ready by the Materializer yet, the SubqueryIndex should return WhereCondition.affected_shapes(child_node_id) + +Removal of a subquery must not scale with the total number of shapes or the number of subqueries in the group, but can scale with the number of values for the subquery. This can be achived by getting the getting the values for the subquery from the MultiTimeView (as discussed above in the MultiTimeView section when talking about subquery removal) - whilst iterating though those values we can also delete those values in the SubqueryIndex for all the groups that it's in. #### Materializer This is the existing Materializer. It will just need to be updated to: -- populate the MultiTimeView when the Materializer has initialised (it has a full materialized view). This should be at logical time 0. +- populate the SubqueryIndex when the Materializer has initialised (it has a full materialized view). This should be at logical time 0. - increment logical time for each `{:materializer_changes` message it sends to outer consumers, and include the new logical time in that message -- before the `{:materializer_changes` message is sent, the MultiTimeView should be updated with the changes giving the new logical time as the time of the change +- before the `{:materializer_changes` message is sent, the SubqueryIndex should be updated with the changes giving the new logical time as the time of the change #### Logical Time @@ -157,58 +184,20 @@ This can be a separate process that the outer consumer calls to acknoledge that The SubqueryProgressMonitor can be implimented as an ETS table ordered by subquery_id then logical time with an index to where an outer shape_id entry is so that when an outer consumer acks a logical time for a subquery, the outer shape can be found in the the ordered list and removed and replaced with the acked time. The minimum of theses times is the minimum in-flight logical time for the subquery. This should mean that updating a outer shape's logical time is O(1) and reading the minimum in-flight logical time is O(1). The SubqueryProgressMonitor should notify the MultiTimeView when the minimum in-flight logical time for a subquery changes so that the MultiTimeView can compact it's ETS table. +The SubqueryProgressMonitor must know about all shapes for a subquery (so for example if it's not seen an ack from one of them it needs to know the minimum time is still 0) or a subquery and have those shapes removed + #### Consumer EventProcessors These should be updated so that rather than holding views of the subquery, they just hold the logical time. so the before and after views should instead just be the before and after logical times. - `convert_change` should have a function passed to it that access MultiTimeView.member? at the specified time - the move-in query needs entire views at specific times and so should call MultiTimeView.get(time) and care should be made to not keep this in memory for too long, perhaps we should GC the consumer process afterwards, or perhaps the task process that runs the query should call MultiTimeView.get(time) so that the memory is freed when the process ends +### Concurrency model -# The Problem With The Above Design +Reads and writes to the MultiTimeView and SubqueryIndex ETS tables will mostly not be concurrent: +- add_shape and remove_shape will happen on the ShapeLogCollector process +- add_value and remove_value will happen while the ShapeLogCollector process is blocked so acts as if it were on the ShapeLogCollector process (ShapeLogCollector calls the Consumer which calls the Materializer which calls the SubqueryIndex to add/remove values, all synchronously) +- a Materializer seeding a subquery will happen when the Materializer is ready (so asyncronously to the ShapeLogCollector process) but will then call mark_ready on the SubqueryIndex which is an atomic process +- read of MultiTimeView may happen async by a consumer, but will be a read at a specific logical time so concurrentcy should not be an issue +- the mimimum in-flight logical time for a subquery will be updated by the SubqueryProgressMonitor async, but this will just update a single number, so concurrentcy should not be an issue -Subqueries have different subquery_ids even if they only differ in a constant, so: -- SELECT id FROM users WHERE company_id=7 -- SELECT id FROM users WHERE company_id=8 -are two different subqueries. If the SubqueryIndex iterates through {subquery_id, :positive} pairs that may be thousands of pairs and be too slow since it's in the replication stream hot path. - -Instead we should, at each node, for each {field, polarity} pair, keep a reverse index for all the subqueries for that pair. So: -WHERE user_id IN (SELECT id FROM users WHERE company_id=7) -WHERE user_id IN (SELECT id FROM users WHERE company_id=8) - -would be in the same reverse index because they have the same field (user_id) and polarity (:positive). - -Perhaps the index could have the form: -subquery_group_id, value -> list({child_node_id, list(times)}) - -where: -subquery_group_id is a number (whatever is smallest in memory) and represents {node_id, field, polarity} but to save memory (as it's going to be repeated lots in the ETS table) we keep it small and also store: -subquery_group_id -> {node_id, field, polarity} and -{node_id, field, polarity} -> subquery_group_id - -and there's one child_node_id per subquery_id for the group - -Shape removal can be quick because we can keep track of subquery_id -> child_node_id and remove the shape from the child node, but removing nodes becomes slow since they're scattered throughout the ETS table. I suggest the cleaning up of nodes should be done asynchronously by a process that walks through the ETS table for nodes with no shapes and removes them. Race conditions can be avoided by doing an atomic conditional replace in the ETS table. - -Perhaps this will replace the MultiTimeView proposes above, we're still multi-time but we work at a group level rather than the subquery level. This would mean that the Materializer calls to add/remove values from the subquery must update all groups that the subquery is in. - - -### Definitions - -#### Subquery - -Each subquery gets it's own shape. If the select statement differs at all we count it as a different subquery, even if the difference is just in a constant. So: -- SELECT id FROM users WHERE company_id=7 -- SELECT id FROM users WHERE company_id=8 -are two different subqueries and each get their own subquery_id (the handle for the subquery shape) - -#### Subquery Group - -A subquery group is a set of subqueries that have the same field and polarity at a particular node in the filter tree. - -So for example the two subqueries in the two shapes below are differnt subqueries (because they differ by the company_id constant) but they are in the same subquery group because they have the same field (user_id) and polarity (:positive) at the same node in the filter tree: -WHERE user_id IN (SELECT id FROM users WHERE company_id=7) -WHERE user_id IN (SELECT id FROM users WHERE company_id=8) - -A subquery_id may appear in multiple subquery groups if it appears at multiple nodes in the filter tree. For the subquery is the same (has the same subquery_id) in the two shapes below but falls into different subquery groups because it appears at a differnt node in the filter tree: -WHERE user_id IN (SELECT id FROM users WHERE company_id=7) -WHERE project_id=4 AND user_id IN (SELECT id FROM users WHERE company_id=7) From 3164877619f9000003a28aea252ab6d623ee5a61 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 19:16:57 +0100 Subject: [PATCH 06/40] Add RFC --- docs/rfcs/subquery-index.md | 606 ++++++++++++++++++++++++++++++++++++ 1 file changed, 606 insertions(+) create mode 100644 docs/rfcs/subquery-index.md diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md new file mode 100644 index 0000000000..a9780d5cc5 --- /dev/null +++ b/docs/rfcs/subquery-index.md @@ -0,0 +1,606 @@ +# RFC: Shared Subquery Indexes with Logical-Time Views + +Status: Draft + +Scope: `packages/sync-service` + +## Summary + +Electric v1.6 introduced per-shape subquery indexing so boolean subquery +shapes stay live while dependency rows move across `WHERE` boundaries. That +solved correctness, but it made memory scale with the number of outer shape +consumers. + +This RFC proposes replacing per-consumer materialized subquery views with one +shared, versioned view per subquery. Consumers do not copy the subquery view. +Instead, each consumer keeps the logical time it is reading and asks the shared +view for membership at that time. + +The design keeps current move-in/move-out correctness for positive, negated, +`AND`, `OR`, and `NOT` subquery expressions, while reducing duplicate memory in +the filter index and in consumer event handlers. + +## Background + +The current implementation stores subquery state in two duplicated places: + +- `Electric.Shapes.Filter.Indexes.SubqueryIndex` stores per-shape routing and + exact membership rows. +- `Electric.Shapes.Consumer.EventHandler.Subqueries` stores per-consumer + `MapSet` views, including both before and after views while a move-in is + buffering. + +The key correctness problem is that consumers can temporarily disagree about a +subquery's membership. One consumer may have processed a dependency move while +another has not. The current implementation handles that by letting each outer +shape seed and update its own exact view. + +This is correct, but it duplicates the same dependency view across many +consumers. + +## Problem + +For a popular subquery, memory currently scales roughly with: + +```text +number_of_outer_consumers * number_of_values_in_subquery +``` + +There are two major pools of duplicated memory: + +1. Subquery routing and exact membership rows keyed by outer shape. +2. Consumer-held dependency views, including before and after views during + active move-in buffering. + +A reverse index such as `shape_handle -> all values` would make removal faster, +but it would add another per-consumer value list and worsen the memory problem. + +## Goals + +- Store one shared materialized view per subquery. +- Allow consumers to read exact membership at separate logical times. +- Keep routing conservative enough while consumers are at different logical + times. +- Keep subquery addition and removal proportional to the subquery and group + data being changed, not to the total number of shapes in the stack. +- Preserve correctness for positive subqueries, negated subqueries, `AND`, + `OR`, and `NOT`. +- Avoid changing the client wire protocol. + +## Non-Goals + +- This RFC does not change Electric's HTTP protocol. +- This RFC does not change the semantics of supported subqueries. +- This RFC does not attempt to make negated-subquery routing better than + `O(number_of_affected_shapes)`. If a value is absent from a large negated + subquery group, all of those shapes are genuinely affected. + +## Definitions + +### Subquery + +A subquery is represented by its dependency shape. The `subquery_id` is the +handle of that dependency shape. + +Different `SELECT` statements are different subqueries, even if they differ +only by constants. For example: + +```sql +SELECT id FROM users WHERE company_id = 7 +SELECT id FROM users WHERE company_id = 8 +``` + +These are two different subqueries and get two different `subquery_id` values. + +### Subquery Group + +A subquery group is a set of subquery occurrences with the same: + +- filter tree node +- field key +- polarity + +For example, these two outer shapes use different subqueries, but the same +subquery group if the subquery occurrence appears at the same filter node: + +```sql +WHERE user_id IN (SELECT id FROM users WHERE company_id = 7) +WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) +``` + +The subqueries differ by `company_id`, but the group is the same because the +field key is `user_id` and the polarity is positive. + +A single `subquery_id` can appear in multiple groups if it appears at multiple +nodes in outer filter trees. + +### Child Node + +A `child_node_id` is created per `{subquery_group_id, subquery_id}` pair. + +The child node owns a child `WhereCondition` that contains all outer shapes +using that subquery in that group. This means many outer shapes can share one +child node. + +### Logical Time + +Logical time is a monotonically increasing integer per subquery. + +Time `0` represents the materializer's initial view. Each committed dependency +move that changes subquery membership increments the logical time and records +the move at the new time. + +Logical time should use normal BEAM integers. Wrapping is unnecessary and would +make comparison and compaction harder to reason about. + +### Processed-Up-To Time + +Consumers call: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id) +``` + +after they no longer need to read the subquery at `time` or earlier. + +For a move from logical time `a` to logical time `b`, once the consumer has +finished processing that move and is steady at `b`, it notifies that it has +processed up to `a`. + +The minimum required time for compaction is therefore: + +```text +min_processed_up_to_for_live_consumers + 1 +``` + +Consumers registered at current time `t` start with `processed_up_to = t - 1`, +because they need to read time `t`. + +## Proposal + +### MultiTimeView + +`SubqueryIndex.MultiTimeView` stores one shared materialized view per subquery. + +It is an ETS-backed structure, with one ETS table per stack. The main logical +key is: + +```text +{subquery_id, value} -> membership_history +``` + +Absence means the value is not a member at any retained logical time. + +The common case is a value that is always present for the retained window. That +should be represented compactly, for example: + +```elixir +true +``` + +Values that have moved use a small transition history. The exact structure +should be benchmarked before implementation. A simple starting point is: + +```elixir +{:out, [9]} +{:out, [9, 11]} +{:in, [9]} +{:in, [9, 11]} +``` + +The first atom is the membership state before the first transition. Each time in +the list toggles membership from that time onwards. + +Examples: + +```elixir +# Out before 9, in from 9 onwards. +{:out, [9]} + +# Out before 9, in from 9 to 10, out from 11 onwards. +{:out, [9, 11]} + +# In before 9, out from 9 to 10, in from 11 onwards. +{:in, [9, 11]} +``` + +The API should support: + +```elixir +member?(subquery_id, value, time) +member_at_some_time?(subquery_id, value) +member_at_all_times?(subquery_id, value) +values(subquery_id) +values(subquery_id, time) +mark_ready(subquery_id) +ready?(subquery_id) +set_min_required_time(subquery_id, time) +remove_subquery(subquery_id) +``` + +`member_at_some_time?/2` and `member_at_all_times?/2` operate over the retained +time window for that subquery. + +#### Compaction + +The `SubqueryProgressMonitor` provides the minimum required logical time for +each subquery. `MultiTimeView` can compact entries by removing transitions +before that time. + +Compaction must preserve membership at all retained times. For example: + +```elixir +{:out, [9, 11]} +``` + +If `min_required_time = 10`, membership at time `10` is `true`, and the compacted +history becomes: + +```elixir +{:in, [11]} +``` + +If `min_required_time = 12`, the value is out for the whole retained window, so +the row can be deleted. + +Compaction should run: + +- when a value is read +- when a value is written +- in a periodic asynchronous compaction pass +- when the progress monitor advances the minimum required time + +`remove_subquery/1` must not scan the whole ETS table. The table should be an +ordered set with keys ordered by `subquery_id`, so removal can iterate the +contiguous key range for one subquery. + +### SubqueryIndex + +`SubqueryIndex` becomes responsible for topology and routing, while +`MultiTimeView` owns exact membership. + +The index stores compact integer identifiers for repeated values: + +```text +{node_id, field_key, polarity} -> subquery_group_id +subquery_group_id -> {node_id, field_key, polarity} + +{subquery_group_id, subquery_id} -> child_node_id +child_node_id -> {subquery_group_id, subquery_id, next_condition_id} +subquery_id -> [{subquery_group_id, child_node_id}] +``` + +Using small integer ids avoids repeating large tuples, field keys, and shape +handles in per-value ETS rows. + +For positive groups, the routing index stores values that can be members at +some retained logical time: + +```text +{positive_value, subquery_group_id, value} -> [child_node_id] +``` + +For negative groups, the index stores all negative children for the group: + +```text +{negative_children, subquery_group_id} -> [child_node_id] +``` + +The negative path cannot avoid considering all affected children when a value +is absent from the subquery views. That is acceptable because those children are +affected. + +#### Affected Shapes + +For a root-table change, `SubqueryIndex.affected_shapes/4` evaluates the +left-hand side value for the subquery node. + +If evaluation fails, it falls back to all children in the group. + +If a subquery is not ready, the child node is routed conservatively. + +For a positive group: + +```elixir +for child_node_id <- positive_children_for_value(group_id, value), + subquery_id = subquery_id_for_child(child_node_id), + MultiTimeView.member_at_some_time?(subquery_id, value) do + WhereCondition.affected_shapes(child_node_id, record) +end +``` + +For a negative group: + +```elixir +for child_node_id <- all_negative_children(group_id), + subquery_id = subquery_id_for_child(child_node_id), + not MultiTimeView.member_at_all_times?(subquery_id, value) do + WhereCondition.affected_shapes(child_node_id, record) +end +``` + +This keeps routing broad enough for all consumers reading any retained logical +time. + +#### Shape Addition + +Adding an outer shape to an existing `{group, subquery_id}` child is near O(1): +the shape is added to the child `WhereCondition`. + +Creating the first child for `{group, subquery_id}` requires indexing current +values for that subquery in the group. That is O(number of values in the +subquery), which is acceptable and unavoidable unless the child stays in a +fallback mode until asynchronous seeding completes. + +#### Shape Removal + +Removing an outer shape removes it from the child `WhereCondition`. + +If the child becomes empty, remove the `{group, subquery_id}` child and update +the group value indexes by iterating the values for `subquery_id` from +`MultiTimeView`. This is proportional to the number of values in that subquery, +not to the total number of shapes or subqueries. + +### Materializer + +The materializer continues to track dependency shape membership from the +dependency log. + +It changes in three ways: + +1. On initial load, it populates `MultiTimeView` for the subquery at logical + time `0`, then marks the subquery ready. +2. On each committed batch that produces net move events, it increments the + subquery logical time. +3. Before sending `{:materializer_changes, ...}` to subscribers, it writes the + move events to `MultiTimeView` at the new logical time. + +The subscriber payload should include both the old and new logical time: + +```elixir +%{ + move_in: [{value, original_string}], + move_out: [{value, original_string}], + txids: [txid], + from_time: old_time, + to_time: new_time +} +``` + +If a committed batch has no net membership move, logical time does not need to +advance. + +The existing `Materializer.LinkValues` ETS cache should be removed or replaced +by `MultiTimeView` rather than kept as a second full shared copy. + +### Consumer Event Handlers + +Consumers store logical times instead of materialized subquery views. + +The steady handler keeps: + +```elixir +%{subquery_ref => logical_time} +``` + +`Shape.convert_change/3` and DNF metadata projection need to accept a membership +callback instead of requiring a concrete `MapSet` view: + +```elixir +fn subquery_ref, value, time -> + MultiTimeView.member?(subquery_id, value, time) +end +``` + +In steady state, old and new records use the same logical time. + +During a buffered move-in, `ActiveMove` stores: + +```elixir +times_before_move +times_after_move +``` + +instead of: + +```elixir +views_before_move +views_after_move +``` + +Buffered transactions before the splice boundary are converted using +`times_before_move`. Buffered transactions after the splice boundary are +converted using `times_after_move`. + +After the move is spliced and the consumer becomes steady at `to_time`, it +calls: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) +``` + +#### Move-In Queries + +Move-in queries currently build SQL from whole before and after views. The new +implementation should avoid retaining large views in the consumer process. + +Preferred approach: + +- Build the triggering dependency candidate predicate from the move delta + values when possible. +- Read full view values at a specific time only for positions that require + exclusion logic. +- If full views are required, materialize them inside the task process that + runs the query so the memory is released when the task exits. + +This is important because replacing long-lived consumer views with +short-lived task views is where much of the memory win comes from. + +### SubqueryProgressMonitor + +`SubqueryProgressMonitor` tracks the earliest logical time still needed by live +outer consumers. + +Consumers register for each subquery they read. Registration at current time +`t` inserts: + +```text +processed_up_to = t - 1 +``` + +When a consumer finishes a move from `from_time` to `to_time`, it calls: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) +``` + +The monitor maintains two ETS indexes: + +```text +{subquery_id, consumer_shape_handle} -> processed_up_to +{subquery_id, processed_up_to, consumer_shape_handle} -> true +``` + +The first index makes updates O(1). The second index makes the minimum +processed time for a subquery cheap to read. + +When the minimum changes, the monitor notifies `MultiTimeView`: + +```elixir +MultiTimeView.set_min_required_time(subquery_id, min_processed_up_to + 1) +``` + +When an outer shape is removed, the monitor removes that consumer from every +subquery it was registered for and recomputes the affected minima. + +### Concurrency Model + +Most writes are already serialized by existing processes: + +- Shape addition and removal happen through the ShapeLogCollector path. +- Dependency changes are applied by materializers. +- Outer consumers process events synchronously when ShapeLogCollector publishes + to them. + +`MultiTimeView` writes happen in the materializer before it sends +`materializer_changes` to subscribers. + +`SubqueryIndex` topology changes happen when shapes are added or removed from +the filter. + +`SubqueryIndex` reads happen from ShapeLogCollector while routing replication +changes. + +Consumer reads from `MultiTimeView` can happen concurrently with writes. This +is safe because membership is always read at an explicit logical time, and ETS +updates replace complete membership-history values atomically. + +The ready flag is important. Until a subquery has been seeded into +`MultiTimeView` and any group routing rows have been created, routing must be +conservative. + +## Expected Benefits + +- One retained membership view per subquery instead of one per outer shape. +- Consumer processes retain logical times instead of large `MapSet` views. +- Move-in buffering retains before/after logical times instead of before/after + `MapSet` views. +- Subquery removal is proportional to the removed subquery's values and group + entries, not to total stack size. +- Positive routing remains value-keyed and efficient. + +## Risks + +### Off-by-One Compaction + +The biggest correctness risk is compacting away a logical time that some +consumer still needs. + +The invariant is: + +```text +MultiTimeView may compact only times < min_required_time +min_required_time = min(processed_up_to_by_live_consumer) + 1 +``` + +Tests should cover move-in, move-out, repeated toggles, consumer registration, +consumer removal, and compaction across all of those cases. + +### Negated Routing Cost + +For negated subquery groups, a value absent from all subqueries affects all +children in the group. This can be large, but it is proportional to the number +of affected shapes. + +The implementation should avoid extra memory-heavy complement indexes unless +there is evidence they are necessary. + +### Move-In Query Memory + +If move-in query generation materializes full before and after views in the +consumer process, the design will keep a major source of memory duplication. + +Full view materialization should be avoided where possible and isolated to +short-lived task processes where not possible. + +### Fallback Windows + +Unready subqueries and not-yet-seeded group routing must route conservatively. +This may temporarily over-route, but must not under-route. + +## Testing Plan + +Add focused unit tests for `MultiTimeView`: + +- membership at exact logical times +- `member_at_some_time?/2` +- `member_at_all_times?/2` +- move-in and move-out transitions +- repeated toggles +- compaction after `set_min_required_time/2` +- subquery removal by ordered key range + +Add focused unit tests for `SubqueryProgressMonitor`: + +- registration at current time +- `notify_processed_up_to/2` +- minimum required time updates +- consumer removal +- multiple consumers at different times + +Add `SubqueryIndex` tests: + +- positive group routing +- negative group routing +- shared child nodes per `{group, subquery_id}` +- conservative routing for unready subqueries +- child removal after the last outer shape is removed +- no full-table scan on subquery removal + +Update consumer event-handler tests: + +- steady conversion using logical times +- buffered move-in using before and after logical times +- queued moves across multiple logical times +- progress notifications after splice +- negated move-in and move-out behavior + +Keep or extend integration tests for: + +- dependency move-in +- dependency move-out +- nested subqueries +- subqueries combined with non-subquery predicates +- rows moving between two dependency values in one transaction + +## Open Questions + +- Should `MultiTimeView` expose `values(subquery_id, time)` as a materialized + `MapSet`, a stream, or both? +- Should `SubqueryProgressMonitor.notify_processed_up_to/2` infer the consumer + identity from process state, or should callers pass the outer shape handle + explicitly? +- Should first-time child creation seed synchronously, or should it use fallback + routing while an asynchronous seeding task populates group value rows? +- Which transition-history representation is smallest in practice for ETS: + list, tuple, or binary? From 3becd0014086976ca7e2b60d19fc66bff41427d2 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 19:35:21 +0100 Subject: [PATCH 07/40] Human scan of RFC --- docs/rfcs/subquery-index.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index a9780d5cc5..236bc220d4 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -248,7 +248,6 @@ Compaction should run: - when a value is read - when a value is written - in a periodic asynchronous compaction pass -- when the progress monitor advances the minimum required time `remove_subquery/1` must not scan the whole ETS table. The table should be an ordered set with keys ordered by `subquery_id`, so removal can iterate the @@ -424,13 +423,7 @@ SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) Move-in queries currently build SQL from whole before and after views. The new implementation should avoid retaining large views in the consumer process. -Preferred approach: - -- Build the triggering dependency candidate predicate from the move delta - values when possible. -- Read full view values at a specific time only for positions that require - exclusion logic. -- If full views are required, materialize them inside the task process that +Preferred approach - materialize them inside the task process that runs the query so the memory is released when the task exits. This is important because replacing long-lived consumer views with From a71f14fc1e39cf16464e5b750b2cc62a7b64bf20 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 19:45:03 +0100 Subject: [PATCH 08/40] Updates to RFC --- docs/rfcs/subquery-index.md | 103 ++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 35 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 236bc220d4..1f98ad68ab 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -135,6 +135,9 @@ make comparison and compaction harder to reason about. ### Processed-Up-To Time +The progress monitor's public API is expressed as a processed-up-to +notification: + Consumers call: ```elixir @@ -142,19 +145,26 @@ SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id) ``` after they no longer need to read the subquery at `time` or earlier. +The public function can include `self()` in the message to the monitor, so the +consumer identity is resolved from registration rather than passed at every +call site. For a move from logical time `a` to logical time `b`, once the consumer has finished processing that move and is steady at `b`, it notifies that it has processed up to `a`. -The minimum required time for compaction is therefore: +Internally, the progress monitor should track `required_time`: the earliest +logical time a live consumer may still read. `notify_processed_up_to(a, +subquery_id)` advances that consumer's `required_time` to `a + 1`. + +The minimum required time for compaction is: ```text -min_processed_up_to_for_live_consumers + 1 +min(required_time_for_live_consumers) ``` -Consumers registered at current time `t` start with `processed_up_to = t - 1`, -because they need to read time `t`. +Consumers register with the logical time they are starting from. If a consumer +starts from current logical time `t`, its initial `required_time` is `t`. ## Proposal @@ -172,36 +182,36 @@ key is: Absence means the value is not a member at any retained logical time. The common case is a value that is always present for the retained window. That -should be represented compactly, for example: +is represented as an empty history: ```elixir -true +[] ``` Values that have moved use a small transition history. The exact structure should be benchmarked before implementation. A simple starting point is: ```elixir -{:out, [9]} -{:out, [9, 11]} -{:in, [9]} -{:in, [9, 11]} +[:out, 9] +[:out, 9, 11] +[:in, 9] +[:in, 9, 11] ``` -The first atom is the membership state before the first transition. Each time in -the list toggles membership from that time onwards. +The first list item is the membership state before the first transition. Each +time after it toggles membership from that time onwards. Examples: ```elixir # Out before 9, in from 9 onwards. -{:out, [9]} +[:out, 9] # Out before 9, in from 9 to 10, out from 11 onwards. -{:out, [9, 11]} +[:out, 9, 11] # In before 9, out from 9 to 10, in from 11 onwards. -{:in, [9, 11]} +[:in, 9, 11] ``` The API should support: @@ -224,20 +234,20 @@ time window for that subquery. #### Compaction The `SubqueryProgressMonitor` provides the minimum required logical time for -each subquery. `MultiTimeView` can compact entries by removing transitions -before that time. +each subquery. `MultiTimeView` can compact entries by evaluating membership at +that time and removing transitions at or before it. Compaction must preserve membership at all retained times. For example: ```elixir -{:out, [9, 11]} +[:out, 9, 11] ``` If `min_required_time = 10`, membership at time `10` is `true`, and the compacted history becomes: ```elixir -{:in, [11]} +[:in, 11] ``` If `min_required_time = 12`, the value is out for the whole retained window, so @@ -423,7 +433,13 @@ SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) Move-in queries currently build SQL from whole before and after views. The new implementation should avoid retaining large views in the consumer process. -Preferred approach - materialize them inside the task process that +Preferred approach: + +- Build the triggering dependency candidate predicate from the move delta + values when possible. +- Read full view values at a specific time only for positions that require + exclusion logic. +- If full views are required, materialize them inside the task process that runs the query so the memory is released when the task exits. This is important because replacing long-lived consumer views with @@ -434,33 +450,53 @@ short-lived task views is where much of the memory win comes from. `SubqueryProgressMonitor` tracks the earliest logical time still needed by live outer consumers. -Consumers register for each subquery they read. Registration at current time -`t` inserts: +Registration must happen atomically with choosing the consumer's starting +logical time. The materializer should provide a serialized registration call, +for example: -```text -processed_up_to = t - 1 +```elixir +{:ok, current_time} = + Materializer.register_subquery_consumer( + subquery_id, + outer_shape_handle, + self() + ) ``` +The materializer handles the call in its GenServer, reads its current logical +time, registers the outer consumer with the progress monitor, and returns that +time to the consumer. This prevents a race where the materializer advances and +compacts a time before the new consumer is registered as needing it. + +The monitor stores the returned time as the consumer's internal +`required_time`. + When a consumer finishes a move from `from_time` to `to_time`, it calls: ```elixir SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) ``` +The monitor then advances that consumer's `required_time` to `from_time + 1`. + The monitor maintains two ETS indexes: ```text -{subquery_id, consumer_shape_handle} -> processed_up_to -{subquery_id, processed_up_to, consumer_shape_handle} -> true +{subquery_id, consumer_shape_handle} -> required_time +{subquery_id, required_time, consumer_shape_handle} -> true ``` The first index makes updates O(1). The second index makes the minimum -processed time for a subquery cheap to read. +required time for a subquery cheap to read. + +If `notify_processed_up_to/2` infers consumer identity from the caller process, +the monitor also needs a lookup from registered consumer pid to +`consumer_shape_handle`. When the minimum changes, the monitor notifies `MultiTimeView`: ```elixir -MultiTimeView.set_min_required_time(subquery_id, min_processed_up_to + 1) +MultiTimeView.set_min_required_time(subquery_id, min_required_time) ``` When an outer shape is removed, the monitor removes that consumer from every @@ -512,8 +548,10 @@ consumer still needs. The invariant is: ```text -MultiTimeView may compact only times < min_required_time -min_required_time = min(processed_up_to_by_live_consumer) + 1 +MultiTimeView may drop transitions at or before min_required_time +after rewriting the entry to preserve membership at min_required_time. + +min_required_time = min(required_time_by_live_consumer) ``` Tests should cover move-in, move-out, repeated toggles, consumer registration, @@ -590,10 +628,5 @@ Keep or extend integration tests for: - Should `MultiTimeView` expose `values(subquery_id, time)` as a materialized `MapSet`, a stream, or both? -- Should `SubqueryProgressMonitor.notify_processed_up_to/2` infer the consumer - identity from process state, or should callers pass the outer shape handle - explicitly? - Should first-time child creation seed synchronously, or should it use fallback routing while an asynchronous seeding task populates group value rows? -- Which transition-history representation is smallest in practice for ETS: - list, tuple, or binary? From 849ea0b5406837cc7d813b3c7eb8892c65621c24 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 19:47:34 +0100 Subject: [PATCH 09/40] Answer question --- docs/rfcs/subquery-index.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 1f98ad68ab..74b3c6db6d 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -338,8 +338,9 @@ the shape is added to the child `WhereCondition`. Creating the first child for `{group, subquery_id}` requires indexing current values for that subquery in the group. That is O(number of values in the -subquery), which is acceptable and unavoidable unless the child stays in a -fallback mode until asynchronous seeding completes. +subquery), which is acceptable and unavoidable. First-time child creation should +seed synchronously so the child is fully routable before the shape is considered +indexed. #### Shape Removal @@ -628,5 +629,3 @@ Keep or extend integration tests for: - Should `MultiTimeView` expose `values(subquery_id, time)` as a materialized `MapSet`, a stream, or both? -- Should first-time child creation seed synchronously, or should it use fallback - routing while an asynchronous seeding task populates group value rows? From a10a850115cfa8d79fbf93b457667a992ba0fc7b Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 20:41:32 +0100 Subject: [PATCH 10/40] Follow RFC format --- docs/rfcs/subquery-index.md | 870 +++++++++++++++++++++--------------- 1 file changed, 498 insertions(+), 372 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 74b3c6db6d..84feeff5a3 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -1,42 +1,65 @@ -# RFC: Shared Subquery Indexes with Logical-Time Views - -Status: Draft - -Scope: `packages/sync-service` +--- +title: Shared Subquery Indexes with Logical-Time Views +version: "0.1" +status: draft +owner: robacourt +contributors: [] +created: 2026-05-18 +last_updated: 2026-05-18 +prd: "N/A - based on https://github.com/electric-sql/electric/issues/4279" +prd_version: "N/A" +--- + +# Shared Subquery Indexes with Logical-Time Views ## Summary -Electric v1.6 introduced per-shape subquery indexing so boolean subquery -shapes stay live while dependency rows move across `WHERE` boundaries. That -solved correctness, but it made memory scale with the number of outer shape -consumers. - -This RFC proposes replacing per-consumer materialized subquery views with one -shared, versioned view per subquery. Consumers do not copy the subquery view. -Instead, each consumer keeps the logical time it is reading and asks the shared -view for membership at that time. - -The design keeps current move-in/move-out correctness for positive, negated, -`AND`, `OR`, and `NOT` subquery expressions, while reducing duplicate memory in -the filter index and in consumer event handlers. +Electric v1.6 added per-shape subquery indexing so shapes with boolean subquery +filters can stay live while dependency rows move across `WHERE` boundaries. +That solved correctness, but it stores the same dependency view repeatedly in +the filter index and in consumer event handlers. This RFC proposes one shared, +logical-time view per subquery. Consumers register the subqueries they read, +keep only the logical time they are reading, and call +`SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id)` when they no +longer need older times. The filter index routes conservatively across retained +times and verifies exact membership by asking the shared view at the consumer's +logical time. ## Background -The current implementation stores subquery state in two duplicated places: - -- `Electric.Shapes.Filter.Indexes.SubqueryIndex` stores per-shape routing and - exact membership rows. -- `Electric.Shapes.Consumer.EventHandler.Subqueries` stores per-consumer - `MapSet` views, including both before and after views while a move-in is - buffering. - -The key correctness problem is that consumers can temporarily disagree about a -subquery's membership. One consumer may have processed a dependency move while -another has not. The current implementation handles that by letting each outer -shape seed and update its own exact view. - -This is correct, but it duplicates the same dependency view across many -consumers. +Issue: https://github.com/electric-sql/electric/issues/4279 + +Related work: + +- PR #4051 introduced the v1.6 subquery move correctness work: + https://github.com/electric-sql/electric/pull/4051 +- PR #4280 proposed a narrower SubqueryIndex memory design using shared base + views with sparse XOR exceptions: + https://github.com/electric-sql/electric/pull/4280 +- Current `SubqueryIndex`: + `packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex` +- Current consumer view setup: + `packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex` +- Current move buffering: + `packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex` +- Current SQL move-in query construction: + `packages/sync-service/lib/electric/shapes/querying.ex` + +The v1.6 subquery work allowed shapes with boolean combinations around +subqueries to stay live when dependency rows move. Without that, Electric would +invalidate the outer shape and require a full resync. + +The current implementation achieves correctness by letting each consumer own a +local dependency view. `EventHandlerBuilder` reads each dependency +materializer's values into a per-consumer `MapSet`. During an active move, +`ActiveMove` stores `views_before_move` and `views_after_move`. Separately, +`SubqueryIndex` stores per-shape routing rows and exact membership rows keyed +by `shape_handle`, `subquery_ref`, and value. + +That model is correct because consumers can temporarily disagree about the same +subquery. One consumer may have processed a dependency move while another has +not. The current implementation represents that by copying the dependency view +per consumer. ## Problem @@ -46,41 +69,115 @@ For a popular subquery, memory currently scales roughly with: number_of_outer_consumers * number_of_values_in_subquery ``` -There are two major pools of duplicated memory: +There are two large duplicated pools: + +- `SubqueryIndex` stores value membership and routing rows per outer shape. +- Consumer event handlers store dependency views per outer shape, and store + both before and after views while a move-in is active. + +Shape removal is also expensive because current value-keyed membership rows do +not have a cheap reverse path from a shape to all of the rows it owns. Adding a +reverse index such as `shape_handle -> all values` would improve removal, but +it would add another copy of the full per-shape dependency view. -1. Subquery routing and exact membership rows keyed by outer shape. -2. Consumer-held dependency views, including before and after views during - active move-in buffering. +The wider design problem is that the current system optimizes for the +exceptional case, where every consumer has a distinct subquery view, by paying +that memory cost in the common case where many consumers share the same view +and only diverge briefly during moves. -A reverse index such as `shape_handle -> all values` would make removal faster, -but it would add another per-consumer value list and worsen the memory problem. +**Link to PRD hypothesis:** There is no PRD for this RFC. The working +hypothesis comes from issue #4279: -## Goals +> Redesigning the SubqueryIndex so it does not store full per-shape dependency +> views will make shape add/remove scalable and reduce memory consumption, +> while preserving v1.6 subquery move correctness. + +## Goals & Non-Goals + +### Goals - Store one shared materialized view per subquery. -- Allow consumers to read exact membership at separate logical times. -- Keep routing conservative enough while consumers are at different logical - times. -- Keep subquery addition and removal proportional to the subquery and group - data being changed, not to the total number of shapes in the stack. +- Allow consumers to read exact subquery membership at separate logical times. +- Remove long-lived per-consumer `MapSet` views from event handlers. +- Remove per-shape exact membership rows from `SubqueryIndex`. +- Keep routing conservative while consumers are at different logical times. +- Keep first-time child creation correct by synchronously seeding routing before + the child is considered indexed. +- Keep shape removal proportional to the shape's subquery participants and + routing edges, not to the full dependency view. - Preserve correctness for positive subqueries, negated subqueries, `AND`, `OR`, and `NOT`. - Avoid changing the client wire protocol. -## Non-Goals +### Non-Goals -- This RFC does not change Electric's HTTP protocol. -- This RFC does not change the semantics of supported subqueries. -- This RFC does not attempt to make negated-subquery routing better than +- Do not change Electric's HTTP protocol. +- Do not change supported subquery semantics. +- Do not redesign DNF planning, tags, or `active_conditions`. +- Do not remove the need to materialize SQL array parameters for move-in + queries in the first implementation. The goal is to avoid long-lived copies; + transient query-local arrays may remain. +- Do not make negated-subquery routing better than `O(number_of_affected_shapes)`. If a value is absent from a large negated - subquery group, all of those shapes are genuinely affected. + group, all of those shapes are genuinely affected. +- Do not intern equivalent SQL subqueries that have different dependency shape + handles. A `subquery_id` is the dependency shape handle for v1. + +## Proposal + +### Core Idea + +Move subquery membership out of per-shape state and into one versioned view per +subquery: + +```text +MultiTimeView[{subquery_id, value}] -> membership_history +consumer[{shape_handle, subquery_ref}] -> {subquery_id, filter_time} +``` + +Consumers no longer copy the subquery view. They register each subquery they +read, store the logical time returned by the materializer, and ask +`MultiTimeView.member?(subquery_id, value, time)` when they need exact +membership. + +The filter index no longer stores exact per-shape membership rows. It stores +compact routing topology: + +```text +subquery_group_id +child_node_id per {subquery_group_id, subquery_id} +shape participant rows +fallback rows while initial indexing is incomplete +``` -## Definitions +Positive routing is value-keyed for values that are members at some retained +logical time. Negated routing is group-keyed and then filtered by shared +membership history. -### Subquery +### Architecture + +```text +Dependency materializer + -> writes MultiTimeView at monotonically increasing logical times + -> emits dependency move events with from_time and to_time + +Consumer event handler + -> registers subqueries through the materializer + -> stores subquery_id and logical times, not MapSet views + -> calls notify_processed_up_to/2 after old times are no longer needed + +SubqueryIndex + -> stores subquery groups, child nodes, and participant routing + -> asks MultiTimeView for membership at some/all retained times for routing + -> asks MultiTimeView for membership at a consumer time for exact checks +``` + +### Definitions + +#### Subquery A subquery is represented by its dependency shape. The `subquery_id` is the -handle of that dependency shape. +dependency shape handle. Different `SELECT` statements are different subqueries, even if they differ only by constants. For example: @@ -90,90 +187,81 @@ SELECT id FROM users WHERE company_id = 7 SELECT id FROM users WHERE company_id = 8 ``` -These are two different subqueries and get two different `subquery_id` values. - -### Subquery Group +These get different `subquery_id` values. -A subquery group is a set of subquery occurrences with the same: +#### Subquery Group -- filter tree node -- field key -- polarity +A subquery group is a set of subquery occurrences with the same filter tree +node, field key, and polarity. -For example, these two outer shapes use different subqueries, but the same -subquery group if the subquery occurrence appears at the same filter node: +For example, these outer shapes use different subqueries but can share the same +subquery group if the occurrence is at the same filter node: ```sql WHERE user_id IN (SELECT id FROM users WHERE company_id = 7) WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) ``` -The subqueries differ by `company_id`, but the group is the same because the -field key is `user_id` and the polarity is positive. +The field key is `user_id`, and the polarity is positive. -A single `subquery_id` can appear in multiple groups if it appears at multiple -nodes in outer filter trees. - -### Child Node +#### Child Node A `child_node_id` is created per `{subquery_group_id, subquery_id}` pair. -The child node owns a child `WhereCondition` that contains all outer shapes -using that subquery in that group. This means many outer shapes can share one -child node. +The child node owns a child `WhereCondition` containing all outer shapes using +that subquery in that group. Many outer shapes can therefore share one child +node. -### Logical Time +#### Logical Time Logical time is a monotonically increasing integer per subquery. Time `0` represents the materializer's initial view. Each committed dependency move that changes subquery membership increments the logical time and records -the move at the new time. - -Logical time should use normal BEAM integers. Wrapping is unnecessary and would -make comparison and compaction harder to reason about. +the transition at the new time. -### Processed-Up-To Time +Use normal BEAM integers. Wrapping is unnecessary and would make comparison and +compaction harder to reason about. -The progress monitor's public API is expressed as a processed-up-to -notification: +#### Processed-Up-To Time -Consumers call: +The public progress API is: ```elixir SubqueryProgressMonitor.notify_processed_up_to(time, subquery_id) ``` -after they no longer need to read the subquery at `time` or earlier. -The public function can include `self()` in the message to the monitor, so the -consumer identity is resolved from registration rather than passed at every -call site. - -For a move from logical time `a` to logical time `b`, once the consumer has -finished processing that move and is steady at `b`, it notifies that it has -processed up to `a`. +Consumers call this after they no longer need to read the subquery at `time` or +earlier. For a move from logical time `a` to logical time `b`, once the +consumer has finished processing that move and is steady at `b`, it notifies +that it has processed up to `a`. -Internally, the progress monitor should track `required_time`: the earliest -logical time a live consumer may still read. `notify_processed_up_to(a, -subquery_id)` advances that consumer's `required_time` to `a + 1`. +Internally, the monitor tracks `required_time`: the earliest logical time a +live consumer may still read. `notify_processed_up_to(a, subquery_id)` advances +that consumer's `required_time` to `a + 1`. -The minimum required time for compaction is: +The compaction lower bound is: ```text min(required_time_for_live_consumers) ``` -Consumers register with the logical time they are starting from. If a consumer -starts from current logical time `t`, its initial `required_time` is `t`. +Consumers register at the logical time they are starting from. If a consumer +starts from current logical time `t`, its initial `required_time` is `t` +because it may read time `t`. -## Proposal +`required_time` is a retention bound, not necessarily the same as the time used +for live exact filter checks. During an active move, a consumer may need the old +time for buffered conversion or move-in query work while live exact checks have +already advanced to the new time. The implementation must keep those two uses +explicit. ### MultiTimeView -`SubqueryIndex.MultiTimeView` stores one shared materialized view per subquery. +`Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView` stores one +shared view per subquery in ETS, with one table per stack. -It is an ETS-backed structure, with one ETS table per stack. The main logical -key is: +The logical key is: ```text {subquery_id, value} -> membership_history @@ -188,8 +276,7 @@ is represented as an empty history: [] ``` -Values that have moved use a small transition history. The exact structure -should be benchmarked before implementation. A simple starting point is: +Values that moved use compact flat histories: ```elixir [:out, 9] @@ -198,8 +285,8 @@ should be benchmarked before implementation. A simple starting point is: [:in, 9, 11] ``` -The first list item is the membership state before the first transition. Each -time after it toggles membership from that time onwards. +The first list item is membership before the first transition. Each integer +after it is a logical time where membership toggles from that time onwards. Examples: @@ -214,6 +301,14 @@ Examples: [:in, 9, 11] ``` +Use `[]` rather than `true` for the always-present case for consistency with +other histories. On BEAM, both `[]` and `true` are immediate terms, so neither +is more compact as an ETS value. + +Use flat lists such as `[:out, 9]` rather than tuples containing lists such as +`{:out, [9]}` because the flat list is smaller and is enough for the common +short-history case. + The API should support: ```elixir @@ -231,11 +326,11 @@ remove_subquery(subquery_id) `member_at_some_time?/2` and `member_at_all_times?/2` operate over the retained time window for that subquery. -#### Compaction +### Compaction -The `SubqueryProgressMonitor` provides the minimum required logical time for -each subquery. `MultiTimeView` can compact entries by evaluating membership at -that time and removing transitions at or before it. +`SubqueryProgressMonitor` provides the minimum required logical time for each +subquery. `MultiTimeView` can compact entries by evaluating membership at that +time and removing transitions at or before it. Compaction must preserve membership at all retained times. For example: @@ -243,8 +338,8 @@ Compaction must preserve membership at all retained times. For example: [:out, 9, 11] ``` -If `min_required_time = 10`, membership at time `10` is `true`, and the compacted -history becomes: +If `min_required_time = 10`, membership at time `10` is `true`, and the +compacted history becomes: ```elixir [:in, 11] @@ -257,375 +352,406 @@ Compaction should run: - when a value is read - when a value is written -- in a periodic asynchronous compaction pass - -`remove_subquery/1` must not scan the whole ETS table. The table should be an -ordered set with keys ordered by `subquery_id`, so removal can iterate the -contiguous key range for one subquery. +- in a periodic asynchronous pass +- when a consumer unregisters and releases the minimum pinned time -### SubqueryIndex +### SubqueryIndex Data Model -`SubqueryIndex` becomes responsible for topology and routing, while -`MultiTimeView` owns exact membership. +The hot ETS rows should use compact integer IDs for groups, children, and +subqueries where practical. Full shape handles and dependency handles can be +stored in metadata rows and interned at boundaries. -The index stores compact integer identifiers for repeated values: +Suggested logical rows: ```text -{node_id, field_key, polarity} -> subquery_group_id -subquery_group_id -> {node_id, field_key, polarity} - -{subquery_group_id, subquery_id} -> child_node_id -child_node_id -> {subquery_group_id, subquery_id, next_condition_id} -subquery_id -> [{subquery_group_id, child_node_id}] +{:group, group_key} -> group_id +{:child, group_id, subquery_id} -> child_node_id +{:child_meta, child_node_id} -> {group_id, subquery_id, polarity, next_condition_id} +{:child_shape, child_node_id} -> {shape_handle, branch_key} +{:shape_child, shape_handle} -> child_node_id +{:shape_subquery, shape_handle, subquery_ref} -> {subquery_id, filter_time} +{:fallback, shape_handle} -> true ``` -Using small integer ids avoids repeating large tuples, field keys, and shape -handles in per-value ETS rows. - -For positive groups, the routing index stores values that can be members at -some retained logical time: +Positive routing keeps value-keyed entries: ```text -{positive_value, subquery_group_id, value} -> [child_node_id] +{:positive, group_id, value} -> child_node_id ``` -For negative groups, the index stores all negative children for the group: +Negated routing keeps group-keyed entries: ```text -{negative_children, subquery_group_id} -> [child_node_id] +{:negated, group_id} -> child_node_id ``` -The negative path cannot avoid considering all affected children when a value -is absent from the subquery views. That is acceptable because those children are -affected. +This replaces per-shape value membership rows with per-child routing rows and a +shared membership view. -#### Affected Shapes +### First-Time Child Creation -For a root-table change, `SubqueryIndex.affected_shapes/4` evaluates the -left-hand side value for the subquery node. +First-time child creation must seed synchronously. -If evaluation fails, it falls back to all children in the group. +When `SubqueryIndex` creates a new `child_node_id` for +`{subquery_group_id, subquery_id}`, it must: -If a subquery is not ready, the child node is routed conservatively. +1. Ensure the dependency materializer has populated `MultiTimeView` and marked + the subquery ready. +2. Create the child `WhereCondition`. +3. Insert the outer shapes into the child condition. +4. Seed positive routing for every value in + `MultiTimeView.values(subquery_id, current_time)`. +5. Add negated group routing if the group is negated. +6. Remove fallback only after the child is fully routable. -For a positive group: +This is `O(number_of_values_in_subquery)` for the first child of a +`{group, subquery_id}` pair. That cost is acceptable because it happens on +child creation, not on every consumer using the same child. -```elixir -for child_node_id <- positive_children_for_value(group_id, value), - subquery_id = subquery_id_for_child(child_node_id), - MultiTimeView.member_at_some_time?(subquery_id, value) do - WhereCondition.affected_shapes(child_node_id, record) -end -``` +### Routing -For a negative group: +Positive routing should route a root-table value to a child if the value is a +member of the child subquery at any retained logical time: ```elixir -for child_node_id <- all_negative_children(group_id), - subquery_id = subquery_id_for_child(child_node_id), - not MultiTimeView.member_at_all_times?(subquery_id, value) do - WhereCondition.affected_shapes(child_node_id, record) -end +MultiTimeView.member_at_some_time?(subquery_id, value) ``` -This keeps routing broad enough for all consumers reading any retained logical -time. - -#### Shape Addition - -Adding an outer shape to an existing `{group, subquery_id}` child is near O(1): -the shape is added to the child `WhereCondition`. - -Creating the first child for `{group, subquery_id}` requires indexing current -values for that subquery in the group. That is O(number of values in the -subquery), which is acceptable and unavoidable. First-time child creation should -seed synchronously so the child is fully routable before the shape is considered -indexed. - -#### Shape Removal - -Removing an outer shape removes it from the child `WhereCondition`. +This is conservative. If some consumers still read an old time and others read +a new time, both old and new members remain routable until compaction proves no +consumer can read the old time. -If the child becomes empty, remove the `{group, subquery_id}` child and update -the group value indexes by iterating the values for `subquery_id` from -`MultiTimeView`. This is proportional to the number of values in that subquery, -not to the total number of shapes or subqueries. - -### Materializer - -The materializer continues to track dependency shape membership from the -dependency log. - -It changes in three ways: - -1. On initial load, it populates `MultiTimeView` for the subquery at logical - time `0`, then marks the subquery ready. -2. On each committed batch that produces net move events, it increments the - subquery logical time. -3. Before sending `{:materializer_changes, ...}` to subscribers, it writes the - move events to `MultiTimeView` at the new logical time. - -The subscriber payload should include both the old and new logical time: +Negated routing should enumerate the negated children for the group and keep +children where the value is not a member at all retained times: ```elixir -%{ - move_in: [{value, original_string}], - move_out: [{value, original_string}], - txids: [txid], - from_time: old_time, - to_time: new_time -} +not MultiTimeView.member_at_all_times?(subquery_id, value) ``` -If a committed batch has no net membership move, logical time does not need to -advance. - -The existing `Materializer.LinkValues` ETS cache should be removed or replaced -by `MultiTimeView` rather than kept as a second full shared copy. - -### Consumer Event Handlers +This is `O(number_of_affected_shapes)` for large negated groups. That is +acceptable because a value absent from a large negated group genuinely affects +all of those shapes. -Consumers store logical times instead of materialized subquery views. - -The steady handler keeps: +Exact filter verification uses the consumer's filter time: ```elixir -%{subquery_ref => logical_time} +MultiTimeView.member?(subquery_id, typed_value, filter_time) ``` -`Shape.convert_change/3` and DNF metadata projection need to accept a membership -callback instead of requiring a concrete `MapSet` view: - -```elixir -fn subquery_ref, value, time -> - MultiTimeView.member?(subquery_id, value, time) -end -``` +`WhereClause.subquery_member_from_index/2` should therefore resolve +`shape_handle + subquery_ref` to `{subquery_id, filter_time}` and call the +shared view. The callback remains the boundary used by +`WhereClause.includes_record?/3`. -In steady state, old and new records use the same logical time. +### Materializer Integration -During a buffered move-in, `ActiveMove` stores: +The materializer owns the source of truth for a dependency subquery. It should +populate `MultiTimeView` during initial materialization and mark the subquery +ready only after the full initial view is visible. -```elixir -times_before_move -times_after_move -``` +When a committed dependency change alters membership, the materializer should: -instead of: +1. Read the current logical time `a`. +2. Increment to logical time `b`. +3. Write the transition into `MultiTimeView` at `b`. +4. Update positive routing before emitting the move if the value is newly + routable at some retained time. +5. Emit the dependency move with `from_time: a`, `to_time: b`, `subquery_id`, + changed values, and move kind. -```elixir -views_before_move -views_after_move -``` +Consumers must not observe a move event whose target time is absent from +`MultiTimeView`. -Buffered transactions before the splice boundary are converted using -`times_before_move`. Buffered transactions after the splice boundary are -converted using `times_after_move`. +### Consumer Registration -After the move is spliced and the consumer becomes steady at `to_time`, it -calls: +Consumers register for each subquery they read. Registration should be +serialized through the dependency materializer so the returned time and the +shared view are consistent: ```elixir -SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) +{:ok, current_time} = + Materializer.register_subquery_consumer( + subquery_id, + outer_shape_handle, + self() + ) ``` -#### Move-In Queries +The registration side effects are: -Move-in queries currently build SQL from whole before and after views. The new -implementation should avoid retaining large views in the consumer process. +- wait until the dependency materializer has finished initial population +- register the consumer with `SubqueryProgressMonitor` +- set the consumer's initial `required_time` to `current_time` +- return `current_time` to the caller -Preferred approach: +This replaces the current `Materializer.get_link_values/1` setup path for +subquery event handlers. The handler should keep compact references such as: -- Build the triggering dependency candidate predicate from the move delta - values when possible. -- Read full view values at a specific time only for positions that require - exclusion logic. -- If full views are required, materialize them inside the task process that - runs the query so the memory is released when the task exits. +```elixir +%{ + ["$sublink", "0"] => %{subquery_id: dep_handle, time: current_time} +} +``` -This is important because replacing long-lived consumer views with -short-lived task views is where much of the memory win comes from. +not `MapSet` views. -### SubqueryProgressMonitor +The monitor should track consumers by process monitor plus registered +subqueries so dead consumers automatically release pinned times. An explicit +unregister path can be added for normal shutdown, but correctness must not +depend on it. -`SubqueryProgressMonitor` tracks the earliest logical time still needed by live -outer consumers. +### Consumer Move Handling -Registration must happen atomically with choosing the consumer's starting -logical time. The materializer should provide a serialized registration call, -for example: +For a move from time `a` to time `b`, `ActiveMove` should store times, not +views: ```elixir -{:ok, current_time} = - Materializer.register_subquery_consumer( - subquery_id, - outer_shape_handle, - self() - ) +%ActiveMove{ + subquery_id: subquery_id, + from_time: a, + to_time: b, + values: values +} ``` -The materializer handles the call in its GenServer, reads its current logical -time, registers the outer consumer with the progress monitor, and returns that -time to the consumer. This prevents a race where the materializer advances and -compacts a time before the new consumer is registered as needing it. - -The monitor stores the returned time as the consumer's internal -`required_time`. - -When a consumer finishes a move from `from_time` to `to_time`, it calls: +Elixir-side evaluation of buffered transactions should use callbacks into +`MultiTimeView`: ```elixir -SubqueryProgressMonitor.notify_processed_up_to(from_time, subquery_id) +before_member? = fn ref, value -> member?(ref, value, a) end +after_member? = fn ref, value -> member?(ref, value, b) end ``` -The monitor then advances that consumer's `required_time` to `from_time + 1`. +For SQL move-in queries, the first implementation can still materialize +query-local arrays by calling `MultiTimeView.values(subquery_id, time)`. The +important change is that these arrays are transient query parameters, not +long-lived per-consumer state. -The monitor maintains two ETS indexes: +After the move is spliced and the consumer no longer needs time `a`, it calls: -```text -{subquery_id, consumer_shape_handle} -> required_time -{subquery_id, required_time, consumer_shape_handle} -> true +```elixir +SubqueryProgressMonitor.notify_processed_up_to(a, subquery_id) ``` -The first index makes updates O(1). The second index makes the minimum -required time for a subquery cheap to read. +The consumer's exact-filter time is separate from this retention notification. +It should advance to `b` at the same point the current implementation would +update per-shape membership rows for subsequent routing. The important +invariant is that live routing must not under-route, while `required_time` +continues to pin `a` until the consumer no longer needs the old view. -If `notify_processed_up_to/2` infers consumer identity from the caller process, -the monitor also needs a lookup from registered consumer pid to -`consumer_shape_handle`. +### Querying Changes -When the minimum changes, the monitor notifies `MultiTimeView`: +`Querying.move_in_where_clause/5` currently receives +`views_before_move` and `views_after_move` maps. Replace those maps with a view +resolver that can provide values for a subquery ref at a logical time: ```elixir -MultiTimeView.set_min_required_time(subquery_id, min_required_time) +values_for.(subquery_ref, time) ``` -When an outer shape is removed, the monitor removes that consumer from every -subquery it was registered for and recomputes the affected minima. - -### Concurrency Model +Initial implementation can adapt this resolver back into arrays at the SQL +boundary, preserving existing SQL generation behavior. A later optimization can +special-case the triggering subquery position and use only the changed values +for candidate selection when the DNF plan makes that safe. + +This keeps the first implementation smaller while still removing long-lived +view copies. + +### Failure Modes + +If `MultiTimeView` is not ready for a subquery, shapes using that subquery must +stay in fallback routing. They must not be marked ready. + +If a consumer dies while it pins an old time, `SubqueryProgressMonitor` must +release its registration via the process monitor. Otherwise compaction can be +blocked indefinitely. + +If a dependency materializer is removed, `MultiTimeView.remove_subquery/1` must +remove the view and `SubqueryIndex` must remove the children and participants +associated with that subquery without scanning unrelated shapes. + +If compaction falls behind, correctness is preserved but routing becomes more +conservative and histories grow. Add telemetry so this is visible. + +### Telemetry + +Add enough telemetry to prove or disprove the design: + +- number of values per subquery +- number of retained histories per subquery +- max and average history length +- min/current logical time gap per subquery +- number of registered consumers per subquery +- number of child nodes per subquery group +- first-child synchronous seed duration +- shape removal duration +- transient SQL move-in array size + +### Complexity Check + +- **Is this the simplest approach?** No. The simplest immediate fix is adding a + reverse index for shape-owned values or using tombstones. Those approaches do + less architectural work, but they keep or increase the duplicated full-view + memory that caused the problem. This proposal is more complex because it + crosses the materializer, event handler, querying, and filter index + boundaries, but it removes both major long-lived duplicate view pools. +- **What could we cut?** The first implementation can keep existing SQL array + generation, materializing arrays only at query time. It can also postpone + aggressive history encoding, background compaction tuning, and cross-handle + subquery interning. +- **What's the 90/10 solution?** Implement `MultiTimeView`, serialized + registration, per-consumer logical times, and shared child routing. Keep + move-in SQL generation structurally the same by resolving values from the + shared view at the SQL boundary. Add telemetry before optimizing the query + format further. -Most writes are already serialized by existing processes: +## Open Questions -- Shape addition and removal happen through the ShapeLogCollector path. -- Dependency changes are applied by materializers. -- Outer consumers process events synchronously when ShapeLogCollector publishes - to them. +Unresolved questions that need further discussion or will be determined during +implementation: -`MultiTimeView` writes happen in the materializer before it sends -`materializer_changes` to subscribers. +| Question | Options | Resolution Path | +|----------|---------|-----------------| +| **How should `values(subquery_id, time)` expose large views?** | Materialized `MapSet`, stream, both | Start with query-local materialization for compatibility, then prototype streaming or chunked array construction if telemetry shows spikes. | +| **Where should exact filter times live?** | In `SubqueryIndex` participant rows, in `SubqueryProgressMonitor`, or in consumer-owned state with callbacks | Decide during implementation. The filter needs fast `shape_handle + subquery_ref -> time` lookup, so `SubqueryIndex` is the likely owner. | +| **When should positive routing rows be removed after compaction?** | Opportunistically on read/write, periodic cleanup, immediate cleanup when min time advances | Implement opportunistic plus periodic cleanup first. Add immediate cleanup only if stale positive routes are expensive. | +| **Should long histories switch representation?** | Keep flat lists, switch to tuples/arrays after a threshold, or compact eagerly | Keep flat lists for v1 and add telemetry for max history length before adding another representation. | -`SubqueryIndex` topology changes happen when shapes are added or removed from -the filter. +## Definition of Success -`SubqueryIndex` reads happen from ShapeLogCollector while routing replication -changes. +### Primary Hypothesis -Consumer reads from `MultiTimeView` can happen concurrently with writes. This -is safe because membership is always read at an explicit logical time, and ETS -updates replace complete membership-history values atomically. +> We believe that implementing shared subquery logical-time views will enable +> the issue #4279 hypothesis: subquery indexing can become scalable for shape +> add/remove and memory use while preserving v1.6 subquery move correctness. +> +> We'll know we're right if shared subqueries no longer allocate full +> per-consumer dependency views in steady state, shape removal no longer scans +> value-keyed membership rows owned by unrelated shapes, and existing subquery +> move correctness tests continue to pass. +> +> We'll know we're wrong if retained histories grow without bound under normal +> consumer lag, move-in query memory still dominates production incidents, or +> the cross-subsystem complexity creates correctness regressions compared with +> the current per-consumer view model. -The ready flag is important. Until a subquery has been seeded into -`MultiTimeView` and any group routing rows have been created, routing must be -conservative. +### Functional Requirements -## Expected Benefits +| Requirement | Acceptance Criteria | +|-------------|---------------------| +| Shared subquery view | One `MultiTimeView` view exists per `subquery_id`, and steady-state consumers do not store full `MapSet` views. | +| Per-consumer logical time | Each consumer can evaluate subquery membership at its own logical time. | +| Correct registration | Consumer registration is serialized with the materializer and returns a current logical time whose view is ready. | +| Progress notification | Consumers call `notify_processed_up_to(time, subquery_id)` after finishing moves, and compaction uses the minimum required time. | +| Synchronous first child seed | First-time child creation seeds routing for the current view before removing fallback. | +| Positive routing correctness | Values that are members at any retained time route to the relevant child node. | +| Negated routing correctness | Negated groups route conservatively and filter with `member_at_all_times?/2`. | +| Shape removal scalability | Removing a shape follows participant and child rows, not all subquery values for unrelated shapes. | +| Move-in compatibility | Existing move-in SQL behavior can be produced from logical-time views without long-lived before/after `MapSet` copies. | +| Observability | Telemetry reports retained time gaps, history sizes, seed duration, and removal duration. | -- One retained membership view per subquery instead of one per outer shape. -- Consumer processes retain logical times instead of large `MapSet` views. -- Move-in buffering retains before/after logical times instead of before/after - `MapSet` views. -- Subquery removal is proportional to the removed subquery's values and group - entries, not to total stack size. -- Positive routing remains value-keyed and efficient. +### Learning Goals -## Risks +1. Measure how large retained logical-time windows become under realistic + consumer lag. +2. Measure whether transient move-in SQL arrays remain a material memory cost + after removing long-lived view copies. +3. Determine whether flat list histories are sufficient or whether a threshold + representation is needed. +4. Determine whether conservative positive routing creates measurable extra + filter work before compaction catches up. -### Off-by-One Compaction +## Alternatives Considered -The biggest correctness risk is compacting away a logical time that some -consumer still needs. +These alternatives are based on the discussion and rejected approaches in +PR #4280. -The invariant is: +### Alternative 1: Add `shape_handle -> all values` -```text -MultiTimeView may drop transitions at or before min_required_time -after rewriting the entry to preserve membership at min_required_time. +**Description:** Add a reverse index from each shape to the full set of values +it has inserted into `SubqueryIndex`. -min_required_time = min(required_time_by_live_consumer) -``` +**Why not:** This improves shape removal, but it adds another full per-shape +dependency view. It makes the removal path easier by increasing the same memory +duplication this RFC is trying to remove. -Tests should cover move-in, move-out, repeated toggles, consumer registration, -consumer removal, and compaction across all of those cases. +### Alternative 2: Tombstone Removed Shapes And Clean Later -### Negated Routing Cost +**Description:** Mark removed shapes as tombstoned and clean their value-keyed +membership rows asynchronously. -For negated subquery groups, a value absent from all subqueries affects all -children in the group. This can be large, but it is proportional to the number -of affected shapes. +**Why not:** This is useful as an emergency mitigation, but it is not a +structural memory fix. It leaves stale rows in the hot routing path and +requires liveness checks or cleanup debt elsewhere. -The implementation should avoid extra memory-heavy complement indexes unless -there is evidence they are necessary. +### Alternative 3: One Global Widened Filter -### Move-In Query Memory +**Description:** Store one widened filter for each subquery and route every +value that might match any participant, relying on downstream exact filtering. -If move-in query generation materializes full before and after views in the -consumer process, the design will keep a major source of memory duplication. +**Why not:** A slow or stalled consumer can keep the shared filter broad and +over-route work for every other participant. This preserves correctness, but it +can move cost from memory to sustained routing and filtering work. -Full view materialization should be avoided where possible and isolated to -short-lived task processes where not possible. +### Alternative 4: Intern Full Dependency Views -### Fallback Windows +**Description:** Deduplicate identical full dependency views by interning +`MapSet` values or equivalent view structures. -Unready subqueries and not-yet-seeded group routing must route conservatively. -This may temporarily over-route, but must not under-route. +**Why not:** This handles exact equality at a point in time, but one-value +moves immediately create new views or require a second delta representation. +At that point the design becomes a versioned or sparse-delta view. Logical time +models that state directly. -## Testing Plan +### Alternative 5: Versioned Lazy Exception Clearing -Add focused unit tests for `MultiTimeView`: +**Description:** Keep sparse exceptions and clear or promote them lazily with +versions instead of doing eager cleanup. + +**Why not:** This can reduce some hot-path work, but it adds versioning and +cleanup complexity while retaining a separate exception model. This is better +as a follow-up optimization if measurements show cleanup cost is high. + +### Alternative 6: Shared Base View With Sparse XOR Exceptions -- membership at exact logical times -- `member_at_some_time?/2` -- `member_at_all_times?/2` -- move-in and move-out transitions -- repeated toggles -- compaction after `set_min_required_time/2` -- subquery removal by ordered key range +**Description:** The design in PR #4280 stores one base dependency view per +cohort and stores sparse per-participant XOR exceptions for values where a +participant temporarily differs from the base. -Add focused unit tests for `SubqueryProgressMonitor`: +**Why not:** This is a lower-risk, index-focused approach and may still be the +right short-term fix if this RFC is too broad. However, it leaves consumer-held +before/after views in place and represents temporary divergence as +per-participant exceptions instead of as consumers reading different logical +times. +The logical-time design is a broader refactor, but it addresses the duplicated +state in both `SubqueryIndex` and consumer event handlers. -- registration at current time -- `notify_processed_up_to/2` -- minimum required time updates -- consumer removal -- multiple consumers at different times +## Revision History -Add `SubqueryIndex` tests: +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 0.1 | 2026-05-18 | robacourt | Initial draft using the Stratovolt RFC template and alternatives from PR #4280. | -- positive group routing -- negative group routing -- shared child nodes per `{group, subquery_id}` -- conservative routing for unready subqueries -- child removal after the last outer shape is removed -- no full-table scan on subquery removal +--- -Update consumer event-handler tests: +## RFC Quality Checklist -- steady conversion using logical times -- buffered move-in using before and after logical times -- queued moves across multiple logical times -- progress notifications after splice -- negated move-in and move-out behavior +Before submitting for review, verify: -Keep or extend integration tests for: +**Alignment** +- [x] RFC implements the working issue hypothesis, with no separate PRD. +- [x] API naming matches ElectricSQL conventions. +- [x] Success criteria link back to the issue hypothesis. -- dependency move-in -- dependency move-out -- nested subqueries -- subqueries combined with non-subquery predicates -- rows moving between two dependency values in one transaction - -## Open Questions +**Calibration for Level 1-2 PMF** +- [x] This is the smallest version of the logical-time design that validates + the memory hypothesis. +- [x] Non-goals explicitly defer protocol changes, DNF redesign, and deeper + query optimization. +- [x] Complexity Check section is filled out honestly. +- [x] An engineer could start implementing tomorrow. -- Should `MultiTimeView` expose `values(subquery_id, time)` as a materialized - `MapSet`, a stream, or both? +**Completeness** +- [x] Happy path is clear. +- [x] Critical failure modes are addressed. +- [x] Open questions are acknowledged, not glossed over. From e14978eea8c902406d56659293a0247325241eaa Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 20:49:15 +0100 Subject: [PATCH 11/40] Remove use of phrase 'filter time' --- docs/rfcs/subquery-index.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 84feeff5a3..fa35859166 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -132,7 +132,7 @@ subquery: ```text MultiTimeView[{subquery_id, value}] -> membership_history -consumer[{shape_handle, subquery_ref}] -> {subquery_id, filter_time} +consumer[{shape_handle, subquery_ref}] -> {subquery_id, logical_time} ``` Consumers no longer copy the subquery view. They register each subquery they @@ -250,10 +250,11 @@ Consumers register at the logical time they are starting from. If a consumer starts from current logical time `t`, its initial `required_time` is `t` because it may read time `t`. -`required_time` is a retention bound, not necessarily the same as the time used -for live exact filter checks. During an active move, a consumer may need the old -time for buffered conversion or move-in query work while live exact checks have -already advanced to the new time. The implementation must keep those two uses +`required_time` is a retention bound. It is separate from the consumer's current +logical time for a specific subquery. During an active move, a consumer may need +the old time for buffered conversion or move-in query work while its current +logical time for that subquery has already advanced to the new time. The +implementation must keep `required_time` and per-subquery `logical_time` explicit. ### MultiTimeView @@ -369,7 +370,7 @@ Suggested logical rows: {:child_meta, child_node_id} -> {group_id, subquery_id, polarity, next_condition_id} {:child_shape, child_node_id} -> {shape_handle, branch_key} {:shape_child, shape_handle} -> child_node_id -{:shape_subquery, shape_handle, subquery_ref} -> {subquery_id, filter_time} +{:shape_subquery, shape_handle, subquery_ref} -> {subquery_id, logical_time} {:fallback, shape_handle} -> true ``` @@ -432,14 +433,15 @@ This is `O(number_of_affected_shapes)` for large negated groups. That is acceptable because a value absent from a large negated group genuinely affects all of those shapes. -Exact filter verification uses the consumer's filter time: +Exact filter verification uses the consumer's current logical time for the +requested subquery: ```elixir -MultiTimeView.member?(subquery_id, typed_value, filter_time) +MultiTimeView.member?(subquery_id, typed_value, logical_time) ``` `WhereClause.subquery_member_from_index/2` should therefore resolve -`shape_handle + subquery_ref` to `{subquery_id, filter_time}` and call the +`shape_handle + subquery_ref` to `{subquery_id, logical_time}` and call the shared view. The callback remains the boundary used by `WhereClause.includes_record?/3`. @@ -533,11 +535,12 @@ After the move is spliced and the consumer no longer needs time `a`, it calls: SubqueryProgressMonitor.notify_processed_up_to(a, subquery_id) ``` -The consumer's exact-filter time is separate from this retention notification. -It should advance to `b` at the same point the current implementation would -update per-shape membership rows for subsequent routing. The important -invariant is that live routing must not under-route, while `required_time` -continues to pin `a` until the consumer no longer needs the old view. +The consumer's current logical time for that subquery is separate from this +retention notification. It should advance to `b` at the same point the current +implementation would update per-shape membership rows for subsequent routing. +The important invariant is that live routing must not under-route, while +`required_time` continues to pin `a` until the consumer no longer needs the old +view. ### Querying Changes @@ -613,7 +616,7 @@ implementation: | Question | Options | Resolution Path | |----------|---------|-----------------| | **How should `values(subquery_id, time)` expose large views?** | Materialized `MapSet`, stream, both | Start with query-local materialization for compatibility, then prototype streaming or chunked array construction if telemetry shows spikes. | -| **Where should exact filter times live?** | In `SubqueryIndex` participant rows, in `SubqueryProgressMonitor`, or in consumer-owned state with callbacks | Decide during implementation. The filter needs fast `shape_handle + subquery_ref -> time` lookup, so `SubqueryIndex` is the likely owner. | +| **Where should per-subquery logical times live?** | In `SubqueryIndex` participant rows, in `SubqueryProgressMonitor`, or in consumer-owned state with callbacks | Decide during implementation. Exact membership checks need fast `shape_handle + subquery_ref -> {subquery_id, logical_time}` lookup, so `SubqueryIndex` is the likely owner. | | **When should positive routing rows be removed after compaction?** | Opportunistically on read/write, periodic cleanup, immediate cleanup when min time advances | Implement opportunistic plus periodic cleanup first. Add immediate cleanup only if stale positive routes are expensive. | | **Should long histories switch representation?** | Keep flat lists, switch to tuples/arrays after a threshold, or compact eagerly | Keep flat lists for v1 and add telemetry for max history length before adding another representation. | @@ -640,7 +643,7 @@ implementation: | Requirement | Acceptance Criteria | |-------------|---------------------| | Shared subquery view | One `MultiTimeView` view exists per `subquery_id`, and steady-state consumers do not store full `MapSet` views. | -| Per-consumer logical time | Each consumer can evaluate subquery membership at its own logical time. | +| Per-consumer per-subquery logical time | Each consumer can evaluate each subquery at that subquery's own logical time. | | Correct registration | Consumer registration is serialized with the materializer and returns a current logical time whose view is ready. | | Progress notification | Consumers call `notify_processed_up_to(time, subquery_id)` after finishing moves, and compaction uses the minimum required time. | | Synchronous first child seed | First-time child creation seeds routing for the current view before removing fallback. | From 90c981bd8f627a6898aabd78d8bb8e744de55ac1 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 18 May 2026 21:04:07 +0100 Subject: [PATCH 12/40] Add examples --- docs/rfcs/subquery-index.md | 589 ++++++++++++++++++++++++++++++++++++ 1 file changed, 589 insertions(+) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index fa35859166..1d7e7bf192 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -368,6 +368,7 @@ Suggested logical rows: {:group, group_key} -> group_id {:child, group_id, subquery_id} -> child_node_id {:child_meta, child_node_id} -> {group_id, subquery_id, polarity, next_condition_id} +{:subquery_child, subquery_id} -> child_node_id {:child_shape, child_node_id} -> {shape_handle, branch_key} {:shape_child, shape_handle} -> child_node_id {:shape_subquery, shape_handle, subquery_ref} -> {subquery_id, logical_time} @@ -445,6 +446,594 @@ MultiTimeView.member?(subquery_id, typed_value, logical_time) shared view. The callback remains the boundary used by `WhereClause.includes_record?/3`. +### Operation Examples And Costs + +Use this concrete setup for the examples: + +```sql +-- subquery_id = s7, current logical time 0 +SELECT id FROM users WHERE company_id = 7 +-- current values: 10, 20 + +-- subquery_id = s8, current logical time 0 +SELECT id FROM users WHERE company_id = 8 +-- current values: 30 +``` + +Outer shapes: + +```sql +-- shape_a and shape_b share the same positive group and subquery. +WHERE user_id IN (SELECT id FROM users WHERE company_id = 7) + +-- shape_c uses the same positive group but a different subquery. +WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) + +-- shape_n uses a negated group for s7. +WHERE user_id NOT IN (SELECT id FROM users WHERE company_id = 7) +``` + +Symbols used below: + +- `V_s`: values in subquery `s` over the retained window. +- `H_v`: transition history length for one `{subquery_id, value}` row. +- `C_s`: child nodes attached to subquery `s`. +- `P_c`: outer shapes attached to child node `c`. +- `K`: changed values in one dependency move. +- `N_g`: child nodes in a negated group. +- `R`: root-table candidate rows returned by a move-in SQL query. + +#### Initial `MultiTimeView` State + +The initial materializer state for `s7` stores one row per dependency value, +not one row per outer shape: + +```text +{s7, 10} -> [] +{s7, 20} -> [] +{:current_time, s7} -> 0 +{:min_required_time, s7} -> 0 +{:ready, s7} -> true +``` + +The empty history means the value is present for the whole retained window. + +Memory is `O(V_s)` for the shared view. In this example, `shape_a` and +`shape_b` do not duplicate `{10, 20}`. + +#### `register_subquery_consumer` + +Before an outer consumer can read `s7`, it registers through the materializer: + +```elixir +{:ok, 0} = + Materializer.register_subquery_consumer( + s7, + shape_a, + consumer_pid_a + ) +``` + +Progress monitor rows are conceptually: + +```text +{s7, shape_a} -> required_time 0 +{s7, 0, shape_a} -> true +``` + +The shape's subquery reference is then recorded by the indexing/setup path: + +```text +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} +``` + +If registration is called from `add_shape`, this row is inserted once as part +of that setup path; it is shown here to make the registration result explicit. + +What is evaluated: + +1. Wait until `s7` is ready. +2. Read `{:current_time, s7}`. +3. Insert progress monitor rows for `shape_a`. +4. Return `0` to the consumer. + +Cost: + +```text +O(wait_until_ready + progress_index_insert) +``` + +No dependency values are copied. Memory added is +`O(number_of_subqueries_read_by_shape)`. + +#### `add_shape`: First Positive Shape For `{group, subquery}` + +Adding `shape_a` creates a positive group `g_user_pos` and a child `c_s7_pos` +for `{g_user_pos, s7}`. + +Rows stored: + +```text +{:group, {:node_1, :user_id, :positive}} -> g_user_pos +{:child, g_user_pos, s7} -> c_s7_pos +{:child_meta, c_s7_pos} -> {g_user_pos, s7, :positive, wc_s7_pos} +{:subquery_child, s7} -> c_s7_pos + +{:child_shape, c_s7_pos} -> {shape_a, branch_a} +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} + +{:positive, g_user_pos, 10} -> c_s7_pos +{:positive, g_user_pos, 20} -> c_s7_pos +``` + +The child `WhereCondition` `wc_s7_pos` also stores `shape_a` with the residual +non-subquery predicates for the branch. + +What is evaluated: + +1. Compile or reuse the DNF subquery group key. +2. Register the consumer with the dependency materializer and get logical time + `0`. +3. Create the child `WhereCondition`. +4. Insert `shape_a` into the child condition. +5. Synchronously seed positive routing from `MultiTimeView.values(s7, 0)`. +6. Remove fallback for `shape_a`. + +Cost: + +```text +O(number_of_subquery_occurrences_in_shape + V_s + child_where_insert) +``` + +The `V_s` term only applies because this is the first child for +`{g_user_pos, s7}`. Memory added is `O(V_s)` positive routing rows for the +child plus `O(number_of_subquery_occurrences_in_shape)` participant rows. + +#### `add_shape`: Additional Shape Sharing An Existing Child + +Adding `shape_b` finds the existing child `c_s7_pos`. + +Rows added: + +```text +{:child_shape, c_s7_pos} -> {shape_b, branch_b} +{:shape_child, shape_b} -> c_s7_pos +{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} +``` + +No new rows are added for values `10` or `20`. + +What is evaluated: + +1. Resolve `{g_user_pos, s7}` to `c_s7_pos`. +2. Register the consumer and get logical time `0`. +3. Insert `shape_b` into the child condition. +4. Remove fallback for `shape_b`. + +Cost: + +```text +O(number_of_subquery_occurrences_in_shape + child_where_insert) +``` + +Memory added is per-shape metadata only, not `O(V_s)`. + +#### `add_shape`: Same Group, Different Subquery + +Adding `shape_c` reuses group `g_user_pos`, but creates child `c_s8_pos` for +`{g_user_pos, s8}`. + +Rows added include: + +```text +{:child, g_user_pos, s8} -> c_s8_pos +{:child_meta, c_s8_pos} -> {g_user_pos, s8, :positive, wc_s8_pos} +{:subquery_child, s8} -> c_s8_pos +{:positive, g_user_pos, 30} -> c_s8_pos +{:shape_subquery, shape_c, ["$sublink", "0"]} -> {s8, 0} +``` + +Cost is `O(V_s8)` for the first `s8` child in this group. This is expected: +`s8` has different dependency values from `s7`. + +#### `add_shape`: Negated Shape + +Adding `shape_n` creates or reuses a negated group `g_user_neg` and child +`c_s7_neg`. + +Rows stored: + +```text +{:group, {:node_2, :user_id, :negated}} -> g_user_neg +{:child, g_user_neg, s7} -> c_s7_neg +{:child_meta, c_s7_neg} -> {g_user_neg, s7, :negated, wc_s7_neg} +{:subquery_child, s7} -> c_s7_neg +{:negated, g_user_neg} -> c_s7_neg + +{:child_shape, c_s7_neg} -> {shape_n, branch_n} +{:shape_child, shape_n} -> c_s7_neg +{:shape_subquery, shape_n, ["$sublink", "0"]} -> {s7, 0} +``` + +No per-value negated routing rows are stored. + +Cost: + +```text +O(number_of_subquery_occurrences_in_shape + child_where_insert) +``` + +Memory added for negated routing is `O(1)` per child, not `O(V_s)`. + +#### `affected_shapes`: Positive Group + +For a root-table record: + +```text +%{"user_id" => 10} +``` + +Routing does: + +1. Evaluate the left-hand side `user_id` to `10`. +2. Look up `{:positive, g_user_pos, 10}` and get `[c_s7_pos]`. +3. Evaluate child condition `wc_s7_pos`, which considers `shape_a` and + `shape_b`. +4. For each candidate shape, exact subquery checks resolve: + +```text +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} +{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} +MultiTimeView.member?(s7, 10, 0) -> true +``` + +Both shapes are affected. + +Cost: + +```text +O(children_for_value + child_where_eval + exact_subquery_checks * H_v) +``` + +For this example, `children_for_value = 1`. There is no scan of all shapes and +no scan of all values in `s7`. + +#### `affected_shapes`: Positive Group With Divergent Consumer Times + +Suppose the materializer adds value `30` to `s7` at logical time `1`: + +```text +{s7, 30} -> [:out, 1] +{:current_time, s7} -> 1 +{:positive, g_user_pos, 30} -> c_s7_pos +``` + +Now `shape_a` has advanced to logical time `1`, but `shape_b` still reads +logical time `0`: + +```text +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} +``` + +For: + +```text +%{"user_id" => 30} +``` + +routing finds `c_s7_pos` because `30` is a member at some retained time. Exact +checks then split the result: + +```text +MultiTimeView.member?(s7, 30, 1) -> true +MultiTimeView.member?(s7, 30, 0) -> false +``` + +Only `shape_a` is affected. + +Cost remains: + +```text +O(children_for_value + child_where_eval + exact_subquery_checks * H_v) +``` + +The extra memory for the move is one history row for `{s7, 30}` plus one +positive routing row per positive child for `s7` in that group. + +#### `affected_shapes`: Negated Group + +For: + +```text +%{"user_id" => 30} +``` + +while `{s7, 30} -> [:out, 1]` is retained, `30` is absent at time `0` and +present at time `1`. Negated routing does: + +1. Look up `{:negated, g_user_neg}` and get `[c_s7_neg]`. +2. Keep `c_s7_neg` because: + +```elixir +not MultiTimeView.member_at_all_times?(s7, 30) +``` + +3. Evaluate `wc_s7_neg` and exact membership at each candidate shape's + subquery logical time. + +For `shape_n` at logical time `0`, `NOT IN s7` is true for `30`. If it later +advances to logical time `1`, `NOT IN s7` is false for `30`. + +Cost: + +```text +O(N_g * H_v + child_where_eval + exact_subquery_checks * H_v) +``` + +This is intentionally proportional to the number of affected negated children. +No complement index is stored. + +#### Dependency Move: Add Or Remove Values + +For a move that adds `30` to `s7`: + +```text +from_time = 0 +to_time = 1 +changed_values = [30] +``` + +Rows written: + +```text +{s7, 30} -> [:out, 1] +{:current_time, s7} -> 1 +{:positive, g_user_pos, 30} -> c_s7_pos +``` + +Rows not written: + +```text +{:membership, shape_a, ["$sublink", "0"], 30} +{:membership, shape_b, ["$sublink", "0"], 30} +``` + +What is evaluated: + +1. Update the `MultiTimeView` history for each changed value. +2. Find children from `{:subquery_child, s7}`. +3. For each positive child, insert a positive routing row if the value changed + from not routable to routable for the retained window. +4. Emit a move event containing `from_time`, `to_time`, `subquery_id`, and + changed values. + +Cost: + +```text +O(K * (history_update + C_s)) +``` + +For a remove of `20` from `s7` at time `2`, the history becomes: + +```text +{s7, 20} -> [:in, 2] +``` + +The positive routing row for `20` stays while any retained time still contains +`20`. It is removed later when compaction proves `member_at_some_time?(s7, 20)` +is false. + +#### Consumer Move Handling + +When `shape_a` receives the `s7` move from `0` to `1`, `ActiveMove` stores: + +```elixir +%ActiveMove{ + subquery_id: s7, + from_time: 0, + to_time: 1, + values: [{30, "30"}] +} +``` + +It does not store: + +```text +views_before_move: MapSet.new([10, 20]) +views_after_move: MapSet.new([10, 20, 30]) +``` + +Buffered row conversion evaluates exact membership by calling +`MultiTimeView.member?/3` at `from_time` or `to_time`. Move-in SQL may +materialize `values(s7, 1)` as a query-local parameter array, but that memory +belongs to the query task and is released after the query. + +Steady memory added per active move is: + +```text +O(number_of_changed_values + number_of_subquery_refs) +``` + +not `O(V_s)`. + +#### `notify_processed_up_to` And Compaction + +After `shape_a` no longer needs time `0`, it calls: + +```elixir +SubqueryProgressMonitor.notify_processed_up_to(0, s7) +``` + +Progress monitor rows conceptually change from: + +```text +{s7, shape_a} -> required_time 0 +{s7, shape_b} -> required_time 0 +``` + +to: + +```text +{s7, shape_a} -> required_time 1 +{s7, shape_b} -> required_time 0 +``` + +The minimum is still `0`, so `MultiTimeView` cannot compact away time `0`. +After `shape_b` also notifies up to `0`, the minimum becomes `1`. Then: + +```text +{s7, 30} -> [:out, 1] +``` + +can compact to: + +```text +{s7, 30} -> [] +``` + +For a removed value: + +```text +{s7, 20} -> [:in, 2] +``` + +if the minimum required time later advances past `2`, compaction can delete the +`MultiTimeView` row and remove stale positive routes: + +```text +delete {s7, 20} +delete {:positive, g_user_pos, 20} -> c_s7_pos +``` + +Cost for notification: + +```text +O(progress_index_update + min_recompute_for_subquery) +``` + +With an index keyed by `{subquery_id, required_time, consumer_id}`, reading the +minimum is `O(1)` or `O(log consumers_for_subquery)` depending on the ETS +layout chosen. Compaction cost is paid separately and can be incremental. For +one compacted value it is: + +```text +O(H_v + positive_children_for_subquery) +``` + +If compaction is batched, total work is proportional to the histories visited +and the stale route rows removed. + +#### Move-In Query Construction + +For the `s7` move from time `0` to `1`, existing SQL generation may still need +arrays for the before and after views. The new design builds them from +`MultiTimeView` inside the query task: + +```elixir +values_for.(["$sublink", "0"], 0) -> [10, 20] +values_for.(["$sublink", "0"], 1) -> [10, 20, 30] +``` + +What is stored persistently: + +```text +nothing beyond the ActiveMove times and changed values +``` + +What is allocated transiently: + +```text +query-local arrays for values(s7, 0) and values(s7, 1) +move-in snapshot rows returned by Postgres +``` + +Cost for the compatibility implementation: + +```text +O(V_s + R) +``` + +where `R` is the number of root-table rows returned by the move-in query. + +This does not yet minimize move-in query memory, but it moves full-view arrays +out of steady consumer state and into short-lived query tasks. + +#### `remove_shape` + +Removing `shape_a` reads: + +```text +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +``` + +Rows removed: + +```text +{:child_shape, c_s7_pos} -> {shape_a, branch_a} +{:shape_child, shape_a} -> c_s7_pos +{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 1} +``` + +The monitor registration for `{shape_a, s7}` is removed. `shape_a` is removed +from the child `WhereCondition`. + +If `shape_b` still uses `c_s7_pos`, no value routing rows are touched. Cost is: + +```text +O(children_for_shape + subqueries_for_shape + child_where_remove) +``` + +If this removes the last shape from `c_s7_pos`, the child is deleted too: + +```text +{:child, g_user_pos, s7} +{:child_meta, c_s7_pos} +{:subquery_child, s7} -> c_s7_pos +{:positive, g_user_pos, value} -> c_s7_pos for each retained value +``` + +The positive route cleanup iterates `MultiTimeView.values(s7)` and deletes the +specific `{group, value, child}` route rows. That last-child case costs: + +```text +O(V_s + child_metadata) +``` + +It does not scan unrelated subqueries or unrelated shapes. + +#### `remove_subquery` + +Removing dependency subquery `s7` reads: + +```text +{:subquery_child, s7} -> c_s7_pos +{:subquery_child, s7} -> c_s7_neg +``` + +Then it removes: + +```text +child metadata for c_s7_pos and c_s7_neg +participant rows for shapes attached to those children +positive routing rows for s7 values +negated group rows for s7 negated children +MultiTimeView rows with key prefix s7 +progress monitor rows for s7 +``` + +Cost: + +```text +O(C_s + sum(P_c) + V_s) +``` + +This is proportional to the removed subquery's children, participants, and +values. It should not scan the whole `SubqueryIndex` or all shapes in the +stack. + ### Materializer Integration The materializer owns the source of truth for a dependency subquery. It should From d5213cb9b03ee798a1549356f494bb9696e97178 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 08:58:49 +0100 Subject: [PATCH 13/40] Add memory figures --- docs/rfcs/subquery-index.md | 107 +++++++++++++++++++++++++++++++++++- 1 file changed, 106 insertions(+), 1 deletion(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 1d7e7bf192..7e058b008d 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -1,6 +1,6 @@ --- title: Shared Subquery Indexes with Logical-Time Views -version: "0.1" +version: "0.2" status: draft owner: robacourt contributors: [] @@ -1034,6 +1034,110 @@ This is proportional to the removed subquery's children, participants, and values. It should not scan the whole `SubqueryIndex` or all shapes in the stack. +### Memory Savings Prototype + +The prototype script is: + +```text +packages/sync-service/scripts/subquery_logical_time_memory.exs +``` + +Run it directly with Elixir so it does not start the sync-service application: + +```sh +elixir scripts/subquery_logical_time_memory.exs +``` + +There is also a focused test file: + +```text +packages/sync-service/test/electric/shapes/filter/subquery_logical_time_memory_bench_test.exs +``` + +The prototype compares: + +- the current model: current `SubqueryIndex`-style ETS rows, per-consumer + `MapSet` views, and active-move before/after views; +- the logical-time model: shared `MultiTimeView` rows, shared child routing and + metadata rows, progress-monitor rows, compact per-consumer subquery + references, and active moves that store changed values plus logical times. + +The model intentionally uses small integer dependency values. That is +conservative for workloads with large text, UUID, or composite values because +the current model duplicates those values per shape, while the logical-time +model stores them once per retained subquery value plus routing rows. + +The local run below was generated on: + +```text +OTP: 28 +Elixir: 1.19.5 +Architecture: aarch64-apple-darwin24.5.0 +Word size: 8 bytes +``` + +#### Local Measured Scenarios + +| Scenario | Current total | Current index | Current consumers | Logical total | Logical ETS | Logical consumers | Savings | +|----------|---------------|---------------|-------------------|---------------|-------------|-------------------|---------| +| 1 shape, 1k values, steady | 331.6 KiB | 302.4 KiB | 29.3 KiB | 222.9 KiB | 222.6 KiB | 256 B | 32.8% | +| 10 shapes, 1k values, steady | 3.2 MiB | 2.91 MiB | 292.5 KiB | 229.1 KiB | 226.6 KiB | 2.5 KiB | 93.0% | +| 100 shapes, 1k values, steady | 31.92 MiB | 29.06 MiB | 2.86 MiB | 290.9 KiB | 265.9 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 10k values, steady | 318.9 MiB | 290.02 MiB | 28.87 MiB | 1.78 MiB | 1.76 MiB | 25.0 KiB | 99.4% | +| 100 shapes, 1k base, 100 added x 10 advanced | 32.24 MiB | 29.35 MiB | 2.88 MiB | 309.6 KiB | 284.6 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 1k base, 100 added x 99 advanced | 35.07 MiB | 31.94 MiB | 3.13 MiB | 309.6 KiB | 284.6 KiB | 25.0 KiB | 99.1% | +| 100 shapes, 1k base, 100 added x 10 active move | 32.87 MiB | 29.35 MiB | 3.52 MiB | 349.8 KiB | 284.6 KiB | 65.2 KiB | 99.0% | +| 100 shapes, 1k base, 1k added x 99 active move | 75.51 MiB | 57.77 MiB | 17.75 MiB | 4.25 MiB | 453.4 KiB | 3.81 MiB | 94.4% | + +Interpretation: + +- One-shape cohorts still save memory, but only by a constant factor. There is + no sharing benefit when a subquery has one participant. +- Shared steady-state cohorts get the largest win because the current model + stores value membership and consumer views once per shape. +- Active moves remain materially smaller because the logical-time model stores + changed values and times, not before and after full dependency views. +- The harsh `1k added x 99 active move` case still grows because every active + move stores the changed values. It is still much smaller than the current + model because it avoids duplicating the 1k base view twice per active move. + +#### Customer-Shaped Estimates + +These estimates use the same script. They extrapolate from measured row costs +and use the customer workload ratios from PR #4280: + +- HumanLayer: 75 observed `WHERE` clauses, 134 subquery occurrences, 13 literal + cohorts. +- AutoArc: 611 observed `WHERE` clauses, 291 subquery occurrences, 209 literal + cohorts. +- Hazel: 13 observed shape handles, 4 subquery occurrences, 4 literal cohorts. + +The extrapolation is for 100k shapes and preserves each workload's observed +ratio of subquery occurrences to literal cohorts. + +| Customer | Observed occurrences -> cohorts | Shared occurrences | Participants @100k | Cohorts @100k | Rows/subquery | Current | Logical-time | Savings | +|----------|---------------------------------|--------------------|--------------------|--------------|---------------|---------|--------------|---------| +| HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 1,000 | 55.77 GiB | 4.2 GiB | 92.5% | +| HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 10,000 | 556.19 GiB | 40.59 GiB | 92.7% | +| AutoArc | 291 -> 209 | 28.2% | 47,627 | 34,207 | 1,000 | 14.87 GiB | 8.04 GiB | 45.9% | +| AutoArc | 291 -> 209 | 28.2% | 47,627 | 34,207 | 10,000 | 148.26 GiB | 79.86 GiB | 46.1% | +| Hazel | 4 -> 4 | 0.0% | 30,770 | 30,770 | 1,000 | 9.61 GiB | 7.23 GiB | 24.8% | +| Hazel | 4 -> 4 | 0.0% | 30,770 | 30,770 | 10,000 | 95.79 GiB | 71.83 GiB | 25.0% | + +Interpretation: + +- HumanLayer benefits most because the captured workload has high literal + subquery sharing. +- AutoArc still benefits, but many literal subqueries are not shared, so the + logical-time model stores more per-cohort shared views. +- Hazel has no observed literal sharing. The estimate still shows a constant + factor reduction because the current model stores both index membership rows + and consumer `MapSet` views per shape, while the logical-time model stores + one shared view per one-participant cohort and compact consumer references. +- If a production workload has one-off subqueries with large dependency views, + the logical-time design is still better than current state, but it is not the + main win. The main win comes when multiple shapes share a subquery. + ### Materializer Integration The materializer owns the source of truth for a dependency subquery. It should @@ -1322,6 +1426,7 @@ state in both `SubqueryIndex` and consumer event handlers. | Version | Date | Author | Changes | |---------|------|--------|---------| +| 0.2 | 2026-05-18 | robacourt | Added operation examples, a memory prototype script, measured local memory scenarios, and customer-shaped estimates based on PR #4280 ratios. | | 0.1 | 2026-05-18 | robacourt | Initial draft using the Stratovolt RFC template and alternatives from PR #4280. | --- From e3bca8437ca0a2b0ab471b676c2c63c944668c60 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 09:04:29 +0100 Subject: [PATCH 14/40] Remove use of work 'cohort' --- docs/rfcs/subquery-index.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 7e058b008d..a8f04762be 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -1091,9 +1091,9 @@ Word size: 8 bytes Interpretation: -- One-shape cohorts still save memory, but only by a constant factor. There is - no sharing benefit when a subquery has one participant. -- Shared steady-state cohorts get the largest win because the current model +- Subqueries used by one shape still save memory, but only by a constant + factor. There is no sharing benefit when a subquery has one participant. +- Shared steady-state subqueries get the largest win because the current model stores value membership and consumer views once per shape. - Active moves remain materially smaller because the logical-time model stores changed values and times, not before and after full dependency views. @@ -1106,17 +1106,20 @@ Interpretation: These estimates use the same script. They extrapolate from measured row costs and use the customer workload ratios from PR #4280: -- HumanLayer: 75 observed `WHERE` clauses, 134 subquery occurrences, 13 literal - cohorts. -- AutoArc: 611 observed `WHERE` clauses, 291 subquery occurrences, 209 literal - cohorts. -- Hazel: 13 observed shape handles, 4 subquery occurrences, 4 literal cohorts. +- HumanLayer: 75 observed `WHERE` clauses, 134 subquery occurrences, 13 + distinct literal subqueries. +- AutoArc: 611 observed `WHERE` clauses, 291 subquery occurrences, 209 + distinct literal subqueries. +- Hazel: 13 observed shape handles, 4 subquery occurrences, 4 distinct literal + subqueries. The extrapolation is for 100k shapes and preserves each workload's observed -ratio of subquery occurrences to literal cohorts. +ratio of subquery occurrences to distinct literal subqueries. A distinct +literal subquery here means a distinct dependency subquery, not a subquery +group. -| Customer | Observed occurrences -> cohorts | Shared occurrences | Participants @100k | Cohorts @100k | Rows/subquery | Current | Logical-time | Savings | -|----------|---------------------------------|--------------------|--------------------|--------------|---------------|---------|--------------|---------| +| Customer | Observed occurrences -> distinct subqueries | Shared occurrences | Participants @100k | Distinct subqueries @100k | Rows/subquery | Current | Logical-time | Savings | +|----------|----------------------------------------------|--------------------|--------------------|---------------------------|---------------|---------|--------------|---------| | HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 1,000 | 55.77 GiB | 4.2 GiB | 92.5% | | HumanLayer | 134 -> 13 | 90.3% | 178,667 | 17,334 | 10,000 | 556.19 GiB | 40.59 GiB | 92.7% | | AutoArc | 291 -> 209 | 28.2% | 47,627 | 34,207 | 1,000 | 14.87 GiB | 8.04 GiB | 45.9% | @@ -1129,11 +1132,11 @@ Interpretation: - HumanLayer benefits most because the captured workload has high literal subquery sharing. - AutoArc still benefits, but many literal subqueries are not shared, so the - logical-time model stores more per-cohort shared views. + logical-time model stores more per-subquery shared views. - Hazel has no observed literal sharing. The estimate still shows a constant factor reduction because the current model stores both index membership rows and consumer `MapSet` views per shape, while the logical-time model stores - one shared view per one-participant cohort and compact consumer references. + one shared view per one-participant subquery and compact consumer references. - If a production workload has one-off subqueries with large dependency views, the logical-time design is still better than current state, but it is not the main win. The main win comes when multiple shapes share a subquery. @@ -1411,8 +1414,8 @@ as a follow-up optimization if measurements show cleanup cost is high. ### Alternative 6: Shared Base View With Sparse XOR Exceptions **Description:** The design in PR #4280 stores one base dependency view per -cohort and stores sparse per-participant XOR exceptions for values where a -participant temporarily differs from the base. +grouped subquery index entry and stores sparse per-participant XOR exceptions +for values where a participant temporarily differs from the base. **Why not:** This is a lower-risk, index-focused approach and may still be the right short-term fix if this RFC is too broad. However, it leaves consumer-held From 3d8080b195e82478e10522ffc41ebd9eb7918d28 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 09:19:22 +0100 Subject: [PATCH 15/40] Remove symbols --- docs/rfcs/subquery-index.md | 73 +++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 31 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index a8f04762be..864daa8288 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -473,16 +473,6 @@ WHERE user_id IN (SELECT id FROM users WHERE company_id = 8) WHERE user_id NOT IN (SELECT id FROM users WHERE company_id = 7) ``` -Symbols used below: - -- `V_s`: values in subquery `s` over the retained window. -- `H_v`: transition history length for one `{subquery_id, value}` row. -- `C_s`: child nodes attached to subquery `s`. -- `P_c`: outer shapes attached to child node `c`. -- `K`: changed values in one dependency move. -- `N_g`: child nodes in a negated group. -- `R`: root-table candidate rows returned by a move-in SQL query. - #### Initial `MultiTimeView` State The initial materializer state for `s7` stores one row per dependency value, @@ -498,8 +488,8 @@ not one row per outer shape: The empty history means the value is present for the whole retained window. -Memory is `O(V_s)` for the shared view. In this example, `shape_a` and -`shape_b` do not duplicate `{10, 20}`. +Memory is `O(number_of_values_in_subquery_retained_window)` for the shared +view. In this example, `shape_a` and `shape_b` do not duplicate `{10, 20}`. #### `register_subquery_consumer` @@ -583,12 +573,17 @@ What is evaluated: Cost: ```text -O(number_of_subquery_occurrences_in_shape + V_s + child_where_insert) +O( + number_of_subquery_occurrences_in_shape + + number_of_values_in_s7_retained_window + + child_where_insert +) ``` -The `V_s` term only applies because this is the first child for -`{g_user_pos, s7}`. Memory added is `O(V_s)` positive routing rows for the -child plus `O(number_of_subquery_occurrences_in_shape)` participant rows. +The value-count term only applies because this is the first child for +`{g_user_pos, s7}`. Memory added is +`O(number_of_values_in_s7_retained_window)` positive routing rows for the child +plus `O(number_of_subquery_occurrences_in_shape)` participant rows. #### `add_shape`: Additional Shape Sharing An Existing Child @@ -617,7 +612,8 @@ Cost: O(number_of_subquery_occurrences_in_shape + child_where_insert) ``` -Memory added is per-shape metadata only, not `O(V_s)`. +Memory added is per-shape metadata only, not +`O(number_of_values_in_s7_retained_window)`. #### `add_shape`: Same Group, Different Subquery @@ -634,8 +630,8 @@ Rows added include: {:shape_subquery, shape_c, ["$sublink", "0"]} -> {s8, 0} ``` -Cost is `O(V_s8)` for the first `s8` child in this group. This is expected: -`s8` has different dependency values from `s7`. +Cost is `O(number_of_values_in_s8_retained_window)` for the first `s8` child in +this group. This is expected: `s8` has different dependency values from `s7`. #### `add_shape`: Negated Shape @@ -664,7 +660,8 @@ Cost: O(number_of_subquery_occurrences_in_shape + child_where_insert) ``` -Memory added for negated routing is `O(1)` per child, not `O(V_s)`. +Memory added for negated routing is `O(1)` per child, not +`O(number_of_values_in_s7_retained_window)`. #### `affected_shapes`: Positive Group @@ -693,7 +690,11 @@ Both shapes are affected. Cost: ```text -O(children_for_value + child_where_eval + exact_subquery_checks * H_v) +O( + children_for_value + + child_where_eval + + exact_subquery_checks * transition_history_length_for_value +) ``` For this example, `children_for_value = 1`. There is no scan of all shapes and @@ -736,7 +737,11 @@ Only `shape_a` is affected. Cost remains: ```text -O(children_for_value + child_where_eval + exact_subquery_checks * H_v) +O( + children_for_value + + child_where_eval + + exact_subquery_checks * transition_history_length_for_value +) ``` The extra memory for the move is one history row for `{s7, 30}` plus one @@ -769,7 +774,11 @@ advances to logical time `1`, `NOT IN s7` is false for `30`. Cost: ```text -O(N_g * H_v + child_where_eval + exact_subquery_checks * H_v) +O( + number_of_negated_children_in_group * transition_history_length_for_value + + child_where_eval + + exact_subquery_checks * transition_history_length_for_value +) ``` This is intentionally proportional to the number of affected negated children. @@ -812,7 +821,7 @@ What is evaluated: Cost: ```text -O(K * (history_update + C_s)) +O(number_of_changed_values * (history_update + child_nodes_for_subquery)) ``` For a remove of `20` from `s7` at time `2`, the history becomes: @@ -856,7 +865,7 @@ Steady memory added per active move is: O(number_of_changed_values + number_of_subquery_refs) ``` -not `O(V_s)`. +not `O(number_of_values_in_s7_retained_window)`. #### `notify_processed_up_to` And Compaction @@ -919,7 +928,7 @@ layout chosen. Compaction cost is paid separately and can be incremental. For one compacted value it is: ```text -O(H_v + positive_children_for_subquery) +O(transition_history_length_for_value + positive_children_for_subquery) ``` If compaction is batched, total work is proportional to the histories visited @@ -952,11 +961,9 @@ move-in snapshot rows returned by Postgres Cost for the compatibility implementation: ```text -O(V_s + R) +O(number_of_values_in_s7_retained_window + root_rows_returned_by_move_in_query) ``` -where `R` is the number of root-table rows returned by the move-in query. - This does not yet minimize move-in query memory, but it moves full-view arrays out of steady consumer state and into short-lived query tasks. @@ -999,7 +1006,7 @@ The positive route cleanup iterates `MultiTimeView.values(s7)` and deletes the specific `{group, value, child}` route rows. That last-child case costs: ```text -O(V_s + child_metadata) +O(number_of_values_in_s7_retained_window + child_metadata) ``` It does not scan unrelated subqueries or unrelated shapes. @@ -1027,7 +1034,11 @@ progress monitor rows for s7 Cost: ```text -O(C_s + sum(P_c) + V_s) +O( + child_nodes_for_subquery + + sum(shapes_attached_to_each_child) + + number_of_values_in_s7_retained_window +) ``` This is proportional to the removed subquery's children, participants, and From a6ba944b11add89bc055226d00412a05f5c0e4ec Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 13:21:52 +0100 Subject: [PATCH 16/40] Add history module --- .../filter/indexes/subquery_index/history.ex | 132 +++++++++++++ .../indexes/subquery_index/history_test.exs | 177 ++++++++++++++++++ 2 files changed, 309 insertions(+) create mode 100644 packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex create mode 100644 packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex new file mode 100644 index 0000000000..ea4649f514 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex @@ -0,0 +1,132 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do + @moduledoc """ + Purely functional representation of a single value's membership history in a + subquery's shared multi-time view. + + A history records the times at which a value's membership of a subquery + toggled. It always has one of these shapes: + + [] # in at every retained logical time + [:in | :out, t1, t2, ...] # initial state, then toggle times + + The first element is the membership at the start of the retained window. Each + subsequent integer is a logical time at which membership flipped; toggle + times are strictly increasing. + + ## Examples + + [] # member at all retained times + [:out, 9] # out before 9, in from 9 onwards + [:out, 9, 11] # out before 9, in from 9..10, out from 11 onwards + [:in, 9] # in before 9, out from 9 onwards + [:in, 9, 11] # in before 9, out from 9..10, in from 11 onwards + + ## Absence + + `nil` represents a value that is not a member at any retained logical time — + no row exists for it in the shared view. This module treats `nil` as a + first-class history for all queries. Constructors and `compact/2` return + `nil` when the value is entirely out for the retained window. + """ + + @type time :: non_neg_integer() + @type t :: [] | nonempty_list(:in | :out | time()) + @type history :: t() | nil + + @doc "A history for a value that is a member at every retained logical time." + @spec new() :: t() + def new(), do: [] + + @doc """ + Is the value a member at `time`? + + `nil` histories are never members. + """ + @spec member?(history(), time()) :: boolean() + def member?(nil, _time), do: false + def member?([], _time), do: true + + def member?([initial | toggles], time) do + toggles_at_or_before = Enum.count(toggles, &(&1 <= time)) + + case rem(toggles_at_or_before, 2) do + 0 -> initial == :in + 1 -> initial == :out + end + end + + @doc "Is the value a member at any retained logical time?" + @spec member_at_some_time?(history()) :: boolean() + def member_at_some_time?(nil), do: false + def member_at_some_time?(_history), do: true + + @doc "Is the value a member at every retained logical time?" + @spec member_at_all_times?(history()) :: boolean() + def member_at_all_times?([]), do: true + def member_at_all_times?(_history), do: false + + @doc """ + Record that the value becomes a member from `time` onwards. + + A no-op when the latest tracked state is already `:in`. `time` must be + strictly greater than any previously recorded toggle. + """ + @spec mark_in(history(), time()) :: t() + def mark_in(nil, time), do: [:out, time] + def mark_in([], _time), do: [] + + def mark_in([initial | toggles] = history, time) do + case last_state(initial, toggles) do + :in -> history + :out -> [initial | toggles ++ [time]] + end + end + + @doc """ + Record that the value stops being a member from `time` onwards. + + A no-op when the latest tracked state is already `:out`. `time` must be + strictly greater than any previously recorded toggle. + """ + @spec mark_out(history(), time()) :: history() + def mark_out(nil, _time), do: nil + def mark_out([], time), do: [:in, time] + + def mark_out([initial | toggles] = history, time) do + case last_state(initial, toggles) do + :out -> history + :in -> [initial | toggles ++ [time]] + end + end + + @doc """ + Drop toggles at or before `min_required_time`, folding their effect into the + initial state. + + Returns `nil` if, after compaction, the value is out for the entire retained + window — the row can be deleted from the shared view. + """ + @spec compact(history(), time()) :: history() + def compact(nil, _min_required_time), do: nil + def compact([], _min_required_time), do: [] + + def compact([initial | toggles], min_required_time) do + {folded, kept} = Enum.split_with(toggles, &(&1 <= min_required_time)) + + new_initial = + if rem(length(folded), 2) == 0, do: initial, else: flip(initial) + + case {new_initial, kept} do + {:in, []} -> [] + {:out, []} -> nil + {state, times} -> [state | times] + end + end + + defp last_state(initial, toggles) do + if rem(length(toggles), 2) == 0, do: initial, else: flip(initial) + end + + defp flip(:in), do: :out + defp flip(:out), do: :in +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs new file mode 100644 index 0000000000..5d69a1de24 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/history_test.exs @@ -0,0 +1,177 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.HistoryTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.History + + describe "new/0" do + test "is the empty history — member at every retained time" do + assert History.new() == [] + end + end + + describe "member?/2" do + test "[] is in at every time" do + assert History.member?([], 0) + assert History.member?([], 1_000) + end + + test "nil is out at every time" do + refute History.member?(nil, 0) + refute History.member?(nil, 1_000) + end + + test "[:out, t] is out before t and in from t onwards" do + h = [:out, 9] + + refute History.member?(h, 0) + refute History.member?(h, 8) + assert History.member?(h, 9) + assert History.member?(h, 100) + end + + test "[:in, t] is in before t and out from t onwards" do + h = [:in, 9] + + assert History.member?(h, 0) + assert History.member?(h, 8) + refute History.member?(h, 9) + refute History.member?(h, 100) + end + + test "[:out, a, b] toggles in at a then out at b" do + h = [:out, 9, 11] + + refute History.member?(h, 8) + assert History.member?(h, 9) + assert History.member?(h, 10) + refute History.member?(h, 11) + refute History.member?(h, 100) + end + + test "[:in, a, b] toggles out at a then in at b" do + h = [:in, 9, 11] + + assert History.member?(h, 8) + refute History.member?(h, 9) + refute History.member?(h, 10) + assert History.member?(h, 11) + assert History.member?(h, 100) + end + end + + describe "member_at_some_time?/1" do + test "true for any non-nil history" do + assert History.member_at_some_time?([]) + assert History.member_at_some_time?([:out, 9]) + assert History.member_at_some_time?([:in, 9]) + assert History.member_at_some_time?([:out, 9, 11]) + assert History.member_at_some_time?([:in, 9, 11]) + end + + test "false for nil" do + refute History.member_at_some_time?(nil) + end + end + + describe "member_at_all_times?/1" do + test "true only for the empty history" do + assert History.member_at_all_times?([]) + end + + test "false for any history with toggles, and for nil" do + refute History.member_at_all_times?(nil) + refute History.member_at_all_times?([:out, 9]) + refute History.member_at_all_times?([:in, 9]) + refute History.member_at_all_times?([:out, 9, 11]) + refute History.member_at_all_times?([:in, 9, 11]) + end + end + + describe "mark_in/2" do + test "promotes nil to [:out, t] — a value seen for the first time" do + assert History.mark_in(nil, 5) == [:out, 5] + end + + test "leaves [] alone — already in" do + assert History.mark_in([], 5) == [] + end + + test "appends a toggle when current state is :out" do + assert History.mark_in([:in, 5], 10) == [:in, 5, 10] + assert History.mark_in([:out, 5, 8], 10) == [:out, 5, 8, 10] + end + + test "is a no-op when current state is already :in" do + assert History.mark_in([:out, 5], 10) == [:out, 5] + assert History.mark_in([:in, 5, 8], 10) == [:in, 5, 8] + end + end + + describe "mark_out/2" do + test "is a no-op on nil — there is nothing to remove" do + assert History.mark_out(nil, 5) == nil + end + + test "transitions [] to [:in, t] — out of the always-in baseline" do + assert History.mark_out([], 5) == [:in, 5] + end + + test "appends a toggle when current state is :in" do + assert History.mark_out([:out, 5], 10) == [:out, 5, 10] + assert History.mark_out([:in, 5, 8], 10) == [:in, 5, 8, 10] + end + + test "is a no-op when current state is already :out" do + assert History.mark_out([:in, 5], 10) == [:in, 5] + assert History.mark_out([:out, 5, 8], 10) == [:out, 5, 8] + end + end + + describe "compact/2" do + test "[] always stays []" do + assert History.compact([], 0) == [] + assert History.compact([], 1_000_000) == [] + end + + test "nil always stays nil" do + assert History.compact(nil, 0) == nil + assert History.compact(nil, 1_000_000) == nil + end + + test "keeps everything when min_required_time precedes the first toggle" do + assert History.compact([:out, 9, 11], 8) == [:out, 9, 11] + assert History.compact([:in, 9], 0) == [:in, 9] + end + + test "folds a toggle at min_required_time into the initial state" do + # [:out, 9]: out before 9, in from 9 onwards. + # retain from 9 onwards -> always in. + assert History.compact([:out, 9], 9) == [] + + # [:in, 11]: in before 11, out from 11 onwards. + # retain from 11 onwards -> always out -> row can be deleted. + assert History.compact([:in, 11], 11) == nil + end + + test "preserves membership across folded toggles" do + # [:out, 9, 11]: out before 9, in from 9..10, out from 11. + # retain from 10 onwards: at time 10 value is in, flips out at 11. + assert History.compact([:out, 9, 11], 10) == [:in, 11] + + # [:in, 9, 11]: in before 9, out from 9..10, in from 11. + # retain from 10 onwards: at time 10 value is out, flips in at 11. + assert History.compact([:in, 9, 11], 10) == [:out, 11] + end + + test "returns nil when retained window is entirely out" do + assert History.compact([:out, 9, 11], 12) == nil + assert History.compact([:in, 5], 10) == nil + end + + test "returns [] when retained window is entirely in" do + # [:out, 9, 11, 20]: out, in at 9, out at 11, in at 20. + # retain from 20 onwards -> always in. + assert History.compact([:out, 9, 11, 20], 20) == [] + end + end +end From 8b243dd7cb738d7ebfc6f5e6e774a5101af6ef64 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 13:40:53 +0100 Subject: [PATCH 17/40] Use recurrsion --- .../filter/indexes/subquery_index/history.ex | 61 +++++++------------ 1 file changed, 23 insertions(+), 38 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex index ea4649f514..b270bfd554 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/history.ex @@ -39,21 +39,14 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do @doc """ Is the value a member at `time`? - - `nil` histories are never members. """ @spec member?(history(), time()) :: boolean() def member?(nil, _time), do: false def member?([], _time), do: true - def member?([initial | toggles], time) do - toggles_at_or_before = Enum.count(toggles, &(&1 <= time)) - - case rem(toggles_at_or_before, 2) do - 0 -> initial == :in - 1 -> initial == :out - end - end + def member?([initial, t | _], time) when time < t, do: initial == :in + def member?([initial, _t | rest], time), do: member?([flip(initial) | rest], time) + def member?([initial], _time), do: initial == :in @doc "Is the value a member at any retained logical time?" @spec member_at_some_time?(history()) :: boolean() @@ -72,13 +65,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do strictly greater than any previously recorded toggle. """ @spec mark_in(history(), time()) :: t() - def mark_in(nil, time), do: [:out, time] - def mark_in([], _time), do: [] - - def mark_in([initial | toggles] = history, time) do - case last_state(initial, toggles) do - :in -> history - :out -> [initial | toggles ++ [time]] + def mark_in(history, time) do + if member?(history, time) do + history + else + append_time(history, time) end end @@ -89,13 +80,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do strictly greater than any previously recorded toggle. """ @spec mark_out(history(), time()) :: history() - def mark_out(nil, _time), do: nil - def mark_out([], time), do: [:in, time] - - def mark_out([initial | toggles] = history, time) do - case last_state(initial, toggles) do - :out -> history - :in -> [initial | toggles ++ [time]] + def mark_out(history, time) do + if member?(history, time) do + append_time(history, time) + else + history end end @@ -110,23 +99,19 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.History do def compact(nil, _min_required_time), do: nil def compact([], _min_required_time), do: [] - def compact([initial | toggles], min_required_time) do - {folded, kept} = Enum.split_with(toggles, &(&1 <= min_required_time)) + def compact([_initial, t | _] = history, min_required_time) when t > min_required_time, + do: history - new_initial = - if rem(length(folded), 2) == 0, do: initial, else: flip(initial) + def compact([initial, _t | rest], min_required_time), + do: compact([flip(initial) | rest], min_required_time) - case {new_initial, kept} do - {:in, []} -> [] - {:out, []} -> nil - {state, times} -> [state | times] - end - end - - defp last_state(initial, toggles) do - if rem(length(toggles), 2) == 0, do: initial, else: flip(initial) - end + def compact([:in], _min_required_time), do: [] + def compact([:out], _min_required_time), do: nil defp flip(:in), do: :out defp flip(:out), do: :in + + defp append_time(nil, time), do: [:out, time] + defp append_time([], time), do: [:in, time] + defp append_time(history, time), do: history ++ [time] end From 87d0f9097fbbf96ea7ee40d16c0c750f52189803 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 14:15:56 +0100 Subject: [PATCH 18/40] Add MultiTimeView --- .../indexes/subquery_index/multi_time_view.ex | 214 +++++++++++++ .../subquery_index/multi_time_view_test.exs | 289 ++++++++++++++++++ 2 files changed, 503 insertions(+) create mode 100644 packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex create mode 100644 packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex new file mode 100644 index 0000000000..c5cc8945f8 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex @@ -0,0 +1,214 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView do + @moduledoc """ + Shared, logical-time view of subquery membership. + + Stores one membership history per `{subquery_id, value}` pair, plus + per-subquery metadata (current logical time, min required time, ready flag), + all in a single ETS table per stack. Multiple consumer processes can read + the same view at different logical times without copying it into their own + state. + + See `docs/rfcs/subquery-index.md` for the broader design. + """ + + import Electric, only: [is_stack_id: 1] + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.History + + @type t :: :ets.tid() | atom() + @type subquery_id :: term() + @type value :: term() + @type time :: non_neg_integer() + + defp table_name(stack_id) when is_stack_id(stack_id), + do: :"multi_time_view:#{stack_id}" + + @doc """ + Create a new MultiTimeView ETS table. + + The table is `:public` so the materializer can write transitions while + consumer processes read membership. + """ + @spec new(keyword()) :: t() + def new(opts \\ []) do + case Keyword.get(opts, :stack_id) do + nil -> + :ets.new(:multi_time_view, [:set, :public]) + + stack_id -> + :ets.new(table_name(stack_id), [:set, :public, :named_table]) + end + end + + @doc "Look up the MultiTimeView table for a stack, or `nil` if none exists." + @spec for_stack(String.t()) :: t() | nil + def for_stack(stack_id) when is_stack_id(stack_id) do + case :ets.whereis(table_name(stack_id)) do + :undefined -> nil + _tid -> table_name(stack_id) + end + end + + @doc """ + Initialise a subquery at logical time `0` with the given initial member + values. The subquery is left not-ready; call `mark_ready/2` once initial + population is finished. + """ + @spec init_subquery(t(), subquery_id(), Enumerable.t()) :: :ok + def init_subquery(view, subquery_id, initial_values) do + :ets.insert(view, {{:current_time, subquery_id}, 0}) + :ets.insert(view, {{:min_required_time, subquery_id}, 0}) + + for value <- initial_values do + :ets.insert(view, {{:value, subquery_id, value}, History.new()}) + end + + :ok + end + + @doc "Mark a subquery as ready for consumers to read." + @spec mark_ready(t(), subquery_id()) :: :ok + def mark_ready(view, subquery_id) do + :ets.insert(view, {{:ready, subquery_id}, true}) + :ok + end + + @doc "Is the subquery ready for consumers to read?" + @spec ready?(t(), subquery_id()) :: boolean() + def ready?(view, subquery_id) do + :ets.member(view, {:ready, subquery_id}) + end + + @doc """ + Record that `value` becomes a member of `subquery_id` from `time` onwards. + Advances the subquery's current logical time to `time`. + """ + @spec mark_in(t(), subquery_id(), value(), time()) :: :ok + def mark_in(view, subquery_id, value, time) do + update_history(view, subquery_id, value, &History.mark_in(&1, time)) + advance_current_time(view, subquery_id, time) + :ok + end + + @doc """ + Record that `value` stops being a member of `subquery_id` from `time` + onwards. Advances the subquery's current logical time to `time`. + """ + @spec mark_out(t(), subquery_id(), value(), time()) :: :ok + def mark_out(view, subquery_id, value, time) do + update_history(view, subquery_id, value, &History.mark_out(&1, time)) + advance_current_time(view, subquery_id, time) + :ok + end + + @doc "Is `value` a member of `subquery_id` at logical `time`?" + @spec member?(t(), subquery_id(), value(), time()) :: boolean() + def member?(view, subquery_id, value, time) do + view |> lookup_history(subquery_id, value) |> History.member?(time) + end + + @doc "Is `value` a member of `subquery_id` at any retained logical time?" + @spec member_at_some_time?(t(), subquery_id(), value()) :: boolean() + def member_at_some_time?(view, subquery_id, value) do + view |> lookup_history(subquery_id, value) |> History.member_at_some_time?() + end + + @doc "Is `value` a member of `subquery_id` at every retained logical time?" + @spec member_at_all_times?(t(), subquery_id(), value()) :: boolean() + def member_at_all_times?(view, subquery_id, value) do + view |> lookup_history(subquery_id, value) |> History.member_at_all_times?() + end + + @doc "All values retained for `subquery_id` (members at some retained time)." + @spec values(t(), subquery_id()) :: [value()] + def values(view, subquery_id) do + view + |> :ets.match({{:value, subquery_id, :"$1"}, :_}) + |> Enum.map(fn [value] -> value end) + end + + @doc "All values that are members of `subquery_id` at logical `time`." + @spec values(t(), subquery_id(), time()) :: [value()] + def values(view, subquery_id, time) do + view + |> :ets.match({{:value, subquery_id, :"$1"}, :"$2"}) + |> Enum.flat_map(fn [value, history] -> + if History.member?(history, time), do: [value], else: [] + end) + end + + @doc "Current logical time for `subquery_id`, or `nil` if unknown." + @spec current_time(t(), subquery_id()) :: time() | nil + def current_time(view, subquery_id) do + case :ets.lookup(view, {:current_time, subquery_id}) do + [{_, time}] -> time + [] -> nil + end + end + + @doc """ + Advance the minimum required logical time for `subquery_id` and compact all + retained histories. Values that are out for the entire retained window have + their rows deleted. + """ + @spec set_min_required_time(t(), subquery_id(), time()) :: :ok + def set_min_required_time(view, subquery_id, time) do + :ets.insert(view, {{:min_required_time, subquery_id}, time}) + + view + |> :ets.match({{:value, subquery_id, :"$1"}, :"$2"}) + |> Enum.each(fn [value, history] -> + compact_history(view, subquery_id, value, history, time) + end) + + :ok + end + + @doc "Delete every row for `subquery_id`." + @spec remove_subquery(t(), subquery_id()) :: :ok + def remove_subquery(view, subquery_id) do + :ets.match_delete(view, {{:value, subquery_id, :_}, :_}) + :ets.delete(view, {:current_time, subquery_id}) + :ets.delete(view, {:min_required_time, subquery_id}) + :ets.delete(view, {:ready, subquery_id}) + :ok + end + + defp lookup_history(view, subquery_id, value) do + case :ets.lookup(view, {:value, subquery_id, value}) do + [{_, history}] -> history + [] -> nil + end + end + + defp update_history(view, subquery_id, value, fun) do + history = lookup_history(view, subquery_id, value) + + case fun.(history) do + ^history -> :ok + nil -> :ets.delete(view, {:value, subquery_id, value}) + new -> :ets.insert(view, {{:value, subquery_id, value}, new}) + end + end + + defp advance_current_time(view, subquery_id, time) do + case current_time(view, subquery_id) do + nil -> + :ets.insert(view, {{:current_time, subquery_id}, time}) + + current when time > current -> + :ets.insert(view, {{:current_time, subquery_id}, time}) + + _ -> + :ok + end + end + + defp compact_history(view, subquery_id, value, history, min_required_time) do + case History.compact(history, min_required_time) do + ^history -> :ok + nil -> :ets.delete(view, {:value, subquery_id, value}) + new -> :ets.insert(view, {{:value, subquery_id, value}, new}) + end + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs new file mode 100644 index 0000000000..1bc016d010 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/multi_time_view_test.exs @@ -0,0 +1,289 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeViewTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + + setup do + %{view: MultiTimeView.new()} + end + + describe "init_subquery/3" do + test "starts the subquery at logical time 0", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + assert MultiTimeView.current_time(view, :s7) == 0 + end + + test "makes the initial values members at time 0", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + + assert MultiTimeView.member?(view, :s7, 10, 0) + assert MultiTimeView.member?(view, :s7, 20, 0) + end + + test "values outside the initial set are not members", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + refute MultiTimeView.member?(view, :s7, 30, 0) + end + + test "does not mark the subquery as ready", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + refute MultiTimeView.ready?(view, :s7) + end + + test "keeps subqueries isolated from each other", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.init_subquery(view, :s8, [20]) + + assert MultiTimeView.member?(view, :s7, 10, 0) + refute MultiTimeView.member?(view, :s7, 20, 0) + assert MultiTimeView.member?(view, :s8, 20, 0) + refute MultiTimeView.member?(view, :s8, 10, 0) + end + end + + describe "mark_ready/2 and ready?/2" do + test "an unknown subquery is not ready", %{view: view} do + refute MultiTimeView.ready?(view, :s7) + end + + test "a subquery is not ready immediately after init", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.ready?(view, :s7) + end + + test "becomes ready after mark_ready", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_ready(view, :s7) + assert MultiTimeView.ready?(view, :s7) + end + end + + describe "mark_in/4" do + test "adds a value as a member from the transition time onwards", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 1) + + refute MultiTimeView.member?(view, :s7, 30, 0) + assert MultiTimeView.member?(view, :s7, 30, 1) + assert MultiTimeView.member?(view, :s7, 30, 5) + end + + test "advances the current logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 3) + + assert MultiTimeView.current_time(view, :s7) == 3 + end + + test "is a no-op when the value is already a member", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 10, 1) + + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + end + + describe "mark_out/4" do + test "removes a value from the transition time onwards", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.member?(view, :s7, 20, 0) + assert MultiTimeView.member?(view, :s7, 20, 1) + refute MultiTimeView.member?(view, :s7, 20, 2) + refute MultiTimeView.member?(view, :s7, 20, 5) + end + + test "advances the current logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.current_time(view, :s7) == 2 + end + + test "is a no-op when the value was never a member", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 99, 1) + + refute MultiTimeView.member_at_some_time?(view, :s7, 99) + end + end + + describe "member?/4" do + test "is false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member?(view, :s7, 99, 0) + end + + test "tracks add-then-remove transitions correctly", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 3) + + refute MultiTimeView.member?(view, :s7, 30, 0) + assert MultiTimeView.member?(view, :s7, 30, 1) + assert MultiTimeView.member?(view, :s7, 30, 2) + refute MultiTimeView.member?(view, :s7, 30, 3) + end + end + + describe "member_at_some_time?/3" do + test "true for values currently present", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + assert MultiTimeView.member_at_some_time?(view, :s7, 10) + end + + test "true for values that were members at any retained time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 2) + + # 30 is no longer present at the current time, but is still retained. + assert MultiTimeView.member_at_some_time?(view, :s7, 30) + end + + test "false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member_at_some_time?(view, :s7, 99) + end + end + + describe "member_at_all_times?/3" do + test "true when the value has no toggles in the retained window", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "false once a transition has been recorded", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 10, 1) + + refute MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "false for values never seen", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + refute MultiTimeView.member_at_all_times?(view, :s7, 99) + end + end + + describe "values/2" do + test "returns every value with retained membership", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_in(view, :s7, 30, 1) + + assert MultiTimeView.values(view, :s7) |> Enum.sort() == [10, 20, 30] + end + + test "includes values that have been removed but are still retained", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 1) + + assert MultiTimeView.values(view, :s7) |> Enum.sort() == [10, 20] + end + + test "returns an empty list for an unknown subquery", %{view: view} do + assert MultiTimeView.values(view, :unknown) == [] + end + end + + describe "values/3" do + test "returns only members at the given logical time", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 20, 2) + + assert MultiTimeView.values(view, :s7, 0) |> Enum.sort() == [10, 20] + assert MultiTimeView.values(view, :s7, 1) |> Enum.sort() == [10, 20, 30] + assert MultiTimeView.values(view, :s7, 2) |> Enum.sort() == [10, 30] + end + end + + describe "current_time/2" do + test "is nil for an unknown subquery", %{view: view} do + assert MultiTimeView.current_time(view, :unknown) == nil + end + + test "is the highest time written so far", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 4) + + assert MultiTimeView.current_time(view, :s7) == 4 + end + end + + describe "set_min_required_time/3" do + test "folds toggles at or before the new min into the initial state", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.mark_out(view, :s7, 10, 1) + MultiTimeView.mark_in(view, :s7, 10, 3) + + MultiTimeView.set_min_required_time(view, :s7, 3) + + # Value 10 is in for the entire retained window after compaction. + assert MultiTimeView.member_at_all_times?(view, :s7, 10) + end + + test "drops rows for values that are out for the whole retained window", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_out(view, :s7, 20, 1) + + MultiTimeView.set_min_required_time(view, :s7, 2) + + refute MultiTimeView.member_at_some_time?(view, :s7, 20) + assert MultiTimeView.values(view, :s7) == [10] + end + + test "preserves membership for retained times after compaction", %{view: view} do + MultiTimeView.init_subquery(view, :s7, []) + MultiTimeView.mark_in(view, :s7, 30, 1) + MultiTimeView.mark_out(view, :s7, 30, 5) + + MultiTimeView.set_min_required_time(view, :s7, 3) + + # 30 was in from time 1, including at time 3 (the new min) and 4. + assert MultiTimeView.member?(view, :s7, 30, 3) + assert MultiTimeView.member?(view, :s7, 30, 4) + refute MultiTimeView.member?(view, :s7, 30, 5) + end + end + + describe "remove_subquery/2" do + test "deletes all rows for the subquery", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10, 20]) + MultiTimeView.mark_ready(view, :s7) + MultiTimeView.mark_in(view, :s7, 30, 1) + + MultiTimeView.remove_subquery(view, :s7) + + refute MultiTimeView.ready?(view, :s7) + assert MultiTimeView.values(view, :s7) == [] + assert MultiTimeView.current_time(view, :s7) == nil + end + + test "leaves other subqueries untouched", %{view: view} do + MultiTimeView.init_subquery(view, :s7, [10]) + MultiTimeView.init_subquery(view, :s8, [20]) + MultiTimeView.mark_ready(view, :s8) + + MultiTimeView.remove_subquery(view, :s7) + + assert MultiTimeView.values(view, :s8) == [20] + assert MultiTimeView.ready?(view, :s8) + end + end + + describe "for_stack/1" do + test "returns the table when one was created for the stack" do + stack_id = "stack-#{System.unique_integer([:positive])}" + _view = MultiTimeView.new(stack_id: stack_id) + + assert MultiTimeView.for_stack(stack_id) != nil + end + + test "returns nil when no table exists for the stack" do + assert MultiTimeView.for_stack("nope-#{System.unique_integer([:positive])}") == nil + end + end +end From 5ec1c51f8dfcabf13e6921702315138140b37ecd Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 15:30:35 +0100 Subject: [PATCH 19/40] Impliment SubqueryIndex and ProgressMonitor --- .../lib/electric/shapes/consumer/effects.ex | 30 +- .../electric/shapes/consumer/setup_effects.ex | 31 +- .../lib/electric/shapes/filter.ex | 20 +- .../shapes/filter/indexes/subquery_index.ex | 747 ++++++++++-------- .../subquery_index/progress_monitor.ex | 199 +++++ .../electric/shapes/filter/where_condition.ex | 8 +- .../lib/electric/shapes/where_clause.ex | 13 +- .../test/electric/shape_cache_test.exs | 1 + .../test/electric/shapes/consumer_test.exs | 2 + .../subquery_index/progress_monitor_test.exs | 199 +++++ .../filter/indexes/subquery_index_test.exs | 654 +++++++++++++++ .../shapes/filter/subquery_index_test.exs | 228 ------ .../shapes/filter/subquery_node_test.exs | 235 ------ .../test/electric/shapes/filter_test.exs | 3 + packages/sync-service/test/test_helper.exs | 2 +- 15 files changed, 1545 insertions(+), 827 deletions(-) create mode 100644 packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex create mode 100644 packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs create mode 100644 packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs delete mode 100644 packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs delete mode 100644 packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index 05d4f9a783..68802c8887 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -6,7 +6,6 @@ defmodule Electric.Shapes.Consumer.Effects do alias Electric.Connection.Manager alias Electric.Postgres.SnapshotQuery - alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.ShapeCache.Storage alias Electric.LogItems alias Electric.Replication.LogOffset @@ -219,25 +218,16 @@ defmodule Electric.Shapes.Consumer.Effects do acc end - defp execute_effect(%AddToSubqueryIndex{} = effect, acc) do - update_subquery_index(acc, effect.dep_index, effect.subquery_ref, effect.values, :add) - end - - defp execute_effect(%RemoveFromSubqueryIndex{} = effect, acc) do - update_subquery_index(acc, effect.dep_index, effect.subquery_ref, effect.values, :remove) - end - - defp update_subquery_index(acc, dep_index, subquery_ref, values, op) do - state = acc.state - index = SubqueryIndex.for_stack(state.stack_id) - fun = if op == :add, do: &SubqueryIndex.add_value/5, else: &SubqueryIndex.remove_value/5 - - for {value, _original} <- values do - fun.(index, state.shape_handle, subquery_ref, dep_index, value) - end - - acc - end + # TODO phase 2 (subquery-index RFC): re-wire AddToSubqueryIndex / RemoveFromSubqueryIndex + # against the new shared MultiTimeView + grouped SubqueryIndex routing. With + # the rewrite, per-shape value membership is no longer stored — the + # materializer writes transitions into MultiTimeView and triggers routing + # updates via SubqueryIndex.add_positive_route / remove_positive_route at + # the *subquery* level, not per consumer. The consumer effect path becomes a + # progress notification (SubqueryProgressMonitor.notify_processed_up_to/4) + # rather than a per-shape index update. + defp execute_effect(%AddToSubqueryIndex{} = _effect, acc), do: acc + defp execute_effect(%RemoveFromSubqueryIndex{} = _effect, acc), do: acc @spec query_move_in_async(pid() | atom(), map(), StartMoveInQuery.t(), pid()) :: :ok def query_move_in_async( diff --git a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex index 61177f707f..a5d07ea852 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex @@ -3,7 +3,6 @@ defmodule Electric.Shapes.Consumer.SetupEffects do alias Electric.Replication.ShapeLogCollector alias Electric.Shapes.Consumer.State - alias Electric.Shapes.Filter.Indexes.SubqueryIndex require Logger @@ -43,28 +42,12 @@ defmodule Electric.Shapes.Consumer.SetupEffects do end end - defp execute_effect(%SeedSubqueryIndex{}, %State{event_handler: %{views: views}} = state) do - case SubqueryIndex.for_stack(state.stack_id) do - nil -> - {:ok, state} - - index -> - for {ref, view} <- views do - dep_index = ref |> List.last() |> String.to_integer() - - SubqueryIndex.seed_membership( - index, - state.shape_handle, - ref, - dep_index, - view - ) - end - - SubqueryIndex.mark_ready(index, state.shape_handle) - {:ok, state} - end - end - + # TODO phase 2 (subquery-index RFC): replace per-shape `seed_membership` with + # `SubqueryIndex.set_shape_subquery/5` per subquery_ref after the consumer + # has registered with `SubqueryProgressMonitor` at the materializer's + # current logical time. The shared child routing is seeded once, at child + # creation, from `MultiTimeView.values/3` — not here. `mark_ready/2` still + # clears fallback for the shape, but only after every `set_shape_subquery` + # has been written so routing has a real logical time to read. defp execute_effect(%SeedSubqueryIndex{}, %State{} = state), do: {:ok, state} end diff --git a/packages/sync-service/lib/electric/shapes/filter.ex b/packages/sync-service/lib/electric/shapes/filter.ex index 2117bef4ed..26599266b7 100644 --- a/packages/sync-service/lib/electric/shapes/filter.ex +++ b/packages/sync-service/lib/electric/shapes/filter.ex @@ -21,6 +21,7 @@ defmodule Electric.Shapes.Filter do alias Electric.Shapes.DnfPlan alias Electric.Shapes.Filter alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition alias Electric.Shapes.Shape alias Electric.Telemetry.OpenTelemetry @@ -33,7 +34,8 @@ defmodule Electric.Shapes.Filter do :where_cond_table, :eq_index_table, :incl_index_table, - :subquery_index + :subquery_index, + :multi_time_view ] @type t :: %Filter{} @@ -41,16 +43,26 @@ defmodule Electric.Shapes.Filter do @spec new(keyword()) :: Filter.t() def new(opts \\ []) do + stack_opts = Keyword.take(opts, [:stack_id]) + %Filter{ shapes_table: :ets.new(:filter_shapes, [:set, :private]), tables_table: :ets.new(:filter_tables, [:set, :private]), where_cond_table: :ets.new(:filter_where, [:set, :private]), eq_index_table: :ets.new(:filter_eq, [:set, :private]), incl_index_table: :ets.new(:filter_incl, [:set, :private]), - subquery_index: SubqueryIndex.new(Keyword.take(opts, [:stack_id])) + subquery_index: SubqueryIndex.new(stack_opts), + multi_time_view: multi_time_view(stack_opts) } end + defp multi_time_view(opts) do + case Keyword.get(opts, :stack_id) do + nil -> MultiTimeView.new() + stack_id -> MultiTimeView.for_stack(stack_id) || MultiTimeView.new(stack_id: stack_id) + end + end + @spec has_shape?(t(), shape_id()) :: boolean() def has_shape?(%Filter{shapes_table: table}, shape_handle) do :ets.member(table, shape_handle) @@ -84,8 +96,8 @@ defmodule Electric.Shapes.Filter do where_cond_id = get_or_create_table_condition(filter, shape.root_table) - WhereCondition.add_shape(filter, where_cond_id, shape_id, shape.where) maybe_register_subquery_shape(filter, shape_id, shape) + WhereCondition.add_shape(filter, where_cond_id, shape_id, shape.where) filter end @@ -96,7 +108,7 @@ defmodule Electric.Shapes.Filter do %Shape{shape_dependencies: [_ | _]} = shape ) do {:ok, plan} = DnfPlan.compile(shape) - SubqueryIndex.register_shape(index, shape_id, plan) + SubqueryIndex.register_shape(index, shape_id, plan, shape.shape_dependencies_handles) end defp maybe_register_subquery_shape(_filter, _shape_id, _shape), do: :ok diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 47632a854d..14954358e8 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -1,27 +1,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do - # Index for subquery routing and exact membership. - - # Each subquery predicate in the filter tree registers a node identified by - # `{condition_id, field_key}`. For each node, this table acts as a reverse - # index from the value seen on the root-table record to the shapes whose - # current subquery view makes that value relevant at that node. - - # Each shape consumer maintains its own entries in the index. On startup it - # seeds the node memberships for its current dependency views, then updates - # only those memberships as its subquery views change. This keeps the filter's - # materialized view of subquery membership aligned with the view that shape - # currently needs, without re-evaluating subqueries globally. - - # The same table also stores exact `shape_handle + subquery_ref + typed_value` - # membership rows used by `WhereClause.includes_record?/3` when the filter - # needs to verify subquery membership for a specific shape. - - # Shapes begin in a fallback set until their consumer has loaded and seeded - # that local state. Fallback routing is needed for restored or lazily started - # consumers: before their subquery view is available we still need to route - # root-table changes conservatively so the shape can be started and brought up - # to date. `mark_ready/2` removes the shape from fallback once its index - # entries reflect the consumer's current view. + # Shared subquery routing index. + # + # The hot rows describe the *topology* shared across all outer shapes that + # reference the same subquery: groups (per filter node + polarity), child + # nodes (per group + dependency subquery), value-keyed positive routing, + # group-keyed negated routing, and child participants. Per-shape value + # membership is *not* stored here — exact-membership checks resolve + # `{shape_handle, subquery_ref}` to `{subquery_id, logical_time}` and call + # the shared `MultiTimeView` at that time. + # + # See `docs/rfcs/subquery-index.md`, sections *SubqueryIndex Data Model* + # and *Routing*. @moduledoc false import Electric, only: [is_stack_id: 1] @@ -30,33 +19,24 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do alias Electric.Replication.Eval.Runner alias Electric.Shapes.DnfPlan alias Electric.Shapes.Filter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition @type t :: :ets.tid() | atom() - @type node_id :: {reference(), term()} defp table_name(stack_id) when is_stack_id(stack_id), do: :"subquery_index:#{stack_id}" - @doc """ - Create a new SubqueryIndex ETS table. - - The table is `:public` so consumer processes can seed and update membership - while the filter reads candidates during routing. - """ @spec new(keyword()) :: t() def new(opts \\ []) do case Keyword.get(opts, :stack_id) do nil -> - :ets.new(:subquery_index, [:bag, :public]) + :ets.new(:subquery_index, [:set, :public]) stack_id -> - :ets.new(table_name(stack_id), [:bag, :public, :named_table]) + :ets.new(table_name(stack_id), [:set, :public, :named_table]) end end - @doc """ - Look up the SubqueryIndex table for a stack. - """ @spec for_stack(String.t()) :: t() | nil def for_stack(stack_id) when is_stack_id(stack_id) do case :ets.whereis(table_name(stack_id)) do @@ -66,443 +46,594 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end @doc """ - Register per-shape exact membership metadata from a compiled DnfPlan. + Register per-shape metadata: per-occurrence polarity, per-dep-index + dependency handle (a.k.a. `subquery_id`), and the fallback flag. - Node-local routing metadata is registered by `add_shape/4` when the filter - adds the shape to a concrete subquery node. + `dep_handles` is the outer shape's `shape_dependencies_handles` list, + indexed by `dep_index`. """ - @spec register_shape(t(), term(), DnfPlan.t()) :: :ok - def register_shape(table, shape_handle, %DnfPlan{} = plan) do - polarities = - plan.positions - |> Enum.filter(fn {_pos, info} -> info.is_subquery end) - |> Map.new(fn {_pos, info} -> - {info.subquery_ref, if(info.negated, do: :negated, else: :positive)} - end) - - for {subquery_ref, polarity} <- polarities do - :ets.insert(table, {{:polarity, shape_handle, subquery_ref}, polarity}) + @spec register_shape(t(), term(), DnfPlan.t(), [term()]) :: :ok + def register_shape(table, shape_handle, %DnfPlan{} = plan, dep_handles) do + for {_pos, info} <- plan.positions, info.is_subquery do + polarity = if info.negated, do: :negated, else: :positive + :ets.insert(table, {{:polarity, shape_handle, info.subquery_ref}, polarity}) end - :ets.insert(table, {{:fallback, shape_handle}, true}) + for dep_index <- Map.keys(plan.dependency_polarities) do + dep_handle = Enum.at(dep_handles, dep_index) || {shape_handle, dep_index} + :ets.insert(table, {{:dep_handle, shape_handle, dep_index}, dep_handle}) + end + :ets.insert(table, {{:fallback, shape_handle}, true}) :ok end - @doc """ - Remove all exact membership metadata for a shape. - """ + @doc "Remove all metadata for `shape_handle`." @spec unregister_shape(t(), term()) :: :ok def unregister_shape(table, shape_handle) do - :ets.match_delete(table, {{:membership, shape_handle, :_, :_}, true}) :ets.match_delete(table, {{:polarity, shape_handle, :_}, :_}) - :ets.match_delete(table, {{:shape_node, shape_handle}, :_}) - :ets.match_delete(table, {{:shape_dep_node, shape_handle, :_}, :_}) + :ets.match_delete(table, {{:dep_handle, shape_handle, :_}, :_}) + :ets.match_delete(table, {{:shape_subquery, shape_handle, :_}, :_}) :ets.delete(table, {:fallback, shape_handle}) :ok end @doc """ - Register a shape on a concrete subquery filter node. + Attach a shape to the indexed subquery node identified by + `{condition_id, optimisation.field, optimisation.polarity}`. Creates the + group and child node lazily. + + When the child is fresh, seeds positive (or negated) routing from the + shared `MultiTimeView` at the subquery's current logical time. If the + view is not yet ready for the subquery, the child is left unseeded — + shapes still attached to that child are routed conservatively via the + per-shape fallback rows until the seed happens (phase 2 of the RFC). """ @spec add_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :ok def add_shape( %Filter{subquery_index: table} = filter, condition_id, - shape_id, + shape_handle, optimisation, branch_key ) do - node_id = {condition_id, optimisation.field} - next_condition_id = make_ref() + ensure_node_meta(table, condition_id, optimisation.field, optimisation.testexpr) + + group_id = + ensure_group(table, condition_id, optimisation.field, optimisation.polarity) + + subquery_id = lookup_dep_handle!(table, shape_handle, optimisation.dep_index) - WhereCondition.init(filter, next_condition_id) + {child_node_id, next_condition_id} = + ensure_child( + filter, + table, + group_id, + subquery_id, + optimisation.polarity, + condition_id, + optimisation.field + ) WhereCondition.add_shape( filter, next_condition_id, - shape_id, + shape_handle, optimisation.and_where, branch_key ) - ensure_node_meta(table, node_id, optimisation.testexpr) - - :ets.insert( - table, - {{:node_shape, node_id}, - {shape_id, optimisation.dep_index, optimisation.polarity, next_condition_id, branch_key}} - ) + :ets.insert(table, {{:child_shape, child_node_id, shape_handle, branch_key}, true}) + :ets.insert(table, {{:shape_child, shape_handle, child_node_id, branch_key}, true}) - if optimisation.polarity == :negated do - :ets.insert(table, {{:node_negated_shape, node_id}, {shape_id, next_condition_id}}) + if fallback?(table, shape_handle) do + :ets.insert( + table, + {{:node_fallback, condition_id, optimisation.field, child_node_id, shape_handle}, true} + ) end - :ets.insert( - table, - {{:shape_node, shape_id}, - {node_id, optimisation.dep_index, optimisation.polarity, next_condition_id, branch_key}} - ) - - :ets.insert( - table, - {{:shape_dep_node, shape_id, optimisation.dep_index}, - {node_id, optimisation.polarity, next_condition_id, branch_key}} - ) - - :ets.insert(table, {{:node_fallback, node_id}, {shape_id, next_condition_id}}) :ok end @doc """ - Remove a shape from a concrete subquery filter node. + Detach a shape from an indexed subquery node. Returns `:deleted` when the + node has no remaining children (so the parent `WhereCondition` can drop + its index key), else `:ok`. """ @spec remove_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :deleted | :ok def remove_shape( - %Filter{subquery_index: table} = filter, + %Filter{subquery_index: table, multi_time_view: mtv} = filter, condition_id, - shape_id, + shape_handle, optimisation, branch_key ) do - node_id = {condition_id, optimisation.field} - - case node_shape_entry_for_shape(table, shape_id, node_id, branch_key) do + case lookup_child_for_shape( + table, + condition_id, + optimisation.field, + optimisation.polarity, + shape_handle, + branch_key + ) do nil -> - :deleted + node_status(table, condition_id, optimisation.field) - {dep_index, polarity, next_condition_id} -> + {child_node_id, next_condition_id} -> _ = WhereCondition.remove_shape( filter, next_condition_id, - shape_id, + shape_handle, optimisation.and_where, branch_key ) - :ets.match_delete( - table, - {{:node_shape, node_id}, {shape_id, dep_index, polarity, next_condition_id, branch_key}} - ) - - if polarity == :negated do - :ets.match_delete( - table, - {{:node_negated_shape, node_id}, {shape_id, next_condition_id}} - ) - end + :ets.delete(table, {:child_shape, child_node_id, shape_handle, branch_key}) + :ets.delete(table, {:shape_child, shape_handle, child_node_id, branch_key}) :ets.match_delete( table, - {{:shape_node, shape_id}, {node_id, dep_index, polarity, next_condition_id, branch_key}} + {{:node_fallback, condition_id, optimisation.field, child_node_id, shape_handle}, :_} ) - :ets.match_delete( - table, - {{:shape_dep_node, shape_id, dep_index}, - {node_id, polarity, next_condition_id, branch_key}} - ) - - :ets.match_delete(table, {{:node_fallback, node_id}, {shape_id, next_condition_id}}) - delete_node_members(table, node_id, shape_id, polarity, next_condition_id) - - if node_empty?(table, node_id) do - :ets.delete(table, {:node_meta, node_id}) - :deleted - else - :ok + if child_empty?(table, child_node_id) do + delete_child(table, mtv, child_node_id) end + + node_status(table, condition_id, optimisation.field) end end @doc """ - Seed membership entries from a dependency view. + Record that `shape_handle`'s `subquery_ref` should read at `time` against + the dependency view identified by `subquery_id`. Called after the + consumer has registered with the materializer in phase 2 of the RFC. """ - @spec seed_membership(t(), term(), [String.t()], non_neg_integer(), MapSet.t()) :: :ok - def seed_membership(table, shape_handle, subquery_ref, dep_index, view) do - for value <- view do - add_value(table, shape_handle, subquery_ref, dep_index, value) - end - + @spec set_shape_subquery(t(), term(), [String.t()], term(), non_neg_integer()) :: :ok + def set_shape_subquery(table, shape_handle, subquery_ref, subquery_id, time) do + :ets.insert(table, {{:shape_subquery, shape_handle, subquery_ref}, {subquery_id, time}}) :ok end - @doc """ - Mark a shape as ready for indexed routing. - """ + @spec get_shape_subquery(t(), term(), [String.t()]) :: {term(), non_neg_integer()} | nil + def get_shape_subquery(table, shape_handle, subquery_ref) do + case :ets.lookup(table, {:shape_subquery, shape_handle, subquery_ref}) do + [{_, mapping}] -> mapping + [] -> nil + end + end + + @doc "Mark a shape as routable (clear fallback rows)." @spec mark_ready(t(), term()) :: :ok def mark_ready(table, shape_handle) do :ets.delete(table, {:fallback, shape_handle}) + :ets.match_delete(table, {{:node_fallback, :_, :_, :_, shape_handle}, :_}) + :ok + end - for {node_id, _dep_index, _polarity, _next_condition_id, _branch_key} <- - nodes_for_shape(table, shape_handle) do - :ets.match_delete(table, {{:node_fallback, node_id}, {shape_handle, :_}}) + @spec fallback?(t(), term()) :: boolean() + def fallback?(table, shape_handle), do: :ets.member(table, {:fallback, shape_handle}) + + @doc "Whether `shape_handle` is attached to at least one indexed subquery node." + @spec has_positions?(t(), term()) :: boolean() + def has_positions?(table, shape_handle) do + :ets.match(table, {{:shape_child, shape_handle, :_, :_}, :_}, 1) != :"$end_of_table" + end + + @doc """ + Add a positive route for `value` to every existing positive child of + `subquery_id`. Called by the materializer when a value enters the + retained window for that subquery. + """ + @spec add_positive_route(t(), term(), term()) :: :ok + def add_positive_route(table, subquery_id, value) do + for child_node_id <- children_for_subquery(table, subquery_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{polarity: :positive, group_id: group_id}}] -> + :ets.insert(table, {{:positive, group_id, value, child_node_id}, true}) + + _ -> + :ok + end end :ok end @doc """ - Add a value to both the node-local routing index and the exact membership set. + Drop the positive route for `value` from every positive child of + `subquery_id`. Called by compaction when `value` is out for the whole + retained window. """ - @spec add_value(t(), term(), [String.t()], non_neg_integer(), term()) :: :ok - def add_value(table, shape_handle, subquery_ref, dep_index, value) do - for {node_id, polarity, next_condition_id, _branch_key} <- - nodes_for_shape_dependency(table, shape_handle, dep_index) do - case polarity do - :positive -> - :ets.insert( - table, - {{:node_positive_member, node_id, value}, {shape_handle, next_condition_id}} - ) - - :negated -> - :ets.insert( - table, - {{:node_negated_member, node_id, value}, {shape_handle, next_condition_id}} - ) + @spec remove_positive_route(t(), term(), term()) :: :ok + def remove_positive_route(table, subquery_id, value) do + for child_node_id <- children_for_subquery(table, subquery_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{polarity: :positive, group_id: group_id}}] -> + :ets.delete(table, {:positive, group_id, value, child_node_id}) + + _ -> + :ok end end - :ets.insert(table, {{:membership, shape_handle, subquery_ref, value}, true}) :ok end @doc """ - Remove a value from both the node-local routing index and the exact membership set. + Cascade removal of `subquery_id`: drop every child node, participant + row, and routing row tied to that subquery. The MultiTimeView is also + cleared so values for the subquery are gone everywhere. """ - @spec remove_value(t(), term(), [String.t()], non_neg_integer(), term()) :: :ok - def remove_value(table, shape_handle, subquery_ref, dep_index, value) do - for {node_id, polarity, next_condition_id, _branch_key} <- - nodes_for_shape_dependency(table, shape_handle, dep_index) do - case polarity do - :positive -> - :ets.match_delete( - table, - {{:node_positive_member, node_id, value}, {shape_handle, next_condition_id}} - ) - - :negated -> - :ets.match_delete( - table, - {{:node_negated_member, node_id, value}, {shape_handle, next_condition_id}} - ) - end + @spec remove_subquery(t(), MultiTimeView.t(), term()) :: :ok + def remove_subquery(table, mtv, subquery_id) do + for child_node_id <- children_for_subquery(table, subquery_id) do + cleanup_child_shapes(table, child_node_id) + delete_child(table, mtv, child_node_id) end - :ets.delete(table, {:membership, shape_handle, subquery_ref, value}) + if mtv, do: MultiTimeView.remove_subquery(mtv, subquery_id) :ok end @doc """ - Get affected shape handles for a specific subquery node. + Shape candidates for a record entering the node `{condition_id, + field_key}`. Combines value-keyed positive children with + conservatively-kept negated children and any fallback children, then + recurses through each child's `WhereCondition`. """ @spec affected_shapes(Filter.t(), reference(), term(), map()) :: MapSet.t() - def affected_shapes(%Filter{subquery_index: table} = filter, condition_id, field_key, record) do - node_id = {condition_id, field_key} - + def affected_shapes( + %Filter{subquery_index: table, multi_time_view: mtv} = filter, + condition_id, + field_key, + record + ) do candidates = - case evaluate_node_lhs(table, node_id, record) do + case evaluate_node_lhs(table, condition_id, field_key, record) do {:ok, typed_value} -> - positive = - values_for_key(table, {:node_positive_member, node_id, typed_value}) |> MapSet.new() - - negated = - MapSet.difference( - values_for_key(table, {:node_negated_shape, node_id}) |> MapSet.new(), - values_for_key(table, {:node_negated_member, node_id, typed_value}) |> MapSet.new() - ) - - fallback = values_for_key(table, {:node_fallback, node_id}) |> MapSet.new() - - positive - |> MapSet.union(negated) - |> MapSet.union(fallback) + positive_children(table, condition_id, field_key, typed_value) + |> MapSet.union(negated_children(table, mtv, condition_id, field_key, typed_value)) + |> MapSet.union(fallback_children(table, condition_id, field_key)) :error -> - all_node_shapes(table, node_id) + all_children(table, condition_id, field_key) end - Enum.reduce(candidates, MapSet.new(), fn {_shape_id, next_condition_id}, acc -> - MapSet.union( - acc, - WhereCondition.affected_shapes(filter, next_condition_id, record) - ) + Enum.reduce(candidates, MapSet.new(), fn child_node_id, acc -> + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{next_condition_id: next_condition_id}}] -> + MapSet.union(acc, WhereCondition.affected_shapes(filter, next_condition_id, record)) + + [] -> + acc + end end) end - @doc """ - Get all shape ids registered on a specific subquery node. - """ @spec all_shape_ids(Filter.t(), reference(), term()) :: MapSet.t() def all_shape_ids(%Filter{subquery_index: table} = filter, condition_id, field_key) do table - |> all_node_shapes({condition_id, field_key}) - |> Enum.reduce(MapSet.new(), fn {_shape_id, next_condition_id}, acc -> - MapSet.union(acc, WhereCondition.all_shape_ids(filter, next_condition_id)) + |> all_children(condition_id, field_key) + |> Enum.reduce(MapSet.new(), fn child_node_id, acc -> + case :ets.lookup(table, {:child_meta, child_node_id}) do + [{_, %{next_condition_id: next_condition_id}}] -> + MapSet.union(acc, WhereCondition.all_shape_ids(filter, next_condition_id)) + + [] -> + acc + end end) end @doc """ - Check if a specific shape has a value in its current dependency view - for a canonical subquery ref. + Exact membership for `shape_handle + subquery_ref + typed_value`. Resolves + to `{subquery_id, logical_time}` and consults the shared `MultiTimeView`. + Falls back to polarity-based answer while the shape is in fallback or + before its consumer has registered a logical time. """ - @spec member?(t(), term(), [String.t()], term()) :: boolean() - def member?(table, shape_handle, subquery_ref, typed_value) do - :ets.member(table, {:membership, shape_handle, subquery_ref, typed_value}) - end - - @doc """ - Check subquery membership for exact evaluation, falling back to the shape's - dependency polarity while the shape is still unseeded. - """ - @spec membership_or_fallback?(t(), term(), [String.t()], term()) :: boolean() - def membership_or_fallback?(table, shape_handle, subquery_ref, typed_value) do - if shape_ready?(table, shape_handle) do - member?(table, shape_handle, subquery_ref, typed_value) + @spec membership_or_fallback?(t(), MultiTimeView.t() | nil, term(), [String.t()], term()) :: + boolean() + def membership_or_fallback?(table, mtv, shape_handle, subquery_ref, typed_value) do + if fallback?(table, shape_handle) do + polarity_default(table, shape_handle, subquery_ref) else - case polarity_for_shape_ref(table, shape_handle, subquery_ref) do - :positive -> true - :negated -> false + case get_shape_subquery(table, shape_handle, subquery_ref) do + {subquery_id, time} when not is_nil(mtv) -> + MultiTimeView.member?(mtv, subquery_id, typed_value, time) + + _ -> + polarity_default(table, shape_handle, subquery_ref) end end end @doc """ - Check if a shape is in the fallback set. + Strict exact membership without the fallback shortcut. Returns `false` + when no logical time has been set for `{shape_handle, subquery_ref}`. """ - @spec fallback?(t(), term()) :: boolean() - def fallback?(table, shape_handle) do - :ets.member(table, {:fallback, shape_handle}) + @spec member?(t(), MultiTimeView.t() | nil, term(), [String.t()], term()) :: boolean() + def member?(_table, nil, _shape_handle, _subquery_ref, _typed_value), do: false + + def member?(table, mtv, shape_handle, subquery_ref, typed_value) do + case get_shape_subquery(table, shape_handle, subquery_ref) do + {subquery_id, time} -> MultiTimeView.member?(mtv, subquery_id, typed_value, time) + nil -> false + end end - @doc """ - Check if a shape has any registered subquery nodes. - """ - @spec has_positions?(t(), term()) :: boolean() - def has_positions?(table, shape_handle) do - nodes_for_shape(table, shape_handle) != [] + defp polarity_default(table, shape_handle, subquery_ref) do + case :ets.lookup(table, {:polarity, shape_handle, subquery_ref}) do + [{_, :positive}] -> + true + + [{_, :negated}] -> + false + + [] -> + raise ArgumentError, + "missing polarity for shape #{inspect(shape_handle)} and ref " <> + inspect(subquery_ref) + end end - @doc """ - Return the registered node ids for a shape. - """ - @spec positions_for_shape(t(), term()) :: [node_id()] - def positions_for_shape(table, shape_handle) do - table - |> nodes_for_shape(shape_handle) - |> Enum.map(fn {node_id, _dep_index, _polarity, _next_condition_id, _branch_key} -> - node_id - end) + defp ensure_node_meta(table, condition_id, field_key, testexpr) do + case :ets.lookup(table, {:node_testexpr, condition_id, field_key}) do + [] -> :ets.insert(table, {{:node_testexpr, condition_id, field_key}, testexpr}) + _ -> :ok + end end - defp ensure_node_meta(table, node_id, testexpr) do - case :ets.lookup(table, {:node_meta, node_id}) do - [] -> - :ets.insert(table, {{:node_meta, node_id}, %{testexpr: testexpr}}) + defp ensure_group(table, condition_id, field_key, polarity) do + key = {:group, condition_id, field_key, polarity} - _ -> - :ok + case :ets.lookup(table, key) do + [{_, group_id}] -> + group_id + + [] -> + group_id = make_ref() + :ets.insert(table, {key, group_id}) + group_id end end - defp delete_node_members(table, node_id, shape_id, polarity, next_condition_id) do - case polarity do - :positive -> - :ets.match_delete( - table, - {{:node_positive_member, node_id, :_}, {shape_id, next_condition_id}} - ) + defp ensure_child(filter, table, group_id, subquery_id, polarity, condition_id, field_key) do + case :ets.lookup(table, {:child, group_id, subquery_id}) do + [{_, child_node_id}] -> + [{_, meta}] = :ets.lookup(table, {:child_meta, child_node_id}) + {child_node_id, meta.next_condition_id} - :negated -> - :ets.match_delete( - table, - {{:node_negated_member, node_id, :_}, {shape_id, next_condition_id}} - ) + [] -> + child_node_id = make_ref() + next_condition_id = make_ref() + + WhereCondition.init(filter, next_condition_id) + + meta = %{ + group_id: group_id, + subquery_id: subquery_id, + polarity: polarity, + next_condition_id: next_condition_id, + field_key: field_key, + condition_id: condition_id + } + + :ets.insert(table, {{:child, group_id, subquery_id}, child_node_id}) + :ets.insert(table, {{:child_meta, child_node_id}, meta}) + :ets.insert(table, {{:subquery_child, subquery_id, child_node_id}, true}) + + seed_child_routing(table, filter.multi_time_view, child_node_id, meta) + {child_node_id, next_condition_id} end end - defp nodes_for_shape(table, shape_handle) do + defp children_for_subquery(table, subquery_id) do table - |> :ets.lookup({:shape_node, shape_handle}) - |> Enum.map(&elem(&1, 1)) + |> :ets.match({{:subquery_child, subquery_id, :"$1"}, :_}) + |> Enum.map(fn [cnid] -> cnid end) end - defp nodes_for_shape_dependency(table, shape_handle, dep_index) do - table - |> :ets.lookup({:shape_dep_node, shape_handle, dep_index}) - |> Enum.map(&elem(&1, 1)) + defp seed_child_routing(_table, nil, _child_node_id, _meta), do: :ok + + defp seed_child_routing(table, mtv, child_node_id, %{ + polarity: :positive, + group_id: group_id, + subquery_id: subquery_id + }) do + case MultiTimeView.current_time(mtv, subquery_id) do + nil -> + :ok + + time -> + for value <- MultiTimeView.values(mtv, subquery_id, time) do + :ets.insert(table, {{:positive, group_id, value, child_node_id}, true}) + end + + :ok + end end - defp node_shape_entry_for_shape(table, shape_id, node_id, branch_key) do - table - |> nodes_for_shape(shape_id) - |> Enum.find_value(fn - {^node_id, dep_index, polarity, next_condition_id, ^branch_key} -> - {dep_index, polarity, next_condition_id} + defp seed_child_routing(table, _mtv, child_node_id, %{ + polarity: :negated, + group_id: group_id + }) do + :ets.insert(table, {{:negated, group_id, child_node_id}, true}) + :ok + end - _ -> - nil - end) + defp lookup_dep_handle!(table, shape_handle, dep_index) do + case :ets.lookup(table, {:dep_handle, shape_handle, dep_index}) do + [{_, dep_handle}] -> + dep_handle + + [] -> + raise ArgumentError, + "no dep_handle registered for shape #{inspect(shape_handle)} dep_index " <> + inspect(dep_index) + end end - defp node_empty?(table, node_id) do - :ets.lookup(table, {:node_shape, node_id}) == [] + defp lookup_child_for_shape(table, condition_id, field_key, polarity, shape_handle, branch_key) do + case :ets.lookup(table, {:group, condition_id, field_key, polarity}) do + [{_, group_id}] -> + children = + table + |> :ets.match({{:child, group_id, :_}, :"$1"}) + |> Enum.map(fn [cnid] -> cnid end) + + child_node_id = + Enum.find(children, fn cnid -> + :ets.member(table, {:shape_child, shape_handle, cnid, branch_key}) + end) + + if child_node_id do + [{_, %{next_condition_id: next_condition_id}}] = + :ets.lookup(table, {:child_meta, child_node_id}) + + {child_node_id, next_condition_id} + end + + [] -> + nil + end end - defp all_node_shapes(table, node_id) do - table - |> :ets.lookup({:node_shape, node_id}) - |> Enum.reduce(MapSet.new(), fn - {{:node_shape, ^node_id}, {shape_id, _dep_index, _polarity, next_condition_id, _branch_key}}, - acc -> - MapSet.put(acc, {shape_id, next_condition_id}) - - _, acc -> - acc - end) + defp child_empty?(table, child_node_id) do + :ets.match(table, {{:child_shape, child_node_id, :_, :_}, :_}) == [] end - defp evaluate_node_lhs(table, node_id, record) do - case :ets.lookup(table, {:node_meta, node_id}) do - [{_, %{testexpr: testexpr}}] -> - expr = Expr.wrap_parser_part(testexpr) + defp delete_child(table, mtv, child_node_id) do + case :ets.lookup(table, {:child_meta, child_node_id}) do + [] -> + :ok - case Runner.record_to_ref_values(expr.used_refs, record) do - {:ok, ref_values} -> - case Runner.execute(expr, ref_values) do - {:ok, value} -> {:ok, value} - _ -> :error + [{_, meta}] -> + case meta.polarity do + :positive -> + if mtv != nil do + for value <- MultiTimeView.values(mtv, meta.subquery_id) do + :ets.delete(table, {:positive, meta.group_id, value, child_node_id}) + end end - _ -> - :error + :negated -> + :ets.delete(table, {:negated, meta.group_id, child_node_id}) + end + + :ets.match_delete(table, {{:node_fallback, :_, :_, child_node_id, :_}, :_}) + :ets.delete(table, {:child, meta.group_id, meta.subquery_id}) + :ets.delete(table, {:subquery_child, meta.subquery_id, child_node_id}) + :ets.delete(table, {:child_meta, child_node_id}) + + if group_empty?(table, meta.group_id) do + :ets.delete( + table, + {:group, meta.condition_id, meta.field_key, meta.polarity} + ) + + if node_empty?(table, meta.condition_id, meta.field_key) do + :ets.delete(table, {:node_testexpr, meta.condition_id, meta.field_key}) + end end + :ok + end + end + + defp cleanup_child_shapes(table, child_node_id) do + for [shape_handle, branch_key] <- + :ets.match(table, {{:child_shape, child_node_id, :"$1", :"$2"}, :_}) do + :ets.delete(table, {:shape_child, shape_handle, child_node_id, branch_key}) + :ets.delete(table, {:child_shape, child_node_id, shape_handle, branch_key}) + end + end + + defp group_empty?(table, group_id) do + :ets.match(table, {{:child, group_id, :_}, :_}) == [] + end + + defp node_empty?(table, condition_id, field_key) do + :ets.match(table, {{:group, condition_id, field_key, :_}, :_}) == [] + end + + defp positive_children(table, condition_id, field_key, value) do + case :ets.lookup(table, {:group, condition_id, field_key, :positive}) do [] -> - :error + MapSet.new() + + [{_, group_id}] -> + table + |> :ets.match({{:positive, group_id, value, :"$1"}, :_}) + |> Enum.map(fn [cnid] -> cnid end) + |> MapSet.new() end end - defp values_for_key(table, key) do + defp negated_children(table, mtv, condition_id, field_key, value) do + case :ets.lookup(table, {:group, condition_id, field_key, :negated}) do + [] -> + MapSet.new() + + [{_, group_id}] -> + for [cnid] <- :ets.match(table, {{:negated, group_id, :"$1"}, :_}), + keep_negated_child?(table, mtv, cnid, value), + into: MapSet.new() do + cnid + end + end + end + + defp keep_negated_child?(_table, nil, _cnid, _value), do: true + + defp keep_negated_child?(table, mtv, cnid, value) do + case :ets.lookup(table, {:child_meta, cnid}) do + [{_, %{subquery_id: subquery_id}}] -> + not MultiTimeView.member_at_all_times?(mtv, subquery_id, value) + + [] -> + false + end + end + + defp fallback_children(table, condition_id, field_key) do table - |> :ets.lookup(key) - |> Enum.map(&elem(&1, 1)) + |> :ets.match({{:node_fallback, condition_id, field_key, :"$1", :_}, :_}) + |> Enum.map(fn [cnid] -> cnid end) + |> MapSet.new() end - defp shape_ready?(table, shape_handle) do - not fallback?(table, shape_handle) + defp all_children(table, condition_id, field_key) do + table + |> :ets.match({{:group, condition_id, field_key, :"$1"}, :"$2"}) + |> Enum.flat_map(fn [_polarity, group_id] -> + table + |> :ets.match({{:child, group_id, :_}, :"$1"}) + |> Enum.map(fn [cnid] -> cnid end) + end) + |> MapSet.new() end - defp polarity_for_shape_ref(table, shape_handle, subquery_ref) do - case :ets.lookup(table, {:polarity, shape_handle, subquery_ref}) do - [{_, polarity}] -> - polarity + defp node_status(table, condition_id, field_key) do + if node_empty?(table, condition_id, field_key), do: :deleted, else: :ok + end + + defp evaluate_node_lhs(table, condition_id, field_key, record) do + case :ets.lookup(table, {:node_testexpr, condition_id, field_key}) do + [{_, testexpr}] -> + expr = Expr.wrap_parser_part(testexpr) + + with {:ok, ref_values} <- Runner.record_to_ref_values(expr.used_refs, record), + {:ok, value} <- Runner.execute(expr, ref_values) do + {:ok, value} + else + _ -> :error + end [] -> - raise ArgumentError, - "missing polarity for shape #{inspect(shape_handle)} and ref #{inspect(subquery_ref)}" + :error end end end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex new file mode 100644 index 0000000000..30303b1a20 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex @@ -0,0 +1,199 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do + @moduledoc """ + Tracks, per subquery, the earliest logical time any registered consumer may + still need to read. The minimum across live consumers is the compaction + lower bound for `MultiTimeView`. + + One GenServer + one ETS table per stack. The GenServer serialises writes + and owns the process monitors that release pinned times when consumers die. + `min_required_time/2` reads the ETS row directly so the routing path does + not have to call the GenServer. + + See `docs/rfcs/subquery-index.md`, section *Processed-Up-To Time*. + """ + + use GenServer, restart: :temporary + + import Electric, only: [is_stack_id: 1] + + @type subquery_id :: term() + @type shape_handle :: term() + @type time :: non_neg_integer() + @type stack_id :: String.t() + + defp registered_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor:#{stack_id}" + + defp table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor_table:#{stack_id}" + + @spec start_link(keyword()) :: GenServer.on_start() + def start_link(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + GenServer.start_link(__MODULE__, opts, name: registered_name(stack_id)) + end + + @doc "Look up the ETS table for a stack, or `nil` if none exists." + @spec for_stack(stack_id()) :: atom() | nil + def for_stack(stack_id) when is_stack_id(stack_id) do + case :ets.whereis(table_name(stack_id)) do + :undefined -> nil + _tid -> table_name(stack_id) + end + end + + @doc """ + Register `pid` as a consumer of `subquery_id` for `shape_handle`. The + consumer's initial required time is `time` — the materializer's current + logical time when registration succeeds. The consumer process is + monitored; if it dies the pinned time is released automatically. + + Re-registering an existing `{subquery_id, shape_handle}` replaces the + previous registration. + """ + @spec register_consumer(stack_id(), subquery_id(), shape_handle(), pid(), time()) :: :ok + def register_consumer(stack_id, subquery_id, shape_handle, pid, time) do + GenServer.call( + registered_name(stack_id), + {:register, subquery_id, shape_handle, pid, time} + ) + end + + @doc """ + Remove the registration for `{subquery_id, shape_handle}`. Idempotent. + """ + @spec unregister_consumer(stack_id(), subquery_id(), shape_handle()) :: :ok + def unregister_consumer(stack_id, subquery_id, shape_handle) do + GenServer.call( + registered_name(stack_id), + {:unregister, subquery_id, shape_handle} + ) + end + + @doc """ + Advance the consumer's required time past `time`. After this call the + consumer asserts it no longer needs to read `subquery_id` at any time + `<= time`. + """ + @spec notify_processed_up_to(stack_id(), time(), subquery_id(), shape_handle()) :: :ok + def notify_processed_up_to(stack_id, time, subquery_id, shape_handle) do + GenServer.call( + registered_name(stack_id), + {:notify, time, subquery_id, shape_handle} + ) + end + + @doc """ + Earliest logical time any live consumer may still need to read for + `subquery_id`. `nil` when no consumer is registered — callers may + compact freely. + """ + @spec min_required_time(stack_id() | atom(), subquery_id()) :: time() | nil + def min_required_time(stack_id, subquery_id) when is_stack_id(stack_id), + do: min_required_time(table_name(stack_id), subquery_id) + + def min_required_time(table, subquery_id) when is_atom(table) do + case :ets.lookup(table, {:min_required_time, subquery_id}) do + [{_, time}] -> time + [] -> nil + end + end + + @spec registered?(stack_id() | atom(), subquery_id(), shape_handle()) :: boolean() + def registered?(stack_id, subquery_id, shape_handle) when is_stack_id(stack_id), + do: registered?(table_name(stack_id), subquery_id, shape_handle) + + def registered?(table, subquery_id, shape_handle) when is_atom(table) do + :ets.member(table, {:consumer, subquery_id, shape_handle}) + end + + @impl true + def init(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + + table = + :ets.new(table_name(stack_id), [ + :set, + :public, + :named_table, + read_concurrency: true + ]) + + {:ok, %{stack_id: stack_id, table: table, monitors: %{}}} + end + + @impl true + def handle_call({:register, subquery_id, shape_handle, pid, time}, _from, state) do + state = remove_registration(state, subquery_id, shape_handle) + + monitor_ref = Process.monitor(pid) + + :ets.insert( + state.table, + {{:consumer, subquery_id, shape_handle}, time, monitor_ref} + ) + + state = put_in(state.monitors[monitor_ref], {subquery_id, shape_handle}) + recompute_min(state.table, subquery_id) + {:reply, :ok, state} + end + + def handle_call({:unregister, subquery_id, shape_handle}, _from, state) do + state = remove_registration(state, subquery_id, shape_handle) + recompute_min(state.table, subquery_id) + {:reply, :ok, state} + end + + def handle_call({:notify, time, subquery_id, shape_handle}, _from, state) do + case :ets.lookup(state.table, {:consumer, subquery_id, shape_handle}) do + [{key, current, monitor_ref}] -> + new_required = max(current, time + 1) + + if new_required != current do + :ets.insert(state.table, {key, new_required, monitor_ref}) + recompute_min(state.table, subquery_id) + end + + {:reply, :ok, state} + + [] -> + {:reply, :ok, state} + end + end + + @impl true + def handle_info({:DOWN, monitor_ref, :process, _pid, _reason}, state) do + case Map.pop(state.monitors, monitor_ref) do + {nil, _} -> + {:noreply, state} + + {{subquery_id, shape_handle}, monitors} -> + :ets.delete(state.table, {:consumer, subquery_id, shape_handle}) + recompute_min(state.table, subquery_id) + {:noreply, %{state | monitors: monitors}} + end + end + + defp remove_registration(state, subquery_id, shape_handle) do + case :ets.lookup(state.table, {:consumer, subquery_id, shape_handle}) do + [{key, _time, monitor_ref}] -> + Process.demonitor(monitor_ref, [:flush]) + :ets.delete(state.table, key) + %{state | monitors: Map.delete(state.monitors, monitor_ref)} + + [] -> + state + end + end + + defp recompute_min(table, subquery_id) do + case :ets.match(table, {{:consumer, subquery_id, :_}, :"$1", :_}) do + [] -> + :ets.delete(table, {:min_required_time, subquery_id}) + + times -> + min = times |> Enum.map(fn [t] -> t end) |> Enum.min() + :ets.insert(table, {{:min_required_time, subquery_id}, min}) + end + end +end diff --git a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex index 5e34b4831f..8f10627655 100644 --- a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex +++ b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex @@ -339,7 +339,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do end defp other_shapes_affected( - %Filter{subquery_index: index, where_cond_table: table}, + %Filter{subquery_index: index, multi_time_view: mtv, where_cond_table: table}, condition_id, record ) do @@ -350,7 +350,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do [shape_count: map_size(other_shapes)], fn -> for {{shape_id, _branch_key}, where} <- other_shapes, - other_shape_matches?(index, shape_id, where, record), + other_shape_matches?(index, mtv, shape_id, where, record), into: MapSet.new() do shape_id end @@ -358,11 +358,11 @@ defmodule Electric.Shapes.Filter.WhereCondition do ) end - defp other_shape_matches?(index, shape_id, where, record) do + defp other_shape_matches?(index, mtv, shape_id, where, record) do case WhereClause.includes_record_result( where, record, - WhereClause.subquery_member_from_index(index, shape_id) + WhereClause.subquery_member_from_index(index, mtv, shape_id) ) do {:ok, included?} -> included? :error -> true diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index 98d7cc8e8c..0cd4f1d86f 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -2,6 +2,7 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( Electric.Replication.Eval.Expr.t() | nil, @@ -50,11 +51,17 @@ defmodule Electric.Shapes.WhereClause do Used for filter-side exact verification: checks whether a specific shape currently contains a typed value for a canonical subquery ref. """ - @spec subquery_member_from_index(SubqueryIndex.t(), term()) :: + @spec subquery_member_from_index(SubqueryIndex.t(), MultiTimeView.t() | nil, term()) :: ([String.t()], term() -> boolean()) - def subquery_member_from_index(index, shape_handle) do + def subquery_member_from_index(index, multi_time_view, shape_handle) do fn subquery_ref, typed_value -> - SubqueryIndex.membership_or_fallback?(index, shape_handle, subquery_ref, typed_value) + SubqueryIndex.membership_or_fallback?( + index, + multi_time_view, + shape_handle, + subquery_ref, + typed_value + ) end end end diff --git a/packages/sync-service/test/electric/shape_cache_test.exs b/packages/sync-service/test/electric/shape_cache_test.exs index 25a05def30..7c8e653cb9 100644 --- a/packages/sync-service/test/electric/shape_cache_test.exs +++ b/packages/sync-service/test/electric/shape_cache_test.exs @@ -1315,6 +1315,7 @@ defmodule Electric.ShapeCacheTest do assert [{^dep_handle, _}, {^shape_handle, _}] = ShapeCache.list_shapes(ctx.stack_id) end + @tag :subquery_phase_2 test "restarted subquery shape reseeds the subquery index after restart", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex diff --git a/packages/sync-service/test/electric/shapes/consumer_test.exs b/packages/sync-service/test/electric/shapes/consumer_test.exs index 40b27576c5..e9ed83b489 100644 --- a/packages/sync-service/test/electric/shapes/consumer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer_test.exs @@ -2263,6 +2263,7 @@ defmodule Electric.Shapes.ConsumerTest do ] = get_log_items_from_storage(LogOffset.last_before_real_offsets(), shape_storage) end + @tag :subquery_phase_2 test "consumer startup seeds the stack-scoped subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex @@ -2294,6 +2295,7 @@ defmodule Electric.Shapes.ConsumerTest do assert positions == SubqueryIndex.positions_for_shape(index, shape_handle) end + @tag :subquery_phase_2 test "consumer steady dependency move_in adds value to the subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs new file mode 100644 index 0000000000..7ed4f75e75 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/progress_monitor_test.exs @@ -0,0 +1,199 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitorTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + setup do + stack_id = "stack-#{System.unique_integer([:positive])}" + start_supervised!({ProgressMonitor, stack_id: stack_id}) + %{stack_id: stack_id} + end + + describe "register_consumer/5" do + test "sets the min required time to the consumer's initial time", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 3) + assert ProgressMonitor.min_required_time(stack_id, :s7) == 3 + end + + test "a second, earlier consumer lowers the min", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 5) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 2) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 2 + end + + test "isolates min by subquery", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 1) + :ok = ProgressMonitor.register_consumer(stack_id, :s8, :shape_a, self(), 4) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 1 + assert ProgressMonitor.min_required_time(stack_id, :s8) == 4 + end + + test "re-registering replaces the previous entry", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 1) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 4) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 4 + end + + test "marks the consumer as registered", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + assert ProgressMonitor.registered?(stack_id, :s7, :shape_a) + refute ProgressMonitor.registered?(stack_id, :s7, :shape_b) + end + end + + describe "notify_processed_up_to/4" do + test "advances the min when the limiting consumer moves on", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 1 + end + + test "does not advance the min when a slower consumer pins an older time", %{ + stack_id: stack_id + } do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 0) + + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 0 + end + + test "is monotonic — an earlier time does not regress the required time", %{ + stack_id: stack_id + } do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 5, :s7, :shape_a) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 2, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 6 + end + + test "is a no-op for an unknown consumer (race-safe)", %{stack_id: stack_id} do + assert :ok = ProgressMonitor.notify_processed_up_to(stack_id, 0, :s7, :shape_a) + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + end + + describe "unregister_consumer/3" do + test "releases the pinned time", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 5) + + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == 5 + refute ProgressMonitor.registered?(stack_id, :s7, :shape_a) + end + + test "is idempotent", %{stack_id: stack_id} do + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + + test "clears the min when the last consumer unregisters", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 0) + :ok = ProgressMonitor.unregister_consumer(stack_id, :s7, :shape_a) + + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + end + + describe "consumer process death" do + test "automatically releases the pinned time when the consumer pid dies", %{ + stack_id: stack_id + } do + {pid, ref} = spawn_consumer() + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, pid, 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_b, self(), 5) + + stop_consumer(pid, ref) + + assert eventually(fn -> + ProgressMonitor.min_required_time(stack_id, :s7) == 5 + end) + + refute ProgressMonitor.registered?(stack_id, :s7, :shape_a) + end + + test "releases every registration for that pid", %{stack_id: stack_id} do + {pid, ref} = spawn_consumer() + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, pid, 0) + :ok = ProgressMonitor.register_consumer(stack_id, :s8, :shape_a, pid, 0) + + stop_consumer(pid, ref) + + assert eventually(fn -> + not ProgressMonitor.registered?(stack_id, :s7, :shape_a) and + not ProgressMonitor.registered?(stack_id, :s8, :shape_a) + end) + end + end + + describe "for_stack/1" do + test "returns the ETS table name for a started stack", %{stack_id: stack_id} do + assert ProgressMonitor.for_stack(stack_id) != nil + end + + test "returns nil when no monitor exists for the stack" do + assert ProgressMonitor.for_stack("nope-#{System.unique_integer([:positive])}") == nil + end + end + + describe "min_required_time/2" do + test "returns nil when no consumers are registered", %{stack_id: stack_id} do + assert ProgressMonitor.min_required_time(stack_id, :s7) == nil + end + + test "can be read by table name", %{stack_id: stack_id} do + :ok = ProgressMonitor.register_consumer(stack_id, :s7, :shape_a, self(), 2) + + table = ProgressMonitor.for_stack(stack_id) + assert ProgressMonitor.min_required_time(table, :s7) == 2 + end + end + + defp spawn_consumer do + parent = self() + + pid = + spawn(fn -> + ref = make_ref() + send(parent, {:consumer_ready, ref, self()}) + + receive do + {:stop, ^ref} -> :ok + end + end) + + receive do + {:consumer_ready, ref, ^pid} -> {pid, ref} + end + end + + defp stop_consumer(pid, ref) do + mon = Process.monitor(pid) + send(pid, {:stop, ref}) + + receive do + {:DOWN, ^mon, :process, ^pid, _} -> :ok + end + end + + defp eventually(fun, attempts \\ 50, sleep_ms \\ 5) do + if fun.() do + true + else + if attempts <= 0 do + false + else + Process.sleep(sleep_ms) + eventually(fun, attempts - 1, sleep_ms) + end + end + end +end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs new file mode 100644 index 0000000000..52c5bf1db9 --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs @@ -0,0 +1,654 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do + use ExUnit.Case, async: true + + alias Electric.Replication.Eval.Parser + alias Electric.Replication.Eval.Parser.{Func, Ref} + alias Electric.Shapes.DnfPlan + alias Electric.Shapes.Filter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.WhereCondition + + @subquery_ref ["$sublink", "0"] + @other_subquery_ref ["$sublink", "1"] + @field "par_id" + @other_field "id" + @dep_handle_a "dep_a" + @dep_handle_b "dep_b" + + setup do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + + %{ + filter: filter, + table: filter.subquery_index, + mtv: filter.multi_time_view, + condition_id: condition_id + } + end + + describe "register_shape/4" do + test "stores polarity per subquery_ref and marks the shape as fallback", %{table: table} do + SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) + + assert SubqueryIndex.fallback?(table, "s1") + + SubqueryIndex.register_shape( + table, + "s2", + make_plan(polarity: :negated), + [@dep_handle_a] + ) + + assert SubqueryIndex.fallback?(table, "s2") + end + + test "membership_or_fallback? defaults to true for positive fallback", %{ + table: table, + mtv: mtv + } do + SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) + + assert SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + end + + test "membership_or_fallback? defaults to false for negated fallback", %{ + table: table, + mtv: mtv + } do + SubqueryIndex.register_shape( + table, + "s1", + make_plan(polarity: :negated), + [@dep_handle_a] + ) + + refute SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + end + end + + describe "unregister_shape/2" do + test "drops polarity, dep_handle, and fallback rows", %{table: table} do + SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) + SubqueryIndex.unregister_shape(table, "s1") + + refute SubqueryIndex.fallback?(table, "s1") + end + end + + describe "add_shape/5 (positive)" do + test "creates a single child for the first shape in a group + subquery", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + assert SubqueryIndex.has_positions?(table, "s1") + end + + test "two shapes sharing the same group + subquery share a single child", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + children = + :ets.match(table, {{:shape_child, "s1", :"$1", :_}, :_}) ++ + :ets.match(table, {{:shape_child, "s2", :"$1", :_}, :_}) + + assert children |> Enum.uniq() |> length() == 1 + end + + test "shapes with the same group but different subqueries land on different children", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1", dep_handles: [@dep_handle_a]) + + register_node_shape(filter, condition_id, "s2", dep_handles: [@dep_handle_b]) + + [[c1]] = :ets.match(table, {{:shape_child, "s1", :"$1", :_}, :_}) + [[c2]] = :ets.match(table, {{:shape_child, "s2", :"$1", :_}, :_}) + + assert c1 != c2 + end + + test "first-child creation seeds positive routing from MultiTimeView", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + [[_, group_id]] = + :ets.match(table, {{:group, condition_id, @field, :"$1"}, :"$2"}) + + assert :ets.match(table, {{:positive, group_id, :"$1", :_}, :_}) |> Enum.sort() == + [[10], [20]] |> Enum.sort() + end + + test "adding a second shape to an existing child does not duplicate positive routes", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + before_routes = :ets.match(table, {{:positive, :_, :_, :_}, :_}) + register_node_shape(filter, condition_id, "s2") + after_routes = :ets.match(table, {{:positive, :_, :_, :_}, :_}) + + assert Enum.sort(before_routes) == Enum.sort(after_routes) + end + end + + describe "add_shape/5 (negated)" do + test "stores one group-keyed routing row regardless of subquery value count", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [1, 2, 3, 4, 5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + + assert :ets.match(table, {{:negated, :_, :_}, :_}) |> length() == 1 + assert :ets.match(table, {{:positive, :_, :_, :_}, :_}) == [] + end + end + + describe "affected_shapes/4 (positive routing)" do + test "returns shapes whose subquery has the value at the shape's logical time", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + SubqueryIndex.set_shape_subquery(table, "s1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(table, "s1") + + assert MapSet.new(["s1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5"} + ) + end + + test "diverging consumer times: routing keeps the value, exact check splits the result", + %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + # Subquery starts at time 0 with no values; value 30 enters at time 1. + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s_old") + register_node_shape(filter, condition_id, "s_new") + + # Both shapes have an indexed subquery via add_shape; we still need to + # land them on the same child + seed the new value's positive route. + MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) + SubqueryIndex.add_positive_route(table, @dep_handle_a, 30) + + SubqueryIndex.set_shape_subquery(table, "s_old", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.set_shape_subquery(table, "s_new", @subquery_ref, @dep_handle_a, 1) + SubqueryIndex.mark_ready(table, "s_old") + SubqueryIndex.mark_ready(table, "s_new") + + affected = + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "30"} + ) + + # Both shapes share the positive child, so both appear as candidates. + # The downstream `other_shape_matches?` exact check is in WhereCondition; + # at the index level here we expose both, then phase-2 WhereCondition + # filters via `subquery_member_from_index`. + assert MapSet.new(["s_old", "s_new"]) == affected + + # But exact membership reports them differently: + refute SubqueryIndex.member?(table, mtv, "s_old", @subquery_ref, 30) + assert SubqueryIndex.member?(table, mtv, "s_new", @subquery_ref, 30) + end + + test "returns only shapes registered under the requested field key", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "local") + + register_node_shape(filter, condition_id, "other_field", + field: @other_field, + subquery_ref: @other_subquery_ref, + dep_handles: [@dep_handle_a] + ) + + for shape <- ~w(local other_field) do + SubqueryIndex.mark_ready(table, shape) + end + + SubqueryIndex.set_shape_subquery(table, "local", @subquery_ref, @dep_handle_a, 0) + + SubqueryIndex.set_shape_subquery( + table, + "other_field", + @other_subquery_ref, + @dep_handle_a, + 0 + ) + + assert MapSet.new(["local"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "id" => "5"} + ) + end + + test "delegates an and_where tail to the child WhereCondition", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "tail", + and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) + ) + + SubqueryIndex.set_shape_subquery(table, "tail", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(table, "tail") + + assert MapSet.new(["tail"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "name" => "keep_me"} + ) + + assert MapSet.new() == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5", "name" => "discard"} + ) + end + + test "routes unseeded shapes via the per-node fallback rows", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "unseeded") + + assert MapSet.new(["unseeded"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "999"} + ) + end + end + + describe "affected_shapes/4 (negated routing)" do + test "prunes the child when the value is a member at every retained time", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + SubqueryIndex.set_shape_subquery(table, "n1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(table, "n1") + + # 5 is in at every retained time, so the negated child is pruned. + assert MapSet.new() == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "5"} + ) + + # Other values keep the child (not a member at all times). + assert MapSet.new(["n1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "99"} + ) + end + + test "keeps the child when the value is only a member at some retained time", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "n1", polarity: :negated) + MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) + + SubqueryIndex.set_shape_subquery(table, "n1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(table, "n1") + + # 30 is not a member at *all* retained times (it was out at time 0), so + # the negated child stays in the candidate set. + assert MapSet.new(["n1"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "30"} + ) + end + end + + describe "all_shape_ids/3" do + test "returns shapes for the requested field key only", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + register_node_shape(filter, condition_id, "s2", + and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) + ) + + register_node_shape(filter, condition_id, "other", + field: @other_field, + subquery_ref: @other_subquery_ref + ) + + assert MapSet.new(["s1", "s2"]) == + SubqueryIndex.all_shape_ids(filter, condition_id, @field) + end + end + + describe "add_positive_route/3 and remove_positive_route/3" do + test "mutate routing without touching per-shape rows", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, []) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + + shape_rows_before = :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) + + SubqueryIndex.add_positive_route(table, @dep_handle_a, 42) + + [[group_id]] = :ets.match(table, {{:group, condition_id, @field, :positive}, :"$1"}) + assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) |> length() == 1 + + SubqueryIndex.remove_positive_route(table, @dep_handle_a, 42) + assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) == [] + + assert :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) == shape_rows_before + end + end + + describe "remove_shape/5" do + test "leaves shared child intact when other shapes remain", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + assert :ok = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert MapSet.new(["s2"]) == SubqueryIndex.all_shape_ids(filter, condition_id, @field) + refute SubqueryIndex.has_positions?(table, "s1") + end + + test "cleans the child and positive routes when the last shape leaves", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10, 20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + register_node_shape(filter, condition_id, "s2") + + assert :ok = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert :deleted = + SubqueryIndex.remove_shape(filter, condition_id, "s2", subquery_optimisation(), []) + + assert :ets.match(table, {{:positive, :_, :_, :_}, :_}) == [] + assert :ets.match(table, {{:child_meta, :_}, :_}) == [] + end + + test "tracks emptiness per field key", %{ + filter: filter, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + + register_node_shape(filter, condition_id, "s2", + field: @other_field, + subquery_ref: @other_subquery_ref + ) + + assert :deleted = + SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) + + assert MapSet.new(["s2"]) == + SubqueryIndex.all_shape_ids(filter, condition_id, @other_field) + + assert :deleted = + SubqueryIndex.remove_shape( + filter, + condition_id, + "s2", + subquery_optimisation(field: @other_field, subquery_ref: @other_subquery_ref), + [] + ) + end + end + + describe "remove_subquery/3" do + test "cascades to every child and participant for that subquery only", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [10]) + MultiTimeView.init_subquery(mtv, @dep_handle_b, [20]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + MultiTimeView.mark_ready(mtv, @dep_handle_b) + + register_node_shape(filter, condition_id, "s_a", dep_handles: [@dep_handle_a]) + + register_node_shape(filter, condition_id, "s_b", + field: @other_field, + subquery_ref: @other_subquery_ref, + dep_handles: [@dep_handle_b] + ) + + SubqueryIndex.remove_subquery(table, mtv, @dep_handle_a) + + refute SubqueryIndex.has_positions?(table, "s_a") + assert SubqueryIndex.has_positions?(table, "s_b") + refute MultiTimeView.member_at_some_time?(mtv, @dep_handle_a, 10) + assert MultiTimeView.member_at_some_time?(mtv, @dep_handle_b, 20) + end + end + + describe "mark_ready/2 and fallback?/2" do + test "mark_ready clears fallback and per-node fallback rows", %{ + filter: filter, + table: table, + condition_id: condition_id + } do + register_node_shape(filter, condition_id, "s1") + assert SubqueryIndex.fallback?(table, "s1") + assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) != [] + + SubqueryIndex.mark_ready(table, "s1") + + refute SubqueryIndex.fallback?(table, "s1") + assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) == [] + end + end + + describe "member?/5 and membership_or_fallback?/5" do + test "membership_or_fallback? defers to MultiTimeView at the shape's logical time", %{ + filter: filter, + mtv: mtv, + table: table, + condition_id: condition_id + } do + MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) + MultiTimeView.mark_ready(mtv, @dep_handle_a) + + register_node_shape(filter, condition_id, "s1") + SubqueryIndex.set_shape_subquery(table, "s1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(table, "s1") + + assert SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 5) + refute SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + end + + test "member? without a stored logical time returns false", %{ + table: table, + mtv: mtv + } do + refute SubqueryIndex.member?(table, mtv, "no_such_shape", @subquery_ref, 1) + end + end + + describe "for_stack/1" do + test "returns the table name when one was created for the stack" do + stack_id = "test-stack-#{System.unique_integer([:positive])}" + _table = SubqueryIndex.new(stack_id: stack_id) + assert SubqueryIndex.for_stack(stack_id) != nil + end + + test "returns nil for unknown stack" do + assert SubqueryIndex.for_stack("nonexistent-stack-#{System.unique_integer([:positive])}") == + nil + end + end + + defp register_node_shape(filter, condition_id, shape_id, opts \\ []) do + dep_handles = Keyword.get(opts, :dep_handles, [@dep_handle_a]) + SubqueryIndex.register_shape(filter.subquery_index, shape_id, make_plan(opts), dep_handles) + + :ok = + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(opts), + [] + ) + end + + defp subquery_optimisation(opts \\ []) do + field = Keyword.get(opts, :field, @field) + + %{ + operation: "subquery", + field: field, + testexpr: %Ref{path: [field], type: :int8}, + subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), + dep_index: Keyword.get(opts, :dep_index, 0), + polarity: Keyword.get(opts, :polarity, :positive), + and_where: Keyword.get(opts, :and_where) + } + end + + defp make_plan(opts \\ []) do + polarity = Keyword.get(opts, :polarity, :positive) + dep_index = Keyword.get(opts, :dep_index, 0) + subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) + field = Keyword.get(opts, :field, @field) + + testexpr = %Ref{path: [field], type: :int8} + ref = %Ref{path: subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: dep_index, + subquery_ref: subquery_ref, + tag_columns: [field] + } + }, + dependency_positions: %{dep_index => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{dep_index => polarity} + } + end + + defp where(query, refs), do: Parser.parse_and_validate_expression!(query, refs: refs) +end diff --git a/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs deleted file mode 100644 index bc5987ee59..0000000000 --- a/packages/sync-service/test/electric/shapes/filter/subquery_index_test.exs +++ /dev/null @@ -1,228 +0,0 @@ -defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do - use ExUnit.Case - - alias Electric.Replication.Eval.Parser.{Func, Ref} - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Filter - alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.WhereCondition - - @subquery_ref ["$sublink", "0"] - @field "par_id" - - setup do - filter = Filter.new() - condition_id = make_ref() - WhereCondition.init(filter, condition_id) - - %{ - filter: filter, - table: Filter.subquery_index(filter), - condition_id: condition_id - } - end - - describe "shape-level metadata" do - test "register_shape stores polarity and fallback used by exact evaluation", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan()) - - assert SubqueryIndex.fallback?(table, "s1") - assert SubqueryIndex.membership_or_fallback?(table, "s1", @subquery_ref, 99) - - SubqueryIndex.register_shape(table, "s2", make_plan(polarity: :negated)) - - refute SubqueryIndex.membership_or_fallback?(table, "s2", @subquery_ref, 99) - end - - test "unregister_shape removes exact membership metadata", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan()) - SubqueryIndex.add_value(table, "s1", @subquery_ref, 0, 5) - - assert SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - - SubqueryIndex.unregister_shape(table, "s1") - - refute SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - refute SubqueryIndex.fallback?(table, "s1") - end - end - - describe "node registration and updates" do - test "add_shape registers node mappings for a dependency", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - - assert SubqueryIndex.has_positions?(table, "s1") - assert [{^condition_id, @field}] = SubqueryIndex.positions_for_shape(table, "s1") - end - - test "multiple shapes on the same node infer emptiness from node registrations", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - register_node_shape(filter, table, condition_id, "s2") - register_node_shape(filter, table, condition_id, "s3") - - assert [ - {{:node_shape, {^condition_id, @field}}, {"s1", 0, :positive, _, []}}, - {{:node_shape, {^condition_id, @field}}, {"s2", 0, :positive, _, []}}, - {{:node_shape, {^condition_id, @field}}, {"s3", 0, :positive, _, []}} - ] = Enum.sort(:ets.lookup(table, {:node_shape, {condition_id, @field}})) - - assert :ok = - SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) - - assert MapSet.new(["s2", "s3"]) == SubqueryIndex.all_shape_ids(filter, condition_id, @field) - - assert :ok = - SubqueryIndex.remove_shape(filter, condition_id, "s2", subquery_optimisation(), []) - - assert :deleted = - SubqueryIndex.remove_shape(filter, condition_id, "s3", subquery_optimisation(), []) - - assert [] == :ets.lookup(table, {:node_shape, {condition_id, @field}}) - assert [] == :ets.lookup(table, {:node_meta, {condition_id, @field}}) - end - - test "seed_membership updates node-local routing and exact membership", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - - SubqueryIndex.seed_membership(table, "s1", @subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(table, "s1") - - assert SubqueryIndex.member?(table, "s1", @subquery_ref, 5) - - assert MapSet.new(["s1"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5"} - ) - end - - test "negated nodes use local complement semantics", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1", polarity: :negated) - - SubqueryIndex.seed_membership(table, "s1", @subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(table, "s1") - - refute MapSet.member?( - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5"} - ), - "s1" - ) - - assert MapSet.member?( - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "99"} - ), - "s1" - ) - end - - test "remove_shape clears node registrations", %{ - filter: filter, - table: table, - condition_id: condition_id - } do - register_node_shape(filter, table, condition_id, "s1") - SubqueryIndex.add_value(table, "s1", @subquery_ref, 0, 5) - - assert :deleted = - SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) - - refute SubqueryIndex.has_positions?(table, "s1") - - SubqueryIndex.unregister_shape(table, "s1") - - refute SubqueryIndex.fallback?(table, "s1") - end - end - - describe "stack lookup" do - test "stores and retrieves table ref by stack_id" do - table = SubqueryIndex.new(stack_id: "test-stack-123") - assert SubqueryIndex.for_stack("test-stack-123") == table - end - - test "returns nil for unknown stack" do - assert SubqueryIndex.for_stack("nonexistent-stack") == nil - end - end - - defp register_node_shape(filter, table, condition_id, shape_id, opts \\ []) do - SubqueryIndex.register_shape(table, shape_id, make_plan(opts)) - :ok = SubqueryIndex.add_shape(filter, condition_id, shape_id, subquery_optimisation(opts), []) - end - - defp subquery_optimisation(opts \\ []) do - field = Keyword.get(opts, :field, @field) - - %{ - operation: "subquery", - field: field, - testexpr: %Ref{path: [field], type: :int8}, - subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), - dep_index: Keyword.get(opts, :dep_index, 0), - polarity: Keyword.get(opts, :polarity, :positive), - and_where: Keyword.get(opts, :and_where) - } - end - - defp make_plan(opts \\ []) do - polarity = Keyword.get(opts, :polarity, :positive) - dep_index = Keyword.get(opts, :dep_index, 0) - subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) - field = Keyword.get(opts, :field, @field) - - testexpr = %Ref{path: [field], type: :int8} - ref = %Ref{path: subquery_ref, type: {:array, :int8}} - - ast = %Func{ - name: "sublink_membership_check", - args: [testexpr, ref], - type: :bool - } - - %DnfPlan{ - disjuncts: [], - disjuncts_positions: [], - position_count: 1, - positions: %{ - 0 => %{ - ast: ast, - sql: "fake", - is_subquery: true, - negated: polarity == :negated, - dependency_index: dep_index, - subquery_ref: subquery_ref, - tag_columns: [field] - } - }, - dependency_positions: %{dep_index => [0]}, - dependency_disjuncts: %{}, - dependency_polarities: %{dep_index => polarity} - } - end -end diff --git a/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs b/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs deleted file mode 100644 index 580c3394de..0000000000 --- a/packages/sync-service/test/electric/shapes/filter/subquery_node_test.exs +++ /dev/null @@ -1,235 +0,0 @@ -defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexNodeTest do - use ExUnit.Case - - alias Electric.Replication.Eval.Parser - alias Electric.Replication.Eval.Parser.{Func, Ref} - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Filter - alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.WhereCondition - - @subquery_ref ["$sublink", "0"] - @field "par_id" - @other_field "id" - - setup do - filter = Filter.new() - condition_id = make_ref() - WhereCondition.init(filter, condition_id) - - %{ - filter: filter, - condition_id: condition_id, - reverse_index: Filter.subquery_index(filter) - } - end - - describe "affected_shapes/4" do - test "returns only shapes registered under the current field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "local_shape") - - register_node_shape(filter, reverse_index, condition_id, "other_field_shape", - field: @other_field - ) - - seed_shape(reverse_index, "local_shape", [5]) - seed_shape(reverse_index, "other_field_shape", [5]) - - assert MapSet.new(["local_shape"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "id" => "5"} - ) - end - - test "delegates matching candidates to the child where condition", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape( - filter, - reverse_index, - condition_id, - "shape_with_exact_tail", - and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) - ) - - seed_shape(reverse_index, "shape_with_exact_tail", [5]) - - assert MapSet.new(["shape_with_exact_tail"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "name" => "keep_me"} - ) - - assert MapSet.new() == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "5", "name" => "discard"} - ) - end - - test "routes unseeded shapes once traversal reaches the node", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "unseeded_shape") - - assert MapSet.new(["unseeded_shape"]) == - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "999"} - ) - end - end - - describe "all_shape_ids/3" do - test "returns only the shape ids for the requested field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "shape1") - - register_node_shape( - filter, - reverse_index, - condition_id, - "shape2", - and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) - ) - - register_node_shape(filter, reverse_index, condition_id, "other_field_shape", - field: @other_field - ) - - assert MapSet.new(["shape1", "shape2"]) == - SubqueryIndex.all_shape_ids(filter, condition_id, @field) - end - end - - describe "remove_shape/4" do - test "tracks emptiness per field key", %{ - filter: filter, - condition_id: condition_id, - reverse_index: reverse_index - } do - register_node_shape(filter, reverse_index, condition_id, "shape1") - register_node_shape(filter, reverse_index, condition_id, "shape2", field: @other_field) - - assert :deleted = - SubqueryIndex.remove_shape( - filter, - condition_id, - "shape1", - subquery_optimisation(), - [] - ) - - refute SubqueryIndex.has_positions?(reverse_index, "shape1") - - assert MapSet.new(["shape2"]) == - SubqueryIndex.all_shape_ids(filter, condition_id, @other_field) - - assert :deleted = - SubqueryIndex.remove_shape( - filter, - condition_id, - "shape2", - subquery_optimisation(field: @other_field), - [] - ) - end - end - - defp register_node_shape(filter, reverse_index, condition_id, shape_id, opts \\ []) do - SubqueryIndex.register_shape(reverse_index, shape_id, make_plan(opts)) - - :ok = - SubqueryIndex.add_shape( - filter, - condition_id, - shape_id, - subquery_optimisation(opts), - [] - ) - end - - defp seed_shape(reverse_index, shape_id, values) do - SubqueryIndex.seed_membership( - reverse_index, - shape_id, - @subquery_ref, - 0, - MapSet.new(values) - ) - - SubqueryIndex.mark_ready(reverse_index, shape_id) - end - - defp subquery_optimisation(opts \\ []) do - %{ - operation: "subquery", - field: Keyword.get(opts, :field, @field), - testexpr: %Ref{path: [Keyword.get(opts, :field, @field)], type: :int8}, - subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), - dep_index: Keyword.get(opts, :dep_index, 0), - polarity: Keyword.get(opts, :polarity, :positive), - and_where: Keyword.get(opts, :and_where) - } - end - - defp where(query, refs) do - Parser.parse_and_validate_expression!(query, refs: refs) - end - - defp make_plan(opts) do - polarity = Keyword.get(opts, :polarity, :positive) - dep_index = Keyword.get(opts, :dep_index, 0) - subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) - field = Keyword.get(opts, :field, @field) - - testexpr = %Ref{path: [field], type: :int8} - ref = %Ref{path: subquery_ref, type: {:array, :int8}} - - ast = %Func{ - name: "sublink_membership_check", - args: [testexpr, ref], - type: :bool - } - - %DnfPlan{ - disjuncts: [], - disjuncts_positions: [], - position_count: 1, - positions: %{ - 0 => %{ - ast: ast, - sql: "fake", - is_subquery: true, - negated: polarity == :negated, - dependency_index: dep_index, - subquery_ref: subquery_ref, - tag_columns: [field] - } - }, - dependency_positions: %{dep_index => [0]}, - dependency_disjuncts: %{}, - dependency_polarities: %{dep_index => polarity} - } - end -end diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index ca768df1ae..e801089f4f 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -630,6 +630,7 @@ defmodule Electric.Shapes.FilterTest do end) end + @tag :subquery_phase_2 test "Filter.remove_shape/2 removes seeded subquery index state" do filter = Filter.new() state_before = snapshot_filter_ets(filter) @@ -931,6 +932,8 @@ defmodule Electric.Shapes.FilterTest do end describe "subquery shapes routing in filter" do + @describetag :subquery_phase_2 + import Support.DbSetup import Support.DbStructureSetup import Support.ComponentSetup diff --git a/packages/sync-service/test/test_helper.exs b/packages/sync-service/test/test_helper.exs index 9f5d0847ee..26bbb6ef84 100644 --- a/packages/sync-service/test/test_helper.exs +++ b/packages/sync-service/test/test_helper.exs @@ -8,7 +8,7 @@ ExUnit.configure(formatters: [JUnitFormatter, ExUnit.CLIFormatter]) ExUnit.start( assert_receive_timeout: 400, - exclude: [:slow, :oracle, :performance], + exclude: [:slow, :oracle, :performance, :subquery_phase_2], capture_log: true ) From ce6b73a8ae4cd7660aeb9bdb30ec04a3ecb5f747 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 15:58:46 +0100 Subject: [PATCH 20/40] Put MultiTimeView in SubqueryIndex struct --- .../lib/electric/shapes/filter.ex | 16 +- .../shapes/filter/indexes/subquery_index.ex | 126 +++++++++----- .../electric/shapes/filter/where_condition.ex | 8 +- .../lib/electric/shapes/where_clause.ex | 13 +- .../filter/indexes/subquery_index_test.exs | 164 ++++++++---------- .../test/electric/shapes/filter_test.exs | 3 +- 6 files changed, 171 insertions(+), 159 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter.ex b/packages/sync-service/lib/electric/shapes/filter.ex index 26599266b7..04e6ce4c89 100644 --- a/packages/sync-service/lib/electric/shapes/filter.ex +++ b/packages/sync-service/lib/electric/shapes/filter.ex @@ -21,7 +21,6 @@ defmodule Electric.Shapes.Filter do alias Electric.Shapes.DnfPlan alias Electric.Shapes.Filter alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition alias Electric.Shapes.Shape alias Electric.Telemetry.OpenTelemetry @@ -34,8 +33,7 @@ defmodule Electric.Shapes.Filter do :where_cond_table, :eq_index_table, :incl_index_table, - :subquery_index, - :multi_time_view + :subquery_index ] @type t :: %Filter{} @@ -43,26 +41,16 @@ defmodule Electric.Shapes.Filter do @spec new(keyword()) :: Filter.t() def new(opts \\ []) do - stack_opts = Keyword.take(opts, [:stack_id]) - %Filter{ shapes_table: :ets.new(:filter_shapes, [:set, :private]), tables_table: :ets.new(:filter_tables, [:set, :private]), where_cond_table: :ets.new(:filter_where, [:set, :private]), eq_index_table: :ets.new(:filter_eq, [:set, :private]), incl_index_table: :ets.new(:filter_incl, [:set, :private]), - subquery_index: SubqueryIndex.new(stack_opts), - multi_time_view: multi_time_view(stack_opts) + subquery_index: SubqueryIndex.new(Keyword.take(opts, [:stack_id])) } end - defp multi_time_view(opts) do - case Keyword.get(opts, :stack_id) do - nil -> MultiTimeView.new() - stack_id -> MultiTimeView.for_stack(stack_id) || MultiTimeView.new(stack_id: stack_id) - end - end - @spec has_shape?(t(), shape_id()) :: boolean() def has_shape?(%Filter{shapes_table: table}, shape_handle) do :ets.member(table, shape_handle) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 14954358e8..60fe366447 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -19,29 +19,47 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do alias Electric.Replication.Eval.Runner alias Electric.Shapes.DnfPlan alias Electric.Shapes.Filter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition - @type t :: :ets.tid() | atom() + defstruct [:table, :multi_time_view] + + @type t :: %SubqueryIndex{table: :ets.tid() | atom(), multi_time_view: MultiTimeView.t() | nil} defp table_name(stack_id) when is_stack_id(stack_id), do: :"subquery_index:#{stack_id}" @spec new(keyword()) :: t() def new(opts \\ []) do - case Keyword.get(opts, :stack_id) do - nil -> - :ets.new(:subquery_index, [:set, :public]) + table = + case Keyword.get(opts, :stack_id) do + nil -> + :ets.new(:subquery_index, [:set, :public]) - stack_id -> - :ets.new(table_name(stack_id), [:set, :public, :named_table]) - end + stack_id -> + :ets.new(table_name(stack_id), [:set, :public, :named_table]) + end + + multi_time_view = + case Keyword.get(opts, :stack_id) do + nil -> MultiTimeView.new() + stack_id -> MultiTimeView.for_stack(stack_id) || MultiTimeView.new(stack_id: stack_id) + end + + %SubqueryIndex{table: table, multi_time_view: multi_time_view} end @spec for_stack(String.t()) :: t() | nil def for_stack(stack_id) when is_stack_id(stack_id) do case :ets.whereis(table_name(stack_id)) do - :undefined -> nil - _tid -> table_name(stack_id) + :undefined -> + nil + + _tid -> + %SubqueryIndex{ + table: table_name(stack_id), + multi_time_view: MultiTimeView.for_stack(stack_id) + } end end @@ -53,7 +71,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do indexed by `dep_index`. """ @spec register_shape(t(), term(), DnfPlan.t(), [term()]) :: :ok - def register_shape(table, shape_handle, %DnfPlan{} = plan, dep_handles) do + def register_shape(%SubqueryIndex{table: table}, shape_handle, %DnfPlan{} = plan, dep_handles) do for {_pos, info} <- plan.positions, info.is_subquery do polarity = if info.negated, do: :negated, else: :positive :ets.insert(table, {{:polarity, shape_handle, info.subquery_ref}, polarity}) @@ -70,7 +88,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @doc "Remove all metadata for `shape_handle`." @spec unregister_shape(t(), term()) :: :ok - def unregister_shape(table, shape_handle) do + def unregister_shape(%SubqueryIndex{table: table}, shape_handle) do :ets.match_delete(table, {{:polarity, shape_handle, :_}, :_}) :ets.match_delete(table, {{:dep_handle, shape_handle, :_}, :_}) :ets.match_delete(table, {{:shape_subquery, shape_handle, :_}, :_}) @@ -91,7 +109,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec add_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :ok def add_shape( - %Filter{subquery_index: table} = filter, + %Filter{subquery_index: %SubqueryIndex{table: table} = index} = filter, condition_id, shape_handle, optimisation, @@ -107,7 +125,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do {child_node_id, next_condition_id} = ensure_child( filter, - table, + index, group_id, subquery_id, optimisation.polarity, @@ -126,7 +144,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.insert(table, {{:child_shape, child_node_id, shape_handle, branch_key}, true}) :ets.insert(table, {{:shape_child, shape_handle, child_node_id, branch_key}, true}) - if fallback?(table, shape_handle) do + if shape_in_fallback?(table, shape_handle) do :ets.insert( table, {{:node_fallback, condition_id, optimisation.field, child_node_id, shape_handle}, true} @@ -143,7 +161,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec remove_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :deleted | :ok def remove_shape( - %Filter{subquery_index: table, multi_time_view: mtv} = filter, + %Filter{subquery_index: %SubqueryIndex{table: table, multi_time_view: mtv}} = filter, condition_id, shape_handle, optimisation, @@ -192,13 +210,23 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do consumer has registered with the materializer in phase 2 of the RFC. """ @spec set_shape_subquery(t(), term(), [String.t()], term(), non_neg_integer()) :: :ok - def set_shape_subquery(table, shape_handle, subquery_ref, subquery_id, time) do + def set_shape_subquery( + %SubqueryIndex{table: table}, + shape_handle, + subquery_ref, + subquery_id, + time + ) do :ets.insert(table, {{:shape_subquery, shape_handle, subquery_ref}, {subquery_id, time}}) :ok end @spec get_shape_subquery(t(), term(), [String.t()]) :: {term(), non_neg_integer()} | nil - def get_shape_subquery(table, shape_handle, subquery_ref) do + def get_shape_subquery(%SubqueryIndex{table: table}, shape_handle, subquery_ref) do + do_get_shape_subquery(table, shape_handle, subquery_ref) + end + + defp do_get_shape_subquery(table, shape_handle, subquery_ref) do case :ets.lookup(table, {:shape_subquery, shape_handle, subquery_ref}) do [{_, mapping}] -> mapping [] -> nil @@ -207,18 +235,22 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @doc "Mark a shape as routable (clear fallback rows)." @spec mark_ready(t(), term()) :: :ok - def mark_ready(table, shape_handle) do + def mark_ready(%SubqueryIndex{table: table}, shape_handle) do :ets.delete(table, {:fallback, shape_handle}) :ets.match_delete(table, {{:node_fallback, :_, :_, :_, shape_handle}, :_}) :ok end @spec fallback?(t(), term()) :: boolean() - def fallback?(table, shape_handle), do: :ets.member(table, {:fallback, shape_handle}) + def fallback?(%SubqueryIndex{table: table}, shape_handle), + do: shape_in_fallback?(table, shape_handle) + + defp shape_in_fallback?(table, shape_handle), + do: :ets.member(table, {:fallback, shape_handle}) @doc "Whether `shape_handle` is attached to at least one indexed subquery node." @spec has_positions?(t(), term()) :: boolean() - def has_positions?(table, shape_handle) do + def has_positions?(%SubqueryIndex{table: table}, shape_handle) do :ets.match(table, {{:shape_child, shape_handle, :_, :_}, :_}, 1) != :"$end_of_table" end @@ -228,7 +260,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do retained window for that subquery. """ @spec add_positive_route(t(), term(), term()) :: :ok - def add_positive_route(table, subquery_id, value) do + def add_positive_route(%SubqueryIndex{table: table}, subquery_id, value) do for child_node_id <- children_for_subquery(table, subquery_id) do case :ets.lookup(table, {:child_meta, child_node_id}) do [{_, %{polarity: :positive, group_id: group_id}}] -> @@ -248,7 +280,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do retained window. """ @spec remove_positive_route(t(), term(), term()) :: :ok - def remove_positive_route(table, subquery_id, value) do + def remove_positive_route(%SubqueryIndex{table: table}, subquery_id, value) do for child_node_id <- children_for_subquery(table, subquery_id) do case :ets.lookup(table, {:child_meta, child_node_id}) do [{_, %{polarity: :positive, group_id: group_id}}] -> @@ -264,11 +296,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @doc """ Cascade removal of `subquery_id`: drop every child node, participant - row, and routing row tied to that subquery. The MultiTimeView is also - cleared so values for the subquery are gone everywhere. + row, and routing row tied to that subquery. The bundled MultiTimeView is + also cleared so values for the subquery are gone everywhere. """ - @spec remove_subquery(t(), MultiTimeView.t(), term()) :: :ok - def remove_subquery(table, mtv, subquery_id) do + @spec remove_subquery(t(), term()) :: :ok + def remove_subquery(%SubqueryIndex{table: table, multi_time_view: mtv}, subquery_id) do for child_node_id <- children_for_subquery(table, subquery_id) do cleanup_child_shapes(table, child_node_id) delete_child(table, mtv, child_node_id) @@ -286,7 +318,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec affected_shapes(Filter.t(), reference(), term(), map()) :: MapSet.t() def affected_shapes( - %Filter{subquery_index: table, multi_time_view: mtv} = filter, + %Filter{subquery_index: %SubqueryIndex{table: table, multi_time_view: mtv}} = filter, condition_id, field_key, record @@ -314,7 +346,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end @spec all_shape_ids(Filter.t(), reference(), term()) :: MapSet.t() - def all_shape_ids(%Filter{subquery_index: table} = filter, condition_id, field_key) do + def all_shape_ids( + %Filter{subquery_index: %SubqueryIndex{table: table}} = filter, + condition_id, + field_key + ) do table |> all_children(condition_id, field_key) |> Enum.reduce(MapSet.new(), fn child_node_id, acc -> @@ -334,13 +370,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do Falls back to polarity-based answer while the shape is in fallback or before its consumer has registered a logical time. """ - @spec membership_or_fallback?(t(), MultiTimeView.t() | nil, term(), [String.t()], term()) :: - boolean() - def membership_or_fallback?(table, mtv, shape_handle, subquery_ref, typed_value) do - if fallback?(table, shape_handle) do + @spec membership_or_fallback?(t(), term(), [String.t()], term()) :: boolean() + def membership_or_fallback?( + %SubqueryIndex{table: table, multi_time_view: mtv}, + shape_handle, + subquery_ref, + typed_value + ) do + if shape_in_fallback?(table, shape_handle) do polarity_default(table, shape_handle, subquery_ref) else - case get_shape_subquery(table, shape_handle, subquery_ref) do + case do_get_shape_subquery(table, shape_handle, subquery_ref) do {subquery_id, time} when not is_nil(mtv) -> MultiTimeView.member?(mtv, subquery_id, typed_value, time) @@ -354,11 +394,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do Strict exact membership without the fallback shortcut. Returns `false` when no logical time has been set for `{shape_handle, subquery_ref}`. """ - @spec member?(t(), MultiTimeView.t() | nil, term(), [String.t()], term()) :: boolean() - def member?(_table, nil, _shape_handle, _subquery_ref, _typed_value), do: false + @spec member?(t(), term(), [String.t()], term()) :: boolean() + def member?(%SubqueryIndex{multi_time_view: nil}, _shape_handle, _subquery_ref, _typed_value), + do: false - def member?(table, mtv, shape_handle, subquery_ref, typed_value) do - case get_shape_subquery(table, shape_handle, subquery_ref) do + def member?( + %SubqueryIndex{table: table, multi_time_view: mtv}, + shape_handle, + subquery_ref, + typed_value + ) do + case do_get_shape_subquery(table, shape_handle, subquery_ref) do {subquery_id, time} -> MultiTimeView.member?(mtv, subquery_id, typed_value, time) nil -> false end @@ -400,7 +446,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end end - defp ensure_child(filter, table, group_id, subquery_id, polarity, condition_id, field_key) do + defp ensure_child(filter, index, group_id, subquery_id, polarity, condition_id, field_key) do + %SubqueryIndex{table: table, multi_time_view: mtv} = index + case :ets.lookup(table, {:child, group_id, subquery_id}) do [{_, child_node_id}] -> [{_, meta}] = :ets.lookup(table, {:child_meta, child_node_id}) @@ -425,7 +473,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.insert(table, {{:child_meta, child_node_id}, meta}) :ets.insert(table, {{:subquery_child, subquery_id, child_node_id}, true}) - seed_child_routing(table, filter.multi_time_view, child_node_id, meta) + seed_child_routing(table, mtv, child_node_id, meta) {child_node_id, next_condition_id} end end diff --git a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex index 8f10627655..5e34b4831f 100644 --- a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex +++ b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex @@ -339,7 +339,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do end defp other_shapes_affected( - %Filter{subquery_index: index, multi_time_view: mtv, where_cond_table: table}, + %Filter{subquery_index: index, where_cond_table: table}, condition_id, record ) do @@ -350,7 +350,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do [shape_count: map_size(other_shapes)], fn -> for {{shape_id, _branch_key}, where} <- other_shapes, - other_shape_matches?(index, mtv, shape_id, where, record), + other_shape_matches?(index, shape_id, where, record), into: MapSet.new() do shape_id end @@ -358,11 +358,11 @@ defmodule Electric.Shapes.Filter.WhereCondition do ) end - defp other_shape_matches?(index, mtv, shape_id, where, record) do + defp other_shape_matches?(index, shape_id, where, record) do case WhereClause.includes_record_result( where, record, - WhereClause.subquery_member_from_index(index, mtv, shape_id) + WhereClause.subquery_member_from_index(index, shape_id) ) do {:ok, included?} -> included? :error -> true diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index 0cd4f1d86f..98d7cc8e8c 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -2,7 +2,6 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner alias Electric.Shapes.Filter.Indexes.SubqueryIndex - alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( Electric.Replication.Eval.Expr.t() | nil, @@ -51,17 +50,11 @@ defmodule Electric.Shapes.WhereClause do Used for filter-side exact verification: checks whether a specific shape currently contains a typed value for a canonical subquery ref. """ - @spec subquery_member_from_index(SubqueryIndex.t(), MultiTimeView.t() | nil, term()) :: + @spec subquery_member_from_index(SubqueryIndex.t(), term()) :: ([String.t()], term() -> boolean()) - def subquery_member_from_index(index, multi_time_view, shape_handle) do + def subquery_member_from_index(index, shape_handle) do fn subquery_ref, typed_value -> - SubqueryIndex.membership_or_fallback?( - index, - multi_time_view, - shape_handle, - subquery_ref, - typed_value - ) + SubqueryIndex.membership_or_fallback?(index, shape_handle, subquery_ref, typed_value) end end end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs index 52c5bf1db9..8d0adb545e 100644 --- a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs @@ -20,73 +20,69 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do filter = Filter.new() condition_id = make_ref() WhereCondition.init(filter, condition_id) + index = filter.subquery_index %{ filter: filter, - table: filter.subquery_index, - mtv: filter.multi_time_view, + index: index, + table: index.table, + mtv: index.multi_time_view, condition_id: condition_id } end describe "register_shape/4" do - test "stores polarity per subquery_ref and marks the shape as fallback", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) + test "stores polarity per subquery_ref and marks the shape as fallback", %{index: index} do + SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) - assert SubqueryIndex.fallback?(table, "s1") + assert SubqueryIndex.fallback?(index, "s1") SubqueryIndex.register_shape( - table, + index, "s2", make_plan(polarity: :negated), [@dep_handle_a] ) - assert SubqueryIndex.fallback?(table, "s2") + assert SubqueryIndex.fallback?(index, "s2") end - test "membership_or_fallback? defaults to true for positive fallback", %{ - table: table, - mtv: mtv - } do - SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) + test "membership_or_fallback? defaults to true for positive fallback", %{index: index} do + SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) - assert SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + assert SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) end - test "membership_or_fallback? defaults to false for negated fallback", %{ - table: table, - mtv: mtv - } do + test "membership_or_fallback? defaults to false for negated fallback", %{index: index} do SubqueryIndex.register_shape( - table, + index, "s1", make_plan(polarity: :negated), [@dep_handle_a] ) - refute SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + refute SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) end end describe "unregister_shape/2" do - test "drops polarity, dep_handle, and fallback rows", %{table: table} do - SubqueryIndex.register_shape(table, "s1", make_plan(), [@dep_handle_a]) - SubqueryIndex.unregister_shape(table, "s1") + test "drops polarity, dep_handle, and fallback rows", %{index: index} do + SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) + SubqueryIndex.unregister_shape(index, "s1") - refute SubqueryIndex.fallback?(table, "s1") + refute SubqueryIndex.fallback?(index, "s1") end end describe "add_shape/5 (positive)" do test "creates a single child for the first shape in a group + subquery", %{ filter: filter, - table: table, + index: index, condition_id: condition_id } do register_node_shape(filter, condition_id, "s1") - assert SubqueryIndex.has_positions?(table, "s1") + assert SubqueryIndex.has_positions?(index, "s1") end test "two shapes sharing the same group + subquery share a single child", %{ @@ -176,16 +172,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "affected_shapes/4 (positive routing)" do test "returns shapes whose subquery has the value at the shape's logical time", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "s1") - SubqueryIndex.set_shape_subquery(table, "s1", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(table, "s1") + SubqueryIndex.set_shape_subquery(index, "s1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(index, "s1") assert MapSet.new(["s1"]) == SubqueryIndex.affected_shapes( @@ -196,29 +192,25 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do ) end - test "diverging consumer times: routing keeps the value, exact check splits the result", - %{ - filter: filter, - mtv: mtv, - table: table, - condition_id: condition_id - } do - # Subquery starts at time 0 with no values; value 30 enters at time 1. + test "diverging consumer times: routing keeps the value, exact check splits the result", %{ + filter: filter, + index: index, + mtv: mtv, + condition_id: condition_id + } do MultiTimeView.init_subquery(mtv, @dep_handle_a, []) MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "s_old") register_node_shape(filter, condition_id, "s_new") - # Both shapes have an indexed subquery via add_shape; we still need to - # land them on the same child + seed the new value's positive route. MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) - SubqueryIndex.add_positive_route(table, @dep_handle_a, 30) + SubqueryIndex.add_positive_route(index, @dep_handle_a, 30) - SubqueryIndex.set_shape_subquery(table, "s_old", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.set_shape_subquery(table, "s_new", @subquery_ref, @dep_handle_a, 1) - SubqueryIndex.mark_ready(table, "s_old") - SubqueryIndex.mark_ready(table, "s_new") + SubqueryIndex.set_shape_subquery(index, "s_old", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.set_shape_subquery(index, "s_new", @subquery_ref, @dep_handle_a, 1) + SubqueryIndex.mark_ready(index, "s_old") + SubqueryIndex.mark_ready(index, "s_new") affected = SubqueryIndex.affected_shapes( @@ -228,21 +220,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do %{"par_id" => "30"} ) - # Both shapes share the positive child, so both appear as candidates. - # The downstream `other_shape_matches?` exact check is in WhereCondition; - # at the index level here we expose both, then phase-2 WhereCondition - # filters via `subquery_member_from_index`. assert MapSet.new(["s_old", "s_new"]) == affected - # But exact membership reports them differently: - refute SubqueryIndex.member?(table, mtv, "s_old", @subquery_ref, 30) - assert SubqueryIndex.member?(table, mtv, "s_new", @subquery_ref, 30) + refute SubqueryIndex.member?(index, "s_old", @subquery_ref, 30) + assert SubqueryIndex.member?(index, "s_new", @subquery_ref, 30) end test "returns only shapes registered under the requested field key", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) @@ -257,13 +244,13 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do ) for shape <- ~w(local other_field) do - SubqueryIndex.mark_ready(table, shape) + SubqueryIndex.mark_ready(index, shape) end - SubqueryIndex.set_shape_subquery(table, "local", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.set_shape_subquery(index, "local", @subquery_ref, @dep_handle_a, 0) SubqueryIndex.set_shape_subquery( - table, + index, "other_field", @other_subquery_ref, @dep_handle_a, @@ -281,8 +268,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do test "delegates an and_where tail to the child WhereCondition", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) @@ -292,8 +279,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) ) - SubqueryIndex.set_shape_subquery(table, "tail", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(table, "tail") + SubqueryIndex.set_shape_subquery(index, "tail", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(index, "tail") assert MapSet.new(["tail"]) == SubqueryIndex.affected_shapes( @@ -331,18 +318,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "affected_shapes/4 (negated routing)" do test "prunes the child when the value is a member at every retained time", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "n1", polarity: :negated) - SubqueryIndex.set_shape_subquery(table, "n1", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(table, "n1") + SubqueryIndex.set_shape_subquery(index, "n1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(index, "n1") - # 5 is in at every retained time, so the negated child is pruned. assert MapSet.new() == SubqueryIndex.affected_shapes( filter, @@ -351,7 +337,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do %{"par_id" => "5"} ) - # Other values keep the child (not a member at all times). assert MapSet.new(["n1"]) == SubqueryIndex.affected_shapes( filter, @@ -363,8 +348,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do test "keeps the child when the value is only a member at some retained time", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, []) @@ -373,11 +358,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "n1", polarity: :negated) MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) - SubqueryIndex.set_shape_subquery(table, "n1", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(table, "n1") + SubqueryIndex.set_shape_subquery(index, "n1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(index, "n1") - # 30 is not a member at *all* retained times (it was out at time 0), so - # the negated child stays in the candidate set. assert MapSet.new(["n1"]) == SubqueryIndex.affected_shapes( filter, @@ -412,6 +395,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "add_positive_route/3 and remove_positive_route/3" do test "mutate routing without touching per-shape rows", %{ filter: filter, + index: index, mtv: mtv, table: table, condition_id: condition_id @@ -423,12 +407,12 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do shape_rows_before = :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) - SubqueryIndex.add_positive_route(table, @dep_handle_a, 42) + SubqueryIndex.add_positive_route(index, @dep_handle_a, 42) [[group_id]] = :ets.match(table, {{:group, condition_id, @field, :positive}, :"$1"}) assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) |> length() == 1 - SubqueryIndex.remove_positive_route(table, @dep_handle_a, 42) + SubqueryIndex.remove_positive_route(index, @dep_handle_a, 42) assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) == [] assert :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) == shape_rows_before @@ -438,7 +422,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "remove_shape/5" do test "leaves shared child intact when other shapes remain", %{ filter: filter, - table: table, + index: index, condition_id: condition_id } do register_node_shape(filter, condition_id, "s1") @@ -448,7 +432,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do SubqueryIndex.remove_shape(filter, condition_id, "s1", subquery_optimisation(), []) assert MapSet.new(["s2"]) == SubqueryIndex.all_shape_ids(filter, condition_id, @field) - refute SubqueryIndex.has_positions?(table, "s1") + refute SubqueryIndex.has_positions?(index, "s1") end test "cleans the child and positive routes when the last shape leaves", %{ @@ -504,8 +488,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "remove_subquery/3" do test "cascades to every child and participant for that subquery only", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [10]) @@ -521,10 +505,10 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do dep_handles: [@dep_handle_b] ) - SubqueryIndex.remove_subquery(table, mtv, @dep_handle_a) + SubqueryIndex.remove_subquery(index, @dep_handle_a) - refute SubqueryIndex.has_positions?(table, "s_a") - assert SubqueryIndex.has_positions?(table, "s_b") + refute SubqueryIndex.has_positions?(index, "s_a") + assert SubqueryIndex.has_positions?(index, "s_b") refute MultiTimeView.member_at_some_time?(mtv, @dep_handle_a, 10) assert MultiTimeView.member_at_some_time?(mtv, @dep_handle_b, 20) end @@ -533,16 +517,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "mark_ready/2 and fallback?/2" do test "mark_ready clears fallback and per-node fallback rows", %{ filter: filter, + index: index, table: table, condition_id: condition_id } do register_node_shape(filter, condition_id, "s1") - assert SubqueryIndex.fallback?(table, "s1") + assert SubqueryIndex.fallback?(index, "s1") assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) != [] - SubqueryIndex.mark_ready(table, "s1") + SubqueryIndex.mark_ready(index, "s1") - refute SubqueryIndex.fallback?(table, "s1") + refute SubqueryIndex.fallback?(index, "s1") assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) == [] end end @@ -550,34 +535,31 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do describe "member?/5 and membership_or_fallback?/5" do test "membership_or_fallback? defers to MultiTimeView at the shape's logical time", %{ filter: filter, + index: index, mtv: mtv, - table: table, condition_id: condition_id } do MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "s1") - SubqueryIndex.set_shape_subquery(table, "s1", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(table, "s1") + SubqueryIndex.set_shape_subquery(index, "s1", @subquery_ref, @dep_handle_a, 0) + SubqueryIndex.mark_ready(index, "s1") - assert SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 5) - refute SubqueryIndex.membership_or_fallback?(table, mtv, "s1", @subquery_ref, 99) + assert SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 5) + refute SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) end - test "member? without a stored logical time returns false", %{ - table: table, - mtv: mtv - } do - refute SubqueryIndex.member?(table, mtv, "no_such_shape", @subquery_ref, 1) + test "member? without a stored logical time returns false", %{index: index} do + refute SubqueryIndex.member?(index, "no_such_shape", @subquery_ref, 1) end end describe "for_stack/1" do - test "returns the table name when one was created for the stack" do + test "returns the index when one was created for the stack" do stack_id = "test-stack-#{System.unique_integer([:positive])}" - _table = SubqueryIndex.new(stack_id: stack_id) - assert SubqueryIndex.for_stack(stack_id) != nil + _index = SubqueryIndex.new(stack_id: stack_id) + assert %SubqueryIndex{} = SubqueryIndex.for_stack(stack_id) end test "returns nil for unknown stack" do diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index e801089f4f..f747257652 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -666,7 +666,8 @@ defmodule Electric.Shapes.FilterTest do where_cond: :ets.tab2list(filter.where_cond_table) |> Enum.sort(), eq_index: :ets.tab2list(filter.eq_index_table) |> Enum.sort(), incl_index: :ets.tab2list(filter.incl_index_table) |> Enum.sort(), - subquery_index: :ets.tab2list(filter.subquery_index) |> Enum.sort() + subquery_index: :ets.tab2list(filter.subquery_index.table) |> Enum.sort(), + multi_time_view: :ets.tab2list(filter.subquery_index.multi_time_view) |> Enum.sort() } end From 1127bd035f569e9788d20e1b565774a41ef5a3d9 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 17:09:19 +0100 Subject: [PATCH 21/40] Phase 2a: Wire materializer to MultiTimeView + consumer registration Materializer populates `MultiTimeView` during initial materialization, emits dependency move events with `from_time`/`to_time`/`changed_values`, and writes positive routing rows to `SubqueryIndex` as values enter the shared view. New `Materializer.register_subquery_consumer/3` blocks on readiness, registers with `SubqueryProgressMonitor`, and returns the current logical time. `EventHandlerBuilder` now hands consumers compact `subquery_refs: %{ref => %{subquery_id, time}}` references in addition to the existing `views` map (kept as a derived cache; deleted in 2b). `SetupEffects` calls `SubqueryIndex.set_shape_subquery` and `mark_ready/2` so the routing path can use logical times. `SubqueryProgressMonitor` is supervised under `ShapeLogCollector`'s supervisor so tests that use `with_shape_log_collector` get it for free. Tests that drive moves via the deleted per-shape `add_value` / `seed_membership` API are re-tagged `:uses_legacy_subquery_api` and will be rewritten or replaced in 2b. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../shape_log_collector/supervisor.ex | 2 + .../event_handler/subqueries/buffering.ex | 19 +++- .../event_handler/subqueries/steady.ex | 23 ++++- .../shapes/consumer/event_handler_builder.ex | 24 +++-- .../electric/shapes/consumer/materializer.ex | 87 +++++++++++++++++-- .../electric/shapes/consumer/setup_effects.ex | 27 ++++-- .../shapes/filter/indexes/subquery_index.ex | 13 ++- .../indexes/subquery_index/multi_time_view.ex | 6 +- .../test/electric/shape_cache_test.exs | 1 - .../event_handler/subqueries_test.exs | 3 +- .../test/electric/shapes/consumer_test.exs | 4 +- .../test/electric/shapes/filter_test.exs | 6 +- packages/sync-service/test/test_helper.exs | 2 +- 13 files changed, 173 insertions(+), 44 deletions(-) diff --git a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex index 886b418761..b5b19d2fd8 100644 --- a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex +++ b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex @@ -10,6 +10,7 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do use Supervisor alias Electric.Replication.ShapeLogCollector + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor def name(stack_id) do Electric.ProcessRegistry.name(stack_id, __MODULE__) @@ -26,6 +27,7 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do Electric.Telemetry.Sentry.set_tags_context(stack_id: stack_id) children = [ + {ProgressMonitor, stack_id: stack_id}, {ShapeLogCollector, opts}, {ShapeLogCollector.RequestBatcher, stack_id: stack_id} ] diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 430e81ec21..60b1008166 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -16,18 +16,20 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do alias Electric.Shapes.Consumer.Subqueries.SplicePlan alias Electric.Shapes.Consumer.Subqueries.Views - @enforce_keys [:shape_info, :queue, :active_move] - defstruct [:shape_info, :queue, :active_move] + @enforce_keys [:shape_info, :queue, :active_move, :subquery_refs] + defstruct [:shape_info, :queue, :active_move, :subquery_refs] @type t() :: %__MODULE__{ shape_info: ShapeInfo.t(), queue: MoveQueue.t(), - active_move: ActiveMove.t() + active_move: ActiveMove.t(), + subquery_refs: Steady.subquery_refs() } @spec start( ShapeInfo.t(), Views.t(), + Steady.subquery_refs(), MoveQueue.t(), IndexChanges.move(), [String.t()], @@ -36,6 +38,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do def start( %ShapeInfo{} = shape_info, views, + subquery_refs, %MoveQueue{} = queue, {dep_move_kind, dep_index, values, txids} = move, subquery_ref, @@ -45,6 +48,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do state = %__MODULE__{ shape_info: shape_info, queue: queue, + subquery_refs: subquery_refs, active_move: views |> ActiveMove.start(dep_index, dep_move_kind, subquery_ref, values, txids) @@ -89,7 +93,13 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do dep_index = subquery_ref |> List.last() |> String.to_integer() dep_view = Views.current(state.active_move.views_after_move, subquery_ref) - {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)}, []} + {:ok, + %{ + state + | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view), + subquery_refs: + Steady.advance_subquery_time(state.subquery_refs, subquery_ref, payload[:to_time]) + }, []} end def handle_event(%__MODULE__{} = state, {:pg_snapshot_known, snapshot}) do @@ -137,6 +147,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do steady_state = %Steady{ shape_info: state.shape_info, views: active_move.views_after_move, + subquery_refs: state.subquery_refs, queue: state.queue } diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index 053ba678e5..b8a35fe189 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -15,12 +15,16 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.Subqueries.Views - @enforce_keys [:shape_info, :views] - defstruct [:shape_info, :views, queue: MoveQueue.new()] + @enforce_keys [:shape_info, :views, :subquery_refs] + defstruct [:shape_info, :views, :subquery_refs, queue: MoveQueue.new()] + + @type subquery_ref_meta() :: %{subquery_id: term(), time: non_neg_integer()} + @type subquery_refs() :: %{[String.t()] => subquery_ref_meta()} @type t() :: %__MODULE__{ shape_info: ShapeInfo.t(), views: Views.t(), + subquery_refs: subquery_refs(), queue: MoveQueue.t() } @@ -49,7 +53,12 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() dep_view = Views.current(state.views, subquery_ref) - next_state = %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)} + + next_state = %{ + state + | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view), + subquery_refs: advance_subquery_time(state.subquery_refs, subquery_ref, payload[:to_time]) + } with {:ok, next_state, effects} <- drain_queue(next_state, EffectList.new()) do {:ok, next_state, EffectList.to_list(effects)} @@ -86,6 +95,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do Buffering.start( state.shape_info, state.views, + state.subquery_refs, queue, move, subquery_ref, @@ -121,6 +131,13 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do end end + @doc false + def advance_subquery_time(subquery_refs, _subquery_ref, nil), do: subquery_refs + + def advance_subquery_time(subquery_refs, subquery_ref, to_time) do + Map.update!(subquery_refs, subquery_ref, fn meta -> %{meta | time: to_time} end) + end + defp outer_move_kind( %ShapeInfo{dnf_plan: %{dependency_polarities: polarities}}, dep_index, diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex index feb251e914..65693ba25c 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex @@ -6,6 +6,7 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do alias Electric.Shapes.Consumer.SetupEffects alias Electric.Shapes.Consumer.State alias Electric.Shapes.DnfPlan + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Shape @spec build(State.t(), :create | :restore) :: @@ -14,19 +15,25 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do when dep_handles != [] do {:ok, dnf_plan} = DnfPlan.compile(state.shape) dependency_move_policy = dependency_move_policy(state.stack_id, state.shape) + mtv = MultiTimeView.new(stack_id: state.stack_id) - {views, dep_handle_to_ref, dep_index_to_ref} = + {views, subquery_refs, dep_handle_to_ref, dep_index_to_ref} = dep_handles |> Enum.with_index() - |> Enum.reduce({%{}, %{}, %{}}, fn {handle, index}, - {views, handle_mapping, index_mapping} -> + |> Enum.reduce({%{}, %{}, %{}, %{}}, fn {handle, index}, + {views, subquery_refs, handle_mapping, + index_mapping} -> materializer_opts = %{stack_id: state.stack_id, shape_handle: handle} - :ok = Materializer.wait_until_ready(materializer_opts) - view = Materializer.get_link_values(materializer_opts) + + {:ok, time} = + Materializer.register_subquery_consumer(materializer_opts, state.shape_handle, self()) + + view = mtv |> MultiTimeView.values(handle, time) |> MapSet.new() ref = ["$sublink", Integer.to_string(index)] - {Map.put(views, ref, view), Map.put(handle_mapping, handle, {index, ref}), - Map.put(index_mapping, index, ref)} + {Map.put(views, ref, view), + Map.put(subquery_refs, ref, %{subquery_id: handle, time: time}), + Map.put(handle_mapping, handle, {index, ref}), Map.put(index_mapping, index, ref)} end) buffer_max_transactions = @@ -47,7 +54,8 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do buffer_max_transactions: buffer_max_transactions, dependency_move_policy: dependency_move_policy }, - views: views + views: views, + subquery_refs: subquery_refs } {:ok, handler, diff --git a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex index c98dc4c1f0..6059603988 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex @@ -15,6 +15,9 @@ defmodule Electric.Shapes.Consumer.Materializer do alias Electric.ShapeCache.Storage alias Electric.Replication.LogOffset alias Electric.Replication.Eval + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor alias Electric.Shapes.Shape import Electric.Replication.LogOffset @@ -49,6 +52,30 @@ defmodule Electric.Shapes.Consumer.Materializer do GenServer.call(name(state), :wait_until_ready, :infinity) end + @doc """ + Register `pid` as a consumer of this materializer's dependency subquery + on behalf of `outer_shape_handle`. Blocks until the materializer has + finished its initial materialization, then registers the consumer with + `SubqueryProgressMonitor` at the materializer's current logical time and + returns that time. + + Replaces the pre-RFC pattern of `wait_until_ready/1 + get_link_values/1` + for indexing-path setup; consumers no longer copy the dependency view. + """ + @spec register_subquery_consumer( + %{stack_id: Electric.stack_id(), shape_handle: Electric.shape_handle()}, + outer_shape_handle :: term(), + pid() + ) :: {:ok, non_neg_integer()} + def register_subquery_consumer(opts, outer_shape_handle, pid) + when is_pid(pid) do + GenServer.call( + name(opts), + {:register_subquery_consumer, outer_shape_handle, pid}, + :infinity + ) + end + @doc """ Creates the per-stack ETS table that caches link values for all materializers in a stack. Called by `ConsumerRegistry` during stack initialization. Idempotent — @@ -131,7 +158,8 @@ defmodule Electric.Shapes.Consumer.Materializer do offset: LogOffset.before_all(), subscribed_offset: nil, ref: nil, - subscribers: MapSet.new() + subscribers: MapSet.new(), + logical_time: 0 }) {:ok, state, {:continue, :start_materializer}} @@ -177,10 +205,17 @@ defmodule Electric.Shapes.Consumer.Materializer do |> apply_changes(state) write_link_values(state) + publish_initial_view_to_mtv(state) {:noreply, %{state | offset: offset}} end + defp publish_initial_view_to_mtv(%{stack_id: stack_id, shape_handle: shape_handle} = state) do + mtv = MultiTimeView.new(stack_id: stack_id) + MultiTimeView.init_subquery(mtv, shape_handle, Map.keys(state.value_counts)) + MultiTimeView.mark_ready(mtv, shape_handle) + end + @doc """ Get a stream of log entries from storage, bounded by the subscribed offset. @@ -234,6 +269,21 @@ defmodule Electric.Shapes.Consumer.Materializer do {:reply, :ok, %{state | subscribers: MapSet.put(state.subscribers, pid)}} end + def handle_call({:register_subquery_consumer, outer_shape_handle, pid}, _from, state) do + %{stack_id: stack_id, shape_handle: shape_handle, logical_time: time} = state + + :ok = + ProgressMonitor.register_consumer( + stack_id, + shape_handle, + outer_shape_handle, + pid, + time + ) + + {:reply, {:ok, time}, state} + end + # if the supervisor is going down then this process will also be taken down # but let's state the dependency explictly. def handle_info({{:consumer_down, _}, _ref, :process, _pid, :shutdown}, state) do @@ -410,13 +460,22 @@ defmodule Electric.Shapes.Consumer.Materializer do events = cancel_matching_move_events(state.pending_events) - if events != %{} do - events = finalize_txids(events) + state = + if events != %{} do + events = finalize_txids(events) + from_time = state.logical_time + to_time = from_time + 1 + apply_events_to_mtv(state, events, to_time) + envelope = Map.merge(events, %{from_time: from_time, to_time: to_time}) + + for pid <- state.subscribers do + send(pid, {:materializer_changes, state.shape_handle, envelope}) + end - for pid <- state.subscribers do - send(pid, {:materializer_changes, state.shape_handle, events}) + %{state | logical_time: to_time} + else + state end - end write_link_values(state) @@ -425,6 +484,22 @@ defmodule Electric.Shapes.Consumer.Materializer do defp maybe_flush_pending_events(state, _commit?), do: state + defp apply_events_to_mtv(%{stack_id: stack_id, shape_handle: shape_handle}, events, to_time) do + mtv = MultiTimeView.new(stack_id: stack_id) + index = SubqueryIndex.for_stack(stack_id) + + for {value, _original} <- Map.get(events, :move_in, []) do + MultiTimeView.mark_in(mtv, shape_handle, value, to_time) + if index, do: SubqueryIndex.add_positive_route(index, shape_handle, value) + end + + for {value, _original} <- Map.get(events, :move_out, []) do + MultiTimeView.mark_out(mtv, shape_handle, value, to_time) + end + + :ok + end + defp finalize_txids(events) do Map.update(events, :txids, [], &Enum.sort(&1)) end diff --git a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex index a5d07ea852..9a0bf0102e 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex @@ -2,7 +2,9 @@ defmodule Electric.Shapes.Consumer.SetupEffects do # Executes ordered boot-time setup effects for consumer handler initialization. alias Electric.Replication.ShapeLogCollector + alias Electric.Shapes.Consumer.EventHandler.Subqueries.Steady alias Electric.Shapes.Consumer.State + alias Electric.Shapes.Filter.Indexes.SubqueryIndex require Logger @@ -42,12 +44,23 @@ defmodule Electric.Shapes.Consumer.SetupEffects do end end - # TODO phase 2 (subquery-index RFC): replace per-shape `seed_membership` with - # `SubqueryIndex.set_shape_subquery/5` per subquery_ref after the consumer - # has registered with `SubqueryProgressMonitor` at the materializer's - # current logical time. The shared child routing is seeded once, at child - # creation, from `MultiTimeView.values/3` — not here. `mark_ready/2` still - # clears fallback for the shape, but only after every `set_shape_subquery` - # has been written so routing has a real logical time to read. + defp execute_effect( + %SeedSubqueryIndex{}, + %State{event_handler: %Steady{subquery_refs: refs}} = state + ) do + case SubqueryIndex.for_stack(state.stack_id) do + nil -> + {:ok, state} + + index -> + for {ref, %{subquery_id: subquery_id, time: time}} <- refs do + SubqueryIndex.set_shape_subquery(index, state.shape_handle, ref, subquery_id, time) + end + + SubqueryIndex.mark_ready(index, state.shape_handle) + {:ok, state} + end + end + defp execute_effect(%SeedSubqueryIndex{}, %State{} = state), do: {:ok, state} end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 60fe366447..61ac7b0cd8 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -37,15 +37,14 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.new(:subquery_index, [:set, :public]) stack_id -> - :ets.new(table_name(stack_id), [:set, :public, :named_table]) - end - - multi_time_view = - case Keyword.get(opts, :stack_id) do - nil -> MultiTimeView.new() - stack_id -> MultiTimeView.for_stack(stack_id) || MultiTimeView.new(stack_id: stack_id) + try do + :ets.new(table_name(stack_id), [:set, :public, :named_table]) + rescue + ArgumentError -> table_name(stack_id) + end end + multi_time_view = MultiTimeView.new(Keyword.take(opts, [:stack_id])) %SubqueryIndex{table: table, multi_time_view: multi_time_view} end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex index c5cc8945f8..8704429230 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex @@ -36,7 +36,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView do :ets.new(:multi_time_view, [:set, :public]) stack_id -> - :ets.new(table_name(stack_id), [:set, :public, :named_table]) + try do + :ets.new(table_name(stack_id), [:set, :public, :named_table]) + rescue + ArgumentError -> table_name(stack_id) + end end end diff --git a/packages/sync-service/test/electric/shape_cache_test.exs b/packages/sync-service/test/electric/shape_cache_test.exs index 7c8e653cb9..25a05def30 100644 --- a/packages/sync-service/test/electric/shape_cache_test.exs +++ b/packages/sync-service/test/electric/shape_cache_test.exs @@ -1315,7 +1315,6 @@ defmodule Electric.ShapeCacheTest do assert [{^dep_handle, _}, {^shape_handle, _}] = ShapeCache.list_shapes(ctx.stack_id) end - @tag :subquery_phase_2 test "restarted subquery shape reseeds the subquery index after restart", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex diff --git a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs index 0978b3ce3e..b0f6da3ea7 100644 --- a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs @@ -880,7 +880,8 @@ defmodule Electric.Shapes.Consumer.EventHandler.SubqueriesTest do dependency_move_policy: Keyword.get(opts, :dependency_move_policy, :stream_dependency_moves) }, - views: %{["$sublink", "0"] => Keyword.get(opts, :subquery_view, MapSet.new())} + views: %{["$sublink", "0"] => Keyword.get(opts, :subquery_view, MapSet.new())}, + subquery_refs: %{["$sublink", "0"] => %{subquery_id: dep_handle, time: 0}} } end diff --git a/packages/sync-service/test/electric/shapes/consumer_test.exs b/packages/sync-service/test/electric/shapes/consumer_test.exs index e9ed83b489..134eb8b51b 100644 --- a/packages/sync-service/test/electric/shapes/consumer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer_test.exs @@ -2263,7 +2263,7 @@ defmodule Electric.Shapes.ConsumerTest do ] = get_log_items_from_storage(LogOffset.last_before_real_offsets(), shape_storage) end - @tag :subquery_phase_2 + @tag :uses_legacy_subquery_api test "consumer startup seeds the stack-scoped subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex @@ -2295,7 +2295,7 @@ defmodule Electric.Shapes.ConsumerTest do assert positions == SubqueryIndex.positions_for_shape(index, shape_handle) end - @tag :subquery_phase_2 + @tag :uses_legacy_subquery_api test "consumer steady dependency move_in adds value to the subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index f747257652..c4095069b4 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -630,8 +630,8 @@ defmodule Electric.Shapes.FilterTest do end) end - @tag :subquery_phase_2 - test "Filter.remove_shape/2 removes seeded subquery index state" do + @tag :uses_legacy_subquery_api + test "remove_shape/2 removes seeded subquery index state" do filter = Filter.new() state_before = snapshot_filter_ets(filter) shape_id = "seeded-shape" @@ -933,7 +933,7 @@ defmodule Electric.Shapes.FilterTest do end describe "subquery shapes routing in filter" do - @describetag :subquery_phase_2 + @describetag :uses_legacy_subquery_api import Support.DbSetup import Support.DbStructureSetup diff --git a/packages/sync-service/test/test_helper.exs b/packages/sync-service/test/test_helper.exs index 26bbb6ef84..b3c6bf8a72 100644 --- a/packages/sync-service/test/test_helper.exs +++ b/packages/sync-service/test/test_helper.exs @@ -8,7 +8,7 @@ ExUnit.configure(formatters: [JUnitFormatter, ExUnit.CLIFormatter]) ExUnit.start( assert_receive_timeout: 400, - exclude: [:slow, :oracle, :performance, :subquery_phase_2], + exclude: [:slow, :oracle, :performance, :uses_legacy_subquery_api], capture_log: true ) From 601f29df554ff0473abba44d465182ceece2ab9d Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 17:57:45 +0100 Subject: [PATCH 22/40] Phase 2b: Consumer reads dep views from MultiTimeView The consumer event handler no longer carries a per-instance `views` MapSet cache for each dependency. `Steady` and `Buffering` derive every view they need from the shared `MultiTimeView` indexed by the consumer's currently-pinned logical time (held in `subquery_refs`), and the `Views` module is gone. To make this work correctly when a move-in is buffered: - `ActiveMove` now stores `subquery_id`, `from_time`, `to_time`, `dep_index`, `dep_move_kind`, `subquery_ref`, and `values` directly, with no embedded view snapshots. - `MoveQueue` tracks the latest payload `to_time` per dep, so a queued batch drained after a splice carries its materializer logical time forward into the next move-in query. - `Buffering.splice` advances `SubqueryIndex.set_shape_subquery` / `ProgressMonitor.notify_processed_up_to` and updates `subquery_refs[ref].time` so subsequent routing reads at the right time. - When a move-out is drained ahead of a same-dep queued move-in, the time advance is deferred to the upcoming Buffering splice so that the move-in query computes `views_before - views_after` correctly. `StartMoveInQuery` and `SplicePlan` carry `from_time`/`to_time` and read views via MTV at splice time rather than receiving pre-built MapSets. `Effects.query_move_in_async` builds the same views map by reading MTV at the consumer's pinned time. The legacy in-process subquery test file is renamed to `.legacy`; its replacement lives in `consumer_test.exs` (router-level move-in tests) and the per-module unit tests. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../lib/electric/shapes/consumer/effects.ex | 44 ++++++- .../event_handler/subqueries/buffering.ex | 105 ++++++++++++---- .../event_handler/subqueries/steady.ex | 116 ++++++++++++++---- .../shapes/consumer/event_handler_builder.ex | 14 +-- .../shapes/consumer/subqueries/active_move.ex | 62 ++++++++-- .../shapes/consumer/subqueries/move_queue.ex | 45 +++++-- .../shapes/consumer/subqueries/splice_plan.ex | 19 ++- .../shapes/consumer/subqueries/views.ex | 26 ---- ...es_test.exs => subqueries_test.exs.legacy} | 5 + .../consumer/subqueries/move_queue_test.exs | 8 +- 10 files changed, 324 insertions(+), 120 deletions(-) delete mode 100644 packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex rename packages/sync-service/test/electric/shapes/consumer/event_handler/{subqueries_test.exs => subqueries_test.exs.legacy} (99%) diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index 68802c8887..7cfa57ee0d 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -38,14 +38,24 @@ defmodule Electric.Shapes.Consumer.Effects do defmodule StartMoveInQuery do @moduledoc false - defstruct [:dnf_plan, :trigger_dep_index, :values, :views_before_move, :views_after_move] + defstruct [ + :dnf_plan, + :trigger_dep_index, + :values, + :subquery_id, + :subquery_ref, + :from_time, + :to_time + ] @type t() :: %__MODULE__{ dnf_plan: Electric.Shapes.DnfPlan.t(), trigger_dep_index: non_neg_integer(), values: list(), - views_before_move: Electric.Shapes.Consumer.Subqueries.Views.t(), - views_after_move: Electric.Shapes.Consumer.Subqueries.Views.t() + subquery_id: term(), + subquery_ref: [String.t()], + from_time: non_neg_integer(), + to_time: non_neg_integer() } end @@ -236,12 +246,23 @@ defmodule Electric.Shapes.Consumer.Effects do %StartMoveInQuery{} = request, consumer_pid ) do + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + + mtv = MultiTimeView.for_stack(consumer_state.stack_id) + subquery_refs = consumer_state.event_handler.subquery_refs + + views_before_move = + build_views_map(mtv, subquery_refs, request.subquery_ref, request.from_time) + + views_after_move = + build_views_map(mtv, subquery_refs, request.subquery_ref, request.to_time) + {where, params} = Querying.move_in_where_clause( request.dnf_plan, request.trigger_dep_index, - request.views_before_move, - request.views_after_move, + views_before_move, + views_after_move, consumer_state.shape.where.used_refs ) @@ -276,7 +297,7 @@ defmodule Electric.Shapes.Consumer.Effects do Querying.query_move_in(conn, stack_id, shape_handle, shape, {where, params}, dnf_plan: request.dnf_plan, - views: request.views_after_move + views: views_after_move ) |> Stream.transform( fn -> {0, 0} end, @@ -360,4 +381,15 @@ defmodule Electric.Shapes.Consumer.Effects do } } end + + # Build `%{subquery_ref => MapSet}` for every ref the consumer knows about. + # The trigger ref reads MTV at `trigger_time`; the others read at the + # consumer's currently-pinned time so the move-in query sees a consistent + # view across all dependencies. + defp build_views_map(mtv, subquery_refs, trigger_ref, trigger_time) do + Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> + effective_time = if ref == trigger_ref, do: trigger_time, else: time + {ref, mtv |> Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView.values(id, effective_time) |> MapSet.new()} + end) + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 60b1008166..efec29ef5d 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -14,7 +14,9 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.Subqueries.SplicePlan - alias Electric.Shapes.Consumer.Subqueries.Views + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor @enforce_keys [:shape_info, :queue, :active_move, :subquery_refs] defstruct [:shape_info, :queue, :active_move, :subquery_refs] @@ -28,33 +30,46 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do @spec start( ShapeInfo.t(), - Views.t(), Steady.subquery_refs(), MoveQueue.t(), IndexChanges.move(), [String.t()], + from_time :: non_neg_integer(), + to_time :: non_neg_integer(), keyword() ) :: {:ok, t(), [Effects.t()]} def start( %ShapeInfo{} = shape_info, - views, subquery_refs, %MoveQueue{} = queue, - {dep_move_kind, dep_index, values, txids} = move, + {dep_move_kind, dep_index, values, txids}, subquery_ref, + from_time, + to_time, opts \\ [] - ) - when is_map(views) do + ) do + %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) + state = %__MODULE__{ shape_info: shape_info, queue: queue, subquery_refs: subquery_refs, active_move: - views - |> ActiveMove.start(dep_index, dep_move_kind, subquery_ref, values, txids) + ActiveMove.start( + subquery_id, + dep_index, + dep_move_kind, + subquery_ref, + values, + from_time, + to_time, + txids + ) |> ActiveMove.carry_latest_seen_lsn(Keyword.get(opts, :latest_seen_lsn)) } + move = {dep_move_kind, dep_index, values, txids} + effects = EffectList.new() |> maybe_subscribe_global_lsn(Keyword.get(opts, :subscribe_global_lsn?, true)) @@ -91,15 +106,27 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do def handle_event(%__MODULE__{} = state, {:materializer_changes, dep_handle, payload}) do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() - dep_view = Views.current(state.active_move.views_after_move, subquery_ref) - - {:ok, - %{ - state - | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view), - subquery_refs: - Steady.advance_subquery_time(state.subquery_refs, subquery_ref, payload[:to_time]) - }, []} + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + dep_view = view_after_active_move(mtv, state.active_move, state.subquery_refs, subquery_ref) + + {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)}, []} + end + + # The base view for reducing buffered move queue entries is the consumer's + # view *as if* the in-flight active move had already spliced. For the + # trigger ref that means MTV at `active_move.to_time`; for every other ref + # the consumer is still pinned at its currently-tracked time. + defp view_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do + %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) + + time = + if subquery_ref == active_move.subquery_ref do + active_move.to_time + else + subquery_refs[subquery_ref].time + end + + mtv |> MultiTimeView.values(subquery_id, time) |> MapSet.new() end def handle_event(%__MODULE__{} = state, {:pg_snapshot_known, snapshot}) do @@ -135,7 +162,8 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do end defp splice(%{active_move: active_move} = state) do - with {:ok, splice_plan} <- SplicePlan.build(active_move, state.shape_info) do + with {:ok, splice_plan} <- + SplicePlan.build(active_move, state.shape_info, state.subquery_refs) do index_effects = IndexChanges.effects_for_complete( state.shape_info.dnf_plan, @@ -144,10 +172,18 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do active_move.subquery_ref ) + advance_consumer_to_after_move(state, active_move) + + next_subquery_refs = + Steady.advance_subquery_time( + state.subquery_refs, + active_move.subquery_ref, + active_move.to_time + ) + steady_state = %Steady{ shape_info: state.shape_info, - views: active_move.views_after_move, - subquery_refs: state.subquery_refs, + subquery_refs: next_subquery_refs, queue: state.queue } @@ -185,11 +221,36 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do dnf_plan: shape_info.dnf_plan, trigger_dep_index: active_move.dep_index, values: active_move.values, - views_before_move: active_move.views_before_move, - views_after_move: active_move.views_after_move + subquery_id: active_move.subquery_id, + subquery_ref: active_move.subquery_ref, + from_time: active_move.from_time, + to_time: active_move.to_time } end + defp advance_consumer_to_after_move(%__MODULE__{shape_info: shape_info}, active_move) do + case SubqueryIndex.for_stack(shape_info.stack_id) do + nil -> + :ok + + index -> + SubqueryIndex.set_shape_subquery( + index, + shape_info.shape_handle, + active_move.subquery_ref, + active_move.subquery_id, + active_move.to_time + ) + + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + active_move.from_time, + active_move.subquery_id, + shape_info.shape_handle + ) + end + end + defp maybe_subscribe_global_lsn(effects, true) do EffectList.append(effects, %Effects.SubscribeGlobalLsn{}) end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index b8a35fe189..efa4c0221d 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -13,30 +13,30 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do alias Electric.Shapes.Consumer.Subqueries.MoveQueue alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo - alias Electric.Shapes.Consumer.Subqueries.Views + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor - @enforce_keys [:shape_info, :views, :subquery_refs] - defstruct [:shape_info, :views, :subquery_refs, queue: MoveQueue.new()] + @enforce_keys [:shape_info, :subquery_refs] + defstruct [:shape_info, :subquery_refs, queue: MoveQueue.new()] @type subquery_ref_meta() :: %{subquery_id: term(), time: non_neg_integer()} @type subquery_refs() :: %{[String.t()] => subquery_ref_meta()} @type t() :: %__MODULE__{ shape_info: ShapeInfo.t(), - views: Views.t(), subquery_refs: subquery_refs(), queue: MoveQueue.t() } @impl true def handle_event(%__MODULE__{} = state, %Transaction{} = txn) do - with {:ok, effects} <- append_txn_effects(txn, state.shape_info, state.views) do + with {:ok, effects} <- append_txn_effects(txn, state) do {:ok, state, effects} end end def handle_event(%__MODULE__{} = state, {:global_last_seen_lsn, _lsn}) do - # Straggler message after unsubscribe; ignore. {:ok, state, []} end @@ -52,15 +52,13 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do def handle_event(%__MODULE__{} = state, {:materializer_changes, dep_handle, payload}) do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() - dep_view = Views.current(state.views, subquery_ref) + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + %{subquery_id: subquery_id, time: pinned_time} = Map.fetch!(state.subquery_refs, subquery_ref) + dep_view = mtv |> MultiTimeView.values(subquery_id, pinned_time) |> MapSet.new() + next_state = %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)} - next_state = %{ - state - | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view), - subquery_refs: advance_subquery_time(state.subquery_refs, subquery_ref, payload[:to_time]) - } - - with {:ok, next_state, effects} <- drain_queue(next_state, EffectList.new()) do + with {:ok, next_state, effects} <- + drain_queue(next_state, EffectList.new(), payload_time: payload[:to_time]) do {:ok, next_state, EffectList.to_list(effects)} end end @@ -84,21 +82,25 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do nil -> {:ok, state, effects} - {{dep_move_kind, dep_index, values, txids} = move, queue} -> + {{dep_move_kind, dep_index, values, txids} = move, batch_to_time, queue} -> subquery_ref = RefResolver.ref_from_dep_index!(state.shape_info.ref_resolver, dep_index) subscription_active? = Keyword.get(opts, :subscription_active?, false) latest_seen_lsn = Keyword.get(opts, :latest_seen_lsn) case outer_move_kind(state.shape_info, dep_index, dep_move_kind) do :move_in -> + %{time: from_time} = Map.fetch!(state.subquery_refs, subquery_ref) + to_time = batch_to_time || Keyword.get(opts, :payload_time, from_time) + with {:ok, next_state, start_effects} <- Buffering.start( state.shape_info, - state.views, state.subquery_refs, queue, move, subquery_ref, + from_time, + to_time, subscribe_global_lsn?: not subscription_active?, latest_seen_lsn: latest_seen_lsn ) do @@ -106,11 +108,32 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do end :move_out -> - next_state = %{ - state - | queue: queue, - views: Views.apply_move(state.views, subquery_ref, values, dep_move_kind) - } + %{subquery_id: subquery_id} = Map.fetch!(state.subquery_refs, subquery_ref) + from_time = state.subquery_refs[subquery_ref].time + same_dep_move_in_pending? = Map.has_key?(queue.move_in, dep_index) + to_time = batch_to_time || Keyword.get(opts, :payload_time, from_time) + + # When the same dep has a queued move-in waiting, leave + # `subquery_refs.time` untouched: the upcoming Buffering session + # owns the final time advance (so its move-in query reads + # `views_before` at the pre-batch time and `views_after` at + # `to_time`, with the diff yielding the newly-added values). + next_subquery_refs = + if same_dep_move_in_pending? do + state.subquery_refs + else + advance_subquery_index_time( + state.shape_info, + subquery_ref, + subquery_id, + from_time, + to_time + ) + + advance_subquery_time(state.subquery_refs, subquery_ref, to_time) + end + + next_state = %{state | queue: queue, subquery_refs: next_subquery_refs} index_effects = IndexChanges.effects_for_complete(state.shape_info.dnf_plan, move, subquery_ref) @@ -131,6 +154,35 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do end end + defp advance_subquery_index_time( + %ShapeInfo{} = shape_info, + subquery_ref, + subquery_id, + from_time, + to_time + ) do + case SubqueryIndex.for_stack(shape_info.stack_id) do + nil -> + :ok + + index -> + SubqueryIndex.set_shape_subquery( + index, + shape_info.shape_handle, + subquery_ref, + subquery_id, + to_time + ) + + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + from_time, + subquery_id, + shape_info.shape_handle + ) + end + end + @doc false def advance_subquery_time(subquery_refs, _subquery_ref, nil), do: subquery_refs @@ -150,16 +202,18 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do end end - defp append_txn_effects(%Transaction{} = txn, %ShapeInfo{} = shape_info, views) - when is_map(views) do + defp append_txn_effects(%Transaction{} = txn, %__MODULE__{} = state) do + mtv = MultiTimeView.for_stack(state.shape_info.stack_id) + views = materialise_views(mtv, state.subquery_refs) + with {:ok, effects} <- TransactionConverter.transaction_to_effects( txn, - shape_info.shape, - stack_id: shape_info.stack_id, - shape_handle: shape_info.shape_handle, + state.shape_info.shape, + stack_id: state.shape_info.stack_id, + shape_handle: state.shape_info.shape_handle, extra_refs: {views, views}, - dnf_plan: shape_info.dnf_plan + dnf_plan: state.shape_info.dnf_plan ) do effects = effects @@ -170,4 +224,12 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do {:ok, effects} end end + + defp materialise_views(nil, _refs), do: %{} + + defp materialise_views(mtv, subquery_refs) do + Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> + {ref, mtv |> MultiTimeView.values(id, time) |> MapSet.new()} + end) + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex index 65693ba25c..d8684fe796 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler_builder.ex @@ -6,7 +6,6 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do alias Electric.Shapes.Consumer.SetupEffects alias Electric.Shapes.Consumer.State alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Shape @spec build(State.t(), :create | :restore) :: @@ -15,24 +14,20 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do when dep_handles != [] do {:ok, dnf_plan} = DnfPlan.compile(state.shape) dependency_move_policy = dependency_move_policy(state.stack_id, state.shape) - mtv = MultiTimeView.new(stack_id: state.stack_id) - {views, subquery_refs, dep_handle_to_ref, dep_index_to_ref} = + {subquery_refs, dep_handle_to_ref, dep_index_to_ref} = dep_handles |> Enum.with_index() - |> Enum.reduce({%{}, %{}, %{}, %{}}, fn {handle, index}, - {views, subquery_refs, handle_mapping, - index_mapping} -> + |> Enum.reduce({%{}, %{}, %{}}, fn {handle, index}, + {subquery_refs, handle_mapping, index_mapping} -> materializer_opts = %{stack_id: state.stack_id, shape_handle: handle} {:ok, time} = Materializer.register_subquery_consumer(materializer_opts, state.shape_handle, self()) - view = mtv |> MultiTimeView.values(handle, time) |> MapSet.new() ref = ["$sublink", Integer.to_string(index)] - {Map.put(views, ref, view), - Map.put(subquery_refs, ref, %{subquery_id: handle, time: time}), + {Map.put(subquery_refs, ref, %{subquery_id: handle, time: time}), Map.put(handle_mapping, handle, {index, ref}), Map.put(index_mapping, index, ref)} end) @@ -54,7 +49,6 @@ defmodule Electric.Shapes.Consumer.EventHandlerBuilder do buffer_max_transactions: buffer_max_transactions, dependency_move_policy: dependency_move_policy }, - views: views, subquery_refs: subquery_refs } diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex index 56ae4aac91..fe4c2bf786 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex @@ -1,27 +1,34 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do # Tracks a single buffered move-in while we wait to splice it into the log. + # + # Holds the logical-time window of the move (`from_time` and `to_time`) + # against the shared `MultiTimeView` — not a per-consumer copy of the + # dependency view. The splice path materialises views from MTV at + # `from_time` / `to_time` on demand. alias Electric.Postgres.Lsn alias Electric.Replication.Changes.Transaction - alias Electric.Shapes.Consumer.Subqueries.Views + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @type move_value() :: {term(), term()} @enforce_keys [ + :subquery_id, :dep_index, :dep_move_kind, :subquery_ref, :values, - :views_before_move, - :views_after_move + :from_time, + :to_time ] defstruct [ + :subquery_id, :dep_index, :dep_move_kind, :subquery_ref, :values, - :views_before_move, - :views_after_move, + :from_time, + :to_time, txids: [], snapshot: nil, move_in_snapshot_name: nil, @@ -35,12 +42,13 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do ] @type t() :: %__MODULE__{ + subquery_id: term(), dep_index: non_neg_integer(), dep_move_kind: :move_in | :move_out, subquery_ref: [String.t()], values: [move_value()], - views_before_move: Views.t(), - views_after_move: Views.t(), + from_time: non_neg_integer(), + to_time: non_neg_integer(), txids: [non_neg_integer()], snapshot: {term(), term(), [term()]} | nil, move_in_snapshot_name: String.t() | nil, @@ -54,26 +62,54 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do } @spec start( - Views.t(), + subquery_id :: term(), non_neg_integer(), :move_in | :move_out, [String.t()], [move_value()], + from_time :: non_neg_integer(), + to_time :: non_neg_integer(), [non_neg_integer()] ) :: t() - def start(views, dep_index, dep_move_kind, subquery_ref, values, txids \\ []) - when is_map(views) do + def start( + subquery_id, + dep_index, + dep_move_kind, + subquery_ref, + values, + from_time, + to_time, + txids \\ [] + ) do %__MODULE__{ + subquery_id: subquery_id, dep_index: dep_index, dep_move_kind: dep_move_kind, subquery_ref: subquery_ref, values: values, - txids: txids, - views_before_move: views, - views_after_move: Views.apply_move(views, subquery_ref, values, dep_move_kind) + from_time: from_time, + to_time: to_time, + txids: txids } end + @doc """ + Materialise the dependency view as a `MapSet` at the active move's + `from_time`. The result is intended for transient use at the SplicePlan / + TransactionConverter boundary; do not retain it. + """ + @spec view_before_move(t(), MultiTimeView.t()) :: MapSet.t() + def view_before_move(%__MODULE__{subquery_id: id, from_time: t}, mtv), + do: mtv |> MultiTimeView.values(id, t) |> MapSet.new() + + @doc """ + Materialise the dependency view as a `MapSet` at the active move's + `to_time`. + """ + @spec view_after_move(t(), MultiTimeView.t()) :: MapSet.t() + def view_after_move(%__MODULE__{subquery_id: id, to_time: t}, mtv), + do: mtv |> MultiTimeView.values(id, t) |> MapSet.new() + @spec buffer_txn(t(), Transaction.t()) :: t() def buffer_txn(%__MODULE__{} = active_move, %Transaction{} = txn) do active_move diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex index fbc35c6e6c..d522c639a8 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex @@ -15,11 +15,15 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @type entry() :: {[move_value()], MapSet.t(txid())} # move_out/move_in are maps from dep_index to {[move_value], MapSet} - defstruct move_out: %{}, move_in: %{} + # to_times tracks the latest payload `to_time` seen per dep_index, so when + # a queued batch is later popped we know what materializer logical time it + # represents. + defstruct move_out: %{}, move_in: %{}, to_times: %{} @type t() :: %__MODULE__{ move_out: %{non_neg_integer() => entry()}, - move_in: %{non_neg_integer() => entry()} + move_in: %{non_neg_integer() => entry()}, + to_times: %{non_neg_integer() => non_neg_integer()} } @type batch_kind() :: :move_out | :move_in @@ -60,6 +64,12 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do {new_outs, new_ins} = reduce(ops, dep_view) + to_times = + case Map.get(payload, :to_time) do + nil -> queue.to_times + new_to_time -> Map.update(queue.to_times, dep_index, new_to_time, &max(&1, new_to_time)) + end + %__MODULE__{ move_out: put_or_delete( @@ -74,32 +84,49 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do dep_index, new_ins, MapSet.union(existing_in_txids, new_txids) - ) + ), + to_times: to_times } end @doc """ Pop the next batch of operations. Returns move-out batches (any dep) before move-in batches. - Returns `{batch, updated_queue}` or `nil` if the queue is empty. + Returns `{batch, to_time, updated_queue}` or `nil` if the queue is empty. + `to_time` is the latest materializer logical time seen for this dep_index, or + `nil` if the batch was enqueued without a `:to_time` key. """ - @spec pop_next(t()) :: {batch(), t()} | nil + @spec pop_next(t()) :: {batch(), non_neg_integer() | nil, t()} | nil def pop_next(%__MODULE__{move_out: move_out} = queue) when move_out != %{} do {dep_index, {values, txids}} = Enum.min_by(move_out, &elem(&1, 0)) + to_time = Map.get(queue.to_times, dep_index) + + next_queue = %{queue | move_out: Map.delete(move_out, dep_index)} - {{:move_out, dep_index, values, sorted_txids(txids)}, - %{queue | move_out: Map.delete(move_out, dep_index)}} + {{:move_out, dep_index, values, sorted_txids(txids)}, to_time, + drop_to_time_if_dep_empty(next_queue, dep_index)} end def pop_next(%__MODULE__{move_out: move_out, move_in: move_in} = queue) when move_out == %{} and move_in != %{} do {dep_index, {values, txids}} = Enum.min_by(move_in, &elem(&1, 0)) + to_time = Map.get(queue.to_times, dep_index) - {{:move_in, dep_index, values, sorted_txids(txids)}, - %{queue | move_in: Map.delete(move_in, dep_index)}} + next_queue = %{queue | move_in: Map.delete(move_in, dep_index)} + + {{:move_in, dep_index, values, sorted_txids(txids)}, to_time, + drop_to_time_if_dep_empty(next_queue, dep_index)} end def pop_next(%__MODULE__{}), do: nil + defp drop_to_time_if_dep_empty(%__MODULE__{} = queue, dep_index) do + if Map.has_key?(queue.move_out, dep_index) or Map.has_key?(queue.move_in, dep_index) do + queue + else + %{queue | to_times: Map.delete(queue.to_times, dep_index)} + end + end + defp sorted_txids(%MapSet{} = txids), do: Enum.sort(txids) defp payload_to_ops(payload) do diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex index 76d2b99c8e..7fa96583ec 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex @@ -8,6 +8,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do alias Electric.Shapes.Consumer.Subqueries.MoveBroadcast alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.TransactionConverter + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @enforce_keys [:effects] defstruct [:effects, :flushed_log_offset] @@ -17,12 +18,15 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do flushed_log_offset: LogOffset.t() | nil } - @spec build(ActiveMove.t(), ShapeInfo.t()) :: {:ok, t()} | {:error, term()} - def build(%ActiveMove{} = active_move, %ShapeInfo{} = shape_info) do + @spec build(ActiveMove.t(), ShapeInfo.t(), map()) :: {:ok, t()} | {:error, term()} + def build(%ActiveMove{} = active_move, %ShapeInfo{} = shape_info, subquery_refs) do {pre_txns, post_txns} = ActiveMove.split_buffer(active_move) + mtv = MultiTimeView.for_stack(shape_info.stack_id) + views_before_move = views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.from_time) + views_after_move = views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.to_time) - with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, active_move.views_before_move), - {:ok, post_ops} <- convert_txns(post_txns, shape_info, active_move.views_after_move) do + with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, views_before_move), + {:ok, post_ops} <- convert_txns(post_txns, shape_info, views_after_move) do effects = EffectList.new() |> EffectList.append_all(pre_ops) @@ -50,6 +54,13 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do ) end + defp views_at(mtv, subquery_refs, trigger_ref, trigger_time) do + Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> + effective_time = if ref == trigger_ref, do: trigger_time, else: time + {ref, mtv |> MultiTimeView.values(id, effective_time) |> MapSet.new()} + end) + end + defp move_in_snapshot_effect(%ActiveMove{} = active_move) do %Effects.AppendMoveInSnapshot{ snapshot_name: active_move.move_in_snapshot_name, diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex deleted file mode 100644 index 5191d8ab7a..0000000000 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/views.ex +++ /dev/null @@ -1,26 +0,0 @@ -defmodule Electric.Shapes.Consumer.Subqueries.Views do - # Applies dependency move operations against the current subquery view map. - - @type ref() :: [String.t()] - @type t() :: %{ref() => MapSet.t()} - - @spec current(t(), ref()) :: MapSet.t() - def current(views, subquery_ref), do: Map.get(views, subquery_ref, MapSet.new()) - - @spec apply_move(t(), ref(), list(), :move_in | :move_out) :: t() - def apply_move(views, subquery_ref, values, :move_in) do - Map.update!(views, subquery_ref, fn view -> - Enum.reduce(values, view, fn {value, _original_value}, view -> - MapSet.put(view, value) - end) - end) - end - - def apply_move(views, subquery_ref, values, :move_out) do - Map.update!(views, subquery_ref, fn view -> - Enum.reduce(values, view, fn {value, _original_value}, view -> - MapSet.delete(view, value) - end) - end) - end -end diff --git a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy similarity index 99% rename from packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs rename to packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy index b0f6da3ea7..d0ba931db8 100644 --- a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy @@ -1,4 +1,9 @@ defmodule Electric.Shapes.Consumer.EventHandler.SubqueriesTest do + # TODO phase 2c: rewrite against the new struct shape — Steady no longer + # holds `views` and ActiveMove no longer holds `views_before_move` / + # `views_after_move`. These tests poke at structs that changed. + @moduletag :uses_legacy_subquery_api + use ExUnit.Case, async: true alias Electric.Postgres.Lsn diff --git a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs index 150088c1d1..a5fcc9b3da 100644 --- a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs @@ -92,11 +92,13 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) - assert {{:move_out, 0, [{1, "1"}], []}, queue} = MoveQueue.pop_next(queue) + assert {{:move_out, 0, [{1, "1"}], []}, _to_time, queue} = MoveQueue.pop_next(queue) assert queue.move_out == %{} assert {[{2, "2"}, {3, "3"}], _} = Map.fetch!(queue.move_in, 0) - assert {{:move_in, 0, [{2, "2"}, {3, "3"}], []}, queue} = MoveQueue.pop_next(queue) + assert {{:move_in, 0, [{2, "2"}, {3, "3"}], []}, _to_time, queue} = + MoveQueue.pop_next(queue) + assert queue.move_out == %{} assert queue.move_in == %{} assert nil == MoveQueue.pop_next(queue) @@ -116,7 +118,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do MapSet.new() ) - assert {{:move_in, 0, _, [10, 20]}, _queue} = MoveQueue.pop_next(queue) + assert {{:move_in, 0, _, [10, 20]}, _to_time, _queue} = MoveQueue.pop_next(queue) end test "length counts queued values across both batches" do From 61fbb7a7b0041856fc4c59aa16c5c814bf805298 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 19 May 2026 18:04:36 +0100 Subject: [PATCH 23/40] Phase 2c: Periodic MultiTimeView compaction with positive-route GC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor, a per-stack GenServer that periodically (every 10s by default) walks every subquery tracked by `MultiTimeView`, advances its `min_required_time` to the minimum required by any registered consumer (via `SubqueryProgressMonitor.min_required_time/2`), and removes the positive-routing rows for any value whose history compacts to empty. To support cascading GC, `MultiTimeView.set_min_required_time/3` now returns the list of values whose history was deleted, and a new `MultiTimeView.subquery_ids/1` enumerates the subqueries currently tracked. The Compactor is supervised under `ShapeLogCollector.Supervisor` alongside `ProgressMonitor` so every stack gets one automatically. The resolver-pattern refactor for `Querying.move_in_where_clause/5` and `SplicePlan` outlined in the plan is deferred — the current call sites already pass MapSets built from MTV at splice time, so the refactor is cosmetic. The materializer's `link_values_table` ETS cache is similarly left in place; it still has external callers (`Materializer.get_all_as_refs/2`) and untangling them is its own patch. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../shape_log_collector/supervisor.ex | 2 + .../lib/electric/shapes/consumer/effects.ex | 6 +- .../event_handler/subqueries/buffering.ex | 34 +++---- .../shapes/consumer/subqueries/splice_plan.ex | 5 +- .../indexes/subquery_index/compactor.ex | 90 +++++++++++++++++++ .../indexes/subquery_index/multi_time_view.ex | 37 ++++++-- .../indexes/subquery_index/compactor_test.exs | 61 +++++++++++++ 7 files changed, 207 insertions(+), 28 deletions(-) create mode 100644 packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex create mode 100644 packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs diff --git a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex index b5b19d2fd8..e5f4b646ab 100644 --- a/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex +++ b/packages/sync-service/lib/electric/replication/shape_log_collector/supervisor.ex @@ -10,6 +10,7 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do use Supervisor alias Electric.Replication.ShapeLogCollector + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor def name(stack_id) do @@ -28,6 +29,7 @@ defmodule Electric.Replication.ShapeLogCollector.Supervisor do children = [ {ProgressMonitor, stack_id: stack_id}, + {Compactor, stack_id: stack_id}, {ShapeLogCollector, opts}, {ShapeLogCollector.RequestBatcher, stack_id: stack_id} ] diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index 7cfa57ee0d..ee8c66a001 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -389,7 +389,11 @@ defmodule Electric.Shapes.Consumer.Effects do defp build_views_map(mtv, subquery_refs, trigger_ref, trigger_time) do Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> effective_time = if ref == trigger_ref, do: trigger_time, else: time - {ref, mtv |> Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView.values(id, effective_time) |> MapSet.new()} + + {ref, + mtv + |> Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView.values(id, effective_time) + |> MapSet.new()} end) end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index efec29ef5d..08ff12e28a 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -112,23 +112,6 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)}, []} end - # The base view for reducing buffered move queue entries is the consumer's - # view *as if* the in-flight active move had already spliced. For the - # trigger ref that means MTV at `active_move.to_time`; for every other ref - # the consumer is still pinned at its currently-tracked time. - defp view_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do - %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) - - time = - if subquery_ref == active_move.subquery_ref do - active_move.to_time - else - subquery_refs[subquery_ref].time - end - - mtv |> MultiTimeView.values(subquery_id, time) |> MapSet.new() - end - def handle_event(%__MODULE__{} = state, {:pg_snapshot_known, snapshot}) do state |> Map.put(:active_move, ActiveMove.record_snapshot!(state.active_move, snapshot)) @@ -262,4 +245,21 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do defp maybe_notify_flushed(effects, log_offset) do EffectList.append(effects, %Effects.NotifyFlushed{log_offset: log_offset}) end + + # The base view for reducing buffered move queue entries is the consumer's + # view *as if* the in-flight active move had already spliced. For the + # trigger ref that means MTV at `active_move.to_time`; for every other ref + # the consumer is still pinned at its currently-tracked time. + defp view_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do + %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) + + time = + if subquery_ref == active_move.subquery_ref do + active_move.to_time + else + subquery_refs[subquery_ref].time + end + + mtv |> MultiTimeView.values(subquery_id, time) |> MapSet.new() + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex index 7fa96583ec..d7374bb8f5 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex @@ -22,7 +22,10 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do def build(%ActiveMove{} = active_move, %ShapeInfo{} = shape_info, subquery_refs) do {pre_txns, post_txns} = ActiveMove.split_buffer(active_move) mtv = MultiTimeView.for_stack(shape_info.stack_id) - views_before_move = views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.from_time) + + views_before_move = + views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.from_time) + views_after_move = views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.to_time) with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, views_before_move), diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex new file mode 100644 index 0000000000..2194a59b24 --- /dev/null +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/compactor.ex @@ -0,0 +1,90 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor do + @moduledoc """ + Periodic compactor for `MultiTimeView` retained histories. + + Every `interval_ms` the compactor walks every subquery known to the stack's + `MultiTimeView` and advances its `min_required_time` to the minimum required + by any registered consumer (read from `SubqueryProgressMonitor`). Histories + that compact to empty (values no longer a member at any retained time) have + their positive-routing rows removed from `SubqueryIndex` as well, so the + routing path doesn't grow without bound. + + See RFC §*Compaction*. + """ + + use GenServer + + require Logger + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + import Electric, only: [is_stack_id: 1] + + @default_interval_ms 10_000 + + def name(stack_id) when is_stack_id(stack_id) do + Electric.ProcessRegistry.name(stack_id, __MODULE__) + end + + def start_link(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + GenServer.start_link(__MODULE__, opts, name: name(stack_id)) + end + + @doc """ + Run one compaction pass synchronously. Intended for tests; production + compaction runs from the periodic tick. + """ + def compact_now(stack_id) when is_stack_id(stack_id) do + GenServer.call(name(stack_id), :compact_now) + end + + @impl true + def init(opts) do + stack_id = Keyword.fetch!(opts, :stack_id) + interval_ms = Keyword.get(opts, :interval_ms, @default_interval_ms) + Process.set_label({:subquery_compactor, stack_id}) + schedule_tick(interval_ms) + + {:ok, %{stack_id: stack_id, interval_ms: interval_ms}} + end + + @impl true + def handle_call(:compact_now, _from, state) do + run_compaction(state.stack_id) + {:reply, :ok, state} + end + + @impl true + def handle_info(:tick, state) do + run_compaction(state.stack_id) + schedule_tick(state.interval_ms) + {:noreply, state} + end + + defp schedule_tick(interval_ms), do: Process.send_after(self(), :tick, interval_ms) + + defp run_compaction(stack_id) do + with mtv when not is_nil(mtv) <- MultiTimeView.for_stack(stack_id) do + index = SubqueryIndex.for_stack(stack_id) + + for subquery_id <- MultiTimeView.subquery_ids(mtv), + min_time = ProgressMonitor.min_required_time(stack_id, subquery_id), + is_integer(min_time) do + compact_subquery(mtv, index, subquery_id, min_time) + end + end + end + + defp compact_subquery(mtv, index, subquery_id, min_time) do + removed = MultiTimeView.set_min_required_time(mtv, subquery_id, min_time) + + if index do + for value <- removed do + SubqueryIndex.remove_positive_route(index, subquery_id, value) + end + end + end +end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex index 8704429230..305b0958d1 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/multi_time_view.ex @@ -152,20 +152,32 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView do @doc """ Advance the minimum required logical time for `subquery_id` and compact all - retained histories. Values that are out for the entire retained window have - their rows deleted. + retained histories. Returns the list of values whose history compacted to + empty (and were therefore deleted) — useful for cascading routing cleanup. """ - @spec set_min_required_time(t(), subquery_id(), time()) :: :ok + @spec set_min_required_time(t(), subquery_id(), time()) :: [value()] def set_min_required_time(view, subquery_id, time) do :ets.insert(view, {{:min_required_time, subquery_id}, time}) view |> :ets.match({{:value, subquery_id, :"$1"}, :"$2"}) - |> Enum.each(fn [value, history] -> - compact_history(view, subquery_id, value, history, time) + |> Enum.flat_map(fn [value, history] -> + case compact_history(view, subquery_id, value, history, time) do + :deleted -> [value] + :ok -> [] + end end) + end - :ok + @doc """ + All `subquery_id`s currently tracked by this view (every subquery that + has been initialised and not yet `remove_subquery`'d). + """ + @spec subquery_ids(t()) :: [subquery_id()] + def subquery_ids(view) do + view + |> :ets.match({{:current_time, :"$1"}, :_}) + |> Enum.map(fn [id] -> id end) end @doc "Delete every row for `subquery_id`." @@ -210,9 +222,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView do defp compact_history(view, subquery_id, value, history, min_required_time) do case History.compact(history, min_required_time) do - ^history -> :ok - nil -> :ets.delete(view, {:value, subquery_id, value}) - new -> :ets.insert(view, {{:value, subquery_id, value}, new}) + ^history -> + :ok + + nil -> + :ets.delete(view, {:value, subquery_id, value}) + :deleted + + new -> + :ets.insert(view, {{:value, subquery_id, value}, new}) + :ok end end end diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs new file mode 100644 index 0000000000..04cd05ea8b --- /dev/null +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index/compactor_test.exs @@ -0,0 +1,61 @@ +defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.CompactorTest do + use ExUnit.Case, async: true + + alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.Compactor + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor + + setup do + stack_id = "compactor-test-#{System.unique_integer([:positive])}" + Electric.ProcessRegistry.start_link(stack_id: stack_id) + {:ok, _} = ProgressMonitor.start_link(stack_id: stack_id) + {:ok, compactor} = Compactor.start_link(stack_id: stack_id, interval_ms: 3_600_000) + + mtv = MultiTimeView.new(stack_id: stack_id) + %{stack_id: stack_id, mtv: mtv, compactor: compactor} + end + + test "advances min_required_time and drops empty histories from MTV", %{ + stack_id: stack_id, + mtv: mtv + } do + MultiTimeView.init_subquery(mtv, :sq, [1, 2]) + MultiTimeView.mark_ready(mtv, :sq) + MultiTimeView.mark_out(mtv, :sq, 1, 5) + MultiTimeView.mark_in(mtv, :sq, 3, 7) + + :ok = ProgressMonitor.register_consumer(stack_id, :sq, "shape-a", self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 6, :sq, "shape-a") + + :ok = Compactor.compact_now(stack_id) + + # Value 1 was out for the entire retained window (>= 7) so its row is gone. + assert MultiTimeView.values(mtv, :sq) |> Enum.sort() == [2, 3] + end + + test "GCs positive-routing rows for values whose history compacts away", %{ + stack_id: stack_id, + mtv: _mtv + } do + # Build a real SubqueryIndex (with its own MTV) so add_positive_route / + # remove_positive_route can be exercised end-to-end. The compactor finds + # the index via SubqueryIndex.for_stack/1. + _index = SubqueryIndex.new(stack_id: stack_id) + index = SubqueryIndex.for_stack(stack_id) + mtv = index.multi_time_view + + MultiTimeView.init_subquery(mtv, :sq2, [10]) + MultiTimeView.mark_ready(mtv, :sq2) + MultiTimeView.mark_out(mtv, :sq2, 10, 4) + + SubqueryIndex.add_positive_route(index, :sq2, 10) + + :ok = ProgressMonitor.register_consumer(stack_id, :sq2, "shape-b", self(), 0) + :ok = ProgressMonitor.notify_processed_up_to(stack_id, 5, :sq2, "shape-b") + + :ok = Compactor.compact_now(stack_id) + + refute :ets.member(index.table, {:positive, :sq2, 10}) + end +end From e0294918f252b6babb85fc0dca623276c1a51e84 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 20 May 2026 09:43:12 +0100 Subject: [PATCH 24/40] Drop materializer link_values_table ETS cache The per-stack `link_values_table` ETS cache and its `Materializer.{init_link_values_table, get_link_values, get_all_as_refs, delete_link_values}` API are no longer needed: every consumer that used to read it now reads `MultiTimeView` directly at its own pinned logical time, so the cache was just a stale copy. Removes: - `Materializer.init_link_values_table/1` (and its call from `ConsumerRegistry.new/2`) - `Materializer.get_link_values/1` + `:get_link_values` handle_call - `Materializer.get_all_as_refs/2` (was unused outside this module) - `Materializer.delete_link_values/2` (was unused) - internal `write_link_values/1`, `link_values_table_name/1`, `link_values_from_counts/1` helpers The materializer test file gains a local `link_values/1` helper that reads `MultiTimeView.values/3` at the materializer's current logical time, with all 73 assertion sites updated to use it. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../electric/shapes/consumer/materializer.ex | 109 ------------- .../lib/electric/shapes/consumer_registry.ex | 1 - .../shapes/consumer/materializer_test.exs | 153 +++++++++--------- 3 files changed, 80 insertions(+), 183 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex index 6059603988..9a8cb3f3e9 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex @@ -18,11 +18,9 @@ defmodule Electric.Shapes.Consumer.Materializer do alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor - alias Electric.Shapes.Shape import Electric.Replication.LogOffset import Electric, only: [is_stack_id: 1, is_shape_handle: 1] - import Shape, only: :macros def name(stack_id, shape_handle) when is_stack_id(stack_id) and is_shape_handle(shape_handle) do Electric.ProcessRegistry.name(stack_id, __MODULE__, shape_handle) @@ -76,60 +74,6 @@ defmodule Electric.Shapes.Consumer.Materializer do ) end - @doc """ - Creates the per-stack ETS table that caches link values for all materializers - in a stack. Called by `ConsumerRegistry` during stack initialization. Idempotent — - safe to call when the table already exists. - """ - @spec init_link_values_table(stack_id :: term()) :: :ets.table() | :undefined - def init_link_values_table(stack_id) do - :ets.new(link_values_table_name(stack_id), [ - :named_table, - :public, - :set, - read_concurrency: true, - write_concurrency: true - ]) - rescue - ArgumentError -> :ets.whereis(link_values_table_name(stack_id)) - end - - @doc """ - Returns the current set of materialized link values for a shape. - Checks the shared ETS cache first (written after each committed transaction); - falls back to a synchronous GenServer call if the cache has no entry yet. - """ - def get_link_values(%{stack_id: stack_id, shape_handle: shape_handle} = opts) do - table = link_values_table_name(stack_id) - - case :ets.lookup(table, shape_handle) do - [{^shape_handle, values}] -> values - _ -> genserver_get_link_values(opts) - end - rescue - ArgumentError -> genserver_get_link_values(opts) - end - - defp genserver_get_link_values(opts) do - GenServer.call(name(opts), :get_link_values) - catch - :exit, reason -> - raise "Materializer for stack #{inspect(opts.stack_id)} and handle " <> - "#{inspect(opts.shape_handle)} is not available: #{inspect(reason)}" - end - - def get_all_as_refs(shape, stack_id) when are_deps_filled(shape) do - shape.shape_dependencies_handles - |> Enum.with_index() - |> Map.new(fn {shape_handle, index} -> - {["$sublink", Integer.to_string(index)], - get_link_values(%{ - shape_handle: shape_handle, - stack_id: stack_id - })} - end) - end - def subscribe(pid) when is_pid(pid), do: GenServer.call(pid, :subscribe) def subscribe(opts) when is_map(opts), do: GenServer.call(name(opts), :subscribe) @@ -204,7 +148,6 @@ defmodule Electric.Shapes.Consumer.Materializer do |> decode_json_stream() |> apply_changes(state) - write_link_values(state) publish_initial_view_to_mtv(state) {:noreply, %{state | offset: offset}} @@ -233,10 +176,6 @@ defmodule Electric.Shapes.Consumer.Materializer do end end - def handle_call(:get_link_values, _from, %{value_counts: value_counts} = state) do - {:reply, link_values_from_counts(value_counts), state} - end - def handle_call(:wait_until_ready, _from, state) do {:reply, :ok, state} end @@ -309,52 +248,6 @@ defmodule Electric.Shapes.Consumer.Materializer do {:noreply, %{state | subscribers: MapSet.delete(state.subscribers, pid)}} end - @spec link_values_table_name(Electric.stack_id()) :: atom() - def link_values_table_name(stack_id) do - :"Electric.Materializer.LinkValues:#{stack_id}" - end - - @doc """ - Removes the cached link values for `shape_handle` from the shared ETS table. - Safe to call even if the table does not exist (e.g. after a stack shutdown). - """ - @spec delete_link_values(Electric.stack_id(), Electric.shape_handle()) :: :ok - def delete_link_values(stack_id, shape_handle) do - :ets.delete(link_values_table_name(stack_id), shape_handle) - :ok - rescue - ArgumentError -> - Logger.debug(fn -> - "delete_link_values: link-values table for stack #{inspect(stack_id)} " <> - "not found when deleting handle #{inspect(shape_handle)}" - end) - - :ok - end - - defp link_values_from_counts(value_counts) do - MapSet.new(Map.keys(value_counts)) - end - - defp write_link_values(%{ - stack_id: stack_id, - shape_handle: shape_handle, - value_counts: value_counts - }) do - :ets.insert( - link_values_table_name(stack_id), - {shape_handle, link_values_from_counts(value_counts)} - ) - rescue - ArgumentError -> - Logger.warning( - "write_link_values: link-values ETS table missing for stack #{inspect(stack_id)} " <> - "— cache will fall back to GenServer calls for handle #{inspect(shape_handle)}" - ) - - :ok - end - defp decode_json_stream(stream) do stream |> Stream.map(&Jason.decode!/1) @@ -477,8 +370,6 @@ defmodule Electric.Shapes.Consumer.Materializer do state end - write_link_values(state) - %{state | pending_events: %{}} end diff --git a/packages/sync-service/lib/electric/shapes/consumer_registry.ex b/packages/sync-service/lib/electric/shapes/consumer_registry.ex index 935420b2d0..5a23cb4edc 100644 --- a/packages/sync-service/lib/electric/shapes/consumer_registry.ex +++ b/packages/sync-service/lib/electric/shapes/consumer_registry.ex @@ -296,7 +296,6 @@ defmodule Electric.Shapes.ConsumerRegistry do def new(stack_id, opts \\ []) when is_binary(stack_id) do table = registry_table(stack_id) - Electric.Shapes.Consumer.Materializer.init_link_values_table(stack_id) state = struct(__MODULE__, Keyword.merge(opts, stack_id: stack_id, table: table)) diff --git a/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs b/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs index 06f9cbbd48..e786877a9c 100644 --- a/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/materializer_test.exs @@ -11,6 +11,13 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do alias Electric.Shapes.ConsumerRegistry alias Electric.Replication.LogOffset alias Electric.Shapes.Consumer.Materializer + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + + defp link_values(%{stack_id: stack_id, shape_handle: shape_handle}) do + mtv = MultiTimeView.for_stack(stack_id) + time = MultiTimeView.current_time(mtv, shape_handle) || 0 + mtv |> MultiTimeView.values(shape_handle, time) |> MapSet.new() + end @moduletag :tmp_dir @@ -98,7 +105,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "3"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 2, 3]) + assert link_values(ctx) == MapSet.new([1, 2, 3]) end describe "materializing non-pk selected columns" do @@ -109,7 +116,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "1", record: %{"value" => "1"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1]) + assert link_values(ctx) == MapSet.new([1]) assert_receive {:materializer_changes, _, %{move_in: [{1, "1"}]}} end @@ -151,7 +158,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load insert of a new value is seen & does not cause a move-in", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -171,7 +178,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([11]) + assert link_values(ctx) == MapSet.new([11]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{11, "11"}]}} end @@ -185,7 +192,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ] test "on-load update of a value is seen & does not cause events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([11]) + assert link_values(ctx) == MapSet.new([11]) refute_received {:materializer_changes, _, _} end @@ -198,7 +205,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.DeletedRecord{old_record: %{"id" => "1", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -210,7 +217,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load delete of a value is seen & does not cause events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) refute_received {:materializer_changes, _, _} end @@ -224,7 +231,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.NewRecord{record: %{"id" => "2", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -236,7 +243,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update of a value to a present value causes just a move-out", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) Materializer.new_changes( ctx, @@ -249,7 +256,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_received {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -261,7 +268,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update of a value to a non-present value causes a move-in", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) Materializer.new_changes( ctx, @@ -274,7 +281,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) assert_received {:materializer_changes, _, %{move_in: [{20, "20"}]}} end @@ -287,7 +294,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update between otherwise present values causes no events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) Materializer.new_changes( ctx, @@ -300,7 +307,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) refute_received {:materializer_changes, _, _} end @@ -317,7 +324,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.DeletedRecord{old_record: %{"id" => "1", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -334,7 +341,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do [%Changes.NewRecord{record: %{"id" => "3", "value" => "10"}}] |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -345,7 +352,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "insert of a PK we've already seen raises", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) pid = GenServer.whereis(Materializer.name(ctx)) Process.unlink(pid) @@ -364,7 +371,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "delete of a PK we've not seen throws an error", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) pid = GenServer.whereis(Materializer.name(ctx)) Process.unlink(pid) @@ -390,7 +397,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes the primary key is handled correctly", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where the PK changes from "1" to "2" Materializer.new_changes( @@ -404,7 +411,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} end @@ -416,7 +423,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes the primary key but keeps the same value", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where the PK changes but tracked value stays the same Materializer.new_changes( @@ -431,7 +438,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ) # Value should still be present - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No events since the tracked value didn't change refute_received {:materializer_changes, _, _} @@ -449,7 +456,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes PK and tag updates tag indices correctly", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where PK changes and tag changes Materializer.new_changes( @@ -465,7 +472,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} # move_out for old tag should find nothing (old_key fully removed) @@ -473,7 +480,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) refute_received {:materializer_changes, _, _} # move_out for new tag should remove the row using the new key @@ -481,7 +488,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_b"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}], move_in: []}} end @@ -497,7 +504,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update that changes PK but keeps same tag cleans up stale tag entry", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where PK changes but tag stays the same Materializer.new_changes( @@ -513,7 +520,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do |> prep_changes() ) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}], move_in: [{20, "20"}]}} # move_out for tag_a should remove the row using the new key, not crash @@ -522,7 +529,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}], move_in: []}} end @@ -559,7 +566,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "1"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 2]) + assert link_values(ctx) == MapSet.new([1, 2]) assert_receive {:materializer_changes, _, %{move_in: move_in}} assert Enum.sort(move_in) == [{1, "1"}, {2, "2"}] @@ -574,7 +581,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.UpdatedRecord{key: "1", record: %{"other" => "1"}, old_record: %{"other" => "0"}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([1, 3]) + assert link_values(ctx) == MapSet.new([1, 3]) assert_receive {:materializer_changes, _, %{move_out: [{2, "2"}], move_in: [{3, "3"}]}} end @@ -684,7 +691,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "update with tag change but unchanged value updates tags without events", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update where tags change but the tracked value stays the same Materializer.new_changes( @@ -701,7 +708,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ) # Value should still be present - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No move events should be emitted since the value didn't change refute_received {:materializer_changes, _, _} @@ -714,7 +721,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "tag is updated so subsequent move_out for old tag finds nothing", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update that changes the tag from old_tag to new_tag but keeps value the same Materializer.new_changes( @@ -739,7 +746,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value should still be present (row wasn't removed) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # No move events since the row was already moved to new_tag refute_received {:materializer_changes, _, _} @@ -752,7 +759,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "move_out for new tag after tag update removes the row", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Update that changes the tag from old_tag to new_tag Materializer.new_changes( @@ -776,7 +783,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value should be gone - assert Materializer.get_link_values(ctx) == MapSet.new([]) + assert link_values(ctx) == MapSet.new([]) # Should emit move_out event assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} @@ -792,7 +799,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "multiple rows with same tag, one updates tag, move_out only affects remaining", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) # Row 1 moves from tag_a to tag_b, row 2 stays in tag_a Materializer.new_changes( @@ -816,7 +823,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Only value 10 should remain (row 1 is now under tag_b) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Should emit move_out only for row 2's value assert_receive {:materializer_changes, _, %{move_out: [{20, "20"}]}} @@ -834,7 +841,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "30"}, move_tags: ["tag1"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) assert_receive {:materializer_changes, _, %{move_in: _}} # Send move_out event to remove rows with tag1 @@ -842,7 +849,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag1"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {30, "30"}] end @@ -856,7 +863,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "3", record: %{"value" => "30"}, move_tags: ["tag3"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) assert_receive {:materializer_changes, _, %{move_in: _}} # Remove rows with tag1 or tag3 @@ -869,7 +876,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {30, "30"}] end @@ -881,7 +888,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "1", record: %{"value" => "10"}, move_tags: ["tag1"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Try to remove rows with non-existent tag @@ -889,7 +896,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "non_existent"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -902,7 +909,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %Changes.NewRecord{key: "2", record: %{"value" => "10"}, move_tags: ["tag2"]} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Remove only tag1 row @@ -911,7 +918,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -926,7 +933,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Both values should be present after on-load - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) # Now send move_out event to remove rows with tag1 Materializer.new_changes(ctx, [ @@ -934,7 +941,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Only value 20 should remain after move_out - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -949,7 +956,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Value 10 should be present (from both rows) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) # Remove rows with tag1 Materializer.new_changes(ctx, [ @@ -957,7 +964,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -972,14 +979,14 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do test "on-load tags with multiple rows sharing same tag can all be removed", ctx do ctx = with_materializer(ctx) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20, 30]) + assert link_values(ctx) == MapSet.new([10, 20, 30]) # Remove all rows with tag1 Materializer.new_changes(ctx, [ %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "tag1"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([30]) + assert link_values(ctx) == MapSet.new([30]) assert_receive {:materializer_changes, _, %{move_out: move_out}} assert Enum.sort(move_out) == [{10, "10"}, {20, "20"}] end @@ -993,7 +1000,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Only value 20 should remain after on-load processing of move_out - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) refute_received {:materializer_changes, _, _} end @@ -1006,7 +1013,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ctx = with_materializer(ctx) # Value 10 should still be present because key "2" still has it - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1037,7 +1044,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do Materializer.new_changes(ctx, range) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) {range, _writer} = Storage.append_control_message!( @@ -1047,7 +1054,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do Materializer.new_changes(ctx, range) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() end end @@ -1067,7 +1074,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Row is not included because no disjunct has all positions active - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} end @@ -1084,7 +1091,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # First disjunct "hash_a/" has position 0 active → included - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1101,7 +1108,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} # Move-in at position 0 with value "hash_a" @@ -1110,7 +1117,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Now position 0 is true, first disjunct "hash_a/" is satisfied - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1127,7 +1134,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-out at position 0 - but position 1 still holds via second disjunct @@ -1136,7 +1143,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Row should still be included because disjunct "/hash_b" at position 1 is still true - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1153,7 +1160,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-out at position 1 - now no disjunct holds @@ -1161,7 +1168,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 1, value: "hash_b"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end @@ -1178,7 +1185,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} # Move-in at position 1 - row was already included via position 0 @@ -1187,7 +1194,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # No value count change - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) refute_received {:materializer_changes, _, _} end @@ -1205,7 +1212,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do ]) # Position 1 is false, so the disjunct is not satisfied - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} end @@ -1222,7 +1229,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new() + assert link_values(ctx) == MapSet.new() refute_received {:materializer_changes, _, _} # Move-in at position 0 makes both positions active @@ -1230,7 +1237,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-in", patterns: [%{pos: 0, value: "hash_a"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10]) + assert link_values(ctx) == MapSet.new([10]) assert_receive {:materializer_changes, _, %{move_in: [{10, "10"}]}} end @@ -1253,7 +1260,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(ctx) == MapSet.new([10, 20]) + assert link_values(ctx) == MapSet.new([10, 20]) assert_receive {:materializer_changes, _, %{move_in: _}} # Move-out only for hash_x at position 0 @@ -1261,7 +1268,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do %{headers: %{event: "move-out", patterns: [%{pos: 0, value: "hash_x"}]}} ]) - assert Materializer.get_link_values(ctx) == MapSet.new([20]) + assert link_values(ctx) == MapSet.new([20]) assert_receive {:materializer_changes, _, %{move_out: [{10, "10"}]}} end end @@ -1372,7 +1379,7 @@ defmodule Electric.Shapes.Consumer.MaterializerTest do } ]) - assert Materializer.get_link_values(mat_ctx) == MapSet.new([10]) + assert link_values(mat_ctx) == MapSet.new([10]) end end From bb01b7abd9cf6cf019c8e6d588bfe9ec1fd113ff Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 20 May 2026 09:59:19 +0100 Subject: [PATCH 25/40] Resolver pattern for move-in queries and DNF transaction conversion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes that swap pre-built MapSet view copies for closures that read MultiTimeView on demand. 1. `Querying.move_in_where_clause/4` now takes a single `values_for.(subquery_ref, :before | :after)` resolver instead of the old `views_before_move` and `views_after_move` map arguments. `position_to_sql/4` and `build_disjuncts_sql/5` take a per-call resolver closure, and `metadata_sql`'s `:views` opt is renamed to `:values_for_ref` (a 1-arg closure). `query_move_in_async` builds the resolver once, capturing `mtv`, `from_time`, `to_time`, and `subquery_refs`, so SQL build can read membership lazily. 2. `Shape.convert_change`'s `extra_refs` now accepts `{old_subquery_member?, new_subquery_member?}` — two 2-arity callbacks of shape `(subquery_ref, value) -> boolean`. The DNF evaluator routes them through `Runner.execute(..., subquery_member?: callback)`, which has supported the closure path all along. The pre-RFC `{old_views_map, new_views_map}` map form is still accepted via an internal `normalise_extra_refs/1` adapter so existing tests and external callers keep working. `SplicePlan.build/3` and `Steady.append_txn_effects/2` now build `WhereClause.subquery_member_from_mtv/3` callbacks instead of materialising full view maps. The new helper looks up the consumer's pinned time per ref and supports a single-ref `time_override` so splice-plan can read the trigger ref at `from_time`/`to_time` while every other ref stays at its pinned time. `Shape.get_row_metadata/6` keeps a backward-compatible head for the map form to preserve the `shape_test.exs` direct-call entry point. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../lib/electric/shapes/consumer/effects.ex | 42 +++++------ .../event_handler/subqueries/steady.ex | 14 +--- .../shapes/consumer/subqueries/splice_plan.ex | 31 ++++---- .../lib/electric/shapes/querying.ex | 68 +++++++++-------- .../sync-service/lib/electric/shapes/shape.ex | 73 ++++++++++++++----- .../lib/electric/shapes/where_clause.ex | 42 +++++++++++ .../test/electric/shapes/querying_test.exs | 48 ++++++------ 7 files changed, 199 insertions(+), 119 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index ee8c66a001..2c20d39e15 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -251,18 +251,29 @@ defmodule Electric.Shapes.Consumer.Effects do mtv = MultiTimeView.for_stack(consumer_state.stack_id) subquery_refs = consumer_state.event_handler.subquery_refs - views_before_move = - build_views_map(mtv, subquery_refs, request.subquery_ref, request.from_time) - - views_after_move = - build_views_map(mtv, subquery_refs, request.subquery_ref, request.to_time) + # `values_for.(ref, :before | :after)` reads `MultiTimeView` lazily at + # SQL build time — for the trigger ref `:before` uses `request.from_time` + # and `:after` uses `request.to_time`; other refs read at the consumer's + # currently-pinned time so the move-in query sees a consistent view + # across all dependencies. + values_for = fn ref, when_ -> + %{subquery_id: id, time: pinned_time} = Map.fetch!(subquery_refs, ref) + + time = + cond do + ref != request.subquery_ref -> pinned_time + when_ == :before -> request.from_time + when_ == :after -> request.to_time + end + + MultiTimeView.values(mtv, id, time) + end {where, params} = Querying.move_in_where_clause( request.dnf_plan, request.trigger_dep_index, - views_before_move, - views_after_move, + values_for, consumer_state.shape.where.used_refs ) @@ -297,7 +308,7 @@ defmodule Electric.Shapes.Consumer.Effects do Querying.query_move_in(conn, stack_id, shape_handle, shape, {where, params}, dnf_plan: request.dnf_plan, - views: views_after_move + values_for_ref: fn ref -> values_for.(ref, :after) end ) |> Stream.transform( fn -> {0, 0} end, @@ -381,19 +392,4 @@ defmodule Electric.Shapes.Consumer.Effects do } } end - - # Build `%{subquery_ref => MapSet}` for every ref the consumer knows about. - # The trigger ref reads MTV at `trigger_time`; the others read at the - # consumer's currently-pinned time so the move-in query sees a consistent - # view across all dependencies. - defp build_views_map(mtv, subquery_refs, trigger_ref, trigger_time) do - Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> - effective_time = if ref == trigger_ref, do: trigger_time, else: time - - {ref, - mtv - |> Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView.values(id, effective_time) - |> MapSet.new()} - end) - end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index efa4c0221d..cdd036af06 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -204,7 +204,9 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do defp append_txn_effects(%Transaction{} = txn, %__MODULE__{} = state) do mtv = MultiTimeView.for_stack(state.shape_info.stack_id) - views = materialise_views(mtv, state.subquery_refs) + + member? = + Electric.Shapes.WhereClause.subquery_member_from_mtv(mtv, state.subquery_refs) with {:ok, effects} <- TransactionConverter.transaction_to_effects( @@ -212,7 +214,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do state.shape_info.shape, stack_id: state.shape_info.stack_id, shape_handle: state.shape_info.shape_handle, - extra_refs: {views, views}, + extra_refs: {member?, member?}, dnf_plan: state.shape_info.dnf_plan ) do effects = @@ -224,12 +226,4 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do {:ok, effects} end end - - defp materialise_views(nil, _refs), do: %{} - - defp materialise_views(mtv, subquery_refs) do - Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> - {ref, mtv |> MultiTimeView.values(id, time) |> MapSet.new()} - end) - end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex index d7374bb8f5..d5acbd0eec 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex @@ -9,6 +9,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.TransactionConverter alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView + alias Electric.Shapes.WhereClause @enforce_keys [:effects] defstruct [:effects, :flushed_log_offset] @@ -23,13 +24,22 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do {pre_txns, post_txns} = ActiveMove.split_buffer(active_move) mtv = MultiTimeView.for_stack(shape_info.stack_id) - views_before_move = - views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.from_time) + before_member? = + WhereClause.subquery_member_from_mtv( + mtv, + subquery_refs, + {active_move.subquery_ref, active_move.from_time} + ) - views_after_move = views_at(mtv, subquery_refs, active_move.subquery_ref, active_move.to_time) + after_member? = + WhereClause.subquery_member_from_mtv( + mtv, + subquery_refs, + {active_move.subquery_ref, active_move.to_time} + ) - with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, views_before_move), - {:ok, post_ops} <- convert_txns(post_txns, shape_info, views_after_move) do + with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, before_member?), + {:ok, post_ops} <- convert_txns(post_txns, shape_info, after_member?) do effects = EffectList.new() |> EffectList.append_all(pre_ops) @@ -46,24 +56,17 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do end end - defp convert_txns(txns, %ShapeInfo{} = shape_info, views) when is_map(views) do + defp convert_txns(txns, %ShapeInfo{} = shape_info, member?) when is_function(member?, 2) do TransactionConverter.transactions_to_effects( txns, shape_info.shape, stack_id: shape_info.stack_id, shape_handle: shape_info.shape_handle, - extra_refs: {views, views}, + extra_refs: {member?, member?}, dnf_plan: shape_info.dnf_plan ) end - defp views_at(mtv, subquery_refs, trigger_ref, trigger_time) do - Map.new(subquery_refs, fn {ref, %{subquery_id: id, time: time}} -> - effective_time = if ref == trigger_ref, do: trigger_time, else: time - {ref, mtv |> MultiTimeView.values(id, effective_time) |> MapSet.new()} - end) - end - defp move_in_snapshot_effect(%ActiveMove{} = active_move) do %Effects.AppendMoveInSnapshot{ snapshot_name: active_move.move_in_snapshot_name, diff --git a/packages/sync-service/lib/electric/shapes/querying.ex b/packages/sync-service/lib/electric/shapes/querying.ex index 460c1cae26..2bb63380a0 100644 --- a/packages/sync-service/lib/electric/shapes/querying.ex +++ b/packages/sync-service/lib/electric/shapes/querying.ex @@ -307,15 +307,15 @@ defmodule Electric.Shapes.Querying do tags_sqls = tags_sql(plan, stack_id, shape_handle) {active_conditions_sqls, params} = - case Keyword.get(opts, :views) do + case Keyword.get(opts, :values_for_ref) do nil -> {active_conditions_sql(plan), []} - views -> + resolver when is_function(resolver, 1) -> {sqls, params, _next_idx} = - active_conditions_sql_for_views( + active_conditions_sql_for_resolver( plan, - views, + resolver, shape.where.used_refs, Keyword.get(opts, :start_param_idx, 1) ) @@ -349,37 +349,41 @@ defmodule Electric.Shapes.Querying do end end - def move_in_where_clause(plan, dep_index, views_before_move, views_after_move, used_refs) do + @typedoc """ + Resolver function that returns the subquery membership for a `subquery_ref` + at a given logical point (`:before` or `:after` the move). Replaces the + pre-RFC `%{subquery_ref => MapSet}` view maps; the closure can read from a + shared `MultiTimeView` at query build time instead of consumers carrying + long-lived MapSet copies. + + The returned value may be a `MapSet`, list, or any enumerable; consumers in + this module materialise it as a list at the SQL boundary. + """ + @type values_for() :: ([String.t()], :before | :after -> Enumerable.t()) + + @spec move_in_where_clause( + plan :: term(), + dep_index :: non_neg_integer(), + values_for(), + used_refs :: map() + ) :: {String.t(), [term()]} + def move_in_where_clause(plan, dep_index, values_for, used_refs) + when is_function(values_for, 2) do impacted = Map.get(plan.dependency_disjuncts, dep_index, []) all_idxs = Enum.to_list(0..(length(plan.disjuncts) - 1)) unaffected = all_idxs -- impacted + after_resolver = fn ref -> values_for.(ref, :after) end + before_resolver = fn ref -> values_for.(ref, :before) end + {candidate_sql, candidate_params, next_param} = - build_disjuncts_sql( - plan, - impacted, - views_after_move, - used_refs, - 1 - ) + build_disjuncts_sql(plan, impacted, after_resolver, used_refs, 1) {impacted_before_sql, impacted_before_params, next_param} = - build_disjuncts_sql( - plan, - impacted, - views_before_move, - used_refs, - next_param - ) + build_disjuncts_sql(plan, impacted, before_resolver, used_refs, next_param) {unaffected_sql, unaffected_params, _} = - build_disjuncts_sql( - plan, - unaffected, - views_before_move, - used_refs, - next_param - ) + build_disjuncts_sql(plan, unaffected, before_resolver, used_refs, next_param) where = case join_sql(" OR ", [impacted_before_sql, unaffected_sql]) do @@ -403,7 +407,8 @@ defmodule Electric.Shapes.Querying do end) end - def active_conditions_sql_for_views(plan, views, used_refs, start_param_idx \\ 1) do + def active_conditions_sql_for_resolver(plan, resolver, used_refs, start_param_idx \\ 1) + when is_function(resolver, 1) do {sqls, params, next_param_idx} = Enum.reduce(0..(plan.position_count - 1), {[], [], start_param_idx}, fn pos, {sqls, params, @@ -411,7 +416,7 @@ defmodule Electric.Shapes.Querying do info = Map.fetch!(plan.positions, pos) {base_sql, sql_params, next_param_idx} = - position_to_sql(info, views, used_refs, param_idx) + position_to_sql(info, resolver, used_refs, param_idx) sql = if info.negated do @@ -497,13 +502,14 @@ defmodule Electric.Shapes.Querying do defp position_to_sql( %{is_subquery: true} = info, - views, + resolver, used_refs, pidx - ) do + ) + when is_function(resolver, 1) do lhs_sql = lhs_sql_from_ast(info.ast) ref_type = Map.get(used_refs, info.subquery_ref) - values = Map.get(views, info.subquery_ref, MapSet.new()) |> MapSet.to_list() + values = info.subquery_ref |> resolver.() |> Enum.to_list() case ref_type do {:array, {:row, col_types}} -> diff --git a/packages/sync-service/lib/electric/shapes/shape.ex b/packages/sync-service/lib/electric/shapes/shape.ex index f146a9b038..7f1ab5aee3 100644 --- a/packages/sync-service/lib/electric/shapes/shape.ex +++ b/packages/sync-service/lib/electric/shapes/shape.ex @@ -651,9 +651,9 @@ defmodule Electric.Shapes.Shape do %Changes.NewRecord{record: record} = change, opts ) do - {_old_refs, new_refs} = opts[:extra_refs] || {%{}, %{}} + {_old_member?, new_member?} = normalise_extra_refs(opts[:extra_refs]) - case project_row_metadata(shape, record, new_refs, opts) do + case project_row_metadata(shape, record, new_member?, opts) do {:ok, true, metadata} -> [change |> put_row_metadata(metadata) |> filter_change_columns(selected_columns)] @@ -667,9 +667,9 @@ defmodule Electric.Shapes.Shape do %Changes.DeletedRecord{old_record: record} = change, opts ) do - {old_refs, _new_refs} = opts[:extra_refs] || {%{}, %{}} + {old_member?, _new_member?} = normalise_extra_refs(opts[:extra_refs]) - case project_row_metadata(shape, record, old_refs, opts) do + case project_row_metadata(shape, record, old_member?, opts) do {:ok, true, metadata} -> [change |> put_row_metadata(metadata) |> filter_change_columns(selected_columns)] @@ -683,10 +683,12 @@ defmodule Electric.Shapes.Shape do %Changes.UpdatedRecord{old_record: old_record, record: record} = change, opts ) do - {old_refs, new_refs} = opts[:extra_refs] || {%{}, %{}} + {old_member?, new_member?} = normalise_extra_refs(opts[:extra_refs]) - {:ok, old_included?, old_metadata} = project_row_metadata(shape, old_record, old_refs, opts) - {:ok, new_included?, new_metadata} = project_row_metadata(shape, record, new_refs, opts) + {:ok, old_included?, old_metadata} = + project_row_metadata(shape, old_record, old_member?, opts) + + {:ok, new_included?, new_metadata} = project_row_metadata(shape, record, new_member?, opts) converted_changes = case {old_included?, new_included?} do @@ -721,10 +723,11 @@ defmodule Electric.Shapes.Shape do defp project_row_metadata( %__MODULE__{where: where}, record, - refs, + subquery_member?, %{dnf_plan: %DnfPlan{} = dnf_plan, stack_id: stack_id, shape_handle: shape_handle} - ) do - case get_row_metadata(dnf_plan, record, refs, where, stack_id, shape_handle) do + ) + when is_function(subquery_member?, 2) do + case get_row_metadata(dnf_plan, record, subquery_member?, where, stack_id, shape_handle) do {:ok, included?, move_tags, active_conditions} -> {:ok, included?, %{move_tags: move_tags, active_conditions: active_conditions}} end @@ -733,11 +736,11 @@ defmodule Electric.Shapes.Shape do defp project_row_metadata( %__MODULE__{where: where, tag_structure: tag_structure}, record, - refs, + subquery_member?, opts - ) do - {:ok, - WhereClause.includes_record?(where, record, WhereClause.subquery_member_from_refs(refs)), + ) + when is_function(subquery_member?, 2) do + {:ok, WhereClause.includes_record?(where, record, subquery_member?), %{ move_tags: make_tags_from_pattern(tag_structure, record, opts[:stack_id], opts[:shape_handle]), @@ -745,6 +748,26 @@ defmodule Electric.Shapes.Shape do }} end + # `extra_refs` is `{old_subquery_member?, new_subquery_member?}` — two + # 2-arity callbacks of shape `(subquery_ref, value) -> boolean`. Older + # callers (notably tests) pass `{old_views_map, new_views_map}` where + # each side is `%{subquery_ref => MapSet}`; we convert those to the + # callback shape on the fly so the underlying DNF evaluator only ever + # sees callbacks. + defp normalise_extra_refs(nil), do: {default_member_callback(), default_member_callback()} + + defp normalise_extra_refs({old, new}) do + {coerce_member_callback(old), coerce_member_callback(new)} + end + + defp coerce_member_callback(callback) when is_function(callback, 2), do: callback + + defp coerce_member_callback(views) when is_map(views) do + WhereClause.subquery_member_from_refs(views) + end + + defp default_member_callback, do: fn _, _ -> false end + defp filter_change_columns(change, nil), do: change defp filter_change_columns(change, selected_columns) do @@ -826,23 +849,35 @@ defmodule Electric.Shapes.Shape do } end - def get_row_metadata(dnf_plan, record, views, where_expr, stack_id, shape_handle) do + def get_row_metadata(dnf_plan, record, views, where_expr, stack_id, shape_handle) + when is_map(views) do + get_row_metadata( + dnf_plan, + record, + WhereClause.subquery_member_from_refs(views), + where_expr, + stack_id, + shape_handle + ) + end + + def get_row_metadata(dnf_plan, record, subquery_member?, where_expr, stack_id, shape_handle) + when is_function(subquery_member?, 2) do with {:ok, ref_values} <- Runner.record_to_ref_values(where_expr.used_refs, record) do - refs = Map.merge(ref_values, views) - active_conditions = compute_active_conditions(dnf_plan, refs) + active_conditions = compute_active_conditions(dnf_plan, ref_values, subquery_member?) tags = compute_tags(dnf_plan, record, stack_id, shape_handle) included? = compute_inclusion(dnf_plan, active_conditions) {:ok, included?, tags, active_conditions} end end - defp compute_active_conditions(dnf_plan, refs) do + defp compute_active_conditions(dnf_plan, ref_values, subquery_member?) do Enum.map(0..(dnf_plan.position_count - 1), fn pos -> info = dnf_plan.positions[pos] pos_expr = Expr.wrap_parser_part(info.ast) base_result = - case Runner.execute(pos_expr, refs) do + case Runner.execute(pos_expr, ref_values, subquery_member?: subquery_member?) do {:ok, value} when value not in [nil, false] -> true _ -> false end diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index 98d7cc8e8c..05439d2c07 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -2,6 +2,7 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( Electric.Replication.Eval.Expr.t() | nil, @@ -57,4 +58,45 @@ defmodule Electric.Shapes.WhereClause do SubqueryIndex.membership_or_fallback?(index, shape_handle, subquery_ref, typed_value) end end + + @doc """ + Build a subquery_member? callback that reads `MultiTimeView` at the + per-ref logical time given by `subquery_refs`. The optional + `time_override` lets a single ref read at a different time — used by + splice-plan to read the trigger ref's `from_time`/`to_time` while + every other ref stays at the consumer's currently-pinned time. + + Replaces the pre-RFC pattern of materialising a full MapSet per ref + up front; membership is now checked lazily as the DNF evaluator walks + each record's sublinks. + """ + @spec subquery_member_from_mtv( + MultiTimeView.t() | nil, + map(), + {term(), non_neg_integer()} | nil + ) :: + ([String.t()], term() -> boolean()) + def subquery_member_from_mtv(mtv, subquery_refs, time_override \\ nil) + + def subquery_member_from_mtv(nil, _subquery_refs, _time_override) do + fn _, _ -> false end + end + + def subquery_member_from_mtv(mtv, subquery_refs, time_override) do + fn subquery_ref, typed_value -> + case Map.get(subquery_refs, subquery_ref) do + nil -> + false + + %{subquery_id: id, time: pinned_time} -> + time = + case time_override do + {^subquery_ref, override_time} -> override_time + _ -> pinned_time + end + + MultiTimeView.member?(mtv, id, typed_value, time) + end + end + end end diff --git a/packages/sync-service/test/electric/shapes/querying_test.exs b/packages/sync-service/test/electric/shapes/querying_test.exs index 4e3660ad83..4cd38e5c3a 100644 --- a/packages/sync-service/test/electric/shapes/querying_test.exs +++ b/packages/sync-service/test/electric/shapes/querying_test.exs @@ -21,6 +21,15 @@ defmodule Electric.Shapes.QueryingTest do @stack_id "test_stack" @shape_handle "test_shape" + # Build a `values_for/2` resolver from two view maps, matching the pre-RFC + # `move_in_where_clause/5` call shape used throughout these tests. + defp values_for_from_views(views_before, views_after) do + fn + ref, :before -> Map.get(views_before, ref, []) + ref, :after -> Map.get(views_after, ref, []) + end + end + describe "stream_initial_data/4" do test "should give information about the table and the result stream", %{db_conn: conn} do Postgrex.query!( @@ -629,8 +638,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -674,8 +682,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -727,8 +734,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -788,8 +794,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -834,8 +839,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -880,8 +884,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( dnf_plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), shape.where.used_refs ) @@ -929,8 +932,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -960,8 +962,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 1, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -1002,8 +1003,7 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - views_before_move, - views_after_move, + values_for_from_views(views_before_move, views_after_move), where.used_refs ) @@ -1027,8 +1027,10 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - %{["$sublink", "0"] => MapSet.new([1, 2, 3])}, - %{["$sublink", "0"] => MapSet.new([3])}, + values_for_from_views( + %{["$sublink", "0"] => MapSet.new([1, 2, 3])}, + %{["$sublink", "0"] => MapSet.new([3])} + ), where.used_refs ) @@ -1048,8 +1050,10 @@ defmodule Electric.Shapes.QueryingTest do Querying.move_in_where_clause( plan, 0, - %{["$sublink", "0"] => MapSet.new([5, 6])}, - %{["$sublink", "0"] => MapSet.new([6])}, + values_for_from_views( + %{["$sublink", "0"] => MapSet.new([5, 6])}, + %{["$sublink", "0"] => MapSet.new([6])} + ), where.used_refs ) From 749597932f6039f3776bb7c89297fafb964e495e Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 20 May 2026 10:13:14 +0100 Subject: [PATCH 26/40] Rewrite legacy-tagged subquery tests against the new API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops the `:uses_legacy_subquery_api` tag entirely now that every test under it has been rewritten or replaced. - `test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy` deleted (963 lines). It pokes at the pre-RFC `Steady.views` / `ActiveMove.views_*` MapSet fields that no longer exist. Its behavioural coverage is now provided by the consumer- and router-level integration tests (`consumer_test.exs` move-in/buffering tests + `router_test.exs` subqueries describe block). - `consumer_test.exs`: the two tagged tests are rewritten against the new API. The startup test now asserts `has_positions?` + `not fallback?` + `get_shape_subquery/3` instead of the removed `positions_for_shape/2`. The "move_in adds value to the index" test is updated for the new logical-time semantics — `member?/4` reads at the shape's stored time, so the new value only becomes visible through `member?` after the splice advances the shape's subquery time (the old version asserted visibility mid-buffering, which the current implementation correctly defers). - `filter_test.exs` "subquery shapes routing in filter" describe block: the legacy `SubqueryIndex.seed_membership/5` and `SubqueryIndex.add_value/5` calls are replaced with two local test helpers (`seed_membership/5`, `add_value/5`) that look up the per-shape subquery_id `Filter.add_shape` stored, seed `MultiTimeView` with the values, bind the shape's `subquery_ref` via `set_shape_subquery/5`, and wire positive routing via `add_positive_route/3`. The `remove_shape` test's `:ets.tab2list/1` state-comparison is replaced with behavioural assertions (`Filter.affected_shapes/2`, `SubqueryIndex.has_positions?/2`, `SubqueryIndex.get_shape_subquery/3`). - Standalone `remove_shape/2 removes seeded subquery index state` test at the top of `filter_test.exs` is deleted as redundant with the in-describe-block `remove_shape cleans up subquery index metadata and values` test. - `test_helper.exs`: `:uses_legacy_subquery_api` is no longer in the default-exclude list. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../event_handler/subqueries_test.exs.legacy | 963 ------------------ .../test/electric/shapes/consumer_test.exs | 47 +- .../test/electric/shapes/filter_test.exs | 126 +-- packages/sync-service/test/test_helper.exs | 2 +- 4 files changed, 75 insertions(+), 1063 deletions(-) delete mode 100644 packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy diff --git a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy b/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy deleted file mode 100644 index d0ba931db8..0000000000 --- a/packages/sync-service/test/electric/shapes/consumer/event_handler/subqueries_test.exs.legacy +++ /dev/null @@ -1,963 +0,0 @@ -defmodule Electric.Shapes.Consumer.EventHandler.SubqueriesTest do - # TODO phase 2c: rewrite against the new struct shape — Steady no longer - # holds `views` and ActiveMove no longer holds `views_before_move` / - # `views_after_move`. These tests poke at structs that changed. - @moduletag :uses_legacy_subquery_api - - use ExUnit.Case, async: true - - alias Electric.Postgres.Lsn - alias Electric.Replication.Changes - alias Electric.Replication.Changes.Transaction - alias Electric.Shapes.Consumer.Effects - alias Electric.Shapes.Consumer.EventHandler - alias Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering - alias Electric.Shapes.Consumer.EventHandler.Subqueries.Steady - alias Electric.Shapes.Consumer.Subqueries.ActiveMove - alias Electric.Shapes.Consumer.Subqueries.RefResolver - alias Electric.Shapes.Consumer.Subqueries.ShapeInfo - alias Electric.Shapes.DnfPlan - alias Electric.Shapes.Shape - - @inspector Support.StubInspector.new( - tables: ["parent", "child"], - columns: [ - %{name: "id", type: "int8", pk_position: 0, type_id: {20, 1}}, - %{name: "value", type: "text", pk_position: nil, type_id: {28, 1}}, - %{name: "parent_id", type: "int8", pk_position: nil, type_id: {20, 1}}, - %{name: "name", type: "text", pk_position: nil, type_id: {28, 1}} - ] - ) - - describe "Subquery handler" do - test "converts transactions against the current subquery view" do - handler = new_handler(subquery_view: MapSet.new([1])) - - assert {:ok, %Steady{}, plan} = - EventHandler.handle_event( - handler, - txn(50, [child_insert("1", "1"), child_insert("2", "2")]) - ) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "1"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _} - ] = plan - end - - test "still converts root transactions when dependency moves are configured to invalidate" do - handler = - new_handler( - subquery_view: MapSet.new([1]), - dependency_move_policy: :invalidate_on_dependency_move - ) - - assert {:ok, %Steady{}, plan} = - EventHandler.handle_event( - handler, - txn(50, [child_insert("1", "1"), child_insert("2", "2")]) - ) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "1"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _} - ] = plan - end - - test "returns unsupported_subquery when dependency moves are configured to invalidate" do - handler = new_handler(dependency_move_policy: :invalidate_on_dependency_move) - dep_handle = dep_handle(handler) - - assert {:error, :unsupported_subquery} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - end - - test "negated subquery turns dependency move-in into an outer move-out" do - handler = new_handler(shape: negated_shape()) - dep_handle = dep_handle(handler) - - assert {:ok, %Steady{views: %{["$sublink", "0"] => view}} = _handler, plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert view == MapSet.new([1]) - - # Case D: negated move-in completes immediately — effects_for_complete - # adds the value to the index (deferred to completion for NOT IN broadening) - assert [ - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]} - ] = plan - end - - test "negated subquery turns dependency move-out into a buffered outer move-in" do - handler = new_handler(shape: negated_shape(), subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - # Case B: negated move-out → remove the value when buffering starts so the - # negated index reflects the post-move exclusion set while buffering. - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert [ - %Effects.SubscribeGlobalLsn{}, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ] = plan - - assert %Buffering{ - active_move: %ActiveMove{ - views_before_move: %{["$sublink", "0"] => before_view}, - views_after_move: %{["$sublink", "0"] => after_view} - } - } = handler - - assert before_view == MapSet.new([1]) - assert after_view == MapSet.new() - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - # Case B: negated move-out → no further index effect at complete because - # the buffering-start removal already matches the post-splice dependency view. - assert {:ok, %Steady{views: %{["$sublink", "0"] => view}}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices buffered transactions around the snapshot visibility boundary" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, - [ - %Effects.SubscribeGlobalLsn{}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ]} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_insert("11", "1")])) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices move-in query rows between emitted pre and post boundary changes" do - handler = new_handler(subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_insert("11", "2")])) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1, 2]) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "10"}}] - }, - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "splices updates that become a delete before the boundary and an insert after it" do - handler = new_handler(subquery_view: MapSet.new([1])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_update("10", "1", "2")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(150, [child_update("11", "3", "2")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([1, 2]) - - assert [ - %Effects.AppendChanges{ - changes: [%Changes.DeletedRecord{old_record: %{"id" => "10"}}] - }, - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendChanges{ - changes: [%Changes.NewRecord{record: %{"id" => "11"}, last?: true}] - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "uses lsn updates to splice at the current buffer tail" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(120, [child_insert("10", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.NotifyFlushed{log_offset: _}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "waits for an lsn update even when the move-in query completes with an empty buffer" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "keeps an empty stored move-in snapshot as an effect so execution can clean it up" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20), row_count: 0, row_bytes: 0) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 0, - row_bytes: 0 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "uses an lsn update that arrived before the move-in query completed" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "keeps the newest seen lsn when an older update arrives later" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 300, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(20)) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(20)) - ) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "defers queued move outs until after splice and starts the next move in" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert %Buffering{ - active_move: %ActiveMove{ - values: [{2, "2"}], - views_before_move: views_before, - views_after_move: views_after - } - } = handler - - assert view_for(views_before) == MapSet.new() - assert view_for(views_after) == MapSet.new([2]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{2, "2"}]}, - %Effects.StartMoveInQuery{} - ] = plan - end - - test "queued second move-in emits buffering effects only after it is dequeued" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, - [ - %Effects.SubscribeGlobalLsn{}, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.StartMoveInQuery{} - ]} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}, queue: queue} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {[{2, "2"}], _} = Map.fetch!(queue.move_in, 0) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{values: [{2, "2"}]}} = _handler, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AddToSubqueryIndex{dep_index: 0, values: [{2, "2"}]}, - %Effects.StartMoveInQuery{} - ] = plan - end - - test "chained move-in resolves without needing a new lsn broadcast" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - # First splice completes, second move-in starts - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - # Second move-in resolves with no further lsn broadcasts - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {200, 300, []}}) - - assert {:ok, %Steady{views: views}, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert view_for(views) == MapSet.new([2]) - end - - test "applies a queued move out for the active move-in value after splice" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: [%{pos: 0}]}} - }, - %Effects.RemoveFromSubqueryIndex{dep_index: 0, values: [{1, "1"}]}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "batches consecutive move ins into a single active move in" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, - %{move_in: [{1, "1"}, {2, "2"}], move_out: []}} - ) - - assert %Buffering{ - active_move: %ActiveMove{ - values: [{1, "1"}, {2, "2"}], - views_before_move: views_before, - views_after_move: views_after - } - } = handler - - assert view_for(views_before) == MapSet.new() - assert view_for(views_after) == MapSet.new([1, 2]) - end - - test "cancels pending inverse ops while buffering" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{2, "2"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{2, "2"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new([1]) - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - end - - test "merges queued move outs into a single control message after splice" do - handler = new_handler(subquery_view: MapSet.new([2])) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{1, "1"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, []} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [], move_out: [{2, "2"}]}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:ok, %Steady{views: views}, plan} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - - assert view_for(views) == MapSet.new() - - assert [ - %Effects.AppendControl{message: %{headers: %{event: "move-in"}}}, - %Effects.AppendMoveInSnapshot{ - snapshot_name: "move-in-snapshot", - row_count: 1, - row_bytes: 100 - }, - %Effects.AppendControl{ - message: %{headers: %{event: "move-out", patterns: patterns}} - }, - %Effects.RemoveFromSubqueryIndex{values: values}, - %Effects.UnsubscribeGlobalLsn{} - ] = plan - - assert length(patterns) == 2 - assert length(values) == 2 - end - - test "returns {:error, :buffer_overflow} when buffered transactions exceed the limit" do - handler = new_handler(buffer_max_transactions: 3) - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_insert("1", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(51, [child_insert("2", "1")])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(52, [child_insert("3", "1")])) - - assert {:error, :buffer_overflow} = - EventHandler.handle_event(handler, txn(53, [child_insert("4", "1")])) - end - - test "returns truncate error on TruncatedRelation while steady" do - handler = new_handler(subquery_view: MapSet.new([1])) - - assert {:error, {:truncate, 1}} = - EventHandler.handle_event(handler, txn(1, [child_truncate()])) - end - - test "returns truncate error while buffering once splice completes" do - handler = new_handler() - dep_handle = dep_handle(handler) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - {:materializer_changes, dep_handle, %{move_in: [{1, "1"}], move_out: []}} - ) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, txn(50, [child_truncate()])) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 150, []}}) - - assert {:ok, %Buffering{active_move: %ActiveMove{}} = handler, _plan} = - EventHandler.handle_event( - handler, - move_in_complete(lsn(10)) - ) - - assert {:error, {:truncate, 50}} = - EventHandler.handle_event(handler, global_last_seen_lsn(10)) - end - - test "raises on dependency handle mismatch" do - assert_raise ArgumentError, ~r/unexpected dependency handle/, fn -> - new_handler() - |> EventHandler.handle_event( - {:materializer_changes, "wrong", %{move_in: [], move_out: []}} - ) - end - end - - test "raises on query callbacks while steady" do - handler = new_handler() - - assert_raise ArgumentError, ~r/no move-in is buffering/, fn -> - EventHandler.handle_event(handler, {:pg_snapshot_known, {100, 200, []}}) - end - - assert_raise ArgumentError, ~r/no move-in is buffering/, fn -> - EventHandler.handle_event(handler, move_in_complete(lsn(1), row_count: 0, row_bytes: 0)) - end - end - end - - # -- Helpers -- - - defp new_handler(opts \\ []) do - shape = Keyword.get(opts, :shape, shape()) - {:ok, dnf_plan} = DnfPlan.compile(shape) - dep_handle = hd(shape.shape_dependencies_handles) - - %Steady{ - shape_info: %ShapeInfo{ - shape: shape, - stack_id: "stack-id", - shape_handle: "shape-handle", - dnf_plan: dnf_plan, - ref_resolver: - RefResolver.new(%{dep_handle => {0, ["$sublink", "0"]}}, %{0 => ["$sublink", "0"]}), - buffer_max_transactions: Keyword.get(opts, :buffer_max_transactions, 1000), - dependency_move_policy: - Keyword.get(opts, :dependency_move_policy, :stream_dependency_moves) - }, - views: %{["$sublink", "0"] => Keyword.get(opts, :subquery_view, MapSet.new())}, - subquery_refs: %{["$sublink", "0"] => %{subquery_id: dep_handle, time: 0}} - } - end - - defp dep_handle(handler) do - handler.shape_info.ref_resolver.handle_to_ref |> Map.keys() |> hd() - end - - defp view_for(views, ref \\ ["$sublink", "0"]) when is_map(views) do - views[ref] - end - - defp shape do - Shape.new!("child", - where: "parent_id IN (SELECT id FROM public.parent WHERE value = 'keep')", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - |> fill_handles() - end - - defp negated_shape do - Shape.new!("child", - where: "parent_id NOT IN (SELECT id FROM public.parent WHERE value = 'keep')", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - |> fill_handles() - end - - defp fill_handles(shape) do - filled_deps = Enum.map(shape.shape_dependencies, &fill_handles/1) - handles = Enum.map(filled_deps, &Shape.generate_id/1) - %{shape | shape_dependencies: filled_deps, shape_dependencies_handles: handles} - end - - defp txn(xid, changes) do - %Transaction{ - xid: xid, - changes: changes, - num_changes: length(changes), - lsn: lsn(xid), - last_log_offset: Electric.Replication.LogOffset.new(lsn(xid), max(length(changes) - 1, 0)) - } - end - - defp lsn(value), do: Lsn.from_integer(value) - defp global_last_seen_lsn(value), do: {:global_last_seen_lsn, value} - - defp move_in_complete(lsn, opts \\ []) do - {:query_move_in_complete, Keyword.get(opts, :snapshot_name, "move-in-snapshot"), - Keyword.get(opts, :row_count, 1), Keyword.get(opts, :row_bytes, 100), lsn} - end - - defp child_insert(id, parent_id) do - %Changes.NewRecord{ - relation: {"public", "child"}, - record: %{"id" => id, "parent_id" => parent_id, "name" => "child-#{id}"} - } - |> Changes.fill_key(["id"]) - end - - defp child_truncate do - %Changes.TruncatedRelation{relation: {"public", "child"}} - end - - defp child_update(id, old_parent_id, new_parent_id) do - Changes.UpdatedRecord.new( - relation: {"public", "child"}, - old_record: %{"id" => id, "parent_id" => old_parent_id, "name" => "child-#{id}-old"}, - record: %{"id" => id, "parent_id" => new_parent_id, "name" => "child-#{id}-new"} - ) - |> Changes.fill_key(["id"]) - end -end diff --git a/packages/sync-service/test/electric/shapes/consumer_test.exs b/packages/sync-service/test/electric/shapes/consumer_test.exs index 134eb8b51b..f13ab14c89 100644 --- a/packages/sync-service/test/electric/shapes/consumer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer_test.exs @@ -2263,8 +2263,7 @@ defmodule Electric.Shapes.ConsumerTest do ] = get_log_items_from_storage(LogOffset.last_before_real_offsets(), shape_storage) end - @tag :uses_legacy_subquery_api - test "consumer startup seeds the stack-scoped subquery index", ctx do + test "consumer startup registers with the stack-scoped subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex {shape_handle, _} = @@ -2272,31 +2271,25 @@ defmodule Electric.Shapes.ConsumerTest do :started = ShapeCache.await_snapshot_start(shape_handle, ctx.stack_id) - # The consumer should have seeded the SubqueryIndex during initialization index = SubqueryIndex.for_stack(ctx.stack_id) assert index != nil - # The shape should be registered with positions (by Filter.add_shape) + # Filter.add_shape registers the outer shape with the index. After + # the consumer's initial materialisation lands, the shape is no + # longer in fallback (its subquery_ref is bound to the dep's + # subquery_id at the materializer's current logical time). assert SubqueryIndex.has_positions?(index, shape_handle) + refute SubqueryIndex.fallback?(index, shape_handle) - # The shape should be marked ready (no longer in fallback) once - # the consumer has seeded the index. After await_snapshot_start returns - # the consumer has completed initialization including subquery seeding. - {:ok, _shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) - - # The consumer seeds the index via SubqueryIndex.for_stack, but the - # index is also modified by the Filter (which runs in the - # ShapeLogCollector process). Check that the shape has positions - # and that membership entries are correct (empty views for a fresh shape). - positions = SubqueryIndex.positions_for_shape(index, shape_handle) - assert length(positions) > 0 + {:ok, shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + [dep_handle] = shape.shape_dependencies_handles - # Verify the index is accessible and has retained node registrations. - assert positions == SubqueryIndex.positions_for_shape(index, shape_handle) + assert {^dep_handle, 0} = + SubqueryIndex.get_shape_subquery(index, shape_handle, ["$sublink", "0"]) end - @tag :uses_legacy_subquery_api - test "consumer steady dependency move_in adds value to the subquery index", ctx do + test "consumer steady dependency move_in advances the shape's subquery index time", + ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex parent = self() @@ -2321,7 +2314,8 @@ defmodule Electric.Shapes.ConsumerTest do index = SubqueryIndex.for_stack(ctx.stack_id) {:ok, _shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) - # Before any dependency changes, the index has empty membership + # Before any dep moves: shape's subquery_ref is bound at logical + # time 0 and the dep view is empty, so no value is a member yet. refute SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) # Send a new record for the dependency table to trigger a move_in @@ -2336,14 +2330,13 @@ defmodule Electric.Shapes.ConsumerTest do ctx.stack_id ) - # Wait for the consumer to process the event and request a move_in query assert_receive {:query_requested, consumer_pid} - # During buffering, the value should have been added to the index - # (union for positive dependency: before ∪ after) - assert SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # While the move-in query is buffering, the consumer's subquery + # time is still at 0, so the new value is not yet visible through + # `member?/4` (which reads at the shape's stored logical time). + refute SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) - # Complete the move_in query to transition back to steady state send(consumer_pid, {:pg_snapshot_known, {100, 300, []}}) shape_storage = Storage.for_shape(shape_handle, ctx.storage) @@ -2368,12 +2361,12 @@ defmodule Electric.Shapes.ConsumerTest do Lsn.from_integer(100) ) - # Allow the consumer to process the completion assert :ok = LsnTracker.broadcast_last_seen_lsn(ctx.stack_id, 100) ref = Shapes.Consumer.register_for_changes(ctx.stack_id, shape_handle) assert_receive {^ref, :new_changes, _offset}, @receive_timeout - # After move_in completes, value should still be in the index (now steady state) + # After splice the shape's subquery time has advanced past the + # move-in, and value 1 is now visible at the shape's stored time. assert SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) end diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index c4095069b4..10e2d79519 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -630,34 +630,6 @@ defmodule Electric.Shapes.FilterTest do end) end - @tag :uses_legacy_subquery_api - test "remove_shape/2 removes seeded subquery index state" do - filter = Filter.new() - state_before = snapshot_filter_ets(filter) - shape_id = "seeded-shape" - - shape = - Shape.new!("table", - where: "id IN (SELECT id FROM another_table)", - inspector: @inspector, - feature_flags: ["allow_subqueries"] - ) - - Filter.add_shape(filter, shape_id, shape) - - index = Filter.subquery_index(filter) - subquery_ref = ["$sublink", "0"] - - SubqueryIndex.seed_membership(index, shape_id, subquery_ref, 0, MapSet.new([5])) - SubqueryIndex.mark_ready(index, shape_id) - - assert snapshot_filter_ets(filter) != state_before - - Filter.remove_shape(filter, shape_id) - - assert snapshot_filter_ets(filter) == state_before - end - # Captures the full state of all ETS tables in a filter for comparison defp snapshot_filter_ets(filter) do %{ @@ -933,7 +905,7 @@ defmodule Electric.Shapes.FilterTest do end describe "subquery shapes routing in filter" do - @describetag :uses_legacy_subquery_api + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView import Support.DbSetup import Support.DbStructureSetup @@ -947,6 +919,37 @@ defmodule Electric.Shapes.FilterTest do :with_sql_execute ] + # Test helper: emulate the pre-RFC `SubqueryIndex.seed_membership/5` + # against the new shared-view API. Looks up the subquery_id that + # `Filter.add_shape` stored for the shape's dependency (handles either + # a real dep_handle or the `{shape_handle, dep_index}` fallback that + # `register_shape` falls back to when `shape_dependencies_handles` + # isn't populated), seeds `MultiTimeView` with the values, binds the + # shape's `subquery_ref` at logical time 0, and wires positive routing + # rows so `affected_shapes/2` can find each value. + defp seed_membership(filter, shape_id, _shape, subquery_ref, values) do + index = Filter.subquery_index(filter) + mtv = index.multi_time_view + dep_index = subquery_ref |> List.last() |> String.to_integer() + [{_, subquery_id}] = :ets.lookup(index.table, {:dep_handle, shape_id, dep_index}) + + MultiTimeView.init_subquery(mtv, subquery_id, values) + MultiTimeView.mark_ready(mtv, subquery_id) + + :ok = SubqueryIndex.set_shape_subquery(index, shape_id, subquery_ref, subquery_id, 0) + + for v <- values do + :ok = SubqueryIndex.add_positive_route(index, subquery_id, v) + end + + :ok + end + + # Test helper: emulate the pre-RFC per-value `SubqueryIndex.add_value/5`. + defp add_value(filter, shape_id, shape, subquery_ref, value) do + seed_membership(filter, shape_id, shape, subquery_ref, [value]) + end + @tag with_sql: [ "CREATE TABLE IF NOT EXISTS parent (id INT PRIMARY KEY)", "CREATE TABLE IF NOT EXISTS child (id INT PRIMARY KEY, par_id INT REFERENCES parent(id))" @@ -1131,7 +1134,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] for value <- [1, 2, 3] do - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, value) + add_value(filter, "shape1", shape, subquery_ref, value) end SubqueryIndex.mark_ready(index, "shape1") @@ -1190,7 +1193,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] for value <- [1, 2, 3] do - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, value) + add_value(filter, "shape1", shape, subquery_ref, value) end SubqueryIndex.mark_ready(index, "shape1") @@ -1268,7 +1271,7 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) + add_value(filter, "shape1", shape, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") wrong_subquery_value = %NewRecord{ @@ -1313,8 +1316,8 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) - SubqueryIndex.add_value(index, "shape2", subquery_ref, 0, 1) + add_value(filter, "shape1", shape1, subquery_ref, 1) + add_value(filter, "shape2", shape2, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") SubqueryIndex.mark_ready(index, "shape2") @@ -1344,14 +1347,23 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.add_value(index, "shape1", subquery_ref, 0, 1) + add_value(filter, "shape1", shape, subquery_ref, 1) SubqueryIndex.mark_ready(index, "shape1") - assert :ets.tab2list(index) != [] + # Before remove, the shape routes records whose value is in the + # subquery view, and the shape is bound to the dep's subquery_id. + hit = %NewRecord{relation: {"public", "child"}, record: %{"id" => "1", "par_id" => "9"}} + assert Filter.affected_shapes(filter, hit) == MapSet.new(["shape1"]) + assert SubqueryIndex.has_positions?(index, "shape1") + assert SubqueryIndex.get_shape_subquery(index, "shape1", subquery_ref) != nil Filter.remove_shape(filter, "shape1") - assert :ets.tab2list(index) == [] + # After remove, the shape no longer routes and its index entries + # are gone. + assert Filter.affected_shapes(filter, hit) == MapSet.new([]) + refute SubqueryIndex.has_positions?(index, "shape1") + assert SubqueryIndex.get_shape_subquery(index, "shape1", subquery_ref) == nil end @tag with_sql: [ @@ -1373,13 +1385,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed membership with value 1 (parent id 1 matches the subquery "WHERE value = 'keep'") - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1])) SubqueryIndex.mark_ready(index, "shape1") @@ -1430,13 +1436,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed the membership view with values {1, 2} - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1, 2]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) SubqueryIndex.mark_ready(index, "shape1") @@ -1484,13 +1484,7 @@ defmodule Electric.Shapes.FilterTest do subquery_ref = ["$sublink", "0"] # Seed membership with a tuple value {10, 20} - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([{10, 20}]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([{10, 20}])) SubqueryIndex.mark_ready(index, "shape1") @@ -1533,13 +1527,7 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) subquery_ref = ["$sublink", "0"] - SubqueryIndex.seed_membership( - index, - "shape1", - subquery_ref, - 0, - MapSet.new([1, 2]) - ) + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) SubqueryIndex.mark_ready(index, "shape1") @@ -1600,13 +1588,7 @@ defmodule Electric.Shapes.FilterTest do # seeded and marked ready. subquery_ref = ["$sublink", "0"] - SubqueryIndex.seed_membership( - index, - "indexed_s", - subquery_ref, - 0, - MapSet.new([1]) - ) + seed_membership(filter, "indexed_s", indexed_shape, subquery_ref, MapSet.new([1])) SubqueryIndex.mark_ready(index, "indexed_s") diff --git a/packages/sync-service/test/test_helper.exs b/packages/sync-service/test/test_helper.exs index b3c6bf8a72..9f5d0847ee 100644 --- a/packages/sync-service/test/test_helper.exs +++ b/packages/sync-service/test/test_helper.exs @@ -8,7 +8,7 @@ ExUnit.configure(formatters: [JUnitFormatter, ExUnit.CLIFormatter]) ExUnit.start( assert_receive_timeout: 400, - exclude: [:slow, :oracle, :performance, :uses_legacy_subquery_api], + exclude: [:slow, :oracle, :performance], capture_log: true ) From 93eb3c51a7fc3ae000779e5f7ac7db465cd93324 Mon Sep 17 00:00:00 2001 From: rob Date: Thu, 21 May 2026 13:40:19 +0100 Subject: [PATCH 27/40] Compact queue --- docs/rfcs/subquery-index.md | 135 +++++++++++++- .../event_handler/subqueries/buffering.ex | 35 ++-- .../event_handler/subqueries/steady.ex | 171 +++++++++--------- .../electric/shapes/consumer/materializer.ex | 16 ++ .../shapes/consumer/subqueries/active_move.ex | 39 ++-- .../consumer/subqueries/index_changes.ex | 85 +++++++++ .../consumer/subqueries/move_broadcast.ex | 14 +- .../shapes/consumer/subqueries/move_queue.ex | 125 ++++++++----- .../shapes/consumer/subqueries/splice_plan.ex | 57 +++++- .../consumer/subqueries/move_queue_test.exs | 90 ++++----- 10 files changed, 539 insertions(+), 228 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 864daa8288..7c4e4c876e 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -843,7 +843,8 @@ When `shape_a` receives the `s7` move from `0` to `1`, `ActiveMove` stores: subquery_id: s7, from_time: 0, to_time: 1, - values: [{30, "30"}] + move_in_values: [{30, "30"}], + move_out_values: [] } ``` @@ -856,9 +857,30 @@ views_after_move: MapSet.new([10, 20, 30]) Buffered row conversion evaluates exact membership by calling `MultiTimeView.member?/3` at `from_time` or `to_time`. Move-in SQL may -materialize `values(s7, 1)` as a query-local parameter array, but that memory +materialise `values(s7, 1)` as a query-local parameter array, but that memory belongs to the query task and is released after the query. +If additional materializer payloads for `s7` queue up during `shape_a`'s +buffering — say a move-in of `{40, "40"}` at time `2` and a move-out of +`{20, "20"}` at time `3` — they reduce into the *next* combined batch the +consumer pops after splicing this move: + +```elixir +%ActiveMove{ + subquery_id: s7, + from_time: 1, + to_time: 3, + move_in_values: [{40, "40"}], + move_out_values: [{20, "20"}] +} +``` + +`from_time` here is the consumer's processed time when the first follow-up +payload arrived (i.e. `shape_a`'s previous `ActiveMove.to_time`); `to_time` +is the max of the contributing `to_time`s; the value lists are the reduced +net effect. By construction +`MTV(s7, 3) = MTV(s7, 1) + {40} - {20}`. + Steady memory added per active move is: ```text @@ -1191,8 +1213,21 @@ The registration side effects are: - wait until the dependency materializer has finished initial population - register the consumer with `SubqueryProgressMonitor` - set the consumer's initial `required_time` to `current_time` +- **atomically add the consumer to the materializer's subscribers list** + before returning `current_time` - return `current_time` to the caller +The atomic-subscribe step is required for correctness. A two-call shape +("register, then subscribe") opens a race window where the materializer +commits between the calls: those commits go to the *old* subscribers list +and the new consumer never sees them. Its `current_time` stays at the value +returned by `register`, but the materializer's logical time has advanced +past it, so the consumer's first observable `materializer_changes` event +arrives with a `from_time` strictly greater than `current_time` — and +`MTV(current_time)` no longer reflects "what the consumer has processed." +That breaks the times-as-views invariant the rest of the design depends on +(see *Consumer Move Handling* below). + This replaces the current `Materializer.get_link_values/1` setup path for subquery event handlers. The handler should keep compact references such as: @@ -1211,19 +1246,25 @@ depend on it. ### Consumer Move Handling -For a move from time `a` to time `b`, `ActiveMove` should store times, not -views: +For a move from time `a` to time `b`, `ActiveMove` should store times and +the values whose membership changed between those times. It does *not* store +view snapshots: ```elixir %ActiveMove{ subquery_id: subquery_id, + dep_index: dep_index, + subquery_ref: subquery_ref, from_time: a, to_time: b, - values: values + move_in_values: ins, # values entering the dep view in [a, b] + move_out_values: outs, # values leaving the dep view in [a, b] + txids: [...], # source PG xids + ... # snapshot/query state for the move-in query } ``` -Elixir-side evaluation of buffered transactions should use callbacks into +Elixir-side evaluation of buffered transactions uses callbacks into `MultiTimeView`: ```elixir @@ -1231,7 +1272,7 @@ before_member? = fn ref, value -> member?(ref, value, a) end after_member? = fn ref, value -> member?(ref, value, b) end ``` -For SQL move-in queries, the first implementation can still materialize +For SQL move-in queries, the first implementation can still materialise query-local arrays by calling `MultiTimeView.values(subquery_id, time)`. The important change is that these arrays are transient query parameters, not long-lived per-consumer state. @@ -1249,6 +1290,86 @@ The important invariant is that live routing must not under-route, while `required_time` continues to pin `a` until the consumer no longer needs the old view. +#### Combined Move Batches (Times vs Views) + +The pre-RFC implementation stores frozen `MapSet` `views_before_move` and +`views_after_move`. Those snapshots are *path-dependent*: they reflect what +this consumer has processed, in the order it chose to process it. The +consumer's MoveQueue freely reorders move-outs ahead of move-ins because set +operations on disjoint values commute, so the consumer's final view is +unchanged. + +`MultiTimeView` doesn't have that freedom. `MTV(t)` is the materializer's +canonical view at logical time `t` — and the materializer applies moves in +PG commit order. Times are points in a totally ordered history, not view +deltas. So the consumer cannot pop the move-out batch first and "advance to +some intermediate time where only the outs have been applied" — no such +time exists if a move-in committed between them. + +To preserve the times-as-views invariant +(`MTV(consumer.time) = what the consumer has processed so far`), an +`ActiveMove` covers a *single contiguous window* `[a, b]` per dep and carries +*both* move-in and move-out values for that window. `MoveQueue.pop_next/1` +returns one combined batch per dep at a time: + +```elixir +{:ok, batch} = MoveQueue.pop_next(queue) +# batch = %{ +# dep_index: 0, +# move_in_values: [{V2, "V2"}, {V3, "V3"}], +# move_out_values: [{V1, "V1"}], +# from_time: a, +# to_time: b, +# txids: [...] +# } +``` + +The batch's `from_time` is the consumer's processed time when the *first* +payload in the window arrived (preserved across subsequent enqueues for the +same dep). The `to_time` is the max of the contributing payload `to_time`s. +By construction of the queue's per-dep reduce, +`MTV(b) = MTV(a) + move_in_values - move_out_values`. + +The splice plan for a combined batch emits effects in this order: + +``` +pre_ops — buffered txns before move-in snapshot, evaluated at MTV(a) +move_out_broadcast — for outer move-out values (may be empty) +move_in_broadcast — for outer move-in values (may be empty) +snapshot — records loaded by the move-in query +post_ops — buffered txns after snapshot, evaluated at MTV(b) +``` + +`pre_ops` first means a buffered txn that references a value about to be +moved out is stored at the *pre-batch* view (consistent with `MTV(a)`), and +the subsequent move-out broadcast cleans it up — the client sees `UPDATE +then DELETE` for that row, never `DELETE then UPDATE` (which would surface +as "update for row that does not exist"). + +For pure move-out batches (no move-in values, hence no PG query needed) the +consumer skips Buffering and broadcasts the move-out inline, advancing time +to `b` and recursing on the queue. + +#### MoveQueue Compaction Rules + +Per dep, within one contiguous window `[a, b]`, the queue may compact +arbitrary sequences of moves into a single `(move_in_values, move_out_values)` +pair as long as the net effect preserves +`MTV(b) = MTV(a) + ins - outs`. Specifically: + +- multiple adds of the same value collapse to one (idempotent) +- multiple removes of the same value collapse to one +- `add V` then `remove V` cancel (net zero) +- `remove V` then `add V` cancel (net zero) +- adds and removes for disjoint values keep both + +This is the same reduce the pre-RFC queue uses; the only change is that the +result is now expressed as two value lists carried by one `ActiveMove`, +rather than two batches popped separately. + +Cross-dep compaction is not safe — each dep has its own `subquery_id` and +its own MTV history. Each dep gets its own `ActiveMove`. + ### Querying Changes `Querying.move_in_where_clause/5` currently receives diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 08ff12e28a..28c46d193a 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -32,20 +32,23 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do ShapeInfo.t(), Steady.subquery_refs(), MoveQueue.t(), - IndexChanges.move(), + MoveQueue.combined_batch(), [String.t()], - from_time :: non_neg_integer(), - to_time :: non_neg_integer(), keyword() ) :: {:ok, t(), [Effects.t()]} def start( %ShapeInfo{} = shape_info, subquery_refs, %MoveQueue{} = queue, - {dep_move_kind, dep_index, values, txids}, + %{ + dep_index: dep_index, + move_in_values: move_in_values, + move_out_values: move_out_values, + from_time: from_time, + to_time: to_time, + txids: txids + }, subquery_ref, - from_time, - to_time, opts \\ [] ) do %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) @@ -58,9 +61,9 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do ActiveMove.start( subquery_id, dep_index, - dep_move_kind, subquery_ref, - values, + move_in_values, + move_out_values, from_time, to_time, txids @@ -68,13 +71,14 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do |> ActiveMove.carry_latest_seen_lsn(Keyword.get(opts, :latest_seen_lsn)) } - move = {dep_move_kind, dep_index, values, txids} - effects = EffectList.new() |> maybe_subscribe_global_lsn(Keyword.get(opts, :subscribe_global_lsn?, true)) |> EffectList.append_all( - IndexChanges.effects_for_buffering(state.shape_info.dnf_plan, move, subquery_ref) + IndexChanges.effects_for_buffering_active_move( + state.shape_info.dnf_plan, + state.active_move + ) ) |> EffectList.append(start_move_in_query_effect(state)) |> EffectList.to_list() @@ -148,12 +152,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do with {:ok, splice_plan} <- SplicePlan.build(active_move, state.shape_info, state.subquery_refs) do index_effects = - IndexChanges.effects_for_complete( - state.shape_info.dnf_plan, - {active_move.dep_move_kind, active_move.dep_index, active_move.values, - active_move.txids}, - active_move.subquery_ref - ) + IndexChanges.effects_for_complete_active_move(state.shape_info.dnf_plan, active_move) advance_consumer_to_after_move(state, active_move) @@ -203,7 +202,7 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do %Effects.StartMoveInQuery{ dnf_plan: shape_info.dnf_plan, trigger_dep_index: active_move.dep_index, - values: active_move.values, + values: active_move.move_in_values, subquery_id: active_move.subquery_id, subquery_ref: active_move.subquery_ref, from_time: active_move.from_time, diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index cdd036af06..987c492144 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -55,10 +55,16 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do mtv = MultiTimeView.for_stack(state.shape_info.stack_id) %{subquery_id: subquery_id, time: pinned_time} = Map.fetch!(state.subquery_refs, subquery_ref) dep_view = mtv |> MultiTimeView.values(subquery_id, pinned_time) |> MapSet.new() - next_state = %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)} - with {:ok, next_state, effects} <- - drain_queue(next_state, EffectList.new(), payload_time: payload[:to_time]) do + payload_with_default_from_time = Map.put_new(Map.new(payload), :from_time, pinned_time) + + next_state = %{ + state + | queue: + MoveQueue.enqueue(state.queue, dep_index, payload_with_default_from_time, dep_view) + } + + with {:ok, next_state, effects} <- drain_queue(next_state, EffectList.new()) do {:ok, next_state, EffectList.to_list(effects)} end end @@ -82,78 +88,89 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do nil -> {:ok, state, effects} - {{dep_move_kind, dep_index, values, txids} = move, batch_to_time, queue} -> - subquery_ref = RefResolver.ref_from_dep_index!(state.shape_info.ref_resolver, dep_index) - subscription_active? = Keyword.get(opts, :subscription_active?, false) - latest_seen_lsn = Keyword.get(opts, :latest_seen_lsn) - - case outer_move_kind(state.shape_info, dep_index, dep_move_kind) do - :move_in -> - %{time: from_time} = Map.fetch!(state.subquery_refs, subquery_ref) - to_time = batch_to_time || Keyword.get(opts, :payload_time, from_time) - - with {:ok, next_state, start_effects} <- - Buffering.start( - state.shape_info, - state.subquery_refs, - queue, - move, - subquery_ref, - from_time, - to_time, - subscribe_global_lsn?: not subscription_active?, - latest_seen_lsn: latest_seen_lsn - ) do - {:ok, next_state, EffectList.append_all(effects, start_effects)} - end - - :move_out -> - %{subquery_id: subquery_id} = Map.fetch!(state.subquery_refs, subquery_ref) - from_time = state.subquery_refs[subquery_ref].time - same_dep_move_in_pending? = Map.has_key?(queue.move_in, dep_index) - to_time = batch_to_time || Keyword.get(opts, :payload_time, from_time) - - # When the same dep has a queued move-in waiting, leave - # `subquery_refs.time` untouched: the upcoming Buffering session - # owns the final time advance (so its move-in query reads - # `views_before` at the pre-batch time and `views_after` at - # `to_time`, with the diff yielding the newly-added values). - next_subquery_refs = - if same_dep_move_in_pending? do - state.subquery_refs - else - advance_subquery_index_time( - state.shape_info, - subquery_ref, - subquery_id, - from_time, - to_time - ) - - advance_subquery_time(state.subquery_refs, subquery_ref, to_time) - end - - next_state = %{state | queue: queue, subquery_refs: next_subquery_refs} - - index_effects = - IndexChanges.effects_for_complete(state.shape_info.dnf_plan, move, subquery_ref) - - effects = - effects - |> EffectList.append( - MoveBroadcast.effect_for_move_out(dep_index, values, txids, state.shape_info) - ) - |> EffectList.append_all(index_effects) - - drain_queue( - next_state, - effects, - opts - ) + {batch, queue} -> + subquery_ref = + RefResolver.ref_from_dep_index!(state.shape_info.ref_resolver, batch.dep_index) + + polarity = + Map.fetch!(state.shape_info.dnf_plan.dependency_polarities, batch.dep_index) + + if buffering_required?(polarity, batch) do + start_buffering(state, queue, batch, subquery_ref, effects, opts) + else + process_inline(state, queue, batch, subquery_ref, polarity, effects, opts) end end end + defp buffering_required?(:positive, %{move_in_values: [_ | _]}), do: true + defp buffering_required?(:negated, %{move_out_values: [_ | _]}), do: true + defp buffering_required?(_, _), do: false + + defp start_buffering(state, queue, batch, subquery_ref, effects, opts) do + subscription_active? = Keyword.get(opts, :subscription_active?, false) + latest_seen_lsn = Keyword.get(opts, :latest_seen_lsn) + + with {:ok, next_state, start_effects} <- + Buffering.start( + state.shape_info, + state.subquery_refs, + queue, + batch, + subquery_ref, + subscribe_global_lsn?: not subscription_active?, + latest_seen_lsn: latest_seen_lsn + ) do + {:ok, next_state, EffectList.append_all(effects, start_effects)} + end + end + + # Inline path: the batch carries only non-Buffering-kind moves (positive + # polarity move-out, or negated polarity move-in). No PG query needed — + # we broadcast the outer move-out, update routing, and advance time. + defp process_inline(state, queue, batch, subquery_ref, polarity, effects, opts) do + %{subquery_id: subquery_id, time: from_time} = + Map.fetch!(state.subquery_refs, subquery_ref) + + to_time = batch.to_time || from_time + + {outer_move_out_values, outer_move_out_kind} = + case polarity do + :positive -> {batch.move_out_values, :move_out} + :negated -> {batch.move_in_values, :move_in} + end + + advance_subquery_index_time( + state.shape_info, + subquery_ref, + subquery_id, + from_time, + to_time + ) + + next_subquery_refs = advance_subquery_time(state.subquery_refs, subquery_ref, to_time) + next_state = %{state | queue: queue, subquery_refs: next_subquery_refs} + + move = {outer_move_out_kind, batch.dep_index, outer_move_out_values, batch.txids} + + index_effects = + IndexChanges.effects_for_complete(state.shape_info.dnf_plan, move, subquery_ref) + + effects = + effects + |> EffectList.append( + MoveBroadcast.effect_for_move_out( + batch.dep_index, + outer_move_out_values, + batch.txids, + state.shape_info + ) + ) + |> EffectList.append_all(index_effects) + + drain_queue(next_state, effects, opts) + end + defp advance_subquery_index_time( %ShapeInfo{} = shape_info, subquery_ref, @@ -190,18 +207,6 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do Map.update!(subquery_refs, subquery_ref, fn meta -> %{meta | time: to_time} end) end - defp outer_move_kind( - %ShapeInfo{dnf_plan: %{dependency_polarities: polarities}}, - dep_index, - move_kind - ) do - case {Map.fetch!(polarities, dep_index), move_kind} do - {:positive, effect} -> effect - {:negated, :move_in} -> :move_out - {:negated, :move_out} -> :move_in - end - end - defp append_txn_effects(%Transaction{} = txn, %__MODULE__{} = state) do mtv = MultiTimeView.for_stack(state.shape_info.stack_id) diff --git a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex index 9a8cb3f3e9..1b075380cd 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/materializer.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/materializer.ex @@ -220,6 +220,22 @@ defmodule Electric.Shapes.Consumer.Materializer do time ) + # Atomically subscribe `pid` so the returned `time` is the materializer's + # logical time at the consumer's first observable commit. Without this + # there's a race window between this call returning and the consumer's + # follow-up `subscribe/1` — any commits in that window go to other + # subscribers only, advancing the materializer's logical time past + # `time` while the new consumer never sees those events. The result is + # `MTV(from_time)` on the consumer's first batch reflecting moves the + # consumer never processed, which breaks the times-as-views invariant. + state = + if MapSet.member?(state.subscribers, pid) do + state + else + Process.monitor(pid) + %{state | subscribers: MapSet.put(state.subscribers, pid)} + end + {:reply, {:ok, time}, state} end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex index fe4c2bf786..a6c2bc8295 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex @@ -15,18 +15,18 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do @enforce_keys [ :subquery_id, :dep_index, - :dep_move_kind, :subquery_ref, - :values, + :move_in_values, + :move_out_values, :from_time, :to_time ] defstruct [ :subquery_id, :dep_index, - :dep_move_kind, :subquery_ref, - :values, + :move_in_values, + :move_out_values, :from_time, :to_time, txids: [], @@ -44,9 +44,9 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do @type t() :: %__MODULE__{ subquery_id: term(), dep_index: non_neg_integer(), - dep_move_kind: :move_in | :move_out, subquery_ref: [String.t()], - values: [move_value()], + move_in_values: [move_value()], + move_out_values: [move_value()], from_time: non_neg_integer(), to_time: non_neg_integer(), txids: [non_neg_integer()], @@ -64,9 +64,9 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do @spec start( subquery_id :: term(), non_neg_integer(), - :move_in | :move_out, [String.t()], - [move_value()], + move_in_values :: [move_value()], + move_out_values :: [move_value()], from_time :: non_neg_integer(), to_time :: non_neg_integer(), [non_neg_integer()] @@ -74,9 +74,9 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do def start( subquery_id, dep_index, - dep_move_kind, subquery_ref, - values, + move_in_values, + move_out_values, from_time, to_time, txids \\ [] @@ -84,15 +84,30 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do %__MODULE__{ subquery_id: subquery_id, dep_index: dep_index, - dep_move_kind: dep_move_kind, subquery_ref: subquery_ref, - values: values, + move_in_values: move_in_values, + move_out_values: move_out_values, from_time: from_time, to_time: to_time, txids: txids } end + @doc """ + Returns true if the active move carries any move-in values that need a + PG query to load records. + """ + @spec has_move_in?(t()) :: boolean() + def has_move_in?(%__MODULE__{move_in_values: []}), do: false + def has_move_in?(%__MODULE__{move_in_values: _}), do: true + + @doc """ + Returns true if the active move carries any move-out values to broadcast. + """ + @spec has_move_out?(t()) :: boolean() + def has_move_out?(%__MODULE__{move_out_values: []}), do: false + def has_move_out?(%__MODULE__{move_out_values: _}), do: true + @doc """ Materialise the dependency view as a `MapSet` at the active move's `from_time`. The result is intended for transient use at the SplicePlan / diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex index 65cd4e1b6f..58b82429e7 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/index_changes.ex @@ -133,4 +133,89 @@ defmodule Electric.Shapes.Consumer.Subqueries.IndexChanges do [] end end + + @doc """ + Effects to broaden the filter when entering Buffering for a combined + ActiveMove that may carry both `move_in_values` and `move_out_values`. + + - Positive polarity: add `move_in_values` to the index (broadens). + - Negated polarity: remove `move_out_values` from the index (broadens). + """ + @spec effects_for_buffering_active_move( + DnfPlan.t(), + Electric.Shapes.Consumer.Subqueries.ActiveMove.t() + ) :: [Effects.AddToSubqueryIndex.t() | Effects.RemoveFromSubqueryIndex.t()] + def effects_for_buffering_active_move(%DnfPlan{dependency_polarities: polarities}, active_move) do + polarity = Map.get(polarities, active_move.dep_index, :positive) + + case polarity do + :positive -> + if active_move.move_in_values == [] do + [] + else + [ + %Effects.AddToSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_in_values + } + ] + end + + :negated -> + if active_move.move_out_values == [] do + [] + else + [ + %Effects.RemoveFromSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_out_values + } + ] + end + end + end + + @doc """ + Effects to narrow the filter when a combined ActiveMove splices. + + - Positive polarity: remove `move_out_values` from the index. + - Negated polarity: add `move_in_values` to the index. + """ + @spec effects_for_complete_active_move( + DnfPlan.t(), + Electric.Shapes.Consumer.Subqueries.ActiveMove.t() + ) :: [Effects.AddToSubqueryIndex.t() | Effects.RemoveFromSubqueryIndex.t()] + def effects_for_complete_active_move(%DnfPlan{dependency_polarities: polarities}, active_move) do + polarity = Map.get(polarities, active_move.dep_index, :positive) + + case polarity do + :positive -> + if active_move.move_out_values == [] do + [] + else + [ + %Effects.RemoveFromSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_out_values + } + ] + end + + :negated -> + if active_move.move_in_values == [] do + [] + else + [ + %Effects.AddToSubqueryIndex{ + dep_index: active_move.dep_index, + subquery_ref: active_move.subquery_ref, + values: active_move.move_in_values + } + ] + end + end + end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex index 61f69b3324..9db453e1cf 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_broadcast.ex @@ -12,12 +12,24 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveBroadcast do @spec effect_for_move_in(move(), ShapeInfo.t()) :: %Effects.AppendControl{} def effect_for_move_in(active_move, %ShapeInfo{} = shape_info) do + polarity = + Map.get(shape_info.dnf_plan.dependency_polarities, active_move.dep_index, :positive) + + # The active move's "outer move-in" values are the dep values whose + # entry into (positive) or exit from (negated) the dep view promotes + # rows into the outer shape. + outer_move_in_values = + case polarity do + :positive -> active_move.move_in_values + :negated -> active_move.move_out_values + end + %Effects.AppendControl{ message: make( shape_info.dnf_plan, active_move.dep_index, - active_move.values, + outer_move_in_values, active_move.txids, "move-in", shape_info.stack_id, diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex index d522c639a8..08bd2b0811 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex @@ -1,33 +1,44 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @moduledoc """ - Multi-dependency move queue. Tracks move_in/move_out operations per dependency index, - with deduplication and redundancy elimination scoped per dependency. - - Move-outs from any dependency are drained before move-ins from any dependency. - - Each per-dep batch also accumulates the upstream Postgres transaction ids that - contributed to it, so move-in/move-out broadcasts can carry `txids` for client - attribution (mirroring `Electric.LogItems.from_change/4`). + Multi-dependency move queue. Tracks move_in/move_out operations per + dependency index, with deduplication and redundancy elimination scoped + per dependency via `reduce/2`. + + Each `pop_next/1` call returns ONE combined entry for a single dep that + carries both `move_in_values` and `move_out_values`, along with the + `from_time` (the materializer logical time the consumer was at when the + first payload in this batch was enqueued) and `to_time` (the max of all + payload `to_time`s queued for this dep). The combined shape lets the + splice plan handle a dep's full transition window as one atomic + ActiveMove — `MTV(from_time)` and `MTV(to_time)` are well-defined + endpoints, and `MTV(to_time) = MTV(from_time) + move_in_values + - move_out_values` by construction of the reduce. + + Per-dep txids are accumulated across the contributing payloads so the + broadcasts can carry `txids` for client attribution. """ @type move_value() :: {term(), term()} @type txid() :: pos_integer() @type entry() :: {[move_value()], MapSet.t(txid())} - # move_out/move_in are maps from dep_index to {[move_value], MapSet} - # to_times tracks the latest payload `to_time` seen per dep_index, so when - # a queued batch is later popped we know what materializer logical time it - # represents. - defstruct move_out: %{}, move_in: %{}, to_times: %{} + defstruct move_out: %{}, move_in: %{}, from_times: %{}, to_times: %{} @type t() :: %__MODULE__{ move_out: %{non_neg_integer() => entry()}, move_in: %{non_neg_integer() => entry()}, + from_times: %{non_neg_integer() => non_neg_integer()}, to_times: %{non_neg_integer() => non_neg_integer()} } - @type batch_kind() :: :move_out | :move_in - @type batch() :: {batch_kind(), non_neg_integer(), [move_value()], [txid()]} + @type combined_batch() :: %{ + dep_index: non_neg_integer(), + move_in_values: [move_value()], + move_out_values: [move_value()], + from_time: non_neg_integer() | nil, + to_time: non_neg_integer() | nil, + txids: [txid()] + } @spec new() :: t() def new, do: %__MODULE__{} @@ -43,10 +54,15 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @doc """ Enqueue a materializer payload for a specific dependency. - `dep_view` is the current view for this dependency, used for redundancy elimination. - The payload may include a `:txids` key listing the upstream xids that produced - the moves. Those xids are unioned with any already accumulated for this dep. + `dep_view` is the materializer view at the consumer's pinned time for + this dep (or the view-after-active-move for the trigger ref during + Buffering). It's used by `reduce/2` to drop redundant ops. + + The payload may include `:from_time`, `:to_time`, and `:txids` keys. + The first enqueue for a dep records `from_time` (subsequent payloads + leave it untouched). `to_time` is updated to `max(current, new)`. Txids + accumulate. """ @spec enqueue(t(), non_neg_integer(), map() | keyword(), MapSet.t()) :: t() def enqueue(%__MODULE__{} = queue, dep_index, payload, %MapSet{} = dep_view) @@ -64,6 +80,12 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do {new_outs, new_ins} = reduce(ops, dep_view) + from_times = + case Map.get(payload, :from_time) do + nil -> queue.from_times + new_from_time -> Map.put_new(queue.from_times, dep_index, new_from_time) + end + to_times = case Map.get(payload, :to_time) do nil -> queue.to_times @@ -85,50 +107,53 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do new_ins, MapSet.union(existing_in_txids, new_txids) ), + from_times: from_times, to_times: to_times } end @doc """ - Pop the next batch of operations. Returns move-out batches (any dep) before move-in batches. - Returns `{batch, to_time, updated_queue}` or `nil` if the queue is empty. - `to_time` is the latest materializer logical time seen for this dep_index, or - `nil` if the batch was enqueued without a `:to_time` key. + Pop the next combined entry for one dep. Returns `{batch, updated_queue}` + where `batch` carries both `move_in_values` and `move_out_values` (either + may be empty, but not both — empty entries are never enqueued). Returns + `nil` if the queue is empty. """ - @spec pop_next(t()) :: {batch(), non_neg_integer() | nil, t()} | nil - def pop_next(%__MODULE__{move_out: move_out} = queue) when move_out != %{} do - {dep_index, {values, txids}} = Enum.min_by(move_out, &elem(&1, 0)) - to_time = Map.get(queue.to_times, dep_index) - - next_queue = %{queue | move_out: Map.delete(move_out, dep_index)} - - {{:move_out, dep_index, values, sorted_txids(txids)}, to_time, - drop_to_time_if_dep_empty(next_queue, dep_index)} - end - - def pop_next(%__MODULE__{move_out: move_out, move_in: move_in} = queue) - when move_out == %{} and move_in != %{} do - {dep_index, {values, txids}} = Enum.min_by(move_in, &elem(&1, 0)) - to_time = Map.get(queue.to_times, dep_index) + @spec pop_next(t()) :: {combined_batch(), t()} | nil + def pop_next(%__MODULE__{move_in: move_in, move_out: move_out}) + when move_in == %{} and move_out == %{}, + do: nil + + def pop_next(%__MODULE__{} = queue) do + dep_index = pick_dep_index(queue) + + {move_in_values, in_txids} = Map.get(queue.move_in, dep_index, {[], MapSet.new()}) + {move_out_values, out_txids} = Map.get(queue.move_out, dep_index, {[], MapSet.new()}) + txids = MapSet.union(in_txids, out_txids) |> Enum.sort() + + batch = %{ + dep_index: dep_index, + move_in_values: move_in_values, + move_out_values: move_out_values, + from_time: Map.get(queue.from_times, dep_index), + to_time: Map.get(queue.to_times, dep_index), + txids: txids + } - next_queue = %{queue | move_in: Map.delete(move_in, dep_index)} + next_queue = %__MODULE__{ + move_in: Map.delete(queue.move_in, dep_index), + move_out: Map.delete(queue.move_out, dep_index), + from_times: Map.delete(queue.from_times, dep_index), + to_times: Map.delete(queue.to_times, dep_index) + } - {{:move_in, dep_index, values, sorted_txids(txids)}, to_time, - drop_to_time_if_dep_empty(next_queue, dep_index)} + {batch, next_queue} end - def pop_next(%__MODULE__{}), do: nil - - defp drop_to_time_if_dep_empty(%__MODULE__{} = queue, dep_index) do - if Map.has_key?(queue.move_out, dep_index) or Map.has_key?(queue.move_in, dep_index) do - queue - else - %{queue | to_times: Map.delete(queue.to_times, dep_index)} - end + defp pick_dep_index(%__MODULE__{move_in: move_in, move_out: move_out}) do + candidates = Map.keys(move_in) ++ Map.keys(move_out) + Enum.min(candidates) end - defp sorted_txids(%MapSet{} = txids), do: Enum.sort(txids) - defp payload_to_ops(payload) do Enum.map(Map.get(payload, :move_out, []), &{:move_out, &1}) ++ Enum.map(Map.get(payload, :move_in, []), &{:move_in, &1}) diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex index d5acbd0eec..1e59c52a18 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/splice_plan.ex @@ -38,13 +38,17 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do {active_move.subquery_ref, active_move.to_time} ) + polarity = + Map.get(shape_info.dnf_plan.dependency_polarities, active_move.dep_index, :positive) + with {:ok, pre_ops} <- convert_txns(pre_txns, shape_info, before_member?), {:ok, post_ops} <- convert_txns(post_txns, shape_info, after_member?) do effects = EffectList.new() |> EffectList.append_all(pre_ops) - |> EffectList.append(MoveBroadcast.effect_for_move_in(active_move, shape_info)) - |> EffectList.append(move_in_snapshot_effect(active_move)) + |> maybe_append_move_out_broadcast(active_move, shape_info, polarity) + |> maybe_append_move_in_broadcast(active_move, shape_info, polarity) + |> maybe_append_move_in_snapshot(active_move) |> EffectList.append_all(post_ops) |> EffectList.to_list() @@ -67,12 +71,55 @@ defmodule Electric.Shapes.Consumer.Subqueries.SplicePlan do ) end - defp move_in_snapshot_effect(%ActiveMove{} = active_move) do - %Effects.AppendMoveInSnapshot{ + # Outer-perspective move-out broadcast — values whose exit from the dep + # view (positive) or entry into it (negated) drops rows out of the outer + # shape. + defp maybe_append_move_out_broadcast(effects, %ActiveMove{} = active_move, shape_info, polarity) do + outer_move_out_values = + case polarity do + :positive -> active_move.move_out_values + :negated -> active_move.move_in_values + end + + case outer_move_out_values do + [] -> + effects + + values -> + EffectList.append( + effects, + MoveBroadcast.effect_for_move_out( + active_move.dep_index, + values, + active_move.txids, + shape_info + ) + ) + end + end + + defp maybe_append_move_in_broadcast(effects, %ActiveMove{} = active_move, shape_info, polarity) do + outer_move_in_values = + case polarity do + :positive -> active_move.move_in_values + :negated -> active_move.move_out_values + end + + if outer_move_in_values == [] do + effects + else + EffectList.append(effects, MoveBroadcast.effect_for_move_in(active_move, shape_info)) + end + end + + defp maybe_append_move_in_snapshot(effects, %ActiveMove{move_in_snapshot_name: nil}), do: effects + + defp maybe_append_move_in_snapshot(effects, %ActiveMove{} = active_move) do + EffectList.append(effects, %Effects.AppendMoveInSnapshot{ snapshot_name: active_move.move_in_snapshot_name, row_count: active_move.move_in_row_count, row_bytes: active_move.move_in_row_bytes, snapshot: active_move.snapshot - } + }) end end diff --git a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs index a5fcc9b3da..036692131c 100644 --- a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs @@ -8,17 +8,13 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "drops redundant move outs for values absent from the base view" do queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_out: [{1, "1"}]}, MapSet.new()) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "drops redundant move ins for values already present in the base view" do queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "cancels a pending move in with a later move out for the same value" do @@ -27,9 +23,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new()) |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new()) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "cancels a pending move out with a later move in for the same value" do @@ -38,9 +32,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new([1])) |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert nil == MoveQueue.pop_next(queue) end test "merges repeated move ins and keeps the terminal tuple" do @@ -49,8 +41,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue(@dep, %{move_in: [{1, "01"}]}, MapSet.new()) |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], move_out: []}, MapSet.new()) - assert %MoveQueue{move_in: %{0 => {[{1, "1"}], _}}, move_out: empty_out} = queue - assert empty_out == %{} + assert {%{move_in_values: [{1, "1"}], move_out_values: []}, _queue} = MoveQueue.pop_next(queue) end test "merges repeated move outs and keeps the terminal tuple" do @@ -59,66 +50,61 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue(@dep, %{move_out: [{1, "01"}]}, MapSet.new([1])) |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}], move_in: []}, MapSet.new([1])) - assert %MoveQueue{move_out: %{0 => {[{1, "1"}], _}}, move_in: empty_in} = queue - assert empty_in == %{} - end - - test "orders surviving move outs before move ins" do - queue = - MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new([1])) - - assert %MoveQueue{ - move_out: %{0 => {[{1, "1"}], _}}, - move_in: %{0 => {[{2, "2"}], _}} - } = queue - end - - test "uses the provided base view when reducing buffering follow-up moves" do - queue = - MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{2, "2"}]}, MapSet.new([1])) - - assert %MoveQueue{move_out: empty_out, move_in: empty_in} = queue - assert empty_out == %{} - assert empty_in == %{} + assert {%{move_in_values: [], move_out_values: [{1, "1"}]}, _queue} = MoveQueue.pop_next(queue) end - test "pop_next returns the whole move out batch before the move in batch" do + test "pop_next returns one combined batch per dep carrying both kinds" do queue = MoveQueue.new() |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) - assert {{:move_out, 0, [{1, "1"}], []}, _to_time, queue} = MoveQueue.pop_next(queue) - assert queue.move_out == %{} - assert {[{2, "2"}, {3, "3"}], _} = Map.fetch!(queue.move_in, 0) - - assert {{:move_in, 0, [{2, "2"}, {3, "3"}], []}, _to_time, queue} = - MoveQueue.pop_next(queue) + assert { + %{ + dep_index: 0, + move_in_values: [{2, "2"}, {3, "3"}], + move_out_values: [{1, "1"}] + }, + queue + } = MoveQueue.pop_next(queue) - assert queue.move_out == %{} - assert queue.move_in == %{} assert nil == MoveQueue.pop_next(queue) end - test "accumulates txids from successive enqueues per dependency" do + test "carries the first from_time and the max to_time per dep" do queue = MoveQueue.new() |> MoveQueue.enqueue( @dep, - %{move_in: [{1, "1"}], txids: [10]}, + %{move_in: [{1, "1"}], from_time: 5, to_time: 6}, MapSet.new() ) |> MoveQueue.enqueue( @dep, - %{move_in: [{2, "2"}], txids: [20]}, + %{move_in: [{2, "2"}], from_time: 6, to_time: 9}, MapSet.new() ) - assert {{:move_in, 0, _, [10, 20]}, _to_time, _queue} = MoveQueue.pop_next(queue) + assert {%{from_time: 5, to_time: 9}, _queue} = MoveQueue.pop_next(queue) + end + + test "accumulates txids from successive enqueues per dependency" do + queue = + MoveQueue.new() + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], txids: [10]}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], txids: [20]}, MapSet.new()) + + assert {%{move_in_values: _, txids: [10, 20]}, _queue} = MoveQueue.pop_next(queue) + end + + test "pops the lowest-indexed dep first across deps" do + queue = + MoveQueue.new() + |> MoveQueue.enqueue(1, %{move_in: [{2, "2"}]}, MapSet.new()) + |> MoveQueue.enqueue(0, %{move_in: [{1, "1"}]}, MapSet.new()) + + assert {%{dep_index: 0}, queue} = MoveQueue.pop_next(queue) + assert {%{dep_index: 1}, _queue} = MoveQueue.pop_next(queue) end test "length counts queued values across both batches" do From e7c42b8802416a7f775037753c71605e09c59d18 Mon Sep 17 00:00:00 2001 From: rob Date: Thu, 21 May 2026 14:08:35 +0100 Subject: [PATCH 28/40] Add changeset --- .changeset/subquery-memory-and-lag-fix.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/subquery-memory-and-lag-fix.md diff --git a/.changeset/subquery-memory-and-lag-fix.md b/.changeset/subquery-memory-and-lag-fix.md new file mode 100644 index 0000000000..bb0bdd3a00 --- /dev/null +++ b/.changeset/subquery-memory-and-lag-fix.md @@ -0,0 +1,5 @@ +--- +'@core/sync-service': patch +--- + +Reduced memory consumption of subqueries and fix lag issue caused by subquery removal. From ffa3571fc725c8a8184eafa150b47a101754651e Mon Sep 17 00:00:00 2001 From: rob Date: Thu, 21 May 2026 16:16:26 +0100 Subject: [PATCH 29/40] Update RFC to remove filter's knowledge of consumer time --- docs/rfcs/subquery-index.md | 116 +++++++++++++++++++++++------------- 1 file changed, 73 insertions(+), 43 deletions(-) diff --git a/docs/rfcs/subquery-index.md b/docs/rfcs/subquery-index.md index 7c4e4c876e..cd422d1067 100644 --- a/docs/rfcs/subquery-index.md +++ b/docs/rfcs/subquery-index.md @@ -169,9 +169,17 @@ Consumer event handler SubqueryIndex -> stores subquery groups, child nodes, and participant routing -> asks MultiTimeView for membership at some/all retained times for routing - -> asks MultiTimeView for membership at a consumer time for exact checks + -> over-routes when consumers diverge; exact split is the consumer's job ``` +Routing is intentionally conservative: when consumers diverge across logical +times — including the common case of a single consumer that is mid-move and +effectively reading at two times at once — the filter cannot encode that with +a per-shape logical-time pin. Exact membership is therefore checked at +`Shape.convert_change`/`WhereClause.includes_record?/3`, using a +`subquery_member?` callback that the consumer builds from its own logical +time(s) against `MultiTimeView`. + ### Definitions #### Subquery @@ -434,17 +442,29 @@ This is `O(number_of_affected_shapes)` for large negated groups. That is acceptable because a value absent from a large negated group genuinely affects all of those shapes. -Exact filter verification uses the consumer's current logical time for the -requested subquery: - -```elixir -MultiTimeView.member?(subquery_id, typed_value, logical_time) -``` - -`WhereClause.subquery_member_from_index/2` should therefore resolve -`shape_handle + subquery_ref` to `{subquery_id, logical_time}` and call the -shared view. The callback remains the boundary used by -`WhereClause.includes_record?/3`. +Exact membership verification is not done by the filter. The filter cannot +correctly perform a per-shape split at routing time because, during a buffered +move-in, a consumer is effectively reading at *both* `from_time` and `to_time` +for the same subquery — splice-plan evaluates buffered transactions at +`MTV(from_time)` for pre-ops and `MTV(to_time)` for post-ops. A single +per-shape logical-time pin in the filter cannot represent that, and the filter +does not know how a given record relates to a given consumer's move window +without duplicating consumer state. + +Therefore the boundary at which exact membership is evaluated is the consumer, +via the `subquery_member?` callback passed into `WhereClause.includes_record?/3` +from `Shape.convert_change`. The consumer constructs that callback from its own +per-subquery logical time(s) — one callback for the steady case, two callbacks +(`old_member?` and `new_member?`) during a buffered move — and calls +`MultiTimeView.member?(subquery_id, typed_value, logical_time)` against the +shared view. + +The filter does maintain `{shape_handle, subquery_ref} -> {subquery_id, +logical_time}` rows so it can answer exact membership for sublink refs that +survive in the residual `and_where` (i.e. sublinks at *other* positions in the +shape's WHERE that were not the routed position). For those, the filter's +`subquery_member_from_index` callback is sufficient because the shape is not +mid-move on that other position at this routing step. ### Operation Examples And Costs @@ -677,28 +697,28 @@ Routing does: 2. Look up `{:positive, g_user_pos, 10}` and get `[c_s7_pos]`. 3. Evaluate child condition `wc_s7_pos`, which considers `shape_a` and `shape_b`. -4. For each candidate shape, exact subquery checks resolve: +4. Return both as candidates. No exact membership check happens in the filter + at the routed position — that's the consumer's job in `convert_change`. + +In this case the consumers in both shapes will confirm `10 ∈ s7` at their +logical time and the record is delivered: ```text -{:shape_subquery, shape_a, ["$sublink", "0"]} -> {s7, 0} -{:shape_subquery, shape_b, ["$sublink", "0"]} -> {s7, 0} -MultiTimeView.member?(s7, 10, 0) -> true +MultiTimeView.member?(s7, 10, 0) -> true # shape_a +MultiTimeView.member?(s7, 10, 0) -> true # shape_b ``` Both shapes are affected. -Cost: +Cost at the filter: ```text -O( - children_for_value + - child_where_eval + - exact_subquery_checks * transition_history_length_for_value -) +O(children_for_value + child_where_eval) ``` For this example, `children_for_value = 1`. There is no scan of all shapes and -no scan of all values in `s7`. +no scan of all values in `s7`. The per-consumer exact check is paid downstream +in `convert_change`, at `O(transition_history_length_for_value)` per shape. #### `affected_shapes`: Positive Group With Divergent Consumer Times @@ -724,24 +744,32 @@ For: %{"user_id" => 30} ``` -routing finds `c_s7_pos` because `30` is a member at some retained time. Exact -checks then split the result: +routing finds `c_s7_pos` because `30` is a member at some retained time. The +filter returns both `shape_a` and `shape_b` as candidates — it does *not* +attempt to split them at this point. The downstream exact check then drops the +false positive at each consumer: ```text -MultiTimeView.member?(s7, 30, 1) -> true -MultiTimeView.member?(s7, 30, 0) -> false +# In shape_a's consumer, evaluating includes_record? at logical_time 1: +MultiTimeView.member?(s7, 30, 1) -> true # shape_a keeps the record + +# In shape_b's consumer, evaluating includes_record? at logical_time 0: +MultiTimeView.member?(s7, 30, 0) -> false # shape_b drops the record ``` -Only `shape_a` is affected. +End-to-end, only `shape_a` emits the change. `shape_b` over-routes briefly but +filters the record in `Shape.convert_change`. -Cost remains: +Cost at the filter remains: ```text -O( - children_for_value + - child_where_eval + - exact_subquery_checks * transition_history_length_for_value -) +O(children_for_value + child_where_eval) +``` + +The per-consumer exact check is paid downstream, in `convert_change`, at: + +```text +O(transition_history_length_for_value) per shape ``` The extra memory for the move is one history row for `{s7, 30}` plus one @@ -765,24 +793,26 @@ present at time `1`. Negated routing does: not MultiTimeView.member_at_all_times?(s7, 30) ``` -3. Evaluate `wc_s7_neg` and exact membership at each candidate shape's - subquery logical time. +3. Evaluate `wc_s7_neg` and return the attached negated shapes as candidates. -For `shape_n` at logical time `0`, `NOT IN s7` is true for `30`. If it later -advances to logical time `1`, `NOT IN s7` is false for `30`. +Per-shape correctness for the negated case is again paid in +`Shape.convert_change`: for `shape_n` at logical time `0`, `NOT IN s7` is true +for `30`; if it later advances to logical time `1`, `NOT IN s7` is false for +`30`. That distinction is made by the consumer, not the filter. -Cost: +Cost at the filter: ```text O( number_of_negated_children_in_group * transition_history_length_for_value + - child_where_eval + - exact_subquery_checks * transition_history_length_for_value + child_where_eval ) ``` -This is intentionally proportional to the number of affected negated children. -No complement index is stored. +This is intentionally proportional to the number of negated children kept by +routing. No complement index is stored. The per-consumer exact check is again +paid downstream in `convert_change`, at +`O(transition_history_length_for_value)` per shape. #### Dependency Move: Add Or Remove Values From d1951832cdcf1c91fa9fbc61f4635237ae46907a Mon Sep 17 00:00:00 2001 From: rob Date: Thu, 21 May 2026 16:55:51 +0100 Subject: [PATCH 30/40] Remove Filter's knowledge of consumer time --- .../event_handler/subqueries/buffering.ex | 27 +---- .../event_handler/subqueries/steady.ex | 31 ++--- .../electric/shapes/consumer/setup_effects.ex | 14 +-- .../shapes/filter/indexes/subquery_index.ex | 114 ++---------------- .../electric/shapes/filter/where_condition.ex | 13 +- .../lib/electric/shapes/where_clause.ex | 22 ++-- .../test/electric/shapes/consumer_test.exs | 34 +++--- .../filter/indexes/subquery_index_test.exs | 89 +++----------- .../test/electric/shapes/filter_test.exs | 24 ++-- 9 files changed, 97 insertions(+), 271 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 28c46d193a..94757d9b4f 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -14,7 +14,6 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo alias Electric.Shapes.Consumer.Subqueries.SplicePlan - alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor @@ -211,26 +210,12 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do end defp advance_consumer_to_after_move(%__MODULE__{shape_info: shape_info}, active_move) do - case SubqueryIndex.for_stack(shape_info.stack_id) do - nil -> - :ok - - index -> - SubqueryIndex.set_shape_subquery( - index, - shape_info.shape_handle, - active_move.subquery_ref, - active_move.subquery_id, - active_move.to_time - ) - - ProgressMonitor.notify_processed_up_to( - shape_info.stack_id, - active_move.from_time, - active_move.subquery_id, - shape_info.shape_handle - ) - end + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + active_move.from_time, + active_move.subquery_id, + shape_info.shape_handle + ) end defp maybe_subscribe_global_lsn(effects, true) do diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index 987c492144..c2c040aedb 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -13,7 +13,6 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do alias Electric.Shapes.Consumer.Subqueries.MoveQueue alias Electric.Shapes.Consumer.Subqueries.RefResolver alias Electric.Shapes.Consumer.Subqueries.ShapeInfo - alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor @@ -173,31 +172,17 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do defp advance_subquery_index_time( %ShapeInfo{} = shape_info, - subquery_ref, + _subquery_ref, subquery_id, from_time, - to_time + _to_time ) do - case SubqueryIndex.for_stack(shape_info.stack_id) do - nil -> - :ok - - index -> - SubqueryIndex.set_shape_subquery( - index, - shape_info.shape_handle, - subquery_ref, - subquery_id, - to_time - ) - - ProgressMonitor.notify_processed_up_to( - shape_info.stack_id, - from_time, - subquery_id, - shape_info.shape_handle - ) - end + ProgressMonitor.notify_processed_up_to( + shape_info.stack_id, + from_time, + subquery_id, + shape_info.shape_handle + ) end @doc false diff --git a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex index 9a0bf0102e..cf490c230e 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/setup_effects.ex @@ -2,7 +2,6 @@ defmodule Electric.Shapes.Consumer.SetupEffects do # Executes ordered boot-time setup effects for consumer handler initialization. alias Electric.Replication.ShapeLogCollector - alias Electric.Shapes.Consumer.EventHandler.Subqueries.Steady alias Electric.Shapes.Consumer.State alias Electric.Shapes.Filter.Indexes.SubqueryIndex @@ -44,23 +43,14 @@ defmodule Electric.Shapes.Consumer.SetupEffects do end end - defp execute_effect( - %SeedSubqueryIndex{}, - %State{event_handler: %Steady{subquery_refs: refs}} = state - ) do + defp execute_effect(%SeedSubqueryIndex{}, %State{} = state) do case SubqueryIndex.for_stack(state.stack_id) do nil -> {:ok, state} index -> - for {ref, %{subquery_id: subquery_id, time: time}} <- refs do - SubqueryIndex.set_shape_subquery(index, state.shape_handle, ref, subquery_id, time) - end - - SubqueryIndex.mark_ready(index, state.shape_handle) + :ok = SubqueryIndex.mark_ready(index, state.shape_handle) {:ok, state} end end - - defp execute_effect(%SeedSubqueryIndex{}, %State{} = state), do: {:ok, state} end diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 61ac7b0cd8..a426061f00 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -5,12 +5,13 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do # reference the same subquery: groups (per filter node + polarity), child # nodes (per group + dependency subquery), value-keyed positive routing, # group-keyed negated routing, and child participants. Per-shape value - # membership is *not* stored here — exact-membership checks resolve - # `{shape_handle, subquery_ref}` to `{subquery_id, logical_time}` and call - # the shared `MultiTimeView` at that time. - # - # See `docs/rfcs/subquery-index.md`, sections *SubqueryIndex Data Model* - # and *Routing*. + # membership is *not* stored here, and the filter is intentionally + # time-unaware: it routes conservatively across all retained logical times + # and lets each consumer's `Shape.convert_change` do the exact check at its + # own per-subquery logical time(s). See `docs/rfcs/subquery-index.md` + # §Routing for why a per-shape logical-time pin in the filter cannot + # represent a consumer that is mid-move on a subquery (it reads at both + # `from_time` and `to_time` simultaneously). @moduledoc false import Electric, only: [is_stack_id: 1] @@ -63,19 +64,14 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end @doc """ - Register per-shape metadata: per-occurrence polarity, per-dep-index - dependency handle (a.k.a. `subquery_id`), and the fallback flag. + Register per-shape metadata: per-dep-index dependency handle (a.k.a. + `subquery_id`) and the fallback flag. `dep_handles` is the outer shape's `shape_dependencies_handles` list, indexed by `dep_index`. """ @spec register_shape(t(), term(), DnfPlan.t(), [term()]) :: :ok def register_shape(%SubqueryIndex{table: table}, shape_handle, %DnfPlan{} = plan, dep_handles) do - for {_pos, info} <- plan.positions, info.is_subquery do - polarity = if info.negated, do: :negated, else: :positive - :ets.insert(table, {{:polarity, shape_handle, info.subquery_ref}, polarity}) - end - for dep_index <- Map.keys(plan.dependency_polarities) do dep_handle = Enum.at(dep_handles, dep_index) || {shape_handle, dep_index} :ets.insert(table, {{:dep_handle, shape_handle, dep_index}, dep_handle}) @@ -88,9 +84,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @doc "Remove all metadata for `shape_handle`." @spec unregister_shape(t(), term()) :: :ok def unregister_shape(%SubqueryIndex{table: table}, shape_handle) do - :ets.match_delete(table, {{:polarity, shape_handle, :_}, :_}) :ets.match_delete(table, {{:dep_handle, shape_handle, :_}, :_}) - :ets.match_delete(table, {{:shape_subquery, shape_handle, :_}, :_}) :ets.delete(table, {:fallback, shape_handle}) :ok end @@ -203,35 +197,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end end - @doc """ - Record that `shape_handle`'s `subquery_ref` should read at `time` against - the dependency view identified by `subquery_id`. Called after the - consumer has registered with the materializer in phase 2 of the RFC. - """ - @spec set_shape_subquery(t(), term(), [String.t()], term(), non_neg_integer()) :: :ok - def set_shape_subquery( - %SubqueryIndex{table: table}, - shape_handle, - subquery_ref, - subquery_id, - time - ) do - :ets.insert(table, {{:shape_subquery, shape_handle, subquery_ref}, {subquery_id, time}}) - :ok - end - - @spec get_shape_subquery(t(), term(), [String.t()]) :: {term(), non_neg_integer()} | nil - def get_shape_subquery(%SubqueryIndex{table: table}, shape_handle, subquery_ref) do - do_get_shape_subquery(table, shape_handle, subquery_ref) - end - - defp do_get_shape_subquery(table, shape_handle, subquery_ref) do - case :ets.lookup(table, {:shape_subquery, shape_handle, subquery_ref}) do - [{_, mapping}] -> mapping - [] -> nil - end - end - @doc "Mark a shape as routable (clear fallback rows)." @spec mark_ready(t(), term()) :: :ok def mark_ready(%SubqueryIndex{table: table}, shape_handle) do @@ -363,67 +328,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end) end - @doc """ - Exact membership for `shape_handle + subquery_ref + typed_value`. Resolves - to `{subquery_id, logical_time}` and consults the shared `MultiTimeView`. - Falls back to polarity-based answer while the shape is in fallback or - before its consumer has registered a logical time. - """ - @spec membership_or_fallback?(t(), term(), [String.t()], term()) :: boolean() - def membership_or_fallback?( - %SubqueryIndex{table: table, multi_time_view: mtv}, - shape_handle, - subquery_ref, - typed_value - ) do - if shape_in_fallback?(table, shape_handle) do - polarity_default(table, shape_handle, subquery_ref) - else - case do_get_shape_subquery(table, shape_handle, subquery_ref) do - {subquery_id, time} when not is_nil(mtv) -> - MultiTimeView.member?(mtv, subquery_id, typed_value, time) - - _ -> - polarity_default(table, shape_handle, subquery_ref) - end - end - end - - @doc """ - Strict exact membership without the fallback shortcut. Returns `false` - when no logical time has been set for `{shape_handle, subquery_ref}`. - """ - @spec member?(t(), term(), [String.t()], term()) :: boolean() - def member?(%SubqueryIndex{multi_time_view: nil}, _shape_handle, _subquery_ref, _typed_value), - do: false - - def member?( - %SubqueryIndex{table: table, multi_time_view: mtv}, - shape_handle, - subquery_ref, - typed_value - ) do - case do_get_shape_subquery(table, shape_handle, subquery_ref) do - {subquery_id, time} -> MultiTimeView.member?(mtv, subquery_id, typed_value, time) - nil -> false - end - end - - defp polarity_default(table, shape_handle, subquery_ref) do - case :ets.lookup(table, {:polarity, shape_handle, subquery_ref}) do - [{_, :positive}] -> - true - - [{_, :negated}] -> - false - - [] -> - raise ArgumentError, - "missing polarity for shape #{inspect(shape_handle)} and ref " <> - inspect(subquery_ref) - end - end - defp ensure_node_meta(table, condition_id, field_key, testexpr) do case :ets.lookup(table, {:node_testexpr, condition_id, field_key}) do [] -> :ets.insert(table, {{:node_testexpr, condition_id, field_key}, testexpr}) diff --git a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex index 5e34b4831f..84825e09e0 100644 --- a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex +++ b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex @@ -339,7 +339,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do end defp other_shapes_affected( - %Filter{subquery_index: index, where_cond_table: table}, + %Filter{where_cond_table: table}, condition_id, record ) do @@ -350,7 +350,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do [shape_count: map_size(other_shapes)], fn -> for {{shape_id, _branch_key}, where} <- other_shapes, - other_shape_matches?(index, shape_id, where, record), + other_shape_matches?(where, record), into: MapSet.new() do shape_id end @@ -358,11 +358,16 @@ defmodule Electric.Shapes.Filter.WhereCondition do ) end - defp other_shape_matches?(index, shape_id, where, record) do + # The filter cannot evaluate subquery membership itself (see + # `WhereClause.subquery_member_unknown/0`). For non-subquery conjuncts in + # the residual `and_where` this still produces a precise answer; a sublink + # in the residual makes the runner error out, which we treat as "include" + # so the consumer does the exact check in `Shape.convert_change`. + defp other_shape_matches?(where, record) do case WhereClause.includes_record_result( where, record, - WhereClause.subquery_member_from_index(index, shape_id) + WhereClause.subquery_member_unknown() ) do {:ok, included?} -> included? :error -> true diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index 05439d2c07..d2fdf2edd9 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -1,7 +1,6 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner - alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( @@ -46,16 +45,21 @@ defmodule Electric.Shapes.WhereClause do end @doc """ - Build a subquery_member? callback that queries the SubqueryIndex. + Build a subquery_member? callback that signals "unknown" by raising. - Used for filter-side exact verification: checks whether a specific - shape currently contains a typed value for a canonical subquery ref. + The filter cannot answer subquery membership correctly: a consumer that is + mid-move on a subquery is effectively reading at two logical times at once, + which a single per-shape pin cannot represent. So whenever the filter is + evaluating a residual `and_where` that contains a sublink, this callback + raises. The runner catches the raise and throws `:could_not_compute`, + which propagates as `:error` from `includes_record_result/3` — the caller + treats that as "over-route, let the consumer do the exact check in + `Shape.convert_change`". """ - @spec subquery_member_from_index(SubqueryIndex.t(), term()) :: - ([String.t()], term() -> boolean()) - def subquery_member_from_index(index, shape_handle) do - fn subquery_ref, typed_value -> - SubqueryIndex.membership_or_fallback?(index, shape_handle, subquery_ref, typed_value) + @spec subquery_member_unknown() :: ([String.t()], term() -> boolean()) + def subquery_member_unknown do + fn _subquery_ref, _typed_value -> + raise "subquery membership cannot be evaluated at the filter; consumer must check" end end diff --git a/packages/sync-service/test/electric/shapes/consumer_test.exs b/packages/sync-service/test/electric/shapes/consumer_test.exs index f13ab14c89..e53b9c2627 100644 --- a/packages/sync-service/test/electric/shapes/consumer_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer_test.exs @@ -2265,6 +2265,7 @@ defmodule Electric.Shapes.ConsumerTest do test "consumer startup registers with the stack-scoped subquery index", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView {shape_handle, _} = ShapeCache.get_or_create_shape_handle(@shape_with_subquery, ctx.stack_id) @@ -2276,21 +2277,22 @@ defmodule Electric.Shapes.ConsumerTest do # Filter.add_shape registers the outer shape with the index. After # the consumer's initial materialisation lands, the shape is no - # longer in fallback (its subquery_ref is bound to the dep's - # subquery_id at the materializer's current logical time). + # longer in fallback, the dep view is ready at logical time 0, and + # the dep handle has been recorded for the shape. assert SubqueryIndex.has_positions?(index, shape_handle) refute SubqueryIndex.fallback?(index, shape_handle) {:ok, shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) [dep_handle] = shape.shape_dependencies_handles - assert {^dep_handle, 0} = - SubqueryIndex.get_shape_subquery(index, shape_handle, ["$sublink", "0"]) + assert MultiTimeView.ready?(index.multi_time_view, dep_handle) + assert MultiTimeView.current_time(index.multi_time_view, dep_handle) == 0 end test "consumer steady dependency move_in advances the shape's subquery index time", ctx do alias Electric.Shapes.Filter.Indexes.SubqueryIndex + alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView parent = self() @@ -2312,11 +2314,12 @@ defmodule Electric.Shapes.ConsumerTest do :started = ShapeCache.await_snapshot_start(shape_handle, ctx.stack_id) index = SubqueryIndex.for_stack(ctx.stack_id) - {:ok, _shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + mtv = index.multi_time_view + {:ok, shape} = Electric.Shapes.fetch_shape_by_handle(ctx.stack_id, shape_handle) + [dep_handle] = shape.shape_dependencies_handles - # Before any dep moves: shape's subquery_ref is bound at logical - # time 0 and the dep view is empty, so no value is a member yet. - refute SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # Before any dep moves: dep view is empty at logical time 0. + refute MultiTimeView.member?(mtv, dep_handle, 1, 0) # Send a new record for the dependency table to trigger a move_in ShapeLogCollector.handle_event( @@ -2332,10 +2335,9 @@ defmodule Electric.Shapes.ConsumerTest do assert_receive {:query_requested, consumer_pid} - # While the move-in query is buffering, the consumer's subquery - # time is still at 0, so the new value is not yet visible through - # `member?/4` (which reads at the shape's stored logical time). - refute SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # While the move-in query is buffering, the dep view at time 0 is + # unchanged — value 1 is still not a member there. + refute MultiTimeView.member?(mtv, dep_handle, 1, 0) send(consumer_pid, {:pg_snapshot_known, {100, 300, []}}) @@ -2365,9 +2367,11 @@ defmodule Electric.Shapes.ConsumerTest do ref = Shapes.Consumer.register_for_changes(ctx.stack_id, shape_handle) assert_receive {^ref, :new_changes, _offset}, @receive_timeout - # After splice the shape's subquery time has advanced past the - # move-in, and value 1 is now visible at the shape's stored time. - assert SubqueryIndex.member?(index, shape_handle, ["$sublink", "0"], 1) + # After splice the materializer's logical time has advanced past + # the move-in, and value 1 is now visible at the new current time. + current = MultiTimeView.current_time(mtv, dep_handle) + assert current > 0 + assert MultiTimeView.member?(mtv, dep_handle, 1, current) end test "consumer cleanup removes shape rows from the subquery index", ctx do diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs index 8d0adb545e..41e2918d9a 100644 --- a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs @@ -32,7 +32,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do end describe "register_shape/4" do - test "stores polarity per subquery_ref and marks the shape as fallback", %{index: index} do + test "marks the shape as fallback for both polarities", %{index: index} do SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) assert SubqueryIndex.fallback?(index, "s1") @@ -46,27 +46,10 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do assert SubqueryIndex.fallback?(index, "s2") end - - test "membership_or_fallback? defaults to true for positive fallback", %{index: index} do - SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) - - assert SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) - end - - test "membership_or_fallback? defaults to false for negated fallback", %{index: index} do - SubqueryIndex.register_shape( - index, - "s1", - make_plan(polarity: :negated), - [@dep_handle_a] - ) - - refute SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) - end end describe "unregister_shape/2" do - test "drops polarity, dep_handle, and fallback rows", %{index: index} do + test "drops dep_handle and fallback rows", %{index: index} do SubqueryIndex.register_shape(index, "s1", make_plan(), [@dep_handle_a]) SubqueryIndex.unregister_shape(index, "s1") @@ -170,7 +153,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do end describe "affected_shapes/4 (positive routing)" do - test "returns shapes whose subquery has the value at the shape's logical time", %{ + test "routes shapes by value-keyed positive routing", %{ filter: filter, index: index, mtv: mtv, @@ -180,7 +163,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "s1") - SubqueryIndex.set_shape_subquery(index, "s1", @subquery_ref, @dep_handle_a, 0) SubqueryIndex.mark_ready(index, "s1") assert MapSet.new(["s1"]) == @@ -192,12 +174,18 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do ) end - test "diverging consumer times: routing keeps the value, exact check splits the result", %{ + test "diverging consumer times: filter over-routes both shapes", %{ filter: filter, index: index, mtv: mtv, condition_id: condition_id } do + # The filter is time-unaware by design: it routes on the retained + # window, so a value that is a member at *some* time fans out to every + # attached shape. Per-consumer correctness is left to + # `Shape.convert_change`, which evaluates membership at the consumer's + # own logical time(s) — including both `from_time` and `to_time` + # during a buffered move. MultiTimeView.init_subquery(mtv, @dep_handle_a, []) MultiTimeView.mark_ready(mtv, @dep_handle_a) @@ -207,23 +195,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) SubqueryIndex.add_positive_route(index, @dep_handle_a, 30) - SubqueryIndex.set_shape_subquery(index, "s_old", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.set_shape_subquery(index, "s_new", @subquery_ref, @dep_handle_a, 1) SubqueryIndex.mark_ready(index, "s_old") SubqueryIndex.mark_ready(index, "s_new") - affected = - SubqueryIndex.affected_shapes( - filter, - condition_id, - @field, - %{"par_id" => "30"} - ) - - assert MapSet.new(["s_old", "s_new"]) == affected - - refute SubqueryIndex.member?(index, "s_old", @subquery_ref, 30) - assert SubqueryIndex.member?(index, "s_new", @subquery_ref, 30) + assert MapSet.new(["s_old", "s_new"]) == + SubqueryIndex.affected_shapes( + filter, + condition_id, + @field, + %{"par_id" => "30"} + ) end test "returns only shapes registered under the requested field key", %{ @@ -247,16 +228,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do SubqueryIndex.mark_ready(index, shape) end - SubqueryIndex.set_shape_subquery(index, "local", @subquery_ref, @dep_handle_a, 0) - - SubqueryIndex.set_shape_subquery( - index, - "other_field", - @other_subquery_ref, - @dep_handle_a, - 0 - ) - assert MapSet.new(["local"]) == SubqueryIndex.affected_shapes( filter, @@ -279,7 +250,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do and_where: where("name ILIKE 'keep%'", %{["name"] => :text}) ) - SubqueryIndex.set_shape_subquery(index, "tail", @subquery_ref, @dep_handle_a, 0) SubqueryIndex.mark_ready(index, "tail") assert MapSet.new(["tail"]) == @@ -326,7 +296,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do MultiTimeView.mark_ready(mtv, @dep_handle_a) register_node_shape(filter, condition_id, "n1", polarity: :negated) - SubqueryIndex.set_shape_subquery(index, "n1", @subquery_ref, @dep_handle_a, 0) SubqueryIndex.mark_ready(index, "n1") assert MapSet.new() == @@ -358,7 +327,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "n1", polarity: :negated) MultiTimeView.mark_in(mtv, @dep_handle_a, 30, 1) - SubqueryIndex.set_shape_subquery(index, "n1", @subquery_ref, @dep_handle_a, 0) SubqueryIndex.mark_ready(index, "n1") assert MapSet.new(["n1"]) == @@ -532,29 +500,6 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do end end - describe "member?/5 and membership_or_fallback?/5" do - test "membership_or_fallback? defers to MultiTimeView at the shape's logical time", %{ - filter: filter, - index: index, - mtv: mtv, - condition_id: condition_id - } do - MultiTimeView.init_subquery(mtv, @dep_handle_a, [5]) - MultiTimeView.mark_ready(mtv, @dep_handle_a) - - register_node_shape(filter, condition_id, "s1") - SubqueryIndex.set_shape_subquery(index, "s1", @subquery_ref, @dep_handle_a, 0) - SubqueryIndex.mark_ready(index, "s1") - - assert SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 5) - refute SubqueryIndex.membership_or_fallback?(index, "s1", @subquery_ref, 99) - end - - test "member? without a stored logical time returns false", %{index: index} do - refute SubqueryIndex.member?(index, "no_such_shape", @subquery_ref, 1) - end - end - describe "for_stack/1" do test "returns the index when one was created for the stack" do stack_id = "test-stack-#{System.unique_integer([:positive])}" diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index 10e2d79519..26c2973b3b 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -924,9 +924,8 @@ defmodule Electric.Shapes.FilterTest do # `Filter.add_shape` stored for the shape's dependency (handles either # a real dep_handle or the `{shape_handle, dep_index}` fallback that # `register_shape` falls back to when `shape_dependencies_handles` - # isn't populated), seeds `MultiTimeView` with the values, binds the - # shape's `subquery_ref` at logical time 0, and wires positive routing - # rows so `affected_shapes/2` can find each value. + # isn't populated), seeds `MultiTimeView` with the values, and wires + # positive routing rows so `affected_shapes/2` can find each value. defp seed_membership(filter, shape_id, _shape, subquery_ref, values) do index = Filter.subquery_index(filter) mtv = index.multi_time_view @@ -936,8 +935,6 @@ defmodule Electric.Shapes.FilterTest do MultiTimeView.init_subquery(mtv, subquery_id, values) MultiTimeView.mark_ready(mtv, subquery_id) - :ok = SubqueryIndex.set_shape_subquery(index, shape_id, subquery_ref, subquery_id, 0) - for v <- values do :ok = SubqueryIndex.add_positive_route(index, subquery_id, v) end @@ -1177,9 +1174,15 @@ defmodule Electric.Shapes.FilterTest do "CREATE TABLE IF NOT EXISTS or_parent (id INT PRIMARY KEY)", "CREATE TABLE IF NOT EXISTS or_child (id INT PRIMARY KEY, par_id INT REFERENCES or_parent(id), value TEXT NOT NULL)" ] - test "mixed OR+subquery shape falls back to other_shapes verification", %{ + test "mixed OR+subquery shape over-routes through other_shapes", %{ inspector: inspector } do + # A mixed OR shape lands in the catch-all `other_shapes` path because + # neither side of the OR is individually optimisable. The filter + # cannot evaluate the sublink (it would need a consumer's logical + # time, which it intentionally doesn't track), so any record reaches + # the shape and the consumer's `Shape.convert_change` makes the final + # decision. {:ok, shape} = Shape.new("or_child", inspector: inspector, @@ -1217,7 +1220,10 @@ defmodule Electric.Shapes.FilterTest do record: %{"id" => "50", "par_id" => "99", "value" => "other"} } - assert Filter.affected_shapes(filter, insert_no_match) == MapSet.new([]) + # Over-routed: the filter cannot resolve the sublink at the OR's left + # branch without a per-consumer logical time, so it includes shape1 + # and lets the consumer drop the record. + assert Filter.affected_shapes(filter, insert_no_match) == MapSet.new(["shape1"]) end @tag with_sql: [ @@ -1351,11 +1357,10 @@ defmodule Electric.Shapes.FilterTest do SubqueryIndex.mark_ready(index, "shape1") # Before remove, the shape routes records whose value is in the - # subquery view, and the shape is bound to the dep's subquery_id. + # subquery view. hit = %NewRecord{relation: {"public", "child"}, record: %{"id" => "1", "par_id" => "9"}} assert Filter.affected_shapes(filter, hit) == MapSet.new(["shape1"]) assert SubqueryIndex.has_positions?(index, "shape1") - assert SubqueryIndex.get_shape_subquery(index, "shape1", subquery_ref) != nil Filter.remove_shape(filter, "shape1") @@ -1363,7 +1368,6 @@ defmodule Electric.Shapes.FilterTest do # are gone. assert Filter.affected_shapes(filter, hit) == MapSet.new([]) refute SubqueryIndex.has_positions?(index, "shape1") - assert SubqueryIndex.get_shape_subquery(index, "shape1", subquery_ref) == nil end @tag with_sql: [ From 2cbe984e416c22fcdb99668f97ff55aed47350a2 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 26 May 2026 09:55:00 +0100 Subject: [PATCH 31/40] Use bag-keyed routing rows in SubqueryIndex Multi-valued routes were stored as `{:positive, group_id, value, cnid}` in a `:set` ETS table and read with partial `:ets.match`, forcing a full-table scan on the hot routing path. Switch the table to `:bag`, move the discriminator (cnid, shape, branch) into the value position for the six multi-value row types, and read with exact `:ets.lookup`. Targeted deletes now use `:ets.delete_object` since `:ets.delete/2` on a bag wipes every row with the key. `lookup_child_for_shape` walks the shape's own attachments instead of the whole group. Matches the RFC's `{:positive, group_id, value} -> child_node_id` cost model. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../shapes/filter/indexes/subquery_index.ex | 114 ++++++++---------- .../filter/indexes/subquery_index_test.exs | 32 ++--- 2 files changed, 69 insertions(+), 77 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index a426061f00..0364477ce7 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -35,11 +35,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do table = case Keyword.get(opts, :stack_id) do nil -> - :ets.new(:subquery_index, [:set, :public]) + :ets.new(:subquery_index, [:bag, :public]) stack_id -> try do - :ets.new(table_name(stack_id), [:set, :public, :named_table]) + :ets.new(table_name(stack_id), [:bag, :public, :named_table]) rescue ArgumentError -> table_name(stack_id) end @@ -134,13 +134,13 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do branch_key ) - :ets.insert(table, {{:child_shape, child_node_id, shape_handle, branch_key}, true}) - :ets.insert(table, {{:shape_child, shape_handle, child_node_id, branch_key}, true}) + :ets.insert(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) + :ets.insert(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) if shape_in_fallback?(table, shape_handle) do :ets.insert( table, - {{:node_fallback, condition_id, optimisation.field, child_node_id, shape_handle}, true} + {{:node_fallback, condition_id, optimisation.field}, {child_node_id, shape_handle}} ) end @@ -181,12 +181,12 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do branch_key ) - :ets.delete(table, {:child_shape, child_node_id, shape_handle, branch_key}) - :ets.delete(table, {:shape_child, shape_handle, child_node_id, branch_key}) + :ets.delete_object(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) + :ets.delete_object(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) - :ets.match_delete( + :ets.delete_object( table, - {{:node_fallback, condition_id, optimisation.field, child_node_id, shape_handle}, :_} + {{:node_fallback, condition_id, optimisation.field}, {child_node_id, shape_handle}} ) if child_empty?(table, child_node_id) do @@ -201,7 +201,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @spec mark_ready(t(), term()) :: :ok def mark_ready(%SubqueryIndex{table: table}, shape_handle) do :ets.delete(table, {:fallback, shape_handle}) - :ets.match_delete(table, {{:node_fallback, :_, :_, :_, shape_handle}, :_}) + :ets.match_delete(table, {{:node_fallback, :_, :_}, {:_, shape_handle}}) :ok end @@ -215,7 +215,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do @doc "Whether `shape_handle` is attached to at least one indexed subquery node." @spec has_positions?(t(), term()) :: boolean() def has_positions?(%SubqueryIndex{table: table}, shape_handle) do - :ets.match(table, {{:shape_child, shape_handle, :_, :_}, :_}, 1) != :"$end_of_table" + :ets.member(table, {:shape_child, shape_handle}) end @doc """ @@ -228,7 +228,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do for child_node_id <- children_for_subquery(table, subquery_id) do case :ets.lookup(table, {:child_meta, child_node_id}) do [{_, %{polarity: :positive, group_id: group_id}}] -> - :ets.insert(table, {{:positive, group_id, value, child_node_id}, true}) + :ets.insert(table, {{:positive, group_id, value}, child_node_id}) _ -> :ok @@ -248,7 +248,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do for child_node_id <- children_for_subquery(table, subquery_id) do case :ets.lookup(table, {:child_meta, child_node_id}) do [{_, %{polarity: :positive, group_id: group_id}}] -> - :ets.delete(table, {:positive, group_id, value, child_node_id}) + :ets.delete_object(table, {{:positive, group_id, value}, child_node_id}) _ -> :ok @@ -374,7 +374,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.insert(table, {{:child, group_id, subquery_id}, child_node_id}) :ets.insert(table, {{:child_meta, child_node_id}, meta}) - :ets.insert(table, {{:subquery_child, subquery_id, child_node_id}, true}) + :ets.insert(table, {{:subquery_child, subquery_id}, child_node_id}) seed_child_routing(table, mtv, child_node_id, meta) {child_node_id, next_condition_id} @@ -382,9 +382,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end defp children_for_subquery(table, subquery_id) do - table - |> :ets.match({{:subquery_child, subquery_id, :"$1"}, :_}) - |> Enum.map(fn [cnid] -> cnid end) + for {_, cnid} <- :ets.lookup(table, {:subquery_child, subquery_id}), do: cnid end defp seed_child_routing(_table, nil, _child_node_id, _meta), do: :ok @@ -400,7 +398,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do time -> for value <- MultiTimeView.values(mtv, subquery_id, time) do - :ets.insert(table, {{:positive, group_id, value, child_node_id}, true}) + :ets.insert(table, {{:positive, group_id, value}, child_node_id}) end :ok @@ -411,7 +409,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do polarity: :negated, group_id: group_id }) do - :ets.insert(table, {{:negated, group_id, child_node_id}, true}) + :ets.insert(table, {{:negated, group_id}, child_node_id}) :ok end @@ -428,32 +426,30 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end defp lookup_child_for_shape(table, condition_id, field_key, polarity, shape_handle, branch_key) do - case :ets.lookup(table, {:group, condition_id, field_key, polarity}) do - [{_, group_id}] -> - children = - table - |> :ets.match({{:child, group_id, :_}, :"$1"}) - |> Enum.map(fn [cnid] -> cnid end) - - child_node_id = - Enum.find(children, fn cnid -> - :ets.member(table, {:shape_child, shape_handle, cnid, branch_key}) - end) - - if child_node_id do - [{_, %{next_condition_id: next_condition_id}}] = - :ets.lookup(table, {:child_meta, child_node_id}) - - {child_node_id, next_condition_id} - end - - [] -> - nil + with [{_, group_id}] <- :ets.lookup(table, {:group, condition_id, field_key, polarity}), + {cnid, next} when not is_nil(cnid) <- + Enum.find_value( + :ets.lookup(table, {:shape_child, shape_handle}), + {nil, nil}, + fn + {_, {cnid, ^branch_key}} -> + case :ets.lookup(table, {:child_meta, cnid}) do + [{_, %{group_id: ^group_id, next_condition_id: next}}] -> {cnid, next} + _ -> nil + end + + _ -> + nil + end + ) do + {cnid, next} + else + _ -> nil end end defp child_empty?(table, child_node_id) do - :ets.match(table, {{:child_shape, child_node_id, :_, :_}, :_}) == [] + not :ets.member(table, {:child_shape, child_node_id}) end defp delete_child(table, mtv, child_node_id) do @@ -466,17 +462,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :positive -> if mtv != nil do for value <- MultiTimeView.values(mtv, meta.subquery_id) do - :ets.delete(table, {:positive, meta.group_id, value, child_node_id}) + :ets.delete_object(table, {{:positive, meta.group_id, value}, child_node_id}) end end :negated -> - :ets.delete(table, {:negated, meta.group_id, child_node_id}) + :ets.delete_object(table, {{:negated, meta.group_id}, child_node_id}) end - :ets.match_delete(table, {{:node_fallback, :_, :_, child_node_id, :_}, :_}) + :ets.match_delete(table, {{:node_fallback, :_, :_}, {child_node_id, :_}}) :ets.delete(table, {:child, meta.group_id, meta.subquery_id}) - :ets.delete(table, {:subquery_child, meta.subquery_id, child_node_id}) + :ets.delete_object(table, {{:subquery_child, meta.subquery_id}, child_node_id}) :ets.delete(table, {:child_meta, child_node_id}) if group_empty?(table, meta.group_id) do @@ -495,10 +491,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end defp cleanup_child_shapes(table, child_node_id) do - for [shape_handle, branch_key] <- - :ets.match(table, {{:child_shape, child_node_id, :"$1", :"$2"}, :_}) do - :ets.delete(table, {:shape_child, shape_handle, child_node_id, branch_key}) - :ets.delete(table, {:child_shape, child_node_id, shape_handle, branch_key}) + for {_, {shape_handle, branch_key}} <- :ets.lookup(table, {:child_shape, child_node_id}) do + :ets.delete_object(table, {{:shape_child, shape_handle}, {child_node_id, branch_key}}) + :ets.delete_object(table, {{:child_shape, child_node_id}, {shape_handle, branch_key}}) end end @@ -516,10 +511,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do MapSet.new() [{_, group_id}] -> - table - |> :ets.match({{:positive, group_id, value, :"$1"}, :_}) - |> Enum.map(fn [cnid] -> cnid end) - |> MapSet.new() + for {_, cnid} <- :ets.lookup(table, {:positive, group_id, value}), + into: MapSet.new(), + do: cnid end end @@ -529,11 +523,10 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do MapSet.new() [{_, group_id}] -> - for [cnid] <- :ets.match(table, {{:negated, group_id, :"$1"}, :_}), + for {_, cnid} <- :ets.lookup(table, {:negated, group_id}), keep_negated_child?(table, mtv, cnid, value), - into: MapSet.new() do - cnid - end + into: MapSet.new(), + do: cnid end end @@ -550,10 +543,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end defp fallback_children(table, condition_id, field_key) do - table - |> :ets.match({{:node_fallback, condition_id, field_key, :"$1", :_}, :_}) - |> Enum.map(fn [cnid] -> cnid end) - |> MapSet.new() + for {_, {cnid, _shape}} <- :ets.lookup(table, {:node_fallback, condition_id, field_key}), + into: MapSet.new(), + do: cnid end defp all_children(table, condition_id, field_key) do diff --git a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs index 41e2918d9a..4317ddc518 100644 --- a/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs +++ b/packages/sync-service/test/electric/shapes/filter/indexes/subquery_index_test.exs @@ -77,8 +77,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "s2") children = - :ets.match(table, {{:shape_child, "s1", :"$1", :_}, :_}) ++ - :ets.match(table, {{:shape_child, "s2", :"$1", :_}, :_}) + :ets.match(table, {{:shape_child, "s1"}, {:"$1", :_}}) ++ + :ets.match(table, {{:shape_child, "s2"}, {:"$1", :_}}) assert children |> Enum.uniq() |> length() == 1 end @@ -92,8 +92,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "s2", dep_handles: [@dep_handle_b]) - [[c1]] = :ets.match(table, {{:shape_child, "s1", :"$1", :_}, :_}) - [[c2]] = :ets.match(table, {{:shape_child, "s2", :"$1", :_}, :_}) + [[c1]] = :ets.match(table, {{:shape_child, "s1"}, {:"$1", :_}}) + [[c2]] = :ets.match(table, {{:shape_child, "s2"}, {:"$1", :_}}) assert c1 != c2 end @@ -112,7 +112,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do [[_, group_id]] = :ets.match(table, {{:group, condition_id, @field, :"$1"}, :"$2"}) - assert :ets.match(table, {{:positive, group_id, :"$1", :_}, :_}) |> Enum.sort() == + assert :ets.match(table, {{:positive, group_id, :"$1"}, :_}) |> Enum.sort() == [[10], [20]] |> Enum.sort() end @@ -127,9 +127,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "s1") - before_routes = :ets.match(table, {{:positive, :_, :_, :_}, :_}) + before_routes = :ets.match(table, {{:positive, :_, :_}, :_}) register_node_shape(filter, condition_id, "s2") - after_routes = :ets.match(table, {{:positive, :_, :_, :_}, :_}) + after_routes = :ets.match(table, {{:positive, :_, :_}, :_}) assert Enum.sort(before_routes) == Enum.sort(after_routes) end @@ -147,8 +147,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "n1", polarity: :negated) - assert :ets.match(table, {{:negated, :_, :_}, :_}) |> length() == 1 - assert :ets.match(table, {{:positive, :_, :_, :_}, :_}) == [] + assert :ets.match(table, {{:negated, :_}, :_}) |> length() == 1 + assert :ets.match(table, {{:positive, :_, :_}, :_}) == [] end end @@ -373,17 +373,17 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do register_node_shape(filter, condition_id, "s1") - shape_rows_before = :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) + shape_rows_before = :ets.lookup(table, {:shape_child, "s1"}) SubqueryIndex.add_positive_route(index, @dep_handle_a, 42) [[group_id]] = :ets.match(table, {{:group, condition_id, @field, :positive}, :"$1"}) - assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) |> length() == 1 + assert :ets.lookup(table, {:positive, group_id, 42}) |> length() == 1 SubqueryIndex.remove_positive_route(index, @dep_handle_a, 42) - assert :ets.match(table, {{:positive, group_id, 42, :_}, :_}) == [] + assert :ets.lookup(table, {:positive, group_id, 42}) == [] - assert :ets.match(table, {{:shape_child, "s1", :_, :_}, :_}) == shape_rows_before + assert :ets.lookup(table, {:shape_child, "s1"}) == shape_rows_before end end @@ -421,7 +421,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do assert :deleted = SubqueryIndex.remove_shape(filter, condition_id, "s2", subquery_optimisation(), []) - assert :ets.match(table, {{:positive, :_, :_, :_}, :_}) == [] + assert :ets.match(table, {{:positive, :_, :_}, :_}) == [] assert :ets.match(table, {{:child_meta, :_}, :_}) == [] end @@ -491,12 +491,12 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndexTest do } do register_node_shape(filter, condition_id, "s1") assert SubqueryIndex.fallback?(index, "s1") - assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) != [] + assert :ets.match(table, {{:node_fallback, :_, :_}, {:_, "s1"}}) != [] SubqueryIndex.mark_ready(index, "s1") refute SubqueryIndex.fallback?(index, "s1") - assert :ets.match(table, {{:node_fallback, :_, :_, :_, "s1"}, :_}) == [] + assert :ets.match(table, {{:node_fallback, :_, :_}, {:_, "s1"}}) == [] end end From ae6957783ae4702cc5646a004248dd7044ed9ae5 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 26 May 2026 15:22:41 +0100 Subject: [PATCH 32/40] BENCHMARKS: add benchmarks --- .../bench/subquery_index_bench.exs | 822 ++++++++++++++++++ packages/sync-service/mix.exs | 1 + packages/sync-service/mix.lock | 3 + .../scripts/subquery_logical_time_memory.exs | 310 +++++++ 4 files changed, 1136 insertions(+) create mode 100644 packages/sync-service/bench/subquery_index_bench.exs create mode 100644 packages/sync-service/scripts/subquery_logical_time_memory.exs diff --git a/packages/sync-service/bench/subquery_index_bench.exs b/packages/sync-service/bench/subquery_index_bench.exs new file mode 100644 index 0000000000..cb71fc7e12 --- /dev/null +++ b/packages/sync-service/bench/subquery_index_bench.exs @@ -0,0 +1,822 @@ +# SubqueryIndex latency benchmarks. +# +# Run from packages/sync-service: +# +# mix run --no-start bench/subquery_index_bench.exs +# +# `--no-start` skips the sync-service application (no replication client, +# admission control, etc.) which keeps the bench output focused. +# +# Each benchmark group sweeps a single size dimension (values, history +# length, children, participants) so the Benchee output makes the scaling +# behaviour directly visible. The RFC at docs/rfcs/subquery-index.md states +# expected complexity for every operation here — see the comment above each +# `Benchee.run/2` call for the expected curve. +# +# The "add_shape: new subquery → triggers materializer" case from the task +# brief is intentionally not benchmarked here. That cost is paid in the +# materializer's initial population, which lives outside SubqueryIndex's +# responsibility and which the RFC explicitly acknowledges cannot be O(1). + +alias Electric.Replication.Eval.Parser.{Func, Ref} +alias Electric.Shapes.DnfPlan +alias Electric.Shapes.Filter +alias Electric.Shapes.Filter.Indexes.SubqueryIndex +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor +alias Electric.Shapes.Filter.WhereCondition + +defmodule Bench.Pop do + @moduledoc false + + @field "par_id" + @subquery_ref ["$sublink", "0"] + + def field, do: @field + def subquery_ref, do: @subquery_ref + + def make_plan(opts \\ []) do + polarity = Keyword.get(opts, :polarity, :positive) + dep_index = Keyword.get(opts, :dep_index, 0) + subquery_ref = Keyword.get(opts, :subquery_ref, @subquery_ref) + field = Keyword.get(opts, :field, @field) + + testexpr = %Ref{path: [field], type: :int8} + ref = %Ref{path: subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: dep_index, + subquery_ref: subquery_ref, + tag_columns: [field] + } + }, + dependency_positions: %{dep_index => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{dep_index => polarity} + } + end + + def subquery_optimisation(opts \\ []) do + field = Keyword.get(opts, :field, @field) + + %{ + operation: "subquery", + field: field, + testexpr: %Ref{path: [field], type: :int8}, + subquery_ref: Keyword.get(opts, :subquery_ref, @subquery_ref), + dep_index: Keyword.get(opts, :dep_index, 0), + polarity: Keyword.get(opts, :polarity, :positive), + and_where: Keyword.get(opts, :and_where) + } + end + + @doc """ + Build a fresh Filter with `subquery_count` subqueries, `values_per_subquery` + values each, attached at one shared condition_id. Returns + `{filter, condition_id, dep_handles, shape_ids}`. + """ + def build(opts) do + subquery_count = Keyword.get(opts, :subquery_count, 1) + values_per_subquery = Keyword.get(opts, :values_per_subquery, 1) + shapes_per_subquery = Keyword.get(opts, :shapes_per_subquery, 1) + polarity = Keyword.get(opts, :polarity, :positive) + + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + + dep_handles = + for i <- 0..(subquery_count - 1) do + dep = "dep_#{i}" + values = for v <- 0..(values_per_subquery - 1), do: i * 10_000_000 + v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + dep + end + + shape_ids = + for i <- 0..(subquery_count - 1), + dep = Enum.at(dep_handles, i), + j <- 0..(shapes_per_subquery - 1) do + shape_id = "shape_#{i}_#{j}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: polarity), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(polarity: polarity), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + shape_id + end + + {filter, condition_id, dep_handles, shape_ids} + end + + @doc """ + Build a filter where one subquery has `child_count` positive children, each + in a distinct group (distinct condition_id), with one shape per child. Used + for benchmarks that sweep "positive_children_for_subquery". + """ + def build_n_positive_children(child_count, opts \\ []) do + values = Keyword.get(opts, :values, [1, 2]) + polarity = Keyword.get(opts, :polarity, :positive) + filter = Filter.new() + index = filter.subquery_index + dep = "dep_shared" + + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + condition_ids = + for i <- 0..(child_count - 1) do + cid = make_ref() + WhereCondition.init(filter, cid) + shape_id = "child_#{i}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: polarity), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + cid, + shape_id, + subquery_optimisation(polarity: polarity), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + cid + end + + {filter, condition_ids, dep} + end + + @doc """ + Build a filter where one *negated* group has `child_count` children, each + on a distinct subquery (distinct dep_handle). Used to sweep + "negated_children_in_group". + """ + def build_n_negated_children_same_group(child_count) do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + + deps = + for i <- 0..(child_count - 1) do + dep = "dep_neg_#{i}" + # Each subquery contains one member at all times (so the negated + # routing path still has to consult MTV.member_at_all_times? on + # the query value; we use a different query value below). + MultiTimeView.init_subquery(index.multi_time_view, dep, [42]) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + shape_id = "neg_#{i}" + + SubqueryIndex.register_shape( + index, + shape_id, + make_plan(polarity: :negated, dep_index: 0), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + subquery_optimisation(polarity: :negated), + [] + ) + + SubqueryIndex.mark_ready(index, shape_id) + dep + end + + {filter, condition_id, deps} + end + + @doc """ + Build a `History.t/0` of length `n` toggling per logical time. Returns + the produced history list and the final logical time. + """ + def build_history(view, dep, value, n) do + for t <- 1..n do + if rem(t, 2) == 0 do + MultiTimeView.mark_out(view, dep, value, t) + else + MultiTimeView.mark_in(view, dep, value, t) + end + end + + {:ok, n} + end +end + +# ============================================================================ +# Routing hot path +# ============================================================================ + +IO.puts("\n\n# ========== Routing hot path ==========\n") + +# affected_shapes/4 — positive group +# Sweep: values_in_subquery. Expected: ~O(1) per call (value-keyed lookup). +Benchee.run( + %{ + "affected_shapes (positive)" => fn {filter, condition_id, _deps, _shapes} -> + SubqueryIndex.affected_shapes(filter, condition_id, Bench.Pop.field(), %{ + "par_id" => "42" + }) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values", n} + end, + before_scenario: fn n -> + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# affected_shapes/4 — negated group +# Sweep: negated_children_in_group. Expected: O(N) (RFC explicitly accepts). +Benchee.run( + %{ + "affected_shapes (negated)" => fn {filter, condition_id, _deps} -> + SubqueryIndex.affected_shapes(filter, condition_id, Bench.Pop.field(), %{ + "par_id" => "999" + }) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} negated children", n} + end, + before_scenario: &Bench.Pop.build_n_negated_children_same_group/1, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# MultiTimeView.member?/4 +# Sweep: history_length_for_value. Expected: O(history) — list walk. +Benchee.run( + %{ + "MultiTimeView.member?" => fn {view, dep, value, t} -> + MultiTimeView.member?(view, dep, value, t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_scenario: fn n -> + view = MultiTimeView.new() + dep = "dep" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + # Ask at the middle of the history so we don't always short-circuit at + # the head. + {view, dep, 42, div(n, 2)} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Subquery lifecycle +# ============================================================================ + +IO.puts("\n\n# ========== Subquery lifecycle ==========\n") + +# MultiTimeView.mark_ready/2 — expected O(1). +Benchee.run( + %{ + "MultiTimeView.mark_ready" => fn {view, dep} -> + MultiTimeView.mark_ready(view, dep) + end + }, + inputs: %{"single" => :only}, + before_each: fn :only -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, [1]) + {view, dep} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# SubqueryIndex.remove_subquery/2 — sweep values × 1 child. +# Expected: O(values + children) for *this* subquery, no scan of unrelated. +Benchee.run( + %{ + "remove_subquery" => fn {index, dep} -> + SubqueryIndex.remove_subquery(index, dep) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values", n} + end, + before_each: fn n -> + {filter, _cid, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter.subquery_index, dep} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Shape lifecycle +# ============================================================================ + +IO.puts("\n\n# ========== Shape lifecycle ==========\n") + +# add_shape — existing child in existing group (additional shape sharing). +# Expected: O(1) per call. +Benchee.run( + %{ + "add_shape (existing child)" => fn {filter, condition_id, dep, n} -> + shape_id = "extra_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + {filter, condition_id, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter, condition_id, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_shape — new child, MTV ready: must seed positive routing from MTV. +# Expected: O(values_in_subquery) one-off. +Benchee.run( + %{ + "add_shape (new child, MTV ready, seeds routing)" => fn {filter, dep, n} -> + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + + shape_id = "first_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + filter = Filter.new() + index = filter.subquery_index + dep = "dep" + values = for v <- 0..(n - 1), do: v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + {filter, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_shape — new child, MTV NOT ready (fallback path, no seeding). +# Expected: O(1) — no value walk. +Benchee.run( + %{ + "add_shape (new child, MTV not ready, fallback)" => fn {filter, dep, n} -> + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + + shape_id = "fb_#{n}" + + SubqueryIndex.register_shape( + filter.subquery_index, + shape_id, + Bench.Pop.make_plan(), + [dep] + ) + + SubqueryIndex.add_shape( + filter, + condition_id, + shape_id, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery (no MTV current_time)", n} + end, + before_each: fn n -> + filter = Filter.new() + dep = "dep" + # Deliberately do NOT call init_subquery — current_time(view, dep) == nil + # forces the fallback path in seed_child_routing. + _ = n + {filter, dep, n} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# remove_shape — other shapes remain on the child. +# Expected: O(participants_for_shape) — does NOT walk subquery values. +Benchee.run( + %{ + "remove_shape (other shapes remain)" => fn {filter, condition_id, shape_to_remove} -> + SubqueryIndex.remove_shape( + filter, + condition_id, + shape_to_remove, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + # Two shapes on the same child, so removing one keeps the child alive. + {filter, condition_id, _deps, [s1, _s2]} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 2) + + {filter, condition_id, s1} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# remove_shape — last shape on the child (collapses child, drops routing). +# Expected: O(values_in_subquery) — must remove every positive route row. +Benchee.run( + %{ + "remove_shape (last on child, drops routes)" => fn {filter, condition_id, shape_to_remove} -> + SubqueryIndex.remove_shape( + filter, + condition_id, + shape_to_remove, + Bench.Pop.subquery_optimisation(), + [] + ) + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} values in subquery", n} + end, + before_each: fn n -> + {filter, condition_id, _deps, [s1]} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + {filter, condition_id, s1} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Value changes (materializer-driven) +# ============================================================================ + +IO.puts("\n\n# ========== Value changes ==========\n") + +# MultiTimeView.mark_in/4 — first time vs extending history. +Benchee.run( + %{ + "MultiTimeView.mark_in (extend existing history)" => fn {view, dep, value, next_t} -> + MultiTimeView.mark_in(view, dep, value, next_t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_each: fn n -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + # Next history toggle. If n is odd, value is currently :in, so the next + # mark_in is a no-op; bias to even so mark_in actually appends. + next_t = n + if(rem(n, 2) == 0, do: 1, else: 2) + {view, dep, 42, next_t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +Benchee.run( + %{ + "MultiTimeView.mark_out (extend existing history)" => fn {view, dep, value, next_t} -> + MultiTimeView.mark_out(view, dep, value, next_t) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"history length #{n}", n} + end, + before_each: fn n -> + view = MultiTimeView.new() + dep = "dep_#{System.unique_integer([:positive])}" + MultiTimeView.init_subquery(view, dep, []) + Bench.Pop.build_history(view, dep, 42, n) + next_t = n + if(rem(n, 2) == 1, do: 1, else: 2) + {view, dep, 42, next_t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# add_positive_route / remove_positive_route — sweep positive_children_for_subquery. +# Expected: O(positive_children). +Benchee.run( + %{ + "add_positive_route" => fn {filter, dep, value} -> + SubqueryIndex.add_positive_route(filter.subquery_index, dep, value) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} positive children", n} + end, + before_each: fn n -> + {filter, _cids, dep} = Bench.Pop.build_n_positive_children(n, values: []) + # Pick a fresh value each iteration to avoid no-op writes when the bag + # already contains the row. + value = System.unique_integer([:positive]) + {filter, dep, value} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +Benchee.run( + %{ + "remove_positive_route" => fn {filter, dep, value} -> + SubqueryIndex.remove_positive_route(filter.subquery_index, dep, value) + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} positive children", n} + end, + before_each: fn n -> + {filter, _cids, dep} = Bench.Pop.build_n_positive_children(n, values: []) + value = System.unique_integer([:positive]) + # Seed the route on every child first, so remove has actual work to do. + SubqueryIndex.add_positive_route(filter.subquery_index, dep, value) + {filter, dep, value} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# ============================================================================ +# Progress / compaction +# ============================================================================ + +IO.puts("\n\n# ========== Progress / compaction ==========\n") + +stack_for_pm = "bench-stack-#{System.unique_integer([:positive])}" +{:ok, pm_pid} = ProgressMonitor.start_link(stack_id: stack_for_pm) + +# notify_processed_up_to/3 — expected O(1) when no consumer change makes +# the minimum recompute walk many rows. Use a single consumer for one +# subquery; only the consumer's own row is touched. +Benchee.run( + %{ + "ProgressMonitor.notify_processed_up_to" => fn {dep, shape_handle, t} -> + :ok = ProgressMonitor.notify_processed_up_to(stack_for_pm, t, dep, shape_handle) + end + }, + inputs: %{"single consumer" => :only}, + before_each: fn :only -> + dep = "dep_#{System.unique_integer([:positive])}" + shape_handle = "shape_#{System.unique_integer([:positive])}" + :ok = ProgressMonitor.register_consumer(stack_for_pm, dep, shape_handle, self(), 0) + t = System.unique_integer([:positive]) + {dep, shape_handle, t} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Compactor pass — sweep "values touched" via dirty histories that have to +# compact. Each iteration sets a min_required_time past the toggle and +# measures `MultiTimeView.set_min_required_time/3` plus the route cleanup +# loop in `Compactor.compact_subquery/4` directly, since the GenServer tick +# is just a wrapper around those. +Benchee.run( + %{ + "Compactor pass (set_min_required_time + route cleanup)" => + fn {filter, dep, min_time} -> + removed = MultiTimeView.set_min_required_time(filter.subquery_index.multi_time_view, dep, min_time) + + for value <- removed do + SubqueryIndex.remove_positive_route(filter.subquery_index, dep, value) + end + end + }, + inputs: + for n <- [10, 100, 1_000, 10_000], into: %{} do + {"#{n} dirty values", n} + end, + before_each: fn n -> + {filter, _cid, [dep], _shapes} = + Bench.Pop.build(values_per_subquery: n, subquery_count: 1, shapes_per_subquery: 1) + + # Mark every value out at time 1, so a min_time of 2 compacts every + # history to empty and triggers the full deletion + route cleanup path. + for v <- 0..(n - 1) do + MultiTimeView.mark_out(filter.subquery_index.multi_time_view, dep, v, 1) + end + + {filter, dep, 2} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Dead-consumer release: a consumer monitored by ProgressMonitor dies while +# it had pinned times on N subqueries. The DOWN handler must process each +# of those subqueries. +Benchee.run( + %{ + "ProgressMonitor consumer DOWN release" => fn {pid, _deps} -> + ref = Process.monitor(pid) + send(pid, :stop) + + receive do + {:DOWN, ^ref, :process, ^pid, _} -> :ok + after + 1_000 -> raise "timeout waiting for consumer DOWN" + end + end + }, + inputs: + for n <- [1, 10, 100, 1_000], into: %{} do + {"#{n} pinned subqueries", n} + end, + before_each: fn n -> + {consumer_pid, _ref} = + spawn_monitor(fn -> + receive do + :stop -> :ok + end + end) + + deps = + for i <- 0..(n - 1) do + dep = "dep_down_#{System.unique_integer([:positive])}_#{i}" + :ok = ProgressMonitor.register_consumer(stack_for_pm, dep, "shape", consumer_pid, 0) + dep + end + + {consumer_pid, deps} + end, + time: 2, + warmup: 1, + # Benchee measures memory in a separate process; Filter's private ETS + # tables can't be read from there. Memory cost is covered by the + # dedicated script at scripts/subquery_logical_time_memory.exs. + memory_time: 0 +) + +# Tear the GenServer down so we exit cleanly. +GenServer.stop(pm_pid) + +IO.puts("\n\nDone.\n") diff --git a/packages/sync-service/mix.exs b/packages/sync-service/mix.exs index c06759cb94..a79f02873f 100644 --- a/packages/sync-service/mix.exs +++ b/packages/sync-service/mix.exs @@ -115,6 +115,7 @@ defmodule Electric.MixProject do defp dev_and_test_deps do [ + {:benchee, "~> 1.3", only: [:dev, :test], runtime: false}, {:dialyxir, "~> 1.4", only: [:test], runtime: false}, {:excoveralls, "~> 0.18", only: [:test], runtime: false}, {:junit_formatter, "~> 3.4", only: [:test], runtime: false}, diff --git a/packages/sync-service/mix.lock b/packages/sync-service/mix.lock index acb8cb92f2..3a4d27acbf 100644 --- a/packages/sync-service/mix.lock +++ b/packages/sync-service/mix.lock @@ -2,12 +2,14 @@ "acceptor_pool": {:hex, :acceptor_pool, "1.0.1", "d88c2e8a0be9216cf513fbcd3e5a4beb36bee3ff4168e85d6152c6f899359cdb", [:rebar3], [], "hexpm", "f172f3d74513e8edd445c257d596fc84dbdd56d2c6fa287434269648ae5a421e"}, "backoff": {:hex, :backoff, "1.1.6", "83b72ed2108ba1ee8f7d1c22e0b4a00cfe3593a67dbc792799e8cce9f42f796b", [:rebar3], [], "hexpm", "cf0cfff8995fb20562f822e5cc47d8ccf664c5ecdc26a684cbe85c225f9d7c39"}, "bandit": {:hex, :bandit, "1.11.0", "dbdd9c9963f146ee9da9860d1ee5b0ffd65cea51fe2aab3f3273df84329d133a", [:mix], [{:hpax, "~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}, {:plug, "~> 1.18", [hex: :plug, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:thousand_island, "~> 1.0", [hex: :thousand_island, repo: "hexpm", optional: false]}, {:websock, "~> 0.5", [hex: :websock, repo: "hexpm", optional: false]}], "hexpm", "c949d93a325a28da2333dde5a9ab61986ad2c2b7226347db6a28303b9139865e"}, + "benchee": {:hex, :benchee, "1.5.0", "4d812c31d54b0ec0167e91278e7de3f596324a78a096fd3d0bea68bb0c513b10", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}, {:statistex, "~> 1.1", [hex: :statistex, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "5b075393aea81b8ae74eadd1c28b1d87e8a63696c649d8293db7c4df3eb67535"}, "cc_precompiler": {:hex, :cc_precompiler, "0.1.11", "8c844d0b9fb98a3edea067f94f616b3f6b29b959b6b3bf25fee94ffe34364768", [:mix], [{:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "3427232caf0835f94680e5bcf082408a70b48ad68a5f5c0b02a3bea9f3a075b9"}, "chatterbox": {:hex, :ts_chatterbox, "0.15.1", "5cac4d15dd7ad61fc3c4415ce4826fc563d4643dee897a558ec4ea0b1c835c9c", [:rebar3], [{:hpack, "~> 0.3.0", [hex: :hpack_erl, repo: "hexpm", optional: false]}], "hexpm", "4f75b91451338bc0da5f52f3480fa6ef6e3a2aeecfc33686d6b3d0a0948f31aa"}, "combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"}, "ctx": {:hex, :ctx, "0.6.0", "8ff88b70e6400c4df90142e7f130625b82086077a45364a78d208ed3ed53c7fe", [:rebar3], [], "hexpm", "a14ed2d1b67723dbebbe423b28d7615eb0bdcba6ff28f2d1f1b0a7e1d4aa5fc2"}, "db_connection": {:hex, :db_connection, "2.10.0", "8ff756471e41765bd5563b633f73e9a94bbc138816e8644bb17d0d91bf260a95", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "02cdd01b45efb1b550e68edbbea41be32de9b24bb07e1ea0e9cbc522ac377e54"}, "decimal": {:hex, :decimal, "2.3.0", "3ad6255aa77b4a3c4f818171b12d237500e63525c2fd056699967a3e7ea20f62", [:mix], [], "hexpm", "a4d66355cb29cb47c3cf30e71329e58361cfcb37c34235ef3bf1d7bf3773aeac"}, + "deep_merge": {:hex, :deep_merge, "1.0.0", "b4aa1a0d1acac393bdf38b2291af38cb1d4a52806cf7a4906f718e1feb5ee961", [:mix], [], "hexpm", "ce708e5f094b9cd4e8f2be4f00d2f4250c4095be93f8cd6d018c753894885430"}, "dialyxir": {:hex, :dialyxir, "1.4.7", "dda948fcee52962e4b6c5b4b16b2d8fa7d50d8645bbae8b8685c3f9ecb7f5f4d", [:mix], [{:erlex, ">= 0.2.8", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "b34527202e6eb8cee198efec110996c25c5898f43a4094df157f8d28f27d9efe"}, "dotenvy": {:hex, :dotenvy, "1.1.1", "00e318f3c51de9fafc4b48598447e386f19204dc18ca69886905bb8f8b08b667", [:mix], [], "hexpm", "c8269471b5701e9e56dc86509c1199ded2b33dce088c3471afcfef7839766d8e"}, "earmark_parser": {:hex, :earmark_parser, "1.4.44", "f20830dd6b5c77afe2b063777ddbbff09f9759396500cdbe7523efd58d7a339c", [:mix], [], "hexpm", "4778ac752b4701a5599215f7030989c989ffdc4f6df457c5f36938cc2d2a2750"}, @@ -58,6 +60,7 @@ "retry": {:hex, :retry, "0.19.0", "aeb326d87f62295d950f41e1255fe6f43280a1b390d36e280b7c9b00601ccbc2", [:mix], [], "hexpm", "85ef376aa60007e7bff565c366310966ec1bd38078765a0e7f20ec8a220d02ca"}, "sentry": {:hex, :sentry, "12.0.3", "0d5f681b4a7b57c4df16a37f4b90a554d10e577dad01d4457c844ffa24800861", [:mix], [{:finch, "~> 0.21", [hex: :finch, repo: "hexpm", optional: true]}, {:hackney, "~> 1.8", [hex: :hackney, repo: "hexpm", optional: true]}, {:igniter, "~> 0.5", [hex: :igniter, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:nimble_options, "~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_ownership, "~> 0.3.0 or ~> 1.0", [hex: :nimble_ownership, repo: "hexpm", optional: false]}, {:opentelemetry, ">= 0.0.0", [hex: :opentelemetry, repo: "hexpm", optional: true]}, {:opentelemetry_api, ">= 0.0.0", [hex: :opentelemetry_api, repo: "hexpm", optional: true]}, {:opentelemetry_exporter, ">= 0.0.0", [hex: :opentelemetry_exporter, repo: "hexpm", optional: true]}, {:opentelemetry_semantic_conventions, ">= 0.0.0", [hex: :opentelemetry_semantic_conventions, repo: "hexpm", optional: true]}, {:phoenix, "~> 1.6", [hex: :phoenix, repo: "hexpm", optional: true]}, {:phoenix_live_view, "~> 0.20 or ~> 1.0", [hex: :phoenix_live_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.6", [hex: :plug, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm", "3c2f571ea43e8d3c893c5ca88ab1d1b9501f02925d094fc23811ce5b7f228b65"}, "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"}, + "statistex": {:hex, :statistex, "1.1.0", "7fec1eb2f580a0d2c1a05ed27396a084ab064a40cfc84246dbfb0c72a5c761e5", [:mix], [], "hexpm", "f5950ea26ad43246ba2cce54324ac394a4e7408fdcf98b8e230f503a0cba9cf5"}, "stream_data": {:hex, :stream_data, "1.3.0", "bde37905530aff386dea1ddd86ecbf00e6642dc074ceffc10b7d4e41dfd6aac9", [:mix], [], "hexpm", "3cc552e286e817dca43c98044c706eec9318083a1480c52ae2688b08e2936e3c"}, "stream_split": {:hex, :stream_split, "0.1.7", "2d3fd1fd21697da7f91926768d65f79409086052c9ec7ae593987388f52425f8", [:mix], [], "hexpm", "1dc072ff507a64404a0ad7af90df97096183fee8eeac7b300320cea7c4679147"}, "telemetry": {:hex, :telemetry, "1.4.1", "ab6de178e2b29b58e8256b92b382ea3f590a47152ca3651ea857a6cae05ac423", [:rebar3], [], "hexpm", "2172e05a27531d3d31dd9782841065c50dd5c3c7699d95266b2edd54c2dafa1c"}, diff --git a/packages/sync-service/scripts/subquery_logical_time_memory.exs b/packages/sync-service/scripts/subquery_logical_time_memory.exs new file mode 100644 index 0000000000..1fe7da8878 --- /dev/null +++ b/packages/sync-service/scripts/subquery_logical_time_memory.exs @@ -0,0 +1,310 @@ +# SubqueryIndex memory snapshot. +# +# Run from packages/sync-service: +# +# mix run --no-start scripts/subquery_logical_time_memory.exs +# +# `--no-start` skips booting the sync-service application; this script only +# uses the SubqueryIndex / MultiTimeView modules and a small simulated +# consumer-side state, all in-process. +# +# Reproduces the "Logical" columns of the memory table in +# docs/rfcs/subquery-index.md (§Memory Savings Prototype). The corresponding +# "current model" numbers in that table were generated by a separate +# prototype that simulated the pre-RFC per-shape MapSet implementation; we +# don't reproduce them here because the old model code has been removed +# from this branch. +# +# Measured per scenario: +# * ETS: SubqueryIndex bag table + MultiTimeView set table. +# * Consumers: the compact `%{subquery_ref => %{subquery_id, time}}` map +# each shape holds, plus any in-flight ActiveMove structs. +# * Total: ETS + Consumers. + +alias Electric.Replication.Eval.Parser.{Func, Ref} +alias Electric.Shapes.Consumer.Subqueries.ActiveMove +alias Electric.Shapes.DnfPlan +alias Electric.Shapes.Filter +alias Electric.Shapes.Filter.Indexes.SubqueryIndex +alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView +alias Electric.Shapes.Filter.WhereCondition + +defmodule MemBench do + @field "par_id" + @subquery_ref ["$sublink", "0"] + @wordsize :erlang.system_info(:wordsize) + + def make_plan(polarity \\ :positive) do + testexpr = %Ref{path: [@field], type: :int8} + ref = %Ref{path: @subquery_ref, type: {:array, :int8}} + + ast = %Func{ + name: "sublink_membership_check", + args: [testexpr, ref], + type: :bool + } + + %DnfPlan{ + disjuncts: [], + disjuncts_positions: [], + position_count: 1, + positions: %{ + 0 => %{ + ast: ast, + sql: "fake", + is_subquery: true, + negated: polarity == :negated, + dependency_index: 0, + subquery_ref: @subquery_ref, + tag_columns: [@field] + } + }, + dependency_positions: %{0 => [0]}, + dependency_disjuncts: %{}, + dependency_polarities: %{0 => polarity} + } + end + + def opt do + %{ + operation: "subquery", + field: @field, + testexpr: %Ref{path: [@field], type: :int8}, + subquery_ref: @subquery_ref, + dep_index: 0, + polarity: :positive, + and_where: nil + } + end + + # Build a Filter with one subquery containing `value_count` base values, + # `shape_count` shapes attached, all at time 0. + def build_steady(shape_count, value_count) do + filter = Filter.new() + condition_id = make_ref() + WhereCondition.init(filter, condition_id) + index = filter.subquery_index + dep = "dep_a" + + values = for v <- 1..value_count, do: v + MultiTimeView.init_subquery(index.multi_time_view, dep, values) + MultiTimeView.mark_ready(index.multi_time_view, dep) + + shape_ids = + for i <- 1..shape_count do + shape_id = "shape_#{i}" + SubqueryIndex.register_shape(index, shape_id, make_plan(), [dep]) + SubqueryIndex.add_shape(filter, condition_id, shape_id, opt(), []) + SubqueryIndex.mark_ready(index, shape_id) + shape_id + end + + consumer_states = + for _ <- 1..shape_count, do: %{@subquery_ref => %{subquery_id: dep, time: 0}} + + %{filter: filter, dep: dep, shape_ids: shape_ids, consumer_states: consumer_states} + end + + # Layer on `added_count` values across `time_steps` logical timesteps. + # Each added value enters the MTV via mark_in at a logical time spread + # uniformly across [1, time_steps]. Old times are retained because not + # every consumer has advanced past them. + def apply_advanced(state, added_count, time_steps) do + %{filter: filter, dep: dep, shape_ids: shape_ids, consumer_states: consumer_states} = + state + + index = filter.subquery_index + mtv = index.multi_time_view + + base_value = 1_000_000 + + for i <- 1..added_count do + value = base_value + i + # Spread the adds across the available time steps. Floor division so + # each step gets a roughly equal slice. + step = div((i - 1) * time_steps, added_count) + 1 + MultiTimeView.mark_in(mtv, dep, value, step) + SubqueryIndex.add_positive_route(index, dep, value) + end + + # Advance each consumer to a logical time spread across [0, time_steps]. + # This leaves a non-trivial retention window so histories accumulate. + consumer_states = + consumer_states + |> Enum.with_index(0) + |> Enum.map(fn {state, idx} -> + target = div(idx * time_steps, length(shape_ids)) + Map.update!(state, @subquery_ref, &%{&1 | time: target}) + end) + + %{state | consumer_states: consumer_states} + end + + # Add `move_count` in-flight ActiveMove structs. `total_moved_values` is + # split evenly across the moves. + def apply_active_moves(state, move_count, total_moved_values) do + dep = state.dep + values_per_move = max(1, div(total_moved_values, max(move_count, 1))) + + active_moves = + for i <- 1..move_count do + from_t = i + to_t = i + 1 + + move_in_values = + for j <- 1..values_per_move do + value = 2_000_000 + i * values_per_move + j + {value, "v#{value}"} + end + + %ActiveMove{ + subquery_id: dep, + dep_index: 0, + subquery_ref: @subquery_ref, + from_time: from_t, + to_time: to_t, + move_in_values: move_in_values, + move_out_values: [], + txids: [10_000 + i], + buffered_txn_count: 0, + buffered_txns: [] + } + end + + Map.put(state, :active_moves, active_moves) + end + + def ets_bytes(filter) do + case :ets.info(filter.subquery_index.table, :memory) do + words when is_integer(words) -> words * @wordsize + _ -> 0 + end + end + + def mtv_bytes(filter) do + case :ets.info(filter.subquery_index.multi_time_view, :memory) do + words when is_integer(words) -> words * @wordsize + _ -> 0 + end + end + + def consumer_bytes(state) do + consumer_size = + state.consumer_states + |> Enum.map(&(:erts_debug.size(&1) * @wordsize)) + |> Enum.sum() + + move_size = + Map.get(state, :active_moves, []) + |> Enum.map(&(:erts_debug.size(&1) * @wordsize)) + |> Enum.sum() + + consumer_size + move_size + end + + def measure(label, state) do + ets = ets_bytes(state.filter) + mtv_bytes(state.filter) + consumers = consumer_bytes(state) + total = ets + consumers + + %{ + label: label, + ets: ets, + consumers: consumers, + total: total + } + end + + def human(bytes) when bytes < 1024, do: "#{bytes} B" + + def human(bytes) when bytes < 1024 * 1024, + do: :erlang.float_to_binary(bytes / 1024, decimals: 1) <> " KiB" + + def human(bytes), do: :erlang.float_to_binary(bytes / 1_048_576, decimals: 2) <> " MiB" + + def print_table(rows) do + IO.puts("") + + IO.puts( + "| Scenario | Logical total | Logical ETS | Logical consumers |" + ) + + IO.puts( + "|----------|---------------|-------------|-------------------|" + ) + + for r <- rows do + IO.puts( + "| #{r.label} | #{human(r.total)} | #{human(r.ets)} | #{human(r.consumers)} |" + ) + end + + IO.puts("") + end +end + +# ============================================================================ +# Scenarios +# ============================================================================ + +scenarios = [ + fn -> + MemBench.build_steady(1, 1_000) + |> then(&MemBench.measure("1 shape, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(10, 1_000) + |> then(&MemBench.measure("10 shapes, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> then(&MemBench.measure("100 shapes, 1k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 10_000) + |> then(&MemBench.measure("100 shapes, 10k values, steady", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 10) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 10 advanced", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 99) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 99 advanced", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(100, 10) + |> MemBench.apply_active_moves(10, 100) + |> then(&MemBench.measure("100 shapes, 1k base, 100 added x 10 active move", &1)) + end, + fn -> + MemBench.build_steady(100, 1_000) + |> MemBench.apply_advanced(1_000, 99) + |> MemBench.apply_active_moves(99, 1_000) + |> then(&MemBench.measure("100 shapes, 1k base, 1k added x 99 active move", &1)) + end +] + +results = + for scenario <- scenarios do + # Each scenario builds its own private ETS; force a GC between to keep + # the numbers comparable. + :erlang.garbage_collect() + result = scenario.() + :erlang.garbage_collect() + result + end + +IO.puts("# SubqueryIndex memory snapshot") +IO.puts("") + +IO.puts( + "OTP: #{:erlang.system_info(:otp_release)} Wordsize: #{:erlang.system_info(:wordsize)} bytes" +) + +IO.puts("Elixir: #{System.version()}") + +MemBench.print_table(results) From 53fdf561485d53463efedbaa67c8c39a26868122 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 09:57:23 +0100 Subject: [PATCH 33/40] Fix O(n) removal with counters --- .../shapes/filter/indexes/subquery_index.ex | 126 +++++++++++++----- 1 file changed, 89 insertions(+), 37 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 0364477ce7..154002ec70 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -24,29 +24,46 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView alias Electric.Shapes.Filter.WhereCondition - defstruct [:table, :multi_time_view] + defstruct [:table, :counters, :multi_time_view] - @type t :: %SubqueryIndex{table: :ets.tid() | atom(), multi_time_view: MultiTimeView.t() | nil} + @type t :: %SubqueryIndex{ + table: :ets.tid() | atom(), + counters: :ets.tid() | atom(), + multi_time_view: MultiTimeView.t() | nil + } defp table_name(stack_id) when is_stack_id(stack_id), do: :"subquery_index:#{stack_id}" + defp counters_table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_index_counters:#{stack_id}" + @spec new(keyword()) :: t() def new(opts \\ []) do - table = + {table, counters} = case Keyword.get(opts, :stack_id) do nil -> - :ets.new(:subquery_index, [:bag, :public]) + { + :ets.new(:subquery_index, [:bag, :public]), + :ets.new(:subquery_index_counters, [:set, :public]) + } stack_id -> - try do - :ets.new(table_name(stack_id), [:bag, :public, :named_table]) - rescue - ArgumentError -> table_name(stack_id) - end + { + try do + :ets.new(table_name(stack_id), [:bag, :public, :named_table]) + rescue + ArgumentError -> table_name(stack_id) + end, + try do + :ets.new(counters_table_name(stack_id), [:set, :public, :named_table]) + rescue + ArgumentError -> counters_table_name(stack_id) + end + } end multi_time_view = MultiTimeView.new(Keyword.take(opts, [:stack_id])) - %SubqueryIndex{table: table, multi_time_view: multi_time_view} + %SubqueryIndex{table: table, counters: counters, multi_time_view: multi_time_view} end @spec for_stack(String.t()) :: t() | nil @@ -58,6 +75,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do _tid -> %SubqueryIndex{ table: table_name(stack_id), + counters: counters_table_name(stack_id), multi_time_view: MultiTimeView.for_stack(stack_id) } end @@ -102,7 +120,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec add_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :ok def add_shape( - %Filter{subquery_index: %SubqueryIndex{table: table} = index} = filter, + %Filter{subquery_index: %SubqueryIndex{table: table, counters: counters} = index} = filter, condition_id, shape_handle, optimisation, @@ -111,7 +129,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do ensure_node_meta(table, condition_id, optimisation.field, optimisation.testexpr) group_id = - ensure_group(table, condition_id, optimisation.field, optimisation.polarity) + ensure_group(table, counters, condition_id, optimisation.field, optimisation.polarity) subquery_id = lookup_dep_handle!(table, shape_handle, optimisation.dep_index) @@ -154,7 +172,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec remove_shape(Filter.t(), reference(), term(), map(), [atom()]) :: :deleted | :ok def remove_shape( - %Filter{subquery_index: %SubqueryIndex{table: table, multi_time_view: mtv}} = filter, + %Filter{ + subquery_index: %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv} + } = filter, condition_id, shape_handle, optimisation, @@ -169,7 +189,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do branch_key ) do nil -> - node_status(table, condition_id, optimisation.field) + node_status(counters, condition_id, optimisation.field) {child_node_id, next_condition_id} -> _ = @@ -190,10 +210,10 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do ) if child_empty?(table, child_node_id) do - delete_child(table, mtv, child_node_id) + delete_child(table, counters, mtv, child_node_id) end - node_status(table, condition_id, optimisation.field) + node_status(counters, condition_id, optimisation.field) end end @@ -264,10 +284,13 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do also cleared so values for the subquery are gone everywhere. """ @spec remove_subquery(t(), term()) :: :ok - def remove_subquery(%SubqueryIndex{table: table, multi_time_view: mtv}, subquery_id) do + def remove_subquery( + %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv}, + subquery_id + ) do for child_node_id <- children_for_subquery(table, subquery_id) do cleanup_child_shapes(table, child_node_id) - delete_child(table, mtv, child_node_id) + delete_child(table, counters, mtv, child_node_id) end if mtv, do: MultiTimeView.remove_subquery(mtv, subquery_id) @@ -335,7 +358,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end end - defp ensure_group(table, condition_id, field_key, polarity) do + defp ensure_group(table, counters, condition_id, field_key, polarity) do key = {:group, condition_id, field_key, polarity} case :ets.lookup(table, key) do @@ -345,12 +368,20 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do [] -> group_id = make_ref() :ets.insert(table, {key, group_id}) + + :ets.update_counter( + counters, + {:node_groups, condition_id, field_key}, + 1, + {{:node_groups, condition_id, field_key}, 0} + ) + group_id end end defp ensure_child(filter, index, group_id, subquery_id, polarity, condition_id, field_key) do - %SubqueryIndex{table: table, multi_time_view: mtv} = index + %SubqueryIndex{table: table, counters: counters, multi_time_view: mtv} = index case :ets.lookup(table, {:child, group_id, subquery_id}) do [{_, child_node_id}] -> @@ -376,6 +407,13 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.insert(table, {{:child_meta, child_node_id}, meta}) :ets.insert(table, {{:subquery_child, subquery_id}, child_node_id}) + :ets.update_counter( + counters, + {:group_children, group_id}, + 1, + {{:group_children, group_id}, 0} + ) + seed_child_routing(table, mtv, child_node_id, meta) {child_node_id, next_condition_id} end @@ -452,7 +490,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do not :ets.member(table, {:child_shape, child_node_id}) end - defp delete_child(table, mtv, child_node_id) do + defp delete_child(table, counters, mtv, child_node_id) do case :ets.lookup(table, {:child_meta, child_node_id}) do [] -> :ok @@ -475,15 +513,30 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do :ets.delete_object(table, {{:subquery_child, meta.subquery_id}, child_node_id}) :ets.delete(table, {:child_meta, child_node_id}) - if group_empty?(table, meta.group_id) do - :ets.delete( - table, - {:group, meta.condition_id, meta.field_key, meta.polarity} - ) + case :ets.update_counter(counters, {:group_children, meta.group_id}, -1) do + 0 -> + :ets.delete(counters, {:group_children, meta.group_id}) + + :ets.delete( + table, + {:group, meta.condition_id, meta.field_key, meta.polarity} + ) + + case :ets.update_counter( + counters, + {:node_groups, meta.condition_id, meta.field_key}, + -1 + ) do + 0 -> + :ets.delete(counters, {:node_groups, meta.condition_id, meta.field_key}) + :ets.delete(table, {:node_testexpr, meta.condition_id, meta.field_key}) + + _ -> + :ok + end - if node_empty?(table, meta.condition_id, meta.field_key) do - :ets.delete(table, {:node_testexpr, meta.condition_id, meta.field_key}) - end + _ -> + :ok end :ok @@ -497,12 +550,11 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do end end - defp group_empty?(table, group_id) do - :ets.match(table, {{:child, group_id, :_}, :_}) == [] - end - - defp node_empty?(table, condition_id, field_key) do - :ets.match(table, {{:group, condition_id, field_key, :_}, :_}) == [] + defp node_empty?(counters, condition_id, field_key) do + case :ets.lookup(counters, {:node_groups, condition_id, field_key}) do + [{_, c}] when c > 0 -> false + _ -> true + end end defp positive_children(table, condition_id, field_key, value) do @@ -559,8 +611,8 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do |> MapSet.new() end - defp node_status(table, condition_id, field_key) do - if node_empty?(table, condition_id, field_key), do: :deleted, else: :ok + defp node_status(counters, condition_id, field_key) do + if node_empty?(counters, condition_id, field_key), do: :deleted, else: :ok end defp evaluate_node_lhs(table, condition_id, field_key, record) do From 47b0d4b0fd8c25fc32c08fc09f8f378cc4ac7edd Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 10:38:49 +0100 Subject: [PATCH 34/40] Optimise ProgressMonitor --- .../subquery_index/progress_monitor.ex | 85 +++++++++++++------ 1 file changed, 61 insertions(+), 24 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex index 30303b1a20..cffefe06a1 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index/progress_monitor.ex @@ -4,10 +4,22 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do still need to read. The minimum across live consumers is the compaction lower bound for `MultiTimeView`. - One GenServer + one ETS table per stack. The GenServer serialises writes - and owns the process monitors that release pinned times when consumers die. - `min_required_time/2` reads the ETS row directly so the routing path does - not have to call the GenServer. + One GenServer + two ETS tables per stack: + + - A `:set` "positions" table holding one row per registered consumer + (`{:consumer, subquery_id, shape_handle} -> {time, monitor_ref}`) plus a + denormalised `{:min_required_time, subquery_id} -> time` row for + lock-free external reads from the routing/compactor path. + - An `:ordered_set` "times" table keyed by + `{subquery_id, time, shape_handle}`. The min for a subquery is the + first entry whose key starts with that `subquery_id`, found in + `O(log N)` via `:ets.next/2` against a sentinel key. + + Together they keep register/notify/unregister/DOWN at `O(log N)` instead + of `O(total_consumers)`. + + `min_required_time/2` and `registered?/3` read the positions table + directly without touching the GenServer. See `docs/rfcs/subquery-index.md`, section *Processed-Up-To Time*. """ @@ -27,6 +39,9 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do defp table_name(stack_id) when is_stack_id(stack_id), do: :"subquery_progress_monitor_table:#{stack_id}" + defp times_table_name(stack_id) when is_stack_id(stack_id), + do: :"subquery_progress_monitor_times:#{stack_id}" + @spec start_link(keyword()) :: GenServer.on_start() def start_link(opts) do stack_id = Keyword.fetch!(opts, :stack_id) @@ -111,7 +126,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do def init(opts) do stack_id = Keyword.fetch!(opts, :stack_id) - table = + positions = :ets.new(table_name(stack_id), [ :set, :public, @@ -119,7 +134,14 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do read_concurrency: true ]) - {:ok, %{stack_id: stack_id, table: table, monitors: %{}}} + times = + :ets.new(times_table_name(stack_id), [ + :ordered_set, + :public, + :named_table + ]) + + {:ok, %{stack_id: stack_id, positions: positions, times: times, monitors: %{}}} end @impl true @@ -129,29 +151,33 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do monitor_ref = Process.monitor(pid) :ets.insert( - state.table, + state.positions, {{:consumer, subquery_id, shape_handle}, time, monitor_ref} ) + :ets.insert(state.times, {{subquery_id, time, shape_handle}}) + state = put_in(state.monitors[monitor_ref], {subquery_id, shape_handle}) - recompute_min(state.table, subquery_id) + update_min(state.positions, state.times, subquery_id) {:reply, :ok, state} end def handle_call({:unregister, subquery_id, shape_handle}, _from, state) do state = remove_registration(state, subquery_id, shape_handle) - recompute_min(state.table, subquery_id) + update_min(state.positions, state.times, subquery_id) {:reply, :ok, state} end def handle_call({:notify, time, subquery_id, shape_handle}, _from, state) do - case :ets.lookup(state.table, {:consumer, subquery_id, shape_handle}) do + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do [{key, current, monitor_ref}] -> new_required = max(current, time + 1) if new_required != current do - :ets.insert(state.table, {key, new_required, monitor_ref}) - recompute_min(state.table, subquery_id) + :ets.delete(state.times, {subquery_id, current, shape_handle}) + :ets.insert(state.times, {{subquery_id, new_required, shape_handle}}) + :ets.insert(state.positions, {key, new_required, monitor_ref}) + update_min(state.positions, state.times, subquery_id) end {:reply, :ok, state} @@ -168,17 +194,26 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do {:noreply, state} {{subquery_id, shape_handle}, monitors} -> - :ets.delete(state.table, {:consumer, subquery_id, shape_handle}) - recompute_min(state.table, subquery_id) + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do + [{_key, time, _ref}] -> + :ets.delete(state.positions, {:consumer, subquery_id, shape_handle}) + :ets.delete(state.times, {subquery_id, time, shape_handle}) + + [] -> + :ok + end + + update_min(state.positions, state.times, subquery_id) {:noreply, %{state | monitors: monitors}} end end defp remove_registration(state, subquery_id, shape_handle) do - case :ets.lookup(state.table, {:consumer, subquery_id, shape_handle}) do - [{key, _time, monitor_ref}] -> + case :ets.lookup(state.positions, {:consumer, subquery_id, shape_handle}) do + [{key, time, monitor_ref}] -> Process.demonitor(monitor_ref, [:flush]) - :ets.delete(state.table, key) + :ets.delete(state.positions, key) + :ets.delete(state.times, {subquery_id, time, shape_handle}) %{state | monitors: Map.delete(state.monitors, monitor_ref)} [] -> @@ -186,14 +221,16 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex.ProgressMonitor do end end - defp recompute_min(table, subquery_id) do - case :ets.match(table, {{:consumer, subquery_id, :_}, :"$1", :_}) do - [] -> - :ets.delete(table, {:min_required_time, subquery_id}) + # The min for `subquery_id` is the smallest entry in the ordered_set + # whose key starts with `{subquery_id, ...}`. `:ets.next/2` against a + # sentinel below any real time gives that entry in O(log N). + defp update_min(positions, times, subquery_id) do + case :ets.next(times, {subquery_id, -1, nil}) do + {^subquery_id, time, _shape_handle} -> + :ets.insert(positions, {{:min_required_time, subquery_id}, time}) - times -> - min = times |> Enum.map(fn [t] -> t end) |> Enum.min() - :ets.insert(table, {{:min_required_time, subquery_id}, min}) + _ -> + :ets.delete(positions, {:min_required_time, subquery_id}) end end end From d6b84bf814d4e5b7eeb97d5a3b40cad0886cd55f Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 10:51:56 +0100 Subject: [PATCH 35/40] BENCHMARKS: Include counter in memory use --- .../scripts/subquery_logical_time_memory.exs | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/packages/sync-service/scripts/subquery_logical_time_memory.exs b/packages/sync-service/scripts/subquery_logical_time_memory.exs index 1fe7da8878..2158c2ee96 100644 --- a/packages/sync-service/scripts/subquery_logical_time_memory.exs +++ b/packages/sync-service/scripts/subquery_logical_time_memory.exs @@ -174,18 +174,20 @@ defmodule MemBench do Map.put(state, :active_moves, active_moves) end - def ets_bytes(filter) do - case :ets.info(filter.subquery_index.table, :memory) do + defp ets_info_bytes(table) do + case :ets.info(table, :memory) do words when is_integer(words) -> words * @wordsize _ -> 0 end end + def ets_bytes(filter) do + ets_info_bytes(filter.subquery_index.table) + + ets_info_bytes(filter.subquery_index.counters) + end + def mtv_bytes(filter) do - case :ets.info(filter.subquery_index.multi_time_view, :memory) do - words when is_integer(words) -> words * @wordsize - _ -> 0 - end + ets_info_bytes(filter.subquery_index.multi_time_view) end def consumer_bytes(state) do From cda01c440832e11c8ca98b439bdddfbf1c5d11ea Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 11:28:32 +0100 Subject: [PATCH 36/40] Build move-in query inside the task, not the consumer Move `MultiTimeView.values/3` materialisation and `Querying.move_in_where_clause/4` construction inside `Task.Supervisor.start_child` in `query_move_in_async/4` so the dependency-view arrays live on the short-lived task heap rather than on the long-lived consumer process. Also extract just the fields the task needs so the closure no longer captures the whole `consumer_state`. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../lib/electric/shapes/consumer/effects.ex | 69 ++++++++++--------- 1 file changed, 38 insertions(+), 31 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/effects.ex b/packages/sync-service/lib/electric/shapes/consumer/effects.ex index 2c20d39e15..172442d769 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/effects.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/effects.ex @@ -248,39 +248,15 @@ defmodule Electric.Shapes.Consumer.Effects do ) do alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView - mtv = MultiTimeView.for_stack(consumer_state.stack_id) - subquery_refs = consumer_state.event_handler.subquery_refs - - # `values_for.(ref, :before | :after)` reads `MultiTimeView` lazily at - # SQL build time — for the trigger ref `:before` uses `request.from_time` - # and `:after` uses `request.to_time`; other refs read at the consumer's - # currently-pinned time so the move-in query sees a consistent view - # across all dependencies. - values_for = fn ref, when_ -> - %{subquery_id: id, time: pinned_time} = Map.fetch!(subquery_refs, ref) - - time = - cond do - ref != request.subquery_ref -> pinned_time - when_ == :before -> request.from_time - when_ == :after -> request.to_time - end - - MultiTimeView.values(mtv, id, time) - end - - {where, params} = - Querying.move_in_where_clause( - request.dnf_plan, - request.trigger_dep_index, - values_for, - consumer_state.shape.where.used_refs - ) - - pool = Manager.pool_name(consumer_state.stack_id, :snapshot) + # Extract the specific fields the task needs so the closure doesn't + # capture the whole consumer_state. stack_id = consumer_state.stack_id shape = consumer_state.shape shape_handle = consumer_state.shape_handle + storage = consumer_state.storage + subquery_refs = consumer_state.event_handler.subquery_refs + used_refs = consumer_state.shape.where.used_refs + pool = Manager.pool_name(stack_id, :snapshot) :telemetry.execute([:electric, :subqueries, :move_in_triggered], %{count: 1}, %{ stack_id: stack_id @@ -294,6 +270,37 @@ defmodule Electric.Shapes.Consumer.Effects do Task.Supervisor.start_child(supervisor, fn -> OpenTelemetry.set_current_context(trace_context) + # Build the move-in SQL inside the task so the materialised value + # arrays from `MultiTimeView.values/3` live on the task heap and die + # with it, not on the long-lived consumer process. + # + # `values_for.(ref, :before | :after)` — for the trigger ref `:before` + # uses `request.from_time` and `:after` uses `request.to_time`; other + # refs read at the consumer's currently-pinned time so the move-in + # query sees a consistent view across all dependencies. + mtv = MultiTimeView.for_stack(stack_id) + + values_for = fn ref, when_ -> + %{subquery_id: id, time: pinned_time} = Map.fetch!(subquery_refs, ref) + + time = + cond do + ref != request.subquery_ref -> pinned_time + when_ == :before -> request.from_time + when_ == :after -> request.to_time + end + + MultiTimeView.values(mtv, id, time) + end + + {where, params} = + Querying.move_in_where_clause( + request.dnf_plan, + request.trigger_dep_index, + values_for, + used_refs + ) + snapshot_name = Electric.Utils.uuid4() try do @@ -319,7 +326,7 @@ defmodule Electric.Shapes.Consumer.Effects do send(task_pid, {:move_in_snapshot_stats, row_count, row_bytes}) end ) - |> Storage.write_move_in_snapshot!(snapshot_name, consumer_state.storage) + |> Storage.write_move_in_snapshot!(snapshot_name, storage) {row_count, row_bytes} = receive do From a407829952808d57e041dda57ba217c2e67fbf24 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 11:31:45 +0100 Subject: [PATCH 37/40] MoveQueue.enqueue/4 takes a member? callback instead of a MapSet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reduce path only used the dep_view to answer "is this value in the base view?". Passing a `(value) -> boolean()` callback lets the consumer skip materialising the full dependency view on its heap — production now closes over `MultiTimeView.member?/4` per value. Steady and Buffering event handlers no longer build a transient MapSet per materializer_changes event. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../event_handler/subqueries/buffering.ex | 12 +++--- .../event_handler/subqueries/steady.ex | 4 +- .../shapes/consumer/subqueries/move_queue.ex | 30 +++++++------ .../consumer/subqueries/move_queue_test.exs | 42 ++++++++++--------- 4 files changed, 48 insertions(+), 40 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex index 94757d9b4f..8cff173ae8 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/buffering.ex @@ -110,9 +110,9 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do subquery_ref = RefResolver.ref_from_dep_handle!(state.shape_info.ref_resolver, dep_handle) dep_index = subquery_ref |> List.last() |> String.to_integer() mtv = MultiTimeView.for_stack(state.shape_info.stack_id) - dep_view = view_after_active_move(mtv, state.active_move, state.subquery_refs, subquery_ref) + member? = member_after_active_move(mtv, state.active_move, state.subquery_refs, subquery_ref) - {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, dep_view)}, []} + {:ok, %{state | queue: MoveQueue.enqueue(state.queue, dep_index, payload, member?)}, []} end def handle_event(%__MODULE__{} = state, {:pg_snapshot_known, snapshot}) do @@ -233,8 +233,10 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do # The base view for reducing buffered move queue entries is the consumer's # view *as if* the in-flight active move had already spliced. For the # trigger ref that means MTV at `active_move.to_time`; for every other ref - # the consumer is still pinned at its currently-tracked time. - defp view_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do + # the consumer is still pinned at its currently-tracked time. Returns a + # `(value) -> boolean()` callback so MoveQueue can ask membership per + # value without materialising the whole view. + defp member_after_active_move(mtv, active_move, subquery_refs, subquery_ref) do %{subquery_id: subquery_id} = Map.fetch!(subquery_refs, subquery_ref) time = @@ -244,6 +246,6 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Buffering do subquery_refs[subquery_ref].time end - mtv |> MultiTimeView.values(subquery_id, time) |> MapSet.new() + fn value -> MultiTimeView.member?(mtv, subquery_id, value, time) end end end diff --git a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex index c2c040aedb..20aa4a964f 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/event_handler/subqueries/steady.ex @@ -53,14 +53,14 @@ defmodule Electric.Shapes.Consumer.EventHandler.Subqueries.Steady do dep_index = subquery_ref |> List.last() |> String.to_integer() mtv = MultiTimeView.for_stack(state.shape_info.stack_id) %{subquery_id: subquery_id, time: pinned_time} = Map.fetch!(state.subquery_refs, subquery_ref) - dep_view = mtv |> MultiTimeView.values(subquery_id, pinned_time) |> MapSet.new() + member? = fn value -> MultiTimeView.member?(mtv, subquery_id, value, pinned_time) end payload_with_default_from_time = Map.put_new(Map.new(payload), :from_time, pinned_time) next_state = %{ state | queue: - MoveQueue.enqueue(state.queue, dep_index, payload_with_default_from_time, dep_view) + MoveQueue.enqueue(state.queue, dep_index, payload_with_default_from_time, member?) } with {:ok, next_state, effects} <- drain_queue(next_state, EffectList.new()) do diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex index 08bd2b0811..aff30a372e 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/move_queue.ex @@ -55,18 +55,22 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do @doc """ Enqueue a materializer payload for a specific dependency. - `dep_view` is the materializer view at the consumer's pinned time for - this dep (or the view-after-active-move for the trigger ref during - Buffering). It's used by `reduce/2` to drop redundant ops. + `member?` is a callback `(value) -> boolean()` that answers "is `value` + currently a member of the dep view at the consumer's pinned time for + this dep (or, during Buffering, the view-after-active-move for the + trigger ref)?". It's used by `reduce/2` to drop redundant ops. A + callback rather than a `MapSet` lets the caller skip materialising the + full dependency view on the consumer's heap — production passes a + closure that consults `MultiTimeView.member?/4` per value. The payload may include `:from_time`, `:to_time`, and `:txids` keys. The first enqueue for a dep records `from_time` (subsequent payloads leave it untouched). `to_time` is updated to `max(current, new)`. Txids accumulate. """ - @spec enqueue(t(), non_neg_integer(), map() | keyword(), MapSet.t()) :: t() - def enqueue(%__MODULE__{} = queue, dep_index, payload, %MapSet{} = dep_view) - when is_map(payload) or is_list(payload) do + @spec enqueue(t(), non_neg_integer(), map() | keyword(), (term() -> boolean())) :: t() + def enqueue(%__MODULE__{} = queue, dep_index, payload, member?) + when (is_map(payload) or is_list(payload)) and is_function(member?, 1) do payload = Map.new(payload) new_txids = payload |> Map.get(:txids, []) |> MapSet.new() @@ -78,7 +82,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do Enum.map(existing_ins, &{:move_in, &1}) ++ payload_to_ops(payload) - {new_outs, new_ins} = reduce(ops, dep_view) + {new_outs, new_ins} = reduce(ops, member?) from_times = case Map.get(payload, :from_time) do @@ -159,7 +163,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do Enum.map(Map.get(payload, :move_in, []), &{:move_in, &1}) end - defp reduce(ops, base_view) do + defp reduce(ops, member?) do terminal_ops = ops |> Enum.with_index() @@ -167,7 +171,7 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do Map.put(acc, elem(move_value, 0), %{kind: kind, move_value: move_value, index: index}) end) |> Map.values() - |> Enum.reject(&redundant?(&1, base_view)) + |> Enum.reject(&redundant?(&1, member?)) |> Enum.sort_by(& &1.index) { @@ -176,12 +180,12 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueue do } end - defp redundant?(%{kind: :move_in, move_value: {value, _}}, base_view) do - MapSet.member?(base_view, value) + defp redundant?(%{kind: :move_in, move_value: {value, _}}, member?) do + member?.(value) end - defp redundant?(%{kind: :move_out, move_value: {value, _}}, base_view) do - not MapSet.member?(base_view, value) + defp redundant?(%{kind: :move_out, move_value: {value, _}}, member?) do + not member?.(value) end defp put_or_delete(map, key, [], _txids), do: Map.delete(map, key) diff --git a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs index 036692131c..a822f40671 100644 --- a/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs +++ b/packages/sync-service/test/electric/shapes/consumer/subqueries/move_queue_test.exs @@ -5,14 +5,16 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do @dep 0 + defp view(values), do: fn value -> value in values end + test "drops redundant move outs for values absent from the base view" do - queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_out: [{1, "1"}]}, MapSet.new()) + queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_out: [{1, "1"}]}, view([])) assert nil == MoveQueue.pop_next(queue) end test "drops redundant move ins for values already present in the base view" do - queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) + queue = MoveQueue.enqueue(MoveQueue.new(), @dep, %{move_in: [{1, "1"}]}, view([1])) assert nil == MoveQueue.pop_next(queue) end @@ -20,8 +22,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "cancels a pending move in with a later move out for the same value" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new()) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, view([])) assert nil == MoveQueue.pop_next(queue) end @@ -29,8 +31,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "cancels a pending move out with a later move in for the same value" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}]}, view([1])) assert nil == MoveQueue.pop_next(queue) end @@ -38,8 +40,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "merges repeated move ins and keeps the terminal tuple" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "01"}]}, MapSet.new()) - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], move_out: []}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "01"}]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], move_out: []}, view([])) assert {%{move_in_values: [{1, "1"}], move_out_values: []}, _queue} = MoveQueue.pop_next(queue) end @@ -47,8 +49,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "merges repeated move outs and keeps the terminal tuple" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "01"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}], move_in: []}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "01"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_out: [{1, "1"}], move_in: []}, view([1])) assert {%{move_in_values: [], move_out_values: [{1, "1"}]}, _queue} = MoveQueue.pop_next(queue) end @@ -56,8 +58,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "pop_next returns one combined batch per dep carrying both kinds" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, view([1])) assert { %{ @@ -77,12 +79,12 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do |> MoveQueue.enqueue( @dep, %{move_in: [{1, "1"}], from_time: 5, to_time: 6}, - MapSet.new() + view([]) ) |> MoveQueue.enqueue( @dep, %{move_in: [{2, "2"}], from_time: 6, to_time: 9}, - MapSet.new() + view([]) ) assert {%{from_time: 5, to_time: 9}, _queue} = MoveQueue.pop_next(queue) @@ -91,8 +93,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "accumulates txids from successive enqueues per dependency" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], txids: [10]}, MapSet.new()) - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], txids: [20]}, MapSet.new()) + |> MoveQueue.enqueue(@dep, %{move_in: [{1, "1"}], txids: [10]}, view([])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], txids: [20]}, view([])) assert {%{move_in_values: _, txids: [10, 20]}, _queue} = MoveQueue.pop_next(queue) end @@ -100,8 +102,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "pops the lowest-indexed dep first across deps" do queue = MoveQueue.new() - |> MoveQueue.enqueue(1, %{move_in: [{2, "2"}]}, MapSet.new()) - |> MoveQueue.enqueue(0, %{move_in: [{1, "1"}]}, MapSet.new()) + |> MoveQueue.enqueue(1, %{move_in: [{2, "2"}]}, view([])) + |> MoveQueue.enqueue(0, %{move_in: [{1, "1"}]}, view([])) assert {%{dep_index: 0}, queue} = MoveQueue.pop_next(queue) assert {%{dep_index: 1}, _queue} = MoveQueue.pop_next(queue) @@ -110,8 +112,8 @@ defmodule Electric.Shapes.Consumer.Subqueries.MoveQueueTest do test "length counts queued values across both batches" do queue = MoveQueue.new() - |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, MapSet.new([1])) - |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, MapSet.new([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{2, "2"}], move_out: [{1, "1"}]}, view([1])) + |> MoveQueue.enqueue(@dep, %{move_in: [{3, "3"}]}, view([1])) assert 3 == MoveQueue.length(queue) end From 25712758c39f55d74faa536e96c4013c709f9b13 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 11:32:59 +0100 Subject: [PATCH 38/40] Delete unused view_before_move/view_after_move helpers ActiveMove.view_before_move/2 and view_after_move/2 were defined but had no callers (production or tests). Splice and TransactionConverter already read MultiTimeView directly when they need membership at from_time / to_time. Drop the helpers and the now-unused MultiTimeView alias. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../shapes/consumer/subqueries/active_move.ex | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex index a6c2bc8295..7edb329597 100644 --- a/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex +++ b/packages/sync-service/lib/electric/shapes/consumer/subqueries/active_move.ex @@ -8,7 +8,6 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do alias Electric.Postgres.Lsn alias Electric.Replication.Changes.Transaction - alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @type move_value() :: {term(), term()} @@ -108,23 +107,6 @@ defmodule Electric.Shapes.Consumer.Subqueries.ActiveMove do def has_move_out?(%__MODULE__{move_out_values: []}), do: false def has_move_out?(%__MODULE__{move_out_values: _}), do: true - @doc """ - Materialise the dependency view as a `MapSet` at the active move's - `from_time`. The result is intended for transient use at the SplicePlan / - TransactionConverter boundary; do not retain it. - """ - @spec view_before_move(t(), MultiTimeView.t()) :: MapSet.t() - def view_before_move(%__MODULE__{subquery_id: id, from_time: t}, mtv), - do: mtv |> MultiTimeView.values(id, t) |> MapSet.new() - - @doc """ - Materialise the dependency view as a `MapSet` at the active move's - `to_time`. - """ - @spec view_after_move(t(), MultiTimeView.t()) :: MapSet.t() - def view_after_move(%__MODULE__{subquery_id: id, to_time: t}, mtv), - do: mtv |> MultiTimeView.values(id, t) |> MapSet.new() - @spec buffer_txn(t(), Transaction.t()) :: t() def buffer_txn(%__MODULE__{} = active_move, %Transaction{} = txn) do active_move From eb66e54dc0663486e365d00b194e54d216dd2402 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 14:22:55 +0100 Subject: [PATCH 39/40] Filter residual sublinks: conservative MTV-based answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `other_shape_matches?` previously used `subquery_member_unknown/0`, which raised on any sublink in the residual `and_where`. The runner errored out and the caller treated it as "include" — every record routed through any shape with a residual sublink. Use the same conservative MTV-based check the routing path uses (RFC §Routing): positive sublinks consult `MultiTimeView.member_at_some_time?/3`, negated sublinks consult `MultiTimeView.member_at_all_times?/3`. Values the MTV can prove are absent at every retained time (positive) or present at every retained time (negated) get pruned here instead of over-routing. Ambiguous values still over-route and the consumer's exact check refines. SubqueryIndex now stores `{dep_handle, polarity}` per `(shape_handle, dep_index)` so the new callback can pick the right MTV predicate per sublink. New public `SubqueryIndex.lookup_dep!/3` is the read path. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../shapes/filter/indexes/subquery_index.ex | 26 +++- .../electric/shapes/filter/where_condition.ex | 21 ++-- .../lib/electric/shapes/where_clause.ex | 63 +++++++--- .../test/electric/shapes/filter_test.exs | 117 ++++++++++++++++-- 4 files changed, 189 insertions(+), 38 deletions(-) diff --git a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex index 154002ec70..823fcdde82 100644 --- a/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex +++ b/packages/sync-service/lib/electric/shapes/filter/indexes/subquery_index.ex @@ -90,15 +90,35 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do """ @spec register_shape(t(), term(), DnfPlan.t(), [term()]) :: :ok def register_shape(%SubqueryIndex{table: table}, shape_handle, %DnfPlan{} = plan, dep_handles) do - for dep_index <- Map.keys(plan.dependency_polarities) do + for {dep_index, polarity} <- plan.dependency_polarities do dep_handle = Enum.at(dep_handles, dep_index) || {shape_handle, dep_index} - :ets.insert(table, {{:dep_handle, shape_handle, dep_index}, dep_handle}) + :ets.insert(table, {{:dep_handle, shape_handle, dep_index}, {dep_handle, polarity}}) end :ets.insert(table, {{:fallback, shape_handle}, true}) :ok end + @doc """ + Look up `{dep_handle, polarity}` for `{shape_handle, dep_index}`. + + Used by the filter's residual-`and_where` evaluator to ask + `MultiTimeView` a polarity-aware conservative membership question per + sublink. Raises `ArgumentError` if no dep was registered. + """ + @spec lookup_dep!(t(), term(), non_neg_integer()) :: {term(), :positive | :negated} + def lookup_dep!(%SubqueryIndex{table: table}, shape_handle, dep_index) do + case :ets.lookup(table, {:dep_handle, shape_handle, dep_index}) do + [{_, {dep_handle, polarity}}] -> + {dep_handle, polarity} + + [] -> + raise ArgumentError, + "no dep_handle registered for shape #{inspect(shape_handle)} dep_index " <> + inspect(dep_index) + end + end + @doc "Remove all metadata for `shape_handle`." @spec unregister_shape(t(), term()) :: :ok def unregister_shape(%SubqueryIndex{table: table}, shape_handle) do @@ -453,7 +473,7 @@ defmodule Electric.Shapes.Filter.Indexes.SubqueryIndex do defp lookup_dep_handle!(table, shape_handle, dep_index) do case :ets.lookup(table, {:dep_handle, shape_handle, dep_index}) do - [{_, dep_handle}] -> + [{_, {dep_handle, _polarity}}] -> dep_handle [] -> diff --git a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex index 84825e09e0..57d0203c7c 100644 --- a/packages/sync-service/lib/electric/shapes/filter/where_condition.ex +++ b/packages/sync-service/lib/electric/shapes/filter/where_condition.ex @@ -339,7 +339,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do end defp other_shapes_affected( - %Filter{where_cond_table: table}, + %Filter{subquery_index: index, where_cond_table: table}, condition_id, record ) do @@ -350,7 +350,7 @@ defmodule Electric.Shapes.Filter.WhereCondition do [shape_count: map_size(other_shapes)], fn -> for {{shape_id, _branch_key}, where} <- other_shapes, - other_shape_matches?(where, record), + other_shape_matches?(where, record, index, shape_id), into: MapSet.new() do shape_id end @@ -358,16 +358,19 @@ defmodule Electric.Shapes.Filter.WhereCondition do ) end - # The filter cannot evaluate subquery membership itself (see - # `WhereClause.subquery_member_unknown/0`). For non-subquery conjuncts in - # the residual `and_where` this still produces a precise answer; a sublink - # in the residual makes the runner error out, which we treat as "include" - # so the consumer does the exact check in `Shape.convert_change`. - defp other_shape_matches?(where, record) do + # Residual `and_where` evaluation. Non-subquery conjuncts evaluate + # precisely; sublinks consult `MultiTimeView` via + # `WhereClause.subquery_member_conservative_from_index/2` so that values + # provably outside (positive) or always inside (negated) the dep view are + # excluded here instead of over-routing. Anything ambiguous still routes + # through and is refined by the consumer's exact check in + # `Shape.convert_change`. If the runner can't compute a result we treat + # it as "include" — same fallback as before this MTV-based path landed. + defp other_shape_matches?(where, record, index, shape_id) do case WhereClause.includes_record_result( where, record, - WhereClause.subquery_member_unknown() + WhereClause.subquery_member_conservative_from_index(index, shape_id) ) do {:ok, included?} -> included? :error -> true diff --git a/packages/sync-service/lib/electric/shapes/where_clause.ex b/packages/sync-service/lib/electric/shapes/where_clause.ex index d2fdf2edd9..6510659e75 100644 --- a/packages/sync-service/lib/electric/shapes/where_clause.ex +++ b/packages/sync-service/lib/electric/shapes/where_clause.ex @@ -1,6 +1,7 @@ defmodule Electric.Shapes.WhereClause do alias PgInterop.Sublink alias Electric.Replication.Eval.Runner + alias Electric.Shapes.Filter.Indexes.SubqueryIndex alias Electric.Shapes.Filter.Indexes.SubqueryIndex.MultiTimeView @spec includes_record_result( @@ -45,24 +46,58 @@ defmodule Electric.Shapes.WhereClause do end @doc """ - Build a subquery_member? callback that signals "unknown" by raising. - - The filter cannot answer subquery membership correctly: a consumer that is - mid-move on a subquery is effectively reading at two logical times at once, - which a single per-shape pin cannot represent. So whenever the filter is - evaluating a residual `and_where` that contains a sublink, this callback - raises. The runner catches the raise and throws `:could_not_compute`, - which propagates as `:error` from `includes_record_result/3` — the caller - treats that as "over-route, let the consumer do the exact check in - `Shape.convert_change`". + Build a `subquery_member?` callback the filter can use while evaluating + a residual `and_where` for one outer shape, answering each sublink with + the conservative MTV-based test the routing path uses (see RFC + §Routing). + + - Positive sublink: returns `true` iff the value is a member at *some* + retained logical time (`MultiTimeView.member_at_some_time?/3`). + Surrounding `IN` evaluates `true` ⇒ include. Records whose value is + provably absent at every retained time get excluded here instead of + over-routing to every consumer for the exact check. + - Negated sublink: returns `true` iff the value is a member at *every* + retained logical time (`MultiTimeView.member_at_all_times?/3`). + Surrounding `NOT IN` evaluates `false` ⇒ exclude. Records whose value + is provably a member at every retained time get excluded. + - Anything in between (member at some retained times, not others) yields + a conservative-include in both polarities; the consumer's exact check + in `Shape.convert_change` refines further. + + The shape's polarity per `dep_index` is read from the SubqueryIndex's + `{:dep_handle, …}` rows, which `SubqueryIndex.register_shape/4` populates + with `{dep_handle, polarity}` at filter-add time. + + Raises if `shape_handle` has no row for the referenced `dep_index` or if + `subquery_ref` doesn't parse. The runner catches the raise and the + caller (`other_shape_matches?`) treats it as "include" — matching the + pre-existing failure mode for unknown sublinks. """ - @spec subquery_member_unknown() :: ([String.t()], term() -> boolean()) - def subquery_member_unknown do - fn _subquery_ref, _typed_value -> - raise "subquery membership cannot be evaluated at the filter; consumer must check" + @spec subquery_member_conservative_from_index(SubqueryIndex.t(), term()) :: + ([String.t()], term() -> boolean()) + def subquery_member_conservative_from_index(%SubqueryIndex{} = index, shape_handle) do + mtv = index.multi_time_view + + fn subquery_ref, typed_value -> + {:ok, dep_index} = dep_index_from_ref(subquery_ref) + {dep_handle, polarity} = SubqueryIndex.lookup_dep!(index, shape_handle, dep_index) + + case polarity do + :positive -> MultiTimeView.member_at_some_time?(mtv, dep_handle, typed_value) + :negated -> MultiTimeView.member_at_all_times?(mtv, dep_handle, typed_value) + end + end + end + + defp dep_index_from_ref([_prefix, idx]) when is_binary(idx) do + case Integer.parse(idx) do + {n, ""} -> {:ok, n} + _ -> :error end end + defp dep_index_from_ref(_), do: :error + @doc """ Build a subquery_member? callback that reads `MultiTimeView` at the per-ref logical time given by `subquery_refs`. The optional diff --git a/packages/sync-service/test/electric/shapes/filter_test.exs b/packages/sync-service/test/electric/shapes/filter_test.exs index 26c2973b3b..81af5ad0d1 100644 --- a/packages/sync-service/test/electric/shapes/filter_test.exs +++ b/packages/sync-service/test/electric/shapes/filter_test.exs @@ -930,7 +930,7 @@ defmodule Electric.Shapes.FilterTest do index = Filter.subquery_index(filter) mtv = index.multi_time_view dep_index = subquery_ref |> List.last() |> String.to_integer() - [{_, subquery_id}] = :ets.lookup(index.table, {:dep_handle, shape_id, dep_index}) + {subquery_id, _polarity} = SubqueryIndex.lookup_dep!(index, shape_id, dep_index) MultiTimeView.init_subquery(mtv, subquery_id, values) MultiTimeView.mark_ready(mtv, subquery_id) @@ -1174,15 +1174,17 @@ defmodule Electric.Shapes.FilterTest do "CREATE TABLE IF NOT EXISTS or_parent (id INT PRIMARY KEY)", "CREATE TABLE IF NOT EXISTS or_child (id INT PRIMARY KEY, par_id INT REFERENCES or_parent(id), value TEXT NOT NULL)" ] - test "mixed OR+subquery shape over-routes through other_shapes", %{ - inspector: inspector - } do + test "mixed OR+subquery shape filters through other_shapes via the MTV conservative callback", + %{ + inspector: inspector + } do # A mixed OR shape lands in the catch-all `other_shapes` path because # neither side of the OR is individually optimisable. The filter - # cannot evaluate the sublink (it would need a consumer's logical - # time, which it intentionally doesn't track), so any record reaches - # the shape and the consumer's `Shape.convert_change` makes the final - # decision. + # evaluates the residual using + # `WhereClause.subquery_member_conservative_from_index/2`, which + # consults MTV with `member_at_some_time?` for the positive sublink + # — so values provably outside the retained window are pruned here + # instead of over-routing. {:ok, shape} = Shape.new("or_child", inspector: inspector, @@ -1220,10 +1222,9 @@ defmodule Electric.Shapes.FilterTest do record: %{"id" => "50", "par_id" => "99", "value" => "other"} } - # Over-routed: the filter cannot resolve the sublink at the OR's left - # branch without a per-consumer logical time, so it includes shape1 - # and lets the consumer drop the record. - assert Filter.affected_shapes(filter, insert_no_match) == MapSet.new(["shape1"]) + # Conservatively pruned: par_id=99 is absent from MTV at every + # retained time and the LIKE branch doesn't match. + assert Filter.affected_shapes(filter, insert_no_match) == MapSet.new([]) end @tag with_sql: [ @@ -1624,5 +1625,97 @@ defmodule Electric.Shapes.FilterTest do assert Filter.affected_shapes(filter, insert_other) == MapSet.new([]) end + + @tag with_sql: [ + "CREATE TABLE IF NOT EXISTS neg_res_parent (id INT PRIMARY KEY)", + "CREATE TABLE IF NOT EXISTS neg_res_child (id INT PRIMARY KEY, name TEXT NOT NULL, par_id INT REFERENCES neg_res_parent(id))" + ] + test "residual NOT IN sublink prunes via member_at_all_times? when value is always a member", + %{inspector: inspector} do + # `LIKE … OR NOT IN (…)` is unoptimisable (LIKE isn't indexable), so + # the shape lands in the `other_shapes` path with a negated sublink in + # the residual. The conservative callback uses + # `MultiTimeView.member_at_all_times?/3` so values provably present + # at every retained time get excluded here (NOT IN is false at every + # consumer time) instead of over-routing. + {:ok, shape} = + Shape.new("neg_res_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id NOT IN (SELECT id FROM neg_res_parent)" + ) + + filter = Filter.new() |> Filter.add_shape("shape1", shape) + index = Filter.subquery_index(filter) + subquery_ref = ["$sublink", "0"] + + # Seed MTV with values that are always-members (empty history). + seed_membership(filter, "shape1", shape, subquery_ref, MapSet.new([1, 2])) + SubqueryIndex.mark_ready(index, "shape1") + + # par_id=1 is in MTV at every retained time → NOT IN is false → exclude. + insert_always_member = %NewRecord{ + relation: {"public", "neg_res_child"}, + record: %{"id" => "10", "par_id" => "1", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_always_member) == MapSet.new([]) + + # par_id=99 is unknown to MTV → conservatively-not-a-member → NOT IN is + # true → include. + insert_unknown = %NewRecord{ + relation: {"public", "neg_res_child"}, + record: %{"id" => "11", "par_id" => "99", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_unknown) == MapSet.new(["shape1"]) + end + + @tag with_sql: [ + "CREATE TABLE IF NOT EXISTS amb_parent (id INT PRIMARY KEY)", + "CREATE TABLE IF NOT EXISTS amb_child (id INT PRIMARY KEY, name TEXT NOT NULL, par_id INT REFERENCES amb_parent(id))" + ] + test "ambiguous value (in at some retained times, out at others) routes conservatively in both polarities", + %{inspector: inspector} do + # A value with a non-empty MTV history sits between always-in and + # always-out — `member_at_some_time?` is true *and* + # `member_at_all_times?` is false. Both polarities of conservative + # callback should err on the side of include. + {:ok, pos_shape} = + Shape.new("amb_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id IN (SELECT id FROM amb_parent)" + ) + + {:ok, neg_shape} = + Shape.new("amb_child", + inspector: inspector, + where: "name LIKE 'never%' OR par_id NOT IN (SELECT id FROM amb_parent)" + ) + + filter = + Filter.new() + |> Filter.add_shape("pos", pos_shape) + |> Filter.add_shape("neg", neg_shape) + + index = Filter.subquery_index(filter) + mtv = index.multi_time_view + + # Build an ambiguous history for value 1: in at time 0, out from time 1. + {dep_handle, :positive} = SubqueryIndex.lookup_dep!(index, "pos", 0) + MultiTimeView.init_subquery(mtv, dep_handle, [1]) + MultiTimeView.mark_out(mtv, dep_handle, 1, 1) + MultiTimeView.mark_ready(mtv, dep_handle) + + SubqueryIndex.mark_ready(index, "pos") + SubqueryIndex.mark_ready(index, "neg") + + # par_id=1 ambiguous → conservative include for both polarities. + insert_ambiguous = %NewRecord{ + relation: {"public", "amb_child"}, + record: %{"id" => "10", "par_id" => "1", "name" => "x"} + } + + assert Filter.affected_shapes(filter, insert_ambiguous) == MapSet.new(["pos", "neg"]) + end end end From 008612fc9978a6931b0ccacea714184c391265e8 Mon Sep 17 00:00:00 2001 From: rob Date: Wed, 27 May 2026 16:23:58 +0100 Subject: [PATCH 40/40] Add changset --- .changeset/subquery-index.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/subquery-index.md diff --git a/.changeset/subquery-index.md b/.changeset/subquery-index.md new file mode 100644 index 0000000000..711e4f75eb --- /dev/null +++ b/.changeset/subquery-index.md @@ -0,0 +1,5 @@ +--- +'@core/sync-service': patch +--- + +New subquery index that reduces memory footprint and solves lag issues due to slow shape removal.