diff --git a/pages/clustering/high-availability/ha-commands-reference.mdx b/pages/clustering/high-availability/ha-commands-reference.mdx index 434aae715..518ded122 100644 --- a/pages/clustering/high-availability/ha-commands-reference.mdx +++ b/pages/clustering/high-availability/ha-commands-reference.mdx @@ -671,17 +671,37 @@ SET COORDINATOR SETTING 'max_replica_read_lag' TO '10' ; ### `deltas_batch_progress_size` -Users can control how often REPLICAs report back to the MAIN that they're still processing the data (transactions, WALs, snapshots) -the MAIN has sent to them. The default value is 100'000, which should be enough for most of your transactions. -However, if processing 100'000 deltas takes more than 30s (because you're dealing with large deltas or you have older CPUs), -you can set the configuration value `deltas_batch_progress_size` to a smaller value. This will avoid timeouts on REPLICAs so -you won't see the query exception "At least one SYNC replica has not committed", but at the cost of lower throughput since REPLICAs -will be sending in-progress messages to the MAIN more often. + +**Deprecated in 3.13: this setting no longer has any effect.** REPLICAs report +progress on a fixed time interval instead of after a fixed number of deltas, so +there is nothing left to tune. The setting remains readable and settable so that +existing configurations and automation keep working across an upgrade, but +changing it does not alter replication behavior. It will be removed in a future +release. + + +Previously, this setting controlled how many deltas a REPLICA processed before +reporting back to the MAIN that it is still working on the data (transactions, +WALs, snapshots) the MAIN sent it. Counting deltas could not cover work that +happens *inside* a single delta — populating an index, validating a constraint, +clearing storage before a snapshot load, or aborting an interrupted two-phase +commit. On a large dataset any one of those can run for minutes while the +REPLICA reports nothing, so the MAIN would hit its RPC timeout and drop the +connection mid-build, and the REPLICA would never converge. + +As of 3.13, a REPLICA emits an in-progress message on a fixed interval for as +long as the operation keeps making progress, regardless of how that work is +divided into deltas. An operation that stops progressing stops reporting, so a +genuinely stuck REPLICA is still caught by the MAIN's timeout rather than masked +by an unconditional keepalive. ``` SET COORDINATOR SETTING 'deltas_batch_progress_size' TO '50000'; ``` +The query above still succeeds and the value is still returned by `SHOW +COORDINATOR SETTINGS`, but it is ignored. + ### `global_read_only` Puts the **entire cluster** into a read-only state. When enabled, the current diff --git a/pages/clustering/replication/how-replication-works.mdx b/pages/clustering/replication/how-replication-works.mdx index a36bb4970..82e8acbe8 100644 --- a/pages/clustering/replication/how-replication-works.mdx +++ b/pages/clustering/replication/how-replication-works.mdx @@ -377,6 +377,40 @@ original role. ![](/pages/clustering/replication/memgraph-replication-ids.png) +### Progress reporting during long recovery steps + +When the MAIN sends a REPLICA a transaction, a set of WAL files, or a snapshot, +it waits a bounded amount of time for the REPLICA's response before dropping the +connection. Some of the work a REPLICA does while answering such a request is not +divided into small steps: populating an index, validating a constraint, loading a +snapshot, clearing storage before a reset, and aborting an interrupted two-phase +commit all walk the entire dataset and can run for minutes on a large tenant. + +To keep those requests alive, a REPLICA emits an in-progress message on a fixed +interval for as long as the operation it is running keeps advancing. The message +carries no data — it only tells the MAIN that the REPLICA is still working, so +the MAIN extends its wait instead of closing the socket. Reporting is tied to +actual advancement rather than to wall-clock time alone: an operation that stops +progressing stops reporting, so a REPLICA that is genuinely stuck is still caught +by the MAIN's timeout rather than kept alive indefinitely by a blind keepalive. + +If the MAIN disappears while a REPLICA is populating an index or validating a +constraint, the REPLICA abandons that build instead of finishing work whose +result can no longer be delivered. Loading a snapshot is the exception: the data +is already on disk at that point, so the REPLICA finishes deriving indices and +constraints from it. That advances its commit timestamp, which means a +reconnecting MAIN may only need to ship WAL deltas from that point rather than +the whole snapshot again. + + + +Progress reporting used to be driven by the number of processed deltas, tunable +through the `deltas_batch_progress_size` coordinator setting. That setting has +had no effect since 3.13. See the [HA commands +reference](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size). + + + ### System data replication We have outlined in the main section of this guide how graph data replication diff --git a/pages/help-center/errors/high-availability.mdx b/pages/help-center/errors/high-availability.mdx index 9df6096e3..1c71064b5 100644 --- a/pages/help-center/errors/high-availability.mdx +++ b/pages/help-center/errors/high-availability.mdx @@ -32,7 +32,14 @@ Below are common causes and how to resolve them: diverged, manual recovery or a force sync may be needed. See the [force sync documentation](/clustering/high-availability/how-high-availability-works#replication-scenarios). 4. **RPC timeout** — If the error mentions an RPC timeout, the replica may be - overloaded or the network latency is too high. Consider adjusting the - `deltas_batch_progress_size` coordinator setting. + overloaded, the network latency may be too high, or the replica may have + stopped making progress altogether. The error message may suggest adjusting + the `deltas_batch_progress_size` coordinator setting; as of 3.13 [that + setting no longer has any + effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size), + because replicas now report progress on a fixed time interval and long single + operations (index population, constraint validation, snapshot loading, + storage clearing) are covered by that reporting. Check the replica's logs and + load instead. diff --git a/pages/help-center/errors/replication.mdx b/pages/help-center/errors/replication.mdx index 48f5ae31c..2db65c711 100644 --- a/pages/help-center/errors/replication.mdx +++ b/pages/help-center/errors/replication.mdx @@ -88,8 +88,15 @@ Failed replicas will be recovered automatically. Check the status of replicas using the `SHOW REPLICAS` query. -If the failure reason is an RPC timeout, the error message includes additional -guidance about adjusting the `deltas_batch_progress_size` coordinator setting. +If the failure reason is an RPC timeout, the error message may include guidance +about adjusting the `deltas_batch_progress_size` coordinator setting. As of 3.13 +[that setting no longer has any +effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size) +— replicas report progress on a fixed time interval instead of after a fixed +number of deltas, and long single operations such as index population, +constraint validation and snapshot loading are covered by that reporting. An RPC +timeout on 3.13 or later points to an overloaded replica, network latency, or a +replica that genuinely stopped making progress, not to a tuning problem. ## Snapshots are disabled for replicas [#warning-1] diff --git a/pages/release-notes.mdx b/pages/release-notes.mdx index 43dacaa9b..a1ddbc533 100644 --- a/pages/release-notes.mdx +++ b/pages/release-notes.mdx @@ -92,6 +92,18 @@ guide. {

🐞 Bug fixes

} +- Replicas no longer time out during recovery steps that run long inside a + single delta, such as populating an index, validating a constraint, loading a + snapshot, clearing storage before a reset, or aborting an interrupted + two-phase commit. A replica now reports progress on a fixed time interval for + as long as the work keeps advancing, instead of after a fixed number of + processed deltas, so the main no longer drops the connection mid-build and + leaves the replica unable to converge. A replica that stops advancing stops + reporting, so a genuinely stuck replica is still caught by the main's timeout. + As a consequence, the `deltas_batch_progress_size` coordinator setting **no + longer has any effect**; it stays readable and settable so existing + configurations keep working across an upgrade, and will be removed in a future + release. [#4557](https://github.com/memgraph/memgraph/pull/4557) - Text and vector search now respect fine-grained label and property permissions. Hits the caller cannot read are filtered out, and `text_search.aggregate` / `text_search.aggregate_edges` return an error unless