Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 26 additions & 6 deletions pages/clustering/high-availability/ha-commands-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -671,17 +671,37 @@ SET COORDINATOR SETTING 'max_replica_read_lag' TO '10' ;

### `deltas_batch_progress_size`

Users can control how often REPLICAs report back to the MAIN that they're still processing the data (transactions, WALs, snapshots)
the MAIN has sent to them. The default value is 100'000, which should be enough for most of your transactions.
However, if processing 100'000 deltas takes more than 30s (because you're dealing with large deltas or you have older CPUs),
you can set the configuration value `deltas_batch_progress_size` to a smaller value. This will avoid timeouts on REPLICAs so
you won't see the query exception "At least one SYNC replica has not committed", but at the cost of lower throughput since REPLICAs
will be sending in-progress messages to the MAIN more often.
<Callout type="warning">
**Deprecated in 3.13: this setting no longer has any effect.** REPLICAs report
progress on a fixed time interval instead of after a fixed number of deltas, so
there is nothing left to tune. The setting remains readable and settable so that
existing configurations and automation keep working across an upgrade, but
changing it does not alter replication behavior. It will be removed in a future
release.
</Callout>

Previously, this setting controlled how many deltas a REPLICA processed before
reporting back to the MAIN that it is still working on the data (transactions,
WALs, snapshots) the MAIN sent it. Counting deltas could not cover work that
happens *inside* a single delta — populating an index, validating a constraint,
clearing storage before a snapshot load, or aborting an interrupted two-phase
commit. On a large dataset any one of those can run for minutes while the
REPLICA reports nothing, so the MAIN would hit its RPC timeout and drop the
connection mid-build, and the REPLICA would never converge.

As of 3.13, a REPLICA emits an in-progress message on a fixed interval for as
long as the operation keeps making progress, regardless of how that work is
divided into deltas. An operation that stops progressing stops reporting, so a
genuinely stuck REPLICA is still caught by the MAIN's timeout rather than masked
by an unconditional keepalive.

```
SET COORDINATOR SETTING 'deltas_batch_progress_size' TO '50000';
```

The query above still succeeds and the value is still returned by `SHOW
COORDINATOR SETTINGS`, but it is ignored.

### `global_read_only`

Puts the **entire cluster** into a read-only state. When enabled, the current
Expand Down
34 changes: 34 additions & 0 deletions pages/clustering/replication/how-replication-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,40 @@ original role.

![](/pages/clustering/replication/memgraph-replication-ids.png)

### Progress reporting during long recovery steps

When the MAIN sends a REPLICA a transaction, a set of WAL files, or a snapshot,
it waits a bounded amount of time for the REPLICA's response before dropping the
connection. Some of the work a REPLICA does while answering such a request is not
divided into small steps: populating an index, validating a constraint, loading a
snapshot, clearing storage before a reset, and aborting an interrupted two-phase
commit all walk the entire dataset and can run for minutes on a large tenant.

To keep those requests alive, a REPLICA emits an in-progress message on a fixed
interval for as long as the operation it is running keeps advancing. The message
carries no data — it only tells the MAIN that the REPLICA is still working, so
the MAIN extends its wait instead of closing the socket. Reporting is tied to
actual advancement rather than to wall-clock time alone: an operation that stops
progressing stops reporting, so a REPLICA that is genuinely stuck is still caught
by the MAIN's timeout rather than kept alive indefinitely by a blind keepalive.

If the MAIN disappears while a REPLICA is populating an index or validating a
constraint, the REPLICA abandons that build instead of finishing work whose
result can no longer be delivered. Loading a snapshot is the exception: the data
is already on disk at that point, so the REPLICA finishes deriving indices and
constraints from it. That advances its commit timestamp, which means a
reconnecting MAIN may only need to ship WAL deltas from that point rather than
the whole snapshot again.

<Callout type="info">

Progress reporting used to be driven by the number of processed deltas, tunable
through the `deltas_batch_progress_size` coordinator setting. That setting has
had no effect since 3.13. See the [HA commands
reference](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size).

</Callout>

### System data replication

We have outlined in the main section of this guide how graph data replication
Expand Down
11 changes: 9 additions & 2 deletions pages/help-center/errors/high-availability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,14 @@ Below are common causes and how to resolve them:
diverged, manual recovery or a force sync may be needed. See the
[force sync documentation](/clustering/high-availability/how-high-availability-works#replication-scenarios).
4. **RPC timeout** — If the error mentions an RPC timeout, the replica may be
overloaded or the network latency is too high. Consider adjusting the
`deltas_batch_progress_size` coordinator setting.
overloaded, the network latency may be too high, or the replica may have
stopped making progress altogether. The error message may suggest adjusting
the `deltas_batch_progress_size` coordinator setting; as of 3.13 [that
setting no longer has any
effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size),
because replicas now report progress on a fixed time interval and long single
operations (index population, constraint validation, snapshot loading,
storage clearing) are covered by that reporting. Check the replica's logs and
load instead.

<CommunityLinks/>
11 changes: 9 additions & 2 deletions pages/help-center/errors/replication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,15 @@ Failed replicas will be recovered automatically. Check the status of replicas
using the `SHOW REPLICAS` query.
</Callout>

If the failure reason is an RPC timeout, the error message includes additional
guidance about adjusting the `deltas_batch_progress_size` coordinator setting.
If the failure reason is an RPC timeout, the error message may include guidance
about adjusting the `deltas_batch_progress_size` coordinator setting. As of 3.13
[that setting no longer has any
effect](/clustering/high-availability/ha-commands-reference#deltas_batch_progress_size)
— replicas report progress on a fixed time interval instead of after a fixed
number of deltas, and long single operations such as index population,
constraint validation and snapshot loading are covered by that reporting. An RPC
timeout on 3.13 or later points to an overloaded replica, network latency, or a
replica that genuinely stopped making progress, not to a tuning problem.

## Snapshots are disabled for replicas [#warning-1]

Expand Down
12 changes: 12 additions & 0 deletions pages/release-notes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,18 @@ guide.

{<h4 className="custom-header">🐞 Bug fixes</h4>}

- Replicas no longer time out during recovery steps that run long inside a
single delta, such as populating an index, validating a constraint, loading a
snapshot, clearing storage before a reset, or aborting an interrupted
two-phase commit. A replica now reports progress on a fixed time interval for
as long as the work keeps advancing, instead of after a fixed number of
processed deltas, so the main no longer drops the connection mid-build and
leaves the replica unable to converge. A replica that stops advancing stops
reporting, so a genuinely stuck replica is still caught by the main's timeout.
As a consequence, the `deltas_batch_progress_size` coordinator setting **no
longer has any effect**; it stays readable and settable so existing
configurations keep working across an upgrade, and will be removed in a future
release. [#4557](https://github.com/memgraph/memgraph/pull/4557)
- Text and vector search now respect fine-grained label and property
permissions. Hits the caller cannot read are filtered out, and
`text_search.aggregate` / `text_search.aggregate_edges` return an error unless
Expand Down