From 95ecb8bb665ad3510c4e73ca454aa76217782ea9 Mon Sep 17 00:00:00 2001 From: David Smiley Date: Thu, 6 Aug 2026 14:08:37 -0400 Subject: [PATCH] Add SolrCloud update consistency documentation - New ref guide page solrcloud-update-consistency.adoc stating the consistency model of /update in SolrCloud: acknowledgment semantics, durability, ordering, atomicity, visibility, optimistic concurrency, retry-ability, and leader failover - New dev-docs/distributed-update-internals.adoc: implementation deep-dive of the distributed update path (routing, fan-out, acknowledgment and error handling, shard terms) and versioning/optimistic concurrency, with an eye toward reasoning about idempotency and retries - AGENTS.md: add a Developer Docs Index so coding agents discover dev-docs/ when working on related topics Co-Authored-By: Claude Fable 5 --- AGENTS.md | 20 ++ dev-docs/distributed-update-internals.adoc | 232 ++++++++++++++++++ .../deployment-guide/deployment-nav.adoc | 1 + ...rcloud-recoveries-and-write-tolerance.adoc | 1 + .../pages/solrcloud-shards-indexing.adoc | 1 + .../pages/solrcloud-update-consistency.adoc | 94 +++++++ 6 files changed, 349 insertions(+) create mode 100644 dev-docs/distributed-update-internals.adoc create mode 100644 solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc diff --git a/AGENTS.md b/AGENTS.md index 5a6ce2d70b8c..3f130f0238f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,6 +51,26 @@ While README.md and CONTRIBUTING.md are mainly written for humans, this file is - New classes should have some javadocs - Changes should not have code comments communicating the change, which are instead great comments to leave for code review / commentary +## Developer Docs Index + +Before diving into code on these topics, read the matching doc in `dev-docs/`. When adding a new dev doc, add a line here. + +Internals: + +- `dev-docs/overseer/overseer.adoc` — Overseer: cluster state updates, ZkStateWriter, collection API message flow +- `dev-docs/shard-split/shard-split.adoc` — SPLITSHARD: shard/replica states, tlog buffering during split +- `dev-docs/distributed-update-internals.adoc` — SolrCloud update path: routing, `_version_`/optimistic concurrency, tlog durability, replication acks, shard terms (user-facing consistency model: ref-guide page `solrcloud-update-consistency.adoc`) +- `dev-docs/plugins-modules-packages.adoc` — plugin/module/package concepts +- `dev-docs/apis.adoc`, `dev-docs/v2-api-conventions.adoc` — API design and v2 conventions +- `dev-docs/ui/` — new Admin UI architecture, component development, testing + +Process & tooling: + +- `dev-docs/solr-source-code.adoc`, `git.adoc`, `IDEs.adoc`, `jvms.adoc` — build and dev environment +- `dev-docs/ref-guide/` — ref-guide authoring (AsciiDoc syntax, Antora templates) +- `dev-docs/dependency-upgrades.adoc`, `lucene-upgrade.md`, `working-between-major-versions.adoc` — upgrades and branch management +- `dev-docs/releasing.adoc`, `changelog.adoc`, `asf-jenkins.adoc` — release and CI process + ## Changelog - We use the "logchange" tooling to manage our changelog. See `dev-docs/changelog.adoc` for details and conventions diff --git a/dev-docs/distributed-update-internals.adoc b/dev-docs/distributed-update-internals.adoc new file mode 100644 index 000000000000..69aaf2284f3d --- /dev/null +++ b/dev-docs/distributed-update-internals.adoc @@ -0,0 +1,232 @@ += Distributed Update Internals (SolrCloud) +:toc: macro +:toclevels: 3 + +toc::[] + +== Why this doc + +The Solr Reference Guide states the user-facing consistency model of SolrCloud updates in +https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc[SolrCloud Update Consistency Model]. +This document explains *how* those guarantees are implemented: the distributed update path from the receiving node through the shard leader to the replicas, and the versioning scheme that everything else leans on. +A particular aim is to give enough precision to reason about idempotency and retry-ability of updates. + +It was written from reading the source of the `main` branch in August 2026. +Things might be off or drift out of date — trust the code over this document, and please fix what you find wrong. + +Class names below are under `solr/core/src/java/org/apache/solr/` unless otherwise noted. + +== Request lifecycle + +An `/update` request is parsed by a loader into a stream of `AddUpdateCommand` / `DeleteUpdateCommand` / `CommitUpdateCommand` objects, each fed through the update request processor (URP) chain. +Everything discussed here happens inside two processors near the end of that chain: `update.processor.DistributedZkUpdateProcessor` (subclass of `DistributedUpdateProcessor`; "DUP" below) and `update.processor.RunUpdateProcessor`. +URPs configured *before* the DUP run only on the node that received the client request; URPs *after* it (and `RunUpdateProcessor`) run on the leader and again on every replica. + +[source,mermaid] +---- +sequenceDiagram + participant C as Client + participant N as Receiving node + participant L as Shard leader + participant R as NRT/TLOG replicas + + C->>N: /update (batch of docs) + N->>L: forward each doc (DistribPhase=TOLEADER) + L->>L: per-doc lock; OCC check; assign _version_ + L->>L: Lucene write + tlog append + L--)R: stream doc (DistribPhase=FROMLEADER, async) + R->>R: drop if stale version, else apply + tlog + L->>L: finish(): await replica responses,
demote failed replicas via shard terms,
flush tlog + L-->>N: per-request response (+ achieved rf) + N-->>C: HTTP 200 (or error) +---- + +=== Routing: setupRequest + +`DistributedZkUpdateProcessor.setupRequest()` decides, per document, what role this node plays. +It computes the target slice via the collection's `DocRouter`, looks up the shard leader with `ZkStateReader.getLeaderRetry(...)`, and compares it to the local core. +The `DISTRIB_UPDATE_PARAM` (`update.distrib`) carries a `DistribPhase` marking where the request came from: + +* `NONE` — an external client request; if this node is not the leader, set `forwardToLeader` and target a single `SolrCmdDistributor.ForwardNode` (the leader). +* `TOLEADER` — forwarded from a peer; this node is (should be) the leader; compute the replica fan-out list. +* `FROMLEADER` — forwarded from the leader; apply locally only, no further distribution. + +The fan-out list comes from `getReplicaNodesForLeader(...)`: only `NRT` and `TLOG` replicas (PULL replicas never receive updates), excluding down/non-live replicas and any replica whose shard term is already behind the leader's (`ZkShardTerms.skipSendingUpdatesTo`) — those are collected in `skippedCoreNodeNames` and will have their terms pushed further down at request end. + +Retry limits for internal hops are asymmetric: forwarding to a leader retries up to `solr.retries.on.forward` (default 25) times, while a leader sending to followers retries only `solr.retries.to.followers` (default 3) times — a follower that can't take the update is demoted instead of retried hard. + +=== Leader-side processing + +For an add, `DistributedUpdateProcessor.versionAdd` runs the whole per-document decision inside `UpdateLocks.runWithLock(id, ...)` — see <>. +`leaderLogic` is true when this core is the leader and the command is not a replay/peer-sync (`leaderLogicWithVersionIntegrityCheck`); a non-leader receiving an update *without* a version is rejected as an invalid state. + +On the leader: + +1. If the command carries a client-supplied version constraint, perform the optimistic-concurrency check (see <>). +2. If the command is an atomic (partial) update, resolve it into a full replacement document (`getUpdatedDocument` → `AtomicUpdateDocumentMerger`), reading the current document through `RealTimeGetComponent.getInputDocument` — which sees uncommitted state via the tlog. +3. Assign a new version: `cmd.setVersion(vinfo.getNewClock())` (see <>). +4. Apply locally: `doLocalAdd` → `RunUpdateProcessor` → `DirectUpdateHandler2.addDoc` (Lucene `updateDocument`) and `UpdateLog.add` (tlog append). +5. Hand the (now fully-resolved, versioned) document to `doDistribAdd` for fan-out. + +Because steps 1–4 happen under the per-document lock, concurrent updates to the same id are serialized at the leader, read-modify-write atomic updates are linearizable per document, and version assignment order matches apply order per document. + +=== Fan-out: SolrCmdDistributor and StreamingSolrClients + +`update.SolrCmdDistributor.distribAdd/distribDelete` submits each command to the target nodes. +Ordinary adds/deletes are *fire-and-forget at the per-document level*: `update.StreamingSolrClients` maintains one `ConcurrentUpdateJettySolrClient` per destination URL (queue size 100, deliberately low thread count — the class comments that more threads "could cause updates to be reordered on a greater scale"), and the document is queued into its stream without waiting for a response. + +Exceptions that are sent synchronously (blocking per command): in-place updates (a dependent in-place update must not overtake its predecessor in a stream), and forwards to sub-shard leaders or routing-rule targets during shard split / migrate. + +The leader therefore does not know a document's replica outcome at the time it processes the next document. +All outcomes are collected at request end. + +=== Request finish: acknowledgment and error triage + +When the loader has fed all commands, `DistributedUpdateProcessor.finish()` runs `doDistribFinish()` and then `RunUpdateProcessor.finish()`. +`DistributedZkUpdateProcessor.doDistribFinish()` is where the acknowledgment semantics live: + +1. If this leader changed its index and skipped any known-stale replicas, bump terms now: `ZkShardTerms.ensureTermsIsHigher(leader, skippedCoreNodeNames)`. +2. `cmdDistrib.finish()` — *block* until every queued replica request has completed (this is the only wait for replica responses in the whole path). +3. Walk `cmdDistrib.getErrors()` and triage: + * Error on a `ForwardNode` (this node → leader): added to `errorsForClient`; the client sees the failure and may retry. + * Error on a `StdNode` (leader → follower): *not* a client error — the code comments "for now we don't error - we assume if it was added locally, we succeeded". + Unless the error is a commit (`commit_end_point` requests never trigger recovery) the follower's coreNodeName is collected for demotion — after double-checking against ZK that we are still the leader and the errored node is still one of our replicas. + * Special case: if the remote error's metadata says `cause=LeaderChanged` (SOLR-6511 — the "follower" now believes it is the leader), the error *is* propagated to the client so it can retry against the new leader. +4. `ensureTermsIsHigher(leader, replicasShouldBeInLowerTerms)` — the demotion. + The demoted replica's term watcher notices it is behind and puts the core into recovery. + This term mechanism (SOLR-11702) is the replacement for the old znode-based "leader-initiated recovery" (LIR); no LIR znodes exist anymore. +5. Compute the achieved replication factor: each shard leader counts itself plus each follower that acked (`LeaderRequestReplicationTracker`), the originating node takes the minimum across shards (`RollupRequestReplicationTracker`), and the result is reported as `rf` in the response header. + It is purely informational; a top-of-class TODO ("optionally fail if n replicas are not reached...") records the unimplemented alternative. + +Client-visible errors are aggregated into `DistributedUpdatesAsyncException` (status: the common code if all agree, else 400 if all 4xx, else 500). + +Finally `RunUpdateProcessor.finish()` calls `UpdateLog.finish(null)` — the per-request tlog flush described next. + +=== Durability: UpdateLog and TransactionLog + +`update.UpdateLog` owns a current `update.TransactionLog` (tlog) plus recent old ones, and an in-memory map from doc id to a `LogPtr` into the tlog — the map that makes uncommitted documents visible to realtime get and to atomic-update resolution. + +Writes: `UpdateLog.add/delete/deleteByQuery` append a record to the current tlog through a buffered `FastOutputStream` over a file channel. +*Nothing is flushed per document.* +Durability happens per *request*, in `TransactionLog.finish(syncLevel)`: + +* `NONE` — do nothing. +* `FLUSH` (the default) — flush the JVM buffer to the OS; survives a JVM crash / `kill -9`, but not an OS crash or power loss. +* `FSYNC` — additionally `channel.force(true)`. + The fsync is deliberately outside the buffer lock; the code notes a partial last record after power failure is expected and tolerated by the reader. + +`syncLevel` is configured on `` in `solrconfig.xml`. +So the ref-guide statement "documents are written to the tlog before the indexing call returns" is true, but with default `FLUSH` the response does not imply the bytes reached the disk platter. + +Commits rotate the tlog: `UpdateLog.preCommit` starts a new tlog (so the old one is definitely fully covered by the index commit), and `postCommit` writes a commit marker into the old one. +On startup, `UpdateLog.recoverFromLog()` replays any tlog tail not covered by a commit — this is what makes acked-but-uncommitted updates survive a restart. +Retention is bounded by `numRecordsToKeep` (default 100) and `maxNumLogsToKeep` (default 10), which also bound how far a replica can fall behind before PeerSync is impossible and full replication is required. + +The UpdateLog also has a state machine (`ACTIVE`, `BUFFERING`, `APPLYING_BUFFERED`, `REPLAYING`) used during recovery and shard split: while a core is recovering, incoming `FROMLEADER` updates are written to a separate buffer tlog *without* being applied, and replayed at the end (`applyBufferedUpdates`). +State transitions quiesce all in-flight updates through `UpdateLocks.blockUpdates()` (the write side of a fair read/write lock; every normal update holds the read side). +See `dev-docs/shard-split/shard-split.adoc` for the shard-split use of buffering. + +=== Replica-side processing + +A replica receiving `DistribPhase.FROMLEADER` runs the same `versionAdd`/`versionDelete` but with `leaderLogic == false`: + +* An update without a `\_version_` is rejected (`missing _version_ on update from leader`) — replicas never mint versions. +* If the local UpdateLog is not `ACTIVE` (the core is recovering), the update is written to the buffer tlog and dropped (no index write). +* Otherwise the *drop rule* runs — the single check that makes asynchronous, possibly-reordered delivery safe: ++ +[source,java] +---- +Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId()); +if (lastVersion != null && Math.abs(lastVersion) >= versionOnUpdate) { + // This update is a repeat, or was reordered. We need to drop this update. + return true; +} +---- ++ +Application on a replica is therefore idempotent (repeats are dropped) and order-insensitive *per document* (an older version arriving late is dropped). +Nothing orders updates across different documents. +* On a TLOG replica (not currently leader), the command additionally gets `UpdateCommand.IGNORE_INDEXWRITER`: it is recorded in the tlog but not indexed — the index arrives later by segment replication, and the tlog exists so the replica can replay it if elected leader. + +Deletes store *negative* versions (hence the `Math.abs`), letting a version lookup distinguish "deleted at version v" from "exists at version v" while still ordering both. + +Two reorder edge cases get dedicated machinery: + +* *Delete-by-query*: on the leader, `versionDeleteByQuery` runs under `UpdateLocks.blockUpdates()` — DBQ quiesces *all* updates on the core, because it can affect any document. + Replicas keep a list of recent DBQs and re-execute them over an add that arrives out of order relative to the DBQ. + A DBQ is also fanned out from the originating node to *all* shard leaders, and is not atomic across shards. +* *In-place updates* carry `distrib.inplace.prevversion`; a replica that has not yet seen that previous version waits for it (`waitForDependentUpdates`, using the per-doc lock's `Condition`), and if it never arrives fetches the full document from the leader (`fetchFullUpdateFromLeader`). + +== Versioning and optimistic concurrency + +=== The _version_ clock + +`update.VersionInfo.getNewClock()` implements a time-based Lamport clock, synchronized per core: + +[source,java] +---- +long time = System.currentTimeMillis(); +long result = time << 20; +if (result <= vclock) { + result = vclock + 1; +} +vclock = result; +---- + +Properties that matter: + +* Strictly increasing per core, so per-document last-writer-wins is well defined under a single leader. +* Wall-clock based so that a restarted or newly elected leader (with an empty in-memory clock) does not go back in time relative to versions already in the index — correctness across leader changes leans on cluster clocks being roughly synchronized. + The low 20 bits are a same-millisecond counter (~1M versions/ms before the clock runs ahead of real time). +* Not contiguous — a commented-out pure-counter alternative in `VersionInfo` notes contiguous versions would make missing-update detection easier; Solr instead detects gaps via PeerSync's version-list exchange. + +The `\_version_` field must exist in the schema, single-valued, indexed-or-docValues and stored-or-docValues (`VersionInfo.getAndCheckVersionField`). +It must be assigned by Solr internally: user-supplied values would break the replica drop rule. +(Use `DocBasedVersionConstraintsProcessorFactory` for application-level version fields.) + +=== Where the version constraint comes from + +`versionOnUpdate` is taken, in priority order, from the command itself, the document's `\_version_` field, or the `\_version_` request parameter. +On internal `FROMLEADER` hops it is the leader-assigned version; on client requests it is the client's optimistic-concurrency constraint (0 when absent). +One subtlety: a leader receiving a document forwarded from *another collection* (`distrib.from.collection`, the MIGRATE path) discards the incoming version and stamps its own. + +=== Leader-side optimistic concurrency + +When a client supplies a nonzero version, the leader checks it against `vinfo.lookupVersion(id)` (tlog first, then index — uncommitted state counts) before assigning the new version: + +* `> 1` — must equal the current version exactly. +* `1` — the document must exist (any positive current version). +* `< 0` — the document must not exist. +* `0` — no check. + +A failed check raises `ErrorCode.CONFLICT` (HTTP 409) — unless `failOnVersionConflicts=false`, which silently drops the update instead (useful for batch loads where any conflicting doc should just be skipped). +The check runs only on the leader, under the per-document lock, so it is atomic with the version assignment: two clients doing conditional updates on the same document cannot both win. + +=== UpdateLocks + +`update.UpdateLocks` (SOLR-14679) replaced the historical fixed-size `VersionBucket` striping. +It keeps a hash-keyed map of pooled, fair `ReentrantLock`+`Condition` pairs with refcounting, so a lock exists only while some thread is operating on that document id, plus the global `blockUpdatesLock` read/write lock described earlier. +Lock acquisition times out after `docLockTimeoutMs`, surfacing pathological contention as an error rather than a hang. + +Everything that must be atomic per document happens inside `runWithLock`: the OCC check, atomic-update read-modify-write, version assignment, tlog append and index write. +This is also what makes realtime get reliable — a concurrent RTG cannot observe a state between the tlog map update and the version assignment. + +=== Implications for idempotency and retries + +What the above buys, and does not buy, for retrying updates: + +* *Internal* redelivery (leader → replica) is fully idempotent: the version travels with the document, and the drop rule discards repeats. + This is why the streaming fan-out can retry without coordination. +* A *client* retry of a full-document add or delete-by-id is a new update: the leader assigns a *fresh* version and the document is applied again. + The end state is the same document, so the retry is effectively idempotent — but it is a second write, and it will overwrite any concurrent write from another client that landed between the attempts (ordinary last-writer-wins). +* A client retry of a non-idempotent atomic update (`inc`, `add`, `remove`) is *not* safe: the first attempt may have been applied even though the response was lost (e.g., the connection died after the leader wrote locally). + The remedy is optimistic concurrency: read the document (RTG), compute the change, and send it conditioned on the read version; a retry after an ambiguous failure then either applies once or fails with 409, never applies twice. +* The failure modes a client can actually observe are: an error forwarding to the leader (nothing was applied for that document — safe to retry), a `LeaderChanged` error (ambiguous — the old leader applied it locally; retry is safe only under the full-document rule above), a version conflict (409), and a lost connection (fully ambiguous). + A replica failing to apply a document is *never* client-visible except through a reduced `rf`. + +== References + +* Ref guide: https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc[SolrCloud Update Consistency Model] (the user-facing contract), plus the pages on shards and indexing, recoveries and write tolerance, commits and transaction logs, partial document updates, and realtime get. +* `dev-docs/shard-split/shard-split.adoc` — tlog buffering during shard split. +* JIRA: https://issues.apache.org/jira/browse/SOLR-11702[SOLR-11702] (shard terms replace LIR), https://issues.apache.org/jira/browse/SOLR-14679[SOLR-14679] (`UpdateLocks` replaces version buckets), https://issues.apache.org/jira/browse/SOLR-6511[SOLR-6511] (`LeaderChanged` propagation), https://issues.apache.org/jira/browse/SOLR-7141[SOLR-7141] (recovery vs. in-flight updates). + +Not covered here (candidates for future documents): replica-type internals (NRT/TLOG/PULL), recovery and leader election (`RecoveryStrategy`, `PeerSync`, `ZkShardTerms` invariants, `leaderVoteWait`), and commit/visibility internals. diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc index 04dfeb766436..83bb2e286ed6 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc @@ -33,6 +33,7 @@ ** SolrCloud Clusters *** xref:solrcloud-shards-indexing.adoc[] *** xref:solrcloud-recoveries-and-write-tolerance.adoc[] +*** xref:solrcloud-update-consistency.adoc[] *** xref:solrcloud-distributed-requests.adoc[] *** xref:node-roles.adoc[] *** xref:aliases.adoc[] diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc index 5980a4ebfafc..fc387be258e0 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-recoveries-and-write-tolerance.adoc @@ -17,6 +17,7 @@ // under the License. SolrCloud is highly available and fault tolerant in reads and writes. +For a precise statement of what a successful update does and does not guarantee, see xref:solrcloud-update-consistency.adoc[]. == Write Side Fault Tolerance diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc index 622875b99f83..97a960f98952 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-shards-indexing.adoc @@ -27,6 +27,7 @@ A different collection might simply use a "hash" on the uniqueKey of each docume There is support for distributing both the index process and the queries automatically, and ZooKeeper provides failover and load balancing. As well as supporting replication, automatic index splitting into shards, there is support for automatic routing of documents to specific shards by a sharding strategy. Additionally, every shard can have multiple replicas for additional robustness. +The guarantees that distributed indexing provides are stated in xref:solrcloud-update-consistency.adoc[]. == Leaders and Replicas diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc new file mode 100644 index 000000000000..3b804f6d0d71 --- /dev/null +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/solrcloud-update-consistency.adoc @@ -0,0 +1,94 @@ += SolrCloud Update Consistency Model +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +This page states the consistency model of SolrCloud update requests (`/update`): what a successful response guarantees, and what it deliberately does not. +Several of these guarantees are intentionally weaker than those of a transactional database; the trade-offs favor indexing throughput and availability. + +Developers interested in how these guarantees are implemented should read the companion document https://github.com/apache/solr/blob/main/dev-docs/distributed-update-internals.adoc[Distributed Update Internals] in the source repository. + +== What a Successful Update Means + +An update request may be sent to any node in the cluster. +Each document is routed to the leader replica of its shard, which assigns the document a version, applies the update locally, and forwards it to the other in-sync NRT and TLOG replicas of the shard. +PULL replicas never receive individual updates; they periodically copy index segments from the leader. + +A successful (HTTP 200) response guarantees that every document in the request was applied and written to the xref:configuration-guide:commits-transaction-logs.adoc#transaction-log[transaction log] on its shard leader, and that Solr attempted to replicate it to the shard's other in-sync replicas. + +It does *not* guarantee that any replica other than the leader applied the update. +By design, a replica that fails to acknowledge an update does not fail the request; instead the leader marks that replica as out-of-date, and the replica must recover (catch up from the leader) before serving queries or becoming a leader. +Clients that need to know how widely an update was replicated can inspect the achieved replication factor (`rf`) in the response header; it is informational only and is never enforced. +See xref:solrcloud-recoveries-and-write-tolerance.adoc[] for details. + +== Durability + +Every update is written to the leader's transaction log before the response is returned, and replayed on startup if the node was stopped before a hard commit. +With the default transaction log `syncLevel` of `flush`, this survives a JVM crash or process kill but not necessarily an operating system crash or power loss; configure `syncLevel` to `fsync` to close that gap at some cost to throughput. +Index files themselves are durable once a hard commit completes. +See xref:configuration-guide:commits-transaction-logs.adoc[] for configuration details. + +An acknowledged update that has reached only the leader can still be lost if the leader is permanently destroyed before any replica receives it. +The durability floor of an acknowledged update is therefore one node, unless the client verifies `rf`. + +== Ordering + +Updates to the *same document* are strictly ordered. +The shard leader serializes concurrent updates to a given document id and assigns each a monotonically increasing `\_version_`; replicas ignore any update older than the version they already have. +The last write accepted by the leader wins. + +There is no ordering guarantee *across different documents*. +Documents sent in one batch may be applied on replicas, and become searchable, in a different order than submitted — especially across shards. + +== Atomicity + +The unit of atomicity is a single document. +A document update fully replaces the previous version of that document (xref:indexing-guide:partial-document-updates.adoc#atomic-updates[atomic updates] are read-modify-write operations performed on the leader, producing a full replacement document). +Queries never observe a partially updated document. + +There are no multi-document transactions. +A batch of documents is processed as independent operations: some may succeed while others fail, and there is no rollback of the documents that succeeded. +Similarly, a delete-by-query spanning multiple shards is not atomic across those shards. + +== Visibility + +An update is not searchable until a commit opens a new searcher; durability (via the transaction log and hard commits) and searchability are independent. +Each replica opens its searcher independently, so a document may briefly be searchable on one replica but not another; applications using `autoSoftCommit` or `commitWithin` must embrace this eventual consistency, as described in xref:solrcloud-shards-indexing.adoc#ignoring-commits-from-client-applications-in-solrcloud[Ignoring Commits from Client Applications in SolrCloud]. + +The exception is xref:configuration-guide:realtime-get.adoc[RealTime Get], which retrieves the latest version of a document by id — including uncommitted updates — directly from the transaction log. + +== Optimistic Concurrency + +Solr supports conditional updates through the `\_version_` field: a client may require that a document already exist, not exist, or exist at an exact version, and a failed condition rejects the update with an HTTP 409 conflict. +This is the supported way to prevent lost updates when multiple clients read, modify, and rewrite the same documents. +See xref:indexing-guide:partial-document-updates.adoc#optimistic-concurrency[Optimistic Concurrency]. + +== Failures and Retries + +Failures that are visible to the client include: the receiving node being unable to reach a shard leader, a shard having no elected leader, a leader change detected mid-request, version conflicts, and per-document errors such as schema violations. +Failed requests may leave earlier documents of the same batch applied. + +Retrying a failed request is safe for full-document adds and deletes-by-id: reapplying the same document produces the same end state (though a retry can overwrite a newer concurrent write from another client, as with any last-write-wins system). +Retrying is *not* inherently safe for atomic updates that are not idempotent, such as `inc`ing a counter or `add`ing a list value, because the original attempt may have been applied even though the response was lost. +Use optimistic concurrency to make such read-modify-write cycles safe to retry. + +== Leader Failover + +When a leader fails, Solr elects a new leader from the replicas that are known to be up-to-date, and the new leader first syncs any recent updates from its peers. +Acknowledged updates therefore survive leader failover whenever at least one up-to-date replica remains. + +If no up-to-date replica is available after a waiting period (`leaderVoteWait`), Solr chooses availability over consistency: a potentially stale replica becomes leader so the shard can continue accepting updates, and updates that only the old leader had may be lost. +Solr logs a "potential data loss" warning when this happens.