CAS: replica HTTP dies on green-path soak after relink NETWORK_ERROR storm

[alsugiliazova]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report - something's broken

Describe the situation

On a two-replica ReplicatedMergeTree over a shared CAS pool, the green-path CAS soak (phase 1: inserts/merges/mutations, no chaos) loses a replica after two successful checkpoints. HTTP to that replica starts resetting or refusing connections. The dying INSERT is deterministic for --seed 1: op_id=102, target=0.

This is an availability failure, not a pool-corruption failure. cas-fsck at the last successful checkpoint and in the failure dump is clean (dangling=0, unreachable=0, stale_edge=0). There is no <Fatal>, no core dump, and (on the regression topology, which does not set a 28g mem_limit) no OOM: Docker still reports the container healthy while the published HTTP port is already dead.

Every clickhouse-server.err.log error on both replicas is the CAS relink confirm refusal:

Code: 210. NETWORK_ERROR
Source … did not prove it still holds the manifest it offered for part … by relink;
the relink is abandoned and the fetch will be retried later.

#2219 documents that this throw is a designed retry-later outcome and should not be logged at Error. That issue also states that replication still converges and lag stays flat. Soak phase 1 disagrees on availability: under the same refusal storm a replica stops answering HTTP, so the cluster is not merely noisy — it is no longer queryable on that node.

This issue:

• Kills replica HTTP during green-path CAS replication (no injected faults)
• Reproduces on both the upstream utils/ca-soak compose (PR #2073) and the cas/soak port in clickhouse-regression
• Leaves the CAS pool consistent (fsck dangling=0)
• Is not explained by a bad soak port: Python diffs vs PR [WIP] CAS draft (adopting to CI/CD, not for review / merge) #2073 are only helpers.cluster env wiring (CA_SOAK_*, chaos restart of clickhouse-server); phase 1 does not use chaos

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20000.altinityantalya
• Build type: release, ARM (build_arm_release)
• Package: PR #2159 187e06612862e0000e7ac0431ced66ded8283bea
  • https://altinity-build-artifacts.s3.amazonaws.com/PRs/2159/187e06612862e0000e7ac0431ced66ded8283bea/build_arm_release/clickhouse-common-static_26.6.2.20000.altinityantalya_arm64.deb
• Also seen on the same PR at f76335f4d1a3933d80fe9812e47eca3f61eaa686
• Topology: 2 replicas, one RustFS CAS pool, ClickHouse Keeper (upstream compose) or ZooKeeper (cas/soak soak_env)

A one-row relink smoke on the same cluster does work (INSERT on replica 1, SELECT on replica 2 returns the row). The failure needs the soak write rate (many small parts, overlapping fetches).

Steps (upstream harness, PR #2073)

1. Mount the 26.6 CAS-capable binary at build/programs/clickhouse (the compose bind-mounts it over /usr/bin/clickhouse in clickhouse/clickhouse-server:25.8).

2. From utils/ca-soak:

docker compose up -d
# wait until http://localhost:8123/ping and :8124/ping answer
PYTHONPATH="$(pwd)" python3 -m soak.run --seed 1 --phase 1 --ops 200 --checkpoint-every 50

3. Same shape via the regression port:

cd cas/soak
./regression.py --local --with-analyzer \
  --clickhouse-binary-path '<the same .deb URL>' \
  --suite phase1 --seed 1

---

Expected behavior

Phase 1 should finish 200 ops with checkpoints at 50 / 100 / 150 / 200. Both replicas should keep answering HTTP. Relink confirm refusals may happen and should retry (see #2219); they must not take a replica offline.

---

Actual behavior

On release builds

Checkpoints 50 and 100 succeed (dangling=0). Immediately afterwards the driver dies:

[soak.run] checkpoint OK: … dangling=0 stale_edge=0 dryrun_count=0
[soak.run] checkpoint GC PHASES [Node(localhost:8123)]: GC PHASES rounds=48 …
[soak.run] checkpoint GC PHASES [Node(localhost:8124)]: unavailable (node unreadable, …)
TRANSPORT FAILURE: [Errno 54] Connection reset by peer
  # regression soak_env: clickhouse1 HTTP resets; clickhouse2 still answers `1`
  # upstream compose: Connection refused after the peer exits / stops listening

Failure dump (failure.json / soak stdout):

"op_id": 102,
"last_op": {"op_id": 102, "type": "insert", "target": 0, "param": 52792},
"error": "TRANSPORT FAILURE: …",
"fsck": { "dangling": 0, "unreachable": 0, "stale_edge": 0 }

clickhouse-server.err.log on both replicas is dominated by:

<Error> ca_soak.ca_stress (…): … processQueueEntry …:
Code: 210. DB::Exception: Source … did not prove it still holds the manifest
it offered for part 20260818_3_3_0 by relink; the relink is abandoned and the
fetch will be retried later. (NETWORK_ERROR)

4. DB::DataPartsExchange::Fetcher::relinkPartToDisk(...)
5. DB::DataPartsExchange::Fetcher::fetchSelectedPart(...)
7. DB::StorageReplicatedMergeTree::fetchPart(...)
8. DB::StorageReplicatedMergeTree::executeFetch(...)

Observed counts on the PR #2073 compose run with the 187e0661… binary: 278 such errors on ch1, 214 on ch2, plus 5× Code: 1000 on each. Zero <Fatal> lines.

On the cas/soak topology (no 28g mem_limit): both containers stayed Up (healthy) while host curl :8123/ping was already Connection reset and :8124 still returned 1. Docker healthcheck ≠ HTTP alive.

---

Root cause analysis

Not fully pinned. The throw site is intentional taxonomy row 3 in DataPartsExchange.cpp:

throw Exception(ErrorCodes::NETWORK_ERROR,
    "Source {} did not prove it still holds the manifest it offered for part {} by relink; "
    "the relink is abandoned and the fetch will be retried later",
    fetch_uri.getHost(), part_name);

CasRefLedger::confirmExactRef returns Unknown when the source ref lane is not quiescent (lane_state != Ready, pending work, or leader_active). The receiver then abandons that relink and the replicated-queue retries later. #2219 is about logging that as Error.

What soak adds: under a burst of small parts the retry loop and confirm HTTP (cas_confirm) run concurrently with live inserts. Something on the source replica then stops serving HTTP (reset / refuse) without a Fatal. Candidates to investigate, not confirmed:

• Confirm / fetch HTTP on the source replica saturating or wedging the server HTTP path (healthcheck inside the container can still pass while the published port resets)
• Retry amplification: NETWORK_ERROR is in the retry-later class, so a busy lane produces a storm of relinkPartToDisk attempts against a source that is itself writing
• Memory: upstream compose sets mem_limit: 28g per node. On a 16 GiB Docker Desktop VM a lowered 4g cap cgroup-OOM-killed ch2 after the relink storm had already started; the regression soak_env run (no such cap) still lost HTTP without OOM. Treat OOM as a possible aggravator, not the primary explanation

---

Additional context

Related

• #2219 — same exception, scoped to log severity; claims no replication/availability impact
• PR #2159 — CAS on antalya-26.6 (binary under test)
• PR #2073 — utils/ca-soak harness (cas-gc-rebuild)

Notes

• Phase 1 sets transport_resilient=False, so the first HTTP reset ends the soak. That is harness policy; the replica going dark is the product bug
• Single-row relink smoke (CREATE + INSERT + SYSTEM SYNC REPLICA + SELECT) passes on the same binary and compose
• Local logs from the PR [WIP] CAS draft (adopting to CI/CD, not for review / merge) #2073 run: utils/ca-soak/logs/phase1_upstream_187e066.log and utils/ca-soak/logs/archive_4g/ch{1,2}.err.log on the machine that reproduced it