Skip to content

Client: a write whose outcome is unknown is no longer sent twice - #146

Merged
bjmeetsfo merged 1 commit into
mainfrom
oss/write-replay-safety
Aug 22, 2026
Merged

Client: a write whose outcome is unknown is no longer sent twice#146
bjmeetsfo merged 1 commit into
mainfrom
oss/write-replay-safety

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

When a request to a datanode failed, the client refreshed the route and sent the
same request again. For a read that is free. For a write it depends entirely on
what the failure was, and the code did not look.

A refused connection is fine to repeat: the peer rejected the handshake, not one
byte was sent, nothing was applied. That is the case the existing tests cover and
it still works exactly as before.

A read timeout is a different thing. It does not mean the datanode never got the
request -- it means it stopped answering. The write may well have been applied.
Sending it again applies it twice, and nothing downstream can tell: an increment
counted twice, an append appended twice. That case was being re-sent too.

This was not behind an off-by-default knob. refresh_route_on_backend_error defaults
to true, so a timed-out write was re-sent in the default configuration, at all
three places that route a request -- execute and both batch_execute paths.

Measured by counting what the datanode actually received: a write that times out
arrived twice before this change and arrives once after.

So the rule is now about what the error proves, not about the verb alone:

  • read, any failure -> send again, unchanged
  • write, connection refused -> send again; it provably never arrived
  • write, anything else -> refresh the route, return the error

The route is still refreshed in every case, so the next request re-resolves and
recovers. What is dropped is only the second copy of a write whose fate is unknown.

The classifier is deliberately conservative in one place worth naming: a connect
timeout and a read timeout are indistinguishable at this layer -- both surface as
TimedOut -- so a connect timeout counts as "unknown" and the write is not repeated.
Wrong in the safe direction.

Worth saying: the command layer above already drew this exact line. It retries a
write only when the backend REFUSED it (safe_budget_free_write_retry requires a
topology retry, a definite "not applied"). The routing layer underneath had no such
reasoning and re-sent regardless, quietly undoing the guarantee the layer above was
maintaining. The write classifier used here is the engine's own is_write, the same
one that gates WAL persistence, rather than a second hand-maintained list -- a
hand-maintained one had already drifted once.

The new test fails with "left: 2, right: 1" if the guard is removed. Checked, not
assumed. My first cut of the guard was too broad -- it blocked the refused case too
-- and the two pre-existing route-refresh tests caught it.

Not addressed here, and worth its own look: the connection pool falls back to a
fresh socket when a pooled one fails, and that fallback cannot currently tell a
socket the server reaped from one that carried a request the server processed.

When a request to a datanode failed, the client refreshed the route and sent the
same request again. For a read that is free. For a write it depends entirely on
what the failure was, and the code did not look.

A refused connection is fine to repeat: the peer rejected the handshake, not one
byte was sent, nothing was applied. That is the case the existing tests cover and
it still works exactly as before.

A read timeout is a different thing. It does not mean the datanode never got the
request -- it means it stopped answering. The write may well have been applied.
Sending it again applies it twice, and nothing downstream can tell: an increment
counted twice, an append appended twice. That case was being re-sent too.

This was not behind an off-by-default knob. refresh_route_on_backend_error defaults
to true, so a timed-out write was re-sent in the default configuration, at all
three places that route a request -- execute and both batch_execute paths.

Measured by counting what the datanode actually received: a write that times out
arrived twice before this change and arrives once after.

So the rule is now about what the error proves, not about the verb alone:

- read, any failure                  -> send again, unchanged
- write, connection refused          -> send again; it provably never arrived
- write, anything else               -> refresh the route, return the error

The route is still refreshed in every case, so the next request re-resolves and
recovers. What is dropped is only the second copy of a write whose fate is unknown.

The classifier is deliberately conservative in one place worth naming: a connect
timeout and a read timeout are indistinguishable at this layer -- both surface as
TimedOut -- so a connect timeout counts as "unknown" and the write is not repeated.
Wrong in the safe direction.

Worth saying: the command layer above already drew this exact line. It retries a
write only when the backend REFUSED it (safe_budget_free_write_retry requires a
topology retry, a definite "not applied"). The routing layer underneath had no such
reasoning and re-sent regardless, quietly undoing the guarantee the layer above was
maintaining. The write classifier used here is the engine's own is_write, the same
one that gates WAL persistence, rather than a second hand-maintained list -- a
hand-maintained one had already drifted once.

The new test fails with "left: 2, right: 1" if the guard is removed. Checked, not
assumed. My first cut of the guard was too broad -- it blocked the refused case too
-- and the two pre-existing route-refresh tests caught it.

Not addressed here, and worth its own look: the connection pool falls back to a
fresh socket when a pooled one fails, and that fallback cannot currently tell a
socket the server reaped from one that carried a request the server processed.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 23:32
@bjmeetsfo
bjmeetsfo merged commit c775731 into main Aug 22, 2026
6 checks passed
@bjmeetsfo
bjmeetsfo deleted the oss/write-replay-safety branch August 22, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant