Client: a write whose outcome is unknown is no longer sent twice - #146
Merged
Conversation
When a request to a datanode failed, the client refreshed the route and sent the same request again. For a read that is free. For a write it depends entirely on what the failure was, and the code did not look. A refused connection is fine to repeat: the peer rejected the handshake, not one byte was sent, nothing was applied. That is the case the existing tests cover and it still works exactly as before. A read timeout is a different thing. It does not mean the datanode never got the request -- it means it stopped answering. The write may well have been applied. Sending it again applies it twice, and nothing downstream can tell: an increment counted twice, an append appended twice. That case was being re-sent too. This was not behind an off-by-default knob. refresh_route_on_backend_error defaults to true, so a timed-out write was re-sent in the default configuration, at all three places that route a request -- execute and both batch_execute paths. Measured by counting what the datanode actually received: a write that times out arrived twice before this change and arrives once after. So the rule is now about what the error proves, not about the verb alone: - read, any failure -> send again, unchanged - write, connection refused -> send again; it provably never arrived - write, anything else -> refresh the route, return the error The route is still refreshed in every case, so the next request re-resolves and recovers. What is dropped is only the second copy of a write whose fate is unknown. The classifier is deliberately conservative in one place worth naming: a connect timeout and a read timeout are indistinguishable at this layer -- both surface as TimedOut -- so a connect timeout counts as "unknown" and the write is not repeated. Wrong in the safe direction. Worth saying: the command layer above already drew this exact line. It retries a write only when the backend REFUSED it (safe_budget_free_write_retry requires a topology retry, a definite "not applied"). The routing layer underneath had no such reasoning and re-sent regardless, quietly undoing the guarantee the layer above was maintaining. The write classifier used here is the engine's own is_write, the same one that gates WAL persistence, rather than a second hand-maintained list -- a hand-maintained one had already drifted once. The new test fails with "left: 2, right: 1" if the guard is removed. Checked, not assumed. My first cut of the guard was too broad -- it blocked the refused case too -- and the two pre-existing route-refresh tests caught it. Not addressed here, and worth its own look: the connection pool falls back to a fresh socket when a pooled one fails, and that fallback cannot currently tell a socket the server reaped from one that carried a request the server processed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a request to a datanode failed, the client refreshed the route and sent the
same request again. For a read that is free. For a write it depends entirely on
what the failure was, and the code did not look.
A refused connection is fine to repeat: the peer rejected the handshake, not one
byte was sent, nothing was applied. That is the case the existing tests cover and
it still works exactly as before.
A read timeout is a different thing. It does not mean the datanode never got the
request -- it means it stopped answering. The write may well have been applied.
Sending it again applies it twice, and nothing downstream can tell: an increment
counted twice, an append appended twice. That case was being re-sent too.
This was not behind an off-by-default knob. refresh_route_on_backend_error defaults
to true, so a timed-out write was re-sent in the default configuration, at all
three places that route a request -- execute and both batch_execute paths.
Measured by counting what the datanode actually received: a write that times out
arrived twice before this change and arrives once after.
So the rule is now about what the error proves, not about the verb alone:
The route is still refreshed in every case, so the next request re-resolves and
recovers. What is dropped is only the second copy of a write whose fate is unknown.
The classifier is deliberately conservative in one place worth naming: a connect
timeout and a read timeout are indistinguishable at this layer -- both surface as
TimedOut -- so a connect timeout counts as "unknown" and the write is not repeated.
Wrong in the safe direction.
Worth saying: the command layer above already drew this exact line. It retries a
write only when the backend REFUSED it (safe_budget_free_write_retry requires a
topology retry, a definite "not applied"). The routing layer underneath had no such
reasoning and re-sent regardless, quietly undoing the guarantee the layer above was
maintaining. The write classifier used here is the engine's own is_write, the same
one that gates WAL persistence, rather than a second hand-maintained list -- a
hand-maintained one had already drifted once.
The new test fails with "left: 2, right: 1" if the guard is removed. Checked, not
assumed. My first cut of the guard was too broad -- it blocked the refused case too
-- and the two pre-existing route-refresh tests caught it.
Not addressed here, and worth its own look: the connection pool falls back to a
fresh socket when a pooled one fails, and that fallback cannot currently tell a
socket the server reaped from one that carried a request the server processed.