fix: client reconnect: messages held before the current buffer aren't replayed - #332
Open
Naros wants to merge 1 commit into
Open
fix: client reconnect: messages held before the current buffer aren't replayed#332Naros wants to merge 1 commit into
Naros wants to merge 1 commit into
Conversation
…ver replayed resetMessageQueue rewound to the start of the buffer the writer was on, so a client that reconnected and asked to continue from an earlier position never received the unconfirmed messages held in the buffers before it. They were skipped silently, since the client is only sent what follows the position it asks for. Rewind to the oldest buffer still held instead. A buffer is only released once the client has confirmed it, so that is the earliest position a client can legitimately ask for, and isNewData() filters forward from there to the requested c_scn/c_idx. A message walked over a second time also had to have its CONFIRMED flag cleared before being queued again, or confirmMessage() dropped it before the reconnecting client had acknowledged it.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a client reconnects and sends
CONTINUEwith a position earlier than the last message it received, any unconfirmed messages sitting in buffers older than the one the writer currently occupies are never re-sent. The client is only given messages that follow the requested position, so those messages are silently skipped with no errors or warnings. For a consumer that resumes from its last committed position rather than its last received position, this results in data loss on restart.Cause
WriterStream::processContinuecallsresetMessageQueue()to rewind the stream. That rewind used the head of whichever buffer the writer happened to be in:builderQueuepoints at the writer's current position, not at the oldest data still retained. Every buffer before it is still held; buffers are only released once the client confirms them, but the read restarts past them, so their contents are never walked again.A second related defect is that
confirmMessagesetsOUTPUT_BUFFER::CONFIRMEDon a message, and the main loop pops confirmed messages off the queue head. A message walked over a second time during a replay still carriesCONFIRMEDfrom the earlier pass, so it is discarded before the reconnecting client has acknowledged it.Proposed solution
Rewind to the oldest buffer still held.
A buffer that has not yet had a message end in it has no message boundary to replay from, so the read resumes at its head.
BUFFER_START_UNDEFINEDmoves from protected to public so that the writer can test for it.This is seen as safe because a buffer is only released once it's confirmed, so
firstBuilderQueueis the earliest position any client can legitimately ask to continue from.Metadata::isNewData()then filters forward to the requestedc_scn/c_idxso the extra messages walked are discarded rather than resent; the change therefore only costs a longer walk, never duplicates.For Debezium, it would be useful if this could be backported to 1.9 and earlier, based on supported versions. This is because Debezium isn't currently compatible with 2.0, which has changed how some parts of the data work.