Skip to content

fix: client reconnect: messages held before the current buffer aren't replayed - #332

Open
Naros wants to merge 1 commit into
bersler:masterfrom
Naros:client-reconnect-replay
Open

fix: client reconnect: messages held before the current buffer aren't replayed#332
Naros wants to merge 1 commit into
bersler:masterfrom
Naros:client-reconnect-replay

Conversation

@Naros

@Naros Naros commented Jul 28, 2026

Copy link
Copy Markdown

Problem

When a client reconnects and sends CONTINUE with a position earlier than the last message it received, any unconfirmed messages sitting in buffers older than the one the writer currently occupies are never re-sent. The client is only given messages that follow the requested position, so those messages are silently skipped with no errors or warnings. For a consumer that resumes from its last committed position rather than its last received position, this results in data loss on restart.

Cause

WriterStream::processContinue calls resetMessageQueue() to rewind the stream. That rewind used the head of whichever buffer the writer happened to be in:

oldSize = builderQueue->start;

builderQueue points at the writer's current position, not at the oldest data still retained. Every buffer before it is still held; buffers are only released once the client confirms them, but the read restarts past them, so their contents are never walked again.

A second related defect is that confirmMessage sets OUTPUT_BUFFER::CONFIRMED on a message, and the main loop pops confirmed messages off the queue head. A message walked over a second time during a replay still carries CONFIRMED from the earlier pass, so it is discarded before the reconnecting client has acknowledged it.

Proposed solution

Rewind to the oldest buffer still held.

builderQueue = builder->firstBuilderQueue;
oldSize = builderQueue->start;
if (unlikely(oldSize == Builder::BUFFER_START_UNDEFINED))
    oldSize = 0;

A buffer that has not yet had a message end in it has no message boundary to replay from, so the read resumes at its head. BUFFER_START_UNDEFINED moves from protected to public so that the writer can test for it.

This is seen as safe because a buffer is only released once it's confirmed, so firstBuilderQueue is the earliest position any client can legitimately ask to continue from. Metadata::isNewData() then filters forward to the requested c_scn / c_idx so the extra messages walked are discarded rather than resent; the change therefore only costs a longer walk, never duplicates.

For Debezium, it would be useful if this could be backported to 1.9 and earlier, based on supported versions. This is because Debezium isn't currently compatible with 2.0, which has changed how some parts of the data work.

…ver replayed

resetMessageQueue rewound to the start of the buffer the writer was on, so
a client that reconnected and asked to continue from an earlier position
never received the unconfirmed messages held in the buffers before it. They
were skipped silently, since the client is only sent what follows the
position it asks for.

Rewind to the oldest buffer still held instead. A buffer is only released
once the client has confirmed it, so that is the earliest position a client
can legitimately ask for, and isNewData() filters forward from there to the
requested c_scn/c_idx.

A message walked over a second time also had to have its CONFIRMED flag
cleared before being queued again, or confirmMessage() dropped it before the
reconnecting client had acknowledged it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant