You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a peer (RS or DS) connected to a replication server is not "following" — i.e. its ServerWriter is catching up from the changelog DB instead of being served from the in-memory message queue — updates delivered to it keep the assured flag, mode and safe-data level of the original sender.
The NotAssuredUpdateMsg substitution that strips the assured flag for peers which are not expected to acknowledge (different group id, level already satisfied, non-eligible status) is only performed on the in-memory queue path:
ReplicationServerDomain.addUpdate() posts NotAssuredUpdateMsg to the queue for non-eligible peers,
but MessageHandler.getNextMessage() in the not-following branch re-reads the update from the changelog DB (fillLateQueue()), where the original wire bytes — including the assured flag — were persisted by publishUpdateMsg(), and ServerWriter publishes whatever it gets.
Every peer starts in catch-up mode right after the handshake (following = false by default), so on a loaded host even a just-connected peer with an empty backlog receives the first updates through the changelog path.
Impact
A peer that must never be asked for an ack (e.g. an RS from another group, or any RS when a safe-data level 1 update has already been satisfied) can receive the update with assuredFlag: true and reply with a spurious AckMsg. The receiving RS drops acks with no matching waitingAcks entry, so this is mostly harmless chatter — but the assured monitoring counters on the peer side get skewed, and the protocol contract ("assured feature does not cross group boundaries") is silently violated.
Conversely, an ack-eligible peer served through the catch-up path acks an update whose ExpectedAcksInfo timer may have long expired.
This made AssuredReplicationServerTest.testSafeDataLevelOne and testSafeReadOneRSComplex fail stochastically on every CI job (fake RSs received assuredFlag: true instead of the expected non-assured forward) while passing on fast local machines — see the analysis in Enable the remaining slow-group tests in the default build #702, worked around there by making the tests wait until all connected peers are following.
Possible fix
Normalize updates served from the late/catch-up path: wrap them in NotAssuredUpdateMsg (or clear the assured flag) before publishing, since by the time a peer catches up from the DB the ack window has passed and no ack will ever be counted. Needs a decision on the corner case where an eligible peer connects while an assured update is still within its ack timeout.
Found while investigating CI failures of the assured replication tests enabled in #702.
Problem
When a peer (RS or DS) connected to a replication server is not "following" — i.e. its
ServerWriteris catching up from the changelog DB instead of being served from the in-memory message queue — updates delivered to it keep the assured flag, mode and safe-data level of the original sender.The
NotAssuredUpdateMsgsubstitution that strips the assured flag for peers which are not expected to acknowledge (different group id, level already satisfied, non-eligible status) is only performed on the in-memory queue path:ReplicationServerDomain.addUpdate()postsNotAssuredUpdateMsgto the queue for non-eligible peers,MessageHandler.getNextMessage()in the not-following branch re-reads the update from the changelog DB (fillLateQueue()), where the original wire bytes — including the assured flag — were persisted bypublishUpdateMsg(), andServerWriterpublishes whatever it gets.Every peer starts in catch-up mode right after the handshake (
following = falseby default), so on a loaded host even a just-connected peer with an empty backlog receives the first updates through the changelog path.Impact
assuredFlag: trueand reply with a spuriousAckMsg. The receiving RS drops acks with no matchingwaitingAcksentry, so this is mostly harmless chatter — but the assured monitoring counters on the peer side get skewed, and the protocol contract ("assured feature does not cross group boundaries") is silently violated.ExpectedAcksInfotimer may have long expired.AssuredReplicationServerTest.testSafeDataLevelOneandtestSafeReadOneRSComplexfail stochastically on every CI job (fake RSs receivedassuredFlag: trueinstead of the expected non-assured forward) while passing on fast local machines — see the analysis in Enable the remaining slow-group tests in the default build #702, worked around there by making the tests wait until all connected peers are following.Possible fix
Normalize updates served from the late/catch-up path: wrap them in
NotAssuredUpdateMsg(or clear the assured flag) before publishing, since by the time a peer catches up from the DB the ack window has passed and no ack will ever be counted. Needs a decision on the corner case where an eligible peer connects while an assured update is still within its ack timeout.Found while investigating CI failures of the assured replication tests enabled in #702.