Skip to content

Remove dead code in dispatcher Unsubscribe so subscriptions are freed#211

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-subscription-unsubscribe-leak
Open

Remove dead code in dispatcher Unsubscribe so subscriptions are freed#211
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-subscription-unsubscribe-leak

Conversation

@damilolaedwards

Copy link
Copy Markdown

Problem

The subscription dispatcher's Unsubscribe returned early on the wrong condition:

if subscription.dispatcher != nil {
    return
}

A live subscription always has dispatcher set (it is assigned in Subscribe and only cleared inside the removal block that follows this guard), so the guard always returned and the removal loop below it was dead code. No subscription was ever removed.

The block, slot, epoch, payload and finalized event subscriptions created by the check and generate tasks each run defer subscription.Unsubscribe(), so every one of those was a no-op. On a long running instance the effects compound:

  • The dispatcher's subscriptions slice grows without bound.
  • Fire runs on every block, slot and epoch and iterates the whole slice under its mutex, so per event work grows with the number of leaked entries.
  • Each leaked subscription keeps a buffered channel that nobody drains, so it pins up to capacity block objects in memory for the lifetime of the process.

Together this trends toward memory and cpu exhaustion.

Fix

Invert the guard so an already removed subscription (dispatcher == nil) is the early return, and a live subscription actually reaches the removal loop. The wrapper Subscription.Unsubscribe already guards double calls, so this remains safe to call more than once.

The dispatcher is duplicated in the execution and consensus client packages, so the same one character fix is applied to both.

Tests

Each package gets a test that drives the real Subscribe, Unsubscribe and Fire, and asserts that removal shrinks the slice, that Fire only reaches still subscribed channels, that a repeated unsubscribe is a safe no-op, and that unsubscribing everything leaves the dispatcher empty. The test fails against the old guard and passes with the fix.

go build ./..., go vet, and the client package tests pass.

The subscription dispatcher's Unsubscribe returned early on the wrong
condition (dispatcher != nil), which is always true for a live
subscription, so the removal loop below it never ran. As a result every
block, slot, epoch and finalized event subscription created by the check
and generate tasks was never removed. Over a long running instance the
dispatcher slice grows without bound, Fire iterates every dead entry on
each event, and the leaked channels pin block objects in memory, trending
to memory and cpu exhaustion.

Invert the guard so an already removed subscription is the early return
and a live one is actually removed. The same defect and fix apply to both
the execution and consensus dispatchers. Add tests covering removal,
repeated unsubscribe, and delivery after removal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant