Remove dead code in dispatcher Unsubscribe so subscriptions are freed#211
Open
damilolaedwards wants to merge 1 commit into
Open
Remove dead code in dispatcher Unsubscribe so subscriptions are freed#211damilolaedwards wants to merge 1 commit into
damilolaedwards wants to merge 1 commit into
Conversation
The subscription dispatcher's Unsubscribe returned early on the wrong condition (dispatcher != nil), which is always true for a live subscription, so the removal loop below it never ran. As a result every block, slot, epoch and finalized event subscription created by the check and generate tasks was never removed. Over a long running instance the dispatcher slice grows without bound, Fire iterates every dead entry on each event, and the leaked channels pin block objects in memory, trending to memory and cpu exhaustion. Invert the guard so an already removed subscription is the early return and a live one is actually removed. The same defect and fix apply to both the execution and consensus dispatchers. Add tests covering removal, repeated unsubscribe, and delivery after removal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The subscription dispatcher's
Unsubscribereturned early on the wrong condition:A live subscription always has
dispatcherset (it is assigned inSubscribeand only cleared inside the removal block that follows this guard), so the guard always returned and the removal loop below it was dead code. No subscription was ever removed.The block, slot, epoch, payload and finalized event subscriptions created by the check and generate tasks each run
defer subscription.Unsubscribe(), so every one of those was a no-op. On a long running instance the effects compound:subscriptionsslice grows without bound.Fireruns on every block, slot and epoch and iterates the whole slice under its mutex, so per event work grows with the number of leaked entries.capacityblock objects in memory for the lifetime of the process.Together this trends toward memory and cpu exhaustion.
Fix
Invert the guard so an already removed subscription (
dispatcher == nil) is the early return, and a live subscription actually reaches the removal loop. The wrapperSubscription.Unsubscribealready guards double calls, so this remains safe to call more than once.The dispatcher is duplicated in the execution and consensus client packages, so the same one character fix is applied to both.
Tests
Each package gets a test that drives the real
Subscribe,UnsubscribeandFire, and asserts that removal shrinks the slice, thatFireonly reaches still subscribed channels, that a repeated unsubscribe is a safe no-op, and that unsubscribing everything leaves the dispatcher empty. The test fails against the old guard and passes with the fix.go build ./...,go vet, and the client package tests pass.