Retry finalizing claimed executions on transient errors - #788
Open
wintan1418 wants to merge 1 commit into
Open
Conversation
When a job finishes, the worker deletes its claimed execution and marks the job finished (or failed) in a small transaction. If that transaction raises — say a transient DB connection drop — the error is swallowed by the pool's thread error handler and the claimed execution stays claimed forever: every recovery mechanism (releasing on deregistration, failing orphaned or pruned claims) assumes the claiming process is gone, but this worker is alive and well, so nothing ever picks the execution up again, even though the job already ran to completion. Finalization is idempotent — it locks the claimed execution row and no-ops if it's already gone — so it's safe to retry. Wait out the hiccup with a few increasingly spaced attempts before giving up and letting the error propagate as before. Each retry emits a retry_finalization.solid_queue event, logged at warn level, so operators can see the instability. Fixes rails#748
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #748
Problem
When a job completes, the worker finalizes it: a small transaction that deletes the
ClaimedExecutionand marks theJobfinished (or failed). If that transaction raises — e.g. a transient DB connection drop, exactly the scenario reproduced in #748 — the error is swallowed by the pool's thread error handler and the claimed execution staysclaimedforever:Fix
Finalization is idempotent: it takes a row lock on the claimed execution and no-ops if it's already gone (
unless_already_finalized). That makes it safe to retry.ClaimedExecution#performnow retries the finalization of both outcomes (finished and failed) through a few increasingly spaced attempts (0.5s, 1s, 2s, 4s) before giving up and letting the error propagate as before.This handles transient instability, which is the case reported in #748 (and what its reproduction script simulates). A persistent DB outage is a different problem with different machinery (#549, #683): once nothing can talk to the database, heartbeats fail too and the process-pruning recovery path takes over when connectivity returns.
Each retry emits a
retry_finalization.solid_queueevent (logged atwarn) with the job, process, attempt number, and error, so operators can see the instability instead of it being invisible.Tests
ActiveRecord::ConnectionNotEstablishedand finalize correctly on the next attempt.All three regression tests reproduce the stuck-
claimedstate onmainand pass with the fix.