Document that retry_on can't intercept process-death errors - #789
Open
wintan1418 wants to merge 1 commit into
Open
Document that retry_on can't intercept process-death errors#789wintan1418 wants to merge 1 commit into
wintan1418 wants to merge 1 commit into
Conversation
Active Job's retry_on and rescue_from only hook into exceptions raised while perform runs. ProcessPrunedError, ProcessExitError and ProcessMissingError are never raised inside the job: the process running it is already gone, and a different process records the error directly as a failed execution after the fact. People coming from other backends expect retry_on to cover this case and are surprised when it silently doesn't, so spell out why it can't work and point to the mechanisms that do: Mission Control and the fail_many_claimed subscription. Fixes rails#786
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #786
retry_on/rescue_fromonly intercept exceptions raised while a job'sperformruns.SolidQueue::Processes::ProcessPrunedError,ProcessExitErrorandProcessMissingErrorare never raised inside the job — the process running it is already gone, and a different process (the one pruning registrations or the supervisor reaping a fork) records the error directly as a failed execution, after the fact. So there's no job execution left for Active Job's retry machinery to hook into, andretry_onsilently does nothing for these errors.As #786 points out (and #313 / #422 before it), this keeps surprising people coming from other backends, especially those running workers on preemptible/spot machines. This adds an explicit note to the "Jobs interrupted by non-graceful process death" README section explaining why
retry_oncan't work there, and pointing to the mechanisms that do: retrying manually via Mission Control, or automatically via thefail_many_claimed.solid_queuesubscription documented just below the note.