Make GCS deletes idempotent and raise delete_many timeout#139
Open
ericmj wants to merge 1 commit into
Open
Conversation
GCS returns 404 when an object was already deleted, for example by a concurrent upload of the same package deleting the same unversioned files. Treat 404 as success in Hexdocs.Store.GS.delete/2 instead of crashing the whole upload with a MatchError. Also raise the delete_many Task.async_stream timeout from 10s to 60s, matching upload_new_files: a single retried GCS request can spend 8s in backoff sleeps alone, so a slow delete could exceed the 10s task timeout and crash the upload.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three Sentry crashes in docs upload processing, all in
Hexdocs.Store.GS:delete/2pattern-matched on{:ok, 204, ...}, but GCS returns 404 when the object is already gone — which happens when two uploads of the same package run concurrently and both try to delete the package-leveldocs_config.js. Deletes now treat 404 as success.delete_many/2ran itsTask.async_streamwith a 10s per-task timeout, butHexdocs.HTTP.retrycan spend 8s in backoff sleeps alone, plus up to 30s per attempt, so a single slow or retried GCS delete crashed the whole upload. The timeout is now 60s, matchingupload_new_files/1inHexdocs.Bucket.Crashed uploads self-healed via SQS redelivery, so the impact was limited to delayed docs publishing.