Skip to content

S3 retries / v0.5.1 - #50

Open
qqmyers wants to merge 3 commits into
mainfrom
S3-retries-v0.5.1
Open

S3 retries / v0.5.1#50
qqmyers wants to merge 3 commits into
mainfrom
S3-retries-v0.5.1

Conversation

@qqmyers

@qqmyers qqmyers commented Jul 23, 2026

Copy link
Copy Markdown
Member

This PR

  • refactors the existing retry logic on per-file calls to Dataverse (handling rate limiting),
  • expands that to include the per-dataset calls as well,
  • adds new retries on the S3 calls,
  • adds instructions on how to install a local copy via git clone, and
  • ups the version to 0.5.1.

FWIW: I haven't heard of big problems with failing S3 calls, e.g. on AWS, but, at QDR, we are now using BackBlaze B2 for larger datasets and their documentation notes that they send 50x responses to S3 calls as a routine response that should be resolved with a retry (TLDR: they reduce costs at the expense of developers having to do automated retries in some cases.).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant