fix: retry complete pinned binary downloads - #499
Conversation
Downloads of pinned binaries (memtrack/exec-harness/mongo-tracer installers, valgrind .deb) intermittently fail in CI with "Failed to download file: error sending request for url (...)", and re-running the job almost always fixes it. The retry middleware on REQUEST_CLIENT only covers send() — up to the response headers — with a ~7s total backoff window, and classifies several transient network errors (e.g. BrokenPipe/UnexpectedEof io errors) as fatal. Body-read failures and torn transfers caught by the SHA-256 check were never retried at all. Wrap the whole download-and-verify in an outer retry loop (3 retries, 2s-30s exponential backoff) that retries any transient failure: request errors, retryable HTTP statuses, body-read errors, and hash mismatches. Client errors like 404 and local filesystem errors still fail immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0184GnvJzAEgQQekzGvZq1ec
Downloads now use the client without retry middleware, so retries are no longer nested: the outer download-and-verify loop is the only retry mechanism, covering send errors, retryable HTTP statuses, body-read failures, and hash mismatches uniformly. Retry count is bumped from 3 to 5 since there are no inner per-request retries anymore. With a single path, retry behavior is deterministic per failure, so the tests now assert exact request counts for aborted connections and transient 500s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0184GnvJzAEgQQekzGvZq1ec
Pinned binary downloads previously wrapped the whole download-and-verify in a manual retry loop. Replace it with a dedicated DOWNLOAD_CLIENT whose middleware uses a custom RetryableStrategy: downloads are idempotent GETs, so every request-level failure is retried, while responses keep the default status-based classification. This is what actually fixes the CI flakiness — the default strategy declines to retry a reqwest "error sending request" whose source is an UnexpectedEof, BrokenPipe or TimedOut io error, so a single GitHub blip failed a run after the benchmarks had already completed. A hash mismatch is now fatal instead of retried. The bytes arrived intact, since a torn transfer fails earlier on the body read, so a mismatch means the pin is wrong and retrying only delays the error. Body-read errors are no longer retried: the middleware only covers the send phase, up to the response headers. Co-Authored-By: Claude <noreply@anthropic.com>
Retry the full request and response-body read so transient truncation does not fail benchmark runs. Keep permanent HTTP, filesystem, and hash errors fatal. Co-Authored-By: Claude <noreply@anthropic.com>
Greptile SummaryThis PR moves pinned-binary retry handling around the complete download attempt so request, HTTP-status, and response-body failures share one exponential-backoff policy.
Confidence Score: 5/5The PR appears safe to merge, with complete-attempt retries preserving fatal handling for permanent HTTP, filesystem, and checksum failures. The download path now retries request and response-body interruptions through one bounded policy, avoids nested client retries, and retains destination integrity checks without an identified blocking or non-blocking defect.
|
| Filename | Overview |
|---|---|
| src/cli/run/helpers/download_file.rs | Introduces complete-download retry handling and comprehensive focused tests without an identified correctness regression. |
| src/request_client.rs | Adds a plain reqwest client for downloads so the new outer retry loop remains the sole retry layer. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Start pinned binary download] --> B[Send request]
B -->|Request error| R[Transient failure]
B -->|HTTP 408, 429, or 5xx| R
B -->|Permanent HTTP error| F[Fatal error]
B -->|Success| C[Read complete response body]
C -->|Body-read error| R
C -->|Success| D[Create and write destination]
D -->|Filesystem error| F
D -->|Success| E[Verify SHA-256]
E -->|Mismatch| G[Delete file and fail]
E -->|Match| S[Success]
R --> H{Retries remaining?}
H -->|Yes| I[Exponential backoff]
I --> B
H -->|No| F
Reviews (1): Last reviewed commit: "fix: retry complete pinned binary downlo..." | Re-trigger Greptile
Merging this PR will not alter performance
|
Retry pinned binary downloads after transient request, HTTP, and response-body failures.
The runner could fail after all benchmarks completed when a GitHub release asset request was interrupted. Retrying only the request phase did not cover transfers interrupted after response headers arrived.
Apply one exponential-backoff policy around the complete download attempt. Request errors, body-read failures,
5xx,408, and429responses are retried up to five times. Permanent client errors, local filesystem failures, and SHA-256 mismatches remain fatal.This addresses the intermittent ruff CI failure.