Skip to content

fix: retry complete pinned binary downloads - #499

Merged
art049 merged 4 commits into
mainfrom
claude/codspeed-memtrack-download-error-slr131
Aug 5, 2026
Merged

fix: retry complete pinned binary downloads#499
art049 merged 4 commits into
mainfrom
claude/codspeed-memtrack-download-error-slr131

Conversation

@art049

@art049 art049 commented Aug 5, 2026

Copy link
Copy Markdown
Member

Retry pinned binary downloads after transient request, HTTP, and response-body failures.

The runner could fail after all benchmarks completed when a GitHub release asset request was interrupted. Retrying only the request phase did not cover transfers interrupted after response headers arrived.

Apply one exponential-backoff policy around the complete download attempt. Request errors, body-read failures, 5xx, 408, and 429 responses are retried up to five times. Permanent client errors, local filesystem failures, and SHA-256 mismatches remain fatal.

This addresses the intermittent ruff CI failure.

claude and others added 4 commits August 5, 2026 07:41
Downloads of pinned binaries (memtrack/exec-harness/mongo-tracer
installers, valgrind .deb) intermittently fail in CI with "Failed to
download file: error sending request for url (...)", and re-running the
job almost always fixes it.

The retry middleware on REQUEST_CLIENT only covers send() — up to the
response headers — with a ~7s total backoff window, and classifies
several transient network errors (e.g. BrokenPipe/UnexpectedEof io
errors) as fatal. Body-read failures and torn transfers caught by the
SHA-256 check were never retried at all.

Wrap the whole download-and-verify in an outer retry loop (3 retries,
2s-30s exponential backoff) that retries any transient failure: request
errors, retryable HTTP statuses, body-read errors, and hash mismatches.
Client errors like 404 and local filesystem errors still fail
immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184GnvJzAEgQQekzGvZq1ec
Downloads now use the client without retry middleware, so retries are
no longer nested: the outer download-and-verify loop is the only retry
mechanism, covering send errors, retryable HTTP statuses, body-read
failures, and hash mismatches uniformly. Retry count is bumped from 3
to 5 since there are no inner per-request retries anymore.

With a single path, retry behavior is deterministic per failure, so the
tests now assert exact request counts for aborted connections and
transient 500s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184GnvJzAEgQQekzGvZq1ec
Pinned binary downloads previously wrapped the whole download-and-verify
in a manual retry loop. Replace it with a dedicated DOWNLOAD_CLIENT whose
middleware uses a custom RetryableStrategy: downloads are idempotent
GETs, so every request-level failure is retried, while responses keep the
default status-based classification. This is what actually fixes the CI
flakiness — the default strategy declines to retry a reqwest "error
sending request" whose source is an UnexpectedEof, BrokenPipe or TimedOut
io error, so a single GitHub blip failed a run after the benchmarks had
already completed.

A hash mismatch is now fatal instead of retried. The bytes arrived
intact, since a torn transfer fails earlier on the body read, so a
mismatch means the pin is wrong and retrying only delays the error.

Body-read errors are no longer retried: the middleware only covers the
send phase, up to the response headers.

Co-Authored-By: Claude <noreply@anthropic.com>
Retry the full request and response-body read so transient truncation does not fail benchmark runs. Keep permanent HTTP, filesystem, and hash errors fatal.

Co-Authored-By: Claude <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 5, 2026

Copy link
Copy Markdown

Greptile Summary

This PR moves pinned-binary retry handling around the complete download attempt so request, HTTP-status, and response-body failures share one exponential-backoff policy.

  • Adds retry classification and complete-attempt retry handling for pinned downloads.
  • Uses a plain download client to avoid nested middleware retries.
  • Adds coverage for aborted connections, truncated bodies, retry exhaustion, retryable server errors, permanent client errors, and checksum mismatches.

Confidence Score: 5/5

The PR appears safe to merge, with complete-attempt retries preserving fatal handling for permanent HTTP, filesystem, and checksum failures.

The download path now retries request and response-body interruptions through one bounded policy, avoids nested client retries, and retains destination integrity checks without an identified blocking or non-blocking defect.

Important Files Changed

Filename Overview
src/cli/run/helpers/download_file.rs Introduces complete-download retry handling and comprehensive focused tests without an identified correctness regression.
src/request_client.rs Adds a plain reqwest client for downloads so the new outer retry loop remains the sole retry layer.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Start pinned binary download] --> B[Send request]
    B -->|Request error| R[Transient failure]
    B -->|HTTP 408, 429, or 5xx| R
    B -->|Permanent HTTP error| F[Fatal error]
    B -->|Success| C[Read complete response body]
    C -->|Body-read error| R
    C -->|Success| D[Create and write destination]
    D -->|Filesystem error| F
    D -->|Success| E[Verify SHA-256]
    E -->|Mismatch| G[Delete file and fail]
    E -->|Match| S[Success]
    R --> H{Retries remaining?}
    H -->|Yes| I[Exponential backoff]
    I --> B
    H -->|No| F
Loading

Reviews (1): Last reviewed commit: "fix: retry complete pinned binary downlo..." | Re-trigger Greptile

@codspeed-hq

codspeed-hq Bot commented Aug 5, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks


Comparing claude/codspeed-memtrack-download-error-slr131 (4d21603) with main (6eac757)

Open in CodSpeed

@art049
art049 merged commit 17d3098 into main Aug 5, 2026
23 checks passed
@art049
art049 deleted the claude/codspeed-memtrack-download-error-slr131 branch August 5, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants