Skip to content

feat(ampup): bounded-concurrency parallel downloads #1762

@LNSD

Description

@LNSD

Summary

Implement bounded-concurrency parallel downloads for installing multiple components simultaneously, with a configurable default of 4 concurrent downloads.

Current behavior

Downloads are fully sequential — each binary is awaited one after the other. There is no concurrency infrastructure for downloads.

Requirements

Core

  • Introduce a download manager that bounds concurrent download tasks using a semaphore (default: 4)
  • Share a connection-pooled HTTP client across all download tasks
  • Use a synchronization barrier so activation (symlink creation) only proceeds after all downloads and verifications complete successfully

Atomicity and error handling

  • No partial installs: if any download in a batch fails, cancel in-flight downloads and clean up all completed artifacts for that batch
  • Each download writes to a staging directory and is moved into the toolchain directory only after verification
  • One retry per failed download before failing the entire batch

Progress reporting

  • TTY: aggregate progress indicator (e.g. [3/7] Installing components...) with per-component status lines that update in place
  • Non-TTY (CI): one line per component completion to avoid garbled output

GitHub rate-limit awareness

  • Respect X-RateLimit-Remaining and Retry-After headers
  • On 429, pause all in-flight requests and back off globally before retrying
  • Token resolution order:
    1. GITHUB_TOKEN environment variable
    2. gh auth token (GitHub CLI)
    3. Unauthenticated (with actionable warning when rate-limited without a token)

Configuration

  • config.toml option: [downloads] max_concurrent = 4
  • CLI flag: --jobs <N> / -j <N> (e.g. ampup install stable -j 8)
  • -j 1 disables concurrency and falls back to sequential mode (useful for debugging)

Verification

  • Per-artifact attestation/checksum verification runs immediately after each download completes, before releasing the concurrency slot
  • Failed verification triggers cleanup of that artifact and fails the batch

Suggested implementation

  • Use Tokio Semaphore to bound concurrency and JoinSet to collect task results
  • Wrap download logic in a DownloadManager struct that encapsulates concurrency, retries, verification, and progress reporting
  • Each spawned task acquires a semaphore permit, downloads + verifies, then releases the permit on completion
  • The batch fails if any task fails; all in-flight tasks are cancelled on first failure

Design rationale

  • Not sequential: installing a full profile with many components is unnecessarily slow; CI pipelines are latency-sensitive; sequential underutilizes bandwidth
  • Not unbounded: GitHub rate limits can trigger 429s; resource consumption scales linearly with concurrency; debugging interleaved failures is harder
  • Default of 4: matches common package manager defaults (cargo, apt); meaningful speedup without saturating connections or hitting rate limits

Context

This feature is most impactful once the component system and profiles are in place, as those introduce multi-artifact installs. However, the download manager can be introduced incrementally — even the current two-binary install benefits from parallelism.

Metadata

Metadata

Assignees

Labels

ampupRelated to the ampup tool

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions