-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
ampupRelated to the ampup toolRelated to the ampup tool
Description
Summary
Implement bounded-concurrency parallel downloads for installing multiple components simultaneously, with a configurable default of 4 concurrent downloads.
Current behavior
Downloads are fully sequential — each binary is awaited one after the other. There is no concurrency infrastructure for downloads.
Requirements
Core
- Introduce a download manager that bounds concurrent download tasks using a semaphore (default: 4)
- Share a connection-pooled HTTP client across all download tasks
- Use a synchronization barrier so activation (symlink creation) only proceeds after all downloads and verifications complete successfully
Atomicity and error handling
- No partial installs: if any download in a batch fails, cancel in-flight downloads and clean up all completed artifacts for that batch
- Each download writes to a staging directory and is moved into the toolchain directory only after verification
- One retry per failed download before failing the entire batch
Progress reporting
- TTY: aggregate progress indicator (e.g.
[3/7] Installing components...) with per-component status lines that update in place - Non-TTY (CI): one line per component completion to avoid garbled output
GitHub rate-limit awareness
- Respect
X-RateLimit-RemainingandRetry-Afterheaders - On 429, pause all in-flight requests and back off globally before retrying
- Token resolution order:
GITHUB_TOKENenvironment variablegh auth token(GitHub CLI)- Unauthenticated (with actionable warning when rate-limited without a token)
Configuration
config.tomloption:[downloads] max_concurrent = 4- CLI flag:
--jobs <N>/-j <N>(e.g.ampup install stable -j 8) -j 1disables concurrency and falls back to sequential mode (useful for debugging)
Verification
- Per-artifact attestation/checksum verification runs immediately after each download completes, before releasing the concurrency slot
- Failed verification triggers cleanup of that artifact and fails the batch
Suggested implementation
- Use Tokio
Semaphoreto bound concurrency andJoinSetto collect task results - Wrap download logic in a
DownloadManagerstruct that encapsulates concurrency, retries, verification, and progress reporting - Each spawned task acquires a semaphore permit, downloads + verifies, then releases the permit on completion
- The batch fails if any task fails; all in-flight tasks are cancelled on first failure
Design rationale
- Not sequential: installing a full profile with many components is unnecessarily slow; CI pipelines are latency-sensitive; sequential underutilizes bandwidth
- Not unbounded: GitHub rate limits can trigger 429s; resource consumption scales linearly with concurrency; debugging interleaved failures is harder
- Default of 4: matches common package manager defaults (cargo, apt); meaningful speedup without saturating connections or hitting rate limits
Context
This feature is most impactful once the component system and profiles are in place, as those introduce multi-artifact installs. However, the download manager can be introduced incrementally — even the current two-binary install benefits from parallelism.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ampupRelated to the ampup toolRelated to the ampup tool