fix(private-org-sync): process repos in parallel with configurable workers#5030
fix(private-org-sync): process repos in parallel with configurable workers#5030petr-muller wants to merge 5 commits intoopenshift:mainfrom
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
Important Review skippedAuto reviews are limited based on label configuration. 🚫 Excluded labels (none allowed) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: petr-muller The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…branch Previously, mirror() called ls-remote for both source and destination on every branch, resulting in 2*N network calls per repo where N is the number of branches. Since ls-remote --heads returns ALL branches at once, we can call it once per repo and pass the results to mirror(). This moves git init, remote setup, and ls-remote calls from mirror() into the main loop where repos are already grouped, reducing network calls from 2*N to 2 per repo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
petr-muller
force-pushed
the
git-sync-speedup-7
branch
from
f4b25d1 to
d7b1783
Compare
Extract the per-repo sync logic from main() into a syncRepo method on gitSyncer. This makes the repo initialization, ls-remote batching, and branch mirroring flow independently testable. Add TestSyncRepo with 6 test cases covering: branches in sync, one branch needing sync, dst/src ls-remote failures (with and without failOnNonexistentDst), and init failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…errors Wrap gitExec with withRetryOnTransientError to transparently retry any git command (fetch, push, ls-remote, etc.) up to 3 times with a 5-second delay when transient network errors are detected (DNS resolution failures, connection timeouts/refused/reset, server 5xx errors). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of querying source and destination branch heads sequentially, run both ls-remote calls concurrently using goroutines. This saves one network round-trip per repo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rkers Extract per-repo sync logic into a syncRepo function and process repos concurrently using a worker pool. The --parallelism flag (default 4) controls the number of concurrent workers. Each worker gets its own copy of the gitSyncer struct, avoiding races on the mutable logger field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
petr-muller
force-pushed
the
git-sync-speedup-7
branch
from
d7b1783 to
908d58a
Compare
|
@petr-muller: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Processes repos in parallel using a configurable worker pool (
--parallelismflag, default 4). Each repo is independent (separate git dir, separate remotes, separate branches), so this is safe to parallelize.Stacked on #5029.