Skip to content

Add cache and parallel run to Git Count Contributors#701

Open
attiasas wants to merge 1 commit intojfrog:devfrom
attiasas:git_cc_parallel_cache
Open

Add cache and parallel run to Git Count Contributors#701
attiasas wants to merge 1 commit intojfrog:devfrom
attiasas:git_cc_parallel_cache

Conversation

@attiasas
Copy link
Collaborator

feat(git): Add parallel scanning and caching to Git Count Contributors

Summary

Introduces parallel repository scanning and a file-based cache layer to the git count-contributors command, significantly improving performance for large organizations with many repositories. Repositories are now scanned concurrently (configurable via --threads, default 10) and results are cached to disk (configurable via --cache-validity, default 3 days) to avoid redundant API calls on repeated runs.

Changes

  • Parallel scanning: Refactored scanAndCollectCommitsInfo to scan repositories concurrently using gofrog/parallel.Runner. Each repo is scanned independently, and results are merged after all tasks complete.
  • File-based cache: New cache.go module persists per-repo scan results as JSON files under ~/.jfrog/contributors-cache/<hash>/. Cache entries are keyed by a SHA256 hash of (scm-type, scm-api-url, owner, months) to avoid collisions. Writes are atomic (tmp + rename).
  • New CLI flags:
    • --cache-validity — number of days a cached result remains valid (0 = skip cache, default 3).
    • --threads — number of parallel threads for scanning (default 10).
  • Result types: Introduced repoScanResult and vcsServerScanResult structs to cleanly aggregate parallel results, along with mergeContributors, mergeDetailedContributors, and mergeDetailedRepos helper functions.
  • Utility: Added GetContributorsCacheDir() in utils/paths.go.
  • Tests: Added unit tests for cache round-trip, expiry, merge helpers, and parallel scanning via a new mockVcsClient test double.

Files changed

File Status
cli/docs/flags.go Modified — new CacheValidity, GitThreads flags
cli/gitcommands.go Modified — parse new flags into params
commands/git/contributors/cache.go Added — cache read/write/sanitize logic
commands/git/contributors/cache_test.go Added — cache unit tests
commands/git/contributors/countcontributors.go Modified — parallel scan, result structs, merge helpers
commands/git/contributors/countcontributors_test.go Modified — merge and parallel scan tests
commands/git/contributors/mock_vcs_client_test.go Added — full VcsClient mock for tests
utils/paths.go Modified — GetContributorsCacheDir helper

Testing

  • New unit tests cover cache write/read, cache expiry, zero-max-age bypass, missing cache files, all three merge helpers (empty src, non-overlapping, collision), and end-to-end parallel scanning with a mock VCS client.
  • Existing tests continue to pass.

Notes

  • Cache entries use the same months parameter in the key, so changing --months automatically invalidates stale cache.
  • Setting --cache-validity 0 fully bypasses caching (no file I/O at all).

  • The pull request is targeting the dev branch.
  • The code has been validated to compile successfully by running go vet ./....
  • The code has been formatted properly using go fmt ./....
  • All static analysis checks passed.
  • All tests have passed. If this feature is not already covered by the tests, new tests have been added.
  • Updated the Contributing page / ReadMe page / CI Workflow files if needed.
  • All changes are detailed at the description. if not already covered at JFrog Documentation, new documentation have been added.

@attiasas attiasas added new feature Automatically generated release notes safe to test Approve running integration tests on a pull request labels Mar 22, 2026
@github-actions github-actions bot removed the safe to test Approve running integration tests on a pull request label Mar 22, 2026
@github-actions
Copy link

👍 Frogbot scanned this pull request and did not find any new security issues.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature Automatically generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant