fetch: add --num option to fetch the last N commits#11047
Open
miloudbelarebia wants to merge 1 commit into
Open
fetch: add --num option to fetch the last N commits#11047miloudbelarebia wants to merge 1 commit into
miloudbelarebia wants to merge 1 commit into
Conversation
Mirrors `dvc gc --num`: `dvc fetch --num N` fetches the cache for the last N commits. On its own it walks back from the current branch's HEAD; combined with --all-branches it applies to each branch tip. Closes treeverse#11033
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11047 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41159 +1364
Branches 3141 3265 +124
==========================================
+ Hits 36087 37449 +1362
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11033
Problem
When merging long-lived branches, DVC can hit missing cache from earlier
commits (
No such file or directory: .../xx/xxx.dir). The current workaroundis
dvc fetch --all-commits, which on a branch likemaincan pull a hugeamount of data that isn't actually needed.
Solution
Add a
--num <N>option todvc fetch, mirroring the existingdvc gc --num:Ncommits walking backfrom the current branch's
HEAD.--all-branches, it applies to each branch tip (consistentwith how
numalready behaves indvc gc/brancher).--numdefaults to1(current behavior, workspace only). Values< 1raise
InvalidArgumentError. A value larger than the available history iscapped at the number of existing commits.
The plumbing reuses the
numparameter already threaded throughRepo.brancher/iter_revs, so the change is small: threadnumthroughfetch()and_collect_indexes(), and expose the CLI flag.Tests
tests/func/test_data_cloud.py::test_fetch_num— fetches the right numberof commit versions for
num=1/2/10.tests/func/test_data_cloud.py::test_fetch_num_invalid—num=0raises.tests/unit/command/test_data_sync.py::test_fetchto cover the flag.All pass locally;
ruff check/ruff format --checkare clean.