compact: free space per pack with --threshold#9801
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9801 +/- ##
==========================================
- Coverage 84.78% 84.77% -0.01%
==========================================
Files 92 92
Lines 15251 15289 +38
Branches 2286 2297 +11
==========================================
+ Hits 12930 12961 +31
- Misses 1621 1630 +9
+ Partials 700 698 -2 ☔ View full report in Codecov by Harness. |
|
Also please rebase on current master (I have merged #9800 now). |
|
Have a look at that, maybe you can implement |
|
I did a test run with slightly modified code (N=2) and got this. So guess the 100% display issue is not fixed yet? Also: even without -v, it is very verbose: I suggest you try this manually to get a feel of it and find such issues yourself. |
Group objects by their pack and act per pack: drop fully-unused packs, and rewrite mixed packs whose unused bytes reach --threshold (default 40%) by copying the survivors forward with compact_pack. Two index scans keep the memory use bounded. refs borgbackup#8572 borgbackup#8514
Rewrite a pack only when its unused bytes reach --threshold percent; --dry-run reports the space compact would free without changing the repository (borgbackup#9379).
mr-raj12
force-pushed
the
pack-files-compact
branch
from
464c91a to
b1862f3
Compare
ThomasWaldmann
left a comment
ThomasWaldmann
left a comment
There was a problem hiding this comment.
no manifest talk please, we do not have a manifest (in the original meaning) any more.
|
About my feedback that it is verbose even without -v: my fault, guess I forgot that I had tweaked borg's default via a settings file of jsongargparse. So, ignore that. |
|
I have seen that Is there a difference now with and without it? IIRC, I introduced that back then because it was much slower with stats than without. |
mr-raj12
force-pushed
the
pack-files-compact
branch
from
b1862f3 to
0586cc7
Compare
yes, --stats does a full repo scan so it is slower but gives size numbers, without it borg uses the cached index so it is faster but shows no sizes |
…ant full scan and fix the epilog
2 participants
Description
Moves
borg compactfrom deleting single objects to compacting whole packs, so it keeps working once a pack holds more than one object (N>1).For each pack:
--thresholdpercent (default 10), copying the survivors into a new pack viaRepository.compact_packand dropping the old one. Below the threshold the pack is left alone, so a large pack is not rewritten just to reclaim a few bytes.--dry-runreports the space compact would free without changing the repository.--statsreports the sum of stored object sizes before and after; the sizes come from the cached chunk index, so no extra full scan over the repository objects is needed.The chunk index is scanned twice to keep memory bounded: first only per-pack byte counts to decide each pack's fate, then the object ids of just the packs that change. The #9748 crash-safety order is preserved: cached chunk indexes are invalidated before the first store change.
At N=1 every pack holds one object, so mixed packs never occur and the behavior matches before. The rewrite path is covered by a test that forces max_count > 1.
This recycles the approach from #9777, which can be closed.
refs #8572 #8514
Checklist
master