borg2: transfer - is it incremental?

[henfri]

Hello,

I have a question on borg transfer. Is it possible to do this incrementally?

Say, I have many backups locally and want to do an offsite backup with parts of this repository. Would a cronjob daily with such command work?

borg --repo=DST_REPO transfer --other-repo=SRC_REPO -a "sh:daily-*"
borg --repo=DST_REPO transfer --other-repo=SRC_REPO -a "sh:meinserver-*"

Best regards,
Hendrik

[ThomasWaldmann]

Answer made by Claude Opus 4.8:

Yes — this is exactly what transfer is designed for. Your cronjob approach works and is explicitly recommended. Each run only copies what's missing, so daily invocations are effectively incremental.

How it's incremental (two levels)

1. Archive level — already-transferred archives are skipped. For every archive, before copying, do_transfer checks whether the destination already has it:

• manifest.archives.exists_name_and_ts(name, ts) — same name + timestamp
• manifest.archives.exists_name_and_id(name, id) — same name + id

If either matches, it prints archive is already present in destination repo, skipping. and moves on. So re-running the same command daily just skips everything old and copies only the new archives.

2. Chunk level — deduplication. Even for a new archive, only chunks the destination doesn't already have are fetched/written (cache.seen_chunk(...) → reuse_chunk vs add_chunk). Chunks shared with already-transferred archives are not re-sent.

Your -a filters are fine

The -a "sh:daily-*" / -a "sh:meinserver-*" selection is applied via the standard archive-filter group. Splitting one big repo into several targeted transfer invocations is directly encouraged in the command's own help:

You could use the misc. archive filter options to limit which archives it will transfer, e.g. using the -a option. This is recommended for big repositories with multiple data sets to keep the runtime per invocation lower.

Caveats worth knowing

• The destination must be a related repo. It has to share the id-hash and chunker secret, otherwise transfer refuses ("You must use the same chunker secret..."). Create it once with:

  borg --repo=DST_REPO repo-create --encryption=DST_ENC --other-repo=SRC_REPO
  

• Interruptions are safe. An archive is only recorded in the destination manifest after archive.save() at the very end. If a cron run is killed mid-archive, that archive simply isn't marked present and gets re-done next run — but the chunks already uploaded stay in the repo and are reused via dedup, so the retry is cheap. No half-archives leak into the manifest.
• Prune the destination separately. transfer never deletes; it only adds. If you want the offsite copy to expire old archives, run borg prune/compact against DST_REPO on its own schedule.
• Tip from the docs: pass --compression=<same as source> --recompress=never (the default is already never) to avoid recompressing data — fastest, since chunks are copied as-is.

A good pattern is to add a --dry-run line (or run one manually now and then) to confirm nothing is unexpectedly left behind:

borg --repo=DST_REPO transfer --other-repo=SRC_REPO -a "sh:daily-*" --dry-run

[ThomasWaldmann]

(I read the answer and think it is fine - at least there are no obvious errors in it)