Structured per-item event reporting for `borg create` (JSON / observability layer)

[YK-03]

Hi Borg maintainers 👋,

I’ve been exploring the borg create flow in depth (from CLI dispatch → do_create → _rec_walk / _process_any → archive.py processing), and I noticed that while Borg provides rich logging for humans, it currently lacks structured, machine-readable observability during backup operations.

---

💡 Motivation

During large backup runs:

• Errors and retries are logged as text, which can be hard to analyze programmatically
• Deduplication decisions (reuse vs new chunks) are not externally visible
• There is no structured way to trace per-file lifecycle (start → processed → skipped → error)

This makes:

• debugging large backups harder
• integration with external tools (dashboards, monitoring, UI) difficult
• automated analysis of backup behavior nearly impossible

---

🚀 Proposal

Introduce an optional structured event reporting system for borg create:

CLI Flag (opt-in)

borg create --log-json ...

---

🧩 Core Idea

Emit structured events during backup execution, for example (JSON Lines format):

{"event": "file_started", "path": "/home/user/file.txt"}
{"event": "chunk_reused", "chunk_id": "...", "size": 4096}
{"event": "file_completed", "path": "/home/user/file.txt", "status": "ok"}
{"event": "file_error", "path": "...", "error": "..."}

---

🏗️ Possible Design Direction

• Introduce a lightweight event emitter inside the create pipeline

• Hook into key points:

  • before/after file processing (_process_any, process_file)
  • deduplication decisions (cache.reuse_chunk, add_chunk)
  • retry / error paths

• Keep default behavior unchanged (text logs remain primary)

---

🎯 Benefits

• Improves observability and debugging

• Enables future tooling:

  • dashboards / visualizations
  • progress trackers
  • integration with external systems

• Keeps Borg backward-compatible (fully opt-in)

---

❓ Questions / Feedback

• Would such an observability layer align with Borg’s design goals?

• Are there existing discussions or constraints I should be aware of before prototyping?

• Preferred direction:

  • integrate with existing logging system
  • or introduce a separate structured event pipeline?

---

If this direction makes sense, I’d be happy to:

• prototype a minimal version (e.g., file-level events)
• iterate based on feedback

Thanks for your time and for maintaining such a powerful tool 🙌

[ThomasWaldmann]

Good idea. Please also check --log-json and --json-lines.

Guess we need to be a bit careful with the per-chunk stuff:

• it's quite high-frequency for little changed or not changed bigger files and especially the chunkids are quite long and maybe not that interesting outside of borg.
• the compressed / uncompressed chunk size, new chunk vs. reused chunk info is interesting for dynamic stats updates.

Rather use file_start and file_end (end means either success or failure).

Status could reuse the status character from human-readable --list output.

If it is a failure, file_end would include the failure reason.

For all file types other than regular files that file_start/file_end would be a bit much, because they don't have content in between. For these, there could be just a "file_end" event.

Could be done together with #9454, that would offer a borg-only way to use / visualize these new events.

master branch.

[YK-03]

The point about per-chunk events being too noisy makes sense — focusing on file-level events feels like a much cleaner approach.

I’m thinking of refining it to something like:

• file_start when processing begins
• file_end with status (reusing existing status char), optional failure reason, and some summary stats (size, reused vs new chunks)

For non-regular files, just a single file_end event sounds reasonable.

I’ll also check how --log-json / --json-lines are currently handled and align with that, and take a look at #9454 to understand the bigger picture.

Planning to start with a small prototype (file-level events + JSON lines, opt-in). Does this direction sound okay?

[ThomasWaldmann]

Sounds good. Have a look at the json api docs also, so it fits in there nicely.

[YK-03]

Hi @ThomasWaldmann ! I’ve opened a small prototype PR ( #9532 ) implementing per-item lifecycle events for file_status JSON output.

It adds a "phase" field:

• "start" when processing begins
• "end" for final status (default)

The implementation keeps JSON construction centralized and avoids changing non-JSON behavior.

Would love feedback on whether this direction aligns with the intended JSON API design.