Skip to content

feat(canopy): N2: blob store backups and paired restore - #810

Open
dannash100 wants to merge 10 commits into
mainfrom
feat/blob-store-backups
Open

feat(canopy): N2: blob store backups and paired restore#810
dannash100 wants to merge 10 commits into
mainfrom
feat/blob-store-backups

Conversation

@dannash100

@dannash100 dannash100 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Implements the bestool side of Tamanu's blob store backup spec (specs/blob-storage/backups.md, id BKUP, on beyondessential/tamanu#10708). The store is a content-addressed, append-only tree, so a tamanu-postgres backup alone no longer captures a server's whole state, and the two captures have to stay ordered (database first, store second) and be restorable as a pair. Today types are independent: nothing orders them, and pairing is an operator eyeballing timestamps. Continues from #809.

Three mechanisms:

  • A def can declare after = "<type>". After a successful run of that type the driver runs its followers, so the store is captured right after each database backup. Needs nothing from Canopy; a schedule on the follower type is a safe backstop since a store capture alone is a superset of any earlier database capture. Chaining lives in run_backup so daemon, CLI-via-daemon, and local runs behave identically.
  • Restore plans the cycle up front, then restores followers from the earliest same-host snapshot of their type started at or after the chosen one, refusing when only earlier ones exist (they can lack referenced content). Snapshots self-identify by the canopy-type tag every backup already carries, with the type in the kopia description as a second signal. kopia start time stands in for the freeze moment; it's conservative in the safe direction.
  • [simple] accepts path_command as an alternative to path, and bestool tamanu blob-root prints the store root from Tamanu's blobStorage.root setting, so the def follows an admin-panel change instead of silently going stale, and a restored store lands where the just-restored database expects it.

Checked on a real repo against the pinned kopia 0.23.1, since pairing rests on it: both signals come back from snapshot list. Tags looked like they didn't because the key comes back spelled tag:canopy-type. Keying on the tag also covers snapshots taken before this branch, and drops a source-path fallback that could only fire on Linux. The proxy e2e asserts both now.

USAGE.md was regenerated on macOS without the iti features and spliced by hand; the check-docs job is the byte-for-byte confirmation.

@dannash100 dannash100 changed the title feat(canopy): blob store backups and paired restore feat(canopy): N2: blob store backups and paired restore Aug 6, 2026
@dannash100

dannash100 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

🤖 No Canopy code change. BackupType is open (Custom(String)), staleness and alert refs are already per (server, type), and restore credentials are issued per (server, type).

One config step, and it's required rather than optional: crates/database/src/backup/staleness.rs only scans types with a non-NULL expected_interval, and a newly-seen type arrives with none. A follower-only blob type would never alert, so a chain that stops leaves the store unbacked and silent.

On Canopy, once the type name is final:

  1. Backup defaults screen, "Add a backup type", enter the name.
  2. Give it an interval, longer than the postgres one. The chain covers the common case; the scheduled run adds a capture on top, cheap under dedup but not free.
  3. Set retention.
  4. Set auto_enable, or leave it off and enable the capability per group on the group's backup page.
  5. Per-group interval override on that page if a group needs a different cadence.

No migration, no PR.

@passcod

passcod commented Aug 10, 2026

Copy link
Copy Markdown
Member

I think it does need something from canopy, because canopy won't allow the creds for the followers

@dannash100

Copy link
Copy Markdown
Contributor Author

🤖 You're right, and my comment above is wrong on that. require_backupable_capability (crates/public-server/src/backup.rs:110) needs the type to be an enabled capability on the server, and a newly-registered type seeds enabled from that type's auto_enable, which is false when no backup_type_defaults row exists. So by default every follower run 409s at issuance.

Still config rather than code, but with an ordering constraint: the type default has to exist, with auto_enable = true, before the follower def ships. register only seeds on insert (crates/database/src/backups.rs:546), so if devices register the type first, the capability rows land disabled and setting the default afterwards doesn't touch them, leaving a per-server flip.

Enabling can't be coded away regardless, since the staleness scan filters on enabled too.

If the ordering risk is worth code, the smallest version is an operator-set flag on the capability so register re-seeds from the current default while it's unset. Own canopy PR, helps any future type. Teaching canopy the chain instead is bigger and means enabling postgres implicitly authorises blob uploads, which is what the per-type gate is there to prevent.

Also correcting the comment above: restore creds aren't gated per type, only by the per-server restore window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants