Skip to content

feat(alertd): blob integrity, correction rate and antivirus checks - #820

Open
dannash100 wants to merge 3 commits into
mainfrom
feat/blob-integrity-antivirus-checks
Open

feat(alertd): blob integrity, correction rate and antivirus checks#820
dannash100 wants to merge 3 commits into
mainfrom
feat/blob-integrity-antivirus-checks

Conversation

@dannash100

@dannash100 dannash100 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The blob storage epic (tamanu#10669) ships a runbook for blob_integrity, blob_correction_rate and blob_antivirus, and docs/healthchecks.md maps all three, but nothing emits them. Support would look up a check that never fires.

Three checks, all registered Tamanu-dependent so they run on facility as well as central: every server that stores blobs scrubs its own and reads its own media, and the quarantine record propagates to all of them. Thresholds come from the runbooks, and where a runbook gives no number the choice carries a comment naming the constraint.

blob_integrity splits faults by whether this server holds the only durable copy: corrupt or absent content that must be durably present here FAILs, a faulty facility cache blob WARNs and self-corrects, and a store nothing has scrubbed for 6h WARNs (the scrub is hourly, so 6 missed passes). blob_antivirus skips entirely where no scanner is named, WARNs on any quarantine, and WARNs on a scan stall that becomes a FAIL under only-known-good, where unscanned means unserved.

A facility cache fault needed a change on the Tamanu side to be observable at all: FacilityBlobHealer deletes the row, which is what makes the repair self-correcting and also what makes it indistinguishable from an eviction, so the registry has nothing left to count. tamanu#10750 counts the drop in local_system_facts instead, and blob_integrity reads it: a run of them with one recently is a WARN, and the lifetime total goes out as a gauge. The counter never resets, so it says how many and how recently, never how many in a window. That check degrades to nothing on a Tamanu without the counter, so this does not have to land second.

Three things a reviewer might read as mistakes:

  • Central corruption FAILs unconditionally rather than only when a live record references it, which is what the runbook actually turns on. blobs is keyed by hash with no back-link to the attachment and asset tables, so the reference cannot be established from the registry.
  • A quarantine is capped at WARN. The runbook says not to delete quarantine rows and that the record is meant to stand, so a FAIL could never be cleared by anything it permits.
  • No trend grading. A sweep is stateless, so the "rising across successive passes" the runbooks escalate on goes out as gauges for munin instead, and the antivirus check substitutes "no verdict for 2h with content waiting" for "flat".

Supersedes #819, whose commit is cherry-picked here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants