feat(alertd): blob integrity, correction rate and antivirus checks - #820
Open
dannash100 wants to merge 3 commits into
Open
feat(alertd): blob integrity, correction rate and antivirus checks#820dannash100 wants to merge 3 commits into
dannash100 wants to merge 3 commits into
Conversation
passcod
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The blob storage epic (tamanu#10669) ships a runbook for
blob_integrity,blob_correction_rateandblob_antivirus, anddocs/healthchecks.mdmaps all three, but nothing emits them. Support would look up a check that never fires.Three checks, all registered Tamanu-dependent so they run on facility as well as central: every server that stores blobs scrubs its own and reads its own media, and the quarantine record propagates to all of them. Thresholds come from the runbooks, and where a runbook gives no number the choice carries a comment naming the constraint.
blob_integritysplits faults by whether this server holds the only durable copy: corrupt or absent content that must be durably present here FAILs, a faulty facility cache blob WARNs and self-corrects, and a store nothing has scrubbed for 6h WARNs (the scrub is hourly, so 6 missed passes).blob_antivirusskips entirely where no scanner is named, WARNs on any quarantine, and WARNs on a scan stall that becomes a FAIL underonly-known-good, where unscanned means unserved.A facility cache fault needed a change on the Tamanu side to be observable at all:
FacilityBlobHealerdeletes the row, which is what makes the repair self-correcting and also what makes it indistinguishable from an eviction, so the registry has nothing left to count. tamanu#10750 counts the drop inlocal_system_factsinstead, andblob_integrityreads it: a run of them with one recently is a WARN, and the lifetime total goes out as a gauge. The counter never resets, so it says how many and how recently, never how many in a window. That check degrades to nothing on a Tamanu without the counter, so this does not have to land second.Three things a reviewer might read as mistakes:
blobsis keyed by hash with no back-link to the attachment and asset tables, so the reference cannot be established from the registry.Supersedes #819, whose commit is cherry-picked here.