verify-before-write

Patterns for letting AI agents write to systems where a wrong number is a reportable event — inspect, propose, apply, verify.

Demo

The committed City of Riverton example, end to end (synthetic data — every entity and figure is fictional):

$ python3 inspect.py --statement work/statement.csv --map work/account_map.csv --out work/findings.json
inspect: 3 finding(s)
  [hardcoded-plug] ASSETS / Capital assets, net / General Fund: face 5,000,000 vs recomputed 4,850,000 (residual +150,000)
  [sign-error] LIABILITIES / Accounts payable / General Fund: face -285,000 vs recomputed 285,000 (residual -570,000)
  [unmapped-account] GL 1350 Prepaid items (General Fund, 45,000): no account_map row — surfaced, not auto-fixed
findings written to work/findings.json

$ python3 propose.py --findings work/findings.json --statement work/statement.csv --map work/account_map.csv --out work/changeset.json
propose: 2 change(s), 1 surfaced finding(s)
  set ASSETS / Capital assets, net / general_fund: 5,000,000 -> 4,850,000  (hardcoded-plug)
  set LIABILITIES / Accounts payable / general_fund: -285,000 -> 285,000  (sign-error)
  surfaced (needs judgment): unmapped-account — GL 1350 Prepaid items (General Fund, 45,000) has no account_map row; its amount is silently absent from every statement line
changeset written to work/changeset.json (no statement bytes touched)

$ CONFIRM=1 python3 apply.py --changeset work/changeset.json --journal work/journal.jsonl
apply: 2 change(s) written, readback matched prediction — adopted (journal: work/journal.jsonl)
  still open (needs judgment): unmapped-account — GL 1350 Prepaid items (General Fund, 45,000) has no account_map row; its amount is silently absent from every statement line

  ... a human adds the GL 1350 mapping row, the loop runs a second pass ...

$ python3 tieout.py --statement work/statement.csv --map work/account_map.csv --report work/tieout_report.md
  [1] Trial balance integrity (sums to zero, overall and per fund): PASS
  [2] Mapping coverage (every account maps to exactly one line): PASS
  [3] Line recompute (face = sign x mapped balances + adjustment; subtotals foot): PASS
  [4] Statement equation (assets = liabilities + net position, per column): PASS
  [5] Cross-foot (total column = General Fund + Water Utility, per row): PASS
tie-out: 5/5 checks PASS — exit 0 (report: work/tieout_report.md)

Why this exists

Agents are good at proposing changes and bad at knowing when they are wrong. In regulated financial reporting the cost of a silent wrong write is asymmetric — a missed defect is embarrassing, a confidently written wrong number in a published statement is a reportable event — so the write path, not the model, must carry the proof. These patterns were developed over 18 months of production agent tooling operating on live government financial reports (platform: Workiva); everything here is the transferable shape.

The loop

flowchart LR
    A["inspect<br/>recompute from sources"] --> B["propose<br/>changeset + predicted poststate"]
    B --> C{"gates<br/>confirm / prestate hash<br/>fixable-only"}
    C -->|dry run| D["diff printed<br/>nothing written"]
    C -->|CONFIRM=1| E["apply"]
    E --> F["readback"]
    F -->|matches prediction| G["adopt + journal"]
    F -->|mismatch| H["revert to prestate<br/>+ journal"]
    G --> I["verify<br/>tie-out oracle, exit 0/1"]
    H --> I

Loading

The six patterns

Pattern	The memorable rule
1. The loop	The oracle is the proof, not the diff.
2. Write gates	Every gate is cheap; a wrong write is not. Dry-run by default, fail closed.
3. Tie-out oracles	Classify every residual; never plug.
4. Journaled changesets	Predict the poststate before you write; adopt on match, revert exactly on mismatch.
5. Detector honesty	A scan that cries wolf is worse than a narrower one that is trusted.
6. Architecture over hardcodes	Route every value through controls; a value the architecture cannot carry is a defect in the architecture.

Worked example: City of Riverton

examples/riverton/ derives a two-fund Statement of Net Position from a synthetic trial balance, an account map, and preparer adjustment cells. The as-found statement carries three seeded defects — a hardcoded plug, a sign error, and an unmapped account — chosen so they do not compensate. The loop fixes the two mechanical defects, surfaces the third for human judgment, and iterates to a passing tie-out.

git clone https://github.com/dbett4/verify-before-write.git
cd verify-before-write/examples/riverton
mkdir -p work && cp statement_before.csv work/statement.csv && cp data/account_map.csv work/account_map.csv

# Pass 1 — fixes the two mechanical defects; the third needs a human
python3 inspect.py --statement work/statement.csv --map work/account_map.csv --out work/findings.json
python3 propose.py --findings work/findings.json --statement work/statement.csv --map work/account_map.csv --out work/changeset.json
python3 apply.py --changeset work/changeset.json          # dry run
CONFIRM=1 python3 apply.py --changeset work/changeset.json
python3 tieout.py --statement work/statement.csv --map work/account_map.csv --report work/tieout_report.md
# expected: 3/5 checks PASS, exit 1 — GL 1350 is still unmapped by design;
# pass 2 below resolves the judgment-call defect

# The human step — adopt the prepared mapping row for GL 1350
python3 -c "from pathlib import Path; p = Path('work/account_map.csv'); p.write_text(p.read_text().replace('# 1350', '1350'))"

# Pass 2 — the loop inserts the newly mapped line; the oracle passes
python3 inspect.py --statement work/statement.csv --map work/account_map.csv --out work/findings.json
python3 propose.py --findings work/findings.json --statement work/statement.csv --map work/account_map.csv --out work/changeset.json
CONFIRM=1 python3 apply.py --changeset work/changeset.json
python3 tieout.py --statement work/statement.csv --map work/account_map.csv --report work/tieout_report.md
# expected: 5/5 checks PASS, exit 0

Stdlib-only Python, no installs, no credentials; the annotated walkthrough is in the example README. CI runs exactly this sequence on every push, byte-compares the result against expected/statement_after.csv, and then runs pytest (tests/test_riverton_loop.py — including the revert-on-mismatch path).

Decision records

Decisions in the source system were recorded as ADRs — measured prevalence tables, rejected candidates with counts, alternatives considered. This repo demonstrates the format with a fresh ADR written for the Riverton detector scope: docs/adr/0001-riverton-detector-scope.md.

Status

Pattern extraction from private production tooling. The private system runs against a commercial reporting platform's REST API; this repo contains no platform code and no client data, and all figures are fictional. What this demonstrates: the write path, not the model, carries the proof.

More at davebettner.com.

License

MIT