feat(cutover): develop directly on the public repo — relocate the leak gate, retire the publish machinery#3
Merged
Merged
Conversation
…st used to protect The cutover retires scripts/publish/publish-denylist.txt. On the private repo CLAUDE.md, .claude/, docs/security/ and the other private-only paths were TRACKED, and publish.ps1 stripped them when producing the mirror. Development now happens directly on the public repo, so nothing strips them any more -- a gitignore rule is the only remaining control, and the cutover procedure runs `git add -A`. Verified before this commit: CLAUDE.md (33KB of internal engineering guidance) and .claude/settings.json were both untracked-and-committable in the fresh clone. Not ignored on purpose: tests/test_scan_forbidden.py, tests/test_anon_core.py and .github/dependabot.yml. They were deny-listed as well, but this change set publishes synthetic/public versions of them; ignoring them would silently drop intended content.
…k gate, retire the publish machinery
Switches MessageFoundry from the private-repo + published-mirror model to developing directly here.
The publish step was the thing enforcing several invariants; each one now needs an explicit control.
SCANNER RELOCATED AND HARDENED (scripts/publish/ -> scripts/security/)
The token list is externalized: the committed scanner carries only STRUCTURAL detectors plus a
synthetic .example, and the real customer/vendor tokens arrive from the MEFOR_FORBIDDEN_TOKENS
secret in CI and a git-ignored local file at commit time.
Requiring a token SOURCE is not sufficient, so this also adds:
* every floor section (names/estate/site_prefixes) must be non-empty -- a partially-mangled secret
previously loaded as few as 1 of 21 detectors and passed a gate calling itself fail-closed;
* a PER-SECTION floor (MEFOR_MIN_DETECTORS=names=7,estate=13,site_prefixes=1). A bare total is a
SUM, so growth in a cheap section masks collapse in an expensive one;
* rejection of entries that PARSE but can never FIRE -- an invisible/zero-width codepoint inside a
token (which survives str.strip(), unlike NBSP, and is what a paste through a rendering surface
produces), the (?!) never-match sentinel, non-ASCII digit prefixes, and duplicates;
* an identifier pass: `_` is a word character, so a \b-anchored pattern cannot see its token inside
OB_TOKEN_ORU. Restricted to SINGLE-TOKEN patterns -- applying it to multi-word patterns matched 9
ordinary snake_case identifiers on the real tree, every one a false positive;
* reason-only reporting on every path, including failures. Parse warnings no longer echo the
offending entry, and a REASON that matches its own pattern is replaced with a generic label --
both would have published a customer token into a world-readable Actions log;
* allowlist breadth validation: one degenerate entry ('.', '.*') vetoed every line BEFORE any
detector ran, disabling the whole gate while the counts line still reported full tables;
* --require-tokens[=N], because pre-commit can pass args to a hook but cannot set env for one.
CI + COMMIT-TIME WIRING
* security.yml: the forbidden-content job was VACUOUS on this repo (it invoked a scanner that was
deny-listed off the mirror, logged "skipping", and exited 0 behind a required check). It now runs
the relocated scanner fail-closed over the whole tree, and hard-errors when the secret is absent
on a non-fork run rather than silently degrading.
* security.yml gains a main-only push trigger: a fork PR is scanned structural-only by design, so
without this nothing fully-loaded ever re-scanned fork-contributed content after merge.
* .pre-commit-config.yaml: the forbidden-content hook is ADDED, not edited -- publish.ps1 stripped
it from every published snapshot, so it has never existed here. It carries --require-tokens, so a
checkout without the token file refuses to commit instead of passing green with zero detectors.
PRIVATE-PATH PROTECTION (see the preceding commit)
CLAUDE.md and .claude/ were TRACKED privately and kept out of the mirror by the deny-list. With the
deny-list retired, gitignore is the only remaining control.
ALSO
* ADR 0030 4h: the anonymizer's site-code prefix is externalized, so it is no longer a literal in
tracked source. test_security_static's unresolvable-regex pin is updated accordingly -- with the
reason, not silently.
* Two docs genericized: they wrote the real site prefix to describe the detector's shape. This is
what takes a whole-tree scan from 3 hits to 0, and is why no legacy path-exemption was ported.
* The gate's own tests no longer trip it: literal probe strings are assembled at runtime, matching
the convention the suite already used for IPs. Allowlisting the test files would have blinded the
gate to the exact classes they exercise.
* Retires release-sync-check.yml and backlog-hygiene.yml (scripts/publish/ and test_publish_gate.py
were already absent here).
VERIFIED
* whole-tree leak scan: exit 0 with all 21 detectors loaded, fail-closed
* full suite 8,891 passed / 0 failed; console suite 338 passed with one failure that reproduces on
a clean checkout of main and is unrelated (tracked separately)
* ruff format clean; ruff check 13 and mypy 21 -- both identical to a baseline measured on main
* 21 targeted mutations of the new guards, 0 undetected
…ip the parity tests
CI caught what local runs could not: test_leak_tables_are_sourced_from_the_guard_when_present failed
on all three test legs while passing locally.
ROOT CAUSE, and it is caused by this change set's own central decision. The test skipped when the
guard was ABSENT, else asserted the token tables were populated. That premise -- guard present implies
tokens populated -- held only while the guard lived under the deny-listed scripts/publish/ AND carried
its tokens as literals. This cutover breaks both halves: it relocates the guard to scripts/security/
(so it IS present on the public repo) and externalizes the tokens (so presence proves nothing). CI
took the non-skip path with no token source and found empty tables. It passed locally only because a
developer working tree has the git-ignored token file.
Now gated on the TOKEN SOURCE -- the condition that actually determines whether there is anything to
source. Verified in all three states: passes with tokens, skips without, and still FAILS when the
bridge in tee/anon/leak.py is blanked. Not traded for a vacuous skip.
TWO PARITY TESTS WERE SILENTLY SKIPPING
_load_scan_forbidden() still pointed at scripts/publish/scan_forbidden.py, which no longer exists, so
pytest.skip fired at module level. Those assertions are the only thing keeping tee/anon/leak.py's
token tables identical to the guard's -- without them the tee copy can drift unnoticed. Repointed;
they now run (10 passed, 0 skipped) and go red under mutation. Same failure shape as the vacuous
forbidden-content job: a check that looks green because it never executed.
REFERENCE SWEEP -- 32 references across 19 files, four distinct treatments
* REPOINT scan_forbidden.py -> scripts/security/, including relative markdown link depth.
* FLAG --published no longer exists (it scanned the deny-list's "publishable subset"); the
scanner now scans the whole tree via --path .
* REWORD publish.ps1, publish-denylist.txt and check_release_sync.py are GONE, with no new path.
Repointing them would have produced links to files that never existed. The Secure Build
Scorecard in particular asserted the gate scans "the publishable subset" via a deny-list
this branch deletes -- a correct path with a false claim is worse than a broken link.
* LEAVE past-tense statements ("formerly enforced by", "the retired ...") were already correct.
release.yml is deliberately COMMENT-ONLY. Its slug-guard logic is release-critical and contains a
now-dead no-op sed; both belong in their own change rather than buried in a cutover sweep. The
comments still had to be fixed -- a false comment next to a guard is how the guard later gets
"normalized" into breaking.
Also updates .github/CODEOWNERS from /scripts/publish/ to /scripts/security/.
VERIFIED: ruff format clean; ruff check 13 and mypy 21, both identical to a baseline measured on main;
whole-tree leak scan exit 0 with all 21 detectors; 129 tests across the affected suites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switches MessageFoundry from the private-repo + published-mirror model to developing directly here.
The publish step was quietly enforcing several invariants. Retiring it means each one needs an explicit control, and this PR adds them.
The gate was vacuous on this repo
forbidden-contentis a required check. Before this PR it invoked a scanner that had been deny-listed off the mirror, loggedscan_forbidden.py absent (OSS mirror) — skipping, and exited 0. A required check that cannot fail.It now runs the relocated scanner fail-closed over the whole tree, and hard-errors when the secret is absent on a non-fork run rather than degrading silently.
Presence is not sufficiency
Requiring a token source only proves one arrived. A partially-mangled secret previously loaded as few as 1 of 21 detectors and passed a gate calling itself fail-closed. So requiring tokens now also requires:
names/estate/site_prefixes) to be non-empty;str.strip(), unlike NBSP — exactly what a paste through a rendering surface produces), the(?!)never-match sentinel, non-ASCII digit prefixes, and duplicates.Detection gaps closed
_is a word character, so a\b-anchored pattern cannot see its token insideOB_TOKEN_ORU. A second pass with_neutralised closes this for every name pattern. Restricted to single-token patterns — applied to multi-word patterns it matched 9 ordinarysnake_caseidentifiers on the real tree, every one a false positive._WORKTREE_SLUG,_HOME_PATH), verified byte-identical to the originals.Reason-only on every path, including failures
Parse warnings no longer echo the offending entry, and a REASON matching its own pattern is replaced with a generic label. Both would have published a customer token into a world-readable Actions log — on the success path, in the second case.
Also
.pre-commit-config.yaml: the hook is added, not edited —publish.ps1stripped it from every published snapshot, so it has never existed here. It carries--require-tokens, because pre-commit can pass args but cannot set env for one hook; without it a fresh clone commits with zero customer detectors and reports "Passed"..,.*) vetoed every line before any detector ran, disabling the gate while the counts line still reported full tables.security.ymlgains a main-onlypushtrigger — a fork PR is scanned structural-only by design, so nothing fully-loaded ever re-scanned fork-contributed content after merge.Verified
main, unrelated, tracked separatelymain