From 519cdbcee54a6631ee3c5984aa863ed8fdf18548 Mon Sep 17 00:00:00 2001 From: Julien Danjou Date: Sun, 26 Jul 2026 10:01:48 +0200 Subject: [PATCH] feat(agents): stop agents leaking internal information into published docs Docs are usually written *from* internal material: a support case, a private PR, an engine source file, a Linear ticket. That material travels with the draft, and nothing in the repo told an agent to leave it behind. A recent billing page was written from a support ticket whose context carried a helpdesk ticket ID, a thread ID, a customer org name, and a private shadow-office source path. None of that belongs on docs.mergify.com, and a leaked identifier cannot be unpublished. This follows the repo's existing agent-guidance structure rather than adding a parallel document: - **AGENTS.md** gains Critical Rule 8 plus a "Never Publish Internal Information" section listing concretely what must never ship (ticket and thread IDs, customer/org/account identifiers, private repo and source paths, internal trackers and admin URLs, credentials, internal framing), and how to fix a leak: state the behavior, drop the provenance. - **`proofread-leaks`** joins the proofread pipeline as a 5th subagent, written to match the existing proofread-* skills. It is the one proofreader that never skips: the pipeline's 10-line threshold and its "code blocks only" / "frontmatter only" exemptions are exactly where a one-line ticket ID hides, so those exemptions no longer apply to it. - **`pnpm check:internal-leaks`** (`scripts/check-internal-leaks.mjs`, plus a CI job) is the deterministic backstop, modelled on validate-config-examples: fixed patterns for ticket/thread IDs, private repo and source paths, and internal trackers and hostnames, with a per-line `internal-leaks: allow ` escape hatch. - **docs-from-code**, **docs-gap-analysis**, and **document-a-feature** each gain a pointer, since those are the skills that pull internal material in. Every rule is narrow enough to leave today's docs clean (146 files scanned, 0 findings) and to ignore the placeholders the docs already use, so `ghp_*`, ``, `Mergifyio/mergify`, and `app.datadoghq.com` do not fire. Credentials stay in the policy and in proofread-leaks' checklist, but the deterministic scan does not look for them: secret detection is not this script's job to reinvent. Judgment call worth a look: the ask was for instructions, and the CI scanner goes beyond that. I added it because the skill-level check only fires when an agent runs the pipeline, which leaves human edits and skipped passes uncovered, and because this is the one class of mistake that is unfixable after the fact. Happy to drop the script and keep the instructions if you would rather not carry the CI surface. The scanner cannot catch customer names, copied examples, or internal framing. That half stays with proofread-leaks, and the docs say so in both places so a green CI run is not mistaken for a clean bill of health. Change-Id: I56a581e7825816dbdc7e7f9be082f39d799cd72f --- .claude/skills/docs-from-code/SKILL.md | 5 + .claude/skills/docs-gap-analysis/SKILL.md | 3 + .claude/skills/document-a-feature/SKILL.md | 19 +- .claude/skills/proofread-leaks/SKILL.md | 202 +++++++++++++++++++++ .github/workflows/ci.yaml | 17 ++ AGENTS.md | 65 ++++++- package.json | 1 + scripts/check-internal-leaks.mjs | 160 ++++++++++++++++ scripts/check-internal-leaks.test.mjs | 76 ++++++++ 9 files changed, 541 insertions(+), 7 deletions(-) create mode 100644 .claude/skills/proofread-leaks/SKILL.md create mode 100644 scripts/check-internal-leaks.mjs create mode 100644 scripts/check-internal-leaks.test.mjs diff --git a/.claude/skills/docs-from-code/SKILL.md b/.claude/skills/docs-from-code/SKILL.md index fb5b0feea1..18a939977c 100644 --- a/.claude/skills/docs-from-code/SKILL.md +++ b/.claude/skills/docs-from-code/SKILL.md @@ -84,6 +84,11 @@ Input: a feature — a PR number, or a code area / config key. docs repo, never add it to `.gitignore` (keep it in the scratch dir). - Document the user-facing contract, not implementation details. Code is the source of truth for *what is true*, not for *how to say it*. +- **The clone is private. Nothing that identifies it may reach the docs** — no + `Mergifyio/monorepo` references, no `mergify_engine/...` paths, no internal + module or function names, no `file:line` citations. Cite the code in your + *report* to the user, never in the page. See `proofread-leaks` and the + "Never Publish Internal Information" section of AGENTS.md. - When code and a doc disagree, the code wins for facts (defaults, enums, behavior) — but if you cannot tell whether a difference is intentional, flag it rather than "fixing" it. diff --git a/.claude/skills/docs-gap-analysis/SKILL.md b/.claude/skills/docs-gap-analysis/SKILL.md index 544c3e8dc7..aff3eb62c6 100644 --- a/.claude/skills/docs-gap-analysis/SKILL.md +++ b/.claude/skills/docs-gap-analysis/SKILL.md @@ -89,6 +89,9 @@ Create a todo for each step. target. - Don't flag internal/refactor PRs as doc gaps. Only user-facing capability changes count. +- The report is internal, the docs are not. Linear IDs, monorepo paths, and + customer names belong in the report and must never survive into a page written + from it — see `proofread-leaks`. - Be honest about uncertainty: if you can't tell whether something is documented, say "needs review" rather than asserting a gap. - The output is a plan, not edits. To actually write a page, hand the item to diff --git a/.claude/skills/document-a-feature/SKILL.md b/.claude/skills/document-a-feature/SKILL.md index 3f6c7f9d7b..23ce6cd820 100644 --- a/.claude/skills/document-a-feature/SKILL.md +++ b/.claude/skills/document-a-feature/SKILL.md @@ -47,6 +47,11 @@ Identify the source of truth for the feature: Establish: what the feature does, who uses it, the exact config/CLI/API surface, and any plan/tier gating. +Every one of these sources is internal. Take the *behavior* from them and leave +the *provenance* behind: no ticket or Linear IDs, no customer or org names, no +monorepo paths, no internal links. Read the "Never Publish Internal Information" +section of AGENTS.md before you write a line. + ## Step 2 — Place the page Read `references/placement-guide.md` for the full section map and decision rules. @@ -106,13 +111,15 @@ exact snippets. Run these before claiming the page is done: -1. **Proofread pipeline** — if total changed/added docs lines ≥ 10, spawn the 4 - proofread subagents in parallel per the repo CLAUDE.md (`proofread-style`, - `proofread-technical`, `proofread-structure`, `proofread-consistency`) on the - diff. -2. **Config examples** — if the page contains any YAML/`.mergify.yml` snippets, +1. **Leak check** — always, whatever the diff size: run + `pnpm check:internal-leaks` and the **proofread-leaks** subagent on the diff. +2. **Proofread pipeline** — if total changed/added docs lines ≥ 10, spawn the + other 4 proofread subagents in parallel per the repo CLAUDE.md + (`proofread-style`, `proofread-technical`, `proofread-structure`, + `proofread-consistency`) on the diff. +3. **Config examples** — if the page contains any YAML/`.mergify.yml` snippets, invoke the **validate-config-examples** skill. -3. **Build check** — run `pnpm check` (astro check + eslint + biome). Fix any +4. **Build check** — run `pnpm check` (astro check + eslint + biome). Fix any errors. For a final SSG sanity check, `pnpm build`. ## Hard rules diff --git a/.claude/skills/proofread-leaks/SKILL.md b/.claude/skills/proofread-leaks/SKILL.md new file mode 100644 index 0000000000..06153be3b1 --- /dev/null +++ b/.claude/skills/proofread-leaks/SKILL.md @@ -0,0 +1,202 @@ +--- +name: proofread-leaks +description: >- + Proofread documentation changes for leaked internal + information. Catches support ticket and thread IDs, + customer and org names, private repo paths, internal + URLs and dashboards, and credentials that came in + from the source material a page was written from. + Designed to run as a subagent on diffs of MDX files. +--- + +# Proofread: Internal Information Leaks + +You are a proofreading subagent. Your job is to make +sure nothing internal ended up in published +documentation. + +Docs are usually written *from* internal material: a +support case, a private PR, an engine source file, a +Linear ticket. That material travels with the draft. +The reader never needs to know where a page came +from, so anything that only describes the *provenance* +of a doc is a leak, not context. + +This check is different from the other proofreaders in +two ways: it runs on **every** docs change regardless +of size, and it covers **code blocks, comments, +frontmatter, alt text, and image contents** — not just +prose. + +## What You Receive + +You will be given a diff or list of changed MDX files +in `src/content/docs/`. Review ONLY the changed/added +lines, including code fences and frontmatter. + +## What You Check + +### Never Publish + +1. **Support ticket and thread IDs.** Plain ticket + refs (`T-1234`), Plain thread IDs + (`th_01JQZK4M8XN2VR7TDY0P3WGA6H`), `app.plain.com` + links, or any equivalent from another helpdesk. + +2. **Customer, org, and account identifiers.** GitHub + org or repo names belonging to a customer, account + IDs, subscription IDs, Stripe customer or invoice + IDs, seat counts of a real account, email + addresses. + +3. **Private repositories and internal code paths.** + `Mergifyio/monorepo`, `mergify_shadow_office/...`, + `mergify_engine/...`, dashboard source paths, + internal module, class, or function names, and + file:line references into private code. + +4. **Internal URLs and tools.** Linear and Notion + links, internal dashboards and admin consoles, + staging or internal Mergify hostnames, cloud + consoles, internal runbooks. + +5. **Credentials.** Real tokens, API keys, private + keys, or webhook secrets. Placeholders like + `ghp_*` or `` are fine; anything that + could be a live secret is not. + +6. **Internal-only framing.** Employee names in an + internal context, quotes from a support + conversation, "as discussed with the customer", + references to unreleased work, or an explanation + that only makes sense if you have read the ticket. + +### Signals a Leak Is Nearby + +Read these lines extra carefully: + +- Anything that explains *why* the page was written, + or that names a specific incident. +- Examples that look copied rather than constructed — + a real-looking org name, a specific date, an odd + number of contributors, a plausible invoice amount. +- "For example, one customer…" openers. +- YAML examples whose `repository`, `owner`, or + `login` values are not obvious placeholders. +- Screenshots and their alt text: a dashboard capture + can show a real org name, a customer's repo list, or + an email address. + +### What Is Fine + +- Public Mergify repos (`Mergifyio/mergify`, + `Mergifyio/gha-mergify-ci`), public product URLs + (`dashboard.mergify.com`, `docs.mergify.com`). +- Public docs of third parties, including their + dashboards (`app.datadoghq.com`, GitHub settings + pages) when the reader is meant to go there. +- Generic placeholders: `my-org`, `your-repo`, + ``, `user@example.com`. +- Product behavior learned from a private source. The + *fact* is publishable; the *source* is not. + +## How to Verify + +1. **Run the deterministic scan first.** It catches + the mechanical patterns in seconds: + + ```bash + pnpm check:internal-leaks + ``` + + It only knows fixed patterns. Customer names, copied + examples, and internal framing are yours to catch — + do not treat a clean run as a pass. + +2. **Re-read every added example.** For each org name, + repo name, number, and URL in the diff, ask: could + this be real? If you cannot tell, replace it with an + obvious placeholder. + +3. **Check the source you wrote from.** If a ticket, a + private PR, or an engine file was in your context, + grep the diff for its identifiers. + +4. **Open added screenshots** and look at the pixels, + not just the filename. + +## How You Fix + +Rewrite so the page states the product behavior and +drops the provenance. Removing the sentence is usually +wrong — the behavior is why the page exists. + +Leak: + +```text +Customer acme-corp (ticket T-1234) cancelled mid-period +and still received an invoice, because prorations are +collected on the next invoice (see +mergify_shadow_office/models/billing.py). +``` + +Fixed: + +```text +If you cancel mid-period, you may still receive an +invoice on the last day of the period. It collects the +prorations recorded during the period, so it is a +true-up rather than a renewal. +``` + +## How You Report + +1. **Always fix, never only flag.** A leak is the one + class of issue you must not leave in the file for + someone else. Redact or rewrite it, then report it. + +2. **Report every removal explicitly**, so a human can + judge whether the value also needs scrubbing from + the branch history or from an already-published + page: + + ```text + LEAK: [file:line] — [what kind] — removed/replaced with [what] + ``` + +3. **Escalate what you cannot judge.** If you cannot + tell whether a name is a real customer, do not + guess. Replace it with a placeholder and say so: + + ```text + CHECK: [file:line] — replaced "acme-corp" with "my-org"; confirm it was not a real customer + ``` + +4. **Follow-up suggestions:** If you spot a probable + leak in surrounding UNCHANGED content, note it + rather than editing: + + ```text + FOLLOW-UP: [file:line] — [description] + ``` + + Say so in your summary even if the diff was clean. + An already-published leak is more urgent than one + you just caught. + +## Scope + +- Review changed/added content in MDX files under + `src/content/docs/`, plus any image added alongside + them. + +- Do NOT edit `src/content/changelog/` — those files + are autogenerated. Report a leak there as a + FOLLOW-UP so a human can fix it upstream. + +- Fix leaks and nothing else. Leave style, structure, + and terminology to the other proofreaders. + +- When you are unsure whether something is internal, + treat it as internal. A placeholder costs the reader + nothing; a leaked identifier cannot be unpublished. diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 68c698688f..4161d55370 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -86,6 +86,23 @@ jobs: pnpm install --frozen-lockfile pnpm check:config-examples + internal-leaks: + timeout-minutes: 5 + runs-on: ubuntu-24.04 + steps: + - name: Checkout 🛎️ + uses: actions/checkout@v7.0.1 + + - name: Setup Node 🔧 + uses: actions/setup-node@v7.0.0 + with: + node-version-file: .node-version + + # No dependencies needed: the scanner is plain Node. Its own unit tests + # run in the `test` job. + - name: Check for leaked internal information + run: node scripts/check-internal-leaks.mjs + build: timeout-minutes: 20 runs-on: ubuntu-24.04 diff --git a/AGENTS.md b/AGENTS.md index 1945a203f4..d482317cc0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,6 +40,9 @@ pnpm format:check # Check formatting without modifying # Testing pnpm test # Run Vitest unit tests ./scripts/detect-broken-links.sh # Check for broken links + +# Safety +pnpm check:internal-leaks # Scan docs for leaked internal information ``` ## Key Directories @@ -147,6 +150,55 @@ Do NOT use regular `git push` - always use `mergify stack push`. option's own reference page (matching existing action pages) is acceptable; dated migration notices are not. +8. **Never publish internal information** - Ticket IDs, customer names, private + repo paths, internal links, and credentials must never reach + `src/content/docs/`. See the section below; it applies to every change, with + no size threshold. + +## Never Publish Internal Information + +Docs are usually written *from* internal material: a support case, a private PR, +an engine source file, a Linear ticket. That material travels with the draft. +The reader never needs to know where a page came from, so anything describing +the *provenance* of a doc is a leak, not context. A leaked identifier cannot be +unpublished. + +**Never write any of these into `src/content/docs/`** - prose, code blocks, +comments, frontmatter, alt text, or screenshots: + +- **Support ticket and thread IDs** - Plain refs (`T-1234`), thread IDs + (`th_01JQ...`), `app.plain.com` links, or the equivalent from any helpdesk. +- **Customer, org, and account identifiers** - a customer's GitHub org or repo, + account and subscription IDs, Stripe customer or invoice IDs, email addresses, + real seat or contributor counts. +- **Private repositories and internal code paths** - `Mergifyio/monorepo`, + `mergify_shadow_office/...`, `mergify_engine/...`, dashboard source paths, + internal module or function names, `file:line` references into private code. +- **Internal URLs and tools** - Linear and Notion links, admin consoles, + internal dashboards, staging or internal Mergify hostnames, runbooks. +- **Credentials** - real tokens, API keys, private keys, webhook secrets. + Placeholders (`ghp_*`, ``) are fine. +- **Internal-only framing** - employee names in an internal context, quotes from + a support conversation, "as discussed with the customer", unreleased work, or + an explanation that only makes sense if you read the ticket. + +The fix is almost never deletion. State the product behavior and drop the +provenance: "customer acme-corp on ticket T-1234 was billed after cancelling" +becomes "if you cancel mid-period, a final invoice may still arrive on the last +day of the period." + +**Enforcement**, in order of what catches what: + +1. `pnpm check:internal-leaks` - deterministic scan for the mechanical patterns + (ticket IDs, internal paths, internal trackers and hosts). Run it before + committing any docs change. It also runs in CI. It cannot catch customer + names, copied examples, or internal framing, so a clean run is not a pass. +2. The `proofread-leaks` subagent - judgment-level review of the diff. **Unlike + the other proofreaders, this one runs on every docs change**, including + sub-10-line edits, frontmatter-only edits, and code-block-only edits. +3. If something internal did reach a commit, say so plainly instead of quietly + amending it. A value that was pushed may need scrubbing beyond the branch. + ## Documentation Proofreading After editing MDX files in `src/content/docs/`, if the @@ -160,7 +212,7 @@ pipeline before considering the work done. MDX files in `src/content/docs/` (use `git diff` for staged/unstaged changes). -2. **Spawn 4 proofreading subagents in parallel** +2. **Spawn 5 proofreading subagents in parallel** using the Agent tool, each with the diff and the list of changed files: - Style (`proofread-style`): AI patterns, banned @@ -175,6 +227,9 @@ pipeline before considering the work done. - Consistency (`proofread-consistency`): terminology drift, capitalization, naming + - Leaks (`proofread-leaks`): ticket IDs, customer + names, private paths, internal links, secrets + 3. **Each subagent:** - Reads the skill with the Skill tool for its full guidelines @@ -215,6 +270,14 @@ When spawning each subagent, include: ### When to Skip +**`proofread-leaks` is never skipped.** None of the +conditions below apply to it: a leaked ticket ID is +usually a one-line change, and it hides in code blocks +and frontmatter as readily as in prose. Run it, plus +`pnpm check:internal-leaks`, on every docs change. + +The other four are skipped when: + - Changes are fewer than 10 lines of docs content - Changes are ONLY to frontmatter metadata diff --git a/package.json b/package.json index eb1e327f8f..2b531bd516 100644 --- a/package.json +++ b/package.json @@ -15,6 +15,7 @@ "format:check": "biome format src integrations plugins scripts", "check": "astro check && eslint . && biome check .", "check:config-examples": "node scripts/validate-config-examples.mjs", + "check:internal-leaks": "node scripts/check-internal-leaks.mjs", "check:links": "linkinator dist/ --recurse --concurrency 25 --verbosity error --skip 'https?://'" }, "devDependencies": { diff --git a/scripts/check-internal-leaks.mjs b/scripts/check-internal-leaks.mjs new file mode 100644 index 0000000000..11ba7d944a --- /dev/null +++ b/scripts/check-internal-leaks.mjs @@ -0,0 +1,160 @@ +#!/usr/bin/env node +/** + * Scan the docs for internal information that must never be published: support + * ticket and thread IDs, private repository and source paths, and internal + * tracker links. + * + * Docs are frequently written *from* internal material — a support case, a + * private PR, an engine source file — and that material travels with the draft. + * This catches the mechanical half of the problem so a leak fails CI instead of + * shipping to docs.mergify.com. + * + * It only knows fixed patterns. Customer and org names, copied examples, and + * "as discussed with the customer" framing cannot be regexed; those are the + * `proofread-leaks` skill's job. A clean run here is not a clean bill of health. + * + * Usage: + * node scripts/check-internal-leaks.mjs [paths...] # scan (default: src/content/docs) + * node scripts/check-internal-leaks.mjs --json [paths...] + * + * False positives are expected to be rare enough to handle one at a time. To + * allow a specific line, put a comment on the line before it naming the rule: + * + * {/* internal-leaks: allow internal-source-path — documenting the OSS engine layout *\/} + * + * In YAML or shell fences, use `# internal-leaks: allow — why`. + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = path.resolve(fileURLToPath(import.meta.url), '..', '..'); +const DEFAULT_TARGETS = ['src/content/docs']; +const SCANNED_EXTENSIONS = ['.mdx', '.md']; + +/** + * Each rule is deliberately narrow: it must not fire on anything already in the + * docs, or on the placeholders docs legitimately use (`my-org`, `your-repo`). + * Public Mergify repos, `dashboard.mergify.com`, and third-party dashboards the + * reader is meant to visit are all fine and must not match. + */ +export const RULES = [ + { + id: 'support-ticket', + label: 'support ticket or thread ID', + re: /\bT-\d{3,6}\b|\bth_[0-9A-Za-z]{16,}\b|\bapp\.plain\.com\b/g, + }, + { + id: 'internal-source-path', + label: 'private repository or internal source path', + re: /\bmergify_shadow_office\b|\bshadow[-_ ]office\b|\bMergifyio\/monorepo\b|\bmergify_engine\b/gi, + }, + { + id: 'internal-tracker', + label: 'internal tracker or wiki link', + re: /\blinear\.app\/|\.notion\.so\/|\bnotion\.site\//gi, + }, + { + id: 'internal-host', + label: 'internal Mergify hostname', + re: /\b(?:admin|internal|staging|shadow(?:-office)?)\.mergify\.(?:com|io)\b/gi, + }, +]; + +const ALLOW_RE = /internal-leaks:\s*allow\s+([\w-]+)/i; + +/** Rule IDs allowed by a directive on the line above `index` (0-based). */ +function allowedOnLine(lines, index) { + const allowed = new Set(); + for (let p = index - 1; p >= 0; p -= 1) { + if (!lines[p].trim()) continue; + const m = lines[p].match(ALLOW_RE); + if (m) allowed.add(m[1].toLowerCase()); + break; + } + return allowed; +} + +/** Scan text; returns [{line, rule, label, match}, ...]. */ +export function scanText(text, rules = RULES) { + const lines = text.split('\n'); + const findings = []; + lines.forEach((line, i) => { + let allowed = null; + for (const rule of rules) { + rule.re.lastIndex = 0; + const matches = line.match(rule.re); + if (!matches) continue; + allowed ??= allowedOnLine(lines, i); + if (allowed.has(rule.id)) continue; + for (const match of new Set(matches)) { + findings.push({ line: i + 1, rule: rule.id, label: rule.label, match }); + } + } + }); + return findings; +} + +export function scanFile(file, rules = RULES) { + return scanText(fs.readFileSync(file, 'utf8'), rules).map((f) => ({ + ...f, + file: path.relative(ROOT, file), + })); +} + +export function* iterFiles(targets) { + for (const t of targets) { + const abs = path.resolve(ROOT, t); + const stat = fs.statSync(abs); + if (stat.isDirectory()) { + for (const entry of fs.readdirSync(abs, { withFileTypes: true, recursive: true })) { + if (entry.isFile() && SCANNED_EXTENSIONS.some((ext) => entry.name.endsWith(ext))) { + yield path.join(entry.parentPath ?? entry.path, entry.name); + } + } + } else if (SCANNED_EXTENSIONS.some((ext) => abs.endsWith(ext))) { + yield abs; + } + } +} + +function main(argv) { + const jsonMode = argv.includes('--json'); + const targets = argv.filter((a) => a !== '--json'); + if (targets.length === 0) targets.push(...DEFAULT_TARGETS); + + const findings = []; + let scanned = 0; + for (const file of iterFiles(targets)) { + scanned += 1; + findings.push(...scanFile(file)); + } + + if (jsonMode) { + process.stdout.write(`${JSON.stringify(findings, null, 2)}\n`); + return findings.length === 0 ? 0 : 1; + } + + console.log(`Scanned ${scanned} file(s) for internal information.`); + if (findings.length === 0) { + console.log('No internal identifiers found.'); + return 0; + } + console.error(`\n${findings.length} possible leak(s):\n`); + for (const f of findings) { + console.error(` ${f.file}:${f.line} — ${f.label} — ${f.match}`); + } + console.error( + '\nRewrite the line so it states the product behavior without the internal\n' + + 'identifier (see .claude/skills/proofread-leaks). If a match is genuinely\n' + + 'public, allow it on the line above:\n' + + ' {/* internal-leaks: allow — why */}' + ); + return 1; +} + +// Run as a CLI only when invoked directly, so tests can import the helpers. +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + process.exit(main(process.argv.slice(2))); +} diff --git a/scripts/check-internal-leaks.test.mjs b/scripts/check-internal-leaks.test.mjs new file mode 100644 index 0000000000..7ef01d2552 --- /dev/null +++ b/scripts/check-internal-leaks.test.mjs @@ -0,0 +1,76 @@ +import { describe, expect, it } from 'vitest'; +import { iterFiles, scanFile, scanText } from './check-internal-leaks.mjs'; + +const rulesOf = (text) => scanText(text).map((f) => f.rule); + +describe('scanText', () => { + it('catches support ticket and thread identifiers', () => { + expect(rulesOf('See ticket T-1234 for details.')).toEqual(['support-ticket']); + expect(rulesOf('thread th_01JQZK4M8XN2VR7TDY0P3WGA6H')).toEqual(['support-ticket']); + expect(rulesOf('https://app.plain.com/workspace/x')).toEqual(['support-ticket']); + }); + + it('catches private repositories and internal source paths', () => { + expect(rulesOf('mergify_shadow_office/models/billing.py')).toEqual(['internal-source-path']); + expect(rulesOf('cloned from Mergifyio/monorepo')).toEqual(['internal-source-path']); + expect(rulesOf('engine/mergify_engine/rules/config.py')).toEqual(['internal-source-path']); + }); + + it('catches internal trackers and hostnames', () => { + expect(rulesOf('https://linear.app/mergifyio/issue/MRGFY-1')).toEqual(['internal-tracker']); + expect(rulesOf('https://www.notion.so/mergify/runbook')).toEqual(['internal-tracker']); + expect(rulesOf('https://admin.mergify.com/orgs')).toEqual(['internal-host']); + }); + + it('leaves public Mergify and third-party references alone', () => { + expect(rulesOf('https://github.com/Mergifyio/mergify/discussions')).toEqual([]); + expect(rulesOf('the `mergifyio/gha-mergify-ci` action')).toEqual([]); + expect(rulesOf('https://dashboard.mergify.com/ci-insights')).toEqual([]); + expect(rulesOf('https://app.datadoghq.com/integrations?search=mergify')).toEqual([]); + expect(rulesOf('owner: my-org\nrepository: your-repo')).toEqual([]); + // The docs document token prefixes as placeholders; those must not fire. + expect(rulesOf('Use a classic `ghp_*` or fine-grained `github_pat_*` token.')).toEqual([]); + expect(rulesOf('Authorization: Bearer ')).toEqual([]); + }); + + it('reports the line number and the matched text', () => { + const findings = scanText('intro\n\nsee T-1234 now\n'); + expect(findings).toEqual([ + { line: 3, rule: 'support-ticket', label: 'support ticket or thread ID', match: 'T-1234' }, + ]); + }); + + it('honors an allow directive naming the rule on the previous line', () => { + expect(rulesOf('{/* internal-leaks: allow support-ticket — public */}\nT-1234\n')).toEqual([]); + expect(rulesOf('# internal-leaks: allow internal-tracker — public\nlinear.app/x\n')).toEqual( + [] + ); + // The directive is scoped: it only allows the rule it names. + expect(rulesOf('{/* internal-leaks: allow internal-tracker */}\nT-1234\n')).toEqual([ + 'support-ticket', + ]); + // ...and only the next non-blank line. + expect(rulesOf('{/* internal-leaks: allow support-ticket */}\nfine\nT-1234\n')).toEqual([ + 'support-ticket', + ]); + }); +}); + +describe('published docs', () => { + it('contain no internal information', () => { + const findings = []; + let scanned = 0; + for (const file of iterFiles(['src/content/docs'])) { + scanned += 1; + findings.push(...scanFile(file)); + } + if (findings.length) { + const detail = findings + .map((f) => `${f.file}:${f.line} — ${f.label} — ${f.match}`) + .join('\n'); + throw new Error(`Internal information found in published docs:\n${detail}`); + } + // Sanity: the scan actually walked the docs tree. + expect(scanned).toBeGreaterThan(50); + }); +});