feat(agents): stop agents leaking internal information into published docs#12204
Merged
mergify[bot] merged 1 commit intoJul 27, 2026
Conversation
Contributor
Merge Protections🟢 All 6 merge protections satisfied — ready to merge. Show 6 satisfied protections🟢 🤖 Continuous Integration
🟢 👀 Review Requirements
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 Reviews
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
There was a problem hiding this comment.
Pull request overview
Adds guardrails to prevent agents (and humans) from leaking internal identifiers into published docs by combining: (1) explicit repository guidance, (2) an always-on leak-focused proofreader skill, and (3) a deterministic CI/CLI scanner for mechanical leak patterns.
Changes:
- Add a Node-based scanner (
pnpm check:internal-leaks) with unit tests to detect common internal identifiers (tickets/threads, private paths, internal trackers/hosts, credential-shaped strings) plus an allow-directive escape hatch. - Wire the scanner into CI as a dedicated
internal-leaksjob and document the policy + enforcement workflow inAGENTS.md. - Introduce a new
proofread-leaksskill and reference it from doc-authoring / code-grounded doc skills to keep internal provenance out of published pages.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/check-internal-leaks.mjs | Implements the deterministic internal-leak scanner and CLI output. |
| scripts/check-internal-leaks.test.mjs | Adds Vitest coverage for rules, allow-directives, and a repo docs-tree sanity scan. |
| package.json | Adds check:internal-leaks script entry. |
| .github/workflows/ci.yaml | Adds CI job to run the leak scanner on docs. |
| AGENTS.md | Documents the “Never publish internal information” rule and enforcement steps. |
| .claude/skills/proofread-leaks/SKILL.md | Adds the always-on leak-focused proofreading subagent guidance. |
| .claude/skills/document-a-feature/SKILL.md | Requires leak checking in the documentation workflow and points to AGENTS.md. |
| .claude/skills/docs-gap-analysis/SKILL.md | Clarifies that internal IDs/paths must not flow from gap reports into published docs. |
| .claude/skills/docs-from-code/SKILL.md | Adds explicit guardrail to avoid citing private clone identifiers/paths in docs output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jd
commented
Jul 26, 2026
… docs Docs are usually written *from* internal material: a support case, a private PR, an engine source file, a Linear ticket. That material travels with the draft, and nothing in the repo told an agent to leave it behind. A recent billing page was written from a support ticket whose context carried a helpdesk ticket ID, a thread ID, a customer org name, and a private shadow-office source path. None of that belongs on docs.mergify.com, and a leaked identifier cannot be unpublished. This follows the repo's existing agent-guidance structure rather than adding a parallel document: - **AGENTS.md** gains Critical Rule 8 plus a "Never Publish Internal Information" section listing concretely what must never ship (ticket and thread IDs, customer/org/account identifiers, private repo and source paths, internal trackers and admin URLs, credentials, internal framing), and how to fix a leak: state the behavior, drop the provenance. - **`proofread-leaks`** joins the proofread pipeline as a 5th subagent, written to match the existing proofread-* skills. It is the one proofreader that never skips: the pipeline's 10-line threshold and its "code blocks only" / "frontmatter only" exemptions are exactly where a one-line ticket ID hides, so those exemptions no longer apply to it. - **`pnpm check:internal-leaks`** (`scripts/check-internal-leaks.mjs`, plus a CI job) is the deterministic backstop, modelled on validate-config-examples: fixed patterns for ticket/thread IDs, private repo and source paths, and internal trackers and hostnames, with a per-line `internal-leaks: allow <rule>` escape hatch. - **docs-from-code**, **docs-gap-analysis**, and **document-a-feature** each gain a pointer, since those are the skills that pull internal material in. Every rule is narrow enough to leave today's docs clean (146 files scanned, 0 findings) and to ignore the placeholders the docs already use, so `ghp_*`, `<your-token>`, `Mergifyio/mergify`, and `app.datadoghq.com` do not fire. Credentials stay in the policy and in proofread-leaks' checklist, but the deterministic scan does not look for them: secret detection is not this script's job to reinvent. Judgment call worth a look: the ask was for instructions, and the CI scanner goes beyond that. I added it because the skill-level check only fires when an agent runs the pipeline, which leaves human edits and skipped passes uncovered, and because this is the one class of mistake that is unfixable after the fact. Happy to drop the script and keep the instructions if you would rather not carry the CI surface. The scanner cannot catch customer names, copied examples, or internal framing. That half stays with proofread-leaks, and the docs say so in both places so a green CI run is not mistaken for a clean bill of health. Change-Id: I56a581e7825816dbdc7e7f9be082f39d799cd72f
Member
Author
Revision history
|
jd
force-pushed
the
devs/jd/jd/llm-no-internal-leak/stop-agents-leaking-internal-info-published-docs--56a581e7
branch
from
July 26, 2026 11:22
16507cd to
519cdbc
Compare
jd
marked this pull request as ready for review
July 26, 2026 13:42
JulianMaurin
approved these changes
Jul 27, 2026
kozlek
approved these changes
Jul 27, 2026
Contributor
Merge Queue Status
This pull request spent 2 minutes 19 seconds in the queue, including 1 minute 46 seconds running CI. Required conditions to merge
|
53 tasks
mergify
Bot
deleted the
devs/jd/jd/llm-no-internal-leak/stop-agents-leaking-internal-info-published-docs--56a581e7
branch
July 27, 2026 07:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs are usually written from internal material: a support case, a private
PR, an engine source file, a Linear ticket. That material travels with the
draft, and nothing in the repo told an agent to leave it behind. A recent
billing page was written from a support ticket whose context carried a
helpdesk ticket ID, a thread ID, a customer org name, and a private
shadow-office source path. None of that belongs on docs.mergify.com, and a
leaked identifier cannot be unpublished.
This follows the repo's existing agent-guidance structure rather than adding
a parallel document:
AGENTS.md gains Critical Rule 8 plus a "Never Publish Internal
Information" section listing concretely what must never ship (ticket and
thread IDs, customer/org/account identifiers, private repo and source
paths, internal trackers and admin URLs, credentials, internal framing),
and how to fix a leak: state the behavior, drop the provenance.
proofread-leaksjoins the proofread pipeline as a 5th subagent,written to match the existing proofread-* skills. It is the one
proofreader that never skips: the pipeline's 10-line threshold and its
"code blocks only" / "frontmatter only" exemptions are exactly where a
one-line ticket ID hides, so those exemptions no longer apply to it.
pnpm check:internal-leaks(scripts/check-internal-leaks.mjs, plus aCI job) is the deterministic backstop, modelled on
validate-config-examples: fixed patterns for ticket/thread IDs, private
repo and source paths, and internal trackers and hostnames, with a
per-line
internal-leaks: allow <rule>escape hatch.docs-from-code, docs-gap-analysis, and document-a-feature each
gain a pointer, since those are the skills that pull internal material in.
Every rule is narrow enough to leave today's docs clean (146 files scanned,
0 findings) and to ignore the placeholders the docs already use, so
ghp_*,<your-token>,Mergifyio/mergify, andapp.datadoghq.comdo not fire.Credentials stay in the policy and in proofread-leaks' checklist, but the
deterministic scan does not look for them: secret detection is not this
script's job to reinvent.
Judgment call worth a look: the ask was for instructions, and the CI scanner
goes beyond that. I added it because the skill-level check only fires when
an agent runs the pipeline, which leaves human edits and skipped passes
uncovered, and because this is the one class of mistake that is unfixable
after the fact. Happy to drop the script and keep the instructions if you
would rather not carry the CI surface.
The scanner cannot catch customer names, copied examples, or internal
framing. That half stays with proofread-leaks, and the docs say so in both
places so a green CI run is not mistaken for a clean bill of health.