Add adversarial-hardening skill (red-team/chaos orchestrator)#1945
Open
danielmeppiel wants to merge 4 commits into
Open
Add adversarial-hardening skill (red-team/chaos orchestrator)#1945danielmeppiel wants to merge 4 commits into
danielmeppiel wants to merge 4 commits into
Conversation
Add a new user-facing orchestrator skill package at packages/adversarial-hardening/ that hardens ONE bounded surface by looping red-team + chaos lenses to a fixpoint, gating every finding against an operator-ratified scope charter, folding accepted findings via shepherd-driver, then reviewing via apm-review-panel and authoring the PR via pr-description-skill. Two deterministic grounding gates ship with it: - Pillar A (scripts/render-dossier.py): the PR findings section is rendered from the persisted ledger, never recalled. - Pillar B (scripts/check-push-hygiene.sh): uncollected tests, orphan fixtures, and wrong-altitude folds are rejected, never shipped. Declares BOTH-mechanism deps on the three siblings (apm.yml local-path edges + use-site probes). Registered in root apm.yml; deployed to .agents/skills/ and lockfile regenerated (apm audit --ci green). Evals: 20 trigger items (val split 1.0/1.0) + 3 content scenarios covering both pillars (delta_anchors >= 3 each). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new user-facing orchestrator skill package, adversarial-hardening, that formalizes a bounded red-team + chaos hardening loop (scope-charter gated) and grounds outputs via a persisted findings ledger plus deterministic dossier render/hygiene gates. The PR also wires the package into the root dependency graph and deploys it into .agents/skills/ with an updated lockfile.
Changes:
- Introduces
packages/adversarial-hardening/with orchestration guidance (SKILL.md), scope-charter template, lens catalogue, findings-ledger schema, and deterministic helper scripts. - Adds a stdlib-only eval harness (
run_evals.py) plus trigger/content fixtures to regression-test dispatcher triggers and grounded-output behaviors. - Registers the new skill in root
apm.ymland regeneratesapm.lock.yamlto deploy it into.agents/skills/adversarial-hardening/.
Show a summary per file
| File | Description |
|---|---|
| packages/adversarial-hardening/SKILL.md | Defines the orchestrator contract and step-by-step hardening flow. |
| packages/adversarial-hardening/scripts/check-push-hygiene.sh | Implements Pillar B hygiene gate for hardening folds. |
| packages/adversarial-hardening/scripts/render-dossier.py | Implements Pillar A deterministic rendering of findings into PR-ready markdown. |
| packages/adversarial-hardening/scripts/run_evals.py | Runs trigger/content evals to validate dispatcher intent matching and output grounding. |
| packages/adversarial-hardening/apm.yml | Declares the skill manifest and direct dependencies on sibling skills. |
| packages/adversarial-hardening/assets/scope-charter.template.md | Provides the operator-ratified scope charter template used to prevent scope creep. |
| packages/adversarial-hardening/references/lens-catalogue.md | Catalogues red-team and chaos lenses and the expected JSON return shape. |
| packages/adversarial-hardening/references/findings-ledger.md | Documents the findings-ledger JSON schema and fixpoint stop predicate. |
| packages/adversarial-hardening/evals/evals.json | Top-level eval manifest: gates and keyword lists for trigger/content evals. |
| packages/adversarial-hardening/evals/triggers.json | Trigger eval dataset (fire/no-fire) for the skill description matching. |
| packages/adversarial-hardening/evals/README.md | Documents eval intent, layout, and matcher behavior. |
| packages/adversarial-hardening/evals/content/out-of-scope-decline.json | Content eval scenario validating explicit OOS-1 declines. |
| packages/adversarial-hardening/evals/content/pillar-a-grounded-findings.json | Content eval scenario validating grounded findings table presence. |
| packages/adversarial-hardening/evals/content/pillar-b-push-hygiene.json | Content eval scenario validating hygiene gate rejection behavior. |
| packages/adversarial-hardening/evals/fixtures/out-of-scope-decline__with_skill.md | "With skill" fixture for OOS decline scenario. |
| packages/adversarial-hardening/evals/fixtures/out-of-scope-decline__without_skill.md | "Without skill" fixture for OOS decline scenario. |
| packages/adversarial-hardening/evals/fixtures/pillar-a-grounded-findings__with_skill.md | "With skill" fixture for Pillar A grounding scenario. |
| packages/adversarial-hardening/evals/fixtures/pillar-a-grounded-findings__without_skill.md | "Without skill" fixture for Pillar A grounding scenario. |
| packages/adversarial-hardening/evals/fixtures/pillar-b-push-hygiene__with_skill.md | "With skill" fixture for Pillar B hygiene scenario. |
| packages/adversarial-hardening/evals/fixtures/pillar-b-push-hygiene__without_skill.md | "Without skill" fixture for Pillar B hygiene scenario. |
| packages/adversarial-hardening/evals/.gitignore | Ignores generated eval result JSON files. |
| packages/adversarial-hardening/evals/results/.gitkeep | Keeps the eval results directory tracked. |
| .agents/skills/adversarial-hardening/SKILL.md | Deployed copy of the orchestrator skill guidance. |
| .agents/skills/adversarial-hardening/scripts/check-push-hygiene.sh | Deployed copy of the Pillar B hygiene gate. |
| .agents/skills/adversarial-hardening/scripts/render-dossier.py | Deployed copy of the Pillar A dossier renderer. |
| .agents/skills/adversarial-hardening/scripts/run_evals.py | Deployed copy of the eval runner. |
| .agents/skills/adversarial-hardening/apm.yml | Deployed copy of the skill manifest. |
| .agents/skills/adversarial-hardening/assets/scope-charter.template.md | Deployed copy of the scope-charter template. |
| .agents/skills/adversarial-hardening/references/lens-catalogue.md | Deployed copy of the lens catalogue reference. |
| .agents/skills/adversarial-hardening/references/findings-ledger.md | Deployed copy of the findings-ledger reference. |
| .agents/skills/adversarial-hardening/evals/evals.json | Deployed copy of the eval manifest. |
| .agents/skills/adversarial-hardening/evals/triggers.json | Deployed copy of the trigger eval dataset. |
| .agents/skills/adversarial-hardening/evals/README.md | Deployed copy of the eval documentation. |
| .agents/skills/adversarial-hardening/evals/content/out-of-scope-decline.json | Deployed copy of the OOS decline scenario. |
| .agents/skills/adversarial-hardening/evals/content/pillar-a-grounded-findings.json | Deployed copy of the Pillar A scenario. |
| .agents/skills/adversarial-hardening/evals/content/pillar-b-push-hygiene.json | Deployed copy of the Pillar B scenario. |
| .agents/skills/adversarial-hardening/evals/fixtures/out-of-scope-decline__with_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/fixtures/out-of-scope-decline__without_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/fixtures/pillar-a-grounded-findings__with_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/fixtures/pillar-a-grounded-findings__without_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/fixtures/pillar-b-push-hygiene__with_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/fixtures/pillar-b-push-hygiene__without_skill.md | Deployed copy of the fixture. |
| .agents/skills/adversarial-hardening/evals/.gitignore | Deployed copy of the eval .gitignore. |
| .agents/skills/adversarial-hardening/evals/results/.gitkeep | Deployed copy of the .gitkeep sentinel. |
| apm.yml | Registers ./packages/adversarial-hardening as a user-facing orchestrator dependency. |
| apm.lock.yaml | Regenerated lockfile to include and deploy the new skill package. |
Review details
- Files reviewed: 44/46 changed files
- Comments generated: 8
- Review effort level: Low
Comment on lines
+78
to
+90
| # -------- arg parse --------------------------------------------------- | ||
| while [ $# -gt 0 ]; do | ||
| case "$1" in | ||
| --base) BASE_REF="${2:?--base needs a value}"; shift 2 ;; | ||
| --test-root) TEST_ROOT="${2:?--test-root needs a value}"; shift 2 ;; | ||
| --src-root) SRC_ROOT="${2:?--src-root needs a value}"; shift 2 ;; | ||
| --collect-cmd) COLLECT_CMD="${2:?--collect-cmd needs a value}"; shift 2 ;; | ||
| --integration-re) INTEGRATION_RE="${2:?--integration-re needs a value}"; shift 2 ;; | ||
| --help|-h) usage; exit 0 ;; | ||
| *) log "[x] unknown argument: $1"; usage; exit 2 ;; | ||
| esac | ||
| done | ||
|
|
Comment on lines
+47
to
+64
| USAGE: | ||
| check-push-hygiene.sh [--base REF] [--test-root DIR] [--src-root DIR] | ||
| [--collect-cmd "CMD"] [--integration-re REGEX] | ||
| [--help] | ||
|
|
||
| OPTIONS: | ||
| --base REF Diff base to compute the fold's changed files | ||
| (default: origin/main). The diff is REF...HEAD. | ||
| --test-root DIR Root under which tests live (default: tests). | ||
| --src-root DIR Root under which source modules live (default: src). | ||
| --collect-cmd "CMD" Command that lists collected test nodeids the way | ||
| the MERGE-QUEUE lane does. Must emit nodeids on | ||
| stdout. Default: "uv run --extra dev pytest | ||
| --collect-only -q". Pass the merge_group lane's | ||
| exact command when it differs. | ||
| --integration-re RE Extended-regex marking a path as an integration | ||
| test (default covers integration/ and it_ names). | ||
| --help Print this help and exit 0. |
Comment on lines
+78
to
+90
| # -------- arg parse --------------------------------------------------- | ||
| while [ $# -gt 0 ]; do | ||
| case "$1" in | ||
| --base) BASE_REF="${2:?--base needs a value}"; shift 2 ;; | ||
| --test-root) TEST_ROOT="${2:?--test-root needs a value}"; shift 2 ;; | ||
| --src-root) SRC_ROOT="${2:?--src-root needs a value}"; shift 2 ;; | ||
| --collect-cmd) COLLECT_CMD="${2:?--collect-cmd needs a value}"; shift 2 ;; | ||
| --integration-re) INTEGRATION_RE="${2:?--integration-re needs a value}"; shift 2 ;; | ||
| --help|-h) usage; exit 0 ;; | ||
| *) log "[x] unknown argument: $1"; usage; exit 2 ;; | ||
| esac | ||
| done | ||
|
|
Comment on lines
+47
to
+64
| USAGE: | ||
| check-push-hygiene.sh [--base REF] [--test-root DIR] [--src-root DIR] | ||
| [--collect-cmd "CMD"] [--integration-re REGEX] | ||
| [--help] | ||
|
|
||
| OPTIONS: | ||
| --base REF Diff base to compute the fold's changed files | ||
| (default: origin/main). The diff is REF...HEAD. | ||
| --test-root DIR Root under which tests live (default: tests). | ||
| --src-root DIR Root under which source modules live (default: src). | ||
| --collect-cmd "CMD" Command that lists collected test nodeids the way | ||
| the MERGE-QUEUE lane does. Must emit nodeids on | ||
| stdout. Default: "uv run --extra dev pytest | ||
| --collect-only -q". Pass the merge_group lane's | ||
| exact command when it differs. | ||
| --integration-re RE Extended-regex marking a path as an integration | ||
| test (default covers integration/ and it_ names). | ||
| --help Print this help and exit 0. |
Comment on lines
+7
to
+11
| * TRIGGER EVALS scoring the SKILL.md `description:` against | ||
| should-fire and should-not-fire queries via a deterministic | ||
| keyword/bigram matcher. The matcher is documented in the | ||
| README; it approximates dispatcher behavior without requiring | ||
| a live LLM. |
Comment on lines
+7
to
+11
| * TRIGGER EVALS scoring the SKILL.md `description:` against | ||
| should-fire and should-not-fire queries via a deterministic | ||
| keyword/bigram matcher. The matcher is documented in the | ||
| README; it approximates dispatcher behavior without requiring | ||
| a live LLM. |
Comment on lines
+197
to
+200
| fixtures = scenario["fixtures"] | ||
| with_path = (scenario_path.parent / fixtures["with_skill"]).resolve() | ||
| without_path = (scenario_path.parent / fixtures["without_skill"]).resolve() | ||
|
|
Comment on lines
+197
to
+200
| fixtures = scenario["fixtures"] | ||
| with_path = (scenario_path.parent / fixtures["with_skill"]).resolve() | ||
| without_path = (scenario_path.parent / fixtures["without_skill"]).resolve() | ||
|
|
The 'A12 gradient inside each lens' section named genesis design-time
role classes ('researcher class', 'implementer class') as if they were
runtime-bindable. The common child-thread spawn affordance seeds a
child with a persona + task description but does NOT guarantee
per-subagent model selection (that is B12 MODEL ROUTER, a per-harness
affordance). On a common-only module that is a PHANTOM AFFORDANCE: the
runtime reads 'promote to implementer class' with nothing to bind it to.
Rewrite the gradient as a portable SEQUENCING discipline (cheap recon
enumeration first, author probes only on surviving vectors) whose
saving holds on whatever model the child gets. Move the model-downgrade
to an explicitly OPTIONAL per-harness note gated on the harness exposing
per-subagent model selection, with a 'never make correctness depend on
the swap' guard.
apm audit --ci 9/9, evals exit 0, pure ASCII, deployed mirror
byte-identical.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Target harness is confirmed as GitHub Copilot, which exposes model selection ONLY on .agent.md custom-agent profiles -- never on skill frontmatter (per the genesis copilot adapter). Make the routing guidance concrete and grounded instead of a hypothetical 'where the harness supports it' note: - SKILL.md fan-out: lenses spawn via the task tool to a NAMED lens custom agent (the same fan-out apm-review-panel uses for its ../../agents/*.agent.md specialists); a per-lens model, if routed, binds on that agent's .agent.md model: field, never in SKILL.md. - lens-catalogue.md: cross-lens B12 routing binds on each lens's custom-agent profile; a lens with no dedicated profile runs at the session default (correct, just unrouted). This realizes the binding site without inventing a per-lens model table the design packet never specified. apm audit --ci 9/9, evals exit 0, ASCII clean, mirror byte-identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Under the 'no bundled lens agents' decision (mirroring apm-review-panel, which ships none and reuses shared profiles), the prior wording implied a NAMED lens custom agent that does not ship -- a phantom reference, the exact unwired-artifact failure this skill exists to prevent. Reword so the lens is seeded from the catalogue charter (which DOES ship) on the session-default model, with the named-agent + .agent.md model: binding as an OPTIONAL routing upgrade when a dedicated lens agent is registered. apm audit --ci 9/9, evals exit 0, ASCII clean, mirror byte-identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Adds a new user-facing orchestrator skill,
adversarial-hardening, atpackages/adversarial-hardening/. It hardens ONE bounded surface (a PR,a module, a CLI command) by looping a red-team + chaos panel until a
full adversarial sweep finds zero new in-scope defects, gating every
finding against an operator-ratified scope charter, folding accepted
fixes via
shepherd-driver, then reviewing withapm-review-panelandauthoring the PR with
pr-description-skill. Registered in rootapm.yml, deployed to.agents/skills/, andapm audit --ciis green.Problem (WHY)
Ad-hoc "harden this feature" runs drift two ways that this skill exists
to prevent, both observed in a real 32-round hardening run:
secret scanner -- detecting another platform's tokens by shape, which
is the script author's shared responsibility, not the target's job.
red_team/tests accreted thatthe merge-queue lane never collected -- dead weight that "cannot get
merged", plus a PR body that narrated findings from recall rather than
from what was actually found and fixed.
There was no reusable primitive that makes "bounded + grounded" a
structural property of an adversarial hardening loop.
Approach (WHAT)
A flat orchestrator package mirroring
packages/shepherd-driver/. Itcomposes three existing siblings (re-implementing none) and adds two
deterministic grounding gates -- the "two pillars":
rendered from a persisted ledger by
scripts/render-dossier.py, neverrecalled.
pr-description-skillembeds the rendered## Hardening findings and resolutionblock VERBATIM.scripts/check-push-hygiene.shrejectsany fold that ships an uncollected test (CI-COLLECTED), an orphan
fixture (DIFF-MINIMAL), or a cross-module change with only unit
coverage (RIGHT-ALTITUDE). A reject loops back to
shepherd-driver;it never advances the finding.
The anti-scope-creep spine is an operator-ratified scope charter: a
finding the charter does not place IN scope is DECLINED with its clause
id (e.g.
OOS-1= no domain scanner) and logged in the PR, never folded.flowchart LR OP[Operator] -->|ratify charter| CH[scope-charter] CH --> L[red-team + chaos lenses] L --> AR[charter-arbiter: gate + dedup] AR -->|accepted| SD[shepherd-driver fold] SD --> HG{push-hygiene gate} HG -->|reject| SD HG -->|pass| LED[(findings-ledger)] LED -->|fixpoint?| L LED --> RD[render-dossier.py] RD --> PD[pr-description-skill embeds verbatim] PD --> RP[apm-review-panel until ship_now] RP --> OPImplementation (HOW)
SKILL.md-- the orchestrator flow (charter ratify -> alignment loop-> lens fan-out -> charter-gated arbiter -> per-finding fold + hygiene
gate -> dossier render -> verbatim PR embed -> panel review). 219
lines, ~2.5k tokens, 985-char dispatch description (all under the
500-line / 5000-token / 1024-char budgets), pure ASCII.
apm.yml-- BOTH-mechanism deps: direct local-path edges on../shepherd-driver,../apm-review-panel,../pr-description-skillAND a use-site activation probe for each in the SKILL body.
assets/scope-charter.template.md-- IN/OUT scope,OOS-*declineclauses,
INV-*invariants, and the five pillar invariants.references/lens-catalogue.md-- RT-1..RT-5 red-team + CH-1..CH-4chaos archetypes (load-on-demand).
references/findings-ledger.md-- the ledger JSON schema, fingerprint/ dedup rule, fixpoint stop-predicate, and dossier column mapping.
scripts/render-dossier.py(Pillar A) andscripts/check-push-hygiene.sh(Pillar B) -- non-interactive,--helpdocumented, structured stdout (MD / JSON), stderrdiagnostics, non-zero exit on hygiene reject.
evals/-- a stdlib-only harness (scripts/run_evals.py), 20 triggeritems, and 3 content scenarios.
apm.ymladds./packages/adversarial-hardening;apm.lock.yamlregenerated by
apm install. The 3 sibling deps were already in theclosure, so only this package is newly deployed.
Trade-offs
entrypoint, so it is declared at root (like the other two
orchestrators); its siblings stay transitive. No new root-level
redundant edge is introduced.
evals/run_evals.pyadapted frompr-description-skill. Thematcher is intentionally a near-copy; the only generalization is a
manifest-driven
secondary_anchor_tokensset (replacing the hardcodedpr/pullanchor) so the trigger heuristic is domain-correct. CI'spylint R0801only scanssrc/apm_cli/, so the parallel underpackages/does not trip duplication.in-scope findings); the per-target budget is a human escalation, never
a silent stop.
Validation evidence
apm install->[*] Installed 8 APM dependencies;adversarial-hardeningintegrated into.agents/skills/.apm audit --ci-> All 9 check(s) passed (lockfile, refconsistency, deployed-files, no-orphans, subset, config, content
integrity, includes-consent, no drift).
python scripts/run_evals.py-> exit 0. Trigger val split:should-fire 1.0, should-not-fire 1.0. Content deltas:
out-of-scope-decline +4, pillar-a +5, pillar-b +3 anchors (gate is
ruff check+ruff format --checkon both.py-> clean.bash scripts/lint-auth-signals.sh-> clean. Pure ASCII across thepackage. All three scripts respond to
--help.uncollected-test / orphan-fixture / cross-module-no-integration all
reject with exit 1 and structured JSON.
This PR is independent of #1798 (already merged into
main); it touchesno other branch. Do NOT auto-merge -- the protected-branch merge stays a
human gate (charter clause
PILL-X1).How to test
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com