Skip to content

docs(demo): rewrite DEMO.md as a console-driven walkthrough of everything - #37

Merged
henleda merged 1 commit into
mainfrom
docs-gui-demo
Aug 8, 2026
Merged

docs(demo): rewrite DEMO.md as a console-driven walkthrough of everything#37
henleda merged 1 commit into
mainfrom
docs-gui-demo

Conversation

@henleda

@henleda henleda commented Aug 5, 2026

Copy link
Copy Markdown
Owner

The demo you asked for: everything driven from the GUI, covering old and new functionality.

Why a rewrite rather than an edit

The old doc predated roughly half the product. It had no Simulate step, no CVE / manifest / OpenAPI inputs, no declarative-WAF emit, no BIG-IP lab, no audit sink, no CWE/OWASP mapping, no benchmark and no MCP. It also described two paths when there are now three meaningfully different levels of proof.

Three demos, not two

needs proves wall clock
A — Offline nothing the whole arc, from a curated dataset 5 min
B — Live on F5 XC XC tenant + a model key a real exploit blocked at a real edge 12 min
C — Declarative WAF B, plus a BIG-IP the same policy emitted for someone else's WAF +6 min

Each states up front what it needs and what it proves, with the instruction to pick one and finish it — running half of each is worse than running one whole.

Demo C exists because the objection you will actually hear is "we are not an F5 shop", and the answer — same finding, same evidence, three vendors' syntax — was not in the doc at all.

The decline table

Every decline in this tool is a deliberate answer: no band-aid fits, MITRE prohibits the obvious CWE, the replay could not be measured, the cure has not merged. In front of a security audience those are the most credible moments available, and the doc now gives the line to say for each rather than leaving you to improvise:

You see Say
a finding with no CWE "MITRE prohibits the obvious mapping here; a blank we can explain beats a guess"
"not measured" in the blast radius "that replay could not be measured — it will not pretend it came back clean"
a refused retire "the cure has not merged; the band-aid is the only thing holding this shut"

A fixture bug, found by running the doc instead of writing it from memory

The curated summary has always claimed "9 candidates, 6 verified, 2 refuted, 1 dropped" while shipping only the 6 verified findings in findings.json. So the report could not render its "Candidates the verify agent did not confirm" section at all — the demo advertised a funnel it was structurally unable to show, and the dataset contradicted its own metrics.

demo/build_demo_out.py now ships the three non-verified candidates in findings.json but not in triage.json, which is exactly how a real run represents them. They are realistic false positives of the kind verify actually catches:

  • an eval() sink that is not reachable
  • a "hardcoded password" that is a test fixture
  • a missing rate limit that already exists upstream

After the change: the report renders "3 of 9 candidates", the severity and OWASP charts still count 6, and the console's Review table still shows 6 rows.

Verified, not asserted

I ran Demo A end to end in a browser and checked each claim before writing it down:

  • the console renders the CWE badges — CWE-89 · mapped under the sqli row
  • the chip row reads candidates: 9 · verified: 6 · band-aids: 5 · code-fix PRs: 6
  • the OWASP chart carries its (no category) bar
  • the refuted section appears below the fold
  • BIGIP_URL reads (set in environment), the three-state fix from fix(secrets): credentials reaching surfaces they must never reach #34

Two claims I had written from memory were wrong and got corrected: the chip row does not include a confirm-rate chip on this dataset, and the refuted section lives in the HTML report rather than the console table. Both now say what the tool actually does.

Suite unchanged at 1085 passed, ruff clean.

🤖 Generated with Claude Code

…hing

The old doc predated roughly half the product. It had no Simulate step, no CVE
/ manifest / OpenAPI inputs, no declarative-WAF emit, no BIG-IP lab, no audit
sink, no CWE/OWASP mapping, no benchmark and no MCP — and it described two
paths when there are now three meaningfully different levels of proof.

Rewritten as a GUI walkthrough:

  A — offline, no cloud and no keys, from the curated dataset
  B — live on F5 XC: a real exploit blocked at a real edge
  C — the declarative WAF: the same policy emitted for BIG-IP AS3 and NGINX
      App Protect, answering "we are not an F5 shop"

Each says up front what it needs and what it proves, with a note to pick one
and finish it rather than run half of each.

Adds a "when something declines mid-demo" table. Every decline in this tool is
a deliberate answer — no band-aid fits, MITRE prohibits the obvious CWE, the
replay could not be measured, the cure has not merged — and in front of a
security audience those are the most credible moments available. The table
gives the line to say for each.

FIXTURE FIX, found while verifying the doc rather than writing it from memory:
the curated summary has always claimed "9 candidates, 6 verified, 2 refuted, 1
dropped" while shipping only the 6 verified findings. So the report could not
render its "Candidates the verify agent did not confirm" section at all — the
demo advertised a funnel it was structurally unable to show, and the dataset
contradicted its own metrics.

demo/build_demo_out.py now ships the three non-verified candidates in
findings.json but NOT in triage.json, which is exactly how a real run
represents them. They are realistic false positives of the kind verify actually
catches: an eval() sink that is not reachable, a "hardcoded password" that is a
test fixture, and a missing rate limit that already exists upstream. The report
now renders "3 of 9 candidates", the severity and OWASP charts still count 6,
and the console's Review table still shows 6 rows.

Verified by running Demo A end to end in a browser, not from memory: the
console renders the CWE badges (`CWE-89 · mapped`), the chip row reads
9/6/5/6, the OWASP chart carries its `(no category)` bar, and the refuted
section appears below the fold.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@henleda
henleda merged commit 4948cca into main Aug 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant