docs(demo): rewrite DEMO.md as a console-driven walkthrough of everything - #37
Merged
Conversation
…hing
The old doc predated roughly half the product. It had no Simulate step, no CVE
/ manifest / OpenAPI inputs, no declarative-WAF emit, no BIG-IP lab, no audit
sink, no CWE/OWASP mapping, no benchmark and no MCP — and it described two
paths when there are now three meaningfully different levels of proof.
Rewritten as a GUI walkthrough:
A — offline, no cloud and no keys, from the curated dataset
B — live on F5 XC: a real exploit blocked at a real edge
C — the declarative WAF: the same policy emitted for BIG-IP AS3 and NGINX
App Protect, answering "we are not an F5 shop"
Each says up front what it needs and what it proves, with a note to pick one
and finish it rather than run half of each.
Adds a "when something declines mid-demo" table. Every decline in this tool is
a deliberate answer — no band-aid fits, MITRE prohibits the obvious CWE, the
replay could not be measured, the cure has not merged — and in front of a
security audience those are the most credible moments available. The table
gives the line to say for each.
FIXTURE FIX, found while verifying the doc rather than writing it from memory:
the curated summary has always claimed "9 candidates, 6 verified, 2 refuted, 1
dropped" while shipping only the 6 verified findings. So the report could not
render its "Candidates the verify agent did not confirm" section at all — the
demo advertised a funnel it was structurally unable to show, and the dataset
contradicted its own metrics.
demo/build_demo_out.py now ships the three non-verified candidates in
findings.json but NOT in triage.json, which is exactly how a real run
represents them. They are realistic false positives of the kind verify actually
catches: an eval() sink that is not reachable, a "hardcoded password" that is a
test fixture, and a missing rate limit that already exists upstream. The report
now renders "3 of 9 candidates", the severity and OWASP charts still count 6,
and the console's Review table still shows 6 rows.
Verified by running Demo A end to end in a browser, not from memory: the
console renders the CWE badges (`CWE-89 · mapped`), the chip row reads
9/6/5/6, the OWASP chart carries its `(no category)` bar, and the refuted
section appears below the fold.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The demo you asked for: everything driven from the GUI, covering old and new functionality.
Why a rewrite rather than an edit
The old doc predated roughly half the product. It had no Simulate step, no CVE / manifest / OpenAPI inputs, no declarative-WAF emit, no BIG-IP lab, no audit sink, no CWE/OWASP mapping, no benchmark and no MCP. It also described two paths when there are now three meaningfully different levels of proof.
Three demos, not two
Each states up front what it needs and what it proves, with the instruction to pick one and finish it — running half of each is worse than running one whole.
Demo C exists because the objection you will actually hear is "we are not an F5 shop", and the answer — same finding, same evidence, three vendors' syntax — was not in the doc at all.
The decline table
Every decline in this tool is a deliberate answer: no band-aid fits, MITRE prohibits the obvious CWE, the replay could not be measured, the cure has not merged. In front of a security audience those are the most credible moments available, and the doc now gives the line to say for each rather than leaving you to improvise:
A fixture bug, found by running the doc instead of writing it from memory
The curated summary has always claimed "9 candidates, 6 verified, 2 refuted, 1 dropped" while shipping only the 6 verified findings in
findings.json. So the report could not render its "Candidates the verify agent did not confirm" section at all — the demo advertised a funnel it was structurally unable to show, and the dataset contradicted its own metrics.demo/build_demo_out.pynow ships the three non-verified candidates infindings.jsonbut not intriage.json, which is exactly how a real run represents them. They are realistic false positives of the kind verify actually catches:eval()sink that is not reachableAfter the change: the report renders "3 of 9 candidates", the severity and OWASP charts still count 6, and the console's Review table still shows 6 rows.
Verified, not asserted
I ran Demo A end to end in a browser and checked each claim before writing it down:
CWE-89 · mappedunder the sqli rowcandidates: 9 · verified: 6 · band-aids: 5 · code-fix PRs: 6(no category)barBIGIP_URLreads (set in environment), the three-state fix from fix(secrets): credentials reaching surfaces they must never reach #34Two claims I had written from memory were wrong and got corrected: the chip row does not include a confirm-rate chip on this dataset, and the refuted section lives in the HTML report rather than the console table. Both now say what the tool actually does.
Suite unchanged at 1085 passed, ruff clean.
🤖 Generated with Claude Code