Secure Bench

Reproducible, evidence-aware benchmarks for static security analyzers.

Important

Secure Bench is an experimental research harness. Its synthetic measurements are not scanner rankings, production-readiness assessments, or complete-coverage claims.

Secure Bench is an independent, local-first Rust benchmark harness. Phase 0 established versioned contracts, scoring-blind adapters, deterministic matching, separate metrics, provenance, and explicit failure accounting using committed mock reports. Phase 1 adds an original JavaScript and TypeScript corpus and a direct black-box runner for an explicitly supplied Secure Engine binary. Phase 1.5 freezes a public neutral taxonomy for prospective reports without changing the historical Phase 1 result. Phase 2 applies that contract prospectively to two isolated, retained Secure Engine 0.1.1 runs while preserving every historical artifact. Phase 3 freezes a separate 56-case holdout examination. Phase 4 provides its one-shot, network-isolated Secure Engine 0.1.2 evaluation. Phase 5 independently freezes a new 112-case orthogonal holdout and tool-neutral evidence contract v2 without executing a scanner. Phase 6 retires the executed Phase 3 corpus and publishes an additive diagnostic package without changing the official Phase 4 result. Phase 7 records the one-shot, per-process network-isolated Secure Engine 0.1.3 evaluation of the frozen Phase 5 holdout, including every nonzero exit and retained report without reruns or result repair. Phase 8 corrects the exit-code adjudication protocol retrospectively from those immutable reports, preserves the original result, and introduces a prospective tool-neutral process-status policy without another scanner execution. Phase 9 freezes a new 224-case counterbalanced orthogonal holdout v3. Phase 10 records its one-shot Secure Engine 0.1.4 evaluation. Phase 11 retires and discloses that corpus, reconstructs an exhaustive offline postmortem from retained evidence, and leaves the official Phase 10 score unchanged. Phase 12 repairs prospective authoring and adapter methodology, and Phase 12.1 establishes stable content-addressed verification for the frozen Phase 0–11 boundary. Phase 13 freezes a new, intentionally unexecuted 112-case holdout v4 without creating a scanner result.

Phases 14–26 preserve and extend that history with execution, failure analysis, sandbox remediation, normalized OpenGrep and Semgrep recovery studies, and independent evidence certification. Phases 27–30 add a new 112-case multi-scanner corpus, frozen scanner bindings, one terminal-failed campaign, a post-open recovery study, and a final scanner-free certification of Secure Engine exit-status semantics. Every exception remains visible in the committed evidence; no failed or unavailable observation is converted to zero or silently rerun.

This remains an intentionally neutral foundation. It is not a production benchmark, scanner comparison, public leaderboard, or basis for claiming that Secure Engine—or any other analyzer—is superior. A Phase 1 baseline measures one explicitly identified binary on a small synthetic corpus and must be reported with its raw artifacts, denominators, environment, and limitations.

Latest certified study

Phase 30 certifies all 336 unique scanner/case combinations derived from Phases 28 and 29, with zero scanner retries. Phase 28 stopped after one consumed scanner attempt because a sealed helper was missing; Phase 29 recovered that retained raw report and executed the remaining 335 combinations once. The result is therefore a post-open recovery study, not a pristine one-shot independent comparison.

Scanner / lane	TP	FP	TN	FN	Precision	Recall	F1	Balanced accuracy
Secure Engine 0.1.7 RC1 / native	23	9	47	33	0.718750	0.410714	0.522727	0.625000
OpenGrep 1.22.0 / capability-normalized	40	32	24	16	0.555556	0.714286	0.625000	0.571429
Semgrep CE 1.170.0 / capability-normalized	40	32	24	16	0.555556	0.714286	0.625000	0.571429

Native and capability-normalized lanes are deliberately separate. OpenGrep and Semgrep tie exactly within the normalized lane; no cross-lane or overall winner is declared. See the Phase 29 recovery report, Phase 30 certification report, machine-readable comparison, and limitations.

The committed Phase 1 Secure Engine Phase 6 bundle is one deterministic black-box measurement with raw public reports and matching decisions. Its exact-match result is preserved without post-execution vocabulary aliases or tool-specific scoring exceptions. The separate Phase 2 bundle records the prospective evaluation, including raw repeat-run stability and isolation provenance. See Phase 1 baseline reporting and Phase 2 prospective evaluation.

The retired Phase 3 cases are now disclosed for development and regression work and are no longer an unseen holdout. See the Phase 6 postmortem. Post-disclosure results from that corpus are not unbiased benchmark measurements. The separate Phase 7 result and its limitations are documented in the Phase 7 one-shot evaluation. The additive offline correction is documented in the Phase 8 exit-code adjudication.

Phase 9 is now a retired, disclosed diagnostic corpus. The immutable one-shot result is documented in the Phase 10 evaluation, and the additive source-and-report analysis is documented in the Phase 11 postmortem. Post-disclosure runs are development regressions, not unbiased holdout measurements.

Secure Bench never downloads, installs, builds, updates, or discovers scanners. Secure Engine receives no internal API access, private fixtures, expected answers, matcher data, or product-specific scoring treatment. No other scanner is installed or executed in the Phase 1 or Phase 2 measurements.

Workspace

secure-bench/
|- apps/secure-bench-cli/       Corpus, runner, evaluation, and summary commands
|- crates/secure-bench-core/    Contracts, adapters, matching, scoring, and runner
|- docs/                        Architecture, methodology, provenance, and verification
|- fixtures/corpus/             Original scanner-visible JavaScript/TypeScript projects
|- fixtures/reports/            Committed Phase 0 mock reports
|- schemas/                     Versioned JSON Schemas
|- taxonomy/                    Frozen prospective neutral taxonomy data
|- prospective/                 Prospective scanner-neutral examinations
|- GOAL.md                      Preserved Phase 0 goal
`- PLAN.md                      Product plan and boundaries

All benchmark executables are Rust. Corpus source files use JavaScript, JSX, TypeScript, and TSX because those are the languages under evaluation; the harness copies but never loads or executes fixture code.

Phase 1 commands

Validate corpus schemas, semantic pairing, answer-leakage controls, provenance, and fingerprints:

cargo run --bin secure-bench -- corpus validate fixtures/corpus-v1.toml

Run an explicitly supplied binary. The default public argument contract is scan {fixture} --format secure-json-v1 --output {report}; repeat --argument to provide an audited alternative template.

cargo run --bin secure-bench -- run fixtures/corpus-v1.toml \
  --tool secure-engine \
  --binary /explicit/path/to/secure \
  --output artifacts/phase-1-run \
  --run-id phase-1-baseline

An optional --configuration file is copied outside the scanned project, fingerprinted, and exposed only through an explicit {configuration} argument placeholder. Configuration text containing matcher-owned categories, invariants, or expectation identifiers is rejected.

Evaluate the retained raw reports and render a summary:

cargo run --bin secure-bench -- evaluate \
  fixtures/corpus-v1.toml artifacts/phase-1-run \
  --output artifacts/phase-1-result.json

cargo run --bin secure-bench -- summary artifacts/phase-1-result.json

Machine-readable JSON is written only to the requested file or standard output. Human diagnostics and summaries use standard error during evaluation when JSON is emitted.

Frozen neutral taxonomy

Taxonomy 1.0.0 defines seven stable category/invariant pairs derived from public security invariants and official MITRE CWE records. It was designed without consulting scanner output and applies only prospectively. Missing, incomplete, unknown, version-mismatched, or conflicting report metadata remains explicitly unmapped; there are no rule aliases, prose fallbacks, or scanner-specific exceptions.

cargo run --bin secure-bench -- taxonomy validate taxonomy/secure-bench-taxonomy-v1.json
cargo run --bin secure-bench -- taxonomy inspect taxonomy/secure-bench-taxonomy-v1.json

See Frozen neutral taxonomy v1. The existing Phase 0 and Phase 1 evaluators remain unchanged in behavior, and the committed Phase 1 baseline is neither recalculated nor retrospectively remapped.

Preserved Phase 0 commands

The original option-based forms remain supported and continue to evaluate only committed recorded reports:

cargo run --bin secure-bench -- evaluate \
  --suite fixtures/suite.toml \
  --run fixtures/reports/runs/native-success.json \
  --output artifacts/native-result.json

cargo run --bin secure-bench -- validate \
  --suite fixtures/suite.toml \
  --run fixtures/reports/runs/native-success.json

Phase 2 prospective evaluation

Phase 2 adds strict taxonomy-profile, network-attestation, and prospective-result contracts. The evaluation command consumes retained run bundles only; it does not start a scanner:

cargo run --release --bin secure-bench -- phase2 evaluate \
  --suite fixtures/corpus-v1.toml \
  --taxonomy taxonomy/secure-bench-taxonomy-v1.json \
  --profile taxonomy/phase-1-corpus-taxonomy-profile-v1.json \
  --network-attestation artifacts/network-isolation.json \
  --phase1-result baselines/phase-1-secure-engine-phase6/result.json \
  --primary-run artifacts/primary \
  --repeat-run artifacts/repeat \
  --binary-sha256 <verified-sha256> \
  --source-rpm-sha256 <recorded-sha256> \
  --output artifacts/phase-2-result.json

Verification

Validate Phase 5 commitments and synthetic evidence-contract tests without executing a scanner:

cargo run --bin secure-bench -- phase5 validate
cargo run --bin secure-bench -- phase5 contract-test

Validate or summarize the independently frozen Phase 9 holdout without executing a scanner:

cargo run --locked --manifest-path phase9/Cargo.toml -- validate .
cargo run --locked --manifest-path phase9/Cargo.toml -- summary .

Validate or summarize the intentionally unexecuted Phase 13 holdout v4:

cargo run --locked --manifest-path prospective/phase-13-holdout-v4/Cargo.toml -- validate .
cargo run --locked --manifest-path prospective/phase-13-holdout-v4/Cargo.toml -- summary .

Verify the retired Phase 11 diagnostic package by deterministic reconstruction without executing a scanner:

cargo run --locked --manifest-path phase11/Cargo.toml -- verify .
cargo run --locked --manifest-path phase11/Cargo.toml -- summary .

Validate or inspect the Phase 3 commitments without exposing per-case answers in terminal output:

cargo run --bin secure-bench -- holdout validate \
  --manifest holdout/phase-3/manifest.json \
  --taxonomy taxonomy/secure-bench-taxonomy-v1.json \
  --ledger holdout/phase-3/execution-ledger.jsonl

Verify the sealed Phase 21 sandbox qualification without starting a scanner or opening holdout inputs:

cargo run --offline --manifest-path phase21/Cargo.toml \
  --bin independent-verify -- .

Verify the sealed Phase 22 post-open recovery study without starting a scanner:

cargo run --offline --manifest-path phase22/Cargo.toml \
  --bin independent-verify -- .

Verify the sealed Phase 23 synthetic Semgrep crash diagnosis and qualification without starting a scanner or opening holdout inputs:

cargo run --offline --locked --manifest-path phase23/Cargo.toml \
  --bin independent-verify -- .

Verify the additive Phase 26 certification of the frozen Phase 25 evidence:

cp .github/allowed_signers "$(git rev-parse --git-common-dir)/allowed_signers"
python3 phase26/reproducer/independent_verify.py verify .

Verify the Phase 29 recovery evidence and Phase 30 exit-status certification without starting a scanner:

python3 phase29/harness.py verify
cargo test --offline --locked --manifest-path phase30/Cargo.toml -- \
  --skip frozen_schema_rejects_malformed_raw
sha256sum --check phase30/SHA256SUMS

The skipped Phase 30 test requires the separately preserved Secure Engine RC schema at its frozen absolute provenance path. Its exact expected hash remains recorded in Phase 30; the self-contained tests and exhaustive Phase 30 checksum inventory run in public CI.

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features
cargo audit --deny warnings
cargo deny check

See Architecture, Methodology, Contracts, Frozen taxonomy, Corpus provenance, Runner boundaries, Phase 2 evaluation, Phase 3 holdout, Phase 4 evaluation, Phase 5 orthogonal holdout, Phase 6 postmortem, Phase 7 evaluation, Phase 8 adjudication, Phase 9 holdout, Phase 10 evaluation, Phase 11 postmortem, Phase 12 methodology, Phase 12.1 historical verification, Phase 13 holdout v4, Phase 21 sandbox remediation, Phase 22 post-open recovery, Phase 23 Semgrep crash remediation, Phase 26 certification, Phase 29 recovery, Phase 30 certification, and Verification.

License

Secure Bench is licensed under the Apache License 2.0. The original synthetic fixtures use the same license and carry explicit authorship, origin, revision, and modification records in their suite manifests.

See CONTRIBUTING.md before proposing changes and SECURITY.md for private vulnerability reporting.