Skip to content

Fix #40/#41: Freeze benchmark scope and audit CnC measurements#48

Merged
isPANN merged 21 commits into
mainfrom
issue-40-freeze-experiment-contract
Jul 17, 2026
Merged

Fix #40/#41: Freeze benchmark scope and audit CnC measurements#48
isPANN merged 21 commits into
mainfrom
issue-40-freeze-experiment-contract

Conversation

@isPANN

@isPANN isPANN commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • freeze benchmark population scope before evaluation mechanics
  • require matched targets across Array, Wallace, Dadda, Booth-Wallace, Booth-Dadda, and Karatsuba multipliers
  • include public multiplier-equivalence suites and EPFL divisor, square, and square-root circuits
  • implement a reproducible data pipeline from targets or Verilog to paired CircuitSAT and DIMACS artifacts
  • add an auditable Cube-and-Conquer evidence bundle and verifier that reconstructs correctness and resource accounting from raw records
  • freeze the 64/96/128 width ladder while deferring tuning, splits, success criteria, and final instance counts to the evaluation protocol

Benchmark scope and pipeline (#40)

The versioned YAML scope is checked by a focused semantic audit. The data pipeline provides deterministic targets, native Array/Karatsuba generators, Yosys-based Verilog import, matched target pinning, equivalence miters, preimage generation, same-circuit Tseitin CNF encoding, source provenance, witness simulation, and manifest collection.

The pipeline uses one CLI (python3 -m benchmarks.pipeline.cli) with subcommands over six responsibility-based library modules. This replaces the previous one-file-per-command layout and centralizes shared JSONL I/O and argument handling.

Generated data stays outside Git. Manifests, measurements, source pins, hashes, Slurm logs, and compressed corpora travel with the externally archived dataset rather than the source tree.

Materialized corpora

  • 24/32-bit corpus: explicitly classified as pipeline smoke-only and excluded from reported results
  • 64-bit scale probe: 1 target x 6 architectures, Slurm job 9992430, 2m52s, 1,556,784 KiB MaxRSS, 535,922,054 extracted bytes, 6/6 witnesses pass
  • 96-bit scale probe: 1 target x 6 architectures, Slurm job 9992451, 6m33s, 3,674,560 KiB MaxRSS, 1,195,510,216 extracted bytes, 6/6 witnesses pass
  • 128-bit scale probe: 1 target x 6 architectures, Slurm job 9992497, 11m39s, 6,925,116 KiB MaxRSS, 2,107,878,363 extracted bytes, 6/6 witnesses pass
  • tagged Yosys v0.66 (86f2ddeb) and Multgen 215fe0a are pinned in provenance
  • scope v4 freezes 64/96/128 as the formal width ladder while leaving final instance counts to the evaluation protocol

Auditable CnC measurements (#41)

benchmarks/cnc/verify_measurements.py verifies a hash-linked evidence bundle containing the input, frontier, monotonic events, per-cube records, executable identities, and SAT witness. It independently reconstructs:

  • exhaustive frontier completeness and non-overlap
  • solved, cancelled, timed-out, and never-started cube lifecycles
  • per-cube verdicts from raw solver output
  • cubing wall/CPU, conquer CPU work, scheduled makespan, orchestration, and end-to-end wall time
  • worker assignments, overlap, and maximum concurrency
  • aggregate verdict and SAT witness validity, including membership in a SAT cube

One small positive fixture reproduces the required output. Tests create negative bundles in temporary directories for uncovered assignments, tampered accounting, and overlapping work, avoiding duplicated generated fixtures in Git.

Repository hygiene

  • no generated benchmark data, probe manifests, measurements, archives, or Slurm logs are committed
  • no HKUST-GZ-specific paths, partitions, environment locks, or submission scripts are committed
  • the CnC verifier lives under tracked benchmark tooling instead of the ignored experiments/ tree
  • Git history replaces a redundant scope digest lock and custom schema implementation
  • pipeline implementation is consolidated from 15 Python modules to 8 files including __init__.py and the unified CLI

Verification

  • python3 -m unittest discover -s tests -p 'test_*.py' (24 passed)
  • cargo test --locked --lib --tests (99 passed)
  • Ruff checks pass for benchmark, scope, CnC verifier, and Python tests
  • benchmark scope audit: 4 focused PASS checks
  • 64/96/128-bit external manifests and measurement hashes match the HPC artifacts
  • every generated multiplier passes private-factor witness simulation
  • all three compressed archives pass zstd -t
  • git diff --check

Closes #40
Closes #41

@isPANN isPANN changed the title Fix #40: Freeze a reviewer-auditable experiment contract Fix #40: Freeze benchmark scope before evaluation protocol Jul 16, 2026
@isPANN isPANN changed the title Fix #40: Freeze benchmark scope before evaluation protocol Fix #40/#41: Freeze benchmark scope and audit CnC measurements Jul 16, 2026
@isPANN
isPANN merged commit d5bdf83 into main Jul 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Establish trustworthy end-to-end CnC measurements Freeze benchmark scope before evaluation protocol

1 participant