Fix #40/#41: Freeze benchmark scope and audit CnC measurements#48
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark scope and pipeline (#40)
The versioned YAML scope is checked by a focused semantic audit. The data pipeline provides deterministic targets, native Array/Karatsuba generators, Yosys-based Verilog import, matched target pinning, equivalence miters, preimage generation, same-circuit Tseitin CNF encoding, source provenance, witness simulation, and manifest collection.
The pipeline uses one CLI (
python3 -m benchmarks.pipeline.cli) with subcommands over six responsibility-based library modules. This replaces the previous one-file-per-command layout and centralizes shared JSONL I/O and argument handling.Generated data stays outside Git. Manifests, measurements, source pins, hashes, Slurm logs, and compressed corpora travel with the externally archived dataset rather than the source tree.
Materialized corpora
9992430, 2m52s, 1,556,784 KiB MaxRSS, 535,922,054 extracted bytes, 6/6 witnesses pass9992451, 6m33s, 3,674,560 KiB MaxRSS, 1,195,510,216 extracted bytes, 6/6 witnesses pass9992497, 11m39s, 6,925,116 KiB MaxRSS, 2,107,878,363 extracted bytes, 6/6 witnesses pass86f2ddeb) and Multgen215fe0aare pinned in provenanceAuditable CnC measurements (#41)
benchmarks/cnc/verify_measurements.pyverifies a hash-linked evidence bundle containing the input, frontier, monotonic events, per-cube records, executable identities, and SAT witness. It independently reconstructs:One small positive fixture reproduces the required output. Tests create negative bundles in temporary directories for uncovered assignments, tampered accounting, and overlapping work, avoiding duplicated generated fixtures in Git.
Repository hygiene
experiments/tree__init__.pyand the unified CLIVerification
python3 -m unittest discover -s tests -p 'test_*.py'(24 passed)cargo test --locked --lib --tests(99 passed)zstd -tgit diff --checkCloses #40
Closes #41