Add sparsify error benchmarking artifacts#263
Open
noajshu wants to merge 1 commit into
Open
Conversation
noajshu
requested review from
LalehB and
arshpreetmaan
and removed request for
a team
LalehB
approved these changes
Jun 22, 2026
| actual_path = alt_path | ||
| else: | ||
| _E_CACHE[circuit_path] = (1, 1) | ||
| return (1, 1) |
Collaborator
There was a problem hiding this comment.
when we are returning (1, 1) here when the circuit file is missing is it possible that it would generate plausible-looking but incorrect plots for some cases? wdyt of raising an error by default, or adding an explicit --allow-missing-circuits mode for offline reproduction?
| _E_CACHE = {} | ||
|
|
||
| # Fallback mapping extracted directly from your provided output. | ||
| _FALLBACK_DETECT_MAP = { |
Collaborator
There was a problem hiding this comment.
For this fallback map that depends on exact circuit filenames. wdyt if we either compute these metrics from the circuit files directly or store E and num_detectors in aggregated_results.jsonl? That would make the analysis less coupled to one benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the benchmarking artifacts used for the error sparsification work in #254 under
benchmarking/sparsify_errors/.This includes: