Add asv benchmarking support - #258
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
I ran the benchmarks locally, but they are failing in the |
|
Also a couple of things about benchmarking -
|
vinayswamik
changed the title
…of the tree - benchmark_downloader: remove the CERT_NONE SSL context and global opener; fetch with default certificate verification. Downloads now cache to tests/benchmarks/qasm/<name> (gitignored) instead of a fresh NamedTemporaryFile per call, so repeated asv setups stop re-fetching. - asv.conf.json: move html_dir under .asv/ so 'asv publish' does not dirty the tree; add Python 3.14 to match the support matrix.
Argus reviewAuto-review is off for this repo. Tick the box below to run a review on this PR.
Estimated cost
Tip: you can also comment |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Refreshed this branch to get it review-ready again (pushed three commits — main advanced mid-refresh, so there are two merge commits):
Verified against current The earlier review asks (benchpress + pinned commit hash, size-parameterized benchmarks, |
The SVG flamegraphs and speedscope JSONs under tests/profiles/ snapshot a specific commit's performance and go stale on any code change, so they would need continuous regeneration to stay truthful. Benchmarks stay reproducible via asv; profiles can be regenerated locally with py-spy when needed.
|
Follow-up: removed the committed py-spy artifacts under |
ryanhill1
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/benchmarks/__init__.py (1)
1-14: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd module docstrings to the new benchmark packages.
Both package modules omit the required purpose documentation.
tests/benchmarks/__init__.py#L1-L14: Add a module docstring that describes the benchmark package.tests/benchmarks/qasm/__init__.py#L1-L14: Add a module docstring that describes the QASM benchmark package.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/benchmarks/__init__.py` around lines 1 - 14, Add a top-level module docstring to tests/benchmarks/__init__.py describing the benchmark package, and add a corresponding top-level module docstring to tests/benchmarks/qasm/__init__.py describing the QASM benchmark package; keep the existing license headers intact.Source: Coding guidelines
🧹 Nitpick comments (1)
tests/benchmarks/qasm/benchmark_downloader.py (1)
28-77: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winComplete the type and documentation contracts.
__init__has no-> Noneannotation.file_info: Dicthas no value type. The methods do not document their parameters and return values. Add annotations forself.cache_dirandself.metadata.As per coding guidelines, “All functions, methods, and class attributes must have type annotations” and docstrings must explain purpose, parameters, and return values.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/benchmarks/qasm/benchmark_downloader.py` around lines 28 - 77, The BenchmarkDownloader API lacks complete type and documentation contracts. Update __init__, get_file_path, _fetch_remote_file, and get_benchmark_file with complete parameter and return annotations, including -> None, a value type for file_info, and explicit annotations for self.cache_dir and self.metadata; expand each method docstring to describe its purpose, parameters, and return value.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/benchmarks/import.py`:
- Around line 23-27: Complete the benchmark annotations and docstrings across
tests/benchmarks/import.py:23-27 (document and annotate
PyqasmImport.time_pyqasm_import), tests/benchmarks/openpulse.py:24-33 (annotate
qasm_file, setup, and time_openpulse and document each method),
tests/benchmarks/pyqasm_functions.py:29-85, tests/benchmarks/unroll.py:26-62,
and tests/benchmarks/validate.py:26-58 (annotate ASV metadata, setup-created
attributes, parameters, and return values, and document every method with its
purpose, parameters, and return value). Ensure all affected classes, methods,
functions, and attributes comply with the project’s typing and docstring
requirements.
- Around line 26-27: Update time_pyqasm_import to use subprocess.check_call
instead of subprocess.call when executing the pyqasm import, so nonzero exit
statuses raise CalledProcessError and fail the ASV benchmark.
In `@tests/benchmarks/qasm/benchmark_downloader.py`:
- Around line 64-68: Update the download logic around urllib.request.urlopen in
the benchmark downloader to pass an explicit timeout and prevent stalled
requests from blocking indefinitely. Catch urllib.error.URLError and
timeout-related exceptions explicitly, convert them to the existing
connection-specific exception with the filename and URL context, and allow
unrelated programming errors to propagate.
In `@tests/benchmarks/qasm/neutral_atom_gate.qasm`:
- Around line 27-29: Update the eom_a_port, eom_b_port, and aod_port
declarations to use extern port instead of port, preserving their names and
order.
In `@tests/benchmarks/unroll.py`:
- Around line 56-62: The benchmark methods unroll.py:56-62 and
pyqasm_functions.py:83-85 reuse modules that benchmark operations mutate. Update
time_unroll and time_qubit_reg_consolidation to load a fresh module from
self.qasm_file for every sample, and update mpl_draw to avoid reusing the loaded
module by restoring or recreating an unmodified module between draws;
alternatively, configure the unroll benchmarks with number = 1 and warmup_time =
0.
---
Outside diff comments:
In `@tests/benchmarks/__init__.py`:
- Around line 1-14: Add a top-level module docstring to
tests/benchmarks/__init__.py describing the benchmark package, and add a
corresponding top-level module docstring to tests/benchmarks/qasm/__init__.py
describing the QASM benchmark package; keep the existing license headers intact.
---
Nitpick comments:
In `@tests/benchmarks/qasm/benchmark_downloader.py`:
- Around line 28-77: The BenchmarkDownloader API lacks complete type and
documentation contracts. Update __init__, get_file_path, _fetch_remote_file, and
get_benchmark_file with complete parameter and return annotations, including ->
None, a value type for file_info, and explicit annotations for self.cache_dir
and self.metadata; expand each method docstring to describe its purpose,
parameters, and return value.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a672e7bf-c0dd-4984-99b6-b1ca4fa41fb2
📒 Files selected for processing (13)
.gitignoreCHANGELOG.mdasv.conf.jsontests/benchmarks/__init__.pytests/benchmarks/import.pytests/benchmarks/openpulse.pytests/benchmarks/pyqasm_functions.pytests/benchmarks/qasm/__init__.pytests/benchmarks/qasm/benchmark_downloader.pytests/benchmarks/qasm/benchmark_metadata.jsontests/benchmarks/qasm/neutral_atom_gate.qasmtests/benchmarks/unroll.pytests/benchmarks/validate.py
| class PyqasmImport: | ||
| """Test the import time of pyqasm.""" | ||
|
|
||
| def time_pyqasm_import(self): | ||
| call((executable, "-c", "import pyqasm")) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add required type annotations and complete docstrings.
Annotate every benchmark method, parameter, return value, and class or instance attribute. Add docstrings that state each method purpose, parameters, and return value.
tests/benchmarks/import.py#L23-L27: annotate and documenttime_pyqasm_import.tests/benchmarks/openpulse.py#L24-L33: annotateqasm_file,setup, andtime_openpulse; add complete method docstrings.tests/benchmarks/pyqasm_functions.py#L29-L85: annotate ASV metadata and setup-created attributes; document and annotate all methods.tests/benchmarks/unroll.py#L26-L62: annotate ASV metadata and setup-created attributes; document and annotate all methods.tests/benchmarks/validate.py#L26-L58: annotate ASV metadata and setup-created attributes; document and annotate all methods.
As per coding guidelines, "**/*.py: Every module, class, method, and function must have a docstring explaining its purpose, parameters, and return values" and "All functions, methods, and class attributes must have type annotations."
📍 Affects 5 files
tests/benchmarks/import.py#L23-L27(this comment)tests/benchmarks/openpulse.py#L24-L33tests/benchmarks/pyqasm_functions.py#L29-L85tests/benchmarks/unroll.py#L26-L62tests/benchmarks/validate.py#L26-L58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/benchmarks/import.py` around lines 23 - 27, Complete the benchmark
annotations and docstrings across tests/benchmarks/import.py:23-27 (document and
annotate PyqasmImport.time_pyqasm_import), tests/benchmarks/openpulse.py:24-33
(annotate qasm_file, setup, and time_openpulse and document each method),
tests/benchmarks/pyqasm_functions.py:29-85, tests/benchmarks/unroll.py:26-62,
and tests/benchmarks/validate.py:26-58 (annotate ASV metadata, setup-created
attributes, parameters, and return values, and document every method with its
purpose, parameters, and return value). Ensure all affected classes, methods,
functions, and attributes comply with the project’s typing and docstring
requirements.
Source: Coding guidelines
| def time_pyqasm_import(self): | ||
| call((executable, "-c", "import pyqasm")) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)tests/benchmarks/import\.py$|benchmark|asv' || true
echo
echo "Import benchmark file:"
if [ -f tests/benchmarks/import.py ]; then
cat -n tests/benchmarks/import.py
fi
echo
echo "Search subprocess usage in benchmark test files:"
rg -n "from subprocess|subprocess\.|check_call|call\\(|Executable|pyqasm" tests/benchmarks benchmarks -g '*.py' 2>/dev/null || trueRepository: qBraid/pyqasm
Length of output: 3643
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import subprocess
import tempfile
import os
executable = getattr(subprocess, '_executable', None)
if executable is None:
executable = os.environ.get('ASV_PYTHON', None)
if executable is None:
executable = "python3"
cmd = (executable, "-E", "-c", "import sys; sys.exit(1)")
print(f"executable={executable!r}")
ret = subprocess.call(cmd)
print(f"call_exit_status={ret}")
try:
subprocess.check_call(cmd)
except subprocess.CalledProcessError as exc:
print(f"check_call_reraised={exc.__class__.__name__}, returncode={exc.returncode}")
PYRepository: qBraid/pyqasm
Length of output: 244
Fail the benchmark when the import fails.
subprocess.call() returns the nonzero exit status instead of raising, so a failed import pyqasm can still report a successful ASV result. Use check_call() so ASV receives CalledProcessError.
Proposed fix
-from subprocess import call
+from subprocess import check_call
...
- call((executable, "-c", "import pyqasm"))
+ check_call((executable, "-c", "import pyqasm"))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def time_pyqasm_import(self): | |
| call((executable, "-c", "import pyqasm")) | |
| def time_pyqasm_import(self): | |
| check_call((executable, "-c", "import pyqasm")) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/benchmarks/import.py` around lines 26 - 27, Update time_pyqasm_import
to use subprocess.check_call instead of subprocess.call when executing the
pyqasm import, so nonzero exit statuses raise CalledProcessError and fail the
ASV benchmark.
| try: | ||
| with urllib.request.urlopen(url) as response: | ||
| content = response.read() | ||
| except Exception as e: | ||
| raise RuntimeError(f"Failed to fetch {filename} from {url}: {e}") from e |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)benchmark_downloader\.py$|qasm' || true
echo
echo "Target file outline:"
ast-grep outline tests/benchmarks/qasm/benchmark_downloader.py || true
echo
echo "Target lines 1-140:"
cat -n tests/benchmarks/qasm/benchmark_downloader.py | sed -n '1,140p'
echo
echo "Imports/usages of benchmark_downloader:"
rg -n "benchmark_downloader|download.*qasm|fetch .*qasm|urlopen|URLError|timeout" tests/benchmarks || true
echo
echo "Python urllib.timeout behavior probe:"
python3 - <<'PY'
import inspect, urllib.request
sig = inspect.signature(urllib.request.urlopen)
print("urlopen signature:", sig)
PYRepository: qBraid/pyqasm
Length of output: 9437
Bound the download and preserve connection failures.
urllib.request.urlopen(url) uses no timeout, so a stalled download can block ASV indefinitely. Add an explicit timeout. Catch urllib.error.URLError and timeout-related errors explicitly, then raise a connection-specific exception; let unrelated programming errors propagate.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 64-64: Request-controlled URL passed to urlopen; validate against an allowlist to prevent SSRF.
Context: urllib.request.urlopen(url)
Note: [CWE-918] Server-Side Request Forgery (SSRF).
(urlopen-unsanitized-data)
🪛 Ruff (0.16.1)
[error] 65-65: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.
(S310)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/benchmarks/qasm/benchmark_downloader.py` around lines 64 - 68, Update
the download logic around urllib.request.urlopen in the benchmark downloader to
pass an explicit timeout and prevent stalled requests from blocking
indefinitely. Catch urllib.error.URLError and timeout-related exceptions
explicitly, convert them to the existing connection-specific exception with the
filename and URL context, and allow unrelated programming errors to propagate.
Source: Coding guidelines
| port eom_a_port; | ||
| port eom_b_port; | ||
| port aod_port; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file context =="
if [ -f tests/benchmarks/qasm/neutral_atom_gate.qasm ]; then
cat -n tests/benchmarks/qasm/neutral_atom_gate.qasm | sed -n '1,80p'
else
echo "missing tests/benchmarks/qasm/neutral_atom_gate.qasm"
fi
echo "== benchmark references =="
rg -n "neutral_atom_gate|BenchmarkDownloader|openpulse|OpenPulse|qasm" tests benchmarks -S 2>/dev/null | head -200 || true
echo "== pyqasm/openpulse parser presence =="
rg -n "openpulse|OpenPulse|pyqasm|qasm" . -g '*.py' -g '*.toml' -g '*.lock' -g '*.txt' -S 2>/dev/null | head -300 || trueRepository: qBraid/pyqasm
Length of output: 44242
🌐 Web query:
OpenQASM 3.0 OpenPulse port extern port declaration bare port
💡 Result:
In OpenQASM 3.0 and OpenPulse, a port is a software abstraction used to represent physical input or output resources for qubit control [1][2]. The declaration extern port is the standard mechanism for accessing these hardware-defined ports [1][2]. This syntax specifies an external linkage that is resolved at compile-time using vendor-supplied translation units [1][2]. For example, a port can be declared as follows: extern port drive_port0; Because these ports represent external physical resources, they are typically defined by the hardware vendor within an include file (often named something like backend.inc), which the user includes in their OpenQASM program [3][4]. There is no standard concept of a "bare port" declaration in the OpenQASM 3.0 specification; all ports must be explicitly declared, typically using the extern keyword to link them to the target system's specific hardware map [1][3]. The term "bare port" appears in other technical contexts (such as networking or RTL design), but it is not a part of the OpenQASM or OpenPulse language grammar [5][6][7][8][9].
Citations:
- 1: https://openqasm.com/versions/3.0/language/openpulse.html
- 2: https://openqasm.com/versions/3.1/language/openpulse.html
- 3: https://openqasm.com/versions/3.0/language/pulses.html
- 4: https://openqasm.com/language/pulses.html
- 5: https://github.com/MHSanaei/3x-ui/blob/53f6ed39/frontend/src/test/stream-wire-normalize.test.ts
- 6: feat: support --allow-net=:4500 denoland/deno#3115
- 7: https://chipverify.ai/learn/output-never-assigned
- 8: openssh/openssh-portable@87ddd67
- 9: https://github.com/browserbase/stagehand/blob/2eb94164/packages/cli/src/local-cdp-discovery.ts
🌐 Web query:
OpenPulse "extern port" "port" declaration
💡 Result:
In OpenPulse, a port represents an abstract I/O channel used for physical resources such as playing pulses or capturing data [1][2][3]. The extern keyword is used to declare a port that is defined externally to the immediate OpenQASM code block, allowing it to be resolved at compile-time via vendor-supplied translation units or mapping files [1][2]. A typical declaration takes the following form: extern port <port_name>; For example: extern port drive_port0; Key points regarding the extern port declaration: 1. External Linkage: By declaring a port as extern, you specify that its physical implementation (e.g., mapping to a specific XY control line on a qubit) will be handled by the hardware provider [1][2]. 2. Scope: These declarations are typically used within cal (calibration) or defcal blocks [1][4][5]. 3. Compilation: The identifier is resolved during the compilation or transpilation process [1][2]. For instance, tools like LabOne Q use an externs dictionary or map to link these program-defined port names to the actual hardware signals during the experiment setup [6]. 4. Purpose: Once declared, these ports are used to construct frames, which are stateful carriers with frequency, phase, and timing information, essential for playing waveforms to or capturing data from the quantum processor [1][2][3].
Citations:
- 1: https://openqasm.com/language/openpulse.html
- 2: https://openqasm.com/versions/3.0/language/openpulse.html
- 3: https://docs.qbraid.com/v2/pyqasm/user-guide/openpulse
- 4: https://openqasm.com/versions/3.1/language/pulses.html
- 5: https://openqasm.com/language/pulses.html
- 6: https://docs.zhinst.com/labone_q_user_manual/core/functionality_and_concepts/08_openqasm/tutorials/00_program_to_experiment.html
🌐 Web query:
site:openqasm.com "Port" "extern port" OpenPulse
💡 Result:
In OpenPulse, an OpenQASM specification for quantum control, the "extern port" syntax is used to declare a hardware port that is linked to an external resource [1][2]. A port serves as a software abstraction representing an input or output component (such as a digital-to-analog converter, amplifier, or local oscillator) used to manipulate and observe qubits [1][2][3]. Because these ports represent specific physical hardware, they are vendor- and device-specific [2][3]. When you declare a port using the "extern" keyword, you are specifying that it has an external linkage [1][3]. This linkage is resolved at compile-time by translation units provided by the hardware vendor [1][2]. Essentially, the vendor provides the specific port names and mappings as configuration information, which the user then accesses within their OpenQASM code using the "extern port" declaration [2][3]. Example syntax: extern port drive_port0; frame driveframe0 = newframe(drive_port0, 5e9, 0.0); [1]
Citations:
- 1: https://openqasm.com/versions/3.1/language/openpulse.html
- 2: https://openqasm.com/language/openpulse.html
- 3: https://openqasm.com/versions/3.0/language/openpulse.html
Declare the OpenPulse ports as extern.
Replace each port declaration at lines 27-29 with extern port. OpenPulse ports are external hardware abstractions declared with extern port; bare port declarations are not useable here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/benchmarks/qasm/neutral_atom_gate.qasm` around lines 27 - 29, Update
the eom_a_port, eom_b_port, and aod_port declarations to use extern port instead
of port, preserving their names and order.
| def time_unroll(self, _): | ||
| """Unroll QASM file of specified size.""" | ||
| _ = self.pyqasm_obj.unroll() | ||
|
|
||
| def time_qubit_reg_consolidation(self, _): | ||
| """Unroll QASM file of specified size with qubit consolidation.""" | ||
| _ = self.pyqasm_obj.unroll(consolidate_qubits=True) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repository files matching benchmarks:\n'
git ls-files 'tests/benchmarks/*.py'
printf '\nRelevant benchmark files:\n'
for f in tests/benchmarks/unroll.py tests/benchmarks/pyqasm_functions.py; do
if [ -f "$f" ]; then
echo "===== $f ====="
wc -l "$f"
sed -n '1,140p' "$f"
fi
done
printf '\nSearch for mpl_draw definition/usages and unroll methods:\n'
rg -n "def mpl_draw|mpl_draw|def unroll|class .*ASV|number|warmup_time" -S --glob '*.py' .Repository: qBraid/pyqasm
Length of output: 13702
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Relevant pyqasm module print/dot unroll methods:\n'
sed -n '370,450p' src/pyqasm/printer.py
printf '\n'
sed -n '320,400p' src/pyqasm/modules/base.py
printf '\nASV benchmark runner behavior/source (installed, if available):\n'
python3 - <<'PY'
try:
import asv_runner.runner as runner
import inspect
print(inspect.getfile(runner))
src = inspect.getsource(runner)
for needle in ["class BenchmarkRunner", "number_times_run", "run_benchmarks", "setup"]:
idx = src.find(needle)
if idx != -1:
start = max(0, src.rfind("\n", 0, idx-1000))
end = min(len(src), src.find("\n", idx+2000))
print(f"\n--- {needle} slice ---\n{src[start:end]}")
except Exception as exc:
print(f"ASV runner import not available or error: {type(exc).__name__}: {exc}")
PYRepository: qBraid/pyqasm
Length of output: 6088
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Top-level unroll implementation:\n'
sed -n '600,720p' src/pyqasm/modules/base.py
printf '\nSearch for unrolled_ast mutation in module code:\n'
rg -n "_unrolled_ast|unrolled_ast|def unroll\\(" src/pyqasm/modules -S
printf '\nPython structural probe for unroll signature and mutation pattern:\n'
python3 - <<'PY'
import ast
from pathlib import Path
path = Path("src/pyqasm/modules/base.py")
tree = ast.parse(path.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == "unroll":
print(ast.get_docstring(node))
print("location:", node.lineno, node.col_offset)
print("args:", [arg.arg for arg in node.args.args])
for child in ast.walk(node):
if isinstance(child, ast.Assign):
names = []
for t in child.targets:
if isinstance(t, ast.Attribute):
names.append(".".join(filter(None, [ast.unparse(n) for n in ([t.value, t.attr] if isinstance(t.value, ast.Attribute) else [t, t.attr])])))
else:
names.append(ast.unparse(t))
print("assign_target:", names)
PYRepository: qBraid/pyqasm
Length of output: 11043
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Printer top-level section and unroll/dot references:\n'
sed -n '1,240p' src/pyqasm/printer.py
printf '\nRelevant Qasm3 unroll implementation:\n'
sed -n '1,120p' src/pyqasm/modules/qasm3.py
printf '\nRead-only benchmark state probe:\n'
python3 - <<'PY'
import ast
from pathlib import Path
for path in [Path("tests/benchmarks/unroll.py"), Path("tests/benchmarks/pyqasm_functions.py")]:
tree = ast.parse(path.read_text())
for cls in [n for n in tree.body if isinstance(n, ast.ClassDef) and n.name in {"Unrolling", "PyqasmFunctions"}]:
print(f"--- {cls.name} ---")
for node in cls.body:
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
print(f"annot_assign {node.target.id}")
elif isinstance(node, ast.Assign):
targets = [t.id if isinstance(t, ast.Name) else ast.unparse(t) for t in node.targets]
print(f"assign {targets}: {ast.unparse(node.value)[:80]}")
elif isinstance(node, ast.FunctionDef):
print(f"def {node.name}")
PYRepository: qBraid/pyqasm
Length of output: 11133
🌐 Web query:
asv benchmarks module setup timed method number warmup_time setup teardown
💡 Result:
In Airspeed Velocity (asv), setup, teardown, warmup_time, and number are key attributes for controlling benchmark execution and timing accuracy. Setup and Teardown - setup: A method or function called before the benchmark runs to perform necessary initialization [1][2][3]. You can define module-level setup functions (run for every benchmark in the module) or class-based setup methods [1][4]. - teardown: A method or function called after the benchmark finishes, useful for cleaning up resources like filesystem changes [1][3]. - Execution flow: For timing benchmarks, the setup runs first, then the benchmark function is called number times, and finally the teardown runs [2][4][5]. Notably, setup and teardown are not executed between iterations of the number parameter [2][5]. - setup_cache: If setup is computationally expensive, you can use setup_cache instead. It runs only once per parameter set and caches the result, avoiding repeated execution across repeats and profiling runs [1][3]. Timing Attributes - number: Specifies the number of iterations to run the benchmarked function within a single sample [2][5]. If this is not manually set, asv adaptively estimates an optimal number of executions [6]. - warmup_time: Defines the duration (in seconds) that asv spends calling the benchmarked function repeatedly before starting official measurements [2][6][5]. This helps account for transient effects like OS scheduling or JIT compiler optimization (defaulting to 0.1s for CPython and 1.0s for PyPy) [2][6][5]. - Dynamic Adjustment: Attributes like number, repeat, and sample_time can be adjusted dynamically within a setup routine, which is particularly useful for parameterized benchmarks [2][5]. The overall execution sequence for a timing benchmark follows this pattern: calibrate (if number is not set) -> setup -> [repeat loops] -> benchmark (called number times) -> teardown [4].
Citations:
- 1: https://asv.readthedocs.io/en/latest/writing_benchmarks.html
- 2: https://asv.readthedocs.io/en/stable/benchmarks.html
- 3: https://asv.readthedocs.io/en/stable/writing_benchmarks.html
- 4: https://github.com/airspeed-velocity/asv/blob/main/docs/source/writing_benchmarks.rst
- 5: https://asv.readthedocs.io/en/v0.6.4/benchmarks.html
- 6: https://asv.readthedocs.io/projects/asv-runner/en/latest/apidocs/asv%5Frunner/asv%5Frunner.benchmarks.time.html
Benchmark each operation from an unmodified module.
ASV runs setup once and then calls the timed method multiple times. unroll() mutates self.pyqasm_obj by replacing _statements with the unrolled AST, so later calls do not measure the operation on the loaded program. mpl_draw() also calls program.unroll() before drawing.
tests/benchmarks/unroll.py#L56-L62: run each sample from a freshload(self.qasm_file)object, or setnumber = 1andwarmup_time = 0for these methods.tests/benchmarks/pyqasm_functions.py#L83-L85: do not reuse the same loaded module formpl_draw()unless you restore an unmodified module between draws.
📍 Affects 2 files
tests/benchmarks/unroll.py#L56-L62(this comment)tests/benchmarks/pyqasm_functions.py#L83-L85
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/benchmarks/unroll.py` around lines 56 - 62, The benchmark methods
unroll.py:56-62 and pyqasm_functions.py:83-85 reuse modules that benchmark
operations mutate. Update time_unroll and time_qubit_reg_consolidation to load a
fresh module from self.qasm_file for every sample, and update mpl_draw to avoid
reusing the loaded module by restoring or recreating an unmodified module
between draws; alternatively, configure the unroll benchmarks with number = 1
and warmup_time = 0.
4 participants
Summary of changes
import,load,draw,unroll,validate,dumpand ,dumps.Qubit Register ConsolidationandOpenpulseprograms.Verification Instructions:
From the project root, you can verify the benchmarks with the following steps:
Creates a benchmarking environment under .asv/.
Generates
htmloutput files containing benchmark results in tests/benchmarking/.Launches a local server to view benchmarking results in your browser.
Notes
Profiling
Profiling data and visualizations for PyQASM functions
openpulse,qubit register consolidation,unroll, andvalidateprofiling.Closes #67
Summary by CodeRabbit
New Features
Documentation
Chores