Skip to content

feat: add --memory_profile_out flag for Chonk memory profiling#22145

Open
johnathan79717 wants to merge 2 commits intomerge-train/barretenbergfrom
jh/memory-profile-out
Open

feat: add --memory_profile_out flag for Chonk memory profiling#22145
johnathan79717 wants to merge 2 commits intomerge-train/barretenbergfrom
jh/memory-profile-out

Conversation

@johnathan79717
Copy link
Copy Markdown
Contributor

@johnathan79717 johnathan79717 commented Mar 30, 2026

Summary

  • Adds --memory_profile_out <file> flag to bb prove that outputs a JSON report of memory consumption during Chonk proving
  • Per-circuit polynomial memory breakdown by category (wires, sigmas, ids, selectors, tables, lookup, databus, etc.)
  • RSS checkpoints at key stages (after polynomial allocation, after accumulate) to identify when peak occurs
  • CRS (commitment key) memory tracking
  • Python extraction script converts JSON into benchmark dashboard entries (stacked area chart for categories, line charts for totals)
  • Integrated into ci_benchmark_ivc_flows.sh for continuous tracking

Refs: AztecProtocol/barretenberg#1641

Dashboard change needed

The extraction script uses extra: "stacked-area:..." which requires a new renderStackedAreaGraph function in benchmark-page-data/bench/index.html. The change is ready at /mnt/user-data/jonathan/benchmark-page-data on branch jh/stacked-area-chart and needs to be pushed by someone with access to AztecProtocol/benchmark-page-data. Until then, the memory metrics still appear as individual line charts (graceful fallback).

Example output

Peak RSS: 256 MiB at after_accumulate (circuit 6)
Peak circuit polynomial memory: 82.5 MB (circuit 6)
  selectors: 26.1 MB, wires: 14.9 MB, sigmas: 14.9 MB, ids: 14.9 MB
  other: 4.4 MB, z_perm: 3.7 MB, lookup: 3.3 MB, databus: 0.2 MB
CRS: 16.0 MB
Total circuits: 13

Test plan

  • Builds cleanly (ninja bb, ninja chonk_tests, ninja ultra_honk_tests)
  • Tested locally with deploy_ecdsar1+sponsored_fpc flow: JSON output is valid, all 13 circuits profiled with RSS checkpoints
  • Extraction script produces correct stacked-area chart entries for benchmark dashboard
  • CI build passes

Add tooling to continually assess memory consumption in Chonk proving.
The new --memory_profile_out flag on `bb prove` outputs a JSON report
with per-circuit polynomial memory breakdown by category (wires, sigmas,
selectors, etc.), CRS size, and RSS checkpoints at key proving stages.

A Python extraction script converts the JSON into benchmark dashboard
entries (stacked charts for polynomial categories, line charts for
totals and peak RSS). Integrated into ci_benchmark_ivc_flows.sh.

Refs: AztecProtocol/barretenberg#1641
@johnathan79717 johnathan79717 added the ci-barretenberg Run all barretenberg/cpp checks. label Mar 30, 2026
Switch from stacked: to stacked-area: prefix in extract_memory_benchmarks.py
so the dashboard renders proper stacked area charts instead of overlaid lines.
This requires the corresponding stacked-area chart support in benchmark-page-data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-barretenberg Run all barretenberg/cpp checks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant