feat(extract): add --memory-limit-mb so a budget overrun aborts cleanly instead of an OOM kill (#3011)

[abhay-codes07]

Closes #3011.

The problem

graphify extract inside a memory-limited container can grow past the cgroup allowance and be OOM-killed. --max-workers bounds the AST pool, but the later JS/TS symbol-resolution passes retain source buffers and syntax trees for the whole corpus, and GRAPHIFY_REBUILD_MEMORY_LIMIT_MB only ever applied to hook/watch rebuilds — never to the extract CLI. The kill leaves no graphify-specific failure, no stable exit status, and whatever had been written behind (the reporter measured 4–6.5 GiB peaks and OOM kills in an 8 GiB pod).

The change

--memory-limit-mb N / GRAPHIFY_MEMORY_LIMIT_MB=N on extract, update, and the bare graphify <path> form (which re-enters extract):

• Applies the cap with setrlimit (RLIMIT_AS; RLIMIT_DATA on macOS, whose allocator ignores RLIMIT_AS) to the CLI process and, through a pool initializer, to every extraction worker. Workers start fresh under spawn, so they read the cap from the environment; the flag is written back there so flag and env behave identically. An existing lower hard limit is never raised.
• Stops demoting MemoryError. Two places used to swallow it: _safe_extract recorded it as a skipped file, and the pool's per-future handler warned and retried the file in-process — which would hit the same wall in the parent. Either way a run could finish and publish a graph silently missing whatever came after. Now the pool cancels its queued work and the error propagates as a typed MemoryBudgetExceeded (a MemoryError subclass, so existing handling still applies).
• Fails cleanly. The CLI reports the configured limit, the phase, and the observed peak RSS, exits with status 3 (distinct from 1 = extraction failed, 2 = bad arguments, so a wrapper can tell "give it more memory" from "the corpus is broken" without parsing stderr), and writes no graph.json — the previous graph is untouched. --allow-partial does not apply here: the operator asked to be stopped.
• Is honest about its limits. A malformed value is refused (exit 2) rather than silently running with no budget — a budget that quietly vanished is the failure this exists to prevent. On Windows, where there is no setrlimit, the CLI says the budget cannot be enforced and continues; it does not pretend.

The enforcement lives in a dependency-free graphify/memory_budget.py. watch._apply_resource_limits now delegates to it and honours the general variable too, with the hook-specific GRAPHIFY_REBUILD_MEMORY_LIMIT_MB keeping precedence on that path, so one setting covers every rebuild.

RLIMIT_AS bounds virtual address space, which over-approximates the RSS a cgroup accounts — the README says to set the budget somewhat below the container limit. It is a per-process cap rather than an aggregate tree budget; that is what the platform offers portably without a dependency, and it is the same semantic the existing hook limit has always had.

What it looks like

$ graphify extract . --memory-limit-mb 6144
[graphify extract] memory budget: 6144 MB (applies to this process and its extraction workers)
...
error: memory budget of 6144 MB exceeded during AST extraction of src/generated/api.ts (peak observed in this process: ~6210 MB)
  configured: 6144 MB (--memory-limit-mb / GRAPHIFY_MEMORY_LIMIT_MB); the previous graph.json, if any, was left untouched.
  Raise the budget, narrow the corpus (.graphifyignore, --exclude), or lower --max-workers to reduce peak usage.
  exit status 3

On Windows:

[graphify extract] warning: memory budget of 512 MB cannot be enforced on this platform (no setrlimit); continuing without one

Tests

tests/test_memory_budget.py — 29 tests: value parsing and the refused env value; the typed error; setrlimit really applied and really biting (a 2 GiB allocation under a 256 MB cap raises MemoryError, run in a subprocess so the test process is never capped) and a lower existing hard limit preserved; the pool initializer and its construction; watch._apply_resource_limits precedence; _safe_extract still swallowing ordinary failures but letting MemoryError through; sequential extraction stopping rather than finishing partial; a worker hitting the budget aborting the pool with the typed error instead of a "worker failed" warning; the CLI exiting 3 with no graph.json for both flag forms and the env var; bad values exiting 2; the unenforceable-platform warning; the ordinary #2445 failure path unchanged; and update taking the flag, exiting 3, and still rejecting unknown options.

With the wiring reverted and only the module kept, 12 of them fail; with it, 27 pass and the 2 setrlimit tests skip on Windows (they run on the Linux CI; I also ran the enforcement path under WSL: cap 256 MB → MemoryError on the 2 GiB allocation). The full suite matches the v8 baseline.

README: an env-table row and a command-reference line.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.

Formal verification. 1 change(s) tested, no difference found (not proven).

---

Graphify review — findings

Adds an opt-in memory budget for extract and update via --memory-limit-mb / GRAPHIFY_MEMORY_LIMIT_MB, which caps the CLI process and its extraction workers with setrlimit so a runaway run aborts with exit status 3 and no partial graph.json instead of being OOM-killed. _arm_memory_budget resolves the flag (which wins over the env var and is written back so spawn workers and nested rebuilds inherit it), rejects a malformed value with exit 2, and warns once and continues where the platform can't enforce a limit. MemoryError now propagates through _safe_extract and pool workers rather than being logged as a skipped file, ensuring the graph is never silently published missing whatever came after the budget was hit.

Worth a look

• Memory budget flag races through process-global environment — graphify/cli.py:613 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Process pool abort still waits for running workers after budget hit — graphify/extract.py:5578 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• ThreadPool substitution runs the process initializer in the pytest process — tests/test_memory_budget.py:227 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1885 functions depend on the 421 functions this change touches.

Health — this change adds coupling hotspots:

• new: extract() — 494 callers, 42 callees
• new: _rebuild_code() — 98 callers, 50 callees
• new: extract_xaml() — 19 callers, 17 callees
• new: dispatch_command() — 2 callers, 127 callees
• new: extract_js() — 80 callers, 3 callees
• new: _get_extractor() — 26 callers, 6 callees
• new: run_pipeline() — 8 callers, 13 callees
• new: collect_files() — 17 callers, 6 callees
• …and 27 more — each is listed as a finding

Verification — 1885 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 1838 function(s) in the blast radius were not formally verified this run

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

Could not verify: Could not verify \_extract\_parallel.

The verifier did not have enough to check \_extract\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set

Could not verify: Could not verify \_safe\_extract.

The verifier did not have enough to check \_safe\_extract, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: parameter `extractor` is annotated `Callable` — outside the synthesizable primitive/collection set

No difference found (not proven): No behavior difference found in \_apply\_resource\_limits (not a proof).

The verifier ran both versions of \_apply\_resource\_limits on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

· 2 grounded finding(s) anchored inline below; 33 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

⚠️ Health regression — _extract_single_file()

fans out to 6 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

[graphify-labs]

⚠️ Health regression — budget_error()

6 callers depend on it (afferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.