feat(llm): openai-cli backend — semantic extraction through the locally authenticated Codex CLI - #3073
feat(llm): openai-cli backend — semantic extraction through the locally authenticated Codex CLI#3073Azeem1985 wants to merge 1 commit into
Conversation
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Graphify review — findings
Adds an openai-cli backend (--backend openai-cli) that runs extraction and community labeling through the locally authenticated Codex CLI via codex exec, billing against a ChatGPT subscription rather than a metered OPENAI_API_KEY; the credential gate accepts a present codex binary in place of a key, mirroring claude-cli. The prompt is piped over stdin to dodge Linux's 128 KB argv cap, the sandbox runs read-only so an agentic Codex can't write to the corpus, every configured MCP server is disabled per call through per-server enabled=false overrides (a blanket mcp_servers={} deep-merges away and leaves them running), and token usage is read from the last turn.completed JSONL event without double-counting cached input. Calls are forced serial unless GRAPHIFY_OPENAI_CLI_PARALLEL=1, and GRAPHIFY_OPENAI_CLI_MODEL (default gpt-5.6-sol) and GRAPHIFY_OPENAI_CLI_EFFORT (default ultra) tune the model and effort.
Worth a look
- Incomplete conditional leaves module syntactically invalid —
graphify/llm.py:2827· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Incomplete openai-cli concurrency branch makes module unloadable —
graphify/llm.py:2827· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1088 functions depend on the 460 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 63 callers, 21 callees - new:
build_merge()— 46 callers, 14 callees - new:
extract_files_direct()— 17 callers, 21 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
_call_claude_cli()— 31 callers, 9 callees - new:
dispatch_command()— 2 callers, 122 callees - new:
_call_llm()— 11 callers, 21 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - …and 19 more — each is listed as a finding
Verification — 1088 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1024 function(s) in the blast radius were not formally verified this run
· 2 grounded finding(s) anchored inline below; 25 more finding(s) on lines outside this diff (see the check run).
| return " | ".join(parts) or "(no stderr or stdout)" | ||
|
|
||
|
|
||
| def _call_claude_cli(user_message: str, max_tokens: int = 8192, *, deep_mode: bool = False, images: list[_ImageRef] | None = None) -> dict: |
There was a problem hiding this comment.
_call_claude_cli()
fans out to 9 callees (efferent coupling); 31 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return args | ||
|
|
||
|
|
||
| def _call_openai_cli(user_message: str, max_tokens: int = 8192, *, deep_mode: bool = False, images: list[_ImageRef] | None = None) -> dict: |
There was a problem hiding this comment.
_call_openai_cli()
fans out to 9 callees (efferent coupling); 11 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
…ly authenticated Codex CLI
Adds `--backend openai-cli`, which runs semantic extraction and community
labeling through `codex exec` — the locally authenticated OpenAI Codex
CLI — so the work rides a ChatGPT OAuth subscription instead of a metered
OPENAI_API_KEY. It mirrors the existing `claude-cli` backend: zero
pricing, forced-serial calls, and a credential gate that accepts a
present CLI in place of an API key.
Four details that took measurement to get right, all in the code:
- The prompt goes in over stdin, not argv. Linux caps a single argv entry
at 128 KB (MAX_ARG_STRLEN) and real extraction chunks reach 240-306 KB,
so passing the prompt as an argument fails with E2BIG on ordinary input.
- The sandbox is read-only, so an agentic Codex cannot write into the
corpus it is reading.
- MCP servers are disabled per server, via Codex's own
`mcp_servers.<name>.enabled=false` override. A blanket `mcp_servers={}`
does not work — Codex deep-merges config, so the key merges away and
every configured server still launches. Measured cost of the naive
version: four servers of ~152 MB each, spawned per `codex exec` call.
- An empty-but-valid graph raises instead of being returned. A hollow
response otherwise feeds the chunk-bisect retry path, which once burned
87% of an hourly quota subdividing a response that was never going to
improve.
Token usage is read from the last `turn.completed` JSONL event, so cached
input is not double-counted. `GRAPHIFY_OPENAI_CLI_MODEL` (default
`gpt-5.6-sol`) and `GRAPHIFY_OPENAI_CLI_EFFORT` (default `ultra`)
configure it; `GRAPHIFY_OPENAI_CLI_PARALLEL=1` opts out of serial calls.
Tests: 15 covering the happy parse, failure raising with vendor detail,
turn-usage accounting, the stdin path, and the per-server MCP disable.
08bf8af to
4e1cfd6
Compare
|
Re the two Checked on a clean shallow clone of
The flagged line is a complete two-line guard, third in the existing run of serial-by-default backend guards: if backend == "openai-cli" and os.environ.get("GRAPHIFY_OPENAI_CLI_PARALLEL", "").strip() != "1":
max_concurrency = 1exactly mirroring the On the two inline coupling notes: |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 2 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an openai-cli backend that runs semantic extraction and community labeling through the locally authenticated Codex CLI (codex exec) so the work rides a ChatGPT subscription instead of a metered OPENAI_API_KEY; the credential gate accepts a codex binary on $PATH in lieu of an API key. The prompt is piped via stdin to clear Linux's 128 KB argv limit, the sandbox is forced read-only, and every configured MCP server is disabled per-call via _codex_disable_mcp_args (best-effort — if codex mcp list is unavailable, the servers stay as the user configured them). Token counts come from the last turn.completed JSONL event without re-adding cached input, and calls run serial unless GRAPHIFY_OPENAI_CLI_PARALLEL=1; GRAPHIFY_OPENAI_CLI_MODEL (default gpt-5.6-sol) and GRAPHIFY_OPENAI_CLI_EFFORT (default ultra) tune it.
Worth a look
- Untrusted corpus text is sent to an agentic CLI that can run tools —
graphify/llm.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- openai-cli concurrency not forced to 1 in code despite docs —
graphify/__main__.py:547· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- MCP server disabling fails open —
graphify/llm.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- extract_corpus_parallel truncated openai-cli concurrency guard likely broken —
graphify/llm.py:2827· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- openai-cli runs an LLM agent with inherited cwd and environment —
graphify/llm.py:3167· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1088 functions depend on the 460 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 63 callers, 21 callees - new:
build_merge()— 46 callers, 14 callees - new:
extract_files_direct()— 17 callers, 21 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
_call_claude_cli()— 31 callers, 9 callees - new:
dispatch_command()— 2 callers, 122 callees - new:
_call_llm()— 11 callers, 21 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - …and 19 more — each is listed as a finding
Verification — 1088 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1024 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_call\_llm.
The verifier did not have enough to check \_call\_llm, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in detect\_backend (not a proof).
The verifier ran both versions of detect\_backend on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract\_corpus\_parallel.
The verifier did not have enough to check extract\_corpus\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify extract\_files\_direct.
The verifier did not have enough to check extract\_files\_direct, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify label\_communities.
The verifier did not have enough to check label\_communities, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 49 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_run\_cli (not a proof).
The verifier ran both versions of \_run\_cli on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 2 grounded finding(s) anchored inline below; 25 more finding(s) on lines outside this diff (see the check run).
| return " | ".join(parts) or "(no stderr or stdout)" | ||
|
|
||
|
|
||
| def _call_claude_cli(user_message: str, max_tokens: int = 8192, *, deep_mode: bool = False, images: list[_ImageRef] | None = None) -> dict: |
There was a problem hiding this comment.
_call_claude_cli()
fans out to 9 callees (efferent coupling); 31 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return args | ||
|
|
||
|
|
||
| def _call_openai_cli(user_message: str, max_tokens: int = 8192, *, deep_mode: bool = False, images: list[_ImageRef] | None = None) -> dict: |
There was a problem hiding this comment.
_call_openai_cli()
fans out to 9 callees (efferent coupling); 11 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
Re the five "extract_corpus_parallel truncated openai-cli concurrency guard likely broken" ( The guard serializes by default and honors the opt-in, exactly as documented. "openai-cli concurrency not forced to 1 in code despite docs" ( "MCP server disabling fails open" ( "Untrusted corpus text is sent to an agentic CLI that can run tools" — true, and inherent to the CLI-backend category: it is the same threat model as the shipped "openai-cli runs an LLM agent with inherited cwd and environment" ( |
1 participant
Replaces #2981, which is the same feature but was opened from a branch that also carried unrelated work from our fork — 11 files and 17 commits. That was my mistake in how I cut the branch. This branch is cut from
v8and carries only this feature. #2981 is being closed with a pointer here.What it adds
--backend openai-cliruns semantic extraction and community labeling throughcodex exec, the locally authenticated OpenAI Codex CLI, so the work rides a ChatGPT OAuth subscription instead of a meteredOPENAI_API_KEY.It is deliberately the same shape as the existing
claude-clibackend: zero pricing (subscription usage is not metered API spend), forced-serial calls unless opted out, and a credential gate that accepts a present CLI in place of an API key.GRAPHIFY_OPENAI_CLI_MODEL(defaultgpt-5.6-sol) andGRAPHIFY_OPENAI_CLI_EFFORT(defaultultra) configure it.GRAPHIFY_OPENAI_CLI_PARALLEL=1opts out of serial.Four things that took measurement to get right
These are in the code with comments, but worth stating because a reviewer would reasonably wonder why the simpler version is not there:
MAX_ARG_STRLEN). Real extraction chunks reach 240-306 KB, so passing the prompt as an argument fails withE2BIGon ordinary input, not on edge cases.mcp_servers.<name>.enabled=falseoverride. A blanketmcp_servers={}does not work: Codex deep-merges its config, so the empty key merges away and every configured server still launches. Measured cost of the naive version on our box: four servers of ~152 MB each, spawned on everycodex execcall.Token usage is read from the last
turn.completedJSONL event, so cached input is not double-counted.Diff
5 files:
graphify/llm.py(+286/-1),tests/test_openai_cli_backend.py(+219),graphify/cli.py(+11),graphify/__main__.py(+2/-2, help text),CHANGELOG.md(+4). 15 tests pass.One honest caveat
This has been running in our fork against real extraction loads, but only against
gpt-5.6-solthrough one person's Codex install. If you want it behind a documented-but-unadvertised flag until more people have exercised it, that is a reasonable call.