Add Vertex AI backend for orgs that disallow raw API keys - #3083
Add Vertex AI backend for orgs that disallow raw API keys#3083eliorerz wants to merge 4 commits into
Conversation
The existing "gemini" backend only talks to the API-key-only Generative Language API, which some GCP orgs disallow outright via policy. Adds a "vertex" backend using google-genai's Vertex AI mode, authenticated via Application Default Credentials (service account, Workload Identity Federation, or `gcloud auth application-default login`) instead of a static key -- no env_key/env_keys, following the same keyless pattern already used for the "bedrock" backend's AWS credential chain. Configured via GOOGLE_CLOUD_PROJECT (required) and GOOGLE_CLOUD_LOCATION (defaults to us-central1), with GRAPHIFY_VERTEX_MODEL to override the default gemini-2.5-flash. Thinking is disabled by default (thinking_budget=0) -- confirmed live that a small max_output_tokens can otherwise be silently consumed entirely by (billed, never-returned) thinking tokens before any extraction JSON is emitted, and thinking traces carry no value for a fixed-schema extraction task anyway. Verified end-to-end against a real GCP project: auto-detection via GOOGLE_CLOUD_PROJECT, the plain-text call path, and the full JSON extraction path (including token accounting) all confirmed working.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a vertex backend that calls Vertex AI's Gemini models through google-genai with Application Default Credentials, requiring only GOOGLE_CLOUD_PROJECT (location defaults to us-central1) so orgs that forbid raw API keys can extract without one. Wires it through extract_files_direct and _call_llm, exempts it from the API-key requirement alongside bedrock/claude-cli, and adds it to detect_backend as a fallback ranked below a static Gemini key. Defaults thinking_budget to 0 since thinking tokens are billed but never surface in the extraction JSON and can exhaust the output budget, and treats a hollow response as truncation so adaptive retry bisects the chunk.
Worth a look
- Vertex autodetection shadows explicit Ollama configuration —
graphify/llm.py:3252· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 770 functions depend on the 178 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 63 callers, 21 callees - new:
build_merge()— 46 callers, 14 callees - new:
extract_files_direct()— 17 callers, 21 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
_call_claude_cli()— 31 callers, 9 callees - new:
dispatch_command()— 2 callers, 122 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - new:
_call_llm()— 11 callers, 19 callees - …and 16 more — each is listed as a finding
Verification — 770 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 491 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_call\_llm.
The verifier did not have enough to check \_call\_llm, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in detect\_backend (not a proof).
The verifier ran both versions of detect\_backend on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract\_files\_direct.
The verifier did not have enough to check extract\_files\_direct, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_format\_backend\_env\_keys.
The verifier did not have enough to check \_format\_backend\_env\_keys, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly KeyError — names the real obstacle, not a sampling gap)
· 2 grounded finding(s) anchored inline below; 22 more finding(s) on lines outside this diff (see the check run).
| return result | ||
|
|
||
|
|
||
| def extract_files_direct( |
There was a problem hiding this comment.
extract_files_direct()
fans out to 21 callees (efferent coupling); 17 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @@ -3103,7 +3230,8 @@ def _validate_ollama_base_url(url: str, *, warn: bool = True) -> None: | |||
| def detect_backend() -> str | None: | |||
There was a problem hiding this comment.
detect_backend()
16 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
GOOGLE_CLOUD_PROJECT is often set globally for other GCP tools (gcloud, terraform, etc.), not specifically for graphify. An explicit Ollama configuration (OLLAMA_BASE_URL/OLLAMA_HOST) should not be shadowed by an ambient GCP environment variable. Fixes the finding: 'Vertex autodetection shadows explicit Ollama configuration'
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a vertex backend that reaches Gemini models through Vertex AI over Application Default Credentials, so orgs whose policy forbids raw API keys can run graphify with a working gcloud/service-account/WIF environment instead. It authenticates keylessly like bedrock (needs GOOGLE_CLOUD_PROJECT, defaults location to us-central1), zeroes the thinking budget by default to stop reasoning tokens silently eating the output budget, and treats a hollow response as truncation so adaptive retry can bisect the chunk. detect_backend picks it up only when GOOGLE_CLOUD_PROJECT is set, ranked after Ollama and all paid keys so an ambient GCP env var never shadows an explicit or paid backend.
Worth a look
- Vertex direct extraction leaks google-genai clients —
graphify/llm.py:1954· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Vertex missing-dependency hint is bypassed —
graphify/llm.py:1955· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Vertex helper call path leaks google-genai clients —
graphify/llm.py:3071· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Ambient GOOGLE_CLOUD_PROJECT silently selects cloud Vertex backend —
graphify/llm.py:3262· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- detect_backend() now returns 'vertex' when GOOGLE_CLOUD_PROJECT is ambiently set, silently changing selection —
graphify/llm.py:3266· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 770 functions depend on the 178 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 63 callers, 21 callees - new:
build_merge()— 46 callers, 14 callees - new:
extract_files_direct()— 17 callers, 21 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
_call_claude_cli()— 31 callers, 9 callees - new:
dispatch_command()— 2 callers, 122 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - new:
_call_llm()— 11 callers, 19 callees - …and 16 more — each is listed as a finding
Verification — 770 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 491 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify \_call\_llm.
The verifier did not have enough to check \_call\_llm, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in detect\_backend (not a proof).
The verifier ran both versions of detect\_backend on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract\_files\_direct.
The verifier did not have enough to check extract\_files\_direct, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_format\_backend\_env\_keys.
The verifier did not have enough to check \_format\_backend\_env\_keys, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly KeyError — names the real obstacle, not a sampling gap)
· 2 grounded finding(s) anchored inline below; 22 more finding(s) on lines outside this diff (see the check run).
| return result | ||
|
|
||
|
|
||
| def extract_files_direct( |
There was a problem hiding this comment.
extract_files_direct()
fans out to 21 callees (efferent coupling); 17 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @@ -3103,13 +3230,15 @@ def _validate_ollama_base_url(url: str, *, warn: bool = True) -> None: | |||
| def detect_backend() -> str | None: | |||
There was a problem hiding this comment.
detect_backend()
16 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
- Vertex AI backend using Application Default Credentials - No API key required (service account/Workload Identity Federation) - Fixes Ollama shadowing issue PR: Graphify-Labs#3083 (upstream)
…tion (#498) ## Summary Enable semantic extraction (docs, images, etc.) in the graphify brain refresh workflow by integrating Vertex AI backend with Workload Identity Federation authentication. ## Changes ### 1. Workload Identity Federation Authentication - Added `google-github-actions/auth@v2` step to authenticate to `osac-ci` GCP project - Added `id-token: write` permission for OIDC token generation - Service account: `osac-ci@osac-ci.iam.gserviceaccount.com` - WIF provider: `projects/1008600636152/locations/global/workloadIdentityPools/github-actions-pool/providers/github-actions-provider` ### 2. Vertex AI Backend Configuration - Set `GOOGLE_CLOUD_PROJECT=osac-ci` - Set `GOOGLE_CLOUD_LOCATION=us-central1` - Backend auto-detection works via these env vars (no `--backend vertex` flag needed) ### 3. Install graphify from Fork (Temporary) **Before:** `pip install --user "graphifyy[sql]==0.9.41"` **After:** `pip install --user "git+https://github.com/eliorerz/graphify@v8#egg=graphifyy[sql,vertex]"` This is temporary until upstream PRs merge: - Graphify-Labs/graphify#3083 (Vertex AI backend) - Graphify-Labs/graphify#3087 (ci-select + fixes) Once merged and a new `graphifyy` release is published, revert to: `pip install --user "graphifyy[sql,vertex]==${GRAPHIFY_VERSION}"` ### 4. Enable Semantic Extraction - **Removed** `--code-only` flag from full extraction (`graphify extract .`) - Now extracts semantic content from docs, images, markdown, etc. via Gemini 2.5 Flash - Incremental updates (`graphify update .`) unchanged (still code-only) ## Rationale Red Hat's GCP org policy **disallows raw API keys entirely** (confirmed live against the osac-ci project). This blocks the existing `gemini` backend (requires `GEMINI_API_KEY`). Vertex AI uses **Application Default Credentials** (service account via Workload Identity Federation) — no static secrets, aligns with security policy. ## Infrastructure Already Deployed - ✅ GCP project: `osac-ci` (project number `1008600636152`) - ✅ Service account with `roles/aiplatform.user` - ✅ WIF pool/provider configured, org-scoped to `osac-project` repos, main branch only - ✅ Vertex backend tested live end-to-end with real extraction ## Testing - [x] YAML validates (`python3 -c "import yaml; yaml.safe_load(...)"`) - [ ] Will test on first scheduled run or manual workflow_dispatch trigger - [ ] Confirm graph.json bundle includes semantic nodes from docs/images ## Rollback Plan If issues occur: 1. Revert this PR 2. Falls back to code-only extraction (AST-only graph, no semantic content) 3. No data loss — graph.json just won't have doc/image nodes <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Improved automated knowledge refresh reliability with more secure cloud authentication. * Refreshes now perform complete content extraction when existing semantic data is incomplete, while continuing to use incremental updates when appropriate. * Enhanced semantic processing delivers more complete and meaningful results. * **Documentation** * Clarified that knowledge reflects committed project content and may not reflect current repository or workflow status. * Added guidance to verify live repository and workflow details directly. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Vertex backend failed at runtime with 'backend vertex requires
GOOGLE_CLOUD_PROJECT to be set' even when GOOGLE_CLOUD_PROJECT was
genuinely set in the environment (confirmed in osac run 32878010795).
Root cause: cli.py lines 3497-3535 pre-flight check for backends
without API keys has special-case exemptions for ollama (localhost),
bedrock (AWS env vars), and claude-cli (binary check), but no
exemption for vertex. Since _get_backend_api_key('vertex') always
returns empty (vertex has no env_key in BACKENDS, uses ADC instead),
allow_no_key stayed False and triggered the error regardless of
GOOGLE_CLOUD_PROJECT being set.
Fix: add vertex branch mirroring bedrock pattern, checking for
GOOGLE_CLOUD_PROJECT env var (same check _vertex_client() does).
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 4 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a vertex LLM backend that talks to Vertex AI Gemini models through google-genai with Application Default Credentials, so orgs whose policy forbids raw API keys can run extraction with just GOOGLE_CLOUD_PROJECT (and optional GOOGLE_CLOUD_LOCATION) set. Wires it through _call_vertex/_call_llm, key-optional validation, and the "no backend" error message, defaulting to gemini-2.5-flash with thinking_budget=0 so billed thinking tokens can't silently drain the output budget, and treating hollow responses as truncation to trigger adaptive retry. Slots vertex last in detect_backend priority — after Ollama and all paid keys — so an ambient GOOGLE_CLOUD_PROJECT never shadows an explicit local or paid backend.
Worth a look
- detect_backend now returns a backend without any API key configured —
graphify/llm.py:3230· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Ambient GOOGLE_CLOUD_PROJECT auto-selects Vertex backend and can send repo data to GCP without explicit graphify opt-in —
graphify/llm.py:3262· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Vertex auto-detection shadows configured custom providers —
graphify/llm.py:3262· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Whitespace-only GOOGLE_CLOUD_PROJECT is detected as Vertex but later rejected —
graphify/llm.py:3262· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 854 functions depend on the 228 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 63 callers, 21 callees - new:
build_merge()— 46 callers, 14 callees - new:
extract_files_direct()— 17 callers, 21 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
_call_claude_cli()— 31 callers, 9 callees - new:
dispatch_command()— 2 callers, 122 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - new:
_call_llm()— 11 callers, 19 callees - …and 18 more — each is listed as a finding
Verification — 854 functions in the blast radius were not formally verified this run (proofs are advisory here).
Health delta baseline: last indexed commit 43d54ac (diverged from this PR's base — delta is approximate).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 743 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_call\_llm.
The verifier did not have enough to check \_call\_llm, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in detect\_backend (not a proof).
The verifier ran both versions of detect\_backend on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract\_files\_direct.
The verifier did not have enough to check extract\_files\_direct, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_format\_backend\_env\_keys.
The verifier did not have enough to check \_format\_backend\_env\_keys, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly KeyError — names the real obstacle, not a sampling gap)
· 2 grounded finding(s) anchored inline below; 24 more finding(s) on lines outside this diff (see the check run).
| return result | ||
|
|
||
|
|
||
| def extract_files_direct( |
There was a problem hiding this comment.
extract_files_direct()
fans out to 21 callees (efferent coupling); 17 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| @@ -3103,13 +3230,15 @@ def _validate_ollama_base_url(url: str, *, warn: bool = True) -> None: | |||
| def detect_backend() -> str | None: | |||
There was a problem hiding this comment.
detect_backend()
16 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Fixes graphify-brain-refresh workflow failure (run 32878010795) where vertex backend rejected GOOGLE_CLOUD_PROJECT even when set. **Problem**: graphify cli.py's pre-flight check had no allow-no-key exemption for vertex backend (unlike ollama/bedrock/claude-cli), so it always rejected vertex regardless of GOOGLE_CLOUD_PROJECT being present. **Fix**: Install graphify from eliorerz/graphify@1a122e1 (feat/vertex-ai-backend branch) which adds the missing vertex check. Will revert to PyPI install once the next graphify release includes this fix. **Related**: - graphify fix: eliorerz/graphify#4 - Upstream PR: Graphify-Labs/graphify#3083 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the automated brain refresh process to use a pinned Graphify source revision. * Improved consistency and reproducibility of refresh runs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Every other backend in this module wires _resolve_max_retries() into its SDK client so a burst of 429s during a large parallel run gets absorbed instead of dropping the chunk -- google-genai needs this set explicitly via http_options.retry_options. Confirmed live that an unconfigured client does NOT retry 429s at all (every field of an empty HttpRetryOptions() defaults to None), which is exactly why a large first-run corpus (~1100 docs across the osac mono-repo) surfaced "Resource exhausted" as hard chunk failures instead of transient delays (9 of 16 semantic chunks failed outright, producing a smaller graph that graphify's own shrink guard correctly refused to publish).
1 participant
Summary
geminibackend only talks to the API-key-only Generative Language API. Some GCP orgs disallow raw API keys entirely via policy (confirmed live against a real org policy), leaving no way to use Gemini through graphify at all.vertexbackend usinggoogle-genai's Vertex AI mode, authenticated via Application Default Credentials (service account, Workload Identity Federation, orgcloud auth application-default login) — no API key, following the same keyless pattern already used forbedrock's AWS credential chain.GOOGLE_CLOUD_PROJECT(required) andGOOGLE_CLOUD_LOCATION(defaults tous-central1), withGRAPHIFY_VERTEX_MODELto override the defaultgemini-2.5-flash.thinking_budget=0) — confirmed live that a smallmax_output_tokenscan otherwise be silently consumed entirely by billed, never-returned thinking tokens before any extraction JSON is emitted. Thinking traces carry no value for a fixed-schema extraction task.extract_files_direct's JSON extraction and_call_llm's plain-text path),detect_backend()auto-detection, vision/image support, and thevertexpip extra (google-genai).Test plan
python3 -m py_compile graphify/llm.pydetect_backend()correctly returnsvertexwhen onlyGOOGLE_CLOUD_PROJECTis set_call_llm(..., backend="vertex")) returns correct text + usage accountingextract_files_direct(..., backend="vertex")) against a real file produces correct nodes/edges/token counts, no truncationGOOGLE_CLOUD_PROJECTraises a clearValueErrorinstead of an opaque SDK error