Feat(model support): ideogram4 support

[Pfannkuchensack]

Summary

Adds first-class Ideogram 4 (text-to-image) support to InvokeAI — a new open-weight 9.3B single-stream DiT with a Qwen3-VL-8B text encoder and flow-matching sampler.

The defining trait of this model is that it is trained on a structured JSON prompt that describes the scene as a list of regions, each with a bounding box ([y_min, x_min, y_max, x_max], normalized 0–1000, origin top-left) and a text description. To stay on the model's trained distribution, InvokeAI always feeds it structured JSON — even a plain prompt is wrapped into a one-element caption (see the safety-filter note below).

The headline feature here is that this JSON is auto-assembled from the existing Canvas Regional Guidance layers: the global prompt becomes the overall description, and each enabled region contributes one element (its drawn rect → bbox, its prompt → description). Users can also paste raw JSON to drive the model directly (passed through unchanged). Assembly happens at generation time in a dedicated ideogram4_caption_builder node, so dynamic prompts / prompt batching that vary the global prompt are folded into the encoded caption, and each generated image records the exact JSON it was given.

⚠️ Built-in safety filter (baked into the weights): Ideogram 4 ships its own content filter inside the model — it is not Invoke's NSFW checker and cannot be disabled from Invoke. Empirically it also false-positives on "degenerate" captions: an empty compositional_deconstruction.elements list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high-level description, returns an "Image blocked by safety filter" placeholder. The caption assembly is built to avoid this: it never emits bare plain text, and when no regions are drawn it synthesizes one element describing the whole scene with a partial bbox [100,100,900,900]. This is documented in models.mdx.

Why the backend only ever sees one prompt string (no mask conditioning): Ideogram 4 does not use spatial attention masks for regions (unlike FLUX/SDXL/Z-Image regional guidance) — the region boxes are encoded as text inside the single JSON string fed to Qwen3-VL. So no mask-conditioning code is touched; the "regional" feature is purely caption assembly.

How

Backend (invokeai/backend/ideogram4/, vendored from the Apache-2.0 reference, copyright headers retained):

• DiT (modeling_ideogram4.py), FLUX2-style KL VAE (autoencoder.py + latent_norm.py), logit-normal flow-match scheduler + presets (scheduler.py, sampler_configs.py), nf4/fp8 quantized loading, and InvokeAI-side denoise.py / text_encoding.py / sampling_utils.py / caption.py.
• Dual-branch asymmetric CFG: positive runs the conditional transformer over [text]+[image] tokens, negative runs the unconditional transformer over image-only tokens with zeroed LLM features (v = gw·pos + (1−gw)·neg). ⇒ no negative prompt. A transformer_pair.py wrapper keeps both transformers co-resident through the loop so the cache doesn't swap them every step (nf4 ≈ 10 GB resident during denoise; fits 24 GB). The guidance schedule caps its polish tail at num_steps-1 so at least one main (guided) step always remains, and requires steps >= 2.
• Step previews: the denoise loop emits a low-res progress image each step (unpatchify + FLUX.2 latent→RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE), so the forming image is visible during generation like the other denoise nodes.
• Non-fp8 text-encoder loading validates the state dict (unexpected keys raise, missing keys warn) instead of silently accepting a partial load.
• Model-manager registration: new BaseModelType.Ideogram4, a Qwen3-VL text-encoder type, config detector + diffusers config, and a loader mirroring Z-Image.
• Five invocations: ideogram4_model_loader, ideogram4_text_encoder, ideogram4_caption_builder (Python port of the JSON assembly), ideogram4_denoise, ideogram4_latents_to_image; new Ideogram4ConditioningInfo + field/output; ideogram4_txt2img generation mode; a declared ideogram4_caption metadata field.

Frontend:

• buildIdeogram4Prompt.ts — collectIdeogram4PromptInputs: gathers the raw inputs (global prompt, enabled regions → {prompt, bbox} clamped/rounded to 0–1000, color palette) for the caption builder node. The JSON assembly itself now lives in the backend node (single source of truth, Python-tested); raw-JSON passthrough and stable key order are handled there.
• buildIdeogram4Graph.ts — text2img-only graph builder + enqueue wiring: prompt (string) → ideogram4_caption_builder → text_encoder. The real prompt node carries the global prompt, so the linear-UI batch injector (dynamic prompts / prompt batching) writes expansions into it and they flow through the caption builder into the encoder — the earlier "decoy string node" hack is removed. The builder's output is wired to the ideogram4_caption metadata field via an edge, so each batched image records its actual caption.
• Params + UI: a Sampler Preset combobox (Quality 48 / Default 20 / Turbo 12, localized) as the primary control, plus Advanced overrides that actually apply to this model — Steps (min 2), Guidance Scale, Schedule Shift (mu) and a Color Palette picker. The irrelevant Advanced controls (VAE, CLIP Skip, CFG Rescale, Seamless, Color Compensation) are hidden for Ideogram 4.
• Metadata recall for all of the above (guarded to only recall onto an Ideogram 4 model).

Dependencies: bumps transformers to >=5.5,<5.6 (Qwen3-VL landed in 4.57; the encoder needs it) and compel to >=2.4.0,<3, with the necessary adaptations to the FLUX / Z-Image loaders, the safety checker, the HF metadata fetcher and model_util.

Out of scope (v1):

• img2img / inpaint / outpaint are not supported — Ideogram 4 is text-to-image only here (the graph asserts txt2img). No latent/image init path exists in this PR.
• ControlNet / IP-Adapter / LoRA.
• The optional local "Magic Prompt" plain-text→JSON expander (parked — see Merge Plan).

Related Issues / Discussions

• Overlaps the transformers 5.x bump tracked in PR feat - Migrate to Transformers 5.5.4 #9248 — this branch carries the same bump (>=5.5,<5.6) plus the cross-model adaptations it requires.

QA Instructions

Requires the gated weights (ideogram-ai/ideogram-4-nf4 — nf4 is the 24 GB path, CUDA/bitsandbytes only) plus the Qwen3-VL encoder + VAE sub-dependencies.

1. Install & select an Ideogram 4 model; open the Canvas/Generate tab. Confirm the model shows under its own group, dimensions default to 1024×1024 (multiples of 16), and the Generation settings show the Sampler Preset control instead of Scheduler/CFG.
2. Regions → JSON: type an overall description, add 1–2 Regional Guidance layers each with a prompt + a drawn box, and Invoke. In the result's metadata, confirm the ideogram4_caption row shows the assembled JSON with the correct key order and elements[*].bbox (0–1000, [y_min, x_min, y_max, x_max]) matching where you drew the boxes, and that element placement in the image roughly matches.
3. Raw-JSON passthrough: paste a hand-written JSON object into the prompt box → it is sent unchanged (no region wrapping, palette ignored).
4. Plain text (now wrapped to JSON): with no regions and a plain prompt, Invoke and confirm it renders (is not blocked). Check the ideogram4_caption metadata: the prompt was wrapped into a one-element caption with a partial bbox [100,100,900,900] — this is the safety-filter workaround. Dynamic prompts / prompt batching still expand (they now flow through the real prompt node → caption builder).
5. Sampler presets: Quality / Default / Turbo produce the expected step counts (min 2).
6. Advanced overrides: Steps / Guidance Scale / mu show the active preset's value as the "auto" default and can be overridden + reset; Color Palette swatches inject style_description.color_palette (auto-build mode only — ignored for raw JSON).
7. Step previews: during generation, confirm a (low-res) progress image is shown each step.
8. Metadata recall: load an Ideogram 4 image and recall — preset, overrides and palette restore (and are guarded to only recall onto an Ideogram 4 model).

Frontend gates (from invokeai/frontend/web/): pnpm lint and pnpm test:no-watch (includes the caption/graph tests). tsc clean. Backend: tests/backend/ideogram4/test_caption.py covers the assembly (raw-JSON passthrough, region mapping, and the no-region default element).

Merge Plan

• Large PR + dependency bump. This raises transformers to >=5.5,<5.6 and compel to >=2.4.0,<3, which touches many models (FLUX, Z-Image, safety checker, HF metadata fetch). Coordinate with / sequence after PR feat - Migrate to Transformers 5.5.4 #9248 (the transformers 5.x bump) to avoid a double-bump conflict, and time it to not collide with a pending release. Broad regression QA across existing model types is warranted, not just Ideogram 4.
• Follow-ups (not in this PR): GGUF loader for the custom DiT (more VRAM headroom), starter_models.py entries, and the optional local Magic Prompt node (blocked on the upstream system-prompts PR feat: add System Prompts library for Expand Prompt button #9152, whose migration_32 collides with main and must be renumbered to 33 first).

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable) — backend unit tests for the JSON caption assembly (test_caption.py) + frontend caption/graph tests
• ❗Changes to a redux slice have a corresponding migration — new params fields use zod defaults; confirm if a migration is needed
• Documentation added / updated (if applicable) — models.mdx documents the model's built-in safety filter
• Updated What's New copy (if doing a release after this PR)

[JPPhoto]

Some findings from the latest group of changes:

• invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:114: structured Ideogram prompts return positive_prompt_decoy as the graph builder's positivePrompt, while the actual ideogram4_text_encoder.prompt input is connected to ideogram4_prompt whose value was assembled before batching at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:57 and invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:100; prepareLinearUIBatch() only injects dynamic prompt expansions into the returned node at invokeai/frontend/web/src/features/nodes/util/graph/buildLinearBatchConfig.ts:74. Trigger: Ideogram 4 generation with a color palette, regional guidance, or raw JSON prompt, plus dynamic prompts or prompt batching. Consequence: the encoded prompt stays at the original static structured caption while metadata positive_prompt changes through the decoy, so batched images can all use the same text conditioning. Test: build an Ideogram graph with ideogram4ColorPalette non-empty and dynamicPrompts.prompts containing two distinct prompts, run prepareLinearUIBatch(), and assert the batch data updates the node connected to ideogram4_text_encoder.prompt or that each structured caption contains the expanded prompt.

• invokeai/backend/model_manager/load/model_loaders/ideogram4.py:178: the non-fp8 text encoder path calls model.load_state_dict(sd, strict=False, assign=True) and discards the returned missing and unexpected keys, unlike the fp8 helper which raises on unexpected keys and warns or raises on missing keys. Trigger: an nf4 or unquantized Ideogram text encoder checkpoint with a typo, extra component key, missing tied-weight exception other than the expected one, or mismatched architecture. Consequence: model import can appear successful with ignored checkpoint mismatch and then fail later during encoding or produce invalid conditioning. Test: monkeypatch AutoModel.from_config() with a tiny model and _load_local_state_dict() with one unexpected key and one non-tolerated missing key, then assert _load_text_encoder() raises or explicitly validates only the known tolerated missing keys.

• invokeai/frontend/web/src/features/parameters/components/Core/ParamIdeogram4SamplerPreset.tsx:11: the Ideogram sampler preset dropdown uses hardcoded English option labels (Quality (48 steps), Default (20 steps), Turbo (12 steps)) even though the component already uses useTranslation() for the field label and invokeai/frontend/web/public/locales/en.json:1635 only adds the generic samplerPreset label. Trigger: running the UI in any non-English locale. Consequence: newly added visible Ideogram controls remain partially untranslated. Test: render ParamIdeogram4SamplerPreset with a non-English i18n resource that translates the three preset labels and assert the combobox options come from translation keys rather than literal English strings.

• invokeai/app/invocations/ideogram4_denoise.py:41: _effective_guidance_schedule() allows num_steps=1 through steps validation at invokeai/app/invocations/ideogram4_denoise.py:72, but polish_count = min(num_steps, ...) makes main_count zero despite the docstring promising at least one polish and one main step. Trigger: user sets the advanced Ideogram step override to 1, optionally with a guidance override. Consequence: the generated schedule is only the polish weight, so the user-specified main guidance_scale is ignored and the denoise behavior contradicts the invocation contract. Test: unit test _effective_guidance_schedule(PRESETS["V4_QUALITY_48"].guidance_schedule, 48, 1, 12.0) and assert the invocation either rejects one-step overrides or returns behavior that reflects the documented guidance override.

[JPPhoto]

While only the first is truly a merge blocker, I would like to see both addressed:

• invokeai/frontend/web/src/features/controlLayers/store/types.ts:826: ideogram4Steps still accepts 1, and Ideogram4Steps.parse() also recalls ideogram4_steps: 1 at invokeai/frontend/web/src/features/metadata/parsing.tsx:942, but the backend invocation now requires steps >= 2 at invokeai/app/invocations/ideogram4_denoise.py:85. A recalled image metadata value or rehydrated client state with ideogram4_steps set to 1 is dispatched without clamping at invokeai/frontend/web/src/features/controlLayers/store/paramsSlice.ts:105 and then passed directly into the graph at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:95. Consequence: the UI can build/enqueue an Ideogram graph that violates the backend schema instead of rejecting or normalizing the stale value before enqueue. Test: recall Ideogram metadata or rehydrate params with ideogram4_steps: 1, build an Ideogram graph, and assert the client rejects/clamps it before enqueue or that parsing refuses the value.

• invokeai/app/invocations/ideogram4_caption.py:16: Ideogram4Region.bbox is declared as Optional[list[int]] with no length or value constraints, while the description and model contract require exactly four normalized coordinates in the 0..1000 range. build_ideogram4_caption() forwards any provided bbox unchanged into the structured JSON at invokeai/backend/ideogram4/caption.py:59, so a workflow/API caller can provide values like [0, 0, 1000], [0, 0, 1000, 1000, 7], or negative/out-of-range coordinates. Consequence: the new public caption-builder node can emit malformed structured prompts that the text encoder/model may misinterpret, ignore, or apply to the wrong region. Test: validate or execute Ideogram4CaptionBuilderInvocation with short, long, and out-of-range bbox arrays and assert validation fails instead of serializing them into the caption.

[JPPhoto]

A few more issues...

Blockers

• invokeai/frontend/web/src/features/queue/hooks/useEnqueueCanvas.ts:73: the canvas enqueue path routes every Ideogram canvas job to buildIdeogram4Graph(), but buildIdeogram4Graph() asserts generationMode === 'txt2img' at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:40. The compositor returns img2img, outpaint, or inpaint when visible raster/inpaint content exists at invokeai/frontend/web/src/features/controlLayers/konva/CanvasCompositorModule.ts:711, while getRasterLayerWarnings() and getInpaintMaskWarnings() still return no problems at invokeai/frontend/web/src/features/controlLayers/store/validators.ts:241 and invokeai/frontend/web/src/features/controlLayers/store/validators.ts:252. Trigger: select an Ideogram 4 model on Canvas with an enabled raster layer or inpaint mask. Consequence: readiness can allow an unsupported canvas mode and enqueue only fails during graph build. Test: create canvas state with an Ideogram model plus an opaque raster layer and with an inpaint mask, then assert canvas readiness reports an unsupported-mode reason before enqueue calls the graph builder.

• invokeai/frontend/web/src/features/queue/store/readiness.ts:562: canvas readiness has model-specific 16/32-pixel bbox checks for Flux, Flux2, CogView4, and Qwen Image through invokeai/frontend/web/src/features/queue/store/readiness.ts:762, but no Ideogram 4 branch. getOriginalAndScaledSizesForTextToImage() uses raw bbox dimensions when scaleMethod === 'none' at invokeai/frontend/web/src/features/nodes/util/graph/graphBuilderUtils.ts:132, buildIdeogram4Graph() passes those dimensions into ideogram4_denoise at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:116, and the backend requires both dimensions to be multiples of 16 at invokeai/app/invocations/ideogram4_denoise.py:81. Trigger: Ideogram 4 on Canvas with bbox scaling disabled and a bbox like 1025x1024. Consequence: readiness can allow a graph whose denoise node fails backend validation. Test: build canvas readiness for Ideogram 4 with bbox.scaleMethod set to none and a non-16-multiple width or height, and assert it blocks with the same incompatible-bbox reason used for other 16-grid models.

• invokeai/frontend/web/src/features/controlLayers/store/validators.ts:50: getRegionalGuidanceWarnings() has no Ideogram 4 branch, but collectIdeogram4PromptInputs() only reads region.positivePrompt and bbox at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Prompt.ts:83. Trigger: an enabled Regional Guidance layer under Ideogram 4 with only a negative prompt, auto-negative, or reference image. Consequence: readiness can allow the layer with no warning, while the Ideogram graph silently drops those inputs and may omit the region entirely. Test: create Ideogram canvas state with regional guidance containing only a negative prompt and/or reference image, assert readiness reports unsupported ignored inputs, and assert graph construction does not silently discard a layer without a warning.

Follow-up PR

• invokeai/app/invocations/ideogram4_caption.py:35: _validate_bbox() enforces length and 0..1000 range, but does not enforce y_min <= y_max and x_min <= x_max; build_ideogram4_caption() forwards the bbox verbatim at invokeai/backend/ideogram4/caption.py:59. Trigger: a custom workflow/API call supplies [900, 900, 100, 100]. Consequence: the caption builder emits an inverted region bbox that satisfies validation but violates the model prompt contract. Test: instantiate Ideogram4Region(prompt="x", bbox=[900, 900, 100, 100]) and assert validation rejects it.

• invokeai/frontend/web/src/features/settingsAccordions/components/AdvancedSettingsAccordion/AdvancedSettingsAccordion.tsx:66: the advanced accordion badge selector only distinguishes Flux and Flux2, so Ideogram 4 falls into the generic !isFlux2 branch at line 77 and can show stale VAE, clip skip, CFG rescale, or seamless badges even though those controls are hidden for Ideogram at lines 116 and 122. Trigger: switch from an SD/SDXL model with VAE/clip skip/rescale/seamless settings to Ideogram 4. Consequence: the collapsed Advanced accordion advertises settings that no longer apply to the selected model. Test: render AdvancedSettingsAccordion with an Ideogram model and stale VAE/clipSkip/cfgRescale/seamless params, and assert only Ideogram-relevant badges appear or no stale badges appear.

• invokeai/frontend/web/src/services/api/schema.ts:14045: Ideogram4Region.bbox is generated as number[] | null with no length or range constraints, even though the backend validator at invokeai/app/invocations/ideogram4_caption.py:24 rejects anything other than exactly four coordinates in 0..1000. Trigger: a workflow/API client generated from OpenAPI can construct [0, 0, 1000] or [0, 0, 1000, 1001] without schema/type feedback. Consequence: clients discover the contract only through runtime invocation validation instead of the public schema. Test: regenerate OpenAPI/types after changing bbox to schema-expressible constraints and assert openapi.json includes minItems, maxItems, and item minimum/maximum for Ideogram4Region.bbox.

[JPPhoto]

Another one I can't test so hopefully you have done so!

Only one minor change if you can do it (or put it in a follow-up), but I am approving this work anyway:

• invokeai/backend/model_manager/load/model_loaders/ideogram4.py:168 and invokeai/backend/model_manager/load/model_loaders/ideogram4.py:179: _load_text_encoder() still allows missing text-encoder keys to proceed as warnings after building the model under accelerate.init_empty_weights(). Any missing non-tied parameter can remain on the meta device, so a bad or mismatched Ideogram text encoder may appear to load and then fail later during device movement or encoding instead of being rejected at load time. Test: monkeypatch AutoModel.from_config() with a tiny meta-built model and _load_local_state_dict() with one missing non-tied parameter for both fp8 and non-fp8 paths, then assert _load_text_encoder() raises or filters only explicitly tolerated tied-weight keys and asserts no meta tensors remain.