fix(ui): make multi-GPU viewer previews survive owner termination and queue lifecycle events#9389
Draft
lstein wants to merge 68 commits into
Draft
fix(ui): make multi-GPU viewer previews survive owner termination and queue lifecycle events#9389lstein wants to merge 68 commits into
lstein wants to merge 68 commits into
Conversation
Run one generation session per configured GPU concurrently, with a tiled progress preview. Multi-user isolation is unchanged. Backed by five seams: - Per-thread device context (TorchDevice.set/get/clear_session_device); choose_torch_device() consults it first, so all device-selecting call sites resolve to the calling worker's GPU with no per-node changes. - Per-device model caches: build_model_manager builds one ModelCache per generation device; ModelLoadService.ram_cache resolves by current thread device; ram_caches fans out clear/drop/shutdown. - Atomic concurrent dequeue: a dequeue lock makes select+claim atomic so concurrent workers never claim the same item (works on FIFO; round-robin from invoke-ai#9086 slots in later). - Worker pool: one _SessionWorker per device, each pinning torch.cuda.set_device and its session device, with its own runner and cancel event; cancellation routes via an {item_id -> worker} lookup. Single-device installs keep the exact legacy single-worker behavior. Profiling disabled when >1 worker. - New config `generation_devices`; unset = legacy single-worker mode. Frontend: the canvas staging area already tiles per queue item; the main ImageViewer now tracks progress per session and renders a tile grid (ProgressImageTiles) when more than one session is active. Also adds a lock to ObjectSerializerForwardCache for concurrent access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_model_load_device_routing mutated the process-wide get_config() singleton (device = "cuda:0") to exercise the per-thread cache routing, but never restored it. The leaked CUDA device was then picked up by a later test (test_model_load::test_loading) via choose_torch_device(), which crashed with "Torch not compiled with CUDA enabled" on the CUDA-less CI runner. Add an autouse fixture to save/restore device and clear any pinned session device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n_devices Regenerate openapi.json (make frontend-openapi) and the frontend schema.ts types (make frontend-typegen) so they include the new generation_devices config field, fixing the openapi-checks and typegen-checks CI jobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`make frontend-openapi` used a bare `python` from a different environment that emitted the CacheStats @DataClass docstring as a schema description. CI generates the schema via `uv run`, which does not, so openapi-checks failed on the diff. Regenerate with the uv-locked environment to drop the stray description while keeping the generation_devices field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o prevent meta-device corruption Parallel multi-GPU session workers could intermittently crash with "unrecognized device meta" (denoise) or "Cannot copy out of meta tensor; no data!" (l2i), because model loading relies on process-global, non-thread-safe monkey-patches. accelerate.init_empty_weights() (used directly by the loaders and implicitly by diffusers' default low_cpu_mem_usage=True in from_pretrained) swaps torch.nn.Module.register_parameter globally for the duration of a load, routing every newly-registered parameter to the meta device. The model cache's VRAM load/unload runs nn.Module.load_state_dict(assign=True), whose assign path does setattr -> __setattr__ -> register_parameter. When one worker's VRAM move overlapped another worker's from_pretrained, the move's real weights got hijacked onto meta and blew up on the next .to(device). Introduce MODEL_LOAD_LOCK, a write-preferring readers-writer lock: - write lock = model construction (_load_and_cache, load_model_from_path), exclusive. - read lock = VRAM load/unload (ModelCache.lock(), repair_required_tensors_on_device). VRAM transfers across GPUs still overlap each other; they only block while a construction holds the write lock. The lock is always acquired before any per-cache lock to keep a consistent order and avoid an AB-BA deadlock with the writer's make_room/put. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ions Image.open() is lazy: it reads the header but defers pixel decoding (and holds the file handle open) until the first .load()/.copy()/.convert(). The opened object was cached and the same object handed to every caller, so in multi-GPU parallel mode two session-processor worker threads could call .copy() on it concurrently and race on the shared file handle and decoder state. This surfaced as "broken data stream when reading image file" and "AssertionError: self.png is not None" during inpainting with batch >1. Force the decode (image.load()) before the object enters the cache so the cached object is safe for concurrent reads, and guard the cache structures (__cache / __cache_ids) with a lock since they are now mutated from multiple threads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generation progress bars (under the Invoke button and the Viewer tab) both read a single global $lastProgressEvent atom, which every session overwrites. With parallel multi-GPU sessions this made the bar jump back and forth between sessions. Track progress per queue item id and render one bar per in-flight session, stacked vertically, each removed as its session reaches a terminal state. - stores.ts: add $progressEvents (map keyed by item_id), $activeProgressEvents (sorted), and set/clear helpers. - setEventListeners.tsx: populate per-item progress on invocation_progress; clear per item on terminal status; clear all on connect/disconnect/queue cleared. - ProgressBar.tsx: render a vertical stack of bars (one per active session) with a single-bar fallback for the idle / model-loading window; add containerProps so dockview tabs can position the stack. - Dockview tab call sites: move positioning into containerProps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
$progressEvents is only referenced within stores.ts (via the $activeProgressEvents computed and the set/clear helpers), so exporting it tripped knip's unused-exports check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With 4 GPUs the stacked per-session progress bars grew past the bottom strip of the dockview tab and overlapped the "Viewer" label. Add a fitHeightPx prop: in fit mode the stack is capped to the available strip (10px below the ~40px tab's centered label) and the bars flex to share it, shrinking below their natural height only once they no longer fit. With 1-2 sessions the bars keep their familiar thin height; with 3+ they scale down to stay within the strip. The sidebar bar is unaffected and continues to stack at natural height (it has the vertical room). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fault generation_devices now accepts "auto" (the new default), which expands to every visible CUDA device — so multi-GPU parallel generation works out of the box without manually listing devices. On GPU-less systems "auto" resolves to the single cpu/mps device, preserving serial behavior. - config_default.py: type is now Union[Literal["auto"], list[str]], default "auto"; validator accepts "auto" or a list of device strings. - devices.py: add TorchDevice.get_generation_devices(), the single resolver that expands "auto", normalizes, and deduplicates. - session_processor / model_manager: both consumers use the resolver instead of iterating the raw config value (which would have iterated the characters of the "auto" string). - Regenerated docs/src/generated/settings.json. - Tests for the resolver (auto-with/without-CUDA, dedup, empty). An explicit single-device list (e.g. [cuda:0]) or an empty list opts out of parallelism. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a badges UI in the Generation section of the Settings dialog for choosing which devices `generation_devices` should use, modeled on the Log Namespaces toggle UI. Backend: - New `GET /api/v1/app/generation_device_options` endpoint listing the selectable devices (cuda:N with GPU names, or the sole mps/cpu fallback). - Add `generation_devices` to the runtime-config update allowlist with validation rejecting invalid device strings and explicit nulls. Frontend: - New SettingsGenerationDevices component with active/inactive badges. "Auto (all GPUs)" is exclusive; removing the last explicit device reverts to auto. Admin/multiuser gated; notes restart requirement. - Wire into the Generation section; regenerate schema; add en strings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the restart sentence into its own string and render it bold so users notice that device changes require restarting InvokeAI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render device badges as "cuda:0 (RTX 3090 #1)" so identical cards can be told apart. Strips the "NVIDIA GeForce" vendor prefix and adds a 1-based "#N" suffix only when multiple cards share a name. The full device name remains available as the badge tooltip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Help users track which CUDA device is processing each session: - Model-load log: "Loaded model ... onto cuda device #N in ..s" - Denoise progress bars: "Denoising (#N)" across all architectures (SD1.5/SDXL, FLUX, FLUX2, Z-Image, Anima, SD3, CogView4) - Progress preview circle: GPU number centered in the ring, via a new `device` field on InvocationProgressEvent (resolved from the worker's thread-local session device) - Session Queue: new "GPU #" column between STATUS and TIME, backed by a `device` column on session_queue (migration_32) recorded when a worker claims an item Adds TorchDevice.get_session_device_label()/get_session_device_index() helpers and a frontend getCudaDeviceIndex() parser (with tests). Shows the number on CUDA only; CPU/MPS show nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # invokeai/frontend/web/src/features/gallery/components/ImageViewer/CurrentImagePreview.tsx # invokeai/frontend/web/src/features/gallery/components/ImageViewer/context.tsx # invokeai/frontend/web/src/services/events/setEventListeners.tsx
Resolve migration_32 conflict: main's migration_32 (model_relationships FK repair) is kept, and the multi-gpu device-column migration is moved to migration_33 and registered after it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rcles - Startup log lists each generation device with its GPU number and id, e.g. "Using torch device: [AMD Radeon PRO W7900 #1 (cuda:0), ...]". Single-device setups keep the bare device name. - Canvas progress circles now show the CUDA device index in the center, matching the viewer panel. - Progress-circle tooltips show the device name and number on hover. - Both are hidden when only a single GPU is available. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… caches In multi-GPU mode the model manager builds one ModelCache per generation device, each with storage_device="cpu" and its own RAM-resident copy of every model. A model loaded on N GPUs therefore occupied N copies in RAM, and each cache sized itself against max_cache_ram_gb independently, so RAM use during the text/reference-image encoding phases skyrocketed and the system swapped — worst when two images rendered at once. This deduplicates the CPU-resident weights and makes RAM accounting global. - SharedCpuWeightsStore: process-/manager-global, refcounted store of one canonical CPU state_dict per model key. The first device to load a key registers its weights; subsequent devices adopt the canonical tensors and re-point their module's params at them (load_state_dict(assign=True)), freeing the duplicate. Weights live once in RAM regardless of GPU count; freed only when the last device releases. Per-device modules are kept (params are device-shuffled in place, so two GPUs need two modules), but their CPU-resident params alias the shared tensors. - RamBudget: single system-wide RAM authority. Splits RAM into shared (counted once via the store) and non-shared (per-instance). ModelCache eviction now runs against the global, deduplicated total and re-checks availability each iteration, since evicting a model another device still holds frees no RAM. build_model_manager wires one store + one budget into all device caches; the cap is max_cache_ram_gb as a true system-wide limit, else the sum of per-cache heuristics. Passing ram_budget=None preserves the prior local accounting. - LoRA/patch safety: direct LoRA patching did an in-place copy_ on the weight, which would corrupt the now-shared canonical tensor (and taint keep_ram_copy even with one GPU) when patching a CPU-resident weight. Switched to an out-of-place add (memory- equivalent) so the canonical tensor is never mutated; fixed the FluxControlLoRA expansion path to target the module's live parameter. Sidecar patching and FreeU/Seamless (which patch forward methods) were already safe. Validated on 2x AMD W7900 / ROCm: correct inference on both GPUs from one shared copy (full + partial load + Q8_0 GGUF quantized), concurrent load/unload without corruption, and LoRA isolation across devices. ~40 new tests; existing suites unchanged. Adds scripts/multigpu_ram_driver.py to drive concurrent dual-GPU generations via the queue API and measure peak RSS / leak drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(multi-GPU) With one session-processor worker per device, multiple queue items can be in_progress at once. cancel_by_batch_ids(), cancel_by_destination() and cancel_by_queue_id() excluded in_progress rows from their bulk UPDATE and then canceled only the single get_current() item (LIMIT 1), so on multi-GPU the other running items kept consuming a GPU and could still produce output after the user requested cancellation. Each running item must be canceled via _set_queue_item_status(), which emits the QueueItemStatusChangedEvent that the processor maps to the worker running that item_id and uses to set its cancel event. Add _cancel_in_progress_matching() to cancel every in-progress item matching the same filter (with user-id scoping preserved) and call it from all three bulk-cancel methods. The returned `canceled` count now includes canceled in-progress items. Adds regression tests that dequeue two items onto separate devices and assert every bulk cancel API moves all matching in_progress items to canceled and emits a cancel event for each (and that user-scoped cancel leaves another user's in-progress item running). Reported by JPPhoto in review of invoke-ai#9263. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vice guards, refcount leak) Fixes from the code review of PR invoke-ai#9263: - Cancellation could be silently lost around dequeue: the per-iteration worker.cancel_event.clear() ran AFTER dequeue + gc.collect() + logging, so a cancel arriving in that window was set by the status handler and then wiped. Move the clear to before dequeue, and after claiming an item re-check (cancel_event + a fresh DB status read via _is_queue_item_terminal) and skip running if it is already terminal, closing both race windows. The runner's stale queue_item.status check could not catch this. - delete_by_destination only stopped one in-progress item (get_current) before deleting all matching rows, leaving other GPU workers running (and then failing to update a deleted row). Cancel every matching in-progress item via _cancel_in_progress_matching first. - generation_devices validation: a bare non-"auto" string (e.g. "cuda:0") was iterated character-by-character; an empty list silently fell back to one device. Reject both with a clear message. - get_generation_devices now fails fast on a CUDA device that does not exist (index past device_count, or CUDA unavailable) instead of starting a worker that errors cryptically at first allocation. - Shared-weights wrappers: if the canonical re-point (load_state_dict assign=True) threw after acquire(), the reference was leaked (the wrapper never entered the cache). Compute size metadata first, make acquire the last step, and release on failure. Adds tests for each: post-dequeue terminal guard, delete_by_destination cancellation, generation_devices validation, absent-device rejection, and acquire-released-on-repoint-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Apply ruff 0.11.2 formatting to the files flagged by `ruff format --check`. - The new fail-fast guard in get_generation_devices() (reject a CUDA device that doesn't exist) made the pre-existing test_get_generation_devices_explicit_list_is_deduplicated fail on CPU-only CI runners, since it passes a cuda list with no CUDA present. Mock torch.cuda.is_available/device_count in that test (matching the existing pattern in this file) so it validates dedup on any runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # invokeai/app/services/session_processor/session_processor_default.py # invokeai/app/services/session_queue/session_queue_common.py # invokeai/app/services/session_queue/session_queue_sqlite.py # invokeai/backend/model_manager/load/model_loaders/qwen_image.py # invokeai/frontend/web/openapi.json # invokeai/frontend/web/src/services/api/schema.ts
- Shared CPU weights: drop_model() now invalidates the model's canonical entries in SharedCpuWeightsStore, so a rebuild on another device can never adopt pre-settings-change weights still aliased by a locked (stale-marked) entry. release() is identity-checked so a stale holder's eviction cannot decrement a newly registered canonical. update_model_record holds MODEL_LOAD_LOCK.write_lock() (off the event loop) across the multi-cache drop to exclude in-flight loads. - Runtime config API: generation_devices is now fully validated at the route boundary — empty lists and unavailable devices (e.g. cuda:99) return 422 without mutating or persisting config, using the same TorchDevice resolution as startup. - Cache stats: /v2/models/stats aggregates per-device caches instead of reporting only the API thread's default cache. - Config/docs contract: session_queue_mode description now documents device-affinity reordering in single-user multi-GPU mode (and that explicit FIFO disables it), and that user rotation outranks priority across users in round_robin mode. Multi-GPU docs no longer claim generation_devices: [] is valid, and describe shared-RAM weight deduplication instead of per-GPU duplication. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # invokeai/app/invocations/anima_denoise.py # invokeai/app/services/image_files/image_files_disk.py # invokeai/frontend/web/src/services/events/setEventListeners.tsx
…tion, MPS validation) - SharedCpuWeightsStore.invalidate() now retires still-referenced entries instead of dropping them from accounting, so RamBudget keeps counting retired weights until the last locked holder releases them. Prevents admitting models past max_cache_ram_gb while a replacement and a stale copy are both resident. - /models/stats aggregation takes max of cache_size and high_watermark across per-device caches (they share one global RamBudget, so summing over-reported an N-GPU system ~N times); event counters are still summed. - TorchDevice.get_generation_devices() rejects 'mps' when MPS is unavailable, so the runtime_config API 422s instead of persisting a device that fails at first tensor op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves conflict with invoke-ai#9367 (auth on model-manager/app-info endpoints): - get_stats keeps multi-GPU cache aggregation, gains the AdminUserOrDefault param - get_generation_device_options gains CurrentUserOrDefault (used by non-admin progress device badges as well as the admin settings modal) - stats/device-option tests patch auth_dependencies.ApiDependencies and provide a single-user configuration
Backend: - layer_patcher: hold MODEL_LOAD_LOCK.read_lock() across patch application so FLUX Control LoRA shape expansion (register_parameter) cannot overlap a concurrent model construction's process-global init_empty_weights patch - flux_redux/flux_denoise: store Redux conditioning on CPU (it may be produced on a borrowed idle GPU) and assign the .to() result when consuming it - model_cache/ram_budget: coordinate eviction across device caches — when a cache's own stack is exhausted and the global budget is still short, peers evict their unlocked entries (non-blocking lock, deadlock-free), so max_cache_ram_gb holds even when RAM is retained only by an idle device - session_queue: 'except current' operations protect the workflow-call chain of EVERY in-progress item, not one arbitrary get_current() row - session_queue: _cancel_in_progress_matching tolerates rows deleted by a concurrent clear between its id SELECT and the per-item cancel - session_processor: the post-dequeue cancel guard cancels the freshly claimed item when skipping it (a stale cancel_event must not abandon it in_progress) - session_processor: _clone_session_runner refuses to downgrade DefaultSessionRunner subclasses or share custom runners across workers - session_processor: an offloaded encoder's cache activity is attributed to the running session's CacheStats (borrowed cache's stale stats pointer swapped for the borrow duration) - events: progress events report the queue item's persisted device, not the thread-local (temporarily borrowed) one - devices/config docs: generation_devices 'auto' defers to an explicitly pinned legacy 'device:' setting so upgrades don't start workers on every GPU Frontend: - ImageViewer context: a terminal status only clears the shared progress event/image globals when that item owns them (multi-GPU: canceling item A no longer blanks item B's live preview) - SettingsGenerationDevices: device tags are keyboard-operable (tabIndex + Enter/Space activation) Each fix has an exposure test per the review's suggestions.
…A on a cold cache apply_smart_model_patches() held MODEL_LOAD_LOCK.read_lock() across its patch loop, but callers pass a lazy generator (e.g. flux_text_encoder._t5_lora_iterator) that constructs each LoRA via context.models.load() on demand. A cold-cache load takes MODEL_LOAD_LOCK.write_lock(); since the lock is non-reentrant and write-preferring, acquiring the write lock while this same thread already holds the read lock deadlocks (write waits for readers==0, but the consuming thread is that reader). The generation hung silently right after the encoder/tokenizer load, whenever a LoRA was applied and not already cached. Materialize the patch iterable before taking the read lock so every LoRA construction takes (and releases) the write lock first; the read lock then covers patch application only, which is its actual purpose (FLUX Control LoRA shape expansion calls register_parameter and must exclude concurrent construction). Compatible with wan_denoise's per-call iterator factory, and unrelated to the SD UNet path, which loads the LoRA before calling the singular patcher (no lock held). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. model_cache: a peer whose lock is contended during cross-cache eviction no longer leaves the shared RAM budget exceeded indefinitely. evict_unlocked_for_peer returns None on contention; the requester records a reconcile request on each skipped peer, and the synchronized-decorator hook honors it as soon as the peer's current operation releases the lock (outermost frame only — the RLock may be held reentrantly). The pending flag stays set until the budget is actually satisfied, so overshoot held by locked entries reconciles when their unlock releases the lock. 2. session_processor: a stale cancellation event from the previous item no longer cancels the freshly claimed, unrelated item. The post-dequeue guard now treats the DB status as the authority: a terminal row is skipped; a set cancel_event with a non-terminal row is a stale signal (a genuine cancel writes the row terminal BEFORE emitting) and is cleared, with a post-clear terminal re-check closing the clear's own race window. A shutdown-raced claim is still canceled so it isn't abandoned in_progress. 3. flux2_klein_text_encoder: conditioning is detached and moved to CPU before context.conditioning.save(), matching flux_text_encoder and flux_redux — the node is idle_gpu_offloadable, and GPU-resident embeddings would pin VRAM on a borrowed device after its pool lock is released. 4. session_queue clear: user-scoped clearing no longer assumes one current item. clear() cancels every in-progress item in scope via _cancel_in_progress_matching (each item's own status-changed event signals the worker running exactly that item) before deleting rows — same pattern as delete_by_destination; the router's arbitrary get_current() check (which could 403 the owner or cancel another user's item) is removed; and _on_queue_cleared honors the event's user_id so a scoped clear cannot stop other users' workers and abandon their rows. Each fix carries the regression test JPPhoto specified: contended-peer budget reconcile, stale-event-runs-item (plus the mid-clear race and shutdown cases), CPU-backed Klein conditioning, and Alice/Bob concurrent clear isolation at both the service and the event-handler layer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The merge-blocker fix 68edb02 reworded the clear route's docstring, which is the OpenAPI operation description — openapi.json and schema.ts must follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cile The deferred reconcile request was recorded pre-admission and honored only by the peer's next lock release. Two interleavings could strand the shared RAM budget above its cap indefinitely: - Lost wakeup: the busy peer releases its lock (running its reconcile hook while the flag is still unset) before request_budget_reconcile() sets the flag; if the peer then stays idle, no future release honors the request. - Pre-admission clearing: a peer's reconcile could run between the request and the new model being counted, see the budget as satisfied, and clear the flag before the admission pushed usage over the cap. Fix both by (1) moving the reconcile request to the end of put(), after the new model is counted, so peers always evaluate the true budget state, and (2) having request_budget_reconcile() attempt the reconcile inline with a non-blocking lock acquire: either the peer's lock is free now and the reconcile runs immediately, or it is still held and the eventual release hook — which runs strictly after the flag is set — performs it. The prior regression test masked the race by touching cache_b.stats after the request; it now emulates the production release hook in the holder thread and asserts reconciliation with no subsequent cache access, and a new test forces the lost-wakeup interleaving by delaying the request until the peer's operation has fully finished. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… queue lifecycle events Two related lifecycle gaps in the image viewer's multi-session preview state (ImageViewer/context.tsx): 1. Terminal-owner fallback: when the session owning the shared $progressEvent/$progressImage globals reached a terminal state, its tile was removed but the globals were cleared (or parked on the finished session's stale frame via the resolve illusion). Since the tiled view only renders with >1 active session, the remaining session's still-running preview disappeared until its next image event. The globals are now handed to the most recently updated remaining session immediately, for every terminal status. 2. Stale lifecycle: $progressData was cleaned only by per-item terminal events. It is now also cleared on queue_cleared (scoped like workflowExecutionCoordinator.onQueueCleared, and marking the cleared items finished so a trailing progress event from a worker stopped only by the clear cannot repopulate the preview), on socket disconnect, and on $socket replacement (auth-token/user change). The store logic is extracted into viewerProgressLifecycle.ts so it can be unit tested without rendering; the provider keeps the socket subscriptions and ownership/scope checks. 16 new vitest cases cover promotion across terminal statuses and auto-switch modes, non-owner termination, clear scoping (own/unscoped/foreign/sanitized), and disconnect resets. Follow-up to PR invoke-ai#9263 (JPPhoto review, 2026-07-25). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
api
python
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #9263, addressing the two ImageViewer preview issues deferred in @JPPhoto's 2026-07-25 review. They share the same store and handler, so they are fixed together:
1. Terminal-owner fallback (
context.tsx): when the session owning the shared$progressEvent/$progressImageglobals reached a terminal state, its tile was removed but the globals were cleared — or, for successful completion with auto-switch, parked on the finished session's stale frame via the resolve illusion. Since the tiled view only renders with more than one active session, the remaining session's still-running preview disappeared until its next image event. The globals are now handed to the most recently updated remaining session immediately, for every terminal status and auto-switch mode.2. Stale progress lifecycle:
$progressDatawas cleaned only by per-item terminal events. It is now also cleared on:queue_cleared— scoped exactly likeworkflowExecutionCoordinator.onQueueCleared(unscoped and own-user clears apply; foreign scoped clears and the sanitizeduser_id="redacted"broadcast do not), and the cleared items are marked finished so a trailinginvocation_progressevent from a worker stopped only by the clear cannot repopulate the preview;setEventListeners);$socketreplacement after an auth-token/user change.Implementation
The store logic is extracted into
viewerProgressLifecycle.tsso it can be unit tested without rendering (per the web CLAUDE.md convention of no UI tests); the provider keeps the socket subscriptions and ownership/scope checks. Sessions gain aseqcounter so "most recently updated" promotion is exact rather than timestamp-granular.Tests
16 vitest cases: preview promotion for completed/canceled/failed and both auto-switch modes, multi-survivor promotion order, non-owner termination leaving the preview alone, last-session clear/resolve behavior, repeat-event idempotence, clear scoping (own/unscoped/foreign/sanitized), trailing-progress suppression after a clear, and disconnect reset.
Merge order
Stacked on #9263 — this branch contains the multi-GPU branch's commits. Draft until #9263 merges; will then rebase onto main and mark ready for review.
Checklist
What's Newcopy (if doing a release after this PR)🤖 Generated with Claude Code