Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agentsroom/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# AgentsRoom: personal files (not committed to git)
*-personal.json
agents-local.json
sessions/
10 changes: 10 additions & 0 deletions .agentsroom/agents.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[
{
"role": "fullstack",
"model": "opus",
"customName": "Full-Stack Developer",
"isPersonal": false,
"id": "agent-1776361243376-3sekdc",
"claudeSessionId": "96773a93-be2a-45a9-a732-ceb224d3d0e5"
}
]
4 changes: 4 additions & 0 deletions .agentsroom/prompts.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"folders": [],
"prompts": []
}
21 changes: 21 additions & 0 deletions .changeset/media-edit-from.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
'@tanstack/ai': minor
'@tanstack/ai-openai': minor
'@tanstack/ai-gemini': minor
'@tanstack/ai-grok': minor
'@tanstack/ai-fal': minor
'@tanstack/ai-client': minor
---

feat: first-class follow-up edits for generated media.

`generateVideo({ ..., previousJobId })` edits a previously generated video instead of generating from scratch. Callers always pass the prior generation's job id; adapters decide how to consume it via a `VideoAdapter` edit-kind map:

- `'job'` — reference the id server-side (OpenAI Sora 2 / Sora 2 Pro remix; Gemini Omni Flash, which maps `previousJobId` onto the Interactions API's `previous_interaction_id` wire field — that field is omitted from Omni `modelOptions`)
- `'media'` — resolve the finished clip via `getVideoUrl(previousJobId)` (xAI `grok-imagine-video` → `/videos/edits`; fal video-to-video endpoints such as `xai/grok-imagine-video/edit-video` and Seedance 2.0 reference-to-video). Fal generate endpoints with a known edit sibling (e.g. Grok text/image-to-video) resolve on the generate model, then submit to the edit endpoint.

Non-editing models (Veo, `grok-imagine-video-1.5`) reject `previousJobId` at compile time. Sora remix and Grok edits accept only a prompt — `size` / `duration` / media inputs are rejected because the output inherits them from the source video.

`generateImage({ ..., previousImage })` is the image-side counterpart: pass a prior result's `GeneratedImage` (or an array, or the whole result) and it is prepended to the prompt as an image part, flowing through each adapter's existing edit path; type-gated to models that accept image inputs.

Breaking for hand-rolled (non-`BaseVideoAdapter`) `VideoAdapter` implementations: the interface gains `supportedEditKind(): 'job' | 'media' | undefined` (and a 7th, defaulted `TModelEditByName` generic — existing 6-argument instantiations keep compiling). `BaseVideoAdapter` supplies a default returning `undefined`, plus `resolvePreviousJobUrl(previousJobId)`. New exports include `VideoEditKind`, `ModelEditKindByName`, `VideoPreviousJobIdForAdapter`, `ImagePreviousSource`, `ImagePreviousImageForModel`, `generatedImageToImagePart`, `generatedVideoUrlToVideoPart`. Client wire types: `VideoGenerateInput.previousJobId`, `ImageGenerateInput.previousImage`.
4 changes: 2 additions & 2 deletions docs/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -288,13 +288,13 @@
"label": "Image Generation",
"to": "media/image-generation",
"addedAt": "2026-04-15",
"updatedAt": "2026-07-07"
"updatedAt": "2026-07-10"
},
{
"label": "Video Generation",
"to": "media/video-generation",
"addedAt": "2026-04-15",
"updatedAt": "2026-07-02"
"updatedAt": "2026-07-10"
},
{
"label": "Generation Hooks",
Expand Down
64 changes: 64 additions & 0 deletions docs/media/image-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ All image adapters support these common options:
| `prompt` | `string \| MediaPromptPart[]` | Description of the image to generate (required). A plain string, or — on models that support image-conditioned generation — an ordered array of content parts interleaving text with image inputs. See [Image-Conditioned Generation](#image-conditioned-generation) below. |
| `numberOfImages` | `number` | Number of images to generate |
| `size` | `string` | Size of the generated image in WIDTHxHEIGHT format |
| `previousImage?` | `GeneratedImage \| GeneratedImage[] \| { images }` | A previously generated image (or images) to edit — prepended to the prompt as image input(s). Only offered (at compile time) for models that accept image inputs — see [Editing generated images](#editing-generated-images-previousimage). |
| `modelOptions?` | `object` | Model-specific options (renamed from `providerOptions`) |

### Size Options
Expand Down Expand Up @@ -183,6 +184,69 @@ The accepted part types are narrowed **per model at compile time**: passing
an image part to a text-only model (e.g. `dall-e-3`, Imagen) is a type
error, not just a runtime throw.

### Editing generated images (previousImage)

To run a **follow-up edit** on something you just generated, pass the prior
result's image as `previousImage` — sugar that prepends it to the prompt as an
image part, so it flows through the model's regular edit path (OpenAI
`/images/edits`, Gemini `generateContent`, xAI `/images/edits`, fal).

**Server:**

```typescript
import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

const adapter = openaiImage('gpt-image-2')

const first = await generateImage({ adapter, prompt: 'A city street at dusk' })

const edited = await generateImage({
adapter,
prompt: 'Same scene, but make it rain',
previousImage: first.images[0],
})
```

`previousImage` accepts a single `GeneratedImage`, an array of them, or the
whole prior result (`{ images }`). URL results pass through as `url`
sources (`data:` URLs are decomposed into raw bytes for adapters that
upload files); `b64Json` results become `data` sources with the mime type
sniffed from the payload (defaulting to `image/png`). Like image parts, the
option is offered **per model at compile time** — text-only models
(`dall-e-3`, Imagen) reject it as a type error.

**Client** — the hook's `ImageGenerateInput.previousImage` is a wire-friendly
`{ url? }` / `{ b64Json? }` shape; your server route should narrow it back
to a `GeneratedImage` before calling `generateImage`:

```tsx
import { useGenerateImage, fetchServerSentEvents } from '@tanstack/ai-react'

function ImageEditor() {
const { generate, result, isLoading } = useGenerateImage({
connection: fetchServerSentEvents('/api/generate/image'),
})

const handleEdit = () => {
const image = result?.images[0]
if (!image) return
void generate({
prompt: 'Same scene, but make it rain',
previousImage: image.url
? { url: image.url }
: { b64Json: image.b64Json },
})
Comment on lines +219 to +239

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include the server endpoint half of this example.

The client snippet sends a wire-friendly { url? } / { b64Json? } value and explicitly says the server must narrow it, but no route showing that conversion is provided. Add a minimal server endpoint that validates the payload and passes a GeneratedImage to generateImage.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/media/image-generation.md` around lines 219 - 239, Extend the
image-generation example with a minimal server endpoint matching the client
request, validating the wire-friendly previousImage shape, narrowing it to a
GeneratedImage, and passing it to generateImage. Reference the route handler and
generateImage usage, and show handling for invalid payloads and both URL and
base64 image variants.

Source: Coding guidelines

}

return (
<button onClick={handleEdit} disabled={isLoading || !result}>
Edit
</button>
)
}
```

### Referencing images from your prompt

**Your prompt text is always sent verbatim — the SDK never injects or
Expand Down
105 changes: 98 additions & 7 deletions docs/media/video-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,7 @@ And returns:
| `prompt` | `string \| MediaPromptPart[]` | Description of the video to generate (required). A plain string, or — on models that support conditioned generation — an ordered array of content parts interleaving text with image / video / audio inputs. See [Image-to-Video](#image-to-video) below. |
| `size` | `string` | Video resolution in WIDTHxHEIGHT format |
| `duration` | `number` | Video duration in seconds (maps to `seconds` parameter in API) |
| `previousJobId?` | `string` | Prior generation's job id to edit instead of generating from scratch. Only offered (at compile time) for models that support follow-up edits — see [Editing Generated Videos](#editing-generated-videos-previousjobid). |
| `modelOptions?` | `object` | Model-specific options (renamed from `providerOptions`) |

## Image-to-Video
Expand Down Expand Up @@ -489,6 +490,91 @@ The API uses the `seconds` parameter. Allowed values:
- `8` seconds (default)
- `12` seconds

## Editing Generated Videos (previousJobId)

Models that support **follow-up runs** can edit a previously generated
video instead of generating from scratch: pass the prior generation's
job id as `previousJobId` and describe the change in the prompt. The
canonical call is the same for every provider.

**Server:**

```typescript ignore
import { generateVideo, getVideoJobStatus } from '@tanstack/ai'
import { openaiVideo } from '@tanstack/ai-openai'

const adapter = openaiVideo('sora-2')

// Turn 1: generate
const first = await generateVideo({ adapter, prompt: 'A city street at dusk' })
// …poll first.jobId to completion…

// Turn 2: edit the result
const edited = await generateVideo({
adapter,
prompt: 'Make it rain',
previousJobId: first.jobId,
})
```

**Client** — pass the completed job's id through the hook; your server
forwards it to `generateVideo`:

```tsx
import { useGenerateVideo, fetchServerSentEvents } from '@tanstack/ai-react'

function VideoEditor() {
const { generate, result, isLoading } = useGenerateVideo({
connection: fetchServerSentEvents('/api/generate/video'),
})

const handleEdit = () => {
if (!result?.jobId) return
void generate({
prompt: 'Make it rain',
previousJobId: result.jobId,
})
Comment on lines +520 to +536

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the server endpoint half of the client example.

The example says the server forwards previousJobId but does not show that endpoint. Add a minimal route that validates the client input and forwards the job ID to generateVideo.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/media/video-generation.md` around lines 520 - 536, Add the corresponding
server endpoint example after the client snippet, defining a minimal route that
validates the incoming prompt and optional previousJobId, then passes both
values to generateVideo. Show the response returned to the client and ensure the
route’s symbols clearly demonstrate forwarding previousJobId.

Source: Coding guidelines

Comment on lines +531 to +536

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use the hook’s job id instead of result.jobId.

The documented hook exposes the job identifier separately (jobId/videoStatus.jobId), while the completed result contains the video result. As written, result?.jobId can be undefined and the edit handler silently returns. Destructure jobId and pass that value as previousJobId.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/media/video-generation.md` around lines 531 - 536, Update handleEdit to
use the hook-provided job identifier rather than result?.jobId: destructure
jobId (or videoStatus.jobId) from the hook and use it for the guard and as
generate’s previousJobId value.

}

return (
<button onClick={handleEdit} disabled={isLoading || !result}>
Edit
</button>
)
}
```

Each model declares **how** it consumes that job id via
`adapter.supportedEditKind()`:

| Kind | How the adapter uses `previousJobId` | Providers |
| --- | --- | --- |
| `'job'` | References the prior job server-side | OpenAI Sora 2 / Sora 2 Pro (remix), Gemini Omni Flash |
| `'media'` | Resolves the finished clip via `getVideoUrl(previousJobId)` | xAI `grok-imagine-video` (`/videos/edits`), fal video-to-video endpoints (`xai/grok-imagine-video/edit-video`, `fal-ai/wan/v2.7/edit-video`, Seedance 2.0 reference-to-video) |

Models without follow-up support (Veo, `grok-imagine-video-1.5`, fal
text/image-to-video endpoints without a known edit sibling) reject
`previousJobId` at compile time.

Provider-specific constraints:

- **OpenAI Sora (remix)** accepts only a text prompt — the output inherits
the source video's size and duration, so `size`, `duration`, and image
parts are rejected when `previousJobId` is set.
- **Grok `/videos/edits`** inherits duration and aspect ratio from the
source (capped at 720p, input truncated to 8 seconds); `size` / `duration`
options are rejected when `previousJobId` is set. The resolved source rides
`video_url` (public URL, base64 `data:` URI, or `file_id`).
- **Gemini Omni Flash** maps `previousJobId` onto the Interactions API's
`previous_interaction_id` wire field internally. That field is **not**
exposed on Omni `modelOptions` — use `previousJobId` only.
- **fal** resolves `previousJobId` via `getVideoUrl` on the generate
model, then routes the URL onto the edit endpoint's video input
(`video_url`, or the endpoint's list field — Seedance 2.0's
reference-to-video takes `video_urls`). Generate endpoints with a known
edit sibling (e.g. Grok text/image-to-video → `edit-video`) do this
automatically.

## Model Options

### OpenAI Model Options
Expand Down Expand Up @@ -618,10 +704,10 @@ the Files API first).

#### Conversational video editing

Omni's headline capability is iterative refinement: pass the interaction id
of a prior generation (its `jobId`) as
`modelOptions.previous_interaction_id` and describe the change — the model
edits the video while preserving everything you didn't mention:
Omni's headline capability is iterative refinement: pass a prior
generation's `jobId` (its interaction id) as `previousJobId` and describe the
change — the model edits the video while preserving everything you didn't
mention (see [Editing Generated Videos](#editing-generated-videos-previousjobid)):

```typescript ignore
import { generateVideo } from '@tanstack/ai'
Expand All @@ -641,12 +727,15 @@ const first = await generateVideo({
const second = await generateVideo({
adapter,
prompt: 'Make the violin invisible',
modelOptions: { previous_interaction_id: first.jobId },
previousJobId: first.jobId,
})
```

`modelOptions` also passes through the Interactions API's request fields
(e.g. `generation_config.video_config.task` to pin
The adapter maps `previousJobId` onto the Interactions API's
`previous_interaction_id` wire field. That field is not available on Omni
`modelOptions` — always use `previousJobId`. `modelOptions` still passes
through the Interactions API's other request fields (e.g.
`generation_config.video_config.task` to pin
`'text_to_video' | 'image_to_video' | 'reference_to_video' | 'edit'`
instead of letting the model infer the task mode).

Expand Down Expand Up @@ -708,6 +797,8 @@ adapter.snapDuration(99) // 15

Generated clips include an audio track. When the job completes, the adapter reports `usage.unitsBilled` (billed seconds of video) and `usage.cost` (exact USD cost as returned by the API) on the result.

`grok-imagine-video` can also edit a previously generated clip via [`previousJobId`](#editing-generated-videos-previousjobid) — pass the prior job id and an edit prompt; the adapter resolves the finished clip and posts to xAI's `/videos/edits` endpoint, then polls like any other job. The output inherits duration and aspect ratio from the source (capped at 720p, input truncated to 8 seconds).

## Response Types

> **Note:** The interfaces below are the underlying adapter-level types. The `getVideoJobStatus()` helper returns a single merged object, `{ status, progress?, url?, error?, usage? }` — it does not return `jobId` or `expiresAt`.
Expand Down
78 changes: 76 additions & 2 deletions examples/ts-react-media/src/components/ImageGenerator.tsx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { useRef, useState } from 'react'
import { ImageIcon, Loader2, Plus, Shuffle, X } from 'lucide-react'
import { ImageIcon, Loader2, Plus, Shuffle, Wand2, X } from 'lucide-react'
import type { ImageGenerationResult } from '@tanstack/ai'
import type { MediaPrompt } from '@tanstack/ai/client'
import type { GeneratedImage, MediaPrompt } from '@tanstack/ai/client'

import { generateImageFn } from '@/lib/server-functions'
import { getRandomImagePrompt } from '@/lib/prompts'
Expand Down Expand Up @@ -37,6 +37,7 @@ export default function ImageGenerator({
const [isLoading, setIsLoading] = useState(false)
const [results, setResults] = useState<Record<string, ModelResult>>({})
const [images, setImages] = useState<Array<AttachedMedia>>([])
const [editPrompts, setEditPrompts] = useState<Record<string, string>>({})
const fileInputRef = useRef<HTMLInputElement>(null)

const currentModel = IMAGE_MODELS.find((m) => m.id === selectedModel)
Expand Down Expand Up @@ -64,6 +65,40 @@ export default function ImageGenerator({
setImages((prev) => prev.filter((image) => image.id !== id))
}

/**
* Follow-up edit of a generated image: the prior image rides `previousImage`
* and the server prepends it to the prompt as an image input for the
* model's edit path.
*/
const handleEditImage = async (modelId: string, image: GeneratedImage) => {
const editPrompt = editPrompts[modelId]?.trim()
if (!editPrompt) return

setResults((prev) => ({ ...prev, [modelId]: { status: 'loading' } }))
try {
const response = await generateImageFn({
data: { prompt: editPrompt, model: modelId, previousImage: image },
})
setResults((prev) => ({
...prev,
[modelId]: { status: 'success', result: response },
}))
setEditPrompts((prev) => ({ ...prev, [modelId]: '' }))
const edited = response.images[0]
if (edited) {
onImageGenerated?.(getImageSrc(edited))
}
} catch (err) {
setResults((prev) => ({
...prev,
[modelId]: {
status: 'error',
error: err instanceof Error ? err.message : 'Failed to edit image',
},
}))
}
}

const handleGenerate = async () => {
if (!prompt.trim()) return
const builtPrompt = buildPrompt()
Expand Down Expand Up @@ -313,6 +348,45 @@ export default function ImageGenerator({
— multiply by the endpoint unit price for USD cost
</p>
)}
{model?.editable && (
<div className="flex gap-2">
<input
type="text"
value={editPrompts[modelId] ?? ''}
onChange={(e) =>
setEditPrompts((prev) => ({
...prev,
[modelId]: e.target.value,
}))
}
onKeyDown={(e) => {
if (e.key === 'Enter')
handleEditImage(
modelId,
modelResult.result!.images[0]!,
)
}}
placeholder="Describe an edit — e.g. 'make it night time'..."
disabled={isLoading}
className="flex-1 px-3 py-2 bg-gray-800 border border-gray-700 rounded-lg text-white text-sm placeholder-gray-500 focus:outline-none focus:ring-2 focus:ring-purple-500 focus:border-transparent disabled:opacity-50"
/>
<button
onClick={() =>
handleEditImage(
modelId,
modelResult.result!.images[0]!,
)
}
Comment on lines +362 to +379

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Remove the redundant modelResult.result! non-null assertion (ESLint error).

Inside the status === 'success' && modelResult.result && ... block, modelResult.result is already narrowed to non-null, so the ! after result is unnecessary and is flagged as an error by @typescript-eslint/no-unnecessary-type-assertion (lines 366 and 377), which will fail lint. Keep the ! on images[0] only.

🧹 Proposed fix
                             onKeyDown={(e) => {
                               if (e.key === 'Enter')
                                 handleEditImage(
                                   modelId,
-                                  modelResult.result!.images[0]!,
+                                  modelResult.result.images[0]!,
                                 )
                             }}
                           onClick={() =>
                             handleEditImage(
                               modelId,
-                              modelResult.result!.images[0]!,
+                              modelResult.result.images[0]!,
                             )
                           }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
onKeyDown={(e) => {
if (e.key === 'Enter')
handleEditImage(
modelId,
modelResult.result!.images[0]!,
)
}}
placeholder="Describe an edit — e.g. 'make it night time'..."
disabled={isLoading}
className="flex-1 px-3 py-2 bg-gray-800 border border-gray-700 rounded-lg text-white text-sm placeholder-gray-500 focus:outline-none focus:ring-2 focus:ring-purple-500 focus:border-transparent disabled:opacity-50"
/>
<button
onClick={() =>
handleEditImage(
modelId,
modelResult.result!.images[0]!,
)
}
onKeyDown={(e) => {
if (e.key === 'Enter')
handleEditImage(
modelId,
modelResult.result.images[0]!,
)
}}
placeholder="Describe an edit — e.g. 'make it night time'..."
disabled={isLoading}
className="flex-1 px-3 py-2 bg-gray-800 border border-gray-700 rounded-lg text-white text-sm placeholder-gray-500 focus:outline-none focus:ring-2 focus:ring-purple-500 focus:border-transparent disabled:opacity-50"
/>
<button
onClick={() =>
handleEditImage(
modelId,
modelResult.result.images[0]!,
)
}
🧰 Tools
🪛 ESLint

[error] 366-366: This assertion is unnecessary since the receiver accepts the original type of the expression.

(@typescript-eslint/no-unnecessary-type-assertion)


[error] 377-377: This assertion is unnecessary since the receiver accepts the original type of the expression.

(@typescript-eslint/no-unnecessary-type-assertion)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/ts-react-media/src/components/ImageGenerator.tsx` around lines 362 -
379, Remove the unnecessary non-null assertions from modelResult.result in both
handleEditImage calls within the success block, while retaining the images[0]!
assertion. Update both the Enter-key handler and button onClick handler.

Source: Linters/SAST tools

disabled={
isLoading || !editPrompts[modelId]?.trim()
}
className="px-4 py-2 bg-purple-600 hover:bg-purple-700 disabled:bg-gray-700 disabled:cursor-not-allowed text-white text-sm font-medium rounded-lg transition-colors flex items-center gap-1.5"
>
<Wand2 className="w-4 h-4" />
Edit
</button>
</div>
)}
</>
)}
</div>
Expand Down
Loading
Loading