Skip to content

feat: generate image, video and audio assets from prompts via a host-registered handler - #151

Closed
dazzatronus wants to merge 1 commit into
mainfrom
derk/sdk-generation-lifecycle
Closed

feat: generate image, video and audio assets from prompts via a host-registered handler#151
dazzatronus wants to merge 1 commit into
mainfrom
derk/sdk-generation-lifecycle

Conversation

@dazzatronus

Copy link
Copy Markdown
Contributor

An image, video or audio asset can carry a prompt instead of a src. Those clips
already render as a placeholder. This makes them generatable from inside the editor.

The SDK owns the lifecycle; the host owns the work:

edit.registerAssetGenerator(async ({ clipId, asset, signal }) => {
	const url = await myBackend.generate(asset, { signal });
	return { url };
});

The SDK tracks pending, generating and failed states, renders them on the canvas and
timeline, and writes the returned URL back through the normal edit path so the change
is undoable and autosaves. A rejection's message is shown as-is beside a retry action.
Which models exist, what they cost and what an error means stay with the host.

Generation state is transient: never saved to the edit, never part of undo, gone on
reload. Deleting a clip mid-generation aborts its request via the signal.

Reviewer notes

No new business logic in the SDK. No model registry, no pricing, no error-code
handling. registerAssetGenerator is the only new integration point.

Without a registered handler there is no generate affordance — not in the toolbar,
not on the canvas placeholder. The prompt stays editable either way, since a prompt is
a document property like any other.

Prompt controls live in the existing media toolbar rather than a separate one, so
generative clips keep every clip-level property (fit, opacity, scale, transitions,
effects, volume).

Two fixes outside the feature, both with regression tests:

  • resolveToNumber used parseFloat, which accepts a numeric prefix. A merge-field
    string resolving to "03 image of a cat" became the number 3, so the clip lost its
    prompt entirely. Now a whole-string parse. This moves merge-field resolution closer to
    the render backend but does not reach parity — see the note below.
  • Deleting a clip aborted in-flight generation only via deleteClipById. The toolbar
    trash button and Delete key call deleteClip(trackIdx, clipIdx) directly, so a
    generation outlived its clip and completed against a deleted target. The abort moved
    to deleteClip, which every deletion funnels through.

Known gap, tracked separately. Merge-field type resolution still differs from the
render backend, which decides the resulting type purely from the type of replace. The
SDK infers it from the key name and whether the resolved text looks numeric, so a string
value like "0xffffff" or "007" becomes a number in preview but stays a string at
render. Closing that requires widening the public MergeField typings, so it is not in
this PR.

Verifying

npx jest && npx tsc --noEmit && npx eslint src tests

Manual: npm run dev, open /smoketest.html. A fake generator with success / failure /
slow-for-abort modes and a template covering every combination — bare prompt, model plus
options, merge-field prompt, prompt with a preview src, plain src, video, audio, and
both legacy generative types. Add ?nogen to see the editor with no generator registered.

@dazzatronus

Copy link
Copy Markdown
Contributor Author

Superseded by #152#153.

At 1245 lines this was over the async-review ceiling and carried a separable fix. The merge-field resolver fix is now #152; the feature targets it in #153, declared as needing a walkthrough with the decision-carrying files called out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant