feat: generate image, video and audio assets from prompts via a host-registered handler - #151
Closed
dazzatronus wants to merge 1 commit into
Closed
feat: generate image, video and audio assets from prompts via a host-registered handler#151dazzatronus wants to merge 1 commit into
dazzatronus wants to merge 1 commit into
Conversation
…registered handler
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An image, video or audio asset can carry a
promptinstead of asrc. Those clipsalready render as a placeholder. This makes them generatable from inside the editor.
The SDK owns the lifecycle; the host owns the work:
The SDK tracks pending, generating and failed states, renders them on the canvas and
timeline, and writes the returned URL back through the normal edit path so the change
is undoable and autosaves. A rejection's message is shown as-is beside a retry action.
Which models exist, what they cost and what an error means stay with the host.
Generation state is transient: never saved to the edit, never part of undo, gone on
reload. Deleting a clip mid-generation aborts its request via the
signal.Reviewer notes
No new business logic in the SDK. No model registry, no pricing, no error-code
handling.
registerAssetGeneratoris the only new integration point.Without a registered handler there is no generate affordance — not in the toolbar,
not on the canvas placeholder. The prompt stays editable either way, since a prompt is
a document property like any other.
Prompt controls live in the existing media toolbar rather than a separate one, so
generative clips keep every clip-level property (fit, opacity, scale, transitions,
effects, volume).
Two fixes outside the feature, both with regression tests:
resolveToNumberusedparseFloat, which accepts a numeric prefix. A merge-fieldstring resolving to
"03 image of a cat"became the number3, so the clip lost itsprompt entirely. Now a whole-string parse. This moves merge-field resolution closer to
the render backend but does not reach parity — see the note below.
deleteClipById. The toolbartrash button and Delete key call
deleteClip(trackIdx, clipIdx)directly, so ageneration outlived its clip and completed against a deleted target. The abort moved
to
deleteClip, which every deletion funnels through.Known gap, tracked separately. Merge-field type resolution still differs from the
render backend, which decides the resulting type purely from the type of
replace. TheSDK infers it from the key name and whether the resolved text looks numeric, so a string
value like
"0xffffff"or"007"becomes a number in preview but stays a string atrender. Closing that requires widening the public
MergeFieldtypings, so it is not inthis PR.
Verifying
Manual:
npm run dev, open/smoketest.html. A fake generator with success / failure /slow-for-abort modes and a template covering every combination — bare prompt, model plus
options, merge-field prompt, prompt with a preview
src, plainsrc, video, audio, andboth legacy generative types. Add
?nogento see the editor with no generator registered.