Skip to content

[2/2] feat: generate image, video and audio assets from prompts via a host-registered handler - #153

Open
dazzatronus wants to merge 1 commit into
derk/sdk-merge-field-resolutionfrom
derk/sdk-generation-feature
Open

[2/2] feat: generate image, video and audio assets from prompts via a host-registered handler#153
dazzatronus wants to merge 1 commit into
derk/sdk-merge-field-resolutionfrom
derk/sdk-generation-feature

Conversation

@dazzatronus

@dazzatronus dazzatronus commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

An image, video or audio asset can carry a prompt instead of a src, and those clips render as an inert placeholder. This lets a host fill them.

The host supplies one async callback and the SDK owns everything else: pending, generating and failed states, the canvas and timeline treatments, and writing the returned URL back through the normal edit path so the change is undoable and autosaves. A rejection's message shows as-is beside a retry. Models, pricing and error semantics stay with the host, which is why the only new integration point is registerAssetGenerator.

The feature is inert until a host registers a handler — no generate control appears anywhere without one. That is the gate, which is why there is no flag as well.

Declared unsplittable at 1204 lines. It does not divide into independently revertible pieces: the lifecycle, the states it drives and the controls that trigger it are one decision, and a core-only PR would ship an API nothing calls. What was separable has been separated into #152, which this targets.

Reading order:

  1. src/core/generation/asset-generator.ts then src/core/edit-session.ts — the lifecycle and the host contract. These two carry the design.
  2. src/core/ui/media-toolbar.ts — prompt and generate folded into the existing toolbar, so generative clips keep every clip-level property. An earlier attempt used a separate toolbar and lost them.
  3. src/components/canvas/players/* and src/components/timeline/* — state plumbing, churn.
  4. smoketest.* and src/templates/generation-smoketest.json — a dev-only harness, 327 lines, not shipped. Skip.

One thing a reviewer would otherwise flag: the abort on clip deletion sits in deleteClip, not the by-id wrapper, because the toolbar trash and the Delete key call the former directly.

Verify: npx jest && npx tsc --noEmit && npx eslint src tests. Manually, npm run dev and open /smoketest.html; a fake generator covers success, failure and slow-for-abort, and ?nogen shows the editor with no handler registered.

Risk: additive. Existing templates without prompts are untouched, and an unregistered host sees today's behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant