Skip to content

feat(cli): add agent-first media treatment workflow#2753

Open
ukimsanov wants to merge 1 commit into
feat/media-treatment-runtimefrom
feat/media-treatment-agent
Open

feat(cli): add agent-first media treatment workflow#2753
ukimsanov wants to merge 1 commit into
feat/media-treatment-runtimefrom
feat/media-treatment-agent

Conversation

@ukimsanov

Copy link
Copy Markdown
Collaborator

What

Adds the canonical hyperframes media-treatment command, progressive capability/detail output, deterministic apply/clear operations, media-use guidance and recipes, public docs, and offline error-diffusion tooling.

Why

Agents should discover a small capability menu, inspect only the relevant treatment, assemble bounded custom payloads, apply them deterministically, and verify results without learning the low-level HTML attribute first.

How

Routes look, effect, animation, and overlay intent through one CLI surface. Skills teach agents to use real media elements, combine compatible operators, preserve user intent, offer unsolicited polish once, and verify representative frames.

Test plan

  • CLI tests: 30/30
  • Media-use, LUT, and dither tests: 61/61
  • CLI typecheck, skill lint/mirror, generated manifest, format, lint, and Fallow gates pass
  • Documentation updated
  • Exact six-PR stack passes the complete workspace build

Stack 3/6. Base: #2752. Registry overlays and Studio UI are optional later layers.

@ukimsanov
ukimsanov marked this pull request as ready for review July 24, 2026 03:42
Copilot AI review requested due to automatic review settings July 24, 2026 03:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an agent-first “media treatment” workflow to HyperFrames by introducing a canonical hyperframes media-treatment CLI surface for capability discovery + deterministic persistence, and by expanding the /media-use skill docs/tests to route broad “footage looks…” feedback into that workflow (including new offline cached error-diffusion tooling).

Changes:

  • Introduces hyperframes media-treatment CLI command (capability overview/detail + apply/clear on a single <img>/<video> via data-color-grading).
  • Updates /media-use guidance, recipes, and tests to treat broad visual feedback as media-treatment intent (plus adds resolve --analyze for side-effect-free grade evidence).
  • Adds exact cached error-diffusion dithering tooling + tests, and expands LUT/preset intent routing synonyms.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
skills/media-use/SKILL.md Expands routing guidance to media treatments and progressive capability discovery.
skills/media-use/scripts/resolve.test.mjs Adds coverage for --analyze help + side-effect-free grade evidence behavior.
skills/media-use/scripts/resolve.mjs Adds --analyze mode for grade evidence without recording a manifest candidate.
skills/media-use/scripts/lib/lut-preset-provider.test.mjs Adds deterministic intent-alias tests for new/curated preset routing.
skills/media-use/scripts/lib/lut-preset-provider.mjs Expands preset IDs/synonyms; keeps legacy IDs readable while de-emphasizing them in fuzzy matching.
skills/media-use/scripts/lib/error-diffusion.test.mjs Adds golden tests for deterministic error-diffusion kernels + contract validation.
skills/media-use/scripts/lib/error-diffusion.mjs Implements error-diffusion kernels and public apply function for RGBA buffers.
skills/media-use/scripts/dither.test.mjs Adds end-to-end tests for dithering images/videos (palette correctness, audio/metadata, HDR rejection).
skills/media-use/scripts/dither.mjs Adds offline cached error-diffusion dither CLI (image/MP4) built on FFmpeg + error-diffusion lib.
skills/media-use/references/operations.md Documents “Exact error-diffusion dither” workflow and when to use it.
skills/media-use/references/media-treatments.md Adds media-treatment policy: lanes, verification workflow, capability discovery, overlay composition rules.
skills/media-use/references/media-treatment-recipes.md Adds a comprehensive set of tested seed recipes for common treatment intents.
skills/media-use/references/grading.md Reframes “grade” usage around the media-treatment workflow and deterministic persistence/verification.
skills/hyperframes/SKILL.md Routes broad photographic-media feedback to /media-use and its media-treatment policy.
skills/hyperframes/references/capability-menu.md Updates capability menu row to include “media treatments” language and routing.
skills/hyperframes-cli/references/compare-and-batch.md Updates grading JSON example to match the canonical nested adjust shape.
skills/general-video/SKILL.md Updates cross-cutting adapter guidance to route vague footage feedback via media-treatments policy.
skills-manifest.json Refreshes skill hashes/file counts after content updates.
packages/cli/src/templates/_shared/CLAUDE.md Adds a new-project instruction gate for media-treatment policy before editing real media looks/reveals.
packages/cli/src/templates/_shared/AGENTS.md Mirrors the new-project instruction gate for media-treatment policy.
packages/cli/src/help.ts Adds media-treatment to CLI help groups.
packages/cli/src/commands/coreSkillContent.test.ts Adds tests to ensure routing/docs/recipes stay synchronized (incl. palette sync with Core).
packages/cli/src/commands/color-grading.ts Implements media-treatment command: capability discovery + deterministic HTML mutation/clear.
packages/cli/src/commands/color-grading.test.ts Adds unit tests for capability discovery and deterministic application/clearing behavior.
packages/cli/src/cli.ts Registers the media-treatment command loader.
packages/cli/src/cli.commands.test.ts Asserts media-treatment is registered and color-grading is not.
packages/cli/src/capture/agentPromptGenerator.ts Adds capture prompt guidance to use /media-use media-treatment policy for later media edits.
docs/guides/color-grading.mdx Expands docs to include effect families/overlays, richer schema examples, performance notes, and CLI guidance.
docs/concepts/variables.mdx Updates example preset enum defaults to the curated clean-studio preset.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/guides/color-grading.mdx Outdated
`--all` only for exhaustive tooling. Recipes are tested shortcuts, not the
complete allowed surface. Persist the final combined payload with the same
`hyperframes media-treatment` command; unknown keys are rejected before the
composition is changed. The legacy `color-grading` command remains an alias.
Comment on lines +567 to +569
const verb = mutationVerb(action, result.changed, payload.dryRun);
console.log(`${c.success("◇")} ${verb} color grading to ${c.accent(selector)}`);
}

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Audited: packages/cli/src/commands/color-grading.ts and tests; skills/media-use/scripts/dither.mjs, lib/error-diffusion.mjs, their tests, the resolve --analyze delta, LUT preset parity, command registration, and CI at exact head d91c2f4b.

Trusting: the 991-line recipe catalog was sampled against its positive-pin tests rather than recipe-reviewed line by line; generated skill-manifest hashes and documentation-only wording outside the new workflow were not independently regenerated.

The canonical core registries are used at the CLI boundary, persisted grading is normalized/serialized deterministically, and the standalone preset mirror has an exhaustive source-parity test. Copilot already covers the stale alias sentence and success-message wording; I did not repeat those.

Important: skills/media-use/scripts/lib/error-diffusion.mjs:131-136 and skills/media-use/scripts/dither.mjs:197-200 allocate three Float32 error slots for every source pixel even though default pointSize=3 reads/writes error only at one block center. At 4K this is ~100 MB for the error grid alone (~133 MB with the RGBA frame), while a block-grid buffer is ~11 MB. Index by block coordinates, or retain only the kernel row window, before this becomes the default agent path for high-resolution footage.

Important: skills/media-use/scripts/dither.test.mjs:126-145 asks ffprobe for frame=pts_time. On FFmpeg 4.2.7 the two keyframes are present but that field is omitted, so the test parses blank output as [NaN]; pkt_pts_time and best_effort_timestamp_time both report 0, 1. Use the portable timestamp field so the regression guard works across supported developer FFmpeg versions. Local focused result: 17 tests passed, this one assertion failed; the exact-head GitHub checks are green.

Ponytail: packages/cli/src/commands/color-grading.ts:558-574: yagni: reportMutation and runMutation are nested one-call wrappers whose only caller is the final branch. Inline the reporting block after prepareMutation() and remove the ReturnType plumbing. net: -12 lines possible.

— Magi

Verdict: APPROVE
Reasoning: No correctness or persistence-contract blocker found; the two important items are bounded performance/test-portability follow-ups, and exact-head CI is green.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additive review at d91c2f4b, weighted toward Ular's focus (duplication / deletable / over-complex / core contract-match). @Copilot commented (the media-treatment success message still says "color grading"; docs claim a color-grading alias the tests say isn't registered). Not repeating those; per Ular, 1/3 issues are being fixed and I'm not re-reporting them.

Good — the CLI is a real core consumer, not a re-implementation. applyColorGradingToHtml routes through core's normalizeHfColorGradingisHfColorGradingActiveserializeHfColorGrading and reuses patchElementInHtml for the mutation; the capability views are a presentation layer over getHfColorGradingCapabilities(). adjust/details/effects validation sets derive from the core registries (color-grading.ts:331-333). The --file path guard (isPathInside + .html, color-grading.ts:518) is correct.

Additive (duplication) — the same hard-coded-key pattern Magi blocked on #2752's lint rule recurs here, and it's rooted in a #2751 gap. GRADING_KEYS (color-grading.ts:320) and LUT_KEYS (:334) are hand-copied literals of the top-level HfColorGrading shape and HfColorGradingLutRef, right next to the three sibling sets that do derive from core. Core exports HF_COLOR_GRADING_{ADJUST,DETAIL,EFFECT}_KEYS but no top-level or lut registry, so every consumer re-hardcodes those two — the lint rule (media.ts, Magi's blocker) and now this CLI. They match today, but the next valid top-level/lut control silently becomes a hard CLI error (assertKnownGradingShape throws) with green CI. The clean fix is one #2751-owned HF_COLOR_GRADING_TOP_LEVEL_KEYS + HF_COLOR_GRADING_LUT_KEYS that both the lint rule and this CLI import — i.e. the #2752 fix should extend here rather than land twice. (Inline below.)

Same family, weaker: EFFECT_DISCOVERY_ROWS (:29-48) is satisfies-checked for membership (every key is a valid active effect) but not for completeness — a new core active effect wouldn't error, it'd just be silently absent from media-treatment --capabilities. An exhaustiveness assertion against HF_COLOR_GRADING_ACTIVE_EFFECT_KEYS would close it.

Verified — no runtime duplication in the offline dither tooling (a duplication Ular explicitly asked about). skills/media-use/scripts/lib/error-diffusion.mjs is CPU error-diffusion (Floyd–Steinberg / Atkinson / Stucki / …) for baking static assets; the runtime's dither effect (#2752) is a GPU ordered dither for live frames. Different algorithm class, execution context, and output — complementary, not overlapping. Its palette validation (2-6 × #rrggbb) matches core, and re-stating it there is correct since skill scripts are intentionally standalone (no workspace-build import). (@magi separately flagged a ~100 MB error-buffer allocation in the default dither path — a perf issue orthogonal to this duplication question.)

Verdict: COMMENT — Ready on the merits once the top-level/lut key source is unified with the #2752 fix (latent today, not blocking on its own). Clean, well-scoped consumer of the core contract. (Deferring the approving stamp to the maintainer.)

Review by Jerrai (hyperframes)

clear?: boolean;
}

const GRADING_KEYS = new Set([

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GRADING_KEYS (here) and LUT_KEYS (:334) are hand-copied literals of core's top-level HfColorGrading shape and HfColorGradingLutRef — the same hard-coded-schema pattern Magi flagged as a blocker on #2752's media.ts lint rule, just in a second consumer. The three sibling sets below (ADJUST_KEYS/DETAIL_KEYS/EFFECT_KEYS, :331-333) already derive from HF_COLOR_GRADING_*_KEYS; these two can't, because #2751 doesn't export a top-level or lut registry.

They match the contract today, but the next valid top-level/lut key becomes a hard assertKnownGradingShape error with green CI. Suggest fixing at the root: add HF_COLOR_GRADING_TOP_LEVEL_KEYS + HF_COLOR_GRADING_LUT_KEYS to core (#2751) and import them here and in the lint rule, so the #2752 fix covers both consumers instead of landing twice.

@ukimsanov
ukimsanov force-pushed the feat/media-treatment-agent branch from d91c2f4 to a96419f Compare July 24, 2026 06:30
@ukimsanov
ukimsanov force-pushed the feat/media-treatment-runtime branch from a2094eb to f9bf4d0 Compare July 24, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants