Skip to content

feat(google): add videoMetadata support for file parts#12755

Open
Pankaj3112 wants to merge 1 commit intovercel:mainfrom
Pankaj3112:feat/google-video-metadata
Open

feat(google): add videoMetadata support for file parts#12755
Pankaj3112 wants to merge 1 commit intovercel:mainfrom
Pankaj3112:feat/google-video-metadata

Conversation

@Pankaj3112
Copy link
Copy Markdown

Background

Google Generative AI supports videoMetadata on file parts (fileData/inlineData) for specifying video clipping offsets and frame rate. Currently the AI SDK's Google provider has no way to pass this metadata through to the API.

Ref: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1beta1/Content

Summary

  • Added GoogleGenerativeAIVideoMetadata type (startOffset, endOffset, fps)
  • Updated GoogleGenerativeAIContentPart union to allow videoMetadata on fileData and inlineData variants
  • Extract videoMetadata from providerOptions.google on file parts in the message conversion, using the same Vertex/Google fallback pattern as thoughtSignature
  • Added documentation example for video clipping with YouTube URLs

Usage

const result = await generateText({
  model: google('gemini-2.5-flash'),
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Summarize this part of the video' },
        {
          type: 'file',
          data: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
          mediaType: 'video/mp4',
          providerOptions: {
            google: {
              videoMetadata: {
                startOffset: '60s',
                endOffset: '600s',
                fps: 1,
              },
            },
          },
        },
      ],
    },
  ],
});

Checklist

  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Closes #8748

@tigent tigent Bot added ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label feature New feature or request provider/google Issues related to the @ai-sdk/google provider labels Feb 21, 2026
@Pankaj3112
Copy link
Copy Markdown
Author

@aayush-kapoor hi, would love a review on this when you get a chance!

Allow users to pass videoMetadata (startOffset, endOffset, fps) via
providerOptions on file parts to control video clipping and frame rate
when using the Google Generative AI provider.

Closes vercel#8748
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label feature New feature or request provider/google Issues related to the @ai-sdk/google provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File part metadata support for Google Generative AI

1 participant