feat: add multimodal embed() activity and provider adapters

.changeset/embed-activity-multimodal.md

@@ -0,0 +1,14 @@
+---
+'@tanstack/ai': minor
+'@tanstack/ai-event-client': minor
+'@tanstack/ai-openai': minor
+'@tanstack/ai-gemini': minor
+'@tanstack/ai-mistral': minor
+'@tanstack/ai-bedrock': minor
+'@tanstack/ai-ollama': minor
+'@tanstack/ai-cohere': minor
+---
+
+Add a multimodal `embed()` activity. A single primitive covers one input or a batch — `input` accepts a string, a text part, an image part, or a fused `{ type: 'content' }` text+image item (one vector per item), with the accepted item types narrowed per model at compile time. Top-level `dimensions` requests Matryoshka output sizes where supported. Results carry `embeddings: [{ vector, index }]` plus `usage` when the provider reports it, and `embed()` participates in generation middleware, debug logging, OTel (`gen_ai.operation.name: embeddings`), and devtools events like every other activity.
+
+Provider adapters: `openaiEmbedding` (text-embedding-3-small/large), `geminiEmbedding` (gemini-embedding-001), `mistralEmbedding` (mistral-embed, codestral-embed), `ollamaEmbedding` (nomic-embed-text and any local model), `bedrockEmbedding` (Titan Text V2, Titan Multimodal G1 with fused text+image, Cohere Embed v3 on Bedrock), and the new `@tanstack/ai-cohere` package's `cohereEmbedding` (embed-v4.0, multimodal text+image with required `inputType`).

docs/adapters/bedrock.md

@@ -204,6 +204,70 @@ for await (const chunk of chat({
 }
 ```
 
+## Embeddings
+
+Generate embedding vectors with Titan or Cohere embedding models via `InvokeModel`:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const result = await embed({
+  adapter: bedrockEmbedding("amazon.titan-embed-text-v2:0"),
+  input: ["a red guitar", "a blue drum kit"],
+  dimensions: 512, // 256 | 512 | 1024
+});
+
+console.log(result.embeddings[0]?.vector);
+```
+
+Titan Multimodal embeds text and images — alone or fused into a single vector:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const productPhoto = "iVBORw0KGgo..."; // base64 image data
+
+const result = await embed({
+  adapter: bedrockEmbedding("amazon.titan-embed-image-v1"),
+  input: {
+    type: "content",
+    content: [
+      { type: "text", content: "a red guitar" },
+      {
+        type: "image",
+        source: {
+          type: "data",
+          value: productPhoto,
+          mimeType: "image/png",
+        },
+      },
+    ],
+  },
+  dimensions: 1024, // 256 | 384 | 1024
+});
+```
+
+Cohere Embed v3 on Bedrock is also supported (text-only, batched, requires `inputType`):
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const result = await embed({
+  adapter: bedrockEmbedding("cohere.embed-english-v3"),
+  input: ["a red guitar", "a blue drum kit"],
+  modelOptions: { inputType: "search_document" },
+});
+```
+
+> Titan models have no batch API — a batch of N items runs as N `InvokeModel`
+> calls under a small concurrency cap. Titan Multimodal does not fetch remote
+> image URLs; pass base64 data (or a `data:` URI).
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Model Availability
 
 The adapter ships with a hand-seeded snapshot catalog (`src/model-catalog.generated.ts`) of confirmed model IDs. This catalog can be refreshed by the maintainer script `scripts/fetch-bedrock-models.ts`, which calls `ListFoundationModels` with AWS credentials.

docs/adapters/cohere.md

@@ -0,0 +1,161 @@
+---
+title: Cohere
+id: cohere-adapter
+order: 18
+description: "Use Cohere's embed-v4.0 multimodal embedding model with TanStack AI via @tanstack/ai-cohere — text and image embeddings for semantic search and RAG."
+keywords:
+  - tanstack ai
+  - cohere
+  - embed-v4
+  - embeddings
+  - multimodal embeddings
+  - semantic search
+  - adapter
+---
+
+The Cohere adapter provides access to Cohere's embed-v4.0 multimodal embedding model — text, images, and fused text+image inputs, each producing a single vector.
+
+## Installation
+
+```bash
+npm install @tanstack/ai-cohere
+```
+
+## Basic Usage
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: ["a red guitar", "a blue drum kit"],
+  modelOptions: { inputType: "search_document" },
+});
+
+console.log(result.embeddings[0]?.vector);
+console.log(result.usage?.promptTokens);
+```
+
+`inputType` is required by Cohere's API — use `search_document` at index time and `search_query` at query time (or `classification` / `clustering` for those workloads). TanStack AI enforces this at the type level: `modelOptions` is required for Cohere embedding calls.
+
+## Basic Usage - Custom API Key
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { createCohereEmbedding } from "@tanstack/ai-cohere";
+
+const adapter = createCohereEmbedding("embed-v4.0", process.env.MY_COHERE_KEY!);
+
+const result = await embed({
+  adapter,
+  input: "a red guitar",
+  modelOptions: { inputType: "search_query" },
+});
+```
+
+## Multimodal Embeddings
+
+embed-v4.0 embeds images alongside text. An image part produces an image vector; a `{ type: "content" }` item fuses text and image into one vector — ideal for product catalogs and screenshot search:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const productPhoto = "iVBORw0KGgo..."; // base64 image data
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: [
+    {
+      type: "image",
+      source: {
+        type: "data",
+        value: productPhoto,
+        mimeType: "image/png",
+      },
+    },
+    {
+      type: "content",
+      content: [
+        { type: "text", content: "Fender Stratocaster, sunburst finish" },
+        {
+          type: "image",
+          source: {
+            type: "data",
+            value: productPhoto,
+            mimeType: "image/png",
+          },
+        },
+      ],
+    },
+  ],
+  modelOptions: { inputType: "search_document" },
+});
+
+console.log(result.embeddings.length); // 2
+```
+
+Cohere's API does not fetch remote image URLs. Pass base64 data (or a `data:` URI), or opt into adapter-side downloading:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const adapter = cohereEmbedding("embed-v4.0", { allowUrlFetch: true });
+
+const result = await embed({
+  adapter,
+  input: {
+    type: "image",
+    source: { type: "url", value: "https://example.com/guitar.png" },
+  },
+  modelOptions: { inputType: "search_document" },
+});
+```
+
+## Requesting Dimensions
+
+embed-v4.0 supports Matryoshka output dimensions via the top-level `dimensions` option:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: "a red guitar",
+  dimensions: 1024, // 256 | 512 | 1024 | 1536
+  modelOptions: { inputType: "search_document" },
+});
+```
+
+## Environment Variables
+
+Set your API key in environment variables:
+
+```bash
+COHERE_API_KEY=...
+```
+
+Get a key from the [Cohere dashboard](https://dashboard.cohere.com/api-keys).
+
+## API Reference
+
+### `cohereEmbedding(model, config?)`
+
+Creates an embedding adapter using `COHERE_API_KEY` from the environment.
+
+- `model`: `"embed-v4.0"`
+- `config.baseUrl`: override the API base URL (default `https://api.cohere.com`)
+- `config.headers`: extra request headers
+- `config.allowUrlFetch`: download `http(s)` image URLs and inline them as base64 (default `false`)
+
+### `createCohereEmbedding(model, apiKey, config?)`
+
+Same as `cohereEmbedding` with an explicit API key.
+
+## Next Steps
+
+- [Embeddings guide](../embeddings.md) — the full `embed()` API
+- [Generation Hooks](../media/generation-hooks.md) — usage and lifecycle middleware

docs/adapters/gemini.md

@@ -381,6 +381,32 @@ const result = await summarize({
 console.log(result.summary);
 ```
 
+## Embeddings
+
+Generate embedding vectors with gemini-embedding-001:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { geminiEmbedding } from "@tanstack/ai-gemini";
+
+const result = await embed({
+  adapter: geminiEmbedding("gemini-embedding-001"),
+  input: "a red guitar",
+  dimensions: 1536,
+  modelOptions: {
+    taskType: "RETRIEVAL_DOCUMENT",
+  },
+});
+
+console.log(result.embeddings[0]?.vector);
+```
+
+> The Gemini API's embedding endpoint does not report token usage, so
+> `result.usage` is absent. The Vertex-only multimodal embedding model
+> (`multimodalembedding@001`) is not supported by this adapter.
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Image Generation
 
 The Gemini adapter supports two types of image generation:

docs/adapters/mistral.md

@@ -262,6 +262,37 @@ const stream = chat({
 > All sampling parameters — including `temperature`, `top_p`, and `max_tokens` —
 > go inside `modelOptions` using Mistral's native (snake_case) names.
 
+## Embeddings
+
+Generate embedding vectors with mistral-embed or codestral-embed:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { mistralEmbedding } from "@tanstack/ai-mistral";
+
+const result = await embed({
+  adapter: mistralEmbedding("mistral-embed"),
+  input: ["a red guitar", "a blue drum kit"],
+});
+
+console.log(result.embeddings[0]?.vector); // 1024 dimensions
+```
+
+`mistral-embed` has fixed 1024-dimension output; `codestral-embed` (tuned for code) supports the top-level `dimensions` option:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { mistralEmbedding } from "@tanstack/ai-mistral";
+
+const result = await embed({
+  adapter: mistralEmbedding("codestral-embed"),
+  input: "function add(a, b) { return a + b }",
+  dimensions: 512,
+});
+```
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Environment Variables
 
 Set your API key in environment variables:

docs/adapters/ollama.md

@@ -218,6 +218,26 @@ const result = await summarize({
 console.log(result.summary);
 ```
 
+## Embeddings
+
+Generate embedding vectors locally with any Ollama embedding model:
+
+```typescript ignore
+import { embed } from "@tanstack/ai";
+import { ollamaEmbedding } from "@tanstack/ai-ollama";
+
+const result = await embed({
+  adapter: ollamaEmbedding("nomic-embed-text"),
+  input: ["a red guitar", "a blue drum kit"],
+});
+
+console.log(result.embeddings[0]?.vector);
+```
+
+Known models (`nomic-embed-text`, `mxbai-embed-large`, `all-minilm`, `snowflake-arctic-embed`, `bge-m3`, `embeddinggemma`) get autocomplete, and any other model name is accepted. Pull the model first with `ollama pull nomic-embed-text`. Output dimensions are fixed per model — the top-level `dimensions` option is not supported.
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Setting Up Ollama
 
 ### 1. Install Ollama

docs/adapters/openai.md

@@ -214,6 +214,38 @@ const result = await summarize({
 console.log(result.summary);
 ```
 
+## Embeddings
+
+Generate embedding vectors with the text-embedding-3 models:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { openaiEmbedding } from "@tanstack/ai-openai";
+
+const result = await embed({
+  adapter: openaiEmbedding("text-embedding-3-small"),
+  input: ["a red guitar", "a blue drum kit"],
+});
+
+console.log(result.embeddings[0]?.vector);
+console.log(result.usage?.promptTokens);
+```
+
+Both models support Matryoshka dimension reduction via the top-level `dimensions` option:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { openaiEmbedding } from "@tanstack/ai-openai";
+
+const result = await embed({
+  adapter: openaiEmbedding("text-embedding-3-large"),
+  input: "a red guitar",
+  dimensions: 1024,
+});
+```
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Image Generation
 
 Generate images with DALL-E: