feat: add multimodal embed() activity and provider adapters#926
Open
jherr wants to merge 9 commits into
Open
Conversation
Introduce a typed embed() primitive with per-model input modalities, Matryoshka dimensions, and adapters for OpenAI, Gemini, Mistral, Ollama, Bedrock, and a new @tanstack/ai-cohere package. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughAdds the shared ChangesEmbedding platform
Estimated code review effort: 5 (Critical) | ~90 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agentsroom/agents.json:
- Around line 7-9: Remove the tracked generated files under .agentsroom,
including agents.json and prompts.json, from version control while leaving the
existing ignore rule intact so future state files remain untracked.
In `@docs/config.json`:
- Around line 309-313: Add an updatedAt field with value "2026-07-10" to both
new documentation entries identified by the "Embeddings" and "Cohere" labels in
docs/config.json, alongside their existing addedAt fields.
In `@docs/embeddings.md`:
- Around line 113-116: Add the missing trailing comma after the text object in
the multimodal example’s content array, before the following image object, so
the array is valid TypeScript.
In `@packages/ai-cohere/src/adapters/embedding.ts`:
- Around line 140-151: Add configurable AbortController-based timeouts with a
sensible default to both the Cohere API fetch in the embedding request method
and the image URL fetch. Create and wire an abort signal for each request,
ensure the timer is cleared after completion, and preserve any existing
caller-provided cancellation behavior if supported by the adapter configuration.
- Around line 228-245: Validate image URLs before fetching when allowUrlFetch is
enabled: parse the value in the URL-fetch branch and reject loopback, private,
link-local, and other non-public IP addresses (including equivalent IPv6 ranges)
before calling fetch in the image URL handling logic. Preserve the existing
error behavior for disallowed fetching and fetch failures, and ensure malformed
or unsupported URLs are rejected clearly.
In `@packages/ai-gemini/src/adapters/embedding.ts`:
- Around line 48-51: Update the Gemini embedding adapter constructor to call
super(model, {}) instead of passing config, so BaseEmbeddingAdapter does not
retain the Gemini apiKey in this.config while createGeminiClient(config) still
receives the credentials.
In `@packages/ai/src/activities/embed/adapter.ts`:
- Around line 108-110: Resolve the generateId signature mismatch by either
updating generateId to accept and use a prefix parameter consistent with
Cohere’s generateId(this.name) call and embed/index.ts createId(prefix), or
changing the downstream call to match the no-argument API. Also replace
substring(7) with the consistent slice(2, 9) random suffix logic.
In `@packages/ai/src/activities/embed/index.ts`:
- Around line 216-226: Add the optional modelOptions field to the event payload
interfaces in the ai-event-client index, specifically
EmbeddingRequestStartedEvent, EmbeddingRequestCompletedEvent, and
EmbeddingRequestErrorEvent, using the appropriate existing model-options type.
Keep the modelOptions fields in all corresponding emissions in the embedding
activity aligned with these interface definitions.
In `@testing/e2e/src/routes/api.embedding.ts`:
- Around line 47-68: Change the factories declaration to a type that allows
missing provider keys, such as an object index signature returning (() => any) |
undefined, while retaining the optional call on factories[provider] in the
embedding factory lookup. This preserves runtime safety for unsupported
providers and resolves the unnecessary optional-chain lint error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e692e98a-9142-4e54-b846-abe1a11d5aa4
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (71)
.agentsroom/.gitignore.agentsroom/agents.json.agentsroom/prompts.json.changeset/embed-activity-multimodal.mddocs/adapters/bedrock.mddocs/adapters/cohere.mddocs/adapters/gemini.mddocs/adapters/mistral.mddocs/adapters/ollama.mddocs/adapters/openai.mddocs/config.jsondocs/embeddings.mddocs/migration/migration-from-vercel-ai.mddocs/migration/migration.mdpackages/ai-bedrock/src/adapters/embedding.tspackages/ai-bedrock/src/embedding/embedding-provider-options.tspackages/ai-bedrock/src/index.tspackages/ai-bedrock/src/model-meta.tspackages/ai-bedrock/tests/embedding-adapter.test.tspackages/ai-cohere/README.mdpackages/ai-cohere/package.jsonpackages/ai-cohere/src/adapters/embedding.tspackages/ai-cohere/src/embedding/embedding-provider-options.tspackages/ai-cohere/src/index.tspackages/ai-cohere/src/model-meta.tspackages/ai-cohere/src/utils/client.tspackages/ai-cohere/tests/embedding-adapter.test.tspackages/ai-cohere/tsconfig.jsonpackages/ai-cohere/vite.config.tspackages/ai-event-client/src/index.tspackages/ai-gemini/src/adapters/embedding.tspackages/ai-gemini/src/embedding/embedding-provider-options.tspackages/ai-gemini/src/index.tspackages/ai-gemini/src/model-meta.tspackages/ai-gemini/tests/embedding-adapter.test.tspackages/ai-mistral/package.jsonpackages/ai-mistral/src/adapters/embedding.tspackages/ai-mistral/src/embedding/embedding-provider-options.tspackages/ai-mistral/src/index.tspackages/ai-mistral/src/model-meta.tspackages/ai-mistral/tests/embedding-adapter.test.tspackages/ai-ollama/src/adapters/embedding.tspackages/ai-ollama/src/embedding/embedding-provider-options.tspackages/ai-ollama/src/index.tspackages/ai-ollama/src/model-meta.tspackages/ai-ollama/tests/embedding-adapter.test.tspackages/ai-openai/src/adapters/embedding.tspackages/ai-openai/src/embedding/embedding-provider-options.tspackages/ai-openai/src/index.tspackages/ai-openai/src/model-meta.tspackages/ai-openai/tests/embedding-adapter.test.tspackages/ai/src/activities/embed/adapter.tspackages/ai/src/activities/embed/index.tspackages/ai/src/activities/index.tspackages/ai/src/activities/middleware/types.tspackages/ai/src/index.tspackages/ai/src/middlewares/otel.tspackages/ai/src/types.tspackages/ai/src/utilities/embedding-input.tspackages/ai/tests/embed-per-model-type-safety.test.tspackages/ai/tests/embed.test.tstesting/e2e/fixtures/embedding/basic.jsontesting/e2e/global-setup.tstesting/e2e/src/components/EmbeddingUI.tsxtesting/e2e/src/lib/feature-support.tstesting/e2e/src/lib/features.tstesting/e2e/src/lib/types.tstesting/e2e/src/routeTree.gen.tstesting/e2e/src/routes/$provider/$feature.tsxtesting/e2e/src/routes/api.embedding.tstesting/e2e/tests/embedding.spec.ts
Comment on lines
+7
to
+9
| "id": "agent-1776361243376-3sekdc", | ||
| "claudeSessionId": "96773a93-be2a-45a9-a732-ceb224d3d0e5" | ||
| } |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify whether .agentsroom files are tracked despite the root .gitignore rule
git ls-files .agentsroom/Repository: TanStack/ai
Length of output: 221
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== root .gitignore ==\n'
if [ -f .gitignore ]; then
cat -n .gitignore
else
echo 'No root .gitignore'
fi
printf '\n== .agentsroom/.gitignore ==\n'
if [ -f .agentsroom/.gitignore ]; then
cat -n .agentsroom/.gitignore
else
echo 'No .agentsroom/.gitignore'
fi
printf '\n== .agentsroom/agents.json ==\n'
cat -n .agentsroom/agents.jsonRepository: TanStack/ai
Length of output: 2498
Remove the tracked .agentsroom state — .agentsroom/agents.json:7-9 is still committed even though .agentsroom/ is ignored, and it includes a claudeSessionId. Untrack the generated .agentsroom files (including agents.json and prompts.json) so personal/session data stays out of git.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.agentsroom/agents.json around lines 7 - 9, Remove the tracked generated
files under .agentsroom, including agents.json and prompts.json, from version
control while leaving the existing ignore rule intact so future state files
remain untracked.
Comment on lines
+309
to
+313
| { | ||
| "label": "Embeddings", | ||
| "to": "embeddings", | ||
| "addedAt": "2026-07-10" | ||
| } |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add updatedAt to the new documentation entries.
The new Embeddings and Cohere pages have addedAt but no updatedAt, although their content is introduced in this change. Per the docs/config.json guideline, add updatedAt: "2026-07-10" to both entries.
Also applies to: 570-572
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/config.json` around lines 309 - 313, Add an updatedAt field with value
"2026-07-10" to both new documentation entries identified by the "Embeddings"
and "Cohere" labels in docs/config.json, alongside their existing addedAt
fields.
Source: Coding guidelines
Comment on lines
+113
to
+116
| type: "content", | ||
| content: [ | ||
| { type: "text", content: "Fender Stratocaster, sunburst finish" }, | ||
| { |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Add the missing comma in the multimodal example.
The text object at Line [115] is immediately followed by the image object, making the content array invalid TypeScript.
Proposed fix
- { type: "text", content: "Fender Stratocaster, sunburst finish" }
+ { type: "text", content: "Fender Stratocaster, sunburst finish" },📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| type: "content", | |
| content: [ | |
| { type: "text", content: "Fender Stratocaster, sunburst finish" }, | |
| { | |
| type: "content", | |
| content: [ | |
| { type: "text", content: "Fender Stratocaster, sunburst finish" }, | |
| { |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/embeddings.md` around lines 113 - 116, Add the missing trailing comma
after the text object in the multimodal example’s content array, before the
following image object, so the array is valid TypeScript.
Source: Coding guidelines
Comment on lines
+48
to
+51
| constructor(config: GeminiEmbeddingConfig, model: TModel) { | ||
| super(model, config) | ||
| this.client = createGeminiClient(config) | ||
| } |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check BaseEmbeddingAdapter class definition and constructor
rg -n 'class BaseEmbeddingAdapter' --type=ts -A 20
# Check if other Gemini adapters pass config to super (package convention check)
rg -n 'super\(' packages/ai-gemini/src/adapters/ --type=ts -A 1
# Check if the base class stores or exposes the second constructor argument
rg -n 'constructor' packages/ai/src/activities/embed/adapter.ts --type=ts -A 15Repository: TanStack/ai
Length of output: 149
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Files under packages/ai-gemini/src/adapters ==\n'
git ls-files 'packages/ai-gemini/src/adapters/**/*.ts' || true
printf '\n== Locate BaseEmbeddingAdapter ==\n'
rg -n 'class BaseEmbeddingAdapter|BaseEmbeddingAdapter' packages -g '*.ts' -A 20 -B 5 || true
printf '\n== Read Gemini embedding adapter ==\n'
sed -n '1,220p' packages/ai-gemini/src/adapters/embedding.ts || true
printf '\n== Read likely base embedding adapter file(s) ==\n'
rg -n 'constructor\(' packages/ai -g '*adapter*.ts' -A 20 -B 5 || trueRepository: TanStack/ai
Length of output: 50369
Pass {} to super() here
BaseEmbeddingAdapter stores the second constructor argument on this.config, so passing config keeps the Gemini apiKey on the adapter instance. Use super(model, {}) here to avoid retaining the secret, matching the other embedding adapters.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-gemini/src/adapters/embedding.ts` around lines 48 - 51, Update
the Gemini embedding adapter constructor to call super(model, {}) instead of
passing config, so BaseEmbeddingAdapter does not retain the Gemini apiKey in
this.config while createGeminiClient(config) still receives the credentials.
These personal agent config files should not ship in the embed PR. Co-authored-by: Cursor <cursoragent@cursor.com>
Align embedding event types with emitted modelOptions, harden generateId, fix e2e factory typing, and add Cohere fetch timeouts plus SSRF guards. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
embed()activity with per-model input modalities, Matryoshkadimensions, middleware/OTel/devtools integration, andembeddings: [{ vector, index }]results@tanstack/ai-coherepackage for embed-v4.0Test plan
pnpm test:pr(sherif, knip, docs, eslint, lib, types, build)@tanstack/aiand provider packagespnpm --filter @tanstack/ai-e2e test:e2e -- tests/embedding.spec.ts(4/4 passed)Made with Cursor
Summary by CodeRabbit
New Features
embed()activity for single or batch text and image inputs, including fused multimodal content.Documentation
Tests