(Openinference Migration: Langchain): Capture multimodal image content (OpenAI `image_url` and Anthropic `image` blocks) as `Blob`/`Uri` message parts. by rads-1996 · Pull Request #296 · open-telemetry/opentelemetry-python-genai

rads-1996 · 2026-07-21T19:59:22Z

Description

Part of the langchain migration PRs - #272

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.

uv run tox -e py312-test-instrumentation-genai-langchain -- -q
uv run tox -e py312-test-instrumentation-genai-langchain-conformance -- -q
uv run --python 3.12 tox -e lint-instrumentation-genai-langchain

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-07-21T20:00:07Z

Pull request dashboard status

Waiting on the author · refreshed 2026-08-07 17:26 UTC

Two things need attention:

Required checks are failing — investigate the failures.
4 review items — respond to each (e.g. link a commit, explain why not, ask a follow-up):
- Inline threads: 1, 2, 3
- Top-level threads: 4

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

Pull request overview

Adds multimodal image support to the LangChain GenAI instrumentation by converting OpenAI image_url and Anthropic image blocks into Blob / Uri message parts, and extends the unit test suite to validate the new parsing behavior.

Changes:

Add multimodal image parsing helpers (_media_part, _image_from_url) to convert LangChain image blocks into Blob/Uri parts.
Extend callback-handler tests to cover OpenAI and Anthropic image content shapes (including data URIs and base64 sources).
Add a changelog fragment documenting the new capability.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/utils.py	Adds image block parsing and base64 decoding to emit `Blob`/`Uri` message parts.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_callback_handler.py	Adds unit tests for data-URI, HTTP-URI, and Anthropic `image/source` parsing into message parts.
instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/296.added	Documents the new multimodal image capture support.

AgentGymLeader · 2026-07-30T20:36:55Z

@rads-1996 One thing I wanted to check on ordering. In on_chat_model_start, to_input_messages() runs before anything consults TelemetryHandler.should_capture_content(), so with the default NO_CONTENT the data URI and Anthropic base64 images still get decoded to bytes via _decode_base64.

That means a config that doesn't record content still copies large images. If that isn't intended, either skip the media decode when capture is off, or add a test asserting _decode_base64 isn't called under NO_CONTENT.

If the parse is meant to run unconditionally so the completion hook has something to work with, that's fine, I just wanted to confirm that's the assumption.

rads-1996 · 2026-07-30T23:46:53Z

@rads-1996 One thing I wanted to check on ordering. In on_chat_model_start, to_input_messages() runs before anything consults TelemetryHandler.should_capture_content(), so with the default NO_CONTENT the data URI and Anthropic base64 images still get decoded to bytes via _decode_base64.

That means a config that doesn't record content still copies large images. If that isn't intended, either skip the media decode when capture is off, or add a test asserting _decode_base64 isn't called under NO_CONTENT.

If the parse is meant to run unconditionally so the completion hook has something to work with, that's fine, I just wanted to confirm that's the assumption.

@AgentGymLeader Thanks for the callout. I have gated decoding and will skipped when the content capture mode is NO_CONTENT.

…mage` blocks) as `Blob`/`Uri` message parts.

lmolkova · 2026-08-06T22:24:25Z

can you please add conformance test? thanks!

lmolkova · 2026-08-06T22:25:52Z

@@ -0,0 +1 @@
+(Openinference Migration: Langchain) - Capture multimodal image content (OpenAI ``image_url`` and Anthropic ``image`` blocks) as ``Blob``/``Uri`` message parts.


please update openai and anthropic instrumentations to hahve feature parity. Ok to do in a separate PR. Thanks!

lmolkova · 2026-08-06T22:28:09Z

+def _decode_base64(data: str) -> bytes | None:
+    # Skip the decode entirely when message content is not being captured;
+    # the resulting bytes would never be emitted under ``NO_CONTENT``.
+    if get_content_capturing_mode() is ContentCapturingMode.NO_CONTENT:


this one actually checks env var and is now used on a hot path, please use handler.should_capture_content() somewhere up the stack and don't call this method when content capturing is disabled

lmolkova · 2026-08-06T22:37:39Z

+    return None
+
+
+def _image_from_url(url: str) -> MessagePart | None:


should we have this whole method in utils? the data:// is pretty generic approach, nothing specific to openai https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data

Yeah make sense, I can then reuse it for openai and anthropic.

lmolkova · 2026-08-06T22:41:50Z

    gen_ai_attributes as GenAIAttributes,
 )
 from opentelemetry.util.genai.types import (
+    Blob,


it's the first time we're using this type in this repo, mind renaming it to BlobPart? Same with Uri, can we rename it to UriPart - that's how they are called in semconv and it seems we're inconsistent - https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/non-normative/models.py

Sounds good. I will make changes in the utils. I believe the same change is needed for some other data classes as well. I think I will make a separate PR to address that.

@lmolkova Blob and Uri both have previously been used in other instrumentations, Blob has been used in the anthropic and google-genai instrumentation and Uri in google-genai. I have a PR out which aligns the names with semconv - #365. Should we have backward compatibility aliases for downstream users with deprecation messages?

opentelemetry-pr-dashboard Bot mentioned this pull request Jul 21, 2026

Pull Request Dashboard #171

Open

rads-1996 marked this pull request as ready for review July 21, 2026 21:18

rads-1996 requested a review from a team as a code owner July 21, 2026 21:18

Copilot AI review requested due to automatic review settings July 21, 2026 21:18

rads-1996 marked this pull request as draft July 21, 2026 21:18

Copilot started reviewing on behalf of rads-1996 July 21, 2026 21:18 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread ...y-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/utils.py Outdated

Comment thread instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/296.added

rads-1996 force-pushed the langchain-multimodal-images branch 3 times, most recently from aa26c4f to ed3949d Compare July 23, 2026 21:41

rads-1996 force-pushed the langchain-multimodal-images branch 2 times, most recently from d72b64e to 501725d Compare July 30, 2026 16:54

rads-1996 marked this pull request as ready for review July 30, 2026 17:20

rads-1996 force-pushed the langchain-multimodal-images branch from 9cd26fe to 1b98bc4 Compare July 31, 2026 16:50

lmolkova mentioned this pull request Aug 3, 2026

openai instrumentation should have parity with langchain openai wrapper #348

Open

rads-1996 force-pushed the langchain-multimodal-images branch from aaea6b5 to d9fe119 Compare August 3, 2026 15:15

lzchen approved these changes Aug 3, 2026

View reviewed changes

rads-1996 force-pushed the langchain-multimodal-images branch from d9fe119 to a61da13 Compare August 4, 2026 22:16

rads-1996 added 8 commits August 5, 2026 15:59

Capture multimodal image content (OpenAI image_url and Anthropic `i…

26dbd61

…mage` blocks) as `Blob`/`Uri` message parts.

Update CHANGELOG number

90a93df

Fix typecheck errors

23e1d0c

Address feedback

24e1696

Fix formatting

4e3cd2f

Add recording tests

eb3f93b

Address feedback

fa3a768

Fix format per new ruff version

0c19a81

rads-1996 force-pushed the langchain-multimodal-images branch from 9ae1c71 to 0c19a81 Compare August 5, 2026 23:05

Merge branch 'main' into langchain-multimodal-images

eb9e373

lmolkova reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

(Openinference Migration: Langchain): Capture multimodal image content (OpenAI `image_url` and Anthropic `image` blocks) as `Blob`/`Uri` message parts. - #296

(Openinference Migration: Langchain): Capture multimodal image content (OpenAI `image_url` and Anthropic `image` blocks) as `Blob`/`Uri` message parts.#296
rads-1996 wants to merge 9 commits into
open-telemetry:mainfrom
rads-1996:langchain-multimodal-images

rads-1996 commented Jul 21, 2026 •

edited

Loading

Uh oh!

opentelemetry-pr-dashboard Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

AgentGymLeader commented Jul 30, 2026

Uh oh!

rads-1996 commented Jul 30, 2026

Uh oh!

lmolkova Aug 6, 2026

Uh oh!

lmolkova Aug 6, 2026

Uh oh!

lmolkova Aug 6, 2026

Uh oh!

lmolkova Aug 6, 2026

Uh oh!

rads-1996 Aug 6, 2026

Uh oh!

lmolkova Aug 6, 2026

Uh oh!

rads-1996 Aug 6, 2026 •

edited

Loading

Uh oh!

rads-1996 Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		@@ -0,0 +1 @@
		(Openinference Migration: Langchain) - Capture multimodal image content (OpenAI ``image_url`` and Anthropic ``image`` blocks) as ``Blob``/``Uri`` message parts.

		return None


		def _image_from_url(url: str) -> MessagePart \| None:

Uh oh!

Conversation

rads-1996 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

AgentGymLeader commented Jul 30, 2026

Uh oh!

rads-1996 commented Jul 30, 2026

Uh oh!

lmolkova Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rads-1996 Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rads-1996 commented Jul 21, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Jul 21, 2026 •

edited

Loading

rads-1996 Aug 6, 2026 •

edited

Loading