feat(datasets): support multimodal dataset items

[wochinge]

Summary

• Adds generated API support for dataset item media references and trace-less media uploads.
• Adds SDK support for creating dataset items with LangfuseMedia in input, expected_output, and metadata.
• Adds optional get_dataset(..., resolve_media_references=True) hydration to replace backend-reported media reference JSONPaths with LangfuseMediaReference objects.
• Adds focused unit coverage and an e2e test using a real JPEG fixture.

Linear

• LFE-10289

Major Decisions

• Keeps get_dataset(...) as the only high-level SDK read surface for resolved dataset media references.
• Uses backend-provided JSONPaths for read hydration and jsonpath-ng recursive discovery for create-path media replacement.
• Splits generated API updates and handwritten SDK changes into separate commits to simplify review.

Review Focus

• Dataset media upload behavior without trace/observation context.
• JSONPath-based replacement for both create and read paths.
• Public SDK surface: LangfuseMediaReference export and get_dataset(..., resolve_media_references=True).

Greptile Summary

This PR adds multimodal support to Langfuse dataset items, allowing LangfuseMedia objects to be embedded in input, expected_output, and metadata fields and uploaded trace-lessly, and adding a resolve_media_references=True option to get_dataset() that hydrates backend-provided JSONPath references into LangfuseMediaReference objects.

• create_dataset_item now recursively discovers LangfuseMedia values via $..this`` (jsonpath-ng), uploads each one synchronously through the updated _upload_media_sync (which accepts `trace_id=None`), deduplicates by `media_id` within a single call, and replaces each value with its reference string before sending to the API.
• get_dataset(resolve_media_references=True) passes include_media_references to the list endpoint, then walks backend-provided JSONPath annotations to replace reference strings with frozen LangfuseMediaReference dataclass instances that expose fetch_bytes(), fetch_base64(), and fetch_data_uri() helpers.
• Generated API types (DatasetItemMediaReference, DatasetItemMediaReferenceField, DatasetItemMediaReferenceMedia) and the GetMediaUploadUrlRequest schema are updated to make trace_id and field optional, enabling trace-less uploads.

Confidence Score: 4/5

Safe to merge; all changed paths work correctly for the expected JSON-like data shapes, and the two main concerns are style/fragility rather than broken behavior.

The media-upload and hydration flows are logically sound and covered by both unit and E2E tests. The two notable findings are: _replace_json_path_value discards the return value of update() and relies implicitly on in-place mutation (works today, diverges from how _process_dataset_item_media handles the same operation); and fetch_bytes() makes a blocking HTTP call with no timeout. Neither produces wrong output in the current code paths, but both leave the door open for subtle failures on refactoring or slow network conditions.

langfuse/_client/client.py (_replace_json_path_value) and langfuse/media.py (LangfuseMediaReference.fetch_bytes) are the two spots that would benefit from a second look before shipping to customers at scale.

Sequence Diagram

sequenceDiagram
    participant User
    participant Langfuse SDK
    participant MediaManager
    participant Langfuse API
    participant Object Storage

    Note over User,Object Storage: create_dataset_item with LangfuseMedia

    User->>Langfuse SDK: create_dataset_item(input={"img": LangfuseMedia(...)})
    Langfuse SDK->>Langfuse SDK: _process_dataset_item_media()<br/>jsonpath-ng discovers LangfuseMedia nodes
    Langfuse SDK->>MediaManager: _upload_media_sync(media)
    MediaManager->>Langfuse API: media.get_upload_url(trace_id=None, field=None)
    Langfuse API-->>MediaManager: upload_url + media_id
    MediaManager->>Object Storage: PUT bytes
    Object Storage-->>MediaManager: 200 OK
    MediaManager->>Langfuse API: media.patch(media_id, upload_http_status=200)
    MediaManager-->>Langfuse SDK: done
    Langfuse SDK->>Langfuse API: dataset_items.create(input={"img": "@@@langfuseMedia:..."})
    Langfuse API-->>Langfuse SDK: DatasetItem
    Langfuse SDK-->>User: DatasetItem (reference strings)

    Note over User,Object Storage: get_dataset with resolve_media_references=True

    User->>Langfuse SDK: get_dataset(name, resolve_media_references=True)
    Langfuse SDK->>Langfuse API: dataset_items.list(includeMediaReferences=true)
    Langfuse API-->>Langfuse SDK: items with mediaReferences[] (JSONPaths + signed URLs)
    loop For each item
        Langfuse SDK->>Langfuse SDK: _hydrate_dataset_item_media_references()<br/>jsonpath-ng update per JSONPath to LangfuseMediaReference
    end
    Langfuse SDK-->>User: DatasetClient (LangfuseMediaReference in fields)
    User->>Langfuse SDK: item.input["img"].fetch_bytes()
    Langfuse SDK->>Object Storage: httpx.get(signed_url)
    Object Storage-->>Langfuse SDK: bytes
    Langfuse SDK-->>User: bytes

Loading

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
langfuse/_client/client.py:3418-3426
`_replace_json_path_value` silently discards the return value of `update()`, relying entirely on in-place mutation of `value`. This works today because API-deserialized data is always a mutable `dict` or `list`, but it diverges from the pattern used in `_process_dataset_item_media` (which correctly captures `data = match.full_path.update(data, ...)`). If `value` were ever an immutable type at a non-root path, the replacement would be silently lost without any log or exception.

```suggestion
        try:
            value = parse_jsonpath(json_path).update(value, replacement)
        except Exception as e:
            langfuse_logger.debug(
                f"Failed to hydrate dataset media reference at JSONPath {json_path}",
                exc_info=e,
            )

        return value
```

### Issue 2 of 3
langfuse/media.py:50-53
`httpx.get(self.url)` is called without a timeout, so it can block indefinitely if the signed URL is slow or the connection hangs. This is especially relevant in hot paths such as evaluation loops that call `fetch_bytes()` on each dataset item's resolved media.

```suggestion
    def fetch_bytes(self, timeout: float = 30.0) -> bytes:
        """Fetch the media content from the signed URL."""
        response = httpx.get(self.url, timeout=timeout)
        response.raise_for_status()
```

### Issue 3 of 3
langfuse/_task_manager/media_manager.py:233-260
**Silent return on incomplete media is undocumented**

`_upload_media_sync` silently returns `None` when any content field is `None`. The caller `_upload_dataset_item_media` then adds `media_id` to `uploaded_media_ids` and returns the reference string as if the upload succeeded — meaning the dataset item would store a reference to media that was never uploaded.

In practice this path is currently unreachable: the earlier check `if reference_string is None or media_id is None: raise ValueError(...)` in `_upload_dataset_item_media` ensures all content fields are populated before this method is called (because `_media_id` is derived from `_content_sha256_hash`, which requires `_content_bytes`). But the silent return offers no signal to future callers that skip that guard, and no explanation is left in a comment. A guarded `raise` or at minimum a comment explaining the invariant would make this safer.

_{Reviews (1): Last reviewed commit: "feat(datasets): support media references" | Re-trigger Greptile}

Greptile also left 3 inline comments on this PR.

[github-actions]

@claude review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd791b64ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[wochinge]

actual (non generated changes)

[wochinge]

in my opinion an okay compromise as the write path doesn't become a hard blocker for users and easy to maintain

Avoid hand rolling jsonpath parsing for the read path -> users can also opt out here so in my opinion fine

[wochinge]

actual (non generated changes)

[wochinge]

actual (non generated changes)

[wochinge]

actual (non generated changes)

[wochinge]

Apache 2.0 license, 731 stars, actively maintained: https://github.com/h2non/jsonpath-ng
Annoying thing is the issues about running optimized python but I think we addressed them enough: https://github.com/h2non/jsonpath-ng#special-note-about-ply-and-docstrings

Next runner up would be jsonpath-python (https://pypi.org/project/python-jsonpath/)

• MIT license
• only 69 stars

I'd avoid hand rolling the parsing. Especially since we have find references via the TS jsonpath library we don't know exactly which syntax might get in there and which cases of the spec we need to handle.