CCM-33801: capture LiteLLM streaming responses in spans

[shreyas70]

Summary

The Harness SDK LiteLLM instrumentation did not capture response telemetry for streaming (stream=True) calls. The litellm_request span was ended as soon as completion() / acompletion() returned — but for streaming, the return value is a not-yet-consumed CustomStreamWrapper. Usage, response id, finish reasons, and content only exist after the stream is iterated, so streaming spans lost all response-side attributes.

This mirrors the deferred-span pattern already used by the OpenAI/Anthropic instrumentors in this SDK.

What changed

• _StreamSpanWrapper (a transparent wrapt.ObjectProxy) now wraps the returned stream. It forwards every chunk to the caller unchanged while accumulating them, and on stream exhaustion (or error) rebuilds the aggregated response via litellm.stream_chunk_builder, copies response metadata onto the span, and ends it.
• _LiteLLMSpanRun.wrap_stream() hands span ownership to the proxy: it detaches the active context and clears the re-dispatch guard without ending the span, so span lifecycle is owned by the stream.
• The sync and async wrappers detect a streaming response (_is_stream_response) and route through the proxy; non-streaming behavior is unchanged.
• A __del__ safety net ends the span if a consumer abandons the stream early.

Why the fix is at the SDK level

The consuming service already injects stream_options={"include_usage": True} and fully consumes the stream. The gap was purely that the SDK closed the span before those chunks were read.

Test plan

• test_litellm_streaming_span_defers_until_consumed — sync: span is not finished before consumption; after consumption the span carries gen_ai.request.streaming=True and aggregated gen_ai.response.finish_reasons.
• test_litellm_async_streaming_span_defers_until_consumed — async equivalent.
• Full LiteLLM instrumentation suite passes (17 tests) — existing non-streaming behavior unchanged.

Made with Cursor

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}