From 61dd1b4be5f013144b6335d39877501359fde7e4 Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 14:20:03 +0530 Subject: [PATCH 1/6] docs: design non-consuming HTTPX response hooks Co-authored-by: Cursor --- ...tpx-non-consuming-response-hooks-design.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md diff --git a/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md b/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md new file mode 100644 index 0000000..383a238 --- /dev/null +++ b/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md @@ -0,0 +1,37 @@ +# HTTPX Non-Consuming Response Hooks + +## Problem + +The HTTPX response hooks synchronously drain the transport response stream to +capture its body, then replace HTTPX's private `_httpcore_stream` state with a +custom replay iterator. This buffers streaming responses, depends on private +HTTPX internals, and can leave a partially consumed stream when draining fails. + +## Design + +HTTPX response hooks will capture response headers and status without reading +the response stream. They will pass `None` as the response body to the generic +response handler. + +The response-draining helpers and replay stream will be removed. Request-body +capture remains unchanged. + +## Compatibility + +HTTPX spans will no longer contain `http.response.body`. Existing response +header and status attributes remain. This intentional telemetry reduction +preserves application correctness and true streaming behavior. + +## Tests + +- A synchronous response hook must not iterate or mutate its response stream. +- An asynchronous response hook must not iterate or mutate its response stream. +- Existing HTTPX integration tests must continue validating request capture, + response headers, and status without expecting response-body capture. +- The complete unit suite must pass. + +## Non-Goals + +- Buffering, truncating, or sampling streaming response bodies. +- Extending span lifetime until application response consumption. +- Changing request-body capture or other instrumentations. From 8c9d82e7b3923986621b3d7c14c318db04638a5c Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 14:24:11 +0530 Subject: [PATCH 2/6] docs: plan non-consuming HTTPX response hooks Co-authored-by: Cursor --- ...7-17-httpx-non-consuming-response-hooks.md | 293 ++++++++++++++++++ 1 file changed, 293 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md diff --git a/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md b/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md new file mode 100644 index 0000000..8c549f8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md @@ -0,0 +1,293 @@ +# HTTPX Non-Consuming Response Hooks Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make synchronous and asynchronous HTTPX response hooks preserve headers and status telemetry without reading or replacing application response streams. + +**Architecture:** Keep request handling unchanged. Both response paths extract only headers and call `generic_response_handler(headers, None, span)`; OpenTelemetry continues owning status capture. Remove response-body decoding, draining, and replay code once no production caller remains. + +**Tech Stack:** Python 3.10+, pytest 7.4, pytest-asyncio, OpenTelemetry HTTPX instrumentation 0.62b1 + +## Global Constraints + +- Apply the change to all HTTPX responses, not only compressed or streaming responses. +- Preserve HTTPX request-body capture. +- Preserve HTTPX response headers and status. +- Remove `http.response.body` from HTTPX spans. +- Never iterate or mutate synchronous or asynchronous response streams. +- Make no changes to other instrumentations. +- Keep the existing design commit `61dd1b4` unchanged. + +--- + +### Task 1: Add response-stream regression tests + +**Files:** +- Create: `test/instrumentation/httpx_client/test_httpx_hooks.py` + +**Interfaces:** +- Consumes: `HTTPXClientInstrumentorWrapper.response_hook(span, request_info, response_info)` and `async_response_hook(span, request_info, response_info)`. +- Produces: focused regression coverage requiring response headers plus a `None` body while preserving the original `_httpcore_stream` object without iteration. + +- [ ] **Step 1: Write the synchronous failing test** + +```python +from types import SimpleNamespace + +import pytest + +from harness_sdk.instrumentation.httpx import HTTPXClientInstrumentorWrapper + + +class _SyncStream: + def __init__(self): + self.iterations = 0 + + def __iter__(self): + self.iterations += 1 + yield b"response" + + +class _AsyncStream: + def __init__(self): + self.iterations = 0 + + async def __aiter__(self): + self.iterations += 1 + yield b"response" + + +def _wrapper_with_response_capture(): + wrapper = HTTPXClientInstrumentorWrapper() + calls = [] + wrapper.generic_response_handler = ( + lambda headers, body, span: calls.append((headers, body, span)) + ) + return wrapper, calls + + +def test_response_hook_does_not_consume_or_replace_stream(): + wrapper, calls = _wrapper_with_response_capture() + span = object() + transport_stream = _SyncStream() + response_stream = SimpleNamespace(_httpcore_stream=transport_stream) + response_info = SimpleNamespace( + status_code=200, + headers={"content-type": "text/plain"}, + stream=response_stream, + extensions={}, + ) + + wrapper.response_hook(span, None, response_info) + + assert calls == [({"content-type": "text/plain"}, None, span)] + assert transport_stream.iterations == 0 + assert response_stream._httpcore_stream is transport_stream +``` + +- [ ] **Step 2: Write the asynchronous failing test** + +```python +@pytest.mark.asyncio +async def test_async_response_hook_does_not_consume_or_replace_stream(): + wrapper, calls = _wrapper_with_response_capture() + span = object() + transport_stream = _AsyncStream() + response_stream = SimpleNamespace(_httpcore_stream=transport_stream) + response_info = SimpleNamespace( + status_code=200, + headers={"content-type": "text/plain"}, + stream=response_stream, + extensions={}, + ) + + await wrapper.async_response_hook(span, None, response_info) + + assert calls == [({"content-type": "text/plain"}, None, span)] + assert transport_stream.iterations == 0 + assert response_stream._httpcore_stream is transport_stream +``` + +- [ ] **Step 3: Run tests to verify RED** + +Run: + +```bash +python3 -m pytest -q \ + test/instrumentation/httpx_client/test_httpx_hooks.py +``` + +Expected: both tests fail because current hooks iterate `_httpcore_stream`, replace it, and pass captured bytes instead of `None`. + +- [ ] **Step 4: Commit the regression tests** + +```bash +git add test/instrumentation/httpx_client/test_httpx_hooks.py +git commit -m "test: cover non-consuming HTTPX response hooks" +``` + +### Task 2: Stop response capture and remove dead helpers + +**Files:** +- Modify: `src/harness_sdk/instrumentation/httpx/__init__.py:8-73` +- Modify: `src/harness_sdk/instrumentation/httpx/utils.py:1-160` +- Delete: `test/instrumentation/httpx_client/test_httpx_utils.py` + +**Interfaces:** +- Consumes: `headers_from_httpx(response_info.headers)` and `generic_response_handler(response_headers, response_body, span)`. +- Produces: `_process_response(span, response_info) -> None`, used by both sync and async response hooks, always forwarding `None` as `response_body`. + +- [ ] **Step 1: Unify response processing without stream access** + +Replace response-specific imports and methods in `src/harness_sdk/instrumentation/httpx/__init__.py` with: + +```python +from harness_sdk.instrumentation.httpx.utils import ( + headers_from_httpx, + read_request_body, + url_from_request_info, +) + + def _process_response(self, span, response_info): + headers = headers_from_httpx(response_info.headers) + self.generic_response_handler(headers, None, span) + + async def async_response_hook(self, span, request_info, response_info): # pylint: disable=unused-argument + '''Capture async client response headers.''' + self._process_response(span, response_info) +``` + +Remove `_process_response_async`. Update the synchronous response-hook docstring to say it captures response headers. + +- [ ] **Step 2: Remove obsolete response helpers** + +Delete from `src/harness_sdk/instrumentation/httpx/utils.py`: + +```python +import gzip +import zlib + +_content_encoding_chain +_apply_content_encoding_decodings +decode_response_body_for_capture +read_response_body +read_response_body_async +_ReplayAsyncStream +``` + +Keep `headers_from_httpx`, `url_from_request_info`, `_body_from_byte_stream`, and `read_request_body` unchanged. Delete `test/instrumentation/httpx_client/test_httpx_utils.py` because every test in it covers the removed response-only decoder. + +- [ ] **Step 3: Run focused tests to verify GREEN** + +Run: + +```bash +python3 -m pytest -q \ + test/instrumentation/httpx_client/test_httpx_hooks.py +``` + +Expected: `2 passed`; neither stream reports iteration or replacement. + +- [ ] **Step 4: Confirm response helpers have no references** + +Run: + +```bash +rg "decode_response_body_for_capture|read_response_body|read_response_body_async|_ReplayAsyncStream" src test +``` + +Expected: no matches. + +- [ ] **Step 5: Commit implementation** + +```bash +git add src/harness_sdk/instrumentation/httpx/__init__.py \ + src/harness_sdk/instrumentation/httpx/utils.py \ + test/instrumentation/httpx_client/test_httpx_utils.py +git commit -m "fix: avoid consuming HTTPX response streams" +``` + +### Task 3: Update telemetry expectations and verify + +**Files:** +- Modify: `test/instrumentation/httpx_client/httpx_integration_test.py:19-127` + +**Interfaces:** +- Consumes: finished HTTPX client span attributes. +- Produces: integration assertions preserving request body, response headers, and status while requiring `http.response.body` to be absent. + +- [ ] **Step 1: Replace response-body assertions** + +In all three HTTPX integration tests, replace: + +```python +assert client_span['attributes']['http.response.body'] == '{ "a": "a", "xyz": "xyz" }' +``` + +or the equivalent `httpx_span` assertion with: + +```python +assert 'http.response.body' not in client_span['attributes'] +``` + +Use `httpx_span` in the async test. Retain all request-body, response-header, and status assertions. Add the existing `tester3` response-header assertion to the POST test so both POST response paths explicitly retain header coverage. + +- [ ] **Step 2: Run the complete HTTPX test directory** + +Run: + +```bash +python3 -m pytest -q test/instrumentation/httpx_client +``` + +Expected: all HTTPX tests pass. + +- [ ] **Step 3: Commit telemetry expectations** + +```bash +git add test/instrumentation/httpx_client/httpx_integration_test.py +git commit -m "test: update HTTPX response telemetry expectations" +``` + +- [ ] **Step 4: Run repository verification** + +Run: + +```bash +python3 -m compileall -q src/harness_sdk +pylint src/harness_sdk --disable=C,R --ignore-patterns=config_pb2.py +./scripts/run-unit-tests.sh +``` + +Expected: each command exits 0. The unit script may report only the repository's configured DB integration-test skips when `RUN_SDK_INTEGRATION_TESTS` is unset. + +- [ ] **Step 5: Review scope and final diff** + +Run: + +```bash +git diff --check origin/main...HEAD +git diff --stat origin/main...HEAD +git diff origin/main...HEAD -- \ + src/harness_sdk/instrumentation/httpx \ + test/instrumentation/httpx_client \ + docs/superpowers +git status --short +``` + +Expected: no whitespace errors; only the approved design, plan, HTTPX source, and HTTPX tests differ; working tree is clean. + +- [ ] **Step 6: Request code review and reverify findings** + +Give a reviewer `origin/main` as the base, current `HEAD` as the head, this plan, and the approved design spec. Fix only valid high-confidence correctness or scope findings, commit those fixes, then rerun the focused HTTPX directory plus any affected lint command. + +- [ ] **Step 7: Push and create the pull request** + +```bash +git push -u origin fix/httpx-non-consuming-response-hooks +gh pr create --base main \ + --title "fix: preserve HTTPX response streams during instrumentation" \ + --body-file /tmp/httpx-stream-safe-pr.md +``` + +The PR body must state the root cause, the intentional removal of `http.response.body` from HTTPX spans, retained headers/status/request capture, exact verification results, and that this removes unsafe stream consumption/private mutation implicated by instrumentation without claiming it alone proves the production gzip issue. From cfb527802df2c234434bb406361d5cce45967513 Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 14:25:47 +0530 Subject: [PATCH 3/6] test: cover non-consuming HTTPX response hooks Co-authored-by: Cursor --- .../httpx_client/test_httpx_hooks.py | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 test/instrumentation/httpx_client/test_httpx_hooks.py diff --git a/test/instrumentation/httpx_client/test_httpx_hooks.py b/test/instrumentation/httpx_client/test_httpx_hooks.py new file mode 100644 index 0000000..0c6205e --- /dev/null +++ b/test/instrumentation/httpx_client/test_httpx_hooks.py @@ -0,0 +1,71 @@ +from types import SimpleNamespace + +import pytest + +from harness_sdk.instrumentation.httpx import HTTPXClientInstrumentorWrapper + + +class _SyncStream: + def __init__(self): + self.iterations = 0 + + def __iter__(self): + self.iterations += 1 + yield b"response" + + +class _AsyncStream: + def __init__(self): + self.iterations = 0 + + async def __aiter__(self): + self.iterations += 1 + yield b"response" + + +def _wrapper_with_response_capture(): + wrapper = HTTPXClientInstrumentorWrapper() + calls = [] + wrapper.generic_response_handler = ( + lambda headers, body, span: calls.append((headers, body, span)) + ) + return wrapper, calls + + +def test_response_hook_does_not_consume_or_replace_stream(): + wrapper, calls = _wrapper_with_response_capture() + span = object() + transport_stream = _SyncStream() + response_stream = SimpleNamespace(_httpcore_stream=transport_stream) + response_info = SimpleNamespace( + status_code=200, + headers={"content-type": "text/plain"}, + stream=response_stream, + extensions={}, + ) + + wrapper.response_hook(span, None, response_info) + + assert calls == [({"content-type": "text/plain"}, None, span)] + assert transport_stream.iterations == 0 + assert response_stream._httpcore_stream is transport_stream + + +@pytest.mark.asyncio +async def test_async_response_hook_does_not_consume_or_replace_stream(): + wrapper, calls = _wrapper_with_response_capture() + span = object() + transport_stream = _AsyncStream() + response_stream = SimpleNamespace(_httpcore_stream=transport_stream) + response_info = SimpleNamespace( + status_code=200, + headers={"content-type": "text/plain"}, + stream=response_stream, + extensions={}, + ) + + await wrapper.async_response_hook(span, None, response_info) + + assert calls == [({"content-type": "text/plain"}, None, span)] + assert transport_stream.iterations == 0 + assert response_stream._httpcore_stream is transport_stream From 285a5786f8e48f09d8d3e3c45b84c42475229a7f Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 14:26:56 +0530 Subject: [PATCH 4/6] fix: avoid consuming HTTPX response streams Co-authored-by: Cursor --- .../instrumentation/httpx/__init__.py | 19 +-- .../instrumentation/httpx/utils.py | 120 ------------------ .../httpx_client/test_httpx_utils.py | 36 ------ 3 files changed, 4 insertions(+), 171 deletions(-) delete mode 100644 test/instrumentation/httpx_client/test_httpx_utils.py diff --git a/src/harness_sdk/instrumentation/httpx/__init__.py b/src/harness_sdk/instrumentation/httpx/__init__.py index e40ef04..955312b 100644 --- a/src/harness_sdk/instrumentation/httpx/__init__.py +++ b/src/harness_sdk/instrumentation/httpx/__init__.py @@ -6,11 +6,8 @@ from harness_sdk.plugins.control import get_control_registry from harness_sdk.instrumentation import BaseInstrumentorWrapper from harness_sdk.instrumentation.httpx.utils import ( - decode_response_body_for_capture, headers_from_httpx, read_request_body, - read_response_body, - read_response_body_async, url_from_request_info, ) @@ -46,22 +43,14 @@ def _process_request(self, span, request_info): def _process_response(self, span, response_info): headers = headers_from_httpx(response_info.headers) - body = read_response_body(response_info.stream) - body = decode_response_body_for_capture(headers, body) - self.generic_response_handler(headers, body, span) - - async def _process_response_async(self, span, response_info): - headers = headers_from_httpx(response_info.headers) - body = await read_response_body_async(response_info.stream) - body = decode_response_body_for_capture(headers, body) - self.generic_response_handler(headers, body, span) + self.generic_response_handler(headers, None, span) def request_hook(self, span, request_info): '''Capture sync client request data and run evaluation.''' self._process_request(span, request_info) def response_hook(self, span, request_info, response_info): # pylint: disable=unused-argument - '''Capture sync client response data.''' + '''Capture sync client response headers.''' self._process_response(span, response_info) async def async_request_hook(self, span, request_info): @@ -69,5 +58,5 @@ async def async_request_hook(self, span, request_info): self._process_request(span, request_info) async def async_response_hook(self, span, request_info, response_info): # pylint: disable=unused-argument - '''Capture async client response data.''' - await self._process_response_async(span, response_info) + '''Capture async client response headers.''' + self._process_response(span, response_info) diff --git a/src/harness_sdk/instrumentation/httpx/utils.py b/src/harness_sdk/instrumentation/httpx/utils.py index 2ea4df3..eab643f 100644 --- a/src/harness_sdk/instrumentation/httpx/utils.py +++ b/src/harness_sdk/instrumentation/httpx/utils.py @@ -1,6 +1,4 @@ '''Helpers for extracting httpx request/response data for tracing.''' -import gzip -import zlib from harness_sdk.custom_logger import get_custom_logger @@ -19,65 +17,6 @@ def url_from_request_info(request_info): return str(request_info.url) -def _content_encoding_chain(headers): - '''Return Content-Encoding tokens in wire order (outer coding first).''' - if not headers: - return [] - lower = {str(k).lower(): v for k, v in headers.items()} - raw = lower.get('content-encoding') or '' - return [t.strip().lower() for t in raw.split(',') if t.strip()] - - -def _apply_content_encoding_decodings(original, chain): - '''Apply decodings in reverse wire order; return ``original`` on failure.''' - out = original - for encoding in reversed(chain): - try: - if encoding in ('gzip', 'x-gzip'): - out = gzip.decompress(out) - elif encoding == 'deflate': - try: - out = zlib.decompress(out, -zlib.MAX_WBITS) - except zlib.error: - out = zlib.decompress(out) - elif encoding == 'br': - try: - import brotli # pylint: disable=import-outside-toplevel - except ImportError: - logger.debug('brotli not installed; skipping br decompression for capture') - return original - out = brotli.decompress(out) - elif encoding in ('identity', 'compress'): - continue - else: - logger.debug('Unsupported content-encoding for capture: %s', encoding) - return original - except Exception: # pylint: disable=broad-except - logger.debug('Response body decompression failed (%s)', encoding, exc_info=True) - return original - return out - - -def decode_response_body_for_capture(headers, body): - '''Decode compressed response bytes for span capture (gzip/deflate/br). - - Transport hooks often see the on-the-wire body while ``Content-Encoding`` - still names the compression. httpx may also surface compressed bytes when - decoding is deferred. If decoding fails or an encoding is unsupported, - the original ``body`` is returned. - ''' - if body in (None, b''): - return body - if not isinstance(body, (bytes, bytearray)): - return body - - chain = _content_encoding_chain(headers) - if not chain: - return body - - return _apply_content_encoding_decodings(bytes(body), chain) - - def _body_from_byte_stream(stream): internal = getattr(stream, "_stream", None) if isinstance(internal, (bytes, bytearray)): @@ -99,62 +38,3 @@ def read_request_body(stream): except Exception: # pylint: disable=broad-except logger.debug("Unable to read httpx request stream body", exc_info=True) return None - - -def read_response_body(stream): - '''Read response body bytes and restore the stream for downstream consumers.''' - if stream is None: - return None - - body = _body_from_byte_stream(stream) - if body is not None: - return body - - httpcore_stream = getattr(stream, "_httpcore_stream", None) - if httpcore_stream is not None: - try: - chunks = list(httpcore_stream) - body = b"".join(chunks) - stream._httpcore_stream = chunks # pylint: disable=protected-access - return body - except Exception: # pylint: disable=broad-except - logger.debug("Unable to read httpx response stream body", exc_info=True) - return None - - return None - - -async def read_response_body_async(stream): - '''Read async response body bytes and restore the stream for downstream consumers.''' - if stream is None: - return None - - body = _body_from_byte_stream(stream) - if body is not None: - return body - - httpcore_stream = getattr(stream, "_httpcore_stream", None) - if httpcore_stream is not None: - try: - chunks = [] - async for part in httpcore_stream: - chunks.append(part) - body = b"".join(chunks) - stream._httpcore_stream = _ReplayAsyncStream(chunks) # pylint: disable=protected-access - return body - except Exception: # pylint: disable=broad-except - logger.debug("Unable to read httpx async response stream body", exc_info=True) - return None - - return None - - -class _ReplayAsyncStream: # pylint: disable=too-few-public-methods - '''Minimal async iterable used to replay captured response chunks.''' - - def __init__(self, chunks): - self._chunks = chunks - - async def __aiter__(self): - for chunk in self._chunks: - yield chunk diff --git a/test/instrumentation/httpx_client/test_httpx_utils.py b/test/instrumentation/httpx_client/test_httpx_utils.py deleted file mode 100644 index 3996fed..0000000 --- a/test/instrumentation/httpx_client/test_httpx_utils.py +++ /dev/null @@ -1,36 +0,0 @@ -import gzip -import zlib - -import pytest - -from harness_sdk.instrumentation.httpx.utils import decode_response_body_for_capture - - -def test_decode_gzip_response_body(): - payload = b'{"hello":"world"}' - compressed = gzip.compress(payload) - headers = {'Content-Encoding': 'gzip', 'content-type': 'application/json'} - assert decode_response_body_for_capture(headers, compressed) == payload - - -def test_decode_deflate_response_body(): - payload = b'plain text body' - compressed = zlib.compress(payload) - headers = {'Content-Encoding': 'deflate'} - assert decode_response_body_for_capture(headers, compressed) == payload - - -def test_no_content_encoding_returns_raw(): - payload = b'not compressed' - assert decode_response_body_for_capture({'content-type': 'text/plain'}, payload) == payload - - -def test_bad_gzip_returns_original(): - payload = b'not gzip at all' - headers = {'Content-Encoding': 'gzip'} - assert decode_response_body_for_capture(headers, payload) == payload - - -@pytest.mark.parametrize('empty', [None, b'']) -def test_decode_empty_body(empty): - assert decode_response_body_for_capture({'Content-Encoding': 'gzip'}, empty) == empty From 5d7f00864a9366e2c46d252a82e8fa0bea61c256 Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 14:32:36 +0530 Subject: [PATCH 5/6] fix: disable HTTPX response body telemetry Co-authored-by: Cursor --- .../instrumentation/httpx/__init__.py | 1 + .../httpx_client/httpx_integration_test.py | 22 ++++++++++--------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/src/harness_sdk/instrumentation/httpx/__init__.py b/src/harness_sdk/instrumentation/httpx/__init__.py index 955312b..c046de3 100644 --- a/src/harness_sdk/instrumentation/httpx/__init__.py +++ b/src/harness_sdk/instrumentation/httpx/__init__.py @@ -23,6 +23,7 @@ def __init__(self): logger.debug('Entering HTTPXClientInstrumentorWrapper.__init__().') HTTPXClientInstrumentor.__init__(self) BaseInstrumentorWrapper.__init__(self) + self._process_response_body = False def _instrument(self, **kwargs) -> None: '''Enable instrumentation with request/response hooks.''' diff --git a/test/instrumentation/httpx_client/httpx_integration_test.py b/test/instrumentation/httpx_client/httpx_integration_test.py index 87b07ff..a6cdfa3 100644 --- a/test/instrumentation/httpx_client/httpx_integration_test.py +++ b/test/instrumentation/httpx_client/httpx_integration_test.py @@ -9,11 +9,12 @@ from test.instrumentation.flask.app import FlaskServer -def _client_span(spans): +def _client_span(spans, url): for span in spans: - if span.kind == SpanKind.CLIENT: - return json.loads(span.to_json()) - raise AssertionError('No client span found') + span_data = json.loads(span.to_json()) + if span.kind == SpanKind.CLIENT and span_data['attributes'].get('http.url') == url: + return span_data + raise AssertionError(f'No client span found for {url}') def test_httpx_client_get(agent, exporter): @@ -38,11 +39,11 @@ def api_example(): spans = exporter.get_finished_spans() assert spans - client_span = _client_span(spans) + client_span = _client_span(spans, url) assert client_span['attributes']['http.method'] == 'GET' assert client_span['attributes']['http.url'] == url - assert client_span['attributes']['http.response.body'] == '{ "a": "a", "xyz": "xyz" }' + assert 'http.response.body' not in client_span['attributes'] assert client_span['attributes']['http.status_code'] == 200 assert client_span['attributes']['http.response.header.tester3'] == 'tester3' finally: @@ -71,15 +72,16 @@ def api_example(): spans = exporter.get_finished_spans() assert spans - client_span = _client_span(spans) + client_span = _client_span(spans, url) assert client_span['kind'] == "SpanKind.CLIENT" assert client_span['attributes']['http.method'] == 'POST' assert client_span['attributes']['http.url'] == url assert client_span['attributes']['http.request.header.content-type'] == 'application/json' assert client_span['attributes']['http.request.body'] == '{"test":"body"}' - assert client_span['attributes']['http.response.body'] == '{ "a": "a", "xyz": "xyz" }' + assert 'http.response.body' not in client_span['attributes'] assert client_span['attributes']['http.status_code'] == 200 + assert client_span['attributes']['http.response.header.tester3'] == 'tester3' finally: server.shutdown() @@ -114,7 +116,7 @@ def api_example(): span_list = exporter.get_finished_spans() assert span_list - httpx_span = _client_span(span_list) + httpx_span = _client_span(span_list, url) assert httpx_span['attributes']['http.method'] == 'POST' assert httpx_span['attributes']['http.url'] == url @@ -122,6 +124,6 @@ def api_example(): assert httpx_span['attributes']['http.request.header.tester2'] == 'tester2' assert httpx_span['attributes']['http.request.body'] == '{ "a":"b", "c": "d" }' assert httpx_span['attributes']['http.response.header.content-type'] == 'application/json' - assert httpx_span['attributes']['http.response.body'] == '{ "a": "a", "xyz": "xyz" }' + assert 'http.response.body' not in httpx_span['attributes'] assert httpx_span['attributes']['http.status_code'] == 200 server.shutdown() From 01c840b19395d613f7af460c7f3663bc19e38696 Mon Sep 17 00:00:00 2001 From: Shreyas Nagaraj Date: Fri, 17 Jul 2026 15:03:22 +0530 Subject: [PATCH 6/6] chore: remove superpowers planning docs from HTTPX fix branch These internal design/plan artifacts should not ship with the SDK change. --- ...7-17-httpx-non-consuming-response-hooks.md | 293 ------------------ ...tpx-non-consuming-response-hooks-design.md | 37 --- 2 files changed, 330 deletions(-) delete mode 100644 docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md delete mode 100644 docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md diff --git a/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md b/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md deleted file mode 100644 index 8c549f8..0000000 --- a/docs/superpowers/plans/2026-07-17-httpx-non-consuming-response-hooks.md +++ /dev/null @@ -1,293 +0,0 @@ -# HTTPX Non-Consuming Response Hooks Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make synchronous and asynchronous HTTPX response hooks preserve headers and status telemetry without reading or replacing application response streams. - -**Architecture:** Keep request handling unchanged. Both response paths extract only headers and call `generic_response_handler(headers, None, span)`; OpenTelemetry continues owning status capture. Remove response-body decoding, draining, and replay code once no production caller remains. - -**Tech Stack:** Python 3.10+, pytest 7.4, pytest-asyncio, OpenTelemetry HTTPX instrumentation 0.62b1 - -## Global Constraints - -- Apply the change to all HTTPX responses, not only compressed or streaming responses. -- Preserve HTTPX request-body capture. -- Preserve HTTPX response headers and status. -- Remove `http.response.body` from HTTPX spans. -- Never iterate or mutate synchronous or asynchronous response streams. -- Make no changes to other instrumentations. -- Keep the existing design commit `61dd1b4` unchanged. - ---- - -### Task 1: Add response-stream regression tests - -**Files:** -- Create: `test/instrumentation/httpx_client/test_httpx_hooks.py` - -**Interfaces:** -- Consumes: `HTTPXClientInstrumentorWrapper.response_hook(span, request_info, response_info)` and `async_response_hook(span, request_info, response_info)`. -- Produces: focused regression coverage requiring response headers plus a `None` body while preserving the original `_httpcore_stream` object without iteration. - -- [ ] **Step 1: Write the synchronous failing test** - -```python -from types import SimpleNamespace - -import pytest - -from harness_sdk.instrumentation.httpx import HTTPXClientInstrumentorWrapper - - -class _SyncStream: - def __init__(self): - self.iterations = 0 - - def __iter__(self): - self.iterations += 1 - yield b"response" - - -class _AsyncStream: - def __init__(self): - self.iterations = 0 - - async def __aiter__(self): - self.iterations += 1 - yield b"response" - - -def _wrapper_with_response_capture(): - wrapper = HTTPXClientInstrumentorWrapper() - calls = [] - wrapper.generic_response_handler = ( - lambda headers, body, span: calls.append((headers, body, span)) - ) - return wrapper, calls - - -def test_response_hook_does_not_consume_or_replace_stream(): - wrapper, calls = _wrapper_with_response_capture() - span = object() - transport_stream = _SyncStream() - response_stream = SimpleNamespace(_httpcore_stream=transport_stream) - response_info = SimpleNamespace( - status_code=200, - headers={"content-type": "text/plain"}, - stream=response_stream, - extensions={}, - ) - - wrapper.response_hook(span, None, response_info) - - assert calls == [({"content-type": "text/plain"}, None, span)] - assert transport_stream.iterations == 0 - assert response_stream._httpcore_stream is transport_stream -``` - -- [ ] **Step 2: Write the asynchronous failing test** - -```python -@pytest.mark.asyncio -async def test_async_response_hook_does_not_consume_or_replace_stream(): - wrapper, calls = _wrapper_with_response_capture() - span = object() - transport_stream = _AsyncStream() - response_stream = SimpleNamespace(_httpcore_stream=transport_stream) - response_info = SimpleNamespace( - status_code=200, - headers={"content-type": "text/plain"}, - stream=response_stream, - extensions={}, - ) - - await wrapper.async_response_hook(span, None, response_info) - - assert calls == [({"content-type": "text/plain"}, None, span)] - assert transport_stream.iterations == 0 - assert response_stream._httpcore_stream is transport_stream -``` - -- [ ] **Step 3: Run tests to verify RED** - -Run: - -```bash -python3 -m pytest -q \ - test/instrumentation/httpx_client/test_httpx_hooks.py -``` - -Expected: both tests fail because current hooks iterate `_httpcore_stream`, replace it, and pass captured bytes instead of `None`. - -- [ ] **Step 4: Commit the regression tests** - -```bash -git add test/instrumentation/httpx_client/test_httpx_hooks.py -git commit -m "test: cover non-consuming HTTPX response hooks" -``` - -### Task 2: Stop response capture and remove dead helpers - -**Files:** -- Modify: `src/harness_sdk/instrumentation/httpx/__init__.py:8-73` -- Modify: `src/harness_sdk/instrumentation/httpx/utils.py:1-160` -- Delete: `test/instrumentation/httpx_client/test_httpx_utils.py` - -**Interfaces:** -- Consumes: `headers_from_httpx(response_info.headers)` and `generic_response_handler(response_headers, response_body, span)`. -- Produces: `_process_response(span, response_info) -> None`, used by both sync and async response hooks, always forwarding `None` as `response_body`. - -- [ ] **Step 1: Unify response processing without stream access** - -Replace response-specific imports and methods in `src/harness_sdk/instrumentation/httpx/__init__.py` with: - -```python -from harness_sdk.instrumentation.httpx.utils import ( - headers_from_httpx, - read_request_body, - url_from_request_info, -) - - def _process_response(self, span, response_info): - headers = headers_from_httpx(response_info.headers) - self.generic_response_handler(headers, None, span) - - async def async_response_hook(self, span, request_info, response_info): # pylint: disable=unused-argument - '''Capture async client response headers.''' - self._process_response(span, response_info) -``` - -Remove `_process_response_async`. Update the synchronous response-hook docstring to say it captures response headers. - -- [ ] **Step 2: Remove obsolete response helpers** - -Delete from `src/harness_sdk/instrumentation/httpx/utils.py`: - -```python -import gzip -import zlib - -_content_encoding_chain -_apply_content_encoding_decodings -decode_response_body_for_capture -read_response_body -read_response_body_async -_ReplayAsyncStream -``` - -Keep `headers_from_httpx`, `url_from_request_info`, `_body_from_byte_stream`, and `read_request_body` unchanged. Delete `test/instrumentation/httpx_client/test_httpx_utils.py` because every test in it covers the removed response-only decoder. - -- [ ] **Step 3: Run focused tests to verify GREEN** - -Run: - -```bash -python3 -m pytest -q \ - test/instrumentation/httpx_client/test_httpx_hooks.py -``` - -Expected: `2 passed`; neither stream reports iteration or replacement. - -- [ ] **Step 4: Confirm response helpers have no references** - -Run: - -```bash -rg "decode_response_body_for_capture|read_response_body|read_response_body_async|_ReplayAsyncStream" src test -``` - -Expected: no matches. - -- [ ] **Step 5: Commit implementation** - -```bash -git add src/harness_sdk/instrumentation/httpx/__init__.py \ - src/harness_sdk/instrumentation/httpx/utils.py \ - test/instrumentation/httpx_client/test_httpx_utils.py -git commit -m "fix: avoid consuming HTTPX response streams" -``` - -### Task 3: Update telemetry expectations and verify - -**Files:** -- Modify: `test/instrumentation/httpx_client/httpx_integration_test.py:19-127` - -**Interfaces:** -- Consumes: finished HTTPX client span attributes. -- Produces: integration assertions preserving request body, response headers, and status while requiring `http.response.body` to be absent. - -- [ ] **Step 1: Replace response-body assertions** - -In all three HTTPX integration tests, replace: - -```python -assert client_span['attributes']['http.response.body'] == '{ "a": "a", "xyz": "xyz" }' -``` - -or the equivalent `httpx_span` assertion with: - -```python -assert 'http.response.body' not in client_span['attributes'] -``` - -Use `httpx_span` in the async test. Retain all request-body, response-header, and status assertions. Add the existing `tester3` response-header assertion to the POST test so both POST response paths explicitly retain header coverage. - -- [ ] **Step 2: Run the complete HTTPX test directory** - -Run: - -```bash -python3 -m pytest -q test/instrumentation/httpx_client -``` - -Expected: all HTTPX tests pass. - -- [ ] **Step 3: Commit telemetry expectations** - -```bash -git add test/instrumentation/httpx_client/httpx_integration_test.py -git commit -m "test: update HTTPX response telemetry expectations" -``` - -- [ ] **Step 4: Run repository verification** - -Run: - -```bash -python3 -m compileall -q src/harness_sdk -pylint src/harness_sdk --disable=C,R --ignore-patterns=config_pb2.py -./scripts/run-unit-tests.sh -``` - -Expected: each command exits 0. The unit script may report only the repository's configured DB integration-test skips when `RUN_SDK_INTEGRATION_TESTS` is unset. - -- [ ] **Step 5: Review scope and final diff** - -Run: - -```bash -git diff --check origin/main...HEAD -git diff --stat origin/main...HEAD -git diff origin/main...HEAD -- \ - src/harness_sdk/instrumentation/httpx \ - test/instrumentation/httpx_client \ - docs/superpowers -git status --short -``` - -Expected: no whitespace errors; only the approved design, plan, HTTPX source, and HTTPX tests differ; working tree is clean. - -- [ ] **Step 6: Request code review and reverify findings** - -Give a reviewer `origin/main` as the base, current `HEAD` as the head, this plan, and the approved design spec. Fix only valid high-confidence correctness or scope findings, commit those fixes, then rerun the focused HTTPX directory plus any affected lint command. - -- [ ] **Step 7: Push and create the pull request** - -```bash -git push -u origin fix/httpx-non-consuming-response-hooks -gh pr create --base main \ - --title "fix: preserve HTTPX response streams during instrumentation" \ - --body-file /tmp/httpx-stream-safe-pr.md -``` - -The PR body must state the root cause, the intentional removal of `http.response.body` from HTTPX spans, retained headers/status/request capture, exact verification results, and that this removes unsafe stream consumption/private mutation implicated by instrumentation without claiming it alone proves the production gzip issue. diff --git a/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md b/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md deleted file mode 100644 index 383a238..0000000 --- a/docs/superpowers/specs/2026-07-17-httpx-non-consuming-response-hooks-design.md +++ /dev/null @@ -1,37 +0,0 @@ -# HTTPX Non-Consuming Response Hooks - -## Problem - -The HTTPX response hooks synchronously drain the transport response stream to -capture its body, then replace HTTPX's private `_httpcore_stream` state with a -custom replay iterator. This buffers streaming responses, depends on private -HTTPX internals, and can leave a partially consumed stream when draining fails. - -## Design - -HTTPX response hooks will capture response headers and status without reading -the response stream. They will pass `None` as the response body to the generic -response handler. - -The response-draining helpers and replay stream will be removed. Request-body -capture remains unchanged. - -## Compatibility - -HTTPX spans will no longer contain `http.response.body`. Existing response -header and status attributes remain. This intentional telemetry reduction -preserves application correctness and true streaming behavior. - -## Tests - -- A synchronous response hook must not iterate or mutate its response stream. -- An asynchronous response hook must not iterate or mutate its response stream. -- Existing HTTPX integration tests must continue validating request capture, - response headers, and status without expecting response-body capture. -- The complete unit suite must pass. - -## Non-Goals - -- Buffering, truncating, or sampling streaming response bodies. -- Extending span lifetime until application response consumption. -- Changing request-body capture or other instrumentations.