fix content-length desync in TextIOPayload - #13552
Conversation
A text stream is decoded and re-encoded before it reaches the wire, so the on-disk st_size is not the body length. Report the size as unknown so chunked framing is used, and stop encoding empty EOF reads so a BOM-prefixed encoding cannot spin the unbounded write loop.
for more information, see https://pre-commit.ci
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #13552 +/- ##
==========================================
- Coverage 99.02% 98.92% -0.11%
==========================================
Files 135 133 -2
Lines 50455 50448 -7
Branches 2647 2639 -8
==========================================
- Hits 49962 49904 -58
- Misses 370 419 +49
- Partials 123 125 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
Merging this PR will degrade performance by 9.51%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | test_list_element_pattern_redos_payload[whitespace_run_after_content] |
29.4 ms | 32.7 ms | -10.1% |
| ❌ | test_read_base64_part |
11 ms | 12.1 ms | -9.53% |
| ❌ | test_list_element_pattern_redos_payload[interleaved_quote_paren_triggers] |
79.8 ms | 87.5 ms | -8.9% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing arshsmith1:textio-content-length-desync (862e772) with master (eb38b3c)
Footnotes
-
83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
What do these changes do?
TextIOPayloadreports its size through the inheritedIOBasePayload.size, which returnsos.fstat().st_size. A text stream is decoded and re-encoded on its way to the wire, so that on-disk byte count is not what actually gets written: universal-newline translation (\r\nto\n) and any gap between the file's encoding and the payload encoding both change the length. Serving an attacker-influenced text file withweb.Response(body=open(path))then sends aContent-Lengththat disagrees with the body. A file ofb"a\r\nb\r\nc\r\n"declares 9 bytes but writes 6, leaving a keep-alive connection three bytes short, and a latin-1 file re-encoded to utf-8 is cut to the too-small length, slicing a character in half.The size genuinely can't be known without reading and encoding the whole stream, so
TextIOPayload.sizenow returnsNone, which selects chunked/close framing instead of a wrong length. While confirming that, I found the unbounded write loop could spin forever for utf-16/utf-32 bodies, because"".encode("utf-16")is the two-byte BOM rather than empty, so an EOF read never looked like EOF. The read helpers now short-circuit an empty read tob"".Are there changes in behavior for the user?
Text file bodies are now sent with chunked transfer-encoding rather than a
Content-Lengththat only matched for pure-ASCII, unix-newline content.StringIOPayloadand binary file payloads are untouched and keep their exactContent-Length.Is it a substantial burden for the maintainers to support this?
No. It is a
sizeoverride plus a one-line EOF guard in each read helper, all insideTextIOPayload. Two existing size tests asserted the on-disk value and are updated to the corrected behavior, and regression tests cover both the short-write and truncation cases.Related issue number
N/A
Checklist
CONTRIBUTORS.txtCHANGES/folder