Skip to content

fix content-length desync in TextIOPayload - #13552

Draft
arshsmith1 wants to merge 3 commits into
aio-libs:masterfrom
arshsmith1:textio-content-length-desync
Draft

fix content-length desync in TextIOPayload#13552
arshsmith1 wants to merge 3 commits into
aio-libs:masterfrom
arshsmith1:textio-content-length-desync

Conversation

@arshsmith1

Copy link
Copy Markdown
Contributor

What do these changes do?

TextIOPayload reports its size through the inherited IOBasePayload.size, which returns os.fstat().st_size. A text stream is decoded and re-encoded on its way to the wire, so that on-disk byte count is not what actually gets written: universal-newline translation (\r\n to \n) and any gap between the file's encoding and the payload encoding both change the length. Serving an attacker-influenced text file with web.Response(body=open(path)) then sends a Content-Length that disagrees with the body. A file of b"a\r\nb\r\nc\r\n" declares 9 bytes but writes 6, leaving a keep-alive connection three bytes short, and a latin-1 file re-encoded to utf-8 is cut to the too-small length, slicing a character in half.

The size genuinely can't be known without reading and encoding the whole stream, so TextIOPayload.size now returns None, which selects chunked/close framing instead of a wrong length. While confirming that, I found the unbounded write loop could spin forever for utf-16/utf-32 bodies, because "".encode("utf-16") is the two-byte BOM rather than empty, so an EOF read never looked like EOF. The read helpers now short-circuit an empty read to b"".

Are there changes in behavior for the user?

Text file bodies are now sent with chunked transfer-encoding rather than a Content-Length that only matched for pure-ASCII, unix-newline content. StringIOPayload and binary file payloads are untouched and keep their exact Content-Length.

Is it a substantial burden for the maintainers to support this?

No. It is a size override plus a one-line EOF guard in each read helper, all inside TextIOPayload. Two existing size tests asserted the on-disk value and are updated to the corrected behavior, and regression tests cover both the short-write and truncation cases.

Related issue number

N/A

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes - N/A, no public API change
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
  • Add a new news fragment into the CHANGES/ folder

A text stream is decoded and re-encoded before it reaches the wire, so
the on-disk st_size is not the body length. Report the size as unknown
so chunked framing is used, and stop encoding empty EOF reads so a
BOM-prefixed encoding cannot spin the unbounded write loop.
@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Aug 26, 2026
@codecov

codecov Bot commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.23810% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.92%. Comparing base (eb38b3c) to head (862e772).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
aiohttp/payload.py 71.42% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #13552      +/-   ##
==========================================
- Coverage   99.02%   98.92%   -0.11%     
==========================================
  Files         135      133       -2     
  Lines       50455    50448       -7     
  Branches     2647     2639       -8     
==========================================
- Hits        49962    49904      -58     
- Misses        370      419      +49     
- Partials      123      125       +2     
Flag Coverage Δ
Autobahn ?
CI-GHA 98.91% <95.23%> (-0.01%) ⬇️
OS-Linux 98.68% <95.23%> (-0.01%) ⬇️
OS-Windows 97.08% <95.23%> (-0.01%) ⬇️
OS-macOS 97.97% <95.23%> (+<0.01%) ⬆️
Py-3.10 98.11% <95.23%> (-0.01%) ⬇️
Py-3.11 98.35% <95.23%> (-0.01%) ⬇️
Py-3.12 98.43% <95.23%> (-0.01%) ⬇️
Py-3.13 98.41% <95.23%> (-0.01%) ⬇️
Py-3.14 98.45% <95.23%> (+<0.01%) ⬆️
Py-3.14t 97.61% <95.23%> (-0.01%) ⬇️
Py-pypy-3.11 97.40% <95.23%> (+<0.01%) ⬆️
VM-macos 97.97% <95.23%> (+<0.01%) ⬆️
VM-ubuntu 98.68% <95.23%> (-0.01%) ⬇️
VM-windows 97.08% <95.23%> (-0.01%) ⬇️
cython-coverage 83.08% <57.14%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@codspeed-hq

codspeed-hq Bot commented Aug 26, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 9.51%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 3 regressed benchmarks
✅ 91 untouched benchmarks
⏩ 83 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
test_list_element_pattern_redos_payload[whitespace_run_after_content] 29.4 ms 32.7 ms -10.1%
test_read_base64_part 11 ms 12.1 ms -9.53%
test_list_element_pattern_redos_payload[interleaved_quote_paren_triggers] 79.8 ms 87.5 ms -8.9%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing arshsmith1:textio-content-length-desync (862e772) with master (eb38b3c)

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant