[release/11.0] Fix DeflateEncoder.GetMaxCompressedLength conservative bound - #132389
[release/11.0] Fix DeflateEncoder.GetMaxCompressedLength conservative bound#132389BrzVlad wants to merge 1 commit into
Conversation
Previous code was using the implementation from compressBound(). This api returns a conservative bound when using the compress() APIs, which our .NET implementation doesn't use (it uses deflate directly). The compress API does a deflate with default configuration, while our API allows custom configuration (via the quality level, window size and a few others). This means that the compressBound is not valid when using the deflate API with custom parameters. zlib exposes the deflateBound method which can be used to obtain a conservative size. The problem with this API is that it requires to know the parameters of the compression, which we don't have available in our .NET API, given GetMaxCompressedLenght is static, applying universally to all compressions. While it is not yet decided if we will add a new API to support obtaining the bound from the library, this commit adds a workaround, by computing the maximum bound for every compression configuration exposed by our .NET APIs The formula is obtained from our zlib-ng sources in deflate.c. Mobile uses zlib however. Validated that this change works correctly on maccatalyst, ios simulator and that the current logic stands with the current open source code for zlib (which applies to android). The actual bound used by Apple and Android can always change, so we do have a small risk that our hardcoded limit will diverge. I believe the only safe solution for this would be to make use of the actual deflateBound method, which requires adjustement to our API.
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression |
|
@steveisok Is this the right way to backport for RC1 ? without any tactics approval ? I see release/11.0 and release/11.0-rc1 are currently identical, so I'm assuming rc1 branch will move forward later today ? |
There was a problem hiding this comment.
Pull request overview
This PR updates the conservative upper-bound calculation used by DeflateEncoder.GetMaxCompressedLength, and adjusts related encoder comments/tests to reflect the new sizing behavior, with the goal of preventing DestinationTooSmall failures (notably on mobile platforms) when callers allocate buffers based on GetMaxCompressedLength.
Changes:
- Replaces the previous
compressBound()-based sizing behavior inDeflateEncoder.GetMaxCompressedLengthwith a managed bound derived fromdeflateBound()-style worst-case formulas. - Updates
ZLibEncoder/GZipEncodercomments to align with the new bound semantics. - Re-enables the “all window logs” roundtrip test on mobile by removing the
[ActiveIssue]skip.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs | Changes max-compressed-length computation to a managed conservative bound. |
| src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs | Updates documentation/comments to reflect new bound source/overhead. |
| src/libraries/System.IO.Compression/src/System/IO/Compression/ZLibEncoder.cs | Updates documentation/comments to reflect new bound semantics. |
| src/libraries/Common/tests/System/IO/Compression/EncoderDecoderTestBase.cs | Removes mobile skip so the window-log coverage test runs on those platforms. |
Previous code was using the implementation from compressBound(). This api returns a conservative bound when using the compress() APIs, which our .NET implementation doesn't use (it uses deflate directly). The compress API does a deflate with default configuration, while our API allows custom configuration (via the quality level, window size and a few others). This means that the compressBound is not valid when using the deflate API with custom parameters.
zlib exposes the deflateBound method which can be used to obtain a conservative size. The problem with this API is that it requires to know the parameters of the compression, which we don't have available in our .NET API, given GetMaxCompressedLenght is static, applying universally to all compressions. While it is not yet decided if we will add a new API to support obtaining the bound from the library, this commit adds a workaround, by computing the maximum bound for every compression configuration exposed by our .NET APIs
The formula is obtained from our zlib-ng sources in deflate.c. Mobile uses zlib however. Validated that this change works correctly on maccatalyst, ios simulator and that the current logic stands with the current open source code for zlib (which applies to android). The actual bound used by Apple and Android can always change, so we do have a small risk that our hardcoded limit will diverge. I believe the only safe solution for this would be to make use of the actual deflateBound method, which requires adjustement to our API.
Fixes issue with new .NET11 APIs (#127563). Backport of #132274