[WIP][SYSTEMDS-3946] Merge PR Enable sending of large (>2GiB) FederatedRequests - #2591
Open
ywcb00 wants to merge 13 commits into
Open
[WIP][SYSTEMDS-3946] Merge PR Enable sending of large (>2GiB) FederatedRequests#2591ywcb00 wants to merge 13 commits into
ywcb00 wants to merge 13 commits into
Conversation
… Responses Federated transfers previously failed for payloads above 2GiB because the single Netty frame size is bounded by a 32-bit length field, capping any request or response at Integer.MAX_VALUE bytes. This patch adds a streaming chunked codec that splits a large payload into bounded frames on the sender and reassembles them on the receiver, so the on-wire size is no longer limited by a single frame. A format detector and format encoder select the chunked path only when the payload exceeds the frame limit, leaving the existing small-message path unchanged to avoid added overhead for the common case. Adds FederatedMaxPayloadTest to exercise the boundary around the former 2GiB cap.
Document the codec on methods and fail fast on an unknown frame type in the chunk decoder.
Both negative tests go through a helper, since writeInbound rethrows if the deserializer thread wins the race.
….java): change class and members to visibility public refactor(test/component/federated/**): use the constants from the federated chunk protocol general refactoring refactor(test/functions/federated/network/FederatedMaxPayloadTest.java): general refactoring
…ove test to io folder instead of network folder and delete the empty network folder
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2591 +/- ##
==========================================
Coverage 71.44% 71.45%
- Complexity 50437 50467 +30
==========================================
Files 1629 1634 +5
Lines 195513 195705 +192
Branches 38118 38136 +18
==========================================
+ Hits 139686 139834 +148
- Misses 44867 44900 +33
- Partials 10960 10971 +11 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…e federated request encoder again fix(main/runtime/controlprogram/federated/FederatedFormatEncoder.java): throw an io exception when the message is not serializable instead of writing it to the channel
…dify the predicate if it is reusable by lineage to always be true when the lineage object is not null chore(test/component/federated/FederatedChunkCodecTest.java): change polling for exceptions etc. to try fewer times and wait longer
apply formatting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intermediate PR for merging #2496