Skip to content

RATIS-2546. Add input stream to DataStreamApi for read operations in Client#1481

Open
peterxcli wants to merge 5 commits into
apache:masterfrom
peterxcli:RATIS-2546-stream-read-client
Open

RATIS-2546. Add input stream to DataStreamApi for read operations in Client#1481
peterxcli wants to merge 5 commits into
apache:masterfrom
peterxcli:RATIS-2546-stream-read-client

Conversation

@peterxcli

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

  • Added a new DataStreamInput interface and its implementation DataStreamInputImpl for asynchronous, zero-copy read-only data streaming, including the readAsync() method for consuming replies and proper resource release on close.
  • Extended the DataStreamApi interface and its implementation to provide streamReadOnly() methods for creating read-only streams.
  • Introduced the new DataStreamReplyByteBuf class to support replies backed by Netty ByteBuf.
  • Added a new overload of streamAsync() in the DataStreamClientRpc interface that takes a reply consumer, supporting multiple replies per request.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2546

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)

Signed-off-by: peterxcli <peterxcli@gmail.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>
@peterxcli

Copy link
Copy Markdown
Member Author

cc @szetszwo

@peterxcli

Copy link
Copy Markdown
Member Author

The no-MD5 sequential reads comparison:

Key size Buffer ReadBlock stream Ratis data stream Ratis / ReadBlock bandwidth
256 MiB 32 MiB 333.23 MB/s 448.04 MB/s 1.34x bandwidth
256 MiB 8 MiB 651.57 MB/s 753.94 MB/s 1.16x bandwidth
256 MiB 1 MiB 346.72 MB/s 726.91 MB/s 2.10x bandwidth
256 MiB 4 KiB 303.35 MB/s 777.43 MB/s 2.56x bandwidth
500 MiB 32 MiB 731.13 MB/s 445.67 MB/s 0.61x bandwidth
500 MiB 8 MiB 587.45 MB/s 1012.59 MB/s 1.72x bandwidth
500 MiB 1 MiB 520.76 MB/s 598.70 MB/s 1.15x bandwidth
500 MiB 4 KiB 373.36 MB/s 846.50 MB/s 2.27x bandwidth
1 GiB 32 MiB 869.43 MB/s 693.04 MB/s 0.80x bandwidth
1 GiB 8 MiB 508.69 MB/s 474.92 MB/s 0.93x bandwidth
1 GiB 1 MiB 432.13 MB/s 784.25 MB/s 1.81x bandwidth
1 GiB 4 KiB 378.85 MB/s 1087.56 MB/s 2.87x bandwidth

The no-MD5 random reads comparison:

Key size Random read size Direct stream Ratis data stream Bandwidth ratio IOPS ratio Direct elapsed Ratis elapsed
256 MiB 1 MiB 28.92 MB/s 103.03 MB/s 3.56x bandwidth 3.56x IOPS 1.106 s 0.311 s
256 MiB 4 KiB 0.12 MB/s 0.38 MB/s 3.19x bandwidth 3.19x IOPS 1.059 s 0.332 s
500 MiB 1 MiB 44.18 MB/s 255.86 MB/s 5.79x bandwidth 5.79x IOPS 0.724 s 0.125 s
500 MiB 4 KiB 0.14 MB/s 0.59 MB/s 4.06x bandwidth 4.06x IOPS 0.865 s 0.213 s
1 GiB 1 MiB 33.08 MB/s 193.64 MB/s 5.85x bandwidth 5.85x IOPS 0.967 s 0.165 s
1 GiB 4 KiB 0.10 MB/s 0.87 MB/s 8.34x bandwidth 8.34x IOPS 1.197 s 0.144 s

I'm looking into why larger buffer would cause throughput to degrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant