HDDS-15179. Support Chunked Transfer without Content Length

[peterxcli]

What changes were proposed in this pull request?

Currently chunked transfer would fail if we don't specify content length

def _ev_add_te_header(request, **kwargs):
    request.headers.add_header('Transfer-Encoding', 'chunked')

def test_object_write_with_chunked_transfer_encoding():
    bucket_name = get_new_bucket()
    client = get_client()

    client.meta.events.register_first('before-sign.*.*', _ev_add_te_header)
    response = client.put_object(Bucket=bucket_name, Key='foo', Body='bar')

    assert response['ResponseMetadata']['HTTPStatusCode'] == 200

with the following error:

botocore.exceptions.ClientError: An error occurred (XAmzContentSHA256Mismatch) when calling the PutObject operation: The provided 'x-amz-content-sha256' header does not match the computed hash.

https://github.com/ceph/s3-tests/blob/fb8b73092bb1dd8db829f1205a9e52e73bf9a232/s3tests/functional/test_s3.py#L1589-L1599

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15179

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)

[peterxcli]

I notice the jira ticket should be as a subtask under https://issues.apache.org/jira/browse/HDDS-8423, is there any way to fix it? I couldn't found the move option.

[adoroszlai]

Thanks @peterxcli for the patch.

[chungen0126]

Thanks @peterxcli for the patch.

Based on my understanding of S3, requests fall into one of two cases: they either include a Content-Length header or an x-amz-decoded-content-length header. That being said, I totally understand that this proposed change adds flexibility to the upload process, and I have no objections to it at all. But for me it is ok.

[chungen0126]

Regarding S3, we should also consider the scenario where the Content-Length header is absent, but the x-amz-decoded-content-length header is provided. In this case, since we already know the exact size of the payload, we wouldn't need to manually calculate the upload length. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

For all requests, you must include the x-amz-decoded-content-length header, specifying the size of the object in bytes.