Skip to content

[feature](s3) Support S3 Express One Zone#65504

Open
0AyanamiRei wants to merge 3 commits into
apache:masterfrom
0AyanamiRei:take-over-63409
Open

[feature](s3) Support S3 Express One Zone#65504
0AyanamiRei wants to merge 3 commits into
apache:masterfrom
0AyanamiRei:take-over-63409

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #63409

Problem Summary:

Doris cannot currently read from or write to Amazon S3 Express One Zone directory buckets.

There are two compatibility gaps:

  1. The bundled AWS SDK for C++ is 1.11.219 and the S3 client is constructed through the legacy client configuration path. This does not enable the endpoint-rules resolver required for S3 Express directory buckets, including bucket-specific endpoint resolution, CreateSession authentication, and the s3express signing service.
  2. Doris always sends Content-MD5 for PutObject and UploadPart. S3 Express One Zone does not support Content-MD5 for these requests and requires a flexible checksum such as CRC32C.

This PR takes over and supersedes #63409. It:

  • Upgrades aws-sdk-cpp from 1.11.219 to 1.11.400.
  • Creates the S3 client with Aws::S3::S3ClientConfiguration and S3EndpointProvider so AWS endpoint rules and S3 Express session authentication are enabled.
  • Uses CRC32C rather than Content-MD5 for S3 Express uploads, while preserving the existing MD5 behavior for regular S3 and S3-compatible endpoints.
  • Adds s3_disable_content_md5 as an explicit compatibility switch for endpoints that do not support Content-MD5.
  • Centralizes S3 Express context detection and addresses the review comments on [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) #63409: directory buckets must end with --x-s3, and endpoints must contain an S3 Express hostname label rather than an arbitrary substring.
  • Adds unit coverage for valid directory bucket names, valid S3 Express endpoints, and false-positive bucket/endpoint cases.

Release note

Support reading from and writing to Amazon S3 Express One Zone directory buckets. S3 Express uploads use CRC32C instead of Content-MD5.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
      • S3 Express directory buckets use SDK endpoint resolution and session authentication.
      • PutObject and UploadPart use CRC32C instead of Content-MD5 in the S3 Express context.
      • Regular S3 behavior remains unchanged by default.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

yinzixin and others added 3 commits July 13, 2026 09:22
### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: apache#63409\n\nProblem Summary: The initial S3 Express implementation detected directory buckets and endpoints with broad substring matches in two separate locations. This could classify ordinary bucket names or endpoint paths as S3 Express and allowed the client configuration and upload checksum behavior to diverge. Centralize the decision in one helper, require the documented --x-s3 bucket suffix or an s3express endpoint host label, and pass the resulting context into the object storage client. Add unit coverage for valid and invalid names.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Unit Test (S3UTILTest.is_s3_express_context; ASAN UT rebuild started)\n- Behavior changed: No, except avoiding false-positive S3 Express detection\n- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei 0AyanamiRei marked this pull request as ready for review July 13, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants