HDDS-14660. Implement new table to store part information for multipart uploads by spacemonkd · Pull Request #9886 · apache/ozone

spacemonkd · 2026-03-07T18:42:13Z

What changes were proposed in this pull request?

HDDS-14660. Implement new table to store part information for multipart uploads

Please describe your PR in detail:

This PR adds a new multipartPartsTable to handle storing of parts for Multipart Uploads.
The existing MultipartKeyInfo is updated to handle new fields like owner and ACLs, volume, bucket and key name.
The codec for PartKey and PartInformation has been added.
The new table is added to the list of compacted tables

How is the PartKey encoded?

Say for the below sample input (we are taking 47 as it has 2F in the lower byte):

uploadId = <abcd-1234...>
partNumber = 47 -> taking 47 because it has 2f in the byte representation

Encoding format:
keyBytes = UTF8(uploadId) + / + int32_be(partNumber)

For 47:
int32 big-endian bytes = 00 00 00 2f

So final tail is:

separator / = 2f
part bytes = 00 00 00 2f
Total tail bytes = 2f 00 00 00 2f

Now we run the following checks:

Check full-key first: if byte at len - 5 is /, decode as full key (used to identify full row)
Else, if byte at len - 1 is /, decode as prefix. (this is used for prefix scan for iterating)
Else invalid.

A logical key will look something like the following in the RocksDB entry:

[61 62 63 31 32 33 2d 75 75 69 64 2d 34 35 36 2f 00 00 00 2f]
[--------------uploadId UTF-8---------------][2f][-int32 BE-]

and will be represented as:

{
  uploadID: "abcd123-uuid-456",
  partNumber: 47
} : {
... 
}

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14660

How was this patch tested?

Patch has been tested by running unit tests.

…rt uploads

…ckstyle and findbug

ChenSammi · 2026-03-10T03:04:30Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/OMDBDefinition.java

          OmMultipartKeyInfo.getCodec());

+  public static final String MULTIPART_PARTS_TABLE = "multipartPartsTable";
+  /** multipartPartsTable: /uploadId/partNumber :- PartKeyInfo. */


The key is "/volume/bucket/key/uploadId/partNumber" in Column Family comments, while it's "/uploadId/partNumber" here. What's the real key format?

Sorry, need to update the comment.
It is actually uploadId/part number.

The multipartInfo table is one with vol/buck/key/uploadID

Can remove the /, /uploadId/partNumber -> uploadId/partNumber

ivandika3

Thanks @spacemonkd for the patch, overall LGTM. Left some comments,

ivandika3 · 2026-03-15T11:08:54Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/OMDBDefinition.java

          OmMultipartKeyInfo.getCodec());

+  public static final String MULTIPART_PARTS_TABLE = "multipartPartsTable";
+  /** multipartPartsTable: /uploadId/partNumber :- PartKeyInfo. */


Can remove the /, /uploadId/partNumber -> uploadId/partNumber

ivandika3 · 2026-03-15T11:45:06Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java


  private Table<String, OmKeyInfo> openKeyTable;
  private Table<String, OmMultipartKeyInfo> multipartInfoTable;
+  private Table<OmMultipartPartKey, OmMultipartPartInfo> multipartPartTable;


Nit: Standardize to multipartPartsTable. Similarly for getMultipartPartTable.

Addressed in the latest commit.

ivandika3 · 2026-03-15T11:46:07Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java

+          if (omMultipartKeyInfo.getSchemaVersion() == 0) {
+            numParts += omMultipartKeyInfo.getPartKeyInfoMap().size();
+          } else {
+            OmMultipartPartKey prefix =
+                OmMultipartPartKey.prefix(expiredMultipartUpload.getUploadId());
+            try (TableIterator<OmMultipartPartKey, ? extends KeyValue<OmMultipartPartKey, OmMultipartPartInfo>>
+                     partIterator = getMultipartPartTable().iterator(prefix)) {
+              while (partIterator.hasNext()) {
+                KeyValue<OmMultipartPartKey, OmMultipartPartInfo> partEntry =
+                    partIterator.next();
+                if (!expiredMultipartUpload.getUploadId().equals(
+                    partEntry.getKey().getUploadId())) {
+                  break;
+                }
+                numParts++;
+              }
+            }
+          }


Since this is not tested yet (we are still creating schemaVersion 0 MPU), let's exclude this first and add this once the MPU flow patches are done and listParts behaving normally.

Addressed this in the latest commit.
I have commented out this part and added a TODO in case I forget about the changes 😅

Will just uncomment the section after the actual request and response modifications are done

ivandika3 · 2026-03-15T11:53:20Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartPartKey.java

+  public static OmMultipartPartKey of(String uploadId, int partNumber) {
+    return new OmMultipartPartKey(uploadId, partNumber);
+  }


Should we also use the boxed Integer instead? If we pass null Integer here, it might throw NPE. We should also always use this static constructor and not normal constructor (in fromPersistedFormat)

Changed it to use Integer and validate that Integer is nonnull in the latest commit.
Also using prefix() and of() methods to return the values instead of direct constructor calls in fromPersistedFormat()

ivandika3 · 2026-03-15T11:58:10Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartPartKey.java

+  private static final class OmMultipartPartKeyCodec
+      implements Codec<OmMultipartPartKey> {


We can support CodecBuffer to prevent buffer copying, should be able to be addressed here, but can raise in another ticket (under HDDS-2274) if you prefer.

Implemented this in the current change itself, added new tests to validate that the change is working fine.

HDDS-14660. Implement new table to store part information for multipa…

7a80416

…rt uploads

spacemonkd requested review from adoroszlai, errose28, ivandika3 and rakeshadr March 7, 2026 18:42

spacemonkd added 4 commits March 8, 2026 00:30

Add checks for schemaVersion 1 to not have parts list inline, fix che…

49ed677

…ckstyle and findbug

Fix TestOmMetadataManager

a14cb69

Add multipartPartsTable defs to OmMetadataManager

c627ce9

Fix build issues

45be606

spacemonkd marked this pull request as ready for review March 8, 2026 18:15

spacemonkd self-assigned this Mar 8, 2026

spacemonkd added the s3 S3 Gateway label Mar 8, 2026

spacemonkd requested a review from ChenSammi March 8, 2026 18:16

ChenSammi reviewed Mar 10, 2026

View reviewed changes

Fix table format comment

12234c3

spacemonkd requested a review from ChenSammi March 12, 2026 18:29

ivandika3 reviewed Mar 15, 2026

View reviewed changes

Address review comments, add more tests

6d0591b

spacemonkd requested a review from ivandika3 March 15, 2026 15:41

		private static final class OmMultipartPartKeyCodec
		implements Codec<OmMultipartPartKey> {

Conversation

spacemonkd commented Mar 7, 2026

What changes were proposed in this pull request?

How is the PartKey encoded?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivandika3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivandika3 Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivandika3 Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ivandika3 Mar 15, 2026 •

edited

Loading

ivandika3 Mar 15, 2026 •

edited

Loading