Introduced support for S3ObjectStore - #91
Open
steph-ieffam wants to merge 1 commit into
Open
Conversation
Author
|
@tdonohue & @nwoodward I have managed to open a new PR for introducing support to replicate data using a S3 bucket |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add S3 support to the Replication Task Suite
Resolves #82
Summary
Adds a new
ObjectStoreimplementation,S3ObjectStore, that allows the Replication TaskSuite to store and retrieve AIPs directly in an Amazon S3 bucket, alongside the existing
LocalObjectStore,MountableObjectStoreandDuraCloudObjectStorebackends.The implementation is based on the AWS SDK for Java v2 (the same SDK already used by
dspace-apiforS3BitStoreService) and usesS3TransferManagerso that AIP uploads anddownloads are not limited by the 5 GB single-
PutObjectcap: multipart transfers arehandled automatically based on file size.
Refs: CST-27589
Changes
src/main/java/org/dspace/ctask/replicate/store/S3ObjectStore.java(new)Full
ObjectStoreimplementation:init()– builds the sync and async S3 clients plus theS3TransferManager, resolves thetarget bucket and creates it if it does not exist yet.
objectExists()/objectAttribute()–HeadObject-based lookups for existence andsizebytes;checksumis computed as MD5 over a temporary local copy of the object.transferObject()/fetchObject()– uploads and downloads viaS3TransferManager,with automatic multipart handling for large AIPs and a synchronous fallback for downloads.
removeObject()/moveObject()– delete, and copy-then-delete to emulate a move betweengroups (e.g. from the AIP store to the delete/replica group).
<group>/<id>, mirroring the folder layout of the existing stores.org.dspace.ctask.replicate.store.S3ObjectStore.operations) is usedfor all S3 operations so that S3 traffic can be traced independently of the rest of the
curation logging.
Three authentication modes are supported, resolved in this order:
replicate.s3.access-key/replicate.s3.secret-key) – kept onlyfor backwards compatibility with existing/legacy buckets.
AssumeRolefor cross-account access (replicate.s3.assume-role-arn), with optionalExternalId(replicate.s3.assume-role-external-id) to protect against confused-deputyattacks. Credentials are refreshed automatically.
task / pod, which is the recommended setup and requires only the region to be configured.
config/modules/replicate.cfgS3ObjectStoreadded to the list of documentedObjectStoreimplementations.### S3 Replicate configuration ###section documentingreplicate.s3.bucket-name,replicate.s3.region-name,replicate.s3.assume-role-arn,replicate.s3.assume-role-external-idand thecommented-out legacy
replicate.s3.access-key/replicate.s3.secret-key.All properties default to empty; if
replicate.s3.bucket-nameis not set the store logs awarning and skips initialization, so existing installations are unaffected.
pom.xmldspace-apionly declaressoftware.amazon.awssdk:s3, so the two artifacts required by thisclass had to be added explicitly:
software.amazon.awssdk:sts–StsClient,StsAssumeRoleCredentialsProvider,AssumeRoleRequestsoftware.amazon.awssdk:s3-transfer-manager–S3TransferManager,UploadFileRequest,DownloadFileRequestBoth are declared with
providedscope (the JARs are expected in the DSpace installation atruntime) and with the same
netty-nio-client/apache-clientexclusions thatdspace-apiapplies to
s3. The version is pinned via a newaws.sdk.versionproperty set to2.43.2,which is the AWS SDK version pulled in by
dspace-api9.3 — keeping them aligned is requiredto satisfy the
DependencyConvergenceenforcer rule.Configuration example