Skip to content

Introduced support for S3ObjectStore - #91

Open
steph-ieffam wants to merge 1 commit into
DSpace:mainfrom
4Science:task/main/replicate-support-S3
Open

Introduced support for S3ObjectStore#91
steph-ieffam wants to merge 1 commit into
DSpace:mainfrom
4Science:task/main/replicate-support-S3

Conversation

@steph-ieffam

@steph-ieffam steph-ieffam commented Jul 27, 2026

Copy link
Copy Markdown

Add S3 support to the Replication Task Suite

Resolves #82

Summary

Adds a new ObjectStore implementation, S3ObjectStore, that allows the Replication Task
Suite to store and retrieve AIPs directly in an Amazon S3 bucket, alongside the existing
LocalObjectStore, MountableObjectStore and DuraCloudObjectStore backends.

The implementation is based on the AWS SDK for Java v2 (the same SDK already used by
dspace-api for S3BitStoreService) and uses S3TransferManager so that AIP uploads and
downloads are not limited by the 5 GB single-PutObject cap: multipart transfers are
handled automatically based on file size.

Refs: CST-27589

Changes

src/main/java/org/dspace/ctask/replicate/store/S3ObjectStore.java (new)

Full ObjectStore implementation:

  • init() – builds the sync and async S3 clients plus the S3TransferManager, resolves the
    target bucket and creates it if it does not exist yet.
  • objectExists() / objectAttribute()HeadObject-based lookups for existence and
    sizebytes; checksum is computed as MD5 over a temporary local copy of the object.
  • transferObject() / fetchObject() – uploads and downloads via S3TransferManager,
    with automatic multipart handling for large AIPs and a synchronous fallback for downloads.
  • removeObject() / moveObject() – delete, and copy-then-delete to emulate a move between
    groups (e.g. from the AIP store to the delete/replica group).
  • Objects are keyed as <group>/<id>, mirroring the folder layout of the existing stores.
  • A dedicated logger (org.dspace.ctask.replicate.store.S3ObjectStore.operations) is used
    for all S3 operations so that S3 traffic can be traced independently of the rest of the
    curation logging.

Three authentication modes are supported, resolved in this order:

  1. Static credentials (replicate.s3.access-key / replicate.s3.secret-key) – kept only
    for backwards compatibility with existing/legacy buckets.
  2. AssumeRole for cross-account access (replicate.s3.assume-role-arn), with optional
    ExternalId (replicate.s3.assume-role-external-id) to protect against confused-deputy
    attacks. Credentials are refreshed automatically.
  3. The default AWS credentials provider chain, i.e. the IAM role attached to the instance /
    task / pod, which is the recommended setup and requires only the region to be configured.

config/modules/replicate.cfg

  • S3ObjectStore added to the list of documented ObjectStore implementations.
  • New ### S3 Replicate configuration ### section documenting
    replicate.s3.bucket-name, replicate.s3.region-name,
    replicate.s3.assume-role-arn, replicate.s3.assume-role-external-id and the
    commented-out legacy replicate.s3.access-key / replicate.s3.secret-key.
    All properties default to empty; if replicate.s3.bucket-name is not set the store logs a
    warning and skips initialization, so existing installations are unaffected.

pom.xml

dspace-api only declares software.amazon.awssdk:s3, so the two artifacts required by this
class had to be added explicitly:

  • software.amazon.awssdk:stsStsClient, StsAssumeRoleCredentialsProvider,
    AssumeRoleRequest
  • software.amazon.awssdk:s3-transfer-managerS3TransferManager, UploadFileRequest,
    DownloadFileRequest

Both are declared with provided scope (the JARs are expected in the DSpace installation at
runtime) and with the same netty-nio-client / apache-client exclusions that dspace-api
applies to s3. The version is pinned via a new aws.sdk.version property set to 2.43.2,
which is the AWS SDK version pulled in by dspace-api 9.3 — keeping them aligned is required
to satisfy the DependencyConvergence enforcer rule.

Configuration example

plugin.single.org.dspace.ctask.replicate.ObjectStore = \
    org.dspace.ctask.replicate.store.S3ObjectStore

replicate.s3.bucket-name = my-dspace-aip-bucket
replicate.s3.region-name = eu-west-1
# optional, for cross-account access
replicate.s3.assume-role-arn = arn:aws:iam::123456789012:role/dspace-replication
replicate.s3.assume-role-external-id = some-shared-secret

@steph-ieffam steph-ieffam added dependencies Pull requests that update a dependency file java Pull requests that update java code labels Jul 27, 2026
@steph-ieffam

steph-ieffam commented Jul 27, 2026

Copy link
Copy Markdown
Author

@tdonohue & @nwoodward I have managed to open a new PR for introducing support to replicate data using a S3 bucket

@steph-ieffam steph-ieffam self-assigned this Jul 27, 2026
@tdonohue tdonohue moved this to 🙋 Needs Reviewers Assigned in DSpace 11.0 Release Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file java Pull requests that update java code

Projects

Status: 🙋 Needs Reviewers Assigned

Development

Successfully merging this pull request may close these issues.

Create new ObjectStore for Amazon S3 storage

2 participants