Research persistent storage in Ocean nodes for users/jobs

[bogdanfazakas]

[alexcos20]

Remote Storage Architecture for Ocean Node

Context

Ocean Node C2D jobs are using docker with temporary volumes for input/output.

After a C2D job is done, Ocean Node takes '/data/output' folder from a container job and currently writes locally when no output request is provided. This is not scalable for long-running nodes or high job throughput.

We also need a persistent dataset storage model where users can upload datasets to Ocean Node and later mount/use them in C2D jobs.
PS: Nice perk would be shared storage with other users

Ocean Node already supports multiple storage backends and already has upload support for:

• s3 via @aws-sdk/client-s3 and multipart uploads
• ftp
• url (HTTP PUT upload)

Requirements

• Scalable storage for docker volumes (during c2d jobs).
• Scalable storage for C2D outputs.
• Persistent storage for user datasets used by C2D.
• TypeScript-friendly API support for file operations (put/list/delete).
• Ability to mount datasets into algorithm containers.
• Clear path for self-hosted and cloud deployments.

Option Comparison

1) S3-compatible object storage (AWS S3, Ceph RGW, MinIO, etc.)

• TypeScript SDK: Yes (first-class via AWS SDK v3).
• Put/list/delete support: Yes.
• Docker volume mount:
  • Not native at Docker engine level for object semantics.
  • Possible via plugins (for example Mountpoint/FUSE-based drivers such as aekis/docker-mount-s3).
• Strengths:
  • Already implemented in Ocean Node storage layer.
  • Strong ecosystem and operational maturity.
  • Works for AWS and most S3-compatible systems (Ceph RGW, MinIO).
  • Good fit for result archives and persistent dataset objects.
• Weaknesses:
  • Mount plugins typically require elevated privileges (/dev/fuse, CAP_SYS_ADMIN, host network) and increase host attack surface.
• Suitable for :
  • user storage (each user gets one bucket): [x]
  • c2d storage (one bucket for all outputs): [x]
  • docker volume storage : []

2) NFS

• TypeScript SDK: No mature direct Node/TypeScript client path (typically mount at OS level).
• Put/list/delete support: Via filesystem APIs after mount.
• Docker volume mount: Yes (Docker local driver with NFS options).
• Strengths:
  • Easy shared filesystem semantics for containers.
  • Good for on-prem/HPC environments requiring POSIX shared paths.
• Weaknesses:
  • Harder multi-tenant isolation and credential model.
  • Performance and locking semantics depend on network/server setup.
  • Not ideal for internet-exposed object workflows.
  • Cannot isolate users
• Suitable for :
  • user storage: []
  • c2d storage (mount nfs path locally): [x]
  • docker volume storage: [x]

3) GlusterFS

• TypeScript SDK: No mainstream TS SDK for object-style operations.
• Docker volume mount: Possible via plugins.
• Strengths:
  • Shared filesystem semantics.
• Weaknesses:
  • Plugin ecosystem maturity/operability is weaker for modern container platforms.
  • Cannot isolate users
• Suitable for :
  • user storage: []
  • c2d storage: []
  • docker volume storage: [x]

4) OpenStack Swift

• TypeScript SDK: Yes (Node clients exist, for example npm swift).
• Put/list/delete support: Yes (object and container operations).
• Docker volume mount: No standard native Docker driver pattern for Swift object semantics.
• Strengths:
  • Good option for OpenStack-centric operators.
  • Object storage model aligns with C2D outputs and dataset persistence.
• Weaknesses:
  • Not currently implemented in Ocean Node storage classes.
  • Adds another auth/config path (Swift/Keystone) alongside S3-compatible setup.
• Suitable for :
  • user storage: []
  • c2d storage: [x]
  • docker volume storage: []

5) FTP

• TypeScript SDK: Yes (Already supported by node).

• Put/list/delete support: Yes

• Docker volume mount: Yes

• Strengths:

  • Easy to use
  • Each user gets it's own folder

• Weaknesses:

  • Not scalable

• Suitable for :

  • user storage: [x]
  • c2d storage: [x]
  • docker volume storage: []

6) Local filesystem

• TypeScript SDK: Yes (Already supported by node).

• Put/list/delete support: Yes

• Docker volume mount: Yes

• Strengths:

  • Easy to use
  • Each user gets it's own folder

• Weaknesses:

  • Not scalable

• Suitable for :

  • user storage: [x]
  • c2d storage: [x] (default case now)
  • docker volume storage: []

Target Architecture & Decision

A) C2D output storage

• allow node owner to specify c2d results storage (s3 is already present, we can later add OpenStack Swift. Nfs can be done with local mounts, so no changes required to node)
• If job output is provided -> use job-level remote storage (current behavior).
• Else if remote results is defined -> stream /data/outputs archive to default remote store and keep reference in db for getResults
• Else -> keep local outputs.tar fallback.

Result: local disk pressure is reduced even when user does not specify output instructions.

B) User dataset storage for C2D

• Each user gets it's own bucket
• Persist datasets as immutable objects in S3-compatible storage (allow user to put/list/delete objects)
• C2D runtime staging flow:
  1. Create job working volume (existing /data mount).
  2. Mount bucket as /data/inputs/ (using aekis/docker-mount-s3 plugin)

C) Docker volumes during C2D

• Add a node level configs for different plugins
• Allow node owner to specify an option
• When creating volumes, use specific options

[peterjiz]

Leaving this paper here as a comment for now. AIStore Will do some more digging towards end of day

[ndrpp]

Perhaps we can use rclone plugin as an abstraction over all user backends and mount them as a local directory (use cases B & C), so the compute engine won't care which is used. It still requires FUSE though.