Skip to content

[filesystem] Add JuiceFS filesystem support - #3870

Open
weizuo93 wants to merge 1 commit into
apache:mainfrom
weizuo93:support-juicefs-filesystem
Open

[filesystem] Add JuiceFS filesystem support#3870
weizuo93 wants to merge 1 commit into
apache:mainfrom
weizuo93:support-juicefs-filesystem

Conversation

@weizuo93

@weizuo93 weizuo93 commented Aug 5, 2026

Copy link
Copy Markdown

Purpose

Linked issue: close #3866

This PR adds support for JuiceFS — a widely-adopted open-source distributed file system that is built on top of object storage and exposes a Hadoop-compatible Java SDK. With this change, users can point remote.data.dir at a jfs:// volume and store Fluss snapshots (Primary-Key Tables) and tiered log segments (Log Tables) on their existing JuiceFS deployments

Brief change log

  • Add a new Maven module fluss-filesystems/fluss-fs-juicefs, registered as a child of fluss-filesystems.
  • Implement JuiceFsPlugin implements FileSystemPlugin:
    • Registers the jfs:// URI scheme.
    • Loads io.juicefs.JuiceFileSystem reflectively via org.apache.hadoop.fs.FileSystem.newInstance, so Fluss has no hard compile-time dependency on io.juicefs.* classes.
    • Forwards any Fluss configuration entry whose key starts with fs.jfs. or juicefs. to the underlying Hadoop Configuration.
    • Injects sensible defaults when not set by the user:
      • fs.jfs.impl = io.juicefs.JuiceFileSystem
      • fs.jfs.impl.disable.cache = false
  • Implement JuiceFsFileSystem extends HadoopFileSystem:
    • Wraps the JuiceFS Hadoop FileSystem.
    • Returns an empty placeholder security token from obtainSecurityToken(), because JuiceFS handles authentication locally on each node against its metadata engine and backing object storage — no STS / delegation-token exchange is required.
  • SPI registration via META-INF/services/org.apache.fluss.fs.FileSystemPlugin.
  • Shading / packaging consistent with sibling filesystem plugins; depends on fluss-fs-hadoop-shaded, fluss-fs-hadoop, and io.juicefs:juicefs-hadoop:1.4.0.
  • Documentation:
    • New page website/docs/maintenance/tiered-storage/filesystems/juicefs.md.
    • Add JuiceFS to the overview.md list of pluggable filesystems.

Tests

Unit tests added in JuiceFsPluginTest:

  • testScheme — verifies the plugin registers the jfs scheme.
  • testHadoopConfigPropagation — verifies fs.jfs.* and juicefs.* entries are forwarded to the Hadoop Configuration, and that unrelated keys are not forwarded.
  • testApplyJuiceFsDefaultsInjectsImpl — verifies default injection of fs.jfs.impl and fs.jfs.impl.disable.cache.
  • testApplyJuiceFsDefaultsDoesNotOverrideUserValues — verifies user-supplied values take precedence over the injected defaults.
  • testGetHadoopConfigurationWithNullFlussConfig — null-safety.

Run locally with:

mvn test -Dtest=JuiceFsPluginTest -DfailIfNoTests=false -pl fluss-filesystems/fluss-fs-juicefs -am

API and Format

No changes to public APIs, wire format, or on-disk / storage format. The change is purely additive — a new implementation of the existing FileSystemPlugin SPI.

Documentation

@weizuo93
weizuo93 force-pushed the support-juicefs-filesystem branch from 9c4d2e9 to 6a7557f Compare August 6, 2026 04:59
@weizuo93
weizuo93 force-pushed the support-juicefs-filesystem branch from 6a7557f to e0ed750 Compare August 6, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[filesystem] Add JuiceFS filesystem support

1 participant