[filesystem] Add JuiceFS filesystem support - #3870
Open
weizuo93 wants to merge 1 commit into
Open
Conversation
weizuo93
force-pushed
the
support-juicefs-filesystem
branch
from
August 6, 2026 04:59
9c4d2e9 to
6a7557f
Compare
weizuo93
force-pushed
the
support-juicefs-filesystem
branch
from
August 6, 2026 06:10
6a7557f to
e0ed750
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #3866
This PR adds support for JuiceFS — a widely-adopted open-source distributed file system that is built on top of object storage and exposes a Hadoop-compatible Java SDK. With this change, users can point
remote.data.dirat ajfs://volume and store Fluss snapshots (Primary-Key Tables) and tiered log segments (Log Tables) on their existing JuiceFS deploymentsBrief change log
fluss-filesystems/fluss-fs-juicefs, registered as a child offluss-filesystems.JuiceFsPlugin implements FileSystemPlugin:jfs://URI scheme.io.juicefs.JuiceFileSystemreflectively viaorg.apache.hadoop.fs.FileSystem.newInstance, so Fluss has no hard compile-time dependency onio.juicefs.*classes.fs.jfs.orjuicefs.to the underlying HadoopConfiguration.fs.jfs.impl = io.juicefs.JuiceFileSystemfs.jfs.impl.disable.cache = falseJuiceFsFileSystem extends HadoopFileSystem:FileSystem.obtainSecurityToken(), because JuiceFS handles authentication locally on each node against its metadata engine and backing object storage — no STS / delegation-token exchange is required.META-INF/services/org.apache.fluss.fs.FileSystemPlugin.fluss-fs-hadoop-shaded,fluss-fs-hadoop, andio.juicefs:juicefs-hadoop:1.4.0.website/docs/maintenance/tiered-storage/filesystems/juicefs.md.overview.mdlist of pluggable filesystems.Tests
Unit tests added in
JuiceFsPluginTest:testScheme— verifies the plugin registers thejfsscheme.testHadoopConfigPropagation— verifiesfs.jfs.*andjuicefs.*entries are forwarded to the HadoopConfiguration, and that unrelated keys are not forwarded.testApplyJuiceFsDefaultsInjectsImpl— verifies default injection offs.jfs.implandfs.jfs.impl.disable.cache.testApplyJuiceFsDefaultsDoesNotOverrideUserValues— verifies user-supplied values take precedence over the injected defaults.testGetHadoopConfigurationWithNullFlussConfig— null-safety.Run locally with:
API and Format
No changes to public APIs, wire format, or on-disk / storage format. The change is purely additive — a new implementation of the existing
FileSystemPluginSPI.Documentation
website/docs/maintenance/tiered-storage/filesystems/juicefs.md— installation, prerequisite (JuiceFS volume creation), required / optionalserver.yamlconfiguration, authentication model, and pointers to the JuiceFS Hadoop Java SDK documentation for advanced tuning.website/docs/maintenance/tiered-storage/filesystems/overview.mdto include JuiceFS in the intro list and add a bullet under the "Pluggable File Systems" section.