0.47.0

Multi-Storage Client (MSC)

Breaking Changes

• s8k provider no longer pins request_checksum_calculation/response_checksum_validation to when_required; botocore defaults now apply, and botocore >= 1.36.0 adds a CRC32 checksum to every upload.

New Features

• Added checksum_algorithm option to s3 and s8k providers for AWS S3 additional upload checksums (CRC32, CRC32C, SHA1, SHA256, CRC64NVME). When rust_client is enabled, only SHA256 is supported.
• End-to-end symlink preservation for uploads and downloads. MSC now treats symbolic links as a first-class concept across all storage providers, so symlinks in a POSIX source tree can be round-tripped through object storage without losing their link semantics. This adds:
  • A new make_symlink API on every StorageProvider (S3, GCS, Azure, OCI, AIS, POSIX), which writes an empty object whose user metadata carries the link target on cloud backends, and a native symlink on POSIX.
  • A symlink_handling option on list / list_files with three modes: FOLLOW (default, backward-compatible), SKIP (exclude symlinks), and PRESERVE (yield symlinks as leaf entries with ObjectMetadata.symlink_target populated).
  • sync_from / msc sync honor symlink_handling=PRESERVE to recreate symlinks at the destination instead of dereferencing them, so uploading a directory to object storage and downloading it back reconstructs the original link structure.
• Add opt-in client-side MD5 verification to the Azure storage provider.
• Support multipart upload and download in the Azure storage provider.

Bug Fixes

• Sort directory entries in the POSIX storage provider to match S3 lexicographical order.
• Improve msc sync performance by skipping per-object metadata lookups for files up to 16MB, avoiding redundant HEAD requests during synchronization.
• Upgrade fastmcp, cryptography, pygments, and axios to address security advisories.

Multi-Storage File System (MSFS)

New Features

• Add fio benchmark script for MSFS performance testing.

Bug Fixes

• Add globalsStruct lock tracking.
• Additional fixes for the fio benchmark.

0.46.0

Multi-Storage Client (MSC)

New Features

• Add batch download_files and upload_files APIs to StorageClient for efficient multi-file transfers.
• Support presigned URL generation for Azure Blob Storage via SAS tokens.
• Use download_files and upload_files in sync_from for efficient transfers.
• Support non-default AWS profile in S3 storage providers through MSC config.
• Support workload identity in Azure storage provider through rclone config.
• Support origin path in CloudFront signed URLs.
• Support security token file in OCI CLI/SDK configs.

Bug Fixes

• Support credential-less rclone configurations.
• Relax google-cloud-storage dependency to >=2.12,<4.
• Setup non-default HTTP client for AWS SDK for Rust config.
• Add cryptography as vault extra dependency.
• Support custom attributes in upload_files.

Multi-Storage File System (MSFS)

New Features

• Integrate PebbleDB as metadata cache on-disk store.
• Add "pseudo" backend to enable arbitrary directory tree extreme testing.
• Support listObjects with StartAfter.
• Avoid HEAD calls to emulate IfMatch.

0.45.1

Multi-Storage Client (MSC)

Bug Fixes

• Restore cryptography as an optional dependency; it was incorrectly added as a core requirement in 0.45.0.
• Lower the minimum cache refresh interval from 300 seconds to 1 second.
• Fix incorrect handling of short final chunks and improve partial file cache reliability.
• Upgrade grpc-go, pyOpenSSL, and Authlib to address CVE-2026-33186, CVE-2026-27459, and CVE-2026-27962.

Multi-Storage File System (MSFS)

New Features

• Adopt sortedmap for inode management.

0.45.0

Multi-Storage Client (MSC)

New Features

• Add presigned URL generation via msc.generate_presigned_url() shortcut and StorageClient.generate_presigned_url(). Supports S3 native signing and CloudFront signed URLs through a pluggable URLSigner abstraction.
• Add dryrun mode to sync_from and msc.sync() that enumerates and compares objects without performing any copy/delete, streaming results to JSONL files via DryrunResult on SyncResult.

Bug Fixes

• Fix sync worker to protect add_file references from concurrent modification.
• Fix handling of directory paths in ManifestMetadataProvider.get_object_metadata.
• Remove placement group overhead from Ray sync workers.

Multi-Storage File System (MSFS)

New Features

• Integrate sortedmap B+Tree data structure for enhanced scalability.
• Shift to pointer-free inodeStruct for improved memory efficiency.

0.44.0

Multi-Storage Client (MSC)

New Features

• Add DefaultAzureCredentialsProvider for Azure Blob authentication via Azure Identity (e.g. managed identity, Azure CLI, environment variables).

Bug Fixes

• Fix ImportError when importing multistorageclient.instrumentation.auth without the [vault] extra installed.
• Preserve metadata-only updates when sync skips copy.

Multi-Storage File System (MSFS)

New Features

• Add GCS backend support.

0.43.0

Multi-Storage Client (MSC)

Breaking Changes

• Drop Python 3.9 support.

New Features

• Add list_recursive method to StorageClient with parallel listing support for S3, Azure, GCS, and OCI providers.
• Pass-through s3 parameter to botocore.config.Config in S3 storage provider.
• Support bulk delete in storage providers and in msc sync.
• Route sync metadata updates through queue-backed provider.
• Add follow_symlinks to list() shortcut.

Bug Fixes

• Resolve IsADirectoryError when syncing a single file to a directory.
• Compare sync last_modified at seconds resolution to avoid spurious re-syncs.
• Ensure directory exists for lock file creation in sync worker.
• Prevent Rust multipart uploads from exceeding the 10k part limit.
• Add logic for invalidating cached mTLS certificates.

Multi-Storage File System (MSFS)

New Features

• Add listObjects() backend API.
• Add support for unlink (file delete).

0.42.0

Multi-Storage Client (MSC)

New Features

The MSC Explorer Web UI is now integrated into MSC, providing a browser-based interface to browse and manage objects across your configured storage backends. See the documentation for more details.

• Bumped OpenTelemetry package version to 1.32.1 for mTLS support.

Bug Fixes

• Avoid HEAD requests when checking source version to reduce unnecessary API calls.
• Resolve sync worker performance regression with bounded backpressure.
• Raise ValueError when base_path is empty in list_objects.
• Resolve multiprocessing metadata loss and related test failures in sync operations.
• Add early validation for cache_line_size exceeding cache size.

Multi-Storage File System (MSFS)

New Features

• Added support for mkdir and rmdir operations.
• Added prefetch of directories when readdir/readdirplus has not already done so, improving directory listing performance.
• MSFS Grafana dashboard now reports READ metrics.

Bug Fixes

• Fixed double locking in directory prefetcher.

0.41.0

Multi-Storage Client (MSC)

Breaking Changes

The default POSIX profile has been renamed from default to __filesystem__. If your code or configuration explicitly sets profile="default", please update it to profile="__filesystem__" (or remove the profile parameter entirely). Users who do not explicitly specify a profile, or who are not using the reserved POSIX profile, are not affected.

New Features

• Added multi-backend configuration support, enabling datasets distributed across multiple storage backends to be accessed through a single unified profile. See configuration reference for usage details.
• Added size-based batching to msc sync operations for improved performance and resource utilization.
• Added automatic ulimit increase for high-concurrency operations to prevent file descriptor exhaustion.
• Added GCS credentials provider support for Rust client with comprehensive refactoring.
• Implemented async metrics collection to significantly reduce OpenTelemetry overhead in telemetry operations.

Bug Fixes

• Fixed sync_from ignoring preserve_source_attributes parameter when using source_files argument.
• Fixed directory creation race conditions on distributed filesystems by implementing proper handling of concurrent mkdir operations.

Multi-Storage File System (MSFS)

New Features

• Added Prometheus/Grafana integration with dashboard support for monitoring file system operations.

Bug Fixes

• Fixed cache ttl_check_interval validation to ensure it is always positive.
• Fixed unlocked cache prune call that could cause race conditions.
• Fixed inode eviction to properly include clean cache lines.
• Fixed missing inode.touch() calls in file system operations.
• Fixed Grafana dashboard job selector for metrics.

0.40.0

New Features

• Add --include and --exclude pattern filtering to msc ls command for selective file listing.
• Enhance SyncResult to separately track files and bytes added/deleted, providing complete visibility into sync operations.
• Add configurable retry settings for Rust client to improve reliability and customization.

Bug Fixes

• Fix automatic retry for HTTP 408 (Request Timeout) errors when accessing Google Cloud Storage via S3-compatible API.
• Add default timeout values for S3 storage providers to prevent indefinite hangs.
• Fix unnecessary directory creation when opening files in read-only mode.

0.39.1

New Features

• Increase default sync concurrency to improve performance, especially on machines with fewer CPU cores.

Bug Fixes

• Update _is_rust_client_enabled() for CompositeStorageClient to return True when all child clients are Rust-enabled.