[improvement](be) Add disk cache for external file metadata#63376
Open
xylaaaaa wants to merge 10 commits into
Open
[improvement](be) Add disk cache for external file metadata#63376xylaaaaa wants to merge 10 commits into
xylaaaaa wants to merge 10 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Stabilize file meta disk cache related BE unit tests by isolating disk resource limit configuration, initializing FD cache for direct FS storage, and ensuring TTL cache test data carries an expiration time. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=FileMetaDiskCacheTest.* -j 8 - ./run-be-ut.sh --run --filter=BlockFileCacheTest.file_cache_path_storage_parse:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:BlockFileCacheTest.meta_queue_can_evict_data_cache_first:BlockFileCacheTest.normal_queue_does_not_evict_meta_cache:BlockFileCacheTest.read_if_cached_returns_downloaded_meta_block -j 8 - Behavior changed: No - Does this need documentation: No
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Large external scans could skip the reader meta cache entirely because the existing file meta cache policy disabled metadata caching when the scan range count exceeded the memory-cache threshold. This prevented the disk-backed file metadata cache from being used for high-file-count workloads. This change separates the L1 memory-cache policy from reader and disk-cache enablement, so large scans can skip the memory cache while still reading from and writing to the disk metadata cache. It also carries the memory-cache policy through row-id file mappings and Iceberg equality readers. ### Release note Improve external Parquet/ORC metadata disk cache behavior for large scans. ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh` - `./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.*:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:IdFileMapTest.*:ParquetReadLinesTest.*:OrcReadLinesTest.*` - Manual test: `plaud_current.DWD.DWD_POSTHOG_EVENTS_DI` CTE cold/hot profiles with external file meta disk cache enabled and disabled. - Behavior changed: Yes. Large external scans can use disk-backed file metadata cache even when L1 memory metadata cache is skipped. - Does this need documentation: No
Contributor
Author
|
run buildall |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 21, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Restore the existing ExternalFileMappingInfo comment style so the file meta disk cache PR does not include unrelated comment formatting noise. ### Release note None ### Check List (For Author) - Test: No need to test (comment-only change) - `git diff --check` - `PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh` - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 21, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: External Parquet/ORC file metadata disk cache profiles exposed hit/miss/write counts but did not report the local disk cache read and write cost. This adds FileFooterReadDiskCacheTime for successful disk-cache reads and FileFooterWriteDiskCacheTime for successful disk-cache writes so cold and hot query profiles can separate local cache overhead from remote footer read and parse time. ### Release note Add profile timers for external Parquet/ORC file metadata disk cache read and write cost. ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh - ./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.*:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:IdFileMapTest.*:ParquetReaderTest.file_footer_disk_cache_time_counters_exist:OrcReaderTest.file_footer_disk_cache_time_counters_exist:ParquetReadLinesTest.*:OrcReadLinesTest.* - Behavior changed: Yes. Query profiles now include external file metadata disk cache read and write timers. - Does this need documentation: No
Contributor
TPC-H: Total hot run time: 31487 ms |
Contributor
TPC-DS: Total hot run time: 169344 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
TPC-H: Total hot run time: 31341 ms |
Contributor
TPC-DS: Total hot run time: 170014 ms |
Contributor
TPC-H: Total hot run time: 31410 ms |
Contributor
TPC-DS: Total hot run time: 170252 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
reviewed
May 25, 2026
| } | ||
|
|
||
| bool FileMetaCache::should_enable_for_reader(int64_t num_scan_ranges) const { | ||
| return should_enable_memory_cache(num_scan_ranges) || |
Contributor
There was a problem hiding this comment.
这个条件只用config::enable_external_file_meta_disk_cache就行
| _current_range, _should_enable_file_meta_cache())); | ||
| auto file_id = id_file_map->get_file_mapping_id(std::make_shared<FileMapping>( | ||
| ((FileScanLocalState*)_local_state)->parent_id(), _current_range, | ||
| _should_enable_file_meta_cache(), _should_enable_file_meta_memory_cache())); |
Contributor
There was a problem hiding this comment.
这里不需要两个开关，我们只需要一个是否开启cache的开关，cache里内存的占比应该是有cache内部机制通过一个比例来控制的
| auto* file_meta_cache_ptr = _should_enable_file_meta_cache() | ||
| ? ExecEnv::GetInstance()->file_meta_cache() | ||
| : nullptr; | ||
| const bool enable_file_meta_memory_cache = _should_enable_file_meta_memory_cache(); |
Gabriel39
reviewed
May 25, 2026
Gabriel39
left a comment
Gabriel39
left a comment
Contributor
There was a problem hiding this comment.
scanner::enable_cache -> 改成通过一个配置项
cache： memory-》memory+disk
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: File meta disk cache reader wiring still carried a separate memory-cache switch, so scan-range admission policy leaked into ORC/Parquet reader setup and could also disable disk-cache use. This change uses enable_external_file_meta_disk_cache as the only reader switch and lets FileMetaCache internals handle memory cache admission and eviction. ### Release note External file meta disk cache reader wiring is now controlled by enable_external_file_meta_disk_cache. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.* -j 8 - ./run-be-ut.sh --run --filter=ParquetReaderTest.file_footer_disk_cache_time_counters_exist:OrcReaderTest.file_footer_disk_cache_time_counters_exist -j 8 - Behavior changed: Yes. The reader file meta cache path no longer has a separate scanner-level memory-cache switch. - Does this need documentation: No
Gabriel39
reviewed
May 25, 2026
| // Local variables can be required because setSerializedFileTail is an assignment operation, not a reference. | ||
| ObjLRUCache::CacheHandle _meta_cache_handle; | ||
| if (_meta_cache->lookup(file_meta_cache_key, &_meta_cache_handle)) { | ||
| const bool use_memory_cache = _meta_cache->enabled(); |
Contributor
There was a problem hiding this comment.
这些接口都不需要改，我们只是替换了一下cache的实现
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Jul 9, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: BE UT crashed in OrcReadLinesTest.test0 because read_lines_from_range can initialize readers without an ExecEnv file meta cache object. The scanner memory-cache toggle still dereferenced ExecEnv::file_meta_cache()->enabled() directly. Use the actual FileMetaCache pointer selected for the reader when enabling memory cache and require an initialized cache object before enabling memory or persistent file meta cache. ### Release note None ### Check List (For Author) - Test: - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.test0' -j 1 - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.*:ParquetReadLinesTest.*:*FileMeta*' -j 1 - Manual test: build-support/clang-format.sh be/src/exec/scan/file_scanner.h be/src/exec/scan/file_scanner.cpp be/src/exec/scan/file_scanner_v2.h be/src/exec/scan/file_scanner_v2.cpp - Manual test: build-support/check-format.sh - Manual test: git diff --check - Manual test: git diff --cached --check - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29506 ms |
Contributor
TPC-DS: Total hot run time: 180345 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29704 ms |
Contributor
TPC-DS: Total hot run time: 180076 ms |
Contributor
ClickBench: Total hot run time: 25.03 s |
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: External file metadata disk cache should be built behind the FileMetaCache boundary instead of exposing a generic complete-value put/get API from BlockFileCache. This change adds a narrow FileMetaCache persistent cache interface, lookup/insert result types, profile counter plumbing, and config gates so upper-layer metadata cache code can depend on FileMetaCache APIs while the storage implementation remains replaceable.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Added focused FileMetaCache interface tests. Attempted targeted BE UT; initial run hit stale PCH after branch switch, clean run was stopped because it triggered a large full rebuild before reaching the target tests.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: External Parquet and ORC readers only kept file metadata in the in-memory FileMetaCache. This change adds a FileMetaDiskCache implementation behind the FileMetaCache persistent-cache interface, stores serialized Parquet footers and ORC file tails in the existing BlockFileCache through its block APIs, and wires readers to read from disk cache, refill memory cache, and write new persistent entries on misses. It also wires the v2 Parquet reader path through FileScannerV2/TableReader so current Parquet scans use the file meta cache, and enables external file meta disk cache by default. ### Release note Enable disk cache for external Parquet/ORC file metadata by default. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - python3 build-support/run_clang_format.py --clang-format-executable clang-format --style file --inplace true -j 1 <changed C++ files> - git diff --check -- <changed C++ files> - ./run-be-ut.sh --run --filter=NewParquetReaderTest.UsesFileMetaCacheForFooterMetadata -j 4 - ./run-be-ut.sh --run --filter=FileMetaCache* -j 4 - ./build.sh --be -j 4 - Manual e2e with output FE/BE: local Parquet TVF returned 30000 before and after BE restart, local ORC TVF returned 10, BE runtime config showed enable_external_file_meta_disk_cache=true, and the post-restart Parquet query added no new file meta disk cache miss log. - Behavior changed: Yes. External Parquet/ORC metadata disk cache is enabled by default and can persist file metadata through the file cache. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: The external file metadata disk cache integration needed stronger correctness boundaries before review. Memory cache keys did not include the metadata format, so legacy Parquet and v2 Parquet could collide while storing incompatible in-memory objects. Persistent entries could also be attempted for files without a stable modification time, and missing disk-cache ranges were treated the same as corrupt entries. This change scopes memory keys by format, adds a PARQUET_V2 persistent format, builds persistent cache identities from path, modification time, and file size, skips persistent cache when the file identity is unstable, handles missing ranges without invalidating concurrent writers, falls back to file reads on invalid persistent payloads, and separates memory-cache gating from persistent-cache gating in file scanners. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - git diff --check - git diff --cached --check - ./run-be-ut.sh --run --filter='*FileMeta*' -j 2 - Manual e2e with output FE/BE: local Parquet TVF returned count 25 twice and second profile showed FileFooterHitMemoryCache=1; local ORC TVF returned count 25 twice and second profile showed FileFooterHitMemoryCache=1. Local TVF does not provide modification_time, so persistent disk cache is intentionally skipped on that path and covered by FileMetaCacheDiskTest. - Behavior changed: Yes. File metadata persistent cache now requires a stable modification time, uses format-scoped identities, and avoids invalidating incomplete in-flight cache writes. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: BE UT crashed in OrcReadLinesTest.test0 because read_lines_from_range can initialize readers without an ExecEnv file meta cache object. The scanner memory-cache toggle still dereferenced ExecEnv::file_meta_cache()->enabled() directly. Use the actual FileMetaCache pointer selected for the reader when enabling memory cache and require an initialized cache object before enabling memory or persistent file meta cache. ### Release note None ### Check List (For Author) - Test: - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.test0' -j 1 - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.*:ParquetReadLinesTest.*:*FileMeta*' -j 1 - Manual test: build-support/clang-format.sh be/src/exec/scan/file_scanner.h be/src/exec/scan/file_scanner.cpp be/src/exec/scan/file_scanner_v2.h be/src/exec/scan/file_scanner_v2.cpp - Manual test: build-support/check-format.sh - Manual test: git diff --check - Manual test: git diff --cached --check - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64961 Problem Summary: Row-id fetch scanners read a mapped range without a split source, but Parquet and ORC reader initialization evaluated the normal scan-range memory file metadata cache admission rule and dereferenced the missing split source. This crashed external, P0, and cloud regression jobs. The unknown-mtime Parquet unit test also accidentally used a synthetic nonzero mtime, enabling page cache while asserting it remained disabled. Disable memory file metadata cache for row-id fetch reader initialization and make the unit test pass an explicit unknown mtime. ### Release note None ### Check List (For Author) - Test: Unit Test / Code style check - `./run-be-ut.sh --run --filter='OrcReadLinesTest.*:ParquetReadLinesTest.*:NewParquetReaderTest.RewriteSameLocalPathDoesNotReuseUnknownMtimePageCache' -j 1` - `build-support/check-format.sh` - `git diff --check` - `git diff --cached --check` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63376 Problem Summary: File meta disk cache already supports external file metadata through the FileMetaCache boundary, but format v2 ORC did not participate in the cache. This change adds an ORC_V2 file meta cache namespace, wires ordinary format v2 ORC readers through FileMetaCache, reuses ORC serialized file tail payloads for memory and persistent cache hits, and refreshes invalid persistent payloads after a successful source-file fallback. Legacy ORC keeps using the ORC namespace so both reader implementations do not share incompatible cached values. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - ./build.sh --be -j 10 - ./run-be-ut.sh --run --filter='FileMetaCacheDiskTest.OrcFormatsUseIndependentKeys:NewOrcReaderTest.*FileMetaCache*:NewOrcReaderTest.OversizedPersistentPayloadRemainsMemoryCacheEligible' -j 10 - git diff --check - Behavior changed: Yes. Format v2 ORC file metadata can use the existing external file meta disk cache when enabled. - Does this need documentation: No
xylaaaaa
force-pushed
the
codex/file-meta-cache-disk-pr
branch
from
a51d8c4 to
48d7b10
Compare
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: Runtime activation could enable persistent metadata lookups after startup without constructing the required cache objects. Nested Iceberg and transactional Hive delete readers also ignored the parent memory-cache policy, and a capacity rejection during a multi-block write could leave downloaded payload blocks behind. Separate startup configuration from the runtime size gate, propagate the reader policy, preclaim complete entries before writing, and cover real disk reinitialization. ### Release note None ### Check List (For Author) - Test: Unit Tests added; modified production and test objects compiled with ASAN and -Werror; full BE UT execution pending the local rebuild - Behavior changed: Yes, runtime cache-size activation is supported and failed writes no longer leave partial entries - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: The file metadata persistent cache interface had no direct unit coverage. Add boundary-level tests for successful round trips, invalidation, profile counters, and read/write failure fallback without depending on BlockFileCache internals. ### Release note None ### Check List (For Author) - Test: Unit Test object compiled with ASAN and -Werror; full execution pending the local BE rebuild - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: Iceberg format-v2 data readers received the shared file metadata cache and memory-cache policy, but format-v2 position and equality delete readers were constructed without either value. Their Parquet and ORC footer metadata therefore bypassed the persistent cache entirely. Pass the cache, memory policy, and timestamp mapping option through the delete-reader construction boundary and add enabled/disabled behavior coverage. ### Release note None ### Check List (For Author) - Test: Unit Test added; modified v2 production and test objects compiled with ASAN and -Werror; full execution pending the local BE rebuild - Behavior changed: Yes, Iceberg v2 delete-file metadata now uses the configured file metadata cache - Does this need documentation: No
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: The disk-cache restart test exercised FSFileCacheStorage without initializing ExecEnv's FDCache, so the restored read dereferenced a null cache under ASAN. Initialize and release the FDCache only when this test owns it, and shorten the TTL worker intervals so destruction does not wait six minutes. ### Release note None ### Check List (For Author) - Test: Unit Test - 20 focused BE tests passed under ASAN - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Jul 12, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: The file metadata persistent cache interface had no direct unit coverage. Add boundary-level tests for successful round trips, invalidation, profile counters, and read/write failure fallback without depending on BlockFileCache internals. ### Release note None ### Check List (For Author) - Test: Unit Test added; execution pending the local BE rebuild - Behavior changed: No - Does this need documentation: No
Contributor
TPC-H: Total hot run time: 29450 ms |
Contributor
TPC-DS: Total hot run time: 180156 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
ClickBench: Total hot run time: 25.07 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64961
Problem Summary:
External Parquet/ORC footer metadata was only cached in BE memory. After memory cache eviction or BE restart, repeated reads from remote storage such as S3/HDFS had to fetch footer metadata remotely again.
This PR builds on the
FileMetaCachepersistent-cache interface from #64961 and adds a default-enabled disk-backed layer for external file metadata. The disk layer is implemented behind theFileMetaCacheboundary: readers call the file-meta-cache lookup/insert APIs, and onlyFileMetaDiskCacheadapts those calls to the existingBlockFileCacheprimitives.Lookup/write flow:
ObjLRUCacheusing a format-scoped memory key.FileMetaCacheasksFileMetaDiskCachefor serialized footer bytes stored in the existingBlockFileCacheINDEX queue.Main changes:
FileMetaDiskCache, aFileMetaPersistentCacheimplementation backed byBlockFileCache.file_cache_path; no new file-cache queue or standalone storage is added.enable_external_file_meta_disk_cache(defaulttrue) andexternal_file_meta_disk_cache_max_entry_bytes(default 64 MiB).Release note
Add a default-enabled disk-backed cache for external Parquet/ORC file metadata. The cache stores eligible serialized footer metadata in the existing local file cache INDEX queue and is controlled by
enable_external_file_meta_disk_cacheandexternal_file_meta_disk_cache_max_entry_bytes.Check List (For Author)
git diff --checkgit diff --cached --check./run-be-ut.sh --run --filter='*FileMeta*' -j 2./build.sh --be -j 4output/FE/BE: Parquet local TVF returnedcount(*) = 25twice; the second query profile showedFileFooterHitMemoryCache = 1andScanBytesdropped from36.65 KBto156 B.output/FE/BE: ORC local TVF returnedcount(*) = 25twice; the second query profile showedFileFooterHitMemoryCache = 1andScanBytesdropped from16.48 KBto488 B.modification_time, so persistent disk cache is intentionally skipped on that path. The persistent disk behavior is covered byFileMetaCacheDiskTestandNewParquetReaderTestin the*FileMeta*BE UT run.enable_external_file_meta_disk_cache,external_file_meta_disk_cache_max_entry_bytes, and that persisted metadata reuses the existing INDEX file cache queue.