[fix](packed-file) Fix packed file cache cleanup issue #59892
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
There was a problem hiding this comment.
Pull request overview
This PR fixes a packed file cache cleanup issue by moving the cache layer from the inner reader to the PackedFileReader wrapper level. Previously, file cache used the packed file path as the cache key, which caused cache entries to become orphaned when stale rowsets were cleaned up (cleanup uses segment path as key).
Changes:
- Modified cache handling in
PackedFileSystem::open_file_impl()to disable caching at the inner reader level and wrapPackedFileReaderwithCachedRemoteFileReaderusing segment path and size - Added asynchronous cache write functions (
do_write_to_file_cacheandwrite_small_file_to_cache_async) to write small files to cache during packed file creation - Changed packed file writer options to disable
write_file_cacheflag, as individual small files are now cached separately - Updated logging from
LOG(INFO)toVLOG_DEBUGfor less verbose output
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| be/src/io/fs/packed_file_system.cpp | Moved cache layer to wrap PackedFileReader instead of inner reader; ensures cache key uses segment path and cache size uses segment size |
| be/src/io/fs/packed_file_manager.cpp | Added async cache writing for small files during packed file creation; disabled write_file_cache for packed files themselves |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
liaoxin01
force-pushed
the
fix/packed-file-cache-cleanup
branch
from
20954cd to
96a95e4
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 30806 ms |
TPC-DS: Total hot run time: 172662 ms |
ClickBench: Total hot run time: 26.64 s |
Contributor
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. 🤖 Generated with Claude Code |
Previously, file cache used packed file path as cache key, which caused cache entries to be orphaned when stale rowsets were cleaned up (cleanup uses segment path as key). This fix moves the cache layer from inner reader to PackedFileReader wrapper level, ensuring: 1. Cache key = hash(segment_path.filename()) - matches cleanup key 2. Cache size = segment size - correct boundary 3. Each segment has independent cache entry - no interference
dataroaring
force-pushed
the
fix/packed-file-cache-cleanup
branch
from
96a95e4 to
2cdcfe2
Compare
Contributor
|
run buildall |
TPC-H: Total hot run time: 31129 ms |
TPC-DS: Total hot run time: 174229 ms |
ClickBench: Total hot run time: 26.97 s |
gavinchou
reviewed
Jan 15, 2026
gavinchou
reviewed
Jan 15, 2026
gavinchou
reviewed
Jan 15, 2026
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32236 ms |
TPC-DS: Total hot run time: 178301 ms |
ClickBench: Total hot run time: 27.3 s |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31795 ms |
TPC-DS: Total hot run time: 174472 ms |
ClickBench: Total hot run time: 26.67 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
luwei16
approved these changes
Jan 17, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
approved
Contributor
|
PR approved by anyone and no changes requested. |
gavinchou
approved these changes
Jan 19, 2026
github-actions bot
pushed a commit
that referenced
this pull request
Jan 19, 2026
Previously, file cache used packed file path as cache key, which caused cache entries to be orphaned when stale rowsets were cleaned up (cleanup uses segment path as key). This fix moves the cache layer from inner reader to PackedFileReader wrapper level, ensuring: 1. Cache key = hash(segment_path.filename()) - matches cleanup key 2. Cache size = segment size - correct boundary 3. Each segment has independent cache entry - no interference
liaoxin01
added a commit
to liaoxin01/doris
that referenced
this pull request
Jan 20, 2026
…pache#59892) Previously, file cache used packed file path as cache key, which caused cache entries to be orphaned when stale rowsets were cleaned up (cleanup uses segment path as key). This fix moves the cache layer from inner reader to PackedFileReader wrapper level, ensuring: 1. Cache key = hash(segment_path.filename()) - matches cleanup key 2. Cache size = segment size - correct boundary 3. Each segment has independent cache entry - no interference
yiguolei
pushed a commit
that referenced
this pull request
Jan 20, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Previously, file cache used packed file path as cache key, which caused cache entries to be orphaned when stale rowsets were cleaned up (cleanup uses segment path as key).
This fix moves the cache layer from inner reader to PackedFileReader wrapper level, ensuring:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)