branch-4.0: [fix](packed-file) Fix packed file cache cleanup issue (#59892) #60052
Conversation
…pache#59892) Previously, file cache used packed file path as cache key, which caused cache entries to be orphaned when stale rowsets were cleaned up (cleanup uses segment path as key). This fix moves the cache layer from inner reader to PackedFileReader wrapper level, ensuring: 1. Cache key = hash(segment_path.filename()) - matches cleanup key 2. Cache size = segment size - correct boundary 3. Each segment has independent cache entry - no interference
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
There was a problem hiding this comment.
Pull request overview
This is a cherry-pick from PR #59892 that fixes a packed file cache cleanup issue. The problem was that file cache used the packed file path as the cache key, causing cache entries to become orphaned when stale rowsets were cleaned up (which uses segment path as the key).
Changes:
- Restructured cache layer to wrap PackedFileReader with CachedRemoteFileReader instead of the reverse
- Added async cache write functionality for small files using segment paths as cache keys
- Updated tests to verify the new cache layer structure
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| be/src/io/fs/packed_file_system.cpp | Reordered cache wrapping to use segment path as cache key and disabled cache for inner packed file reader |
| be/src/io/fs/packed_file_manager.cpp | Added async small file cache write functionality with proper cache key generation and disabled write cache for packed files |
| be/test/io/fs/packed_file_concurrency_test.cpp | Updated test to use unique file names and verify new cache wrapper structure |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
skip buildall |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
4 participants
Cherry-picked from #59892